Irregular Gears
This article is kind of long! I split it into three pages.
The Mathematics of Irregular Gears
I'm basing my approach to this problem on the humble epitrochoid. Unlike the point-based approach, here we are keeping one gear stationary and rolling the other around its surface.
For a standard epitrochoid, the paramaterized coordinates are found using a parameter \(t_1 \in \left[ -\pi, \pi \right]\). Don't be fooled by the limits of \(t_1\), it is NOT the polar angle of the resulting epitrochoid. Anyway: $$ \begin{array}{l} \displaystyle\phantom{y}x(t_1) = (R+r)\cos{(t_1)}-d\cos{\left(\frac{R+r}{r}t_1\right)}\\ \displaystyle\phantom{x}y(t_1) = (R+r)\sin{(t_1)}-d\sin{\left(\frac{R+r}{r}t_1\right)} \end{array} $$ where \(x\) and \(y\) are the cartesian coordinates with respect to \(t_1\); \(R\) and \(r\) are the radii of the fixed (central) circle and the rolling circle, respectively; and \(d\) is the distance from the center of the rolling circle to the point being traced.
Applying this to irregular gear design, we must consider that the distance \(d\) varies along the profile of the input gear. Unfortunately, just calculating the epitrochoid radius at each \(t_1\) will probably not give us a gear that meshes. We must also consider the profile of the input gear in nearby the contact point. To do this, we need to introduce a second parameter, \(t_2\), which will be some kind of "sweep angle" that allows us to get the \(x\) and \(y\) positions of any point on the input gear, instead of a single point per \(t_1\) value. This is shown below. $$ \begin{array}{l} \displaystyle\phantom{y}x(t_1,\,t_2) = (R+r)\cos{(t_1)}-d(t_2)\cos{\left(\frac{R+r}{r}t_1+t_2\right)}\\ \displaystyle\phantom{x}y(t_1,\,t_2) = (R+r)\sin{(t_1)}-d(t_2)\sin{\left(\frac{R+r}{r}t_1+t_2\right)}\\ \end{array} $$
I can smell that this is going to turn into a parameter optimization problem, so we're going to need at least a new function and a new variable. Let \(\theta\) be the polar angle of the output gear, and let \(G(\theta)\) be the radius of said output gear at that polar angle.
We want \(G(\theta)\) to be equal to the smallest radius returned by any combination of \(t_1\) and \(t_2\), provided the radius is reachable at angle \(\theta\). As an equation, this becomes $$ \begin{array}{l} \displaystyle G(\theta) = \min_{\substack{t_1 \in \left[-\pi,\,\pi\right]\\ t_2 \in \left[-\pi,\, \pi\right]}}{\left\{ x(t_1,\,t_2)^2 + y(t_1,\, t_2)^2 \right\}}\\ \displaystyle\operatorname{subject\,to:}\,\tan^{-1}{\left(\frac{y(t_1,\,t_2)}{x(t_1,\,t_2)}\right)}=\theta. \end{array} $$
After some trigonometric identities, the equations above give the following. $$ \begin{array}{l} \displaystyle G(\theta) = \min_{\substack{t_1 \in \left[-\pi,\,\pi\right]\\ t_2 \in \left[-\pi,\, \pi\right]}} \left\{ \left(d(t_2)\right)^2 -2d(t_2)(R+r)\cos{\left( \frac{R}{r}t_1+t_2 \right)} \right\}\\ \displaystyle\operatorname{subject\,to:}\,\tan^{-1}\left(\frac {(R+r)\sin{(t_1)}-d(t_2)\sin{\left( \frac{R+r}{r}t_1+t_2 \right)}} {(R+r)\cos{(t_1)}-d(t_2)\cos{\left( \frac{R+r}{r}t_1+t_2 \right)}} \right)-\theta = 0. \end{array} $$
It's hard to go further without knowing the actual function \(d\) to be used. But at least now we've got something to plop into MATLAB's optimization function fmincon. You know. If you were so inclined. But that's another post for another time.