Undistorting night sky star motion (given that stars don't move)

Question

My ultimate goal: out of 500 night-sky photos, I'd like to distort 499 of them to the exact star position of the first one. (Imagine a fancy motorized star-tracking rig, but done with math instead of motors)

But I don't know my camera's distortion, so is there a way to brute-force solve this?

What I've got to work with:

1. 500 2-second nighttime photos over a several hour range.
2. Perfectly stationary camera (normal field of view)
3. Extracted star locations as x,y pixel coordinates
4. Star correlation, for the ~15 brightest stars I know: frameTimeLocation(time, bright_star_number) = (x,y)
5. Exact time each of the 500 photos were taken
6. The knowledge that stars don't move relative to each other
7. The knowledge that stars are rotating around a fixed point (which may be outside the camera's FOV)
8. The assumption that the distortion of the camera is smooth

Assuming everything above, I could either

1. Solve the camera distortion equation
2. Solve some general distortion field matrix thing

The problem is that I'm not sure how to start with either. I'm very much a programmer and not as much a mathematician, and eventually I need to end up with pixelValuesAcrossAllPhotosMappedToTime0(targetx, targety) = 500 pixels of data.