It's all very fun to use differential equations to show that the path of a particle follows a parabola under constant acceleration, a circle when the angular momentum is just enough to allow for zero radial acceleration and an ellipse when it is in a general orbit. (still haven't done the hyperbola).
But a question that has always bugged me is why do these objects all follow conic sections under gravity? Intuitively, it seems like the inverse square law in polar coordinates is responsible. But an object accelerating under constant force still takes on a parabola. Is this related to the fact that angular momentum is constant under gravity due to zero torque? (Kepler's second law)
Here's my take on the question:
First I assume that by "orbit" you really mean the image of the orbit, because the parametrized orbit (as a function of time) itself contains more information describing your physical system, for instance its time derivative(velocity).
An important feature for planetary orbits is conservation of mechanical energy, so that its position also determines its kinetic energy, so in this case position is the minimal information you need to reconstruct your physical system.
Now conic sections also have convenient alternative descriptions involving "distances", for instance an ellipse is the set of points that has the same total distance to the two foci, and parabolas are the set of points that are equidistant from both the directrix and the focus. This provides us with the necessary tools to parametrize both potential energy and kenetic energy at the same time.
For example consider a celestial body orbiting a planet. Given their relative positions and the velocity of the celestial body, think of the planet as the first focus parametrizing potential energy, and a second focus parametrizing kinetic energy. This second focus will lie on the circle around the celestial body with radius its kinetic energy, and its position on this circle will be determined by the direction of velocity, so that the velocity is tangential to the resulting eclipse.
The same exercise can of course also be done for objects under a constant force.