Law of reflection when tangent is undefined

Question

The law of reflection also holds for non-plane mirrors, provided that the normal at any point on the mirror is understood to be the outward pointing normal to the local tangent plane of the mirror at that point

While the law of reflection is based in physics, interpreting it purely mathematically seems useful. Though this produces a problem for surfaces where the tangent is undefined. In real life, I don't think such a surface could exist. But in a world where surfaces with undefined tangents can exist, I have no idea what would happen.

What would happen if a light/billiard hit a point with an undefined tangent? Is it undefined?

e.g Some ray hitting smooth surface & corner

Answer 1

I am not sure how appropriate this is for the math section, but here it is.

I think an insightful approach is to use Huygens principle for refraction. Use this video and the many others that show rather than compute how things work in this model. It is not the ray-based model of optics that you mentioned, and it is not the best model that there is in physics, but for the purpose of this question I think it is good enough to get the right picture.

So if you look at a couple of animations with Huygens principle you will notice that the sources of light generate some wave-like things that propagate with a spherical symmetry around the source point. Put a bunch of these source points on a surface and let them generate their wave-like patterns and you get a superposition of multiple waves that travel through the medium in question. Now if those waves happen to "hit" an object, on each point on the surface of that object you will get a "secondary" source, in the sense that each point is treated as starting to generate waves itself. There are some observations to be made here, such as the phase difference between two secondary sources due to a difference in the distance between the main source and the two secondary sources, but what matters for now is that the surface of an object hit by waves becomes a secondary source and starts generating waves itself.

Now back to your question, in this model you can imagine that at the sharp tip of your mirror surface ideally you have only one point. If it is hit by the wave from a main source, it will become a secondary source and start generating a spherical wave. Since in the image you uploaded you have two surfaces that create that sharp edge, the main contribution to the superposition of all the secondary waves will be given by the two plane surfaces and a negligible amount from the tip.

So the take away is that given the more physical nature of the question, the answer is based on physics also, in the sense that you don't apply the ray model for this scenario, but a more general one namely the wave-based one. Sure, the two can be linked via approximations, but the idea of this answer was to give a direction and maybe a mental image of a possible solution to your question.

Answer 2

The law of reflection itself is undefined, so depending on the area of study, people choose whatever is most appropriate.

As an example, in Dynamical Billiards, it seems that it is convention to "terminate" or "stop" the light/billiard when it hits a point with an undefined tangent.

From "Geometry and Billiards" by Serge Tabachnikov

Of course, we assume that the reflection occurs at a smooth point of the boundary. For example, if the billiard ball hits a corner of the billiard table, the reflection is not defined and the motion of the ball terminates right there.

And from a paper on dynamic billiards:

The reflection rule is not defined at the corners of the boundary. The standard convention is to “stop the ball” when it reaches a corner.

In some fields such as microlocal analysis, this may not be satisfactory and more elaborate solutions may be required, or in physics where it may not be appropriate to consider light as balls (point-particles). So it seems dependent on the field of study.