- There are five possible answers to a multiple-choice question. Given that the student does not know the answer, what is the probability that the student chooses the first answer?
That is not a well formed math problem, to say the least, but I once told a student that it is not actually a math problem and he couldn't understand why, and thought it was obviously a math problem.
But I wonder if our understanding of what constitutes a mathematics problem should be expanded somewhat beyond what mathematicians conventionally think it is, in order to include instances of reasoning that can be done by mathematicians and is unknown to others, to the extent to which those others are not mathematicians.
Obviously this calls for examples, and at this moment I have only two:
Clearly Regiomontanus's angle maximization problem is a mathematics problem. One posits a painting hanging on a wall with its lowest and highest edges above your eye level. Given those two heights above your eye level, how far from the wall should you stand to maximize the angle whose vertex is at your eye and whose rays are incident to the top and bottom of the painting? (This has a solution by elementary geometry not involving calculus, another solution by a somewhat unconventional way of completing the square, and another by calculus, in which perhaps the most efficient way is to directly maximize the tangent of the angle.) However, often people phrase this incorrectly: they ask what distance from the wall gives you the best view of the painting. So let us suppose that I announce the following "theorem":
The distance from the wall that maximizes the angle does not generally coincide with the distance that gives you the best view of the painting.
If this is a theorem then it has a proof. Here it is:
First suppose that the lower edge of the painting is exactly at your eye level. Then you would maximize the angle by placing your eyelid in contact with the lower edge of the painting. As you approach that point, the angle approaches $90^\circ,$ and cannot get more than that. But this is clearly not the best possible view. Next, suppose the lower edge is a tenth of a millimeter above your eye level. If the top of the painting were one meter above eye level, then the angle is maximized by making the distance one centimeter. Clearly still not the best view. The quality of the view varies continuously with the distance, and the angle also varies continuously with the distance, so the angle cannot suddenly coincide with the quality as you slowly back away. Quod erat demonstrandum.
Suppose cheese and chalk are two commodities that you are in the habit of buying. If I steal some of your cheese or some of your chalk, I leave you worse off than you were, and if you gain some of either your lot is improved. Now suppose if you have $20$ units of cheese and $30$ units of chalk, you are just as well off as if you have $50$ units of cheese and $20$ of chalk. If $f(x,y)$ is your utility of $x$ units of cheese and $y$ of chalk, then the level set of $f$ passing through the two points just described could have any of many shapes; this is an "indifference curve". Discovering these curves is an empirical matter, not a matter of mathematics, and economists have found that in realistic situations they may have any of many shapes. None of that is a mathematical theorem; rather it is an empirical finding. But now suppose if you invest your wealth in a certain way, then the probability distribution of the amounts of cheese and chalk you will have next February 30th is thus-and-so, but if you invest your wealth in another way, you have a different probability distribution. Suppose you are indifferent between those two probability distributions. Now we state a "theorem":
The indifference curve between your two equally valuable probability distributions must be a straight line.
Proof: Since you are indifferent between them, you don't care if I throw dice to decide which one you get. Therefore any weighted average of the two distributions has the same utility as either of them. Q.E.D.
Why should we regard these "theorems" and "proofs" as belonging within mathematics rather than being located in some other region of the intellectual realm? To which I reply: Because only an understanding of mathematics can make it possible to understand them.
This is not a position of which I am convinced, but I do think the foundations of the subject are not as well understood as many feel they are.
Questions: Do such examples bear upon the question of where the boundaries of the discipline are?
You seem to be asking two related questions: (1) are the boundaries of mathematics known? (2) have the foundations of mathematics been well understood?
Rather than answering these from first philosophical principles such as the realism/nominalism dichotomy, I would suggest that the answer lies in an examination of the history of mathematics. At the turn of the (19th) century mathematics was understood as arithmetized analysis as practiced by Weierstrass and his students. In 1900 in his famous lecture at the world congress of mathematicians in Paris, David Hilbert turned the tables on the traditional view by offering a broad scope of problems including set theory, logic, and axiomatisation of physics (only a minority of which relates to arithmetized analysis). This is of course not the only but perhaps one of the most famous instances when the boundaries of the subject have changed dramatically over time.
If the subject matter of mathematics evolves surely its foundations necessarily evolve as well, as is illustrated by the historical development of category theory starting in the 1950s, challenging the traditional set-theoretic foundations.
An important aspect of the picture is the fact that the boundaries of mathematics are not always growing; sometimes they are shrinking. Thus, infinitesimals were part and parcel of the landscape of mathematical analysis for about two centuries: 1670-1870. Sometime after 1870 a more-or-less official ban against them was issued (that was not uniformly followed however) that held for about 90 years. Around 1960 infinitesimals were restored to respectability though some still seek to enforce the obsolete ban.
Therefore my answer to both questions is negative.