Often in mathematics one constructs a set of some sort, let's name it $A$. We've constructed it in an abstract way, so, a priori, structural aspects of $A$ are yet unknown to us, until we prove them. Lets say that, after some effort, we've proven that $A$ in fact has a set of properties $P$. If we happen to have previously studied, in general, the implications of the properties $P$, then we can now apply everything we studied about $P$ to $A$, and we all live happily ever after. It generally works this way, and people are considered satisfied with applying what we know about $P$ in $A$ if and only if we have proven that we can. Sometimes we can't, but in these cases there's an massive collective effort to do so.
In physics, is this the case? I'm very, very interested in finding out if anyone has made efforts to justify why we use mathematics to study the world outside of our minds. Because in doing so, we are treating the world as if it were $A$ in the example, and we are applying $P$, but without justification. Most likely there isn't such justification, because it'll all bubble down to some parmenides-style paradox about what-is-not, maybe... but I still would like to know about attempts, whether historical or modern.
Any books on the matter? online pdf's? I won't consider it an invalid answer if someone recommends some philosopher or some other, or literarian, or who-be-it.
To the extent I understand your question (I have to protect myself against those waters of accusation too!), it made me think of the following classic:
Mathematics and Plausible Reasoning, by G. Polya. Volume 1: Patterns of Plausible Inference; Volume II: Induction and Analogy in Mathematics.
Polya develops a structure around what he calls undemonstrable inferences. So for example, instead of $A \implies B$ and $\neg B$ together imply $\neg A$, you might have $A \implies B$ and $B$ together imply A more credible.