I have a biased coin whose bias I don't know. I know a fair coin has $p=0.5$ but I don't know if that's the case of my coin. I try and estimate the coin's real $p$ value by running a sample series of $N$ tosses. I get $k$ heads.
At this point two questions arise
How can I estimate a confidence interval for the true mean?
Should I run multiple series of $N$ tosses, collect each $p$ value, and then use those values to calculate a sample mean and standard error? Or is it enough that I have one $p$ value as $k/N$? Should I then estimate the confidence interval as $k/N \pm Z*SE_\frac{k}{N}$? What would be a correct value of $Z$ in this case for a, say, 95% interval? The 1.96 normal? What is the $SE_\frac{k}{N}$?
How can I numerically determine the probability of each number in this interval as being the true $p$ of my coin?
So what is the probability that the $k/N$ number is the true $p$ of this coin? What is the probability of $k/N+0.01$? Probability of $k/N-0.01$? This is basically the question in the title. What does this function look like? Does it even exist? Is my question number 1 relevant in any way?
The method we describe below is basic, and can be improved upon substantially, particularly for $p$ that are fairly close to $0$ or $1$.
Imagine tossing the coin $n$ times. Then the number $X$ of heads has binomial distribution. The mean of $X$ is $np$, and the variance of $X$ is $np(1-p)$.
Let $Y=\frac{X}{n}$. So $Y$ is the sample proportion. It too is a random variable, different experients will (usually) yields different numbers of heads.
The random variable $Y$ has mean $p$ and variance $\frac{1}{n^2}(np(1-p)=\frac{1}{np(1-p)}$. Thus $Y$ has standard deviation $\frac{1}{\sqrt{np(1-p)}}$.
If $n$ is large, and $p$ is not close to $0$ or $1$, then the distribution of $X$, and hence of $Y$, is closely approximated by the normal distribution.
So for suitable $n$ and $p$, we have $$\Pr\left(\left|\frac{X}{n} -p\right| \le \frac{1.96}{\sqrt{np(1-p)}}\right)\approx 0.95.$$ We can rewrite this by saying that with probability $0.95$, we have $$ \frac{X}{n} - \frac{1.96}{\sqrt{np(1-p)}}\le p \le \frac{X}{n}+ \frac{1.96}{\sqrt{np(1-p)}}.\tag{1}$$
The inequalities (1) are nice, but suffer from a serious problem: we don't know $p$, and the bounds for $p$ involve $p$ itself.
There are several ways around this problem. One is to note that $t(1-t)$ reaches a maximum value in the interval $(0,1)$ at $t=1/2$, where its value is $1/4$. And importantly, if $t$ is not far from $1/2$, then $t(1-t)$ is not far from $1/4$. Thus the variance of $\frac{X}{n}$ is biggest if $p=1/2$. It then follows from (1) that $$ \frac{X}{n} - \frac{1.96}{\sqrt{n/4}}\le p \le \frac{X}{n}+ \frac{1.96}{\sqrt{n/4}}.\tag{2}$$
The inequalities (2) give a useful $95\%$ confidence interval for $p$. It is traditional, though slightly confusing, to use the notation $\hat{p}$ for $\frac{X}{n}$.
We can do somewhat better, and use the sample proportion $\hat{p}$ we got in our experiment to approximate $p$ in the expression for the standard deviation. Then we get the $95\%$ confidence interval $$\hat{p} - \frac{1.96}{\sqrt{n\hat{p}(1-\hat{p})}}\le p \le \hat{p}+ \frac{1.96}{\sqrt{n\hat{p}(1-\hat{p})}}.\tag{3}$$
Either (2) or (3) can be used as long as the sample size is reasonably large. The version (3) can give tighter confidence intervals for the same cost (sample size).
For quite small samples, or $p$ close to $0$ and $1$, the above techniques need to be modified.