After taking a self-learning course in Probability, I wanted to learn Statistics by myself. I thought one of the best practical ways I could learn if I take a coin and apply statistics to figure out the probability of getting a head. With this, I ask if you could outline the steps I should follow to obtain a number say P(H) is 0.3 or 0.8 etc. It would be useful to mention related topics/theories so that I can read them before I take up this practical experiment.
I was thinking something like this -- repeat tossing for some N times and then use frequency to come to a number for P(H). However, we might need some further theory to decide the exact value of N for which the result would be within say 95% confidence level.
I'd say that, by the definition of a coin being biased, you're assigning each outcome a certain probability. We denote the probability of (the union of) all possible events to be 1, and then we pick a certain event out of all the possibilities. The union of all possible events is called the sample space (denote it $\mathcal{S}$, for instance). It contains all possible outcomes of your "experiment".
In this case, we have two possiblities: heads or tails. So we can write the sample space as: $\mathcal{S} = \{H, \; T\}$, where $H$ means "heads" and $T$ means "tails".
Now, the probability that anything within a sample space occurs is 1. We write this as:
$$\operatorname{P} (\mathcal{S}) = 1$$
In this case, we can also write $\mathcal{S}$ as a union of the two events that compose it:
$$\mathcal{S} = H \cup T$$
So:
$$\operatorname{P} (\mathcal{S}) = \operatorname{P} (H \cup T) = 1$$
These two elements are disjoint, which means that they can't happen at the same time. This means that the chances of $H$ or $T$ happening are the same as the chances of $H$ happening plus the chances of $T$ happening. This is a bit of a difficult concept to wrap around your head, especially in more complex cases. Hence,
$$\operatorname{P} (H \cup T) = \operatorname{P} (H) + \operatorname{P} (T) = 1$$
For an unbiased coin, $\operatorname{P} (H) = \operatorname{P} (T)$, which would make:
$$\operatorname{P} (H) + \operatorname{P} (T) = \operatorname{P} (H) + \operatorname{P} (H) = 2\operatorname{P} (H) = 1$$
$$\Rightarrow \operatorname{P} (H) = 0.5$$
However, for a biased coin, by definition $\operatorname{P} (H) \not = \operatorname{P} (T)$, which means that:
$$\operatorname{P} (H) = 1 - \operatorname{P} (T)$$
I hope this clarifies your understanding of the matter.
EDIT:
Now I finally understand what you mean. What you need in this case is testing, or proving, with a certain confidence, that the coin is biased or not.
We start with the alternative hypothesis: "The coin is biased", and then the null hypothesis is the opposite. To show that the coin is biased, we will try to show (with a certain confidence) that it is not unbiased. Hence, we're trying to disprove the opposite of the alternative hypothesis, the null hypothesis: "The coin is not biased."
$$H_a: \text{The coin is biased.}$$ $$H_0: \text{The coin is not biased.}$$
To "disprove" $H_0$ we calculate the probability of the actual outcome we observed, assuming that the coin is unbiased. Obviously, you'd require some observations. I'll provide some as an example.
So suppose we observed that, out of 100 coin tosses, 57 were heads and 43 were tails. Now, we assume that each toss is independent and that the probability of each toss does not change throughout the experiment. These are the characteristics of a Binomial distribution.
Now we can see that $H_0$ is equivalent to saying that $X$, the number of observed heads, is distributed $\sim \operatorname{BIN}(100, p = 0.5)$
So, because by our question of whether the coin is biased, we're asking whether heads has a higher chance of occurring than tails, we calculate
$$\begin{align} \operatorname{P}(X \ge 57) & = \operatorname{P}(X = 57) + \operatorname{P}(X = 58) + \cdots + \operatorname{P}(X = 100) \\ & = \binom{100}{57} (0.5)^{57} (0.5)^{43} + \binom{100}{58} (0.5)^{58} (0.5)^{42} + \cdots + \binom{100}{100} (0.5)^100 (0.5)^0 \end{align}$$
Typically, we'd approximate this probability with the Normal distribution, but modern computers can handle calculating that (you can check the sum here). So
$$\operatorname{P}(X \ge 57) \approx 0.09667$$
Now, depending on your confidence interval, you go on to concluding whether you can reject $H_0$ and therefore accept $H_a$ or whether you cannot reject $H_0$ and therefore cannot accept $H_a$.
A popular choice for a confidence interval is 0.05, but there is a lot of debate on whether it should be that popular or not (some studies show that there is a lot of space for the so-called p-value hacking for $p = 0.05$, but this is beyond the question). For some tests you'll need a much smaller confidence interval (aka it is more difficult to reject $H_0$; i.e. medical illness detection procedures), while for others you can be much more lenient (i.e. tossing a coin).
In this case if your confidence level is at 0.10, because 0.09667 < 0.10, you can reject $H_0$ and accept $H_a$, but if it were at 0.05, you wouldn't be able to do this.
Basically, you're calculating the probability of the outcome you observe if the distribution is binomial with $n = 100$ and $p = 0.5$. If this probability is high, it means that it is likely that the distribution is appropriate for what you are measuring, and therefore the coin is not biased as $\operatorname{P}(H) = \operatorname{P}(T) = p = 0.5$.
However, if such probability is low, it means that it is not likely that what you're observing follows the binomial distribution. You can then go on to conjecture why this is: maybe the coin is biased ( $\operatorname{P}(H) \not = \operatorname{P}(T)$ ), maybe the throws are not independent, maybe the probability changes from throw to throw...
And you're done.