Why is $\Delta = b^2 - 4ac$ so special that it deserves a name of its own ? I saw it only once in quadratic formula and nowhere else.
Does higher polynomials also have discriminant ? if so then is there a general formula for it in terms of coefficients ?
It was mentioned already that the discriminant of a polynomial is zero if and only if the polynomial has multiple roots (i.e. it has some factor of $(x-a)^n$ for $n>1$).
The discriminant is defined for higher polynomials (there is a formula on wikipedia in terms of the polynomial and its derivative). Perhaps an easier to explain definition is as follows: given a polynomial $$p(x)=\prod_{i=1}^d(x-\alpha_i)$$ (recall from the fundamental theorem of algebra than any polynomial with real coefficients [and leading coefficient 1] can be written in this form, with $\alpha_i$ complex). Then we say $$\text{disc}(p(x))=\prod_{i<j}(\alpha_i-\alpha_j)^2$$ where the product is taken over all pairs $(i,j)$ with $1\le i<j\le d$. (Simple exercise - verify this definition agrees with the discrimiant in the quadratic formula.)
In abstract algebra (particularly in Galois Theory), we can do a bit more with the discriminant than just check that no roots are the same. Consider a polynomial with rational coefficients, $$p(x)=\sum_{i=0}^d a_ix^i=a_dx^d+\dots+a_1x+a_0$$ for some $a_i\in\mathbb{Q}$. In some cases, we can factor such polynomials into linear factors with coefficients in $\mathbb{Q}$: $x^2-2x+1=(x-1)^2$. In other cases, we cannot: $x^2+1=(x+i)(x-i)$. In this case, the polynomial factors into linear polynomials with coefficients in $\mathbb{C}$. Both $\mathbb{C}$ and $\mathbb{Q}$ are sets called fields - essentially they are sets where you can do multiplication, division (except by $0$), addition, and subtraction. There are many other examples of fields, for instance $\mathbb{R}$. Notice that $\mathbb{Q}\subset\mathbb{R}\subset\mathbb{C}$. We could add a lot of other fields into this chain of inclusions. There are also fields between $\mathbb{Q}$ and $\mathbb{C}$ that do not include $\mathbb{R}$ or have $\mathbb{R}$ as a subset. One of the fundamental questions of Galois theory is: "What is the smallest field over which we can write a polynomial as a product of linear factors?" This smallest field is called the splitting field. It turns out that the discriminant of a polynomial can be used to compute some interesting properties of splitting fields of polynomials.
As a quick note, the splitting field of $x^2+1$ viewed as a polynomial with rational coefficients is the set $\{a+bi:a,b\in\mathbb{Q}\}\subset\mathbb{C}$. This is usually denoted by $\mathbb{Q}(i)$ (or $\mathbb{Q}[i]$ if we are not emphasizing the fact that it is a field). Also, notice that $\mathbb{R}\not\subset\mathbb{Q}(i)$ and $\mathbb{Q}(i)\not\subset\mathbb{R}$.
I am not sure what your background is, so I will leave it at this for now, but feel free to ask if you want me to expand on more things. Also, I am by no means an expert. Hopefully someone with more experience could talk more about what discriminants are useful for.