This is what I have
Let $\phi$ be a symmetric bilinear form on $E$, and $F$ a vector subspace of $E$. Given $n=\dim E$, prove that $$\dim F^\perp = n - \dim F + \dim F \cap \ker \phi$$ where $F^\perp = \{ x \in E: \forall y \in F, \phi (y,x) = 0 \}$ is the orthogonal complement of $F$.
I tried to use the linear transformation $$ \begin{alignat}{2} \Psi\ :\ &E \to F^* && \\ &y \mapsto \phi (y,\cdot) \end{alignat} $$ where $F^*$ is the dual space of $F$, so that I can use the fact that $$n = \dim E = \dim\ker\Psi + \dim\operatorname{im}\Psi = \dim F^\perp + \dim\operatorname{im}\Psi$$ using the Rank–nullity theorem, but I got stuck here. How do I complete the proof?
In addition to what you have, we'll also apply the rank nullity theorem to $\Psi^*:F\to E^*$ and we'll use the fact that $\dim \text{im}(\Psi)=\dim \text{im}(\Psi^*)$.
Note that, by symmetry, $\Psi^*:F\to E^*$ is given by $\Psi^*(f)(e)=\phi(e,f)$.
Let $r=\dim\text{im}(\Psi)$ and fix $e_1, \ldots, e_r\in E$ such that $\Psi(e_i)$, $i=1, \ldots, r$ is a basis for $\text{im}(\Psi)$. Necessarily $e_1, \ldots, e_r$ is a linearly independent set.
Choose a linearly independent set $f_1, \ldots, f_r$ in $F$ such that $$\Psi(e_i)(f_j)=\left\{\begin{array}{ll} 1 & : i=j \\ 0 & : i\neq j.\end{array}\right.$$
Let $e_{r+1}, \ldots, e_n$ be a basis for $\ker(\Psi)$ and note that $e_1, \ldots, e_n$ is a basis for $E$ (this is in the proof of the rank nullity theorem). Extend $f_1, \ldots, f_r$ to a basis $f_1, \ldots, f_d$ for $F$ such that $\Phi(e_i)(f_j)=0$ for all $1\leqslant i\leqslant r$ and $r<j\leqslant d$. We can do this by first extending $f_1, \ldots, f_r$ to a basis $f_1, \ldots, f_r, f_{r+1}', \ldots, f_d'$ for $F$ and then replacing $$f_j=f_j'-\sum_{i=1}^r \Psi(e_i)(f_j')f_i$$ for each $r<j\leqslant d$.
I claim that $\dim\text{im}(\Psi^*)=r=\dim \text{im}(\Psi)$. For this we need to show that $\Psi^*(f_j)$, $1\leqslant j\leqslant r$, is a basis for $\text{im}(\Psi^*)$.
We first show linear independence of $\Psi^*(f_j)$, $1\leqslant j \leqslant r$. Suppose that $\Psi^*(\sum_{j=1}^r a_jf_j)=\sum_{j=1}^r a_j\Psi^*(f_j)=0$. Then for each $1\leqslant i\leqslant r$, $$0=\sum_{j=1}^r a_j\Psi^*(f_j)(e_i)=\sum_{j=1}^r a_j\phi(e_i,f_j)\sum_{j=1}^r a_j\Psi(e_i)(f_j)=a_i.$$ So $a_1=\ldots = a_r=0$.
Next, next note that $\Psi^*(f_j)=0$ for all $r<j\leqslant d$. To see this, fix $1\leqslant i\leqslant d$. If $r<i\leqslant d$, then $e_i\in \text{ker}(\Psi)$, and $$\Psi^*(f_j)(e_i)=\phi(e_i,f_j)=\Psi(e_i)(f_j)=0.$$ If $1\leqslant i\leqslant r$, $$\Psi^*(f_j)(e_i)=\phi(e_i,f_j)=\Psi(e_i)(f_j)=0$$ by our choice of $f_j$, $r<j\leqslant d$. Therefore for all $r<j\leqslant d$ and $e=\sum_{i=1}^n b_ie_i\in E$, $$\Psi^*(f_j)(e)=\sum_{i=1}^n b_i\Psi^*(f_j)(e_i)=\sum_{i=1}^n b_i\cdot 0=0,$$ and $\Phi^*(f_j)=0$.
Now for any $f=\sum_{j=1}^d c_jf_j$, $$\Psi^*(f)=\sum_{j=1}^r c_j\Psi^*(f_j)+\sum_{j=r+1}^d c_j\cdot 0=\sum_{j=1}^r c_j\Psi^*(f_j).$$ Therefore $\Psi^*(f_j)$, $1\leqslant j\leqslant r$ is a spanning set for $\text{im}(\Psi^*)$. Since this set is linearly independent, $\dim \text{im}(\Psi^*)=r=\dim \text{im}(\Psi)$.
Therefore we have shown that $\dim \text{im}(\Psi)=\dim\text{im}(\Psi^*)$. Apply the rank nullity theorem to $\Psi^*$ to deduce that $$\dim F=\dim \ker(\Psi^*)+\dim\text{im}(\Psi^*)=\dim (F\cap \ker(\phi))+\dim \text{im}(\Psi).$$ Here we used the fact that $\ker \Psi^*=F\cap \ker(\phi)$ and $\dim \text{im}(\Psi^*)=\dim \text{im}(\Psi)$.
Now plug in $\dim F=\dim (F\cap \ker(\phi))+\dim \text{im}(\Psi)$ into what you have: $$\dim F^\perp = n- \dim \text{im}(\Psi)=n-(\dim F-\dim (F\cap \ker(\phi)))=n-\dim F + \dim (F\cap \ker(\phi)).$$