From Serge Lang's Linear Algebra, I've been just introduced to the concept of index of nullity and Sylvester's theorem based on non positive-definite scalar products:
Let $V$ be a finite dimensional vector space over $\mathbb{R}$, with a scalar product. Assume $\textrm{dim} \, V > 0$. Let $V_0$ be the subspace of $V$ consisting of all vectors $v \in V$ such that $\langle{v}, w \rangle = 0$ for all $w \in V$. Let $\{v_1, ... , v_n\}$ be an orthogonal basis for $V$. Then the number of integers $i$ such that $\langle{v_i} , v_i \rangle$ is equal to the dimension of $V_0$.
The proof is fairly simple, suppose $\{v_1, ... , v_n\}$ is ordered so that:
$\langle v_1, v_1 \rangle \neq 0, ... ,\langle v_s, v_s \rangle \neq 0$ but $\langle v_i, v_i \rangle = 0$ for all $i > s$.
Considering that $\{v_1, ... , v_n\}$ is orthogonal basis, it is obvious that $\{v_{s+1}, ... , v_n\}$ lies in $V_0$. Any element $v \in V_0$ can be thus written as:
$$v = x_1v_1 + ... + x_sv_s + ... + x_nv_n$$
with $x_i \in X \in \mathbb{R}^n$. Taking scalar product of $v$ with any $v_j$ such that $j \leq s$, it can be seen by bilinearity that:
$$0=\langle v, v_{j} \rangle = x_j \langle v_j, v_j \rangle$$
Considering that $\langle v_j, v_j \rangle \neq 0$, by trivial factor rule $x_j = 0$. Hence $\{v_{s+1}, ... , v_n\}$ forms an orthogonal basis for $V_0$.
Contradiction by orthogonal complement:
I've studied much before a concept of orthogonal complement in positive-definite cases, such that:
$$\textrm{dim} \, W + \textrm{dim} \, W^{\perp} = \textrm{dim} \, V$$
if $W$ is a subspace of $V$ and $W^{\perp}$ is its orthogonal complement.
But in this case, $V_0$ is an orthogonal complement of $V$, and thus:
$$\textrm{dim} \, V + \textrm{dim} \, V^{\perp} = \textrm{dim} \, V$$ $$\textrm{dim} \, V^{\perp} = \textrm{dim} \, V - \textrm{dim} \, V$$ $$\textrm{dim} \, V^{\perp} = 0$$
Thus this contradicts the proof above, because instead of $\{v_{s+1}, ... ,v_n\}$, basis of trivial vector space must be $\{0\}$.
Am I missing something? The note on index of nullity does not mention whether or not scalar product is positive-definite. Perhaps basis of $V_0$ is $\{0\}$ iff $V$ has positive-definite scalar product?
Thank you!
By definition $V_0^\perp = V$, so $\dim V_0 + \dim V_0^\perp = \dim V_0 + \dim W$. In particular, if $V_0 \neq \{ 0 \}$ then $\dim V_0 + \dim V_0^{\perp} > \dim W$, and so the formula you recall from the case of a positive definite scalar product does not apply.
Remark Any scalar product $\langle\,\cdot\,,\,\cdot\,\rangle$ on $V$ determines a nondegenerate scalar product $\langle\!\langle\,\cdot\,,\,\cdot\,\rangle\!\rangle$ on $V / V_0$ by setting $\langle\!\langle v + V_0, w + V_0 \rangle\!\rangle = \langle v, w \rangle$. Then, replacing $V$ and $W$ in the identity $\dim W + \dim W^\perp = \dim V$ (which applies just as well to general nondegenerate scalar products as to positive definite ones) respectively with $V / V_0$ and $W / (W \cap V_0)$ gives the identity $$\dim W + \dim W^\perp = \dim V + \dim (W \cap V_0) $$ which holds even for degenerate scalar products on $V$.