Consider the set $K=\{x\in \mathbb{R}^4: Ax=a,~x_i\geq 0\}$ with $$A=\begin{pmatrix}1&-1&-1&2\\ \:\:\:\:0&1&0&3\\ \:\:\:\:2&0&-2&10\end{pmatrix}\quad a=\begin{pmatrix}2\\ 7\\ 18\end{pmatrix}$$ Which of the vectors is a vertex of the set?
$$x_1=\begin{pmatrix}10\\ \:7\\ \:0\\ \:0\end{pmatrix},~x_2=\begin{pmatrix}0\\ \:\:1\\ \:\:1\\ \:\:2\end{pmatrix},~x_3=\begin{pmatrix}1\\ \:\:\:1\\ \:\:\:2\\ \:\:\:2\end{pmatrix},~ x_4=\begin{pmatrix}9\\ \:\:\:7\\ \:\:\:\:0\\ \:\:\:\:0\end{pmatrix}$$
So I checked that $Ax_2=a$, $Ax_3=a$ and $Ax_4=a$. But $Ax_1=\begin{pmatrix}3\\ 7\\ 20\end{pmatrix}$ so $x_1$ is not in $K$. How do I check if the other points are vertices?
Vertex; $$G\subseteq \mathbb{R}^n~\text{convex}, z\in G ~\text{vertex} \iff~x,y\in G: z=\frac{x+y}{2} \Rightarrow x=y=z$$
First, consider the vector $x_2$ . . .
Let's attempt to represent $x_2$ as the midpoint of two distinct points of $K$.
Thus, suppose $x_2\pm v\in K$, for some nonzero $v\in\mathbb{R}^4$.
Equivalently, $v\in\mathbb{R}^4$ is such that $$ \begin{cases} Av=0\\[4pt] x_2\pm v \ge 0\\ \end{cases} $$ Since $x_2[1]=0$, the conditions $x_2\pm v \ge 0$ force $v[1]=0$, hence $$v=\begin{pmatrix}0\\q\\r\\s\end{pmatrix}$$ for some $q,r,s\in\mathbb{R}$, not all zero.
Expressing the condition $Av=0$ as a system of equations, we get $$ \begin{cases} q+3s=0\\[4pt] -q-r+2s=0\\[4pt] -r+5s = 0\\[4pt] \end{cases} $$ which has the general solution \begin{cases} q=-3s=0\\[4pt] r=5s \end{cases} where $s\in\mathbb{R}$ is arbitrary.
If $s$ is sufficiently small, but not zero (e.g., $s={\large{\frac{1}{10}}}$), we get $x_2\pm v\ge 0$.
It follows that $x_2$ is not a vertex of $K$.
Next, consider the vector $x_3$ . . .
Let's attempt to represent $x_3$ as the midpoint of two distinct points of $K$.
Thus, suppose $x_3\pm v\in K$, for some nonzero $v\in\mathbb{R}^4$.
Equivalently, $v\in\mathbb{R}^4$ is such that $$ \begin{cases} Av=0\\[4pt] x_3\pm v \ge 0\\ \end{cases} $$ Write $$v=\begin{pmatrix}p\\q\\r\\s\end{pmatrix}$$ for some $p,q,r,s\in\mathbb{R}$, not all zero.
Expressing the condition $Av=0$ as a system of equations, we get $$ \begin{cases} p-q-r+2s= 0\\[4pt] p-r+5s=0\\[4pt] q+3s=0\\ \end{cases} $$ which has the general solution \begin{cases} r=p+5s\\[4pt] q=-3s\\[4pt] \end{cases} where $p,s\in\mathbb{R}$ are arbitrary.
If $p,s$ are sufficiently small, and $s,p+5s$ are not both zero (e.g., $p=s={\large{\frac{1}{10}}}$), we get $x_3\pm v\ge 0$.
It follows that $x_3$ is not a vertex of $K$.
Finally, consider the vector $x_4$ . . .
Let's attempt to represent $x_4$ as the midpoint of two distinct points of $K$.
Thus, suppose $x_4\pm v\in K$, for some nonzero $v\in\mathbb{R}^4$.
Equivalently, $v\in\mathbb{R}^4$ is such that $$ \begin{cases} Av=0\\[4pt] x_4\pm v \ge 0\\ \end{cases} $$ Since $x_4[3]=x_4[4]=0$, the conditions $x_4\pm v \ge 0$ force $v[3]=v[4]=0$, hence $$v=\begin{pmatrix}p\\q\\0\\0\end{pmatrix}$$ for some $p,q\in\mathbb{R}$, not both zero.
Expressing the condition $Av=0$ as a system of equations, we get $$ \begin{cases} q=0\\[4pt] p-q=0\\[4pt] p= 0\\ \end{cases} $$ which has no solutions other than $p=q=0$.
It follows that $x_4$ is a vertex of $K$.