I don't know if there is a certain name for this matrix. but I want to show
$\begin{pmatrix}1&\gamma&\gamma^2& \ldots & \gamma^n\\ &1&\gamma&\ddots&\vdots\\ &&\ddots&\ddots&\gamma^2\\ &&&1&\gamma\\ &&&&1\end{pmatrix}^{-1}= \begin{pmatrix}1&-\gamma&& & \\ &1&-\gamma&&\\ &&\ddots&\ddots&\\ &&&1&-\gamma\\ &&&&1\end{pmatrix}\,\in\mathbb{C}^{(n+1)\times (n+1)}$
Where the blank spaces of the matrices represent zero entries.
I am not sure how to give a concise proof for this. When I try to use the blockwise inversion or directly multiply these matrices together, I get bogged down in computations. Is any theorem or property I should consider for this proof?
We can find a nice formula for $M^{-1}$, where $$ M = \begin{pmatrix}1&-\gamma&& & \\ &1&-\gamma&&\\ &&\ddots&\ddots&\\ &&&1&-\gamma\\ &&&&1\end{pmatrix} $$ In particular, it is useful to note that $M = I - N$, where $$ N = \begin{pmatrix}0&\gamma&& & \\ &0&\gamma&&\\ &&\ddots&\ddots&\\ &&&0&\gamma\\ &&&&0\end{pmatrix} $$ from there, we could use the Neumann series to compute $$ M^{-1} = (I - N)^{-1} = I + N + N^2 + N^3 + \cdots $$ where we note that $N^k = 0$ whenever $k \geq n+1$. With that, you can conclude that $$ M^{-1} = \pmatrix{1&\gamma&\gamma^2& \ldots & \gamma^n\\ &1&\gamma&\ddots&\vdots\\ &&\ddots&\ddots&\gamma^2\\ &&&1&\gamma\\ &&&&1} $$ which is equivalent to the result you're looking for.
As for the name of this kind of matrix, I would say that it's an upper triangular Toeplitz matrix.
A formal (inductive proof) for the formula of $N^k$: we wish to show that $$ [N^k]_{i,j} = \begin{cases} \gamma^k & j-i = k\\ 0 & \text{otherwise} \end{cases} $$ where $[A]_{i,j}$ denotes the $i,j$ entry of $A$. The base case (either $k=0$ or $k=1$) holds trivially. For the inductive step: we note that if $i,j$ are between $1$ and $n+1$ $$ [N^{k+1}]_{i,j} = [N N^{k}]_{i,j} = \sum_{p=1}^{n+1} N_{ip}[N^k]_{pj} $$ We note that $N_{ip}[N^k]_{pj}$ is only non-zero if $N_{ip} \neq 0$ and $[N^k]_{pj} \neq 0$. By our definition of $N$, $N_{ip}$ will only be non-zero if $p = i+1$. On the other hand: by our inductive hypothesis, $[N^k]_{pj}$ will only be non-zero if $p = j-k$. These can only be simultaneously true if $i+1 = j-k$, which is to say that $j-i = k+1$. Thus, we conclude that $[N^{k+1}]_{i,j} = 0$ whenever $j-i \neq k+1$.
Whenever $j - i = k+1$, we compute $$ [N^{k+1}]_{i,j} = \sum_{p=1}^{n+1} N_{ip}[N^k]_{pj} = N_{i,(i+1)}[N^k]_{(j-k),j} = \gamma \cdot \gamma^k = \gamma^{k+1} $$ The conclusion follows.