I'm getting to tensor algebra. However, there's one expression I wasn't quite sure: Suppose $\displaystyle expression\cdot \frac{\partial x_i}{\partial \tilde{x}_j}$ sum over the $i,j$ index, I used the usual trick that $$\displaystyle expression \cdot (\frac{\partial \tilde{x}_j}{\partial x_i})^{-1}=expression\cdot (M_{ji})^{-1}.$$
However, something doesn't feel right. Because it's easy to see from the matrix form and the canonical relationship $$\displaystyle \frac{\partial \tilde{x}_i}{\partial x_j}\cdot \frac{\partial x_j}{\partial \tilde{x}_l}=\delta^i_l$$ that $$(\frac{\partial \tilde{x}_l}{\partial x_j})^{-1}=(M^{-1})_{jl}$$ was the correct formula.
But the I'm still confused about this issue. Because, from the perspective of tensor, which was still element-wise operation, it seemed to lead to element wise inverse: $$(\frac{\partial \tilde{x}_j}{\partial x_i})^{-1}=(M_{ji})^{-1}$$ but not the entire inverse of the matrix labeled in $j,i$.
Could you explain to me why we can't simply take the element wise inverse to the tensor?
Further, how to prove that $$(\frac{\partial \tilde{x}_j}{\partial x_i})^{-1}=(M^{-1})_{ij}$$ purely from an tonsorial method/perspective?
I think your confusion stems from assuming $\frac{\partial\tilde{x}_l}{\partial x_j}=\left(\frac{\partial x_j}{\partial\tilde{x}_l}\right)^{-1}$. Partial derivatives don't work like that. Actually, let me clarify that. Each partial derivative is only well-defined when you specify which variables are held fixed. The above equation works if the derivatives are defined according to the same convention, but they're typically not.
Let's take a 2D example: $x_1=x,\,x_2=y,\,\tilde{x}_1=r,\,\tilde{x}_2=\theta$. Then $\frac{\partial x}{\partial r}$ is usually assumed to mean $\left(\frac{\partial x}{\partial r}\right)_\theta$, i.e. $\theta$ is constant, whereas $\frac{\partial r}{\partial x}:=\left(\frac{\partial r}{\partial x}\right)_y$. These two derivatives aren't reciprocals; in fact, they're equal (they're both $\frac{x}{r}$).
If you're familiar with relativistic "paradoxes" such as the twin paradox, you can actually explain those in the same way: the quantities found to be equal instead of having product $1$ are expressible as partial derivatives that hold different things constant. The Lorentz transformation satisfies$$\left(\frac{\partial t^\prime}{\partial t}\right)_{x}=\left(\frac{\partial t}{\partial t^\prime}\right)_{x^\prime}=\left(\frac{\partial x^\prime}{\partial x}\right)_{t}=\left(\frac{\partial x}{\partial x^\prime}\right)_{t^\prime}=\gamma,\\\left(\frac{\partial t^\prime}{\partial t}\right)_{x^\prime}=\left(\frac{\partial t}{\partial t^\prime}\right)_{x}=\left(\frac{\partial x^\prime}{\partial x}\right)_{t^\prime}=\left(\frac{\partial x}{\partial x^\prime}\right)_{t}=\frac{1}{\gamma}.$$
Returning to your problem, $\frac{\partial\tilde{x}_l}{\partial x_j}$ holds the $x_k$ with $k\ne j$ constant, while $\frac{\partial x_j}{\partial\tilde{x}_l}$ holds the $\tilde{x}_m$ with $m\ne l$ constant.
So that answers your first question. Your second is about how a "purely tensorial" method can get the right answer. I'm not sure what your criteria are, but here's how I look at it. The chain rule $dx_j=\frac{\partial x_j}{\partial\tilde{x}_l}d\tilde{x}_l=M_{jl}d\tilde{x}_l$ can be rewritten as $d\vec{x}=Md\vec{\tilde{x}}$, so $d\vec{\tilde{x}}=M^{-1}d\vec{x}$ i.e. $d\tilde{x}_l=\left(M^{-1}\right)_{lj}dx_j$. Comparing this with the chain rule $d\tilde{x}_l=\frac{\partial \tilde{x}_l}{\partial x_j}dx_j$, $\frac{\partial \tilde{x}_l}{\partial x_j}=\left(M^{-1}\right)_{lj}$.