$$x-\frac{1}{x} - \frac{1}{x-\frac{1}{x}} - \frac{1} {x-\frac{1}{x} - \frac{1}{x-\frac{1}{x}}} - \dots$$ this expression is the result of recursively subtracting $1/x$ from $x$,
i.e.
$$x_{n+1} = x_n - 1/x_n$$
when executing the previous expression multiple times it does not converge, as the number of executions increase the value of $x$ tends to change slower and slower.
I wonder if this expression can be represented by some integral or summation
On
As user @jjagmath has already noticed, there is no (convergence to a) fixpoint, or one might say, the only fixpoint is infinity.
But there can be said more: there are periodic points of any period length, and they can be numerically approximated to any precision. Since the iteration-process in the question produces divergence, any small error in these approximations shall escalate after a certain finite number of iterations. However, such periodic points can be found as roots of polynomials, and are thus algebraic numbers.
It came out to be helpful to decode the variable $x$ as fractional representation, for instance $x = \frac ab$. I'll use that below.
The iteration $\frac ab \to { a^2-b^2\over ab}$ can be inverted; however the results in the reverse iteration are ambiguous having either $ \frac ab $ or $ \frac {-a}{-b}$ . So with the route of reverse iterations we must supply a vector containing the signs for each iteration. If the vector is finite of length, say, $h$ and the provided signs are taken periodically, then the reverse iteration converges to a set of $h$-periodic points. I've done the calculation of iterations in $(a,b)$ in terms of a 2x2-matrix $A = \begin{bmatrix}a&-b\\b&a \end{bmatrix} $ and $D = \small \begin{bmatrix}1&2\\2&1 \end{bmatrix} $ and $D_r = \small \begin{bmatrix}1&1/2\\1/2&1 \end{bmatrix} $ to iterate $ A \to A^2 \star D_r $ where the "$\star$" means Hadamard-multiplication (=elementwise multiplication), and the reverse iteration $ A \to_\pm \sqrt{D \star A} $ where -if we iterate- the signs are taken from a predefined vector or list.
So for instance I got with the Pari/GP-routine (containing a simple matrix-root-routine which also includes the Hadamard-multiplication with constant matrix $D$)
{M2=[a0,-b0;b0,a0];
il=0;list=vectorv(1000); \\ takes the protocol
for(k=0,199,
M2=matroot(M2);
M2*=[1,-1][k % 2+1]; \\ vector of signs is [1,-1] and taken as infinitely repeating
il++;list[il]=M2[1,]; \\ document current values of [a,b]
);
list=Mat(VE(list,il)); } \\ reshape list to get it as matrix
This got me the protocol:
a b
-----------------------------------
... ... (a couple of hundred lines of
... ... not yet well approximates)
-1.34150376263 -1.02674252883 (approximated to more than 170 dec digits)
0.745432124647 1.37737896568
-1.34150376263 1.02674252883
0.745432124647 -1.37737896568
-1.34150376263 -1.02674252883 (repeating previous values precisely)
... ...
where, after $800$ iterations the difference from $ |a_{k+4}-a_k| \lt 1e-170$.
The last entry cutted to $40$ visible dec digits protocol gives
a =-1.341503762630577719675693514485165986352 and b=-1.026742528828304264759792215760033619885 .
You might try iteration starting at this values (with high enough internal precision) to see the periodicity.
Periodic points in the reals can be found solving polynomials as they occur from 2- or more steps iteration.
The assumed fixpoint iteration at $L$ leads to $L=L-\frac 1L$ as mentioned correctly by user @jjagmath: if $x$ is expressed as $\frac ab$ then we find the fixpoint $$ \frac ab = \frac ab - \frac ba = {a^2-b^2\over ab } \\ \text{solving} \\ \begin{array}{} (1)& a &= a^2 - b^2 \\ (2)& b &= ab &\qquad \to & a=1 \\ \to & 1 &= 1^2 - b^2 &\qquad \to & b=0\\ \end{array} $$ which represents of course the singularity for the expression $x = \frac ab$.
However, this does not exhaust all interesting properties of the iteration in the OP.
Using the polynomials in $(a,b)$ which occur if we iterate 2 or more times symbolically (with vector of signs provided) allows to get infinitely many solutions over algebraic numbers defining periodic iterations.
For instance the above numerical approximation, initially been found by reverse iteration, gives the ratio $$ {a \over b} = \sqrt{1+\sqrt{1/2}} = x $$ as value, which defines a 4-periodic set of values. We find this value if we write $ (a_0,b_0) \to (a_1,b_1)=(a_0^2-b_0^2,a_0 b_0) \to (a_2,b_2)=(a_1^2-b_1^2, a_1 b_1)$ and solve $(a_0,b_0)=(a_2,-b_2)$
The above heuristical protocol more detailed:
a_k b_k x_k x_k symbolic
-1.34150376263 ,-1.02674252883, 1.30656296488, sqrt(1+sqrt(1/2))
0.745432124647, 1.37737896568, 0.541196100146, sqrt(1-sqrt(1/2))
-1.34150376263 , 1.02674252883,-1.30656296488, -sqrt(1+sqrt(1/2))
0.745432124647,-1.37737896568,-0.541196100146,-sqrt(1-sqrt(1/2))
P.S. Wolfram Alpha decomposes the following values (from second row in the protocol)
[a_1 ≈] 0.74543212464725619 ≈ 1/(1 + sqrt(2))^(1/3) `
[b_1 ≈] 1.377378965676004 ≈ 2^(1/4) (1 + sqrt(2))^(1/6) `
Final remark: since we have periodic points for those points the infinite iteration-series (as intended in the OP) might be regularizable by the $\zeta(0)$ or $\eta(0)$ application.
If we do not assume a fixpoint $L$ to which the partial sums converge in the sense that in the limit $L=L - 1/L$ (which can only hold for infinite $L$) but a 2- periodic point, then we get for the limit: $$L = M - 1/M \text{ and } M= L - 1/L \tag 1 $$ Let's see, whether we can find some $L$: $$ \begin{array} {} L &= L-1/L - 1/(L-1/L) \\ &=(L^2-1)/L - L/(L^2-1) \\ &=((L^2-1)^2-L^2)/(L(L^2-1)) \\ L^2(L^2-1)&=(L^2-1)^2-L^2 \\ L^4&=(L^2-1)^2 \\ L^4&=L^4-2L^2+1 \\ 2L^2&=1 \\ L&=\sqrt{1/2} \\ M &= \sqrt{1/2}-\sqrt2 \\ &=\sqrt{1/2}(1-2)=-\sqrt{1/2}=-L \end{array} \tag 2 $$ With this we find the partial sums of the OP's expression, when $x=L$ as $$ \begin{array}{r|r|l} \text{ps} & \text{ps} & \text{partial expression in OP ; ps:partial sum} \\ \hline L & L & x \\ M & -L & x - 1/x \\ L & L & x - 1/x - 1/(x-1/x)\\ M & -L & x - 1/x - 1/(x-1/x) - 1/(...)\\ \vdots & \vdots & \vdots \end{array}\tag 3 $$
One might consider to assign a regularized value in the sense of the alternating $\zeta(0)$-series; if I didn't mess it up we should get the (Euler-) regularized value for $f(x)$ at $x=L=1/\sqrt 2$ as $$f(L) \underset{\mathfrak E}=0 \tag 4$$
In 2020 I've done a (formally) similar analyis (without the final attempt to regularization) on the question of periodic points in the iteration of the (complex-valued) function $z_{k+1} =\exp(z_k)$ and have discussed this to some extent here in MSE because I had some open questions with this. I find it quite funny to find here such a familarity with the derivations...
Form my earlier comment at the OP one may take the external references discussing interesting aspects of the OP's problem:
See oeis.org/A147985 for more information. Clark Kimberling has done an article on this in JIS see: cs.uwaterloo.ca/journals/JIS/VOL12/Kimberling/kimberling56.pdf
If $x_{n+1} = x_n - \frac{1}{x_n}$ were to converge to a limit $L$ for some value of $x_1$, taking limits in the recurrence relation we would have $L = L - \frac{1}{L}$ which is absurd.