For any $x,y\in L_2([0,1],\lambda)$, define the inner product $\langle. , . \rangle$ by
\begin{equation} \langle x, y \rangle=\int_{[0,1]} x(t) y(t) \lambda (dt) \end{equation}
Is it proper to define a line and the symmetry w.r.t a line as follows?
Definition: For any $x,y\in L_2 ([0,1],\lambda)$, a line $l(x, y)$ is the set of all points of the form $x+k y$ for some $k\in R$.
A set $W\subset L_2([0,1],\lambda) $ is symmetric with respect to $l( x, y )$ if for every $z\in L_2([0,1],\lambda)$ satisfying $\langle z, y \rangle=0$, we have $x+ky+z\in W$ implies $x+ky-z\in W$.
Yes, that looks correct. And you can write down the reflection operator $R$ and then define $W$ to be symmetric about the line if $RW=W$. Using orthogonal projection, you can write $w$ as $$ \begin{align} w & = w-\left((w-x)-\frac{\langle w-x,y\rangle}{\langle y,y\rangle}y\right)+\left((w-x)-\frac{\langle w-x,y\rangle}{\langle y,y\rangle}y\right) \\ & = x+\frac{\langle w-x,y\rangle}{\langle y,y,\rangle}y+\left((w-x)-\frac{\langle w-x,y\rangle}{\langle y,y\rangle}y\right) \end{align} $$ The final expression is a point on the line + a vector that is orthogonal to the direction vector $y$. Therefore, the reflection operator is $$ Rw = x+\frac{\langle w-x,y\rangle}{\langle y,y\rangle}y-\left((w-x)-\frac{\langle w-x,y\rangle}{\langle y,y\rangle}y\right) \\ = 2x-w+2\frac{\langle w-x,y\rangle}{\langle y,y\rangle}y $$