How to approach this question?
Let's say we have 40 l of water. Operation 1) 4 l water take out and 4 l juice replaced Operation-2)5 l water taken out and 5 l juice replaced and so on what would be the water's concentration at the end of nth operation
On
Your question seems confused on two counts:
(a) After operation 1, you can only take out a mixture and replace by juice.
(b) Operation 1 mentions 4L exchange, operation 2 mentions 5L and so on. I am answering on the assumption that in each of n operations, you intended an exchange of 4L from a cask of 40L.
After operation 1, fraction of water left in cask = 36/40 = 0.9.
Any subsequent operation will lower the amount of water to 0.9 of the water that was left, hence after operation 2, fraction of water left in cask = $0.9^2$.
Thus after n operations, fraction of water left in cask = $0.9^n$.
Generalising, if you leave behind water = fraction f of that remaining each time, the concentration (= fraction) of water after n operations will be $f^n$
Finally, if you really meant different fractions left behind in each operation, concentration will simply be $f_1\times f_2\times f_3\times....\times f_n$
Hint
If you have $$w+j=40$$ liters of well-mixed water ($w$) and juice ($j$) and replace $k$ liters of it by $k$ liters of juice then you end up with $$w'+j'=40$$
where $$w'=(1-\frac{k}{40})w$$ and $$j'=(1-\frac{k}{40})j+k$$