Filtering of fish or planted tanks is often given as amount of tank volumes / hour. E.g. a tank having volume of 400 L and a pump of 1000 L/h would result in filtering of 2.5 volumes / hour.
What is the formula to calculate fraction of water which is being filtered within given time (or remains unfiltered), assuming the water mixes instantly?
Based on the continuous interest formula, I come up to the following, given the flow f (volumes / hour) and time t (hours):
$$fractionFiltered = 1 - \frac{1}{e^{f*t}}$$
Filtering of 1 volume / hour would result in $1 - \frac{1}{e^{1*1}} = 1 - 1/e = 0.632 = 63.2 \%$ of water passing the filter per hour.
Is the formula correct?
P.S. I know, this could be a homework exercise, but I'm just a hobby aquariumist with math courses way in the past.
I don't think that you are quite asking the "right" question. It does not make sense to talk about "filtered" water vs "unfiltered" water—these two volumes are being constantly mixed, and there is no way to meaningfully distinguish between them.
The goal of filtering water is to remove contaminants from the water (fish poop, microorganisms, etc). Instead of thinking about the proportion of water which has been filtered, I think it is better to think about the quantity of contaminants in the water. So define
\begin{align*} V(t) &= \text{total volume of the tank (in liters) at time $t$,} \\ P(t) &= \text{the volume of water pumped through the filter (in liters) at time $t$, and} \\ C(t) &= \text{the amount of contamination in the water (in, say, grams) at time $t$.} \end{align*}
Further assume that the filter is perfectly effective—the water coming out of the filter is perfectly pure, no matter how contaminated it is going in; and that the water is always perfectly mixed. This means that the rate at which contamination is being removed from the water is equal to the concentration of the contamination times the rate at which water is passing through the filter. In other words
$$ C'(t) = -\frac{C(t)}{V(t)} P(t). $$
Note that this is negative, since contaminants are being removed. In the given problem, both $V(t)$ and $P(t)$ are constant (they don't need to be—maybe the tank is slowly being filled while being filtered, or maybe the pump works at variable speeds). This radically simplifies the problem: instead, write
$$ C'(t) = -\frac{P}{V} C(t). $$
This is a separable differential equation. The "usual" solution (with the "usual" abuses of notation) looks something like
\begin{align*} \frac{1}{C(t)} C'(t) = -\frac{P}{V} &\implies \frac{1}{C} \frac{\mathrm{d}C}{\mathrm{d}t} = -\frac{P}{V} \\ &\implies \frac{1}{C}\, \mathrm{d}C = -\frac{P}{V}\, \mathrm{d}t \\ &\implies \int \frac{1}{C} \,\mathrm{d}C = -\int \frac{P}{V} \,\mathrm{d}t \\ &\implies \log(C) = -\frac{P}{V} t + k' \\ &\implies C = \mathrm{e}^{-(P/V)t + k'} = k \mathrm{e}^{-(P/V)t}, \end{align*}
where $k'$ is a constant of integration, and $k = \mathrm{e}^{k'}$ is another constant. In other words, the amount of contaminant in the tank at time $t$ is given by $$ C(t) = k \mathrm{e}^{(P/V)t}. $$ Note that $C(0) = k$, hence $k$ is the amount of contaminant in the tank at time zero. To keep this straight, it might be helpful to replace $k$ with $C_0$, and write $$ C(t) = C_0 \mathrm{e}^{(P/V) t}. $$ This gives a model for how "contaminated" the water in the tank is at any particular moment in time.
As an example, consider a $500$ liter tank which initially contains $100$ grams of contamination, and a filter which processes $1000$ liters per hour. The amount of contamination in the tank at any given time is
$$ C(t) = 100 \mathrm{e}^{-2t}. $$
GeoGebra sketch.
A slightly more meaningful or easily understood measure might be the "half-life" of contamination; how long does it take for the filter to remove half of the contaminants in the water? The half-life, which I'll denote by $t_{1/2}$, is the solution to the equation
$$ C_0 \mathrm{e}^{-(P/V)t_{1/2}} = C(t_{1/2}) = \frac{1}{2} C_0 \iff \mathrm{e}^{-(P/V)t_{1/2}} = \frac{1}{2} \iff t_{1/2} = -\frac{V}{P} \log\!\left( \frac{1}{2} \right) = \frac{V}{P} \log(2), $$
where $\log(2)$ denotes the natural logarithm of $2$. In the previous example, with $V=500$ and $P=1000$, this gives $$ t_{1/2} = \frac{1}{2} \log(2) \approx 0.3466~\text{ hours}. $$ That is, the amount of contamination will be reduced by one half every $20$ minutes or so.