So I have the closed formula of:
(n(2n+1)(2n-1))/3
This closed form is the summation of all odd squares to n. I am now trying to figure out how to subtract something less than or equal to n.
I am trying to get the closed form of all odd squares up to n, so it should only add odd squares that are less than or equal to n. For example if n=9 then the summation should equal 10, because the perfect odd squares less than or equal to 9 are: 1 and 9, 1+9=10.
First note that your formula can be interpreted as the nth term of a sequence:
$$\{n(2n+1)(2n-1)/3\}=1,10,35,84,165,...$$
You claim that your nth term formula gives the summation (more properly, the partial sums) of all squares of odd numbers, up to $n$. This implies that term $n=3$, for example, is the sum of the first three odd squares:
$$35=1+9+25=1^2+3^2+5^2$$
This appears to be correct, but I have not tested beyond the first four terms. I would be interested to see a proof of the claim that this sequence is the partial sums of the squares of odd numbers.
This is the nth partial sum of the squares of odd numbers:
$$S_n=\sum_{i=1}^{n}(2i-1)^2=1+9+25+...+(2n-1)^2$$
Note that odd numbers are produced by the moiety $\{2i+1\}$ or $\{2i-1\}$ and even numbers by $\{2i\}$; this may help you construct sequences and series in the future.
Let's examine some partial sums of this series:
$$S_1=\sum_{i=1}^{1}(2i-1)^2=1$$ $$S_2=\sum_{i=1}^{2}(2i-1)^2=1+9=10$$ $$S_3=\sum_{i=1}^{3}(2i-1)^2=1+9+25=35$$ $$S_4=\sum_{i=1}^{4}(2i-1)^2=1+9+25+49=84$$
Notice that the partial sums are indeed following your sequence (for the first four terms at least):
$$1,10,35,84,...=\{n(2n+1)(2n-1)/3\}$$
So, we may suspect an equation between the partial sums of the squares of odd numbers and the sequence of the partial sums:
$$\sum_{i=1}^{n}(2i-1)^2=\frac{n(2n+1)(2n-1)}{3}\equiv S_n$$
where "$\equiv$" means "define as", so your function $S_n$ can be calculated by either of the above expressions.
UPDATE
Given an arbitrary integer $N$, you seem to be seeking the sum of the squares of odd numbers up to but not exceeding $N$. That is, you desire the partial sum $S_n$ where none of the $n$ terms (all squares of odd numbers) exceeds $N$. (Correct me if I'm wrong).
I propose the following algorithm as a solution to your problem:
For arbitrary input integer $N$, define the integer index $n$:
$$n\equiv \Big\lfloor\frac{\sqrt N+1}{2}\Big\rfloor$$
where $\lfloor x\rfloor$ is the floor function, rounding down the input x to the nearest integer. I suppose I would show the derivation for the above expression if you request it, but I've already spent more time than I should have on this little puzzle.
When the appropriate parameter value $n$ is obtained from arbitrary input $N$, the value you seek is the $n$th partial sum of the squares of odd numbers:
$$S_n\equiv\sum_{i=1}^{n}(2i-1)^2 \\=\frac{n(2n+1)(2n-1)}{3} \\=\frac{1}{3}\Big(4n^3-n\Big)$$
where whichever form seems most convenient may be used for calculation.
TL;DR I think the function you desire is
$$f(N)\equiv\sum_{i=1}^{\lfloor(\sqrt N+1)/2\rfloor}(2i-1)^2 \\=\frac{1}{3}\Big(4\Big\lfloor\frac{\sqrt N+1}{2}\Big\rfloor^3-\Big\lfloor\frac{\sqrt N+1}{2}\Big\rfloor\Big)$$
I have listed two equivalent forms for convenience. This function is intended to return the sum of the squares of odd numbers, truncated so that no term (odd number square) is larger than the input $N$. I invite you to tabulate values of $f(N)$ for integer inputs $N$ and let me know if this function fulfills your your needs. If I may ask, what motivates your interest in such a function?