I have an vector in space that I want to use to 'restrict' other vectors, but I've no idea how. Since I don't know how to explain this using math terms, here's a picture:

The blue line represents the vector that will 'restrict' the other vectors.
Green lines represent other vectors that should be 'restricted'.
The red line is perpendicular to the 'restricting' vector (blue line) - green lines passing through this line should be 'restricted' to this line.
For more clarification: If the limiting vector was something like $(0, 5)$, all other vectors would have their $Y$ values limited to $(-\infty, 5]$, but it's not that simple when the blue vector is not axis-aligned.
There are a couple of ways to interpret what you mean by “limiting” a vector, but based on your example of $(0,5)$ as the limiter, it looks like you want to trim the component that’s parallel to the limiting vector. This is easily done by projecting onto the limiter.
Let $\mathbf v$ be the “limit” vector. Given a vector $\mathbf x$ that is to be trimmed, first compute the scalar projection $p$ of $\mathbf x$ onto $\mathbf v$: $$p = {\mathbf x\cdot\mathbf v \over \|\mathbf v\|}.$$ If $p\le\|\mathbf v\|$ then there’s nothing to do. Otherwise, decompose $\mathbf x$ into its components $\mathbf x_\parallel$ and $\mathbf x_\perp$ parallel and orthogonal, respectively, to $\mathbf v$, and replace $\mathbf x_\parallel$ with $\mathbf v$. The orthogonal component is $$\mathbf x_\perp = \mathbf x - {p\over\|\mathbf v\|}\mathbf v,$$ the orthogonal rejection of $\mathbf x$ from $\mathbf v$ and so the trimmed vector is $$\mathbf x_\perp+\mathbf v = \mathbf x + \left(1 - {p\over\|\mathbf v\|}\right)\mathbf v = \mathbf x + \left(1-{\mathbf x\cdot\mathbf v \over \mathbf v\cdot\mathbf v}\right)\mathbf v.$$ You can even combine this expression with the test if you like: $$\mathbf x + \min\left(0,1-{\mathbf x\cdot\mathbf v \over \mathbf v\cdot\mathbf v}\right)\mathbf v$$
If instead you just want to shorten $\mathbf x$ without changing its direction, simply multiply the entire vector by ${\|\mathbf v\|\over p}$ without performing the above decomposition.