Note: I'm not too experienced with branches of mathematics like topology, so a lot of the terminology I use is used pretty loosely.
I think we all understand the fact that a point in $n$-dimensional space is described with $n$ parameters. In 2d space, you have $(p_1, p_2)$ (where $p_k \in \mathbb{R}$), in 3d space you have $(p_1, p_2, p_3)$, so in an $n$-dimensional space you have $(p_1, p_2, ..., p_n)$. I know the way I'm stating this with the notation I use isn't as robust as it could be, but you get the idea.
Going into the world of fractal geometry, we find non-integer dimensional spaces. Applying what I said previously, how would you even express a point in such a space? What does it even mean to be that kind of space? Are you unable to express a "point" in such a space?
Dimensions in Vector Spaces
The notion of dimension that you put forward is, roughly speaking, the correct way of understanding the dimension of a vector space. A vector space consists of two basic sets of objects: a base field $\mathbb{F}$, and a set of vectors $\mathcal{V}$. These two sets are linked via some algebraic structures:
The basic example of a vector space that you should have in your head is $\mathbb{R}^n$ (vectors are tuples of real numbers, and the scalars are also real numbers). However, there are other examples, such as $\mathbb{C}^n$ (a complex vector space), $\mathbb{Q}_p^n$ (a $p$-adic vector space), and function spaces such as $C(\mathbb{R})$ and $L^p(\mathbb{R})$ (vectors are certain types of functions, and the scalars are real numbers).
Assuming the Axiom of Choice, it can be shown that every vector space has a basis. The basis of a vector space is a collection of vectors that span the vector space. In particular, we say that $\mathcal{B} = \{\vec{e}_i\}_{i=1}^{N}$ is a basis for the vector space $\mathcal{V}$, then we can write every element of $\mathcal{V}$ as a linear combinations of elements of $\mathcal{B}$. That is, if $\vec{v} \in \mathcal{V}$, then there is a collection of scalars $\{\alpha_i\}_{i=1}^{N}$ such that $$ \vec{v} = \sum_{i=1}^{N} \alpha_i \vec{e}_i. $$ Moreover, the elements of $\mathcal{B}$ are linearly independent, which means that if $$ \sum_{i=1}^{N} \alpha_i \vec{e}_i = 0, $$ then $\alpha_i = 0$ for all $i$. Even more impressively, while a vector space may have many different bases, every single one of them has the same cardinality. That is, if $\{\vec{e}_i\}_{i=1}^{M}$ and $\{\vec{f}_j\}_{j=1}^{N}$ are both bases for $\mathcal{V}$, then $M=N$ (note that we could have $M = N = \infty$—bases need not be finite; this is why we require AoC).
This is all a long explanation, but the punchline is that we can think of the vectors in a basis as describing the different "directions" in which it is possible to travel in a vector space. Each direction requires a different parameter to describe, which gives the dimension of the vector space. That is, the dimension of a vector space is the cardinality of any basis.
This is a purely algebraic notion, and has purely algebraic generalizations (such as the Krull dimension). However, this is not the kind of dimension that people who study fractals generally think about when they say "dimension."
Dimensions through Scalings
To talk about dimensions the way that most fractal geometers think about them, you need both more and less structure. On the one hand, we don't really need a vector space—typically, all that is required is a metric [measure] space. A metric is a way of determining the distance between two points, while a measure assigning a size (or volume) to subsets. We typically want both a metric (which we can use to describe how sets scale) and a measure (though we can make due without an a priori measure).
The basic intuition is as follows:
The pattern is as follows: let $X$ be an $n$-dimensional set with measure $\mu(X)$ (if $X$ is a segment, then $\mu(X)$ is the length of that segment; if $X$ is a cube, then $\mu(X)$ is the volume of that cube; and so on). Scale $X$ by a factor of $\alpha$. Then $$ \mu(\alpha X) = \alpha^n \mu(X). $$ The goal is to describe the dimension in terms of the measures of scaled sets, so we solve for $n$: $$ \mu(\alpha X) = \alpha^n \mu(X) \implies \alpha^n = \frac{\mu(\alpha X)}{\mu(X)} \implies n = \log_{\alpha} \left( \frac{\mu(\alpha X)}{\mu(X)} \right). $$ We can use this to define a notion of dimension in terms of scaled copies of sets and measures. Roughly speaking, $\frac{\mu(\alpha X)}{\mu(X)}$ is the number of copies of $X$ are required to "cover" $\alpha X$. If we take the logarithm of this quantity, we get the dimension.
For example, consider the ternary Cantor set $\mathscr{C}$. This set is constructed by removing the middle third of an interval, then removing the middle third of each of the remaining intervals, and so on. Wikipedia gives the first several steps in the construction:
Notice that below the initial interval, the left- and right-halves of the Cantor set look the same. Indeed, the Cantor set is made up of two identical copies of itself, scaled down by a factor of $\frac{1}{3}$. In the language developed above, we have $$ \frac{\mu\left(\frac{1}{3} \mathscr{C} \right)}{\mu(\mathscr{C})} = \frac{1}{2} \implies \dim(\mathscr{C}) = \log_{\frac{1}{3}} \left( \frac{1}{2} \right) = \log_3(2). $$ This we can reasonably state that the dimension of the Cantor set is $\log_3(2) \approx 0.631$.
Note, however, that the notion of dimension here is quite different from the notion of a vector space. Indeed, while the ternary Cantor set lives in $\mathbb{R}$ (a vector space), the Cantor set itself doesn't have an obvious vector space structure. Instead, it has a way of measuring distances between points (the absolute value from $\mathbb{R}$), and a rough way of measuring the sizes of subsets (there is actually something very subtle going on here, related to something called the Hausdorff measure, which is a way of getting a measure from a metric). In this context, the dimension is a way of describing how the measure of a set changes with scaling.
Concluding Thoughts
Typically, when we talk about spaces with non-integer dimension, we are thinking about something like the notion of dimension described above. That is, we have in mind some way of quantifying how the volumes or measures of sets change as the sets are scaled.
That being said, there are actually many other notions of dimension running around. There are purely topological definitions, such as the covering dimension, big and little inductive dimensions, and so on. The dimension of a manifold is an integer related to the local structure of that manifold. The previously mentioned notion of Krull dimension is algebraic. There are notions of dimension somewhat related to, but more general than, the idea of relating measures and scales. There are even ways of assigning complex valued dimensions to sets (this is my own area of research).