The question is specific to this document: http://people.missouristate.edu/jrebaza/assets/10compression.pdf
It is chapter 6 from book "A First Course in Applied Mathematics" by Jorge Rebaza. It is, a to the point explaination of DCT that is easier for an engineer to grasp. However, somethings are not explained and thus here I am.
The questions are about the notation used when the book starts talking about 1D DCT; on pg.3 of the pdf it shows x that is the input to be transformed. This x is a vector i.e a matrix with n rows and 1 column. We then multiply this with C that is n x n matrix to get a n x 1 matrix y, that contains the DCT coefficients.
Q: Why is x described fundamentally as a column matrix and not a row matrix on pg.3 of pdf? If the x was row matrix i.e it was not transposed, making it 1 x n matrix, how would the equation to calculate y change? Right now it is y=Cx.
Q: Why is the equation y=Cx and not written as y'=Cx' where the ' symbol means transpose? Since we are describing both x and y as being transposes?
Q: When we move to 2D DCT (pg.8 in pdf) the equation is derived like this y=C*(Cx')' which gives y=CxC'. How come here we explicitly write the transpose of the operand x but not in the 1D equation?
The descriptions are looking inconsistent.
Q1: The only fundamental characteristic of signals $x$ and $y$ is that both are one-dimensional and consist of $n$ elements. It is only for notational convenience (which allows easier incorporation of linear algebra methods/techniques such as transposition) to represent $x$ and $y$ as column or row vectors. The choice between column or row representation is completely arbitrary, however, it is more common in engineering literature to employ column vectors.
If you would like to transform a "column-notation" equation to a "row-notation" equation, simply apply transposition. For example, applying the transposition operator to both sides of the equation $y=Cx$ results in $$ \begin{align} y^T &= \left(Cx \right)^T\\ &= x^T C^T \end{align} $$ Note that $y^T$ and $x^T$ are rows, therefore, the above equality represents the DCT transform in row notation. Note that both "column" and "row" notation are equivalent. They both describe the exact same operations in different representations.
Q2: You are obviously confused with the notation. Note that in Theorem 6.2, signal $x$ is defined as $x=[x_0, x_1, \ldots, x_{n-1}]^T$. This is a column vector because it is the transpose of $[x_0, x_1, \ldots, x_{n-1}]$ which represents a row vector.
Q3: In 2D DCT, you no longer consider one-dimensional signals (which can be naturally represented as either columns or rows) but two-dimensional signals, which can be naturally represented by matrices. Therefore, signal $X$ is a matrix (note the use of capital letter, to differentiate from the column/row vector notation using small letters). As stated in the beginning of Sec. 6.1.2,
"Given an input matrix X, the DCT is applied to the rows of $X$, and then the DCT is applied again to the rows of the resulting matrix, i.e., $X=C(CX^T)^T=CXC^T$."
Let's verify that the notation indeed corresponds to the series of operation described.
First note that the statement "the DCT is applied to the rows of $X$" means the following: Treat each row of $X$ as a one-dimensional signal and apply the DCT on it. Now, since the author chose the column vector represenation of the DCT transform, the rows of $X$ must be converted in columns in order to apply the DCT. This is exactly what the operation $X^T$ does. Now application of DCT to the rows of $X$ is exactly the operation $CX^T$, with the columns of the resulting matrix representing the DCT of the rows of $X$.
The statement "the DCT is applied again to the rows of the resulting matrix" means the following: Convert the rows of the resulting matrix (i.e., $CX^T$) to columns, which is easily performed by simple transposition, and apply the DCT on the resulting columns. This series of operations is represented as $C(CX^T)^T$, thus verifying that $CXC^T$ is indeed the matrix representation of the DCT applied on signal $X$.