Here is notation used in the explanation of the question.
Suppose that $A$ is an $m \times n$ matrix. The $(i, j)$-entry of $A$ is denoted as $[A]_{i,j}$.
Suppose that $A$ is an $m \times n$ matrix. Suppose that $i_1$, $i_2$, $\dots$, $i_s$ are distinct integers less than or equal to $m$, and that $j_1$, $j_2$, $\dots$, $j_t$ are distinct integers less than or equal to $n$. The matrix obtained from $A$ by removing rows $i_1$, $i_2$, $\dots$, $i_s$ and columns $j_1$, $j_2$, $\dots$, $j_t$ is denoted as $A({i_1,\dots,i_s}|{j_1,\dots,j_t})$.
Suppose that $A$ is an $m \times n$ matrix. Suppose that $i_1$, $i_2$, $\dots$, $i_s$ are distinct integers less than or equal to $m$, and that $j_1$, $j_2$, $\dots$, $j_t$ are distinct integers less than or equal to $n$. The matrix obtained from $A$ by removing all rows other than rows $i_1$, $i_2$, $\dots$, $i_s$ and all columns other than $j_1$, $j_2$, $\dots$, $j_t$ is denoted as $\displaystyle A\binom{i_1,\dots,i_s}{j_1,\dots,j_t}$.
Suppose that $i$, $j$ are integers. Define $$ \rho(i, j) = \begin{cases} 0, & i < j; \\ 1, & i \geq j. \end{cases} $$
Suppose that $i_1$, $i_2$, $\dots$, $i_n$ are integers. Define $$ \tau (i_1, i_2, \dots, i_n) = \sum_{1 \leq u < v \leq n} {\rho (i_u, i_v)}. $$ (If $n = 1$, the sum above is understood to be $0$.)
Suppose that $a_1$, $a_2$, $\dots$, $a_n$ are $m \times 1$ matrices. Then $[a_1, a_2, \dots, a_n]$ is the $m \times n$ matrix whose $(i, j)$-entry is equal to $[a_j]_{i,1}$. The notation allows us to display a matrix using its columns.
Suppose that $R$ is a commutative ring with unity. Suppose that $A$ is an $n \times n$ matrix with entries in $R$. It is well known that the determinant is equal to (or is defined to be) $$ \det {(A)} = \begin{cases} [A]_{1,1}, & n = 1; \\ \displaystyle \sum_{i = 1}^{n} {(-1)^{i+1} [A]_{i,1} \det {(A(i|1))}}, & n \geq 2. \end{cases} $$
Fix $\mu \in R$ such that $\mu^2 = 1$. Define the "modified determinant" of $A$ by $$ \operatorname{mdt} {(A)} = \begin{cases} [A]_{1,1}, & n = 1; \\ \displaystyle \sum_{i = 1}^{n} {{\color{red}{\mu}}^{i+1} [A]_{i,1} \operatorname{mdt} {(A(i|1))}}, & n \geq 2. \end{cases} $$
Direct computations prove the following results:
Suppose that $A$ is an $n \times n$ matrix and that $j$ is a positive integer less than or equal to $n$. Then $$ \operatorname{mdt} {(A)} = \sum_{i = 1}^{n} {{\mu}^{i+j} [A]_{i,j} \operatorname{mdt} {(A(i|j))}}. $$
Suppose that $A$ is an $n \times n$ matrix, that $j_1$, $j_2$, $\dots$, $j_k$ are distinct positive integers less than or equal to $n$, and that $j_1 < j_2 < \dots < j_k$. Then $$ \begin{aligned} \operatorname{mdt} {(A)} = \sum_{1 \leq i_1 < i_2 < \dots < i_k \leq n} {\operatorname{mdt} {\left( A\binom{i_1, i_2, \dots, i_k} {j_1, j_2, \dots, j_k} \right)}} {\mu}^{i_1 + i_2 + \dots + i_k + j_1 + j_2 + \dots + j_k} \operatorname{mdt} {(A({i_1,i_2,\dots,i_k}|{j_1,j_2,\dots,j_k}))}. \end{aligned} $$
Suppose that $A$ is an $n \times n$ matrix and that $j_1$, $j_2$, $\dots$, $j_n$ are $n$ distinct positive integers less than or equal to $n$. Then $$ \operatorname{mdt} {(A)} = \sum_{\substack{ 1 \leq i_1, i_2, \dots, i_n \leq n \\ i_1, i_2, \dots, i_n\,\text{are distinct} }} {{\mu}^{\tau (i_1, i_2, \dots, i_n)}\, {\mu}^{\tau (j_1, j_2, \dots, j_n)}\, [A]_{i_1,j_1} [A]_{i_2,j_2} \dots [A]_{i_n,j_n}}. $$
Suppose that $I_n$ is the $n \times n$ identity matrix. Then $\operatorname{mdt} {(I_n)} = 1$.
Suppose that $j$ is a positive integer less than or equal to $n$. Suppose that $a_1$, $\dots$, $a_{j-1}$, $a_{j+1}$, $\dots$, $a_n$ are $n \times 1$ matrices. Suppose that $x$, $y$ are $n \times 1$ matrices. Suppose that $s$, $t \in R$. Then $$ \begin{aligned} & \operatorname{mdt} {[a_1, \dots, a_{j-1}, \overset{\text{column}\,j}{sx + ty}, a_{j+1}, \dots, a_n]} \\ = {} & s \operatorname{mdt} {[a_1, \dots, a_{j-1}, x, a_{j+1}, \dots, a_n]} + t \operatorname{mdt} {[a_1, \dots, a_{j-1}, y, a_{j+1}, \dots, a_n]}. \end{aligned} $$
Suppose that $q$ is a positive integer less than or equal to $n$. Suppose that $p$ is a positive integer less than $q$. Suppose that $a_1$, $a_2$, $\dots$, $a_n$ are $n \times 1$ matrices. Then $$ \operatorname{mdt} {[a_1, \dots, \overset{\text{column}\,p}{a_q}, \dots, \overset{\text{column}\,q}{a_p}, \dots, a_n]} = {\mu}\operatorname{mdt} {[\dots, a_p, \dots, a_q, \dots]}. $$
Suppose that $A$ is an $n \times n$ matrix. Then the "modified determinant" of the transpose of $A$ is equal to that of $A$: $$ \operatorname{mdt} {(A^{\mathrm{T}})} = \operatorname{mdt} {(A)}. $$
I notice that $\operatorname{mdt}$ is not alternating (in general), which means that it is not necessarily true that "if $A$ has two equal columns, then $\operatorname{mdt} {(A)} = 0$": $$ \operatorname{mdt} { \begin{bmatrix}1 & 1\\1 & 1\\\end{bmatrix} } = 1 + \mu \neq 0\, \quad\text{unless $\mu = -1$}. $$
Accordingly, $\operatorname{mdt}$ is not multiplicative (in general), which means that it is not necessarily true that "if $A$, $B$ are square matrices of the same size, then $\operatorname{mdt} {(AB)} = \operatorname{mdt} {(A)} \operatorname{mdt} {(B)}$": $$ \begin{aligned} & \begin{bmatrix}1 & 1\\1 & 1\\\end{bmatrix} \begin{bmatrix}0 & 0\\1 & 1\\\end{bmatrix} = \begin{bmatrix}1 & 1\\1 & 1\\\end{bmatrix}, \\ & \operatorname{mdt} {\begin{bmatrix}1 & 1\\1 & 1\\\end{bmatrix}} = 1 + \mu, \\ & \operatorname{mdt} {\begin{bmatrix}0 & 0\\1 & 1\\\end{bmatrix}} = 0, \\ & (1 + \mu) \neq (1 + \mu) \cdot 0\, \quad\text{unless $\mu = -1$}. \end{aligned} $$
Here is my question: Is there literature on the "modified determinant"?
I am sorry if the above explanation is poor.