Let $A$ be a $4\ x\ 4$ binary matrix and $Z=\pmatrix {s&s&s&s \\ s&s&s&s \\s&s&s&s \\s&s&s&s}$
Then $\det(A+Z)=\det(A)=1\ $ (independent of s, so $\det(A)$ is "shift-invariant") can only hold if the number of zeros in $A$ is exact 8. This condition is not sufficient because only $72$ out of the $12\ 870$ matrices with exactly 8 zeros do the job. I checked this funny and surprising condition by brute force. Is there a proof without checking all $65\ 536$ matrices ?
Some thoughts on your question : denote by $C_1,C_2,C_3,C_4$ the columns of $A$. Let $V$ be the all-ones column $(1,1,1,1)$, and let $U$ be the matrix of all whose columns are equal to $V$ (so that $Z=sU$). Then
$\det (A + sU) = $ $$\begin{array}{lcl} &=& [C_1+sV,C_2+sV,C_3+sV,c_4+sV] \\ &=& [C_1,C_2,C_3,C_4] +s([V,C_2,C_3,C_4]+[C_1,V,C_3,C_4]+ [C_1,C_2,V,C_4]+[C_1,C_2,C_3,V]) \end{array}\tag{1} $$
So your "shift-invariance" property $\det(A+sU)=\det(A)$ will hold iff $$ [V,C_2,C_3,C_4]+[C_1,V,C_3,C_4]+ [C_1,C_2,V,C_4]+[C_1,C_2,C_3,V]=0 $$
Hope this helps.