Suppose we are given a matrix $B\in \mathbb{R}^{n\times n}$. I would like to find $n$ real values $\{a_i\}_{i=1}^n$ that form a diagonal matrix $A=\text{diag}(\{a_i\}_{i=1}^n)$ to design the kernel of the matrix $(AB-\mathbb{1})$.
Take the following as a concrete example:
Find $\{a_i\}_{i=1}^n$ such that there exist $\lambda \in \text{ker}(AB-\mathbb{1})$ with $|\lambda_i| = 1 \forall i\in \{1,...,n\}$.
Is this problem well posed?
For the specific example, can one determine assumptions on $B$ that guarantee that a solution $(\{a_i\}_{i=1}^n,\lambda)$ exists?
Can one cast this a standard optimization problem such that it can be solved with for example Matlab?
Let me summarize your problem:
You want to design $A=\text{diag}(\{a_i\}_{i=1}^n)$ and find $\lambda$ such that $(AB-I)\lambda = 0$ and $|\lambda_i| = 1 \forall i\in \{1,...,n\}$.
It is clear that $\lambda $ is an eigenvector with eigenvalue 1 to the matrix $AB$ since $AB \lambda = \lambda $ which follows from $(AB-I)\lambda = 0$. Now, we want $1$ to be an eigenvector of $AB$. Let $y=B\mathbf{1}$. Let us choose $A=\text{diag}(\{a_i\}_{i=1}^n)$ such that $AB\mathbf{1}=Ay = \mathbf{1}$. The latter equality is achieved by
$$ a_i = 1/y_i$$
For the latter choice, it is of course necessary that $y_i \neq 0$.