I am an engineer so forgive me if my terminology/explanations are a bit off.
I am trying to combine weights using AHP (having people doing a survey) and while I am capable of obtaining weights for parts within the lists themselves, the issue comes from combining the multiple lists together. The lists are about the same topic but slightly different categories of stuff. Think company with different departments. Each department is a list. Within each department each element is a person. The weights used are their importance for the well-being/success of a company.
For example:
I have a set of lists $A,B,C,D$ with weights $a,b,c,d$ ($x_N$). The weights always sum up to 1 due to used scaling. Within each list there are elements, whose independent weights we will call $x_1, x_2, x_3$ and so on (substitute $a$/$b$/$c$/$d$ for $x$). As both $a+b+c+d=1$ and $x_1+...+x_n=1$ for each list, I have considered simply multiplying $x_N*x_n=w_n$ for the final weight of element, as the sum of all elements of all lists will then be $1$ and all the element weights will be scaled correctly right away.
Problem
However further research and surveying has shown that transitivity does not work as expected. There is a result mismatch between the weights calculated vs when asked to compare directly, meaning the abovementioned technique calculates that $A_1$ element is $2\times$ as important compared to $B_2$, however when asked to compare directly the surveyee thinks that $A_1$ is $4\times$ more important than $B_2$.
Attempted/proposed solutions
I believe a non linear system is needed to account for such issues, as a first look at results shows that the person places a larger importance to the weight (importance) of the element within the list itself. Originally a simple coefficient less multiplication that gives equal importance was attempted, but didn't work, so adding coefficients or possibly going further and using a polynomial system was an idea of mine. However I do not know enough and am looking for more.
What I'm looking for:
Papers, topics or similar problems that try to resolve this issue will all be helpful
Mathematical example:
HR department weight $= 0.2$
Importance of John in HR $= 0.6$
IT department weight $= 0.4$
Importance of Nina in IT $= 0.1$
When asked to compare directly by the people being surveyed; "How important to success of company is John from HR compared to Nina in IT?", John was found to be $2\times$ as important as Nina on average.
However, $0.2*0.6=0.12$ was the calculated weight for John and $0.4*0.1=0.04$ weight for Nina. This shows that the original system values John $0.14/0.04=3.5$ as much instead of direct comparison where it was $2\times$. So I'm looking for a mathematical way to approximate from the combination of two readings above the closest way to estimate their weight.
I surmise that the simplest model is as follows: you have 2 departments with two people in each and two surveyors ($a$ and $b$) providing two opinions about all 4 people. Let's say that the opinions are given by the matrices $\begin{bmatrix} a_{11} & a_{12}\\ a_{21}& a_{22}\end{bmatrix}$ and $\begin{bmatrix} b_{11} & b_{12}\\ b_{21}& b_{22}\end{bmatrix}$ (rows are the departments and columns are people).
Now, the sums are normalized to the total weight $1$ for both surveyors, so $\sum_{i,j}a_{ij}=\sum_{i,j}b_{ij}=1$. The department $i$ importance is then, apparently, computed as $D_i=\sum_j(a_{ij}w_a+b_{ij}w_b)$ where $w_a$ and $w_b$ are the weights you give to the opinion of the surveyors $a$ and $b$. Now the importance of the person within the department can be reported in two ways.
The first one is that you compute the importance of each person $(i,j)$ within the whole firm as an average and then report the ratio of that to $D_i$. That way would agree perfectly with the product rule you started with and cause no discrepancy.
The second way, however, is to compute the importance of each person within the department first renormalizing the surveyor ranking of each department to $1$, i.e., to get $$ P_{ij}^i=w_a\frac{a_{ij}}{\sum_j a_{ij}}+w_b\frac{b_{ij}}{\sum_j b_{ij}} $$ and you see that the product rule $w_a a_{ij}+w_b b_{ij}=P_{ij}=D_iP_{ij}^i$ fails in this case.
Moreover, in this case just the information about department rankings and the rankings within each department alone is insufficient for the recovery of individual average rankings (because the product rule gives another possibility consistent with all averages and achievable if both surveyors have exactly the same opinion). So, you cannot recover them by any clever algebraic tricks but have to request and use additional information instead.