I believe that it is safer that each member should have his or her encryption and decryption keys that no one else knows. IN this case a message $m$ is sent as $m^{e_1}$ the receiver sends $t$ back as $m^{e_1e_2}$ the sender decrypts his/her by $m^{e_1e_2d_1}$ and the final decryption is $m^{e_1e_2d_1d_2}=m$ by Fermat little theorem.
Thank you for your help
You're using the Shamir three-pass protocol. Here both $e_1,d_1$ are only known to the sender, and $e_2,d_2$ only to the receiver. So I don't understand what your problem is. Or are you describing another protocol?