I have a plane defined in space by 3 points and I would like to rotate this plane around the axis which is formed by the first two points of this plane. To do this, I used the rotation matrix suited for this purpose, taken from wikipedia. (Rotation matrix). However, I seem to get wrong results, and can not find what I am doing wrong. I used the following code in Matlab:
clc
clear all
Here I am defining the 3 points for the plane
lm1 = [1,0,6];
lm2 = [2,3,2];
lm3 = [1.5,2,1];
I want to rotate pont lm3 around the axis between lm1 and lm2
rotation = 90;
theta = degtorad(rotation);
Defining the rotation axis between lm1 and lm2 and make a unit vector of it
rot_axis = [lm2(1)-lm1(1), lm2(2) - lm1(2), lm2(3) - lm1(3)];
urot = rot_axis/norm(rot_axis);
Defining the rotation matrix (as taken from Wiki)
R = [cos(theta) + urot(1)^2*(1-cos(theta)), urot(1)*urot(2)*(1-cos(theta))-urot(3)*sin(theta), urot(1)*urot(3)*(1-cos(theta)) + urot(2)*sin(theta);...
urot(2)*urot(1)*(1-cos(theta)) + urot(3)*sin(theta), cos(theta) + urot(2)^2*(1-cos(theta)), urot(2)*urot(3)*(1-cos(theta)) - urot(1)*sin(theta);...
urot(3)*urot(1)*(1-cos(theta))-urot(2)*sin(theta), urot(3)*urot(2)*(1-cos(theta))+ urot(1)*sin(theta), cos(theta) + urot(3)^2*(1-cos(theta))]
Calculate new lm3 after rotation around the axis between lm1 and lm2
lm3_new = lm3*R
Plotting to check the results
plane_initial = [lm1', lm2', lm3'];
plane_rotated = [lm1', lm2', lm3_new'];
figure
fill3(plane_initial(1,:),plane_initial(2,:),plane_initial(3,:),'r')
hold on
fill3(plane_rotated(1,:),plane_rotated(2,:),plane_rotated(3,:),'c')
grid on
xlabel('X')
ylabel('Y')
zlabel('Z')
vector on old plane
vec_old = [lm3(1)-lm2(1), lm3(2) - lm2(2), lm3(3) - lm2(3)];
vector on new plane
vec_new = [lm3_new(1)-lm2(1), lm3_new(2) - lm2(2), lm3_new(3) - lm2(3)];
Checking the angle between those two vectors on both planes
angle_check = atan2d(norm(cross(vec_old,vec_new)),dot(vec_old,vec_new))
The planes should now have an angle of 90 degrees with each other. However, both the anglecheck (= 41 degrees) and the plot see here for 3D-plot show different results. I have checked the rotation matrix multiple times for hours but I think it should be correct. I was wondering if anyone has experience with this and can see the mistake. Thanks in advance!

Your rotation matrix rotates about an axis that passes through the origin. Whatever source you cribbed it from most likely mentions that somewhere. Unless you happen to be very lucky, the rotation axis defined by your two points doesn’t. So, what you’re doing is rotating about an axis that’s parallel to the one you want.
There are several ways to fix this problem. The simplest for your code would be to translate
lm3by an amount that puts the rotation axis through the origin, rotate using the method you already have, then translate back. That is, rotate eitherlm3-lm1orlm3-lm2, then addlm1orlm2as appropriate to the result.