I'm trying to make a simplified sailing ship simulation and want to get the angel-of-attack between a surface normal vector (representing a sail) and a vector (representing the wind velocity). The result should preferably be in the range of 0-180 deg.
I can calculate the angle between the surface normal and the wind vector, but that is always the positive angle between the two, but I need to differentiate between the angle-of-attacks according to the illustration below.
For example I need to figure out the difference between situation 2 and 4 (since one is generating positive lift and the other negative lift). Both give a 45 deg angle between the two vectors, but I need to differentiate the two, the former should be 45 deg and the latter should be 135 deg (see the desired output written in orange next to each situation).
Just for some more context. I model in 3D space, and the 2D illustration is just to give a general idea of my problem. One could picture the examples in the illustration as a sail viewed from above where for example situation 1 would generate no lift and no drag, situation 2 would generate medium lift and drag, and situation 3 would generate no lift but maximum drag, etc.
I believe I need an additional vector representing the concept of "up" or something in those lines, but I'm unsure. I'm not that good at linear algebra... any ideas how to calculate this.
Assuming the sail is a flat plane as shown in the figures (it isn't, but let's ignore that problem for a moment), you have two choices for the normal vector of that plane. Whatever choice you made when you initially modeled the plane, you can always find the other choice of normal vector by simply reversing the modeled normal vector (multiply by $-1$).
If you want to have a vector representing the "up" direction (the direction of positive lift provided by the sail) for the interaction between the flat-plate sail and the wind, you can choose the normal vector that has a positive dot product with the wind vector. In other words, you can easily change examples 6, 7, and 8 in your diagram to examples 2, 3, and 4 respectively. This gives good results because (for example) the angle you want to find in example 2 is the same as the angle in example 6.
If the chosen normal vector is $\hat{\mathbf n}$ and the wind vector is $\mathbf w$, project $\hat{\mathbf n}$ onto $\mathbf w$ and subtract the result from $\hat{\mathbf n}$. That gives you a vector perpendicular to the wind in the "up" direction:
$$ \mathbf n_\perp = \hat{\mathbf n} - \frac{\hat{\mathbf n}\cdot\mathbf w}{\mathbf w\cdot\mathbf w} \mathbf w. $$
The angle of the sail to the wind also can be found by using the dot product: $$ \theta = \arcsin\left(\frac1{\lVert\mathbf w\rVert}\mathbf w \cdot \hat{\mathbf n}\right).$$
Usually we use arccos to get an angle here, but you want $\theta$ to be zero when the wind is perpendicular to the normal vector. Also be careful about whether your angles are in degrees or radians; usually arcsin gives you the angle in radians.
This means you get an angle in the range of $0$ to $90$ degrees, not the $0$ to $180$ degrees you were asking for. For the flat plate sail, however, this should not be a problem. Your "$135$ degree" example (case 4) is just what you get if you reflect your "$45$ degree" example (case 2), including the sail and the direction of the "up" vector, across the line of the wind vector. The reversal of the "up" vector means that we can treat case 4 as a "$45$ degree" example too and get the correct amount of lift in the correct direction.
But sails are not flat plates. They are typically made of fabric that gets "filled" by the wind.
For some sails, such as the mainsail of a Bermuda-rigged sloop when it's close-hauled, the filling of the sail may affect the coefficients of lift and drag, but its shape is essentially independent of which side of the sail the wind comes from; if you mirror-image the wind relative to the rigging of the sail, it mirror-images the shape of the sail.
On the other hand, for a typical square-rigged sail, a wind on one side of the sail fills the sail out in a nice smooth shape, but a wind on the other side of the sail blows the sail against the mast, which causes the sail to form separate pockets on each side of the mast.
In other cases there may be spars or stays in the way of the sail on one side that make the sail not perform symmetrically when the wind is on the "other" side of the sail.
In short, masts, spars, and other rigging on one side of the sail can mean that case 2 in your example is not really the same as case 6. If the normal vector points away from any rigging that would interfere with the shape of the sail, case 2 is the case where the sail has a "good" shape (with no interference from the rigging) and case 6 is the case where the sail is blown against the rigging.
Many sails also have asymmetric edges. A typical square-rigged sail probably performs the same regardless of whether the left edge or the right edge is upwind. But a Genoa jib has a luff (an edge attached to the forestay, which holds that edge nearly along a straight line) and a leach (a vertical edge not attached to the forestay), which can bend quite a lot more. When the boat is sailing upwind the airflow runs along the sail from the luff to the leach; downwind, the airflow is reversed. Depending on how closely you want your simulation to follow real life, you might need to take effects like this into account.
If you do want to account for effects like this in all possible cases of sails rigged a particular way with wind coming from any direction, $0$ to $180$ degrees is not enough. You need something like $0$ to $360$ degrees or $-180$ to $180$ degrees. (I would probably choose $-180$ to $180$.)
Not knowing how much fidelity you want in your simulation, I've presented just some of the things that affect how sails interact with the wind. (I haven't even yet touched on the way the wind near the water tends to be different from the wind higher up, or the three-dimensional shapes of sails.) The flat-plate model probably gives a reasonable result if you pick suitable coefficients of lift and drag at each angle of incidence.