I'd like to get the width and height of the red rectangle with this constraints:
- Maximize the area of the red rectangle
- The center of the rotation is the center of the original (dotted) rectangle.
- The red rectangle must have the same proportions as the original (dotted) rectangle
- The sides of the red rectangle are parralel to the dotted one.
- The red rectangle is contained in the blue one.
any help ?

Clearly the maximal inner (red) rectangle will have the endpoints of at least one of its diagonals on the sides of the rotated (blue) rectangle. This suggests the following procedure: Find the intersections of the two diagonals of the original rectangle with the rotated one. Take the shorter of the two resulting line segments as the diagonal of the inscribed rectangle. This rectangle will be unique up to translation.