The task
I need to convert the coordinate system to +X forward, +Y right and +Z up (left-handed, like the one in Unreal Engine). The crucial part is that I want my camera to face its forward axis (again, like in Unreal Engine). Here is how it works in Unreal:

The problem
I have managed to achieve both things: my camera now faces its forward direction and world coordinates are the same as in UE, HOWEVER, I've stumbled upon a big issue. For each object:
- pitch rotation (around RightVector) is now clockwise
- yaw rotation (around UpVector) is now clockwise
- roll rotation (around ForwardVector) is counter-clockwise (as it was before)
I need these rotations to be all counter-clockwise (as per standard) and keep the camera facing the forward vector.
My current solution attempt
My current setup relies on rotating the projection and camera view matrices (model matrix in my code is called entity matrix):
#define GLM_FORCE_LEFT_HANDED // it's defined somewhere else but so you're aware
// Transform::VectorForward = (1, 0, 0) // these are used below
// Transform::VectorRight = (0, 1, 0)
// Transform::VectorUp = (0, 0, 1)
// The entity matrix update function which updates the model matrix for each object
virtual void UpdateEntityMatrix()
{
auto m = glm::translate(glm::mat4(1.f), m_Transform.Location);
m = glm::rotate(m, glm::radians(m_Transform.Rotation[2]), Transform::VectorUp); // yaw rotation
m = glm::rotate(m, glm::radians(m_Transform.Rotation[1]), Transform::VectorRight); // pitch rotation
m = glm::rotate(m, glm::radians(m_Transform.Rotation[0]), Transform::VectorForward); // roll rotation
m_EntityMatrix = glm::scale(m, m_Transform.Scale);
}
// The entity matrix update in my camera class (child of entity) which also updates the view matrix
void Camera::UpdateEntityMatrix()
{
SceneEntity::UpdateEntityMatrix();
auto m = glm::translate(glm::mat4(1.f), m_Transform.Location);
m = glm::rotate(m, glm::radians(180.f), GetRightVector()); // NOTE: I'm getting the current Forward/Right/Up vectors of the entity
m = glm::rotate(m, glm::radians(m_Transform.Rotation[2]), GetUpVector()); // yaw rotation
m = glm::rotate(m, glm::radians(m_Transform.Rotation[1]), GetRightVector()); // pitch rotation
m = glm::rotate(m, glm::radians(m_Transform.Rotation[0]), GetForwardVector()); // roll rotation
m = glm::scale(m, m_Transform.Scale);
m_ViewMatrix = glm::inverse(m);
m_ViewProjectionMatrix = m_ProjectionMatrix * m_ViewMatrix;
}
void PerspectiveCamera::UpdateProjectionMatrix()
{
m_ProjectionMatrix = glm::perspective(glm::radians(m_FOV), m_Width / m_Height, m_NearClip, m_FarClip);
// we want the camera to face +x:
m_ProjectionMatrix = glm::rotate(m_ProjectionMatrix, glm::radians(-90.f), Transform::VectorUp);
m_ProjectionMatrix = glm::rotate(m_ProjectionMatrix, glm::radians(90.f), Transform::VectorRight);
}
This is my result so far (the camera visualization thing shows the rotation of currently used camera):

What else I tried
I tried modifying the model matrix (note the -):
virtual void UpdateEntityMatrix()
{
auto m = glm::translate(glm::mat4(1.f), m_Transform.Location);
m = glm::rotate(m, glm::radians(-m_Transform.Rotation[2]), Transform::VectorUp); // yaw rotation
m = glm::rotate(m, glm::radians(-m_Transform.Rotation[1]), Transform::VectorRight); // pitch rotation
m = glm::rotate(m, glm::radians(m_Transform.Rotation[0]), Transform::VectorForward); // roll rotation
m_EntityMatrix = glm::scale(m, m_Transform.Scale);
}
But it makes my camera not face forward anymore (the view is the camera's rotation but mirrored on the Y and Z axis):

Itried to fix it by applying the same change when calculating the camera view matrix but it didn't help (still mirrored or some other issues).
On top of that, I tried experimenting with the glm::lookAt functions to create the view matrix but never achieved anything.
Update: I've found a solution
I think my problem was actually defining a counter-clockwise rotation. It turns out Unreal's rotations are not consistently counter-clockwise. I think it's best to define a counter-clockwise rotation when looking in the direction pointed by the direction arrow - as if you were behind it. Here is my conclusion:

Conclusion
Unless somebody finds an errorin the system I defined, I'll stick to it. The code is contained in the My current solution attempt section. It seems I was correct from the get-go.
However, the answer provided by @paddy does solve my original issue - converting clockwise to counter-clockwise. Upon further attempts, I was able to correctly replicate Unreal's system while keeping my camera facing forward.