Mat not enough in EmguCV? Matrix<> load an image from a file itself? For instance,
Mat img = new Mat(path);
is a valid operation. But,
Matrix<byte> img = new Matrix<byte>(path);
or,
Matrix<byte> img = Matrix<byte>.FromFile(path);
aren't valid operations.
Based on the information from the Emgu Wiki the fundamental difference between the two types is whether the underlying data array is managed or not.
Mat is a wrapper around the C++ cv::Mat class. Generally this class acts as a smart pointer which manages the memory allocated for the data array it owns (although it's able to just observe as well -- a good example of this capability is the ability to return a Mat header for a Matrix instance in C#). This means that OpenCV is able to (re)allocate the memory as necesssary. The trade-off is that in such cases it's more difficult to access the underlying data effectively in C#.
The Matrix class uses a managed array to hold the data. That means you can easily access the underlying data array in C#.
Honestly, the best person to tell you why it's not possible to load Matrix from an image file would be the author. My guess would be that it's intended to represent other things than images. Technically this could be added in the same way as the ability to load an image file was given to the Mat wrapper (the C++ equivalent has no such feature).