You could write a library that makes calls directly to the graphics drivers, or you could write a custom pipeline that runs on the cpu, which would functionally be an emulator for dedicated graphics hardware, but it would be complex, unreliable, slow, and would likely cost you a lot of time to implement, with very little return on investment.
As has been mentioned in comments, GPU's work the way they do, because they were designed on a hardware level to do it that way. The OpenGL/DirectX/Vulkan api's just allow you to have more, or less control over that pipeline, and to plug custom shader stages in.
The reason you cannot write your own rasterizer, is because it is a stage in the pipeline, in which is is pretty much as efficient as it can be. Your version is unlikely to be better.
Don't concern yourself too much with the non-programmable parts of the pipeline. That is the purview of hardware engineers and driver programmers.
If you want lots of control over the existing hardware, try DX12 or Vulkan/Metal. They're about as close as you can get to the hardware as a programmer, without writing custom drivers.