I've heard that Smalltalk is also an OS. How can I install Smalltalk on my X86 instead of Linux or windows?

I don't know if there is a standalone smalltalk environment, but you can try squeak, which runs above your os (linux, windows or mac). Not just an os but a complete working environment.

There was a project called SqueakNOS. That was the only post-PARC experiment to use Smalltalk directly loaded over the hardware, with no other operating system. It had even a version to work on an OLPC.

I don't know if currently someone follows that development, but it's very interesting and an excellent place to learn about hardware, in human-readable (but executable) terms. You can see and debug how a device driver works, for instance.

Smalltalk Virtual Machine is handling memory allocation, access to file systems, network connections, process (green thread)...

So viewed from Smalltalk world, it is very similar to an OS.

But how to implement the virtual machine without an underlying OS?

There is a project SqueakNOS and a fork PharoNOS meaning No Operating System, whose goal is to completely bypass the OS, and have Squeak/Pharo running on bare hardware.
So yes, it's possible.
But what you'll get with this is still a Smalltalk world, maybe not what you expect from a linux or windows distribution...