Garbage collectors either use tracing or are reference counted. (The EpsilonGC is a no-reclamation GC which uses neither, but that will simply blow up when it reaches the memory limit.)
Generally the JVM uses tracing reference collectors with a stop-the-world pause occasionally because it adds the least impact to the runtime. Reference counted garbage collectors are more predictable, but incur runtime costs during the application's threads to deal with the reference counts.
It's worth noting that whichever garbage collector is picked, it will have different effects on the code generation. For example, JVMs which have single threaded garbage collectors (such as when running on a single CPU) may have less overhead with assigning references than when a multi-threaded or multi-generational garbage collector runs. In those cases, mark cards or region pointers may be needed to highlight where and how references need to be checked. The exact requirements are dependent upon the garbage collector itself; for example, with Shenandoah and ZGC the generated code is different from using G1GC or CMS/Parallel.
So it would be possible to have a JVM GC that used reference counting; instead of generating a reference to a card mark, you could increase or decrease the object count.
However, reference counting GCs have a particular weakness in that they can't detect cycles. If you have A pointing to B and B pointing to A, but otherwise are unreachable, they can't be GC'd if you're using a pure reference counting strategy. Different languages that use reference counting deal with this in different ways; for example, Swift requires the programmer to determine a 'weak' reference which will be broken thereby allowing the cycle to be cleaned up.
The JVM doesn't (easily) provide a way of annotating such weak circles, so it's possible that a reference counting GC with Java would end up leaking such references, or it may require a full tracing pass periodically to evict such cycles. (Yes, you can use various Weak/Phantom references but they're almost a code smell in themselves.)
The other thing you can say is that a reference counted GC doesn't have any additional overhead when in a steady state, because once the object graphs are constructed, you don't need to visit them again. This may result in better cache locality for objects.
If a JVM gets into an allocation free state then the GC doesn't need to run, but likely young GCs will continue, and there are other periodic operations (biased lock revocation etc.) which may inadvertently trigger a GC. Generally though a JVM can be tuned so that its object creation set is in the eden gen only, which is a fairly minimal operation and won't require re-scanning the whole heap periodically.
If you're interested, you can give writing your own GC a go to find out.