The key difference between 'synchronized' and 'volatile', is that 'synchronized' can make threads pause, whereas volatile can't.

'caches and registers' isn't a thing. The book says that because in practice that's usually how things are implemented, and it makes it easier (or perhaps not, given these questions) to understand the how & why of the JMM (java memory model).

The JMM doesn't name them, however. All it says is that the VM is free to give each thread its own local copy of any variable, or not, to be synchronized at some arbitrary time with some or all other threads, or not... unless there is a happens-before relationship anywhere, in which case the VM must ensure that at the point of execution between the two threads where a happens before relationship has been established, they observe all variables in the same state.

In practice that would presumably mean to flush caches. Or not; it might mean the other thread overwrites its local copy.

The VM is free to implement this stuff however it wants, and differently on every architecture out there. As long as the VM sticks to the guaranteed that the JMM makes, it's a good implementation, and as a consequence, your software must work given solely those guarantees and no other assumptions; because what works on your machine might not work on another if you rely on assumptions that aren't guaranteed by the JMM.

Reordering

Reordering is also not in the VM spec, at all. What IS in the VM spec are the following two concepts:

1. Within the confines of a single thread, all you can observe from inside it is consistent with an ordered view. That is, if you write 'x = 5; y = 10;' it is not possible to observe, from within the same thread, y being 10 but x being its old value. Regardless of synchronized or volatile. So, anytime it can reorder things without that being observable, then the VM is free to. Will it? Up to the VM. Some do, some don't.

2. When observing effects caused by other threads, and you have not established a happens-before relationship, you may see some, all, or none of these effects, in any order. Really, anything can happen here. In practice, then: Do NOT attempt to observe effects caused by other threads without establishing a happens-before, because the results are arbitrary and untestable.

Happens-before relationships are established by all sorts of things; synchronized blocks obviously do it (if your thread is frozen trying to acquire a lock, and it then runs, any synchronized blocks on that object that finished 'happened before', and anything they did you can now observe, with the guarantee that what you observe is consistent with those things having run in order, and where all data they wrote you can see (as in, you won't get an older 'cache' or whatnot). Volatile accesses do as well.

Atomicity

Yes, your interpretation of why x++ is not atomic even if x is volatile, is correct.

I'm not sure what your Q4 is trying to ask.

In general, if you want to atomically increment an integer, or do any of many other concurrent-esque operations, look at the java.util.concurrent package. These contain efficient and useful implementations of various concepts. AtomicInteger, for example, can be used to atomically increment something, in a way that is visible from other threads, while still being quite efficient (for example, if your CPU supports Compare-And-Set (CAS) operations, Atomicinteger will use it; not something you can do from general java without resorting to Unsafe).

Just to complement rzwitserloot excellent answer:

A1. You can think of it like so: synchronized guarantees that all cashed changes will become visible to other threads that enter a synchronized block (flushed from the cache) once the first thread exits the synchronized block and before the other enters.

A2. Operations executed by a thread T1 within a synchronized block appear to some other thread T2 as not reordered if and only if T2 synchronizes on the same guard.

A3. I'm not sure what you understand by that. It may happen that when incrementing both threads will first perform a read of the variable a which will yield some value v, then both threads will locally increase their local copy of the value v producing v' = v + 1, then both threads will write v' to a. Thus finally the value of a could be v + 1 instead of v + 2.

A4. Basically yes, although in a synchronized block you can perform atomically many operations, while atomic variables allow you to only do a certain single operation like an atomic increment. Moreover, the difference is that when using the synchronized block incorrectly, i.e. by reading variables outside of a synchronized block which are modified by another thread within a synchronized block, you can observe them not-atomically and reordered. Something which is impossible with atomic variables. Locking is exactly the same as synchronized.

Q1. I feel second paragraph above (of volatile) corresponds to what book said about synchronized.

Sure. volatile access can be regarded as synchronization lite.

But is there synchronized-equivalent to volatile's first paragraph? In other words, does using synchronized ensures any / some variables not getting cached in processor caches and registers?

The book has confused you by mixing levels. volatile access has nothing directly to do with processor caches or registers, and in fact the book is surely incorrect about the caches. Volatility and synchronization are about the inter-thread visibility of certain actions, especially of writes to shared variables. How the semantics are implemented is largely a separate concern.

In any case, no, synchronization does not put any constraints on storage of variables. Everything to do with synchronized semantics happens at the boundaries of synchronized regions. This is why all accesses to a given variable from a set of concurrently running threads must be synchronized on the same object in order for a program to be properly synchronized with respect to that variable.

Book says following about volatile in the context of reordering:

When a field is declared volatile, the compiler and runtime are put on notice that this variable is shared and that operations on it should not be reordered with other memory operations.

Q2. Book does not say anything about reordering in the context of synchronized. Can someone explain what can be said of reordering in the context of synchronized?

But that already does say something (not everything) about synchronized access. You need to understand that a "memory operation" in this sense is a read or write of a shared variable, or acquiring or releasing any object's monitor. Entry to a synchronized region involves acquiring a monitor, so already the book says, correctly, that volatile accesses will not be reordered across the boundaries of a synchronized region.

More generally, reads of shared variables will not be reordered relative to the beginning of a synchronized region, and writes will not be reordered relative to the end of one.

Q3. I guess this means two threads can see volatile int a together, both will increment it and then save it. But only one last read will have effect, thus making whole "read-increment-save" non atomic. Am I correct with this interpretation on non-atomic-ness of volatile?

Yes. The autoincrement operator performs both a read and a write of the variable to which it is applied. If that variable is volatile then volatile semantics apply to those individually, so other operations on the same variable can happen between if there is no other protection.

Q4. Does all locking-equivalent are comparable and have same visibility, ordering and atomicity property: synchronized blocks, atomic variables, locks?

Huh? This sub-question is much too broad. You're reading a whole book about it. Generally speaking, though, no these mechanisms have some characteristics in common and some that differ. All have effects on on the visibility of memory operations and their ordering, but they are not identical. "Atomicity" is a function of the other two.