Multiple sub-projects depending on the same library

I have a project main_project. It depends on two libraries - libA and libB, which in turn depend on lib_common each.

Graphically project's structure looks like this:

main_project
   |
   +-libA
   |   |
   |   +-CMakeLists.txt
   |
   +-libB
   |   |
   |   +-CMakeLists.txt
   |
   +-lib_common
   |   |
   |   +-CMakeLists.txt
   |
   +-CMakeLists.txt

Since libA uses lib_common it's pretty natural to call add_subdirectory(lib_common) inside libA's CMakeLists.txt. But the libB also depends on lib_common. If I put it in a subdirectory again it will compile twice. And I don't want that. Also I want to keep all my subprojects clean and self-contained - that's why I don't want to put lib_common in top-level main_project's CMakeLists.txt (because lib_common have nothing with main_project).

Which options do I have?