Compiling 32-bit vs 64-bit project using CMake

How do I specify that CMake should use a different link_directories value depending on whether the target is 32-bit or 64-bit? For example, 32-bit binaries need to link with 32-bit Boost, 64-bit binaries need to link with 64-bit Boost.