Failed to apply plugin "com.google.gms.google-services"

Here is my project build.gradle:

buildscript {
    ...
    dependencies {
        ...
        classpath 'com.google.gms:google-services:3.1.0'
    }
}

In one of the module's build.gradle I have this:

apply plugin: 'com.google.gms.google-services'

which results into the following error:

Could not get unknown property 'LibraryVariants' for object of type com.android.build.gradle.LibraryExtension.

Fun fact, if I set google-services version to 3.0.0, this specific error disappears, but Gradle asks me to downgrade versions of other google libraries, but I really do not want to do that.

How do I deal with these LibraryVariants?