pytest warningsfilter breaks tests using pytest.warns(..)

I have a strange error using filterwarnings in pytest. I'm trying to filter the warnings that are raise on MDAnalysis. Currently I have the following config.

[tool:pytest]
filterwarnings= always

Using that config all the tests pass on python 2 and 3 travis. This results in a lot of warnings since MDAnalysis uses a lot of warnings to inform the user about various things. Now to filter the warnings down to see everything except User and some RuntimeWarnings I use the following config.

[tool:pytest]
filterwarnings= all
                  ignore::UserWarning
                  ignore:invalid value encountered in.*:RuntimeWarning

Now the some tests fail only on python 2 Travis. The tests check that if a UserWarning is raised. Strangely this only applies to a handful of tests. The mosts tests that use pytest.warns(UserWarning) still work fine. I do not understand why this only affects a handful of tests on python 2. Are warnings treated differently on python 2 vs python 3?