Threads hang until i attach debugger

I have a service using WCF. Internally it has a dictionary with lists that you can add to or get a subset of from different endpoints.

The code is something like this:

        List<Data> list = null;
        try
        {
            locker.EnterReadLock();
            list = internalData[Something].Where(x => x.hassomething()).ToList();
        }
        finally
        {
            locker.ExitReadLock();
        }

        foreach (var y in list)
        {
            result[y.proprty1].Add(y.property2); // <-- here it hangs
        }

        return result;

So the internalData is locked with a ReaderWriterLockSlim for all operations, readerlock for reading and writerlock for adding. I make a copy of the items inside the lock and work on this copy later.

The issue is after a while, more and more cpu-cores goes to 100% and finally is uses all cores. It can run perfectly for days and million calls before it stops.

Attaching debugger and pausing shows that one treads hang on adding to the result dictionary. But as soon as I resume all threads will continue and a lot of memory is being released.

Is there something special happening when a debugger is attached, pausing and resuming that will release something like this?