Session gets lost when enabling OutputCache

I have a (working) MVC-application that uses Session properties on multiple parts:

return httpContext.Session[SPContextKey] as SharePointAcsContext;

(ignore that this is sharepoint; This problem isn't be SP-specific)

This works fine until I try to enable Outputcaching:

[OutputCache (Duration =600)]
public ActionResult Select() {
  DoSelect();
}

When the content is cached, httpContext.Session becomes NULL.

Is there a way to keep the Session data and also use caching?