App config for dynamically loaded assemblies

I'm trying to load modules into my application dynamically, but I want to specify separate app.config files for each one.

Say I have following app.config setting for main app:

<appSettings>
  <add key="House" value="Stark"/>
  <add key="Motto" value="Winter is coming."/>
</appSettings>

And another for library that I load using Assembly.LoadFrom:

<appSettings>
  <add key="House" value="Lannister"/>
  <add key="Motto" value="Hear me roar!"/>
</appSettings>

Both libraries have a class implementing the same interface, with the following method:

public string Name
{
    get { return ConfigurationManager.AppSettings["House"]; }
}

And sure enough calls to Name from both main class and loaded assembly class output Stark.

Is there a way to make main app use its own app.config and each loaded assembly use theirs? Names of config files are different in the output, so that should be possible I think.