Sandbox against malicious code in a Java application

1. Run the untrusted code in its own thread. This for example prevents problems with infinite loops and such, and makes the future steps easier. Have the main thread wait for the thread to finish, and if takes too long, kill it with Thread.stop. Thread.stop is deprecated, but since the untrusted code shouldn't have access to any resources, it would be safe to kill it.

2. Set a SecurityManager on that Thread. Create a subclass of SecurityManager which overrides checkPermission(Permission perm) to simply throw a SecurityException for all permissions except a select few. There's a list of methods and the permissions they require here: Permissions in the Java^TM 6 SDK.

3. Use a custom ClassLoader to load the untrusted code. Your class loader would get called for all classes which the untrusted code uses, so you can do things like disable access to individual JDK classes. The thing to do is have a white-list of allowed JDK classes.

4. You might want to run the untrusted code in a separate JVM. While the previous steps would make the code safe, there's one annoying thing the isolated code can still do: allocate as much memory as it can, which causes the visible footprint of the main application to grow.

JSR 121: Application Isolation API Specification was designed to solve this, but unfortunately it doesn't have an implementation yet.

This is a pretty detailed topic, and I'm mostly writing this all off the top of my head.

But anyway, some imperfect, use-at-your-own-risk, probably buggy (pseudo) code:

ClassLoader

class MyClassLoader extends ClassLoader {
  @Override
  public Class<?> loadClass(String name) throws ClassNotFoundException {
    if (name is white-listed JDK class) return super.loadClass(name);
    return findClass(name);
  }
  @Override
  public Class findClass(String name) {
    byte[] b = loadClassData(name);
    return defineClass(name, b, 0, b.length);
  }
  private byte[] loadClassData(String name) {
    // load the untrusted class data here
  }
}

SecurityManager

class MySecurityManager extends SecurityManager {
  private Object secret;
  public MySecurityManager(Object pass) { secret = pass; }
  private void disable(Object pass) {
    if (pass == secret) secret = null;
  }
  // ... override checkXXX method(s) here.
  // Always allow them to succeed when secret==null
}

Thread

class MyIsolatedThread extends Thread {
  private Object pass = new Object();
  private MyClassLoader loader = new MyClassLoader();
  private MySecurityManager sm = new MySecurityManager(pass);
  public void run() {
    SecurityManager old = System.getSecurityManager();
    System.setSecurityManager(sm);
    runUntrustedCode();
    sm.disable(pass);
    System.setSecurityManager(old);
  }
  private void runUntrustedCode() {
    try {
      // run the custom class's main method for example:
      loader.loadClass("customclassname")
        .getMethod("main", String[].class)
        .invoke(null, new Object[]{...});
    } catch (Throwable t) {}
  }
}

Obviously such a scheme raises all sorts of security concerns. Java has a rigorous security framework, but it isn't trivial. The possibility of screwing it up and letting an unprivileged user access vital system components shouldn't be overlooked.

That warning aside, if you're taking user input in the form of source code, the first thing you need to do is compile it to Java bytecode. AFIAK, this cannot be done natively, so you'll need to make a system call to javac, and compile the source code to bytecode on disk. Here's a tutorial that can be used as a starting point for this. Edit: as I learned in the comments, you actually can compile Java code from source natively using javax.tools.JavaCompiler

Once you have JVM bytecode, you can load it into the JVM using a ClassLoader's defineClass function. To set a security context for this loaded class you will need to specify a ProtectionDomain. The minimal constructor for a ProtectionDomain requires both a CodeSource and a PermissionCollection. The PermissionCollection is the object of primary use to you here- you can use it to specify the exact permissions the loaded class has. These permissions should be ultimately enforced by the JVM's AccessController.

There's a lot of possible points of error here, and you should be extremely careful to completely understand everything before you implement anything.

Java-Sandbox is a library for executing Java code with a limited set of permissions.

It can be used to allow access to only a set of white-listed classes and resources. It doesn't seem to be able to restrict access to individual methods. It uses a system with a custom class loader and security manager to achieve this.

I have not used it but it looks well designed and reasonably well documented.

@waqas has given a very interesting answer explaining how this is possible to implement yourself. But it is much safer to leave such security critical and complex code to experts.

Note: The project has not been updated since 2013 and the the creators describe it as "experimental". Its home page has disappeared but the Source Forge entry remains.

Example code adapted from the project web site:

SandboxService sandboxService = SandboxServiceImpl.getInstance();

// Configure context 
SandboxContext context = new SandboxContext();
context.addClassForApplicationLoader(getClass().getName());
context.addClassPermission(AccessType.PERMIT, "java.lang.System");

// Whithout this line we get a SandboxException when touching System.out
context.addClassPermission(AccessType.PERMIT, "java.io.PrintStream");

String someValue = "Input value";

class TestEnvironment implements SandboxedEnvironment<String> {
    @Override
    public String execute() throws Exception {
        // This is untrusted code
        System.out.println(someValue);
        return "Output value";
    }
};

// Run code in sandbox. Pass arguments to generated constructor in TestEnvironment.
SandboxedCallResult<String> result = sandboxService.runSandboxed(TestEnvironment.class, 
    context, this, someValue);

System.out.println(result.get());

You will probably need to use a custom SecurityManger and/or AccessController. For lots of detail, see Java Security Architecture and other security documentation from Sun.