prototype changes from globalscope to enginescope in nashorn

Why your code doesn't work

The global scope can only be used for simple variable mapping. For example:

ScriptContext defCtx = engine.getContext();
defCtx.getBindings(ScriptContext.GLOBAL_SCOPE).put("foo", "hello");

Object lives in the engine scope and thus the global scope is not even searched for any mappings related to it (Object.prototype.test in your case).

An excerpt from the documentation:

The default context's ENGINE_SCOPE is a wrapped instance of ECMAScript "global" object - which is the "this" in top level script expressions. So, you can access ECMAScript top-level objects like "Object", "Math", "RegExp", "undefined" from this scope object. Nashorn Global scope object is represented by an internal implementation class called jdk.nashorn.internal.objects.Global. Instance of this class is wrapped as a jdk.nashorn.api.scripting.ScriptObjectMirror instance. ScriptObjectMirror class implements javax.script.Bindings interface. Please note that the context's GLOBAL_SCOPE Bindings and nashorn global object are different. Nashorn's global object is associated with ENGINE_SCOPE and not with GLOBAL_SCOPE. GLOBAL_SCOPE object of default script context is a javax.script.SimpleBindings instance.The user can fill it with name, value pairs from the java code.

https://wiki.openjdk.java.net/display/Nashorn/Nashorn+jsr223+engine+notes

Solutions

1. Keep using engine scope
2. Use --global-per-engine option by specifying -Dnashorn.args=--global-per-engine in your Java command line. Then Nashorn will use a single instance of the global object for all script evaluations regardless of ScriptContext passed.
3. Use full-blown ScriptContext instead of Bindings:

ScriptEngine engine = new ScriptEngineManager().getEngineByName("nashorn");
SimpleScriptContext context = new SimpleScriptContext();
engine.eval("Object.prototype.test = function(arg){print(arg);}", context);
engine.eval("var x = {}; x.test('hello');", context);

How to run several scripts, every time with the libraries loaded, but nothing left from previous executions

Every time you need a new context with libraries simply create it:

public static void main(String[] args) throws ScriptException {
    ScriptEngine engine = new ScriptEngineManager().getEngineByName("nashorn");
    SimpleScriptContext context1 = createContextWithLibraries(engine);
    //works as expected, printing "hello"
    engine.eval("var x = {}; x.test('hello'); var y = 'world';", context1);
    SimpleScriptContext context2 = createContextWithLibraries(engine);
    //works as expected, printing "hello"
    engine.eval("var x = {}; x.test('hello');", context2);
    //works as expected, printing "world"
    engine.eval("print(y);", context1);
    //fails with exception since there is no "y" variable in context2
    engine.eval("print(y);", context2);
}

private static SimpleScriptContext createContextWithLibraries(ScriptEngine engine) throws ScriptException {
    SimpleScriptContext context = new SimpleScriptContext();
    engine.eval("Object.prototype.test = function(arg){print(arg);}", context);
    return context;
}

Denis has already explained the issue so I won't repeat what he said. I will however give another solution. To avoid the overhead of parsing the same libraries over and over again, you can compile them. Here is a way to do it:

import java.util.*;
import javax.script.*;

public class Test
{
    public static List<CompiledScript> compileScripts(ScriptEngine engine, String... scripts) throws ScriptException
    {
        Compilable compilable = (Compilable)engine;
        ArrayList<CompiledScript> list = new ArrayList<>();
        for(String script : scripts)
            list.add(compilable.compile(script));
        return list;
    }

    public static void execute(ScriptEngine engine, List<CompiledScript> libs, String script) throws ScriptException
    {
        engine.setBindings(engine.createBindings(), ScriptContext.ENGINE_SCOPE);
        for(CompiledScript lib : libs)
            lib.eval();
        engine.eval(script);
    }

    public static void main(String[] args) throws ScriptException
    {
        ScriptEngineManager manager = new ScriptEngineManager();
        ScriptEngine engine = manager.getEngineByName("nashorn");
        List<CompiledScript> libs = compileScripts(engine, "var x = 1", "Object.prototype.test = function(arg){print(arg)}");

        // Prints 'hello'
        execute(engine, libs, "x.test('hello')");

        // Defines y
        execute(engine, libs, "var y = 2");

        // Checks that executions have a clean (non-polluted) context
        // Throws ReferenceError: "y" is not defined in <eval> at line number 1
        execute(engine, libs, "print(y)");
    }
}