Type 'org/eclipse/equinox/http/servlet/HttpServiceServlet' (current frame, stack[0]) is not assignable to 'javax/servlet/http/HttpServlet'

user_8757350 0
Viewed 328

I have a very strange behavior, somethimes it starts without errors or it will give me java.lang.VerifyError: Bad type on operand stack

It's AdoptOpenJDK 11.0.3 i tried also 11.0.9 without changes, it's running in Tomcat 9, i forced to use JSP 2.2 with Servlet 3.0, because it's a very old application.

What are my options for debugging? It's JVM Bug?

Classloading logs:

[157.897s][info   ][class,load] org.eclipse.equinox.http.servletbridge.internal.Activator source: file:/appservers/tomcat/webapps/app/work/eclipse/plugins/org.eclipse.equinox.http.servletbridge_1.1.100.v20180827-1235.jar
[157.899s][info   ][class,load] javax.servlet.Servlet source: file:/appservers/tomcat/webapps/app/work/eclipse/plugins/javax.servlet-3.0.0.v201112011016.jar
[157.899s][info   ][class,load] javax.servlet.ServletConfig source: file:/appservers/tomcat/webapps/app/work/eclipse/plugins/javax.servlet-3.0.0.v201112011016.jar
[157.899s][info   ][class,load] javax.servlet.GenericServlet source: file:/appservers/tomcat/webapps/app/work/eclipse/plugins/javax.servlet-3.0.0.v201112011016.jar
[157.899s][info   ][class,load] javax.servlet.http.HttpServlet source: file:/appservers/tomcat/webapps/app/work/eclipse/plugins/javax.servlet-3.0.0.v201112011016.jar
[157.899s][info   ][class,load] org.eclipse.equinox.http.servlet.internal.servlet.ProxyServlet source: file:/appservers/tomcat/webapps/app/work/eclipse/plugins/org.eclipse.equinox.http.servlet_1.2.0.v20150519-1816.jar
[157.899s][info   ][class,load] org.eclipse.equinox.http.servlet.HttpServiceServlet source: file:/appservers/tomcat/webapps/app/work/eclipse/plugins/org.eclipse.equinox.http.servlet_1.2.0.v20150519-1816.jar
[157.900s][info   ][class,load] org.eclipse.equinox.http.servlet.internal.Activator source: file:/appservers/tomcat/webapps/app/work/eclipse/plugins/org.eclipse.equinox.http.servlet_1.2.0.v20150519-1816.jar
[157.901s][info   ][class,load] org.osgi.service.http.HttpService source: file:/appservers/tomcat/webapps/app/work/eclipse/plugins/org.eclipse.osgi.services_3.7.100.v20180827-1536.jar
[157.901s][info   ][class,load] org.eclipse.equinox.http.servlet.ExtendedHttpService source: file:/appservers/tomcat/webapps/app/work/eclipse/plugins/org.eclipse.equinox.http.servlet_1.2.0.v20150519-1816.jar
[157.901s][info   ][class,load] java.lang.VerifyError source: jrt:/java.base

The error logged in separeted file:

!ENTRY org.eclipse.equinox.http.servletbridge 4 0 2020-12-18 08:29:43.240
!MESSAGE FrameworkEvent ERROR
!STACK 0
org.osgi.framework.BundleException: Error starting module.
    at org.eclipse.osgi.container.Module.doStart(Module.java:593)
    at org.eclipse.osgi.container.Module.start(Module.java:452)
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1685)
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1665)
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1627)
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1558)
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:233)
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:343)
Caused by: java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    org/eclipse/equinox/http/servletbridge/internal/Activator.start(Lorg/osgi/framework/BundleContext;)V @15: invokestatic
  Reason:
    Type 'org/eclipse/equinox/http/servlet/HttpServiceServlet' (current frame, stack[0]) is not assignable to 'javax/servlet/http/HttpServlet'
  Current Frame:
    bci: @15
    flags: { }
    locals: { 'org/eclipse/equinox/http/servletbridge/internal/Activator', 'org/osgi/framework/BundleContext' }
    stack: { 'org/eclipse/equinox/http/servlet/HttpServiceServlet' }
  Bytecode:
    0000000: 2abb 0012 59b7 0021 b500 1f2a b400 1fb8
    0000010: 0022 b1                                

    at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
    at java.base/java.lang.Class.getConstructor0(Class.java:3342)
    at java.base/java.lang.Class.newInstance(Class.java:556)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.loadBundleActivator(BundleContextImpl.java:766)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:719)
    at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1005)
    at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:357)
    at org.eclipse.osgi.container.Module.doStart(Module.java:584)
    ... 8 more
Root exception:
java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    org/eclipse/equinox/http/servletbridge/internal/Activator.start(Lorg/osgi/framework/BundleContext;)V @15: invokestatic
  Reason:
    Type 'org/eclipse/equinox/http/servlet/HttpServiceServlet' (current frame, stack[0]) is not assignable to 'javax/servlet/http/HttpServlet'
  Current Frame:
    bci: @15
    flags: { }
    locals: { 'org/eclipse/equinox/http/servletbridge/internal/Activator', 'org/osgi/framework/BundleContext' }
    stack: { 'org/eclipse/equinox/http/servlet/HttpServiceServlet' }
  Bytecode:
    0000000: 2abb 0012 59b7 0021 b500 1f2a b400 1fb8
    0000010: 0022 b1                                

    at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
    at java.base/java.lang.Class.getConstructor0(Class.java:3342)
    at java.base/java.lang.Class.newInstance(Class.java:556)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.loadBundleActivator(BundleContextImpl.java:766)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:719)
    at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1005)
    at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:357)
    at org.eclipse.osgi.container.Module.doStart(Module.java:584)
    at org.eclipse.osgi.container.Module.start(Module.java:452)
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1685)
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1665)
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1627)
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1558)
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:233)
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:343)
osgi tomcat jsp java
2 Answers

The application is very old and has alot of technical debts, but we are forced to update it every year.... (the approach would be better "don't touch a running system")

So i debugged whole OSGI bundle classloader and found out, that it resolves somethimes the javax.servlet in two different ways, one is the new version and the other is the bundle javax.servlet-3.0. Why osgi randomly switches on restart, i don't know, i don't debugged it, its pure magic. I think its an osgi bug, because it should, throw an error showing up that has two dependency chains, modifying the meta-inf caused somethimes to throw the dependency chain error.

About the problem:
When "Caused by: java.lang.VerifyError: Bad type on operand stack" occours, it tries to assign an old/new version of the class with the same class with another version that has obviously a different byte code.

org.osgi.service.http -> javax.servlet of Tomcat (server-api.jar) javax.servlet -> javax.servlet Bundle

Example:
Class A(V1.0)
Class A(V1.1)

Class B uses Class A(V1.1)
Class C uses Class A(V1.0)
Class C gets reference of Class B and calls method with Class A(V1.0)
Class A(V1.0) != Class A(V1.1) will throw java.lang.VerifyError: Bad type on operand stack

How i resolved it:

we have own bundle that imports, exports javax.servlet, that causes this javax.servlet switching that somethimes fails with java.lang.VerifyError. I removed the javax.servlet import because org.osgi.service.http does import javax.servlet, thats it, now it seems to work.

Update 01.07.2021: So after long time, i resolved the error after debugging the whole osgi plugin / resolver / loading mechanism etc. I found out that a Bundle/Plugin is dynamically created at runtime aka "org.eclipse.equinox.servletbridge.extensionbundle_1.3.0.jar ". from "org.eclipse.equinox.servletbridge" plugin.

From the OSGI documention:

If an OSGi servlet is to be loaded, the ServletBridge which passes on the queries to the respective OSGi Servlet is called by Tomcat. For the OSGi servlets, which are actually only conventional HttpServlets, also to know or use the servlet API, there exists a special Equinox bundle "org.eclipse.equinox.servlet.bridge.extensionbundle", which defines or releases the servlet API or the Tomcat packages in its manifest file. This bundle is generated by the ServletBridge at runtime at every deploy of the plug-ins, i.e. each time Tomcat is started. In order to add extensions, that is to release additional Tomcat packages or to use another servlet API version, this bundle must be patched. Here, in the Incubator bundle "org.eclipse.equinox.servletbridge" there exists the possibility of generating a separate extension bundle with the help of an Ant script.

So i created a dummy file "org.eclipse.equinox.servletbridge.extensionbundle_block_deployment.dummy" and copied into the plugin folder. So that it blocks the creation and registration of another "javax.servlet.http" Code: org.eclipse.equinox.servletbridge.FrameworkLauncher.deployExtensionBundle()

And now the error is gone, because it retrieves "javax.servlet.http" classes only from javax.servlet_3.1.0.v201410161800.jar

I could not delete the javax.servlet_3.1.0.v201410161800.jar because is needed by other Plugins and binded by the plugin name and not over the package import in the manifest file.

user_87573500

Very interesting. What's happening here is that your class is being loaded (the BundleActivator), and then it's being verified with the JVM class loader, and it's failing that verification. It's complaining that the top of the stack is an HttpServiceServlet and that that isn't assignable to javax.servlet.http.HttpServlet.

It may be that you're using a newer version of the JakartaEE spec, which moved from the javax. namespace to the jakartaee. namespace; that would give the error seen. It's likely that the version of the binaries that you're using were compiled against an older version of the spec, and are trying to run it in a newer version of the spec. If you use just the older version that corresponds with the version you need to run against, you should be fine.

user_1306590
Related