Jenkins not running any jobs

user_1140783 0
Viewed 826

I've managed to break Jenkins while trying to fix some other problems. Now whenever I run a job it immediately fails with the following error:

java.lang.NullPointerException
at java.util.TreeMap.putAll(TreeMap.java:313)
at hudson.slaves.EnvironmentVariablesNodeProperty.buildEnvVars(EnvironmentVariablesNodeProperty.java:85)
at hudson.model.Computer.buildEnvironment(Computer.java:1195)
at hudson.model.Job.getEnvironment(Job.java:385)
at hudson.model.Run.getEnvironment(Run.java:2419)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getEnvironment(WorkflowRun.java:500)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:112)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:68)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:310)
at hudson.model.ResourceController.execute(ResourceController.java:99)
at hudson.model.Executor.run(Executor.java:432)
Finished: FAILURE

I found this answer Jenkins - Problem: hudson.model.Queue$MaintainTask failed but it suggests rewriting the config.xml file. When I do that and reload config from the disk, config.xml files gets overwritten by jenkins. I have upgraded jenkins to the latest available version and also updated most of the plugins. I also tried to add back the environment variables through the UI but doing that throws the following exception:

2022-01-06 17:21:50.894+0000 [id=10]    WARNING o.e.j.s.h.ContextHandler$Context#log: Error while serving https://<jenkins-url>:<port>/configSubmit
java.lang.ClassCastException: java.lang.Integer cannot be cast to hudson.model.Describable
        at hudson.util.DescribableList.get(DescribableList.java:126)
        at hudson.util.DescribableList.rebuild(DescribableList.java:168)
        at jenkins.model.GlobalNodePropertiesConfiguration.configure(GlobalNodePropertiesConfiguration.java:25)
        at jenkins.model.Jenkins.configureDescriptor(Jenkins.java:3846)
        at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:3810)
        at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
Caused: java.lang.reflect.InvocationTargetException
        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:400)
        at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
        at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
        at org.kohsuke.stapler.SelectionInterceptedFunction$Adapter.invoke(SelectionInterceptedFunction.java:36)

which looks similar to the first exception where it is unable to recognize the Integer field. How can I get this configuration fixed?

How it started I had several scheduled jobs running perfectly fine. But suddenly after weeks of successful runs they started to fail without any obvious reason. But they were still running to a certain point. I also had a plugin installed called build-monitor-plugin but after installing that I had not restarted jenkins. I decided to restart jenkins to see if that resolves the problem but that landed me into a different set of issues and I never got to figure out why the jobs started to fail after running successfully for several weeks. To find that out I have to resolve the NPE issue first (which I described in the beginning). The folder that had all my jobs now shows a broken view:

enter image description here

However, I am still able to access the jobs using a direct url to them. This folder had a view added to it using the build-monitor-plugin and I suspected it messed up something after the restart. So I uninstalled the plugin but that did not fix the view. I have also manually removed all files related to this plugin but that also does not help. When I visit the folder I see following error in the log:

2022-01-06 05:29:47.232+0000 [id=87]    WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID cb5bdf8c-2a97-4360-8f80-95df2c5c719b
org.apache.commons.jelly.JellyTagException: jar:file:/app/jenkins/war/WEB-INF/lib/jenkins-core-2.319.1.jar!/hudson/model/View/index.jelly:42:43: <st:include> org.apache.commons.jelly.JellyTagException: jar:file:/app/jenkins/war/WEB-INF/lib/jenkins-core-2.319.1.jar!/lib/hudson/projectView.jelly:67:24: <d:invokeBody> `java.lang.NullPointerException
    at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:726)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:281)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)`
jenkins
2 Answers

Finally managed to fix both issues.

What happened? Sequence of installing the plugin-in build-monitor-plugin, restarting Jenkins and then uninstalling the plug-in left some of the files in Jenkins in a corrupted state. A state from which Jenkins version 2.263.3 was unable to recover even though it kept on recreating the files but every time it was creating the same corrupted files again.

Corrupted Files There were two files that got corrupted and both are named config.xml. First one is in the top most level directory of Jenkins. It is supposed to be as follows in case you have environment variables:

  <globalNodeProperties>
    <hudson.slaves.EnvironmentVariablesNodeProperty>
      <envVars serialization="custom">
        <unserializable-parents/>
        <tree-map>
          <default>
            <comparator class="java.lang.String$CaseInsensitiveComparator"/>
          </default>
          <int>2</int> <!-- This number indicates how many key/value pairs will follow it -->
          <string></string>
          <string></string>
          ...
        </tree-map>
      </envVars>
    </hudson.slaves.EnvironmentVariablesNodeProperty>
  </globalNodeProperties>

Something similar was also suggested in Jenkins - Problem: hudson.model.Queue$MaintainTask failed but the value of <comparator> class mentioned there did not work. I figured out the correct value by installing a separate instance of Jenkins and comparing the new config.xml with the old one.

Similarly the other config.xml that was broken was for the folder which was not showing up properly as can be seen in question. For that I created another folder and compared it's config.xml with the broken one and found following two lines missing from the <folderViews>:

  ...
  <folderViews>
    ...
    <primaryView>All</primaryView>
    <tabBar class="hudson.views.DefaultViewsTabBar"/>
  </folderViews>
  ...

Adding these two lines resolved the issue.

user_11407830

Piggybacking on this answer since we just had a very similar error pop up and we had the build-monitor-plugin plugin installed, too:

java.lang.NullPointerException
    at java.util.TreeMap.putAll(Unknown Source)
    at hudson.slaves.EnvironmentVariablesNodeProperty.buildEnvVars(EnvironmentVariablesNodeProperty.java:85)
    at hudson.model.Computer.buildEnvironment(Computer.java:1210)
    at hudson.model.Job.getEnvironment(Job.java:385)
    at hudson.model.AbstractProject.getEnvironment(AbstractProject.java:334)
    at hudson.model.Run.getEnvironment(Run.java:2419)
    at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:943)
    at CUSTOM_PLUGIN

I immediately thought our custom plugin was to blame but reverting to the previous stable version did not change this behavior. Then, finding @Syed Ali's answer, I looked at our config.xml files but they seemed fine.

The problem was that, in the Configuration of the administrative section, the Environment Variables section's checkbox was checked but no key-value pairs had been added. For some reason, unchecking that option fixed any related problem.

user_200430
Related