Liquibase/Springboot startup exception

user_5841981 0
Viewed 1385

Starting yesterday (Sunday) morning my production app fails to start, with no code changes from my side. It's running Springboot 2.3.4, Liquibase-core 3.8.0 and is hosted on Amazon linux2. Funny thing is there are no exceptions locally, only when deployed.

Here is the relevant stack trace:

Caused by: liquibase.exception.UnexpectedLiquibaseException: java.nio.file.NoSuchFileException: /tmp/agent12302722365010540729.jar
 at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:129)
 at liquibase.servicelocator.ServiceLocator.<init>(ServiceLocator.java:69)
 at liquibase.servicelocator.CustomResolverServiceLocator.<init>(CustomResolverServiceLocator.java:16)
 at org.springframework.boot.liquibase.LiquibaseServiceLocatorApplicationListener$LiquibasePresent.replaceServiceLocator(LiquibaseServiceLocatorApplicationListener.java:55)
 at org.springframework.boot.liquibase.LiquibaseServiceLocatorApplicationListener.onApplicationEvent(LiquibaseServiceLocatorApplicationListener.java:44)
 at org.springframework.boot.liquibase.LiquibaseServiceLocatorApplicationListener.onApplicationEvent(LiquibaseServiceLocatorApplicationListener.java:36)
...
Caused by: java.nio.file.NoSuchFileException: /tmp/agent801508645517312012.jar
  at liquibase.resource.ClassLoaderResourceAccessor.getResourcesAsStream(ClassLoaderResourceAccessor.java:53)
  at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:115)

I double checked all application related files and env variables and they are all the same. The file in question is in no way related to my app.

Do you have any idea what this file is and why is Liquibase trying to find it all of the sudden?

liquibase spring-boot java
4 Answers

I had the same problem. On the startup of the amazon linux 2, there is a security patch that is installed.

The package causing the problem is log4j-cve-2021-44228-hotpatch.noarch (you can check that in /var/log/yum.log)

A temporary solution is to uninstall the patch and install another java version.

yum remove log4j-cve-2021-44228-hotpatch.noarch
yum install java-11-openjdk-11.0.12.0.7-0.amzn2.0.2.x86_64

Thanks to @mihristov for the solution.

user_4786630

To elaborate a bit more on what's happening here:

AWS has developed a systemd service that, every second, looks for any JVMs running on the system. For all that it finds and supports it connect a java agent that at runtime replaces the log4j JNDI lookup() method with a hardcoded safe string.

Why liquibase fails on this is not clear to me. I do see that the already given answers seem to be too invasive. There's no need to remove/reinstall/update your JVM. You can work around this by:

  • Stopping the systemd patch service: sudo service log4j-cve-2021-44228-hotpatch status|stop|start

  • Best solution - create the kill file that, if read by the service, stops applying the patch: sudo touch /etc/log4j-cve-2021-44228-hotpatch.kill

user_32540760

For us moving to latest liquibase (4+) resolved this. They use a more standard ServiceLocator in 4+ and that seems to make the difference.

user_9688780

This is a temporary issue with: java-11-amazon-corretto-headless-11.0.13+8-2.amzn2.aarch64

To fix:

yum remove log4j-cve-2021-44228-hotpatch.noarch
yum install java-11-amazon-corretto-headless-11.0.13+8-1.amzn2.aarch64

Thanks to @Reda and @mihristov for pointing the issue out.

Our actual fix was just to install a specific version of corretto instead of the general package

yum install java-11-amazon-corretto-headless-11.0.13+8-1.amzn2.aarch64

You should only do this temporarily until a fix is out!

user_2147150
Related