Artifactory fails to get response from /api/v1/system/ping during startup

Viewed 3004

Artifactory(6.16.0) was running on the system (Amazon Linux AMI 2018.03, RHEL) until the disk storage run out of memory. After increasing the disk space and a rebooting the system, the artifactory was not able to start anylonger.

Output of /var/opt/jfrog/artifactory/logs/artifactory.log

2020-11-25 16:02:23,125 [art-init] [INFO ] (o.a.s.a.ArtifactoryAccessClientConfigStore:625) - Using Access Server URL: http://localhost:8040/access (bundled) source: detected
2020-11-25 16:02:23,337 [art-init] [INFO ] (o.a.s.a.AccessServiceImpl:387) - Waiting for access server...
2020-11-25 16:02:23,458 [art-init] [WARN ] (o.j.a.c.AccessClientHttpException:41) - Unrecognized ErrorsModel by Access. Original message: Failed on executing /api/v1/system/ping, with response: Not Found
2020-11-25 16:02:23,462 [art-init] [ERROR] (o.a.s.a.AccessServiceImpl:398) - Could not ping access server: {}
org.jfrog.access.client.AccessClientHttpException: HTTP response status 404:Failed on executing /api/v1/system/ping, with response: Not Found
        at org.jfrog.access.client.http.AccessHttpClient.createRestResponse(AccessHttpClient.java:162)
        at org.jfrog.access.client.http.AccessHttpClient.restCall(AccessHttpClient.java:114)
        at org.jfrog.access.client.http.AccessHttpClient.ping(AccessHttpClient.java:96)
        at org.jfrog.access.client.AccessClientImpl.ping(AccessClientImpl.java:211)
        at org.artifactory.security.access.AccessServiceImpl.waitForAccessServer(AccessServiceImpl.java:393)
        at org.artifactory.security.access.AccessServiceImpl.waitForAccessServer(AccessServiceImpl.java:366)
        at org.artifactory.security.access.AccessServiceImpl.bootstrapAccessClient(AccessServiceImpl.java:1250)
        at org.artifactory.security.access.AccessServiceImpl.initAccessService(AccessServiceImpl.java:339)
        at org.artifactory.security.access.AccessServiceImpl.initAccessClientIfNeeded(AccessServiceImpl.java:328)
        at org.artifactory.security.access.AccessServiceImpl.init(AccessServiceImpl.java:321)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
        at org.artifactory.storage.fs.lock.aop.LockingAdvice.invoke(LockingAdvice.java:76)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy56.init(Unknown Source)
        at org.artifactory.spring.ArtifactoryApplicationContext.refresh(ArtifactoryApplicationContext.java:273)
        at org.artifactory.spring.ArtifactoryApplicationContext.<init>(ArtifactoryApplicationContext.java:153)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.artifactory.webapp.servlet.ArtifactoryContextConfigListener.configure(ArtifactoryContextConfigListener.java:211)
        at org.artifactory.webapp.servlet.ArtifactoryContextConfigListener.access$200(ArtifactoryContextConfigListener.java:67)
        at org.artifactory.webapp.servlet.ArtifactoryContextConfigListener$1.run(ArtifactoryContextConfigListener.java:92)

Tail of /var/opt/jfrog/artifactory/access/logs/access.log

2020-11-25 02:00:00,863 [http-nio-8040-exec-7] [INFO ] (o.j.a.s.s.b.AccessImporterExporterImpl:63) - Exporting users...
2020-11-25 02:00:00,962 [http-nio-8040-exec-7] [INFO ] (o.j.a.s.s.b.AccessImporterExporterImpl:63) - Exporting tokens...
2020-11-25 02:00:00,976 [http-nio-8040-exec-7] [INFO ] (o.j.a.s.s.b.AccessImporterExporterImpl:63) - Exporting federation_events...
2020-11-25 02:00:00,985 [http-nio-8040-exec-7] [INFO ] (o.j.a.s.s.b.AccessImporterExporterImpl:63) - Exporting federation_servers...
2020-11-25 02:00:00,993 [http-nio-8040-exec-7] [INFO ] (o.j.a.s.s.b.AccessImporterExporterImpl:63) - Exporting permissions...
2020-11-25 02:00:01,036 [http-nio-8040-exec-7] [INFO ] (o.j.a.s.s.b.AccessImporterExporterImpl:63) - Exporting groups...
2020-11-25 02:00:01,040 [http-nio-8040-exec-7] [INFO ] (o.j.a.s.s.b.AccessImporterExporterImpl:63) - Exporting config...
2020-11-25 02:00:01,092 [http-nio-8040-exec-7] [INFO ] (o.j.a.s.s.b.AccessImporterExporterImpl:63) - Exporting encryption...
2020-11-25 02:00:01,097 [http-nio-8040-exec-7] [INFO ] (o.j.a.s.s.b.AccessImporterExporterImpl:63) - Exporting home...

Tail of /var/opt/jfrog/artifactory/access/logs/request.log

2020-11-25T09:27:38.801+0000|127.0.0.1|jfrt@01cpj3tjark29x0mpgvnyj1bs1|GET|http://localhost:8040/access/api/v1/permissions/last_updated|200|0|0|JFrog Access Java Client/4.10.0
2020-11-25T10:23:58.146+0000|127.0.0.1|jfrt@01cpj3tjark29x0mpgvnyj1bs1|POST|http://localhost:8040/access/api/v1/auth/authenticate|200|38|3|JFrog Access Java Client/4.10.0
2020-11-25T10:23:58.149+0000|127.0.0.1|jfrt@01cpj3tjark29x0mpgvnyj1bs1|GET|http://localhost:8040/access/api/v1/permissions/last_updated|200|0|1|JFrog Access Java Client/4.10.0
2020-11-25T11:03:21.363+0000|127.0.0.1|jfrt@01cpj3tjark29x0mpgvnyj1bs1|POST|http://localhost:8040/access/api/v1/auth/authenticate|200|38|2|JFrog Access Java Client/4.10.0
2020-11-25T11:03:21.364+0000|127.0.0.1|jfrt@01cpj3tjark29x0mpgvnyj1bs1|GET|http://localhost:8040/access/api/v1/permissions/last_updated|

(But as you can see the last update was done before the disk went out of space and since then no new entries)

I already removed /var/opt/jfrog/artifactory/data/derby/db.lck ,/var/opt/jfrog/artifactory/data/derby/dbex.lck and /var/opt/jfrog/artifactory/data/derby/log/* as well as the /opt/jfrog/artifactory/tomcat/webapps/access, but the result was always the same.

2 Answers

From the logs shared, it clearly tells that the Artifactory could not ping the Access server "Could not ping access server" and resulting in 404 when Artifactory is pinging the Access server. Navigate to the /var/opt/jfrog/artifactory/access/logs and check the access.log, if possible share the access.log from the access location.

Navigate to the /var/opt/jfrog/artifactory/access/data and remove all file in this directory, then restart jfrog service.

Related