Timeout for artifacts and ui self-hosted Artifactory

Enrico B. 0
Viewed 51

We are experiencing intermittent crashes of our Artifactory instance. There is no visibility in the logs.

UI is timing out as well as managing artifacts but health endpoints work

leviness, readiness, ping API calls all return OK status check shows healthy for each service

I am considering adding PUT of a dummy artifact as healthcheck to keep that instance operational (let ecs do the keepalive while restarting tasks)

some background:

  • artifactory as ecs task
  • efs with 60mb/s provisioned and ia-tiering (infrequent access)
  • filesystem derby db
  • used for some local repos and docker cache
  • artifact data storage currently at 2tb
  • latest version
  • maybe 10-20 clients (ci, developers)

I will start monitoring jvm, tomcat as next step to get more insights

update 220922:

  • added ecs container insights - have to wait on this to get some useful data
  • added thread dumping as suggested
  • i have monitoring data (from some time before and throughout and during crash) from jstat -gc/-gcutil and ps ax -T|grep java - seeing no out of normal values - gc working as it should be
  • for 4cpu/16gb task i have added JAVA_OPTS - -server -Xms7g -Xmx7g -Xss500m -XX:PermSize=500m -XX:+UseG1GC -XX:NewSize=1g -XX:MaxNewSize=1g -XX:OnOutOfMemoryError='kill -9 %p' -Djruby.compile.invokedynamic=false -Dfile.encoding=UTF8 -Dartdist=zip -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true -Djava.security.egd=file:/dev/./urandom

where i am struggling - i see no issue inside the logs ? which log's severity should i increase (from info to debug) to maybe find an issue ?

during crash (artifactory is timing out) i can access the ui (/ui/login) and some api (i.e. checking ping,liveness,readiness,route health,service health) but it times out on deployment of artefacts and going further during login (seeing rest of the ui)

how can i debug the derby db - maybe its time to move to a separate db ?

tomcat artifactory continuous-integration
1 Answers

Considering the microservices status are healthy, I would like to suggest reviewing the resource usage [CPU, Memory] as you have already planned and the communication to Artifactory from the proxy layers (nginx/apache) if available in this architecture. Just to rule out that the reverse-proxy is not timing out the connections, can you try accessing the Artifactory just with the IP:PORT combination from the browser [if the direct access is not restricted]?

Consider collecting thread dumps as well to review if the system is clogged due to thread exhaustion by any chance.

Yuvarajan0
Related