Hybris: What causes several Backoffice Long Operation items, which seems to be causing performance issues?

user_90513 0
Viewed 1383

Hybris: 1905.14

I'm having performance issues on a Hybris instance hosted in CCV2. It is slowing down the storefront and the backoffice. If I go to HAC > Monitoring > Suspend, I see several Backoffice Long Operation items. The thread dump also show several threads related to backoffice.

There are no cronjobs running and the Triggers have been set to active=false. After some time, the server needs to be restarted, since the backoffice no longer loads. Lastly, the server cannot be initialized, since it contains data.

There is minimal configuration in backoffice, just some XML configuration to customize the treeview of different usergroups.

I am not able to replicate the performance issue on my local. Any ideas what may be causing these Backoffice Long Operation items?

enter image description here

Blocked threads look like this:

priority:5 - threadId:0x2095 - nativeId:0x82f - nativeId (decimal):2095 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED
at java.base@11.0.6/java.util.Collections$SynchronizedMap.put(Collections.java:2598)
- waiting to lock java.util.Collections$SynchronizedMap@1e60f80f
at com.hybris.cockpitng.util.cache.WidgetAsyncWarmUpCache$WarmUpOperation.lambda$execute$0(WidgetAsyncWarmUpCache.java:122)
at com.hybris.cockpitng.util.cache.WidgetAsyncWarmUpCache$WarmUpOperation$$Lambda$1481/0x00000008020d4c40.accept(Unknown Source)
at java.base@11.0.6/java.util.ArrayList.forEach(ArrayList.java:1540)
at com.hybris.cockpitng.util.cache.WidgetAsyncWarmUpCache$WarmUpOperation.execute(WidgetAsyncWarmUpCache.java:122)
at com.hybris.cockpitng.engine.impl.DefaultWidgetInstanceManager.lambda$prepareLongOperation$2(DefaultWidgetInstanceManager.java:223)
at com.hybris.cockpitng.engine.impl.DefaultWidgetInstanceManager$$Lambda$1466/0x00000008020d0040.get(Unknown Source)
at com.hybris.cockpitng.engine.operations.CockpitNGBackgroundOperation.runInternal(CockpitNGBackgroundOperation.java:125)
at com.hybris.cockpitng.engine.operations.CockpitNGBackgroundOperation.run(CockpitNGBackgroundOperation.java:93)
at com.hybris.backoffice.cockpitng.util.BackofficeThreadContextCreator$RunnableWithParentThreadContext.run(BackofficeThreadContextCreator.java:100)
at java.base@11.0.6/java.lang.Thread.run(Thread.java:834)
at de.hybris.platform.core.threadregistry.RegistrableThread.internalRun(RegistrableThread.java:141)
at de.hybris.platform.core.threadregistry.RegistrableThread.run(RegistrableThread.java:131)
Locked synchronizers: count = 0

Thread count from fastthread.io:

enter image description here

enter image description here

sap-commerce-cloud
2 Answers

UPDATE 5/27 15:29: Hybris confirmed it to be a bug in 1905.14: ECP-5030 WidgetAsyncWarmUpCache causing CPU saturation with certain category structure. A workaround for CCV2 is in the JIRA tiket:

  1. Upload attached cockpitframework-19.05.14-RC4.jar to your repository under root/CUSTOMIZE/modules/backoffice-framework/backoffice/web/webroot/WEB-INF/lib
  2. Build
  3. Deploy

However, we are choosing to use 1905.13 for now.

xxx

As a temporary resolution (which removed the performance issue), we have:

  • Downgraded Hybris 1905.14 to 1905.13
  • Removed/Disabled the hotfolder extensions

At this time, we can't say if the issue is due to Hybris version or due to hotfolder extension.

We have another server that is running on Hybris 1905.14 with hotfolders enabled, and it doesn't have the "Backoffice Long Operation" issue. So, at this time, we're just waiting for SAP to provide some response (or investigation of the issue).

user_905130

I had a similar situation(from what I remember my thread dump was identical to yours) in the past and the problem was caused because of a Variant who had itself as a baseProduct. Because of this when sync or indexing was triggered, a lot of stackoverflow errors appeared since the code was trying to also synch/index the base product who had itself as baseProduct to infinity and beyond.

In order to check whether you have a similar scenario you can run the following flexible search:

select {VP:code} from {VariantProduct as VP} where {VP:baseProduct}={VP:pk}

PS: I checked and the issue still reproduces(Hybris still lets you create this cyclic dependency) on Hybris 1905.11

user_54689900
Related