My company is using Coldfusion 2018 (updated with latest patch Update 14) running on windows server 2019.
Lately it kept getting into the issue of intermittent down time which the client will see Error 503 service unavailable, but the service is still running.
Checking in FusionReactor I notice the active request is peaked at 256 and seems like any new request that came in after that will be responded with error 503 until all the existing request is resolved.
From the thread dump, I also notice that most of the thread stuck on reading files, and on this particular FileInputStream.readBytes method or SocketInputStream.socketRead0 method.
"ajp-nio-127.0.0.1-8018-exec-672" Id=6054 in RUNNABLE (running in native)
java.lang.Thread.State: RUNNABLE
prio=5 blockedtime=3771 blockedcount=3780 waitedtime=4599268 waitedcount=6229
at java.base@11.0.16/java.io.FileInputStream.readBytes(Native Method)
at java.base@11.0.16/java.io.FileInputStream.read(FileInputStream.java:279)
at java.base@11.0.16/java.io.BufferedInputStream.read1(BufferedInputStream.java:290)
at java.base@11.0.16/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at coldfusion.tagext.net.ContentTag.doStartTag(ContentTag.java:264)
at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:4329)
"ajp-nio-127.0.0.1-8018-exec-556" Id=2815 in RUNNABLE (running in native)
java.lang.Thread.State: RUNNABLE
prio=5 blockedtime=7179 blockedcount=6734 waitedtime=11790236 waitedcount=10845
at java.base@11.0.16/java.net.SocketInputStream.socketRead0(Native Method)
at java.base@11.0.16/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base@11.0.16/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base@11.0.16/java.net.SocketInputStream.read(SocketInputStream.java:140)
Also by tracing the thread dump, we located the code that it is stuck at:
<cfif type IS "JPG" OR type IS "JPE" OR type IS "JPEG">
<cfcontent reset="yes" type="image/jpeg" file="#FNAME#" deletefile="no">
<cfelseif type IS "GIF">
<cfcontent reset="yes" type="image/gif" file="#FNAME#" deletefile="no">
<cfelseif type IS "PNG">
<cfcontent reset="yes" type="image/x-png" file="#FNAME#" deletefile="no">
<cfelseif type IS "TIF" or type IS "TIFF">
<cfcontent reset="yes" type="image/tiff" file="#FNAME#" deletefile="no">
<cfelseif type IS "PDF">
<cfcontent reset="yes" type="application/pdf" file="#FNAME#" deletefile="no">
<cfelse>
<cfcontent reset="yes" type="text/plain" file="#FNAME#" deletefile="no">
</CFIF>
This chunk of code we have not done anything new to it for quite some time and it has been working well for a long time.
We tried contacting Adobe ColdFusion support, which took them a while and they provide us with a hotfix file named "hf201800-4212487". After applying the hotfix, the issue did went away for a while until recently the issue happens again.
Other method that I've tried including changing the number of concurrent thread, increase the number of cached template, upgrading java from java version 11.0.8 to 11.0.16, but nothing really helps.
Just curious if anyone happens to faced similar issue or have any solution for it.
Edit :
Here is some screen shot of latest thread dump uploaded to fastthread.io for visualization
