How to stop a process running in tmux printing thread dumps periodically?

user_1357569 0
Viewed 84

I am running a pyspark shell from a bash session running in tmux, previously we just ran it from bash without a terminal multiplexer but since this runs in a (Jupyter) container, people would lose work (when their login session expired and they logged back in what was on screen would be gone) and I want to use tmux as it's a really powerful tool.

However I have noticed an odd behaviour that I don't understand, but I can reproduce. Every now and then, when the pyspark shell is running inside tmux a thread dump is generated, which fills the screen with hundreds of lines of text, meaning, what was on screen would normally be a long scroll up for the user.

e.g.

>>> 2022-05-10 18:06:43                                                                                              
Full thread dump OpenJDK 64-Bit Server VM (25.332-b08 mixed mode):                                                                   
                                                                                                                                 
"IPC Client (394840392) connection to ip-10-51-112-44/10.51.112.44:8020 from root" #157 daemon prio=5 os_prio=0 tid=0x00007f378802e000 nid=0x693 in Object.wait() [0x00007f38d86f6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)                                                                 
        at java.lang.Object.wait(Native Method)                                                                                                 
        at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1034)                                                               
        - locked <0x00007f418c689e48> (a org.apache.hadoop.ipc.Client$Connection)                                                                                               
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1078)                                                                                                               
                                                                                                                                                                       
"SparkUI-156" #156 daemon prio=5 os_prio=0 tid=0x00007f37e0009800 nid=0x676 waiting on condition [0x00007f38ea8bc000]                                                     
   java.lang.Thread.State: TIMED_WAITING (parking)                                                                                                           
        at sun.misc.Unsafe.park(Native Method)                                                                                                                                             
        - parking to wait for  <0x00007f390bb01968> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)                                
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)                                                                               
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)                                                      
        at org.sparkproject.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)                                                       
        at org.sparkproject.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974)

This continues for around 500 lines or more.

We never had this issue without tmux, and I tested in screen and got a similar behaviour, so I think it's related to the use of a multiplexer.

I am able to generate a similar thread dump if I find the PID of the java process running the spark shell, I can send a kill -3 signal to that PID and I get the same kind of thread dump generated.

I suspect this is an internal mechanism inside the multiplexer to make sure a process is still running, but I don't know for sure.

My question is, is there any way to stop this output from appearing in the terminal window? Can I re-route just this output to /dev/null while keeping the normal interactive session? Can I stop tmux from polling the running application in this way?

pyspark tmux java
1 Answers

If you have access to the runtime command, try adding something like this to the command. It would leave other console output alone (that is, still viewable in the console):

-XX:+LogVMOutput -XX:LogFile=/tmp/jvm-output.log

Another option you asked about – redirecting to /dev/null – is possible, but it would apply to all stdout, including anything you might actually want to see. I would encourage you to find alternatives to this, but it's doable by redirecting stdout ("1") like this (assuming, again, that you have access to the runtime command):

java xyz 1>/dev/null

One final option: redirect stdout to a file (vs. /dev/null) so that you have a chance of seeing other console output if/when the need arises.

java xyz 1>/path/to/file.txt
user_113749570
Related