GDB not interrupting program immediately

user_2087442 0
Viewed 1334

When debugging a large C application, I see strange behavior from gdb: I can always interrupt the program pressing Ctrl+C:

^C
Program received signal SIGINT, Interrupt.
0x76f58964 in select () at ../sysdeps/unix/syscall-template.S:81
81      in ../sysdeps/unix/syscall-template.S
(gdb)

However, after a sufficient runtime of the program (like > 1 day), I cannot easily interrupt the program anymore. When trying to interrupt the program with Ctrl+C, gdb just shows

^C
Program received signal SIGINT, Interrupt.

and hangs there for several minutes up to hours. If it takes longer than a few minutes, I usually open another terminal and kill gdb manually to be able to continue.

Question: Is this expected behavior from gdb? Can I set an option to avoid this?

Further details:

  • The application is FTL (https://github.com/pi-hole/FTL)
  • It is multi-threaded using pthreads
  • During the waiting time after hitting Ctrl+C, gdb is at 100% CPU.

Edit: Further details

I ran perf record -p $(pidof gdb) for about 10 seconds while gdb was frozen. perf report returns:

90,82%  gdb      gdb                [.] find_thread_ptid                                                                                                   
 9,13%  gdb      gdb                [.] ptid_equal                                                                                                         
 0,02%  gdb      gdb                [.] iterate_over_threads                                                                                               
 0,01%  gdb      [kernel.kallsyms]  [k] run_timer_softirq                                                                                                  
 0,01%  gdb      gdb                [.] 0x0016a9a4                                                                                                         
 0,00%  gdb      gdb                [.] 0x0015a480                                                                                                         
 0,00%  gdb      gdb                [.] 0x0016a998                                                                                                         
 0,00%  gdb      gdb                [.] is_exited

After a few minutes, gdb finished and I ran info threads which still showed only three threads (as before):

(gdb) info threads
  Id   Target Id         Frame 
  3    Thread 0x764b8460 (LWP 10114) "socket listener" 0x76f60260 in accept () at ../sysdeps/unix/syscall-template.S:81
  2    Thread 0x76cb8460 (LWP 10113) "loganalyzer" 0x76f58964 in select () at ../sysdeps/unix/syscall-template.S:81
* 1    Thread 0x76e65000 (LWP 10098) "pihole-FTL" 0x76f58964 in select () at ../sysdeps/unix/syscall-template.S:81
i386 signals gdb linux c
1 Answers

gdb just shows ... and hangs there for several minutes up to hours.

A guess: your program creates, but does not properly join and terminate threads.

You can confirm or disprove this by running the program for a few hours, interrupting it with Control-C, and issuing info threads command.

On Linux, threads are just processes that happen to share virtual memory and file descriptors (and controlling terminal). When you hit Control-C, only one of your threads receives SIGINT.

In the default all-stop mode, GDB gets notified (by the kernel) that one thread has a pending SIGINT. GDB then needs to stop all the other threads of your process, and that can take non-trivial amount of time.

Not only that, but GDB may have to repeat this several times: while the threads were running, they may have created new threads, which now must be stopped as well.

Employed_Russian_506170
Related