Recently I had to adapt one open source project to the needs of the client I am currently working to. The way the code was written has proven a nightmare to track the function calls, value and type of the variables (since the program was written using a LOT of void pointers and function pointers).
To help me understanding the program's flow I used GDB (Gnu Debugger). It wasn't my first time using it, but in the past I mostly used it with core files. This time I learned some new tricks that I want to share.
In this post I will talk about the preparation and the process of using GDB. It will be divided into a few sections: introduction to GDB, the debug flag, the core files and useful commands.
1) Introduction to GDB:
GBD is a debugging program that allows you to retrieve several information about your application. You can check your stack trace, variable values, add breakpoints, run a program step by step and so on.
There are three ways of using GDB that I will explain in this post. The first one is straight running it, which is useful when your program is not behaving as expected (so you can add breakpoints or run step by step) or you can reproduce the case where it crashes. The second is when your program breaks sometimes, but you don't know how it broke (very common if you have a client-server application) - in this case you will need to allow your system to create a core file. The third method is by informing the program pid, this is very useful to debug client-server applications that spawn children process to run a specific task.
2) The Debug Flag:
In order to use gdb you must have a few things set in your code. The most important thing that you need to do is turn the debug flag of your compiler on.
If you are compiling your program in the command line all you have to is add one -g in the command line. For instance, if your line is:
gcc -O2 example.c -o example
It should become:
gcc -g -O2 example.c -o example
If you are using a Makefile, you have to do pretty much the same thing. Find the line with the flags and add the "-g" option to it. Don't forget to run the make clean before running the make, since it won't recompile the project by only changing the make file.
If you are using an interface, you will have to find where you turn the debug flag on. When I use code::blocks the flag is on Settings > Compiler and debugger... > Compiler settings > Compiler Flags; the option is "Produce debugging symbols [-g]".
Finally, you should know that there are several debug flags. -g is the most general one and has always worked fine for me. If you want to learn more about the other debug flags, you may find a full explanation here.
3) Core Files
If your program is crashing under an unknown situation, you can configure your linux to create a core file. This file has all the information of registers, variables and stack trace of the program when it crashed and can be used with GDB to find out why your program crashed.
In order to create a core file you must first enable it. To do so you have to run the following command:
ulimit -c unlimited
To turn it off:
ulimit -c 0
This will allow your system to create core files of any size. It is important to notice that this command limit only lasts for your current section (in other words, when you logout/turn off you PC, it will reset to the default value) and is only valid to your user. If the program is run by other user, you must change this value under its section.
By default the core will be written at the running binary directory. So the user running the program must have write permission there. If you don't want to change the permissions on the binary directory (to avoid security issues), you can change the path by editing the configuration file at /proc/sys/kernel/core_pattern. You can also change the format (for instance, ubuntu's default format is "core", that doesn't say much huh?), I like to use core.<binary>.<pid> and to save then all at the tmp directory (so I don't have to manually remove them). To do so you must run:
echo "/tmp/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern
For more information on cores, you can check this link or simply type on your terminal:
Finally, if your system is still not creating cores, it may be because your program is changing its user during its execution (again, to avoid security issues). In this case you must change your code to call the function prctl, with PR_SET_DUMPABLE as option and the first argument as 1. Other arguments are not used. So the line added should be (this function returns an integer, so you may want to check it):
prctl(PR_SET_DUMPABLE, 1, 0, 0, 0);
For more information on prtcl check this link or type man prtcl on your terminal.
As I said in the introduction, I will describe how to use GDB in three ways. If you just want to run your program simply use:
gdb ./<your binary>
If your application requires arguments to be passed, you must use:
gdb --args ./<your binary> <arg 1> <arg 2> ... <arg n>
If you are going to use a core file you need to run:
gdb -c <core file> ./<your binary>
Finally, if you want to detach a running process, you should use:
gdb -p <process pid>
Following I have compiled a list with the commands that I find most useful (in no particular order). For convenience I will add a number after each command that represents which kind of use they are valid. (1) is for running your binary on gdb, (2) is for running from a core file and (3) for detaching a process.
- break <file:line> - Add a breakpoint, the program will stop if that line is reached. (1)(3)
- run - Start running the program. (1)
- step - run one operation (mostly a code sequence with no function calls, such as an if with many statements or a mathematical operation; if there is a function call, it will step to the first operation of the given function). (1)(3)
- stepi - run one assembly operation (for instance, an if with only one simple comparison - in other words, comparing two variables, two constants or a variable and a constant - will take two stepi, one for the compare and other for the branch). (1)(3)
- until - run the operation in a line completely, for instance if I have x = <some function>, it will run until x receives a value, no matter how many operations <some function> runs.(1)(3)
- finish - run the current function until it returns (and prints the value returned). (1)(3)
- continue - run the program until a breakpoint is reached or the program is finished.(1)(3)
- bt - prints the stack trace. (1)(2)(3)
- bt full - prints the stack trace and the values of the arguments of each function. (1)(2)(3)
- print <variable name> - print the value of the variable. You may use pointer arithmetic (such as *<variable name>) and struct accessing (<variable name>.<element> or <variable name>-><element>). (1)(2)(3)
- up - goes up one level of the stack. (2)
- down - goes down one level of the stack (2).
- help -print a help message, you can also use help <command> to read a help message about that specific command, including arguments and syntax. (1)(2)(3).
Another useful tip is that simply pressing enter will repeat the last command, also there are shorter versions for some commands (such as "s" for step).
Well, that is it for today's post, hope it helps.