Debugging TORO with BOCHS and GDB

In the following lines I'll tray to explain how debug Toro using bochs+gdb. I often used qemu instance of bochs but the latest days I figured out some rare behaviors so I replaced qemu for bochs with good results. 

Before to start we have to compile bochs for such purpose, after download the source run the next command in order to compile it:

> ./configure --enable-cpu-level=6 --enable-all-optimizations --enable-x86-64 --enable-pci --enable-vmx --enable-disasm --enable-debugger-gui --enable-logging --enable-fpu --enable-3dnow --enable-sb16=dummy --enable-cdrom --enable-x86-debugger --enable-iodebug --disable-plugins --disable-docbook --enable-gdb-stub     

> make install

It is not possible to enable the smp support and gdb stub at the same time. If the compilation was right we'll be able to run bochs with gdb support.

Now It is needed to compile the toro application within the debug symbols, if there're any old .o or .ppu file they must be deleted because they don't have symbols information. We should execute in the toro source directory the next commands:

> fpc ToroHello.pas -g -oToroHello -Fu../rtl/ -Fu../rtl/drivers

> ./build 4 ToroHello boot.o ToroHello.img

This procedure is for ToroHello.pas but it's the same for other toro's app.

So far, we've the bochs and the Toro's image, now we have to build the .bochsrc file and launch bochs.  The following lines may be useful:

megs: 32

ata0-master: type=disk, path="ToroHello.img"

boot: c

log: bochsout.txt

gdbstub: enabled=1

Check that we've indeed to enable the gdbstub in the bochs' source file.

The next step is to run bochs and then GDB:

> bochs

> gdb ToroHello

If we run gdb in the toro/test directory .gdbinit will be used, otherwise we have to connect to bochs manually as follow:

> (gdb) target remote localhost:1234 

If everything goes well we are able to set breakpoints and uses all the tools of gdb. For instance we could do:

> (gdb) b KERNELSTART

> (gdb) c

In the first line a breakpoint is set at KERNELSTART and then the virtual machine continues until comes back when the breakpoint is reached.

Many commands could be usefull in this point like n, for running line by line, step for stepping into, info registers and soon on.

There're a lot of information that we can get at this point but that's for another tutorial ;)

Matias E. Vara

www.torokernel.org

Toro debug with ECLIPSE

Hi everyone! I figured out that the server where I had the video about "TORO debug with ECLIPSE" is down, so I've uploaded it again. It shows the TORO Builder running on Windows 2003 x64. It is interesting how we can do "step by step" debugging and "set a breakpoint". Qemu is emulating a x86_64 arch.

Enjoy!   

Matias E. Vara
www.torokernel.org

Toro's article in Microelectronic Congress

Here it is the link to the article that I've shown at Microelectronic Congress 2011, Faculty of Engineering, University of La Plata, Argentina. The only thing is it is in spanish. It talks about TORO project and it shows a benchmarks that I made for my graduated project. Enjoy!

Matias E. Vara
www.torokernel.org

Patching GDB 7.3 for QEMU remote kernel debug

This time I will try to explain how patch GDB 7.3 in order to debug a kernel using QEMU through remote debuging. If we try to debug remotely, we'll find a error message like:

Remote packet too long: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 ...

I am not sure about problem but I suppose it's about register size. When the virtual machine jumps from real mode to long/protect mode, the register size changes but GDB doesn't know that. Thus, when GDB receives a bigger packet than it expects, it fails. Therefore, The patch just increments the buffer in those cases.

The first step is to download GDB 7.3 from http://www.gnu.org/s/gdb/download/, I've implemented the patch on 7.3 version but I think it works in oldest too.

Once downloaded and uncompressed, edit the file gdb-7.3/gdb/remote.c and go to 5693 line. That's the process_g_packet procedure. Now, look for and replace the original source with the following lines:

/* Further sanity checks, with knowledge of the architecture. */

//if (buf_len > 2 * rsa->sizeof_g_packet)

// error (_("Remote 'g' packet reply is too long: %s"), rs->buf);

if (buf_len > 2 * rsa->sizeof_g_packet)

{

rsa->sizeof_g_packet = buf_len;

for (i = 0; i < gdbarch_num_regs (gdbarch); i++)

{

if (rsa->regs[i].pnum == -1)

continue;

if (rsa->regs[i].offset >= rsa->sizeof_g_packet)

rsa->regs[i].in_g_packet = 0;

else

rsa->regs[i].in_g_packet = 1;

}

}

Finally, it just remains to execute:

$ ./configure

$ make


In some systems may be necessary to install termcap library, simply execute:

$ sudo apt-get install libncurses5-dev

After compilation, the binary could be found in gdb-7.3/gdb/gdb, It must be enough to run GDB correctly.

Matias E. Vara

www.torokernel.org

Threads migration without Lock in Toro

In a Multicore environment, the programmer needs to create local and remote threads. In TORO create a remote threads is easy, you just have to use BeginThread() with the appropriate CPU identification. On that basis, there are two important procedures in TORO:

- Thread Emigrating: is when the threads are created in a remote processor.
- Thread Inmigrating: is when the guest processor enque in its scheduler the threads that they are comming from others processors.

This is the unique kernel point which needs syncronization between the cores. The mechanism is called "Exchange Slot" and it works without any atomic operation. In this case it used for send and receiv threads but it works with any kind of data.

For every processor in TORO there is an structure called TSchedulerExchangeSlot:

TSchedulerExchangeSlot = record
DispatcherArray: array[0..MAX_CPU-1] of PThread;
EmigrateArray: array[0..MAX_CPU-1] of PThread;
end;

Where MAX_CPU is the number of processors and PThread is a pointer to TThread structure. From the structure declaration we can see that every processor has two arrays

(DispatcherArray y EmigrateArray), and every entry in the array is a pointer to a thread´s queu.

The procedure to send threads to remote processor has three stages:
1-The user calls to BeginThread()for create a new one, if the parameter CPUID is different to local CPU then the kernel enque it to DispatcherArray[CPUID].
2-During Scheduling (cause SysThreadSwitch syscall). The procedure Emigrating()moves all threads from DispatcherArray[] to EmigrateArray[] (only if EmigrateArray[] is nil)
3-During Scheduling of the Remote CPU, the procedure Inmigrating() look for a not nil entry in EmigrateArray[LocalCPUID] in every TSchedulerExchangeSlot processor structure. If it is not nil Then import all the threads to local scheduler and become EmigrateArray[LocalCpuid] to nil.
Local processor just writes and read to DispatcherArray[]. While the local and remote processor write and read to EmigrateArray[], but the access is synchronized using nil pointer.
The “Exchange Slot” doesn´t need "LOCK" instruction.


The Inmigrating and Emigrating procedures are called from the Scheduler. The scheduler makes a few system task, for example in the picture, we can see the scheduler´s flow diagram. There, first it calls Inmigrating(), after that it calls Emigrating() and At the end a new thread is scheduling.


Matias E. Vara
www.torokernel.org