Toro kernel

A dedicated kernel for multi-threading applications.

Sunday, August 16, 2020

Debugging by using QEMU trace-events

Hello folks! In this post I am going to talk a bit about QEMU trace-events. I found this mechanism during the development of a virtio driver. Roughly speaking, trace-events are generated by different components of QEMU like the virtio queues, the ioapic, lapics, etc. To enable it, you have to compile QEMU with the following option:


Then, you have to add the following line to command-line:

-trace events=./events

The file named events contains a list of events that we are interesting to observe. For example, this is mine:




In my case, I am interesting on checking if irqs are correcly acknowledged. To see this, I get all the events related with apic, ioapic and virtio. To output the logs in a file, I have to get QEMU monitor and first do 'trace-events on' and second 'trace-events flush'. I am not sure why this is not automatically done. You end up getting a file named 'trace-PID' in which PID is the corresponding PID of the QEMU process. To read this file, you have just to run the following python script:

python3 ~/qemulast/scripts/ ~/qemulast/build/trace-events-all trace-30572   

You will get something like:

virtio_mmio_read 131.447 pid=2451 offset=0x60

virtio_mmio_write_offset 141.046 pid=2451 offset=0x64 value=0x1

virtio_mmio_setting_irq 8.345 pid=2451 level=0x0

ioapic_set_irq 4.359 pid=2451 vector=0xc level=0x0

ioapic_eoi_broadcast 29.005 pid=2451 vector=0x2c

ioapic_clear_remote_irr 1.683 pid=2451 n=0xc vector=0x2c

In this example, we can see that when an IRQ is captured, the handler reads the status register and writes it to ack the irq. Then, the virtio sets the irq level at 0x0. The handler ends up by sending the EOI to the LAPIC. You can find more information about trace-events at:;a=blob_plain;f=docs/devel/tracing.txt;hb=HEAD.

Saturday, June 27, 2020

Status of the port of Toro to microvm

Since May I am working on porting TORO to the new microvm machine, which is a simplified QEMU machine with a reduced device model and an improved booting time, among others very interesting features (see For Toro, I am interested in removing all the support for legacy hardware and to have virtio-vsocket and virtio-fs working on this kind of machine. I splitted the work into the following items:
1. Compile Toro as a PVH kernel and support PVH configuration during booting
  - Issue #390
  - Issue #391
2. Add support for multicore by identifying cores on the MP table.
  - Issue #392
3. Add support for LAPIC and IOAPIC
 - Issue #395
4. Use KVM clock to get current time
- Issue #366
5. Add mmio transport layer for virtio-vsocket
- Issue #403
6. Add mmio transport layer for virtio-fs
- Issue #404
Work items from 1 to 4 are already implemented. These were tasks that removed support for legacy hardware like 8259 and the CMOS. IRQs are now handled by the LAPIC and the IOAPIC. The issues 5 and 6 mainly add support for the virtio-mmio transport layer for these drivers. The detection of mmio devices is simpler than by using PCI. The information about virtio-devices is passed in the kernel command line. The driver has to parse the kernel command line and gets the base address and the irq base. The driver for virtio-vsocket has been already ported. I am currently working on porting the driver for virtio-fs. I hope this work is finished in about a month. Stay tuned! 

Matias E. Vara Larsen

Sunday, November 10, 2019

My first patch to Linux kernel

Hello everyone! during September I submitted my first patch to the Linux kernel. This was an amazing experience in which I learnt a lot! the main problem was that information is spread in many documents so there is no a single place where are the steps are covered. I wrote down some of these steps here. Bear in mind that the patch was on a kernel module.

Where to code the patch?
My patch was on "vhost/virtio" subsystem so I cloned net-next and I applied the changes there (see

How to try the patch?
To try it, I backported the changes to my current Ubuntu installation. First, I get the headers of the current Ubuntu by doing:

sudo apt-get install linux-headers-`uname -r`

Then, I got the source code that corresponds with the headers. To know which repository to clone I checked from here

git checkout -b vsockttest Ubuntu-2.6.27-7.13

I applied the changes and then I compiled only the modules by doing:

make -C /lib/modules/4.15.0-45-generic/build M=$(pwd) modules

Finally, you have to remove old modules and install new ones.

How to write the commit message and correct patch code style automatically?

I titled the commit as “vhost/virtio:”. At the end, I added "Signed-off-by: Matias Ezequiel Vara Larsen ". I added a hook to check code style during commit (see I had to configure vim to use the right identation and to limit the number of characters of a line. 

How to generate the patch?
To generate a patch from last commit, do:

git format-patch -v2 --subject-prefix='PATCH net-next' -o ../ HEAD^

The patch has the tag "net-next" that indicates that the patch is ready for "net-next". Net-next gets patches in a 2 week windows which go to the next release. Do not send net-next packets if window is not open! (see
The “-v2” indicates that it is the second version of the patch. If you patch is a POC you can tagged with "RFC PATCH".

How to send it?

You can get a list of maintainers by doing:

./scripts/ 0001-x86-build-don-t-add-maccumulate-outgoing-args-w-o-co.patc

Use git-send-email to send the patch:

git send-email --to -cc -cc -cc -cc -cc -cc -cc -cc ../v2-0001-vsock-virtio-add-support-for-MSG_PEEK.patch

How to answer feedback?

To answer feedback configure mutt and answer from there. Don’t use gmail!
It is possible that gmail wont work with mutt. You have to configure your gmail account to allow you to use an unknown device.


Friday, July 05, 2019

QProfiler: A profiler for guests in QEMU/KVM

In this article, I am going to talk about QProfiler which is a tool to profile a guest running on top of QEMU/KVM. The source code is hosted at I started this project because I was interested in profiling Toro running as a guest on QEMU/KVM. Roughly speaking, Profiling is to count how often each function is executed. This gives an idea about where the execution time is spent. I am not an expert on this area but I will sum up my research. There are two mechanisms to profile:
   1) by counting how often each function is invoked.
   2) by sampling a process and counting which function is executed in that time.
The mechanism number 1) is intrusive since the code must be modified. The executable must be compiled with the "-pg" option that makes each function to invoke mcount() thus counting the number of times a function is executed. The main benefit of mechanism number 2) is it can profile a process without any modification. However, the result may be not accurate and limited by the maximum sample frequency. In my case, I decided to use the mechanism number 2) by implementing an script that samples a VM by using the Qemu Monitor Protocol. The script gets the %rip register and the %rbp register thus enabling to get current function and the invoked function. It is also possible to get a full backtrace but it remains a TODO work. The only change in the code is to compile by using the “-g” option to add debugging information to the binary. Then, by using addr2line is possible to get the name of the function from an address. The scripts accepts as parameter the duration of the sampling and the sampling frequency. For example, if the script samples during 10 seconds and the sampling frequency is 1s, we end up with 10 samples.
Using QProfile on StaticWebServer shows that 96% of the time the guest is executing Move(). This means that most the time the application is copying data from one block to other. For example, this happens when a new packet arrives and the content is moved to the user’s buffer. This means the networking is not very well optimized and there are too many copies between the kernel’s buffers and the user’s buffers.
There are still open questions regarding with the use of this mechanism: 
  - How fast the script can sample?
  - How does QMP actually work? And does it affect the guest execution?
  - May be more accurate to count the number of times a function is invoked?