Toro compiled with FPC 3.0.4 and Lazarus 1.8.0

I am glad to announce that Toro compiles smoothly by using FPC 3.0.4 from Lazarus 1.8.0. There is only one issue when I want to run qemu from the IDE which I am going to report. Apart of that, no other issue has been observed.

Matias 

Toro in FOSDEM'18!

I am glad to announce that Toro will be in FOSDEM'18. This time I will talk about the improvement in the scheduler to reduce the impact of idle loops thus reducing the cpu usage of VMs. For more information, you can find the abstract here.

Reducing CPU usage on VMs that run Toro

Last days I worked on reducing the CPU usage of Toro. I observed that VMs that run Toro consume 100% of CPU which makes any solution based on Toro impossible for production. I identified four situations in which an idle loop is the issue:

1. During a spin locks
2. When there is no a thread in ready state 
3. When there is no thread 
4. When threads only poll a variable

Cases 1, 2 and 3 are in the kernel code. However, case 4 is when a user thread does idle work by polling a variable. So in this case, the solution would be harder since the scheduler has to figure out that the thread is only polling. Intel proposes different mechanisms to reduce the impact of idle loops [1, 2]. In particular, I was interested on the use of mwait/monitor instructions. However, I found this is not very well supported on all Hypervisors. So I have to base on the instructions hlt (halt) and pause. I want to highlight that hlt is a privilege instruction so only ring0 can use it. However, since in Toro both the kernel and the application run in ring0, hlt can be used either by the kernel or the user. Following cases correspond with the use by the kernel.

First, I tackled case 1 by introducing the pause instruction inside the loop. This relaxes the CPU when a thread is getting exclusive access to a resource. Cases 2 and 3 were improved by using hlt which just halts the CPU until next interruption. To tackle case 4, I proposed two APIs to tell the scheduler when a thread is polling a variable. When scheduler figures out that all threads in a core are polling a variable, it just turns the core off.

I tested this in my baremetal host (4 cores, 8 GB, 2GHz) in Scaleway with KVM and a VM running Toro. I also installed Monitorix to monitor the state of the host. To stress the system, I generate http traffic and monitor the CPU usage. A few seconds after the stress stopped, the CPU usage of the qemu process is only about 1%. This usage goes up and down depending on the stress. By topping, I get a patter like this:   

CPU%
6.6    0.6     0:46.92 qemu-system-x86
63.8  0.6     0:48.84 qemu-system-x86 (stress)
99.7  0.6     0:51.84 qemu-system-x86 (stress)
45.5  0.6     0:53.21 qemu-system-x86 (stress)
2.3    0.6     0:53.28 qemu-system-x86
4.0    0.6     0:53.40 qemu-system-x86
6.6    0.6     0:53.60 qemu-system-x86

This is not always the case and sometimes Toro takes longer to turn off. This may happen when a socket is not correctly closed and it ends up by timeout. I need to experiment more to measure how much reaction the system losses when the core is halted. This recent work, however, seems very promising!

[1] https://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/IdleTalk.pdf
[2] Intel Volume 3, section 8.10.2 and 8.10.4

Running Toro on top of VirtualBox

I am glad to announce that Toro is supported by VirtualBox. After bug fixing in the bootloader, Toro perfectly boots on VirtualBox:

This way Toro can run on Hyper-V, VMWare, QEMU, KVM and now VirtualBox. 

Running Toro on top of Hyper-V

Hello folks! I am working on the bootloader of Toro to be able to boot on HyperV. First experiments seem very interesting! I could run the example ToroHello on top of HyperV.

I found that the problem was the use of mm0 register in the bootloader. While this works well in KVM or QEMU, in Hyper-V the system hangs/crashes. By using a general purpose register like EBX, the problem is fixed and the kernel is loaded. I could not figure out why this is happening. Also, it was a bit tricky to wake up the other cores but, finally, I got them up. In the the near future, Hyper-V will be installed on every Windows in this context Toro can be an option of containers in a Windows environment. 

The four steps to deploy a Toro application in the Cloud

Hi folks! Toro makes very simple the deployment of standalone-applications on Clouds. I just committed the script SetupCloudGuest.sh which is meant to install the needed tools to run a Toro application in a VM and then makes it public on internet by relying on port forwarding. These are the four steps:

1) Get a machine in Scaleway that will be used to run KVM and the Toro's guests.

2) Clone Toro repo. This contains the needed scripts to setup the host.

3) Run torokernel/tests/SetupCloudGuest.sh. This installs KVM and other needed tools.

3) Download the image that run the guest. For example, you can download TorowithFileSystem.img from here. The image must be copied to torokernel/tests. This workflow supposes that the development is done in other machine and the Toro image is just copied from your local machine to the host.

4) Finally, run Cloudit.sh TorowithFileSystem onlykvm

 

That's all folks! You can get the guest's console by using a VNC Viewer to port 5900, or connect to port 80 of guest by connecting to port 80 of host.

Deploying a TORO guest in the Cloud

Hi folks! this weekend I deployed a TORO guest in the Scaleway's Cloud. This can be checked by connecting a VNC client to  51.15.142.20:5900

Figure1. VNC Viewer on Windows connected to the Toro guest.

What you see is a TORO guest running on top of KVM. This example runs the ToroHello.pas which just prints a message to the screen. I am working to add more examples that use the filesystem and the network stack. Here I am writing down all the procedure to compile Toro in Ubuntu and run the example on KVM. This example, yet very simple, shows how easy is to deploy an application which is compiled within Toro kernel and run it in a VM without interference of an OS.

Testing Toro on KVM and Xen

Hi folks! These days I have been experimenting with Toro on KVM and Xen. This is still experimental work but it demonstrates how flexible Toro is. In KVM, I had very good results and ,with minor changes, Toro was running without any problem.

In Xen, I tried with an HVM guest. However, I can't go further than booting the kernel. It seems that I am getting a lot of exceptions when Toro tries to wake up the cores in the system.

I will focus on Xen in the next days. In addition, I am very interested in test Toro in Hyper-V so stay tuned!

Matias

E1000 driver is alive!

Hi folks! I spent this weekend trying to make the driver e1000 works. I started with this piece of code in 2011. I based on the code of Minix3. However, I could not go further than getting the MAC address, and then, I just gave up. I restarted last weekend and I did a great progress. I have almost the ToroPing.pas example working (see Figure 1). The key was to enable E1000_DEBUG in QEMU and put the hands in the code to understand as much as possible what the network card was doing. I will give more details about how enable debugging in QEMU in future posts.

Figure 1. ToroPing.pas running with E1000 driver.

 I will continue progressing on E1000 in the following weeks so stay tuned!

Matias

Progress on Toro Virtual Filesystem and tooling

Hi folks! the purpose of this post is to highlight the work done on Torokernel Virtual Filesystem.

Writing Operation support in VFS:
I implemented three new VFS functions: SysWriteFile(), SysCreateFile(), SysCreateDir(). I will not detail the aim of these functions which I think is obvious. However, their semantics could vary from other kernels. For example, SysCreateFile() first tries to create  a new file, if the file already exists it fails. In the case of SysCreateDir(), if the directory exists, it also returns with error.

Writing Operation support in Ext2 driver:
I added three new functions to give support to the new vfs features. These are Ext2WriteFile(), Ext2CreateFile() and Ext2CreateDirectory().

Pci unit:
I moved all the pci device detection to a new unit named Pci.pas. This makes the code in FileSystem.pas much more clear.

Tool to create an populate ext2 images in Windows 10:
To ease the filesystem manipulation, I compiled a simple version of genext2fs for Windows 10. This is a common tool in Linux system which allows to create and populate an ext2 image. In Windows 10, I could not find such a tool so I decided to port it by myself. Again, this is a simple port which may not work in all cases. However, it is working fine in my tests. You can find an user guide here.

TorowithFileSystem.pas project:
I updated this example to show the use of new vfs features. The example still relies on ToroFiles.img which is mounted and used to get /web/index.html. I updated the application to log in the directory /web/logs each time that client requests a page.  

This is more or less all the progress done in the last three months. Future work is the implementation of Continuous Integration to compile the tests examples so stay tuned!

Matias

Continuous Integration in torokernel (travis)

Hi folks, this weekend I spent some time playing around Travis. This is a tool for continuous integration which is very well integrated into Github. Travis makes very easy to build and run tests after a push or before a pull request for a given branch. 

I started my experiments with a simple idea: make a test to check if a simple application in Toro Kernel is correctly executed. To do this, I choose a simple application named ToroHello.pas which just writes "Hello World" to the serial console. To check this application, I set up Travis to do two things:

1. To compile ToroHello example and generate the booting image ToroHello.img.

2. To run the image in Qemu and test if the example worked.  

I based on a previous work in https://github.com/nielsAD/travis-lazarus. So 1) was easy to do and I got a ToroHello.img compiled by using Travis very quickly (see Figure 1). 

The task 2) took me a bit more time. I set up to run qemu after the building and to output the serial port to a file. Then, I tested if the output contains 'Hello world". In such a case, the test passed. Following pictures show the output of the job:

Figure 1. Travis compiling ToroHello.pas

Figure 2. Travis running ToroHello.img in Qemu and outputting the serial

Figure 2 shows the output from the serial port. We can see Toro's initialization at line 3574.

Figure 3. Result of the building

To summarizes, in a very short time I was able to compile ToroHello and then produce a test to verify its correct behavior. This is a simple example but it allows to show how powerful might be CI for a kernel. So I am planing to use Travis in two different levels of testing. The first level is to verify the compilation of the examples after a push and/or pull request. The second level is to write specific tests to measure performance, leaks, wrong behaviors, exception faults, etc. This needs however a lot more experimentation.

Matias

Porting genext2fs to Windows 10

Last weekend I spent some time to find a tool to create ext2 images in Windows. I found a wonderful tool named genext2fs which allows to create an ext2 image and to populate it with a directory structure. I could not find a way to compile it in windows so I put hands on it and I got a windows version that seems working. You can find the patch at the end of this post. However, it is a horrible patch that only makes the thinks work. If anyone knows a better way to do it, please don't hesitate to tell me. I compiled without any problem in cygwin64. We have to be careful when we run it since, for a reason that I don't understand, the program tries to put all current directory into the image. To prevent this, I just run it from the directory that I want to put into the image. To summarize, I am now able to create a ext2 image and populate it with a directory structure. For example, I am experimenting by populating ToroFiles.img with the files that are used by the webserver in Toro. To do so, I am running the following command: 

../genext2fs.exe -v -d ../ToroFiles -b 8192 ../ToroFiles.img

This generates a 8MB image named ToroFiles.img that contains the files in the directory ToroFiles. I executed the command inside the directory ToroFiles. In the next days, I will commit the example TorowithFilesystem.pas that uses this tool to populate ToroFiles.img. Together with the example, I will upload the instruction to run it so stay tuned!

Matias Vara

--------------------------------------------------------------------------------------------------------
This is a patch for genext2fs-1.4.1/genext2fs.c
--------------------------------------------------------------------------------------------------------

761c761
< readsize = readlink(path, buf, bufsize); /* 1st try */
---
> // readsize = readlink(path, buf, bufsize); /* 1st try */
1458c1458
< if(st->st_mode & S_IRGRP)
---
> /* if(st->st_mode & S_IRGRP)
1475c1475
< mode |= FM_ISVTX;
---
> mode |= FM_ISVTX; */
1631c1631
< lstat(dent->d_name, &st);
---
> stat(dent->d_name, &st);
1645c1645
< case S_IFLNK:
---
> // case S_IFLNK:
1652c1652
< case S_IFSOCK:
---
> // case S_IFSOCK:
1669c1669
< if (!S_ISDIR(st.st_mode) && !S_ISLNK(st.st_mode) && st.st_nlink > 1) {
---
> if (!S_ISDIR(st.st_mode) /* && !S_ISLNK(st.st_mode) */ && st.st_nlink > 1) {
1682c1682
< nod = mknod_fs(fs, this_nod, name, mode|FM_IFCHR, uid, gid, major(st.st_rdev), minor(st.st_rdev), ctime, mtime);
---
> // nod = mknod_fs(fs, this_nod, name, mode|FM_IFCHR, uid, gid, major(st.st_rdev), minor(st.st_rdev), ctime, mtime);
1685c1685
< nod = mknod_fs(fs, this_nod, name, mode|FM_IFBLK, uid, gid, major(st.st_rdev), minor(st.st_rdev), ctime, mtime);
---
> // nod = mknod_fs(fs, this_nod, name, mode|FM_IFBLK, uid, gid, major(st.st_rdev), minor(st.st_rdev), ctime, mtime);
1691c1691
< case S_IFSOCK:
---
> /* case S_IFSOCK:
1698c1698
< break;
---
> break; */
2392,2395c2392,2395
< if((pdir = open(".", O_RDONLY)) < 0)
< perror_msg_and_die(".");
< if(chdir(dopt[i]) < 0)
< perror_msg_and_die(dopt[i]);
---
> //if((pdir = open(".", O_RDONLY)) < 0)
> // perror_msg_and_die(".");
> //if(chdir(dopt[i]) < 0)
> // perror_msg_and_die(dopt[i]);
2397,2400c2397,2400
< if(fchdir(pdir) < 0)
< perror_msg_and_die("fchdir");
< if(close(pdir) < 0)
< perror_msg_and_die("close");
---
> //if(fchdir(pdir) < 0)
> // perror_msg_and_die("fchdir");
> //if(close(pdir) < 0)
> // perror_msg_and_die("close");
2593d2592
< 
2595d2593
< 

Example of Toro Virtual Filesystem

Hi folks! I have spent some time to write an example of the use of the toro virtual filesystem. After fixing a couple of bugs, I manage to get an example named TorowithFileSystem.pas. This example together with others is hosted in torokernel/tests. The example can be tested by following the instructions here. I also committed a small ext2 image named ToroFiles.img that contains the files to make it run. For the moment, I am providing the whole image. I hope to provide a tool to create ext2 images. The example is very simple, it opens a file name index.hml located in the ext2 image and then it listens on port 80. When a connection arrives, it sends the content of the file. Briefly speaking, it is a sort of webserver. The example is a slightly modification of the ToroHttp.pas example. The only difference is that this example read the content from a file. To illustrate, I highlighted the differences in code in read and green (see Figure 1).  

Figure1. Differences between ToroHttp.pas and TorowithFilesystem.pas

          
In red, we can see the initialization part in which the disk ata0 is dedicated to the core 0 (local cpu). By doing this, only core can access to this device. Once dedicated, we are able to mount the first partition, which is identified with the minor number 6, by using the ext2 drive. After this, we are able to open the file index.html and read it until EOF. Then, the content of this file is sent when a new connection arrives. Enjoy!

Matias

Compiling and Running Tests by using Lazarus 1.6.2

Hi folks! I am spending some time working on Lazarus and making all tests examples to compile and run directly from Lazarus. For the moment, I am only working on Windows 10. So far I managed to:
1. Compile ToroHello.pas directly from the IDE thus avoiding to execute any make from command line:

2. Run on Qemu ToroHello.img: 

I illustrated by using ToroHello, however, this also applies to all the tests. The changes are not still in master, instead I am working on the branch https://github.com/MatiasVara/torokernel/tree/supportlazaruscompilation. Fell free to clone and try. This only needs to install Lazarus 1.6.2 and FreePascal 3.0.0.


Matias

Toro on Github

Hi folks! I moved toro repository to github, i.e., https://github.com/MatiasVara/torokernel. I am trying to be more organized in the TODO work so fell free to check Projects and Issue sections. I am looking forward to more collaborative work!

Matias

Toro kernel compiled with FPC 3.0.0!

Hi folks! I just fixed system.pas to compile Toro by using FPC 3.0.0. Most of the work has been done on the way that system.pas handles AnsiStrings. In addition, I moved the project to github, so fell free to participate. I tested the tests examples and they seem working. However, I only tested on Win 10. I will test in Linux in the following days. 

Matias 

How compile and run Toro in Windows 10

Hi folks! This time I want to present how compile and run Toro in Windows 10. What am I going to explain? How to compile the example named ToroHello.pas and run it on a QEMU VM. In thw following, I explain this procedure in 7 steps. My environment is in a Windows 10 in a x86-64 architecture.

1) Install Git Bash for Windows:
I am using this as the command line to get the compilation on Windows. You can download it from https://git-for-windows.github.io/

2) Install Yasm:
Yasm is the compilator for assembler code. This is needed to compile the bootloader. You can download it from http://yasm.tortall.net/Download.html

3) Install FPC 2.6.4:
FPC is used to compile the code of toro. For the moment toro is based on FPC 2.6.4. You can download it from https://sourceforge.net/projects/freepascal/files/Win32/2.6.4/fpc-2.6.4.x86_64-win64.exe/download

4) Install QEMU:
QEMU is used to run the application compiled into Toro. You can download it from https://qemu.weilnetz.de/w64/

So far we have all the tools to compile and run ToroHello.pas. We need just to clone it and compile it.

5) Clone Toro code:
Go to the Git Bash for Windows and clone the toro code by doing:

git clone git@github.com:MatiasVara/torokernel.git

6) Compile the bootloader and tools:
Before compile any application with Toro we have to compile the bootloader and the tools to generate an image for the VM. To do so, just run Make in toro-code/test directory.

7) Compile ToroHello.pas
To do this, we rely on the script named CloudIt.sh. We have to run the following command in the toro-code/test directory:

./CloudIt.sh ToroHello

That's all folks! If everything goes well you will get a QEMU VM running ToroHello.pas. Please fell free to comment if any of the steps did not work for you. Matias

HTTP server by using socket multiplexing

Hi folks!
In this post I will show you part of the last experiments on sockets multiplexing. This has been added in Toro a couple or years ago, however, only now I could make an example that works. This is a simple example, i.e., a simple http server that prints a welcome message. To make it works, I spent the last month fixing bugs in the networking. Also, I fixed an important bug in the memory manager. In the following, I will show how socket multiplexing is used in the case of a http server. The code of the example can be found in tests/ToroHttp.pas. In this post, I am not going into details about how socket multiplexing works. I will try only to give a big picture.
Let's first discuss the code of ToroHttp.pas (see Figure 1). This is the main program. It defines the network handler, i.e., HttpHandler. Then, it sets the right methods, e.g., DoInit(), DoAccept().

Figure 1.

After that, the handler is registered by using SysRegisterNetworkService(). The kernel will execute the different methods. For example, the first method invoked will be HttpInit().

Figure 2.

The method creates a new socket that listens on local port 80. Then, when a new connection arrives, the kernel will invoke HttpAccept().

This method prints something and waits for new data from the remote host. When some data is received, the kernel executes HttpReceive().

Figure 3.

We implemented a simple Receiv method. It only reads all the data from the remote host, then it sends a welcome message, and finally it closes the connection. To tray the server, we use telnet to the IP of the guest. In the telnet client, we get:

In the Qemu VM, we see that the connection is received and then closed:

As I said before, I only explained some details about socket multiplexing. I used the http server as a running example. The whole example is hosted in toro repo so please fell free to clone it.

Matias Vara