Progress on Toro Virtual Filesystem and tooling

Hi folks! the purpose of this post is to highlight the work done on Torokernel Virtual Filesystem.

Writing Operation support in VFS:
I implemented three new VFS functions: SysWriteFile(), SysCreateFile(), SysCreateDir(). I will not detail the aim of these functions which I think is obvious. However, their semantics could vary from other kernels. For example, SysCreateFile() first tries to create  a new file, if the file already exists it fails. In the case of SysCreateDir(), if the directory exists, it also returns with error.

Writing Operation support in Ext2 driver:
I added three new functions to give support to the new vfs features. These are Ext2WriteFile(), Ext2CreateFile() and Ext2CreateDirectory().

Pci unit:
I moved all the pci device detection to a new unit named Pci.pas. This makes the code in FileSystem.pas much more clear.

Tool to create an populate ext2 images in Windows 10:
To ease the filesystem manipulation, I compiled a simple version of genext2fs for Windows 10. This is a common tool in Linux system which allows to create and populate an ext2 image. In Windows 10, I could not find such a tool so I decided to port it by myself. Again, this is a simple port which may not work in all cases. However, it is working fine in my tests. You can find an user guide here.

TorowithFileSystem.pas project:
I updated this example to show the use of new vfs features. The example still relies on ToroFiles.img which is mounted and used to get /web/index.html. I updated the application to log in the directory /web/logs each time that client requests a page.  

This is more or less all the progress done in the last three months. Future work is the implementation of Continuous Integration to compile the tests examples so stay tuned!

Matias

Continuous Integration in torokernel (travis)

Hi folks, this weekend I spent some time playing around Travis. This is a tool for continuous integration which is very well integrated into Github. Travis makes very easy to build and run tests after a push or before a pull request for a given branch. 

I started my experiments with a simple idea: make a test to check if a simple application in Toro Kernel is correctly executed. To do this, I choose a simple application named ToroHello.pas which just writes "Hello World" to the serial console. To check this application, I set up Travis to do two things:

1. To compile ToroHello example and generate the booting image ToroHello.img.

2. To run the image in Qemu and test if the example worked.  

I based on a previous work in https://github.com/nielsAD/travis-lazarus. So 1) was easy to do and I got a ToroHello.img compiled by using Travis very quickly (see Figure 1). 

The task 2) took me a bit more time. I set up to run qemu after the building and to output the serial port to a file. Then, I tested if the output contains 'Hello world". In such a case, the test passed. Following pictures show the output of the job:

Figure 1. Travis compiling ToroHello.pas

Figure 2. Travis running ToroHello.img in Qemu and outputting the serial

Figure 2 shows the output from the serial port. We can see Toro's initialization at line 3574.

Figure 3. Result of the building

To summarizes, in a very short time I was able to compile ToroHello and then produce a test to verify its correct behavior. This is a simple example but it allows to show how powerful might be CI for a kernel. So I am planing to use Travis in two different levels of testing. The first level is to verify the compilation of the examples after a push and/or pull request. The second level is to write specific tests to measure performance, leaks, wrong behaviors, exception faults, etc. This needs however a lot more experimentation.

Matias

Porting genext2fs to Windows 10

Last weekend I spent some time to find a tool to create ext2 images in Windows. I found a wonderful tool named genext2fs which allows to create an ext2 image and to populate it with a directory structure. I could not find a way to compile it in windows so I put hands on it and I got a windows version that seems working. You can find the patch at the end of this post. However, it is a horrible patch that only makes the thinks work. If anyone knows a better way to do it, please don't hesitate to tell me. I compiled without any problem in cygwin64. We have to be careful when we run it since, for a reason that I don't understand, the program tries to put all current directory into the image. To prevent this, I just run it from the directory that I want to put into the image. To summarize, I am now able to create a ext2 image and populate it with a directory structure. For example, I am experimenting by populating ToroFiles.img with the files that are used by the webserver in Toro. To do so, I am running the following command: 

../genext2fs.exe -v -d ../ToroFiles -b 8192 ../ToroFiles.img

This generates a 8MB image named ToroFiles.img that contains the files in the directory ToroFiles. I executed the command inside the directory ToroFiles. In the next days, I will commit the example TorowithFilesystem.pas that uses this tool to populate ToroFiles.img. Together with the example, I will upload the instruction to run it so stay tuned!

Matias Vara

--------------------------------------------------------------------------------------------------------
This is a patch for genext2fs-1.4.1/genext2fs.c
--------------------------------------------------------------------------------------------------------

761c761
< readsize = readlink(path, buf, bufsize); /* 1st try */
---
> // readsize = readlink(path, buf, bufsize); /* 1st try */
1458c1458
< if(st->st_mode & S_IRGRP)
---
> /* if(st->st_mode & S_IRGRP)
1475c1475
< mode |= FM_ISVTX;
---
> mode |= FM_ISVTX; */
1631c1631
< lstat(dent->d_name, &st);
---
> stat(dent->d_name, &st);
1645c1645
< case S_IFLNK:
---
> // case S_IFLNK:
1652c1652
< case S_IFSOCK:
---
> // case S_IFSOCK:
1669c1669
< if (!S_ISDIR(st.st_mode) && !S_ISLNK(st.st_mode) && st.st_nlink > 1) {
---
> if (!S_ISDIR(st.st_mode) /* && !S_ISLNK(st.st_mode) */ && st.st_nlink > 1) {
1682c1682
< nod = mknod_fs(fs, this_nod, name, mode|FM_IFCHR, uid, gid, major(st.st_rdev), minor(st.st_rdev), ctime, mtime);
---
> // nod = mknod_fs(fs, this_nod, name, mode|FM_IFCHR, uid, gid, major(st.st_rdev), minor(st.st_rdev), ctime, mtime);
1685c1685
< nod = mknod_fs(fs, this_nod, name, mode|FM_IFBLK, uid, gid, major(st.st_rdev), minor(st.st_rdev), ctime, mtime);
---
> // nod = mknod_fs(fs, this_nod, name, mode|FM_IFBLK, uid, gid, major(st.st_rdev), minor(st.st_rdev), ctime, mtime);
1691c1691
< case S_IFSOCK:
---
> /* case S_IFSOCK:
1698c1698
< break;
---
> break; */
2392,2395c2392,2395
< if((pdir = open(".", O_RDONLY)) < 0)
< perror_msg_and_die(".");
< if(chdir(dopt[i]) < 0)
< perror_msg_and_die(dopt[i]);
---
> //if((pdir = open(".", O_RDONLY)) < 0)
> // perror_msg_and_die(".");
> //if(chdir(dopt[i]) < 0)
> // perror_msg_and_die(dopt[i]);
2397,2400c2397,2400
< if(fchdir(pdir) < 0)
< perror_msg_and_die("fchdir");
< if(close(pdir) < 0)
< perror_msg_and_die("close");
---
> //if(fchdir(pdir) < 0)
> // perror_msg_and_die("fchdir");
> //if(close(pdir) < 0)
> // perror_msg_and_die("close");
2593d2592
< 
2595d2593
<