Unikraft Porting Libraries Walkthrough

In order to port an existing application on top of Unikraft, one has to port its dependencies using Unikraft external libraries. Each external library will have its own project or repository and, ideally, no source file will be necessary. Of course, there are exceptions which we will talk about later in this article. The most difficult part when creating an external library is translating the original build system of the library to Unikraft build system. In other words, one has to understand how the original build systems works in order to identify which source files are compiled, how the generated file objects are linked together and which flags are used for these operations. If the original build system is Makefile based, then this task is pretty straight-forward. However, with more complex build systems this task becomes more difficult, especially if the person doing the porting is not a developer of the original code.

This article presents the steps involved in creating an external library. We will be using the pcre library as an example throughout this article. We chose pcre because it is a popular dependency among Debian/Ubuntu packages, it has a small number of source files and it required only 3 commits to fully port it for Unikraft.

Steps for creating the external library

Every external library should have the following files:

.
├── .gitignore
├── CODING_STYLE.md
├── CONTRIBUTING.md
├── COPYING.md
├── Config.uk
├── MAINTAINERS.md
├── Makefile.uk
└── README.md

Some of the files will be exactly the same for all external libraries, some will have some small changes and some will be completely different. For simplicity, we propose the commits we created for pcre to be adapted for your libraries as well. We will describe each commit in the following subsections.

Initial empty commit (optional)

It is always a good idea to start with an empty commit because it helps on rebasing when amending commits.

Introducing the library skeleton

You can simply copy and adapt the first commit of pcre, which introduces the library skeleton. The .gitignore, CODING_STYLE.md and CONTRIBUTING.md files will be exactly the same. All you have to do is:

1. adapt the symbol names in Config.uk
2. change the maintainers in MAINTAINERS.md
3. pick a license to be used throughout the external library and copy it in COPYING.md; for example, pcre uses BSD-3-clause license
4. adapt the library references in README.md

Add Makefile.uk

The second commit should add the Unikraft Makefile.uk file. This is the most important file, similar to regular Makefiles, it contains the list of source files to be compiled and the building flags. The Makefile.uk file of pcre external library has the following sections:

• Library registration - This is where you register your new library to the Unikraft build system. We have two registration directives there, the second one is commented out because it would register the glue code as a library of its own. In exceptional case, one would need to add some glue code in order to make the ported library work with Unikraft (we will revisit this below). Fortunately, for pcre this wasn’t the case.
• Original sources - Here are the variables which keep the information about the location of the archive containing the original code. The fetching directive does just that: fetching the compressed archive and decompressing it. You will notice that there is also a patching directive which is not doing anything for pcre. There are other exceptional cases when patches must be applied to the original code in order to make it work with Unikraft. Usually, these patches correct some wrong doings in the origin code: bugs, code that doesn’t comply with standards, code that is enabled for well-known operating systems and should also be enabled for Unikraft, etc. You can find more information about patches below.
• Helpers - Here is where you define auxiliary variables, if you need any.
• Library includes - These are the include paths that should be captured from the original build. The external library might need its own headers directory (here represented by $(LIBPCRE_BASE)/include), but this is not the case for pcre.
• Global flags - You will find that quite a lot of libraries have a lot of warnings. Patching the code wouldn’t be the smartest idea in that case, so that’s why we should choose to disable warnings that don’t indicate critical issues. Here is also where you should add the compilation flags and the preprocessing symbol definitions (e.g. HAVE_CONFIG_H for pcre) which are required for the ported library.
• Glue between Unikraft and pcre - As we mentioned it above, there might be some cases where some glue code is required to make things work. This is where you write which source files contain the glue code.
• Library code - This contains the list of original source files. Be careful to select only those source files which are required for the library you want to build. An original build may generate multiple libraries and executables. For example, pcre original code contains two libraries and two test executables, so we had to pick only the source files of the main library.
• Library prepare - Some libraries might require additional steps before compiling. For example, autotools based libraries need ./configure to be run before running make. Some source files might be generated, other might be just symbol links to existing files. This section contains all the custom rules required for building the original code. Notice that the first custom rule should depend on the $(LIBNAME_BUILD)/.origin file which will be created after decompressing the archive. In the pcre Makefile we used LIBPCRE_PREPARED_DEPS just as an auxiliary variable, this is not mandatory. The last two directives are always mandatory for prepare rules in order to be properly registered in the Unikraft build system.

Add patches for the original code (optional)

As mentioned in the previous section, sometimes the original code has to changed in order to make it compile or to fix some bugs. What you have to do is to create patches which will be automatically applied by the Unikraft build system after downloading the source code. All the patches should be put in the patches/ subdirectory, named in order. You can find plenty of examples in our previous projects, e.g. newlib, pthread-embedded. In order to create the patches, one way is to create a local git repository for the original code after the Unikraft build systems downloads it and commit the changes there.

Export symbols

Unikraft uses exportsyms.uk file for exporting library symbols, such as function and variable names. The list of exported symbols should contain only the API symbols, meaning that one has to get familiar with the library in order to know the API. The library itself might have more global symbols, but the extra ones will have the visibility updated to private by the Unikraft build system. For pcre we used the symbols listed in its API documentation.

Please keep the symbols sorted alphabetically so that the reviewing may be done easier.

Update: We currently skip exporting symbols, so this step should be ignored.

Adding extra configuration symbols (optional)

An optional step after making the changes above would be to add some configuration symbols which would be used in enabling and configuring features of the ported library. You may find a good example in the Config.uk file of our pthread-embedded port which adds configuration symbols for two features in the original code and for building its unit tests.

Adding extra information (optional)

For some external libraries, one would need additional information about how to use it. For example, pthread-embedded library needs to stay before newlib when adding them to the value of LIBS variable in Makefile file of the application. Information like this would have to stay in the README.md, therefore a new commit adding this information would be needed.

Further information

• A walkthrough of the build system can be found here. The slide set explains how Unikraft's build system can be fed to do the job.
• You can find more information about the syntax and the predefined variable names of Config.uk and Makefile.uk files on the Unikraft developer's guide or in the guides under the doc/ subdirectory in the source tree.