Embedded and Automotive/Archived/PV Drivers/Project Proposal

This proposal condenses the discussion on this e-mail thread and follow-up as well as tries to follow up on 2013 Dev Summit BoF notes : Seeding an Android and Embedded ecosystem.

Embedded and Automotive PV Drivers Project Proposal

Roles

• Project Lead: Artem Mygaiev, EPAM Systems
• Project Sponsor: Philippe Robin, ARM representative on Xen Project Advisory Board
• Project Mentor: Lars Kurth - Lars is the Community Manager for xenproject.org and has agreed to act as the project’s Mentor.

Background

To build an automotive/embedded platform that involves the Xen Hypervisor a number of challenges need to be resolved. For example:

• Co-processor virtualization (GPU, DSP, IPU, etc.)
• Peripheral PV drivers support for guest OSes
• IOMMUs support for peripherals pass-through
• TEE integration support
• Deprivileged execution for "native" applications
• Real-time support
• Certification

Many of the issues (e.g. real-time support) are already being addressed by the Hypervisor team/subproject and *should be addressed there*.

PV Drivers for which there is no upstream: A number of vendors that care about embedded use-cases (e.g. EPAM, GlobalLogic, Galois, Samsung) started to develop Linux user space drivers for Android and started to add PV support for operating systems relevant to embedded (e.g. QNX and FreeRTOS). However, there is no official git repository where such drivers can be hosted. This subproject proposal is aiming to address this issue and

• Provide locations for PV drivers with no upstream
• Provide a mechanism, which allows coordination of several strands of work across different open source projects and Xen project teams

Community Focalpoint: One of the challenges in seeding a new subproject (recent examples within the Xen Project community were Xen on ARM) is to provide a focalpoint within the Xen Project community as well as within the wider open source community. It can over time also provide test, integration and other capabilities.

The experience trying to bootstrap automotive/embedded activity within the Xen Project since the topic first arose in November 2013, has been that existing communication channels (in particular xen-devel) have been too noisy for newcomers: in essence creating a barrier of entry. One way of addressing this, is to provide a quieter discussion space, a web portal, etc. - all of which comes with a subproject.

Subprojects further provide a clear mechanism to market the growth of the Xen Project codebase into new markets, which in turn helps such initiatives succeed.

Relevance to Xen and its Community

The purpose of this subproject is to enable seeding of a diverse community of vendors and end-users that intend to use Xen Project technology in automotive, embedded, mobile and other non-datacenter oriented use-cases. It is already clear (as the last 7 months have shown and similar experience from within Linux), that this will benefit the core Xen Project use-cases also.

Subproject Proposal

The proposal is to create a pvdrivers subproject with a strong initial emphasis on automotive and embedded use-cases in terms of messaging and the subproject portal due to the initial contributions from GlobalLogic and other vendors that need drivers. However longer term, it is likely and desirable that the project may be useful for other use cases.

The proposal would include have:

• A pvdrivers directory on xenbits
• A webportal on xenproject.org pointing to all other pages
• A pvdrivers category and portal on the wiki
• It's own mailing list pvdrivers-devel (note that as we agreed not to merge with Windows_PV_Drivers_Incubation_Project_Proposal we can't use the generic namespace pvdrivers-devel - in other words maybe something like embedded-pv-devel may be more appropriate) for the subproject, addressing concerns related to barrier of entry. However CCing other relevant lists (aka xen-devel/lkml) is common practice for discussions affecting upstreams: the expectation is that the subproject will follow these conventions.

Current Status

The following drivers are discussed as candidates for initial contribution. Note that the list of drivers is not final.

pv_audio driver provide audio driver back- and front-end for different instances of an operating systems (Linux and qnx)
Drivers for pvdrivers/*.git

  lnx-app-alsa-be/ Linux application back-end based on alsa
  qnx-be/          Qnx back-end
  qnx-fe/          Qnx front-end

Drivers to be upstreamed to Linux kernel (may live in pvdrivers specific development repositories temporarily)

  lnx-kd-be/       Linux kernel driver back-end
  lnx-kd-fe/       Linux kernel driver front-end

pv_fb driver provide audio driver back- and front-end for different instances of an operating systems (lnx and qnx)
Drivers for pvdrivers/*.git

  lnx-app-v4l-be/  Linux application back-end based on v4l framework 
  qnx-be/          Qnx back-end
  qnx-fe/          Qnx front-end

pv_events driver provide events driver back- and front-end for different instances of an operating systems (lnx and qnx)
Drivers for pvdrivers/*.git

  lnx-app-be/  Linux application back-end
  qnx-be/      Qnx back-end
  qnx-fe/      Qnx front-end

pv_usb driver provide events driver back- and front-end for different instances of an operating systems (lnx and qnx)
Drivers for pvdrivers/*.git

  qnx-be/       Qnx back-end
  qnx-fe/       Qnx front-end

Drivers to be upstreamed to Linux kernel (may live in pvdrivers specific development repositories temporarily)

  lnx-kd-be/    Linux kernel driver back-end
  lnx-kd-fe/    Linux kernel driver back-end

pv_gpu driver provide events driver back- and front-end for different instances of an operating systems (lnx and qnx)
Drivers for pvdrivers/*.git

  qnx-fe/
  qnx-be/
  lnx-alsa-be/

Drivers to be upstreamed to Linux kernel (may live in pvdrivers specific development repositories temporarily)

  lnx-be/
  lnx-fe/

A number of code changes to xen.git and linux.git are also candidates for upstreaming to Xen and Linux, but may temporarily be hosted in pvdrivers specific development repositories. This is necessary, because some of the drivers above depend on a small number of modifications to Xen and Linux that are not yet upstreamed and some will need tidying and maybe re-factoring before they are upstreamable.

Licenses

Linux

Most of the Xen PV frontends/backends inside of the Linux kernel are under a dual-license like the following (this one is from blkfront). The proposal is to follow this convention.

 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License version 2
 * as published by the Free Software Foundation; or, when distributed
 * separately from the Linux kernel or incorporated into other
 * software packages, subject to the following license:
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this source file (the "Software"), to deal in the Software without
 * restriction, including without limitation the rights to use, copy, modify,
 * merge, publish, distribute, sublicense, and/or sell copies of the Software,
 * and to permit persons to whom the Software is furnished to do so, subject to
 * the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
 * IN THE SOFTWARE.

This would allow driver code and headers to be imported/shared with other OSes (e.g. BSD's).

Some backend code may be under Apache 2 license, so can be integrated into software with commercial licenses.

QNX and other OSes

QNX recommends Apache 2.0 license for QNX drivers (which are OSI approved). GlobalLogic intends to make the code available as part of the subproject. To build these drivers, the QNX GNU libc variant is used (licensed under LGPL 2.1). Other tools, libraries and header files that may need to be used are listed in QNX license guide and [1] for reference.

• GPL (v2 and v3) depending on tools: QNX GNU Compiler Collection, GNU Binutils, GNU Debugger
• Misc open source licenses: System libraries, see QNX license guide for a detailed breakdown

Important note: we will need to run a license check for the whole imported codebase to build the drivers to make sure nothing unexpected or problematic is imported.

Build Dependencies and 3rd Party Tools

• QNX drivers (and software in general) can be entirely built with tools provided under open source licenses. More specifically QNX GNU Compiler Collection, GNU Binutils, GNU Debugger
• QNX provides the Eclipse based Momentics Tool Suite: this suite is not necessary for development. Commercial organisations have to procure a license for Momentics, but a free variant is available for for non-commercial developers (see [2]).

Other OSes (future proofing)

Must have OSI approved licenses that are required to link with and use the drivers within the native OS. This is a requirement for all software hosted by the Linux Foundation (and thus the Xen Project).

Incubation

The aim of the (Embedded and Automotive) PV Drivers subproject is to

• Provide a collaborative space for vendors and individuals to collaborate on embedded/automotive/mobile use-cases using Xen Project technology
• Build a library of PV drivers for different operating systems that work with upstream Xen and Linux
• Raise the profile of embedded/automotive/mobile use-cases to members of the Xen Project community and the wider open source community
• Build a sustainable and diverse developer community around these technologies
• Longer term extend the focus to be more general

Although we do not have any strict criteria for graduation in general, the conventions that have emerged in the context of Mirage OS are: at least 3 vendors are contributing to the project, project code works with upstream Linux, Xen and other dependent upstreams, there has been one official release (in this case it would be a set of PV drivers working with Xen 4.x and a specific version of Linux and QNX). However, due to the nature of PV drivers, it is likely that sets of drivers for different guest operating systems Typically we aim to time graduation with a release.

Community members have also requested that any development trees have been retired (e.g. merged upstream or otherwise obsolete) before graduation.

Related Projects and Proposals (Open Question)

We could - and probably should - at least share resources between the Windows PV Drivers Incubation Project Proposal to avoid fragmentation.

We should also consider to merge the Windows PV Drivers Incubation Project Proposal with this proposal, but provide separate repositories for Windows PV Drivers and build infrastructure because of dependencies on proprietary Microsoft software.

Required Infrastructure

Repositories

Official Repositories: Repositories will be created on http://xenbits.xenproject.org/ under a pvdrivers directory (which is proposed not to be browsable through gitweb). This would look as follows:

  git://xenbits.xenproject.org/pvdrivers/repo-1.git
  ...
  git://xenbits.xenproject.org/pvdrivers/repo-n.git

Code in these directories is expected to work with upstream Linux and Xen.

Initial repositories to be created:

• linux-user-drivers.git to host Linux user space PV drivers
• qnx-drivers.git to host QNX user space drivers

Development Repositories: (optional, may decide to use personal trees instead)
To help with ongoing development and to provide a location for the initial code contribution the subproject may own a number of development repos in

  git://xenbits.xenproject.org/pvdrivers/development/*.git

More specifically

  git://xenbits.xenproject.org/pvdrivers/development/xen.git
  git://xenbits.xenproject.org/pvdrivers/development/linux.git
  git://xenbits.xenproject.org/pvdrivers/development/xen-linux-user-drivers.git
  git://xenbits.xenproject.org/pvdrivers/development/xen-qnx-drivers.git

following the convention of other Xen Project teams.

Personal Branches: will be handled as for all other subprojects and will be stored

  git://xenbits.xenproject.org/people/*.git

Root namespace on xenbits

This subproject will share the

  git://xenbits.xenproject.org/pvdrivers/

namespace with the already approved Windows PV Drivers Incubation Project Proposal, which will use

  git://xenbits.xenproject.org/win/*.git

for its repository. The rationale is that this will make it easier to locate PV drivers repos on xenbits, while keeping developer communities separate.

Mailing Lists and Web Real Estate

The following infrastructure shall be created for the subproject:

• A embedded-pv-devel mailing list
• A webportal on xenproject.org pointing to all repositories and highlighting use-cases
• A pvdrivers and embedded category and portal on the wiki

Note that the exact naming can still be changed after the project has been created.

Build and Test

Build infrastructure for the embedded and automotive repository should be established in the short term Tests shall be performed by GlobalLogic on a regular basis in the short term (or more generally by owners of repositories in the subproject tree), until an automated solution can be found.

Due Diligence prior to initial contribution

Verify that

• all of the pre-requisites source files currently available and under a suitable license? (license check)
• are the build instructions complete and up to date?
• does the software compile and build cleanly?

Review/Approval Status

• Approved July 29th
• From 1-11 July : Community Review (see [3])
• Modifications made based on feedback
• Community Vote from July 15 - 23 (see [4])

TODO to bootstrap project

• Create win-pv-devel@lists@xen.project.org or [5] ✓
• Lars to create web portal ✓
• Lars to create wiki portal page ✓
• Lars to plan PR ✓
• Lars to request creation of embedded-pv-devel mailing list ✓
• Lars and Artem to nominate initial set of maintainers for the project ✓
• Artem and Andrii to follow up on Andrii's personal repo (it is not currently listed on xenbits) ✓
• Artem to send list of repository names to Lars and Ian Jackson such that these (as well as the directory structure) can be created ✓
• Artem, Lars and Brian to work on due diligence for initial contribution ✓ note that these all occurred in upstream Xen and Linux locations
• Lars to work on further PR ✓