OpenStack CI Loop for Xen-Libvirt: Difference between revisions
Lars.kurth (talk | contribs) |
(→CI Loop: Add the repo that is used to deploy the ci loop) |
||
| (30 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
{{TODO|Put the content of this page together - aka a user manual for how to interpret results and how to investigate issues}} |
|||
= CI Loop = |
= CI Loop = |
||
| Line 7: | Line 6: | ||
* [https://www.mirantis.com/blog/setting-external-openstack-testing-system-part-2/ Setting Up an External OpenStack Testing System – Part 2] |
* [https://www.mirantis.com/blog/setting-external-openstack-testing-system-part-2/ Setting Up an External OpenStack Testing System – Part 2] |
||
* [https://github.com/openstack/tempest The Test Suite we run (OpenStack Tempest)] |
* [https://github.com/openstack/tempest The Test Suite we run (OpenStack Tempest)] |
||
* [http://docs.openstack.org/developer/tempest/overview.html Tempest Overview] |
|||
* [https://blog.xenproject.org/2015/05/20/xen-project-now-in-openstack-nova-hypervisor-driver-quality-group-b/ Xen Project now in OpenStack Nova Hypervisor Driver Quality Group B] |
|||
Information on how this CI loop is deployed can be found in this repo: |
|||
* [https://xenbits.xen.org/gitweb/?p=openstack/ci-loop-config.git https://xenbits.xen.org/git-http/openstack/ci-loop-config.git] |
|||
It follows instruction provide at [http://docs.openstack.org/infra/openstackci/third_party_ci.html third_party_ci]. |
|||
== Baseline == |
== Baseline == |
||
Our baseline for testing is ... |
|||
Ubuntu Xenial Xerus 16.04 LTS. It comes with: |
|||
* Xen 4.6 |
|||
* Libvirt 1.3.1 |
|||
* Linux 4.4 |
|||
== Usage Instructions == |
== Usage Instructions == |
||
Our OpenStack CI loop interface is available at [http://jenkins.openstack.xenproject.org/ jenkins.openstack.xenproject.org]. |
Our OpenStack CI loop interface is available at [http://jenkins.openstack.xenproject.org/ jenkins.openstack.xenproject.org]. The CI loop is now integrated with OpenStack Gerrit, in '''voting mode''' (Group B). This means that functional testing provided by our CI loop does not gate commits, but advises patch authors and reviewers of results in gerrit. |
||
{{WarningLeft|Note that the DNS is broken, but you can access the CI loop via http://104.130.29.226 and http://104.130.29.226:8080}} |
|||
The CI loop tests our [[#Baseline|baseline]] against each Nova OpenStack commit. |
The CI loop tests our [[#Baseline|baseline]] against each Nova OpenStack commit. |
||
| Line 21: | Line 34: | ||
<br clear=all> |
<br clear=all> |
||
'''OpenStack review''': Each build maps onto a changeset on [https://review.openstack.org/]. The following figure shows how to map a test run on our CI loop to a specific OpenStack review. |
'''OpenStack review''': Each build maps onto a changeset on [https://review.openstack.org/ review.openstack.org]. The following figure shows how to map a test run on our CI loop to a specific OpenStack review. |
||
<br clear=all> |
<br clear=all> |
||
[[File:OSCI-Review.png|left|700px|thumb|Shows how you get from a test to an OpenStack review]] |
[[File:OSCI-Review.png|left|700px|thumb|Shows how you get from a test to an OpenStack review]] |
||
<br clear=all> |
<br clear=all> |
||
'''Aborted builds''': Sometimes, jobs are aborted. The main reason for aborted jobs is that the OpenStack CI loop run at the OpenStack Foundation (or another voting 3rd party CI loop) may have determined that a change breaks a test. In other words, it is not necessary to complete the test in this case. Aborted jobs are typically marked with a grey circle. Note that this is nothing to worry about |
|||
'''Aborted builds''': |
|||
=== Checking up on Test Failures === |
=== Checking up on Test Failures === |
||
Test failures can be caused by |
|||
* an OpenStack commit [https://review.openstack.org/ review.openstack.org] |
|||
* an issue in libvirt |
|||
* an issue in Xen |
|||
The first thing to do is to check whether a failure has been caused by an OpenStack commit [https://review.openstack.org/ review.openstack.org]. If that is the case, the OpenStack community will fix the issue and re-submit the changeset. |
|||
<br clear=all> |
|||
[[File:OSCI-Gerrit.png|left|700px|thumb|Shows a test failure against the OpenStack CI Loop hosted by the OpenStack Foundation]] |
|||
<br clear=all> |
|||
That is unless, the issue is specific to the Xen Project. In that case, there is an expectation that we help the committer investigate the issue. Note that you can also use [http://zuul.openstack.xenproject.org/scoreboard zuul.openstack.xenproject.org/scoreboard] to get a comparison which tests failures were Xen specific. |
|||
The figure below, shows how you can find the test reports and how you verify that an issue is Xen+Libvirt specific. |
|||
<br clear=all> |
|||
[[File:OSCI-Failure1.png|left|700px|thumb|In this figure you see that the change was verified on the CI Loop hosted by the OpenStack Foundation. The figure also shows you how you get to the test reports.]] |
|||
<br clear=all> |
|||
The figure below, shows you how you get to the libvirt and libxl libvirt driver log files, which you will need to investigate and reproduce an issue. |
|||
<br clear=all> |
|||
[[File:OSCI-Failure2.png|left|700px|thumb|In this figure you see how you get to the libvirt and libxl libvirt driver log files.]] |
|||
<br clear=all> |
|||
=== Scripts to investigate issues === |
|||
A simple couple of bash lines to download the full console logs from the jobs and create a histogram of tempest tests caused the majority of the failures: |
|||
<pre> |
|||
# for i in {1106..1374}; do wget http://104.239.148.40:8080/job/dsvm-tempest-xen/${i}/consoleFull -O ${i}_console; done |
|||
# grep -h "\.\.\. FAIL" * | sed -e 's/.*\(tempest[^) ]*\).*/\1/' | sort | uniq -c | sort -n |
|||
</pre> |
|||
Note that the <pre>in {<start test-run>..<end test-run>}</pre> portion of the script needs to be adjusted to the start and end test run. |
|||
= Community Activities = |
= Community Activities = |
||
| Line 36: | Line 76: | ||
== Meeting Minutes == |
== Meeting Minutes == |
||
* [http://lists.xenproject.org/archives/html/wg-test-framework/2015-04/msg00096.html Minutes of monthly Xen+Libvirt OpenStack meeting (April 22nd)] |
|||
* [http://lists.xenproject.org/archives/html/wg-test-framework/2015-03/msg00086.html Minutes of monthly Xen+Libvirt OpenStack meeting (March 25th)] |
* [http://lists.xenproject.org/archives/html/wg-test-framework/2015-03/msg00086.html Minutes of monthly Xen+Libvirt OpenStack meeting (March 25th)] |
||
Latest revision as of 12:35, 25 November 2016
CI Loop
This article provides some information about jenkins.openstack.xenproject.org, which is the front-end for the Xen Project OpenStack CI loop, which is a 3rd party OpenStack CI loop (3rd party because it is operated by the Xen Project, not the OpenStack Foundation). You can find more information about 3rd party OpenStack testing systems in the following articles
- Setting Up an External OpenStack Testing System – Part 1
- Setting Up an External OpenStack Testing System – Part 2
- The Test Suite we run (OpenStack Tempest)
- Tempest Overview
- Xen Project now in OpenStack Nova Hypervisor Driver Quality Group B
Information on how this CI loop is deployed can be found in this repo:
It follows instruction provide at third_party_ci.
Baseline
Our baseline for testing is ...
Ubuntu Xenial Xerus 16.04 LTS. It comes with:
- Xen 4.6
- Libvirt 1.3.1
- Linux 4.4
Usage Instructions
Our OpenStack CI loop interface is available at jenkins.openstack.xenproject.org. The CI loop is now integrated with OpenStack Gerrit, in voting mode (Group B). This means that functional testing provided by our CI loop does not gate commits, but advises patch authors and reviewers of results in gerrit.
|
Note that the DNS is broken, but you can access the CI loop via http://104.130.29.226 and http://104.130.29.226:8080 |
The CI loop tests our baseline against each Nova OpenStack commit.
Dashboard
The most useful view to get an overview over test failures can be found at the Build History page.
OpenStack review: Each build maps onto a changeset on review.openstack.org. The following figure shows how to map a test run on our CI loop to a specific OpenStack review.
Aborted builds: Sometimes, jobs are aborted. The main reason for aborted jobs is that the OpenStack CI loop run at the OpenStack Foundation (or another voting 3rd party CI loop) may have determined that a change breaks a test. In other words, it is not necessary to complete the test in this case. Aborted jobs are typically marked with a grey circle. Note that this is nothing to worry about
Checking up on Test Failures
Test failures can be caused by
- an OpenStack commit review.openstack.org
- an issue in libvirt
- an issue in Xen
The first thing to do is to check whether a failure has been caused by an OpenStack commit review.openstack.org. If that is the case, the OpenStack community will fix the issue and re-submit the changeset.
That is unless, the issue is specific to the Xen Project. In that case, there is an expectation that we help the committer investigate the issue. Note that you can also use zuul.openstack.xenproject.org/scoreboard to get a comparison which tests failures were Xen specific.
The figure below, shows how you can find the test reports and how you verify that an issue is Xen+Libvirt specific.
The figure below, shows you how you get to the libvirt and libxl libvirt driver log files, which you will need to investigate and reproduce an issue.
Scripts to investigate issues
A simple couple of bash lines to download the full console logs from the jobs and create a histogram of tempest tests caused the majority of the failures:
# for i in {1106..1374}; do wget http://104.239.148.40:8080/job/dsvm-tempest-xen/${i}/consoleFull -O ${i}_console; done
# grep -h "\.\.\. FAIL" * | sed -e 's/.*\(tempest[^) ]*\).*/\1/' | sort | uniq -c | sort -n
Note that the
in {<start test-run>..<end test-run>}portion of the script needs to be adjusted to the start and end test run.
Community Activities
Training Session (week of May 4th)
We are planning a training session the week of May 4th or the week after, For more details see here. This will also include a discussion on how to integrate the CI loop into the project's workflow.
