Difference between revisions of "OpenStack CI Loop for Xen-Libvirt"

From Xen
m (Scripts to investigate issues)
m (Scripts to investigate issues)
Line 67: Line 67:
 
# grep -h "\.\.\. FAIL" * | sed -e 's/.*\(tempest[^) ]*\).*/\1/' | sort | uniq -c | sort -n
 
# grep -h "\.\.\. FAIL" * | sed -e 's/.*\(tempest[^) ]*\).*/\1/' | sort | uniq -c | sort -n
 
</pre>
 
</pre>
Note that the <pre>in {<start test-run>..<end-test-run>}</pre> portion of the script needs to be adjusted to the start and end test run.
+
Note that the <pre>in {<start test-run>..<end test-run>}</pre> portion of the script needs to be adjusted to the start and end test run.
   
 
= Community Activities =
 
= Community Activities =

Revision as of 10:27, 24 April 2015

Icon todo.png To Do:

Update, when we have enabled the voting feature in Jenkins. Right now we have been given voting rights for Nova. But we want to monitor for a few days to ensure we will vote sensibly.


CI Loop

This article provides some information about jenkins.openstack.xenproject.org, which is the front-end for the Xen Project OpenStack CI loop, which is a 3rd party OpenStack CI loop (3rd party because it is operated by the Xen Project, not the OpenStack Foundation). You can find more information about 3rd party OpenStack testing systems in the following articles

Baseline

Our baseline for testing is ...

Ubuntu xen package 4.4.1-0ubuntu0.14.04.4 (Xen 4.4.1), with the following 4 patches applied

  • 9369988 libxl: event handling: Break out ao_work_outstanding
  • f1335f0 libxl: event handling: ao_inprogress does waits while reports outstanding
  • 4783c99 libxl: In domain death search, start search at first domid we want
  • 188e9c5 libxl: Domain destroy: fork

We are using the Ubuntu libvirt package 1.2.2 libvirt_1.2.2-0ubuntu13.1.7 as baseline, updated it to the most recent libvirt release, version 1.2.14, with the 3 missing libvirt patches from Jim Fehlig in the series outlined in this mail

Usage Instructions

Our OpenStack CI loop interface is available at jenkins.openstack.xenproject.org. Currently we are ramping up and the CI loop is not yet voting on OpenStack Nova. In other words, changes to OpenStack Nova which break Xen, will not be automatically rejected. Bot hopefully, this will happen soon.

The CI loop tests our baseline against each Nova OpenStack commit.

Dashboard

The most useful view to get an overview over test failures can be found at the Build History page.

Screenshot of the Dashboard


OpenStack review: Each build maps onto a changeset on review.openstack.org. The following figure shows how to map a test run on our CI loop to a specific OpenStack review.

Shows how you get from a test to an OpenStack review


Aborted builds: Sometimes, jobs are aborted. The main reason for aborted jobs is that the OpenStack CI loop run at the OpenStack Foundation (or another voting 3rd party CI loop) may have determined that a change breaks a test. In other words, it is not necessary to complete the test in this case. Aborted jobs are typically marked with a grey circle. Note that this is nothing to worry about

Checking up on Test Failures

Test failures can be caused by

The first thing to do is to check whether a failure has been caused by an OpenStack commit review.openstack.org. If that is the case, the OpenStack community will fix the issue and re-submit the changeset.

Shows a test failure against the OpenStack CI Loop hosted by the OpenStack Foundation


That is unless, the issue is specific to the Xen Project. In that case, there is an expectation that we help the committer investigate the issue.

The figure below, shows how you can find the test reports and how you verify that an issue is Xen+Libvirt specific.

In this figure you see that the change was verified on the CI Loop hosted by the OpenStack Foundation. The figure also shows you how you get to the test reports.


The figure below, shows you how you get to the libvirt and libxl libvirt driver log files, which you will need to investigate and reproduce an issue.

In this figure you see how you get to the libvirt and libxl libvirt driver log files.


Scripts to investigate issues

A simple couple of bash lines to download the full console logs from the jobs and create a histogram of tempest tests caused the majority of the failures:

# for i in {1106..1374}; do wget http://104.239.148.40:8080/job/dsvm-tempest-xen/${i}/consoleFull -O ${i}_console; done
# grep -h "\.\.\. FAIL" * | sed -e 's/.*\(tempest[^) ]*\).*/\1/' | sort | uniq -c | sort -n

Note that the

in {<start test-run>..<end test-run>}

portion of the script needs to be adjusted to the start and end test run.

Community Activities

Training Session (week of May 4th)

We are planning a training session the week of May 4th or the week after, For more details see here. This will also include a discussion on how to integrate the CI loop into the project's workflow.

Meeting Minutes