AB Votes/COLO-Contracts-June2015

Voting Summary

From: Lars Kurth <lars.kurth@xxx>
Date: Wednesday, 24 June 2015 11:06
Subject: Re: [Advisory-board] Urgent: AllNet Xen Project Contract Renewal (Action required before June 23rd)

All,
in summary, it seems we have all votes that are required.
We need 6 positive votes and have them from
* Oracle
* Citrix
* Intel
* ARM
* Rackspace
* Cavium
Best Regards
Lars

3 Resolutions Re AllNet Xen Project Contract Renewal (Action required before June 23rd)

From: Lars Kurth <lars.kurth@xxx>
Date: Thursday, 18 June 2015 17:20
Subject: [Advisory-board] Urgent: AllNet Xen Project Contract Renewal (Action required before June 23rd)

Dear Advisory Board Members,

at this week's AllNet – Xen Project call the question of whether the work that AllNet performed for the Project to set up the Xen Project Test Lab is complete. I will need your input, ideally before June 23rd. The first two sections contain background and recommendations. Section "3) plan of record" contains how we plan to approach AllNet and communicate section "4) Resolutions" contains the section which requires votes. We need 6 positive votes. 

Please vote with "+1", "0", "-1 explanation" as usual 

= 1) Closing / Accepting AllNet initial installation of HW in the COLO =

The relevant sections in the attached agreement  are sections 7 (Installation), 8 (Testing and Certification) and 9 (Acceptance) describe the conditions to close the contract. AllNet makes the case that each machine has undergone 7 (Installation) and 8 (Testing and Certification) and that we should Accept (see 9) their work by signing the Final Certificate of Acceptance.

The remaining 3 issues that we are dealing with are due to either HW or SW faults with some of the test machines in the test lab. AllNet argues, that they have tried to resolve issues on a best effort basis and that that the remaining 3 issues are not their fault, but faults with the HW or SW that surfaced after 7 and 8 had been performed initially. In other words, this type of activity should in future be covered by the maintenance agreement, with hardware problems covered by the hardware warranty.

We currently have 3 open issues:
A) The motherboard of one set of test machines died under warranty and is currently being resolved. The affected machine was in production for a very short period of time. 
B) Another HW or BIOS issue surfaced in another set of machines after the machine was being installed and has undergone Testing and Certification, which failed
C) We have a yet unidentified HW issue that recently started affecting another machine which was already in production
D) We do have sufficient capacity to run testing on behalf of the community despite issues A-C

Both Ian and I believe that this viewpoint is acceptable and that we should "accept" that the initial contract was completed. The only grounds on which we could refuse completion is possibly B.

= 2) Recommendation to *not* start the maintenance contract with AllNet =

However, both Ian and I recommend *not to start a new maintenance agreement* with AllNet for another 12 months that cannot be terminated before the 12 months at a cost of $1474 per month. We initially didn't sign the agreement as it was valid for 12 months from signing.

The basis for Ian's and my recommendation is that
* AllNet have not proved as competent as we originally anticipated and a string of re-work (sometimes several times) was required : this led to a 3 months project being delayed by 4 months
* Many mistakes were made, which led to significant delays and to the production system falling over several times
* AllNet has not shown to be as competent in Linux system administration as they claimed and is needed for the task: members of the Xen Project community had to step in and help out to get the project off the ground. This has affected the capability of the project to take in patches.

We have started to investigate alternative suppliers, who can a) perform HW and SW maintenance, b) install new HW and c) in addition look after software services such as Wiki, Mailing Lists, etc. We do have a number of leads (currently 2) and are investigating further leads, but we do not know how long this process will take (in the best case 2 weeks, worst case several months).

This exposes the project to the risk that we do not have on-ground support, should there be significant HW issues in the Test Lab. There are a number of possible scenarios:
A) SW issues : these should all be resolvable remotely
B) HW issues in one of the 24 test machines : in this case we would take the machines off-line leading to a reduction of capacity
C) HW issues in one of the 2 control machines and remote serial access : in this case, we would try to work with Earthlink (our centre provider) and have a remote hands service. In the worst case scenario Ian could fly out to Marlborough and fix the issue. However we may have a few days down-time in such a case.
D) If we can't find an alternative supplier within 2-3 months, expansion plans may be impacted
 
Ian believes the risk of B) and in particular C) is fairly small. Also, taking the risk is preferably to continuing to work with AllNet: some of the recent issues  we have seen with the Production Test system were caused by switching off parts of the live machines by mistake and other problems that should not occur. 

= 3) Plan of Record =

Unless, there is disagreement by the board, we suggest the following action
* On June 23rd we will rescind AllNet access to systems as a precaution
* In the meeting on the 24th, we will inform AlNet that we (that is Ian and I) will *recommend* to the Test WG and the Advisory Board …
** To Accept the Initial Install of the COLO
** But that we will not recommend to start the maintenance contract 
* We will try and work on a plan for handover as it is likely that AllNet will still have 2-4 machines which will not sit in the rack in the COLO at this time

Alternatively we could stall the discussion with AllNet, but this seems unethical and unfair.

= 4) Resolutions = 

Resolution 1: Accept the completion of the original contract with AllNet to procure, test, and install SW and HW for the 

Resolution 2:  Support the recommendation of Ian J and Lars K not to start the AllNet maintenance contract despite the risks involved

Resolution 3: Agree with the plan of Record

= Practical and Legal Issues (for the Linux Foundation) =
These are questions to Laura Kempke and Mike Woster
* Can you check whether the proposal above is acceptable from the point of view of Linux Foundation Code and Conduct and legally
* Who would need to sign the necessary formal agreements to close item 1 – in other word, who would sign the Final Certificate of Acceptance?

Best Regards
Lars