Category Archives: Third-party CI

Sharing Best Practices BoF: Third Party CI Working Group at Liberty Summit

·

We held our second Third Party CI Working Group meet up at the Vancouver OpenStack Liberty Summit. Because of limited space and a proposed user session, we were grouped together with the previously proposed Sharing Best Practices in Setting up and Operating OpenStack CI Loops Birds of a Feather session. Despite the fact that we have more that 100 CI systems, and that we had more than 100 attendees sign up, we had plenty of space. I was worried about the available space since the room only seated about 50 and we had 80-100 in Paris. I’m not exactly sure why we didn’t have the same turnout, maybe too short of an advertising runway after the cross-project sessions were finally decided, or the small space assigned in the back of the second building, regardless, it was really nice to meet more friendly CI operators and share experience. I estimate about 35-40 attended.

The Xen folks welcomed us with open arms into their BoF session. We decided to split the meeting into their CI overview and experience, and then to leave time for working group discussions. Stefano Stabellini and Bob Ball, both from Citrix, presented their experience bringing up a CI system for testing Xen + Libvirt, and explained the difference between XenServer driver testing and what they were doing for Xen + Libvirt.

IMG_20150520_140428796_a

Bob Ball presenting

Following this, we had some time to discuss some of the agenda items agreed upon in the weekly Third-party CI working group meetings. These items are listed in the etherpad for the session, so I won’t repeat them here. As always, time passed way too fast and we got little decided to further the work items identified in the etherpad.

Just like I did at the Paris session, I asked those attending a few questions and did a quick show of hands and came up with some somewhat surprising answers:

  1. The majority attending had already set up, or were in the process of setting up a third party CI system to test a driver or OpenStack on a different platform.
  2. There were only a few that were there to learn about OpenStack third party CI in advance of starting their test system.
  3. From those in #1 that had a system, only a few had even heard of the third party CI meetings we hold every week.
  4. Most did not monitor the openstack-dev maillist or ever get on IRC. Wow.
  5. Everyone I talked to said they would be willing to share tools they had written.

As a result, I did a quick overview of the meetings that are held weekly and urged everyone to get involved. My takeaway actions are:

  1. We must increase participation. The questions is, how do we reach teams if they do not monitor email or get on IRC?
  2.  We need to do a better job of advertising that help is available.

If you are reading this, please come to the weekly meetings that fit into your schedule. We can improve the consumability of OpenStack CI and our test systems by sharing our experience and tools. We have a repository now, and there are already some good tools there contributed from other systems. Spread the word.

Thanks everyone that attended the BoF session in Vancouver!

 

Third Party CI Tools Repo

·
The OpenStack third party CI working group has been discussing a way to share tools, configurations, and plugins best practices. This was an idea that started at the Paris summit because teams that are operating external CI systems have each created tools that make their life as CI operators easier. These tools are potentially valuable to all CI systems, and need to be shared. Now we have a way to share them.

I created this new repo with the following CI operators as initial cores in addition to myself. They were selected and voted in because they have consistently attended the working group meetings, helped fellow operators, and pushed to move work items forward in the community. They are: Ramy Asselin (asselin), Patrick East (patrickeast), Steve Weston (sweston), and Mikhail Medvedev (mmedvede).

Since this is a StackForge project, it shares the OpenStack patch process.  If you are interested in sharing something you have written that makes your CI life easier, please come to a working group meeting and discuss, or just submit a patch.

Clean up that swift object store

·

My team is using an OpenStack swift object storage service through SoftLayer as a way to publish the required logs to the community from our PowerKVM Continuous Integration testing. The test system ran like this for several months before we determined how to set the expiration on an object to have swift expire and remove a log after 30 days. No big deal really until we started to run out of space on our object store allocation and were being charged overages. This was not major expense, but it wasn’t right to leave all these old logs around from before we started expiring objects. They needed to be cleaned up.

I started out thinking that I would just go back and set the X-Delete-After header for the object, in this case a log file, and the swift-object-expirer daemon would eventually remove it when it tidied up. It seamed like the right thing to do, but did not work in practice. Try as I might, I could not programmatically set the X-Delete-After header for these old existing objects. Then I found this statement, “As expiring objects are added to the system, the object servers will record the expirations in a hidden .expiring_objects account for the swift-object-expirer to handle later”. I believe the key here is “As expiring objects are added”, meaning when we originally push the log file object to swift. Ah…so apparently a X-Delete-After header cannot be added to an existing object.

Well, fine. Then I’ll just create a tool that removes all the old objects. I decided that the way to do this was to look at the last_modified attribute of an object, compare this with a provided expire-older-than date, and remove anything older using the python swift client delete_object method. It just took a few iterations to get this working properly. I created a test container and filled it with a few random text files to use as an test target. Once I was happy with my new delete tool taking a supplied container and a delete-older-than date and having it remove all objects in my test container, I set it loose on the containers holding our PowerKVM CI logs. This was not without problems, as it would randomly fail after 20 minutes or so. I just caught the exceptions to continue, but then later did some debugging. A quick investigation indicated a possible scenario that when there are a lot of requests coming in, they can be queued. The client would then retry on the delete and fail, because the object was then marked for deletion from the queued request before the retry could complete. I have contemplated opening a defect on this, but looking at other swift client code, it seems to be a common practice to catch these HTTP errors and continue. All in all, no loss, it was a great learning experience for me.

Overall the process result was a success! My new del_objects.py tool cleaned out the old log objects and in the process removed more than 380 GB of old log files. I will be happy to share this and other swift tools that I put together, they aren’t much really, but might be good for an example to use for creating your own super utility. I must get approval from IBM to share them, and that is in progress. I will share them either in my personal github account or maybe on a new Third Party CI Working Group repository created for this, stay tuned.

Third-party CI work group at Kilo Summit

·

At the beginning of this year,  I was being asked to share my team’s experience standing up PowerKVM CI with teams that were just getting started.  Jay Pipes, Anita Kuno and the OpenStack infrastructure team were feeling overwhelmed by all the new third-party CI systems. So, after a brief discussion, we agreed to start this work group after the Atlanta Design Summit. We set out to help improve the third-party testing experience in OpenStack and have now grown into a large group of CI operators.

As a result, we held our very first meeting at the Paris Kilo Design Summit Third-Party CI work group session. The session event page showed 89 signed up and that number was close since I was estimating around 80-100 attending.  It was such a pleasure to get to meet everyone I have been interacting with for the past several months.

Third party CI discussion led by @anteaya and @kurtrtaylor

(Picture by Elizabeth K. Joseph)

The session started with a background overview by Anita. I then led the discussion on the agenda items agreed upon in the weekly Third-party CI meetings. These items are listed in the etherpad for the session, so I won’t repeat them here.

During the session, I asked those attending a few questions and did a quick show of hands and came up with some interesting answers:

  1.  The majority of CI operators attending had contributed at least one patch to OpenStack. I was somewhat surprised with this response. After discussing it and thinking it through, I realized that this made sense. The operators that understand how to participate in an open source community, understand how important it is to come to design summit.
  2.  Almost everyone attending the session knew about the Third-party CI weekly meetings. But, not nearly that many actually attend.
  3. Most Third-party CI systems have some functionality that they did not get from upstream CI, that they had to create for themselves.  Most of those system teams are willing to share what they have created and learned with the third-party community.

Time in the session went way too quickly. We spent a good portion discussing how to bootstrap teams, improve docs and finding alternate meeting times, but then had to rush through improving reliability and better self-regulating. I spent the rest of the week in numerous hall discussions on these topics.

My takeaways are:

  1.  Figure out how to help the operators that didn’t come to summit become contributors to OpenStack community, and to participate in the Third-party CI work group.
  2.  Get other third-party operators to start contributing to the care and feeding of this work group and the weekly meetings.
  3.  Create an additional meeting time for those teams that would like to attend the weekly meeting, but can’t because of time zone differences.
  4.  Help coordinate and participate in a work group focused on improving third-party CI documentation.
  5.  Pull together the efforts to contribute a third-party CI monitoring tool and dashboard (more on that in a later post).

We are now at a place where we as third-party CI systems can organize and help improve the consumability of OpenStack CI and ourselves in the process.  If you were not able to attend the summit session, I encourage you to get involved, attend a weekly meeting if you can, and be on the lookout for an additional meeting time that would be more convenient.

Thanks everyone that attended!