I created this new repo with the following CI operators as initial cores in addition to myself. They were selected and voted in because they have consistently attended the working group meetings, helped fellow operators, and pushed to move work items forward in the community. They are: Ramy Asselin (asselin), Patrick East (patrickeast), Steve Weston (sweston), and Mikhail Medvedev (mmedvede).
My team is using an OpenStack swift object storage service through SoftLayer as a way to publish the required logs to the community from our PowerKVM Continuous Integration testing. The test system ran like this for several months before we determined how to set the expiration on an object to have swift expire and remove a log after 30 days. No big deal really until we started to run out of space on our object store allocation and were being charged overages. This was not major expense, but it wasn’t right to leave all these old logs around from before we started expiring objects. They needed to be cleaned up.
I started out thinking that I would just go back and set the X-Delete-After header for the object, in this case a log file, and the swift-object-expirer daemon would eventually remove it when it tidied up. It seamed like the right thing to do, but did not work in practice. Try as I might, I could not programmatically set the X-Delete-After header for these old existing objects. Then I found this statement, “As expiring objects are added to the system, the object servers will record the expirations in a hidden .expiring_objects account for the swift-object-expirer to handle later”. I believe the key here is “As expiring objects are added”, meaning when we originally push the log file object to swift. Ah…so apparently a X-Delete-After header cannot be added to an existing object.
Well, fine. Then I’ll just create a tool that removes all the old objects. I decided that the way to do this was to look at the last_modified attribute of an object, compare this with a provided expire-older-than date, and remove anything older using the python swift client delete_object method. It just took a few iterations to get this working properly. I created a test container and filled it with a few random text files to use as an test target. Once I was happy with my new delete tool taking a supplied container and a delete-older-than date and having it remove all objects in my test container, I set it loose on the containers holding our PowerKVM CI logs. This was not without problems, as it would randomly fail after 20 minutes or so. I just caught the exceptions to continue, but then later did some debugging. A quick investigation indicated a possible scenario that when there are a lot of requests coming in, they can be queued. The client would then retry on the delete and fail, because the object was then marked for deletion from the queued request before the retry could complete. I have contemplated opening a defect on this, but looking at other swift client code, it seems to be a common practice to catch these HTTP errors and continue. All in all, no loss, it was a great learning experience for me.
Overall the process result was a success! My new del_objects.py tool cleaned out the old log objects and in the process removed more than 380 GB of old log files. I will be happy to share this and other swift tools that I put together, they aren’t much really, but might be good for an example to use for creating your own super utility. I must get approval from IBM to share them, and that is in progress. I will share them either in my personal github account or maybe on a new Third Party CI Working Group repository created for this, stay tuned.
At the beginning of this year, I was being asked to share my team’s experience standing up PowerKVM CI with teams that were just getting started. Jay Pipes, Anita Kuno and the OpenStack infrastructure team were feeling overwhelmed by all the new third-party CI systems. So, after a brief discussion, we agreed to start this work group after the Atlanta Design Summit. We set out to help improve the third-party testing experience in OpenStack and have now grown into a large group of CI operators.
As a result, we held our very first meeting at the Paris Kilo Design Summit Third-Party CI work group session. The session event page showed 89 signed up and that number was close since I was estimating around 80-100 attending. It was such a pleasure to get to meet everyone I have been interacting with for the past several months.
(Picture by Elizabeth K. Joseph)
The session started with a background overview by Anita. I then led the discussion on the agenda items agreed upon in the weekly Third-party CI meetings. These items are listed in the etherpad for the session, so I won’t repeat them here.
During the session, I asked those attending a few questions and did a quick show of hands and came up with some interesting answers:
- The majority of CI operators attending had contributed at least one patch to OpenStack. I was somewhat surprised with this response. After discussing it and thinking it through, I realized that this made sense. The operators that understand how to participate in an open source community, understand how important it is to come to design summit.
- Almost everyone attending the session knew about the Third-party CI weekly meetings. But, not nearly that many actually attend.
- Most Third-party CI systems have some functionality that they did not get from upstream CI, that they had to create for themselves. Most of those system teams are willing to share what they have created and learned with the third-party community.
Time in the session went way too quickly. We spent a good portion discussing how to bootstrap teams, improve docs and finding alternate meeting times, but then had to rush through improving reliability and better self-regulating. I spent the rest of the week in numerous hall discussions on these topics.
My takeaways are:
- Figure out how to help the operators that didn’t come to summit become contributors to OpenStack community, and to participate in the Third-party CI work group.
- Get other third-party operators to start contributing to the care and feeding of this work group and the weekly meetings.
- Create an additional meeting time for those teams that would like to attend the weekly meeting, but can’t because of time zone differences.
- Help coordinate and participate in a work group focused on improving third-party CI documentation.
- Pull together the efforts to contribute a third-party CI monitoring tool and dashboard (more on that in a later post).
We are now at a place where we as third-party CI systems can organize and help improve the consumability of OpenStack CI and ourselves in the process. If you were not able to attend the summit session, I encourage you to get involved, attend a weekly meeting if you can, and be on the lookout for an additional meeting time that would be more convenient.
Thanks everyone that attended!
If you have followed the Linux Sound blog in the past and found your way here, welcome!
I still am very interested in embedded Linux and audio/music/sound support, but my current assignment to OpenStack keeps me very busy. Since I have not updated that blog in some time, and wanted to continue posting about an even broader set of topics, I have moved and merged that mission and my new interests into this one consolidated place.
Stay tuned for more entries on OpenStack, Linux, and of course, Linux Sound. Thanks!
Now there are two. Yes, there are now two different UCM prototypes for Android on Pandaboard. But why? Well, the first, tinyHAL, came out of conversations with Mark Brown and others, as described in my earlier post. The second came from later discovering that Gabriel Beddingfield had implemented a similar PoC for Blaze. They are both good starts and both have their pluses, and now both run on Pandaboard with the Linaro Android builds.
The first, tinyHAL is available at here. The second, android-ucm, is available here. I have also facilitated the relicensing of the dependencies for android-ucm and deposited them here and here. Please refer to the README in each of these repositories for the instructions on how to build these components in a Linaro Android build (edit: Linaro has removed these git repos, contact them directly for this source).
So, which one should be used and enhanced? Well, I like the XML mapping from android-ucm, and I like the XML configuration formats from tinyHAL. I suspect that the way forward should be to decide the best features, and then combine, refactor, and merge efforts. For now, try them both and see which one you prefer. As always, contributions are always welcome and feel free to contact me or email@example.com if you have comments.
Following the ASoC Embedded ALSA Conference and the Linaro Multimedia Summit last year, I put plans in place to bring UCM to Linaro for both the Ubuntu-based builds and the Android builds. We are seeing that happen. UCM functionality is improving in Ubuntu and now is coming to Android. Based on the initial drop of tinyhal by Mark Brown, I now have a prototype running on Pandaboard. This code should be viewed as a proof-of-concept framework for UCM, not a complete fully-functioning implementation. The code supports basic playback and will need to be enhanced for a full dynamic verb set. I have created a git repository for my work here (edit: Linaro has removed this git repo, contact them directly for this source).
To give it a try, you will need to follow the Linaro Android instructions for building from source, which I recently blogged here. Follow the README instructions for copying files to their needed locations. You can test playback with tinyplay or the Android music player app. The configuration file sets the initial volume, but the Android settings work normally for adjusting this level.
The remaining needed work will continue over the next several weeks, adding support for other mixing and routes. I am also looking to unify the file formats between Ubuntu and Android. Stay tuned for that proposal. As always, contributions are welcome.
Yes, it has been too long since I blogged, I won’t bore you with the details of why. Instead, I’ll kick this off again with a step-by-step post on how to set up a system to do your first Android build.
Linaro has a great new Android homepage with lots of useful links on it, go explore the info there http://wiki.linaro.org/Platform/Android
I did my installation on Ubuntu 11.10 Oneiric, so this post will make the most sense if you do too. Put away that P3 box in the corner you were planning on using. It seems to be a wise requirement to use a quick box with x86_64 support. You will also need ~30G of free space on your drive an a good amount of memory, plan for it. So, go do your AMD64 install of Ubuntu Oneiric desktop and then come back.
The AOSP “Initializing a Build Environment” page is your next stop. It is slightly out of date, but work through it http://source.android.com/source/initializing.html Under the “Installing required packages” section, there is a package change for Oneiric. The package lib32readline5-dev is not available, use lib32readline-gplv2-dev instead.
Next, go get repo. To do this, follow the first 3 steps under the Quickstart section at http://wiki.linaro.org/Platform/Android/BuildSource Wait on running the repo init command for now.
Install gcc-4.5. I used software center, and uninstalled 4.6 first. You may not have to do this as it may have caused me some extra fun for me. Regardless, when you have it installed, make sure that you have /usr/bin/cc and /usr/bin/gcc symlinked to gcc-4.5, not 4.6.
I also experienced a problem that was well documented here: http://unforgivendevelopment.com/2011/10/05/solved-cyanogenmod-build-issues-on-debian-wheezy-testing/ As this post suggests, running “sudo ln -s /usr/include/`uname -m`-linux-gnu/asm/ /usr/include/asm” fixed the problem.
Now it is time to decide on the code to build. I was interested in reproducing the 12.01 Linaro Android “tracking” build because it had the patches needed to make audio work on the Pandaboard. The best way to do this is to follow the “How to Get and Build the Source” on the android-build.linaro.org page for the build you are interested in, in my case, https://android-build.linaro.org/builds/~linaro-android/tracking-panda-12.01-release/ If you follow this exactly (with any changes for your environment, of course), you shouldn’t have any problems. Also, I hope you aren’t in a hurry. The repo sync alone will take over an hour unless you have a super fast connection and the build will depend on your host, but mine takes a good long time.
When the build is complete, check the build log in the root directory to make sure there wasn’t any issues. To use the build you just made, refer to the instructions “How to Use Prebuilt Images” on the same Linaro Build page you used in the step above. The boot, system and userdata compressed tarballs are available in the ~/android/out/target/product/<board> directory. Substitute <board> with your dev platform, in my case that was “pandaboard”. Blast these onto a SD card just like you’d do with the prebuilt images and you’ll be up an running on your own Android build.
That’s all there is to it. Hope this helps, I tried to capture everything that I went through to get it working. Please let me know if you find any omissions or problems so that I can edit this post with any corrections.
Since the Linaro multimedia working group didn’t get to plan at the last Linaro Developers Summit, we held our own mini-summit at the beautiful IBM campus in Austin. It has been 3 weeks and I am just now able to think about other things like updating this blog due to the flurry of planning activities that followed. There are other factors at play too, one of them is I have agreed to take on the role of team lead for the multimedia working group. Even though my view now encompasses all multimedia, I will try to keep to this blogs original intent of writing about Linux sound and specifically ARM audio.
The Linaro multimedia working group, often referred to as MMWG, needed to hold the summit to share our plans and discuss the requirements for the upcoming development cycle. It was very well attended by key technical folks from all areas of ARM-based multimedia. Attending were representatives from Linaro, IBM, NVidia, Freescale, Texas Instruments, Collabora, ST-Ericsson, ARM and others as well as several who were able to dial-in. I had organized meet-ups before, but it was the first time I had organized an event this size. All in all, it went very well and I think everyone enjoyed the experience of meeting to discuss the challenges ahead of us.
I’ve been a little busy lately with new assignments and travel. I just returned from two back-to-back conferences, in two different countries, in less than one week.
The first was the ASoC and Embedded Linux Audio Conference in Edinburgh. This was a gathering of about 30-40 multimedia developers from all over the world representing almost every company involved in this space and I was honored to be invited. After introductions, the 2 days worth of in-depth technical discussions got started with a description of the ASoC DSP support, focused on the description and charts shown here -> http://omapedia.org/wiki/Audio_Drive_Arch We then shifted to OS integration topics that mainly centered around Android’s need for a different ALSA user space lib, with more business favorable license terms. Several options were discussed. The UCM talk was next and with an initial brief overview and then enumerating the needs moving forward, including PulseAudio work. Next were discussions on QoS, run-time coefficients, firmware and events. The meeting drew to a close with a summary, questions time and assignments review. I drew the assignment for looking into getting an install script and git set up for the UCM profile config files, something we will need for Linaro. All in all, a great technical conference that was well prepared and useful. Also, it was a fantastic opportunity to meet everyone I have been interacting with in email and IRC, plus many new contacts. I look forward to attending the next one!
The second was the Linux Audio Conference in Dublin, aka, LAC2011. I worked with the planners of this conference to get the PulseAudio folks, including Lennart Poettering, a place to work. The space we were allocated was large enough to allow others to come and listen and ask a few questions. I want to apologize to the people that waited so patiently to ask questions. We had such an active discussion between the developers that they could hardly get a word in edge wise. I learned a lot and hopefully contributed to the overall discussion, direction and schedule. The rest of the conference was more general purpose for all things Linux Audio including: sound generation, sequencing, pro audio, and composition. I have followed this for many years now and it was really interesting to see how it continues to evolve. Even though it was a Linux conference, there were constant references to Apple and iPad integration with the applications used, even a session on running PulseAudio on Mac OS X. I think that mobile Linux has an opportunity to reach these folks.
Even though the trip was jam-packed, it was a perfect technical requirements review for going into the next development cycle for Linaro. They were both worth the hectic travel schedule.