My team is using an OpenStack swift object storage service through SoftLayer as a way to publish the required logs to the community from our PowerKVM Continuous Integration testing. The test system ran like this for several months before we determined how to set the expiration on an object to have swift expire and remove a log after 30 days. No big deal really until we started to run out of space on our object store allocation and were being charged overages. This was not major expense, but it wasn’t right to leave all these old logs around from before we started expiring objects. They needed to be cleaned up.
I started out thinking that I would just go back and set the X-Delete-After header for the object, in this case a log file, and the swift-object-expirer daemon would eventually remove it when it tidied up. It seamed like the right thing to do, but did not work in practice. Try as I might, I could not programmatically set the X-Delete-After header for these old existing objects. Then I found this statement, “As expiring objects are added to the system, the object servers will record the expirations in a hidden .expiring_objects account for the swift-object-expirer to handle later”. I believe the key here is “As expiring objects are added”, meaning when we originally push the log file object to swift. Ah…so apparently a X-Delete-After header cannot be added to an existing object.
Well, fine. Then I’ll just create a tool that removes all the old objects. I decided that the way to do this was to look at the last_modified attribute of an object, compare this with a provided expire-older-than date, and remove anything older using the python swift client delete_object method. It just took a few iterations to get this working properly. I created a test container and filled it with a few random text files to use as an test target. Once I was happy with my new delete tool taking a supplied container and a delete-older-than date and having it remove all objects in my test container, I set it loose on the containers holding our PowerKVM CI logs. This was not without problems, as it would randomly fail after 20 minutes or so. I just caught the exceptions to continue, but then later did some debugging. A quick investigation indicated a possible scenario that when there are a lot of requests coming in, they can be queued. The client would then retry on the delete and fail, because the object was then marked for deletion from the queued request before the retry could complete. I have contemplated opening a defect on this, but looking at other swift client code, it seems to be a common practice to catch these HTTP errors and continue. All in all, no loss, it was a great learning experience for me.
Overall the process result was a success! My new del_objects.py tool cleaned out the old log objects and in the process removed more than 380 GB of old log files. I will be happy to share this and other swift tools that I put together, they aren’t much really, but might be good for an example to use for creating your own super utility. I must get approval from IBM to share them, and that is in progress. I will share them either in my personal github account or maybe on a new Third Party CI Working Group repository created for this, stay tuned.