Killing the redundancy with automation

- Comments

In the past three weeks, the openSUSE community KDE team has been pretty busy to package the latest release of Applications from KDE, 16.12. It was a pretty large task, due to the number of programs involved, and the fact that several monolithic projects were split (in particular KDE PIM). This post goes through what we did, and how we improved our packaging workflow.

Some prerequisites

In openSUSE speak, packages are developed in “projects”, which are separate repositories maintained on the OBS. Projects whose packages will end up in the distro, that is where they are being developed to land in the distribution, are called devel projects. The KDE team uses a number of these to package and test:

  • KDE:Qt for currently-released Qt versions
  • KDE:Frameworks5 for Frameworks and Plasma packages
  • KDE:Applications for the KDE Applications packages
  • KDE:Extra for additional software not part of the above categories

The last three have also an Unstable equivalent (KDE:Unstable:XXX) where packages built straight off git master are made, and used in the Argon and Krypton live images.

A new development approach

With the release of Leap 42.2, we also needed a way to keep only Long Term Support packages in a place were could test and adjust fixes for Leap (which, having a frozen base, will not easily accept version upgrades), so we created an additional repository with the LTS suffix to track Plasma 5.8 versions (and KF 5.26, which is what Leap ships).

As you can see, the number of repositories was starting to get large, and we’re still a very low number team, with everyone contributing their spare time to this task. Therefore, a new approach was proposed, prototyped by Hrvoje “shumski” Senjan and spearheaded by Fabian Vogt during the Leap development cycle.

The idea was to use only one repository as an authoritative source of spec files (for those not in the know, spec files are files needed to build RPM packages, and describe the structure of the package, it sources, whether it should be split in sub-packages, and so on), only do changes there and then sync back the changes to the other repositories.

In this case, KDE:Frameworks5 was used as source. All changes were then synced to both the LTS repository and to the Unstable variant with some simple scripting and the use of the osc command, which allows interacting with the OBS from the CLI. This significantly reduced divergences between packages and eased maintenance.

Enter Applications 16.12

When we started packaging Applications, we faced a number of problems that involved the existence of kdelibs 4.x applications that were now obsoleted by KF5 based versions (see Okular, but not only that). Additionally, there were a major number of splits, meaning that we had to track and adjust packaging to keep in mind that what used to be there wasn’t around anymore.

We had already a source that kept track of these changes: the KDE:Unstable:Applications repository, which followed git master. A major problem was that its development had gone in a different direction than the original KDE:Applications, meaning that there was a significant divergence in the two.

Initially we set up a test project and tried to figure out how to lay out a migration path for the existing packages. It didn’t work too well by hand: too many changes, too many packages to keep track of. That is when Raymond Wooninck had the idea to automate the whole process and change the development workflow of Applications packaging.

The new workflow worked as such:

  1. The authoritative source of changes is the Unstable repository, because that’s where the changes end up first, before release
  2. On beta release time, packages would be copied from Unstable to the stable repository, dropping any patches present that were upstreamed
  3. openSUSE specific patches (integration, build system, etc.) would stay in both repositories
  4. upstream patches (patches already committed but not part of a release) would only stay in the stable repository

In order to ensure that this would be done automatically, Raymond created a repository and wrote scripts to do both the Unstable->stable transition and to automate packaging of new minor releases. Once we switched to this workflow, adjustments were much easier, and we were able to finish the job, at least: yesterday (as of writing) the new Applications release was checked in Tumbleweed.

This new workflow requires some discipline (to avoid “ad hoc” solutions) but dramatically reduces the maintenance required, and allows us to track changes in the packages “incrementally” as they happen during a release cycle. At the same time, this guarantees that all the openSUSE packaging policies are followed also in the Unstable project (which was more lax, due to the fact that it would never end up in the distro).

Final icing on the cake

The last bit was to ensure timely updates of all the Unstable project hierarchy avoiding to watch git commits like hawks. I took up the challenges and wrote a script, which, coupled with a repository mapping file would cache the latest “seen” (past 24h) git revision of the KDE repositories, and trigger updates only if something changed (using git ls-remote to avoid hitting the KDE servers too hard).

I put this on a cron job which runs every day at 20 UTC+1, meaning that even the updates are now fully automated. Of course I have to check every now and then for added dependencies and build faiures, but the workload is definitely less than before.

Final wrap up

A handful of tools and some quick thinking can make a massive collection of software manageable by a small bunch of people. At the same time, there’s always need for more helping hands! Should you want to help in openSUSE packaging, drop by in the #opensuse-kde IRC channel on Freenode.

Comments