If I look back at the last post I made on ths blog… let’s say quite a lot of time has passed. The reason? Well, first of all, one would call it a lack of motivation1, and afterwards, the emergence of a small yet quite annoying pathogen which caused a bit of a stir worldwide. But today I’m not going to talk about viruses: perhaps some other time when you can avoid hear about it for breakfast, lunch, and dinner.
As you know, since ages, the openSUSE KDE team provides a series of repositories which track the latest state of the git repositories in KDE, be them either Frameworks, Plasma, or the applications part of the Release Service. This also allows to create Live CDs which can be useful for testing out the software.
But the question that I’ve seen every now and then is… how it is actually done? Is everything provided by the OBS, or does someone need to add some glue on top of that?
Using the OBS to provide updates to KDE packages from git
Source Services are tools to validate, generate or modify sources in a trustable way.
Ok, that doesn’t tell us much, does it? Luckily, the concept is simple. It is that, for packages built in the OBS, we can use tools (internal or external) to perform operations on the source(s) the packages are built from. These are done before any actual building starts.
The openSUSE wiki has some examples of what a source service can do, of which one immediately jumps to our eye:
Checkout service - checks out from SVC system (svn, git, …) and creates a tarball.
That means that we can, theoretically, ask the OBS to do a checkout from git, and automatically generate a tarball from there. In addition, a source service can add version information to the RPM spec file - the blueprint of an openSUSE package - including some data taken off git, for example the date and the revision hash of the checkout.
Source services are implemented as files named _service which live in the OBS for each package. Let’s take a look at the one for kcoreaddons in the KDE:Unstable::
The first line inside service tells us that we’re using the obs_scm service, which is a built-in service in openSUSE’s OBS instance to checkout the sources from the url, using git. The versionformat parameter sets the version to a literal VERSION (more on that later) and adds the git suffix, and then adds %ci, which is replaced by the checkout date (example: 20210102T122329) and %h, which puts the short git revision hash in the version (example: 5d069715).
The whole data is compressed in a cpio archive with the obscpio extension. At this point, we don’t have a tarball yet.
After the checkout
THe next services run when the package is actually built, as you can see by the mode="builtime" in their definitions. The first one (set_version_kf5) is actually a service made by Fabian Vogt (fellow member of the KDE team) which replaces VERSION by the current version present in the KDE git (done manually: it is read from the OBS project configuration, and bumped every time it happens upstream). The following lines set the version in the spec file, and compress the whole thing into a tarball.
At this point, the build system starts building the actual source and creating the package.
Just when are these kind of source services run? If left by themselves, never. You need to actually signal the OBS (trigger, in OBS-speak) to run the service. An example is with the osc command line utility:
osc service remoterun KDE:Unstable:Frameworks kcoreaddons
Or there’s a button in the OBS web interface which can allow you to do just that:
There’s just a little problem. When there are more than 200 packages to handle, updating in this way gets a complicated. Also, while the OBS is smart enough to avoid updating a package that has not changed, it has no way to tell you before the service is actually run.
Luckily, the OBS has features which, used with some external tools, can solve the problem in a hopefully effective way.
First of all, I generated a token, and I wanted to make sure that it could only trigger updates. Nothing more, nothing less. This was fairly easy with osc:
osc token --create -o runservice
I did not specify a project, so the token works with all the repositories I have access to. This was a requirement, because the KDE Unstable repositories are actually three different OBS projects.
To trigger an update in the OBS, one needs to call the https://api.opensuse.org/trigger/runservice endpoint, doing a POST request and include the project name (project parameter) and the package name (package parameter). An example (I know, I didn’t encode the URL for simplicity, bear with me):
The token needs to be passed as an Authorization header, in the form Token <your token here>. You get a 200 response if the trigger is successful, and 404 in other cases (including an incorrect token).
Like this, I was able to find a way to reliably trigger the updates. In fact, it would be probably easy to conjure a script to do that. But I wanted to do something more.
A step further
I wanted to actually make sure to trigger the update only if there were any new commits. But at the same time I did not want to have a full checkout of the KDE git repositories: that would have been a massive waste of space.
Enter git ls-remote, which can give you the latest revision of a repository for all its branches (and tags), without having to clone it. For example:
In this case I limited the list to branches to branches (--heads). As you can see, the latest revision in master for kitinerary at the time of writing is f12f2cb73e3229a9a9dafb347a2f5fe9bd6c7975. So, the idea I had in mind was:
Get the state of the master branch in all repositories part of the KDE Unstable hierarchy;
Save the latest revision on disk;
On the following check (24 hours later) compare the revisions between what’s stored on disk and the remote;
If the revisions differ, trigger an update;
Save the new revision to disk;
This was slightly complicated by the fact that package names on the OBS and source repositories in KDE can differ (example: plasma-workspace is plasma5-workspace or akonadi is akonadi-server). My over-engineered idea was to create a JSON mapping of the whole thing (excerpt):
Mmm… cake. Wait. We’re not talking about food here, just about how the whole idea was put into “production” (include several hundred of quotes around that word). I created a separate user on my server (the same which runs dennogumi.org) with minimal privileges, put the token into a file with 0600 permissions, and set up a cron job which runs the script every day at 20 UTC+1.
There was just one extra step. Historically (but this is not the case anymore nowadays) the OBS would fail to perform a git checkout. This would leave a package in the broken state, and the only way to recover would be to force an update again (if that did not fix it, it would be time to poke the friendly people in #opensuse-buildservice).
Inspired by what a former KDE team member (Raymond “tittiatcoke” Wooninck) did, I wrote a stupid script which checks the packages in broken state (at the moment just for one repository and one architecture: a broken clone would affect all of them equally) and forces an update. It needs to use tokens rather than osc, but I’ll get to that soon, hopefully.
There, you have it. This is how (at the moment) I can handle updating all KDE packages from git in the OBS with (now) minimal effort. Probably it is an imperfect solution, but it does the job well. Shout out to Fabian who mentioned tokens and prompted the idea.