Posts on swick's blog

How Hard Is It To Open a File?

Sebastian Wick — Thu, 23 Apr 2026 22:41:11 +0200

It’s a question I had to ask myself multiple times over the last few months. Depending on the context the answer can be:

very simple, just call the standard library function
extremely hard, don’t trust anything

If you are an app developer, you’re lucky and it’s almost always the first answer. If you develop something with a security boundary which involves files in any way, the correct answer is very likely the second one.

Opening a File, the Hard Way

Like so often, the details depend on the specifics, but in the worst-case scenario, there is a process on either side of the security boundary, which operate on a filesystem tree which is shared by both processes.

Let’s say that the process with more privileges operates on a file on behalf of the process with less privileges. You might want to restrict this to files in a certain directory, to prevent the less privileged process from, for example, stealing your SSH key, and thus take a subpath that is relative to that directory.

The first obvious problem is that the subpath can refer to files outside of the directory if it contains ... If the privileged process gets called with a subpath of ../.ssh/id_ed25519, you are in trouble. Easy fix: normalize the path, and if we ever go outside of the directory, fail.

The next issue is that every component of the path might be a symlink. If the privileged process gets called with a subpath of link, and link is a symlink to ../.ssh/id_ed25519, you might be in trouble. If the process with less privileges cannot create files in that part of the tree, it cannot create a malicious symlink, and everything is fine. In all other scenarios, nothing is fine. Easy fix: resolve the symlinks, expand the path, then normalize it.

This is usually where most people think we’re done, opening a file is not that hard after all, we can all do more fun things now. Really, this is where the fun begins.

The fix above works, as long as the less privileged process cannot change the file system tree anywhere in the file’s path while the more privileged process tries to access it. Usually this is the case if you unpack an attacker-provided archive into a directory the attacker does not have access to. If it can however, we have a classic TOCTOU (time-of-check to time-of-use) race.

We have the path foo/id_ed25519, we resolve the smlinks, we expand the path, we normalize it, and while we did all of that, the other process just replaced the regular directory foo that we just checked with a symlink which points to ../.ssh. We just checked that the path resolves to a path inside the target directory though, and happily open the path foo/id_ed25519 which now points to your ssh key. Not an easy fix.

So, what is the fundamental issue here? A path string like /home/user/.local/share/flatpak/app/org.example.App/deploy describes a location in a filesystem namespace. It is not a reference to a file. By the time you finish speaking the path aloud, the thing it names may have changed.

The safe primitive is the file descriptor. Once you have an fd pointing at an inode, the kernel pins that inode. The directory can be unlinked, renamed, or replaced with a symlink; the fd does not care. A common misconception is that file descriptors represent open files. It is true that they can do that, but fds opened with O_PATH do not require opening the file, but still provide a stable reference to an inode.

The lesson that should be learned here is that you should not call any privileged process with a path. Period. Passing in file descriptors also has the benefit that they serve as proof that the calling process actually has access to the resource.

Another important lesson is that dropping down from a file descriptor to a path makes everything racy again. For example, let’s say that we want to bind mount something based on a file descriptor, and we only have the traditional mount API, so we convert the fd to a path, and pass that to mount. Unfortunately for the user, the kernel resolves the symlinks in the path that an attacker might have managed to place there. Sometimes it’s possible to detect the issue after the fact, for example by checking that the inode and device of the mounted file and the file descriptor match.

With that being said, sometimes it is not entirely avoidable to use paths, so let’s also look into that as well!

In the scenario above, we have a directory in which we want all the paths to resolve in, and that the attacker does not control. We can thus open it with O_PATH and get a file descriptor for it without the attacker being able to redirect it somewhere else.

With the openat syscall, we can open a path relative to the fd we just opened. It has all the same issues we discussed above, except that we can also pass O_NOFOLLOW. With that flag set, if the last segment of the path is a symlink, it does not follow it and instead opens the actual symlink inode. All the other components can still be symlinks, and they still will be followed. We can however just split up the path, and open the next file descriptor for the next path segment and resolve symlinks manually until we have done so for the entire path.

libglnx chase

libglnx is a utility library for GNOME C projects that provides fd-based filesystem operations as its primary API. Functions like glnx_openat_rdonly, glnx_file_replace_contents_at, and glnx_tmpfile_link_at all take directory fds and operate relative to them. The library is built around the discipline of “always have an fd, never use an absolute path when you can use an fd.”

The most recent addition is glnx_chaseat, which provides safe path traversal, and was inspired by systemd’s chase(), and does precisely what was described above.

int glnx_chaseat (int              dirfd,
                  const char      *path,
                  GlnxChaseFlags   flags,
                  GError         **error);

It returns an O_PATH | O_CLOEXEC fd for the resolved path, or -1 on error. The real magic is in the flags:

typedef enum _GlnxChaseFlags {
  /* Default */
  GLNX_CHASE_DEFAULT = 0,
  /* Disable triggering of automounts */
  GLNX_CHASE_NO_AUTOMOUNT = 1 << 1,
  /* Do not follow the path's right-most component. When the path's right-most
   * component refers to symlink, return O_PATH fd of the symlink. */
  GLNX_CHASE_NOFOLLOW = 1 << 2,
  /* Do not permit the path resolution to succeed if any component of the
   * resolution is not a descendant of the directory indicated by dirfd. */
  GLNX_CHASE_RESOLVE_BENEATH = 1 << 3,
  /* Symlinks are resolved relative to the given dirfd instead of root. */
  GLNX_CHASE_RESOLVE_IN_ROOT = 1 << 4,
  /* Fail if any symlink is encountered. */
  GLNX_CHASE_RESOLVE_NO_SYMLINKS = 1 << 5,
  /* Fail if the path's right-most component is not a regular file */
  GLNX_CHASE_MUST_BE_REGULAR = 1 << 6,
  /* Fail if the path's right-most component is not a directory */
  GLNX_CHASE_MUST_BE_DIRECTORY = 1 << 7,
  /* Fail if the path's right-most component is not a socket */
  GLNX_CHASE_MUST_BE_SOCKET = 1 << 8,
} GlnxChaseFlags;

While it doesn’t sound too complicated to implement, a lot of details are quite hairy. The implementation uses openat2, open_tree and openat depending on what is available and what behavior was requested, it handles auto-mount behavior, ensures that previously visited paths have not changed, and a few other things.

An Aside on Standard Libraries

The POSIX APIs are not great at dealing with the issue. The GLib/Gio APIs (GFile, etc.) are even worse and only accept paths. Granted, they also serve as a cross-platform abstraction where file descriptors are not a universal concept. Unfortunately, Rust also has this cross-platform abstraction which is based entirely on paths.

If you use any of those APIs, you very likely created a vulnerability. The deeper issue is that those path-based APIs are often the standard way to interact with files. This makes it impossible to reason about the security of composed code. You can audit your own code meticulously, open everything with O_PATH | O_NOFOLLOW, chain *at() calls carefully — and then call a third-party library that calls open(path) internally. The security property you established in your code does not compose through that library call.

This means that any system-level code that cares about filesystem security has to audit all transitive dependencies or avoid them in the first place.

So what would a better GLib cross-platform API look like? I would say not too different from chaseat(), but returning opaque handles instead of file descriptors, which on Unix would carry the O_PATH file descriptor and a path that can be used for printing, debugging and things like that. You would open files from those handles, which would yield another kind of opaque handle for reading, writing, and so on.

The current GFile was also designed to implement GVfs: g_file_new_for_uri("smb://server/share/file") gives you a GFile you can g_file_read() just like a local file. This is the right goal, but the wrong abstraction layer. Instead, this kind of access should be provided by FUSE, and the URI should be translated to a path on a specific FUSE mount. This would provide a few benefits:

The fd-chasing approach works everywhere because it is a real filesystem managed by the kernel
The filesystem becomes independent of GLib and can be used for example from Rust as well
It stacks with other FUSE filesystems, such as the XDG Desktop Document Portal used by Flatpak

Wait, Why Are You Talking About This?

Nowadays I maintain a small project called Flatpak. Codean Labs recently did a security analysis on it and found a number of issues. Even though Flatpak developers were aware of the dangers of filesystems, and created libglnx because of it, most of the discovered issues were just about that. One of them (CVE-2026-34078) was a complete sandbox escape.

flatpak run was designed as a command-line tool for trusted users. When you type flatpak run org.example.App, you control the arguments. The code that processes the arguments was written assuming the caller is legitimate. It accepted path strings, because that’s what command-line tools accept.

The Flatpak portal was then built as a D-Bus service that sandboxed apps could call to start subsandboxes — and it did this by effectively constructing a flatpak run invocation and executing it. This connected a component designed for trusted input directly to an untrusted caller (the sandboxed app).

Once that connection exists, every assumption baked into flatpak run about caller trustworthiness becomes a potential vulnerability. The fix wasn’t “change one function” — it was “audit the entire call chain from portal request to bubblewrap execution and replace every path string with an fd.” That’s commits touching the portal, flatpak-run, flatpak_run_app, flatpak_run_setup_base_argv, and the bwrap argument construction, plus new options (--app-fd, --usr-fd, --bind-fd, --ro-bind-fd) threaded through all of them.

If the GLib standard file and path APIs were secure, we would not have had this issue.

Another annoyance here is that the entire subsandboxing approach in Flatpak comes from 15 years ago, when unprivileged user namespaces were not common. Nowadays we could (and should) let apps use kernel-native unprivileged user namespaces to create their own subsandboxes.

Unfortunately with rather large changes comes a high likelihood of something going wrong. For a few days we scrambled to fix a few regressions that prevented Steam, WebKit, and Chromium-based apps from launching. Huge thanks to Simon McVittie!

In the end, we managed to fix everything, made Flatpak more secure, the ecosystem is now better equipped to handle this class of issues, and hopefully you learned something as well.

Three Little Rust Crates

Sebastian Wick — Fri, 27 Mar 2026 02:15:49 +0200

I published three Rust crates:

name-to-handle-at: Safe, low-level Rust bindings for Linux name_to_handle_at and open_by_handle_at system calls
pidfd-util: Safe Rust wrapper for Linux process file descriptors (pidfd)
listen-fds: A Rust library for handling systemd socket activation

They might seem like rather arbitrary, unconnected things – but there is a connection!

systemd socket activation passes file descriptors and a bit of metadata as environment variables to the activated process. If the activated process exec’s another program, the file descriptors get passed along because they are not CLOEXEC. If that process then picks them up, things could go very wrong. So, the activated process is supposed to mark the file descriptors CLOEXEC, and unset the socket activation environment variables. If a process doesn’t do this for whatever reason however, the same problems can arise. So there is another mechanism to help prevent it: another bit of metadata contains the PID of the target. Processes can check it against their own PID to figure out if they were the target of the activation, without having to depend on all other processes doing the right thing.

PIDs however are racy because they wrap around pretty fast, and that’s why nowadays we have pidfds. They are file descriptors which act as a stable handle to a process and avoid the ID wrap-around issue. Socket activation with systemd nowadays also passes a pidfd ID. A pidfd ID however is not the same as a pidfd file descriptor! It is the 64 bit inode of the pidfd file descriptor on the pidfd filesystem. This has the advantage that systemd doesn’t have to install another file descriptor in the target process which might not get closed. It can just put the pidfd ID number into the $LISTEN_PIDFDID environment variable.

Getting the inode of a file descriptor doesn’t sound hard. fstat(2) fills out struct stat which has the st_ino field. The problem is that it has a type of ino_t, which is 32 bits on some systems so we might end up with a process identifier which wraps around pretty fast again.

We can however use the name_to_handle syscall on the pidfd to get a struct file_handle with a f_handle field. The man page helpfully says that “the caller should treat the file_handle structure as an opaque data type”. We’re going to ignore that, though, because at least on the pidfd filesystem, the first 64 bits are the 64 bit inode. With systemd already depending on this and the kernel rule of “don’t break user-space”, this is now API, no matter what the man page tells you.

So there you have it. It’s all connected.

Obviously both pidfds and name_to_handle have more exciting uses, many of which serve my broader goal: making Varlink services a first-class citizen. More about that another time.

Redefining Content Updates in Wayland

Sebastian Wick — Tue, 10 Mar 2026 23:56:11 +0100

The Wayland core protocol has described surface state updates the same way since the beginning: requests modify pending state, commits either apply that state immediately or cache it into the parent for synchronized subsurfaces. Compositors implemented this model faithfully. Then things changed.

Buffer Readiness and Compositor Deviation

The problem emerged from GPU work timing. When a client commits a surface with a buffer, that buffer might still have GPU rendering in progress. If the compositor applies the commit immediately, it would display incomplete content—glitches. If the compositor submits its own GPU work with a dependency on the unfinished client work, it risks missing the deadlines for the next display refresh cycles and even worse stalling in some edge cases.

To get predictable timing, the compositor needs to defer applying commits until the GPU work finishes. This requires tracking readiness constraints on committed state.

Mutter was the first compositor to address this by implementing constraints and dependency tracking of content updates internally. Instead of immediately applying or caching commits, Mutter queued the changes in what we now call content updates, and only applied them when ready. Critically, this was an internal implementation detail. From the client’s perspective, the protocol semantics remained unchanged. Mutter had deviated from the implementation model implied by the specification while maintaining the observable behavior.

New Protocols on Unstable Foundations

When we wanted better frame timing control and a proper FIFO presentation modes on Wayland, we suddenly required explicit queuing of content updates to describe the behavior of the protocols. You can’t implement FIFO and scheduling of content updates without a queue, so both the fifo and commit-timing protocols were designed around the assumption that compositors maintain per-surface queues of content updates.

These protocols were implemented in compositors on top of their internal queue-based architectures, and added to wayland-protocols. But the core protocol specification was never updated. It still described the old “apply or cache into parent state” model that has no notion of content updates, and per-surface queues.

We now had a situation where the core protocol described one model, extension protocols assumed a different model, and compositors implemented something that sort of bridged both.

Implementation and Theory

That situation is not ideal: If the internal implementation follows the design which the core protocol implies, you can’t deal properly with pending client GPU work, and you can’t properly implement the latest timing protocols. To understand and implement the per-surface queue model, you would have to read a whole bunch of discussions, and most likely an implementation such as the one in mutter. The implementations in compositors also evolved organically, making them more complex than they actually have to be. To make matter worse, we also lacked a shared vocabulary for discussing the behavior.

The obvious solution to this is specifying a general model of the per-surface content update queues in the core protocol. Easier said than done though. Coming up with a model that is sufficient to describe the new behavior while also being compatible with the old behavior when no constraints on content updates defer their application was harder than I expected.

Together with Julian Orth, we managed to change the Wayland core protocol, and I wrote documentation about the system.

Recently Pekka Paalanen and Julian Orth reviewed the work, which allowed it to land. The updated and improved Wayland book should get deployed soon, as well.

The end result is that if you ever have to write a Wayland compositor, one of the trickier parts to get right should now be almost trivial. Implement the rules as specified, and things should just work. Edge cases are handled by the general rules rather than requiring special knowledge.

Best Practices for Ownership in GLib

Sebastian Wick — Wed, 21 Jan 2026 16:31:23 +0100

For all the rightful criticisms that C gets, GLib does manage to alleviate at least some of it. If we can’t use a better language, we should at least make use of all the tools we have in C with GLib.

This post looks at the topic of ownership, and also how it applies to libdex fibers.

Ownership

In normal C usage, it is often not obvious at all if an object that gets returned from a function (either as a real return value or as an out-parameter) is owned by the caller or the callee:

MyThing *thing = my_thing_new ();

If thing is owned by the caller, then the caller also has to release the object thing. If it is owned by the callee, then the lifetime of the object thing has to be checked against its usage.

At this point, the documentation is usually being consulted with the hope that the developer of my_thing_new documented it somehow. With gobject-introspection, this documentation is standardized and you can usually read one of these:

The caller of the function takes ownership of the data, and is responsible for freeing it.

The returned data is owned by the instance.

If thing is owned by the caller, the caller now has to release the object or transfer ownership to another place. In normal C usage, both of those are hard issues. For releasing the object, one of two techniques are usually employed:

single exit

MyThing *thing = my_thing_new ();
gboolean c;
c = my_thing_a (thing);
if (c)
  c = my_thing_b (thing);
if (c)
  my_thing_c (thing);
my_thing_release (thing); /* release thing */

goto cleanup

  MyThing *thing = my_thing_new ();
  if (!my_thing_a (thing))
    goto out;
  if (!my_thing_b (thing))
    goto out;
  my_thing_c (thing);
out:
  my_thing_release (thing); /* release thing */

Ownership Transfer

GLib provides automatic cleanup helpers (g_auto, g_autoptr, g_autofd, g_autolist). A macro associates the function to release the object with the type of the object (e.g. G_DEFINE_AUTOPTR_CLEANUP_FUNC). If they are being used, the single exit and goto cleanup approaches become unnecessary:

g_autoptr(MyThing) thing = my_thing_new ();
if (!my_thing_a (thing))
  return;
if (!my_thing_b (thing))
  return;
my_thing_c (thing);

The nice side effect of using automatic cleanup is that for a reader of the code, the g_auto helpers become a definite mark that the variable they are applied on own the object!

If we have a function which takes ownership over an object passed in (i.e. the called function will eventually release the resource itself) then in normal C usage this is indistinguishable from a function call which does not take ownership:

MyThing *thing = my_thing_new ();
my_thing_finish_thing (thing);

If my_thing_finish_thing takes ownership, then the code is correct, otherwise it leaks the object thing.

On the other hand, if automatic cleanup is used, there is only one correct way to handle either case.

A function call which does not take ownership is just a normal function call and the variable thing is not modified, so it keeps ownership:

g_autoptr(MyThing) thing = my_thing_new ();
my_thing_finish_thing (thing);

A function call which takes ownership on the other hand has to unset the variable thing to remove ownership from the variable and ensure the cleanup function is not called. This is done by “stealing” the object from the variable:

g_autoptr(MyThing) thing = my_thing_new ();
my_thing_finish_thing (g_steal_pointer (&thing));

By using g_steal_pointer and friends, the ownership transfer becomes obvious in the code, just like ownership of an object by a variable becomes obvious with g_autoptr.

Ownership Annotations

Now you could argue that the g_autoptr and g_steal_pointer combination without any conditional early exit is functionally exactly the same as the example with the normal C usage, and you would be right. We also need more code and it adds a tiny bit of runtime overhead.

I would still argue that it helps readers of the code immensely which makes it an acceptable trade-off in almost all situations. As long as you haven’t profiled and determined the overhead to be problematic, you should always use g_auto and g_steal!

The way I like to look at g_auto and g_steal is that it is not only a mechanism to release objects and unset variables, but also annotations about the ownership and ownership transfers.

Scoping

One pattern that is still somewhat pronounced in older code using GLib, is the declaration of all variables at the top of a function:

static void
foobar (void)
{
  MyThing *thing = NULL;
  size_t i;

  for (i = 0; i < len; i++) {
    g_clear_pointer (&thing);
    thing = my_thing_new (i);
    my_thing_bar (thing);
  }
}

We can still avoid mixing declarations and code, but we don’t have to do it at the granularity of a function, but of natural scopes:

static void
foobar (void)
{
  for (size_t i = 0; i < len; i++) {
    g_autoptr(MyThing) thing = NULL;

    thing = my_thing_new (i);
    my_thing_bar (thing);
  }
}

Similarly, we can introduce our own scopes which can be used to limit how long variables, and thus objects are alive:

static void
foobar (void)
{
  g_autoptr(MyOtherThing) other = NULL;

  {
    /* we only need `thing` to get `other` */
    g_autoptr(MyThing) thing = NULL;

    thing = my_thing_new ();
    other = my_thing_bar (thing);
  }

  my_other_thing_bar (other);
}

Fibers

When somewhat complex asynchronous patterns are required in a piece of GLib software, it becomes extremely advantageous to use libdex and the system of fibers it provides. They allow writing what looks like synchronous code, which suspends on await points:

g_autoptr(MyThing) thing = NULL;

thing = dex_await_object (my_thing_new_future (), NULL);

If this piece of code doesn’t make much sense to you, I suggest reading the libdex Additional Documentation.

Unfortunately the await points can also be a bit of a pitfall: the call to dex_await is semantically like calling g_main_loop_run on the thread default main context. If you use an object which is not owned across an await point, the lifetime of that object becomes critical. Often the lifetime is bound to another object which you might not control in that particular function. In that case, the pointer can point to an already released object when dex_await returns:

static DexFuture *
foobar (gpointer user_data)
{
  /* foo is owned by the context, so we do not use an autoptr */
  MyFoo *foo = context_get_foo ();
  g_autoptr(MyOtherThing) other = NULL;
  g_autoptr(MyThing) thing = NULL;

  thing = my_thing_new ();
  /* side effect of running g_main_loop_run */
  other = dex_await_object (my_thing_bar (thing, foo), NULL);
  if (!other)
    return dex_future_new_false ();

  /* foo here is not owned, and depending on the lifetime
   * (context might recreate foo in some circumstances),
   * foo might point to an already released object
   */
  dex_await (my_other_thing_foo_bar (other, foo), NULL);
  return dex_future_new_true ();
}

If we assume that context_get_foo returns a different object when the main loop runs, the code above will not work.

The fix is simple: own the objects that are being used across await points, or re-acquire an object. The correct choice depends on what semantic is required.

We can also combine this with improved scoping to only keep the objects alive for as long as required. Unnecessarily keeping objects alive across await points can keep resource usage high and might have unintended consequences.

static DexFuture *
foobar (gpointer user_data)
{
  /* we now own foo */
  g_autoptr(MyFoo) foo = g_object_ref (context_get_foo ());
  g_autoptr(MyOtherThing) other = NULL;

  {
    g_autoptr(MyThing) thing = NULL;

    thing = my_thing_new ();
    /* side effect of running g_main_loop_run */
    other = dex_await_object (my_thing_bar (thing, foo), NULL);
    if (!other)
      return dex_future_new_false ();
  }

  /* we own foo, so this always points to a valid object */
  dex_await (my_other_thing_bar (other, foo), NULL);
  return dex_future_new_true ();
}

static DexFuture *
foobar (gpointer user_data)
{
  /* we now own foo */
  g_autoptr(MyOtherThing) other = NULL;

  {
    /* We do not own foo, but we only use it before an
     * await point.
     * The scope ensures it is not being used afterwards.
     */
    MyFoo *foo = context_get_foo ();
    g_autoptr(MyThing) thing = NULL;

    thing = my_thing_new ();
    /* side effect of running g_main_loop_run */
    other = dex_await_object (my_thing_bar (thing, foo), NULL);
    if (!other)
      return dex_future_new_false ();
  }

  {
    MyFoo *foo = context_get_foo ();

    dex_await (my_other_thing_bar (other, foo), NULL);
  }

  return dex_future_new_true ();
}

One of the scenarios where re-acquiring an object is necessary, are worker fibers which operate continuously, until the object gets disposed. Now, if this fiber owns the object (i.e. holds a reference to the object), it will never get disposed because the fiber would only finish when the reference it holds gets released, which doesn’t happen because it holds a reference. The naive code also suspiciously doesn’t have any exit condition.

static DexFuture *
foobar (gpointer user_data)
{
  g_autoptr(MyThing) self = g_object_ref (MY_THING (user_data));

  for (;;)
    {
      g_autoptr(GBytes) bytes = NULL;

      bytes = dex_await_boxed (my_other_thing_bar (other, foo), NULL);

      my_thing_write_bytes (self, bytes);
    }
}

So instead of owning the object, we need a way to re-acquire it. A weak-ref is perfect for this.

static DexFuture *
foobar (gpointer user_data)
{
  /* g_weak_ref_init in the caller somewhere */
  GWeakRef *self_wr = user_data;

  for (;;)
    {
      g_autoptr(GBytes) bytes = NULL;

      bytes = dex_await_boxed (my_other_thing_bar (other, foo), NULL);

      {
        g_autoptr(MyThing) self = g_weak_ref_get (&self_wr);
        if (!self)
          return dex_future_new_true ();

        my_thing_write_bytes (self, bytes);
      }
    }
}

Conclusion

Always use g_auto/g_steal helpers to mark ownership and ownership transfers (exceptions do apply)
Use scopes to limit the lifetime of objects
In fibers, always own objects you need across await points, or re-acquire them

Improving the Flatpak Graphics Drivers Situation

Sebastian Wick — Tue, 06 Jan 2026 00:30:46 +0100

Graphics drivers in Flatpak have been a bit of a pain point. The drivers have to be built against the runtime to work in the runtime. This usually isn’t much of an issue but it breaks down in two cases:

If the driver depends on a specific kernel version
If the runtime is end-of-life (EOL)

The first issue is what the proprietary Nvidia drivers exhibit. A specific user space driver requires a specific kernel driver. For drivers in Mesa, this isn’t an issue. In the medium term, we might get lucky here and the Mesa-provided Nova driver might become competitive with the proprietary driver. Not all hardware will be supported though, and some people might need CUDA or other proprietary features, so this problem likely won’t go away completely.

Currently we have runtime extensions for every Nvidia driver version which gets matched up with the kernel version, but this isn’t great.

The second issue is even worse, because we don’t even have a somewhat working solution to it. A runtime which is EOL doesn’t receive updates, and neither does the runtime extension providing GL and Vulkan drivers. New GPU hardware just won’t be supported and the software rendering fallback will kick in.

How we deal with this is rather primitive: keep updating apps, don’t depend on EOL runtimes. This is in general a good strategy. A EOL runtime also doesn’t receive security updates, so users should not use them. Users will be users though and if they have a goal which involves running an app which uses an EOL runtime, that’s what they will do. From a software archival perspective, it is also desirable to keep things working, even if they should be strongly discouraged.

In all those cases, the user most likely still has a working graphics driver, just not in the flatpak runtime, but on the host system. So one naturally asks oneself: why not just use that driver?

That’s a load-bearing “just”. Let’s explore our options.

Exploration

Attempt #1: Bind mount the drivers into the runtime.

Cool, we got the driver’s shared libraries and ICDs from the host in the runtime. If we run a program, it might work. It might also not work. The shared libraries have dependencies and because we are in a completely different runtime than the host, they most likely will be mismatched. Yikes.

Attempt #2: Bind mount the dependencies.

We got all the dependencies of the driver in the runtime. They are satisfied and the driver will work. But your app most likely won’t. It has dependencies that we just changed under its nose. Yikes.

Attempt #3: Linker magic.

Until here everything is pretty obvious, but it turns out that linkers are actually quite capable and support what’s called linker namespaces. In a single process one can load two completely different sets of shared libraries which will not interfere with each other. We can bind mount the host shared libraries into the runtime, and dlmopen the driver into its own namespace. This is exactly what libcapsule does. It does have some issues though, one being that the libc can’t be loaded into multiple linker namespaces because it manages global resources. We can use the runtime’s libc, but the host driver might require a newer libc. We can use the host libc, but now we contaminate the apps linker namespace with a dependency from the host.

Attempt #4: Virtualization.

All of the previous attempts try to load the host shared objects into the app. Besides the issues mentioned above, this has a few more fundamental issues:

The Flatpak runtimes support i386 apps; those would require a i386 driver on the host, but modern systems only ship amd64 code.
We might want to support emulation of other architectures later
It leaks an awful lot of the host system into the sandbox
It breaks the strict separation of the host system and the runtime

If we avoid getting code from the host into the runtime, all of those issues just go away, and GPU virtualization via Virtio-GPU with Venus allows us to do exactly that.

The VM uses the Venus driver to record and serialize the Vulkan commands, sends them to the hypervisor via the virtio-gpu kernel driver. The host uses virglrenderer to deserializes and executes the commands.

This makes sense for VMs, but we don’t have a VM, and we might not have the virtio-gpu kernel module, and we might not be able to load it without privileges. Not great.

It turns out however that the developers of virglrenderer also don’t want to have to run a VM to run and test their project and thus added vtest, which uses a unix socket to transport the commands from the mesa Venus driver to virglrenderer.

It also turns out that I’m not the first one who noticed this, and there is some glue code which allows Podman to make use of virgl.

You can most likely test this approach right now on your system by running two commands:

rendernodes=(/dev/dri/render*)
virgl_test_server --venus --use-gles --socket-path /tmp/flatpak-virgl.sock --rendernode "${rendernodes[0]}" &
flatpak run --nodevice=dri --filesystem=/tmp/flatpak-virgl.sock --env=VN_DEBUG=vtest --env=VTEST_SOCKET_NAME=/tmp/flatpak-virgl.sock org.gnome.clocks

If we integrate this well, the existing driver selection will ensure that this virtualization path is only used if there isn’t a suitable driver in the runtime.

Implementation

Obviously the commands above are a hack. Flatpak should automatically do all of this, based on the availability of the dri permission.

We actually already start a host program and stop it when the app exits: xdg-dbus-proxy. It’s a bit involved because we have to wait for the program (in our case virgl_test_server) to provide the service before starting the app. We also have to shut it down when the app exits, but flatpak is not a supervisor. You won’t see it in the output of ps because it just execs bubblewrap (bwrap) and ceases to exist before the app even started. So instead we have to use the kernel’s automatic cleanup of kernel resources to signal to virgl_test_server that it is time to shut down.

The way this is usually done is via a so called sync fd. If you have a pipe and poll the file descriptor of one end, it becomes readable as soon as the other end writes to it, or the file description is closed. Bubblewrap supports this kind of sync fd: you can hand in a one end of a pipe and it ensures the kernel will close the fd once the app exits.

One small problem: only one of those sync fds is supported in bwrap at the moment, but we can add support for multiple in Bubblewrap and Flatpak.

For waiting for the service to start, we can reuse the same pipe, but write to the other end in the service, and wait for the fd to become readable in Flatpak, before exec’ing bwrap with the same fd. Also not too much code.

Finally, virglrenderer needs to learn how to use a sync fd. Also pretty trivial. There is an older MR which adds something similar for the Podman hook, but it misses the code which allows Flatpak to wait for the service to come up, and it never got merged.

Overall, this is pretty straight forward.

Conclusion

The virtualization approach should be a robust fallback for all the cases where we don’t get a working GPU driver in the Flatpak runtime, but there are a bunch of issues and unknowns as well.

It is not entirely clear how forwards and backwards compatible vtest is, if it even is supposed to be used in production, and if it provides a strong security boundary.

None of that is a fundamental issue though and we could work out those issues.

It’s also not optimal to start virgl_test_server for every Flatpak app instance.

Given that we’re trying to move away from blanket dri access to a more granular and dynamic access to GPU hardware via a new daemon, it might make sense to use this new daemon to start the virgl_test_server on demand and only for allowed devices.

Flatpak Pre-Installation Approaches

Sebastian Wick — Sat, 13 Dec 2025 18:17:48 +0100

Together with my then-colleague Kalev Lember, I recently added support for pre-installing Flatpak applications. It sounds fancy, but it is conceptually very simple: Flatpak reads configuration files from several directories to determine which applications should be pre-installed. It then installs any missing applications and removes any that are no longer supposed to be pre-installed (with some small caveats).

For example, the following configuration tells Flatpak that the devel branch of the app org.test.Foo from remotes which serve the collection org.test.Collection, and the app org.test.Bar from any remote should be installed:

[Flatpak Preinstall org.test.Foo]
CollectionID=org.test.Collection
Branch=devel

[Flatpak Preinstall org.test.Bar]

By dropping in another confiuration file with a higher priority, pre-installation of the app org.test.Foo can be disabled:

[Flatpak Preinstall org.test.Foo]
Install=false

The installation procedure is the same as it is for the flatpak-install command. It supports installing from remotes and from side-load repositories, which is to say from a repository on a filesystem.

This simplicity also means that system integrators are responsible for assembling all the parts into a functioning system, and that there are a number of choices that need to be made for installation and upgrades.

The simplest way to approach this is to just ship a bunch of config files in /usr/share/flatpak/preinstall.d and config files for the remotes from which the apps are available. In the installation procedure, flatpak-preinstall is called and it will download the Flatpaks from the remotes over the network into /var/lib/flatpak. This works just fine, until someone needs one of those apps but doesn’t have a suitable network connection.

The next way one could approach this is exactly the same way, but with a sideload repository on the installation medium which contains the apps that will get pre-installed. The flatpak-preinstall command needs to be pointed at this repository at install time, and the process which creates the installation medium needs to be adjusted to create this repository. The installation process now works without a network connection. System updates are usually downloaded over the network, just as new pre-installed applications will be.

It is also possible to simply skip flatpak-preinstall, and use flatpak-install to create a Flatpak installation containing the pre-installed apps which get shipped on the installation medium. This installation can then be copied over from the installation medium to /var/lib/flatpak in the installation process. It unfortunately also makes the installation process less flexible because it becomes impossible to dynamically build the configuration.

On modern, image-based operating systems, it might be tempting to just ship this Flatpak installation on the image because the flexibility is usually neither required nor wanted. This currently does not work for the simple reason that the default system installation is in /var/lib/flatpak, which is not in /usr which is the mount point of the image. If the default system installation was in the image, then it would be read-only because the image is read-only. This means we could not update or install anything new to the system installation. If we make it possible to have two different system installations — one in the image, and one in /var — then we could update and install new things, but the installation on the image would become useless over time because all the runtimes and apps will be in /var anyway as they get updated.

All of those issues mean that even for image-based operating systems, pre-installation via a sideload repository is not a bad idea for now. It is however also not perfect. The kind of “pure” installation medium which is simply an image now contains a sideload repository. It also means that a factory reset functionality is not possible because the image does not contain the pre-installed apps.

In the future, we will need to revisit these approaches to find a solution that works seamlessly with image-based operating systems and supports factory reset functionality. Until then, we can use the systems mentioned above to start rolling out pre-installed Flatpaks.

Flatpak Happenings

Sebastian Wick — Tue, 04 Nov 2025 21:28:26 +0100

Yesterday I released Flatpak 1.17.0. It is the first version of the unstable 1.17 series and the first release in 6 months. There are a few things which didn’t make it for this release, which is why I’m planning to do another unstable release rather soon, and then a stable release still this year.

Back at LAS this year I talked about the Future of Flatpak and I started with the grim situation the project found itself in: Flatpak was stagnant, the maintainers left the project and PRs didn’t get reviewed.

Some good news: things are a bit better now. I have taken over maintenance, Alex Larsson and Owen Taylor managed to set aside enough time to make this happen and Boudhayan Bhattcharya (bbhtt) and Adrian Vovk also got more involved. The backlog has been reduced considerably and new PRs get reviewed in a reasonable time frame.

I also listed a number of improvements that we had planned, and we made progress on most of them:

It is now possible to define which Flatpak apps shall be pre-installed on a system, and Flatpak will automatically install and uninstall things accordingly. Our friends at Aurora and Bluefin already use this to ship core apps from Flathub on their bootc based systems (shout-out to Jorge Castro).
The OCI support in Flatpak has been enhanced to support pre-installing from OCI images and remotes, which will be used in RHEL 10
We merged the backwards-compatible permission system. This allows apps to use new, more restricting permissions, while not breaking compatibility when the app runs on older systems. Specifically access to input devices such as gamepads, and access to the USB portal can now be granted in this way. It will also help us to transition to PipeWire.
We have up-to-date docs for libflatpak again

Besides the changes directly in Flatpak, there are a lot of other things happening around the wider ecosystem:

bbhtt released a new version of flatpak-builder
Enhanced License Compliance Tools for Flathub
Adrian and I have made plans for a service which allows querying running app instances (systemd-appd). This provides a new way of authenticating Flatpak instances and is a prerequisite for nested sandboxing, PipeWire support, and getting rid of the D-Bus proxy. My previous blog post went into a few more details.
Our friends at KDE have started looking into the XDG Intents spec, which will hopefully allow us to implement deep-linking, thumbnailing in Flatpak apps, and other interesting features
Adrian made progress on the session save/restore Portal
Some rather big refactoring work in the Portals frontend, and GDBus and libdex integration work which will reduce the complexity of asynchronous D-Bus

What I have also talked about at my LAS talk is the idea of a Flatpak-Next project. People got excited about this, but I feel like I have to make something very clear:

If we redid Flatpak now, it would not be significantly better than the current Flatpak! You could still not do nested sandboxing, you would still need a D-Bus proxy, you would still have a complex permission system, and so on.

Those problems require work outside of Flatpak, but have to integrate with Flatpak and Flatpak-Next in the future. Some of the things we will be doing include:

Work on the systemd-appd concept
Make varlink a feasible alternative to D-Bus
D-Bus filtering in the D-Bus daemons
Network sandboxing via pasta
PipeWire policy for sandboxes
New Portals

So if you’re excited about Flatpak-Next, help us to improve the Flatpak ecosystem and make Flatpak-Next more feasible!

SO_PEERPIDFD Gets More Useful

Sebastian Wick — Fri, 10 Oct 2025 17:04:34 +0200

A while ago I wrote about the limited usefulness of SO_PEERPIDFD. for authenticating sandboxed applications. The core problem was simple: while pidfds gave us a race-free way to identify a process, we still had no standardized way to figure out what that process actually was - which sandbox it ran in, what application it represented, or what permissions it should have.

The situation has improved considerably since then.

cgroup xattrs

Cgroups now support user extended attributes. This feature allows arbitrary metadata to be attached to cgroup inodes using standard xattr calls.

We can change flatpak (or snap, or any other container engine) to create a cgroup for application instances it launches, and attach metadata to it using xattrs. This metadata can include the sandboxing engine, application ID, instance ID, and any other information the compositor or D-Bus service might need.

Every process belongs to a cgroup, and you can query which cgroup a process belongs to through its pidfd - completely race-free.

Standardized Authentication

Remember the complexity from the original post? Services had to implement different lookup mechanisms for different sandbox technologies:

For flatpak: look in /proc/$PID/root/.flatpak-info
For snap: shell out to snap routine portal-info
For firejail: no solution
…

All of this goes away. Now there’s a single path:

Accept a connection on a socket
Use SO_PEERPIDFD to get a pidfd for the client
Query the client’s cgroup using the pidfd
Read the cgroup’s user xattrs to get the sandbox metadata

This works the same way regardless of which sandbox engine launched the application.

A Kernel Feature, Not a systemd One

It’s worth emphasizing: cgroups are a Linux kernel feature. They have no dependency on systemd or any other userspace component. Any process can manage cgroups and attach xattrs to them. The process only needs appropriate permissions and is restricted to a subtree determined by the cgroup namespace it is in. This makes the approach universally applicable across different init systems and distributions.

To support non-Linux systems, we might even be able to abstract away the cgroup details, by providing a varlink service to register and query running applications. On Linux, this service would use cgroups and xattrs internally.

Replacing Socket-Per-App

The old approach - creating dedicated wayland, D-Bus, etc. sockets for each app instance and attaching metadata to the service which gets mapped to connections on that socket - can now be retired. The pidfd + cgroup xattr approach is simpler: one standardized lookup path instead of mounting special sockets. It works everywhere: any service can authenticate any client without special socket setup. And it’s more flexible: metadata can be updated after process creation if needed.

For compositor and D-Bus service developers, this means you can finally implement proper sandboxed client authentication without needing to understand the internals of every container engine. For sandbox developers, it means you have a standardized way to communicate application identity without implementing custom socket mounting schemes.

XDG Intents Updates

Sebastian Wick — Wed, 24 Sep 2025 18:57:45 +0200

Andy Holmes wrote an excellent overview of XDG Intents in his “Best Intentions” blog post, covering the foundational concepts and early proposals. Unfortunately, due to GNOME Foundation issues, this work never fully materialized. As I have been running into more and more cases where this would provide a useful primitive for other features, I tried to continue the work.

The specifications have evolved as I worked on implementing them in glib, desktop-file-utils and ptyxis. Here’s what’s changed:

Intent-Apps Specification

Andy showed this hypothetical syntax for scoped preferences:

[Default Applications]
org.freedesktop.Thumbnailer=org.gimp.GIMP
org.freedesktop.Thumbnailer[image/svg+xml]=org.gnome.Loupe;org.gimp.GIMP

We now use separate groups instead:

[Default Applications]
org.freedesktop.Thumbnailer=org.gimp.GIMP

[org.freedesktop.Thumbnailer]
image/svg+xml=org.gnome.Loupe;org.gimp.GIMP

This approach creates a dedicated group for each intent, with keys representing the scopes. This way, we do not have to abuse the square brackets which were meant for translatable keys and allow only very limited values.

The updated specification also adds support for intent.cache files to improve performance, containing up-to-date lists of applications supporting particular intents and scopes. This is very similar to the already existing cache for MIME types. The update-desktop-database tool is responsible for keeping the cache up-to-date.

This is implemented in glib!4797, desktop-file-utils!27, and the updated specification is in xdg-specs!106.

Terminal Intent Specification

While Andy mentioned the terminal intent as a use case, Zander Brown tried to upstream the intent in xdg-specs!46 multiple years ago. However, because it depended on the intent-apps specification, it unfortunately never went anywhere. With the fleshed-out version of the intent-apps specification, and an implementation in glib, I was able to implement the terminal-intent specification in glib as well. With some help from Christian, we also added support for the intent in the ptyxis terminal.

This revealed some shortcomings in the proposed D-Bus interface. In particular, when a desktop file gets activated with multiple URIs, and the Exec line in the desktop entry only indicates support for a limited number of URIs, multiple commands need to be launched. To support opening those commands in a single window but in multiple tabs in the terminal emulator, for example, those multiple commands must be part of a single D-Bus method call. The resulting D-Bus interface looks like this:

<interface name="org.freedesktop.Terminal1">
  <method name="LaunchCommand">
    <arg type='aa{sv}' name='commands' direction='in' />
    <arg type='ay' name='desktop_entry' direction='in' />
    <arg type='a{sv}' name='options' direction='in' />
    <arg type='a{sv}' name='platform_data' direction='in' />
  </method>
</interface>

This is implemented in glib!4797, ptyxis!119 and the updated specification is in xdg-specs!107.

Deeplink Intent

Andy’s post discussed a generic “org.freedesktop.UriHandler” with this example:

[org.freedesktop.UriHandler]
Supports=wise.com;
Patterns=https://*.wise.com/link?urn=urn%3Awise%3Atransfers;

The updated specification introduces a specific org.freedesktop.handler.Deeplink1 intent where the scheme is implicitly http or https and the host comes from the scope (i.e., the Supports part). The pattern matching is done on the path alone:

[org.freedesktop.handler.Deeplink1]
Supports=example.org;extensions.gnome.org
example.org=/login;/test/a?a
extensions.gnome.org=/extension/*/*/install;/extension/*/*/uninstall

This allows us to focus on deeplinking alone and allows the user to set the order of handlers for specific hosts.

In this example, the app would handle the URIs http://example.org/login, http://example.org/test/aba, http://extensions.gnome.org/extension/123456/BestExtension/install and so on.

There is a draft implementation in glib!4833 and the specification is in xdg-specs!109.

Deeplinking Issues and Other Handlers

I am still unsure about the Deeplink1 intent. Does it make sense to allow schemes other than http and https? If yes, how should the priority of applications be determined when opening a URI? How complex does the pattern matching need to be?

Similarly, should we add an org.freedesktop.handler.Scheme1 intent? We currently abuse MIME handlers for this, so it seems like a good idea, but then we need to take backwards compatibility into account. Maybe we can modify update-desktop-database to add entries from org.freedesktop.handler.Scheme1 to mimeapps.list for that?

If we go down that route, is there a reason not to also do the same for MIME handlers and add an org.freedesktop.handler.Mime1 intent for that purpose with the same backwards compatibility mechanism?

Deeplinking to App Locations

While working on this, I noticed that we are not great at allowing linking to locations in our apps. For example, most email clients do not have a way to link to a specific email. Most calendars do not allow referencing a specific event. Some apps do support this. For example, Zotero allows linking to items in the app with URIs of the form zotero://select/items/0_USN95MJC.

Maybe we can improve on this? If all our apps used a consistent scheme and queries (for example xdg-app-org.example.appid:/some/path/in/the/app?name=Example), we could render those links differently and finally have a nice way to link to an email in our calendar.

This definitely needs more thought, but I do like the idea.

Security Considerations

Allowing apps to describe more thoroughly which URIs they can handle is great, but we also live in a world where security has to be taken into account. If an app wants to handle the URI https://bank.example.org, we better be sure that this app actually is the correct banking app. This unfortunately is not a trivial issue, so I will leave it for the next time.

Integrating libdex with GDBus

Sebastian Wick — Thu, 18 Sep 2025 20:58:43 +0200

Writing asynchronous code in C has always been a challenge. Traditional callback-based approaches, including GLib’s async/finish pattern, often lead to the so-called callback hell that’s difficult to read and maintain. The libdex library offers a solution to this problem, and I recently worked on expanding the integration with GLib’s GDBus subsystem.

The Problem with the Sync and Async Patterns

Writing C code involving tasks which can take non-trivial amount of time has traditionally required choosing between two approaches:

Synchronous calls - Simple to write but block the current thread
Asynchronous callbacks - Non-blocking but result in callback hell and complex error handling

Often the synchronous variant is chosen to keep the code simple, but in a lot of cases, blocking for potentially multiple seconds is not acceptable. Threads can be used to prevent the other threads from blocking, but it creates parallelism and with it the need for locking. It also can potentially create a huge amount of threads which mostly sit idle.

The asynchronous variant has none of those problems, but consider a typical async D-Bus operation in traditional GLib code:

static void
on_ping_ready (GObject      *source_object,
               GAsyncResult *res,
               gpointer      data)
{
  g_autofree char *pong = NULL;

  if (!dex_dbus_ping_pong_call_ping_finish (DEX_BUS_PING_PONG (source_object),
                                            &pong,
                                            res, NULL))
    return; // handle error

  g_print ("client: %s\n", pong);
}

static void
on_ping_pong_proxy_ready (GObject      *source_object,
                          GAsyncResult *res,
                          gpointer      data)
{
  DexDbusPingPong *pp dex_dbus_ping_pong_proxy_new_finish (res, NULL);
  if (!pp)
    return; // Handle error

  dex_dbus_ping_pong_call_ping (pp, "ping", NULL,
                                on_ping_ready, NULL);
}

This pattern becomes unwieldy quickly, especially with multiple operations, error handling, shared data and cleanup across multiple callbacks.

What is libdex?

Dex provides Future-based programming for GLib. It provides features for application and library authors who want to structure concurrent code in an easy to manage way. Dex also provides Fibers which allow writing synchronous looking code in C while maintaining the benefits of asynchronous execution.

At its core, libdex introduces two key concepts:

Futures: Represent values that will be available at some point in the future
Fibers: Lightweight cooperative threads that allow writing synchronous-looking code that yields control when waiting for asynchronous operations

Futures alone already simplify dealing with asynchronous code by specefying a call chain (dex_future_then(), dex_future_catch(), and dex_future_finally()), or even more elaborate flows (dex_future_all(), dex_future_all_race(), dex_future_any(), and dex_future_first()) at one place, without the typical callback hell. It still requires splitting things into a bunch of functions and potentially moving data through them.

static DexFuture *
lookup_user_data_cb (DexFuture *future,
                     gpointer   user_data)
{
  g_autoptr(MyUser) user = NULL;
  g_autoptr(GError) error = NULL;

  // the future in this cb is already resolved, so this just gets the value
  // no fibers involved 
  user = dex_await_object (future, &error);
  if (!user)
    return dex_future_new_for_error (g_steal_pointer (&error));

  return dex_future_first (dex_timeout_new_seconds (60),
                           dex_future_any (query_db_server (user),
                                           query_cache_server (user),
                                           NULL),
                           NULL);
}

static void
print_user_data (void)
{
  g_autoptr(DexFuture) future = NULL;

  future = dex_future_then (find_user (), lookup_user_data_cb, NULL, NULL);
  future = dex_future_then (future, print_user_data_cb, NULL, NULL);
  future = dex_future_finally (future, quit_cb, NULL, NULL);

  g_main_loop_run (main_loop);
}

The real magic of libdex however lies in fibers and the dex_await() function, which allows you to write code that looks synchronous but executes asynchronously. When you await a future, the current fiber yields control, allowing other work to proceed while waiting for the result.

g_autoptr(MyUser) user = NULL;
g_autoptr(MyUserData) data = NULL;
g_autoptr(GError) error = NULL;

user = dex_await_object (find_user (), &error);
if (!user)
  return dex_future_new_for_error (g_steal_pointer (&error));

data = dex_await_boxed (dex_future_first (dex_timeout_new_seconds (60),
                                          dex_future_any (query_db_server (user),
                                                          query_cache_server (user),
                                                          NULL),
                                          NULL), &error);
if (!data)
  return dex_future_new_for_error (g_steal_pointer (&error));

g_print ("%s", data->name);

Christian Hergert wrote pretty decent documentation, so check it out!

Bridging libdex and GDBus

With the new integration, you can write D-Bus client code that looks like this:

g_autoptr(DexDbusPingPong) *pp = NULL;
g_autoptr(DexDbusPingPongPingResult) result = NULL;

pp = dex_await_object (dex_dbus_ping_pong_proxy_new_future (connection,
                                                            G_DBUS_PROXY_FLAGS_NONE,
                                                            "org.example.PingPong",
                                                            "/org/example/pingpong"),
                       &error);
if (!pp)
  return dex_future_new_for_error (g_steal_pointer (&error));

res = dex_await_boxed (dex_dbus_ping_pong_call_ping_future (pp, "ping"), &error);
if (!res)
  return dex_future_new_for_error (g_steal_pointer (&error));

g_print ("client: %s\n", res->pong);

This code is executing asynchronously, but reads like synchronous code. Error handling is straightforward, and there are no callbacks involved.

On the service side, if enabled, method handlers will run in a fiber and can use dex_await() directly, enabling complex asynchronous operations within service implementations:

static gboolean
handle_ping (DexDbusPingPong       *object,
             GDBusMethodInvocation *invocation,
             const char            *ping)
{
  g_print ("service: %s\n", ping);

  dex_await (dex_timeout_new_seconds (1), NULL);
  dex_dbus_ping_pong_complete_ping (object, invocation, "pong");

  return G_DBUS_METHOD_INVOCATION_HANDLED;
}

static void
dex_dbus_ping_pong_iface_init (DexDbusPingPongIface *iface)
{
  iface->handle_ping = handle_ping;
}

pp = g_object_new (DEX_TYPE_PING_PONG, NULL);
dex_dbus_interface_skeleton_set_flags (DEX_DBUS_INTERFACE_SKELETON (pp),
                                       DEX_DBUS_INTERFACE_SKELETON_FLAGS_HANDLE_METHOD_INVOCATIONS_IN_FIBER);

This method handler includes a 1-second delay, but instead of blocking the entire service, it yields control to other fibers during the timeout.

The merge request contains a complete example of a client and service communicating with each other.

Implementation Details

The integration required extending GDBus’s code generation system. Rather than modifying it directly, the current solution introduces a very simple extension system to GDBus’ code generation.

The generated code includes:

Future-returning functions: For every _proxy_new() and _call_$method() function, corresponding _future() variants are generated
Result types: Method calls return boxed types containing all output parameters
Custom skeleton base class: Generated skeleton classes inherit from DexDBusInterfaceSkeleton instead of GDBusInterfaceSkeleton, which implements dispatching method handlers in fibers

Besides the GDBus code generation extension system, there are a few more changes required in GLib to make this work. This is not merged at the time of writing, but I’m confident that we can move this forward.

Future Directions

I hope that this work convinces more people to use libdex! We have a whole bunch of existing code bases which will have to stick with C in the foreseeable future, and libdex provides tools to make incremental improvements. Personally, I want to start using in in the xdg-desktop-portal project.

Testing with Portals

Sebastian Wick — Thu, 21 Aug 2025 23:00:55 +0200

At the Linux App Summit (LAS) in Albania three months ago, I gave a talk about testing in the xdg-desktop-portal project. There is a recording of the presentation, and the slides are available as well.

To give a quick summary of the work I did:

Revamped the CI
Reworked and improved the pytest based integration test harness
Added integration tests for new portals
Ported over all the existing GLib/C based integration tests
Support ASAN for detecting memory leaks in the tests
Made tests pretend to be either a host, Flatpak or Snap app

The hope I had is that this will result in:

Fewer regressions
Tests for new features and bug fixes
More confidence in refactoring
More activity in the project

While it’s hard to get definite data on those points, at least some of it seems to have become reality. I have seen an increase in activity (there are other factors to this for sure), and a lot of PRs already come with tests without me even having to ask for it. Canonical is involved again, taking care of the Snap side of things. So far it seems like we didn’t introduce any new regressions, but this usually shows after a new release. The experience of refactoring portals also became a lot better because there is a baseline level of confidence when the tests pass, as well as the possibility to easily bisect issues. Overall I’m already quite happy with the results.

Two weeks ago, Georges merged the last piece of what I talked about in the LAS presentation, so we’re finally testing the code paths that are specific to host, Flatpak and Snap applications! I also continued a bit with improving the tests, and now they can be run with Valgrind, which is super slow and that’s why we’re not doing it in the CI, but it tends to find memory leaks which ASAN does not. With the existing tests, it found 9 small memory leaks.

If you want to improve the Flatpak story, come and contribute to xdg-desktop-portal. It’s now easier than ever!

Display Next Hackfest 2025

Sebastian Wick — Fri, 15 Aug 2025 17:41:49 +0200

A few weeks ago, a bunch of display driver and compositor developers met once again for the third iteration of the Display Next Hackfest. The tradition was started by Red Hat, followed by Igalia (thanks Melissa), and now AMD (thanks Harry). We met in the AMD offices in Markham, Ontario, Canada; and online, to discuss issues, present things we worked on, figure out future steps on a bunch of topics related to displays, GPUs, and compositors.

The Display Next Hackfest in the AMD Markham offices

It was really nice meeting everyone again, and also seeing some new faces! Notably, Charles Poynton who “decided that HD should have 1080 image rows, and square pixels”, and Keith Lee who works for AMD and designed their color pipeline, joined us this year. This turned out to be invaluable. It was also great to see AMD not only organizing the event, but also showing genuine interest and support for what we are trying to achieve.

This year’s edition is likely going to be the last dedicated Display Next Hackfest, but we’re already plotting to somehow fuse it with XDC next year in some way.

If you’re looking for a more detailed technical rundown of what we were doing there, you can read Xaver’s, or Louis’ blog posts, or our notes.

With all that being said, here is an incomplete list of things I found exciting:

The biggest update of the Atomic KMS API (used to control displays) is about to get merged. The Color Pipeline API is something I came up with three years ago, and thanks to the tireless efforts of AMD, Intel and Igalia and others, this is about to become reality. Read Melissa’s blog post for more details.
As part of the work enabling displaying HDR content in Wayland compositors, we’ve been unhappy with the current HDR modes in displays, as they are essentially created for video playback and have lots of unpredictable behavior. To address this, myself and Xaver have since last year been lobbying for displays to allow the use of Source Based Tone Mapping (SBTM), and this year, it seems that what we have asked for have made it to the right people. Let’s see!
In a similar vein, on mobile devices we want to dynamically increase or decrease the HDR headroom, depending on what content applications want to show. This requires backlight changes to be somewhat atomic and having a mapping to luminance. The planned KMS backlight API will allow us to expose this, if the platform supports it. I worked a lot on backlight support in mutter this year so we can immediately start using this when it becomes available.
Charles, Christopher, and I had a discussion about compositing HDR and SDR content, and specifically about how to adjust content that was mastered for a dark viewing environment that is being shown in a bright viewing environment, so that the perception is maintained. I believe that we now have a complete picture of how compositing should work, and I’m working on documenting this in the color-and-hdr repo.
For Variable Refresh Rates (VRR) we want a new KMS API to set the minimum and maximum refresh cycle, where setting min=max gives us a fixed refresh rate without a mode set. To make use of VRR in more than the single-fullscreen-window case, we also agreed that a Wayland protocol letting clients communicate their preferred refresh rate would be a good idea.

Like always, lots of work ahead of us, but it’s great to actually see the progress this year with the entire ecosystem having HDR support now.

Sampling local craft beers

See you all at XDC this year (or at least the one next year)!

GNOME 49 Backlight Changes

Sebastian Wick — Fri, 08 Aug 2025 16:26:23 +0100

One of the things I’m working on at Red Hat is HDR support. HDR is inherently linked to luminance (brightness, but ignoring human perception) which makes it an important parameter for us that we would like to be in control of.

One reason is rather stupid. Most external HDR displays refuse to let the user control the luminance in their on-screen-display (OSD) if the display is in HDR mode. Why? Good question. Read my previous blog post.

The other reason is that the amount of HDR headroom we have available is the result of the maximum luminance we can achieve versus the luminance that we use as the luminance a sheet of white paper has (reference white level). For power consumption reasons, we want to be able to dynamically change the available headroom, depending on how much headroom the content can make use of. If there is no HDR content on the screen, there is no need to crank up the backlight to give us more headroom, because the headroom will be unused.

To work around the first issue, mutter can change the signal it sends to the display, so that white is not a signal value of 1.0 but somewhere else between 0 and 1. This essentially emulates a backlight in software. The drawback is that we’re not using a bunch of bits in the signal anymore and issues like banding might become more noticeable, but since we’re using this only with 10 or 12 bits HDR signals, this isn’t an issue in practice.

This has been implemented in GNOME 48 already, but it was an API that mutter exposed and Settings showed as “HDR Brightness”.

"HDR Brightness" in GNOME Settings

The second issue requires us to be able to map the backlight value to luminance, and change the backlight atomically with an update to the screen. We could work towards adding those things to the existing sysfs backlight API but it turns out that there are a number of problems with it. Mapping the sysfs entry to a connected display is really hard (GNOME pretends that there is only one single internal display that can ever be controlled), and writing a value to the backlight requires root privileges or calling a logind dbus API. One internal panel can expose multiple backlights, the value of 0 can mean the display turns off or is just really dim.

So a decision was made to create a new API that will be part of KMS, the API that we use to control the displays.

The sysfs backlight has been controlled by gnome-settings-daemon and GNOME Shell called a dbus API when the screen brightness slider was moved.

To recap:

There is a sysfs backlight API for basically one internal panel requires logind or a setuid helper executable
gnome-settings-daemon controlled this single screen sysfs backlight
Mutter has a “software backlight” feature
KMS will get a new backlight API that needs to be controlled by mutter

Overall, this is quite messy, so I decided to clean this up.

Over the last year, I moved the sysfs backlight handling from gnome-settings-daemon into mutter, added logic to decide which backlight to use (sysfs or the “software backlight”), and made it generic so that any screen can have a backlight. This means mutter is the single source of truth for the backlight itself. The backlight itself gets its value from a number of sources. The user can configure the screen brightness in the quick settings menu and via keyboard shortcuts. Power saving features can kick in and dim the screen. Lastly, an Ambient Light Sensor (ALS) can take control over the screen brightness. To make things more interesting, a single “logical monitor” can have multiple hardware monitors which each can have a backlight. All of that logic is now neatly sitting in GNOME Shell which takes signals from gnome-settings-daemon about the ALS and dimming. I also changed the Quick Settings UI to make it possible to control the brightness on multiple screens, and removed the old “HDR Brightness” from Settings.

There should have been a video here but your browser does not seem to support it.

All of this means that we can now handle screen brightness on multiple monitors and when the new KMS backlight API makes it upstream, we can just plug it in, and start to dynamically create HDR headroom.

Blender HDR and the reference white issue

Sebastian Wick — Sun, 13 Jul 2025 15:26:01 +0100

The latest alpha of the upcoming Blender 5.0 release comes with High Dynamic Range (HDR) support for Linux on Wayland which will, if everything works out, make it into the final Blender 5.0 release on October 1, 2025. The post on the developer forum comes with instructions on how to enable the experimental support and how to test it.

If you are using Fedora Workstation 42, which ships GNOME version 48, everything is already included to run Blender with HDR. All that is required is an HDR compatible display and graphics driver, and turning on HDR in the Display Settings.

It’s been a lot of personal blood, sweat and tears, paid for by Red Hat across the Linux graphics stack for the last few years to enable applications like Blender to add HDR support. From kernel work, like helping to get the HDR mode working on Intel laptops, and improving the Colorspace and HDR_OUTPUT_METADATA KMS properties, to creating a new library for EDID and DisplayID parsing, and helping with wiring things up in Vulkan.

I designed the active color management paradigm for Wayland compositors, figured out how to properly support HDR, created two wayland protocols to let clients and compositors communicate the necessary information for active color management, and created documentation around all things color in FOSS graphics. This would have also been impossible without Pekka Paalanen from Collabora and all the other people I can’t possibly list exhaustively.

For GNOME I implemented the new API design in mutter (the GNOME Shell compositor), and helped my colleagues to support HDR in GTK.

Now that everything is shipping, applications are starting to make use of the new functionality. To see why Blender targeted Linux on Wayland, we will dig a bit into some of the details of HDR!

HDR, Vulkan and the reference white level

Blender’s HDR implementation relies on Vulkan’s VkColorSpaceKHR, which allows applications to specify the color space of their swap chain, enabling proper HDR rendering pipeline integration. The key color space in question is VK_COLOR_SPACE_HDR10_ST2084_EXT, which corresponds to the HDR10 standard using the ST.2084 (PQ) transfer function.

However, there’s a critical challenge with this Vulkan color space definition: it has an undefined reference white level.

Reference white indicates the luminance or a signal level at which a diffuse white object (such as a sheet of paper, or the white parts of a UI) appears in an image. If images with different reference white levels end up at different signal levels in a composited image, the result is that “white” in one of the images is still being perceived as white, while the “white” from the other image is now being perceived as gray. If you ever scrolled through Instagram on an iPhone or played an HDR game on Windows, you will probably have noticed this effect.

The solution to this issue is called anchoring. The reference white level of all images needs to be normalized in order for “white” ending up on the same signal level in the composited image.

Another issue with the reference white level specific to PQ is the prevalent myth, that the absolute luminance of a PQ signal must be replicated on the actual display a user is viewing the content at. PQ is a bit of a weird transfer characteristic because any given signal level corresponds to an absolute luminance with the unit cd/m² (also known as nit). However, the absolute luminance is only meaningful for the reference viewing environment! If an image is being viewed in the reference viewing environment of ITU-R BT.2100, (essentially a dark room) and the image signal of 203 nits is being shown at 203 nits on the display, it makes the image appear as the artist intended. The same is not true when the same image is being viewed on a phone with the summer sun blasting on the screen from behind.

PQ is no different from other transfer characteristics in that the reference white level needs to be anchored, and that the anchoring point does not have to correspond to the luminance values that the image encodes.

Coming back to the Vulkan color space VK_COLOR_SPACE_HDR10_ST2084_EXT definition: “HDR10 (BT2020) color space, encoded according to SMPTE ST2084 Perceptual Quantizer (PQ) specification”. Neither ITU-R BT.2020 (primary chromaticity) nor ST.2084 (transfer characteristics), nor the closely related ITU-R BT.2100 define the reference white level. In practice, the reference level of 203 cd/m² from ITU-R BT.2408 (“Suggested guidance for operational practices in high dynamic range television production”) is used. Notable, this is however not specified in the Vulkan definition of VK_COLOR_SPACE_HDR10_ST2084_EXT.

The consequences of this? On almost all platforms, VK_COLOR_SPACE_HDR10_ST2084_EXT implicitly means that the image the application submits to the presentation engine (what we call the compositor in the Linux world) is assumed to have a reference white level of 203 cd/m², and the presentation engine adjusts the signal in such a way that the reference white level of the composited image ends up at a signal value that is appropriate for the actual viewing environment of the user. On GNOME, the way to control this currently is the “HDR Brightness” slider in the Display Settings, but will become the regular screen brightness slider in the Quick Settings menu.

On Windows, the misunderstanding that a PQ signal value must be replicated one to one on the actual display has been immortalized in the APIs. It was only until support for HDR was added to laptops that this decision was revisited, but changing the previous APIs was already impossible at this point. Their solution was exposing the reference white level in the Win32 API and tasking applications to continuously query the level and adjust the image to match the new level. Few applications actually do this, with most games providing a built-in slider instead.

The reference white level of VK_COLOR_SPACE_HDR10_ST2084_EXT on Windows is essentially a continuously changing value that needs to be queried from Windows APIs outside of Vulkan.

This has two implications:

It is impossible to write a cross-platform HDR application in Vulkan (if Windows is one of the targets)
On Wayland, a “standard” HDR signal can just be passed on to the compositor, while on Windows, more work is required

While the cross-platform issue is solvable, and something we’re working on, the way Windows works also means that the cross-platform API might become harder to use because we cannot change the underlying Windows mechanisms.

No Windows Support

The result is that Blender currently does not support HDR on Windows.

Jeroen-Bakker explaining the lack of Windows support

The design of the Wayland color-management protocol, and the resulting active color-management paradigm of Wayland compositors was a good choice, making it easy for developers to do the right thing, while also giving them more control if they so chose.

Looking forward

We have managed to transition the compositor model from a dumb blitter to a component which takes an active part in color management, we have image viewers and video players with HDR support and now we have tools for producing HDR content! While it is extremely exciting for me that we have managed to do this properly, we also have a lot of work ahead of us, some of which I will hopefully tell you about in a future blog post!

Booting into Toolbox Containers

Sebastian Wick — Tue, 30 Jan 2024 13:23:30 +0100

There are a lot of tangible benefits in using toolbox containers for development to the point that I don’t want to use anything else anymore. Even with a bunch of tricks at our disposal, there are still downsides. The containers are not complete sessions but rather try to integrate with the host session. If you’re working on something that is part of a session it might be possible to run a test suite and even more elaborate setups but running it fully integrated often becomes a problem.

If the host is a traditional mutable system it’s possible to just not use toolbox. If you’re on an immutable system they often offer some way to make it mutable temporarily using some kind of overlay at which point they behave mostly like the traditional mutable systems. The unfortunate side effect is that you don’t get the benefits of toolbox anymore.

It’s also possible to develop in the toolbox and on the host system at the same time, depending on what specifically you’re working on right now to get the benefits of both systems. The drawback is that the toolbox container and the host are different systems. You’re setting up, compiling, etc. everything twice and run your project in different environments. Also not ideal.

We can do better. Toolbox can, in theory, use arbitrary OCI images. In practice there are assumptions from toolbox on how an image looks and behaves. Fedora Silverblue, or rather rpm-ostree, can also, in theory, boot arbitrary OCI images but also comes with its assumptions.

It turns out that in practice the unofficial OCI image variant of Fedora Silverblue can be used as a toolbox image and the images of such containers can be booted into with rpm-ostree.

$ toolbox create -i quay.io/fedora-ostree-desktops/silverblue:39 my-silverblue-toolbox
$ toolbox enter my-silverblue-toolbox
⬢ # install dnf to make it behave like a usual toolbox container
⬢ sudo rpm-ostree install -y dnf
⬢ sudo dnf update -y
⬢ # Let's install strace and gdb. Do whatever you want with your container! 
⬢ sudo dnf install -y strace gdb
⬢ exit
$ # some magic to convert the running container into something rpm-ostree understands
$ # there are probably ways to do this with less copying (tell me if you know)
$ podman commit my-silverblue-toolbox my-silverblue-toolbox-image
$ sudo mkdir -p /var/lib/my-silverblue-toolbox-image
$ podman save --format=oci-archive "my-silverblue-toolbox-image" | sudo tar -x -C "/var/lib/my-silverblue-toolbox-image"
$ sudo rpm-ostree rebase "ostree-unverified-image:oci:/var/lib/my-silverblue-toolbox-image"
$ # boot into our toolbox
$ sudo systemctl reboot -i

One toolbox container to develop in and one reboot to test the changes in a full session on real hardware. This is all unsupported and might break in interesting ways but it shows the power of OCI based operating systems and toolbox.

On the Usefulness of SO_PEERPIDFD

Sebastian Wick — Thu, 21 Sep 2023 18:11:03 +0200

Update Oct. 2025: Things have changed since writing this. In a follow-up post I explain how SO_PEERPIDFD Gets More Useful.

Kernel 6.5 added a few new pidfd functions: SCM_PIDFD and SO_PEERPIDFD. The idea behind them is the same as SCM_CREDENTIALS and SO_PEERCRED respectively. The only difference is that the PIDFD functions return not a plain, numerical PID but a file descriptor instead.

A plain PID is small number of type pid_t that is incremented for each new process and wraps over when too many processes have been created. This PID is usually used to look up some information about the process via files in /proc/$PID. While a process is looking up some information, it is possible that the process that PID initially referred to has terminated and a new process with this PID has been created. The looked up information is now incorrect, possibly resulting in a security vulnerability.

The pidfd on the other hand always refers to one process and can be queried about the state of the process. This allows one to look up information from /proc/$PID without the race mentioned earlier. The SO_PEERCRED functionality in particular is interesting because it allows a service to query the pidfd of a connected client.

Or so it seems. For flatpak, wayland compositors and D-Bus services also want to authenticate their clients but they do not rely on this functionality. Instead the preferred approach taken here (implemented in wayland as the security-context protocol, still in discussion for D-Bus as the org.freedesktop.DBus.Containers1 interface) is to create a new wayland or D-Bus socket for each application instance and make sure that those sockets are the only way to connect to the services (specifically the “normal” host sockets must not be made available to the application instances). Flatpak is responsible for creating those sockets, attaching some metadata, and then mounting them. Currently the metadata is a triple: the sandboxing engine (flatpak, snap, …), the application id and the application instance. There are plans to extend this for further metadata.

The question is, why add all of this complexity to authenticate a process when the pidfd approach is so much easier. Turns out that the hard part is knowing what to do with the pid, where to look up the information that you need. For flatpak, /proc/$PID/root/.flatpak-info is a file that cannot be changed from processes in the sandbox and contains among other things the flatpak instance-id which can be used to look up more data from $XDG_RUNTIME_DIR/.flatpak/$instance-id. For snap the whole process is very different and other technologies like firejail have basically no way to do this lookup (don’t take my word on it).

(Aside: xdg-desktop-portal does does look up flatpak information from /proc/$PID and I just said it doesn’t use SO_PEERCRED so why is this not broken? Flatpak makes sure there is a process in the sandbox that stays alive the entire time and acts as a proxy between the app and the D-Bus broker to enforce access control. The PID of all connections from inside the sandbox are the PID of the proxy. This is still technically racy but it becomes much harder to pull off an attack. Implementing SO_PEERCRED to get rid of the race entirely in xdg-desktop-portal would be nice.)

Implementing all kinds of different, subtle mechanisms to look up information from different sandbox engines you might not even know exist doesn’t scale and that makes pidfd with SO_PEERCRED much less useful than one would expect in a lot of cases.

Fedora Silverblue Development Utils

Sebastian Wick — Thu, 31 Aug 2023 00:00:00 +0000

In the previous post we explored all the different ways to develop software on Fedora Silverblue when Toolbx and Flatpak are not enough. Some of the ideas there are interesting, some are dead-ends and some are extremely useful.

I’ve extracted all the useful parts to a small script I’m calling silverblue-devel-utils.

$ silverblue-devel-utils 
Utils for developing on a Silverblue system

Syntax:
silverblue-devel-utils make_mutable [--temporary]
silverblue-devel-utils unmake_mutable
silverblue-devel-utils cleanup_mutable
silverblue-devel-utils rebase_image <IMAGE>

Let’s go over the functionality: make_mutable makes /usr writable and installs dnf. By default the mutable changes persist until unmake_mutable is called and the machine is rebooted. If you want the changes to automatically revert on the next boot you can use make_mutable --temporary. When you have booted into the previous state, you can call cleanup_mutable do get rid of the deployment containing your changes.

Last but not least, rebase_image can rebase your Silverblue install to any image in your local repository. This can be useful if you plan on keeping your changes for a long time, if creating all changes as rpms is hard or you just want to make very specific changes. Using a Dockerfile with the Silverblue OCI image as a base is the easiest way to create such images.

Let’s see how we can actually use the script! First of all, let’s install it:

curl -o ~/.local/bin/silverblue-devel-utils https://gist.githubusercontent.com/swick/968453815a1e918f0f0f56faff310a80/raw/65cb9b0581c22d9978daace0f850cbc5c5018d4b/silverblue-devel-utils && chmod +x ~/.local/bin/silverblue-devel-utils

Now, let’s try temporarily making our system mutable:

$ silverblue-devel-utils make_mutable --temporary
$ sudo dnf install -y gdb
$ sudo dnf debuginfo-install -y mutter gnome-shell
$ # do some serious debugging
$ dbus-run-session gdb --args gnome-shell --wayland --nested
$ systemctl reboot
...
$ # and we're back on our old system!

Let’s do the same, but this time with the changes persisting across boots:

$ silverblue-devel-utils make_mutable
$ sudo dnf builddep -y mutter
$ sudo dnf install -y libei-devel libeis-devel
$ meson setup mybuild --prefix=/usr
$ ninja -C mybuild/
$ sudo ninja -C mybuild/ install
$ # our new mutter is here!
$ mutter --version
mutter 45.beta.1
$ systemctl reboot
...
$ mutter --version
mutter 45.beta.1
$ # still the version we build!
$ silverblue-devel-utils unmake_mutable
$ systemctl reboot
...
$ silverblue-devel-utils cleanup_mutable
$ # our old version again
$ mutter --version
mutter 44.3

And finally, let’s build our own OS image based on Silverblue and boot into it:

ARG IMAGE_NAME="${IMAGE_NAME:-silverblue}"
ARG SOURCE_IMAGE="${SOURCE_IMAGE:-silverblue}"
ARG BASE_IMAGE="quay.io/fedora-ostree-desktops/${SOURCE_IMAGE}"
ARG FEDORA_MAJOR_VERSION="${FEDORA_MAJOR_VERSION:-38}"

FROM ${BASE_IMAGE}:${FEDORA_MAJOR_VERSION} AS base

ARG IMAGE_NAME="${IMAGE_NAME}"
ARG FEDORA_MAJOR_VERSION="${FEDORA_MAJOR_VERSION}"

# setup dnf
RUN rpm-ostree install -y dnf
RUN dnf install -y 'dnf-command(builddep)'

# install development packages
RUN dnf groupinstall -y 'Development Tools'
RUN ln -s /usr/bin/ld.gold /usr/bin/ld # ???
RUN dnf install -y meson strace gdb valgrind sysprof

# clean and commit
RUN rm -rf /tmp/* /var/*
RUN ostree container commit
RUN mkdir -p /var/tmp && chmod -R 1777 /var/tmp

$ podman build -f Dockerfile -t my-very-own-os-image
$ silverblue-devel-utils rebase_image my-very-own-os-image
$ systemctl reboot
...
$ # our OS has strace installed!
$ strace test

The system will stay on this image until another one is installed with silverblue-devel-utils rebase_image (every now and then, you might want to podman pull the base image and rebuild your own image to receive updates) or you manually rebase to another origin with rpm-ostree rebase.

Thanks to Jordan Petridis for his great blog post on systemd-sysext which made me explore all the available tools! Thanks to Ivan Molodetskikh and Tomáš Popela for the feedback on the previous post!

I hope you find this tool useful!

Developing Gnome Shell on Fedora Silverblue

Sebastian Wick — Tue, 22 Aug 2023 00:45:53 +0200

Fedora Silverblue is one of the immutable operating systems. Somewhere the OS is being built, resulting in an ostree commit that gets distributed to the machines running Silverblue. On the machines this results in a new deployment which will become active on the next boot, while the previous deployment is still around, ready to be booted into if anything goes wrong. The deployments are read-only.

The resulting stabilility and fault-tolerance is, compared with the traditional single directory tree, mutated by packages, nothing short of incredible.

There are drawbacks, though. The read-only nature of the OS, and /usr in particular, makes some development work-flows unusable. Want to install a -devel package to build your software or just need a compiler? Technically you could manage to do so with rpm-ostree’s layering capabilities (man rpm-ostree) but this is slow, requires a reboot (the apply-live option exists but there are restrictions) and is generally frowned upon.

The general solution to this problem has been two-fold:

Embrace containers
Toolbx

Silverblue ships with podman and other container tools out of the box which allows you to build your software using Dockerfiles or other container native technologies. It allows you to run containers with podman run. For a lot of uses cases this is sufficient.

When this is not sufficient or otherwise undesirable Toolbx will come to the rescue! It allows you to create a pet container with toolbox create $name and to enter it with toolbox enter $name. The great thing about it: it looks and feels just like any fedora system (other operating systems are supported as well), so you can dnf install your compiler and -devel packages. It also shares your home directory and is integrated with the host.

Building and running mutter and gnome-shell inside a toolbox has been possible for a while now thanks to work both on Toolbx and mutter. I’ve continued making sure everything is working as smoothly as possible. It’s possible to run mutter on a TTY and even run all the complicated testing setups (some patches pending).

As great as this is, there are situations where the developed version of gnome-shell needs to to be run in an actual, real session. Maybe even as part of an actual boot, maybe to run it long enough to find some issue that shows up very infrequently. In those cases Toolbx is not enough and we need to somehow get our gnome-shell running on the host.

systemd-sysext and rpm-ostree usroverlay

There are two more mechanisms to modify files in /usr besides rpm-ostree overlays:

rpm-ostree usroverlay
systemd-sysext

Conceptually they are very similar. They use some overlay filesystem on top of the otherwise read-only /usr tree. With rpm-ostree usroverlay this is really all there is to it. You enable it and now you can modify files in /usr.

If we build gnome-shell for example in our toolbox with --prefix=/usr and then call ninja install we’re installing it into /usr of the toolbox container. The host OS is mounted in /run/host in the toolbox so if we just use DESTDIR=/run/host/usr ninja install then it… still doesn’t work because we can’t get the right permissions to modify files on the host from inside the toolbox (root is not mapped to any user, /run/host is nobody). But with some clever use of DESTDIR=/tmp/gnome-shell-install ninja install and rsyncing /tmp/gnome-shell-install to /usr, it is possible to install our gnome-shell from inside the toolbox onto our host OS. At least temporarily. The rpm-ostree usroverlay disappears when the machine powers down.

If we need persistence across reboots, then systemd-sysext seems exactly like what we need. We can just create a new folder in /var/lib/extensions and drop our files in there (it needs a special file to be picked up as extension), use systemd-sysext list to show the available extensions, status to show if they are currently active and merge and unmerge to overlay and remove the overlay of the extensions. Enabling systemd-sysext.service makes systemd overlay the extensions on boot. We can use the same process as before to get our installation from the toolbox to the right place on the host.

The tiny problem

Now that we’ve successfully installed gnome-shell to our host, we can for example just run gnome-shell --nested and see… missing dependencies?

In this GNOME development cycle mutter grew a dependency on libei and this is no issue in the toolbox. We just dnf install libei-devel and we’re done. The problem though: this is not our host and libei will only be available in the toolbox. We can clone, build and install the dependency just like we did for gnome-shell but this gets ugly fast when the project you’re building has a lot of dependencies that are not installed on the host. In this case, not a big issue though, so we keep pushing ahead and install what we need.

This time gnome-shell starts, just to exit again. This cycle mutter gained the cancel-input-capture keybinding which is defined in a gschema file, which gets installed correctly, but needs to be compiled with all the other gschemas on the host system and that’s not happening. It can’t be happening because the toolbox has its own set of gschemas. As part of the build process, the compiler will build them into the toolbox, instead of compiling the host gschemas and the new gschema to the host. We can be clever and run the compiler on the host and install the results either to /usr for rpm-ostree usroverlay or the extension directory for systemd-sysext.

This doesn’t seem very robust.

Where is the root issue here? We’re building on one operating system and installing the result to another. This can’t work well.

Embracing containers again

We can’t build gnome-shell in our toolbox! Instead, we have to build it on our host. At least, something that looks exactly like our host. The nice thing about the immutable system here is that we know exactly how our host looks like and there is an ostree commit which describes it. Not only that, we can also build a native container (OCI) image from that ostree commit and someone already did that.

We can create our own Dockerfile which builds gnome-shell and mutter based on the Silverblue image:

ARG IMAGE_NAME="${IMAGE_NAME:-silverblue}"
ARG SOURCE_IMAGE="${SOURCE_IMAGE:-silverblue}"
ARG BASE_IMAGE="quay.io/fedora-ostree-desktops/${SOURCE_IMAGE}"
ARG FEDORA_MAJOR_VERSION="${FEDORA_MAJOR_VERSION:-38}"

FROM ${BASE_IMAGE}:${FEDORA_MAJOR_VERSION} AS builder

ARG IMAGE_NAME="${IMAGE_NAME}"
ARG FEDORA_MAJOR_VERSION="${FEDORA_MAJOR_VERSION}"

# setup dnf
RUN rpm-ostree install -y dnf
RUN dnf install -y 'dnf-command(builddep)'

# install development packages
RUN dnf groupinstall -y 'Development Tools'
RUN ln -s /usr/bin/ld.gold /usr/bin/ld # ???
RUN dnf install -y meson strace gdb valgrind sysprof

# install gnome shell and mutter specific dependencies
RUN dnf builddep -y gnome-shell mutter
RUN dnf install -y libei-devel libeis-devel asciidoc sassc

# dnf won't work on the running system, let's not confuse ourselves
RUN rpm-ostree uninstall -y dnf

# build mutter
ADD ./mutter /tmp/mutter
RUN cd /tmp/mutter && \
    meson setup _container_build/ --prefix=/usr && \
    ninja -C _container_build/ && \
    ninja -C _container_build/ install

# build gnome-shell
ADD ./gnome-shell /tmp/gnome-shell
RUN cd /tmp/gnome-shell && \
    meson setup _container_build/ --prefix=/usr && \
    ninja -C _container_build/ && \
    ninja -C _container_build/ install

RUN rm -rf /tmp/* /var/*
RUN ostree container commit
RUN mkdir -p /var/tmp && chmod -R 1777 /var/tmp

And then build our own Silverblue based OS which includes our own gnome-shell: podman build -f GnomeShell.containerfile -t silverblue-gnome-shell-main.

Pretty neat, but how do we get the changes between the Silverblue image and our own image deployed onto our host?

OCI images consist of layers. Each layer can add, remove and modify files from the previous layer. When we build our own image based on another image, our changes become new layers in the new image. With a bit of massaging we can extract those layers into a filesystem tree, such as /var/lib/extensions:

#! /bin/bash

set -ouex pipefail

BASE_IMAGE=${BASE_IMAGE-quay.io/fedora-ostree-desktops/silverblue:38}
EXT_IMAGE=${EXT_IMAGE-localhost/silverblue-gnome-shell-main:latest}
SYSEXT_NAME=${SYSEXT_NAME-test-ext}

SYSEXT_PATH=/var/lib/extensions/$SYSEXT_NAME

function fail {
    printf '%s\n' "$1" >&2 ## Send message to stderr.
    exit "${2-1}" ## Return a code specified by $2, or 1 by default.
}

tmpdir="$(mktemp -d)"
trap 'rm -rf -- "$tmpdir"' EXIT

skopeo copy containers-storage:$BASE_IMAGE dir:$tmpdir/base-image
skopeo copy containers-storage:$EXT_IMAGE dir:$tmpdir/ext-image

base_info=$(skopeo inspect dir:$tmpdir/base-image --raw)
ext_info=$(skopeo inspect dir:$tmpdir/ext-image --raw)

base_layer_count=$(jq '.layers | length' <<< "$base_info")
ext_layer_count=$(jq '.layers | length' <<< "$ext_info")

if [[ $ext_layer_count -le $base_layer_count ]]; then
  fail "ext image needs more layers than base image"
fi

for (( i=0; i<$base_layer_count; ++i)); do
  base_digest=$(jq -r ".layers[${i}].digest" <<< "$base_info")
  ext_digest=$(jq -r ".layers[${i}].digest" <<< "$ext_info")

  if [[ "$base_digest" != "$ext_digest" ]]; then
    fail "layer $i digest mismatch: base $base_digest, ext $ext_digest"
  fi
done

sudo rm -rf "$SYSEXT_PATH"
sudo mkdir -p $SYSEXT_PATH/usr/lib/extension-release.d/
echo "ID=_any" | sudo tee $SYSEXT_PATH/usr/lib/extension-release.d/extension-release.$SYSEXT_NAME

for (( i=$base_layer_count; i<$ext_layer_count; ++i)); do
  ext_digest=$(jq -r ".layers[${i}].digest" <<< "$ext_info")
  ext_digest="${ext_digest#sha256:}"
  
  sudo tar xf $tmpdir/ext-image/$ext_digest -C $SYSEXT_PATH
done

This script is horribly inefficient, copies around all files multiple times. It also doesn’t deal with so-called whiteout files (the real filename, prefixed with .wh.) which are used to indicate a file from the previous layers was removed.

One could expand on this concept and create a real program which doesn’t have those inefficiencies and handles removal of files properly. However, we can’t handle whiteout files which remove files from the base image (i.e. our Silverblue system) with systemd-sysext simply because it doesn’t support removing files from the base system. It only works with the temporary rpm-ostree usroverlay.

All in all, this isn’t too bad. With a bit more investment this workflow could become usable. Can we do better?

Embracing new Operating Systems

Usually rpm-ostree pulls updates/commits from a ostree repository but did you know that it can also pull OCI images from container registries?

Did you notice we have an OCI image laying around which contains exactly everything we want?

#! /bin/sh

set -ouex pipefail

IMAGE_NAME=$1
OCI_IMAGE_STORAGE="/var/development-images/${IMAGE_NAME}"

sudo rm -rf "${OCI_IMAGE_STORAGE}"
sudo mkdir -p "${OCI_IMAGE_STORAGE}"
podman save --format=oci-archive "${IMAGE_NAME}" | sudo tar -x -C "${OCI_IMAGE_STORAGE}"
sudo rpm-ostree rebase "ostree-unverified-image:oci:${OCI_IMAGE_STORAGE}"
# revert: rpm-ostree rebase fedora:fedora/38/x86_64/silverblue

We can deploy this image using the script above with rpm-ostree-deploy-image silverblue-gnome-shell-main and reboot into it.

This is easy and robust! The only thing that’s annoying is that everything here takes a bit of time and requires a reboot. In a lot of cases this is fine. You develop in the toolbox and if a specific case comes up that requires testing in the entire session or under real conditions for a long amount of time, you can build your own OS like this and boot into it. When everything is done we can just rebase onto the upstream silverblue repository using rpm-ostree rebase fedora:fedora/38/x86_64/silverblue.

Can we do even better and get rid of the slow development loop?

Instead of installing gnome-shell directly into the image, we can stop after installing the dependencies and build tools, boot into this image, then build gnome-shell on our new host, install it into /var/lib/extensions and then activate the extension with systemd-sysext refresh. After the initial build of the image the development loop is almost the same as on a traditional system and just as fast.

Let’s start by copying the small helper scripts to ~/.local/bin.

rpm-ostree-deploy-image

#! /bin/sh

set -ouex pipefail

if [ "$#" -ne 1 ]; then
    echo "Illegal number of parameters"
fi

IMAGE_NAME=$1
OCI_IMAGE_STORAGE="/var/development-images/${IMAGE_NAME}"

sudo rm -rf "${OCI_IMAGE_STORAGE}"
sudo mkdir -p "${OCI_IMAGE_STORAGE}"
podman save --format=oci-archive "${IMAGE_NAME}" | sudo tar -x -C "${OCI_IMAGE_STORAGE}"
sudo rpm-ostree rebase "ostree-unverified-image:oci:${OCI_IMAGE_STORAGE}"
# revert: rpm-ostree rebase fedora:fedora/38/x86_64/silverblue

sysext-install

#! /bin/bash

set -eux

if [ "$#" -ne 2 ]; then
    echo "Illegal number of parameters"
fi

BUILDDIR="$1"
EXTENSION_NAME="$2"

DESTDIR="/var/lib/extensions/$EXTENSION_NAME"
RELEASE_DIR="$DESTDIR/usr/lib/extension-release.d"

sudo mkdir -p "$DESTDIR"
sudo meson install --destdir="$DESTDIR" -C "$BUILDDIR" --no-rebuild

sudo mkdir -p "$RELEASE_DIR"
echo ID=_any | sudo tee "$RELEASE_DIR/extension-release.$EXTENSION_NAME"

Adjust the Dockerfile a bit (make sure to put it in a directory containing both gnome-shell and mutter): GnomeShellDevelopment.containerfile.

ARG IMAGE_NAME="${IMAGE_NAME:-silverblue}"
ARG SOURCE_IMAGE="${SOURCE_IMAGE:-silverblue}"
ARG BASE_IMAGE="quay.io/fedora-ostree-desktops/${SOURCE_IMAGE}"
ARG FEDORA_MAJOR_VERSION="${FEDORA_MAJOR_VERSION:-38}"

FROM ${BASE_IMAGE}:${FEDORA_MAJOR_VERSION} AS base

ARG IMAGE_NAME="${IMAGE_NAME}"
ARG FEDORA_MAJOR_VERSION="${FEDORA_MAJOR_VERSION}"

# setup dnf
RUN rpm-ostree install -y dnf
RUN dnf install -y 'dnf-command(builddep)'

# install development packages
RUN dnf groupinstall -y 'Development Tools'
RUN ln -s /usr/bin/ld.gold /usr/bin/ld # ???
RUN dnf install -y meson strace gdb valgrind sysprof

# install gnome shell and mutter specific dependencies
RUN dnf builddep -y gnome-shell mutter
RUN dnf install -y libei-devel libeis-devel asciidoc sassc

# dnf won't work on the running system, let's not confuse ourselves
RUN rpm-ostree uninstall -y dnf

RUN rm -rf /tmp/* /var/*
RUN ostree container commit
RUN mkdir -p /var/tmp && chmod -R 1777 /var/tmp


FROM base AS build

# build mutter
ADD ./mutter /tmp/mutter
RUN cd /tmp/mutter && \
    meson setup _container_build/ --prefix=/usr && \
    ninja -C _container_build/ && \
    ninja -C _container_build/ install

# build gnome-shell
ADD ./gnome-shell /tmp/gnome-shell
RUN cd /tmp/gnome-shell && \
    meson setup _container_build/ --prefix=/usr && \
    ninja -C _container_build/ && \
    ninja -C _container_build/ install

RUN rm -rf /tmp/*

Then run the entire Dockerfile to make sure we actually got all the dependencies we need, tag the base stage as silverblue-38-gnome-shell-development, deploy this image, and then finally boot into it.

podman build -f GnomeShellDevelopment.containerfile
podman build -f GnomeShellDevelopment.containerfile --target=base -t silverblue-38-gnome-shell-development
rpm-ostree-deploy-image silverblue-38-gnome-shell-development
systemctl reboot

After the reboot we can then actually build mutter and gnome-shell on the host, install it into the extension with the sysext-install script and make it active with systemd-sysext refresh. Finally we can enable systemd-sysext.service to make it persist across reboots.

cd mutter

meson setup buildhost --prefix=/usr
ninja -C buildhost/
sysext-install buildhost/ gnome-shell-test
sudo systemd-sysext refresh

cd ../gnome-shell

meson setup buildhost --prefix=/usr
ninja -C buildhost/
sysext-install buildhost/ gnome-shell-test
sudo systemd-sysext refresh

gnome-shell --version
sudo systemctl enable systemd-sysext.service

Back to the basics

Did you know that rpm-ostree usroverlay is not the only way to get a mutable overlay on top of the read-only /usr tree? Me neither, until Ivan Molodetskikh pointed this out to me. ostree admin unlock --hotfix gets us a persistent overlay even!

This way we can get around the biggest drawback of using systemd-sysext: we can install directly into the filesystem tree and install-time build system integration such as the gschema compiler will do its job perfectly.

So, instead of installing all the dependencies and build tools into our OS, we just layer rpm on top of it, enable the persistent overlay with ostree admin unlock --hotfix, and start using it like a traditional system. We can install the dependencies with dnf builddep, anything else we might need with dnf install and then build and install everything as usual.

rpm-ostree install dnf
systemctl reboot

sudo ostree admin unlock --hotfix
sudo dnf install 'dnf-command(builddep)'
sudo dnf builddep gnome-shell mutter --allowerasing
sudo dnf install libei-devel libeis-devel asciidoc sassc

cd mutter
meson setup buildhost --prefix=/usr
ninja -C buildhost/
sudo ninja -C buildhost/ install

cd ../gnome-shell
meson setup buildhost --prefix=/usr
ninja -C buildhost/
sudo ninja -C buildhost/ install

The dependencies in the dnf repo and in the silverblue image can be slightly different which might require passing --allowerasing when installing new stuff with dnf.

ostree admin unlock --hotfix creates a new rollback target. To abandon the experiment and rollback to our previous state, we can simply use rpm-ostree rollback -r and clean up our messy deployment with rpm-ostree cleanup -r after successfully booting into the previous, good version.

Tomáš Popela pointed out that you can get dnf even without overlaying it using rpm-ostree by downloading and installing microdnf and install dnf with it. This way we can keep our base system clean, don’t have to reboot and don’t accidentally run dnf when /usr is read-only.

#!/bin/sh

set -e

tmp=$(mktemp -d)
cleanup () {
	rm -rf $tmp
}
trap cleanup EXIT

set -x

curl --silent --show-error --remote-name-all --output-dir $tmp \
  https://kojipkgs.fedoraproject.org//packages/microdnf/3.9.0/2.fc38/x86_64/microdnf-3.9.0-2.fc38.x86_64.rpm \
  https://kojipkgs.fedoraproject.org//packages/libpeas/1.34.0/3.fc38/x86_64/libpeas-1.34.0-3.fc38.x86_64.rpm \
  https://kojipkgs.fedoraproject.org//packages/dnf/4.14.0/2.fc38/noarch/dnf-data-4.14.0-2.fc38.noarch.rpm \
  https://kojipkgs.fedoraproject.org//packages/libdnf/0.68.0/2.fc38/x86_64/libdnf-0.68.0-2.fc38.x86_64.rpm

successful=$?

if [ $successful != 0 ] ; then
  exit 1
fi

if ! sudo rpm -i "$tmp/*.rpm"; then
  echo "Can't install microdnf!"
  exit 1
fi

if ! sudo microdnf --assumeyes install dnf python3-dnf-plugins*; then
  exit 1
fi

All in all, this seems to be the best option. It transforms the silverblue setup temporarily into a traditional package based, mutable directory tree like system which can be rolled back at any point. It’s easy to use, works well and doesn’t require any further kind of trickery.

Setting up a personal server in 2023

Sebastian Wick — Thu, 17 Aug 2023 21:22:35 +0200

Sometimes I just want to do a small thing, like creating a blog, and then end up noticing everything is horrible and I should probably start fixing things at the root.

In this instance, my personal server from a hoster didn’t receive an update in a long time.

$ cat /etc/os-release | grep PRETTY_NAME
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"

Debian 9… that seems a bit old.

Stretch also had benefited from Long Term Support (LTS) until the end of June 2022.

Crap.

I also didn’t update my mail server, mail frontend, rss reader and anything else really in a long time. Time for some change.

Hetzner cloud

For whatever reason I decided to go with Hetzner as my hoster. They have some reasonably priced low-end on-demand servers in their cloud. Both the Cloud and DNS services have ansible plugins to work with them and expose a nice API.

What they do not provide however is an official way to install CoreOS on their servers. That won’t stop us though: one can boot the server into a recovery mode where one can download an image and write it to the disk.

# CoreOS
- name: Check if CoreOS is installed
  shell: |
    ssh -oStrictHostKeyChecking=no root@{{ swick_server.hcloud_server.ipv4_address }} << 'EOF'
    cat /etc/os-release | grep CoreOS
    EOF
  register: check_os
  failed_when: check_os.rc != 1 and check_os.rc != 0

- name: Install CoreOS
  block:
  - name: Boot into rescue system
    hcloud_server:
      name: "{{ swick_server.hcloud_server.name }}"
      rescue_mode: linux64
      ssh_keys: "{{ project.ssh.name }}"
      state: restarted

  - name: Wait for SSH to come up
    wait_for:
      host: "{{ swick_server.hcloud_server.ipv4_address }}"
      port: 22
      timeout: 1800

  - name: Copy ignition config
    command: scp -oStrictHostKeyChecking=no {{ cache }}/config.ign root@{{ swick_server.hcloud_server.ipv4_address }}:./config.ign

  # TODO: make the install medium selection more robust (look at id, label?)
  - name: Install CoreOS
    shell: |
      ssh -oStrictHostKeyChecking=no root@{{ swick_server.hcloud_server.ipv4_address }} << 'EOF'
      wget "{{ fcos_image_url }}"
      echo "{{ fcos_image_checksum }}" > checksum
      sha256sum -c checksum || reboot
      xz -dc "{{ fcos_image_filename }}" | dd of={{ server_install_device }} status=progress
      mount {{ focs_image_boot_partition }} /mnt/
      mkdir -p /mnt/ignition
      cp config.ign /mnt/ignition/
      umount /mnt
      EOF

  - name: Reboot to CoreOS
    hcloud_server:
      name: "{{ swick_server.hcloud_server.name }}"
      state: restarted

  - name: Wait for SSH to come up
    wait_for:
      host: "{{ swick_server.hcloud_server.ipv4_address }}"
      port: 22
      sleep: 10
      timeout: 1800

  when: check_os.rc

Ignition, podman and docker-compose

Now all we need is our ignition file for CoreOS to configure itself. I’m going for a docker-compose setup which gets controlled by systemd and ansible. All docker services should run as the core user in the user session and connect to an unprivileged podman service. I also need a way to store podman volumes on persistent memory.

So, first of all we need some tools that are not on the default CoreOS install, such as python for ansible and semanage for setting up the correct labels for our podman volume storage.

systemd:
  units:

    # disable ssh, enable when we're ready
    - name: sshd.service
      enabled: false

    # install python and semanage as a layered package with rpm-ostree
    # also re-enable ssh when we're done
    - name: rpm-ostree-install-tools.service
      enabled: true
      contents: |
        [Unit]
        Description=Layer python and semanage with rpm-ostree
        Wants=network-online.target
        After=network-online.target
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/rpm-ostree install --apply-live --allow-inactive python3 policycoreutils-python-utils
        ExecStart=/usr/bin/systemctl enable --now sshd.service
        ExecStart=/bin/touch /var/lib/%N.stamp

        [Install]
        WantedBy=default.target

One trick I’m using here is to disable sshd in the ignition file to avoid ansible connecting before python is installed. Only after rpm-ostree install --apply-live succeeds sshd will get enabled and started. This allows ansible to use a simple wait_for task to wait for the server to become fully functional.

The other trick here is using the ConditionPathExists together with ExecStart=/bin/touch to avoid unnecessary work on later boots.

storage:
  files:

    # store volumes on persistent storage
    - path: /home/core/.config/containers/storage.conf
      mode: 0644
      user:
        name: core
      group:
        name: core
      contents:
        inline: |
          [storage]
            graphroot = "/var/storage/core-volumes"

systemd:
  units:

    # mount persistent storage
    - name: var-storage.mount
      enabled: true
      contents: |
        [Unit]
        Description=Mount persistent storage

        [Install]
        WantedBy=storage-directories.service

        [Mount]
        What={{ swick_server_volume.hcloud_volume.linux_device }}
        Where=/var/storage

    # create volume storage directory on persistent storage
    - name: storage-directories.service
      enabled: true
      contents: |
        [Unit]
        Description=Create volume storage directory on persistent storage
        After=var-storage.mount
        After=rpm-ostree-install-tools.service

        [Install]
        WantedBy=default.target

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/mkdir -p /var/storage/core-volumes
        ExecStart=/usr/bin/chown core:core /var/storage/core-volumes
        ExecStart=/usr/sbin/semanage fcontext -a -e /var/home/core/.local/share/containers/storage /var/storage/core-volumes

For storing the podman volumes on persistent memory we change the container storage config file for the core user to point the graphroot to a directory on our hetzner volume.

This volume is mounted by a system service, a mount point is created and the selinux context from the default graphroot is copied to that mount point.

At this point it should be possible to create podman volumes, pull images and run containers as the core user. We can even throw away the entire server, create a new one, attach the hetzner volume to the new server and all the volumes and images will still be available on the new server.

storage:
  files:

    # docker-compose
    - path: /usr/local/bin/docker-compose
      mode: 0755
      user:
        name: root
      group:
        name: root
      contents:
        source: {{ docker_compose_binary_url }}
        verification:
          hash: sha256-{{ docker_compose_binary_sha256 }}

    # docker-compose instanced service
    - path: /etc/systemd/user/docker-compose@.service
      mode: 0644
      contents:
        inline: |
          [Unit]
          Description=%i service with docker compose
          After=podman.socket

          [Service]
          Type=exec
          WorkingDirectory=%E/docker-compose/%i
          ExecStart=/usr/local/bin/docker-compose up --remove-orphans

          [Install]
          WantedBy=default.target

    # set DOCKER_HOST environemnt variable to the podman socket
    - path: /home/core/.config/environment.d/podman-docker-socket.conf
      mode: 0644
      user:
        name: core
      group:
        name: core
      contents:
        inline: "DOCKER_HOST=unix://{{ docker.host }}"

  links:
    # create the podman system service socket
    - path: /etc/systemd/user/sockets.target.wants/podman.socket
      target: /usr/lib/systemd/user/podman.socket
      mode: 0644
      hard: false

systemd:
  units:

    # disable podman socket
    - name: podman.socket
      enabled: false

    # disable docker socket
    - name: docker.socket
      enabled: false

It’s time to get docker-compose running. We can just download the release binary but docker-compose needs to communicate with the docker service, or in our case with the compatible podman service. Our podman service has to run in the core user session which we can achieve simply by symlinking the socket to /etc/systemd/user/sockets.target.wants/podman.socket. We can also just disable the system podman and docker sockets to avoid accidentally talking with them.

We also have to make sure docker-compose picks up the socket which should be available as /run/user/1000/podman/podman.sock. For that we create the podman-docker-socket.conf file in ~/.config/environment.d/ and set DOCKER_HOST as required.

The last piece of the puzzle is the docker-compose@.service instanced service: with it we can just throw a docker-compose.yaml file into ~/.config/docker-compose/$SERVICE and start the service with systemctl --user enable --now docker-compose@$SERVICE.

Running docker-compose services

version: "3.3"

services:
  traefik:
    image: "traefik:v2.10"
    container_name: "traefik"
    command:
      - "--log.level=WARN"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.myresolver.acme.httpchallenge=true"
      - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.myresolver.acme.email={{ admin.mail }}"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    security_opt:
      - label=disable
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "{{ docker.host }}:/var/run/docker.sock:ro"

  whoami:
    image: "traefik/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`{{ domain }}`)"
      - "traefik.http.routers.whoami.entrypoints=web"

This docker-compose.yaml file starts traefik as reverse proxy with SSL/let’s encrypt support and the whoami service which will print something when connected to. Traefik will monitors for containers with specific labels to create certificates and route traffic and thus needs access to the docker socket. For this to work we need to turn off labels with security_opt: label=disable.

After all of this it should be possible to start the service with systemctl --user enable --now docker-compose@traefik and then see the whoami output with curl {{ domain }}.

Conclusion

All in all, this setup seems to be robust and as low maintenance as possible. CoreOS will update itself. Watchtower can be installed to automatically update containers when new images become available. The podman volumes are nicely separated on a detachable, persistent storage device with snapshot and backup functionality. The most unreliable part here is installing CoreOS via the recovery mode.