Resize2fs, Mount, Systemd and Namespaces

[2026.03.29]

This rant started with a dumb-as-f*** “no space left on device” situation, which I naively thought I would address in matter of seconds (before starting my skiing day).

lvresize, resize2fs, right? What can be more simple and straight-forward? Seconds! Nope…

# Add 1G to my /var partition
lvresize -L +1G /dev/vg.local/lv.var

# And make its ext4 filesystem aware of it
resize2fs /dev/vg.local/lv.var
: resize2fs: Device or resource busy while trying to open /dev/vg.local/lv.var

What the hell!?! Come on, this is 2026! We’ve been able to resize ext4 online for ages!!!

Oh well…

# Identify services using /var
lsof /var

# Stop all services using /var
for service in \
  mariadb \
  rsyslog.service syslog.socket \
  systemd-journald.service systemd-journald.socket systemd-journald-dev-log.socket \
; do
  systemctl stop "${service}"
done

# Unmount /var partition
umount /var

Now, that should do it, right?

# "And make its ext4 filesystem aware of it" [bis]
resize2fs /dev/vg.local/lv.var
: resize2fs: Device or resource busy while trying to open /dev/vg.local/lv.var

… What the f***?!?…

# WTF!?! WTF!?! WTF!?!
findmnt /var
lsof /var
grep /var /etc/mtab
: (nil)

# Has the device mapper gone berseck ?
ls -l /dev/vg.local/lv.var 
: /dev/vg.local/lv.var -> ../dm-1

# Identify processes holding /dm-1
lsof | grep /dm-1
: jbd2/dm-1 386 root txt unknown /proc/386/exe

cat /proc/386/comm 
:jbd2/dm-1-8

!!!???!!! Why the heck does the kernel still run /var journaling process, even though no trace of a mounted /var can be found ???!!!???

(we all know jbd2 stands for ext4 journal, right?)

And now is the time to acknowledge those brillant HIs (Human Intelligence) on StackOverflow, who beat all those f***ing AIs to it:

Kudos to A.B: “If the filesystem was inherited in an other mount namespace, chances are it’s still mounted there. systemd can create namespaces for additional isolation.”

In-f***ing-deed!!!

# List mount namespaces
lsns -t mnt
: 4026532187 mnt 1 253 root             ├─/usr/lib/systemd/systemd-udevd
: 4026532236 mnt 1 396 systemd-timesync ├─/usr/lib/systemd/systemd-timesyncd
: 4026532315 mnt 1 498 root             └─/usr/lib/systemd/systemd-logind

# Find the culprit
nsenter -m -t 498 -- findmnt /dev/vg.local/lv.var
: /var /dev/mapper/vg.local-lv.var ext4 ro,nosuid,relatime
: /var/lib/systemd/linger /dev/mapper/vg.local-lv.var[/lib/systemd/linger] ext4 rw,nosuid,relatime
: /var/tmp /dev/mapper/vg.local-lv.var[/tmp/systemd-private-...-systemd-logind.service-.../tmp]

Qu'à cela ne tienne!

# Stop namespaced service that mounted /var
systemctl stop systemd-logind

# "And make its ext4 filesystem aware of it" [ter]
resize2fs /dev/vg.local/lv.var
: Please run 'e2fsck -f /dev/vg.local/lv.var' first.

AAAaaahhh!!! Now we’re rolling!

(no surprise here that resize2fs prompted a force e2fsck, given the origin of all these frustations was a system ungracefully running out of space; and I’m betting this is the reason why it could not be performed online, albeit with a rather misleading message)

# Check the filesystem for errors
e2fsck -f /dev/vg.local/lv.var
: (there were indeed a few minor fixes begging to be applied)

# "And make its ext4 filesystem aware of it" [quar]
resize2fs /dev/vg.local/lv.var
: SUCCESS!!!

Voilà!

Now, here is a good opportunity to understand the root cause of these headaches.

systemd-logind is the service that tracks users sessions and does plenty of nifty things along them.

One can understand hardening this service might indeed be a good idea. (like everything that deals with users and permissions, right?)

And systemd comes with plenty of clever tricks to do so, leveraging long-existing kernel mechanisms like:

cgroups (Control Groups): resources quotas, which systemd manages as Slices
capabilities: privileges, managed by systemd with ad-hoc Capability settings
namespaces: isolation (aka. “jail”), which systemd calls Sandboxing

# systemd and cgroups/slices
systemctl cat systemd-logind.service | grep -F slice
: Wants=user.slice
: After=user.slice

# systemd and capabilities
systemctl cat systemd-logind.service | grep -F Capability
: CapabilityBoundingSet=CAP_SYS_ADMIN CAP_MAC_ADMIN CAP_AUDIT_CONTROL ...

# systemd and namespaces
systemctl cat systemd-logind.service | grep -E '(Private|Privileges|Protect)'
: NoNewPrivileges=yes
: PrivateTmp=yes
: ProtectClock=yes
: ProtectControlGroups=yes
: ProtectHome=yes
: ProtectHostname=yes
: ProtectKernelLogs=yes
: ProtectKernelModules=yes
: ProtectSystem=strict

Where in particular:

PrivateTmp entails: “If enabled, a new file system namespace will be set up for the executed processes”

Voilà! (for real! I can now go skiing!)