My WebDAV server for Zotero archives stopped working yesterday after a
system update.
I’ve set it up so that an rclone instance serves WebDAV to a local
Unix socket, and then an nginx server proxies the requests from my
local LAN and VPN network to the local socket.
Either or both of nginx and rclone could have gone wrong. I wasted
much of my time looking at the log of rclone as it has been broken
for other reasons before. The log did include scary messages
containing Error, but it was quite misleading as the service turned
out to run just fine.
The culprit turned out to be nginx’s way of accessing the unix socket, which you
would normally specify in your config file as follows.
proxy_pass http://unix:/run/zotero/socket;
The above line proxies the http requests to the Unix socket under the
path /run/zotero/socket and requires the nginx process, which by
default runs its workers as the user http, to have both read and
write access to the socket.
After checking the journal, I saw the following error message:
2025/10/31 01:26:44 [alert] 229120#229120: *91659 writev() failed (13: Permission denied)
while sending request to upstream, ...
That’s odd, because the http user can definitely read and write to
the socket, which I was able to test by manually sending requests to
the socket.
In addition to the traditional directory access control (DAC), the
unix rwx permissions that come with each file and directory, I’ve
set up apparmor for an extra layer of protection to prevent nginx
from accessing files it doesn’t need, even if it has the right DAC
permission for it.
Still, it wouldn’t make much sense for it to break, as I’ve already
granted nginx access to the socket file in the apparmor profile as
well.
It never hurts, though, to try the aa-logprof utility, which detects
the accesses denied by apparmor that aren’t added explicitly to the
apparmor profile, and allows the user to interactively set the rule,
either to allow or reject. Sadly, the aa-logprof command yielded no
result.
I was about to give up and ended up browsing online searching for the error messages, but then saw this thread where someone else had a similar permission issue but with SELinux, another mandatory access control system like Apparmor.
I decided to revisit my apparmor setup and started reading the audit
log stored under /var/log/audit/audit.log.
Grepping the log by nginx, I finally found the issue:
type=AVC msg=audit(1762153660.596:6145): apparmor="DENIED" operation="file_perm" class="file"
info="Failed name lookup - disconnected path" error=-13 profile="/usr/bin/nginx"
name="run/zotero/socket" pid=727535 comm="nginx" requested_mask="w" denied_mask="w"
fsuid=33 ouid=960^]FSUID="http" OUID="zotero"
The info entry says “Failed name lookup - disconnected path”. Why is
the path disconnected? If we look at the name field, it says
“run/zotero/socket” but without the root slash at the ver beginning!
Note that this denial has nothing to do with relative paths. The path
that nginx tries to access misses the root, which tends to
happen with containers like docker or incus.
That explains why the rule in my profile that allows
“/run/zotero/socket” never triggered, because the detached path
without the slash simply wouldn’t match! It is still unfortunate,
though, that aa-logprof is completely silent on this kind of
detached path error.
To fix it, I need to add the attach_disconnected flag to my nginx apparmor
profile.
/usr/bin/nginx flags=(attach_disconnected) {
From now on, whenever apparmor sees a path that is not full, it
attaches the root directory in front of it. This solution, although
harmless most of the time, is sadly not ideal and has certain security
risks. You should read about the security implications here:
https://gitlab.com/apparmor/apparmor/-/issues/125.
I have ran into a similar issue with firefox lately, which started using the sandboxing tool bubblewrap whenever you try to do stuff like downloading. It’s unclear to me what caused this issue for nginx and I might try doing some debugging to identify the patch that have caused this.
A tale of caution is that if you use apparmor, you should definitely remind yourself to look into it whenever there’s a permission error. Applications get rewritten and their behaviors change and the your apparmor profile must evolve alongside as well.