Troubleshooting

When software that works on another distribution does not work on Sulka, a hardening measure is usually responsible. This page maps the symptoms you are likely to see onto the measure that causes them.

Narrowing It Down

Two checks narrow things down quickly.

First, build the same image with the development configuration described in Building With Sulka and disable SELinux and firewall to see whether the problem goes away:

setenforce 0
nft flush ruleset

Second, look at the logs. Most of the hardening in Sulka reports refusals there:

# Kernel log: module loading failures, failures caused by kernel hardening, read-only rootfs failures, etc.
dmesg
# Audit log: SELinux related failures, look for lines with denial messages
grep -i avc /var/log/audit/audit.log
# System log: User space failures, login failures, access control failures, etc.
cat /var/log/syslog
# On systemd systems, check journalctl as well
journalctl

SELinux Denials

Symptoms. A service fails to start for no obvious reason. A file cannot be opened even though the ownership and permission bits are correct. Something works when run by hand but not when run as a service.

Check whether SELinux is enforcing, and look for denials:

getenforce
sestatus
grep -i avc /var/log/audit/audit.log
# Early SELinux denials may go to dmesg instead of audit log
dmesg | grep -i avc

Denials are recorded as AVC messages naming the source context, the target context and the permission that was refused.

Confirming the cause. On an image built with development mode, put SELinux into permissive mode temporarily and retry:

setenforce 0

If the problem disappears, it is a policy problem rather than a bug in your software. Note that this only works on an image built with development mode enabled, as described in Development Mode. On a production image the kernel refuses the request, and the boot parameter selinux=0 has no effect either.

Fixing it. Your own services will usually need a policy module of their own. The audit2allow tool generates a starting point from the denials you have collected, but it is only installed by the development configuration, not on a default image. Treat its output as a draft: it grants exactly what was asked for, which is not always what you should allow.

grep -i avc /var/log/audit/audit.log | sudo audit2allow

See SELinux for the policy Sulka ships and the patches applied to it.

Network Traffic Is Blocked

Symptoms. Nothing can reach the device, and the device cannot reach anything either. Outgoing connections, DNS lookups and package downloads all fail.

This is the default firewall doing its job. Sulka ships a rule set that drops all traffic in every direction until you configure it.

nft list ruleset

To confirm this, you can try flushing the firewall:

nft flush ruleset

Fixing it. Select a template that suits you with SULKA_NFTABLES_CONF, or ship your own rule set. See Firewall for the bundled templates.

Write Failures On The Root File System

Symptoms. Writes fail with a read-only file system error, or an application that expects to create files at startup fails immediately.

The root file system is mounted read-only by default:

mount | grep ' / '

Fixing it. Write to one of the designated writable locations instead (check the /var tmpfs mounts from mount output), or arrange one for your application. Read-Only Root File System covers the options, including writable partitions, temporary file systems, bind mounts and symlinks. Note that the overlayfs approach conflicts with SELinux, which is enabled by default.

Cannot Execute A Script From /tmp, /run Or /var

Symptoms. A script or binary with the execute bit set fails with a permission error, specifically when it lives in /tmp, /run, /var/lib or /var/cache.

Sulka mounts the volatile file systems with noexec, along with nodev and nosuid:

mount | grep -E ' /tmp | /run | /var/volatile | /var/lib | /var/cache '

/var/lib and /var/cache are affected as well, which is less obvious. With the read-only root file system they are bind mounted from /var/volatile, and they inherit the mount options from their source, so the noexec on /var/volatile carries over to both.

/tmp reaches the same end result through two different routes, depending on the init manager. On systemd systems /tmp is its own tmpfs, mounted by systemd’s tmp.mount unit, which Sulka patches to add noexec. On sysvinit systems /tmp is a symbolic link into /var/volatile, so it inherits the noexec from that mount.

Fixing it. Prefer moving the executable somewhere persistent and running it from there. If that is not possible, setting SULKA_HARDEN_MOUNTS to 0 disables the mount option hardening, but you then take on setting safe options yourself. See the description of that variable in Configuration Variables.

ps Shows Only Your Own Processes

Symptoms. ps and top show almost nothing, and tools that inspect other processes report that they do not exist.

/proc is mounted with hidepid=2, which hides other users’ processes. Run the command through sudo to see the whole system, or remount with hidepid=0 to verify. This is also controlled by SULKA_HARDEN_MOUNTS.

A Kernel Module Will Not Load

Symptoms. modprobe or insmod fails, and the kernel log mentions a missing or invalid signature.

Module signing is enforced by default, so the kernel refuses any module it cannot verify against a trusted key. Modules built through their own Yocto recipes are signed during the build. Prebuilt binary modules and anything compiled by hand are not.

Fixing it. Sign the module with the same key pair the kernel trusts, as described in Signing External Modules. If a correctly signed external module still fails to load, kernel configuration CONFIG_RANDSTRUCT_FULL may be the cause, as it changes internal structure layout.

If you cannot sign your modules at all, SULKA_ENABLE_MODULE_SIGNING can be disabled to disable the enforcement, at a real cost to the security of the system.

A Kernel Feature Is Missing

Symptoms. A syscall returns ENOSYS, a binary refuses to run, a file system or protocol is unavailable, or a library reports that a kernel facility is not supported.

Sulka removes a substantial amount of kernel functionality to cut the attack surface. The options below are the ones most likely to affect ordinary software:

Disabled option	What stops working
`CONFIG_COMPAT`	32-bit binaries on a 64-bit system
`CONFIG_USER_NS`	User namespaces, and with them containers and rootless sandboxing
`CONFIG_IO_URING`	Applications built on `io_uring`
`CONFIG_AIO`	The POSIX asynchronous I/O interface
`CONFIG_COREDUMP`	Core dumps, in addition to the process limits that also disable them
`CONFIG_DEBUG_FS`	`debugfs`, and the tools that rely on it
`CONFIG_FTRACE`, `CONFIG_KPROBES`, `CONFIG_UPROBE_EVENTS`	Kernel tracing, and tooling built on it
`CONFIG_CHECKPOINT_RESTORE`	Checkpoint and restore, such as CRIU
`CONFIG_MAGIC_SYSRQ`	The SysRq key combinations
`CONFIG_IP_SCTP`, `CONFIG_TIPC`, `CONFIG_RDS`, `CONFIG_MPTCP`	Those network protocols

You can try disabling the kernel hardening to see if that helps by adding the following to your build configuration:

SULKA_HARDEN_KERNEL = "0"

Fixing it. The cuts live in the kernel metadata of meta-sulka-kernel, under the sulka-cut-attack-surface feature. Re-enable what you genuinely need with a kernel configuration fragment of your own, from your own layer, rather than by editing the Sulka metadata. Turning the whole feature off is possible with SULKA_HARDEN_KERNEL, but that gives back far more attack surface than you probably intend.

Debuggers And Profilers Do Not Work

Symptoms. gdb cannot attach to a running process, strace and ltrace fail immediately, perf reports insufficient permissions, and no core dump appears after a crash.

This is several measures acting together:

kernel.yama.ptrace_scope is set to 3, which disables ptrace entirely. No process can attach to another, and a process cannot mark itself traceable either. This stops attaching debuggers and tracers outright, rather than merely restricting them.
kernel.perf_event_paranoid is set to 3, which restricts performance monitoring.
kernel.unprivileged_bpf_disabled is set to 1, so unprivileged BPF is unavailable.
Core dumps are disabled both in the kernel and through the process limits.
Kernel build configuration affects these as well.

Fixing it. These are runtime settings in the sysctl configuration that meta-sulka-kernel installs, so they can be overridden from your own layer for development images. ptrace_scope in particular cannot be relaxed by simply raising a limit at runtime once it has been set, so plan to build a development image rather than expecting to loosen it on a shipped device.

You can also try disabling the kernel hardening to see if that helps by adding the following to your build configuration:

SULKA_HARDEN_KERNEL = "0"

Consider whether you can reproduce the problem on a development host instead. The restrictions exist precisely because these interfaces are powerful, and a device that permits debugger attachment in the field is a device that permits it for an attacker too.

The Build Fails

A few build failures are Sulka-specific rather than ordinary Yocto problems:

Missing module signing keys. Module signing is enabled by default and the build fails without keys. See Module Signing.
A dangling bbappend. Sulka bbappends wildcard the patch release of the recipe version they target, so a version outside that range fails the build rather than silently dropping the hardening. Match the Yocto release that Sulka targets.
The fstab hardening did not apply. If you ship your own fstab, the hardening may not match it, and the build fails rather than leaving the mounts unhardened. Set SULKA_HARDEN_MOUNTS to 0 and harden your own fstab instead. See meta-sulka-distro for guidance.
Many kernel configuration warnings. Disabling kernel modules converts many options from modules to built-ins, which Yocto reports as a mismatch between the requested and resulting configuration. See Disabling Kernel Modules.