systemd status=203/EXEC: Causes and Fixes

What status 203/EXEC actually means

systemd assigns its own exit statuses for failures that happen while preparing a service process. 203/EXEC, also identified as EXIT_EXEC, means the final execution call failed. The important distinction is timing: systemd loaded the unit and attempted to execute ExecStart, but the kernel did not start that program successfully.

A typical status contains both the configured command and the synthetic systemd result:

Process: 4182 ExecStart=/opt/backup/run.sh (code=exited, status=203/EXEC)
backup.service: Failed at step EXEC spawning /opt/backup/run.sh: Permission denied

The second line is more useful than the final Failed with result 'exit-code' summary. Always capture the full line because the text after spawning distinguishes missing paths, denied execution, and other kernel errors.

Inspect the effective unit first

Do not assume the file you opened is the configuration systemd loaded. Vendor units, local overrides, generated units, and drop-ins can change ExecStart. Read the effective unit and the resolved command:

systemctl cat your-service.service
systemctl show your-service.service -p FragmentPath -p DropInPaths -p ExecStart
systemd-analyze verify /etc/systemd/system/your-service.service

After changing a unit file, run systemctl daemon-reload. A restart alone does not make the manager reread unit definitions.

Cause 1: the executable path is wrong

The first token in ExecStart must resolve to an executable. Use an absolute path for custom programs and scripts. Shell features such as pipes, redirection, command substitution, and a tilde home shortcut are not automatically interpreted as they would be in an interactive shell. If you genuinely need a shell expression, invoke the shell explicitly and keep the command controlled.

ls -l /opt/backup/run.sh
namei -l /opt/backup/run.sh
file /opt/backup/run.sh

namei -l is valuable because execution requires directory traversal permission at every level, not only permission on the final file. A correct mode on the script does not help if the service identity cannot traverse /opt/backup.

Cause 2: the script has an invalid interpreter line

A script needs a valid shebang on its first line, such as #!/bin/bash or #!/usr/bin/env python3. The named interpreter must exist inside the service's runtime environment. A virtual environment, container-like root directory, or filesystem namespace can make an interpreter visible in your shell but unavailable to the service.

head -n 1 /opt/backup/run.sh
file /opt/backup/run.sh
ls -l /bin/bash /usr/bin/env

Run the script through its interpreter only as a diagnostic comparison. If /bin/bash script.sh works but direct execution does not, inspect the shebang, line endings, file mode, and mount options.

Cause 3: Windows line endings damaged the shebang

A script copied from Windows can carry CRLF line endings. The kernel may then read the interpreter as a path ending in an invisible carriage return. Depending on the environment, the message may look like a missing interpreter or an execution-format failure.

file /opt/backup/run.sh
sed -n '1l' /opt/backup/run.sh
dos2unix /opt/backup/run.sh

Rebuild or redeploy the corrected file through the normal delivery path. A one-off conversion on the server can be overwritten by the next deployment.

Cause 4: file or directory permissions block execution

Check the unit's User and Group, then test access as that identity. Avoid chmod 777. It grants unrelated write access and often hides the real directory, ownership, or policy problem.

systemctl show your-service.service -p User -p Group
sudo -u service-user test -x /opt/backup/run.sh
namei -l /opt/backup/run.sh

If DynamicUser=yes or other sandboxing features are used, read the full unit and the manager's security settings rather than assuming a permanent local account.

Why the command works in your shell but not in the service

An interactive shell can supply a current directory, login profile, aliases, functions, virtual-environment activation, and a customized PATH. A system service normally receives none of those conveniences unless the unit or its executable provides them explicitly. Confirm the command path and required environment in the effective unit. Use Environment or EnvironmentFile for controlled values, and keep secrets in a facility designed for the service rather than embedding them in a world-readable unit.

If ExecStart names a wrapper script, make the wrapper self-contained. It should use valid absolute paths where ambiguity matters, return meaningful exit codes, and avoid depending on shell startup files. Test it as the configured service user with the same working directory and environment, then compare that result with the journal.

Cause 5: a noexec mount blocks the file

Files on a filesystem mounted with noexec cannot be executed directly. This often affects temporary, removable, network, or hardened application mounts. Check the actual mount containing the file:

findmnt -T /opt/backup/run.sh
mount | grep noexec

Move the executable to an appropriate executable filesystem or correct the intended mount policy. Do not remove a hardening option without understanding why it was set.

Cause 6: SELinux or AppArmor denied execution

Traditional Unix permissions can look correct while a mandatory access-control policy still denies the operation. Check the audit trail at the failure timestamp. On SELinux systems, review AVC denials and file contexts. On AppArmor systems, look for profile denials in the journal or kernel log.

journalctl -b --no-pager | grep -i denied
ausearch -m AVC -ts recent
ls -Z /opt/backup/run.sh

Do not disable the security system as the fix. Correct the label, deployment location, or policy for the intended service behavior.

A focused recovery sequence

1. Capture full status and journal output before changing anything.
2. Read the effective unit with systemctl cat.
3. Verify the absolute path, file format, shebang, and line endings.
4. Check every directory permission and test as the configured service user.
5. Inspect mount options and SELinux or AppArmor denials.
6. Reload the manager, reset the failed state if necessary, and start the unit again.

systemctl daemon-reload
systemctl reset-failed your-service.service
systemctl start your-service.service
systemctl status your-service.service --full --no-pager

Authoritative reference

The systemd.exec manual documents systemd's process exit codes, including EXIT_EXEC. Use it together with the exact journal error, because 203 identifies the failed stage rather than one universal underlying cause.