Create and harden a custom systemd service
Use a dedicated identity, explicit paths, bounded restarts, sandboxing, logs, and a verified rollback.
Run a long-lived process without unnecessary privilege or filesystem access.
- systemd 249+, 255+
- Ubuntu Server 22.04 LTS, 24.04 LTS
- Versioned executable Install a known binary outside writable application data.
- Path inventory List every required read and write path.
- Health signal Define process and application success.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A custom long-running application installed under root-owned executable and configuration paths with a dedicated non-login service identity and one narrow writable state directory.
- A validated systemd unit with explicit dependencies, bounded restart behavior, filesystem and privilege sandboxing, structured journal access, and enablement at boot.
- A staged health, failure, restart-limit, and rollback procedure that separates process state from actual application readiness.
- systemd-analyze verify reports no unknown directive, missing executable, invalid dependency, or unit syntax error.
- The effective process runs as myapp, writes only the intended state path, cannot gain new privileges, and receives private temporary and protected home/system views.
- The service becomes active, returns the expected local health payload, logs to journald, and demonstrates one bounded automatic restart in a non-production test.
- A previous binary, configuration, and unit revision can be restored without guessing which daemon state or files changed.
Architecture
How the parts fit together
systemd PID 1 parses the unit, establishes credentials and namespaces, then execs the root-owned application as the myapp account. The application reads root-owned configuration, writes only /var/lib/myapp, exposes a loopback health endpoint, and sends stdout/stderr to journald. Restart policy reacts to failure but is rate-limited by unit-level controls.
- systemd loads and validates the unit after daemon-reload.
- PID 1 constructs filesystem, temporary-directory, kernel, home, and privilege restrictions.
- The application starts as myapp with absolute executable and configuration paths.
- systemd records lifecycle and output in the journal and restarts only qualifying failures.
- Operators verify both unit state and application health, then compare exposure and failure behavior.
Assumptions
- The example application is network-dependent, writes only /var/lib/myapp, reads /etc/myapp/config.yml, and serves health on loopback 8080.
- The executable and configuration are versioned artifacts already tested outside systemd.
- Every required file, device, socket, capability, environment variable, credential, and network dependency has been inventoried.
- Sandbox directives are supported by the target systemd release and tested against real application behavior.
- Failure injection occurs only in a non-production or approved maintenance environment.
Key concepts
- Unit ordering
- After controls startup order while Wants adds a weak dependency; neither proves an external service is actually ready.
- Restart policy
- Rules deciding when systemd starts a failed process again; it is not a substitute for correcting crash loops.
- Start limit
- A unit-level rate limit that stops repeated activation attempts within a configured interval.
- Filesystem sandbox
- A service-specific mount namespace that makes broad paths read-only or inaccessible and then opens narrow exceptions.
- NoNewPrivileges
- A kernel promise that the process and descendants cannot gain privilege through execve mechanisms such as setuid or file capabilities.
- Readiness
- Application ability to serve its intended function, which active (running) alone does not establish.
Security and production boundaries
- A low systemd-analyze security exposure score is comparative guidance, not proof that the application or complete host is secure.
- Keep executable, unit, environment, and configuration files root-owned; a service that can replace its own command defeats the identity boundary.
- Never place secrets directly in the unit because systemctl show and process environments can expose them. Use a protected credential mechanism supported by the deployment.
- Add sandbox controls incrementally and test behavior; disabling every control after one failure creates an undocumented privilege regression.
Stop before continuing if
- Stop if the executable or configuration is writable by myapp or an untrusted deployment identity.
- Stop if systemd-analyze verify reports any issue or effective directives differ from the reviewed unit.
- Stop if sandboxing blocks an undocumented dependency; inventory and add the narrow requirement instead of disabling the boundary broadly.
- Stop if health fails, logs show repeated crashes, or restart count increases unexpectedly.
- Do not enable at boot until a manual start, health check, failure test, and rollback rehearsal pass.
command
Create a service identity
Use a non-login system user.
Why this step matters
A dedicated non-login identity separates application state from administrators and prevents the service from inheriting broad personal files, groups, and interactive access.
What to understand
--system selects a system account range; --create-home provisions the declared state path and nologin rejects normal interactive sessions.
Review supplementary groups carefully because each group expands file or device access. The account should not own executable, unit, or configuration files.
System changes
- Creates the persistent myapp account, group, and /var/lib/myapp home directory.
Syntax explained
--system- Creates a system service account rather than a regular human login.
--home-dir /var/lib/myapp --create-home- Defines and creates the state directory.
--shell /usr/sbin/nologin- Rejects ordinary interactive login for this identity.
sudo useradd --system --home-dir /var/lib/myapp --create-home --shell /usr/sbin/nologin myappmyapp system user created
Checkpoint: Verify least-privilege identity
getent passwd myapp && id myapp && stat -c '%U:%G %a %n' /var/lib/myappContinue whenmyapp has nologin, only approved groups, and owns its mode-750 state directory.
Stop whenThe account already exists with unknown files, login shell, UID, or supplementary groups.
If this step fails
useradd reports that myapp already exists.
Likely causeA previous deployment or unrelated service owns the name.
getent passwd myappid myappfind / -xdev -user myapp -ls 2>/dev/null | head
ResolutionInventory the existing identity and choose a migration or unique service account; never overwrite ownership blindly.
Security notes
- nologin is one layer; filesystem ownership and systemd sandboxing still enforce runtime boundaries.
Alternatives
- Use DynamicUser and managed StateDirectory for an application designed for transient service identities.
Stop conditions
- Do not share a service account between unrelated applications.
command
Install binary and state paths
Keep binaries root-owned and mutable data isolated.
Why this step matters
Root-owned code and configuration prevent a compromised service from replacing what systemd executes, while a dedicated writable state path makes mutation explicit.
What to understand
install applies owner, group, and mode atomically to the copied binary and configuration. Validate their checksums against the reviewed release before placement.
/etc/myapp is root-controlled and /var/lib/myapp is service-controlled. Logs go to journald rather than a broad writable /var/log path in this baseline.
If the application needs cache or runtime sockets, use dedicated cache/runtime directories and expose them narrowly in the unit.
System changes
- Installs the executable and configuration root-owned and creates the service-owned persistent state directory.
Syntax explained
install -o root -g root -m 0755- Copies the executable as root-owned and executable but not service-writable.
install -d -m 0750 /etc/myapp- Creates a restricted root-controlled configuration directory.
install -m 0640 config.yml- Installs configuration readable by root and the service group but not world-readable.
install -d -o myapp -g myapp -m 0750 /var/lib/myapp- Creates the one persistent service-writable state path.
sudo install -o root -g root -m 0755 myapp /usr/local/bin/myapp && sudo install -d -o root -g myapp -m 0750 /etc/myapp && sudo install -o root -g myapp -m 0640 config.yml /etc/myapp/config.yml && sudo install -d -o myapp -g myapp -m 0750 /var/lib/myapproot:root 755 /usr/local/bin/myapp root:myapp 640 /etc/myapp/config.yml myapp:myapp 750 /var/lib/myapp
Checkpoint: Prove ownership boundary
namei -l /usr/local/bin/myapp /etc/myapp/config.yml /var/lib/myappContinue whenEvery code/config parent is root-controlled; only the state directory is myapp-writable.
Stop whenmyapp can write its executable, configuration, unit file, or any parent directory.
If this step fails
The application requires write access to /etc/myapp.
Likely causeMutable runtime state is mixed with administrator-owned configuration.
sudo -u myapp /usr/local/bin/myapp --check-config /etc/myapp/config.ymljournalctl -u myapp -n 100 --no-pager
ResolutionMove generated state to /var/lib/myapp or a managed runtime/cache path; keep policy configuration immutable to the service.
Security notes
- Never solve a write failure with recursive chown of /usr/local or /etc.
Alternatives
- Install a signed distribution package that owns binary, configuration, sysusers, and unit lifecycle.
Stop conditions
- Do not start a service that can replace its own next executable.
config
Write the hardened unit
Use absolute paths, bounded restarts, and the narrowest compatible sandbox.
Why this step matters
The unit is the executable security contract: it binds identity, dependency order, restart rate, command path, and the narrow filesystem and privilege view the process receives.
What to understand
StartLimitIntervalSec and StartLimitBurst belong in [Unit] and stop crash loops. Restart=on-failure plus RestartSec handles unexpected exits without restarting an intentional stop.
ProtectSystem=strict makes the host filesystem read-only for the service, with ReadWritePaths reopening only /var/lib/myapp. ProtectHome hides user data and PrivateTmp isolates temporary files.
NoNewPrivileges and ProtectKernelTunables reduce privilege and kernel mutation. Add other controls only after compatibility tests and documented requirements.
Use absolute ExecStart paths. Systemd does not interpret a shell command unless a shell is explicitly invoked, which this unit avoids.
System changes
- Creates /etc/systemd/system/myapp.service; no process starts until daemon-reload and start.
Syntax explained
[Unit]- Declares description, dependency ordering, and activation-rate controls.
User=myapp / Group=myapp- Drops the process to the dedicated service identity.
ExecStart=- Executes the absolute binary directly with explicit arguments.
Restart=on-failure- Restarts unexpected failures but not clean deliberate stops.
ProtectSystem=strict + ReadWritePaths=- Makes broad filesystem paths read-only and reopens one reviewed state path.
[Install] WantedBy=multi-user.target- Defines the boot target used when the unit is enabled.
/etc/systemd/system/myapp.service[Unit]
Description=My application
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60s
StartLimitBurst=5
[Service]
User=myapp
Group=myapp
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yml
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ReadWritePaths=/var/lib/myapp
[Install]
WantedBy=multi-user.targetUnit saved
Checkpoint: Review the complete unit contract
systemd-analyze cat-config systemd/system/myapp.serviceContinue whenThe effective unit contains the intended identity, command, restart limits, sandbox, state path, and boot target with no unexpected drop-in.
Stop whenAn unknown drop-in overrides policy or any required path/capability remains undocumented.
If this step fails
systemd-analyze verify reports that a directive is in the wrong section.
Likely causeA service, unit, or install option was placed under an incompatible heading.
systemd-analyze verify /etc/systemd/system/myapp.servicesystemd --version
ResolutionMove the directive according to the official systemd.unit/service/exec documentation and validate again.
Security notes
- Do not use Environment= for long-lived secrets; unit properties are observable by administrators and often other local tooling.
Alternatives
- Use LoadCredential and StateDirectory on supported systemd releases.
Stop conditions
- No unit with unreviewed ExecStart, writable code, or broad privilege proceeds.
verification
Verify unit syntax
Resolve every unknown directive, dependency, and missing path.
Why this step matters
Static verification catches syntax, missing executables, invalid dependencies, and unsupported directives before PID 1 replaces the loaded unit definition.
What to understand
systemd-analyze verify parses the unit and related dependencies. No output with exit zero is the expected evidence.
Validation does not prove runtime permissions, network dependencies, application configuration, or health, so it gates rather than replaces later tests.
Run the target host's systemd-analyze because available directives differ by release.
System changes
- No persistent change; parses the candidate unit and referenced paths.
Syntax explained
systemd-analyze verify- Checks unit files for syntax, dependencies, executable paths, and directive validity.
/etc/systemd/system/myapp.service- Validates the exact candidate file before reload.
sudo systemd-analyze verify /etc/systemd/system/myapp.serviceNo output; exit code 0
Checkpoint: Require zero validation findings
sudo systemd-analyze verify /etc/systemd/system/myapp.service; echo $?Continue whenNo diagnostic output and exit status 0.
Stop whenAny warning, error, missing path, or unsupported option appears.
If this step fails
The verifier cannot find /usr/local/bin/myapp.
Likely causeDeployment order or path differs from ExecStart.
namei -l /usr/local/bin/myappfile /usr/local/bin/myapp
ResolutionInstall the reviewed artifact at the declared path or update the unit intentionally; do not suppress the finding.
Security notes
- Validation should run in CI and again on the target release.
Alternatives
- Use systemd-run for an isolated transient compatibility experiment, then still validate the persistent unit.
Stop conditions
- Never daemon-reload a candidate that fails verify.
verification
Review security exposure
Use the score as a comparison aid, not a security guarantee.
Why this step matters
The exposure report highlights missing systemd sandbox controls and gives reviewers a repeatable comparison between revisions without claiming application-level security.
What to understand
systemd-analyze security inspects service directives and calculates an exposure score. A lower score usually means more kernel-enforced restrictions.
Some recommendations are incompatible with legitimate device, network, namespace, or filesystem needs. Document each exception with runtime evidence.
Compare the score and detailed rows before and after a revision; do not optimize a number by breaking the service.
System changes
- No persistent change; evaluates the effective service sandbox configuration.
Syntax explained
systemd-analyze security- Analyzes selected service hardening properties and prints exposure details.
myapp.service- Targets this unit rather than all loaded services.
systemd-analyze security myapp.serviceOverall exposure level: 3.1 OK
Checkpoint: Record exposure and exceptions
systemd-analyze security myapp.serviceContinue whenA saved detailed report shows the reviewed controls and documented exceptions; score is compared with the previous revision.
Stop whenA high-impact missing control has no tested operational justification.
If this step fails
The score is unchanged after adding a control.
Likely causeA drop-in overrides it, the directive is unsupported, or the scoring model weights another property.
systemctl cat myappsystemctl show myapp -p NoNewPrivileges -p ProtectSystem -p ProtectHome
ResolutionInspect effective properties and behavior rather than adding arbitrary directives for the score.
Security notes
- The report does not scan application code, credentials, network policy, or host patch state.
Alternatives
- Combine with application threat modeling, syscall observation, and host security policy.
Stop conditions
- Do not waive a boundary solely because the process currently needs broad access.
command
Start and inspect
Reload units, start the service, and read recent logs.
Why this step matters
Reload, enable, start, unit-state inspection, journal review, and listener checks together prove that PID 1 loaded the intended revision and the process survived initialization.
What to understand
daemon-reload makes systemd reread unit files. enable creates boot-target links; --now starts immediately. These are distinct state changes.
status is a snapshot and may truncate logs. journalctl supplies recent service-specific evidence; ss confirms the intended loopback listener and owning process.
Record FragmentPath, DropInPaths, MainPID, User, and the binary checksum so the running revision is attributable.
System changes
- Reloads systemd manager configuration, enables myapp at boot, and starts the service process.
Syntax explained
daemon-reload- Reloads unit definitions without restarting every service.
enable --now- Adds boot enablement and starts myapp immediately.
--no-pager status- Prints current state without interactive paging.
journalctl -u myapp -n 50- Reads the fifty most recent unit log records.
sudo systemctl daemon-reload && sudo systemctl enable --now myapp && systemctl --no-pager status myapp && journalctl -u myapp -n 50 --no-pagerActive: active (running)
Checkpoint: Confirm the intended running revision
systemctl show myapp -p ActiveState -p SubState -p MainPID -p FragmentPath -p DropInPaths && ss -lntp 'sport = :8080'Continue whenactive/running, expected unit path and drop-ins, nonzero MainPID, and only the intended loopback listener.
Stop whenStartup is degraded, logs contain repeated errors, listener is public, or effective unit differs.
If this step fails
The service exits immediately with status 1.
Likely causeApplication configuration, permissions, dependencies, or sandbox access failed during initialization.
systemctl status myapp --no-pagerjournalctl -u myapp -n 100 --no-pagersudo -u myapp /usr/local/bin/myapp --check-config /etc/myapp/config.yml
ResolutionCorrect the specific preflight or permission and restart once; do not disable the entire sandbox.
Security notes
- Inspect whether journal records contain secrets before forwarding them.
Alternatives
- Start without enabling, complete all verification, then enable as a separate approved step.
Stop conditions
- Do not proceed from active state alone when logs or listener disagree.
verification
Test application behavior
Confirm behavior, not only process state.
Why this step matters
A local application request verifies listener, routing, handler, dependency initialization, and response semantics that systemd process state cannot observe.
What to understand
--fail returns nonzero for HTTP errors, --silent suppresses progress, and the explicit loopback URL prevents accidentally probing another environment.
The response includes status and version so the test proves the intended release rather than any process on the port.
Add dependency checks carefully: a health endpoint should be fast, bounded, and meaningful without causing writes.
System changes
- No persistent change; sends one read-only local health request.
Syntax explained
curl --fail- Returns a nonzero status for HTTP 400 and 500 responses.
--silent- Suppresses progress output while retaining response content.
127.0.0.1:8080/health- Targets the intended local-only readiness endpoint.
curl --fail --silent http://127.0.0.1:8080/health{"status":"ok","version":"1.2.3"}Checkpoint: Require application readiness
curl --fail --silent http://127.0.0.1:8080/healthContinue whenThe payload reports status ok and the reviewed application version.
Stop whenHTTP fails, version is wrong, response is slow, or required dependency status is degraded.
If this step fails
curl gets connection refused while systemd says active.
Likely causeThe application has not bound yet, uses another address/port, or its main process delegates incorrectly.
ss -lntp 'sport = :8080'journalctl -u myapp -n 100 --no-pagersystemctl show myapp -p MainPID -p Type
ResolutionCorrect startup/readiness semantics and listener configuration; do not expose another port as a shortcut.
Security notes
- Keep administrative health detail on loopback or an authenticated management network.
Alternatives
- Use a Unix-socket health command or sd_notify readiness for non-HTTP services.
Stop conditions
- No traffic promotion or boot enablement sign-off without readiness.
decision
Test bounded failure behavior
In non-production, terminate once and confirm one restart rather than a loop.
Why this step matters
A controlled non-production failure proves that on-failure restarts once, preserves logs, returns to readiness, and remains bounded rather than entering an invisible crash loop.
What to understand
SIGKILL intentionally creates an abnormal termination that Restart=on-failure should handle. Run only in an approved test because active requests are interrupted.
After RestartSec, NRestarts should increment exactly once and ActiveState return to active. Repeat the health check and inspect the old/new invocation logs.
StartLimitBurst and interval protect the host from rapid retries. Testing the full limit needs a disposable environment and should not be automated against production.
System changes
- Kills the current test process, causes systemd to create one replacement process, and increments restart counters.
Syntax explained
systemctl kill --signal=SIGKILL --kill-who=main- Terminates only the unit's main process with an abnormal signal for the controlled test.
Restart=on-failure- Causes systemd to restart after the qualifying signal.
systemctl show -p- Reads exact restart, result, and active-state properties.
sudo systemctl kill --signal=SIGKILL --kill-who=main myapp && sleep 6 && systemctl show myapp -p NRestarts -p Result -p ActiveStateNRestarts=1 Result=success ActiveState=active
Checkpoint: Prove one bounded recovery
systemctl show myapp -p NRestarts -p Result -p ActiveState && curl --fail --silent http://127.0.0.1:8080/healthContinue whenNRestarts increases by one, state is active, health succeeds, and logs show one old plus one new invocation.
Stop whenThe service does not restart, loops repeatedly, loses state, or health remains unavailable.
If this step fails
The unit remains failed after SIGKILL.
Likely causeRestart policy, start limits, or a replacement startup failure prevented recovery.
systemctl show myapp -p Restart -p Result -p NRestarts -p StartLimitBurst -p StartLimitIntervalUSecjournalctl -u myapp --since '-5 minutes' --no-pager
ResolutionRestore service health manually, correct policy or startup failure, and repeat only in a safe test window.
Security notes
- Failure injection is an availability-impacting change and belongs in a test or approved maintenance environment.
Alternatives
- Exercise the same unit in a disposable VM or integration environment.
Stop conditions
- Never test repeated crash-loop limits on an unapproved production service.
Finish line
Verification checklist
systemd-analyze verify /etc/systemd/system/myapp.serviceExit code 0.systemctl is-active myappReturns active.curl --fail http://127.0.0.1:8080/healthReturns the healthy payload.Recovery guidance
Common problems and safe checks
The unit fails with status 203/EXEC.
Likely causeExecStart path is missing, non-executable, has a bad interpreter, or is blocked by mount/security policy.
systemctl status myapp --no-pagernamei -l /usr/local/bin/myappfile /usr/local/bin/myappjournalctl -u myapp -n 100 --no-pager
ResolutionInstall the reviewed executable with correct interpreter and root ownership, verify paths, then start again without loosening unrelated sandboxing.
The service starts without sandboxing but fails with ProtectSystem=strict.
Likely causeThe application writes to an unlisted path or expects mutable configuration, cache, log, or runtime directories.
journalctl -u myapp -n 100 --no-pagersystemctl show myapp -p ReadWritePaths -p ProtectSystemstrace -f -e trace=file -o /tmp/myapp.files /usr/local/bin/myapp --config /etc/myapp/config.yml
ResolutionClassify the required write, provision a dedicated state/cache/runtime directory, and add the narrowest explicit path.
systemd reports active but the health endpoint fails.
Likely causeThe process exists but initialization, dependency connection, listener bind, or application state is not ready.
systemctl status myapp --no-pagerss -lntp 'sport = :8080'journalctl -u myapp -n 100 --no-pager
ResolutionCorrect application readiness and dependencies; do not route traffic or mark deployment successful from process state alone.
The service enters start-limit-hit.
Likely causeRepeated failures exceeded StartLimitBurst within StartLimitIntervalSec.
systemctl show myapp -p Result -p NRestarts -p StartLimitBurst -p StartLimitIntervalUSecjournalctl -u myapp --since '-10 minutes' --no-pager
ResolutionFix the underlying failure, verify manually, then use systemctl reset-failed once; do not loop reset-failed in automation.
Reference
Frequently asked questions
Should every service use the same hardening directives?
No. Begin from a strong baseline, inventory actual requirements, and document narrow exceptions. Kernel, device, network, and filesystem needs differ.
Why use Restart=on-failure instead of always?
A deliberate clean stop should normally remain stopped, while unexpected nonzero exits and signals can restart. The correct policy depends on service semantics.
Does After=network-online.target guarantee the network dependency is reachable?
No. It orders startup after the target; application retries and an explicit readiness check still handle DNS, routing, TLS, and remote service availability.
Recovery
Rollback
Restore the prior unit and executable revision.
- Stop and disable the new unit.
- Restore the prior unit and executable.
- Run daemon-reload and systemd-analyze verify.
- Start and verify the previous version.
Evidence