Recommendations

  • Target environment: these patterns assume bash ≥4.4. macOS ships bash 3.2 by default; the getopts-only and explicit-checking paths still work there, but inherit_errexit, mapfile, and read -r -a do not. Linux distributions and modern containers (Debian 11+, Ubuntu 20.04+, Alpine with bash package) ship 5.x.
  • Prefer explicit error checking, but acknowledge that strict-mode + ERR-trap + inherit_errexit is a legitimate alternative on bash 4.4+ — the position is genuinely contested. Wooledge BashFAQ recommends against set -e for non-trivial scripts because it is silently suppressed in conditionals, pipelines, and (pre-4.4) command substitution. Aaron Maxwell and many practitioner guides recommend the strict-mode pattern. On bash 4.4+, shopt -s inherit_errexit closes the command-substitution gap, narrowing the suppression contexts from four to three (conditionals, pipelines, local var=$(cmd) still leak). Explicit cmd || die is the more conservative default; strict-mode-plus is defensible when the team commits to its rules. See set -e and the strict-mode question.
  • Tradeoff: strict-mode (set -euo pipefail + inherit_errexit + ERR trap) buys terse scripts at the cost of remaining silent failure modes (conditionals, pipelines, local-decl); explicit checking buys correctness at the cost of verbosity. For a sysadmin agent that values reproducible execution over brevity, explicit checking wins by default. See set -e and the strict-mode question.
  • For container entrypoints, use exec form ENTRYPOINT or exec "$@" at script tail so the application becomes PID 1 directly and receives signals. Only fall back to the app & ; trap ... ; wait wrapper pattern when the script genuinely needs cleanup logic after the app exits. When monitoring or cgroup pid-limits are tight, prefer exec — Pattern B leaves a shell PID in the cgroup that affects pids.max accounting and shows up in ps. See Signal handling and bash as PID 1.
  • Use flock(1) with a numeric file descriptor (exec 9>/var/lock/foo; flock -n 9 || exit) for single-instance enforcement. Do not use PID files for mutual exclusion; do not place flock files on NFS pre-2.6.12. See Idempotency and mutual exclusion.
  • For structured logging from shell, use logger --journald when writing multiple structured fields, systemd-cat -t TAG -p PRIORITY for whole-script redirection, and stderr for errors regardless. See Structured logging from shell.
  • Run shellcheck with -x (external-sources) and # shellcheck source=path directives to follow sourced files; commit a .shellcheckrc rather than relying on inline disables. SC2086/SC2068/SC2206 are the same word-splitting failure in three syntactic positions; SC2128 is the related but distinct array-default bug ($arr silently means ${arr[0]}). See shellcheck beyond the basics.
  • Decision point: for getopts vs GNU getopt, the tradeoff is portability vs long-flag support. Use getopts when short flags suffice and the script must run on macOS/BSD; use GNU getopt (with the getopt --test capability check) when long flags improve user-facing scripts. See Input validation and argument parsing.

Summary

Bash’s defaults are wrong for production; the patterns that work are the ones that respect bash’s actual semantics around error propagation, signals, and file descriptors — not the semantics people assume. Each section below documents a specific gap between “what most scripts do” and “what bash actually does,” with the canonical pattern that closes the gap.

  • set -e is silently suppressed in conditionals, pipelines (except last stage), command substitution subshells (pre-bash-4.4 or without inherit_errexit), local var=$(cmd) declarations, and process substitution — the expert community is split, with Wooledge BashFAQ recommending explicit error checking and Aaron Maxwell’s strict-mode pattern still widely adopted (set -e and the strict-mode question).
  • Bash as PID 1 does not process signals while a foreground child runs — the wrapper must either exec the app or run it in the background with trap + wait (Signal handling and bash as PID 1).
  • flock(2) locks attach to the open file description, not the file descriptor — fork() and dup() share the same lock, and the lock releases when the last FD from that open() closes (Idempotency and mutual exclusion).
  • logger --journald and systemd-cat are not interchangeable: --journald ignores -p because priority is embedded in the structured-field stream, while systemd-cat wraps whole stdout/stderr with a single priority (Structured logging from shell).

Key Findings

  • set -e rules change between bash versions and the POSIX definition is “extremely slippery” — relying on it for correctness in long-lived scripts is fragile (BashFAQ/105). [consensus]
  • The community is split on whether to use strict mode at all. Wooledge BashFAQ/105 recommends against set -e for non-trivial scripts and advocates explicit error checking. Aaron Maxwell’s “unofficial strict mode” (set -euo pipefail; IFS=$'\n\t') is widely adopted and advocates the opposite. The Google Shell Style Guide takes no firm position. The fault line is real engineering disagreement, not consensus (BashFAQ/105; redsymbol.net; Google Shell Style Guide). [contested]
  • local var=$(cmd) masks the exit status of cmd because local itself returns 0, defeating set -e on the assignment (BashFAQ/105). Declare the variable on one line and assign on the next when the exit code matters. [consensus]
  • set -e and ERR trap are equally suppressed in conditional contexts — switching to ERR trap solves the silent forcing problem (no context on failure) but does not solve the suppressed-in-conditionals problem (BashFAQ/105; disconnected.systems). [consensus]
  • (( i++ )) as a standalone statement exits under set -e when i=0 — bash sets the exit status to 1 (false) when the arithmetic expression evaluates to 0, and set -e triggers on the non-zero exit (Google Shell Style Guide). Use (( ++i )) (pre-increment) or i=$((i+1)) instead. [consensus]
  • Shell form ENTRYPOINT makes the shell PID 1; exec form makes the app PID 1 — only exec form delivers signals to the app on docker stop (CloudBees; Suraj Deshmukh). [consensus]
  • Bash does not process trapped signals while a foreground child runs — the script must either background the child (app & ; pid=$! ; wait "$pid") or exec to replace itself with the app (CloudBees; Suraj Deshmukh). [consensus]
  • flock locks are advisory only — any process can bypass them by not calling flock; they enforce nothing at the filesystem layer (flock(2) man page). [consensus]
  • flock does not detect deadlock — two processes each holding a lock and waiting for the other will hang indefinitely (flock(2) man page). [consensus]
  • flock on NFS pre-kernel 2.6.12 is local-only (no cross-host coordination); since 2.6.12 it is emulated as an fcntl whole-file byte-range lock (flock(2) man page). [consensus]
  • logger --journald ignores the -p flag — priority must be embedded as a PRIORITY=N field in the structured-data stdin (logger(1) man page). A common gotcha when migrating from logger -p syntax.
  • SC2086, SC2068, SC2206 are the same word-splitting failure in three syntactic positions — unquoted expansion subject to IFS-split and glob-expansion, whether bare ($var), positional/array spread ($@), or array literal (array=( $var )). SC2128 is a related-but-distinct array-default bug: a bare $arr references only ${arr[0]} regardless of array size, which is a near-certain bug but not a word-splitting issue (shellcheck wiki SC2086 and siblings). [consensus]
  • Pipeline stages run in subshells — variable modifications inside cmd | while read x; do FOO=$x; done do not persist to the caller; use process substitution while read x; do FOO=$x; done < <(cmd) to keep the loop in the parent shell (BashPitfalls). [consensus]
  • Command substitution strips trailing newlinesvar=$(cmd) followed by echo "$var" re-adds exactly one trailing newline, which is rarely what you want for multi-line output (BashPitfalls). [consensus]

Open Questions

  • When is set -e genuinely the better choice over explicit checking? The Wooledge position is “never for production,” but Maxwell’s strict-mode and the set -euo pipefail + inherit_errexit + ERR-trap pattern are widely used in practice. Is there a defensible decision rule — perhaps “always for scripts under N lines” or “always when the team commits to never using conditionals on commands that may fail” — that would let this stop being a religious debate?
  • For long-running daemons launched from shell wrappers, what is the precise cgroup/ps cost of Pattern B (app & ; wait) versus exec? Pattern B leaves a shell PID in the cgroup that consumes a slot against pids.max and shows up in ps. The recommendation already says to prefer exec when those limits are tight, but the threshold at which Pattern B’s overhead actually matters in practice is not documented.
  • What is the right policy for # shellcheck disable=SCxxxx inline disables? The wiki documents the mechanism but offers no guidance on when an inline disable is acceptable vs when the code should be rewritten. A vault-wide convention would help.
  • At what scale does a .shellcheckrc with external-sources=true start producing false positives from sourced files that genuinely don’t exist at lint time (e.g., generated config)? The escape hatch is # shellcheck source=/dev/null but the threshold for when to reach for it is not documented.

Details

set -e and the strict-mode question

set -e is a leaky abstraction over error propagation; the patterns that survive production either work around its leaks or replace it with explicit checks. The “unofficial strict mode” (set -euo pipefail; IFS=$'\n\t') is widely cited but the canonical Bash reference (BashFAQ/105) recommends against it for non-trivial scripts.

What each flag actually does

FlagCatchesDoes NOT catch
set -eTop-level commands with non-zero exit, in non-conditional contextsAnything in if/&&/`
set -uReference to unset variables ($undefined)Set-but-empty variables (var=); requires ${var:?} for that
set -o pipefailNon-zero exit anywhere in a pipeline (returns rightmost non-zero stage)Surfaces SIGPIPE-killed upstream commands as pipeline failures even when the downstream command intentionally closed early (e.g., head, grep -q) — a false-failure foot-gun. yes | head -1 exits 141 under pipefail, not 0.
shopt -s inherit_errexit (bash ≥4.4)Propagates set -e into command-substitution subshells ($(...))Conditionals, pipelines, local var=$(cmd), process substitution — closes one of four suppression contexts, not all
IFS=$'\n\t'Space-splitting of filenames in unquoted for-loopsAnything else word-splitting can do — quote your variables

The local var=$(cmd) trap. Because local is itself a builtin that returns 0, the exit status of cmd is discarded and set -e cannot fire. This is the single most common silent-failure pattern in shell scripts that think they are running strict-mode.

# WRONG — exit code of cmd is masked, set -e cannot trigger
some_func() {
  local result=$(cmd_that_might_fail)
  echo "$result"
}
 
# RIGHT — separate declaration from assignment
some_func() {
  local result
  result=$(cmd_that_might_fail)
  echo "$result"
}

Legitimate workarounds when you must run a command that may fail

# Pattern 1: explicit allow-failure
some_command || true
 
# Pattern 2: capture exit code for branching
some_command && rc=0 || rc=$?
case $rc in
  0) ;;
  3) echo "expected condition" >&2 ;;
  *) echo "unexpected: $rc" >&2; exit "$rc" ;;
esac
 
# Pattern 3: early-return from function
some_command || return

The ERR trap as alternative. Some practitioners prefer dropping set -e and using an ERR trap that captures context on failure:

set -uo pipefail
IFS=$'\n\t'
trap 's=$?; echo "$0: error on line $LINENO: $BASH_COMMAND" >&2; exit $s' ERR

This solves the silent forcing problem — without the trap, set -e just exits with no indication where the failure occurred. The trap captures $LINENO and $BASH_COMMAND for diagnostics. But the ERR trap is subject to the same conditional-context suppression as set -e — it does not fire inside if/&&/||/pipelines either (disconnected.systems).

The contested recommendation. BashFAQ/105 advocates “don’t use set -e, add your own error checking” on the grounds that any non-trivial script will eventually need a command that may fail in a conditional, forcing the author to either disable set -e, accept silent failure, or work around it with || true. Aaron Maxwell’s “unofficial strict mode” advocates the opposite — that the brevity and crash-on-error default catch more bugs than the suppression contexts cause. Both are defensible; this is genuine engineering disagreement, not consensus. The vault’s working assumption (explicit checking) is one position among two legitimate ones.

inherit_errexit (bash ≥4.4) partially closes the command-substitution gap. Setting shopt -s inherit_errexit propagates set -e into command-substitution subshells, so x=$(false; echo unreachable) aborts the parent script instead of silently completing. This is the single most useful strict-mode addition since bash 4.4 — pair it with set -euo pipefail. It does not address the other three suppression contexts: conditionals (if cmd; then ...), pipelines (set -o pipefail is separate), or local var=$(cmd) (the local builtin’s own zero exit status still masks the inner command). It also does not enter process substitutions. Treat inherit_errexit as a partial mitigation that makes the strict-mode position more defensible without making it complete.

(( i++ )) and set -e. The expression (( i++ )) evaluates i (yielding its current value as the result of the arithmetic expression), then increments. Bash sets the exit status to 1 (false) when the evaluated value is 0, and 0 (true) otherwise. Under set -e, exit status 1 aborts the script. So i=0; (( i++ )) under strict mode exits the script; i=1; (( i++ )) does not. Workaround: use (( ++i )) (pre-increment increments first and yields the new value, so the expression is never 0 until i wraps), i=$((i+1)) (bash treats this as a plain assignment), or ((i++)) || true to mask the exit code (Google Shell Style Guide).

Why IFS hardening matters. The default IFS=$' \t\n' makes unquoted for f in $files split on spaces — filenames with spaces become two iterations. Setting IFS=$'\n\t' removes space as a separator, so for f in $files splits only on newlines and tabs. This is a partial mitigation; the real fix is quoting (for f in "$@" with arrays) and never letting unquoted variables undergo word-splitting in the first place (redsymbol.net strict mode; BashPitfalls).

Signal handling and bash as PID 1

Bash was not designed to be PID 1, and the gap shows up at signal-delivery time. A wrapper script that does not understand bash’s foreground-blocking signal model will swallow SIGTERM from docker stop and the container will be SIGKILL’d after the grace period — visible as exit code 137 instead of the expected 143.

The PID 1 problem in containers. When ENTRYPOINT is the shell form (a string, parsed by /bin/sh -c), the shell becomes PID 1 and the app is a child. docker stop sends SIGTERM to PID 1 (the shell), not to the app. The shell does not forward the signal unless explicitly trapped, and even when trapped, bash does not process trapped signals while a foreground process is running (CloudBees signal trapping; Suraj Deshmukh on exec).

ENTRYPOINT formExamplePID 1Signal behavior
Shell formENTRYPOINT /app/start.sh/bin/sh running the scriptApp receives no signals from docker stop
Exec formENTRYPOINT ["/app/start.sh"]The script’s shellSame problem unless the script execs the app
Exec form + execScript ends with exec /app/binThe app itselfApp receives signals directly

The two correct patterns

# Pattern A: exec — replace the shell with the app entirely
#!/bin/bash
set -uo pipefail
 
# ... do setup, env munging, config rendering ...
 
exec /app/bin "$@"   # shell is now gone; app is PID 1
# Pattern B: wrapper with signal forwarding — only when cleanup must run AFTER app exits
#!/bin/bash
set -uo pipefail
 
cleanup() {
  echo "cleanup running" >&2
  rm -rf /tmp/work
}
 
shutdown() {
  echo "forwarding SIGTERM to $pid" >&2
  kill -SIGTERM "$pid" 2>/dev/null || true
}
 
trap shutdown SIGTERM SIGINT
trap cleanup EXIT
# Workload-specific: add SIGHUP if you want the wrapper to forward terminal
# disconnects, and SIGQUIT if you want graceful handling of Ctrl-\.
 
/app/bin "$@" &
pid=$!
 
# First wait: interrupted by the SIGTERM trap, returns 128+SIG immediately
# (this is the signal-delivery quirk, NOT the child's real exit code)
wait "$pid"
 
# Second wait: now actually blocks until the child has fully exited
# (after the trap forwarded SIGTERM to it), returning the child's true exit code
wait "$pid"
exit_code=$?
exit "$exit_code"

Why pattern B needs & + two waits: if the script ran /app/bin "$@" in the foreground, the trap on SIGTERM would not fire until the app returned. Backgrounding the app makes bash’s signal-processing loop active. The wait builtin is interruptible by trapped signals (unlike most foreground commands), which is what makes the pattern work — but that interruptibility is also why a single wait is not enough. When SIGTERM arrives, the first wait is interrupted and returns 128 + signal_number immediately, before the child has actually exited. The trap handler runs (forwarding SIGTERM to the child), and the script then needs a second wait to actually block on the child’s true exit. The second wait’s $? is the value you want to propagate.

SIGKILL and SIGSTOP cannot be trapped — they are delivered by the kernel directly and the process has no opportunity to handle them. This is why docker kill (which sends SIGKILL) bypasses any cleanup logic, and why graceful-shutdown windows matter.

EXIT trap ordering. trap '...' EXIT fires after the script reaches its natural end or exits via exit N or exits via an untrapped fatal signal. Multiple cleanup steps in one EXIT trap run in script order; if one fails, the remaining steps still run (the trap does not honor set -e). For ordered cleanup with dependencies, consider a single cleanup function with explicit error handling rather than chained traps.

exec does not just chain commands — it replaces the current process. exec /app/bin causes the bash process to be overlaid by /app/bin with the same PID. The shell is gone. Any code after exec is unreachable. This is the operational property that makes exec the right tool for entrypoint scripts: the app inherits PID 1 and the file descriptors, signal masks, and environment of the shell that called it (Suraj Deshmukh on exec).

Idempotency and mutual exclusion

Scope note: this section covers mutual exclusion (preventing concurrent execution) and atomic file replacement (avoiding partial-write windows). True rerun-idempotency — making a script safe to re-execute regardless of where the previous run failed — is a separate problem typically solved with idempotency keys, content-addressed outputs, or transactional state stores, and is out of scope here.

Shell’s primitives for the in-scope problems are three: atomic check-and-act (which doesn’t exist for most filesystem operations), lockfiles via flock(1), and the temp-write-then-rename pattern for transactional writes. Each has subtle semantics that produce real production bugs when misunderstood.

Check-then-act races. The naive pattern if [ ! -f /lock ]; then touch /lock; ...; rm /lock; fi has a TOCTOU race between the test and the touch. Two processes can both pass the test, both create the file, and both proceed. Do not use this pattern for anything that matters.

flock(1) — the canonical Linux primitive. flock is a thin wrapper over the flock(2) syscall, which provides advisory whole-file locking with BSD semantics (flock(1); flock(2)).

Usage modes

# Mode 1: subprocess form — flock holds the lock for the duration of the command
flock -n /var/lock/myjob -c 'long_running_command'
 
# Mode 2: FD form — lock held until FD closes (script exit or explicit close)
exec 9>/var/lock/myjob
flock -n 9 || { echo "another instance is running" >&2; exit 1; }
# ... do work ...
# lock releases automatically when script exits (FD 9 closes)
 
# Mode 3: subshell form — lock held only for the subshell
(
  flock -n 9 || exit 1
  # ... work ...
) 9>/var/lock/myjob

Flag reference

FlagMeaningUse case
-nNon-blocking — fail immediately if lock unavailableSingle-instance enforcement
-xExclusive (write) lock — defaultMost cases
-sShared (read) lockMultiple readers, one writer
-w NWait up to N seconds (decimal OK)Bounded retry
-E NCustom exit code on lock-acquisition failureDistinguish “couldn’t lock” from “command failed”
-oClose FD before exec’ing command (don’t pass lock to child)When the locked command spawns long-lived children that should not hold the lock
-uDrop the lock explicitlyManual lock management within a script

Critical semantic from flock(2): locks attach to the open file description, not the file descriptor. This means:

  • Within a process, all FDs derived from the same open() (via dup() or carried through fork()) share one lock — the child does not get an independent lock, and the lock releases only when all such FDs close.
  • Across processes, locks on the same inode conflict regardless of which open() produced them — that is what makes flock useful for cross-process mutual exclusion in the first place. The kernel tracks locks per open file description but checks for conflicts at the inode level.

The “open file description” property matters for sharing a lock within a process (a forked child that should also hold the lock, or a dup’d FD); it does not mean cross-process locks via separate open()s are independent.

PID files are not locks. A common pattern is echo $$ > /var/run/foo.pid and then checking kill -0 $(cat /var/run/foo.pid) to see if the previous instance is alive. This has multiple failure modes: stale PID files from crashes, PID reuse (the OS recycles PIDs and the recycled process may have nothing to do with your service), and the same TOCTOU race as check-then-act. Use flock on a FD; if you need the PID for observability, write it inside the locked region but rely on the lock for mutual exclusion, not the PID file.

Critical caveats from flock(2)

  • Advisory only. A process that doesn’t call flock can read and write the file without restriction. flock enforces nothing — it is a convention all participating processes must follow.
  • No deadlock detection. If process A holds lock 1 and waits for lock 2, while process B holds lock 2 and waits for lock 1, both will wait forever. The kernel does not detect the cycle.
  • NFS pre-2.6.12 is local-only. flock calls on an NFS mount before kernel 2.6.12 do not coordinate across hosts — each client sees only its own locks. Since 2.6.12, flock on NFS is emulated as an fcntl byte-range lock over the whole file, which has different semantics (per-process rather than per-open-file-description) and may surprise scripts that rely on the BSD-style behavior.
  • CIFS since kernel 5.5 has mandatory semantics — I/O on a locked region fails with EACCES, which is the opposite of advisory.

Transactional writes via temp + rename. The atomic primitive Linux provides for “publish a new version of a file without partial-write windows” is rename(2), which is atomic within a filesystem:

# Atomic file replacement
tmp=$(mktemp -p /etc/myapp/) || exit 1
trap 'rm -f "$tmp"' EXIT
generate_config > "$tmp" || exit 1
chmod 644 "$tmp"
mv "$tmp" /etc/myapp/config.yaml   # atomic within filesystem
trap - EXIT   # don't delete after successful rename

This is the only correct pattern for “update a config file” in any script that another process might read concurrently. A direct cmd > /etc/myapp/config.yaml opens a window where the file is truncated or partially written. The mktemp -p ensures the temp file lives on the same filesystem as the target so mv is atomic (mv falls back to copy-and-delete across filesystems, which is not atomic).

Structured logging from shell

Shell scripts have three viable logging substrates: stdout/stderr (the floor), logger(1) to syslog/journald (the middle), and systemd-cat / logger --journald to journald with structured fields (the ceiling). Pick by what the consumer needs.

The stdout/stderr discipline. Errors go to stderr; normal output goes to stdout. This is not a stylistic preference — 2>&1 redirection, pipeline composition, and CI log capture all depend on it. The Google Shell Style Guide codifies this with an error helper (Google Shell Style Guide):

err() {
  echo "[$(date +'%Y-%m-%dT%H:%M:%S%z')] $*" >&2
}

logger(1) for syslog/journald. Standard syslog has 8 priority levels (0=emerg, 1=alert, 2=crit, 3=err, 4=warning, 5=notice, 6=info, 7=debug) and facility codes (user, daemon, mail, etc.). On systemd systems, logger writes to journald, which preserves the syslog metadata as journal fields (logger(1); terminalzone.eu).

# Basic — tag and priority
logger -t myscript -p user.err "config render failed: $reason"
 
# Helper functions
SCRIPT=$(basename "$0")
log_info()  { logger -t "$SCRIPT" -p user.info  "$*"; }
log_warn()  { logger -t "$SCRIPT" -p user.warning "$*"; }
log_error() { logger -t "$SCRIPT" -p user.err   "$*"; }

systemd-cat for whole-script redirection. When you want every line the script writes to land in journald with a consistent tag, redirect once at the top (terminalzone.eu):

# Whole-script stdout to journald at info priority
exec > >(systemd-cat -t myscript -p info)
# Stderr separately, at err priority
exec 2> >(systemd-cat -t myscript -p err)

This requires no per-line logger calls — just write to stdout/stderr normally. The >(...) is bash process substitution; systemd-cat runs as a coprocess and feeds journald.

Buffering caveat. Process-substitution coprocesses inherit pipe buffering, so journal entries may not appear until the buffer flushes or the script exits. For tail-following during debugging, run journalctl -f -t myscript and accept that entries arrive in bursts. The canonical fix for live tailing is stdbuf -oL -eL to force line-buffering on the producer: exec > >(stdbuf -oL systemd-cat -t myscript -p info). journalctl --flush forces a journald-side flush but does not address upstream pipe buffering.

logger --journald for true structured logging. This is the only shell-native way to write multi-field structured entries to journald (logger(1)):

logger --journald <<EOF
PRIORITY=3
MESSAGE=database connection failed
DB_HOST=prod-replica-3
RETRY_COUNT=5
SCRIPT=$(basename "$0")
EOF

The stdin format is one KEY=VALUE per line. --journald ignores -p — priority must be embedded as the PRIORITY=N field (syslog numeric levels 0-7). Custom fields appear in journalctl -o json and are queryable via journalctl FIELD=value. This is the right substrate when downstream consumers (Loki, Promtail, journal-based alerting) parse structured fields.

Use caseToolNotes
One-off log line with tag and prioritylogger -t TAG -p user.info "msg"Simplest
Redirect whole script to journaldexec > >(systemd-cat -t TAG -p info)One-time setup at script top
Structured multi-field entrieslogger --journald with heredocOnly way to set custom fields
Plain stderr from helperecho ... >&2Floor; always works

Choosing key=value vs JSON. For journald, logger --journald gives you structured fields natively — no need for JSON or key=value strings. For non-journald systems (Loki via stdout, file-based logging), key=value is parseable by Promtail’s logfmt stage and is readable in raw form. JSON requires more careful escaping in shell (quotes inside quotes, unset variables expanding to empty inside braces) and is rarely worth the friction for shell-emitted logs.

Input validation and argument parsing

Bash gives you getopts (built-in, POSIX, short flags) and GNU getopt(1) (external, long flags, requires a capability check). The choice is determined by portability requirements and whether long flags add user-facing value.

getopts — the portable built-in. POSIX-compliant, available everywhere, but supports only single-character flags (Google Shell Style Guide shows only getopts):

verbose=0
output=""
 
while getopts ":vo:h" opt; do
  case "$opt" in
    v) verbose=1 ;;
    o) output="$OPTARG" ;;
    h) usage; exit 0 ;;
    \?) echo "unknown flag: -$OPTARG" >&2; exit 2 ;;
    :)  echo "flag -$OPTARG requires an argument" >&2; exit 2 ;;
  esac
done
shift $((OPTIND - 1))
 
# Remaining positional args are in "$@"

The leading : in :vo:h enables silent error handling (the \? and : cases above). Without it, getopts writes its own error messages to stderr.

GNU getopt(1) — long flags, requires capability check. Linux distributions ship “enhanced” GNU getopt that supports --long-flags, optional arguments, and proper handling of --. BSD/macOS ships a different getopt without these features. Always probe with getopt --test before using:

# Capability probe — GNU getopt returns exit code 4 on --test
getopt --test > /dev/null 2>&1
if [ $? -ne 4 ]; then
  echo "this script requires GNU getopt; on macOS install via 'brew install gnu-getopt'" >&2
  exit 2
fi
 
opts=$(getopt -o vo:h --long verbose,output:,help -n "$0" -- "$@") || exit 2
eval set -- "$opts"
 
verbose=0
output=""
while true; do
  case "$1" in
    -v|--verbose) verbose=1; shift ;;
    -o|--output)  output="$2"; shift 2 ;;
    -h|--help)    usage; exit 0 ;;
    --)           shift; break ;;
    *)            echo "internal error parsing $1" >&2; exit 2 ;;
  esac
done
 
# Remaining positional args are in "$@"

The eval set -- is required because GNU getopt returns a single string with shell quoting that must be re-parsed into "$@". This is the one place in shell where eval is the canonical pattern.

Manual -- handling. The convention that -- terminates flag parsing (everything after is positional, even if it starts with -) is honored by both getopts (via getopts returning a non-option and OPTIND pointing past --) and GNU getopt (via the explicit -- case above). For hand-rolled parsing, treat -- as a special token that breaks the parse loop.

Env-var fallbacks. A common pattern is “flag overrides env-var overrides default”:

output="${OUTPUT_PATH:-/var/log/default.log}"   # env-var with default
 
while getopts "o:" opt; do
  case "$opt" in
    o) output="$OPTARG" ;;   # flag overrides
  esac
done

The ${var:-default} parameter expansion handles “unset OR empty” — use ${var-default} (no colon) for “unset ONLY.”

Conflicting-flag detection. When two flags are mutually exclusive, detect at the end of parsing rather than mid-parse:

mode=""
while getopts "qv" opt; do
  case "$opt" in
    q) mode_q=1 ;;
    v) mode_v=1 ;;
  esac
done
if [ -n "${mode_q:-}" ] && [ -n "${mode_v:-}" ]; then
  echo "-q and -v are mutually exclusive" >&2
  exit 2
fi

Mid-parse detection forces an order-dependent error (the second flag triggers, the first doesn’t), which is confusing for users.

shellcheck beyond the basics

ShellCheck catches the entire class of word-splitting bugs that bash invites by default. The three most-common rules (SC2086, SC2068, SC2206) are the same failure mode in three syntactic positions — collapsing them to one concept makes the surrounding noise tractable. SC2128 is a closely-related-but-distinct gotcha about array-default semantics that travels with the cluster in practice.

The word-splitting mechanism. An unquoted variable expansion in bash undergoes three steps (SC2086):

  1. Split on IFS characters (default: space, tab, newline). "foo bar" becomes two tokens.
  2. Each fragment is glob-expanded. *.txt matches files in the current directory.
  3. Rejoined with spaces when passed to a command.

Quoting ("$var") suppresses all three steps. This is the single most important defensive habit in bash scripting. Every unquoted variable is a bug waiting for an input with whitespace or glob characters.

The three-rule word-splitting cluster (one bug, three positions)

RulePositionExampleFix
SC2086Bare variablecp $src $dstcp "$src" "$dst"
SC2068Positional/array spreadcp $@ ~/cp "$@" ~/
SC2206Array literalarr=( $var )arr=( "$var" ) or mapfile -t arr <<< "$var"

SC2128 — the related array-default bug: $arr and ${arr[0]} are identical — both give only the first element of the array regardless of size. This is not a word-splitting bug; it is bash’s silent fallback when you reference an array without an index. "${arr[@]}" expands all elements as separate arguments; "${arr[*]}" concatenates with the first IFS character. The bare $arr is not a syntax error, just a near-certain bug — and it travels with the SC2086 cluster in practice because both are “I forgot how bash treats this expansion” mistakes (SC2128).

For intentional splitting, use arrays, not unquoted variables

# WRONG — unquoted, vulnerable to IFS and globs
flags="--verbose --output=/tmp/x"
some_command $flags
 
# RIGHT — array preserves argument boundaries explicitly
flags=(--verbose --output=/tmp/x)
some_command "${flags[@]}"
 
# RIGHT for newline-split — mapfile is the canonical pattern
mapfile -t lines <<< "$multiline_var"
 
# RIGHT for space-split with controlled IFS
IFS=" " read -r -a parts <<< "$space_separated"

Following source statements with -x. ShellCheck does not follow source by default because it cannot know what files are valid input. To follow (SC1091; Vidar Holen on v0.7.0):

# Option 1: explicit per-statement directive
# shellcheck source=lib/common.sh
. "$(dirname "$0")/lib/common.sh"
 
# Option 2: skip the sourced file entirely
# shellcheck source=/dev/null
. "$dynamic_path"
 
# Option 3: source-path directive (v0.7.0+)
# shellcheck source-path=SCRIPTDIR
. lib/common.sh

The SCRIPTDIR identifier resolves to the directory of the file containing the directive, which works around hard-coding absolute paths. Pair with -x on the command line or external-sources=true in .shellcheckrc to actually follow the source.

Inline disables with scope. A # shellcheck disable=SCxxxx comment disables the rule for the next command:

# shellcheck disable=SC2086  # word-splitting is intentional here
some_command $intentionally_split

Use sparingly. Each disable is a claim that you understand the warning and choose to override it; reviewers and future-you should be able to verify the claim from the surrounding context.

# shellcheck shell=bash for files without a shebang. Library files meant to be sourced often have no shebang. Without a shebang ShellCheck guesses sh, which over-reports compatibility issues for bash-specific syntax. The directive # shellcheck shell=bash at the top of the file fixes this (Vidar Holen on v0.7.0).

The .shellcheckrc for project-wide settings. A .shellcheckrc at the project root configures ShellCheck globally:

external-sources=true
shell=bash
disable=SC1091

This is preferable to commenting every file. Commit it.

The optional-checks ladder. ShellCheck has off-by-default checks (--list-optional shows them) that catch advanced issues like missing braces (SC2250), quote-the-rhs-of-assignment (SC2249), etc. Enable on a per-file basis with # shellcheck enable=SC2250 or globally via .shellcheckrc's enable= directive (Vidar Holen on v0.7.0).

Sources

Accepted

SourceTierWhy Credible
BashFAQ/105 - Why doesn’t set -e do what I expected? (retrieved: 2026-05-24)HighWooledge community wiki; primary authoritative reference on Bash semantics; community-maintained with multiple expert contributors
Aaron Maxwell - Unofficial Bash Strict Mode (retrieved: 2026-05-24)HighOriginating source for the “strict mode” term; widely cited across shell-scripting community; practitioner-authored with working examples
Disconnected Systems - Another Bash Strict Mode (retrieved: 2026-05-24)EstablishedPractitioner blog with the ERR-trap alternative; cited in shellcheck and bash discussions; working code shown
Google Shell Style Guide (retrieved: 2026-05-24)HighAuthoritative corporate style guide; enforced across Google’s open-source projects; actively maintained
flock(1) man page (retrieved: 2026-05-24)HighLinux man-pages project; authoritative upstream documentation for the command
flock(2) man page (retrieved: 2026-05-24)HighLinux man-pages project; primary kernel syscall documentation
logger(1) man page (retrieved: 2026-05-24)HighLinux man-pages project; authoritative upstream documentation; only place that documents --journald accurately
ShellCheck wiki - SC2086 (retrieved: 2026-05-24)HighOfficial tool documentation maintained by ShellCheck’s author Vidar Holen
ShellCheck wiki - SC2068 (retrieved: 2026-05-24)HighOfficial tool documentation
ShellCheck wiki - SC2206 (retrieved: 2026-05-24)HighOfficial tool documentation
ShellCheck wiki - SC2128 (retrieved: 2026-05-24)HighOfficial tool documentation
ShellCheck wiki - SC1091 (retrieved: 2026-05-24)HighOfficial tool documentation; covers source-following and directives
Vidar Holen blog - ShellCheck v0.7.0 release notes (retrieved: 2026-05-24)HighPrimary changelog source by ShellCheck’s author; covers source-path, optional checks, .shellcheckrc
CloudBees - Trapping Signals in Docker Containers (retrieved: 2026-05-24)EstablishedWidely-cited engineering blog; canonical reference for Docker signal patterns; reproducible patterns
Suraj Deshmukh - Shell Exec (retrieved: 2026-05-24)EstablishedTechnical practitioner blog with container focus; explains exec semantics with PID-1 framing
Terminal Zone - Bash and journald (retrieved: 2026-05-24)EstablishedPractitioner journald-integration reference; documents systemd-cat and logger patterns with examples
BashPitfalls (retrieved: 2026-05-24)HighWooledge community wiki; primary catalog of common shell mistakes; community-maintained

Rejected

SourceWhy Rejected
Various Medium/Dev.to “10 bash tips” postsAggregator content rephrasing the same set of basics; primary sources (BashFAQ, man pages, shellcheck wiki) used instead
Stack Overflow answers on set -e semanticsUseful as orientation but not citable for canonical claims; BashFAQ/105 is the upstream reference these answers cite
Vendor-specific container runtime docs on signal handlingScoped to one runtime; CloudBees and Suraj Deshmukh references generalize to all OCI runtimes

Revision History

DateScopeKey Changes
2026-05-24Initial researchCreated from 17 web sources (13 High-tier, 4 Established). Covers set -euo pipefail semantics and suppression contexts, signal handling and bash-as-PID-1, idempotency via flock(1)/flock(2) with NFS/CIFS caveats, structured logging via logger and systemd-cat, getopts vs GNU getopt input parsing, and the SC2086/SC2068/SC2206/SC2128 word-splitting rule cluster plus source-following directives. Scoped to a sysadmin-engineer agent that already knows bash basics; emphasizes why and edge cases over what.
2026-05-24Reviewer-driven revisionAddressed HIGH findings from copy-editor, validate, and fact-check: (1) corrected (( i++ )) explanation — bash sets exit status to 1 (not 0) when the arithmetic value is 0, and ++i is pre-increment not post-increment; (2) fixed Pattern B wait-order — the first wait returns 128+SIG from trap interruption, the second wait returns the child’s real exit; (3) rewrote the flock open-file-description bullet to remove draft-thinking artifact and correctly state within-process vs cross-process semantics; (4) corrected getopt --test capability probe (two-line form); (5) added inherit_errexit mitigation as a partial close on the command-substitution suppression context, with explicit note that conditionals/pipelines/local-decl still leak; (6) reframed the strict-mode recommendation as contested rather than consensus, acknowledging Maxwell’s strict-mode advocacy alongside Wooledge’s explicit-checking position; (7) corrected pipefail+SIGPIPE direction — produces false failure, not false success (yes | head -1 exits 141); (8) added cgroup/ps caveat to Pattern B recommendation; (9) added target-environment baseline (bash ≥4.4 assumption, macOS 3.2 caveat); (10) sharpened SC2128 framing as array-default bug distinct from the SC2086/2068/2206 word-splitting cluster; (11) added stdbuf -oL as canonical live-tailing fix in journald buffering note; (12) added scope note to Idempotency section clarifying that true rerun-idempotency is out of scope.
2026-07-18Garden projection scrub + style passPublished in place as a garden page (publish: true, stage: budding): dropped 3 internal-only cross-reference sentences per the scrub checklist; converted 8 bold pseudo-headings to real headings and tagged a languageless code fence per the Vault Style Guide.