Skip to content

Bash & Shell Scripting

Turn repeated commands into repeatable scripts. Bash is the glue of data engineering and MLOps: scheduling, ETL wrappers, and model-serving one-liners are all shell.

Table of Contents

Installation / Setup

Any Linux/macOS has bash (Windows: WSL or Git Bash). Scripts start with a shebang and need execute permission:

#!/usr/bin/env bash
set -euo pipefail          # fail fast on errors (see Best Practices)

echo "hello"

chmod +x script.sh
./script.sh                # run
bash script.sh             # run without chmod

Core Syntax

Variables & expansion

name="Ada"                 # NO spaces around =
echo "$name"               # Ada  (quote to keep as one word)
echo "${name}_suffix"      # Ada_suffix
readonly CONFIG="prod"     # constant

# Command substitution
files=$(ls *.csv)          # or `ls *.csv`
echo "found: $files"

# Arithmetic
n=$((2 + 3 * 4))           # 14
Expansion Result
$var / ${var} variable value
$(cmd) command output
$((expr)) arithmetic
$1, $2 ... $@ script/function arguments
$? exit code of last command
$$ current PID
* / ? / [abc] filename globbing

Conditionals

if [ -f "$file" ]; then
    echo "file exists"
elif [ -d "$dir" ]; then
    echo "dir exists"
else
    echo "neither"
fi

if [ "$name" = "Ada" ]; then echo "hi Ada"; fi      # string compare
if [ "$n" -gt 10 ]; then echo "big"; fi             # numeric: -gt -lt -eq -ge -le

# test shortcuts:  && = and, || = or
[ -z "$var" ] && echo "var is empty"
grep -q error log || echo "no errors"

# modern [[ ]] (bash): =~ regex, no quoting worries
if [[ "$url" =~ ^https ]]; then echo "secure"; fi

Common test flags: -f file exists, -d directory, -e any path, -z empty string, -n non-empty, -x executable, -r readable.

Loops

for f in *.csv; do
    echo "processing $f"
    python parse.py "$f"
done

for i in $(seq 1 5); do echo $i; done
for ((i=0; i<10; i++)); do ...; done

while read -r line; do           # read a file line by line
    echo "$line"
done < data.txt

while true; do                   # until Ctrl-C
    python monitor.py
    sleep 60
done

Functions

greet() {
    local name="$1"              # local = scope inside function
    echo "hello $name"
    return 0                     # exit status
}
greet "Ada"

Pipes & redirection

# pipes: send stdout of one command into stdin of the next
ps aux | grep python | awk '{print $2}'

# redirection
cmd > out.txt          # stdout  -> file (overwrite)
cmd >> out.log         # stdout  -> file (append)
cmd 2> err.txt         # stderr  -> file
cmd > all.log 2>&1     # stdout AND stderr -> file
cmd < input.txt        # stdin   <- file
cmd &>/dev/null        # discard everything
Piece Means
\| pipe stdout → next command's stdin
> / >> overwrite / append a file
2> / 2>&1 redirect stderr / merge into stdout
&> both streams
$( ) capture output as text
&& / \|\| run next only if success / failure

Job control & background

python train.py &            # background
jobspid=$!
wait $pid                    # wait for it
kill %1                      # kill job 1
nohup python train.py &      # survive logout (see linux-commands.md)

Practical Examples

1. Data-prep wrapper (safe, parameterized)

#!/usr/bin/env bash
set -euo pipefail

DATA_DIR="${1:-data}"                    # first arg, default "data"
OUT="${2:-clean.csv}"

echo "Processing $DATA_DIR"
for f in "$DATA_DIR"/*.csv; do
    echo "  → $(basename "$f")"
    python prep.py "$f"
done
echo "Done → $OUT"

2. Retry until success (train/upload resilience)

attempt=0
until curl -sf https://api.example.com/health; do
    attempt=$((attempt + 1))
    echo "waiting for service (${attempt})..."
    sleep 5
done
echo "service is up"

3. Parallel-safe daily backup

#!/usr/bin/env bash
set -euo pipefail
stamp=$(date +%Y-%m-%d)
tar -czf "backups/models-$stamp.tar.gz" models/
gsutil cp "backups/models-$stamp.tar.gz" gs://my-bucket/models/   # or rsync
echo "backup complete"

4. Loop over every GPU experiment

for lr in 0.001 0.0001; do
    for seed in 1 2 3; do
        nohup python train.py --lr "$lr" --seed "$seed" \
            > "logs/lr${lr}_s${seed}.log" 2>&1 &
    done
done

Common Patterns

# Exit if a required command is missing
command -v python3 >/dev/null || { echo "python3 missing"; exit 1; }

# Read TSV/CSV columns in a loop
while IFS=',' read -r name score; do
    echo "$name scored $score"
done < scores.csv

# Fail if previous step didn't produce output
test -s "$OUT" || { echo "empty output!"; exit 1; }

# Dry-run flag for destructive scripts
if [[ "${DRY_RUN:-0}" == "1" ]]; then
    echo "DRY RUN: would rm $TARGET"
else
    rm -rf "$TARGET"
fi

Best Practices

  • Start every script with set -euo pipefail: exit on error, fail on unset variables and broken pipes.
  • Always quote variables ("$file") — unquoted splits on spaces.
  • Use local in functions to avoid leaking globals.
  • Check inputs: if [ $# -lt 2 ]; then echo "usage: ..."; exit 1; fi.
  • Validate exit codes ($?, &&/||, test -s) — a silent failure is worse than a crash.
  • Prefer [[ ]] over [ ] in bash for regex and fewer quoting bugs.
  • Keep scripts idempotent (re-running shouldn't double-apply) and add echo progress lines.
  • Lint with shellcheck — it catches most classic bash bugs.

Common Mistakes

Mistake Consequence Fix
var = value (spaces) Runs var as a command var=value
Forgetting set -e Script continues after failure set -euo pipefail
Unquoted "$file" Paths with spaces break Quote every expansion
[ vs [[ confusion Regex/empty-string bugs Use [[ ]] in bash
rm -rf $DIR with empty var rm -rf /-adjacent disaster Guard: ${DIR:?unset!} and quote
cat file \| grep (useless cat) Extra process grep pattern file
if [ "$x" = pattern ] Literal compare, not glob [[ $x == *pattern* ]]
Editing a script with CRLF (Windows) $'\r': command not found sed -i 's/\r$//' script.sh

Troubleshooting

Symptom Cause Fix
syntax error near unexpected token CRLF endings / bad quoting file script.sh; strip CR; check quotes
command not found mid-script PATH changed or typo command -v cmd; full path
Script exits but seems fine Unset var with -u Default it: ${VAR:-default}
set -e exits on a "failing" grep grep returns 1 on no match Append \|\| true or use if grep -q
Loop reads only first line stdin consumed by inner command < file at loop end; use -r
Background jobs die on logout Shell sends SIGHUP nohup / setsid / tmux

Official Documentation

Last Updated

2026-09-06 — Initial version.