Shell scripts are surprisingly brittle, and what looks like a simple sequence of commands can unravel due to tiny, often overlooked, details.
Let’s see a script in action:
#!/bin/bash
# A simple script to process a file
INPUT_FILE="data.txt"
OUTPUT_DIR="/tmp/processed_data"
DATE_SUFFIX=$(date +%Y%m%d)
# Create output directory if it doesn't exist
mkdir -p "$OUTPUT_DIR"
# Process the file
grep "ERROR" "$INPUT_FILE" > "$OUTPUT_DIR/errors_$DATE_SUFFIX.log"
wc -l "$OUTPUT_DIR/errors_$DATE_SUFFIX.log" >> "$OUTPUT_DIR/summary_$DATE_SUFFIX.log"
echo "Processing complete. Check $OUTPUT_DIR for logs."
This script aims to find lines containing "ERROR" in data.txt, save them to a dated log file, and then count those error lines, appending the count to a summary log. On the surface, it seems straightforward.
The core problem this script addresses is automating repetitive command-line tasks, making them repeatable, less error-prone, and executable on a schedule. It acts as a glue language, stringing together the power of individual Unix utilities. Internally, a shell script is simply a text file containing a sequence of commands that the shell interpreter (like Bash) reads and executes one by one. The shell manages process creation, input/output redirection, variable expansion, and control flow (loops, conditionals).
Here are the key levers you control:
- Variables: Used to store data, filenames, configuration settings.
INPUT_FILE="data.txt". - Command Execution: The sequence of commands like
grep,wc,mkdir. - Input/Output Redirection: Using
>to overwrite,>>to append,<to read from a file, and|to pipe output from one command to another. - Control Flow:
if,for,whilestatements to make decisions and repeat actions. - Functions: To group commands for reusability and modularity.
When you run bash my_script.sh, the shell reads the script line by line. #!/bin/bash (the shebang) tells the system to use /bin/bash to execute the script. mkdir -p creates the directory. grep "ERROR" "$INPUT_FILE" > "$OUTPUT_DIR/errors_$DATE_SUFFIX.log" searches data.txt for lines with "ERROR" and writes them to the specified log file. wc -l ... >> ... counts the lines in the error log and appends that count to the summary log.
The most surprising thing about shell scripting is how often scripts fail not because of a complex logic error, but because of unexpected whitespace or special characters in filenames, or the simple absence of a file that the script assumes exists.
To make scripts more robust, several best practices are crucial. First, always quote your variables. "$INPUT_FILE" prevents issues if the filename has spaces or special characters. Second, use set -euo pipefail. set -e exits immediately if a command exits with a non-zero status. set -u treats unset variables as an error. set -o pipefail ensures that a pipeline command fails if any command in the pipe fails, not just the last one. Third, check command success explicitly when set -e might not be enough or when you want specific error handling. For example, if ! grep "ERROR" "$INPUT_FILE" > "$OUTPUT_DIR/errors_$DATE_SUFFIX.log"; then echo "ERROR: grep command failed!" >&2; exit 1; fi. Fourth, use mktemp for temporary files to avoid race conditions and security issues. Instead of TMP_FILE=$(date +%s%N).tmp, use TMP_FILE=$(mktemp). Finally, use >&2 to redirect error messages to standard error, distinguishing them from normal output.
The next concept to grapple with is handling signals gracefully, like Ctrl+C, to clean up temporary files or close connections.