# Why Bash? The simple answer? Pretty much any Unix-based system is going to have some version of Bash on it. Even languages such as python aren't guaranteed because not every system uses `systemd` and thus doesn't require it (eg. AIX) - but you'll be hard-pressed to find a system without bash in it. # Bash version differences cheatsheet Each new version of bash adds new features and changes or removes others. It's important to know the capabilities of **all** of the systems you plan on running your bash scripts on. I took the liberty of summarizing the most notable changes in each version of bash. The full list can be found [here](https://git.savannah.gnu.org/cgit/bash.git/tree/NEWS). ## 5.2 (09/26/2022) | Feature | Summary | | ------------------------ | ------------------------------------------------------------------------- | | `read` Timeout | `read` command now supports fractional timeouts. | | Quoting | Adds `...'` and `quot;..."` quoting options. | | Parameter Transformation | Adds `@k` operator for splitting strings into words after transformation. | | `printf` Specifier | Adds `%Q` specifier for quoting strings. | | Globskip Dots | Prevents `.` and `..` from appearing in pathname expansions by default. | | Suspend Option | `suspend -f` now forces suspension even without job control. | ## 5.1 (12/07/2020) | Feature | Summary | | ---------------------------- | ------------------------------------------------------------------- | | `read` with File Descriptors | `read -e` now supports custom file descriptors (e.g., `read -u N`). | | `wait` PID Storage | `wait` can store the PID of processes using `-p VARNAME`. | | Case Conversion | Adds `U`, `u`, and `L` options to change string case. | | Enhanced Job Reporting | `jobs` can show completed jobs when used with `-c`. | | Array Variable Display | `PROMPT_COMMAND` can now be an array, with each item as a command. | ## 5.0 (01/07/2019) | Feature | Summary | |-----------------------------|----------------------------------------------------------------------------| | `wait` with Process Substitution | `wait` now supports waiting for the last process substitution. | | Epoch Time Variables | Adds `EPOCHSECONDS` and `EPOCHREALTIME` for Unix epoch time. | | `history -d` Enhancements | Allows deleting ranges of history entries with `-d start-end`. | | Case-Insensitive Completion | Command completion matches aliases and functions case-insensitively. | | Flexible `umask` Values | `umask` supports modes and masks beyond octal 777. | ## 4.4 (09/15/2016) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | `mapfile` Enhancements | `mapfile` supports custom delimiters with `-d` and can strip delimiters with `-t`. | | Command Timing | Supports `time ; othercommand` to time null commands. | | Filename Ignore List | `EXECIGNORE` variable can ignore specific filenames when searching for commands. | | Prompt String Expansion | New `PS0` string for showing prompts before command execution. | | `unset` Enhancement | `unset` can now unset scalar variables using subscript `0`. | ## 4.3 (02/26/2014) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | Unlimited History Option | Setting `HISTSIZE` or `HISTFILESIZE` to negative values enables unlimited history. | | Enhanced `read` | `read` now skips NUL bytes in input and is interruptible by signals in POSIX mode. | | Negative Array Indexing | Allows referencing array elements using negative indices (e.g., `array[-1]`). | | Wait for Any Child | `wait -n` waits for any child process to change state. | | New Special Variable | `BASH_COMPAT` sets compatibility level for different Bash versions. | ## 4.2 (02/13/2011) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | Unicode Escapes | `...'`, `echo`, and `printf` now support `\u` and `\U` Unicode escapes. | | `test` Enhancement | Adds `-v variable` option to check if a variable is set. | | Negative Array Indexes | Indexed arrays support negative subscripts, counting from the end. | | `printf` Time Formatting | New `%(fmt)T` format specifier for time formatting with `strftime`-like syntax. | | Last Pipe Option | `lastpipe` option allows last pipeline command to run in the current shell. | ## 4.1 (12/31/2009) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | `printf` Array Support | `printf -v` can assign values to array indices. | | Enhanced `read` | New `-N nchars` option reads exact character counts, ignoring delimiters. | | Conditional String Comparison | `[[ <` and `>` comparisons respect locale settings. | | Syslog History Option | New option to forward all command history entries to syslog. | ## 4.0 (02/20/2009) | Feature | Summary | | ------------------------ | --------------------------------------------------------------------------------- | | `BASHPID` Variable | New `$BASHPID` variable provides the process ID of the current shell. | | Automatic `cd` | The `autocd` option allows automatic directory changes with directory name input. | | `globstar` Option | Enables `**` to match directories and files recursively. | | Append Redirect Operator | `&>>` appends both stdout and stderr to a file. | | Case Modification | Adds expansions for uppercase (`^`) and lowercase (`,`), applicable globally. | ## 3.2 (10/11/2006) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | Pattern Matching Update | `[[ =~ ]]` now supports quoted strings for exact string matching. | | Home Variable in POSIX Mode | `$HOME` is no longer automatically set in POSIX mode. | | MacOS Compatibility | Loadable builtins are now supported on MacOS 10.3 and 10.4. | ## 3.1 (12/08/2005) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | Append Operator | Adds `+=` operator for appending to strings and arrays. | | Case-Insensitive Matching | New `nocasematch` option for case-insensitive matching in `case` and `[[` commands.| | `printf` Output to Variable | `printf` now supports `-v var` option to write output directly to a variable. | | Environment Inheritance | Inherits `$_` from environment if set at startup. | | POSIX Conformance Options | Strict POSIX mode added, setting compliance by default if enabled. | ## 3.0 (08/03/2004) | Feature | Summary | | ------------------------------- | ------------------------------------------------------------------------ | | Brace Expansion | Adds `{x..y}` expansion for sequences, e.g., `{1..5}`. | | Case-Insensitive History Ignore | `HISTCONTROL`'s `erasedups` is now case-insensitive. | | Regular Expression Matching | `[[ =~ ]]` supports extended regex matching in conditional expressions. | | Fail Glob Option | New `failglob` option triggers error if pathname expansion has no match. | | `pipefail` Option | `set -o pipefail` returns failure if any command in a pipeline fails. | ## 2.05a/b (04/09/2001, 11/16/2001, 07/17/2002) | Feature | Summary | | -------------------------- | ------------------------------------------------------------------------------- | | `/dev/tcp` and `/dev/udp` | Supports service names in redirections, not just port numbers. | | `complete` Options | Adds options for fallback completion (e.g., directory or filename) if no match. | | Here-String Redirection | Adds `<<<` operator for here-strings, redirecting a string as stdin. | | `printf` Escape Updates | New `%q` format for quoting special characters and `\\cX` for Control-X. | ## 2.04 (03/21/2000) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | Enhanced `read` Options | `read` command adds `-t` (timeout), `-n` (character limit), `-d` (delimiter), and `-s` (silent). | | TCP and UDP Redirection | Supports `/dev/tcp/host/port` and `/dev/udp/host/port` for network connections. | | Programmable Completion | Adds `complete` and `compgen` builtins for custom autocompletions. | | `FUNCNAME` Variable | New variable, `FUNCNAME`, stores the name of the currently executing function. | | `HISTCONTROL` Enhancements | Adds `erasedups` to remove duplicate entries from command history. | ## 2.03 (02/19/1999) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | Restricted Shell Option | New `shopt` option, `restricted_shell`, to indicate restricted shell mode. | | Auto-Export `OLDPWD` | `OLDPWD` is now automatically exported, aligning with POSIX.2 requirements. | | Login Option for Non-Interactive Shells | Non-interactive shells with `--login` source login startup files. | ## 2.02 (04/18/1998) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | Extended Pattern Matching | Adds ksh-style extended globbing (e.g., `[@+*?!](pattern)`) with `extglob` option. | | `[[` Command | New `[[` command adds extended test functionality. | | `printf` Builtin | Adds `printf` as a builtin, following POSIX standards. | | Command Substitution | `$(<filename)` syntax for reading a file’s contents without `cat`. | | Case-Insensitive Globbing | New `nocaseglob` option for case-insensitive file pattern matching. | ## 2.01 (06/05/1997) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | `GROUPS` Variable | Adds `GROUPS` array variable, listing all groups the user belongs to. | ## 2.0 (12/31/1996) | Feature | Summary | |--------------------------------|------------------------------------------------------------------------------------| | `time` Reserved Word | Adds `time` keyword for timing pipelines, commands, and functions. | | ANSI and Locale Quoting | `...'` and `quot;..."` quoting for ANSI-C escapes and locale-specific translations. | | Array Support | Adds integer-indexed arrays with flexible indexing and assignment. | | Enhanced Prompt Customization | New prompt expansions: `\a`, `\e`, `\H`, `\T`, `\@`, `\v`, `\V`. | | Indirect Expansion | Adds `${!var}` syntax for indirect variable references (equivalent to `eval`). | # Basics This is intended as a cheatsheet and crash-course for SAs looking to automate things with Bash, so I won't be going into great detail about any particular thing. If you're an SA, you know how to use Google. I hope. Bash scripts and the command-line are effectively the same thing. You can paste an entire script into your terminal and it will run fine, and vice-versa. When I write bash scripts, I'll usually test each line I'm unsure of on the machine or machines on which I plan to run the script- as long as the line doesn't actually make any changes to the OS that I can't easily undo. The main thing to note is the shebang at the top of a script file (`#!/bin/bash` etc) - it's usually recommended to use `#!/usr/bin/env bash` for portability reasons but there's a *lot* of history and debate online around anything that you can put into a script and its "portability" - so, using `#!/bin/bash`, as long as it works for your use-case, is perfectly fine. ## Variables Variables can be set for the current shell by simply doing the following: ```bash variable_1="x" VARIABLE_2="y" ``` You can set variables for the current shell *and any subprocesses it spawns* by doing the following instead: ```bash export variable_1="x" export VARIABLE_2="y" ``` or, if the variable was already set somewhere else: ```bash export variable_1 export VARIABLE_2 ``` To use variables in your script later, it's usually recommended to use the `"${}"` syntax. eg. ```bash echo "${variable_1}" ``` this allows for fancy things like replacements directly in the variable and also avoids problems like shell expansions. Sometimes, though, you really need to do weird tuff like run `yum` or `dnf` based on the OS: ```bash if command -v dnf; then pkg_cmd="dnf" else pkg_cmd="yum" fi $pkg_command -y update ``` The above script would run `dnf -y update` if the system has `dnf`, and `yum -y update` if not. ## If/elif/else The basic syntax for an if/else if/else statement is: ```bash if condition-1; then do-thing-1 elif condition-2; then do-thing-2 elif condition-3; then do-thing-3 else do-thing-4 fi ``` ## Return codes Return codes in Bash indicate the success or failure of a command. A return code of `0` generally means success, while any non-zero code indicates an error. You can access the return code of the previous command using `$?`. eg. ```bash ls /nonexistent_directory if [[ $? -ne 0 ]]; then echo "Command failed." fi ``` ## Checking if a command exists You can use `command -v` or `which` to check if a command is available: ```bash if command -v cat &>/dev/null; then echo "cat is installed." else echo "cat is not installed." fi ``` Note that this works for Bash builtins (eg. `timeout`) as well as "real" commands located on the system. ## $() The `$()` syntax is used for command substitution in Bash. It allows you to capture the output of a command and use it as part of another command or assign it to a variable. eg. ```bash date_output=$(date) echo "The current date and time is: $date_output" ``` This is equivalent to the older backtick syntax, but `$()` is preferred as it is more readable and can be nested easily. The older syntax would look like this: ```bash date_output=`date` echo "The current date and time is: $date_output" ``` Combining the previous two sections, a handy trick I use sometimes is `ls -l $(which cat)` or `vi $(which custom-command)` to quickly find and edit a particular script registered as a command (eg. for Bash scripts located in `~/.local/bin`) ## Checking OS type and version Almost every Unix-based distro will support `os-release`, which is really just a file located at `/etc/os-release`. For example, to determine if a system is RHEL-based or Debian-based, you can use the `ID_LIKE` variable located in it: ```bash ID_LIKE=$(grep ^ID_LIKE /etc/os-release | awk -F'[="]' '{print $3}') ``` or, even better: ```bash source /etc/os-release ``` Sourcing the file will put all of the content into variables like `NAME`, `VERSION`, `ID`, `ID_LIKE`, etc for use in your script. Then, you can do something like the following: ```bash if [[ "${ID_LIKE}" == *rhel* ]]; then echo "RHEL OS!" elif [[ "${ID_LIKE}" == *debian* ]]; then echo "Debian OS!" fi ``` ## Strings Strings in Bash can be enclosed with single (`'`) or double (`"`) quotes. Single quotes preserve literal value, while double quotes allow variable interpolation. ```bash name="John" echo 'Hello, $name' # Output: Hello, $name echo "Hello, $name" # Output: Hello, John ``` ## Numbers & Math Bash allows simple arithmetic using `$((...))` syntax: ```bash num1=5 num2=10 sum=$((num1 + num2)) echo $sum # Output: 15 ``` These operators are used for numeric comparisons in conditional expressions: - `-eq`: Equal - `-ne`: Not equal - `-gt`: Greater than - `-lt`: Less than eg. ```bash if [[ num1 -gt 2 ]]; then echo "$num1 > 2" else echo "2 > $num1" fi ``` ## -n and -z `-n` and `-z` are used to test strings: - `-n` checks if a string is non-empty. - `-z` checks if a string is empty. eg. ```bash if [[ -n "$MY_VAR" ]]; then echo "MY_VAR is not empty." fi if [[ -z "$MY_VAR" ]]; then echo "MY_VAR is empty." fi ``` ## Arrays Bash supports indexed arrays and (from version 4) associative arrays: ```bash # Indexed Array my_array=("apple" "banana" "cherry") echo ${my_array[1]} # Output: banana # Associative Array declare -A fruits fruits["yellow"]="banana" echo ${fruits["yellow"]} # Output: banana ``` ## Difference between \[, \[\[, and no brackets - `[` is a synonym for `test` and is used for basic conditional tests. - `[[` is a Bash-specific keyword that supports more complex conditions and pattern matching. - No brackets are used for simple commands or variable assignment. eg. ```bash if command -v cat; then echo "Using no brackets." fi if [ "$var" = "value" ]; then echo "Using single bracket." fi if [[ "$var" == "value" ]]; then echo "Using double brackets." fi ``` ## To terminal or not to terminal? You can check if a script is connected to a terminal to decide whether to enable terminal-specific features, like colored output: ```bash if [[ -t 1 ]]; then TERMINAL=true else TERMINAL=false fi ``` ## Inline if/then Bash supports inline `if/then` statements using the `&&` and `||` operators: ```bash [[ -n "$MY_VAR" ]] && echo "MY_VAR is not empty" || echo "MY_VAR is empty" ``` ## Output redirection Output redirection in Bash allows you to direct the output of commands to files or other outputs. The most common operators are: - `>`: Redirect standard output to a file (overwrites the file). - `>>`: Redirect standard output to a file (appends to the file). - `2>`: Redirect standard error to a file. - `&>`: Redirect both standard output and standard error to a file. eg. ```bash # Overwrite output to a file ls > output.txt # Append output to a file echo "Additional line" >> output.txt # Redirect error messages to a file ls /nonexistent_directory 2> error.log # Redirect both output and errors echo "Test" &> combined.log ``` or, if you're feeling extra fancy, you can do some cool stuff with `cat`: ```bash cat <<EOF > myfile.txt # look, ma, a comment! do x and y plz and try ${favorite_piza}! EOF ``` You can also output to devices, like `/dev/null`. As a useful example, to run a find command on your entire (local) system but skip any errors: ```bash find / -xdev -type f -name "*xyz*" -ls 2> /dev/null ``` # Checking connectivity Connectivity checks can be done in a few ways, but the most reliable would be with a pure socket implementation, which has been supported in bash since version 2.04. ## Pure sockets w/ timeout This method uses `timeout` with Bash's `/dev/tcp` feature to check if a port is reachable: ```bash timeout 10 bash -c "cat < /dev/null > /dev/tcp/server/port" &>/dev/null ``` This only works in modern Bash versions where `timeout` is available, however, so it's best to check if `timeout` is available with a quick `if command -v timeout; then` beforehand. ## Pure sockets w/o timeout This method uses `/dev/tcp` but has no set timeout, meaning it will block for the OS's default time if the connection doesn't succeed: ```bash </dev/tcp/server/port &>/dev/null ``` In either case, you can get the result with a simple `$?` on the next line to see the return code. If it's non-zero, you have a problem. # Fancy techniques Sometimes you need some fancy stuff to make your scripts pretty or make it work well. Here's a couple fun ones. ## Getting current script path To reliably determine the current script's path in a one-liner: ```bash SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd)" ``` This method works even if the script is being sourced. Thanks to [this SO answer](https://stackoverflow.com/a/4774063). ## Stripping ANSI colors To strip ANSI color codes from a string: ```bash echo -e "$1" | sed 's/\x1B\[[0-9]*\(\?:;[0-9]\)*\?m//g' ``` This is useful when you want to remove color formatting for logging or processing output, or when you don't have a terminal. You can check for all of these by simply doing a `if [[ -t 1 ]]; then` if/else case. Largely thanks to the answers in [this SO post](https://stackoverflow.com/questions/17998978/removing-colors-from-output), but I had to hand-edit the regex for it to work reliably across multiple versions and OSes. # For your bashrc Here's a few useful things you can throw in your `.bashrc` (or `.bash_profile`) ## enhanced grep ```bash alias grep="grep --color=auto" alias lgrepi="ls -la | grep --color=auto -i" alias rgrepi="find \${PWD} -name '.*' -prune -o -type f -print0 | xargs -r0 -P2 grep --color=auto -HIni" alias grepi="find \${PWD} -maxdepth 1 -name '.*' -prune -o -type f -print0 | xargs -r0 -P2 grep --color=auto -HIni" ``` And a quick summary of the aliases: - `grep` has been updated to add colors to the output when it can - `lgrepi` searches your current directory for filenames matching your query (case-insensitive) - `rgrepi` searches your current directory (and subdirectories) for content matching your query (case-insensitive) - `grepi` is the same as `rgrepi`, but won't recursively search subdirectories ## rclone copy Sometimes you need to copy a bunch of files fast, and with progress bars. `rclone` is a great way to do that but it doesn't really match the syntax of `cp` at all. Here's an `rcp` command that does just that: ```bash function is_remote_mount() { dir="$1" grep -E ' nfs[3,4]* | cifs ' /proc/mounts | awk '{print $2}' | while read -r mount_point; do if [[ "$dir" == "$mount_point"* ]]; then return 0 fi done return 1 } function rclone_cp() { args=("$@") last_index=$(( $# - 1 )) dest="${args[$last_index]}" unset 'args[$last_index]' base_flags="--verbose --update --copy-links" # Enable progress if running in an interactive terminal if [ -t 1 ]; then base_flags="$base_flags --progress" fi temp_dir=$(mktemp -d) echo "Created temporary directory ${temp_dir}" trap 'rm -rf -- "$temp_dir"' EXIT for src in "${args[@]}"; do # Convert relative path to absolute path if [[ "$src" != /* ]]; then src="$PWD/$src" fi if [[ -d "$src" ]] && [[ "$src" != */ ]]; then # If source is a directory and does not end with '/', copy the directory itself ln -s "$src" "$temp_dir/" elif [[ -f "$src" ]]; then # If source is a file ln -s "$src" "$temp_dir/" fi done temp_flags="$base_flags" # Determine if source or destination is a remote mount if is_remote_mount "$dest"; then temp_flags="$temp_flags --transfers=16 --checkers=16 --buffer-size=512M" else temp_flags="$temp_flags --transfers=4 --checkers=4 --buffer-size=256M" fi # Determine if source or destination is remote (via rclone) if [[ "$dest" == *":"* ]]; then temp_flags="$temp_flags --transfers=32 --checkers=32 --buffer-size=1G --fast-list" fi rclone copy "$temp_dir" "$dest" $temp_flags } alias rcp='rclone_cp' ```