Introducing BASH3 Boilerplate

There are many cases where using BASH to code ends up overcomplicating things, but due to it's portability there are many cases that make it the best tool for the job.

When I do hack on BASH scripts, I often find there are things like logging, configuration, commandline argument parsing that:

  • I need everytime
  • Take quite some effort to get right, and
  • Keep you from your actual work.

Here's an attempt at bundling those things in a generalized way so that they are reusable as-is in most of my (and hopefully your) programs.

Goals

I'm going to be pragmatic and be violating some good practices like DRY and the things I know about sourcing, submodules, package management, etc.

I feel that when a program consists of multiple files, BASH may very well not be the right tool for the job.

I propose people copy-paste main.sh, then delete the parts they don't need, and write the rest of their script below ### Runtime.

Copy-pasting and removing what you don't need may not seem very rocket-scientific or 2013, but in fact, neither is BASH.

Keeping things simple, small and lightweight should be key here.

Aiming for portability, I'm targetting BASH 3 (OSX still ships with 3 for instance). If you're going to ask people to install BASH 4 first, you might as well pick a more advanced language as a dependency.

The main template should only have features that are needed 80% of the time you write BASH.

Other cool functions can be adopted by the repository, but should not be referenced by the main.sh template and function purely as a resource.

Features

  • Structure
  • Configuration by environment variables
  • Configuration by commandline arguments (definitions parsed from help info, so no duplication needed)
  • Magic variables like __FILE__ and __DIR__
  • Logging that supports colors and is compatible with Syslog Severity levels

Conventions / Recommendations

  • It's the task of the script's caller to: Redirect the output to appropriate locations
  • It's the task of the script's caller to: Correctly set the PATH variable (think cronjobs)
  • Use spaces vs tabs so you can copy-paste parts directly into the console, without BASH's automcomplete firing on every tab character
  • Use set -eu, so it will break on every error and undefined variable. If you're expecting an error, add ||true to your command. set -eu is better that relying on #!/bin/bash -eu, that will be ignored when sourced or executed like bash main.sh.
  • Checkout The 10 Commandments of Logging
  • Adhere Syslog. Sure, skip notice, alert, critical, if you need just 3 levels. You should still crash in emergency, debug in debug, inform in info. The world has had plenty smart folks creating new cool logstandards, dealing with them does not make life easier though.

Code

#!/bin/bash
#
# Template to write better bash scripts. More info: http://kvz.io
# Version 0.0.1
#
# Usage:
#  LOG_LEVEL=7 ./template.sh first_arg second_arg
#
# Licensed under MIT
# Copyright (c) 2013 Kevin van Zonneveld
# http://twitter.com/kvz
#

### Configuration
#####################################################################

# Environment variables
[ -z "${LOG_LEVEL}" ] && LOG_LEVEL="6" # 7 = debug -> 0 = emergency

# Commandline options. This defines the usage page, and is used to parse cli opts & defaults from.
# the parsing is unforgiving so be precise in your syntax:
read -r -d '' usage <<-'EOF'
  -f   [arg] Filename to process.
  -t   [arg] Location of tempfile. Default="/tmp/x"
  -d         Enables debug mode
  -h         This page
EOF

# Set magic variables for current FILE & DIR
__FILE__="$(test -L "$0" && readlink "$0" || echo "$0")"
__DIR__="$(cd "$(dirname "${__FILE__}")"; echo $(pwd);)"


### Functions
#####################################################################

function _fmt ()      {
  color_ok="\x1b[32m"
  color_bad="\x1b[31m"

  color="${color_bad}"
  if [ "${1}" = "debug" ] || [ "${1}" = "info" ] || [ "${1}" = "notice" ]; then
    color="${color_ok}"
  fi

  color_reset="\x1b[0m"
  if [ "${TERM}" != "xterm" ] || [ -t 1 ]; then
    # Don't use colors on pipes or non-recognized terminals
    color=""
    color_reset=""
  fi
  echo -e "$(date -u +"%Y-%m-%d %H:%M:%S UTC") ${color}$(printf "[%9s]" ${1})${color_reset}";
}
function emergency () { echo "$(_fmt emergency) ${@}" || true; exit 1; }
function alert ()     { [ "${LOG_LEVEL}" -ge 1 ] && echo "$(_fmt alert) ${@}" || true; }
function critical ()  { [ "${LOG_LEVEL}" -ge 2 ] && echo "$(_fmt critical) ${@}" || true; }
function error ()     { [ "${LOG_LEVEL}" -ge 3 ] && echo "$(_fmt error) ${@}" || true; }
function warning ()   { [ "${LOG_LEVEL}" -ge 4 ] && echo "$(_fmt warning) ${@}" || true; }
function notice ()    { [ "${LOG_LEVEL}" -ge 5 ] && echo "$(_fmt notice) ${@}" || true; }
function info ()      { [ "${LOG_LEVEL}" -ge 6 ] && echo "$(_fmt info) ${@}" || true; }
function debug ()     { [ "${LOG_LEVEL}" -ge 7 ] && echo "$(_fmt debug) ${@}" || true; }

function help () {
  echo ""
  echo " ${@}"
  echo ""
  echo "  ${usage}"
  echo ""
  exit 1
}

function cleanup_before_exit () {
  info "Cleaning up. Done"
}
trap cleanup_before_exit EXIT


### Parse commandline options
#####################################################################

# Translate usage string -> getopts arguments, and set $arg_<flag> defaults
while read line; do
  opt="$(echo "${line}" |awk '{print $1}' |sed -e 's#^-##')"
  if ! echo "${line}" |egrep '\[.*\]' >/dev/null 2>&1; then
    init="0" # it's a flag. init with 0
  else
    opt="${opt}:" # add : if opt has arg
    init=""  # it has an arg. init with ""
  fi
  opts="${opts}${opt}"

  varname="arg_${opt:0:1}"
  if ! echo "${line}" |egrep '\. Default=' >/dev/null 2>&1; then
    eval "${varname}=\"${init}\""
  else
    match="$(echo "${line}" |sed 's#^.*Default=\(\)##g')"
    eval "${varname}=\"${match}\""
  fi
done <<< "${usage}"

# Reset in case getopts has been used previously in the shell.
OPTIND=1

# Overwrite $arg_<flag> defaults with the actual CLI options
while getopts "${opts}" opt; do
  line="$(echo "${usage}" |grep "\-${opt}")"


  [ "${opt}" = "?" ] && help "Invalid use of script: ${@} "
  varname="arg_${opt:0:1}"
  default="${!varname}"

  value="${OPTARG}"
  if [ -z "${OPTARG}" ] && [ "${default}" = "0" ]; then
    value="1"
  fi

  eval "${varname}=\"${value}\""
  debug "cli arg ${varname} = ($default) -> ${!varname}"
done

shift $((OPTIND-1))

[ "$1" = "--" ] && shift


### Switches
#####################################################################

# debug mode
if [ "${arg_d}" = "1" ]; then
  set -x
  LOG_LEVEL="7"
fi

# help mode
if [ "${arg_h}" = "1" ]; then
  help "Help using ${0}"
fi


### Validation
#####################################################################

[ -z "${arg_f}" ]     && help   "Setting a filename with -f is required"
[ -z "${LOG_LEVEL}" ] && emergency "Cannot continue without loglevel. "


### Runtime
#####################################################################

# Exit on error. Append ||true if you expect an error.
# set -e is safer than #!/bin/bash -e because that is nutralised if
# someone runs your script like `bash yourscript.sh`
set -e

# Bash will remember & return the highest exitcode in a chain of pipes.
# This way you can catch the error in case mysqldump fails in `mysqldump |gzip`
set -o pipefail

debug "Info useful to developers for debugging the application, not useful during operations."
info "Normal operational messages - may be harvested for reporting, measuring throughput, etc. - no action required."
notice "Events that are unusual but not error conditions - might be summarized in an email to developers or admins to spot potential problems - no immediate action required."
warning "Warning messages, not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full - each item must be resolved within a given time. This is a debug message"
error "Non-urgent failures, these should be relayed to developers or admins; each item must be resolved within a given time."
critical "Should be corrected immediately, but indicates failure in a primary system, an example is a loss of a backup ISP connection."
alert "Should be corrected immediately, therefore notify staff who can fix the problem. An example would be the loss of a primary ISP connection."
emergency "A \"panic\" condition usually affecting multiple apps/servers/sites. At this level it would usually notify all tech staff on call."

Contribute: