- Published on
Best Practices for Writing Bash Scripts
- Authors
- Name
- Kevin van Zonneveld
- @kvz
This project now has its own homepage at bash3boilerplate.sh.
I recently tweeted a few best practices that I picked up over the years and got some good feedback. I decided to write them all down in a blogpost. Here goes
-
Use long options (
logger --priorityvslogger -p). If you're on cli, abbreviations make sense for efficiency. but when you're writing reusable scripts a few extra keystrokes will pay off in readability and avoid ventures into man pages in the future by you or your collaborators. -
Use
set -o errexit(a.k.a.set -e) to make your script exit when a command fails. -
Then add
|| trueto commands that you allow to fail. -
Use
set -o nounset(a.k.a.set -u) to exit when your script tries to use undeclared variables. -
Use
set -o xtrace(a.k.aset -x) to trace what gets executed. Useful for debugging. -
Use
set -o pipefailin scripts to catchmysqldumpfails in e.g.mysqldump |gzip. The exit status of the last command that threw a non-zero exit code is returned. -
#!/usr/bin/env bashis more portable than#!/bin/bash. -
Avoid using
#!/usr/bin/env bash -e(vsset -e), because when someone runs your script asbash ./script.sh, the exit on error will be ignored. -
Surround your variables with
{}. Otherwise bash will try to access the$ENVIRONMENT_appvariable in/srv/$ENVIRONMENT_app, whereas you probably intended/srv/${ENVIRONMENT}_app. -
You don't need two equal signs when checking
if [ "${NAME}" = "Kevin" ]. -
Surround your variable with
"inif [ "${NAME}" = "Kevin" ], because if$NAMEisn't declared, bash will throw a syntax error (also seenounset). -
Use
:-if you want to test variables that could be undeclared. For instance:if [ "${NAME:-}" = "Kevin" ]will set$NAMEto be empty if it's not declared. You can also set it tononamelike soif [ "${NAME:-noname}" = "Kevin" ] -
Set magic variables for current file, basename, and directory at the top of your script for convenience.
Summarizing, why not start your next bash script like this:
#!/usr/bin/env bash
# Bash3 Boilerplate. Copyright (c) 2014, kvz.io
set -o errexit
set -o pipefail
set -o nounset
# set -o xtrace
# Set magic variables for current file & dir
__dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
__file="${__dir}/$(basename "${BASH_SOURCE[0]}")"
__base="$(basename ${__file} .sh)"
__root="$(cd "$(dirname "${__dir}")" && pwd)" # ← change this as it depends on your app
arg1="${1:-}"
If you have additional tips, please share and I'll update this post.
Legacy Comments (38)
These comments were imported from the previous blog system (Disqus).
7. /bin/sh is more portable. Learn to write portable shell script, not portable bash script.
10. No, [[ needs two equals. Two equals with the command "test" will failed.
11. You should write [ "x$NAME" = "xKevin" ] to be portable. Always initialize variable.
shell is very limited and not even the subject. the titles says BASH best practices. Loosen your geek suspenders
My bash works with either a single or double '=' character in the test, [ ... ], and [[ ... ]] instances. I don't know if this is a recent Bash change or not.
You don't need to use the archaic [[ "x${NAME}" = "xKevin" ]] ... test if you remember to quote the variable. The 'x' was used if you don't quote the variable and the variable happens to be Null or undefined. It is unnecessary if you remember to quote the variables.
Nice, I came to use many of the same tips over time, wish I had known them all since the beginning.
Otoh now I write all my scripts in higher level languages and do not regret shell scripts at all :-)
I think the correct option for exiting on undefined variables is 'nounset' rather than 'nounsets'. So it should be 'set -o nounset'
thanks for the Tips Kevin, looking at your profile and experience for shell scripting I would like to ask is there some prefered pattern you use when you specify functions, variables?
If the script is sourced, $0 will not be correct. Instead, use:
[[ "${BASH_SOURCE[0]}" != "${0}" ]] && __SELF__=`basename "${BASH_SOURCE[0]}"` || __SELF__=`basename "${0}"`;
__DIR__="$(cd "$(dirname "${__SELF__}")"; echo $(pwd))";
__BASE__="$(basename "${__SELF__}")";
Thanks, it's fixed in the article!
I noticed when I try to set the '_dir' variable, it doesn't work if there are spaces in the names of any of the directories.
For example, if I tried to set _dir, and the directory's name was "Test Application", and then tried to run 'ls ${_dir}', it would run 'ls [PathToDir]Test Application', in which case it would say:
ls: cannot access /[PathToDir]/Test: No such file or directory
ls: cannot access Application: No such file or directory
Is there any way to fix that?
Yes, with any kind of command, always put the path inside quotes. In this case:
`ls "${__dir}"`
This is true for all path variables, not just the $__dir one.
One problem here, running diff on files that diff exits the script ;)
Catch expected nonzero exit codes with an if statement, or with || true
ok I found the issue. pipefail doesn't affect errexit, I had a diff with grep which normally exits with 0 code even if the files differ. With pipefail it exits with 1, but the script continues to work as errexit doesnt catch that
You could use ${PIPESTATUS[@]} to see the results of elements in a pipe. I've posted a simple example to test pipe results that may help.
Nice article, it will help me a lot in the future !
However, I don't understand the last line in the boilerplate code example.
Can you explain this special form "${1:-}" ?
Found myself the answer. Simply RTFM
From bash manual"${parameter:-word}
Use Default Values. If
parameter
is unset or null, the expansion of
word
is substituted. Otherwise, the value of
parameter
is substituted."
this is excellent. one thing i might suggest is to create a test script that provides some examples of how to use things like arguments etc. while there are people who wont benefit from this, im sure there are many more people who will benefit.
We have, it's called BASH 3 Boilerplate and can be found here https://github.com/kvz/bash... : )
What is the intended purpose of `__root`, and why is it marked "change this"?
The root of your project. Often useful when referring to other files. But since this depends on the script's location, best change it accordingly
Very helpful. Just found out that 'errexit' is ignored in functions, so it might be a good idea to also set 'errtrace'.
I recommand to set locale in script :
(if u want utf8 ans en_US: )
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
Ok, Why ?
For example GNU Grep regex compile change from locale. (or no locale)
test :
$ echo a |egrep -o '[a-Z]'
a
and test:
$ unset LC_MESSAGES ; unset LC_ALL; unset LANG; echo a |egrep -o '[a-Z]'
"grep: Invalid range end"
Okay regex '[a-Z]' can be change, but if you dont want / cant change...?
For unset locale, or some local grep change (not really why yet). If you use a client ssh in python for example paramiko for exec some command on server. You will have some issue with some grep regex.
And maybe other utils can fail without right locale.
#sorryforenglish.
a-Z is the problem, use the predefined classes to write portable code.
Very need! I use
__file=$(readlink -f "$0")
and
__dir=$(dirname "$__file") are there any disadvantages about that?
When echoing error messages user a line like the following:
echo -e "\n\tERROR[${LINENO}]: This *(^&*^% thing isn't working!\n"
The number in the [] will show the line number in the code that failed and make it easier to debug. The error message details following the ':' is up to the user to select.
ls -lad ${GOOD_DIR} | cat -
[[ "${PIPESTATUS[@]}" != [0\ ]* ]] && echo -e "Bad" 1>&2
The PIPESTATUS array will contain the return code from each element of a pipe; while ${?} will contain only the return code of the last element of a pipe. This will test all the values in the pipe at one time. Anything other than zero return code will result in executing the echo statement (i.e. [[ ... ]] && <cmd> is a cheap and dirty if statement).
Use the "source" command instead of "." which is easy to overlook.
Use ${0##*/} instead of the "basename" command as it is faster (beware of scripts that are sourced though[ e.g. "source junk.sh"] as ${0} will not be what you expect).
If you are not sure whether or not your script will be sourced, use ${BASH_SOURCE[0]##*/} all the time.
Creating directories is an atomic process. If you need to set an exclusive lock for a process use something like:
if mkdir -m 754 ${LOCK} 2>/dev/null
then
echo "Lock established."
else
echo "Lock failed."
fi
If the lock directory doesn't exist, it will be created. If the lock directory does exist, the if statement will fail. An example of implementing this (including a random wait time span) is included here.
#####
# Variables, etc.
#####
26 set -o errtrace
27 typeset -i RET=0
28 typeset -i CNT_LIMIT=10 SLEEP=30 CNT
29 typeset -r NAME="${BASH_SOURCE[0]##*/}"
30 typeset -r LOCK="/tmp/${NAME}.lck"
31 typeset LOCK_MESSAGE="# Please wait patiently..."
...
#####
# Set Lock
#####
71 for (( CNT=1; CNT<=${CNT_LIMIT}; CNT++ ))
72 do
73
74 # Set lock.
75 if mkdir -m 754 ${LOCK} 2>/dev/null
76 then
77 # No matter how you exit the process,
78 # remove the lock file and print the date and time and execution time.
79 trap "(( RET+=\${?} )); rm -rf ${LOCK}; echo -e \"# Info: Lock Removed\n# Info: Returning: \${RET}\"; date \"+# Remote End Time: %m/%d/%Y %R (\${SECONDS} Sec
80 echo "${$}" > ${LOCK}/PID
81 echo -e "# Info: \"${LOCK}\" Lock established."
82 break
83 else
84 ps h -p $(<${LOCK}/PID) || { LOCK_MESSAGE="# ERROR[${LINENO}]: That process is not running."; ((RET++)); }
85 echo -e "# NOTE: a lock from process \"$(<${LOCK}/PID)\" already exists.\n${LOCK_MESSAGE}"
86 fi
87 ((RET)) && break
88 # Sleep 30-60 seconds before trying again.
89 sleep $(expr ${SLEEP} + ${RANDOM} % \( ${SLEEP} + 1 \))
90 done
91 [[ \${CNT} -gt 10 ]] && ((RET++))
92 ((RET)) && { echo -e "# ERROR[${LINENO}]: Unable to establish a lock.\n\t# You need to check on the \"${LOCK}\" lock directory."; exit ${RET}; }
93 unset CNT
#####
# Main Logic
#####
Put the main logic of the code after the previous code lines. If a lock can't be established in ten tries, the code will exit with a non-zero return code. The "trap" is set to make sure the lock is removed no matter how you exit the script. The "set -o errtrace" allows the trap to be seen in any functions you might call later (traps are normally reset in functions).
The process id is saved in a file under the lock directory in case something weird happens and the lock directory is left in place, but the script dies. This lock code will try to read the PID file and check to see if the process is still running. if the PID file doesn't exist, you'll see an error message. Resolving this possible issue is an exercise left for the user to resolve (shouldn't happen, but weird things do happen sometimes).
The variables in the trap command have the '$' characters escaped because I don't want their value at the time the trap is set, I want the value when the trap is actually executed.
Use "$(...)" instead of "`...`" (i.e. back ticks). This is easier to see and match when debugging.
Use "$(<${FILE})" to put the contents of a file in the command line itself.
"$(cat ${FILE})" is equivalent to "$(<${FILE})".
It really is a lot easier to read:
myfile=`ls -l | grep `date``
myfile=$(ls -l | grep $(date))
Use here documents with ssh commands for long sets of commands.
typeset -i VAR=1
ssh host_name <<-!EOF_0
typeset -i VAR_1=3
cd /some/directory
ls -la
cd /some/other/directory
ls -la
echo -e "Doing some work. ${VAR}, \${VAR_1}"
....
!EOF_0
If you use variables in the ssh command list, you don't have to escape the '$' unless the variable value changes within the here document. If the variable value changes in the here document, you must escape the '$' character. The reason is that the local shell will interpret the variables before handing the here document contents to the ssh command. Escaping the '$' (e.g. "\${VAR}") will pass the literal ${VAR} in the script and the remote shell interpreter will take care of the value substitution.
I would (with all due respect) not agree with #10 as a 'best practice'. Though it is true that only an "=" is needed for string equality comparison, the "=" is also used as the assignment operator. Hence, so as to reduce possible confusion, I would use "==" for string equality . It so happens that "==" is just a synonym for "=" but I prefer not to use the same operator for multiple purposes.
with all due respect, "==" is not for string equality check if you are using only 1 bracket "[]". The "==" is only valid syntax if you are using two: "[[ $expr == $target ]]". In other words, "==" is syntax error(!) in old format.
But isn't using double brackets better than using single brackets?
better is relative, depends what you have to do. [ "string" = "string" ] doesn't need the overhead of "[["
I generally just use
__dir=$(dirname $(readlink -nf $0))
__dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
__file="${__dir}/$(basename "${BASH_SOURCE[0]}")"
__base="$(basename ${__file} .sh)"
This code should be replaced with the following. These suggestions should run faster and don't involve spining up subshells for no reason. The usual way to write variable names is with majescule, not miniscule, characters. You might also try using ${0} in place of ${BASH_SOURCE[0]}. Lastly you will need to define or typeset your variables since you are using the "nounset" option.
__DIR="${BASH_SOURCE[0]%/*}"
__BASE="${BASH_SOURCE[0]##*/}"
__FILE="${BASH_SOURCE[0]}"
I find that is you use function in your code this helps alot to help in debugging your code
export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
This will print the the function name and the line number of the the script.
This really helps if you use lots of functions in your code