[gsu.git] / README.md

Introduction
------------
gsu is a small library of bash functions intended to ease the task of
writing and documenting large shell scripts with multiple subcommands,
each providing different functionality. gsu is known to work on Linux,
FreeBSD, NetBSD and MacOS.

This document describes how to install and use the gsu library.

Setting up gsu
--------------
gsu is very easy to install:

___Requirements___

gsu is implemented in bash, and thus gsu depends on bash. Bash version
4.3 is required. Besides bash, gsu depends only on programs which are
usually installed on any Unix system (awk, grep, sort, ...). Care has
been taken to not rely on GNU specific behavior of these programs,
so it should work on non GNU systems (MacOS, FreeBSD, NetBSD) as
well. The gui module depends on the dialog utility.

___Download___

All gsu modules are contained in a git repository. Get a copy with

                git clone git://git.tuebingen.mpg.de/gsu.git

There is also a [gitweb](http://git.tuebingen.mpg.de/gsu.git) page.

___Installation___

gsu consists of several independent modules which are all located
at the top level directory of the git repository. gsu requires no
installation beyond downloading. In particular it is not necessary
to make the downloaded files executable.  The library modules can
be sourced directly, simply tell your application where to find
it. The examples of this document assume that gsu is installed in
`/usr/local/lib/gsu` but this is not mandatory. `~/.gsu` is another
reasonable choice.

Conventions
-----------
___Public and private functions and variables___

Although there is no way in bash to annotate symbols (functions
and variables) as private or public, gsu distinguishes between the
two. The `gsu_*` name space is reserved for public symbols while all
private symbols start with `_gsu`.

Private symbols are meant for internal use only. Applications should
never use them directly because name and semantics might change
between gsu versions.

The public symbols, on the other hand, define the gsu API. This API
must not change in incompatible ways that would break existing
applications.

___`$ret` and `$result`___

All public gsu functions set the $ret variable to an integer value
to indicate success or failure. As a convention, `$ret < 0` means
failure while a non-negative value indicates success.

The `$result` variable contains either the result of a function (if any)
or further information in the error case. A negative value of `$ret` is
in fact an error code similar to the errno variable used in C programs.
It can be turned into a string that describes the error. The public
`gsu_err_msg()` function can be used to pretty-print a suitable error
message provided `$ret` and `$result` are set appropriately.

The subcommand module
---------------------
This gsu module provides helper functions to ease the repetitious task
of writing applications which operate in several related modes, where
each mode of operation corresponds to a subcommand of the application.

With gsu, for each subcommand one must only write a _command handler_
which is simply a function that implements the subcommand. All
processing is done by the gsu library. Functions starting with the
string `com_` are automatically recognized as subcommand handlers.

The startup part of the script has to source the subcommand file of
gsu and must then call

                gsu "$@"

Minimal example:

                #!/bin/bash
                com_world()
                {
                        echo 'hello world'
                }
                . /usr/local/lib/gsu/subcommand || exit 1
                gsu "$@"

Save this code in a file called `hello` (adjusting the installation
directory if necessary), make it executable (`chmod +x hello`) and try

                ./hello
                ./hello world
                ./hello invalid

Here, we have created a bash script (`hello`) that has a single "mode"
of operation, specified by the subcommand `world`.

gsu automatically generates several reserved subcommands, which should
not be specified: `help, man, prefs, complete`.

___Command handler structure___

For the automatically generated help and man subcommands to work
properly, all subcommand handlers must be documented. In order to be
recognized as subcommand help text, comments must be prefixed with
two `#` characters and the subcommand documentation must be located
between the function "declaration", `com_world()` in the example above,
and the opening brace that starts the function body.

Example:

                com_world()
                ##
                ##
                ##
                {
                        echo 'hello world'
                }

The subcommand documentation consists of the following parts:

- The summary. One line of text,
- the usage/synopsis string,
- free text section 1,
- options (if any),
- free text section 2.

The last three parts are optional. All parts should be separated by
lines consisting of two `#` characters only. Example:

                com_world()
                ##
                ## Print the string "hello world" to stdout.
                ##
                ## Usage: world [-v]
                ##
                ## Any arguments to this function are ignored.
                ##
                ## -v: Enable verbose mode
                ##
                ## Warning: This subcommand may cause the top most line of your terminal to
                ## disappear and may cause DATA LOSS in your scrollback buffer. Use with
                ## caution.
                {
                        printf 'hello world'
                        [[ "$1" == '-v' ]] && printf '!'
                        printf '\n'
                }

Replace `hello` with the above and try:

                ./hello help
                ./hello help world
                ./hello help invalid
                ./hello man

to check the automatically generated help and man subcommands.

___Error codes___

As mentioned above, all public functions of gsu return an error code
in the `$ret` variable. A negative value indicates failure, and in this
case `$result` contains more information about the error. The same
convention applies for subcommand handlers: gsu will automatically
print an error message to stderr if a subcommand handler returns with
`$ret` set to a negative value.

To allow for error codes defined by the application, the
`$gsu_errors` variable must be set before calling `gsu()`. Each
non-empty line in this variable should contain an identifier and error
string. Identifiers are written in upper case and start with `E_`. For
convenience the `$GSU_SUCCESS` variable is defined to non-negative
value. Subcommand handlers should set `$ret` to `$GSU_SUCCESS` on
successful return.

To illustrate the `$gsu_errors` variable, assume the task is to
print all mount points which correspond to an ext3 file system in
`/etc/fstab`. We'd like to catch two possible errors: (a) the file
does not exist or is not readable, and (b) it contains no ext3 entry.
A possible implementation of the ext3 subcommand could look like this
(documentation omitted):

                #!/bin/bash

                gsu_errors='
                        E_NOENT         No such file or directory
                        E_NOEXT3        No ext3 file system detected
                '

                com_ext3()
                {
                        local f='/etc/fstab'
                        local ext3_lines

                        if [[ ! -r "$f" ]]; then
                                ret=-$E_NOENT
                                result="$f"
                                return
                        fi
                        ext3_lines=$(awk '{if ($3 == "ext3") print $2}' "$f")
                        if [[ -z "$ext3_lines" ]]; then
                                ret=-$E_NOEXT3
                                result="$f"
                                return
                        fi
                        printf 'ext3 mount points:\n%s\n' "$ext3_lines"
                        ret=$GSU_SUCCESS
                }

___Printing diagnostic output___

gsu provides a couple of convenience functions for output. All
functions write to stderr.

- `gsu_msg()`. Writes the name of the application and the given text.

- `gsu_short_msg()`. Like `gsu_msg()`, but does not print the application name.

- `gsu_date_msg()`. Writes application name, date, and the given text.

- `gsu_err_msg()`. Prints an error message according to `$ret` and `$result`.

___Subcommands with options___

Bash's getopts builtin provides a way to define and parse command line
options, but it is cumbersome to use because one must loop over all
given arguments and check the `OPTIND` and `OPTARG` variables during
each iteration. The `gsu_getopts()` function makes this repetitive
task easier.

`gsu_getopts()` takes a single argument: the optstring which contains
the option characters to be recognized. As usual, if a character is
followed by a colon, the option is expected to have an argument. On
return `$result` contains bash code that should be eval'ed to parse
the position parameters `$1`, `$2`, ... of the subcommand according
to the optstring.

The shell code returned by `gsu_getopts()` creates a local variable
`$o_x` for each defined option `x`. It contains `true/false` for
options without argument and either the empty string or the given
argument for options that take an argument.

To illustrate `gsu_getopts()`, assume the above `com_ext3()` subcommand
handler is to be extended to allow for arbitrary file systems, and
that it should print either only the mount point as before or the
full line of `/etc/fstab`, depending on whether the verbose switch
`-v` was given at the command line.

Hence our new subcommand handler must recognize two options: `-t` for
the file system type and `-v`. Note that `-t` takes an argument but
`-v` does not. Hence we shall use the optstring `t:v` as the argument
for `gsu_getopts()` as follows:

                com_fs()
                {
                        local f='/etc/fstab'
                        local fstype fstab_lines
                        local -i awk_field=2

                        gsu_getopts 't:v'
                        eval "$result"
                        ((ret < 0)) && return

                        [[ -z "$o_t" ]] && o_t='ext3' # default to ext3 if -t is not given
                        [[ "$o_v" == 'true' ]] && awk_field=0 # $0 is the whole line
                        fstab_lines=$(awk -v fstype="$o_t" -v n="$awk_field" \
                                '{if ($3 == fstype) print $n}' "$f")
                        printf '%s entries:\n%s\n' "$o_t" "$fstab_lines"
                        ret=$GSU_SUCCESS
                }

Another repetitive task is to check the number of non-option arguments
and to report an error if this number turns out to be invalid for the
subcommand in question. The `gsu_check_arg_count()` function performs
this check and sets `$ret` and `$result` as appropriate. This function
takes three arguments: the actual argument count and the minimal and
maximal number of non-option arguments allowed. The last argument may
be omitted in which case any number of arguments is considered valid.

Our `com_world()` subcommand handler above ignored any given
arguments. Let's assume we'd like to handle this case and
print an error message if one or more arguments are given. With
`gsu_check_arg_count()` this can be achieved as follows:

                com_world()
                {
                        gsu_check_arg_count $# 0 0 # no arguments allowed
                        ((ret < 0)) && return
                        echo 'hello world'
                }

___Global documentation___

Besides the documentation for subcommands, one might also want to
include an overall description of the application which provides
general information that is not related to any particular subcommand.

If such a description is included at the top of the script, the
automatically generated man subcommand will print it. gsu recognizes
the description only if it is enclosed by two lines consisting of at
least 70 # characters.

Example:

                #/bin/bash

                #######################################################################
                # gsu-based hello - a cumbersome way to write a hello world program
                # -----------------------------------------------------------------
                # It not only requires one to download and install some totally weird
                # git repo, it also takes about 50 lines of specially written code
                # to perform what a simple echo 'hello world' would do equally well.
                #######################################################################

___HTML output___

The auto-generated man subcommand produces plain text, html, or
roff output.

                ./hello man -m html > index.html

is all it takes to produce an html page for your
application. Similarly,

                ./hello man -m roff > hello.1

creates a manual page.

___Interactive completion___

The auto-generated `complete` subcommand provides interactive bash
completion.  To activate completion for the hello program, it is
enough to put the following into your `~/.bashrc`:

                _hello()
                {
                        eval $(hello complete 2>/dev/null)
                }
                complete -F _hello hello

This will give you completion for the first argument of the hello
program: the subcommand.

In order to get subcommand-sensitive completion you must provide a
_completer_ in your application for each subcommand that is to support
completion. Like subcommand handlers, completers are recognized by name:
If a function `xxx_complete()` is defined, gsu will call it on the
attempt to complete the `xxx` subcommand at the subcommand line. gsu
has a few functions to aid you in writing a completer.

Let's have a look at the completer for the above `fs` subcommand.

                complete_fs()
                {
                        local f='/etc/fstab'
                        local optstring='t:v'

                        gsu_complete_options $optstring "$@"
                        ((ret > 0)) && return

                        gsu_cword_is_option_parameter $optstring "$@"
                        [[ "$result" == 't' ]] && awk '{print $3}' "$f"
                }

Completers are always called with `$1` set to the index into the array
of words in the current command line when tab completion was attempted
(see `COMP_CWORD` in the bash manual). These words are passed to the
completer as `$2`, `$3`,...

`gsu_complete_options()` receives the option string as `$1`, the word
index as `$2` and the individual words as `$3`, `$4`,... Hence we
may simply pass the `$optstring` and `"$@"`. `gsu_complete_options()`
checks if the current word begins with `-`, i.e., whether an attempt
to complete an option was performed. If yes `gsu_complete_options()`
prints all possible command line options and sets `$ret` to a
positive value.

The last two lines of `complete_fs()` check whether the word preceding
the current word is an option that takes an argument. If it is,
that option is returned in `$result`, otherwise `$result` is the empty
string. Hence, if we are completing the argument to `-t`, the awk
command is executed to print all file system types of `/etc/fstab` as
the possible completions.

See the comments to `gsu_complete_options()`,
`gsu_cword_is_option_parameter()` and `gsu_get_unnamed_arg_num()`
(which was not covered here) in the `subcommand` file for a more
detailed description.

The gui module
--------------
This module can be employed to create interactive dialog boxes from a
bash script. It depends on the dialog(1) utility which is available on
all Unix systems. On Debian and Ubuntu Linux it can be installed with

                apt-get install dialog

The core of the gui module is the `gsu_gui()` function which receives
a _menu tree_ as its single argument. The menu tree defines a tree
of menus for the user to navigate with the cursor keys. As for a
file system tree, internal tree nodes represent folders. Leaf nodes,
on the other hand, correspond to _actions_. Pressing enter activates a
node. On activation, for internal nodes a new menu with the contents of
the subfolder is shown. For leaf nodes the associated _action handler_
is executed.

Hence the application has to provide a menu tree and an action handler
for each leaf node defined in the tree. The action handler is simply a
function which is named according to the node. In most cases the action
handler will run dialog(1) to show some dialog box on its own. Wrappers
for some widgets of dialog are provided by the gui module, see below.

___Menu trees___

The concept of a menu tree is best illustrated by an example. Assume
we'd like to write a system utility for the not-so-commandline-affine
Linux sysadmin next door. For the implementation we confine ourselves
with giving some insight in the system by running lean system commands
like `df` to show the list of file system, or `dmesg` to print the
contents of the kernel log buffer. Bash code which defines the menu
tree could look like this:

                menu_tree='
                        load_average System load
                        processes Running processes of a user
                        hardware/ Hardware related information
                                cpu Show prozessor type and features
                                scsi Show SCSI devices
                        storage/ Filesystems and software raid
                                df List of mounted filesystems
                                mdstat Status of software raid arrays
                        log/ System and kernel logs
                                syslog System log
                                dmesg Kernel log
                '

Each line of the menu tree consists of an identifier, suffixed with an
optional slash, and a description. The identifier becomes part of the
name of a bash function and should only contain alphabetic characters
and underscores. The description becomes the text shown as the menu
item. Identifiers suffixed with a slash are regarded as internal nodes
which represent submenus. In the above tree, `hardware/`, `storage/`
and `log/` are internal nodes. All entries of the menu tree must be
properly indented by tab characters.

___Action handlers___

Action handlers are best explained via example:

Our application, let's call it `lsi` for _lean system information_,
must provide action handlers for all leaf nodes. Here is the action
handler for the `df` node:

                lsi_df()
                {
                        gsu_msgbox "$(df -h)"
                }

The function name `lsi_df` is derived from the name of the script
(`lsi`) and the identifier of the leaf node (`df`). The function simply
passes the output of the `df(1)` command as the first argument to the
public gsu function `gsu_msgbox()` which runs dialog(1) to display
a message box that shows the given text.

`gsu_msgbox()` is suitable for small amounts of output. For essentially
unbounded output like log files that can be arbitrary large, it is
better to use `gsu_textbox()` instead which takes a path to the file
that contains the text to show.

To illustrate `gsu_input_box()` function, assume the action handler
for the `processes` leaf node should ask for a username, and display
all processes owned by the given user. This could be implemented
as follows.

                lsi_processes()
                {
                        local username

                        gsu_inputbox 'Enter username' "$LOGNAME"
                        ((ret != 0)) && return
                        username="$result"
                        gsu_msgbox "$(pgrep -lu "$username")"
                }

Once all other action handlers have been defined, the only thing left
to do is to source the gsu gui module and to call `gsu_gui()`:

                . /usr/local/lib/gsu/gui || exit 1
                gsu_gui "$menu_tree"

___Example___

The complete lsi script below can be used as a starting point for your
own gsu gui application. If you cut and paste it, be sure to not turn
tab characters into space characters. The script must be named "lsi".

                #!/bin/bash

                menu_tree='
                        load_average System load
                        processes Running processes of a user
                        hardware/ Hardware related information
                                cpu Show prozessor type and features
                                scsi Show SCSI devices
                        storage/ Filesystems and software raid
                                df List of mounted filesystems
                                mdstat Status of software raid arrays
                        log/ System and kernel logs
                                syslog System log
                                dmesg Kernel log
                '

                lsi_load_average()
                {
                        gsu_msgbox "$(cat /proc/loadavg)"
                }

                lsi_processes()
                {
                        local username

                        gsu_inputbox 'Enter username' "$LOGNAME"
                        ((ret < 0)) && return
                        username="$result"
                        gsu_msgbox "$(pgrep -lu "$username")"
                }

                lsi_cpu()
                {
                        gsu_msgbox "$(lscpu)"
                }

                lsi_scsi()
                {
                        gsu_msgbox "$(lsscsi)"
                }

                lsi_df()
                {
                        gsu_msgbox "$(df -h)"
                }

                lsi_mdstat()
                {
                        gsu_msgbox "$(cat /proc/mdstat)"
                }

                lsi_dmesg()
                {
                        local tmp="$(mktemp)" || exit 1

                        trap "rm -f $tmp" EXIT
                        dmesg > $tmp
                        gsu_textbox "$tmp"
                }

                lsi_syslog()
                {
                        gsu_textbox '/var/log/syslog'
                }

                . /usr/local/lib/gsu/gui || exit 1
                gsu_gui "$menu_tree"

The config module
-----------------
Some applications need config options which are not related to
any particular subcommand, like the URL of a web service, the path
to some data directory, or a default value which is to be used by
several subcommands. Such options do not change frequently and are
hence better stored in a configuration file rather than passed to
every subcommand that needs the information.

The config module of gsu makes it easy to maintain such options and
performs routine tasks like reading and checking the values given in
the config file, or printing out the current configuration. It can
be used stand-alone, or in combination with either the subcommand or
the gui module.

___Defining config options___

To use the config module, you must define the `$gsu_options`
bash array.  Each config option is represented by one slot in this
array. Here is an example which defines two options:

                gsu_options=(
                "
                name=fs_type
                option_type=string
                default_value=ext3
                required=false
                description='file system type to consider'
                help_text='
                        This option is used in various contexts. All
                        subcommands which need a file system type
                        use the value specified here as the default.
                '
                "
                "
                name=limit
                option_type=num
                default_value=3
                required=no
                description='print at most this many lines of output'
                "
                )

Each config option consists of the following fields:

- `name`. This must be a valid bash variable name. Hence no special
characters are allowed.

- `option_type`. Only `string` and `num` are supported but additional
types might be supported in future versions. While string variables
may have arbitrary content, only integers are accepted for variables
of type `num`.

- `default_value`. The value to use if the option was not specified.

- `required`. Whether gsu considers it an error if the option was
not specified. It does not make sense to set this to `true` and set
`default_value` at the same time.

- `description`. Short description of the variable. It is printed by
the `prefs` subcommand.

- `help_text`. Optional long description, also printed by `prefs`.

To enable the config module you must source the config module of gsu
after `$gsu_options` has been defined:

                . /usr/local/lib/gsu/config || exit 1

___Passing config options to the application___

There are two ways to pass the value of an option to a gsu application:
environment variable and config file. The default config file is
`~/.$gsu_name.rc` where `$gsu_name` is the basename of the application,
but this can be changed by setting `$gsu_config_file`. Thus, the
following two statements are equivalent

                fs_type=xfs hello fs
                echo 'fs_type=xfs' > ~/.hello.rc && hello fs

If an option is set both in the environment and in the config file,
the environment takes precedence.

The `$gsu_config_file` variable can actually contain more than one
filename, separated by spaces. The config files are processed in
order, so that an option that is specified in the second config file
overwrites the definition given in the first. This is useful for
applications which implement a system-wide config file in addition
to a per-user config file.

___Checking config options___

The gsu config module defines two public functions for this purpose:
`gsu_check_options()` and `gsu_check_options_or_die()`. The latter
function exits on errors while the former function only sets `$ret`
and `$result` as appropriate and lets the application deal with the
error. The best place to call one of these functions is after sourcing
the config module but before calling `gsu()` or `gsu_gui()`.

___Using config values___

The name of an option as specified in `$gsu_options` (`fs_type` in
the example above) is what users of your application may specify at
the command line or in the config file. This leads to a mistake that
is easy to make and difficult to debug: The application might use a
variable name which is also a config option.

To reduce the chance for this to happen, `gsu_check_options()` creates
a different set of variables for the application where each variable
is prefixed with `${gsu_name}`. For example, if `$gsu_options` as above
is part of the hello script, `$hello_fs_type` and `$hello_limit` are
defined after `gsu_check_options()` returned successfully. Only the
prefixed variants are guaranteed to contain the proper value, so this
variable should be used exclusively in the application. The
prefix may be changed by setting `$gsu_config_var_prefix` before calling
`gsu_check_options()`.

___com_prefs()___

For scripts which source both the subcommand and the config module, the
auto-generated `prefs` subcommand prints out the current configuration
and exits. The description and help text of the option as specified
in the `description` and `help_text` fields of `$gsu_options` are shown
as comments in the output. Hence this output can be used as a template
for the config file.

List of public variables
------------------------
- `$gsu_dir`. Where gsu is installed. If unset, gsu guesses
its installation directory by examining the `$BASH_SOURCE` array.

- `$gsu_name`. The name of the application. Defaults to `$0` with
all leading directories removed.

- `$gsu_banner_txt`. Used by both the subcommand and the gui
module. It is printed by the man subcommand, and as the title for
dialog windows.

- `$gsu_errors`. Identifier/text pairs for custom error reporting.

- `$gsu_config_file`. The name of the config file of the application.
Defaults to `~/.${gsu_name}.rc`.

- `$gsu_options`. Array of config options, used by the config module.

- `$gsu_config_var_prefix`. Used by the config module to set up
the variables defined in `$gsu_options`.

- `$gsu_package`. Text shown at the bottom left of the man page,
usually the name and version number of the software package. Defaults
to `$gsu_name`.

License
-------
gsu is licensed under the  GNU LESSER GENERAL PUBLIC LICENSE (LGPL), version 3.
See COPYING and COPYING.LESSER.

Contact
-------
Send beer, pizza, patches, improvements, bug reports, flames,
(in this order), to Andre Noll <maan@tuebingen.mpg.de>.

References
----------
- [bash](http://www.gnu.org/software/bash/bash.html)
- [dialog](http://www.invisible-island.net/dialog/dialog.html)
- [The author's home page](http://people.tuebingen.mpg.de/maan/)