require bash-4.3
[gsu.git] / README.md
1 Introduction
2 ------------
3 gsu is a small library of bash functions intended to ease the task of
4 writing and documenting large shell scripts with multiple subcommands,
5 each providing different functionality. gsu is known to work on Linux,
6 FreeBSD, NetBSD and MacOS.
7
8 This document describes how to install and use the gsu library.
9
10 Setting up gsu
11 --------------
12 gsu is very easy to install:
13
14 ___Requirements___
15
16 gsu is implemented in bash, and thus gsu depends on bash. Bash version
17 4.3 is required. Besides bash, gsu depends only on programs which are
18 usually installed on any Unix system (awk, grep, sort, ...). Care has
19 been taken to not rely on GNU specific behavior of these programs,
20 so it should work on non GNU systems (MacOS, FreeBSD, NetBSD) as
21 well. The gui module depends on the dialog utility.
22
23 ___Download___
24
25 All gsu modules are contained in a git repository. Get a copy with
26
27 git clone git://git.tuebingen.mpg.de/gsu.git
28
29 There is also a [gitweb](http://git.tuebingen.mpg.de/gsu.git) page.
30
31 ___Installation___
32
33 gsu consists of several independent modules which are all located
34 at the top level directory of the git repository. gsu requires no
35 installation beyond downloading. In particular it is not necessary
36 to make the downloaded files executable. The library modules can
37 be sourced directly, simply tell your application where to find
38 it. The examples of this document assume that gsu is installed in
39 `/usr/local/lib/gsu` but this is not mandatory. `~/.gsu` is another
40 reasonable choice.
41
42 Conventions
43 -----------
44 ___Public and private functions and variables___
45
46 Although there is no way in bash to annotate symbols (functions
47 and variables) as private or public, gsu distinguishes between the
48 two. The `gsu_*` name space is reserved for public symbols while all
49 private symbols start with `_gsu`.
50
51 Private symbols are meant for internal use only. Applications should
52 never use them directly because name and semantics might change
53 between gsu versions.
54
55 The public symbols, on the other hand, define the gsu API. This API
56 must not change in incompatible ways that would break existing
57 applications.
58
59 ___`$ret` and `$result`___
60
61 All public gsu functions set the $ret variable to an integer value
62 to indicate success or failure. As a convention, `$ret < 0` means
63 failure while a non-negative value indicates success.
64
65 The `$result` variable contains either the result of a function (if any)
66 or further information in the error case. A negative value of `$ret` is
67 in fact an error code similar to the errno variable used in C programs.
68 It can be turned into a string that describes the error. The public
69 `gsu_err_msg()` function can be used to pretty-print a suitable error
70 message provided `$ret` and `$result` are set appropriately.
71
72 The subcommand module
73 ---------------------
74 This gsu module provides helper functions to ease the repetitious task
75 of writing applications which operate in several related modes, where
76 each mode of operation corresponds to a subcommand of the application.
77
78 With gsu, for each subcommand one must only write a _command handler_
79 which is simply a function that implements the subcommand. All
80 processing is done by the gsu library. Functions starting with the
81 string `com_` are automatically recognized as subcommand handlers.
82
83 The startup part of the script has to source the subcommand file of
84 gsu and must then call
85
86 gsu "$@"
87
88 Minimal example:
89
90 #!/bin/bash
91 com_world()
92 {
93 echo 'hello world'
94 }
95 . /usr/local/lib/gsu/subcommand || exit 1
96 gsu "$@"
97
98 Save this code in a file called `hello` (adjusting the installation
99 directory if necessary), make it executable (`chmod +x hello`) and try
100
101 ./hello
102 ./hello world
103 ./hello invalid
104
105 Here, we have created a bash script (`hello`) that has a single "mode"
106 of operation, specified by the subcommand `world`.
107
108 gsu automatically generates several reserved subcommands, which should
109 not be specified: `help, man, prefs, complete`.
110
111 ___Command handler structure___
112
113 For the automatically generated help and man subcommands to work
114 properly, all subcommand handlers must be documented. In order to be
115 recognized as subcommand help text, comments must be prefixed with
116 two `#` characters and the subcommand documentation must be located
117 between the function "declaration", `com_world()` in the example above,
118 and the opening brace that starts the function body.
119
120 Example:
121
122 com_world()
123 ##
124 ##
125 ##
126 {
127 echo 'hello world'
128 }
129
130 The subcommand documentation consists of three parts:
131
132 - The summary. One line of text,
133 - the usage/synopsis string,
134 - free text section.
135
136 The three parts should be separated by lines consisting of two `#` characters
137 only. Example:
138
139 com_world()
140 ##
141 ## Print the string "hello world" to stdout.
142 ##
143 ## Usage: world
144 ##
145 ## Any arguments to this function are ignored.
146 ##
147 ## Warning: This subcommand may cause the top most line of your terminal to
148 ## disappear and may cause DATA LOSS in your scrollback buffer. Use with
149 ## caution.
150 {
151 echo 'hello world'
152 }
153
154 Replace `hello` with the above and try:
155
156 ./hello help
157 ./hello help world
158 ./hello help invalid
159 ./hello man
160
161 to check the automatically generated help and man subcommands.
162
163 ___Error codes___
164
165 As mentioned above, all public functions of gsu return an error code
166 in the `$ret` variable. A negative value indicates failure, and in this
167 case `$result` contains more information about the error. The same
168 convention applies for subcommand handlers: gsu will automatically
169 print an error message to stderr if a subcommand handler returns with
170 `$ret` set to a negative value.
171
172 To allow for error codes defined by the application, the
173 `$gsu_errors` variable must be set before calling `gsu()`. Each
174 non-empty line in this variable should contain an identifier and error
175 string. Identifiers are written in upper case and start with `E_`. For
176 convenience the `$GSU_SUCCESS` variable is defined to non-negative
177 value. Subcommand handlers should set `$ret` to `$GSU_SUCCESS` on
178 successful return.
179
180 To illustrate the `$gsu_errors` variable, assume the task is to
181 print all mount points which correspond to an ext3 file system in
182 `/etc/fstab`. We'd like to catch two possible errors: (a) the file
183 does not exist or is not readable, and (b) it contains no ext3 entry.
184 A possible implementation of the ext3 subcommand could look like this
185 (documentation omitted):
186
187 #!/bin/bash
188
189 gsu_errors='
190 E_NOENT No such file or directory
191 E_NOEXT3 No ext3 file system detected
192 '
193
194 com_ext3()
195 {
196 local f='/etc/fstab'
197 local ext3_lines
198
199 if [[ ! -r "$f" ]]; then
200 ret=-$E_NOENT
201 result="$f"
202 return
203 fi
204 ext3_lines=$(awk '{if ($3 == "ext3") print $2}' "$f")
205 if [[ -z "$ext3_lines" ]]; then
206 ret=-$E_NOEXT3
207 result="$f"
208 return
209 fi
210 printf 'ext3 mount points:\n%s\n' "$ext3_lines"
211 ret=$GSU_SUCCESS
212 }
213
214 ___Printing diagnostic output___
215
216 gsu provides a couple of convenience functions for output. All
217 functions write to stderr.
218
219 - `gsu_msg()`. Writes the name of the application and the given text.
220
221 - `gsu_short_msg()`. Like `gsu_msg()`, but does not print the application name.
222
223 - `gsu_date_msg()`. Writes application name, date, and the given text.
224
225 - `gsu_err_msg()`. Prints an error message according to `$ret` and `$result`.
226
227 ___Subcommands with options___
228
229 Bash's getopts builtin provides a way to define and parse command line
230 options, but it is cumbersome to use because one must loop over all
231 given arguments and check the `OPTIND` and `OPTARG` variables during
232 each iteration. The `gsu_getopts()` function makes this repetitive
233 task easier.
234
235 `gsu_getopts()` takes a single argument: the optstring which contains
236 the option characters to be recognized. As usual, if a character is
237 followed by a colon, the option is expected to have an argument. On
238 return `$result` contains bash code that should be eval'ed to parse
239 the position parameters `$1`, `$2`, ... of the subcommand according
240 to the optstring.
241
242 The shell code returned by `gsu_getopts()` creates a local variable
243 `$o_x` for each defined option `x`. It contains `true/false` for
244 options without argument and either the empty string or the given
245 argument for options that take an argument.
246
247 To illustrate `gsu_getopts()`, assume the above `com_ext3()` subcommand
248 handler is to be extended to allow for arbitrary file systems, and
249 that it should print either only the mount point as before or the
250 full line of `/etc/fstab`, depending on whether the verbose switch
251 `-v` was given at the command line.
252
253 Hence our new subcommand handler must recognize two options: `-t` for
254 the file system type and `-v`. Note that `-t` takes an argument but
255 `-v` does not. Hence we shall use the optstring `t:v` as the argument
256 for `gsu_getopts()` as follows:
257
258 com_fs()
259 {
260 local f='/etc/fstab'
261 local fstype fstab_lines
262 local -i awk_field=2
263
264 gsu_getopts 't:v'
265 eval "$result"
266 (($ret < 0)) && return
267
268 [[ -z "$o_t" ]] && o_t='ext3' # default to ext3 if -t is not given
269 [[ "$o_v" == 'true' ]] && awk_field=0 # $0 is the whole line
270 fstab_lines=$(awk -v fstype="$o_t" -v n="$awk_field" \
271 '{if ($3 == fstype) print $n}' "$f")
272 printf '%s entries:\n%s\n' "$o_t" "$fstab_lines"
273 ret=$GSU_SUCCESS
274 }
275
276 Another repetitive task is to check the number of non-option arguments
277 and to report an error if this number turns out to be invalid for the
278 subcommand in question. The `gsu_check_arg_count()` function performs
279 this check and sets `$ret` and `$result` as appropriate. This function
280 takes three arguments: the actual argument count and the minimal and
281 maximal number of non-option arguments allowed. The last argument may
282 be omitted in which case any number of arguments is considered valid.
283
284 Our `com_world()` subcommand handler above ignored any given
285 arguments. Let's assume we'd like to handle this case and
286 print an error message if one or more arguments are given. With
287 `gsu_check_arg_count()` this can be achieved as follows:
288
289 com_world()
290 {
291 gsu_check_arg_count $# 0 0 # no arguments allowed
292 (($ret < 0)) && return
293 echo 'hello world'
294 }
295
296 ___Global documentation___
297
298 Besides the documentation for subcommands, one might also want to
299 include an overall description of the application which provides
300 general information that is not related to any particular subcommand.
301
302 If such a description is included at the top of the script, the
303 automatically generated man subcommand will print it. gsu recognizes
304 the description only if it is enclosed by two lines consisting of at
305 least 70 # characters.
306
307 Example:
308
309 #/bin/bash
310
311 #######################################################################
312 # gsu-based hello - a cumbersome way to write a hello world program
313 # -----------------------------------------------------------------
314 # It not only requires one to download and install some totally weird
315 # git repo, it also takes about 50 lines of specially written code
316 # to perform what a simple echo 'hello world' would do equally well.
317 #######################################################################
318
319 ___HTML output___
320
321 The output of the auto-generated man subcommand is a suitable input for the
322 grutatxt plain text to html converter. Hence
323
324 ./hello man | grutatxt > index.html
325
326 is all it takes to produce an html page for your application.
327
328 ___Interactive completion___
329
330 The auto-generated `complete` subcommand provides interactive bash
331 completion. To activate completion for the hello program, it is
332 enough to put the following into your `~/.bashrc`:
333
334 _hello()
335 {
336 eval $(hello complete 2>/dev/null)
337 }
338 complete -F _hello hello
339
340 This will give you completion for the first argument of the hello
341 program: the subcommand.
342
343 In order to get subcommand-sensitive completion you must provide a
344 _completer_ in your application for each subcommand that is to support
345 completion. Like subcommand handlers, completers are recognized by name:
346 If a function `xxx_complete()` is defined, gsu will call it on the
347 attempt to complete the `xxx` subcommand at the subcommand line. gsu
348 has a few functions to aid you in writing a completer.
349
350 Let's have a look at the completer for the above `fs` subcommand.
351
352 complete_fs()
353 {
354 local f='/etc/fstab'
355 local optstring='t:v'
356
357 gsu_complete_options $optstring "$@"
358 (($ret > 0)) && return
359
360 gsu_cword_is_option_parameter $optstring "$@"
361 [[ "$result" == 't' ]] && awk '{print $3}' "$f"
362 }
363
364 Completers are always called with `$1` set to the index into the array
365 of words in the current command line when tab completion was attempted
366 (see `COMP_CWORD` in the bash manual). These words are passed to the
367 completer as `$2`, `$3`,...
368
369 `gsu_complete_options()` receives the option string as `$1`, the word
370 index as `$2` and the individual words as `$3`, `$4`,... Hence we
371 may simply pass the `$optstring` and `"$@"`. `gsu_complete_options()`
372 checks if the current word begins with `-`, i.e., whether an attempt
373 to complete an option was performed. If yes `gsu_complete_options()`
374 prints all possible command line options and sets `$ret` to a
375 positive value.
376
377 The last two lines of `complete_fs()` check whether the word preceding
378 the current word is an option that takes an argument. If it is,
379 that option is returned in `$result`, otherwise `$result` is the empty
380 string. Hence, if we are completing the argument to `-t`, the awk
381 command is executed to print all file system types of `/etc/fstab` as
382 the possible completions.
383
384 See the comments to `gsu_complete_options()`,
385 `gsu_cword_is_option_parameter()` and `gsu_get_unnamed_arg_num()`
386 (which was not covered here) in the `subcommand` file for a more
387 detailed description.
388
389 The gui module
390 --------------
391 This module can be employed to create interactive dialog boxes from a
392 bash script. It depends on the dialog(1) utility which is available on
393 all Unix systems. On Debian and Ubuntu Linux it can be installed with
394
395 apt-get install dialog
396
397 The core of the gui module is the `gsu_gui()` function which receives
398 a _menu tree_ as its single argument. The menu tree defines a tree
399 of menus for the user to navigate with the cursor keys. As for a
400 file system tree, internal tree nodes represent folders. Leaf nodes,
401 on the other hand, correspond to _actions_. Pressing enter activates a
402 node. On activation, for internal nodes a new menu with the contents of
403 the subfolder is shown. For leaf nodes the associated _action handler_
404 is executed.
405
406 Hence the application has to provide a menu tree and an action handler
407 for each leaf node defined in the tree. The action handler is simply a
408 function which is named according to the node. In most cases the action
409 handler will run dialog(1) to show some dialog box on its own. Wrappers
410 for some widgets of dialog are provided by the gui module, see below.
411
412 ___Menu trees___
413
414 The concept of a menu tree is best illustrated by an example. Assume
415 we'd like to write a system utility for the not-so-commandline-affine
416 Linux sysadmin next door. For the implementation we confine ourselves
417 with giving some insight in the system by running lean system commands
418 like `df` to show the list of file system, or `dmesg` to print the
419 contents of the kernel log buffer. Bash code which defines the menu
420 tree could look like this:
421
422 menu_tree='
423 load_average
424 processes
425 hardware/
426 cpu
427 scsi
428 storage/
429 df
430 mdstat
431 log/
432 syslog
433 dmesg
434 '
435
436 In this tree, `hardware/`, `block_devices/` and `log/` are the only
437 internal nodes. Note that these are written with a trailing slash
438 character while the leaf nodes have no slash at the end. All entries
439 of the menu tree must be indented by tab characters.
440
441 ___Action handlers___
442
443 Action handlers are best explained via example:
444
445 Our application, let's call it `lsi` for _lean system information_,
446 must provide action handlers for all leaf nodes. Here is the action
447 handler for the `df` node:
448
449 lsi_df()
450 {
451 gsu_msgbox "$(df -h)"
452 }
453
454 The function name `lsi_df` is derived from the name of the script
455 (`lsi`) and the name of the leaf node (`df`). The function simply
456 passes the output of the `df(1)` command as the first argument to the
457 public gsu function `gsu_msgbox()` which runs dialog(1) to display
458 a message box that shows the given text.
459
460 `gsu_msgbox()` is suitable for small amounts of output. For essentially
461 unbounded output like log files that can be arbitrary large, it is
462 better to use `gsu_textbox()` instead which takes a path to the file
463 that contains the text to show.
464
465 To illustrate `gsu_input_box()` function, assume the action handler
466 for the `processes` leaf node should ask for a username, and display
467 all processes owned by the given user. This could be implemented
468 as follows.
469
470 lsi_processes()
471 {
472 local username
473
474 gsu_inputbox 'Enter username' "$LOGNAME"
475 (($ret != 0)) && return
476 username="$result"
477 gsu_msgbox "$(pgrep -lu "$username")"
478 }
479
480 Once all other action handlers have been defined, the only thing left
481 to do is to source the gsu gui module and to call `gsu_gui()`:
482
483 . /usr/local/lib/gsu/gui || exit 1
484 gsu_gui "$menu_tree"
485
486 ___Example___
487
488 The complete lsi script below can be used as a starting point
489 for your own gsu gui application. If you cut and paste it, be
490 sure to not turn tab characters into space characters.
491
492 #!/bin/bash
493
494 menu_tree='
495 load_average
496 processes
497 hardware/
498 cpu
499 scsi
500 storage/
501 df
502 mdstat
503 log/
504 syslog
505 dmesg
506 '
507
508 lsi_load_average()
509 {
510 gsu_msgbox "$(cat /proc/loadavg)"
511 }
512
513 lsi_processes()
514 {
515 local username
516
517 gsu_inputbox 'Enter username' "$LOGNAME"
518 (($ret < 0)) && return
519 username="$result"
520 gsu_msgbox "$(pgrep -lu "$username")"
521 }
522
523 lsi_cpu()
524 {
525 gsu_msgbox "$(lscpu)"
526 }
527
528 lsi_scsi()
529 {
530 gsu_msgbox "$(lsscsi)"
531 }
532
533 lsi_df()
534 {
535 gsu_msgbox "$(df -h)"
536 }
537
538 lsi_mdstat()
539 {
540 gsu_msgbox "$(cat /proc/mdstat)"
541 }
542
543 lsi_dmesg()
544 {
545 local tmp="$(mktemp)" || exit 1
546
547 trap "rm -f $tmp" EXIT
548 dmesg > $tmp
549 gsu_textbox "$tmp"
550 }
551
552 lsi_syslog()
553 {
554 gsu_textbox '/var/log/syslog'
555 }
556
557 . /usr/local/lib/gsu/gui || exit 1
558 gsu_gui "$menu_tree"
559
560 The config module
561 -----------------
562 Some applications need config options which are not related to
563 any particular subcommand, like the URL of a web service, the path
564 to some data directory, or a default value which is to be used by
565 several subcommands. Such options do not change frequently and are
566 hence better stored in a configuration file rather than passed to
567 every subcommand that needs the information.
568
569 The config module of gsu makes it easy to maintain such options and
570 performs routine tasks like reading and checking the values given in
571 the config file, or printing out the current configuration. It can
572 be used stand-alone, or in combination with either the subcommand or
573 the gui module.
574
575 ___Defining config options___
576
577 To use the config module, you must define the `$gsu_options`
578 bash array. Each config option is represented by one slot in this
579 array. Here is an example which defines two options:
580
581 gsu_options=(
582 "
583 name=fs_type
584 option_type=string
585 default_value=ext3
586 required=false
587 description='file system type to consider'
588 help_text='
589 This option is used in various contexts. All
590 subcommands which need a file system type
591 use the value specified here as the default.
592 '
593 "
594 "
595 name=limit
596 option_type=num
597 default_value=3
598 required=no
599 description='print at most this many lines of output'
600 "
601 )
602
603 Each config option consists of the following fields:
604
605 - `name`. This must be a valid bash variable name. Hence no special
606 characters are allowed.
607
608 - `option_type`. Only `string` and `num` are supported but additional
609 types might be supported in future versions. While string variables
610 may have arbitrary content, only integers are accepted for variables
611 of type `num`.
612
613 - `default_value`. The value to use if the option was not specified.
614
615 - `required`. Whether gsu considers it an error if the option was
616 not specified. It does not make sense to set this to `true` and set
617 `default_value` at the same time.
618
619 - `description`. Short description of the variable. It is printed by
620 the `prefs` subcommand.
621
622 - `help_text`. Optional long description, also printed by `prefs`.
623
624 To enable the config module you must source the config module of gsu
625 after `$gsu_options` has been defined:
626
627 . /usr/local/lib/gsu/config || exit 1
628
629 ___Passing config options to the application___
630
631 There are two ways to pass the value of an option to a gsu application:
632 environment variable and config file. The default config file is
633 `~/.$gsu_name.rc` where `$gsu_name` is the basename of the application,
634 but this can be changed by setting `$gsu_config_file`. Thus, the
635 following two statements are equivalent
636
637 fs_type=xfs hello fs
638 echo 'fs_type=xfs' > ~/.hello.rc && hello fs
639
640 If an option is set both in the environment and in the config file,
641 the environment takes precedence.
642
643 ___Checking config options___
644
645 The gsu config module defines two public functions for this purpose:
646 `gsu_check_options()` and `gsu_check_options_or_die()`. The latter
647 function exits on errors while the former function only sets `$ret`
648 and `$result` as appropriate and lets the application deal with the
649 error. The best place to call one of these functions is after sourcing
650 the config module but before calling `gsu()` or `gsu_gui()`.
651
652 ___Using config values___
653
654 The name of an option as specified in `$gsu_options` (`fs_type` in
655 the example above) is what users of your application may specify at
656 the command line or in the config file. This leads to a mistake that
657 is easy to make and difficult to debug: The application might use a
658 variable name which is also a config option.
659
660 To reduce the chance for this to happen, `gsu_check_options()` creates
661 a different set of variables for the application where each variable
662 is prefixed with `${gsu_name}`. For example, if `$gsu_options` as above
663 is part of the hello script, `$hello_fs_type` and `$hello_limit` are
664 defined after `gsu_check_options()` returned successfully. Only the
665 prefixed variants are guaranteed to contain the proper value, so this
666 variable should be used exclusively in the application. The
667 prefix may be changed by setting `$gsu_config_var_prefix` before calling
668 `gsu_check_options()`.
669
670 ___com_prefs()___
671
672 For scripts which source both the subcommand and the config module, the
673 auto-generated `prefs` subcommand prints out the current configuration
674 and exits. The description and help text of the option as specified
675 in the `description` and `help_text` fields of `$gsu_options` are shown
676 as comments in the output. Hence this output can be used as a template
677 for the config file.
678
679 List of public variables
680 ------------------------
681 - `$gsu_dir`. Where gsu is installed. If unset, gsu guesses
682 its installation directory by examining the `$BASH_SOURCE` array.
683
684 - `$gsu_name`. The name of the application. Defaults to `$0` with
685 all leading directories removed.
686
687 - `$gsu_banner_txt`. Used by both the subcommand and the gui
688 module. It is printed by the man subcommand, and as the title for
689 dialog windows.
690
691 - `$gsu_errors`. Identifier/text pairs for custom error reporting.
692
693 - `$gsu_config_file`. The name of the config file of the application.
694 Defaults to `~/.${gsu_name}.rc`.
695
696 - `$gsu_options`. Array of config options, used by the config module.
697
698 - `$gsu_config_var_prefix`. Used by the config module to set up
699 the variables defined in `$gsu_options`.
700
701 License
702 -------
703 gsu is licensed under the GNU LESSER GENERAL PUBLIC LICENSE (LGPL), version 3.
704 See COPYING and COPYING.LESSER.
705
706 Contact
707 -------
708 Send beer, pizza, patches, improvements, bug reports, flames,
709 (in this order), to Andre Noll <maan@tuebingen.mpg.de>.
710
711 References
712 ----------
713 - [bash](http://www.gnu.org/software/bash/bash.html)
714 - [dialog](http://www.invisible-island.net/dialog/dialog.html)
715 - [grutatxt](http://triptico.com/software/grutatxt.html)