Add README.
[gsu.git] / README
1 The global subcommand utility
2 =============================
3 gsu is a small library of bash functions intended to ease the task of
4 writing and documenting large shell scripts with multiple subcommands,
5 each providing different functionality. gsu is known to work on Linux,
6 FreeBSD, NetBSD and MacOS.
7
8 This document describes how to install and use the gsu library.
9
10 Setting up gsu
11 --------------
12 gsu is very easy to install:
13
14 Requirements
15 ~~~~~~~~~~~~
16 gsu is implemented in bash, and thus gsu depends on bash. Bash version
17 3 is required, version 4 is recommended. Besides bash, gsu depends
18 only on programs which are usually installed on any Unix system (awk,
19 grep, sort, ...). Care has been taken to not rely on GNU specific
20 behavior of these programs, so it should work on non GNU systems
21 (MacOS, *BSD) as well. The gui module depends on the dialog utility.
22
23 Download
24 ~~~~~~~~
25 All gsu modules are contained in a git repository. Get a copy with
26
27 git clone git://git.tuebingen.mpg.de/gsu.git
28
29 There is also a http://ilm.eb.local/gitweb/?p=gsu;a=summary (gitweb) page.
30
31 Installation
32 ~~~~~~~~~~~~
33 gsu consists of several independent modules which are all located
34 at the top level directory of the git repository. gsu requires no
35 installation beyond downloading. In particular it is not necessary
36 to make the downloaded files executable. The library modules can
37 be sourced directly, simply tell your application where to find
38 it. The examples of this document assume that gsu is installed in
39 `/usr/local/lib/gsu' but this is not mandatory.`~/.gsu' is another
40 reasonable choice.
41
42 Conventions
43 -----------
44 Public and private functions and variables
45 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
46 Although there is no way in bash to annotate symbols (functions
47 and variables) as private or public, gsu distinguishes between the
48 two. The `gsu_*' name space is reserved for public symbols while all
49 private symbols start with `_gsu'.
50
51 Private symbols are meant for internal use only. Applications should
52 never use them directly because name and semantics might change
53 between gsu versions.
54
55 The public symbols, on the other hand, define the gsu API. This API
56 must not change in incompatible ways that would break existing
57 applications.
58
59 $ret and $result
60 ~~~~~~~~~~~~~~~~
61 All public gsu functions set the $ret variable to an integer value to
62 indicate success or failure. As a convention, $ret < 0 means failure
63 while a non-negative value indicates success.
64
65 The $result variable contains either the result of a function (if any)
66 or further information in the error case. A negative value of $ret is
67 in fact an error code similar to the errno variable used in C programs.
68 It can be turned into a string that describes the error. The public
69 gsu_err_msg() function can be used to pretty-print a suitable error
70 message provided $ret and $result are set appropriately.
71
72 The subcommand module
73 ---------------------
74 This gsu module provides helper functions to ease the repetitious task
75 of writing applications which operate in several related modes, where
76 each mode of operation corresponds to a subcommand of the application.
77
78 With gsu, for each subcommand one must only write a _command handler_
79 which is simply a function that implements the subcommand. All
80 processing is done by the gsu library. Functions starting with the
81 string `com_' are automatically recognized as subcommand handlers.
82
83 The startup part of the script has to source the subcommand file of
84 gsu and must then call
85
86 gsu "$@"
87
88 Minimal example:
89
90 #!/bin/bash
91 com_world()
92 {
93 echo 'hello world'
94 }
95 . /usr/local/lib/gsu/subcommand || exit 1
96 gsu "$@"
97
98 Save this code in a file called `hello' (adjusting the installation
99 directory if necessary), make it executable (`chmod +x hello') and try
100
101 ./hello
102 ./hello world
103 ./hello invalid
104
105 Here, we have created a bash script ("hello") that has a single "mode"
106 of operation, specified by the subcommand "world".
107
108 gsu automatically generates several reserved subcommands, which should
109 not be specified: `help, man, prefs, complete'.
110
111 Command handler structure
112 ~~~~~~~~~~~~~~~~~~~~~~~~~
113 For the automatically generated help and man subcommands to work
114 properly, all subcommand handlers must be documented. In order to be
115 recognized as subcommand help text, comments must be prefixed with
116 two `#' characters and the subcommand documentation must be located
117 between the function "declaration", com_world() in the example above,
118 and the opening brace that starts the function body.
119
120 Example:
121
122 com_world()
123 ##
124 ##
125 ##
126 {
127 echo 'hello world'
128 }
129
130 The subcommand documentation consists of three parts:
131
132 - The summary. One line of text,
133 - the usage/synopsis string,
134 - free text section.
135
136 The three parts should be separated by lines consisting of two # characters
137 only. Example:
138
139 com_world()
140 ##
141 ## Print the string "hello world" to stdout.
142 ##
143 ## Usage: world
144 ##
145 ## Any arguments to this function are ignored.
146 ##
147 ## Warning: This subcommand may cause the top most line of your terminal to
148 ## disappear and may cause DATA LOSS in your scrollback buffer. Use with
149 ## caution.
150 {
151 echo 'hello world'
152 }
153
154 Replace 'hello' with the above and try:
155
156 ./hello help
157 ./hello help world
158 ./hello help invalid
159 ./hello man
160
161 to check the automatically generated help and man subcommands.
162
163 Error codes
164 ~~~~~~~~~~~
165 As mentioned above, all public functions of gsu return an error code
166 in the $ret variable. A negative value indicates failure, and in this
167 case $result contains more information about the error. The same
168 convention applies for subcommand handlers: gsu will automatically
169 print an error message to stderr if a subcommand handler returns with
170 $ret set to a negative value.
171
172 To allow for error codes defined by the application, the $gsu_errors
173 variable must be set before calling gsu(). Each non-empty line in this
174 variable should contain an identifier and error string. Identifiers
175 are written in upper case and start with `E_'. For convenience the
176 $GSU_SUCCESS variable is defined to non-negative value. Subcommand
177 handlers should set $ret to $GSU_SUCCESS on successful return.
178
179 To illustrate the $gsu_errors variable, assume the task is to
180 print all mount points which correspond to an ext3 file system in
181 `/etc/fstab'. We'd like to catch two possible errors: (a) the file
182 does not exist or is not readable, and (b) it contains no ext3 entry.
183 A possible implementation of the ext3 subcommand could look like this
184 (documentation omitted):
185
186 #!/bin/bash
187
188 gsu_errors='
189 E_NOENT No such file or directory
190 E_NOEXT3 No ext3 file system detected
191 '
192
193 com_ext3()
194 {
195 local f='/etc/fstab'
196 local ext3_lines
197
198 if [[ ! -r "$f" ]]; then
199 ret=-$E_NOENT
200 result="$f"
201 return
202 fi
203 ext3_lines=$(awk '{if ($3 == "ext3") print $2}' "$f")
204 if [[ -z "$ext3_lines" ]]; then
205 ret=-$E_NOEXT3
206 result="$f"
207 return
208 fi
209 printf 'ext3 mount points:\n%s\n' "$ext3_lines"
210 ret=$GSU_SUCCESS
211 }
212
213 Printing diagnostic output
214 ~~~~~~~~~~~~~~~~~~~~~~~~~~
215 gsu provides a couple of convenience functions for output. All
216 functions write to stderr.
217
218 - *gsu_msg()*. Writes the name of the application and the given text.
219
220 - *gsu_short_msg()*. Like gsu_msg(), but does not print the application name.
221
222 - *gsu_date_msg()*. Writes application name, date, and the given text.
223
224 - *gsu_err_msg()*. Prints an error message according to $ret and $result.
225
226 Subcommands with options
227 ~~~~~~~~~~~~~~~~~~~~~~~~
228 Bash's getopts builtin provides a way to define and parse command line
229 options, but it is cumbersome to use because one must loop over all
230 given arguments and check the OPTIND and OPTARG variables during each
231 iteration. The gsu_getopts() function makes this repetitive task easier.
232
233 gsu_getopts() takes a single argument: the optstring which contains
234 the option characters to be recognized. As usual, if a character is
235 followed by a colon, the option is expected to have an argument. On
236 return $result contains bash code that should be eval'ed to parse the
237 position parameters $1, $2, ... of the subcommand according to the
238 optstring.
239
240 The shell code returned by gsu_getopts() creates a local variable $o_x
241 for each defined option `x'. It contains `true/false' for options
242 without argument and either the empty string or the given argument for
243 options that take an argument.
244
245 To illustrate gsu_getopts(), assume the above com_ext3() subcommand
246 handler is to be extended to allow for arbitrary file systems, and
247 that it should print either only the mount point as before or the
248 full line of `/etc/fstab', depending on whether the verbose switch
249 `-v' was given at the command line.
250
251 Hence our new subcommand handler must recognize two options: `-t' for
252 the file system type and `-v'. Note that `-t' takes an argument but `-v'
253 does not. Hence we shall use the optstring `t:v' as the argument for
254 gsu_getopts() as follows:
255
256 com_fs()
257 {
258 local f='/etc/fstab'
259 local fstype fstab_lines
260 local -i awk_field=2
261
262 gsu_getopts 't:v'
263 eval "$result"
264 (($ret < 0)) && return
265
266 [[ -z "$o_t" ]] && o_t='ext3' # default to ext3 if -t is not given
267 [[ "$o_v" == 'true' ]] && awk_field=0 # $0 is the whole line
268 fstab_lines=$(awk -v fstype="$o_t" -v n="$awk_field" \
269 '{if ($3 == fstype) print $n}' "$f")
270 printf '%s entries:\n%s\n' "$o_t" "$fstab_lines"
271 ret=$GSU_SUCCESS
272 }
273
274 Another repetitive task is to check the number of non-option arguments
275 and to report an error if this number turns out to be invalid for
276 the subcommand in question. The gsu_check_arg_count() function performs
277 this check and sets $ret and $result as appropriate. This function
278 takes three arguments: the actual argument count and the minimal and
279 maximal number of non-option arguments allowed. The last argument may
280 be omitted in which case any number of arguments is considered valid.
281
282 Our com_world() subcommand handler above ignored any given
283 arguments. Let's assume we'd like to handle this case and
284 print an error message if one or more arguments are given. With
285 gsu_check_arg_count() this can be achieved as follows:
286
287 com_world()
288 {
289 gsu_check_arg_count $# 0 0 # no arguments allowed
290 (($ret < 0)) && return
291 echo 'hello world'
292 }
293
294 Global documentation
295 ~~~~~~~~~~~~~~~~~~~~
296 Besides the documentation for subcommands, one might also want to
297 include an overall description of the application which provides
298 general information that is not related to any particular subcommand.
299
300 If such a description is included at the top of the script, the
301 automatically generated man subcommand will print it. gsu recognizes
302 the description only if it is enclosed by two lines consisting of at
303 least 70 # characters.
304
305 Example:
306
307 #/bin/bash
308
309 #######################################################################
310 # gsu-based hello - a cumbersome way to write a hello world program
311 # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
312 # It not only requires one to download and install some totally weird
313 # git repo, it also takes about 50 lines of specially written code
314 # to perform what a simple echo 'hello world' would do equally well.
315 #######################################################################
316
317 HTML output
318 ~~~~~~~~~~~
319 The output of the auto-generated man subcommand is a suitable input for the
320 grutatxt plain text to html converter. Hence
321
322 ./hello man | grutatxt > index.html
323
324 is all it takes to produce an html page for your application.
325
326 Interactive completion
327 ~~~~~~~~~~~~~~~~~~~~~~
328 The auto-generated `complete' subcommand provides interactive bash
329 completion. To activate completion for the hello program, it is
330 enough to put the following into your `~/.bashrc':
331
332 _hello()
333 {
334 eval $(hello complete 2>/dev/null)
335 }
336 complete -F _hello hello
337
338 This will give you completion for the first argument of the hello
339 program: the subcommand.
340
341 In order to get subcommand-sensitive completion you must provide a
342 _completer_ in your application for each subcommand that is to support
343 completion. Like subcommand handlers, completers are recognized by name:
344 If a function xxx_complete() is defined, gsu will call it on the
345 attempt to complete the `xxx' subcommand at the subcommand line. gsu
346 has a few functions to aid you in writing a completer.
347
348 Let's have a look at the completer for the above `fs' subcommand.
349
350 complete_fs()
351 {
352 local f='/etc/fstab'
353 local optstring='t:v'
354
355 gsu_complete_options $optstring "$@"
356 (($ret > 0)) && return
357
358 gsu_cword_is_option_parameter $optstring "$@"
359 [[ "$result" == 't' ]] && awk '{print $3}' "$f"
360 }
361
362 Completers are always called with $1 set to the index into the array
363 of words in the current command line when tab completion was attempted
364 (see `COMP_CWORD' in the bash manual). These words are passed to the
365 completer as $2, $3,...
366
367 gsu_complete_options() receives the option string as $1, the word
368 index as $2 and the individual words as $3, $4,... Hence we may simply
369 pass the $optstring and `"$@"'. gsu_complete_options() checks if the
370 current word begins with `-', i.e., whether an attempt to complete
371 an option was performed. If yes gsu_complete_options() prints all
372 possible command line options and sets $ret to a positive value.
373
374 The last two lines of complete_fs() check whether the word preceding
375 the current word is an option that takes an argument. If it is,
376 that option is returned in $result, otherwise $result is the empty
377 string. Hence, if we are completing the argument to `-t', the awk
378 command is executed to print all file system types of /etc/fstab as
379 the possible completions.
380
381 See the comments to gsu_complete_options(),
382 gsu_cword_is_option_parameter() and gsu_get_unnamed_arg_num()
383 (which was not covered here) in the `subcommand' file for a more
384 detailed description.
385
386 The gui module
387 --------------
388 This module can be employed to create interactive dialog boxes from a
389 bash script. It depends on the dialog(1) utility which is available on
390 all Unix systems. On Debian and Ubuntu Linux it can be installed with
391
392 apt-get install dialog
393
394 The core of the gui module is the gsu_gui() function which receives
395 a _menu tree_ as its single argument. The menu tree defines a tree
396 of menus for the user to navigate with the cursor keys. As for a
397 file system tree, internal tree nodes represent folders. Leaf nodes,
398 on the other hand, correspond to _actions_. Pressing enter activates a
399 node. On activation, for internal nodes a new menu with the contents of
400 the subfolder is shown. For leaf nodes the associated _action handler_
401 is executed.
402
403 Hence the application has to provide a menu tree and an action handler
404 for each leaf node defined in the tree. The action handler is simply a
405 function which is named according to the node. In most cases the action
406 handler will run dialog(1) to show some dialog box on its own. Wrappers
407 for some widgets of dialog are provided by the gui module, see below.
408
409 Menu trees
410 ~~~~~~~~~~
411 The concept of a menu tree is best illustrated by an example. Assume
412 we'd like to write a system utility for the not-so-commandline-affine
413 Linux sysadmin next door. For the implementation we confine ourselves
414 with giving some insight in the system by running lean system commands
415 like `df' to show the list of file system, or `dmesg' to print the
416 contents of the kernel log buffer. Bash code which defines the menu
417 tree could look like this:
418
419 menu_tree='
420 load_average
421 processes
422 hardware/
423 cpu
424 scsi
425 storage/
426 df
427 mdstat
428 log/
429 syslog
430 dmesg
431 '
432
433 In this tree, `hardware/', `block_devices/' and `log/' are the only
434 internal nodes. Note that these are written with a trailing slash
435 character while the leaf nodes have no slash at the end. All entries
436 of the menu tree must be indented by tab characters.
437
438 Action handlers
439 ~~~~~~~~~~~~~~~
440 Action handlers are best explained via example:
441
442 Our application, let's call it `lsi' for _lean system information_,
443 must provide action handlers for all leaf nodes. Here is the action
444 handler for the `df' node:
445
446 lsi_df()
447 {
448 gsu_msgbox "$(df -h)"
449 }
450
451 The function name `lsi_df' is derived from the name of the script
452 (`lsi') and the name of the leaf node (`df'). The function simply
453 passes the output of the `df(1)' command as the first argument to
454 the public gsu function gsu_msgbox() which runs dialog(1) to display
455 a message box that shows the given text.
456
457 gsu_msgbox() is suitable for small amounts of output. For essentially
458 unbounded output like log files that can be arbitrary large, it is
459 better to use gsu_textbox() instead which takes a path to the file
460 that contains the text to show.
461
462 To illustrate gsu_input_box() function, assume the action handler
463 for the `processes' leaf node should ask for a username, and display
464 all processes owned by the given user. This could be implemented
465 as follows.
466
467 lsi_processes()
468 {
469 local username
470
471 gsu_inputbox 'Enter username' "$LOGNAME"
472 (($ret != 0)) && return
473 username="$result"
474 gsu_msgbox "$(pgrep -lu "$username")"
475 }
476
477 Once all other action handlers have been defined, the only thing left
478 to do is to source the gsu gui module and to call gsu_gui():
479
480 . /usr/local/lib/gsu/gui || exit 1
481 gsu_gui "$menu_tree"
482
483 Example
484 ~~~~~~~
485 The complete lsi script below can be used as a starting point
486 for your own gsu gui application. If you cut and paste it, be
487 sure to not turn tab characters into space characters.
488
489 #!/bin/bash
490
491 menu_tree='
492 load_average
493 processes
494 hardware/
495 cpu
496 scsi
497 storage/
498 df
499 mdstat
500 log/
501 syslog
502 dmesg
503 '
504
505 lsi_load_average()
506 {
507 gsu_msgbox "$(cat /proc/loadavg)"
508 }
509
510 lsi_processes()
511 {
512 local username
513
514 gsu_inputbox 'Enter username' "$LOGNAME"
515 (($ret < 0)) && return
516 username="$result"
517 gsu_msgbox "$(pgrep -lu "$username")"
518 }
519
520 lsi_cpu()
521 {
522 gsu_msgbox "$(lscpu)"
523 }
524
525 lsi_scsi()
526 {
527 gsu_msgbox "$(lsscsi)"
528 }
529
530 lsi_df()
531 {
532 gsu_msgbox "$(df -h)"
533 }
534
535 lsi_mdstat()
536 {
537 gsu_msgbox "$(cat /proc/mdstat)"
538 }
539
540 lsi_dmesg()
541 {
542 local tmp="$(mktemp)" || exit 1
543
544 trap "rm -f $tmp" EXIT
545 dmesg > $tmp
546 gsu_textbox "$tmp"
547 }
548
549 lsi_syslog()
550 {
551 gsu_textbox '/var/log/syslog'
552 }
553
554 . /usr/local/lib/gsu/gui || exit 1
555 gsu_gui "$menu_tree"
556
557 The config module
558 -----------------
559 Some applications need config options which are not related to
560 any particular subcommand, like the URL of a web service, the path
561 to some data directory, or a default value which is to be used by
562 several subcommands. Such options do not change frequently and are
563 hence better stored in a configuration file rather than passed to
564 every subcommand that needs the information.
565
566 The config module of gsu makes it easy to maintain such options and
567 performs routine tasks like reading and checking the values given in
568 the config file, or printing out the current configuration. It can
569 be used stand-alone, or in combination with either the subcommand or
570 the gui module.
571
572 Defining config options
573 ~~~~~~~~~~~~~~~~~~~~~~~
574 To use the config module, you must define the $gsu_options bash array.
575 Each config option is represented by one slot in this array. Here is
576 an example which defines two options:
577
578 gsu_options=(
579 "
580 name=fs_type
581 option_type=string
582 default_value=ext3
583 required=false
584 description='file system type to consider'
585 help_text='
586 This option is used in various contexts. All
587 subcommands which need a file system type
588 use the value specified here as the default.
589 '
590 "
591 "
592 name=limit
593 option_type=num
594 default_value=3
595 required=no
596 description='print at most this many lines of output'
597 "
598 )
599
600 Each config option consists of the following fields:
601
602 - *name*. This must be a valid bash variable name. Hence no special
603 characters are allowed.
604
605 - *option_type*. Only `string' and `num' are supported but additional
606 types might be supported in future versions. While string variables
607 may have arbitrary content, only integers are accepted for variables
608 of type `num'.
609
610 - *default_value*. The value to use if the option was not specified.
611
612 - *required*. Whether gsu considers it an error if the option was
613 not specified. It does not make sense to set this to `true' and set
614 *default_value* at the same time.
615
616 - *description*. Short description of the variable. It is printed by
617 the `prefs' subcommand.
618
619 - *help_text*. Optional long description, also printed by `prefs'.
620
621 To enable the config module you must source the config module of gsu
622 after $gsu_options has been defined:
623
624 . /usr/local/lib/gsu/config || exit 1
625
626 Passing config options to the application
627 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
628 There are two ways to pass the value of an option to a gsu application:
629 environment variable and config file. The default config file is
630 ~/.$gsu_name.rc where $gsu_name is the basename of the application,
631 but this can be changed by setting $gsu_config_file. Thus, the
632 following two statements are equivalent
633
634 fs_type=xfs hello fs
635 echo 'fs_type=xfs' > ~/.hello.rc && hello fs
636
637 If an option is set both in the environment and in the config file,
638 the environment takes precedence.
639
640 Checking config options
641 ~~~~~~~~~~~~~~~~~~~~~~~
642 The gsu config module defines two public functions for this purpose:
643 gsu_check_options() and gsu_check_options_or_die(). The latter function
644 exits on errors while the former function only sets $ret and $result
645 as appropriate and lets the application deal with the error. The best
646 place to call one of these functions is after sourcing the config
647 module but before calling gsu() or gsu_gui().
648
649 Using config values
650 ~~~~~~~~~~~~~~~~~~~
651 The name of an option as specified in $gsu_options (`fs_type' in
652 the example above) is what users of your application may specify at
653 the command line or in the config file. This leads to a mistake that
654 is easy to make and difficult to debug: The application might use a
655 variable name which is also a config option.
656
657 To reduce the chance for this to happen, gsu_check_options() creates
658 a different set of variables for the application where each variable
659 is prefixed with ${gsu_name}. For example, if $gsu_options as above
660 is part of the hello script, $hello_fs_type and $hello_limit are
661 defined after gsu_check_options() returned successfully. Only the
662 prefixed variants are guaranteed to contain the proper value, so this
663 variable should be used exclusively in the application. The
664 prefix may be changed by setting $gsu_config_var_prefix before calling
665 gsu_check_options().
666
667 com_prefs()
668 ~~~~~~~~~~~
669 For scripts which source both the subcommand and the config module, the
670 auto-generated 'prefs' subcommand prints out the current configuration
671 and exits. The description and help text of the option as specified
672 in the `description' and `help_text' fields of $gsu_options are shown
673 as comments in the output. Hence this output can be used as a template
674 for the config file.
675
676 List of public variables
677 ------------------------
678 - *$gsu_dir*. Where gsu is installed. If unset, gsu guesses
679 its installation directory by examining the $BASH_SOURCE array.
680
681 - *$gsu_name*. The name of the application. Defaults to $0 with
682 all leading directories removed.
683
684 - *$gsu_banner_txt*. Used by both the subcommand and the gui
685 module. It is printed by the man subcommand, and as the title for
686 dialog windows.
687
688 - *$gsu_errors*. Identifier/text pairs for custom error reporting.
689
690 - *$gsu_config_file*. The name of the config file of the application.
691 Defaults to `~/.${gsu_name}.rc'.
692
693 - *$gsu_options*.
694
695 - *$gsu_config_var_prefix*. Used by the config module to set up
696 the variables defined in $gsu_options.
697
698 License
699 -------
700 Contact
701 -------
702 Send beer, pizza, patches, improvements, bug reports, flames,
703 (in this order), to Andre Noll `<maan@tuebingen.mpg.de>'.
704
705 References
706 ----------
707 - http://www.gnu.org/software/bash/bash.html (bash)
708 - http://www.invisible-island.net/dialog/dialog.html (dialog)
709 - http://triptico.com/software/grutatxt.html (grutatxt)