OSLV: Replace LXC section.
[aple.git] / Bash.m4
1 TITLE(«
2
3 Write programs that do one thing and do it well. Write programs
4 to work together. Write programs to handle text streams,
5 because that is a universal interface. -- Doug MacIlroy
6
7 », __file__)
8
9 SECTION(«CMD(«sh») versus CMD(«bash»)»)
10
11 - Bash scripts begin with a sha-bang: CMD(«#!/bin/bash»)
12 - sh is the POSIX defined shell specification.
13 - Bash is one implementation of the sh specification.
14 - /bin/sh links to the default shell of your system.
15 - This can be different from your user shell!
16 - Each shell has its idiosyncracies.
17 - Using a sha-bang pointing to bash is safer CMD(«#!/bin/bash»)
18 - < 10.3, the default Mac OS X shell is tcsh (bad!)
19 - Scripts need to be executable (chmod u+x script).
20
21 EXERCISES()
22 - Your current shell is stored the CMD($SHELL) environment variable.
23 Use echo to find out what it is.
24 - Similarly, find out your CMD($BASH_VERSION).
25 - Use readlink to find out what CMD(/bin/sh) points to in your system.
26
27 HOMEWORK(«
28 Explain why bash does not exit when you type CMD(«Ctrl+C») on the
29 command line.
30 »)
31
32 SECTION(«Variables»)
33 - Variables are defined using CMD(«=») (no spaces!).
34 - Variables are read using CMD(«$»).
35 - Spaces are the enemy of variables. Spaces break variables.
36 - Double quotes CMD(«"») a the defense against spaces.
37 - braces (curly brackets) CMD(«{}») can also protect variables from
38 ambiguity. eg: CMD(«${foo}»)bar. They also group commands.
39 - Single quotes CMD(«'») are like double quotes, but are literal.
40 - Bash scripts have special variables:
41 - CMD(«$0»): script full path and name.
42 - CMD(«$1»): first command line argument.
43 - CMD(«$2»): second argument ... etc.
44 - CMD(«$#»): number of command line arguments.
45 - CMD(«$*»): list of arguments as a single string.
46 - CMD(«$@»): list of arguments as a delimited list.
47 - Parentheses CMD(«()») execute a command in a sub-shell.
48 - Double parentheses return the result of arithmetic expansion
49 (positive integer only).
50
51 EXERCISES()
52
53 - Write a simple script in which you define a variable with the string
54 "Hello World!". echo this variable without quotes, with single and
55 double quotes, and with braces (again with and without different
56 quotes). Become comfortable with the results.
57 - How do you return the results of a sub-shell ()?
58 - Write a simple script to add two positive integers.
59 - Write a simple script to add two positive integers supplied as
60 arguments to the script.
61 »)
62
63 HOMEWORK(«
64 Write a script using your favorite editor. The script should display
65 the path to your home directory and the terminal type that you
66 are using.
67 »)
68
69 SECTION(«Tests»)
70
71 - CMD(«[...]») is the POSIX sh test function.
72 - CMD(«[[...]]») is the Bash test function (more powerful).
73 - These tests are logical: they return TRUE or FALSE.
74 - Tests use logical operators.
75 - Spaces are a must!
76 - There are three types of operators: File, String and Integer.
77 - A few single file operators eg: CMD(«[[ -e somefile ]]»)
78 - CMD(«-e»): file exists
79 - CMD(«-s»): file not zero size
80 - CMD(«-d»): file is a directory
81 - CMD(«-r»): you have read permission
82 - CMD(«-O»): you are the owner of the file
83 - A few multiple file operators eg: CMD(«[[ file1 -nt file2 ]]»)
84 - CMD(«-nt»): first file newer than second
85 - CMD(«-ot»): first file older than second
86 - A few integer operators:
87 - CMD(«-eq»): equal to
88 - CMD(«-ne»): not equal to
89 - CMD(«-gt»): greater than
90 - CMD(«-ge»): greater than or equal to
91 - CMD(«-lt»): less than
92 - CMD(«-le»): less than or equal to
93 - A few string operators:
94 - CMD(«==»): equal to
95 - CMD(«!=»): not equal to
96 - CMD(«=~»): regex match (Bash specific)
97 - CMD(«-z»): string is null (zero length)
98 - CMD(«-n»): string is not null (zero length)
99 - When you understand how these operators work, you will have a good
100 idea of the kinds of things you can do in Bash.
101 - Tests can be combined with CMD(«&&»): "and" and CMD(«||») "or".
102
103 HOMEWORK(«
104 Write a script that checks whether or not you own a file, and reports
105 back if you do not. (This is useful if you are working on multiple
106 user systems and need your script to remove many files.
107 »)
108
109 SECTION(«Conditionals»)
110
111 - The most commonly used Bash conditional structure is: CMD(«if»)
112 ... CMD(«then») ... CMD(«fi»)
113 - A shorter version uses logic in place of if..then..fi. eg: CMD(«[[
114 test ]] && { execute if TRUE; also execute }»)
115
116 EXERCISES()
117
118 - Modify your calculator script to check for valid inputs:
119 - There must be two.
120 - They should not have letters
121 - Write a script to check for a directory, and create it if it
122 doesn't exist.
123 - Write a script to remove a file specfied as an argument, but only
124 if you are its owner.
125
126
127 HOMEWORK(«
128 Write a script that takes exactly one argument, a directory name. If
129 the number of arguments is more or less than one, print a usage
130 message. If the argument is not a directory, print another message. For
131 the given directory, print the five biggest files and the five files
132 that were most recently modified.
133 »)
134
135 SECTION(«Loops»)
136
137 - The most commonly used Bash loop structure is: CMD(for ... do
138 ... done)
139 - The CMD(for) statement behaves a lot like a variable assignment.
140 - File globbing works: CMD(«for file in *.txt; do; done»)
141 - Sequences are also useful:
142 - CMD(«for num in {1..5}; do; echo $num; done 1 2 3 4 5»)
143 - CMD(«for num in {1..10..2}; do; echo $num; done 1 3 5 7 9»)
144
145 EXERCISES()
146 - Write a script that stores the results of a arithmetic in files
147 named by the inputs.
148
149 HOMEWORK(«
150 Come up with a for loop which prints the first 10 squares (1, 4,
151 9, ...).
152 »)
153
154 SECTION(«Pipes and Here Strings»)
155
156 - Here strings are an alternative to conventional piping.
157 - Because they do not spawn a sub-shell, they retain variables in
158 the shell the script is running in. eg: instead of CMD(head file1 |
159 cut -f 1) write CMD(head | cut -f 1 <<< file1)
160 - They can be easier or more difficult to read, depending on your
161 taste.
162
163 EXERCISES()
164 Write a script that uses pipes, change it to use a Here string.
165
166
167 HOMEWORK(«
168 Tie all the above together with the following task:
169
170 Let's say that you want to perform an analysis by population
171 (k-means) cluster of some accessions (ecotypes). You want to
172 generate a separate bed file with the SNPs of the members of
173 each cluster (which you have previously calculated).
174
175 The relevant plink argument is: CMD(«--keep “$keeplist”») where
176 keeplist is a two column file specifying family and ecotype
177 (made for human data). We will just duplicate the ecotype
178 in the two columns. e.g.:
179
180 > cat keeplist
181 88 88
182 107 107
183 etc.
184
185 I have provided a comma-separated file of which cluster each ecotype
186 belongs to: CMD(«/tmp/cluster_course/admix_pop.csv») Take a look at
187 this file. You will see that it is a comma separated file with
188 ecotype ID numbers in the first column and the cluster assignment
189 in the second.
190
191 Use the 1001_cluster_course files as your test dataset.
192
193 You suspect that your clusters might change, (in assignment
194 and number ), so you want to write a Bash script to generate
195 separate bed files for a given clustering.
196
197 Hints:
198
199 - Your script will be called something like this:
200
201 sep_clust.sh all_snps_bedfile_root cluster_assignment.csv
202
203 - You will have a loop.
204 - You will generate a list of cluster numbers from the
205 CMD(«cluster_assignment.csv») file.
206 - The cluster file has a header! CMD(«tail -n+2») will skip to the
207 second line.
208 - CMD(«grep “something$”») matches CMD(«something») at the
209 end of a line.
210 - You will generate a “keep” list for each cluster and supply
211 that to plink.
212 - CMD(«cut») expects tab-delimited input, but ours is
213 comma-delimited. Use CMD(«-d ","»).
214 - The keep list needs the ecotypes printed twice per line. The easiest
215 thing to use in this case is CMD(«awk»):
216
217 awk -v OFS='\t' '{print $1, $1}'
218
219 - Here, CMD(«-v») changes an internal value (CMD(«OFS»), the
220 “Output Field Separator”), CMD(«\t») specifies the delimiter
221 (a tab), CMD(«{...}») is the command, and CMD(«print $1, $1»)
222 is the command to print column 1, column 1.
223
224 - Remember:
225 - CMD(«uniq») requires sorted input.
226 - CMD(«sort -n») specifies a numerical sort.
227 - Generate as few temporary files as possible.
228 »)
229
230 HOMEWORK(«
231 If important data have been copied from one system to another,
232 one might want to check that both copies are identical. This is
233 fairly easy if they both live on the same system, but can be quite
234 tricky if they don't. For example, imagine files copied from an
235 NFS-mounted directory to the local hard drive.
236
237 <ul>
238 <li> Write a bash script which checks that all files have been
239 copied and that all copies have the same size as the original. Hint:
240 You might want to check out <code>find</code>. Generate two almost
241 identical folders with <code> mkdir a b && touch {a,b}/{1..10} &&
242 rm b/4 && echo foo > a/7 </code> to try out the script. </li>
243
244 <li> Run <code>echo bar > b/7</code>. Will the script detect that
245 <code>a/7</code> and <code>b/7</code> are different? </li>
246
247 <li> Come up with a different script which runs <code>cmp</code>
248 to detect content changes. Analyze and compare the running time of
249 both scripts. </li>
250
251 <li> Now suppose the two directories are stored on different
252 systems. Assume further that they are so large (or the network
253 so slow) that transferring the file contents over the network
254 would take too long. Argue how a cryptographic hash function can
255 be employed to detect content changes. Write a script which runs
256 <code>sha1sum</code> to implement this idea and analyze its running
257 time. </li>
258 </ul>
259 », «
260
261 <ul>
262 <li> Script which prints file names and sizes of all regular files
263 in the given two directories and compares the result.
264
265 <pre>
266 #!/bin/bash
267 list_files() { (cd "$1" && find -type f -printf '%h/%f\t%s\n' | sort); }
268 (($# != 2)) && exit 1
269 { list_files "$1" && list_files "$2"; } | sort | uniq -u
270 </pre>
271 </li>
272
273 <li> The above script will not look at file contents. Since
274 <code>a/7</code> and <code>b/7</code> have the same size and the same
275 base name, the script won't notice they are different. </li>
276
277 <li> Script which compares the contents of all regular files
278 in directory <code>$1</code> against the files in directory
279 <code>$2</code>:
280
281 <pre>
282
283 (($# != 2)) && exit 1
284 { cd "$1" && find -type f; } | while read -r file; do
285 cmp "$1/$file" "$2/$file"
286 done
287 </pre>
288
289 <li> The running time of the find command in first script is proportional
290 to the number of files, <code>n</code>. The sort command runs in time
291 proportional to <code>n * log(n</code>). Since the two commands run
292 in parallel, the total running time is the maximum of the two. In
293 practice, since the <code>find</code> command needs at least one
294 system call per file (<code>stat(2)</code>) to get the metadata, and the
295 kernel must load this information from disk, the running time of the
296 pipeline is dominated by the running time of <code>find</code>. Note
297 that it is independent of the file sizes. The running time of the
298 second script is proportional to the number of files, plus the sum of
299 all file sizes, since in the worst case <code>cmp</code> must read
300 all files completely to perform its task. In the common situation
301 where files are much bigger than a few hundred bytes, the sum of all
302 file sizes dominates the running time. The second script might easily
303 run orders of magnitute slower than the first. </li>
304
305 <li> Script which computes the sha1 hash of all regular files in
306 directory <code>$1</code> and checks the hashes against the files in
307 directory <code>$2</code> on host <code>$3</code>:
308
309 <pre>
310 #!/bin/bash
311 (($# != 3)) && exit 1
312 cd "$1" && find -type f -print0 \
313 | xargs -0 sha1sum \
314 | ssh "$3" "cd $2 && sha1sum -c"
315 </pre>
316
317 The <code>sha1sum</code> command reads the full contents of each file,
318 so the running time is the same as for the script that executed
319 <code>cmp</code>. However, only the hashes but no contents are
320 transferred over the network, and the hashes are computated locally on
321 each system. Therefore, this approach performs best in practice. </li>
322 </ul>
323
324 »)
325
326 SECTION(«Substitution and Expansion»)
327
328 - expansion is performed on the command line after it has been split
329 into words
330 - several kinds of expansion: tilde, brace, arithmetic, pathname,
331 parameter and variable, history
332 - command substitution
333
334 EXERCISES()
335
336 - Give an example for each type of expansion.
337 - Which expansions can change the number of words?
338 - Create a list of "words" with CMD(«apg -c 42 -n 10000 >
339 foo»). Transform each word to upper case, using the case-modification
340 operator CMD(«^^») as follows: CMD(«while read a; do echo ${a^^};
341 done < foo > bar»). Compare the running time of this command with (a)
342 CMD(«tr [a-z] [A-Z] < foo > bar») and (b) CMD(«while read a; do tr
343 [a-z] [A-Z] <<< "$a"; done < foo > bar»). Try to beat the fastest
344 implementation using your favorite tool (CMD(«sed»), CMD(«perl»),
345 CMD(«python»), ...).
346 - The command CMD(«find . -maxdepth 1 -mindepth 1 -type d») lists
347 all directories in the CWD. Describe an CMD(«ls») command which
348 does the same.
349 - Scripts often contain code like CMD(«find . | while read f; do
350 something with "$f"; done»). While this code works for file names
351 which contain spaces or tab characters, it is not bullet-proof because
352 file names may also contain the newline character. The only character
353 that can not be part of a file name is the null character. This is
354 why CMD(«find(1)») has the CMD(«--print0») option to separate the
355 file names in its output by null characters rather than the newline
356 character. Find a way to make the CMD(«while») loop work when it
357 is fed a file list produced with CMD(«find --print0»). Check the
358 correctness of your command by using CMD(«printf 'file\n1\0file 2'»)
359 as the left hand side of the pipe.
360 »)
361
362 HOMEWORK(«
363 - Write a shell script CMD(«varstate») which takes the name of a
364 variable as CMD(«$1») and determines whether the named variable
365 is (1) set to a non-empty string, (2) set to the empty string, or
366 (3) unset. Verify the correctness of your script with the following
367 commands:
368 - CMD(«foo=bar ./varstate foo # case (1)»)
369 - CMD(«foo= ./varstate foo # case (2)»)
370 - CMD(«unset foo; ./varstate foo # case (3)»)
371 »)
372
373 SECTION(«Functions»)
374
375 - code block that implements a set of operations
376 - for tasks which repeat with only slight variations
377 - syntax: CMD(«f() {commands; }»)
378 - positional parameters (CMD(«$1»), CMD(«$2»), ...)
379 - special parameters: CMD(«$*»), CMD(«$@»), CMD(«$#»)
380
381 EXERCISES()
382
383 - Understand your smiley of the day (and run it if you are brave):
384 CMD(«:() { :& :& };:»)
385 - Write a function which checks whether the passed string is a
386 decimal number.
387 - Consider this simple function which returns its first argument:
388 CMD(«foo() { return $1; }»). Find the largest positive integer this
389 function can return.
390 - Write a function which returns the sum of the first and the 10th
391 argument.
392
393 SECTION(«Arrays and Hashes»)
394
395 - bash-2: arrays, bash-4: hashes (associative arrays)
396 - zero-based, one-dimensional only
397 - three ways of assigning values
398 - negative parameters in arrays and string-extraction (bash-4.2)
399
400 EXERCISES()
401
402 - The following three array assignments are equivalent:
403 CMD(arr=(one two three)), CMD(«arr=([0]=one [1]=two [2]=three)»),
404 CMD(«arr[0]=one; arr[1]=two; arr[2]=three»). Discuss the pros and
405 cons of each version.
406 - Define an array with CMD(«arr=(one two three)»).
407 - Learn how to determine the number of elements that have been assigned
408 (three in this example).
409 - Convert all entries to upper case without iterating.
410 - Print all entries which do not contain an CMD(«"o"») character,
411 again without iterating (result: CMD(«three»)).
412 - Use arrays to write a bash script that lists itself, including
413 line numbers, and does not call any external command (CMD(«sed,
414 awk, perl, python, ...»)). Try to get rid of the loop in this
415 REFERENCE(«self-list.bash», «solution»),
416 - CMD(«rot13») is a simple "encryption" algorithm which shifts each
417 letter in the alphabet a-z by 13 characters and leaves non-letter
418 characters unchanged. That is, CMD(«a») maps to CMD(«n»),
419 CMD(«b») maps to CMD(«o»), ..., CMD(«m») maps to CMD(«z»),
420 CMD(«n») maps to CMD(«a»), and so on. Implement CMD(«rot13»)
421 using an associative array. Compare your solution with this
422 REFERENCE(«rot13.bash», «implementation») which reads from
423 stdin and writes to stdout. Verify that "encrypting" twice with
424 CMD(«rot13») is a no-op.
425 - Examine the CMD(BASH_VERSINFO) array variable to check whether the
426 running bash instance supports negative array indices.
427 - Write a bash script which reads from stdin and prints the last word
428 of the input.
429
430 HOMEWORK(«
431 Bash-4.2 added support for negative array indices and string
432 extraction (count backward from the last element). Apply this feature
433 to print all but the first and last character of the last word of
434 each input line.
435 », «
436 The script below implements a loop which reads lines from stdin into
437 an array. In each interation of the loop we use CMD(«${arr[-1]}»)
438 to get the last word of the line. Substring expansion with -1 as the
439 offset value refers to the last character within the word.
440
441 #!/bin/bash
442 # The -a option assigns the words of each line to an array variable.
443 while read -a arr; do
444 #
445 # If the input line contains only whitespace, there is
446 # nothing to do.
447 ((${#arr[@]} == 0)) && continue
448 #
449 # Negative array indices count back from the end of the
450 # array. In particular the index -1 references the last
451 # element. Hence ${arr[-1]} is the last word.
452 #
453 # To print the first and the last character of the last
454 # word, we use substring expansion:
455 # ${parameter:offset:length} expands to up to length
456 # characters of the value of parameter starting at the
457 # character specified by offset. As for array indices,
458 # negative offsets are allowed and instruct bash to use
459 # the value as an offset from the *end* of the value,
460 # with -1 being the last character.
461 #
462 # A negative offset must be separated from the colon by
463 # a space to avoid confusion with the :- expansion (use
464 # default values). For example, ${var:-1:1} expands to
465 # the string "1:1" if a is unset (and the value of var
466 # otherwise).
467 echo "${arr[-1]: 0: 1} ${arr[-1]: -1: 1}"
468 done
469
470 »)
471
472 SECTION(«Signals»)
473
474 - trap
475 - exit code 128 + n
476
477 EXERCISES()
478
479 - Run CMD(«sleep 10»), interrupt the command with CMD(«CTRL+C») and
480 examine CMD(«$?»). Hint: CMD(«trap -l») prints all signal numbers.
481 - The REFERENCE(«stale_tmpfile.bash», «script») below is flawed
482 in that it leaves a stale temporary file when interrupted with
483 CMD(«CTRL+C»). Fix this flaw by trapping CMD(«SIGINT»).
484
485 SECTION(«Shell Options»)
486
487 - Confusing:
488 - _many_ options, some really weird ones
489 - two ways to set options: CMD(«set»), CMD(«shopt»)
490 - CMD(«set +option») _disables_ CMD(«option»)
491 - aim: Introduce examples for the most useful options
492 - CMD(«-x»): debugging
493 - CMD(«-u»): parameter expansion is treated as error for unset variables
494 - CMD(«-e»): exit on first error
495 - pipefail: Get _all_ exit codes of a pipeline
496 - nullglob: avoid common pitfalls with pathname expansion
497 - extglob: activate extended pattern matching features
498
499 EXERCISES()
500
501 - Find at least two bugs in the REFERENCE(«catch_the_bug.bash»,
502 «script») below. Run the script twice, once with
503 CMD(«bash catch_the_bug.bash») and once with CMD(«bash -x
504 catch_the_bug.bash»). Compare the output.
505 - There is a subtle bug in the the
506 REFERENCE(«HungarianCamelSquare.bash», «HungarianCamelSquare.bash»)
507 script below. Run the script with and without bash's CMD(«-u») option
508 and compare the error messages. Discuss whether it is reasonable to
509 add CMD(«set -u») to existing scripts.
510 - What's the exit code of the pipeline CMD(«/notthere | wc -l»)?
511 Run CMD(«set -o pipefail»), then repeat the command. Search the bash
512 man page for CMD(«pipefail») and learn about the CMD(«PIPESTATUS»)
513 array variable. Repeat the above command and examine the contents
514 of CMD(«PIPESTATUS»).
515 - Assume that CMD(«/etc») contains only "reasonable" file
516 names (without space or other "funny" characters). Yet the
517 REFERENCE(«count_config_files.bash», «count_config_files.bash»)
518 script is buggy. Point out the flaw _before_ you try it out, then run
519 it to confirm. Insert CMD(«shopt -s nullglob») before the loop and
520 run the script again. Search the bash manual page for CMD(«nullglob»)
521 for an explanation.
522
523 HOMEWORK(«
524 The REFERENCE(«rm_tmp.bash», «rm_tmp.bash») script is seriously
525 flawed and would sooner or later create major grief if the CMD(«rm»)
526 command was not commented out. Find at least three bugs in it. Run
527 CMD(«bash rm_tmp.bash /notthere») and CMD(«bash -e rm_tmp.bash
528 /notthere») to see the CMD(«-e») option in action.
529 », «
530
531 - If the CMD(«cd») command fails, the CMD(«rm») command will be
532 executed in the current directory. This can happen for several reasons:
533 - CMD(«$1») does not exist,
534 - CMD(«$1») is not a directory,
535 - The executing user has no permissions to change into CMD(«$1»),
536 - CMD(«$1») contains whitespace characters,
537 - CMD(«$1») is a directory on a network share which is currently
538 unavailable. This does not happen with NFS, but may happen with CIFS
539 (Microsoft's Common Internet File System).
540 - If no argument is given, the CMD(«rm») command will be executed
541 in the home directory.
542 - The CMD(«rm») command does not remove all files: filenames starting
543 with a dot will be omitted.
544 - If the directory contains more files than the maximal number of
545 arguments in a command line, the CMD(«rm») command fails. The limit
546 depends on the system, but is often as low as 32768.
547 - If the directory contains a file named CMD(«-r»), the directory
548 will be removed recursively.
549 - If CMD(«$1») is an empty directory, the command fails because
550 there is no file named CMD(«"*"»). See the CMD(«nullglob») shell
551 option if you don't know why.
552 - The command fails if CMD(«$1») contains subdirectories.
553 - Even the CMD(«echo») command is buggy: If there is a file
554 CMD(«-n»), it will be treated as an option to CMD(«echo»).
555 »)
556
557 HOMEWORK(«
558 - Suppose you'd like to remove all leading occurences of the character
559 CMD(«"a"») from each input line. The script should read input lines
560 from CMD(«stdin») and write its output to CMD(«stdout»). For
561 example, the input line CMD(«aabba») should be transformed into
562 CMD(«bba»).
563
564 - Write a bash script that runs a suitable external command of
565 your choice (e.g., CMD(«sed»), CMD(«awk»), CMD(«perl») or
566 CMD(«python»)) for each input line.
567 - Come up with an alternative script that does not run any commands.
568 - Implement yet another version that uses extended globbing.
569
570 - Create a suitable input file with 100000 lines by running
571 CMD(«base64 < /dev/urandom | head -n 100000 > foo»). Test the
572 performance of the three implementations of the above script by
573 executing CMD(«time script < foo») and discuss the result.
574 », «
575 - Bash script with external command:
576
577 #!/bin/bash
578
579 while read line; do
580 sed -e 's/^a\+//' <<< "$line"
581 done
582
583 - Bash script without external command (note that CMD(«printf»)
584 is a shell builtin):
585
586 #!/bin/bash
587
588 while read line; do
589 n=0
590 while [[ "${line:$n:1}" == 'a' ]]; do
591 let n++
592 done
593 printf '%s\n' "${line:$n}"
594 done
595
596 - Bash script with extended globbing:
597
598 #!/bin/bash
599
600 shopt -s extglob
601
602 while read line; do
603 printf '%s\n' "${line/*(a)}"
604 done
605
606 - Running times:
607 - external command: 289s
608 - without external command, without extglob: 4s
609 - extglob: 8s
610
611 - Discussion: External commands hurt plenty. Try to avoid them
612 inside of loops which execute many times. The extglob feature is
613 handy but is still twice as expensive than the open-coded version
614 which avoids pattern matching alltogether. Note that the simple
615 CMD(«sed -e 's/^a\+//' foo») also does the job, and is even two
616 orders of magnitude faster than the fastest bash version. However,
617 this approach is not very flexible, hence unsuitable for real world
618 applications which do more than just write the transformed string
619 to stdout.
620 »)
621
622 SECTION(«Miscellaneous»)
623
624 - IFS
625 - read -ie
626 - ** (globbing)
627 - prompt
628 - Indirect variable referencing (eval, ${!x}, nameref)
629
630 EXERCISES()
631
632 - Write a bash script which prints the username and login shell of
633 each user defined in CMD(«/etc/passwd»). Hint: Set CMD(«IFS»)
634 and employ the bash CMD(«read») builtin with suitable options to
635 read each line of CMD(«/etc/passwd») into an array. Compare your
636 solution with this REFERENCE(«print_login_shells.bash», «script»).
637 - Run CMD(«read -p "> " -ei "kill -9 -1" t; echo "you entered:
638 $t"») and note how it provides nice readline-based editing. Check
639 CMD(«bash») man page for other options to the CMD(«read») builtin,
640 like CMD(«-s») and CMD(«-t»).
641 - Run CMD(«ls ~/**/*.pdf»). Search the bash manual page for
642 CMD(«**») and CMD(«globstar») to understand the meaning of the
643 CMD(«**») pattern in pathname expansion. Next, run CMD(«shopt -s
644 globstar && ls ~/**/*.pdf») and marvel.
645 - Is there a way in bash to distinguish between undefined variables
646 and variables which have been set to the emtpy string? Hint: examine
647 the difference between CMD(«${x-42}») and CMD(«${x:-42}»).
648 - Setting the environment variable CMD(«PROMPT_COMMAND»)
649 to a function instructs bash to call this function prior to
650 issuing the prompt. Run CMD(«prompt_command() { PS1="$PWD > ";
651 }; PROMPT_COMMAND=prompt_command») to change your prompt. Modify
652 the function to replace the middle part of the path by '...' if
653 CMD(«$PWD») exceeds 10 characters.
654 - During parameter expansion, if the first character of a parameter
655 is an exclamation point (!), bash uses the value of the variable
656 formed from the rest of parameter as the name of the variable rather
657 than the value of the parameter itself. This is known as _indirect
658 expansion_. Run CMD(«a=42; x=a; echo ${!x}») to see the effect.
659 - Examine and run the REFERENCE(«minmax.bash», «minmax script»)
660 whose CMD(«minmax()») function is given the _name_ CMD(«X») of a
661 variable, and a sequence of positive integers. The function computes
662 the minimum and the maximum given value and sets the variables
663 CMD(«X_min») and CMD(«X_max») accordingly.
664
665 HOMEWORK(«
666 Get rid of the CMD(«eval») statement in the
667 REFERENCE(«minmax.bash», «minmax script») by passing
668 variables declared with CMD(«-n») to assign the CMD(«namref»)
669 attribute. Hint: search for (nameref) in the bash manual.
670 »)
671
672 HOMEWORK(«
673 Read the CMD(«bashbug») manual page and discuss
674 under which circumstances one should file a bug report.
675 Download the source code of latest version of bash from
676 XREFERENCE(«ftp://ftp.gnu.org/pub/gnu/bash», «gnu ftp server»),
677 apply all patches found in the CMD(«bash-4.3-patches») subdirectory
678 and compile the package. Run the compiled executable and execute
679 CMD(«echo ${BASH_VERSINFO[@]}»).
680 »)
681
682 SECTION(«Job Control»)
683
684 - suspend/resume selected processes
685 - POSIX.1 (1988)
686 - aim: understand foreground/background jobs, Ctrl+Z, Ctrl+C,
687 CMD(«fg»), CMD(«bg»)
688 - job <=> process group <=> pipeline (+descendants) <=> PGID
689 - (interactive) session := collection of process groups
690 - setsid() syscall creates new session, PGID := PID of calling process
691 - session leader: process which called setsid(), SID: PID of session
692 leader
693 - terminal's current process group (TPGID)
694 - TPGID determines foreground PG = CMD(«{P: PGID(P) == TPGID}»)
695
696 EXERCISES()
697
698 - Examine all fields in the output of CMD(«ps j»).
699 - Assume a typical scenario with one background process and another
700 process running in the foreground. How many sessions are there? Which
701 of the three processes are session leaders? Determine all process
702 groups. Verify your result by running CMD(«sleep 100 & ps j»).
703 - What happens if a background process tries to read from
704 CMD(«stdin»)? Verify your answer by executing CMD(«cat &»).
705 - What happens if the session leader terminates while there are
706 still processes running in a background process group? To find out,
707 open a terminal, run CMD(«sleep 100&») and kill the session leader
708 (the shell) with CMD(«kill -9 $$»). Open another terminal and
709 execute CMD(«ps -aj») and examine the row that corresponds to the
710 CMD(«sleep») process.
711 - Look at how bash handles a pipeline by executing CMD(«ps xj |
712 cat»).
713 - Verify that in the output of CMD(«ps j») the TPGID and the PID
714 columns coincide while the two columns differ if the command is run
715 in the background (CMD(«ps j &»)). Determine the foreground process
716 group in both cases.
717 - Read the section on job control in the bash manual and make yourself
718 familiar with the various ways to refer to a job in bash (CMD(«%»),
719 CMD(«%n»), CMD(«%-,»), CMD(«%+»)).
720
721 SUPPLEMENTS()
722
723 SUBSECTION(«stale_tmpfile.bash»)
724
725 <pre>
726
727 #!/bin/bash
728 f=$(mktemp) || exit 1
729 echo "starting analysis, temporary file: $f"
730 sleep 100
731 echo "done, removing $f"
732 rm -f "$f"
733
734 </pre>
735
736 SUBSECTION(«self-list.bash»)
737
738 <pre>
739
740 #!/bin/bash
741 IFS='
742 '
743 a=($(cat $0))
744 for ((i = 0; i < ${#a[@]}; i++)); do
745 echo "$((i + 1)): ${a[$i]}"
746 done
747
748 </pre>
749
750 SUBSECTION(«rot13.bash»)
751
752 <pre>
753
754 #!/bin/bash
755 declare -A h=(
756 [a]=n [b]=o [c]=p [d]=q [e]=r [f]=s [g]=t [h]=u [i]=v [j]=w [k]=x
757 [l]=y [m]=z [n]=a [o]=b [p]=c [q]=d [r]=e [s]=f [t]=g [u]=h [v]=i
758 [w]=j [x]=k [y]=l [z]=m
759 )
760
761 while read -r line; do
762 for ((i =0; i < ${#line}; i++)); do
763 c="${line:$i:1}"
764 echo -n ${h[$c]:-$c}
765 done
766 echo
767 done
768
769 </pre>
770
771 SUBSECTION(«catch_the_bug.bash»)
772
773 <pre>
774
775 #!/bin/bash
776 if (($# == 0)); then
777 # no argument given, choose a random number instead
778 x=$(($RANDOM / 3276 + 1)) # between 1 an 10
779 else
780 x=$1
781 fi
782 echo "1/$x is approximately $((100 / $x))%"
783
784 </pre>
785
786 SUBSECTION(«HungarianCamelSquare.bash»)
787
788 <pre>
789
790 #!/bin/bash
791 declare -i ThisVariableIsATemporaryCounter
792
793 for ((ThisVariableIsATemporaryCounter=0; ThisVariableIsATemporaryCounter < 10; ThisVariableIsATemporaryCounter++)); do
794 echo "$ThisVariableIsATemporaryCounter * $ThisVariableIsATemporaryCounter is $(($ThisVariableIsATenporaryCounter * $ThisVariableIsATemporaryCounter))"
795 done
796
797 </pre>
798
799 SUBSECTION(«rm_tmp.bash»)
800
801 <pre>
802
803 #!/bin/bash
804 echo "removing all temporary files in $1"
805 cd $1
806 echo removing *
807 # rm *
808
809 </pre>
810
811 SUBSECTION(«count_config_files.bash»)
812
813 <pre>
814
815 #!/bin/bash
816 for c in {a..z}; do
817 files=(/etc/$c*.conf)
818 echo "There are ${#files[@]} config files in /etc that start with $c: ${files[@]}"
819 done
820
821 </pre>
822
823 SUBSECTION(«print_login_shells.bash»)
824
825 <pre>
826
827 #!/bin/bash
828 while IFS=: read -ra a; do
829 echo "${a[0]} ${a[6]}"
830 done < /etc/passwd
831
832 </pre>
833
834 SUBSECTION(«minmax.bash»)
835
836 <pre>
837 minmax()
838 {
839 local var min max
840
841 var="$1"
842 shift
843 min=$1
844 max=$1
845 shift
846 while (($#)); do
847 (($1 < $min)) && min=$1
848 (($1 > $max)) && max=$1
849 shift
850 done
851 eval ${var}_min=$min
852 eval ${var}_max=$max
853 }
854
855 print_minmax()
856 {
857 local var="$1"
858 local min="${var}_min" max="${var}_max"
859
860 echo "min: ${!min}, max: ${!max}"
861 }
862
863 minmax a 3 4 2 9 4
864 print_minmax a
865
866
867 </pre>