Fix two broken links.
[aple.git] / Command_Line_Utilities.m4
1 TITLE(«
2
3 Free yourself from all the confusion «...» Forget about desktop
4 decor, senseless glitter, and animations «....» Say goodbye «...» to
5 the rodent and welcome the ultimate interface to your computer. --
6 Ratpoison propaganda
7
8 », __file__)
9
10
11 SECTION(«Essential Command Line Tools»)
12
13 - man, apropos
14 - cd, pwd, ls, stat
15 - rm, cp, mv, ln
16 - echo, printf, cat
17 - less
18 - tar
19 - file
20 - find, xargs
21 - cut, sort, uniq, wc
22 - mkdir, rmdir
23 - df, du
24 - nice, ionice
25 - ps, kill, top, htop, jobs
26 - ping, wget
27
28
29 SECTION(«CMD(«gzip»), CMD(«bzip2») and CMD(«xz»)»)
30
31 EXERCISES()
32
33 - create the file CMD(reads.fq) from the supplement using CMD(cat
34 > reads.fq). Discuss the usage of CMD(Ctrl-d) versus CMD(Ctrl-c)
35 to end the CMD(cat) process.
36 - run CMD(file) on CMD(reads.fq). Read it using CMD(less). Compress
37 the file using CMD(gzip). Run CMD(file) again. Try reading it.
38
39 HOMEWORK(«
40 Reading from the special file CMD(«/dev/urandom») returns random
41 data. Explain what the command CMD(«head -c 10000000 /dev/urandom |
42 base64 > foo») does. Execute CMD(«cp foo bar») and CMD(«cp foo
43 baz») to create two copies of CMD(«foo»). Use the CMD(«gzip»),
44 CMD(«bzip2») and CMD(«xz») commands to compress the three files
45 using three different compression algorithms. Hand in the running
46 time of each command, the sizes of the compressed files and your
47 conclusion about which algorithm/program should be preferred.
48 », «
49 The CMD(«head | base64») command reads 10M of random data and encodes
50 these with the base64 algorithm, which sequentially encodes three
51 input bytes into four output bytes, representing the six significant
52 bits of each output byte as a character in the set CMD(«{a-z, A-Z,
53 0-9, +, /}»). The result is written to the file named CMD(«foo»).
54 The commands CMD(«gzip foo»), CMD(«bzip2 bar») and CMD(«xz baz»)
55 each compress one of the files. Running times were 0.9s, 3.1s, 8.9s
56 respectively. It's surprising to see how big the differences of the
57 three running times are, although it is kind of expected that the most
58 "modern" program, CMD(«xz»), has the highest running time. At first
59 sight it might also be surprising that the sizes of the compressed
60 files are almost identical (all three were almost exactly 10M, the
61 size of the unencoded data). But when taking into account that we
62 are dealing with random data, it is clear that one can only expect
63 a compression factor of 3/4 due to the base64 encoding, no matter
64 which algorithm is used. For non-random data the file sizes would have
65 differed much more, and it depends on the use case which algorithm is
66 the best. For example, if the compressed file will be downloaded many
67 times from an FTP server, it might be worth to spend extra CPU time
68 during compression to make the file as small as possible in order to
69 save bandwith.
70 »)
71
72 SECTION(«CMD(«sed»), CMD(«grep»), CMD(«tr»)»)
73
74 EXERCISES()
75
76 - Create a file with three lines (CMD(printf "one\ntwo\nthree\n" >
77 foo)). Use CMD(sed) or CMD(tr) to replace all newline characters with
78 spaces. Discuss cases where it is not an option to open the file in
79 an editor.
80 - Unpack (or recreate) the file CMD(reads.fq) from the previous
81 exercise.
82 - Extract only read2 from CMD(reads.fq) using CMD(grep). To do that,
83 check CMD(man grep) for the CMD(-A) and CMD(-B) options.
84 - Use CMD(sed) to extract only the lines containing the ID. What are
85 the dangers of doing it the intuitive way using grep?
86 - A little more advanced: use sed to write a very short, highly
87 efficient FastQ to FastA converter.
88
89 HOMEWORK(«
90 Find a command which prints the usage of all file systems which are
91 NFS-mounted from a specific server (e.g., neckar, or arc).
92 », «
93 This exercise shows that things can be trickier than they look like at
94 first glance. The complications are (a) the string "CMD(«arc»)" may
95 well be part of an unrealated mount point, and (b) the server name can
96 be specified as either CMD(«arc») or CMD(«arc.eb.local»). Hence,
97 the simple CMD(«df -t nfs | grep arc») is not robust enough, at
98 least for scripts for which subtle future breakage (when another file
99 system is mounted) should be avoided. A better variant is
100
101 df -t nfs | grep '^arc\(\.eb\.local\)*:'
102 »)
103
104 SECTION(«CMD(«chmod»), CMD(«chgrp»)»)
105
106 - Change permissions or group.
107
108 EXERCISES()
109
110 - Discuss the implications of project directories with mode 777.
111 - Use the -group option to CMD(find) to locate files and directories
112 whose GID is ebio. Discuss whether changing the GID to e.g. abt6 with
113 CMD(chgrp -R) would trigger a backup of these files. For the CMD(chgrp)
114 command, which flags besides -R would you specify in order to handle
115 symbolic links correctly?
116 - Discuss why CMD(chown) is root-only.
117
118 HOMEWORK(«
119 Come up with a CMD(find | xargs chmod) command that turns on the
120 group-executable bit for all files which are executable for its owner,
121 but leaves non-executable files unchanged. Does your command work
122 with filenames containing spaces. Does it work if a filename contains
123 a newline character?
124 », «
125 CMD(«find ~ -perm /u+x -not -perm /g+x -print0 | xargs -0 chmod
126 g+x»). The CMD(«-not -perm /g+x») part is not strictly necessary
127 but it may speed up the command, and it preserves the ctime of those
128 files which are already group-executable. The CMD(«-print0»)
129 option is the only way to make this type of command safe because any
130 other character can be part of a path. So it is strongly recommend
131 to always use it whenever possible. Unfortuntately this feature is
132 not part of the POSIX standard. It is supported on Linux and FreeBSD
133 (hence MacOS) though.
134 »)
135
136 SECTION(«The LDAP user database»)
137 - LDAP: campus-wide user database
138 - stores your personal information: name, password, phone number,...
139 - read-only access w/o auth
140 - write access through web app
141 - ldapsearch, id, finger, last, w
142
143 EXERCISES()
144
145 - Run the commands CMD(finger $LOGNAME) and discuss the meaning of
146 all output fields.
147 - Run the command CMD(id). What's the meaning of the terms CMD(uid)
148 and CMD(gid) shown in the output?
149 - Show your own LDAP entry: CMD(ldapsearch -x uid=$LOGNAME). Use a
150 similar command to show the entry of somebody who left the
151 institute. How can one tell whether an account is active?
152 - List all abt6 users: CMD(ldapsearch -x cn=abt6)
153 - use CMD(ldapsearch -x) to find other people at our institute with
154 the same surname ("sn") as you.
155 - use CMD(id) to figure out what groups you are in.
156
157 HOMEWORK(«
158 Find all former members of your department or group.
159 », «
160 Example for department 1:
161
162 ldapsearch -x homeDirectory | grep -A 1 vault | grep '^homeDirectory: /ebio/abt1'
163 »)
164
165 SECTION(«CMD(«rsync»)»)
166
167 - file-copying tool by Andrew Tridgell (1996)
168 - remote copying via ssh
169 - synchronization
170 - performant, sends only differences
171 - aim: know the most common options
172
173 EXERCISES()
174
175 - Preparation: Create a scratch directory in /tmp and store 10 10M
176 text files there: CMD(«for i in $(seq 10); do (head -c 7500000
177 /dev/urandom | base64 > $RANDOM &) ; done»).
178 - Create a copy (also in CMD(«/tmp»)) with CMD(«rsync») and measure
179 the running time: CMD(«time rsync -ax $src $dest»). Check the rsync
180 manual for the meaning of these two options.
181 - Modify and remove some files in source directory, run rsync again,
182 this time with the CMD(«-ax --delete») options to propagate your
183 changes to the destination.
184 - Suppose files which contain a "1" in its file name are generated
185 any you don't want to synchronize these. Find the correct argument
186 for rsync's CMD(«--exclude») option. Use the CMD(«--dry-run -v»)
187 options to check.
188 - Find a way to use rsync for a non-local copy where the remote ssh
189 server listens on a port different than 22. To try this, forward
190 the port 24865 + $UID to port 22 with CMD(«ssh -NfL $((24865 +
191 $UID)):localhost:22 localhost») so you can ssh into localhost through
192 this port.
193 - Look at the XREFERENCE(http://people.tuebingen.mpg.de/maan/dss/,
194 dyadic snapshot scheduler).
195 - Suppose source and destination directories are on different hosts and
196 contain slightly different versions of a single huge file. For example,
197 somewhere near the beginning a small part of the file was deleted,
198 and another small part was appended to the end of the file. Suppose
199 further that the file is so large (or the network so slow) that
200 copying all of it would take days. Think about an algorithm that
201 finds the above differences without transferring the whole file.
202
203 HOMEWORK(«
204 The XREFERENCE(https://en.wikipedia.org/wiki/MD4, Wikipedia page) on
205 the CMD(«MD4») message-digest algorithm states that the security of
206 CMD(«MD4») has been severely compromised. The CMD(«MD5») algorithm
207 is also known to be broken for years. Yet these two algorithms are
208 an essential part of rsync. Is this a problem for the correctness
209 of CMD(«rsync»)?
210 », «
211 The fact that one can find hash collisions for CMD(«MD4»)
212 and CMD(«MD5») is usually not a problem for the integrity of
213 CMD(«rsync»). First, the weak hash algorithms are not used for
214 any type of authentication, since CMD(«rsync») relies on SSH
215 for this. Second, why would a malicious CMD(«rsync») user want
216 to modify the files on the source or destination to create a hash
217 collision if he already has access to these files? On the other hand,
218 for non-manipulated files, the probability of a hash collision is so
219 small that other types of failures are much more likely. Finally, the
220 CMD(«MD4») and CMD(«MD5») algorithms are used in combination with
221 other checksums, so modifying a file while keeping its CMD(«MD5»)
222 hash the same is not enough to fool CMD(«rsync»).
223 »)
224
225 HOMEWORK(«
226 Describe the idea of rolling checksums and how they are used in
227 CMD(«rsync»).
228 »)
229
230 SECTION(«The Cron service»)
231
232 - cron daemon (CMD(«crond»)) executes scheduled commands
233 - CMD(«crontab») (file): table used to drive crond
234 - CMD(«crontab») (program): command to install, update, or list tables
235
236 EXERCISES()
237
238 - Preparation: Set the CMD(«EDITOR») variable to your favorite editor.
239 - Run CMD(«crontab -l») to see your current crontab.
240 - Set up a cron job which runs every minute and appends the current
241 date to a file.
242 - Note that there are two CMD(«crontab») manual pages. The command
243 CMD(«man crontab») gives the manual for the crontab program. Run
244 CMD(«man 8 crontab») to see the manual page for the configuration
245 file. Look at the examples at the end.
246 - Write a script which checks if a device is mounted and if yes
247 execute some operations.
248
249 HOMEWORK(«
250 Discuss the difference between a cron job and a script which does
251 something like CMD(«while :; do run_experiment; sleep $TIMEOUT;
252 done»). Describe what happens when
253
254 - the user logs out or closes the terminal,
255 - the server gets rebooted,
256 - CMD(«run_experiment») runs for more than CMD(«$TIMEOUT») seconds.
257 », «
258 There are many differences:
259
260 - A cron job is started by the cron daemon and is executed in different
261 environment where some variables like CMD(«PATH») might have a
262 different value.
263 - The cron deamon is started automatically during boot, so a cron
264 script will still be run after a reboot while the command loop will
265 not be restarted automatically in this case.
266 - If the user logs out or closes the terminal, the command loop might
267 be killed. This depends on whether it was started in the background
268 and whether CMD(«stdin»), CMD(«stdout») and CMD(«stderr»)
269 were redirected.
270 - The cron job might be started twice if the previous invocation is
271 still running. This can happen if a file system is slow or the job
272 is running for longer than the scheduling period.
273 - The timing of the command loop will drift over time because the
274 running time of the script is not taken into account. The cron script,
275 on the other hand, will always run at the specified times.
276 »)
277
278 SECTION(«Regular Expressions»)
279
280 - regex: sequence of characters that forms a search pattern
281 - Kleene (1956)
282 - compilation: RE -> finite automaton
283 - Chomsky hierarchy
284 - basic, extended, perl
285
286 EXERCISES()
287
288 - Understand the difference between basic, extended and perl regular
289 expressions. Discuss which ones should be preferred in various cases.
290 - In a web service, is it safe to match a user-specified
291 expression against a known input? Does it help if both
292 regex and input are bounded by, say, 1000 characters? Read the
293 XREFERENCE(«https://en.wikipedia.org/wiki/ReDoS», «ReDos») wikipedia
294 page for the answer.
295 - Match the "evil" extended regex CMD(«(a|aa)+») against a long
296 string which contains only "CMD(«a»)" characters. Try various
297 regex implementations (CMD(«sed»), CMD(«grep»), CMD(«awk»),
298 CMD(«perl»), CMD(«python»)).
299 - Discuss the consequences of the fact that perl regular expressions
300 do EMPH(«not») necessarily correspond to regular languages in the
301 Chomsky hierarchy.
302 - Is it possible to construct a regular expression that matches a
303 line of text if and only if the line is a syntactically correct perl
304 or C program?
305
306 HOMEWORK(«
307 Find a regular expression which matches any number of "CMD(«a»)"
308 characters, followed by the EMPH(same) number of "CMD(«b»)"
309 characters. Alternatively, prove that no such regular expression
310 exists.
311 », «
312 If there was a regular expression that matched any number of
313 "CMD(«a»)'s" followed by the same number of "CMD(«b»)'s" there would
314 exist a finite automaton that describes the characters that have been
315 matched so far. But apparently such an automaton would necessarily
316 have infinitely many states. Hence no regular expression exists that
317 matches all such strings. Another way to prove this is to apply the
318 so-called EMPH(«pumping lemma»).
319
320 The take-away of this exercise is (a) to get a feeling about what
321 can be described by regular expressions, and (b) that understanding
322 the underlying theory can help you to avoid wasting a lot of effort
323 on trying to do provably impossible things.
324 »)
325
326 SECTION(«CMD(«awk»)»)
327
328 - general purpose text utility
329 - Aho, Weinberger, Kernighan (1977)
330 - search files for certain patterns, perform actions on matching lines
331 - specified in POSIX, here: gawk (GNU awk)
332
333 EXERCISES()
334
335 - What does the following awk program do? CMD(«ls -l | awk '{x +=
336 $5} END {print x}'»)
337 - Check the awk uses in the
338 XREFERENCE(«http://ilm.eb.local/gitweb/?p=user-info;a=blob;f=scripts/admin/cmt;hb=HEAD»,
339 «cluster management tool»).
340 - Write an CMD(«awk») program which prints the length of the longest
341 input line.
342 - Write an awk program which uses an associative array to print how
343 often different words appear in the input.
344 - Check the
345 XREFERENCE(«http://www.gnu.org/software/gawk/manual/gawk.html»,
346 «gawk manual») to get an idea of CMD(«awk») features and its
347 complexity.
348 - Sometimes it is useful to initialize an awk variable from a shell
349 environment variable. Learn to use the CMD(«--assign») option to
350 achieve this.
351 - Discuss what the CMD(«--traditional») option to CMD(«awk»)
352 does and when it should be used.
353
354 HOMEWORK(«
355 On any system, determine the average file system size and the average
356 percentage of used space.
357 », «
358 The command CMD(«df -P») prints both the size and the percentage
359 of used space for each file system as the second and fifth column.
360 The following pipeline prints the desired average percentages:
361
362 df -P | awk '{s2 += $2; s5 += $5} END {print s2 / NR, s5 / NR}'
363 »)
364
365 SECTION(«CMD(«screen»)»)
366
367 - terminal multiplexer
368 - Laumann, Bormann (1987), GNU
369 - sessions, detach, attach
370 - multiuser sessions
371
372 EXERCISES()
373
374 - Start screen, run CMD(«ls»), detach the session, log out, log in,
375 re-attach the session.
376 - In a screen session with multiple windows, type CMD(«CTRL+a "»)
377 to see the window list.
378 - The key to move the cursor to the beginning of the command
379 line is mapped to the same character as the screen escape key,
380 CMD(«CTRL+a»). Learn how to workaround this ambiguity.
381 - Type CMD(«CTRL+a :title») to set the window title.
382 - Put the following line to your .screenrc: CMD(«caption always
383 "%{cb} «%{wb}Act: %-w%{cb}%>%n(%t)%{-}%+w%<%{cb}»%{-}"») and see
384 what it does.
385 - Learn to copy and paste from the screen scrollback buffer
386 (CMD(«CTRL+a ESCAPE»)). Increase the size of the scrollback buffer
387 in your .screenrc file.
388 - Learn how to use CMD(«split») and CMD(«split -v»).
389 - To allow your colleage CMD(«nubi») to attach your screen session
390 in read-only mode, create a suitable REFERENCE(«.screenrc.multi»,
391 «multiuser screenrc») file and start a session with this config file:
392 CMD(«screen -C ~/.screenrc.multi -S multi»). Ask CMD(«nubi»)
393 to run CMD(«screen -x $OWNER/multi»), where CMD(«OWNER») is your
394 username. Read the section on the CMD(«aclchg») command in the
395 screen manual for how to allow write access. Discuss the security
396 implications.
397
398 SECTION(«CMD(«adu»)»)
399
400 - advanced disk usage
401 - creates database, only slow "once"
402 - produces summary or list of largest directories
403
404 EXERCISES()
405
406 - Create an adu database from your project directory: CMD(«adu
407 -C -d $DB -b $BASE»), where CMD(«$DB») is the (non-existing)
408 directory to create for the database, and CMD(«$BASE») is a existing
409 (sub-)directory of your storage project.
410 - List the 10 largest directories: CMD(«adu -S -d $DB -s "-m
411 global_list -l 10"»), then the top-10 directories with respect to
412 file count: CMD(«adu -S -d $DB -s "-m global_list -l 10 -s f"»).
413 - Print the 10 largest directories, but consider only those which
414 contain the letter "CMD(«a»)".
415 - Read the adu manual page to learn how to customize the output with
416 format strings.
417
418 SECTION(«CMD(«make»)»)
419
420 - most often used to build software, useful also in other situations
421 - Stuart Feldman (1976), POSIX, GNU
422 - regenerates dependent files with minimal effort
423 - keeps generated files up-to-date without running your entire workflow
424 - Makefile abstracts out dependency tracking
425 - rule, recipe, target, prerequisites
426
427 EXERCISES()
428
429 - Look at this
430 XREFERENCE(«http://ilm.eb.local/gitweb/?p=user-info;a=blob;f=backup/Makefile;hb=HEAD»,
431 «simple Makefile») which creates the bareos configuration files
432 from a (public) template file and a (secret) file that contains
433 passwords. Identify targets, recipes, rules, prerequisites. -
434 Create a couple of text files with CMD(«head -c 100 /dev/urandom |
435 base64 -w $(($RANDOM / 1000 + 1)) > $RANDOM.txt»). Write a Makefile
436 which creates for each CMD(«.txt») file in the current directory a
437 corresponding CMD(«.wc») file which contains the number of lines
438 in the text file. Extend the Makefile by a rule which reads the
439 CMD(«.wc») files to create an additional file CMD(«sum»), which
440 contains the sum of the line counts (CMD(«cat *.wc | awk "{s += $1}
441 END {print s}"»)). Draw the dependency graph which is described in
442 your Makefile. Modify some files (CMD(«echo hello >> $NUMBER.txt»))
443 and run CMD(«make») again. Discuss the time to compute the new
444 sum. Add a CMD(«clean») target which lets you remove all derived
445 files (CMD(«.wc»), CMD(«sum»)) with CMD(«make clean»).
446 - There are two flavors of CMD(«make») variables. Read the
447 XREFERENCE(«http://www.gnu.org/software/make/manual/make.html»,
448 «make documentation») to understand the difference.
449 - Look at the list of automatic CMD(«make»)
450 variables in section on implicit rules of the
451 XREFERENCE(«http://www.gnu.org/software/make/manual/make.html»,
452 «make manual»).
453
454 HOMEWORK(«
455 Explain the difference between simply and recursively expanded make
456 variables. Discuss under which circumstances either of the two should
457 be used in a Makefile.
458 »)
459
460 SECTION(«CMD(«autoconf»)»)
461
462 - creates shell scripts that configure software packages
463 - makes life easier for the EMPH(users) of your software
464 - Mackenzie (1991), GNU, written in perl, uses m4
465 - idea: test for features, not for versions
466
467 EXERCISES()
468
469 - Preparation: Assume for simplicity that your software package
470 consists only of a single script REFERENCE(«s1l», «s1l») which
471 prints the sha1sum of each file in the given directory. Create this
472 script in a scratch directory and add the REFERENCE(«configure.ac»,
473 «configure.ac») and REFERENCE(«Makefile.in», «Makefile.in»)
474 files as well.
475 - Run CMD(«autoconf») to create the configure script, then
476 run CMD(«./configure -h») to see the automatically generated
477 options. Run CMD(«configure --prefix $HOME»), CMD(«make») and
478 CMD(«make install») to install the "package". Notice how the value
479 specified to the CMD(«--prefix») option propagates from the command
480 line to CMD(«Makefile»).
481 - Draw a graph which illustrates how the generated files
482 depend on each other. Compare your graph with diagram on the
483 XREFERENCE(«https://en.wikipedia.org/wiki/Autoconf», «autoconf
484 wikipedia page»). In this larger diagram, identify those parts of
485 the graph which are present in the minimal CMD(«s1l») example.
486 - Suppose you spent a lot of hard work to improve your program to
487 use the much more secure sha2 checksum instead of sha1 (CMD(«sed
488 -i s/sha1/sha2/g *; mv s1l s2l»)). To give your customers the best
489 possible experience, you'd like the configuration step to fail
490 on systems where the CMD(«sha2sum») program is not installed
491 (notice the mantra: check for features, not versions). Add this
492 REFERENCE(«AC_PATH_PROG», «test») to CMD(«configure.ac»)
493 and run CMD(«autoconf») and CMD(«./configure») again to see it
494 failing gracefully at configure time (rather than at runtime).
495 - Discuss the pros and cons of configure-time checks vs. run-time
496 checks.
497 - Change CMD(«configure.ac») to not fail any more but to create
498 a Makefile which installs either s1l or s2l, depending on whether
499 CMD(«sha2sum») is installed.
500 - Read the "Making configure Scripts"
501 and "Writing configure.ac" sections of the
502 XREFERENCE(«http://www.gnu.org/software/autoconf/manual/autoconf.html»,
503 «autoconf manual»).
504 - Notice that CMD(«configure.ac») is in
505 fact written in the m4 macro language. Look at the
506 XREFERENCE(«http://www.gnu.org/software/m4/manual/m4.html», «m4
507 manual») to get an overview.
508
509
510 SUPPLEMENTS()
511
512 SUBSECTION(«FastQ File»)
513
514 <pre>
515 @read1
516 ATGCCAGTACA
517 +
518 DDDDDDDDDDD
519 @read2
520 ATCGTCATGCA
521 +
522 DDDDDDDDDDD
523
524 </pre>
525
526 SUBSECTION(«.screenrc.multi»)
527
528 <pre>
529 multiuser on
530 umask ?-wx
531 acladd nubi
532
533 </pre>
534
535
536 SUBSECTION(«configure.ac»)
537
538 <pre>
539
540 AC_INIT(«MPI sha1 list», «0.0.0», «me@s1l.org», «s1l», «http://s1l.mpg.de/»)
541 AC_CONFIG_FILES(«Makefile»)
542 AC_OUTPUT
543
544 </pre>
545
546
547 SUBSECTION(«Makefile.in»)
548
549 <pre>
550 all:
551 @echo "run make install to install s1l in @prefix@"
552 install:
553 install -m 755 s1l @prefix@/bin/
554
555 </pre>
556
557 SUBSECTION(«s1l»)
558
559 <pre>
560
561 #!/bin/sh
562 find "$1" -type f -print0 | xargs -0 sha1sum
563
564 </pre>
565
566 SUBSECTION(«AC_PATH_PROG»)
567
568 <pre>
569 AC_PATH_PROG(«SHA2SUM», «sha2sum»)
570 test -z "$SHA2SUM" && AC_MSG_ERROR(sha2sum required)
571 </pre>