2 Tell me and I forget. Teach me and I remember. Involve me
3 and I learn. – Benjamin Franklin
4 », __file__, «Advanced Problems in the Linux Environment»)
7 This practical training course covers basic and advanced Unix
8 and Linux topics. It targets scientists, programmers and system
9 administrators. Readers will find plenty of material and exercises
10 of varying difficulty. Move the pointer to the icon at the top left
11 corner to open the navigation menu.
14 SECTION(«Table of Contents»)
15 TABLE_OF_CONTENTS(«chapter_list_introduction»)
19 ifelse(PUBLIC(), «true», «dnl public version of the pages
21 <p> These pages were originally written to provide the necessary
22 background for using the IT infrastructure of the Max Planck Institute
23 for Developmental Biology and the Friedrich Miescher Laboratory in
24 Tübingen, Germany. Over time they morphed into a generic document
25 that was made public in December 2019. </p>
27 <p> The title is of course a pun on the famous book "Advanced
28 Programming in the Unix Environment" by W. Richard Stevens. While
29 Stevens' book centers around C programming, we try to convey
30 fundamental ideas without assuming substantial programming skills.
31 An elementary knowledge of shell and C programming is certainly
32 helpful, though. We also put an emphasis on Linux, which was still
33 in its infancy when Stevens' book was published in 1992. </p>
35 <p> All pages are served as static html files with no active
36 contents. They do not require javascript, work with any browser and do
37 not track the user in any way. In particular, we don't use cookies,
38 there is no "like" button, and we do not employ any web analysis
39 service like google analytics. Also, there are no advertisements of
42 SUBSECTION(«Exercises and Homeworks»)
44 <p> The exercises generally try to encourage the reader to
45 think about a specific topic rather than solve meaningless
46 problems mechanically. Many exercises suggest to examine further
47 literature. Longer or more challenging exercises are labelled as
48 homework. Solutions to homework exercises are only provided if at least
49 one person hands in a draft of a solution. To do so, send plain text
50 email to <a href="mailto:maan@tuebingen.mpg.de">Andre Noll</a>. </p>
52 SUBSECTION(«Feedback»)
54 <p> These pages get updated when errors are found, contents become
55 obsolete, or improvements are suggested. Feedback via the above
56 mailto link is appreciated. Besides solutions of homework exercises,
57 suggestions for additional topics or improvements of existing contents
58 are welcome. Please also point out unclear wording, grammar mistakes
63 <p> This work is published under the
64 <a href="https://www.gnu.org/licenses/fdl-1.3.html">GNU Free
65 Documentation License</a> (GFDL). You may copy, redistribute, and
66 modify these pages but all copies and derivatives are required to be
67 available under the same license. </p>
69 SUBSECTION(«See Also»)
71 <p> Go back to the author's <a href="..">home page</a>. </p>
73 », « dnl internal version: different Introduction and Motivation
75 <p> These pages aim to provide the necessary background for using the
76 IT infrastructure of the MPI for developmental biology and the FML.
79 <a href="http://ilm.eb.local/user-guide/">User guide</a>,
81 but are meant to be a practical training course rather than just a
82 text document. While the contents of the user guide are relevant to
83 almost every member of the institute, the primary target audience of
84 the Unix course is scientists with an emphasis on IT. In particular,
85 computer scientists who intent to use the compute cluster will find
86 plenty of material of varying difficulty to learn basic and advanced
87 topics related to Unix and Linux. </p>
89 <p> We first cover the general concepts of the Unix operating system while
90 later chapters focus on Linux specific topics and selected command
91 line tools. The exercises aim to convey understanding by inviting the
92 reader to read background information and to think about the topic
93 at hand. This is in contrast to many other tutorials which provide
94 quick solutions to frequently asked questions, targeting users who
95 do not wish to spend the time necessary to gain insight. Longer or
96 more challenging exercises are labelled as homework. </p>
98 <p> Feedback is appreciated. In fact, significant changes to these
99 pages are always triggered by the users asking questions. On one
100 hand, this makes sure that new material stays relevant to the target
101 audience. On the other hand, it also helps to fine-tune the degree
102 of difficulty and the detail of the solutions. As a general rule,
103 solutions to existing exercises are only provided if at least one
104 person hands in a proposal. To do so, send plain text email to <a
105 href="mailto:maan@tuebingen.mpg.de">Andre Noll</a>. </p>
107 SECTION(«Motivation»)
109 A quick glance at the table of contents reveals that the topics
110 of this course center around command line utilities rather than
111 software which comes with a <em> Graphical User Interface </em>
112 (GUI). While GUIs serve a purpose, especially for graphical tasks
113 (image manipulation, presentation writing, etc.), they are harmful
114 for scientific analysis. The remainder of this section explains why,
115 and tries to convince the reader that although <em> Command Line
116 Interfaces </em> (CLIs) can be scary in the beginning, learning how
117 to use them is worth the work.
119 SUBSECTION(«Why GUIs are harmful»)
122 <li> GUIs give people comfort by presenting choices. </li>
124 <li> GUIs limit you by limiting choices. </li>
126 <li> Science is a unique situation. </li>
128 <li> As a scientist, you don't want to be influenced by the GUI
129 choices of the programmer. </li>
131 <li> As a scientist, you know which analyses you want to
134 <li> In this situation, hand-holding GUIs hurt you. </li>
136 <li> GUIs make you dependent on their style of hand-holding. </li>
138 <li> GUIs change from version to version. </li>
140 <li> GUI-based software is often commercial, so your data can
141 be trapped if new versions cannot read the data files of old
144 <li> GUIs make you less expert in your field </li>
146 <li> You will get bad habits </li>
150 <li> GUIs make it easy to make bad choices (analysis,
151 plotting. etc). </li>
153 <li> GUIs make it easy to add unstructured data to your dataset
154 (e.g. spreadsheets with formatting as data). </li>
157 <li> Your colleagues do not use the GUI </li>
160 <li> Even if you do not program yourself, you will generate data that
161 is easy for programmers to work on. </li>
165 SUBSECTION(«Advantages of CLIs»)
168 <li> You can do more without a GUI </li>
171 <li> As you work more with the CLI, you will see problems in a different
172 way. You will realize that you can do new things. </li>
174 <li> Automation: </li>
178 <li> Reduces your manual labor. </li>
180 <li> Produces more consistent results. </li>
182 <li> Allows you to work on bigger datasets. </li>
187 <li> You are faster without GUI </li>
190 <li> You will perform similar analyses over and over again. Using
191 GUI-based software requires many clicks. Each click is an opportunity
194 <li> If you have to perform an analysis or make a conversion again,
195 you don't have to figure it out again if you have a little script. </li>
197 <li> As you continue to work with the CLI, you will see problems
198 differently, and realize there are many ways. </li>
202 <li> You will make fewer errors </li>
206 <li> GUIs are easy, but there is no record of your clicks. It can be
207 difficult to detect a mis-click, so sometimes it is impossible
208 to tell what you did (for example, which options you selected for
211 <li> GUI-based software such as Excel can provide comfort, because
212 you "see" the data, but it hides the calculations, connections
213 between cells, and even which cells are entered data and which are
214 calculated. This has led to huge, world changing mistakes. </li>
218 <li> You need the CLI </li>
222 <li> Data is too big for GUIs. </li>
224 <li> Almost all cutting-edge tools are CLI. </li>
226 <li> You will probably need to convert formats to use new tools. </li>
228 <li> You will need to do this over dozens or hundreds or thousands
229 of files to use the servers and the cluster. </li>
233 <li> Using the CLI makes you a better scientist </li>
236 <li> CLI brings you closer to your data. </li>
238 <li> CLI brings you closer to your analysis. </li>
240 <li> CLI lets you use the most current tools. </li>
242 <li> CLI makes your analyses more repeatable. </li>
244 <li> CLI give you more control and accountability in your analyses. </li>
246 <li> CLI makes you more expert in your field. </li>
248 <li> Do not fear the CLI. </li>
250 <li> Learning the CLI takes work and time, but you will only have less
251 time as you progress in you career. </li>