3 Unix is user-friendly. It's just very selective about who
4 its friends are. -- Unknown
8 SECTION(«History and Philosophy»)
10 SUBSECTION(«Early Unix History»)
12 <p> Unix was created in 1969 as a successor of Multics, the
13 <em>MULTiplexed Information and Computing Service</em>, which had been
14 in use since the mid 1960s as the successor of CTSS, the <em>Compatible
15 Time-Sharing System</em> of the early 1960s. Multics aimed to get
16 CTSS right, but failed in this regard and was eventually discontinued
17 because of its complexity. The Unix approach was very different as it
18 was brainstormed by only three people and then implemented by Ken
19 Thompson at Bell Laboratories in two days. Unlike its predecessors it
20 focused on elegance and simplicity. The name was originally spelt
21 UNICS (<em>UNiplexed Information and Computing Service</em>) to
22 emphasize the contrast to Multics. </p>
24 <p> The original Unix implementation was written in assembly
25 language for the 18 bit processor of the PDP-7 "minicomputer", a
26 device of the size of a wardrobe which was considered small by the
27 standards of the day. Like all computers of this era, the PDP-7 was
28 not connected to a video screen. Instead, input had to be typed in on
29 the <em>console</em>, a device which looked much like an electric
30 typewriter. Output from the computer was printed on rolls of paper.
31 Since the assembly instructions could not easily be ported to different
32 hardware, Dennis Ritchie invented the C programming language in
33 1971. By 1973 the complete Unix implementation had been rewritten in C.
34 The C language was another corner stone in the history of Unix which
35 turned out to be very successful. While other programming languages
36 of that time have long been abandoned or play merely a niche role, C
37 is still one of the most widely used programming languages today. The
38 first Unix application was <code>roff</code>, a typesetting program
39 which is still ubiquitous as the manual pages which ship with every
40 Unix system are formatted with roff. </p>
42 <p> From the beginning, Thompson and his early collaborators encouraged
43 close communication between programmers, creating an early form of
44 community. Up to the present day, this "hacker culture" has been a
45 stimulus for countless improvements. Copies of Unix were distributed
46 on tapes, hand-signed with "Love, Ken". Over time many universities
47 contributed to Unix. By the end of the 1970s, Unix accumulated a
48 whole bunch of utilities that made it a fully flavored operating
49 system which was also free of any copyright claims. </p>
51 <p> Despite the primitive hardware of the time, the early Unix was
52 remarkably similar to modern Linux systems. For example, the task
53 scheduler, the hierarchical filesystem tree and the shell already
54 existed back then. </p>
56 SUBSECTION(«Networking»)
58 <p> The <em>Advanced Research Projects Agency</em> (ARPA) was a
59 military research unit that was part of the USA's department of
60 defence. It was established in the early 1960s with the mandate to
61 create systems that could survive a nuclear war. The agency created
62 the <em>arpanet</em>, the predecessor of today's internet, which was
63 designed to stay operational after subordinate network losses. By
64 the end of the 1960s and the early 1970s, the fundamental networking
65 protocols were established: telnet for remote login was standardized
66 in 1969, email (SMTP) in 1971, and the file transfer protocol (FTP)
69 <p> By the end of the 1970s many Unix installations existed in
70 all parts of the world. However, the arpanet was mostly powered by
71 commercial Multics systems because Unix only had rudimentary network
72 support (UUCP, the <em>Unix to Unix copy</em>) which could copy files
73 over telephone lines via modems but not much more. This changed in
74 1983 when TCP/IP networking was developed by the ARPA to replace the
75 arpanet. Unix support for TCP/IP was developed at Berkeley University
76 which had become the "Mecca" for Unix development and started already
77 in 1977 to release their own Unix system named BSD, the <em>Berkeley
78 Software Distribution</em>. </p>
80 SUBSECTION(«Commercialization, POSIX and GNU»)
82 <p> With excellent networking support and no licensing issues, it
83 was only a matter of time until companies became interested in Unix
84 in order to make money. Several companies started to commercialize
85 Unix by adding features to the common code base but keeping their
86 improvements closed, effectively stopping the source code from being
87 freely distributable. At the same time Microsoft began to sell their
88 DOS operating system, targeting small businesses and the home market.
89 DOS lacked many features that Unix already had for a decade, like
90 multi-tasking and multi-user support, but it did run on the cheap
91 Intel 286 processors that were too weak for Unix. </p>
93 <p> By 1985 the commercialization of Unix and the success of Microsoft
94 had damaged the Unix community badly. But also the various companies
95 that sold their particular proprietary Unix brand realized that
96 too many incompatible Unix implementations would only hurt their
97 business. That's where the Computer Society of the <em>Institute of
98 Electrical and Electronics Engineers</em> (IEEE) became involved
99 in Unix. The IEEE is an organization which was already founded in
100 1946 to advance the theory, practice, and application of computer
101 technology. This organization created POSIX, the <em>Portable Operating
102 System Interface for Unix</em>, which is a family of specifications
103 for maintaining compatibility between operating systems. The first
104 version of POSIX was published in 1988. It covered several command
105 line utilities including <code>vi(1)</code> and <code>awk(1)</code>,
106 the shell scripting language, application programmer interfaces (APIs)
107 for I/O (input/output) and networking, and more. Up to the present
108 day POSIX is maintained by the IEEE and new revisions of the POSIX
109 standard are published regularly. </p>
111 <p> In 1983 Richard Stallman launched the GNU project and the Free
112 Software Foundation as a reaction to the ongoing commercialization
113 of Unix. GNU, which is is a recursive acronym for "GNU's not Unix",
114 aimed to keep the Unix source code free, or to replace non-free parts
115 by open source equivalents. To this aim the GNU project created
116 the <em>GNU General Public License</em> (GPL), which requires not
117 only the source code to stay free, but also that all subsequent
118 modifications to the code base remain free. By the end of the 80s,
119 the GNU toolset had become a full developer software stack licensed
120 under the GPL. This set of software packages was complemented by
121 the <em>X window system</em>, which was also released under a free
122 license and enabled programmers to build graphical applications for
123 desktop systems. Moreover, the first open source scripting language,
124 <em>perl</em>, was released in 1987. </p>
128 <p> In 1985 Intel announced the 386 processor which, unlike its 286
129 predecessor, was powerful enough to run Unix. There were efforts to
130 port the Unix operating system kernel to this hardware, but these
131 efforts were impaired by pending lawsuits about who owns the copyright
132 on the BSD source code. Due to the unclear legal situation of the BSD
133 code, the major missing piece in the GNU toolset was a free operating
134 system kernel. This hole was filled in 1991 when Linus Torvalds,
135 a student from Helsinki in Finland, announced the first version of
136 his <em>Linux</em> kernel. </p>
138 <p> Linux did not repeat the licensing problems of the original Unix
139 because the Linux source code was written from scratch and licensed
140 under the GPL. Due to this difference many developers moved from
141 Unix to Linux, so Linux grew quickly and started soon to outperform
142 the commercial Unix kernels in almost every benchmark. The cheap 386
143 hardware, the Linux kernel, the GNU toolset and the graphical user
144 interface based on the X window system facilitated cheap workstations
145 which ran a complete open source software stack. </p>
147 <p> The success of Linux, or <em>GNU/Linux</em> as some prefer to
148 call it for reasons that should now be clear, steadily increased over
149 time. In 2003 the SCO group, a company which sold a proprietary Unix
150 system, was unhappy about this progress and sued IBM, which offered
151 various Linux products. SCO claimed to be the owner of Unix, and that
152 Linux contained "millions of lines" of code copied from Unix. SCO's
153 lawyers argued that the success of Linux originated from this theft
154 of intellectual property and asked for $5 billion as compensation
155 for the resulting losses. The company also tried to collect taxes
156 from other Linux users. Microsoft funded SCO in these efforts. </p>
158 <p> In the end SCO lost the lawsuit since it was evident that all that
159 copied code never existed. In fact, the court ruled that SCO did not
160 even own the Unix copyrights to begin with. Another fun fact is that
161 the large number of bugs in the early Linux code actually helped to
162 prove that Linux was original work. The long term effects of this
163 lawsuit, an improved position of Linux and its ecosystem, last until
164 the presence. Commercial Unix systems have become irrelevant as Linux
165 runs on a wide variety of machines ranging from supercomputers to
166 workstations, smart phones and IOT (internet of things) devices with
167 very limited resources. </p>
169 <p> While SCO went bankrupt eventually, some of the companies which
170 almost killed Unix by maximizing their own profit still exist, and
171 make money with Linux today. However, they had to adjust their
172 business model in order to comply with the GPL. Rather than selling
173 proprietary software, they bundle open source software and sell
174 support to paying customers. Some companies also sell hardware with
175 Linux pre-installed. </p>
177 SUBSECTION(«Linux Distributions»)
179 <p> A <em>Linux Distribution</em> is a conglomeration of free software,
180 including the Linux kernel, the GNU toolset and the X window system,
181 plus possibly other, proprietary software on top of that. Usually a
182 distribution also includes an installer and a package manager to
183 make it easy to install and update packages according to the users'
186 <p> There are hundreds of Linux distributions, and new distributions
187 are created all the time while others are discontinued. Many
188 distributions are backed by companies which target specific
189 classes of users or hardware, but there are also non-commercial
190 Linux distributions which are solely driven by a community of
193 <p> One of the most popular company-backed Linux distributions is
194 <em>Ubuntu</em>, which is led since 2004 by the UK-based Canonical Ltd.
195 It targets unskilled desktop users which would like to switch away
196 from Microsoft Windows. One reason for the popularity of Ubuntu
197 is that it is very easy to install on standard desktop and laptop
198 hardware. A distinguishing feature of Ubuntu is its strict release
199 cycles: New versions are released in April and October of each year,
200 and every fourth release is a <em>long-term support</em> (LTS) release
201 which will be supported for at least five years. Ubuntu also features
202 a variant for server hardware which contains a different Linux kernel
203 and ships with most desktop packages excluded. </p>
205 <p> The main community-driven Linux distribution is
206 <em>Debian</em>. The Debian project was founded in 1993 and the first
207 stable version was released in 1996. Debian is used as the basis for
208 many other distributions. In fact, Ubuntu is based on Debian. The
209 development of Debian closely follows the Unix culture in that it
210 is developed openly and distributed freely. A team of about 1000
211 core developers work together with countless package maintainers
212 according to the Debian Social Contract, the Debian Constitution,
213 and the Debian Free Software Guidelines. </p>
218 <li> Run <code>uname -a</code> on various Unix machines to see the
219 OS type and the kernel version. </li>
222 href="https://upload.wikimedia.org/wikipedia/commons/7/77/Unix_history-simple.svg">Unix
226 href="http://www.gnu.org/cgi-bin/license-quiz.cgi">Free Software
227 licensing quiz</a>. </li>
229 <li> On a Debian or Ubuntu system, run <code>aptitude search
230 python</code> to list all python-related Ubuntu packages. Run
231 <code>aptitude show python-biopython</code> to see the description
232 of the biopython package. Repeat with different search patterns and
235 <li> The Debian Social Contract (DSC) describes the agenda of Debian.
236 Find the DSC online, read it and form your own opinion about the key
237 points stated in this document. </li>
240 SECTION(«Characteristics of a Unix system»)
242 <p> After having briefly reviewed the history of Unix, we now look
243 closer at the various components which comprise a Unix system and
244 which distinguish Unix from other operating systems. We focus on
245 general design patterns that have existed since the early Unix days
246 and are still present on recent Linux systems. </p>
248 SUBSECTION(«Single Hierarchy of Files»)
250 <p> The most striking difference between Unix and Windows is perhaps
251 that on Unix the files of all devices are combined to form a single
252 hierarchy with no concept of drive letters. When the system boots,
253 there is only one device, the <em>root device</em>, which contains the
254 <em>root directory</em>. To make the files of other devices visible,
255 the <em>mount</em> operation must be employed. This operation attaches
256 the file hierarchy of the given device to the existing hierarchy
257 at a given location which is then called the <em>mountpoint</em>
258 of the device. Mountpoints are thus the locations in the hierarchy
259 where the underlying storage device changes. </p>
261 <p> The root directory contains a couple of well-known subdirectories,
262 each of which is supposed to contain files of a certain type or for
263 a certain purpose. The following table lists a subset:
266 <li> <code>/bin</code>: Essential commands for all users </li>
267 <li> <code>/sbin</code>: Essential system binaries </li>
268 <li> <code>/lib</code>: Essential libraries </li>
269 <li> <code>/usr</code>: Non-essential read-only user data </li>
270 <li> <code>/etc</code>: Static configuration files </li>
271 <li> <code>/home</code>: Home directories </li>
272 <li> <code>/tmp</code>: Temporary files </li>
273 <li> <code>/run</code>: Files which describe the state of running programs </li>
274 <li> <code>/var</code>: Log and spool files </li>
277 <p> The <em>Filesystem Hierarchy Standard</em> describes the various
278 subdirectories in more detail. The exercises ask the reader to become
279 acquainted with this directory structure. </p>
281 SUBSECTION(«POSIX Commands and Shell»)
283 <p> The Filesystem Hierarchy Standard lists <code>/bin</code>
284 and <code>/sbin</code> and several other directories for executable
285 files. The POSIX standard defines which executables must exist in one
286 of these directories for the system to be POSIX-compliant. Well over
287 100 <em>POSIX commands</em> are listed in the XCU volume of this
288 standard. Besides the names of the commands, the general behaviour
289 of each and the set of command line options and their semantics are
290 described. POSIX versions are designed with backwards compatibility
291 in mind. For example, a new POSIX version might require a command
292 to support additional command line options, but existing options are
293 never dropped and never change semantics in incompatible ways. The
294 target audience of the POSIX document are programmers who implement
295 and maintain the POSIX commands and users which want to keep their
296 software portable across different Unix flavors. </p>
298 <p> One of the POSIX commands is the <em>shell</em>,
299 <code>/bin/sh</code>, an interpreter that reads input expressed in
300 the <em>shell command language</em>, which is also part of POSIX.
301 The shell transforms the input in various ways to produce commands
302 and then executes these commands. The user may enter shell code
303 (i.e., code written in the shell command language) interactively at
304 the <em>command prompt</em>, or supply the input for the shell as a
305 <em>shell script</em>, a text file which contains shell code. Shell
306 scripts which only contain POSIX commands and use only POSIX options
307 are portable between different shell implementations and between
308 different Unix flavors. They should therefore never cease to work after
309 an upgrade. Among the many available POSIX shell implementations,
310 <em>GNU bash</em> is one of the more popular choices. Bash is fully
311 POSIX compatible and offers many more features on top of what is
312 required by POSIX. </p>
314 <p> Several implementations of the POSIX commands exist. On Linux
315 the GNU implementation is typically installed while FreeBSD, NetBSD
316 and MacOS contain the BSD versions. Although all implementations
317 are POSIX-compliant, they differ considerably because different
318 implementations support different sets of additional features and
319 options which are not required by POSIX. These extensions are not
320 portable, and should thus be avoided in shell scripts that must work
321 on different Unix flavors. </p>
323 <p> In addition to the POSIX commands, a typical Unix system might well
324 contain thousands of other commands. This illustrates another aspect
325 that is characteristic for Unix: Tools should do only one specific
326 task, and do it well. The operating system provides mechanisms
327 to combine the simple commands in order to form more powerful
328 programs. For example, commands can be <em>chained</em> together so
329 that the output of one command becomes the input for the next command
330 in the chain. This is the idea behind <em>pipes</em>, a Unix concept
331 which dates back to 1973 and which is also covered by POSIX. We shall
332 come back to pipes and related concepts in a later section. </p>
334 SUBSECTION(«Multi-User, Multi-Tasking, Isolation»)
336 <p> From the very beginning Unix was designed to be a multi-user
337 and a multi-tasking operating system. That is, it could run multiple
338 programs on behalf of different users independently of each other and
339 isolated from each other. This design was chosen to improve hardware
340 utilization and robustness. In contrast, DOS and early versions
341 of Windows were designed for <em>personal computing</em> (PC) and
342 had no notion of user accounts, access permissions or isolation.
343 This resulted in an unstable system because a single misbehaving
344 program was enough to take down the whole system. Therefore these
345 features had to be retrofitted later. </p>
347 <p> While multi-tasking makes all tasks appear to run simultaneously
348 even if there are more tasks than CPUs, isolation refers to
349 <em>memory protection</em>, a mechanism which prevents applications
350 from interfering with each other and with the internals of the
351 operating system. A running Unix system maintains two sets of
352 running tasks: besides the <em>application tasks</em> there is
353 also a set of <em>kernel tasks</em>. Unlike the application tasks,
354 the kernel tasks are privileged in that they can access the memory
355 of the application tasks while applications tasks can only access
356 their own memory. Isolation is achieved by a hardware concept called
357 <em>protection domains</em>, which existed already in Multics and thus
358 predates Unix. In the simplest case, there are only two protection
359 domains: a privileged domain called <em>ring 0</em> for the kernel
360 tasks, and an unprivileged domain for application tasks, also called
361 <em>user processes</em> in this context. The CPU is always aware
362 of the current protection domain as this information is stored in a
363 special CPU register.
365 SUBSECTION(«System Calls and the C POSIX Library»)
367 <p> Only when the CPU is running in the privileged ring 0 domain
368 (also known as <em>kernel mode</em>, as opposed to <em>user mode</em>
369 for application tasks), it can interact directly with hardware and
370 memory. If an application wants to access hardware, for example read a
371 data block from a storage device, it can not do so by itself. Instead,
372 it has to ask the operating system to perform the read operation
373 on behalf of the application. This is done by issuing a <em>system
374 call</em>. Like function calls, system calls interrupt the current
375 program, continue execution at a different address and eventually
376 return to the instruction right after the call. However, in addition
377 to this, they also cause the CPU to enter kernel mode so that it can
378 perform the privileged operation. When the system call has done its
379 work and is about to return to the application, the protection domain
380 is changed again to let the CPU re-enter user mode. </p>
382 <p> The system calls thus define the interface between applications
383 and the operating system. For backwards compatibility it is of utmost
384 importance that system calls never change semantics in an incompatible
385 way. Moreover, system calls must never be removed because this would
386 again break existing applications. The syntax and the semantics
387 of many system calls are specified in POSIX, although POSIX does
388 not distinguish between functions and system calls and refers to
389 both as <em>system functions</em>. This is because system calls are
390 typically not performed by the application directly. Instead, if an
391 application calls, for example, <code>read()</code>, it actually calls
392 the compatibility wrapper for the <code>read</code> system call which
393 is implemented as a function in the <em>C POSIX Library</em> (libc),
394 which ships with every Unix system. It is this library which does the
395 hard work, like figuring out which system calls are supported on the
396 currently running kernel, and how kernel mode must be entered on this
397 CPU type. Like the POSIX commands, the system functions described in
398 POSIX never change in incompatible ways, so programs which exclusively
399 use POSIX system functions are portable between different Unix flavors
400 and stay operational after an upgrade. </p>
402 SUBSECTION(«Multi-Layer Configuration Through Text Files»)
404 <p> On a multi-user system it becomes necessary to configure programs
405 according to each user's personal preferences. The Unix way to
406 achieve this is to provide four levels of configuration options
407 for each program. First, there are the built-in defaults which are
408 provided by the author of the program. Next, there is the system-wide
409 configuration that is controlled by the administrator. Third,
410 there is the user-defined configuration, and finally there are the
411 command line options. Each time the program is executed, the four
412 sets of configuration options are applied one after another so
413 that the later sets of options override the earlier settings. The
414 system-wide configuration is stored in <code>/etc</code> while the
415 user-defined configuration is stored in that user's home directory.
416 Both are are simple text files that can be examined and modified with
417 any text editor. This makes it easy to compare two configurations
418 and to transfer the configuration across different machines or user
421 SUBSECTION(«Everything is a File»)
423 <p> Another mantra which is often heard in connection with Unix is
424 <em>everything is a file</em>. This phrase, while certainly catchy,
425 is slightly incorrect. A more precise version would be <em>everything
426 is controlled by a file descriptor</em>, or, as Ritchie and Thompson
427 stated it, Unix has <em>compatible file, device, and inter-process
428 I/O</em>. Modern Unix systems have pushed this idea further and employ
429 file descriptors also for networking, process management, system
430 configuration, and for certain types of events. The file descriptor
431 concept is thus an abstraction which hides the differences between the
432 objects the file descriptors refer to. It provides a uniform interface
433 for the application programmer who does not need to care about whether
434 a file descriptor refers to a file, a network connection, a peripheral
435 device or something else because the basic I/O operations like open,
436 read, write are the same. </p>
438 <p> File descriptors are ubiquitous since every Unix program uses
439 them, albeit perhaps implicitly via higher-level interfaces provided
440 by a scripting language. We shall return to this topic when we discuss
443 SUBSECTION(«Manual Pages»)
445 <p> All POSIX commands and most other programs are installed along
446 with one or more <em>man pages</em> (short for <em>manual pages</em>),
447 which are plain text files that can be formatted and displayed in
448 various ways. This concept was introduced in 1971 as part of the
449 <em>Unix Programmer's Manual</em>. The characteristic page layout
450 and the typical sections (NAME, SYNOPSIS, DESCRIPTION, EXAMPLES,
451 SEE ALSO) of a man page have not changed since then. The POSIX
452 <code>man</code> command is used to view man pages in a terminal. For
453 example, the command <code>man ls</code> opens the man page of the
454 <code>ls</code> command, and <code>man man</code> shows the man page
455 of the <code>man</code> command itself. Most implementations also
456 maintain a database of the existing man pages and provide additional
457 commands to query this database. For example, the <code>whatis</code>
458 command prints the one-line description of all man pages which match
459 a pattern while the <code>apropos</code> command searches the manual
460 page names and descriptions. </p>
462 <p> In addition to the man pages for commands, there are man pages for
463 system calls, library functions, configuration files and more. Each
464 man page belongs to one of several <em>man sections</em>. For example,
465 the aforementioned man pages for <code>ls</code> and <code>man</code>
466 are part of section 1 (user commands) while section 2 is reserved for
467 system calls and section 8 for administration commands that can only be
468 executed by privileged users. By convention, to indicate which section
469 a command or a function belongs to, the man section is appended in
470 parenthesis as in <code>mount(8)</code>. Most Unix systems also offer
471 translated man pages for many languages as an optional package. Note
472 that the same name may refer to more than one man page. For example
473 there is <code>kill(1)</code> for the user command that kills processes
474 and also <code>kill(2)</code> which describes the corresponding system
475 call. To open the man page of a specific section, one may use a command
476 like <code>man 2 kill</code>. The <code>MANSECT</code> environment
477 variable can be set to a colon-delimited list of man sections to
478 change the order in which the man sections are searched. </p>
480 <p> Consulting the local man pages rather than searching the web has
481 some advantages. Most importantly, the local pages will always give
482 correct answers since they always match the installed software while
483 there is no such relationship between a particular web documentation
484 page and the version of the software package that is installed on the
485 local computer. Working with man pages is also faster, works offline
486 and helps the user to stay focused on the topic at hand. </p>
491 <li> Run <code>df</code> on as many systems as possible to see the
492 mount points of each filesystem. Then discuss the pros and cons of
493 a single file hierarchy as opposed to one hierarchy per device. </li>
495 <li> Run <code>ls /</code> to list all top-level subdirectories of
496 the root file system and discuss the purpose of each. Consult the
497 Filesystem Hierarchy Standard if in doubt. </li>
499 <li> Execute <code>cd / && mc</code> and start surfing at the root
502 <li> Compare the list of top-level directories that exist on different
503 Unix systems, for example Linux and MacOS. </li>
505 <li> Find out which type of files are supposed to be stored in
506 <code>/usr/local/bin</code>. Run <code>ls /usr/local/bin</code>
507 to list this directory. </li>
509 <li> Find out what the term <em>bashism</em> means and learn how to
510 avoid bashishms. </li>
512 <li> Find the POSIX specification of the <code>cp(1)</code> command
513 online and compare the set of options with the options supported by
514 the GNU version of that command, as obtained with <code>man cp</code>
515 on a Linux system. </li>
519 <li> Run <code>time ls /</code> and discuss the meaning of
520 the three time values shown at the end of the output (see
521 <code>bash(1)</code>). </li>
523 <li> Guess the user/real and the sys/real ratios for the following
524 commands. Answer, before you run the commands.
527 <li> <code>time head -c 100000000 /dev/urandom > /dev/null</code> </li>
529 <li> <code>i=0; time while ((i++ < 1000000)); do :; done</code>
534 <li> Run the above two commands again, this time run
535 <code>htop(1)</code> in parallel on another terminal and observe the
540 <li> On a Linux system, check the list of all system calls in
541 <code>syscalls(8)</code>. </li>
543 <li> The <code>strace(1)</code> command prints the system calls that
544 the given command performs. Guess how many system calls the command
545 <code>ls -l</code> will make. Run <code>strace -c ls -l</code> for
546 the answer. Read the <code>strace(1)</code> man page to find suitable
547 command line options to only see the system calls which try to open
550 <li> Guess how many man pages a given system has. Run <code>whatis -w
551 '*' | wc -l</code> to see how close your guess was. </li>
553 <li> Search the web for "cp(1) manual page" and count how many
554 <em>different</em> manual pages are shown in the first 20 hits. </li>
559 Think about printers, sound cards, or displays as a file. Specifically,
560 describe what <code>open</code>, <code>read</code>, and <code>write</code>
561 should mean for these devices.
565 Opening would establish a (probably exclusive) connection
566 to the device. Reading from the file descriptor returned by
567 <code>open(2)</code> could return all kinds of status information,
568 like the type, model and capabilities of the device. For example,
569 printers could return the number of paper trays, the amount of toner
570 left etc. Writing to the file descriptor would cause output on the
571 device. This would mean to print the text that is written, play the
572 audio samples, or show the given text on the display. The point to
573 take away is that the <code>open</code>, <code>read</code>,
574 <code>write</code> interface is a generic concept that works for
575 different kinds of devices, not only for storing data in a file on a
580 SECTION(«Paths, Files and Directories»)
582 In this section we look in some detail at paths, at a matching
583 language for paths, and at the connection between paths and files. We
584 then describe the seven Unix file types and how file metadata are
585 stored. We conclude with the characteristics of soft and hard links.
589 <p> The path concept was introduced in the 1960s with the Multics
590 operating system. Paths will be familiar to the reader because
591 they are often specified as arguments to commands. Also many
592 system calls receive a path argument. A path is a non-empty
593 string of <em>path components</em> which are separated by slash
594 characters. An <em>absolute path</em> is a path that starts with a
595 slash, all other paths are called <em>relative</em>. A relative path
596 has to be interpreted within a context that implies the leading
597 part of the path. For example, if the implied leading part is
598 <code>/foo/bar</code>, the relative path <code>baz/qux</code> is
599 equivalent to the absolute path <code>/foo/bar/baz/qux</code>. </p>
601 <p> Given a path, there may or may not exist a file or a
602 directory that corresponds to the path. <em>Path lookup</em> is the
603 operation which determines the answer to this question, taking the
604 implied leading part into account in case of relative paths. This
605 operation is always performed within the kernel and turns out to
606 be surprisingly complex due to concurrency and performance issues.
607 Consult <code>path_resolution(7)</code> on a Linux system to learn
608 more about how pathnames are resolved to files. </p>
610 <p> If a path was successfully looked up, each path component up to the
611 second-last refers to an existing directory while the last component
612 refers to either a file or a directory. In both cases the directory
613 identified by the second-last component contains an entry named by the
614 last component. We call those paths <em>valid</em>. The valid paths
615 give rise to a rooted tree whose interior nodes are directories and
616 whose leaf nodes are files or directories. Note that the validity of a
617 path depends on the set of existing files, not just on the path itself,
618 and that a valid path may become invalid at any time, for example if
619 a file is deleted or renamed. Many system calls which receive a path
620 argument perform path lookup and fail with the <code>No such file or
621 directory</code> error if the lookup operation fails. </p>
623 <p> It depends on the underlying filesystem whether the path components
624 are <em>case-sensitive</em> or <em>case-insensitive</em>. That is,
625 whether paths which differ only in capitalization (for example
626 <code>foo</code> and <code>Foo</code>) refer to the same file.
627 Since the hierarchy of files may be comprised of several filesystems,
628 some components of the path may be case-sensitive while others are
629 case-insensitive. As a rule of thumb, Unix filesystems are case
630 sensitive while Microsoft filesystems are case-insensitive even when
631 mounted on a Unix system. </p>
633 <p> Path components may contain every character except the Null
634 character and the slash. In particular, space and newline characters
635 are allowed. However, while dots are allowed in path components if
636 they are used together with other characters, the path components
637 <code>.</code> and <code>..</code> have a special meaning: every
638 directory contains two subdirectories named <code>.</code> and
639 <code>..</code> which refer to the directory itself and its parent
640 directory, respectively. </p>
642 SUBSECTION(«Globbing»)
644 <p> Globbing, also known as <em>pathname expansion</em>, is a pattern
645 matching language for paths which was already present in the earliest
646 Unix versions. The glob operation generates a set of valid paths from
647 a <em>glob pattern</em> by replacing the pattern by all <em>matching
650 <p> Glob patterns may contain special characters called
651 <em>wildcards</em>. The wildcard characters are: </p>
654 <li> <code>*</code>: match any string, </li>
655 <li> <code>?</code>: match any simple character, </li>
656 <li> <code>[...]</code>: match any of the enclosed characters. </li>
659 <p> The complete syntax rules for glob patterns and the exact
660 semantics for pattern matching are described in POSIX and in
661 <code>glob(7)</code>. Any POSIX-compliant shell performs globbing
662 to construct the command to be executed from the line entered at
663 the prompt. However, POSIX also demands system functions which make
664 globbing available to other applications. These are implemented as
668 <p> There are a few quirks related to globbing which are worth to
669 point out. First, if no valid path matches the given pattern, the
670 expansion of the pattern is, by definition according to POSIX, the
671 pattern itself. This can lead to unexpected results. Second, files
672 which start with a dot (so-called <em>hidden</em> files) must be
673 matched explicitly. For example, <code>rm *</code> does <em>not</em>
674 remove these files. Third, the tilde character is <em>no</em> wildcard,
675 although it is also expanded by the shell. See the exercises for more
678 <p> POSIX globbing has some limitations. For example, there is no
679 glob pattern which matches exactly those files that start with an
680 arbitrary number of <code>a</code> characters. To overcome these
681 limitations, some shells extend the matching language by implementing
682 <em>extended glob patterns</em> which are not covered by POSIX. For
683 example, if extended globbing feature of <code>bash(1)</code> is
684 activated via the <code>extglob</code> option, the extended glob
685 pattern <code>+(a)</code> matches the above set of files. </p>
687 SUBSECTION(«File Types»)
689 We have seen that all but the last component of a valid path refer
690 to directories while the last component may refer to either a file
691 or a directory. The first character in the output of <code>ls
692 -l</code> indicates the type of the last path component: for
693 directories a <code>d</code> character is shown while files (also
694 called <em>regular</em> files in this context) get a hyphen character
695 (<code>-</code>). Besides directories and regular files, the following
696 special file types exist:
699 <dt> Soft link (<code>l</code>) </dt>
701 <dd> A file which acts as a pointer to another file. We shall cover
702 links in a dedicated subsection below. </dd>
704 <dt> Device node (<code>c</code> and <code>b</code>) </dt>
706 <dd> Also called <em>device special</em>. These files refer to devices
707 on the local system. Device nodes come in two flavors: character
708 devices (<code>c</code>) and block devices (<code>b</code>). Regardless
709 of the flavor, each device node has a major and a minor number
710 associated with it. The major number indicates the type of the
711 device (e.g. a hard drive, a serial connector, etc.) while the
712 minor number enumerates devices of the same type. On most systems
713 the device nodes are created and deleted on the fly as the set of
714 connected devices changes, for example due to a USB device being
715 added or removed. However, device nodes can also be created manually
716 with the <code>mknod(1)</code> command or the <code>mknod(2)</code>
717 system call. Device nodes do not necessarily correspond to physical
718 devices. In fact, POSIX demands the existence of a couple of
719 <em>virtual devices</em> with certain properties. We look at some of
720 these in the exercises. The access to device nodes which do correspond
721 to physical devices is usually restricted to privileged users. </dd>
723 <dt> Socket (<code>s</code>) </dt>
725 <dd> Sockets provide an interface between a running program and the
726 network stack of the kernel. They are subdivided into <em>address
727 families</em> which correspond to the various network protocols. For
728 example, the <code>AF_INET</code> and <code>AF_INET6</code> address
729 families are for internet protocols (IP) while <code>AF_LOCAL</code>
730 (also known as <code>AF_UNIX</code>) is used for communication between
731 processes on the same machine. These local sockets are also called
732 <em>Unix domain sockets</em>. They can be bound to a path which
733 refers to a file of type socket. Regardless of the address family,
734 processes can exchange data via sockets in both directions, but
735 the local sockets support additional features, like passing process
736 credentials to other processes. </dd>
738 <dt> Fifo (<code>p</code>) </dt>
740 <dd> Files of type <em>fifo</em> are also known as <em>named
741 pipes</em>. They associate a path with a kernel object that provides a
742 <em>First In, First Out</em> data channel for user space programs. Data
743 written to the fifo by one program can be read back by another program
744 in the same order. Fifos are created with the <code>mkfifo(1)</code>
745 command or the <code>mkfifo(3)</code> library function. </dd>
748 <p> Note that the type of a file is never inferred from the path.
749 In particular the suffix of the path (everything after the last
750 dot) is just a convention and has no strict connection to the file
751 type. Also there is no difference between text and binary files. </p>
753 SUBSECTION(«Metadata and Inodes»)
755 <p> The <code>stat(2)</code> system call returns the metadata
756 of the file or directory that corresponds to the given path. The
757 <code>stat(1)</code> command is a simple program which executes this
758 system call and prints the thusly obtained metadata in human-readable
759 form, including the file type. This is done without looking at the
760 contents of the file because metadata are stored in a special area
761 called the <em>inode</em>. All types of files (including directories)
762 have an associated inode. Besides the file type, the inode stores
763 several other properties prescribed by POSIX. For example the file
764 size, the owner and group IDs, and the access permissions are all
765 stored in the inode. Moreover, POSIX requires to maintain three
766 timestamps in each inode: </p>
769 <li> modification time (mtime): time of last content change. </li>
771 <li> access time (atime): time of last access. </li>
773 <li> status change time (ctime): time of last modification to the
777 <p> To illustrate the difference between the mtime and the ctime,
778 consider the <code>chgrp(1)</code> command which changes the group
779 ID of the file or directory identified by its path argument. This
780 command sets the ctime to the current time while the mtime is left
781 unmodified. On the other hand, commands which modify the contents of
782 a file, such as <code>echo foo >> bar</code>, change both the mtime
785 <p> The inode of each file or directory contains twelve <em>mode
786 bits</em>, nine of which are the <em>permission bits</em> which
787 control who is allowed to access the file or directory, and how. The
788 permission bits are broken up into three classes called <em>user</em>
789 (<code>u</code>), <em>group</em> (<code>g</code>) and <em>others</em>
790 (<code>o</code>). Some texts refer to the first and last class as
791 "owner" and "world" instead, but we won't use this naming to avoid
792 confusion. Each class contains three bits. The bits of the "user"
793 class apply to the file owner, that is, the user whose ID is stored in
794 the inode. The "group" category applies to all non-owners who belong
795 to the group whose ID is stored in the inode. The third category
796 applies to all remaining users. The three bits of each class refer to
797 read/write/execute permission. They are therefore named <code>r</code>,
798 <code>w</code> and <code>x</code>, respectively. The permission
799 bits mean different things for directories and non-directories,
800 as described below. </p>
804 <th> </th>
805 <th> Directories </th>
806 <th> Non-directories </th>
808 <th> <code>r</code> </th>
810 <td> The permission to list the directory contents. More precisely,
811 this bit grants the permission to call <code>opendir(3)</code>
812 to obtain a handle to the directory which can then be passed to
813 <code>readdir(3)</code> to obtain the directory contents. </td>
815 <td> If read permission is granted, the <code>open(2)</code> system
816 call does not fail with the <code>permission denied</code> error,
817 provided the file is opened in read-only mode. The system call may
818 fail for other reasons, though.
821 <th> <code>w</code> </th>
823 <td> The permission to add or remove directory entries. That is,
824 to create new files or to remove existing files. Note that write
825 permission is not required for the file that is being removed. </td>
827 <td> Permission to open the file in write-only mode in order
828 to perform subsequent operations like <code>write(2)</code>
829 and <code>truncate(2)</code> which change the contents of the
830 file. Non-directories are often opened with the intention to both
831 read and write. Naturally, such opens require both read and write
835 <th> <code>x</code> </th>
837 <td> The permission to <em>search</em> the directory. Searching
838 a directory means to access its entries, either by retrieving
839 inode information with <code>stat(2)</code> or by calling
840 <code>open(2)</code> on a directory entry. </td>
842 <td> Run the file. This applies to <em>binary executables</em> as well
843 as to text files which start with a <em>shebang</em>, <code>#!</code>,
844 followed by the path to an interpreter. We shall cover file execution
845 in more detail below. </td>
850 <p> To run the regular file <code>/foo/bar/baz</code>, search
851 permission is needed for both <code>foo</code> and <code>bar</code>,
852 and execute permission is needed for <code>baz</code>. Similarly, to
853 open the regular file <code>foo/bar</code> for reading, we need execute
854 permissions on the current working directory and on <code>foo</code>,
855 and read permissions on <code>bar</code>. </p>
857 <p> A <em>numeric permission mode</em> is a three octal digit (0-7)
858 number, where the digits correspond to the user, group, other classes
859 described above, in that order. The value of each digit is derived by
860 adding up the bits with values 4 (read), 2 (write), and 1 (execute).
861 The following table lists all eight possibilities for each of the
866 <th> Octal Value </th>
867 <th> Symbolic Representation </th>
871 <td> <code>---</code> </td>
872 <td> no permissions at all </td>
875 <td> <code>--x</code> </td>
876 <td> only execute permission </td>
879 <td> <code>-w-</code> </td>
880 <td> only write permission </td>
883 <td> <code>-wx</code> </td>
884 <td> write and execute permission </td>
887 <td> <code>r--</code> </td>
888 <td> only read permission </td>
891 <td> <code>r-x</code> </td>
892 <td> read and execute permission </td>
895 <td> <code>rw-</code> </td>
896 <td> read and write permission </td>
899 <td> <code>rwx</code> </td>
900 <td> read, write and execute permission </td>
904 <p> The <code>chmod(1)</code> command changes the permission
905 bits of the file identified by the path argument. For example,
906 <code>chmod 600 foo</code> sets the permissions of <code>foo</code> to
907 <code>rw-------</code>. Besides the octal values, <code>chmod(1)</code>
908 supports symbolic notation to address the three classes described
909 above: <code>u</code> selects the user class, <code>g</code> the
910 group class, <code>o</code> the class of other users. The symbolic
911 value <code>a</code> selects all three classes. Moreover, the letters
912 <code>r</code>, <code>w</code> and <code>x</code> are used to set or
913 unset the read, write and execute permission, respectively. The above
914 command is equivalent to <code>chmod u=rw,g=---,o=--- foo</code>. The
915 <code>+</code> and <code>-</code> characters can be specified instead
916 of <code>=</code> to set or unset specific permission bits while
917 leaving the remaining bits unchanged. For example <code>chmod go-rw
918 foo</code> turns off read and write permissions for non-owners. </p>
920 <p> Unprivileged users can only change the mode bits of their own
921 files or directories while there is no such restriction for the
924 SUBSECTION(«Hard and Soft Links»)
926 <p> Links make it possible to refer to identical files through
927 different paths. They come in two flavors: hard and soft. Both
928 types of links have advantages and disadvantages, and different
929 limitations. We start with hard links because these existed already
930 in the earliest Unix versions. </p>
932 <p> A file can have more than one directory entry that points to its
933 inode. If two directory entries point to the same inode, they are
934 said to be <em> hard links</em> of each other. The two entries are
935 equivalent in that they refer to the same file. It is impossible
936 to tell which of the two is the "origin" from which the "link"
937 was created. Hard links are created with the <code>link(2)</code>
938 system call or the <code>ln(1)</code> command. Both take two path
939 arguments, one for the existing file and one for the directory entry
940 to be created. The filesystem maintains in each inode a link counter
941 which keeps track of the number of directory entries which point to the
942 inode. The <code>link(2)</code> system call increases the link count
943 while <code>unlink(2)</code> decrements the link count and removes
944 the directory entry. If the decremented counter remains positive,
945 there is still at least one other directory entry which points to
946 the inode. Hence the file is still accessible through this other
947 directory entry and the file contents must not be released. Otherwise,
948 when the link counter reached zero, the inode and the file contents
949 may be deleted (assuming the file is not in use). </p>
951 <p> There are several issues with hard links. For one, hard links
952 can not span filesystems. That is, the two path arguments for
953 <code>link(2)</code> have to refer to files which reside on the
954 same filesystem. Second, it is problematic to create hard links to
955 directories. Early Unix systems allowed this for the superuser,
956 but on Linux the attempt to hard-link a directory always fails.
957 To address the limitations of hard links, <em>soft links</em>, also
958 called <em>symbolic links</em> (or <em>symlinks</em> for short),
959 were introduced. A soft link can be imagined as a special text file
960 containing a single absolute or relative path, the <em>target</em> of
961 the link. For relative paths the implied leading part is the directory
962 that contains the link. A soft link is thus a named reference in
963 the global hierarchy of files. Unlike hard links, the soft link
964 and its target do not need to reside on the same filesystem, and
965 there is a clear distinction between the link and its target. Soft
966 links are created with <code>symlink(2)</code> or by specifying the
967 <code>-s</code> option to the <code>ln(1)</code> command. </p>
969 <p> A soft link and its target usually point to different inodes. This
970 raises the following question: Should system calls which receive a
971 path argument that happens to be a soft link operate on the link
972 itself, or should they <em>follow</em> (or <em>dereference</em>)
973 the link and perform the operation on the target? Most system calls
974 follow soft links, but some don't. For example, if the path argument
975 to <code>chdir(2)</code> happens to be a soft link, the link is
976 dereferenced and the working directory is changed to the target of
977 the link instead. The <code>rename(2)</code> system call, however,
978 does not follow soft links and renames the link rather than its
979 target. Other system calls, including <code>open(2)</code>, allow
980 the caller to specify the desired behaviour by passing a flag to the
981 system call. For yet others there is a second version of the system
982 call to control the behaviour. For example, <code>lstat(2)</code> is
983 identical to <code>stat(2)</code>, but does not follow soft links. </p>
985 <p> It is possible for a soft link to refer to an invalid path. In
986 fact, <code>ln(1)</code> and <code>symlink(2)</code> do not consider
987 it an error if the target does not exist, and happily create a soft
988 link which points to an invalid path. Such soft links are called
989 <em>dangling</em> or <em>broken</em>. Dangling soft links also occur
990 when the target file is removed or renamed, or after a mount point
993 <p> Soft links may refer to other soft links. System calls which
994 follow soft links must therefore be prepared to resolve chains of
995 soft links to determine the file to operate on. However, this is not
996 always possible because soft links can easily introduce loops into the
997 hierarchy of files. For example, the commands <code>ln -s foo bar;
998 ln -s bar foo</code> create such a loop. System calls detect this
999 and fail with the <code>Too many levels of symbolic links</code>
1000 error when they encounter a loop. </p>
1002 <p> Another issue with both soft and hard links is that there is no
1003 simple way to find all directory entries which point to the same path
1004 (soft links) or inode (hard links). The only way to achieve this is
1005 to traverse the whole hierarchy of files. This may be prohibitive
1006 for large filesystems, and the result is unreliable anyway unless
1007 the filesystems are mounted read-only. </p>
1012 <li> A path can lack both slashes and components. Give an example
1013 of a path that lacks a slash and another example of a path that has
1014 no components. </li>
1016 <li> Assume <code>foo</code> is an existing directory. Guess what the
1017 command <code>mv foo bar</code> will do in each of the following cases:
1018 (a) <code>bar</code> does not exist, (b) <code>bar</code> exists and
1019 is a regular file, (c) <code>bar</code> exists and is a directory.
1020 Verify your guess by running the command. </li>
1022 <li> Many programs check if a path is valid and act differently
1023 according to the result. For example, a shell script might
1024 check for the existence of a file with code like <code>if test
1025 -e "$file"; do something_with "$file"; fi</code>. Explain
1026 why this approach is not bullet-proof. How could this be
1029 <li> Run <code>touch file-{1..100}</code> to create 100 files. Guess
1030 what the following commands will print. Run each command to confirm.
1033 <li> <code>ls file-</code> </li>
1034 <li> <code>ls file-*</code> </li>
1035 <li> <code>ls file-?</code> </li>
1036 <li> <code>ls file-[1-4]</code> </li>
1037 <li> <code>ls file-[1,3,5,7,9]*</code> </li>
1041 <li> Find an extended glob pattern for <code>bash(1)</code>
1042 that matches all valid paths whose last component starts with
1043 <code>file-</code>, followed by any number of odd digits (1, 3, 5,
1046 <li> Point out the flaw in the following shell code: <code>for
1047 f in file-*; do something_with "$f"; done</code>. Hint: Search
1048 <code>bash(1)</code> for "nullglob".
1050 <li> Create a file named <code>-r</code> with <code>echo >
1051 -r</code>. Try to remove the file with <code>rm -r</code> and
1052 discuss why this doesn't work as expected. Find a way to get rid
1053 of the file. Discuss what happens if you run <code>rm *</code> in a
1054 directory which contains a file named <code>-r</code>. </li>
1056 <li> The content of the <code>PATH</code> variable is a
1057 colon-separated list of directories in which the shell looks for
1058 commands to execute. Discuss the dangers of including the current
1059 working directory in this list. </li>
1061 <li> Run <code>id</code> to determine a group <code>G</code> you
1062 belong to but is not your primary group. Consider the following
1063 commands <code>mkdir foo; chgrp $G foo; touch foo/bar</code>. What
1064 is the group ID of <code>foo/bar</code>? Run the same commands, but
1065 insert <code>chmod g+s foo</code> as the second-to-last command. </li>
1067 <li> Run <code>man null</code> and <code>man zero</code> to learn
1068 about the properties of these two character devices. </li>
1070 <li> Assume the modification time stored in the inode of some file
1071 suggests that the file was last modified two years ago. How sure
1072 can you be that the file was never changed since then? Hint: See the
1073 <code>-d</code> option of <code>touch(1)</code>. </li>
1075 <li> Run the following commands <code>echo hello > foo</code>,
1076 <code>cat foo</code>, <code>chmod 600 foo</code>, <code>echo world >>
1077 foo</code>. Check the three timestamps with <code>stat foo</code>
1078 after each command. </li>
1080 <li> Determine the state of the permission bits of your own
1081 home directory by running <code>ls -ld ~</code>. Who can
1082 access its contents? Also look at the permission bits of
1083 other people's home directory. </li>
1085 <li> A file or directory is called <em>world-writeable</em>
1086 if the <code>w</code> bit is set in the <code>others</code>
1087 class of the permission bits. Create a world-writable
1088 directory with <code>mkdir foo; chmod 777 foo</code>
1089 and create a file in the new directory: <code>echo hello
1090 > foo/bar</code>. Is a different user allowed to create
1091 another file there (<code>echo world > foo/baz</code>)? Can
1092 he remove it again (<code>rm foo/baz</code>)? Will he succeed
1093 in removing <code>foo/bar</code> although it is owned by you
1094 and <em>not</em> writable to him? Try the same with the sticky
1095 bit turned on (<code>chmod 1777 foo</code>). </li>
1097 <li> Translate <code>rw-r--r--</code> into octal, and 755 into
1098 <code>rwx</code>-notation. </li>
1100 <li> Create a <a href="#hello_world">hello world script</a>, make it
1101 executable and run it. Create a subdirectory of your home directory
1102 and move the script into this directory. Set the permissions of the
1103 directory to <code>r--------</code>, check whether you still can
1104 list/execute it. Do the same with <code>--x------</code>. </li>
1106 <li> Create a file with <code>echo hello > foo</code>,
1107 create soft and hard links with <code>ln -s foo soft</code>
1108 and <code>ln foo hard</code>. Examine the inode numbers
1109 and link counts using the command <code>stat foo soft
1110 hard</code>. Remove <code>foo</code> and repeat. Try to
1111 access the file content through both links. Guess what
1112 <code>realpath foo soft hard</code> will print. Run the
1113 command to confirm. </li>
1115 <li> Create a dangling symlink with <code>ln -s /nope foo</code>. Would
1116 you expect the commands <code>ls -l foo</code> and <code>cat foo</code>
1117 succeed? Run these commands to verify your guess. </li>
1119 <li> One of the seven Unix file types is symlink. Why is there no
1120 file type for hard links? </li>
1124 How many paths are there that refer to the same file?
1126 Given the path <code>/foo/bar</code>, one may construct different paths
1127 which refer to the same file by inserting any number of <code>/.</code>
1128 or <code>../foo</code> after the first component. For example,
1129 <code>/foo/./bar</code> and <code>/foo/../foo/bar</code> both refer
1130 to the same file. If relative paths have to be taken into account as
1131 well, even more paths can be constructed easily. Hence the answer is:
1134 This illustrates the fundamental difference between a path and a
1135 file. Paths can be mapped to files, but not the other way around. In
1136 particular, there is no such thing like "the list of paths which have
1137 changed since yesterday".
1139 The concept of hard- and soft links complicates the situation further.
1143 Given two paths, how can one tell if they refer to the same file?
1146 Among other information, the metadata record of each file contains the
1147 so-called <em>inode number</em>, which uniquely identifies the file
1148 within the file system that contains the file. Therefore, if both
1149 paths are known to refer to files stored on the same file system,
1150 a comparison of the two inode numbers is sufficient to tell whether
1151 the two paths refer to the same file. The inode number can be obtained
1152 with the command <code>ls -i</code>.
1154 In the general case one additionally has to check that the
1155 two <code>device IDs</code> which identify the underlying file
1156 systems are also identical. Like the inode number, the device ID
1157 is part of the metadata of the file. It can be obtained by running
1158 <code>stat(1)</code>.
1163 Device nodes come in two flavors: Character and block devices. Explain
1164 the difference between the two device flavors.
1170 <li> Nine of the 12 mode bits of each file are the permission
1171 bits. The remaining three are the <em>sticky</em>, <em>setuid</em>
1172 and <em>setgid</em> bits. Explain the purpose of each. </li>
1174 <li> Run <code>find /usr/bin/ -perm -2000 -ls</code> to see all SUID
1175 executables in <code>/usr/bin</code>. Discuss why those programs have
1176 the SUID bit set. </li>
1182 How many possible permission modes exist for a file or directory on
1185 There are nine permission bits that can be turned on and off
1186 independently. Hence we have 2^9=512 possibilities. When taking into
1187 account the three special bits (sticky, setuid, setgid), the number
1188 increases to 2^12=4096.
1192 Explain each command of the <a href="«#»symlink_madness">script</a>
1193 below. Show the arrangement of all files and links in a figure,
1194 drawing a directory as a circle and a file as a square. How many
1195 different paths exist for the file <code> a</code>? Discuss whether
1196 the question "What's the path of a given file?" makes sense.
1202 width="150" height="125"
1203 xmlns="http://www.w3.org/2000/svg"
1204 xmlns:xlink="http://www.w3.org/1999/xlink"
1208 viewBox="0 0 10 10" refX="5" refY="5"
1209 markerWidth="4" markerHeight="4"
1210 orient="auto-start-reverse">
1211 <path d="M 0 0 L 10 5 L 0 10 z" />
1225 text-anchor="middle"
1242 text-anchor="middle"
1258 text-anchor="middle"
1280 C 150,90 150,30 80,20
1285 marker-end="url(#slm_arrow)"
1290 Since <code> foo/a</code>, <code> foo/testdir/a</code>, <code>
1291 foo/testdir/testdir/a </code> etc. all refer to the same file, there
1292 are infinitely many paths for the file <code> a</code>. Hence the
1293 question makes no sense: There is no such thing as <em> the </em>
1300 Recall that the path component <code>..</code> refers to the
1301 parent directory. Give an example of a path to a directory where
1302 "parent directory" and "directory identified by the path obtained by
1303 removing the last path component" are different. Which of the two
1304 interpretations of <code>..</code> does bash apply when you type
1305 <code>cd ..</code> at the bash prompt?
1311 Is it possible to choose among all possible paths which refer to the
1312 same file a <em>canonical</em> path? That is, a shortest (counting
1313 characters) absolute path which does not contain any soft links?
1317 <p> The POSIX standard requires each Unix system library to provide
1318 the <code>realpath()</code> function which performs the following
1319 substitutions on the given path: First, the path to the current
1320 working directory is prepended if the given path is relative
1321 (does not begin with a slash). Second, symbolic links are replaced
1322 by their targets. Third, any occurrences of <code>/.</code> and
1323 <code>foo/..</code> are removed. The thusly transformed path is
1324 returned by the function as the canonical path. </p>
1326 <p> Although each path can be canonicalized in this way, not all paths
1327 which refer to the same file give rise to the same canonical path. For
1328 example, <code>/tmp/foo</code> and <code>/tmp/bar</code> could refer
1329 to regular files which are hard links of each other. In this case the
1330 paths refer to the same file, yet the paths are different and already
1331 canonicalized. The same can happen when a file system (or a subtree
1332 of it) is <em>bind mounted</em>. That is, the file system tree is
1333 visible at two or more locations in the global directory tree. </p>
1335 The message of this exercise is to convince the reader that it is
1336 incorrect to assume that two files are different because their paths
1341 SECTION(«Processes»)
1343 <p> A <em>program</em> consists of instructions and data stored in
1344 a regular file. A <em> user process</em> is an instance of a running
1345 program. This is in contrast to <em>kernel processes</em> which are
1346 created directly by the kernel and have no relationship to executable
1347 files. Since we shall only be concerned with user processes, we will
1348 refer to these as "processes" from now on. In this section we'll see
1349 how processes are created and removed. We will then take a closer look
1350 at the enviroment of a process and discuss how processes communicate
1351 with each other. </p>
1353 SUBSECTION(«Process Tree, Zombies and Orphans»)
1355 <p> When the system boots, there is only one process, the
1356 <em>init</em> process, which is created by the kernel at the end
1357 of the boot sequence by executing <code>/sbin/init</code>. All
1358 other processess are created from existing processes by means of
1359 the <code>fork(2)</code> system call. The process which called
1360 <code>fork(2)</code> is said to be the <em>parent</em> of the newly
1361 created <em>child</em>. After <code>fork(2)</code> returns, both
1362 parent and child are executing independently of each other. Both
1363 processes may call <code>fork(2)</code> again to create further
1364 processes. This gives rise to a tree structure where the processes
1365 are the nodes of the tree with init being the root node. The edges
1366 of the tree describe the parent-child relationships. </p>
1368 <p> If there are more processes than CPUs, not all processes can
1369 run simultaneously. It is the mission of the kernel's <em>task
1370 scheduler</em> to assign processes to CPUs and to perform <em>context
1371 switches</em>. That is, to take away the CPU from a running process
1372 in order to give another process the chance to run. The scheduler
1373 has to choose the duration of each process' time slice and it must
1374 pick the process to switch to when the time slice of the current
1375 process has expired or the process gives up the CPU voluntarily, for
1376 example because it is waiting for an I/O operation to complete. This
1377 is a non-trivial task at least for modern multi-CPU systems with
1378 <em>non-uniform memory access</em> (NUMA) where the memory access times
1379 depend on the memory location and the processor core. Things don't
1380 get easier if the CPU clock speed can vary and/or scheduling must be
1381 power-aware to optimize battery time. To make good decisions, some
1382 information has to be provided by the processes or by a system-wide
1383 policy. One elementary way to prioritize certain processes over others
1384 is via <em>nice levels</em> which we shall discuss below. </p>
1386 <p> The normal way for a process to terminate is to call
1387 <code>exit(3)</code> after it has done its work. This function
1388 transfers an integer value, the <em>exit status</em>, to the
1389 kernel. The exit status can only be retrieved by the parent of
1390 the terminating process. To illustrate this concept, imagine an
1391 interactive shell which creates one child each time the user enters
1392 a command. The children are short living while the parent, the shell,
1393 stays around for much longer. During command execution the parent needs
1394 to wait for its child to terminate. It may then want to tell whether
1395 the child has terminated successfully. To achieve this, the parent
1396 calls one of the <em>wait</em> system calls (<code>wait(2)</code>,
1397 <code>waitpid(2)</code>, <code>waitid(2)</code>) which block until the
1398 child has terminated, then return the child's exit status. After the
1399 child called <code>exit(3)</code> but before the parent has called
1400 one of the wait functions, the kernel needs to keep at least the
1401 exit status (and possibly further information) about the terminated
1402 child. During this time window the child has already terminated but
1403 a part of it still exists in kernel memory. Processes in this state
1404 are aptly called <em>zombies</em>. </p>
1406 <p> Unlike in the shell scenario outlined above, a process might well
1407 have any number of children at the time it terminates. Its children
1408 then become <em>orphans</em> as they lose their parent. The kernel
1409 cannot simply remove the terminated process from the process tree
1410 because this would disconnect its orphaned children from the other
1411 processes in the tree, destroying the tree structure. To avoid this,
1412 orphans are <em>reparented to init</em>, that is, made children
1413 of the init process. This works because the init process never
1416 <p> There are several programs which show information about
1417 processes. The POSIX <code>ps(1)</code> command prints a list of
1418 processes. It has many options that control which processes are
1419 shown and how the output is formatted. Similar programs which are not
1420 covered by POSIX are <code>pstree(1)</code>, <code>top(1)</code> and
1421 <code>htop(1)</code>. The former shows the tree structure while the
1422 latter two provide a dynamic real-time view of the process tree. The
1423 exercises of this section invite the reader to become familiar with
1424 these essential programs. </p>
1426 SUBSECTION(«File Execution»)
1428 <p> When a process calls <code>fork(2)</code>, the newly created
1429 child process starts out as a copy of the calling process. However,
1430 the reason to create a new process is usually to let the child do
1431 something different than the parent. Therefore, <code>fork(2)</code>
1432 is often followed by a second system call which replaces the child
1433 process with a different program. There are several similar system
1434 calls which do this, with slight semantic differences. We refer to
1435 this family of system calls as the <em>exec system calls</em>. </p>
1437 <p> All exec system calls receive a path argument from which they
1438 determine an executable file that contains the program to run. Linux
1439 and BSD store executables in <em>Executable and Linkable Format</em>
1440 (ELF). Executables are typically linked <em>dynamically</em>.
1441 That is, the dependent libraries (at least libc, but possibly many
1442 more) are loaded at runtime from separate files. This is in contrast
1443 to <em>static linking</em> which includes all dependencies in the
1444 executable, making the executable self-contained but also larger
1445 and harder to maintain. Regardless of the type of linking, when the
1446 program is loaded, it completely replaces the previously running
1447 program. Note that there can be more than one process at the same
1448 time which executes the same program. </p>
1450 <p> Files in ELF format are called <em>native</em> executables because
1451 they contain machine instructions which can be executed directly
1452 by the CPU. Another type of executables are <em>scripts</em>,
1453 also known as <em>interpreter files</em>. Scripts are text files
1454 which start with the <em>shebang</em> (<code>#!</code>). They can
1455 not run directly but have to be <em>interpreted</em> at runtime
1456 by another program, the <em>interpreter</em>. Nevertheless, it is
1457 possible to execute a script as if it were a native executable:
1458 by passing the path to one of the exec system calls or by typing
1459 the path at the shell prompt. The exec system call recognizes that
1460 the given file is a script by investigating the first line, which
1461 contains the path to the interpreter after the shebang, optionally
1462 followed by options to the interpreter. The kernel then executes the
1463 interpreter rather than the script, passing the path to the script as
1464 an argument. For example, if <code>/foo/bar</code> is being executed,
1465 and the first line of this file is <code>#!/bin/sh</code>, the kernel
1466 executes <code>/bin/sh /foo/bar</code> instead. Popular interpreters
1467 besides <code>/bin/sh</code> include <code>/bin/bash</code>,
1468 <code>/usr/bin/perl</code>, <code>/usr/bin/python</code> and
1469 <code>/usr/bin/awk</code>. </p>
1471 SUBSECTION(«File Descriptions and File Descriptors»)
1473 <p> The kernel must always be aware of the set of all objects which are
1474 currently in use. This set is often called the <em>system-wide table
1475 of open files</em> although not all entries refer to files. In fact, an
1476 entry may refer to any object that supports I/O operations, for example
1477 a network socket. Each entry is called a <em>file description</em>,
1478 which is a somewhat unfortunate term that was coined by POSIX. A
1479 file description records information about the object itself as well
1480 as the current state of the reference, including the file offset,
1481 if applicable, and the <em>status flags</em> which affect how future
1482 I/O operations are going to be performed through this reference. </p>
1484 <p> The kernel maintains for each process an array of pointers to file
1485 descriptions. Each such pointer is a <em>file descriptor</em>. Unlike
1486 files and file descriptions, a file descriptor always corresponds
1487 to a process and is identified by a non-negative number, the index
1488 into the pointer array of that process. This index is returned by
1489 system calls like <code>open(2)</code> or <code>socket(2)</code>.
1490 As far as user space programs are concerned, a file descriptor is
1491 synonymous with this integer. It can be regarded as an abstract
1492 <em>handle</em> that must be supplied to subsequent I/O operations
1493 like <code>read(2)</code> or <code>write(2)</code> to tell the system
1494 call the target object of the operation. </p>
1496 <p> The shell automatically creates three file descriptors for each
1497 process which are identified by the integers 0, 1 and 2. They are
1498 called <em>stdin</em>, <em>stdout</em>, and <em>stderr</em>, which is
1499 short for <em>standard input/output/error</em>. It is possible, and in
1500 fact common, that all three file descriptors point to the same file
1501 description: the terminal device. Many command line tools read their
1502 input from stdin, write normal output to stdout, and error messages
1503 to stderr. For example, when the POSIX command <code>cat(1)</code>
1504 is invoked with no arguments, it reads data from stdin and writes
1505 the same data to stdout. </p>
1507 SUBSECTION(«Signals»)
1509 <p> Signals are another ancient Unix concept that dates back to the
1510 early 1970s and was standardized in POSIX long ago. This concept
1511 facilitates a rudimentary form of <em>inter process communication</em>
1512 (IPC) between unrelated processes. Signals can be regarded as software
1513 interrupts that transmit a notification event from the sending process
1514 to the target process. The event is sent <em>asynchronously</em>,
1515 meaning that the interruption can happen at any location of the code
1518 <p> It is fair to say that most non-trivial programs, including
1519 scripts, have to deal with signals. All major scripting languages
1520 (bash, python, perl, ruby, etc.) provide an API for signal
1521 handling. The interpreter of the scripting language ends up calling
1522 the POSIX system functions, so we will only look at these. </p>
1524 <p> Signals are identified by name or by a numerical ID. For example,
1525 <code>SIGINT</code> (interrupt from keyboard) is the name for signal
1526 number 2. POSIX defines 31 <em>standard signals</em> plus at least
1527 eight <em>real-time signals</em>. The standard signals can be
1528 subdivided according to the origin of the signal as follows. </p>
1531 <dt> hardware related signals </dt>
1533 <dd> These signals originate from <em>hardware traps</em> that force
1534 the CPU back into kernel mode. The kernel responds to the trap by
1535 generating a signal for the process that caused the trap. For example,
1536 a division by zero in a user space program triggers a hardware trap
1537 in the <em>floating point unit</em> (FPU) of the CPU. The kernel
1538 then generates the <code>SIGFPE</code> (floating-point exception)
1539 signal for the process. Another example for a signal that originates
1540 from a hardware trap is <code>SIGSEGV</code> (segmentation fault)
1541 which occurs when a process attempts to reference a memory address
1542 that has not been mapped (i.e., marked as valid) by the <em>memory
1543 management unit</em> (MMU) of the CPU. </dd>
1545 <dt> kernel generated signals </dt>
1547 <dd> Signals which originate from the kernel rather than from
1548 hardware. One example is <code>SIGCHLD</code> (child terminated),
1549 which is sent to the parent process when one of its child processes
1550 terminates. Another example is <code>SIGWINCH</code> (window resize),
1551 which is generated when the geometry of the controlling terminal of
1552 a process changes. </dd>
1554 <dt> user-space generated signals </dt>
1556 <dd> These signals can only originate from user space when a process,
1557 for example <code>kill(1)</code>, calls <code>raise(2)</code>
1558 or <code>kill(2)</code> to instruct the kernel to generate a
1559 signal. Examples are <code>SIGTERM</code>, which issues a termination
1560 request, and <code>SIGUSR1</code> and <code>SIGUSR2</code> which are
1561 reserved for use by application programs. </dd>
1564 The following signals are used frequently and deserve to the described
1565 explicitly. We refer to <code>signal(7)</code> for the full list of
1566 signals and their semantics.
1569 <dt> <code>SIGINT, SIGTERM</code> and <code>SIGKILL</code> </dt>
1571 <dd> All three signals terminate the process by default.
1572 <code>SIGINT</code> is generated for the <em>foreground processes</em>
1573 when the <em>interrupt character</em> (CTRL+C) is pressed in a
1574 terminal. For example, if CTRL+C is pressed while the shell pipeline
1575 <code>find | wc </code> is executing, <code>SIGINT</code> is sent
1576 to both processes of the pipeline. <code>SIGTERM</code> is the
1577 default signal for the <code>kill(1)</code> command. It requests
1578 the target process to run its shutdown procedure, if any, then
1579 terminate. <code>SIGKILL</code> instructs the kernel to terminate the
1580 target process immediately, without giving the process the chance to
1581 clean up. This signal can originate from a process or from the kernel
1582 in response to running out of memory. To keep the system working, the
1583 kernel invokes the infamous <em>out of memory killer</em> (OOM killer)
1584 which terminates one memory-hungry process to free some memory. </dd>
1586 <dt> <code>SIGSTOP</code>, <code>SIGTSTP</code> and <code>SIGCONT</code> </dt>
1588 <dd> <code>SIGSTOP</code> instructs the task scheduler of the kernel to
1589 no longer assign any CPU time to the target process until the process
1590 is woken up by a subsequent <code>SIGCONT</code>. <code>SIGTSTP</code>
1591 (stop typed at terminal) stops all foreground processes of a terminal
1592 session. It is generated when the <em>stop character</em> (CTRL+Z)
1593 is pressed in a terminal. </dd>
1596 <p> Processes may set the <em>signal disposition</em> of most signals
1597 to control what happens when the signal arrives. When no disposition
1598 has been set, the signal is left at its <em>default disposition</em> so
1599 that the <em>default action</em> is performed to deliver the signal.
1600 For most signals the default action is to terminate the process,
1601 but for others the default action is to <em>ignore</em> the signal.
1602 If the signal is neither ignored nor left at its default disposition,
1603 it is said to be <em>caught</em> by the process. To catch a signal the
1604 process must tell the kernel the address of a function, the <em>signal
1605 handler</em>, to call in order to deliver the signal. The set of
1606 signal dispositions of a process can thus be imagined as an array
1607 of function pointers with one pointer per possible signal. If the
1608 process catches the signal, the pointer points to the corresponding
1609 signal handler. A NULL pointer represents a signal that was left at
1610 its default disposition while the special value <code>SIG_IGN</code>
1611 indicates that the process explicitly asked to ignore this signal. </p>
1613 <p> Signals can also be <em>blocked</em> and <em>unblocked</em>. When
1614 a signal is generated for a process that has it blocked, it remains
1615 <em>pending</em>. Pending signals cause no action as long as the
1616 signal remains blocked but the action will be performed once the
1617 signal gets unblocked. <code>SIGKILL</code> and <code>SIGSTOP</code>
1618 can not be caught, blocked, or ignored. </p>
1620 <p> Real-time signals are similar to <code>SIGUSR1</code> and
1621 <code>SIGUSR2</code> in that they have no predefined meaning but
1622 can be used for any purpose. However, they have different semantics
1623 than the standard signals and support additional features. Most
1624 importantly, real-time signals are <em>queued</em>, meaning that in
1625 contrast to standard signals the same real-time signal can be pending
1626 multiple times. Also, the sending process may pass an <em>accompanying
1627 value</em> along with the signal. The target process can obtain this
1628 value from its signal handler along with additional information like
1629 the PID and the UID of the process that sent the signal. </p>
1631 <p> Some system calls including <code>read(2)</code> and
1632 <code>write(2)</code> may block for an indefinite time. For
1633 example, reading from a network socket blocks until there is data
1634 available. What should happen when a signal arrives while the process
1635 is blocked in a system call? There are two reasonable answers: Either
1636 the system call is <em>restarted</em>, or the call fails with the
1637 <code>Interrupted system call</code> error. Unfortunately, different
1638 flavors of Unix handle this case differently by default. However,
1639 applications may request either behaviour by setting or clearing the
1640 <code>SA_RESTART</code> flag on a per-signal basis. </p>
1642 SUBSECTION(«Environment of a Process»)
1644 <p> Now that we have a rough understanding of processes we look
1645 closer at the information the kernel maintains for each process. We
1646 already discussed the array of file descriptors and the array of
1647 signal dispositions. Clearly both are process specific properties.
1648 As we shall see, there is much more what constitutes the environment
1651 <p> Each process is identified by a unique <em>process ID</em>
1652 (PID), which is a positive integer. The <code>init</code> process is
1653 identified by PID 1. PIDs are assigned in ascending order, but are
1654 usually restricted to the range 1..32767. After this many processes
1655 have been created, PIDs wrap and unused PIDs are recycled for new
1656 processes. Thus, on a busy system on which processes are created and
1657 terminated frequently, the same PID is assigned to multiple processes
1660 <p> Not all processes call <code>fork(2)</code> to create a child
1661 process, but each process except the init process has a unique
1662 parent. As described before, this is either the "real" parent (the
1663 process which created the process earlier) or the init process that
1664 "adopted" the orphaned process in case the real parent terminated
1665 before the child. The process ID of the parent (PPID) is thus
1666 well-defined. A process can obtain its PID and PPID with the
1667 <code>getpid(2)</code> and <code>getppid(2)</code> system calls. </p>
1669 <p> Each process runs on behalf of a user (possibly the superuser)
1670 which is identified by its <em>user ID</em> (UID) and belongs to
1671 one or more groups, identified by one or more <em>group IDs</em>
1672 (GIDs). The superuser is identified by UID zero. When we talked
1673 about the permission bits of files and directories, we said that
1674 suitable permissions are needed for system calls which operate on
1675 files (<code>open(2)</code>, <code>stat(2)</code>, etc.). A more
1676 precise statement is that the <em>process</em> which calls, say,
1677 <code>open(2)</code> needs to have these permissions. To decide this,
1678 the kernel needs to take into account the UID and GIDs of the process
1679 that called <code>open(2)</code>, the UID and the GID stored in the
1680 inode of the file that is being opened, and the permission bits of
1681 this file. The UID is also taken into account for <code>kill(2)</code>
1682 because unprivileged processes (non-zero UID) can only send signals
1683 to processes which run on behalf of the same user while the superuser
1684 may target any process. </p>
1686 <p> Each process has a <em>current working directory</em> (CWD)
1687 associated with it. When the user logs in, the CWD of the login shell
1688 process is set to his <em>home directory</em>, which should always
1689 exist and have the read, write and execute permission bits set for
1690 the user. The CWD can later be changed with <code>chdir(2)</code>
1691 and be retrieved with <code>getcwd(3)</code>. The CWD is used as the
1692 starting point for path searches for relative paths. It affects most
1693 system calls which receive a path argument. For example, if the CWD
1694 is <code>/foo/bar</code> and the relative path <code>baz/qux</code>
1695 is passed to <code>open(2)</code>, the kernel will attempt to open
1696 the file which is identified by <code>/foo/bar/baz/qux</code>. </p>
1698 <p> Many programs accept arguments to control their behavior.
1699 In addition to the path to the program that is to be executed,
1700 all variants of the exec system calls receive an array of arguments
1701 called the <em>argument vector</em>. For example, when the command
1702 <code>ls -l foo</code> is executed, the argument vector contains
1703 the two strings <code>"-l"</code> and <code>"foo"</code>. Note that
1704 the argument vector is not part of the program but is tied to the
1705 process. It is passed to the main function at startup so that the
1706 program may evaluate it and act accordingly. </p>
1708 <p> Another way to pass information to a program is via <em>environment
1709 variables</em>. Environment variables are strings of the form
1710 <code>name=value</code>. POSIX describes the API to maintain the
1711 environment variables of a process. Environment variables are set
1712 with <code>setenv(3)</code> or <code>putenv(3)</code>, the value of a
1713 variable can be retrieved with <code>getenv(3)</code>, and a variable
1714 and its value can be deleted with <code>unsetenv(3)</code>. The set of
1715 environment variables is sometimes called the <em>environment</em>
1716 of the process, although we use this term in a broader sense to
1717 describe the entirety of all metadata maintained by the kernel about
1718 the process, including but not limited to environment variables. </p>
1720 <p> Each process also has about a dozen <em>resource limits</em>
1721 that can be set and queried with the POSIX <code>setrlimit(2)</code>
1722 and <code>getrlimit(2)</code> functions. Each limit controls a
1723 different aspect. For example, <code>RLIMIT_CPU</code> limits the
1724 CPU time the process is allowed to use and <code>RLIMIT_NOFILE</code>
1725 controls how many open files it may have at a time. For each resource
1726 there is a <em>soft</em> and a <em>hard</em> limit. The kernel
1727 enforces the value stored as the soft limit. This value may be set
1728 by an unprivileged process to any value between zero and the hard
1729 limit. Unprivileged processes may also reduce (but not increase) their
1730 hard limits. Once a hard limit is reduced, it can not be increased
1731 any more. For <code>RLIMIT_CPU</code> a special convention applies:
1732 If the soft limit is reached, the kernel sends <code>SIGXCPU</code>
1733 (CPU time limit exceeded) to notify the process about this fact so
1734 that it can terminate orderly (e.g., remove temporary files). When
1735 the hard limit is reached, the kernel terminates the process as if
1736 it received <code>SIGKILL</code>. </p>
1738 <p> The <em>nice level</em> of a process provides a hint for
1739 the task scheduler to give the process a lower or higher priority
1740 relative to other processes. Nice levels range between -20 and 19. A
1741 high nice level means that the process wants to be nice to other
1742 processes, that is, should run with reduced priority. Low nice levels
1743 indicate important processes that should be prioritized over other
1744 processes. The default nice level is zero. Unprivileged users may
1745 set the nice level of new processes with the <code>nice(1)</code>
1746 command to any non-negative value. They can also increase the nice
1747 level of their existing processes with <code>renice(1)</code>, but
1748 never decrease it. The superuser, however, may set the nice level
1749 of any process to an arbitrary value in the valid range. </p>
1751 <p> The bulk of the properties discussed so far are inherited by the
1752 child after a <code>fork(2)</code>. Specifically, the child gets the
1753 same array of file descriptors and signal dispositions as its parent,
1754 runs on behalf of the same user, has the same working directory,
1755 the same resource limits and nice level, and also the same set
1756 of environment variables with identical values. The PID and PPID,
1757 however, are different of course. </p>
1759 <p> After a process has called an exec function to replace itself with
1760 a new program, its signal handlers no longer exist because they were
1761 part of the program code which has been replaced. Therefore the exec
1762 system calls reset the disposition of all signals that were caught to
1763 the default disposition. Signals that were being ignored keep being
1764 ignored, however. </p>
1766 SUBSECTION(«The Process Filesystem»)
1768 <p> Although not covered by POSIX, at least Linux, NetBSD and FreeBSD
1769 provide information about processes via the <em>process filesystem</em>
1770 (procfs), which is usually mounted on <code>/proc</code>. The process
1771 filesystem is a <em>pseudo-filesystem</em>, i.e., it has no underlying
1772 storage device. Files and directories are faked by the kernel as they
1773 are accessed. Each process is represented by a numerical subdirectory
1774 of <code>/proc</code> which is named by the PID. For example,
1775 <code>/proc/1</code> represents the init process. The aforementioned
1776 process utilities (<code>ps(1)</code>, <code>top(1)</code>, etc.) read
1777 the contents of the process filesystem in order to do their job. <p>
1779 <p> Each <code>/proc/[pid]</code> directory contains the same set
1780 of files although this set is different between Unixes. These files
1781 expose much of the environment of the process to user space. The Linux
1782 procfs implementation provides text files named <code>environ</code>
1783 and <code>limits</code> which contain the current environment and
1784 the resource limits of the process, respectively. Moreover, the
1785 file descriptor array of each process is exposed in the files of
1786 the <code>/proc/[pid]/fd</code> directory. Linux and NetBSD (but not
1787 FreeBSD) also provide a <code>cwd</code> soft link which points to
1788 the current working directory of the process. </p>
1790 SUBSECTION(«Pipes and Redirections»)
1792 <p> The <code>pipe(2)</code> system call takes no arguments and
1793 creates two file descriptors for the calling process which are tied
1794 together as a unidirectional first in, first out data channel. One
1795 file descriptor is the <em>read end</em> of the pipe, the other is
1796 the <em>write end</em>. Data written to the write end is buffered by
1797 the kernel and can be obtained by reading from the read end. </p>
1799 <p> One application of pipes is communication between
1800 related processes. A process first creates a pipe, then calls
1801 <code>fork(2)</code> to create a child process. The child inherits
1802 a copy of both pipe file descriptors. Hence the parent process can
1803 communicate with the child by writing a message to the write end of
1804 the pipe for the child to read. </p>
1806 <p> This approach depends on file descriptor inheritance across
1807 <code>fork(2)</code>, so it does not work in the situation
1808 where neither process is an ancestor of the other. Files of
1809 type <em>fifo</em> (named pipes) overcome this restriction. To
1810 establish a connection between two <em>unrelated</em> processes,
1811 both processes call <code>open(2)</code> to obtain a file
1812 descriptor which is associated with the fifo. One process passes
1813 the <code>O_WRONLY</code> flag to open the file for writing while
1814 the other passes <code>O_RDONLY</code> to open it for reading. The
1815 two processes may then communicate in the same way as with the
1816 <code>pipe(2)/fork(2)</code> approach. </p>
1818 <p> The POSIX <code>dup(2)</code> and <code>dup2(2)</code> system
1819 calls allow a process to manipulate the entries of its file descriptor
1820 array. In particular the standard file descriptors 0, 1, and 2 can be
1821 replaced. By doing so before performing an exec system call, it can
1822 be arranged that the replacement program starts with, say, its stdout
1823 file descriptor be redirected to the write end of a pipe. Note that
1824 the replacement program does not need any modifications for this to
1825 work, and might not even be aware of the fact that it is not writing
1826 its output to the terminal as usual. </p>
1828 <p> Shells employ this technique to implement the <code>|</code>
1829 operator which "pipes" the output of one command "into" another
1830 command. For example, the pipeline <code>ls | wc</code> works
1831 as follows: First the shell creates a pipe, then it calls
1832 <code>fork(2)</code> twice to create two processes which both
1833 get a copy of the two pipe file descriptors. The first process
1834 replaces its stdout file descriptor with the write end of the
1835 pipe and performs an exec system call to replace itself with the
1836 <code>ls(1)</code> program. The second process replaces its stdin
1837 file descriptor with the read end of the pipe and replaces itself
1838 with <code>wc(1)</code>. Since <code>ls(1)</code> writes to stdout
1839 and <code>wc(1)</code> reads from stdin, <code>wc(1)</code> processes
1840 the output of <code>ls(1)</code>. </p>
1844 <p> The POSIX standard requires a compliant Unix system to provide
1845 a collection of functions that let applications perform input and
1846 output by means of operations on <em>streams</em>. This programming
1847 interface, known as <em>stdio</em> for <em>standard input/output</em>,
1848 is part of every Unix system since 1979. Every program which contains
1849 a <code>printf(3)</code> statement relies on stdio. </p>
1851 <p> The stdio functions are implemented as part of libc on top of the
1852 <code>open(2)</code>, <code>read(2)</code> and <code>write(2)</code>
1853 system calls which are implemented in the kernel. Roughly speaking,
1854 stdio replaces the file descriptor API by a more abstract API
1855 which centers around streams. A stream is an opaque data structure
1856 which comprises a file descriptor and an associated data buffer for
1857 I/O. Each program has three predefined streams which correspond to
1858 the three standard file descriptors (stdin, stdout and stderr). The
1859 stdio API contains well over 50 functions to create and maintain
1860 streams and to perform I/O on streams. These functions take care of
1861 the characteristics of the underlying file description. For example,
1862 they automatically try to select the optimal I/O buffer size. </p>
1864 <p> Many applications rely on stdio because of convenience. For
1865 one, the buffers for <code>read(2)</code> and <code>write(2)</code>
1866 must be allocated and freed explicitly by the application, and care
1867 must be taken to not overflow these buffers. With stdio, this task
1868 is done by the stdio library. Second, <em>formatted</em> I/O is
1869 much easier to do with the stdio functions because the programmer
1870 only has to provide a suitable <em>format string</em> to convert
1871 between the machine representation and the textual representation of
1872 numbers. For example, by passing the format string <code>"%f"</code>
1873 to <code>scanf(3)</code>, the programmer tells the stdio library to
1874 read a floating point number stored in textual representation from the
1875 specified stream, convert it to machine representation and store it
1876 in the given variable. The <code>fprintf(3)</code> function works the
1877 other way round: the value is converted from machine representation
1878 to text, and this text is written to the stream. Both functions can
1879 deal with various formats, like scientific notation for floating
1880 point numbers (e.g., 0.42E-23). With stdio it is easy to customize
1881 the formatted output, for example add leading zeros or select the
1882 number of decimal digits in the textual representation of a floating
1885 <p> Another reason why many programs rely on stdio is that it performs
1886 <em>buffered</em> I/O. With buffered I/O not each read/write operation
1887 results in a system call. Instead, data read from or written to the
1888 stream is first stored in the user space buffer that is associated
1889 with the stream. This is a performance improvement for applications
1890 which perform many small I/O operations because every system call
1891 incurs some overhead. Buffers may be <em>flushed</em> explicitly by
1892 calling <code>fflush(3)</code>, or implicitly by the stdio library. The
1893 criteria which cause an implicit flush depend on the <em>buffering
1894 type</em> of the stream. Three buffering types exist. </p>
1897 <dt> unbuffered </dt>
1899 <dd> The stdio library does not buffer any reads or writes. Instead,
1900 each I/O operation results in a <code>read(2)</code> or
1901 <code>write(2)</code> system call. By default the stderr stream is
1902 unbuffered to display error messages as quickly as possible. </dd>
1904 <dt> line buffered </dt>
1906 <dd> System calls are performed when a newline character is
1907 encountered. This buffering type applies by default to interactive
1908 sessions where the file descriptor of the stream refers to a terminal
1909 device (as determined by <code>isatty(3)</code>). </dd>
1911 <dt> fully buffered </dt>
1913 <dd> I/O takes place only if the buffer of the stream is empty/full. By
1914 default, if the file descriptor refers to a regular file, the
1915 stream is fully buffered. POSIX requires that stderr is never fully
1919 <p> The exercises on stdio focus on the three different buffering
1920 types because this is a common source of confusion. </p>
1922 SUBSECTION(«The Virtual Address Space of a Process»)
1924 <p> Isolation refers to the concept that each process gets its own
1925 <em>virtual address space</em>. A rough understanding of the memory
1926 management system and the layout of the virtual address space of
1927 a process helps to locate the source of common problems like the
1928 infamous <code>segmentation fault</code> error, and to realize that
1929 putatively simple questions such as "how much memory is my process
1930 currently using?" are in fact not simple at all, and need to be made
1931 more precise before they can be answered. </p>
1935 define(«vas_width», «200»)
1936 define(«vas_height», «300»)
1937 define(«vas_vmem_left_margin», «5»)
1938 define(«vas_vmem_top_margin», «5»)
1939 define(«vas_mem_width», «20»)
1940 define(«vas_gap_width», «30»)
1941 define(«vas_vmem_height», «140»)
1942 define(«vas_vmem_color», «#34b»)
1943 define(«vas_pmem_height», «100»)
1944 define(«vas_pmem_color», «#7e5»)
1945 define(«vas_vmem_unmapped_color», «#a22»)
1946 define(«vas_vmem_swapped_color», «yellow»)
1947 define(«vas_pmem_unavail_color», «orange»)
1948 define(«vas_disk_gap», «15»)
1949 define(«vas_disk_height», «20»)
1950 define(«vas_disk_color», «grey»)
1951 define(«vas_x1», «vas_vmem_left_margin()»)
1952 define(«vas_x2», «eval(vas_x1() + vas_mem_width())»)
1953 define(«vas_x3», «eval(vas_x2() + vas_gap_width())»)
1954 define(«vas_x4», «eval(vas_x3() + vas_mem_width())»)
1956 define(«vas_membox», «
1958 fill="$1" stroke="black" stroke-width="1"
1959 x="eval(vas_vmem_left_margin() + $3)"
1960 y="vas_vmem_top_margin()"
1961 width="vas_mem_width()" height="$2"
1964 define(«vas_vmem_unmapped_box», «
1966 fill="vas_vmem_unmapped_color()" stroke="black" stroke-width="1"
1967 x="eval(vas_vmem_left_margin())"
1968 y="eval(vas_vmem_top_margin() + $1)"
1969 width="vas_mem_width()"
1974 define(«vas_vmem_swapped_box», «
1976 fill="vas_vmem_swapped_color()" stroke="black" stroke-width="1"
1977 x="eval(vas_vmem_left_margin())"
1978 y="eval(vas_vmem_top_margin() + $1)"
1979 width="vas_mem_width()"
1984 define(«vas_pmem_unavail_box», «
1986 fill="vas_pmem_unavail_color()" stroke="black" stroke-width="1"
1987 x="eval(vas_vmem_left_margin() + vas_mem_width() + vas_gap_width())"
1988 y="eval(vas_vmem_top_margin() + $1)"
1989 width="vas_mem_width()"
1994 define(«vas_vmem_hline», «
1996 x1="vas_vmem_left_margin()"
1997 y1="eval(vas_vmem_top_margin() + $1)"
1998 x2="eval(vas_vmem_left_margin() + vas_mem_width())"
1999 y2="eval(vas_vmem_top_margin() + $1)"
2006 define(«vas_pmem_hline», «
2007 «<!-- pmem hline -->»
2009 x1="eval(vas_vmem_left_margin() + vas_mem_width() + vas_gap_width())"
2010 y1="eval(vas_vmem_top_margin() + $1)"
2011 x2="eval(vas_vmem_left_margin() + 2 * vas_mem_width() + vas_gap_width())"
2012 y2="eval(vas_vmem_top_margin() + $1)"
2018 define(«vas_arrow», «
2020 x1="eval(vas_vmem_left_margin() + vas_mem_width())"
2021 y1="eval(vas_vmem_top_margin() + $1)"
2022 x2="eval(vas_vmem_left_margin() + vas_mem_width() + vas_gap_width() - 2)"
2023 y2="eval(vas_vmem_top_margin() + $2)"
2026 marker-end="url(#arrow)"
2029 define(«vas_disk», «
2031 fill="vas_disk_color()" stroke="black" stroke-width="1"
2033 y="eval(vas_vmem_top_margin() + vas_pmem_height()
2035 width="eval(vas_x4() - vas_x3())"
2036 height="eval(vas_disk_height())"
2039 cx="eval(vas_x3() + vas_mem_width() / 2)"
2040 cy="eval(vas_vmem_top_margin() + vas_pmem_height() + vas_disk_gap())"
2041 rx="eval(vas_mem_width() / 2)"
2042 ry="eval(vas_mem_width() / 4)"
2043 fill="vas_disk_color()" stroke="black"
2046 cx="eval(vas_x3() + vas_mem_width() / 2)"
2047 cy="eval(vas_vmem_top_margin() + vas_pmem_height()
2048 + vas_disk_gap() + vas_disk_height())"
2049 rx="eval(vas_mem_width() / 2)"
2050 ry="eval(vas_mem_width() / 4)"
2051 fill="vas_disk_color()" stroke="black"
2056 width="vas_width()" height="vas_height()"
2057 viewBox="0 0 100 eval(100 * vas_height() / vas_width())"
2058 xmlns="http://www.w3.org/2000/svg"
2059 xmlns:xlink="http://www.w3.org/1999/xlink"
2063 viewBox="0 0 10 10" refX="5" refY="5"
2064 markerWidth="4" markerHeight="4"
2065 orient="auto-start-reverse">
2066 <path d="M 0 0 L 10 5 L 0 10 z" />
2068 vas_membox(«vas_vmem_color()», «vas_vmem_height()», «0»)
2069 vas_membox(«vas_pmem_color()», «vas_pmem_height()»,
2070 «eval(vas_gap_width() + vas_mem_width())»)
2071 vas_vmem_hline(«10»)
2072 vas_vmem_hline(«40»)
2073 vas_vmem_unmapped_box(«40», «20»)
2074 vas_vmem_swapped_box(«60», «60»)
2076 vas_pmem_unavail_box(«0», «10»)
2077 vas_pmem_hline(«20»)
2078 vas_pmem_unavail_box(«20», «30»)
2079 vas_pmem_hline(«80»)
2081 vas_arrow(«5», «15»)
2082 vas_arrow(«25», «65»)
2083 vas_arrow(«130», «90»)
2085 vas_arrow(«90», «eval(vas_pmem_height() + vas_disk_gap()
2086 + vas_disk_height() / 2)»)
2090 <p> Virtual memory is an abstraction of the available memory resources.
2091 When a process reads from or writes to a memory location, it refers
2092 to <em>virtual addresses</em> (illustrated as the left box of the
2093 diagram). Virtual addresses are mapped by the MMU to <em>physical
2094 addresses</em> which refer to physical memory locations (right
2095 box). The <em>mapped</em> virtual address space of a process is a
2096 collection of ranges of virtual addresses which correspond to physical
2097 memory addresses (blue areas). By storing less frequently-accessed
2098 chunks of virtual memory (yellow) on the swap area (grey), applications
2099 can use more memory than is physically available. In this case the
2100 size of the valid virtual addresses (blue and yellow areas together)
2101 exceeds the amount of physical memory (orange and green areas). Any
2102 attempt to access an unmapped memory location (red and yellow areas)
2103 results in a <em>page fault</em>, a hardware trap which forces the CPU
2104 back into kernel mode. The kernel then checks whether the address is
2105 valid (yellow) or invalid (red). If it is invalid, the kernel sends
2106 <code>SIGSEGV</code>, which usually terminates the process with
2107 the <code>segmentation fault</code> error. Otherwise it allocates
2108 a chunk of unused physical memory, copies the chunk from the swap
2109 area to the newly allocated memory and adjusts the mapping (i.e.,
2110 a yellow part becomes blue). The virtual memory concept increases
2111 stability and security because no process can access physical memory
2112 which belongs to the kernel or to other processes (orange areas). </p>
2114 <p> We've already seen that the <code> fork(2) </code> system call
2115 creates a new process as a duplicate of the calling process. Since
2116 the virtual address space of the calling process (a) might be large
2117 and (b) is likely to be replaced in the child by a subsequent call
2118 to an exec function, it would be both wasteful and pointless to
2119 copy the full address space of the parent process to the child. To
2120 implement <code> fork(2) </code> efficiently, operating systems
2121 employ an optimization strategy known as <em> Copy on Write </em>
2122 (CoW). The idea of CoW is that if multiple callers ask for resources
2123 which are initially indistinguishable, you can give them pointers to
2124 the same resource. This function can be maintained until a caller
2125 tries to modify its copy of the resource, at which point a true
2126 private copy is created to prevent the changes becoming visible to
2127 everyone else. The primary advantage is that if a caller never makes
2128 any modifications, no private copy needs ever be created. The <code>
2129 fork(2) </code> system call marks the pages of the virtual address
2130 space of both the parent and the child process as CoW by setting a
2131 special bit in the <em> page table entry </em> which describes the
2132 mapping between virtual and physical addresses of the MMU. As for
2133 invalid memory accesses, the attempt to write to a CoW page results
2134 in a page fault that puts the CPU back into kernel mode. The kernel
2135 then allocates a new memory page on behalf of the process, copies
2136 the contents of the page which caused the fault, changes the page
2137 table mappings for the process accordingly and returns to user space.
2138 This all happens transparently to the process. </p>
2141 define(«asl_width», «300»)
2142 define(«asl_height», «400»)
2143 define(«asl_top_margin», «10»)
2144 define(«asl_text_width», «35»)
2145 define(«asl_mem_width», «25»)
2146 define(«asl_mem_color_env», «#fc8»)
2147 define(«asl_mem_color_stack», «#8fc»)
2148 define(«asl_mem_color_empty», «#ccc»)
2149 define(«asl_mem_color_heap», «#c8f»)
2150 define(«asl_mem_color_bss», «#8cf»)
2151 define(«asl_mem_color_data», «#cf8»)
2152 define(«asl_mem_color_text», «#f8c»)
2153 define(«asl_font_size», «5»)
2155 define(«asl_arrow», «
2159 x2="eval(asl_text_width() - 2)"
2163 marker-end="url(#arrow)"
2166 define(«asl_arrow_text», «
2170 font-size="asl_font_size()"
2176 dnl $1: y0, $2; height, $3: color, $4: high arrow text
2177 dnl $5: low arrow text, $6: desc
2183 x="asl_text_width()"
2184 y="eval($1 + asl_top_margin())"
2187 width="asl_mem_width()"
2189 ifelse(«$4», «», «», «
2190 asl_arrow(«eval($1 + asl_top_margin())»)
2191 asl_arrow_text(«eval($1 + asl_top_margin() - 2)», «$4»)
2193 ifelse(«$5», «», «», «
2194 asl_arrow(«eval($1 + $2 + asl_top_margin())»)
2195 asl_arrow_text(«eval(asl_top_margin()
2196 + $1 + $2 - 2)», «$5»)
2199 x="eval(asl_text_width() + asl_mem_width() + 2)"
2200 y="eval($1 + $2 / 2 + asl_top_margin())"
2202 font-size="asl_font_size()"
2209 width="asl_width()" height="asl_height()"
2210 viewBox="0 0 100 eval(100 * asl_height() / asl_width())"
2211 xmlns="http://www.w3.org/2000/svg"
2212 xmlns:xlink="http://www.w3.org/1999/xlink"
2214 asl_box(«0», «10», «asl_mem_color_env», «2^64 - 1», «»,
2216 asl_box(«10», «15», «asl_mem_color_stack», «», «base pointer»,
2218 asl_box(«25», «30», «asl_mem_color_empty», «», «break point»,
2220 asl_box(«55», «35», «asl_mem_color_heap», «», «», «Heap»)
2221 asl_box(«90», «10», «asl_mem_color_bss», «», «», «BSS»)
2222 asl_box(«100», «10», «asl_mem_color_data», «», «», «Data»)
2223 asl_box(«110», «10», «asl_mem_color_text», «», «0», «Text»)
2227 <p> The diagram on the left illustrates the layout of the virtual
2228 address space of a process. At the top of the address space are the
2229 argument vector and the environment variables. The <em>stack</em>
2230 stores the local variables of the functions which are currently
2231 being called, plus house-keeping data like the return addresses
2232 of these functions. As more functions are called, the stack grows
2233 downwards towards the lower addresses. Its current lower end is
2234 called the <em> base pointer</em>. The other variable area of the
2235 address space is the <em>heap</em>, which contains the memory that
2236 has been allocated on behalf of the process, for example with <code>
2237 malloc(3)</code>. As the process allocates more memory, the heap grows
2238 upwards, towards the stack. The current end of the heap is called the
2239 <em> break point</em>. The lower part of the address space contains
2240 three segments of fixed size. The <em>text</em> segment contains the
2241 compiled machine instructions of the executable, the <em>data</em>
2242 segment contains the initialized variables which are already known
2243 at compile time. Finally, the <em>BSS</em> segment is allocated and
2244 zeroed at execution time. This segment contains variables which should
2245 initialized to zero at startup. Unlike the data segment it is not
2246 stored in the executable. BSS stands for "Block Started by Symbol",
2247 which is a historic name coined in the 1950s. It has no relation to
2248 the real meaning of the segment. </p>
2250 The exercises of this section invite the reader to look at the virtual
2251 address space of running processes to learn what happens when a
2252 dynamically-linked executable is being executed and how the resulting
2253 memory maps affect the virtual address space of the newly created
2259 <li> Examine your own processes with <code>htop</code>, <code>ps
2260 ux</code> and <code>pstree -panuch $LOGNAME</code>. </li>
2262 <li> Run <code>ls -l /proc/$$</code> and examine the environment of
2263 your shell process. </li>
2265 <li> Run <code>kill -l</code> and discuss the meaning of signals
2266 1-15. Use <code>signal(7)</code> as a reference. </li>
2268 <li> Create a zombie process: run <code>sleep 100&</code>. From
2269 another terminal, send <code>SIGSTOP</code> to the parent process
2270 of the sleep process (the shell), then send <code>SIGKILL</code>
2271 to the sleep process. Run <code>cat /proc/$PID/status</code> where
2272 <code>$PID</code> is the process ID of the sleep process. </li>
2274 <li> Run <code>echo $$</code> to obtain the PID of an interactive
2275 shell that is running in a terminal. Send the <code>SIGSTOP</code>
2276 and <code>SIGCONT</code> signals to this PID from another terminal
2277 and see what happens when you type in the terminal that contains the
2278 stopped shell process. </li>
2280 <li> The <code>ping(8)</code> utility catches <code>SIGQUIT</code>.
2281 In one terminal execute <code>ping localhost</code>. While this
2282 command runs in an endless loop, send <code>SIGQUIT</code> to the
2283 ping process from another terminal and see what happens. </li>
2285 <li> Read <code>kill(2)</code> to learn what <code>kill -9 -1</code>
2286 does. Run this command if you are brave. </li>
2288 <li> Why doesn't the <a href="«#»cd_script">cd script</a> work as
2291 <li> Explain the difference between the two commands <code>X=foo
2292 bar</code> and <code>X=foo; bar</code>. </li>
2294 <li> Run <code>set</code> and examine the environment variables of
2295 an interactive shell session. </li>
2298 href="https://public-inbox.org/git/Pine.LNX.4.64.0609141023130.4388@g5.osdl.org/">email</a>
2299 from Linus Torvalds about why stdio is not that simple at all. </li>
2301 <li> Run the command <code>ls / /does-not-exist</code>, redirect
2302 stdout and stderr to different files. </li>
2304 <li> Consider the following shell code which uses stdio to first write
2305 to stdout, then to stderr. <code>echo foo; echo bar 1>&2</code>. Which
2306 circumstances guarantee that the "foo" line appears before the "bar"
2307 line in the output? </li>
2309 <li> In the pipeline <code> foo | bar</code>, what is the
2310 buffering type of the file descriptor which corresponds to
2311 the <code> stdin </code> stream of the <code> bar </code>
2314 <li> Assume <code>foo</code> is a log file which increases due to
2315 some process appending data to it. Explain why the command <code>
2316 tail -f foo | while read; do echo new data; done </code> does not
2317 work as expected. Fix the flaw by changing the buffering type with
2318 <code>stdbuf(1)</code>. </li>
2320 <li> Run <code>sleep 100 > /dev/null &</code>, examine open
2321 files of the sleep process by looking at suitable files in
2322 <code>/proc</code>. Do the same with <code>sleep 100 | head
2325 <li> Run <code>ldd /bin/sh</code> and explain what happens when a
2326 shell is executed. </li>
2328 <li> On a Linux system, run <code>cat /proc/$$/maps</code> or
2329 <code>pmap -x $$</code> to see the address space layout of your
2330 shell. Check <code>Documentation/filesystems/proc.txt</code>
2331 in the linux kernel source tree for the format of
2332 <code>/proc/$$/maps</code>. </li>
2334 <li> Run <code>cat /proc/$$/smaps</code> and examine the values of
2335 the heap section. </li>
2337 <li> Assume some program allocates a lot of memory so that the size of
2338 the valid virtual addresses is 1T large. Assume further that a software
2339 bug causes the content of a pointer variable to be overwritten with
2340 random garbage. Determine the probability that this pointer variable
2341 contains a valid address (assuming a 64 bit system). </li>
2346 Explain how <code>PR_SET_CHILD_SUBREAPER</code> works and possible
2347 use-cases for this (Linux) feature.
2353 Explain in one paragraph of text the purpose of the <em>file
2354 creation mask</em> (also known as <em>umask</em>) of a process.
2360 When we said that each process runs on behalf of a user and that the
2361 ID of this user is part of the process metadata, we were simplifying
2362 matters. There are actually three different UIDs and three different
2363 GIDs: the <em>real UID</em>, the <em>effective UID</em>, and the
2364 <em>saved set-user ID</em>, and analogous for the group IDs. Explain
2365 the purpose of the three UIDs.
2372 On a multi-CPU system the performance of a program can be
2373 enhanced by allowing for multiple flows of control. This is the
2374 idea behind <em>threads</em>, which are also called <em>lightweight
2375 processes</em>. Give an overview of threads, summarize the POSIX thread
2376 API (see <code>pthreads(7)</code>) and explain how the Linux-specific
2377 <code>clone(2)</code> system call can used to implement threads.
2383 Explain what the command <code>find /etc > /dev/null</code> does,
2384 and why you get some error messages. Assume you'd like to extract
2385 only those error messages which contain the string "lvm". Explain
2386 why <code>find /etc > /dev/null | grep lvm</code> does not work as
2387 expected. Come up with a similiar command that works.
2391 The command traverses the <code>/etc</code> directory recursively and
2392 prints all files and directories it encounters during the traversal to
2393 stdout. Since stdout is redirected to the NULL device by the <code>>
2394 /dev/null</code> construct, only the stderr stream containing the error
2395 messages makes it to the terminal. This includes all subdirectories
2396 of <code>/etc</code> which cannot be traversed due to insufficient
2397 permissions (no "r" bit set). The proposed <code>find | grep</code>
2398 command does not work since the <code>|</code> operator is evaluated
2399 <em>before</em> any redirections specified by the find command
2400 take place. More precisely, stdout of the find process is redirected
2401 <em>twice</em>: First to one end of the pipe due to the <code>|</code>,
2402 then to the NULL device due to the <code>> /dev/null</code>. The
2403 last redirection "wins", so the <code>grep</code> process does not
2404 see any input. The command <code>find /etc 2>&1 > /dev/null | grep
2405 lvm</code> works. The following four redirections take place: First
2406 stdout of the <code>find</code> process and stdin of <code>grep</code>
2407 process are redirected to the two ends of the pipe. Next, due to
2408 the <code>2>&1</code> the stderr stream of the <code>find</code>
2409 process is redirected to the current destination of stdout, i.e.,
2410 to the pipe. Finally the <code>> /dev/null</code> redirects stdout
2411 of the find process to the NULL device. Hence error messages go to
2412 the pipe and are processed by <code>grep</code> as desired.
2417 Run <code>ulimit -n</code> to see the maximal number of file descriptors you
2418 are allowed to create. Explain what this limit means with respect
2419 to multiple processes, multiple logins, and the <code>fork(2</code>) system
2420 call. Write a program in your language of choice which creates file
2421 descriptors in a loop until it fails due to the file descriptor
2422 limit. Then print the number of file descriptors the program was able
2425 On our systems the limit is set to 1024. This means a single process
2426 can only have this many files open at any given time. Independent
2427 processes (like those coming from different login sessions) have no
2428 common file descriptors, even though they may open the same files. In
2429 this sense the file descriptor limit is a per-process limit. However,
2430 when a process calls <code>«fork(</code>») to create a new process, the new
2431 process inherits all open file descriptors from the parent. This can
2432 lead to the situation where a newly created process is unable to open
2433 <em>any</em> files. This property was actually used to break computer
2434 security. The <code>«O_CLOEXEC»</code> flag was introduced not too long
2435 ago to deal with this problem. See <code>open(2</code>) for details.
2437 C program that opens the maximal possible number of file descriptors:
2444 for (i == 0; open("/dev/null", O_RDONLY) >= 0; i++)
2446 printf("opened %d file descriptors\n", i);
2454 Search the web for the document called
2455 <code>vm/overcommit-accounting</code>. Discuss the pros and cons of
2456 the three possible overcommit handling modes.
2464 href="https://utcc.utoronto.ca/~cks/space/blog/unix/MemoryOvercommit">blog
2465 posting</a> on the virtual memory overcommit issue. Explain the
2466 catch-22 situation described there in no more than two sentences.
2472 Describe, in a single paragraph of text, what a virtual dynamic
2473 shared object (VDSO) is and which type of applications benefit most
2480 Describe the concept of <em> huge pages </em> and the Linux-specific
2481 implementation of <em> transparent </em> huge pages. Discuss the pros
2482 and cons of huge tables and explain the workloads which would benefit
2483 from a setup with huge pages enabled.
2489 <li> Explain the concept of <em>address space layout randomization</em>
2492 <li> Run <code>bash -c</code> '<code>cat /proc/$$/maps</code>'
2493 repeatedly to see address space layout randomization in action. Discuss
2494 the pros and cons of ASLR. </li>
2500 SUBSECTION(«cd_script»)
2504 echo "changing CWD to $1"
2508 SUBSECTION(«hello_world»)
2515 SUBSECTION(«symlink_madness»)
2521 ln -s ../foo foo/testdir
2522 ls -l foo/a foo/testdir/a foo/testdir/testdir/a
2525 SECTION(«Further Reading»)
2528 href="http://www.catb.org/~esr/writings/taoup/html/ch02s01.html">Origins
2529 and History of Unix, 1969-1995</a> by Eric Steven Raymond. </li>
2532 href="https://www.newyorker.com/business/currency/the-gnu-manifesto-turns-thirty">
2533 The GNU Manifesto Turns Thirty</a>, by Maria Bustillos. </li>
2535 <li> <a href="https://lwn.net/Articles/924577/">The SCO lawsuit,
2536 20 years later</a> </li>
2539 href="http://www.catb.org/~esr/writings/unix-koans/end-user.html">
2540 The Koan of Master Foo and the End User</a>. </li>
2542 <li> <a href="https://lwn.net/Articles/411845/">Ghosts of Unix Past:
2543 a historical search for design patterns</a>, by Neil Brown. </li>
2545 <li> W. Richard Stevens: Advanced Programming in the Unix
2546 Environment. Addison Wesley. </li>