From: Andre Noll Date: Thu, 13 May 2010 18:13:02 +0000 (+0200) Subject: Merge remote branch 'fml/master' X-Git-Tag: v0.4.3~23 X-Git-Url: http://git.tuebingen.mpg.de/?p=paraslash.git;a=commitdiff_plain;h=840725e10602abd8187428924040f4bf3e04594c;hp=7cb8fa26cfdcb3d34f8be2b86c48ffc6d7b9d84a Merge remote branch 'fml/master' --- diff --git a/INSTALL b/INSTALL index aba5c35f..0f4c18b6 100644 --- a/INSTALL +++ b/INSTALL @@ -1,258 +1,13 @@ -INSTALL -======= - ----- Any knowledge of how to work with mouse and icons is not required. ---------------------------- -Install all needed packages ---------------------------- -See -<< - REQUIREMENTS ->> -for a list of required software. You don't need everything listed -there. In particular, mp3, ogg vorbis and aac support are all -optional. The configure script will detect what is installed on your -system and will only try to build those executables that can be built -with your setup. - -Note that no special library (not even the mp3 decoding library -libmad) is needed for para_server if you only want to stream mp3 or -wma files. Also, it's fine to use para_server on a box without sound -card as para_server only sends the audio stream to connected clients. - -------------------------- -Install server and client -------------------------- - -Install the paraslash package on all machines, you'd like this software -to run on: - - (./configure && make) > /dev/null - -There should be no errors but probably some warnings about missing -software packages which usually implies that not all audio formats will -be supported. If headers or libs are installed at unusual locations -you might need to tell the configure script where to find them. Try - - ./configure --help - -to see a list of options. If the paraslash package was compiled -successfully, execute as root, - - make install - ------------------------------------ -Setup user list and create RSA keys ------------------------------------ - -Note that the RSA keys for paraslash 0.3.x will not work for version -0.4.x as the new version requires stronger (2048 bit) keys. If you -already have your 2048 bit keys, skip this step. If you are new to -paraslash, you have to generate a key pair for each user you want to -allow to connect. You need at least one user. - -Let's assume that you'd like to run the server on host server_host -as user foo, and that you want to connect from client_host as user bar. - -As foo@server_host, create ~/.paraslash/server.users by typing the -following commands: - - user=bar - target=~/.paraslash/server.users - key=~/.paraslash/key.pub.$user - perms=AFS_READ,AFS_WRITE,VSS_READ,VSS_WRITE - mkdir -p ~/.paraslash - echo "user $user $key $perms" >> $target - -This gives "bar" the full privileges. - -Change to the "bar" account on client_host and generate the key-pair -with the commands - - key=~/.paraslash/key.$LOGNAME - mkdir -p ~/.paraslash - (umask 077 && openssl genrsa -out $key 2048) - -Next, extract its public part: - - pubkey=~/.paraslash/key.pub.$LOGNAME - openssl rsa -in $key -pubout -out $pubkey - -and copy the public key just created to server_host (you may -skip this step for a single-user setup, i.e. if foo=bar and -server_host=client_host): - - scp $pubkey foo@server_host:.paraslash/ - -Finally, tell para_client to connect to server_host: - - conf=~/.paraslash/client.conf - echo 'hostname server_host' > $conf - ------------------ -Start para_server ------------------ - -Before starting the server make sure you have write permissions to -the directory /var/paraslash. - - sudo chown $LOGNAME /var/paraslash - -Alternatively, use the --afs_socket Option to specify a different -location for the afs command socket. - -For this first try, we'll use the info loglevel to make the output -of para_server more verbose. - - para_server -l info - -Now you can use para_client to connect to the server and issue -commands. Open a new shell (as "bar" on "client_host" in the above -example) and try - - para_client help - para_client si - -to retrieve the list of available commands and some server info. -Don't proceed if this doesn't work. - -------------------- -Create the database -------------------- - - para_client init - -This creates some empty tables under ~/.paraslash/afs_database-0.4. -You normally don't need to look at these tables, but it's good to -know that you can start from scratch with - - rm -rf ~/.paraslash/afs_database-0.4 - -in case something went wrong. - -Next, you need to fill the audio file table of that database with -contents so that para_server knows about your audio files. Choose an -absolute path to a directory containing some audio files and add them -to the audio file table: - - para_client add /my/mp3/dir - -This might take a while, so it is a good idea to start with a directory -containing not too many audio files. Note that the table only contains -data about the audio files found, not the files themselves. - -Print a list of all audio files found with - - para_client ls - ------------------------- -Start streaming manually ------------------------- - - para_client play - para_client -- stat -n=2 - -This starts streaming and dumps some information about the current -audio file to stdout. - -You should now be able to receive the stream and listen to it. If -you have mpg123 or xmms handy, execute on client_host - - mpg123 http://server_host:8000/ -or - xmms http://server_host:8000/ - -Paraslash comes with its own receiving and playing software, which -will be described next. Try the following on client_host (assuming -Linux/ALSA and an mp3 stream): - - para_recv -r 'http -i server_host' > file.mp3 - # (interrupt with CTRL+C after a few seconds) - ls -l file.mp3 # should not be empty - para_filter -f mp3dec -f wav < file.mp3 > file.wav - ls -l file.wav # should be much bigger than file.mp3 - para_write -w alsa < file.wav - -If this works, proceed. Otherwise double check what is logged by -para_server and use the --loglevel option of para_recv, para_filter -and para_write to increase verbosity. - -Next, put the pieces together: - - para_recv -r 'http -i server_host' \ - | para_filter -f mp3dec -f wav \ - | para_write -w alsa - ---------------------- -Configure para_audiod ---------------------- - -In order to automatically start the right decoder at the right time -and to offer to the clients some information on the current audio -stream and on paraslash's internal state, you should run the local -audio daemon, para_audiod, on every machine in your network which is -supposed to play the audio stream. Try - - para_audiod -h - -for help. Usually you have to specify only server_host as the receiver -specifier for each supported audio format, like this: - - para_audiod -l info -r 'mp3:http -i server_host' - -The preferred way to use para_audiod is to run it once at system start -as an unprivileged user. para_audiod needs to create a "well-known" -socket for the clients to connect to. The default path for this -socket is - - /var/paraslash/audiod_socket.$HOSTNAME - -so the /var/paraslash directory should be writable for the user who -runs para_audiod. - -If you want to change the location of the socket, use the --socket -option for para_audiod or the config file ~/.paraslash/audiod.conf -to change the default. Note that in this case you'll also have to -specify the same value for para_audioc's --socket option. - -If para_server is playing, you should be able to listen to the audio -stream as soon as para_audiod is started. Once it is running, try - - para_audioc stat - -That should dump some information to stdout. Other commands include +From tarball: - para_audioc off - para_audioc on - para_audioc sb - para_audioc term - para_audioc cycle + ./configure && make && sudo make install --------------- -Start para_gui --------------- +From git: -para_gui reads the output of "para_audioc stat" and displays that -information in a curses window. It also allows you to bind keys to -arbitrary commands. There are several flavours of key-bindings: + ./autogen.sh && sudo make install - - internal: These are the built-in commands that can not be - changed (help, quit, loglevel, version...). - - external: Shutdown curses before launching the given command. - Useful for starting other ncurses programs from within - para_gui, e.g. aumix or dialog scripts. Or, use the mbox - output format to write a mailbox containing one mail for each - (admissible) file the audio file selector knows about. Then - start mutt from within para_gui to browse your collection! - - display: Launch the command and display its stdout in - para_gui's bottom window. - - para: Like display, but start "para_client " instead of "". +For details see the user manual: -This concludes the installation notes. Next thing you might to have a look -at is how to use paraslash's audio file selector. See -<< - README.afs ->> + http://paraslash.systemlinux.org/manual.html diff --git a/NEWS b/NEWS index 227bf7fb..c17a543b 100644 --- a/NEWS +++ b/NEWS @@ -6,6 +6,8 @@ NEWS --------------------------------------------- - Fix an end-of-file detection bug in the oggdec filter. + - New user manual + - The new nonblock API ------------------------------------------ 0.4.2 (2010-04-23) "associative expansion" diff --git a/README b/README index 883eeee1..e991b734 100644 --- a/README +++ b/README @@ -6,162 +6,10 @@ Paraslash is an acronym for _Play, archive, rate and stream large audio sets happily_ -It contains the following programs: +The paraslash package contains server and client software for +network streaming as well as stand-alone utilities for decoding +mp3, ogg, aac and wma files. See the user manual for details. ------------ -para_server ------------ - -para_server streams binary audio data (mp3/oggvorbis/m4a/wma files) -over local and/or remote networks. It listens on a tcp port and -accepts commands such as play, stop, pause, next from authenticated -clients. However, there are many more commands. - -It supports three built-in network streaming methods (senders): http, dccp, -or udp. - - * The http sender is recommended for public streams that can be played - by any player like mpg123, xmms, itunes, winamp... - - * The dccp sender requires kernel support for the datagram congestion - control protocol. - - * The udp sender is recommended for multicast LAN streaming. - -It is possible to activate more than one sender simultaneously. - -The built-in audio file selector of paraslash is used to manage your -audio files. It maintains statistics on the usage of all available audio -files such as last played time, and the number of times each file was -selected. - -Its features include - - * attributes. Allow fine-grained audio file selection. - - * image table. For storing e.g. album cover art. - - * lyrics table. For storing lyrics. - - * playlist table. Stores arbitrary many playlists for later use. - - * mood mode. Audio file selection works by specifying mood - methods involving attributes, pattern matching for file names - and more. This allows rather sophisticated configurations - and is explained in more detail in -<< - README.afs ->> - - * rename detection. If files are moved or renamed, afs will - recognize them despite of this change. - ------------ -para_client ------------ - -The client program to connect to para_server. paraslash commands -are sent to para_server and the response is dumped to stdout. This -can be used by any scripting language to produce user interfaces with -little programming effort. - -All connections between para_server and para_client are encrypted by -default. For each user of paraslash you must create a public/secret -RSA key pair for authentication. The authenticated connection is -encrypted with a symmetric rc4 session key. - ---------- -para_recv ---------- - -A command line http/dccp/udp stream grabber. The http mode of this -tool can be used to receive data from any http streaming source. - ------------ -para_filter ------------ - -A filter program that converts from stdin and writes to stdout. - -para_filter combines several decoders (mp3, oggvorbis, aac, wma), -a volume normalizer and a cpuple of other filters. New filters can -be added easily. It is possible to "chain" any number of filters, -like UNIX pipes. - -para_filter does not depend on other parts of paraslash, so it can -be used as a stand-alone command line tool for audio decoding and -volume normalization. - --------- -para_afh --------- - -A small stand-alone program that prints tech info about the given -audio file to stdout. It can be instructed to print a "chunk table", -an array of offsets within the audio file or to write the content of -the audio file in complete chunks 'just in time'. - -This allows third-party streaming software that is unaware of -the particular audio format to send complete frames in real -time. Currently, mp3, ogg vorbis, aac and wma are supported. - ----------- -para_write ----------- - -A modular audio stream writer. It supports a simple file writer -output plug-in and optional wav/raw players for ALSA (Linux) and for -coreaudio (Mac OS). para_write can also be used as a stand-alone wav -or raw audio player. - ------------ -para_audiod ------------ - -The local daemon that collects information from para_server. - -It runs on the client side and connects to para_server. As soon as -para_server announces the availability of an audio stream, para_audiod -starts an appropriate receiver, any number of filters and a paraslash -writer to play the stream. It is possible to capture the stream at -any position in the filter chain. - -Moreover, para_audiod listens on a local socket and sends status -information about para_server and para_audiod to local clients on -request. Access via this local socket may be restricted by using Unix -socket credentials, if available. - ------------ -para_audioc ------------ - -The client program which talks to para_audiod. Used to control -para_audiod, to receive status info, or to grab the stream at any -point in the filter chain. - -para_audioc (hence para_audiod) is needed by para_gui see below. - --------- -para_gui --------- - -Themable ncurses-based gui. It calls para_audioc and presents -the obtained information in an ncurses window. para_gui provides -key-bindings for the most common commands and new key-bindings can -be added easily. - ---------- -para_fade ---------- - -A (oss-only) alarm clock and volume-fader. - ---------------- -bash_completion ---------------- - -A small bash script for inclusion in ~/.bashrc. It gives you command -line completion for some paraslash commands. ------- LICENSE diff --git a/README.afs b/README.afs deleted file mode 100644 index 51625667..00000000 --- a/README.afs +++ /dev/null @@ -1,279 +0,0 @@ -The audio file selector -======================= - -Paraslash comes with a sophisticated audio file selector called *afs*. -In the -<< -installation notes, ->> -only the "dummy" mode of afs was used which gets activated automatically if -nothing else was specified. In this section the various features of afs are -described. - ----------- -Attributes -~~~~~~~~~~ - -An attribute is simply a bit which can be set for each audio -file individually. Up to 64 different attributes may be -defined. For example, "pop", "rock", "blues", "jazz", "instrumental", -"german_lyrics", "speech", whatever. It's up to you how many attributes -you define and how you call them. - -A new attribute "test" is created by - - para_client addatt test -and - para_client lsatt - -lists all available attributes. You can set the "test" attribute for -an audio file by executing - - para_client setatt test+ /path/to/the/audio/file - -Similarly, the "test" bit can be removed from an audio file with - - para_client setatt test- /path/to/the/audio/file - -Instead of a path you may use a shell wildcard pattern. The attribute -is applied to all audio files matching that pattern: - - para_client setatt test+ '/test/directory/*' - -The command - - para_client -- ls -lv - -gives you a verbose listing of your audio files which contains also -which attributes are set. - -In case you wonder why the double-dash in the above command is needed: -It tells para_client to not interpret the options after the dashes. If -you find this annoying, just say - - alias para='para_client --' - -and be happy. In what follows we shall use this alias. - -The "test" attribute can be dropped from the database with - - para rmatt test - -Read the output of - - para help ls - para help setatt - -for more information and a complete list of command line options to -these commands. - - ----------------------- -Abstract mood nonsense -~~~~~~~~~~~~~~~~~~~~~~ - -[skip this part if you don't like formal definitions] - -A mood consists of a unique name and its *mood definition*, which is -a set of *mood lines* containing expressions in terms of attributes -and other data contained in the database. - -A mood defines a subset of audio files called the *admissible audio files* -for that mood. At any time, at most one mood can be *active* which -means that para_server is going to select only files from that subset -of admissible files. - -So in order to create a mood definition one has to write a set of -mood lines. Mood lines come in three flavours: Accept lines, deny -lines and score lines. - -The general syntax of the three types of mood lines is - - - accept [with score ] [if] [not] [options] - deny [with score ] [if] [not] [options] - score [if] [not] [options] - - -Here is either an integer or the string "random" which assigns -a random score to all matching files. The score value changes the -order in which admissible files are going to be selected, but is of -minor importance for this introduction. - -So we concentrate on the first two forms, i.e. accept and deny -lines. As usual, everything in square brackets is optional, i.e. -accept/deny lines take the following form when ignoring scores: - - accept [if] [not] [options] - -and analogously for the deny case. The "if" keyword is purely cosmetic -and has no function. The "not" keyword just inverts the result, so -the essence of a mood line is the mood method part and the options -following thereafter. - -A *mood method* is realized as a function which takes an audio file -and computes a number from the data contained in the database. -If this number is non-negative, we say the file *matches* the mood -method. The file matches the full mood line if it either - - - matches the mood method and the "not" keyword is not given, -or - - does not match the mood method, but the "not" keyword is given. - -The set of admissible files for the whole mood is now defined as those -files which match at least one accept mood line, but no deny mood line. -More formally, an audio file F is admissible if and only if - - (F ~ AL1 or F ~ AL2...) and not (F ~ DL1 or F ~ DN2 ...) - -where AL1, AL2... are the accept lines, DL1, DL2... are the deny -lines and "~" means "matches". - -The cases where no mood lines of accept/deny type are defined need -special treatment: - - - Neither accept nor deny lines: This treats all files as admissible - (in fact, that is the definition of the dummy mood which is activated - automatically if no moods are available). - - - Only accept lines: A file is admissible iff it matches at least one - accept line: - - F ~ AL1 or F ~ AL2 or ... - - - Only deny lines: A file is admissible iff it matches no deny line: - - not (F ~ DL1 or F ~ DN2 ...) - - - --------------------- -List of mood_methods -~~~~~~~~~~~~~~~~~~~~ - - no_attributes_set - -Takes no arguments and matches an audio file if and only if no -attributes are set. - - is_set - -Takes the name of an attribute and matches iff that attribute is set. - - path_matches - -Takes a filename pattern and matches iff the path of the audio file -matches the pattern. - - artist_matches - album_matches - title_matches - comment_matches - -Takes an extended regular expression and matches iff the text of the -corresponding tag of the audio file matches the pattern. If the tag -is not set, the empty string is matched against the pattern. - - year ~ - bitrate ~ - frequency ~ - channels ~ - num_played ~ - -Takes a comparator ~ of the set {<, =, <=, >, >=, !=} and a number -. Matches an audio file iff the condition ~ is -satisfied where val is the corresponding value of the audio file -(value of the year tag, bitrate in kbit/s, frequency in Hz, channel -count, play count). - -The year tag is special as its value is undefined if the audio file -has no year tag or the content of the year tag is not a number. Such -audio files never match. Another difference is the special treatment -if the year tag is a two-digit number. In this case either 1900 or -2000 are added to the tag value depending on whether the number is -greater than 2000 plus the current year. - - ----------- -Mood usage -~~~~~~~~~~ - -To create a new mood called "my_mood", write its definition into -some temporary file, say "tmpfile", and add it to the mood table -by executing - - para addmood my_mood < tmpfile - -If the mood definition is really short, you may just pipe it to the -client instead of using temporary files. Like this: - - echo "$MOOD_DEFINITION" | para addmood my_mood - -There is no need to keep the temporary file since you can always use -the catmood command to get it back: - - para catmood my_mood - -A mood can be activated by executing - - para select m/my_mood - -Once active, the list of admissible files is shown by the ls command -if the "-a" switch is given: - - para ls -a - ------------------------ -Example mood definition -~~~~~~~~~~~~~~~~~~~~~~~ - -Suppose you have defined attributes "punk" and "rock" and want to define -a mood containing only Punk-Rock songs. That is, an audio file should be -admissible if and only if both attributes are set. Since - - punk and rock - -is obviously the same as - - not (not punk or not rock) - -(de Morgan's rule), a mood definition that selects only Punk-Rock -songs is - - deny if not is_set punk - deny if not is_set rock - - ---------- -Troubles? ---------- - -Use the debug loglevel (option -l debug for most commands) to show -debugging info. Almost all paraslash executables have a brief online -help which is displayed by using the -h switch. The --detailed-help -option prints the full help text. - -If para_server crashed or was killed by SIGKILL (signal 9), it -may refuse to start again because of "dirty osl tables". In this -case you'll have to run the oslfsck program of libosl to fix your -database. It might be necessary to use --force (even if your name -isn't Luke). However, make sure para_server isn't running before -executing oslfsck --force. - -If you don't mind to recreate your database you can start -from scratch by removing the entire database directory, i.e. - - rm -rf ~/.paraslash/afs_database-0.4 - -Be aware that this removes all attribute definitions, all playlists -and all mood definitions. - -Although oslfsck fixes inconsistencies in database tables it doesn't -care about the table contents. To check for invalid table contents, use - - para_client check - -This prints out references to missing audio files as well as invalid -playlists and mood definitions. - -Still having problems? mailto: Andre Noll diff --git a/REQUIREMENTS b/REQUIREMENTS deleted file mode 100644 index a9aedcda..00000000 --- a/REQUIREMENTS +++ /dev/null @@ -1,61 +0,0 @@ -Requirements -============ - -In any case you'll need - - - *libosl*, the object storage layer: Used by para_server. It is - available from http://git.tuebingen.mpg.de/osl. Alternatively, - execute "git clone git://git.tuebingen.mpg.de/osl". - - - *gcc*, the gnu compiler collection (shipped with distro): gcc-3.3 - or newer is required. - - - *gnu make* (shipped with disto, might be called gmake on BSD systems) - - - *bash* (most likely already installed) - - - *openssl* (needed by server, client): usually shipped with - distro, but you might have to install the "development package" - (called libssl-dev on debian systems) as well: - http://www.openssl.org/ - - - *help2man* (for man page creation) ftp://ftp.gnu.org/pub/gnu/help2man - -Optional features: - - - *mp3*: The mp3 decoder of para_filter is based on libmad: - http://www.underbit.com/products/mad/. If you prefer to - use the libmad package provided by your distributor, make - sure to install the corresponding development package as - well. It is called libmad0-dev on debian-based systems. - Note that libmad is not necessary for the server side, - i.e. for sending mp3 files. - - - *id3 tags*: - For version-2 id3 tag support, you'll need libid3tag which - is also available through the above link (alternatively: - install package libid3tag0-dev). Without libid3tag, only - version one tags are recognized. - - - *ogg vorbis*: For ogg vorbis streams you'll need libogg, - libvorbis, libvorbisfile: http://www.xiph.org/downloads/. - The corresponding Debian packages are called libogg-dev - libvorbis-dev, other distributors chose similar names. - - - *aac*: - For aac files (m4a) you'll need libfaad. Get it at - http://www.audiocoding.com/. - Debian package: libfaad-dev. - - - On Linux, you'll need to have ALSA's development package - installed. The Debian package is called libasound2-dev. - -Hacking the source: - - - gengetopt: ftp://ftp.gnu.org/pub/gnu/gengetopt/ - - autoconf: ftp://ftp.gnu.org/pub/gnu/autoconf/ - - git http://git.or.cz/ - - grutatxt http://www.triptico.com/software/grutatxt.html - - doxygen http://www.stack.nl/~dimitri/doxygen/ - - global ftp://ftp.gnu.org/pub/gnu/global - - m4: ftp://ftp.gnu.org/pub/gnu/m4/ diff --git a/afs.c b/afs.c index d738c3d9..2b748f25 100644 --- a/afs.c +++ b/afs.c @@ -709,15 +709,16 @@ static void signal_pre_select(struct sched *s, struct task *t) static void afs_signal_post_select(struct sched *s, struct task *t) { - struct signal_task *st = container_of(t, struct signal_task, task); + int signum; + if (getppid() == 1) { PARA_EMERG_LOG("para_server died\n"); goto shutdown; } - if (!FD_ISSET(st->fd, &s->rfds)) + signum = para_next_signal(&s->rfds); + if (signum == 0) return; - st->signum = para_next_signal(); - if (st->signum == SIGHUP) { + if (signum == SIGHUP) { close_afs_tables(); parse_config_or_die(1); t->error = open_afs_tables(); @@ -726,7 +727,7 @@ static void afs_signal_post_select(struct sched *s, struct task *t) init_admissible_files(current_mop); return; } - PARA_EMERG_LOG("terminating on signal %d\n", st->signum); + PARA_EMERG_LOG("terminating on signal %d\n", signum); shutdown: sched_shutdown(); t->error = -E_AFS_SIGNAL; @@ -842,57 +843,56 @@ static int call_callback(int fd, int query_shmid) return shm_detach(query_shm); } -static int execute_server_command(void) +static int execute_server_command(fd_set *rfds) { char buf[8]; - int ret = recv_bin_buffer(server_socket, buf, sizeof(buf) - 1); + size_t n; + int ret = read_nonblock(server_socket, buf, sizeof(buf) - 1, rfds, &n); - if (ret <= 0) { - if (!ret) - ret = -ERRNO_TO_PARA_ERROR(ECONNRESET); - goto err; - } - buf[ret] = '\0'; - PARA_DEBUG_LOG("received: %s\n", buf); - ret = -E_BAD_CMD; + if (ret < 0 || n == 0) + return ret; + buf[n] = '\0'; if (strcmp(buf, "new")) - goto err; - ret = open_next_audio_file(); -err: - return ret; + return -E_BAD_CMD; + return open_next_audio_file(); } -static void execute_afs_command(int fd, uint32_t expected_cookie) +/* returns 0 if no data available, 1 else */ +static int execute_afs_command(int fd, fd_set *rfds, uint32_t expected_cookie) { uint32_t cookie; int query_shmid; char buf[sizeof(cookie) + sizeof(query_shmid)]; - int ret = recv_bin_buffer(fd, buf, sizeof(buf)); + size_t n; + int ret = read_nonblock(fd, buf, sizeof(buf), rfds, &n); if (ret < 0) goto err; - if (ret != sizeof(buf)) { + if (n == 0) + return 0; + if (n != sizeof(buf)) { PARA_NOTICE_LOG("short read (%d bytes, expected %lu)\n", ret, (long unsigned) sizeof(buf)); - return; + return 1; } cookie = *(uint32_t *)buf; if (cookie != expected_cookie) { - PARA_NOTICE_LOG("received invalid cookie(got %u, expected %u)\n", + PARA_NOTICE_LOG("received invalid cookie (got %u, expected %u)\n", (unsigned)cookie, (unsigned)expected_cookie); - return; + return 1; } query_shmid = *(int *)(buf + sizeof(cookie)); if (query_shmid < 0) { PARA_WARNING_LOG("received invalid query shmid %d)\n", query_shmid); - return; + return 1; } ret = call_callback(fd, query_shmid); if (ret >= 0) - return; + return 1; err: PARA_NOTICE_LOG("%s\n", para_strerror(-ret)); + return 1; } /** Shutdown connection if query has not arrived until this many seconds. */ @@ -905,20 +905,16 @@ static void command_post_select(struct sched *s, struct task *t) struct afs_client *client, *tmp; int fd, ret; - if (FD_ISSET(server_socket, &s->rfds)) { - ret = execute_server_command(); - if (ret < 0) { - PARA_EMERG_LOG("%s\n", para_strerror(-ret)); - sched_shutdown(); - return; - } + ret = execute_server_command(&s->rfds); + if (ret < 0) { + PARA_EMERG_LOG("%s\n", para_strerror(-ret)); + sched_shutdown(); + return; } - /* Check the list of connected clients. */ list_for_each_entry_safe(client, tmp, &afs_client_list, node) { - if (FD_ISSET(client->fd, &s->rfds)) - execute_afs_command(client->fd, ct->cookie); - else { /* prevent bogus connection flooding */ + ret = execute_afs_command(client->fd, &s->rfds, ct->cookie); + if (ret == 0) { /* prevent bogus connection flooding */ struct timeval diff; tv_diff(now, &client->connect_time, &diff); if (diff.tv_sec < AFS_CLIENT_TIMEOUT) @@ -930,14 +926,11 @@ static void command_post_select(struct sched *s, struct task *t) free(client); } /* Accept connections on the local socket. */ - if (!FD_ISSET(ct->fd, &s->rfds)) - return; - ret = para_accept(ct->fd, &unix_addr, sizeof(unix_addr)); - if (ret < 0) { + ret = para_accept(ct->fd, &s->rfds, &unix_addr, sizeof(unix_addr), &fd); + if (ret < 0) PARA_NOTICE_LOG("%s\n", para_strerror(-ret)); + if (ret <= 0) return; - } - fd = ret; ret = mark_fd_nonblocking(fd); if (ret < 0) { PARA_NOTICE_LOG("%s\n", para_strerror(-ret)); diff --git a/audioc.c b/audioc.c index 3134faa1..f91f41ff 100644 --- a/audioc.c +++ b/audioc.c @@ -71,7 +71,7 @@ int main(int argc, char *argv[]) { int ret = -E_AUDIOC_SYNTAX, fd; char *cf, *buf = NULL, *args; - size_t bufsize, loaded = 0; + size_t bufsize; if (audioc_cmdline_parser(argc, argv, &conf)) goto out; @@ -93,72 +93,33 @@ int main(int argc, char *argv[]) args = conf.inputs_num? concat_args(conf.inputs_num, conf.inputs) : para_strdup("stat"); - bufsize = conf.bufsize_arg; - buf = para_malloc(bufsize); - if (conf.socket_given) { + if (conf.socket_given) ret = create_remote_socket(conf.socket_arg); - } else { - char *hn = para_hostname(), - *socket_name = make_message("/var/paraslash/audiod_socket.%s", hn); - + else { + char *hn = para_hostname(), *socket_name = make_message( + "/var/paraslash/audiod_socket.%s", hn); ret = create_remote_socket(socket_name); free(hn); free(socket_name); } - if (ret < 0) + if (ret < 0) { + PARA_EMERG_LOG("failed to create remote socket\n"); goto out; + } fd = ret; - ret = mark_fd_nonblocking(fd); - if (ret < 0) - goto out; - ret = mark_fd_nonblocking(STDOUT_FILENO); - if (ret < 0) - goto out; ret = send_cred_buffer(fd, args); if (ret < 0) goto out; - for (;;) { - int max_fileno = -1, check_write = 0; - ssize_t len; - fd_set rfd, wfd; - FD_ZERO(&rfd); - FD_ZERO(&wfd); - if (loaded < bufsize) - para_fd_set(fd, &rfd, &max_fileno); - if (loaded > 0) { - para_fd_set(STDOUT_FILENO, &wfd, &max_fileno); - check_write = 1; - } - ret = -E_AUDIOC_OVERRUN; - if (max_fileno < 0) - goto out; - ret = para_select(max_fileno + 1, &rfd, &wfd, NULL); - if (ret < 0) - goto out; - if (loaded < bufsize && FD_ISSET(fd, &rfd)) { - len = recv_bin_buffer(fd, buf + loaded, - bufsize - loaded); - if (len <= 0) { - ret = len < 0? -E_AUDIOC_READ : 0; - goto out; - } - loaded += len; - } - if (check_write && FD_ISSET(STDOUT_FILENO, &wfd)) { - ret = write(STDOUT_FILENO, buf, loaded); - if (ret < 0) { - ret = -E_AUDIOC_WRITE; - goto out; - } - loaded -= ret; - if (loaded && ret) - memmove(buf, buf + ret, loaded); - } - } + bufsize = conf.bufsize_arg; + buf = para_malloc(bufsize); + do { + size_t n = ret = recv_bin_buffer(fd, buf, bufsize); + if (ret <= 0) + break; + ret = write_all(STDOUT_FILENO, buf, &n); + } while (ret >= 0); out: - if (!ret && loaded && buf) - ret = write(STDOUT_FILENO, buf, loaded); if (ret < 0) PARA_ERROR_LOG("%s\n", para_strerror(-ret)); return ret < 0? EXIT_FAILURE : EXIT_SUCCESS; diff --git a/audiod.c b/audiod.c index 4a4a2ae7..778318ce 100644 --- a/audiod.c +++ b/audiod.c @@ -980,19 +980,16 @@ static void signal_pre_select(struct sched *s, struct task *t) para_fd_set(st->fd, &s->rfds, &s->max_fileno); } -static void signal_post_select(struct sched *s, struct task *t) +static void signal_post_select(struct sched *s, __a_unused struct task *t) { - struct signal_task *st = container_of(t, struct signal_task, task); - - if (!FD_ISSET(st->fd, &s->rfds)) - return; + int signum; - st->signum = para_next_signal(); - switch (st->signum) { + signum = para_next_signal(&s->rfds); + switch (signum) { case SIGINT: case SIGTERM: case SIGHUP: - PARA_EMERG_LOG("terminating on signal %d\n", st->signum); + PARA_EMERG_LOG("terminating on signal %d\n", signum); clean_exit(EXIT_FAILURE, "caught deadly signal"); } } @@ -1023,9 +1020,7 @@ static void command_post_select(struct sched *s, struct task *t) last_status_dump = *now; } - if (!FD_ISSET(ct->fd, &s->rfds)) - return; - ret = handle_connect(ct->fd); + ret = handle_connect(ct->fd, &s->rfds); if (ret < 0) PARA_ERROR_LOG("%s\n", para_strerror(-ret)); audiod_status_dump(); diff --git a/audiod.h b/audiod.h index f0cb6cae..44b430c8 100644 --- a/audiod.h +++ b/audiod.h @@ -70,7 +70,7 @@ extern struct audiod_args_info conf; extern int audiod_status; void __noreturn clean_exit(int status, const char *msg); -int handle_connect(int accept_fd); +int handle_connect(int accept_fd, fd_set *rfds); void audiod_status_dump(void); char *get_time_string(int slot_num); struct btr_node *audiod_get_btr_root(void); diff --git a/audiod_command.c b/audiod_command.c index 8024ec56..ce1aff68 100644 --- a/audiod_command.c +++ b/audiod_command.c @@ -421,31 +421,32 @@ static int check_perms(uid_t uid) } /** - * handle arriving connections on the local socket + * Handle arriving connections on the local socket. * - * \param accept_fd the fd to call accept() on + * \param accept_fd The fd to accept connections on. + * \param rfds If \a accept_fd is not set in \a rfds, do nothing. * - * This is called whenever para_audiod's main task detects an incoming - * connection by the readability of \a accept_fd. This function reads the - * command sent by the peer, checks the connecting user's permissions by using - * unix socket credentials (if supported by the OS) and calls the corresponding - * command handler if permissions are OK. + * This is called in each iteration of the select loop. If there is an incoming + * connection on \a accept_fd, this function reads the command sent by the peer, + * checks the connecting user's permissions by using unix socket credentials + * (if supported by the OS) and calls the corresponding command handler if + * permissions are OK. * - * \return positive on success, negative on errors + * \return Positive on success, negative on errors, zero if there was no + * connection to accept. * * \sa para_accept(), recv_cred_buffer() * */ -int handle_connect(int accept_fd) +int handle_connect(int accept_fd, fd_set *rfds) { - int i, argc, ret, clifd = -1; + int i, argc, ret, clifd; char buf[MAXLINE], **argv = NULL; struct sockaddr_un unix_addr; uid_t uid; - ret = para_accept(accept_fd, &unix_addr, sizeof(struct sockaddr_un)); - if (ret < 0) - goto out; - clifd = ret; + ret = para_accept(accept_fd, rfds, &unix_addr, sizeof(struct sockaddr_un), &clifd); + if (ret <= 0) + return ret; ret = recv_cred_buffer(clifd, buf, sizeof(buf) - 1); if (ret < 0) goto out; diff --git a/client_common.c b/client_common.c index f34f81bf..593cb2c0 100644 --- a/client_common.c +++ b/client_common.c @@ -107,17 +107,29 @@ static void client_pre_select(struct sched *s, struct task *t) } } -static ssize_t client_recv_buffer(struct client_task *ct, char *buf, size_t len) +static int client_recv_buffer(struct client_task *ct, fd_set *rfds, + char *buf, size_t sz, size_t *n) { - ssize_t ret; + int ret; if (ct->status < CL_SENT_CH_RESPONSE) - ret = recv_buffer(ct->rc4c.fd, buf, len); - else - ret = rc4_recv_buffer(&ct->rc4c, buf, len); + return read_nonblock(ct->rc4c.fd, buf, sz, rfds, n); + + *n = 0; + ret = rc4_recv_buffer(&ct->rc4c, buf, sz); + /* + * rc4_recv_buffer is used with blocking fds elsewhere, so it + * does not use the nonblock-API. Therefore we need to + * check for EOF and EAGAIN. + */ if (ret == 0) return -E_SERVER_EOF; - return ret; + if (ret == -ERRNO_TO_PARA_ERROR(EAGAIN)) + return 0; + if (ret < 0) + return ret; + *n = ret; + return 0; } /** @@ -138,6 +150,7 @@ static void client_post_select(struct sched *s, struct task *t) struct client_task *ct = container_of(t, struct client_task, task); struct btr_node *btrn = ct->btrn; int ret = 0; + size_t n; char buf[CLIENT_BUFSIZE]; t->error = 0; @@ -145,11 +158,9 @@ static void client_post_select(struct sched *s, struct task *t) return; switch (ct->status) { case CL_CONNECTED: /* receive welcome message */ - if (!FD_ISSET(ct->rc4c.fd, &s->rfds)) - return; - ret = client_recv_buffer(ct, buf, sizeof(buf)); - if (ret < 0) - goto err; + ret = client_recv_buffer(ct, &s->rfds, buf, sizeof(buf), &n); + if (ret < 0 || n == 0) + goto out; ct->status = CL_RECEIVED_WELCOME; return; case CL_RECEIVED_WELCOME: /* send auth command */ @@ -173,14 +184,12 @@ static void client_post_select(struct sched *s, struct task *t) /* the SHA1 of the decrypted challenge */ unsigned char challenge_sha1[HASH_SIZE]; - if (!FD_ISSET(ct->rc4c.fd, &s->rfds)) - return; - ret = client_recv_buffer(ct, buf, sizeof(buf)); - if (ret < 0) - goto err; - PARA_INFO_LOG("<-- [challenge] (%d bytes)\n", ret); + ret = client_recv_buffer(ct, &s->rfds, buf, sizeof(buf), &n); + if (ret < 0 || n == 0) + goto out; + PARA_INFO_LOG("<-- [challenge] (%zu bytes)\n", n); ret = para_decrypt_buffer(ct->key_file, crypt_buf, - (unsigned char *)buf, ret); + (unsigned char *)buf, n); if (ret < 0) goto err; sha1_hash((char *)crypt_buf, CHALLENGE_SIZE, challenge_sha1); @@ -199,19 +208,15 @@ static void client_post_select(struct sched *s, struct task *t) } case CL_SENT_CH_RESPONSE: /* read server response */ { - size_t bytes_received; - if (!FD_ISSET(ct->rc4c.fd, &s->rfds)) - return; - ret = client_recv_buffer(ct, buf, sizeof(buf)); - if (ret < 0) - goto err; - bytes_received = ret; + ret = client_recv_buffer(ct, &s->rfds, buf, sizeof(buf), &n); + if (ret < 0 || n == 0) + goto out; /* check if server has sent "Proceed" message */ ret = -E_CLIENT_AUTH; - if (bytes_received < PROCEED_MSG_LEN) - goto err; + if (n < PROCEED_MSG_LEN) + goto out; if (!strstr(buf, PROCEED_MSG)) - goto err; + goto out; ct->status = CL_RECEIVED_PROCEED; return; } @@ -239,23 +244,20 @@ static void client_post_select(struct sched *s, struct task *t) case CL_SENT_COMMAND: { char *buf2; - if (!FD_ISSET(ct->rc4c.fd, &s->rfds)) - return; /* can not use "buf" here because we need a malloced buffer */ buf2 = para_malloc(CLIENT_BUFSIZE); - ret = client_recv_buffer(ct, buf2, CLIENT_BUFSIZE); - if (ret < 0) { + ret = client_recv_buffer(ct, &s->rfds, buf2, CLIENT_BUFSIZE, &n); + if (n > 0) { + if (strstr(buf2, AWAITING_DATA_MSG)) { + free(buf2); + ct->status = CL_SENDING; + return; + } + ct->status = CL_RECEIVING; + btr_add_output(buf2, n, btrn); + } else free(buf2); - goto err; - } - if (strstr(buf2, AWAITING_DATA_MSG)) { - free(buf2); - ct->status = CL_SENDING; - return; - } - ct->status = CL_RECEIVING; - btr_add_output(buf2, ret, btrn); - return; + goto out; } case CL_SENDING: { @@ -283,20 +285,24 @@ static void client_post_select(struct sched *s, struct task *t) goto err; if (ret == 0) return; + /* + * The FD_ISSET() is not strictly necessary, but is allows us + * to skip the malloc below if there is nothing to read anyway. + */ if (!FD_ISSET(ct->rc4c.fd, &s->rfds)) return; buf2 = para_malloc(CLIENT_BUFSIZE); - ret = client_recv_buffer(ct, buf2, CLIENT_BUFSIZE); - if (ret < 0) { + ret = client_recv_buffer(ct, &s->rfds, buf2, CLIENT_BUFSIZE, &n); + if (n > 0) { + buf2 = para_realloc(buf2, n); + btr_add_output(buf2, n, btrn); + } else free(buf2); - goto err; - } - buf2 = para_realloc(buf2, ret); - btr_add_output(buf2, ret, btrn); - return; + goto out; } } err: +out: t->error = ret; if (ret < 0) { if (ret != -E_SERVER_EOF && ret != -E_BTR_EOF) diff --git a/dccp_recv.c b/dccp_recv.c index f71a7253..2ab9fcab 100644 --- a/dccp_recv.c +++ b/dccp_recv.c @@ -153,34 +153,27 @@ static void dccp_recv_post_select(struct sched *s, struct task *t) struct btr_node *btrn = rn->btrn; struct iovec iov[2]; int ret, iovcnt; + size_t num_bytes; ret = btr_node_status(btrn, 0, BTR_NT_ROOT); - if (ret < 0) - goto err; - if (ret == 0) - return; - if (!FD_ISSET(pdd->fd, &s->rfds)) - return; /* nothing to do */ + if (ret <= 0) + goto out; iovcnt = btr_pool_get_buffers(pdd->btrp, iov); ret = -E_DCCP_OVERRUN; if (iovcnt == 0) - goto err; - ret = para_readv(pdd->fd, iov, iovcnt); - /* EAGAIN is possible even if FD_ISSET */ - if (ret < 0 && is_errno(-ret, EAGAIN)) - return; - if (ret == 0) - ret = -E_RECV_EOF; - if (ret < 0) - goto err; - if (ret <= iov[0].iov_len) /* only the first buffer was filled */ - btr_add_output_pool(pdd->btrp, ret, btrn); + goto out; + ret = readv_nonblock(pdd->fd, iov, iovcnt, &s->rfds, &num_bytes); + if (num_bytes == 0) + goto out; + if (num_bytes <= iov[0].iov_len) /* only the first buffer was filled */ + btr_add_output_pool(pdd->btrp, num_bytes, btrn); else { /* both buffers contain data */ btr_add_output_pool(pdd->btrp, iov[0].iov_len, btrn); - btr_add_output_pool(pdd->btrp, ret - iov[0].iov_len, btrn); + btr_add_output_pool(pdd->btrp, num_bytes - iov[0].iov_len, btrn); } - return; -err: +out: + if (ret >= 0) + return; btr_remove_node(rn->btrn); t->error = ret; } diff --git a/dccp_send.c b/dccp_send.c index fb2eafc9..6248ae80 100644 --- a/dccp_send.c +++ b/dccp_send.c @@ -70,9 +70,7 @@ static void dccp_post_select(fd_set *rfds, __a_unused fd_set *wfds) struct sender_client *sc; int tx_ccid; - if (dss->listen_fd < 0 || !FD_ISSET(dss->listen_fd, rfds)) - return; - sc = accept_sender_client(dss); + sc = accept_sender_client(dss, rfds); if (!sc) return; diff --git a/error.h b/error.h index d92c9d6e..31903062 100644 --- a/error.h +++ b/error.h @@ -228,7 +228,6 @@ extern const char **para_errlist[]; PARA_ERROR(SENDMSG, "sendmsg() failed"), \ PARA_ERROR(RECVMSG, "recvmsg() failed"), \ PARA_ERROR(SCM_CREDENTIALS, "did not receive SCM credentials"), \ - PARA_ERROR(RECV_PATTERN, "did not receive expected pattern"), \ #define UDP_RECV_ERRORS \ @@ -374,6 +373,8 @@ extern const char **para_errlist[]; #define FD_ERRORS \ PARA_ERROR(FGETS, "fgets error"), \ + PARA_ERROR(EOF, "end of file"), \ + PARA_ERROR(READ_PATTERN, "did not read expected pattern"), \ #define WRITE_ERRORS \ diff --git a/fd.c b/fd.c index 46be2289..7336bd51 100644 --- a/fd.c +++ b/fd.c @@ -11,12 +11,12 @@ #include #include #include -#include #include #include "para.h" #include "error.h" #include "string.h" +#include "fd.h" /** * Write a buffer to a file descriptor, re-write on short writes. @@ -83,23 +83,143 @@ int write_nonblock(int fd, const char *buf, size_t len, } /** - * Simple wrapper for readv(). + * Read from a non-blocking file descriptor into multiple buffers. * * \param fd The file descriptor to read from. * \param iov Scatter/gather array used in readv(). * \param iovcnt Number of elements in \a iov. + * \param rfds An optional fd set pointer. + * \param num_bytes Result pointer. Contains the number of bytes read from \a fd. + * + * If \a rfds is not \p NULL and the (non-blocking) file descriptor \a fd is + * not set in \a rfds, this function returns early without doing anything. + * Otherwise The function tries to read up to \a sz bytes from \a fd. As for + * write_nonblock(), EAGAIN is not considered an error condition. However, EOF + * is. + * + * \return Zero or a negative error code. If the underlying call to readv(2) + * returned zero (indicating an end of file condition) or failed for some + * reason other than \p EAGAIN, a negative return value is returned. + * + * In any case, \a num_bytes contains the number of bytes that have been + * successfully read from \a fd (zero if the first readv() call failed with + * EAGAIN). Note that even if the function returns negative, some data might + * have been read before the error occured. In this case \a num_bytes is + * positive. + * + * \sa \ref write_nonblock(), read(2), readv(2). + */ +int readv_nonblock(int fd, struct iovec *iov, int iovcnt, fd_set *rfds, + size_t *num_bytes) +{ + int ret, i, j; + + *num_bytes = 0; + /* + * Avoid a shortcoming of select(): Reads from a non-blocking fd might + * return EAGAIN even if FD_ISSET() returns true. However, FD_ISSET() + * returning false definitely means that no data can currently be read. + * This is the common case, so it is worth to avoid the overhead of the + * read() system call in this case. + */ + if (rfds && !FD_ISSET(fd, rfds)) + return 0; + + for (i = 0, j = 0; i < iovcnt;) { + + /* fix up the first iov */ + assert(j < iov[i].iov_len); + iov[i].iov_base += j; + iov[i].iov_len -= j; + ret = readv(fd, iov + i, iovcnt - i); + iov[i].iov_base -= j; + iov[i].iov_len += j; + + if (ret == 0) + return -E_EOF; + if (ret < 0) { + if (errno == EAGAIN) + return 0; + return -ERRNO_TO_PARA_ERROR(errno); + } + *num_bytes += ret; + while (ret > 0) { + if (ret < iov[i].iov_len - j) { + j += ret; + break; + } + ret -= iov[i].iov_len - j; + j = 0; + if (++i >= iovcnt) + break; + } + } + return 0; +} + +/** + * Read from a non-blocking file descriptor into a single buffer. + * + * \param fd The file descriptor to read from. + * \param buf The buffer to read data to. + * \param sz The size of \a buf. + * \param rfds \see \ref readv_nonblock(). + * \param num_bytes \see \ref readv_nonblock(). + * + * This is a simple wrapper for readv_nonblock() which uses an iovec with a single + * buffer. + * + * \return The return value of the underlying call to readv_nonblock(). + */ +int read_nonblock(int fd, void *buf, size_t sz, fd_set *rfds, size_t *num_bytes) +{ + struct iovec iov = {.iov_base = buf, .iov_len = sz}; + return readv_nonblock(fd, &iov, 1, rfds, num_bytes); +} + +/** + * Read a buffer and check its content for a pattern. + * + * \param fd The file descriptor to receive from. + * \param pattern The expected pattern. + * \param bufsize The size of the internal buffer. + * \param rfds Passed to read_nonblock(). * - * \return A negative error code on errors, the return value of the underlying - * call to readv() otherwise. + * This function tries to read at most \a bufsize bytes from the non-blocking + * file descriptor \a fd. If at least \p strlen(\a pattern) bytes have been + * received, the beginning of the received buffer is compared with \a pattern, + * ignoring case. * - * \sa readv(2). + * \return Positive if \a pattern was received, negative on errors, zero if no data + * was available to read. + * + * \sa \ref read_nonblock(), \sa strncasecmp(3). */ -int para_readv(int fd, struct iovec *iov, int iovcnt) +int read_pattern(int fd, const char *pattern, size_t bufsize, fd_set *rfds) { - int ret = readv(fd, iov, iovcnt); + size_t n, len; + char *buf = para_malloc(bufsize + 1); + int ret = read_nonblock(fd, buf, bufsize, rfds, &n); + buf[n] = '\0'; if (ret < 0) - return -ERRNO_TO_PARA_ERROR(errno); + goto out; + ret = 0; + if (n == 0) + goto out; + ret = -E_READ_PATTERN; + len = strlen(pattern); + if (n < len) + goto out; + if (strncasecmp(buf, pattern, len) != 0) + goto out; + ret = 1; +out: + if (ret < 0) { + PARA_NOTICE_LOG("%s\n", para_strerror(-ret)); + PARA_NOTICE_LOG("recvd %zu bytes: %s\n", n, buf); + } + free(buf); return ret; } diff --git a/fd.h b/fd.h index 68092308..c21a7d14 100644 --- a/fd.h +++ b/fd.h @@ -7,7 +7,6 @@ /** \file fd.h exported symbols from fd.c */ int write_all(int fd, const char *buf, size_t *len); -int para_readv(int fd, struct iovec *iov, int iovcnt); int file_exists(const char *); int para_select(int n, fd_set *readfds, fd_set *writefds, struct timeval *timeout_tv); @@ -27,6 +26,10 @@ int mmap_full_file(const char *filename, int open_mode, void **map, int para_munmap(void *start, size_t length); int write_ok(int fd); void valid_fd_012(void); +int readv_nonblock(int fd, struct iovec *iov, int iovcnt, fd_set *rfds, + size_t *num_bytes); +int read_nonblock(int fd, void *buf, size_t sz, fd_set *rfds, size_t *num_bytes); +int read_pattern(int fd, const char *pattern, size_t bufsize, fd_set *rfds); int write_nonblock(int fd, const char *buf, size_t len, size_t max_bytes_per_write); int for_each_file_in_dir(const char *dirname, diff --git a/gui.c b/gui.c index 64fab61b..fa1538b0 100644 --- a/gui.c +++ b/gui.c @@ -54,7 +54,7 @@ static int cmd_died, curses_active; static pid_t cmd_pid; static int command_pipe = -1; -static int audiod_pipe = -1; +static int stat_pipe = -1; static struct gui_args_info conf; enum {GETCH_MODE, COMMAND_MODE, EXTERNAL_MODE}; @@ -191,20 +191,6 @@ static struct gui_command command_list[] = { } }; -static int para_open_audiod_pipe(char *cmd) -{ - int fds[3] = {0, 1, 0}; - pid_t pid; - int ret = para_exec_cmdline_pid(&pid, cmd, fds); - if (ret < 0) - return ret; - ret = mark_fd_nonblocking(fds[1]); - if (ret > 0) - return fds[1]; - close(fds[1]); - return ret; -} - static int find_cmd_byname(char *name) { int i; @@ -708,12 +694,15 @@ print: return 1; } -static int read_audiod_pipe(int fd) +static int read_stat_pipe(fd_set *rfds) { static char *buf; static int bufsize, loaded; - int ret; + int ret, ret2; + size_t sz; + if (stat_pipe < 0) + return 0; if (loaded >= bufsize) { if (bufsize > 1000 * 1000) { loaded = 0; @@ -723,16 +712,18 @@ static int read_audiod_pipe(int fd) buf = para_realloc(buf, bufsize); } assert(loaded < bufsize); - ret = read(fd, buf + loaded, bufsize - loaded); - if (ret <= 0) - return ret; - loaded += ret; - ret = for_each_stat_item(buf, loaded, update_item); - if (ret < 0) - return ret; - if (ret > 0 && ret < loaded) - memmove(buf, buf + loaded - ret, ret); - loaded = ret; + ret = read_nonblock(stat_pipe, buf + loaded, bufsize - loaded, + rfds, &sz); + loaded += sz; + ret2 = for_each_stat_item(buf, loaded, update_item); + if (ret < 0 || ret2 < 0) { + loaded = 0; + return ret2 < 0? ret2 : ret; + } + sz = ret2; /* what is left */ + if (sz > 0 && sz < loaded) + memmove(buf, buf + loaded - sz, sz); + loaded = sz; return 1; } @@ -902,15 +893,24 @@ static void handle_signal(int sig) } } -static int open_audiod_pipe(void) +static int open_stat_pipe(void) { static int init = 1; + int ret, fds[3] = {0, 1, 0}; + pid_t pid; if (init) init = 0; else sleep(1); - return para_open_audiod_pipe(conf.stat_cmd_arg); + ret = para_exec_cmdline_pid(&pid, conf.stat_cmd_arg, fds); + if (ret < 0) + return ret; + ret = mark_fd_nonblocking(fds[1]); + if (ret >= 0) + return fds[1]; + close(fds[1]); + return ret; } /* @@ -931,7 +931,7 @@ static int do_select(int mode) { fd_set rfds; int ret; - int max_fileno, cp_numread = 1; + int max_fileno; char command_buf[4096] = ""; int cbo = 0; /* command buf offset */ struct timeval tv; @@ -941,11 +941,10 @@ repeat: // ret = refresh_status(); FD_ZERO(&rfds); max_fileno = 0; - /* audiod pipe */ - if (audiod_pipe < 0) - audiod_pipe = open_audiod_pipe(); - if (audiod_pipe >= 0) - para_fd_set(audiod_pipe, &rfds, &max_fileno); + if (stat_pipe < 0) + stat_pipe = open_stat_pipe(); + if (stat_pipe >= 0) + para_fd_set(stat_pipe, &rfds, &max_fileno); /* signal pipe */ para_fd_set(signal_pipe, &rfds, &max_fileno); /* command pipe only for COMMAND_MODE */ @@ -955,46 +954,41 @@ repeat: if (ret <= 0) goto check_return; /* skip fd checks */ /* signals */ - if (FD_ISSET(signal_pipe, &rfds)) { - int sig_nr = para_next_signal(); - if (sig_nr > 0) - handle_signal(sig_nr); - } + ret = para_next_signal(&rfds); + if (ret > 0) + handle_signal(ret); /* read command pipe if ready */ - if (command_pipe >= 0 && mode == COMMAND_MODE && - FD_ISSET(command_pipe, &rfds)) { - cp_numread = read(command_pipe, command_buf + cbo, - sizeof(command_buf) - 1 - cbo); - if (cp_numread >= 0) - cbo += cp_numread; - else { - if (cp_numread < 0) - PARA_ERROR_LOG("read error (%d)", cp_numread); + if (command_pipe >= 0 && mode == COMMAND_MODE) { + size_t sz; + ret = read_nonblock(command_pipe, command_buf + cbo, + sizeof(command_buf) - 1 - cbo, &rfds, &sz); + cbo += sz; + sz = cbo; + cbo = for_each_line(command_buf, cbo, &add_output_line, NULL); + if (sz != cbo) + wrefresh(bot.win); + if (ret < 0) { + PARA_NOTICE_LOG("closing command pipe: %s", + para_strerror(-ret)); close(command_pipe); command_pipe = -1; + return 0; } } - if (audiod_pipe >= 0 && FD_ISSET(audiod_pipe, &rfds)) - if (read_audiod_pipe(audiod_pipe) <= 0) { - close(audiod_pipe); - audiod_pipe = -1; - clear_all_items(); - free(stat_content[SI_BASENAME]); - stat_content[SI_BASENAME] = - para_strdup("audiod not running!?"); - print_all_items(); - } + ret = read_stat_pipe(&rfds); + if (ret < 0) { + PARA_NOTICE_LOG("closing stat pipe: %s\n", para_strerror(-ret)); + close(stat_pipe); + stat_pipe = -1; + clear_all_items(); + free(stat_content[SI_BASENAME]); + stat_content[SI_BASENAME] = + para_strdup("stat command terminated!?"); + print_all_items(); + } check_return: switch (mode) { case COMMAND_MODE: - if (cp_numread <= 0 && !cbo) /* command complete */ - return 0; - if (cbo) - cbo = for_each_line(command_buf, cbo, - &add_output_line, NULL); - if (cp_numread <= 0) - cbo = 0; - wrefresh(bot.win); ret = wgetch(top.win); if (ret != ERR && ret != KEY_RESIZE) { if (command_pipe) { diff --git a/http_recv.c b/http_recv.c index cc376dd9..9ade8dfb 100644 --- a/http_recv.c +++ b/http_recv.c @@ -95,7 +95,7 @@ static void http_recv_post_select(struct sched *s, struct task *t) struct btr_node *btrn = rn->btrn; int ret; char *buf; - size_t sz; + size_t sz, n; t->error = 0; ret = btr_node_status(btrn, 0, BTR_NT_ROOT); @@ -116,12 +116,12 @@ static void http_recv_post_select(struct sched *s, struct task *t) phd->status = HTTP_SENT_GET_REQUEST; return; } - if (!FD_ISSET(phd->fd, &s->rfds)) - return; if (phd->status == HTTP_SENT_GET_REQUEST) { - ret = recv_pattern(phd->fd, HTTP_OK_MSG, strlen(HTTP_OK_MSG)); + ret = read_pattern(phd->fd, HTTP_OK_MSG, strlen(HTTP_OK_MSG), &s->rfds); if (ret < 0) goto err; + if (ret == 0) + return; PARA_INFO_LOG("received ok msg, streaming\n"); phd->status = HTTP_STREAMING; return; @@ -130,13 +130,11 @@ static void http_recv_post_select(struct sched *s, struct task *t) sz = btr_pool_get_buffer(phd->btrp, &buf); if (sz == 0) goto err; - ret = recv_bin_buffer(phd->fd, buf, sz); - if (ret == 0) - ret = -E_RECV_EOF; - if (ret < 0) - goto err; - btr_add_output_pool(phd->btrp, ret, btrn); - return; + ret = read_nonblock(phd->fd, buf, sz, &s->rfds, &n); + if (n > 0) + btr_add_output_pool(phd->btrp, n, btrn); + if (ret >= 0) + return; err: btr_remove_node(rn->btrn); t->error = ret; diff --git a/http_send.c b/http_send.c index 828d99e2..424d63b2 100644 --- a/http_send.c +++ b/http_send.c @@ -96,23 +96,20 @@ static void http_post_select(fd_set *rfds, __a_unused fd_set *wfds) { struct sender_client *sc, *tmp; struct private_http_sender_data *phsd; + int ret; - if (hss->listen_fd < 0) - return; list_for_each_entry_safe(sc, tmp, &hss->client_list, node) { phsd = sc->private_data; switch (phsd->status) { case HTTP_STREAMING: /* nothing to do */ break; case HTTP_CONNECTED: /* need to recv get request */ - if (FD_ISSET(sc->fd, rfds)) { - if (recv_pattern(sc->fd, HTTP_GET_MSG, MAXLINE) - < 0) { - phsd->status = HTTP_INVALID_GET_REQUEST; - } else { - phsd->status = HTTP_GOT_GET_REQUEST; - PARA_INFO_LOG("received get request\n"); - } + ret = read_pattern(sc->fd, HTTP_GET_MSG, MAXLINE, rfds); + if (ret < 0) + phsd->status = HTTP_INVALID_GET_REQUEST; + else if (ret > 0) { + phsd->status = HTTP_GOT_GET_REQUEST; + PARA_INFO_LOG("received get request\n"); } break; case HTTP_GOT_GET_REQUEST: /* need to send ok msg */ @@ -125,9 +122,7 @@ static void http_post_select(fd_set *rfds, __a_unused fd_set *wfds) break; } } - if (!FD_ISSET(hss->listen_fd, rfds)) - return; - sc = accept_sender_client(hss); + sc = accept_sender_client(hss, rfds); if (!sc) return; phsd = para_malloc(sizeof(*phsd)); diff --git a/net.c b/net.c index 92ae217d..59b7f367 100644 --- a/net.c +++ b/net.c @@ -756,23 +756,36 @@ int recv_buffer(int fd, char *buf, size_t size) * Wrapper around the accept system call. * * \param fd The listening socket. + * \param rfds An optional fd_set pointer. * \param addr Structure which is filled in with the address of the peer socket. * \param size Should contain the size of the structure pointed to by \a addr. + * \param new_fd Result pointer. * - * Accept incoming connections on \a addr. Retry if interrupted. + * Accept incoming connections on \a addr, retry if interrupted. If \a rfds is + * not \p NULL, return 0 if \a fd is not set in \a rfds without calling accept(). * - * \return The new file descriptor on success, negative on errors. + * \return Negative on errors, zero if no connections are present to be accepted, + * one otherwise. * * \sa accept(2). */ -int para_accept(int fd, void *addr, socklen_t size) +int para_accept(int fd, fd_set *rfds, void *addr, socklen_t size, int *new_fd) { - int new_fd; + int ret; + if (rfds && !FD_ISSET(fd, rfds)) + return 0; do - new_fd = accept(fd, (struct sockaddr *) addr, &size); - while (new_fd < 0 && errno == EINTR); - return new_fd < 0? -ERRNO_TO_PARA_ERROR(errno) : new_fd; + ret = accept(fd, (struct sockaddr *) addr, &size); + while (ret < 0 && errno == EINTR); + + if (ret >= 0) { + *new_fd = ret; + return 1; + } + if (errno == EAGAIN || errno == EWOULDBLOCK) + return 0; + return -ERRNO_TO_PARA_ERROR(errno); } /** @@ -1017,41 +1030,3 @@ int recv_cred_buffer(int fd, char *buf, size_t size) return result; } #endif /* HAVE_UCRED */ - -/** - * Receive a buffer and check for a pattern. - * - * \param fd The file descriptor to receive from. - * \param pattern The expected pattern. - * \param bufsize The size of the internal buffer. - * - * \return Positive if \a pattern was received, negative otherwise. - * - * This function tries to receive at most \a bufsize bytes from file descriptor - * \a fd. If at least \p strlen(\a pattern) bytes were received, the beginning - * of the received buffer is compared with \a pattern, ignoring case. - * - * \sa recv_buffer(), \sa strncasecmp(3). - */ -int recv_pattern(int fd, const char *pattern, size_t bufsize) -{ - size_t len = strlen(pattern); - char *buf = para_malloc(bufsize + 1); - int ret = -E_RECV_PATTERN, n = recv_buffer(fd, buf, bufsize + 1); - - if (n < len) - goto out; - if (strncasecmp(buf, pattern, len)) - goto out; - ret = 1; -out: - if (ret < 0) { - PARA_NOTICE_LOG("did not receive pattern '%s'\n", pattern); - if (n > 0) - PARA_NOTICE_LOG("recvd %d bytes: %s\n", n, buf); - else if (n < 0) - PARA_NOTICE_LOG("%s\n", para_strerror(-n)); - } - free(buf); - return ret; -} diff --git a/net.h b/net.h index 93c0c5e8..457c24dc 100644 --- a/net.h +++ b/net.h @@ -139,13 +139,12 @@ __printf_2_3 int send_va_buffer(int fd, const char *fmt, ...); int recv_bin_buffer(int fd, char *buf, size_t size); int recv_buffer(int fd, char *buf, size_t size); -int para_accept(int, void *addr, socklen_t size); +int para_accept(int fd, fd_set *rfds, void *addr, socklen_t size, int *new_fd); int create_local_socket(const char *name, struct sockaddr_un *unix_addr, mode_t mode); int create_remote_socket(const char *name); int recv_cred_buffer(int, char *, size_t); ssize_t send_cred_buffer(int, char*); -int recv_pattern(int fd, const char *pattern, size_t bufsize); /** * Functions and definitions to support \p IPPROTO_DCCP diff --git a/oss_write.c b/oss_write.c index 10a64497..d0cff015 100644 --- a/oss_write.c +++ b/oss_write.c @@ -53,7 +53,8 @@ static void oss_close(struct writer_node *wn) { struct private_oss_write_data *powd = wn->private_data; - close(powd->fd); + if (powd->fd >= 0) + close(powd->fd); free(powd); } @@ -137,7 +138,7 @@ static int oss_init(struct writer_node *wn, unsigned samplerate, unsigned channe return 1; err: close(powd->fd); - free(powd); + powd->fd = -1; return ret; } diff --git a/send.h b/send.h index 85e5ed1f..acf62db4 100644 --- a/send.h +++ b/send.h @@ -140,7 +140,7 @@ void generic_com_deny(struct sender_command_data *scd, int generic_com_on(struct sender_status *ss, unsigned protocol); void generic_com_off(struct sender_status *ss); char *generic_sender_help(void); -struct sender_client *accept_sender_client(struct sender_status *ss); +struct sender_client *accept_sender_client(struct sender_status *ss, fd_set *rfds); int send_queued_chunks(int fd, struct chunk_queue *cq, size_t max_bytes_per_write); int parse_fec_url(const char *arg, struct sender_command_data *scd); diff --git a/send_common.c b/send_common.c index f931fdaf..b44c8133 100644 --- a/send_common.c +++ b/send_common.c @@ -348,15 +348,18 @@ void generic_com_off(struct sender_status *ss) * \sa \ref para_accept(), \ref mark_fd_nonblocking(), \ref acl_check_access(), * \ref cq_new(), \ref add_close_on_fork_list(). */ -struct sender_client *accept_sender_client(struct sender_status *ss) +struct sender_client *accept_sender_client(struct sender_status *ss, fd_set *rfds) { struct sender_client *sc; - int fd, ret = para_accept(ss->listen_fd, NULL, 0); - if (ret < 0) { + int fd, ret; + + if (ss->listen_fd < 0) + return NULL; + ret = para_accept(ss->listen_fd, rfds, NULL, 0, &fd); + if (ret < 0) PARA_ERROR_LOG("%s\n", para_strerror(-ret)); + if (ret <= 0) return NULL; - } - fd = ret; ret = -E_MAX_CLIENTS; if (ss->max_clients > 0 && ss->num_clients >= ss->max_clients) goto err_out; diff --git a/server.c b/server.c index 89a81372..bc143039 100644 --- a/server.c +++ b/server.c @@ -276,15 +276,13 @@ static void handle_sighup(void) kill(mmd->afs_pid, SIGHUP); } -static void signal_post_select(struct sched *s, struct task *t) +static void signal_post_select(struct sched *s, __a_unused struct task *t) { - struct signal_task *st = container_of(t, struct signal_task, task); + int signum = para_next_signal(&s->rfds); - if (!FD_ISSET(st->fd, &s->rfds)) + switch (signum) { + case 0: return; - - st->signum = para_next_signal(); - switch (st->signum) { case SIGHUP: handle_sighup(); break; @@ -304,7 +302,7 @@ static void signal_post_select(struct sched *s, struct task *t) /* die on sigint/sigterm. Kill all children too. */ case SIGINT: case SIGTERM: - PARA_EMERG_LOG("terminating on signal %d\n", st->signum); + PARA_EMERG_LOG("terminating on signal %d\n", signum); kill(0, SIGTERM); /* * We must wait for afs because afs catches SIGINT/SIGTERM. @@ -366,12 +364,9 @@ static void command_post_select(struct sched *s, struct task *t) pid_t child_pid; uint32_t *chunk_table; - if (!FD_ISSET(sct->listen_fd, &s->rfds)) - return; - ret = para_accept(sct->listen_fd, NULL, 0); - if (ret < 0) + ret = para_accept(sct->listen_fd, &s->rfds, NULL, 0, &new_fd); + if (ret <= 0) goto out; - new_fd = ret; peer_name = remote_name(new_fd); PARA_INFO_LOG("got connection from %s, forking\n", peer_name); mmd->num_connects++; diff --git a/signal.c b/signal.c index bded532e..0b4b6f0b 100644 --- a/signal.c +++ b/signal.c @@ -151,27 +151,24 @@ void para_install_sighandler(int sig) /** * Return the number of the next pending signal. * - * This should be called if the fd for the signal pipe is ready for reading. + * \param rfds Th fd_set containing the signal pipe. * - * \return On success, the number of the received signal is returned. If the - * read returned zero or was interrupted by another signal the function returns - * 0. Otherwise, a negative error value is returned. + * \return On success, the number of the received signal is returned. If there + * is no signal currently pending, the function returns zero. On read errors + * from the signal pipe, the process is terminated. */ -int para_next_signal(void) +int para_next_signal(fd_set *rfds) { - int s; - ssize_t r = read(signal_pipe[0], &s, sizeof(s)); + size_t n; + int s, ret = read_nonblock(signal_pipe[0], &s, sizeof(s), rfds, &n); - if (!r) { - PARA_CRIT_LOG("read from signal pipe returned zero\n"); - return 0; - } - if (r < 0) { - if (errno == EAGAIN || errno == EINTR) - return 0; - return -ERRNO_TO_PARA_ERROR(errno); + if (ret < 0) { + PARA_EMERG_LOG("%s\n", para_strerror(-ret)); + exit(EXIT_FAILURE); } - assert(r == sizeof(s)); + if (n == 0) + return 0; + assert(n == sizeof(s)); PARA_DEBUG_LOG("next signal: %d\n", s); return s; } diff --git a/signal.h b/signal.h index 7be960ef..1dbfc981 100644 --- a/signal.h +++ b/signal.h @@ -12,8 +12,6 @@ struct signal_task { /** The signal pipe. */ int fd; - /** The number of the most recent signal. */ - int signum; /** The associated task structure. */ struct task task; }; @@ -22,5 +20,5 @@ int para_signal_init(void); void para_sigaction(int sig, void (*handler)(int)); void para_install_sighandler(int); int para_reap_child(pid_t *pid); -int para_next_signal(void); +int para_next_signal(fd_set *rfds); void para_signal_shutdown(void); diff --git a/stdin.c b/stdin.c index 5fc91c9d..ca5eb0e1 100644 --- a/stdin.c +++ b/stdin.c @@ -61,7 +61,7 @@ static void stdin_post_select(struct sched *s, struct task *t) { struct stdin_task *sit = container_of(t, struct stdin_task, task); ssize_t ret; - size_t sz; + size_t sz, n; char *buf = NULL; t->error = 0; @@ -70,8 +70,6 @@ static void stdin_post_select(struct sched *s, struct task *t) goto err; if (ret == 0) return; - if (!FD_ISSET(STDIN_FILENO, &s->rfds)) - return; sz = btr_pool_get_buffer(sit->btrp, &buf); if (sz == 0) return; @@ -81,15 +79,11 @@ static void stdin_post_select(struct sched *s, struct task *t) * reference can not be freed, we're stuck. */ sz = PARA_MIN(sz, btr_pool_size(sit->btrp) / 2); - ret = read(STDIN_FILENO, buf, sz); - if (ret < 0) - ret = -ERRNO_TO_PARA_ERROR(errno); - if (ret == 0) - ret = -E_STDIN_EOF; - if (ret < 0) - goto err; - btr_add_output_pool(sit->btrp, ret, sit->btrn); - return; + ret = read_nonblock(STDIN_FILENO, buf, sz, &s->rfds, &n); + if (n > 0) + btr_add_output_pool(sit->btrp, n, sit->btrn); + if (ret >= 0) + return; err: btr_remove_node(sit->btrn); //btr_pool_free(sit->btrp); diff --git a/udp_recv.c b/udp_recv.c index 00ad3e27..5520c6fb 100644 --- a/udp_recv.c +++ b/udp_recv.c @@ -67,43 +67,36 @@ static void udp_recv_post_select(__a_unused struct sched *s, struct task *t) struct receiver_node *rn = container_of(t, struct receiver_node, task); struct private_udp_recv_data *purd = rn->private_data; struct btr_node *btrn = rn->btrn; - size_t packet_size; + size_t num_bytes; struct iovec iov[2]; - int ret, iovcnt; + int ret, readv_ret, iovcnt; t->error = 0; ret = btr_node_status(btrn, 0, BTR_NT_ROOT); - if (ret < 0) - goto err; - if (ret == 0) - return; - if (!FD_ISSET(purd->fd, &s->rfds)) - return; + if (ret <= 0) + goto out; iovcnt = btr_pool_get_buffers(purd->btrp, iov); ret = -E_UDP_OVERRUN; if (iovcnt == 0) - goto err; - ret = para_readv(purd->fd, iov, iovcnt); - /* EAGAIN is possible even if FD_ISSET */ - if (ret < 0 && is_errno(-ret, EAGAIN)) - return; - if (ret == 0) - ret = -E_RECV_EOF; + goto out; + ret = readv_nonblock(purd->fd, iov, iovcnt, &s->rfds, &num_bytes); + if (num_bytes == 0) + goto out; + readv_ret = ret; + ret = udp_check_eof(num_bytes, iov); if (ret < 0) - goto err; - packet_size = ret; - ret = udp_check_eof(packet_size, iov); - if (ret < 0) - goto err; - if (iov[0].iov_len >= packet_size) - btr_add_output_pool(purd->btrp, packet_size, btrn); + goto out; + if (iov[0].iov_len >= num_bytes) + btr_add_output_pool(purd->btrp, num_bytes, btrn); else { /* both buffers contain data */ btr_add_output_pool(purd->btrp, iov[0].iov_len, btrn); - btr_add_output_pool(purd->btrp, packet_size - iov[0].iov_len, + btr_add_output_pool(purd->btrp, num_bytes - iov[0].iov_len, btrn); } - return; -err: + ret = readv_ret; +out: + if (ret >= 0) + return; btr_remove_node(btrn); t->error = ret; close(purd->fd); diff --git a/vss.c b/vss.c index bfb0f0a1..898180c0 100644 --- a/vss.c +++ b/vss.c @@ -781,16 +781,20 @@ static int recv_afs_msg(int afs_socket, int *fd, uint32_t *code, uint32_t *data) return 1; } -static void recv_afs_result(struct vss_task *vsst) +static void recv_afs_result(struct vss_task *vsst, fd_set *rfds) { int ret, passed_fd, shmid; uint32_t afs_code = 0, afs_data = 0; struct stat statbuf; - vsst->afsss = AFS_SOCKET_READY; + if (!FD_ISSET(vsst->afs_socket, rfds)) + return; ret = recv_afs_msg(vsst->afs_socket, &passed_fd, &afs_code, &afs_data); + if (ret == -ERRNO_TO_PARA_ERROR(EAGAIN)) + return; if (ret < 0) goto err; + vsst->afsss = AFS_SOCKET_READY; PARA_DEBUG_LOG("fd: %d, code: %u, shmid: %u\n", passed_fd, afs_code, afs_data); ret = -E_NOFD; @@ -920,10 +924,9 @@ static void vss_post_select(struct sched *s, struct task *t) senders[sender_num].client_cmds[num](&mmd->sender_cmd_data); mmd->sender_cmd_data.cmd_num = -1; } - if (vsst->afsss != AFS_SOCKET_CHECK_FOR_WRITE) { - if (FD_ISSET(vsst->afs_socket, &s->rfds)) - recv_afs_result(vsst); - } else if (FD_ISSET(vsst->afs_socket, &s->wfds)) { + if (vsst->afsss != AFS_SOCKET_CHECK_FOR_WRITE) + recv_afs_result(vsst, &s->rfds); + else if (FD_ISSET(vsst->afs_socket, &s->wfds)) { PARA_NOTICE_LOG("requesting new fd from afs\n"); ret = send_buffer(vsst->afs_socket, "new"); if (ret < 0) diff --git a/web/documentation.in.html b/web/documentation.in.html index 90f3bb67..2789112a 100644 --- a/web/documentation.in.html +++ b/web/documentation.in.html @@ -11,20 +11,8 @@ the pieces of paraslash work together. -
  • REQUIREMENTS, - list of required and optional software. -
    • - -
  • README, - the paraslash executables, with brief descriptions. -
    • - -
  • INSTALL, - installation and configuration notes. -
    • - -
  • README.afs, - audio file selector documentation. +
  • user manual, + Installation, Configuration and Usage.

    • diff --git a/web/manual.m4 b/web/manual.m4 new file mode 100644 index 00000000..0ff7e9b7 --- /dev/null +++ b/web/manual.m4 @@ -0,0 +1,2122 @@ +dnl To generate the html version, execute +dnl m4 web/manual.m4 | grutatxt --toc + +define(`LOCAL_LINK_NAME', `translit(`$1', `A-Z +', `a-z__')') +define(`REMOVE_NEWLINE', `translit(`$1',` +', ` ')') + +define(`REFERENCE', ./``#''`LOCAL_LINK_NAME($1)' (`REMOVE_NEWLINE($2)')) +define(`XREFERENCE', `$1' (`REMOVE_NEWLINE($2)')) +define(`EMPH', ``_''`REMOVE_NEWLINE($1)'``_'') + +Paraslash user manual +===================== + +This document describes how to install, configure and use the paraslash +network audio streaming system. Most chapters start with a chapter +overview and conclude with an example section. We try to focus on +general concepts and on the interaction of the various pieces of the +paraslash package. Hence this user manual is not meant as a replacement +for the manual pages that describe all command line options of each +paraslash executable. + +------------ +Introduction +------------ + +In this chapter we give an REFERENCE(Overview, overview) of the +interactions of the two main programs contained in the paraslash +package, followed by REFERENCE(The paraslash executables, brief +descriptions) of all executables. + +Overview +~~~~~~~~ + +The core functionality of the para suite is provided by two main +executables, para_server and para_audiod. The former maintains a +database of audio files and streams these files to para_audiod which +receives and plays the stream. + +In a typical setting, both para_server and para_audiod act as +background daemons whose functionality is controlled by client +programs: the para_audioc client controls para_audiod over a local +socket while the para_client program connects to para_server over a +local or remote networking connection. + +Typically, these two daemons run on different hosts but a local setup +is also possible. + +A simplified picture of a typical setup is as follows +<< +
    + server_host                                  client_host
+ ~~~~~~~~~~~                                  ~~~~~~~~~~~
+ 
+ +-----------+         audio stream           +-----------+
+ |para_server| -----------------------------> |para_audiod|
+ +-----------+                                +-----------+
+      ^                                            ^
+      |                                            |
+      |                                            | connect
+      |                                            |
+      |                                            |
+      |                                       +-----------+
+      |                                       |para_audioc|
+      |                                       +-----------+
+      |
+      |
+      |                  connect              +-----------+
+      +-------------------------------------- |para_client|
+                                              +-----------+
+
    +>> + +The paraslash executables +~~~~~~~~~~~~~~~~~~~~~~~~~ + +*para_server* + +para_server streams binary audio data (MP3, OGG/Vorbis, M4A, WMA +files) over local and/or remote networks. It listens on a TCP port and +accepts commands such as play, stop, pause, next from authenticated +clients. There are many more commands though, see the man page of +para_server for a description of all commands. + +It supports three built-in network streaming protocols +(senders/receivers): HTTP, DCCP, or UDP. This is explained in more +detail in the section on REFERENCE(Networking, networking). + +The built-in audio file selector of paraslash is used to manage your +audio files. It maintains statistics on the usage of all available +audio files such as last-played time, and the number of times each +file was selected. + +Additional information may be added to the database to allow +fine-grained selection based on various properties of the audio file, +including information found in (ID3) tags. However, old-fashioned +playlists are also supported. + +It is also possible to store images (album covers) and lyrics in the +database and associate these to the corresponding audio files. + +The section on the REFERENCE(The audio file selector, audio file +selector) discusses this topic. + + +*para_client* + +The client program to connect to para_server. paraslash commands +are sent to para_server and the response is dumped to STDOUT. This +can be used by any scripting language to produce user interfaces with +little programming effort. + +All connections between para_server and para_client are encrypted +with a symmetric RC4 session key. For each user of paraslash you must +create a public/secret RSA key pair for authentication. + + +*para_audiod* + +The local daemon that collects information from para_server. + +It runs on the client side and connects to para_server. As soon as +para_server announces the availability of an audio stream, para_audiod +starts an appropriate receiver, any number of filters and a paraslash +writer to play the stream. + +Moreover, para_audiod listens on a local socket and sends status +information about para_server and para_audiod to local clients on +request. Access via this local socket may be restricted by using Unix +socket credentials, if available. + + +*para_audioc* + +The client program which talks to para_audiod. Used to control +para_audiod, to receive status info, or to grab the stream at any +point of the decoding process. + +*para_recv* + +A command line HTTP/DCCP/UDP stream grabber. The http mode is +compatible with arbitrary HTTP streaming sources (e.g. icecast). + +*para_filter* + +A filter program that reads from STDIN and writes to STDOUT. +Like para_recv, this is an atomic building block which can be used +to assemble higher-level audio receiving facilities. It combines +several different functionalities in one tool: decoders for multiple +audio formats (MP3, OGG/Vorbis, AAC, WMA) and a number of processing +filters, among these a normalizer for audio volume. + +*para_afh* + +A small stand-alone program that prints tech info about the given +audio file to STDOUT. It can be instructed to print a "chunk table", +an array of offsets within the audio file or to write the content of +the audio file in complete chunks 'just in time'. + +This allows third-party streaming software that is unaware of the +particular audio format to send complete frames in real time. + +*para_write* + +A modular audio stream writer. It supports a simple file writer +output plug-in and optional WAV/raw players for ALSA (Linux) and for +coreaudio (Mac OS). para_write can also be used as a stand-alone WAV +or raw audio player. + + +*para_gui* + +Curses-based gui that presents status information obtained in a curses +window. Appearance can be customized via themes. para_gui provides +key-bindings for the most common server commands and new key-bindings +can be added easily. + + +*para_fade* + +An (OSS-only) alarm clock and volume-fader. + +----------- +Quick start +----------- + +This chapter lists the REFERENCE(Requirements, necessary software) +that must be installed to compile the paraslash package, describes +how to REFERENCE(Installation, compile and install) the paraslash +source code and the steps that have to be performed in order to +REFERENCE(Quick start, set up) a typical server and client. + +Requirements +~~~~~~~~~~~~ + +In any case you'll need + + - XREFERENCE(http://systemlinux.org/~maan/osl/, libosl). + The _object storage layer_ library is used by para_server. To + clone the source code repository, execute + + git clone git://git.tuebingen.mpg.de/osl + + - XREFERENCE(ftp://ftp.gnu.org/pub/gnu/gcc, gcc). The + EMPH(gnu compiler collection) is usually shipped with the + distro. gcc-3.3 or newer is required. + + - XREFERENCE(ftp://ftp.gnu.org/pub/gnu/make, gnu make) is + also shipped with the disto. On BSD systems the gnu make + executable is often called gmake. + + - XREFERENCE(ftp://ftp.gnu.org/pub/gnu/bash, bash). Some + scripts which run during compilation require the EMPH(Bourne + again shell). It is most likely already installed. + + - XREFERENCE(http://www.openssl.org/, openssl). The EMPH(Secure + Sockets Layer) library is needed for cryptographic routines + on both the server and the client side. It is usually shipped + with the distro, but you might have to install the "development + package" (called libssl-dev on debian systems) as well. + + - XREFERENCE(ftp://ftp.gnu.org/pub/gnu/help2man, help2man) + is used to create the man pages. + +Optional: + + - XREFERENCE(http://www.underbit.com/products/mad/, libmad). + To compile in MP3 support for paraslash, the development + package must be installed. It is called libmad0-dev on + debian-based systems. Note that libmad is not necessary on + the server side, i.e. for sending MP3 files. + + - XREFERENCE(http://www.underbit.com/products/mad/, + libid3tag). For version-2 ID3 tag support, you'll need + the libid3tag development package libid3tag0-dev. Without + libid3tag, only version one tags are recognized. + + - XREFERENCE(http://www.xiph.org/downloads/, ogg vorbis). + For ogg vorbis streams you'll need libogg, libvorbis, + libvorbisfile. The corresponding Debian packages are called + libogg-dev and libvorbis-dev. + + - XREFERENCE(http://www.audiocoding.com/, libfaad). For aac + files (m4a) you'll need libfaad (libfaad-dev). + + - XREFERENCE(ftp://ftp.alsa-project.org/pub/lib/, alsa-lib). On + Linux, you'll need to have ALSA's development package + libasound2-dev installed. + +Installation +~~~~~~~~~~~~ + +First make sure all non-optional packages listed in the section on +REFERENCE(Requirements, required software) are installed on your +system. + +You don't need everything listed there. In particular, MP3, OGG/Vorbis +and AAC support are all optional. The configure script will detect +what is installed on your system and will only try to build those +executables that can be built with your setup. + +Note that no special decoder library (not even the MP3 decoding library +libmad) is needed for para_server if you only want to stream MP3 or WMA +files. Also, it's fine to use para_server on a box without sound card. + +Next, install the paraslash package on all machines, you'd like this +software to run on: + + (./configure && make) > /dev/null + +There should be no errors but probably some warnings about missing +packages which usually implies that not all audio formats will be +supported. If headers or libs are installed at unusual locations you +might need to tell the configure script where to find them. Try + + ./configure --help + +to see a list of options. If the paraslash package was compiled +successfully, execute as root, + + make install + +to install executables under /usr/local/bin and the man pages under +/usr/local/man. + +Configuration +~~~~~~~~~~~~~ + +*Step 1*: Create a paraslash user + +In order to control para_server at runtime you must create a paraslash +user. As authentication is based on the RSA crypto system you'll have +to create an RSA key pair. If you already have a user and an RSA key +pair, you may skip this step. + +In this section we'll assume a typical setup: You would like to run +para_server on some host called server_host as user foo, and you want +to connect to para_server from another machine called client_host as +user bar. + +As foo@server_host, create ~/.paraslash/server.users by typing the +following commands: + + user=bar + target=~/.paraslash/server.users + key=~/.paraslash/key.pub.$user + perms=AFS_READ,AFS_WRITE,VSS_READ,VSS_WRITE + mkdir -p ~/.paraslash + echo "user $user $key $perms" >> $target + +Next, change to the "bar" account on client_host and generate the +key pair with the commands + + key=~/.paraslash/key.$LOGNAME + mkdir -p ~/.paraslash + (umask 077 && openssl genrsa -out $key 2048) + +para_server only needs to know the public key of the key pair just +created. It can be extracted with + + pubkey=~/.paraslash/key.pub.$LOGNAME + openssl rsa -in $key -pubout -out $pubkey + +Copy the public key just created to server_host (you may skip this step +for a single-user setup, i.e. if foo=bar and server_host=client_host): + + scp $pubkey foo@server_host:.paraslash/ + +Finally, tell para_client to connect to server_host: + + conf=~/.paraslash/client.conf + echo 'hostname server_host' > $conf + + +*Step 2*: Start para_server + +Before starting the server make sure you have write permissions to +the directory /var/paraslash that has been created during installation: + + sudo chown $LOGNAME /var/paraslash + +Alternatively, use the --afs_socket Option to specify a different +location for the AFS command socket. + +For this first try, we'll use the info loglevel to make the output +of para_server more verbose. + + para_server -l info + +Now you can use para_client to connect to the server and issue +commands. Open a new shell as bar@client_host and try + + para_client help + para_client si + +to retrieve the list of available commands and some server info. +Don't proceed if this doesn't work. + +*Step 3*: Create and populate the database + +An empty database is created with + + para_client init + +This initializes a couple of empty tables under +~/.paraslash/afs_database-0.4. You normally don't need to look at these +tables, but it's good to know that you can start from scratch with + + rm -rf ~/.paraslash/afs_database-0.4 + +in case something went wrong. + +Next, you need to add some audio files to that database so that +para_server knows about them. Choose an absolute path to a directory +containing some audio files and add them to the audio file table: + + para_client add /my/mp3/dir + +This might take a while, so it is a good idea to start with a directory +containing not too many files. Note that the table only contains data +about the audio files found, not the files themselves. + +You may print the list of all known audio files with + + para_client ls + +*Step 4*: Configure para_audiod + +para_audiod needs to create a "well-known" socket for the clients to +connect to. The default path for this socket is + + /var/paraslash/audiod_socket.$HOSTNAME + +In order to make this directory writable for para_audiod, execute +as bar@client_host + + sudo chown $LOGNAME /var/paraslash + + +We will also have to tell para_audiod that it should receive the +audio stream from server_host: + + para_audiod -l info -r 'mp3:http -i server_host' + +You should now be able to listen to the audio stream once para_server +starts streaming. To activate streaming, execute + + para_client play + +Since no playlist has been specified yet, the "dummy" mode which +selects all known audio files is activated automatically. See the +section on the REFERENCE(The audio file selector, audio file selector) +for how to use playlists and moods to specify which files should be +streamed in which order. + +*Troubleshooting* + +It did not work? To find out why, try to receive, decode and play the +stream manually using para_recv, para_filter and para_write as follows. + +For simplicity we assume that you're running Linux/ALSA and that only +MP3 files have been added to the database. + + para_recv -r 'http -i server_host' > file.mp3 + # (interrupt with CTRL+C after a few seconds) + ls -l file.mp3 # should not be empty + para_filter -f mp3dec -f wav < file.mp3 > file.wav + ls -l file.wav # should be much bigger than file.mp3 + para_write -w alsa < file.wav + +Double check what is logged by para_server and use the --loglevel +option of para_recv, para_filter and para_write to increase verbosity. + +--------------- +User management +--------------- + +para_server uses a challenge-response mechanism to authenticate +requests from incoming connections, similar to ssh's public key +authentication method. Authenticated connections are encrypted using +the RC4 stream cipher. + +In this chapter we briefly describe RSA and RC4 and sketch the +REFERENCE(Client-server authentication, authentication handshake) +between para_client and para_server. User management is discussed +in the section on REFERENCE(The user_list file, the user_list file). +These sections are all about communication between the client and the +server. Connecting para_audiod is a different matter and is described +in a REFERENCE(Connecting para_audiod, separate section). + + + +RSA and RC4 +~~~~~~~~~~~ + +RSA is an asymmetric block cipher which is used in many applications, +including ssh and gpg. An RSA key consists in fact of two keys, +called the public key and the private key. A message can be encrypted +with either key and only the counterpart of that key can decrypt +the message. While RSA can be used for both signing and encrypting +a message, paraslash only uses RSA only for the latter purpose. The +RSA public key encryption and signatures algorithms are defined in +detail in RFC 2437. + +RC4 is a stream cipher, i.e. the input is XORed with a pseudo-random +key stream to produce the output. Decryption uses the same function +calls as encryption. While RC4 supports variable key lengths, +paraslash uses a fixed length of 256 bits, which is considered a +strong encryption by today's standards. Since the same key must never +be used twice, a different, randomly-generated key is used for every +new connection. + +Client-server authentication +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The authentication handshake between para_client and para_server goes +as follows: + + - para_client connects to para_server and sends an + authentication request for a user. It does so by connecting + to para_server, TCP 2990, the control port of para_server. + + - para_server accepts the connection and forks a child process + which is supposed to handle the connection. The parent process + keeps listening on the control port while the child process + (also called para_server below) continues as follows. + + - para_server loads the RSA public key of that user, fills a + fixed-length buffer with random bytes, encrypts that buffer + using the public key and sends the encrypted buffer to the + client. The first part of the buffer is the challenge which + is used for authentication while the second part is the RC4 + session key. + + - para_client receives the encrypted buffer and decrypts it + using the user's private key, thereby obtaining the challenge + buffer and the session key. It sends the SHA1 hash value of + the challenge back to para_server and stores the session key + for further use. + + - para_server also computes the SHA1 hash of the challenge + and compares it against what was sent back by the client. + + - If the two hashes do not match, the authentication has + failed and para_server closes the connection. + + - Otherwise the user is considered authenticated and the client + is allowed to proceed by sending a command to be executed. From + this point on the communication is encrypted using the RC4 + stream cipher with the session key known to both peers. + +paraslash relies on the quality of openssl's cryptographically strong +pseudo-random bytes, on the security of the implementation of the +openssl RSA and RC4 crypto routines and on the infeasibility to invert +the SHA1 function. + +Neither para_server or para_client create RSA keys on their own. This +has to be done once for each user as sketched in REFERENCE(Quick start, +Quick start) and discussed in more detail REFERENCE(The user_list +file, below). + +The user_list file +~~~~~~~~~~~~~~~~~~ + +At startup para_server reads the user list file which must contain +one line per user. The default location of the user list file may be +changed with the --user_list option. + +There should be at least one user in this file. Each user must have +an RSA key pair. The public part of the key is needed by para_server +while the private key is needed by para_client. Each line of the +user list file must be of the form + + user + +where _username_ is an arbitrary string (usually the user's login +name), _key_ is the full path to that user's public RSA key, and +_perms_ is a comma-separated list of zero or more of the following +permission bits: + + +---------------------------------------------------------+ + | AFS_READ | read the contents of the databases | + +-----------+---------------------------------------------+ + | AFS_WRITE | change database contents | + +-----------+---------------------------------------------+ + | VSS_READ | obtain information about the current stream | + +-----------+---------------------------------------------+ + | VSS_WRITE | change the current stream | + +---------------------------------------------------------+ + +The permission bits specify which commands the user is allowed to +execute. The output of + + para_client help + +contains in the third column the permissions needed to execute the +command. + +A new RSA key can be created with + + openssl genrsa -out 2048 + +and the public part may be extracted with + + openssl rsa -in -pubout -out + +Note that para_server refuses to use a key if it is shorter than 2048 +bits. In particular, the RSA keys of paraslash 0.3.x will not work +with version 0.4.x. Moreover, para_client refuses to use a (private) +key which is world-readable. + +It is possible to make para_server reread the user_list file by +executing the paraslash "hup" command or by sending SIGHUP to the +PID of para_server. + + +Connecting para_audiod +~~~~~~~~~~~~~~~~~~~~~~ + +para_audiod listens on a Unix domain socket. Those sockets are +for local communication only, so only local users can connect to +para_audiod. The default is to let any user connect but this can be +restricted on platforms that support UNIX socket credentials which +allow para_audiod to obtain the Unix credentials of the connecting +process. + +Use para_audiod's --user_allow option to allow connections only for +a limited set of users. + +----------------------- +The audio file selector +----------------------- + +paraslash comes with a sophisticated audio file selector (AFS), +whose main task is to determine which file to stream next, based on +information on the audio files stored in a database. It communicates +also with para_client whenever an AFS command is executed, for example +to answer a database query. + +Besides the traditional playlists, AFS supports audio file selection +based on _moods_ which act as a filter that limits the set of all +known audio files to those which satisfy certain criteria. It also +maintains tables containing images (e.g. album cover art) and lyrics +that can be associated with one or more audio files. + +AFS uses libosl, the object storage layer, as the backend library +for storing information on audio files, playlists, etc. This library +offers functionality similar to a relational database, but is much +more lightweight than a full database backend. + +In this chapter we sketch the setup of the REFERENCE(The AFS process, +AFS process) during server startup and proceed with the description +of the REFERENCE(Database layout, layout) of the various database +tables. The section on REFERENCE(Playlists and moods, playlists +and moods) explains these two audio file selection mechanisms +in detail and contains pratical examples. The way REFERENCE(File +renames and content changes, file renames and content changes) are +detected is discussed briefly before the REFERENCE(Troubleshooting, +Troubleshooting) section which concludes the chapter. + +The AFS process +~~~~~~~~~~~~~~~ + +On startup, para_server forks to create the AFS process which opens +the OSL database tables. The server process communicates with the +AFS process via pipes and shared memory. Usually, the AFS process +awakes only briefly whenever the current audio file changes. The AFS +process determines the next audio file, opens it, verifies it has +not been changed since it was added to the database and passes the +open file descriptor to the server process, along with audio file +meta-data such as file name, duration, audio format and so on. The +server process then starts to stream the audio file. + +The AFS process also accepts connections from local clients via +a well-known socket. However, only child processes of para_server +may connect through this socket. All server commands that have the +AFS_READ or AFS_WRITE permission bits use this mechanism to query or +change the database. + +Database layout +~~~~~~~~~~~~~~~ + +*The audio file table* + +This is the most important and usually also the largest table of the +AFS database. It contains the information needed to stream each audio +file. In particular the following data is stored for each audio file. + + - SHA1 hash value of the audio file contents. This is computed + once when the file is added to the database. Whenever AFS + selects this audio file for streaming the hash value is + recomputed and checked against the value stored in the + database to detect content changes. + + - The time when this audio file was last played. + + - The number of times the file has been played so far. + + - The attribute bitmask. + + - The image id which describes the image associated with this + audio file. + + - The lyrics id which describes the lyrics associated with + this audio file. + + - The audio format id (MP3, OGG, AAC, WMA). + + - An amplification value that can be used by the amplification + filter to pre-amplify the decoded audio stream. + + - The chunk table. It describes the location and the timing + of the building blocks of the audio file. This is used by + para_server to send chunks of the file at appropriate times. + + - The duration of the audio file. + + - Tag information contained in the audio file (ID3 tags, + Vorbis comments, ...). + + - The number of channels + + - The encoding bitrate. + + - The sampling frequency. + +To add or refresh the data contained in the audio file table, the _add_ +command is used. It takes the full path of either an audio file or a +directory. In the latter case, the directory is traversed recursively +and all files which are recognized as valid audio files are added to +the database. + +*The attribute table* + +The attribute table contains two columns, _name_ and _bitnum_. An +attribute is simply a name for a certain bit number in the attribute +bitmask of the audio file table. + +Each of the 64 bits of the attribute bitmask can be set for each +audio file individually. Hence up to 64 different attributes may be +defined. For example, "pop", "rock", "blues", "jazz", "instrumental", +"german_lyrics", "speech", whatever. You are free to choose as +many attributes as you like and there are no naming restrictions +for attributes. + +A new attribute "test" is created by + + para_client addatt test +and + para_client lsatt + +lists all available attributes. You can set the "test" attribute for +an audio file by executing + + para_client setatt test+ /path/to/the/audio/file + +Similarly, the "test" bit can be removed from an audio file with + + para_client setatt test- /path/to/the/audio/file + +Instead of a path you may use a shell wildcard pattern. The attribute +is applied to all audio files matching that pattern: + + para_client setatt test+ '/test/directory/*' + +The command + + para_client -- ls -lv + +gives you a verbose listing of your audio files also showing which +attributes are set. + +In case you wonder why the double-dash in the above command is needed: +It tells para_client to not interpret the options after the dashes. If +you find this annoying, just say + + alias para='para_client --' + +and be happy. In what follows we shall use this alias. + +The "test" attribute can be dropped from the database with + + para rmatt test + +Read the output of + + para help ls + para help setatt + +for more information and a complete list of command line options to +these commands. + +*Blob tables* + +The image, lyrics, moods and playlists tables are all blob tables. +Blob tables consist of three columns each: The identifier which is +a positive non-negative number that is auto-incremented, the name +(an arbitrary string) and the content (the blob). + +All blob tables support the same set of actions: cat, ls, mv, rm +and add. Of course, _add_ is used for adding new blobs to the table +while the other actions have the same meaning as the corresponding +Unix commands. The paraslash commands to perform these actions are +constructed as the concatenation of the table name and the action. For +example addimg, catimg, lsimg, mvimg, rmimg are the commands that +manipulate or query the image table. + +The add variant of these commands is special as these commands read +the blob contents from stdin. To add an image to the image table the +command + + para addimg image_name < file.jpg + +can be used. + +Note that the images and lyrics are not interpreted at all, and also +the playlist and the mood blobs are only investigated when the mood +or playlist is activated by using the select command. + +*The score table* + +Unlike all other tables the contents of the score table remain in +memory and are never stored on disk. The score table contains two +columns: The SHA1 hash value (of an audio file) and its current +score. + +However, only those files which are admissible for the current mood +or playlist are contained in the score table. The audio file selector +always chooses the row with the highest score as the file to stream +next. While doing so, it computes the new score and updates the +last_played and the num_played fields in the audio file table. + +The score table is recomputed by the select command which loads a +new mood or playlist. + +Playlists and moods +~~~~~~~~~~~~~~~~~~~ + +Playlists and moods offer two different ways of specifying the set of +admissible files. A playlist in itself describes a set of admissible +files. A mood, in contrast, describes the set of admissible files in +terms of attributes and other type of information available in the +audio file table. As an example, a mood can define a filename pattern, +which is then matched against the names of audio files in the table. + +Selecting a mood or playlist means the generation of a ranking +(a score table) for the set of admissible files. Audio files are +then selected on a highest-score-first basis. The score table is +recomputed at the moment the mood or playlist is selected. + +*Playlists* + +Playlists are accommodated in the playlist table of the afs database, +using the aforementioned blob format for tables. A new filelist is +created using the addpl command, by specifying the full (absolute) +paths of all desired audio files, separated by newlines. For example + + find /my/mp3/dir -name "*.mp3" | para addpl my_playlist + +If _my_playlist_ already exists it is overwritten. To activate the +new playlist, execute + + para select p/my_playlist + +The audio file selector will assign scores to each entry of the list, +in descending order so that files will be selected in order. If a +file could not be opened for streaming, its entry is removed from +the score table (but not from the playlist). + +*Moods* + +A mood consists of a unique name and its *mood definition*, which is +a set of *mood lines* containing expressions in terms of attributes +and other data contained in the database. + +At any time, at most one mood can be *active* which means that +para_server is going to select only files from that subset of +admissible files. + +So in order to create a mood definition one has to write a set of +mood lines. Mood lines come in three flavours: Accept lines, deny +lines and score lines. + +The general syntax of the three types of mood lines is + + + accept [with score ] [if] [not] [options] + deny [with score ] [if] [not] [options] + score [if] [not] [options] + + +Here is either an integer or the string "random" which assigns +a random score to all matching files. The score value changes the +order in which admissible files are going to be selected, but is of +minor importance for this introduction. + +So we concentrate on the first two forms, i.e. accept and deny +lines. As usual, everything in square brackets is optional, i.e. +accept/deny lines take the following form when ignoring scores: + + accept [if] [not] [options] + +and analogously for the deny case. The "if" keyword is only syntactic +sugar and has no function. The "not" keyword just inverts the result, +so the essence of a mood line is the mood method part and the options +following thereafter. + +A *mood method* is realized as a function which takes an audio file +and computes a number from the data contained in the database. +If this number is non-negative, we say the file *matches* the mood +method. The file matches the full mood line if it either + + - matches the mood method and the "not" keyword is not given, +or + - does not match the mood method, but the "not" keyword is given. + +The set of admissible files for the whole mood is now defined as those +files which match at least one accept mood line, but no deny mood line. +More formally, an audio file F is admissible if and only if + + (F ~ AL1 or F ~ AL2...) and not (F ~ DL1 or F ~ DN2 ...) + +where AL1, AL2... are the accept lines, DL1, DL2... are the deny +lines and "~" means "matches". + +The cases where no mood lines of accept/deny type are defined need +special treatment: + + - Neither accept nor deny lines: This treats all files as + admissible (in fact, that is the definition of the dummy mood + which is activated automatically if no moods are available). + + - Only accept lines: A file is admissible iff it matches at + least one accept line: + + F ~ AL1 or F ~ AL2 or ... + + - Only deny lines: A file is admissible iff it matches no + deny line: + + not (F ~ DL1 or F ~ DN2 ...) + + + +*List of mood_methods* + + no_attributes_set + +Takes no arguments and matches an audio file if and only if no +attributes are set. + + is_set + +Takes the name of an attribute and matches iff that attribute is set. + + path_matches + +Takes a filename pattern and matches iff the path of the audio file +matches the pattern. + + artist_matches + album_matches + title_matches + comment_matches + +Takes an extended regular expression and matches iff the text of the +corresponding tag of the audio file matches the pattern. If the tag +is not set, the empty string is matched against the pattern. + + year ~ + bitrate ~ + frequency ~ + channels ~ + num_played ~ + +Takes a comparator ~ of the set {<, =, <=, >, >=, !=} and a number +. Matches an audio file iff the condition ~ is +satisfied where val is the corresponding value of the audio file +(value of the year tag, bitrate in kbit/s, frequency in Hz, channel +count, play count). + +The year tag is special as its value is undefined if the audio file +has no year tag or the content of the year tag is not a number. Such +audio files never match. Another difference is the special treatment +if the year tag is a two-digit number. In this case either 1900 or +2000 are added to the tag value depending on whether the number is +greater than 2000 plus the current year. + + +*Mood usage* + +To create a new mood called "my_mood", write its definition into +some temporary file, say "tmpfile", and add it to the mood table +by executing + + para addmood my_mood < tmpfile + +If the mood definition is really short, you may just pipe it to the +client instead of using temporary files. Like this: + + echo "$MOOD_DEFINITION" | para addmood my_mood + +There is no need to keep the temporary file since you can always use +the catmood command to get it back: + + para catmood my_mood + +A mood can be activated by executing + + para select m/my_mood + +Once active, the list of admissible files is shown by the ls command +if the "-a" switch is given: + + para ls -a + + +*Example mood definition* + +Suppose you have defined attributes "punk" and "rock" and want to define +a mood containing only Punk-Rock songs. That is, an audio file should be +admissible if and only if both attributes are set. Since + + punk and rock + +is obviously the same as + + not (not punk or not rock) + +(de Morgan's rule), a mood definition that selects only Punk-Rock +songs is + + deny if not is_set punk + deny if not is_set rock + + + +File renames and content changes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Since the audio file selector knows the SHA1 of each audio file that +has been added to the afs database, it recognizes if the content of +a file has changed, e.g. because an ID3 tag was added or modified. +Also, if a file has been renamed or moved to a different location, +afs will detect that an entry with the same hash value already exists +in the audio file table. + +In both cases it is enough to just re-add the new file. In the +first case (file content changed), the audio table is updated, while +metadata such as the num_played and last_played fields, as well as +the attributes, remain unchanged. In the other case, when the file +is moved or renamed, only the path information is updated, all other +data remains as before. + +It is possible to change the behaviour of the add command by using the +"-l" (lazy add) or the "-f" (force add) option. + +Troubleshooting +~~~~~~~~~~~~~~~ + +Use the debug loglevel (option -l debug for most commands) to show +debugging info. Almost all paraslash executables have a brief online +help which is displayed by using the -h switch. The --detailed-help +option prints the full help text. + +If para_server crashed or was killed by SIGKILL (signal 9), it +may refuse to start again because of "dirty osl tables". In this +case you'll have to run the oslfsck program of libosl to fix your +database. It might be necessary to use --force (even if your name +isn't Luke). However, make sure para_server isn't running before +executing oslfsck --force. + +If you don't mind to recreate your database you can start +from scratch by removing the entire database directory, i.e. + + rm -rf ~/.paraslash/afs_database-0.4 + +Be aware that this removes all attribute definitions, all playlists +and all mood definitions and requires to re-initialize the tables. + +Although oslfsck fixes inconsistencies in database tables it doesn't +care about the table contents. To check for invalid table contents, use + + para_client check + +This prints out references to missing audio files as well as invalid +playlists and mood definitions. + +--------------------------------------- +Audio formats and audio format handlers +--------------------------------------- + +Audio formats +~~~~~~~~~~~~~ + +The following audio formats are supported by paraslash: + +*MP3* + +Mp3, MPEG-1 Audio Layer 3, is a common audio format for audio storage, +designed as part of its MPEG-1 standard. An MP3 file is made up of +multiple MP3 frames, which consist of a header and a data block. The +size of an MP3 frame depends on the bit rate and on the number +of channels. For a typical CD-audio file (sample rate of 44.1 kHz +stereo), encoded with a bit rate of 128 kbit, an MP3 frame is about +400 bytes large. + +*OGG/Vorbis* + +OGG is a standardized audio container format, while Vorbis is an +open source codec for lossy audio compression. Since Vorbis is most +commonly made available via the OGG container format, it is often +referred to as OGG/Vorbis. The OGG container format divides data into +chunks called OGG pages. A typical OGG page is about 4KB large. The +Vorbis codec creates variable-bitrate (VBR) data, where the bitrate +may vary considerably. + +*AAC* + +Advanced Audio Coding (AAC) is a standardized, lossy compression +and encoding scheme for digital audio which is the default audio +format for Apple's iPhone, iPod, iTunes. Usually MPEG-4 is used as +the container format and audio files encoded with AAC have the .m4a +extension. A typical AAC frame is about 700 bytes large. + +*WMA* + +Windows Media Audio (WMA) is an audio data compression technology +developed by Microsoft. A WMA file is usually encapsulated in the +Advanced Systems Format (ASF) container format, which also specifies +how meta data about the file is to be encoded. The bit stream of WMA +is composed of superframes, each containing one or more frames of +2048 samples. For 16 bit stereo a WMA superframe is about 8K large. + +Meta data +~~~~~~~~~ + +Unfortunately, each audio format has its own conventions how meta +data is added as tags to the audio file. + +For MP3 files, ID3, version 1 and 2 are widely used. ID3 version 1 +is rather simple but also very limited as it supports only artist, +title, album, year and comment tags. Each of these can only be at most +32 characters long. ID3, version 2 is much more flexible but requires +a separate library being installed for paraslash to support it. + +Ogg vorbis files contain meta data as Vorbis comments, which are +typically implemented as strings of the form "[TAG]=[VALUE]". Unlike +ID3 version 1 tags, one may use whichever tags are appropriate for +the content. + +AAC files usually use the MPEG-4 container format for storing meta +data while WMA files wrap meta data as special objects within the +ASF container format. + +paraslash only tracks the most common tags that are supported by +all tag variants: artist, title, year, album, comment. When a file +is added to the AFS database, the meta data of the file is extracted +and stored in the audio file table. + +Chunks and chunk tables +~~~~~~~~~~~~~~~~~~~~~~~ + +paraslash uses the word "chunk" as common term for the building +blocks of an audio file. For MP3 files, a chunk is the same as an +MP3 frame, while for OGG/Vorbis files, a chunk is an OGG page, etc. +Therefore the chunk size varies considerably between audio formats, +from a few hundred bytes (MP3) up to 8K (WMA). + +The chunk table contains the offsets within the audio file that +correspond to the chunk boundaries of the file. Like the meta data, +the chunk table is computed and stored in the database whenever an +audio file is added. + +The paraslash senders (see below) always send complete chunks. The +granularity for seeking is therefore determined by the chunk size. + +Audio format handlers +~~~~~~~~~~~~~~~~~~~~~ + +For each audio format paraslash contains an audio format handler whose +first task is to tell whether a given file is a valid audio file of +this type. If so, the audio file handler extracts some technical data +(duration, sampling rate, number of channels etc.), computes the +chunk table and reads the meta data. + +The audio format handler code is linked into para_server and executed +via the _add_ command. The same code is also available as a stand-alone +tool, para_afh, which can be used to print the technical data, the +chunk table and the meta data of a file. Furthermore, one can use +para_afh to cut an audio file, i.e. to select some of its chunks to +produce a new file containing only these chunks. + +---------- +Networking +---------- + +Paraslash uses different network connections for control and data. +para_client communicates with para_server over a dedicated TCP control +connection. To transport audio data, separate data connections are +used. For these data connections, a variety of transports (UDP, DCCP, +HTTP) can be chosen. + +The chapter starts with the REFERENCE(The paraslash control +service, control service), followed by a section on the various +REFERENCE(Streaming protocols, streaming protocols) in which the data +connections are described. The way audio file headers are embedded into +the stream is discussed REFERENCE(Streams with headers and headerless +streams, briefly) before the REFERENCE(Networking examples, example +section) which illustrates typical commands for real-life scenarios. + +Both IPv4 and IPv6 are supported. + +The paraslash control service +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +para_server is controlled at runtime via the paraslash control +connection. This connection is used for server commands (play, stop, +...) as well as for afs commands (ls, select, ...). + +The server listens on a TCP port and accepts connections from clients +that connect the open port. Each connection causes the server to fork +off a client process which inherits the connection and deals with that +client only. In this classical accept/fork approach the server process +is unaffected if the child dies or goes crazy for whatever reason. In +fact, the child process can not change address space of server process. + +The section on REFERENCE(Client-server authentication, client-server +authentication) above described the early connection establishment +from the crypto point of view. Here it is described what happens +after the connection (including crypto setup) has been established. +There are four processes involved during command dispatch as sketched +in the following diagram. + +<< +
    + server_host                                   client_host
+ ~~~~~~~~~~~                                   ~~~~~~~~~~~
+
+ +-----------+             connect            +-----------+
+ |para_server|<------------------------------ |para_client|
+ +-----------+                                +-----------+
+      |                                             ^
+      |     fork   +---+                            |
+      +----------> |AFS|                            |
+      |            +---+                            |
+      |              ^                              |
+      |              |                              |
+      |              | connect (cookie)             |
+      |              |                              |
+      |              |                              |
+      |    fork   +-----+    inherited connection   |
+      +---------->|child|<--------------------------+
+                  +-----+
+
    +>> + +Note that the child process is not a child of the afs process, +so communication of these two processes has to happen via local +sockets. In order to avoid abuse of the local socket by unrelated +processes, a magic cookie is created once at server startup time just +before the server process forks off the AFS process. This cookie is +known to the server, AFS and the child, but not to unrelated processes. + +There are two different kinds of commands: First there are commands +that cause the server to respond with some answer such as the list +of all audio files. All but the addblob commands (addimg, addlyr, +addpl, addmood) are of this kind. The addblob commands add contents +to the database, so they need to transfer data the other way round, +from the client to the server. + +There is no knowledge about the server commands built into para_client, +so it does not know about addblob commands. Instead, it inspects the +first data package sent by the server for a magic string. If this +string was found, it sends STDIN to the server, otherwise it dumps +data from the server to STDOUT. + +Streaming protocols +~~~~~~~~~~~~~~~~~~~ + +A network (audio) stream usually consists of one streaming source, +the _sender_, and one or more _receivers_ which read data over the +network from the streaming source. + +Senders are thus part of para_server while receivers are part of +para_audiod. Moreover, there is the stand-alone tool para_recv which +can be used to manually download a stream, either from para_server +or from a web-based audio streaming service. + +The following three streaming protocols are supported by paraslash: + + - HTTP. Recommended for public streams that can be played by + any player like mpg123, xmms, itunes, winamp, etc. The HTTP + sender is supported on all operating systems and all platforms. + + - DCCP. Recommended for LAN streaming. DCCP is currently + available only for Linux. + + - UDP. Recommended for multicast LAN streaming. + +See the Appendix on REFERENCE(Network protocols, network protocols) +for brief descriptions of the various protocols relevant for network +audio streaming with paraslash. + +It is possible to activate more than one sender simultaneously. +Senders can be controlled at run time and via config file and command +line options. + +Note that audio connections are _not_ encrypted. Transport or Internet +layer encryption should be used if encrypted data connections are +needed. + +Since DCCP and TCP are both connection-oriented protocols, connection +establishment/teardown and access control are very similar between +these two streaming protocols. UDP is the most lightweight option, +since in contrast to TCP/DCCP it is connectionless. It is also the +only protocol supporting IP multicast. + +The HTTP and the DCCP sender listen on a (TCP/DCCP) port waiting for +clients to connect and establish a connection via some protocol-defined +handshake mechanism. Both senders maintain two linked lists each: +The list of all clients which are currently connected, and the list +of access control entries which determines who is allowed to connect. +IP-based access control may be configured through config file and +command line options and via the "allow" and "deny" sender subcommands. + +Upon receiving a GET request from the client, the HTTP sender sends +back a status line and a message. The body of this message is the +audio stream. This is common practice and is supported by many popular +clients which can thus be used to play a stream offered by para_server. +For DCCP things are a bit simpler: No messages are exchanged between +the receiver and sender. The client simply connects and the sender +starts to stream. + +DCCP is an experimental protocol which offers a number of new features +not available for TCP. Both ends can negotiate these features using +a built-in negotiation mechanism. In contrast to TCP/HTTP, DCCP is +datagram-based (no retransmissions) and thus should not be used over +lossy media (e.g. WiFi networks). One useful feature offered by DCCP +is access to a variety of different congestion-control mechanisms +called CCIDs. Two different CCIDs are available per default on Linux: + + + - _CCID 2_. A Congestion Control mechanism similar to that + of TCP. The sender maintains a congestion window and halves + this window in response to congestion. + + + - _CCID-3_. Designed to be fair when competing for bandwidth. + It has lower variation of throughput over time compared with + TCP, which makes it suitable for streaming media. + +Unlike the HTTP and DCCP senders, the UDP sender maintains only a +single list, the _target list_. This list describes the set of clients +to which the stream is sent. There is no list for access control and +no "allow" and "deny" commands for the UDP sender. Instead, the "add" +and "delete" commands can be used to modify the target list. + +Since both UDP and DCCP offer an unreliable datagram-based transport, +additional measures are necessary to guard against disruptions over +networks that are lossy or which may be subject to interference (as +is for instance the case with WiFi). Paraslash uses FEC (Forward +Error Correction) to guard against packet losses and reordering. The +stream is FEC-encoded before it is sent through the UDP socket and +must be decoded accordingly on the receiver side. + +The packet size and the amount of redundancy introduced by FEC can +be configured via the FEC parameters which are dictated by server +and may also be configured through the "sender" command. The FEC +parameters are encoded in the header of each network packet, so no +configuration is necessary on the receiver side. See the section on +REFERENCE(Forward error correction, FEC) below. + +Streams with headers and headerless streams +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For ogg vorbis and wma streams, not all information needed to decode +the stream is contained in each data chunk but only in the audio +file header of the container format. Therefore clients must be able +to obtain this information in case streaming starts in the middle of +the file or if para_audiod is started while para_server is already +sending a stream. + +This is accomplished in different ways, depending on the streaming +protocol. For connection-oriented streams (HTTP, DCCP) the audio file +header is sent prior to audio file data. This technique however does +not work for the connectionless UDP transport. Hence the audio file +header is periodically being embedded into the UDP audio data stream. +By default, the header is resent after five seconds. The receiver has +to wait until the next header arrives before it can start decoding +the stream. + +Examples +~~~~~~~~ + +The sender command of para_server allows to (de-)activate senders +and to change the access permissions senders at runtime. The "si" +(server info) command is used to list the streaming options of the +currently running server as well as the various sender access lists. + +-> Show client/target/access lists: + + para_client si + +-> Obtain general help for the sender command: + + para_client help sender + +-> Get help for a specific sender (contains further examples): + + s=http # or dccp or udp + para_client sender $s help + +By default para_server activates both the HTTP and th DCCP sender on +startup. This can be changed via command line options or para_server's +config file. + +-> List config file options for senders: + + para_server -h + +All senders share the "on" and "off" commands, so senders may be +activated and deactivated independently of each other. + +-> Switch off the http sender: + + para_client sender http off + +-> Receive a DCCP stream using CCID2 and write the output into a file: + + host=foo.org; ccid=2; filename=bar + para_recv --receiver "dccp --host $host --ccid $ccid" > $filename + +Note the quotes around the arguments for the dccp receiver. Each +receiver has its own set of command line options and its own command +line parser, so arguments for the dccp receiver must be protected +from being interpreted by para_recv. + +-> Start UDP multicast, using the default multicast address: + + para_client sender udp add 224.0.1.38 + +-> Receive FEC-encoded multicast stream and write the output into a file: + + filename=foo + para_recv -r udp > $filename + +-> Add an UDP unicast for a client to the target list of the UDP sender: + + t=client.foo.org + para_client sender udp add $t + +-> Receive this (FEC-encoded) unicast stream: + + filename=foo + para_recv -r 'udp -i 0.0.0.0' > $filename + +-> Create a minimal config for para_audiod for HTTP streams: + + c=$HOME/.paraslash/audiod.conf.min; s=server.foo.com + formats="mp3 ogg aac wma" # remove what you do not have + for f in $formats; do echo receiver \"$f:http -i $s\"; done > $c + para_audiod --config $c + +------- +Filters +------- + +A paraslash filter is a module which transforms an input stream into +an output stream. Filters are included in the para_audiod executable +and in the stand-alone tool para_filter which usually contains the +same modules. + +While para_filter reads its input stream from STDIN and writes +the output to STDOUT, the filter modules of para_audiod are always +connected to a receiver which produces the input stream and a writer +which absorbs the output stream. + +Some filters depend on a specific library being installed and are +not compiled in if this library was not found at compile time. To +see the list of supported filters, run para_filter and para_audiod +with the --help option. The output looks similar to the following: + + Available filters: + compress wav amp fecdec wmadec prebuffer oggdec aacdec mp3dec + +Out of these filter modules, a chain of filters can be constructed, +much in the way Unix pipes can be chained, and analogous to the use +of modules in gstreamer: The output of the first filter becomes the +input of the second filter. There is no limitation on the number of +filters and the same filter may occur more than once. + +Like receivers, each filter has its own command line options which +must be quoted to protect them from the command line options of +the driving application (para_audiod or para_filter). Example: + + para_filter -f 'mp3dec --ignore-crc' -f 'compress --damp 1' + +For para_audiod, each audio format has its own set of filters. The +name of the audio format for which the filter should be applied is +used as the prefix for the filter option. Example: + + para_audiod -f 'mp3:prebuffer --duration 300' + +Decoders +~~~~~~~~ + +For each supported audio format there is a corresponding filter +which decodes audio data in this format to 16 bit PCM data which +can be directly sent to the sound device or any other software that +operates on undecoded PCM data (visualizers, equalizers etc.). Such +filters are called _decoders_ in general, and xxxdec is the name of +the paraslash decoder for the audio format xxx. For example, the mp3 +decoder filter is called mp3dec. + +Note that the output of the decoder is about 10 times larger than +its input. This means that filters that operate on the decoded audio +stream have to deal with much more data than filters that transform +the audio stream before it is fed to the decoder. + +Paraslash relies on external libraries for most decoders, so these +libraries must be installed for the decoder to be included in the +para_filter and para_audiod executables. The oggdec filter depends +on the libogg and libvorbis libraries for example. + +Forward error correction +~~~~~~~~~~~~~~~~~~~~~~~~ + +As already mentioned REFERENCE(Streaming protocols, earlier), +paraslash uses forward error correction (FEC) for the unreliable +UDP transport. FEC is a technique which was invented already in +1960 by Reed and Solomon and which is widely used for the parity +calculations of storage devices (RAID arrays). It is based on the +algebraic concept of finite fields, today called Galois fields, in +honour of the mathematician Galois (1811-1832). The FEC implementation +of paraslash is based on code by Luigi Rizzo. + +Although the details require a sound knowledge of the underlying +mathematics, the basic idea is not hard to understand: For positive +integers k and n with k < n it is possible to compute for any k given +data bytes d_1, ..., d_k the corresponding r := n -k parity bytes p_1, +..., p_r such that all data bytes can be reconstructed from *any* +k bytes of the set + + {d_1, ..., d_k, p_1, ..., p_r}. + +FEC-encoding for unreliable network transports boils down to slicing +the audio stream into groups of k suitably sized pieces called _slices_ +and computing the r corresponding parity slices. This step is performed +in para_server which then sends both the data and the parity slices +over the unreliable network connection. If the client was able +to receive at least k of the n = k + r slices, it can reconstruct +(FEC-decode) the original audio stream. + +From these observations it is clear that there are three different +FEC parameters: The slice size, the number of data slices k, and the +total number of slices n. It is crucial to choose the slice size +such that no fragmentation of network packets takes place because +FEC only guards against losses and reodering but fails if slices are +received partially. + +FEC decoding in paralash is performed through the fecdec filter which +usually is the first filter (there can be other filters before fecdec +if these do not alter the audio stream). + + +Volume adjustment (amp and compress) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The amp and the compress filter both adjust the volume of the audio +stream. These filters operate on uncompressed audio samples. Hence +they are usually placed directly after the decoding filter. Each +sample is multiplied with a scaling factor (>= 1) which makes amp +and compress quite expensive in terms of computing power. + +*amp* + +The amp filter amplifies the audio stream by a fixed scaling factor +that must be known in advance. For para_audiod this factor is derived +from the amplification field of the audio file's entry in the audio +file table while para_filter uses the value given at the command line. + +The optimal scaling factor F for an audio file is the largest real +number F >= 1 such that after multiplication with F all samples still +fit into the sample interval [-32768, 32767]. One can use para_filter +in combination with the sox utility to compute F: + + para_filter -f mp3dec -f wav < file.mp3 | sox -t wav - -e stat -v + +The amplification value V which is stored in the audio file table, +however, is an integer between 0 and 255 which is connected to F +through the formula + + V = (F - 1) * 64. + +To store V in the audio file table, the command + + para_client -- touch -a=V file.mp3 + +is used. The reader is encouraged to write a script that performs +these computations :) + +*compress* + +Unlike the amplification filter, the compress filter adjusts the volume +of the audio stream dynamically without prior knowledge about the peak +value. It maintains the maximal volume of the last n samples of the +audio stream and computes a suitable amplification factor based on that +value and the various configuration options. It tries to chose this +factor such that the adjusted volume meets the desired target level. + +Note that it makes sense to combine amp and compress. + +Misc filters (wav and prebuffer) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These filters are rather simple and do not modify the audio stream at +all. The wav filter is only useful with para_filter and in connection +with a decoder. It asks the decoder for the number of channels and the +sample rate of the stream and adds a Microsoft wave header containing +this information at the beginning. This allows to write wav files +rather than raw PCM files (which do not contain any information about +the number of channels and the sample rate). + +The prebuffer filter simply delays the output until the given time has +passed (starting from the time the first byte was available in its +input queue) or until the given amount of data has accumulated. It +is mainly useful for para_audiod if the standard parameters result +in buffer underruns. + +Both filters require almost no additional computing time, even when +operating on uncompressed audio streams, since data buffers are simply +"pushed down" rather than copied. + +Examples +~~~~~~~~ + +-> Decode an mp3 file to wav format: + + para_filter -f mp3dec -f wav < file.mp3 > file.wav + +-> Amplify a raw audio file by a factor of 1.5: + + para_filter -f amp --amp 32 < foo.raw > bar.raw + +------ +Output +------ + +Once an audio stream has been received and decoded to PCM format, +it can be sent to a sound device for playback. This part is performed +by paraslash _writers_ which are described in this chapter. + +Writers +~~~~~~~ + +A paraslash writer acts as a data sink that consumes but does not +produce audio data. Paraslash writers operate on the client side and +are contained in para_audiod and in the stand-alone tool para_write. + +The para_write program reads uncompressed 16 bit audio data from +STDIN. If this data starts with a wav header, sample rate and channel +count are read from the header. Otherwise CD audio (44.1KHz stereo) +is assumed but this can be overridden by command line options. +para_audiod, on the other hand, obtains the sample rate and the number +of channels from the decoder. + +Like receivers and filters, each writer has an individual set of +command line options, and for para_audiod writers can be configured +per audio format separately. It is possible to activate more than +one writer for the same stream simultaneously. + +OS-dependent APIs +~~~~~~~~~~~~~~~~~ + +Unfortunately, the various flavours of Unix on which paraslash +runs on have different APIs for opening a sound device and starting +playback. Hence for each such API there is a paraslash writer that +can play the audio stream via this API. + +*ALSA*. The _Advanced Linux Sound Architecture_ is only available on +Linux systems. Although there are several mid-layer APIs in use by +the various Linux distributions (ESD, Jack, PulseAudio), paraslash +currently supports only the low-level ALSA API which is not supposed +to be change. ALSA is very feature-rich, in particular it supports +software mixing via its DMIX plugin. ALSA is the default writer on +Linux systems. + +*OSS*. The _Open Sound System_ is the only API on *BSD Unixes and +is also available on Linux systems, usually provided by ALSA as an +emulation for backwards compatibility. This API is rather simple but +also limited. For example only one application can open the device +at any time. The OSS writer is activated by default on BSD Systems. + +*OSX*. Mac OS X has yet another API called CoreAudio. The OSX writer +for this API is only compiled in on such systems and is of course +the default there. + +*FILE*. The file writer allows to capture the audio stream and +write the PCM data to a file on the file system rather than playing +it through a sound device. It is supported on all platforms and is +always compiled in. + +Examples +~~~~~~~~ + +-> Use the OSS writer to play a wav file: + + para_write --writer oss < file.wav + +-> Enable ALSA software mixing for mp3 streams + + para_audiod --writer 'mp3:alsa -d plug:swmix' + + +--- +Gui +--- + +para_gui executes an arbitrary command which is supposed to print +status information to STDOUT. It then displays this information in +a curses window. By default the command + + para_audioc -- stat -p + +is executed, but this can be customized via the --stat_cmd option. In +particular it possible to use + + para_client -- stat -p + +to make para_gui work on systems on which para_audiod is not running. + +Key bindings +~~~~~~~~~~~~ + +It is possible to bind keys to arbitrary commands via custom +key-bindings. Besides the internal keys which can not be changed (help, +quit, loglevel, version...), the following flavours of key-bindings +are supported: + + - external: Shutdown curses before launching the given command. + Useful for starting other ncurses programs from within + para_gui, e.g. aumix or dialog scripts. Or, use the mbox + output format to write a mailbox containing one mail for each + (admissible) file the audio file selector knows about. Then + start mutt from within para_gui to browse your collection! + + - display: Launch the command and display its stdout in + para_gui's bottom window. + + - para: Like display, but start "para_client " instead of "". + +The general form of a key binding is + + key_map k:m:c + +which maps key k to command c using mode m. Mode may be x, d or p +for external, display and paraslash commands, respectively. + +Themes +~~~~~~ + +Currently there are only two themes for para_gui. It is easy, however, +to add more themes. To create a new theme one has to define the +position, color and geometry for for each status item that should be +shown by this theme. See gui_theme.c for examples. + +The "." and "," keys are used to switch between themes. + +Examples +~~~~~~~~ + +-> Show server info: + + key_map "i:p:si" + +-> Jump to the middle of the current audio file by pressing F5: + + key_map ":p:jmp 50" + +-> vi-like bindings for jumping around: + + key_map "l:p:ff 10" + key_map "h:p:ff 10-" + key_map "w:p:ff 60" + key_map "b:p:ff 60-" + +-> Print the current date and time: + + key_map "D:d:date" + +-> Call other curses programs: + + key_map "U:x:aumix" + key_map "!:x:/bin/bash" + key_map "^E:x:/bin/sh -c 'vi ~/.paraslash/gui.conf'" + +----------- +Development +----------- + +Tools +~~~~~ + +In order to compile the sources from the git repository (rather than +from tar balls) and for contributing non-trivial changes to the +paraslash project, some additional tools should be installed on a +developer machine. + +http://git.or.cz/ (git). As described in more detail REFERENCE(Git +branches, below), the git source code management tool is used for +paraslash development. It is necessary for cloning the git repository +and for getting updates. + +ftp://ftp.gnu.org/pub/gnu/gengetopt/ (gengetopt). The C code for +the command line parsers of all paraslash executables is generated +by gengetopt. The generated C files are shipped in the tarballs but +are not contained in the git repository. + +ftp://ftp.gnu.org/pub/gnu/m4/ (m4). Some input files for gengetopt +are generated from templates by the m4 macro processor. + +ftp://ftp.gnu.org/pub/gnu/autoconf/ (autoconf) GNU autoconf creates +the configure file which is shipped in the tarballs but has to be +generated when compiling from git. + +http://www.triptico.com/software/grutatxt.html (grutatxt). The +HTML version of this manual and some of the paraslash web pages are +generated by the grutatxt plain text to HTML converter. If changes +are made to these text files the grutatxt package must be installed +to regenerate the HTML files. + +http://www.stack.nl/~dimitri/doxygen/ (doxygen). The documentation +of paraslash's C sources uses the doxygen documentation system. The +conventions for documenting the source code is described in the +REFERENCE(Doxygen, Doxygen section). + +ftp://ftp.gnu.org/pub/gnu/global (global). This is used to generate +browsable HTML from the C sources. It is needed by doxygen. + +Git branches +~~~~~~~~~~~~ + +Paraslash has been developed using the git source code management +tool since 2006. Development is organized roughly in the same spirit +as the git development itself, as described below. + +The following text passage is based on "A note from the maintainer", +written by Junio C Hamano, the maintainer of git. + +There are four branches in the paraslash repository that track the +source tree: "master", "maint", "next", and "pu". + +The "master" branch is meant to contain what is well tested and +ready to be used in a production setting. There could occasionally be +minor breakages or brown paper bag bugs but they are not expected to +be anything major, and more importantly quickly and easily fixable. +Every now and then, a "feature release" is cut from the tip of this +branch, named with three dotted decimal digits, like 0.4.2. + +Whenever changes are about to be included that will eventually lead to +a new major release (e.g. 0.5.0), a "maint" branch is forked off from +"master" at that point. Obvious, safe and urgent fixes after the major +release are applied to this branch and maintenance releases are cut +from it. New features never go to this branch. This branch is also +merged into "master" to propagate the fixes forward. + +A trivial and safe enhancement goes directly on top of "master". +New development does not usually happen on "master", however. +Instead, a separate topic branch is forked from the tip of "master", +and it first is tested in isolation; Usually there are a handful such +topic branches that are running ahead of "master". The tip of these +branches is not published in the public repository, to keep the number +of branches that downstream developers need to worry about low. + +The quality of topic branches varies widely. Some of them start out as +"good idea but obviously is broken in some areas" and then with some +more work become "more or less done and can now be tested by wider +audience". Luckily, most of them start out in the latter, better shape. + +The "next" branch is to merge and test topic branches in the latter +category. In general, this branch always contains the tip of "master". +It might not be quite rock-solid production ready, but is expected to +work more or less without major breakage. The maintainer usually uses +the "next" version of paraslash for his own pleasure, so it cannot +be _that_ broken. The "next" branch is where new and exciting things +take place. + +The two branches "master" and "maint" are never rewound, and "next" +usually will not be either (this automatically means the topics that +have been merged into "next" are usually not rebased, and you can find +the tip of topic branches you are interested in from the output of +"git log next"). You should be able to safely build on top of them. + +The "pu" (proposed updates) branch bundles the remainder of the +topic branches. The "pu" branch, and topic branches that are only in +"pu", are subject to rebasing in general. By the above definition +of how "next" works, you can tell that this branch will contain quite +experimental and obviously broken stuff. + +When a topic that was in "pu" proves to be in testable shape, it +graduates to "next". This is done with + + git checkout next + git merge that-topic-branch + +Sometimes, an idea that looked promising turns out to be not so good +and the topic can be dropped from "pu" in such a case. + +A topic that is in "next" is expected to be polished to perfection +before it is merged to "master". Similar to the above, this is +done with + + git checkout master + git merge that-topic-branch + git branch -d that-topic-branch + +Note that being in "next" is not a guarantee to appear in the next +release (being in "master" is such a guarantee, unless it is later +found seriously broken and reverted), nor even in any future release. + +Coding Style +~~~~~~~~~~~~ + +The preferred coding style for paraslash coincides more or less +with the style of the Linux kernel. So rather than repeating what is +written XREFERENCE(http://www.kernel.org/doc/Documentation/CodingStyle, +there), here are the most important points. + + - Burn the GNU coding standards. + - Never use spaces for indentation. + - Tabs are 8 characters, and thus indentations are also 8 characters. + - Don't put multiple assignments on a single line. + - Avoid tricky expressions. + - Don't leave whitespace at the end of lines. + - The limit on the length of lines is 80 columns. + - Use K&R style for placing braces and spaces: + + if (x is true) { + we do y + } + + - Use a space after (most) keywords. + - Do not add spaces around (inside) parenthesized expressions. + - Use one space around (on each side of) most binary and ternary operators. + - Do not use cute names like ThisVariableIsATemporaryCounter, call it tmp. + - Mixed-case names are frowned upon. + - Descriptive names for global variables are a must. + - Avoid typedefs. + - Functions should be short and sweet, and do just one thing. + - The number of local variables shouldn't exceed 10. + - Gotos are fine if they improve readability and reduce nesting. + - Don't use C99-style "// ..." comments. + - Names of macros defining constants and labels in enums are capitalized. + - Enums are preferred when defining several related constants. + - Always use the paraslash wrappers for allocating memory. + - If the name of a function is an action or an imperative. + command, the function should return an error-code integer + (<0 means error, >=0 means success). If the name is a + predicate, the function should return a "succeeded" boolean. + + +Doxygen +~~~~~~~ + +Doxygen is a documentation system for various programming +languages. The paraslash project uses Doxygen for generating the API +reference on the web pages, but good source code documentation is +also beneficial to people trying to understand the code structure +and the interactions between the various source files. + +It is more illustrative to look at the source code for examples than +to describe the conventions for documenting the source in this manual, +so we only describe which parts of the code need doxygen comments, +but leave out details on documentation conventions. + +As a rule, only the public part of the C source is documented with +Doxygen. This includes structures, defines and enumerations in header +files as well as public (non-static) C functions. These should be +documented completely. For example each parameter and the return +value of a public function should get a descriptive comment. + +No doxygen comments are necessary for static functions and for +structures and enumerations in C files (which are used only within +this file). This does not mean, however, that those entities need +no documentation at all. Instead, common sense should be applied to +document what is not obvious from reading the code. + +-------- +Appendix +-------- + +Network protocols +~~~~~~~~~~~~~~~~~ + +*IP*. The _Internet Protocol_ is the primary networking protocol +used for the Internet. All protocols described below use IP as the +underlying layer. Both the prevalent IPv4 and the next-generation +IPv6 variant are being deployed actively worldwide. + +*Connection-oriented and connectionless protocols*. Connectionless +protocols differ from connection-oriented ones in that state +associated with the sending/receiving endpoints is treated +implicitly. Connectionless protocols maintain no internal knowledge +about the state of the connection. Hence they are not capable of +reacting to state changes, such as sudden loss or congestion on the +connection medium. Connection-oriented protocols, in contrast, make +this knowledge explicit. The connection is established only after +a bidirectional handshake which requires both endpoints to agree +on the state of the connection, and may also involve negotiating +specific parameters for the particular connection. Maintaining an +up-to-date internal state of the connection also in general means +that the sending endpoints perform congestion control, adapting to +qualitative changes of the connection medium. + +*Reliability*. In IP networking, packets can be lost, duplicated, +or delivered out of order, and different network protocols handle +these problems in different ways. We call a transport-layer protocol +_reliable_, if it turns the unreliable IP delivery into an ordered, +duplicate- and loss-free delivery of packets. Sequence numbers +are used to discard duplicates and re-arrange packets delivered +out-of-order. Retransmission is used to guarantee loss-free +delivery. Unreliable protocols, in contrast, do not guarantee ordering +or data integrity. + +*Classification*. With these definitions the protocols which are used +by paraslash for steaming audio data may be classified as follows. + + - HTTP/TCP: connection-oriented, reliable, + - UDP: connectionless, unreliable, + - DCCP: connection-oriented, unreliable. + +Below we give a short descriptions of these protocols. + +*TCP*. The _Transmission Control Protocol_ provides reliable, +ordered delivery of a stream and a classic window-based congestion +control. In contrast to UDP and DCCP (see below), TCP does not have +record-oriented or datagram-based syntax, i.e. it provides a stream +which is unaware and independent of any record (packet) boundaries. +TCP is used extensively by many application layers. Besides HTTP (the +Hypertext Transfer Protocol), also FTP (the File Transfer protocol), +SMTP (Simple Mail Transfer Protocol), SSH (Secure Shell) all sit on +top of TCP. + +*UDP*. The _User Datagram Protocol_ is the simplest transport-layer +protocol, built as a thin layer directly on top of IP. For this reason, +it offers the same best-effort service as IP itself, i.e. there is no +detection of duplicate or reordered packets. Being a connectionless +protocol, only minimal internal state about the connection is +maintained, which means that there is no protection against packet +loss or network congestion. Error checking and correction (if at all) +are performed in the application.' + +*DCCP*. The _Datagram Congestion Control Protocol_ combines the +connection-oriented state maintenance known from TCP with the +unreliable, datagram-based transport of UDP. This means that it +is capable of reacting to changes in the connection by performing +congestion control, offering multiple alternative approaches. But it +is bound to datagram boundaries (the maximum packet size supported +by a medium), and like UDP it lacks retransmission to protect +against loss. Due to the use of sequence numbers, it is however +able to react to loss (interpreted as a congestion indication) and +to ignore out-of-order and duplicate packets. Unlike TCP it allows +to negotiate specific, binding features for a connection, such as +the choice of congestion control: classic, window-based congestion +control known from TCP is available as CCID-2, rate-based, "smooth" +congestion control is offered as CCID-3. + +*HTTP*. _The Hypertext Transfer Protocol_ is an application layer +protocol on top of TCP. It is spoken by web servers and is most often +used for web services. However, as can be seen by the many Internet +radio stations and YouTube/Flash videos, http is by far not limited to +the delivery of web pages only. Being a simple request/response based +protocol, the semantics of the protocol also allow the delivery of +multimedia content, such as audio over http. + +*Multicast*. IP multicast is not really a protocol but a technique +for one-to-many communication over an IP network. The challenge is to +deliver information to a group of destinations simultaneously using +the most efficient strategy to send the messages over each link of +the network only once. This has benefits for streaming multimedia: +the standard one-to-one unicast offered by TCP/DCCP means that +n clients listening to the same stream also consume n-times the +resources, whereas multicast requires to send the stream just once, +irrespective of the number of receivers. Since it would be costly to +maintain state for each listening receiver, multicast often implies +connectionless transport, which is the reason that it is currently +only available via UDP. + +License +~~~~~~~ + +Paraslash is licensed under the GPL, version 2. Most of the code +base has been written from scratch, and those parts are GPL V2 +throughout. Notable exceptions are FEC and the WMA decoder. See the +corresponding source files for licencing details for these parts. Some +code sniplets of several other third party software packages have +been incorporated into the paraslash sources, for example log message +coloring was taken from the git sources. These third party software +packages are all published under the GPL or some other license +compatible to the GPL. + +Acknowledgements +~~~~~~~~~~~~~~~~ + +Many thanks to Gerrit Renker who read an early draft of this manual +and contributed significant improvements. + +---------- +References +---------- + +Articles +~~~~~~~~ + - Reed, Irving S.; Solomon, Gustave (1960), + XREFERENCE(http://kom.aau.dk/~heb/kurser/NOTER/KOFA01.PDF, + Polynomial Codes over Certain Finite Fields), Journal of the + Society for Industrial and Applied Mathematics (SIAM) 8 (2): + 300-304, doi:10.1137/0108018) + +RFCs +~~~~ + + - XREFERENCE(http://www.ietf.org/rfc/rfc768.txt, RFC 768) (1980): + User Datagram Protocol + - XREFERENCE(http://www.ietf.org/rfc/rfc791.txt, RFC 791) (1981): + Internet Protocol + - XREFERENCE(http://www.ietf.org/rfc/rfc2437.txt, RFC 2437) (1998): + RSA Cryptography Specifications + - XREFERENCE(http://www.ietf.org/rfc/rfc4340.txt, RFC 4340) + (2006): Datagram Congestion Control Protocol (DCCP) + - XREFERENCE(http://www.ietf.org/rfc/rfc4341.txt, RFC 4341) (2006): + Congestion Control ID 2: TCP-like Congestion Control + - XREFERENCE(http://www.ietf.org/rfc/rfc4342.txt, RFC 4342) (2006): + Congestion Control ID 3: TCP-Friendly Rate Control (TFRC) + +Application web pages +~~~~~~~~~~~~~~~~~~~~~ + + - XREFERENCE(http://paraslash.systemlinux.org/, paraslash) + - XREFERENCE(http://xmms2.org/wiki/Main_Page, xmms) + - XREFERENCE(http://www.mpg123.de/, mpg123) + - XREFERENCE(http://gstreamer.freedesktop.org/, gstreamer) + - XREFERENCE(http://www.icecast.org/, icecast) + - XREFERENCE(http://beesbuzz.biz/code/audiocompress.php, Audio Compress) + +External documentation +~~~~~~~~~~~~~~~~~~~~~~ + + - XREFERENCE(http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf, + H. Peter Anvin: The mathematics of Raid6) + - XREFERENCE(http://info.iet.unipi.it/~luigi/fec_ccr.ps.gz, + Luigi Rizzo: Effective Erasure Codes for reliable Computer + Communication Protocols) + +Code +~~~~ + - XREFERENCE(http://info.iet.unipi.it/~luigi/vdm.tar.gz, + Original FEC implementation by Luigi Rizzo) +