Main Body

3 Input and Output: Redirection and Pipes

By design most Unix commands are small and simple in their functionality.  To solve beyond the trivial requires the use of several steps or commands.  How to sequence and combine commands in Unix requires an understanding of how input and output is managed.  The next sections will introduce these concepts and show how more complex problems may be solved.

Filters

One may think of a filter as a black box with an input and an output.  Most Unix commands can be thought of as a filter.  The inputs and outputs have been given formal names. The input is named standard input (STDIN); the output is named standard output (STDOUT); and there is a secondary output named standard error  (STDERR) which will be discussed in more detail later.  These inputs and outputs are associated with file descriptors or stream numbers 0, 1, and 2 respectively.

By default, Unix commands read from standard input and print to standard output.   Any error messages are sent to standard error

arrow on left points from standard input to box containing word "filter"; two arrows project from right of box 1. standard output 2. standard error

Examples of Unix commands as filters

Command

Description

cut

reads from standard input and passes selected portions (columns, fields) to standard output

grep

reads from standard input and prints matching lines to standard output

head & tail

reads from standard input and prints the first (last) few lines to standard output

cat 

a transparent filter: reads from standard input and prints the same to standard output

wc

reads from standard input and prints summary information to standard output

...

not an exhaustive list

Redirection

When working interactively in a Unix session, the default setup is to have standard input draw from the keyboard, and standard output (as well as standard error) directed at the screen.  In other words, a command reading from standard input will wait for keystrokes. A command printing to standard output will have its output appear on the terminal screen.

arrow points from keyboard to filter box; two arrows emanate from filter both directed at screen
Default association of keyboard and screen to filter.

While this arrangement works well, there will come situations where the user will want to save the output of a command to a file, or substitute a file for keyboard input.  This is accomplished through the concept of redirection where one of the inputs or outputs is associated with a file.

Redirection of standard output ( > operator)

 The user can save the standard output of any Unix command by redirecting standard output to a file using the > operator.  Graphically, the concept is illustrated as follows.

arrow points from keyboard to filter box; standard output arrow aims at file; standard error arrow aims at screen.
Redirection of Standard Output
Operator syntax

Examples and explanation

cmd > some_file

e.g.

cat chapter1 > book
  •  
    if the file already exists it will overwrite it
cmd >> some_file

e.g.

cat chapter3 >> book
  • the >> operator appends an existing file

Redirecting standard error ( 2> operator)

The user can save the standard error of any Unix command by redirecting standard error to a file using the 2> operator.  Graphically, the concept is illustrated as follows.

arrow points from keyboard to filter box; standard output arrow aims at screen; standard error arrow aims at file.
Redirection of Standard Error
Operator syntax

Examples and explanation

cmd 2> some_file
e.g.
cat chapter1 chapter4 2> errors

 

  • saves error messages in file errors but standard output is displayed on the screen
cmd > some_file 2> another_file
e.g.
cat chapter1 > book 2> errors

 

  • saves standard output to file book, and saves standard error messages in file errors

Merging two streams ( >& operator)

 The user can save both standard output and standard error of any Unix command.  This is accomplished by first redirecting standard error to a file, and then merging standard error with standard output.  The syntax of the merge operator is m>&n where stream m is merged with wherever stream n is already going.   Graphically, the concept is illustrated as follows.  Here stream 2 (standard error) is merged with stream 1 (standard output).

 arrow points from keyboard to filter box; two arrows emanate from filter both directed to a plus symbol showing a merge; one arrow goes from plus symbol to file.

Standard Output and Standard Error are merged (combined) and re-directed to a file

Operator syntax

Examples and explanation

cmd > out_file 2>&1
e.g.
cat text1 junk text3 > both 2>&1
  • saves standard output and standard error to file both
  • Explanation: Unix parses left to right.  First standard output is redirected to file “both”.  Then the merge operator (>&) merges (blends) standard error (stream 2) with where standard output (stream 1) is already going (to file “both”).
  • Caution: How about
    cat text1 junk text3 > both 2> both
  • This is an incorrect method of trying to merge two streams.  This causes a race condition (two streams competing for the same file) and correct results are not guaranteed.

Redirecting standard input ( < operator)

The user can redirect standard input from a file instead of the keyboard to any Unix command using the < operator.  Graphically, the concept is illustrated as follows.

arrow points from file to filter box; two arrows emanate from filter both directed at screen
Redirection of Standard Input from a file
Operator syntax

Examples and explanation

cmd < some_file
cat < appendix

While it may appear that the “<” in the above command does not seem to do anything (works the same without the “<“), the reason is that the cat command is smart and knows to look for input on the command line.  Unless a command is specifically designed to inspect the command line for input arguments, it is necessary to use the “<” for redirection of standard input.  

Consider a more basic example of a simple script requesting input from the keyboard.  To substitute a file, it would be necessary to issue the following command, where the file “keystrokes” contains what the user would have typed.

myscript < keystrokes

Here “myscript” represents a user-written Unix script (program), not a Unix command.

Pipes

It is often the case that a problem in Unix is solved with multiple commands.  Typically the output of the first command is saved in a file which is then used as input to a subsequent command.  The use of a pipe is considered a refinement of this approach potentially simplifying the solution.

Problem: To determine the number of entries in a directory

Method 1

Graphical view

code

explanation

ls ->
file_list
-> wc

ls /etc > file_list
wc -l file_list
rm file_list
  • save output of ls command in file_list
  • run wc command to count lines
  • delete temporary file file_list

Method 2

Graphical view

code

explanation

ls ->

 -> wc

 

ls /etc | wc -l
  • save output of ls command is sent directly to input of wc command
  • no temporary file needed

 

Definition:

A pipe connects STDOUT of previous command to STDIN of next command

You can use a pipe multiple times creating a pipeline.

e.g.
cmd1 | cmd2 | cmd3 | cmd4

Building a pipeline should be an iterative process. Condense stepwise as you know the solutions work, otherwise there might be errors that might be difficult to detect from a single pipeline

Start out like this:


cmd1 > out1
cmd2 < out1 > out2
cmd3 < out2

 

Key Takeaways

  1.  Redirection: Use between a command and a file
  2.  Pipe: Use between commands

 Making your script executable

  1. create file containing unix commands
  2. chmod u+x myscript (once per file)
  3. To run, type ./myscript

 

 

License

Share This Book