Main Body
3 Input and Output: Redirection and Pipes
By design most Unix commands are small and simple in their functionality. To solve beyond the trivial requires the use of several steps or commands. How to sequence and combine commands in Unix requires an understanding of how input and output is managed. The next sections will introduce these concepts and show how more complex problems may be solved.
Filters
One may think of a filter as a black box with an input and an output. Most Unix commands can be thought of as a filter. The inputs and outputs have been given formal names. The input is named standard input (STDIN); the output is named standard output (STDOUT); and there is a secondary output named standard error (STDERR) which will be discussed in more detail later. These inputs and outputs are associated with file descriptors or stream numbers 0, 1, and 2 respectively.
By default, Unix commands read from standard input and print to standard output. Any error messages are sent to standard error
Examples of Unix commands as filters
| Command |
Description |
|---|---|
cut
|
reads from standard input and passes selected portions (columns, fields) to standard output |
grep
|
reads from standard input and prints matching lines to standard output |
head & tail
|
reads from standard input and prints the first (last) few lines to standard output |
cat
|
a transparent filter: reads from standard input and prints the same to standard output |
wc
|
reads from standard input and prints summary information to standard output |
...
|
not an exhaustive list |
Redirection
When working interactively in a Unix session, the default setup is to have standard input draw from the keyboard, and standard output (as well as standard error) directed at the screen. In other words, a command reading from standard input will wait for keystrokes. A command printing to standard output will have its output appear on the terminal screen.
While this arrangement works well, there will come situations where the user will want to save the output of a command to a file, or substitute a file for keyboard input. This is accomplished through the concept of redirection where one of the inputs or outputs is associated with a file.
Redirection of standard output ( > operator)
The user can save the standard output of any Unix command by redirecting standard output to a file using the > operator. Graphically, the concept is illustrated as follows.
| Operator syntax |
Examples and explanation |
|---|---|
cmd > some_file
|
e.g. cat chapter1 > book
|
cmd >> some_file
|
e.g. cat chapter3 >> book
|
Redirecting standard error ( 2> operator)
The user can save the standard error of any Unix command by redirecting standard error to a file using the 2> operator. Graphically, the concept is illustrated as follows.
| Operator syntax |
Examples and explanation |
|---|---|
cmd 2> some_file
|
e.g. cat chapter1 chapter4 2> errors
|
cmd > some_file 2> another_file
|
e.g.
cat chapter1 > book 2> errors
|
Merging two streams ( >& operator)
The user can save both standard output and standard error of any Unix command. This is accomplished by first redirecting standard error to a file, and then merging standard error with standard output. The syntax of the merge operator is m>&n where stream m is merged with wherever stream n is already going. Graphically, the concept is illustrated as follows. Here stream 2 (standard error) is merged with stream 1 (standard output).
Standard Output and Standard Error are merged (combined) and re-directed to a file
| Operator syntax |
Examples and explanation |
|---|---|
cmd > out_file 2>&1
|
e.g.
cat text1 junk text3 > both 2>&1
|
Redirecting standard input ( < operator)
The user can redirect standard input from a file instead of the keyboard to any Unix command using the < operator. Graphically, the concept is illustrated as follows.
| Operator syntax |
Examples and explanation |
|---|---|
cmd < some_file
|
cat < appendix
While it may appear that the “<” in the above command does not seem to do anything (works the same without the “<“), the reason is that the cat command is smart and knows to look for input on the command line. Unless a command is specifically designed to inspect the command line for input arguments, it is necessary to use the “<” for redirection of standard input. Consider a more basic example of a simple script requesting input from the keyboard. To substitute a file, it would be necessary to issue the following command, where the file “keystrokes” contains what the user would have typed. myscript < keystrokes
Here “myscript” represents a user-written Unix script (program), not a Unix command. |
Pipes
It is often the case that a problem in Unix is solved with multiple commands. Typically the output of the first command is saved in a file which is then used as input to a subsequent command. The use of a pipe is considered a refinement of this approach potentially simplifying the solution.
Problem: To determine the number of entries in a directory
Method 1
|
Graphical view |
code |
explanation |
|---|---|---|
|
ls -> |
ls /etc > file_list wc -l file_list rm file_list |
|
Method 2
|
Graphical view |
code |
explanation |
|---|---|---|
ls -> -> wc |
ls /etc | wc -l
|
|
Definition:
A pipe connects STDOUT of previous command to STDIN of next command
You can use a pipe multiple times creating a pipeline.
e.g.
cmd1 | cmd2 | cmd3 | cmd4
Building a pipeline should be an iterative process. Condense stepwise as you know the solutions work, otherwise there might be errors that might be difficult to detect from a single pipeline
Start out like this:
cmd1 > out1
cmd2 < out1 > out2
cmd3 < out2
…
Key Takeaways
- Redirection: Use between a command and a file
- Pipe: Use between commands
Making your script executable
- create file containing unix commands
- Once per file, type either:
- chmod u+x myscript
- chmod 700 myscript
- To run, type ./myscript
