Unix is primarily a command line oriented operating system.  Most commands are an action which is performed on an object, typically a file or directory.  In order to be productive in a Unix environment, one must be intimately familiar with the concept of the hierarchical file system.  Mastering this concept is fundamental to successful work in a Unix environment and is the key takeaway of this section.

Hierarchical File System

The Unix file system is that of an inverted tree.  Imagine a tree with leaves and branches but turned upside down with the root or main trunk at the top.  The main trunk branches off to smaller branches and eventually leaves.  By analogy, the root directory typically contains several directories (folders) which in turn contain other directories (subdirectories) and/or files.  Just as a tree branch can have smaller branches or leaves, a leaf cannot have other branches or leaves.  Similarly, the difference between a directory and a file is that a directory can contain other directories or files, but a file cannot contain other directories or files -- a file is a terminal node.

The figure below shows the typical layout of a Unix file system.  While systems vary greatly in size, most will have at least these directories.  The convention in this diagram uses an ellipse to depict a directory and a rectangle to depict a file.

Explanation of common subdirectories:

How to Specify a File or Directory in Unix

As most Unix commands act on files or directories, it is necessary to be able to specify such an entity.

Absolute

To specify a file or directory using the absolute method, start with the root directory (/) and write each directory that is encountered on the path to the directory or file being specified.  Separate each directory with a "/" forward slash character.

/etc

/etc/passwd

/bin/ls

/users/raj/work/mydata

Relative

When working on Unix, the user will always "be'' at some logical position within the hierarchy.  This position is termed the "current working directory" or simply the "current directory". The specification of a relative path is relative to this current directory position.  Note that it is possible to change one's current directory while working; this will be discussed in a later chapter.

To specify a file or directory using the relative method, start with the current directory and write each directory that is encountered on the path to the directory or file being specified.  Separate each directory with a "/" forward slash character.

work/mydata

mydata

../report

When should one use absolute vs. relative specifications?  In many cases both are acceptable.  One may notice that a relative specification usually requires less typing.  Who likes more typing? An absolute path is preferred when the user or programmer values portability allowing the specification to be used from any position on the system by any user.

Points to Consider

• Names for files and directories are case-sensitive.  Thus report, Report, and REPORT are all distinct.
• Names may include any letters, digits, and some special characters (period, comma, underscore, etc.) but not /, <, >, &, :, |.
• Names may be up to 255 characters in length.
• There is no requirement for file extensions (few characters after a period).  All of the following are valid names:

• • report

  • letter_to_bob.text

  • forecast.July,Version1

  • notes.doc

• Names must be unique within a directory (no duplicates allowed).  This is automatically enforced by the operating system.  In the raj directory, it would not be possible to create another file or directory called work.  Consider the analogy of human families:  Two siblings would not share the same name, but a cousin, uncle, or grandparent could share the same name without conflict.

Home Directory

The home directory is a private area for the user files and directories. Each user will have a directory name matching the login name assigned by the system administrator.

While it is correct and possible to specify a home directory using the absolute and relative methods aforementioned, there exists an abbreviation consisting of a ~ (tilde) followed by the user’s login name.  Advantage: This avoids having to know the name of the user file area which is non-standard:  some installations call it users, others call it home, still others have their own conventions.  

~raj

~raj/report

All commands are case sensitive.

ls /etc

ls/etc

ls

ls

ls /etc

ls g*

ls -l

ls -a

cat

cat report

cat chapter1 chapter2

head

head longfile

tail

tail logfile

cp

cp report work/report_v2

mv

mv report presentation

mv mydata ../yourdata

rm

rm report

cd

cd work

cd /bin

cd ..

cd ~ahmed

pwd

pwd

mkdir

mkdir unix_exercises

rmdir

rmdir unix_exercises

who

who

whoami

whoami

date

date

man

man ls

man -k directory

More Unix Commands

cut

cut -c1-10,20-23 myfile

cut -f3,5 -d: /etc/passwd

paste

paste file1 file2 file3

wc

wc chapter3

wc -l chapter3

grep

grep Total sales

sort

sort namelist

Filters

One may think of a filter as a black box with an input and an output. Most Unix commands can be thought of as a filter.  The inputs and outputs have been given formal names. The input is named standard input (STDIN); the output is named standard output (STDOUT); and there is a secondary output named standard error  (STDERR) which will be discussed in more detail later.  These inputs and outputs are associated with file descriptors or stream numbers 0, 1, and 2 respectively.

By default, Unix commands read from standard input and print to standard output.   Any error messages are sent to standard error.

Examples of Unix commands as filters

cut

grep

head & tail

cat 

wc

...

Redirection

When working interactively in a Unix session, the default setup is to have standard input draw from the keyboard, and standard output (as well as standard error) directed at the screen.  In other words, a command reading from standard input will wait for keystrokes. A command printing to standard output will have its output appear on the terminal screen.

While this arrangement works well, there will come situations where the user will want to save the output of a command to a file, or substitute a file for keyboard input.  This is accomplished through the concept of redirection where one of the inputs or outputs is associated with a file.

Redirection of standard output ( > operator)

 The user can save the standard output of any Unix command by redirecting standard output to a file using the > operator.  Graphically, the concept is illustrated as follows.

cmd > some_file

cat chapter1 > book

cmd >> some_file

cat chapter3 >> book

Redirecting standard error ( 2> operator)

The user can save the standard error of any Unix command by redirecting standard error to a file using the 2> operator. Graphically, the concept is illustrated as follows.

cmd 2> some_file

e.g.
cat chapter1 chapter4 2> errors

cmd > some_file 2> another_file

e.g.
cat chapter1 > book 2> errors

Merging two streams ( >& operator)

 The user can save both standard output and standard error of any Unix command.  This is accomplished by first redirecting standard error to a file, and then merging standard error with standard output.  The syntax of the merge operator is m>&n where stream m is merged with wherever stream n is already going.   Graphically, the concept is illustrated as follows.  Here stream 2 (standard error) is merged with stream 1 (standard output).

Standard Output and Standard Error are merged (combined) and re-directed to a file

cmd > out_file 2>&1

e.g.
cat text1 junk text3 > both 2>&1

Redirecting standard input ( < operator)

The user can redirect standard input from a file instead of the keyboard to any Unix command using the < operator.  Graphically, the concept is illustrated as follows.

cmd < some_file

cat < appendix

myscript < keystrokes

Pipes

It is often the case that a problem in Unix is solved with multiple commands.  Typically the output of the first command is saved in a file which is then used as input to a subsequent command.  The use of a pipe is considered a refinement of this approach potentially simplifying the solution.

Problem: To determine the number of entries in a directory

Method 1

ls /etc > file_list
wc -l file_list
rm file_list

Method 2

ls ->



 -> wc 

 

ls /etc | wc -l

You can use a pipe multiple times creating a pipeline.

e.g.
cmd1 | cmd2 | cmd3 | cmd4

Building a pipeline should be an iterative process. Condense stepwise as you know the solutions work, otherwise there might be errors that might be difficult to detect from a single pipeline

Start out like this:

cmd1 > out1
cmd2 < out1 > out2
cmd3 < out2

 Making your script executable

1. create file containing unix commands
2. Once per file, type either:
   • chmod u+x myscript
   • chmod 700 myscript
3. To run, type ./myscript

Unix is an operating system but also the name given to its scripting (programming) language. In order to be useful, a computer language needs to be able to store data. Unix supports this capability with shell variables.

There are three kinds of shell variables:

1. special
2. environment
3. program

Special variables

Special variables are unlike what you may have seen in other programming languages. Rather special variables are created and set by the operating system automatically. Consider the following examples:

Positional parameters ($1, $2, …)

Recall that the cat program allows the user to specify inputs as command-line arguments:  (In the following examples, the leading dollar sign represents the command prompt; do not type the leading dollar sign.)

$ cat ch1 ch2 ch3

Imagine that you would like to write your own script allowing the user to specify inputs as command-line arguments as in.

$ myscript apple cherry

$ cat arg_demo
#!/bin/bash
echo The 1st argument is $1
echo The 2nd argument is $2
$
$ ./arg_demo apple cherry
The 1st argument is apple
The 2nd argument is cherry

number of command line arguments ($#)

The special variable $# holds the number of command-line arguments specified when your script was run.

$ cat numeg
#!/bin/bash
echo The number of command line arguments is $#
$
$ ./numeg apple cherry
The number of command line arguments is 2
$

return code of previous command ($?)

Every Unix command generates a return code typically indicating success or failure. The value of this return code is stored in the special variable $?. Thus each time a Unix command is run the $? variable is updated automatically to hold the return code of the most recently executed Unix command.  The use of the return code is important and will become apparent in the chapter on control structures.

Environment variables

Environment variables hold information about the users's current settings and configuration. By convention, they are typically all UPPERCASE. To display them, type ‘env’

lhiraki@metis:~$ env
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin:/usr/courses/bin/x86_64
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
EDITOR=vi
LANGUAGE=en_CA:en
LPDEST=eng206c
lhiraki@metis:~$

Program variables

Program variables are the type of variable that typically come to mind when one thinks of variables in a computer programming language. Program variables are variables which you as the programmer create and set. As Unix is a prototyping language, it is common to dispense with many of the formalities required for variable declarations, etc.

Program variables hold a string, i.e. text, and can be used anywhere text could appear in a program such as a filename, part of the file name, or even a Unix command itself.  A variable can hold only one value at a time.

The naming convention for program variables is to use lowercase. Specifically uppercase program variable names should be avoided so as to prevent confusion with environment variables and accidentally overwriting an environment variable.

temp_name=/usr/temp
cp myfile $temp_name

Assignment statements copy the value to the right of the equal sign to the variable on the left.  Important:  Assignment statements must not contain spaces, especially around the equal sign.

month=september 
echo the current month is $month

When to use $ with variables

When writing to a variable, i.e. setting or changing its contents, do not use the dollar sign. When reading a variable, i.e. accessing its contents, you must use the dollar sign.

read

The read command collects characters from standard input (STDIN) and stores them in a variable.  The read command is typically used in an interactive script to collect user input after an appropriate prompt message.  As the read command draws from standard input, and standard input can be redirected from a file, it is possible to prepare inputs (answers) in a file and run the script in a non-interactive fashion.

$ cat readeg
echo -n 'What is your name? '
read name
echo Hello $name, pleased to meet you!
$
$ ./readeg
What is your name? Mohammed
Hello Mohammed, pleased to meet you!
$

Quotes

Single quotes:

Problem:

$ grep Al Shaji employee_list
grep: can’t open Shaji

Solution:

$ grep 'Al Shaji' employee_list

Double quotes

• Recognize $, \, ` (backtick or backquote)

heading='    Name     Addr     Phone'
echo $heading
echo "$heading"

read operator # enter asterisk *
echo $operator
echo "$operator"

The double quotes are similar to single quotes in that Unix takes what is within the quotes literally.  However, double quotes are "smarter" in that variables and selected meta-characters are interpreted and expanded in spite of the usual literal nature of quotes. 

Home directory potential tricky issue (tilde is protected i.e. not expanded within quotes).

Instead of:

datafile="~jasmin/rawdata"
cat $datafile  # produces "No such file or directory" error

Say (solution 1):

datafile=~jasmin/rawdata
cat $datafile

Say (solution 2):

datafile="~jasmin/rawdata"
eval cat $datafile

Command substitution

Sometimes a programmer wishes to run a command and use its output at some point within a program.  While it is possible to redirect output to a temporary file, load the contents of the file, and then promptly delete it, for small tasks, it is more convenient and efficient to use the technique of command substitution  and avoid extra disc access.

today=date
echo $today

today=$(date)  (old version today=`date`)
echo $today

$(unix_command)

The mechanics of command substitution works as follows:

1. Unix will run the command within the parentheses as if it were typed at the keyboard.  The command may include options, command line arguments, or even be a script.
2. Instead of being displayed on the screen, the standard output (STDOUT) of the command is substituted at the exact position of the call ($(unix_command)).  

The command substitution may be made anywhere in a program; however, it is often used on the right-hand side of an assignment statement (to save the output in a variable), or within an echo statement.

test

What does the test command do?  It evaluates a condition and sets the special shell variable $?, the return code.  Much to the confusion of new users, the test command is silent in that it does not print anything to standard output.  Thus when running the test command, it appears as if to do nothing.  To check the return code, one may simply print its value with an echo statement.

$ test -r myfile
$ echo $?
0

How does one interpret the return code?  

Unix convention:

$ test "this" = "that"
$ echo $?
1

Synonym to the test command:  [

For reasons of readability, many programmers with use the synonym or abbreviation for the test command which is the left square bracket: [ .  In all examples above, replace the word test with the left square bracket.  Note that a matching right square bracket needs to be added for syntactic reasons.  As with all Unix commands, spaces are a big deal and a space is required after the left square bracket and before the right square bracket.

$ [ -r myfile ] # note the space after the left bracket
$ echo $?
0 

Control Structures

if

The if statement in Unix is the basic two-way branch.

if unix_statements
then
   actions_true
else  # optional
   actions_false
fi

How does the if statement work?  Here is the sequence of operations.  The if statement will:

1. Run all the unix_statements.  
2. Check the return code ($?) of the last statement in the list (just prior to the keyword "then".
3. If the return code is true. the "then" clause is executed ( actions_true).  If the return code is false, the "else" clause is executed (actions_false).  The else clause is optional (can be left out if not needed).

#!/bin/bash
comfort=20
temperature=18
if echo The current temperature is $temperature
   echo Temperature for comparison is $comfort
   [ $temperature -lt $comfort ]
then
  echo It is cold.
else
  echo It is warm.
fi 

Nested conditions are supported in Unix.  Any statement in the "then" or "else" clause can itself be another if statement.

elif

Although the if statement is primarily a two-way branch (e.g. true or false), a multi-way branch (e.g. red, yellow, green) can be coded using a set of nested if statements.  Some computer languages support an "else-if"-type clause; Unix is one of them.

There is an "else-if" clause called "elif" which requires a statement just like the if clause.

if [ $temperature -lt $comfort ]
then
  echo It is cold.
elif [ $temperature -eq $comfort ]
then
  echo It is perfect.
else
  echo It is warm.
fi
 

case

Many computer languages support a multi-way branch. Unix is included as one of them.

case $variable 
in
  val1) action1;;
  val2) action2;;
  *) default action;;
esac 

$ ./myscript # user omits filename; give second chance
$ ./myscript chapter3 # preferred syntax; just proceed
$ ./myscript chapter3 chapter5 # multiple arguments not supported; 
inform user

Place this code snippet at the beginning of your script like this:
$ cat myscript
case $# in
  0) echo Enter file name:
           read arg1;;
  1) arg1=$1;;
  *) echo invalid number of arguments
     echo "Syntax: $0 filename"
     exit 1;;
esac
# rest of program continues after esac
$

case match_string_expr in
  match_pattern) action1;;
  match_pattern) action2;;
  ...

esac

lhiraki@metis:~/test$ cat case_month
case $(date '+%m') in
01|03|05|07|08|10|12)
     echo This is a long month;;
04|06|09|11)
     echo This is a short month;;
02)
     echo This is the shortest month;;
*)
     echo Something wrong with date command;;
esac

# Example run in September

lhiraki@metis:~/test$ ./case_month
This is a short month
lhiraki@metis:~/test$

Defensive programming

Well the previous example is rather trivial and the date command has been well tested over the years, it is considered good practice to always have a default clause even if you think you have covered all possible conditions.

Summary

 The match_string_expr is matched against each match_pattern in the order coded.  At the first match, the corresponding action is taken.  After one action is completed, the case statement terminates and execution continues after the esac (end case).  There is no "fall-through".  The case statement will not execute multiple actions.

shift

When processing multiple command-line arguments, it may be necessary to manipulate them to facilitate processing.

The shift command moves all command-line arguments one position to the left.  For example the second command line argument is moved into the first position; the third command line argument is moved into the second position, and so forth.

lhiraki@thebe:~/test$ cat shifter

#!/bin/bash
echo The 1st arg is $1
echo The 2nd arg is $2
shift
echo The 2nd arg is $1
shift
echo The 3rd arg is $1 

lhiraki@thebe:~/test$ ./shifter apple pear grape
The 1st arg is apple
The 2nd arg is pear
The 2nd arg is pear
The 3rd arg is grape
lhiraki@thebe:~/test$

The exit command

The exit command does two things:

1. It terminates the current shell (or script), returning control to the calling shell, if any.
2. It sets the return code ($?) for your script.

$ cat exit_example
#!/bin/bash
exit 3
echo This line never executed.

$ ./exit_example
$ echo $?
3
$

The exit command is typically used to terminate a script midway through often due to an errror condition. The other primary use of the exit command is for a sub-script to communicate information back to the calling script.

./child 
ret_val=$?
case $ret_val

...
   exit 2
...
exit 0

Shebang line

To specify which interpreter Unix should use when executing your script, as the first line of the file place the path to the interpreter after "#!".  While the number sign character (#) normally introduces a comment, when used with the exclamation mark at the beginning of a file, Unix will load the interpreter specified in the path to run the rest of the script file.

This is especially important to make your script portable in Unix environments.  If you write your script in bash, and you give your script for someone else to use who works in a c-shell or Korn shell environment, your script may not work properly.  To ensure that the script is run under bash, you must specify the shebang line.

Careful:  The shebang line must be the first line of the file, not just the first line of text, or the first line of code.  A common mistake is to have a blank line as the first line, or some comments above the shebang line.  Unix does not look beyond the first line of the file in order to identify the expected interpreter.

1:#!/bin/bash
2:#comments
3:#comments
4:code begins here        

1:#comments
2:#comments
3:#!/bin/bash
4:code begins here

1:
2:#!/bin/bash
3:
4:code begins here

Loops in Unix

There are three commands in Unix to support loops.

• while
• until        
• for

while command

The while command in Unix implements the General Loop construct discussed earlier.

while list1
do
  list2
done

while block1
      check
do
   block 2
done

Unix runs the list1 and looks at the return code ($?) of the last command in list.  (If there's only one command in the list1, then that's the last command.)  If the return code of this last command is TRUE, then list2 (loop body) is run.  Once the keyword "done" is reached, control returns to re-run list1 and the return code of the last command in list1 is checked.   If the return code of this last command is FALSE, then the while loop terminates, and control continues after the keyword "done".

month=1                                
while echo Checking limit against month ${month}
  [ ${month} -le 12 ]   # The test is the last command in list1
do
   echo Performing action for month ${month}
   month=$(( ${month} + 1))
done 
echo Value of month outside loop is ${month}

Preamble: Redirecting input from a file

$ cat novel
It was a
dark and
stormy
$ read oneline < novel
$ echo $oneline
It was a
$ read oneline < novel
$ echo $oneline
It was a
$

Be reminded that redirection operators in Unix (<, >, etc.) apply only to the current command.  Once the command is over, redirection is restored to the default configuration (STDIN from keyboard, STDOUT to screen, etc.).   Thus, observe that each time the read command is run, redirection is applied anew, and the first line of the file is read in.

$ cat badwhile
while read wholeline < novel
do
        something $wholeline         #e.g. echo $wholeline
done
$ ./badwhile
It was a
It was a
It was a
It was a
...

while read wholeline
do
        something $wholeline  #e.g. echo $wholeline
done < inputfile

cat ch1 ch2 ch3 ...

mycmd parm1 parm2 ...

while [ $1 ]
do
   someprocess $1
   shift
done

$ ./whilecmdlist
What is the password? happy
--- Sorry, that is not the right password.
What is the password? access
--- Sorry, that is not the right password.
What is the password? Strawberry
Welcome to the system.
$

until command

The until command in Unix implements the General Loop construct discussed earlier.

The syntax is identical to the while command with the only difference that the logic of the conditional test is reversed.  If the last command in list1 is TRUE, the loop terminates.

Here's the same explanation for the until command:

Unix runs the list1 and looks at the return code ($?) of the last command in list.  (If there's only one command in the list1, then that's the last command.)  If the return code of this last command is FALSE, then list2 (loop body) is run.  Once the keyword "done" is reached, control returns to re-run list1 and the return code of the last command in list1 is checked.   If the return code of this last command is TRUE, then the until loop terminates, and control continues after the keyword "done".

for command

The for command in Unix operates quite differently than what is common in other computer languages like BASIC or Pascal.  In particular, it is not a counted loop, rather one should think "iterative substitution".

for name in word ...
do
   list
done

ren assig* assig*.bak

mv assig* assig*.backup

for filevar in assig*
do
   mv ${filevar} ${filevar}.backup
done 

count=0
for files in $(ls /etc)
do
   count=$(( $count +1 ))
done
echo The number of entries is $count 

mycmd parm1 parm2 …

for cmdarg in $*
do
        myprocess $cmdarg
done
 

break

i=0
while true
do
  if [ $i -eq 5 ]
  then
    echo Reached 5
    break
  fi
  i=$(( $i + 1 ))
done

i=0
while [ $i -ne 5 ]
do
  i=$(( $i + 1 ))
done
echo Reached 5

grep 'Al Shaji' namelist

echo "Price: $value"

today=$(date)

echo ${alpha:3:5}

a=$(( $a + 1 ))

if [ $op = MR ]

FAQs

• chmod 700 myscript

• chmod u+x myscript