Rolf Jansen's Intro to Linux (or Unix)

An OS is an OS is an OS?

Linux is a new (and better), open source implementation of the commercial UNIX® Operating System (OS). An OS translates your high level commands into the actual series of operations that the hardware (CPU, memory, hard disk, etc..) in your computer has to perform to implement that command and return a —hopefully meaningful— result. Command-line based operating systems (such as DOS®) interpret the text commands you type. Graphical User Interface (GUI) based operating systems (such as MacOS® or Microsoft® Windows®) allow you to interact with the computer through pictures and menus, via mouse-clicks or by pressing a key on the keyboard.
The larger flexibility of Unix and Linux generally gives you the option of using both command-lines and GUIs. The former offer more control and flexibility — but you need to learn and remember a fairly large set of commands and command options. The latter can be easier and more intuitive for complex operations that don't need to be frequently modified or repeated many times — but are often not flexible enough for the task at hand.

A Short History

Unix has been in use for more than three decades and resulted from the efforts of two groups, at Bell Labs and at UC Berkeley, to develop a reliable, multi-user, true multi-tasking (none of which apply to, e.g., DOS or Windows) operating system. It originally ran mainly on large mainframe computers and workstations, but was exceptional in that the OS itself was written in a high-level programming language (C) and could therefore be made to run different types of computers (hardware architectures). Windows, on the other hand, gained popularity in the mid-1980's as the only major operating system designed specifically for Intel-compatible processors as found in the, then new, Personal Computers (PC's). As a mass-marketed, single user OS, it was designed to look good and be intuitive to use by people with little knowledge of computers, not for stability, flexibility, and multi-tasking.

Starting around 1991, a new implementation of Unix called Linux (named after originator Linus Torvalds) begain gaining popularity with PC users, particularly for high performance and mission-critical operations (e.g., Internet Service Providers, hospital administrations) and for users with serious computational needs (in the exact sciences, PC's running Linux have replaced many of the more expensive Unix workstations and their recurring software license fees). The Linux® kernel and GNU C compiler and suite of application software form the basis of the Linux OS, which is available for a large variety of architectures and has in many ways improved upon UNIX. The unique feature of Linux, and reason for its rapid spread, is that all source code is freely available (see: GNU General Public License) to anyone who wishes to modify or add functionality to existing programs or write and integrate new ones (as opposed to proprietary operating systems, where the source code remains a well-guarded trade secret, preventing users to modify programs to suit their needs, or solve common problems with existing programs). This means that the Linux OS benefits from the work of thousands of users who help in further developing the kernel and application programs. Despite claims to the contrary by proprietary OS vendors, this large number of Linux developers has yielded an operating system of unprecedented efficiency and robustness, with many CDROM's worth of freely available applications software packages for both business and pleasure. Juergen Haas nicely sums up why Linux is the Ultimate Unix.

Look at all them linuces

Dozens of companies package the Linux kernel and C compiler, the core of the OS, with large quantities of applications software and a convenient helper program that unpacks and installs everything onto your computer's hard-disk. The market leaders of these are Redhat (including its offshoot Fedora; particularly in the US) and SuSE (particularly in Europe), but Debian, Mandrake, and Slackware also have a sizeable followings. Although the packages of application software (including the desktop GUI) and/or the place where they wind up on your hard-disk may differ between these distributions, each generation runs the same Linux kernel (give or take a minor version number).

Living in a Shell

A shell is a program that acts as an intermediary between the user and the guts of the OS (much like the OS acts as an intermediary between the shell and the nuts and bolts of the hardware). A shell provides a command line. In DOS, command.com acts as the shell (albeit a limited one). Linux shells (e.g., tcsh, csh, bash, sh, ksh) perform the same function, but besides translating your commands into something the kernel can understand and act upon, they also add some important functions that the base OS doesn't supply. Whenever you open a new terminal window (like xterm), a tcsh shell is automatically started for normal users; the root account for the system administrator typically runs a bash shell. Some basic features of all Linux shells are:

Prompts   A prompt is a character or string of characters (such as a "%", or ">" for csh and tcsh; "$" for the other shells) that the shell displays when it is ready to receive a new command. The prompt may be customized (to, e.g., "user@machine>").
Command resolution   When you enter a command, the shell must determine which program to run in order to perform that command. This command resolution process can be customized to suit the user's needs.
Job control   Linux lets you multi-task (run more than one command at the same time). Individual tasks may be started, stopped, suspended, and executed in either the foreground or background. One can start many shells, in each of which different tasks can be running simultaneously.
Command history and completion   Sometimes you want to repeat a command you've executed before, or issue a similar one. Previously executed commands can be recalled and modified. Also, the shell can automatically complete commands and file names for you.
Wildcards and aliases   Wildcards let you process a whole bunch of files at once, instead of having to repeat the same command for each file individually. Commonly used commands, or modifications of commands with different sets of command options can be stored as aliases.
Piping and I/O redirection   Connecting programs together by sending the output of one program directly to the input of another (piping), or reading input from and/or sending output to a file (I/O redirect) can save time and keystrokes.

Computer, I command thee

Although the user may remain oblivious to this fact (since a shell is started when we open a terminal window), there are two types of commands: Unix/Linux commands and Shell commands.

Shell commands are commands that are built into a shell program. Shell commands are entered at the shell-prompt in a terminal window program (e.g., xterm, kterm, gnome-terminal). Although the command set built into different shells differs, the commands within a given shell are the same across Unix/Linux distributions.

Unix/Linux commands are not part of a shell, but correspond to separate executable programs. Each such program is written in a programming language like C (or C++, Fortran, Java, etc..). These programs tend to be located in system directories where shells can automatically find them, but where only the system administrator can create or modify files. The shell program itself is a Unix/Linux command. The set of available commands tends to vary from one Unix/Linux distribution to the next, and depends on which software packages the system administrator chose to install.

Most commands (both of the shell and Unix/Linux variety) accept one or more options and/or arguments. Options modify the default actions performed by a command, or the format of their output. Usually, single character options need to be preceded by a single dash (-), while most multi-character options need to be preceded by a double dash (--). For example:

user@machine>   du --version
user@machine>   du -h
user@machine>   du -s -h

In the latter command, we could have merged the two single character options (this is why the double dash is needed for multi-character options):

user@machine>   du -sh

Arguments are typically used to specify what file(s) the command should operate on and/or what file(s) to create. For example:

user@machine>   du -sh /home/user/
user@machine>   du -sh *

where, in the latter command, we used the wildcard *, which matches any and all files or directories.

Some common shell commands
cd,chdir change directory go to your home directory: cd or cd ~ or cd /home/user
go to your data directory:   cd /data#/user
return to previous directory: cd -
echo write arguments to the standard output of the shell (usually the screen) echo "Hello, world!"
echo -n "Hello, " ; echo "world!"
exit exits the shell  
history displays the history list of prior commands show last 10 commands: history 10
kill sends a termination signal to a process kill process#       dangerous: don't mistype process number!
printenv,setenv print or set environment variables printenv HOSTTYPE
source reads a series of (shell)commands from a script source ~/my_macro.txt
which shows the full path of (shell) commands which arg narg range

Some common Unix/Linux commands
acroread display and read PDF documents (see also gv, xpdf, and evince) acroread document.pdf
apropos search for available commands that match one or more search string[s] apropos 'search_keyword[s]'
awk pattern scanning and processing language excise columns 3 and 4 from an ASCII table:
   cat table.txt | awk '{print $3,$4}' > mytable.txt
excise cols 1, 3 and 4 from ASCII table, and reformat:
   cat table.txt | \
   awk '{printf("%-12s %4d %7.4f\n",tolower($1),$3,$4)}' \
   > mytable.txt
cat concatenate files and print on the standard output cat ~/AA_README
chmod change file access permissions make a script executable:
   chmod +x myscript     or       chmod 755 myscript
set reasonable file permissions for a text file copied from a Microsoft® file system (VFAT,FAT32,NTFS) to a linux file system (ext2,ext3,ext4):
   chmod 644 some_file.txt
clear clear the terminal screen  
cp

copy files and directories move large downloaded data files to your data directory:
   cp ~/Desktop/bigfile*.fits /data#/user/Project/.
csh, tcsh start a C shell (when starting 'xterm' using the button on the desktop, 'tcsh' will be started up automatically)  
cut remove sections from each line in files cut -d/ -f2-4 some_file_list | sort -u
date print (or set) the system date and time print the current local and UTC time with default format:
   date ; date -u
print date+time as a Y2K-compliant FITS date string:
   date -u +%Y-%m-%dT%T
df report filesystem disk space usage report on all mounted file systems, use human-readable sizes:
   df -ah
diff find differences between two files diff file1.txt file2.txt
du estimate file space usage summarize file space usage, use human-readable sizes:
   du -sh ~/ ; du -sh /data#/user/
dvips convert a LaTeX DVI file to Postscript convert a LaTeX DVI file to PS suitable for later conversion to PDF, while including all fonts and specifying the paper type to be US Letter:
   dvips -Ppdf -Pamz -Pcmz -t letter ms1.dvi \
   -o ms1_20130614.ps
emacs GNU Emacs text editor
See: tutorial, beginners, survival.pdf and refcard.pdf.
create and edit a ASCII text file in current 'xterm' window:
   emacs -nw newfile.txt
open existing text file and use Emacs' own GUI window:
   emacs oldfile.txt
enscript print plain text files (nicer formatting than lpr) print ASCII text (e.g., source code) in a 2-pages per page landscape format, printing in duplex mode:
   enscript -2r program.c -p program.c.ps
   lpr -Pps -o Duplex=DuplexNoTumble program.c.ps
evince display, read, and print PDF documents evince document.pdf                (as of CentOS 6.x)
find search for files in a directory hierarchy find all PDF files newer than 30 days in your home dir:
   find ~/ -type f -mtime -30 -name '*.pdf'
g77, gfortran GNU Fortran77 (+ some newer Fortran extensions) compiler g77 -W -Wall -o program program.f -L/home/lib -lmytools
(assuming some dependency on a nonstandard library "libmytools.a" that can be found in your ~/lib/ directory)
gcc GNU C compiler — without it, no Linux gcc -W -Wall -o program program.c
grep, fgrep, egrep search for expressions in a file search for upper-, lower-, or mixed case instances of "flatR" in all files with names of the form "flat*.lis":
   grep -i 'flatR' `find /data#/user/ -type f -name 'flat*.lis'`
gs Ghostscript (PostScript and PDF language interpreter) use as a basic PostScript viewer to view file "plot.ps":
   gs -sDEVICE=x11 plot.ps
use as a filter to merge multiple PDF files into one:
   gs -dNOPAUSE -sDEVICE=pdfwrite \
     -sOUTPUTFILE=file123.pdf -dBATCH \
     file1.pdf file2.pdf file3.pdf
gtar GNU version of the tar archiving utility create a gzip-compressed backup in archive mode (preserve permissions and time stamps) of directory "Project":
   gtar -cpzf /tmp/backup.tgz Project
inspect the contents of the tar-ball just created:
   gtar -tvpzf /tmp/backup.tgz
restore this backup (some place else):
   gtar -xvpzf backup.tgz
gv Ghostview (PostScript and PDF viewer) view a (possibly gzip-compressed) Postscript or PDF file:
   gv plot.ps    or    gv plot.ps.gz    or    gv plot.pdf
gzip, gunzip compress or expand files compress all Postscript files and raw (integer valued) FITS files of a project:
   gzip -v `find /data#/user/Project/ -type f -name '*.ps' \
     -or -name '*.eps' -or '-name '*.cps' -or -name '*.fit' \
     -or '-name '*.fits'`
compress all files larger than 1 Mb in size, except files that are poorly compressible or already compressed:
   gzip -v `find /data#/user/Project/ -type f -size +1M | \
     egrep -ve '(.pdf$|.bin$|.jar$|.exe$|.gz$|.tgz$|.zip$|.bz$)'`
head output the first few lines of files display the first 22 lines in file "img.lis":
   head -22 img.lis
display the 3rd batch of 10 lines in file "img.lis":
   head -30 img.lis | tail -10
info read Info documents info command
latex LaTeX, macro package for the TeX typesetting program (see the online LaTeX manual); produces DVI file. (see also dvips). latex ms1.tex ; latex ms1.tex
(Note: latex needs to be run twice, or even trice, when you request automatic generation of a table of contents, figure and table numbering, and/or citations)
less, more page through a text file less ~/AA_README
ln make links between files create a symbolic link to a project directory that resides on your data partition in your home directory:
   cd ; ln -s /data#/user/Project Project
lpq show printer queue status lpq -Pps
lpr print plain text or Postscript files ASCII text: better to use enscript;   Postscript files:
   lpr -Pps -o Duplex=DuplexNoTumble program.c.ps
lprm cancel print jobs find Job number (#) using lpq, then:   lprm -Pps #
ls list directory contents print a listing of all files in the current directory, for both block and human-readable file sizes (include hidden files):
   ls -la   and   ls -lah
the same in a directory other than where you are now (no need to change directories first):
   ls -lh ~/Project/src/python/   or   ls -lh ../../data/new/
print a single-column listing of file names without colors or other markup (needed when creating input file lists):
   \ls -1 /data#/user/Project/*.fits
man command format and display the on-line manual pages for a command — singlemost important Unix/Linux command. man ls
mkdir make directories mkdir /data#/user/Project2
mozilla, firefox web-browsers (offshoots of Netscape) (usually started by clicking the icon on your desktop)
mv move (rename) files mv /data#/user/Project/whatIdid20130614.txt ~/Notes/.
paste merge lines of files paste the lines from two input files using a "/" as delimiting character:
   paste -d/ paths.lis files.lis
ps2pdf distill a PDF from a Postscript document convert your manuscript to PDF before sending it as an email attachment:
   ps2pdf ms1_20130614.ps
pwd print name of current/working directory where am i?   pwd
rm, rmdir remove (irrecoverably!) files or directories remove a file with confirmation:   rm testimage.fits
recursively remove the contents of a directory and that directory without any confirmation (any typo and you may be royally screwed):   rm -fr ./testreduction
rsync copy/sync files to/from another machine via Secure Shell copy all files that exist in the local directory but do not exist in the corresponding remote directory, and update all files in the remote directory that differ from the ones in the local directory. Use compression (-z) to speed up transfers over slow networks, and show (-v) what is going on:
   rsync -avz --rsh "ssh -l user" ./
         hostname.hostdomain:/destination_path/.
update local files using remote directory as source:
   rsync -avz --rsh "ssh -l user"
         hostname.hostdomain:/source_path/ .
synchronise all but the hidden (configuration) files from your home directory on a remote machine:
   rsync -avz --rsh "ssh -l user"
         hostname.hostdomain:/home/user/\[0-9A-Za-z\]\* .
sed unix command-line editor append the file extension ".fits" to all image root names in a list of such names, and prepend a directory path:
   sed 's:$:.fits:g' imroot.lis | \
     sed 's:^:/path/:g' > files.lis
sleep delay for a specified number of seconds Test data transfer in Mb per minute:
   du -sm ./ ; sleep 60 ; du -sm ./
sort sort lines of text files Find unique file sizes:
   du -m `find . -type f` | awk '{print $1}' | sort -u
ssh, scp connect to or copy files to/from another machine via Secure Shell login to a remote machine via a securely encripted connection:
   ssh username@hostname.hostdomain
(you may need to specify the -X or -Y options to ssh to allow X11 forwarding, i.e., spawning applications from the remote host onto the local display)
securely copy a local file to a remote host, preserving attributes:
   scp -p /source_path/filename
         user@hostname.hostdomain:/destination_path/.
securely copy a remote file to current directory on local host:
   scp -p user@hostname.hostdomain:/source_path/filename .
tail output the last few lines of files
top display top CPU processes
vi, vim Unix text editor
w show who is logged on and what they are doing
wc print the number of bytes, words, and lines in files count # of objects (lines) in list: wc -l objects.lis
who shows who is logged on
xpdf display, read, and print PDF documents xpdf document.pdf
xv, eog view and/or edit pictures in various image formats (e.g., gif,jpg,png,bmp,ppm,tiff,ras,ico,svg) xv picture.jpg   or   eog picture.png

To be, or not to be ... but where?

The Unix/Linux directory path structure differs somewhat from Windows'.

Would ls by any other case be as sweet?

Note, that Unix/Linux (contrary to, e.g., DOS) is case sensitive. So beware of inadvertently hitting your [Caps Lock] key (on most keyboards, there is a LED indicating whether it's on or off). In short, LS differs from Ls, lS and ls.

Tips-'n-tricks

(file name) completion
Often, it's not necessary to type the full name of a file. Type the first few characters, then hit the [Tab] key. If you typed enough characters to provide a unique match, the shell will add the remaining characters of the name. If there are no files, or if there is more than one file that starts with these first few characters, the shell will beep at you. In that case, hit the [D] key while depressing the [Ctrl] key — commonly denoted as ^D and pronounced "Control D" — and the shell will give you a list of all possible matches. It is also possible that the shell will append additional characters to the characters already typed, up to the character where more than one possibility exists.

Example:
I want to edit a text file with a long name, that starts with an "i", and that I know resides in a directory with absolute path /data1/me/ .

    me@mymachine>  vi /data1/me/i
Hitting [Tab] sounds a beep, so either the file name didn't start with an "i" after all, or there are more than one files or subdirectories with names that start with an "i". Hitting ^D produces:
    me@mymachine>  vi /data1/me/i
    idl/     iraf/    itinerary   info_ast494_bck050901v1.txt
    me@mymachine>  vi /data1/me/i
So to edit the "info" file with the long name, I only have to add an "n" to what I already typed and then hit [Tab], i.e.:
    me@mymachine>  vi /data1/me/in[Tab]
Had there been also a file named "info_ast494_bck050901v2.txt" in the same directory, and if that were the file I wished to edit, then I would only need to type in the last step above:
    me@mymachine>  vi /data1/me/in[Tab]2[Tab]
Similarly, if you know the first few characters of a command, then hitting ^D will give a listing of all available commands starting with these characters.

Copy/Paste using your mouse
To copy text from either the terminal window you are working in right now or from another window (like the browser you're viewing this web-page with) to your command-line or to a text file that you are editing, you can use your mouse in several ways. A Unix/Linux mouse generally has three (or more) buttons. If the mouse is a wheel-mouse, depressing (rather than rotating) the wheel serves the function of a middle button.

Method 1:   Drag your mouse with the left-button depressed over the portion of the text that you want to copy. The selected text will become highlighted.
Method 2:   Left-click on the first character, then right-click on the last character of the portion of text to copy. The selected text will become highlighted.
Method 3:   Double-clicking the left mouse button anywhere in a word will select it. What is considered a word may depend on the application.
Method 4:   To select an entire line, triple-click with the left mouse button.

Then move you mouse to the window were the text should be copied to, and hit the middle mouse button. Note that, if the text should be copied into a text editor, then (depending on the editor program) you either have to move the text cursor or the mouse pointer to the location where the text should appear before hitting the middle mouse button.

Command-line editing
Why re-type a previous command that was almost what you want to do now? You can browse through the command history using the up-arrow [↑] and down-arrow [↓] keys. Instead of the arrow keys, one can also use ^P and ^N (remember: ^P denotes hitting the [Ctrl] and [P] keys simultaneously). If the command was issued a while back, try searching the command history using grep (note the pipe operation — the vertical bar — that allows us to pass the output of the history command to the input of the grep command):

    me@mymachine>  history | grep scp
    1292  22:08   scp -p linux_intro.html me@myothermachine:/afs/asu.edu/users/me/www/linux/.

This command may be copied using the mouse. If I actually want to scp another file than the one I copied before, try hitting the ^A. This results in the text cursor jumping from the end to the start (left) of the command-line. You can now move the cursor forward to the begin of the file name with either the right-arrow [→] key (opposite: left-arrow [←]) or by hitting ^F (opposite: ^B). To delete the old file name hit ^D repeatedly. Note, that anywhere except at the end of the command line, ^D means delete-next-character. Now you can type the name of the file you do want to copy (typing the first few unique characters followed by [Tab] suffices). You can now move to the end of the command-line by hitting ^E. If the original destination directory differs from the one you want to copy to now, hit [Backspace] to delete the original name and then type the correct destination directory. Since the destination directory resides on another machine, directory name completion using [Tab] won't work in this example. Finally, hit [Enter] or [Return] to execute the modified command. Once you have a little practice with this, you'll find this way faster than re-typing every command or editing the command line with only the arrow and [Backspace] keys.

In case of a vi scare
I inadvertently started the vi editor and opened up a file; I don't know vi commands yet and was just hitting some keys and suddenly some text changed and the file may have been ruined. How can I get out, hopefully restoring the file to its original content?

Don't worry. Start by hitting the [ESC] key once or twice to clear any of the modes that you may have inadvertently entered. Then hit the colon key, [ : ]. If a colon appears as the first character in the bottom line of the window, type q (i.e., hit the [Q] key). If a colon did not appear, hit [ESC] again, then type :q. This will exit vi without changing the original — all editing up to this point had only been performed in a copy of the file in the computer's memory.

Exiting editor programs
Exit editor programs correctly using the commands/keystrokes appropriate for that editor. Many editor programs open up their own window instead of running within a terminal window. For such programs, the warning below may not apply. But never ever close an editor program that runs within a terminal window by killing that window, e.g., by clicking the "cross button" at the top right of the window bar. The communication of the editor program with the window/shell from which it was launched is lost that way and the now orphaned editor program starts to take up all CPU time (try the top command; hit q to exit). If this happens, carefully note the process number (leftmost column in top) of your editor program (e.g., vi, emacs, mem) and issue a kill process_number command. If thereafter, the process still runs, issue a kill −9 process_number command.

How can I read data from CDROM
Normal users, in general, will not have permission to change the hardware configuration of a linux PC. Since mounting a CDROM falls technically in this category, on some computers users are not allowed to mount and umount a CDROM, while on others a special program automatically detects when a CDROM has been inserted and will transparently mount it for the user. On most, you can right-click on the CDROM icon on your desktop and select the "Mount" option. If you wish (or need) to manually mount the CDROM, users can temporarilly gain the extra privileges required using the sudo command:

    me@mymachine>  sudo mount -t iso9660 /dev/cdrom /mnt/cdrom
or: me@mymachine> sudo mount -t iso9660 /dev/scd0 /mnt/cdrom
depending on the machine and version of Linux. You will be prompted for your user-password. After the CDROM is mounted on /mnt/cdrom, you can cd, ls and cp your data to a data directory on the hard disk. When you're done, type:
    me@mymachine>  sudo umount /mnt/cdrom
or you can right-click on the CDROM icon on you desktop and select "Eject", which will first automatically dismount the CDROM. In either case, make sure you are no longer in the /mnt/cdrom/ directory, or a "Device busy" error will be returned.

For more info, tutorials, tips and tricks, see:
William Knottenbelt's "Introduction to UNIX course"
Juergen Haas', "Linux 101: A Newcomer's Guide"
Rafael Irizarry's, "Introduction to unix/linux"
Doctor Bob's, "Linux Tutorial"
Rolf Jansen's, "Using (C)shell scripts to make life more manageable"
And various guides available from www.codecoffee.com/tipsforlinux/ :
        Using Emacs – part I and Using Emacs – part II
        Wildcards, Quotes, Back Quotes and Apostrophes in shell commands
        How to use pipes – get the most out of your shell
        Input/Output redirection in Unix
        How to use the 'grep' command
        How to find files using 'find'

Last updated: Aug 11, 2014


Back to the top of this page.
In case of problems with this page, contact: Rolf.Jansen@asu.edu , tel.: (+1) 480  727 7119