Tips

A Collection of Little Known Coding For UNIX

November 9, 2009 • By

The popular UNIX® operating system is almost like a huge, uncharted desert. When you traverse the area, you can pick up tools that will help you. These tools help in making full use of the powerful command line interface of UNIX. The command line interface is not only lightweight, requiring very less power as compared to a typical graphical user interface, but is also more secure when computers are connected in networks. Additionally, running batch processes is a simple task in the command interface which cuts down the amount of work and time required drastically. Imagine having to resize 100 images – with the command line it will take you few simple commands, compare it to opening the 100 images in a photo editor and resizing them manually.

With the advantages of the command line interface well established, let’s now take a look at some of the little known yet highly useful commands in UNIX. We will start with a little brief about using variable names, setting a good coding practice which will help in not just the commands we discuss further but also in your general coding. Next we will discuss about the ever helpful ‘find’ command. After we discover some of its secrets we will move on to some file manipulation commands using grep and stream editor. Then we discuss some commands which are required for compression and archives and see how we can resize images in a batch as mentioned above. After some useful tricks for image manipulation we will take a look into various ways by which you can speed up your command line usage – file name completion, using history commands, aliases, quicker directory navigation etc. And in the end we will discuss a neat way to copy the Screen’s copy buffer to the X’s copy buffer.

Using variable names with caution – a good coding practice

Always be careful when using variables and variable names. Additionally, if a variable name is being directly followed with alphanumeric text, then remember to enclose the variable name in curly braces ({}), this distinguishes the variable from the proceeding text, and avoids it being interpreted as a part of the variable name.
Below are some examples to better illustrate the idea:

~ $ ls /tmpList the files present in ‘tmp’ directory
a b                                       – Output is the names of the files – ‘a’ and ‘b’
~ $ VAR=/tmp/*Declare a variable ‘VAR’ and assign it value ‘tmp/*’
~ $ echo $VAREchoes the value pointed to by the value stored in ‘VAR’
/tmp/kde-user /tmp/kde-root /tmp/ksocket-user /tmp/pulse-uavuaOb9vyJc
~ $ echo “$VAR” Echoes the value of ‘VAR’
/tmp/*
~ $ echo $VARa Treats the variable name as ‘VARa’ which doesn’t exist,
and hence returns a null value
~ $ echo “$VARa”Treats the variable name as ‘VARa’ which doesn’t exist,
and hence returns a null value
~ $ echo “${VAR}a”Echoes the value of VAR and ‘a’
/tmp/*a
~ $ echo ${VAR}a –  The same result as above
/tmp/*a

The difference in using quotation marks and brackets in variables becomes obvious from the above examples, and hence you should always remember to be careful about using the same. The main difference is that if you echo a variable with quotations it will echo the name of the variable and not the value pointed by the variable. Now let’s take a look at the Find command.

The Find Command

The find command can simplify your work online, and it is much more than just locating files; it can automatically execute certain and designated sequences of other obscure UNIX commands, using the input filenames.
The most basic task of the find command is to find a file on your system. You can use various options and wildcards to obtain files with a particular suffix, in a particular directory, or ignore case in finding the filename. For instance the following command will let you look for files by the name of  “MyZProgram.c”, irrespective of the case in the current directory and all its sub-directories.
# find -iname “MyZProgram.c”
To try out the above command, you can create a few empty sample files and structure using the commands below:

# vim create_sample_files.sh
touch Program.c
touch myzprogram.c
touch MybashProgram.sh
touch MyZProgram.c
mkdir backup
cd backup
touch Program.c
touch myzprogram.c
touch MybashProgram.sh
touch MyZProgram.c
# chmod +x create_sample_files.sh
# ./create_sample_files.sh

After making these files, if you run the above find command, the output you get will be:
./myzprogram.c
./backup/myzprogram.c
./backup/MyZProgram.c
./MyZProgram.c
Now, instead of discussing just the typical uses of the find command, let’s see some of its more advanced and creative uses:
Using the Find command to maintain TMP and LOG file storage spaces
There are certain applications which create a number of temporary and log files which use up space on your computer. By putting the below mentioned commands in a daily cron job you can maintain these files properly on your system.

Command One:
$find $LOGSDIR -type d -mtime +1 -exec compress -r {} ;
Command Two:
$find $LOGSDIR -type d -mtime +7 -exec rm -f {} ;

Command one checks all directories (-type d) which are found in $LOGSDIR directory for a file whose content has been changed more than 48 hours ago (-mtime +1) to compress these files (compress -r {}) and free up some disk space on the system. The next command deletes these files (rm -f {}) in case they are older than one week (-mtime +7), to further free up space. This way you don’t lose disk space on old and unnecessary files.
Now let us take a look at some other advanced ways in which you can use the find command to make your work simpler. The table below lists some such useful commands:

Finding a particular file in complete system
find / -type f -name <file> -print
Finding a certain file pattern in the current directory
find . -type f -name *<too>* –print
Delete all the files with the name “core” present in the system
find / -type f -name core -exec /bin/rm -f {} ;
Finding files in the current directory having a particular <someword>
find . -type f -exec grep -l <someword> {} ;
Find the files in the current directory which were modified more than one month back
find . -type f -ctime +30
Look for the files which are larger than 1 MB in size
find /path -size +1048576c –print
Follow links when searching
find . –L …
Search in the current directory excluding nfs mounted file systems
find . –local …
Find all the directories which have the write permission
find / -type d -perm -002 –print
Find all the manual page directories
find / -type d -print | egrep ‘.*/(catman|man)$’

Working with Files – The grep Command

Under the find command discussion above, we saw how find can be used to find files which have a particular word or pattern in them. It required the use of grep command along with the find command. The grep command is a ‘search’ command for locating text strings within one or more files. Below we will look at some more advanced uses of the command.

We know that using the command, grep ‘fred’ * will search for all occurrences of the text string ‘fred’ in all the files of the current directory. It will also return results which have string like ‘alfred’. To restrict it to returning only occurrences of ‘fred’ we use the carat character. For instance grep ‘^fred’ * makes it look for records where ‘f’ is in the first letter, hence a string like ‘alfred’ is not returned.

By default, grep searches are case sensitive, to make them case insensitive we can add –i to the command. So strings like ‘FrEd’ will also get returned in the above example. Alternatively you can use square brackets to specify different alternatives like

grep “[Ff]red” *

Now suppose we want to count the number of lines that contain a particular string, it is as simple as adding the –c to the command. And by adding –n you can also get the line numbers. And finally, if you want to search for a line with only one type of character, then you can use the asterisk which searches for zero or more repetitions of the preceding match. For example, the command to search for lines which have only a’s or is empty in the file specified by filename would be as follows:

grep “^a*$” filename

Advanced Stream Editor Commands

Next we take a look at some advanced UNIX stream editor commands which are of utmost importance in making your job simpler in file manipulations. While grep helps you look for certain patterns in files, Stream Editor is a UNIX utility used for parsing text files and applying textual transformations to them. Following are some highly useful commands of this category:

sed -e ‘s/[0-9]/number/g’ <file>Replace a specific pattern in the file text
sed -e ‘s/rita/john/g’ <file>Replace a certain string in the file text
ls -1 | grep ‘.sufa$’ | sed -e ‘s/(.*.)sufa/mv & 1sufb/’ | shRename the files having the suffix .sufa to files having the suffix .sufb
sed -e ‘/C/s/X/Y/’ <infile > > <outfile >Change ‘X’ to ‘Y’ on the lines which have C
sed -e ‘s!A!<font color=”#FF0000″>A</font>!g’ fileChange all the  “A” in a html file to the color red
sed -f <commandfilename> <infile> > <outfile>Place multiple edit commands in single file (commandfilename)
sed -e ‘/i owe you/d’ <infile> > <outfile>Delete all the lines which contain “i owe ou”

Compression and Archives

Now that you have found the files you are looking for and modified them as required, you may realize that you don’t really need all of them on a routine basis, or maybe you just need more space on your system, or may be you have to send them to someone over the internet. A good option in such cases is to compress your files and make them into an archive. It will free up space on your hard disc and take less time in being sent over the network.
Following are some useful commands which will help you achieve the same. These have been tested both on Ubuntu and Fedora.

tar -c dir/ | bzip2 > dir.tar.bz2Create a compressed archive of the dir /
tar -c dir/ | gzip | ssh username@remote ‘dd of= dir.tar.gz’Create an encrypted archive of dir / on the remote machine
bzip2 -dc dir.tar.bz2 | tar -xExtract the archive (In place of bzip2 for tar.gz files use gzip)
find dir/ -name ‘*.txt’ | xargs cp -a –target-directory=dir_txt/ –parentsCreate a copy of the subset of dir/ and contents below
( tar -c /dir/to/copy ) | ssh -C username@remote ‘cd /where/to/ && tar –x -p’Copy (alongwith permission) copy /dir to remote :/where/to/dir
( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p )Copy (along with permission) contents of copy /dir to /where/to/
dd bs=1M if=/dev/sda | gzip | ssh username@remote ‘dd of=sda.gz’Backup the hard drive to a remote machine
gpg file.gpgDecrypt a particular file
gpg -c fileEncrypt a particular file

Batch Image Resizing

When discussing the advantages of command line earlier, we had mentioned resizing images in a batch, let us take a look at how we can do that. We will make use of Image Magick, which is an open source graphics suite. Using Image Magick, resizing or even changing the quality of an image is just one command away:

convert image_name -resize  (dimension) -quality  (quality)% output_name

For instance:

convert myImage.jpg -resize 750×750 -quality 90% myImage_new.jpg

The above command will convert ‘myImage.jpg’ to a size 750 by 750 and reduce its quality to 90%, and finally save it as ‘myImage_new.jpg’.
To do the same with a number of image files, we can use the following command:

mogrify -resize 750×750 -quality 90% “*” *.jpg

The above command resizes and changes the quality of all the .jpg files in the directory.

More Image Manipulation

While talking about Image Magick, we should also mention the other advanced uses that we can put it to. We can easily convert one image type to another using the mogrify command as below

mogrify -format png *.tiff

The above command converts all the .tiff images files to png type and saves them.

Now suppose we had to make thumbnails of all the jpeg files in the directory, we could do so using
convert -size 160×160 *.jpg -resize 160×160 +profile ‘*’ $$-thumb.jpg

The ‘-size 160×160’ helps run the jpeg decoder faster by telling it that the images are being downgraded and hence high resolution is not required. The ‘-resize 160×160’ specifies the dimensions for the thumbnails, and finally, ‘+profile ‘*’’ removes any profile from the image such as EXIF, ICM etc which is not required in a thumbnail, the $$-thumb.jpg will create a new filename for the thumb image.

Quickly determine the shell you are using

Unix supports a lot of different shells, each has its own distinct advantages and disadvantages. While they all support majority of the commands used in UNIX they can have different functionality when it comes to more advanced commands or tools. Thus it becomes imperative to know the UNIX shell that you are using so as to determine the outcome of your commands. Finding out the shell takes just one simple command:

$echo $0
– bash                 – Output in case of bash shell

File Name Completion

We have often seen how advanced search engines or word editors auto complete words for us. Imagine if there was a way of completing file names on the command line without having to type them manually. Not only will it help save time and effort but will also reduce the chances of errors. Well such a tool is available. It functions differently in different shells, so we will discuss its functionality in some of the most popular ones below. You can find out which shell you are using the command discussed above.

If you are using C Shell, then you can configure the shell for file name completion by the command set filec. Then to use it, simply press Escape while typing in the file name and shell will fill in as much of the filename as it can for you. For example if you have three files named as myfile1, myfile2, and myfile3 then pressing Escape after typing ‘m’ will auto complete to myfile and you can complete the file name by mentioning the appropriate number.

For Bash Shell users, the file name completion is set by default, but instead of the Escape key, you need to press the tab key. So in the above example, the shell will fill in myfile for you. An additional feature of Bash shell is that if you press the tab key again now, it will list for you the files which match what you have typed so far, myfile1 myfile2 myfile3 in our example.

Korn Shell users with EDITOR set to vi need to press Escape followed by backslash to fill in the file name. Where as with the EDITOR set to emacs pressing the Escape key twice will do the trick.

Quicker Directory Navigation

In a multi directory system, as our systems mostly are, navigating from one directory to another specially to deep nested directory and back to starting directory can take quite a bit of effort and time. However by using pushd and popd you can speed up your directory navigation.

The pushd command changes your current directory and stores it onto a stack, where as the popd command removes the directory from the stack and takes you back to that directory.

The following commands illustrate the use of pushd and popd and the contents of the stack with it.

$ pushd .
~ ~
$ pushd /etc
/etc ~ ~
$ pushd /var
/var /etc ~ ~
$ pushd /usr/local/bin
/usr/local/bin /var /etc ~ ~
$ dirs
/usr/local/bin /var /etc ~ ~
$ popd
/var /etc ~ ~
$ popd
/etc ~ ~
$ popd
~ ~
$ popd

To just see the stack and the directories present in it without adding or removing any directory from it you can use the dirs command.

Now to make it more helpful, you can use the alias command to change the meaning of the simple cd command and to increase its functionality.
alias cd = ‘pushd’
alias cd – = ‘popd’
alian cd — = ‘popd 2’
So navigating to the second last directory you were at would simply require cd –.

Commands History

We have discussed so many useful commands above, wouldn’t it be great if we could keep a history of all the commands we use. Well we can, UNIX calls them events and assigns an event number to each of them. We can have the number displayed with the command prompt by using
Set prompt= ‘!%’
If you would like to have the event numbers displayed by default every time you log on, you can simply add the above command to the hidden .cshrc start up file. You can even change the number of commands you would like to be kept in the history by changing it in the .cshrc file or using the command
Set history=90
Suppose you want to save 10 most recent commands across the login sessions, then the command to be used is
Set savehist=10
To look at the commands stored in your history, simply type the command history. Not only does UNIX let us maintain the history of events, it also lets us repeat any command by simply following the event number. We will discuss this in the next section.

Repeating Commands

Using the event numbers discussed above, we can repeat any of the commands in the history list and even add variations and supply different arguments to them. To repeat a command you can refer to it in one of the following ways:

  1. Using its exact event number
  2. Using its relative number as compared to the current event
  3. Using the text present in the command

Whichever way you choose to follow, you have to precede your reference with the exclamation sign.
For instance, !! will repeat the previous command, where as !-5 will repeat the 5th preceding command in the current window. !gre repeats the most recent command which began with ‘gre’. The shorthand !$ can be used to refer to the last word of the preceeding command.
Suppose you have to modify a previous command before executing it, then you can use a colon and modifier to do so. For instance, say you mistyped a command as

cay firstfile secondfile
cay: Command not found
!!:s/cay/cat
The ‘s’ used here is a modifier which replaces the first occurrence of first string specified with the second string.
You can use the following modifiers to change your commands:

ModifierStands ForAction
RRootTo remove the filename extension
HHeadTo remove the last element from a path name
TTailTo remove all the elements except the last part of a path name
PPrintTo not execute the changed event

Command Aliases

The above reuse of commands through the history is a very handy feature, but in case you need to use commands which may not fall in the history range, or want to shorten a long command to a shorter one for ease of use, or simply to give it a more memorable name, you can use alias.

For instance, ‘alias bye logout’ command will let you use bye in place of logout to leave the session. To make permanent aliases you can add them to your hidden .cshrc start up file. You may find other aliases too in the file and can comment them out using ‘#’ if you so wish.

You can have multiple commands grouped into a single alias by separating the commands with a semicolon at the time of declaration. This is illustrated below:
alias pls pwd;ls –sFC
The above command makes an alias ‘pls’ which displays not just the current working directory, but also the files in it.

You can also add arguments inside aliased commands by using the !* place holder. To display a list of all the aliases in effect, simply type ‘alias’ command. To delink any alias from its command use the unalias command, for instance,
unalias bye

Controlling Jobs

When working on complex programs or doing many things at once, say even trying out all the commands we have discussed above, it can be quite frustrating to wait for a command to finish before we can start a new one. But with Unix being multitasking, we can have multiple jobs running at the same time. They can be running in the foreground (user will have to wait for the job to finish before he can enter any other command), or in the background (user does not have to wait for the job to finish before he can give new commands), or be suspended. By default, commands run in the foreground.

We define jobs as a group of one or many commands. The command
$pwd; ls

consists of one job with two commands, where as

$pwd
$ls
are two jobs with one command each.

To make any command or job run in the background, an ampersand sign must be appended at the end of the command. For instance, $who & will make the command run in the background.
If you want to free up memory, you can suspend any foreground or background job which is running. To suspend a foreground job, simply type ^z. For suspending a background job, you need to enter the command:
$stop %jobnumber
The jobnumber can be obtained by using the ‘jobs’ command which gives a list of all the stopped and background processes.

You can resume the job at a later time by the command $fg [%jobnumber] for a foreground job and $bg [%jobnumber] for a background job.

Using the kill command, you can terminate any job.

Copy Buffer Commands

Before we finish this article, lets take a look at how we can copy the screen copy buffer to X’s copy buffer using xsel (or pbcopy for Mac OSX)from Conrad Parker, available at http://www.vergenet.net/

Using this command, you can type c-a-b, that is ctrl followed by a then b, and save the buffer copy to /tmp/screen-exchange which then gets copied to X clipboard using xsel.

  1. Start by installing xsel from the above mentioned link.
  2. Add the following line to the .screenrc file

bind b eval writebuf ‘exec/bin/sh-c “xsel -I –b < /tmp/screen-exchange”’’exec/bin/sh –c “killall xsel”’

  1. Re-start the screen.
  2. Enter copy mode by pressing C-a.
  3. Using the vi movement keystrokes, that is j,k,h,l etc, and space bar to start and end the selection, select some text.
  4. Enter C-a-b and xsel will copy the screen buffer to X buffer.
  5. You can now easily the paste the copy buffer to any X program you require.

That’s it for now. Hope you find all this information useful and are able to make good use of it.

Resources:
http://www.ibm.com/developerworks/aix/library/au-unix-find.html
http://www.thegeekstuff.com/2009/03/15-practical-linux-find-command-examples/
http://www.pixelbeat.org/cmdline.html
http://www.ibm.com/developerworks/aix/library/au-badunixhabits.html
http://www.commandlinefu.com/commands/browse
http://members.unine.ch/philippe.renard/unix2.html
http://www.ssel.montana.edu/HowTo/adv_unix.html
http://unixfaq.blogspot.com/2008/07/advanced-unix-commands.html
http://members.unine.ch/philippe.renard/unix2.html
http://linux.about.com/od/commands/l/blcmdl1_mogrify.htm
http://www.cs.nott.ac.uk/~dxs/advanced-unix.pdf
http://www.hawaii.edu/itsdocs/cen/unxadv.pdf
http://www.ibm.com/developerworks/aix/library/au-unixtips/index.html?S_TACT=105AGX20&S_CMP=EDU
http://www.brad.ac.uk/lss/documentation/unix-adws1/unix-adws1.pdf

(Visited 68 times, 1 visits today)