Arkham's Eyrie » shell

Sed by examples, Part 2

Arkham — Mon, 29 Jun 2009 16:19:48 +0000

sed (stream editor) is a Unix utility which (a) parses text files and (b) implements a programming language which can apply textual transformations to such files.
It reads input files line by line (sequentially), applying the operation which has been specified via the command line (or a sed script), and then outputs the line.

Substitution

Substitute, for every line, the first occurence of foo (if any) with spam:

# test.txt
foo bar spam foo
spam foo bar egg
sausage foo spam egg
spam spam spam

sed -e 's/foo/spam/' test.txt

Substitute every occurence of foo with spam:
```
sed -e 's/foo/spam/g' test.txt
```
Substitute every occurence of foo with spam in the first two lines:
```
sed -e '1,2s/foo/spam/g' test.txt
```
Substitute every occurence of foo with spam in every line that starts with sausage and ends with egg:
```
sed -e '/^sausage.*egg$/s/foo/spam/g' test.txt
```

Substitute paths:

sed -e 's|/usr/bin/|/usr/local/bin|g' script.sh

Remove html tags:



Hello World!

sed -e 's/<[^>]*>//g' hello.html

Advanced substitution

Append (that’s what she said) to every line ( & represents what matches):
```
sed -e "s/^.*$/& (that's what she said)/g" test.txt
```

Append lol, rofl and lmao to the first three words of every line:

sed -e 's/\(^[^ ]*\) \([^ ]*\) \([^ ]*\)/\1lol \2rofl \3lmao/' test.txt

Multiple Commands

Print a file alternating row numbers:
```
sed -n -e '=;p' test.txt
```

Print some information about your cpu first core:

# script.sed
1,/^$/{
    /model name/p
    /flags/p
    /bogomips/p
}

sed -n -f script.sed /proc/cpuinfo

Add a line before/after each line or replace line:

# script.sed
i\
This line will be inserted before each line

# script.sed
a\
This line will be inserted after each line

# script.sed
c\
This line will be inserted in each line

sed -f script.sed test.txt

Sed by examples

Arkham — Sun, 28 Jun 2009 23:16:40 +0000

sed (stream editor) is a Unix utility which (a) parses text files and (b) implements a programming language which can apply textual transformations to such files.
It reads input files line by line (sequentially), applying the operation which has been specified via the command line (or a sed script), and then outputs the line.

Delete

Delete the first line of a file:
```
sed -e '1d' /etc/fstab
```
Delete from the second to the tenth line:
```
sed -e '2,10d' /etc/fstab
```
Delete lines starting with #:
```
sed -e '/^#/d' /etc/fstab
```

Print

Print lines starting with #:
```
sed -n -e '/^#/p' /etc/fstab
```
Print blocks of text enclosed by BEGIN and END:
```
sed -n -e '/BEGIN/,/END/p' script.awk
```

Print Device section in xorg.conf:

sed -n -e '/Section "Device"/,/EndSection/p' /etc/X11/xorg.conf

Print main function in a C source file:

sed -n -e '/main[[:space:]]*(/,/^}/p' source.c

How to colorize bash prompt easily

Arkham — Wed, 24 Jun 2009 18:02:34 +0000

Sometimes, the bash prompt can look a little dull and it may become difficult to recognize where the output of a command ends. Here I provide with a very simple way to customize your bash prompt by adding some colors.

First of all, add these colors definition to your .bashrc

## Fancy colors
red='\[\e[0;31m\]'
RED='\[\e[1;31m\]'
blue='\[\e[0;34m\]'
BLUE='\[\e[1;34m\]'
cyan='\[\e[0;36m\]'
CYAN='\[\e[1;36m\]'
black='\[\e[0;30m\]'
BLACK='\[\e[1;30m\]'
green='\[\e[0;32m\]'
GREEN='\[\e[1;32m\]'
yellow='\[\e[0;33m\]'
YELLOW='\[\e[1;33m\]'
magenta='\[\e[0;35m\]'
MAGENTA='\[\e[1;35m\]'
white='\[\e[0;37m\]'
WHITE='\[\e[1;37m\]'
NC='\[\e[0m\]' # No Color

The variables you want to use are

\u – username
\h – host name
\w – current absolute path
\W for current relative path
\$ – te prompt character (eg. ‘#’)

For example, my current setup looks like:

PS1="${green}\u${NC}@${green}\h ${BLUE}\w${NC} ${GREEN}\$ ${NC} "

Enjoy

Awk by examples

Arkham — Tue, 14 Apr 2009 14:27:40 +0000

AWK is a language for processing files of text. A file is treated as a sequence of records, and by default each line is a record. Each line is broken up into a sequence of fields, so we can think of the first word in a line as the first field, the second word as the second field, and so on. An AWK program is of a sequence of pattern-action statements. AWK reads the input a line at a time. A line is scanned for each pattern in the program, and for each pattern that matches, the associated action is executed.

Basics

Print /etc/mtab (awk reads a line at time and prints it) :
```
awk '{ print }' /etc/mtab
```
Print /etc/mtab ($0 denotes the whole line) :
```
awk '{ print $0 }' /etc/mtab
```
List mounted filesystems ($1 denotes the first element of the line) :
```
awk '{ print $1 }' /etc/mtab
```
List groups (-F chooses the Field Separator) :
```
awk -F":" '{ print $1 }' /etc/group
```
List groups and id (awk concatenates print() arguments) :
```
awk -F":" '{ print $1 " " $3 }' /etc/group
```

List groups and id (Nicer format) :

awk -F":" '{ print "group: " $1 "\tid: " $3 }' /etc/group

Launching external awk scripts :

# first.awk
BEGIN {
    FS=":"
}
{ print $1 }

awk -f first.awk /etc/mtab

List IPv4 addressess :
```
ifconfig | awk '/inet / { print $2 }'
```

List processes run by root :

# psroot.awk
$1 == "root" { 
    printf("ROOT: "); 
    for (i=11; i<=NF; i++) 
        printf("%s ", $i);
    printf("\n")
}

ps au | awk -f psroot.awk

Print X warnings :

awk '$1 ~ /(WW)/ { print }' /var/log/Xorg.0.log

Print a file removing comments :
```
awk '! /^#/ { print }' /etc/fstab
```

Print number of files/directories :

ls -lA |  awk 'BEGIN{ x=0 } { x=x+1 } END{ print x-1 }'

Awk as calculator :
```
echo | awk '{ print ((2*5^2-1)%7) }'
```

Count empty lines :

# blanklines.awk
BEGIN { x=0 } 
/^$/  { x=x+1 } 
END   { print "I found " x " blank lines :)" }

awk -f blanklines.awk /etc/profile

Use regexp as FS (note the difference though) :

echo '   a   b c  d '| awk -F"[ \t\n]+" '{ print $2 }'

echo '   a   b c  d '| awk -F" " '{ print $2 }'

Understand NR and NF :

# nrnf.awk
BEGIN { x=0 } 
      { print "Words on line " NR ": " NF ; x+=NF } 
END   { print "Total lines: " NR ; print "Total words: " x }

awk -f nrnf.awk nrnf.awk