Sed by examples, Part 2

posted by Arkham on June 29th, 2009

sed_logo_white

sed (stream editor) is a Unix utility which (a) parses text files and (b) implements a programming language which can apply textual transformations to such files.
It reads input files line by line (sequentially), applying the operation which has been specified via the command line (or a sed script), and then outputs the line.

Substitution

  1. Substitute, for every line, the first occurence of foo (if any) with spam:

    # test.txt
    foo bar spam foo
    spam foo bar egg
    sausage foo spam egg
    spam spam spam
    sed -e 's/foo/spam/' test.txt
  2. Substitute every occurence of foo with spam:

    sed -e 's/foo/spam/g' test.txt
  3. Substitute every occurence of foo with spam in the first two lines:

    sed -e '1,2s/foo/spam/g' test.txt
  4. Substitute every occurence of foo with spam in every line that starts with sausage and ends with egg:

    sed -e '/^sausage.*egg$/s/foo/spam/g' test.txt
  5. Substitute paths:

    sed -e 's|/usr/bin/|/usr/local/bin|g' script.sh
  6. Remove html tags:

    <html>
    <body>
    Hello World!
    </body>
    </html>
    sed -e 's/<[^>]*>//g' hello.html

Advanced substitution

  1. Append (that’s what she said) to every line ( & represents what matches):

    sed -e "s/^.*$/& (that's what she said)/g" test.txt
  2. Append lol, rofl and lmao to the first three words of every line:

    sed -e 's/\(^[^ ]*\) \([^ ]*\) \([^ ]*\)/\1lol \2rofl \3lmao/' test.txt

Multiple Commands

  1. Print a file alternating row numbers:

    sed -n -e '=;p' test.txt
  2. Print some information about your cpu first core:

    # script.sed
    1,/^$/{
        /model name/p
        /flags/p
        /bogomips/p
    }
    sed -n -f script.sed /proc/cpuinfo
  3. Add a line before/after each line or replace line:

    # script.sed
    i\
    This line will be inserted before each line
    # script.sed
    a\
    This line will be inserted after each line
    # script.sed
    c\
    This line will be inserted in each line
    sed -f script.sed test.txt
  4. Tags: , , , ,

Sed by examples

posted by Arkham on June 29th, 2009

Canon Tosh SED

sed (stream editor) is a Unix utility which (a) parses text files and (b) implements a programming language which can apply textual transformations to such files.
It reads input files line by line (sequentially), applying the operation which has been specified via the command line (or a sed script), and then outputs the line.

Delete

  1. Delete the first line of a file:

    sed -e '1d' /etc/fstab
  2. Delete from the second to the tenth line:

    sed -e '2,10d' /etc/fstab
  3. Delete lines starting with #:

    sed -e '/^#/d' /etc/fstab

Print

  1. Print lines starting with #:

    sed -n -e '/^#/p' /etc/fstab
  2. Print blocks of text enclosed by BEGIN and END:

    sed -n -e '/BEGIN/,/END/p' script.awk
  3. Print Device section in xorg.conf:

    sed -n -e '/Section "Device"/,/EndSection/p' /etc/X11/xorg.conf
  4. Print main function in a C source file:

    sed -n -e '/main[[:space:]]*(/,/^}/p' source.c

Tags: , , , ,

Awk by examples

posted by Arkham on April 14th, 2009

auk

AWK is a language for processing files of text. A file is treated as a sequence of records, and by default each line is a record. Each line is broken up into a sequence of fields, so we can think of the first word in a line as the first field, the second word as the second field, and so on. An AWK program is of a sequence of pattern-action statements. AWK reads the input a line at a time. A line is scanned for each pattern in the program, and for each pattern that matches, the associated action is executed.

Basics

  1. Print /etc/mtab (awk reads a line at time and prints it) :

    awk '{ print }' /etc/mtab
  2. Print /etc/mtab ($0 denotes the whole line) :

    awk '{ print $0 }' /etc/mtab
  3. List mounted filesystems ($1 denotes the first element of the line) :

    awk '{ print $1 }' /etc/mtab
  4. List groups (-F chooses the Field Separator) :

    awk -F":" '{ print $1 }' /etc/group
  5. List groups and id (awk concatenates print() arguments) :

    awk -F":" '{ print $1 " " $3 }' /etc/group
  6. List groups and id (Nicer format) :

    awk -F":" '{ print "group: " $1 "\tid: " $3 }' /etc/group
  7. Launching external awk scripts :

    # first.awk
    BEGIN {
        FS=":"
    }
    { print $1 }
    awk -f first.awk /etc/mtab
  8. List IPv4 addressess :

    ifconfig | awk '/inet / { print $2 }'
  9. List processes run by root :

    # psroot.awk
    $1 == "root" { 
        printf("ROOT: "); 
        for (i=11; i<=NF; i++) 
            printf("%s ", $i);
        printf("\n")
    }
    ps au | awk -f psroot.awk
  10. Print X warnings :

    awk '$1 ~ /(WW)/ { print }' /var/log/Xorg.0.log
  11. Print a file removing comments :

    awk '! /^#/ { print }' /etc/fstab
  12. Print number of files/directories :

    ls -lA |  awk 'BEGIN{ x=0 } { x=x+1 } END{ print x-1 }'
  13. Awk as calculator :

    echo | awk '{ print ((2*5^2-1)%7) }'
  14. Count empty lines :

    # blanklines.awk
    BEGIN { x=0 } 
    /^$/  { x=x+1 } 
    END   { print "I found " x " blank lines :)" }
    awk -f blanklines.awk /etc/profile
  15. Use regexp as FS (note the difference though) :

    echo '   a   b c  d '| awk -F"[ \t\n]+" '{ print $2 }'
    echo '   a   b c  d '| awk -F" " '{ print $2 }'
  16. Understand NR and NF :

    # nrnf.awk
    BEGIN { x=0 } 
          { print "Words on line " NR ": " NF ; x+=NF } 
    END   { print "Total lines: " NR ; print "Total words: " x }
    awk -f nrnf.awk nrnf.awk

Tags: , , , ,

GNU Screen cheat-sheet

posted by Arkham on January 26th, 2008

Screen Home:
http://www.gnu.org/software/screen/

Basics:

  • Ctrl+a c -> create new window
  • Ctrl+a A -> set window name
  • Ctrl+a w -> show all window
  • Ctrl+a 1|2|3|… -> switch to window n
  • Ctrl+a “ -> choose window
  • Ctrl+a Ctrl+a -> switch between last two windows
  • Ctrl+a n -> switch to next window
  • Ctrl+a p -> switch to previous window
  • Ctrl+a d -> detach window
  • Ctrl+a ? -> help
  • Ctrl+a [ -> start copy, move cursor to the copy location, press ENTER, select the chars, press ENTER to copy the selected characters to the buffer
  • Ctrl+a ] -> paste from buffer
  • Ctrl+a : screen foo -> create a new window executing foo and set its name

How to use screen:

  • screen -ls -> list of detached screen
  • screen -r PID -> attach detached screen session
  • screen -S MySession -> start a screen session with name MySession
  • screen -r MySession -> attach screen session with name MySession
  • screen -S MySession -md /usr/bin/foo -> start a screen session called MySession, launching /usr/bin/foo inside of it, and then detach it
  • screen -DR -> if a session is running, reattach; if necessary detach and logout remotely first. If it was not running, create it

Advanced:

  • Ctrl+a S -> create split screen
  • Ctrl+a TAB -> switch between split screens
    If you created a new split screen, the current window is empty. either select an existing window (ctrl a “) or create a new split screen (ctrl a n).
  • Ctrl+a Q -> Kill all regions but the current one.
  • Ctrl+a X -> remove active window from split screen
  • Ctrl+a O -> logout active window (disable output)
  • Ctrl+a I -> login active window (enable output)

Tags: , ,