"awk" Command

Learn all about awk command

HOME

one of the most prominent text-processing utility on GNU/Linux. It can solve complex text processing tasks with a few lines of code.AWK is an interpreted programming language. It is very powerful and specially designed for text processing. Its name is derived from the family names of its authors - Alfred Aho, Peter Weinberger, and Brian Kernighan.

To become an expert AWK programmer, you need to know its internals. AWK follows a simple workflow - Read, Execute, and Repeat.

Read
AWK reads a line from the input stream (file, pipe, or stdin) and stores it in memory.

Execute
All AWK commands are applied sequentially on the input. By default AWK execute commands on every line. We can restrict this by providing patterns.

Repeat
This process repeats until the file reaches its end.

The basic syntax of AWK:

awk 'BEGIN {BEGIN_BLOCK_ACTION} {BODY_BLOCK_FOR_EACHLINE} END {END_BLOCK_ACTION}' filename

(1) BEGIN block Syntax

BEGIN {awk-commands}

The BEGIN block gets executed once at program start-up. It's good place to initialize variables or print some meaningful statements. 'BEGIN' is an awk keyword so it must be in upper-case; kindly note that this block is optional.

(2) Body Block Syntax

/pattern/ {awk-commands}

AWK command applies the body block on every input line. By Default it executes commands on every line; but it can be restricted by providing patterns. Note that there are no keywords for the Body block.

(3) END Block Syntax

END {awk-commands}

The END block executes once at the end of the program. END is an AWK keyword so it must be in upper-case. Kindly, note that this block is optional.

More Information on awk can be found through its manual page or help:

root:~# awk --help

root:~# man awk

Let us create cricketer_details.txt file which has the following content, which will be used in examples.

root:~#cat cricketer_details.txt
10  Virat      OneDown     Right   5200
20  Rohit      Opener      Right   5540
30  Dhawan     Opener      Left    7340
40  Yuvaraj    Middle      Left    9300
50  Rahane     OneDown     Right   6200

Example-1

1.1 Default behavior of awk. As discussed, by default awk prints every line from the file.

root:~#awk '{print ;}' cricketer_details.txt
10  Virat      OneDown     Right   5200
20  Rohit      Opener      Right   5540
30  Dhawan     Opener      Left    7340
40  Yuvaraj    Middle      Left    9300
50  Rahane     OneDown     Right   6200

1.2 For print the lines that matches with the pattern; provide following command:

root:~# awk '/Rohit/
> /Rahane/' cricketer_details.txt

In the above example it prints all the line which matches with the "Rohit" or "Rahane"; It has two patterns. Awk accepts any number of patterns, but each set (patterns and its corresponding actions) has to be separated by newline.

1.3 Print only specific field.

awk has number of built in variables. For each record (i.e line), it splits the record delimited by whitespace character by default and stores it in the $n variables. If the line has 4 words, it will be stored in $1, $2, $3 and $4. $0 represents whole line. NF is a built in variable that represents total number of fields in a record.

root:~# awk '{print $2,$5;}' cricketer_details.txt
Virat 5200
Rohit 5540
Dhawan 7340
Yuvaraj 9300
Rahane 6200

root:~# awk '{print $2,$NF;}' cricketer_details.txt
Virat 5200
Rohit 5540
Dhawan 7340
Yuvaraj 9300
Rahane 6200

In the above example $2 and $5 represents Name and Run respectively. We can get the Run by using $NF also, where $NF represents last field. In the print statement , is a concatenator.

Example-2

2.1 Initialization and Final Action.

Awk has two important patterns which are specified by the keyword called BEGIN and END. Actions specified in the BEGIN section will be executed before starts reading the lines from the input. END actions will be performed after completing the reading and processing the lines from the input.

root:~# awk 'BEGIN {print "Name\tPosition\tHanded\tRun";}
{print $2,"\t",$3,"\t",$4,"\t",$NF;}
END{print "Report Generated\n--------------";}
' cricketer_details.txt
Name    Position    Handed    Run
Virat      OneDown      Right      5200
Rohit      Opener      Right      5540
Dhawan      Opener      Left      7340
Yuvaraj      Middle      Left      9300
Rahane      OneDown      Right      6200
Report Generated
--------------

2.2 Find the batsman who has total run greater than 6000

root:~# awk '$5 > 6000' cricketer_details.txt
30  Dhawan      Opener      Left    7340
40  Yuvaraj     Middle      Left    9300
50  Rahane      OneDown     Right   6200

In the above example, last field ($5) is total run. So if $5 is greater than 6000, then just do the default print action to print the whole line.

2.3 Print the list of Right Handed batsman

root:~# awk '$4 ~/Right/' cricketer_details.txt
10  Virat       OneDown     Right   5200
20  Rohit       Opener      Right   5540
50  Rahane      OneDown     Right   6200

Operator ~ is for comparing with the regular expressions. If it matches the default action i.e print whole line will be performed.

2.4 Print number of Left Handed Batsman

In below example, checks if the Batsman is Left Handed; if yes then in the Action, just increment the count variable, which was initialized with zero in the BEGIN section.

root:~# awk 'BEGIN { count=0;}
$4 ~ /Left/ { count++; }
END { print "Number of Left Handed Batsman =",count;}' cricketer_details.txt

Example-3

There are two types of built-in variables in Awk.

1. Variable which defines values which can be changed such as field separator and record separator.
2. Variable which can be used for processing and reports such as Number of records, number of fields.

3.1 Input field separator variable - 'FS' Example

awk reads and parses each line from input based on whitespace character by default and set the variables $1,$2 and etc. awk FS variable is used to set the field separator for each record. awk FS can be set to any single character or regular expression. You can use input field separator using one of the following two options:

Using -F command line option.
Awk FS can be set like normal variable.
Syntax:

$ awk -F 'FS' 'commands' inputfilename

(or)

$ awk 'BEGIN{FS="FS";}'

awk FS is any single character or regular expression which you want to use as a input field separator. awk FS can be changed any number of times, it retains its values until its explicitly changed. If you want to change the field separator, its better to change before you read the line, So that change affects the line what you read.
Here is an awk FS example to read the /etc/passwd file which has ":" as field delimiter.

root:~# cat etc_passwd.awk
BEGIN{
FS=":";
print "Name\tUserID\tGroupID\tHomeDirectory";
}
{
    print $1"\t"$3"\t"$4"\t"$6;
}
END {
    print NR,"Records Processed";
}

root:~# awk -f etc_passwd.awk /etc/passwd
Name    UserID    GroupID    HomeDirectory
root    0    0    /root
syslog    102    102    /home/syslog
klog    103    103    /home/klog
nobody    65534    65534    /home
4 Records Processed

Example-4

4.1 OFS (Output Field Separator) Example

awk OFS is an output equivalent of awk FS input variable. By default awk OFS is a single space character as shown in following example.

root:~# awk -F':' '{print $3,$4;}' /etc/passwd
305 216
1 1
2 2
3 4
4 7
5 0

Concatenator in the print statement "," concatenates two parameters with a space which is the value of awk OFS by default. awk OFS value will be inserted between fields in the output as shown below in example.

root:~# awk -F':' 'BEGIN {OFS="="} {print $3,$4;}' /etc/passwd
305=216
1=1
2=2
3=4
4=7
5=0

Example-5

5.1 RS (Record Separator) Example

The first `My_List', represents a list of computer bulletin board systems together with information about those systems. each record contains the name of a computer bulletin board, its phone number, the board's baud rate(s), and a code for the number of hours it is operational. An `A' in the last column means the board operates 24 hours a day. A `B' in the last column means the board only operates on evening and weekend hours. A `C' means the board operates only on weekends:

root:~# cat My_List
aardvark     555-5553     1200/300          A
alpo-net     555-3412     2400/1200/300     B
camelot      555-0542     300               C
core         555-2912     1200/300          A
sdace        555-3430     2400/1200/300     B
sabafoo      555-2127     1200/300          C

root:~# awk 'BEGIN {RS="/"} {print $0 }' My_List
aardvark     555-5553     1200
300          A
alpo-net     555-3412     2400
1200
300     B
camelot      555-0542     300               C
core         555-2912     1200
300          A
sdace        555-3430     2400
1200
300     B
sabafoo      555-2127     1200
300          C

5.2 RS (Record Separator) Example

Create one address file as shown below:

root:~# cat address.txt
Parth Shah
123 Main Street
Anywhere, IN 12345-67890

Gopal Shah
456 Tree-lined Avenue
Smallville, GJ 98765-43210

root:~# cat address.awk
# address.awk --- simple address list program

# Records are separated by blank lines.
# Each line is one field.
BEGIN { RS = "" ; FS = "\n" }

{
      print "Name is:", $1
      print "Address is:", $2
      print "City and State are:", $3
      print ""
}

Here, adresses are separated by new-line hence FS is given '\n' and Records (RS) are separated by runs of blank lines.

root:~# awk -f address.awk address.txt
Name is: Parth Shah
Address is: 123 Main Street
City and State are: Anywhere, IN 12345-67890

Name is: Gopal Shah
Address is: 456 Tree-lined Avenue
City and State are: Smallville, GJ 98765-43210

Example-6

6.1 ORS (Output Record Separator) Example

ORS is an Output equivalent of RS. Each record in the output will be printed with this delimiter. Following is an awk ORS example: Refer Example-5, for what contents should be available in address.txt file. In this example, ORS is set as \n---\n.

root:~# cat address.awk
# addrs.awk --- simple mailing list program

# Records are separated by blank lines.
# Each line is one field.
BEGIN {ORS="\n---\n"; RS = "" ; FS = "\n" }

{
      print "Name is:", $1
      print "Address is:", $2
      print "City and State are:", $3
      print ""
}

root:~# awk -f address.awk address.txt
Name is: Parth Shah
---
Address is: 123 Main Street
---
City and State are: Anywhere, IN 12345-67890
---

---
Name is: Gopal Shah
---
Address is: 456 Tree-lined Avenue
---
City and State are: Smallville, GJ 98765-43210
---

---

Example-7

7.1 NR (Number of Records) Example

NR gives you the total number of records being processed or line number. In the following example, NR variable has line number, in the END section awk NR tells you the total number of records in a file. Contents of My_list file can be found in Example-5.

root:~# awk '{print "Processing Record - ",NR;}END {print NR, "Records are processed";}' My_List
Processing Record -  1
Processing Record -  2
Processing Record -  3
Processing Record -  4
Processing Record -  5
Processing Record -  6
6 Records are processed

7.2 NF (Number of Fields) Example

NF gives you the total number of fields in a record. It will be very useful for validating whether all the fields are exist in a record.

root:~# awk '{print NR,"--",NF}' My_List
1 -- 4
2 -- 4
3 -- 4
4 -- 4
5 -- 4
6 -- 4

7.3 FILENAME Example - Name of the current input file

FILENAME variable gives the name of the file being read. Awk can accept number of input files to process.

root:~# awk '{print FILENAME}' My_List address.txt
My_List
My_List
My_List
My_List
My_List
My_List
address.txt
address.txt
address.txt
address.txt
address.txt
address.txt
address.txt

7.4 FNR Example - Number of Records relative to the current input file

When awk reads from the multiple input file, NR variable will give the total number of records relative to all the input file; where FNR will give you number of records for each input file.

root:~# awk '{print FILENAME,FNR}' My_List address.txt
My_List 1
My_List 2
My_List 3
My_List 4
My_List 5
My_List 6
address.txt 1
address.txt 2
address.txt 3
address.txt 4
address.txt 5
address.txt 6
address.txt 7

root:~# awk '{print FILENAME,NR}' My_List address.txt
My_List 1
My_List 2
My_List 3
My_List 4
My_List 5
My_List 6
address.txt 7
address.txt 8
address.txt 9
address.txt 10
address.txt 11
address.txt 12
address.txt 13