Awk - A nifty little tool for text manipulation and more.

[briandams]

Dealing with date and time is more or less a common task when batch scripting. Awk provides simple date and time function for basic time/date manipulation needs.
1) systime()
2) strftime()
3) mktime()

1) systime().
This is the the number of seconds since the system epoch. systime is commonly used to create a random number seed.

Code: [Select]

C:\>awk "BEGIN{ print systime(); } "
1389169226

2) strftime().
This is a function to format a timestamp based on the contents of the format string. This is useful if you want to create a time stamp on windows.eg To get the full 4-digits year, use the "%Y" format

Code: [Select]

C:\>awk "BEGIN{ print strftime(\"%Y\") } "
2014


To get YYYY-MM-DD-HH-mm-ss timestamp

Code: [Select]

C:\>awk "BEGIN{ print strftime(\"%Y-%m-%d-%H-%M-%S\") } "
2014-01-08-16-24-23

you can then capture the results in the usual DOS for loop.


3) mktime( date specs )
"date specs" argument to mktime is a string of the form YYYY MM DD HH MM SS.
YYYY = full year
MM = month, 1 to 12
DD = day, 1 to 31
HH = hour, 0 to 23
mm = minute, 0 to 59
SS = seconds, 0 to 59
mktime will create a timestamp similar to systime()
eg

Code: [Select]

C:\>awk "BEGIN{string=\"2014 01 01 0 0 0\"; print mktime(string) } "
1388505600

mktime is commonly use to get time difference. eg compare the date "2014 01 01 0 0 0 " against today's date and get their difference (in secs)

Code: [Select]

C:\>awk "BEGIN{string=\"2014 01 01 0 0 0\"; s=mktime(string); print (systime() - s) } "
664866
this is useful if for example, you are parsing a log file and filtering the date/time column for a specific date.


to be continued

- brianadams

[briandams]

Sometimes you many want to merge a collection of similar items. eg

Code: [Select]

C_1,KOG0155
C_1,KOG0306
C_2,KOG3259
C_3,KOG0931
C_2,KOG3638
C_4,KOG0956
C_6,KOG0155
C_1,KOG0306
C_3,KOG3259
C_4,KOG0931
C_5,KOG3638
C_1,KOG0956

to become something like this:

Code: [Select]

C_1,KOG0155 ,KOG0306,KOG0306,KOG0956
C_2,KOG3259, KOG3638
C_3,KOG0931, KOG3259
C_4,KOG0956, KOG0931
C_6,KOG0155
C_5,KOG3638

You can make use of associative arrays in awk

Code: [Select]

C:\>awk -F"," "{ array[$1] = array[$1]\",\"$2 }END{ for(idx in array) print idx, a[idx]}"
C_3 ,KOG0931,KOG3259
C_4 ,KOG0956,KOG0931
C_5 ,KOG3638
C_6 ,KOG0155
C_1 ,KOG0155,KOG0306,KOG0306,KOG0956
C_2 ,KOG3259,KOG3638


[Salmon Trout]

Lots of awk stuff lately from you.

[Squashman]

That's how you can call an external DOS command and have it displayed inside awk program itself.

getline returns 1 if it finds a record, and 0 if the end of the file is encountered. If there is some error in getting a record, such as a file that cannot be opened, then getline returns -1. It is generally good practice to always explicitly test for >0 while reading a file or handling input from pipes.

to be continued

- brianadams

The error is stored within AWK's error variable.  It does not pass the error back to the calling batch file or CMD window you have open.

[briandams]

The error is stored within AWK's error variable.  It does not pass the error back to the calling batch file or CMD window you have open.

awk internally doesn't have a mechanism for checking file existence such as -f test for linux. so most of the time if you want to do that then have to make a system call , OR to call getline and check -1.

Code: [Select]


C:\>awk "BEGIN{ x=getline < \"ddd\"   ; print x  }"
-1
ERRNO is just a string internal for awk.

Code: [Select]

C:\>awk "BEGIN{ getline < \"ddd\"   ; print ERRNO  }"
No such file or directory

so it doesn't get returned to DOS errorlevel. you can capture it though using exit().

Code: [Select]

C:\>awk "BEGIN{ x=getline < \"ddd\"   ; exit(x)  }"
C:\>echo %errorlevel%
-1

or

Code: [Select]

C:\> awk "BEGIN{ if ((\"ddd\" | getline) <= 0 ) exit(-1) ;   }"  2>nul
C:\>echo %errorlevel%
-1


[Squashman]

awk internally doesn't have a mechanism for checking file existence such as -f test for linux. so most of the time if you want to do that then have to make a system call , OR to call getline and check -1.

Then why not use the shells built-in functionality to check for the file existence before running your AWK command.

Code: [Select]

IF EXIST foo.txt awk.........

[BC_Programmer]

The error is stored within AWK's error variable.  It does not pass the error back to the calling batch file or CMD window you have open.

awk internally doesn't have a mechanism for checking file existence such as -f test for linux. so most of the time if you want to do that then have to make a system call , OR to call getline and check -1.

I can see copy-pasting your posts from another forum practically verbatim, because they had never really been posted here so could be valuable to some.But when responses like the above are copy-pasted verbatim to rather different questions, that's just a bit weird, I think.

[Salmon Trout]

copy-pasting your posts from another forum

I wondered about that.

[briandams]

Then why not use the shells built-in functionality to check for the file existence before running your AWK command.

Code: [Select]

IF EXIST foo.txt awk.........

this can be done in awk as well as shown in the examples but if you want to do it in the shell , thats up to individual.

[briandams]

But when responses like the above are copy-pasted verbatim to rather different questions, that's just a bit weird, I think.

The author for that dostips thread is yours truly . Hence I can copy and paste all I want. I don't have a blog, if not, i would just redirect readers there.  That's not a different question. I just felt the response for the question look a bit similar as i had answered it in dostips. hence the copy and paste.

[briandams]

I wondered about that.

as explained. I am the original author of that dostip thread.

[Squashman]

Not that hard to start a free blog or free website.

[briandams]

Sometimes you may need to filter a file using keywords from another file. say you have file1.txt and file2.txt

Code: [Select]


C:\>type file1.txt
cheese
milk
sausage

C:\>type file2.txt
milk
cheese
popcorn
pasta
milk
sausage
cheese
melon

you want to filter file2.txt with file1.txt such that only those not matching remains. eg

Code: [Select]

popcorn
pasta
melon

We can do this with awk one liner.

Code: [Select]

C:\>awk "FNR==NR{ a[$1] ;next} { if ( !($0 in a) ) { print }    }" file1.txt file2.txt
popcorn
pasta
melon

Explanation:
FNR==NR   : FNR means the number of records read so far. NR means the TOTAL number of records read from all files. Hence, the idiom FNR==NR means to read all the records from the first file and store to array.
When awk finish processing the first file, the FNR and NR would be different values, so the 2nd file will be processed. In this case the

Code: [Select]

if ( !($0 in a) ) { print }
statement just says to compare the item inside the array and print the record if not found.

[briandams]

Here are some commonly used one liners for file/text parsing

1) Deleting last line of a file
2) Deleting first line of file
3) Print a range of lines
4) Print lines not in a range
5) Concatenating two files
6) Transposing a file
7) Print first and last line
8) Print the line above and below a pattern
9) Print all lines until a matched pattern
10) Print  from a matched pattern till the end of file


1) Deleting last line of a file

Code: [Select]

C:\>type myFile.txt
CAT
MAT
RAT

C:\>awk "BEGIN{ RS=\"\0\"} { for(i=1;i<NF;i++) print $i  } " myFile.txt
CAT
MAT


2) Deleting first line of file

Code: [Select]

C:\> awk "NR>1 { print  } " myFile.txt
MAT
RAT


3) Print a range of lines. eg print line 3 to line 5

Code: [Select]

C:\> type myFile.txt
CAT
MAT
RAT
BAT
TAT
DAT
PAT

C:\> awk "NR==3,NR==5{ print  } " myFile.txt
RAT
BAT
TAT

4) Print lines not in a range . eg don't  print lines number 3 to 5

Code: [Select]

C:\>awk "!(NR>=3 && NR<=5) { print  }"  myFile.txt
CAT
MAT
DAT
PAT

5) Concatenating two files

Code: [Select]

C:\>awk "{print}"  file1 file2 > newFile.txt


6) Transposing a file

Code: [Select]

C:\> awk "BEGIN{ORS=\" \"}{print}" myFile.txt
CAT MAT RAT BAT TAT DAT PAT

7) Print first and last line

Code: [Select]

C:\> awk "NR==1;END{print}" myFile.txt
CAT
PAT

8) Print the line above and below a pattern. eg Search for "RAT" and print the lines above and below

Code: [Select]

C:\> type myFile.txt
CAT
MAT
RAT
BAT
TAT
DAT
PAT

C:\> awk "/RAT/{print y;print;f=1;next}f{print;f=0}{y=$0}" myFile.txt
MAT
RAT
BAT


9) Print all lines until a matched pattern. eg Print until the word "BAT" is found

Code: [Select]

C:\> awk "/BAT/{exit}{print}" myFile.txt

10) Print  from a matched pattern till the end of file

Code: [Select]

C:\> awk "/TAT/,0" myFile.txt
TAT
DAT
PAT