Linux gawk command

Updated: 06/30/2017 by Computer Hope

About gawk

gawk is a powerful pattern-matching and processing language. It is based on the language AWK.

Several kinds of tasks occur repeatedly when working with text files. You might want to extract certain lines and discard the rest. Or you may need to make changes wherever certain patterns appear, but leave the rest of the file alone. Writing single-use programs for these tasks in languages such as C, C++, or Java is time-consuming and inconvenient. Such jobs are often easier with awk. The awk utility interprets a special-purpose programming language that makes it easy to handle simple data-reformatting jobs.

The GNU implementation of awk is called gawk; if you invoke it with the proper options or environment variables (see Options), it is fully compatible with the POSIX specification of the awk language and with the Unix version of awk maintained by Brian Kernighan.

Using awk (or gawk) allows you to:

  • Manage small, personal databases
  • Generate reports
  • Validate data
  • Produce indexes and perform other document preparation tasks
  • Experiment with algorithms that you can adapt later to other computer languages

In addition, gawk provides facilities that make it easy to:

  • Extract bits and pieces of data for processing
  • Sort data
  • Perform simple network communications

gawk syntax

gawk [ POSIX or GNU style options ] -f program-file [ -- ] file ... 
gawk [ POSIX or GNU style options ] [ -- ] program-text file ...
pgawk [ POSIX or GNU style options ] -f program-file [ -- ] file ...
pgawk [ POSIX or GNU style options ] [ -- ] program-text file ...
dgawk [ POSIX or GNU style options ] -f program-file [ -- ] file ...

Option Format

gawk options may be either traditional POSIX-style one letter options, or GNU-style long options. POSIX options start with a single "-", while long options start with "--". Long options are provided for both GNU-specific features and for POSIX-mandated features.

gawk-specific options are typically used in long-option form. Arguments to long options are either joined with the option by an = sign, with no intervening spaces, or they may be provided in the next command line argument. Long options may be abbreviated, as long as the abbreviation remains unique.

Additionally, each long option has a corresponding short option, so that the option's functionality may be used from within #! executable scripts.

Options

-f program-file, --file program-file

Read the AWK program source from the file program-file, instead of from the first command line argument. Multiple -f (or --file) options may be used.

-F fs, --field-separator fs

Use fs for the input field separator (the value of the FS predefined variable).

-v var=val, --assign var=val

Assign the value val to the variable var, before execution of the program begins. Such variable values are available to the BEGIN block of an AWK program.

-b, --characters-as-bytes

Treat all input data as single-byte characters. In other words, don't pay any attention to the locale information when attempting to process strings as multibyte characters. The --posix option overrides this option.

-c, --traditional

Run in compatibility mode. In compatibility mode, gawk behaves identically to UNIX awk; none of the GNU-specific extensions are recognized. See GNU EXTENSIONS, below, for more information.

-C, --copyright

Print the short version of the GNU copyright information message on the standard output and exit successfully.

-d[file], --dump-variables[=file]

Print a sorted list of global variables, their types and final values to file. If no file is provided, gawk uses a file named awkvars.out in the current directory. Having a list of all the global variables is a good way to look for typographical errors in your programs. You would also use this option if you have a large program with a lot of functions, and you want to be sure that your functions don't inadvertently use global variables that you meant to be local. This is a particularly easy mistake to make with simple variable names like i, j, and so on.

-e program-text, --source program-text

Use program-text as AWK program source code. This option allows the easy intermixing of library functions (used via the -f and --file options) with source code entered on the command programs used in shell scripts.

-E file, --exec file

Similar to -f, however, this is option is the last one processed. This should be used with #! scripts, particularly for CGI applications, to avoid passing in options or source code (!!!) on the command line from a URL. This option disables command-line variable assignments.

-g, --gen-pot

Scan and parse the AWK program, and generate a GNU .pot (Portable Object Template) format file on standard output with entries for all localizable strings in the program. The program itself is not executed. See the GNU gettext distribution for more information on .pot files.

-h, --help

Print a relatively short summary of the available options on the standard output. Per the GNU Coding Standards, these options cause an immediate, successful exit.

-L [value], --lint[=value]

Provide warnings about constructs that are dubious or non-portable to other AWK implementations. With an optional argument of fatal, lint warnings become fatal errors. This may be drastic, but its use will certainly encourage the development of cleaner AWK programs. With an optional argument of invalid, only warnings about things that are actually invalid are issued. Note: This is not fully implemented yet.

-n, --non-decimal-data

Recognize octal and hexadecimal values in input data. Use this option with great caution!

-N, --use-lc-numeric

This forces gawk to use the locale's decimal point character when parsing input data. Although the POSIX standard requires this behavior, and gawk does so when --posix is in effect, the default is to follow traditional behavior and use a period as the decimal point, even in locales where the period is not the decimal point character. This option overrides the default behavior, without the full draconian strictness of the --posix option.

-O, --optimize

Enable optimizations upon the internal representation of the program. Currently, this includes just simple constant-folding. The gawk maintainer hopes to add additional optimizations over time.

-p[prof_file], --profile[=prof_file]

Send profiling data to prof_file. The default is awkprof.out. When run with gawk, the profile is just a "pretty printed" version of the program. When run with pgawk, the profile contains execution counts of each statement in the program in the left margin and function call counts for each user-defined function.

-P, --posix

This turns on compatibility mode, with the following additional restrictions:

\x escape sequences are not recognized

Only space and tab act as field separators when FS is set to a single space, newline does not

You cannot continue lines after ? and :

The synonym func for the keyword function is not recognized

The operators ** and **= cannot be used in place of ^ and ^=

The fflush() function is not available

-r, --re-interval

Enable the use of interval expressions in regular expression matching. Interval expressions were not traditionally available in the AWK language. The POSIX standard added them, to make awk and egrep consistent with each other. They are enabled by default, but this option remains for use with --traditional.

-R, --command file

Dgawk only. Read stored debugger commands from file.

-S, --sandbox

Runs gawk in sandbox mode, disabling the system() function, input redirection with getline, output redirection with print and printf, and loading dynamic extensions. Command execution (through pipelines) is also disabled. This effectively blocks a script from accessing local resources (except for the files specified on the command line).

-t, --lint-old

Provide warnings about constructs that are not portable to the original version of Unix awk.

-V, --version

Print version information for this particular copy of gawk on the standard output. This is useful mainly for knowing if the current copy of gawk on your system is up to date with respect to whatever the Free Software Foundation is distributing. This is also useful when reporting bugs. Per the GNU Coding Standards, these options cause an immediate, successful exit.

--

Signal the end of options. This is useful to allow further arguments to the AWK program itself to start with a "-". This provides consistency with the argument parsing convention used by most other POSIX programs.

In compatibility mode, any other options are flagged as invalid, but are otherwise ignored. In normal operation, as long as program text has been supplied, unknown options are passed on to the AWK program in the ARGV array for processing. This is particularly useful for running AWK programs via the "#!" executable interpreter mechanism.

AWK Program Execution

An AWK program consists of a sequence of pattern-action statements and optional function definitions.

@include "filename" pattern { action statements }
function name(parameter list) { statements }

gawk first reads the program source from the program-file(s) if specified, from arguments to --source, or from the first non-option argument on the command line. The -f and --source options may be used multiple times on the command line. gawk reads the program text as if all the program files and command line source texts had been concatenated. This is useful for building libraries of AWK functions, without having to include them in each new AWK program that uses them. It also provides the ability to mix library functions with command line programs.

In addition, lines beginning with @include may be used to include other source files into your program, making library use even easier.

The environment variable AWKPATH specifies a search path to use when finding source files named with the -f option. If this variable does not exist, the default path is ".:/usr/local/share/awk". The actual directory may vary, depending upon how gawk was built and installed. If a file name given to the -f option contains a "/" character, no path search is performed.

gawk executes AWK programs in the following order. First, all variable assignments specified via the -v option are performed. Next, gawk compiles the program into an internal form. Then, gawk executes the code in the BEGIN block(s) (if any), and then proceeds to read each file named in the ARGV array (up to ARGV[ARGC]). If there are no files named on the command line, gawk reads the standard input.

If a filename on the command line has the form var=val it is treated as a variable assignment. The variable var will be assigned the value val. This happens after any BEGIN block(s) have been run. Command line variable assignment is most useful for dynamically assigning values to the variables AWK uses to control how input is broken into fields and records. It is also useful for controlling state if multiple passes are needed over a single data file.

If the value of a particular element of ARGV is empty (""), gawk skips over it.

For each input file, if a BEGINFILE rule exists, gawk executes the associated code before processing the contents of the file. Similarly, gawk executes the code associated with ENDFILE after processing the file.

For each record in the input, gawk tests to see if it matches any pattern in the AWK program. For each pattern that the record matches, the associated action is executed. The patterns are tested in the order they occur in the program.

Finally, after all the input is exhausted, gawk executes the code in the END block(s) (if any).

Official gawk User's Guide

If you want to learn more about this incredibly powerful language, check out the GNU gawk User Guide.

gawk examples

gawk '{ num_fields = num_fields + NF }
END { print num_fields }' 

Print the total number of fields in all input lines.

gawk 'length($0) > 80'

Prints every line longer than 80 characters. The sole rule has a relational expression as its pattern, and has no action (so the default action, printing the record, is used).

ls -l files | awk '{ x += $4 } ; END { print "total bytes: " x }'

Prints the total number of bytes used by files.

awk — Interpreter for the AWK text processing programming language.
sed — A utility for filtering and transforming text.