GoNomicon — Learn Go. Get hired.

The idea

Picture a web server that wrote 50,000 lines overnight, and your manager asks which page visitors opened most often. No single tool answers that. But small tools that each do one job can be chained: one picks out a column, one groups equal values, one counts them, one ranks the counts. A pipe (|) is the conveyor belt you met in the first lesson: the left tool's output becomes the right tool's input. Chain a few and messy text comes out shaped.

Setup: Windows users run these in the Ubuntu (WSL2) app from the first lesson, in their home folder (~). macOS and Linux use Terminal. A few flags (cat -A, sed -i.bak) behave slightly differently on macOS; when that matters, the lesson says so. Paste this block to make the practice files:

mkdir -p ~/pipes && cd ~/pipes
printf '%s\n' "08:01:12 login alice ok" "08:01:40 login bob denied" "08:02:05 login bob denied" "08:02:31 login Bob retry" "08:03:10 logout alice ok" "08:04:55 login carol ok" "08:05:20 login dave denied" "08:06:02 login dave retry" "08:07:30 login bobby ok" > auth.log
printf '%s\n' "10.0.0.7 GET /home 200 512" "10.0.0.8 GET /help/404 200 2048" "10.0.0.9 GET /cart 404 404" "10.0.0.7 POST /pay 302 88" "10.0.0.8 GET /missing 404 150" "10.0.0.9 GET /home 200 512" "10.0.0.7 GET /cart 200 740" > web.log
printf 'pen,3,2\nruler,1,5\npen,2,2\ncap,4,7\nruler,2,5\n' > orders.csv
printf '10\n9\n100\n' > nums.txt
printf 'Zed\nalice\nBob\nalice\n' > names.txt
printf 'name,city\r\nann,rome\r\nbo,oslo\r\n' > win.csv
printf 'a  b   c\nd e  f\n' > gaps.txt
mkdir -p t
printf 'x\ny\n' > t/one.txt
printf 'z\n' > t/two.txt
printf 'q\nr\ns\n' > "t/three four.txt"

auth.log is who logged in, web.log is web requests (visitor, method, page, status, bytes), and orders.csv is a comma-separated order list with no header: item, quantity, price. win.csv has Windows line endings on purpose (more below). The labs use different files; these are for learning the tools.

sort and uniq -c

sort lays lines out in order. uniq -c stacks identical lines and stamps each stack with its count, but only neighbors stack, so sort first.

sort nums.txt                  # text order, so 100 comes before 9 (the character 1 sorts before 9)
sort -n nums.txt               # -n sorts as numbers: 9, 10, 100
sort -nr nums.txt              # -r reverses, so the largest number is first
sort -u names.txt              # -u keeps just one copy of each line
sort -f names.txt              # -f folds case, so Bob and alice sort as if both were lowercase
sort -t, -k2,2nr orders.csv    # -t sets the separator, -k2,2 means "sort by field 2 only", n = numeric, r = biggest first
sort names.txt | uniq -c       # sort first, then count each run of identical lines
sort names.txt | uniq -d       # -d prints only the lines that appear more than once

Real output:

$ sort -n nums.txt
9
10
100
$ sort -t, -k2,2nr orders.csv
cap,4,7
pen,3,2
pen,2,2
ruler,2,5
ruler,1,5
$ sort names.txt | uniq -c
      1 Bob
      1 Zed
      2 alice
$ sort names.txt | uniq -d
alice

Look at the sort -t, -k2,2nr result: pen,3,2 comes before pen,2,2 because the field-2 values are 3 and 2. Two lines tie on field 2 (pen,2,2 and ruler,2,5), and sort decides between tied lines with its own fallback rule that you did not ask for. That fallback differs between flag styles and between machines, so never rely on it. To control ties, add a second key: sort -t, -k2,2nr -k1,1 sorts by field 2 biggest first, then by field 1 alphabetically.

Two facts about uniq -c. First, it only collapses lines that touch: uniq -c names.txt without sorting prints four lines, each with a count of 1, because the two alice lines are not neighbors. Second, it pads the count with spaces to line up the columns ( 2 alice: the number sits at the right edge of a wide column). Those spaces are real characters. Sometimes that is fine and sometimes it breaks the next step. To get a clean COUNT VALUE line with one space and no padding, rebuild it with awk (you meet it below) or strip the leading spaces with sed:

$ sort names.txt | uniq -c | awk '{print $1, $2}'
1 Bob
1 Zed
2 alice
$ sort names.txt | uniq -c | sed 's/^ *//'
1 Bob
1 Zed
2 alice

One more trap: the order of capital and lowercase letters depends on the computer's language setting. Some machines print Bob Zed alice alice (capitals first), others alice alice Bob Zed. When a result must come out the same everywhere, put LC_ALL=C in front: LC_ALL=C sort names.txt always puts capitals first, in plain byte order. (This sets a variable for just that one command.)

cut and tr

cut is a paper trimmer that keeps one chosen column. tr swaps one set of characters for another, letter by letter.

cut -d, -f1 orders.csv           # -d sets the separator; -f picks field 1
cut -d, -f2,3 orders.csv         # fields 2 and 3 (the separator comes along between them)
cut -d, -f1-2 orders.csv         # a range: fields 1 to 2
cut -c1-5 auth.log               # -c cuts by character position instead: the first 5 characters
echo hello | tr 'el' 'ip'        # translate: every e becomes i, every l becomes p  ->  hippo
echo hello | tr 'a-c' 'A-C'      # ranges work: a->A, b->B, c->C (hello has none, so it is unchanged)
echo Hello | tr '[:lower:]' '[:upper:]'   # [:lower:] and [:upper:] mean all lowercase / all uppercase letters
echo 'a b c' | tr ' ' '\n'       # turn each space into a newline: one word per line
echo hello | tr -d 'l'           # -d deletes every l  ->  heo
echo aaabbb | tr -s 'a'          # -s squeezes a run of the same character into one  ->  abbb

cut counts fields from 1, and a line with no separator is printed whole. tr pairs its two sets up position by position: the first character of the first set becomes the first character of the second, and so on. tr works on its standard input only, so give it a file with < or a pipe: tr 'a-c' 'A-C' < file.

cut treats every single separator as a boundary. With a space as separator, two spaces in a row make an empty field between them:

$ cut -d' ' -f2 gaps.txt

e
$ awk '{print $2}' gaps.txt
b
e
$ tr -s ' ' < gaps.txt | cut -d' ' -f2
b
e

gaps.txt holds a b c and d e f. cut on the first line sees a, an empty field, then b, so field 2 is empty. awk (below) treats any run of spaces as one separator, which is why awk is the usual tool for space-aligned columns. The other fix is to squeeze the runs with tr -s ' ' first.

Carriage returns. Files made on Windows often end each line with an invisible extra character: a carriage return, written \r. A file's lines end in \r\n instead of just \n. It rides along on the last field of every line and quietly breaks comparisons. cat -A shows hidden characters: \r appears as ^M and the real line end as $ (on macOS use cat -e). tr -d '\r' removes them:

$ cat -A win.csv
name,city^M$
ann,rome^M$
bo,oslo^M$
$ tr -d '\r' < win.csv | cut -d, -f2 | cat -A
city$
rome$
oslo$

sed substitutions

sed is a rubber stamp that replaces text it recognizes. s/old/new/ means substitute. A trailing g means every match on the line; without it only the first match per line changes. The patterns work like grep's (^, $, ., [ ], *).

echo a-b-c | sed 's/-/+/g'            # a+b+c   (g: every match)
echo a-b-c | sed 's/-/+/'             # a+b-c   (no g: only the first)
sed 's/^/> /' orders.csv              # ^ is the start of the line: put "> " in front of every line
sed 's/[0-9]//g' orders.csv           # replace every digit with nothing: delete them
echo hello | sed 's/l/[&]/g'          # & stands for whatever was matched  ->  he[l][l]o
echo a/b | sed 's#/#-#'               # any character can be the divider; use # when the text contains /
sed -e 's/pen/PEN/' -e 's/cap/CAP/' orders.csv   # -e adds several steps; each runs on every line
sed -E 's/([a-z]+),([0-9]+)/\2 x \1/' orders.csv # -E: groups ( ) are remembered, \1 and \2 paste them back

Real output of the last one, first two lines: 3 x pen,2 and 1 x ruler,5. Everything the pattern did not match (,2) is left alone.

sed can also delete lines and choose lines. -n silences sed's normal "print every line" so only lines you ask for are shown, with p to print:

sed '/ruler/d' orders.csv     # d deletes lines containing ruler
sed '1d' orders.csv           # delete line 1 (drop a header row)
sed -n '2,3p' orders.csv      # print only lines 2 to 3
sed -n '/cap/p' orders.csv    # print only lines containing cap
sed -n '$p' orders.csv        # $ means the last line

sed normally prints its result and leaves the file alone. sed -i.bak 's/pen/pencil/' orders.csv edits the file itself and keeps the old version as orders.csv.bak. That one changes your data, so look at the plain version first (without -i) and keep the .bak. On macOS sed -i needs a backup suffix, as in -i.bak, so writing it that way works in both places. On Linux, s/pen/x/I makes the match ignore case; macOS sed does not support the I.

awk fields and sums

awk reads each line as columns and can add a column up. $0 is the whole line, $1 the first field, $2 the second, NF is the number of fields on the line, $NF is the last field, and NR is the number of the line being read.

awk '{print $1}' web.log                 # first whitespace-separated field (any run of spaces is one separator)
awk '{print NR, $3}' auth.log            # line number, then field 3 (a comma between items prints a space)
awk '{print $NF}' auth.log               # the last field of each line
awk -F, '{print $1, $2 * $3}' orders.csv # -F, sets the separator; arithmetic works on fields
awk -F, '{total += $2 * $3} END {print total}' orders.csv   # sum quantity x price over all lines
awk -F, '{q += $2; m += $2 * $3} END {print q, m}' orders.csv  # keep two running totals at once
awk 'END {print NR}' auth.log            # after the last line, NR is how many lines there were

Real output of the arithmetic ones:

$ awk -F, '{print $1, $2 * $3}' orders.csv
pen 6
ruler 5
pen 4
cap 28
ruler 10
$ awk -F, '{total += $2 * $3} END {print total}' orders.csv
53
$ awk -F, '{q += $2; m += $2 * $3} END {print q, m}' orders.csv
12 53

awk reads line by line. The part in { } runs on every line. END { } runs once, after the last line. BEGIN { } runs once, before the first. A variable you never set starts at zero (or empty), so total += ... works without declaring it. Separate steps with ;.

A test or a pattern before the braces makes awk run them only on lines that pass. Text is compared with == in double quotes; numbers with >, >=, <, <=, ==, !=:

$ awk '$4 >= 404 {print $3, $4}' web.log
/cart 404
/missing 404
$ awk '$2 == "POST"' web.log
10.0.0.7 POST /pay 302 88
$ awk '$3 == "bob" {print NR}' auth.log
2
3
$ awk '/denied/ {n++} END {print n}' auth.log
3

With no { } at all, awk prints the whole line when the test passes. Between slashes is a pattern, like grep's. Notice that $4 >= 404 is a test on ONE column, so a 404 in the page name (/help/404) is not fooled into matching. If no line matched /denied/, n would never be set and print n would print an empty line; write print n + 0 to always get a number.

Skipping a header row: awk -F, 'NR == 1 {next} {s += $2} END {print s}' file means "on line 1 skip to the next line; otherwise add field 2". NR > 1 as a test does the same.

printf for formatted output. print is quick; printf gives you control. The pattern in quotes has %s for text, %d for a whole number, %.2f for a number with 2 decimals, and %-6s for text padded to 6 wide. Unlike print, printf adds no newline by itself, so end the pattern with \n:

$ awk -F, '{printf "%-6s %3d\n", $1, $2}' orders.csv
pen      3
ruler    1
pen      2
cap      4
ruler    2
$ awk -F, '{printf "%s costs %.2f\n", $1, $3}' orders.csv | head -n 2
pen costs 2.00
ruler costs 5.00

When a total could be very large, printf "%d\n", total prints it as a whole number, never as 1e+06-style shorthand.

xargs

xargs gathers loose lines and hands them to the next machine as arguments. Many commands, like wc and rm, want file names as arguments, not as lines on standard input. A pipe delivers lines on standard input, so xargs bridges the two.

printf 'a\nb\nc\n' | xargs echo               # a b c        (all lines become one command)
printf 'a\nb\nc\n' | xargs -n 1 echo item:     # -n 1: one argument per run: three runs of echo
printf 'a\nb\nc\n' | xargs -I{} echo '<{}>'    # -I{}: run once per line, putting the line where {} is
grep -l x t/*.txt | xargs wc -c                # run wc -c on every file name grep -l printed
find t -name "*.txt" -print0 | xargs -0 wc -l  # -print0 and -0: separate names with an invisible null instead of whitespace
printf '' | xargs -r echo hi                   # -r: do nothing when there is no input (GNU/WSL; macOS does this by default)

Real output of the first three:

a b c
item: a
item: b
item: c
<a>
<b>
<c>

xargs keeps the order of the lines it was given, so if the order matters, put sort before it: printf 'b\na\n' | sort | xargs echo prints a b. (sort cannot sit in front of -print0's null-separated names, which is a trade-off to remember.)

Names with spaces. xargs splits input on whitespace. t/three four.txt is one file, but plain xargs sees two names:

$ find t -name '*.txt' | xargs wc -l
wc: t/three: No such file or directory
wc: four.txt: No such file or directory
1 t/two.txt
2 t/one.txt
3 total

The -print0 / -0 pair from above fixes it: find t -name '*.txt' -print0 | xargs -0 wc -l counts all three files and reports 6 total. Also: wc -l on several files adds a total line at the end; and for a huge list, xargs may split the work into several runs, each with its own total.

Preview before you destroy. find . -name "*.tmp" | xargs rm deletes everything find found, permanently. Run the find alone first and read the list. A safe way to rehearse is to put echo in front of the command (... | xargs -n 1 echo rm): it prints what would run without running it.

Building a pipeline one stage at a time

A pipeline of five tools is easy if you build it like a real assembly line: add one station, look at what comes out, then add the next. Never write all five and hope.

Question: which pages were requested most often in web.log?

$ awk '{print $3}' web.log
/home
/help/404
/cart
/pay
/missing
/home
/cart
$ awk '{print $3}' web.log | sort | uniq -c
      2 /cart
      1 /help/404
      2 /home
      1 /missing
      1 /pay

Stage 1 picks out the page column. Stage 2 sorts equal pages next to each other. Stage 3 counts each stack. If a result looks wrong, run the pipeline again with the last stage removed and look at what that stage received: the bug is where the output stops matching what you expected. Every stage is a command you already know, and you can run any prefix of the pipeline on its own.

Putting it together: top-N questions

"Which pages were most popular?" is a top-N question. The recipe is always: pick the column, sort so equal values touch, count with uniq -c, rank the counts with sort -nr, and keep the first few with head.

$ awk '{print $3}' web.log | sort | uniq -c | sort -nr | head -n 2
      2 /home
      2 /cart

sort -nr sorts the padded count as a number, biggest first. head -n 2 keeps the top two lines. Here two pages tie on 2 requests, and ties come out in whatever order sort happens to use (on the machine that produced this output /home came before /cart; yours may differ). When a tie must resolve predictably, add a second key as you saw in the sort section:

$ awk '{print $3}' web.log | sort | uniq -c | sort -k1,1nr -k2,2 | head -n 2
      2 /cart
      2 /home

Another example with a different column and a filter first: who was denied the most?

$ grep denied auth.log | awk '{print $3}' | sort | uniq -c | sort -rn | head -n 1
      2 bob

To count how many distinct values there are instead of ranking them: awk '{print $3}' auth.log | sort -u | wc -l prints 6. To get one line like bob: 2, finish with a reshaping awk: ... | awk '{print $2 ": " $1}'.

Common mistakes

Running uniq without sort first. Only lines that touch collapse.
Forgetting that uniq -c pads the count with spaces. Normalize it with awk '{print $1, $2}' or sed 's/^ *//' when the next step needs a clean line.
sort without -n on numbers: 100 lands before 9.
cut -d' ' on columns separated by runs of spaces: every single space is a boundary, so you get empty fields. Use awk or squeeze first with tr -s ' '.
Omitting the g in sed, so only the first match per line changes.
Confusing awk's $0 (the whole line) with $1 (the first field), or NF (how many fields) with $NF (the last one).
Hidden \r from Windows files. If a comparison fails for no visible reason, run cat -A on the file.
Trusting find order. Pipe it through sort before comparing results, and put the sort before xargs.
xargs splits on whitespace, so a filename with a space becomes two arguments. Use -print0 and -0.
Running xargs rm without looking first. Run the find on its own, because deletion is permanent.
Building a long pipeline in one go. Add one stage at a time and read the output.

Try it

Do these in ~/pipes before the graded labs.

Run every command from this lesson and compare your output with the output printed here.
Print the orders.csv item names in alphabetical order, each only once.
Count how many orders each item has, with a clean COUNT ITEM line (single space, no padding).
Print only the item and quantity columns of orders.csv, separated by a tab instead of a comma (hint: tr ',' '\t' on the result of cut).
Use sed to print orders.csv with every comma replaced by |, without changing the file.
Use awk to print each line of orders.csv as ITEM: TOTAL where total is quantity times price.
Use awk to print the total quantity ordered, and then the total of lines with a quantity of at least 2.
Use awk to print the last field of every line in web.log, then the sum of that column with printf "%d\n".
Make win.csv readable: show the hidden characters with cat -A, remove them, and show the result.
Find the most common status code in web.log (field 4), then the top 2 status codes, resolving ties by the smaller code first.
Make an .txt file whose name has a space, find it with find, and run wc -l on it through xargs the safe way.
Rebuild step 10 one stage at a time, running the pipeline after every | you add.

answer.sh is a plain text file of commands that sh answer.sh runs top to bottom. Whatever it prints is graded, so match each lab's exact output format. The sample files live in testdata/: look at them with cat first. Each lab states exactly what your script must print.