GoNomicon — Learn Go. Get hired.

The idea

A real filing cabinet holds thousands of papers, and a busy server writes thousands of new lines every minute. You cannot read them all, so this lesson is about finding the few that matter: the one failed request, the one oversized file, the one line that explains an outage. Support engineers do this all day.

A log file is a diary. A program adds one line each time something happens: oldest at the top, newest at the bottom. To answer "did anything fail?", you read the diary.

Setup: Windows users work in the Ubuntu (WSL2) app from the last lesson, inside their home folder (~). macOS and Linux use Terminal. Every command in this lesson is meant to be typed, so first make some practice files. Paste this whole block into your terminal (printf and mkdir -p came from the last lesson; head -c 3000 /dev/zero > file makes a file that is exactly 3000 bytes of nothing, so we can test sizes):

mkdir -p ~/practice && cd ~/practice
printf '%s\n' "08:01:12 login alice ok" "08:01:40 login bob denied" "08:02:05 login bob denied" "08:02:31 login Bob retry" "08:03:10 logout alice ok" "08:04:55 login carol ok" "08:05:20 login dave denied" "08:06:02 login dave retry" "08:07:30 login bobby ok" > auth.log
printf 'name,qty\nbolt,4\nnut,9\n' > stock.csv
printf '%s\n' "10.0.0.7 GET /home 200 512" "10.0.0.8 GET /help/404 200 2048" "10.0.0.9 GET /cart 404 404" "10.0.0.7 POST /pay 302 88" "10.0.0.8 GET /missing 404 150" "10.0.0.9 GET /home 200 512" > web.log
mkdir -p site/docs/old site/img site/empty
printf 'todo: fix login page\ntodo: update footer\nlogin works\n' > "site/docs/my notes.txt"
printf 'how to login\nhow to logout\n' > site/docs/readme.txt
printf 'login v1 retired\n' > site/docs/old/2024.txt
printf '<h1>Welcome</h1>\n' > site/index.html
printf 'body { color: red; }\n' > site/style.css
: > site/empty/blank.txt
head -c 3000 /dev/zero > site/img/big.bin
head -c 1025 /dev/zero > site/img/just-over.bin
head -c 1024 /dev/zero > site/img/exactly-1k.bin

You now have stock.csv (a tiny table with a header row), auth.log (who logged in), web.log (web requests: visitor, method, page, status, bytes) and a small site/ folder with files of different sizes. The : > file line makes an empty file. Every example below runs against these, and its real output is printed under it, so you can check yourself. The labs use different files, so the point is the skill, not these particular lines.

Reading files: cat, less, head, and tail

cat dumps the whole diary on your desk. less is a window you scroll through, one screen at a time. head shows the first pages and tail the newest.

cat auth.log              # print the whole file
cat -n stock.csv          # -n numbers each line as it prints
head -n 3 auth.log        # first 3 lines (without -n 3, you get 10)
tail -n 2 auth.log        # last 2 lines
tail -n +2 stock.csv      # from line 2 to the end: skips line 1, handy for a header row
head -n 1 auth.log web.log   # several files: each one gets a ==> name <== banner
tail -f auth.log          # follow: keep printing new lines as they arrive; Ctrl+C stops
less auth.log             # scroll one screen at a time

tail -n 2 auth.log prints:

08:06:02 login dave retry
08:07:30 login bobby ok

tail -n +2 is different from tail -n 2: the plus sign means "start at line 2", not "the last 2 lines". It is how you drop a header row.

To watch tail -f work, open a second terminal tab, go to ~/practice, and run echo "08:09:00 login erin ok" >> auth.log: the new line appears in the first tab at once. tail -f never ends by itself, so press Ctrl+C when you have seen enough.

Inside less: Space moves down a screen, b back up, /login then Enter searches for login, n jumps to the next match, and q quits. less is the right tool for a file too big to cat.

Counting with wc and the < redirect

wc is a tally counter for lines, words, and bytes.

wc -l < auth.log        # count lines; prints just the number
wc -l auth.log          # prints the number and the name
wc -l auth.log web.log  # several files, plus a total line
wc -w auth.log          # count words
wc -c < auth.log        # count bytes

Real output of the first three:

9
9 auth.log
  9 auth.log
  6 web.log
 15 total

(The first line is from wc -l < auth.log, the second from wc -l auth.log, the last three from the two-file version.) The < feeds a file into a command's input. With <, wc never sees the file's name, which is why it prints only 9.

One trap: wc -l counts newline characters, not lines of text. A last line that does not end with a newline is not counted: printf "a\nb" | wc -l prints 1, even though there are two lines. Files written by programs nearly always end with a newline, so you will rarely notice, but it explains an off-by-one when you do.

Searching text with grep

grep is a highlighter: it shows only the lines containing what you asked for. grep PATTERN FILE. The pattern is case-sensitive unless you say otherwise.

grep -n denied auth.log     # matching lines, each starting with its line number
grep -i bob auth.log        # -i: ignore case, so bob, Bob and BOB all match
grep -c login auth.log      # -c: count matching lines instead of printing them
grep -v ok auth.log         # -v: invert, keep lines that do NOT contain ok
grep -w bob auth.log        # -w: whole word only
grep -o denied auth.log     # -o: print only the matching part, one per match
grep -m 2 login auth.log    # -m 2: stop after the first 2 matching lines

Real output:

$ grep -n denied auth.log
2:08:01:40 login bob denied
3:08:02:05 login bob denied
7:08:05:20 login dave denied
$ grep -c login auth.log
8
$ grep -w bob auth.log
08:01:40 login bob denied
08:02:05 login bob denied
$ grep -m 2 login auth.log
08:01:12 login alice ok
08:01:40 login bob denied

Notice grep -c login says 8, not 9: logout does not contain login, so that line is not counted. Notice too that the line numbers shown by -n are followed by a colon, and the line's own 08:01:40 colons come after that.

Combining flags. Single-letter flags can be glued together: grep -in bob auth.log means -i -n. Order does not matter, but a flag that takes a value goes last: -m 2 or -C 2.

$ grep -in bob auth.log
2:08:01:40 login bob denied
3:08:02:05 login bob denied
4:08:02:31 login Bob retry
9:08:07:30 login bobby ok

Line 9 is bobby, which contains bob. That is the "inside a longer word" trap. -w fixes it for words, and padding the pattern with spaces fixes it for anything:

$ grep -c bob auth.log
3
$ grep -c " bob " auth.log
2

" bob " (with a space on each side) matches only a standalone bob. Whenever the thing you are counting could also appear inside something longer, such as a short number inside a longer number or a page name, surround it with spaces or anchors (see the next section) so you count what you mean. -w is not enough for numbers in a web log: it counts / and . as word boundaries, so grep -w 404 web.log also matches the line 10.0.0.8 GET /help/404 200 2048, where 404 is part of the page name.

Context lines. One line rarely explains a failure; the lines around it do. -A N shows N lines After each match, -B N shows N lines Before, and -C N shows both (Context). N is any number you choose.

$ grep -A 1 retry auth.log
08:02:31 login Bob retry
08:03:10 logout alice ok
--
08:06:02 login dave retry
08:07:30 login bobby ok
$ grep -C 2 carol auth.log
08:02:31 login Bob retry
08:03:10 logout alice ok
08:04:55 login carol ok
08:05:20 login dave denied
08:06:02 login dave retry

When two groups of context lines are not touching, grep prints a -- line between them. When they touch or overlap (as in the second example), the groups merge and no -- appears.

Searching many files. Give grep several files, or let -r (recursive) walk a whole folder and every folder inside it. With more than one file, grep puts the file name in front of every line:

$ grep -c login auth.log web.log
auth.log:8
web.log:0
$ grep -r login site | sort
site/docs/my notes.txt:login works
site/docs/my notes.txt:todo: fix login page
site/docs/old/2024.txt:login v1 retired
site/docs/readme.txt:how to login

grep -r visits files in no particular order, so add | sort when you need a fixed order. A few more flags change what -r prints:

grep -rl login site                       # -l: only the NAMES of files that match
grep -rL login site                       # -L: only the names of files that do NOT match
grep -rh login site                       # -h: hide the file-name prefix
grep -rc login site                       # -c per file: every file gets a count, even 0
grep -r --include="*.txt" login site      # only look inside files whose name fits the pattern
grep -r --exclude-dir=old login site      # skip a folder

grep -rl login site | sort prints the three files that contain login (site/docs/my notes.txt, site/docs/old/2024.txt, site/docs/readme.txt). Without -r, grep refuses a folder: grep login site/docs prints grep: site/docs: Is a directory.

Pipes. A pipe hands one command's output to the next, so each step narrows the result:

$ grep login auth.log | grep -vc ok
5

That reads: take the login lines, then count how many of those do not contain ok.

Exit status. Every command hands back a number when it finishes, and you can see it by typing echo $? right after. grep hands back 0 when it found something, 1 when it found nothing, and 2 when it hit a real error:

$ grep zzz auth.log; echo "exit code: $?"
exit code: 1
$ grep login nofile.txt; echo "exit code: $?"
grep: nofile.txt: No such file or directory
exit code: 2

"Nothing found" is not an error, but scripts treat any non-zero number as "this command failed". The shell scripting lesson uses that.

Patterns: a tiny bit of regular expression

The pattern you give grep is more than plain text. It is a regular expression (regex): plain letters plus a few special characters with special meanings. You need only a handful:

^ means the start of the line: grep "^08:04" auth.log
$ means the end of the line: grep "ok$" auth.log
. means any one character: grep "20.8" web.log
[abc] means one character from the set, and [0-9] is a range: grep "^08:0[1-2]" auth.log
* means the character before it, zero or more times: grep "colou*r" matches color and colour
\. means a literal dot (a backslash turns a special character back into plain text): grep "v1\.2"

With -E (extended), three more work as you would expect: | means or, + means one or more of the character before, ? means zero or one, and ( ) group things:

$ grep "^08:0[1-2]" auth.log
08:01:12 login alice ok
08:01:40 login bob denied
08:02:05 login bob denied
08:02:31 login Bob retry
$ grep -E "^08:0[4-9]" auth.log
08:04:55 login carol ok
08:05:20 login dave denied
08:06:02 login dave retry
08:07:30 login bobby ok
$ grep "ok$" auth.log | wc -l
4
$ grep -E "denied|retry" auth.log | wc -l
5

The first example anchors with ^ so the time must be at the very start of the line. The dot is the sneaky one: grep "20.8" web.log matches the line with 2048, because . stands for any character (the 4). To search for text that really contains special characters, escape them (\.) or tell grep to treat the whole pattern as plain text with -F:

$ grep "20.8" web.log
10.0.0.8 GET /help/404 200 2048
$ grep -F "20.8" web.log; echo "exit code: $?"
exit code: 1

Two warnings. First, * in a grep pattern means "repeat the previous character", but * in a shell command (ls *.txt) means "anything"; they are different languages that share a symbol. Second, always put the pattern in quotes so the shell does not rewrite *, |, $ or [ before grep sees them.

Finding files with find

find is a search party that walks the cabinet drawer by drawer, checking each paper's name, kind, size, and date. The shape is find WHERE TESTS: the folder to start in comes first (. means "the drawer I have open"), then the tests every result must pass.

find . -name "*.txt"        # names matching the pattern, from here down
find . -iname "README*"     # -iname ignores case, so readme.txt matches
find . -type d              # directories only (-type f means regular files only)
find . -type f ! -name "*.txt"   # ! means NOT: files whose name is not .txt
find . -type f -empty       # files with zero bytes
find . -maxdepth 1          # look only 1 level down: this folder's direct contents
find docs -name "*.txt"     # start in a different folder; paths start with that folder

Run from inside ~/practice/site, find . -name "*.txt" | sort prints:

./docs/my notes.txt
./docs/old/2024.txt
./docs/readme.txt
./empty/blank.txt

Every test is joined with "and". To mean "or", group with $ ... -o ... $:

$ find . -type f \( -name "*.css" -o -name "*.html" \) | sort
./index.html
./style.css

Always quote the name pattern: -name "*.txt". Unquoted, the shell may expand the * against files in the current folder before find runs. find has no order of its own, so add | sort when order matters. If the starting folder does not exist, find prints find: 'nope': No such file or directory and exits with 1.

Size. -size takes a number and a unit: c bytes, k kilobytes (1024 bytes), M megabytes, G gigabytes. A plain number with no sign means exactly that many units, +N means more than N, and -N means less than N. The catch: find rounds each file UP to a whole number of units before comparing. So a 1025-byte file counts as 2k, and a 6-byte file counts as 1k:

$ find . -type f -size +2k
./img/big.bin
$ find . -type f -size 2k
./img/just-over.bin
$ find . -type f -size -1k
./empty/blank.txt
$ find . -type f -size +1024c | sort
./img/big.bin
./img/just-over.bin

(big.bin is 3000 bytes = 3k. just-over.bin is 1025 bytes = 2k. exactly-1k.bin is 1024 bytes = exactly 1k, so it is not "more than 1k".) That is why -size -1k finds only empty files: anything with even one byte rounds up to 1k. When you need an exact byte comparison, use c: -size +1024c means more than 1024 bytes.

Age. -mtime N is about days since the file's contents last changed; -mmin N is the same in minutes. The signs work the same way (+N more than, -N less than, plain N exactly), but days are rounded DOWN to whole 24-hour periods, which surprises almost everyone. To see it, make files with known ages (touch -d sets a date; this form works on Linux and WSL. macOS's touch has no -d "30 hours ago"; there use WSL or skip this part):

mkdir -p ~/practice/age && cd ~/practice/age
touch now.txt
touch -d "30 hours ago" h30.txt
touch -d "49 hours ago" h49.txt
touch -d "5 days ago" d5.txt
touch -d "9 days ago" d9.txt

$ find . -type f -mtime -2 | sort
./h30.txt
./now.txt
$ find . -type f -mtime 2 | sort
./h49.txt
$ find . -type f -mtime +2 | sort
./d5.txt
./d9.txt
$ find . -type f -mmin -90 | sort
./now.txt
$ find . -type f -mmin +120 | sort
./d5.txt
./d9.txt
./h30.txt
./h49.txt

Read the middle two carefully. h49.txt is 49 hours old, which rounds down to 2 days, so it matches -mtime 2 and matches NEITHER -mtime -2 NOR -mtime +2. -mtime -2 means "fewer than 2 whole days old", and -mtime +2 means "3 or more whole days old". When the question is about hours or minutes, use -mmin and skip the rounding.

Doing something to each file found. -exec COMMAND {} \; runs a command once per result, with {} standing in for the path:

$ find . -type f -name "*.txt" -exec wc -c {} \; | sort
0 ./empty/blank.txt
17 ./docs/old/2024.txt
27 ./docs/readme.txt
53 ./docs/my notes.txt

(Run from ~/practice/site.) It handles names with spaces correctly. find can also delete (-delete), which is permanent: run the find on its own first and read the list before you add anything destructive.

Searching a huge tree like / as a normal user floods the screen with Permission denied lines. Those are error messages, not results; hide them with 2>/dev/null (from the terminal lesson): find / -name "*.conf" 2>/dev/null.

Answering a question from a log

Put the tools together. Support questions usually sound like "how many logins were turned away?", "who is the visitor behind the last request?" or "what happened just before the first failure?". The recipe is always the same: narrow with grep, shrink with head/tail, count with -c or wc -l, and sanity-check by reading the real lines before trusting a number.

grep -c denied auth.log                  # how many lines say denied
tail -n 3 web.log                        # the 3 newest requests
grep -n retry auth.log | tail -n 1       # the LAST retry, with its line number
grep login auth.log | grep -vc ok        # logins that did not end in ok
grep -E "denied|retry" auth.log | wc -l  # lines that mention either word

Before you trust any count, ask what could be matched by accident (a short word inside a longer one, a number inside a path), print the matching lines once and read them, and only then count. When the question has two parts, print the two answers one after another: the shell runs your commands top to bottom, and each command's output just follows the last. To print a separator line between results, use echo "---".

Common mistakes

grep matches anywhere in a line, so bob also finds bobby. Use -w, spaces around the pattern, or ^ and $ anchors.
-w still treats /, . and - as word boundaries, so it is not enough for a number inside a path.
-c counts matching lines, not matching words. grep -o PATTERN FILE | wc -l counts every occurrence.
. in a pattern means any character. Use \. or grep -F for a literal dot.
grep PATTERN FOLDER fails with "Is a directory". Add -r.
-size +1k means MORE than one kilobyte after rounding up, and -mtime -2 means fewer than two WHOLE days. Test with files whose age and size you know.
find prints in no set order. Add | sort before comparing results.
Leaving patterns unquoted: the shell expands * and | first and the command gets the wrong thing.
tail -n +2 and tail -n 2 are not the same thing.
tail -f never ends by itself. Press Ctrl+C.
wc -l counts newlines. A final line with no newline is not counted.

Try it

Do these in ~/practice so you can see every result with your own eyes before the graded labs.

Re-run every command from this lesson and compare your output with the output printed here.
How many lines of auth.log mention dave? Count with grep -c, then print the lines and count them by eye to make sure they agree.
Print only the lines of auth.log that start with 08:05 or 08:06, using one grep pattern.
Print the line of auth.log that contains carol, with 2 lines before it and none after.
List every file under site whose name ends in .html or .css, sorted.
Make ~/practice/age as shown above, then list the files changed within the last 6 hours (use minutes), and the files at least 3 whole days old.
List the files under site bigger than 2 kilobytes. Then explain why exactly-1k.bin is missing from find . -type f -size +1k.
In web.log, count how many requests transferred exactly 512 bytes (the last column). Print the matching lines first, then count, and make sure a 5120 or 1512 could not be counted by accident.
Save the first 3 lines of auth.log into a file with head -n 3 auth.log > top.txt, then check it with wc -l < top.txt.
Run grep login site/docs, read the error, and fix it.

The labs here are shell labs. In each one you edit answer.sh, a plain text file of commands that sh answer.sh runs from top to bottom. Whatever it prints is what gets graded, and each lab states the exact output it wants. The sample files live in testdata/: read them with the commands above, then run the lab's test to see how close you are. If a lab fails, go test shows what your script printed and what it wanted: compare the two line by line.