Lesson 3 of 7 · 0 of 2 exercises solved

stdin, stdout & Pipes

Every program starts with three data streams: input, output and . Learn how and connect those streams to files or to other programs.

Bash

# Three standard streams, by number:
#   0 stdin  — bytes in (default: your keyboard)
#   1 stdout — results out (default: your screen)
#   2 stderr — errors out (default: also your screen)

ls > files.txt        # stdout -> file (overwrite)
ls >> files.txt       # stdout -> file (append)
ls /nope 2> err.txt   # stderr -> file
ls /nope > all.txt 2>&1   # both -> one file
sort < names.txt      # file -> stdin

# stderr is separate ON PURPOSE — errors stay visible
# even when you redirect the results:
myprogram > results.txt     # errors still hit your screen

The pipe | connects one program's stdout to the next one's stdin, no temp files involved. Both programs run simultaneously; the shuttles between them and pauses the fast one when the slow one falls behind. Small tools, each doing one job, connected in a chain.

Bash

# Build analysis pipelines out of small tools:
history | grep git | wc -l       # how many git commands you've run

# Classic shape: extract | filter | transform | count
cat access.log \
  | grep " 500 "        \
  | awk '{print $1}'    \
  | sort | uniq -c      \
  | sort -rn | head     # top IPs causing server errors

# tee splits a stream: to a file AND onward
long_build 2>&1 | tee build.log | grep -i error

Common question

Are small single-purpose programs worse than one program that does everything?

Every process starts with three streams: input, output and errors. Because programs read and write these instead of naming files, the can connect them freely: a file into a program, one program into another, output to a file while errors go somewhere else. That one convention is what lets small tools combine into powerful pipelines.

Learn the operators (|, >, >>, 2>, < and 2>&1) and the handful of filters that make up most pipelines: grep, sort, uniq, cut, head, tail and wc. Then build pipelines one stage at a time, checking the output at each step before adding the next.

Source: Doug McIlroy, who invented the Unix pipe: "Write programs that do one thing and do it well. Write programs to work together."

Common mix-ups

Each of these sounds reasonable. Think about why it is wrong, then open the explanation.

One tool that sorts, filters and counts is more convenient than three joined by .
write to a temporary file between the stages.
Redirecting output with > also captures error messages.

Practice

Shell exercise

Redirection and a pipe

Redirect and pipe. Save the output of ls -l to files.txt, append the line "end of listing" to it, then use a pipe to count how many lines in data/songs.txt mention Track 3.

3 checks

Practice shell — type help to see what is available.

Shell exercise

A pipeline that transforms data

Build a small pipeline: print the names from data/team.csv (the first field, skipping the header line), sorted alphabetically, and save the result to names.txt.

2 checks

Practice shell — type help to see what is available.

Finished reading?2 exercises still open. You can come back to them later.

Tip: use the ← and → keys to move between lessons.