Terminal Text Utility Tools: A Comprehensive Guide

May 28, 2026 · by 0xKiireguidefilesiostoragelogswaldurabilitycdata structuresalgorithmsDSA

A deep dive into the Unix/Linux text toolchain — grep, sed, awk, cat, echo, cut, tr, sort, uniq, wc, tee, less, head, tail, printf, xargs, find, diff, patch, strings, hexdump, and more.

Philosophy: The Unix Text Model
echo — Printing to Standard Output
printf — Formatted Output
cat — Concatenate and Print Files
tac — Reverse cat
less and more — Paging Through Text
head — First N Lines
tail — Last N Lines
wc — Word, Line, and Byte Count
grep — Pattern Matching and Searching
egrep and fgrep — grep Variants
sed — Stream Editor
awk — Pattern-Action Text Processor
cut — Extract Fields and Characters
paste — Merge Lines Side by Side
tr — Translate or Delete Characters
sort — Sort Lines of Text
uniq — Filter Duplicate Lines
tee — Split Output to File and Stdout
find — Search Filesystem and Act on Files
xargs — Build and Execute Commands from Input
diff — Compare Files Line by Line
patch — Apply Diff Patches
strings — Extract Printable Strings from Binaries
hexdump and xxd — Inspect Raw Bytes
od — Octal Dump
expand and unexpand — Tabs and Spaces
fold and fmt — Rewrap Lines
column — Align Output in Columns
nl — Number Lines
rev — Reverse Lines Character by Character
join — Join Lines of Two Files on a Common Field
comm — Compare Two Sorted Files Line by Line
look — Find Lines Starting with a String
Combining Tools: Real-World Pipelines
Regex Reference for Text Tools
Performance Notes and Gotchas

1. Philosophy: The Unix Text Model

Unix was designed around a deceptively simple idea: everything is a file, and all communication flows through text streams. Programs read from standard input (stdin, file descriptor 0), write to standard output (stdout, fd 1), and report errors to standard error (stderr, fd 2). This uniformity makes programs composable through the pipe operator |.

stdin ──► [program A] ──► stdout
                               │
                               ▼  pipe
                          [program B] ──► stdout

This design philosophy — called the Unix Philosophy — has three core rules (from Doug McIlroy):

Write programs that do one thing and do it well.
Write programs to work together.
Write programs to handle text streams, because that is a universal interface.

Every tool in this guide was designed with these rules in mind. A command like grep only matches lines — it does not sort them, count them, or reformat them. To count matching lines, you pipe grep into wc -l. To sort them, you pipe into sort. This is not a limitation; it is the architecture's greatest strength.

Standard Streams

# Redirect stdout to a file
command > output.txt

# Redirect stderr to a file
command 2> errors.txt

# Redirect both stdout and stderr
command > output.txt 2>&1
# Or shorthand (Bash 4+):
command &> output.txt

# Discard output entirely
command > /dev/null

# Discard both stdout and stderr
command > /dev/null 2>&1

# Read stdin from a file
command < input.txt

# Here-string: feed a string as stdin
command <<< "some string"

# Here-document: feed multiple lines as stdin
command << EOF
line one
line two
EOF

The Pipe

The pipe | connects the stdout of one process directly to the stdin of the next. Both processes run concurrently — the kernel manages a kernel buffer between them.

# Classic pipe chain
cat /var/log/syslog | grep "ERROR" | sort | uniq -c | sort -rn | head -20

Understanding this model deeply makes every tool in this guide 10x more useful.

2. echo — Printing to Standard Output

echo is the simplest output tool. It prints its arguments to stdout followed by a newline.

Basic Usage

echo "Hello, World!"
# Hello, World!

echo Hello World
# Hello World   (shell splits on whitespace first)

echo ""          # prints empty line
echo            # also prints empty line

Flags

Flag	Meaning
`-n`	Do not print trailing newline
`-e`	Enable interpretation of backslash escapes
`-E`	Disable escape interpretation (default)

echo -n "no newline at end"
# prompt appears on same line

echo -e "line one\nline two\nline three"
# line one
# line two
# line three

echo -e "tab\there"
# tab     here

echo -e "\033[1;32mGreen Bold Text\033[0m"
# (renders in green bold on a terminal)

Escape Sequences with -e

Sequence	Effect
`\n`	Newline
`\t`	Horizontal tab
`\r`	Carriage return
`\\`	Literal backslash
`\a`	Alert/bell
`\b`	Backspace
`\0NNN`	Octal character
`\xHH`	Hex character

ANSI Color Codes

# Format: \033[STYLE;COLORm
# Styles: 0=reset, 1=bold, 2=dim, 4=underline, 7=reverse
# Colors (fg): 30=black,31=red,32=green,33=yellow,34=blue,35=magenta,36=cyan,37=white
# Colors (bg): 40-47 (same order)

echo -e "\033[1;31mERROR:\033[0m Something went wrong"
echo -e "\033[1;32mSUCCESS:\033[0m Operation complete"
echo -e "\033[1;33mWARN:\033[0m Check your input"
echo -e "\033[4;36mUnderlined Cyan\033[0m"

# 256-color mode
echo -e "\033[38;5;208mOrange text\033[0m"

# True color (24-bit)
echo -e "\033[38;2;0;255;100mRGB Green\033[0m"

Practical Applications

# Write a file with echo
echo "Hello" > file.txt

# Append to a file
echo "Another line" >> file.txt

# Use in scripts for status output
echo "[INFO] Starting deployment..."
echo "[WARN] Low disk space"
echo "[ERROR] Connection refused" >&2  # write to stderr

# Print variable values
NAME="kiire"
echo "Hello, $NAME"

# Print without variable expansion (single quotes)
echo 'Cost is $5.00'
# Cost is $5.00

# Brace expansion
echo file{1,2,3}.txt
# file1.txt file2.txt file3.txt

# Generate a sequence of lines
echo -e "$(seq 1 5 | tr '\n' ',')"
# 1,2,3,4,5,

Portability Note

echo behavior differs between shells (bash, sh, dash, zsh). For portable, well-defined output formatting, prefer printf over echo. The -e flag is not POSIX-standard and may not work in /bin/sh scripts.

3. printf — Formatted Output

printf is the proper tool for formatted output. It follows C-style format strings and is consistent across shells and systems.

Basic Syntax

printf FORMAT [ARGUMENTS...]

Format Specifiers

Specifier	Meaning
`%s`	String
`%d`	Integer (decimal)
`%f`	Floating-point
`%e`	Scientific notation
`%g`	Shorter of `%f` and `%e`
`%o`	Octal
`%x`	Hexadecimal (lowercase)
`%X`	Hexadecimal (uppercase)
`%c`	Character
`%%`	Literal percent sign

Examples

# Basic string formatting
printf "Hello, %s!\n" "world"
# Hello, world!

# Multiple arguments: format string repeats
printf "%s\n" apple banana cherry
# apple
# banana
# cherry

# Integer formatting
printf "Count: %d\n" 42
# Count: 42

# Floating point
printf "Pi: %.4f\n" 3.14159265
# Pi: 3.1416

# Width and alignment
printf "%-20s %5d\n" "Item" 100
# Item                   100

printf "%20s %5d\n" "Item" 100
#                 Item   100

# Zero padding
printf "%05d\n" 42
# 00042

# Hex output
printf "%x\n" 255
# ff
printf "%X\n" 255
# FF
printf "0x%08X\n" 255
# 0x000000FF

# Multiple columns in a table
printf "%-15s %-10s %8s\n" "Name" "Type" "Size"
printf "%-15s %-10s %8s\n" "----------" "----" "----"
printf "%-15s %-10s %8d\n" "main.c" "source" 4096
printf "%-15s %-10s %8d\n" "Makefile" "build" 512

# Output:
# Name            Type         Size
# ----------      ----         ----
# main.c          source       4096
# Makefile        build         512

# Write output to a file
printf "line1\nline2\nline3\n" > output.txt

# Generate hex escape
printf "\x41\x42\x43\n"
# ABC

# Null-terminate strings (useful with xargs -0)
printf "%s\0" "file one" "file two" "file three"

printf in Scripts

#!/bin/bash
# Logging function using printf
log() {
    local level="$1"
    local msg="$2"
    printf "[%s] %-8s %s\n" "$(date +%H:%M:%S)" "$level" "$msg"
}

log "INFO"  "Starting process"
log "WARN"  "Disk at 80%"
log "ERROR" "Cannot open file"

# [14:32:01] INFO     Starting process
# [14:32:01] WARN     Disk at 80%
# [14:32:01] ERROR    Cannot open file

4. cat — Concatenate and Print Files

cat stands for "concatenate." It reads files and writes them to stdout. When given no arguments (or -), it reads from stdin.

Basic Usage

# Print file contents
cat file.txt

# Print multiple files (concatenated)
cat file1.txt file2.txt file3.txt

# Concatenate files into a new file
cat file1.txt file2.txt > combined.txt

# Read from stdin
echo "hello" | cat

# Mix files and stdin
cat file1.txt - file2.txt   # stdin is inserted between file1 and file2

# Create a file interactively (Ctrl+D to end)
cat > newfile.txt

Flags

Flag	Meaning
`-n`	Number all output lines
`-b`	Number non-blank lines only
`-A`	Show all: equivalent to `-vET`
`-v`	Show non-printing characters (except tab and newline)
`-E`	Display `$` at end of each line
`-T`	Display TAB as `^I`
`-s`	Squeeze multiple blank lines into one

# Number lines
cat -n script.sh

# Show line endings (useful for debugging Windows \r\n vs Unix \n)
cat -A file.txt
# Lines ending in \r\n show as: content^M$
# Lines ending in \n show as:   content$

# Squeeze blank lines
cat -s file_with_gaps.txt

# Show tabs and EOL markers together
cat -A script.py | head -20

# Display binary content (printable representation)
cat -v binary_file

Practical Applications

# View a config file
cat /etc/hosts
cat /proc/cpuinfo | head -50

# Combine log files
cat /var/log/app.log /var/log/app.log.1 > combined_logs.txt

# Append one file to another
cat appendix.txt >> main_document.txt

# Create a here-document file
cat > config.json << 'EOF'
{
    "host": "localhost",
    "port": 8080,
    "debug": true
}
EOF

# Pipeline origin: feed a file into a pipeline
cat access.log | grep "404" | awk '{print $1}' | sort | uniq -c

# Copy a file (less efficient than cp but demonstrates streams)
cat source.bin > destination.bin

# Print file with line numbers and squeeze blanks
cat -ns README.md

The "Useless Use of cat" Anti-pattern

Many pipelines begin with cat file | grep .... This is called a "useless use of cat" because grep can read files directly:

# Useless use of cat
cat access.log | grep "ERROR"

# Better: grep reads the file itself
grep "ERROR" access.log

# But cat is valid when you need to combine files first
cat access.log access.log.1 | grep "ERROR"

# Or when you want to force stdin behavior explicitly
cat < access.log | grep "ERROR"

5. tac — Reverse cat

tac prints files in reverse line order (last line first). The name is "cat" spelled backwards.

# Reverse line order
tac file.txt

# Reverse a log to read newest first
tac /var/log/syslog | less

# Reverse a numbered sequence
seq 1 10 | tac
# 10
# 9
# 8
# ...
# 1

# Use a different line separator
tac --separator=: file.txt

# Reverse sections of a file
# (useful for reversing paragraph order)
tac --before --separator=$'\n\n' document.txt

6. less and more — Paging Through Text

more is the original Unix pager. It shows one screen of text at a time.

more file.txt

Key commands inside more:

Space — next page
Enter — next line
q — quit
/pattern — search forward
b — page backward (in some versions)

less

less is a superset of more with many more features. The name comes from the joke "less is more." It is the preferred pager for interactive use.

less file.txt
less /var/log/syslog
man grep | less

Key Bindings Inside less

Key	Action
`Space` / `f`	Forward one page
`b`	Backward one page
`g`	Go to beginning
`G`	Go to end
`j` / `↓`	Next line
`k` / `↑`	Previous line
`/pattern`	Search forward
`?pattern`	Search backward
`n`	Next search match
`N`	Previous search match
`q`	Quit
`h`	Help
`-N`	Toggle line numbers
`=`	Show file info (position, line number)
`F`	Follow mode (like `tail -f`)
`&pattern`	Filter: show only matching lines

Useful less Flags

# Show line numbers
less -N file.txt

# Don't clear screen on exit
less -X file.txt

# Follow mode (like tail -f)
less +F /var/log/syslog

# Open at a specific line
less +100 file.txt

# Open at first match of a pattern
less +/ERROR logfile.txt

# Quit if entire file fits on one screen
less -F file.txt

# Strip color codes from ANSI sequences
less -R file.txt     # raw: pass ANSI codes through
less -r file.txt     # raw control: pass all control chars through

# Combine: colored output with paging
ls --color=always | less -R
grep --color=always "pattern" file.txt | less -R

LESSOPEN: Preprocessor

less supports a preprocessor pipeline via the LESSOPEN environment variable. This allows less to transparently open compressed files:

# In .bashrc/.zshrc
export LESSOPEN="| /usr/bin/lesspipe %s"

# Now you can directly page compressed files
less file.gz
less archive.tar.bz2

7. head — First N Lines

head prints the first N lines (default 10) of a file or stdin.

Basic Usage

# Default: first 10 lines
head file.txt

# First 20 lines
head -n 20 file.txt

# First 5 lines (shorthand)
head -5 file.txt

# First N bytes
head -c 100 file.txt     # first 100 bytes
head -c 1K file.txt      # first 1024 bytes
head -c 1M file.txt      # first 1,048,576 bytes

Excluding Trailing Lines

The -n flag also accepts negative values, meaning "all lines except the last N":

# All lines except the last 5
head -n -5 file.txt

# All lines except the last line (useful for removing trailing commas in JSON arrays)
head -n -1 array.json

Multiple Files

head -n 5 file1.txt file2.txt file3.txt
# ==> file1.txt <==
# line1
# line2
# ...
# ==> file2.txt <==
# ...

Practical Applications

# Check the format/header of a CSV
head -n 3 data.csv

# Preview a log file
head -n 50 /var/log/syslog

# Get first line (e.g. shebang check)
head -1 script.sh

# Extract CSV header
head -1 data.csv > header.csv

# Peek at a large binary's magic bytes (in hex)
head -c 8 image.png | xxd

# Generate top-N list from sorted data
sort -rn scores.txt | head -10

# Remove last line from file
head -n -1 file.txt > file_trimmed.txt

8. tail — Last N Lines

tail prints the last N lines (default 10) of a file or stdin.

Basic Usage

# Default: last 10 lines
tail file.txt

# Last 20 lines
tail -n 20 file.txt

# Last 5 lines
tail -5 file.txt

# Last N bytes
tail -c 100 file.txt

Starting from a Line Number

# All lines from line 10 onward (skip first 9)
tail -n +10 file.txt

# Skip the header line of a CSV
tail -n +2 data.csv

# Skip the first N bytes
tail -c +1024 binary.bin

Follow Mode

The -f (follow) flag makes tail watch the file and print new lines as they are appended. This is essential for monitoring live logs.

# Follow a log in real time
tail -f /var/log/syslog

# Follow multiple files simultaneously
tail -f /var/log/nginx/access.log /var/log/nginx/error.log

# Follow and retry if file doesn't exist yet
tail -F /var/log/app.log

# Follow with grep — watch for errors in real time
tail -f /var/log/app.log | grep --line-buffered "ERROR"

# Follow with context highlighting
tail -f /var/log/app.log | grep --line-buffered --color "ERROR\|WARN\|"

Practical Applications

# View end of a large file
tail -n 100 huge_log.txt

# Extract lines 50-60 from a file (combine head and tail)
head -n 60 file.txt | tail -n 11

# Extract lines N through M (sed is better for this, see §12)
sed -n '50,60p' file.txt

# Remove first line (header) from CSV for processing
tail -n +2 data.csv | awk -F, '{sum += $3} END {print sum}'

# Monitor a growing log and filter in real time
tail -f /var/log/auth.log | grep --line-buffered "Failed password"

# Get the last inserted line in a log
tail -1 access.log

9. wc — Word, Line, and Byte Count

wc (word count) counts lines, words, characters, and bytes in files.

Flags

Flag	Count
`-l`	Lines
`-w`	Words
`-c`	Bytes
`-m`	Characters (multibyte-aware)
`-L`	Length of longest line

# Full output: lines words bytes filename
wc file.txt
# 42  358 2048 file.txt

# Count lines only
wc -l file.txt

# Count words only
wc -w file.txt

# Count bytes
wc -c file.txt

# Count characters (differs from bytes in UTF-8 text)
wc -m file.txt

# Longest line length
wc -L file.txt

# Multiple files
wc -l *.c
# Shows count per file plus total

# stdin input
echo "hello world foo" | wc -w
# 3

cat /var/log/syslog | wc -l

Practical Applications

# Count lines matching a grep pattern
grep -c "ERROR" logfile.txt        # grep -c is faster for this
grep "ERROR" logfile.txt | wc -l   # equivalent

# Count unique users in an access log
awk '{print $1}' access.log | sort -u | wc -l

# Check line count before processing
LINES=$(wc -l < data.csv)
echo "Processing $LINES records"

# Find longest function name in C code
grep -oP 'int \K\w+(?=\()' code.c | sort | awk '{ print length, $0 }' | sort -rn | head

# Word count of your own writing
wc -w essay.txt

# Count total lines of code in a project
find . -name "*.c" -o -name "*.h" | xargs wc -l | tail -1

# Count files in a directory
ls | wc -l

10. grep — Pattern Matching and Searching

grep is one of the most powerful and widely used Unix tools. It searches input for lines matching a pattern and prints those lines. The name stands for global regular expression print, from the ed command g/re/p.

Basic Syntax

grep [OPTIONS] PATTERN [FILE...]

Basic Examples

# Search for a string in a file
grep "error" logfile.txt

# Case-insensitive search
grep -i "error" logfile.txt

# Search multiple files
grep "TODO" *.c *.h

# Read from stdin
cat file.txt | grep "pattern"
dmesg | grep "usb"

Essential Flags

Flag	Meaning
`-i`	Case-insensitive matching
`-v`	Invert match (print non-matching lines)
`-n`	Print line numbers
`-c`	Count matching lines
`-l`	Print only filenames with matches
`-L`	Print only filenames WITHOUT matches
`-r`	Recursive search
`-R`	Recursive, follow symlinks
`-w`	Match whole words only
`-x`	Match whole lines only
`-m N`	Stop after N matches
`-A N`	Print N lines After each match
`-B N`	Print N lines Before each match
`-C N`	Print N lines of Context (before+after)
`-o`	Print only the matching part, not full line
`-s`	Suppress error messages
`-q`	Quiet: exit 0 if match found, 1 if not
`-e`	Specify pattern (allows multiple `-e`)
`-f`	Read patterns from file
`-F`	Treat pattern as fixed string (no regex)
`-E`	Extended regex (same as `egrep`)
`-P`	Perl-compatible regex (PCRE)
`--color`	Highlight matches

Searching Files and Directories

# Search recursively from current directory
grep -r "TODO" .

# Recursive, case-insensitive, with line numbers
grep -rin "error" /var/log/

# Only show filenames (not matching lines)
grep -rl "deprecated" src/

# Search only specific file types
grep -r "malloc" --include="*.c"
grep -r "import" --include="*.py" --include="*.js"

# Exclude directories
grep -r "pattern" . --exclude-dir=".git" --exclude-dir="node_modules"

# Exclude file types
grep -r "pattern" . --exclude="*.min.js"

Context Lines

# 3 lines before and after each match
grep -C 3 "segfault" /var/log/kern.log

# 5 lines before
grep -B 5 "ERROR" app.log

# 2 lines after
grep -A 2 "Connection refused" app.log

Counting and Testing

# Count occurrences
grep -c "404" access.log

# Test if a pattern exists (for scripts)
if grep -q "FAILED" build.log; then
    echo "Build failed!"
fi

# Exit status: 0=match found, 1=no match, 2=error
grep "pattern" file.txt
echo $?

# Only print matching part of each line
grep -o "[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}" access.log

Multiple Patterns

# Match either pattern (logical OR)
grep -e "error" -e "warning" -e "critical" logfile.txt

# OR using regex alternation
grep -E "error|warning|critical" logfile.txt

# Match pattern from a file (one pattern per line)
cat patterns.txt
# error
# warning
# timeout
grep -f patterns.txt logfile.txt

# Logical AND: chain two greps
grep "user" logfile.txt | grep "failed"

# Match a pattern but exclude another
grep "connection" logfile.txt | grep -v "closed"

Whole Word and Whole Line Matching

# Whole word: matches "log" but not "login" or "catalog"
grep -w "log" file.txt

# Whole line: only lines that are exactly "success"
grep -x "success" results.txt

# Match blank lines
grep "^$" file.txt

# Match non-blank lines
grep -v "^$" file.txt

# Match lines with only whitespace
grep "^\s*$" file.txt
grep -P "^\s*$" file.txt

Regex Patterns in grep

# Any character: .
grep "b.g" words.txt        # bag, big, bog, bug...

# Start of line: ^
grep "^Error" logfile.txt   # lines starting with "Error"

# End of line: $
grep "\.c$" filelist.txt    # lines ending in .c

# Zero or more of preceding: *
grep "fo*" words.txt        # f, fo, foo, fooo...

# Character class
grep "[aeiou]" words.txt    # lines with a vowel
grep "[0-9]" data.txt       # lines with a digit
grep "[^0-9]" data.txt      # lines with a non-digit

# Escape special character
grep "\." file.txt          # literal dot (not any char)
grep "3\.14" constants.c

# Range in character class
grep "[a-z]" file.txt
grep "[A-Za-z0-9_]" identifiers.txt

# Repetition (BRE requires backslash)
grep "x\{3\}" file.txt          # exactly 3 x's: xxx
grep "x\{2,4\}" file.txt        # 2 to 4 x's

# Extended regex with -E (no backslash for quantifiers)
grep -E "x{3}" file.txt
grep -E "^(ERROR|WARN|INFO):" logfile.txt
grep -E "[0-9]{4}-[0-9]{2}-[0-9]{2}" dates.txt   # date format YYYY-MM-DD

# PCRE with -P (Perl-compatible, very powerful)
grep -P "(?<=user=)\w+" access.log     # lookbehind
grep -P "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" access.log  # IP addresses
grep -P "^\d+\s+\w+" data.txt

Named Capture Groups and -oP

# Extract only the matched portion using -o
grep -oE "[0-9]+" numbers.txt

# Extract specific field using PCRE lookbehind
grep -oP '(?<=Host: )\S+' http_headers.txt

# Extract timestamps from logs
grep -oP '\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}' syslog

# Extract email addresses
grep -oP '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' emails.txt

# Extract URLs
grep -oP 'https?://[^\s"]+' html_file.txt

grep in Scripts

#!/bin/bash

# Check if service is running
if systemctl status nginx | grep -q "active (running)"; then
    echo "nginx is up"
else
    echo "nginx is DOWN" >&2
    exit 1
fi

# Count errors in last hour's log
ERROR_COUNT=$(grep -c "ERROR" /var/log/app.log)
if [ "$ERROR_COUNT" -gt 100 ]; then
    echo "Alert: $ERROR_COUNT errors found"
fi

# Find all TODO comments and list their files
grep -rn "TODO\|FIXME\|HACK\|XXX" src/ --include="*.c" | sort

11. egrep and fgrep — grep Variants

These are traditionally separate binaries but are now usually implemented as grep -E and grep -F.

egrep (Extended grep)

egrep uses Extended Regular Expressions (ERE), avoiding the need for backslash escaping of quantifiers and groups:

egrep "error|warning" logfile.txt     # alternation
egrep "(GET|POST|PUT|DELETE)" access.log
egrep "[0-9]{4}" file.txt            # no backslash needed
egrep "^(start|begin)" file.txt

fgrep (Fixed-string grep)

fgrep treats the pattern as a literal fixed string — no regex interpretation. This is much faster for large files when you don't need regex, and essential when searching for strings that contain regex metacharacters:

# Search for literal dots, brackets, parentheses without escaping
fgrep "array[0].value" code.c
fgrep "printf("hello")" code.c
fgrep "1.2.3.4" access.log         # literal IP (no regex)

# Performance: fgrep is faster for large files
fgrep "exact_string" huge_file.txt

12. sed — Stream Editor

sed is a non-interactive stream editor. It reads input line by line, applies a script of editing commands, and writes the result to stdout. It is one of the most powerful text transformation tools available.

Basic Syntax

sed [OPTIONS] 'SCRIPT' [FILE...]
sed [OPTIONS] -e 'CMD1' -e 'CMD2' [FILE...]
sed [OPTIONS] -f script.sed [FILE...]

The Substitution Command: s

The most-used sed command is s (substitute):

s/PATTERN/REPLACEMENT/FLAGS

# Replace first occurrence on each line
sed 's/foo/bar/' file.txt

# Replace ALL occurrences (global flag)
sed 's/foo/bar/g' file.txt

# Case-insensitive replacement
sed 's/foo/bar/gi' file.txt

# Replace only on line 5
sed '5s/foo/bar/' file.txt

# Replace on lines 1-10
sed '1,10s/foo/bar/g' file.txt

# Replace the 2nd occurrence on each line
sed 's/foo/bar/2' file.txt

# Replace 2nd and all subsequent occurrences
sed 's/foo/bar/2g' file.txt

# Print lines where replacement was made (-n + /p flag)
sed -n 's/foo/bar/p' file.txt

In-Place Editing

# Edit file in place (GNU sed)
sed -i 's/old/new/g' file.txt

# Edit in place with backup (.bak extension)
sed -i.bak 's/old/new/g' file.txt

# On macOS, -i requires an argument (can be empty string)
sed -i '' 's/old/new/g' file.txt

Delimiters

You can use any delimiter character instead of /. This avoids escaping slashes in paths:

# Replace a path (using | as delimiter)
sed 's|/usr/local/bin|/opt/bin|g' config.txt

# Replace URL (using # as delimiter)
sed 's#http://old.example.com#https://new.example.com#g' bookmarks.txt

Addresses

sed commands apply to all lines unless you specify an address:

# Line number
sed '5d' file.txt              # delete line 5

# Range of lines
sed '5,10d' file.txt           # delete lines 5 through 10

# First to last: $
sed '5,$d' file.txt            # delete from line 5 to end

# Regex address: lines matching pattern
sed '/error/d' file.txt        # delete lines containing "error"

# Regex range: from match to match
sed '/START/,/END/d' file.txt  # delete from START line to END line

# Negation with !
sed '/error/!d' file.txt       # delete lines NOT containing error
                                # (keep only error lines)

# Every Nth line (step)
sed -n '0~2p' file.txt         # print every 2nd line (0, 2, 4...)
sed -n '1~2p' file.txt         # print odd lines (1, 3, 5...)

Delete Command: d

# Delete blank lines
sed '/^$/d' file.txt

# Delete blank lines and lines with only whitespace
sed '/^\s*$/d' file.txt

# Delete comment lines (starting with #)
sed '/^#/d' config.conf

# Delete comments and blank lines
sed '/^#/d; /^$/d' config.conf

# Delete first line (header)
sed '1d' data.csv

# Delete last line
sed '$d' file.txt

# Delete from line 5 to end
sed '5,$d' file.txt

# Delete lines matching a pattern (with context)
sed '/START_BLOCK/,/END_BLOCK/d' file.txt

Print Command: p

# Print lines matching pattern (-n suppresses default output)
sed -n '/error/p' logfile.txt

# Equivalent to grep
sed -n '/pattern/p' file.txt

# Print specific line
sed -n '5p' file.txt

# Print line range
sed -n '5,10p' file.txt

# Print from pattern to end
sed -n '/START/,$p' file.txt

Append, Insert, Change Commands

# Append line after matching line
sed '/pattern/a\This line is appended after' file.txt

# Insert line before matching line
sed '/pattern/i\This line is inserted before' file.txt

# Change line to new text
sed '/pattern/c\This replaces the matching line' file.txt

# Add a blank line after every line
sed 'G' file.txt

# Add a blank line before every line
sed '' file.txt   # (use awk 'NR>1{print ""} {print}' instead)

Multiple Commands

# Semicolon separator
sed 's/foo/bar/g; s/baz/qux/g' file.txt

# Multiple -e flags
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt

# Script file
cat fixes.sed
# s/foo/bar/g
# s/baz/qux/g
# /DEBUG/d
sed -f fixes.sed file.txt

The Hold Space

sed has two buffers: the pattern space (current line) and the hold space (auxiliary buffer). Commands for working with them:

Command	Meaning
`h`	Copy pattern space to hold space
`H`	Append pattern space to hold space
`g`	Copy hold space to pattern space
`G`	Append hold space to pattern space
`x`	Exchange pattern and hold spaces

# Reverse lines in a file (tac equivalent)
sed -n '1!G; h; $p' file.txt

# Double-space a file (add blank line after each line)
sed 'G' file.txt

# Remove duplicate consecutive lines
sed '$!N; /^\(.*\)\n\1$/!P; D' file.txt

Practical Applications

# Remove trailing whitespace
sed 's/[[:space:]]*$//' file.txt

# Remove leading whitespace
sed 's/^[[:space:]]*//' file.txt

# Remove both leading and trailing whitespace (trim)
sed 's/^[[:space:]]*//; s/[[:space:]]*$//' file.txt

# Add line numbers to a file
sed = file.txt | sed 'N; s/\n/\t/'

# Replace tabs with 4 spaces
sed 's/\t/    /g' file.txt

# Convert Windows line endings to Unix
sed 's/\r//' file.txt
sed 's/\r$//' file.txt   # safer: only trailing \r

# Extract lines between two patterns (inclusive)
sed -n '/BEGIN/,/END/p' file.txt

# Extract lines between patterns (exclusive)
sed -n '/BEGIN/,/END/{/BEGIN/d; /END/d; p}' file.txt

# Comment out lines matching a pattern
sed '/pattern/s/^/# /' config.conf

# Uncomment lines
sed 's/^# //' config.conf

# Prepend text to each line
sed 's/^/PREFIX: /' file.txt

# Append text to each line
sed 's/$/ :SUFFIX/' file.txt

# Quote each line for shell use
sed "s/'/'\\\\''/g; s/.*/'&'/" file.txt

# Extract CSV column 3 (for simple non-quoted CSV)
sed 's/[^,]*,[^,]*,\([^,]*\).*/\1/' data.csv

# Batch rename: change file extension
ls *.txt | sed 's/\.txt$/.md/' | xargs -I{} echo "rename: {}"

# Mass replace across all source files
find . -name "*.c" -exec sed -i 's/old_func/new_func/g' {} \;

13. awk — Pattern-Action Text Processor

awk is a complete programming language for text processing. It reads input line by line (or record by record), splits each line into fields, and applies pattern-action rules. The name comes from its creators: Aho, Weinberger, and Kernighan.

Basic Model

awk 'PATTERN { ACTION }' file.txt

For each input record, if the pattern matches, the action is executed. Both pattern and action are optional. Without a pattern, the action runs on every line. Without an action, the default is { print }.

Built-in Variables

Variable	Meaning
`$0`	Entire current record (line)
`$1`-`$N`	Field 1 through N
`NF`	Number of fields in current record
`NR`	Current record number (line number)
`FNR`	Record number within current file
`FS`	Input field separator (default: whitespace)
`OFS`	Output field separator (default: space)
`RS`	Record separator (default: newline)
`ORS`	Output record separator (default: newline)
`FILENAME`	Name of current input file

Basic Examples

# Print entire file
awk '{ print }' file.txt
awk '1' file.txt              # shorthand (1 is always true)

# Print first field
awk '{ print $1 }' file.txt

# Print first and third fields
awk '{ print $1, $3 }' file.txt    # separated by OFS (space)

# Print last field
awk '{ print $NF }' file.txt

# Print line number and line
awk '{ print NR, $0 }' file.txt

# Print number of fields per line
awk '{ print NF }' file.txt

# Print total number of lines
awk 'END { print NR }' file.txt

Field Separator

# CSV: comma separated
awk -F, '{ print $2 }' data.csv

# Tab separated
awk -F'\t' '{ print $1, $3 }' data.tsv

# Colon separated (like /etc/passwd)
awk -F: '{ print $1, $6 }' /etc/passwd    # username and home dir

# Multiple separators using regex
awk -F'[,;|]' '{ print $2 }' mixed.txt

# Set output field separator
awk -F, 'BEGIN{OFS="|"} { print $1, $2, $3 }' data.csv

Patterns

# Print lines matching regex
awk '/error/' logfile.txt        # equivalent to grep "error"

# Print lines NOT matching
awk '!/error/' logfile.txt

# Conditional: field comparison
awk '$3 > 100' data.txt           # lines where field 3 > 100
awk '$1 == "root"' /etc/passwd
awk 'NF > 5' file.txt             # lines with more than 5 fields
awk 'NR % 2 == 0' file.txt        # even-numbered lines
awk 'NR >= 10 && NR <= 20' file.txt  # lines 10-20

# Range pattern
awk '/START/,/END/' file.txt      # print from START to END (inclusive)

BEGIN and END Blocks

# BEGIN: runs before first line
# END: runs after last line

awk 'BEGIN { print "=== Report ===" }
     { print }
     END { print "=== Done ===" }' file.txt

# Count lines
awk 'END { print "Total lines:", NR }' file.txt

# Sum a column
awk -F, '{ sum += $3 } END { print "Total:", sum }' data.csv

# Average
awk '{ sum += $1; count++ } END { print sum/count }' numbers.txt

# Min and max
awk 'NR==1 { min=$1; max=$1 }
     { if ($1 < min) min=$1; if ($1 > max) max=$1 }
     END { print "Min:", min, "Max:", max }' numbers.txt

Arithmetic and String Operations

# Arithmetic operators: + - * / % ^
awk '{ print $1 * $2 }' products.txt
awk '{ print $1 / $2 }' data.txt
awk '{ print $1 ^ 2 }' numbers.txt       # squared

# String concatenation (just juxtapose)
awk '{ print $1 " " $2 }' names.txt

# String functions
awk '{ print length($0) }' file.txt      # length of line
awk '{ print toupper($1) }' file.txt     # uppercase
awk '{ print tolower($1) }' file.txt     # lowercase
awk '{ print substr($1, 2, 3) }' file.txt  # substring(str, start, len)
awk '{ gsub(/foo/, "bar"); print }' file.txt  # global substitution
awk '{ sub(/foo/, "bar"); print }' file.txt   # first substitution only

# Index of substring
awk '{ print index($0, "error") }' file.txt   # 0 if not found

# Split string into array
awk '{ n = split($0, arr, ":"); for (i=1; i<=n; i++) print arr[i] }' file.txt

# Printf formatting in awk
awk '{ printf "%-20s %8.2f\n", $1, $2 }' data.txt

Arrays

# Count occurrences of each word
awk '{ for (i=1; i<=NF; i++) count[$i]++ }
     END { for (word in count) print count[word], word }' file.txt

# Count unique values in field 1
awk '{ seen[$1]++ } END { print length(seen) " unique values" }' data.txt

# Word frequency, sorted
awk '{ for (i=1; i<=NF; i++) count[$i]++ }
     END { for (w in count) print count[w], w }' file.txt | sort -rn | head -20

# Invert a key-value file
awk '{ print $2, $1 }' pairs.txt

# Group by field 2 and collect field 1
awk '{ groups[$2] = groups[$2] " " $1 }
     END { for (g in groups) print g ":" groups[g] }' data.txt

Control Flow

# if-else
awk '{ if ($3 > 90) print $1, "PASS"; else print $1, "FAIL" }' scores.txt

# for loop
awk '{ for (i=1; i<=NF; i++) printf "%s ", $i; printf "\n" }' file.txt

# while loop
awk '{ i=1; while (i<=NF) { print $i; i++ } }' file.txt

# next: skip to next record
awk '/^#/ { next } { print }' config.txt   # skip comment lines

# getline: read next line
awk '/HEADER/ { getline; print "After header:", $0 }' file.txt

Practical awk Applications

# Extract specific columns from a log
# Apache log: IP - - [date] "METHOD URL PROTO" STATUS SIZE
awk '{ print $1, $9 }' access.log         # IP and status code

# Count HTTP status codes
awk '{ status[$9]++ } END { for (s in status) print s, status[s] }' access.log | sort -k2 -rn

# Calculate average response time
awk -F'"' '{ print $NF }' access.log | awk '{ sum+=$1; n++ } END { print sum/n }'

# /etc/passwd analysis
awk -F: '$3 >= 1000 { print $1 }' /etc/passwd    # non-system users
awk -F: '{ shells[$7]++ } END { for (s in shells) print shells[s], s }' /etc/passwd

# Filter CSV rows and reformat
awk -F, '$5 > 1000 { printf "%s,%s,%.2f\n", $1, $2, $5/100 }' data.csv

# Running total
awk '{ total += $1; print $0, total }' transactions.txt

# Deduplicate by first field (keep first occurrence)
awk '!seen[$1]++' data.txt

# Deduplicate entire lines
awk '!seen[$0]++' file.txt     # much faster than sort | uniq for large files

# Transpose a file (rows become columns)
awk '{ for (i=1; i<=NF; i++) a[i][NR]=$i }
     END { for (i=1; i<=NF; i++) { for (j=1; j<=NR; j++) printf "%s ", a[i][j]; printf "\n" } }' matrix.txt

# Print lines between N and M
awk 'NR>=10 && NR<=20' file.txt

# Sum column 3 for rows where column 1 matches pattern
awk '/^server/ { sum += $3 } END { print sum }' metrics.txt

# Multi-file: process header from first file only
awk 'FNR==1 && NR!=1 { next } { print }' file1.csv file2.csv file3.csv

14. cut — Extract Fields and Characters

cut extracts columns from each line of input by delimiter, byte position, or character position.

Flags

Flag	Meaning
`-d DELIM`	Field delimiter (default: tab)
`-f LIST`	Select fields
`-c LIST`	Select character positions
`-b LIST`	Select byte positions
`--complement`	Invert selection
`-s`	Suppress lines without delimiter

List Format

A LIST can be: a single number 3, a range 2-5, multiple 1,3,5, or mixed 1,3-5,7.

# Extract field 2 (tab-delimited by default)
cut -f2 data.tsv

# Extract fields 1, 3, and 5
cut -f1,3,5 data.tsv

# Extract fields 2 through 4
cut -f2-4 data.tsv

# Extract fields from 3 to end
cut -f3- data.tsv

# Extract up to field 3
cut -f-3 data.tsv

# Comma-delimited
cut -d, -f2 data.csv
cut -d, -f1,3 data.csv

# Colon-delimited: get usernames from /etc/passwd
cut -d: -f1 /etc/passwd

# Get home directories
cut -d: -f6 /etc/passwd

# Get shells
cut -d: -f1,7 /etc/passwd

# Character positions
cut -c1-10 file.txt        # first 10 characters of each line
cut -c5 file.txt            # 5th character
cut -c1,5,10 file.txt

# Byte positions (differs from -c for multibyte UTF-8)
cut -b1-10 file.txt

# Remove first field (complement)
cut --complement -f1 data.tsv
cut --complement -d, -f1 data.csv

# Suppress lines without delimiter
cut -d, -f2 -s data.csv    # skip lines without commas

Practical Applications

# Extract hostname from access log (field 1)
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head

# Get PATH directories
echo $PATH | tr ':' '\n'

# Extract first word of each line
cut -d' ' -f1 file.txt

# Get file extensions from filenames
ls | cut -d'.' -f2-    # everything after first dot

# Trim long lines to 80 characters for display
cut -c1-80 wide_output.txt

# Extract columns from /etc/passwd
cut -d: -f1,3,4 /etc/passwd | sort -t: -k2 -n

# Get IP addresses (first field, space-delimited)
cut -d' ' -f1 access.log

# Extract year from date column (YYYY-MM-DD)
cut -d- -f1 dates.txt

15. paste — Merge Lines Side by Side

paste is the inverse of cut. It merges lines from multiple files horizontally.

# Paste two files side by side (tab-separated)
paste file1.txt file2.txt

# Use a different delimiter
paste -d, file1.txt file2.txt

# Multiple delimiters (cycled)
paste -d',;' file1.txt file2.txt file3.txt

# Merge all lines into one (serial mode)
paste -s file.txt           # all lines joined with tab
paste -s -d, file.txt       # all lines joined with comma

# Practical: create key=value pairs
paste -d= keys.txt values.txt

# Number lines (add sequence number column)
seq 1 $(wc -l < file.txt) | paste - file.txt

# Combine stdout columns
seq 1 5 | paste - -          # two columns (reads two lines per row)
seq 1 9 | paste - - -        # three columns

16. tr — Translate or Delete Characters

tr translates, squeezes, or deletes characters from stdin. It does not read files; always uses stdin.

Basic Syntax

tr [OPTIONS] SET1 [SET2]

Basic Translation

# Uppercase to lowercase
echo "HELLO WORLD" | tr 'A-Z' 'a-z'
# hello world

# Lowercase to uppercase
cat file.txt | tr 'a-z' 'A-Z'

# Replace spaces with underscores
echo "hello world foo" | tr ' ' '_'
# hello_world_foo

# Replace colons with tabs
cat /etc/passwd | tr ':' '\t'

# ROT13 encoding
echo "Hello, World!" | tr 'A-Za-z' 'N-ZA-Mn-za-m'

Character Classes

# [:alpha:]  - letters
# [:digit:]  - digits
# [:alnum:]  - alphanumeric
# [:space:]  - whitespace (space, tab, newline, etc.)
# [:upper:]  - uppercase letters
# [:lower:]  - lowercase letters
# [:punct:]  - punctuation
# [:print:]  - printable characters

echo "Hello, World! 123" | tr '[:upper:]' '[:lower:]'
echo "abc123DEF" | tr '[:lower:]' '[:upper:]'

Deletion with -d

# Delete digits
echo "abc123def456" | tr -d '0-9'
# abcdef

# Delete all non-digits
echo "Phone: (555) 123-4567" | tr -cd '0-9'
# 5551234567

# Delete newlines (join all lines)
cat file.txt | tr -d '\n'

# Delete carriage returns (fix Windows files)
tr -d '\r' < windows_file.txt > unix_file.txt
cat -v file.txt | tr -d '\r' > fixed.txt

# Delete punctuation
echo "Hello, World!" | tr -d '[:punct:]'
# Hello World

Squeeze with -s

-s replaces sequences of the same character with a single instance.

# Squeeze multiple spaces into one
echo "hello     world" | tr -s ' '
# hello world

# Squeeze multiple newlines into one
cat file.txt | tr -s '\n'

# Squeeze digits
echo "aaa1111bbb222" | tr -s '0-9'
# aaa1bbb2

# Normalize whitespace (spaces and tabs)
echo "hello	   world" | tr -s '[:space:]' ' '
# hello world

Complement with -c

-c uses the complement of SET1 (everything NOT in SET1).

# Keep only alphanumeric characters
echo "Hello, World! 123" | tr -cd '[:alnum:]'
# HelloWorld123

# Keep only printable ASCII
cat binary_file.txt | tr -cd '[:print:]\n'

# Keep only digits and newlines
cat file.txt | tr -cd '0-9\n'

Practical Applications

# Convert spaces to newlines (one word per line)
echo "one two three four" | tr ' ' '\n'

# Convert newlines to spaces (single-line output)
cat file.txt | tr '\n' ' '

# Count unique characters in a file
cat file.txt | tr -d '\n' | fold -w1 | sort | uniq -c | sort -rn

# Extract numbers from text
echo "I have 42 apples and 17 oranges" | tr -cd '0-9 \n'

# Simple Caesar cipher
echo "hello" | tr 'a-z' 'n-za-m'

# Remove null bytes from file
tr -d '\000' < file_with_nulls.bin > clean_file.bin

17. sort — Sort Lines of Text

sort sorts lines of text from stdin or files. It is stable by default in GNU sort.

Basic Flags

Flag	Meaning
`-r`	Reverse sort
`-n`	Numeric sort
`-h`	Human-readable number sort (1K, 2M, 1G)
`-g`	General numeric (handles scientific notation)
`-f`	Case-insensitive
`-u`	Unique: remove duplicate lines
`-k N`	Sort by field N
`-k N,M`	Sort by fields N to M
`-t DELIM`	Field delimiter
`-s`	Stable sort
`-R`	Random shuffle
`-o FILE`	Write to file (can be input file)
`-b`	Ignore leading blanks
`-V`	Version sort (1.2 < 1.10)
`-c`	Check if sorted (exit 1 if not)
`-m`	Merge already-sorted files

Basic Sorting

# Alphabetical sort
sort file.txt

# Reverse sort
sort -r file.txt

# Numeric sort
sort -n numbers.txt

# Sort and remove duplicates
sort -u file.txt

# Sort in-place (overwrite input file)
sort file.txt -o file.txt

# Sort multiple files together
sort file1.txt file2.txt file3.txt

Sorting by Field

# Sort by 2nd field (whitespace-delimited)
sort -k2 file.txt

# Sort by 2nd field numerically
sort -k2 -n file.txt

# Sort by 2nd field, reverse numerically
sort -k2 -rn file.txt

# Sort CSV by field 3
sort -t, -k3 -n data.csv

# Sort /etc/passwd by UID (field 3)
sort -t: -k3 -n /etc/passwd

# Multi-key sort: primary key field 2, secondary key field 1
sort -k2,2 -k1,1 file.txt

# Sort by field 2 numerically, then field 1 alphabetically
sort -t, -k2,2n -k1,1 data.csv

# Sort by last field (using awk to reverse fields first)
awk '{ for(i=NF;i>=1;i--) printf $i" "; print "" }' file.txt | sort | \
    awk '{ for(i=NF;i>=1;i--) printf $i" "; print "" }'

Human-Readable Sort

# Sort du output correctly
du -sh * | sort -h
# 1.0K    file.txt
# 45K     script.sh
# 1.2M    image.png
# 3.4G    backup.tar.gz

# Sort ls -lh output by size
ls -lh | sort -k5 -h

Version Sort

# Incorrect numeric sort for versions
echo -e "1.2\n1.10\n1.9" | sort -n
# 1.2
# 1.9
# 1.10    <- wrong (9 < 10)

# Correct version sort
echo -e "1.2\n1.10\n1.9" | sort -V
# 1.2
# 1.9
# 1.10

Practical Applications

# Find most frequent words in a text
tr ' ' '\n' < document.txt | tr -d '[:punct:]' | tr 'A-Z' 'a-z' | \
    sort | uniq -c | sort -rn | head -20

# Sort processes by memory usage
ps aux | sort -k4 -rn | head -20

# Sort log entries by timestamp (field 4)
sort -k4 access.log

# Find duplicate files by size
du -b * | sort -n

# Sort IP addresses correctly
sort -t. -k1,1n -k2,2n -k3,3n -k4,4n iplist.txt

# Randomly shuffle lines (e.g., for sampling)
sort -R file.txt | head -100

# Check if a file is sorted
sort -c file.txt && echo "sorted" || echo "not sorted"

# Merge pre-sorted files efficiently
sort -m sorted1.txt sorted2.txt sorted3.txt > merged.txt

18. uniq — Filter Duplicate Lines

uniq filters adjacent duplicate lines. Input must be sorted for it to find all duplicates (use sort | uniq or sort -u).

Flags

Flag	Meaning
`-c`	Prefix lines with count
`-d`	Print only duplicate lines
`-u`	Print only unique lines (no duplicates)
`-i`	Case-insensitive comparison
`-f N`	Skip first N fields
`-s N`	Skip first N characters
`-w N`	Compare only first N characters

# Remove duplicates (input must be sorted)
sort file.txt | uniq

# Count occurrences of each line
sort file.txt | uniq -c

# Most frequent lines
sort file.txt | uniq -c | sort -rn | head

# Print only lines that appear more than once
sort file.txt | uniq -d

# Print only lines that appear exactly once
sort file.txt | uniq -u

# Case-insensitive deduplication
sort -f file.txt | uniq -i

# Compare only first 5 characters
sort file.txt | uniq -w 5

Practical Applications

# Most common IPs hitting your server
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

# Most common HTTP status codes
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# Find duplicate filenames (regardless of directory)
find /path -type f -printf "%f\n" | sort | uniq -d

# Words that appear exactly once in a document
tr ' ' '\n' < doc.txt | tr -d '[:punct:]' | tr 'A-Z' 'a-z' | sort | uniq -u

# Count unique values per column in CSV
awk -F, '{print $2}' data.csv | sort | uniq -c | sort -rn

# Find users who logged in multiple times
last | awk '{print $1}' | sort | uniq -d

19. tee — Split Output to File and Stdout

tee reads stdin and writes to both stdout AND one or more files simultaneously. Named after the T-pipe fitting that splits flow in two directions.

# Write to file and print to terminal
command | tee output.txt

# Append to file (don't overwrite)
command | tee -a output.txt

# Write to multiple files
command | tee file1.txt file2.txt

# Continue the pipeline AND save intermediate output
cat data.csv | tee raw_data.csv | awk -F, '{print $3}' | tee column3.txt | sort -n

# Write to file AND pass to next command
grep "ERROR" logfile.txt | tee errors.txt | wc -l

# Write to a file as root (via sudo tee)
echo "127.0.0.1 myhost" | sudo tee -a /etc/hosts

# Overwrite a root-owned file
echo "new content" | sudo tee /etc/config_file

# Suppress stdout while writing to file
command | tee output.txt > /dev/null

20. find — Search Filesystem and Act on Files

find searches for files in a directory hierarchy based on name, type, size, modification time, permissions, and more. It can also execute commands on each found file.

Basic Syntax

find [PATH...] [EXPRESSION]

Finding by Name

# Find files named exactly "foo.txt"
find . -name "foo.txt"

# Case-insensitive name match
find . -iname "*.txt"

# Find by extension
find . -name "*.log"
find . -name "*.c" -o -name "*.h"

# Find files matching a pattern
find /var/log -name "*.log"

# Exclude a pattern (! negates)
find . -name "*.js" ! -name "*.min.js"

Finding by Type

# Regular files
find . -type f

# Directories
find . -type d

# Symbolic links
find . -type l

# Named pipes
find . -type p

# Sockets
find . -type s

Finding by Size

# Exactly 100 bytes
find . -size 100c

# Greater than 10MB
find . -size +10M

# Less than 1KB
find . -size -1k

# Between 1MB and 100MB
find . -size +1M -size -100M

# Find empty files
find . -empty -type f

# Find empty directories
find . -empty -type d

Finding by Time

# Modified in the last day (24 hours)
find . -mtime -1

# Modified more than 7 days ago
find . -mtime +7

# Modified exactly 3 days ago
find . -mtime 3

# Access time (last read)
find . -atime -1

# Change time (inode metadata changed)
find . -ctime -1

# Modified in last 60 minutes
find . -mmin -60

# Modified in last N minutes
find . -mmin -30

# Newer than a reference file
find . -newer reference.txt

Finding by Permissions

# Find files with specific permissions (exact)
find . -perm 755

# Find files with at least these permissions
find . -perm -644

# Find world-writable files
find / -perm -o+w -type f

# Find SUID files
find / -perm /4000

# Find SGID files
find / -perm /2000

# Find SUID or SGID files
find / -perm /6000 -type f

Finding by Owner

# Files owned by user "alice"
find . -user alice

# Files owned by group "wheel"
find . -group wheel

# Files not owned by anyone (dangling UID)
find . -nouser

Depth Control

# Maximum depth of 2 directories
find . -maxdepth 2 -name "*.txt"

# Minimum depth of 3 directories
find . -mindepth 3 -type f

# Only immediate children (not recursive)
find . -maxdepth 1 -type f

Logical Operators

# AND (default, also -a)
find . -name "*.c" -mtime -7

# OR (-o)
find . -name "*.c" -o -name "*.h"

# NOT (! or -not)
find . ! -name "*.txt"
find . -not -name "*.txt"

# Grouping with parentheses
find . \( -name "*.c" -o -name "*.h" \) -mtime -7

Executing Commands

# Execute a command on each found file
find . -name "*.log" -exec rm {} \;

# The {} is the placeholder; \; ends the command

# More efficient: pass all files at once to one command invocation
find . -name "*.log" -exec rm {} +

# Execute and prompt before each action
find . -name "*.txt" -ok rm {} \;

# Combine with xargs for more control
find . -name "*.py" -print0 | xargs -0 wc -l

# Find and grep
find . -name "*.c" -exec grep -l "malloc" {} +

# Find and change permissions
find . -name "*.sh" -exec chmod +x {} +

# Find and copy to a directory
find . -name "*.jpg" -exec cp {} /backup/photos/ \;

# Find and rename (complex: use xargs and sed)
find . -name "*.txt" | xargs -I{} bash -c 'mv "$1" "${1%.txt}.md"' -- {}

# Delete files (carefully!)
find . -name "*.tmp" -delete          # safer: atomic delete
find . -name "*.tmp" -exec rm {} +   # also fine

# Find and print with format
find . -name "*.c" -printf "%s\t%p\n"   # size and path
find . -type f -printf "%TY-%Tm-%Td\t%p\n"  # date and path

Practical Applications

# Find files modified in last day
find /var/www -type f -mtime -1

# Find large files (>100MB) and list by size
find / -size +100M -type f -printf "%s %p\n" | sort -rn | head

# Find and delete old log files
find /var/log -name "*.log" -mtime +30 -delete

# Find all broken symlinks
find . -type l ! -e

# Find duplicate file sizes (potential duplicates)
find . -type f -printf "%s %p\n" | sort -n | uniq -D -w 10

# Find recently modified config files
find /etc -type f -mtime -7 -name "*.conf"

# Find world-writable files (security audit)
find / -type f -perm -o+w -not -path "/proc/*" -not -path "/sys/*"

# Count files by extension
find . -type f | sed 's/.*\.//' | sort | uniq -c | sort -rn

# Find all C source files and count total lines
find . -name "*.c" -exec wc -l {} + | tail -1

# Find and compress old logs
find /var/log -name "*.log" -mtime +7 -exec gzip {} \;

# Find files between two dates
find . -type f -newer 2024-01-01 ! -newer 2024-06-01

21. xargs — Build and Execute Commands from Input

xargs reads items from stdin (one per line by default) and executes a command with those items as arguments. It handles the limits on argument list length automatically.

Basic Usage

# Pass each line as argument to echo
echo "file1 file2 file3" | xargs echo

# Delete files from a list
cat files_to_delete.txt | xargs rm

# Count lines in many files
find . -name "*.c" | xargs wc -l

Flags

Flag	Meaning
`-n N`	Use at most N arguments per command
`-I {}`	Replace `{}` with each argument
`-0`	Null-delimited input (for filenames with spaces)
`-P N`	Run N processes in parallel
`-d DELIM`	Use custom delimiter
`-t`	Print commands before executing
`-p`	Prompt before each execution
`-r`	Don't run if input is empty
`-a FILE`	Read from file instead of stdin

Safe Handling of Filenames with Spaces

# Problem: spaces in filenames break xargs
find . -name "*.txt" | xargs rm        # WRONG if filenames have spaces

# Solution: null-terminate with -print0 and -0
find . -name "*.txt" -print0 | xargs -0 rm    # CORRECT

# Or use find's -exec directly
find . -name "*.txt" -exec rm {} +

Argument Replacement with -I

# Replace {} with each argument
find . -name "*.jpg" | xargs -I{} convert {} {}.png

# Run a command template with the argument
ls *.sql | xargs -I{} bash -c 'psql mydb < {}'

# Rename files
find . -name "*.txt" -print0 | xargs -0 -I{} mv {} {}.bak

Parallel Execution

# Run 4 processes in parallel
find . -name "*.c" -print0 | xargs -0 -P4 gcc -c

# Parallel image conversion
find photos/ -name "*.jpg" -print0 | xargs -0 -P8 -I{} convert {} -resize 1024x1024 resized/{}

# Parallel grep (useful for huge files)
find . -name "*.log" -print0 | xargs -0 -P4 grep -l "ERROR"

Practical Applications

# Remove all .pyc files
find . -name "*.pyc" -print0 | xargs -0 rm -f

# Run tests in parallel
find tests/ -name "test_*.py" -print0 | xargs -0 -P4 python3

# Lint all JavaScript files
find src/ -name "*.js" -not -path "*/node_modules/*" -print0 | xargs -0 eslint

# Download a list of URLs
cat urls.txt | xargs -n1 wget

# Batch rename: add prefix
ls | xargs -I{} mv {} "prefix_{}"

# Check if a list of servers is up
cat servers.txt | xargs -I{} -P10 ping -c1 {}

# Mass search and replace across files
grep -rl "oldterm" . | xargs sed -i 's/oldterm/newterm/g'

# Create many directories
echo "dir1 dir2 dir3" | xargs mkdir -p

# Remove empty directories
find . -type d -empty -print0 | xargs -0 rmdir

22. diff — Compare Files Line by Line

diff compares two files and outputs the differences. It is the foundation of version control.

Basic Usage

diff file1.txt file2.txt

Output Format

2c2          <- line 2 Changed to line 2
< old text   <- content from file1
---
> new text   <- content from file2

5d4          <- line 5 Deleted (line 4 in file2)
< deleted line

7a8          <- After line 7 in file1, line 8 Added
> new line

Flags

Flag	Meaning
`-u N`	Unified format with N lines context (default 3)
`-c`	Context format
`-s`	Report when files are identical
`-q`	Report only whether files differ
`-i`	Case-insensitive
`-w`	Ignore whitespace
`-b`	Ignore changes in amount of whitespace
`-B`	Ignore blank line changes
`-r`	Recursive directory diff
`-N`	Treat absent files as empty
`--color`	Colorize output
`-y`	Side-by-side output
`--suppress-common-lines`	With -y, only show differences

Unified Format (Most Common)

# -u3 is the standard for patches
diff -u original.txt modified.txt

# Output:
# --- original.txt   2024-01-01 12:00:00
# +++ modified.txt   2024-01-02 15:30:00
# @@ -5,7 +5,9 @@
#  context line
# -removed line
# +added line
#  more context

# More context lines
diff -u10 original.txt modified.txt

# No context lines
diff -u0 original.txt modified.txt

Directory Comparison

# Compare two directories
diff -r dir1/ dir2/

# Show only file list differences, not content
diff -rq dir1/ dir2/

# Ignore .git directory
diff -r --exclude=".git" dir1/ dir2/

Side-by-Side Comparison

# Side-by-side
diff -y file1.txt file2.txt

# Side-by-side, only differences
diff -y --suppress-common-lines file1.txt file2.txt

# Set column width
diff -y --width=120 file1.txt file2.txt

Practical Applications

# Generate a patch
diff -u original.c modified.c > fix.patch

# Apply a patch
patch original.c < fix.patch

# Compare configuration files
diff /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak

# Find new files in a directory
diff -rq dir_old/ dir_new/ | grep "^Only in dir_new"

# Compare command output
diff <(ls dir1/) <(ls dir2/)
diff <(sort file1.txt) <(sort file2.txt)

# Check if two files are identical (exit 0=same, 1=different)
diff -q file1.txt file2.txt > /dev/null && echo "identical"

# Visual diff with colordiff
diff -u file1.txt file2.txt | colordiff

# Create patch for entire directory tree
diff -ruN original_dir/ modified_dir/ > changes.patch

23. patch — Apply Diff Patches

patch applies changes described in a diff file to original files.

# Apply a patch
patch < changes.patch

# Apply to specific file
patch original.txt < changes.patch

# Reverse a patch
patch -R < changes.patch

# Dry run (test without applying)
patch --dry-run < changes.patch

# Apply patch to a directory tree
patch -p1 < feature.patch    # strip 1 leading directory from paths

# Create and apply patch workflow
diff -u old.c new.c > fix.patch
patch old.c < fix.patch

# For directory patches
diff -ruN old_src/ new_src/ > update.patch
patch -p1 < update.patch

24. strings — Extract Printable Strings from Binaries

strings searches binary files for sequences of printable characters (at least 4 by default) and prints them. Essential for binary analysis and debugging.

# Extract strings from a binary
strings /usr/bin/ls

# Minimum string length of 8
strings -n 8 binary_file

# Show file offsets
strings -t x binary_file     # hex offset
strings -t d binary_file     # decimal offset

# Search for specific strings in binary
strings malware.bin | grep "password"
strings library.so | grep "http"
strings firmware.bin | grep -i "version"
strings executable | grep -E "[0-9]+\.[0-9]+\.[0-9]+"  # version numbers

# Find embedded URLs
strings binary | grep -oE "https?://[^\x00-\x1f ]+"

# Find embedded paths
strings binary | grep "^/"

# Find environment variable names
strings binary | grep "^[A-Z_][A-Z0-9_]*$"

# Search for copyright/license info
strings binary | grep -i "copyright\|license\|GPL\|MIT"

25. hexdump and xxd — Inspect Raw Bytes

hexdump

# Default hex + ASCII view
hexdump file.bin

# Canonical format (16 bytes/line, hex + ASCII)
hexdump -C file.bin

# View first 64 bytes
hexdump -C -n 64 file.bin

# Only ASCII output
hexdump -e '"%c"' file.txt

# Raw hex bytes
hexdump -v -e '/1 "%02x "' file.bin

# Inspect file magic bytes (header)
hexdump -C -n 16 unknown_file

xxd

xxd is more versatile — it can also convert hex back to binary.

# Hex dump
xxd file.bin

# First 64 bytes
xxd -l 64 file.bin

# Starting at offset 256
xxd -s 256 file.bin

# Plain hex output (no ASCII column)
xxd -p file.bin

# Compact hex output
xxd -p file.bin | tr -d '\n'

# Reverse: hex to binary
echo "48656c6c6f" | xxd -r -p > hello.bin

# In-place binary editing (classic hex patch technique)
xxd binary | sed 's/deadbeef/cafebabe/g' | xxd -r > patched_binary

# View a specific section of a file
xxd -s 0x100 -l 0x40 file.bin

# Group bytes differently
xxd -g 4 file.bin    # group by 4 bytes
xxd -g 1 file.bin    # one byte per group

26. od — Octal Dump

od dumps files in various formats (octal, hex, decimal, ASCII, etc.).

# Default: octal format
od file.bin

# Hex bytes
od -t x1 file.bin

# Hex shorts (2 bytes)
od -t x2 file.bin

# Unsigned decimal integers
od -t u4 file.bin

# ASCII with escape sequences
od -c file.txt

# Characters and octal
od -An -c file.txt    # no address, just chars

# View line endings
echo "Hello" | od -c
# 0000000   H   e   l   l   o  \n
# 0000006

echo -e "Hello\r\n" | od -c
# 0000000   H   e   l   l   o  \r  \n
# 0000007

27. expand and unexpand — Tabs and Spaces

# Convert tabs to spaces (default: 8 spaces per tab)
expand file.txt

# 4 spaces per tab
expand -t 4 file.txt

# Convert spaces back to tabs
unexpand file.txt

# Only convert leading spaces to tabs
unexpand -a file.txt

28. fold and fmt — Rewrap Lines

fold

fold wraps long lines to a specified width.

# Wrap at 80 characters
fold -w 80 file.txt

# Break at word boundaries
fold -w 80 -s file.txt

# Wrap a long string for display
echo "This is a very long string that should be wrapped for display purposes" | fold -w 40 -s

fmt

fmt is a smarter text formatter that tries to produce "nice" line lengths.

# Reformat to 72-char lines (default)
fmt file.txt

# Set width
fmt -w 80 file.txt

# Format and maintain paragraph structure
fmt -p '#' code_comments.txt    # prefix: treat lines starting with # as paragraphs

# Uniform spacing between sentences
fmt -u file.txt

29. column — Align Output in Columns

column formats its input into columns.

# Format into columns (fills columns then rows)
ls | column

# Specify number of columns
ls | column -c 3

# Table mode with delimiter
cat /etc/passwd | column -t -s ':'

# Table from CSV
cat data.csv | column -t -s ','

# Auto-format mount output
mount | column -t

# Format df output
df -h | column -t

# Custom delimiter output
cat file.txt | column -t -s '|'

# Practical: pretty-print key=value config
grep -v "^#" config.conf | grep "=" | column -t -s '='

30. nl — Number Lines

nl numbers lines with more control than cat -n.

# Number all lines
nl file.txt

# Number only non-blank lines
nl -b t file.txt

# Number only lines matching regex
nl -b p'^[A-Z]' file.txt

# Custom format
nl -n rz file.txt      # right-justified, zero-padded
nl -n ln file.txt      # left-justified
nl -n rn file.txt      # right-justified (default)

# Custom width
nl -w 4 file.txt

# Custom separator
nl -s ' | ' file.txt

# No header/footer numbering
nl -h n -f n -b a file.txt   # body only

31. rev — Reverse Lines Character by Character

# Reverse each line character by character
echo "hello" | rev
# olleh

rev file.txt

# Reverse field order (combined with tr)
echo "one:two:three" | rev | tr ':' '\n' | rev
# three
# two
# one

# Get last field (reverse, cut, reverse)
echo "path/to/some/file.txt" | rev | cut -d'/' -f1 | rev
# file.txt

32. join — Join Lines of Two Files on a Common Field

join performs a relational join on two sorted files sharing a common field.

# Files must be sorted on the join field
sort -k1 file1.txt > sorted1.txt
sort -k1 file2.txt > sorted2.txt
join sorted1.txt sorted2.txt

# Specify join field
join -1 2 -2 3 file1.txt file2.txt    # field 2 from file1, field 3 from file2

# Delimiter
join -t, file1.csv file2.csv

# Also print unmatched lines (like outer join)
join -a1 -a2 file1.txt file2.txt

# Print only unmatched lines from file 1
join -v1 file1.txt file2.txt

# Format output
join -o 1.1,2.2,1.3 file1.txt file2.txt   # field order in output

33. comm — Compare Two Sorted Files Line by Line

comm compares two sorted files and produces three columns:

Column 1: lines only in file1
Column 2: lines only in file2
Column 3: lines in both files

# Full three-column output
comm sorted_file1.txt sorted_file2.txt

# Suppress column 1 (lines only in file1)
comm -1 sorted_file1.txt sorted_file2.txt

# Suppress column 2 (lines only in file2)
comm -2 sorted_file1.txt sorted_file2.txt

# Show only common lines (intersection)
comm -12 sorted_file1.txt sorted_file2.txt

# Show only lines unique to file1 (difference: file1 - file2)
comm -23 sorted_file1.txt sorted_file2.txt

# Show only lines unique to file2 (difference: file2 - file1)
comm -13 sorted_file1.txt sorted_file2.txt

# Practical: find common packages between two systems
comm -12 <(dpkg --get-selections | sort) <(ssh other "dpkg --get-selections" | sort)

# Find files present in both directories
comm -12 <(ls dir1/ | sort) <(ls dir2/ | sort)

34. look — Find Lines Starting with a String

look searches a sorted file for lines beginning with a string. Uses binary search and is very fast.

# Search the system dictionary
look "pre"          # words starting with "pre"
look "inter"

# Search a custom file
look "Error" logfile.txt

# Case-insensitive
look -f "error" logfile.txt

# Specify delimiter for field comparison
look -t: "root" /etc/passwd

# Check if a word is in the dictionary
look "$word" /usr/share/dict/words && echo "valid word" || echo "not a word"

35. Combining Tools: Real-World Pipelines

This section demonstrates the power of combining these tools in real-world scenarios.

Log Analysis

# Top 10 IP addresses hitting your webserver
awk '{print $1}' /var/log/nginx/access.log | \
    sort | uniq -c | sort -rn | head -10

# Count requests per HTTP method
awk '{print $6}' /var/log/nginx/access.log | \
    tr -d '"' | sort | uniq -c | sort -rn

# All 5xx error URLs in the last hour
awk -v start="$(date -d '1 hour ago' +'[%d/%b/%Y:%H')" \
    '$4 > start && $9 ~ /^5/' /var/log/nginx/access.log | \
    awk '{print $7}' | sort | uniq -c | sort -rn

# Bandwidth consumed per IP
awk '{bytes[$1]+=$10} END {for (ip in bytes) print bytes[ip], ip}' access.log | \
    sort -rn | head | awk '{printf "%s\t%.2f MB\n", $2, $1/1048576}'

# Response time analysis (Apache combined log, field 11 = response time in ms)
awk '{sum+=$11; n++; if ($11>max) max=$11} END {print "Avg:", sum/n, "Max:", max}' access.log

# Find slow queries in MySQL slow log
grep -A3 "Query_time" mysql-slow.log | grep "^SELECT\|^INSERT\|^UPDATE" | \
    sort | uniq -c | sort -rn | head

Code Analysis

# Count lines of code by extension
find . -type f | grep -oP '\.\K[a-z]+$' | sort | uniq -c | sort -rn

# Find all TODOs with file and line
grep -rn "TODO\|FIXME\|HACK" --include="*.{c,h,py,go,rs}" . | \
    sed 's/:[^:]*:/: /'

# Find the most complex functions (most lines between { and })
# (approximate, works for C-style languages)
awk '/^[a-zA-Z].*\(/{name=$0; count=0} /^{/{inside=1} inside{count++} \
     /^}/{if(inside) print count, name; inside=0}' *.c | sort -rn | head

# Find all #include dependencies
grep -h "#include" *.c | sort -u

# Check for debug prints left in code
grep -rn "printf\|console\.log\|print\|dbg!" --include="*.c" --include="*.js" . | \
    grep -v "//.*printf"   # exclude commented-out ones

# Find files with the most lines (complexity proxy)
find . -name "*.c" -exec wc -l {} + | sort -rn | head -20

# Find all magic numbers in C code
grep -oP '\b\d{3,}\b' *.c | grep -v "^\s*//" | sort -t: -k2 -n

System Administration

# Find the top disk consumers
du -sh /home/* 2>/dev/null | sort -h -r | head -10

# Monitor a log file and alert on errors
tail -f /var/log/app.log | grep --line-buffered "CRITICAL" | \
    while IFS= read -r line; do
        echo "ALERT: $line" | mail -s "Critical Error" admin@example.com
    done

# Count failed SSH logins per IP
grep "Failed password" /var/log/auth.log | \
    awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20

# Find processes consuming the most memory
ps aux --sort=-%mem | head -10 | awk '{printf "%-20s %s%%\n", $11, $4}'

# Generate a report of open ports
ss -tlnp | awk 'NR>1 {print $4}' | cut -d: -f2 | sort -n | uniq

# Disk usage by file type
find /path -type f -printf "%s %f\n" | \
    awk '{split($2,a,"."); ext=a[length(a)]; size[ext]+=$1; count[ext]++}
         END {for (e in size) printf "%10d %6d %s\n", size[e], count[e], e}' | \
    sort -rn | head -20

Data Processing

# Calculate column statistics from CSV
awk -F, 'NR>1 {sum+=$3; n++; if($3>max) max=$3; if(NR==2||$3<min) min=$3}
         END {printf "Count: %d\nSum: %.2f\nAvg: %.2f\nMin: %.2f\nMax: %.2f\n",
              n, sum, sum/n, min, max}' data.csv

# Join two CSVs on a common field
join -t, -1 1 -2 1 \
    <(sort -t, -k1 file1.csv) \
    <(sort -t, -k1 file2.csv)

# Transpose a CSV
awk -F, '{for(i=1;i<=NF;i++) a[i][NR]=$i}
         END {for(i=1;i<=NF;i++){for(j=1;j<=NR;j++) printf "%s%s",a[i][j],(j<NR?",":""); print ""}}' data.csv

# Find rows where column 4 exceeds a threshold
awk -F, '$4 > 1000 {print}' data.csv | tee filtered.csv | wc -l

# Remove CSV rows with empty fields
awk -F, '{valid=1; for(i=1;i<=NF;i++) if($i=="") valid=0; if(valid) print}' data.csv

# Pivot: count occurrences of (col1, col2) pairs
awk -F, '{count[$1","$2]++}
         END {for (k in count) print count[k], k}' data.csv | sort -rn

# Normalize whitespace in a TSV
sed 's/\t\+/\t/g; s/^\t//; s/\t$//' data.tsv

Text Transformation

# Convert markdown list to numbered list
grep "^-" list.md | nl -v1 -nrn -w3 | sed 's/-/. /'

# Wrap text to 72 columns and format
cat essay.txt | fmt -w 72 | sed 's/^/    /'    # indent 4 spaces

# Extract all email addresses from text
grep -oP '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' messy_text.txt | \
    sort -u

# Extract all URLs
grep -oP 'https?://[^\s"<>]+' document.html | sort -u

# Convert CSV to markdown table
awk -F, 'NR==1 {
    printf "|"; for(i=1;i<=NF;i++) printf " %s |", $i; printf "\n"
    printf "|"; for(i=1;i<=NF;i++) printf " --- |"; printf "\n"
}
NR>1 {
    printf "|"; for(i=1;i<=NF;i++) printf " %s |", $i; printf "\n"
}' data.csv

# Count word frequency in a document
tr '[:upper:]' '[:lower:]' < document.txt | \
    tr -cs '[:alpha:]' '\n' | \
    grep -v '^$' | \
    sort | uniq -c | sort -rn | head -30

# Strip HTML tags
sed 's/<[^>]*>//g; /^$/d' page.html

# Extract text from simple HTML
sed 's/<[^>]*>//g' page.html | sed '/^\s*$/d' | fmt -w 80

36. Regex Reference for Text Tools

Different tools use different regex dialects. Here's a quick reference:

BRE (Basic Regular Expressions) — grep, sed default

Pattern	Meaning
`.`	Any character
`^`	Start of line
`$`	End of line
`*`	Zero or more of preceding
`\+`	One or more (GNU extension)
`\?`	Zero or one (GNU extension)
`\{n\}`	Exactly n
`\{n,m\}`	n to m
`$…$`	Grouping
`\1`	Backreference
`[abc]`	Character class
`[^abc]`	Negated class

ERE (Extended Regular Expressions) — grep -E, awk, sed -E

Pattern	Meaning
`+`	One or more
`?`	Zero or one
`{n}`	Exactly n
`{n,m}`	n to m
`(…)`	Grouping (no backslash)
`	`

PCRE — grep -P

Adds to ERE:

Pattern	Meaning
`\d`	Digit
`\D`	Non-digit
`\w`	Word character
`\W`	Non-word character
`\s`	Whitespace
`\S`	Non-whitespace
`\b`	Word boundary
`(?=…)`	Lookahead
`(?!…)`	Negative lookahead
`(?<=…)`	Lookbehind
`(?<!…)`	Negative lookbehind
`(?:…)`	Non-capturing group
`(?P<name>…)`	Named capture group

POSIX Character Classes (work in BRE, ERE, awk)

Class	Meaning
`[:alpha:]`	Letters
`[:digit:]`	Digits
`[:alnum:]`	Letters and digits
`[:space:]`	Whitespace
`[:upper:]`	Uppercase letters
`[:lower:]`	Lowercase letters
`[:punct:]`	Punctuation
`[:print:]`	Printable chars
`[:graph:]`	Printable non-space
`[:blank:]`	Space and tab
`[:cntrl:]`	Control characters
`[:xdigit:]`	Hex digits

37. Performance Notes and Gotchas

General Performance

# fgrep is faster than grep for fixed strings
fgrep "literal" huge.log                # fast
grep "literal" huge.log                 # slower (compiles regex)
grep -F "literal" huge.log              # same as fgrep

# awk is faster than multiple sed/grep pipes
# Instead of:
grep "pattern" file | sed 's/foo/bar/' | cut -d: -f2

# Use:
awk '/pattern/ { gsub(/foo/, "bar"); print $2 }' FS=: file

# For large files, avoid reading twice
# Instead of: wc -l file && grep -c "X" file
awk '{lines++; if (/X/) matches++} END {print lines, matches}' file

# sort's parallel merge sort
sort --parallel=4 huge_file.txt

# Use -S to give sort more memory
sort -S 2G huge_file.txt

# For very large files, use parallel GNU tools
parallel grep "pattern" ::: file1 file2 file3 file4

Common Gotchas

# Gotcha 1: grep -c counts lines, not occurrences
grep -c "word" file.txt       # counts lines containing "word"
grep -o "word" file.txt | wc -l  # counts actual occurrences

# Gotcha 2: sed -i behavior differs between GNU and BSD (macOS)
sed -i 's/old/new/' file.txt          # GNU: works
sed -i '' 's/old/new/' file.txt       # BSD/macOS: requires '' argument
sed -i.bak 's/old/new/' file.txt      # works on both (creates backup)

# Gotcha 3: Uninitialized awk variables are 0/""
awk '{print $5}' file_with_4_fields.txt   # prints empty string, not error

# Gotcha 4: cut doesn't handle repeated delimiters
echo "a  b  c" | cut -d' ' -f2   # returns empty (between first two spaces)
echo "a  b  c" | awk '{print $2}' # returns "b" (awk handles this correctly)

# Gotcha 5: sort must precede uniq
uniq file.txt          # only removes ADJACENT duplicates
sort file.txt | uniq   # removes ALL duplicates

# Gotcha 6: Trailing newline in command substitution
# $(cat file.txt) strips trailing newlines!
content=$(cat file.txt)     # trailing newlines stripped
content=$(cat file.txt; echo x); content="${content%x}"  # preserve them

# Gotcha 7: grep -r follows symlinks by default with -R but not -r
grep -r "pattern" .    # does NOT follow symlinks
grep -R "pattern" .    # DOES follow symlinks

# Gotcha 8: echo -e is not portable
echo -e "line\nline"   # works in bash, not in POSIX sh
printf "line\nline\n"  # portable alternative

# Gotcha 9: find -exec vs find -exec ... +
find . -name "*.txt" -exec rm {} \;   # one rm per file (slow)
find . -name "*.txt" -exec rm {} +    # one rm for all files (fast)
find . -name "*.txt" -print0 | xargs -0 rm   # fastest for huge lists

# Gotcha 10: xargs with spaces in filenames
find . -name "*.txt" | xargs rm      # BREAKS on spaces in filenames
find . -name "*.txt" -print0 | xargs -0 rm   # SAFE

Quick Reference Card

echo      - print to stdout
printf    - formatted print (portable)
cat       - concatenate/display files
tac       - cat in reverse
less      - interactive pager
head      - first N lines
tail      - last N lines / follow log
wc        - count lines/words/bytes
grep      - search lines by pattern
sed       - stream editing (substitute, delete, print)
awk       - field-based text processing and computation
cut       - extract fields by delimiter or position
paste     - merge files horizontally
tr        - translate/delete characters
sort      - sort lines
uniq      - deduplicate adjacent lines
tee       - split output to file and stdout
find      - search filesystem
xargs     - build command from stdin
diff      - compare files
patch     - apply diff patches
strings   - extract printable strings from binary
hexdump   - binary file viewer
xxd       - hex dump + reverse
od        - octal/various format dump
expand    - tabs to spaces
fold      - wrap long lines
fmt       - reformat text
column    - align into columns
nl        - number lines
rev       - reverse characters per line
join      - relational join on files
comm      - compare sorted files set-theoretically
look      - binary search in sorted file

This guide was written in the spirit of the Unix philosophy: each tool sharp, each pipe clean. Master these, and you hold a text processing toolkit more powerful than most GUI applications.

Table of Contents

1. Philosophy: The Unix Text Model

Standard Streams

The Pipe

2. echo — Printing to Standard Output

Basic Usage

Flags

Escape Sequences with -e

ANSI Color Codes

Practical Applications

Portability Note

3. printf — Formatted Output

Basic Syntax

Format Specifiers

Examples

printf in Scripts

4. cat — Concatenate and Print Files

Basic Usage

Flags

Practical Applications

The "Useless Use of cat" Anti-pattern

5. tac — Reverse cat

6. less and more — Paging Through Text

more

less

Key Bindings Inside less

Useful less Flags

LESSOPEN: Preprocessor

7. head — First N Lines

Basic Usage

Excluding Trailing Lines

Multiple Files

Practical Applications

8. tail — Last N Lines

Basic Usage

Starting from a Line Number

Follow Mode

Practical Applications

9. wc — Word, Line, and Byte Count

Flags

Practical Applications

10. grep — Pattern Matching and Searching

Basic Syntax

Basic Examples

Essential Flags

Searching Files and Directories

Context Lines

Counting and Testing

Multiple Patterns

Whole Word and Whole Line Matching

Regex Patterns in grep

Named Capture Groups and -oP

grep in Scripts

11. egrep and fgrep — grep Variants

egrep (Extended grep)

fgrep (Fixed-string grep)

12. sed — Stream Editor

Basic Syntax

The Substitution Command: s

In-Place Editing

Delimiters

Addresses

Delete Command: d

Print Command: p

Append, Insert, Change Commands

Multiple Commands

The Hold Space

Practical Applications

13. awk — Pattern-Action Text Processor

Basic Model

Built-in Variables

Basic Examples

Field Separator

Patterns

BEGIN and END Blocks

Arithmetic and String Operations

Arrays

Control Flow

Practical awk Applications

14. cut — Extract Fields and Characters