Build a Complete Web Framework From Scratch — Architecture, Design Patterns & Complete Checklist

· by 0xKiireguidewebweb frameworkframeworkfrom scratchdjangolaravelrailszigdesign patterns

Table of Contents

  1. Understand the Architecture: A Bird's-Eye View
  2. The HTTP Protocol Layer — What the Framework Must Speak
  3. The TCP Server & Socket Management
  4. The HTTP Parser — Turning Bytes into Structure
  5. The WSGI / ASGI Application Interface
  6. The Application Object — The Framework's Front Door
  7. The Router — URL Dispatch and Pattern Matching
  8. The Request Object
  9. The Response Object
  10. The Middleware Pipeline
  11. The View & Handler System
  12. The Template Engine — Lexer
  13. The Template Engine — Parser & Abstract Syntax Tree
  14. The Template Engine — Renderer & Context
  15. The Template Engine — Tags, Filters & Template Inheritance
  16. The ORM — Database Abstraction & Connection Management
  17. The ORM — Model Definition & Metaclass System
  18. The ORM — Field Types, Validators & Descriptors
  19. The ORM — QuerySet & Lazy Evaluation
  20. The ORM — SQL Generation & Query Compilation
  21. The ORM — Relationships: ForeignKey, ManyToMany, OneToOne
  22. The ORM — Migrations System
  23. The ORM — Connection Pooling & Transaction Management
  24. The Form System — Declaration & Binding
  25. The Validation System — Field & Cross-Field Rules
  26. The Authentication & Authorization System
  27. The Session System
  28. Cookie & State Management
  29. The Security Subsystem — CSRF, XSS, Headers & More
  30. The Caching System
  31. The Configuration & Settings System
  32. The Application Registry & Plugin System
  33. Static Files & the Asset Pipeline
  34. The Admin Interface
  35. The Email Subsystem
  36. Background Tasks & the Job Queue
  37. WebSocket & Real-Time Support
  38. The CLI & Management Commands
  39. Logging, Monitoring & Observability
  40. Internationalization & Localization (i18n / l10n)
  41. File Upload & Storage Backends
  42. Signals — Decoupled Application Events
  43. Test Suites
  44. Recommended Build Order

SOLID Principles: Your Constant Compass

A web framework is one of the largest, most interconnected software systems you will ever build. Without deliberate adherence to the SOLID principles, it collapses into a tangle of circular dependencies, untestable globals, and classes that break every time a neighbouring module changes. Before reading a single step below, internalize these five principles and how they manifest specifically in a framework context.

Principle One-line definition What it prevents in a web framework
Single Responsibility Every class has one reason to change The Request class parsing cookies, validating CSRF tokens, running business logic, and writing to a database
Open/Closed Open for extension, closed for modification Adding a new database backend requires editing every ORM query class
Liskov Substitution Subtypes are always substitutable for their base type A ReadOnlySession backend that silently fails on session.set() instead of raising a meaningful error
Interface Segregation No client depends on methods it doesn't use A simple JSON API handler being forced to implement HTML rendering methods from a fat base view class
Dependency Inversion High-level modules depend on abstractions, not concretions The authentication system importing PostgreSQLUserRepository directly instead of depending on a UserRepository interface

These are not academic obligations — every section below identifies exactly which principle is at risk and how the proposed design avoids the violation. A web framework that adheres to SOLID is a framework where a developer can add a MongoDB backend, a new template engine, or a custom authentication provider without ever touching framework-core source files.

GoF Patterns Used in This Project (Master Map)

Pattern Category Primary step(s) where used Core problem it solves in the framework
Chain of Responsibility Behavioral Steps 10, 29 Middleware pipeline; security header pipeline
Command Behavioral Steps 22, 38 Migration operations; management commands
Iterator Behavioral Steps 19, 24 QuerySet lazy evaluation; form field traversal
Observer Behavioral Steps 42, 6 Signals; application event hooks
State Behavioral Steps 5, 37 WSGI/ASGI connection state; WebSocket session state
Strategy Behavioral Steps 16, 25, 30 Database backends; validation strategies; cache backends
Template Method Behavioral Steps 11, 34 Base view lifecycle; admin model admin customization
Visitor Behavioral Steps 13, 20 AST node rendering; SQL query compilation
Adapter Structural Steps 5, 16 WSGI/ASGI adapter; database driver adapter
Composite Structural Steps 7, 15, 24 URL patterns; template inheritance tree; form fieldsets
Decorator Structural Steps 10, 26, 29 Middleware as decorators; permission decorators
Facade Structural Steps 6, 16, 30 Application object; ORM high-level API; cache API
Flyweight Structural Steps 12, 17 Shared token objects; model field instances
Proxy Structural Steps 19, 27 Lazy QuerySet; session lazy loading
Abstract Factory Creational Steps 16, 30, 35 Database backend factory; cache backend factory; email backend factory
Builder Creational Steps 19, 20 QuerySet construction; SQL query builder
Factory Method Creational Steps 8, 9, 11 Request factory; Response factory; View factory
Singleton Creational Steps 6, 31 Application registry; settings object (use sparingly)
Prototype Creational Step 19 QuerySet cloning for chained filters

1. Understand the Architecture: A Bird's-Eye View

The Philosophy of a Web Framework

Before drawing boxes and arrows, you must deeply understand what a web framework actually is philosophically. It is not a collection of utilities. It is an inversion of control container — the framework calls your code, not the other way around. This is the Hollywood Principle: "Don't call us, we'll call you." You define routes, views, models, and templates. The framework orchestrates when and how they are invoked, handles the plumbing between the network and your business logic, enforces security policies, and provides the runtime scaffolding so that your application code only ever needs to think about what to do, not how to talk to a TCP socket.

This inversion has profound architectural implications. Because the framework controls the call stack, it must be extremely careful about what it couples together. Circular dependencies between framework modules are fatal — if the ORM imports the template engine and the template engine imports the ORM, nothing can ever be loaded in isolation. The framework must be designed as a directed acyclic graph of dependencies, flowing from infrastructure at the bottom up to application features at the top.

The Full Stack Layered Architecture

A complete web framework has at least seven distinct layers, each with a precisely bounded responsibility:

┌───────────────────────────────────────────────────────────────────────────────┐
│                           APPLICATION LAYER                                   │
│  Your Views  │  Your Models  │  Your Templates  │  Your Forms  │  Your URLs   │
│  (User code — the framework never touches this layer directly)                │
├───────────────────────────────────────────────────────────────────────────────┤
│                          FRAMEWORK FEATURE LAYER                              │
│  Admin Interface  │  Auth System  │  Form Engine  │  Signal Dispatcher        │
│  CLI Commands     │  i18n Engine  │  File Storage │  Email Subsystem          │
├───────────────────────────────────────────────────────────────────────────────┤
│                          REQUEST/RESPONSE LAYER                               │
│  Router  │  Middleware Pipeline  │  View Dispatch  │  Response Rendering      │
├───────────────────────────────────────────────────────────────────────────────┤
│                            DATA LAYER                                         │
│  ORM Models  │  QuerySet API  │  SQL Compiler  │  Migration Engine            │
│  Connection Pool  │  Transaction Manager  │  Schema Introspection             │
├───────────────────────────────────────────────────────────────────────────────┤
│                          TEMPLATE / RENDERING LAYER                           │
│  Lexer  │  Parser  │  AST  │  Renderer  │  Tag Library  │  Filter Library     │
├───────────────────────────────────────────────────────────────────────────────┤
│                        CROSS-CUTTING CONCERNS LAYER                           │
│  Caching  │  Logging  │  Security (CSRF, XSS)  │  Session  │  Config          │
├───────────────────────────────────────────────────────────────────────────────┤
│                          INFRASTRUCTURE LAYER                                 │
│  HTTP Parser  │  TCP Server  │  WSGI/ASGI Interface  │  Socket I/O            │
│  DB Drivers (Postgres, MySQL, SQLite)  │  Cache Drivers (Redis, Memcached)    │
└───────────────────────────────────────────────────────────────────────────────┘

The Request Journey (End-to-End)

Understanding what happens to a single HTTP request from the moment it arrives at a socket until the response bytes are sent back is the most important mental model in the entire framework. Every architectural decision follows from this journey. Memorize it:

Network socket
     │  Raw bytes (TCP stream)
     ▼
TCP Server
     │  Reads bytes until HTTP message boundary
     ▼
HTTP Parser
     │  Parses start line, headers, body
     │  Produces: method, path, version, headers dict, body bytes
     ▼
WSGI / ASGI Adapter
     │  Wraps parsed data into environ dict or ASGI scope
     ▼
Application Object  (Framework entry point)
     │  Creates Request object from environ/scope
     ▼
Middleware Pipeline  (Chain of Responsibility)
     │  Each middleware can: modify request, short-circuit, modify response
     ▼
Router  (URL matching)
     │  Finds the view callable registered for this URL path + method
     │  Extracts URL parameters (e.g., /users/42/ → user_id=42)
     ▼
View Dispatch
     │  Calls the matched view function or class-based view method
     │  Provides: request object, URL parameters
     ▼
Your View Code  (User code)
     │  Reads request data, queries models, performs business logic
     │  Returns a Response object
     ▼
Middleware Pipeline (response phase, reverse order)
     │  Each middleware can inspect/modify the response
     ▼
Response Serializer
     │  Converts Response object back to HTTP bytes
     │  (status line + headers + body)
     ▼
WSGI / ASGI Adapter
     │  Calls start_response() and yields body chunks
     ▼
TCP Server
     │  Writes bytes back to socket
     ▼
Network socket → client

Horizontal Concerns (Cross-Cutting)

Some responsibilities do not belong to a single layer — they cut horizontally across all of them. These require special architectural treatment because they must be accessible everywhere without creating spaghetti imports:

  • Logging: Every layer needs to log. The solution is a global logger registry (accessed by name, not by direct import of a singleton).
  • Configuration: Every component needs settings. The solution is a lazy-loaded settings object injected at startup.
  • Caching: Can be used in views, ORM queries, template rendering, and middleware. The solution is a cache abstraction with a unified API.
  • Signals: Any layer may need to notify any other layer of an event without creating a direct dependency. The solution is a publish/subscribe signal dispatcher.

Checklist

  • Draw the full layer diagram for your specific implementation before writing any code.
  • Define the dependency rule: upper layers may import from lower layers; lower layers may never import from upper layers.
  • Map the full request journey in a document that every contributor must read first.
  • Identify cross-cutting concerns upfront and decide how they will be globally accessible without creating circular imports.
  • Choose your concurrency model at the start: synchronous WSGI, asynchronous ASGI, or both (a hybrid framework). This decision pervades every I/O operation in the framework.
  • Define the minimum viable deliverable for each phase before writing code (see Step 44).
  • Establish a code review checklist that explicitly checks for SOLID violations in every pull request.

2. The HTTP Protocol Layer — What the Framework Must Speak

HTTP as a Text Protocol Built on TCP

HTTP (HyperText Transfer Protocol) is, at its core, a plain-text protocol layered on top of TCP. Understanding it precisely — not approximately — is essential for building a correct framework. Many subtle bugs in web frameworks trace back to imprecise HTTP parsing: mishandling chunked transfer encoding, incorrectly processing multi-value headers, or failing to read the exact number of body bytes the Content-Length header specifies.

An HTTP/1.1 request message has the following structure, and your parser must handle every part of it correctly:

REQUEST LINE: <METHOD> <Request-URI> <HTTP-Version>\r\n
HEADERS:      <Header-Name>: <Header-Value>\r\n
              <Header-Name>: <Header-Value>\r\n
              ... (zero or more headers)
BLANK LINE:   \r\n
BODY:         <body bytes — only present if Content-Length > 0 or Transfer-Encoding: chunked>

HTTP Methods and Their Semantics

Your framework must understand the semantic contract of each HTTP method, because this determines how the router matches requests, how CSRF protection is applied, and what assumptions the ORM or view layer is allowed to make:

  • GET: Retrieves a resource. Must be safe (no side effects) and idempotent (calling it multiple times produces the same state). Browsers cache GET responses. GET requests must never have a meaningful body — they use query string parameters only.
  • POST: Submits data to be processed. Not idempotent. Used for form submissions, creating resources, RPC-style actions. Has a body; its type is described by the Content-Type header.
  • PUT: Replaces a resource entirely. Idempotent. If you PUT the same data twice, the state is the same as if you PUT it once.
  • PATCH: Partially updates a resource. Not necessarily idempotent — two identical PATCH requests may produce different results depending on the server's logic.
  • DELETE: Removes a resource. Should be idempotent.
  • HEAD: Identical to GET but the server must not send a response body. Used to check if a resource exists and inspect its metadata.
  • OPTIONS: Returns the communication options for the target resource. Your framework must respond to OPTIONS for CORS pre-flight requests.

HTTP Status Codes Your Framework Must Generate

Your response system must be capable of generating every status code in the relevant categories. Understanding which codes to use is not optional:

  • 2xx Success: 200 OK, 201 Created (after POST that creates a resource), 204 No Content (successful DELETE or PUT that returns nothing), 206 Partial Content (range requests for file streaming).
  • 3xx Redirection: 301 Moved Permanently (permanent redirect; browsers and crawlers cache it), 302 Found (temporary redirect; POST/Redirect/GET pattern uses this), 304 Not Modified (conditional GET when ETag or Last-Modified matches; no body sent).
  • 4xx Client Error: 400 Bad Request, 401 Unauthorized (authentication required), 403 Forbidden (authenticated but not permitted), 404 Not Found, 405 Method Not Allowed (the URL exists but not for this method; must include Allow header), 409 Conflict, 422 Unprocessable Entity (form validation failures), 429 Too Many Requests.
  • 5xx Server Error: 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout.

HTTP Headers: The Most Important Ones Your Framework Must Handle

Headers are the metadata of HTTP. Your framework does not just pass them through — it must actively process, validate, generate, and enforce many of them:

  • Content-Type: Tells the receiver the format of the body. Your parser uses this to decide how to decode the body (JSON, form data, multipart, etc.).
  • Content-Length: The exact size of the body in bytes. Your parser reads exactly this many bytes after the blank line.
  • Transfer-Encoding: chunked: Alternative to Content-Length for streaming responses where the total size is not known upfront. Each chunk is prefixed with its size in hexadecimal.
  • Authorization: Carries authentication credentials (Bearer token, Basic auth).
  • Cookie / Set-Cookie: Carries session cookies. The Set-Cookie header has a rich set of attributes (HttpOnly, Secure, SameSite, Path, Domain, Expires) that your session system must manage.
  • Accept: What content types the client can handle. Your view layer should respect this for content negotiation (returning JSON vs. HTML based on what the client asks for).
  • ETag / If-None-Match: Cache validation. If the ETag matches, return 304 and no body.
  • Last-Modified / If-Modified-Since: Alternative cache validation using timestamps.
  • Cache-Control: Directives for caching behaviour (no-store, no-cache, max-age, private, public).
  • Location: Used with 3xx responses to tell the client where to redirect.
  • X-Forwarded-For / X-Real-IP: Set by reverse proxies; your framework must trust these (conditionally) to determine the real client IP address.
  • Host: The domain the client is trying to reach. Essential for virtual hosting and CSRF validation.
  • Origin / Referer: Used by CSRF protection to verify that POST requests come from your domain.
  • Access-Control-Allow-Origin: CORS response header.

HTTP/2 and HTTP/3 (Brief Overview)

Your framework should be designed so that the HTTP version is an infrastructure concern, not an application concern. A view that returns a Response(200, "Hello") should not need to know whether it will be delivered over HTTP/1.1, HTTP/2, or HTTP/3:

  • HTTP/2: Multiplexes multiple requests over a single TCP connection using binary framing instead of plain text. Also supports server push. The application protocol (headers, methods, status codes, cookies) remains identical to HTTP/1.1.
  • HTTP/3: Built on QUIC (UDP-based) instead of TCP. Same application protocol as HTTP/1.1 and HTTP/2.

For your framework, this means the WSGI/ASGI adapter (Step 5) must absorb version differences. Your application code — every view, every model, every form — must be completely unaware of the transport version.

Checklist

  • Implement or integrate a correct HTTP/1.1 request line parser (method, URI, version).
  • Implement header parsing that handles multi-value headers (e.g., Set-Cookie can appear multiple times; Accept can list multiple MIME types comma-separated).
  • Implement correct body reading: Content-Length-based and Transfer-Encoding: chunked.
  • Define an enumeration of all HTTP methods your framework will support.
  • Define constants for all status codes your framework will generate.
  • Implement response serialization: status line + headers + blank line + body.
  • Handle HEAD requests: call the same view as GET but strip the body from the response.
  • Handle OPTIONS requests for CORS preflight.
  • Write a test for every edge case: missing Content-Length with a body, chunked encoding, multi-line header folding, percent-encoded URIs.

3. The TCP Server & Socket Management

What the TCP Server Does

The TCP server is the most fundamental component of the framework's infrastructure layer. It is the boundary between the network and your application code. Its responsibilities are tightly scoped: accept incoming connections, read raw bytes from those connections, hand the bytes to the HTTP parser, receive the serialized response bytes back, write them to the connection, and manage the connection's lifetime.

The TCP server has absolutely no knowledge of HTTP, routing, views, or any application-level concept. This is a textbook application of the Single Responsibility Principle: the server's one reason to change is if the way it handles socket I/O must change. It does not change because a new view was added or a new database backend was introduced.

The Lifecycle of a TCP Connection

A TCP connection goes through a well-defined lifecycle that your server must manage correctly:

FUNCTION tcp_server_lifecycle():
    // Phase 1: Bind and listen
    socket = create_tcp_socket()
    socket.bind(host, port)
    socket.listen(backlog=1024)   // How many unaccepted connections to queue
    
    // Phase 2: Accept loop — runs forever
    WHILE server_is_running:
        client_socket, client_address = socket.accept()  // Blocks until a client connects
        handle_connection(client_socket, client_address)  // Dispatch to a handler

    // Phase 3: Graceful shutdown
    // Stop accepting new connections
    // Wait for in-flight requests to complete
    // Close all sockets

Connection Handling Strategies

How handle_connection works determines the server's concurrency model. This is arguably the most important infrastructure decision you make, because it affects the entire programming model of your framework:

Forking (One Process Per Connection): The server forks a new OS process for each connection. The simplest model; no shared state issues; excellent isolation. Terrible scalability — processes are expensive to create; memory is not shared between workers.

Threading (One Thread Per Connection): The server spawns a new OS thread for each connection. Simpler than async; shares memory between threads (useful for caches). Threads have overhead; too many concurrent connections means too many threads; requires careful locking everywhere shared state is touched.

Thread Pool (Pre-Spawned Workers): A fixed pool of worker threads is created at startup. Incoming connections are placed in a queue; idle workers pick them up. This is the model used by classic WSGI servers like gunicorn with sync workers. The pool size is configurable; it sets the maximum concurrency. If all workers are busy, new requests queue (up to a limit) and then are rejected.

Event Loop (Non-Blocking I/O): A single thread runs an event loop. All I/O is non-blocking: instead of blocking a thread waiting for bytes to arrive, the socket is registered with the OS's I/O notification system (epoll on Linux, kqueue on macOS). When bytes arrive, the event loop is notified and dispatches to a callback or coroutine. This is the ASGI model. A single event loop thread can handle tens of thousands of concurrent connections, but all handlers must be written as coroutines — blocking I/O in a handler freezes the entire server.

Hybrid (Event Loop + Thread Pool): The event loop handles I/O; blocking operations (database queries, file reads) are offloaded to a thread pool. This gives async throughput without requiring pure async code everywhere.

Key Server Configuration Parameters

A production-ready server has a set of configuration knobs that directly affect reliability and performance. Your framework's CLI (Step 38) must expose these:

STRUCT ServerConfig:
    host:              string     // e.g., "0.0.0.0" to listen on all interfaces
    port:              integer    // e.g., 8000
    workers:           integer    // number of worker processes/threads
    worker_type:       ENUM { SYNC, ASYNC, HYBRID }
    backlog:           integer    // OS listen queue depth
    keepalive:         integer    // seconds to keep idle connections alive (HTTP/1.1)
    timeout:           integer    // seconds before a slow worker is killed
    max_request_size:  integer    // maximum body size in bytes (DoS protection)
    graceful_timeout:  integer    // seconds to wait for in-flight requests on shutdown
    ssl_cert:          path       // path to TLS certificate
    ssl_key:           path       // path to TLS private key

Graceful Shutdown

A production server must handle SIGTERM gracefully — it must not abruptly close connections mid-response. The graceful shutdown sequence is:

  1. Stop calling socket.accept() — no new connections are accepted.
  2. Signal all workers to finish their current request and then exit.
  3. Wait up to graceful_timeout seconds for workers to finish.
  4. Force-kill any workers that have not finished within the timeout.
  5. Close the listening socket.

Keep-Alive Connections

HTTP/1.1 defaults to persistent connections (Connection: keep-alive). After sending a response, the server does not close the socket — it keeps it open and waits for the next request. This dramatically reduces latency for pages that make many small HTTP requests (loading CSS, JS, images). Your server must manage the keep-alive timeout: if no new request arrives within the timeout window, close the socket.

Checklist

  • Implement create_tcp_socket(), bind(), listen(), and accept() with correct error handling.
  • Choose and implement your concurrency model (thread pool recommended for a first implementation).
  • Implement read_request(socket) that reads bytes until the HTTP header block ends (blank line \r\n\r\n).
  • Implement write_response(socket, response_bytes).
  • Implement keep-alive connection management with configurable timeout.
  • Implement maximum request size enforcement (return 413 Payload Too Large if exceeded).
  • Implement graceful shutdown signal handler.
  • Implement connection timeout: abort connections that take too long to send their request headers.
  • Log connection open/close events at DEBUG level with client IP and port.

4. The HTTP Parser — Turning Bytes into Structure

Why the HTTP Parser Deserves Its Own Component

The HTTP parser is a deceptively complex piece of engineering. Its job — converting a raw byte stream into a structured representation of an HTTP message — sounds simple but is filled with edge cases, security considerations, and correctness requirements. Security vulnerabilities like HTTP request smuggling exist specifically because different parsers interpret ambiguous HTTP constructs differently. Your parser must be strict, correct, and efficient.

The HTTP parser is a pure function from a standpoint of design purity: it takes bytes in and produces a structured ParsedRequest data object out. It has no knowledge of routing, views, models, or any application concept. This keeps it independently testable and replaceable.

The Parsing State Machine

The cleanest way to implement an HTTP parser is as a state machine. The parser maintains a current state and transitions between states as it reads bytes. This approach handles streaming input naturally — if the bytes arrive in multiple chunks, the state machine can be suspended and resumed without losing progress.

ENUM ParseState:
    READING_REQUEST_LINE       // Reading "GET /path HTTP/1.1\r\n"
    READING_HEADERS            // Reading header lines
    READING_BODY_CONTENT_LENGTH // Reading body using Content-Length
    READING_BODY_CHUNKED       // Reading chunked transfer encoding
    READING_CHUNK_SIZE         // Reading the hex size of the next chunk
    READING_CHUNK_DATA         // Reading chunk body bytes
    COMPLETE                   // Parsing is done
    ERROR                      // Invalid HTTP encountered

STRUCT HttpParser:
    state:             ParseState
    buffer:            ByteBuffer        // accumulates bytes read from socket
    parsed_method:     string
    parsed_path:       string
    parsed_version:    string
    parsed_headers:    MultiDict<string, string>  // allows duplicate keys
    parsed_body:       bytes
    content_length:    integer | null
    is_chunked:        boolean
    bytes_read_so_far: integer

Parsing the Request Line

The first line of an HTTP request encodes three pieces of information separated by single spaces, terminated by \r\n. Your parser must validate each piece independently:

The method must be one of a known set of uppercase ASCII strings. Reject anything else with a 400 Bad Request. Do not accept methods with lowercase letters or special characters.

The Request-URI is the URL the client is requesting. It may be an absolute path (/users/42/), an absolute URI (http://example.com/users/42/), or a bare asterisk (* for OPTIONS requests). It must be percent-decoded before being passed to the router, but the raw value must also be preserved for security logging purposes.

The HTTP version must be HTTP/1.0 or HTTP/1.1 for your initial implementation. Reject anything else.

FUNCTION parse_request_line(line: string):
    parts = line.split(' ')
    IF len(parts) != 3:
        RAISE ParseError("Invalid request line: expected METHOD PATH VERSION")
    
    method  = parts[0].strip()
    raw_uri = parts[1].strip()
    version = parts[2].strip()
    
    IF method NOT IN VALID_METHODS:
        RAISE ParseError(f"Unknown method: {method}")
    IF NOT version STARTS WITH "HTTP/":
        RAISE ParseError(f"Invalid HTTP version: {version}")
    
    path, query_string = split_path_and_query(raw_uri)
    path = percent_decode(path)
    
    RETURN ParsedRequestLine(method, path, query_string, version)

Parsing Headers

Headers are the most subtle part of HTTP to parse correctly. There are several important edge cases:

Header folding (obsolete but must be rejected or handled): Historical HTTP/1.1 allowed multi-line headers where continuation lines started with whitespace. RFC 7230 obsoleted this. Your parser should reject folded headers.

Multiple values for the same header name: Some headers can legally appear multiple times (Set-Cookie, Accept, Via). Others should not (e.g., Content-Length). Your parser must use a data structure that preserves all values — a MultiDict (a dictionary where each key can map to a list of values).

Header name case insensitivity: HTTP header names are case-insensitive. Content-Type and content-type are the same header. Your parser must normalize header names (e.g., to lowercase) before storing them.

Header value whitespace: Leading and trailing whitespace in header values should be stripped. Internal whitespace is significant and must be preserved.

Security limits on header count and size: Limit the total number of headers (e.g., 100 maximum) and the maximum size of any single header (e.g., 8 KB). Reject requests that exceed these limits to prevent DoS attacks.

FUNCTION parse_headers(raw_header_block: string) → MultiDict:
    headers = MultiDict()
    lines   = raw_header_block.split('\r\n')
    
    IF len(lines) > MAX_HEADER_COUNT:
        RAISE ParseError("Too many headers")
    
    FOR line IN lines:
        IF line is empty: BREAK   // Blank line signals end of headers
        
        IF NOT ':' IN line:
            RAISE ParseError(f"Malformed header line: {line}")
        
        name, _, value = line.partition(':')
        name  = name.strip().lower()
        value = value.strip()
        
        IF len(name) + len(value) > MAX_HEADER_SIZE:
            RAISE ParseError("Header too large")
        
        IF has_invalid_characters(name):
            RAISE ParseError(f"Invalid characters in header name: {name}")
        
        headers.add(name, value)
    
    RETURN headers

Parsing the Body

Body parsing depends on the transfer mechanism specified by the headers. There are exactly three scenarios your parser must handle:

No body: The request has no Content-Length header and no Transfer-Encoding header. This is the case for GET, HEAD, and DELETE requests. The body is empty bytes.

Content-Length body: Read exactly Content-Length bytes from the socket. If fewer bytes arrive (connection closed early), raise a parse error. If you read successfully, the body is complete. Never read more than Content-Length bytes — the extra bytes may belong to the next pipelined request.

Chunked Transfer Encoding: The body arrives in chunks. Each chunk is preceded by its size in hexadecimal, followed by \r\n, then the chunk data, then another \r\n. A chunk of size 0 signals the end. Optional trailing headers may follow the final zero-length chunk.

FUNCTION read_chunked_body(socket) → bytes:
    body = b""
    WHILE true:
        chunk_size_line = socket.readline()    // Read hex size
        chunk_size      = parse_hex(chunk_size_line.strip())
        
        IF chunk_size == 0:
            socket.readline()   // Consume the final \r\n
            BREAK               // End of chunked body
        
        IF chunk_size > MAX_CHUNK_SIZE:
            RAISE ParseError("Chunk too large")
        
        chunk_data = socket.read(chunk_size)
        socket.readline()  // Consume \r\n after chunk data
        body += chunk_data
    
    RETURN body

Query String Parsing

The query string is the portion of the URL after the ? character. It consists of key=value pairs separated by &. Both keys and values are percent-encoded. Your parser must decode them correctly, including handling + as a space (in application/x-www-form-urlencoded encoding only) and handling duplicate keys (which should produce a list of values, not silently overwrite earlier ones).

FUNCTION parse_query_string(raw_qs: string) → MultiDict:
    params = MultiDict()
    IF raw_qs is empty: RETURN params
    
    pairs = raw_qs.split('&')
    FOR pair IN pairs:
        IF '=' IN pair:
            key, _, value = pair.partition('=')
        ELSE:
            key   = pair
            value = ""
        
        key   = percent_decode(key.replace('+', ' '))
        value = percent_decode(value.replace('+', ' '))
        params.add(key, value)
    
    RETURN params

Checklist

  • Implement HttpParser as a state machine with well-defined states and transitions.
  • Implement parse_request_line() with strict method, URI, and version validation.
  • Implement parse_headers() using a MultiDict with case normalization.
  • Implement read_content_length_body(socket, length).
  • Implement read_chunked_body(socket) with chunk size parsing.
  • Implement parse_query_string() with percent-decoding and multi-value support.
  • Implement parse_multipart_body() for file upload handling (referenced in Step 41).
  • Implement parse_urlencoded_body() for form POST data.
  • Enforce limits on header count, header size, URI length, and body size.
  • Write security-focused tests: request smuggling attempts, header injection, extremely long headers, zero Content-Length with body, etc.

5. The WSGI / ASGI Application Interface

GoF Pattern: Adapter + State

The WSGI and ASGI specifications are interfaces — formal contracts between your framework and the web server that runs it. A server that speaks WSGI can run any framework that implements the WSGI interface (Django, Flask, Falcon), and a framework that implements WSGI can run on any compliant server (gunicorn, uWSGI, mod_wsgi). This standardization is one of the greatest architectural achievements in the Python web ecosystem, and it directly embodies the Dependency Inversion Principle: both the server and the framework depend on the WSGI abstraction, not on each other.

The Adapter pattern (GoF) is at work here: the WSGI/ASGI adapter converts between the server's native interface (the environ dict or scope dict) and your framework's internal interface (the Request and Response objects). The adapter is the only place in the entire framework that knows about the specifics of WSGI or ASGI.

WSGI in Detail

WSGI (Web Server Gateway Interface, PEP 3333) defines the interface for synchronous Python web applications. The entire WSGI specification boils down to a single callable:

FUNCTION wsgi_application(environ, start_response):
    // environ: a dictionary containing all HTTP request data and server variables
    // start_response: a callable used to begin the HTTP response
    
    // Your framework must:
    // 1. Read data from environ to build a Request object
    // 2. Process the request through its pipeline
    // 3. Call start_response(status_string, response_headers)
    // 4. Return an iterable of byte strings (the response body)
    
    response_body = b"Hello, World!"
    start_response("200 OK", [("Content-Type", "text/plain"),
                               ("Content-Length", str(len(response_body)))])
    RETURN [response_body]

The environ dictionary is densely packed with information. The most important keys your adapter must know how to read:

environ['REQUEST_METHOD']     → "GET", "POST", etc.
environ['PATH_INFO']          → "/users/42/" (decoded URL path)
environ['QUERY_STRING']       → "sort=name&page=2" (raw query string)
environ['SERVER_NAME']        → "example.com"
environ['SERVER_PORT']        → "8000"
environ['HTTP_HOST']          → "example.com:8000"
environ['HTTP_ACCEPT']        → "text/html,application/xhtml+xml"
environ['HTTP_COOKIE']        → "session=abc123; theme=dark"
environ['CONTENT_TYPE']       → "application/json"
environ['CONTENT_LENGTH']     → "42"
environ['wsgi.input']         → file-like object to read request body
environ['wsgi.errors']        → file-like object for error output
environ['wsgi.multithread']   → True if server may run multiple threads
environ['wsgi.multiprocess']  → True if server may run multiple processes
environ['wsgi.url_scheme']    → "http" or "https"
environ['wsgi.run_once']      → True if called only once per process (CGI)

// HTTP headers are added with HTTP_ prefix, dashes converted to underscores:
environ['HTTP_X_FORWARDED_FOR'] → "203.0.113.1"
environ['HTTP_AUTHORIZATION']   → "Bearer eyJhbGci..."

ASGI in Detail

ASGI (Asynchronous Server Gateway Interface) is the async evolution of WSGI. It supports long-lived connections like WebSockets and HTTP/2 server push in addition to regular HTTP. The interface is more complex because it must represent three connection types:

// ASGI application: a callable that takes scope, receive, send
ASYNC FUNCTION asgi_application(scope, receive, send):
    // scope: dict describing the connection (type, path, headers, etc.)
    // receive: async callable that yields events from the client
    // send: async callable to send events to the client
    
    IF scope['type'] == 'http':
        await handle_http(scope, receive, send)
    ELIF scope['type'] == 'websocket':
        await handle_websocket(scope, receive, send)
    ELIF scope['type'] == 'lifespan':
        await handle_lifespan(scope, receive, send)  // startup/shutdown events

The scope dict for HTTP requests:

scope['type']         → "http"
scope['asgi']         → {"version": "3.0"}
scope['http_version'] → "1.1" or "2"
scope['method']       → "GET"
scope['path']         → "/users/42/"
scope['query_string'] → b"sort=name&page=2" (bytes)
scope['headers']      → list of (name_bytes, value_bytes) tuples
scope['client']       → ("203.0.113.1", 52345) or None
scope['server']       → ("example.com", 8000)

The WSGI-to-Framework Adapter

Your framework's WSGI adapter is the bridge between the server's environ dict and your framework's internal Request object. It encapsulates all knowledge of the WSGI specification so nothing else in your framework ever needs to know what an environ dict is:

CLASS WsgiAdapter:
    FIELDS:
        framework_app:  ApplicationObject   // the framework's internal routing engine

    FUNCTION __call__(environ, start_response):
        // Convert WSGI environ to framework Request
        request = self.build_request(environ)
        
        // Run the full framework pipeline
        response = framework_app.handle(request)
        
        // Convert framework Response back to WSGI format
        status_string = f"{response.status_code} {response.status_phrase}"
        headers_list  = [(k, v) FOR k, v IN response.headers.items()]
        start_response(status_string, headers_list)
        RETURN [response.body_bytes]

    FUNCTION build_request(environ) → Request:
        method  = environ['REQUEST_METHOD']
        path    = environ.get('PATH_INFO', '/')
        qs      = environ.get('QUERY_STRING', '')
        body    = environ['wsgi.input'].read(content_length)
        headers = extract_headers_from_environ(environ)
        scheme  = environ.get('wsgi.url_scheme', 'http')
        RETURN Request(method, path, qs, headers, body, scheme)

The ASGI Lifespan Protocol

ASGI introduces a lifecycle protocol that WSGI lacks. The lifespan scope allows the server to notify the framework when the server is starting up and shutting down. This is the correct place to initialize the database connection pool, load configuration, and run startup checks:

ASYNC FUNCTION handle_lifespan(scope, receive, send):
    event = await receive()    // Wait for 'lifespan.startup' event
    
    IF event['type'] == 'lifespan.startup':
        TRY:
            await framework_app.startup()  // Initialize connections, caches, etc.
            await send({'type': 'lifespan.startup.complete'})
        EXCEPT Exception AS e:
            await send({'type': 'lifespan.startup.failed', 'message': str(e)})
    
    event = await receive()    // Wait for 'lifespan.shutdown' event
    
    IF event['type'] == 'lifespan.shutdown':
        await framework_app.shutdown()  // Clean up connections, flush caches
        await send({'type': 'lifespan.shutdown.complete'})

Checklist

  • Implement the WSGI callable interface (environ, start_response).
  • Implement WsgiAdapter.build_request(environ) that converts every environ key to the appropriate Request field.
  • Implement WsgiAdapter.__call__() that calls the framework pipeline and converts the response back.
  • Implement the ASGI interface (scope, receive, send) for HTTP connections.
  • Implement the ASGI lifespan protocol (startup and shutdown).
  • Implement the ASGI WebSocket handling scaffold (detailed in Step 37).
  • Write a test that runs your WSGI app through a real WSGI test client (a callable that simulates a server).
  • Support both WSGI and ASGI from the same framework by having two separate adapter classes.
  • Handle wsgi.errors write for uncaught exceptions.

6. The Application Object — The Framework's Front Door

GoF Pattern: Facade + Singleton + Observer

The Application object is the central hub of your entire framework. It is the one thing that the server knows about, and through it, the server gets access to the router, the middleware pipeline, the settings, the signal dispatcher, the plugin registry, and every other subsystem. It is a Facade (GoF) that presents a single, clean entry point to a complex subsystem — the framework itself.

The Application object is also the system's top-level Observer (GoF) in the sense that it wires up all the subsystems during initialization, connecting events to handlers, middleware to the pipeline, and routes to views. Once initialized, it is immutable — you do not add routes or middleware after the server has started handling requests. This immutability after startup is a critical safety property.

What the Application Object Knows About

The Application object holds references to all the major subsystems, but it is not their implementation — it delegates to them:

CLASS Application:
    FIELDS:
        settings:        Settings           // The configuration object (Step 31)
        router:          Router             // URL dispatch (Step 7)
        middleware_stack: MiddlewareStack   // The request pipeline (Step 10)
        registry:        AppRegistry        // Installed apps/plugins (Step 32)
        signal_dispatcher: SignalDispatcher // Signal pub/sub (Step 42)
        db_connections:  ConnectionPool    // ORM connections (Step 23)
        cache:           CacheBackend      // Caching layer (Step 30)
        template_engine: TemplateEngine    // Template rendering (Steps 12-15)
        static_files:    StaticFilesHandler // Static file serving (Step 33)

The Initialization Sequence

The order in which subsystems are initialized matters enormously, because later subsystems depend on earlier ones. Violating the initialization order leads to subtle bugs where a subsystem tries to use something that has not yet been configured. The canonical order for a Django-like framework is:

FUNCTION Application.initialize():
    // Phase 1: Settings first — everything else reads from settings
    settings.load()
    
    // Phase 2: Logging — so all subsequent phases can log errors
    logging.configure(settings.LOGGING)
    
    // Phase 3: App registry — discover installed apps and their models, views, etc.
    registry.populate(settings.INSTALLED_APPS)
    
    // Phase 4: Database — connect pool, run connection checks
    db_connections.initialize(settings.DATABASES)
    
    // Phase 5: Caching
    cache.initialize(settings.CACHES)
    
    // Phase 6: Template engine — load template directories from installed apps
    template_engine.initialize(settings.TEMPLATES, registry)
    
    // Phase 7: Signal connections — let installed apps connect their signal handlers
    registry.connect_signals(signal_dispatcher)
    
    // Phase 8: URL routing — collect URL patterns from installed apps
    router.load_urls(settings.ROOT_URLCONF)
    
    // Phase 9: Middleware — build the pipeline from the settings list
    middleware_stack.build(settings.MIDDLEWARE, self)
    
    // Phase 10: Static files
    static_files.configure(settings.STATIC_URL, settings.STATIC_ROOT)
    
    // Initialization complete — mark as ready
    self.ready = true
    signal_dispatcher.send("framework_ready", sender=self)

Handling Uncaught Exceptions

The Application object is also responsible for handling any exception that propagates all the way up through the view and middleware layers without being caught. This is the last line of defense before an unhandled error crashes the connection:

FUNCTION Application.handle(request) → Response:
    TRY:
        response = middleware_stack.process(request)
        RETURN response
    
    EXCEPT Http404 AS e:
        RETURN self.handle_404(request, e)
    
    EXCEPT PermissionDenied AS e:
        RETURN self.handle_403(request, e)
    
    EXCEPT Exception AS e:
        // Log the full traceback immediately
        logger.exception("Internal server error during request: " + request.path)
        
        // Send a signal so monitoring systems can be notified
        signal_dispatcher.send("request_exception", exception=e, request=request)
        
        IF settings.DEBUG:
            RETURN self.render_debug_page(request, e)  // Show traceback in browser
        ELSE:
            RETURN self.render_500_page(request)        // Generic error page

The Debug Mode vs. Production Mode Distinction

One of the most important design decisions in a web framework is what happens in debug mode vs. production mode. Debug mode must never be enabled in production because it exposes:

  • Full Python tracebacks with local variable values
  • Database query logs
  • Template context data
  • All loaded settings (including secrets)

Your framework must enforce this with a loud warning at startup if DEBUG=True and ALLOWED_HOSTS contains anything other than localhost.

Checklist

  • Define the Application class with fields for all major subsystems.
  • Implement initialize() in the exact order specified (settings → logging → registry → database → cache → templates → signals → urls → middleware → static).
  • Implement handle(request) that runs the middleware stack and catches uncaught exceptions.
  • Implement render_debug_page(request, exception) that shows the full traceback with source context.
  • Implement render_500_page(request) for production error pages (configurable by the user).
  • Implement render_404_page(request) and render_403_page(request).
  • Emit a framework_ready signal after initialization for plugins to hook into.
  • Warn loudly at startup if DEBUG=True in a production-like environment.
  • Implement a health check endpoint (/health/) that returns the application's readiness status.

7. The Router — URL Dispatch and Pattern Matching

GoF Pattern: Composite + Chain of Responsibility + Factory Method

The router is one of the most user-visible parts of the framework. Its API is what developers interact with daily when adding new URLs. Its implementation must be fast (routing happens on every single request), correct (it must never match the wrong view or extract wrong parameters), and flexible (it must support plain strings, parameterized paths, regex patterns, and nested URL includes).

The router's job can be summarized in one sentence: given an HTTP method and a URL path, find the registered view callable and extract any parameters embedded in the URL. If no match is found, it raises Http404. If a match is found but the method is not allowed, it raises MethodNotAllowed.

URL Pattern Types

A well-designed router must support at least three kinds of URL patterns, each with increasing power and complexity:

Literal patterns: Exact string matches. Fast because they can be stored in a hash table and looked up in O(1). Used for fixed URLs that have no variable components. Example: /about/, /api/v1/status/.

Parameterized patterns: URL segments that match any value and capture that value for the view. The pattern describes the shape of the segment. Example: /users/<user_id:int>/ captures the integer after /users/ and passes it to the view as user_id=42. Common parameter types include int, str, slug, uuid, path (which matches slashes).

Regex patterns: The most powerful but most expensive option. Any valid regular expression can be used as a URL pattern. Capture groups become view parameters. Example: /articles/(?P<year>[0-9]{4})/(?P<month>[0-9]{2})/.

The URLPattern and URLResolver Data Structures

The Composite pattern (GoF) is the natural fit for URL configuration because URL patterns can be nested — a URLResolver includes a set of sub-patterns (which can themselves be URLResolver instances), enabling the include() mechanism that allows app-level URL configs to be mounted under a prefix:

INTERFACE URLNode:                  // GoF Composite
    resolve(path: string, method: string) → ResolvedURL | null

CLASS URLPattern IMPLEMENTS URLNode:   // GoF Leaf
    FIELDS:
        pattern:   CompiledPattern      // The compiled pattern (regex or trie)
        view:      Callable             // The view function or class
        name:      string | null        // Optional name for reverse URL lookup
        methods:   Set<string>          // Allowed HTTP methods

    FUNCTION resolve(path, method):
        match = pattern.match(path)
        IF match is null: RETURN null
        IF method NOT IN methods AND methods is not empty:
            RAISE MethodNotAllowed(allowed=methods)
        RETURN ResolvedURL(view=view, kwargs=match.groups, url_name=name)

CLASS URLResolver IMPLEMENTS URLNode:  // GoF Composite
    FIELDS:
        prefix:   CompiledPattern    // The prefix pattern this resolver handles
        patterns: List<URLNode>      // Child patterns (can be URLPattern or URLResolver)
        namespace: string | null     // For namespaced URL reversal

    FUNCTION resolve(path, method):
        prefix_match = prefix.match(path)
        IF prefix_match is null: RETURN null
        
        remaining_path = path[len(prefix_match.match):]
        
        FOR child IN patterns:
            result = child.resolve(remaining_path, method)
            IF result is not null:
                IF namespace: result.add_namespace(namespace)
                RETURN result
        
        RETURN null

The Route Registration API

The user-facing API for registering routes must be clean, readable, and expressive. Most modern frameworks offer both a decorator API and a list-based configuration API:

// Decorator-style registration (popular in Flask-like frameworks)
FUNCTION route(path_pattern, methods=["GET"]):
    FUNCTION decorator(view_func):
        router.register(URLPattern(path_pattern, view_func, methods))
        RETURN view_func   // Return unmodified; decorator is for registration only
    RETURN decorator

// List-based configuration (Django style — more explicit, easier to test)
urlpatterns = [
    path("",                 views.home,      name="home"),
    path("users/",           views.user_list, name="user-list"),
    path("users/<int:pk>/",  views.user_detail, name="user-detail"),
    include("myapp.urls",    prefix="api/"),
]

The Route Matching Algorithm

For performance, the router should not iterate through every registered URL pattern on every request. A well-optimized router uses a radix tree (also called a patricia trie) to group patterns by common prefixes, enabling O(k) matching where k is the length of the URL path:

CLASS RadixTree:
    FIELDS:
        root: RadixNode

    FUNCTION insert(pattern: string, handler: URLPattern):
        // Walk the tree, splitting nodes where the pattern diverges
        // from existing nodes. At the leaf, store the handler.

    FUNCTION match(path: string) → (URLPattern, dict) | null:
        // Walk the tree following the path.
        // At parameter nodes (<int:pk>), match any value and capture it.
        // At the leaf, return the handler and captured parameters.

URL Reversal — Going from Name to URL

Every URL that your framework generates (redirects, links in templates) should be generated by a reversal function, not by hard-coding URL strings. This is essential for maintainability: if you rename /users/ to /accounts/, only the URL config changes, not every template and redirect in the codebase.

FUNCTION reverse(url_name: string, kwargs: dict = {}) → string:
    // Walk all registered URLPatterns, find the one with matching name
    // Substitute kwargs into the pattern to produce the URL
    pattern = find_pattern_by_name(url_name)
    IF pattern is null:
        RAISE NoReverseMatch(f"No URL pattern named '{url_name}'")
    RETURN pattern.reverse(kwargs)

// Template usage: {% url "user-detail" pk=42 %} → "/users/42/"

Checklist

  • Implement URLPattern (leaf) and URLResolver (composite) with a shared resolve() interface.
  • Implement pattern compilation for literal, parameterized, and regex patterns.
  • Implement the radix tree or equivalent optimized data structure for matching.
  • Implement the path() and re_path() registration functions.
  • Implement include() for mounting sub-URL configs under a prefix.
  • Implement URL namespaces for apps that may be installed multiple times.
  • Implement reverse(name, kwargs) for URL generation.
  • Handle MethodNotAllowed correctly: return 405 with an Allow header listing valid methods.
  • Handle Http404 for paths that match no pattern.
  • Implement URL reversal in the template engine as a built-in tag.
  • Implement trailing slash handling: optionally redirect /users → /users/.
  • Write performance tests: measure routing time for 1000 registered patterns.

8. The Request Object

GoF Pattern: Factory Method

The Request object is the framework's representation of an incoming HTTP request. It is created once per request by the WSGI/ASGI adapter, passed through the entire middleware pipeline, and eventually handed to the view. It is a rich data object that provides lazy access to all the information about the request: method, path, headers, body, parsed form data, cookies, session, uploaded files, and more.

The Request object's design is governed by two principles that pull against each other and must be carefully balanced:

Laziness: Parsing the body, decoding cookies, reading the session from a database, and parsing uploaded files are all expensive operations. If your request handler returns a JSON response in 5 milliseconds, it would be wasteful to have spent 10 milliseconds parsing a multipart body that the view never looked at. All expensive properties should be computed lazily — only when accessed for the first time, and then cached on the object.

Immutability: The request object represents an immutable fact — what the client sent. Views should be able to read the request but never modify it in ways that affect other middleware or the response pipeline. However, middleware legitimately needs to attach data to the request (e.g., the authentication middleware attaches request.user). The solution is a small, controlled set of mutable attributes that middleware is allowed to set, and the rest is immutable.

The Request Object Fields

CLASS Request:
    // Core HTTP fields — set directly from the parsed HTTP message, immutable
    FIELDS:
        method:          string          // "GET", "POST", etc.
        path:            string          // "/users/42/" (decoded)
        full_path:       string          // "/users/42/?sort=name"
        scheme:          string          // "http" or "https"
        host:            string          // "example.com:8000"
        headers:         ImmutableMultiDict  // All HTTP headers
        query_string:    string          // raw "sort=name&page=2"
        
    // Lazy-parsed properties
    LAZY PROPERTY query_params:
        // Parsed from query_string on first access
        RETURN parse_query_string(query_string)
    
    LAZY PROPERTY body:
        // Read from wsgi.input on first access
        RETURN wsgi_input.read(content_length)
    
    LAZY PROPERTY data:
        // Parse body based on Content-Type:
        // "application/json" → parse_json(body)
        // "application/x-www-form-urlencoded" → parse_urlencoded(body)
        // "multipart/form-data" → parse_multipart(body, content_type)
        RETURN dispatch_body_parser(content_type, body)
    
    LAZY PROPERTY files:
        // Only populated if content_type is multipart/form-data
        RETURN extract_files_from_multipart(body, content_type)
    
    LAZY PROPERTY cookies:
        // Parse the "Cookie" header into a dict
        RETURN parse_cookie_header(headers.get("cookie", ""))
    
    LAZY PROPERTY content_type:
        RETURN parse_content_type(headers.get("content-type", ""))
    
    // Middleware-injectable fields — mutable but only by middleware
    MUTABLE FIELDS:
        user:      User | AnonymousUser  // Set by AuthenticationMiddleware
        session:   Session              // Set by SessionMiddleware
        auth:      AuthToken | null     // Set by TokenAuthMiddleware
        META:      dict                 // Arbitrary metadata from middleware

Content Negotiation

A well-designed request object facilitates content negotiation — the HTTP mechanism by which the client tells the server what content formats it can accept, and the server responds in the best matching format. The Accept header contains a comma-separated list of MIME types with optional quality values (q=0.9).

FUNCTION accepts(request, content_type: string) → boolean:
    // Check if the client's Accept header includes this content type
    accepted = parse_accept_header(request.headers.get("accept", "*/*"))
    RETURN any_match(accepted, content_type)

FUNCTION preferred_content_type(request, offered: List<string>) → string:
    // Given a list of content types your view can produce,
    // return the one the client most prefers
    accepted = parse_accept_header(request.headers.get("accept", "*/*"))
    FOR mime_type IN ranked_by_quality(accepted):
        IF mime_type IN offered: RETURN mime_type
    RAISE NotAcceptable(f"Client does not accept any of: {offered}")

Request Security Properties

The Request object also exposes security-relevant properties that views and middleware use to make authorization decisions:

PROPERTY is_secure:
    // True if the request came over HTTPS
    // Must also check X-Forwarded-Proto when behind a trusted proxy
    RETURN scheme == "https" OR headers.get("x-forwarded-proto") == "https"

PROPERTY ip_address:
    // The client's IP address
    // Behind a trusted proxy, read from X-Forwarded-For
    IF settings.TRUSTED_PROXY_IPS includes server_ip:
        forwarded_for = headers.get("x-forwarded-for", "")
        RETURN parse_first_ip(forwarded_for)  // leftmost IP is the original client
    RETURN direct_client_ip

PROPERTY is_ajax:
    // Whether this is an XMLHttpRequest (legacy but still useful)
    RETURN headers.get("x-requested-with") == "XMLHttpRequest"

Checklist

  • Implement Request with all core HTTP fields set from the parsed HTTP message.
  • Implement lazy parsing for query_params, body, data, files, cookies.
  • Implement parse_content_type() to extract the MIME type and charset from the Content-Type header.
  • Implement body parsers for JSON, URL-encoded form data, and multipart form data.
  • Implement accepts(content_type) and preferred_content_type(offered) for content negotiation.
  • Implement is_secure, ip_address, and is_ajax security properties.
  • Implement build_absolute_uri(path) that constructs a full URL from the request's scheme and host.
  • Make the core HTTP fields immutable after construction.
  • Design the META dict API for middleware-injected data.
  • Test lazy properties: ensure body is not read if request.data is never accessed.

9. The Response Object

GoF Pattern: Factory Method + Builder

The Response object represents what the server will send back to the client. Unlike the Request, which is created by the framework and handed to the user's code, the Response is created by the user's view code and returned to the framework. The framework then runs it through the middleware pipeline (response phase) and serializes it.

Because Response objects are created very frequently (every request produces one), and because there are many specialized variants (HTML responses, JSON responses, redirects, file downloads, streaming responses), the design uses the Factory Method and Builder patterns to make the creation of the most common types concise.

The Base Response

CLASS Response:
    FIELDS:
        status_code:   integer           // e.g., 200, 302, 404
        headers:       MutableMultiDict  // Response headers
        body:          bytes | None      // Serialized response body
        cookies:       List<Cookie>      // Cookies to set on the client
        charset:       string            // Default: "utf-8"
    
    FUNCTION set_header(name, value):
        headers.set(name, value)
    
    FUNCTION delete_header(name):
        headers.delete(name)
    
    FUNCTION set_cookie(name, value, max_age, expires, path, domain,
                        secure, http_only, same_site):
        cookies.add(Cookie(name, value, ...))
    
    FUNCTION delete_cookie(name):
        // Set the cookie with max_age=0 and an expired date to tell
        // the browser to delete it
        set_cookie(name, "", max_age=0, expires=EPOCH)
    
    FUNCTION to_bytes() → bytes:
        status_line = f"HTTP/1.1 {status_code} {STATUS_PHRASES[status_code]}\r\n"
        header_bytes = serialize_headers(headers, cookies)
        RETURN status_line.encode() + header_bytes + b"\r\n" + (body or b"")

Specialized Response Subclasses

The Factory Method pattern gives rise to a family of pre-configured response subclasses. Each subclass represents one common use case and sets all the necessary headers automatically, removing boilerplate from view code:

CLASS HtmlResponse EXTENDS Response:
    CONSTRUCTOR(content: string, status=200):
        super(status_code=status)
        headers.set("Content-Type", "text/html; charset=utf-8")
        body = content.encode("utf-8")
        headers.set("Content-Length", str(len(body)))

CLASS JsonResponse EXTENDS Response:
    CONSTRUCTOR(data: any, status=200, encoder=None):
        super(status_code=status)
        headers.set("Content-Type", "application/json")
        body = json_encode(data, encoder=encoder).encode("utf-8")
        headers.set("Content-Length", str(len(body)))

CLASS RedirectResponse EXTENDS Response:
    CONSTRUCTOR(url: string, permanent=False):
        super(status_code=301 IF permanent ELSE 302)
        headers.set("Location", url)
        body = b""

CLASS NotFoundResponse EXTENDS Response:
    CONSTRUCTOR(message="Not Found"):
        super(status_code=404)
        headers.set("Content-Type", "text/plain")
        body = message.encode()

CLASS StreamingResponse EXTENDS Response:
    FIELDS:
        stream_generator: Generator<bytes>  // yields chunks
    
    FUNCTION to_bytes():
        // This response does not buffer — it yields chunks as they become available
        // The WSGI adapter yields from stream_generator directly
        RAISE NotImplementedError("Use iter_content() instead of to_bytes()")
    
    FUNCTION iter_content():
        FOR chunk IN stream_generator:
            YIELD chunk

CLASS FileResponse EXTENDS StreamingResponse:
    CONSTRUCTOR(filepath: Path, attachment_name=None):
        content_type = guess_content_type(filepath)
        headers.set("Content-Type", content_type)
        file_size = os.path.getsize(filepath)
        headers.set("Content-Length", str(file_size))
        
        IF attachment_name:
            headers.set("Content-Disposition",
                        f'attachment; filename="{attachment_name}"')
        
        stream_generator = read_file_in_chunks(filepath)

Response Caching Headers

Setting cache-related headers correctly on responses is one of the most impactful things a framework can do for application performance. Your framework should provide a clean API for setting these headers:

FUNCTION cache_response(response, max_age=0, private=True, must_revalidate=True):
    directives = []
    IF private:    directives.add("private")
    ELSE:          directives.add("public")
    IF max_age:    directives.add(f"max-age={max_age}")
    IF must_revalidate: directives.add("must-revalidate")
    response.set_header("Cache-Control", ", ".join(directives))

FUNCTION no_cache_response(response):
    response.set_header("Cache-Control", "no-store, no-cache, must-revalidate")
    response.set_header("Pragma", "no-cache")   // For HTTP/1.0 compatibility
    response.set_header("Expires", "0")

Checklist

  • Implement the base Response class with status code, headers (as MutableMultiDict), body, and cookies.
  • Implement HtmlResponse, JsonResponse, RedirectResponse subclasses.
  • Implement StreamingResponse for large file delivery and server-sent events.
  • Implement FileResponse with correct MIME type detection and Content-Length.
  • Implement set_cookie() with all RFC 6265 attributes: max_age, expires, path, domain, secure, httponly, samesite.
  • Implement delete_cookie().
  • Implement to_bytes() serialization including proper header serialization.
  • Implement cache header helpers: cache_for(seconds), no_cache(), set_etag().
  • Implement conditional response support: check If-None-Match header; return 304 if ETag matches.
  • Test that Content-Length is always set correctly for non-streaming responses.

10. The Middleware Pipeline

GoF Pattern: Chain of Responsibility + Decorator

The middleware system is perhaps the most powerful architectural feature of a web framework. It allows third parties — and your own framework modules — to transparently participate in every request and response without any coupling to the views or models that process the request. Authentication, session management, CSRF protection, GZip compression, security headers, rate limiting, and request logging are all implemented as middleware.

The Chain of Responsibility pattern (GoF) is the precise fit: each middleware in the chain has the opportunity to handle the request, modify it, pass it to the next middleware, receive the response, modify it, and return it. The chain ends with the actual view.

The Middleware Interface

Every middleware must implement the same interface. This is the key to the pipeline being composable: the chain-building code does not care what each middleware does, only that it implements process(request, next_handler):

INTERFACE Middleware:
    // Called for every request.
    // next_handler is a callable that represents the rest of the pipeline.
    // The middleware calls next_handler(request) to pass control forward,
    // or returns a Response directly to short-circuit the pipeline.
    FUNCTION process(request: Request, next_handler: Callable) → Response

CLASS LoggingMiddleware IMPLEMENTS Middleware:
    FUNCTION process(request, next_handler):
        start_time = now()
        logger.info(f"→ {request.method} {request.path}")
        
        response = next_handler(request)   // Pass to next middleware (or view)
        
        elapsed = now() - start_time
        logger.info(f"← {response.status_code} in {elapsed:.1f}ms")
        RETURN response

CLASS GzipMiddleware IMPLEMENTS Middleware:
    FUNCTION process(request, next_handler):
        response = next_handler(request)
        
        // Only compress if client accepts gzip and response is large enough
        IF "gzip" IN request.headers.get("accept-encoding", ""):
            IF response.content_length > GZIP_MIN_SIZE:
                response.body = gzip_compress(response.body)
                response.set_header("Content-Encoding", "gzip")
                response.set_header("Content-Length", str(len(response.body)))
        RETURN response

Building the Pipeline

The pipeline is built once at application startup, not on every request. The configuration specifies the middleware in order from outermost to innermost — the first middleware in the list is the outermost (it sees the request first and the response last).

CLASS MiddlewareStack:
    FIELDS:
        handler: Callable    // The final, fully-wrapped callable

    FUNCTION build(middleware_classes: List<Type>, view_router: Callable):
        // Start with the view router as the innermost handler
        handler = view_router
        
        // Wrap each middleware around the handler, from inside out
        // (reverse order so that the first middleware listed is outermost)
        FOR middleware_class IN REVERSED(middleware_classes):
            middleware = middleware_class()
            // Use a closure to capture the current handler
            previous_handler = handler
            handler = LAMBDA request: middleware.process(request, previous_handler)
        
        self.handler = handler

    FUNCTION process(request: Request) → Response:
        RETURN self.handler(request)

Visually, if settings.MIDDLEWARE = [SecurityMiddleware, SessionMiddleware, AuthMiddleware], the call stack looks like:

SecurityMiddleware.process()
    └── SessionMiddleware.process()
            └── AuthMiddleware.process()
                    └── router.dispatch() → view()
                    ← view returns Response
            ← AuthMiddleware augments response
        ← SessionMiddleware saves session to response
    ← SecurityMiddleware adds security headers
← Response returned to client

Short-Circuiting the Pipeline

A critical feature of the Chain of Responsibility is that any middleware can return a response without calling next_handler. This is called "short-circuiting" and is used for:

  • Rate limiting: If the client has exceeded their request limit, return 429 immediately without running the view.
  • CSRF protection: If a POST request fails the CSRF check, return 403 immediately.
  • IP blocking: If the client's IP is on a blocklist, return 403 immediately.
  • Maintenance mode: If the site is in maintenance mode, return 503 for all requests except the maintenance page.
CLASS MaintenanceModeMiddleware IMPLEMENTS Middleware:
    FUNCTION process(request, next_handler):
        IF settings.MAINTENANCE_MODE:
            IF request.path NOT IN settings.MAINTENANCE_EXEMPT_URLS:
                RETURN HtmlResponse(render_maintenance_page(), status=503)
        RETURN next_handler(request)

The Standard Middleware Suite

Your framework should ship with a set of pre-built middleware that covers the most common needs. Detailed implementations are covered in their respective steps:

Middleware Responsibility Step
SecurityMiddleware HTTPS redirect, HSTS, clickjacking protection 29
SessionMiddleware Load/save session on each request 27
AuthenticationMiddleware Attach request.user from session 26
CsrfMiddleware Validate CSRF tokens on unsafe methods 29
GzipMiddleware Compress responses with gzip 10
CommonMiddleware Trailing slash redirect, URL normalization 7
CacheMiddleware Full-page response caching 30
LocaleMiddleware Detect user's language preference 40
LoggingMiddleware Request/response logging 39
RateLimitMiddleware Request rate limiting per IP or user 29

Checklist

  • Define the Middleware interface with process(request, next_handler).
  • Implement MiddlewareStack.build() that wraps middleware in reverse order.
  • Implement LoggingMiddleware as the simplest example and test it first.
  • Implement GzipMiddleware with minimum-size threshold and content-type filtering.
  • Implement CommonMiddleware for trailing slash normalization.
  • Implement the short-circuit mechanism and test with MaintenanceModeMiddleware.
  • Ensure middleware can set attributes on the request object (like request.user).
  • Ensure middleware runs in the correct order: request phase = list order; response phase = reverse list order.
  • Implement async middleware variants for ASGI.
  • Profile middleware overhead: the entire middleware stack (with 10 middleware) should add less than 1ms of latency.

11. The View & Handler System

GoF Pattern: Template Method + Strategy + Factory Method

The view is the point where user code meets the framework. Everything the framework has done up to this point — parsing HTTP, routing, middleware — has been preparing to call the view. Everything after the view — serializing the response, running response-phase middleware — is preparing the view's return value for the client.

The framework must support two kinds of views, both with equal first-class support:

Function-based views (FBVs): A simple callable that takes a Request and returns a Response. Maximally flexible, minimal boilerplate, easy to read. The downside is that common patterns (authentication checking, permission checking, form handling) must be repeated or composed manually using decorators.

Class-based views (CBVs): A class with methods corresponding to HTTP methods (get, post, put, delete). The framework dispatches the request to the correct method automatically. Common patterns are implemented once in base classes and inherited. The Template Method pattern (GoF) governs CBVs: the base class defines the skeleton of the request-handling algorithm, and subclasses override specific steps.

Function-Based Views

A function-based view is the simplest possible view. It is just a Python (or language-equivalent) callable:

FUNCTION user_detail_view(request, user_id):
    IF request.method == "GET":
        user = User.objects.get(pk=user_id)
        RETURN JsonResponse(user.to_dict())
    ELSE:
        RETURN Response(status=405)

// Registered in urls.py:
path("users/<int:user_id>/", views.user_detail_view)

View Decorators — Composing Cross-Cutting Concerns

Decorators (GoF Decorator pattern applied in function composition) are the FBV mechanism for attaching cross-cutting concerns without modifying the view function itself:

// The login_required decorator is a higher-order function that wraps a view
FUNCTION login_required(redirect_url="/login/"):
    FUNCTION decorator(view_func):
        FUNCTION wrapper(request, **kwargs):
            IF NOT request.user.is_authenticated:
                RETURN RedirectResponse(redirect_url + "?next=" + request.path)
            RETURN view_func(request, **kwargs)
        RETURN wrapper
    RETURN decorator

// Usage:
@login_required
@permission_required("myapp.view_user")
FUNCTION user_list_view(request):
    users = User.objects.all()
    RETURN JsonResponse({"users": [u.to_dict() FOR u IN users]})

Class-Based Views — The Template Method Pattern

A CBV defines a class. The framework calls its dispatch() method, which is the Template Method (GoF): it defines the skeleton of the algorithm (check permissions, dispatch to the right HTTP method handler, handle errors) while allowing subclasses to override specific steps.

CLASS View:
    // Base class for all class-based views.
    // Implements the Template Method pattern.
    
    http_method_names: List<string> = ["get", "post", "put", "patch", "delete", "head", "options"]
    
    // The template method — defines the algorithm skeleton.
    // Subclasses should NOT override this; they override the individual method handlers.
    FUNCTION dispatch(request, **url_kwargs) → Response:
        method_name = request.method.lower()
        
        IF method_name NOT IN http_method_names:
            RETURN Response(status=405)
        
        handler = getattr(self, method_name, null)
        
        IF handler is null:
            RETURN self.http_method_not_allowed(request, **url_kwargs)
        
        RETURN handler(request, **url_kwargs)
    
    // Default implementation for unsupported methods
    FUNCTION http_method_not_allowed(request, **url_kwargs):
        allowed = [m.upper() FOR m IN http_method_names IF has_method(self, m)]
        response = Response(status=405)
        response.set_header("Allow", ", ".join(allowed))
        RETURN response
    
    // Subclasses implement these:
    FUNCTION get(request, **kwargs):    RAISE NotImplemented
    FUNCTION post(request, **kwargs):   RAISE NotImplemented
    // etc.

// How the router calls a CBV:
// It calls View.as_view(), which returns a function.
CLASS UserDetailView EXTENDS View:
    FUNCTION get(request, user_id):
        user = User.objects.get(pk=user_id)
        RETURN JsonResponse(user.to_dict())
    
    FUNCTION delete(request, user_id):
        User.objects.filter(pk=user_id).delete()
        RETURN Response(status=204)

Mixins — Composing Behaviour

Mixins allow CBV behaviour to be composed from multiple reusable pieces. Each mixin provides one specific behaviour (authentication check, permission check, form handling, etc.) by overriding specific hooks in the Template Method chain:

CLASS LoginRequiredMixin:
    login_url:        string = "/login/"
    redirect_field:   string = "next"
    
    FUNCTION dispatch(request, **kwargs):
        IF NOT request.user.is_authenticated:
            return redirect_to_login(request.path, self.login_url, self.redirect_field)
        RETURN super().dispatch(request, **kwargs)

CLASS PermissionRequiredMixin:
    permission_required: string | List<string>
    
    FUNCTION dispatch(request, **kwargs):
        IF NOT request.user.has_perms(self.permission_required):
            RAISE PermissionDenied
        RETURN super().dispatch(request, **kwargs)

// A view that requires login and a specific permission:
CLASS SecretDocumentView EXTENDS LoginRequiredMixin, PermissionRequiredMixin, View:
    permission_required = "myapp.view_secret_document"
    
    FUNCTION get(request, doc_id):
        doc = SecretDocument.objects.get(pk=doc_id)
        RETURN HtmlResponse(render_template("doc.html", {"doc": doc}))

Generic Class-Based Views

The most powerful application of CBVs is the generic view library — a set of pre-built views for the most common patterns in web development. Every CRUD interface, list page, and detail page follows one of these patterns:

CLASS ListView EXTENDS View:
    // Renders a list of objects from the database
    model:         Model class   // Which model to list
    template_name: string        // Template to render
    paginate_by:   integer       // Items per page; None = no pagination
    ordering:      string        // Default ordering
    
    FUNCTION get_queryset():
        RETURN model.objects.all().order_by(ordering)
    
    FUNCTION get(request):
        queryset = self.get_queryset()
        IF paginate_by:
            paginator = Paginator(queryset, paginate_by)
            page_number = request.query_params.get("page", 1)
            page_obj = paginator.get_page(page_number)
        context = self.get_context_data(object_list=page_obj)
        RETURN HtmlResponse(render_template(template_name, context))

CLASS DetailView EXTENDS View:
    model:         Model class
    template_name: string
    pk_url_kwarg:  string = "pk"
    
    FUNCTION get_object():
        pk = self.url_kwargs[pk_url_kwarg]
        RETURN model.objects.get(pk=pk)  // Raises Http404 if not found
    
    FUNCTION get(request, **url_kwargs):
        self.url_kwargs = url_kwargs
        obj = self.get_object()
        context = {"object": obj}
        RETURN HtmlResponse(render_template(template_name, context))

CLASS CreateView EXTENDS View:
    model:         Model class
    form_class:    Form class
    template_name: string
    success_url:   string
    
    FUNCTION get(request, **url_kwargs):
        form = form_class()
        RETURN HtmlResponse(render_template(template_name, {"form": form}))
    
    FUNCTION post(request, **url_kwargs):
        form = form_class(data=request.data)
        IF form.is_valid():
            obj = form.save()
            RETURN RedirectResponse(success_url)
        RETURN HtmlResponse(render_template(template_name, {"form": form}), status=422)

Checklist

  • Implement the base View class with dispatch() as the Template Method.
  • Implement http_method_not_allowed() that returns 405 with the Allow header.
  • Implement View.as_view() that returns a function the router can call.
  • Implement LoginRequiredMixin and PermissionRequiredMixin.
  • Implement ListView with pagination support.
  • Implement DetailView that raises Http404 for missing objects.
  • Implement CreateView, UpdateView, and DeleteView backed by the form system.
  • Implement @login_required, @permission_required, @require_http_methods decorators for FBVs.
  • Implement content negotiation in views: check Accept header and return JSON or HTML accordingly.
  • Ensure that exceptions raised in views propagate correctly to the Application object's exception handler.

12. The Template Engine — Lexer

GoF Pattern: Flyweight (Token Pool)

The template engine is one of the most intellectually interesting subsystems in a web framework. It is, in effect, a domain-specific language (DSL) compiler: it takes a string written in the template language (which contains HTML, template tags, filters, and variable expressions) and transforms it into a data structure that, when evaluated with a context, produces an HTML string. This process has three phases, each implemented as a separate component: the Lexer (Step 12), the Parser (Step 13), and the Renderer (Step 14).

The Lexer's job is tokenization: it reads the template source string character by character and produces a flat sequence of tokens. A token is the smallest meaningful unit in the template language — a chunk of plain HTML text, the opening delimiter of a tag, the content of a variable expression, or a comment.

Template Language Delimiters

Your template language must define distinct delimiters for its three special constructs. These should be visually distinct from HTML so they are easy to read and write:

Construct Opening Closing Example
Variable expression {{ }} {{ user.name }}
Template tag {% %} {% for item in items %}
Comment {# #} {# This is a comment #}

Everything else is a TEXT token — raw content to be output verbatim to the rendered result.

Token Types

ENUM TokenType:
    TEXT        // Raw HTML or text content — output verbatim
    VARIABLE    // Contents of {{ ... }} — evaluate and output
    BLOCK       // Contents of {% ... %} — a tag command
    COMMENT     // Contents of {# ... #} — ignored in output

STRUCT Token:
    type:     TokenType
    content:  string    // The token's text content (stripped of delimiters)
    source:   string    // The template file name (for error reporting)
    line:     integer   // The line number (for error reporting)
    column:   integer   // The column number

The Lexer State Machine

The Lexer is implemented as a state machine that scans the template source left-to-right. It alternates between two states: IN_TEXT (reading raw content until a delimiter is encountered) and IN_TOKEN (reading token content until the closing delimiter):

CLASS TemplateLexer:
    FIELDS:
        source:       string       // The template source code
        position:     integer      // Current position in source
        line:         integer      // Current line number
        column:       integer      // Current column number
        source_name:  string       // File name for error messages

    FUNCTION tokenize() → List<Token>:
        tokens = []
        
        WHILE position < len(source):
            // Look ahead for a token opening delimiter
            IF source starts with "{{" at position:
                tokens.add(read_variable_token())
            ELIF source starts with "{%" at position:
                tokens.add(read_block_token())
            ELIF source starts with "{#" at position:
                read_comment()   // Comments produce no token; just advance
            ELSE:
                tokens.add(read_text_token())
        
        RETURN tokens

    FUNCTION read_text_token() → Token:
        start = position
        start_line = line
        
        // Read characters until we hit a delimiter opening or end of source
        WHILE position < len(source):
            IF source[position:position+2] IN ["{{", "{%", "{#"]:
                BREAK
            IF source[position] == '\n':
                line++; column = 1
            ELSE:
                column++
            position++
        
        content = source[start:position]
        RETURN Token(TEXT, content, source_name, start_line)

    FUNCTION read_variable_token() → Token:
        position += 2    // Skip "{{"
        start_line = line
        
        end = find_closing_delimiter("}}")
        IF end == -1:
            RAISE TemplateSyntaxError(f"Unclosed variable tag at line {line}")
        
        content = source[position:end].strip()
        position = end + 2   // Skip "}}"
        RETURN Token(VARIABLE, content, source_name, start_line)

Whitespace Control

One of the most user-visible features of a template lexer is whitespace handling. Template tags on their own line often produce unwanted blank lines in the output. Most template engines offer a whitespace-stripping mechanism through a special token modifier. In Jinja2 this is done with - inside the delimiters ({%- if ... -%}). Your lexer must detect this modifier and mark the token accordingly, so the renderer can strip surrounding whitespace.

FUNCTION parse_whitespace_control(token_content: string) → (string, bool, Bool):
    strip_before = token_content.startswith('-')
    strip_after  = token_content.endswith('-')
    cleaned      = token_content.strip('-').strip()
    RETURN (cleaned, strip_before, strip_after)

Token Flyweights

Because many tokens are identical across different templates (e.g., the text <div>, <p>, </div>), the Flyweight pattern (GoF) can be applied to intern token content strings. This reduces memory usage when thousands of templates are loaded, because all tokens with the same content string share a single string object in memory. This is typically handled automatically by your language's string interning mechanism, but you should be aware of it and not break it by unnecessary string concatenation.

Checklist

  • Define the Token struct with type, content, source file, and line number.
  • Define TokenType as an enum with TEXT, VARIABLE, BLOCK, and COMMENT.
  • Implement TemplateLexer as a scanning state machine.
  • Implement read_text_token(), read_variable_token(), read_block_token(), and read_comment().
  • Track line and column numbers for every token (essential for error messages).
  • Implement whitespace control (- modifier in delimiters).
  • Handle nested delimiters correctly: {{ "{{literal}}" }} should not confuse the lexer.
  • Handle unterminated tokens with helpful error messages that include the file and line number.
  • Write exhaustive lexer tests: empty template, template with only text, only variables, all three types mixed, whitespace control.
  • Benchmark the lexer on a large template (10,000 lines): ensure it tokenizes in milliseconds.

13. The Template Engine — Parser & Abstract Syntax Tree

GoF Pattern: Composite (AST Nodes) + Visitor (AST Traversal)

The Parser takes the flat stream of tokens produced by the Lexer and produces an Abstract Syntax Tree (AST) — a hierarchical data structure that represents the logical structure of the template. While the Lexer knows nothing about what the tokens mean (it just classifies raw text), the Parser understands the grammar of the template language: that a for block tag must eventually be followed by an endfor block tag, that if can have elif and else clauses, and that an extends tag must appear at the very beginning of the template.

The AST is the crucial intermediate representation between parsing and rendering. It allows the template to be parsed once and rendered many times efficiently (caching the parsed tree and re-using it with different contexts). It also enables static analysis of templates — checking for undefined variables, missing end-tags, and other structural errors before the template is ever rendered.

The Node Hierarchy (GoF: Composite)

Every node in the AST implements the same TemplateNode interface. Leaf nodes produce output directly; composite nodes contain child nodes and produce output by rendering those children. This is the Composite pattern (GoF) in its purest form:

INTERFACE TemplateNode:                     // GoF Composite component
    render(context: RenderContext) → string  // Render this node to a string
    get_nodes_by_type(node_type) → List<TemplateNode>  // For tree traversal

CLASS TextNode IMPLEMENTS TemplateNode:     // GoF Leaf
    FIELDS:
        content: string     // The raw text content
    
    FUNCTION render(context):
        RETURN content      // Text nodes output their content verbatim

CLASS VariableNode IMPLEMENTS TemplateNode: // GoF Leaf
    FIELDS:
        expression: FilterExpression  // The parsed variable expression + filters
    
    FUNCTION render(context):
        value = expression.resolve(context)
        RETURN str(value) IF value is not null ELSE ""

CLASS BlockNode IMPLEMENTS TemplateNode:    // GoF Composite
    // A block node contains a list of child nodes
    FIELDS:
        children: List<TemplateNode>
    
    FUNCTION render(context):
        RETURN "".join(child.render(context) FOR child IN children)

CLASS ForNode IMPLEMENTS TemplateNode:      // GoF Composite
    FIELDS:
        loop_var:      string               // The loop variable name ("item")
        sequence_expr: FilterExpression     // The sequence to iterate ("items")
        body_nodes:    List<TemplateNode>   // Rendered for each iteration
        empty_nodes:   List<TemplateNode>   // Rendered if sequence is empty
    
    FUNCTION render(context):
        sequence = sequence_expr.resolve(context)
        IF sequence is empty OR null: RETURN render_children(empty_nodes, context)
        
        output = []
        FOR index, item IN enumerate(sequence):
            // Add loop metadata to context
            loop_context = context.new_child({
                loop_var: item,
                "forloop": {
                    "counter":      index + 1,
                    "counter0":     index,
                    "first":        index == 0,
                    "last":         index == len(sequence) - 1,
                    "parentloop":   context.get("forloop"),
                }
            })
            output.add(render_children(body_nodes, loop_context))
        RETURN "".join(output)

CLASS IfNode IMPLEMENTS TemplateNode:       // GoF Composite
    FIELDS:
        conditions_and_nodelists: List<(Expression, List<TemplateNode>)>
        // Each tuple is (condition, nodes_to_render_if_true)
        // The last tuple has condition=None for the else clause
    
    FUNCTION render(context):
        FOR condition, nodelist IN conditions_and_nodelists:
            IF condition is null OR condition.evaluate(context):
                RETURN render_children(nodelist, context)
        RETURN ""   // No condition matched and no else clause

The Parser State Machine

The parser processes tokens from the lexer in a single left-to-right pass. It uses a stack to handle nested structures (for loops inside if blocks, if blocks inside for loops):

CLASS TemplateParser:
    FIELDS:
        tokens:      List<Token>     // From the lexer
        position:    integer         // Current position in token list
        tag_library: TagLibrary      // Maps tag names to tag parsers

    FUNCTION parse(until: List<string> = []) → List<TemplateNode>:
        // until: stop parsing when a block tag with one of these names is found
        // (used for parsing the body of for/if/block tags)
        nodes = []
        
        WHILE position < len(tokens):
            token = tokens[position]
            position++
            
            IF token.type == TEXT:
                nodes.add(TextNode(token.content))
            
            ELIF token.type == VARIABLE:
                expression = parse_variable_expression(token.content)
                nodes.add(VariableNode(expression))
            
            ELIF token.type == BLOCK:
                tag_name, rest = split_first_word(token.content)
                
                IF tag_name IN until:
                    position--   // Put the token back; the caller handles it
                    RETURN nodes
                
                tag_parser = tag_library.get(tag_name)
                IF tag_parser is null:
                    RAISE TemplateSyntaxError(f"Unknown tag: '{tag_name}' at line {token.line}")
                
                node = tag_parser(self, rest, token)
                nodes.add(node)
        
        IF until is not empty:
            RAISE TemplateSyntaxError(f"Missing end tag for: {until}")
        
        RETURN nodes

Parsing the for Tag

The for tag parser is a good example of how complex tags are parsed recursively. It calls parser.parse() to parse its body, telling it to stop at empty or endfor tags:

FUNCTION parse_for_tag(parser, rest, token) → ForNode:
    // rest = "item in items" or "key, value in items.items"
    parts = rest.split(" in ", maxsplit=1)
    IF len(parts) != 2:
        RAISE TemplateSyntaxError(f"Malformed for tag: {rest}")
    
    loop_var     = parts[0].strip()
    sequence_str = parts[1].strip()
    sequence_expr = parse_variable_expression(sequence_str)
    
    // Parse the body nodes (until "empty" or "endfor")
    body_nodes = parser.parse(until=["empty", "endfor"])
    
    empty_nodes = []
    next_token = parser.current_token()
    IF split_first_word(next_token.content)[0] == "empty":
        parser.advance()   // Consume the "empty" tag
        empty_nodes = parser.parse(until=["endfor"])
    
    parser.advance()   // Consume the "endfor" tag
    RETURN ForNode(loop_var, sequence_expr, body_nodes, empty_nodes)

The FilterExpression

Variable expressions in templates can include a chain of filters applied to the variable: {{ user.name|lower|truncate:50 }}. The FilterExpression object represents this chain:

CLASS FilterExpression:
    FIELDS:
        variable:  LookupExpression     // The base variable ("user.name")
        filters:   List<(Filter, args)> // Ordered list of (filter_func, filter_args)
    
    FUNCTION resolve(context: RenderContext):
        value = variable.resolve(context)   // Look up "user.name" in the context dict
        FOR filter_func, args IN filters:
            value = filter_func(value, *args)  // Apply each filter in sequence
        RETURN value

CLASS LookupExpression:
    FIELDS:
        lookups: List<string>   // ["user", "name"] for "user.name"
    
    FUNCTION resolve(context):
        // Walk the lookup chain through the context
        current = context
        FOR part IN lookups:
            IF isinstance(current, dict):
                current = current.get(part)
            ELIF isinstance(current, object):
                current = getattr(current, part, null)
            IF current is null: RETURN null
        RETURN current

Checklist

  • Define the TemplateNode interface with render(context).
  • Implement TextNode, VariableNode, ForNode, IfNode as the core node types.
  • Implement the TemplateParser with recursive descent and the until parameter.
  • Implement parse_for_tag() and parse_if_tag() as examples of complex tag parsers.
  • Implement FilterExpression and LookupExpression with the dotted-path lookup.
  • Implement the TagLibrary as a registry that maps tag names to parser functions.
  • Detect and report structural errors: missing end tags, mismatched block tags, invalid variable expressions.
  • Cache parsed AST trees keyed by template name and modification time.
  • Implement get_nodes_by_type() for static analysis and extends block collection.
  • Write parser tests: nested for/if, for with empty clause, if/elif/else, comments inside blocks.

14. The Template Engine — Renderer & Context

GoF Pattern: Visitor + Strategy (Auto-Escaping)

The Renderer walks the AST produced by the Parser and produces the final HTML string. It is the most performance-sensitive part of the template engine because it is called on every request where a template is rendered. The Visitor pattern (GoF) governs the relationship between the renderer and the AST nodes: the renderer visits each node and produces output, without the nodes needing to know anything about the renderer.

The Render Context is the data structure that carries variable values from the view into the template. It must support:

  • Dictionary-style access: context["user_name"] looks up the variable.
  • Dotted-path access: The LookupExpression (Step 13) handles user.profile.bio.
  • Layering: New scopes (created by for loops and with blocks) push a new layer onto the context stack. When the scope ends, that layer is popped.
  • Built-in template variables: Things like request, user, csrf_token, STATIC_URL are automatically available in every template through a context processor system.

The Render Context Stack

CLASS RenderContext:
    FIELDS:
        stack: List<dict>   // A stack of dictionaries, most-specific first
    
    CONSTRUCTOR(initial: dict = {}):
        // The bottom of the stack is always populated by context processors
        builtin_context = {}
        FOR processor IN settings.TEMPLATE_CONTEXT_PROCESSORS:
            builtin_context.update(processor())
        stack = [builtin_context, initial]
    
    FUNCTION get(key, default=null):
        // Walk the stack from top (most specific) to bottom (least specific)
        FOR layer IN REVERSED(stack):
            IF key IN layer:
                RETURN layer[key]
        RETURN default
    
    FUNCTION new_child(bindings: dict) → RenderContext:
        // Create a new context with an additional scope layer pushed on top
        new_context = RenderContext.__new__()   // Don't call __init__; share the stack
        new_context.stack = self.stack + [bindings]
        RETURN new_context
    
    FUNCTION update(bindings: dict):
        // Push a new layer (used by {% with %} tag)
        stack.append(bindings)
    
    FUNCTION pop():
        // Pop the top layer (used by {% endwith %})
        stack.pop()

Context Processors — Universal Template Variables

Context processors are functions that return a dictionary of values that should be available in every template. They are called once per request and merged into the bottom of the context stack. The user never has to pass request, user, or STATIC_URL to render() explicitly — the context processors do it automatically:

// Built-in context processors (each returns a dict)

FUNCTION request_context_processor(request) → dict:
    RETURN {"request": request}

FUNCTION auth_context_processor(request) → dict:
    RETURN {
        "user": request.user,
        "perms": PermissionWrapper(request.user),
    }

FUNCTION csrf_context_processor(request) → dict:
    RETURN {"csrf_token": get_csrf_token(request)}

FUNCTION static_context_processor(request) → dict:
    RETURN {"STATIC_URL": settings.STATIC_URL}

// Configured in settings:
TEMPLATE_CONTEXT_PROCESSORS = [
    "framework.context_processors.request",
    "framework.context_processors.auth",
    "framework.context_processors.csrf",
    "framework.context_processors.static",
]

Auto-Escaping — The Single Most Important Security Feature of a Template Engine

Auto-escaping is non-negotiable. Without it, every template is a potential XSS (Cross-Site Scripting) vulnerability. If a user submits <script>alert('XSS')</script> as their username and the template renders {{ user.name }} without escaping, the attacker's script executes in every visitor's browser.

Auto-escaping means that every variable rendered by {{ ... }} has HTML special characters converted to their safe entity equivalents by default. The only way to render unescaped HTML is to explicitly mark it as safe using the safe filter or the mark_safe() function:

// HTML escaping mapping:
// & → &amp;    (must be first to avoid double-escaping)
// < → &lt;
// > → &gt;
// " → &quot;
// ' → &#x27;

FUNCTION html_escape(value: string) → SafeString:
    escaped = value
        .replace("&", "&amp;")
        .replace("<", "&lt;")
        .replace(">", "&gt;")
        .replace('"', "&quot;")
        .replace("'", "&#x27;")
    RETURN SafeString(escaped)   // Mark as already-escaped; don't escape again

CLASS SafeString EXTENDS string:
    // A string subclass that is marked as already HTML-safe.
    // The renderer checks isinstance(value, SafeString) before escaping.
    // If True, skip escaping. If False, escape.
    pass

FUNCTION mark_safe(value: string) → SafeString:
    // Used by developers who know their string is safe HTML.
    // This is a serious security responsibility — use only for trusted content.
    RETURN SafeString(value)

The renderer must check for SafeString before escaping:

FUNCTION render_variable_value(value) → string:
    IF isinstance(value, SafeString):
        RETURN value   // Already safe; output verbatim
    ELSE:
        RETURN html_escape(str(value))   // Escape before output

The Rendering Algorithm

CLASS TemplateRenderer:
    FUNCTION render(node_list: List<TemplateNode>, context: RenderContext) → string:
        parts = []
        FOR node IN node_list:
            TRY:
                rendered = node.render(context)
                parts.add(rendered)
            EXCEPT Exception AS e:
                IF settings.TEMPLATE_DEBUG:
                    RAISE   // In debug mode, let the exception propagate with traceback
                ELSE:
                    logger.warning(f"Template error in {node.source}:{node.line}: {e}")
                    parts.add("")   // In production, silently skip errored nodes
        RETURN "".join(parts)

Template Caching

The most expensive operation in template rendering is parsing. Parsing happens at the lexer and parser level and produces the AST. Once the AST is built, rendering is fast. Your template engine must cache parsed templates and only re-parse when the source file changes:

CLASS TemplateCache:
    FIELDS:
        cache: dict   // Maps template_name → (mtime, parsed_ast)
    
    FUNCTION get(template_name) → List<TemplateNode> | null:
        entry = cache.get(template_name)
        IF entry is null: RETURN null
        
        cached_mtime, parsed_ast = entry
        current_mtime = os.path.getmtime(template_path(template_name))
        
        IF current_mtime != cached_mtime:
            // Template file has changed; invalidate and re-parse
            cache.remove(template_name)
            RETURN null
        
        RETURN parsed_ast
    
    FUNCTION set(template_name, parsed_ast):
        mtime = os.path.getmtime(template_path(template_name))
        cache[template_name] = (mtime, parsed_ast)

Checklist

  • Implement RenderContext with a stack, get(), new_child(), update(), and pop().
  • Implement the context processor system and call processors on every render.
  • Implement html_escape() and SafeString for auto-escaping.
  • Ensure auto-escaping is enabled by default; provide {% autoescape off %} tag for opt-out.
  • Implement mark_safe() and the safe filter.
  • Implement TemplateRenderer.render() that calls node.render(context) for each node.
  • Implement TemplateCache with mtime-based invalidation.
  • Implement the top-level render_template(name, context_dict, request) function used by views.
  • Implement render_to_string(name, context_dict) for use outside of views (email templates, etc.).
  • Test auto-escaping: ensure <script> in user data is rendered as &lt;script&gt;.

15. The Template Engine — Tags, Filters & Template Inheritance

GoF Pattern: Composite (Template Inheritance) + Decorator (Filters)

This step covers the richest and most user-visible part of the template engine: the built-in tag and filter library, and the critically important template inheritance system. Template inheritance is what allows you to define a single base layout (the base.html with the site header, footer, navigation, and CSS imports) and have every other template extend it, overriding only the parts that differ.

Built-in Filters

A filter is a function that transforms a value. Filters are applied using the pipe character in variable expressions: {{ article.created_at|date:"Y-m-d" }}. Every filter has the signature filter_func(value, *args) → any. The filter library should be extensible — user code and plugins can register new filters.

// The filter registry:
FILTER_LIBRARY: dict<string, Callable> = {}

FUNCTION register_filter(name: string):
    FUNCTION decorator(func):
        FILTER_LIBRARY[name] = func
        RETURN func
    RETURN decorator

// Core built-in filters:

@register_filter("lower")
FUNCTION filter_lower(value): RETURN str(value).lower()

@register_filter("upper")
FUNCTION filter_upper(value): RETURN str(value).upper()

@register_filter("length")
FUNCTION filter_length(value): RETURN len(value)

@register_filter("default")
FUNCTION filter_default(value, default_value):
    RETURN value IF (value is not null AND value != "") ELSE default_value

@register_filter("truncate")
FUNCTION filter_truncate(value, length, suffix="..."):
    s = str(value)
    IF len(s) <= length: RETURN s
    RETURN s[:length - len(suffix)] + suffix

@register_filter("date")
FUNCTION filter_date(value, format_string):
    IF isinstance(value, datetime): RETURN value.strftime(format_string)
    RETURN str(value)

@register_filter("join")
FUNCTION filter_join(value, separator=", "):
    RETURN separator.join(str(item) FOR item IN value)

@register_filter("safe")
FUNCTION filter_safe(value): RETURN mark_safe(str(value))

@register_filter("escape")
FUNCTION filter_escape(value): RETURN html_escape(str(value))

@register_filter("json")
FUNCTION filter_json(value, indent=null):
    RETURN mark_safe(json_encode(value, indent=indent))  // Safe: developer controls this

@register_filter("pluralize")
FUNCTION filter_pluralize(value, suffix="s"):
    singular, _, plural = suffix.partition(",")
    RETURN singular IF int(value) == 1 ELSE (plural OR singular + "s")

The with Tag — Creating Template Scopes

The with tag allows binding expensive expressions to shorter variable names within a block. It uses context.update() / context.pop() to create a new scope:

CLASS WithNode IMPLEMENTS TemplateNode:
    FIELDS:
        bindings:  dict<string, FilterExpression>  // variable_name → expression
        body_nodes: List<TemplateNode>
    
    FUNCTION render(context):
        resolved_bindings = {}
        FOR name, expr IN bindings.items():
            resolved_bindings[name] = expr.resolve(context)
        
        context.update(resolved_bindings)
        output = render_children(body_nodes, context)
        context.pop()
        RETURN output

Template Inheritance — The extends and block System

Template inheritance is the most architecturally complex feature of the template engine. Understanding it requires understanding what extends, block, and include actually do:

  • {% extends "base.html" %}: This template is a child of base.html. The child template's entire content is essentially a collection of block overrides. Anything in the child template that is not inside a {% block %} is ignored.
  • {% block content %} ... {% endblock %}: Defines a named block. In the base template, this defines a placeholder with a default value. In a child template, this defines the content that replaces the placeholder.
  • {% include "partials/header.html" %}: Renders another template and inserts its output at this point. Unlike extends, include is a simple insertion — no inheritance is involved.

The implementation of extends requires two passes through the template tree:

First pass (at parse time): Collect all block nodes from the child template into a dictionary keyed by block name. If the parent template also extends a grandparent, recurse up the chain.

Second pass (at render time): Start rendering the base (root) template. When a block node is encountered in the base template, check if the child has overridden it. If yes, render the child's version. If no, render the base template's default content.

CLASS BlockNode IMPLEMENTS TemplateNode:
    FIELDS:
        name:       string
        body_nodes: List<TemplateNode>
        parent:     BlockNode | null    // The parent block (for {{ block.super }})
    
    FUNCTION render(context):
        // The current block override is injected into the context by ExtendsNode
        override = context.get(f"__block_{name}")
        IF override is not null:
            // Temporarily set this block as the parent for {{ block.super }} resolution
            context.update({f"__block_{name}_parent": self})
            result = render_children(override.body_nodes, context)
            context.pop()
            RETURN result
        ELSE:
            RETURN render_children(body_nodes, context)

CLASS ExtendsNode IMPLEMENTS TemplateNode:
    FIELDS:
        parent_name:  FilterExpression   // The parent template name
        child_blocks: dict<string, BlockNode>  // All blocks from the child template
    
    FUNCTION render(context):
        parent_name_resolved = parent_name.resolve(context)
        parent_template      = template_loader.load(parent_name_resolved)
        
        // Inject child block overrides into context
        FOR block_name, block_node IN child_blocks.items():
            context.update({f"__block_{block_name}": block_node})
        
        // Render the parent template — block nodes will find their overrides in context
        RETURN render_children(parent_template.node_list, context)

The include Tag

CLASS IncludeNode IMPLEMENTS TemplateNode:
    FIELDS:
        template_expr:    FilterExpression   // May be a variable ("include var")
        extra_context:    dict               // Optional extra context variables
        isolated_context: boolean            // If True, don't pass parent context
    
    FUNCTION render(context):
        template_name   = template_expr.resolve(context)
        included_template = template_loader.load(template_name)
        
        IF isolated_context:
            child_context = RenderContext(extra_context)
        ELSE:
            child_context = context.new_child(extra_context)
        
        RETURN render_children(included_template.node_list, child_context)

Multi-Level Template Inheritance

A real application typically has three or more levels of inheritance:

base.html           ← Defines html, head, body, main header, footer
  └── layout.html   ← Extends base; defines sidebar, main-content columns
        └── article_detail.html  ← Extends layout; fills main-content with article

base.html:
    {% block body %}
        <header>...</header>
        {% block content %}{% endblock %}
        <footer>...</footer>
    {% endblock %}

layout.html:
    {% extends "base.html" %}
    {% block content %}
        <aside>{% block sidebar %}{% endblock %}</aside>
        <main>{% block main %}{% endblock %}</main>
    {% endblock %}

article_detail.html:
    {% extends "layout.html" %}
    {% block main %}
        <article>{{ article.body|safe }}</article>
    {% endblock %}

Checklist

  • Implement the filter registry with register_filter().
  • Implement all core built-in filters: lower, upper, length, default, truncate, date, join, safe, escape, json, pluralize, floatformat, linebreaks, urlize.
  • Implement the with tag using context push/pop.
  • Implement the block tag and BlockNode.
  • Implement the extends tag and ExtendsNode with recursive parent loading.
  • Implement {{ block.super }} — rendering a block's parent content inside the child override.
  • Implement the include tag with optional isolated context.
  • Implement the load tag for loading custom tag/filter libraries.
  • Implement the url tag using the router's reverse() function.
  • Implement the static tag for generating static asset URLs.
  • Implement the csrf_input tag that renders the CSRF hidden input field.
  • Write inheritance tests: two-level, three-level, block.super, sibling blocks, block in included template.

16. The ORM — Database Abstraction & Connection Management

GoF Pattern: Abstract Factory + Adapter + Facade

The Object-Relational Mapper (ORM) is the most complex subsystem in any full-stack web framework. It is responsible for mapping between the relational world of tables, rows, and SQL queries and the object-oriented world of classes, instances, and method calls. A well-designed ORM makes database interactions feel natural and idiomatic in the host language while remaining efficient, secure, and database-agnostic.

The ORM's database abstraction layer is built on the Abstract Factory pattern (GoF): there is one abstract factory per database backend (PostgreSQL, MySQL, SQLite, Oracle), and each factory produces a suite of related objects (connection, cursor, query compiler, schema editor) that work together. Swapping from PostgreSQL to MySQL means substituting one factory for another — the upper layers of the ORM are completely unaffected.

The Database Backend Interface

The backend interface is the contract that every database driver adapter must fulfill. It is intentionally narrow — it covers only what is truly database-specific and cannot be shared:

INTERFACE DatabaseBackend:      // GoF Abstract Factory
    // Connection management
    create_connection(config: DatabaseConfig) → Connection
    
    // Schema introspection (for migrations)
    get_table_names(connection) → List<string>
    get_column_info(connection, table_name) → List<ColumnInfo>
    get_index_info(connection, table_name) → List<IndexInfo>
    get_constraint_info(connection, table_name) → List<ConstraintInfo>
    
    // SQL dialect specifics
    quote_name(name: string) → string     // Wraps identifiers: "users" or `users`
    get_placeholder() → string            // "?" for SQLite, "%s" for psycopg2
    supports_returning() → boolean        // Can INSERT ... RETURNING ... ?
    supports_json_field() → boolean
    max_varchar_length() → integer
    
    // Query compiler (GoF Factory Method)
    get_compiler() → SqlCompiler
    
    // Schema operations
    get_schema_editor() → SchemaEditor

CLASS PostgreSQLBackend IMPLEMENTS DatabaseBackend:
    quote_name(name) → RETURN f'"{name}"'
    get_placeholder() → RETURN "%s"
    supports_returning() → RETURN true

CLASS SQLiteBackend IMPLEMENTS DatabaseBackend:
    quote_name(name) → RETURN f'"{name}"'
    get_placeholder() → RETURN "?"
    supports_returning() → RETURN false  // SQLite < 3.35 doesn't support RETURNING

CLASS MySQLBackend IMPLEMENTS DatabaseBackend:
    quote_name(name) → RETURN f'`{name}`'
    get_placeholder() → RETURN "%s"
    supports_returning() → RETURN false

The Connection Abstraction

A Connection object wraps a single underlying database driver connection. It provides a consistent interface regardless of which database driver is being used (psycopg2, sqlite3, pymysql, etc.). The Adapter pattern (GoF) is used here: the raw driver connection is adapted to the framework's Connection interface:

CLASS Connection:
    FIELDS:
        backend:   DatabaseBackend
        raw_conn:  driver_connection   // The actual psycopg2/sqlite3/pymysql connection
        config:    DatabaseConfig
        in_transaction: boolean
        savepoint_counter: integer

    FUNCTION execute(sql: string, params: list = []) → Cursor:
        log_query(sql, params)    // For debug logging
        cursor = raw_conn.cursor()
        cursor.execute(sql, params)
        RETURN Cursor(cursor, backend)

    FUNCTION begin_transaction():
        IF NOT in_transaction:
            execute("BEGIN")
            in_transaction = true

    FUNCTION commit():
        raw_conn.commit()
        in_transaction = false

    FUNCTION rollback():
        raw_conn.rollback()
        in_transaction = false

    FUNCTION create_savepoint(name: string):
        execute(f"SAVEPOINT {name}")
        savepoint_counter++

    FUNCTION release_savepoint(name: string):
        execute(f"RELEASE SAVEPOINT {name}")

    FUNCTION rollback_to_savepoint(name: string):
        execute(f"ROLLBACK TO SAVEPOINT {name}")

    FUNCTION close():
        raw_conn.close()

Database Configuration

STRUCT DatabaseConfig:
    ENGINE:   string    // "framework.backends.postgresql"
    NAME:     string    // Database name (or file path for SQLite)
    HOST:     string    // "" means localhost
    PORT:     integer   // 5432 for PostgreSQL, 3306 for MySQL, 0 for SQLite
    USER:     string
    PASSWORD: string
    OPTIONS:  dict      // Driver-specific options
    
    CONN_MAX_AGE:    integer  // Seconds to reuse a connection (0 = close after each request)
    CONN_HEALTH_CHECKS: bool  // Check connection health before using
    
    ATOMIC_REQUESTS: bool     // Wrap every request in a transaction
    AUTOCOMMIT:     bool      // Default: True (each query is its own transaction)

Checklist

  • Define the DatabaseBackend abstract factory interface.
  • Implement PostgreSQLBackend, SQLiteBackend, and MySQLBackend.
  • Implement the Connection wrapper with execute, transaction, and savepoint support.
  • Implement DatabaseConfig with all connection parameters.
  • Implement a backend loader that instantiates the correct backend from the ENGINE setting string.
  • Add query logging in debug mode: log every SQL statement with its parameters and execution time.
  • Implement health check: test the connection before using it from the pool.
  • Write integration tests for each backend: create table, insert, select, update, delete, transaction, rollback.

17. The ORM — Model Definition & Metaclass System

GoF Pattern: Template Method + Factory Method + Flyweight (Field Instances)

The Model class is the user-facing API of the ORM. A developer defines a model by creating a class that inherits from Model and declares fields as class attributes. The framework then does a remarkable amount of work transparently: it discovers the fields, determines the database table name, builds SQL for creating and querying the table, and wires up many convenience methods like save(), delete(), objects.all(), and objects.filter().

The mechanism that makes this work — reading the class body at class-creation time and transforming it — is the metaclass. When Python (or any language with this capability) creates a class, the metaclass's __new__ or __init_subclass__ method is called with the class name, bases, and namespace (the dict of everything defined in the class body). The ORM's metaclass uses this opportunity to inspect the class body, extract field declarations, and augment the class with generated methods and metadata.

The ModelBase Metaclass

METACLASS ModelBase:
    FUNCTION __new__(meta, name, bases, namespace):
        // Step 1: Separate field objects from non-field attributes
        fields = {}
        non_field_attrs = {}
        
        FOR attr_name, attr_value IN namespace.items():
            IF isinstance(attr_value, Field):
                fields[attr_name] = attr_value
                attr_value.contribute_to_class(name, attr_name)  // Tell the field its name
            ELSE:
                non_field_attrs[attr_name] = attr_value
        
        // Step 2: Create the class normally (without the field attrs as raw class attrs)
        cls = type.__new__(meta, name, bases, non_field_attrs)
        
        // Step 3: Store the field definitions in class-level metadata
        cls._meta = Options(
            model_name = name.lower(),
            db_table   = non_field_attrs.get("__table__") OR to_snake_case(name) + "s",
            fields     = fields,
            ordering   = non_field_attrs.get("Meta", {}).get("ordering", []),
            unique_together = non_field_attrs.get("Meta", {}).get("unique_together", []),
        )
        
        // Step 4: Install field descriptors on the class
        // (so that model.field_name = value triggers validation and tracking)
        FOR field_name, field_obj IN fields.items():
            descriptor = FieldDescriptor(field_name, field_obj)
            setattr(cls, field_name, descriptor)
        
        // Step 5: Attach the default manager
        IF NOT has_manager(cls):
            cls.objects = Manager(cls)
        
        // Step 6: Register the model in the app registry
        app_registry.register_model(cls)
        
        RETURN cls

The Options Object (_meta)

Every Model class has a _meta attribute (an Options instance) that is the framework's canonical source of truth about a model's structure. It is used extensively by the migration system, the SQL compiler, the admin interface, and the form generator:

CLASS Options:
    FIELDS:
        model_name:      string        // "user"
        db_table:        string        // "myapp_user"
        app_label:       string        // "myapp"
        fields:          OrderedDict<string, Field>  // All declared fields
        pk:              Field         // The primary key field
        ordering:        List<string>  // Default ordering e.g., ["-created_at"]
        unique_together: List<tuple>   // e.g., [("first_name", "last_name")]
        indexes:         List<Index>
        constraints:     List<Constraint>
        abstract:        bool          // True if this is an abstract base class
        proxy:           bool          // True if this is a proxy model
        managed:         bool          // True if the ORM manages the schema
        
        // Relationship reverse accessors (populated after all models are loaded)
        related_objects: List<RelatedObject>

    FUNCTION get_field(name: string) → Field:
        IF name IN fields: RETURN fields[name]
        RAISE FieldDoesNotExist(f"'{model_name}' has no field '{name}'")
    
    FUNCTION get_all_field_names() → List<string>:
        RETURN list(fields.keys())

Field Descriptors — The __get__/__set__ Protocol

A field descriptor is the mechanism that makes instance.field_name = value work like a Python attribute assignment but with ORM magic underneath. When a developer writes user.name = "Alice", the descriptor's __set__ method is called. It validates the value, tracks that the field has been modified (for partial updates), and stores the value in the instance's __dict__:

CLASS FieldDescriptor:
    FIELDS:
        field_name: string
        field:      Field
    
    FUNCTION __get__(instance, owner):
        IF instance is null:
            RETURN self   // Accessing via class, not instance; return the descriptor itself
        RETURN instance.__dict__.get(field_name, field.default)
    
    FUNCTION __set__(instance, value):
        // Validate and convert the value
        converted = field.to_python(value)
        field.validate(converted)
        
        // Track the modification for partial updates
        IF NOT hasattr(instance, "_changed_fields"):
            instance._changed_fields = set()
        IF instance.__dict__.get(field_name) != converted:
            instance._changed_fields.add(field_name)
        
        instance.__dict__[field_name] = converted

The Model Instance Lifecycle

CLASS Model:
    // Every model subclass inherits these methods.
    // The metaclass has already configured _meta.
    
    CONSTRUCTOR(**kwargs):
        // Set field values from keyword arguments
        FOR field_name, value IN kwargs.items():
            setattr(self, field_name, value)  // Uses FieldDescriptor.__set__
        
        // Set defaults for unspecified fields
        FOR field_name, field IN _meta.fields.items():
            IF field_name NOT IN kwargs AND field.has_default():
                setattr(self, field_name, field.get_default())
        
        self._state = ModelState(adding=True, db=None)
        self._changed_fields = set()
    
    FUNCTION save(force_insert=False, force_update=False, update_fields=None):
        // Validate all fields before saving
        self.full_clean()
        
        IF self._state.adding OR force_insert:
            self._do_insert()
        ELSE:
            self._do_update(update_fields)
        
        self._state.adding = False
        self._changed_fields.clear()
    
    FUNCTION _do_insert():
        sql, params = compiler.as_sql_insert(self)
        cursor = connection.execute(sql, params)
        
        // If the backend supports RETURNING, extract the new PK
        IF backend.supports_returning():
            self.pk = cursor.fetchone()[0]
        ELSE:
            self.pk = cursor.lastrowid
    
    FUNCTION _do_update(update_fields=None):
        IF update_fields:
            fields_to_update = [_meta.fields[f] FOR f IN update_fields]
        ELSE:
            fields_to_update = [f FOR f IN _meta.fields.values() IF NOT f.primary_key]
        
        sql, params = compiler.as_sql_update(self, fields_to_update)
        connection.execute(sql, params)
    
    FUNCTION delete():
        // Collect related objects that must also be deleted (CASCADE)
        collector = DeletionCollector(connection)
        collector.collect([self])
        collector.delete()
    
    FUNCTION full_clean():
        FOR field_name, field IN _meta.fields.items():
            value = getattr(self, field_name)
            field.clean(value, self)   // May raise ValidationError

Checklist

  • Implement ModelBase metaclass with field extraction and class augmentation.
  • Implement Options (_meta) with all model metadata.
  • Implement FieldDescriptor with __get__ / __set__ and changed-field tracking.
  • Implement Model.__init__() with keyword argument field setting and default application.
  • Implement Model.save() with insert/update discrimination based on _state.adding.
  • Implement Model.delete() with cascade collection.
  • Implement Model.full_clean() that validates all fields.
  • Register every concrete (non-abstract) model in the application registry.
  • Implement abstract model base classes that are inherited but not registered in the schema.
  • Implement proxy models that reuse a parent table but add Python-level behavior.

18. The ORM — Field Types, Validators & Descriptors

GoF Pattern: Strategy (Validators) + Flyweight (Shared Field Instances)

Fields are the atomic unit of the ORM model definition. Every column in a database table corresponds to a field declaration on a Model class. The field system has three distinct responsibilities that must be clearly separated: the database representation (what column type to generate in SQL), the Python representation (what Python type to return when reading the field), and the validation (what constraints the value must satisfy before being saved).

The Field Interface

INTERFACE Field:
    // Identity
    name:             string       // Set by the metaclass when the field is attached
    column:           string       // Database column name (defaults to name)
    primary_key:      boolean
    unique:           boolean
    null:             boolean      // NULL allowed in database
    blank:            boolean      // Empty value allowed in validation
    default:          any | MISSING
    editable:         boolean
    verbose_name:     string       // Human-readable label
    help_text:        string       // Longer description for forms and docs
    
    // Type conversion — the core responsibility of every field
    FUNCTION to_python(value: any) → any
        // Convert a raw value (from the database, a form, or user code) to Python type
    
    FUNCTION get_db_prep_value(value: any, connection) → any
        // Convert a Python value to what the database driver expects
    
    FUNCTION get_internal_type() → string
        // Return the canonical type name: "CharField", "IntegerField", etc.
    
    FUNCTION db_type(connection) → string
        // Return the SQL column type for this backend: "VARCHAR(255)", "INTEGER", etc.
    
    // Validation
    FUNCTION validate(value: any, model_instance)
        // Raise ValidationError if invalid
    
    FUNCTION run_validators(value: any)
        // Run all registered validators against the value
    
    FUNCTION clean(value: any, model_instance)
        // to_python() + validate() + run_validators() in sequence
    
    // Contribution to class
    FUNCTION contribute_to_class(cls_name: string, field_name: string)
        // Called by the metaclass; field can set up reverse accessors, etc.

Core Field Implementations

Each field type encapsulates everything about how values of that type behave. The following are the most important fields every ORM must include:

CharField: Stores a string up to a maximum length. The max_length parameter is mandatory.

CLASS CharField EXTENDS Field:
    FIELDS:
        max_length: integer
    
    FUNCTION to_python(value):
        IF value is null: RETURN null IF null ELSE ""
        RETURN str(value)
    
    FUNCTION db_type(connection):
        RETURN f"VARCHAR({max_length})"
    
    FUNCTION validate(value, model_instance):
        IF value and len(value) > max_length:
            RAISE ValidationError(f"Value too long: max {max_length} chars")

IntegerField / BigIntegerField / SmallIntegerField: Stores integers of varying sizes.

FloatField / DecimalField: Float uses floating-point arithmetic (imprecise). DecimalField stores exact decimal numbers using a fixed-point representation and is mandatory for monetary values. DecimalField requires max_digits and decimal_places parameters.

BooleanField: Stores True/False. Many databases use 0/1 or a BOOLEAN type.

DateField / DateTimeField / TimeField: Store date and time values. DateTimeField must handle timezone awareness — whether the stored value includes timezone information or is "naive" (assumed to be UTC).

TextField: Stores arbitrarily long text. Maps to TEXT in SQL. No max_length validation, but may have performance implications for indexing.

EmailField: A CharField with an email validator pre-applied.

URLField: A CharField with a URL validator.

SlugField: A CharField restricted to URL-safe characters (letters, numbers, hyphens, underscores).

UUIDField: Stores a UUID. In PostgreSQL, uses the native UUID type. In SQLite/MySQL, stores as a 36-character VARCHAR.

JSONField: Stores arbitrary JSON data. In PostgreSQL, uses the native JSONB type (indexable, queryable). In other backends, serializes to a TEXT column.

FileField / ImageField: Stores a file path. Does not store the file itself in the database — stores only the path within the storage backend. ImageField adds image validation on top of FileField.

ForeignKey / ManyToManyField / OneToOneField: Relationship fields covered in Step 21.

The Validator System — GoF Strategy Pattern

Validators are the Strategy pattern applied to field validation. A validator is a callable that takes a value and raises ValidationError if the value is invalid. Validators are completely decoupled from fields — you can mix and match them freely:

INTERFACE Validator:              // GoF Strategy
    FUNCTION __call__(value: any) → void    // Raise ValidationError or do nothing

CLASS MinValueValidator IMPLEMENTS Validator:
    FIELDS: min_value: any
    FUNCTION __call__(value):
        IF value < min_value:
            RAISE ValidationError(f"Value must be at least {min_value}")

CLASS MaxValueValidator IMPLEMENTS Validator:
    FIELDS: max_value: any
    FUNCTION __call__(value):
        IF value > max_value:
            RAISE ValidationError(f"Value must be at most {max_value}")

CLASS RegexValidator IMPLEMENTS Validator:
    FIELDS: regex: CompiledRegex; message: string
    FUNCTION __call__(value):
        IF NOT regex.match(str(value)):
            RAISE ValidationError(message)

CLASS EmailValidator IMPLEMENTS Validator:
    FUNCTION __call__(value):
        IF NOT is_valid_email(value):
            RAISE ValidationError(f"'{value}' is not a valid email address")

// Fields register validators in their constructor:
CLASS EmailField EXTENDS CharField:
    CONSTRUCTOR(**kwargs):
        super(max_length=254, **kwargs)
        validators.append(EmailValidator())

Checklist

  • Define the Field base class interface with all required methods.
  • Implement CharField, TextField, IntegerField, FloatField, DecimalField.
  • Implement BooleanField, DateField, DateTimeField, TimeField.
  • Implement EmailField, URLField, SlugField, UUIDField.
  • Implement JSONField with backend-specific storage (JSONB for PostgreSQL).
  • Implement AutoField (auto-incrementing integer primary key) and UUIDAutoField.
  • Implement FileField that stores a path and delegates to the storage backend.
  • Implement the validator protocol with MinValueValidator, MaxValueValidator, RegexValidator, EmailValidator.
  • Implement ValidationError with field-level error messages and non-field errors.
  • Implement Field.choices for enum-like fields (validates against a list of allowed values).
  • Test each field's to_python(), db_type(), and validation logic independently.

19. The ORM — QuerySet & Lazy Evaluation

GoF Pattern: Builder + Iterator + Proxy + Prototype (Cloning)

The QuerySet is the most important and most elegant part of the ORM's user-facing API. It represents a database query that has not yet been executed — a lazy, composable query builder that allows you to chain filters, orderings, annotations, and joins without sending a single SQL query to the database until the results are actually needed.

The Builder pattern (GoF) is at work because the QuerySet is constructed incrementally through method chaining — each method call adds a new constraint or transformation to the query description, returning a new QuerySet object. The Proxy pattern is at work because the QuerySet represents a set of database rows without actually fetching them. The Iterator pattern is at work when the results are eventually consumed. The Prototype pattern is at work in the internal _clone() method that creates a copy of the current QuerySet before applying new constraints — this is what makes chaining safe and non-destructive.

What Makes a QuerySet Lazy

Laziness is the defining characteristic of a QuerySet. The QuerySet object holds a description of a query (table, filters, orderings, limits, joins) but does not execute it until one of the following "evaluation triggers" occurs:

  • Iteration: for user in User.objects.all() triggers the query.
  • Slicing with a step or explicit bounds: User.objects.all()[0:10] triggers the query.
  • len(queryset): Triggers the query and returns the count.
  • bool(queryset): Triggers the query; True if any results exist.
  • list(queryset): Explicitly forces evaluation.
  • repr(queryset): For debugging; triggers evaluation.
  • .get(), .first(), .last(), .exists(), .count(): Each is an explicit evaluation.

The key insight is that chaining filter methods does NOT trigger evaluation. This is what makes queries like User.objects.filter(active=True).filter(age__gte=18).order_by("-created_at")[:10] efficient — one SQL query is generated and executed, not three.

The QuerySet Data Structure

CLASS QuerySet:
    FIELDS:
        model:        Model class      // Which model this QuerySet is for
        _db:          string           // Which database alias to use
        _result_cache: List | None     // None = not yet evaluated; list = results
        
        // Query description (built up through chaining)
        _filter_conditions:    List<Q>      // WHERE conditions
        _exclude_conditions:   List<Q>      // Negated WHERE conditions
        _ordering:             List<string> // ORDER BY clauses
        _select_fields:        List<string> // SELECT ... (None = select all)
        _limit:                int | None
        _offset:               int | None
        _distinct:             boolean
        _select_related:       dict | boolean  // JOIN related tables
        _prefetch_related:     List<string>    // Separate queries for M2M
        _annotations:          dict<string, Expression>
        _group_by:             List<string>
        _having:               Q | None
        _joins:                List<Join>

    FUNCTION _clone() → QuerySet:
        // Create a copy of this QuerySet without evaluation cache
        // This is the Prototype pattern — create a new QuerySet with the same
        // query description, so that chaining does not mutate the original
        clone = QuerySet.__new__()
        clone.model             = model
        clone._db               = _db
        clone._result_cache     = None    // New clone has not been evaluated
        clone._filter_conditions  = list(_filter_conditions)   // Shallow copy
        clone._ordering         = list(_ordering)
        // ... copy all other query description fields
        RETURN clone

The QuerySet API — Method Chaining

Every method that adds a constraint MUST return a new cloned QuerySet rather than modifying the current one. This is essential for safety: a QuerySet stored in a variable must not be unexpectedly mutated by code that chains off it.

FUNCTION filter(**kwargs) → QuerySet:
    // kwargs: field lookups like name="Alice", age__gte=18, status__in=["active"]
    clone = self._clone()
    q = Q(**kwargs)   // Q object encapsulates conditions (Step 20)
    clone._filter_conditions.append(q)
    RETURN clone

FUNCTION exclude(**kwargs) → QuerySet:
    clone = self._clone()
    q = ~Q(**kwargs)  // Negate the conditions
    clone._filter_conditions.append(q)
    RETURN clone

FUNCTION order_by(*fields) → QuerySet:
    // fields: "name", "-created_at" (prefix - = descending)
    clone = self._clone()
    clone._ordering = list(fields)
    RETURN clone

FUNCTION distinct() → QuerySet:
    clone = self._clone()
    clone._distinct = true
    RETURN clone

FUNCTION values(*fields) → ValuesQuerySet:
    // Returns dicts instead of model instances
    clone = self._clone()
    clone._select_fields = list(fields)
    clone.__class__ = ValuesQuerySet
    RETURN clone

FUNCTION values_list(*fields, flat=False) → ValuesListQuerySet:
    // Returns tuples (or flat values if flat=True and only one field)

FUNCTION annotate(**annotations) → QuerySet:
    // Add computed values to each result: User.objects.annotate(post_count=Count("posts"))
    clone = self._clone()
    FOR name, expression IN annotations.items():
        clone._annotations[name] = expression
    RETURN clone

FUNCTION select_related(*fields) → QuerySet:
    // Perform a JOIN to prefetch related objects in a single query
    clone = self._clone()
    IF fields:
        clone._select_related = {f: {} FOR f IN fields}
    ELSE:
        clone._select_related = true  // Follow all non-null foreign keys
    RETURN clone

FUNCTION prefetch_related(*lookups) → QuerySet:
    // Schedule additional queries after the main query for M2M/reverse FK
    clone = self._clone()
    clone._prefetch_related.extend(lookups)
    RETURN clone

Terminal Methods — Evaluation Triggers

Terminal methods consume the QuerySet and return concrete results:

FUNCTION get(**kwargs) → model instance:
    // Fetch exactly one result; raise DoesNotExist or MultipleObjectsReturned
    clone = self.filter(**kwargs) IF kwargs ELSE self._clone()
    clone._limit = 2   // Only fetch 2 to detect MultipleObjectsReturned
    results = list(clone)
    IF len(results) == 0:
        RAISE model.DoesNotExist(f"{model.__name__} matching query does not exist")
    IF len(results) > 1:
        RAISE model.MultipleObjectsReturned("get() returned more than one instance")
    RETURN results[0]

FUNCTION first() → model instance | None:
    clone = self.order_by(*_meta.ordering or ["pk"])._clone()
    clone._limit = 1
    results = list(clone)
    RETURN results[0] IF results ELSE null

FUNCTION last() → model instance | None:
    // Reverse ordering and take first
    clone = self.order_by(*["-" + f IF NOT f.startswith("-") ELSE f[1:]
                             FOR f IN (_meta.ordering or ["pk"])])
    clone._limit = 1
    results = list(clone)
    RETURN results[0] IF results ELSE null

FUNCTION exists() → boolean:
    // Efficient existence check: SELECT 1 FROM ... WHERE ... LIMIT 1
    IF _result_cache is not null: RETURN len(_result_cache) > 0
    clone = self._clone()
    clone._select_fields = ["1"]
    clone._limit = 1
    sql, params = compiler.as_sql_exists(clone)
    cursor = connection.execute(sql, params)
    RETURN cursor.fetchone() is not null

FUNCTION count() → integer:
    // SELECT COUNT(*) FROM ... WHERE ...
    IF _result_cache is not null: RETURN len(_result_cache)
    sql, params = compiler.as_sql_count(self)
    cursor = connection.execute(sql, params)
    RETURN cursor.fetchone()[0]

FUNCTION create(**kwargs) → model instance:
    // Create and save a new model instance in one operation
    obj = model(**kwargs)
    obj.save()
    RETURN obj

FUNCTION get_or_create(defaults={}, **kwargs) → (model instance, boolean):
    // Atomic get-or-create operation
    TRY:
        RETURN (self.get(**kwargs), False)   // False = not created
    EXCEPT model.DoesNotExist:
        params = {**kwargs, **defaults}
        obj = self.create(**params)
        RETURN (obj, True)   // True = created

FUNCTION bulk_create(objs: List<model>, batch_size=None) → List<model>:
    // Insert many rows in a single SQL statement: much faster than looping save()
    batches = chunk_list(objs, batch_size or len(objs))
    FOR batch IN batches:
        sql, params = compiler.as_sql_bulk_insert(model, batch)
        connection.execute(sql, params)
    RETURN objs

FUNCTION update(**kwargs) → integer:
    // UPDATE table SET field=value WHERE ... — returns number of rows updated
    sql, params = compiler.as_sql_update_queryset(self, kwargs)
    cursor = connection.execute(sql, params)
    RETURN cursor.rowcount

FUNCTION delete() → (integer, dict):
    // DELETE FROM table WHERE ... — with cascade collection
    collector = DeletionCollector(connection)
    collector.collect(list(self))
    return collector.delete()

Checklist

  • Implement the QuerySet class with full internal state as described.
  • Implement _clone() that deep-copies query description fields but not the result cache.
  • Implement all filtering methods: filter(), exclude(), Q objects, | and & operators on Q objects.
  • Implement ordering: order_by(), reverse(), latest(), earliest().
  • Implement evaluation triggers: iteration, slicing, len(), bool().
  • Implement terminal methods: get(), first(), last(), exists(), count().
  • Implement mutating methods: create(), get_or_create(), update_or_create(), bulk_create(), update(), delete().
  • Implement values(), values_list(), only(), defer() for field selection.
  • Implement annotate() with aggregate functions: Count, Sum, Avg, Min, Max.
  • Implement select_related() (JOIN-based) and prefetch_related() (separate query-based).
  • Write tests that confirm no SQL is executed until an evaluation trigger is hit.

20. The ORM — SQL Generation & Query Compilation

GoF Pattern: Visitor + Builder

SQL generation is the bridge between the abstract query description inside a QuerySet and the concrete SQL string that the database will execute. It must be correct, efficient, and produce parameterized queries (never string interpolation — that is the source of SQL injection).

The Visitor pattern (GoF) governs how the compiler processes the QuerySet's expression tree: the compiler visits each expression node (a filter condition, an annotation, a JOIN) and produces the corresponding SQL fragment, without the nodes knowing anything about the compiler.

The Q Object — Composable Filter Conditions

The Q object is how complex, composable WHERE conditions are represented:

CLASS Q:
    // Encapsulates one or more filter conditions with a logical connector
    FIELDS:
        conditions:  List<(string, any)>   // [("field__lookup", value)]
        connector:   ENUM { AND, OR }      // How conditions are combined
        negated:     boolean               // If True: NOT (...)
    
    FUNCTION __and__(other: Q) → Q:
        // self & other → Q with connector=AND containing [self, other]
        RETURN Q.combine(self, other, AND)
    
    FUNCTION __or__(other: Q) → Q:
        RETURN Q.combine(self, other, OR)
    
    FUNCTION __invert__() → Q:
        // ~Q(...) → negated Q
        clone = self._clone()
        clone.negated = NOT clone.negated
        RETURN clone

Field Lookups — The __ Notation

The double-underscore notation is the key to expressive ORM filtering. User.objects.filter(age__gte=18) means "where users.age >= 18". The part after __ is the lookup type. Your SQL compiler must translate each lookup type to the appropriate SQL operator:

LOOKUP_MAP = {
    "exact":       "= %s",
    "iexact":      "UPPER({col}) = UPPER(%s)",
    "contains":    "{col} LIKE %s",          // params: ("%value%",)
    "icontains":   "UPPER({col}) LIKE UPPER(%s)",
    "startswith":  "{col} LIKE %s",          // params: ("value%",)
    "endswith":    "{col} LIKE %s",          // params: ("%value",)
    "in":          "{col} IN (%s)",          // %s expands to N placeholders
    "gt":          "> %s",
    "gte":         ">= %s",
    "lt":          "< %s",
    "lte":         "<= %s",
    "isnull":      "{col} IS NULL" OR "{col} IS NOT NULL",
    "range":       "{col} BETWEEN %s AND %s",
    "year":        "EXTRACT(YEAR FROM {col}) = %s",
    "month":       "EXTRACT(MONTH FROM {col}) = %s",
    "day":         "EXTRACT(DAY FROM {col}) = %s",
    "regex":       "{col} REGEXP %s",
    "iregex":      "{col} REGEXP (?i)%s",
}

The SQL Compiler

CLASS SqlCompiler:
    FIELDS:
        connection: Connection
        backend:    DatabaseBackend
    
    FUNCTION as_sql(queryset: QuerySet) → (string, list):
        // Build the full SELECT statement
        select_clause  = self.build_select_clause(queryset)
        from_clause    = self.build_from_clause(queryset)
        join_clauses   = self.build_joins(queryset)
        where_clause, where_params = self.build_where(queryset._filter_conditions)
        group_by       = self.build_group_by(queryset)
        having_clause, having_params = self.build_having(queryset)
        order_clause   = self.build_order_by(queryset)
        limit_offset   = self.build_limit_offset(queryset)
        
        parts = [select_clause, from_clause]
        IF join_clauses:   parts.extend(join_clauses)
        IF where_clause:   parts.append(f"WHERE {where_clause}")
        IF group_by:       parts.append(f"GROUP BY {group_by}")
        IF having_clause:  parts.append(f"HAVING {having_clause}")
        IF order_clause:   parts.append(f"ORDER BY {order_clause}")
        IF limit_offset:   parts.append(limit_offset)
        
        sql    = " ".join(parts)
        params = where_params + having_params
        RETURN (sql, params)
    
    FUNCTION build_select_clause(queryset) → string:
        table = backend.quote_name(queryset.model._meta.db_table)
        IF queryset._select_fields:
            cols = [f"{table}.{backend.quote_name(f)}" FOR f IN queryset._select_fields]
        ELSE:
            cols = [f"{table}.*"]
        
        IF queryset._distinct:
            RETURN f"SELECT DISTINCT {', '.join(cols)}"
        RETURN f"SELECT {', '.join(cols)}"
    
    FUNCTION build_where(conditions: List<Q>) → (string, list):
        IF NOT conditions: RETURN ("", [])
        
        parts = []
        params = []
        FOR q IN conditions:
            fragment, frag_params = self.compile_q(q)
            parts.append(f"({fragment})")
            params.extend(frag_params)
        
        RETURN ("AND ".join(parts), params)
    
    FUNCTION compile_q(q: Q) → (string, list):
        parts = []
        params = []
        FOR field_lookup, value IN q.conditions:
            field_name, _, lookup_type = field_lookup.rpartition("__")
            IF NOT lookup_type: lookup_type = "exact"
            
            // Resolve field and column name (handles __ traversals for related fields)
            column = resolve_column(q.model, field_name)
            sql_template = LOOKUP_MAP.get(lookup_type)
            
            sql_part = sql_template.format(col=column, ph=backend.get_placeholder())
            parts.append(sql_part)
            params.extend(prepare_value(value, lookup_type))
        
        connector = " AND " IF q.connector == AND ELSE " OR "
        combined  = connector.join(parts)
        
        IF q.negated: combined = f"NOT ({combined})"
        RETURN (combined, params)

Aggregate Functions

Aggregates are expressions that collapse many rows into a single value. They are used with annotate() and aggregate():

CLASS Aggregate:
    FIELDS:
        source_expression: string   // The field or expression to aggregate
        alias:             string   // Output name
    
    FUNCTION as_sql(compiler) → (string, list):
        RAISE NotImplementedError

CLASS Count EXTENDS Aggregate:
    FIELDS: distinct: boolean
    FUNCTION as_sql(compiler):
        col = compiler.compile_expression(source_expression)
        IF distinct: RETURN (f"COUNT(DISTINCT {col})", [])
        RETURN (f"COUNT({col})", [])

CLASS Sum   EXTENDS Aggregate:
    FUNCTION as_sql(compiler): RETURN (f"SUM({compiler.compile(source_expression)})", [])

CLASS Avg   EXTENDS Aggregate:
    FUNCTION as_sql(compiler): RETURN (f"AVG({compiler.compile(source_expression)})", [])

Checklist

  • Implement the Q object with &, |, ~ operators.
  • Implement the full LOOKUP_MAP for all lookup types.
  • Implement SqlCompiler.as_sql(queryset) assembling SELECT, FROM, WHERE, GROUP BY, HAVING, ORDER BY, LIMIT/OFFSET.
  • Implement build_where() with recursive Q compilation.
  • Implement JOIN generation for select_related().
  • Implement subquery support: filter(pk__in=Subquery(...)).
  • Implement aggregate functions: Count, Sum, Avg, Min, Max.
  • Implement window functions: Row_Number, Rank, DenseRank (advanced).
  • Ensure all user-provided values go through parameterized queries (never string interpolation).
  • Test SQL output for complex queries: joins + filters + annotations + ordering + pagination.

21. The ORM — Relationships: ForeignKey, ManyToMany, OneToOne

GoF Pattern: Proxy + Composite (M2M Through Tables)

Relationships between models are the heart of relational database design. The ORM must represent them in a way that feels natural in the object model while generating correct and efficient SQL. There are three fundamental relationship types: ForeignKey (many-to-one), OneToOneField (one-to-one), and ManyToManyField (many-to-many).

ForeignKey — Many-to-One

A ForeignKey represents a reference from many instances of one model to one instance of another. In SQL, this is a column that contains the primary key of the related table.

CLASS ForeignKey EXTENDS Field:
    FIELDS:
        to:            Model class | string   // The related model (or its string name for lazy resolution)
        on_delete:     ENUM { CASCADE, SET_NULL, SET_DEFAULT, PROTECT, DO_NOTHING, RESTRICT }
        related_name:  string | null          // Name for the reverse accessor
        related_query_name: string | null
        db_constraint: boolean                // Whether to create a FK constraint in the DB
    
    FUNCTION contribute_to_class(cls, field_name):
        // Install a forward accessor on the model (instance.related_model)
        setattr(cls, field_name, ForwardManyToOneDescriptor(self))
        
        // Install a reverse accessor on the related model (related_instance.model_set)
        related_name = self.related_name OR (cls.__name__.lower() + "_set")
        setattr(to, related_name, ReverseManyToOneDescriptor(self))
    
    FUNCTION db_type(connection):
        // Use the same type as the related model's primary key
        pk_field = to._meta.pk
        RETURN pk_field.db_type(connection)

// The forward descriptor: user.profile fetches the related Profile instance
CLASS ForwardManyToOneDescriptor:
    FUNCTION __get__(instance, owner):
        IF instance is null: RETURN self
        pk_value = instance.__dict__.get(field.attname)  // The raw FK column value
        IF pk_value is null: RETURN null
        // Fetch from cache first; otherwise query
        IF "__cache__" NOT IN instance.__dict__:
            instance.__dict__["__cache__"] = {}
        IF field.name NOT IN instance.__dict__["__cache__"]:
            instance.__dict__["__cache__"][field.name] = field.to.objects.get(pk=pk_value)
        RETURN instance.__dict__["__cache__"][field.name]
    
    FUNCTION __set__(instance, value):
        // Allow assignment of the related object directly
        IF value is null:
            instance.__dict__[field.attname] = null
            instance.__dict__["__cache__"][field.name] = null
        ELSE:
            instance.__dict__[field.attname] = value.pk
            instance.__dict__["__cache__"][field.name] = value

// The reverse descriptor: author.article_set returns a RelatedManager
CLASS ReverseManyToOneDescriptor:
    FUNCTION __get__(instance, owner):
        IF instance is null: RETURN self
        RETURN RelatedManager(field.model, field, instance)

ManyToManyField — Many-to-Many Through a Junction Table

A ManyToManyField represents a many-to-many relationship. In SQL, this requires a junction table (also called an association table or through table) with two foreign key columns, one pointing to each side of the relationship. The ORM creates and manages this table automatically unless the user specifies a custom through model:

CLASS ManyToManyField EXTENDS RelatedField:
    FIELDS:
        to:          Model class
        through:     Model class | null   // Custom junction model; auto-generated if null
        related_name: string | null
        through_defaults: dict            // Default values for auto-created junction table rows
    
    FUNCTION contribute_to_class(cls, field_name):
        // If no through model is specified, auto-generate one
        IF through is null:
            through = create_auto_through_model(cls, to, field_name)
        
        // Install a ManyRelatedManager on both sides
        setattr(cls, field_name, ManyToManyDescriptor(self, reverse=False))
        setattr(to, related_name, ManyToManyDescriptor(self, reverse=True))

// ManyRelatedManager: handles add(), remove(), set(), clear(), all()
CLASS ManyRelatedManager:
    FIELDS:
        source_model: Model class
        target_model: Model class
        through:      Model class
        source_instance: model instance
    
    FUNCTION all() → QuerySet:
        // SELECT target.* FROM target
        // JOIN through ON through.target_id = target.id
        // WHERE through.source_id = self.source_instance.pk
        RETURN target_model.objects.filter(
            **{f"{through_target_field}__source": source_instance}
        )
    
    FUNCTION add(*objs, through_defaults={}):
        // INSERT OR IGNORE INTO through_table (source_id, target_id) VALUES ...
        FOR obj IN objs:
            through.objects.get_or_create(
                **{source_field: source_instance, target_field: obj},
                defaults=through_defaults
            )
    
    FUNCTION remove(*objs):
        // DELETE FROM through_table WHERE source_id=... AND target_id IN (...)
        through.objects.filter(
            **{source_field: source_instance, f"{target_field}__in": objs}
        ).delete()
    
    FUNCTION set(objs, clear=False):
        // Replace the entire set of related objects
        IF clear:
            self.clear()
            self.add(*objs)
        ELSE:
            old_ids = set(self.all().values_list("pk", flat=True))
            new_ids = set(obj.pk FOR obj IN objs)
            self.remove(*[obj FOR obj IN self.all() IF obj.pk IN old_ids - new_ids])
            self.add(*[obj FOR obj IN objs IF obj.pk IN new_ids - old_ids])
    
    FUNCTION clear():
        through.objects.filter(**{source_field: source_instance}).delete()

OneToOneField

A OneToOneField is a ForeignKey with unique=True. It represents a one-to-one relationship: each instance of the source model corresponds to exactly one instance of the target model. Commonly used to extend a model's data without modifying it (e.g., UserProfile extending User):

CLASS OneToOneField EXTENDS ForeignKey:
    FUNCTION contribute_to_class(cls, field_name):
        // Forward: instance.profile returns the related instance directly (not a manager)
        setattr(cls, field_name, ForwardOneToOneDescriptor(self))
        // Reverse: profile.user returns the source instance directly
        setattr(to, related_name, ReverseOneToOneDescriptor(self))
    
    FUNCTION db_type(connection):
        RETURN super().db_type(connection)   // Same as ForeignKey
    
    // Additional unique constraint added to the database
    FUNCTION get_internal_type(): RETURN "OneToOneField"

Checklist

  • Implement ForeignKey with ForwardManyToOneDescriptor and ReverseManyToOneDescriptor.
  • Implement all on_delete behaviours: CASCADE (delete the referencing row), SET_NULL, SET_DEFAULT, PROTECT (raise ProtectedError), DO_NOTHING (let the DB handle it), RESTRICT.
  • Implement ManyToManyField with auto-generated through table creation.
  • Implement ManyRelatedManager with all(), add(), remove(), set(), clear().
  • Implement custom through model support for M2M with extra data.
  • Implement OneToOneField.
  • Implement the CASCADE deletion collector: when a model instance is deleted, collect all related instances that must also be deleted (respecting on_delete settings) before executing any DELETE statements.
  • Implement select_related() for ForeignKey and OneToOneField (JOIN in the main query).
  • Implement prefetch_related() for ManyToManyField and reverse ForeignKey (separate queries, merged in Python).

22. The ORM — Migrations System

GoF Pattern: Command + Composite (Migration DAG)

The migrations system is one of the most architecturally sophisticated features of a full-stack ORM. Its purpose is to track the history of schema changes (adding a table, adding a column, changing a field's type, adding an index) as a series of ordered, reversible operations. Developers run migrate to apply pending migrations and makemigrations to generate new migrations by comparing the current model definitions to the last known schema state.

The Command pattern (GoF) is the governing pattern: each migration operation (CreateTable, AddField, AlterField, DeleteModel, AddIndex, etc.) is a Command object with an apply() method (forward migration) and an unapply() method (reverse migration). A migration file is a Composite command containing a list of such operations.

The Migration File Structure

A migration file is a code file (or declarative YAML/JSON) that describes a set of schema operations. It has a dependency list (which migrations it builds on top of), an apply() function, and an unapply() function:

CLASS Migration:
    FIELDS:
        name:          string             // e.g., "0001_initial"
        app_label:     string             // Which app this migration belongs to
        dependencies:  List<(app, name)>  // Migrations that must run first
        operations:    List<Operation>    // Ordered list of Command objects
        replaces:      List<(app, name)>  // Squashed migrations (advanced)

// The migration history table — one row per applied migration
STRUCT MigrationRecord:
    app:             string   // App label
    name:            string   // Migration name
    applied_at:      datetime // When it was applied

Migration Operations — The Command Pattern

INTERFACE Operation:                      // GoF Command
    FUNCTION apply(schema_editor, app_state)
    FUNCTION unapply(schema_editor, app_state)
    FUNCTION state_forwards(app_label, state)   // Update the in-memory state model
    FUNCTION describe() → string               // Human-readable description

CLASS CreateModel IMPLEMENTS Operation:
    FIELDS:
        name:         string
        fields:       List<(name, Field)>
        options:      dict      // Meta options (ordering, unique_together)
    
    FUNCTION apply(schema_editor, app_state):
        model = app_state.apps.get_model(app_label, name)
        schema_editor.create_table(model)  // Executes CREATE TABLE SQL
    
    FUNCTION unapply(schema_editor, app_state):
        model = app_state.apps.get_model(app_label, name)
        schema_editor.delete_table(model)  // Executes DROP TABLE SQL
    
    FUNCTION describe():
        RETURN f"Create model {name}"

CLASS AddField IMPLEMENTS Operation:
    FIELDS: model_name: string; name: string; field: Field
    
    FUNCTION apply(schema_editor, app_state):
        model = app_state.get_model(model_name)
        schema_editor.add_column(model, field)   // ALTER TABLE ... ADD COLUMN
    
    FUNCTION unapply(schema_editor, app_state):
        model = app_state.get_model(model_name)
        schema_editor.remove_column(model, field)  // ALTER TABLE ... DROP COLUMN

CLASS AlterField IMPLEMENTS Operation:
    FIELDS: model_name: string; name: string; field: Field; old_field: Field
    
    FUNCTION apply(schema_editor, app_state):
        model = app_state.get_model(model_name)
        schema_editor.alter_column(model, old_field, field)  // ALTER TABLE ... ALTER COLUMN

CLASS RenameField IMPLEMENTS Operation
CLASS DeleteModel IMPLEMENTS Operation
CLASS AddIndex IMPLEMENTS Operation
CLASS RemoveIndex IMPLEMENTS Operation
CLASS AddConstraint IMPLEMENTS Operation
CLASS RunSQL IMPLEMENTS Operation:     // Escape hatch for custom SQL
CLASS RunPython IMPLEMENTS Operation:  // Escape hatch for data migrations

The Migration Autodetector — makemigrations

The autodetector compares the current state of the model definitions (reading Python source files) against the last known state (stored in existing migration files) and generates the minimal set of operations needed to bring the schema in sync:

CLASS MigrationAutodetector:
    FIELDS:
        from_state: ProjectState   // State after all existing migrations
        to_state:   ProjectState   // Current state from model definitions
    
    FUNCTION detect_changes() → dict<app, List<Operation>>:
        changes = {}
        
        // Step 1: Find new models (in to_state but not from_state)
        FOR app_label, model_name IN to_state.models - from_state.models:
            changes[app_label].add(CreateModel(model_name, ...))
        
        // Step 2: Find deleted models
        FOR app_label, model_name IN from_state.models - to_state.models:
            changes[app_label].add(DeleteModel(model_name))
        
        // Step 3: Find field changes in surviving models
        FOR model IN intersection(from_state.models, to_state.models):
            old_fields = from_state.get_fields(model)
            new_fields = to_state.get_fields(model)
            
            FOR field_name IN new_fields - old_fields:
                changes[model.app].add(AddField(model, field_name, new_fields[field_name]))
            
            FOR field_name IN old_fields - new_fields:
                changes[model.app].add(RemoveField(model, field_name))
            
            FOR field_name IN intersection(old_fields, new_fields):
                IF old_fields[field_name] != new_fields[field_name]:
                    changes[model.app].add(AlterField(model, field_name, new_fields[field_name]))
        
        // Step 4: Sort operations to resolve dependencies
        RETURN sort_by_dependencies(changes)

The Migration Executor — migrate

CLASS MigrationExecutor:
    FIELDS:
        connection:   Connection
        loader:       MigrationLoader   // Reads migration files from disk
    
    FUNCTION migrate(targets: List<(app, name)> | null):
        // Load the migration graph from disk
        graph = loader.build_graph()
        
        // Determine what is already applied
        applied = get_applied_migrations(connection)
        
        // Determine what needs to run
        plan = create_plan(graph, applied, targets)
        
        // Apply each migration in order
        FOR migration, backwards IN plan:
            IF backwards:
                unapply_migration(migration)
            ELSE:
                apply_migration(migration)
    
    FUNCTION apply_migration(migration: Migration):
        schema_editor = connection.backend.get_schema_editor(connection)
        
        WITH atomic_transaction():
            state = ProjectState()
            FOR operation IN migration.operations:
                operation.apply(schema_editor, state)
            
            record_applied(connection, migration.app_label, migration.name)

Checklist

  • Implement the Operation interface with apply(), unapply(), state_forwards().
  • Implement all core operations: CreateModel, AddField, AlterField, RenameField, DeleteModel, AddIndex, RemoveIndex, AddConstraint.
  • Implement RunSQL and RunPython for escape-hatch migrations.
  • Implement ProjectState — an in-memory snapshot of all models after a given migration.
  • Implement MigrationAutodetector that detects differences and generates operations.
  • Implement MigrationLoader that reads migration files and builds a dependency graph.
  • Implement the migration plan: resolve the dependency graph to produce an ordered list.
  • Implement MigrationExecutor with atomic application and record-keeping.
  • Implement migrate --fake (record as applied without executing) and migrate --plan (show what would run).
  • Implement migration squashing to merge many migrations into one.

23. The ORM — Connection Pooling & Transaction Management

GoF Pattern: Singleton (Pool) + Decorator (atomic())

The connection pool is responsible for reusing database connections across multiple requests rather than creating and destroying them on every request. Database connection setup is expensive (TCP handshake, authentication, SSL negotiation), taking tens to hundreds of milliseconds. For an application handling 1,000 requests per second, creating a new connection for every request would make the database connection overhead dwarf the actual query time.

The Connection Pool

CLASS ConnectionPool:
    FIELDS:
        backend:         DatabaseBackend
        config:          DatabaseConfig
        max_connections: integer        // Maximum pool size
        available:       Queue<Connection>  // Idle connections ready to use
        in_use:          Set<Connection>    // Currently checked out
        lock:            threading.Lock     // Thread-safe pool access
    
    FUNCTION get_connection() → Connection:
        WITH lock:
            // Try to get an idle connection
            WHILE NOT available.empty():
                conn = available.get()
                IF is_connection_alive(conn):
                    in_use.add(conn)
                    RETURN conn
                ELSE:
                    conn.close()   // Dead connection; discard and try next
            
            // No idle connections; create a new one if under limit
            IF len(in_use) < max_connections:
                conn = backend.create_connection(config)
                in_use.add(conn)
                RETURN conn
            
            // Pool is full; wait for a connection to be returned
            RAISE PoolExhausted("Connection pool exhausted; try again later")
    
    FUNCTION return_connection(conn: Connection):
        WITH lock:
            in_use.discard(conn)
            IF conn.is_closed() OR len(available) >= max_connections:
                conn.close()
            ELSE:
                available.put(conn)

    FUNCTION is_connection_alive(conn: Connection) → boolean:
        TRY:
            conn.execute("SELECT 1")
            RETURN true
        EXCEPT:
            RETURN false

Per-Request Connection Strategy

The recommended pattern for web applications (used by Django's CONN_MAX_AGE setting) is per-thread connection reuse: each thread gets one connection that is reused for the duration of the request and potentially held open for subsequent requests (up to a configurable age). This avoids the overhead of pool checkout/return while keeping the pool simple:

// Thread-local connection storage
THREAD_LOCAL_CONNECTIONS: ThreadLocal<dict<string, Connection>> = ThreadLocal()

FUNCTION get_connection(db_alias="default") → Connection:
    connections = THREAD_LOCAL_CONNECTIONS.get_or_create({})
    IF db_alias NOT IN connections:
        connections[db_alias] = pool.get_connection()
    RETURN connections[db_alias]

FUNCTION close_thread_connections():
    // Called at the end of each request by a middleware
    connections = THREAD_LOCAL_CONNECTIONS.get({})
    FOR alias, conn IN connections.items():
        pool.return_connection(conn)
    THREAD_LOCAL_CONNECTIONS.set({})

The atomic() Transaction Decorator / Context Manager

atomic() is the user-facing API for transaction management. It can be used as both a decorator and a context manager. It supports nesting via savepoints — inner atomic() blocks use database savepoints if an outer transaction is already active:

CLASS atomic:
    FIELDS:
        using:    string   // Database alias
        savepoint: boolean // Whether to use savepoints for nesting
    
    FUNCTION __enter__():
        connection = get_connection(using)
        
        IF connection.in_transaction:
            // We're inside an existing transaction; use a savepoint
            IF savepoint:
                name = f"savepoint_{connection.savepoint_counter}"
                connection.create_savepoint(name)
                self.savepoint_name = name
                self.needs_rollback = False
        ELSE:
            // Start a new transaction
            connection.begin_transaction()
            self.savepoint_name = null
    
    FUNCTION __exit__(exc_type, exc_val, exc_tb):
        connection = get_connection(using)
        
        IF exc_type is not null:
            // An exception occurred; roll back
            IF self.savepoint_name:
                connection.rollback_to_savepoint(self.savepoint_name)
                connection.release_savepoint(self.savepoint_name)
            ELSE:
                connection.rollback()
        ELSE:
            // Success; commit
            IF self.savepoint_name:
                connection.release_savepoint(self.savepoint_name)
            ELSE:
                connection.commit()
    
    FUNCTION __call__(func):
        // Used as a decorator: @atomic or @atomic(using="secondary")
        FUNCTION wrapper(*args, **kwargs):
            WITH self:
                RETURN func(*args, **kwargs)
        RETURN wrapper

Checklist

  • Implement ConnectionPool with thread-safe get/return using a lock.
  • Implement per-thread connection storage using thread-local storage.
  • Implement connection health checking before reuse.
  • Implement atomic() as both a context manager and a decorator.
  • Implement savepoint nesting for inner atomic() blocks.
  • Implement CONN_MAX_AGE: close and reopen connections that have been idle longer than configured.
  • Implement ATOMIC_REQUESTS middleware: wrap every request in a transaction, roll back on any unhandled exception.
  • Test transaction isolation: changes inside an atomic block are not visible outside until committed.
  • Test savepoint rollback: an inner atomic() rollback should not roll back the outer transaction.

24. The Form System — Declaration & Binding

GoF Pattern: Composite (Fieldsets) + Iterator (Field Traversal) + Template Method (Validation Lifecycle)

The form system is the framework's mechanism for safely accepting, validating, and processing user input. A form is a collection of fields that describes what data is expected, how to validate it, and how to render an HTML <form> element. The form system has an extremely strong security purpose: it ensures that all user input is validated before it can reach the ORM or any business logic.

The form system also provides the ModelForm — a form that is automatically generated from a Model's field definitions, dramatically reducing boilerplate for CRUD interfaces.

The Form Declaration

A form is declared by defining a class that inherits from Form, with fields declared as class attributes (exactly like Model field declarations, using the metaclass pattern):

CLASS ContactForm EXTENDS Form:
    subject = CharField(max_length=100, label="Subject")
    message = TextField(label="Message", help_text="Please be specific.")
    sender  = EmailField(label="Your email address")
    cc_me   = BooleanField(required=False, label="Send me a copy")
    
    // Optional custom validation:
    FUNCTION clean_subject():
        // clean_<fieldname> methods are called for individual field validation
        data = self.cleaned_data["subject"]
        IF "spam" IN data.lower():
            RAISE ValidationError("Please do not send spam.")
        RETURN data  // Must return the cleaned value
    
    FUNCTION clean():
        // Cross-field validation: called after all individual fields are clean
        cleaned = super().clean()
        IF cleaned.get("message") == cleaned.get("subject"):
            RAISE ValidationError("Message cannot be the same as the subject.")
        RETURN cleaned

The Form Metaclass

Like Model, Form also uses a metaclass to extract field declarations from the class body and store them in an ordered dict:

METACLASS FormMetaclass:
    FUNCTION __new__(meta, name, bases, namespace):
        // Collect field declarations from this class and all base classes (in MRO order)
        declared_fields = OrderedDict()
        
        // First, collect from base classes (so subclass fields override base fields)
        FOR base IN bases:
            IF hasattr(base, "declared_fields"):
                declared_fields.update(base.declared_fields)
        
        // Then, collect from this class's namespace
        FOR attr_name, attr_value IN namespace.items():
            IF isinstance(attr_value, Field):
                declared_fields[attr_name] = attr_value
                del namespace[attr_name]   // Remove from class body; access via Form API
        
        cls = type.__new__(meta, name, bases, namespace)
        cls.declared_fields = declared_fields
        RETURN cls

The Form Instance — Binding, Validation & Cleaned Data

A form instance can be either unbound (no data; used to display a blank form) or bound (has data; can be validated). Binding happens by passing data to the constructor:

CLASS Form:
    FIELDS:
        data:          dict | null   // Input data (bound if not null)
        files:         dict | null   // Uploaded files
        prefix:        string | null // Form prefix for multiple forms on one page
        initial:       dict          // Initial values for unbound display
        
        // Set by validation:
        cleaned_data:  dict          // Validated and cleaned values (populated after is_valid())
        errors:        ErrorDict     // Keyed by field name; also has "__all__" for non-field errors
        _is_bound:     boolean       // True if data was passed to constructor
    
    PROPERTY is_bound:
        RETURN data is not null
    
    FUNCTION is_valid() → boolean:
        // Trigger full validation if not already done
        RETURN is_bound AND len(self.errors) == 0
    
    // The Template Method for the full validation lifecycle:
    FUNCTION full_clean():
        errors = ErrorDict()
        cleaned_data = {}
        
        // Phase 1: Field-level validation
        FOR field_name, field IN declared_fields.items():
            raw_value = data.get(prefix_key(field_name), MISSING)
            
            TRY:
                cleaned_value = field.clean(raw_value)
                
                // Call clean_<fieldname>() if defined on the form
                custom_cleaner = getattr(self, f"clean_{field_name}", null)
                IF custom_cleaner:
                    cleaned_value = custom_cleaner()
                
                cleaned_data[field_name] = cleaned_value
            
            EXCEPT ValidationError AS e:
                errors[field_name] = e.messages
        
        // Phase 2: Cross-field validation (only if all fields passed phase 1)
        IF NOT errors:
            TRY:
                self.cleaned_data = cleaned_data
                self.clean()
            EXCEPT ValidationError AS e:
                errors["__all__"] = e.messages
        
        self.errors = errors

ModelForm — Generating a Form from a Model

ModelForm automatically generates form fields from a Model's field definitions, saving the developer from declaring the same information twice:

CLASS ModelFormMetaclass EXTENDS FormMetaclass:
    FUNCTION __new__(meta, name, bases, namespace):
        cls = super().__new__(meta, name, bases, namespace)
        
        IF hasattr(cls, "Meta") AND hasattr(cls.Meta, "model"):
            model = cls.Meta.model
            fields = cls.Meta.fields   // "__all__" or a list of field names
            exclude = getattr(cls.Meta, "exclude", [])
            
            // Generate form fields from model fields
            generated_fields = {}
            FOR field_name, model_field IN model._meta.fields.items():
                IF fields != "__all__" AND field_name NOT IN fields: CONTINUE
                IF field_name IN exclude: CONTINUE
                form_field = model_field.formfield()  // Each ModelField knows its form field type
                IF form_field: generated_fields[field_name] = form_field
            
            // Merge generated fields with explicitly declared fields
            // (declared fields take precedence over generated ones)
            cls.declared_fields = {**generated_fields, **cls.declared_fields}
        
        RETURN cls

CLASS ModelForm EXTENDS Form:
    FUNCTION save(commit=True) → model instance:
        // Create or update a model instance from cleaned_data
        IF self.instance:
            // Update existing instance
            FOR field_name, value IN cleaned_data.items():
                setattr(self.instance, field_name, value)
        ELSE:
            // Create new instance
            self.instance = self.Meta.model(**cleaned_data)
        
        IF commit:
            self.instance.save()
            // Save M2M relationships (can't be saved until instance has a PK)
            self.save_m2m()
        
        RETURN self.instance

Checklist

  • Implement FormMetaclass with field declaration extraction and MRO inheritance.
  • Implement Form with is_bound, is_valid(), full_clean(), errors, cleaned_data.
  • Implement the two-phase validation lifecycle: field-level then cross-field.
  • Implement clean_<fieldname>() hook dispatch.
  • Implement clean() for cross-field validation.
  • Implement form rendering: form.as_p(), form.as_table(), form.as_ul() methods.
  • Implement per-field error rendering in templates: {{ form.field_name.errors }}.
  • Implement ModelFormMetaclass that generates form fields from model fields.
  • Implement ModelForm.save(commit=True).
  • Implement ModelFormSet for handling multiple instances of the same form.
  • Implement form prefix support for multiple forms on one page.
  • Implement the Widget system: each form field has a widget that controls HTML rendering.

25. The Validation System — Field & Cross-Field Rules

GoF Pattern: Strategy (Validators) + Chain of Responsibility (Validation Pipeline)

The validation system is shared between the form layer and the ORM model layer. The same validators that check email addresses in a form field also check email addresses when User.full_clean() is called before save. This sharing is essential for the DRY principle and is made possible by the Strategy pattern: validators are independent callables that can be attached to any field in any context.

The ValidationError Architecture

ValidationError is the single exception type used throughout the entire validation system. It can carry:

  • A single string message: For simple, one-field errors.
  • A list of messages: When multiple validation rules fail simultaneously.
  • A dictionary: For non-field errors and multi-field errors on model or form level.
CLASS ValidationError EXTENDS Exception:
    FIELDS:
        message:   string | list | dict
        code:      string | null     // Machine-readable error code: "required", "invalid", "max_length"
        params:    dict | null       // Parameters for message formatting
    
    FUNCTION messages() → List<string>:
        // Always return a flat list, regardless of how the error was created
        IF isinstance(message, dict):
            return flatten([v FOR v IN message.values()])
        IF isinstance(message, list):
            return message
        RETURN [message]
    
    FUNCTION update_error_dict(error_dict: dict):
        // Merge this error into an existing error dictionary
        IF isinstance(message, dict):
            FOR field, errors IN message.items():
                error_dict.setdefault(field, []).extend(errors)
        ELSE:
            error_dict.setdefault("__all__", []).extend(messages())

Cross-Field Validation Patterns

Cross-field validation — where the validity of one field depends on the value of another — is one of the trickiest parts of the validation system. There are two patterns for handling it:

Form-level clean() method: The most common approach for form validation. After all individual fields have been validated, the form's clean() method is called with access to all cleaned values. This is appropriate when the dependency is between form fields.

Model-level clean() method: For business rule validation that belongs at the data layer. Called by full_clean() before save. Appropriate for validating that an Event's end_date is after its start_date, regardless of how the data arrived (form, API, management command, or fixture).

CLASS Event EXTENDS Model:
    name:       CharField(max_length=200)
    start_date: DateTimeField()
    end_date:   DateTimeField()
    capacity:   IntegerField(min_value=0)
    
    FUNCTION clean():
        // Cross-field validation at the model level
        IF start_date AND end_date AND end_date <= start_date:
            RAISE ValidationError({
                "end_date": ValidationError(
                    "End date must be after start date.",
                    code="invalid_date_range"
                )
            })

The Unique Validation — Database-Level Constraints

Some validations require querying the database — for example, validating that a username is not already taken. The ORM must perform this validation as part of full_clean(), but it must be careful to exclude the current instance from the uniqueness check (otherwise updating an existing record would always fail):

FUNCTION validate_unique(model_instance):
    // Check unique=True fields
    FOR field_name, field IN model_instance._meta.fields.items():
        IF field.unique AND NOT field.primary_key:
            value = getattr(model_instance, field_name)
            IF value is null AND field.null: CONTINUE
            
            qs = field.model.objects.filter(**{field_name: value})
            IF model_instance.pk:
                qs = qs.exclude(pk=model_instance.pk)  // Exclude self
            
            IF qs.exists():
                RAISE ValidationError({field_name: f"'{value}' is already in use."})
    
    // Check unique_together constraints
    FOR unique_set IN model_instance._meta.unique_together:
        filter_kwargs = {f: getattr(model_instance, f) FOR f IN unique_set}
        qs = field.model.objects.filter(**filter_kwargs)
        IF model_instance.pk: qs = qs.exclude(pk=model_instance.pk)
        IF qs.exists():
            RAISE ValidationError(f"The combination of {unique_set} must be unique.")

Checklist

  • Implement ValidationError with message, code, params, and dictionary form.
  • Implement ErrorDict and ErrorList for structured error storage and rendering.
  • Implement the full built-in validator library: MinValue, MaxValue, MinLength, MaxLength, Regex, Email, URL, Slug, FileExtension, ProhibitNullCharacters.
  • Implement validate_unique() that queries the database for uniqueness violations.
  • Implement validate_unique_together() for composite unique constraints.
  • Implement model-level clean() and the full_clean() pipeline: clean_fields() → clean() → validate_unique().
  • Ensure form clean() receives already-converted Python values, not raw strings.
  • Test that updating an existing record does not trigger false uniqueness violations.

26. The Authentication & Authorization System

GoF Pattern: Strategy (Authentication Backends) + Decorator (Permission Checking) + Template Method (User Model)

The authentication system answers two distinct questions: Authentication asks "who are you?" — verifying a user's identity via credentials. Authorization asks "are you allowed?" — checking whether an authenticated user has permission to perform a specific action. These are often conflated but must be architecturally separated.

The User Model

The user model is the central entity of the authentication system. Crucially, the framework must allow the application developer to swap out the user model entirely — some applications need extra fields (profile picture, phone number), different authentication identifiers (email instead of username), or a completely different user hierarchy. Django solved this with AUTH_USER_MODEL setting and the AbstractUser / AbstractBaseUser base classes:

CLASS AbstractBaseUser EXTENDS Model:
    // The bare minimum: a password field and authentication mechanics.
    // Concrete user models inherit from this and add their own identifying fields.
    password:          CharField(max_length=128)  // Stored as hash:salt:algo
    last_login:        DateTimeField(null=True)
    
    ABSTRACT FUNCTION get_username() → string     // The field used for login
    ABSTRACT FUNCTION get_email_field_name() → string
    
    FUNCTION set_password(raw_password: string):
        self.password = make_password(raw_password)  // Hash with salt
    
    FUNCTION check_password(raw_password: string) → boolean:
        RETURN check_password_hash(raw_password, self.password)
    
    FUNCTION get_session_auth_hash() → string:
        // Used to invalidate sessions when password changes
        RETURN hmac(settings.SECRET_KEY, self.password, sha256)
    
    @property
    FUNCTION is_anonymous() → boolean: RETURN False
    @property
    FUNCTION is_authenticated() → boolean: RETURN True

CLASS AbstractUser EXTENDS AbstractBaseUser:
    // A more opinionated base with common fields pre-defined.
    username:          CharField(max_length=150, unique=True)
    email:             EmailField(unique=True)
    first_name:        CharField(max_length=150, blank=True)
    last_name:         CharField(max_length=150, blank=True)
    is_active:         BooleanField(default=True)
    is_staff:          BooleanField(default=False)     // Can access admin
    is_superuser:      BooleanField(default=False)     // Bypasses all permissions
    date_joined:       DateTimeField(auto_now_add=True)
    groups:            ManyToManyField("Group", blank=True)
    user_permissions:  ManyToManyField("Permission", blank=True)

CLASS AnonymousUser:
    // Represents a not-logged-in visitor.
    is_authenticated = False
    is_anonymous     = True
    is_active        = False
    is_staff         = False
    is_superuser     = False
    pk               = None
    
    FUNCTION has_perm(perm): RETURN False
    FUNCTION has_perms(perms): RETURN False

Password Hashing — The Most Security-Critical Function

Storing passwords as plaintext is an unforgivable security failure. Passwords must be stored as a hash produced by a password hashing algorithm designed for this purpose — not a general-purpose hash like SHA-256, which is too fast and enables brute-force attacks. The correct algorithms are Argon2 (the winner of the Password Hashing Competition, recommended), bcrypt, scrypt, or PBKDF2-SHA256.

FUNCTION make_password(raw_password: string,
                        salt: string = null,
                        hasher: string = settings.PASSWORD_HASHER) → string:
    IF raw_password is null:
        RAISE ValueError("Password cannot be null. Use 'set_unusable_password()' instead.")
    
    hasher_obj = get_hasher(hasher)
    salt = salt OR hasher_obj.generate_salt()
    hash = hasher_obj.encode(raw_password, salt)
    RETURN f"{hasher}${salt}${hash}"   // Stored format: "argon2id$salt$hash"

FUNCTION check_password_hash(raw_password: string, stored: string) → boolean:
    algorithm, salt, hash = stored.split("$", 2)
    hasher_obj = get_hasher(algorithm)
    computed = hasher_obj.encode(raw_password, salt)
    RETURN constant_time_compare(computed, hash)  // Must use constant-time comparison!

// Constant-time comparison prevents timing attacks:
FUNCTION constant_time_compare(a: string, b: string) → boolean:
    IF len(a) != len(b): RETURN False   // Must pad if different lengths to avoid leaking length
    result = 0
    FOR char_a, char_b IN zip(a, b):
        result |= (char_a XOR char_b)
    RETURN result == 0

Authentication Backends — GoF Strategy

The authentication backend is a Strategy: multiple backends can be configured, and they are tried in order until one succeeds or all fail. This allows supporting multiple authentication methods (username/password, LDAP, OAuth, API key) simultaneously:

INTERFACE AuthenticationBackend:      // GoF Strategy
    FUNCTION authenticate(request, **credentials) → User | None
    FUNCTION get_user(user_id) → User | None
    FUNCTION has_perm(user, perm, obj=None) → boolean
    FUNCTION has_module_perms(user, app_label) → boolean

CLASS ModelBackend IMPLEMENTS AuthenticationBackend:
    // Authenticates against the database user model
    FUNCTION authenticate(request, username=null, password=null):
        IF username is null OR password is null: RETURN null
        UserModel = get_user_model()
        TRY:
            user = UserModel.objects.get(**{UserModel.USERNAME_FIELD: username})
        EXCEPT UserModel.DoesNotExist:
            // Run the hasher anyway to prevent username enumeration via timing
            UserModel().set_password(password)
            RETURN null
        
        IF user.check_password(password) AND user.is_active:
            RETURN user
        RETURN null
    
    FUNCTION has_perm(user, perm, obj=None):
        IF NOT user.is_active: RETURN False
        IF user.is_superuser: RETURN True
        app_label, codename = perm.split(".")
        RETURN Permission.objects.filter(
            codename=codename,
            content_type__app_label=app_label,
            user=user OR group__user=user
        ).exists()

FUNCTION authenticate(request, **credentials) → User | None:
    // Try each configured backend in order
    FOR backend_path IN settings.AUTHENTICATION_BACKENDS:
        backend = load_backend(backend_path)
        user = backend.authenticate(request, **credentials)
        IF user is not null:
            user._auth_backend = backend_path   // Remember which backend authenticated
            RETURN user
    RETURN null

The Permission System

The permission system uses a code-based approach: each permission is identified by a string of the form "app_label.codename" (e.g., "blog.add_article", "blog.change_article", "blog.delete_article", "blog.view_article"). The framework auto-creates these four CRUD permissions for every model. Custom permissions can be added via Meta.permissions:

CLASS Permission EXTENDS Model:
    name:          CharField(max_length=255)    // Human-readable: "Can add article"
    codename:      CharField(max_length=100)    // Machine: "add_article"
    content_type:  ForeignKey(ContentType)     // Which model this permission applies to

CLASS Group EXTENDS Model:
    // A named collection of permissions. Users can belong to many groups.
    name:         CharField(max_length=150, unique=True)
    permissions:  ManyToManyField(Permission)

CLASS ContentType EXTENDS Model:
    // Represents a model class: (app_label, model_name) → model class
    app_label:  string
    model:      string

Checklist

  • Implement AbstractBaseUser with password hashing and session auth hash.
  • Implement AbstractUser with all standard fields.
  • Implement AnonymousUser with all permission methods returning False.
  • Implement make_password() with Argon2 as the default hasher.
  • Implement check_password_hash() with constant-time comparison.
  • Implement AuthenticationBackend interface and ModelBackend.
  • Implement authenticate(request, **credentials) that tries all backends.
  • Implement login(request, user) that sets the user in the session.
  • Implement logout(request) that clears the session.
  • Implement the Permission, Group, and ContentType models.
  • Auto-create CRUD permissions for every model after migration.
  • Implement user.has_perm() and user.has_perms() with backend delegation.
  • Implement @login_required and @permission_required decorators.
  • Implement LoginView, LogoutView, PasswordChangeView, PasswordResetView as generic views.

27. The Session System

GoF Pattern: Proxy (Lazy Loading) + Strategy (Session Backends)

The session system gives your stateless HTTP protocol the illusion of state. Because HTTP is stateless (each request is independent), the only way for the server to remember who a user is across multiple requests is to give the browser a session identifier (stored in a cookie) and remember the corresponding data server-side.

The session must be designed so that the session data is not loaded from the backend until it is actually accessed — a lazy-loading Proxy pattern — because most requests may not need session data at all (static file requests, public pages, etc.).

Session Backends — GoF Strategy

The session data can be stored in many places, each with different trade-offs:

INTERFACE SessionBackend:                // GoF Strategy
    FUNCTION load(session_key) → dict
    FUNCTION save(session_key, data, expiry: datetime)
    FUNCTION delete(session_key)
    FUNCTION exists(session_key) → boolean
    FUNCTION create_session_key() → string

CLASS DatabaseSessionBackend IMPLEMENTS SessionBackend:
    // Stores sessions in the database. Durable; requires periodic cleanup.
    // Best for: applications with an existing database; sessions with complex data.

CLASS CacheSessionBackend IMPLEMENTS SessionBackend:
    // Stores sessions in the cache (Redis/Memcached). Fast; volatile.
    // Best for: high-traffic applications; sessions that can be lost (re-login is acceptable).

CLASS CachedDatabaseSessionBackend IMPLEMENTS SessionBackend:
    // Write-through cache: writes to both cache and database.
    // Reads from cache first; falls back to database on cache miss.
    // Best for: high-traffic with durability requirements.

CLASS FileSessionBackend IMPLEMENTS SessionBackend:
    // Stores sessions as files in a configured directory.
    // Simple; requires file system access; not suitable for multi-server deployments.

CLASS SignedCookieSessionBackend IMPLEMENTS SessionBackend:
    // Stores session data in the cookie itself (signed + encrypted).
    // No server-side storage; stateless. Limit: cookie max size ~4KB.
    // Best for: small, simple session data; applications with many servers.

The Session Object

CLASS Session:
    FIELDS:
        session_key:   string
        backend:       SessionBackend
        _data:         dict | None    // None = not yet loaded (Proxy pattern)
        modified:      boolean        // True if data has changed and needs saving
        accessed:      boolean        // True if data was read (update cookie expiry)
        _session_cache: dict | None
    
    PROPERTY _session:
        // Lazy loading — only hits the backend when data is first accessed
        IF _session_cache is null:
            _session_cache = backend.load(session_key)
        RETURN _session_cache
    
    FUNCTION get(key, default=null):
        self.accessed = True
        RETURN _session.get(key, default)
    
    FUNCTION set(key, value):
        self.accessed = True
        self.modified = True
        _session[key] = value
    
    FUNCTION delete(key):
        self.modified = True
        _session.pop(key, null)
    
    FUNCTION flush():
        // Delete the current session and create a new one.
        // Called on logout to prevent session fixation attacks.
        backend.delete(session_key)
        session_key = backend.create_session_key()
        _session_cache = {}
        modified = True
    
    FUNCTION cycle_key():
        // Create a new session key while preserving data.
        // Called on login to prevent session fixation.
        data = dict(_session)
        backend.delete(session_key)
        session_key = backend.create_session_key()
        _session_cache = data
        modified = True

Session Security

Sessions are a prime target for security attacks. The framework must address these:

Session Fixation: An attacker sets a known session key before the user logs in, then hijacks the session after login. Prevention: call session.cycle_key() immediately after a successful login.

Session Hijacking: An attacker steals the session cookie. Prevention: use HttpOnly and Secure cookie flags; use SameSite=Strict or Lax; use HTTPS.

Session Invalidation on Password Change: If a user's password is changed, all existing sessions should be invalidated to prevent a compromised session from persisting. Django uses get_session_auth_hash() stored in the session to detect password changes.

Checklist

  • Define the SessionBackend strategy interface.
  • Implement DatabaseSessionBackend, CacheSessionBackend, and SignedCookieSessionBackend.
  • Implement Session as a lazy-loading proxy over the backend.
  • Implement session.cycle_key() for login anti-fixation.
  • Implement session.flush() for logout.
  • Implement SessionMiddleware that loads the session from the cookie, attaches it to the request, and saves it to the response cookie after the view runs.
  • Implement session expiry: SESSION_COOKIE_AGE, SESSION_EXPIRE_AT_BROWSER_CLOSE.
  • Implement session invalidation on password change by storing and checking session_auth_hash.
  • Implement a management command to clear expired sessions from the database.
  • Test that cycle_key() is called on login and that the old key is deleted.

Understanding Cookies in Depth

Cookies are small pieces of data that the server sends to the browser in a Set-Cookie response header, and the browser sends back on every subsequent request in the Cookie header. They are the primary mechanism for session management, user preferences, tracking (which your framework should not build in!), and CSRF protection.

A cookie is more than just a key-value pair. The Set-Cookie header has a rich set of attributes that control the cookie's lifetime, scope, and security:

Set-Cookie: session_id=abc123; 
            Max-Age=3600; 
            Path=/; 
            Domain=example.com; 
            Secure; 
            HttpOnly; 
            SameSite=Lax

Each attribute has precise semantics your framework must implement correctly:

Max-Age vs. Expires: Max-Age specifies the cookie's lifetime in seconds from the time of receipt. Expires specifies an absolute date. Max-Age takes precedence; use it instead of Expires. A cookie with neither attribute is a "session cookie" — the browser deletes it when closed.

Path: The cookie is only sent for requests whose URL path starts with this value. Restricting the path can limit cookie exposure but also causes confusion. Default to /.

Domain: If set, the cookie is sent for this domain and all its subdomains. If not set, the cookie is only sent for the exact host that set it. Never set Domain=example.com if you don't intend to share cookies across subdomains.

Secure: The cookie is only sent over HTTPS connections. This attribute is mandatory for any cookie containing sensitive data. Without it, cookies are transmitted in plaintext over HTTP.

HttpOnly: The cookie cannot be accessed by JavaScript (document.cookie). This protects session cookies from XSS attacks — even if an attacker injects JavaScript, they cannot steal the session cookie.

SameSite: Controls when cookies are sent on cross-site requests. Strict = never on cross-site requests (breaks OAuth flows and some links). Lax = sent on top-level navigations (GET) but not on cross-site POST requests. None = always sent (requires Secure flag). Lax is the recommended default for session cookies.

CLASS Cookie:
    FIELDS:
        name:      string
        value:     string
        max_age:   integer | null   // seconds; null = session cookie
        expires:   datetime | null
        path:      string = "/"
        domain:    string | null
        secure:    boolean = False
        http_only: boolean = False
        same_site: string | null = "Lax"
    
    FUNCTION to_header_value() → string:
        parts = [f"{name}={quote_cookie(value)}"]
        IF max_age is not null:  parts.append(f"Max-Age={max_age}")
        IF expires:              parts.append(f"Expires={to_http_date(expires)}")
        IF path:                 parts.append(f"Path={path}")
        IF domain:               parts.append(f"Domain={domain}")
        IF secure:               parts.append("Secure")
        IF http_only:            parts.append("HttpOnly")
        IF same_site:            parts.append(f"SameSite={same_site}")
        RETURN "; ".join(parts)

CLASS CookieJar:
    FIELDS:
        cookies: dict<string, Cookie>
    
    FUNCTION set(name, value, **attrs) → Cookie:
        cookie = Cookie(name, value, **attrs)
        cookies[name] = cookie
        RETURN cookie
    
    FUNCTION delete(name, path="/", domain=null):
        // Tell browser to delete by setting expired date
        set(name, "", max_age=0, expires=EPOCH, path=path, domain=domain)
    
    FUNCTION to_headers() → List<(string, string)>:
        RETURN [("Set-Cookie", cookie.to_header_value()) FOR cookie IN cookies.values()]

Signed Cookies — Tamper-Proof State

Sometimes you want to store state in a cookie without using a server-side session. For example, flash messages (one-time notifications after a redirect) can be stored in a signed cookie. The cookie is signed with the server's SECRET_KEY so that any tampering is detected:

FUNCTION sign_cookie_value(value: string) → string:
    // Serialize the value and sign it
    serialized = json_encode(value)
    signature  = hmac(settings.SECRET_KEY, serialized, sha256)
    RETURN base64_encode(serialized) + ":" + signature

FUNCTION unsign_cookie_value(signed_value: string) → string | null:
    TRY:
        parts     = signed_value.rsplit(":", 1)
        data_b64  = parts[0]
        signature = parts[1]
        serialized = base64_decode(data_b64)
        expected   = hmac(settings.SECRET_KEY, serialized, sha256)
        IF NOT constant_time_compare(signature, expected):
            RAISE BadSignature("Cookie signature mismatch")
        RETURN json_decode(serialized)
    EXCEPT:
        RETURN null   // Invalid; ignore

Checklist

  • Implement the Cookie class with all RFC 6265 attributes.
  • Implement Cookie.to_header_value() with correct serialization.
  • Implement CookieJar on the Response object with set() and delete().
  • Implement parse_cookie_header() for the Request's Cookie header.
  • Implement sign_cookie_value() and unsign_cookie_value() using HMAC.
  • Set HttpOnly=True and SameSite=Lax as defaults for session cookies.
  • Set Secure=True as default when settings.SESSION_COOKIE_SECURE=True.
  • Implement flash message storage using signed cookies.
  • Test that delete() sets the correct expiry to instruct the browser to delete the cookie.

29. The Security Subsystem — CSRF, XSS, Headers & More

GoF Pattern: Chain of Responsibility (Security Header Pipeline) + Decorator (View-Level Security)

Security is not a feature you add after building the framework — it is a property that must be designed in from the beginning. This step covers the most critical security threats that a web framework must defend against: CSRF, XSS, SQL injection (handled by the ORM's parameterized queries), clickjacking, and MIME sniffing.

CSRF (Cross-Site Request Forgery) Protection

CSRF is an attack where a malicious website tricks an authenticated user's browser into making requests to your website. Because the browser automatically sends cookies with every request, the server cannot distinguish between requests that the user initiated and requests that the attacker triggered.

The standard defense is the Synchronizer Token Pattern: the server generates a secret random token, stores it in the session (or in a cookie), and requires it to be present in every state-changing request (POST, PUT, PATCH, DELETE) as a hidden form field or a custom HTTP header. The token is bound to the user's session so that an attacker cannot use a stolen token.

// CsrfMiddleware: Chain of Responsibility
CLASS CsrfMiddleware IMPLEMENTS Middleware:
    SAFE_METHODS = {"GET", "HEAD", "OPTIONS", "TRACE"}
    
    FUNCTION process(request, next_handler):
        // Step 1: Ensure the request has a CSRF cookie
        csrf_cookie = request.cookies.get(settings.CSRF_COOKIE_NAME)
        IF csrf_cookie is null:
            csrf_cookie = generate_csrf_token()    // Cryptographically random 64 bytes
        request.META["CSRF_COOKIE"] = csrf_cookie
        
        // Step 2: Skip verification for safe methods
        IF request.method IN SAFE_METHODS:
            response = next_handler(request)
            set_csrf_cookie(response, csrf_cookie)
            RETURN response
        
        // Step 3: For unsafe methods, verify the token
        // First, check the Origin or Referer header (defense-in-depth)
        IF NOT verify_origin_or_referer(request):
            RETURN Response(status=403, body="CSRF origin check failed")
        
        // Then, check the token in the request
        request_token = (
            request.data.get(settings.CSRF_INPUT_NAME) OR   // Form POST
            request.headers.get("X-CSRFToken")              // AJAX/API
        )
        
        IF NOT constant_time_compare(csrf_cookie, request_token):
            RETURN Response(status=403, body="CSRF token invalid or missing")
        
        // Step 4: Pass to next handler; set cookie on response
        response = next_handler(request)
        set_csrf_cookie(response, csrf_cookie)
        RETURN response
    
    FUNCTION verify_origin_or_referer(request) → boolean:
        origin   = request.headers.get("origin")
        referer  = request.headers.get("referer")
        
        IF origin:
            RETURN origin IN settings.CSRF_TRUSTED_ORIGINS
        IF referer:
            parsed = parse_url(referer)
            RETURN parsed.host IN settings.ALLOWED_HOSTS
        
        // No Origin or Referer header: for safety, reject
        RETURN False

Security HTTP Headers

A suite of HTTP response headers instructs the browser to apply additional protections. Your SecurityMiddleware must add these to every response:

CLASS SecurityMiddleware IMPLEMENTS Middleware:
    FUNCTION process(request, next_handler):
        // Redirect HTTP to HTTPS
        IF settings.SECURE_SSL_REDIRECT AND NOT request.is_secure:
            new_url = f"https://{request.host}{request.full_path}"
            RETURN RedirectResponse(new_url, permanent=True)
        
        response = next_handler(request)
        
        // HSTS: tell browsers to only connect via HTTPS for N seconds
        IF settings.SECURE_HSTS_SECONDS > 0:
            hsts = f"max-age={settings.SECURE_HSTS_SECONDS}"
            IF settings.SECURE_HSTS_INCLUDE_SUBDOMAINS: hsts += "; includeSubDomains"
            IF settings.SECURE_HSTS_PRELOAD:            hsts += "; preload"
            response.set_header("Strict-Transport-Security", hsts)
        
        // X-Content-Type-Options: prevent MIME sniffing attacks
        response.set_header("X-Content-Type-Options", "nosniff")
        
        // X-Frame-Options: prevent clickjacking
        IF settings.X_FRAME_OPTIONS:
            response.set_header("X-Frame-Options", settings.X_FRAME_OPTIONS)
        
        // Content-Security-Policy: restrict sources of scripts, styles, etc.
        IF settings.CONTENT_SECURITY_POLICY:
            response.set_header("Content-Security-Policy", settings.CONTENT_SECURITY_POLICY)
        
        // Referrer-Policy: control what is sent in the Referer header
        response.set_header("Referrer-Policy", "same-origin")
        
        // Permissions-Policy: disable unnecessary browser features
        response.set_header("Permissions-Policy", "camera=(), microphone=(), geolocation=()")
        
        RETURN response

Host Header Validation

The Host header is user-controlled and can be set to any value by an attacker. Without validation, password reset emails might contain malicious URLs like https://evil.com/reset/?token=abc. Your framework must validate the Host header against a whitelist:

CLASS CommonMiddleware IMPLEMENTS Middleware:
    FUNCTION process(request, next_handler):
        host = request.host
        
        IF NOT is_allowed_host(host):
            RETURN Response(status=400, body=f"Invalid Host header: {host}")
        
        RETURN next_handler(request)

FUNCTION is_allowed_host(host: string) → boolean:
    FOR pattern IN settings.ALLOWED_HOSTS:
        IF pattern == "*": RETURN True
        IF pattern.startswith("."):
            // Wildcard: ".example.com" matches "api.example.com" and "example.com"
            IF host == pattern[1:] OR host.endswith(pattern): RETURN True
        ELSE:
            IF host == pattern: RETURN True
    RETURN False

Checklist

  • Implement CsrfMiddleware with synchronizer token validation.
  • Implement {% csrf_input %} template tag that renders the CSRF hidden field.
  • Implement X-CSRFToken header support for AJAX/JSON requests.
  • Implement @csrf_exempt decorator for views that must opt out.
  • Implement SecurityMiddleware with HTTPS redirect and HSTS.
  • Set X-Content-Type-Options: nosniff on all responses.
  • Set X-Frame-Options (DENY or SAMEORIGIN) to prevent clickjacking.
  • Implement Content Security Policy header generation.
  • Implement Host header validation against ALLOWED_HOSTS.
  • Implement SQL injection protection: ensure all ORM queries use parameterized statements.
  • Implement rate limiting middleware with configurable limits per IP and per user.
  • Document that mark_safe() should only be called on trusted content.

30. The Caching System

GoF Pattern: Abstract Factory (Cache Backends) + Facade (Cache API) + Decorator (View/Fragment Caching)

Caching is one of the highest-leverage performance optimizations available to a web framework. A well-designed cache can reduce database load by 90% or more for read-heavy applications. The caching system must be flexible (multiple backends), composable (cache specific views, specific querysets, or specific template fragments), and consistent (the cache API must be identical regardless of which backend is configured).

Cache Backends — GoF Abstract Factory

INTERFACE CacheBackend:                      // GoF Abstract Factory
    FUNCTION get(key: string) → any | null
    FUNCTION set(key: string, value: any, timeout: int | null)
    FUNCTION delete(key: string)
    FUNCTION get_many(keys: List<string>) → dict
    FUNCTION set_many(mapping: dict, timeout: int | null)
    FUNCTION delete_many(keys: List<string>)
    FUNCTION has_key(key: string) → boolean
    FUNCTION incr(key: string, delta: int = 1) → int
    FUNCTION decr(key: string, delta: int = 1) → int
    FUNCTION clear()

CLASS MemcachedBackend IMPLEMENTS CacheBackend:
    // Connect to Memcached using pylibmc or pymemcache
    // Advantages: extremely fast; battle-tested; supports automatic eviction
    // Disadvantages: no persistence; no data types beyond key-value; no pub/sub

CLASS RedisBackend IMPLEMENTS CacheBackend:
    // Connect to Redis using redis-py or aioredis
    // Advantages: persistence; rich data types; pub/sub; sorted sets
    // Disadvantages: more memory per key; requires Redis server

CLASS DatabaseCacheBackend IMPLEMENTS CacheBackend:
    // Store cache entries in a database table
    // Advantages: uses existing infrastructure; durable; inspectable
    // Disadvantages: much slower than Memcached/Redis; adds DB load

CLASS LocMemCacheBackend IMPLEMENTS CacheBackend:
    // In-process Python dict — purely for development and testing
    // Zero configuration; zero performance overhead; does not persist between requests

CLASS DummyCacheBackend IMPLEMENTS CacheBackend:
    // A no-op cache — never stores anything; all gets return None
    // Useful to disable caching without changing application code

The Cache Key Construction

Raw cache keys must be properly namespaced and version-stamped to prevent collisions between different applications sharing the same cache and to enable mass invalidation:

FUNCTION make_cache_key(key: string, version: int | null = null) → string:
    prefix  = settings.CACHE_KEY_PREFIX    // e.g., "myapp"
    version = version OR settings.CACHE_VERSION  // Global cache version
    // Normalize the key: hash long keys to avoid exceeding max key length
    safe_key = md5(key.encode()).hexdigest() IF len(key) > 200 ELSE key.replace(" ", "_")
    RETURN f"{prefix}:{version}:{safe_key}"

View-Level Caching

Full-page caching stores the entire rendered HTML response for a given URL, serving subsequent requests from the cache without running the view or any database queries:

CLASS CacheMiddleware IMPLEMENTS Middleware:
    FUNCTION process(request, next_handler):
        // Only cache GET and HEAD requests
        IF request.method NOT IN {"GET", "HEAD"}: RETURN next_handler(request)
        // Don't cache authenticated requests by default
        IF request.user.is_authenticated AND NOT settings.CACHE_AUTHENTICATED:
            RETURN next_handler(request)
        
        cache_key = make_cache_key(f"view:{request.full_path}")
        cached = cache.get(cache_key)
        
        IF cached is not null:
            RETURN cached   // Serve from cache without calling the view
        
        response = next_handler(request)
        
        // Only cache successful responses without cookies
        IF response.status_code == 200 AND NOT response.cookies:
            timeout = get_max_age_from_cache_control(response) OR settings.CACHE_MIDDLEWARE_SECONDS
            cache.set(cache_key, response, timeout)
        
        RETURN response

// Per-view caching decorator:
FUNCTION cache_page(timeout: int, key_prefix: string = ""):
    FUNCTION decorator(view_func):
        FUNCTION wrapper(request, **kwargs):
            key = make_cache_key(f"{key_prefix}:{request.full_path}")
            cached = cache.get(key)
            IF cached: RETURN cached
            response = view_func(request, **kwargs)
            IF response.status_code == 200:
                cache.set(key, response, timeout)
            RETURN response
        RETURN wrapper
    RETURN decorator

Template Fragment Caching

Often you want to cache only part of a page — the navigation menu, the product list, the sidebar — not the entire response:

// Template tag: {% cache 3600 "product-list" %}...{% endcache %}
CLASS CacheNode IMPLEMENTS TemplateNode:
    FIELDS:
        timeout_expr:  FilterExpression
        fragment_name: FilterExpression
        vary_on:       List<FilterExpression>
        body_nodes:    List<TemplateNode>
    
    FUNCTION render(context):
        timeout = timeout_expr.resolve(context)
        name    = fragment_name.resolve(context)
        vary    = [v.resolve(context) FOR v IN vary_on]
        key     = make_cache_key(f"fragment:{name}:{':'.join(vary)}")
        
        cached = cache.get(key)
        IF cached is not null: RETURN cached
        
        result = render_children(body_nodes, context)
        cache.set(key, result, timeout)
        RETURN result

Checklist

  • Define the CacheBackend interface with all required methods.
  • Implement LocMemCacheBackend (in-memory dict) and DummyCacheBackend first for testing.
  • Implement RedisBackend and MemcachedBackend.
  • Implement DatabaseCacheBackend with a cache_table model and periodic cleanup.
  • Implement make_cache_key() with prefix, versioning, and key normalization.
  • Implement CacheMiddleware for full-page caching.
  • Implement the @cache_page(timeout) decorator.
  • Implement the {% cache %} template tag for fragment caching.
  • Implement cache.get_or_set(key, default_callable, timeout) convenience method.
  • Implement cache versioning and cache.incr_version() for mass invalidation.
  • Implement vary_on_headers and vary_on_cookie decorators (adds Vary response header).
  • Implement per-site and per-user cache key variations.

31. The Configuration & Settings System

GoF Pattern: Singleton (Settings Object) + Proxy (Lazy Loading) + Composite (Layered Settings)

The configuration system is the foundation every other subsystem depends on. It must be available before any other part of the framework initializes, it must be immutable after the application starts (to prevent accidental runtime mutations), and it must provide sensible defaults so that developers only need to specify what differs from the default.

The Settings Object Architecture

The settings object exposes all configuration as attributes. It is lazy-loaded — it does not read the settings file until the first attribute access. This lazy loading is implemented as a Proxy (GoF): the settings proxy forwards attribute access to the underlying settings module, which is only loaded on first access:

CLASS LazySettings:
    FIELDS:
        _settings_module:  module | null   // The loaded settings module
        _wrapped:          Settings | null  // The wrapped Settings object
    
    PROPERTY wrapped:
        IF _wrapped is null:
            module_name = get_env("FRAMEWORK_SETTINGS_MODULE")
            IF NOT module_name:
                RAISE ImproperlyConfigured(
                    "FRAMEWORK_SETTINGS_MODULE environment variable is not set."
                )
            self._wrapped = Settings(module_name)
        RETURN _wrapped
    
    FUNCTION __getattr__(name: string):
        RETURN getattr(self.wrapped, name)

CLASS Settings:
    FIELDS:
        SETTINGS_MODULE: string   // The settings file that was loaded
    
    CONSTRUCTOR(settings_module: string):
        // Load global defaults first
        FOR key, value IN global_settings.items():
            setattr(self, key, value)
        
        // Load user settings file (overrides defaults)
        user_settings = import_module(settings_module)
        FOR key IN dir(user_settings):
            IF key.isupper():  // Convention: settings are UPPERCASE
                setting_value = getattr(user_settings, key)
                setattr(self, key, setting_value)
        
        self.SETTINGS_MODULE = settings_module
        
        // Validate critical settings
        self._validate()
    
    FUNCTION _validate():
        IF NOT self.SECRET_KEY:
            RAISE ImproperlyConfigured("SECRET_KEY must be set.")
        IF len(self.SECRET_KEY) < 50:
            RAISE ImproperlyConfigured("SECRET_KEY is too short. Use at least 50 random chars.")
        IF self.DEBUG AND self.ALLOWED_HOSTS == []:
            RAISE ImproperlyConfigured(
                "ALLOWED_HOSTS is empty. Set it to ['*'] for development or your real domains."
            )

Global Defaults

Every setting must have a documented default. The global defaults file is the canonical documentation of all available settings:

// global_settings.py (or equivalent)

// Core
DEBUG =                False
SECRET_KEY =           ""       // Must be overridden
ALLOWED_HOSTS =        []

// Database
DATABASES =            {}       // Must be configured
DEFAULT_AUTO_FIELD =   "AutoField"   // Or "BigAutoField"

// Caching
CACHES = {
    "default": {
        "BACKEND": "framework.cache.backends.locmem.LocMemCache"
    }
}

// Templates
TEMPLATES = [
    {
        "BACKEND": "framework.template.backends.FrameworkTemplates",
        "DIRS": [],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "framework.template.context_processors.request",
                "framework.template.context_processors.auth",
                "framework.template.context_processors.csrf",
                "framework.template.context_processors.static",
            ],
        }
    }
]

// Authentication
AUTH_USER_MODEL =              "auth.User"
AUTHENTICATION_BACKENDS =      ["framework.auth.backends.ModelBackend"]
PASSWORD_HASHER =              "argon2"
LOGIN_URL =                    "/login/"
LOGIN_REDIRECT_URL =           "/dashboard/"
LOGOUT_REDIRECT_URL =          null  // null = don't redirect; show logout page

// Sessions
SESSION_ENGINE =               "framework.sessions.backends.db"
SESSION_COOKIE_NAME =          "sessionid"
SESSION_COOKIE_AGE =           1209600  // 2 weeks in seconds
SESSION_COOKIE_SECURE =        False
SESSION_COOKIE_HTTPONLY =      True
SESSION_COOKIE_SAMESITE =      "Lax"
SESSION_EXPIRE_AT_BROWSER_CLOSE = False
SESSION_SAVE_EVERY_REQUEST =   False

// Security
CSRF_COOKIE_NAME =             "csrftoken"
CSRF_COOKIE_SECURE =           False
CSRF_COOKIE_HTTPONLY =         False
CSRF_INPUT_NAME =              "csrfmiddlewaretoken"
CSRF_TRUSTED_ORIGINS =         []
SECURE_SSL_REDIRECT =          False
SECURE_HSTS_SECONDS =          0
SECURE_HSTS_INCLUDE_SUBDOMAINS = False
SECURE_HSTS_PRELOAD =          False
X_FRAME_OPTIONS =              "DENY"

// Email
EMAIL_BACKEND =   "framework.email.backends.smtp.EmailBackend"
EMAIL_HOST =      "localhost"
EMAIL_PORT =      25
EMAIL_USE_TLS =   False
DEFAULT_FROM_EMAIL = "webmaster@localhost"

// Static files
STATIC_URL =          "/static/"
STATIC_ROOT =         null   // Where collectstatic puts files
STATICFILES_DIRS =    []     // Additional directories to search
MEDIA_URL =           "/media/"
MEDIA_ROOT =          null   // Where uploaded files are stored

// Internationalization
LANGUAGE_CODE =       "en-us"
TIME_ZONE =           "UTC"
USE_I18N =            True
USE_TZ =              True   // Store datetimes in UTC

Settings Validation and Checks

A framework that silently accepts invalid configuration and then fails mysteriously at runtime is a terrible developer experience. Your framework should include a system check framework that validates settings, detects potential issues, and reports them with clear messages:

INTERFACE SystemCheck:
    FUNCTION run(app_registry, settings) → List<CheckMessage>

STRUCT CheckMessage:
    id:       string         // e.g., "security.W001" for a security warning
    level:    ENUM { DEBUG, INFO, WARNING, ERROR, CRITICAL }
    message:  string
    hint:     string | null
    obj:      any            // The model, view, or setting the check pertains to

// Example checks:
CLASS SecurityChecks IMPLEMENTS SystemCheck:
    FUNCTION run(app_registry, settings):
        messages = []
        IF settings.DEBUG:
            messages.add(CheckMessage("security.W001", WARNING, 
                                       "DEBUG=True should not be used in production."))
        IF NOT settings.SECRET_KEY:
            messages.add(CheckMessage("security.E001", ERROR, 
                                       "SECRET_KEY must be set."))
        IF settings.ALLOWED_HOSTS == ["*"]:
            messages.add(CheckMessage("security.W002", WARNING,
                                       "ALLOWED_HOSTS = ['*'] is insecure."))
        RETURN messages

Checklist

  • Implement LazySettings proxy that loads settings on first access.
  • Implement Settings loader that merges global defaults with user settings.
  • Implement global_settings with documented defaults for every setting.
  • Implement settings validation: SECRET_KEY, ALLOWED_HOSTS, database config.
  • Implement the system check framework with SystemCheck interface.
  • Implement security-specific checks (using DEBUG, secret key length, HTTPS settings).
  • Implement model-specific checks (field type validation, relationship consistency).
  • Implement check management command that runs all checks and reports results.
  • Ensure settings are effectively read-only after application startup.
  • Support environment variable overrides for settings (useful for 12-factor app deployment).

32. The Application Registry & Plugin System

GoF Pattern: Observer + Singleton + Factory Method

The application registry is the framework's catalog of all installed applications (called "apps"). Each app is a modular unit of functionality — a collection of models, views, URLs, templates, static files, management commands, and signal handlers. The registry manages their lifecycle: discovery, initialization, and ready-state signaling.

What an App Is

An app is a directory with at least an apps.py file (or equivalent configuration) that declares its metadata. A well-designed app is self-contained: it carries its own models, its own URL patterns, its own templates, and its own static files. It does not reach into other apps' internals.

CLASS AppConfig:
    FIELDS:
        name:           string    // The Python module path: "myapp"
        label:          string    // Short name for prefix: "myapp"
        verbose_name:   string    // Human-readable: "My Application"
        path:           string    // Filesystem path to the app directory
        models_module:  module    // The app's models.py (or equivalent)
        
    FUNCTION ready():
        // Called after all apps have been loaded.
        // The correct place to register signal handlers.
        // Do NOT do database queries here — they may not be ready.
        from .signals import connect_handlers
        connect_handlers()
    
    FUNCTION get_models() → List<Model>:
        RETURN all model classes that are registered under this app

// Example app config:
CLASS BlogAppConfig EXTENDS AppConfig:
    name         = "blog"
    verbose_name = "Blog"
    
    FUNCTION ready():
        from .models import Article
        from .handlers import on_article_published
        from myapp.signals import article_published
        article_published.connect(on_article_published, sender=Article)

The App Registry

CLASS Apps:
    FIELDS:
        app_configs:      OrderedDict<string, AppConfig>  // label → config
        all_models:       dict<string, dict<string, Model>>  // label → {name → Model}
        ready:            boolean
        _pending_lookups: dict    // For resolving lazy model references in ForeignKey
    
    FUNCTION populate(installed_apps: List<string>):
        IF ready: RETURN
        
        // Phase 1: Create AppConfig objects from string names
        FOR entry IN installed_apps:
            config = create_app_config(entry)  // Factory Method
            app_configs[config.label] = config
        
        // Phase 2: Import all models modules
        FOR config IN app_configs.values():
            config.import_models()
        
        // Phase 3: Run ready() on each app
        FOR config IN app_configs.values():
            config.ready()
        
        self.ready = True
    
    FUNCTION register_model(model: Model class):
        app_label = model._meta.app_label
        model_name = model._meta.model_name
        
        IF app_label NOT IN all_models:
            all_models[app_label] = {}
        
        all_models[app_label][model_name] = model
        
        // Resolve any pending lazy references to this model
        resolve_pending_lookups(app_label, model_name, model)
    
    FUNCTION get_model(app_label: string, model_name: string) → Model:
        TRY:
            RETURN all_models[app_label][model_name.lower()]
        EXCEPT KeyError:
            RAISE LookupError(f"No model '{model_name}' in app '{app_label}'")
    
    FUNCTION get_models(include_auto_created=False) → List<Model>:
        models = []
        FOR app_config IN app_configs.values():
            models.extend(app_config.get_models(include_auto_created))
        RETURN models

Lazy Model References

When a ForeignKey is declared with a string (ForeignKey("blog.Article")), the referenced model may not have been imported yet. The registry resolves these lazy references:

FUNCTION resolve_lazy_model_reference(model_string: string) → Model | None:
    IF "." IN model_string:
        app_label, model_name = model_string.split(".", 1)
    ELSE:
        // Assume same app
        app_label = current_app.label
        model_name = model_string
    
    TRY:
        RETURN apps.get_model(app_label, model_name)
    EXCEPT LookupError:
        // Model not yet registered; add to pending lookups
        pending_lookups.setdefault(f"{app_label}.{model_name}", []).append(current_field)
        RETURN null

Checklist

  • Implement AppConfig with name, label, verbose_name, path, and ready().
  • Implement Apps.populate() with three-phase initialization (discover → import models → ready).
  • Implement Apps.register_model() with lazy reference resolution.
  • Implement Apps.get_model(app_label, model_name).
  • Implement Apps.get_models() for migration and admin use.
  • Implement the INSTALLED_APPS loading loop with proper error handling for missing apps.
  • Ensure ready() is only called after all models in all apps have been imported.
  • Implement the check framework integration: collect checks from all apps.
  • Implement app template and static file directory registration for the template and static file finders.

33. Static Files & the Asset Pipeline

The Fundamental Problem: Serving Static Files

Static files are files that are served as-is without any processing: CSS files, JavaScript files, images, fonts. In development, the framework should serve them directly for convenience. In production, they should be served by a dedicated server (nginx, a CDN) for performance — serving static files through Python is 100x slower than serving them from nginx's sendfile.

The framework provides three mechanisms: a development file server (should never be used in production), a file finder system that locates static files from all installed apps and configured directories, and a collectstatic management command that gathers all static files into a single directory for deployment.

The Static File Finder System

INTERFACE StaticFilesFinder:
    FUNCTION find(path: string, all=False) → string | List<string> | null
    FUNCTION list(ignore_patterns: List<string>) → Iterator<(path, storage)>

CLASS FileSystemFinder IMPLEMENTS StaticFilesFinder:
    // Searches STATICFILES_DIRS
    FIELDS:
        locations: List<(prefix, root)>   // From STATICFILES_DIRS setting
    
    FUNCTION find(path, all=False):
        results = []
        FOR prefix, root IN locations:
            // path may start with prefix; strip it
            IF path.startswith(prefix):
                matched_path = os.path.join(root, path[len(prefix):])
                IF os.path.isfile(matched_path):
                    IF NOT all: RETURN matched_path
                    results.append(matched_path)
        RETURN results IF all ELSE null

CLASS AppDirectoriesFinder IMPLEMENTS StaticFilesFinder:
    // Looks in <app>/static/ directories within each installed app
    FUNCTION find(path, all=False):
        results = []
        FOR app_config IN apps.get_app_configs():
            static_dir = os.path.join(app_config.path, "static")
            full_path  = os.path.join(static_dir, path)
            IF os.path.isfile(full_path):
                IF NOT all: RETURN full_path
                results.append(full_path)
        RETURN results IF all ELSE null

The collectstatic Command

The collectstatic command gathers all static files from all finders and copies them to STATIC_ROOT. This directory is what nginx serves in production:

FUNCTION collectstatic(clear=False, dry_run=False):
    storage = StaticFilesStorage(settings.STATIC_ROOT)
    
    IF clear: storage.clear()
    
    copied = 0
    FOR finder IN configured_finders:
        FOR path, file_storage IN finder.list(ignore_patterns=settings.STATICFILES_IGNORE_PATTERNS):
            destination = storage.path(path)
            source      = file_storage.path(path)
            
            IF NOT dry_run:
                IF NOT os.path.exists(destination) OR is_newer(source, destination):
                    copy_file(source, destination)
                    copied++
            
            log(f"  {path}")
    
    log(f"{copied} static files copied to '{settings.STATIC_ROOT}'.")

Cache Busting — Content Hashing

In production, static files are often served with very long cache headers (Cache-Control: max-age=31536000, immutable). But this means that when you update a CSS file, users' browsers continue serving the old cached version. The solution is content-based hashing: include a hash of the file's content in its URL so that changed files always have new URLs.

// Instead of: /static/css/main.css
// Use:        /static/css/main.abc123.css  (where abc123 is a hash of the content)

CLASS ManifestStaticFilesStorage EXTENDS StaticFilesStorage:
    FIELDS:
        manifest: dict<original_path, hashed_path>  // Loaded from staticfiles.json
    
    FUNCTION url(name: string) → string:
        // Replace original path with content-hashed path
        hashed_name = manifest.get(name, name)
        RETURN settings.STATIC_URL + hashed_name
    
    FUNCTION post_process(paths):
        // After collecting, compute content hashes and update the manifest
        FOR path IN paths:
            content = read_file(path)
            hash    = md5(content).hexdigest()[:8]
            hashed_path = add_hash_to_filename(path, hash)
            rename_file(path, hashed_path)
            manifest[path] = hashed_path
        
        write_manifest(manifest)

Checklist

  • Implement StaticFilesFinder interface.
  • Implement FileSystemFinder for STATICFILES_DIRS.
  • Implement AppDirectoriesFinder for <app>/static/ directories.
  • Implement the development static file server view (only enabled when DEBUG=True).
  • Implement collectstatic management command.
  • Implement ManifestStaticFilesStorage for content-hashed static files.
  • Implement the {% static %} template tag.
  • Implement the MEDIA_URL/MEDIA_ROOT serving for development (user-uploaded files).
  • Document that static files should never be served by the framework in production.

34. The Admin Interface

GoF Pattern: Template Method + Factory Method + Composite

The admin interface is one of the most beloved features of Django and a significant competitive advantage for any framework. It is a fully functional, automatically generated CRUD interface for all registered models. Every developer who has built a web application has spent hours writing admin-like interfaces by hand — the framework generates them for free.

The admin is built on top of every system we have built: the ORM, the form system, the template engine, the authentication system, and the view system. It is the ultimate integration test of the entire framework.

The ModelAdmin — Template Method Pattern

ModelAdmin defines the configuration and behaviour for managing one model in the admin. It uses the Template Method pattern: the base class defines the lifecycle of admin operations (list, add, change, delete), and the developer overrides specific hooks to customize behaviour:

CLASS ModelAdmin:
    FIELDS:
        model:              Model class
        list_display:       List<string>   // Fields shown in the list view columns
        list_filter:        List<string>   // Fields to create sidebar filters for
        list_per_page:      integer = 100
        search_fields:      List<string>   // Fields to search when using the search box
        ordering:           List<string>   // Default ordering
        readonly_fields:    List<string>   // Fields shown but not editable
        fieldsets:          List<tuple>    // Organize form fields into sections
        inlines:            List<InlineModelAdmin>  // Inline editing of related objects
        actions:            List<Callable>    // Bulk actions (checkboxes in list view)
        form:               Form class | null  // Custom form class
        exclude:            List<string>       // Fields to exclude from the form
        raw_id_fields:      List<string>       // Use ID input for FK fields
        autocomplete_fields: List<string>      // Use AJAX autocomplete for FK
    
    // Template Method hooks — override these to customize admin behaviour:
    
    FUNCTION get_queryset(request) → QuerySet:
        // Override to customize which objects are shown
        RETURN model.objects.all()
    
    FUNCTION has_add_permission(request) → boolean:
        RETURN request.user.has_perm(f"{app_label}.add_{model_name}")
    
    FUNCTION has_change_permission(request, obj=null) → boolean:
        RETURN request.user.has_perm(f"{app_label}.change_{model_name}")
    
    FUNCTION has_delete_permission(request, obj=null) → boolean:
        RETURN request.user.has_perm(f"{app_label}.delete_{model_name}")
    
    FUNCTION save_model(request, obj, form, change):
        // Override to add custom save logic (e.g., set created_by = request.user)
        obj.save()
    
    FUNCTION delete_model(request, obj):
        obj.delete()
    
    FUNCTION get_form(request, obj=null, **kwargs) → Form class:
        // Generate or return the form class for this model
        RETURN modelform_factory(model, fields=get_fields_for_form(), exclude=exclude, **kwargs)
    
    FUNCTION save_formset(request, form, formset, change):
        formset.save()
    
    // Admin views — these implement the CRUD views:
    FUNCTION changelist_view(request) → Response    // List all objects
    FUNCTION add_view(request) → Response          // Form to add a new object
    FUNCTION change_view(request, object_id) → Response   // Form to edit an object
    FUNCTION delete_view(request, object_id) → Response   // Confirm deletion
    FUNCTION history_view(request, object_id) → Response  // Change history log

The Admin Site — Registry

The AdminSite is the registry of all ModelAdmin instances. When an app registers a model with the admin, it associates the model with a ModelAdmin configuration:

CLASS AdminSite:
    FIELDS:
        _registry:    dict<Model, ModelAdmin>   // model class → admin config
        name:         string = "admin"           // URL namespace
        site_header:  string = "Admin"
        site_title:   string = "Admin Site"
        index_title:  string = "Site Administration"
    
    FUNCTION register(model_or_models, admin_class=ModelAdmin, **options):
        models = model_or_models IF isinstance(model_or_models, list) ELSE [model_or_models]
        FOR model IN models:
            IF model IN _registry:
                RAISE AlreadyRegistered(f"{model.__name__} is already registered.")
            admin_obj = admin_class(model, self)
            _registry[model] = admin_obj
    
    FUNCTION unregister(model_or_models):
        models = model_or_models IF isinstance(model_or_models, list) ELSE [model_or_models]
        FOR model IN models:
            IF model NOT IN _registry:
                RAISE NotRegistered(f"{model.__name__} is not registered.")
            del _registry[model]
    
    FUNCTION get_urls() → URLResolver:
        patterns = [path("", self.index_view, name="index")]
        FOR model, admin_obj IN _registry.items():
            info = (model._meta.app_label, model._meta.model_name)
            patterns.extend([
                path(f"{info[0]}/{info[1]}/",          admin_obj.changelist_view, name=f"{info[0]}_{info[1]}_changelist"),
                path(f"{info[0]}/{info[1]}/add/",      admin_obj.add_view,        name=f"{info[0]}_{info[1]}_add"),
                path(f"{info[0]}/{info[1]}/<pk>/change/", admin_obj.change_view,  name=f"{info[0]}_{info[1]}_change"),
                path(f"{info[0]}/{info[1]}/<pk>/delete/", admin_obj.delete_view,  name=f"{info[0]}_{info[1]}_delete"),
                path(f"{info[0]}/{info[1]}/<pk>/history/", admin_obj.history_view, name=f"{info[0]}_{info[1]}_history"),
            ])
        RETURN URLResolver("/admin/", patterns)

// The default admin site:
site = AdminSite()

// How apps register models:
FROM myapp.models IMPORT Article, Author
site.register(Author)
site.register(Article, ArticleAdmin)   // ArticleAdmin extends ModelAdmin with custom config

Inline Editing

Inlines allow editing related objects on the same page as the parent object — for example, editing all OrderItems on the Order change page:

CLASS InlineModelAdmin:
    FIELDS:
        model:          Model class
        extra:          integer = 3      // Extra empty forms shown
        min_num:        integer = 0
        max_num:        integer | null = null
        can_delete:     boolean = True
        show_change_link: boolean = False
        fk_name:        string | null

CLASS TabularInline EXTENDS InlineModelAdmin:
    // Renders inline forms in a table layout
    template = "admin/edit_inline/tabular.html"

CLASS StackedInline EXTENDS InlineModelAdmin:
    // Renders inline forms stacked vertically (one field per row)
    template = "admin/edit_inline/stacked.html"

Checklist

  • Implement ModelAdmin with all configuration options and Template Method hooks.
  • Implement AdminSite.register(), unregister(), and get_urls().
  • Implement changelist_view with pagination, search, list filters, and sortable columns.
  • Implement add_view and change_view with form generation and save.
  • Implement delete_view with confirmation and cascade information.
  • Implement inline model admin (TabularInline and StackedInline).
  • Implement bulk actions (with a checkbox per row and a dropdown of registered actions).
  • Implement change history logging (LogEntry model recording who changed what and when).
  • Implement admin authentication: require is_staff=True for all admin views.
  • Implement the admin index page showing all registered apps and models.
  • Implement admin autocomplete for ForeignKey fields.

35. The Email Subsystem

GoF Pattern: Abstract Factory (Email Backends) + Builder (Email Message)

The email subsystem provides a consistent API for sending emails regardless of how they are delivered. In development, emails should be printed to the console or written to a file. In testing, they should be captured in memory. In production, they should be sent via SMTP or an API (Mailgun, SendGrid, AWS SES).

Email Backends — GoF Abstract Factory

INTERFACE EmailBackend:
    FUNCTION send_messages(messages: List<EmailMessage>) → integer  // Returns count sent
    FUNCTION open()    // Open connection (for batch sending)
    FUNCTION close()   // Close connection

CLASS SMTPEmailBackend IMPLEMENTS EmailBackend:
    FIELDS:
        host:     string
        port:     integer
        username: string
        password: string
        use_tls:  boolean
        timeout:  integer
        connection: SMTPConnection | null
    
    FUNCTION send_messages(messages):
        IF NOT connection:
            self.open()
        count = 0
        FOR message IN messages:
            TRY:
                message.send_via_smtp(connection)
                count++
            EXCEPT SMTPException AS e:
                IF not fail_silently: RAISE
        RETURN count

CLASS ConsoleEmailBackend IMPLEMENTS EmailBackend:
    // Prints emails to stdout/stderr — for development
    FUNCTION send_messages(messages):
        FOR message IN messages:
            print(f"\n{'='*70}\nTo: {message.to}\nSubject: {message.subject}\n{message.body}\n{'='*70}")
        RETURN len(messages)

CLASS InMemoryEmailBackend IMPLEMENTS EmailBackend:
    // Stores sent emails in a class-level list — for testing
    outbox: List<EmailMessage> = []   // Read from tests to assert email was sent
    FUNCTION send_messages(messages):
        InMemoryEmailBackend.outbox.extend(messages)
        RETURN len(messages)

CLASS FileEmailBackend IMPLEMENTS EmailBackend:
    // Writes each email to a .eml file — for development

The EmailMessage Builder

CLASS EmailMessage:
    FIELDS:
        subject:      string
        body:         string           // Plain text body
        from_email:   string           // Defaults to DEFAULT_FROM_EMAIL
        to:           List<string>
        cc:           List<string>
        bcc:          List<string>
        reply_to:     List<string>
        headers:      dict<string, string>
        attachments:  List<Attachment>
        alternatives: List<(content, mimetype)>  // For HTML alternatives
    
    FUNCTION attach(filename: string, content: bytes, mimetype: string):
        attachments.add(Attachment(filename, content, mimetype))
    
    FUNCTION attach_file(path: string, mimetype=null):
        mimetype = mimetype OR guess_type(path)
        content  = read_file_bytes(path)
        attach(os.path.basename(path), content, mimetype)
    
    FUNCTION send(fail_silently=False) → boolean:
        backend = get_email_backend()
        RETURN backend.send_messages([self]) == 1

CLASS EmailMultiAlternatives EXTENDS EmailMessage:
    // An email that includes both a plain text and an HTML version
    // The email client chooses which to display
    FUNCTION attach_alternative(content: string, mimetype: string):
        alternatives.add((content, mimetype))
    
    // Usage:
    // msg = EmailMultiAlternatives("Subject", "Plain text", "from@example.com", ["to@example.com"])
    // msg.attach_alternative("<h1>Hello</h1>", "text/html")
    // msg.send()

Template-Based Email Rendering

Real applications rarely send plain text emails. They render email templates (often both HTML and plain text versions):

FUNCTION send_templated_email(template_prefix: string,
                               context: dict,
                               to: List<string>,
                               subject: string | null = null,
                               from_email: string | null = null):
    // Renders both {prefix}.txt and {prefix}.html templates
    context_obj = RenderContext(context)
    
    subject   = subject OR render_to_string(f"{template_prefix}_subject.txt", context).strip()
    body_text = render_to_string(f"{template_prefix}_body.txt", context)
    body_html = render_to_string(f"{template_prefix}_body.html", context)
    
    msg = EmailMultiAlternatives(
        subject=subject,
        body=body_text,
        from_email=from_email OR settings.DEFAULT_FROM_EMAIL,
        to=to
    )
    msg.attach_alternative(body_html, "text/html")
    msg.send()

Checklist

  • Implement the EmailBackend interface.
  • Implement SMTPEmailBackend with TLS support and connection reuse.
  • Implement ConsoleEmailBackend and InMemoryEmailBackend.
  • Implement FileEmailBackend.
  • Implement EmailMessage with attachments and custom headers.
  • Implement EmailMultiAlternatives for HTML email.
  • Implement send_templated_email() for template-based emails.
  • Implement send_mail() shortcut function for simple one-off emails.
  • Implement send_mass_mail() for efficient batch sending.
  • Implement mail_admins() and mail_managers() for framework-generated error alerts.
  • Test that InMemoryEmailBackend.outbox is populated correctly in tests.

36. Background Tasks & the Job Queue

GoF Pattern: Command + Observer + Strategy (Queue Backends)

Long-running tasks — sending emails, generating PDFs, processing images, calling external APIs — should never be performed during a web request. A user should not wait 30 seconds for a PDF to be generated before seeing their confirmation page. The solution is a job queue: the web request enqueues a task (a Command object) describing the work to be done, returns a response immediately, and a separate worker process picks up and executes the task asynchronously.

The Task System

INTERFACE Task:                   // GoF Command
    FUNCTION execute(args: dict)   // The actual work
    FUNCTION on_success(result)    // Called after successful execution
    FUNCTION on_failure(error)     // Called after failed execution
    FUNCTION on_retry(error, countdown)   // Called when task is being retried
    
    max_retries:    integer    // How many times to retry on failure
    retry_backoff:  boolean    // Exponential backoff between retries
    priority:       integer    // Higher = processed sooner

CLASS FunctionTask EXTENDS Task:
    // Wraps a plain function as a Task
    FIELDS:
        func:       Callable
        func_args:  tuple
        func_kwargs: dict
    
    FUNCTION execute(args):
        RETURN func(*func_args, **func_kwargs)

// The @task decorator converts a function into a Task descriptor:
FUNCTION task(max_retries=3, priority=5, queue="default"):
    FUNCTION decorator(func):
        RETURN TaskDescriptor(func, max_retries, priority, queue)
    RETURN decorator

CLASS TaskDescriptor:
    FUNCTION delay(*args, **kwargs):
        // Enqueue the task to be run asynchronously
        task_instance = FunctionTask(self.func, args, kwargs)
        job_queue.enqueue(task_instance, queue=self.queue)
    
    FUNCTION apply_async(*args, **kwargs, countdown=0, eta=null):
        // Enqueue with optional delay or ETA
        job_queue.enqueue(FunctionTask(self.func, args, kwargs),
                          queue=self.queue, countdown=countdown, eta=eta)
    
    FUNCTION __call__(*args, **kwargs):
        // Call synchronously (useful in tests)
        RETURN self.func(*args, **kwargs)

Queue Backends

INTERFACE QueueBackend:
    FUNCTION enqueue(task, queue, priority, countdown, eta)
    FUNCTION dequeue(queue, timeout) → TaskMessage | null
    FUNCTION ack(task_id)          // Mark task as successfully processed
    FUNCTION nack(task_id)         // Return task to queue (for retry)
    FUNCTION get_status(task_id) → TaskStatus

CLASS RedisQueueBackend IMPLEMENTS QueueBackend:
    // Uses Redis lists (LPUSH/BRPOP) for simple FIFO queues
    // Uses sorted sets with the ETA as the score for scheduled tasks

CLASS DatabaseQueueBackend IMPLEMENTS QueueBackend:
    // Uses a database table to store pending tasks
    // SELECT ... FOR UPDATE SKIP LOCKED for safe concurrent dequeue

CLASS InMemoryQueueBackend IMPLEMENTS QueueBackend:
    // For testing: tasks run synchronously or are stored in a list

The Worker Process

CLASS Worker:
    FIELDS:
        queues:     List<string>    // Which queues this worker listens to
        backend:    QueueBackend
        max_tasks:  integer         // Maximum tasks before respawning (prevents memory leaks)
        
    FUNCTION run():
        tasks_processed = 0
        
        WHILE tasks_processed < max_tasks:
            // Poll all configured queues in priority order
            message = null
            FOR queue IN queues:
                message = backend.dequeue(queue, timeout=5)
                IF message: BREAK
            
            IF message is null: CONTINUE   // Timeout; poll again
            
            TRY:
                task = deserialize_task(message)
                result = task.execute(message.args)
                task.on_success(result)
                backend.ack(message.id)
            EXCEPT Exception AS e:
                task.on_failure(e)
                IF task.retry_count < task.max_retries:
                    countdown = 2 ** task.retry_count IF task.retry_backoff ELSE 0
                    backend.enqueue(task, countdown=countdown)
                backend.nack(message.id)
            
            tasks_processed++

Checklist

  • Implement the Task interface and FunctionTask class.
  • Implement the @task decorator and TaskDescriptor.
  • Implement QueueBackend with RedisQueueBackend and DatabaseQueueBackend.
  • Implement InMemoryQueueBackend for testing.
  • Implement Worker.run() with retry and backoff logic.
  • Implement countdown (run after N seconds) and eta (run at a specific datetime) scheduling.
  • Implement task result storage: store the return value or exception for task.delay().get().
  • Implement task revocation: cancel a pending task before it runs.
  • Implement a management command to start worker processes.
  • Implement task monitoring: a view or command that shows queue length and task status.

37. WebSocket & Real-Time Support

GoF Pattern: State + Observer

WebSockets provide a persistent, bidirectional communication channel between the browser and the server. Unlike HTTP requests (which are one request → one response), a WebSocket connection stays open for the life of a session and either side can send messages at any time. This is used for live notifications, chat, collaborative editing, and live data dashboards.

WebSocket support requires ASGI (Step 5) — it cannot be built on top of synchronous WSGI.

The WebSocket Connection State Machine

A WebSocket connection passes through distinct states, and the server must manage these transitions correctly. This is the State pattern (GoF):

ENUM WebSocketState:
    CONNECTING    // TCP connection established; HTTP upgrade handshake in progress
    OPEN          // Handshake complete; messages can flow in both directions
    CLOSING       // Either side has initiated the closing handshake
    CLOSED        // Connection is fully closed

CLASS WebSocketConnection:
    FIELDS:
        state:      WebSocketState
        scope:      dict          // ASGI scope with URL, headers, user, etc.
        receive_fn: Callable      // ASGI receive; awaitable; yields events
        send_fn:    Callable      // ASGI send; awaitable; accepts events
        groups:     Set<string>   // Channel groups this connection is subscribed to
    
    ASYNC FUNCTION accept(subprotocol=null):
        IF state != CONNECTING: RAISE RuntimeError("Can only accept in CONNECTING state")
        await send_fn({"type": "websocket.accept", "subprotocol": subprotocol})
        state = OPEN
    
    ASYNC FUNCTION send_text(text: string):
        IF state != OPEN: RAISE RuntimeError("Cannot send on closed WebSocket")
        await send_fn({"type": "websocket.send", "text": text})
    
    ASYNC FUNCTION send_json(data: any):
        await send_text(json_encode(data))
    
    ASYNC FUNCTION receive_text() → string:
        IF state != OPEN: RAISE RuntimeError("Cannot receive on closed WebSocket")
        event = await receive_fn()
        IF event["type"] == "websocket.disconnect":
            state = CLOSED
            RAISE WebSocketDisconnect(code=event.get("code", 1000))
        RETURN event.get("text", "")
    
    ASYNC FUNCTION close(code=1000):
        await send_fn({"type": "websocket.close", "code": code})
        state = CLOSED

The WebSocket Consumer — Template Method

The WebSocket consumer is the view equivalent for WebSocket connections. It defines the template method lifecycle:

CLASS WebSocketConsumer:
    ASYNC FUNCTION __call__(scope, receive, send):
        self.scope = scope
        self.connection = WebSocketConnection(scope, receive, send)
        await self.websocket_connect()
    
    // Template Method hooks:
    ASYNC FUNCTION websocket_connect():
        await self.connection.accept()
        await self.connect()
        await self._receive_loop()
    
    ASYNC FUNCTION _receive_loop():
        TRY:
            WHILE True:
                text = await self.connection.receive_text()
                IF is_json(text):
                    await self.receive_json(json_decode(text))
                ELSE:
                    await self.receive(text)
        EXCEPT WebSocketDisconnect AS e:
            await self.disconnect(e.code)
    
    // Override these in your consumer class:
    ASYNC FUNCTION connect():    PASS    // Called when connection is established
    ASYNC FUNCTION disconnect(code): PASS   // Called when connection is closed
    ASYNC FUNCTION receive(text): PASS   // Called when text message arrives
    ASYNC FUNCTION receive_json(data): PASS   // Called when JSON message arrives

// Usage example:
CLASS ChatConsumer EXTENDS WebSocketConsumer:
    ASYNC FUNCTION connect():
        room_name = scope["url_route"]["kwargs"]["room_name"]
        self.room_group = f"chat_{room_name}"
        await channel_layer.group_add(self.room_group, self.connection)
        await self.connection.accept()
    
    ASYNC FUNCTION disconnect(code):
        await channel_layer.group_discard(self.room_group, self.connection)
    
    ASYNC FUNCTION receive_json(data):
        message = data["message"]
        await channel_layer.group_send(self.room_group, {
            "type": "chat.message",
            "message": message,
            "sender": self.scope["user"].username
        })
    
    ASYNC FUNCTION chat_message(event):
        await self.send_json({"message": event["message"], "sender": event["sender"]})

Channel Layers — Broadcasting to Groups

To send a message to multiple connected clients (e.g., everyone in a chat room), you need a channel layer — a message-passing backend shared across all server processes. Redis is the standard backend:

INTERFACE ChannelLayer:
    ASYNC FUNCTION send(channel_name: string, message: dict)
    ASYNC FUNCTION receive(channel_name: string) → dict
    ASYNC FUNCTION group_add(group: string, channel_name: string)
    ASYNC FUNCTION group_discard(group: string, channel_name: string)
    ASYNC FUNCTION group_send(group: string, message: dict)

CLASS RedisChannelLayer IMPLEMENTS ChannelLayer:
    // Each WebSocket connection has a unique channel name (a Redis list key)
    // group_send() looks up all channel names in the group and sends to each

Checklist

  • Implement WebSocketConnection with accept(), send_text(), receive_text(), close().
  • Implement the WebSocketConsumer base class with the Template Method pattern.
  • Implement the ASGI WebSocket handler that routes connections to consumers.
  • Implement ChannelLayer interface and RedisChannelLayer.
  • Implement group_add(), group_discard(), group_send() for broadcasting.
  • Integrate WebSocket URL routing with the main URL router.
  • Implement authentication for WebSocket connections (read user from session in scope).
  • Implement WebSocket heartbeat (ping/pong) to detect dead connections.
  • Implement graceful server shutdown that closes all open WebSocket connections.

38. The CLI & Management Commands

GoF Pattern: Command + Composite (Command Groups)

The framework provides a command-line interface (CLI) for development and operational tasks: running the development server, creating database migrations, applying migrations, collecting static files, creating a superuser, clearing caches, running shell sessions, and more. Third-party apps and user applications can add their own management commands.

The Management Command Interface

INTERFACE ManagementCommand:      // GoF Command
    help:        string    // Short description shown in help
    
    FUNCTION add_arguments(parser: ArgumentParser)  // Register CLI arguments
    FUNCTION handle(**options)                       // The command logic
    FUNCTION execute(**options)                      // Handles setup before handle()
    
    output_transaction: boolean  // If True, wrap output in a transaction

CLASS BaseCommand IMPLEMENTS ManagementCommand:
    FIELDS:
        stdout: file-like object
        stderr: file-like object
        style:  ColorStyle   // Terminal color styling utilities
    
    FUNCTION execute(**options):
        // Setup: configure settings, check system (if --check was passed)
        IF options.get("check"):
            system_checks.run_checks()
        
        output = self.handle(**options)
        IF output:
            self.stdout.write(str(output))
    
    FUNCTION print_help():
        // Print the command's help text and argument list

// Example built-in commands:
CLASS RunServerCommand EXTENDS BaseCommand:
    help = "Starts a lightweight Web server for development."
    
    FUNCTION add_arguments(parser):
        parser.add_argument("addrport", nargs="?", default="127.0.0.1:8000")
        parser.add_argument("--reload", action="store_true")
    
    FUNCTION handle(addrport, reload, **options):
        host, port = parse_addrport(addrport)
        server = DevelopmentServer(host, int(port), reload=reload)
        server.run()

CLASS MakeMigrationsCommand EXTENDS BaseCommand:
    FUNCTION handle(app_labels, **options):
        autodetector = MigrationAutodetector(
            from_state=loader.project_state(),
            to_state=ProjectState.from_apps(registry)
        )
        changes = autodetector.detect_changes()
        IF NOT changes:
            self.stdout.write("No changes detected.")
        ELSE:
            writer.write_migration_files(changes)

CLASS MigrateCommand EXTENDS BaseCommand:
    FUNCTION handle(app_label, migration_name, **options):
        executor = MigrationExecutor(connections["default"])
        executor.migrate(targets=targets)

The CLI Entry Point

FUNCTION execute_from_command_line(argv: List<string>):
    // The main entry point: parse the command name and dispatch
    IF len(argv) < 2:
        print_main_help()
        sys.exit(1)
    
    command_name = argv[1]
    command = find_command(command_name)   // Look in built-in and app commands
    
    IF command is null:
        raise CommandError(f"Unknown command: '{command_name}'")
    
    command.run_from_argv(argv)

Auto-Discovery of App Commands

Each app can define management commands in a management/commands/ directory. The framework discovers these automatically:

FUNCTION find_command(name: string) → ManagementCommand | null:
    // First look in built-in commands
    IF name IN built_in_commands:
        RETURN built_in_commands[name]
    
    // Then look in each installed app's management/commands directory
    FOR app_config IN registry.get_app_configs():
        command_path = os.path.join(app_config.path, "management", "commands", f"{name}.py")
        IF os.path.exists(command_path):
            module = import_module_from_path(command_path)
            RETURN module.Command()
    
    RETURN null

Checklist

  • Implement the ManagementCommand interface.
  • Implement BaseCommand with argument parsing, execute-before-handle, stdout/stderr.
  • Implement all built-in commands: runserver, shell, makemigrations, migrate, collectstatic, createsuperuser, changepassword, check, flush, dumpdata, loaddata.
  • Implement auto-discovery of app-provided commands from management/commands/.
  • Implement --verbosity (0-3) for controlling output level.
  • Implement --settings and --pythonpath for specifying settings from the CLI.
  • Implement color output with --no-color to disable it.
  • Implement the help command and --help flag on all commands.

39. Logging, Monitoring & Observability

GoF Pattern: Observer (Log Handlers as Observers) + Chain of Responsibility (Log Level Filtering)

Logging is a cross-cutting concern that must be accessible from every layer of the framework. The logging system must be fast (a log statement that never produces visible output should have near-zero overhead), structured (log records should contain machine-parseable data, not just human-readable strings), and configurable (different environments need different log levels and destinations).

The Logging Architecture

// The logging system is organized as a hierarchy of loggers.
// Each logger has a name that forms a dotted hierarchy: "myapp", "myapp.views", "framework.orm"
// Propagation: if "myapp.views" has no handler, the log record goes up to "myapp", then to the root.

STRUCT LogRecord:
    logger_name:  string
    level:        ENUM { DEBUG=10, INFO=20, WARNING=30, ERROR=40, CRITICAL=50 }
    message:      string
    timestamp:    datetime
    process_id:   integer
    thread_id:    integer
    pathname:     string
    lineno:       integer
    exc_info:     ExceptionInfo | null
    extra:        dict    // Custom structured fields

CLASS Logger:
    FIELDS:
        name:     string
        level:    integer
        handlers: List<LogHandler>
        parent:   Logger | null
        propagate: boolean
    
    FUNCTION debug(message, *args, **kwargs):   log(DEBUG, message, *args, **kwargs)
    FUNCTION info(message, *args, **kwargs):    log(INFO, message, *args, **kwargs)
    FUNCTION warning(message, *args, **kwargs): log(WARNING, message, *args, **kwargs)
    FUNCTION error(message, *args, **kwargs):   log(ERROR, message, *args, **kwargs)
    FUNCTION exception(message, *args, **kwargs):
        // Shorthand for error() with exc_info=True
        log(ERROR, message, *args, exc_info=True, **kwargs)
    
    FUNCTION log(level, message, *args, exc_info=False, extra={}):
        IF level < self.level: RETURN   // Below threshold; fast exit
        
        record = LogRecord(name, level, format_message(message, args), now(), ...)
        self.handle(record)
    
    FUNCTION handle(record):
        FOR handler IN handlers:
            IF record.level >= handler.level:
                handler.emit(record)
        IF propagate AND parent:
            parent.handle(record)

Structured Logging for Production

In production, log records should be emitted as JSON objects, not formatted strings. This allows log aggregation systems (Elasticsearch, Splunk, Datadog) to index and query log data efficiently:

CLASS JsonFormatter:
    FUNCTION format(record: LogRecord) → string:
        data = {
            "timestamp":   record.timestamp.isoformat(),
            "level":       record.level_name,
            "logger":      record.logger_name,
            "message":     record.message,
            "process":     record.process_id,
        }
        IF record.exc_info:
            data["exception"] = format_exception(record.exc_info)
        IF record.extra:
            data.update(record.extra)
        RETURN json_encode(data)

Request Logging Middleware

Every request should produce a structured log record that includes the duration, status code, and user:

CLASS RequestLoggingMiddleware IMPLEMENTS Middleware:
    logger = get_logger("framework.request")
    
    FUNCTION process(request, next_handler):
        start = now()
        
        response = next_handler(request)
        
        elapsed_ms = (now() - start).milliseconds
        
        user_id = request.user.pk IF request.user.is_authenticated ELSE null
        
        logger.info("HTTP request", extra={
            "method":         request.method,
            "path":           request.path,
            "status_code":    response.status_code,
            "duration_ms":    elapsed_ms,
            "user_id":        user_id,
            "remote_addr":    request.ip_address,
            "content_length": response.content_length,
        })
        
        RETURN response

Checklist

  • Implement the Logger hierarchy with propagation.
  • Implement log handlers: StreamHandler (stdout/stderr), FileHandler, RotatingFileHandler, SMTPHandler (email on CRITICAL).
  • Implement formatters: SimpleFormatter (human-readable) and JsonFormatter (structured).
  • Implement the LOGGING settings dict for configuration.
  • Implement RequestLoggingMiddleware with structured request/response data.
  • Implement slow query logging in the ORM.
  • Implement error monitoring integration: send ERROR and CRITICAL records to a monitoring service (Sentry, Datadog).
  • Implement performance metrics: request latency histogram, database query count per request.
  • Implement DEBUG_TOOLBAR-style query inspection for development.

40. Internationalization & Localization (i18n / l10n)

GoF Pattern: Strategy (Translation Backends) + Proxy (Lazy Translation Strings)

Internationalization (i18n) is designing the framework so that text, dates, numbers, and currencies can be adapted for different languages and regions. Localization (l10n) is the process of actually creating those translations and locale-specific formats.

The Translation System

// Marking strings for translation:
FUNCTION gettext(message: string) → string:
    // Look up the current thread's active language
    // Find the translation catalog for that language
    // Return the translated string, or the original if no translation found
    lang    = get_current_language()
    catalog = get_translation_catalog(lang)
    RETURN catalog.gettext(message) OR message

FUNCTION ngettext(singular: string, plural: string, count: integer) → string:
    // Plural forms handling
    lang    = get_current_language()
    catalog = get_translation_catalog(lang)
    RETURN catalog.ngettext(singular, plural, count) OR (singular IF count == 1 ELSE plural)

// Lazy translation: evaluated at rendering time, not import time
CLASS LazyTranslationString:
    FIELDS:
        message:  string
    
    FUNCTION __str__() → string:
        RETURN gettext(message)

FUNCTION gettext_lazy(message) → LazyTranslationString:
    // Use for strings at module level that must survive import before language is set
    RETURN LazyTranslationString(message)

// Aliased for brevity (standard convention):
_ = gettext
_l = gettext_lazy

The Locale Middleware

The LocaleMiddleware detects the user's preferred language and activates it for the duration of the request:

CLASS LocaleMiddleware IMPLEMENTS Middleware:
    FUNCTION process(request, next_handler):
        language = self.get_language_from_request(request)
        activate_language(language)
        request.LANGUAGE_CODE = language
        
        response = next_handler(request)
        
        // Set the Content-Language header
        response.set_header("Content-Language", language)
        
        // Set the language cookie so it persists
        response.set_cookie(settings.LANGUAGE_COOKIE_NAME, language)
        
        RETURN response
    
    FUNCTION get_language_from_request(request) → string:
        // Check 1: URL prefix (if USE_I18N_URL_PREFIXES is True)
        // Check 2: Cookie
        lang = request.cookies.get(settings.LANGUAGE_COOKIE_NAME)
        IF lang AND is_language_supported(lang): RETURN lang
        
        // Check 3: Accept-Language header
        accept_language = request.headers.get("accept-language", "")
        lang = parse_accept_language(accept_language)
        IF lang: RETURN lang
        
        // Fall back to default language
        RETURN settings.LANGUAGE_CODE

Number and Date Localization

Different locales format numbers and dates differently. German uses comma as decimal separator (1.234,56); English uses period (1,234.56). Dates are ordered differently (DD/MM/YYYY vs. MM/DD/YYYY). The framework must provide locale-aware formatting functions:

FUNCTION format_number(value, decimal_pos=2, grouping=True) → string:
    locale = get_current_locale()
    parts = []
    // Use locale's decimal separator and grouping separator
    RETURN format_with_locale(value, locale.decimal_separator, locale.thousands_separator)

FUNCTION format_date(value, format="SHORT_DATE_FORMAT") → string:
    locale  = get_current_locale()
    fmt_str = locale.date_formats.get(format, format)
    RETURN value.strftime(fmt_str)

Checklist

  • Implement gettext() with thread-local language activation.
  • Implement gettext_lazy() for module-level strings.
  • Implement ngettext() for plural forms.
  • Implement pgettext() for context-disambiguated translations (same string, different meanings).
  • Implement LocaleMiddleware with Accept-Language parsing.
  • Implement .po/.mo file format support for translation catalogs.
  • Implement the makemessages management command to extract translatable strings.
  • Implement the compilemessages management command to compile .po → .mo.
  • Implement locale-aware number, date, and currency formatting.
  • Implement timezone support: USE_TZ=True stores all datetimes as UTC; display in user's local timezone.
  • Implement the {% trans %} and {% blocktrans %} template tags.

41. File Upload & Storage Backends

GoF Pattern: Strategy (Storage Backends) + Proxy (FieldFile)

File upload handling has two phases: receiving the uploaded file from the HTTP request (handled by the HTTP parser in Step 4) and storing it for later retrieval. The storage layer is abstracted behind a Storage interface so that files can be stored locally, in object storage (Amazon S3, Google Cloud Storage), or anywhere else without changing application code.

The Storage Interface

INTERFACE Storage:
    FUNCTION save(name: string, content: File) → string       // Returns actual saved name
    FUNCTION open(name: string, mode="rb") → File
    FUNCTION delete(name: string)
    FUNCTION exists(name: string) → boolean
    FUNCTION size(name: string) → integer
    FUNCTION url(name: string) → string                        // URL to access the file
    FUNCTION listdir(path: string) → (List<dirs>, List<files>)

CLASS FileSystemStorage IMPLEMENTS Storage:
    FIELDS:
        location: string    // Absolute filesystem path (MEDIA_ROOT)
        base_url: string    // URL prefix (MEDIA_URL)
    
    FUNCTION save(name, content):
        full_path = os.path.join(location, name)
        os.makedirs(os.path.dirname(full_path), exist_ok=True)
        
        // Handle name collisions: if "uploads/photo.jpg" exists, try "uploads/photo_1.jpg"
        name = get_available_name(name)
        
        WITH open(full_path, "wb") AS f:
            FOR chunk IN content.chunks():
                f.write(chunk)
        RETURN name
    
    FUNCTION url(name):
        RETURN urljoin(base_url, filepath_to_uri(name))

CLASS S3Storage IMPLEMENTS Storage:
    // Uses boto3 to store files in Amazon S3
    // Files are publicly accessible via S3 URLs or CloudFront CDN

The FieldFile Proxy

When a model has a FileField, accessing instance.file_field returns a FieldFile object, not a raw filename. FieldFile is a Proxy (GoF) over the storage backend that provides a convenient interface:

CLASS FieldFile:
    FIELDS:
        instance: model instance
        field:    FileField
        name:     string    // The stored path/filename
    
    PROPERTY url:
        RETURN field.storage.url(name)
    
    PROPERTY size:
        RETURN field.storage.size(name)
    
    FUNCTION open(mode="rb"):
        RETURN field.storage.open(name, mode)
    
    FUNCTION save(name: string, content: File, save=True):
        self.name = field.generate_filename(instance, name)
        field.storage.save(self.name, content)
        IF save: instance.save()
    
    FUNCTION delete(save=True):
        field.storage.delete(name)
        self.name = ""
        IF save: instance.save()

Handling Uploaded Files in Views

FUNCTION handle_upload_view(request):
    IF request.method == "POST":
        form = FileUploadForm(data=request.data, files=request.files)
        IF form.is_valid():
            instance = form.save()
            RETURN JsonResponse({"url": instance.file.url})
    ELSE:
        form = FileUploadForm()
    RETURN HtmlResponse(render_template("upload.html", {"form": form}))

Checklist

  • Implement FileSystemStorage with MEDIA_ROOT-based storage and collision handling.
  • Implement S3Storage (or equivalent cloud storage backend).
  • Implement FieldFile proxy with url, size, open, save, delete.
  • Implement FileField and ImageField (with image validation using Pillow or equivalent).
  • Implement chunked file upload reading (do not read the entire file into memory at once).
  • Implement file name sanitization (prevent directory traversal attacks via ../ in filenames).
  • Implement configurable upload_to callable for organizing files into subdirectories.
  • Implement InMemoryUploadedFile and TemporaryUploadedFile for small and large uploads respectively.
  • Test that large file uploads do not cause memory exhaustion.

42. Signals — Decoupled Application Events

GoF Pattern: Observer + Singleton (Signal Registry)

Signals are the framework's decoupled event system for application-level communication. While the middleware system handles cross-cutting concerns at the request level, and the ORM's save() method accepts pre_save/post_save hooks, signals provide a general-purpose publish/subscribe mechanism that allows any piece of code to notify other pieces of code that an event has occurred, without the sender needing to know who (if anyone) is listening.

Signals differ from the event bus in Step 6 (which is for framework-internal communication) — signals are part of the user-facing API and are intended for use in application code.

The Signal Architecture

CLASS Signal:
    FIELDS:
        receivers:   List<(id, weakref_to_receiver)>
        use_caching: boolean
        _receiver_cache: dict  // For caching sender-specific receivers
        lock:        threading.Lock
    
    FUNCTION connect(receiver: Callable,
                     sender: any = null,
                     weak: boolean = True,
                     dispatch_uid: string | null = null):
        // receiver: the callable to call when signal is sent
        // sender: if specified, only call receiver when this sender fires the signal
        // weak: use a weak reference so the receiver doesn't prevent garbage collection
        // dispatch_uid: prevent duplicate registration with a unique ID
        
        WITH lock:
            id = dispatch_uid OR id(receiver)
            
            // Check for duplicate
            FOR existing_id, _ IN receivers:
                IF existing_id == id: RETURN
            
            ref = weakref(receiver) IF weak ELSE strong_ref(receiver)
            receivers.append((id, ref))
            _receiver_cache.clear()   // Invalidate cache
    
    FUNCTION disconnect(receiver=null, sender=null, dispatch_uid=null):
        WITH lock:
            id = dispatch_uid OR id(receiver)
            receivers = [(i, r) FOR i, r IN receivers IF i != id]
            _receiver_cache.clear()
    
    FUNCTION send(sender: any, **kwargs) → List<(receiver, response)>:
        // Fire the signal synchronously; return list of (receiver, response) pairs
        responses = []
        FOR receiver IN self._get_receivers(sender):
            response = receiver(signal=self, sender=sender, **kwargs)
            responses.append((receiver, response))
        RETURN responses
    
    FUNCTION send_robust(sender, **kwargs) → List<(receiver, response_or_exception)>:
        // Like send(), but catches exceptions from receivers and returns them
        // instead of letting one bad receiver break all others
        responses = []
        FOR receiver IN self._get_receivers(sender):
            TRY:
                response = receiver(signal=self, sender=sender, **kwargs)
                responses.append((receiver, response))
            EXCEPT Exception AS e:
                responses.append((receiver, e))
        RETURN responses

Built-in Framework Signals

The framework ships with pre-defined signals that users can connect to without modifying framework code:

// ORM signals (from django.db.models.signals)
pre_init    = Signal()    // Before Model.__init__
post_init   = Signal()    // After Model.__init__
pre_save    = Signal()    // Before Model.save()
post_save   = Signal()    // After Model.save() — provides created=True/False
pre_delete  = Signal()    // Before Model.delete()
post_delete = Signal()    // After Model.delete()
m2m_changed = Signal()    // When a ManyToManyField is changed
pre_migrate = Signal()    // Before running migrations
post_migrate = Signal()   // After running migrations

// Request/response signals (from framework.core.signals)
request_started  = Signal()
request_finished = Signal()
got_request_exception = Signal()   // Unhandled exception during request

// Auth signals
user_logged_in   = Signal()
user_logged_out  = Signal()
user_login_failed = Signal()

Connecting Signals in App.ready()

The correct place to connect signal handlers is in AppConfig.ready(), which runs after all models are imported:

CLASS OrdersAppConfig EXTENDS AppConfig:
    name = "orders"
    
    FUNCTION ready():
        from .signals import handlers  // Import handlers module to register them
    
// handlers.py
from framework.signals import post_save
from .models import Order

FUNCTION on_order_created(sender, instance, created, **kwargs):
    IF created:
        // Send confirmation email
        send_templated_email(
            "orders/emails/confirmation",
            context={"order": instance},
            to=[instance.customer.email]
        )

post_save.connect(on_order_created, sender=Order)

// Or using the @receiver decorator:
@receiver(post_save, sender=Order)
FUNCTION on_order_created(sender, instance, created, **kwargs):
    IF created:
        send_templated_email(...)

Checklist

  • Implement Signal with connect(), disconnect(), send(), send_robust().
  • Implement weak reference support to prevent signal handlers from leaking memory.
  • Implement sender-specific filtering in _get_receivers().
  • Implement dispatch_uid to prevent duplicate handler registration.
  • Implement all ORM signals: pre_save, post_save, pre_delete, post_delete, m2m_changed.
  • Implement request lifecycle signals: request_started, request_finished.
  • Implement auth signals: user_logged_in, user_logged_out, user_login_failed.
  • Implement the @receiver(signal, sender) decorator shorthand.
  • Test send_robust(): ensure one crashing receiver does not prevent others from receiving.
  • Test weak reference cleanup: receiver should be garbage collected when its owner is deleted.

43. Test Suites

Test Suite 1: ORM, Forms & Middleware

These tests validate the core data layer and request pipeline. All database tests should use a test-specific database that is created fresh and torn down after each test suite.

TEST GROUP: ORM — QuerySet

  TEST "filter() returns only matching records":
      Article.objects.create(title="Hello", published=True)
      Article.objects.create(title="World", published=False)
      results = Article.objects.filter(published=True)
      ASSERT len(results) == 1
      ASSERT results[0].title == "Hello"

  TEST "filter() with no match returns empty queryset":
      result = User.objects.filter(username="doesnotexist")
      ASSERT len(result) == 0
      ASSERT bool(result) == False

  TEST "no SQL executed until evaluation trigger":
      query_count_before = connection.queries_count()
      qs = User.objects.filter(is_active=True).order_by("-created_at")
      query_count_after = connection.queries_count()
      ASSERT query_count_before == query_count_after   // No query yet
      list(qs)   // Evaluation trigger
      ASSERT connection.queries_count() == query_count_before + 1

  TEST "filter chaining produces single SQL query":
      qs = (User.objects
                .filter(is_active=True)
                .filter(age__gte=18)
                .order_by("-created_at"))
      with connection.capture_queries() AS queries:
          list(qs)
      ASSERT len(queries) == 1
      ASSERT "WHERE" IN queries[0].sql
      ASSERT "ORDER BY" IN queries[0].sql

  TEST "QuerySet _clone() is independent of original":
      base = User.objects.filter(is_active=True)
      filtered = base.filter(is_staff=True)
      // base should not have the is_staff filter
      ASSERT filtered._filter_conditions != base._filter_conditions

  TEST "get() raises DoesNotExist for missing object":
      ASSERT_RAISES(User.DoesNotExist):
          User.objects.get(username="nobodyhere")

  TEST "get() raises MultipleObjectsReturned for multiple results":
      User.objects.create(is_active=True, username="a")
      User.objects.create(is_active=True, username="b")
      ASSERT_RAISES(User.MultipleObjectsReturned):
          User.objects.get(is_active=True)

  TEST "bulk_create is faster than N individual saves":
      objects = [Article(title=f"Post {i}") FOR i IN range(100)]
      start = now()
      Article.objects.bulk_create(objects)
      elapsed_bulk = now() - start
      
      Article.objects.all().delete()
      start = now()
      FOR obj IN objects: obj.save()
      elapsed_individual = now() - start
      
      ASSERT elapsed_bulk < elapsed_individual

  TEST "transaction rollback on exception":
      TRY:
          WITH atomic():
              Article.objects.create(title="Will be rolled back")
              RAISE Exception("Force rollback")
      EXCEPT: PASS
      ASSERT Article.objects.filter(title="Will be rolled back").count() == 0

  TEST "select_related produces a single query":
      author = Author.objects.create(name="Alice")
      Article.objects.create(title="Hello", author=author)
      WITH connection.capture_queries() AS queries:
          articles = list(Article.objects.select_related("author"))
          // Access author — should NOT cause additional query
          _ = articles[0].author.name
      ASSERT len(queries) == 1

  TEST "prefetch_related produces exactly two queries":
      article = Article.objects.create(title="Hello")
      article.tags.add(Tag(name="python"), Tag(name="django"))
      WITH connection.capture_queries() AS queries:
          articles = list(Article.objects.prefetch_related("tags"))
          _ = [a.tags.all() FOR a IN articles]
      ASSERT len(queries) == 2


TEST GROUP: Form System

  TEST "unbound form has no errors and is not valid":
      form = ContactForm()
      ASSERT NOT form.is_bound
      ASSERT NOT form.is_valid()

  TEST "bound form with valid data is valid":
      form = ContactForm(data={
          "subject": "Hello",
          "message": "This is a test.",
          "sender":  "test@example.com",
          "cc_me":   False
      })
      ASSERT form.is_bound
      ASSERT form.is_valid()
      ASSERT form.cleaned_data["sender"] == "test@example.com"

  TEST "required field missing produces field-level error":
      form = ContactForm(data={"message": "Hi"})  // missing subject, sender
      ASSERT NOT form.is_valid()
      ASSERT "subject" IN form.errors
      ASSERT "sender" IN form.errors
      ASSERT "This field is required" IN form.errors["subject"][0]

  TEST "email field validates format":
      form = ContactForm(data={
          "subject": "Test", "message": "Hi",
          "sender": "not-an-email"
      })
      ASSERT NOT form.is_valid()
      ASSERT "sender" IN form.errors

  TEST "clean_<field> method is called":
      form = ContactForm(data={
          "subject": "Buy spam now!",
          "message": "Click here",
          "sender": "test@example.com"
      })
      ASSERT NOT form.is_valid()
      ASSERT "subject" IN form.errors
      ASSERT "spam" IN form.errors["subject"][0].lower()

  TEST "ModelForm creates model instance on save":
      form = ArticleForm(data={"title": "New Article", "body": "Content"})
      ASSERT form.is_valid()
      article = form.save()
      ASSERT article.pk is not null
      ASSERT Article.objects.filter(pk=article.pk).exists()

  TEST "ModelForm with existing instance performs update":
      article = Article.objects.create(title="Old Title", body="Old Body")
      form = ArticleForm(instance=article, data={"title": "New Title", "body": "Old Body"})
      ASSERT form.is_valid()
      updated = form.save()
      ASSERT updated.pk == article.pk    // Same record
      ASSERT updated.title == "New Title"


TEST GROUP: Middleware Pipeline

  TEST "middleware runs in order for request, reverse for response":
      log = []
      CLASS First IMPLEMENTS Middleware:
          FUNCTION process(request, next):
              log.append("first:request")
              response = next(request)
              log.append("first:response")
              RETURN response
      CLASS Second IMPLEMENTS Middleware:
          FUNCTION process(request, next):
              log.append("second:request")
              response = next(request)
              log.append("second:response")
              RETURN response
      
      stack = MiddlewareStack()
      stack.build([First, Second], dummy_view)
      stack.process(dummy_request)
      
      ASSERT log == ["first:request", "second:request", "second:response", "first:response"]

  TEST "middleware short-circuit prevents view from running":
      view_called = False
      CLASS BlockMiddleware IMPLEMENTS Middleware:
          FUNCTION process(request, next):
              RETURN Response(status=403, body="Blocked")
      
      FUNCTION dummy_view(request):
          view_called = True
          RETURN Response(status=200)
      
      stack = MiddlewareStack()
      stack.build([BlockMiddleware], dummy_view)
      response = stack.process(dummy_request)
      
      ASSERT response.status_code == 403
      ASSERT NOT view_called

  TEST "CSRF middleware blocks POST without token":
      middleware = CsrfMiddleware()
      post_request = Request(method="POST", path="/test/", data={})
      // No CSRF token in data or headers
      response = process_with_middleware(middleware, post_request)
      ASSERT response.status_code == 403

  TEST "CSRF middleware passes POST with valid token":
      token = generate_csrf_token()
      post_request = Request(method="POST", path="/test/",
                             cookies={"csrftoken": token},
                             data={"csrfmiddlewaretoken": token})
      response = process_with_middleware(CsrfMiddleware(), post_request)
      ASSERT response.status_code != 403

  TEST "SecurityMiddleware redirects HTTP to HTTPS when configured":
      settings.SECURE_SSL_REDIRECT = True
      request = Request(method="GET", path="/test/", scheme="http")
      response = process_with_middleware(SecurityMiddleware(), request)
      ASSERT response.status_code == 301
      ASSERT response.headers["location"].startswith("https://")
      settings.SECURE_SSL_REDIRECT = False   // Reset

Test Suite 2: Router, Views, Templates & Authentication

TEST GROUP: Router

  TEST "literal path matches exactly":
      router = Router()
      router.path("/about/", lambda req: Response(200))
      result = router.resolve("/about/", "GET")
      ASSERT result is not null

  TEST "parameterized path extracts integer":
      router = Router()
      router.path("/users/<int:user_id>/", lambda req, user_id: Response(200))
      result = router.resolve("/users/42/", "GET")
      ASSERT result is not null
      ASSERT result.kwargs["user_id"] == 42
      ASSERT isinstance(result.kwargs["user_id"], int)

  TEST "unmatched path raises Http404":
      router = Router()
      ASSERT_RAISES(Http404):
          router.resolve("/does-not-exist/", "GET")

  TEST "wrong method raises MethodNotAllowed":
      router = Router()
      router.path("/create/", lambda req: Response(201), methods=["POST"])
      ASSERT_RAISES(MethodNotAllowed):
          router.resolve("/create/", "GET")

  TEST "reverse() generates correct URL":
      router = Router()
      router.path("/users/<int:pk>/", lambda: null, name="user-detail")
      url = router.reverse("user-detail", kwargs={"pk": 42})
      ASSERT url == "/users/42/"

  TEST "include() mounts sub-patterns under prefix":
      sub_patterns = [path("list/", lambda: null, name="list")]
      main = Router()
      main.include("api/", sub_patterns)
      result = main.resolve("/api/list/", "GET")
      ASSERT result is not null


TEST GROUP: Class-Based Views

  TEST "CBV routes GET to get() method":
      class MyView EXTENDS View:
          FUNCTION get(request):
              RETURN Response(status=200, body="hello")
      
      response = MyView.as_view()(make_request("GET"))
      ASSERT response.status_code == 200

  TEST "CBV returns 405 for unimplemented method":
      class ReadOnlyView EXTENDS View:
          FUNCTION get(request):
              RETURN Response(status=200)
      
      response = ReadOnlyView.as_view()(make_request("POST"))
      ASSERT response.status_code == 405
      ASSERT "GET" IN response.headers["allow"]

  TEST "LoginRequiredMixin redirects anonymous user":
      class ProtectedView EXTENDS LoginRequiredMixin, View:
          FUNCTION get(request): RETURN Response(200)
      
      request = make_request("GET")
      request.user = AnonymousUser()
      response = ProtectedView.as_view()(request)
      ASSERT response.status_code == 302
      ASSERT response.headers["location"].startswith("/login/")

  TEST "ListView paginates correctly":
      FOR i IN range(25): Article.objects.create(title=f"Article {i}")
      
      class ArticleListView EXTENDS ListView:
          model = Article
          paginate_by = 10
      
      request = make_request("GET", query_string="page=2")
      response = ArticleListView.as_view()(request)
      ASSERT response.status_code == 200
      ASSERT len(response.context["object_list"]) == 10


TEST GROUP: Template Engine

  TEST "variable node renders context value":
      template = compile_template("Hello, {{ name }}!")
      result = render(template, {"name": "World"})
      ASSERT result == "Hello, World!"

  TEST "auto-escaping prevents XSS":
      template = compile_template("{{ user_input }}")
      result = render(template, {"user_input": "<script>alert('xss')</script>"})
      ASSERT "<script>" NOT IN result
      ASSERT "&lt;script&gt;" IN result

  TEST "safe filter bypasses auto-escaping":
      template = compile_template("{{ html|safe }}")
      result = render(template, {"html": "<strong>Bold</strong>"})
      ASSERT "<strong>Bold</strong>" IN result

  TEST "for tag iterates correctly":
      template = compile_template("{% for item in items %}{{ item }},{% endfor %}")
      result = render(template, {"items": ["a", "b", "c"]})
      ASSERT result == "a,b,c,"

  TEST "for...empty renders empty clause for empty list":
      template = compile_template("{% for item in items %}{{ item }}{% empty %}none{% endfor %}")
      ASSERT render(template, {"items": []}) == "none"
      ASSERT render(template, {"items": ["x"]}) == "x"

  TEST "if tag evaluates conditions":
      template = compile_template("{% if value > 5 %}big{% else %}small{% endif %}")
      ASSERT render(template, {"value": 10}) == "big"
      ASSERT render(template, {"value": 3})  == "small"

  TEST "template inheritance works at two levels":
      base_template = "{% block content %}DEFAULT{% endblock %}"
      child_template = "{% extends 'base.html' %}{% block content %}CHILD{% endblock %}"
      
      register_template("base.html", base_template)
      register_template("child.html", child_template)
      
      result = render_template("child.html", {})
      ASSERT result == "CHILD"
      ASSERT "DEFAULT" NOT IN result

  TEST "template caching avoids re-parsing":
      parse_count = 0
      original_parse = TemplateLexer.tokenize
      TemplateLexer.tokenize = LAMBDA self: parse_count += 1 AND original_parse(self)
      
      render_template("test.html", {})
      render_template("test.html", {})   // Second render
      
      ASSERT parse_count == 1   // Only parsed once (cached)


TEST GROUP: Authentication

  TEST "set_password stores a hash, not the raw password":
      user = User()
      user.set_password("mysecretpassword")
      ASSERT user.password != "mysecretpassword"
      ASSERT user.password STARTS WITH "argon2"

  TEST "check_password returns True for correct password":
      user = User()
      user.set_password("correctpassword")
      ASSERT user.check_password("correctpassword") == True
      ASSERT user.check_password("wrongpassword")   == False

  TEST "login() sets user in session":
      request = make_request()
      user = User.objects.create(username="alice")
      user.set_password("password")
      login(request, user)
      ASSERT request.session.get("_auth_user_id") == str(user.pk)

  TEST "logout() clears session":
      request = make_request()
      user = User.objects.create(username="bob")
      login(request, user)
      logout(request)
      ASSERT request.session.get("_auth_user_id") is null

  TEST "password change invalidates old sessions":
      user = User.objects.create(username="charlie")
      user.set_password("oldpassword")
      user.save()
      
      old_hash = user.get_session_auth_hash()
      user.set_password("newpassword")
      user.save()
      new_hash = user.get_session_auth_hash()
      
      ASSERT old_hash != new_hash

  TEST "anonymous user has no permissions":
      anon = AnonymousUser()
      ASSERT NOT anon.has_perm("blog.add_article")
      ASSERT NOT anon.has_perm("blog.change_article")
      ASSERT NOT anon.is_authenticated

  TEST "superuser passes all permission checks":
      user = User.objects.create(username="admin", is_superuser=True)
      ASSERT user.has_perm("blog.add_article")
      ASSERT user.has_perm("nonexistent.fictional_permission")

  TEST "ModelBackend prevents timing attack via constant-time comparison":
      // Both valid and invalid username lookups should take similar time
      start1 = now()
      authenticate(request, username="validuser", password="wrong")
      time1 = now() - start1
      
      start2 = now()
      authenticate(request, username="invaliduser", password="wrong")
      time2 = now() - start2
      
      // Time difference should be < 50ms (both run the hasher)
      ASSERT abs(time1 - time2) < 50ms

The recommended order minimizes wasted work by always having a working, testable system at each phase. You should be able to demo something at the end of every phase.

Phase 1 — You Can Handle an HTTP Request
  Goal: A working HTTP server that can receive a request and return a hardcoded response.
  
  [x] Step 3:  TCP server (bind, listen, accept, read, write)
  [x] Step 4:  HTTP parser (request line, headers, body)
  [x] Step 2:  HTTP protocol knowledge (methods, status codes, key headers)
  [x] Step 5:  WSGI adapter (environ construction, response serialization)
  [x] Step 6:  Application object (basic initialization, call to a hardcoded handler)
  [x] Step 9:  Response object (status, headers, body, to_bytes())
  [x] Step 8:  Request object (basic; lazy body parsing comes later)
  
  Milestone: curl http://localhost:8000/ returns "Hello, World!"

Phase 2 — You Can Route and Handle Requests
  Goal: URL routing, views, and middleware.
  
  [x] Step 7:  Router (literal + parameterized patterns; Http404; MethodNotAllowed)
  [x] Step 10: Middleware pipeline (Chain of Responsibility; at minimum LoggingMiddleware)
  [x] Step 11: View system (FBVs with decorators; base CBV with dispatch())
  [x] Step 31: Settings system (LazySettings, global defaults, validation)
  
  Milestone: Multiple URLs handled by different views; request/response logged.

Phase 3 — You Can Render Templates
  Goal: A full template engine.
  
  [x] Step 12: Lexer (tokenize template source into tokens)
  [x] Step 13: Parser (build AST from token stream)
  [x] Step 14: Renderer + Context (render AST with data; auto-escaping)
  [x] Step 15: Tags, filters, template inheritance
  
  Milestone: An HTML page rendered from a template with context variables.

Phase 4 — You Can Persist Data
  Goal: A working ORM with at least one database backend.
  
  [x] Step 16: Database backend abstraction (Connection, PostgreSQLBackend)
  [x] Step 17: Model metaclass and Options
  [x] Step 18: Field types and validators
  [x] Step 19: QuerySet (lazy evaluation, chaining, terminal methods)
  [x] Step 20: SQL compiler (SELECT, WHERE, ORDER BY, LIMIT)
  [x] Step 21: Relationships (ForeignKey first; M2M later)
  [x] Step 22: Migrations (makemigrations, migrate)
  [x] Step 23: Connection pooling and atomic()
  
  Milestone: Define a User model, create the table, insert and query rows.

Phase 5 — You Can Handle Forms and Authentication
  Goal: Secure user login/logout and form validation.
  
  [x] Step 24: Form system (declaration, binding, validation)
  [x] Step 25: Validation system (validators, ValidationError, unique checks)
  [x] Step 26: Authentication (User model, password hashing, backends, permissions)
  [x] Step 27: Session system (at least DatabaseSessionBackend)
  [x] Step 28: Cookie management (set/delete, attributes)
  [x] Step 29: Security (CSRF middleware, SecurityMiddleware, Host header validation)
  
  Milestone: A login/logout flow with CSRF protection working end to end.

Phase 6 — You Have a Complete Framework
  Goal: All remaining production-grade features.
  
  [x] Step 30: Caching (LocMem and Redis backends; @cache_page; {% cache %} tag)
  [x] Step 32: App registry (INSTALLED_APPS loading, model registration)
  [x] Step 33: Static files (finders, collectstatic, development server)
  [x] Step 34: Admin interface (ModelAdmin, AdminSite, changelist/change/add/delete views)
  [x] Step 35: Email (SMTP backend; ConsoleEmailBackend; InMemoryEmailBackend)
  [x] Step 36: Background tasks (job queue, worker, retry logic)
  [x] Step 37: WebSocket (ASGI consumer, channel layers)
  [x] Step 38: Management commands (CLI, built-in commands, auto-discovery)
  [x] Step 39: Logging (structured JSON logging, request logging middleware)
  [x] Step 40: i18n/l10n (gettext, LocaleMiddleware, template tags)
  [x] Step 41: File upload and storage backends
  [x] Step 42: Signals (Observer for decoupled app events)
  
  Milestone: A full-featured web application running on your framework.

Quick Reference: Design Patterns Index

GoF Pattern Category Step(s) Problem Solved
Abstract Factory Creational 16, 30, 35 Swap database, cache, and email backends without touching application code
Builder Creational 19, 20 Incrementally construct a QuerySet; assemble SQL from parts
Factory Method Creational 8, 9, 11 Create the right Request, Response, or View type without hardcoding
Prototype Creational 19 Clone a QuerySet for safe non-destructive chaining
Singleton Creational 6, 31 Application registry; settings object
Adapter Structural 5, 16 WSGI/ASGI adapter; database driver adapter
Composite Structural 7, 15, 24 URL resolver tree; template inheritance; form fieldsets
Decorator Structural 10, 26, 29 Middleware as functional decorators; permission decorators
Facade Structural 6, 16, 30 Application object; ORM high-level API; unified cache API
Flyweight Structural 12, 17 Shared template tokens; shared model field instances
Proxy Structural 19, 27 Lazy QuerySet evaluation; lazy session loading
Chain of Responsibility Behavioral 10, 29 Middleware pipeline; security header chain
Command Behavioral 22, 38 Migration operations; management commands
Iterator Behavioral 19, 24 QuerySet row traversal; form field iteration
Observer Behavioral 42, 6 Application signals; app startup event hooks
State Behavioral 5, 37 WSGI/ASGI connection type dispatch; WebSocket lifecycle
Strategy Behavioral 16, 25, 30 Swappable database, validation, cache backends
Template Method Behavioral 11, 34 CBV dispatch lifecycle; ModelAdmin CRUD lifecycle
Visitor Behavioral 13, 20 AST node rendering; SQL query compilation

Quick Reference: SOLID Violations to Actively Avoid

Violation Symptom Fix
SRP: Request parses cookies AND runs business logic request.get_current_user() queries DB Move user lookup to AuthenticationMiddleware
SRP: Model validates AND generates HTML forms model.render_form() exists Keep validation in validators; form rendering in Form classes
OCP: Adding a new DB backend requires editing ORM core if backend == "postgres": ... in compiler Backend-specific behaviour in backend class; compiler calls backend.quote_name()
OCP: New template tag requires editing the parser if tag_name == "mynewtag": in parser Use a tag registry; tags are registered, not hardcoded
LSP: ReadOnlySession.set() raises NotImplementedError silently Setting session data silently fails Raise ReadOnlySessionError explicitly; or use a separate interface
ISP: Every view inherits a huge base class with 30 methods View subclasses override pass on 25 methods Use mixins for specific behaviours; keep the base View minimal
DIP: UserService imports PostgreSQLUserRepository directly FROM db.postgres IMPORT UserRepo at top of service Inject UserRepository interface; bind to concrete at startup
DIP: Template engine imports the ORM to resolve model names Circular import between template and ORM layers Pass resolved data in the context dict; template engine is data-agnostic

Quick Reference: Security Checklist (Non-Negotiable)

Threat Framework Defence Step
SQL Injection All ORM queries use parameterized placeholders (never f-string SQL) 20
XSS Auto-escaping in template engine; mark_safe() documented as dangerous 14
CSRF Synchronizer token middleware on all unsafe methods 29
Session Fixation session.cycle_key() called on every login 27
Session Hijacking HttpOnly, Secure, SameSite=Lax cookie flags 28
Clickjacking X-Frame-Options: DENY header by default 29
MIME Sniffing X-Content-Type-Options: nosniff header 29
Sensitive Data in URLs Never put session IDs or tokens in URL parameters 27
Password Storage Argon2 hashing with salt; constant-time comparison 26
Host Header Attack Validate against ALLOWED_HOSTS on every request 29
Timing Attacks Constant-time string comparison for tokens, passwords, CSRFs 26, 29
Directory Traversal in Uploads Sanitize filenames; reject ../ components 41
Secret Key Exposure Validate SECRET_KEY at startup; never log it 31
DoS via Large Uploads Enforce max_request_size at TCP layer 3
Open Redirect Validate next parameter against ALLOWED_HOSTS before redirect 26

Useful Resources

© 2026 0xkiire.