HTTP

HTTP Clients

Two clients dominate operator use of HTTP: curl (batteries-included, scriptable) and wget (focused on downloads). The sections below cover each HTTP method through both, with the most common option flags for headers, auth, and TLS.

The structure of an HTTP exchange. Both halves are ASCII over TCP (or TLS-wrapped TCP); curl -v prints the request and response lines as > ... and < ... so the operator can see exactly what crossed the wire:

── request ──────────────────────────────────────────────────────────
GET /v1/users/42 HTTP/1.1                ◄── method  path  version
Host: api.example.com                    ◄──┐
Authorization: Bearer eyJ...              ◄──┤  headers (key: value)
Accept: application/json                  ◄──┤
User-Agent: curl/8.4.0                    ◄──┘
                                         ◄── blank line ends headers
{"hint":"only on POST/PUT/PATCH"}        ◄── body (optional)

── response ─────────────────────────────────────────────────────────
HTTP/1.1 200 OK                          ◄── version  code  reason
Content-Type: application/json           ◄──┐
Content-Length: 73                        ◄──┤  response headers
Date: Sun, 04 May 2026 14:00:00 GMT      ◄──┘
                                         ◄── blank line ends headers
{"id":42,"name":"operator","tier":2}     ◄── body

curl -v       ► prints both halves with > and < markers
curl -i       ► includes response headers in the output
curl -I       ► sends HEAD; response headers only

Request anatomy

Three pieces, separated by a blank line:

Component	Format
Request line	METHOD /path HTTP/version
Headers	Header-Name: value (one per line)
Body	Optional bytes; type and length declared by headers

The request line:

Field	Examples / notes
Method	GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS
Path	/v1/users/42?format=json, absolute path on the server
Version	HTTP/1.1 text, HTTP/2 binary multiplexed, HTTP/3 over QUIC

Common request headers:

Header	Purpose
Host	Target server (mandatory in HTTP/1.1; pseudo-header in HTTP/2)
Authorization	Bearer <jwt>, Basic <b64>, Digest ...
Accept	What the client wants back (application/json, text/html)
Accept-Encoding	Compression: gzip, br, zstd
Content-Type	For request bodies: application/json, multipart/form-data
Content-Length	Body size in bytes (or use Transfer-Encoding: chunked)
Cookie	Previously-set session state
User-Agent	Client software identification
If-None-Match	Cache validator paired with ETag from a prior response
X-Forwarded-For	Original client IP across a chain of proxies

Response anatomy

Same structure as the request, with a status line in place of the request line:

Component	Format
Status line	HTTP/version  code  reason-phrase
Headers	Header-Name: value
Body	Bytes (HTML, JSON, binary, …)

Status code classes:

Code range	Class	Examples
1xx	Informational	100 Continue, 101 Switching Protocols (WebSocket)
2xx	Success	200 OK, 201 Created, 204 No Content, 206 Partial Content
3xx	Redirection	301 Moved Permanently, 302 Found, 304 Not Modified
4xx	Client error	400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 429 Too Many Requests
5xx	Server error	500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout

Common response headers:

Header	Purpose
Content-Type	Body MIME type (application/json, text/html)
Content-Length	Body size; or Transfer-Encoding: chunked
Content-Encoding	Compression applied (gzip, br)
Set-Cookie	Server hands the client a cookie
Cache-Control	max-age=3600, no-store, private, …
ETag	Opaque version tag for cache validation
Last-Modified	Validator alternative to ETag
Location	Target of a 3xx redirect or 201 Created
WWW-Authenticate	Challenge accompanying a 401
Strict-Transport-Security	Force future requests to HTTPS (HSTS)
Server	Server software (nginx, cloudflare)

GET

The default and safest method. Fetches a resource without modifying server state. Idempotent and cacheable.

$ curl https://example.com/
$ curl -fsSL https://api.example.com/v1/users/42
$ curl -G --data-urlencode 'q=operator handbook' \
     https://api.example.com/v1/search
$ wget https://example.com/file.tar.gz

POST

Submits data to the server, usually creating a new resource. Not idempotent; two POSTs create two records.

$ curl -X POST -d '{"name":"alpha"}' \
     -H 'content-type: application/json' \
     https://api.example.com/v1/items
$ curl -X POST --data-urlencode 'user=op' --data-urlencode 'pass=s3cret' \
     https://example.com/login
$ curl -X POST -F 'file=@./report.pdf' \
     https://api.example.com/v1/uploads

PUT

Replaces a resource at a known URL. Idempotent; N identical PUTs have the same effect as one.

$ curl -X PUT -d '{"name":"alpha","tier":2}' \
     -H 'content-type: application/json' \
     https://api.example.com/v1/items/42
$ curl -X PUT --data-binary @./config.yaml \
     -H 'content-type: application/yaml' \
     https://api.example.com/v1/configs/main

DELETE

Removes a resource. Idempotent; deleting an already-deleted resource still returns a “gone” status, not a state error.

$ curl -X DELETE https://api.example.com/v1/items/42
$ curl -X DELETE -H "authorization: Bearer $TOKEN" \
     https://api.example.com/v1/sessions/current

HEAD

Like GET but only the response headers come back, no body. Used to check existence, size, content-type, or last-modified without downloading the resource.

$ curl -I https://example.com/
$ curl -I -fsS https://api.example.com/v1/items/42
$ curl -I https://example.com/file.iso

OPTIONS

Asks the server which methods and headers are allowed for a URL. Most often seen as a CORS preflight from a browser before a non-trivial cross-origin request.

$ curl -X OPTIONS -i https://api.example.com/v1/items
$ curl -X OPTIONS -i \
     -H 'origin: https://app.example.com' \
     -H 'access-control-request-method: PUT' \
     https://api.example.com/v1/items/42

PATCH

Partial update; send only the fields that change. Often uses a diff format (JSON Merge Patch RFC 7396 or JSON Patch RFC 6902).

$ curl -X PATCH -d '{"tier":3}' \
     -H 'content-type: application/merge-patch+json' \
     https://api.example.com/v1/items/42
$ curl -X PATCH \
     -d '[{"op":"replace","path":"/tier","value":3}]' \
     -H 'content-type: application/json-patch+json' \
     https://api.example.com/v1/items/42

TRACE

Diagnostic loopback; the server echoes the request as it was received, so the operator can see what proxies altered. Almost always disabled in production (Cross-Site Tracing is an attack vector); useful as part of a security check.

$ curl -X TRACE -i https://example.com/
$ curl -X TRACE -i http://target.example.com/

CONNECT

Tells an HTTP proxy to open a raw TCP tunnel to a host:port. Used by HTTPS through a proxy so the proxy never sees cleartext.

$ curl --proxy http://proxy.lan:3128 https://example.com/
$ curl -v --proxy http://proxy.lan:3128 https://example.com/ 2>&1 | grep -i CONNECT

$ { printf 'CONNECT example.com:443 HTTP/1.1\r\nHost: example.com:443\r\n\r\n'; sleep 1; } \
    | nc proxy.lan 3128

Quick HTTP servers

When you need to serve a directory or stand up a temporary endpoint (file delivery during a pentest, mobile-app testing, log retrieval over HTTP), one-liners beat installing nginx.

Command	Notes
python3 -m http.server 8000	Stdlib; serves $PWD
python3 -m http.server 8000 --bind 127.0.0.1	Localhost only
python3 -m http.server 8000 -d /var/log	Serve a different directory
python3 -m http.server 8000 --cgi	Enable cgi-bin/ execution
php -S 0.0.0.0:8000	PHP CLI; runs index.php if present
ruby -run -e httpd . -p 8000	Ruby stdlib
npx http-server -p 8000	Node; CORS, listings, gzip flags
npx serve -l 8000	Node alternative; SPA-friendly
caddy file-server --listen :8000	Caddy single binary
caddy file-server --listen :8443 --tls internal	Local HTTPS with auto self-signed
busybox httpd -f -p 8000	BusyBox / Alpine
darkhttpd /var/www -p 8000	Tiny C server (~50 KB)
socat TCP-LISTEN:8000,fork,reuseaddr SYSTEM:'cat /tmp/f'	Single-shot byte stream over TCP
go install ... && simple-server -port 8000	Various Go alternatives

Python with TLS in one block:

# python3 -c '...'
import http.server, ssl
srv = http.server.HTTPServer(("0.0.0.0", 4443),
                             http.server.SimpleHTTPRequestHandler)
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_cert_chain(certfile="server.pem")
srv.socket = ctx.wrap_socket(srv.socket, server_side=True)
srv.serve_forever()

Expose a local server publicly:

$ ssh -R 80:localhost:8000 nokey@localhost.run         # localhost.run
$ cloudflared tunnel --url http://localhost:8000       # Cloudflare quick tunnel
$ ngrok http 8000                                      # ngrok

Warning

These servers expose $PWD (or the chosen directory) verbatim. Don’t run them in directories with secrets, dotfiles, or .git. Bind to 127.0.0.1 unless you actually want external access.

Cloning a website

Pull a site (or a portion of it) to disk for offline reading, analysis, or air-gapped use. The classic tool is wget --mirror; httrack is the heavier-weight alternative with a GUI.

Tool	Strength
wget --mirror	Recursive download; rewrites links for local viewing
httrack	Full-site mirroring; resume, filters, depth control
wget2	Modern wget rewrite; HTTP/2, parallel downloads
puppeteer / playwright	Render JS-heavy pages to static HTML / PDF / PNG
single-file-cli	Save a fully-rendered page (with all assets inlined) to one file
monolith	Same idea in Rust; one self-contained .html
Archivebox	Self-hosted archive: HTML + WARC + screenshot per URL
warcprox + wget --warc-file	Capture into the WARC web-archive format

# mirror with link rewriting
$ wget --mirror --convert-links --adjust-extension \
       --page-requisites --no-parent https://example.com/

# depth-limited, polite (1 req/s), include images / CSS / JS
$ wget -r -l 3 -np -k -p -w 1 --random-wait \
       -U 'Mozilla/5.0' https://example.com/

# httrack
$ httrack https://example.com/ -O /tmp/site \
          '+*.example.com/*' '-*.pdf' --depth=3

# JS-rendered single-page archive
$ npx single-file-cli https://example.com/ output.html
$ monolith https://example.com/ -o output.html

# WARC capture
$ wget --warc-file=mirror --warc-cdx --recursive https://example.com/

Web scraping

Programmatic extraction of structured data from web pages. Three families: shell pipelines (curl + jq / htmlq), Python libraries (requests, httpx, BeautifulSoup, lxml, scrapy), and headless browsers for JS-heavy sites (playwright, selenium, puppeteer).

Tool	Use
curl + jq	JSON APIs from the shell
curl + htmlq	CSS selector queries on HTML, jq-style
curl + pup	Same idea, Go-based; less maintained
xidel	XPath / CSS selectors / JSONiq, command-line
hxselect / hxnormalize (html-xml-utils)	CSS selector + clean
BeautifulSoup	Python; forgiving HTML parser
lxml	Python; fast XPath / XSLT
requests / httpx	Python HTTP clients (httpx adds HTTP/2 + async)
scrapy	Python framework; spiders, pipelines, throttling, dedup
playwright / selenium	Drive a real browser (Chromium, Firefox, WebKit)
puppeteer	Node alternative for Chromium
crawl4ai / firecrawl	LLM-friendly markdown extraction

Shell pipelines:

# JSON API
$ curl -s https://api.github.com/repos/torvalds/linux | jq '.stargazers_count'

# CSS selector on HTML
$ curl -s https://news.ycombinator.com/ \
    | htmlq --text '.titleline > a'
$ curl -s https://news.ycombinator.com/ \
    | pup '.titleline > a text{}'

# XPath
$ xidel -e '//a/@href' https://example.com/

Python (BeautifulSoup):

import httpx
from bs4 import BeautifulSoup

r = httpx.get("https://example.com/", headers={"User-Agent": "ops/1.0"})
soup = BeautifulSoup(r.text, "lxml")
for a in soup.select("article h2 a"):
    print(a.get("href"), a.get_text(strip=True))

Python (scrapy spider, spider.py):

import scrapy

class HNSpider(scrapy.Spider):
    name = "hn"
    start_urls = ["https://news.ycombinator.com/"]

    def parse(self, response):
        for row in response.css("tr.athing"):
            yield {
                "rank":  row.css("span.rank::text").get(),
                "title": row.css(".titleline > a::text").get(),
                "url":   row.css(".titleline > a::attr(href)").get(),
            }

# scrapy runspider spider.py -O hn.json

Headless browser (playwright):

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/")
    page.wait_for_selector("article")
    html = page.content()
    page.screenshot(path="page.png")
    browser.close()

Warning

Respect robots.txt, the site’s terms of service, and any rate limits. Hammering an origin without throttling is itself a DoS. Identify your scraper in User-Agent and add backoff on 429 / 503.

HAR format

HAR (HTTP Archive) is a JSON file that captures every request and response a browser made loading a page, timings, headers, cookies, bodies. The browser dev-tools “Network” tab can export and import HAR.

Field path (subset)	Meaning
log.version	HAR version (1.2)
log.creator.name	Browser / tool that produced it
log.pages[i].id	Page identifier (page_1, …)
log.pages[i].pageTimings.onLoad	Page-load milestone in ms
log.entries[i].startedDateTime	ISO 8601 timestamp
log.entries[i].time	Total request time, ms
log.entries[i].request.method	HTTP method
log.entries[i].request.url	Full URL
log.entries[i].request.headers[]	Header list (name / value)
log.entries[i].request.queryString[]	Parsed query parameters
log.entries[i].request.postData.text	Request body
log.entries[i].response.status	HTTP status code
log.entries[i].response.content.mimeType	Content-Type
log.entries[i].response.content.text	Response body (may be base64 for binary)
log.entries[i].timings	blocked / dns / connect / ssl / send / wait / receive

Capture a HAR

• Chrome / Edge / Firefox dev-tools → Network tab → right-click → Save all as HAR with content.

• Browser CLI, headless capture without the GUI:

# Chrome via puppeteer-har
$ npx puppeteer-har -u https://example.com/ -o page.har

# Playwright
$ npx playwright open --har=page.har https://example.com/

# browsertime (instrumented metrics + HAR)
$ docker run --rm sitespeedio/browsertime --har page.har https://example.com/

# mitmproxy can record HAR live
$ mitmweb --set hardump=session.har

Read / convert / replay

# all URLs and status codes
$ jq -r '.log.entries[] | "\(.response.status) \(.request.method) \(.request.url)"' page.har

# only failed requests (>= 400)
$ jq '.log.entries[] | select(.response.status >= 400)' page.har

# convert each request into a curl command
$ npx har-to-curl page.har > requests.sh

# convert HAR → k6 / locust / Postman / OpenAPI
$ npx har-to-k6 page.har -o load.js
$ npx har2openapi page.har > openapi.yaml

Mitmproxy and harlib both read and write HAR programmatically; pyppeteer / playwright can both record HAR while driving a real browser, so a HAR is often the common interchange between “what the browser did” and “what the load-test should re-do”.

See also: HAR 1.2 spec (W3C Web Performance WG), ahmadnassri/har-spec.