Why we care: we are Geonode and we sell proxies, so people use HEAD requests through us constantly to check links, sizes and availability cheaply. The honest caution is that HEAD is a different request, not a lightweight GET, and treating it as one produces confident wrong conclusions. A URL that returns 200 to a HEAD may return 403 to a GET. A resource that shows no Content-Length under HEAD may have one under GET. And an anti-bot layer may treat an unusual HEAD as its own signal. HEAD is excellent for what it is for; it is a poor proxy for "what would happen if I actually fetched this".
The Correct Way
curl -I https://example.com
The curl manual documents -I, --head as: "(HTTP FTP FILE) Fetch the headers only. HTTP-servers feature the command HEAD which this uses to get nothing but the header of a document. When used on an FTP or FILE URL, curl displays the file size and last modification time only."
Output:
HTTP/2 200
content-type: text/html; charset=UTF-8
content-length: 1256
last-modified: Thu, 17 Oct 2019 07:18:26 GMT
cache-control: max-age=604800
That is the whole answer to the headline question. What follows is the part that saves time.
Why -X HEAD Is Wrong
The curl manual addresses this directly under -X, --request:
This option only changes the actual word used in the HTTP request, it does not alter the way curl behaves. For example if you want to make a proper HEAD request, using
-X HEADdoes not suffice. You need to use the--headoption.
The mechanism: -X swaps the method string and nothing else. curl still behaves as though it is doing a GET, which means it still expects a response body. The server, correctly implementing HEAD, sends headers and no body. curl waits for content that will never arrive, and the command appears to hang until a timeout or a connection close ends it.
The manual also warns about a second -X behaviour that catches people out with redirects: "If --location is used, the method string you set with --request is used for all requests". So -X POST -L re-sends a POST to every hop in a redirect chain, which is rarely what anyone intends.
The general principle, stated by the manual itself: "Normally you do not need this option. All sorts of GET, HEAD, POST and PUT requests are rather invoked by using dedicated command line options." Use -I for HEAD, -d for POST, -T for PUT, and reserve -X for genuinely unusual methods such as PROPFIND.
What HEAD Actually Is
RFC 9110 §9.3.2 defines it in one sentence:
The HEAD method is identical to GET except that the server MUST NOT send content in the response.
And states its purpose: "HEAD is used to obtain metadata about the selected representation without transferring its representation data, often for the sake of testing hypertext links or finding recent modifications."
That is a strong requirement on servers — MUST NOT send content — and it is why curl hangs when told to expect one.
The header rule is deliberately weaker, and this is the part people get wrong:
The server SHOULD send the same header fields in response to a HEAD request as it would have sent if the request method had been GET. However, a server MAY omit header fields for which a value is determined only while generating the content.
The RFC gives a concrete example: servers that buffer dynamic responses may produce Content-Length and Vary on a GET that are "not generated within a HEAD response". It calls these "minor inconsistencies" and considers them "preferable to generating and discarding the content for a HEAD request, since HEAD is usually requested for the sake of efficiency."
So a missing Content-Length on a HEAD is not necessarily a bug and not necessarily meaningful. It may simply be a server declining to compute something it would only have known by rendering the page.
There is also a rule about request bodies worth knowing if you are building tooling. Content in a HEAD request "has no generally defined semantics, cannot alter the meaning or target of the request, and might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack". The RFC says a client "SHOULD NOT generate content in a HEAD request" absent a specific prior arrangement. Do not send a body with a HEAD.
What HEAD Is Good For
Genuinely useful cases, all of which trade a full transfer for a few hundred bytes.
Checking whether a URL is alive:
curl -sI -o /dev/null -w '%{response_code}\n' https://example.com
Finding a file's size before downloading:
curl -sI https://example.com/large.iso | grep -i content-length
Following and reporting a redirect chain:
curl -sIL -o /dev/null -w '%{num_redirects} hops -> %{url_effective}\n' https://example.com
Checking whether a server supports range requests, which determines whether an interrupted download can resume:
curl -sI https://example.com/file.zip | grep -i accept-ranges
Checking freshness without downloading:
curl -sI https://example.com/data.json | grep -iE 'last-modified|etag'
Bulk link checking, which is the classic use and where the bandwidth saving compounds:
while read -r url; do
code=$(curl -sIL -o /dev/null -w '%{response_code}' --max-time 10 "$url")
echo "$code $url"
done < urls.txt
Through metered bandwidth the saving is real: a link check that would transfer 500 KB per URL transfers a few hundred bytes instead. Across ten thousand URLs that is the difference between five gigabytes and a few megabytes.
When HEAD Misleads You
The failure modes, which are the reason for the caution at the top.
The server rejects HEAD entirely. 405 Method Not Allowed on a URL where GET works perfectly. Uncommon on static content, not rare on APIs and application endpoints.
The server handles HEAD differently. Different status codes, different headers, sometimes a different code path in the application entirely. The RFC permits omitting content-derived headers, and implementations vary in how far they take that.
Caches and CDNs may key HEAD separately. Cache headers on a HEAD can reflect a different cache entry from the one a GET would hit, so a HEAD is not a reliable way to inspect caching behaviour.
Anti-bot systems respond differently. A HEAD from an unfamiliar client is itself a signal, and the response you get may not be the one a browser-shaped GET would receive.
Content-Length may be absent or wrong. Permitted by the spec, common with dynamic content, and a poor basis for a download-size estimate on anything generated.
Redirect chains may differ. Some servers redirect GET and HEAD to different places, particularly where content negotiation is involved.
When you need to know what a real request would do, do a real request and throw the body away:
curl -sS -o /dev/null -D - https://example.com
A genuine GET, headers on stdout, body discarded. You pay the bandwidth, and you get an accurate answer. Choose deliberately between the two: -I when you want cheap, -o /dev/null -D - when you want true.
A middle option exists for large resources — request one byte instead of the whole thing:
curl -sS -r 0-0 -o /dev/null -D - https://example.com/large.iso
-r, --range retrieves "a byte range (i.e. a partial document)", so 0-0 fetches the first byte only. This is a real GET with real GET behaviour, at almost no bandwidth cost. The caveat from the manual: "Many HTTP/1.1 servers do not have this feature enabled", so check for Accept-Ranges: bytes first and expect a full response when it is absent.
Patterns Worth Copying Into Scripts
The commands above become considerably more useful with a little structure around them.
A link checker that reports honestly. The naive version treats every non-200 as a broken link, which produces false alarms on redirects and on servers that reject HEAD. This one distinguishes them:
check() {
local url="$1" code
code=$(curl -sIL -o /dev/null --max-time 10 -w '%{response_code}' "$url")
case "$code" in
200) echo "OK $url" ;;
405) code=$(curl -sSL -o /dev/null --max-time 10 -w '%{response_code}' "$url")
echo "GET:$code $url" ;;
000) echo "TIMEOUT $url" ;;
*) echo "$code $url" ;;
esac
}
The 405 branch matters: a server refusing HEAD is not a broken link, and retrying as a GET is the only way to know. 000 is curl's way of reporting that no HTTP response arrived at all, which distinguishes a network failure from a server error.
Parallelism, carefully. Link checking is embarrassingly parallel and the temptation is to run it wide. Resist:
xargs -P 8 -I{} sh -c 'check "$1"' _ {} < urls.txt
Eight is a reasonable default. The ceiling here is the politeness of hammering someone else's server rather than your own capacity, and a link checker that generates a rate-limiting incident has cost more than it saved.
Always set a timeout. A HEAD against an unresponsive host hangs exactly as long as a GET would. --max-time 10 with --connect-timeout 5 bounds it, and in a loop over thousands of URLs that bound is what makes the job finish.
Record the effective URL, not just the status. %{url_effective} after -L tells you where a link actually landed, which turns "this link works" into "this link works and now points somewhere else" — usually the more interesting finding.
Cache your results. Re-checking every URL on every run wastes bandwidth and goodwill. Store the status and the ETag or Last-Modified, and use conditional requests on subsequent passes so unchanged resources cost a 304 rather than a full check.
HEAD Through a Proxy
Three things change, all worth knowing.
curl -I -x http://user:pass@proxy.example.com:9000 https://example.com
The CONNECT happens first for HTTPS. curl establishes a tunnel before the HEAD, and in verbose output the proxy's response to that appears before the target's. --suppress-connect-headers hides it; %{http_connect} reports the proxy's status separately from the target's.
The bandwidth saving is the point. Through metered traffic, a HEAD costs a few hundred bytes against a full page's worth. For link validation, availability monitoring and size checks at volume, this is the difference between an affordable job and an expensive one. It is one of the few genuinely large optimisations available on per-gigabyte pricing.
But blocks and challenges behave differently. An anti-bot layer that would serve a challenge page to a GET may simply refuse a HEAD, or vice versa. If you are using HEAD to check whether a target is accessible, verify the conclusion with a real GET on a sample before trusting it across a whole list. This is the silent-failure pattern we wrote about in why testing proxies matters: the request succeeds, the answer is wrong, and nothing tells you.
One curl detail worth knowing: -G, --get combines with --head. The manual notes that when -G is used with --head, "the POST data is instead appended to the URL with a HEAD request" — useful when you need query parameters built from key-value pairs on a HEAD.
Reading the Response Properly
Getting the most from what comes back.
Status first. 200 exists. 301/302 moved — add -L to follow. 403 refused. 404 gone. 405 means the server does not accept HEAD, so retry as a GET. 429 means slow down.
Content-Length if present, remembering the spec permits omitting it.
Accept-Ranges: bytes means resumable downloads and range requests are available.
Last-Modified and ETag enable conditional requests. -z sends If-Modified-Since — the manual describes it as requesting "a file that has been modified later than the given time and date" — and --etag-compare handles the ETag side. A 304 Not Modified costs almost nothing and is the correct way to poll a resource repeatedly.
Content-Type tells you what you would have received. text/html where you expected JSON usually means an error or login page.
For machine consumption, skip the text parsing entirely:
curl -sI -o /dev/null -w '%{header_json}' https://example.com | jq
%{header_json} emits all response headers as JSON with lowercase names and array values, which handles repeated headers correctly and removes the need for a parser. We covered it and the other inspection options in showing response headers with curl.
People Also Ask
How do I send a HEAD request with curl?
curl -I https://example.com. The long form is --head. Do not use -X HEAD — the manual states explicitly that it "does not suffice" for a proper HEAD request, because it changes only the method string while curl continues to expect a response body.
Why does curl -X HEAD hang?
Because -X changes only the word in the request line, not curl's behaviour. curl still waits for a response body, while the server correctly sends none, since RFC 9110 requires that a server "MUST NOT send content" in a HEAD response. Use -I instead.
What is the difference between HEAD and GET?
HEAD is identical to GET except the server must not send the body. It exists to obtain metadata without transferring content, typically for link checking or freshness testing. Servers should send the same headers as they would for GET, but may omit ones only computed while generating content.
Does HEAD always return the same headers as GET?
No. The spec says servers SHOULD send the same headers but MAY omit those "for which a value is determined only while generating the content" — it names Content-Length and Vary as examples. It calls these minor inconsistencies preferable to generating and discarding a body.
How do I get a file size without downloading it?
curl -sI URL | grep -i content-length. Be aware the header may be absent for dynamically generated content, which the spec permits. For a more reliable answer at almost no cost, request a single byte with -r 0-0 and read the Content-Range header.
Why does a URL work in a browser but return 405 to curl -I?
Because the server does not accept HEAD on that endpoint. 405 Method Not Allowed is a valid response to HEAD from a server that handles GET fine. Retry as a GET with the body discarded: curl -sS -o /dev/null -D - URL.
Can I send a HEAD request with a body?
You should not. RFC 9110 says content in a HEAD request "has no generally defined semantics", cannot alter the request's meaning, and "might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack". Clients SHOULD NOT generate content in a HEAD.
Is HEAD useful for checking whether a proxy is working?
Partially. It confirms connectivity and returns the status code cheaply, which is a good smoke test. It does not tell you whether a real GET would succeed, because anti-bot layers frequently treat the two differently. Validate with real GETs on a sample before trusting HEAD results across a list.
Wrapping Up
Two commands cover this entirely. curl -I URL for a proper HEAD request, and curl -sS -o /dev/null -D - URL when you want the headers a real GET would produce. What does not work is -X HEAD, and the manual says so in terms: it changes the word and not the behaviour, so curl waits for a body the server is required not to send.
The judgement call is which of those two you want. HEAD is dramatically cheaper — a few hundred bytes against a full page — which makes it the right tool for link checking, availability monitoring and size estimation at any volume, and especially on metered bandwidth. It is the wrong tool for predicting what a real fetch would return, because servers are permitted to omit content-derived headers, may reject HEAD outright, and frequently route it through different logic.
And if you need the accuracy of a GET without the bandwidth, -r 0-0 is the underused middle path: a genuine GET that fetches one byte. Check for Accept-Ranges: bytes first, since plenty of servers will hand you the whole file anyway.