Our stake, plainly: we are Geonode and we sell proxies, and user-agent questions usually arrive alongside "why am I being blocked". The honest answer is that the user agent is one of the weaker signals available, and a browser string on a request whose TLS fingerprint says otherwise is worse than no disguise at all — you have created an inconsistency that a real browser never produces. There is a section on that below. The strategy that works more often than people expect is the opposite one: say who you are, provide a contact URL, and behave reasonably. Anonymous automation gets blocked far more readily than identified automation, and identifying yourself is free.
The Syntax
curl -A "MyBot/1.0" https://example.com
The curl manual documents -A, --user-agent <name>: "Specify the User-Agent string to send to the HTTP server. To encode blanks in the string, surround the string with single or double quote marks."
The default is stated too: "By default, curl uses curl/VERSION, such as User-Agent: curl/8.22.0."
So without any option, every request you make announces itself as curl and its version. Some servers respond differently on that basis, which is worth knowing before you conclude a site is broken.
The equivalent using a generic header:
curl -H "User-Agent: MyBot/1.0" https://example.com
Both produce the same result — the manual notes the header "can also be set with the --header or the --proxy-header options". -A is shorter; -H is consistent with how you set every other header, which matters if you are generating commands programmatically.
If you give it several times, "the last set value is used".
Removing the Header Entirely
A case people do not know exists, and the manual is specific about the distinction:
If you give an empty argument to
--user-agent(""), it removes the header completely from the request. If you prefer a blank header, you can set it to a single space (" ").
curl -A "" https://example.com # no User-Agent header at all
curl -A " " https://example.com # User-Agent: (empty value)
These are genuinely different requests. Sending no header is not the same as sending an empty one, and servers can distinguish them.
Both are unusual, and unusual is itself a signal. A request with no user agent at all is rarer than one identifying as curl, so removing the header to seem less conspicuous generally achieves the opposite.
The same mechanism works through -H, and the manual explains the syntax for both cases: "Remove an internal header by giving a replacement without content on the right side of the colon, as in: -H "Host:"", while a header with an empty value needs a semicolon — -H "X-Custom-Header;" sends X-Custom-Header:.
Why the Default Matters
curl/8.22.0 is a perfectly honest string, and it has consequences.
Some servers block it outright. A blanket rule against known automation tools is easy to write and common enough that you will meet it.
Some serve different content. Simplified markup, no JavaScript-dependent sections, occasionally an entirely different page.
Some log it and do nothing. The most common case by far.
Some CDNs treat it as a signal among many rather than a decision on its own.
The practical consequence is that "it works in my browser but not in curl" has several possible causes, and the user agent is only one. Before changing it, check whether the difference is actually JavaScript — curl does not execute it, so a page assembled client-side will look nearly empty to curl no matter what header you send. That is not a blocking problem and no user agent fixes it.
Why Impersonating a Browser Often Backfires
The section worth reading before you paste a Chrome string.
A user agent is a claim. A modern anti-bot system checks that claim against evidence, and there is plenty of it:
TLS fingerprint. How a client negotiates TLS — cipher suite order, extensions, supported groups — produces a signature commonly summarised as JA3 or JA4. curl's is not Chrome's, and no header changes it. A request claiming to be Chrome with curl's TLS handshake is more identifiable than one honestly claiming to be curl, because real Chrome never produces that combination.
Header set and ordering. Browsers send a characteristic set of headers in a characteristic order — Accept, Accept-Language, Accept-Encoding, Sec-Fetch-*, and more. curl sends three or four. Claiming Chrome while sending curl's header set is a visible contradiction.
HTTP version and behaviour. Connection reuse, multiplexing, header compression details.
Behaviour. Real browsers fetch the CSS, the images and the scripts. A client that fetches one HTML document and nothing else does not look like a browser regardless of what it says.
So the honest hierarchy is: an accurate user agent is consistent and unremarkable; a fabricated one is inconsistent and remarkable. If you genuinely need browser-shaped requests, you need a browser — Playwright or a similar tool — not a header. We covered the surrounding trade-offs in taking screenshots with Playwright.
There is a narrow legitimate middle ground: testing your own site's user-agent handling, or fetching content that a site serves differently to mobile clients. Those are checks against your own systems or benign content negotiation, and they are fine.
What to Send Instead
For automated clients, the string that gets blocked least is one that says what you are.
curl -A "AcmePriceBot/1.2 (+https://acme.example.com/bot)" https://example.com
The convention has three parts: a name, a version, and a URL where someone can find out what you are and how to reach you. It works because it gives the site operator options other than blocking. An unexplained pattern of requests is a problem to be stopped; an identified crawler with a contact page is a decision to be made, and frequently the decision is to allow it.
Three practical benefits, all real:
robots.txt can address you specifically. Rules are matched by user-agent token, so a site can grant your crawler an allowance it does not grant everyone. That cannot happen if you are anonymous.
Operators can contact you before blocking. This happens more often than people expect, and it is a much better outcome than discovering a block three weeks later.
It supports asking for access. "We are the crawler identified as AcmePriceBot; here is what we do" is a conversation that can go somewhere. "We are an unidentified script" is not.
Match it with behaviour that fits the claim: honour robots.txt, respect Crawl-delay and Retry-After, keep your rate modest, and cache so you never fetch the same unchanged resource twice.
Setting It Consistently in Scripts
For anything beyond one command, put the string in one place.
UA="AcmePriceBot/1.2 (+https://acme.example.com/bot)"
curl -sS --fail --location \
--user-agent "$UA" \
--connect-timeout 5 --max-time 30 \
"$URL"
Or set it once in a curl config file, which keeps every invocation consistent without repeating the flag:
# bot.conf
--user-agent "AcmePriceBot/1.2 (+https://acme.example.com/bot)"
--location
--show-error
curl -K bot.conf "$URL"
A caution about ~/.curlrc specifically: it applies to every curl invocation by that user, including commands you did not write. Setting a bot user agent there means every ad hoc request you ever make identifies as your crawler, which produces confusing results months later. Use a named config file with -K for anything workload-specific.
If you need several user agents across a workload, keep them in an array and select deliberately rather than at random — random rotation within a session produces a client that appears to change browser mid-visit, which is another inconsistency rather than a disguise.
Checking What You Actually Sent
The habit worth building, because assumptions about headers are wrong surprisingly often.
curl -v -A "MyBot/1.0" https://example.com 2>&1 | grep -i '^> user-agent'
Verbose output goes to stderr, hence the 2>&1. The > lines are what curl transmitted.
Or ask a service to echo it back:
curl -sS -A "MyBot/1.0" https://httpbin.org/user-agent
This matters more than it sounds because of a specific behaviour the manual describes: "if you should add a custom header that has the same name as one of the internal ones curl would use, your externally set header is used instead of the internal one". So combining -A with -H "User-Agent: ..." means one silently wins, and knowing which requires looking. The manual's own advice is worth repeating: "You should not replace internally set headers without knowing perfectly well what you are doing."
The same check applies through a proxy, where --proxy-header sets headers on the proxy connection rather than the target request — a distinction that catches people when a header they set appears not to arrive.
Reading a User-Agent String
Worth understanding, both because you may need to construct one and because the format explains why browser strings look so strange.
A modern Chrome string looks roughly like this:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36
Almost none of that is true. It is not Mozilla, it is not Safari, and AppleWebKit/537.36 has been frozen for years. The string is a fossil record of two decades of content negotiation: every browser added the tokens of its predecessors so that servers sniffing for a competitor would serve it the good version of the page. The result is a format that carries almost no reliable information and is nonetheless parsed by a great deal of software.
The structural grammar underneath is simple: a sequence of Product/Version tokens, each optionally followed by a parenthesised comment. That is the whole specification, and it is why a string like AcmePriceBot/1.2 (+https://acme.example.com/bot) is well-formed — a product token, a version, and a comment containing a URL. The + prefix on the URL is a convention rather than a requirement, and it is widely recognised.
The mobile question. Many sites serve different markup to mobile clients, and the distinguishing token is usually Mobile somewhere in the string. If you are legitimately checking how a page renders for mobile visitors, sending a mobile user agent is ordinary content negotiation rather than impersonation — though the same caveat applies as always: a mobile string on a desktop-shaped connection with a desktop viewport is only half a claim, and a real mobile browser would differ in several other ways.
And the version-freezing trend. Browsers have been progressively reducing the detail they expose in this header, moving capability information to structured client hints instead. The practical implication is that user-agent parsing is a declining source of information for everyone — including the sites deciding what to do about your requests, which is another reason it is a weak signal to build a strategy around.
When the User Agent Is Not the Problem
Cases where changing it will not help, listed because people try it first.
When the content is rendered by JavaScript. curl does not execute scripts. A near-empty response means the page is assembled client-side, and the fix is a browser or the underlying API, not a header.
When you are rate-limited. A 429 is about volume, not identity. Slowing down helps; a new user agent does not.
When the address is the issue. If a whole range is blocked, every request from it fails regardless of headers.
When authentication is required. A 401 wants credentials.
When the TLS fingerprint gives you away. Covered above, and the reason browser impersonation via headers alone tends to disappoint.
When the site simply does not want automated traffic. Some sites state this in their terms and enforce it. Changing a header does not change the terms, and a site that has told you not to crawl it has given you information rather than a puzzle.
The diagnostic that separates these quickly: request the same URL from a normal browser on the same connection. If the browser works and curl does not, compare the two requests header by header — and if the only difference that matters turns out to be something you cannot change from the command line, that is your answer.
People Also Ask
How do I set a User-Agent in curl?
curl -A "MyBot/1.0" URL, or equivalently curl -H "User-Agent: MyBot/1.0" URL. Quote the string if it contains spaces. If given more than once, the last value wins.
What is curl's default User-Agent?
curl/VERSION — for example curl/8.22.0. It is sent on every request unless you override or remove it, and some servers respond differently on the basis of it.
How do I remove the User-Agent header in curl?
curl -A "" URL removes the header entirely. curl -A " " URL sends it with an empty value, which is a different request. Note that sending no user agent is more unusual than sending curl's default, so it draws more attention rather than less.
Should I fake a browser User-Agent with curl?
Usually not. A browser string on a request whose TLS fingerprint, header set and ordering are all curl's is an inconsistency that real browsers never produce, which makes you more identifiable rather than less. If you need browser-shaped requests, use a browser.
Will changing the User-Agent stop me being blocked?
Rarely on its own. It helps against blanket rules that reject known tools, and does nothing against rate limits, address-based blocks, TLS fingerprinting or behavioural analysis. Identifying yourself honestly with a contact URL often works better than disguise.
What should a bot's User-Agent look like?
Name, version and a contact URL: AcmePriceBot/1.2 (+https://acme.example.com/bot). This lets robots.txt address you specifically, lets operators contact you instead of blocking you, and gives you standing to ask for access.
Can I set a different User-Agent per request?
Yes — -A applies to the invocation, so give a different value each time, or use --next to run several operations with different options in one command. Avoid rotating randomly within a session, which produces a client that appears to change browser mid-visit.
Why does my custom User-Agent not appear?
Most likely you set it twice by different means, since curl uses your externally set header instead of its internal one and the last definition wins. Check with curl -v ... 2>&1 | grep -i '^> user-agent' to see what was actually transmitted.
Wrapping Up
Setting a user agent in curl is one flag, and the choice of value matters more than the syntax.
The default announces curl honestly, and honesty is a defensible position: consistent, unremarkable, and occasionally the reason a site treats you reasonably. Removing the header is possible — an empty argument drops it entirely, a single space sends it blank — and both are more unusual than the default, which is the opposite of what people usually intend.
Copying a browser string is the common instinct and the weakest option, because the claim is checkable. TLS fingerprints, header composition and ordering, and whether you fetch the page's subresources all contradict it, and a contradiction is more identifiable than a plain admission. If browser-shaped requests are genuinely required, a browser is the tool.
The approach that works better than most people expect costs nothing: a name, a version and a URL where someone can find out what you are. It lets robots.txt address you, lets an operator email you instead of blocking you, and turns "unexplained traffic" into "a crawler with an owner" — which is a much better position to be in when someone is deciding what to do about you.