A test published on November 12, 2026 by Massive, a company that sells residential proxies and page rendering services for scraping, checked which User-Agent header 17 open-source scraping tools send by default and whether they honor robots.txt. The results matter to anyone running web servers: ten of the tools clearly identify themselves, four pose as a regular browser, and only one of the 19 setups tested fetched robots.txt.
Scraper User-Agents in 30 seconds
- 10 of 17 tools name themselves in the User-Agent, such as python-requests or Scrapy.
- Playwright and Puppeteer give themselves away with HeadlessChrome.
- Scrapling, Crawl4AI and both Crawlee versions send a browser-like string by default.
- Only Scrapy, with the settings its project generator writes, read robots.txt; three link-following setups fetched a disallowed page.
The author, Massive community lead Franklin Uche, ran the tests on November 9 against a local server that logged every header. That server spoke only HTTP/1.1, so TLS and HTTP/2 fingerprints, which are also used to detect bots, were not captured. The study comes from a vendor that sells scraping infrastructure, which is worth keeping in mind when reading its advice, but the header data is useful on the server side.
What shows up in your logs
HTTP libraries use a simple product/version format: python-requests/2.34.2, python-httpx/0.28.1, axios/1.21.0 or Python/3.12 aiohttp/3.14.5. Scrapy adds a URL in parentheses, (+https://scrapy.org), the long-standing convention for telling site owners who runs a crawler. These clients send between 2 and 7 headers.
Headless browsers are harder to tell apart, but by default they still give themselves away: Playwright and Puppeteer announce HeadlessChrome/156, and Playwright repeats it in the sec-ch-ua header. curl_cffi sends no User-Agent at all unless its Chrome impersonation mode is on. Browser-like tools send 11 to 18 headers, including client hints and Sec-Fetch-*.
Counting User-Agents in your access log is enough to see which of these clients hit your server. With the nginx or Apache combined log format:
# Top 20 User-Agents
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
# Libraries and headless browsers only
grep -Ei 'python-requests|python-httpx|aiohttp|axios|scrapy|headlesschrome|curl/|wget'
/var/log/nginx/access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rnCode language: PHP (php)
From there you can rate-limit specific clients, block those that add nothing, or ask a named crawler’s operator to behave. Be careful, though: curl and python-requests are also used by monitoring checks, integrations and legitimate API clients.
robots.txt is a request, not a control
Of the 19 setups, only Scrapy with ROBOTSTXT_OBEY = True read robots.txt. That is the value scrapy startproject writes into new projects, although Scrapy’s built-in default is False. Crawlee and Crawl4AI also ship with it off, and HTTP libraries never read it because they only fetch the URLs they are given.
The practical takeaway is that robots.txt states preferences for crawlers that choose to follow them; it is not access control. According to the article, robots.txt is not binding on its own in the US, while in the EU it can act as a text and data mining opt-out, and the European Data Protection Board’s draft scraping guidelines tell scrapers to skip sites that clearly object through it. Those are the author’s readings, not legal advice.
The study itself concedes that the User-Agent is one header among many: sites also look at the TLS handshake, HTTP/2 settings, header order, JavaScript-visible browser properties and IP reputation. Protecting content or controlling load takes rate limits, WAF rules or bot management, not just robots.txt and User-Agent filters.
FAQ
Do scrapers respect robots.txt?
Rarely by default, according to this test: one of 19 setups read it. HTTP libraries never check it and several crawlers ship with it disabled.
How do I spot scrapers in my logs?
Count the User-Agents in your access log and look for libraries such as python-requests, aiohttp, axios or Scrapy, and headless browsers announcing HeadlessChrome.
Is blocking by User-Agent enough?
No. A User-Agent can be changed with one line of code and some tools mimic browsers by default. You also need rate limiting and other signals.
Sources:
