Access Log Bot Analyzer:
Which Crawler Claims Do the Addresses Support?
Paste raw access log lines — Apache, nginx, or anything close — and every line that claims to be a crawler gets checked against that operator's published IP ranges. The lines that claim Googlebot and come from addresses Google does not publish are grouped at the top, with the offending IPs ready to copy. All of it happens on your machine: the log text never leaves the browser.
0
Impersonating
0
Verified crawler
0
Unverifiable
0
Lines read
Claimed a crawler, address says otherwise
These requests named a crawler and arrived from addresses that operator does not publish. Strong evidence, not absolute proof — confirm anything you intend to act on before you act on it.
| IP | Claimed | Requests | Example path | Responses | Confirm |
|---|
Crawlers that checked out
| Crawler | Requests | Unique IPs | Responses | Most requested paths |
|---|
Unverifiable — not the same as fake
These operators publish no machine-readable IP ranges, so nobody outside those companies can confirm or deny these claims. Do not build a blocklist from this table.
| Claimed | Requests | Unique IPs | Why |
|---|
Run details
Quick Answer: How Do You Find Fake Bots in an Access Log?
Pair each line's user agent with the address on the same line, then check that address against the operator's published ranges. A line claiming Googlebot from an address Google does not publish is the finding. Both halves have to come from the same request.
Jessica Wright
Cybersecurity Threat Researcher
Jessica works on bot verification and IP reputation, separating traffic that is what it says from traffic that only claims to be.
This page is the one I wanted first and built last, because it needed the other eight to exist. A bulk address checker cannot see what each request claimed. A user agent checker handles one line at a time. The access log is where the two halves sit together on the same row, and that pairing is the only thing that turns a suspicion into a finding.
The one decision I would defend hardest is that it runs in your browser. Asking somebody to upload a server log to a web form is asking for a record of who visited their site and what they read. Nobody should hand that over casually, and I did not want to be the reason they did. The vendor ranges come down when the page loads; everything after that happens on your machine. Open the network tab and check — there is nothing going back.
Last reviewed 24 August 2026 · 2488 published prefixes loaded · Ranges built 2026-08-26 23:10 UTC
View all articles by Jessica WrightWhy Is the Log Line the Only Unit That Settles This?
Crawler verification needs two facts that have to belong to the same request: what it claimed to be, and where it came from. A user agent on its own is a sentence the visitor wrote about itself. An IP address on its own tells you whose network sent the packets but nothing about what they were pretending to be.
An access log line carries both, side by side, guaranteed to be the same request. That is why this page takes log lines rather than a list of addresses, and it is the difference between "this address is not Google's" and "this request said it was Googlebot and it is not".
The pairing has to be genuine. Taking an address from one line and a user agent from another produces a confident answer to a question nobody asked. It sounds obvious and it is a mistake people make constantly when reading logs by eye, which is most of the reason to let something else do the pairing.
What that pairing catches
HUMAN Security tracked sixteen well-known AI crawler user agents over a two-week window and found 5.7% of the traffic carrying them was forged. For the ChatGPT-User string it was 16.7% — roughly one request in six claiming to be that agent was not. Those numbers are invisible in a log until something checks the claim against the address.
The cost of not looking runs both ways, and unevenly. Imperva has written that accidentally blocking Googlebot loses rankings sometimes worth millions in brand equity. Letting a forged crawler through costs bandwidth and some scraped content. That asymmetry is why this page groups unverifiable traffic separately rather than lumping it in with impersonation — the two demand very different responses.
For a single suspicious line, the user agent checker takes one string and one address and adds forward-confirmed reverse DNS, which is a stronger check than anything possible at log scale.
What Should You Paste, and How Do You Trim a Big Log?
Raw lines, unedited. The parser finds the client address across the first few fields rather than insisting it comes first, and that matters more than it sounds.
Apache's vhost_combined format — the default on a lot of cPanel hosting — puts the domain and port ahead of the address. IIS and CloudFront put a timestamp there instead. All three work here, alongside Apache combined and common and nginx's default. Anything the parser cannot read is counted and reported rather than quietly dropped.
Trimming a log that is too big
Twenty thousand lines is the ceiling here, which is twenty times what Screaming Frog's free tier allows. A real log is still much larger than that, and mostly irrelevant to this question — the great majority of lines are human traffic and static assets.
One grep reduces most logs by an order of magnitude while keeping exactly what matters. awk does the same job if you prefer it, and both ship with every Unix-like system you are likely to have a log on:
grep -iE "bot|crawler|spider" access.log > bots.log— keeps only lines whose user agent mentions a bot, which is every line this page can say anything about.grep -i "googlebot" access.log | head -5000— narrower still, if you are chasing one specific claim.- Add
| grep -vE "\.(js|css|png|jpg|svg|woff2?)"to drop static asset requests, which are noise for this purpose. awk '$0 ~ /bot|crawler|spider/' access.log > bots.log— the same filter in awk, if that is the tool already in your fingers.- Logs are usually gzipped once rotated. This page reads
.gzfiles directly, soaccess.log.1.gzcan go straight into the file picker without unpacking it first.
Where the log lives. On most Apache installs it is /var/log/apache2/access.log; on nginx, /var/log/nginx/access.log. On shared hosting it is usually downloadable from the control panel under raw access logs. Cloudflare, CloudFront and similar produce their own logs, and those are the ones to use if you sit behind a CDN — your origin log will show the CDN's addresses rather than the real client.
If you only have addresses and no user agents, the bulk checker takes 200 at once and collapses the verified ones into a prefix allowlist.
What Do the Three Groups Actually Mean?
Results split three ways, and confusing the second with the first is how people block traffic they wanted.
Impersonating
The request named a crawler, and the address is in none of the prefixes that operator publishes. This is the strongest signal available from a log, and it is still not absolute: published lists go stale, and an address an operator has used and released reads exactly the same way. Microsoft's list has not been revised since January 2024, which is worth remembering before treating a Bingbot miss as certain.
Verified
The claim and the address agree. Nobody outside the operator can arrange that, so this group can be trusted. What it does not tell you is whether you want the traffic — a genuine crawler can still be hitting you hard enough to cause problems, and the answer to that is a crawl rate setting rather than a block.
The response column beside it is where the useful reading usually is. A verified crawler that spends a fifth of its visits on 404s is not a security problem, it is a crawl budget problem — every one of those requests is a page it will not fetch instead. The results call it out when it crosses that line, and it is often the first time anyone has noticed.
Unverifiable
The operator publishes no machine-readable ranges at all. Apple, Amazon, Yandex, ByteDance, Meta, DuckDuckGo, Baidu and the SEO crawlers all sit here. Nobody outside those companies can confirm or deny a claim naming them, and treating this group as impersonation would block a great deal of legitimate traffic.
The unverifiable table is not a blocklist. It is a list of claims that cannot be tested, which is a statement about the operators rather than about your visitors. If you want to act on that traffic, act on its behaviour — request rate, paths, whether robots.txt was fetched first — not on the fact that nobody publishes a way to check it.
Report from the verified set, not the raw match
This is the part most crawl reports get wrong, and it costs more than the security question does.
The usual way to count crawler activity is to grep the log for Googlebot and total the lines. That number is inflated by every scraper wearing the name — Incapsula's measurement put roughly one Googlebot in twenty-five as fake, and HUMAN's more recent AI-crawler figures run higher. Present that inflated total to a client or a capacity planner and every decision downstream inherits the error. Crawl frequency looks healthier than it is. Server sizing gets padded for traffic that was never Google. And content that appears well-crawled may barely be crawled at all.
The verified group above is the number that belongs in a report. It is smaller, it is real, and it is the one that tracks with what actually gets indexed.
Read the response column, not just the counts
Classifying a crawler is half the picture. What your server returned to it decides the indexing outcome, and that is what the response mix beside each crawler shows. A verified Googlebot spending a fifth of its visits on 404 responses is burning crawl budget on pages that are not there. A wall of 5xx status codes or soft 404s is worse — it is the kind of thing that quietly stalls indexing while everything looks fine in a dashboard.
The results call that out automatically when a verified crawler crosses a fifth of its requests on missing or broken pages, because nobody goes looking for it unprompted.
What to do with the impersonation list
Confirm before acting. An ASN lookup names the network an address really belongs to, and that is the durable thing — whoever is doing this can change addresses in seconds but not networks. A blacklist check shows whether an address already has a reported abuse history. And for the addresses that matter most, the operator pages run reverse DNS as a second check: Googlebot across its five categories, Bingbot where Microsoft's own two checks disagree, and PerplexityBot where a non-match means three different things.
Once you know what you want to allow and refuse, the crawler blocker writes robots.txt and firewall rules from the same live ranges this page just matched against.
What this tool cannot do
This is a narrow tool wearing a broad name, so the boundaries matter.
It is not a crawl budget analyser
Directory-level breakdowns, orphan page detection, response time analysis, imported crawl comparison — all of that is what Screaming Frog's Log File Analyser and the enterprise tools do, and they do it properly. This page answers one question about identity.
No reverse DNS
The stronger check needs a DNS round trip per address, which cannot run in a browser at all. Published ranges are the half that works client-side. Take individual addresses to the operator pages for the other half.
Ranges are as fresh as the page load
The vendor files come down when you open the page and stay in memory. That is fresher than a snapshot baked in weeks ago, and it is not live during a long session. Reload before a serious run.
A feed that fails makes real bots look fake
If a vendor file does not load, every request from that operator lands in the impersonation table. The banner at the top names which operators are affected when it happens, and that banner is the most important thing on the page when it appears.
Behind a CDN, the addresses are wrong
An origin log behind Cloudflare or CloudFront records the CDN's edge addresses, not the visitor's. Every line will read as unmatched. Use the CDN's own logs, or configure the origin to record the forwarded client address.
20,000 lines a run
A ceiling to keep the browser responsive rather than a technical limit — five thousand lines parse and match in about fifty milliseconds. For a bigger log, filter with grep first; most shrink by an order of magnitude once static assets and human traffic are out.
And the guide to IP spoofing covers why a header is never identity in the first place.
Frequently asked questions about access log bot analysis
How do I find fake Googlebot in my access logs?
Paste the log lines into the box above. Every line is split into its address and its user agent, and any line whose user agent claims a crawler is checked against that operator’s published IP ranges. Lines that claim a crawler and come from an address the operator does not publish are grouped together at the top as impersonation. That pairing only works because both halves come from the same log line.
Is my log file uploaded to your server?
No. The vendor IP ranges are sent to your browser when the page loads, and the parsing and matching then happen entirely on your machine in JavaScript. Your log text is never transmitted, never written to disk here, and never logged. You can confirm that by opening your browser’s network tab while you run it — there is no request.
What log formats does it read?
Apache combined and common, nginx’s default, and most variants of both. The parser looks for an IP address at the start of the line and a quoted user agent at the end, which covers the great majority of real-world access logs. Tab-separated CloudFront-style lines work too. Lines it cannot read are counted and reported rather than dropped silently.
How is this different from Screaming Frog's Log File Analyser?
Screaming Frog’s tool is far more capable and worth its licence — crawl budget by directory, orphan pages, response times, imported crawl comparison. It is a desktop install, free to 1,000 log events and then about GBP 99 a year. This page does one narrow slice of that job: it pairs the crawler claim with the address and tells you which requests do not add up. No install, no event cap, and the ranges are read live rather than from a snapshot.
Why do some bots show as unverifiable rather than fake?
Because their operators publish nothing to check against. Apple, Amazon, Yandex, ByteDance, Meta, DuckDuckGo, Baidu and the SEO crawlers like AhrefsBot and SemrushBot publish no machine-readable IP range file. A request claiming to be one of those cannot be confirmed or denied from outside, so it is reported as unverifiable. Treating that group as fake would block a lot of legitimate traffic.
How many log lines can it handle?
The interface is capped at 5,000 lines a run, which keeps the browser responsive. That is a deliberate ceiling rather than a technical one — the matching itself is fast. For a bigger log, filter it first: grep -iE "bot|crawler|spider" access.log reduces most logs by an order of magnitude and keeps exactly the lines this page cares about.
What does the impersonation count actually prove?
That a request named a crawler and arrived from an address that crawler’s operator does not publish. It is strong evidence and not absolute proof — published lists go stale, and an address an operator has retired reads the same way. For anything you intend to act on, confirm the individual address on the operator’s own page, where forward-confirmed reverse DNS runs as a second check.
Can I see which pages the crawlers are hitting?
The results show the most-requested paths per crawler, which is enough to spot a bot stuck in a parameter loop or hammering one section. It is not a full crawl budget analysis — directory-level breakdowns, orphan page detection and response-time analysis are what the dedicated log file analysers are for, and they do it properly.
Related IP & crawler tools
This page reads whole log lines. These take it further one address at a time.
Browse the full set on the TrustMyIP tools directory.
Found impersonators? Now decide what to do
Blocking single addresses is whack-a-mole. Find the network behind them, then write rules that hold — from the same live vendor ranges this page just used.
Last updated 24 August 2026 · 2488 prefixes from 14 of 14 vendor feeds · Your log is never uploaded