Crawler
RankMascotBot
The short version
- User agent:
RankMascotBot/0.1 (+https://rankmascot.com/bot) - Crawls from:
51.79.158.36, reverse DNSbot.rankmascot.com - Why: we audit a site only when its owner has asked us to. We are not a general web crawler and we do not build a public index.
- Problem with us? Email abuse@rankmascot.com. We would much rather hear from you directly than from your hosting provider.
On this page
1. What RankMascotBot does
Rank Mascot is an SEO auditing tool. When someone signs up and asks us to audit a website, this crawler fetches that site's pages so we can check them for technical problems: broken links, redirect chains, missing canonicals, duplicate titles, pages that search engines cannot reach, and so on.
We only crawl sites our customers have asked us to audit. We do not crawl the web at large, we do not build a search index, and we do not sell or republish the content we fetch. If we are on your site, someone with access to your project asked us to look at it.
We store a copy of the page content we fetch so we can compare crawls over time and tell the site owner what changed. That copy is private to their account.
2. Verifying it is really us
Anyone can put our name in a user agent string. Do not trust the user agent on its own. Verify the IP instead, the same way you would verify Googlebot: a reverse lookup, then a forward lookup, and check that they agree.
# 1. what hostname does the IP claim?
$ dig +short -x 51.79.158.36
bot.rankmascot.com.
# 2. does that hostname resolve back to the same IP?
$ dig +short bot.rankmascot.com
51.79.158.36If both steps agree, the request is genuinely from us. If they do not, it is not us, and you should feel free to block it.
Our full published range is a single address today: 51.79.158.36. If we add more, they will be listed here before they are used, and they will always have matching reverse DNS under rankmascot.com.
3. How fast we crawl
Politeness is a correctness requirement for us, not a courtesy. A crawler that knocks a site over has damaged the thing it was hired to measure.
| Behaviour | What we do |
|---|---|
| Concurrent requests | At most 6 open connections to a single site at a time, usually fewer |
| Request rate | Rate limited per host, with a token bucket sized well below what a small site can absorb |
Crawl-delay | Honoured where your robots.txt sets it, and it overrides our own default |
| On HTTP 429 | We back off exponentially and slow down for the rest of the crawl |
| On repeated 5xx | We assume we are part of the problem and stop |
| Rendering | Most pages are fetched as plain HTML. We only run a browser on pages that need JavaScript to show their content |
If we are still too aggressive for your setup, set a Crawl-delay or email us and we will lower our ceiling for your domain specifically.
4. What we never do
- We do not scan for vulnerabilities. We never probe for
/.env,/.git, exposed backups, admin panels or anything whose absence would be a security finding. If something is requesting those paths, it is not us. - We do not ignore
robots.txt. If you disallow us, we stop, including for sites our own customers own. - We do not disguise ourselves. We never send a browser user agent to get past bot detection, and we never crawl from residential or rotating proxy addresses.
- We do not submit forms, follow logout links, or send anything other than
GETandHEAD. - We do not attempt to log in, and we do not use credentials unless a customer has explicitly configured them for their own site.
5. Letting us through a CDN or WAF
If you are a Rank Mascot customer and your audit is coming back with missing or blocked pages, your CDN is probably challenging us. Allowing the IP through fixes it.
| Platform | Where to add 51.79.158.36 |
|---|---|
| Cloudflare | Security → WAF → Tools → IP Access Rules, action Allow. If you use Bot Fight Mode, add a Configuration Rule to skip bot protection for this IP |
| AWS WAF | Create an IP set with the address, then an Allow rule above your bot control rules |
| Akamai | Add to a Network List, referenced from a Bot Manager exception |
| Sucuri | Firewall → Access Control → Allowlist IP Address |
| Wordfence | Firewall → All Firewall Options → Allowlisted IP addresses |
| Imperva / Incapsula | Security → IP Allowlist, plus a bot exception if Advanced Bot Protection is enabled |
| nginx / Apache | Allow the address ahead of any rate limiting or bot rules you have written yourself |
Allowlisting by user agent alone is weaker than allowlisting by IP, because a user agent is trivial to copy. If your platform supports it, prefer the IP, or verify with reverse DNS as described above.
6. Blocking us
You are entitled to. Add this to your robots.txt and we will stop on our next crawl:
User-agent: RankMascotBot
Disallow: /We re-read robots.txt at the start of every crawl and cache it briefly, so a block takes effect within a few minutes rather than instantly. If you need us gone immediately, email us and we will stop the crawl and add your domain to a permanent exclusion list.
Before you block: if you got here because a crawl was slowing your site down, tell us instead. We can lower our rate for your domain to something you are comfortable with, which keeps the audit working for whoever asked for it.
7. Contact
Abuse reports, rate complaints, exclusion requests and anything else about this crawler: abuse@rankmascot.com. We read it, and we would genuinely rather resolve something with you directly than have it reach our hosting provider first.
If you are reporting a problem, the most useful things to include are the timestamps, the source IP, a few of the requested paths, and the domain affected. That lets us find the exact crawl and stop it.