Skip to content
mavirank
Register
Open app

MavirankBot

MavirankBot is the program mavirank uses to fetch web pages. It reads a page so that the page can be checked for SEO, technology and security. It does not build a search index: only the evaluated values of a page end up in a report.

How to recognize it

Every request MavirankBot sends carries the user agent below. The address in brackets is this page.

MavirankBot/1.0 (+https://mavirank.com/bot)
  • Requests are plain GET requests over HTTP or HTTPS. The bot does not run JavaScript, does not keep cookies and follows at most 10 redirects.
  • It connects over IPv4. We do not publish a list of IP addresses; recognize the bot by MavirankBot in the user agent.
  • Customers can set a different user agent for audits of their own pages, for example for a staging system. Those requests follow the same limits.

When it fetches pages

MavirankBot does not roam the web on its own. It fetches a page only when someone asks mavirank to check it.

Audits ordered by customers
A customer enters a URL, or a list of URLs, to have it checked. The bot fetches that page and a few supporting files of the same site, such as robots.txt and the favicon.
Recurring audits of websites
Customers register a website. The bot reads its sitemap and audits the pages listed there on the schedule the customer chose.
Public SEO tools
A visitor of our free tools enters a URL or a host name. The bot fetches what that one tool needs, once.

robots.txt

Scheduled audits, sitemap crawls and bulk audits obey robots.txt. The rules for MavirankBot apply; without such a group, the rules for *. A URL that is disallowed there is not fetched.

Allow and Disallow are read as in RFC 9309. Crawl-delay is not evaluated; the limits below always apply. The file is cached for up to one hour, so a change takes effect within the hour.

When a person starts a single audit or uses a public tool for one URL, the bot fetches that one page even if robots.txt disallows it, as that person's browser would, and names the rule in the result.

Request rate

  • At most 2 requests per second to one host name, counted over all customers together.
  • At most 8 requests per second to one IP address, so many host names on one server do not multiply the load.
  • A single audit makes at most 20 requests.

How to keep it out

To keep MavirankBot away from your site, add this group to your robots.txt. To exclude only a part, name its path instead of /.

User-agent: MavirankBot
Disallow: /

This stops scheduled audits, sitemap crawls and bulk audits. Single checks started by a person are refused when your web server or firewall rejects requests whose user agent contains MavirankBot.

Report a problem

Is MavirankBot causing trouble on your site, or do you think someone is misusing it? Tell us the host name and the time of the requests, and we will look into it.

Go to the contact page