SearchNivoBot

SearchNivoBot reads a small number of pages from a website when somebody submits that website to our free visibility check, to score its SEO, answer-engine and AI-visibility readiness. It does not collect content for training AI models, and it does not resell or republish page content. Results are deleted after 30 days, apart from a short record that the site was checked, when, and its scores.

Identifying the crawler

Requests are sent with this user agent:

SearchNivoBot/1.0 (+https://searchlens-web.onrender.com/bot)

We don't issue tokens, keys or verification headers, and there is nothing to sign up for or install. A user-agent string can be copied by anyone, so if you let SearchNivoBot past your firewall by its user agent, keep the rule in place only while you run a check.

How it behaves

  • Fetches and obeys robots.txt, including crawl-delay directives.
  • Reads XML sitemaps to discover URLs.
  • Requests pages with a delay between fetches, at most two at a time.
  • Backs off when a site asks it to slow down (see below).
  • Sends only GET requests. It never submits forms, logs in, or modifies content.
  • Follows redirects, revalidating each hop, and caps response bodies at 10 MB.

Controlling the crawl rate

To space requests out, set a crawl delay in seconds. A group for SearchNivoBot is used if there is one; otherwise the * group's delay applies:

User-agent: SearchNivoBot
Crawl-delay: 5

With a crawl delay set, SearchNivoBot fetches one page at a time and waits at least that long after each request finishes before starting the next. Delays longer than 30 seconds are treated as 30 seconds. A crawl that cannot read its page allowance at that pace in the time it has stops early, and its report says so.

If your server answers 429 Too Many Requests, or 503 with a Retry-After header, the whole crawl pauses — for the Retry-After time (seconds or an HTTP date, at most 60 seconds), or 2, 4 and then 8 seconds if none is given — continues with fewer parallel requests, and retries that page at most three times. If three pages in a row are still refused after their retries, the crawl stops.

Why your site might block it

A check that ends with "your website turned our crawler away" means your site answered SearchNivoBot with 401, 403, 429 or 503 instead of a page, even where robots.txt allows it. That is almost always bot protection in your CDN, firewall or a security plugin doing its job. It flags our requests because they:

  • come from a cloud server's IP address, not a home or office connection;
  • identify themselves as a crawler rather than a web browser;
  • cannot solve JavaScript or CAPTCHA challenges, because the crawler doesn't run them;
  • read several pages in quick succession, which a strict rate limit can count as too many.

A 401 means the site asked for a login: our crawler never logs in, so a password-protected or unpublished site can't be checked. None of this affects how Google or your real visitors see your site.

Letting the crawler in

You don't need anything from us. Make one change in whatever protects your site, run the check again straight away, and undo the change once the check has finished if you like. When a check is blocked, its page names the service we detected in front of your site.

Wherever a rule can match on the user agent, match requests whose user agent contains SearchNivoBot.

Cloudflare

  1. In the Cloudflare dashboard, open your domain and go to Security → WAF → Custom rules (Security → Security rules in the newer dashboard), then create a rule.
  2. Match requests whose User Agent contains "SearchNivoBot" — or paste the expression below.
  3. Set the action to Skip and tick All remaining custom rules, All rate limiting rules, All managed rules and All Super Bot Fight Mode rules where they are offered. Put the rule first.
  4. If “I’m Under Attack” mode is on, turn it off while the check runs: it challenges every visitor, crawlers included.
(http.user_agent contains "SearchNivoBot")

Cloudflare Free plan: Bot Fight Mode runs outside custom rules, so Skip has no effect on it. Switch it off under Security → Settings (filter for “Bot traffic”) for the few minutes the check runs, then switch it back on.

Wordfence (WordPress)

  1. In WordPress, go to Wordfence → All Options.
  2. If our crawler page lists our IP addresses, add them to “Allowlisted IP addresses that bypass all rules” under Advanced Firewall Options.
  3. Otherwise, under Rate Limiting, temporarily turn off “Enable Rate Limiting and Advanced Blocking”, run the check, then turn it back on.

Sucuri

  1. In the Sucuri Firewall dashboard, add our IP addresses (if our crawler page lists them) to the allowlist under Access Control.
  2. If no addresses are listed, ask Sucuri support to let requests whose user agent contains "SearchNivoBot" through for the check.

Amazon CloudFront / AWS WAF

  1. In AWS WAF, open the web ACL attached to your CloudFront distribution.
  2. Add a rule above your Bot Control and rate-based rules that matches the User-Agent header containing "SearchNivoBot", with the action Allow.

Akamai

  1. Akamai's bot management is usually set up by your developer or IT team. Ask them to add an exception that lets requests whose user agent contains "SearchNivoBot" past bot management and rate limiting.
  2. The exception only has to be in place while the check runs; it can be removed straight after.

Imperva

  1. Imperva's bot management is usually set up by your developer or IT team. Ask them to add an exception that lets requests whose user agent contains "SearchNivoBot" past bot management and rate limiting.
  2. The exception only has to be in place while the check runs; it can be removed straight after.

Fastly

  1. Fastly's bot management is usually set up by your developer or IT team. Ask them to add an exception that lets requests whose user agent contains "SearchNivoBot" past bot management and rate limiting.
  2. The exception only has to be in place while the check runs; it can be removed straight after.

Vercel

  1. A 401 from Vercel is Deployment Protection: check your production domain, not a preview address, or turn protection off for the deployment you want checked.
  2. Otherwise open your project's Firewall. Turn off Attack Challenge Mode if it is on, or add a custom rule matching User Agent contains "SearchNivoBot" with the action Bypass.

Netlify

  1. A 401 from Netlify means the site is password-protected: remove the password (or basic authentication) from the published site, or check a site that is public.
  2. If you use Netlify's firewall or traffic rules, add an exception for requests whose user agent contains "SearchNivoBot".

Shopify

  1. Shopify runs its own protection, which site owners cannot configure.
  2. Make sure the site is published and not behind a password or “coming soon” page — our crawler never logs in.
  3. If the site is public and still refuses us, Shopify is limiting traffic from cloud servers. Try again later, or ask Shopify support.

Wix

  1. Wix runs its own protection, which site owners cannot configure.
  2. Make sure the site is published and not behind a password or “coming soon” page — our crawler never logs in.
  3. If the site is public and still refuses us, Wix is limiting traffic from cloud servers. Try again later, or ask Wix support.

Squarespace

  1. Squarespace runs its own protection, which site owners cannot configure.
  2. Make sure the site is published and not behind a password or “coming soon” page — our crawler never logs in.
  3. If the site is public and still refuses us, Squarespace is limiting traffic from cloud servers. Try again later, or ask Squarespace support.

Webflow

  1. Webflow runs its own protection, which site owners cannot configure.
  2. Make sure the site is published and not behind a password or “coming soon” page — our crawler never logs in.
  3. If the site is public and still refuses us, Webflow is limiting traffic from cloud servers. Try again later, or ask Webflow support.

Other firewalls, hosts and security plugins

  1. Common culprits are a WordPress security plugin (Wordfence, Solid Security, All In One WP Security), your host's firewall (ModSecurity, Imunify360) or a rule that blocks traffic from cloud providers.
  2. Ask your host or developer to allow requests whose user agent contains "SearchNivoBot" — or our IP addresses, if our crawler page lists them — for the few minutes the check runs.

Blocking the crawler

To disallow it entirely, add this to your robots.txt:

User-agent: SearchNivoBot
Disallow: /

The crawler honours this on its next run. If you are the site owner and want a result page about your site removed, contact us (below) with the domain.

Contact

If this crawler is causing load problems or you believe a site is being crawled without authorisation, send us a message with the site's domain.

SearchNivoBot — crawler information · SearchNivo