Screaming Frog SEO Spider is the most-installed SEO tool in agency workflows. Most audits we've seen referenced over the past decade started with a Screaming Frog crawl. It's a good piece of software. It is also a desktop application that has to be running on someone's laptop, that has to be configured per-project, that has to be re-run when anything changes, and that produces a CSV which then has to be opened in Excel and triaged by hand.

This post is for agencies and in-house teams asking the obvious question: what does the API equivalent look like, and is it actually cheaper? The short answer is yes, with a different shape of workflow. The longer answer is below.

What Screaming Frog is good at

Crediting where credit is due. Screaming Frog does several things well:

  • Crawl discovery. Point it at a URL and it walks every internal link. You don't need to know your sitemap, your sitemap doesn't have to be correct, you'll find pages the sitemap forgot.
  • Per-page detail. It pulls headings, meta tags, internal/external links, response codes, redirect chains, and a long tail of properties.
  • One-off depth. Configure once, crawl once, walk away.

If you're doing a single deep audit of a single site and never re-running it, Screaming Frog is fine. Pay the £199/year and move on.

What it isn't good at

Where it breaks down for an agency or in-house team running ongoing audits:

  1. It's a desktop app. Someone's laptop has to be on, awake, and not crashing. For a 50,000-URL site that's a 6-hour crawl on a tuned machine.
  2. Crawls don't compose. Last week's crawl is a CSV on someone's hard drive. There is no built-in "what changed since last week" view that isn't manual diffing.
  3. No native scoring or grading. It tells you a <title> is 67 characters long. It does not tell you "this page scores 84 and your competitor scores 91 and here's why."
  4. No API surface. You cannot trigger it from a CI pipeline, schedule it from a server, or hand a button to a non-technical colleague.
  5. Per-seat licensing. Each analyst needs their own license, their own configuration, their own copy of the CSV.

For a team of three doing weekly audits across twelve clients, those constraints stop being theoretical and start eating real hours.

The API-equivalent workflow

Same outcome — a scored audit of every URL on the site — restructured as a server-side job:

import requests, csv, os
import xml.etree.ElementTree as ET

API_KEY = os.environ["SEOSCORE_API_KEY"]
BASE = "https://api.seoscoreapi.com"

def crawl(site):
    """Pull URLs from the sitemap, recursing into indexes."""
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
    def _collect(url):
        root = ET.fromstring(requests.get(url, timeout=20).text)
        if root.tag.endswith("sitemapindex"):
            urls = []
            for child in root.findall(".//sm:sitemap/sm:loc", ns):
                urls.extend(_collect(child.text))
            return urls
        return [l.text for l in root.findall(".//sm:url/sm:loc", ns)]
    return _collect(f"{site}/sitemap.xml")

def audit_all(urls):
    rows = []
    for i in range(0, len(urls), 50):
        r = requests.post(
            f"{BASE}/audit/batch",
            headers={"X-API-Key": API_KEY},
            json={"urls": urls[i:i+50]},
            timeout=300,
        )
        r.raise_for_status()
        rows.extend(r.json()["results"])
    return rows

urls = crawl("https://client.com")
print(f"Discovered {len(urls)} URLs")
results = audit_all(urls)
print(f"Audited {len(results)} pages")

Sixty lines, end to end. Run it on a server. Output a CSV. Hand the CSV to the analyst, or pipe it into a dashboard, or post it to Slack.

The crawl-discovery gap (and how to close it)

The honest tradeoff: a sitemap-based crawl is not a true crawl. If a page exists but isn't in the sitemap, you won't find it. Screaming Frog walks every link and finds those orphans.

For most sites this matters less than you'd think. Marketing sites publish complete sitemaps. Shopify, WooCommerce, WordPress with Yoast — all generate exhaustive sitemaps automatically. If you've audited the sitemap, you've audited the site that Google can see.

For sites where it does matter (legacy CMS, fragments of static HTML hand-rolled into Apache, archives that nobody linked into the nav), a one-time Screaming Frog crawl to generate the URL list, plus the API for ongoing scoring, is a perfectly reasonable hybrid. The API doesn't have to replace every part of the workflow to be a 10x improvement on the parts it does replace.

What the API does that the desktop can't

The interesting tradeoff isn't "what we lost" — it's "what we gained that wasn't possible before."

Historical comparison. Every audit's score is stored against your key. Run the same audit weekly and you have a per-URL trend line, with deltas surfaced automatically. The historical tracking post has the full API. There is no equivalent in Screaming Frog without manually saving and diffing CSVs.

Composability. The audit results are JSON. Pipe them into anything. We've seen teams build:

  • Slack alerts that fire when any tracked client drops 3+ points.
  • HTML dashboards generated from /history/domains for monthly client reviews.
  • Re-audits triggered automatically when a new deploy lands in production.

Side-by-side comparisons. /compare audits two URLs in parallel and returns a diff. Use it for client-vs-competitor pitches, before/after migration checks, or A/B preview/production validation. The Stripe/Square/Adyen comparison post shows the format.

AI readability and GEO scoring. SEO Score API runs the same 54 SEO checks Screaming Frog covers plus an additional layer specifically for how LLMs read your content. That's not in Screaming Frog at all and isn't going to be — it's a different category of analysis. The GEO optimization guide and AI readability post cover what's in there.

Shareable reports and badges. Every audited URL gets a publicly shareable report at /report/{domain} and an SVG score badge. Drop the badge in client decks; drop the report URL in emails. There's no Screaming Frog equivalent — its outputs are CSVs that only the analyst sees.

Cost comparison, honestly

Screaming Frog SEO Spider: £199/year per seat (~$250/year).

SEO Score API:

Tier Audits/mo Annual cost Use case
Basic 1,000 $180/year Solo SEO, 3-5 small clients
Pro 5,000 $468/year Agency with 10-20 clients, weekly audits
Ultra 25,000 $1,188/year Agency with 30+ clients or one large enterprise site

A two-person agency replacing two Screaming Frog seats (~$500/year) gets the same audit coverage on Pro for less money, plus everything in the "what the API does that the desktop can't" section. A larger agency replacing 4-5 seats (~$1,200/year) lands on Ultra and gets unlimited history retention, which on its own is worth the difference.

The break-even is rough but real: somewhere around 3 seats of Screaming Frog, Ultra is cheaper and strictly more capable.

What we keep Screaming Frog around for

We don't pretend the API replaces 100% of use cases. Where Screaming Frog still wins:

  • Crawling sites with broken or missing sitemaps.
  • Following redirect chains across external links to figure out what an old domain ended up redirecting to.
  • One-shot deep audits where you want every property of every page in a single CSV and you'll never re-run the audit.
  • Legacy site migrations where you need to map old URLs → new URLs by content hash.

Use the right tool for the job. The point isn't to eliminate Screaming Frog — it's to stop using it for the weekly and per-deploy and cross-client work that it was never designed to do, where its constraints turn into real cost.

The migration path

If you're an agency moving from Screaming Frog-only to API-first, the order we'd suggest:

  1. Pick your three highest-effort clients. Audit their sites with the API. Compare the output to your most recent Screaming Frog audit. Confirm the coverage is equivalent for your purposes.
  2. Stand up a weekly job. Cron + a Python script + Slack webhook. The agency monitor setup post has the full walkthrough.
  3. Move quarterly client reports onto the API output. The HTML report at /report/{domain} is shareable; the badge is droppable; the history endpoint generates trend charts.
  4. Use Screaming Frog only for the deep one-off audits where its discovery wins. That's roughly twice a year per client, not weekly.

Most agencies that do this end up shifting 80–90% of their crawler-hours to the API and keeping Screaming Frog around for the long-tail cases. Total cost drops, total throughput goes up, and the analyst time freed up goes into actual fixes instead of running spider crawls overnight.

Getting started

If you're a solo or small agency, Basic ($15/mo) covers 3-5 sites at weekly cadence. If you're running an agency at scale and Screaming Frog has become a bottleneck, Pro ($39/mo) is the sweet spot. For the largest agencies with 30+ clients, Ultra ($99/mo) replaces the seat-licensing math entirely.