--- title: "Free Screaming Frog Alternative: Crawl 1,625 Pages Free" slug: "free-screaming-frog-alternative-freecrawl-guide" published: "2026-07-30" updated: "2026-10-26" validated: "2026-10-26" categories: - "Tools" tags: - "Screaming Frog alternative" - "FreeCrawl" - "FreeCrawl SEO Tool" - "FreeCrawl Mac download" - "FreeCrawl CLI" - "FreeCrawl MCP server" - "open source SEO crawler" - "SQLite crawl database" - "WordPress migration" - "headless website crawling" llm-intent: "reference" audience-level: "intermediate" framework-versions: - "freecrawl" - "node.js" - "npm" - "git" - "typescript" status: "stable" llm-purpose: "Screaming Frog alternative FreeCrawl: install from source, render JS with Playwright, and query the SQLite crawl DB to inventory 1,625 WordPress pages." llm-prereqs: - "Access to FreeCrawl" - "Access to Node.js" - "Access to npm" - "Access to Git" - "Access to TypeScript" llm-outputs: - "Completed outcome: Screaming Frog alternative FreeCrawl: install from source, render JS with Playwright, and query the SQLite crawl DB to inventory 1,625 WordPress pages." --- **Summary Triples** - (FreeCrawl, stores crawl output in, a local SQLite database) - (FreeCrawl, renders JavaScript with, Playwright Chromium in headless mode) - (Workflow, was tested on, macOS 15.7.1 (Apple Silicon) with Node.js 24.12.0, npm 11.6.2, Git 2.52.0) - (Repository, used commit, b91a2b7eae32f0688b11a62a0ce954beea0813df (CLI-only path)) - (Crawl result, recovered, 1,625 HTML pages) - (Crawl result, captured, 92,071 internal link relationships) - (Crawl result, produced a database of, 441 MB) - (Project requirement, declares Node.js, 22 or newer) - (Installation method, is, clone the repo and run npm install (no packaged release in the tested workflow)) - (Database access, can be done, while the crawler is running using the sqlite3 client) - (Execution choices, in the guide, Electron app and Docker were not used — CLI-only) ### {GOAL} Screaming Frog alternative FreeCrawl: install from source, render JS with Playwright, and query the SQLite crawl DB to inventory 1,625 WordPress pages. ### {PREREQS} - Access to FreeCrawl - Access to Node.js - Access to npm - Access to Git - Access to TypeScript ### {STEPS} 1. Confirm toolchain and environment 2. Clone repository and install deps 3. Build TypeScript packages first 4. Build the CLI binary 5. Run a small test crawl 6. Add exclusion regex for external hosts 7. Execute full raw HTML crawl 8. Query the SQLite crawl database 9. Test JavaScript rendering on a sample # Free Screaming Frog Alternative: Crawl 1,625 Pages Free > Use FreeCrawl instead of Screaming Frog: free Mac app, CLI and MCP. Tested commands and SQLite queries from a 1,625-page WordPress crawl. Matija Žiberna · 2026-07-30 I needed a full URL inventory for a WordPress-to-Payload migration, and Screaming Frog's free version stops at 500 URLs. The site had more pages than that in its blog archive alone, so the choice was a licence or a different tool. I went with FreeCrawl, an open-source crawler. In my original Apple Silicon CLI test, it recovered **1,625 HTML pages, 92,071 internal link relationships and a 441 MB SQLite database**. Getting there took some debugging. This article preserves the commands, failures and SQL queries I actually tested, rather than presenting them as a fresh benchmark of the current release. **Updated October 26, 2026:** FreeCrawl has moved on significantly since that test. The current GitHub release is **v1.1.1 (October 25, 2026)**. It now offers downloadable Mac applications, a much more complete desktop SEO interface, an improved CLI and an MCP server for AI-assisted crawling. **You no longer need to build from source or use a terminal just to crawl a website.** The walkthrough further down remains a record of my earlier CLI run. ## FreeCrawl in 2026: desktop app, CLI or MCP? Choose the entry point based on what you are trying to accomplish. | What you need | Best option | What it gives you | | --- | --- | --- | | Inspect a site like Screaming Frog | Desktop app | Crawl a domain and explore SEO issues through tables and reports | | Inventory thousands of URLs for a migration | CLI | Repeatable crawl parameters, exports and a queryable SQLite project | | Ask an AI agent to crawl and investigate issues | MCP server | Tools to start crawls, inspect URL details, query issues and export results | The current project advertises **181 SEO checks**, JavaScript rendering, a desktop link visualiser, Google Search Console and GA4 integrations, scheduled crawls, multiple export formats and an MCP server with 100+ tools. Those are **maintainer-documented capabilities**, not features I have personally retested. The project's stated target of 1M+ URLs is likewise not a benchmark I have validated. Sources: [repository and installation guide](https://github.com/kemalai/FreeCrawl-SEO-Tool), [changelog](https://github.com/kemalai/FreeCrawl-SEO-Tool/blob/main/CHANGELOG.md) and [releases](https://github.com/kemalai/FreeCrawl-SEO-Tool/releases). ### Easiest setup on an Apple Silicon Mac 1. Open the [official GitHub Releases page](https://github.com/kemalai/FreeCrawl-SEO-Tool/releases). 2. Download the **macOS Apple Silicon .dmg** for your M-series Mac (or the Intel build for an older Intel Mac). 3. Open the application, create or open a crawl project, enter the starting website URL and configure the scope, crawl rate and limits. 4. Start crawling and use the URL and issue tables or export the results. The installer includes Chromium for JavaScript rendering. **The releases are not yet notarized for macOS**; read the project's own Gatekeeper instructions before bypassing any warning, and obtain the app only from the official release page. The packaged GUI is the most sensible starting point if you do not need automation. ### Fetch all discoverable pages from the command line If you prefer a headless or reproducible crawl, the standalone CLI still exists. The current README documents building it with `npx tsc -b apps/cli`, after installing the repository dependencies. This is a documented current workflow; the crawl commands below were run on my older commit, not re-benchmarked on v1.1.1. ```bash git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git cd FreeCrawl-SEO-Tool npm install npx tsc -b apps/cli mkdir -p output/example node apps/cli/dist/index.js \ https://example.com/ \ --depth 20 \ --max 100000 \ --concurrency 4 \ --rps 2 \ --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \ --db output/example/crawl.seoproject \ --out output/example/pages.json \ --json ``` Replace `example.com` in the start URL and exclusion expression. Check the robots policy and use a request rate the site owner allows. The `--max 100000` option raises the ceiling; it does not force the crawler to discover 100,000 pages. Link-following only finds reachable URLs. Compare the output against XML sitemaps, Search Console and your CMS export to identify orphaned or unpublished content. To export just successful internal HTML URLs from the SQLite project, the query is: ```bash sqlite3 -header -csv output/example/crawl.seoproject \ "SELECT url FROM urls WHERE content_kind = 'html' AND status_code = 200 AND is_external = 0;" > pages.csv ``` This produces a **URL inventory**, not necessarily a full offline copy of every page's HTML. Also remember that images, scripts and stylesheets count as fetched resources but are not additional pages. ### AI agents: FreeCrawl now has an MCP server One particularly useful development since my test is the project's local **Model Context Protocol (MCP)** server. According to the current documentation, it exposes crawl summaries, per-URL data, broken-link queries, exports, configuration and live start/pause/resume/stop controls. Live desktop control requires the desktop app running; the server can also work with saved crawl projects, and newer versions support separate agent sessions. For example, the documented source setup is: ```bash npm run build:mcp # Claude Code example claude mcp add freecrawl -- node /absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js ``` This does **not** mean FreeCrawl is automatically connected to your ChatGPT account. You must configure the local MCP server in a compatible client. The benefit for developers is the ability to ask an agent questions such as "which old URLs have the most broken internal links?" without manually filtering every report. ## The original CLI test (August 2026) Everything below is the firsthand test and troubleshooting record. Where I say "the tested version," I mean commit `b91a2b7eae32f0688b11a62a0ce954beea0813df`, not today's release. ## What I ran it on The workflow below was verified on a MacBook Pro, macOS 15.7.1, Apple Silicon, 10 cores and 16 GB of memory, with Node.js 24.12.0, npm 11.6.2 and Git 2.52.0. FreeCrawl ran CLI-only against commit `b91a2b7eae32f0688b11a62a0ce954beea0813df`. The Electron desktop application was never opened, Docker was never touched, and JavaScript rendering used Playwright Chromium in true headless mode. That distinction matters for the Ubuntu section near the end. The macOS path is tested. The Linux path is reasoned from the project structure, not verified. ## Installing from source This is the installation method I used for the original CLI test. There are now [prebuilt macOS downloads](https://github.com/kemalai/FreeCrawl-SEO-Tool/releases), so use those if you want the desktop interface. Build from source when you need the CLI or MCP server, want to inspect the code, or need to reproduce this older test. ```bash node --version npm --version git --version ``` Node.js 24 worked in my original test. The October 2026 changelog now specifies **Node.js 22.13 or newer for the CLI**, so check the current project's requirements before following these older steps. ```bash git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git cd FreeCrawl-SEO-Tool npm install ``` Keep the crawler outside the repository you are auditing. It pulls in its own build output and browser binaries, and there is no reason for Chrome for Testing to end up in a client project. In my original installation, the post-install script downloaded Chrome for Testing, Chrome Headless Shell and FFmpeg, and the installed footprint came to roughly 1.9 GB before a single URL was crawled: about 1.4 GB for the repository, build output and dependencies, and another 534 MB for the Playwright browser cache. That is an observation from my older source build, not a current v1.1.1 disk-space measurement. That same install reported 18 npm vulnerabilities, one critical. I did not investigate whether they were exploitable in my setup, and this count should **not** be interpreted as the current audit result. Run a fresh dependency audit if you deploy a new source build, especially on a shared server. ```bash npm audit ``` Do not reflexively run `npm audit fix` here. Review the proposed dependency changes first. ## The CLI build-order problem in my original test On my tested commit, the obvious command below failed from a clean clone because its dependent workspace packages had not been built. The current README instead documents `npx tsc -b apps/cli` to build the CLI and dependencies. Treat the failure here as an historical troubleshooting note, not a claim that v1.1.1 has the same bug. ```bash npm run build:cli ``` From a clean clone, that produced module resolution errors: ```text Cannot find module '@freecrawl/shared-types' Cannot find module '@freecrawl/core' Cannot find module '@freecrawl/db' ``` FreeCrawl is a TypeScript monorepo, and the CLI package depends on shared packages that have not been compiled yet. Running the TypeScript project build first resolves every one of those errors. ```bash npx tsc -b npm run build:cli node apps/cli/dist/index.js --help ``` For someone installing **the current** source version just for headless crawling, follow the README's shorter CLI build sequence: ```bash git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git cd FreeCrawl-SEO-Tool npm install npx tsc -b apps/cli node apps/cli/dist/index.js --help git rev-parse HEAD ``` I still recommend recording the Git commit. When you upgrade a crawler, it is useful to know whether a change in output came from the website, your configuration or the crawler itself. The older `npx tsc -b` plus `npm run build:cli` sequence above is retained solely to explain my original fix. ## The external-link behavior that cost me a crawl My first production crawl on the **August test commit** went in the bin because FreeCrawl started making real requests to social-sharing and citation domains in page content. In that tested version, the `--external` flag did not prevent the initial status-check request to discovered external URLs; it controlled whether FreeCrawl then followed them recursively. I verified that behavior against the source at the time. **I have not retested this exact behavior in v1.1.1**, whose changelog includes changes to external-domain checks. The defensive solution for a tightly scoped audit is still an explicit exclusion pattern and a small verification crawl: ```bash --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" ``` That negative lookahead permits `https://example.com/` and `https://www.example.com/` and excludes other hosts. I recommend keeping it for tightly scoped client audits and checking the requested hosts in a 20-URL test before a full crawl. Do not assume flags or saved settings behave identically across versions. ## Run twenty URLs before you run twenty thousand A ten to twenty URL test crawl would have caught the external-domain problem in under a minute, which is the main reason this section exists. ```bash mkdir -p output/example ``` ```bash node apps/cli/dist/index.js \ https://example.com/ \ --depth 2 \ --max 20 \ --concurrency 2 \ --rps 1 \ --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \ --db output/example/test.seoproject ``` Replace both instances of `example.com` with the domain you are authorized to crawl. Then open the resulting database and check which hosts were actually requested, whether images and scripts are being stored, whether query parameters are generating duplicate URLs, and whether the site is returning throttling responses. Only move on once the scope is provably correct. While you are here, read the robots file properly rather than assuming. ```bash curl -fsSL https://example.com/robots.txt ``` The production site I crawled declared a very high crawl delay. The crawl was authorized by the site owner, so we agreed on a conservative two requests per second instead, and the corrected run returned no 429, 502 or 503 responses at all. Without that authorization, respect what is published. Sitemaps are worth locating at the same time, because a link-following crawl and a CMS-generated sitemap answer two different questions. The crawl finds pages reachable through links. The sitemap lists URLs the site claims should exist. The gap between them is where orphan pages, stale URLs and navigation holes live, which is exactly what a migration inventory needs. ```bash curl -I https://example.com/sitemap.xml curl -I https://example.com/sitemap_index.xml curl -I https://example.com/wp-sitemap.xml ``` A `200`, `301` or `302` response tells you a URL responds, not that it contains valid XML. Download it and look at it. ## The primary raw HTML crawl With scope confirmed, the real run is the same command with the limits opened up and exports requested. ```bash node apps/cli/dist/index.js \ https://example.com/ \ --depth 20 \ --max 100000 \ --concurrency 4 \ --rps 2 \ --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \ --db output/example/example-text.seoproject \ --out output/example/example-text.json \ --json ``` The depth of 20 lets the crawler follow deeply nested internal paths, and the 100,000 ceiling replaces the artificial 500-URL limit that sent me here in the first place. Concurrency and requests per second are the two politeness controls, and raising concurrency does nothing useful when the rate limit or the origin server is the real bottleneck, so start low. The `--db` flag writes the SQLite project, and `--out` plus `--json` request an export and a machine-readable summary once the crawl finishes naturally. That last condition matters more than it sounds. I will come back to it. ## A URL count is not a page count The crawl reached 5,554 internal URLs. Only 1,625 of them were pages. | Content type | Count | | ---------------- | ----: | | HTML pages | 1,625 | | Images | 3,427 | | JavaScript files | 233 | | CSS files | 223 | | Fonts | 32 | | PDFs | 2 | | Other resources | 12 | | Total | 5,554 | FreeCrawl records linked resources in the same table as documents, so a headline number like "5,554 URLs crawled" will quietly inflate your migration scope by a factor of three if nobody separates them. For a content migration, treat HTML and PDFs as documents and everything else as resources. The resource count still matters for media analysis, but it is not the page inventory. ## Querying the crawl database The `.seoproject` file is an ordinary SQLite database, and this turned out to be the single best thing about the tool. Instead of being limited to whatever reports someone decided to build, every question I had became a query. ```bash brew install sqlite sqlite3 output/example/example-text.seoproject ``` Start with the same breakdown as above, straight from the database. ```sql SELECT content_kind, COUNT(*) AS total FROM urls WHERE is_external = 0 GROUP BY content_kind ORDER BY total DESC; ``` Filtering on `is_external = 0` is your check that the exclusion rule held. If foreign hostnames appear anywhere in this table, the crawl scope leaked and the run is not trustworthy. From there, the audit questions are all variations on filtering `content_kind = 'html'`. ```sql SELECT status_code, COUNT(*) AS total FROM urls WHERE content_kind = 'html' GROUP BY status_code ORDER BY status_code; ``` ```sql SELECT url FROM urls WHERE content_kind = 'html' AND status_code = 200 AND (title IS NULL OR TRIM(title) = ''); ``` ```sql SELECT title, COUNT(*) AS occurrences FROM urls WHERE content_kind = 'html' AND status_code = 200 AND title IS NOT NULL AND TRIM(title) <> '' GROUP BY title HAVING COUNT(*) > 1 ORDER BY occurrences DESC; ``` Missing titles come from a null-or-empty check restricted to successful HTML responses, so redirects and errors do not pollute the result. Duplicate titles come from grouping the same filtered set and keeping only the groups with more than one member. Neither needs an export step, and both can be rerun against the database a month later when someone asks whether the problem was fixed. The database is also readable while the crawler is still writing to it, which is how I worked out when to stop. ```sql SELECT COUNT(*) FROM crawl_queue; ``` Watching the HTML count sit still at 1,625 while thousands of image requests remained queued told me the page discovery was finished and the remaining work was resources I did not need. ## Read the links table, not the convenience columns FreeCrawl stores the discovered link graph in a `links` table, which held 92,071 internal relationships in my crawl. The `urls` table also has an `inlinks` column, and mine was zero across the board, because that column is populated by an end-of-crawl aggregation step that never ran. If you interrupt a crawl, calculate from the raw graph instead. ```sql SELECT to_url, COUNT(*) AS inlinks FROM links WHERE is_internal = 1 GROUP BY to_url ORDER BY inlinks DESC; ``` The more useful version joins that graph against response status, which gives you every broken page ranked by how many internal links still point at it. That is your redirect priority list for a migration. ```sql SELECT l.to_url, COUNT(*) AS internal_links FROM links AS l JOIN urls AS u ON u.url = l.to_url WHERE l.is_internal = 1 AND u.status_code >= 400 GROUP BY l.to_url ORDER BY internal_links DESC; ``` A static export cannot answer that question without a script. A relational link graph answers it in six lines. ## Stopping early changes what you get I stopped the primary crawl deliberately, once HTML discovery had plateaued and the queue was mostly images. ```bash kill -INT ``` The process exited with code 130, which correctly signals an interrupt rather than a crash, and the SQLite database remained fully usable. Two things did not happen: the JSON export I requested with `--out` was never written, and the aggregate fields like `inlinks` were never populated. The crawl was still successful. The pages, metadata and link graph were all there. The missing export was a consequence of how I ended the process, not evidence of failure. Treat the SQLite database as the primary artifact and the JSON as a bonus you only trust when the process exits on its own. Exit codes need the same skepticism. During one JavaScript sample run, FreeCrawl returned exit code `1` simply because the crawl had encountered HTTP error responses, and the database and summary were both complete. If you are wrapping this in automation, check whether the database exists, how many rows it holds and what the status summary says, rather than treating any non-zero exit as an unusable crawl. ## Test JavaScript rendering on a sample A browser-rendered crawl is far more expensive than a raw HTML one, so the question is not whether FreeCrawl can render JavaScript but whether your site needs it. Put a representative set of URLs in a plain text file at `output/example/js-sample-urls.txt`: ```text https://example.com/ https://example.com/products/example-product/ https://example.com/blog/example-article/ https://example.com/recipes/example-recipe/ https://example.com/category/example/ ``` Then a config file at `output/example/js-config.json`: ```json { "renderingMode": "js", "maxDepth": 0, "maxUrls": 250, "maxConcurrency": 2, "maxRps": 1, "respectRobotsTxt": true, "crawlExternal": false, "jsRender": { "headless": true } } ``` The `maxDepth` of 0 is the important line. It tells FreeCrawl to render exactly the URLs in your list and not follow links out of them, which keeps an expensive rendering pass from turning into a second full crawl. ```bash node apps/cli/dist/index.js \ --list output/example/js-sample-urls.txt \ --config output/example/js-config.json \ --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \ --db output/example/js-sample.seoproject \ --out output/example/js-sample.json \ --json ``` I compared 39 URLs this way, covering the homepage, navigation pages, products, blog posts, recipes, taxonomy pages and anything likely to contain interactive elements. Chromium launched in true headless mode without any display server, and across all 39 pages the rendered crawl found no meaningful difference in titles, meta descriptions, H1s, canonicals, hreflang counts, structured data types, link counts or word counts. That is not a rendering failure. It is evidence that this particular WordPress site already served every migration-relevant field in its initial HTML, which meant a full JavaScript crawl would have cost hours and told me nothing new. Crawl raw HTML first, render a sample, compare, and only escalate if the sample shows you something missing. ## What the crawl actually produced For the record, here is what came out of the corrected production run. | Result | Count | | ------------------- | ----: | | HTML pages | 1,625 | | 200 responses | 1,216 | | 301 redirects | 299 | | 404 responses | 110 | | 5xx responses | 0 | | Indexable pages | 1,088 | | Non-indexable pages | 537 | Alongside that, it flagged 474 missing titles, 399 duplicate titles, 699 missing meta descriptions, 588 missing H1s, 691 pages with multiple H1s, 1,113 sitemap-only URLs and 607 crawled URLs absent from the sitemaps. Those numbers prove FreeCrawl can build a real inventory. They do not prove that every line is a defect. Archive pages, redirects, utility routes and deliberately non-indexable content can legitimately lack the metadata you would demand of a landing page. A crawler produces candidates, and the audit still needs a human deciding which ones matter. The other boundary is worth stating plainly, because it caught a colleague out on a previous project. A public crawl tells you which URLs are reachable, which are declared, how pages link to each other and what metadata renders. It cannot tell you which WordPress post type owns a URL, which custom fields produced the page, which drafts and private records exist, which page-builder components were used or which translations are incomplete. For a migration you need the crawl and a database or API export from the CMS. One is the external view, the other is the source of truth. ## Running it on a server Ubuntu Server was not part of this implementation, so treat the following as a path to test rather than a confirmed reproduction. ```bash sudo apt-get update sudo apt-get install -y curl git ca-certificates curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash - sudo apt-get install -y nodejs git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git cd FreeCrawl-SEO-Tool npm install sudo npx playwright install-deps chromium npx tsc -b npm run build:cli node apps/cli/dist/index.js --help ``` The one addition over the macOS sequence is `playwright install-deps chromium`, which pulls the system libraries Chromium needs on Linux. Because Playwright Chromium supports genuinely headless execution, this should not require GNOME, VNC or any visible browser session. On a shared machine I would also run it under a dedicated non-root user, keep it well away from application repositories, restrict outbound network access and preserve crawl databases somewhere that is not a temporary directory. ## What I would tell you before you start **For a first crawl in October 2026, start with the packaged desktop app.** You no longer need to be comfortable with the terminal to get Screaming Frog-style URL tables and SEO issue reports. The source-built CLI remains attractive when you want deterministic crawls, custom SQL, scheduled jobs or CI integration. The MCP server is a promising third path when you want agents to query and operate the crawler. The original CLI test still offers three practical lessons. Test your crawl scope on 10 to 20 URLs before committing to thousands. Verify whether external domains are being requested rather than trusting a flag name. And if you interrupt a run early, check the SQLite database directly because the final export and aggregation steps may not have completed. Those are results from my older test, not a statement that all three limitations remain in v1.1.1. For the WordPress-to-Payload migration I was tackling, the valuable result was not simply escaping a 500-URL limit. It was getting a queryable list of pages, redirects, errors and inbound link relationships that I could compare with sitemaps and the WordPress database. If I were starting that task again now, I would keep the same data-driven approach while evaluating the newer desktop and MCP interfaces as well. Let me know in the comments if you have questions, and subscribe for more practical development guides. Thanks, Matija ## LLM Response Snippet ```json { "goal": "Screaming Frog alternative FreeCrawl: install from source, render JS with Playwright, and query the SQLite crawl DB to inventory 1,625 WordPress pages.", "responses": [ { "question": "What does the article \"Free Screaming Frog Alternative: Crawl 1,625 Pages Free\" cover?", "answer": "Screaming Frog alternative FreeCrawl: install from source, render JS with Playwright, and query the SQLite crawl DB to inventory 1,625 WordPress pages." } ] } ```