Free Screaming Frog Alternative: Crawl 1,625 Pages Free

I needed a full URL inventory for a WordPress-to-Payload migration, and Screaming Frog's free version stops at 500 URLs. The site had more pages than that in its blog archive alone, so the choice was a licence or a different tool.

I went with FreeCrawl, an open-source crawler. In my original Apple Silicon CLI test, it recovered 1,625 HTML pages, 92,071 internal link relationships and a 441 MB SQLite database. Getting there took some debugging. This article preserves the commands, failures and SQL queries I actually tested, rather than presenting them as a fresh benchmark of the current release.

Updated October 26, 2026: FreeCrawl has moved on significantly since that test. The current GitHub release is v1.1.1 (October 25, 2026). It now offers downloadable Mac applications, a much more complete desktop SEO interface, an improved CLI and an MCP server for AI-assisted crawling. You no longer need to build from source or use a terminal just to crawl a website. The walkthrough further down remains a record of my earlier CLI run.

FreeCrawl in 2026: desktop app, CLI or MCP?

Choose the entry point based on what you are trying to accomplish.

What you need	Best option	What it gives you
Inspect a site like Screaming Frog	Desktop app	Crawl a domain and explore SEO issues through tables and reports
Inventory thousands of URLs for a migration	CLI	Repeatable crawl parameters, exports and a queryable SQLite project
Ask an AI agent to crawl and investigate issues	MCP server	Tools to start crawls, inspect URL details, query issues and export results

The current project advertises 181 SEO checks, JavaScript rendering, a desktop link visualiser, Google Search Console and GA4 integrations, scheduled crawls, multiple export formats and an MCP server with 100+ tools. Those are maintainer-documented capabilities, not features I have personally retested. The project's stated target of 1M+ URLs is likewise not a benchmark I have validated. Sources: repository and installation guide, changelog and releases.

Easiest setup on an Apple Silicon Mac

Open the official GitHub Releases page.
Download the macOS Apple Silicon .dmg for your M-series Mac (or the Intel build for an older Intel Mac).
Open the application, create or open a crawl project, enter the starting website URL and configure the scope, crawl rate and limits.
Start crawling and use the URL and issue tables or export the results.

The installer includes Chromium for JavaScript rendering. The releases are not yet notarized for macOS; read the project's own Gatekeeper instructions before bypassing any warning, and obtain the app only from the official release page. The packaged GUI is the most sensible starting point if you do not need automation.

Fetch all discoverable pages from the command line

If you prefer a headless or reproducible crawl, the standalone CLI still exists. The current README documents building it with npx tsc -b apps/cli, after installing the repository dependencies. This is a documented current workflow; the crawl commands below were run on my older commit, not re-benchmarked on v1.1.1.

bash

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npx tsc -b apps/cli

mkdir -p output/example

node apps/cli/dist/index.js \
  https://example.com/ \
  --depth 20 \
  --max 100000 \
  --concurrency 4 \
  --rps 2 \
  --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
  --db output/example/crawl.seoproject \
  --out output/example/pages.json \
  --json

Replace example.com in the start URL and exclusion expression. Check the robots policy and use a request rate the site owner allows. The --max 100000 option raises the ceiling; it does not force the crawler to discover 100,000 pages. Link-following only finds reachable URLs. Compare the output against XML sitemaps, Search Console and your CMS export to identify orphaned or unpublished content.

To export just successful internal HTML URLs from the SQLite project, the query is:

bash

sqlite3 -header -csv output/example/crawl.seoproject \
  "SELECT url FROM urls
   WHERE content_kind = 'html'
     AND status_code = 200
     AND is_external = 0;" > pages.csv

This produces a URL inventory, not necessarily a full offline copy of every page's HTML. Also remember that images, scripts and stylesheets count as fetched resources but are not additional pages.

AI agents: FreeCrawl now has an MCP server

One particularly useful development since my test is the project's local Model Context Protocol (MCP) server. According to the current documentation, it exposes crawl summaries, per-URL data, broken-link queries, exports, configuration and live start/pause/resume/stop controls. Live desktop control requires the desktop app running; the server can also work with saved crawl projects, and newer versions support separate agent sessions.

For example, the documented source setup is:

bash

npm run build:mcp

# Claude Code example
claude mcp add freecrawl -- node /absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js

This does not mean FreeCrawl is automatically connected to your ChatGPT account. You must configure the local MCP server in a compatible client. The benefit for developers is the ability to ask an agent questions such as "which old URLs have the most broken internal links?" without manually filtering every report.

The original CLI test (August 2026)

Everything below is the firsthand test and troubleshooting record. Where I say "the tested version," I mean commit b91a2b7eae32f0688b11a62a0ce954beea0813df, not today's release.

What I ran it on

The workflow below was verified on a MacBook Pro, macOS 15.7.1, Apple Silicon, 10 cores and 16 GB of memory, with Node.js 24.12.0, npm 11.6.2 and Git 2.52.0. FreeCrawl ran CLI-only against commit b91a2b7eae32f0688b11a62a0ce954beea0813df. The Electron desktop application was never opened, Docker was never touched, and JavaScript rendering used Playwright Chromium in true headless mode.

That distinction matters for the Ubuntu section near the end. The macOS path is tested. The Linux path is reasoned from the project structure, not verified.

Installing from source

This is the installation method I used for the original CLI test. There are now prebuilt macOS downloads, so use those if you want the desktop interface. Build from source when you need the CLI or MCP server, want to inspect the code, or need to reproduce this older test.

bash

node --version
npm --version
git --version

Node.js 24 worked in my original test. The October 2026 changelog now specifies Node.js 22.13 or newer for the CLI, so check the current project's requirements before following these older steps.

bash

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install

Keep the crawler outside the repository you are auditing. It pulls in its own build output and browser binaries, and there is no reason for Chrome for Testing to end up in a client project.

In my original installation, the post-install script downloaded Chrome for Testing, Chrome Headless Shell and FFmpeg, and the installed footprint came to roughly 1.9 GB before a single URL was crawled: about 1.4 GB for the repository, build output and dependencies, and another 534 MB for the Playwright browser cache. That is an observation from my older source build, not a current v1.1.1 disk-space measurement.

That same install reported 18 npm vulnerabilities, one critical. I did not investigate whether they were exploitable in my setup, and this count should not be interpreted as the current audit result. Run a fresh dependency audit if you deploy a new source build, especially on a shared server.

bash

npm audit

Do not reflexively run npm audit fix here. Review the proposed dependency changes first.

The CLI build-order problem in my original test

On my tested commit, the obvious command below failed from a clean clone because its dependent workspace packages had not been built. The current README instead documents npx tsc -b apps/cli to build the CLI and dependencies. Treat the failure here as an historical troubleshooting note, not a claim that v1.1.1 has the same bug.

bash

npm run build:cli

From a clean clone, that produced module resolution errors:

text

Cannot find module '@freecrawl/shared-types'
Cannot find module '@freecrawl/core'
Cannot find module '@freecrawl/db'

FreeCrawl is a TypeScript monorepo, and the CLI package depends on shared packages that have not been compiled yet. Running the TypeScript project build first resolves every one of those errors.

bash

npx tsc -b
npm run build:cli
node apps/cli/dist/index.js --help

For someone installing the current source version just for headless crawling, follow the README's shorter CLI build sequence:

bash

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npx tsc -b apps/cli
node apps/cli/dist/index.js --help
git rev-parse HEAD

I still recommend recording the Git commit. When you upgrade a crawler, it is useful to know whether a change in output came from the website, your configuration or the crawler itself. The older npx tsc -b plus npm run build:cli sequence above is retained solely to explain my original fix.

The external-link behavior that cost me a crawl

My first production crawl on the August test commit went in the bin because FreeCrawl started making real requests to social-sharing and citation domains in page content.

In that tested version, the --external flag did not prevent the initial status-check request to discovered external URLs; it controlled whether FreeCrawl then followed them recursively. I verified that behavior against the source at the time. I have not retested this exact behavior in v1.1.1, whose changelog includes changes to external-domain checks. The defensive solution for a tightly scoped audit is still an explicit exclusion pattern and a small verification crawl:

bash

--exclude "^https?://(?!(www\.)?example\.com([:/]|$))"

That negative lookahead permits https://example.com/ and https://www.example.com/ and excludes other hosts. I recommend keeping it for tightly scoped client audits and checking the requested hosts in a 20-URL test before a full crawl. Do not assume flags or saved settings behave identically across versions.

Run twenty URLs before you run twenty thousand

A ten to twenty URL test crawl would have caught the external-domain problem in under a minute, which is the main reason this section exists.

bash

mkdir -p output/example

bash

node apps/cli/dist/index.js \
  https://example.com/ \
  --depth 2 \
  --max 20 \
  --concurrency 2 \
  --rps 1 \
  --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
  --db output/example/test.seoproject

Replace both instances of example.com with the domain you are authorized to crawl. Then open the resulting database and check which hosts were actually requested, whether images and scripts are being stored, whether query parameters are generating duplicate URLs, and whether the site is returning throttling responses. Only move on once the scope is provably correct.

While you are here, read the robots file properly rather than assuming.

bash

curl -fsSL https://example.com/robots.txt

The production site I crawled declared a very high crawl delay. The crawl was authorized by the site owner, so we agreed on a conservative two requests per second instead, and the corrected run returned no 429, 502 or 503 responses at all. Without that authorization, respect what is published.

Sitemaps are worth locating at the same time, because a link-following crawl and a CMS-generated sitemap answer two different questions. The crawl finds pages reachable through links. The sitemap lists URLs the site claims should exist. The gap between them is where orphan pages, stale URLs and navigation holes live, which is exactly what a migration inventory needs.

bash

curl -I https://example.com/sitemap.xml
curl -I https://example.com/sitemap_index.xml
curl -I https://example.com/wp-sitemap.xml

A 200, 301 or 302 response tells you a URL responds, not that it contains valid XML. Download it and look at it.

The primary raw HTML crawl

With scope confirmed, the real run is the same command with the limits opened up and exports requested.

bash

node apps/cli/dist/index.js \
  https://example.com/ \
  --depth 20 \
  --max 100000 \
  --concurrency 4 \
  --rps 2 \
  --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
  --db output/example/example-text.seoproject \
  --out output/example/example-text.json \
  --json

The depth of 20 lets the crawler follow deeply nested internal paths, and the 100,000 ceiling replaces the artificial 500-URL limit that sent me here in the first place. Concurrency and requests per second are the two politeness controls, and raising concurrency does nothing useful when the rate limit or the origin server is the real bottleneck, so start low. The --db flag writes the SQLite project, and --out plus --json request an export and a machine-readable summary once the crawl finishes naturally.

That last condition matters more than it sounds. I will come back to it.

A URL count is not a page count

The crawl reached 5,554 internal URLs. Only 1,625 of them were pages.

Content type	Count
HTML pages	1,625
Images	3,427
JavaScript files	233
CSS files	223
Fonts	32
PDFs	2
Other resources	12
Total	5,554

FreeCrawl records linked resources in the same table as documents, so a headline number like "5,554 URLs crawled" will quietly inflate your migration scope by a factor of three if nobody separates them. For a content migration, treat HTML and PDFs as documents and everything else as resources. The resource count still matters for media analysis, but it is not the page inventory.

Querying the crawl database

The .seoproject file is an ordinary SQLite database, and this turned out to be the single best thing about the tool. Instead of being limited to whatever reports someone decided to build, every question I had became a query.

bash

brew install sqlite
sqlite3 output/example/example-text.seoproject

Start with the same breakdown as above, straight from the database.

sql

SELECT
  content_kind,
  COUNT(*) AS total
FROM urls
WHERE is_external = 0
GROUP BY content_kind
ORDER BY total DESC;

Filtering on is_external = 0 is your check that the exclusion rule held. If foreign hostnames appear anywhere in this table, the crawl scope leaked and the run is not trustworthy.

From there, the audit questions are all variations on filtering content_kind = 'html'.

sql

SELECT
  status_code,
  COUNT(*) AS total
FROM urls
WHERE content_kind = 'html'
GROUP BY status_code
ORDER BY status_code;

sql

SELECT
  url
FROM urls
WHERE content_kind = 'html'
  AND status_code = 200
  AND (title IS NULL OR TRIM(title) = '');

sql

SELECT
  title,
  COUNT(*) AS occurrences
FROM urls
WHERE content_kind = 'html'
  AND status_code = 200
  AND title IS NOT NULL
  AND TRIM(title) <> ''
GROUP BY title
HAVING COUNT(*) > 1
ORDER BY occurrences DESC;

Missing titles come from a null-or-empty check restricted to successful HTML responses, so redirects and errors do not pollute the result. Duplicate titles come from grouping the same filtered set and keeping only the groups with more than one member. Neither needs an export step, and both can be rerun against the database a month later when someone asks whether the problem was fixed.

The database is also readable while the crawler is still writing to it, which is how I worked out when to stop.

sql

SELECT COUNT(*) FROM crawl_queue;

Watching the HTML count sit still at 1,625 while thousands of image requests remained queued told me the page discovery was finished and the remaining work was resources I did not need.

Read the links table, not the convenience columns

FreeCrawl stores the discovered link graph in a links table, which held 92,071 internal relationships in my crawl. The urls table also has an inlinks column, and mine was zero across the board, because that column is populated by an end-of-crawl aggregation step that never ran.

If you interrupt a crawl, calculate from the raw graph instead.

sql

SELECT
  to_url,
  COUNT(*) AS inlinks
FROM links
WHERE is_internal = 1
GROUP BY to_url
ORDER BY inlinks DESC;

The more useful version joins that graph against response status, which gives you every broken page ranked by how many internal links still point at it. That is your redirect priority list for a migration.

sql

SELECT
  l.to_url,
  COUNT(*) AS internal_links
FROM links AS l
JOIN urls AS u
  ON u.url = l.to_url
WHERE l.is_internal = 1
  AND u.status_code >= 400
GROUP BY l.to_url
ORDER BY internal_links DESC;

A static export cannot answer that question without a script. A relational link graph answers it in six lines.

Stopping early changes what you get

I stopped the primary crawl deliberately, once HTML discovery had plateaued and the queue was mostly images.

bash

kill -INT <PROCESS_ID>

The process exited with code 130, which correctly signals an interrupt rather than a crash, and the SQLite database remained fully usable. Two things did not happen: the JSON export I requested with --out was never written, and the aggregate fields like inlinks were never populated.

The crawl was still successful. The pages, metadata and link graph were all there. The missing export was a consequence of how I ended the process, not evidence of failure.

Treat the SQLite database as the primary artifact and the JSON as a bonus you only trust when the process exits on its own.

Exit codes need the same skepticism. During one JavaScript sample run, FreeCrawl returned exit code 1 simply because the crawl had encountered HTTP error responses, and the database and summary were both complete. If you are wrapping this in automation, check whether the database exists, how many rows it holds and what the status summary says, rather than treating any non-zero exit as an unusable crawl.

Test JavaScript rendering on a sample

A browser-rendered crawl is far more expensive than a raw HTML one, so the question is not whether FreeCrawl can render JavaScript but whether your site needs it.

Put a representative set of URLs in a plain text file at output/example/js-sample-urls.txt:

text

https://example.com/
https://example.com/products/example-product/
https://example.com/blog/example-article/
https://example.com/recipes/example-recipe/
https://example.com/category/example/

Then a config file at output/example/js-config.json:

json

{
  "renderingMode": "js",
  "maxDepth": 0,
  "maxUrls": 250,
  "maxConcurrency": 2,
  "maxRps": 1,
  "respectRobotsTxt": true,
  "crawlExternal": false,
  "jsRender": {
    "headless": true
  }
}

The maxDepth of 0 is the important line. It tells FreeCrawl to render exactly the URLs in your list and not follow links out of them, which keeps an expensive rendering pass from turning into a second full crawl.

bash

node apps/cli/dist/index.js \
  --list output/example/js-sample-urls.txt \
  --config output/example/js-config.json \
  --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
  --db output/example/js-sample.seoproject \
  --out output/example/js-sample.json \
  --json

I compared 39 URLs this way, covering the homepage, navigation pages, products, blog posts, recipes, taxonomy pages and anything likely to contain interactive elements. Chromium launched in true headless mode without any display server, and across all 39 pages the rendered crawl found no meaningful difference in titles, meta descriptions, H1s, canonicals, hreflang counts, structured data types, link counts or word counts.

That is not a rendering failure. It is evidence that this particular WordPress site already served every migration-relevant field in its initial HTML, which meant a full JavaScript crawl would have cost hours and told me nothing new. Crawl raw HTML first, render a sample, compare, and only escalate if the sample shows you something missing.

What the crawl actually produced

For the record, here is what came out of the corrected production run.

Result	Count
HTML pages	1,625
200 responses	1,216
301 redirects	299
404 responses	110
5xx responses	0
Indexable pages	1,088
Non-indexable pages	537

Alongside that, it flagged 474 missing titles, 399 duplicate titles, 699 missing meta descriptions, 588 missing H1s, 691 pages with multiple H1s, 1,113 sitemap-only URLs and 607 crawled URLs absent from the sitemaps.

Those numbers prove FreeCrawl can build a real inventory. They do not prove that every line is a defect. Archive pages, redirects, utility routes and deliberately non-indexable content can legitimately lack the metadata you would demand of a landing page. A crawler produces candidates, and the audit still needs a human deciding which ones matter.

The other boundary is worth stating plainly, because it caught a colleague out on a previous project. A public crawl tells you which URLs are reachable, which are declared, how pages link to each other and what metadata renders. It cannot tell you which WordPress post type owns a URL, which custom fields produced the page, which drafts and private records exist, which page-builder components were used or which translations are incomplete. For a migration you need the crawl and a database or API export from the CMS. One is the external view, the other is the source of truth.

Running it on a server

Ubuntu Server was not part of this implementation, so treat the following as a path to test rather than a confirmed reproduction.

bash

sudo apt-get update
sudo apt-get install -y curl git ca-certificates

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt-get install -y nodejs

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool

npm install
sudo npx playwright install-deps chromium

npx tsc -b
npm run build:cli

node apps/cli/dist/index.js --help

The one addition over the macOS sequence is playwright install-deps chromium, which pulls the system libraries Chromium needs on Linux. Because Playwright Chromium supports genuinely headless execution, this should not require GNOME, VNC or any visible browser session. On a shared machine I would also run it under a dedicated non-root user, keep it well away from application repositories, restrict outbound network access and preserve crawl databases somewhere that is not a temporary directory.

What I would tell you before you start

For a first crawl in October 2026, start with the packaged desktop app. You no longer need to be comfortable with the terminal to get Screaming Frog-style URL tables and SEO issue reports. The source-built CLI remains attractive when you want deterministic crawls, custom SQL, scheduled jobs or CI integration. The MCP server is a promising third path when you want agents to query and operate the crawler.

The original CLI test still offers three practical lessons. Test your crawl scope on 10 to 20 URLs before committing to thousands. Verify whether external domains are being requested rather than trusting a flag name. And if you interrupt a run early, check the SQLite database directly because the final export and aggregation steps may not have completed. Those are results from my older test, not a statement that all three limitations remain in v1.1.1.

For the WordPress-to-Payload migration I was tackling, the valuable result was not simply escaping a 500-URL limit. It was getting a queryable list of pages, redirects, errors and inbound link relationships that I could compare with sitemaps and the WordPress database. If I were starting that task again now, I would keep the same data-driven approach while evaluating the newer desktop and MCP interfaces as well.

Let me know in the comments if you have questions, and subscribe for more practical development guides.

Thanks, Matija

FreeCrawl in 2026: desktop app, CLI or MCP?

Choose the entry point based on what you are trying to accomplish.

What you need	Best option	What it gives you
Inspect a site like Screaming Frog	Desktop app	Crawl a domain and explore SEO issues through tables and reports
Inventory thousands of URLs for a migration	CLI	Repeatable crawl parameters, exports and a queryable SQLite project
Ask an AI agent to crawl and investigate issues	MCP server	Tools to start crawls, inspect URL details, query issues and export results

Easiest setup on an Apple Silicon Mac

Open the official GitHub Releases page.
Download the macOS Apple Silicon .dmg for your M-series Mac (or the Intel build for an older Intel Mac).
Open the application, create or open a crawl project, enter the starting website URL and configure the scope, crawl rate and limits.
Start crawling and use the URL and issue tables or export the results.

Fetch all discoverable pages from the command line

bash

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npx tsc -b apps/cli

mkdir -p output/example

node apps/cli/dist/index.js \
  https://example.com/ \
  --depth 20 \
  --max 100000 \
  --concurrency 4 \
  --rps 2 \
  --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
  --db output/example/crawl.seoproject \
  --out output/example/pages.json \
  --json

To export just successful internal HTML URLs from the SQLite project, the query is:

bash

sqlite3 -header -csv output/example/crawl.seoproject \
  "SELECT url FROM urls
   WHERE content_kind = 'html'
     AND status_code = 200
     AND is_external = 0;" > pages.csv

AI agents: FreeCrawl now has an MCP server

For example, the documented source setup is:

bash

npm run build:mcp

# Claude Code example
claude mcp add freecrawl -- node /absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js

The original CLI test (August 2026)

Everything below is the firsthand test and troubleshooting record. Where I say "the tested version," I mean commit b91a2b7eae32f0688b11a62a0ce954beea0813df, not today's release.

What I ran it on

That distinction matters for the Ubuntu section near the end. The macOS path is tested. The Linux path is reasoned from the project structure, not verified.

Installing from source

bash

node --version
npm --version
git --version

Node.js 24 worked in my original test. The October 2026 changelog now specifies Node.js 22.13 or newer for the CLI, so check the current project's requirements before following these older steps.

bash

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install

Keep the crawler outside the repository you are auditing. It pulls in its own build output and browser binaries, and there is no reason for Chrome for Testing to end up in a client project.

bash

npm audit

Do not reflexively run npm audit fix here. Review the proposed dependency changes first.

The CLI build-order problem in my original test

bash

npm run build:cli

From a clean clone, that produced module resolution errors:

text

Cannot find module '@freecrawl/shared-types'
Cannot find module '@freecrawl/core'
Cannot find module '@freecrawl/db'

FreeCrawl is a TypeScript monorepo, and the CLI package depends on shared packages that have not been compiled yet. Running the TypeScript project build first resolves every one of those errors.

bash

npx tsc -b
npm run build:cli
node apps/cli/dist/index.js --help

For someone installing the current source version just for headless crawling, follow the README's shorter CLI build sequence:

bash

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npx tsc -b apps/cli
node apps/cli/dist/index.js --help
git rev-parse HEAD

The external-link behavior that cost me a crawl

My first production crawl on the August test commit went in the bin because FreeCrawl started making real requests to social-sharing and citation domains in page content.

bash

--exclude "^https?://(?!(www\.)?example\.com([:/]|$))"

Run twenty URLs before you run twenty thousand

A ten to twenty URL test crawl would have caught the external-domain problem in under a minute, which is the main reason this section exists.

bash

mkdir -p output/example

bash

node apps/cli/dist/index.js \
  https://example.com/ \
  --depth 2 \
  --max 20 \
  --concurrency 2 \
  --rps 1 \
  --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
  --db output/example/test.seoproject

While you are here, read the robots file properly rather than assuming.

bash

curl -fsSL https://example.com/robots.txt

bash

curl -I https://example.com/sitemap.xml
curl -I https://example.com/sitemap_index.xml
curl -I https://example.com/wp-sitemap.xml

A 200, 301 or 302 response tells you a URL responds, not that it contains valid XML. Download it and look at it.

The primary raw HTML crawl

With scope confirmed, the real run is the same command with the limits opened up and exports requested.

bash

node apps/cli/dist/index.js \
  https://example.com/ \
  --depth 20 \
  --max 100000 \
  --concurrency 4 \
  --rps 2 \
  --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
  --db output/example/example-text.seoproject \
  --out output/example/example-text.json \
  --json

That last condition matters more than it sounds. I will come back to it.

A URL count is not a page count

The crawl reached 5,554 internal URLs. Only 1,625 of them were pages.

Content type	Count
HTML pages	1,625
Images	3,427
JavaScript files	233
CSS files	223
Fonts	32
PDFs	2
Other resources	12
Total	5,554

Querying the crawl database

bash

brew install sqlite
sqlite3 output/example/example-text.seoproject

Start with the same breakdown as above, straight from the database.

sql

SELECT
  content_kind,
  COUNT(*) AS total
FROM urls
WHERE is_external = 0
GROUP BY content_kind
ORDER BY total DESC;

Filtering on is_external = 0 is your check that the exclusion rule held. If foreign hostnames appear anywhere in this table, the crawl scope leaked and the run is not trustworthy.

From there, the audit questions are all variations on filtering content_kind = 'html'.

sql

SELECT
  status_code,
  COUNT(*) AS total
FROM urls
WHERE content_kind = 'html'
GROUP BY status_code
ORDER BY status_code;

sql

SELECT
  url
FROM urls
WHERE content_kind = 'html'
  AND status_code = 200
  AND (title IS NULL OR TRIM(title) = '');

sql

SELECT
  title,
  COUNT(*) AS occurrences
FROM urls
WHERE content_kind = 'html'
  AND status_code = 200
  AND title IS NOT NULL
  AND TRIM(title) <> ''
GROUP BY title
HAVING COUNT(*) > 1
ORDER BY occurrences DESC;

The database is also readable while the crawler is still writing to it, which is how I worked out when to stop.

sql

SELECT COUNT(*) FROM crawl_queue;

Watching the HTML count sit still at 1,625 while thousands of image requests remained queued told me the page discovery was finished and the remaining work was resources I did not need.

Read the links table, not the convenience columns

If you interrupt a crawl, calculate from the raw graph instead.

sql

SELECT
  to_url,
  COUNT(*) AS inlinks
FROM links
WHERE is_internal = 1
GROUP BY to_url
ORDER BY inlinks DESC;

sql

SELECT
  l.to_url,
  COUNT(*) AS internal_links
FROM links AS l
JOIN urls AS u
  ON u.url = l.to_url
WHERE l.is_internal = 1
  AND u.status_code >= 400
GROUP BY l.to_url
ORDER BY internal_links DESC;

A static export cannot answer that question without a script. A relational link graph answers it in six lines.

Stopping early changes what you get

I stopped the primary crawl deliberately, once HTML discovery had plateaued and the queue was mostly images.

bash

kill -INT <PROCESS_ID>

The crawl was still successful. The pages, metadata and link graph were all there. The missing export was a consequence of how I ended the process, not evidence of failure.

Treat the SQLite database as the primary artifact and the JSON as a bonus you only trust when the process exits on its own.

Test JavaScript rendering on a sample

A browser-rendered crawl is far more expensive than a raw HTML one, so the question is not whether FreeCrawl can render JavaScript but whether your site needs it.

Put a representative set of URLs in a plain text file at output/example/js-sample-urls.txt:

text

https://example.com/
https://example.com/products/example-product/
https://example.com/blog/example-article/
https://example.com/recipes/example-recipe/
https://example.com/category/example/

Then a config file at output/example/js-config.json:

json

{
  "renderingMode": "js",
  "maxDepth": 0,
  "maxUrls": 250,
  "maxConcurrency": 2,
  "maxRps": 1,
  "respectRobotsTxt": true,
  "crawlExternal": false,
  "jsRender": {
    "headless": true
  }
}

bash

node apps/cli/dist/index.js \
  --list output/example/js-sample-urls.txt \
  --config output/example/js-config.json \
  --exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
  --db output/example/js-sample.seoproject \
  --out output/example/js-sample.json \
  --json

What the crawl actually produced

For the record, here is what came out of the corrected production run.

Result	Count
HTML pages	1,625
200 responses	1,216
301 redirects	299
404 responses	110
5xx responses	0
Indexable pages	1,088
Non-indexable pages	537

Running it on a server

Ubuntu Server was not part of this implementation, so treat the following as a path to test rather than a confirmed reproduction.

bash

sudo apt-get update
sudo apt-get install -y curl git ca-certificates

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt-get install -y nodejs

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool

npm install
sudo npx playwright install-deps chromium

npx tsc -b
npm run build:cli

node apps/cli/dist/index.js --help

What I would tell you before you start

Let me know in the comments if you have questions, and subscribe for more practical development guides.

Thanks, Matija

FreeCrawl in 2026: desktop app, CLI or MCP?

Easiest setup on an Apple Silicon Mac

Fetch all discoverable pages from the command line

AI agents: FreeCrawl now has an MCP server

The original CLI test (August 2026)

What I ran it on

Installing from source

The CLI build-order problem in my original test

The external-link behavior that cost me a crawl

Run twenty URLs before you run twenty thousand

The primary raw HTML crawl

A URL count is not a page count

Querying the crawl database

Read the links table, not the convenience columns

Stopping early changes what you get

Test JavaScript rendering on a sample

What the crawl actually produced

Running it on a server

What I would tell you before you start

📚 Get Practical Development Guides

Frequently Asked Questions

Is FreeCrawl a true replacement for Screaming Frog?

Why did external domains get status-checked even without --external?

Is the SQLite .seoproject file usable while crawling?

Can I run FreeCrawl on Ubuntu Server the same way as macOS?

Comments

Comments

FreeCrawl in 2026: desktop app, CLI or MCP?

Easiest setup on an Apple Silicon Mac

Fetch all discoverable pages from the command line

AI agents: FreeCrawl now has an MCP server

The original CLI test (August 2026)

What I ran it on

Installing from source

The CLI build-order problem in my original test

The external-link behavior that cost me a crawl

Run twenty URLs before you run twenty thousand

The primary raw HTML crawl

A URL count is not a page count

Querying the crawl database

Read the links table, not the convenience columns

Stopping early changes what you get

Test JavaScript rendering on a sample

What the crawl actually produced

Running it on a server

What I would tell you before you start

Comments