Step-by-step FreeCrawl CLI guide: install from source, render JS with Playwright, and query the SQLite crawl DB.
Join developers getting comprehensive guides, code examples, optimization tips, and time-saving prompts to accelerate their development workflow.
No spam. Unsubscribe anytime.
I needed a full URL inventory for a WordPress to Payload migration, and Screaming Frog's free version stops at 500 URLs. The site had more pages than that in its blog archive alone, so the choice was a licence or a different tool.
I went with FreeCrawl, an open-source crawler that runs from the command line, stores everything in a local SQLite database and renders JavaScript through Playwright. It worked. The crawl recovered 1,625 HTML pages, 92,071 internal link relationships and a 441 MB database.
It also failed in three places that cost me a full crawl, and none of those failures are in the documentation. This guide walks through the installation and crawl sequence that actually worked, the two commands that did not, and how to read the crawl database while the crawler is still running.
The workflow below was verified on a MacBook Pro, macOS 15.7.1, Apple Silicon, 10 cores and 16 GB of memory, with Node.js 24.12.0, npm 11.6.2 and Git 2.52.0. FreeCrawl ran CLI-only against commit b91a2b7eae32f0688b11a62a0ce954beea0813df. The Electron desktop application was never opened, Docker was never touched, and JavaScript rendering used Playwright Chromium in true headless mode.
That distinction matters for the Ubuntu section near the end. The macOS path is tested. The Linux path is reasoned from the project structure, not verified.
There is no packaged release in this workflow. You clone the repository and build it yourself, so start by confirming your toolchain is present.
node --version
npm --version
git --version
The project declares Node.js 22 or newer. Node 24 worked without complaint.
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
Keep the crawler outside the repository you are auditing. It pulls in its own build output and browser binaries, and there is no reason for Chrome for Testing to end up in a client project.
That npm install is heavier than it looks. The post-install script downloads Chrome for Testing, Chrome Headless Shell and FFmpeg, and the installed footprint came to roughly 1.9 GB before a single URL was crawled: about 1.4 GB for the repository, build output and dependencies, and another 534 MB for the Playwright browser cache.
The install also reported 18 npm vulnerabilities, one of them critical. I did not investigate them, so I cannot tell you whether they were exploitable in this context. If you are putting this on a shared or production server, read the report yourself before deciding how to isolate the tool.
npm audit
Do not reflexively run npm audit fix here. Review the proposed dependency changes first.
The obvious command is the one that fails.
npm run build:cli
From a clean clone, that produced module resolution errors:
Cannot find module '@freecrawl/shared-types'
Cannot find module '@freecrawl/core'
Cannot find module '@freecrawl/db'
FreeCrawl is a TypeScript monorepo, and the CLI package depends on shared packages that have not been compiled yet. Running the TypeScript project build first resolves every one of those errors.
npx tsc -b
npm run build:cli
node apps/cli/dist/index.js --help
So the complete installation, from nothing to a working binary, is this:
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npx tsc -b
npm run build:cli
node apps/cli/dist/index.js --help
git rev-parse HEAD
That last line is worth keeping. FreeCrawl is under active development, and recording the commit means you can tell later whether a behavior change came from the tool or from your own configuration.
My first production crawl went in the bin because FreeCrawl started making real requests to social-sharing and citation domains that appeared in the page content.
The --external flag reads as though it governs whether external URLs are requested at all. It does not. In the tested version, FreeCrawl made one real status-check request to every discovered external URL whether or not the flag was passed. What the flag controls is whether the crawler then continues recursively through those external pages. I only confirmed this by reading the crawler source after foreign hostnames turned up in the database.
The fix is an explicit exclusion pattern that allows your domain and blocks everything else.
--exclude "^https?://(?!(www\.)?example\.com([:/]|$))"
That negative lookahead permits https://example.com/ and https://www.example.com/ and rejects any other host. Add it to every single invocation. Configuration does not carry over between commands, so a crawl you run tomorrow without the flag will start hitting third-party domains again.
A ten to twenty URL test crawl would have caught the external-domain problem in under a minute, which is the main reason this section exists.
mkdir -p output/example
node apps/cli/dist/index.js \
https://example.com/ \
--depth 2 \
--max 20 \
--concurrency 2 \
--rps 1 \
--exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
--db output/example/test.seoproject
Replace both instances of example.com with the domain you are authorized to crawl. Then open the resulting database and check which hosts were actually requested, whether images and scripts are being stored, whether query parameters are generating duplicate URLs, and whether the site is returning throttling responses. Only move on once the scope is provably correct.
While you are here, read the robots file properly rather than assuming.
curl -fsSL https://example.com/robots.txt
The production site I crawled declared a very high crawl delay. The crawl was authorized by the site owner, so we agreed on a conservative two requests per second instead, and the corrected run returned no 429, 502 or 503 responses at all. Without that authorization, respect what is published.
Sitemaps are worth locating at the same time, because a link-following crawl and a CMS-generated sitemap answer two different questions. The crawl finds pages reachable through links. The sitemap lists URLs the site claims should exist. The gap between them is where orphan pages, stale URLs and navigation holes live, which is exactly what a migration inventory needs.
curl -I https://example.com/sitemap.xml
curl -I https://example.com/sitemap_index.xml
curl -I https://example.com/wp-sitemap.xml
A 200, 301 or 302 response tells you a URL responds, not that it contains valid XML. Download it and look at it.
With scope confirmed, the real run is the same command with the limits opened up and exports requested.
node apps/cli/dist/index.js \
https://example.com/ \
--depth 20 \
--max 100000 \
--concurrency 4 \
--rps 2 \
--exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
--db output/example/example-text.seoproject \
--out output/example/example-text.json \
--json
The depth of 20 lets the crawler follow deeply nested internal paths, and the 100,000 ceiling replaces the artificial 500-URL limit that sent me here in the first place. Concurrency and requests per second are the two politeness controls, and raising concurrency does nothing useful when the rate limit or the origin server is the real bottleneck, so start low. The --db flag writes the SQLite project, and --out plus --json request an export and a machine-readable summary once the crawl finishes naturally.
That last condition matters more than it sounds. I will come back to it.
The crawl reached 5,554 internal URLs. Only 1,625 of them were pages.
| Content type | Count |
|---|---|
| HTML pages | 1,625 |
| Images | 3,427 |
| JavaScript files | 233 |
| CSS files | 223 |
| Fonts | 32 |
| PDFs | 2 |
| Other resources | 12 |
| Total | 5,554 |
FreeCrawl records linked resources in the same table as documents, so a headline number like "5,554 URLs crawled" will quietly inflate your migration scope by a factor of three if nobody separates them. For a content migration, treat HTML and PDFs as documents and everything else as resources. The resource count still matters for media analysis, but it is not the page inventory.
The .seoproject file is an ordinary SQLite database, and this turned out to be the single best thing about the tool. Instead of being limited to whatever reports someone decided to build, every question I had became a query.
brew install sqlite
sqlite3 output/example/example-text.seoproject
Start with the same breakdown as above, straight from the database.
SELECT
content_kind,
COUNT(*) AS total
FROM urls
WHERE is_external = 0
GROUP BY content_kind
ORDER BY total DESC;
Filtering on is_external = 0 is your check that the exclusion rule held. If foreign hostnames appear anywhere in this table, the crawl scope leaked and the run is not trustworthy.
From there, the audit questions are all variations on filtering content_kind = 'html'.
SELECT
status_code,
COUNT(*) AS total
FROM urls
WHERE content_kind = 'html'
GROUP BY status_code
ORDER BY status_code;
SELECT
url
FROM urls
WHERE content_kind = 'html'
AND status_code = 200
AND (title IS NULL OR TRIM(title) = '');
SELECT
title,
COUNT(*) AS occurrences
FROM urls
WHERE content_kind = 'html'
AND status_code = 200
AND title IS NOT NULL
AND TRIM(title) <> ''
GROUP BY title
HAVING COUNT(*) > 1
ORDER BY occurrences DESC;
Missing titles come from a null-or-empty check restricted to successful HTML responses, so redirects and errors do not pollute the result. Duplicate titles come from grouping the same filtered set and keeping only the groups with more than one member. Neither needs an export step, and both can be rerun against the database a month later when someone asks whether the problem was fixed.
The database is also readable while the crawler is still writing to it, which is how I worked out when to stop.
SELECT COUNT(*) FROM crawl_queue;
Watching the HTML count sit still at 1,625 while thousands of image requests remained queued told me the page discovery was finished and the remaining work was resources I did not need.
FreeCrawl stores the discovered link graph in a links table, which held 92,071 internal relationships in my crawl. The urls table also has an inlinks column, and mine was zero across the board, because that column is populated by an end-of-crawl aggregation step that never ran.
If you interrupt a crawl, calculate from the raw graph instead.
SELECT
to_url,
COUNT(*) AS inlinks
FROM links
WHERE is_internal = 1
GROUP BY to_url
ORDER BY inlinks DESC;
The more useful version joins that graph against response status, which gives you every broken page ranked by how many internal links still point at it. That is your redirect priority list for a migration.
SELECT
l.to_url,
COUNT(*) AS internal_links
FROM links AS l
JOIN urls AS u
ON u.url = l.to_url
WHERE l.is_internal = 1
AND u.status_code >= 400
GROUP BY l.to_url
ORDER BY internal_links DESC;
A static export cannot answer that question without a script. A relational link graph answers it in six lines.
I stopped the primary crawl deliberately, once HTML discovery had plateaued and the queue was mostly images.
kill -INT <PROCESS_ID>
The process exited with code 130, which correctly signals an interrupt rather than a crash, and the SQLite database remained fully usable. Two things did not happen: the JSON export I requested with --out was never written, and the aggregate fields like inlinks were never populated.
The crawl was still successful. The pages, metadata and link graph were all there. The missing export was a consequence of how I ended the process, not evidence of failure.
Treat the SQLite database as the primary artifact and the JSON as a bonus you only trust when the process exits on its own.
Exit codes need the same skepticism. During one JavaScript sample run, FreeCrawl returned exit code 1 simply because the crawl had encountered HTTP error responses, and the database and summary were both complete. If you are wrapping this in automation, check whether the database exists, how many rows it holds and what the status summary says, rather than treating any non-zero exit as an unusable crawl.
A browser-rendered crawl is far more expensive than a raw HTML one, so the question is not whether FreeCrawl can render JavaScript but whether your site needs it.
Put a representative set of URLs in a plain text file at output/example/js-sample-urls.txt:
https://example.com/
https://example.com/products/example-product/
https://example.com/blog/example-article/
https://example.com/recipes/example-recipe/
https://example.com/category/example/
Then a config file at output/example/js-config.json:
{
"renderingMode": "js",
"maxDepth": 0,
"maxUrls": 250,
"maxConcurrency": 2,
"maxRps": 1,
"respectRobotsTxt": true,
"crawlExternal": false,
"jsRender": {
"headless": true
}
}
The maxDepth of 0 is the important line. It tells FreeCrawl to render exactly the URLs in your list and not follow links out of them, which keeps an expensive rendering pass from turning into a second full crawl.
node apps/cli/dist/index.js \
--list output/example/js-sample-urls.txt \
--config output/example/js-config.json \
--exclude "^https?://(?!(www\.)?example\.com([:/]|$))" \
--db output/example/js-sample.seoproject \
--out output/example/js-sample.json \
--json
I compared 39 URLs this way, covering the homepage, navigation pages, products, blog posts, recipes, taxonomy pages and anything likely to contain interactive elements. Chromium launched in true headless mode without any display server, and across all 39 pages the rendered crawl found no meaningful difference in titles, meta descriptions, H1s, canonicals, hreflang counts, structured data types, link counts or word counts.
That is not a rendering failure. It is evidence that this particular WordPress site already served every migration-relevant field in its initial HTML, which meant a full JavaScript crawl would have cost hours and told me nothing new. Crawl raw HTML first, render a sample, compare, and only escalate if the sample shows you something missing.
For the record, here is what came out of the corrected production run.
| Result | Count |
|---|---|
| HTML pages | 1,625 |
| 200 responses | 1,216 |
| 301 redirects | 299 |
| 404 responses | 110 |
| 5xx responses | 0 |
| Indexable pages | 1,088 |
| Non-indexable pages | 537 |
Alongside that, it flagged 474 missing titles, 399 duplicate titles, 699 missing meta descriptions, 588 missing H1s, 691 pages with multiple H1s, 1,113 sitemap-only URLs and 607 crawled URLs absent from the sitemaps.
Those numbers prove FreeCrawl can build a real inventory. They do not prove that every line is a defect. Archive pages, redirects, utility routes and deliberately non-indexable content can legitimately lack the metadata you would demand of a landing page. A crawler produces candidates, and the audit still needs a human deciding which ones matter.
The other boundary is worth stating plainly, because it caught a colleague out on a previous project. A public crawl tells you which URLs are reachable, which are declared, how pages link to each other and what metadata renders. It cannot tell you which WordPress post type owns a URL, which custom fields produced the page, which drafts and private records exist, which page-builder components were used or which translations are incomplete. For a migration you need the crawl and a database or API export from the CMS. One is the external view, the other is the source of truth.
Ubuntu Server was not part of this implementation, so treat the following as a path to test rather than a confirmed reproduction.
sudo apt-get update
sudo apt-get install -y curl git ca-certificates
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt-get install -y nodejs
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
sudo npx playwright install-deps chromium
npx tsc -b
npm run build:cli
node apps/cli/dist/index.js --help
The one addition over the macOS sequence is playwright install-deps chromium, which pulls the system libraries Chromium needs on Linux. Because Playwright Chromium supports genuinely headless execution, this should not require GNOME, VNC or any visible browser session. On a shared machine I would also run it under a dedicated non-root user, keep it well away from application repositories, restrict outbound network access and preserve crawl databases somewhere that is not a temporary directory.
FreeCrawl is genuinely usable as a free Screaming Frog alternative, provided you are the kind of person who is comfortable with a terminal, a regular expression, a SQL query and occasionally reading source code when the documentation runs out. The absence of a URL limit and the presence of a queryable database made it a better fit for a developer-led migration than a graphical application would have been.
Three things will bite you, and all three are avoidable now that you know about them. The CLI build fails from a clean clone unless you run npx tsc -b first. External domains get status-checked unless you exclude them explicitly on every command. And an interrupted crawl silently skips the JSON export and the aggregate columns, so the SQLite file is the artifact you should trust.
Use Screaming Frog when you want a mature, polished application and an interface that a non-technical colleague can operate. Use FreeCrawl when you want a crawler you can automate, inspect and query as part of a technical workflow. For the migration inventory I needed, the second was worth more than the first.
Let me know in the comments if you have questions, and subscribe for more practical development guides.
Thanks, Matija