Norconex Crawlers
August 30, 2026 · View on GitHub
Norconex Web and File System Crawlers collect content from websites, file systems, cloud storage, and content management systems, process it, and commit it to the repository of your choice — a search engine, a database, a vector store, or your own pipeline.
They run from the command line with file-based configuration on any OS, or embed into Java applications through documented APIs.
Website and documentation: crawler.norconex.com
Which version do I want?
| Status | Where | |
|---|---|---|
| Version 3 | Current stable release — recommended for production | 3.x-branch · docs |
| Version 4 | Beta — a ground-up rewrite, open for early adopters | main (this branch) · docs |
Version 4 is a beta: interfaces, configuration, and packaging may still change before the final release. Version 3 configurations are not compatible as-is and need conversion — the online configurator can import a V3 configuration and convert it for you.
Feedback on the beta is very welcome — please open an issue or start a discussion.
Getting started with V4
Docker
Images are published for each release. There is no latest tag yet — the
moving tags are reserved for the first stable 4.0.0.
| Image | Purpose |
|---|---|
norconex/crawler-web | Web crawler (includes WebDriver support for JavaScript-rendered pages) |
norconex/crawler-fs | File system crawler |
norconex/crawler-web-playwright | Web crawler with Playwright and Chromium bundled, as an alternative to WebDriver |
docker run --rm \
-v "$PWD/configs:/opt/norconex/crawler/configs" \
-v "$PWD/logs:/opt/norconex/crawler/logs" \
norconex/crawler-web:4.0.0-beta.1
Images are also mirrored to ghcr.io/norconex/.
Distributions
Ready-to-run zips for the web and file system crawlers, plus each committer, are attached to every GitHub release.
Maven
All modules share the group id com.norconex.crawler and one version number.
<dependency>
<groupId>com.norconex.crawler</groupId>
<artifactId>nx-crawler-web</artifactId>
<version>4.0.0-beta.1</version>
</dependency>
Visual configurator
configurator.norconex.com builds and validates crawler configurations in your browser, and can convert an existing V3 configuration to V4.
Projects
This is a mono-repo: every Norconex crawler project that used to live in its own repository is here, and they are all released together under one version.
| Folder | Artifact Id | Quality gate |
|---|---|---|
| crawler/core/ | nx-crawler-core | |
| crawler/web/ | nx-crawler-web | |
| crawler/fs/ | nx-crawler-fs | |
| importer/ | nx-importer | |
| committer/core/ | nx-committer-core | |
| committer/apachekafka/ | nx-committer-apachekafka | |
| committer/azurecognitivesearch/ | nx-committer-azurecognitivesearch | |
| committer/elasticsearch/ | nx-committer-elasticsearch | |
| committer/googlecloudsearch/ | nx-committer-googlecloudsearch | |
| committer/idol/ | nx-committer-idol | |
| committer/neo4j/ | nx-committer-neo4j | |
| committer/solr/ | nx-committer-solr | |
| committer/sql/ | nx-committer-sql | |
| 🪦 committer/amazoncloudsearch/ | nx-committer-amazoncloudsearch | Deprecated |
Contributing
Contributions are welcome — a bug report, a fix, a new feature, or better documentation. Please read CONTRIBUTING.md first: all contributions need a sign-off and signed commits.
Sponsors
Norconex Crawler is free and Apache-2.0 licensed, built and maintained by Norconex Inc. Sponsorship funds development beyond what we already invest, and puts your name in front of the people using it.
👉 Become a sponsor — tiers and what each includes are listed there. See SPONSORS.md for our sponsors, and for the many ways to support the project that cost nothing.
Sponsorship is recognition, not a support contract. If your team needs guaranteed response times or committed engineering, see Support.
License
Apache License 2.0 — see LICENSE.