📦 stormcrawler/ apache

Quickstart

NOTE: These instructions assume that you have Apache Maven installed. You will need to install Apache Storm 3.0.0 to run the crawler.

StormCrawler requires Java 25 or above. To execute tests, it requires you to have a locally installed and working Docker environment.

Once Storm is installed, the easiest way to get started is to generate a new StormCrawler project following the instructions below:

mvn archetype:generate -DarchetypeGroupId=org.apache.stormcrawler -DarchetypeArtifactId=stormcrawler-archetype -DarchetypeVersion=<CURRENT_VERSION>

Be sure to replace <CURRENT_VERSION> with the latest released version, which you can find on search.maven.org.

You'll be asked to enter a groupId (e.g. com.mycompany.crawler), an artefactId (e.g. stormcrawler), a version, a package name and details about the user agent to use.

This will not only create a fully formed project containing a POM with the dependency above but also the default resource files, a default CrawlTopology class and a configuration file. Enter the directory you just created (should be the same as the artefactId you specified earlier) and follow the instructions on the README file.

Alternatively if you can't or don't want to use the Maven archetype above, you can simply copy the files from archetype-resources.

…