CrawlyExamples

May 8, 2020 ยท View on GitHub

Running the spiders

  1. Run mix deps.get
  2. Start interactive console: iex -S mix
  3. Schedule one of the spiders:
    • Crawly.Engine.start_spider(CrawlyExamples.Spider.ClassicCars) or
    • Crawly.Engine.start_spider(CrawlyExamples.Spider.Esl) or
    • Crawly.Engine.start_spider(CrawlyExamples.Spider.Homebase) or
    • Crawly.Engine.start_spider(CrawlyExamples.Spider.WorldwideVintageAutos)
  4. Find the data in /tmp/