Scraping

Scraping Limitations / Road Blockers:

• How many requests can their server take
  • How long does it take for the server to handle a request
  • Request/second
• How many requests can you parallelise:
  • From a single process
  • From multiple process
• How do you track what needs to be scraped
• Authentication
  • Watch out for
    • Password change reminders
    • Account being locked out
• If the page uses JavaScript

Goals:

• Maximize number of requests/sec
• Less compute resources used

Tools:

• Database
• Queue

Database:

• Avoid nosql
• Use a SQL database from the start, since you’ll most likely be exporting/querying it
  • Easier to change field names
  • Run SQL queries to fix
• One table per “type” of page
  • One table for the pagination results
  • Another table for page results
• Another table for consolidated results
  • This can be the source of truth
  • Hard part may be figuring out what should exist in the consolidated table, but doesn’t

Log to the console:

• page/id scraped
• time scraped
• Time it took to scrape page