yad2 Apartment Scraper

Self-hostable scraper for yad2.co.il that monitors rental listings and sends Telegram alerts when new apartments matching your criteria are posted. Runs on a schedule via Docker on a home server or VPS.

What it does

• Fetches listings from the yad2 rental API on a configurable schedule
• Filters by price, room count, city, neighborhood, amenities, and listing type
• Stores listings in a local SQLite database and alerts only on new ones
• Sends a Telegram message for each new matching listing
• Sends a daily summary with stats
• Handles yad2's anti-bot protection (DataDome + ShieldSquare) via TLS fingerprint rotation

Requirements

• Python 3.12
• Docker and Docker Compose
• A Telegram bot token and your Telegram chat ID

Setup

1. Clone the repo

git clone https://github.com/guyvolvo/yad2-apartment-scraper
cd yad2

2. Create your config

cp config.example.yaml config.yaml

Open config.yaml and fill in your values. See the Configuration section below.

3. Build and run a test scan

docker compose build
docker compose run --rm scraper python main.py --scan

This scrapes once, applies your filters, stores results in the database, and sends Telegram alerts for any matches. Use this to verify everything works before starting the continuous scheduler.

Configuration

All settings live in config.yaml. The file is gitignored - never committed.

Search filters

search:
  price_min: 4000        # minimum monthly rent (ILS)
  price_max: 8000        # maximum monthly rent (ILS)
  rooms_min: 2.5         # Israeli room count (1.5, 2, 2.5, 3, 3.5, 4...)
  rooms_max: 4
  listing_type: apartment  # "apartment" or "roommates"
  min_size_m2: null      # minimum apartment size in square meters (null = no filter)
  max_floor: null        # maximum floor number (null = no filter)

  locations:
    - city: "תל אביב יפו"        # Hebrew city name as it appears on yad2
      neighborhoods: []           # empty = all neighborhoods
    - city: "רמת גן"
      neighborhoods: ["בורוכוב"] # specific neighborhood filter

  amenities:             # only list what you strictly require - unspecified = ignored
    parking: true
    elevator: true
    # balcony: true
    # safe_room: true
    # air_conditioning: true
    # furnished: true
    # pets_allowed: true
    # storeroom: true
    # bars: true
    # handicapped_access: true

  property_types: []    # empty = all types
  conditions: []        # empty = all conditions

City names must be in Hebrew. Supported cities are listed in scraper/cities.py. Common ones: תל אביב יפו, ירושלים, חיפה, ראשון לציון, פתח תקווה, רמת גן, נתניה, באר שבע, בני ברק, חולון, בת ים, הרצליה, כפר סבא, רעננה, רחובות.

Telegram

telegram:
  bot_token: "YOUR_BOT_TOKEN"
  chat_id: YOUR_CHAT_ID

To get these values:

1. Message @BotFather on Telegram and send /newbot to create a bot. Copy the token.
2. Send any message to your new bot (required before it can message you).
3. Open https://api.telegram.org/bot<YOUR_TOKEN>/getUpdates in a browser. Your chat_id is the "id" field inside "chat" in the response.

Schedule

schedule:
  interval_minutes: 60         # how often to scrape (Docker mode only)
  daily_summary_time: "08:00"  # time to send the daily summary (24h, local time)

Database

database:
  path: "data/listings.db"   # leave as-is

Scraper

scraper:
  request_timeout_seconds: 10
  retry_attempts: 3
  proxy: null   # residential proxy URL - only needed if your server IP gets blocked by yad2

Deployment

Docker (recommended for home server or VPS)

Start the continuous scheduler:

docker compose up -d

The scraper runs every interval_minutes and sends a daily summary at daily_summary_time. Logs go to stdout and are visible via docker compose logs -f.

Stop it:

docker compose down

Running tests

pytest tests/ -v --ignore=tests/test_api.py

The live API test (tests/test_api.py) hits the real yad2 API. Run it manually:

pytest -m live

Project structure

scraper/
  api.py        - yad2 API client, TLS fingerprint rotation, anti-bot evasion
  cities.py     - Hebrew city name to numeric yad2 ID mapping
  config.py     - config loader and validator
  db.py         - SQLite database layer
  filters.py    - listing filter engine
  models.py     - Listing dataclass
  notifier.py   - Telegram alert sender
  scheduler.py  - APScheduler loop and scrape cycle logic
main.py         - entrypoint
config.yaml     - your personal config (gitignored)
config.example.yaml - template to copy from

Notes

• The first scan after a long gap will alert all listings in the database that haven't been seen before. Subsequent runs only alert genuinely new listings.
• If you see captcha warnings in the logs, your IP is being rate-limited by ShieldSquare. The scraper retries automatically. Persistent blocks on a cloud/VPS IP require a residential proxy.
• config.yaml is gitignored. Never commit it - it contains your Telegram credentials.