Integration - tool to run Erigon stages in custom way: run/reset single stage, run all stages but reorg every X blocks, etc...

Examples

All commands require parameter --datadir=<datadir> - I will skip it for readability.

integration --help
integration print_stages

# Run single stage
integration stage_senders
integration stage_exec
integration stage_exec --block=1_000_000 # stop at 1M block
integration stage_exec --sync.mode.chaintip # every block: `ComputeCommitment`, `rwtx.Commit()`, write diffs/changesets

# Unwind single stage 10 blocks backward
integration stage_exec --unwind=10

# Drop data of single stage
integration stage_exec --reset

# Unwind single stage N blocks backward
integration stage_exec --unwind=N

# Run stage prune to block N
integration stage_exec --prune.to=N

# To remove all blocks (together with bodies/txs) from db 
integration stage_headers --reset --datadir=<my_datadir> --chain=<my_chain>

# Run txn replay with domains [requires 6th stage to be done before run]
integration read_domains --chain sepolia account <addr> <addr> ... # read values for given accounts

For testing run all stages in "N blocks forward M blocks re-org" loop

Pre-requirements of state_stages command:

• Headers/Bodies must be downloaded
• TxSenders stage must be executed

make all
./build/bin/integration state_stages --datadir=<datadir> --unwind=10 --unwind.every=20 --pprof
integration reset_state # drops all stages after Senders stage (including it's db tables DB tables)

For example:

--unwind=1 --unwind.every=10  # 10 blocks forward, 1 block back, 10 blocks forward, ...
--unwind=10 --unwind.every=1  # 1 block forward, 10 blocks back, 1 blocks forward, ...
--unwind=10  # 10 blocks back, then stop
--integrity.fast=false --integrity.slow=false # it performs DB integrity checks each step. You can disable slow or fast checks.
--block # stop at exact blocks
--chaindata.reference # When finish all cycles, does comparison to this db file.

How to unwind node

In Erigon3 - better do rm -rf chaindata (for bor maybe also need remove polygon-bridge, bor, heimdall folders until https://github.com/erigontech/erigon/issues/13674 is fixed)

Copy data to another db

In Erigon3 - better do rm -rf chaindata (for bor maybe also need remove polygon-bridge, bor, heimdall folders until https://github.com/erigontech/erigon/issues/13674 is fixed)

0. You will need 2x disk space (can be different disks).
1. Stop Erigon
2. Create new db with new --db.pagesize:
ONLY_CREATE_DB=true ./build/bin/erigon --datadir=/erigon-new/ --chain="$CHAIN" --db.pagesize=8kb --db.size.limit=12T
# if erigon doesn't stop after 1 min. just stop it.
3. Build integration: cd erigon; make integration
5. Run: ./build/bin/integration mdbx_to_mdbx --chaindata /existing/erigon/path/chaindata/ --chaindata.to /erigon-new/chaindata/
6. cp -R /existing/erigon/path/snapshots /erigon-new/snapshots
7. start erigon in new datadir as usually

Recover db from some bad state

If you face db-open error like MDBX_PROBLEM: Unexpected internal error. First: use tools like https://www.memtest86.com to test RAM and tools like https://www.smartmontools.org to test Disk. If hardware is fine: can try manually recover db (see ./build/bin/mdbx_chk -h for more details):

make db-tools

./build/bin/mdbx_chk -0 -d /erigon/chaindata
./build/bin/mdbx_chk -1 -d /erigon/chaindata
./build/bin/mdbx_chk -2 -d /erigon/chaindata

# if all 3 commands return success - then remove `-d` parameter and run again
# if all 1 command is fail but other success. choose successful number - for example 2 - and switch db manually to it:  
./build/bin/mdbx_chk -2 -d -t -w /erigon/chaindata  

# if all 3 commands are fail - game over. use backups.

Clear bad blocks markers table in the case some block was marked as invalid after some error

It allows to process this blocks again

1. ./build/bin/integration clear_bad_blocks --datadir=<datadir>

FAQ

How to re-exec all blocks

# Option 1 (on empty datadir):
erigon --snap.skip-state-snapshot-download

# Option 2 (on synced datadir):
erigon snapshots rm-all-state-state
integration stage_exec --reset
integration stage_exec 

# Option 2 is good 

How to re-gen CommitmentDomain

integration commitment rebuild

How to re-generate optional Domain/Index

# By parallel executing blocks on existing historical state. Can be 1 or many domains:
erigon snapshots rm-state-snapshots --domain=receipt,rcache,logtopics,logaddrs,tracesfrom,tracesto
integration stage_custom_trace --domain=receipt,rcache,logtopics,logaddrs,tracesfrom,tracesto --reset
integration stage_custom_trace --domain=receipt,rcache,logtopics,logaddrs,tracesfrom,tracesto

How to remove history snapshots only (keep domain data)

# Remove all history files (SnapHistory + SnapIdx + SnapAccessors) without touching domain data (.kv files):
erigon snapshots rm-state --only-history

# Remove a specific step range of history files only:
erigon snapshots rm-state --only-history --step=0-900

# Dry-run first to see what would be deleted:
erigon snapshots rm-state --only-history --step=0-900 --dry-run

How to re-gen bor checkpoints

rm -rf datadir/heimdall
rm -rf datadir/snapshots/*borch*
# Start erigon, it will gen. Then:
erigon snapshots integrity --datadir /erigon-data/ --check=BorCheckpoints

Compact chaindata in-place

src=<datadir>/chaindata && ./build/bin/mdbx_copy -c -u "$src" "${src}/mdbx.dat.tmp" && mv "${src}/mdbx.dat.tmp" "${src}/mdbx.dat" || rm -f "${src}/mdbx.dat.tmp"

See tables size

./build/bin/mdbx_stat -efa  /erigon-data/mainnet_archive/chaindata/ | awk '
    BEGIN { pagesize = 4096 }
    /^  Pagesize:/ { pagesize = \$2 }
    /^Status of/ { table = \$3 }
    /Branch pages:/ { branch = \$3 }
    /Leaf pages:/ { leaf = \$3 }
    /Overflow pages:/ { overflow = \$3 }
    /Entries:/ {
      table_pages = branch + leaf + overflow
      size_gb = (table_pages * pagesize) / ($1024^{3}$)
      printf "%-30s %.2fG\n", table, size_gb
    }
    /Reclaimable:/ {
      table = "Reclaimable" 
      table_pages = \$2
      size_gb = (table_pages * pagesize) / ($1024^{3}$)
      printf "%-30s %.2fG\n", table, size_gb
    }
  ' | grep -v '0.0'