crusty-backup

June 25, 2026 · View on GitHub

Living scope + plan for crusty-backup's (cb-dos) networked backup/restore and its disk-state / resume / change-detection model. This is the companion to cb_dos.md (the overall cb-dos scope); that doc deferred networking to §6 / Phase 7 and left "incremental-ish" state tracking unspecified. This file captures the design discussion that fleshed both out.

Unified-daemon note (2026-06-20): the networked transport here is now Family B of the single rb-cli serve daemon described in the umbrella plan remote_transfer_plan.md. The host listener is rb-cli serve (not a separate net-serve verb); cb-dos reaches it as a JSON-free Family-B client over a shared binary Hello handshake, and the same chunk-stream now also serves desktop-pulls-remote-disk and restore-over- the-wire (producer/consumer symmetry — see the umbrella §0, §7). The chunk container, resume map, fingerprint, and manifest specced below are unchanged and remain the source of truth for Family B's internals.

Two intertwined subjects live here:

  1. Transport — how cb-dos moves the native backup folder to/from the modern desktop over a wire (not just removable media).
  2. Disk-state calculations — how cb-dos and the desktop decide whether a transfer is resumable, whether a source disk is unchanged, what files are volatile (swap), and how backup↔restore stays idempotent.

These are coupled: resume, fingerprinting, the chunked container, and the file manifest are all the same machinery viewed from different angles.

Status legend: [ ] not started · [~] in progress · [x] done · [-] dropped/out of scope

Nothing here is built yet. This is a scoping document. It slots into cb_dos.md's Phase 7 (deferred) and should not be started until the local removable-media round-trip (cb_dos Phases 2–4) proves the format.

0. Confirmed scope decisions

From the scoping discussion (2026-06-03):

DecisionChoice
Transport familyTCP/IP over a packet driver — not raw L2 Ethernet, not serial/parallel
Why TCP (not raw L2 or UDP)Reliable in-order stream → no hand-rolled ACK/retransmit; testable with stock tools (nc, Wireshark, loopback sockets); host becomes a plain unprivileged socket (no pcap/Npcap/BPF/root)
Stack on DOSA small TCP/IP stack over the Crynwr packet-driver API (INT 60h). Reference / candidate-to-borrow: mTCP (GPLv3, AGPL-compatible)
Host listener homerb-cli serve daemon, Family B (unified — was net-serve) — reuses format + restore code, scriptable, cross-platform
On-disk artifactA single chunked container (.cbk), not a folder-of-files on the wire. Container doubles as the resume log; folder is materialized at finalize for the desktop restore path
Resume modelddrescue-style transfer map of committed chunks; fsync-before-record ordering; truncate-to-last-committed on resume
Chunk unitOne independently-compressed gzip member covering a fixed span of uncompressed source data (~1–4 MB). Self-verifying (gzip CRC trailer); producer resumes by source-offset seek, never recompresses
"Same source?" gateCheap composite fingerprint (geometry + MBR/ptable CRC + volume serials + allocation-bitmap CRC), not a full-disk CRC
Change detectionrsync-tier: size + mtime + archive bit, per file, via a manifest. Best-effort gate; streamed per-member/per-partition CRC is the real integrity guarantee
IdempotencyRestore replays recorded mtime + attributes (incl. archive bit) so a fresh restore does not look like a fresh modification
Boot protectionManifest fingerprints + round-trips MBR boot code, VBR/hidden sectors, and IO.SYS/MSDOS.SYS/COMMAND.COM
Swap filesExcluded from the change counter always; zeroed in the payload by default (--keep-swap to opt back in)

Out of scope (for now): raw L2 Ethernet, serial, parallel; hand-rolled TCP (borrow/port mTCP if we want TCP-the-library); whole-disk content CRC as a gate; Level-2 swap deallocation (see §6).

1. Transport — TCP/IP over the packet driver

1a. Why TCP, and why over a packet driver

DOS networking layers as:

cb-dos transfer protocol
   TCP                ← reliable stream (borrowed or minimal)
   IP + ARP + ICMP
   packet driver      ← Crynwr INT 60h (send/recv raw frames)
   NIC

The packet-driver API (Russ Nelson / Crynwr) is the thin, universal L2 substrate every DOS NIC ships a driver for, and the standard base for mTCP / WATTCP. It is not what DOS games used — those were mostly IPX/ODI (Doom, Descent, etc.); the packet driver is the TCP/IP-tools world.

TCP over raw L2 was chosen because TCP's reliable, in-order byte stream means we write zero sequencing/ACK/retransmit code, and — critically — it moves all the painful verification problems off the host:

Raw L2 planTCP plan
Host privilegepcap/Npcap, root, BPF permsplain unprivileged socket
Wi-Fibreaks (AP drops spoofed-MAC L2)fine (routable)
86Box test modemust be bridged/pcap on a wired iface (SLiRP drops frames)SLiRP "just works" via port-forward
Host test toolingbespokenc/socat/Wireshark/loopback unit tests
Reliability layerhand-rolledfree (TCP)

The cost of going IP is paid on the DOS side (carry ARP/IP/TCP; the box needs an IP — one AUTOEXEC.BAT line or DHCP). That trade — a bit more DOS code for a trivial host and painless testing — is judged worth it.

1b. mTCP — reference, and borrow-vs-rewrite

mTCP (Michael Brutman, GPLv3, compatible with our AGPL-3.0) is the canonical "small TCP/IP stack on a packet driver." It implements both TCP and UDP over IP (ARP, IP fragments, DNS, DHCP client, listen/accept, Karn's-algorithm retransmit, compile-in/out features for size). Docs:

Decision (2026-06-21) — WATT-32, not mTCP. mTCP is Open Watcom / C++; cb-dos is DJGPP / C, and the borrow-vs-port question (build mTCP under Watcom and link, switch cb-dos to Watcom, or port mTCP's UDP/IP to DJGPP and hand-roll reliability) all carried real cost. WATT-32 (Gisle Vanem / Erick Engelke, the Waterloo-TCP lineage) dissolves it: it builds natively under DJGPP and exposes a BSD-sockets API (socket/connect/send/recv/closesocket) over the same Crynwr packet driver. One toolchain, real TCP, no Watcom.

  • Acquisition: the prebuilt DJGPP binary package (wat3211b.zip on the DJGPP mirrors) ships libwatt.a + headers — no cross-build of the stack from source. Fetched by ../crusty-backup/net/fetch-watt32.sh into net/watt32/ (gitignored, like the mTCP apps); the make net target links -Lnet/watt32/lib -lwatt.
  • License: WATT-32 is BSD (Regents of California). The historical advertising-acknowledgment clause is a permitted additional term under AGPL-3.0 §7, so it is compatible with this project's AGPL-3.0 — the license file travels with the library.
  • Startup: sock_init() brings up the packet driver + IP (DHCP or static from WATTCP.CFG); blocking recv/send drive the stack internally — no hand-rolled tick loop in the client.
  • Considered-and-rejected: borrow mTCP (Watcom toolchain split), port mTCP UDP + hand-rolled reliability (gives up free TCP), hand-roll TCP (a real project — the whole reason a stack exists). Reserve raw-L2/UDP only if a reason resurfaces.

mTCP's prebuilt apps (DHCP/FTP/PING) still ride the CD for manual/diagnostic use (cb_dos_networking.md); WATT-32 is specifically the library the streaming client links.

1c. The DOS driver requirement (no way around some driver)

cb-dos does not eliminate the per-NIC driver — nothing in DOS can auto-drive arbitrary NIC silicon. It requires a packet driver (Crynwr API), not NDIS or ODI:

ModelUsed byUs
Packet drivermTCP, WATTCP, EtherDFSyes — one small TSR, e.g. NE2000 0x60 3 0x300
NDIS 2.xMS LAN Managerno
ODINetWare/IPXno

NDIS/ODI-only cards can still feed us via a shim (DIS_PKT.DOS for NDIS, ODIPKT.COM for ODI). The boot-media plan softens the burden: bundle the common packet drivers (NE2000, 3C5x9, PCNTPK, RTSPKT…) + a boot-menu "Network" profile that autodetects mainstream cards. The long tail loads its own driver manually — documented, detected, refused gracefully if absent.

1d. Test rig — 86Box / DOSBox-X networking mode (gotcha)

The VM NIC must be in PCap / bridged mode (true L2 bridge via libpcap/Npcap) — not SLiRP — for raw frames to reach the host. However, because we chose TCP/IP, SLiRP works too (it NATs IP): set a port-forward and the whole loop runs on the default emulator networking. Caveats if bridged:

  • Wired, not Wi-Fi (Wi-Fi drivers drop bridged spoofed-MAC frames).
  • Cleanest rig: host listener on a second physical machine on the same switch (removes same-host loopback ambiguity).
  • macOS BPF / /dev/bpf* access for the emulator (and for pcap if ever used).

The emulated NE2000 / PCnet has a stock packet driver, so the entire path is exercisable on the Mac before real 486 hardware.

1e. Security posture

Intended use is an isolated host↔vintage-box LAN. The protocol is unauthenticated/unencrypted — a 486 can't afford crypto. Document "don't run on an untrusted network"; do not bolt on encryption.

2. The chunked container (.cbk) — wire framing = on-disk artifact = resume log

The key idea: separate the wire stream from the on-disk artifact, then make the on-disk artifact a single chunked container that is the resume journal.

2a. On the wire — one multiplexed chunk stream

Never N separate file transfers. One ordered sequence of chunks, each with a small header:

chunk hdr: { magic, version, logical_id, src_offset, len, crc32 }

logical_id names the logical member (metadata / mbr / partition-0 / …). One stream, one resume cursor, regardless of how many logical files exist.

2b. On disk — append-only container + index

The host writes one file, MYDISK.cbk, = chunks appended in arrival order plus a trailer/sidecar index:

[chunk hdr][payload][chunk hdr][payload] ...   append-only log; payload = gzip member
[index/trailer]                                logical_id -> {chunk offsets, crc}

This one file does triple duty: wire framing, resume log, and deliverable. A small .idx sidecar (updated via atomic rename) avoids rescanning a multi-GB container on every resume.

Bridge to the desktop restore path: the desktop's resize-on-restore consumes the native PerPartition folder, not .cbk. Two options:

  • (a) Materialize the folder from the container at finalize (fast local untar). Lower effort, desktop unchanged. Ship first.
  • (b) Teach the desktop to read partitions directly out of .cbk as a backup source. Nicer if the single file should be the permanent artifact. Later.

2c. The chunk = an independent, source-keyed gzip member

Instead of one continuous gzip stream per partition, emit the partition as a sequence of independent gzip members, each covering a fixed span of uncompressed source data (~1–4 MB; a bit bigger than the 400 KB first floated, to balance ratio vs. fsync frequency vs. resume granularity).

Why this exact shape:

  • gzip is multi-member by spec — concatenated members decompress as one file. partition-1.gz = member₀++member₁++… is valid gzip. Desktop must decode with flate2::MultiGzDecoder, not GzDecoder (the latter stops after member 0) — a one-line change in compress.rs alongside the planned Gzip codec.
  • Each member self-verifies — gzip's CRC32 + length trailer means a fully written member is provably complete. Free per-chunk integrity.
  • Producer resume is a seek, not a recompress — member K maps to source range [K·S, (K+1)·S); resuming = int 13h-seek the card to K·S and continue. Never recompresses already-sent data (critical on a slow 486).
  • No determinism dependency — resuming at source boundaries sidesteps any "skip the first N compressed bytes" fragility across zlib versions.

Tradeoff: independent members reset the deflate dictionary each span — negligible ratio loss at MB granularity. Tune span up for ratio/fewer fsyncs, down for finer resume.

2d. Frozen v1 on-disk format (shipped desktop-first, 2026-06-24)

Implemented on the desktop ahead of the network transport — src/rbformats/cbk.rs (pack_folder_to_cbk / materialize_cbk_to_folder), rb-cli cbk pack|unpack, and rb-cli restore <x>.cbk (materializes a temp folder, then restores). This freezes the format the DOS Family-B producer (7b) must emit. All integers big-endian (the RBK convention).

file = [chunk]* [index] [footer:16]

chunk (24-byte header + payload):
  magic       u32 = "RBKC" (0x5242_4B43)
  version     u16 = 1
  logical_id  u16              member index (into the index table)
  src_offset  u64              uncompressed offset of this span in the member
  len         u32              payload byte length
  crc32       u32              CRC32 of the payload bytes
  payload     [len]            one independent gzip member

index:
  magic        u32 = "RBKI" (0x5242_4B49)
  version      u16 = 1
  member_count u16
  per member:  name_len u16, name (UTF-8, folder-relative),
               kind u8  (0=Gz: payloads concatenated ARE the file, e.g.
                         partition-N.gz; 1=Raw: file = gunzip(concat payloads),
                         e.g. metadata.json / mbr.bin),
               logical_id u16, chunk_count u32, chunk_count × chunk_offset u64

footer (last 16 bytes): magic u32 = "RBKF" (0x5242_4B46), index_offset u64,
                        index_len u32

A reader seeks EOF−16, reads the footer, jumps to the index — no full scan. The desktop packer emits one chunk per member (simplest valid chunking); the DOS producer will append many chunks per partition (each a ~1–4 MB-span gzip member) for the resume granularity §2c/§3 need. Both are valid .cbk this materializer reads. The network .idx sidecar (§2b/§3) layers on top of this: it is the same per-member chunk-offset list written incrementally for crash-safe resume, with the trailer index as the finalized form.

2e. .cbk as a first-class image — native inspect/browse/restore (planned)

Decision (2026-06-24): a .cbk must behave like any other disk image everywhere in the app — inspect, ls/get (browse + extract files), fsck, GUI Inspect/Restore — with no user-visible "extract first" step. Restore from .cbk is treated exactly like a folder restore. Editing a .cbk may do extra legwork (materialize → edit → repack) and is in scope but lower priority.

The whole app reads disks through one abstraction — Box<dyn ReadSeek> (ReadSeek: Read + Seek + Send, src/rbformats/mod.rs) — and a partition is just a byte offset into that whole-disk reader (open_filesystem(reader, offset, …)). So native .cbk support = present the container as a reconstructed whole disk (table at sector 0, partitions at their byte_offset(), gaps zero) behind that trait, then hook it into the one open/detect dispatch every consumer funnels through.

Phased plan (this is the work, in order):

  • (i) CbkReader: ReadSeek — open a .cbk as a seekable disk. v1 implementation reuses the existing compressed-image precedent (ZipDiskReader / GzipTempReader): materialize the container to a temp folder, reconstruct the disk into a sparse temp file (reconstruct_disk_from_backup, src/rbformats/mod.rs), and delegate Read/Seek to it. Transparent — the user never runs an extract step; the temp disk is the app's business, exactly like opening a .zip-wrapped image today. (Future: a truly lazy reader that decompresses only the chunked members a seek touches — the §2c/§2d span layout is designed for this; it's a reader upgrade, no format change.)
  • (ii) Hook the dispatch — add an is_cbk_path arm to open_read_dispatch (src/model/source_reader.rs) and .cbk to is_container_path. That single arm lights up inspect, the CLI resolve layer (ls/get/fsck via open_peeled_read), and GUI Inspect automatically. Add a .cbk arm to BrowseSession::open_image (src/model/browse_session.rs) for browse/extract.
  • (iii) GUI picker + associations — add "cbk" to DISK_IMAGE_EXTS (src/model/file_types.rs; not NON_ASSOCIATED_EXTS — a .cbk should double-click-open) + a regression test. The Inspect/Restore tabs then accept a .cbk with no further change.
  • (iv) Restore parity — already done for the CLI (materialize-to-temp arm in restore.rs); GUI restore picks up .cbk for free once it's in DISK_IMAGE_EXTS and routed like a folder.
  • (v) Edit — DONE (2026-06-24). resolve_partition_rw_forced detects a .cbk, materializes it to a temp folder, edits the partition via the existing backup-folder RW path (open_backup_partition_rw), and on commit (RwCommit::Cbk) recompresses partition-N.gz + rewrites metadata.json and repacks the folder over the original .cbk (write-to-.tmp + rename). gzip was added to the editable-codec whitelist (decompress/recompress already handled it). So rb-cli put/rm/mkdir/cp X.cbk@N … all work; verified put → ls/get → restore round-trips with both old and new files intact. Like editing a .zst backup, the edit decompresses to full size and drops cb-dos's smart compaction (the container grows) — correct, just larger.

3. Resume — the transfer map

3a. The crash-consistency rule

TCP guarantees order only while the connection lives; a crash/reboot kills the socket and TCP gives nothing across it. Resume is entirely application-layer.

The host owns durable resume state (stable storage, assembly point, survives the DOS box rebooting; the DOS side stays ~stateless and can't write the source card anyway). The receive loop, with strict ordering:

  1. Receive a full member into the container at its offset.
  2. fsync the container — bytes durable.
  3. Then append the member record to the index, written atomically (write .idx.tmp, fsync, rename() over the real file; ReplaceFile/MoveFileEx on Windows).

Because data is fsynced before it's recorded committed, a crash can never mark a member committed that isn't on disk. The reverse (durable but unrecorded) is harmless — re-receive that one member.

3b. The resume handshake

  1. cb-dos reconnects: RESUME session_id, source=<fingerprint> (see §4).
  2. Host loads the index, verifies source fingerprint (refuse/warn on swap), and truncates each in-progress member back to its last committed end offset — discards any half-written trailing member. Disk state now == recorded state.
  3. Host replies per logical member: "complete" or "resume at member K (source offset K·S)".
  4. cb-dos seeks the source and resumes emitting members; host continues the checkpoint loop.
  5. At finalize, host verifies each member against the per-partition checksum the native format already expects (end-to-end gate). Per-member CRCs caught mid-flight corruption.

Ordering: send big resumable members first; write tiny metadata.json (needs final compressed sizes/checksums) last — if a crash precedes it, regenerate it on resume (cheap; not worth checkpointing).

3c. Prior art

This is GNU ddrescue's mapfile pattern (durable offset size status map → resume; periodic rewrite bounds loss) made content-aware. Reference its mapfile format when specing the .idx. partclone's per-N-block embedded CRC32 is the direct analog of our per-member CRC.

4. Disk-state fingerprint — "same source?" (cheap gate)

Never take a full-disk CRC — reading the whole card over int13h is the exact work resume exists to avoid, and you'd pay it every check. "Resumable?" only needs "is this the same, unchanged source I started?", answered by a composite fingerprint from a few seconds of reads.

4a. The layered fingerprint, by cost vs. power

IngredientCostCatches a change when…
Geometry + LBA count + MBR/ptable CRC~free (1 sector)layout / repartition
Volume serial(s) (from BPB)~free (1 sector/vol)reformat
Free-cluster count~freeadd/delete/resize — UX only, subsumed below
Allocation-bitmap CRC (FAT / $Bitmap)~free — already walked for compactionany allocation change: add/delete/move/defrag/resize
Directory-tree hash (mtime/size/start cluster)dir walkmost same-size edits
Full content hashwhole-disk readeverything

The allocation-bitmap CRC is the workhorse and it's free (cb-dos walks the FAT/$Bitmap for compaction regardless). It subsumes free-space (popcount of the same bitmap) — keep free-space for UX/preflight only, not as the detector.

Recorded per partition:

disk:  { geometry, lba_count, mbr_crc, ptable_crc }
vol[N]:{ serial, free_clusters, alloc_bitmap_crc }   // bitmap_crc = workhorse

4b. Why this isn't redundant with the per-member CRCs

Per-member CRCs verify wire integrity (sent == received). They cannot detect that the source changed between the original session and the resume — each member is individually valid, just from different points in time. Splicing old + new members yields an internally inconsistent, split-vintage image with every CRC passing. The bitmap fingerprint is the only cheap thing that catches that. Not duplication — it guards a hole the content CRCs structurally can't see.

4c. Honest limit

Even bitmap-CRC + dir-hash miss a same-cluster, same-size, timestamp-preserving overwrite. Only a content hash catches that — and we do compute it, during transfer (per-member + per-partition), not as an upfront gate. So:

  • Fingerprint = cheap pre-flight gate ("probably same, unchanged → proceed").
  • Streamed per-member / per-partition CRC = the integrity guarantee.

Given cb-dos's quiescent-source model (boot from floppy → the card is a passive int13h device nothing writes; the DOS substitute for a snapshot), the realistic threat is a swapped card or one booted-and-edited between resume attempts — both caught by the fingerprint. Optional extra: on resume, re-hash just the last committed member's source range and compare (one ~MB read).

4d. Bonus: the fingerprint is the incremental index

The same per-partition {serial, bitmap_crc} compared against a prior completed backup answers "is this partition unchanged → skip re-imaging, copy from prior." Resume gate and incremental index are the same artifact. v1 = resume only; incremental falls out later.

5. File manifest — "did it change?" + idempotency + boot protection

Change-detection is rsync-tier best-effort: size + mtime + archive bit, per file. The payload stays a block image (preserves bootability + desktop resize); we add a file-level manifest sidecar — file-awareness without giving up block fidelity. This is the "more of a filesystem backup" feel.

5a. Manifest contents (per partition)

files[]: { path, size, mtime, attribs{archive,hidden,system,readonly}, start_cluster }
system:  { mbr_boot_code_hash, vbr_hash, hidden_sectors_hash,
           sysfiles:[ {name:"IO.SYS",  size, mtime, attribs, first_cluster, hash},
                      {name:"MSDOS.SYS", ...}, {name:"COMMAND.COM", ...} ] }

Built by one directory-tree walk (the same walk that finds swap files, §6).

5b. Change detection

Diff new manifest vs. stored:

  • Whole-disk identity gate — all files (size+mtime+archive) and the system section equal a prior backup → "identical to backup B1, nothing to do" and stop. (This is the "prevent a pointless re-backup right after a restore.")
  • Per-file incremental — otherwise the diff is the changed set (combine with §4 bitmap CRC for the resume gate).

5c. Idempotency — the archive bit is round-tripped, not computed

The footgun: a restore writes every file, and DOS sets the archive bit on every write → a naïve restore leaves every file archive=set → re-backup thinks "everything changed," defeating the gate.

Fix — make mtime + attribs a round-tripped attribute:

  1. Backup records each file's mtime + attribs (incl. the archive bit's actual state).
  2. Restore replays them — after writing data, set original mtime (int 21h/5701h) and attributes incl. archive bit (int 21h/4301h) from the manifest. A restored file looks exactly as at backup time, not "freshly written."
  3. Re-backup sees no change → backup↔restore is idempotent.

So the archive bit's job here is surviving the round-trip, not detecting modification (it's too spoofable for that — see 5e).

5d. Boot protection ("check boot stuff")

A backup that loses bootability is worthless for the vintage use case. The system block fingerprints + round-trips exactly what the DOS SYS command babysits: MBR boot code (446 bytes), each VBR + hidden sectors, and IO.SYS/MSDOS.SYS/COMMAND.COM (name, size, mtime, attribs, first cluster + contiguity, content hash). First-cluster/contiguity is what actually determines boot and is the natural pre-flight for the Phase-B defrag (which must keep those first and contiguous). Mismatch vs. prior backup ⇒ flag a bootability change.

5e. Honest caveat (don't oversell)

size+mtime+archive is fooled by a same-size, timestamp-preserving, bit-cleared overwrite, and the archive bit can be cleared by another DOS backup tool (MSBACKUP/XCOPY /M) between sessions. Only content hashing is airtight — which we still stream. Manifest = cheap gate + idempotency mechanism; streamed CRCs = the guarantee.

5f. Prior art

rsync's default quick-check (size+mtime, no content read; --checksum to read) is exactly our two-tier split. The DOS-native archive bit (BACKUP/MSBACKUP/ XCOPY /M set-on-write, clear-on-backup) is the historical file-level change primitive — we round-trip it rather than rely on it. Snapshots (VSS/LVM/ zfs-send) are the "proper" solution DOS lacks; our boot-from-floppy quiescence is the substitute.

6. Swap-file exclusion

The manifest walk identifies swap files (you can only exclude what you can find the cluster chain for). Two separable actions.

6a. Recognized swap/page files

SystemFile(s)Location / attribs
Windows 3.x (386 Enh.)386SPART.PAR (perm), WIN386.SWP (temp)root, hidden; SPART.PAR in Win dir points to it
Windows 9xWIN386.SWPWindows dir or root
Windows NT/2k/XP (NTFS or FAT)pagefile.sys, hiberfil.sysroot, hidden+system
OS/2SWAPPER.DATusually \OS2\SYSTEM

hiberfil.sys (= RAM size) and pagefile.sys/386SPART.PAR are the big wins.

DO NOT EXCLUDE DBLSPACE.000 / DRVSPACE.000 / STACVOL.DSK — those are the compressed filesystem; excluding them destroys the volume. The exclusion list is a strict allowlist of known names in expected locations with expected attributes — never a fuzzy match.

6b. Exclude from the change counter (always)

Swap churns every boot but carries no meaningful state. Keep it listed in the manifest (name/size/attribs/chain) flagged volatile:true, but exclude from the change signal — else every backup looks "changed" and idempotency (§5c) collapses. Unambiguously correct, free.

6c. Exclude from the payload (default on, --keep-swap to opt out)

The payload images allocated clusters; a swap file's clusters are allocated. Level 1 (ship this) — zero the content, keep the allocation (what Ghost/ntfsclone do):

  • Keep the directory entry / MFT record, FAT/$Bitmap allocation, and cluster chain as-is → filesystem stays fully consistent, file still exists full-size.
  • Emit zeros for those clusters in the stream → gzip crushes them.
  • Restore brings the file back full-size, zero-filled; OS reinitializes swap on next boot. (hiberfil.sys consequence: restored disk cold-boots instead of resuming from hibernation — almost always desired; document it.)

Level 2 (optional, later) — actually deallocate (free the FAT chain + drop the dir entry) so the volume's minimum size shrinks and resize-restore reclaims it. A real FS mutation in the image (more code/risk); the OS recreates swap anyway, so the only gain over Level 1 is a smaller resize-down minimum. Defer.

6d. Behavior

  • On by default, --keep-swap escape hatch.
  • Conservative ID: exact name + expected location + expected attribs.
  • Log every exclusion ("Excluded WIN386.SWP (12 MB, zeroed)") — never silent.
  • Record excluded:true, content:zeroed in the manifest so restore recreates the file and the counter keeps ignoring it.

6e. Feature-parity note — shipped (2026-06-25)

Per CLAUDE.md's GUI/CLI parity + shared-business-logic rules, swap exclusion is not a cb-dos-only trick — the Rust desktop backup now does the same Level-1 zeroing. CompactFatReader::new_excluding_swap (src/fs/fat.rs) runs the same strict allowlist over root / \WINDOWS / \OS2\SYSTEM and zeros the swap files' content while keeping their allocation; keep_swap threads through BackupConfig to an rb-cli --keep-swap flag and a GUI checkbox (default = exclude). It works in both plain and defrag compaction (the set is keyed by old cluster number, which survives the defrag remap — unlike cb-dos's by-position mask, which /DEFRAG skips). single-file-CHD is the one desktop format that still images swap verbatim (a documented follow-up).

7. Prior-art map (for the record)

Our pieceEstablished pattern
Transfer map / resumeGNU ddrescue mapfile
Used-blocks-only imagepartimage / partclone / Ghost "smart" copy
Chunked / spanned imageGhost segments, multi-volume ARJ/ZIP/RAR
Per-chunk CRCpartclone embedded CRC32 every N blocks
Cheap change gatersync size+mtime quick-check (vs. --checksum)
File-level change primitiveDOS archive bit (BACKUP/MSBACKUP/XCOPY /M)
Snapshot we lackVSS / LVM / zfs-send → substituted by boot-from-floppy quiescence
TCP/IP on packet drivermTCP / WATTCP

No single tool does resumable + fingerprint-gated + networked block imaging on DOS — the combination is the new part; the primitives are all proven.

8. Open risks / unknowns

  • mTCP toolchain mixing — Open Watcom/C++ vs our DJGPP/C. Resolved (§1b): WATT-32 (DJGPP-native BSD sockets, prebuilt libwatt.a) — no Watcom, one toolchain. The nethello client links it and compiles clean under DJGPP.
  • DOS IP config UX — static IP (one AUTOEXEC.BAT line) vs DHCP client; detect/guide.
  • Container ↔ folder bridge — ship (a) materialize-at-finalize first; (b) desktop reads .cbk directly is a later, larger change.
  • MultiGzDecoder — confirm the desktop reads multi-member .gz everywhere the Gzip codec lands (not just one call site).
  • Fingerprint false-negative — same-cluster/same-size/timestamp-preserving overwrite escapes the gate; mitigated by streamed CRC + quiescent-source model. Document the ceiling.
  • Archive-bit cleared by 3rd-party tools — the gate weakens if another backup tool ran between sessions; not a correctness bug (streamed CRC backstops), but the "identical, nothing to do" shortcut could mis-fire. Keep it advisory.
  • Swap ID safety — strict allowlist; never touch DBLSPACE/DRVSPACE/STACVOL.
  • 86Box networking mode — SLiRP+port-forward (TCP) works; if ever bridged, wired-only + ideally a second host.

9. Phased plan (within cb_dos Phase 7)

Prerequisite: cb_dos Phases 2–4 (local removable-media round-trip) must prove the native format first. Networking only swaps the destination under a working engine.

  • 7a — Frame/socket hello-world. Done — handshake round-trips end-to-end (real FreeDOS in qemu over an NE2000 packet driver + SLiRP, 2026-06-24). - Host (done, headless-tested): rb-cli serve now accepts a binary Family-B handshake alongside the JSON Family-F one on the same port — read_handshake peeks the 4-byte magic (b"RBK0") to disambiguate, and write_binary_hello replies (magic + version + caps, big-endian). src/remote/{protocol,server}.rs; loopback test family_b_binary_handshake_over_loopback. - DOS client (done, runtime-verified): crusty-backup/src/net_hello.c (NETHELLO.EXE) — WATT-32 sock_init + BSD socket/connect/ send/recv, sends the binary Hello to <agent-ip>:7341, prints the reply. Builds clean under DJGPP (make -C crusty-backup net). - Runtime check (done, 2026-06-24, emulator): booted the FreeDOS 1.4 image headless in qemu-system-i386 (-device ne2k_isa,iobase=0x300, irq=3 + -netdev user SLiRP), loaded the Crynwr NE2000.COM packet driver (NE2000 0x60 3 0x300), and ran NETHELLO 10.0.2.2 7341 against a host rb-cli serve. Handshake round-tripped: client printed "Connected. Agent protocol v2, capabilities 0x0001 [file]" (exit 0), host logged "Family-B client connected (version 2, caps 0x0000)". WATTCP.CFG UX: a one-line my_ip = dhcp is enough — SLiRP's DHCP leases the guest and 10.0.2.2 reaches the host listener. DOS gotcha: FreeCOM doesn't honor 2>/2>&1 (parses the 2 as an argv) — pass the port explicitly and use single > redirects, or all output goes to stderr/console. CWSDPMI.EXE must sit next to NETHELLO.EXE (real FreeDOS has no DPMI host).
  • 7b — Chunk protocol + container. Done (2026-06-25). The Family-B chunk-PUT wire framing (the §2a/§2d chunk shape) + the host receiver. src/remote/protocol.rs: PutHeader/MemberHeader/ChunkHeader + read_put_header/read_member_header/read_chunk_header and the stop-and-go ack/result. src/remote/server.rs: handle_family_b reads the post-handshake frame (EOF = bare handshake → clean close; RBKP → PUT) and receive_put streams each member's chunks to a temp folder (CRC re-checked, fsync-before-ack) then assembles the frozen .cbk with the existing pack_folder_to_cbk writer (atomic .tmp + rename) — no second format. CAP_FAMILY_B advertised. Loopback test family_b_chunk_put_assembles_cbk_over_loopback (multi-chunk members). (An interim standalone NETPUT.EXE that read a saved backup folder and shipped its files was the first cut; it was the wrong shape — file-level and it needed a spare local DOS disk to stage the folder — so it was replaced by the integrated block-level producer in 7c. The host/protocol here are unchanged and reused.)
  • 7c — Whole-disk backup over the wire (block-level, integrated). Done — qemu-verified (2026-06-25). Networked backup is baked into CRUSTYBK BACKUP: a rb://HOST[:PORT]/NAME destination images the disk block-by-block over int13h and streams it straight to the agent — no intermediate folder on the DOS box, so a machine with no spare storage can back itself up. Each partition is smart-compacted and compressed into independent 1 MiB gzip-member spans (cbnet.c's span streamer, zlib deflateInit2 gzip), each shipped as a chunk {src_offset, len, crc32} stop-and-go; mbr.bin + metadata.json ride as small Raw members (metadata last, carrying the per-partition gz CRC accumulated mid-stream). The host's receive_put concatenates the span chunks into a valid multi-member partition-N.gz and packs the .cbk — unchanged from 7b. CRUSTYBK.EXE now links WATT-32 (alongside zlib + lz4). Verified on qemu (FreeDOS 1.4, NE2000 + SLiRP): CRUSTYBK BACKUP rb://10.0.2.2:7341/MYDISK 81 on a FAT16 disk → host MYDISK.cbk; a 3.5 MiB used-data disk streamed as 4 gzip-member spans that the host assembled into one multi-member .gz; rb-cli restore MYDISK.cbk rebuilt a disk with files byte-identical to the source. (Primaries + FAT/NTFS, gzip only; extended/logical, /DEFRAG, and LZ4 over the wire compose later. Materialize-at-finalize is the host's pack_folder_to_cbk; desktop restore is unchanged — §2b option a.)
  • 7d — Resume. Done — qemu-verified (2026-06-25). A killed transfer resumes instead of restarting. The PUT header carries the §4 source fingerprint; the daemon replies with a resume map (RBKR) of per-member committed chunks. receive_put assembles into a persistent per-container staging dir guarded by a durable journal.json — each Gz chunk's bytes are fsynced before the journal records it committed (the ddrescue-mapfile pattern), and a reconnect with a matching fingerprint truncates each member to its last committed chunk and resumes (a mismatched fingerprint discards the stale staging and starts fresh). The producer (cbnet) skips committed spans by seeking the source to committed·CBNET_SPAN, never recompressing. The daemon owns the gz checksum (the rebooted client can't recompute a CRC across a resume) and fills it at finalize. Loopback test family_b_chunk_put_resumes_after_drop; qemu proof: a backup dropped after 2 committed spans (RB_SERVE_TEST_DROP_ AFTER_CHUNKS=2), reconnected, was told to resume at span 2, finished, and rb-cli restore rebuilt a byte-identical disk.
  • 7e — Restore over wire. Done — qemu-verified (2026-06-25). Closes the producer/consumer loop: CRUSTYBK RESTORE rb://HOST/NAME 81 /Y pulls a .cbk back from the agent and rebuilds the disk over int13h, no local folder. A GET op (RBKG) joins PUT under one post-handshake dispatcher (read_family_b_op); the daemon (serve_get) advertises the container's members and streams each on request — a *.gz member as its raw gzip bytes (the client inflates), a Raw member decompressed — reusing the native .cbk readers (CbkPayloadReader / cbk_member_content_reader), no new format. cb-dos (cbnet GET client + cmd_netrestore) streams each partition gz straight to the disk (manual multi-member inflateReset), same-size: zero-pad to original, rebuild the EBR chain, write the MBR verbatim. Verified on qemu: one boot backed disk 0x81 up over the wire then restored it over the wire to a blank 0x82 — files byte-identical. Loopback test family_b_get_serves_cbk_members_over_loopback. (Same-size only — target ≥ original; resize-over-the-wire is a follow-up, since the socket is forward-only and the local peek-then-resize two-pass doesn't apply yet. gzip members, FAT + NTFS, primaries + logicals.)
  • 7f — Manifest + idempotency. Done — qemu-verified (2026-06-25). Emit a per-FAT-partition manifest-N.json sidecar (§5a files[] + the §5d system boot-fingerprint block) and prove backup→restore→backup is a no-op. Idempotency is structural, not replayed: cb-dos restore is block-level (write_lba only), so every dir entry — mtime, attribs, the archive bit — round-trips verbatim inside the image; re-building the manifest from the restored disk yields the byte-identical document, so the §5c int-21h attribute replay a file-level tool would need is unnecessary here. The manifest rides the folder / .cbk / network PUT as an ordinary Raw member. Also bundled DOSLFN on the boot media so the non-8.3 member names work on a bare DOS host (the FreeDOS kernel has no LFN API of its own).
  • 7g — Boot section + swap exclusion. Done — qemu-verified (2026-06-25). Deepened the §5d system block with each DOS sysfile's content hash (CRC32, FAT-chain order) -- a stable idempotency fingerprint that round-trips a block-level restore. Added Level-1 swap zeroing (§6c): new cbswap.{c,h} allowlists swap/page files (386SPART.PAR / WIN386.SWP / PAGEFILE.SYS / HIBERFIL.SYS / SWAPPER.DAT; never DBLSPACE/DRVSPACE/STACVOL) by name+location+attribs and marks their cluster chains; the shared compaction (local + net) zeros that content while keeping the allocation, the manifest flags them volatile/content:zeroed, and /KEEPSWAP opts out.
  • 7h — Incremental backup. Done — qemu-verified (2026-06-25). Two parts. 7h(a): host-side change detection + the §5d bootability-change flag — a networked PUT over a prior NAME.cbk logs which partitions are unchanged / changed and whether the boot chain differs (src/remote/manifest.rs diff + server.rs::compare_to_prior_cbk). 7h(b): opt-in whole-disk skip (/INCREMENTAL) — if the §4 disk fingerprint matches the prior backup's (recorded in a <name>.fp sidecar), the daemon replies skip in the RBKR reply and cb-dos sends nothing; the prior .cbk stands. Opt-in, so a §4c fingerprint false-negative can never silently drop a real change. (Optional refinement: per-partition skip — per-partition fingerprints → a partition skip-map → copy unchanged partition members from the prior .cbk — for multi-partition disks where only some partitions change.)
  • [~] 7i (optional) — Level-2 swap deallocation; desktop reads .cbk directly. Desktop swap parity DONE (§6e, 2026-06-25) — the Rust backup now Level-1 excludes swap too (--keep-swap / GUI checkbox). Remaining: Level-2 dealloc (free the chain so resize-down shrinks) + single-file-CHD swap exclusion.

Progress log

  • 2026-06-25 — Phase 7h(b) — opt-in incremental whole-disk skip, qemu-verified. The streaming-skip half of incremental: a networked backup of an unchanged disk now skips the whole transfer instead of re-imaging it. Opt-in via /INCREMENTAL (without it, every backup is full — safe by default), so a §4c fingerprint false-negative can never silently drop a real change. Wire: the PUT header gains a flags byte (bit0 = incremental requested) and the daemon's RBKR resume-map reply gains a flags byte (bit0 = skip). On an incremental PUT, if the daemon holds a prior NAME.cbk and a recorded fingerprint sidecar (<name>.fp) equal to the new header fingerprint and no partial staging is in flight, it replies skip=1 + the result frame and returns; cb-dos (cbnet_skip() → cbnet_finish) sends no members and the prior .cbk stands. The daemon writes <name>.fp on every completed PUT. Reuses the existing §4 disk-wide fingerprint as the gate (per-partition skip is a later refinement). cb-dos: cbnet_start(...,incremental), cmd_netbackup short-circuit, /INCREMENTAL parsed in cmd_backup (rb:// only). Verified on FreeDOS/qemu: three /INCREMENTAL backups of one disk — boot 1 (no prior) full PUT + writes MYDISK.fp; boot 2 (unchanged) logs "source unchanged … skipping, prior MYDISK.cbk stands" and streams nothing; boot 3 (after adding SECOND.TXT) the fingerprint differs → full PUT. Protocol round-trip unit tests (PUT incremental flag, RBKR skip flag) + family_b loopback + clippy (incl. --features remote) green; cb-dos builds clean. 7h complete.
  • 2026-06-25 — Phase 7h(a) — host-side incremental change detection, qemu-verified. First slice of incremental (§5b/§5d): when a networked PUT arrives for a NAME that already has a NAME.cbk, the daemon diffs the freshly staged manifest-N.json against the prior backup's and logs which partitions are unchanged + whether the boot chain changed — the cheap-gate / bootability-flag half, on which a later streaming-skip optimization builds. Reuses the 7f/7g manifests, so no wire or cb-dos change. New src/remote/manifest.rs parses + diffs a partition (§5b rsync-tier: size+mtime+attr+start-cluster per file; the whole system block for the §5d bootability flag); swap files (volatile) are excluded from the change counter (§6b). server.rs::compare_to_prior_cbk (testable) reads the prior members via the existing cbk readers; report_incremental logs at PUT finalize before the pack renames over the prior .cbk. Verified on FreeDOS/qemu: three sequential networked backups of one disk — first creates MYDISK.cbk; the second (unchanged) logs "partition 0: unchanged => identical to prior backup"; a third after adding NEWFILE.TXT logs "partition 0: 1 file(s) changed". Unit tests (identical / changed-file / changed-sysfile-hash / placeholder + a .cbk round-trip glue test) + family_b loopback + clippy (incl. --features remote) green. Remaining for 7h: the streaming-skip optimization (per-partition fingerprints → daemon skip-map → cb-dos skips unchanged partitions → host copies them from the prior .cbk).
  • 2026-06-25 — Desktop swap parity (§6e / part of 7i) — shipped. Brought 7g's Level-1 swap exclusion to the Rust desktop backup (CLAUDE.md GUI/CLI parity). CompactFatReader::new_excluding_swap (src/fs/fat.rs) runs the same strict allowlist (386SPART.PAR / WIN386.SWP / PAGEFILE.SYS / HIBERFIL.SYS / SWAPPER.DAT, mirroring cbswap.c) over root / \WINDOWS / \OS2\SYSTEM, collecting the swap files' cluster chains; load_cluster emits zeros for them, keeping the allocation. Keyed by old cluster number, so it works in both plain and defrag compaction (cb-dos's by-position mask had to skip /DEFRAG; the desktop doesn't). Threaded as keep_swap: bool through BackupConfig → compact_partition_reader / defrag_fat_partition_reader / packed_partition_reader_padded, surfaced as an rb-cli --keep-swap flag + a GUI "Keep swap/page-file content" checkbox (default off = exclude). Unit test proves a packed image zeros WIN386.SWP full-size while a normal file + plain new are byte-preserved; lib + clippy -D warnings green. single-file-CHD still images swap verbatim (a documented follow-up); the remaining 7i piece is Level-2 dealloc.
  • 2026-06-25 — Phase 7g complete — boot section deepening + swap exclusion, qemu-verified (local + network). Two parts. (a) Boot deepening (§5d): each DOS system file in the manifest system block now carries a hash (CRC32 over its content, first size bytes in FAT chain order) alongside the size/mtime/attr/first_cluster/contiguity 7f recorded -- a pure live read, deterministic across a block-level restore (data + chain round-trip verbatim), so it stays a stable idempotency fingerprint. (b) Level-1 swap exclusion (§6c): new cbswap.{c,h} identifies swap/page files by a strict allowlist (exact name + expected location + expected attribs -- 386SPART.PAR, WIN386.SWP, PAGEFILE.SYS, HIBERFIL.SYS, SWAPPER.DAT; DBLSPACE/DRVSPACE/ STACVOL deliberately absent, those ARE the filesystem) and marks their cluster chains. The shared FAT compaction (local backup_fat_partition + networked netstream_fat_partition) now zeros that content alongside free clusters -- the allocation is kept (the file survives full-size, the OS reinitializes swap on boot), gzip crushes the zeros. Every exclusion is logged (never silent). The manifest flags the same files volatile:true (so the §5b change counter ignores them) and, unless --keep-swap, content:zeroed, sharing the one cbswap_is_swap() predicate so flags and payload never disagree; /KEEPSWAP images swap verbatim. Skipped under /DEFRAG (the repacker relocates clusters). Verified on FreeDOS/qemu: a FAT16 disk (IO.SYS/MSDOS.SYS/COMMAND.COM with distinct content, a nested tree, an LFN file, a hidden 386SPART.PAR + a WIN386.SWP) → BACKUP C:\BK1 81 → RESTORE C:\BK1 82 /Y → BACKUP C:\BK2 82: the restored target's swap files are full-size but all-zero (0 non-zero bytes) while IO.SYS is byte-identical to source, and BK1 == BK2 manifest byte-identical (sysfile hashes + swap flags round-trip). BACKUP C:\BK3 81 /KEEPSWAP images swap verbatim (no exclusions, manifest drops the two content:zeroed, 42 bytes shorter). Network half: BACKUP rb://…/MYDISK 81 PUT (4 members) assembled a .cbk whose manifest-0.json is byte-identical to the local BK1 manifest -- the shared compaction behaves the same over the wire. Lib (2105) + cbk round-trip + clippy green. Next: optional 7h (incremental + the boot-change flag) / 7i (Level-2 dealloc; desktop swap parity per §6e).
  • 2026-06-25 — Phase 7f complete — per-partition file manifest + idempotency, qemu-verified. Backup now emits a manifest-N.json sidecar per FAT partition: a depth-first files[] list (path / size / mtime / attr / start_cluster, dirs flagged) plus a system boot-fingerprint block (MBR boot-code CRC, the partition's reserved/boot-sectors CRC, and the DOS sysfiles IO.SYS/MSDOS.SYS/COMMAND.COM/… with size/mtime/attr/first_cluster/contiguity). New cbmanifest.{c,h} walks the live source read-only via the cbbrowse FAT reader (extended to carry the dir write-time + date); cmd_backup writes it locally, cmd_netbackup ships it as a Raw member (the PUT member count grew by one per FAT partition), so it rides the folder / .cbk / network PUT for free — pack_folder_to_cbk and receive_put already carry arbitrary members. FAT only (NTFS has no on-DOS directory reader). Idempotency turned out to be structural: cb-dos restore is block-level (write_lba), so dir entries — mtime, attribs, the archive bit — round-trip verbatim in the image; a same-size backup→restore→backup is a no-op and re-building the manifest yields the byte-identical document (the §5b gate), making the §5c int-21h attribute replay unnecessary. Also bundled DOSLFN on the boot media (vendored adoxa/doslfn v0.42 doslfn.com + attribution, shipped at the media root via mkmedia.sh and auto-loaded in cbdos-autoexec.bat): the backup-folder names are not 8.3-clean (metadata.json / partition-N.gz / manifest-N.json) and the FreeDOS kernel has no LFN API, so a local-folder backup/restore on a bare DOS host needs the LFN TSR (the .cbk + network paths don't — those names never become DOS files). Verified on qemu (FreeDOS 1.4): a FAT32 disk (system files, nested dirs, an LFN file) → BACKUP 81 → RESTORE 82 /Y → BACKUP 82 produced a byte-identical manifest-0.json (vendored DOSLFN wrote every non-8.3 name cleanly), and a BACKUP rb://…/MYDISK 81 PUT (4 members) assembled a MYDISK.cbk whose unpacked manifest-0.json is byte-identical to the local one. Lib (2093) + cbk pack/unpack round-trip (now covers a manifest member) + clippy green. Next: 7g (boot section deepening + swap exclusion, §5d/§6).
  • 2026-06-25 — Phase 7e complete — restore over the wire (loop closed), qemu-verified. A vintage box with a blank disk now pulls a .cbk back from the agent and rebuilds the disk over the network — the mirror of backup rb://..., no intermediate folder either side. Wire (src/remote/protocol.rs): a GET op (RBKG) joins PUT under one post-handshake dispatcher (read_family_b_op/FamilyBOp). GET-open advertises the container's members; the client fetches each by name, streamed as big-endian length-delimited frames; a *.gz member is served as its raw gzip bytes (client inflates), a Raw member decompressed. Host (serve_get) reuses the native .cbk readers (read_cbk_index, CbkPayloadReader, cbk_member_content_reader) — read-only, no new format. DOS (cbnet.{h,c} GET client + cmd_restore.c's cmd_netrestore): cbnet gained cbnet_start_get / cbnet_get_raw / a streaming cbnet_get_member_* that inflates a multi-member gzip from the framed socket (manual inflateReset across members); restore rb://... GETs metadata.json + mbr.bin, then streams each partition gz straight to int13h same-size (zero-pad to original), rebuilds the EBR chain, writes the MBR verbatim. Reuses the metadata scanner + part_t + write_ebr_chain; the rb:// parser moved to cbnet_parse_url (shared by backup + restore). Verified on qemu (FreeDOS 1.4 + NE2000 + SLiRP): one boot ran CRUSTYBK BACKUP rb://…/MYDISK 81 then CRUSTYBK RESTORE rb://…/MYDISK 82 /Y — disk 0x81 backed up over the wire, restored over the wire to a blank 0x82, files byte-identical (a byte-exact restore of a bootable disk boots). Lib (2093) + the 4 family_b loopback tests (incl. family_b_get_serves_cbk_members_over_loopback) + clippy green. Deferred: resize-over-the-wire (the socket is forward-only; the local peek-then-resize two-pass doesn't apply). Next: 7f (manifest + idempotency, §5).
  • 2026-06-25 — Phase 7d complete — resumable networked backup, qemu-verified. A killed transfer now resumes instead of restarting from zero. Host (src/remote/{protocol,server}.rs): the PUT header gained the §4 source fingerprint, and the daemon answers with a resume map (RBKR) of per-member committed-chunk counts. receive_put was reworked from a throwaway temp dir into a persistent per-container staging dir + journal.json: each Gz chunk's bytes are fsynced, then the journal records it committed (atomic rename), then the ack goes out — so the resume cursor only ever advances past durable data (§3a). A reconnect loads the journal, and if the fingerprint matches, truncates each member to committed_bytes (discarding a half-written trailing chunk) and resumes; a mismatch discards the stale staging. The daemon owns each partition's gz checksum — it fills metadata.json at finalize (the rebooted client can't CRC a member it streamed across two sessions), packs with pack_folder_to_cbk, and clears the staging. DOS (cbnet.{h,c} + cmd_backup.c): cmd_netbackup computes the fingerprint (CRC over MBR + disk size + each partition's boot sector + FAT — the allocation-map workhorse); cbnet reads the resume map and skips committed spans by seeking the source to committed·CBNET_SPAN; the producer dropped its gz-CRC accumulation (ships a placeholder) and Raw members are re-sent fresh. A test-only RB_SERVE_TEST_DROP_AFTER_CHUNKS knob makes the drop deterministic. Verified on qemu (FreeDOS 1.4 + NE2000 + SLiRP): RUN 1 dropped after 2 of 4 committed spans (journal.json = committed_chunks:2), RUN 2 reconnected — "PUT resuming (fingerprint 0xd4be9d7d matches)" — resumed at span 2 and completed; rb-cli restore rebuilt a disk with both files byte-identical and the agent-filled checksum (2a6324cc) equal to the real partition gz CRC. Lib (2093) + loopback resume test + clippy green. Next: 7e (restore over the wire) or 7f (manifest + idempotency).
  • 2026-06-25 — Phase 7c complete — networked backup baked into CRUSTYBK, block-level, qemu-verified. Reworked the producer after the first 7b cut went the wrong way: the interim standalone NETPUT.EXE read a saved backup folder off a DOS disk and shipped its files — exactly the "folder-of-files on the wire" §0 rules out (file-level, and it needed a spare local disk to stage the folder first). Replaced it with the right shape: CRUSTYBK BACKUP rb://HOST[:PORT]/NAME images the disk block-by-block over int13h and streams it straight to the agent, no intermediate folder. New cbnet.{h,c} (WATT-32 sockets + the chunk-PUT framing + a zlib gzip span streamer): each partition is smart-compacted and compressed into independent 1 MiB gzip-member spans, each sent as a chunk {src_offset, len, crc32} stop-and-go (memory bounded to one span); mbr.bin + metadata.json ride as Raw members (metadata last, with the per-partition gz CRC accumulated during the stream). cmd_backup.c detects the rb:// dest, pre-scans the MBR for selected primary FAT/NTFS partitions, and build_metadata is now shared by the local-folder and networked paths so the JSON never drifts. CRUSTYBK.EXE links WATT-32 now (the Makefile + docker/cb-dos.Dockerfile fetch it alongside zlib/lz4). The host side (receive_put, pack_folder_to_cbk) is unchanged — it already concatenates multi-chunk members and assembles the .cbk. Verified on qemu (FreeDOS 1.4 + NE2000 + SLiRP, a live rb-cli serve --root): a 3.5 MiB used-data FAT16 disk streamed as 4 gzip-member spans, the host assembled a valid multi-member MYDISK.cbk, and rb-cli restore rebuilt a disk with BIG.BIN/HELLO.TXT byte-identical to the source. Lib suite (2093) + the loopback test (now multi-chunk) green; clippy clean. Deferred (compose later): extended/logical partitions, /DEFRAG, LZ4, and 7d's per-span resume .idx. Next: 7d (resume) or 7e (restore over the wire).
  • 2026-06-25 — Phase 7b complete — the chunk PUT round-trips on real FreeDOS in qemu. Built the Family-B chunk-PUT protocol (the §2a/§2d chunk shape on the wire) and proved it end-to-end. DOS producer: NETPUT.EXE (crusty-backup/src/net_put.c, make net — WATT-32, no zlib) enumerates a CRUSTYBK BACKUP folder, does the 7a handshake, then streams it as one ordered member stream: per member an RBKM descriptor {kind, name, chunk_count}, per chunk a {src_offset, len, crc32} header + payload, stop-and-go (waits for the daemon's ack per chunk). One chunk per member (cb-dos writes a single gzip member per partition). Host: src/remote/protocol.rs gained the PUT framing (read_put_header/read_member_header/read_chunk_header + the client-side writers for the loopback test); src/remote/server.rs's handle_family_b now reads the post-handshake frame — EOF (a bare NETHELLO) closes cleanly, an RBKP PUT goes to receive_put, which stages each member to a temp folder (CRC re-checked, fsync-before-ack) and assembles the frozen .cbk via the existing pack_folder_to_cbk (atomic .tmp + rename) — reuse, not a second format. CAP_FAMILY_B now advertised. Verified on qemu (FreeDOS 1.4, NE2000 + SLiRP): a FAT16 disk imaged on DOS → NETPUT 10.0.2.2 7341 C:\BK MYDISK → host MYDISK.cbk byte-identical to a desktop rb-cli cbk pack of the mcopy'd folder, and rb-cli restore MYDISK.cbk rebuilt a disk with both files byte-identical to the source. Lib suite (2093) + the loopback test family_b_chunk_put_assembles_cbk_over_loopback green; clippy clean. Next: 7c (whole-folder backup straight off the live disk over the wire) then 7d (resume: per-span chunks + fsync-before-record + the incremental .idx).
  • 2026-06-24 — .cbk is now fully first-class — native read + edit (§2e). Two steps. (1) Native read: a CbkTempReader arm in open_read_dispatch (source_reader.rs) + is_container_path + a BrowseSession::open_image arm
    • "cbk" in DISK_IMAGE_EXTS — so inspect, ls/get, fsck, GUI Inspect, and restore all treat a .cbk like a flat disk image (it reconstructs the disk into a temp file behind the trait; no user-visible extract). (2) Edit: resolve_partition_rw_forced materializes the .cbk, edits the partition via the backup-folder RW path, and RwCommit::Cbk recompresses + repacks over the original on commit; gzip added to the editable-codec whitelist. Verified: inspect/ls/get read a cb-dos .cbk natively, and put → ls/get → restore round-trips an edit with both files intact. Lib suite (2086) + clippy green. The user's whole .cbk ask (read/inspect/browse/extract/restore like native, edit with extra legwork) is delivered.
  • 2026-06-24 — .cbk container frozen + shipped desktop-first (§2d). Built src/rbformats/cbk.rs (pack_folder_to_cbk / materialize_cbk_to_folder), the rb-cli cbk pack|unpack verb, and taught rb-cli restore to read a .cbk directly (materialize a temp folder, then restore — §2b option a). The on-disk format is now frozen at v1 (RBKC chunk headers = gzip members, RBKI trailer index, RBKF footer; big-endian) so the future DOS Family-B producer (7b) emits the same bytes. Verified: a cb-dos backup folder → .cbk (1255 B vs 1882 B folder) → rb-cli restore MYDISK.cbk rebuilds a valid disk (FS mounts, file intact), and cbk unpack round-trips every folder file byte-for-byte. Desktop packer emits one chunk/member; the network producer will append many (per-span) for resume — both valid. The §3 .idx sidecar layers on top (incremental form of the same chunk-offset table). 3 unit tests; clippy clean. Bridges 7b: the container exists, so 7b is now just the wire framing + incremental index.
  • 2026-06-24 — Phase 7a complete — handshake round-trips on real FreeDOS. Closed the only open 7a item (the runtime check) on a headless Linux box, no 86Box/DOSBox-X GUI needed: qemu-system-i386 booting the FreeDOS 1.4 image with an emulated NE2000 (-device ne2k_isa,iobase=0x300,irq=3) + SLiRP usermode net (-netdev user). Pulled the Crynwr NE2000.COM out of the FreeDOS image's own crynwr.zip, loaded it on int 0x60, and ran NETHELLO 10.0.2.2 7341 against rb-cli serve. Both ends confirmed: client "Connected. Agent protocol v2, capabilities 0x0001 [file]" (exit 0), host "Family-B client connected (version 2, caps 0x0000)". The binary RBK0 Family-B handshake is now proven over a true DOS TCP stack, not just the loopback unit test. Two lessons worth keeping: (1) FreeCOM mis-parses 2> / 2>&1 — it treats the 2 as a program argument (which silently sent NETHELLO to port 2), so pass the port explicitly and stick to single > redirects; (2) my_ip = dhcp in WATTCP.CFG is the whole network config — SLiRP's DHCP does the rest. CWSDPMI.EXE must travel next to the exe (real FreeDOS provides no DPMI host; only the disk-spike's DOSBox-X faked one). Next: 7b (the .cbk chunk protocol + container). The transport is unblocked.
  • 2026-06-21 — Phase 7a started (socket + handshake hello-world). Resolved the long-open TCP-stack question: WATT-32 (DJGPP-native BSD sockets, prebuilt libwatt.a, BSD/AGPL-compatible) over mTCP/Watcom (§1b). Host: rb-cli serve gained a binary Family-B handshake path that coexists with the JSON Family-F handshake on one port (peek the b"RBK0" magic) — read_handshake / write_binary_hello in src/remote/protocol.rs, handle_family_b in server.rs; headless test family_b_binary_handshake_over_loopback. DOS client: crusty-backup/src/net_hello.c (NETHELLO.EXE) connects to an agent IP and exchanges the binary Hello — compiles clean under DJGPP via the new make net target + net/fetch-watt32.sh. Runtime check on real NIC / DOSBox-X is the next step (the only remaining 7a item). The chunk/.cbk data protocol (7b+) is untouched.
  • 2026-06-03 — Scoping discussion captured into this doc. Transport: TCP/IP over the Crynwr packet driver (not raw L2 / serial / parallel) — chosen for TCP's free reliability + a plain-socket, unprivileged, SLiRP-friendly host (vs. pcap/Npcap/BPF/root for raw L2). mTCP (GPLv3, AGPL-OK) is the reference / borrow candidate; borrow-vs-port-to-DJGPP is open. Host = rb-cli serve (Family B of the unified daemon; was net-serve). Designed the chunked .cbk container (wire framing = on-disk artifact = ddrescue-style resume log) with chunks = independent source-keyed gzip members (MultiGzDecoder on the desktop; producer resumes by source seek, no recompress). Resume via fsync-before-record + truncate-to-last-committed. Disk-state fingerprint = geometry+MBR/ptable CRC+serials+allocation-bitmap CRC (no full-disk CRC; bitmap CRC subsumes free-space, doubles as the incremental index). Change detection = rsync-tier size+mtime+archive bit via a file manifest sidecar (block image stays the payload); archive bit is round-tripped by restore for backup↔restore idempotency; manifest also fingerprints/round-trips boot (MBR code, VBR, IO.SYS/MSDOS.SYS/COMMAND.COM). Swap files (386SPART.PAR / WIN386.SWP / pagefile.sys / hiberfil.sys / SWAPPER.DAT) excluded from the counter always and zeroed in the payload by default (--keep-swap); strict allowlist — never DBLSPACE/DRVSPACE/STACVOL. Prior-art map recorded. No code yet.