https://gitlab.synchro.net/main/sbbs/-/commit/7471d1b776cb2d7ea274a41e
Added Files:
docs/zmodem_comparison.md
Log Message:
docs: add ZMODEM implementation comparison (sexyz vs lrzsz vs rzsz)

A performance, scalability, and robustness analysis of Synchronet's ZMODEM (sexyz + the shared zmodem.c, which SyncTERM also uses) benchmarked against lrzsz 0.12.21rc and Chuck Forsberg's final rzsz 3.73 (modern branch).

Headline findings, all empirically measured:

- sexyz's sender is ~25x slower than lrzsz's lsz on a fast link (~8 vs
~204 MB/s), from two stacked overheads isolated by linking the real
zmodem.o behind two send paths: zmodem.c's per-byte callback + escape +
CRC design caps any sender at ~26 MB/s (shared, so SyncTERM inherits it),
and sexyz.c's ring-buffer + output-thread drops that a further ~3x to
~8 MB/s (sexyz-only; a futex storm + ~84-byte writes). SyncTERM avoids
the second layer, so its upload runs ~26 MB/s.
- A windowed (-w) send hangs the instant the file offset crosses 2 GiB:
zmodem.c narrows transmit-window/ACK positions to signed int32_t. Shared
with SyncTERM. The data path (fseeko/int64) is otherwise 64-bit clean;
4 GiB is a hard protocol ceiling (32-bit wire position field).
- sexyz's receiver is fine past 2 GB; Forsberg's sender runs ~92 MB/s but
its receiver won't complete headlessly (serial-tty assumptions).

Includes a conditions matrix (block size, CRC, latency, bandwidth
asymmetry, injected errors) and per-finding attribution to sexyz.c vs the shared zmodem.c vs SyncTERM's term.c. Relates to the sexyz-throughput and 2-GiB-window GitLab issues.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/f29244a40663ce7f99d3e4e6
Modified Files:
docs/zmodem_comparison.md
Log Message:
docs: note the sexyz SINGLE_THREADED build is not a throughput fix

Document that sexyz.c:84 still carries a source-level
`#define SINGLE_THREADED FALSE` (no makefile toggle), but its
single-threaded send_byte (sexyz.c:682) writes one byte per write()
syscall via sendbuf() Ä trading the two-thread futex storm for a
write-syscall storm, generally worse. Add a three-way send-architecture
table (threaded / single-threaded / buffered) and a recommendation caveat:
the fix is output buffering (flush per subpacket, as SyncTERM's term.c
does), not thread count.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/5c0e802235bfc797afbdadb1
Modified Files:
docs/zmodem_comparison.md
Log Message:
docs/zmodem: correct throughput magnitudes to steady-state; version-tag

The initial throughput figures were measured on 32 MB transfers whose
short duration was dominated by per-transfer startup, which deflated and distorted the rates. Re-measured at 256 MB steady-state:

- sexyz sender was ~18x (not ~25x) slower than lsz: ~11 vs ~204 MB/s.
- The two layers are ~7.5x sexyz.c (ring per-byte, 85->11, now fixed by
the send-batching change -> ~66 MB/s) and ~2.4x zmodem.c (per-byte send
design, 204->85, shared with SyncTERM), not the ~3x/~8x quoted before.
- The SyncTERM-model (ztx_buf) sender runs ~85 MB/s, not ~26; sexyz
receiver ~130; Forsberg sender ~107.

The mechanism findings (futex storm, ~84-byte writes, ring-per-byte) were unaffected -- only the magnitudes changed.

Also: tag every number to a component version (sexyz.c revision 3.3->3.4, zmodem.c rev 2.2->2.3) since Synchronet's side is actively changing, and
mark the recommendations with status (Fix A and the 2 GB fix done, Fix B
and the adaptive-block-length model pending).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/9c73c787d0f036e0eff35085
Modified Files:
docs/zmodem_comparison.md
Log Message:
docs/zmodem: record that the send-batching fixes were reverted

Multi-run error-recovery testing showed both batching prototypes regress
error recovery under heavy injected corruption: the shipped per-byte
sender passes 3/3 (~50 s), the Fix A prototype (batch spans into the ring, sexyz.c) timed out 0/3, and the Fix B prototype (send_buf span callback, zmodem.c) hard-failed 1/3. Root cause: batching a whole subpacket before
one blocking flush starves the back-channel during retransmit storms (the sender is stuck in the flush and can't service ZRPOS), stalling the
transfer; the original per-byte-to-ring keeps the pipeline draining and
stays responsive.

Both prototypes were reverted. The only shipped code change remains the
2 GB signed-position fix in zmodem.c (rev 2.3, GitLab #1196); sexyz.c is unchanged (stays 3.3). The ~66/~88 MB/s figures are retained in the doc, relabeled as reverted prototypes, because the root-cause analysis (ring per-byte -> futex storm / 84-byte writes) is still valid and the eventual
fix must service the back-channel while sending.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/142d6aa509d91e348bd28dd2
Modified Files:
docs/zmodem_comparison.md src/bench/zmodem/README.md
Log Message:
docs/zmodem, bench: correct SyncTERM/localhost/recall claims (per Deuce)

Deuce (SyncTERM's author) corrected three framings; retained with caveats:

- "SyncTERM ~85 MB/s (faithful ztx_buf model)" is wrong. ztx_buf models the
SHAPE of SyncTERM's send path but under-measures its throughput ~3x
(Python-relay harness + poll-per-subpacket). SyncTERM's speed is a
deliberate BDP / socket-buffer choice (~75% of a 1Gb LAN), not an emergent
send-path fact. Relabelled ztx_buf as a "buffered floor (harness)" and
flagged ~85 as a localhost floor, not SyncTERM, throughout.
- The "recall stale bytes / abort-aware purge on ZRPOS" root-cause framing
(bench README) was a wrong garden path: you can't recall bytes once they
hit the network buffers, and lrz doesn't either yet recovers. The batched-
sender failures were implementation bugs, not inherent to batching.
- These are localhost CPU microbenchmarks; below ~8ms RTT the socket buffer
dominates the send loop. Optimizing it for localhost is largely beside the
point; the send-loop rewrite pays off only on fast LANs.

Also fixed a stale "fixed" on the ring-per-byte row (the batching prototype
was reverted, not shipped).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/9627d6504cecd3c53ff66c47
Modified Files:
docs/zmodem_comparison.md src/bench/zmodem/README.md
Log Message:
docs/zmodem, bench: re-measure after Deuce's send-path work; root-cause sexyz

Every throughput figure is re-measured in one interleaved batch (256 MB,
three passes, spread under 2%) so the tables are internally consistent
instead of splicing runs from different days: lrzsz 203.9, zmodem.c
rev 2.3 buffered floor 115.8, sexyz receiving 113.3, Forsberg 96.9,
zmodem.c rev 2.2 buffered floor 91.9, sexyz sending 11.5 MB/s.

Deuce's 2026-07-24 work on the shared send path (class-table byte
classifier, slicing-by-4 CRC-32, hoisted escape mask with noinline cold
paths, buffered fcrc32) moved the buffered floor 91.9 -> 115.8, +26%,
which SyncTERM inherits. The doc's old "second, shared lever, pending"
framing is replaced with what actually shipped.

Root-cause the sexyz sender properly rather than asserting it.
send_byte() takes the ring mutex twice per byte while output_thread
hot-loops on the same mutex: 44 CPU-seconds per 256 MB against lsz's
0.95, 1,464,130 voluntary context switches against 1,402, one futex per
17 bytes with 37% contended, and ~94-byte socket writes. A full sweep
of OutbufSize, OutbufHighwaterMark and OutbufDrainTimeout is flat inside
noise, so this is not a tuning problem and should not be chased as one.

Record the prototype ladder and what each one proved. Buffering the
producer removes the entire sexyz-specific penalty -- 115.8 MB/s at 1.0 CPU-second, exactly the engine ceiling -- but six variants now fail the
3x error-injection gate that the shipped per-byte sender passes. Two
results are worth keeping: rate-capping the batched sender to the
shipped sender's own 11 MB/s still fails while the shipped sender passes
3/3, so it is not an artifact of the higher data rate; and going single-threaded with a buffered send_byte holds full speed, drops
context switches to lsz's level, and reaches 2/3, leaving blocking
sendbuf() as the one missing piece. The recommendation is now what lrz actually is: single-threaded, buffered, non-blocking output with select
on both directions.

Also note the failure modes that cost time to rediscover (a void flush
callback swallowing a failed write, a sticky error latch spinning on zmodem_send_raw ERROR, position accounting desync) and that a
single-stream localhost benchmark cannot see per-byte CPU wins above
~100 MB/s -- two of Deuce's optimizations measure zero here because his
six interleaved 1 GiB transfers are CPU-saturated and this is not.

No code changes: all prototypes were measured out-of-tree and discarded.
Refs #1195, #1196.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/3d5c82cfc2265f8f131969c1
Modified Files:
docs/zmodem_comparison.md src/bench/zmodem/README.md
Log Message:
docs/zmodem, bench: 115.8 MB/s is a harness stall, not an engine ceiling

The previous commit treated the 115.8 MB/s "buffered floor" as zmodem.c's
CPU limit and built three conclusions on it. Measuring what the senders actually consume shows that is wrong:

ztx_buf (rev 2.3 + buffered send) 115.8 MB/s 0.69 CPU-s 30% CPU
lsz 203.9 MB/s 0.98 CPU-s 77% CPU

The engine does the same 256 MB in 30% less CPU than lrzsz and sits idle
for 70% of the wall clock; extrapolated to their CPU limits it is roughly
371 MB/s against lrzsz's 261. The receiver is not the constraint either,
since the same lrz absorbs 203.9 from lsz. So after Deuce's 2026-07-24 send-path work the engine is cheaper per byte than lrzsz's inlined loop,
and 115.8 is where ztx_buf's own I/O pattern stalls -- a blocking write()
per zmodem_flush and a poll() per subpacket in data_waiting. Under
strace -w, 93% of its in-syscall wall time lands in those polls; the
exact stall mechanism is not isolated and the doc says so.

Three claims are retracted. The 115.8 figure is not an engine ceiling.
The residual gap to lrzsz is not an escape/CRC deficit, so handing
send_byte a span would shave CPU that is not being spent -- that
recommendation is downgraded from "the second lever" to "done, and no
longer the lever". And the sexyz prototypes were not pinned by the
engine: all of them ran at 30-36% CPU because they share the same blocking-flush-per-subpacket shape, so a genuinely non-blocking sender's ceiling is unmeasured and plausibly above lrzsz's 203.9. That makes the transport rewrite more attractive than the previous commit implied, and
makes the next useful measurement a non-blocking prototype rather than
more engine micro-optimization.

The bench README gains the lesson that caused this: always record CPU
alongside goodput. A goodput number with no CPU figure beside it cannot distinguish fast code from idle code, and ztx_buf needs its own I/O
pattern fixed before it can serve as a reference at all.

No code changes. Refs #1195.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/c55eef72078c25e2b7322309
Modified Files:
docs/zmodem_comparison.md src/bench/zmodem/README.md
Log Message:
docs/zmodem, bench: lrzsz uses blocking stdio; a real flush() is not the fix

Two revisions of these documents told the reader that lrzsz's sender uses non-blocking output with select() on both directions, and that supplying
sexyz with the same would close the error-recovery gap. Reading the
source shows the first claim is false, and building the thing shows the
second one does not follow.

What lsz actually does: output is putchar/fwrite into stdout
(zm.c:109-110), flushmo() is fflush(stdout) (zglobal.h:411), and there is
no O_NONBLOCK, fcntl or FIONBIO anywhere in lsz.c or zm.c. The only
select() in the sender (lsz.c:754) is the pre-handshake purge that drains
stale input before ZRQINIT, not part of the data path. Escaping is a
lookup table with fwrite() of unescaped runs (zm.c:285-320) -- both the class-table idea Deuce landed this week and the "hand send_byte a span"
idea, present since the 1990s. Its per-subpacket loop (lsz.c:2093-2120)
is ZSDATA, then fflush(stdout), then a while (rdchk(fd)) loop that drains
the back-channel to empty. The property is "everything is on the wire
before you look for a reply", with blocking I/O throughout.

That did suggest a real difference: sexyz's flush() callback is a no-op
on the socket path -- it only fflush(stdout)s, and only in stdio mode --
so zmodem_flush() returns with the bytes still sitting in the ring.
zmodem.c already has the matching drain loop and already calls
zmodem_flush() per subpacket, so sexyz was the only piece not holding up
its end. Prototype v8 supplies a flush() that waits for the ring to
drain: full speed at 115.8 MB/s, and the error gate still fails, 1/3.
Worth fixing on its own merits, but it is not the cause.

Eight prototypes in, the useful output is the exclusion list, now
recorded: it is not the data rate (rate-capped control), not the batch
size (64 B to 4 KB all fail), not dropped or latched errors, not
flush-to-wire ordering, and only partly the output thread (removing it
moves 0-1/3 to 2/3 without fixing it). The mechanism is unidentified,
and the recommendation changes accordingly -- the next step is to
instrument a failing run and find what the receiver sees after a ZRPOS
that it does not see with the per-byte sender, rather than build a ninth prototype.

No code changes. Refs #1195.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/85956c15d7e9a6d5a389f859
Modified Files:
docs/zmodem_comparison.md src/sbbs3/zmodem.c
Log Message:
docs/zmodem: the "115.8 MB/s buffered floor" was a one-second sleep

The comparison doc's headline mystery was that every sender built on
zmodem.c stalled at 115.8 MB/s on a 256 MiB localhost send while lrzsz
reached 203.9, on less CPU per byte and only 30% utilization. Successive revisions called it an engine CPU limit, then an unexplained I/O stall in
the measuring tool. It was the 1-second receive-buffer purge on the normal
send path: 1.32 s of work plus 1.00 s of sleep is 2.32 s, and 256 MiB over
2.32 s is 115.8 MB/s. Every buffered sender reported the same number
because each was paying the same fixed second.

Re-measured with that fixed and with the ZFILE retransmit behind it: sexyz 203.9 MB/s, ztx_buf 203.9, lsz 203.8, all in 1.32 s. The engine now ties lrzsz's own inlined loop, which retires the "what could a non-blocking
sender reach" question this doc kept open.

Record why the benchmark could not see it, since the blind spot outlived several rounds of analysis: the harness sends one file per run, so a
per-file cost lands once in a goodput figure built to divide fixed costs
away; the rate-capped rows refunded it from banked tokens and read as link-bound; and the CPU/context-switch diagnostics cannot see a sleeping
poll() by construction. Section 2 now says to time a batch of small files
for anything suspected of costing per file.

Bump zmodem_ver() to 2.5 for the two protocol fixes.

Found and reported by Uwe Ohse, maintainer of lrzsz, who measured sexyz at
5.9 KB/s against lrzsz's 10.2 KB/s sending five small files over a
simulated 115200 bps line, and supplied the protocol dump and strace.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/9e7bbc3092f96252da7cb44e
Modified Files:
docs/zmodem_comparison.md src/sbbs3/sexyz.c
Log Message:
Increment SEXYZ version to 3.5 for the zmodem.c 2.5 fixes

Has the ZRPOS purge and ZFILE retransmit fixes. Both live in zmodem.c,
already at 2.5; sexyz.c is otherwise unchanged. Its startup banner prints
only its own revision, with zmodem.c's appearing solely under "sexyz v",
so the two move together the way they did for the ZFIN/OO fix in
9e95decd0b (edit-3-eyes, 2026-03-06).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/8d8af7d5d090c3077ef615d5
Modified Files:
docs/zmodem_comparison.md src/bench/zmodem/README.md
Log Message:
docs/zmodem: re-baseline the throughput tables on a receiver that can
keep up

Every sender figure this document ever published was measured against
`lrz`. Section 3.5 then measured `lrz` itself and found it receives 256
MB for 1.28 CPU-seconds, with receiver CPU equal to the wall clock --
more expensive than any sender it was scoring. So the sender table was reporting `lrz`'s ceiling four times over and reading it as a four-way
tie ("every sender built on zmodem.c now runs level with lrzsz").

Deuce spotted it from the outside: switch the baseline receiver, lrz
can't handle the load. He is right about the premise. `zmrx` is the
obvious replacement at 0.94 CPU-s, but as of zmodem.c 2.7 it is no
longer the cheapest receiver -- the span work put sexyz itself at
0.55 s. So the baseline receiver is now sexyz.

Re-measured (256 MB, five passes, interleaved, medians), the senders
spread over 1.74x where they had sat inside 1.6 %:

zmtx 487.2 MB/s 0.47 CPU-s
ztx_buf 385.4 0.70
sexyz 335.3 1.00
lsz 279.9 0.96

sexyz sends 20 % faster than lsz, which the old baseline could not show.
The document says plainly that this is concurrency rather than
efficiency -- sexyz spends more CPU than lsz and wins on wall clock with
two threads -- and that the engine's efficiency claim belongs to
ztx_buf, single-threaded and cheaper than lsz on both axes.

The receive table of 3.5 had the same defect in the other direction: its
fixed `lsz` sender costs 0.96 CPU-s, so after the span fix the three
fastest receivers all read ~280 MB/s and the fix looked like it had
merely drawn level with `zmrx`. Driven by `zmtx` (0.47 s) instead, sexyz receives at 486.5 MB/s for 0.55 s -- 1.70x `zmrx`, the fastest receiver measured, and 4.2x its own pre-fix 117.2 at 24 % of the CPU. The
pre-fix rows were never affected, being the slow end either way, so the
deficit that section diagnoses stands as measured.

New section 3.0.2 states the rule both halves of this document have now
broken: the fixed end of a comparison must be cheaper in CPU than
everything at the varied end, and that must be shown rather than
assumed. The per-endpoint `wait4` CPU columns added on 2026-08-24 make
it checkable -- if the fixed end's CPU equals the wall clock, the run
measured the fixed end. The bench README carries the same lesson, and
its recipes no longer default to `lrz`.

Also recorded: the span receiver's cost depends on the sender's write granularity. Behind zmtx's 8 KB subpackets it receives for 0.55 CPU-s;
behind Forsberg's 1 KB blocks the same receiver spends 1.34 s and 256 k
context switches for the same bytes.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/77503916899457263ce9bb86
Modified Files:
docs/zmodem_comparison.md src/hash/crc32.h src/sbbs3/zmodem.c
Log Message:
zmodem: CRC received spans with crc32_update(), not a duplicate of it

The bulk receive path added in the commit that introduced the optional recv_span callback CRC'd each span with a new static inline
ucrc32_span() in crc32.h. That function was byte-for-byte the loop crc32_update() (src/hash/crc32.c:170) has always been -- slicing-by-4
for the bulk, byte-at-a-time for the tail -- differing only in argument
order. zmodem.c already includes crc32.h, so the helper it needed was
already declared in the header it was editing.

Call crc32_update() and delete ucrc32_span(). No functional change:
5 clean 256 MB receives verify byte-identical and the 3e-6 corruption
gate passes 5 of 5. Throughput is unmoved -- 484.2 MB/s against
486.5 for the inline version, receiver CPU 0.55 s either way, a
difference well inside the +/-5 % code-layout noise documented in docs/zmodem_comparison.md 3.5. A span averages 36 bytes, so trading an
inlined loop for one call per span costs nothing measurable.

The send path keeps its open-coded loop: zmodem_send_data_subpkt()
interleaves the CRC with zmodem_tx_masked() four bytes at a time, so it
needs the per-4-byte granularity and cannot call a whole-span helper.

Found by Rob Swindell, who asked whether crc32_update() did not already
do this. It did.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/218c474f6d616d63799bdac5
Modified Files:
docs/zmodem_comparison.md
Log Message:
docs/zmodem: re-measure everything against the current releases

The document had accumulated five measuring sessions between 2026-07-23
and 2026-08-25, comparing components at versions that no longer ship and carrying figures taken against a baseline now known to have been the bottleneck. Re-measure the whole thing in one sitting on one host and
rewrite around it.

Structural change: measure the full 20-pair sender x receiver matrix
rather than holding one end fixed. Fixing an end is what produced the
old sender table's four-way tie -- every row was scored against lrz,
which receives for more CPU than any sender in the table spends to send,
so 1.74x of real difference reported as 1.8%. The matrix has no such
blind spot, and printing both endpoints' CPU makes the bottleneck visible
in every cell.

All twenty pairs interoperate byte-identically and wire overhead is equal
to two decimals at equal block size, so nothing here is a compatibility problem. zmtx/zmrx 2.04 is the cheapest implementation in both
directions by roughly a factor of two -- 10.0 instructions per data byte sending and 12.1 receiving, against the next best of 23.5 either way.
sexyz's receiver is second at 24.6, where the same code cost 126.8 eight
days earlier, and its error recovery is the strongest measured: 10 of 10
in both directions where the all-lrzsz pair manages 7 of 10.

Two findings belong to zmtx/zmrx rather than to us. Its whole-file retry budget resets only on the ZRPOS path, so NAK- and timeout-driven recovery
still drains it and the sender cannot complete a transfer at or above a
5e-7 error rate. And its ESCCTL escaping omits carriage return, so no
receiver that requests escaping can download from it -- the same defect
our own send path carried until 2026-08-24.

New sections cover what the old document never measured: feature and
option support side by side, ESCCTL interoperability, and per-endpoint instruction counts for all eight endpoints.

Method notes that cost real time to learn are written down. The
corruption model seeds its PRNG with a constant, so repeated runs at one
rate are not independent samples and a low enough rate injects nothing at
all -- a rate sweep is the instrument, not repetition. Error gates are
far more load-sensitive than throughput runs: a first pass taken while a profiler was still finishing scored the same lrzsz pair 0/3 and 2/3 in
two identical invocations, where it passes in half a second on an idle
host. And the first cut of this measurement used a zmtx checkout 32
commits stale, which is why every version in section 1.1 now carries its release date.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/f8442b0765fd2eab167a8257
Modified Files:
docs/zmodem_comparison.md
Log Message:
docs/zmodem: add Forsberg's DSZ as a reference for the negotiated options

The four implementations compared here cannot settle what ZMODEM's option
bits were meant to do, because for ESC8 none of them agrees with another. Forsberg's commercial DSZ can: he wrote the protocol, and the binary still runs.

Add section 7.4 covering DSZ.EXE 1997-05-25 under DOSBox, with its COM1
bridged to TCP. It is deliberately not in the throughput tables -- 16-bit
DOS, no source, nothing to build or profile -- and appears only for the negotiated options. The base protocol interoperates perfectly: a
4096-byte random file sent by that 1997 binary arrives byte-identical at
both sexyz and zmrx.

Its ZRINIT settles ESC8. "rz -E" sets ZF0 bit 0x80, so ESC8 is the
negotiation bit for the "8th bit quoting" that DSZ.DOC describes as half
of ZMODEM-90's 7-bit mode; "rz -e" sets ESCCTL; "rz -P" sets neither.
CANRLE is set unconditionally, which matches the other half of that 7-bit
mode being present in Forsberg's free source while the quoting half never
was.

And the two implementations of ESC8 that exist do not interoperate. DSZ
quotes a high-bit byte as 0x0E followed by the byte with bit 7 cleared -- Kermit-style prefixing, and it switches to frame type 0x31, which is not
among the eight types any published zmodem.h defines -- while zmtx/zmrx
2.04 sends ZDLE and the byte XOR 0x40. Recovered by known-plaintext
capture: 0x0E occurs 4,991 times and is followed by a byte below 0x80 in
100% of them, and decoding on that rule alone recovers 92 contiguous
bytes of the known plaintext while restoring the stream to 45.6%
high-bit.

Neither encoding is wrong, because ZMODEM.DOC names ESC8 once and never
says how a high-bit byte is escaped, where ESCCTL gets a paragraph. Two implementers 29 years apart read that line and built unrelated mechanisms.

Also record the method, since the obvious route is the wrong one.
DSZ.EXE is a plain unpacked MZ image and disassembles fine, but its
escape table is not in the file -- it is built at runtime into BSS,
exactly as Forsberg's free zsendline_init() does. Running the binary
against a deliberately permissive receiver with known plaintext settled
it in three runs.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/f06bafe5ee11ad2f2056050a
Modified Files:
docs/zmodem_comparison.md
Log Message:
docs/zmodem: record option-by-option compatibility with DSZ

Section 7.4 had DSZ answering what the negotiated option bits mean, but
the only transfers behind it were a pair of defaults. In particular the
BBS download case -- sexyz sending to DSZ receiving -- had never been run
once. Run the whole option set in both directions and record it.

Everything interoperates. Sending to DSZ: default, -8/-4/-2, -o, -s,
-w4096, -e, -l, -y, -p and -n all transfer an 8 KiB random file
byte-for-byte or, where the option says not to, correctly decline to.
Receiving from DSZ: default, -m (MobyTurbo), -Q@ and -e likewise.

Two results are worth more than the tick in the box. The file-management options added this month produce the documented behaviour at a
third-party receiver: -p leaves DSZ's existing file alone and -n skips an
older source. -n only demonstrates itself once the timestamps are forced apart, because FAT's two-second granularity makes a same-minute
comparison ambiguous -- the first run of it looked like a failure and was
not. And ESCCTL works in both directions, which is the path that carried
the carriage-return defect until 2026-08-24.

DSZ's 7-bit options turn out to be receiver-driven despite their names:
"sz -E" and "sz -P" sending to sexyz both transfer identically and leave
46.9% of the wire carrying high-bit bytes, the same as with no option at
all. They request the mode when DSZ receives and impose nothing when it
sends.

State plainly what was not tested rather than letting the table imply
coverage: -m as a local size limit, the X/YMODEM modes, and -telnet.
Say too what "we do not support ESC8" actually means -- sexyz no longer
sets the bit, and it ignores a remote receiver's request rather than
refusing it, there being no way to decline a ZRINIT capability, which is
what lrzsz does as well.

And note a capability we decline unnecessarily: DSZ advertises CANRLE in
every ZRINIT and implements RLE both ways, while sexyz never advertises
it, so the compression is never used.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/6a06088bd80f6b71f9681cba
Modified Files:
docs/zmodem_comparison.md
Log Message:
docs/zmodem: sexyz no longer advertises ESC8, so stop saying it does

The document was written while the Escape8thBit option still existed and
kept describing it in the present tense in four places: the feature
table, the summary, section 7.2's analysis and section 8's findings. The option and the ZRINIT advertisement were both removed the same day the
last of those was written.

Put the sexyz half in the past tense and say what the current behaviour
is -- the bit is never set, and a remote receiver's request is ignored
rather than refused, there being no way to decline a ZRINIT capability.
The measurements stay as they were taken; only the claims about what
sexyz does today change.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/7bc6f244d87a80c113f2fbfc
Modified Files:
docs/zmodem_comparison.md src/sbbs3/zmodem.h
Log Message:
zmodem: note that ZF0 bit 0x08 is contested, and document two gaps

Audit of every sexyz option -- command-line, sexyz.ini and negotiated -- prompted by Escape8thBit having been inert for 21 years. The parsed
surface came out clean: all 43 ini keys and all 16 command-line options
reach something live, and the six knobs that should show up in the
negotiated ZRINIT all do (FullDuplex clears CANFDX, CRC32=false clears
CANFC32, EscapeCtrlChars sets ESCCTL, RecvBufSize sets the buffer field, Streaming=false clears CANOVIO and sets it to the block size).
MaxFileSize refuses an oversized file and RecvTimeout changes the
engine's per-read timeout. No second dead option.

The gaps are all in the negotiated layer, and two are worth writing down
in the feature table rather than leaving for the next person to
rediscover:

- We never send a ZSINIT, and a received one is ACKed with its flags and
Attn string discarded. lrzsz handles it both ways and DSZ sends one
when it has something to declare -- observed once, for sz -e -- while
zmtx never sends one either.
- As a receiver we ignore the sender's ZFILE management request and let
local policy decide, where lrz, zmrx and DSZ all honour it. DSZ
honours ours, so the asymmetry is ours alone.

And correct a misleading comment: we call ZF0 bit 0x08 CANCRY, as lrzsz
does, but Forsberg's own rzsz calls it CANRLE and his DSZ sets it on
every connection meaning "receiver can decode RLE". Say the bit is
contested rather than implying it is settled; we neither set nor honour
it under either reading.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/c9fb4003646dcbbaf56bb784
Modified Files:
docs/zmodem_comparison.md
Log Message:
docs/zmodem: re-measure zmtx/zmrx at 2.05, which fixes both reported defects

Deuce released 2.05 the day after this document reported two defects
against 2.04, and fixed both, plus implemented the ESC8 encoding the DSZ investigation had recovered. Re-measure everything the document claims
about zmtx/zmrx.

ESCCTL is fixed (82be5ad). CR is folded into the control class --
action = TX_ESCAPE_CONTROL | TX_ESCAPE_CR -- with the '@' conditional
kept as a separate test, which is the same shape as our own fix six days earlier. zmtx 2.05 now transfers byte-identically to lrz -e, to sexyz
-e and to its own zmrx -e, putting 5,245,010 bytes on the wire where
sexyz puts 5,245,034 and lsz 5,245,032 for the same file. The
32,400-byte shortfall, which was the unescaped CR count, is gone.

The retry budget is fixed (6e32e97). The gate at 3e-6 goes from 0 of 5
to 3 of 5 and the rate sweep now completes where 2.04 failed from 5e-7
up. Still the weakest of the three -- sexyz manages 10 of 10 and lrzsz
7 of 10 -- but no longer structurally unable to finish.

And ESC8 now interoperates (399e3b6). The commit adopts Forsberg's
encoding rather than inventing one; its comment names DSZ.EXE's
ZMODEM-90 seven-bit encoder and the code emits SO followed by the byte
with bit 7 cleared, which is what the known-plaintext capture recovered. Verified against the 1997 binary both ways on a 16 KiB file: zmtx 2.05
to DSZ rz -E and DSZ sz to zmrx -b both arrive identical. So a de facto normative ESC8 now exists, with two implementations agreeing on it,
where days ago there were two encodings and no agreement.

The features cost the sender something: 0.38 CPU-seconds per 256 MiB at
2.04, 0.48 at the 2.05 tag, 0.44 after dfa97f0 "Recover standard
transfer performance", which is measured here because that commit lands
after the tag. Still the cheapest sender measured, but the margin over
lsz narrowed from 2.5x to 2.2x.

The 2.02 and 2.04 measurements are kept throughout rather than
overwritten, since they are how the encoding was established and why the options are the way they are.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/60b70e526213285b61ae11f8
Modified Files:
docs/zmodem_comparison.md
Log Message:
docs/zmodem: zmtx/zmrx supports CRC-16, it just cannot be asked for

The feature table listed CRC-16 as "--" for zmtx/zmrx, which reads as
not supported. That is wrong in a way that matters to anyone pairing it
with a CRC-16-only peer.

want_fcs_32 is hard-coded true in zmdm.c and neither program offers a
CRC flag, so it cannot be asked to prefer 16-bit -- but the machinery is present in both the header and data paths and the selection is
can_fcs_32 && want_fcs_32, so it falls back whenever the peer omits
CANFC32. Distinguish "supported but not selectable" from "absent",
since sexyz and lsz can both force CRC-16 and zmtx/zmrx can only fall
into it.

Measured rather than read: zmtx to sexyz -o puts 4,311,016 bytes on the
wire against 4,312,065 with CRC-32, and sexyz -o to zmrx 4,310,744
against 4,311,816. A 4 MiB file is 512 subpackets at 8 KiB and two CRC
bytes saved each is about 1,024, which is what those deltas are. Both
verified byte-identical, so the fallback is real in both directions.

Also corrects a stale count in the prose above the table.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/0822029d905accebd45d3265
Modified Files:
docs/zmodem_comparison.md
Log Message:
docs/zmodem: measure what MobyTurbo is worth

Rob asked what sexyz would gain by supporting MobyTurbo. zmtx/zmrx 2.05 implements it, so it can be measured rather than estimated.

It is worth about 2.3% on compressed or otherwise incompressible data and essentially nothing on text. ZMODEM must escape seven byte values --
ZDLE, 0x10, 0x11, 0x13, 0x90, 0x91 and 0x93 -- at one extra wire byte
each, which is 7/256 = 2.73% of uniformly distributed data; MobyTurbo
drops that escaping on links known to be transparent and recovers exactly
it. Measured on 1 MiB: random data goes from +2.84% overhead to +0.49%,
while text goes from +0.09% to +0.09%. The split is the escape rate
itself -- those seven values occupy 2.746% of the random file and 0.000%
of the text one, none of them occurring in ordinary ASCII.

Three bounds on what that is worth, all recorded: it is a wire saving, so
it converts to throughput only where bandwidth binds rather than in this document's CPU-bound regime; it needs a transparent link, the same
precondition ESC8 exists to work around, inverted; and it is negotiated
by the receiver, so DSZ sz -m to sexyz puts exactly the same bytes on the
wire as without it.

Forsberg measured about the same himself. DSZ.DOC's benchmark table has Pro-YAM at 231 cps standard against 237 with MobyTurbo on a 2400 bps
link, +2.6%, within half a point of the figure here thirty-six years
later.

Also adds feature-table rows for MobyTurbo and Pack-7, both of which
zmtx/zmrx 2.05 now implements and neither sexyz nor lrzsz does.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net