https://gitlab.synchro.net/main/sbbs/-/commit/17846dff73b55f7d3303f588
Modified Files:
ctrl/chat_llm.ini exec/chat_llm.js
Log Message:
chat_llm: stop discarding known-good URLs and repeating whole replies
Three defects in the reply pipeline, all visible in a single 2026-08-09 #synchronet exchange where the bot cited a wiki page correctly, then
answered "what were those links again?" four times running with the
link silently deleted from every reply.
strip_fake_urls() validated citations against ctx._valid_wiki_urls,
which is rebuilt each turn from that turn's BM25 hits alone. A URL the
bot had already cited was therefore treated as fabricated the moment
retrieval moved on -- and a back-reference question ("what was that
link again?") carries no content words, so it retrieves unrelated pages
by construction and could never re-validate the link it was asking
about. URLs appearing verbatim in the transcript the model was handed
now count as valid: quoting itself is not fabrication. The transcript
holds post-strip text, so a stripped URL never enters it and cannot
launder itself into the allowlist, and memory is per-speaker, so one
user's transcript cannot validate another's citations.
"Cite this URL verbatim:" is RAG scaffolding prepended to wiki chunks
to make their URLs visible to the model. qwen2.5:7b echoes it back,
and with the adjacent URL stripped the bare instruction was all that
reached the channel. It is now removed from replies unconditionally --
it is an instruction to the model and has no business in output.
A cornered small model also re-emits its previous answer nearly
verbatim; two consecutive replies to the same user were token-identical,
and the same closing boilerplate ended three of six replies in a row.
Raising temperature (the earlier mitigation) varies the wording without noticing the repetition. Each finished reply is now compared against
that speaker's recent bot turns by word-set overlap; on a match the bot regenerates once with an explicit don't-repeat instruction, and if that
is still a duplicate it answers from the guru file instead -- the
pre-LLM Guru's keyword-matched responses, so a bot with nothing to add
deflects in character rather than repeating itself. A second
deflection in a row stays silent.
The 0.75 default is word-set Jaccard, calibrated against real repeats
in data/guru_irc_chat.log: observed duplicates scored 0.83-1.00,
genuinely-new answers on the same topic 0.42-0.62. Replies under 12
words are exempt, since "ok" and "yep, that works" are legitimately
repeatable. Explicit requests to repeat bypass dedup entirely, and it
is skipped in streaming (private 1:1) mode, where the first attempt is
already on the user's screen and swapping the stored string would leave
them reading one answer while the log records another.
The guru file parser follows sbbs_t::guruchat()/guruexp() in src/sbbs3/chat.cpp, with two deliberate divergences. Sections carrying
an access-requirement test are skipped: an IRC nick has no user record
to evaluate SEX/FLAG/LEVEL against, and guessing would put the stock
file's gendered variants in front of the wrong people. Chat-control
codes are stripped rather than obeyed, since `q ends the session and `h
hangs up, neither of which a bot may do to a channel. Dropped codes
leave a space, because the file uses `_ pauses between words for timing
("write to`_`_Rush Limbaugh") where deleting them outright runs the
words together.
---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net