LumiBooks: Hierarchical Chat Memory

A common failure mode in long LLM chats is unbounded context growth: every reply has to re-attend the whole history, latency climbs, and eventually the context window evicts the early scene-setting. The usual workaround (system-prompt summaries) loses narrative position because the summary lives at the top instead of where the events happened. LumiBooks compresses in place: older windows of messages get rewritten into summaries that occupy the same slot in the prompt as the originals.

Two-tier compression.

• Chapter: a short summary of a contiguous window of older messages, spliced back at the same position. Default 15% of input length, configurable as either % of input (scales with how much you fed it) or token budget (hard cap).
• Arc: several chapters bound into one larger summary once chapter count or token budget crosses a threshold. Default 20% per pass; arcs compress less per pass since the input is already-compressed text. Trigger is one of after N chapters, after N tokens, or manual only.

In-place splicing. Original messages are never deleted. They get marked hidden-on-file: the model reads the summary in their place during prompt assembly, but the rows stay in the database, greyed out in the UI so the user can see where the compacted section begins and ends. Reverting a chapter removes the summary and surfaces the originals again, intact.

Storage. Chapters and arcs live as entries in a per-chat world book (the host’s per-character knowledge-base format), so they’re editable as plain text after the fact (fix a mis-summary, retitle, re-thread) without leaving the editor. Deleting a chapter restores the hidden messages it covered.

Automation policy. Two independent toggles for auto-file chapters and auto-bind arcs, each with its own threshold + lag (the newest N messages or chapters are off-limits to compression so the freshest material stays untouched). Per-side counting is configurable (default: bot replies only count toward thresholds, with an opt-in to count user turns too). Manual “Make” buttons always work regardless of the automation toggles.

Profiles & hooks. Multiple named profiles can be saved and switched per chat/scenario (different connection, samplers, ratios, prompts). Optional regex hooks run on the prompt going to the summarizer (outgoing) and on the summary coming back (incoming); handy for stripping system noise before the model reads, or normalizing the output format after. Preview-before-save stages each new chapter/arc as a draft for approval before it touches the shelf. Failed attempts retry up to a configurable count, then back off and re-pick the same messages on the next trigger.

Defaults tuned for the task. Ships with temperature 0.4 for factual summaries, 128k input cap, 32k output cap; user-overridable but the defaults are sent on the wire so a misconfigured connection preset doesn’t silently break the summarizer.

~200+ unique users.

GitHub