SOFAR: An Adapter I Built So Transformers Stop Forgetting

What this is

Here's the thing about language models: they technically "see" your whole input. Feed one a 100-page contract, a 50-file codebase, a 90-minute conversation transcript, and yes, all of it is sitting there in the context window. But by the time the model gets to the end of that input, it's reliably lost track of what was at the beginning. That's the long-context degradation problem, and it shows up in pretty much any LLM application that runs long enough.

The usual fix is RAG, retrieval-augmented generation: build a search index, look up the relevant snippets, paste them back into the prompt. RAG works, but it doesn't actually fix the problem, it works around it. You're bolting a second system onto a model that still can't reliably use the context you already gave it.

I built SOFAR (Signal-Optimized Frequency-Aligned Routing) to try a different angle. It attaches to an existing HuggingFace transformer as a small adapter, it doesn't touch the host model's weights at all, and it nudges attention toward the internal directions that hold onto information across layers instead of losing it. It's not trying to replace RAG for information that lives outside the prompt. It's aimed at something narrower: making sure the model can actually use the context it was already handed.

The numbers

These come straight from the project's published benchmark recipe, run against plain HuggingFace baselines on a single GPU. The recipe itself, analyze the model, train the adapter, run the needle-in-a-haystack retrieval benchmark (hide a fact somewhere in a pile of text, see if the model can still find it), is identical no matter what size model you point it at. The README notes that the lift scales up with model size.

The CLI that produced this table ships with the project, so you can reproduce it yourself or point it at a different model:

sofar map gpt2                                           # analyze the model
sofar train gpt2 gpt2_channels.json --epochs 3           # train the adapter
sofar bench gpt2 --patched --channel-map gpt2_channels.json
sofar report results/                                    # markdown comparison

Behind those numbers, the codebase has 365 unit and integration tests at 100% line coverage, running under strict-warning CI (pytest -W error), including dedicated pickle, deepcopy, and multiprocessing round-trip tests. I care about this part almost as much as the benchmark table. A research result nobody can trust to run twice isn't worth much.

Why this might matter

I hold two versions of this in my head at once, and I think both are fair.

The careful version: SOFAR shows that a lightweight, weights-only look at a model's internal structure can find which parts of its attention are worth reinforcing, and that a small trained adapter built along those lines produces a measurable needle-retrieval lift on GPT-2 small and medium, across three different context lengths. The adapter is tiny, it never touches the host weights, and the benchmark recipe is public, anyone can re-run it and check my numbers.

The version I'd love to be true: the README states the lift scales with model size, and the same approach already runs across GPT-2, Llama, Mistral, Qwen, and quantized model formats through one code path. If that holds up as models and context windows keep growing, it's a cheaper complement to reaching for RAG every time a model loses the thread of its own input. I built SOFAR to work alongside persistent memory systems, not compete with them: the memory layer pulls the right window of context, and SOFAR helps the model actually use the whole thing once it's pulled.

Honestly, the real answer sits somewhere between those two: a genuine, reproducible result on two GPT-2 sizes at context lengths up to 768 tokens, and an open question about whether it holds at the model sizes and context lengths people actually run in production. I don't know yet. That's part of why it's out in the open.

What it isn't

Trust matters more to me than a good demo, so here's what SOFAR does not do, straight from my own disclosures in the project:

• It won't work on closed-weight APIs. Claude, GPT-4, Gemini, all out of scope. SOFAR needs white-box access to what's happening inside the model as it processes text, so it only works on HuggingFace causal language models you can actually load and run locally.
• It's not a substitute for RAG or persistent memory. If the information you need lives outside the current prompt, or you need the model to remember something across sessions, SOFAR doesn't touch that. It helps with coherence inside the context you already gave it, not memory across conversations.
• It's not free at inference time. On CPU, a trained adapter adds roughly 60% latency on GPT-2-class models. An untrained adapter costs nothing, it's bit-exact identical to the base model through a fast path. On GPU the overhead is smaller relative to everything else the model is doing, but it's not zero, and I say plainly in the README that this is a poor fit if you need every millisecond at inference time.
• The published numbers stop at 768 tokens. I haven't measured behavior at 32k+ context against published results yet.
• This is still a work in progress. Recent releases produce stronger needle-retrieval results than earlier ones did.
• The published baseline leaves out an optional add-on. The benchmark numbers above were measured using only the adapter's core mechanism, not its opt-in extensions.
• This is a portfolio and research project, not a product I'm selling. Issues and pull requests are welcome, but support is best-effort, there's no SLA behind it.

How it actually works, in plain English

Whales talk to each other across thousands of miles of open ocean. Not by calling louder. There's a horizontal layer in the ocean called the SOFAR channel where sound travels enormous distances with almost no loss, and whales drop their calls into it and let the water carry the message. That's where the name comes from.

I went looking for the equivalent inside a transformer: directions in the model's internal math where information can travel across many layers without degrading, the way sound travels through that ocean channel. Find those directions, route attention through them, and the model holds onto more of what you gave it.

SOFAR runs as a small adapter attached to the model, and it never modifies the model's own weights. It first looks at the model's own internal structure to find the directions that reliably carry information across layers, then trains a lightweight adapter, about 0.03% of the model's size, that steers attention along those directions. It's built to hold up across different long-context retrieval shapes, not just the needle-in-a-haystack setup used for the benchmark table above. If you want the full technical detail, it's all in the open-source repo, I didn't hide anything behind this writeup.

It hooks into the model non-invasively, so the model's own forward pass, the actual computation it runs on every input, is never touched or rewritten. The adapters are picklable (you can save and load them like any Python object), multiprocessing-safe, aware of DDP / FSDP / DataParallel wrapping (the common ways people spread a model across multiple GPUs; SOFAR unwraps them automatically), and quantization-aware, meaning it works whether the model has been shrunk down with bitsandbytes 8-bit and 4-bit, GPTQ, or AWQ. In code, attaching it looks like this:

from transformers import AutoModelForCausalLM
from sofar import map_channels, patch

model = AutoModelForCausalLM.from_pretrained("gpt2")
channel_map = map_channels(model)        # analyze the model, offline
patched = patch(model, channel_map)      # attach the adapter

Where to go deeper

If you want more than the plain-English version, here's where to look:

• The code, the full test suite, and EVAL_GUIDE.md (the actual CPU and GPU latency numbers behind the "not free at inference time" point above) all live in the repo: github.com/NORTHTEKDevs/SOFAR.
• The benchmark method above is the needle-in-a-haystack style of long-context evaluation, hide a fact somewhere in a big pile of text, see if the model can still find it. It's a standard way people test long-context recall; I didn't invent it, I just ran it against SOFAR.
• You'll need Python 3.10 or newer to run it. Check your version at python.org if you're not sure.
• Want to poke at it or contribute? CONTRIBUTING.md in the repo covers how to get involved.

Built by Kristian Baer / Northtek in Anchorage, Alaska. Research and portfolio code, Apache-2.0 licensed. If you use this in your own work, there's a citation in the repo README.