No official torrent yet β requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from BananaMind/BananaMind-2.1-Unified
All models by BananaMind β[Image blocked: Banner]
BananaMind-2.1-Unified is a three-tower decoder-only causal language model trained from scratch by BananaMind on a 38B-token flat mix inspired by the brain having two hemispheres. It is the successor experiment to BananaMind-2-Unified, and the first BananaMind model where the towers cannot talk to each other directly at all.
Three transformer stacks share one d=384 embedding. A and C are the outer towers and each owns an output head; the next token is a probability-space mixture of the two. B is the relay β it has no output head, no loss term of its own, and is the only path between A and C. Everything the outer towers share has to survive a trip through the middle, and B is trained entirely by gradient arriving through its four bridge directions.
The model has 34,999,041 parameters, a 4,096 token context window, and the same custom 8k-token digit-aware byte-level BPE tokenizer as BananaMind-2-Mini.
This is a base model. It is not instruction tuned.
[Image blocked: Architecture]
Tower C reads the shared embedding natively at d=384; A and B get a linear input projection. The four bridge families total 1,232,640 parameters.
Three rounds, 1-indexed layer numbers. Bridge output is added to the residual before the receiving block, which guarantees the relay always has real processing between taking a signal in and handing one back out.
Placement is biased late on the outer towers because that is where the 2.0 run's gates actually grew. Note the two structural consequences: A's layers 13-14 run after the final bridge read and can never influence C, and C's layer 6 runs after B is finished and can never influence A.
Every bridge is per-channel gated and initialised to 0.01 rather than 0. In 2.0 a zero init was correct because both towers had their own loss and the bridges were a bonus. Here B has no loss term, so a zero init risks the middle receiving no gradient on step 0 and never waking up.
It woke up. Mean |gate| in the final checkpoint, against the 0.01 init:
Every one of the twelve gates grew, by 4x to 47x. Two patterns are visible: traffic is
strongly biased toward the deep rounds in three of the four directions (the same
concentration 2.0 showed), and the into-B directions carry noticeably larger gates than
the out-of-B ones, with B -> C the quietest channel in the model.
The next-token distribution is a two-way probability-space mixture, not a logit sum:
alpha is the per-token weight on tower A. Measured on an 84-token mixed
history/math/science passage: mean 0.433, range 0.049 to 0.865, with 41.7% of tokens
weighted toward A. The mixer is genuinely token-dependent, not collapsed onto one head.
Identical to the BananaMind-2-Mini tokenizer: a custom 8k byte-level BPE trained on FineWeb-Edu text with digit-aware pre-tokenization. Digits are kept as separate tokens so numbers do not collapse into large number tokens.
Digit IDs:
Examples:
Special token IDs:
38B tokens, streamed, flat mix, no curriculum ramp. 2.1 changes topology, and a moving data distribution on top of that would make the comparison against 2.0 unreadable. The shares are Mini's final aggregate targets held constant from the first token.
Dataset revisions are pinned in checkpoint_metadata.json.
There is no L_B term. B trains entirely on gradient arriving through its four bridge
directions.
Final training-batch losses at step 72,470:
These are training-batch numbers, not held-out. Note also that L_A and L_C are computed
with the bridges live, so they do not predict standalone tower performance β that was
the central misreading in the 2.0 run. The ablation modes below answer it properly, and the
gap is enormous: solo head A logs 2.643 with the bridges live and 10.386 without them.
Full per-step history is in training_metrics.jsonl.
lm_eval 0.4.12, zero-shot, fp32 on one RTX 5070 Ti, --batch_size auto (settled at 64), full
test sets (ARC-Easy 2,376 / PIQA 1,838 / HellaSwag 10,042). All task scores are acc_norm,
matching the BananaMind-2-Mini card. Every ablation mode was evaluated on the same run.
Approximate standard errors: ARC-Easy Β±1.0, PIQA Β±1.1, HellaSwag Β±0.5 points.
Against the previous generation on the same three tasks:
The three-tower model is roughly level with Mini overall β ahead on PIQA by 2.1 points, behind on ARC-Easy by 1.4 β for 39% more parameters and 27% more tokens. On aggregate benchmarks the three-tower model is roughly level with Mini. The contribution of this architecture is not a benchmark number β it is what the ablations and lens data reveal about how integration, specialisation, and understanding organise themselves when the only path between two output towers is a silent relay that has no voice of its own.
full is +10.1 points over the chance
average. Everything else falls between +0.7 and +6.5, and the bottom four modes sit within a
few points of chance on all three tasks.a_only is indistinguishable from chance (34.01 vs 33.33; PIQA 50.11 against a 50.00
floor). Tower A alone, with a 14-layer stack and its own trained head, has essentially no
standalone ability.cut_bridges (35.76) and
c_only (35.91) are within noise of each other. Three towers that cannot communicate are
worth no more than tower C on its own β which is what you would expect if C's head carries
the model whenever the relay is dead.bypass_b (39.61) sits 3.9 points
above cut_bridges while performing zero computation in the middle, and B's actual
computation is worth a further 3.8 on top.The NLL ablations put ab_only (4.845) well ahead of cb_only (7.373). The benchmarks reverse
it: cb_only averages 39.84 against ab_only's 35.43. Both measurements are correct and they
are measuring different things β NLL is absolute calibration over running text, while
acc_norm is length-normalised ranking among a fixed set of candidate answers. A+relay
predicts ordinary text more accurately; C+relay discriminates better between multiple-choice
options. C reads the shared embedding natively at d=384 and owns the larger head (3.15M vs
2.10M), which is the likeliest explanation for the discrimination advantage.
The practical reading: do not treat either metric alone as "which tower matters more". They rank the halves of this model in opposite orders.
Four-choice base-text continuation scored by mean conditional token log-probability, 350 cases across 7 categories, run per mode. Chance accuracy is 25%.
Per-category Elo (50 cases each, so single-category gaps under ~100 Elo are noise):
Two things here that the lm_eval table does not show:
ab_only (867) and a_only (866) are the same score. On this benchmark, giving tower A a
live relay and a running tower B buys essentially nothing over running A completely alone.
Whatever the relay contributes, it needs tower C at the other end of it β consistent with the
lens finding that B's representation only becomes readable when both outer towers feed it.ab_only/a_only 867/866 over cb_only/c_only 803/774), while lm_eval acc_norm ranked
them the other way (cb_only 39.84 over ab_only 35.43). Three metrics have now ordered the
two halves of this model three different ways β NLL favours A, acc_norm favours C, Base Bench
Elo favours A. None of them is wrong; "which tower matters more" is simply not a
metric-independent question here.full is the only mode meaningfully clear of chance on accuracy, and the only one above 900
Elo. Raw reports and per-case predictions for every mode are in
eval_results/base_bench_1.1/<mode>/.
Fitted with Anthropic's jacobian-lens method:
lens_l(h) = unembed(J_l @ h) with J_l = E[βh_final / βh_l], averaged over 6 prompts and all
valid source/target positions. For the two-headed modes the target basis is the joint
vector z = [x_a ; x_c] (256 + 384 = 640), since the mixture is a deterministic function of
that single vector and no individual tower's basis can express it. Single-head modes use that
head's basis alone.
Scale-normalised mean βJβ_F per tower β each half divided by RMS Β· βd, because the towers
have different widths and different residual scales, so raw Frobenius norms would just measure
residual magnitude. The bracketed figure is the share of that tower's influence landing in C's
half of the joint basis.
Three of these rows are self-validating. Under cut_bridges tower B's mass is exactly
0.000 β B owns no head, so with the bridges dead it cannot reach the output at all, and the
lens recovers that from the gradients without being told. The same row shows A contributing
0.0% to C's half and C contributing 100% to it: the two outer towers are perfectly decoupled.
And a_only (3.505) and c_only (5.873) reproduce the cut_bridges tower masses to the
digit, which is the only thing they could do if bridge-cutting truly isolates the towers.
Four things fall out:
full. B has no head and no loss term, yet in
full its deepest block reads Paris/ France, oxygen/ hydrogen, blue. This is
the README's original "does B wake up?" question answered from the representation rather
than from gate magnitudes.cut_bridges B's readout
collapses to the four special tokens on every prompt β the residual of a stack that receives
nothing and reaches nothing.ab_only and cb_only the
bridges are live and B still runs, but its readout is junk ( High Tem Cal, dom
vere ure). B's task content is not something A alone or C alone puts there; it appears
only at the confluence.bypass_b sharpens the outer towers while making the model worse. With B reduced to the
identity, tower A's own readout gets more directly predictive ( France French, versus
the B in in full) and both tower masses rise sharply (A 3.572 β 5.138, C 8.370 β
13.205). The outer towers compensate by carrying more themselves β and still lose 3.8 points
of benchmark average. What B computes is not replaceable by the outer towers working harder.Caveats: this is a linearised, corpus-averaged sensitivity, not a causal contribution β a gate-ablation KL would be the confirming experiment. The normalisation is also a choice; raw Frobenius mass gives a different picture because tower A's final residual runs ~3.3x hotter than C's. Fit settings (6 prompts, 48 tokens, first 8 positions skipped) are lighter than a publication-grade run, so treat small differences between adjacent rows as noise.
The lens readout reveals something unexpected about how the relay processes information. Tracing B's representations layer by layer across multiple prompts shows a consistent pattern: B processes significance before facts.
The early layers (B-L0, B-L1) consistently produce evaluative and abstract terms β not factual content, not input echoes, but something closer to significance assessment. By the final layer (B-L4), B has arrived at the correct answer: Paris/France, Water/hydrogen, blue/green.
This trajectory β from evaluation to categorisation to answer β is consistent across prompts and mirrors the ordering of affective and cognitive processing observed in biological neural systems, where emotional evaluation precedes and shapes factual retrieval. No part of the architecture or training objective was designed to produce this ordering. B discovered it.
B's influence distribution across the two output towers is nearly perfectly balanced (49.1% toward C), confirming that B functions as a symmetric integration hub rather than favouring either side.
The aggregate benchmark comparison with Mini understates what the ablations reveal about physical reasoning. PIQA measures physical intuition β understanding that you pour water into a cup, not a fork β and it is the benchmark where the relay topology produces its clearest separation.
Tower A alone scores 50.11% on PIQA: indistinguishable from the 50.00% chance floor. Tower C alone scores 52.12%: barely above chance. The full system scores 61.75%. The entire physical reasoning capability of this model β all 11.75 points above chance β is a product of integration. Neither output tower can reason about the physical world on its own.
This makes PIQA the sharpest measure of what the relay contributes. It is not a capability that either tower possesses and the relay merely enhances. It is a capability that exists only in the integration and nowhere else.
For comparison, the single-tower BananaMind-2-Medium at 50M parameters scores 59.41% on PIQA. The 35M three-tower model exceeds it by 2.3 points, with a silent middle tower that produces no output and consumes roughly 5.8M of the parameter budget on pure integration.
Three findings from the ablation and lens data point to integration as an emergent rather than additive phenomenon:
B needs both outer towers to become semantic. Under ab_only and cb_only, B's readout collapses to junk despite having live bridges and running its full computation. B's task-relevant representations β Paris, oxygen, blue β appear only when both A and C feed it simultaneously. This is not A's knowledge or C's knowledge routed through B. It is something new that exists only at the confluence.
Dedicated understanding parameters outperform general-purpose ones. Tower B has no output head and no loss term. Its 5.8M parameters are trained entirely by indirect gradient arriving through four bridge directions. Yet removing B (the bypass_b ablation) costs 5.82 PIQA points and 59 Elo on Base Bench. Those 5.8M parameters, freed from the output objective and devoted entirely to integration, contribute more per parameter than any equivalent allocation to the output towers could.
The channel matters, but so does the computation. bypass_b replaces B with the identity, keeping the bridges live but performing zero computation in the middle. This recovers 3.59 nats over cut_bridges, showing that a path between A and C is valuable even without processing. But B's actual computation adds a further 2.83 nats on top. The relay is not merely a conduit; it transforms what passes through it.
This one predates the sweep above: it was run against the unpinned main of the source repo on
2026-08-18, before the 100% checkpoint in this folder was pulled, so it is not confirmed to
be this exact checkpoint, and it was run only in full mode. Treat it as indicative.
This model uses custom architecture code, so load it with trust_remote_code=True.
Install dependencies:
Run inference:
Supported, and on by default β generate(), use_cache=True, and a manually supplied
DynamicCache all work. Decoding is O(n) per token instead of a full three-tower re-forward.
A relay model has no single residual stack, so the three towers share one flat cache index space, in the order the config documents:
config.num_hidden_layers is 25, the sum of the three tower depths. It exists purely so
generic Transformers tooling β DynamicCache above all β can size a per-layer cache. It is
derived in the config class, never read back from a serialised config.json, so a stale value
cannot desync it from the layer mapping. Each tower keeps its own entries; nothing is shared
between them.
Two properties of the architecture make this work without any relay-specific machinery:
Edge is a per-channel gate on a linear map, so every
bridge contribution for a newly arriving token is computable from that token's own tower
states. No bridge output has to be cached alongside the KV states.Measured on CPU, fp32, from a 221-token prompt:
Verified equivalences: token-by-token cached decode reproduces the full uncached forward
(max abs error 3.1e-05 on fp32 log-probs), chunked prefill in 5/4/5-token pieces reproduces it
too, greedy generation is bit-identical with and without the cache over 48 tokens, and
left-padded batched generation matches the same prompts run singly. cut_bridges=True was
checked separately and also matches.
Scoring passes are unaffected: when labels is supplied, forward() skips cache construction,
since a loss pass consumes the whole sequence at once and would otherwise allocate 25 layers of
state for nothing.
.logits contains.logits holds a normalised log-probability vector, not unnormalised logits, because the
two heads are mixed in probability space. Verified on this checkpoint: logsumexp over the
vocabulary is 1.2e-06, i.e. zero.
log_softmax is the identity on it, so loglikelihood scoring, generate() and temperature-1
sampling all behave correctly. The one thing that is not meaningful is treating the numbers
as unnormalised scores with an arbitrary additive offset β they are already calibrated.
hidden_states() returns the two final tower residuals and the raw mixer logit, which is the
entry point for any interpretability work on the relay:
Because A and C have no direct path to each other, "what is the relay worth?" is only
answerable by cutting the model apart. relay_mode selects where to cut. All seven modes run
on the same weights, with no retraining.
bypass_b is the interesting one: tower B's five blocks are replaced by the identity, but
every bridge stays live. A and C still exchange signal β through a relay that does no
computation. That separates "the relay computes something" from "a channel exists at all".
Three ways to select a mode, checked against each other so a contradiction raises rather than resolving by some invisible precedence rule:
cut_bridges=True predates relay_mode and still works, selecting the cut_bridges mode.
In modes where a tower is off, hidden_states() returns None in that tower's slot, and
mix_logit is None whenever only one head is live. In full mode all three are always
tensors, so existing callers are unaffected.
Mean NLL over 12 held-out encyclopedic passages, 519 predicted tokens, fp32:
Four things fall out of this, and they are the answer to the question 2.0 left open:
a_only and c_only both score worse than
uniform over the 8,192-token vocabulary. Without its partner each head is not merely
degraded, it is confidently wrong. 2.0's tower-B bridge dependence reappears here in the
extreme, and on both towers at once.cut_bridges at 8.669
sits just under the 9.011 uniform baseline.cut_bridges to
bypass_b recovers 3.59 nats with tower B performing zero computation β merely existing
as a path. B's actual computation is then worth a further 2.83 nats on top.ab_only (4.845) beats cb_only (7.373) by 2.5
nats, even though C is the wider tower and owns the larger head. A plus a computing relay,
with C absent entirely, also edges out all three towers with a non-computing relay.Note that this measures the mode's loss, not a benchmark score, on a small sample. It is a
sharp instrument for relative comparison between modes and not a substitute for a full
lm_eval sweep.
For stable continuations:
do_sample=Falserepetition_penalty=1.1max_new_tokens=64 to 160For more varied text:
do_sample=Truetemperature=0.6 to 0.8top_p=0.9top_k=50repetition_penalty=1.1max_new_tokens=64 to 192BananaMind-2.1-Unified is intended for multi-tower architecture research, relay and information-bottleneck experiments, interpretability work on cross-tower routing, workspace and integration research, consciousness research, lightweight language-model research, local experimentation, and text continuation.
2.0 answered one question β two towers with cuttable bridges do beat either tower alone β and raised a sharper one. Its gate data showed traffic concentrated deep and heavily asymmetric, and the bench then showed one tower had traded away standalone ability for bridge dependence. 2.1 asks what happens when the exchange is not a side channel between two peers, but the only path between them.
We're releasing BananaMind-2-Unified soon when its done training.
The architecture, trainer, and export tooling live in BananaMind-2.1-Unified-Arch/.
Apache 2.0
alpha = sigmoid(mix_head([ln_f_a(h_a) ; ln_f_c(h_c)]))log p = logaddexp(log alpha + log_softmax(lm_head_a(h_a)), log(1 - alpha) + log_softmax(lm_head_c(h_c)))18 -> [20, 27]227 -> [21, 21, 26]L = L_mix + 0.3 * (L_A + L_C)pip install -U transformers safetensors torchimport torchfrom transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "BananaMind/BananaMind-2.1-Unified"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
device = "cuda" if torch.cuda.is_available() else "cpu"dtype = ( torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float32)
model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, dtype=dtype,).to(device).eval()
prompt = "The capital city of France is"inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.no_grad(): output = model.generate( **inputs, max_new_tokens=96, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.1, pad_token_id=tokenizer.eos_token_id, eos_token_id=tokenizer.eos_token_id, )
print(tokenizer.decode(output[0], skip_special_tokens=True))h_a, h_c, mix_logit = model.hidden_states(input_ids)alpha = torch.sigmoid(mix_logit) # per-token weight on tower A# 1. at load timemodel = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, relay_mode="bypass_b")
# 2. the single-tower alias ("a" or "c"; B has no head and cannot run alone)model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, use_single_tower="a")
# 3. per call, for sweeping without reloadinglogits = model(input_ids, relay_mode="ab_only").logitsmodel.generate(input_ids, max_new_tokens=64, relay_mode="cb_only")
# or switch in place (start a fresh KV cache afterwards)model.set_relay_mode("cut_bridges")