No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from SupraLabs/SupraElegans-500k
All models by SupraLabs →SupraElegans-500K is a ~500,000-parameter causal language model built around a sparse, signed, recurrent neural graph instead of a Transformer. It has no attention mechanism, no positional encoding, and no KV cache. Context is carried by a persistent per-neuron membrane potential that is updated token by token.
The architecture is loosely inspired by ideas from the C. elegans nervous system: sparse connectivity, distinct neuron populations, excitatory/inhibitory signaling, and persistent recurrent state. It is not a biological simulation and makes no claim of biological equivalence.
This is an experimental first release. The goal of the project is to test whether this kind of architecture can do useful language modeling at very small scale, not to compete with Transformers on quality.
Neuron populations: sensory, interneuron/association, output — contiguous index ranges over a fixed pool of neurons.
Connectivity: sparse, directed, signed edge list (fan-in/out on the order of 10-20 per neuron). No dense weight matrix is ever materialized; propagation is a scatter-add over edges.
Neuron dynamics: for each neuron i, at every propagation micro-step,
leak, bias, and threshold are learned per neuron. incoming is the scatter-summed signal from all edges pointing at neuron i, scaled by 1/sqrt(average fan-in) to keep variance controlled across neurons with different in-degree.
Per-token processing: a token's embedding is projected into the sensory population, then the graph runs a fixed number of propagation micro-steps (3 by default) before the output population is read out and projected to vocabulary logits. The membrane potential itself is not reset between tokens — it persists across the whole sequence, which is what gives the model its context window.
Generation: autoregressive, driven entirely by the recurrent state. There is no cache to maintain beyond the current (v, a) state tensors.
Full derivation of the update equations and the rationale for the scaling/clamping choices are in the training notebook.
inference.py registers the local SupraElegansConfig/SupraElegansForCausalLM classes with AutoConfig/AutoModelForCausalLM before loading, so model_type: "supraelegans" in config.json resolves locally — this works whether or not the specific checkpoint's config.json contains an auto_map entry, and does not depend on fetching or executing a remote modeling_supraelegans.py from the Hub. If you're loading with your own script instead of inference.py, either do the same registration (shown below) or use trust_remote_code=True, which relies on the repo's config.json pointing at modeling_supraelegans.py via auto_map:
The tokenizer is loaded with PreTrainedTokenizerFast rather than AutoTokenizer. Depending on which transformers/tokenizers version the checkpoint was saved with, tokenizer_config.json may contain a tokenizer_class value (e.g. TokenizersBackend) that older transformers installs don't recognize, which makes AutoTokenizer.from_pretrained raise a ValueError. Loading the concrete fast-tokenizer class directly avoids depending on that field.
A ready-to-run CLI script (inference.py) is included in the repo:
Since context lives in the recurrent state rather than a KV cache, you can drive the model token by token and inspect or reset that state directly:
model.reset_state() is a no-op kept for API symmetry — in practice you just call model.init_state(...) to start a fresh sequence.
1/sqrt(average fan-in), and clamping the membrane potential to [-6, 6]. Without both, training destabilizes at this graph size — loss spikes and does not reliably converge, even on small overfitting runs.Full training code, including the parameter-budget search, sanity tests, overfitting check, and bio-inspired training metrics (activation sparsity, mean membrane potential, excitatory/inhibitory activity fraction), is in the accompanying notebook.
Apache 2.0.
token → embedding → sensory neurons → sparse recurrent graph → output neurons → vocab logitsv[t+1] = clamp(leak_i * v[t] + incoming[t] + bias_i, -6, 6)a[t+1] = tanh(v[t+1] - threshold_i)pip install torch transformersimport torchfrom transformers import AutoConfig, AutoModelForCausalLM, PreTrainedTokenizerFast
from modeling_supraelegans import SupraElegansConfig, SupraElegansForCausalLM
model_id = "SupraLabs/SupraElegans-500k"
AutoConfig.register("supraelegans", SupraElegansConfig)AutoModelForCausalLM.register(SupraElegansConfig, SupraElegansForCausalLM)
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id)model.eval()
prompt = "Once upon a time"input_ids = torch.tensor([[tokenizer.bos_token_id] + tokenizer.encode(prompt)])
with torch.no_grad(): output_ids, _ = model.generate(input_ids, max_new_tokens=100, temperature=0.8, top_k=50, top_p=0.9)
print(tokenizer.decode(output_ids[0].tolist(), skip_special_tokens=True))from modeling_supraelegans import SupraElegansConfig, SupraElegansForCausalLMfrom transformers import AutoConfig, AutoModelForCausalLM
AutoConfig.register("supraelegans", SupraElegansConfig)AutoModelForCausalLM.register(SupraElegansConfig, SupraElegansForCausalLM)
model = AutoModelForCausalLM.from_pretrained(model_id)python inference.py --prompt "The little robot" --max_new_tokens 150 --temperature 0.7python inference.py --interactivestate = model.init_state(batch_size=1)logits, state = model.nervous_system.step_token(torch.tensor([token_id]), state)