[Image blocked: banner]
BananaMind-1.5-Base
BananaMind-1.5-Base is a small English causal language model trained from scratch by BananaMind.
It is our first fully pretrained medium model
Model Details
[Image blocked: Benchmarks]
A instruction tuned version is coming very soon.
Usage
Generation Settings
Recommended starting settings:
For deterministic sanity tests:
Training
BananaMind-1.5-Base was trained from scratch on approximately 27B tokens of FineWeb-Edu-style English web text.
The model uses a custom 32k byte-level BPE tokenizer and a compact Llama-style architecture with grouped-query attention.
Architecture
BananaMind-1.5-Base uses a compact Llama-style decoder architecture:
- 12 Transformer layers
- 640 hidden size
- 1728 intermediate size
- 10 attention heads
- 5 key-value heads
- grouped-query attention
- SiLU activation
- RMSNorm
- tied input/output embeddings
- 4096 token context length
Evaluation
Our model performs very good in comparison to other models:
[Image blocked: Benchmarks]
Parameter vs Size
[Image blocked: Parameter vs size Chart]
Citation