17
followers ·
5 following None yet
reacted to ucr-max 'spost with 🔥 2 days ago Introducing Limen0.2B
We are releasing Limen0.2B, a 222.5M-parameter base language model developed as a research platform for efficient pretraining and superword tokenization at smaller scales.
Limen0.2B was trained from scratch on 50B tokens and uses a compact 16K BoundlessBPE vocabulary. The project explores whether SuperBPE-style tokenization can remain effective in a substantially smaller model and vocabulary regime than those examined in earlier large-scale experiments.
The model also combines a deep-and-narrow transformer design with Exclusive Self-Attention, grouped-query attention, and tied embeddings. Its compact vocabulary reduces the embedding footprint and leaves a larger share of the parameter budget available to the transformer layers.
Despite its relatively modest training budget, Limen0.2B achieves competitive results for its scale across the reported language understanding, commonsense reasoning, and grammatical evaluation tasks. Comparisons with other compact models are provided as context rather than strict rankings, as their training data, token budgets, architectures, and evaluation settings differ.
The release includes the model weights, implementation, training configuration, checkpoint progression, and evaluation results, all under Apache 2.0.
https://huggingface.co/UniversalComputingResearch/Limen0.2B
Technical feedback, independent evaluations, and further experiments with the model and tokenizer are welcome.
reacted to ucr-max 'spost with 🔥 2 days ago Introducing Limen0.2B
We are releasing Limen0.2B, a 222.5M-parameter base language model developed as a research platform for efficient pretraining and superword tokenization at smaller scales.
Limen0.2B was trained from scratch on 50B tokens and uses a compact 16K BoundlessBPE vocabulary. The project explores whether SuperBPE-style tokenization can remain effective in a substantially smaller model and vocabulary regime than those examined in earlier large-scale experiments.
The model also combines a deep-and-narrow transformer design with Exclusive Self-Attention, grouped-query attention, and tied embeddings. Its compact vocabulary reduces the embedding footprint and leaves a larger share of the parameter budget available to the transformer layers.
Despite its relatively modest training budget, Limen0.2B achieves competitive results for its scale across the reported language understanding, commonsense reasoning, and grammatical evaluation tasks. Comparisons with other compact models are provided as context rather than strict rankings, as their training data, token budgets, architectures, and evaluation settings differ.
The release includes the model weights, implementation, training configuration, checkpoint progression, and evaluation results, all under Apache 2.0.
https://huggingface.co/UniversalComputingResearch/Limen0.2B
Technical feedback, independent evaluations, and further experiments with the model and tokenizer are welcome.
reacted to ucr-max 'spost with 🔥 2 days ago Introducing Limen0.2B
We are releasing Limen0.2B, a 222.5M-parameter base language model developed as a research platform for efficient pretraining and superword tokenization at smaller scales.
Limen0.2B was trained from scratch on 50B tokens and uses a compact 16K BoundlessBPE vocabulary. The project explores whether SuperBPE-style tokenization can remain effective in a substantially smaller model and vocabulary regime than those examined in earlier large-scale experiments.
The model also combines a deep-and-narrow transformer design with Exclusive Self-Attention, grouped-query attention, and tied embeddings. Its compact vocabulary reduces the embedding footprint and leaves a larger share of the parameter budget available to the transformer layers.
Despite its relatively modest training budget, Limen0.2B achieves competitive results for its scale across the reported language understanding, commonsense reasoning, and grammatical evaluation tasks. Comparisons with other compact models are provided as context rather than strict rankings, as their training data, token budgets, architectures, and evaluation settings differ.
The release includes the model weights, implementation, training configuration, checkpoint progression, and evaluation results, all under Apache 2.0.
https://huggingface.co/UniversalComputingResearch/Limen0.2B
Technical feedback, independent evaluations, and further experiments with the model and tokenizer are welcome.
View all activity view post Wiki-SmartBotLM Benchmarked. Here are the results: Benchmark WikiSB | Gemma3_270M BoolQ 37.8 | 61.4 ARC Challenge 24.2 | 28.2 HellaSwag 26.3 | 37.7 PIQA 53.5 | 66.2 Winogrande 50.2 | 52.3 Summary: Closest benchmark: Winogrande (2.1-point gap) Strong performance: ARC Challenge (4.0-point gap) Moderate gap: HellaSwag and PIQA Largest gap: BoolQ (Reading Comprehension) It shows despite the training difference between Gemma3 270M and WikiSmartBotLM, It does come close to Gemma 3 in some benchmarks. pranavupadhyaya52/Wiki-SmartBotLM-Instruct See translation
view post After several weeks of experimenting, debugging, and iterating, I am excited to share WikiSmartBotLM. WikiSmartBotLM is a compact decoder only language model built from the ground up as an educational and practical project. The goal was not simply to train another language model, but to create one that is easy to understand, modify, and experiment with while following many of the architectural ideas used in modern LLMs. The model is built on a custom Transformer architecture featuring Rotary Positional Embeddings, RMSNorm, SwiGLU feed forward layers, grouped query attention, and an efficient autoregressive decoder optimized for local inference. The repository includes the complete model implementation, configuration files, tokenizer integration, training pipeline, inference scripts, checkpoint conversion utilities, and examples that demonstrate how each component works together. Whether you want to understand the forward pass, train your own model, or build applications on top of WikiSmartBotLM, everything is designed to be approachable. You can directly run the model via the models Huggingfaces space, which I've included in the post. Model Repository: pranavupadhyaya52/Wiki-SmartBotLM-Instruct I hope WikiSmartBotLM becomes a useful resource for anyone who enjoys learning by building. Feedback, issues, feature requests, and contributions are always welcome. https://huggingface.co/spaces/pranavupadhyaya52/WikiSmartBot See translation
