Silia: A Tiny Transformer Architecture for Sub-10M Parameter Models
r/LocalLLaMA top day3 days agoPaperA student from India shared their first paper on r/LocalLLaMA, proposing Silia, a Transformer architecture for extremely small models. The idea is to merge attention-style dynamic mixing with SwiGLU-like nonlinear transformation, aiming to save parameters in models under roughly 10M parameters. The author frames the work as an early, small-scale exploration, limited by old hardware and restricted access to larger compute.
How LLMs Actually Work
Hacker News (AI keywords)10 days agoTutorialThe article explains how modern LLMs convert text into token IDs, embeddings, and position-aware vectors before passing them through stacked transformer blocks. It covers attention, multi-head attention, KV cache, GQA, feed-forward networks, MoE, residual streams, normalization, and decoding. Its goal is educational: helping readers understand the common architecture behind many current model families and read model cards or papers more confidently.
Transformer 架構下的編碼器-解碼器（Encoder-Decoder）模型深度解析★ 70
Hugging Face Blog2,073 days agoTutorialThis classic blog post written by Hugging Face researcher Patrick von Platen takes a deep dive into the Transformer-based Encoder-Decoder model architecture…