Quick note on recent QAT issues
r/LocalLLaMA top day5 days agoCommentaryThe post argues that recent Google QAT quantization has several implementation problems, including token embeddings being quantized to q6k instead of using a pure mode. It also claims llama-quantize has a hardcoded parameter that mismatches some optimized groups, and that 32-block groups are misaligned. The author recommends Unsloth UD Q4_K_XL as a temporary option and says they are working on a patch.
llama.cpp PR adds MTP support for Gemma-4 E2B and E4B assistants
r/LocalLLaMA top day5 days agoReleaseThe Reddit post links to ggml-org/llama.cpp Pull Request #24282, which adds MTP support for Gemma-4 E2B and E4B assistants. The submitter frames it as useful for tiny Gemma models on phones, low-end machines, Raspberry Pi, or similarly constrained devices. The post does not include benchmarks, merge status, or setup instructions, so it should be treated as a development signal rather than a finished release.
Introducing FrontierCode★ 78
Hacker News (AI keywords)5 days agoBenchmarkCognition launched FrontierCode, a coding benchmark focused on mergeability rather than only functional correctness. It evaluates correctness, tests, scope discipline, style, and repository-specific quality standards. Built with open-source maintainers and extensive quality control, it shows current frontier models still struggle: Claude Opus 4.8 scores 13.4% on the hardest Diamond subset, ahead of GPT-5.5 and Gemini 3.1 Pro.
Qwen3.6-35B-A3B Tool Calling Benchmark: ByteShape vs Unsloth GGUFs
r/LocalLLaMA top day6 days agoBenchmarkThe post benchmarks eight Qwen3.6-35B-A3B GGUF quants from ByteShape and Unsloth using llama.cpp and tool-eval-bench. It compares f16, q8_0, and q4_0 KV cache quantization under short and long-context pressure, totaling 144 runs and roughly 300 GPU-hours. The author reports no clear ByteShape versus Unsloth winner, q8_0 as close to a free lunch, q4_0 as weaker, and long context as a major tool-calling degradation factor.
Was BitNet a dead end? What happened to ternary LLMs?
r/LocalLLaMA top day6 days agoCommentaryA r/LocalLLaMA user questions whether BitNet and ternary LLMs were a dead end after earlier promise around efficient low-bit models. The post notes that the largest ternary model appears to remain around 2B parameters. It asks why frontier open-weight AI labs are not visibly pursuing the approach, but provides no technical evidence or definitive answer.
LocalLLaMA post tier list
r/LocalLLaMA top day6 days agoOpinionThe author proposes a tier list for r/LocalLLaMA posts in response to complaints about declining post quality. Top-tier posts include new local model releases with GGUF/MLX or benchmark data, meaningful optimizations, complete hardware performance reports, and well-analyzed research. Low-tier posts include repeated toy benchmarks, unrelated cloud AI chatter, AI-generated slop, and thinly disguised ads for Claude-wrapper startups.
When every other post is an AI benchmark, best-model question, or slop app
r/LocalLLaMA top day6 days agoCommentaryThis r/LocalLLaMA post is a meme-like complaint about the subreddit’s recent content quality. The author points to repeated AI-generated benchmark reports, recurring “best model” questions, and hastily built apps or engines presented as groundbreaking. It is not a technical release or evidence-based analysis, but it reflects frustration with noise, hype, and low-effort AI-generated discussion in local model communities.
LocalLLaMA post urges users not to join SpaceX, OpenAI, Anthropic IPOs
r/LocalLLaMA top day6 days agoOpinionA popular r/LocalLLaMA post urges local LLM supporters not to invest in IPOs tied to SpaceX, OpenAI, or Anthropic. The author argues that frontier labs drive up demand and prices for GPUs, RAM, SSDs, HDDs, and NAS hardware, making local inference harder. The post also questions AI company valuations, but its claims are mostly opinion and speculation without cited evidence.
Show HN: Gitdot – a better GitHub, open-source, anti-AI, written in Rust
Hacker News (AI keywords)6 days agoNew ToolGitdot appeared on Hacker News as a Show HN project claiming to be “a better GitHub.” The title says it is open-source, written in Rust, and explicitly anti-AI. No article body was provided, so details about features, licensing, deployment, maturity, and how it differs from GitHub cannot be confirmed from the source.
An Implementation of NanoQuant: A Flexible Binary Quantization Method
r/LocalLLaMA top day6 days agoNew ToolA r/LocalLLaMA post presents an unofficial PyTorch implementation of NanoQuant, a 2026 post-training quantization method for dense transformers. The method factorizes weights into scaling vectors and binary matrices, then quantizes and fine-tunes blocks sequentially to reduce hardware requirements. Early Qwen3-0.6B and Qwen3-4B experiments are promising for base models, but instruct quality remains weak and highly dependent on calibration data.
Xiaomi Claims 1,000+ TPS on a 1T Model Using a Standard 8-GPU Server★ 72
r/LocalLLaMA top day6 days agoBenchmarkXiaomi announced MiMo-V2.5-Pro-UltraSpeed with TileRT, claiming over 1,000 tokens/s decode speed on a 1-trillion-parameter MoE model. The company says it runs on a single standard 8-GPU commodity node, not wafer-scale or SRAM-heavy specialized hardware. The claimed stack combines FP4 MoE expert quantization, DFlash speculative decoding, and TileRT low-latency inference kernels, but independent validation is still needed.
Luce Spark: a 35B MoE on a 16 GB GPU, without the offload tax★ 72
r/LocalLLaMA top day6 days agoNew ToolLuce Spark is an open-source MoE offload system for running 33B-35B A3B models on 16GB-class GPUs. It keeps frequently routed experts on GPU, stores the long tail in system RAM, and swaps cold experts through a bounded async cache. The author reports 13.3 GiB for Qwen3.6 35B-A3B and about 100 tok/s with Spark optimizations, but notes real 16GB GPU testing is still missing.
OpenEnv coordination expands to HF, PyTorch, Unsloth, Modal, and more
r/LocalLLaMA top day6 days agoNew ToolOpenEnv is a tool for creating agentic execution environments such as terminals, browsers, or other systems an agent can interact with. The project will now be coordinated by a committee including Meta-PyTorch, Reflection, Unsloth, Modal, Prime Intellect, Nvidia, Mercor, Fleet AI, and Hugging Face. The post also lists many AI organizations supporting or adopting OpenEnv, positioning it as infrastructure for open-source agent training.
[3090] Gemma4 QAT + MTP quick TPS numbers
r/LocalLLaMA top day6 days agoBenchmarkA r/LocalLLaMA user shared quick throughput numbers for Gemma4 QAT with MTP speculative decoding on an RTX 3090 24GB setup. They report roughly 1.2-1.8x TPS improvement, with Gemma 4 31B moving from about 40 tok/s to 70-80 tok/s. The author frames this as a rough benchmark, using 11 task categories and noting stochastic variation from temp 1.0.
mtmd adds video input support in llama.cpp★ 72
r/LocalLLaMA top day6 days agoReleaseggml-org/llama.cpp merged PR #24269, adding video input support to mtmd through mtmd-cli and /chat/completions, which also enables the web UI path. The implementation invokes a locally installed ffmpeg subprocess instead of bundling codec support, and currently extracts visual frames only, with no audio support yet. It was tested with Qwen3-VL-2B in CLI and Gemma 4 E4B in web UI, making local multimodal video experiments more accessible.
What was your local daily driver for coding last week?
r/LocalLLaMA top day6 days agoCommentaryThis r/LocalLLaMA post is a brief community poll asking users what their local coding daily driver was last week. The post asks commenters to share their favorite model and quant, but the provided text does not include poll options, results, or specific model names. Its value is mainly as a community signal for tracking local LLM coding preferences.
llama.cpp PR #24277 avoids KV cell copies in kv-cache
r/LocalLLaMA top day6 days agoReleaseggml-org/llama.cpp merged PR #24277 by ggerganov, titled “kv-cache: avoid kv cells copies.” The Reddit post says the change improves MTP performance for Gemma-4 and was merged the previous day. It is available starting with the b9551 release, making it relevant for local inference users tracking llama.cpp performance updates.
Magistral★ 78
Mistral AI News6 days agoReleaseMistral AI announced Magistral, its first reasoning model family, with Magistral Small as a 24B open-weight Apache 2.0 model and Magistral Medium for enterprise use. The company emphasizes traceable multilingual reasoning, professional-domain use cases, and faster reasoning in Le Chat through Think mode and Flash Answers. Magistral Small is available on Hugging Face, while Magistral Medium is available in Le Chat preview and via La Plateforme API.
Upgrading agentic coding capabilities with the new Devstral models★ 72
Mistral AI News6 days agoReleaseMistral AI announced two Devstral updates focused on agentic coding workflows: Devstral Small 1.1 and Devstral Medium. Devstral Small 1.1 remains a 24B Apache 2.0 open model and reaches 53.6% on SWE-Bench Verified. Devstral Medium reaches 61.6%, is available through Mistral’s API, and supports private deployment and custom finetuning for enterprises.
Voxtral★ 78
Mistral AI News6 days agoReleaseMistral AI introduces Voxtral, a speech understanding model family with 24B and 3B variants under Apache 2.0. The models support long-context transcription, audio Q&A, summarization, multilingual detection, and function calling from voice. Mistral says Voxtral is competitive across transcription and audio understanding benchmarks, with API access starting at $0.001 per minute and local downloads available on Hugging Face.
Introducing Mistral 3★ 84
Mistral AI News6 days agoReleaseMistral AI introduced Mistral 3, a new open model family under Apache 2.0. It includes Mistral Large 3, a 675B-parameter sparse MoE with 41B active parameters, plus Ministral 3 models at 3B, 8B, and 14B. The release targets frontier open-weight use, multimodal and multilingual workflows, enterprise customization, and efficient local or edge deployments.
Introducing Devstral 2 and Mistral Vibe CLI★ 76
Mistral AI News6 days agoNew ToolMistral introduced Devstral 2, a 123B coding model, and Devstral Small 2, a 24B variant for lighter deployment. The company reports 72.2% and 68.0% on SWE-bench Verified, respectively, with permissive open-source licensing. It also launched Mistral Vibe CLI, an open-source terminal agent for codebase exploration, multi-file edits, command execution, and IDE integration.
Heaps do lie: debugging a memory leak in vLLM
Mistral AI News6 days agoTutorialMistral AI published an engineering deep dive on a memory leak found during vLLM disaggregated serving tests. The leak appeared only with a specific stack involving Mistral Medium 3.1, NIXL, UCX, graph compilation, and P/D disaggregation, with RSS growing steadily despite heap profilers looking normal. The team used pmap, BPFtrace, and targeted GDB automation to trace the issue to UCX mmap hooks and applied configuration fixes plus a vLLM patch.
Leanstral: Open-Source Foundation for Trustworthy Vibe-Coding★ 76
Mistral AI News6 days agoReleaseMistral AI introduced Leanstral, an open-source code agent designed for Lean 4 and formal proof engineering. The model is available through Apache 2.0 weights, Mistral Vibe, and a Labs API endpoint. Mistral positions it as a cost-efficient alternative for verified coding workflows, with FLTEval benchmarks comparing it against Claude family models and large open-source competitors.
Mistral AI partners with NVIDIA to accelerate open frontier models★ 74
Mistral AI News6 days agoBusinessMistral AI announced it is a founding member of the NVIDIA Nemotron Coalition, a global initiative for open frontier foundation models. The partnership combines Mistral AI’s model architecture, training techniques, multimodal capabilities, and enterprise fine-tuning tools with NVIDIA compute, development tools, and synthetic data pipelines. The coalition’s first initiative is a DGX Cloud-trained base model that will support the upcoming NVIDIA Nemotron 4 family and be open-sourced for specialization.
Introducing Mistral Small 4★ 76
Mistral AI News6 days agoReleaseMistral AI introduced Mistral Small 4 as the next major release in the Mistral Small family. It combines reasoning, multimodal, and agentic coding capabilities into one open model with configurable reasoning effort. The model uses a MoE architecture, supports a 256k context window and text-image inputs, and is available through Mistral API, AI Studio, Hugging Face, NVIDIA NIM, and common inference stacks.
Voxtral TTS: Open-Weights, Low-Latency Text-to-Speech from Mistral AI★ 78
Mistral AI News6 days agoReleaseMistral AI introduced Voxtral TTS, its first text-to-speech model, focused on realistic multilingual voice generation. The 4B-parameter model supports nine languages, quick voice adaptation from short references, and low-latency streaming for voice agents. Mistral says human evaluations show stronger naturalness than ElevenLabs Flash v2.5, with API access, Studio testing, Le Chat access, and open weights on Hugging Face.
Voxtral TTS★ 76
Mistral AI News6 days agoReleaseMistral AI introduced Voxtral TTS, its first text-to-speech model, targeting natural multilingual voice generation across nine languages. The 4B-parameter model supports voice adaptation from short references, emotional expressiveness, dialect handling, and low-latency streaming. It is available through API, Mistral Studio, and Le Chat, with open weights on Hugging Face under a non-commercial CC BY NC 4.0 license.
Introducing Mistral 3★ 78
Mistral AI News6 days agoReleaseMistral AI introduced Mistral 3, a new open model family including Mistral Large 3 and Ministral 3 models at 3B, 8B, and 14B sizes. Large 3 is a 675B-parameter sparse MoE model with 41B active parameters, while Ministral 3 targets local and edge use cases. The models are released under Apache 2.0 and are available through Mistral AI Studio, Hugging Face, Amazon Bedrock, and other platforms.
Introducing Mistral Small 4★ 78
Mistral AI News6 days agoReleaseMistral Small 4 is the next major release in the Mistral Small family, unifying Magistral-style reasoning, Pixtral-style multimodality, and Devstral-style coding agents. It uses a MoE architecture with 119B total parameters, 6B active parameters per token, a 256k context window, and configurable reasoning effort. The model is available via Mistral API, AI Studio, Hugging Face, open-source serving stacks, and NVIDIA deployment options.