TTS Benchmark Revamped with Objective Standards and Blind ELO Voting (46 Models)
r/LocalLLaMA top day5 days agoBenchmarkReddit user UkieTechie has revamped their TTS benchmark platform with objective scoring standards and live blind voting, now covering 46 speech synthesis models. Hosted on Hugging Face Space, the arena lets users vote on audio quality without knowing the model name, generating a dynamic ELO leaderboard. The project is open-source on GitHub and welcomes community submissions of new models.
Can LLMs Beat Classical Hyperparameter Optimization Algorithms?
Hacker News (AI keywords)5 days agoBenchmarkThis paper investigates whether LLMs can serve as effective hyperparameter optimization (HPO) agents, competing with established classical methods such as Bayesian optimization, TPE, and random search. The study likely employs a systematic evaluation framework where LLMs iteratively suggest hyperparameter configurations based on task descriptions and historical evaluation results. Findings aim to clarify the practical potential and limitations of LLMs in AutoML pipelines.
Jetson Orin NX Build for Hermes Agent + Benchmarking
r/LocalLLaMA top day5 days agoHardwareThe post describes turning an unused Jetson Orin NX into a compact local LLM server for Hermes Agent testing. The goals were low noise, over 10 tok/s generation, 300 tok/s prompt processing, at least 65K context, and a custom case. After testing Gemma 4, Qwen 3.6, and many quant variants, the author reports Gemma 4 26B A4B UD Q2_K_XL reaching 66K context and 10.21 tok/s near 60K context.
Omi Med STT v1: Open-Weight Medical ASR Fine-Tuned from Parakeet 0.6B★ 72
r/LocalLLaMA top day5 days agoReleaseOmi Health’s founder says he fine-tuned NVIDIA Parakeet TDT 0.6B v2 for clinical speech and released Omi Med STT v1 under CC-BY-4.0. The runtime supports Mac, Windows, and Linux, auto-selecting MLX, NeMo, or GGUF/parakeet.cpp backends. In the author’s held-out medical benchmark, it reports 2.37% medical-WER and 145× realtime on local A10 compute.
How much do amd64 microarchitecture levels help in Go?
Hacker News (AI keywords)6 days agoBenchmarkDaniel Lemire tests Go’s GOAMD64 levels using Roaring Bitmaps on a modern Intel Xeon. v2 brings strong gains where popcnt matters, while v3 adds further speedups in dense bitmap and set-operation workloads through AVX2. v4, despite implying AVX-512 support, shows no meaningful improvement in these benchmarks, likely due to current Go compiler limitations.
Qwen 3.6 27B DeepSWE Benchmark Results Highlight Gap Between Local and Closed-Source Models
r/LocalLLaMA top day6 days agoBenchmarkA community benchmark of Qwen 3.6 27B on DeepSWE yielded a score of 1.79% (18/20th place), slightly outperforming Haiku 4.5. Run on a single RTX 6000 Blackwell GPU via vLLM with reasoning enabled, the test averaged 32 minutes and 44k output tokens per task. The author notes that while Qwen 3.6 27B represents a 'poor man's local SOTA,' the massive gap compared to frontier closed models suggests local LLMs are struggling to keep pace in complex coding.
EVA-Bench Data 2.0: 3 Domains, 121 Tools, 213 Scenarios
Hugging Face Blog10 days agoBenchmarkServiceNow AI published a Hugging Face Blog post titled “EVA-Bench Data 2.0: 3 Domains, 121 Tools, 213 Scenarios.” Based only on the title, it appears to be a benchmark dataset update involving tool-use or scenario-based AI evaluation. The exact domains, tools, scenario design, licensing, supported models, and evaluation methodology cannot be confirmed without the full article.
Hugging Face 與 IBM 聯合推出 Open Agent Leaderboard：開源 AI 智能體效能評測全新基準★ 80
Hugging Face Blog27 days agoReleaseHugging Face and IBM Research have jointly announced the launch of the "Open Agent Leaderboard," aimed at establishing an objective, standardized, and fully…
Hugging Face 為 Open ASR 排行榜引入「防刷榜機制」，使用私有測試數據打擊 Benchmaxxer★ 75
Hugging Face Blog39 days agoReleaseHugging Face has recently made a major update to its popular Open ASR (Automatic Speech Recognition) leaderboard, aimed at combating the increasingly serious…
QIMMA ⛰：首個品質優先的阿拉伯語大型語言模型（LLM）排行榜
Hugging Face Blog54 days agoReleaseThe Technology Innovation Institute (TII) of the United Arab Emirates — the organization behind the well-known open-source model Falcon — has officially…
深入解析 VAKRA：IBM Research 評估 AI Agent 推理、工具調用與失敗模式的全新基準測試★ 75
Hugging Face Blog60 days agoReleaseAs generative AI technology has evolved, the industry's focus has shifted from pure "Large Language Models (LLMs)" to "AI Agents" capable of autonomously…
EVA：ServiceNow AI 推出全新語音 Agent 評估框架★ 75
Hugging Face Blog82 days agoReleaseWith the proliferation of GPT-4o, Gemini Live, and various end-to-end voice models, Voice Agents have become an important frontier in AI applications. However…
IBM 與柏克萊加州大學推出 IT-Bench 與 MAST：診斷企業級 AI Agent 失敗原因的全新基準與框架★ 80
Hugging Face Blog115 days agoRelease### The Pain Points of Enterprise AI Agents in Production: Why Do They Keep Failing? As large language models (LLMs) have rapidly advanced, enterprises have…
OpenEnv 實戰：在真實世界環境中評估具備工具使用能力的 AI Agent★ 75
Hugging Face Blog122 days agoNew ToolAs AI Agent (intelligent agent) technology advances rapidly, evaluating how these agents perform in the real world has become one of the greatest challenges…
Alyah ⭐️：邁向阿拉伯語大型語言模型中阿聯酋方言能力的強健評估
Hugging Face Blog138 days agoReleaseAs Arabic large language models (LLMs) develop rapidly, accurately evaluating model performance across different regional dialects has become a significant…
AssetOpsBench：彌合 AI Agent 評估基準與工業實際應用差距的全新基準測試★ 75
Hugging Face Blog144 days agoReleaseIn today's era of rapid development in AI Agent technology, how to evaluate the performance of these Agents in real-world settings — particularly in industrial…
Google DeepMind 推出 FACTS 基準測試套件：系統化評估大型語言模型的真實性★ 80
Google DeepMind Blog187 days agoReleaseAs large language models (LLMs) are deployed across a wide range of industries, ensuring the "factuality" of model outputs and reducing "hallucination" has…
Hugging Face 推出 BigCodeArena：透過實際執行程式碼進行端到端 Code LLM 評測★ 75
Hugging Face Blog250 days agoReleaseHugging Face and the BigCode community have jointly launched a new code model evaluation platform called "BigCodeArena." As AI-assisted coding (such as Copilot…
Hugging Face 推出 RTEB：全新檢索評估標準，為 RAG 系統打造更真實的測試基準★ 80
Hugging Face Blog256 days agoReleaseAs Retrieval-Augmented Generation (RAG) becomes the dominant architecture for enterprises deploying large language models (LLMs), accurately evaluating the…
Hugging Face 推出 Gaia2 與 ARE：賦能社群深入研究 AI Agent★ 85
Hugging Face Blog265 days agoReleaseAI agents are currently the hottest research direction in the AI field, but how to objectively, safely, and reproducibly evaluate agent capabilities has long…
FilBench 發布：大型語言模型真的懂菲律賓語嗎？全新評測基準登場
Hugging Face Blog306 days agoReleaseThe Hugging Face team and community have collaborated to launch a new evaluation benchmark called "FilBench," aimed at answering a key question: do large…
TextQuests：LLM 在文字冒險遊戲中的表現究竟如何？Hugging Face 推出全新評估基準★ 75
Hugging Face Blog306 days agoReleaseHugging Face has recently introduced a new benchmark called "TextQuests," designed to evaluate the performance of large language models (LLMs) in text-based…
在 DeepResearch Bench 評測開源 Llama Nemotron 模型：NVIDIA 打造頂尖且可移植的深度研究 Agent★ 80
Hugging Face Blog313 days agoReleaseThis article provides a detailed look at how NVIDIA is using its open-source Llama Nemotron series of models to evaluate and build top-performing, portable…
📚 3LM：針對阿拉伯語大語言模型在 STEM 與程式碼能力的全新評估基準
Hugging Face Blog317 days agoReleaseThe Technology Innovation Institute (TII) of the UAE — the organization behind the Falcon models — has announced on the Hugging Face blog the launch of a new…
TimeScope：評估影片大型多模態模型（Video LMM）長影片理解極限的新基準★ 75
Hugging Face Blog326 days agoReleaseAs large multimodal models (LMMs) have achieved breakthroughs in image and short-video understanding, the industry has gradually shifted its attention to the…
回到未來：Hugging Face 推出 FutureBench 評估 AI Agent 的未來事件預測能力★ 75
Hugging Face Blog332 days agoRelease### What is FutureBench? As large language models (LLMs) and AI agents have rapidly advanced, traditional static benchmarks (such as MMLU and GSM8K) face a…
ScreenEnv：部署你的全端桌面 AI 代理（Desktop Agent）環境★ 82
Hugging Face Blog339 days agoNew ToolWith the rise of Anthropic's Claude 3.5 Sonnet "Computer Use" and various GUI-oriented multimodal models, "desktop agents" have become one of the hottest areas…
Hugging Face 推出 ScreenSuite：最全面的 GUI Agent 評估套件！★ 80
Hugging Face Blog373 days agoNew ToolAs artificial intelligence moves beyond simple "text-based conversation" into the era of Agents (intelligent agents) that actively execute tasks, enabling AI…
介紹 HELMET：全面評估長文本語言模型（Long-context LLMs）的新一代基準測試★ 80
Hugging Face Blog424 days agoRelease### Background and Pain Points: Moving Beyond the Overly Simple "Needle in a Haystack" Test In recent years, the context window length supported by large…
Hugging Face 推出 DABStep：評估數據代理多步驟推理能力的全新基準測試★ 75
Hugging Face Blog495 days agoReleaseAs large language model (LLM) technology has evolved, AI has transformed from a simple question-answering assistant into an "AI agent" capable of proactively…