Tag: hardware

Detailed build log for a $3k home inference server with 128GB VRAM (4x V620) and 256GB system RAM. Power consumption runs 700-900W during prefill and 500-600W during decode, but enables running large models locally. Build uses server-grade components after abandoning a Lenovo P620 workstation.

A user launched vram.wiki, a community database for comparing local LLM hardware setups with actual performance metrics. Users can contribute their own setups (hardware, model, quant, runtime, harness, context size) and see why similar configurations achieve different speeds.

The RTX 5090 has sold out across US retailers with scalpers charging up to $9,500 (6x+ MSRP). This suggests either genuine demand from AI workloads or speculative hoarding, with major implications for local AI deployment costs.

Xiaomi unveiled a prototype AI Cube featuring three chips (Xuanjie O3, O100, and D100) with impressive specifications including 1.22TB/s memory bandwidth and support for up to 160GB RAM. This represents a significant move by a consumer electronics company into dedicated AI hardware, potentially democratizing access to high-performance local AI inference.

Apple announced the new Mac Studio with M5 Max and M5 Ultra chips supporting up to 512GB of unified memory. This massive memory capacity in consumer hardware makes it viable to run large language models locally without specialized server equipment, potentially democratizing access to high-performance local AI.

Announcement or leak of Apple's M5 Server chip, suggesting Apple is developing server-class silicon optimized for AI workloads. This could represent Apple's entry into the AI infrastructure market beyond consumer devices.

A researcher simulated cosmic ray bit flips on LLM weights to test resilience in space/high-radiation environments. Results show LLMs are highly sensitive to bit flips, degrading rapidly under radiation exposure—important for deploying AI in satellites, spacecraft, or other radiation-prone environments.

This post checks in on the status of Huawei GPUs nearly a year after initial hype about breaking NVIDIA's monopoly. The discussion reveals the reality of hardware alternatives in the AI acceleration space and provides ground truth on whether alternative GPU architectures have materialized for local AI workloads.

A detailed account of building extreme local hardware infrastructure to run GLM-5.2, escalating from a single 5090 to a multi-GPU setup with full PCIe 5.0 x16 across all slots. This post offers valuable insights into the practical challenges and cost escalation of running frontier-scale models locally.

Tensordyne announces breakthrough inference chip using logarithmic math hardware for dramatically improved efficiency: 17x better power efficiency and 13x higher throughput versus NVIDIA Blackwell. Claims efficient log-space computation as the key mathematical breakthrough.

Analysis of optimal budget hardware for running Qwen 3.6 models (27B and 35B-A3B) targeting 40+ tok/s. Compares RTX 3090 24GB, RTX 3080 20GB, and controversial Tesla V100 32GB options. Community consensus favors RTX 3090 for broader future compatibility.

A practical tip for running ~30B parameter models on consumer hardware: combining a modern 16GB card (like 5070Ti) with an older 6GB card (like RTX 2060) enables running larger models by splitting layers across GPUs. The key insight is that fitting everything in VRAM matters more than having matching GPUs, even if one card is significantly weaker.

Leaks describe OpenAI's wearable audio device: metal "eggstone" design worn behind the ear, powered by custom 2nm Samsung Exynos chip designed to command Siri and replace iPhone actions. Bill of materials closer to smartphone than earbuds. The Jony Ive collaboration has apparently prioritized this project.

Qwen team lead publicly states that Chinese AI companies are severely bottlenecked by inference compute rather than training compute. This suggests export controls on inference chips may be more impactful than training restrictions. The comment provides rare insight into how sanctions affect Chinese AI development.