Original Summary

I’ve been building an open-source side project called TensorSharp , a local LLM inference engine and agent runtime written in C#/.NET: TensorSharp on GitHub One of the problems I’ve been experimenting with is: How far can we push huge MoE models on ordinary consumer hardware? My latest test is Qwen3.8 Flash Next (176B parameters) . And surprisingly, this hardware is enough to run it: RTX 3080 Laptop — 16GB VRAM + 32GB system RAM + SSD The main challenge isn't simply quantizing the model until it fits. A 176B model is still far larger than the available VRAM and RAM. So I implemented an MoE-aware memory/offloading architecture that coordinates: GPU cache / VRAM ↔ system RAM ↔ SSD Instead of treating SSD as a last-resort swap space, TensorSharp schedules and caches model data across the memory hierarchy based on the execution characteristics of MoE models. I also benchmarked the implementation against Strata on the same machine. Results are in the attached screenshot: Measurement TensorSharp Strata Decode 11.09 tok/s 10.24 tok/s Whole process 16.54s 62.15s Peak GPU memory 14,832.5 MiB 15,729 MiB Peak OS working set 19.74 GiB 18.51 GiB What surprised me most wasn't the decode speed—the two are actually fairly close there. It was that a 176B model can be made usable on a laptop with only 16GB VRAM and 32GB RAM if the inference engine treats VRAM, RAM, SSD, caching, and MoE scheduling as one system rather than independent pieces. TensorSharp started as an experiment in understanding LLM inference from the bottom up, but it has gradually grown into a much larger project covering local inference, multimodal models, image generation, and agentic runtimes. It’s open source, so if this kind of low-level LLM engineering interests you, I’d love feedback: GitHub: github.com/zhongkaifu/TensorSharp And yes, the benchmark in the screenshot really is from an RTX 3080 laptop. 😄   submitted by   /u/fuzhongkai [link]   [comments]

  • 情报分类：硬件与数码
  • 分类依据：内容涉及硬件、数码产品或通信卡
  • 信息来源：Reddit · SideProject
  • 发布时间：2026/10/4 04:59:20