Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

使用29GB内存以0.50 tok/s速度运行Kimi K3

HN 235 分 · 96 条评论 · 作者 marcobambini · 来源 github.com · HN 讨论

【摘要】

Kimi K3, a model with 2.78 trillion parameters, can now run on a consumer laptop using WASTE, an embeddable inference engine written in C with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses remaining RAM as a bounded expert cache.

⋯ 继续阅读请登录会员 ⋯