moonshotai

/

Kimi-K3

like 10.5k

Follow

Moonshot AI 16.7k

goated

This is a hallucination. First, it's not possible to get that hardware anywhere near that cheaply. Second, it'd be very slow, likely entirely unusable for a model of this size.

In which universe are you living ? Currently 32gb of DDR5 RDIMM costs you around 1k at least. so for 1.5TB ~ 50k.

What are you daydreaming about? Spitting out words one by one is already pretty fast.

keen to see tokens/sec on the hardware you described

I created an account just to mention. 128gb ddr5 in aliexpress second hand is arround ~2k usd. and DDR4 is ~1.8k USD. So Yeah, impossible

That's a great idea, and I'm already putting it into practice. Given the exorbitant price of DDR5 server memory, I settled on the final generation DDR4 server platform as a more cost-effective alternative.

1. Hardware Selection & Costs
   I selected the Intel C621A chipset paired with 3rd Gen Intel® Xeon® Scalable Processors. Thanks to the abundance of ES (Engineering Sample) server CPUs on the Chinese second-hand market—and the fact that 3rd Gen Xeon ES versions are known for having relatively few bugs—I can easily acquire an 8368 ES (38 cores, 76 threads, Ice Lake architecture, Stepping 6—same as the retail version) for just $100–150. Matching DDR4 16GB 3200 memory modules cost about $90 per stick.
   Adding a decommissioned 1U server chassis (including the full set of power supplies, coolers, and motherboard) comes to around $400.
   All in all, the total cost to assemble a single dual-socket server is approximately $2,200.
2. Networking & Cluster Architecture
   For networking, I'm using Mellanox CX5 series cards with MCP1600 100G cables to attempt an InfiniBand interconnect.
   The total investment for building two dual-socket servers with this InfiniBand setup is around $5,000.
   The entire cluster possesses approximately 512GB of DDR4 memory, with a theoretical concurrent bandwidth of 800G (effectively around 600G).
3. Software Stack Comparison
   Regarding the software stack, I evaluated two main approaches:
   Option 1: Standard, Commercial-Grade Server Frameworks
   This involves vLLM and SGLang. There are some enhanced forks on GitHub tailored for personal deployment, such as lvllm, which are extensions of Tsinghua University's KTransformers framework.
   Cons: Most of these inference backends do not support GGUF and rely on standard w4a16 computation, leading to excessive memory requirements. Furthermore, they basically don't support pure CPU inference, and the performance is mediocre.
   Benchmarks: With a dual-socket 8368 setup and two RTX 3090s, the generation speed (TG) is barely above 10 tokens/s（Dspark Enable will be ≈30toks）.
   Option 2: Heavily CPU-Optimized llama.cpp Fork (My Current Focus)
   I had Kimi-k3 review the code and discovered that the original version missed many optimization opportunities. So, I rewrote significant portions:
   Optimizations: Added repack support for low-quantization operators; fixed the compute path to properly utilize AVX-512; enabled newer acceleration instructions like VNNI and VBMI; implemented operator fusion and barrier optimizations; and added support for weight/KV mirroring as well as CPU EP.
   Benchmarks (DeepSeek V4 Flash @ Q2):
   Pure CPU: Prefill (PP) speed is approx. 50 tokens/s; Generation (TG) speed is approx. 15 tokens/s.
   With Dual 3090s: Generation (TG) speed reaches approx. 30+ tokens/s.
4. Current Progress & Goals
   Currently, I am working hard to implement Tensor Parallelism within llama.cpp.
   Right now, only one RTX 3090 is handling the main compute acceleration, while the second card is solely dedicated to MoE computation. If I can successfully offload the Dense layers, Router, and Attention mechanisms of such a massive MoE model across the GPUs, I expect to see a significant boost in actual performance.

• Modified and translate by LLM, because im a Chinese. I promise the data is TRUE and NOT the AI Hallucinations.
  

Let me know where you're getting hardware at that price 😭

Let me know where you're getting hardware at that price 😭
China's biggist second-hand online market Xianyu& ebay actually

@heiketu that's actually really cool, I didn't know that mb- just saw the setup :)

@heiketu Huge thanks to you for your expansion on this theory/methodology with real data! and even photos! I feel like I did something here 😄! I am usually always working in the stealth background on these things, alone. I was just compelled to share something and tell everyone it is not hopeless! personally my first success in the deep planing and iterative process of making local & powerful AI possible at home (risky as it was as I have less money than average). Was with my 4xV100 NVlinked 128GB HBM2 system! I have spend weeks to months, but more and more heavily focused the last few weeks setting up my first first absolutely incredible system that was less than half the price of 2x RTX5090's for double the vram and when patched is way more performant, especially memory bandwidth wise, not to mention possesses real ECC memory and enterprise reliability AND the real potential to TRAIN your own models as you cannot reliably do with RTX cards.

Let's keep this going, I will try to post some photos and plan some more when I have finally got some sleep! it is now 6am here...

lets go, i will continue follow your process!

The V100 does seem viable. Maybe you could snag a few more V100s on Xianyu. But honestly, the AI performance of the V100 was never anything to write home about, and on top of that there's no NVLink... Still, just getting it running is already pretty impressive. I think pairing it with something like MTP might make it a bit more usable, though.

Interesting thread, but obviously you guys forget Kimi K3 has reasoning. So 3 tokens/sec means you likely wait AT LEAST 5 mintes (for 1000 tokens of reasoning) before begining to see the answer.

https://github.com/heiketu/PeoplesLLM
try this

I want

I want too