The 3-Tier MoE Streaming Architecture: Unlocking NVMe for Local Inference
Open-source developer mw00 (peasantsmith) has published Project Maya, an MIT-licensed specialized Mixture-of-Experts (MoE) inference engine built as a derivative of the Strata architecture. Engineered specifically to execute Zhipu AI's flagship GLM-5.3-Flash—a model housing 321 billion total parameters with approximately 18 billion active parameters routed per token—Maya removes the requirement for multi-GPU data center infrastructure through an innovative 3-tier hierarchical expert offloading strategy.
Under Maya's architecture, memory management is split across physical hardware boundaries. Frequently routed ('hot') MoE experts are pinned directly inside GPU VRAM, medium-frequency experts reside in host system DDR RAM, and cold tail experts are dynamically streamed on demand from high-speed NVMe storage. By shuttling experts in and out of execution memory during token decoding, Maya allows a single machine to host and run a 321B frontier-class foundation model entirely on premises.
Quantization Profiles and Benchmark Accuracy: Validating Real-World Throughput
Alongside the inference runtime, peasant-smith published four specialized GGUF quantization builds on Hugging Face: Maya-S24 (94.7 GB, engineered for 24 GB cards like the RTX 3090 and RTX 4090), Maya-S (96.5 GB), Maya-M (116 GB), and Maya-L (156.3 GB). Empirical benchmarks indicate that Maya-L retains 99.2% of full FP8 zero-shot evaluation accuracy across standard benchmark suites including ARC, HellaSwag, WinoGrande, and PIQA, while the highly compressed Maya-S retains 97.9%.
In terms of generation speed, Maya achieves over 30 tokens per second (tok/s) on a single high-end consumer GPU such as the RTX 5090 or dual older workstation cards, scaling up to 118 tok/s on quad-GPU server topologies. The engine natively supports context windows up to 1 million tokens and provides drop-in API compatibility for existing OpenAI and Anthropic client SDKs. While Linux remains the primary production target, experimental AMD ROCm support was merged into the mainline in release v1.0.11.
Developer and Practitioner Reaction: Homelab Autonomy vs. Hardware Price Squeeze
The release has ignited substantial interest across engineering forums and homelab communities. Practitioners voiced enthusiasm that massive 300B+ MoE models are transitioning from theoretical data-center artifacts into usable daily drivers for software development on sub-$2,000 workstations, liberating teams from vendor API quotas and cloud-imposed censorship constraints. Independent testing verified that pairing high-capacity system memory (such as 256 GB DDR4/DDR5) with dual consumer GPUs yielded responsive decoding speeds and sustained high prefill rates.
At the same time, practitioners highlighted significant operational caveats. Inference throughput remains deeply bounded by system bus bandwidth (PCIe Gen 4 or Gen 5) and NVMe I/O operations per second. Systems equipped with less than 64 GB to 128 GB of host RAM experience severe generation throttling whenever cold experts must be paged from disk. Furthermore, developers noted the broader market irony: while software optimizations continue to compress memory footprints, physical hardware availability is tightening, with gaming GPUs, enterprise DDR5 memory, and fast NVMe drives facing rapid price inflation as enterprises aggressively secure supply.
Strategic Implications for Thai Enterprises: Sovereign AI and Data Governance
For enterprise technology leaders in Thailand—particularly within regulated verticals such as financial services, telecommunications, and healthcare governed by the Personal Data Protection Act (PDPA)—architectures like Project Maya offer a direct path toward data sovereignty. Running a 321B foundation model on local on-premises hardware ensures that customer PII and proprietary internal datasets remain strictly within company firewalls without transiting cross-border cloud APIs.
Furthermore, the economic implications for Thai digital transformation budgets are profound. Rather than allocating capital expenditures toward six-figure specialized AI server clusters or absorbing unpredictable API token billing, organizations can deploy internal coding and automation engines on upgraded local workstations equipped with RTX-tier GPUs and high-density RAM. This provides domestic software teams with access to top-tier reasoning capabilities while retaining total control over their operational infrastructure.
Project Maya dismantles the enterprise hardware barrier for data-sensitive workflows by bringing frontier-class 321B inference to on-premises workstations and local homelabs without cloud lock-in.