TerEffic introduces a breakthrough approach for running large language models on edge devices through specialized hardware design and extreme quantization techniques.

• Achieves on-chip inference for LLMs by reducing memory footprint with ternary quantization (weights as -1, 0, or 1)
• Co-designs memory architecture and computational units specifically for ternary models
• Enables edge deployment with lower power consumption and higher throughput
• Demonstrates how specialized hardware can overcome traditional LLM deployment constraints

This innovation matters because it opens possibilities for running sophisticated AI models in environments where cloud connectivity, power, or latency constraints previously made LLM deployment impractical.

TerEffic: Highly Efficient Ternary LLM Inference on FPGA