Kog builds a co-designed inference stack for real-time AI agents on standard NVIDIA and AMD datacenter GPUs, spanning model architecture, inference engine, compilers, and low-level GPU kernels.
We developed Laneformer 2B and Delayed Tensor Parallelism (DTP), a Transformer architecture designed to overlap communication with computation.
Our inference stack generates 3,500 tokens/s per request on 8 AMD MI300X GPUs and 2,100 tokens/s per request on 8 NVIDIA H200 GPUs, in FP16 at batch size 1, without quantization or speculative decoding.
Our next challenge is making model optimization faster and more scalable. New models are released constantly, and spending months adapting each one to a specific GPU setup is not sustainable.
This is why we're developing AGCO, our agentic compiler initiative. The goal is to build on our existing optimization techniques to dramatically reduce the time needed to port and optimize models across different GPU architectures, progressively automating parts of that process.
We're a team of 10 people, including three PhD holders and one PhD candidate.
Try our technology at playground.kog.ai and read more on the Kog Labs blog.
