описание
The client is building a real-time AI voice platform and developing large language model inference capabilities for production use.
задачи
- Build and optimize a vLLM-based inference stack for low latency, high throughput, and maximum GPU utilization
- Optimize continuous batching, scheduling, prefill/decode, prefix caching, and KV-cache allocation and reuse
- Optimize or implement CUDA and Triton kernels for attention, GEMMs, sampling, normalization, and other critical model operations
- Evaluate and integrate FlashAttention, FlashInfer, CUDA Graphs, torch.compile, speculative decoding, and quantization
- Optimize distributed inference using tensor, data, and expert parallelism, NCCL, NVLink/NVSwitch, and InfiniBand
- Work with researchers to bring new dense and MoE model architectures into production
- Take ownership from profiling through production deployment
требования
- Deep hands-on experience with LLM inference, ideally inside vLLM, SGLang, TensorRT-LLM, or similar systems, not just deploying them
- Understanding of TTFT, inter-token latency, throughput, continuous batching, PagedAttention, KV caching, and prefill versus decode performance
- Ability to profile GPUs and reason about compute, memory bandwidth, kernel launches, synchronization, and communication bottlenecks
- Experience with CUDA, Triton, PyTorch, NCCL, and modern NVIDIA GPU architectures
- Understanding of Transformer internals, including MHA/GQA, ROPE, KV cache, quantization, and MoE
- Experience optimizing distributed, performance-critical systems in production
- Care about performance, simplicity, and reliability
условия
- Условий в вакансии нет