Main topics I worked on in HKRC Huawei

Created on July 15, 2025

2025

Agentic RAG acceleration

  • For the central control unit of XiaoYi AI Agent system, we accelerate the performance (TTFT) to more than 4x, by using more efficient attention mechanism
  • SFT (32B model) pipeline which guarantees the accuracy loss < 1%

Hardware-based inference optimization

  • Optimize the decoding attention kernel (a key kernel of the inference framework) based on the GQA and NPU architecture
  • This solution achieved 4x speed-up and was delivered as Ascend NPU Kernel (decoding attention) for xiaoyi commercial vLLM inference framework