Microsoft Research Team Proposes LLM Accelerator LLMA

20 Nis 2023

A group of researchers at Microsoft proposes the LLM Accelerator LLMA. It is reported that. This inference decoding technique with references can speed up LLM inference in many real-world settings by exploiting the overlap between the output of the LLM and the references. LLMA works by selecting a span of text from the reference, copying its tokens into the LLM decoder, and then doing efficient parallel inspection based on the output token probabilities.