Benchmarks for Text Generation with Large Language Models

大语言模型文本生成评价进展

   The text generation capability of Large Language Models (LLMs) is extremely powerful and not limited to any type of topic, style, language, etc. This makes it necessary to consider a variety of different dimensions in the evaluation of the generation capabilities of LLMs. Moreover, due to the diversity of language expressions, there is usually no single ground truth for text generation, which also makes the evaluation more difficult. The evaluation of LLM text generation also faces issues such as safety. This report will introduce the main challenges and methods in the evaluation of LLM text generation, as well as some of the progresses we have achieved in this field.

大语言模型（LLMs）的文本生成能力非常强大，并且不限定于任何类型的话题、风格、语言等，这使得对大语言模型生成能力的评价需要考虑多种不同的维度。另外，由于语言表达的多样性，文本生成通常并不存在统一的标准答案，这也使得这种评价变得更加困难。大语言模型文本生成的评价还面临安全性等问题。本报告将介绍大语言模型文本生成评价面临的主要挑战和方法，并介绍我们在这一领域取得的一些成果。

LIU Qun, Chief Scientist of Speech and Language Computing of Huawei, Professor, ACL Fellow. From 2018, He has been leading the Speech and NLP group of Huawei Noah's Ark Lab, which supports Huawei's products and services by developing a series of technologies including machine translation, dialog systems, speech recognition and synthesis, pre-trained large language models, etc. Before joining Huawei, he was a Full Professor at the School of Computing of Dublin City University and the NLP Theme Leader of the ADAPT Centre, Ireland from 2012. Prior to that, he was as a Researcher and Professor in the Institute of Computing Technology, Chinese Academy of Sciences for 20 years, where he founded and led the NLP Research Group. He obtained his B.Sc., M.Sc. and Ph.D. in computer science in the University of Science and Technology of China, Chinese Academy of Sciences, and Peking University respectively. His research interests lie in the area of Natural Language Processing, with achievements on Chinese word segmentation and POS tagging, statistical and neural machine translation, pre-trained language models, question-answering and dialog systems, etc. He has authored or co-authored more than 300 peer-reviewed research publications, with over 20,000 citations. He has supervised more than 50 students to the completion of their M.Sc. or Ph.D. degrees. He has received numerous awards, including Google Research Award (2012), ACL Best Long Paper (2018), Qian Weichang Chinese Information Processing Award (2010), China's National Science & Technology Progress Prize (2015), IAMT Honor Award (2023).

刘群，华为语音语义首席科学家，教授，ACL Fellow。从2018年起，他领导了华为诺亚方舟实验室的语音语义团队，该团队开发了包括机器翻译、对话系统、语音识别与合成、预训练大语言模型等一系列技术，为华为公司的产品和服务提供了有力支持。加入华为之前，从2012年起，他是爱尔兰都柏林城市大学教授、爱尔兰ADAPT中心自然语言处理主题负责人。在此之前，他在中国科学院计算技术研究所工作了20年并担任研究员职位，创建了自然语言处理研究组并担任负责人。他分别在中国科学技术大学、中科院计算所、北京大学获得计算机学士、硕士和博士学位。他的主要研究方向是自然语言处理，研究成果包括汉语词语切分和词性标注系统、统计和神经机器翻译、预训练语言模型、问答和对话系统等。他在专业会议和期刊上发表300多篇论文被引用20000多次，培养国内外博士硕士毕业生50多人。他获得过Google Research Award、ACL Best Long Paper、钱伟长中文信息处理科学技术奖一等奖、国家科技进步二等奖、IAMT Honor Award等众多奖项。