Artificial Intelligence Fundamental Model Support Platform and Evaluation Technology (National Key R&D Program)

Nov 1, 2022

This project focuses on the development of large models, with the aim of promoting the advancement of cutting-edge technologies for ultra-large-scale intelligent models in China and facilitating the empowerment of artificial intelligence for economic and social development. It aims to build an internationally leading open-source technology system for fundamental models and establish an open innovation ecosystem centered around foundational models in artificial intelligence.

Yequan Wang

Researcher, Team Leader

My research interests include large model, emboddied AI and NLP.

Publications

Not All Layers of LLMs Are Necessary During Inference

We propose AdaInfer, a lightweight algorithm that adaptively stops LLM inference early based on statistical cues, cutting up to 43% of computation with <1% performance loss and no model modification.

Siqi Fan, Xin Jiang, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang

Few-Shot Learner Generalizes Across AI-Generated Image Detection

We propose Few-Shot Detector (FSD), a novel AI-generated image detector that learns a metric space to recognize unseen fake images with only a few samples, achieving +11.6% higher accuracy and strong generalization without retraining.

Shiyu Wu, Jing Liu, Jing Li, Yequan Wang

52B to 1T: Lessons Learned via Tele-FLM Series

As scaling laws underscore the potential of increasing model sizes, the academic community has intensified its investigations into LLMs with capacities exceeding 50 billion parameters. This technical report builds on our prior work with Tele-FLM (also known as FLM-2), a publicly available 52-billion-parameter model.

Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, China Telecom, Yequan Wang, Zhongjiang He, Zhongyuan Wang, Xuelong Li, Tiejun Huang

Masked Structural Growth for 2x Faster Language Model Pre-training

To lower the computional cost of training large model, we focus on speeding up pre-training by progressively growing from a small Transformer structure to a large one.

Yiqun Yao, Zheng Zhang, Jing Li, Yequan Wang

Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scales

We find that Maximal Update parametrization (uP) enables accurate fitting of scaling laws for hyperparameters close to common loss basins, without any search. Thus, different models can be directly compared on large scales with loss prediction even before the training starts. We propose a new paradigm as a first step towards reliable academic research for any model scale without heavy computation.

Yiqun Yao, Yequan Wang

Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scales