Foundation Model

We propose AdaInfer, a lightweight algorithm that adaptively stops LLM inference early based on statistical cues, cutting up to 43% of computation with <1% performance loss and no model modification.

We propose Few-Shot Detector (FSD), a novel AI-generated image detector that learns a metric space to recognize unseen fake images with only a few samples, achieving +11.6% higher accuracy and strong generalization without retraining.

As scaling laws underscore the potential of increasing model sizes, the academic community has intensified its investigations into LLMs with capacities exceeding 50 billion parameters. This technical report builds on our prior work with Tele-FLM (also known as FLM-2), a publicly available 52-billion-parameter model.

To lower the computional cost of training large model, we focus on speeding up pre-training by progressively growing from a small Transformer structure to a large one.

We find that Maximal Update parametrization (uP) enables accurate fitting of scaling laws for hyperparameters close to common loss basins, without any search. Thus, different models can be directly compared on large scales with loss prediction even before the training starts. We propose a new paradigm as a first step towards reliable academic research for any model scale without heavy computation.