Foundation Model

我们提出了 AdaInfer，一种基于统计特征自适应提前终止 LLM 推理的轻量算法，可在不修改模型的情况下减少 最高 43% 的计算量，性能下降不足 1%。

我们提出了 Few-Shot Detector (FSD)，一种通过学习度量空间、仅需少量样本即可识别未见伪造图像的检测器，能在无需再训练的情况下实现 准确率提升 11.6% 并保持强泛化能力。

As scaling laws underscore the potential of increasing model sizes, the academic community has intensified its investigations into LLMs with capacities exceeding 50 billion parameters. This technical report builds on our prior work with Tele-FLM (also known as FLM-2), a publicly available 52-billion-parameter model.

To lower the computional cost of training large model, we focus on speeding up pre-training by progressively growing from a small Transformer structure to a large one.

We find that Maximal Update parametrization (uP) enables accurate fitting of scaling laws for hyperparameters close to common loss basins, without any search. Thus, different models can be directly compared on large scales with loss prediction even before the training starts. We propose a new paradigm as a first step towards reliable academic research for any model scale without heavy computation.