Skip to content
CaMLSysUniversity of Cambridge
Home People Publications Projects Contact

← All publications

AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling

  • Preslav AleksandrovUniversity of CambridgeMeta
  • Meghdad KurmanjiUniversity of Cambridge
  • Fernando Garcia RedondoIMEC
  • David O'SheaUniversity of Cambridge
  • William ShenUniversity of Cambridge
  • Alex IacobUniversity of Cambridge
  • Lorenzo SaniUniversity of Cambridge
  • Xinchi QiuUniversity of CambridgeMeta
  • Nicola CanceddaMeta
  • Nicholas D. LaneUniversity of Cambridge

Preprint arXiv, 2025

Abstract

We introduce the Autoregressive Block-Based Iterative Encoder (AbbIE), a novel recursive generalization of the encoder-only Transformer architecture, which achieves better perplexity than a standard Transformer and allows for the dynamic scaling of compute resources at test time. This simple, recursive approach is a complement to scaling large language model (LLM) performance through parameter and token counts. AbbIE performs its iterations in latent space, but unlike latent reasoning models, does not require a specialized dataset or training protocol. We show that AbbIE upward generalizes (ability to generalize to arbitrary iteration lengths) at test time by only using 2 iterations during train time, far outperforming alternative iterative methods. AbbIE’s ability to scale its computational expenditure based on the complexity of the task gives it an up to 12% improvement in zero-shot in-context learning tasks versus other iterative and standard methods and up to 5% improvement in language perplexity. The results from this study open a new avenue to Transformer performance scaling. We perform all of our evaluations on model sizes up to 350M parameters.

Materials

  • PDF
BibTeX
@misc{aleksandrov2025abbie,
  title = {{AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling}},
  author = {Aleksandrov, Preslav and Kurmanji, Meghdad and Redondo, Fernando Garcia and O'Shea, David and Shen, William and Iacob, Alex and Sani, Lorenzo and Qiu, Xinchi and Cancedda, Nicola and Lane, Nicholas D.},
  year = {2025},
  url = {https://arxiv.org/abs/2507.08567},
  eprint = {2507.08567},
  archiveprefix = {arXiv}
}
Cambridge Machine Learning Systems Group
Department of Computer Science and Technology
University of Cambridge
ContactGitHub