Towards Auto-Regressive Next-Token Prediction: In-context Learning Emerges from Generalization
Zixuan Gong*, Xiaolin Hu*, Huayi Tang, Yong Liu (* Equal contribution)

• We explore the emergence of in-context learning (ICL) capabilities in auto-regressive next-token prediction models.
• To bridge the pre-training and ICL phases, we introduce a two-level expectation over data and topic distributions, providing PAC-Bayes generalization bounds to support our analysis.
• Additionally, we model the training process using Stochastic Differential Equations (SDEs), demonstrating that ICL arises from the exceptional generalization across sequences and topics.

ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
Pengwei Tang, Xiaolin Hu, Yong Liu

• We propose Adaptive Decomposed Prompt Tuning (ADePT), which can produce unique token embedding offset for each token.
• ADePT addresses the limitations of DePT, enabling better optimization and generalization without increasing inference time or parameters.
• Experiments on 23 NLP tasks and 4 PLMs show ADePT outperforms leading PEFT methods and even full fine-tuning in some cases.

Enhancing In-Context Learning with just SVD-Based Pruning: A Theoretical Perspective
Xinhao Yao, Xiaolin Hu, Shenzhi Yang, Yong Liu

• We show an exciting phenomenon that SVD-based weight pruning can enhance In-Context Learning (ICL) performance.
• we conduct theoretical analysis by presenting the implicit gradient descent (GD) of ICL and giving generalization bounds of ICL.
• We further propose a simple, derivative-free algorithm to enhance ICL. Experiments demonstrate its effectiveness.

PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
Qibin Wang, Xiaolin Hu, Weikai Xu, Wei Liu, Jian Luan, Bin Wang

• We propose PMSS, enabling high-rank updates at low costs by selecting skeletons from pre-trained weights.
• PMSS overcomes LoRA’s low-rank limitations and optimizes initialization to utilize semantic and linguistic information.
• Experiments show PMSS outperforms LoRA and excels in tasks like DROP and math reasoning with fewer trainable parameters.

Neural Retrievers are Biased Towards LLM-Generated Content
Sunhao Dai, Yuqi Zhou, Liang Pang, Weihao Liu, Xiaolin Hu, Yong Liu, Xiao Zhang, Gang Wang, Jun Xu

• We explore how LLM-generated texts influence IR systems, revealing a source bias where neural models favor LLM-generated documents.
• We use information theory to explain this bias, showing it arises from the focused semantics of LLM-generated content.

Chemical knowledge-informed framework for privacy-aware retrosynthesis learning
Guikun Chen, Xu Zhang, Xiaolin Hu, Yong Liu, Yi Yang & Wenguan Wang

• CKIF presents a privacy-preserving framework for decentralized retrosynthesis learning across chemical organizations.
• It employs chemical knowledge-informed aggregation, where molecular properties guide adaptive weighting of local models.
• Experiments on multiple reaction datasets show CKIF achieves higher accuracy than strong baselines.

Stability and Generalization of Zeroth-Order Decentralized Stochastic Gradient Descent with Changing Topology
Xiaolin Hu, Zixuan Gong, Gengze Xu, Wei Liu, Jian Luan, Bin Wang, Yong Liu (Oral)

• This paper provides the first generalization analysis of ZO-DSGD with changing topology.
• The obtained generalization bounds align with SGD in (strongly) convex cases and with DSGD in non-convex cases.
• The results reflect the impact of client count, sample size, and topology on generalization performance.

Generalization Bounds for Federated Learning: Fast Rates, Unparticipating Clients and Unbounded Losses
Xiaolin Hu, Shaojie Li, Yong Liu

Video

• We present a theoretical analysis of the generalization error for non-participating clients in federated learning.
• The obtained generalization bounds in high probability form capture the performance of a single trial, rather than the average over multiple trials.
• We derive generalization bounds for heavy-tail losses, applicable to federated learning with unbounded losses, such as cross-entropy.

A Deep Learning Framework for Solving Rectangular Waveguide Problems
Xiaolin Hu, Nicholas E. Buri, APMC 2020 (Oral) |

Project

• We employ Physics Informed Neural Networks (PINNs) to solve rectangular waveguide problems.
• We successfully apply PINNs to the task of solving electric and magnetic fields, which can be described by partial differential equations (PDEs).
• We also show the applicability of the framework for predicting the unknown parameters such as wavenumber.