EMNLP 2025 Papers — Page 2
Conference on Empirical Methods in Natural Language Processing · 1809 papers
Annotating Training Data for Conditional Semantic Textual Similarity Measurement using Large Language Models
Gaifan Zhang (University of Liverpool), Danushka Bollegala (University of Liverpool)
Data-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: Utilize large language models to automatically clean and re-annotate the conditions and similarity scores of the C-STS dataset, combining them with human annotations to construct a higher-quality training set.
Answer Convergence as a Signal for Early Stopping in Reasoning
Xin Liu (University of Michigan), Lu Wang (University of Michigan)
Computational EfficiencyRecurrent Neural NetworkLarge Language ModelTextBenchmarkChain-of-Thought
🎯 What it does: This paper investigates the phenomenon of answer convergence in Chain-of-Thought (CoT) and proposes three methods for dynamically early stopping during the reasoning process to reduce LLM inference costs.
Answering Narrative-Driven Recommendation Queries via a Retrieve–Rank Paradigm and the OCG-Agent
Yunxiao Shi (University of Technology Sydney), Min Xu (University of Technology Sydney)
Recommendation SystemLarge Language ModelAgentic AIPrompt EngineeringTextBenchmarkRetrieval-Augmented GenerationChain-of-Thought
🎯 What it does: Propose a retrieval-ranking framework for narrative-driven recommendation, and develop OCG-Agent to achieve wide and deep candidate retrieval.
AnyMAC: Cascading Flexible Multi-Agent Collaboration via Next-Agent Prediction
Song Wang (University of Virginia), Jundong Li (University of Virginia)
OptimizationTransformerLarge Language ModelReinforcement LearningAgentic AIText
🎯 What it does: Propose the ANYMAC framework, which employs a sequential communication protocol based on LLM for multi-agent collaboration, dynamically predicts the next agent (Next-Agent Prediction) and context (Next-Context Selection), and constructs task-adaptive communication links.
APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport
Zhuo Li (Chinese University of Hong Kong, Shenzhen), Xiang Wan (Chinese University of Hong Kong, Shenzhen)
Reinforcement Learning from Human FeedbackLarge Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: Designed an adaptive marginal mechanism based on Optimal Transport to enhance the robustness of reward models in distinguishing human preferences;
AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs
Xiaopeng Ke (Harbin Institute of Technology), Min Zhang (Harbin Institute of Technology)
Data SynthesisTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Based on the AQuilt framework, automatically generate instruction-tuning data containing answers, questions, logical reasoning, inspection results, and task types from unannotated text.
AraEval: An Arabic Multi-Task Evaluation Suite for Large Language Models
Alhanoof Althnian (HUMAIN), Nora Al-Twairesh (HUMAIN)
Large Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: Created AraEval, a large-scale multi-task evaluation suite for Arabic, containing 24,378 samples covering dimensions such as knowledge, reasoning, truthfulness, and instruction following.
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?
Momoka Furuhashi (Tohoku University), Saku Sugawara (National Institute of Informatics)
GenerationLarge Language ModelTextChain-of-Thought
🎯 What it does: Investigated the necessity, generation methods, and effectiveness of using checklists in automatically evaluating generated tasks, exploring when to use them, how to generate them, and which items can improve consistency with human assessments.
Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probability
Tu Anh Dinh (Karlsruhe Institute of Technology), Jan Niehues (Karlsruhe Institute of Technology)
GenerationLarge Language ModelText
🎯 What it does: Propose a model probability-based quality estimation method called BOOSTEDPROB to address the low confidence issue caused by excessively low model probabilities in free-text generation tasks.
Are Language Models Consequentialist or Deontological Moral Reasoners?
Keenan Samway (Max Planck Institute for Intelligent Systems), Zhijing Jin (Max Planck Institute for Intelligent Systems)
Autonomous DrivingExplainability and InterpretabilityLarge Language ModelTextChain-of-Thought
🎯 What it does: This paper systematically analyzes the moral reasoning trajectories of large language models (LLMs) by providing over 600 rewritten autonomous vehicle moral dilemmas (Trolley Problem) and prompting the models to generate their thinking processes;
Are Large Language Models Chronically Online Surfers? A Dataset for Chinese Internet Meme Explanation
Yubo Xie (Shanghai Maritime University), Fahui Miao (Ecole Polytechnique Federale De Lausanne)
Explainability and InterpretabilityTransformerLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: This paper constructs the CHIME dataset to explain Chinese internet emoticons and evaluate the understanding ability of large language models (LLMs) without fine-tuning.
Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance
Omer Nahum (Technion Institute of Technology), Roi Reichart (Technion Institute of Technology)
Data-Centric LearningTransformerLarge Language ModelPrompt EngineeringMixture of ExpertsTextBenchmark
🎯 What it does: Detect and correct label errors in factual consistency and summary quality datasets to enhance model training and evaluation effectiveness.
Are Stereotypes Leading LLMs’ Zero-Shot Stance Detection ?
Anthony Dubreuil (Université de Saint-Etienne), Amine Trabelsi (Université de Sherbrooke)
ClassificationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper systematically evaluates and quantifies the implicit biases of large language models toward dialects (AAE vs SAE) and text readability (complexity) in zero-shot stance detection.
Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study
DongGeon Lee (Pohang University of Science and Technology), Hwanjo Yu (Pohang University of Science and Technology)
Safty and PrivacyLarge Language ModelVision Language ModelImageTextMultimodalityBenchmark
🎯 What it does: Constructed MEMESAFETYBENCH, a benchmark dataset containing 50,430 real-world meme images paired with corresponding harmful/neutral instructions, to evaluate the safety of vision-language models (VLMs).
Arena-lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons
Seonil Son (NC AI), KunTae Kim
Computational EfficiencyTransformerLarge Language ModelTextBenchmark
🎯 What it does: Propose the Arena-Lite framework, which directly compares LLMs through a single-elimination tournament format, eliminating the need for baseline outputs to enhance evaluation reliability.
ArgCMV: An Argument Summarization Benchmark for the LLM-era
Omkar Gurjar (DoorDash, Inc.), Eshwar Chandrasekharan (University of Illinois Urbana-Champaign)
TransformerLarge Language ModelTextBenchmark
🎯 What it does: Constructed a new key point extraction benchmark, Arg CMV, by collecting approximately 12K multi-round, long-form debate arguments from Reddit's r/ChangeMyView. Key points (KPs) were automatically generated using LLMs and manually verified, forming a publicly available dataset.
Argument Summarization and its Evaluation in the Era of Large Language Models
Moritz Altemeyer, Benjamin Schiller (Bielefeld University)
GenerationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Integrate large language models (LLMs) into the argumentative summarization (ArgSum) system, propose two new LLM-based summarization systems, and design a prompt-based evaluation scheme.
AROMA: Autonomous Rank-one Matrix Adaptation
Hao Nan Sheng (City University of Hong Kong), Mingrui Yang (University of Hong Kong)
Computational EfficiencyRepresentation LearningTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Proposes an adaptive Rank-One Matrix Adaptation framework named AROMA for parameter-efficient fine-tuning on large language models.
Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressions
Nicholas Deas (Columbia University), Kathleen McKeown (Columbia University)
Explainability and InterpretabilityRepresentation LearningTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This study proposes and evaluates the LLM's 'human impressions' of prompt authors by decoding warmth and capability dimensions through linear probes in the hidden layer.
Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoning
Jiayuan Zhu (University of Oxford), Junde Wu (University of Oxford)
TransformerLarge Language ModelTextBiomedical DataRetrieval-Augmented Generation
🎯 What it does: Proposed a multi-round LLM medical assistant Dr.APP, leveraging medical guidelines for grounded reasoning, employing Bayesian active learning to automatically select the most informative questions, and enhancing patient experience through empathetic dialogue.
AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarification
Xuan Zhang (Zhejiang University), Weiming Lu (Zhejiang University)
TransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Propose the ASKTOACT framework, which leverages automated construction of intent clarification data and incorporates a self-correction mechanism, significantly improving the accuracy of LLMs in tool calls and clarification efficiency.
Aspect-Oriented Summarization for Psychiatric Short-Term Readmission Prediction
WonJin Yoon (Boston Children's Hospital), Timothy Miller (Boston Children's Hospital)
ClassificationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBiomedical DataElectronic Health Records
🎯 What it does: Preprocess psychiatric discharge notes using aspect-oriented LLM summarization techniques, and fine-tune a Transformer model on the summaries to predict 30-day readmission risk
Assay2Mol: Large Language Model-based Drug Design Using BioAssay Context
Yifan Deng (University of Wisconsin-Madison), Anthony Gitter (University of Wisconsin-Madison)
Drug DiscoveryTransformerLarge Language ModelTabularBiomedical DataRetrieval-Augmented Generation
🎯 What it does: Utilize large language models (LLMs) to retrieve and parse PubChem BioAssay data, generating candidate molecules targeting specific targets or phenotypes based on the retrieved experimental context.
Assessing effective de-escalation of crisis conversations using transformer-based models and trend statistics
Ignacio J. Tripodi (Crisis Text Line), Elizabeth A. Olson (Crisis Text Line)
ClassificationData SynthesisTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Developed and trained a BERT-based sentiment valuation scoring model (BERT-EV), which numerically evaluates the sentiment of each message in crisis text conversations, and quantifies the emotional de-escalation level and predicts clinical outcomes by analyzing sentiment trends over time using the Mann-Kendall test.
Assessing French Readability for Adults with Low Literacy: A Global and Local Perspective
Wafa Aissa (UCLouvain), Thomas François (UCLouvain)
ClassificationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBenchmark
🎯 What it does: A readability assessment framework for French texts targeting low-literate adults was constructed at both document and paragraph levels, along with the release of a dataset containing 461 texts annotated with overall difficulty and fine-grained difficulty labels.
AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity
Yifan Liu (Sun Yat-sen University), Wushao Wen (Sun Yat-sen University)
Large Language ModelPrompt EngineeringVision Language ModelDiffusion modelMultimodalityBenchmark
🎯 What it does: Propose the AssoCiAm benchmark, systematically eliminating internal and external ambiguities in multi-modal association reasoning, and constructing 25 categories with 2025 test samples, employing hybrid computational methods to achieve automated sample generation and option optimization.
Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms
Orfeas Menis Mastromichalakis (National Technical University of Athens), Giorgos Stamou (National Technical University of Athens)
TransformerLarge Language ModelTextBenchmark
🎯 What it does: Examines gender bias in machine translation when handling gender-neutral occupational terms, proposing an aggregated evaluation framework and metrics.
ASTRA: A Negotiation Agent with Adaptive and Strategic Reasoning via Tool-integrated Action for Dynamic Offer Optimization
Deuksin Kwon (University of Southern California), Gale Lucas
OptimizationTransformerLarge Language ModelAgentic AIText
🎯 What it does: Propose the ASTRA framework, achieving dynamic proposal optimization in multi-issue negotiation based on opponent modeling and Tit-for-Tat recursion.
Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following
Yueen Ma (Chinese University of Hong Kong), Irwin King (Chinese University of Hong Kong)
Computational EfficiencyRepresentation LearningRobotic IntelligenceTransformerVision-Language-Action ModelContrastive LearningMultimodality
🎯 What it does: Proposes Astra Transformer, a specialized model for efficient multimodal trajectory modeling, incorporating trajectory attention and learnable action queries, along with contrastive dynamics learning to enhance environmental dynamic understanding and multimodal alignment.
Attacking Misinformation Detection Using Adversarial Examples Generated by Language Models
Piotr Przybyła (Universitat Pompeu Fabra), Horacio Saggion (Universitat Pompeu Fabra)
Adversarial AttackTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This study proposes the TREPAT framework, which generates semantically preserved rewrites using large language models (LLMs), decomposes them into atomic edits, and employs beam search to generate adversarial examples for misinformation detection classifiers within a limited number of queries.
Attacks by Content: Automated Fact-checking is an AI Security Issue
Michael Sejr Schlichtkrull (Queen Mary University of London)
Safty and PrivacyAdversarial AttackTransformerLarge Language ModelTextRetrieval-Augmented Generation
🎯 What it does: This paper explains how attackers implement attacks by providing misleading or forged content to retrieval-augmented agents, and proposes integrating automated fact-checking technology into a complete self-defense pipeline for agents.
Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignment
Pedram Zaree (University of California Riverside), Nael Abu-Ghazaleh (University of California Riverside)
Safty and PrivacyAdversarial AttackTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Propose the Attention Eclipse framework, which manipulates the attention patterns of large language models (LLMs) to enhance jailbreak attacks, making models more susceptible to bypassing safety alignment;
Attention-guided Self-reflection for Zero-shot Hallucination Detection in Large Language Models
Qiang Liu (Chinese Academy of Sciences), Liang Wang (Chinese Academy of Sciences)
Anomaly DetectionExplainability and InterpretabilityComputational EfficiencyTransformerLarge Language ModelText
🎯 What it does: Proposed an attention-guided self-reflection zero-shot hallucination detection method (AGSER), which splits the input query into attentive and non-attentive parts, generates answers separately for each part, calculates consistency scores with the original answer, and uses the difference between the two as a hallucination estimator.
Audio-centric Video Understanding Benchmark without Text Shortcut
Yudong Yang (Tsinghua University), Chao Zhang (ByteDance)
Large Language ModelVideoTextMultimodalityBenchmarkAudio
🎯 What it does: This paper proposes the AVUT audio-centric video understanding benchmark, designing multiple tasks focusing on audio content and audio-visual alignment, and eliminating text shortcuts through an answer permutation filtering mechanism.
Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models
Xie Zhifei (Nanyang Technological University), Chunyan Miao (Nanyang Technological University)
ClassificationRecognitionExplainability and InterpretabilityTransformerLarge Language ModelChain-of-ThoughtAudio
🎯 What it does: Developed Audio-Reasoner, a large language model tailored for audio, capable of deep chain-of-thought reasoning, supporting multi-task question answering and emotion recognition across three domains: audio, speech, and music.
Augmenting Multi-Agent Communication with State Delta Trajectory
Yichen Tang (Tsinghua University), Qingyao Ai (Tsinghua University)
TransformerLarge Language ModelAgentic AIText
🎯 What it does: Proposes a multi-agent communication protocol based on State Delta Encoding, which jointly conveys information using natural language and sequences of hidden state changes to enhance collaboration in single LLM multi-agent systems.
AutoCT: Automating Interpretable Clinical Trial Prediction with LLM Agents
Fengze Liu (University of Pennsylvania), Andrew Lo
Explainability and InterpretabilityTransformerLarge Language ModelBiomedical DataBenchmarkRetrieval-Augmented GenerationChain-of-Thought
🎯 What it does: Propose the AUTOCT framework, which leverages Large Language Model (LLM) agents to automatically generate, evaluate, and iteratively optimize predictive features for clinical trials, ultimately using traditional machine learning models for prediction.
Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions
Lan Zhang (University of Manchester), Andre Freitas
AI Code AssistantLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: This paper investigates the performance of large language models (LLMs) in the autoformalization of complex mathematical definitions from the real world, and proposes two new datasets (Def_Wiki and Def_Arxiv) to evaluate the models.
Automated Knowledge Graph Construction using Large Language Models and Sentence Complexity Modelling
Sydney Anuyah (Indiana University), Sunandan Chakraborty (Indiana University)
Representation LearningTransformerLarge Language ModelPrompt EngineeringMixture of ExpertsTextBiomedical DataChain-of-Thought
🎯 What it does: Propose CoDe-KG end-to-end open-source pipeline, automatically constructing knowledge graphs from medical literature through coreference resolution, sentence type classification, sentence simplification, and relation extraction, while publicly releasing a 150k triple dataset.
Automating Steering for Safe Multimodal Large Language Models
Lyucheng Wu (Zhejiang University), Shumin Deng (National University of Singapore NUS NCS Joint Lab)
Safty and PrivacyLarge Language ModelContrastive LearningMultimodality
🎯 What it does: Developed AutoSteer, which automatically performs safe regulation for multimodal large language models during inference, achieving safe intervention without fine-tuning by leveraging safety awareness scores, toxicity detectors, and rejection heads.
AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists
Yifei Li (University of Wisconsin-Madison), Huan Sun (University of Wisconsin-Madison)
AI Code AssistantTransformerLarge Language ModelTextBenchmark
🎯 What it does: Propose the AutoSDT automated pipeline for large-scale collection of real scientific code generation tasks across four disciplines (bioinformatics, chemistry, geographic information science, psychology and cognitive neuroscience), constructing AutoSDT-5K (5,404 tasks)
Avoidance Decoding for Diverse Multi-Branch Story Generation
Kyeongman Park (Seoul National University), Kyomin Jung (Seoul National University)
GenerationTransformerLarge Language ModelPrompt EngineeringContrastive LearningText
🎯 What it does: Proposed the Avoidance Decoding strategy to generate multi-branch diverse stories given a single story prompt;
BabyLM’s First Constructions: Causal interventions provide a signal of learning
Joshua Rozner (Stanford University), Cory Shain (Stanford University)
Explainability and InterpretabilityRepresentation LearningTransformerLarge Language ModelContrastive LearningTextBenchmark
🎯 What it does: In the 2024 BabyLM challenge, a masked language model trained with 100M words (or fewer) is used, employing the causal probing method (global/local affinity) proposed by Rozner et al. to evaluate the model's mastery of various structural syntax (e.g., causal remainder structures, fixed slots, metaphors, and literal usages).
Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models
Zeping Yu (University of Manchester), Sophia Ananiadou (University of Manchester)
Explainability and InterpretabilityComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: By conducting interpretability analysis of the internal workings of large language models (LLMs), researchers revealed key mechanisms of multi-hop reasoning and proposed a novel mechanism called back attention, significantly enhancing the multi-hop reasoning performance of LLMs.
BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism
Qinzhuo Wu (Xiaomi Inc), Jian Luan (Xiaomi Inc)
Computational EfficiencyReinforcement Learning from Human FeedbackSupervised Fine-TuningReinforcement LearningAgentic AIVision Language ModelTextMultimodality
🎯 What it does: Proposes the BacktrackAgent framework, integrating backtracking mechanisms such as validator, judger, and reflector to explicitly detect and correct errors in GUI agents, thereby improving task completion rate and step accuracy.
Balanced Multi-Factor In-Context Learning for Multilingual Large Language Models
Masahiro Kaneko (Mohamed bin Zayed University of Artificial Intelligence), Timothy Baldwin (Mohamed bin Zayed University of Artificial Intelligence)
TransformerLarge Language ModelTextBenchmark
🎯 What it does: This paper proposes the BMF-ICL method, which enhances the context learning performance of multilingual LLMs by weighted selection of multilingual examples.
Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models
Benyamin Jamialahmadi (Huawei Noah's Ark Lab), Marzieh S. Tahaei (Huawei Noah's Ark Lab)
GenerationComputational EfficiencyKnowledge DistillationTransformerLarge Language ModelText
🎯 What it does: Designed a depth-based dynamic inference framework called Balcony, which achieves real-time adaptation to different computational budgets by inserting a single-layer Transformer after specific layers in a pre-trained LLM and freezing the main model.
BANMIME : Misogyny Detection with Metaphor Explanation on Bangla Memes
Md Ayon Mia (Dhaka International University), Akmmahbubur Rahman
ClassificationExplainability and InterpretabilityData-Centric LearningSupervised Fine-TuningPrompt EngineeringVision Language ModelMultimodalityChain-of-Thought
🎯 What it does: Constructed the BANMIME dataset, containing 2000 Bangla memes, with multi-modal annotations (types of gender discrimination, humor types, metaphor localization, and explanations) and evaluated multiple vision-language models.
BannerAgency: Advertising Banner Design with Multimodal LLM Agents
Heng Wang (Sony Group Corporation), Shingo Takamatsu (Sony Group Corporation)
GenerationTransformerLarge Language ModelAgentic AIDiffusion modelImageMultimodalityBenchmark
🎯 What it does: Propose BannerAgency, a training-free multimodal large language model agent system that can automatically generate editable banner designs based on brand logos and advertising requirements, and export them as SVG or Figma plugin code.
Batched Self-Consistency Improves LLM Relevance Assessment and Ranking
Anton Korikov (University of Toronto), Navid Rekabsaz (University of Toronto)
RetrievalLarge Language ModelText
🎯 What it does: This paper studies batched pointwise evaluation and ranking methods in large language models (LLMs), combined with self-consistency techniques, to enhance relevance judgment and sorting performance in retrieval tasks.
BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation
Tianhao Zhang (University of Minnesota Twin Cities), Dongyeop Kang (University of Minnesota Twin Cities)
Computational EfficiencyRepresentation LearningTransformerScore-based ModelContrastive LearningTextStochastic Differential Equation
🎯 What it does: Propose a random representation based on the Brownian bridge and design the BBScoreV2 metric to evaluate the temporal consistency and structural integrity of long texts.
Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks
Dongjun Kim (Korea University), Heuiseok Lim (Korea University)
Explainability and InterpretabilityTransformerTextBenchmark
🎯 What it does: Propose the Benchmark Profiling framework, which quantifies the contribution of each cognitive ability of LLMs on standard benchmarks through gradient importance scoring and parameter ablation.
Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models
Md. Atabuzzaman (Virginia Tech), Chris Thomas (Virginia Tech)
TransformerLarge Language ModelPrompt EngineeringImageTextBenchmark
🎯 What it does: This paper systematically evaluates the selection bias of large vision-language models (LVLMs) in multiple-choice question answering (MCQA) and proposes a debiasing method based on logit correction during inference.
Benchmarking Debiasing Methods for LLM-based Parameter Estimates
Nicolas Audinet de Pieuchon (Chalmers University of Technology), Richard Johansson (Chalmers University of Technology)
TransformerLarge Language ModelTextBenchmark
🎯 What it does: Compare and evaluate the performance of two LLM bias mitigation methods (DSL and PPI) in estimating downstream statistical parameters under finite sample settings, investigating the impact of expert annotation proportion and total sample size on method effectiveness.
Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation
Simin Chen, Baishakhi Ray
Large Language ModelTextReview/Survey PaperBenchmark
🎯 What it does: This paper reviews the 'data pollution' issue in large language models (LLMs) during evaluation, systematically assesses the current state of static and dynamic benchmarks, and proposes evaluation criteria and design principles for dynamic benchmarks.
Benchmarking LLMs for Translating Classical Chinese Poetry: Evaluating Adequacy, Fluency, and Elegance
Andong Chen, Min Zhang (Harbin Institute of Technology)
GenerationTransformerLarge Language ModelTextBenchmarkRetrieval-Augmented Generation
🎯 What it does: Proposed a retrieval-enhanced translation method RAT and GPT-4 evaluation metrics to assess and improve the translation of classical Chinese poetry into English
Benchmarking LLMs on Semantic Overlap Summarization
John Salvador (University of Central Florida), Santu Karmaker (University of Central Florida)
Safty and PrivacyTransformerLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: Evaluate the performance of large language models on the Semantic Overlap Summary (SOS) task and introduce a new privacy policy pair dataset (PrivacyPolicyPairs, 3P).
BeSimulator: A Large Language Model Powered Text-based Behavior Simulator
Jianan Wang (National University of Defense Technology), Lihanxun
Robotic IntelligenceLarge Language ModelTextBenchmarkChain-of-Thought
🎯 What it does: Proposed and implemented BeSimulator, a text environment behavior simulation framework based on large language models (LLMs), for pre-validation of robot behavior planning; the framework generates text scenarios, simulates actions step-by-step according to control logic, and evaluates whether task goals are achieved;
Beyond A Single AI Cluster: A Survey of Decentralized LLM Training
Haotian Dong (Tsinghua University), Zhi Wang (Tsinghua University)
Federated LearningTransformerLarge Language ModelTextReview/Survey Paper
🎯 What it does: Systematically reviewed the research progress in decentralized LLM training, proposing two major paradigms based on resource distribution (community-driven and organization-driven) along with their corresponding optimization objectives and technical frameworks.
Beyond Averages: Learning with Annotator Disagreement in STS
Alejandro Benito-Santos (Universidad Nacional de Educacion a Distancia), Adrian Ghajari
RetrievalTransformerText
🎯 What it does: This paper proposes two methods to model annotator inconsistency in semantic text similarity tasks, rather than simply taking the average, thereby improving the model's calibration and ranking performance in the face of human uncertainty.
Beyond Checkmate: Exploring the Creative Choke Points for AI Generated Texts
Nafis Irtiza Tripto (Pennsylvania State University), Dongwon Lee (Pennsylvania State University)
ClassificationLarge Language ModelText
🎯 What it does: Explore the subtle differences between human writing and LLM-generated text in three paragraphs (introduction, body, and conclusion), and evaluate the contribution of these paragraphs to AI text detection.
Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning
Qianxi He (Shanghai Artificial Intelligence Laboratory), Yingchun Wang (Shanghai Artificial Intelligence Laboratory)
Reinforcement Learning from Human FeedbackTransformerSupervised Fine-TuningReinforcement LearningTextChain-of-Thought
🎯 What it does: Designed and trained a reward model (C2RM) that simultaneously considers correctness and confidence, and applied it to optimize chain-of-thought (CoT) reasoning in small-scale LLMs, followed by reinforcement learning using PPO.
Beyond Demographics: Enhancing Cultural Value Survey Simulation with Multi-Stage Personality-Driven Cognitive Reasoning
Haijiang Liu (Wuhan University of Science and Technology), Jinguang Gu (Wuhan University of Science and Technology)
GenerationData SynthesisExplainability and InterpretabilityTransformerLarge Language ModelPrompt EngineeringTextTabularChain-of-Thought
🎯 What it does: This study designs and implements MARK (Multi-stAge Reasoning framework), a multi-stage reasoning framework based on MBTI personality dynamics, to simulate respondents' answers to cultural value surveys. The framework generates responses closer to real respondents by integrating demographic information, psychological theories, and large language models through four stages: pressure analysis, personality prediction, cognitive reasoning, and comprehensive synthesis.
Beyond Demonstrations: Dynamic Vector Construction from Latent Representations
Wang Cai (Peking University), Yunfang Wu (Peking University)
ClassificationGenerationTransformerLarge Language ModelReinforcement LearningPrompt EngineeringText
🎯 What it does: This paper proposes a dynamic vector construction and injection framework called DyVec, which can extract and inject task-related information from the internal representations of large language models without updating parameters, achieving performance in zero-shot reasoning that is even better than few-shot prompt learning.
Beyond Hate Speech: NLP’s Challenges and Opportunities in Uncovering Dehumanizing Language
Hamidreza Saffari (Politecnico di Milano), Nafise Sadat Moosavi (University of Sheffield)
ClassificationExplainability and InterpretabilityTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Systematically evaluate four LLMs (Claude, GPT-4.1mini, Mistral, Qwen) on dehumanizing language detection tasks, explore three strategies (zero-shot, few-shot, and interpretable prompting), and analyze performance differences across target groups.
Beyond Human Labels: A Multi-Linguistic Auto-Generated Benchmark for Evaluating Large Language Models on Resume Parsing
Zijian Ling (Apply U UK), Xiangjian He (University of Nottingham Ningbo)
Data-Centric LearningTransformerLarge Language ModelTextBenchmark
🎯 What it does: Constructed the first multilingual, structured resume benchmark ResumeBench, containing 2,500 synthetic resumes manually reviewed for compliance, covering 50 templates, 30 career fields, and 5 languages;
Beyond Input Activations: Identifying Influential Latents by Gradient Sparse Autoencoders
Dong Shu (Northwestern University), Ninghao Liu (University of Georgia)
Explainability and InterpretabilityRepresentation LearningLarge Language ModelAuto EncoderText
🎯 What it does: Studies how to identify latent features in sparse autoencoders (SAE) that have causal influence on large language model outputs, and proposes the Gradient SAE (GradSAE) method.
Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs
Zhang Zhang (Peking University), Wei Wu (Ant Group)
Reinforcement Learning from Human FeedbackLarge Language ModelText
🎯 What it does: Propose a dynamic data transformation method that converts offline aligned data into online equivalent data, achieving LLM alignment without requiring a reward model.
Beyond Outlining: Heterogeneous Recursive Planning for Adaptive Long-form Writing with Language Models
Ruibin Xiong (Independent Researcher), Jürgen Schmidhuber (Center of Excellence for Generative AI KAUST)
GenerationLarge Language ModelTextRetrieval-Augmented GenerationChain-of-Thought
🎯 What it does: Propose a general-purpose long-form writing framework called WriteHERE, which achieves adaptive writing through recursive task decomposition and dynamic integration of retrieval, reasoning, and creation tasks.
Beyond Pairwise: Global Zero-shot Temporal Graph Generation
Alon Eirew (Bar Ilan University), Ido Dagan (Bar Ilan University)
GenerationOptimizationTransformerLarge Language ModelPrompt EngineeringTextGraphChain-of-Thought
🎯 What it does: Propose a zero-shot LLM method that can generate a complete temporal graph for a document in one go and ensure consistency through integer linear programming.
Beyond Seen Data: Improving KBQA Generalization Through Schema-Guided Logical Form Generation
Shengxiang Gao (University of Melbourne), Jianzhong Qi (University of Melbourne)
TransformerLarge Language ModelSupervised Fine-TuningTextRetrieval-Augmented Generation
🎯 What it does: Propose the SG-KBQA model, enhancing the generalization ability of knowledge base question answering under non-IID settings through Schema-guided Entity Retrieval and Schema-guided Logical Form Generation.
Beyond Static Testbeds: An Interaction-Centric Agent Simulation Platform for Dynamic Recommender Systems
Song Jin (Renmin University of China), Rui Yan (Renmin University of China)
Recommendation SystemSafty and PrivacyLarge Language ModelTextChain-of-Thought
🎯 What it does: Built a simulation platform named RecInter based on LLM, supporting dynamic updates of product attributes through user-merchant interactions and implementing multi-dimensional user profiles and advanced agent architecture.
Beyond Task-Oriented and Chitchat Dialogues: Proactive and Transition-Aware Conversational Agents
Yejin Yoon (Hanyang University), Taeuk Kim (Hanyang University)
Data SynthesisReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningAgentic AITextSequential
🎯 What it does: Constructed a dialogue dataset TACT containing a mix of multi-turn tasks and casual conversations with recoverable transfers, and trained a model to realize an active and mode-aware dialogue agent.
Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generation
Jiankun Zhang (Jilin University), Yi Chang (Michigan State University)
RetrievalSafty and PrivacyTransformerLarge Language ModelPrompt EngineeringImageMultimodalityRetrieval-Augmented GenerationAudio
🎯 What it does: This paper systematically evaluates the privacy risks of multi-modal retrieval-augmented generation (MRAG) systems and proposes an attack method that can be conducted in a black-box environment through composite structured prompts, revealing direct and indirect privacy leakage in both visual and speech modalities.
Beyond the Leaderboard: Understanding Performance Disparities in Large Language Models via Model Diffing
Sabri Boughorbel (Qatar Computing Research Institute, HBKU), Majd Hawasly (Qatar Computing Research Institute, HBKU)
Safty and PrivacyExplainability and InterpretabilityRepresentation LearningTransformerAuto EncoderText
🎯 What it does: Using model diffing and crosscoders methods, conduct a fine-grained analysis of the latent representations of Gemma-2-9b-it and its SimPO-enhanced version, revealing specific capability differences in safety, instruction following, and multilingual aspects.
Beyond the Score: Uncertainty-Calibrated LLMs for Automated Essay Assessment
Ahmed Karim (King's College London), Zheng Yuan (University of Sheffield)
ClassificationExplainability and InterpretabilityTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper applies distribution-free Conformal Prediction on three public grading corpora (ASAP, TOEFL11, and Cambridge-FCE) after performing LoRA fine-tuning on Llama-3 8B and Qwen-2.5 3B, enabling uncertainty calibration for automatic essay scoring models and generating interpretable prediction sets.
Beyond the Surface: Measuring Self-Preference in LLM Judgments
Zhi-Yuan Chen (Renmin University of China), Yankai Lin (Huawei)
Explainability and InterpretabilityTransformerLarge Language ModelText
🎯 What it does: Studied the self-preference bias of large language models (LLMs) in evaluation tasks and proposed the DBG (Difference Between Gold and Judge) score to accurately measure this bias.
Beyond WER: Probing Whisper’s Sub‐token Decoder Across Diverse Language Resource Levels
Siyu Liang (University of Washington), Richard Wright (University of Washington)
RecognitionExplainability and InterpretabilityRepresentation LearningTransformerAudio
🎯 What it does: Conduct a fine-grained analysis of the subword-level components of the Whisper multilingual decoder.
Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendations
Giorgos Filandrianos (National Technical University of Athens), Giorgos Stamou (National Technical University of Athens)
Recommendation SystemAdversarial AttackTransformerLarge Language ModelText
🎯 What it does: By embedding cognitive biases (such as social proof, scarcity, exclusivity, etc.) into product descriptions, the study investigates their impact on product recommendation systems driven by large language models (LLMs).
Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Dataset
Taisei Yamamoto (University of Tokyo), Hitomi Yanaka (University of Liverpool)
TransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBenchmark
🎯 What it does: Constructed a unified evaluation benchmark called SOBACO for Japanese social bias and cultural commonsense, and assessed LLMs' bias and cultural commonsense on this benchmark.
Biased Tales: Cultural and Topic Bias in Generating Children’s Stories
Donya Rooein (Bocconi University), Dirk Hovy (Bocconi University)
GenerationTransformerLarge Language ModelTextBenchmark
🎯 What it does: Investigate biases in large language models when generating personalized children's bedtime stories with respect to social cultural factors such as gender, ethnicity, and religion, and propose an evaluation framework.
BIRD: Bronze Inscription Restoration and Dating
Wenjie Hua (Wuhan University), Gangyan Ge (Wuhan University)
ClassificationRestorationDomain AdaptationGraph Neural NetworkTransformerLarge Language ModelTextGraph
🎯 What it does: To address the damage and dating uncertainty of Chinese bronze inscriptions, the BIRD dataset is constructed and a restoration and dating framework based on pre-trained language models is proposed.
Bit-Flip Error Resilience in LLMs: A Comprehensive Analysis and Defense Framework
Yuhang Chen (University of North Carolina at Chapel Hill), Tianlong Chen (University of North Carolina at Chapel Hill)
Safty and PrivacyTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper systematically analyzes the impact of bit-flip errors on large language models (LLMs) and proposes the FlipGuard defense framework.
Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs
Dawid Jan Kopiczko (University Of Technology Nuremberg), Yuki M Asano (University Of Technology Nuremberg)
TransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Introduce bidirectional attention to a pre-trained decoder-only LLM during instruction tuning, thereby enhancing the model's performance on reasoning, arithmetic, and language understanding tasks.
Blind Men and the Elephant: Diverse Perspectives on Gender Stereotypes in Benchmark Datasets
Mahdi Zakizadeh (Khatam University), Mohammad Taher Pilehvar (Cardiff University)
Data-Centric LearningTransformerTextBenchmark
🎯 What it does: Investigated how data distribution differences between two mainstream gender stereotype benchmarks (StereoSet and CrowS-Pairs) lead to inconsistent measurements, and improved their correlation through manual revision and balancing.
Boosting Data Utilization for Multilingual Dense Retrieval
Chao Huang (Beijing Jiaotong University), Kaiyu Huang (Beijing Jiaotong University)
RetrievalData-Centric LearningTransformerLarge Language ModelContrastive LearningText
🎯 What it does: This paper proposes a method to enhance data utilization in multilingual dense retrieval by leveraging high-quality hard negative sample selection and generation, along with constructing mini-batches balanced by language and topic.
Boosting Multi-modal Keyphrase Prediction with Dynamic Chain-of-Thought in Vision-Language Models
Qihang Ma (ByteDance Douyin Content Group), Jiao Ran (ByteDance Douyin Content Group)
Supervised Fine-TuningVision Language ModelImageTextMultimodalityChain-of-Thought
🎯 What it does: This paper explores the application of vision-language models (VLM) to multimodal key phrase prediction (MMKP), proposing three strategies: zero-shot learning, supervised fine-tuning, Fine-tune-CoT (Chain-of-Thought fine-tuning), and dynamic chain-of-thought (Dynamic CoT), aiming to enhance the model's reasoning and generalization capabilities in scenarios with missing text information and unseen phrases.
BOUQuET : dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation
Pierre Andrews (Meta), Shireen Yates (Meta)
Data-Centric LearningTextBenchmark
🎯 What it does: Created the Source-BOUQuET dataset, covering 8 non-English languages, aligned paragraphs, initiated a community translation open plan, and provided a multi-faceted evaluation benchmark.
BrailleLLM: Braille Instruction Tuning with Large Language Models for Braille Domain Tasks
Tianyuan Huang (Zhejiang University), Jiajun Bu (Zhejiang University)
TransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Propose BrailleLLM, a large language model using instruction fine-tuning, specialized in bidirectional Braille and mixed text (including formulas) conversion;
Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs
Mohsinul Kabir (University of Manchester), Sophia Ananiadou (University of Manchester)
TransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper evaluates and challenges the limitations of traditional closed-ended multiple-choice surveys in assessing cultural consistency of large language models.
Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification
Boyang Zhang (CISPA Helmholtz Center for Information Security), Yang Zhang (CISPA Helmholtz Center for Information Security)
Adversarial AttackTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper proposes a new attack method—inducing dysfunction in LLM agents by misleading them to perform repetitive or irrelevant operations, thereby preventing them from completing tasks.
Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders
Agam Goyal (University of Illinois Urbana-Champaign), Hari Sundaram (University of Illinois Urbana-Champaign)
Safty and PrivacyTransformerLarge Language ModelAuto EncoderText
🎯 What it does: Use sparse autoencoders to identify toxic dimensions in the residual flow of LLMs, and achieve detoxification for GPT-2 Small and Gemma-2-2B through feature elimination and targeted activation suppression;
Breaking the Noise Barrier: LLM-Guided Semantic Filtering and Enhancement for Multi-Modal Entity Alignment
Chenglong Lu (Northeastern University), Fu Zhang (Northeastern University)
Convolutional Neural NetworkGraph Neural NetworkTransformerLarge Language ModelVision Language ModelContrastive LearningMultimodality
🎯 What it does: Propose the LGEA framework to address the noise problem in multi-modal entity alignment. First, use LLM for semantic-level visual filtering and attribute summarization, then perform multi-modal fusion and alignment.
Bridging External and Parametric Knowledge: Mitigating Hallucination of LLMs with Shared-Private Semantic Synergy in Dual-Stream Knowledge
Yi Sui, Qiuchi Li (Beijing Institute Of Technology)
RetrievalExplainability and InterpretabilityTransformerLarge Language ModelTextBiomedical DataBenchmarkRetrieval-Augmented Generation
🎯 What it does: Proposes a Dual-Stream Knowledge-Augmented Framework for Shared-Private Semantic Synergy (DSSP-RAG), which achieves external and internal knowledge sharing and private semantic synergy in retrieval-augmented generation through unsupervised distortion detection, energy quotient filtering, and hybrid attention mechanisms, thereby reducing hallucinations and knowledge conflicts in large language models (LLMs).
Bridging the Gap Between Molecule and Textual Descriptions via Substructure-aware Alignment
Hyuntae Park (Korea University), SangKeun Lee (Korea University)
RetrievalRepresentation LearningDrug DiscoveryLarge Language ModelContrastive LearningTextGraphBiomedical Data
🎯 What it does: This paper proposes the MolBridge framework, which learns fine-grained representations of molecules and text by explicitly aligning molecular substructures with chemical phrases.
BRSpeech-DF: A Deep Fake Synthetic Speech Dataset for Portuguese Zero-Shot TTS
Alexandre Costa Ferro Filho (Advanced Knowledge Center in Immersive Technologies), Arlindo Rodrigues Galvão Filho (Advanced Knowledge Center in Immersive Technologies)
Data SynthesisAnomaly DetectionTransformerBenchmarkAudio
🎯 What it does: Constructed and publicly released the BRSpeech-DF dataset for Portuguese speech audio deepfake detection, containing real read speech and multi-model zero-shot text-to-speech synthesized speech.
BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Training
WenJie Zhou, Xueqi Cheng (Chinese Academy of Sciences)
OptimizationConvolutional Neural NetworkTransformerImageText
🎯 What it does: By separating the feature subspace of the loss Hessian matrix, we construct a pluggable optimization framework called BSFA. It scales the gradient updates of the principal subspace (sharp directions) and the volume subspace (flat directions) with different ratios, thereby improving training stability and convergence speed.
BTC-SAM: Leveraging LLMs for Generation of Bias Test Cases for Sentiment Analysis Models
Zsolt T. Kardkovács (Dublin City University), Walid Gaaloul (Télécom SudParis)
ClassificationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Proposed the BTC-SAM framework, which automatically generates diverse bias test cases for evaluating social bias in sentiment analysis models using large language models with only minimal specifications provided.
BTS: Harmonizing Specialized Experts into a Generalist LLM
Qizhen Zhang (University of Oxford), Mike Lewis (Meta Superintelligence Labs)
Computational EfficiencyKnowledge DistillationTransformerLarge Language ModelMixture of ExpertsText
🎯 What it does: Proposes a BRANCH-TRAIN-STITCH (BTS) algorithm that integrates multiple pre-trained domain experts into a stronger general-purpose large language model by inserting lightweight stitch layers between a frozen expert model and a seed model.
Building Trust in Clinical LLMs: Bias Analysis and Dataset Transparency
Svetlana Maslenkova (M42), Praveenkumar Kanithi (M42)
Explainability and InterpretabilityData-Centric LearningTransformerLarge Language ModelTextBiomedical Data
🎯 What it does: Constructed an 8.9 billion-word medical text corpus HC4, and trained nine small-scale models (124–179M parameters) including GPT-2, Llama-3, and Mistral on HC4, SlimPajama, and FineWeb; subsequently conducted general bias evaluation (BOLD) and medical-specific bias evaluation (opioid prescription tendencies across different races, genders, and ages), proposing the Net Bias Prescription Score (NBPS) metric;