Conference on Empirical Methods in Natural Language Processing · 435 papers
Inference Helps PLMs’ Conceptual Understanding: Improving the Abstract Inference Ability with Hierarchical Conceptual Entailment Graphs
Juncai Li, Jeff Z. Pan (Shanxi University)
CodeExplainability and InterpretabilityRepresentation LearningTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Construct a hierarchical concept entailment graph (HiCon-EG) by simplifying complex sentences, pyramidizing concepts, and selecting concepts based on entropy to mine multi-level predicate and noun entailment relationships, thereby enhancing the abstract reasoning and common-sense inference capabilities of pre-trained language models (PLMs).
Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes
Kosuke Nishida (NTT Human Informatics Laboratories, NTT Corporation), Kuniko Saito (NTT Human Informatics Laboratories, NTT Corporation)
CodeTransformerLarge Language ModelText
🎯 What it does: Proposes the WeSaR method, which introduces a trainable gating parameter α for each parameter matrix to achieve reparameterization with a unified standard deviation, thereby eliminating loss spikes during LLM pre-training and enhancing training stability and speed.
🎯 What it does: Propose a task selection method (INSTA) that uses only task instruction text to optimize instruction tuning, enhancing zero-shot performance on specific tasks
CodeData SynthesisRepresentation LearningTransformerLarge Language ModelSupervised Fine-TuningTextFinance Related
🎯 What it does: Explored supervised multi-task learning using instruction-response pairs during pre-training, proposing the Instruction Pre-Training framework
InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context
Ziyi Liu (University of Southern California), Jieyu Zhao (University of Southern California)
CodeTransformerLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: Developed the INTERINTENT framework, systematically evaluating the social intelligence of LLMs in the Avalon social reasoning game through four dimensions of intention understanding (situational awareness, self-regulation, self-awareness, theory of mind).
🎯 What it does: This study explores the performance of large language models (LLMs) in complex word identification (CWI) and lexical complexity prediction (LCP) tasks across multiple languages and domains, combining zero-shot, few-shot, and fine-tuning settings;
Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?
Alexander Arno Weber (Lamarr Institute), Mehdi Ali (Lamarr Institute)
CodeData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: This study systematically evaluates the impact of different language combinations, data scales, and generation methods on the instruction-following performance of multilingual large language models (medium 7B and large Mixtral-8x7B) by constructing a high-quality parallel multilingual instruction fine-tuning dataset (Lima-X) and an evaluation benchmark (MT-Bench-X).
Is Child-Directed Speech Effective Training Data for Language Models?
Steven Y. Feng (Stanford University), Michael C. Frank (Stanford University)
CodeRepresentation LearningData-Centric LearningTransformerLarge Language ModelTextBenchmark
🎯 What it does: This paper investigates the effectiveness of child-directed speech (CHILDES) in language model training by pretraining on different corpora with GPT-2 and RoBERTa, and compares it with synthetic dialogue data TinyDialogues, as well as benchmark datasets such as OpenSubtitles, Wikipedia, and BabyLM; a 'controlled rearing' experiment is conducted to examine the impact of global and local corpus ordering on model performance.
Joint Pre-Encoding Representation and Structure Embedding for Efficient and Low-Resource Knowledge Graph Completion
Chenyu Qiu (Jiangnan University), Eddie-Yin-Kwee Ng (Nanyang Technological University)
CodeComputational EfficiencyRepresentation LearningTransformerLarge Language ModelContrastive LearningTextGraphBenchmark
🎯 What it does: Proposes a knowledge graph completion method based on Pre-Encoded Masked Language Model (PEMLM), and improves the prediction effectiveness of 1-N relations through structural embedding fusion.
Jump Starting Bandits with LLM-Generated Prior Knowledge
Parand A. Alamdari (University of Toronto), Kevin H. Wilson (Borealis AI)
CodeTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningPrompt EngineeringText
🎯 What it does: Proposed and validated a method that uses synthetic user preference data generated by large language models (LLMs) to pretrain contextual bandits (CBLI), significantly reducing cumulative regret during the early stages of online learning.
Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization
Aseem Srivastava, Md Shad Akhtar
CodeGenerationRecurrent Neural NetworkGraph Neural NetworkTransformerLarge Language ModelTextBiomedical DataBenchmark
🎯 What it does: A PIECE framework is proposed for mental health counseling dialogues, generating more professionally compliant counseling summaries through a planning-then-generation approach, utilizing knowledge filtering, knowledge scaffolding, and structured knowledge (sheaf learning).
Knowledge Verification to Nip Hallucination in the Bud
Fanqi Wan (Sun Yat-sen University), Shuming Shi (Tencent)
CodeExplainability and InterpretabilityTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: This paper studies the hallucination problem caused by inconsistencies between external knowledge and pre-training knowledge in LLM alignment data, proposing a method based on Knowledge Consistency Alignment (KCA).
KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Server
Wenhao Wang (Zhejiang University), Yanfeng Wang (Shanghai Jiao Tong University)
CodeData SynthesisFederated LearningSafty and PrivacyKnowledge DistillationTextFinance Related
🎯 What it does: Propose a client-server framework KnowledgeSG that generates high-quality synthetic text by locally fine-tuning models with differential privacy and performing knowledge distillation through server-side specialized models;
Label Confidence Weighted Learning for Target-level Sentence Simplification
Xin Ying Qiu, Jingshen Zhang (Guangdong University of Foreign Studies)
CodeGenerationData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Proposed Label Confidence Weighted Learning (LCWL), which leverages weakly supervised pseudo labels and incorporates label confidence weighting into the encoder-decoder loss to address label noise in multi-level sentence simplification.
Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level
Zhaopeng Feng (Zhejiang University), Zuozhu Liu (Zhejiang University)
CodeGenerationTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Propose MT-Ladder, a model-agnostic, cost-effective method that leverages intermediate translations generated by existing LLMs and reference translations to construct pseudo-refinement triplets, performing instruction-based fine-tuning of LLMs to significantly improve translation quality.
Large Language Models Can Be Contextual Privacy Protection Learners
Yijia Xiao (University of California, Los Angeles), Wei Cheng (NEC Laboratories America)
CodeSafty and PrivacyData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningTextBiomedical Data
🎯 What it does: Propose Contextual Privacy Protection Language Models (CPPLM), which fine-tune large language models using multiple methods to protect context-sensitive PII while injecting domain knowledge.
CodeTransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: Propose a zero-shot prompting method called PROCO, which enables large language models to self-correct reasoning errors without relying on external feedback through an iterative verify-then-correct process.
Leading Whitespaces of Language Models’ Subword Vocabulary Pose a Confound for Calculating Word Probabilities
Byung-Doh Oh (New York University), William Schuler (Ohio State University)
CodeRepresentation LearningTransformerLarge Language ModelText
🎯 What it does: This paper points out that subword tokenization-based language models may encounter issues where the sum of word probability distributions exceeds 1 during word probability calculation, due to leading spaces in subword vocabularies, leading to incorrect allocation of word-level surprisal values; it proposes a 'post-spacing decoding (WT)' method that reassigns the probability of trailing spaces to the preceding word, restoring consistency in word probabilities.
Learning Personalized Alignment for Evaluating Open-ended Text Generation
Danqing Wang (Carnegie Mellon University), Yuandong Tian (Meta AI)
CodeExplainability and InterpretabilityData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: Designed and trained PERSE — a personalized evaluation framework based on LLMs, used to measure the consistency between generated text and individual preferences, and provide interpretable scores and explanations.
CodeData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper proposes an entity-centric framework for structured entity extraction (SEE), designs a new evaluation metric called AESOP, and subsequently introduces the MuSEE multi-stage language model to accomplish the task.
LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models
Renzhi Wang (Nanjing University of Aeronautics and Astronautics), Piji Li (Nanjing University of Aeronautics and Astronautics)
CodeComputational EfficiencyMeta LearningTransformerMixture of ExpertsText
🎯 What it does: This paper proposes LEMoE, an advanced Mixture of Experts (MoE) adapter designed for lifelong model editing, achieving continuous knowledge updates for large language models (LLMs) through module insertion, KV Anchor routing, and clustering-based order planning;
Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
Zihan Wang (DeepSeek AI), Yu Wu (DeepSeek AI)
CodeComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningMixture of ExpertsText
🎯 What it does: This paper proposes ESFT, a parameter-efficient fine-tuning method for sparse Mixture-of-Experts (MoE) large language models, which can fine-tune only the experts most relevant to downstream tasks, maintaining expert specialization while significantly reducing computational costs.
Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Ranking
Jun Bai (Beihang University), Wenge Rong (Beihang University)
CodeRetrievalRepresentation LearningTransformerLarge Language ModelText
🎯 What it does: Propose a text ranking model selection method called AiRTran based on pre-trained models, which calculates the expected ranking using sentence embeddings from pre-trained models as a transferability metric, and improves the accuracy of this metric through adaptive isotropic normalization (AdaIso).
🎯 What it does: This paper proposes three improved subword segmentation schemes aimed at enhancing the segmentation's ability to capture morphology, thereby improving the performance of downstream NLP tasks.
Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models
Philipp Mondorf (MaiNLP, Center for Information and Language Processing, LMU Munich), Barbara Plank (MaiNLP, Center for Information and Language Processing, LMU Munich)
CodeLarge Language ModelTextBenchmarkChain-of-Thought
🎯 What it does: Proposed the TruthQuest benchmark, using knight-and-knave logic puzzles to evaluate the hypothetical reasoning ability of LLMs.
🎯 What it does: Propose a dynamic contrast weight (β) selection method based on linear extrapolation to improve inter-layer contrast reasoning in Transformer for fine-grained sentiment classification.
Link, Synthesize, Retrieve: Universal Document Linking for Zero-Shot Information Retrieval
Dae Yon Hwang (Amazon AGI), Yaroslav Nechaev (Amazon AGI)
CodeRetrievalTransformerLarge Language ModelSupervised Fine-TuningTextRetrieval-Augmented Generation
🎯 What it does: Propose a Universal Document Linking (UDL) algorithm that utilizes entropy-based selection of similarity models and employs Named Entity Recognition (NER) to determine document links, thereby generating cross-document synthetic queries to enhance zero-shot information retrieval;
CodeOptimizationReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper systematically evaluates and optimizes the complete three-stage training pipeline from supervised fine-tuning (SFT) to offline preference learning (DPO) and then to online preference learning (DPO), addressing the alignment issue of large language models. Based on publicly available models Gemma-2b and LLaMA-3-8b, the LION series of models were constructed.
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-Training
Tong Zhu (Soochow University), Yu Cheng (Chinese University of Hong Kong)
CodeComputational EfficiencyTransformerMixture of ExpertsText
🎯 What it does: Split the FFN of LLaMA-2-7B into multiple experts and recover performance through continuous pre-training, resulting in the LLaMA-MoE model capable of efficient inference.
LLM4Decompile: Decompiling Binary Code with Large Language Models
Hanzhuo Tan (Southern University of Science and Technology), Yuqun Zhang (Southern University of Science and Technology)
CodeGenerationAI Code AssistantTransformerLarge Language ModelTextBenchmark
🎯 What it does: Developed the LLM4Decompile series of models, utilizing large-scale LLMs to directly convert binary code into readable C source code, and integrated with traditional decompilation tools to improve executable rates.
LLMs Are Zero-Shot Context-Aware Simultaneous Translators
Roman Koshkin (Okinawa Institute of Science and Tenchnology), Satoshi Nakamura (Nara Institute of Science and Technology)
CodeGenerationTransformerLarge Language ModelPrompt EngineeringTextAudio
🎯 What it does: This paper proposes a zero-shot, context-aware real-time translation method that leverages the open-source large language model (Llama-3-70B-Instruct) and Whisper ASR for incremental inference, achieving simultaneous translation without training through response priming and minimal background information injection.
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Jiangshu Du (University of Illinois Chicago), Wenpeng Yin (Penn State University)
CodeTransformerLarge Language ModelPrompt EngineeringTextReview/Survey PaperBenchmark
🎯 What it does: Construct the ReviewCritique dataset to compare human-generated and LLM-generated paper reviews (and meta-reviews), annotate defects and explanations at the sentence level, and evaluate LLM performance in review and meta-review tasks.
Locating Information Gaps and Narrative Inconsistencies Across Languages: A Case Study of LGBT People Portrayals on Wikipedia
Farhan Samir (University of British Columbia), Yulia Tsvetkov (University of Washington)
CodeSafty and PrivacyComputational EfficiencyData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningContrastive LearningText
🎯 What it does: Propose the INFOGAP method to automatically compare factual-level information differences in biographies across different language Wikipedias.
LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models
Yuxuan Wan (Chinese University of Hong Kong), Michael Lyu
CodeTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBenchmarkChain-of-Thought
🎯 What it does: Propose the LogicAsker framework, which generates automated minimal functional tests (MFT) using 34 atomic logic rules and their 208 extended variants. The system evaluates the formal reasoning capabilities of LLMs, generates targeted examples and fine-tuning data based on error cases, thereby improving the model's reasoning accuracy.
LONGAGENT: Achieving Question Answering for 128k-Token-Long Documents through Multi-Agent Collaboration
Jun Zhao (Fudan University), Xuanjing Huang (Fudan University)
CodeRetrievalLarge Language ModelSupervised Fine-TuningAgentic AIPrompt EngineeringTextBenchmarkRetrieval-Augmented GenerationChain-of-Thought
🎯 What it does: Proposes LONGAGENT, a long-text question-answering framework based on multi-agent collaboration, which first splits documents into small chunks processed by member agents, then the leader agent derives the final answer through multi-round instructions and discussions.
LongEmbed: Extending Embedding Models for Long Context Retrieval
Dawei Zhu (Peking University), Sujian Li (Peking University)
CodeRetrievalTransformerLarge Language ModelTextBenchmark
🎯 What it does: Built the LONGEMBED benchmark and investigated training-agnostic context window expansion methods, enabling existing embedding models to extend from 512 tokens to a maximum of 32,768 tokens.
LUQ: Long-text Uncertainty Quantification for LLMs
Caiqi Zhang (University of Cambridge), Nigel Collier (University of Cambridge)
CodeExplainability and InterpretabilityTransformerLarge Language ModelTextBiomedical DataBenchmark
🎯 What it does: Propose the LUQ method for uncertainty quantification in long-text generation, and enhance the factual accuracy of LLMs through LUQ-ENSEMBLE and selective answering.
M^2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning
Taowen Wang (Rochester Institute of Technology), Dongfang Liu (Rochester Institute of Technology)
CodeComputational EfficiencyRepresentation LearningMeta LearningTransformerLarge Language ModelPrompt EngineeringImageTextMultimodality
🎯 What it does: Proposed a multi-modal prompt tuning (M PT) method that achieves zero-shot instruction learning for multi-modal large language models by utilizing a small number of learnable visual and text prompts.
MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration
Lin Xu (National University of Singapore), Jiashi Feng (National University of Singapore)
CodeTransformerLarge Language ModelAgentic AITextBenchmarkChain-of-Thought
🎯 What it does: Designed and implemented a competition-based multi-agent evaluation framework called MAgIC to quantify seven capabilities of large language models (LLMs) in multi-agent environments, including judgment, reasoning, deception, self-awareness, cooperation, coordination, and rationality.
Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training
Yixuan Wang (Harbin Institute of Technology), Wanxiang Che (Harbin Institute of Technology)
CodeComputational EfficiencyAI Code AssistantTransformerLarge Language ModelSupervised Fine-TuningTextRetrieval-Augmented Generation
🎯 What it does: Introduce noise input during the supervised fine-tuning phase to train a large language model (LLM) capable of parallel reasoning, and propose a tree-structured retrieval-enhanced Jacobi decoding strategy
Making Large Language Models Better Reasoners with Orchestrated Streaming Experiences
Xiangyang Liu (Fudan University), Xipeng Qiu (Fudan University)
CodeTransformerLarge Language ModelTextBenchmarkChain-of-Thought
🎯 What it does: Propose the RoSE framework, which utilizes a streaming experience pool to continuously collect and organize answered questions and reasoning paths of LLMs, achieving automatic, self-improving reasoning enhancement.
MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension
Ting Liu (National University of Defense Technology), Quanjun Yin (Hefei University of Technology)
CodeRecognitionComputational EfficiencyTransformerPrompt EngineeringVision Language ModelContrastive LearningMultimodality
🎯 What it does: Propose a multi-modal prior-guided parameter-efficient fine-tuning framework called MaPPER for referring expression understanding tasks.
MAR: Matching-Augmented Reasoning for Enhancing Visual-based Entity Question Answering
Zhengxuan Zhang (Hong Kong University of Science and Technology), Nan Tang (Hong Kong University of Science and Technology)
CodeRetrievalGraph Neural NetworkPrompt EngineeringVision Language ModelImageTextMultimodalityBenchmarkRetrieval-Augmented GenerationChain-of-Thought
🎯 What it does: Propose the Matching-Augmented Reasoning (MAR) method, using a matching graph to perform fine-grained retrieval and reasoning on Visual Entity Question Answering (VEQA), addressing the challenges of MLLMs in identifying individual entities.
MEANT: Multimodal Encoder for Antecedent Information
Benjamin Irving, Annika Marie Schoene (Northeastern University)
CodeClassificationTransformerVision Language ModelImageTextMultimodalityTime SeriesFinance Related
🎯 What it does: Proposed the MEANT model—a multimodal encoder integrating language, visual, and temporal dimensions for predicting momentum signals in stock markets, and released a new dataset called TempStock.
Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?
Daniel P Jeong, Michael Oberst (Johns Hopkins University)
CodeDomain AdaptationTransformerLarge Language ModelPrompt EngineeringVision Language ModelImageTextBiomedical Data
🎯 What it does: Compared the performance of medical LLMs and VLMs obtained through domain-adapted pretraining (DAPT) with their original base models on zero/one-shot medical question answering (QA) tasks.
MediTOD: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotations
Vishal Vivek Saley, Mausam .
CodeData-Centric LearningTransformerLarge Language ModelTextBiomedical DataBenchmark
🎯 What it does: Developed an English doctor-patient dialogue dataset called MediTOD, and constructed a Comprehensive Medical Attribute Schema (CMAS) using a questionnaire-based annotation method, achieving precise association between symptoms and attributes;
MedReadMe: A Systematic Study for Fine-grained Sentence Readability in Medical Domain
Chao Jiang (Georgia Institute of Technology), Wei Xu (Georgia Institute of Technology)
CodeTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBiomedical DataBenchmark
🎯 What it does: Propose the MEDREADME dataset, containing 4,520 medical sentences with readability scores and fine-grained complex span annotations, and conduct in-depth analysis through the 'Google-Easy'/'Google-Hard' classification of medical terminology; perform systematic experiments on medical text readability, investigating the impact of vocabulary, syntax, and terminology on difficulty; improve traditional readability formulas by introducing term counting to enhance correlation with human evaluation; train and evaluate complex span identification models.
MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classification
Siddhant Bikram Shah (Northeastern University), Haohan Wang (University of Illinois Urbana-Champaign)
CodeClassificationTransformerVision Language ModelContrastive LearningMultimodality
🎯 What it does: Proposes the MemeCLIP framework, which achieves multi-faceted (hate, target, stance, humor) meme classification using a frozen CLIP model and lightweight modules.
MIND: Multimodal Shopping Intention Distillation from Large Vision-language Models for E-commerce Purchase Understanding
Baixuan Xu (Hong Kong University of Science and Technology), Yangqiu Song (Amazon.com Inc)
CodeRecommendation SystemTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelMultimodality
🎯 What it does: Propose the MIND framework, leveraging large vision-language models to generate multimodal (text-image) shopping intents based on product co-purchase records, and automatically filter them using a human-like role-aware filter, constructing a 1.26M intention knowledge base.
MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
Liyan Tang (University of Texas at Austin), Greg Durrett (University of Texas at Austin)
CodeData SynthesisComputational EfficiencyData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: The study proposes a lightweight fact-checking model called MiniCheck based on synthetic training data, which can verify facts in LLM-generated content against supporting documents without relying on sentence splitting or context resolution.
CodeClassificationTransformerLarge Language ModelSupervised Fine-TuningContrastive LearningTextBenchmark
🎯 What it does: Propose the MiniConGTS method, combining a minimalist grid tagging scheme and token-level contrastive learning to achieve end-to-end efficient extraction in the Aspect Sentiment Triplet Extraction (ASTE) task.
MIPD: Exploring Manipulation and Intention In a Novel Corpus of Polish Disinformation
Arkadiusz Modzelewski (Polish-Japanese Academy of Information Technology), Adam Wierzbicki (Polish-Japanese Academy of Information Technology)
CodeClassificationTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: This paper constructs the Polish MIPD dataset, collecting and annotating 15,356 web articles with annotations on credibility, manipulation techniques, intent types, and topics, and trains various baseline models on this dataset.
🎯 What it does: Propose the HiCore framework, which leverages multi-channel hypergraphs to learn multi-layer user interests and enhances both recommendation and dialogue tasks in dialogue recommendation systems through a self-supervised approach, aiming to alleviate the worsening Matthew effect over time.
CodeGenerationLarge Language ModelReinforcement LearningDiffusion modelImageTextMultimodalityBenchmark
🎯 What it does: Proposed the OpenCHAIR benchmark and the MOCHa method for quantifying and mitigating hallucinations in open-vocabulary image captioning.
Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing
Weichuan Wang (City University of Hong Kong), Ying Wei (Zhejiang University)
CodeGenerationTransformerLarge Language ModelText
🎯 What it does: When using large language models (LLM) for machine translation, language mismatch and repetition errors frequently occur. The authors attempt to mitigate these errors through model editing techniques (Function Vectors and Knowledge Neurons). However, direct model editing has limited effectiveness or may negatively impact overall translation quality. To address this, the study proposes identifying modules across different language settings, taking their intersection to screen attention heads and FFN neurons genuinely associated with errors, and then performing targeted edits. This approach significantly reduces error rates while maintaining or improving overall translation quality.
🎯 What it does: By splitting queries into sub-queries, documents into propositions, and computing similarity at multiple granularities followed by RRF fusion, the generalization performance of document retrieval in scientific domains is improved.
Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules
Zhuocheng Gong (Peking University), Rui Yan (Renmin University of China)
CodeComputational EfficiencyTransformerMixture of ExpertsText
🎯 What it does: Redesign the Transformer by decomposing it into dynamically selectable modules, including multi-head attention, feed-forward networks, and skip modules, which are assembled on-demand during the forward pass, breaking away from the traditional depth-sequential structure.
MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts
Haofei Yu (Carnegie Mellon University), Paul Pu Liang (Massachusetts Institute of Technology)
CodeRepresentation LearningTransformerMixture of ExpertsVision Language ModelMultimodality
🎯 What it does: Propose the MMOE (Multimodal Mixtures of Experts) framework, which trains specialized expert models for different types of multimodal interactions (redundancy, uniqueness, collaboration) and dynamically fuses experts based on input during inference to enhance multimodal model performance.
Model Balancing Helps Low-data Training and Fine-tuning
Zihang Liu (Dartmouth College), Yaoqing Yang (Dartmouth College)
CodeOptimizationTransformerSupervised Fine-TuningTextBiomedical Data
🎯 What it does: Under low-data training and fine-tuning scenarios, this paper leverages Heavy-Tail Self-Regularization (HT-SR) theory to detect imbalanced training quality across model layers, and proposes the TempBalance method based on hierarchical learning rate scheduling to balance the training progress of each layer, thereby enhancing model performance.
Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue
Jia-Chen Gu (University of California, Los Angeles), Nanyun Peng (University of California, Los Angeles)
CodeExplainability and InterpretabilityComputational EfficiencyTransformerLarge Language ModelText
🎯 What it does: The paper studies the potential side effects of large language models (LLMs) on their general capabilities (reasoning, question answering, sentiment analysis, etc.) during knowledge editing, and proposes a regularization method called RECT to mitigate overfitting caused by editing.
Model-based Preference Optimization in Abstractive Summarization without Human Feedback
Jaepill Choi (Seoul National University), Taesup Kim (Seoul National University)
CodeGenerationOptimizationTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: Construct an automated preference optimization process by using summaries generated by the model under different decoding strategies (deterministic beam search and stochastic temperature sampling) as preference pairs, enhancing the accuracy and reliability of abstractive summarization.
Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation
Sweta Agrawal, Andre Martins
CodeData-Centric LearningReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: This paper proposes and constructs the MT-PREF translation preference dataset, combining the automatic quality evaluation metric XCOMET-XL+XXL with human ratings from professional translators to automatically induce translation preference relationships. The dataset is used to perform preference learning (CPO, DPO, etc.) on the TOWER decoder-only large language model (LLM), thereby improving machine translation quality.
Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration
Shangbin Feng (University of Washington), Yulia Tsvetkov (University of Washington)
CodeLarge Language ModelSupervised Fine-TuningPrompt EngineeringMixture of ExpertsText
🎯 What it does: Proposes the MODULAR PLURALISM framework, achieving multi-dimensional diversified alignment by collaborating large LLMs with multiple specialized community small LLMs.
🎯 What it does: Expand and verify the largest Word Usage Graph (WUG) dataset from SemEval-2020 Task 1: Add two additional rounds of annotation (totaling six rounds) on DWUG (English, German, Swedish), further intensifying the graph structure; supplement annotations for DiscoWUG; conduct comparative experiments on DWUG DE Sense; verify reproducibility through resampling and re-annotation.
More Insightful Feedback for Tutoring: Enhancing Generation Mechanisms and Automatic Evaluation
Wencke Liermann (Electronics and Telecommunications Research Institute), Kong Joo Lee (Chungnam National University)
CodeGenerationTransformerLarge Language ModelReinforcement LearningText
🎯 What it does: This study proposes two training extensions based on generating personalized student feedback: ① utilizing KL regularization to internalize key sentence information, ② adopting direct preference optimization (DPO) based on NLI to reduce entailment between student answers and generated feedback.
Wenlong Fei (HeFei University of Technology), Hongbo Li (HeFei University of Technology)
CodeRepresentation LearningTransformerLarge Language ModelMixture of ExpertsContrastive LearningImageText
🎯 What it does: By rendering text into glyphs (pixel maps) and constructing symbol embeddings (SSS embedding), then jointly pre-training with pre-trained language models (BERT, RoBERTa), replacing the original token embedding layer to enhance the model's multilingual capabilities.
🎯 What it does: This study proposes the ViMD corpus, covering 63 provincial dialects in Vietnam, and conducts two benchmark experiments on dialect identification and speech recognition based on this dataset.
Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models
Nisarg Patel (Arizona State University), Chitta Baral (Arizona State University)
CodeTransformerLarge Language ModelTextBenchmarkChain-of-Thought
🎯 What it does: Constructed a multi-step logical reasoning evaluation dataset named Multi-LogiEval, covering propositional logic, predicate logic, and non-monotonic logic, containing 33 inference rules and their combinations, with a maximum of 5-step reasoning depth, and evaluated the reasoning capabilities of various large language models using zero-shot chain-of-thought (CoT) assessment.
Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation
Juhwan Choi (Chung-Ang University), YoungBin Kim (Chung-Ang University)
CodeData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBenchmarkChain-of-Thought
🎯 What it does: Utilize large language models (LLM) through chain-of-thought and majority voting mechanisms to automatically identify and remove irrelevant or noisy documents from the Multi-News dataset, generating the cleaned MULTI-NEWS+ dataset.
🎯 What it does: Created the NeuroTrialNER corpus, annotating neurological diseases, treatment interventions, and control interventions entities in 1093 clinical trial abstracts.
NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian
Peng Liu (Norwegian University of Science and Technology), Zhirong Yang (Norwegian University of Science and Technology)
CodeGenerationTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningTextBenchmarkChain-of-Thought
🎯 What it does: This study constructs a multi-scale Norwegian generative language model (NorGLM) and releases a comprehensive benchmark dataset for Norwegian language, NLEBench, covering seven tasks: dialogue, news summarization, instruction generation, natural language understanding, toxicity and bias evaluation, and multi-task QA+summarization.
NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition
Elena Merdjanovska (Humboldt-Universität zu Berlin), Alan Akbik (Humboldt-Universität zu Berlin)
CodeRecognitionTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: Construct the NOISEBENCH benchmark, containing six types of real label noise (expert errors, crowdsourcing errors, remote inference, weak supervision, LLM-generated labels, and expert errors), to evaluate their impact on NER performance and compare multiple noise-robust learning methods.
Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation
Ruotong Pan (Chinese Academy of Sciences), Le Sun (Meituan)
CodeGenerationRetrievalExplainability and InterpretabilityTransformerSupervised Fine-TuningTextBenchmarkRetrieval-Augmented Generation
🎯 What it does: Proposed and implemented a 'Credibility-aware Generation (CAG)' framework to identify and leverage the credibility of retrieved documents in retrieval-augmented generation (RAG), thereby reducing the negative impact of noise, outdated, or misleading information on answers.
Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment
Zhipeng Chen (Renmin University of China), Ji-Rong Wen (Renmin University of China)
CodeOptimizationReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: This paper proposes a low-redundancy optimization alignment method called ALLO. It first filters the most important neurons for human preferences through gradient estimation, then splits the alignment process into two stages: the forgetting stage (using NPO and token-level reward models to eliminate misaligned knowledge) and the learning stage (using DPO with token-level weights to focus on key tokens) for efficient fine-tuning.
OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer
Lu Zhang (Om AI Research), Kyusong Lee (Om AI Research)
CodeRetrievalTransformerLarge Language ModelAgentic AIVideoTextMultimodalityRetrieval-Augmented GenerationChain-of-Thought
🎯 What it does: Propose OmAgent, a framework integrating multi-modal RAG with a general-purpose agent, specifically designed for complex question answering on long videos;
On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning
Geewook Kim (NAVER Cloud AI), Minjoon Seo (KAIST AI)
CodeComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringVision Language ModelImageTextMultimodalityBenchmark
🎯 What it does: Developed an efficient vision-language model ELVA by enhancing the visual encoder and training strategies to improve visual-text understanding.
On Eliciting Syntax from Language Models via Hashing
Yiran Wang (National Institute of Information and Communications Technology), Masao Utiyama (National Institute of Information and Communications Technology)
CodeTransformerLarge Language ModelContrastive LearningText
🎯 What it does: Propose an unsupervised compositional syntactic parsing method called Parserker2, which extracts syntactic structures from pre-trained language models using binary hashing and first-order CKY.
On Fake News Detection with LLM Enhanced Semantics Mining
Xiaoxiao Ma (Macquarie University), Hao Fan (Northwestern University)
CodeClassificationGraph Neural NetworkLarge Language ModelTextGraph
🎯 What it does: Proposed the LESS4FD model, which extracts entities and topics from news using LLMs, constructs a heterogeneous graph, and performs local and global semantic propagation through Generalized PageRank to achieve fake news detection.
🎯 What it does: Conduct a large-scale, systematic experimental evaluation of active learning methods in text classification tasks, exploring their stability and effectiveness under different configurations such as datasets, text representations, classifiers, and batch/seed sizes.
On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Models
Abhilasha Sancheti (University of Maryland College Park), Rachel Rudinger (University of Maryland College Park)
CodeClassificationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This study investigates gender and racial biases in large language models when predicting romantic relationships through controlled replacement of character names in movie dialogues.
On the Reliability of Psychological Scales on Large Language Models
Jen-tse Huang (Chinese University of Hong Kong), Michael Lyu (Chinese University of Hong Kong)
CodeTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper systematically evaluates the reliability of large language models (LLMs) in answering the Big Five Inventory (BFI) questionnaire across multiple dimensions (instruction templates, question phrasing, language, option labels, option order), and explores how to modulate LLM personality expression through prompting techniques such as environmental context creation, personality assignment, and role-playing.
CodeExplainability and InterpretabilityData-Centric LearningTransformerLarge Language ModelText
🎯 What it does: Investigated the impact of training data on GPT model performance and proposed the GPTfluence simulator to predict various performance metrics during training (loss, BLEU, ROUGE, etc.)
Open-world Multi-label Text Classification with Extremely Weak Supervision
Xintong Li (University of California San Diego), Jingbo Shang (Cisco)
CodeClassificationTransformerLarge Language ModelText
🎯 What it does: Proposed the X-MLClass framework, achieving open-world multi-label text classification under extremely weak supervision (only user-provided classification target descriptions).
Optimizing Chinese Lexical Simplification Across Word Types: A Hybrid Approach
ZiHao Xiao, Wei Song (Capital Normal University)
CodeOptimizationKnowledge DistillationTransformerLarge Language ModelSupervised Fine-TuningTextRetrieval-Augmented Generation
🎯 What it does: Proposed a Chinese vocabulary simplification method based on word types, generated training data through automatic knowledge distillation (PivotKD), and enhanced out-of-dictionary (OOD) word handling using retrieval-based interpretability augmentation (RIA);
Order of Magnitude Speedups for LLM Membership Inference
Rongting Zhang (AWS AI), Aaron Roth (AWS AI)
CodeComputational EfficiencyAdversarial AttackTransformerLarge Language ModelText
🎯 What it does: Developed a low-cost membership inference attack for large language models, utilizing quantile regression ensemble to predict thresholds for determining whether a document belongs to the model's training set.
ORPO: Monolithic Preference Optimization without Reference Model
Jiwoo Hong (KAIST AI), James Thorne (KAIST AI)
CodeOptimizationReinforcement Learning from Human FeedbackTransformerSupervised Fine-TuningText
🎯 What it does: Proposed a no-reference model, single-step preference alignment algorithm ORPO, which introduces an odds ratio penalty term into supervised fine-tuning (SFT) to directly achieve alignment on pre-trained language models.
🎯 What it does: Propose Ouroboros under the speculative decoding framework, which enhances draft efficiency and length by using phrase-level parallel generation on the draft model, concatenating phrases to expand the draft, and reusing phrases from validation results and historical context, thereby significantly accelerating large model inference.
PANDA: Persona Attributes Navigation for Detecting and Alleviating Overuse Problem in Large Language Models
Jinsung Kim (Korea University), Heuiseok Lim (Korea University)
CodeExplainability and InterpretabilityTransformerPrompt EngineeringTextChain-of-Thought
🎯 What it does: Built the PANDA framework to quantify and detect attribute overuse issues in persona-grounded dialogues of large language models (LLMs), proposed two quantitative criteria, 'off-topic' and 'excessive quantity,' and classified attributes using fine-grained conversation topics.
CodeTransformerLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: Constructed 20 questions (5 health + 5 finance + 10 culture) across 10 Indian languages (Hindi, Tamil, Telugu, Malayalam, Kannada, Marathi, Odia, Bengali, Gujarati, Punjabi), collected 90K human evaluations and 30K GPT-4-32K evaluations, conducted pairwise (Elo) and direct assessment (LA, TQ, H) evaluations on 30 multilingual models, generated leaderboards for human and LLM evaluators, and analyzed their consistency and biases.
PATIENT-\psi: Using Large Language Models to Simulate Patients for Training Mental Health Professionals
Ruiyi Wang (Carnegie Mellon University), Zhiyu Chen (Carnegie Mellon University)
CodeTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper proposes PATIENTΨ, a virtual patient that integrates CBT cognitive models with large language models (LLMs), designed to train mental health professionals.
Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models
Kushal Tatariya (KU Leuven), Miryam de Lhoneux (Sailplane AI)
CodeClassificationRecognitionTransformerSupervised Fine-TuningVision Language ModelAuto EncoderImageTextMultimodality
🎯 What it does: This paper systematically evaluates PIXEL (a pixel-level language model) through a series of language and visual probing tasks, exploring its positioning in visual and language processing;
Please note that I’m just an AI: Analysis of Behavior Patterns of LLMs in (Non-)offensive Speech Identification
Esra Dönmez (University of Stuttgart), Agnieszka Falenska (University of Stuttgart)
CodeClassificationSafty and PrivacyTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Evaluated and compared the performance of 16 popular large language models (LLMs) in identifying online non-aggressive/aggressive language (including hate speech and microaggressions), and conducted in-depth analysis of their failure behavior patterns.
PostMark: A Robust Blackbox Watermark for Large Language Models
Yapei Chang (University of Massachusetts Amherst), Mohit Iyyer (University of Massachusetts Amherst)
CodeGenerationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Proposed a post-processing black-box watermarking method called POSTMARK, which utilizes text semantic embedding to select keywords and inserts them into LLM-generated text via an instruction-based LLM to achieve detection of LLM-generated text.
PREDICT: Multi-Agent-based Debate Simulation for Generalized Hate Speech Detection
Someen Park (Hanyang University), Kyungsik Han (Hanyang University)
CodeClassificationTransformerLarge Language ModelAgentic AITextRetrieval-Augmented Generation
🎯 What it does: Proposed the PREDICT framework, which leverages multi-agent debate simulations to enhance the generalization performance of hate speech detection.
Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion for Efficient Inference Intervention in Large Language Model
Chenhan Yuan (University of Manchester), Jingren Zhou (Alibaba Group)
CodeComputational EfficiencyReinforcement Learning from Human FeedbackTransformerLarge Language ModelText
🎯 What it does: Propose a non-destructive parameter insertion method called Otter, which can predict synchronized calibration signals (e.g., rewards) aligned with tokens to achieve inference intervention without interfering with the original LLM output.
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality
Youngtaek Oh (KAIST), Junmo Kim (KAIST)
CodeRepresentation LearningSupervised Fine-TuningVision Language ModelContrastive LearningMultimodality
🎯 What it does: Proposes the FSC-CLIP fine-tuning framework, which enhances the compositional reasoning ability of pre-trained Vision-Language Models (VLMs) while maintaining multi-modal task performance through local hard negative loss and selective calibration regularization.