EMNLP 2024 Papers — Page 9
Conference on Empirical Methods in Natural Language Processing · 1268 papers
Neuron-Level Knowledge Attribution in Large Language Models
Zeping Yu (University of Manchester), Sophia Ananiadou (University of Manchester)
Explainability and InterpretabilityRepresentation LearningTransformerLarge Language ModelText
🎯 What it does: Proposed a static method based on log probability increments to identify 'value neurons' in large language models (LLMs) that directly influence final predictions, and used an inner product method to locate 'query neurons' that activate these 'value neurons'. Subsequently, they conducted a quantitative analysis on the storage characteristics of six types of knowledge in attention layers and FFN layers.
NeuroTrialNER: An Annotated Corpus for Neurological Diseases and Therapies in Clinical Trial Registries
Simona Emilova Doneva (University of Zurich), Benjamin Victor Ineichen (University of Zurich)
ClassificationRecognitionTransformerSupervised Fine-TuningTextBiomedical DataBenchmark
🎯 What it does: Created the NeuroTrialNER corpus, annotating neurological diseases, treatment interventions, and control interventions entities in 1093 clinical trial abstracts.
NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian
Peng Liu (Norwegian University of Science and Technology), Zhirong Yang (Norwegian University of Science and Technology)
GenerationTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningTextBenchmarkChain-of-Thought
🎯 What it does: This study constructs a multi-scale Norwegian generative language model (NorGLM) and releases a comprehensive benchmark dataset for Norwegian language, NLEBench, covering seven tasks: dialogue, news summarization, instruction generation, natural language understanding, toxicity and bias evaluation, and multi-task QA+summarization.
No Culture Left Behind: ArtELingo-28, a Benchmark of WikiArt with Captions in 28 Languages
Youssef Mohamed (King Abdullah University of Science and Technology), Mohamed Elhoseiny (King Abdullah University of Science and Technology)
Large Language ModelSupervised Fine-TuningVision Language ModelImageTextMultimodalityBenchmark
🎯 What it does: Constructed a multilingual sentiment image description benchmark named ArtELingo-28, collecting approximately 200K emotional tags and subjective descriptions in 28 languages for 2000 WikiArt works.
Noise, Novels, Numbers. A Framework for Detecting and Categorizing Noise in Danish and Norwegian Literature
Ali Al-Laith (University of Copenhagen), Timothy R Tangherlini (University of California)
ClassificationTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Developed and validated a framework for detecting and classifying noise in 19th-century Danish and Norwegian novels.
NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition
Elena Merdjanovska (Humboldt-Universität zu Berlin), Alan Akbik (Humboldt-Universität zu Berlin)
RecognitionTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: Construct the NOISEBENCH benchmark, containing six types of real label noise (expert errors, crowdsourcing errors, remote inference, weak supervision, LLM-generated labels, and expert errors), to evaluate their impact on NER performance and compare multiple noise-robust learning methods.
Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation
Ruotong Pan (Chinese Academy of Sciences), Le Sun (Meituan)
GenerationRetrievalExplainability and InterpretabilityTransformerSupervised Fine-TuningTextBenchmarkRetrieval-Augmented Generation
🎯 What it does: Proposed and implemented a 'Credibility-aware Generation (CAG)' framework to identify and leverage the credibility of retrieved documents in retrieval-augmented generation (RAG), thereby reducing the negative impact of noise, outdated, or misleading information on answers.
Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment
Zhipeng Chen (Renmin University of China), Ji-Rong Wen (Renmin University of China)
OptimizationReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: This paper proposes a low-redundancy optimization alignment method called ALLO. It first filters the most important neurons for human preferences through gradient estimation, then splits the alignment process into two stages: the forgetting stage (using NPO and token-level reward models to eliminate misaligned knowledge) and the learning stage (using DPO with token-level weights to focus on key tokens) for efficient fine-tuning.
Null-Shot Prompting: Rethinking Prompting Large Language Models With Hallucination
Pittawat Taveekitworachai (Ritsumeikan University), Ruck Thawonmas (Ritsumeikan University)
TransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: The study introduces hallucinations in large language model prompts and proposes the method of enhancing multi-task performance under a zero-shot setting through 'Null-Shot Prompting'.
NumeroLogic: Number Encoding for Enhanced LLMs’ Numerical Reasoning
Eli Schwartz, Assaf Arbelle (IBM Research)
TransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextChain-of-Thought
🎯 What it does: Proposed the NumeroLogic digital encoding scheme, which prefixes numbers with a digit count prefix to enhance numerical reasoning and generation capabilities in large language models.
NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data
Sergei Bogdanov (NuMind), Etienne P Bernard (NuMind)
RecognitionTransformerLarge Language ModelContrastive LearningText
🎯 What it does: Leverage large language models (LLM) to automatically annotate massive text, building a training set with 200k dimensions of entity types, and pre-train RoBERTa using contrastive learning to obtain NuNER.
OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants
Jaspreet Ranjit (University of Southern California), Swabha Swayamdipta (University of Southern California)
ClassificationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: Built and utilized the OATH-Frames framework, combining experts and large language models to perform multi-label emotion framework annotation on millions of tweets about homeless people in the United States, and conducted large-scale attitude analysis based on the prediction results.
Oddballs and Misfits: Detecting Implicit Abuse in Which Identity Groups are Depicted as Deviating from the Norm
Michael Wiegand (University of Vienna), Josef Ruppenhofer (FernUniversität in Hagen)
ClassificationAnomaly DetectionTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper studies implicit insults targeting identity groups in sentences, focusing on scenarios where groups are portrayed as deviating from social norms, and constructs two datasets (a constructed dataset and a Twitter real sentences collection) while evaluating the detection performance of multiple models.
OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer
Lu Zhang (Om AI Research), Kyusong Lee (Om AI Research)
RetrievalTransformerLarge Language ModelAgentic AIVideoTextMultimodalityRetrieval-Augmented GenerationChain-of-Thought
🎯 What it does: Propose OmAgent, a framework integrating multi-modal RAG with a general-purpose agent, specifically designed for complex question answering on long videos;
On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning
Geewook Kim (NAVER Cloud AI), Minjoon Seo (KAIST AI)
Computational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringVision Language ModelImageTextMultimodalityBenchmark
🎯 What it does: Developed an efficient vision-language model ELVA by enhancing the visual encoder and training strategies to improve visual-text understanding.
On Eliciting Syntax from Language Models via Hashing
Yiran Wang (National Institute of Information and Communications Technology), Masao Utiyama (National Institute of Information and Communications Technology)
TransformerLarge Language ModelContrastive LearningText
🎯 What it does: Propose an unsupervised compositional syntactic parsing method called Parserker2, which extracts syntactic structures from pre-trained language models using binary hashing and first-order CKY.
On Fake News Detection with LLM Enhanced Semantics Mining
Xiaoxiao Ma (Macquarie University), Hao Fan (Northwestern University)
ClassificationGraph Neural NetworkLarge Language ModelTextGraph
🎯 What it does: Proposed the LESS4FD model, which extracts entities and topics from news using LLMs, constructs a heterogeneous graph, and performs local and global semantic propagation through Generalized PageRank to achieve fake news detection.
On Mitigating Performance Disparities in Multilingual Speech Recognition
Monorama Swain (University of Copenhagen), Anders Søgaard (University of Copenhagen)
RecognitionTransformerLarge Language ModelSupervised Fine-TuningAudio
🎯 What it does: Evaluate the impact of various fine-tuning algorithms (including fairness-promoting algorithms) on the performance and gender gap of multilingual ASR models.
On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choices
Branislav Pecher (Brno University of Technology), Maria Bielikova (Kempelen Institute of Intelligent Technologies)
ClassificationMeta LearningTransformerSupervised Fine-TuningText
🎯 What it does: Proposes a new systematic evaluation method for random factors, which considers interactions between random factors by 'weakening' the influence of other uninvestigated factors, and provides the relative importance of each factor to overall performance fluctuations; subsequently, comprehensive experiments were conducted on seven text classification tasks (including three binary classification and four multi-class classification) and three meta-learning tasks, focusing on context learning, fine-tuning, and meta-learning under few-label scenarios.
On the Fragility of Active Learners for Text Classification
Abhishek Ghose ([24]7.ai), Emma Thuong Nguyen ([24]7.ai)
ClassificationRepresentation LearningTransformerContrastive LearningText
🎯 What it does: Conduct a large-scale, systematic experimental evaluation of active learning methods in text classification tasks, exploring their stability and effectiveness under different configurations such as datasets, text representations, classifiers, and batch/seed sizes.
On the In-context Generation of Language Models
Zhongtao Jiang (Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences), Kang Liu (Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences)
GenerationData SynthesisExplainability and InterpretabilityTransformerLarge Language ModelText
🎯 What it does: This paper systematically studies the performance of large language models in in-context generation (ICG) through theoretical analysis and controlled synthetic data experiments, and explores the impact of different data attributes and model structures on ICG and its generalization ability.
On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Models
Abhilasha Sancheti (University of Maryland College Park), Rachel Rudinger (University of Maryland College Park)
ClassificationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This study investigates gender and racial biases in large language models when predicting romantic relationships through controlled replacement of character names in movie dialogues.
On the Proper Treatment of Tokenization in Psycholinguistics
Mario Giulianelli (ETH Zurich), Ryan Cotterell (ETH Zurich)
TransformerLarge Language ModelText
🎯 What it does: Propose converting word-level language models into character-level models through marginalization, and introduce the concept of 'focal area' to more precisely calculate surprisal of subword units in psycholinguistic experiments.
On the Relationship between Truth and Political Bias in Language Models
Suyash Fulay (MIT), Jad Kabbara (MIT)
OptimizationExplainability and InterpretabilityReinforcement Learning from Human FeedbackLarge Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: By training and evaluating reward models, the study investigates the relationship between the truthfulness of language models and political bias, and finds that reward models still exhibit left-leaning political bias even when trained on objective factual data.
On the Reliability of Psychological Scales on Large Language Models
Jen-tse Huang (Chinese University of Hong Kong), Michael Lyu (Chinese University of Hong Kong)
TransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper systematically evaluates the reliability of large language models (LLMs) in answering the Big Five Inventory (BFI) questionnaire across multiple dimensions (instruction templates, question phrasing, language, option labels, option order), and explores how to modulate LLM personality expression through prompting techniques such as environmental context creation, personality assignment, and role-playing.
On the Robustness of Editing Large Language Models
Xinbei Ma (Shanghai Jiao Tong University), Yulong Wang (Baichuan Intelligent Technology)
TransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: This paper systematically evaluates the robustness of large language models (LLMs) after editing, particularly in real-world application scenarios as conversational AI. Through experiments with existing editing methods (e.g., MEMIT, ROME, SERAC, MEND, IKE, etc.), the study finds: ① During multi-round dialogues with human users, edited models exhibit confusion and hallucinations; ② After attacks such as adding context to edited knowledge, rewriting queries, or introducing suspicious questions, the editing effectiveness significantly decreases; ③ When editing more popular knowledge (high frequency, strong connectivity, high co-occurrence), robustness is worse.
On the Role of Context in Reading Time Prediction
Andreas Opedal (ETH Zürich), Ethan Wilcox (ETH Zürich)
Explainability and InterpretabilityLarge Language ModelText
🎯 What it does: Investigated the impact of context on reading time, compared the relationships between Surprisal, PMI, and word frequency, and proposed a predictor that removes collinearity in word frequency through orthogonalization.
On the Universal Truthfulness Hyperplane Inside LLMs
Junteng Liu (Hong Kong University of Science and Technology), Junxian He (Hong Kong University of Science and Technology)
Explainability and InterpretabilityTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: The study investigates whether there exists a universal truth hyperplane inside large language models, proposing to test this hypothesis by training linear probes on multi-task, multi-domain datasets.
On Training Data Influence of GPT Models
Yekun Chai (Baidu Inc), Hua Wu (Baidu Inc)
Explainability and InterpretabilityData-Centric LearningTransformerLarge Language ModelText
🎯 What it does: Investigated the impact of training data on GPT model performance and proposed the GPTfluence simulator to predict various performance metrics during training (loss, BLEU, ROUGE, etc.)
One Thousand and One Pairs: A “novel” challenge for long-context language models
Marzena Karpinska (UMass Amherst), Mohit Iyyer (UMass Amherst)
ClassificationTransformerLarge Language ModelPrompt EngineeringTextBenchmarkRetrieval-Augmented GenerationChain-of-Thought
🎯 What it does: Construct the NOCHA dataset and evaluate the performance of various long-text LLMs on the task of verifying the authenticity of novel content.
One-to-Many Communication and Compositionality in Emergent Communication
Heeyoung Lee (Sungkyunkwan University)
Representation LearningRecurrent Neural NetworkReinforcement LearningImageTabular
🎯 What it does: In a one-to-many communication environment, the study investigates how listener interest differences and collaborative pressure affect language compositionality when a single speaker broadcasts information to multiple listeners.
One2Set + Large Language Model: Best Partners for Keyphrase Generation
Liangying Shao (Xiamen University), Jinsong Su (Xiamen University)
GenerationTransformerLarge Language ModelText
🎯 What it does: This paper proposes a generate-then-select framework, which first uses the ONE2SET model to generate a candidate keyword set, and then employs a large language model (LLM) to filter the candidates, completing the full keyword generation task.
OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Prompting
Xukai Liu (University of Science and Technology of China), Enhong Chen (University of Science and Technology of China)
RetrievalTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Proposes OneNet, a framework that does not require fine-tuning for few-shot entity linking through prompts from large language models.
Ontologically Faithful Generation of Non-Player Character Dialogues
Nathaniel Weir (Johns Hopkins University), Harsh Jhamtani (Microsoft)
GenerationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: Built the KNUDGE dataset based on the game 'The Outer Worlds' and proposed the DialogueWriter method, which automatically generates NPC dialogue trees using knowledge constraints (task details, character biographies, participant lists).
Open-world Multi-label Text Classification with Extremely Weak Supervision
Xintong Li (University of California San Diego), Jingbo Shang (Cisco)
ClassificationTransformerLarge Language ModelText
🎯 What it does: Proposed the X-MLClass framework, achieving open-world multi-label text classification under extremely weak supervision (only user-provided classification target descriptions).
OpenSep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation
Tanvir Mahmud (University of Texas at Austin), Diana Marculescu (University of Texas at Austin)
SegmentationConvolutional Neural NetworkTransformerLarge Language ModelPrompt EngineeringAudio
🎯 What it does: Developed the OpenSep framework to achieve open-world audio source separation without manual intervention.
Optimized Speculative Sampling for GPU Hardware Accelerators
Dominik Wagner (Technische Hochschule Nürnberg Georg Simon Ohm), Tobias Bocklet (Technische Hochschule Nürnberg Georg Simon Ohm)
OptimizationComputational EfficiencyTransformerTextAudio
🎯 What it does: Two acceleration optimizations for speculative sampling inference on GPUs for autoregressive models were proposed: exact parallelization of intermediate matrices and using sigmoid to approximate softmax;
Optimizing Chinese Lexical Simplification Across Word Types: A Hybrid Approach
ZiHao Xiao, Wei Song (Capital Normal University)
OptimizationKnowledge DistillationTransformerLarge Language ModelSupervised Fine-TuningTextRetrieval-Augmented Generation
🎯 What it does: Proposed a Chinese vocabulary simplification method based on word types, generated training data through automatic knowledge distillation (PivotKD), and enhanced out-of-dictionary (OOD) word handling using retrieval-based interpretability augmentation (RIA);
Optimizing Code Retrieval: High-Quality and Scalable Dataset Annotation through Large Language Models
Rui Li (University of Science and Technology of China), Zhenya Huang (University of Science and Technology of China)
RetrievalOptimizationData-Centric LearningTransformerLarge Language ModelTextBenchmark
🎯 What it does: This paper proposes a high-quality code retrieval query annotation method based on large language models and constructs the Query4Code dataset;
Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
Krista Opsahl-Ong (Stanford University), Omar Khattab (Stanford University)
OptimizationHyperparameter SearchData-Centric LearningLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper proposes a framework for optimizing prompts in multi-stage language model (LM) programs, defining prompt parameters (instructions and a few examples) as adjustable variables and designing a series of algorithms (Bootstrap Random Search, module-level OPRO, MIPRO, and its variants) to find optimal prompt configurations under gradient-free and label-free conditions.
Optimizing Language Models with Fair and Stable Reward Composition in Reinforcement Learning
Jiahui Li (Zhejiang University), Jun Zhou (Zhejiang University)
OptimizationComputational EfficiencyReinforcement Learning from Human FeedbackLarge Language ModelReinforcement LearningText
🎯 What it does: Propose a Fast RL method based on equity theory, which integrates multiple reward functions in reinforcement learning through learnable dynamic weights, avoiding imbalance and overfitting caused by a single dominant reward;
Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach
Siqi Li (University of California Irvine), Jan Niehues (Karlsruhe Institute of Technology)
RetrievalOptimizationTransformerSupervised Fine-TuningMultimodalityRetrieval-Augmented Generation
🎯 What it does: To improve translation accuracy for rare words in direct speech translation (ST), a retrieval-demonstration framework is proposed, injecting retrieved examples as context into the ST model.
Order of Magnitude Speedups for LLM Membership Inference
Rongting Zhang (AWS AI), Aaron Roth (AWS AI)
Computational EfficiencyAdversarial AttackTransformerLarge Language ModelText
🎯 What it does: Developed a low-cost membership inference attack for large language models, utilizing quantile regression ensemble to predict thresholds for determining whether a document belongs to the model's training set.
ORPO: Monolithic Preference Optimization without Reference Model
Jiwoo Hong (KAIST AI), James Thorne (KAIST AI)
OptimizationReinforcement Learning from Human FeedbackTransformerSupervised Fine-TuningText
🎯 What it does: Proposed a no-reference model, single-step preference alignment algorithm ORPO, which introduces an odds ratio penalty term into supervised fine-tuning (SFT) to directly achieve alignment on pre-trained language models.
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
Weilin Zhao (Tsinghua University), Maosong Sun (Tsinghua University)
GenerationComputational EfficiencyTransformerText
🎯 What it does: Propose Ouroboros under the speculative decoding framework, which enhances draft efficiency and length by using phrase-level parallel generation on the draft model, concatenating phrases to expand the draft, and reusing phrases from validation results and historical context, thereby significantly accelerating large model inference.
Outcome-Constrained Large Language Models for Countering Hate Speech
Lingzi Hong (University of North Texas), Xiaoying Song (University of North Texas)
GenerationReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningPrompt EngineeringTextBenchmark
🎯 What it does: Studies how to incorporate constraints on dialogue outcomes when generating responses against hate speech to reduce the likelihood of subsequent uncivil or hateful comments reoccurring.
Overcome Noise and Bias: Segmentation-Aided Multi-Granularity Denoising and Debiasing for Enhanced Quarduples Extraction in Dialogue
Xianlong Luo (Sun Yat-Sen University), Yihao Wang (Sun Yat-Sen University)
RecognitionTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: The study proposes the Segmentation-Aided multi-Granularity Denoising and Debiasing (SADD) method for the dialogue emotion quadruple extraction task, combining multi-granularity denoising generation with segmentation-assisted debiasing to significantly enhance model robustness.
PairDistill: Pairwise Relevance Distillation for Dense Retrieval
Chao-Wei Huang (National Taiwan University), Yun-Nung Chen (National Taiwan University)
RetrievalKnowledge DistillationTransformerLarge Language ModelContrastive LearningText
🎯 What it does: This paper proposes the PAIRDISTILL method, which enhances retrieval effectiveness by utilizing fine-grained comparison signals from pairwise rerankers to perform knowledge distillation on dense retrieval models.
PALM: Few-Shot Prompt Learning for Audio Language Models
Asif Hanif (Mohamed Bin Zayed University of Artificial Intelligence), Hanan Aldarmaki (Mohamed Bin Zayed University of Artificial Intelligence)
ClassificationRecognitionTransformerSupervised Fine-TuningPrompt EngineeringAudio
🎯 What it does: This paper proposes the PALM method, achieving few-shot prompt learning on audio language models by optimizing the text encoder's feature space.
PANDA: Persona Attributes Navigation for Detecting and Alleviating Overuse Problem in Large Language Models
Jinsung Kim (Korea University), Heuiseok Lim (Korea University)
Explainability and InterpretabilityTransformerPrompt EngineeringTextChain-of-Thought
🎯 What it does: Built the PANDA framework to quantify and detect attribute overuse issues in persona-grounded dialogues of large language models (LLMs), proposed two quantitative criteria, 'off-topic' and 'excessive quantity,' and classified attributes using fine-grained conversation topics.
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
Haoyuan Wu (Chinese University of Hong Kong), Bei Yu (Chinese University of Hong Kong)
Computational EfficiencyKnowledge DistillationTransformerSupervised Fine-TuningMixture of ExpertsText
🎯 What it does: This paper proposes the Parameter-Efficient Sparsity Crafting (PESC) method, which efficiently converts a dense Transformer model into a Mixture-of-Experts (MoE) sparse model using adapters, and further performs instruction fine-tuning on this basis.
Paraphrase Types Elicit Prompt Engineering Capabilities
Jan Philip Wahle (University of Göttingen), Bela Gipp (University of Göttingen)
Large Language ModelPrompt EngineeringText
🎯 What it does: The study systematically evaluates the impact of different linguistic variations (morphology, syntax, vocabulary, etc.) on LLM prompts, conducting experiments across 120 tasks with five large models.
PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data
Ishaan Watts (Microsoft Corporation), Sunayana Sitaram (Karya)
TransformerLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: Constructed 20 questions (5 health + 5 finance + 10 culture) across 10 Indian languages (Hindi, Tamil, Telugu, Malayalam, Kannada, Marathi, Odia, Bengali, Gujarati, Punjabi), collected 90K human evaluations and 30K GPT-4-32K evaluations, conducted pairwise (Elo) and direct assessment (LA, TQ, H) evaluations on 30 multilingual models, generated leaderboards for human and LLM evaluators, and analyzed their consistency and biases.
PATIENT-\psi: Using Large Language Models to Simulate Patients for Training Mental Health Professionals
Ruiyi Wang (Carnegie Mellon University), Zhiyu Chen (Carnegie Mellon University)
TransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper proposes PATIENTΨ, a virtual patient that integrates CBT cognitive models with large language models (LLMs), designed to train mental health professionals.
Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similarity
Bowen Zhang (Tsinghua University), Chunping Li (Tsinghua University)
Representation LearningSupervised Fine-TuningContrastive LearningText
🎯 What it does: This paper proposes a two-stage fine-tuning framework called Pcc-tuning to break through the performance ceiling of contrastive learning in semantic text similarity (STS) tasks.
PCQPR: Proactive Conversational Question Planning with Reflection
Shasha Guo (Renmin University of China), Hong Chen (Renmin University of China)
GenerationTransformerLarge Language ModelText
🎯 What it does: Redefine traditional conversational question generation (CQG) as conclusion-driven conversational question generation (CCQG), and propose the PCQPR framework to achieve proactive planning and question generation.
Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification
Pritish Sahu (SRI International), Ajay Divakaran (SRI International)
Explainability and InterpretabilityLarge Language ModelVision Language ModelMultimodalityTabularChain-of-Thought
🎯 What it does: Proposes the Pelican framework, which detects and corrects hallucinations in large vision-language models (LVLMs) during visual question answering by decomposing visual propositions and verifying through Program-of-Thought (PoT) reasoning.
PepRec: Progressive Enhancement of Prompting for Recommendation
Yakun Yu, Di Niu
Recommendation SystemLarge Language ModelPrompt EngineeringTabular
🎯 What it does: Propose a training-free, large language model-based progressive enhancement prompting (PepRec) framework to integrate content filtering and collaborative filtering while generating interpretable recommendations.
Perceptions of Linguistic Uncertainty by Language Models and Humans
Catarina G Belém (University of California Irvine), Padhraic Smyth (University of California Irvine)
Explainability and InterpretabilityTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This study evaluates the numerical interpretation ability of large language models (LLMs) regarding expressions of uncertainty in language, and compares them with humans;
Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models
Chani Jung (KAIST), Hyunwoo Kim (Allen Institute for AI)
Explainability and InterpretabilityLarge Language ModelTextChain-of-Thought
🎯 What it does: Constructed two perception-enhanced ToM datasets, Percept-ToMi and Percept-FANToM, to evaluate the capabilities of LLMs in two stages: perceptual reasoning and percept-belief reasoning, and proposed the PercepToM framework to improve the ToM reasoning performance of LLMs.
Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale
Flavio Di Palo (Amazon), Bilal H Fadlallah
ClassificationComputational EfficiencyKnowledge DistillationLarge Language ModelText
🎯 What it does: Using a large language model (LLM) as a teacher, gradually distill its knowledge into a small task-specific model through an active learning cycle (including validation metrics and hard negative samples), achieving efficient text classification.
Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems
Zhengyuan Liu (Nanyang Technological University), Nancy F. Chen (Nanyang Technological University)
TransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Built a student personalization simulation framework based on large language models, achieving multi-dimensional modeling of cognitive and non-cognitive features in language learning dialogues.
Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts
Zhaoxuan Tan (University of Notre Dame), Meng Jiang (University of Notre Dame)
Federated LearningSafty and PrivacyComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: Proposes the PER-PCS framework, enabling users to securely share and assemble personalized PEFT segments for efficiently constructing personalized LLMs; implements automated gating and weighted aggregation processes.
Personas as a Way to Model Truthfulness in Language Models
Nitish Joshi (New York University), He He (New York University)
Data SynthesisExplainability and InterpretabilityRepresentation LearningData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: This paper proposes and verifies the 'human persona hypothesis,' which posits that large language models (LLMs) learn the concept of truth by identifying common features among (true/false) text generators (i.e., personas) in pre-training data and can infer corresponding personas during inference based on context, thereby controlling the sincerity of generated text.
PhiloGPT: A Philology-Oriented Large Language Model for Ancient Chinese Manuscripts with Dunhuang as Case Study
Yuqing Zhang (Zhejiang University), Fei Wu (Zhejiang University)
RestorationTransformerLarge Language ModelSupervised Fine-TuningTextBenchmarkChain-of-Thought
🎯 What it does: Proposed a domain-specific large model called PhiloGPT tailored for ancient Chinese manuscripts, and constructed the corresponding ancient Chinese corpus PhiloCorpus-ZH, evaluation benchmark PhiloBenchmark, and chained reasoning framework PhiloCoP;
Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models
Kushal Tatariya (KU Leuven), Miryam de Lhoneux (Sailplane AI)
ClassificationRecognitionTransformerSupervised Fine-TuningVision Language ModelAuto EncoderImageTextMultimodality
🎯 What it does: This paper systematically evaluates PIXEL (a pixel-level language model) through a series of language and visual probing tasks, exploring its positioning in visual and language processing;
Please note that I’m just an AI: Analysis of Behavior Patterns of LLMs in (Non-)offensive Speech Identification
Esra Dönmez (University of Stuttgart), Agnieszka Falenska (University of Stuttgart)
ClassificationSafty and PrivacyTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Evaluated and compared the performance of 16 popular large language models (LLMs) in identifying online non-aggressive/aggressive language (including hate speech and microaggressions), and conducted in-depth analysis of their failure behavior patterns.
Position Engineering: Boosting Large Language Models through Positional Information Manipulation
Zhiyuan He (Microsoft Research), Lili Qiu (Microsoft Research)
ClassificationGenerationRetrievalTransformerPrompt EngineeringTextRetrieval-Augmented Generation
🎯 What it does: Propose a technique called Position Engineering, which regulates positional information by inserting placeholders into prompts, and validates its effectiveness on retrieval-augmented generation (RAG) and in-context learning (ICL) tasks.
PostMark: A Robust Blackbox Watermark for Large Language Models
Yapei Chang (University of Massachusetts Amherst), Mohit Iyyer (University of Massachusetts Amherst)
GenerationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Proposed a post-processing black-box watermarking method called POSTMARK, which utilizes text semantic embedding to select keywords and inserts them into LLM-generated text via an instruction-based LLM to achieve detection of LLM-generated text.
Pragmatic Norms Are All You Need – Why The Symbol Grounding Problem Does Not Apply to LLMs
Reto Gubelmann (University of Zurich)
Large Language ModelReview/Survey Paper
🎯 What it does: Discusses whether LLMs are affected by the symbol grounding problem, proposes two categories of SCP (philosophical and empirical) and clarifies their origins.
Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation
Yuhui Zhang (Stanford University), Alexander T Toshev
GenerationTransformerAuto EncoderContrastive LearningMultimodality
🎯 What it does: Attempt to migrate pre-trained language models to autoregressive text-to-image generation tasks, expand the vocabulary to accommodate image tokens, and fine-tune on a large-scale image-text pair dataset;
Pre-training Cross-lingual Open Domain Question Answering with Large-scale Synthetic Supervision
Fan Jiang (University of Melbourne), Trevor Cohn (University of Melbourne)
RetrievalKnowledge DistillationRepresentation LearningTransformerLarge Language ModelSupervised Fine-TuningContrastive LearningTextBenchmarkRetrieval-Augmented Generation
🎯 What it does: Propose a unified encoder-decoder model that can simultaneously perform cross-lingual retrieval and multilingual open-domain question answering, and enhance the model's retrieval and answer generation capabilities through two-stage self-supervised pre-training.
PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment
Jiahuan Li (National Key Laboratory for Novel Software Technology, Nanjing University), Jiajun Chen (National Key Laboratory for Novel Software Technology, Nanjing University)
Domain AdaptationRepresentation LearningTransformerLarge Language ModelContrastive LearningText
🎯 What it does: Propose the PREALIGN framework, which initializes alignment before language model pre-training by constructing a word alignment table, and maintains multi-lingual alignment during pre-training using an input-only code-switching strategy, thereby enhancing cross-lingual transfer and knowledge sharing capabilities.
Precise Model Benchmarking with Only a Few Observations
Riccardo Fogliato (Amazon Web Services), Mathew Monfort (Amazon Web Services)
TransformerLarge Language ModelTextMultimodalityBenchmark
🎯 What it does: This paper proposes an estimation method based on experience Bayesian (EB) for accurately evaluating the performance of large language models (LLM) on different subgroups (such as topics, tasks, etc.) under scenarios with only a limited number of samples.
Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancement
Yuxuan Wang (Nanyang Technological University), Xiaoyuan Liu (Nanyang Technological University)
ClassificationObject DetectionGenerationTransformerVision Language ModelMultimodality
🎯 What it does: Combine pre-trained vision-language models with scene graph generation models, propose an LM estimation method for post-correction of predicate bias, and enhance model performance on unseen samples through confidence-aware dynamic integration.
PREDICT: Multi-Agent-based Debate Simulation for Generalized Hate Speech Detection
Someen Park (Hanyang University), Kyungsik Han (Hanyang University)
ClassificationTransformerLarge Language ModelAgentic AITextRetrieval-Augmented Generation
🎯 What it does: Proposed the PREDICT framework, which leverages multi-agent debate simulations to enhance the generalization performance of hate speech detection.
Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion for Efficient Inference Intervention in Large Language Model
Chenhan Yuan (University of Manchester), Jingren Zhou (Alibaba Group)
Computational EfficiencyReinforcement Learning from Human FeedbackTransformerLarge Language ModelText
🎯 What it does: Propose a non-destructive parameter insertion method called Otter, which can predict synchronized calibration signals (e.g., rewards) aligned with tokens to achieve inference intervention without interfering with the original LLM output.
Preference-Guided Reflective Sampling for Aligning Language Models
Hai Ye (National University of Singapore), Hwee Tou Ng (National University of Singapore)
Reinforcement Learning from Human FeedbackTransformerSupervised Fine-TuningReinforcement LearningText
🎯 What it does: Proposes the Preference-Guided Reflective Sampling (PRS) method, using tree-shaped self-reflective generation to improve the quality of offline RL alignment data sampling for large language models (LLMs).
Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization
Seungwoo Son (POSTECH), Jaeho Lee (Google)
Computational EfficiencyTransformerLarge Language ModelText
🎯 What it does: This paper proposes a method to reduce outliers in the activation of large language models by inserting special cushion tokens (CushionCache) before the input sequence prefix, thereby enhancing the effectiveness of activation quantization.
Preserving Generalization of Language models in Few-shot Continual Relation Extraction
Quyen Tran (VinAI Research), Thien Huu Nguyen (University of Oregon)
ClassificationTransformerLarge Language ModelPrompt EngineeringContrastive LearningTextBenchmark
🎯 What it does: This paper addresses the problem of few-shot continuous relation extraction (FCRE) by proposing a mutual information maximization (MIM) strategy on the pre-trained language model's head, aiming to retain the general knowledge of the pre-trained backbone network and enhance the representation learning of the main classifier, thereby reducing catastrophic forgetting and overfitting.
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality
Youngtaek Oh (KAIST), Junmo Kim (KAIST)
Representation LearningSupervised Fine-TuningVision Language ModelContrastive LearningMultimodality
🎯 What it does: Proposes the FSC-CLIP fine-tuning framework, which enhances the compositional reasoning ability of pre-trained Vision-Language Models (VLMs) while maintaining multi-modal task performance through local hard negative loss and selective calibration regularization.
Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method
Weichao Zhang (CAS Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences), Xueqi Cheng (CAS Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences)
Anomaly DetectionTransformerLarge Language ModelText
🎯 What it does: Propose a discrete calibration method (DC-PDD) based on the cross-entropy between token probability and word frequency distribution, for detecting whether training data in black-box LLMs contains a given text.
Pretraining Language Models Using Translationese
Meet Doshi (Indian Institute of Technology Bombay), Pushpak Bhattacharyya (Indian Institute of Technology Bombay)
Data SynthesisTransformerSupervised Fine-TuningText
🎯 What it does: This paper proposes a complete workflow for pretraining low-resource languages using synthetic data generated from machine translation, known as 'translationese,' and verifies the effectiveness of this method on various downstream tasks.
PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation
Christoph Leiter (University of Mannheim), Steffen Eger (University of Technology Nuremberg)
TransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: This study constructs and evaluates 720 hierarchical prompt templates, generating 6.6 million prompts to assess the metric performance of open-source LLMs in machine translation and summarization evaluation;
Private Language Models via Truncated Laplacian Mechanism
Tianhao Huang (Provable Responsible AI and Data Analytics (PRADA) Lab), Di Wang (Provable Responsible AI and Data Analytics (PRADA) Lab)
Safty and PrivacyRepresentation LearningLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Propose a high-dimensional truncated Laplacian mechanism for word-level differential privacy embedding, addressing the issue of performance degradation caused by traditional Laplacian/Gaussian noise in high privacy scenarios.
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
Seungone Kim (KAIST AI), Minjoon Seo (KAIST AI)
TransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: Propose PROMETHEUS 2, a unified evaluation language model that merges model weights trained separately on direct assessment and pairwise ranking tasks, enabling simultaneous support for both evaluation formats.
PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Sampling
Yongchao Chen (MIT), Chuchu Fan (MIT)
OptimizationReinforcement Learning from Human FeedbackTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Developed the PROMST framework, which automatically optimizes LLM prompts for multi-step tasks using human feedback rules and predictive models, significantly improving task success rates.
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
Shengyao Zhuang (CSIRO), Guido Zuccon (University of Waterloo)
RetrievalRepresentation LearningTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Propose PromptReps, a method for zero-shot document retrieval by generating dense and sparse representations through prompting large language models (LLMs);
Prompts have evil twins
Rimon Melamed (George Washington University), Enric Boix-Adserà (Massachusetts Institute of Technology)
Safty and PrivacyExplainability and InterpretabilityAdversarial AttackTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper discovers 'evil twins'—uninterpretable prompts that cannot be read by humans but can generate the same functionality as original natural language prompts in language models, through maximum likelihood optimization.
Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?
Guillermo Marco (UNED), Ramón Del Castillo Santos
GenerationData SynthesisTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This study centers on a creative writing competition between Patricio Pron (a world-class novelist) and GPT-4 (a state-of-the-art LLM), generating 60 movie titles (30 from Pron and 30 from GPT-4). Both parties were required to write a 600-word plot synopsis for each title. A scoring scale based on Boden's creativity framework (novelty, surprise, value) was designed, and six literary experts evaluated the results.
Prove Your Point!: Bringing Proof-Enhancement Principles to Argumentative Essay Generation
Ruiyu Xiao (Harbin Institute of Technology), Ting Liu (Harbin Institute of Technology)
GenerationTransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: Proposed a two-phase Proof-Enhancement and Self-Annotation framework named PESA for generating logically rigorous and persuasive argumentative essays.
Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging
Deyuan Liu (Harbin Institute of Technology), Dianbo Sui (Harbin Institute of Technology)
CompressionTransformerLarge Language ModelTextBenchmark
🎯 What it does: This paper proposes the MKA method, utilizing manifold learning and NPIB metrics to achieve hierarchical knowledge alignment and merging within LLMs, thus enabling model compression.
PSC: Extending Context Window of Large Language Models via Phase Shift Calibration
Wenqiao Zhu (HiThink Research), Jun Wu (HiThink Research)
Computational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Propose a Phase Shift Calibration (PSC) module to fine-tune the frequencies of existing RoPE extensions, thereby enhancing the performance of large language models in long-context scenarios.
PsFuture: A Pseudo-Future-based Zero-Shot Adaptive Policy for Simultaneous Machine Translation
Libo Zhao (South China University of Technology), Ziqian Zeng (South China University of Technology)
TransformerText
🎯 What it does: Proposed a zero-training adaptive read/write strategy called PsFuture, which makes real-time read/write decisions by leveraging the inherent potential information of the translation model itself, while introducing the Prefix-to-Full (P2F) training method to enhance the performance of the offline model in SiMT tasks.
PsyGUARD: An Automated System for Suicide Detection and Risk Assessment in Psychological Counseling
Huachuan Qiu (Zhejiang University), Zhenzhong Lan (Westlake University)
ClassificationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: Built the PsyGUARD automated system, integrating fine-grained suicide intent detection with a risk assessment framework to enable real-time monitoring in psychological counseling texts.
PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL
Ruilin Luo (Tsinghua University), Yujiu Yang (Tsinghua University)
OptimizationData-Centric LearningAI Code AssistantTransformerSupervised Fine-TuningPrompt EngineeringTextBenchmark
🎯 What it does: Propose PTD-SQL, which first divides queries into four categories—multi-set, combination, filtering, and simple—based on SQL keywords, then constructs corresponding target practice libraries for each category, using a small number of examples to guide LLMs in generating SQL.
Puzzle Solving using Reasoning of Large Language Models: A Survey
Panagiotis Giadikiaroglou (National Technical University of Athens), Giorgos Stamou (National Technical University of Athens)
Large Language ModelSupervised Fine-TuningPrompt EngineeringTextReview/Survey PaperBenchmark
🎯 What it does: Reviews the performance of large language models in puzzles, proposes two classifications: rule-based (deterministic/random) and rule-free (riddles, programming, common sense reasoning), evaluates methods such as prompting, neuro-symbolic, fine-tuning, and compiles related datasets and benchmarks;
Python is Not Always the Best Choice: Embracing Multilingual Program of Thoughts
Xianzhen Luo (Harbin Institute of Technology), Wanxiang Che (Harbin Institute of Technology)
AI Code AssistantPrompt EngineeringTextChain-of-Thought
🎯 What it does: This paper explores the use of multiple programming languages within the Program of Thoughts (PoT) framework and proposes a task- and model-agnostic multilingual PoT (MultiPoT) method.
QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation
Weiping Fu (Xi'an Jiaotong University), Jun Liu (Xi'an Jiaotong University)
GenerationTextBenchmark
🎯 What it does: Proposes QGEval, a multi-dimensional human evaluation benchmark containing 3000 questions generated by 15 QG models based on 200 passages, with evaluation dimensions covering fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency;
Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs
Shadi Iskander (Amazon), Zohar Karnin (Technology Innovation Institute)
Data SynthesisData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: This paper proposes two methods for automatically evaluating the quality of LLM training data (human-defined intrinsic evaluation and In-Context Evaluation), and uses them to filter high-quality synthetic data, verifying that a small number of high-quality samples can match or surpass the original large-scale unverified data.