ACL 2023 Papers — Page 9
Annual Meeting of the Association for Computational Linguistics · 1074 papers
Rethinking Annotation: Can Language Learners Contribute?
Haneul Yoo (KAIST), Alice Oh (KAIST)
ClassificationRecognitionTransformerSupervised Fine-TuningText
🎯 What it does: By designing controlled experiments, 36 language learners of varying proficiency levels were recruited to annotate data across four NLP tasks in English, Korean, and Indonesian (sentiment analysis, natural language inference, named entity recognition, and machine reading comprehension), while assessing annotation quality and improvements in learners' language proficiency.
Rethinking Masked Language Modeling for Chinese Spelling Correction
Hongqiu Wu (Shanghai Jiao Tong University), Hai Zhao (Shanghai Jiao Tong University)
Data-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: This paper decomposes Chinese Spelling Correction (CSC) into a language model and an error model, discovering that traditional BERT fine-tuning overfits the error model and underfits the language model, leading to poor generalization on unseen error patterns.
Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View
Changmeng Zheng (Hong Kong Polytechnic University), Qing Li (Hong Kong Polytechnic University)
RecognitionImage TranslationTransformerVision Language ModelDiffusion modelContrastive LearningMultimodality
🎯 What it does: This paper re-examines the task of multi-modal entity and relation extraction, proposing to address alignment errors between text and images by treating text-image pairs as mutual translations. It achieves multi-modal reverse translation through generative diffusion models and constructs a high-resource bridging multi-modal bias estimator to obtain fine-grained alignment confidence.
Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale
Hritik Bansal (University of California, Los Angeles), Dan Roth (AWS AI Labs)
Explainability and InterpretabilityTransformerLarge Language ModelText
🎯 What it does: Conduct interpretability analysis on attention heads and feed-forward networks in the OPT-66B large language model, evaluate their in-context learning capabilities across multiple downstream tasks, and verify which components significantly contribute to task performance through structured pruning.
Retrieval-free Knowledge Injection through Multi-Document Traversal for Dialogue Models
Rui Wang (Harbin Institute of Technology), Ruifeng Xu (Huawei Noah's Ark Lab)
GenerationGraph Neural NetworkTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Propose a retrieval-agnostic knowledge injection method KiDG, which automatically converts knowledge documents into simulated dialogues through multi-document traversal to enhance the knowledge generation capability of dialogue models.
Retrieve-and-Sample: Document-level Event Argument Extraction via Hybrid Retrieval Augmentation
Yubing Ren (Chinese Academy of Sciences), Zheng Lin (Chinese Academy of Sciences)
GenerationRetrievalTransformerTextRetrieval-Augmented Generation
🎯 What it does: This paper proposes a retrieval-enhanced generative model for document-level event argument extraction, and designs three retrieval strategy settings (context consistency, pattern consistency, adaptive hybrid). The model improves analogy capability by generating pseudo demonstrations in continuous space through adaptive hybrid retrieval.
RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models
Zheng Liu (Huawei Technologies Ltd Co), Zhao Cao (Beijing University Of Posts And Telecommunications)
RetrievalKnowledge DistillationTransformerAuto EncoderContrastive LearningText
🎯 What it does: Propose a dual mask autoencoder (DupMAE) framework that jointly trains the embeddings of CLS and regular tokens to generate more optimal retrieval semantic representations.
REV: Information-Theoretic Evaluation of Free-Text Rationales
Hanjie Chen (University of Virginia), Swabha Swayamdipta (University of Southern California)
Explainability and InterpretabilityTransformerLarge Language ModelText
🎯 What it does: Proposed a new information-theoretic metric, REV (Rationale Evaluation with conditional V-information), to assess novel and label-related information in free-text reasoning rationales.
Revealing Single Frame Bias for Video-and-Language Learning
Jie Lei (University of North Carolina at Chapel Hill), Mohit Bansal (University of North Carolina at Chapel Hill)
RetrievalTransformerVision Language ModelContrastive LearningVideoTextMultimodality
🎯 What it does: Explore the effectiveness of single-frame training in video-language tasks, and propose strategies to achieve competitive performance through early fusion and large-scale pre-training; meanwhile, introduce two new tasks to examine the model's temporal modeling capabilities.
Revisiting Automated Prompting: Are We Actually Doing Better?
Yulin Zhou (University of Cambridge), Yarin Gal (University of Oxford)
TransformerSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: Systematically evaluate the performance of automatic prompting techniques, manual prompting, and no-prompt fine-tuning in multi-task K-shot learning, exploring their effectiveness and sustainability on RoBERTa-large.
Revisiting Commonsense Reasoning in Machine Translation: Training, Evaluation and Challenge
Xuebo Liu (Harbin Institute of Technology), Min Zhang (Harbin Institute of Technology)
GenerationTransformerTextBenchmark
🎯 What it does: This paper systematically studies the common-sense reasoning (CR) capability in neural machine translation (NMT), exploring pre-training, evaluation methods, and challenges in enhancing CR.
Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation
Yulong Chen (Zhejiang University), Yue Zhang (Westlake University)
GenerationTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: This paper proposes a cross-lingual summarization annotation protocol combining original source text and monolingual summaries, and constructs the ConvSumX benchmark based on this.
Revisiting Event Argument Extraction: Can EAE Models Learn Better When Being Aware of Event Co-occurrences?
Yuxin He (Harbin Institute of Technology), Buzhou Tang (Harbin Institute of Technology)
Representation LearningTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Reframe event argument extraction as a table generation task, using the non-autoregressive decoding framework TabEAE to parallelly extract arguments of multiple events.
Revisiting non-English Text Simplification: A Unified Multilingual Benchmark
Michael J Ryan, Wei Xu (Georgia Institute of Technology)
GenerationData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBenchmark
🎯 What it does: Proposed the MULTISIM multilingual text simplification benchmark, collecting and unifying 27 parallel simplified corpora (covering 12 languages, 1.75M sentence pairs), and conducting fine-tune, zero-shot transfer, and few-shot pre-prompting experiments on multilingual models (mT5, BLOOM) on this benchmark.
Revisiting Relation Extraction in the era of Large Language Models
Somin Wadhwa (Northeastern University), Byron Wallace
ClassificationTransformerLarge Language ModelSupervised Fine-TuningTextChain-of-Thought
🎯 What it does: The study re-examines relation extraction in the era of large language models, comparing the few-shot and fine-tuning performance of GPT-3 and Flan-T5, and proposes leveraging Chain-of-Thought (CoT) explanations generated by GPT-3 to enhance Flan-T5 training, achieving SOTA results.
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
Yixin Liu (Yale University), Dragomir Radev (Yale University)
GenerationTextBenchmark
🎯 What it does: Propose the ACU (Atomic Content Unit) fine-grained evaluation protocol and construct the RoSE large-scale human evaluation benchmark based on this protocol, conducting systematic comparisons of multiple evaluation protocols and automatic evaluation metrics.
Revisiting Token Dropping Strategy in Efficient BERT Pretraining
Qihuang Zhong (Wuhan University), Dacheng Tao (University of Sydney)
Computational EfficiencyKnowledge DistillationRepresentation LearningTransformerLarge Language ModelText
🎯 What it does: Studied the semantic loss problem caused by the token dropping strategy and proposed a semantic consistency learning (SCTD) method to improve pre-training efficiency.
Reward Gaming in Conditional Text Generation
Richard Yuanzhe Pang (New York University), He He (New York University)
GenerationReinforcement Learning from Human FeedbackTransformerReinforcement LearningText
🎯 What it does: This paper investigates the 'reward gaming' problem in conditional text generation models when training with reward functions learned from human annotations, exploring three typical scenarios: noise-induced, naturally induced, and covariate shift.
RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs
Afra Feyza Akyurek, Niket Tandon (Allen Institute for Artificial Intelligence)
Reinforcement Learning from Human FeedbackTransformerLarge Language ModelReinforcement LearningText
🎯 What it does: Propose the RL4F framework, enabling a language model to generate natural language feedback to improve the output of another fixed task model (e.g., GPT-3).
RMLM: A Flexible Defense Framework for Proactively Mitigating Word-level Adversarial Attacks
Zhaoyang Wang (Sun Yat-sen University), Jiahai Wang (Sun Yat-sen University)
ClassificationAdversarial AttackTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: The study targets word-level adversarial attacks, proposing the RMLM framework that utilizes random synonym substitution, BERT correction, and detection methods during inference to proactively confuse attackers and repair disturbed contexts;
Robust Learning for Multi-party Addressee Recognition with Discrete Addressee Codebook
Pengcheng Zhu (Alibaba Group), Haiqing Chen (Alibaba Group)
RecognitionTransformerAuto EncoderText
🎯 What it does: Proposed a robust multi-party dialogue coreference resolution model RARM, which achieves noise robustness by discretizing the audience into a codebook and using VQ-VAE.
Robust Multi-bit Natural Language Watermarking through Invariant Features
KiYoon Yoo (Seoul National University), Nojun Kwak (Seoul National University)
Large Language ModelSupervised Fine-TuningText
🎯 What it does: Propose a multi-bit watermark embedding and extraction framework based on text semantics and syntactic invariant features for copyright protection and leakage tracking of natural language content.
Robust Representation Learning with Reliable Pseudo-labels Generation via Self-Adaptive Optimal Transport for Short Text Clustering
Xiaolin Zheng (Zhejiang University), Xinting Liao (Zhejiang University)
Representation LearningTransformerContrastive LearningText
🎯 What it does: Proposed a robust short text clustering model RSTC, achieving adaptive clustering for imbalanced and noisy data through two modules: pseudo-label generation and robust representation learning.
RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations
Yilun Zhao (Yale University), Dragomir Radev (Yale University)
Adversarial AttackTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTabularBenchmarkChain-of-Thought
🎯 What it does: Constructed the ROBUT benchmark for systematic robustness evaluation in Table QA, containing 138,149 human-annotated adversarial perturbations across 10 dimensions based on table headers, content, questions, and mixed layers; and proposed the LETA framework for robustness enhancement using large language models (LLMs) to generate adversarial samples.
Rogue Scores
Max Grusky
TextReview/Survey Paper
🎯 What it does: Systematically evaluated the use of ROUGE metrics in machine learning papers, revealing widespread issues in reproducibility, comparability, and correctness; audited 2,834 papers and 831 code repositories, validated defects in 17 non-standard ROUGE implementations, uncovering the 'rogue scores' phenomenon; demonstrated through case studies that erroneous evaluations can inflate model performance; proposed Rogue-3 baseline to show how misconfigurations can artificially produce top scores.
Rule By Example: Harnessing Logical Rules for Explainable Hate Speech Detection
Christopher Clarke (University of Michigan), Mei Chen (Microsoft)
ClassificationExplainability and InterpretabilityTransformerContrastive LearningText
🎯 What it does: This paper proposes the Rule By Example (RBE) framework, which utilizes example-based contrastive learning combined with a rule encoder and text encoder to train a model capable of learning from logical rules for hate speech detection.
S2ynRE: Two-stage Self-training with Synthetic data for Low-resource Relation Extraction
Benfeng Xu (University of Science and Technology of China), Zhendong Mao (Institute of Artificial Intelligence, Hefei Comprehensive National Science Center)
ClassificationData SynthesisKnowledge DistillationTransformerLarge Language ModelText
🎯 What it does: Generate synthetic training samples that match the target domain using large language models, and enhance low-resource relation extraction performance by alternately learning from synthetic data and golden data through a two-stage self-training algorithm.
S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answering
Fangyu Lei (Laboratory of Cognition and Decision Intelligence for Complex Systems Institute of Automation Chinese Academy of Sciences), Kang Liu (Laboratory of Cognition and Decision Intelligence for Complex Systems Institute of Automation Chinese Academy of Sciences)
TransformerLarge Language ModelPrompt EngineeringTextTabularRetrieval-Augmented GenerationChain-of-Thought
🎯 What it does: Proposes a three-stage HybridQA framework (retriever → selector → generative reasoner) to address multi-hop text and table hybrid question answering problems.
SafeConv: Explaining and Correcting Conversational Unsafe Behavior
Mian Zhang (Soochow University), Dong Yu (Tencent AI Lab)
Safty and PrivacyExplainability and InterpretabilityTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningTextBenchmark
🎯 What it does: Constructed a Chinese dialogue safety dataset named SAFECONV containing dialogue-level safety labels, unsafe spans, and safe alternative responses, and trained safety detectors, unsafe span annotators, and context rewriters on it.
Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge
Jiangjie Chen (Fudan University), Yanghua Xiao (Fudan University)
Explainability and InterpretabilityTransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: Designed two probing tasks, keyword-constrained sentence generation (CG) and yes/no question answering (QA), to systematically evaluate large language models' understanding of negative common sense using the CSK-PN dataset.
Scaling in Cognitive Modelling: a Multilingual Approach to Human Reading Times
Andrea de Varda, Marco Marelli (University of Milano - Bicocca)
TransformerLarge Language ModelText
🎯 What it does: Compare the predictive power of different parameter-scale Transformer language models (XGLM) on eye movement data across ten languages for early and late reading processes.
Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination
Hao Fei (National University of Singapore), Tat-Seng Chua (National University of Singapore)
Representation LearningGraph Neural NetworkContrastive LearningTextMultimodalityGraph
🎯 What it does: Studied unsupervised multimodal machine translation (UMMT) without requiring images during inference, and proposed using visual and language scene graphs (SG) as pivots, while designing a Visual Scene Hallucination (VSH) mechanism to generate pseudo visual SGs from source language text alone, thereby enabling translation from text to the target language.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Yue Feng (University College London), Gabriella Kazai (University College London)
ClassificationExplainability and InterpretabilityTransformerLarge Language ModelText
🎯 What it does: Proposes a user satisfaction modeling framework (SG-USM) based on task attribute satisfaction, predicting user satisfaction by explicitly calculating the satisfaction level and importance of each task attribute in the dialogue.
ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning
Jingyuan S. She, Atticus Geiger (Stanford University)
TransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBenchmark
🎯 What it does: Constructed two new benchmarks, ScoNe-NLI and ScoNe-NLG, to probe models' understanding of the scope of negation words.
SCOTT: Self-Consistent Chain-of-Thought Distillation
Peifeng Wang (University of Southern California), Xiang Ren (University of Southern California)
Explainability and InterpretabilityKnowledge DistillationTransformerLarge Language ModelContrastive LearningTextChain-of-Thought
🎯 What it does: Learn a small self-consistent chain-of-thought (CoT) model through knowledge distillation from a large teacher model, ensuring consistency between the student's generated explanations and its predictions during training.
Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities
Sina Ahmadi (George Mason University), Antonios Anastasopoulos (George Mason University)
Data SynthesisTransformerText
🎯 What it does: For low-resource languages using non-traditional writing systems in bilingual communities, this paper proposes a script normalization method based on character-level Transformers.
Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM’s Translation Capability
Eleftheria Briakou (University of Maryland), George Foster (Google)
GenerationExplainability and InterpretabilityTransformerLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: Conduct a large-scale analysis of PaLM's training data to quantify and investigate its incidental exposure to bilingual and translated examples, thereby explaining the capabilities of LLMs in zero/one-shot machine translation.
SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models
Akshita Jha (Virginia Tech), Sunipa Dev (Google Research)
TransformerLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: Leverage large language models (PaLM, GPT-3, T0) to generate stereotypes at global and regional levels, which are subsequently validated for authenticity and offensiveness through a globally diverse human review, constructing SeeGULL, a large-scale, cross-cultural stereotype benchmark dataset.
Seen to Unseen: Exploring Compositional Generalization of Multi-Attribute Controllable Dialogue Generation
Weihao Zeng (Beijing University of Posts and Telecommunications), Weiran Xu (Meituan)
GenerationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper proposes a decoupled controllable dialogue generation model based on prompts, DCG, which can learn from seen attribute values and achieve compositional generalization on unseen attribute combinations.
Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering
Zhiyong Wu (Shanghai AI Laboratory), Lingpeng Kong (The University of Hong Kong)
CompressionData-Centric LearningMeta LearningTransformerLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: Investigated adaptive context learning methods, proposing a two-stage example selection and ranking framework that utilizes the principle of information compression (MDL) for unsupervised optimization of context organization;
Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models
James O’Neill, Sourav Dutta (Huawei Ireland Research Center)
CompressionComputational EfficiencyTransformerLarge Language ModelTextBenchmark
🎯 What it does: Investigate the impact of post-training quantization and quantization-aware training on generalization in Transformer language models, proposing Self-Distilled Quantization (SDQ) to reduce 8-bit integer quantization error.
Self-Edit: Fault-Aware Code Editor for Code Generation
Kechi Zhang (Peking University), Zhi Jin (Peking University)
AI Code AssistantTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: This paper proposes a generation-editing (Self-Edit) framework that leverages large language models (LLMs) to first generate competition problem code, then obtain error information through execution examples, generates supplementary comments, and subsequently modifies the generated code using a specifically trained fault-aware code editor, ultimately producing higher quality programs.
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Yizhong Wang (University Of Washington), Hannaneh Hajishirzi (University Of Washington)
Data SynthesisTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
🎯 What it does: This paper proposes the SELF-INSTRUCT framework, which automatically constructs a massive and diverse set of instruction tasks using the generation capability of pre-trained language models, and then fine-tunes the model on these synthetic data.
Semantic Structure Enhanced Event Causality Identification
Zhilei Hu (University of Chinese Academy of Sciences), Xueqi Cheng (Institute of Computing Technology, Chinese Academy of Sciences)
ClassificationRecurrent Neural NetworkGraph Neural NetworkLarge Language ModelText
🎯 What it does: Propose the SemSIn model, which utilizes semantic graphs derived from AMR parsing, combining event-centric structures and event-related structures to enhance event causality recognition.
Sequence Parallelism: Long Sequence Training from System Perspective
Shenggui Li (School of Computing National University of Singapore), Yang You (School of Computing National University of Singapore)
Computational EfficiencyText
🎯 What it does: Designed and implemented a Sequence Parallelism scheme, utilizing multiple GPUs to split long sequences into sub-sequences. Each GPU stores only its sub-sequence, and employs Ring Self-Attention to compute cross-GPU attention, breaking the single-card sequence length limit and enabling training of infinitely long sequences.
SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes
Wenda Xu (University of California Santa Barbara), William Yang Wang (University of California Santa Barbara)
GenerationData SynthesisTransformerLarge Language ModelTextRetrieval-Augmented Generation
🎯 What it does: Propose a self-supervised evaluation metric called SESCORE2, which utilizes retrieval-enhanced synthetic error generation to create training data and estimates text generation quality without human scoring.
Should you marginalize over possible tokenizations?
Nadezhda Chirkova (Naver Labs Europe), Marc Dymetman (Independent Researcher)
TransformerLarge Language ModelText
🎯 What it does: Investigated whether marginalization over all possible tokenization schemes is necessary in autoregressive language models.
Shrinking Embeddings for Hyper-Relational Knowledge Graphs
Bo Xiong (University of Stuttgart), Steffen Staab (University of Stuttgart)
Representation LearningGraph Neural NetworkGraph
🎯 What it does: Proposed a hyper-relational knowledge graph embedding model called ShrinkE for addressing link prediction with qualifiers.
Similarity-weighted Construction of Contextualized Commonsense Knowledge Graphs for Knowledge-intense Argumentation Tasks
Moritz Plenz (Heidelberg University), Anette Frank (Heidelberg University)
Representation LearningTransformerContrastive LearningTextGraph
🎯 What it does: This paper proposes an unsupervised contextual common sense knowledge graph (CCKG) construction method to supplement implicit common sense reasoning in argumentative texts, thereby improving performance on knowledge-intensive argument tasks.
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Liang Wang (Microsoft Corporation), Furu Wei (Microsoft Corporation)
RetrievalKnowledge DistillationRepresentation LearningTransformerContrastive LearningText
🎯 What it does: Proposed the SIMLM pre-training framework to enhance representation learning in dense passage retrieval models.
SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams
Te-Lin Wu (University of California Los Angeles), Seungwhan Moon (Meta AI)
GenerationData SynthesisTransformerLarge Language ModelPrompt EngineeringVision Language ModelVideoTextMultimodalityBenchmark
🎯 What it does: Created the SIMMC-VR dataset with a two-stage data collection pipeline (first generating object-centric dialogues and virtual perspective videos via programs, then having human language experts naturalize them), and defined four new tasks on this dataset (multimodal dialogue state tracking, multimodal coreference resolution, failure mode prediction, and dialogue response generation).
SimOAP: Improve Coherence and Consistency in Persona-based Dialogue Generation via Over-sampling and Post-evaluation
Junkai Zhou (Institute of Computing Technology Chinese Academy of Sciences), Xueqi Cheng (Institute of Computing Technology Chinese Academy of Sciences)
GenerationComputational EfficiencyKnowledge DistillationTransformerLarge Language ModelText
🎯 What it does: Proposes a two-phase method (SimOAP) that enhances the coherence and consistency of persona-based dialogue generation through large-scale oversampling and post-evaluation.
Simple and Effective Unsupervised Speech Translation
Changhan Wang (Meta), Juan Pino (Meta)
Domain AdaptationTransformerSupervised Fine-TuningContrastive LearningTextAudio
🎯 What it does: This paper proposes a simple and effective unsupervised speech translation framework. It first generates pseudo labels using unsupervised ASR, MT, and TTS models, then trains end-to-end S2TT and S2ST models. The recognition performance is further improved through unsupervised domain adaptation of wav2vec 2.0.
Simple Augmentations of Logical Rules for Neuro-Symbolic Knowledge Graph Completion
Ananjan Nandi (Indian Institute of Technology Delhi), Mausam (Indian Institute of Technology Delhi)
Explainability and InterpretabilityRepresentation LearningRecurrent Neural NetworkGraph Neural NetworkGraph
🎯 What it does: Investigate simple and general rule expansion techniques to improve the rule coverage and performance of neural symbolic knowledge graph completion (NS-KGC) models. The method converts existing rules into abductive forms, generates equivalent rules using inverse relationships, and creates new rules through local random walks. Subsequently, PCA scoring is used to filter low-quality rules, ultimately obtaining a high-quality, high-coverage rule set.
Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions
Satwik Bhattamishra (University of Oxford), Phil Blunsom (University of Oxford)
Representation LearningRecurrent Neural NetworkTransformerText
🎯 What it does: Analyze the preference of Transformer and LSTM during random initialization and training for low-sensitivity Boolean functions, and evaluate their generalization ability on sparse Boolean functions.
SIMSUM: Document-level Text Simplification via Simultaneous Summarization
Sofia Blinova (EPFL), Seyed Ali Bahrainian (EPFL)
GenerationTransformerLarge Language ModelPrompt EngineeringTextRetrieval-Augmented Generation
🎯 What it does: Propose SIMSUM, a two-stage document-level text simplification model that first generates a summary and then simplifies it;
Single Sequence Prediction over Reasoning Graphs for Multi-hop QA
Gowtham Ramesh (University of Wisconsin-Madison), Junjie Hu (University of Wisconsin-Madison)
Graph Neural NetworkTransformerLarge Language ModelTextGraphChain-of-Thought
🎯 What it does: This paper proposes a single-sequence prediction method called SEQGRAPH, which constructs a local entity-document graph and integrates graph neural network representations into the T5 encoder to generate sequences that include both reasoning paths and answers.
SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT
Aditya Yadavalli (Karya Inc.), Vera Tobin (Case Western Reserve University)
Representation LearningData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Construct the SLABERT framework, using child-directed speech (CDS) in different languages as L1 pre-training, then freezing the embedding layer for fine-tuning on English AD S L2, and evaluating the model's mastery of English grammar with BLiMP; simultaneously creating the MAO-CHILDES multilingual CDS corpus.
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks
Suwon Shon (ASAPP), Shinji Watanabe (Carnegie Mellon University)
ClassificationRecognitionTransformerTextBenchmarkAudio
🎯 What it does: Propose the SLUE Phase-2 benchmark suite, which includes four diversified spoken language understanding tasks (dialogue act classification, question answering, summarization, and named entity localization), and provides datasets, human annotations, pipeline and end-to-end baseline models, as well as evaluation metrics for each task.
Small Data, Big Impact: Leveraging Minimal Data for Effective Machine Translation
Jean Maillard (Meta AI), Francisco Guzman
GenerationData SynthesisData-Centric LearningTransformerTextBenchmark
🎯 What it does: For 39 low-resource languages, 6,000 professional translation sentence pairs were manually collected and publicly released for each language, constructing a high-quality seed dataset, and extensive experiments were conducted to evaluate the improvement in machine translation performance.
Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization
Ze-Feng Gao (Renmin University of China), Ji-Rong Wen (Renmin University of China)
Computational EfficiencyTransformerSupervised Fine-TuningTextBenchmark
🎯 What it does: Propose an over-parameterization (OPF) method for small pre-trained language models during the fine-tuning phase through matrix decomposition, significantly enhancing model performance without increasing inference latency.
Social-Group-Agnostic Bias Mitigation via the Stereotype Content Model
Ali Omrani (University of Southern California), Morteza Dehghani (University of Southern California)
Representation LearningTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: Propose a community-irrelevant bias mitigation method based on the social psychology stereotype content model (SCM), implemented on both static word embeddings and large language models;
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Artidoro Pagnoni (University of Washington), Chien-Sheng Wu (Salesforce AI Research)
GenerationTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Propose SOCRATIC pre-training — an unsupervised pre-training objective based on question generation and answering, enhancing query compliance in controllable summarization.
Soft Alignment Objectives for Robust Adaptation of Language Generation
Michal Štefánik (Masaryk University), Petr Sojka (Masaryk University)
GenerationDomain AdaptationTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Propose a soft alignment objective based on domain-agnostic semantic similarity to improve domain adaptation in language generation models and reduce catastrophic forgetting.
Soft Language Clustering for Multilingual Model Pre-training
Jiali Zeng (Tencent Inc), Jie Zhou (Tencent Inc)
Computational EfficiencyRepresentation LearningTransformerLarge Language ModelPrompt EngineeringContrastive LearningText
🎯 What it does: Propose the XLM-P model, which enhances the model in multilingual pre-training by using context-retrieved soft prompts as dynamic language clustering information.
Solving Math Word Problems via Cooperative Reasoning induced Language Models
Xinyu Zhu (Tsinghua University), Yujiu Yang (Tsinghua University)
Explainability and InterpretabilityComputational EfficiencyTransformerLarge Language ModelTextBenchmarkChain-of-Thought
🎯 What it does: Propose the CoRe framework, which utilizes a dual-system collaborative model composed of a generator (System 1) and a verifier (System 2), achieving mathematical word problem reasoning through Monte Carlo Tree Search (MCTS) and real-time scoring, while introducing a self-thinking mechanism to enhance model generalization.
Songs Across Borders: Singable and Controllable Neural Lyric Translation
Longshen Ou (National University of Singapore), Ye Wang (National University of Singapore)
GenerationTransformerSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: Explored how to use prompt-driven neural machine translation models to generate singable lyrics that comply with musical constraints.
Span-level Aspect-based Sentiment Analysis via Table Filling
Mao Zhang (University of Science and Technology of China), Linli Xu (Tencent)
ClassificationTransformerLarge Language ModelText
🎯 What it does: Propose a span-level ABSA model called TF-BERT based on table filling, which uses BERT to encode sentences and aspects, constructs an upper triangular table to record the intensity of all span pairs for each sentiment polarity, and then predicts sentiment polarity through two methods: table decoding or table aggregation. Additionally, sentiment consistency regularization is introduced to ensure consistency of the same span across different polarity tables.
Span-Selective Linear Attention Transformers for Effective and Robust Schema-Guided Dialogue State Tracking
Björn Bebensee (Samsung Research), Haejun Lee (Samsung Research)
TransformerContrastive LearningText
🎯 What it does: Proposed a novel linear attention Transformer called SPLAT for performing dialogue state tracking given a natural language service schema.
Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment
Tianshu Yu (Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences), Yongbin Li (Alibaba Group)
ClassificationRecognitionRepresentation LearningTransformerTextMultimodalityBenchmarkAudio
🎯 What it does: Proposed SPECTRA—the first pre-trained model for speech-text dialogues, aiming to enhance speech dialog understanding performance.
SPEECH: Structured Prediction with Energy-Based Event-Centric Hyperspheres
Shumin Deng (National University of Singapore), Bryan Hooi (National University of Singapore)
ClassificationRecognitionTransformerLarge Language ModelText
🎯 What it does: Propose a model called SPEECH that combines energy networks and event-centric hyperspheres for event detection, classification, and relation extraction.
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations
Paul-Ambroise Duquenne (Meta AI Research), Holger Schwenk (Meta AI Research)
Mixture of ExpertsTextAudio
🎯 What it does: This paper proposes SpeechMatrix, a large-scale multilingual speech-to-speech translation corpus mined from the VoxPopuli corpus, covering 17 languages with approximately 418k hours;
Split-NER: Named Entity Recognition via Two Question-Answering-based Classifications
Jatin Arora (Nuro Inc.), Youngja Park (IBM)
RecognitionComputational EfficiencyConvolutional Neural NetworkTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Decompose the named entity recognition task into two steps: first using a question-answering model for entity boundary detection, then using another question-answering model to classify the detected entities.
SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created through Human-Machine Collaboration
Hwaran Lee (NAVER AI Lab), Jung-Woo Ha (NAVER AI Lab)
Safty and PrivacyData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBenchmark
🎯 What it does: This paper generates and annotates a massive dataset named SQUARE containing Korean sensitive questions and acceptable responses through human-AI collaboration, and proposes a filtering-based regulation method based on an acceptable response classifier to enhance LLM safety.
SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control
Xiaochuang Han (University of Washington), Yulia Tsvetkov (University of Washington)
GenerationTransformerLarge Language ModelDiffusion modelText
🎯 What it does: Proposed a semi-autoregressive, simplex-based diffusion language model (SSD-LM) that can generate multiple token blocks in a single step during text generation and supports offline multi-modal control;
Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
Yasmine Karoui (Technical University of Munich), Karl Aberer (EPFL)
RetrievalDomain AdaptationTransformerVision Language ModelContrastive LearningMultimodality
🎯 What it does: This paper proposes the CLiCoTEA method, which utilizes a small amount of parallel corpus obtained from machine translation. By aligning context word embeddings, it transfers existing monolingual vision-language pre-training models to unseen languages, achieving zero-shot cross-lingual visual tasks.
StoryARG: a corpus of narratives and personal experiences in argumentative texts
Neele Falk (University of Stuttgart), Gabriella Lapesa (University of Stuttgart)
Data-Centric LearningTextBenchmark
🎯 What it does: Investigated the functions and effects of stories (personal experiences) in argumentative texts and constructed the StoryARG dataset.
StoryTrans: Non-Parallel Story Author-Style Transfer with Discourse Representations and Content Enhancing
Xuekai Zhu (Tsinghua University), Juan Liu (Wuhan University)
GenerationRepresentation LearningTransformerLarge Language ModelGenerative Adversarial NetworkText
🎯 What it does: Developed the StoryTrans model to achieve non-parallel story author style transfer, utilizing paragraph-level representations and learnable style embeddings, combined with mask-and-fill to enhance content preservation.
StoryWars: A Dataset and Instruction Tuning Baselines for Collaborative Story Understanding and Generation
Yulun Du (Columbia University), Lydia Chilton (Columbia University)
RecognitionGenerationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBenchmark
🎯 What it does: This paper proposes a collaborative story dataset and multi-task benchmark named STORYWARS, which constructs 101 tasks covering understanding and generation. Based on this, we design and evaluate a two-stage instruction-tuned model called INSTRUCTSTORY.
STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions
Michel Plüss, Mark Cieliebak (University of Applied Sciences and Arts Northwestern Switzerland)
RecognitionTransformerSupervised Fine-TuningTextBenchmarkAudio
🎯 What it does: Constructed a 343-hour Swiss German speech corpus STT4SG-350 covering 7 dialect regions, providing standard German text alignment along with training/validation/test sets.
Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning
Tharindu Cyril Weerasooriya (Rochester Institute of Technology), Christopher Homan (Rochester Institute of Technology)
Representation LearningData-Centric LearningConvolutional Neural NetworkTransformerText
🎯 What it does: Propose the CrowdOpinion method, which combines text features with label distribution through clustering, retaining subjective inconsistencies in human annotations via a two-phase approach (unsupervised clustering + supervised learning);
Subset Retrieval Nearest Neighbor Machine Translation
Hiroyuki Deguchi (Nara Institute of Science and Technology), Eiichiro Sumita (National Institute of Information and Communications Technology)
GenerationRetrievalDomain AdaptationComputational EfficiencyTransformerTextRetrieval-Augmented Generation
🎯 What it does: Propose Subset k NN-MT, which improves decoding speed while maintaining or even enhancing translation quality by first retrieving a subset of nearest neighbor sentences for the input sentence, then retrieving k nearest neighbor target words within that subset.
Substitution-based Semantic Change Detection using Contextual Embeddings
Dallas Card (University of Michigan)
Computational EfficiencyRepresentation LearningTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Use a contextual masked language model to obtain the top-k most probable substitutes for each word, statistically analyze their distribution, and detect semantic changes through Jensen-Shannon Divergence.
Summarizing, Simplifying, and Synthesizing Medical Evidence using GPT-3 (with Varying Success)
Chantal Shaib (Northeastern University), Byron Wallace
TransformerLarge Language ModelPrompt EngineeringTextBiomedical Data
🎯 What it does: Evaluated the performance of GPT-3-D3 in single and multi-document abstracts, simplification, and evidence synthesis for medical RCTs, and assessed factual accuracy through manual annotations by medical experts.
Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization
Yunlong Liang (Beijing Jiaotong University), Jie Zhou (Tencent Inc)
GenerationTransformerLarge Language ModelVision Language ModelMultimodality
🎯 What it does: Proposed a multimodal abstractive summarization framework SOV-MAS based on summary-oriented visual features, and constructed a large-scale multilingual multimodal summarization dataset MM-Sum.
Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversations
Dou Hu (Chinese Academy of Sciences), Songlin Hu (Chinese Academy of Sciences)
RecognitionRecurrent Neural NetworkTransformerGenerative Adversarial NetworkContrastive LearningText
🎯 What it does: Proposed the Supervised Adversarial Contrastive Learning (SACL) framework and implemented the SACL-LSTM model for the Emotion Recognition in Conversations (ERC) task;
Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models
Ehsan Doostmohammadi (Linköping University), Richard Johansson (Chalmers University of Technology)
RetrievalTransformerTextRetrieval-Augmented Generation
🎯 What it does: Investigate the retrieval mechanism of RETRO, demonstrating that its performance improvement mainly stems from surface-level token overlap. Subsequently, replace dense retrieval with BM25 for retrieval and propose a hybrid strategy that only uses BM25 for re-ranking.
SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages
Philippe Laban (Salesforce AI), Chien-Sheng Wu
GenerationTransformerLarge Language ModelTextBenchmark
🎯 What it does: Constructed the SWIPE dataset by leveraging the complete revision history of English Wikipedia and its simplified version to achieve document-level simplification alignment, and annotated over 5,000 document pairs with 19 fine-grained edit labels; simultaneously developed an automated editing recognition model;
Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step
Liunian Harold Li (University of California, Los Angeles), Yejin Choi (Allen Institute for Artificial Intelligence)
Knowledge DistillationTransformerLarge Language ModelSupervised Fine-TuningTextChain-of-Thought
🎯 What it does: Generate a large number of chain-of-thought reasoning examples using a teacher model, then distill a small language model with these examples as training data to enable self-reasoning;
Syntax and Geometry of Information
Raphaël Bailly (Centre National de la Recherche Scientifique Université Paris 1), Kata Gábor (Institut National des Langues et Civilisations Orientales)
Representation LearningData-Centric LearningText
🎯 What it does: Propose an information-theoretic syntactic generalization model that decomposes the corpus into semantic context and syntactic structure, introducing 'abstraction' as a new unsupervised learning objective, and experimentally validating it on unsupervised part-of-speech tagging tasks.
Synthesize, Prompt and Transfer: Zero-shot Conversational Question Generation with Pre-trained Language Model
Hongwei Zeng (Xi'an Jiaotong University), Weiping Fu (Xi'an Jiaotong University)
GenerationData SynthesisTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextRetrieval-Augmented Generation
🎯 What it does: Propose a zero-shot conversational question answering generation framework (SPARTA) that does not require manually annotated dialog data;
Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe
Xiang Yue (Ohio State University), Robert Sim (Microsoft)
ClassificationData SynthesisSafty and PrivacyTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Synthetic text generation with differential privacy by fine-tuning pre-trained language models (e.g., GPT-2) using DP-SGD and incorporating control codes.
Table and Image Generation for Investigating Knowledge of Entities in Pre-trained Vision and Language Models
Hidetaka Kamigaito (Nara Institute of Science and Technology), Taro Watanabe (Nara Institute of Science and Technology)
GenerationTransformerVision Language ModelImageMultimodalityTabular
🎯 What it does: This paper proposes a table and image generation task to examine the retention of entity knowledge by visual-language models, and constructs the WikiTIG dataset.
TableVLM: Multi-modal Pre-training for Table Structure Recognition
Leiyuan Chen (Fudan University), Xuanjing Huang (Fudan University)
RecognitionData SynthesisTransformerVision Language ModelImageTextMultimodality
🎯 What it does: Proposed TableVLM, a multi-modal pre-training model and the ComplexTable dataset, for identifying table structures from table images.
Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation
Matthieu Futeral (Inria Paris), Rachel Bawden (Inria Paris)
TransformerLarge Language ModelVision Language ModelContrastive LearningImageTextMultimodality
🎯 What it does: Proposed the VGAMT (Visually Guided and Adapted Machine Translation) model, which combines lightweight adapters and visual projections with vision-conditioned masked language modeling (VMLM) and multimodal translation (MMT) dual objectives for joint training on a frozen strong text-based MT model (mBART), while constructing the CoMMuTE benchmark dataset to evaluate the role of images in disambiguation.
Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment Detection
Yiwei Wei (Tianjin University), Meng Chen (JD AI Research)
ClassificationTransformerVision Language ModelMultimodality
🎯 What it does: This paper proposes a Multi-View Calibration Network (MVCN), addressing the modality heterogeneity issue in text-image sentiment detection by designing three modules: text-guided fusion, emotional consistency constraints, and adaptive loss calibration.
TAGPRIME: A Unified Framework for Relational Structure Extraction
I-Hung Hsu (University of Southern California), Nanyun Peng (University of California, Los Angeles)
TransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Propose the TAGPRIME unified framework, which injects conditional and relational information into the input via priming to achieve relation structure extraction.
Tailor: A Soft-Prompt-Based Approach to Attribute-Based Controlled Text Generation
Kexin Yang (Alibaba Group), Jun Xie (Alibaba Group)
GenerationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: Propose a control text generation method called Tailor based on soft prompts, which achieves single-attribute and multi-attribute generation by learning continuous prompts for each attribute and enabling their combination.
Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation
Yuxin Ren (Tsinghua University), Mu Li (Boson AI)
ClassificationKnowledge DistillationTransformerText
🎯 What it does: Propose a learning-to-teach framework called LGTM based on 'distillation influence,' which dynamically weights teacher training according to the impact of each training sample on student validation performance, thereby reducing student overfitting.
Target-Based Offensive Language Identification
Marcos Zampieri (George Mason University), Preslav Nakov (IBM Research)
ClassificationRecognitionRecurrent Neural NetworkTransformerLarge Language ModelText
🎯 What it does: This paper creates a new dataset called TBO (Target-Based Offensive Language Identification), which annotates English tweets with triplets of target, offensive phrases, and harmfulness, and conducts experimental evaluations on multiple models.