By: Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, Ajay Divakaran
Vision-language models (VLMs) have recently demonstrated strong efficacy as
visual assistants that can parse natural queries about the visual content and
generate human-like outputs. In this work, we explore the ability of these
models to demonstrate human-like reasoning based on the perceived information.
To address a crucial concern regarding the extent to which their reasoning
capabilities are fully consistent and grounded, we also measu... more
Vision-language models (VLMs) have recently demonstrated strong efficacy as
visual assistants that can parse natural queries about the visual content and
generate human-like outputs. In this work, we explore the ability of these
models to demonstrate human-like reasoning based on the perceived information.
To address a crucial concern regarding the extent to which their reasoning
capabilities are fully consistent and grounded, we also measure the reasoning
consistency of these models. We achieve this by proposing a chain-of-thought
(CoT) based consistency measure. However, such an evaluation requires a
benchmark that encompasses both high-level inference and detailed reasoning
chains, which is costly. We tackle this challenge by proposing a
LLM-Human-in-the-Loop pipeline, which notably reduces cost while simultaneously
ensuring the generation of a high-quality dataset. Based on this pipeline and
the existing coarse-grained annotated dataset, we build the CURE benchmark to
measure both the zero-shot reasoning performance and consistency of VLMs. We
evaluate existing state-of-the-art VLMs, and find that even the best-performing
model is unable to demonstrate strong visual reasoning capabilities and
consistency, indicating that substantial efforts are required to enable VLMs to
perform visual reasoning as systematically and consistently as humans. As an
early step, we propose a two-stage training framework aimed at improving both
the reasoning performance and consistency of VLMs. The first stage involves
employing supervised fine-tuning of VLMs using step-by-step reasoning samples
automatically generated by LLMs. In the second stage, we further augment the
training process by incorporating feedback provided by LLMs to produce
reasoning chains that are highly consistent and grounded. We empirically
highlight the effectiveness of our framework in both reasoning performance and
consistency.
less
By: Sharath Srivatsa, Srinath Srinivasa
Media houses reporting on public figures, often come with their own biases
stemming from their respective worldviews. A characterization of these
underlying patterns helps us in better understanding and interpreting news
stories. For this, we need diverse or subjective summarizations, which may not
be amenable for classifying into predefined class labels. This work proposes a
zero-shot approach for non-extractive or generative characterizat... more
Media houses reporting on public figures, often come with their own biases
stemming from their respective worldviews. A characterization of these
underlying patterns helps us in better understanding and interpreting news
stories. For this, we need diverse or subjective summarizations, which may not
be amenable for classifying into predefined class labels. This work proposes a
zero-shot approach for non-extractive or generative characterizations of person
entities from a corpus using GPT-2. We use well-articulated articles from
several well-known news media houses as a corpus to build a sound argument for
this approach. First, we fine-tune a GPT-2 pre-trained language model with a
corpus where specific person entities are characterized. Second, we further
fine-tune this with demonstrations of person entity characterizations, created
from a corpus of programmatically constructed characterizations. This twice
fine-tuned model is primed with manual prompts consisting of entity names that
were not previously encountered in the second fine-tuning, to generate a simple
sentence about the entity. The results were encouraging, when compared against
actual characterizations from the corpus.
less
nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resources
0upvotes
By: Piotr Nawrot
State-of-the-art language models like T5 have revolutionized the NLP
landscape, but their computational demands hinder a large portion of the
research community. To address this challenge, we present nanoT5, a
specially-optimized PyTorch framework for efficient pre-training and
fine-tuning of T5 models. Drawing on insights from optimizer differences and
prioritizing efficiency, nanoT5 allows a T5-Base model to be pre-trained on a
single GPU... more
State-of-the-art language models like T5 have revolutionized the NLP
landscape, but their computational demands hinder a large portion of the
research community. To address this challenge, we present nanoT5, a
specially-optimized PyTorch framework for efficient pre-training and
fine-tuning of T5 models. Drawing on insights from optimizer differences and
prioritizing efficiency, nanoT5 allows a T5-Base model to be pre-trained on a
single GPU in just 16 hours, without any loss in performance. With the
introduction of this open-source framework, we hope to widen the accessibility
to language modelling research and cater to the community's demand for more
user-friendly T5 (Encoder-Decoder) implementations. Our contributions,
including configurations, codebase, software/hardware insights, and pre-trained
models, are available to the public, aiming to strike a balance between
research accessibility and resource constraints in NLP.
less
By: Yuansheng Ni, Sichao Jiang, Xinyu wu, Hui Shen, Yuli Zhou
Recently, Instruction fine-tuning has risen to prominence as a potential
method for enhancing the zero-shot capabilities of Large Language Models (LLMs)
on novel tasks. This technique has shown an exceptional ability to boost the
performance of moderately sized LLMs, sometimes even reaching performance
levels comparable to those of much larger model variants. The focus is on the
robustness of instruction-tuned LLMs to seen and unseen tasks.... more
Recently, Instruction fine-tuning has risen to prominence as a potential
method for enhancing the zero-shot capabilities of Large Language Models (LLMs)
on novel tasks. This technique has shown an exceptional ability to boost the
performance of moderately sized LLMs, sometimes even reaching performance
levels comparable to those of much larger model variants. The focus is on the
robustness of instruction-tuned LLMs to seen and unseen tasks. We conducted an
exploration of six models including Alpaca, Vicuna, WizardLM, and Traditional
Task-oriented Models(Flan-T5-XL/XXL, T0++) using real-world relation extraction
datasets as case studies. We carried out a comprehensive evaluation of these
instruction-following LLMs which have been tuned based on open-domain
instructions and task-oriented instructions. The main discussion is their
performance and robustness towards instructions. We have observed that in most
cases, the model's performance in dealing with unfamiliar instructions tends to
worsen significantly, and the robustness of the model for RE instructions
deteriorates compared to QA. Further, we discovered that up until a certain
parameter size threshold (3B), the performance of the FLAN-T5 model improves as
the parameter count increases. The robustness of different scales of FLAN-T5
models to RE instruction is worse than the robustness to QA instruction.
less
By: Shuly Wintner, Safaa Shehadi, Yuli Zeira, Doreen Osmelak, Yuval Nov
Why do bilingual speakers code-switch (mix their two languages)? Among the
several theories that attempt to explain this natural and ubiquitous
phenomenon, the Triggering Hypothesis relates code-switching to the presence of
lexical triggers, specifically cognates and proper names, adjacent to the
switch point. We provide a fuller, more nuanced and refined exploration of the
triggering hypothesis, based on five large datasets in three langua... more
Why do bilingual speakers code-switch (mix their two languages)? Among the
several theories that attempt to explain this natural and ubiquitous
phenomenon, the Triggering Hypothesis relates code-switching to the presence of
lexical triggers, specifically cognates and proper names, adjacent to the
switch point. We provide a fuller, more nuanced and refined exploration of the
triggering hypothesis, based on five large datasets in three language pairs,
reflecting both spoken and written bilingual interactions. Our results show
that words that are assumed to reside in a mental lexicon shared by both
languages indeed trigger code-switching; that the tendency to switch depends on
the distance of the trigger from the switch point; and on whether the trigger
precedes or succeeds the switch; but not on the etymology of the trigger words.
We thus provide strong, robust, evidence-based confirmation to several
hypotheses on the relationships between lexical triggers and code-switching.
less
The Poison of Alignment
0upvotes
By: Aibek Bekbayev, Sungbae Chun, Yerzat Dulat, James Yamazaki
From the perspective of content safety issues, alignment has shown to limit
large language models' (LLMs) harmful content generation. This intentional
method of reinforcing models to not respond to certain user inputs seem to be
present in many modern open-source instruction tuning datasets such as
OpenAssistant or Guanaco. We introduce a novel insight to an instruction-tuned
model's performance affected by the presence of alignment in supe... more
From the perspective of content safety issues, alignment has shown to limit
large language models' (LLMs) harmful content generation. This intentional
method of reinforcing models to not respond to certain user inputs seem to be
present in many modern open-source instruction tuning datasets such as
OpenAssistant or Guanaco. We introduce a novel insight to an instruction-tuned
model's performance affected by the presence of alignment in supervised
fine-tuning dataset. To be specific, we noticed that alignment acts as if it is
poisoning the instruction dataset. Experimentally, we demonstrate that aligned
answers significantly worsen the performance of the resulting fine-tuned
model's on various reasoning benchmarks such as Big Bench (BBH), Massive
Multitask Language Understanding (MMLU), Human Eval, and Discrete Reasoning
Over Paragraphs (DROP), performing worse than the counterpart tuned without
alignment by 4-33%.
less
By: Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, Jing Xiao
In the realm of Large Language Models, the balance between instruction data
quality and quantity has become a focal point. Recognizing this, we introduce a
self-guided methodology for LLMs to autonomously discern and select cherry
samples from vast open-source datasets, effectively minimizing manual curation
and potential cost for instruction tuning an LLM. Our key innovation, the
Instruction-Following Difficulty (IFD) metric, emerges as a ... more
In the realm of Large Language Models, the balance between instruction data
quality and quantity has become a focal point. Recognizing this, we introduce a
self-guided methodology for LLMs to autonomously discern and select cherry
samples from vast open-source datasets, effectively minimizing manual curation
and potential cost for instruction tuning an LLM. Our key innovation, the
Instruction-Following Difficulty (IFD) metric, emerges as a pivotal tool to
identify discrepancies between a model's expected responses and its autonomous
generation prowess. Through the adept application of IFD, cherry samples are
pinpointed, leading to a marked uptick in model training efficiency. Empirical
validations on renowned datasets like Alpaca and WizardLM underpin our
findings; with a mere 10% of conventional data input, our strategy showcases
improved results. This synthesis of self-guided cherry-picking and the IFD
metric signifies a transformative leap in the optimization of LLMs, promising
both efficiency and resource-conscious advancements.
less
By: Quentin Brabant, Gwenole Lecorve, Lina M. Rojas-Barahona, Claire Gardent
We present KGConv, a large, conversational corpus of 71k conversations where
each question-answer pair is grounded in a Wikidata fact. Conversations contain
on average 8.6 questions and for each Wikidata fact, we provide multiple
variants (12 on average) of the corresponding question using templates, human
annotations, hand-crafted rules and a question rewriting neural model. We
provide baselines for the task of Knowledge-Based, Conversatio... more
We present KGConv, a large, conversational corpus of 71k conversations where
each question-answer pair is grounded in a Wikidata fact. Conversations contain
on average 8.6 questions and for each Wikidata fact, we provide multiple
variants (12 on average) of the corresponding question using templates, human
annotations, hand-crafted rules and a question rewriting neural model. We
provide baselines for the task of Knowledge-Based, Conversational Question
Generation. KGConv can further be used for other generation and analysis tasks
such as single-turn question generation from Wikidata triples, question
rewriting, question answering from conversation or from knowledge graphs and
quiz generation.
less
Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models
0upvotes
By: Joseph Marvin Imperial, Harish Tayyar Madabushi
Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL)
and the Common European Framework of Reference for Languages (CEFR) exist to
guide teachers and educators to properly assess the complexity of educational
materials before administering them for classroom use. In this study, we select
a diverse set of open and closed-source instruction-tuned language models and
investigate their performances in writing story complet... more
Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL)
and the Common European Framework of Reference for Languages (CEFR) exist to
guide teachers and educators to properly assess the complexity of educational
materials before administering them for classroom use. In this study, we select
a diverse set of open and closed-source instruction-tuned language models and
investigate their performances in writing story completions and simplifying
narratives$-$tasks that teachers perform$-$using standard-guided prompts
controlling text readability. Our extensive findings provide empirical proof of
how globally recognized models like ChatGPT may be considered less effective
and may require more refined prompts for these generative tasks compared to
other open-sourced models such as BLOOMZ and FlanT5$-$which have shown
promising results.
less
Advancing Text-to-GLOSS Neural Translation Using a Novel Hyper-parameter Optimization Technique
0upvotes
By: Younes Ouargani, Noussaima El Khattabi
In this paper, we investigate the use of transformers for Neural Machine
Translation of text-to-GLOSS for Deaf and Hard-of-Hearing communication. Due to
the scarcity of available data and limited resources for text-to-GLOSS
translation, we treat the problem as a low-resource language task. We use our
novel hyper-parameter exploration technique to explore a variety of
architectural parameters and build an optimal transformer-based architectu... more
In this paper, we investigate the use of transformers for Neural Machine
Translation of text-to-GLOSS for Deaf and Hard-of-Hearing communication. Due to
the scarcity of available data and limited resources for text-to-GLOSS
translation, we treat the problem as a low-resource language task. We use our
novel hyper-parameter exploration technique to explore a variety of
architectural parameters and build an optimal transformer-based architecture
specifically tailored for text-to-GLOSS translation. The study aims to improve
the accuracy and fluency of Neural Machine Translation generated GLOSS. This is
achieved by examining various architectural parameters including layer count,
attention heads, embedding dimension, dropout, and label smoothing to identify
the optimal architecture for improving text-to-GLOSS translation performance.
The experiments conducted on the PHOENIX14T dataset reveal that the optimal
transformer architecture outperforms previous work on the same dataset. The
best model reaches a ROUGE (Recall-Oriented Understudy for Gisting Evaluation)
score of 55.18% and a BLEU-1 (BiLingual Evaluation Understudy 1) score of
63.6%, outperforming state-of-the-art results on the BLEU1 and ROUGE score by
8.42 and 0.63 respectively.
less
By: Anirudh Mittal, Timo Schick, Mikel Artetxe, Jane Dwivedi-Yu
As increasingly sophisticated language models emerge, their trustworthiness
becomes a pivotal issue, especially in tasks such as summarization and
question-answering. Ensuring their responses are contextually grounded and
faithful is challenging due to the linguistic diversity and the myriad of
possible answers. In this paper, we introduce a novel approach to evaluate
faithfulness of machine-generated text by computing the longest noncontin... more
As increasingly sophisticated language models emerge, their trustworthiness
becomes a pivotal issue, especially in tasks such as summarization and
question-answering. Ensuring their responses are contextually grounded and
faithful is challenging due to the linguistic diversity and the myriad of
possible answers. In this paper, we introduce a novel approach to evaluate
faithfulness of machine-generated text by computing the longest noncontinuous
substring of the claim that is supported by the context, which we refer to as
the Longest Supported Subsequence (LSS). Using a new human-annotated dataset,
we finetune a model to generate LSS. We introduce a new method of evaluation
and demonstrate that these metrics correlate better with human ratings when LSS
is employed, as opposed to when it is not. Our proposed metric demonstrates an
18% enhancement over the prevailing state-of-the-art metric for faithfulness on
our dataset. Our metric consistently outperforms other metrics on a
summarization dataset across six different models. Finally, we compare several
popular Large Language Models (LLMs) for faithfulness using this metric. We
release the human-annotated dataset built for predicting LSS and our fine-tuned
model for evaluating faithfulness.
less
By: Zhen Bi, Ningyu Zhang, Yinuo Jiang, Shumin Deng, Guozhou Zheng, Huajun Chen
The reasoning capabilities of Large Language Models (LLMs) play a pivotal
role in the realm of embodied artificial intelligence. Although there are
effective methods like program-of-thought prompting for LLMs which uses
programming language to tackle complex reasoning tasks, the specific impact of
code data on the improvement of reasoning capabilities remains under-explored.
To address this gap, we propose complexity-impacted reasoning scor... more
The reasoning capabilities of Large Language Models (LLMs) play a pivotal
role in the realm of embodied artificial intelligence. Although there are
effective methods like program-of-thought prompting for LLMs which uses
programming language to tackle complex reasoning tasks, the specific impact of
code data on the improvement of reasoning capabilities remains under-explored.
To address this gap, we propose complexity-impacted reasoning score (CIRS),
which combines structural and logical attributes, to measure the correlation
between code and reasoning abilities. Specifically, we use the abstract syntax
tree to encode the structural information and calculate logical complexity by
considering the difficulty and the cyclomatic complexity. Through an empirical
analysis, we find not all code data of complexity can be learned or understood
by LLMs. Optimal level of complexity is critical to the improvement of
reasoning abilities by program-aided prompting. Then we design an
auto-synthesizing and stratifying algorithm, and apply it to instruction
generation for mathematical reasoning and code data filtering for code
generation tasks. Extensive results demonstrates the effectiveness of our
proposed approach. Code will be integrated into the EasyInstruct framework at
https://github.com/zjunlp/EasyInstruct.
less
By: Harish Tayyar Madabushi, Laurence Romain, Petar Milin, Dagmar Divjak
Recent progress in deep learning and natural language processing has given
rise to powerful models that are primarily trained on a cloze-like task and
show some evidence of having access to substantial linguistic information,
including some constructional knowledge. This groundbreaking discovery presents
an exciting opportunity for a synergistic relationship between computational
methods and Construction Grammar research. In this chapter, w... more
Recent progress in deep learning and natural language processing has given
rise to powerful models that are primarily trained on a cloze-like task and
show some evidence of having access to substantial linguistic information,
including some constructional knowledge. This groundbreaking discovery presents
an exciting opportunity for a synergistic relationship between computational
methods and Construction Grammar research. In this chapter, we explore three
distinct approaches to the interplay between computational methods and
Construction Grammar: (i) computational methods for text analysis, (ii)
computational Construction Grammar, and (iii) deep learning models, with a
particular focus on language models. We touch upon the first two approaches as
a contextual foundation for the use of computational methods before providing
an accessible, yet comprehensive overview of deep learning models, which also
addresses reservations construction grammarians may have. Additionally, we
delve into experiments that explore the emergence of constructionally relevant
information within these models while also examining the aspects of
Construction Grammar that may pose challenges for these models. This chapter
aims to foster collaboration between researchers in the fields of natural
language processing and Construction Grammar. By doing so, we hope to pave the
way for new insights and advancements in both these fields.
less
By: Elena Volodina University of Gothenburg, Simon Dobnik University of Gothenburg, Therese Lindström Tiedemann University of Helsinki, Xuan-Son Vu Umeå university
Accessibility of research data is critical for advances in many research
fields, but textual data often cannot be shared due to the personal and
sensitive information which it contains, e.g names or political opinions.
General Data Protection Regulation (GDPR) suggests pseudonymization as a
solution to secure open access to research data, but we need to learn more
about pseudonymization as an approach before adopting it for manipulation of
... more
Accessibility of research data is critical for advances in many research
fields, but textual data often cannot be shared due to the personal and
sensitive information which it contains, e.g names or political opinions.
General Data Protection Regulation (GDPR) suggests pseudonymization as a
solution to secure open access to research data, but we need to learn more
about pseudonymization as an approach before adopting it for manipulation of
research data. This paper outlines a research agenda within pseudonymization,
namely need of studies into the effects of pseudonymization on unstructured
data in relation to e.g. readability and language assessment, as well as the
effectiveness of pseudonymization as a way of protecting writer identity, while
also exploring different ways of developing context-sensitive algorithms for
detection, labelling and replacement of personal information in unstructured
data. The recently granted project on pseudonymization Grandma Karl is 27 years
old addresses exactly those challenges.
less
By: Jarno Vrolijk, David Graus
The increased digitization of the labour market has given researchers,
educators, and companies the means to analyze and better understand the labour
market. However, labour market resources, although available in high volumes,
tend to be unstructured, and as such, research towards methodologies for the
identification, linking, and extraction of entities becomes more and more
important. Against the backdrop of this quest for better labour m... more
The increased digitization of the labour market has given researchers,
educators, and companies the means to analyze and better understand the labour
market. However, labour market resources, although available in high volumes,
tend to be unstructured, and as such, research towards methodologies for the
identification, linking, and extraction of entities becomes more and more
important. Against the backdrop of this quest for better labour market
representations, resource constraints and the unavailability of large-scale
annotated data cause a reliance on human domain experts. We demonstrate the
effectiveness of prompt-based tuning of pre-trained language models (PLM) in
labour market specific applications. Our results indicate that cost-efficient
methods such as PTR and instruction tuning without exemplars can significantly
increase the performance of PLMs on downstream labour market applications
without introducing additional model layers, manual annotations, and data
augmentation.
less
Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languages
0upvotes
By: Baban Gain, Dibyanayan Bandyopadhyay, Samrat Mukherjee, Chandranath Adak, Asif Ekbal
The study investigates the effectiveness of utilizing multimodal information
in Neural Machine Translation (NMT). While prior research focused on using
multimodal data in low-resource scenarios, this study examines how image
features impact translation when added to a large-scale, pre-trained unimodal
NMT system. Surprisingly, the study finds that images might be redundant in
this context. Additionally, the research introduces synthetic noi... more
The study investigates the effectiveness of utilizing multimodal information
in Neural Machine Translation (NMT). While prior research focused on using
multimodal data in low-resource scenarios, this study examines how image
features impact translation when added to a large-scale, pre-trained unimodal
NMT system. Surprisingly, the study finds that images might be redundant in
this context. Additionally, the research introduces synthetic noise to assess
whether images help the model deal with textual noise. Multimodal models
slightly outperform text-only models in noisy settings, even with random
images. The study's experiments translate from English to Hindi, Bengali, and
Malayalam, outperforming state-of-the-art benchmarks significantly.
Interestingly, the effect of visual context varies with source text noise: no
visual context works best for non-noisy translations, cropped image features
are optimal for low noise, and full image features work better in high-noise
scenarios. This sheds light on the role of visual context, especially in noisy
settings, opening up a new research direction for Noisy Neural Machine
Translation in multimodal setups. The research emphasizes the importance of
combining visual and textual information for improved translation in various
environments.
less
By: Maximilian Li, Xander Davies, Max Nadeau
Language models often exhibit behaviors that improve performance on a
pre-training objective but harm performance on downstream tasks. We propose a
novel approach to removing undesirable behaviors by ablating a small number of
causal pathways between model components, with the intention of disabling the
computational circuit responsible for the bad behavior. Given a small dataset
of inputs where the model behaves poorly, we learn to ablate ... more
Language models often exhibit behaviors that improve performance on a
pre-training objective but harm performance on downstream tasks. We propose a
novel approach to removing undesirable behaviors by ablating a small number of
causal pathways between model components, with the intention of disabling the
computational circuit responsible for the bad behavior. Given a small dataset
of inputs where the model behaves poorly, we learn to ablate a small number of
important causal pathways. In the setting of reducing GPT-2 toxic language
generation, we find ablating just 12 of the 11.6K causal edges mitigates toxic
generation with minimal degradation of performance on other inputs.
less
By: György Orosz, Gergő Szabó, Péter Berkecz, Zsolt Szántó, Richárd Farkas
This paper presents a set of industrial-grade text processing models for
Hungarian that achieve near state-of-the-art performance while balancing
resource efficiency and accuracy. Models have been implemented in the spaCy
framework, extending the HuSpaCy toolkit with several improvements to its
architecture. Compared to existing NLP tools for Hungarian, all of our
pipelines feature all basic text processing steps including tokenization,
sen... more
This paper presents a set of industrial-grade text processing models for
Hungarian that achieve near state-of-the-art performance while balancing
resource efficiency and accuracy. Models have been implemented in the spaCy
framework, extending the HuSpaCy toolkit with several improvements to its
architecture. Compared to existing NLP tools for Hungarian, all of our
pipelines feature all basic text processing steps including tokenization,
sentence-boundary detection, part-of-speech tagging, morphological feature
tagging, lemmatization, dependency parsing and named entity recognition with
high accuracy and throughput. We thoroughly evaluated the proposed
enhancements, compared the pipelines with state-of-the-art tools and
demonstrated the competitive performance of the new models in all text
preprocessing steps. All experiments are reproducible and the pipelines are
freely available under a permissive license.
less
Thesis Distillation: Investigating The Impact of Bias in NLP Models on Hate Speech Detection
0upvotes
By: Fatma Elsafoury
This paper is a summary of the work in my PhD thesis. In which, I investigate
the impact of bias in NLP models on the task of hate speech detection from
three perspectives: explainability, offensive stereotyping bias, and fairness.
I discuss the main takeaways from my thesis and how they can benefit the
broader NLP community. Finally, I discuss important future research directions.
The findings of my thesis suggest that bias in NLP models i... more
This paper is a summary of the work in my PhD thesis. In which, I investigate
the impact of bias in NLP models on the task of hate speech detection from
three perspectives: explainability, offensive stereotyping bias, and fairness.
I discuss the main takeaways from my thesis and how they can benefit the
broader NLP community. Finally, I discuss important future research directions.
The findings of my thesis suggest that bias in NLP models impacts the task of
hate speech detection from all three perspectives. And that unless we start
incorporating social sciences in studying bias in NLP models, we will not
effectively overcome the current limitations of measuring and mitigating bias
in NLP models.
less
By: Pengzhi Gao, Ruiqing Zhang, Zhongjun He, Hua Wu, Haifeng Wang
Consistency regularization methods, such as R-Drop (Liang et al., 2021) and
CrossConST (Gao et al., 2023), have achieved impressive supervised and
zero-shot performance in the neural machine translation (NMT) field. Can we
also boost end-to-end (E2E) speech-to-text translation (ST) by leveraging
consistency regularization? In this paper, we conduct empirical studies on
intra-modal and cross-modal consistency and propose two training strateg... more
Consistency regularization methods, such as R-Drop (Liang et al., 2021) and
CrossConST (Gao et al., 2023), have achieved impressive supervised and
zero-shot performance in the neural machine translation (NMT) field. Can we
also boost end-to-end (E2E) speech-to-text translation (ST) by leveraging
consistency regularization? In this paper, we conduct empirical studies on
intra-modal and cross-modal consistency and propose two training strategies,
SimRegCR and SimZeroCR, for E2E ST in regular and zero-shot scenarios.
Experiments on the MuST-C benchmark show that our approaches achieve
state-of-the-art (SOTA) performance in most translation directions. The
analyses prove that regularization brought by the intra-modal consistency,
instead of modality gap, is crucial for the regular E2E ST, and the cross-modal
consistency could close the modality gap and boost the zero-shot E2E ST
performance.
less
By: Frederick Riemenschneider, Anette Frank
Intertextual allusions hold a pivotal role in Classical Philology, with Latin
authors frequently referencing Ancient Greek texts. Until now, the automatic
identification of these intertextual references has been constrained to
monolingual approaches, seeking parallels solely within Latin or Greek texts.
In this study, we introduce SPhilBERTa, a trilingual Sentence-RoBERTa model
tailored for Classical Philology, which excels at cross-lingual... more
Intertextual allusions hold a pivotal role in Classical Philology, with Latin
authors frequently referencing Ancient Greek texts. Until now, the automatic
identification of these intertextual references has been constrained to
monolingual approaches, seeking parallels solely within Latin or Greek texts.
In this study, we introduce SPhilBERTa, a trilingual Sentence-RoBERTa model
tailored for Classical Philology, which excels at cross-lingual semantic
comprehension and identification of identical sentences across Ancient Greek,
Latin, and English. We generate new training data by automatically translating
English texts into Ancient Greek. Further, we present a case study,
demonstrating SPhilBERTa's capability to facilitate automated detection of
intertextual parallels. Our models and resources are available at
https://github.com/Heidelberg-NLP/ancient-language-models.
less
By: Ruiqi Li, Liesbeth Allein, Damien Sileo, Marie-Francine Moens
The capabilities and use cases of automatic natural language processing (NLP)
have grown significantly over the last few years. While much work has been
devoted to understanding how humans deal with discourse connectives, this
phenomenon is understudied in computational systems. Therefore, it is important
to put NLP models under the microscope and examine whether they can adequately
comprehend, process, and reason within the complexity of n... more
The capabilities and use cases of automatic natural language processing (NLP)
have grown significantly over the last few years. While much work has been
devoted to understanding how humans deal with discourse connectives, this
phenomenon is understudied in computational systems. Therefore, it is important
to put NLP models under the microscope and examine whether they can adequately
comprehend, process, and reason within the complexity of natural language. In
this chapter, we introduce the main mechanisms behind automatic sentence
processing systems step by step and then focus on evaluating discourse
connective processing. We assess nine popular systems in their ability to
understand English discourse connectives and analyze how context and language
understanding tasks affect their connective comprehension. The results show
that NLP systems do not process all discourse connectives equally well and that
the computational processing complexity of different connective kinds is not
always consistently in line with the presumed complexity order found in human
processing. In addition, while humans are more inclined to be influenced during
the reading procedure but not necessarily in the final comprehension
performance, discourse connectives have a significant impact on the final
accuracy of NLP systems. The richer knowledge of connectives a system learns,
the more negative effect inappropriate connectives have on it. This suggests
that the correct explicitation of discourse connectives is important for
computational natural language processing.
less
Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
0upvotes
By: Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, Sara Hooker
The Mixture of Experts (MoE) is a widely known neural architecture where an
ensemble of specialized sub-models optimizes overall performance with a
constant computational cost. However, conventional MoEs pose challenges at
scale due to the need to store all experts in memory. In this paper, we push
MoE to the limit. We propose extremely parameter-efficient MoE by uniquely
combining MoE architecture with lightweight experts.Our MoE architect... more
The Mixture of Experts (MoE) is a widely known neural architecture where an
ensemble of specialized sub-models optimizes overall performance with a
constant computational cost. However, conventional MoEs pose challenges at
scale due to the need to store all experts in memory. In this paper, we push
MoE to the limit. We propose extremely parameter-efficient MoE by uniquely
combining MoE architecture with lightweight experts.Our MoE architecture
outperforms standard parameter-efficient fine-tuning (PEFT) methods and is on
par with full fine-tuning by only updating the lightweight experts -- less than
1% of an 11B parameters model. Furthermore, our method generalizes to unseen
tasks as it does not depend on any prior task knowledge. Our research
underscores the versatility of the mixture of experts architecture, showcasing
its ability to deliver robust performance even when subjected to rigorous
parameter constraints. Our code used in all the experiments is publicly
available here: https://github.com/for-ai/parameter-efficient-moe.
less
By: Shaltiel Shmidman, Avi Shmidman, Moshe Koppel
We present DictaBERT, a new state-of-the-art pre-trained BERT model for
modern Hebrew, outperforming existing models on most benchmarks. Additionally,
we release two fine-tuned versions of the model, designed to perform two
specific foundational tasks in the analysis of Hebrew texts: prefix
segmentation and morphological tagging. These fine-tuned models allow any
developer to perform prefix segmentation and morphological tagging of a Hebrew... more
We present DictaBERT, a new state-of-the-art pre-trained BERT model for
modern Hebrew, outperforming existing models on most benchmarks. Additionally,
we release two fine-tuned versions of the model, designed to perform two
specific foundational tasks in the analysis of Hebrew texts: prefix
segmentation and morphological tagging. These fine-tuned models allow any
developer to perform prefix segmentation and morphological tagging of a Hebrew
sentence with a single call to a HuggingFace model, without the need to
integrate any additional libraries or code. In this paper we describe the
details of the training as well and the results on the different benchmarks. We
release the models to the community, along with sample code demonstrating their
use. We release these models as part of our goal to help further research and
development in Hebrew NLP.
less
By: Songhua Yang, Chenghao Zhang, Hongfei Xu, Yuxiang Jia
The Biomedical Entity Normalization (BEN) task aims to align raw,
unstructured medical entities to standard entities, thus promoting data
coherence and facilitating better downstream medical applications. Recently,
prompt learning methods have shown promising results in this task. However,
existing research falls short in tackling the more complex Chinese BEN task,
especially in the few-shot scenario with limited medical data, and the vast
... more
The Biomedical Entity Normalization (BEN) task aims to align raw,
unstructured medical entities to standard entities, thus promoting data
coherence and facilitating better downstream medical applications. Recently,
prompt learning methods have shown promising results in this task. However,
existing research falls short in tackling the more complex Chinese BEN task,
especially in the few-shot scenario with limited medical data, and the vast
potential of the external medical knowledge base has yet to be fully harnessed.
To address these challenges, we propose a novel Knowledge-injected Prompt
Learning (PL-Knowledge) method. Specifically, our approach consists of five
stages: candidate entity matching, knowledge extraction, knowledge encoding,
knowledge injection, and prediction output. By effectively encoding the
knowledge items contained in medical entities and incorporating them into our
tailor-made knowledge-injected templates, the additional knowledge enhances the
model's ability to capture latent relationships between medical entities, thus
achieving a better match with the standard entities. We extensively evaluate
our model on a benchmark dataset in both few-shot and full-scale scenarios. Our
method outperforms existing baselines, with an average accuracy boost of
12.96\% in few-shot and 0.94\% in full-data cases, showcasing its excellence in
the BEN task.
less
By: Sakayo Toadoum Sari, Angela Fan, Lema Logamou Seknewna
In Africa, and the world at large, there is an increasing focus on developing
Neural Machine Translation (NMT) systems to overcome language barriers. NMT for
Low-resource language is particularly compelling as it involves learning with
limited labelled data. However, obtaining a well-aligned parallel corpus for
low-resource languages can be challenging. The disparity between the
technological advancement of a few global languages and the la... more
In Africa, and the world at large, there is an increasing focus on developing
Neural Machine Translation (NMT) systems to overcome language barriers. NMT for
Low-resource language is particularly compelling as it involves learning with
limited labelled data. However, obtaining a well-aligned parallel corpus for
low-resource languages can be challenging. The disparity between the
technological advancement of a few global languages and the lack of research on
NMT for local languages in Chad is striking. End-to-end NMT trials on
low-resource Chad languages have not been attempted. Additionally, there is a
dearth of online and well-structured data gathering for research in Natural
Language Processing, unlike some African languages. However, a guided approach
for data gathering can produce bitext data for many Chadian language
translation pairs with well-known languages that have ample data. In this
project, we created the first sba-Fr Dataset, which is a corpus of
Ngambay-to-French translations, and fine-tuned three pre-trained models using
this dataset. Our experiments show that the M2M100 model outperforms other
models with high BLEU scores on both original and original+synthetic data. The
publicly available bitext dataset can be used for research purposes.
less
By: Michael Ginn, Alexis Palmer
Morpheme glossing is a critical task in automated language documentation and
can benefit other downstream applications greatly. While state-of-the-art
glossing systems perform very well for languages with large amounts of existing
data, it is more difficult to create useful models for low-resource languages.
In this paper, we propose the use of a taxonomic loss function that exploits
morphological information to make morphological glossing ... more
Morpheme glossing is a critical task in automated language documentation and
can benefit other downstream applications greatly. While state-of-the-art
glossing systems perform very well for languages with large amounts of existing
data, it is more difficult to create useful models for low-resource languages.
In this paper, we propose the use of a taxonomic loss function that exploits
morphological information to make morphological glossing more performant when
data is scarce. We find that while the use of this loss function does not
outperform a standard loss function with regards to single-label prediction
accuracy, it produces better predictions when considering the top-n predicted
labels. We suggest this property makes the taxonomic loss function useful in a
human-in-the-loop annotation setting.
less
By: Lu Chen, Ruqing Zhang, Wei Huang, Wei Chen, Jiafeng Guo, Xueqi Cheng
The mainstream of data-driven abstractive summarization models tends to
explore the correlations rather than the causal relationships. Among such
correlations, there can be spurious ones which suffer from the language prior
learned from the training corpus and therefore undermine the overall
effectiveness of the learned model. To tackle this issue, we introduce a
Structural Causal Model (SCM) to induce the underlying causal structure of the... more
The mainstream of data-driven abstractive summarization models tends to
explore the correlations rather than the causal relationships. Among such
correlations, there can be spurious ones which suffer from the language prior
learned from the training corpus and therefore undermine the overall
effectiveness of the learned model. To tackle this issue, we introduce a
Structural Causal Model (SCM) to induce the underlying causal structure of the
summarization data. We assume several latent causal factors and non-causal
factors, representing the content and style of the document and summary.
Theoretically, we prove that the latent factors in our SCM can be identified by
fitting the observed training data under certain conditions. On the basis of
this, we propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq)
to learn the causal representations that can mimic the causal factors, guiding
us to pursue causal information for summary generation. The key idea is to
reformulate the Variational Auto-encoder (VAE) to fit the joint distribution of
the document and summary variables from the training corpus. Experimental
results on two widely used text summarization datasets demonstrate the
advantages of our approach.
less
By: Chau-Thang Phan, Quoc-Nam Nguyen, Kiet Van Nguyen
Link prediction task is vital to automatically understanding the structure of
large knowledge bases. In this paper, we present our system to solve this task
at the Data Science and Advanced Analytics 2023 Competition "Efficient and
Effective Link Prediction" (DSAA-2023 Competition) with a corpus containing
948,233 training and 238,265 for public testing. This paper introduces an
approach to link prediction in Wikipedia articles by formulati... more
Link prediction task is vital to automatically understanding the structure of
large knowledge bases. In this paper, we present our system to solve this task
at the Data Science and Advanced Analytics 2023 Competition "Efficient and
Effective Link Prediction" (DSAA-2023 Competition) with a corpus containing
948,233 training and 238,265 for public testing. This paper introduces an
approach to link prediction in Wikipedia articles by formulating it as a
natural language inference (NLI) task. Drawing inspiration from recent
advancements in natural language processing and understanding, we cast link
prediction as an NLI task, wherein the presence of a link between two articles
is treated as a premise, and the task is to determine whether this premise
holds based on the information presented in the articles. We implemented our
system based on the Sentence Pair Classification for Link Prediction for the
Wikipedia Articles task. Our system achieved 0.99996 Macro F1-score and 1.00000
Macro F1-score for the public and private test sets, respectively. Our team
UIT-NLP ranked 3rd in performance on the private test set, equal to the scores
of the first and second places. Our code is publicly for research purposes.
less
By: Nguyen Anh Tu, Hoang Thi Thu Uyen, Tu Minh Phuong, Ngo Xuan Bach
Multiple intent detection and slot filling are two fundamental and crucial
tasks in spoken language understanding. Motivated by the fact that the two
tasks are closely related, joint models that can detect intents and extract
slots simultaneously are preferred to individual models that perform each task
independently. The accuracy of a joint model depends heavily on the ability of
the model to transfer information between the two tasks so t... more
Multiple intent detection and slot filling are two fundamental and crucial
tasks in spoken language understanding. Motivated by the fact that the two
tasks are closely related, joint models that can detect intents and extract
slots simultaneously are preferred to individual models that perform each task
independently. The accuracy of a joint model depends heavily on the ability of
the model to transfer information between the two tasks so that the result of
one task can correct the result of the other. In addition, since a joint model
has multiple outputs, how to train the model effectively is also challenging.
In this paper, we present a method for multiple intent detection and slot
filling by addressing these challenges. First, we propose a bidirectional joint
model that explicitly employs intent information to recognize slots and slot
features to detect intents. Second, we introduce a novel method for training
the proposed joint model using supervised contrastive learning and
self-distillation. Experimental results on two benchmark datasets MixATIS and
MixSNIPS show that our method outperforms state-of-the-art models in both
tasks. The results also demonstrate the contributions of both bidirectional
design and the training method to the accuracy improvement. Our source code is
available at https://github.com/anhtunguyen98/BiSLU
less
Multi-party Goal Tracking with LLMs: Comparing Pre-training, Fine-tuning, and Prompt Engineering
0upvotes
By: Angus Addlesee, Weronika Sieińska, Nancie Gunson, Daniel Hernández Garcia, Christian Dondrup, Oliver Lemon
This paper evaluates the extent to which current Large Language Models (LLMs)
can capture task-oriented multi-party conversations (MPCs). We have recorded
and transcribed 29 MPCs between patients, their companions, and a social robot
in a hospital. We then annotated this corpus for multi-party goal-tracking and
intent-slot recognition. People share goals, answer each other's goals, and
provide other people's goals in MPCs - none of which oc... more
This paper evaluates the extent to which current Large Language Models (LLMs)
can capture task-oriented multi-party conversations (MPCs). We have recorded
and transcribed 29 MPCs between patients, their companions, and a social robot
in a hospital. We then annotated this corpus for multi-party goal-tracking and
intent-slot recognition. People share goals, answer each other's goals, and
provide other people's goals in MPCs - none of which occur in dyadic
interactions. To understand user goals in MPCs, we compared three methods in
zero-shot and few-shot settings: we fine-tuned T5, created pre-training tasks
to train DialogLM using LED, and employed prompt engineering techniques with
GPT-3.5-turbo, to determine which approach can complete this novel task with
limited data. GPT-3.5-turbo significantly outperformed the others in a few-shot
setting. The `reasoning' style prompt, when given 7% of the corpus as example
annotated conversations, was the best performing method. It correctly annotated
62.32% of the goal tracking MPCs, and 69.57% of the intent-slot recognition
MPCs. A `story' style prompt increased model hallucination, which could be
detrimental if deployed in safety-critical settings. We conclude that
multi-party conversations still challenge state-of-the-art LLMs.
less
By: Daniel Russo, Serra Sinem Tekiroglu, Marco Guerini
Fighting misinformation is a challenging, yet crucial, task. Despite the
growing number of experts being involved in manual fact-checking, this activity
is time-consuming and cannot keep up with the ever-increasing amount of Fake
News produced daily. Hence, automating this process is necessary to help curb
misinformation. Thus far, researchers have mainly focused on claim veracity
classification. In this paper, instead, we address the gener... more
Fighting misinformation is a challenging, yet crucial, task. Despite the
growing number of experts being involved in manual fact-checking, this activity
is time-consuming and cannot keep up with the ever-increasing amount of Fake
News produced daily. Hence, automating this process is necessary to help curb
misinformation. Thus far, researchers have mainly focused on claim veracity
classification. In this paper, instead, we address the generation of
justifications (textual explanation of why a claim is classified as either true
or false) and benchmark it with novel datasets and advanced baselines. In
particular, we focus on summarization approaches over unstructured knowledge
(i.e. news articles) and we experiment with several extractive and abstractive
strategies. We employed two datasets with different styles and structures, in
order to assess the generalizability of our findings. Results show that in
justification production summarization benefits from the claim information,
and, in particular, that a claim-driven extractive step improves abstractive
summarization performances. Finally, we show that although cross-dataset
experiments suffer from performance degradation, a unique model trained on a
combination of the two datasets is able to retain style information in an
efficient manner.
less
By: Chenrui Zhang, Lin Liu, Jinpeng Wang, Chuyuan Wang, Xiao Sun, Hongyu Wang, Mingchen Cai
As an effective tool for eliciting the power of Large Language Models (LLMs),
prompting has recently demonstrated unprecedented abilities across a variety of
complex tasks. To further improve the performance, prompt ensemble has
attracted substantial interest for tackling the hallucination and instability
of LLMs. However, existing methods usually adopt a two-stage paradigm, which
requires a pre-prepared set of prompts with substantial manu... more
As an effective tool for eliciting the power of Large Language Models (LLMs),
prompting has recently demonstrated unprecedented abilities across a variety of
complex tasks. To further improve the performance, prompt ensemble has
attracted substantial interest for tackling the hallucination and instability
of LLMs. However, existing methods usually adopt a two-stage paradigm, which
requires a pre-prepared set of prompts with substantial manual effort, and is
unable to perform directed optimization for different weak learners. In this
paper, we propose a simple, universal, and automatic method named PREFER (Pompt
Ensemble learning via Feedback-Reflect-Refine) to address the stated
limitations. Specifically, given the fact that weak learners are supposed to
focus on hard examples during boosting, PREFER builds a feedback mechanism for
reflecting on the inadequacies of existing weak learners. Based on this, the
LLM is required to automatically synthesize new prompts for iterative
refinement. Moreover, to enhance stability of the prompt effect evaluation, we
propose a novel prompt bagging method involving forward and backward thinking,
which is superior to majority voting and is beneficial for both feedback and
weight calculation in boosting. Extensive experiments demonstrate that our
PREFER achieves state-of-the-art performance in multiple types of tasks by a
significant margin. We have made our code publicly available.
less
By: Tongtao Ling, Chen Liao, Zhipeng Yu, Lei Chen, Shilei Huang, Yi Liu
In clinical dictation, utterances after automatic speech recognition (ASR)
without explicit punctuation marks may lead to the misunderstanding of dictated
reports. To give a precise and understandable clinical report with ASR,
automatic punctuation restoration is required. Considering a practical
scenario, we propose a fast and light pre-trained model for Chinese medical
punctuation restoration based on 'pretraining and fine-tuning' paradig... more
In clinical dictation, utterances after automatic speech recognition (ASR)
without explicit punctuation marks may lead to the misunderstanding of dictated
reports. To give a precise and understandable clinical report with ASR,
automatic punctuation restoration is required. Considering a practical
scenario, we propose a fast and light pre-trained model for Chinese medical
punctuation restoration based on 'pretraining and fine-tuning' paradigm. In
this work, we distill pre-trained models by incorporating supervised
contrastive learning and a novel auxiliary pre-training task (Punctuation Mark
Prediction) to make it well-suited for punctuation restoration. Our experiments
on various distilled models reveal that our model can achieve 95% performance
while 10% model size relative to state-of-the-art Chinese RoBERTa.
less
By: Peiying Wang, Sunlu Zeng, Junqing Chen, Lu Fan, Meng Chen, Youzheng Wu, Xiaodong He
Multimodal emotion recognition (MER) aims to detect the emotional status of a
given expression by combining the speech and text information. Intuitively,
label information should be capable of helping the model locate the salient
tokens/frames relevant to the specific emotion, which finally facilitates the
MER task. Inspired by this, we propose a novel approach for MER by leveraging
label information. Specifically, we first obtain the repre... more
Multimodal emotion recognition (MER) aims to detect the emotional status of a
given expression by combining the speech and text information. Intuitively,
label information should be capable of helping the model locate the salient
tokens/frames relevant to the specific emotion, which finally facilitates the
MER task. Inspired by this, we propose a novel approach for MER by leveraging
label information. Specifically, we first obtain the representative label
embeddings for both text and speech modalities, then learn the label-enhanced
text/speech representations for each utterance via label-token and label-frame
interactions. Finally, we devise a novel label-guided attentive fusion module
to fuse the label-aware text and speech representations for emotion
classification. Extensive experiments were conducted on the public IEMOCAP
dataset, and experimental results demonstrate that our proposed approach
outperforms existing baselines and achieves new state-of-the-art performance.
less
A Multi-Task Semantic Decomposition Framework with Task-specific Pre-training for Few-Shot NER
0upvotes
By: Guanting Dong, Zechen Wang, Jinxu Zhao, Gang Zhao, Daichi Guo, Dayuan Fu, Tingfeng Hui, Chen Zeng, Keqing He, Xuefeng Li, Liwen Wang, Xinyue Cui, Weiran Xu
The objective of few-shot named entity recognition is to identify named
entities with limited labeled instances. Previous works have primarily focused
on optimizing the traditional token-wise classification framework, while
neglecting the exploration of information based on NER data characteristics. To
address this issue, we propose a Multi-Task Semantic Decomposition Framework
via Joint Task-specific Pre-training (MSDP) for few-shot NER. D... more
The objective of few-shot named entity recognition is to identify named
entities with limited labeled instances. Previous works have primarily focused
on optimizing the traditional token-wise classification framework, while
neglecting the exploration of information based on NER data characteristics. To
address this issue, we propose a Multi-Task Semantic Decomposition Framework
via Joint Task-specific Pre-training (MSDP) for few-shot NER. Drawing
inspiration from demonstration-based and contrastive learning, we introduce two
novel pre-training tasks: Demonstration-based Masked Language Modeling (MLM)
and Class Contrastive Discrimination. These tasks effectively incorporate
entity boundary information and enhance entity representation in Pre-trained
Language Models (PLMs). In the downstream main task, we introduce a multi-task
joint optimization framework with the semantic decomposing method, which
facilitates the model to integrate two different semantic information for
entity classification. Experimental results of two few-shot NER benchmarks
demonstrate that MSDP consistently outperforms strong baselines by a large
margin. Extensive analyses validate the effectiveness and generalization of
MSDP.
less
By: Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, Tie-Yan Liu
Pre-trained language models have attracted increasing attention in the
biomedical domain, inspired by their great success in the general natural
language domain. Among the two main branches of pre-trained language models in
the general language domain, i.e., BERT (and its variants) and GPT (and its
variants), the first one has been extensively studied in the biomedical domain,
such as BioBERT and PubMedBERT. While they have achieved great s... more
Pre-trained language models have attracted increasing attention in the
biomedical domain, inspired by their great success in the general natural
language domain. Among the two main branches of pre-trained language models in
the general language domain, i.e., BERT (and its variants) and GPT (and its
variants), the first one has been extensively studied in the biomedical domain,
such as BioBERT and PubMedBERT. While they have achieved great success on a
variety of discriminative downstream biomedical tasks, the lack of generation
ability constrains their application scope. In this paper, we propose BioGPT, a
domain-specific generative Transformer language model pre-trained on large
scale biomedical literature. We evaluate BioGPT on six biomedical NLP tasks and
demonstrate that our model outperforms previous models on most tasks.
Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI
end-to-end relation extraction tasks respectively, and 78.2% accuracy on
PubMedQA, creating a new record. Our larger model BioGPT-Large achieves 81.0%
on PubMedQA. Our case study on text generation further demonstrates the
advantage of BioGPT on biomedical literature to generate fluent descriptions
for biomedical terms. Code is available at https://github.com/microsoft/BioGPT.
less
By: Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, Soheil Feizi
The rapid progress of Large Language Models (LLMs) has made them capable of
performing astonishingly well on various tasks including document completion
and question answering. The unregulated use of these models, however, can
potentially lead to malicious consequences such as plagiarism, generating fake
news, spamming, etc. Therefore, reliable detection of AI-generated text can be
critical to ensure the responsible use of LLMs. Recent work... more
The rapid progress of Large Language Models (LLMs) has made them capable of
performing astonishingly well on various tasks including document completion
and question answering. The unregulated use of these models, however, can
potentially lead to malicious consequences such as plagiarism, generating fake
news, spamming, etc. Therefore, reliable detection of AI-generated text can be
critical to ensure the responsible use of LLMs. Recent works attempt to tackle
this problem either using certain model signatures present in the generated
text outputs or by applying watermarking techniques that imprint specific
patterns onto them. In this paper, both empirically and theoretically, we show
that these detectors are not reliable in practical scenarios. Empirically, we
show that paraphrasing attacks, where a light paraphraser is applied on top of
the generative text model, can break a whole range of detectors, including the
ones using the watermarking schemes as well as neural network-based detectors
and zero-shot classifiers. We then provide a theoretical impossibility result
indicating that for a sufficiently good language model, even the best-possible
detector can only perform marginally better than a random classifier. Finally,
we show that even LLMs protected by watermarking schemes can be vulnerable
against spoofing attacks where adversarial humans can infer hidden watermarking
signatures and add them to their generated text to be detected as text
generated by the LLMs, potentially causing reputational damages to their
developers. We believe these results can open an honest conversation in the
community regarding the ethical and reliable use of AI-generated text.
less
By: Abhilash Datta, Souvic Chakraborty, Animesh Mukherjee
Personality types are important in various fields as they hold relevant
information about the characteristics of a human being in an explainable
format. They are often good predictors of a person's behaviors in a particular
environment and have applications ranging from candidate selection to marketing
and mental health. Recently automatic detection of personality traits from
texts has gained significant attention in computational linguisti... more
Personality types are important in various fields as they hold relevant
information about the characteristics of a human being in an explainable
format. They are often good predictors of a person's behaviors in a particular
environment and have applications ranging from candidate selection to marketing
and mental health. Recently automatic detection of personality traits from
texts has gained significant attention in computational linguistics. Most
personality detection and analysis methods have focused on small datasets
making their experimental observations often limited. To bridge this gap, we
focus on collecting and releasing the largest automatically curated dataset for
the research community which has 152 million tweets and 56 thousand data points
for the Myers-Briggs personality type (MBTI) prediction task. We perform a
series of extensive qualitative and quantitative studies on our dataset to
analyze the data patterns in a better way and infer conclusions. We show how
our intriguing analysis results often follow natural intuition. We also perform
a series of ablation studies to show how the baselines perform for our dataset.
less
By: Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, Gabriel Synnaeve
We release Code Llama, a family of large language models for code based on
Llama 2 providing state-of-the-art performance among open models, infilling
capabilities, support for large input contexts, and zero-shot instruction
following ability for programming tasks. We provide multiple flavors to cover a
wide range of applications: foundation models (Code Llama), Python
specializations (Code Llama - Python), and instruction-following models ... more
We release Code Llama, a family of large language models for code based on
Llama 2 providing state-of-the-art performance among open models, infilling
capabilities, support for large input contexts, and zero-shot instruction
following ability for programming tasks. We provide multiple flavors to cover a
wide range of applications: foundation models (Code Llama), Python
specializations (Code Llama - Python), and instruction-following models (Code
Llama - Instruct) with 7B, 13B and 34B parameters each. All models are trained
on sequences of 16k tokens and show improvements on inputs with up to 100k
tokens. 7B and 13B Code Llama and Code Llama - Instruct variants support
infilling based on surrounding content. Code Llama reaches state-of-the-art
performance among open models on several code benchmarks, with scores of up to
53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python
7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform
every other publicly available model on MultiPL-E. We release Code Llama under
a permissive license that allows for both research and commercial use.
less
LLM2KB: Constructing Knowledge Bases using instruction tuned context aware Large Language Models
0upvotes
By: Anmol Nayak, Hari Prasad Timmapathini
The advent of Large Language Models (LLM) has revolutionized the field of
natural language processing, enabling significant progress in various
applications. One key area of interest is the construction of Knowledge Bases
(KB) using these powerful models. Knowledge bases serve as repositories of
structured information, facilitating information retrieval and inference tasks.
Our paper proposes LLM2KB, a system for constructing knowledge base... more
The advent of Large Language Models (LLM) has revolutionized the field of
natural language processing, enabling significant progress in various
applications. One key area of interest is the construction of Knowledge Bases
(KB) using these powerful models. Knowledge bases serve as repositories of
structured information, facilitating information retrieval and inference tasks.
Our paper proposes LLM2KB, a system for constructing knowledge bases using
large language models, with a focus on the Llama 2 architecture and the
Wikipedia dataset. We perform parameter efficient instruction tuning for
Llama-2-13b-chat and StableBeluga-13B by training small injection models that
have only 0.05 % of the parameters of the base models using the Low Rank
Adaptation (LoRA) technique. These injection models have been trained with
prompts that are engineered to utilize Wikipedia page contexts of subject
entities fetched using a Dense Passage Retrieval (DPR) algorithm, to answer
relevant object entities for a given subject entity and relation. Our best
performing model achieved an average F1 score of 0.6185 across 21 relations in
the LM-KBC challenge held at the ISWC 2023 conference.
less
By: Lingjiao Chen, Matei Zaharia, James Zou
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM)
services. However, when and how these models are updated over time is opaque.
Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on
several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3)
opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating
code, 6) US Medical License tests, and 7) visual reasoning.... more
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM)
services. However, when and how these models are updated over time is opaque.
Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on
several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3)
opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating
code, 6) US Medical License tests, and 7) visual reasoning. We find that the
performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time.
For example, GPT-4 (March 2023) was reasonable at identifying prime vs.
composite numbers (84% accuracy) but GPT-4 (June 2023) was poor on these same
questions (51% accuracy). This is partly explained by a drop in GPT-4's amenity
to follow chain-of-thought prompting. Interestingly, GPT-3.5 was much better in
June than in March in this task. GPT-4 became less willing to answer sensitive
questions and opinion survey questions in June than in March. GPT-4 performed
better at multi-hop questions in June than in March, while GPT-3.5's
performance dropped on this task. Both GPT-4 and GPT-3.5 had more formatting
mistakes in code generation in June than in March. Overall, our findings show
that the behavior of the "same" LLM service can change substantially in a
relatively short amount of time, highlighting the need for continuous
monitoring of LLMs.
less
By: Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, Kaokao Lv
Large Language Models (LLMs) have proven their exceptional capabilities in
performing language-related tasks. However, their deployment poses significant
challenges due to their considerable memory and storage requirements. In
response to this issue, weight-only quantization, particularly 3 and 4-bit
weight-only quantization, has emerged as one of the most viable solutions. As
the number of bits decreases, the quantization grid broadens, th... more
Large Language Models (LLMs) have proven their exceptional capabilities in
performing language-related tasks. However, their deployment poses significant
challenges due to their considerable memory and storage requirements. In
response to this issue, weight-only quantization, particularly 3 and 4-bit
weight-only quantization, has emerged as one of the most viable solutions. As
the number of bits decreases, the quantization grid broadens, thus emphasizing
the importance of up and down rounding. While previous studies have
demonstrated that fine-tuning up and down rounding with the addition of
perturbations can enhance accuracy in some scenarios, our study is driven by
the precise and limited boundary of these perturbations, where only the
threshold for altering the rounding value is of significance. Consequently, we
propose a concise and highly effective approach for optimizing the weight
rounding task. Our method, named SignRound, involves lightweight block-wise
tuning using signed gradient descent, enabling us to achieve outstanding
results within 400 steps. SignRound outperforms the established baseline of
rounding-to-nearest (RTN) and competes impressively against recent methods,
without introducing additional inference overhead. The source code will be
publicly available at https://github.com/intel/neural-compressor soon.
less
By: Sergei Monakhov
In this paper I propose a new way of measuring linguistic productivity that
objectively assesses the ability of an affix to be used to coin new complex
words and, unlike other popular measures, is not directly dependent upon token
frequency. Specifically, I suggest that linguistic productivity may be viewed
as the probability of an affix to combine with a random base. The advantages of
this approach include the following. First, token frequ... more
In this paper I propose a new way of measuring linguistic productivity that
objectively assesses the ability of an affix to be used to coin new complex
words and, unlike other popular measures, is not directly dependent upon token
frequency. Specifically, I suggest that linguistic productivity may be viewed
as the probability of an affix to combine with a random base. The advantages of
this approach include the following. First, token frequency does not dominate
the productivity measure but naturally influences the sampling of bases.
Second, we are not just counting attested word types with an affix but rather
simulating the construction of these types and then checking whether they are
attested in the corpus. Third, a corpus-based approach and randomised design
assure that true neologisms and words coined long ago have equal chances to be
selected. The proposed algorithm is evaluated both on English and Russian data.
The obtained results provide some valuable insights into the relation of
linguistic productivity to the number of types and tokens. It looks like
burgeoning linguistic productivity manifests itself in an increasing number of
types. However, this process unfolds in two stages: first comes the increase in
high-frequency items, and only then follows the increase in low-frequency
items.
less
By: Alexander Tsvetkov. Alon Kipnis
We propose an unsupervised method to extract keywords and keyphrases from
texts based on a pre-trained language model (LM) and Shannon's information
maximization. Specifically, our method extracts phrases having the highest
conditional entropy under the LM. The resulting set of keyphrases turns out to
solve a relevant information-theoretic problem: if provided as side
information, it leads to the expected minimal binary code length in compr... more
We propose an unsupervised method to extract keywords and keyphrases from
texts based on a pre-trained language model (LM) and Shannon's information
maximization. Specifically, our method extracts phrases having the highest
conditional entropy under the LM. The resulting set of keyphrases turns out to
solve a relevant information-theoretic problem: if provided as side
information, it leads to the expected minimal binary code length in compressing
the text using the LM and an entropy encoder. Alternately, the resulting set is
an approximation via a causal LM to the set of phrases that minimize the
entropy of the text when conditioned upon it. Empirically, the method provides
results comparable to the most commonly used methods in various keyphrase
extraction benchmark challenges.
less
By: Şükrü Ozan, D. Emre Taşar
In this study, we aim to find a method to auto-tag sentences specific to a
domain. Our training data comprises short conversational sentences extracted
from chat conversations between company's customer representatives and web site
visitors. We manually tagged approximately 14 thousand visitor inputs into ten
basic categories, which will later be used in a transformer-based language
model with attention mechanisms for the ultimate goal of d... more
In this study, we aim to find a method to auto-tag sentences specific to a
domain. Our training data comprises short conversational sentences extracted
from chat conversations between company's customer representatives and web site
visitors. We manually tagged approximately 14 thousand visitor inputs into ten
basic categories, which will later be used in a transformer-based language
model with attention mechanisms for the ultimate goal of developing a chatbot
application that can produce meaningful dialogue. We considered three different
state-of-the-art models and reported their auto-tagging capabilities. We
achieved the best performance with the bidirectional encoder representation
from transformers (BERT) model. Implementation of the models used in these
experiments can be cloned from our GitHub repository and tested for similar
auto-tagging problems without much effort.
less
By: Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, Rif A. Saurous
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module. Building these components often requires extensive domain expertise and may contain brittle design choices. In this paper, we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters. Given <text, audio> pairs, the model can be trai... more
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module. Building these components often requires extensive domain expertise and may contain brittle design choices. In this paper, we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters. Given <text, audio> pairs, the model can be trained completely from scratch with random initialization. We present several key techniques to
make the sequence-to-sequence framework perform well for this challenging task. Tacotron achieves a 3.82 subjective 5-scale mean opinion score on US English, outperforming a production parametric system in terms of naturalness. In addition, since Tacotron generates speech at the frame level, it's substantially faster than sample-level autoregressive methods.
less
By: Michael Beukman, Manuel Fokam
Transfer learning has led to large gains in performance for nearly all NLP
tasks while making downstream models easier and faster to train. This has also
been extended to low-resourced languages, with some success. We investigate the
properties of cross-lingual transfer learning between ten low-resourced
languages, from the perspective of a named entity recognition task. We
specifically investigate how much adaptive fine-tuning and the choi... more
Transfer learning has led to large gains in performance for nearly all NLP
tasks while making downstream models easier and faster to train. This has also
been extended to low-resourced languages, with some success. We investigate the
properties of cross-lingual transfer learning between ten low-resourced
languages, from the perspective of a named entity recognition task. We
specifically investigate how much adaptive fine-tuning and the choice of
transfer language affect zero-shot transfer performance. We find that models
that perform well on a single language often do so at the expense of
generalising to others, while models with the best generalisation to other
languages suffer in individual language performance. Furthermore, the amount of
data overlap between the source and target datasets is a better predictor of
transfer performance than either the geographical or genetic distance between
the languages.
less
Attention Is All You Need
1upvote
By: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
The dominant sequence transduction models are based on complex recurrent or
convolutional neural networks in an encoder-decoder configuration. The best
performing models also connect the encoder and decoder through an attention
mechanism. We propose a new simple network architecture, the Transformer, based
solely on attention mechanisms, dispensing with recurrence and convolutions
entirely. Experiments on two machine translation tasks show ... more
The dominant sequence transduction models are based on complex recurrent or
convolutional neural networks in an encoder-decoder configuration. The best
performing models also connect the encoder and decoder through an attention
mechanism. We propose a new simple network architecture, the Transformer, based
solely on attention mechanisms, dispensing with recurrence and convolutions
entirely. Experiments on two machine translation tasks show these models to be
superior in quality while being more parallelizable and requiring significantly
less time to train. Our model achieves 28.4 BLEU on the WMT 2014
English-to-German translation task, improving over the existing best results,
including ensembles by over 2 BLEU. On the WMT 2014 English-to-French
translation task, our model establishes a new single-model state-of-the-art
BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction
of the training costs of the best models from the literature. We show that the
Transformer generalizes well to other tasks by applying it successfully to
English constituency parsing both with large and limited training data.
less
![[GPT-4] Technical Report](https://cdn.sciencecast.org/storage/blobs/proxy/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBbUlaIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--64c904e26cedc6cce38e37060f7fb8affb54f97c/20230424-5-6bj8h2)
[GPT-4] Technical Report
0upvotes
By: OpenAI
We report the development of GPT-4, a large-scale, multimodal model which can
accept image and text inputs and produce text outputs. While less capable than
humans in many real-world scenarios, GPT-4 exhibits human-level performance on
various professional and academic benchmarks, including passing a simulated bar
exam with a score around the top 10% of test takers. GPT-4 is a
Transformer-based model pre-trained to predict the next token in... more
We report the development of GPT-4, a large-scale, multimodal model which can
accept image and text inputs and produce text outputs. While less capable than
humans in many real-world scenarios, GPT-4 exhibits human-level performance on
various professional and academic benchmarks, including passing a simulated bar
exam with a score around the top 10% of test takers. GPT-4 is a
Transformer-based model pre-trained to predict the next token in a document.
The post-training alignment process results in improved performance on measures
of factuality and adherence to desired behavior. A core component of this
project was developing infrastructure and optimization methods that behave
predictably across a wide range of scales. This allowed us to accurately
predict some aspects of GPT-4's performance based on models trained with no
more than 1/1,000th the compute of GPT-4.
less
ChatGPT as Data Augmentation for Compositional Generalization: A Case Study in Open Intent Detection
0upvotes
By: Yihao Fang, Xianzhi Li, Stephen W. Thomas, Xiaodan Zhu
Open intent detection, a crucial aspect of natural language understanding,
involves the identification of previously unseen intents in user-generated
text. Despite the progress made in this field, challenges persist in handling
new combinations of language components, which is essential for compositional
generalization. In this paper, we present a case study exploring the use of
ChatGPT as a data augmentation technique to enhance compositio... more
Open intent detection, a crucial aspect of natural language understanding,
involves the identification of previously unseen intents in user-generated
text. Despite the progress made in this field, challenges persist in handling
new combinations of language components, which is essential for compositional
generalization. In this paper, we present a case study exploring the use of
ChatGPT as a data augmentation technique to enhance compositional
generalization in open intent detection tasks. We begin by discussing the
limitations of existing benchmarks in evaluating this problem, highlighting the
need for constructing datasets for addressing compositional generalization in
open intent detection tasks. By incorporating synthetic data generated by
ChatGPT into the training process, we demonstrate that our approach can
effectively improve model performance. Rigorous evaluation of multiple
benchmarks reveals that our method outperforms existing techniques and
significantly enhances open intent detection capabilities. Our findings
underscore the potential of large language models like ChatGPT for data
augmentation in natural language understanding tasks.
less
By: Thoralf Mildenberger
We propose a resampling-based approach for assessing keyness in corpus
linguistics based on suggestions by Gries (2006, 2022). Traditional approaches
based on hypothesis tests (e.g. Likelihood Ratio) model the copora as
independent identically distributed samples of tokens. This model does not
account for the often observed uneven distribution of occurences of a word
across a corpus. When occurences of a word are concentrated in few documen... more
We propose a resampling-based approach for assessing keyness in corpus
linguistics based on suggestions by Gries (2006, 2022). Traditional approaches
based on hypothesis tests (e.g. Likelihood Ratio) model the copora as
independent identically distributed samples of tokens. This model does not
account for the often observed uneven distribution of occurences of a word
across a corpus. When occurences of a word are concentrated in few documents,
large values of LLR and similar scores are in fact much more likely than
accounted for by the token-by-token sampling model, leading to false positives.
We replace the token-by-token sampling model by a model where corpora are
samples of documents rather than tokens, which is much closer to the way
corpora are actually assembled. We then use a permutation approach to
approximate the distribution of a given keyness score under the null hypothesis
of equal frequencies and obtain p-values for assessing significance. We do not
need any assumption on how the tokens are organized within or across documents,
and the approach works with basically *any* keyness score. Hence, appart from
obtaining more accurate p-values for scores like LLR, we can also assess
significance for e.g. the logratio which has been proposed as a measure of
effect size.
An efficient implementation of the proposed approach is provided in the `R`
package `keyperm` available from github.
less
By: Ciprian-Octavian Truică, Victor Tudose, Elena-Simona Apostol
Automatic semantic change methods try to identify the changes that appear
over time in the meaning of words by analyzing their usage in diachronic
corpora. In this paper, we analyze different strategies to create static and
contextual word embedding models, i.e., Word2Vec and ELMo, on real-world
English and Romanian datasets. To test our pipeline and determine the
performance of our models, we first evaluate both word embedding models on an... more
Automatic semantic change methods try to identify the changes that appear
over time in the meaning of words by analyzing their usage in diachronic
corpora. In this paper, we analyze different strategies to create static and
contextual word embedding models, i.e., Word2Vec and ELMo, on real-world
English and Romanian datasets. To test our pipeline and determine the
performance of our models, we first evaluate both word embedding models on an
English dataset (SEMEVAL-CCOHA). Afterward, we focus our experiments on a
Romanian dataset, and we underline different aspects of semantic changes in
this low-resource language, such as meaning acquisition and loss. The
experimental results show that, depending on the corpus, the most important
factors to consider are the choice of model and the distance to calculate a
score for detecting semantic change.
less
By: Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, Xipeng Qiu
Current speech large language models build upon discrete speech
representations, which can be categorized into semantic tokens and acoustic
tokens. However, existing speech tokens are not specifically designed for
speech language modeling. To assess the suitability of speech tokens for
building speech language models, we established the first benchmark,
SLMTokBench. Our results indicate that neither semantic nor acoustic tokens are
ideal fo... more
Current speech large language models build upon discrete speech
representations, which can be categorized into semantic tokens and acoustic
tokens. However, existing speech tokens are not specifically designed for
speech language modeling. To assess the suitability of speech tokens for
building speech language models, we established the first benchmark,
SLMTokBench. Our results indicate that neither semantic nor acoustic tokens are
ideal for this purpose. Therefore, we propose SpeechTokenizer, a unified speech
tokenizer for speech large language models. SpeechTokenizer adopts the
Encoder-Decoder architecture with residual vector quantization (RVQ). Unifying
semantic and acoustic tokens, SpeechTokenizer disentangles different aspects of
speech information hierarchically across different RVQ layers. Furthermore, We
construct a Unified Speech Language Model (USLM) leveraging SpeechTokenizer.
Experiments show that SpeechTokenizer performs comparably to EnCodec in speech
reconstruction and demonstrates strong performance on the SLMTokBench
benchmark. Also, USLM outperforms VALL-E in zero-shot Text-to-Speech tasks.
Code and models are available at
https://github.com/ZhangXInFD/SpeechTokenizer/.
less
By: Sebastian Gehrmann, Hendrik Strobelt, Alexander M. Rush
The rapid improvement of language models has raised the specter of abuse of
text generation systems. This progress motivates the development of simple
methods for detecting generated text that can be used by and explained to
non-experts. We develop GLTR, a tool to support humans in detecting whether a
text was generated by a model. GLTR applies a suite of baseline statistical
methods that can detect generation artifacts across common sampli... more
The rapid improvement of language models has raised the specter of abuse of
text generation systems. This progress motivates the development of simple
methods for detecting generated text that can be used by and explained to
non-experts. We develop GLTR, a tool to support humans in detecting whether a
text was generated by a model. GLTR applies a suite of baseline statistical
methods that can detect generation artifacts across common sampling schemes. In
a human-subjects study, we show that the annotation scheme provided by GLTR
improves the human detection-rate of fake text from 54% to 72% without any
prior training. GLTR is open-source and publicly deployed, and has already been
widely used to detect generated outputs
less
Simple is Better and Large is Not Enough: Towards Ensembling of Foundational Language Models
0upvotes
By: Nancy Tyagi, Aidin Shiri, Surjodeep Sarkar, Abhishek Kumar Umrawal, Manas Gaur
Foundational Language Models (FLMs) have advanced natural language processing
(NLP) research. Current researchers are developing larger FLMs (e.g., XLNet,
T5) to enable contextualized language representation, classification, and
generation. While developing larger FLMs has been of significant advantage, it
is also a liability concerning hallucination and predictive uncertainty.
Fundamentally, larger FLMs are built on the same foundations as... more
Foundational Language Models (FLMs) have advanced natural language processing
(NLP) research. Current researchers are developing larger FLMs (e.g., XLNet,
T5) to enable contextualized language representation, classification, and
generation. While developing larger FLMs has been of significant advantage, it
is also a liability concerning hallucination and predictive uncertainty.
Fundamentally, larger FLMs are built on the same foundations as smaller FLMs
(e.g., BERT); hence, one must recognize the potential of smaller FLMs which can
be realized through an ensemble. In the current research, we perform a reality
check on FLMs and their ensemble on benchmark and real-world datasets. We
hypothesize that the ensembling of FLMs can influence the individualistic
attention of FLMs and unravel the strength of coordination and cooperation of
different FLMs. We utilize BERT and define three other ensemble techniques:
{Shallow, Semi, and Deep}, wherein the Deep-Ensemble introduces a
knowledge-guided reinforcement learning approach. We discovered that the
suggested Deep-Ensemble BERT outperforms its large variation i.e. BERTlarge, by
a factor of many times using datasets that show the usefulness of NLP in
sensitive fields, such as mental health.
less
By: Baoli Zhang, Haining Xie, Pengfan Du, Junhao Chen, Pengfei Cao, Yubo Chen, Shengping Liu, Kang Liu, Jun Zhao
The unprecedented performance of large language models (LLMs) requires
comprehensive and accurate evaluation. We argue that for LLMs evaluation,
benchmarks need to be comprehensive and systematic. To this end, we propose the
ZhuJiu benchmark, which has the following strengths: (1) Multi-dimensional
ability coverage: We comprehensively evaluate LLMs across 7 ability dimensions
covering 51 tasks. Especially, we also propose a new benchmark th... more
The unprecedented performance of large language models (LLMs) requires
comprehensive and accurate evaluation. We argue that for LLMs evaluation,
benchmarks need to be comprehensive and systematic. To this end, we propose the
ZhuJiu benchmark, which has the following strengths: (1) Multi-dimensional
ability coverage: We comprehensively evaluate LLMs across 7 ability dimensions
covering 51 tasks. Especially, we also propose a new benchmark that focuses on
knowledge ability of LLMs. (2) Multi-faceted evaluation methods collaboration:
We use 3 different yet complementary evaluation methods to comprehensively
evaluate LLMs, which can ensure the authority and accuracy of the evaluation
results. (3) Comprehensive Chinese benchmark: ZhuJiu is the pioneering
benchmark that fully assesses LLMs in Chinese, while also providing equally
robust evaluation abilities in English. (4) Avoiding potential data leakage: To
avoid data leakage, we construct evaluation data specifically for 37 tasks. We
evaluate 10 current mainstream LLMs and conduct an in-depth discussion and
analysis of their results. The ZhuJiu benchmark and open-participation
leaderboard are publicly released at http://www.zhujiu-benchmark.com/ and we
also provide a demo video at https://youtu.be/qypkJ89L1Ic.
less
By: Ruibo Chen, Zhiyuan Zhang, Yi Liu, Ruihan Bao, Keiko Harimoto, Xu Sun
Multimodal stock trading volume movement prediction with stock-related news
is one of the fundamental problems in the financial area. Existing multimodal
works that train models from scratch face the problem of lacking universal
knowledge when modeling financial news. In addition, the models ability may be
limited by the lack of domain-related knowledge due to insufficient data in the
datasets. To handle this issue, we propose the Prompt-ba... more
Multimodal stock trading volume movement prediction with stock-related news
is one of the fundamental problems in the financial area. Existing multimodal
works that train models from scratch face the problem of lacking universal
knowledge when modeling financial news. In addition, the models ability may be
limited by the lack of domain-related knowledge due to insufficient data in the
datasets. To handle this issue, we propose the Prompt-based MUltimodal Stock
volumE prediction model (ProMUSE) to process text and time series modalities.
We use pre-trained language models for better comprehension of financial news
and adopt prompt learning methods to leverage their capability in universal
knowledge to model textual information. Besides, simply fusing two modalities
can cause harm to the unimodal representations. Thus, we propose a novel
cross-modality contrastive alignment while reserving the unimodal heads beside
the fusion head to mitigate this problem. Extensive experiments demonstrate
that our proposed ProMUSE outperforms existing baselines. Comprehensive
analyses further validate the effectiveness of our architecture compared to
potential variants and learning mechanisms.
less
By: Jaeseok Yoon, Seunghyun Hwang, Ran Han, Jeonguk Bang, Kee-Eung Kim
Although there have been remarkable advances in dialogue systems through the
dialogue systems technology competition (DSTC), it remains one of the key
challenges to building a robust task-oriented dialogue system with a speech
interface. Most of the progress has been made for text-based dialogue systems
since there are abundant datasets with written corpora while those with spoken
dialogues are very scarce. However, as can be seen from voic... more
Although there have been remarkable advances in dialogue systems through the
dialogue systems technology competition (DSTC), it remains one of the key
challenges to building a robust task-oriented dialogue system with a speech
interface. Most of the progress has been made for text-based dialogue systems
since there are abundant datasets with written corpora while those with spoken
dialogues are very scarce. However, as can be seen from voice assistant systems
such as Siri and Alexa, it is of practical importance to transfer the success
to spoken dialogues. In this paper, we describe our engineering effort in
building a highly successful model that participated in the speech-aware
dialogue systems technology challenge track in DSTC11. Our model consists of
three major modules: (1) automatic speech recognition error correction to
bridge the gap between the spoken and the text utterances, (2) text-based
dialogue system (D3ST) for estimating the slots and values using slot
descriptions, and (3) post-processing for recovering the error of the estimated
slot value. Our experiments show that it is important to use an explicit
automatic speech recognition error correction module, post-processing, and data
augmentation to adapt a text-based dialogue state tracker for spoken dialogue
corpora.
less
By: Thanh Thi Nguyen, Campbell Wilson, Janis Dalins
Detecting online sexual predatory behaviours and abusive language on social
media platforms has become a critical area of research due to the growing
concerns about online safety, especially for vulnerable populations such as
children and adolescents. Researchers have been exploring various techniques
and approaches to develop effective detection systems that can identify and
mitigate these risks. Recent development of large language models... more
Detecting online sexual predatory behaviours and abusive language on social
media platforms has become a critical area of research due to the growing
concerns about online safety, especially for vulnerable populations such as
children and adolescents. Researchers have been exploring various techniques
and approaches to develop effective detection systems that can identify and
mitigate these risks. Recent development of large language models (LLMs) has
opened a new opportunity to address this problem more effectively. This paper
proposes an approach to detection of online sexual predatory chats and abusive
language using the open-source pretrained Llama 2 7B-parameter model, recently
released by Meta GenAI. We fine-tune the LLM using datasets with different
sizes, imbalance degrees, and languages (i.e., English, Roman Urdu and Urdu).
Based on the power of LLMs, our approach is generic and automated without a
manual search for a synergy between feature extraction and classifier design
steps like conventional methods in this domain. Experimental results show a
strong performance of the proposed approach, which performs proficiently and
consistently across three distinct datasets with five sets of experiments. This
study's outcomes indicate that the proposed method can be implemented in
real-world applications (even with non-English languages) for flagging sexual
predators, offensive or toxic content, hate speech, and discriminatory language
in online discussions and comments to maintain respectful internet or digital
communities. Furthermore, it can be employed for solving text classification
problems with other potential applications such as sentiment analysis, spam and
phishing detection, sorting legal documents, fake news detection, language
identification, user intent recognition, text-based product categorization,
medical record analysis, and resume screening.
less
![[GPT-3] Language Models are Few-Shot Learners](https://cdn.sciencecast.org/storage/blobs/proxy/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBbUFaIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--269eaa44d0e9d3bc7e6d2b0dc3fad69b02940e17/20230424-5-bv3szu)
By: Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei
Recent work has demonstrated substantial gains on many NLP tasks and
benchmarks by pre-training on a large corpus of text followed by fine-tuning on
a specific task. While typically task-agnostic in architecture, this method
still requires task-specific fine-tuning datasets of thousands or tens of
thousands of examples. By contrast, humans can generally perform a new language
task from only a few examples or from simple instructions - somet... more
Recent work has demonstrated substantial gains on many NLP tasks and
benchmarks by pre-training on a large corpus of text followed by fine-tuning on
a specific task. While typically task-agnostic in architecture, this method
still requires task-specific fine-tuning datasets of thousands or tens of
thousands of examples. By contrast, humans can generally perform a new language
task from only a few examples or from simple instructions - something which
current NLP systems still largely struggle to do. Here we show that scaling up
language models greatly improves task-agnostic, few-shot performance, sometimes
even reaching competitiveness with prior state-of-the-art fine-tuning
approaches. Specifically, we train GPT-3, an autoregressive language model with
175 billion parameters, 10x more than any previous non-sparse language model,
and test its performance in the few-shot setting. For all tasks, GPT-3 is
applied without any gradient updates or fine-tuning, with tasks and few-shot
demonstrations specified purely via text interaction with the model. GPT-3
achieves strong performance on many NLP datasets, including translation,
question-answering, and cloze tasks, as well as several tasks that require
on-the-fly reasoning or domain adaptation, such as unscrambling words, using a
novel word in a sentence, or performing 3-digit arithmetic. At the same time,
we also identify some datasets where GPT-3's few-shot learning still struggles,
as well as some datasets where GPT-3 faces methodological issues related to
training on large web corpora. Finally, we find that GPT-3 can generate samples
of news articles which human evaluators have difficulty distinguishing from
articles written by humans. We discuss broader societal impacts of this finding
and of GPT-3 in general.
less
By: Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, Maosong Sun
Recently there has been a significant surge in multimodal learning in terms
of both image-to-text and text-to-image generation. However, the success is
typically limited to English, leaving other languages largely behind. Building
a competitive counterpart in other languages is highly challenging due to the
low-resource nature of non-English multimodal data (i.e., lack of large-scale,
high-quality image-text data). In this work, we propose ... more
Recently there has been a significant surge in multimodal learning in terms
of both image-to-text and text-to-image generation. However, the success is
typically limited to English, leaving other languages largely behind. Building
a competitive counterpart in other languages is highly challenging due to the
low-resource nature of non-English multimodal data (i.e., lack of large-scale,
high-quality image-text data). In this work, we propose MPM, an effective
training paradigm for training large multimodal models in low-resource
languages. MPM demonstrates that Multilingual language models can Pivot
zero-shot Multimodal learning across languages. Specifically, based on a strong
multilingual large language model, multimodal models pretrained on English-only
image-text data can well generalize to other languages in a zero-shot manner
for both image-to-text and text-to-image generation, even surpassing models
trained on image-text data in native languages. Taking Chinese as a practice of
MPM, we build large multimodal models VisCPM in image-to-text and text-to-image
generation, which achieve state-of-the-art (open-source) performance in
Chinese. To facilitate future research, we open-source codes and model weights
at https://github.com/OpenBMB/VisCPM.git.
less
By: Hossein Amirkhani, Mohammad AzariJafari, Zohreh Pourjafari, Soroush Faridan-Jahromi, Zeinab Kouhkan, Azadeh Amirak
Natural language inference (NLI) is known as one of the central tasks in
natural language processing (NLP) which encapsulates many fundamental aspects
of language understanding. With the considerable achievements of data-hungry
deep learning methods in NLP tasks, a great amount of effort has been devoted
to develop more diverse datasets for different languages. In this paper, we
present a new dataset for the NLI task in the Persian language... more
Natural language inference (NLI) is known as one of the central tasks in
natural language processing (NLP) which encapsulates many fundamental aspects
of language understanding. With the considerable achievements of data-hungry
deep learning methods in NLP tasks, a great amount of effort has been devoted
to develop more diverse datasets for different languages. In this paper, we
present a new dataset for the NLI task in the Persian language, also known as
Farsi, which is one of the dominant languages in the Middle East. This dataset,
named FarsTail, includes 10,367 samples which are provided in both the Persian
language as well as the indexed format to be useful for non-Persian
researchers. The samples are generated from 3,539 multiple-choice questions
with the least amount of annotator interventions in a way similar to the
SciTail dataset. A carefully designed multi-step process is adopted to ensure
the quality of the dataset. We also present the results of traditional and
state-of-the-art methods on FarsTail including different embedding methods such
as word2vec, fastText, ELMo, BERT, and LASER, as well as different modeling
approaches such as DecompAtt, ESIM, HBMP, and ULMFiT to provide a solid
baseline for the future research. The best obtained test accuracy is 83.38%
which shows that there is a big room for improving the current methods to be
useful for real-world NLP applications in different languages. We also
investigate the extent to which the models exploit superficial clues, also
known as dataset biases, in FarsTail, and partition the test set into easy and
hard subsets according to the success of biased models. The dataset is
available at https://github.com/dml-qom/FarsTail
less
By: Linkai Peng, Baorian Nuchged, Yingming Gao
People have long hoped for a conversational system that can assist in
real-life situations, and recent progress on large language models (LLMs) is
bringing this idea closer to reality. While LLMs are often impressive in
performance, their efficacy in real-world scenarios that demand expert
knowledge remains unclear. LLMs are believed to hold the most potential and
value in education, especially in the development of Artificial intelligence
... more
People have long hoped for a conversational system that can assist in
real-life situations, and recent progress on large language models (LLMs) is
bringing this idea closer to reality. While LLMs are often impressive in
performance, their efficacy in real-world scenarios that demand expert
knowledge remains unclear. LLMs are believed to hold the most potential and
value in education, especially in the development of Artificial intelligence
(AI) based virtual teachers capable of facilitating language learning. Our
focus is centered on evaluating the efficacy of LLMs in the realm of education,
specifically in the areas of spoken language learning which encompass
phonetics, phonology, and second language acquisition. We introduce a new
multiple-choice question dataset to evaluate the effectiveness of LLMs in the
aforementioned scenarios, including understanding and application of spoken
language knowledge. In addition, we investigate the influence of various
prompting techniques such as zero- and few-shot method (prepending the question
with question-answer exemplars), chain-of-thought (CoT, think step-by-step),
in-domain exampler and external tools (Google, Wikipedia). We conducted
large-scale evaluation on popular LLMs (20 distinct models) using these
methods. We achieved significant performance improvements compared to the
zero-shot baseline in the practical questions reasoning (GPT-3.5, 49.1% ->
63.1%; LLaMA2-70B-Chat, 42.2% -> 48.6%). We found that models of different
sizes have good understanding of concepts in phonetics, phonology, and second
language acquisition, but show limitations in reasoning for real-world
problems. Additionally, we also explore preliminary findings on conversational
communication.
less
By: Daniel Deutsch, Juraj Juraska, Mara Finkelstein, and Markus Freitag
As research on machine translation moves to translating text beyond the
sentence level, it remains unclear how effective automatic evaluation metrics
are at scoring longer translations. In this work, we first propose a method for
creating paragraph-level data for training and meta-evaluating metrics from
existing sentence-level data. Then, we use these new datasets to benchmark
existing sentence-level metrics as well as train learned metric... more
As research on machine translation moves to translating text beyond the
sentence level, it remains unclear how effective automatic evaluation metrics
are at scoring longer translations. In this work, we first propose a method for
creating paragraph-level data for training and meta-evaluating metrics from
existing sentence-level data. Then, we use these new datasets to benchmark
existing sentence-level metrics as well as train learned metrics at the
paragraph level. Interestingly, our experimental results demonstrate that using
sentence-level metrics to score entire paragraphs is equally as effective as
using a metric designed to work at the paragraph level. We speculate this
result can be attributed to properties of the task of reference-based
evaluation as well as limitations of our datasets with respect to capturing all
types of phenomena that occur in paragraph-level translations.
less
By: Yue Wang, Xinrui Wang, Juntao Li, Jinxiong Chang, Qishen Zhang, Zhongyi Liu, Guannan Zhang, Min Zhang
Instruction tuning is instrumental in enabling Large Language Models~(LLMs)
to follow user instructions to complete various open-domain tasks. The success
of instruction tuning depends on the availability of high-quality instruction
data. Owing to the exorbitant cost and substandard quality of human annotation,
recent works have been deeply engaged in the exploration of the utilization of
powerful closed-source models to generate instructio... more
Instruction tuning is instrumental in enabling Large Language Models~(LLMs)
to follow user instructions to complete various open-domain tasks. The success
of instruction tuning depends on the availability of high-quality instruction
data. Owing to the exorbitant cost and substandard quality of human annotation,
recent works have been deeply engaged in the exploration of the utilization of
powerful closed-source models to generate instruction data automatically.
However, these methods carry potential risks arising from the usage
requirements of powerful closed-source models, which strictly forbid the
utilization of their outputs to develop machine learning models. To deal with
this problem, in this work, we explore alternative approaches to generate
high-quality instruction data that do not rely on closed-source models. Our
exploration includes an investigation of various existing instruction
generation methods, culminating in the integration of the most efficient
variant with two novel strategies to enhance the quality further. Evaluation
results from two benchmarks and the GPT-4 model demonstrate the effectiveness
of our generated instruction data, which can outperform Alpaca, a method
reliant on closed-source models. We hope that more progress can be achieved in
generating high-quality instruction data without using closed-source models.
less
By: Yachao Zhao, Bo Wang, Dongming Zhao, Kun Huang, Yan Wang, Ruifang He, Yuexian Hou
Recent researches indicate that Pre-trained Large Language Models (LLMs)
possess cognitive constructs similar to those observed in humans, prompting
researchers to investigate the cognitive aspects of LLMs. This paper focuses on
explicit and implicit social bias, a distinctive two-level cognitive construct
in psychology. It posits that individuals' explicit social bias, which is their
conscious expression of bias in the statements, may diff... more
Recent researches indicate that Pre-trained Large Language Models (LLMs)
possess cognitive constructs similar to those observed in humans, prompting
researchers to investigate the cognitive aspects of LLMs. This paper focuses on
explicit and implicit social bias, a distinctive two-level cognitive construct
in psychology. It posits that individuals' explicit social bias, which is their
conscious expression of bias in the statements, may differ from their implicit
social bias, which represents their unconscious bias. We propose a two-stage
approach and discover a parallel phenomenon in LLMs known as "re-judge
inconsistency" in social bias. In the initial stage, the LLM is tasked with
automatically completing statements, potentially incorporating implicit social
bias. However, in the subsequent stage, the same LLM re-judges the biased
statement generated by itself but contradicts it. We propose that this re-judge
inconsistency can be similar to the inconsistency between human's unaware
implicit social bias and their aware explicit social bias. Experimental
investigations on ChatGPT and GPT-4 concerning common gender biases examined in
psychology corroborate the highly stable nature of the re-judge inconsistency.
This finding may suggest that diverse cognitive constructs emerge as LLMs'
capabilities strengthen. Consequently, leveraging psychological theories can
provide enhanced insights into the underlying mechanisms governing the
expressions of explicit and implicit constructs in LLMs.
less
By: Maximilian Mozes, Xuanli He, Bennett Kleinberg, Lewis D. Griffin
Spurred by the recent rapid increase in the development and distribution of
large language models (LLMs) across industry and academia, much recent work has
drawn attention to safety- and security-related threats and vulnerabilities of
LLMs, including in the context of potentially criminal activities.
Specifically, it has been shown that LLMs can be misused for fraud,
impersonation, and the generation of malware; while other authors have
con... more
Spurred by the recent rapid increase in the development and distribution of
large language models (LLMs) across industry and academia, much recent work has
drawn attention to safety- and security-related threats and vulnerabilities of
LLMs, including in the context of potentially criminal activities.
Specifically, it has been shown that LLMs can be misused for fraud,
impersonation, and the generation of malware; while other authors have
considered the more general problem of AI alignment. It is important that
developers and practitioners alike are aware of security-related problems with
such models. In this paper, we provide an overview of existing - predominantly
scientific - efforts on identifying and mitigating threats and vulnerabilities
arising from LLMs. We present a taxonomy describing the relationship between
threats caused by the generative capabilities of LLMs, prevention measures
intended to address such threats, and vulnerabilities arising from imperfect
prevention measures. With our work, we hope to raise awareness of the
limitations of LLMs in light of such security concerns, among both experienced
developers and novel users of such technologies.
less
By: Qiao Jin, Yifan Yang, Qingyu Chen, Zhiyong Lu
While large language models (LLMs) have been successfully applied to various
tasks, they still face challenges with hallucinations. Augmenting LLMs with
domain-specific tools such as database utilities can facilitate easier and more
precise access to specialized knowledge. In this paper, we present GeneGPT, a
novel method for teaching LLMs to use the Web APIs of the National Center for
Biotechnology Information (NCBI) for answering genomics... more
While large language models (LLMs) have been successfully applied to various
tasks, they still face challenges with hallucinations. Augmenting LLMs with
domain-specific tools such as database utilities can facilitate easier and more
precise access to specialized knowledge. In this paper, we present GeneGPT, a
novel method for teaching LLMs to use the Web APIs of the National Center for
Biotechnology Information (NCBI) for answering genomics questions.
Specifically, we prompt Codex to solve the GeneTuring tests with NCBI Web APIs
by in-context learning and an augmented decoding algorithm that can detect and
execute API calls. Experimental results show that GeneGPT achieves
state-of-the-art performance on eight tasks in the GeneTuring benchmark with an
average score of 0.83, largely surpassing retrieval-augmented LLMs such as the
new Bing (0.44), biomedical LLMs such as BioMedLM (0.08) and BioGPT (0.04), as
well as GPT-3 (0.16) and ChatGPT (0.12). Our further analyses suggest that: (1)
API demonstrations have good cross-task generalizability and are more useful
than documentations for in-context learning; (2) GeneGPT can generalize to
longer chains of API calls and answer multi-hop questions in GeneHop, a novel
dataset introduced in this work; (3) Different types of errors are enriched in
different tasks, providing valuable insights for future improvements.
less
By: V. K. Cody Bumgardner, Aaron Mullen, Sam Armstrong, Caylin Hickey, Jeff Talbert
This paper introduces an approach that combines the language reasoning
capabilities of large language models (LLMs) with the benefits of local
training to tackle complex, domain-specific tasks. Specifically, the authors
demonstrate their approach by extracting structured condition codes from
pathology reports. The proposed approach utilizes local LLMs, which can be
fine-tuned to respond to specific generative instructions and provide
struct... more
This paper introduces an approach that combines the language reasoning
capabilities of large language models (LLMs) with the benefits of local
training to tackle complex, domain-specific tasks. Specifically, the authors
demonstrate their approach by extracting structured condition codes from
pathology reports. The proposed approach utilizes local LLMs, which can be
fine-tuned to respond to specific generative instructions and provide
structured outputs. The authors collected a dataset of over 150k uncurated
surgical pathology reports, containing gross descriptions, final diagnoses, and
condition codes. They trained different model architectures, including LLaMA,
BERT and LongFormer and evaluated their performance. The results show that the
LLaMA-based models significantly outperform BERT-style models across all
evaluated metrics, even with extremely reduced precision. The LLaMA models
performed especially well with large datasets, demonstrating their ability to
handle complex, multi-label tasks. Overall, this work presents an effective
approach for utilizing LLMs to perform domain-specific tasks using accessible
hardware, with potential applications in the medical domain, where complex data
extraction and classification are required.
less
By: Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, Timothy Baldwin
With the rapid evolution of large language models (LLMs), new and
hard-to-predict harmful capabilities are emerging. This requires developers to
be able to identify risks through the evaluation of "dangerous capabilities" in
order to responsibly deploy LLMs. In this work, we collect the first
open-source dataset to evaluate safeguards in LLMs, and deploy safer
open-source LLMs at a low cost. Our dataset is curated and filtered to consist
on... more
With the rapid evolution of large language models (LLMs), new and
hard-to-predict harmful capabilities are emerging. This requires developers to
be able to identify risks through the evaluation of "dangerous capabilities" in
order to responsibly deploy LLMs. In this work, we collect the first
open-source dataset to evaluate safeguards in LLMs, and deploy safer
open-source LLMs at a low cost. Our dataset is curated and filtered to consist
only of instructions that responsible language models should not follow. We
annotate and assess the responses of six popular LLMs to these instructions.
Based on our annotation, we proceed to train several BERT-like classifiers, and
find that these small classifiers can achieve results that are comparable with
GPT-4 on automatic safety evaluation. Warning: this paper contains example data
that may be offensive, harmful, or biased.
less
By: Sharath Chandra Guntuku, Thomas Talhelm, Garrick Sherman, Angel Fan, Salvatore Giorgi, Liuqing Wei, Lyle H. Ungar
We used natural language processing to analyze a billion words to study
cultural differences on Weibo, one of China's largest social media platforms.
We compared predictions from two common explanations about cultural differences
in China (economic development and urban-rural differences) against the
less-obvious legacy of rice versus wheat farming. Rice farmers had to
coordinate shared irrigation networks and exchange labor to cope with hi... more
We used natural language processing to analyze a billion words to study
cultural differences on Weibo, one of China's largest social media platforms.
We compared predictions from two common explanations about cultural differences
in China (economic development and urban-rural differences) against the
less-obvious legacy of rice versus wheat farming. Rice farmers had to
coordinate shared irrigation networks and exchange labor to cope with higher
labor requirements. In contrast, wheat relied on rainfall and required half as
much labor. We test whether this legacy made southern China more
interdependent. Across all word categories, rice explained twice as much
variance as economic development and urbanization. Rice areas used more words
reflecting tight social ties, holistic thought, and a cautious, prevention
orientation. We then used Twitter data comparing prefectures in Japan, which
largely replicated the results from China. This provides crucial evidence of
the rice theory in a different nation, language, and platform.
less
By: Jian Hu, Li Tao, June Yang, Chandler Zhou
Learning from human preferences is crucial for language models (LMs) to
effectively cater to human needs and societal values. Previous research has
made notable progress by leveraging human feedback to follow instructions.
However, these approaches rely primarily on online reinforcement learning (RL)
techniques like Proximal Policy Optimization (PPO), which have been proven
unstable and challenging to tune for language models. Moreover, PPO... more
Learning from human preferences is crucial for language models (LMs) to
effectively cater to human needs and societal values. Previous research has
made notable progress by leveraging human feedback to follow instructions.
However, these approaches rely primarily on online reinforcement learning (RL)
techniques like Proximal Policy Optimization (PPO), which have been proven
unstable and challenging to tune for language models. Moreover, PPO requires
complex distributed system implementation, hindering the efficiency of
large-scale distributed training. In this study, we propose an offline
reinforcement learning from human feedback (RLHF) framework to align LMs using
pre-generated samples without interacting with RL environments. Specifically,
we explore maximum likelihood estimation (MLE) with filtering, reward-weighted
regression (RWR), and Decision Transformer (DT) to align language models to
human preferences. By employing a loss function similar to supervised
fine-tuning, our methods ensure more stable model training than PPO with a
simple machine learning system~(MLSys) and much fewer (around 12.3\%) computing
resources. Experimental results demonstrate the DT alignment outperforms other
Offline RLHF methods and is better than PPO.
less
By: Yijin Liu, Xianfeng Zeng, Fandong Meng, Jie Zhou
Large language models (LLMs) are capable of performing conditional sequence
generation tasks, such as translation or summarization, through instruction
fine-tuning. The fine-tuning data is generally sequentially concatenated from a
specific task instruction, an input sentence, and the corresponding response.
Considering the locality modeled by the self-attention mechanism of LLMs, these
models face the risk of instruction forgetting when ge... more
Large language models (LLMs) are capable of performing conditional sequence
generation tasks, such as translation or summarization, through instruction
fine-tuning. The fine-tuning data is generally sequentially concatenated from a
specific task instruction, an input sentence, and the corresponding response.
Considering the locality modeled by the self-attention mechanism of LLMs, these
models face the risk of instruction forgetting when generating responses for
long input sentences. To mitigate this issue, we propose enhancing the
instruction-following capability of LLMs by shifting the position of task
instructions after the input sentences. Theoretical analysis suggests that our
straightforward method can alter the model's learning focus, thereby
emphasizing the training of instruction-following capabilities. Concurrently,
experimental results demonstrate that our approach consistently outperforms
traditional settings across various model scales (1B / 7B / 13B) and different
sequence generation tasks (translation and summarization), without any
additional data or annotation costs. Notably, our method significantly improves
the zero-shot performance on conditional sequence generation, e.g., up to 9.7
BLEU points on WMT zero-shot translation tasks.
less
By: Tyler A. Chang, Zhuowen Tu, Benjamin K. Bergen
How do language models learn to make predictions during pre-training? To
study this question, we extract learning curves from five autoregressive
English language model pre-training runs, for 1M tokens in context. We observe
that the language models generate short repetitive phrases before learning to
generate longer and more coherent text. We quantify the final surprisal,
within-run variability, age of acquisition, forgettability, and cros... more
How do language models learn to make predictions during pre-training? To
study this question, we extract learning curves from five autoregressive
English language model pre-training runs, for 1M tokens in context. We observe
that the language models generate short repetitive phrases before learning to
generate longer and more coherent text. We quantify the final surprisal,
within-run variability, age of acquisition, forgettability, and cross-run
variability of learning curves for individual tokens in context. More frequent
tokens reach lower final surprisals, exhibit less variability within and across
pre-training runs, are learned earlier, and are less likely to be "forgotten"
during pre-training. Higher n-gram probabilities further accentuate these
effects. Independent of the target token, shorter and more frequent contexts
correlate with marginally more stable and quickly acquired predictions. Effects
of part-of-speech are also small, although nouns tend to be acquired later and
less stably than verbs, adverbs, and adjectives. Our work contributes to a
better understanding of language model pre-training dynamics and informs the
deployment of stable language models in practice.
less
By: Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, Dacheng Tao
Adapter tuning, which updates only a few parameters, has become a mainstream
method for fine-tuning pretrained language models to downstream tasks. However,
it often yields subpar results in few-shot learning. AdapterFusion, which
assembles pretrained adapters using composition layers tailored to specific
tasks, is a possible solution but significantly increases trainable parameters
and deployment costs. Despite this, our preliminary study ... more
Adapter tuning, which updates only a few parameters, has become a mainstream
method for fine-tuning pretrained language models to downstream tasks. However,
it often yields subpar results in few-shot learning. AdapterFusion, which
assembles pretrained adapters using composition layers tailored to specific
tasks, is a possible solution but significantly increases trainable parameters
and deployment costs. Despite this, our preliminary study reveals that even
single adapters can outperform Adapterfusion in few-shot learning, urging us to
propose \textbf{\texttt{Merging Pretrained Adapters}} (MerA) that efficiently
incorporates pretrained adapters to a single model through model fusion.
Extensive experiments on two PLMs demonstrate that MerA achieves substantial
improvements compared to both single adapters and AdapterFusion. To further
enhance the capacity of MerA, we also introduce a simple yet effective
technique, referred to as the "\textit{same-track}" setting, that merges
adapters from the same track of pretraining tasks. With the implementation of
the "\textit{same-track}" setting, we observe even more impressive gains,
surpassing the performance of both full fine-tuning and adapter tuning by a
substantial margin, e.g., 3.5\% in MRPC and 5.0\% in MNLI.
less
By: Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, Zhang Xiong
Equipped with Chain-of-Thought (CoT), Large language models (LLMs) have shown
impressive reasoning ability in various downstream tasks. Even so, suffering
from hallucinations and the inability to access external knowledge, LLMs often
come with incorrect or unfaithful intermediate reasoning steps, especially in
the context of answering knowledge-intensive tasks such as KBQA. To alleviate
this issue, we propose a framework called Knowledge-Dr... more
Equipped with Chain-of-Thought (CoT), Large language models (LLMs) have shown
impressive reasoning ability in various downstream tasks. Even so, suffering
from hallucinations and the inability to access external knowledge, LLMs often
come with incorrect or unfaithful intermediate reasoning steps, especially in
the context of answering knowledge-intensive tasks such as KBQA. To alleviate
this issue, we propose a framework called Knowledge-Driven Chain-of-Thought
(KD-CoT) to verify and modify reasoning traces in CoT via interaction with
external knowledge, and thus overcome the hallucinations and error propagation.
Concretely, we formulate the CoT rationale process of LLMs into a structured
multi-round QA format. In each round, LLMs interact with a QA system that
retrieves external knowledge and produce faithful reasoning traces based on
retrieved precise answers. The structured CoT reasoning of LLMs is facilitated
by our developed KBQA CoT collection, which serves as in-context learning
demonstrations and can also be utilized as feedback augmentation to train a
robust retriever. Extensive experiments on WebQSP and ComplexWebQuestion
datasets demonstrate the effectiveness of proposed KD-CoT in task-solving
reasoning generation, which outperforms the vanilla CoT ICL with an absolute
success rate of 8.0% and 5.1%. Furthermore, our proposed feedback-augmented
retriever outperforms the state-of-the-art baselines for retrieving knowledge,
achieving significant improvement in Hit performance.
less
By: Jianzhe Lin, Maurice Diesendruck, Liang Du, Robin Abraham
Many LLMs are trained to perform zero-shot or few-shot inference using
instruction-based prompts. Crafting prompts for these LLMs typically requires
the user to provide a detailed task description, examples of context and
completion, and single example of context for inference. This regular prompt
baseline is referred to as SinglePrompt in this paper. However, for NLP tasks
where each data point for inference is not necessarily lengthy, the... more
Many LLMs are trained to perform zero-shot or few-shot inference using
instruction-based prompts. Crafting prompts for these LLMs typically requires
the user to provide a detailed task description, examples of context and
completion, and single example of context for inference. This regular prompt
baseline is referred to as SinglePrompt in this paper. However, for NLP tasks
where each data point for inference is not necessarily lengthy, the token count
for instructions and few-shot examples in the prompt may be considerably larger
than that of the data point, resulting in lower token-resource utilization
compared with encoder-based models like fine-tuned BERT. This cost-efficiency
issue, affecting inference speed and compute budget, counteracts the many
benefits LLMs have to offer. This paper aims to alleviate the preceding problem
by batching multiple data points into a single prompt, a prompting strategy we
refer to as BatchPrompt. This strategy increases the density of data points,
which in turn leads to improved token utilization. Applying BatchPrompt
naively, however, is very challenging due to significant performance
degradation, as observed in our experiments. We also noticed varying inference
outcomes for the same data point appearing in different positions within a
prompt. To address the quality issue while remain high token-resource
utilization, we introduce Batch Permutation and Ensembling for BatchPrompt, a
simple way that recovers labeling quality through majority votes from data
points placed in varying positions in a batch at the price of more token usage.
To counterbalance the additional token usage caused by the voting process, we
further propose Self-reflection-guided EArly Stopping, which can terminate the
voting process early for data points the LLM confidently handles.
less
By: Hina Raja, Asim Munawar, Mohammad Delsoz, Mohammad Elahi, Yeganeh Madadi, Amr Hassan, Hashem Abu Serhan, Onur Inam, Luis Hermandez, Sang Tran, Wuqas Munir, Alaa Abd-Alrazaq, Hao Chen, SiamakYousefi
Purpose: In this paper, we present an automated method for article
classification, leveraging the power of Large Language Models (LLM). The
primary focus is on the field of ophthalmology, but the model is extendable to
other fields. Methods: We have developed a model based on Natural Language
Processing (NLP) techniques, including advanced LLMs, to process and analyze
the textual content of scientific papers. Specifically, we have employed
... more
Purpose: In this paper, we present an automated method for article
classification, leveraging the power of Large Language Models (LLM). The
primary focus is on the field of ophthalmology, but the model is extendable to
other fields. Methods: We have developed a model based on Natural Language
Processing (NLP) techniques, including advanced LLMs, to process and analyze
the textual content of scientific papers. Specifically, we have employed
zero-shot learning (ZSL) LLM models and compared against Bidirectional and
Auto-Regressive Transformers (BART) and its variants, and Bidirectional Encoder
Representations from Transformers (BERT), and its variant such as distilBERT,
SciBERT, PubmedBERT, BioBERT. Results: The classification results demonstrate
the effectiveness of LLMs in categorizing large number of ophthalmology papers
without human intervention. Results: To evalute the LLMs, we compiled a dataset
(RenD) of 1000 ocular disease-related articles, which were expertly annotated
by a panel of six specialists into 15 distinct categories. The model achieved
mean accuracy of 0.86 and mean F1 of 0.85 based on the RenD dataset.
Conclusion: The proposed framework achieves notable improvements in both
accuracy and efficiency. Its application in the domain of ophthalmology
showcases its potential for knowledge organization and retrieval in other
domains too. We performed trend analysis that enables the researchers and
clinicians to easily categorize and retrieve relevant papers, saving time and
effort in literature review and information gathering as well as identification
of emerging scientific trends within different disciplines. Moreover, the
extendibility of the model to other scientific fields broadens its impact in
facilitating research and trend analysis across diverse disciplines.
less
By: Yongrui Chen, Haiyun Jiang, Xinting Huang, Shuming Shi, Guilin Qi
High-quality instruction-tuning data is critical to improving LLM
capabilities. Existing data collection methods are limited by unrealistic
manual labeling costs or by the hallucination of relying solely on LLM
generation. To address the problems, this paper presents a scalable method to
automatically collect high-quality instructional adaptation data by training
language models to automatically design tasks based on human-written texts.
In... more
High-quality instruction-tuning data is critical to improving LLM
capabilities. Existing data collection methods are limited by unrealistic
manual labeling costs or by the hallucination of relying solely on LLM
generation. To address the problems, this paper presents a scalable method to
automatically collect high-quality instructional adaptation data by training
language models to automatically design tasks based on human-written texts.
Intuitively, human-written text helps to help the model attenuate illusions
during the generation of tasks. Unlike instruction back-translation-based
methods that directly take the given text as a response, we require the model
to generate the \textit{instruction}, \textit{input}, and \textit{output}
simultaneously to filter the noise. The results of the automated and manual
evaluation experiments demonstrate the quality of our dataset.
less
By: Hai Pham, Young Jin Kim, Subhabrata Mukherjee, David P. Woodruff, Barnabas Poczos, Hany Hassan Awadalla
Mixture-of-experts (MoE) architecture has been proven a powerful method for
diverse tasks in training deep models in many applications. However, current
MoE implementations are task agnostic, treating all tokens from different tasks
in the same manner. In this work, we instead design a novel method that
incorporates task information into MoE models at different granular levels with
shared dynamic task-based adapters. Our experiments and ana... more
Mixture-of-experts (MoE) architecture has been proven a powerful method for
diverse tasks in training deep models in many applications. However, current
MoE implementations are task agnostic, treating all tokens from different tasks
in the same manner. In this work, we instead design a novel method that
incorporates task information into MoE models at different granular levels with
shared dynamic task-based adapters. Our experiments and analysis show the
advantages of our approaches over the dense and canonical MoE models on
multi-task multilingual machine translations. With task-specific adapters, our
models can additionally generalize to new tasks efficiently.
less
By: Ha-Thanh Nguyen, Randy Goebel, Francesca Toni, Kostas Stathis, Ken Satoh
The evolution of Generative Pre-trained Transformer (GPT) models has led to
significant advancements in various natural language processing applications,
particularly in legal textual entailment. We present an analysis of GPT-3.5
(ChatGPT) and GPT-4 performances on COLIEE Task 4 dataset, a prominent
benchmark in this domain. The study encompasses data from Heisei 18 (2006) to
Reiwa 3 (2021), exploring the models' abilities to discern entail... more
The evolution of Generative Pre-trained Transformer (GPT) models has led to
significant advancements in various natural language processing applications,
particularly in legal textual entailment. We present an analysis of GPT-3.5
(ChatGPT) and GPT-4 performances on COLIEE Task 4 dataset, a prominent
benchmark in this domain. The study encompasses data from Heisei 18 (2006) to
Reiwa 3 (2021), exploring the models' abilities to discern entailment
relationships within Japanese statute law across different periods. Our
preliminary experimental results unveil intriguing insights into the models'
strengths and weaknesses in handling legal textual entailment tasks, as well as
the patterns observed in model performance. In the context of proprietary
models with undisclosed architectures and weights, black-box analysis becomes
crucial for evaluating their capabilities. We discuss the influence of training
data distribution and the implications on the models' generalizability. This
analysis serves as a foundation for future research, aiming to optimize
GPT-based models and enable their successful adoption in legal information
extraction and entailment applications.
less
By: Xi Yan, Cedric Möller, Ricardo Usbeck
Linking biomedical entities is an essential aspect in biomedical natural
language processing tasks, such as text mining and question answering. However,
a difficulty of linking the biomedical entities using current large language
models (LLM) trained on a general corpus is that biomedical entities are
scarcely distributed in texts and therefore have been rarely seen during
training by the LLM. At the same time, those LLMs are not aware of h... more
Linking biomedical entities is an essential aspect in biomedical natural
language processing tasks, such as text mining and question answering. However,
a difficulty of linking the biomedical entities using current large language
models (LLM) trained on a general corpus is that biomedical entities are
scarcely distributed in texts and therefore have been rarely seen during
training by the LLM. At the same time, those LLMs are not aware of high level
semantic connection between different biomedical entities, which are useful in
identifying similar concepts in different textual contexts. To cope with
aforementioned problems, some recent works focused on injecting knowledge graph
information into LLMs. However, former methods either ignore the relational
knowledge of the entities or lead to catastrophic forgetting. Therefore, we
propose a novel framework to pre-train the powerful generative LLM by a corpus
synthesized from a KG. In the evaluations we are unable to confirm the benefit
of including synonym, description or relational information.
less
By: Mu-Tien Kuo, Chih-Chung Hsueh, Richard Tzong-Han Tsai
While large language models have made strides in natural language processing,
their proficiency in complex reasoning tasks requiring formal language
comprehension, such as chess, remains less investigated. This paper probes the
performance of ChatGPT, a sophisticated language model by OpenAI in tackling
such complex reasoning tasks, using chess as a case study. Through robust
metrics examining both the legality and quality of moves, we asse... more
While large language models have made strides in natural language processing,
their proficiency in complex reasoning tasks requiring formal language
comprehension, such as chess, remains less investigated. This paper probes the
performance of ChatGPT, a sophisticated language model by OpenAI in tackling
such complex reasoning tasks, using chess as a case study. Through robust
metrics examining both the legality and quality of moves, we assess ChatGPT's
understanding of the chessboard, adherence to chess rules, and strategic
decision-making abilities. Our evaluation identifies limitations within
ChatGPT's attention mechanism that affect its formal language comprehension and
uncovers the model's underdeveloped self-regulation abilities. Our study also
reveals ChatGPT's propensity for a coherent strategy in its gameplay and a
noticeable uptick in decision-making assertiveness when the model is presented
with a greater volume of natural language or possesses a more lucid
understanding of the state of the chessboard. These findings contribute to the
growing exploration of language models' abilities beyond natural language
processing, providing valuable information for future research towards models
demonstrating human-like cognitive abilities.
less
By: Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Tangjie Lv, Zhipeng Hu, Wen Zhang
Large-scale vision-language pre-training has shown promising advances on
various downstream tasks and achieved significant performance in multi-modal
understanding and generation tasks. However, existing methods often perform
poorly on image-text matching tasks that require a detailed semantics
understanding of the text. Although there have been some works on this problem,
they do not sufficiently exploit the structural knowledge present in... more
Large-scale vision-language pre-training has shown promising advances on
various downstream tasks and achieved significant performance in multi-modal
understanding and generation tasks. However, existing methods often perform
poorly on image-text matching tasks that require a detailed semantics
understanding of the text. Although there have been some works on this problem,
they do not sufficiently exploit the structural knowledge present in sentences
to enhance multi-modal language representations, which leads to poor
performance. In this paper, we present an end-to-end framework Structure-CLIP,
which integrates latent detailed semantics from the text to enhance
fine-grained semantic representations. Specifically, (1) we use scene graphs in
order to pay more attention to the detailed semantic learning in the text and
fully explore structured knowledge between fine-grained semantics, and (2) we
utilize the knowledge-enhanced framework with the help of the scene graph to
make full use of representations of structured knowledge. To verify the
effectiveness of our proposed method, we pre-trained our models with the
aforementioned approach and conduct experiments on different downstream tasks.
Numerical results show that Structure-CLIP can often achieve state-of-the-art
performance on both VG-Attribution and VG-Relation datasets. Extensive
experiments show its components are effective and its predictions are
interpretable, which proves that our proposed method can enhance detailed
semantic representation well.
less
By: Abul Hasan, Mark Levene, David Weston
We investigate the potential benefit of incorporating dictionary information
into a neural network architecture for natural language processing. In
particular, we make use of this architecture to extract several concepts
related to COVID-19 from an on-line medical forum. We use a sample from the
forum to manually curate one dictionary for each concept. In addition, we use
MetaMap, which is a tool for extracting biomedical concepts, to ident... more
We investigate the potential benefit of incorporating dictionary information
into a neural network architecture for natural language processing. In
particular, we make use of this architecture to extract several concepts
related to COVID-19 from an on-line medical forum. We use a sample from the
forum to manually curate one dictionary for each concept. In addition, we use
MetaMap, which is a tool for extracting biomedical concepts, to identify a
small number of semantic concepts. For a supervised concept extraction task on
the forum data, our best model achieved a macro $F_1$ score of 90\%. A major
difficulty in medical concept extraction is obtaining labelled data from which
to build supervised models. We investigate the utility of our models to
transfer to data derived from a different source in two ways. First for
producing labels via weak learning and second to perform concept extraction.
The dataset we use in this case comprises COVID-19 related tweets and we
achieve an $F_1$ score 81\% for symptom concept extraction trained on weakly
labelled data. The utility of our dictionaries is compared with a COVID-19
symptom dictionary that was constructed directly from Twitter. Further
experiments that incorporate BERT and a COVID-19 version of BERTweet
demonstrate that the dictionaries provide a commensurate result. Our results
show that incorporating small domain dictionaries to deep learning models can
improve concept extraction tasks. Moreover, models built using dictionaries
generalize well and are transferable to different datasets on a similar task.
less
By: Li Yunxiang, Li Zihan, Zhang Kai, Dan Ruilong, Zhang You
Recent large language models (LLMs) in the general domain, such as ChatGPT,
have shown remarkable success in following instructions and producing
human-like responses. However, such language models have not been tailored to
the medical domain, resulting in poor answer accuracy and inability to give
plausible recommendations for medical diagnosis, medications, etc. To address
this issue, we collected more than 700 diseases and their correspo... more
Recent large language models (LLMs) in the general domain, such as ChatGPT,
have shown remarkable success in following instructions and producing
human-like responses. However, such language models have not been tailored to
the medical domain, resulting in poor answer accuracy and inability to give
plausible recommendations for medical diagnosis, medications, etc. To address
this issue, we collected more than 700 diseases and their corresponding
symptoms, required medical tests, and recommended medications, from which we
generated 5K doctor-patient conversations. By fine-tuning LLMs using these
tailored doctor-patient conversations, the resulting models emerge with great
potential to understand patients' needs, provide informed advice, and offer
valuable assistance in a variety of medical-related fields. The integration of
these advanced language models into healthcare can revolutionize the way
healthcare professionals and patients communicate, ultimately improving the
overall efficiency and quality of patient care and outcomes. In addition, we
made public all the source codes, datasets, and model weights to facilitate the
further development of dialogue models in the medical field. The training data,
codes, and weights of this project are available at:
https://github.com/Kent0n-Li/ChatDoctor.
less
By: Daoguang Zan, Ailun Yu, Bo Shen, Jiaxin Zhang, Taihong Chen, Bing Geng, Bei Chen, Jichuan Ji, Yafen Yao, Yongji Wang, Qianxiang Wang
When human programmers have mastered a programming language, it would be
easier when they learn a new programming language. In this report, we focus on
exploring whether programming languages can boost each other during the
instruction fine-tuning phase of code large language models. We conduct
extensive experiments of 8 popular programming languages (Python, JavaScript,
TypeScript, C, C++, Java, Go, HTML) on StarCoder. Results demonstrate ... more
When human programmers have mastered a programming language, it would be
easier when they learn a new programming language. In this report, we focus on
exploring whether programming languages can boost each other during the
instruction fine-tuning phase of code large language models. We conduct
extensive experiments of 8 popular programming languages (Python, JavaScript,
TypeScript, C, C++, Java, Go, HTML) on StarCoder. Results demonstrate that
programming languages can significantly improve each other. For example,
CodeM-Python 15B trained on Python is able to increase Java by an absolute
17.95% pass@1 on HumanEval-X. More surprisingly, we found that CodeM-HTML 7B
trained on the HTML corpus can improve Java by an absolute 15.24% pass@1. Our
training data is released at https://github.com/NL2Code/CodeM.
less
By: Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, Yemin Shi
Multi-modal large language models have garnered significant interest
recently. Though, most of the works focus on vision-language multi-modal models
providing strong capabilities in following vision-and-language instructions.
However, we claim that speech is also an important modality through which
humans interact with the world. Hence, it is crucial for a general-purpose
assistant to be able to follow multi-modal speech-and-language instru... more
Multi-modal large language models have garnered significant interest
recently. Though, most of the works focus on vision-language multi-modal models
providing strong capabilities in following vision-and-language instructions.
However, we claim that speech is also an important modality through which
humans interact with the world. Hence, it is crucial for a general-purpose
assistant to be able to follow multi-modal speech-and-language instructions. In
this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an
end-to-end trained large multi-modal speech-language model with cross-modal
conversational abilities, capable of following speech-and-language
instructions. Our early experiments show that LLaSM demonstrates a more
convenient and natural way for humans to interact with artificial intelligence.
Specifically, we also release a large Speech Instruction Following dataset
LLaSM-Audio-Instructions. Code and demo are available at
https://github.com/LinkSoul-AI/LLaSM and
https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions
dataset is available at
https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.
less
By: Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Juho Kim, Alice Oh
English datasets predominantly reflect the perspectives of certain
nationalities, which can lead to cultural biases in models and datasets. This
is particularly problematic in tasks heavily influenced by subjectivity, such
as hate speech detection. To delve into how individuals from different
countries perceive hate speech, we introduce CReHate, a cross-cultural
re-annotation of the sampled SBIC dataset. This dataset includes annotations
fr... more
English datasets predominantly reflect the perspectives of certain
nationalities, which can lead to cultural biases in models and datasets. This
is particularly problematic in tasks heavily influenced by subjectivity, such
as hate speech detection. To delve into how individuals from different
countries perceive hate speech, we introduce CReHate, a cross-cultural
re-annotation of the sampled SBIC dataset. This dataset includes annotations
from five distinct countries: Australia, Singapore, South Africa, the United
Kingdom, and the United States. Our thorough statistical analysis highlights
significant differences based on nationality, with only 59.4% of the samples
achieving consensus among all countries. We also introduce a culturally
sensitive hate speech classifier via transfer learning, adept at capturing
perspectives of different nationalities. These findings underscore the need to
re-evaluate certain aspects of NLP research, especially with regard to the
nuanced nature of hate speech in the English language.
less
By: John Mendonça, Patrícia Pereira, João Paulo Carvalho, Alon Lavie, Isabel Trancoso
Despite significant research effort in the development of automatic dialogue
evaluation metrics, little thought is given to evaluating dialogues other than
in English. At the same time, ensuring metrics are invariant to semantically
similar responses is also an overlooked topic. In order to achieve the desired
properties of robustness and multilinguality for dialogue evaluation metrics,
we propose a novel framework that takes advantage of t... more
Despite significant research effort in the development of automatic dialogue
evaluation metrics, little thought is given to evaluating dialogues other than
in English. At the same time, ensuring metrics are invariant to semantically
similar responses is also an overlooked topic. In order to achieve the desired
properties of robustness and multilinguality for dialogue evaluation metrics,
we propose a novel framework that takes advantage of the strengths of current
evaluation models with the newly-established paradigm of prompting Large
Language Models (LLMs). Empirical results show our framework achieves state of
the art results in terms of mean Spearman correlation scores across several
benchmarks and ranks first place on both the Robust and Multilingual tasks of
the DSTC11 Track 4 "Automatic Evaluation Metrics for Open-Domain Dialogue
Systems", proving the evaluation capabilities of prompted LLMs.
less
By: Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang
In recent years, there have been remarkable advancements in the performance
of Transformer-based Large Language Models (LLMs) across various domains. As
these LLMs are deployed for increasingly complex tasks, they often face the
needs to conduct longer reasoning processes or understanding larger contexts.
In these situations, the length generalization failure of LLMs on long
sequences become more prominent. Most pre-training schemes truncat... more
In recent years, there have been remarkable advancements in the performance
of Transformer-based Large Language Models (LLMs) across various domains. As
these LLMs are deployed for increasingly complex tasks, they often face the
needs to conduct longer reasoning processes or understanding larger contexts.
In these situations, the length generalization failure of LLMs on long
sequences become more prominent. Most pre-training schemes truncate training
sequences to a fixed length (such as 2048 for LLaMa). LLMs often struggle to
generate fluent texts, let alone carry out downstream tasks, after longer
contexts, even with relative positional encoding which is designed to cope with
this problem. Common solutions such as finetuning on longer corpora often
involves daunting hardware and time costs and requires careful training process
design. To more efficiently leverage the generation capacity of existing LLMs,
we theoretically and empirically investigate the main out-of-distribution (OOD)
factors contributing to this problem. Inspired by this diagnosis, we propose a
simple yet effective solution for on-the-fly length generalization,
LM-Infinite, which involves only a $\Lambda$-shaped attention mask and a
distance limit while requiring no parameter updates or learning. We find it
applicable to a variety of LLMs using relative-position encoding methods.
LM-Infinite is computational efficient with $O(n)$ time and space, and
demonstrates consistent fluency and generation quality to as long as 32k tokens
on ArXiv and OpenWebText2 datasets, with 2.72x decoding speedup. On downstream
task such as passkey retrieval, it continues to work on inputs much longer than
training lengths where vanilla models fail immediately.
less
By: Patrick Eickhoff, Matthias Möller, Theresa Pekarek Rosin, Johannes Twiefel, Stefan Wermter
In recent research, in the domain of speech processing, large End-to-End
(E2E) systems for Automatic Speech Recognition (ASR) have reported
state-of-the-art performance on various benchmarks. These systems intrinsically
learn how to handle and remove noise conditions from speech. Previous research
has shown, that it is possible to extract the denoising capabilities of these
models into a preprocessor network, which can be used as a frontend... more
In recent research, in the domain of speech processing, large End-to-End
(E2E) systems for Automatic Speech Recognition (ASR) have reported
state-of-the-art performance on various benchmarks. These systems intrinsically
learn how to handle and remove noise conditions from speech. Previous research
has shown, that it is possible to extract the denoising capabilities of these
models into a preprocessor network, which can be used as a frontend for
downstream ASR models. However, the proposed methods were limited to specific
fully convolutional architectures. In this work, we propose a novel method to
extract the denoising capabilities, that can be applied to any encoder-decoder
architecture. We propose the Cleancoder preprocessor architecture that extracts
hidden activations from the Conformer ASR model and feeds them to a decoder to
predict denoised spectrograms. We train our pre-processor on the Noisy Speech
Database (NSD) to reconstruct denoised spectrograms from noisy inputs. Then, we
evaluate our model as a frontend to a pretrained Conformer ASR model as well as
a frontend to train smaller Conformer ASR models from scratch. We show that the
Cleancoder is able to filter noise from speech and that it improves the total
Word Error Rate (WER) of the downstream model in noisy conditions for both
applications.
less
By: Wei Du, Laksh Advani, Yashmeet Gambhir, Daniel J Perry, Prashant Shiralkar, Zhengzheng Xing, Aaron Colak
Large language models (LLMs) have demonstrated significant capability to
generalize across a large number of NLP tasks. For industry applications, it is
imperative to assess the performance of the LLM on unlabeled production data
from time to time to validate for a real-world setting. Human labeling to
assess model error requires considerable expense and time delay. Here we
demonstrate that ensemble disagreement scores work well as a proxy ... more
Large language models (LLMs) have demonstrated significant capability to
generalize across a large number of NLP tasks. For industry applications, it is
imperative to assess the performance of the LLM on unlabeled production data
from time to time to validate for a real-world setting. Human labeling to
assess model error requires considerable expense and time delay. Here we
demonstrate that ensemble disagreement scores work well as a proxy for human
labeling for language models in zero-shot, few-shot, and fine-tuned settings,
per our evaluation on keyphrase extraction (KPE) task. We measure fidelity of
the results by comparing to true error measured from human labeled ground
truth. We contrast with the alternative of using another LLM as a source of
machine labels, or silver labels. Results across various languages and domains
show disagreement scores provide a better estimation of model performance with
mean average error (MAE) as low as 0.4% and on average 13.8% better than using
silver labels.
less
By: Olga Kellert, Mahmud Uz Zaman, Nicholas Hill Matlis, Carlos Gómez-Rodríguez
This paper summarizes the results of experimenting with Universal
Dependencies (UD) adaptation of an Unsupervised, Compositional and Recursive
(UCR) rule-based approach for Sentiment Analysis (SA) submitted to the Shared
Task at Rest-Mex 2023 (Team Olga/LyS-SALSA) (within the IberLEF 2023
conference). By using basic syntactic rules such as rules of modification and
negation applied on words from sentiment dictionaries, our approach exploits... more
This paper summarizes the results of experimenting with Universal
Dependencies (UD) adaptation of an Unsupervised, Compositional and Recursive
(UCR) rule-based approach for Sentiment Analysis (SA) submitted to the Shared
Task at Rest-Mex 2023 (Team Olga/LyS-SALSA) (within the IberLEF 2023
conference). By using basic syntactic rules such as rules of modification and
negation applied on words from sentiment dictionaries, our approach exploits
some advantages of an unsupervised method for SA: (1) interpretability and
explainability of SA, (2) robustness across datasets, languages and domains and
(3) usability by non-experts in NLP. We compare our approach with other
unsupervised approaches of SA that in contrast to our UCR rule-based approach
use simple heuristic rules to deal with negation and modification. Our results
show a considerable improvement over these approaches. We discuss future
improvements of our results by using modality features as another shifting rule
of polarity and word disambiguation techniques to identify the right sentiment
words.
less
By: Lucas Cassiel Jacaruso
Topic modeling and text mining are subsets of Natural Language Processing
with relevance for conducting meta-analysis (MA) and systematic review (SR).
For evidence synthesis, the above NLP methods are conventionally used for
topic-specific literature searches or extracting values from reports to
automate essential phases of SR and MA. Instead, this work proposes a
comparative topic modeling approach to analyze reports of contradictory resul... more
Topic modeling and text mining are subsets of Natural Language Processing
with relevance for conducting meta-analysis (MA) and systematic review (SR).
For evidence synthesis, the above NLP methods are conventionally used for
topic-specific literature searches or extracting values from reports to
automate essential phases of SR and MA. Instead, this work proposes a
comparative topic modeling approach to analyze reports of contradictory results
on the same general research question. Specifically, the objective is to find
topics exhibiting distinct associations with significant results for an outcome
of interest by ranking them according to their proportional occurrence and
consistency of distribution across reports of significant results. The proposed
method was tested on broad-scope studies addressing whether supplemental
nutritional compounds significantly benefit macular degeneration (MD). Eight
compounds were identified as having a particular association with reports of
significant results for benefitting MD. Six of these were further supported in
terms of effectiveness upon conducting a follow-up literature search for
validation (omega-3 fatty acids, copper, zeaxanthin, lutein, zinc, and
nitrates). The two not supported by the follow-up literature search (niacin and
molybdenum) also had the lowest scores under the proposed methods ranking
system, suggesting that the proposed method's score for a given topic is a
viable proxy for its degree of association with the outcome of interest. These
results underpin the proposed methods potential to add specificity in
understanding effects from broad-scope reports, elucidate topics of interest
for future research, and guide evidence synthesis in a systematic and scalable
way.
less
By: D. Emre Taşar, Şükrü Ozan, Umut Özdil, M. Fatih Akca, Oğuzhan Ölmez, Semih Gülüm, Seçilay Kutal, Ceren Belhan
The problem of categorizing short speech sentences according to their
semantic features with high accuracy is a subject studied in natural language
processing. In this study, a data set created with samples classified in 46
different categories was used. Examples consist of sentences taken from chat
conversations between a company's customer representatives and the company's
website visitors. The primary purpose is to automatically tag ques... more
The problem of categorizing short speech sentences according to their
semantic features with high accuracy is a subject studied in natural language
processing. In this study, a data set created with samples classified in 46
different categories was used. Examples consist of sentences taken from chat
conversations between a company's customer representatives and the company's
website visitors. The primary purpose is to automatically tag questions and
requests from visitors in the most accurate way for 46 predetermined categories
for use in a chat application to generate meaningful answers to the questions
asked by the website visitors. For this, different BERT models and one GPT-2
model, pre-trained in Turkish, were preferred. The classification performances
of the relevant models were analyzed in detail and reported accordingly.
less
By: Yajing Wang, Zongwei Luo
Large Language Models (LLMs) have made significant strides in both scientific
research and practical applications. Existing studies have demonstrated the
state-of-the-art (SOTA) performance of LLMs in various natural language
processing tasks. However, the question of how to further enhance LLMs'
performance in specific task using prompting strategies remains a pivotal
concern. This paper explores the enhancement of LLMs' performance in sen... more
Large Language Models (LLMs) have made significant strides in both scientific
research and practical applications. Existing studies have demonstrated the
state-of-the-art (SOTA) performance of LLMs in various natural language
processing tasks. However, the question of how to further enhance LLMs'
performance in specific task using prompting strategies remains a pivotal
concern. This paper explores the enhancement of LLMs' performance in sentiment
analysis through the application of prompting strategies. We formulate the
process of prompting for sentiment analysis tasks and introduce two novel
strategies tailored for sentiment analysis: RolePlaying (RP) prompting and
Chain-of-thought (CoT) prompting. Specifically, we also propose the RP-CoT
prompting strategy which is a combination of RP prompting and CoT prompting. We
conduct comparative experiments on three distinct domain datasets to evaluate
the effectiveness of the proposed sentiment analysis strategies. The results
demonstrate that the adoption of the proposed prompting strategies leads to a
increasing enhancement in sentiment analysis accuracy. Further, the CoT
prompting strategy exhibits a notable impact on implicit sentiment analysis,
with the RP-CoT prompting strategy delivering the most superior performance
among all strategies.
less
By: David Oniani, Jordan Hilsman, Hang Dong, Fengyi Gao, Shiven Verma, Yanshan Wang
The emergence of generative Large Language Models (LLMs) emphasizes the need
for accurate and efficient prompting approaches. LLMs are often applied in
Few-Shot Learning (FSL) contexts, where tasks are executed with minimal
training data. FSL has become popular in many Artificial Intelligence (AI)
subdomains, including AI for health. Rare diseases, affecting a small fraction
of the population, inherently require FSL techniques due to limite... more
The emergence of generative Large Language Models (LLMs) emphasizes the need
for accurate and efficient prompting approaches. LLMs are often applied in
Few-Shot Learning (FSL) contexts, where tasks are executed with minimal
training data. FSL has become popular in many Artificial Intelligence (AI)
subdomains, including AI for health. Rare diseases, affecting a small fraction
of the population, inherently require FSL techniques due to limited data
availability, though manual data collection and annotation is costly and
time-consuming. In this paper, we propose Models-Vote Prompting (MVP), a
flexible prompting approach for improving the performance of LLM queries in FSL
settings. MVP works by prompting numerous LLMs to perform the same tasks and
then conducting a majority vote on the resulting outputs. This method achieves
improved results to any one model in the ensemble on one-shot rare disease
identification and classification tasks. We also release a novel rare disease
dataset for FSL, available to those who agreed to the MIMIC-IV Data Use
Agreement (DUA). Furthermore, in using MVP, each model is prompted multiple
times, substantially increasing the time needed for manual annotation, and to
address this, we assess the feasibility of using JSON for automating generative
LLM evaluation.
less
By: Aydar Bulatov, Yuri Kuratov, Mikhail S. Burtsev
This technical report presents the application of a recurrent memory to
extend the context length of BERT, one of the most effective Transformer-based
models in natural language processing. By leveraging the Recurrent Memory
Transformer architecture, we have successfully increased the model's effective
context length to an unprecedented two million tokens, while maintaining high
memory retrieval accuracy. Our method allows for the storage a... more
This technical report presents the application of a recurrent memory to
extend the context length of BERT, one of the most effective Transformer-based
models in natural language processing. By leveraging the Recurrent Memory
Transformer architecture, we have successfully increased the model's effective
context length to an unprecedented two million tokens, while maintaining high
memory retrieval accuracy. Our method allows for the storage and processing of
both local and global information and enables information flow between segments
of the input sequence through the use of recurrence. Our experiments
demonstrate the effectiveness of our approach, which holds significant
potential to enhance long-term dependency handling in natural language
understanding and generation tasks as well as enable large-scale context
processing for memory-intensive applications.
less
Text-to-OverpassQL: A Natural Language Interface for Complex Geodata Querying of OpenStreetMap
0upvotes
By: Michael Staniek, Raphael Schumann, Maike Züfle, Stefan Riezler
We present Text-to-OverpassQL, a task designed to facilitate a natural
language interface for querying geodata from OpenStreetMap (OSM). The Overpass
Query Language (OverpassQL) allows users to formulate complex database queries
and is widely adopted in the OSM ecosystem. Generating Overpass queries from
natural language input serves multiple use-cases. It enables novice users to
utilize OverpassQL without prior knowledge, assists experienc... more
We present Text-to-OverpassQL, a task designed to facilitate a natural
language interface for querying geodata from OpenStreetMap (OSM). The Overpass
Query Language (OverpassQL) allows users to formulate complex database queries
and is widely adopted in the OSM ecosystem. Generating Overpass queries from
natural language input serves multiple use-cases. It enables novice users to
utilize OverpassQL without prior knowledge, assists experienced users with
crafting advanced queries, and enables tool-augmented large language models to
access information stored in the OSM database. In order to assess the
performance of current sequence generation models on this task, we propose
OverpassNL, a dataset of 8,352 queries with corresponding natural language
inputs. We further introduce task specific evaluation metrics and ground the
evaluation of the Text-to-OverpassQL task by executing the queries against the
OSM database. We establish strong baselines by finetuning sequence-to-sequence
models and adapting large language models with in-context examples. The
detailed evaluation reveals strengths and weaknesses of the considered learning
strategies, laying the foundations for further research into the
Text-to-OverpassQL task.
less
By: Loukas Ilias, Ioannis Michail Kazelidis, Dimitris Askounis
Although not all bots are malicious, the vast majority of them are
responsible for spreading misinformation and manipulating the public opinion
about several issues, i.e., elections and many more. Therefore, the early
detection of social spambots is crucial. Although there have been proposed
methods for detecting bots in social media, there are still substantial
limitations. For instance, existing research initiatives still extract a large
... more
Although not all bots are malicious, the vast majority of them are
responsible for spreading misinformation and manipulating the public opinion
about several issues, i.e., elections and many more. Therefore, the early
detection of social spambots is crucial. Although there have been proposed
methods for detecting bots in social media, there are still substantial
limitations. For instance, existing research initiatives still extract a large
number of features and train traditional machine learning algorithms or use
GloVe embeddings and train LSTMs. However, feature extraction is a tedious
procedure demanding domain expertise. Also, language models based on
transformers have been proved to be better than LSTMs. Other approaches create
large graphs and train graph neural networks requiring in this way many hours
for training and access to computational resources. To tackle these
limitations, this is the first study employing only the user description field
and images of three channels denoting the type and content of tweets posted by
the users. Firstly, we create digital DNA sequences, transform them to 3d
images, and apply pretrained models of the vision domain, including
EfficientNet, AlexNet, VGG16, etc. Next, we propose a multimodal approach,
where we use TwHIN-BERT for getting the textual representation of the user
description field and employ VGG16 for acquiring the visual representation for
the image modality. We propose three different fusion methods, namely
concatenation, gated multimodal unit, and crossmodal attention, for fusing the
different modalities and compare their performances. Extensive experiments
conducted on the Cresci '17 dataset demonstrate valuable advantages of our
introduced approaches over state-of-the-art ones reaching Accuracy up to
99.98%.
less
By: Etienne Labbé IRIT-SAMoVA, Thomas Pellegrini IRIT-SAMoVA, Julien Pinquier IRIT-SAMoVA
Automated Audio Captioning (AAC) aims to develop systems capable of
describing an audio recording using a textual sentence. In contrast, Audio-Text
Retrieval (ATR) systems seek to find the best matching audio recording(s) for a
given textual query (Text-to-Audio) or vice versa (Audio-to-Text). These tasks
require different types of systems: AAC employs a sequence-to-sequence model,
while ATR utilizes a ranking model that compares audio and ... more
Automated Audio Captioning (AAC) aims to develop systems capable of
describing an audio recording using a textual sentence. In contrast, Audio-Text
Retrieval (ATR) systems seek to find the best matching audio recording(s) for a
given textual query (Text-to-Audio) or vice versa (Audio-to-Text). These tasks
require different types of systems: AAC employs a sequence-to-sequence model,
while ATR utilizes a ranking model that compares audio and text representations
within a shared projection subspace. However, this work investigates the
relationship between AAC and ATR by exploring the ATR capabilities of an
unmodified AAC system, without fine-tuning for the new task. Our AAC system
consists of an audio encoder (ConvNeXt-Tiny) trained on AudioSet for audio
tagging, and a transformer decoder responsible for generating sentences. For
AAC, it achieves a high SPIDEr-FL score of 0.298 on Clotho and 0.472 on
AudioCaps on average. For ATR, we propose using the standard Cross-Entropy loss
values obtained for any audio/caption pair. Experimental results on the Clotho
and AudioCaps datasets demonstrate decent recall values using this simple
approach. For instance, we obtained a Text-to-Audio R@1 value of 0.382 for
Au-dioCaps, which is above the current state-of-the-art method without external
data. Interestingly, we observe that normalizing the loss values was necessary
for Audio-to-Text retrieval.
less
By: Hai-Long Nguyen, Dieu-Quynh Nguyen, Hoang-Trung Nguyen, Thu-Trang Pham, Huu-Dong Nguyen, Thach-Anh Nguyen, Thi-Hai-Yen Vuong, Ha-Thanh Nguyen
In recent years, natural language processing has gained significant
popularity in various sectors, including the legal domain. This paper presents
NeCo Team's solutions to the Vietnamese text processing tasks provided in the
Automated Legal Question Answering Competition 2023 (ALQAC 2023), focusing on
legal domain knowledge acquisition for low-resource languages through data
enrichment. Our methods for the legal document retrieval task empl... more
In recent years, natural language processing has gained significant
popularity in various sectors, including the legal domain. This paper presents
NeCo Team's solutions to the Vietnamese text processing tasks provided in the
Automated Legal Question Answering Competition 2023 (ALQAC 2023), focusing on
legal domain knowledge acquisition for low-resource languages through data
enrichment. Our methods for the legal document retrieval task employ a
combination of similarity ranking and deep learning models, while for the
second task, which requires extracting an answer from a relevant legal article
in response to a question, we propose a range of adaptive techniques to handle
different question types. Our approaches achieve outstanding results on both
tasks of the competition, demonstrating the potential benefits and
effectiveness of question answering systems in the legal field, particularly
for low-resource languages.
less
By: Xiangci Li, Yi-Hui Lee, Jessica Ouyang
Automatic related work generation must ground their outputs to the content of
the cited papers to avoid non-factual hallucinations, but due to the length of
scientific documents, existing abstractive approaches have conditioned only on
the cited paper \textit{abstracts}. We demonstrate that the abstract is not
always the most appropriate input for citation generation and that models
trained in this way learn to hallucinate. We propose to co... more
Automatic related work generation must ground their outputs to the content of
the cited papers to avoid non-factual hallucinations, but due to the length of
scientific documents, existing abstractive approaches have conditioned only on
the cited paper \textit{abstracts}. We demonstrate that the abstract is not
always the most appropriate input for citation generation and that models
trained in this way learn to hallucinate. We propose to condition instead on
the \textit{cited text span} (CTS) as an alternative to the abstract. Because
manual CTS annotation is extremely time- and labor-intensive, we experiment
with automatic, ROUGE-based labeling of candidate CTS sentences, achieving
sufficiently strong performance to substitute for expensive human annotations,
and we propose a human-in-the-loop, keyword-based CTS retrieval approach that
makes generating citation texts grounded in the full text of cited papers both
promising and practical.
less
By: Sanaz Saki Norouzi, Mohammad Saeid Mahdavinejad, Pascal Hitzler
This study evaluates the applicability and efficiency of ChatGPT for ontology
alignment using a naive approach. ChatGPT's output is compared to the results
of the Ontology Alignment Evaluation Initiative 2022 campaign using conference
track ontologies. This comparison is intended to provide insights into the
capabilities of a conversational large language model when used in a naive way
for ontology matching, and to investigate the potential... more
This study evaluates the applicability and efficiency of ChatGPT for ontology
alignment using a naive approach. ChatGPT's output is compared to the results
of the Ontology Alignment Evaluation Initiative 2022 campaign using conference
track ontologies. This comparison is intended to provide insights into the
capabilities of a conversational large language model when used in a naive way
for ontology matching, and to investigate the potential advantages and
disadvantages of this approach.
less
By: Lingbing Guo, Zhuo Chen, Jiaoyan Chen, Huajun Chen
Recent embedding-based methods have achieved great successes on exploiting
entity alignment from knowledge graph (KG) embeddings of multiple modals. In
this paper, we study embedding-based entity alignment (EEA) from a perspective
of generative models. We show that EEA is a special problem where the main
objective is analogous to that in a typical generative model, based on which we
theoretically prove the effectiveness of the recently deve... more
Recent embedding-based methods have achieved great successes on exploiting
entity alignment from knowledge graph (KG) embeddings of multiple modals. In
this paper, we study embedding-based entity alignment (EEA) from a perspective
of generative models. We show that EEA is a special problem where the main
objective is analogous to that in a typical generative model, based on which we
theoretically prove the effectiveness of the recently developed generative
adversarial network (GAN)-based EEA methods. We then reveal that their
incomplete objective limits the capacity on both entity alignment and entity
synthesis (i.e., generating new entities). We mitigate this problem by
introducing a generative EEA (abbr., GEEA) framework with the proposed mutual
variational autoencoder (M-VAE) as the generative model. M-VAE can convert an
entity from one KG to another and generate new entities from random noise
vectors. We demonstrate the power of GEEA with theoretical analysis and
empirical experiments on both entity alignment and entity synthesis tasks.
less
By: Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, Quanquan Gu
The recent surge of generative AI has been fueled by the generative power of
diffusion probabilistic models and the scalable capabilities of large language
models. Despite their potential, it remains elusive whether diffusion language
models can solve general language tasks comparable to their autoregressive
counterparts. This paper demonstrates that scaling diffusion models w.r.t.
data, sizes, and tasks can effectively make them strong lan... more
The recent surge of generative AI has been fueled by the generative power of
diffusion probabilistic models and the scalable capabilities of large language
models. Despite their potential, it remains elusive whether diffusion language
models can solve general language tasks comparable to their autoregressive
counterparts. This paper demonstrates that scaling diffusion models w.r.t.
data, sizes, and tasks can effectively make them strong language learners. We
build competent diffusion language models at scale by first acquiring knowledge
from massive data via masked language modeling pretraining thanks to their
intrinsic connections. We then reprogram pretrained masked language models into
diffusion language models via diffusive adaptation, wherein task-specific
finetuning and instruction finetuning are explored to unlock their versatility
in solving general language tasks. Experiments show that scaling diffusion
language models consistently improves performance across downstream language
tasks. We further discover that instruction finetuning can elicit zero-shot and
few-shot in-context learning abilities that help tackle many unseen tasks by
following natural language instructions, and show promise in advanced and
challenging abilities such as reasoning
less
By: Ronald Seoh, Haw-Shiuan Chang, Andrew McCallum
Many useful tasks on scientific documents, such as topic classification and
citation prediction, involve corpora that span multiple scientific domains.
Typically, such tasks are accomplished by representing the text with a vector
embedding obtained from a Transformer's single CLS token. In this paper, we
argue that using multiple CLS tokens could make a Transformer better specialize
to multiple scientific domains. We present Multi2SPE: it e... more
Many useful tasks on scientific documents, such as topic classification and
citation prediction, involve corpora that span multiple scientific domains.
Typically, such tasks are accomplished by representing the text with a vector
embedding obtained from a Transformer's single CLS token. In this paper, we
argue that using multiple CLS tokens could make a Transformer better specialize
to multiple scientific domains. We present Multi2SPE: it encourages each of
multiple CLS tokens to learn diverse ways of aggregating token embeddings, then
sums them up together to create a single vector representation. We also propose
our new multi-domain benchmark, Multi-SciDocs, to test scientific paper vector
encoders under multi-domain settings. We show that Multi2SPE reduces error by
up to 25 percent in multi-domain citation prediction, while requiring only a
negligible amount of computation in addition to one BERT forward pass.
less
By: Shutong Feng, Nurul Lubis, Benjamin Ruppik, Christian Geishauser, Michael Heck, Hsien-chin Lin, Carel van Niekerk, Renato Vukovic, Milica Gašić
Emotion recognition in conversations (ERC) is a crucial task for building
human-like conversational agents. While substantial efforts have been devoted
to ERC for chit-chat dialogues, the task-oriented counterpart is largely left
unattended. Directly applying chit-chat ERC models to task-oriented dialogues
(ToDs) results in suboptimal performance as these models overlook key features
such as the correlation between emotions and task complet... more
Emotion recognition in conversations (ERC) is a crucial task for building
human-like conversational agents. While substantial efforts have been devoted
to ERC for chit-chat dialogues, the task-oriented counterpart is largely left
unattended. Directly applying chit-chat ERC models to task-oriented dialogues
(ToDs) results in suboptimal performance as these models overlook key features
such as the correlation between emotions and task completion in ToDs. In this
paper, we propose a framework that turns a chit-chat ERC model into a
task-oriented one, addressing three critical aspects: data, features and
objective. First, we devise two ways of augmenting rare emotions to improve ERC
performance. Second, we use dialogue states as auxiliary features to
incorporate key information from the goal of the user. Lastly, we leverage a
multi-aspect emotion definition in ToDs to devise a multi-task learning
objective and a novel emotion-distance weighted loss function. Our framework
yields significant improvements for a range of chit-chat ERC models on EmoWOZ,
a large-scale dataset for user emotion in ToDs. We further investigate the
generalisability of the best resulting model to predict user satisfaction in
different ToD datasets. A comparison with supervised baselines shows a strong
zero-shot capability, highlighting the potential usage of our framework in
wider scenarios.
less
By: Jiuhai Chen, Jonas Mueller
We introduce BSDetector, a method for detecting bad and speculative answers
from a pretrained Large Language Model by estimating a numeric confidence score
for any output it generated. Our uncertainty quantification technique works for
any LLM accessible only via a black-box API, and combines intrinsic and
extrinsic assessments of confidence into a single trustworthiness estimate for
any LLM response to a given prompt. Our method is extreme... more
We introduce BSDetector, a method for detecting bad and speculative answers
from a pretrained Large Language Model by estimating a numeric confidence score
for any output it generated. Our uncertainty quantification technique works for
any LLM accessible only via a black-box API, and combines intrinsic and
extrinsic assessments of confidence into a single trustworthiness estimate for
any LLM response to a given prompt. Our method is extremely general and can
applied to all of the best LLMs available today (whose training data remains
unknown). By expending a bit of extra computation, users of any LLM API can now
get the same response as they would ordinarily, as well as a confidence
estimate that caution when not to trust this response. Experiments on both
closed and open-form Question-Answer benchmarks reveal that BSDetector more
accurately identifies incorrect LLM responses than alternative uncertainty
estimation procedures (for both GPT-3 and ChatGPT). By sampling multiple
responses from the LLM and considering the one with the highest confidence
score, we can additionally obtain more accurate responses from the same LLM,
without any extra training steps.
less
Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts
0upvotes
By: Zhi-Yi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen, Wei-Chen Chiu
Text-to-image diffusion models, e.g. Stable Diffusion (SD), lately have shown
remarkable ability in high-quality content generation, and become one of the
representatives for the recent wave of transformative AI. Nevertheless, such
advance comes with an intensifying concern about the misuse of this generative
technology, especially for producing copyrighted or NSFW (i.e. not safe for
work) images. Although efforts have been made to filter i... more
Text-to-image diffusion models, e.g. Stable Diffusion (SD), lately have shown
remarkable ability in high-quality content generation, and become one of the
representatives for the recent wave of transformative AI. Nevertheless, such
advance comes with an intensifying concern about the misuse of this generative
technology, especially for producing copyrighted or NSFW (i.e. not safe for
work) images. Although efforts have been made to filter inappropriate
images/prompts or remove undesirable concepts/styles via model fine-tuning, the
reliability of these safety mechanisms against diversified problematic prompts
remains largely unexplored. In this work, we propose Prompting4Debugging (P4D)
as a debugging and red-teaming tool that automatically finds problematic
prompts for diffusion models to test the reliability of a deployed safety
mechanism. We demonstrate the efficacy of our P4D tool in uncovering new
vulnerabilities of SD models with safety mechanisms. Particularly, our result
shows that around half of prompts in existing safe prompting benchmarks which
were originally considered "safe" can actually be manipulated to bypass many
deployed safety mechanisms, including concept removal, negative prompt, and
safety guidance. Our findings suggest that, without comprehensive testing, the
evaluations on limited safe prompting benchmarks can lead to a false sense of
safety for text-to-image models.
less
By: Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, Wenhu Chen
We introduce MAmmoTH, a series of open-source large language models (LLMs)
specifically tailored for general math problem-solving. The MAmmoTH models are
trained on MathInstruct, our meticulously curated instruction tuning dataset.
MathInstruct is compiled from 13 math datasets with intermediate rationales,
six of which have rationales newly curated by us. It presents a unique hybrid
of chain-of-thought (CoT) and program-of-thought (PoT) ra... more
We introduce MAmmoTH, a series of open-source large language models (LLMs)
specifically tailored for general math problem-solving. The MAmmoTH models are
trained on MathInstruct, our meticulously curated instruction tuning dataset.
MathInstruct is compiled from 13 math datasets with intermediate rationales,
six of which have rationales newly curated by us. It presents a unique hybrid
of chain-of-thought (CoT) and program-of-thought (PoT) rationales, and also
ensures extensive coverage of diverse fields in math. The hybrid of CoT and PoT
not only unleashes the potential of tool use but also allows different thought
processes for different math problems. As a result, the MAmmoTH series
substantially outperform existing open-source models on nine mathematical
reasoning datasets across all scales with an average accuracy gain between 13%
and 29%. Remarkably, our MAmmoTH-7B model reaches 35% on MATH (a
competition-level dataset), which exceeds the best open-source 7B model
(WizardMath) by 25%, and the MAmmoTH-34B model achieves 46% accuracy on MATH,
even surpassing GPT-4's CoT result. Our work underscores the importance of
diverse problem coverage and the use of hybrid rationales in developing
superior math generalist models.
less
By: Zachary Horvitz, Ajay Patel, Chris Callison-Burch, Zhou Yu, Kathleen McKeown
Textual style transfer is the task of transforming stylistic properties of
text while preserving meaning. Target "styles" can be defined in numerous ways,
ranging from single attributes (e.g, formality) to authorship (e.g,
Shakespeare). Previous unsupervised style-transfer approaches generally rely on
significant amounts of labeled data for only a fixed set of styles or require
large language models. In contrast, we introduce a novel diffus... more
Textual style transfer is the task of transforming stylistic properties of
text while preserving meaning. Target "styles" can be defined in numerous ways,
ranging from single attributes (e.g, formality) to authorship (e.g,
Shakespeare). Previous unsupervised style-transfer approaches generally rely on
significant amounts of labeled data for only a fixed set of styles or require
large language models. In contrast, we introduce a novel diffusion-based
framework for general-purpose style transfer that can be flexibly adapted to
arbitrary target styles at inference time. Our parameter-efficient approach,
ParaGuide, leverages paraphrase-conditioned diffusion models alongside
gradient-based guidance from both off-the-shelf classifiers and strong existing
style embedders to transform the style of text while preserving semantic
information. We validate the method on the Enron Email Corpus, with both human
and automatic evaluations, and find that it outperforms strong baselines on
formality, sentiment, and even authorship style transfer.
less
Large Language Models
0upvotes
By: Michael R. Douglas
Artificial intelligence is making spectacular progress, and one of the best
examples is the development of large language models (LLMs) such as OpenAI's
GPT series. In these lectures, written for readers with a background in
mathematics or physics, we give a brief history and survey of the state of the
art, and describe the underlying transformer architecture in detail. We then
explore some current ideas on how LLMs work and how models trai... more
Artificial intelligence is making spectacular progress, and one of the best
examples is the development of large language models (LLMs) such as OpenAI's
GPT series. In these lectures, written for readers with a background in
mathematics or physics, we give a brief history and survey of the state of the
art, and describe the underlying transformer architecture in detail. We then
explore some current ideas on how LLMs work and how models trained to predict
the next word in a text are able to perform other tasks displaying
intelligence.
less
AKEM: Aligning Knowledge Base to Queries with Ensemble Model for Entity Recognition and Linking
0upvotes
By: Di Lu, Zhongping Liang, Caixia Yuan, Xiaojie Wang
This paper presents a novel approach to address the Entity Recognition and
Linking Challenge at NLPCC 2015. The task involves extracting named entity
mentions from short search queries and linking them to entities within a
reference Chinese knowledge base. To tackle this problem, we first expand the
existing knowledge base and utilize external knowledge to identify candidate
entities, thereby improving the recall rate. Next, we extract feat... more
This paper presents a novel approach to address the Entity Recognition and
Linking Challenge at NLPCC 2015. The task involves extracting named entity
mentions from short search queries and linking them to entities within a
reference Chinese knowledge base. To tackle this problem, we first expand the
existing knowledge base and utilize external knowledge to identify candidate
entities, thereby improving the recall rate. Next, we extract features from the
candidate entities and utilize Support Vector Regression and Multiple Additive
Regression Tree as scoring functions to filter the results. Additionally, we
apply rules to further refine the results and enhance precision. Our method is
computationally efficient and achieves an F1 score of 0.535.
less
By: Neel Jitesh Bhate, Ansh Mittal, Zhe He, Xiao Luo
Demographics, Social determinants of health, and family history documented in
the unstructured text within the electronic health records are increasingly
being studied to understand how this information can be utilized with the
structured data to improve healthcare outcomes. After the GPT models were
released, many studies have applied GPT models to extract this information from
the narrative clinical notes. Different from the existing work... more
Demographics, Social determinants of health, and family history documented in
the unstructured text within the electronic health records are increasingly
being studied to understand how this information can be utilized with the
structured data to improve healthcare outcomes. After the GPT models were
released, many studies have applied GPT models to extract this information from
the narrative clinical notes. Different from the existing work, our research
focuses on investigating the zero-shot learning on extracting this information
together by providing minimum information to the GPT model. We utilize
de-identified real-world clinical notes annotated for demographics, various
social determinants, and family history information. Given that the GPT model
might provide text different from the text in the original data, we explore two
sets of evaluation metrics, including the traditional NER evaluation metrics
and semantic similarity evaluation metrics, to completely understand the
performance. Our results show that the GPT-3.5 method achieved an average of
0.975 F1 on demographics extraction, 0.615 F1 on social determinants
extraction, and 0.722 F1 on family history extraction. We believe these results
can be further improved through model fine-tuning or few-shots learning.
Through the case studies, we also identified the limitations of the GPT models,
which need to be addressed in future research.
less
GADePo: Graph-Assisted Declarative Pooling Transformers for Document-Level Relation Extraction
0upvotes
By: Andrei C. Coman, Christos Theodoropoulos, Marie-Francine Moens, James Henderson
Document-level relation extraction aims to identify relationships between
entities within a document. Current methods rely on text-based encoders and
employ various hand-coded pooling heuristics to aggregate information from
entity mentions and associated contexts. In this paper, we replace these rigid
pooling functions with explicit graph relations by leveraging the intrinsic
graph processing capabilities of the Transformer model. We propo... more
Document-level relation extraction aims to identify relationships between
entities within a document. Current methods rely on text-based encoders and
employ various hand-coded pooling heuristics to aggregate information from
entity mentions and associated contexts. In this paper, we replace these rigid
pooling functions with explicit graph relations by leveraging the intrinsic
graph processing capabilities of the Transformer model. We propose a joint
text-graph Transformer model, and a graph-assisted declarative pooling (GADePo)
specification of the input which provides explicit and high-level instructions
for information aggregation. This allows the pooling process to be guided by
domain-specific knowledge or desired outcomes but still learned by the
Transformer, leading to more flexible and customizable pooling strategies. We
extensively evaluate our method across diverse datasets and models, and show
that our approach yields promising results that are comparable to those
achieved by the hand-coded pooling functions.
less
By: Seongmin Lee, Benjamin Hoover, Hendrik Strobelt, Zijie J. Wang, ShengYun Peng, Austin Wright, Kevin Li, Haekyu Park, Haoyang Yang, Duen Horng Chau
Diffusion-based generative models' impressive ability to create convincing
images has captured global attention. However, their complex internal
structures and operations often make them difficult for non-experts to
understand. We present Diffusion Explainer, the first interactive visualization
tool that explains how Stable Diffusion transforms text prompts into images.
Diffusion Explainer tightly integrates a visual overview of Stable Diff... more
Diffusion-based generative models' impressive ability to create convincing
images has captured global attention. However, their complex internal
structures and operations often make them difficult for non-experts to
understand. We present Diffusion Explainer, the first interactive visualization
tool that explains how Stable Diffusion transforms text prompts into images.
Diffusion Explainer tightly integrates a visual overview of Stable Diffusion's
complex components with detailed explanations of their underlying operations,
enabling users to fluidly transition between multiple levels of abstraction
through animations and interactive elements. By comparing the evolutions of
image representations guided by two related text prompts over refinement
timesteps, users can discover the impact of prompts on image generation.
Diffusion Explainer runs locally in users' web browsers without the need for
installation or specialized hardware, broadening the public's education access
to modern AI techniques. Our open-sourced tool is available at:
https://poloclub.github.io/diffusion-explainer/. A video demo is available at
https://youtu.be/Zg4gxdIWDds.
less
By: František Kmječ, Ondřej Bojar
Many meetings require creating a meeting summary to keep everyone up to date.
Creating minutes of sufficient quality is however very cognitively demanding.
Although we currently possess capable models for both audio speech recognition
(ASR) and summarization, their fully automatic use is still problematic. ASR
models frequently commit errors when transcribing named entities while the
summarization models tend to hallucinate and misinterpret... more
Many meetings require creating a meeting summary to keep everyone up to date.
Creating minutes of sufficient quality is however very cognitively demanding.
Although we currently possess capable models for both audio speech recognition
(ASR) and summarization, their fully automatic use is still problematic. ASR
models frequently commit errors when transcribing named entities while the
summarization models tend to hallucinate and misinterpret the transcript. We
propose a novel tool -- Minuteman -- to enable efficient semi-automatic meeting
minuting. The tool provides a live transcript and a live meeting summary to the
users, who can edit them in a collaborative manner, enabling correction of ASR
errors and imperfect summary points in real time. The resulting application
eases the cognitive load of the notetakers and allows them to easily catch up
if they missed a part of the meeting due to absence or a lack of focus. We
conduct several tests of the application in varied settings, exploring the
worthiness of the concept and the possible user strategies.
less
By: Anthony Colas, Jun Araki, Zhengyu Zhou, Bingqing Wang, Zhe Feng
Explanations accompanied by a recommendation can assist users in
understanding the decision made by recommendation systems, which in turn
increases a user's confidence and trust in the system. Recently, research has
focused on generating natural language explanations in a human-readable format.
Thus far, the proposed approaches leverage item reviews written by users, which
are often subjective, sparse in language, and unable to account for ... more
Explanations accompanied by a recommendation can assist users in
understanding the decision made by recommendation systems, which in turn
increases a user's confidence and trust in the system. Recently, research has
focused on generating natural language explanations in a human-readable format.
Thus far, the proposed approaches leverage item reviews written by users, which
are often subjective, sparse in language, and unable to account for new items
that have not been purchased or reviewed before. Instead, we aim to generate
fact-grounded recommendation explanations that are objectively described with
item features while implicitly considering a user's preferences, based on the
user's purchase history. To achieve this, we propose a knowledge graph (KG)
approach to natural language explainable recommendation. Our approach draws on
user-item features through a novel collaborative filtering-based KG
representation to produce fact-grounded, personalized explanations, while
jointly learning user-item representations for recommendation scoring.
Experimental results show that our approach consistently outperforms previous
state-of-the-art models on natural language explainable recommendation.
less
By: Abdelrahman "Boda" Sadallah, Omar Ahmed, Shimaa Mohamed, Omar Hatem, Doaa Hesham, Ahmed H. Yousef
One of the main tasks of Natural Language Processing (NLP), is Named Entity
Recognition (NER). It is used in many applications and also can be used as an
intermediate step for other tasks. We present ANER, a web-based named entity
recognizer for the Arabic, and Arabizi languages. The model is built upon BERT,
which is a transformer-based encoder. It can recognize 50 different entity
classes, covering various fields. We trained our model on ... more
One of the main tasks of Natural Language Processing (NLP), is Named Entity
Recognition (NER). It is used in many applications and also can be used as an
intermediate step for other tasks. We present ANER, a web-based named entity
recognizer for the Arabic, and Arabizi languages. The model is built upon BERT,
which is a transformer-based encoder. It can recognize 50 different entity
classes, covering various fields. We trained our model on the WikiFANE\_Gold
dataset which consists of Wikipedia articles. We achieved an F1 score of
88.7\%, which beats CAMeL Tools' F1 score of 83\% on the ANERcorp dataset,
which has only 4 classes. We also got an F1 score of 77.7\% on the
NewsFANE\_Gold dataset which contains out-of-domain data from News articles.
The system is deployed on a user-friendly web interface that accepts users'
inputs in Arabic, or Arabizi. It allows users to explore the entities in the
text by highlighting them. It can also direct users to get information about
entities through Wikipedia directly. We added the ability to do NER using our
model, or CAMeL Tools' model through our website. ANER is publicly accessible
at \url{http://www.aner.online}. We also deployed our model on HuggingFace at
https://huggingface.co/boda/ANER, to allow developers to test and use it.
less
By: Harini S I, Somesh Singh, Yaman K Singla, Aanisha Bhattacharyya, Veeky Baths, Changyou Chen, Rajiv Ratn Shah, Balaji Krishnamurthy
Marketers spend billions of dollars on advertisements but to what end? At the
purchase time, if customers cannot recognize a brand for which they saw an ad,
the money spent on the ad is essentially wasted. Despite its importance in
marketing, until now, there has been no study on the memorability of ads in the
ML literature. Most studies have been conducted on short-term recall (<5 mins)
on specific content types like object and action vide... more
Marketers spend billions of dollars on advertisements but to what end? At the
purchase time, if customers cannot recognize a brand for which they saw an ad,
the money spent on the ad is essentially wasted. Despite its importance in
marketing, until now, there has been no study on the memorability of ads in the
ML literature. Most studies have been conducted on short-term recall (<5 mins)
on specific content types like object and action videos. On the other hand, the
advertising industry only cares about long-term memorability (a few hours or
longer), and advertisements are almost always highly multimodal, depicting a
story through its different modalities (text, images, and videos). With this
motivation, we conduct the first large scale memorability study consisting of
1203 participants and 2205 ads covering 276 brands. Running statistical tests
over different participant subpopulations and ad-types, we find many
interesting insights into what makes an ad memorable - both content and human
factors. For example, we find that brands which use commercials with fast
moving scenes are more memorable than those with slower scenes (p=8e-10) and
that people who use ad-blockers remember lower number of ads than those who
don't (p=5e-3). Further, with the motivation of simulating the memorability of
marketing materials for a particular audience, ultimately helping create one,
we present a novel model, Sharingan, trained to leverage real-world knowledge
of LLMs and visual knowledge of visual encoders to predict the memorability of
a content. We test our model on all the prominent memorability datasets in
literature (both images and videos) and achieve state of the art across all of
them. We conduct extensive ablation studies across memory types, modality,
brand, and architectural choices to find insights into what drives memory.
less
By: Davut Emre Tasar, Ceren Ocal Tasar
With the increasing use of cloud-based services for training and deploying
machine learning models, data privacy has become a major concern. This is
particularly important for natural language processing (NLP) models, which
often process sensitive information such as personal communications and
confidential documents. In this study, we propose a method for training NLP
models on encrypted text data to mitigate data privacy concerns while
ma... more
With the increasing use of cloud-based services for training and deploying
machine learning models, data privacy has become a major concern. This is
particularly important for natural language processing (NLP) models, which
often process sensitive information such as personal communications and
confidential documents. In this study, we propose a method for training NLP
models on encrypted text data to mitigate data privacy concerns while
maintaining similar performance to models trained on non-encrypted data. We
demonstrate our method using two different architectures, namely
Doc2Vec+XGBoost and Doc2Vec+LSTM, and evaluate the models on the 20 Newsgroups
dataset. Our results indicate that both encrypted and non-encrypted models
achieve comparable performance, suggesting that our encryption method is
effective in preserving data privacy without sacrificing model accuracy. In
order to replicate our experiments, we have provided a Colab notebook at the
following address: https://t.ly/lR-TP
less
By: Yun-Cheng Wang, Xiou Ge, Bin Wang, C. -C. Jay Kuo
Knowledge graph entity typing (KGET) is a task to predict the missing entity
types in knowledge graphs (KG). Previously, KG embedding (KGE) methods tried to
solve the KGET task by introducing an auxiliary relation, 'hasType', to model
the relationship between entities and their types. However, a single auxiliary
relation has limited expressiveness for diverse entity-type patterns. We
improve the expressiveness of KGE methods by introducing ... more
Knowledge graph entity typing (KGET) is a task to predict the missing entity
types in knowledge graphs (KG). Previously, KG embedding (KGE) methods tried to
solve the KGET task by introducing an auxiliary relation, 'hasType', to model
the relationship between entities and their types. However, a single auxiliary
relation has limited expressiveness for diverse entity-type patterns. We
improve the expressiveness of KGE methods by introducing multiple auxiliary
relations in this work. Similar entity types are grouped to reduce the number
of auxiliary relations and improve their capability to model entity-type
patterns with different granularities. With the presence of multiple auxiliary
relations, we propose a method adopting an Asynchronous learning scheme for
Entity Typing, named AsyncET, which updates the entity and type embeddings
alternatively to keep the learned entity embedding up-to-date and informative
for entity type prediction. Experiments are conducted on two commonly used KGET
datasets to show that the performance of KGE methods on the KGET task can be
substantially improved by the proposed multiple auxiliary relations and
asynchronous embedding learning. Furthermore, our method has a significant
advantage over state-of-the-art methods in model sizes and time complexity.
less
By: Yubo Wang, Xueguang Ma, Wenhu Chen
Large-scale language models (LLMs), such as ChatGPT, are capable of
generating human-like responses for various downstream tasks, such as
task-oriented dialogues and question answering. However, applying LLMs to
medical domains remains challenging due to their inability to leverage
domain-specific knowledge. In this study, we present the Large-scale Language
Models Augmented with Medical Textbooks (LLM-AMT), which integrates
authoritative m... more
Large-scale language models (LLMs), such as ChatGPT, are capable of
generating human-like responses for various downstream tasks, such as
task-oriented dialogues and question answering. However, applying LLMs to
medical domains remains challenging due to their inability to leverage
domain-specific knowledge. In this study, we present the Large-scale Language
Models Augmented with Medical Textbooks (LLM-AMT), which integrates
authoritative medical textbooks as the cornerstone of its design, enhancing its
proficiency in the specialized domain through plug-and-play modules, comprised
of a Hybrid Textbook Retriever, supplemented by the Query Augmenter and the LLM
Reader. Experimental evaluation on three open-domain medical question-answering
tasks reveals a substantial enhancement in both the professionalism and
accuracy of the LLM responses when utilizing LLM-AMT, exhibiting an improvement
ranging from 11.4% to 13.2%. Despite being 100 times smaller, we found that
medical textbooks as the retrieval corpus serves as a more valuable external
knowledge source than Wikipedia in the medical domain. Our experiments show
that textbook augmentation results in a performance improvement ranging from
9.7% to 12.2% over Wikipedia augmentation.
less
By: Tanjim Mahmud, Michal Ptaszynski, Fumito Masui
The negative effects of online bullying and harassment are increasing with
Internet popularity, especially in social media. One solution is using natural
language processing (NLP) and machine learning (ML) methods for the automatic
detection of harmful remarks, but these methods are limited in low-resource
languages like the Chittagonian dialect of Bangla.This study focuses on
detecting vulgar remarks in social media using supervised ML and... more
The negative effects of online bullying and harassment are increasing with
Internet popularity, especially in social media. One solution is using natural
language processing (NLP) and machine learning (ML) methods for the automatic
detection of harmful remarks, but these methods are limited in low-resource
languages like the Chittagonian dialect of Bangla.This study focuses on
detecting vulgar remarks in social media using supervised ML and deep learning
algorithms.Logistic Regression achieved promising accuracy (0.91) while simple
RNN with Word2vec and fastTex had lower accuracy (0.84-0.90), highlighting the
issue that NN algorithms require more data.
less
By: Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, Xuanjing Huang
Spiking neural networks (SNNs) offer a promising avenue to implement deep
neural networks in a more energy-efficient way. However, the network
architectures of existing SNNs for language tasks are too simplistic, and deep
architectures have not been fully explored, resulting in a significant
performance gap compared to mainstream transformer-based networks such as BERT.
To this end, we improve a recently-proposed spiking transformer (i.e.,
... more
Spiking neural networks (SNNs) offer a promising avenue to implement deep
neural networks in a more energy-efficient way. However, the network
architectures of existing SNNs for language tasks are too simplistic, and deep
architectures have not been fully explored, resulting in a significant
performance gap compared to mainstream transformer-based networks such as BERT.
To this end, we improve a recently-proposed spiking transformer (i.e.,
Spikformer) to make it possible to process language tasks and propose a
two-stage knowledge distillation method for training it, which combines
pre-training by distilling knowledge from BERT with a large collection of
unlabelled texts and fine-tuning with task-specific instances via knowledge
distillation again from the BERT fine-tuned on the same training examples.
Through extensive experimentation, we show that the models trained with our
method, named SpikeBERT, outperform state-of-the-art SNNs and even achieve
comparable results to BERTs on text classification tasks for both English and
Chinese with much less energy consumption.
less
By: Hongyin Zhu
Passage reranking is a crucial task in many applications, particularly when
dealing with large-scale documents. Traditional neural architectures are
limited in retrieving the best passage for a question because they usually
match the question to each passage separately, seldom considering contextual
information in other passages that can provide comparison and reference
information. This paper presents a list-context attention mechanism to ... more
Passage reranking is a crucial task in many applications, particularly when
dealing with large-scale documents. Traditional neural architectures are
limited in retrieving the best passage for a question because they usually
match the question to each passage separately, seldom considering contextual
information in other passages that can provide comparison and reference
information. This paper presents a list-context attention mechanism to augment
the passage representation by incorporating the list-context information from
other candidates. The proposed coarse-to-fine (C2F) neural retriever addresses
the out-of-memory limitation of the passage attention mechanism by dividing the
list-context modeling process into two sub-processes, allowing for efficient
encoding of context information from a large number of candidate answers. This
method can be generally used to encode context information from any number of
candidate answers in one pass. Different from most multi-stage information
retrieval architectures, this model integrates the coarse and fine rankers into
the joint optimization process, allowing for feedback between the two layers to
update the model simultaneously. Experiments demonstrate the effectiveness of
the proposed approach.
less
By: Samuel Rhys Cox, Ashraf Abdul, Wei Tsang Ooi
Large language models (LLMs) are increasingly capable and prevalent, and can
be used to produce creative content. The quality of content is influenced by
the prompt used, with more specific prompts that incorporate examples generally
producing better results. On from this, it could be seen that using
instructions written for crowdsourcing tasks (that are specific and include
examples to guide workers) could prove effective LLM prompts. To e... more
Large language models (LLMs) are increasingly capable and prevalent, and can
be used to produce creative content. The quality of content is influenced by
the prompt used, with more specific prompts that incorporate examples generally
producing better results. On from this, it could be seen that using
instructions written for crowdsourcing tasks (that are specific and include
examples to guide workers) could prove effective LLM prompts. To explore this,
we used a previous crowdsourcing pipeline that gave examples to people to help
them generate a collectively diverse corpus of motivational messages. We then
used this same pipeline to generate messages using GPT-4, and compared the
collective diversity of messages from: (1) crowd-writers, (2) GPT-4 using the
pipeline, and (3 & 4) two baseline GPT-4 prompts. We found that the LLM prompts
using the crowdsourcing pipeline caused GPT-4 to produce more diverse messages
than the two baseline prompts. We also discuss implications from messages
generated by both human writers and LLMs.
less
By: Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, Abhinav Rastogi
Reinforcement learning from human feedback (RLHF) is effective at aligning
large language models (LLMs) to human preferences, but gathering high quality
human preference labels is a key bottleneck. We conduct a head-to-head
comparison of RLHF vs. RL from AI Feedback (RLAIF) - a technique where
preferences are labeled by an off-the-shelf LLM in lieu of humans, and we find
that they result in similar improvements. On the task of summarization... more
Reinforcement learning from human feedback (RLHF) is effective at aligning
large language models (LLMs) to human preferences, but gathering high quality
human preference labels is a key bottleneck. We conduct a head-to-head
comparison of RLHF vs. RL from AI Feedback (RLAIF) - a technique where
preferences are labeled by an off-the-shelf LLM in lieu of humans, and we find
that they result in similar improvements. On the task of summarization, human
evaluators prefer generations from both RLAIF and RLHF over a baseline
supervised fine-tuned model in ~70% of cases. Furthermore, when asked to rate
RLAIF vs. RLHF summaries, humans prefer both at equal rates. These results
suggest that RLAIF can yield human-level performance, offering a potential
solution to the scalability limitations of RLHF.
less
By: Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou
Large Language Models (LLMs) present strong general capabilities, and a
current compelling challenge is stimulating their specialized capabilities,
such as machine translation, through low-cost instruction tuning. The standard
instruction-following data is sequentially organized as the concatenation of an
instruction, an input, and a response. As the attention mechanism of LLMs has
limitations on local focus, LLMs tend to focus more on the ... more
Large Language Models (LLMs) present strong general capabilities, and a
current compelling challenge is stimulating their specialized capabilities,
such as machine translation, through low-cost instruction tuning. The standard
instruction-following data is sequentially organized as the concatenation of an
instruction, an input, and a response. As the attention mechanism of LLMs has
limitations on local focus, LLMs tend to focus more on the words or sentences
nearby at each position. This leads to a high risk of instruction forgetting
during decoding. To alleviate the above issues, We propose SWIE
(Segment-Weighted Instruction Embedding) and an instruction-following dataset
OVERMISS. SWIE improves the model instruction understanding by adding a global
instruction representation on the following input and response representations.
OVERMISS improves model faithfulness by comparing over-translation and
miss-translation results with the correct translation. We apply our methods to
two main-stream open-source LLMs, BLOOM and LLaMA. The experimental results
demonstrate significant improvements in translation performance with SWIE based
on BLOOMZ-3b, particularly in zero-shot and long text translations due to
reduced instruction forgetting risk. Additionally, OVERMISS outperforms the
baseline in translation performance (e.g. an increase in BLEU scores from 0.69
to 3.12 and an average improvement of 0.48 percentage comet scores for
LLaMA-7b) with further enhancements seen in models combining OVERMISS and SWIE
(e.g. the BLUE scores increase up to 0.56 from English to German across three
different backbones), and both exhibit improvements in the faithfulness metric
based on word alignment.
less
BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models
0upvotes
By: Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, William Chandra Tjhi
The rapid development of Large Language Models (LLMs) and the emergence of
novel abilities with scale have necessitated the construction of holistic,
diverse and challenging benchmarks such as HELM and BIG-bench. However, at the
moment, most of these benchmarks focus only on performance in English and
evaluations that include Southeast Asian (SEA) languages are few in number. We
therefore propose BHASA, a holistic linguistic and cultural ev... more
The rapid development of Large Language Models (LLMs) and the emergence of
novel abilities with scale have necessitated the construction of holistic,
diverse and challenging benchmarks such as HELM and BIG-bench. However, at the
moment, most of these benchmarks focus only on performance in English and
evaluations that include Southeast Asian (SEA) languages are few in number. We
therefore propose BHASA, a holistic linguistic and cultural evaluation suite
for LLMs in SEA languages. It comprises three components: (1) a NLP benchmark
covering eight tasks across Natural Language Understanding (NLU), Generation
(NLG) and Reasoning (NLR) tasks, (2) LINDSEA, a linguistic diagnostic toolkit
that spans the gamut of linguistic phenomena including syntax, semantics and
pragmatics, and (3) a cultural diagnostics dataset that probes for both
cultural representation and sensitivity. For this preliminary effort, we
implement the NLP benchmark only for Indonesian, Vietnamese, Thai and Tamil,
and we only include Indonesian and Tamil for LINDSEA and the cultural
diagnostics dataset. As GPT-4 is purportedly one of the best-performing
multilingual LLMs at the moment, we use it as a yardstick to gauge the
capabilities of LLMs in the context of SEA languages. Our initial experiments
on GPT-4 with BHASA find it lacking in various aspects of linguistic
capabilities, cultural representation and sensitivity in the targeted SEA
languages. BHASA is a work in progress and will continue to be improved and
expanded in the future.
less
By: Alessandra Polimeno, Myrthe Reuver, Sanne Vrijenhoek, Antske Fokkens
News recommender systems play an increasingly influential role in shaping
information access within democratic societies. However, tailoring
recommendations to users' specific interests can result in the divergence of
information streams. Fragmented access to information poses challenges to the
integrity of the public sphere, thereby influencing democracy and public
discourse. The Fragmentation metric quantifies the degree of fragmentation ... more
News recommender systems play an increasingly influential role in shaping
information access within democratic societies. However, tailoring
recommendations to users' specific interests can result in the divergence of
information streams. Fragmented access to information poses challenges to the
integrity of the public sphere, thereby influencing democracy and public
discourse. The Fragmentation metric quantifies the degree of fragmentation of
information streams in news recommendations. Accurate measurement of this
metric requires the application of Natural Language Processing (NLP) to
identify distinct news events, stories, or timelines. This paper presents an
extensive investigation of various approaches for quantifying Fragmentation in
news recommendations. These approaches are evaluated both intrinsically, by
measuring performance on news story clustering, and extrinsically, by assessing
the Fragmentation scores of different simulated news recommender scenarios. Our
findings demonstrate that agglomerative hierarchical clustering coupled with
SentenceBERT text representation is substantially better at detecting
Fragmentation than earlier implementations. Additionally, the analysis of
simulated scenarios yields valuable insights and recommendations for
stakeholders concerning the measurement and interpretation of Fragmentation.
less
By: Shaoxiong Ji, Wei Sun, Pekka Marttinen
Many computational linguistic methods have been proposed to study the
information content of languages. We consider two interesting research
questions: 1) how is information distributed over long documents, and 2) how
does content reduction, such as token selection and text summarization, affect
the information density in long documents. We present four criteria for
information density estimation for long documents, including surprisal,
ent... more
Many computational linguistic methods have been proposed to study the
information content of languages. We consider two interesting research
questions: 1) how is information distributed over long documents, and 2) how
does content reduction, such as token selection and text summarization, affect
the information density in long documents. We present four criteria for
information density estimation for long documents, including surprisal,
entropy, uniform information density, and lexical density. Among those
criteria, the first three adopt the measures from information theory. We
propose an attention-based word selection method for clinical notes and study
machine summarization for multiple-domain documents. Our findings reveal the
systematic difference in information density of long text in various domains.
Empirical results on automated medical coding from long clinical notes show the
effectiveness of the attention-based word selection method.
less
By: Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, Zhifang Sui
Reasoning is a cognitive process of using evidence to reach a sound
conclusion. The reasoning capability is essential for large language models
(LLMs) to serve as the brain of the artificial general intelligence agent.
Recent studies reveal that fine-tuning LLMs on data with the chain of thought
(COT) reasoning process can significantly enhance their reasoning capabilities.
However, we find that the fine-tuned LLMs suffer from an \textit{As... more
Reasoning is a cognitive process of using evidence to reach a sound
conclusion. The reasoning capability is essential for large language models
(LLMs) to serve as the brain of the artificial general intelligence agent.
Recent studies reveal that fine-tuning LLMs on data with the chain of thought
(COT) reasoning process can significantly enhance their reasoning capabilities.
However, we find that the fine-tuned LLMs suffer from an \textit{Assessment
Misalignment} problem, i.e., they frequently assign higher scores to subpar
COTs, leading to potential limitations in their reasoning abilities. To address
this problem, we introduce an \textit{Alignment Fine-Tuning (AFT)} paradigm,
which involves three steps: 1) fine-tuning LLMs with COT training data; 2)
generating multiple COT responses for each question, and categorizing them into
positive and negative ones based on whether they achieve the correct answer; 3)
calibrating the scores of positive and negative responses given by LLMs with a
novel constraint alignment loss. Specifically, the constraint alignment loss
has two objectives: a) Alignment, which guarantees that positive scores surpass
negative scores to encourage answers with high-quality COTs; b) Constraint,
which keeps the negative scores confined to a reasonable range to prevent the
model degradation. Beyond just the binary positive and negative feedback, the
constraint alignment loss can be seamlessly adapted to the ranking situations
when ranking feedback is accessible. Furthermore, we also delve deeply into
recent ranking-based alignment methods, such as DPO, RRHF, and PRO, and
discover that the constraint, which has been overlooked by these approaches, is
also crucial for their performance. Extensive experiments on four reasoning
benchmarks with both binary and ranking feedback demonstrate the effectiveness
of AFT.
less
By: Huimin Wang, Wai-Chung Kwan, Kam-Fai Wong
Recent works usually address Dialog policy learning DPL by training a
reinforcement learning (RL) agent to determine the best dialog action. However,
existing works on deep RL require a large volume of agent-user interactions to
achieve acceptable performance. In this paper, we propose to make full use of
the plain text knowledge from the pre-trained language model to accelerate the
RL agent's learning speed. Specifically, we design a dialo... more
Recent works usually address Dialog policy learning DPL by training a
reinforcement learning (RL) agent to determine the best dialog action. However,
existing works on deep RL require a large volume of agent-user interactions to
achieve acceptable performance. In this paper, we propose to make full use of
the plain text knowledge from the pre-trained language model to accelerate the
RL agent's learning speed. Specifically, we design a dialog action-aware
transformer encoder (DaTrans), which integrates a new fine-tuning procedure
named masked last action task to encourage DaTrans to be dialog-aware and
distils action-specific features. Then, DaTrans is further optimized in an RL
setting with ongoing interactions and evolves through exploration in the dialog
action space toward maximizing long-term accumulated rewards. The effectiveness
and efficiency of the proposed model are demonstrated with both simulator
evaluation and human evaluation.
less
By: Jiawen Liu, Kan Li
Exposure bias poses a common challenge in numerous natural language
processing tasks, particularly in the dialog generation. In response to this
issue, researchers have devised various techniques, among which scheduled
sampling has proven to be an effective method for mitigating exposure bias.
However, the existing state-of-the-art scheduled sampling methods solely
consider the current sampling words' quality for threshold truncation sampli... more
Exposure bias poses a common challenge in numerous natural language
processing tasks, particularly in the dialog generation. In response to this
issue, researchers have devised various techniques, among which scheduled
sampling has proven to be an effective method for mitigating exposure bias.
However, the existing state-of-the-art scheduled sampling methods solely
consider the current sampling words' quality for threshold truncation sampling,
which overlooks the importance of sentence-level information and the method of
threshold truncation warrants further discussion. In this paper, we propose a
bilevel scheduled sampling model that takes the sentence-level information into
account and incorporates it with word-level quality. To enhance sampling
diversity and improve the model's adaptability, we propose a smooth function
that maps the combined result of sentence-level and word-level information to
an appropriate range, and employ probabilistic sampling based on the mapped
values instead of threshold truncation. Experiments conducted on the
DailyDialog and PersonaChat datasets demonstrate the effectiveness of our
proposed methods, which significantly alleviate the exposure bias problem and
outperform state-of-the-art scheduled sampling methods.
less
Recurrent Memory Transformer
2upvotes
By: Aydar Bulatov, Yuri Kuratov, Mikhail S. Burtsev
Transformer-based models show their effectiveness across multiple domains and
tasks. The self-attention allows to combine information from all sequence
elements into context-aware representations. However, global and local
information has to be stored mostly in the same element-wise representations.
Moreover, the length of an input sequence is limited by quadratic computational
complexity of self-attention.
In this work, we propose and st... more
Transformer-based models show their effectiveness across multiple domains and
tasks. The self-attention allows to combine information from all sequence
elements into context-aware representations. However, global and local
information has to be stored mostly in the same element-wise representations.
Moreover, the length of an input sequence is limited by quadratic computational
complexity of self-attention.
In this work, we propose and study a memory-augmented segment-level recurrent
Transformer (RMT). Memory allows to store and process local and global
information as well as to pass information between segments of the long
sequence with the help of recurrence.
We implement a memory mechanism with no changes to Transformer model by
adding special memory tokens to the input or output sequence. Then the model is
trained to control both memory operations and sequence representations
processing.
Results of experiments show that RMT performs on par with the Transformer-XL
on language modeling for smaller memory sizes and outperforms it for tasks that
require longer sequence processing. We show that adding memory tokens to Tr-XL
is able to improve its performance. This makes Recurrent Memory Transformer a
promising architecture for applications that require learning of long-term
dependencies and general purpose in memory processing, such as algorithmic
tasks and reasoning.
less
By: Mohsin Ali, Kandukuri Sai Teja, Neeharika Gupta, Parth Patwa, Anubhab Chatterjee, Vinija Jain, Aman Chadha, Amitava Das
The mixing of two or more languages is called Code-Mixing (CM). CM is a
social norm in multilingual societies. Neural Language Models (NLMs) like
transformers have been very effective on many NLP tasks. However, NLM for CM is
an under-explored area. Though transformers are capable and powerful, they
cannot always encode positional/sequential information since they are
non-recurrent. Therefore, to enrich word information and incorporate posi... more
The mixing of two or more languages is called Code-Mixing (CM). CM is a
social norm in multilingual societies. Neural Language Models (NLMs) like
transformers have been very effective on many NLP tasks. However, NLM for CM is
an under-explored area. Though transformers are capable and powerful, they
cannot always encode positional/sequential information since they are
non-recurrent. Therefore, to enrich word information and incorporate positional
information, positional encoding is defined. We hypothesize that Switching
Points (SPs), i.e., junctions in the text where the language switches (L1 -> L2
or L2-> L1), pose a challenge for CM Language Models (LMs), and hence give
special emphasis to switching points in the modeling process. We experiment
with several positional encoding mechanisms and show that rotatory positional
encodings along with switching point information yield the best results.
We introduce CONFLATOR: a neural language modeling approach for code-mixed
languages. CONFLATOR tries to learn to emphasize switching points using smarter
positional encoding, both at unigram and bigram levels. CONFLATOR outperforms
the state-of-the-art on two tasks based on code-mixed Hindi and English
(Hinglish): (i) sentiment analysis and (ii) machine translation.
less
By: Helena Bonaldi, Giuseppe Attanasio, Debora Nozza, Marco Guerini
Recent computational approaches for combating online hate speech involve the
automatic generation of counter narratives by adapting Pretrained
Transformer-based Language Models (PLMs) with human-curated data. This process,
however, can produce in-domain overfitting, resulting in models generating
acceptable narratives only for hatred similar to training data, with little
portability to other targets or to real-world toxic language. This pap... more
Recent computational approaches for combating online hate speech involve the
automatic generation of counter narratives by adapting Pretrained
Transformer-based Language Models (PLMs) with human-curated data. This process,
however, can produce in-domain overfitting, resulting in models generating
acceptable narratives only for hatred similar to training data, with little
portability to other targets or to real-world toxic language. This paper
introduces novel attention regularization methodologies to improve the
generalization capabilities of PLMs for counter narratives generation.
Overfitting to training-specific terms is then discouraged, resulting in more
diverse and richer narratives. We experiment with two attention-based
regularization techniques on a benchmark English dataset. Regularized models
produce better counter narratives than state-of-the-art approaches in most
cases, both in terms of automatic metrics and human evaluation, especially when
hateful targets are not present in the training data. This work paves the way
for better and more flexible counter-speech generation models, a task for which
datasets are highly challenging to produce.
less
By: Kushal Tirumala, Daniel Simig, Armen Aghajanyan, Ari S. Morcos
Over recent years, an increasing amount of compute and data has been poured
into training large language models (LLMs), usually by doing one-pass learning
on as many tokens as possible randomly selected from large-scale web corpora.
While training on ever-larger portions of the internet leads to consistent
performance improvements, the size of these improvements diminishes with scale,
and there has been little work exploring the effect of d... more
Over recent years, an increasing amount of compute and data has been poured
into training large language models (LLMs), usually by doing one-pass learning
on as many tokens as possible randomly selected from large-scale web corpora.
While training on ever-larger portions of the internet leads to consistent
performance improvements, the size of these improvements diminishes with scale,
and there has been little work exploring the effect of data selection on
pre-training and downstream performance beyond simple de-duplication methods
such as MinHash. Here, we show that careful data selection (on top of
de-duplicated data) via pre-trained model embeddings can speed up training (20%
efficiency gains) and improves average downstream accuracy on 16 NLP tasks (up
to 2%) at the 6.7B model scale. Furthermore, we show that repeating data
intelligently consistently outperforms baseline training (while repeating
random data performs worse than baseline training). Our results indicate that
clever data selection can significantly improve LLM pre-training, calls into
question the common practice of training for a single epoch on as much data as
possible, and demonstrates a path to keep improving our models past the limits
of randomly sampling web data.
less
By: Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, Yiming Qian
With the increasing size of pre-trained language models (PLMs), fine-tuning
all the parameters in the model is not efficient, especially when there are a
large number of downstream tasks, which incur significant training and storage
costs. Many parameter-efficient fine-tuning (PEFT) approaches have been
proposed, among which, Low-Rank Adaptation (LoRA) is a representative approach
that injects trainable rank decomposition matrices into ever... more
With the increasing size of pre-trained language models (PLMs), fine-tuning
all the parameters in the model is not efficient, especially when there are a
large number of downstream tasks, which incur significant training and storage
costs. Many parameter-efficient fine-tuning (PEFT) approaches have been
proposed, among which, Low-Rank Adaptation (LoRA) is a representative approach
that injects trainable rank decomposition matrices into every target module.
Yet LoRA ignores the importance of parameters in different modules. To address
this problem, many works have been proposed to prune the parameters of LoRA.
However, under limited training conditions, the upper bound of the rank of the
pruned parameter matrix is still affected by the preset values. We, therefore,
propose IncreLoRA, an incremental parameter allocation method that adaptively
adds trainable parameters during training based on the importance scores of
each module. This approach is different from the pruning method as it is not
limited by the initial number of training parameters, and each parameter matrix
has a higher rank upper bound for the same training overhead. We conduct
extensive experiments on GLUE to demonstrate the effectiveness of IncreLoRA.
The results show that our method owns higher parameter efficiency, especially
when under the low-resource settings where our method significantly outperforms
the baselines. Our code is publicly available.
less
By: Renlong Jie, Xiaojun Meng, Lifeng Shang, Xin Jiang, Qun Liu
Recently, large language models (LLMs) like ChatGPT and GPT-4 have attracted
great attention given their surprising improvement and performance. Length
controlled generation of LLMs emerges as an important topic, which also enables
users to fully leverage the capability of LLMs in more real-world scenarios
like generating a proper answer or essay of a desired length. In addition, the
autoregressive generation in LLMs is extremely time-consu... more
Recently, large language models (LLMs) like ChatGPT and GPT-4 have attracted
great attention given their surprising improvement and performance. Length
controlled generation of LLMs emerges as an important topic, which also enables
users to fully leverage the capability of LLMs in more real-world scenarios
like generating a proper answer or essay of a desired length. In addition, the
autoregressive generation in LLMs is extremely time-consuming, while the
ability of controlling this generated length can arbitrarily reduce the
inference cost by limiting the length, and thus satisfy different needs.
Therefore, we aim to propose a prompt-based length control method to achieve
this length controlled generation, which can also be widely applied in
GPT-style LLMs. In particular, we adopt reinforcement learning with the reward
signal given by either trainable or rule-based reward model, which further
affects the generation of LLMs via rewarding a pre-defined target length.
Experiments show that our method significantly improves the accuracy of
prompt-based length control for summarization task on popular datasets like
CNNDM and NYT. We believe this length-controllable ability can provide more
potentials towards the era of LLMs.
less
By: Yanjun Gao, Ruizhe Li, John Caskey, Dmitriy Dligach, Timothy Miller, Matthew M. Churpek, Majid Afshar
Electronic Health Records (EHRs) and routine documentation practices play a
vital role in patients' daily care, providing a holistic record of health,
diagnoses, and treatment. However, complex and verbose EHR narratives overload
healthcare providers, risking diagnostic inaccuracies. While Large Language
Models (LLMs) have showcased their potential in diverse language tasks, their
application in the healthcare arena needs to ensure the mini... more
Electronic Health Records (EHRs) and routine documentation practices play a
vital role in patients' daily care, providing a holistic record of health,
diagnoses, and treatment. However, complex and verbose EHR narratives overload
healthcare providers, risking diagnostic inaccuracies. While Large Language
Models (LLMs) have showcased their potential in diverse language tasks, their
application in the healthcare arena needs to ensure the minimization of
diagnostic errors and the prevention of patient harm. In this paper, we outline
an innovative approach for augmenting the proficiency of LLMs in the realm of
automated diagnosis generation, achieved through the incorporation of a medical
knowledge graph (KG) and a novel graph model: Dr.Knows, inspired by the
clinical diagnostic reasoning process. We derive the KG from the National
Library of Medicine's Unified Medical Language System (UMLS), a robust
repository of biomedical knowledge. Our method negates the need for
pre-training and instead leverages the KG as an auxiliary instrument aiding in
the interpretation and summarization of complex medical concepts. Using
real-world hospital datasets, our experimental results demonstrate that the
proposed approach of combining LLMs with KG has the potential to improve the
accuracy of automated diagnosis generation. More importantly, our approach
offers an explainable diagnostic pathway, edging us closer to the realization
of AI-augmented diagnostic decision support systems.
less
By: Jorge Martinez-Gil
Semantic similarity measures are widely used in natural language processing
to catalyze various computer-related tasks. However, no single semantic
similarity measure is the most appropriate for all tasks, and researchers often
use ensemble strategies to ensure performance. This research work proposes a
method for automatically designing semantic similarity ensembles. In fact, our
proposed method uses grammatical evolution, for the first ti... more
Semantic similarity measures are widely used in natural language processing
to catalyze various computer-related tasks. However, no single semantic
similarity measure is the most appropriate for all tasks, and researchers often
use ensemble strategies to ensure performance. This research work proposes a
method for automatically designing semantic similarity ensembles. In fact, our
proposed method uses grammatical evolution, for the first time, to
automatically select and aggregate measures from a pool of candidates to create
an ensemble that maximizes correlation to human judgment. The method is
evaluated on several benchmark datasets and compared to state-of-the-art
ensembles, showing that it can significantly improve similarity assessment
accuracy and outperform existing methods in some cases. As a result, our
research demonstrates the potential of using grammatical evolution to
automatically compare text and prove the benefits of using ensembles for
semantic similarity tasks. The source code that illustrates our approach can be
downloaded from https://github.com/jorge-martinez-gil/sesige.
less
By: Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
Pre-trained language models (PLMs) contain vast amounts of factual knowledge,
but how the knowledge is stored in the parameters remains unclear. This paper
delves into the complex task of understanding how factual knowledge is stored
in multilingual PLMs, and introduces the Architecture-adapted Multilingual
Integrated Gradients method, which successfully localizes knowledge neurons
more precisely compared to current methods, and is more uni... more
Pre-trained language models (PLMs) contain vast amounts of factual knowledge,
but how the knowledge is stored in the parameters remains unclear. This paper
delves into the complex task of understanding how factual knowledge is stored
in multilingual PLMs, and introduces the Architecture-adapted Multilingual
Integrated Gradients method, which successfully localizes knowledge neurons
more precisely compared to current methods, and is more universal across
various architectures and languages. Moreover, we conduct an in-depth
exploration of knowledge neurons, leading to the following two important
discoveries: (1) The discovery of Language-Independent Knowledge Neurons, which
store factual knowledge in a form that transcends language. We design
cross-lingual knowledge editing experiments, demonstrating that the PLMs can
accomplish this task based on language-independent neurons; (2) The discovery
of Degenerate Knowledge Neurons, a novel type of neuron showing that different
knowledge neurons can store the same fact. Its property of functional overlap
endows the PLMs with a robust mastery of factual knowledge. We design
fact-checking experiments, proving that the degenerate knowledge neurons can
help the PLMs to detect wrong facts. Experiments corroborate these findings,
shedding light on the mechanisms of factual knowledge storage in multilingual
PLMs, and contribute valuable insights to the field. The source code will be
made publicly available for further research.
less
By: Chengyu Wang, Zhongjie Duan, Bingyan Liu, Xinyi Zou, Cen Chen, Kui Jia, Jun Huang
Text-to-image synthesis for the Chinese language poses unique challenges due
to its large vocabulary size, and intricate character relationships. While
existing diffusion models have shown promise in generating images from textual
descriptions, they often neglect domain-specific contexts and lack robustness
in handling the Chinese language. This paper introduces PAI-Diffusion, a
comprehensive framework that addresses these limitations. PAI-... more
Text-to-image synthesis for the Chinese language poses unique challenges due
to its large vocabulary size, and intricate character relationships. While
existing diffusion models have shown promise in generating images from textual
descriptions, they often neglect domain-specific contexts and lack robustness
in handling the Chinese language. This paper introduces PAI-Diffusion, a
comprehensive framework that addresses these limitations. PAI-Diffusion
incorporates both general and domain-specific Chinese diffusion models,
enabling the generation of contextually relevant images. It explores the
potential of using LoRA and ControlNet for fine-grained image style transfer
and image editing, empowering users with enhanced control over image
generation. Moreover, PAI-Diffusion seamlessly integrates with Alibaba Cloud's
Machine Learning Platform for AI, providing accessible and scalable solutions.
All the Chinese diffusion model checkpoints, LoRAs, and ControlNets, including
domain-specific ones, are publicly available. A user-friendly Chinese WebUI and
the diffusers-api elastic inference toolkit, also open-sourced, further
facilitate the easy deployment of PAI-Diffusion models in various environments,
making it a valuable resource for Chinese text-to-image synthesis.
less
By: Fabian Lechner, Allison Lahnala, Charles Welch, Lucie Flek
The potential to provide patients with faster information access while
allowing medical specialists to concentrate on critical tasks makes medical
domain dialog agents appealing. However, the integration of large-language
models (LLMs) into these agents presents certain limitations that may result in
serious consequences. This paper investigates the challenges and risks of using
GPT-3-based models for medical question-answering (MedQA). We ... more
The potential to provide patients with faster information access while
allowing medical specialists to concentrate on critical tasks makes medical
domain dialog agents appealing. However, the integration of large-language
models (LLMs) into these agents presents certain limitations that may result in
serious consequences. This paper investigates the challenges and risks of using
GPT-3-based models for medical question-answering (MedQA). We perform several
evaluations contextualized in terms of standard medical principles. We provide
a procedure for manually designing patient queries to stress-test high-risk
limitations of LLMs in MedQA systems. Our analysis reveals that LLMs fail to
respond adequately to these queries, generating erroneous medical information,
unsafe recommendations, and content that may be considered offensive.
less
By: Zhijie Bao, Wei Chen, Shengze Xiao, Kuang Ren, Jiaao Wu, Cheng Zhong, Jiajie Peng, Xuanjing Huang, Zhongyu Wei
We propose DISC-MedLLM, a comprehensive solution that leverages Large
Language Models (LLMs) to provide accurate and truthful medical response in
end-to-end conversational healthcare services. To construct high-quality
Supervised Fine-Tuning (SFT) datasets, we employ three strategies: utilizing
medical knowledge-graphs, reconstructing real-world dialogues, and
incorporating human-guided preference rephrasing. These datasets are
instrumental... more
We propose DISC-MedLLM, a comprehensive solution that leverages Large
Language Models (LLMs) to provide accurate and truthful medical response in
end-to-end conversational healthcare services. To construct high-quality
Supervised Fine-Tuning (SFT) datasets, we employ three strategies: utilizing
medical knowledge-graphs, reconstructing real-world dialogues, and
incorporating human-guided preference rephrasing. These datasets are
instrumental in training DISC-MedLLM, surpassing existing medical LLMs in both
single-turn and multi-turn consultation scenarios. Extensive experimental
results demonstrate the effectiveness of the proposed model in bridging the gap
between general language models and real-world medical consultation.
Additionally, we release the constructed dataset and model weights to further
contribute to research and development. Further details and resources can be
found at https://github.com/FudanDISC/DISC-MedLLM
less
By: Yuxuan Hu, Jing Zhang, Chen Zhao, Cuiping Li, Hong Chen
We propose TCSP, a novel method for compressing a transformer model by
focusing on reducing the hidden size of the model. By projecting the whole
transform model into a subspace, we enable matrix operations between the weight
matrices in the model and features in a reduced-dimensional space, leading to
significant reductions in model parameters and computing resources. To
establish this subspace, we decompose the feature matrix, derived fro... more
We propose TCSP, a novel method for compressing a transformer model by
focusing on reducing the hidden size of the model. By projecting the whole
transform model into a subspace, we enable matrix operations between the weight
matrices in the model and features in a reduced-dimensional space, leading to
significant reductions in model parameters and computing resources. To
establish this subspace, we decompose the feature matrix, derived from
different layers of sampled data instances, into a projection matrix. For
evaluation, TCSP is applied to compress T5 and BERT models on the GLUE and
SQuAD benchmarks. Experimental results demonstrate that TCSP achieves a
compression ratio of 44\% with at most 1.6\% degradation in accuracy,
surpassing or matching prior compression methods. Furthermore, TCSP exhibits
compatibility with other methods targeting filter and attention head size
compression.
less
Foresight -- Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRs
2upvotes
By: Zeljko Kraljevic, Dan Bean, Anthony Shek, Rebecca Bendayan, Harry Hemingway, Joshua Au Yeung, Alexander Deng, Alfie Baston, Jack Ross, Esther Idowu, James T Teo, Richard J Dobson
Background: Electronic Health Records hold detailed longitudinal information
about each patient's health status and general clinical history, a large
portion of which is stored within the unstructured text. Existing approaches
focus mostly on structured data and a subset of single-domain outcomes. We
explore how temporal modelling of patients from free text and structured data,
using deep generative transformers can be used to forecast a wi... more
Background: Electronic Health Records hold detailed longitudinal information
about each patient's health status and general clinical history, a large
portion of which is stored within the unstructured text. Existing approaches
focus mostly on structured data and a subset of single-domain outcomes. We
explore how temporal modelling of patients from free text and structured data,
using deep generative transformers can be used to forecast a wide range of
future disorders, substances, procedures or findings. Methods: We present
Foresight, a novel transformer-based pipeline that uses named entity
recognition and linking tools to convert document text into structured, coded
concepts, followed by providing probabilistic forecasts for future medical
events such as disorders, substances, procedures and findings. We processed the
entire free-text portion from three different hospital datasets totalling
811336 patients covering both physical and mental health. Findings: On tests in
two UK hospitals (King's College Hospital, South London and Maudsley) and the
US MIMIC-III dataset precision@10 0.68, 0.76 and 0.88 was achieved for
forecasting the next disorder in a patient timeline, while precision@10 of
0.80, 0.81 and 0.91 was achieved for forecasting the next biomedical concept.
Foresight was also validated on 34 synthetic patient timelines by five
clinicians and achieved relevancy of 97% for the top forecasted candidate
disorder. As a generative model, it can forecast follow-on biomedical concepts
for as many steps as required. Interpretation: Foresight is a general-purpose
model for biomedical concept modelling that can be used for real-world risk
forecasting, virtual trials and clinical research to study the progression of
disorders, simulate interventions and counterfactuals, and educational
purposes.
less
By: Jorge Martinez-Gil
The issue of word sense ambiguity poses a significant challenge in natural
language processing due to the scarcity of annotated data to feed machine
learning models to face the challenge. Therefore, unsupervised word sense
disambiguation methods have been developed to overcome that challenge without
relying on annotated data. This research proposes a new context-aware approach
to unsupervised word sense disambiguation, which provides a flex... more
The issue of word sense ambiguity poses a significant challenge in natural
language processing due to the scarcity of annotated data to feed machine
learning models to face the challenge. Therefore, unsupervised word sense
disambiguation methods have been developed to overcome that challenge without
relying on annotated data. This research proposes a new context-aware approach
to unsupervised word sense disambiguation, which provides a flexible mechanism
for incorporating contextual information into the similarity measurement
process. We experiment with a popular benchmark dataset to evaluate the
proposed strategy and compare its performance with state-of-the-art
unsupervised word sense disambiguation techniques. The experimental results
indicate that our approach substantially enhances disambiguation accuracy and
surpasses the performance of several existing techniques. Our findings
underscore the significance of integrating contextual information in semantic
similarity measurements to manage word sense ambiguity in unsupervised
scenarios effectively.
less
By: Thibault Douzon, Stefan Duffner, Christophe Garcia, Jérémy Espinas
Since their release, Transformers have revolutionized many fields from
Natural Language Understanding to Computer Vision. Document Understanding (DU)
was not left behind with first Transformer based models for DU dating from late
2019. However, the computational complexity of the self-attention operation
limits their capabilities to small sequences. In this paper we explore multiple
strategies to apply Transformer based models to long multi... more
Since their release, Transformers have revolutionized many fields from
Natural Language Understanding to Computer Vision. Document Understanding (DU)
was not left behind with first Transformer based models for DU dating from late
2019. However, the computational complexity of the self-attention operation
limits their capabilities to small sequences. In this paper we explore multiple
strategies to apply Transformer based models to long multi-page documents. We
introduce 2 new multi-modal (text + layout) long-range models for DU. They are
based on efficient implementations of Transformers for long sequences.
Long-range models can process whole documents at once effectively and are less
impaired by the document's length. We compare them to LayoutLM, a classical
Transformer adapted for DU and pre-trained on millions of documents. We further
propose 2D relative attention bias to guide self-attention towards relevant
tokens without harming model efficiency. We observe improvements on multi-page
business documents on Information Retrieval for a small performance cost on
smaller sequences. Relative 2D attention revealed to be effective on dense text
for both normal and long-range models.
less
By: Kin Ian Lo University College London, London, UK, Mehrnoosh Sadrzadeh University College London, London, UK, Shane Mansfield Quandela, Paris, France
Ambiguities in natural language give rise to probability distributions over
interpretations. The distributions are often over multiple ambiguous words at a
time; a multiplicity which makes them a suitable topic for sheaf-theoretic
models of quantum contextuality. Previous research showed that different
quantitative measures of contextuality correlate well with Psycholinguistic
research on lexical ambiguities. In this work, we focus on coref... more
Ambiguities in natural language give rise to probability distributions over
interpretations. The distributions are often over multiple ambiguous words at a
time; a multiplicity which makes them a suitable topic for sheaf-theoretic
models of quantum contextuality. Previous research showed that different
quantitative measures of contextuality correlate well with Psycholinguistic
research on lexical ambiguities. In this work, we focus on coreference
ambiguities and investigate the Winograd Schema Challenge (WSC), a test
proposed by Levesque in 2011 to evaluate the intelligence of machines. The WSC
consists of a collection of multiple-choice questions that require
disambiguating pronouns in sentences structured according to the Winograd
schema, in a way that makes it difficult for machines to determine the correct
referents but remains intuitive for human comprehension. In this study, we
propose an approach that analogously models the Winograd schema as an
experiment in quantum physics. However, we argue that the original Winograd
Schema is inherently too simplistic to facilitate contextuality. We introduce a
novel mechanism for generalising the schema, rendering it analogous to a
Bell-CHSH measurement scenario. We report an instance of this generalised
schema, complemented by the human judgements we gathered via a crowdsourcing
platform. The resulting model violates the Bell-CHSH inequality by 0.192, thus
exhibiting contextuality in a coreference resolution setting.
less
By: Haochun Wang, Sendong Zhao, Zewen Qiang, Zijian Li, Nuwa Xi, Yanrui Du, MuZhen Cai, Haoqiang Guo, Yuhan Chen, Haoming Xu, Bing Qin, Ting Liu
Large Language Models (LLMs) have demonstrated remarkable success in diverse
natural language processing (NLP) tasks in general domains. However, LLMs
sometimes generate responses with the hallucination about medical facts due to
limited domain knowledge. Such shortcomings pose potential risks in the
utilization of LLMs within medical contexts. To address this challenge, we
propose knowledge-tuning, which leverages structured medical knowle... more
Large Language Models (LLMs) have demonstrated remarkable success in diverse
natural language processing (NLP) tasks in general domains. However, LLMs
sometimes generate responses with the hallucination about medical facts due to
limited domain knowledge. Such shortcomings pose potential risks in the
utilization of LLMs within medical contexts. To address this challenge, we
propose knowledge-tuning, which leverages structured medical knowledge bases
for the LLMs to grasp domain knowledge efficiently and facilitate reliable
response generation. We also release cMedKnowQA, a Chinese medical knowledge
question-answering dataset constructed from medical knowledge bases to assess
the medical knowledge proficiency of LLMs. Experimental results show that the
LLMs which are knowledge-tuned with cMedKnowQA, can exhibit higher levels of
accuracy in response generation compared with vanilla instruction-tuning and
offer a new reliable way for the domain adaptation of LLMs.
less
By: Roman Klinger
The term emotion analysis in text subsumes various natural language
processing tasks which have in common the goal to enable computers to
understand emotions. Most popular is emotion classification in which one or
multiple emotions are assigned to a predefined textual unit. While such setting
is appropriate to identify the reader's or author's emotion, emotion role
labeling adds the perspective of mentioned entities and extracts text spans
... more
The term emotion analysis in text subsumes various natural language
processing tasks which have in common the goal to enable computers to
understand emotions. Most popular is emotion classification in which one or
multiple emotions are assigned to a predefined textual unit. While such setting
is appropriate to identify the reader's or author's emotion, emotion role
labeling adds the perspective of mentioned entities and extracts text spans
that correspond to the emotion cause. The underlying emotion theories agree on
one important point; that an emotion is caused by some internal or external
event and comprises several subcomponents, including the subjective feeling and
a cognitive evaluation. We therefore argue that emotions and events are related
in two ways. (1) Emotions are events; and this perspective is the fundament in
NLP for emotion role labeling. (2) Emotions are caused by events; a perspective
that is made explicit with research how to incorporate psychological appraisal
theories in NLP models to interpret events. These two research directions, role
labeling and (event-focused) emotion classification, have by and large been
tackled separately. We contributed to both directions with the projects SEAT
(Structured Multi-Domain Emotion Analysis from Text) and CEAT (Computational
Event Evaluation based on Appraisal Theories for Emotion Analysis), both funded
by the German Research Foundation. In this paper, we consolidate the findings
and point out open research questions.
less
By: Jorge Martinez-Gil
This research presents ORUGA, a method that tries to automatically optimize
the readability of any text in English. The core idea behind the method is that
certain factors affect the readability of a text, some of which are
quantifiable (number of words, syllables, presence or absence of adverbs, and
so on). The nature of these factors allows us to implement a genetic learning
strategy to replace some existing words with their most suitable... more
This research presents ORUGA, a method that tries to automatically optimize
the readability of any text in English. The core idea behind the method is that
certain factors affect the readability of a text, some of which are
quantifiable (number of words, syllables, presence or absence of adverbs, and
so on). The nature of these factors allows us to implement a genetic learning
strategy to replace some existing words with their most suitable synonyms to
facilitate optimization. In addition, this research seeks to preserve both the
original text's content and form through multi-objective optimization
techniques. In this way, neither the text's syntactic structure nor the
semantic content of the original message is significantly distorted. An
exhaustive study on a substantial number and diversity of texts confirms that
our method was able to optimize the degree of readability in all cases without
significantly altering their form or meaning. The source code of this approach
is available at https://github.com/jorge-martinez-gil/oruga.
less
By: Rabindra Lamsal, Maria Rodriguez Read, Shanika Karunasekera
Social media platforms play an essential role in crisis communication, but
analyzing crisis-related social media texts is challenging due to their
informal nature. Transformer-based pre-trained models like BERT and RoBERTa
have shown success in various NLP tasks, but they are not tailored for
crisis-related texts. Furthermore, general-purpose sentence encoders are used
to generate sentence embeddings, regardless of the textual complexities ... more
Social media platforms play an essential role in crisis communication, but
analyzing crisis-related social media texts is challenging due to their
informal nature. Transformer-based pre-trained models like BERT and RoBERTa
have shown success in various NLP tasks, but they are not tailored for
crisis-related texts. Furthermore, general-purpose sentence encoders are used
to generate sentence embeddings, regardless of the textual complexities in
crisis-related texts. Advances in applications like text classification,
semantic search, and clustering contribute to effective processing of
crisis-related texts, which is essential for emergency responders to gain a
comprehensive view of a crisis event, whether historical or real-time. To
address these gaps in crisis informatics literature, this study introduces
CrisisTransformers, an ensemble of pre-trained language models and sentence
encoders trained on an extensive corpus of over 15 billion word tokens from
tweets associated with more than 30 crisis events, including disease outbreaks,
natural disasters, conflicts, and other critical incidents. We evaluate
existing models and CrisisTransformers on 18 crisis-specific public datasets.
Our pre-trained models outperform strong baselines across all datasets in
classification tasks, and our best-performing sentence encoder improves the
state-of-the-art by 17.43% in sentence encoding tasks. Additionally, we
investigate the impact of model initialization on convergence and evaluate the
significance of domain-specific models in generating semantically meaningful
sentence embeddings. All models are publicly released
(https://huggingface.co/crisistransformers), with the anticipation that they
will serve as a robust baseline for tasks involving the analysis of
crisis-related social media texts.
less
By: Tiezheng Yu, Ziwei Ji, Pascale Fung
Query-Focused Meeting Summarization (QFMS) aims to generate a summary of a
given meeting transcript conditioned upon a query. The main challenges for QFMS
are the long input text length and sparse query-relevant information in the
meeting transcript. In this paper, we propose a knowledge-enhanced two-stage
framework called Knowledge-Aware Summarizer (KAS) to tackle the challenges. In
the first stage, we introduce knowledge-aware scores to i... more
Query-Focused Meeting Summarization (QFMS) aims to generate a summary of a
given meeting transcript conditioned upon a query. The main challenges for QFMS
are the long input text length and sparse query-relevant information in the
meeting transcript. In this paper, we propose a knowledge-enhanced two-stage
framework called Knowledge-Aware Summarizer (KAS) to tackle the challenges. In
the first stage, we introduce knowledge-aware scores to improve the
query-relevant segment extraction. In the second stage, we incorporate
query-relevant knowledge in the summary generation. Experimental results on the
QMSum dataset show that our approach achieves state-of-the-art performance.
Further analysis proves the competency of our methods in generating relevant
and faithful summaries.
less
By: Damian Hodel, Jevin West
In their recent Nature Human Behaviour paper, "Emergent analogical reasoning
in large language models," (Webb, Holyoak, and Lu, 2023) the authors argue that
"large language models such as GPT-3 have acquired an emergent ability to find
zero-shot solutions to a broad range of analogy problems." In this response, we
provide counterexamples of the letter string analogies. In our tests, GPT-3
fails to solve even the easiest variants of the prob... more
In their recent Nature Human Behaviour paper, "Emergent analogical reasoning
in large language models," (Webb, Holyoak, and Lu, 2023) the authors argue that
"large language models such as GPT-3 have acquired an emergent ability to find
zero-shot solutions to a broad range of analogy problems." In this response, we
provide counterexamples of the letter string analogies. In our tests, GPT-3
fails to solve even the easiest variants of the problems presented in the
original paper. Zero-shot reasoning is an extraordinary claim that requires
extraordinary evidence. We do not see that evidence in our experiments. To
strengthen claims of humanlike reasoning such as zero-shot reasoning, it is
important that the field develop approaches that rule out data memorization.
less