Artificial IntelligenceComputational Engineering, Finance, and ScienceComputation and LanguageComputers and SocietyComputer Vision and Pattern RecognitionCryptography and SecurityDigital Libraries
Large Language Models
0upvotes
By: Michael R. Douglas
Artificial intelligence is making spectacular progress, and one of the best
examples is the development of large language models (LLMs) such as OpenAI's
GPT series. In these lectures, written for readers with a background in
mathematics or physics, we give a brief history and survey of the state of the
art, and describe the underlying transformer architecture in detail. We then
explore some current ideas on how LLMs work and how models trai... more
Artificial intelligence is making spectacular progress, and one of the best
examples is the development of large language models (LLMs) such as OpenAI's
GPT series. In these lectures, written for readers with a background in
mathematics or physics, we give a brief history and survey of the state of the
art, and describe the underlying transformer architecture in detail. We then
explore some current ideas on how LLMs work and how models trained to predict
the next word in a text are able to perform other tasks displaying
intelligence.
less
By: Ashish Singh, Antonio Bevilacqua, Timilehin B. Aderinola, Thach Le Nguyen, Darragh Whelan, Martin O'Reilly, Brian Caulfield, Georgiana Ifrim
Wearable sensors such as Inertial Measurement Units (IMUs) are often used to
assess the performance of human exercise. Common approaches use handcrafted
features based on domain expertise or automatically extracted features using
time series analysis. Multiple sensors are required to achieve high
classification accuracy, which is not very practical. These sensors require
calibration and synchronization and may lead to discomfort over longer... more
Wearable sensors such as Inertial Measurement Units (IMUs) are often used to
assess the performance of human exercise. Common approaches use handcrafted
features based on domain expertise or automatically extracted features using
time series analysis. Multiple sensors are required to achieve high
classification accuracy, which is not very practical. These sensors require
calibration and synchronization and may lead to discomfort over longer time
periods. Recent work utilizing computer vision techniques has shown similar
performance using video, without the need for manual feature engineering, and
avoiding some pitfalls such as sensor calibration and placement on the body. In
this paper, we compare the performance of IMUs to a video-based approach for
human exercise classification on two real-world datasets consisting of Military
Press and Rowing exercises. We compare the performance using a single camera
that captures video in the frontal view versus using 5 IMUs placed on different
parts of the body. We observe that an approach based on a single camera can
outperform a single IMU by 10 percentage points on average. Additionally, a
minimum of 3 IMUs are required to outperform a single camera. We observe that
working with the raw data using multivariate time series classifiers
outperforms traditional approaches based on handcrafted or automatically
extracted features. Finally, we show that an ensemble model combining the data
from a single camera with a single IMU outperforms either data modality. Our
work opens up new and more realistic avenues for this application, where a
video captured using a readily available smartphone camera, combined with a
single sensor, can be used for effective human exercise classification.
less
By: Lingjiao Chen, Matei Zaharia, James Zou
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM)
services. However, when and how these models are updated over time is opaque.
Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on
several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3)
opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating
code, 6) US Medical License tests, and 7) visual reasoning.... more
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM)
services. However, when and how these models are updated over time is opaque.
Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on
several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3)
opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating
code, 6) US Medical License tests, and 7) visual reasoning. We find that the
performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time.
For example, GPT-4 (March 2023) was reasonable at identifying prime vs.
composite numbers (84% accuracy) but GPT-4 (June 2023) was poor on these same
questions (51% accuracy). This is partly explained by a drop in GPT-4's amenity
to follow chain-of-thought prompting. Interestingly, GPT-3.5 was much better in
June than in March in this task. GPT-4 became less willing to answer sensitive
questions and opinion survey questions in June than in March. GPT-4 performed
better at multi-hop questions in June than in March, while GPT-3.5's
performance dropped on this task. Both GPT-4 and GPT-3.5 had more formatting
mistakes in code generation in June than in March. Overall, our findings show
that the behavior of the "same" LLM service can change substantially in a
relatively short amount of time, highlighting the need for continuous
monitoring of LLMs.
less
By: M. Andrecut
Predicting the dynamics of chaotic systems is one of the most challenging
tasks for neural networks, and machine learning in general. Here we aim to
predict the spatiotemporal chaotic dynamics of a high-dimensional non-linear
system. In our attempt we use the TensorFlow library, representing the state of
the art for deep neural networks training and prediction. While our results are
encouraging, and show that the dynamics of the considered ... more
Predicting the dynamics of chaotic systems is one of the most challenging
tasks for neural networks, and machine learning in general. Here we aim to
predict the spatiotemporal chaotic dynamics of a high-dimensional non-linear
system. In our attempt we use the TensorFlow library, representing the state of
the art for deep neural networks training and prediction. While our results are
encouraging, and show that the dynamics of the considered system can be
predicted for short time, we also indirectly discovered an unexpected and
undesirable behavior of the TensorFlow library. More specifically, the longer
term prediction of the system's chaotic behavior quickly deteriorates and blows
up due to the nondeterministic behavior of the TensorFlow library. Here we
provide numerical evidence of the short time prediction ability, and of the
longer term predictability blow up.
less
By: Lianke Qin, Zhao Song, Baocheng Sun
A rising trend in theoretical deep learning is to understand why deep
learning works through Neural Tangent Kernel (NTK) [jgh18], a kernel method
that is equivalent to using gradient descent to train a multi-layer
infinitely-wide neural network. NTK is a major step forward in the theoretical
deep learning because it allows researchers to use traditional mathematical
tools to analyze properties of deep neural networks and to explain various
... more
A rising trend in theoretical deep learning is to understand why deep
learning works through Neural Tangent Kernel (NTK) [jgh18], a kernel method
that is equivalent to using gradient descent to train a multi-layer
infinitely-wide neural network. NTK is a major step forward in the theoretical
deep learning because it allows researchers to use traditional mathematical
tools to analyze properties of deep neural networks and to explain various
neural network techniques from a theoretical view. A natural extension of NTK
on graph learning is \textit{Graph Neural Tangent Kernel (GNTK)}, and
researchers have already provide GNTK formulation for graph-level regression
and show empirically that this kernel method can achieve similar accuracy as
GNNs on various bioinformatics datasets [dhs+19]. The remaining question now is
whether solving GNTK regression is equivalent to training an infinite-wide
multi-layer GNN using gradient descent. In this paper, we provide three new
theoretical results. First, we formally prove this equivalence for graph-level
regression. Second, we present the first GNTK formulation for node-level
regression. Finally, we prove the equivalence for node-level regression.
less
By: Paul M Baggenstoss
In this paper, we exploit the unique properties of a deterministic projected
belief network (D-PBN) to take full advantage of trainable compound activation
functions (TCAs). A D-PBN is a type of auto-encoder that operates by "backing
up" through a feed-forward neural network. TCAs are activation functions with
complex monotonic-increasing shapes that change the distribution of the data so
that the linear transformation that follows is more ... more
In this paper, we exploit the unique properties of a deterministic projected
belief network (D-PBN) to take full advantage of trainable compound activation
functions (TCAs). A D-PBN is a type of auto-encoder that operates by "backing
up" through a feed-forward neural network. TCAs are activation functions with
complex monotonic-increasing shapes that change the distribution of the data so
that the linear transformation that follows is more effective. Because a D-PBN
operates by "backing up", the TCAs are inverted in the reconstruction process,
restoring the original distribution of the data, thus taking advantage of a
given TCA in both analysis and reconstruction. In this paper, we show that a
D-PBN auto-encoder with TCAs can significantly out-perform standard
auto-encoders including variational auto-encoders.
less
By: Pierre Gaillard Thoth, Sébastien Gerchinovitz IMT, Étienne de Montbrun TSE-R
We study the classical problem of approximating a non-decreasing function $f:
\mathcal{X} \to \mathcal{Y}$ in $L^p(\mu)$ norm by sequentially querying its
values, for known compact real intervals $\mathcal{X}$, $\mathcal{Y}$ and a
known probability measure $\mu$ on $\cX$. For any function~$f$ we characterize
the minimum number of evaluations of $f$ that algorithms need to guarantee an
approximation $\hat{f}$ with an $L^p(\mu)$ error below $... more
We study the classical problem of approximating a non-decreasing function $f:
\mathcal{X} \to \mathcal{Y}$ in $L^p(\mu)$ norm by sequentially querying its
values, for known compact real intervals $\mathcal{X}$, $\mathcal{Y}$ and a
known probability measure $\mu$ on $\cX$. For any function~$f$ we characterize
the minimum number of evaluations of $f$ that algorithms need to guarantee an
approximation $\hat{f}$ with an $L^p(\mu)$ error below $\epsilon$ after
stopping. Unlike worst-case results that hold uniformly over all $f$, our
complexity measure is dependent on each specific function $f$. To address this
problem, we introduce GreedyBox, a generalization of an algorithm originally
proposed by Novak (1992) for numerical integration. We prove that GreedyBox
achieves an optimal sample complexity for any function $f$, up to logarithmic
factors. Additionally, we uncover results regarding piecewise-smooth functions.
Perhaps as expected, the $L^p(\mu)$ error of GreedyBox decreases much faster
for piecewise-$C^2$ functions than predicted by the algorithm (without any
knowledge on the smoothness of $f$). A simple modification even achieves
optimal minimax approximation rates for such functions, which we compute
explicitly. In particular, our findings highlight multiple performance gaps
between adaptive and non-adaptive algorithms, smooth and piecewise-smooth
functions, as well as monotone or non-monotone functions. Finally, we provide
numerical experiments to support our theoretical results.
less
By: Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, Haoxing Ren
The increasing popularity of large language models (LLMs) has paved the way
for their application in diverse domains. This paper proposes a benchmarking
framework tailored specifically for evaluating LLM performance in the context
of Verilog code generation for hardware design and verification. We present a
comprehensive evaluation dataset consisting of 156 problems from the Verilog
instructional website HDLBits. The evaluation set consists... more
The increasing popularity of large language models (LLMs) has paved the way
for their application in diverse domains. This paper proposes a benchmarking
framework tailored specifically for evaluating LLM performance in the context
of Verilog code generation for hardware design and verification. We present a
comprehensive evaluation dataset consisting of 156 problems from the Verilog
instructional website HDLBits. The evaluation set consists of a diverse set of
Verilog code generation tasks, ranging from simple combinational circuits to
complex finite state machines. The Verilog code completions can be
automatically tested for functional correctness by comparing the transient
simulation outputs of the generated design with a golden solution. We also
demonstrate that the Verilog code generation capability of pretrained language
models could be improved with supervised fine-tuning by bootstrapping with LLM
generated synthetic problem-code pairs.
less
By: Cristina Pinneri, Sarah Bechtle, Markus Wulfmeier, Arunkumar Byravan, Jingwei Zhang, William F. Whitney, Martin Riedmiller
We present a novel approach to address the challenge of generalization in
offline reinforcement learning (RL), where the agent learns from a fixed
dataset without any additional interaction with the environment. Specifically,
we aim to improve the agent's ability to generalize to out-of-distribution
goals. To achieve this, we propose to learn a dynamics model and check if it is
equivariant with respect to a fixed type of transformation, nam... more
We present a novel approach to address the challenge of generalization in
offline reinforcement learning (RL), where the agent learns from a fixed
dataset without any additional interaction with the environment. Specifically,
we aim to improve the agent's ability to generalize to out-of-distribution
goals. To achieve this, we propose to learn a dynamics model and check if it is
equivariant with respect to a fixed type of transformation, namely translations
in the state space. We then use an entropy regularizer to increase the
equivariant set and augment the dataset with the resulting transformed samples.
Finally, we learn a new policy offline based on the augmented dataset, with an
off-the-shelf offline RL algorithm. Our experimental results demonstrate that
our approach can greatly improve the test performance of the policy on the
considered environments.
less
By: Ahmad Chamma Inria Universite Paris Saclay CEA, Denis A. Engemann Roche Pharma Research and Early Development, Neuroscience and Rare Diseases, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland, Bertrand Thirion Inria Universite Paris Saclay CEA
Variable importance assessment has become a crucial step in machine-learning
applications when using complex learners, such as deep neural networks, on
large-scale data. Removal-based importance assessment is currently the
reference approach, particularly when statistical guarantees are sought to
justify variable inclusion. It is often implemented with variable permutation
schemes. On the flip side, these approaches risk misidentifying unim... more
Variable importance assessment has become a crucial step in machine-learning
applications when using complex learners, such as deep neural networks, on
large-scale data. Removal-based importance assessment is currently the
reference approach, particularly when statistical guarantees are sought to
justify variable inclusion. It is often implemented with variable permutation
schemes. On the flip side, these approaches risk misidentifying unimportant
variables as important in the presence of correlations among covariates. Here
we develop a systematic approach for studying Conditional Permutation
Importance (CPI) that is model agnostic and computationally lean, as well as
reusable benchmarks of state-of-the-art variable importance estimators. We show
theoretically and empirically that $\textit{CPI}$ overcomes the limitations of
standard permutation importance by providing accurate type-I error control.
When used with a deep neural network, $\textit{CPI}$ consistently showed top
accuracy across benchmarks. An empirical benchmark on real-world data analysis
in a large-scale medical dataset showed that $\textit{CPI}$ provides a more
parsimonious selection of statistically significant variables. Our results
suggest that $\textit{CPI}$ can be readily used as drop-in replacement for
permutation-based methods.
less
By: Nafis Sajid, Md Rashidul Hasan, Muhammad Ibrahim
Community question answering (CQA) forums are Internet-based platforms where
users ask questions about a topic and other expert users try to provide
solutions. Many CQA forums such as Quora, Stackoverflow, Yahoo!Answer,
StackExchange exist with a lot of user-generated data. These data are leveraged
in automated CQA ranking systems where similar questions (and answers) are
presented in response to the query of the user. In this work, we empi... more
Community question answering (CQA) forums are Internet-based platforms where
users ask questions about a topic and other expert users try to provide
solutions. Many CQA forums such as Quora, Stackoverflow, Yahoo!Answer,
StackExchange exist with a lot of user-generated data. These data are leveraged
in automated CQA ranking systems where similar questions (and answers) are
presented in response to the query of the user. In this work, we empirically
investigate a few aspects of this domain. Firstly, in addition to traditional
features like TF-IDF, BM25 etc., we introduce a BERT-based feature that
captures the semantic similarity between the question and answer. Secondly,
most of the existing research works have focused on features extracted only
from the question part; features extracted from answers have not been explored
extensively. We combine both types of features in a linear fashion. Thirdly,
using our proposed concepts, we conduct an empirical investigation with
different rank-learning algorithms, some of which have not been used so far in
CQA domain. On three standard CQA datasets, our proposed framework achieves
state-of-the-art performance. We also analyze importance of the features we use
in our investigation. This work is expected to guide the practitioners to
select a better set of features for the CQA retrieval task.
less
Multi-Source Domain Adaptation meets Dataset Distillation through Dataset Dictionary Learning
0upvotes
By: Eduardo Fernandes Montesuma, Fred Ngolè Mboula, Antoine Souloumiac
In this paper, we consider the intersection of two problems in machine
learning: Multi-Source Domain Adaptation (MSDA) and Dataset Distillation (DD).
On the one hand, the first considers adapting multiple heterogeneous labeled
source domains to an unlabeled target domain. On the other hand, the second
attacks the problem of synthesizing a small summary containing all the
information about the datasets. We thus consider a new problem called ... more
In this paper, we consider the intersection of two problems in machine
learning: Multi-Source Domain Adaptation (MSDA) and Dataset Distillation (DD).
On the one hand, the first considers adapting multiple heterogeneous labeled
source domains to an unlabeled target domain. On the other hand, the second
attacks the problem of synthesizing a small summary containing all the
information about the datasets. We thus consider a new problem called MSDA-DD.
To solve it, we adapt previous works in the MSDA literature, such as
Wasserstein Barycenter Transport and Dataset Dictionary Learning, as well as DD
method Distribution Matching. We thoroughly experiment with this novel problem
on four benchmarks (Caltech-Office 10, Tennessee-Eastman Process, Continuous
Stirred Tank Reactor, and Case Western Reserve University), where we show that,
even with as little as 1 sample per class, one achieves state-of-the-art
adaptation performance.
less
By: Mengge Du, Longfeng Nie, Siyu Lou, Yuntian Chenc, Dongxiao Zhang
Unveiling the underlying governing equations of nonlinear dynamic systems
remains a significant challenge, especially when encountering noisy
observations and no prior knowledge available. This study proposes R-DISCOVER,
a framework designed to robustly uncover open-form partial differential
equations (PDEs) from limited and noisy data. The framework operates through
two alternating update processes: discovering and embedding. The discoveri... more
Unveiling the underlying governing equations of nonlinear dynamic systems
remains a significant challenge, especially when encountering noisy
observations and no prior knowledge available. This study proposes R-DISCOVER,
a framework designed to robustly uncover open-form partial differential
equations (PDEs) from limited and noisy data. The framework operates through
two alternating update processes: discovering and embedding. The discovering
phase employs symbolic representation and a reinforcement learning (RL)-guided
hybrid PDE generator to efficiently produce diverse open-form PDEs with tree
structures. A neural network-based predictive model fits the system response
and serves as the reward evaluator for the generated PDEs. PDEs with superior
fits are utilized to iteratively optimize the generator via the RL method and
the best-performing PDE is selected by a parameter-free stability metric. The
embedding phase integrates the initially identified PDE from the discovering
process as a physical constraint into the predictive model for robust training.
The traversal of PDE trees automates the construction of the computational
graph and the embedding process without human intervention. Numerical
experiments demonstrate our framework's capability to uncover governing
equations from nonlinear dynamic systems with limited and highly noisy data and
outperform other physics-informed neural network-based discovery methods. This
work opens new potential for exploring real-world systems with limited
understanding.
less
Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis
0upvotes
By: Mehdi Zadem, Sergio Mover, Sao Mai Nguyen
Open-ended learning benefits immensely from the use of symbolic methods for
goal representation as they offer ways to structure knowledge for efficient and
transferable learning. However, the existing Hierarchical Reinforcement
Learning (HRL) approaches relying on symbolic reasoning are often limited as
they require a manual goal representation. The challenge in autonomously
discovering a symbolic goal representation is that it must preserv... more
Open-ended learning benefits immensely from the use of symbolic methods for
goal representation as they offer ways to structure knowledge for efficient and
transferable learning. However, the existing Hierarchical Reinforcement
Learning (HRL) approaches relying on symbolic reasoning are often limited as
they require a manual goal representation. The challenge in autonomously
discovering a symbolic goal representation is that it must preserve critical
information, such as the environment dynamics. In this paper, we propose a
developmental mechanism for goal discovery via an emergent representation that
abstracts (i.e., groups together) sets of environment states that have similar
roles in the task. We introduce a Feudal HRL algorithm that concurrently learns
both the goal representation and a hierarchical policy. The algorithm uses
symbolic reachability analysis for neural networks to approximate the
transition relation among sets of states and to refine the goal representation.
We evaluate our approach on complex navigation tasks, showing the learned
representation is interpretable, transferrable and results in data efficient
learning.
less
deepFDEnet: A Novel Neural Network Architecture for Solving Fractional Differential Equations
0upvotes
By: Ali Nosrati Firoozsalari, Hassan Dana Mazraeh, Alireza Afzal Aghaei, Kourosh Parand
The primary goal of this research is to propose a novel architecture for a
deep neural network that can solve fractional differential equations
accurately. A Gaussian integration rule and a $L_1$ discretization technique
are used in the proposed design. In each equation, a deep neural network is
used to approximate the unknown function. Three forms of fractional
differential equations have been examined to highlight the method's
versatility... more
The primary goal of this research is to propose a novel architecture for a
deep neural network that can solve fractional differential equations
accurately. A Gaussian integration rule and a $L_1$ discretization technique
are used in the proposed design. In each equation, a deep neural network is
used to approximate the unknown function. Three forms of fractional
differential equations have been examined to highlight the method's
versatility: a fractional ordinary differential equation, a fractional order
integrodifferential equation, and a fractional order partial differential
equation. The results show that the proposed architecture solves different
forms of fractional differential equations with excellent precision.
less
By: Francisco Nunes Ferreira Quialheiro Simoes, Mehdi Dastani, Thijs van Ommen
Artificial intelligence models and methods commonly lack causal
interpretability. Despite the advancements in interpretable machine learning
(IML) methods, they frequently assign importance to features which lack causal
influence on the outcome variable. Selecting causally relevant features among
those identified as relevant by these methods, or even before model training,
would offer a solution. Feature selection methods utilizing informat... more
Artificial intelligence models and methods commonly lack causal
interpretability. Despite the advancements in interpretable machine learning
(IML) methods, they frequently assign importance to features which lack causal
influence on the outcome variable. Selecting causally relevant features among
those identified as relevant by these methods, or even before model training,
would offer a solution. Feature selection methods utilizing information
theoretical quantities have been successful in identifying statistically
relevant features. However, the information theoretical quantities they are
based on do not incorporate causality, rendering them unsuitable for such
scenarios. To address this challenge, this article proposes information
theoretical quantities that incorporate the causal structure of the system,
which can be used to evaluate causal importance of features for some given
outcome variable. Specifically, we introduce causal versions of entropy and
mutual information, termed causal entropy and causal information gain, which
are designed to assess how much control a feature provides over the outcome
variable. These newly defined quantities capture changes in the entropy of a
variable resulting from interventions on other variables. Fundamental results
connecting these quantities to the existence of causal effects are derived. The
use of causal information gain in feature selection is demonstrated,
highlighting its superiority over standard mutual information in revealing
which features provide control over a chosen outcome variable. Our
investigation paves the way for the development of methods with improved
interpretability in domains involving causation.
less
By: Haochong Xia, Shuo Sun, Xinrun Wang, Bo An
Financial simulators play an important role in enhancing forecasting
accuracy, managing risks, and fostering strategic financial decision-making.
Despite the development of financial market simulation methodologies, existing
frameworks often struggle with adapting to specialized simulation context. We
pinpoint the challenges as i) current financial datasets do not contain context
labels; ii) current techniques are not designed to generate f... more
Financial simulators play an important role in enhancing forecasting
accuracy, managing risks, and fostering strategic financial decision-making.
Despite the development of financial market simulation methodologies, existing
frameworks often struggle with adapting to specialized simulation context. We
pinpoint the challenges as i) current financial datasets do not contain context
labels; ii) current techniques are not designed to generate financial data with
context as control, which demands greater precision compared to other
modalities; iii) the inherent difficulties in generating context-aligned,
high-fidelity data given the non-stationary, noisy nature of financial data. To
address these challenges, our contributions are: i) we proposed the Contextual
Market Dataset with market dynamics, stock ticker, and history state as
context, leveraging a market dynamics modeling method that combines linear
regression and Dynamic Time Warping clustering to extract market dynamics; ii)
we present Market-GAN, a novel architecture incorporating a Generative
Adversarial Networks (GAN) for the controllable generation with context, an
autoencoder for learning low-dimension features, and supervisors for knowledge
transfer; iii) we introduce a two-stage training scheme to ensure that
Market-GAN captures the intrinsic market distribution with multiple objectives.
In the pertaining stage, with the use of the autoencoder and supervisors, we
prepare the generator with a better initialization for the adversarial training
stage. We propose a set of holistic evaluation metrics that consider alignment,
fidelity, data usability on downstream tasks, and market facts. We evaluate
Market-GAN with the Dow Jones Industrial Average data from 2000 to 2023 and
showcase superior performance in comparison to 4 state-of-the-art time-series
generative models.
less
By: Marcos Eduardo Valle
Despite the many successful applications of deep learning models for
multidimensional signal and image processing, most traditional neural networks
process data represented by (multidimensional) arrays of real numbers. The
intercorrelation between feature channels is usually expected to be learned
from the training data, requiring numerous parameters and careful training. In
contrast, vector-valued neural networks are conceived to process a... more
Despite the many successful applications of deep learning models for
multidimensional signal and image processing, most traditional neural networks
process data represented by (multidimensional) arrays of real numbers. The
intercorrelation between feature channels is usually expected to be learned
from the training data, requiring numerous parameters and careful training. In
contrast, vector-valued neural networks are conceived to process arrays of
vectors and naturally consider the intercorrelation between feature channels.
Consequently, they usually have fewer parameters and often undergo more robust
training than traditional neural networks. This paper aims to present a broad
framework for vector-valued neural networks, referred to as V-nets. In this
context, hypercomplex-valued neural networks are regarded as vector-valued
models with additional algebraic properties. Furthermore, this paper explains
the relationship between vector-valued and traditional neural networks.
Precisely, a vector-valued neural network can be obtained by placing
restrictions on a real-valued model to consider the intercorrelation between
feature channels. Finally, we show how V-nets, including hypercomplex-valued
neural networks, can be implemented in current deep-learning libraries as
real-valued networks.
less
By: Emanuele Marconato, Andrea Passerini, Stefano Teso
Focus in Explainable AI is shifting from explanations defined in terms of
low-level elements, such as input features, to explanations encoded in terms of
interpretable concepts learned from data. How to reliably acquire such concepts
is, however, still fundamentally unclear. An agreed-upon notion of concept
interpretability is missing, with the result that concepts used by both
post-hoc explainers and concept-based neural networks are acqui... more
Focus in Explainable AI is shifting from explanations defined in terms of
low-level elements, such as input features, to explanations encoded in terms of
interpretable concepts learned from data. How to reliably acquire such concepts
is, however, still fundamentally unclear. An agreed-upon notion of concept
interpretability is missing, with the result that concepts used by both
post-hoc explainers and concept-based neural networks are acquired through a
variety of mutually incompatible strategies. Critically, most of these neglect
the human side of the problem: a representation is understandable only insofar
as it can be understood by the human at the receiving end. The key challenge in
Human-interpretable Representation Learning (HRL) is how to model and
operationalize this human element. In this work, we propose a mathematical
framework for acquiring interpretable representations suitable for both
post-hoc explainers and concept-based neural networks. Our formalization of HRL
builds on recent advances in causal representation learning and explicitly
models a human stakeholder as an external observer. This allows us to derive a
principled notion of alignment between the machine representation and the
vocabulary of concepts understood by the human. In doing so, we link alignment
and interpretability through a simple and intuitive name transfer game, and
clarify the relationship between alignment and a well-known property of
representations, namely disentanglment. We also show that alignment is linked
to the issue of undesirable correlations among concepts, also known as concept
leakage, and to content-style separation, all through a general
information-theoretic reformulation of these properties. Our conceptualization
aims to bridge the gap between the human and algorithmic sides of
interpretability and establish a stepping stone for new research on
human-interpretable representations.
less
Beta Diffusion
0upvotes
By: Mingyuan Zhou, Tianqi Chen, Zhendong Wang, Huangjie Zheng
We introduce beta diffusion, a novel generative modeling method that
integrates demasking and denoising to generate data within bounded ranges.
Using scaled and shifted beta distributions, beta diffusion utilizes
multiplicative transitions over time to create both forward and reverse
diffusion processes, maintaining beta distributions in both the forward
marginals and the reverse conditionals, given the data at any point in time.
Unlike tra... more
We introduce beta diffusion, a novel generative modeling method that
integrates demasking and denoising to generate data within bounded ranges.
Using scaled and shifted beta distributions, beta diffusion utilizes
multiplicative transitions over time to create both forward and reverse
diffusion processes, maintaining beta distributions in both the forward
marginals and the reverse conditionals, given the data at any point in time.
Unlike traditional diffusion-based generative models relying on additive
Gaussian noise and reweighted evidence lower bounds (ELBOs), beta diffusion is
multiplicative and optimized with KL-divergence upper bounds (KLUBs) derived
from the convexity of the KL divergence. We demonstrate that the proposed KLUBs
are more effective for optimizing beta diffusion compared to negative ELBOs,
which can also be derived as the KLUBs of the same KL divergence with its two
arguments swapped. The loss function of beta diffusion, expressed in terms of
Bregman divergence, further supports the efficacy of KLUBs for optimization.
Experimental results on both synthetic data and natural images demonstrate the
unique capabilities of beta diffusion in generative modeling of range-bounded
data and validate the effectiveness of KLUBs in optimizing diffusion models,
thereby making them valuable additions to the family of diffusion-based
generative models and the optimization techniques used to train them.
less
Some notes concerning a generalized KMM-type optimization method for density ratio estimation
0upvotes
By: Cristian Daniel Alecsa
In the present paper we introduce new optimization algorithms for the task of
density ratio estimation. More precisely, we consider extending the well-known
KMM method using the construction of a suitable loss function, in order to
encompass more general situations involving the estimation of density ratio
with respect to subsets of the training data and test data, respectively. The
associated codes can be found at https://github.com/CDAlec... more
In the present paper we introduce new optimization algorithms for the task of
density ratio estimation. More precisely, we consider extending the well-known
KMM method using the construction of a suitable loss function, in order to
encompass more general situations involving the estimation of density ratio
with respect to subsets of the training data and test data, respectively. The
associated codes can be found at https://github.com/CDAlecsa/Generalized-KMM.
less
By: Rüdiger Brecht, Dmytro R. Popovych, Alex Bihlo, Roman O. Popovych
Current physics-informed (standard or operator) neural networks still rely on
accurately learning the initial conditions of the system they are solving. In
contrast, standard numerical methods evolve such initial conditions without
needing to learn these. In this study, we propose to improve current
physics-informed deep learning strategies such that initial conditions do not
need to be learned and are represented exactly in the predicted s... more
Current physics-informed (standard or operator) neural networks still rely on
accurately learning the initial conditions of the system they are solving. In
contrast, standard numerical methods evolve such initial conditions without
needing to learn these. In this study, we propose to improve current
physics-informed deep learning strategies such that initial conditions do not
need to be learned and are represented exactly in the predicted solution.
Moreover, this method guarantees that when a DeepONet is applied multiple times
to time step a solution, the resulting function is continuous.
less
By: Tilahun M. Getu, Georges Kaddoum
Although deep learning (DL) has led to several breakthroughs in many
disciplines as diverse as chemistry, computer science, electrical engineering,
mathematics, medicine, neuroscience, and physics, a comprehensive understanding
of why and how DL is empirically successful remains fundamentally elusive. To
attack this fundamental problem and unravel the mysteries behind DL's empirical
successes, significant innovations toward a unified theory... more
Although deep learning (DL) has led to several breakthroughs in many
disciplines as diverse as chemistry, computer science, electrical engineering,
mathematics, medicine, neuroscience, and physics, a comprehensive understanding
of why and how DL is empirically successful remains fundamentally elusive. To
attack this fundamental problem and unravel the mysteries behind DL's empirical
successes, significant innovations toward a unified theory of DL have been
made. These innovations encompass nearly fundamental advances in optimization,
generalization, and approximation. Despite these advances, however, no work to
date has offered a way to quantify the testing performance of a DL-based
algorithm employed to solve a pattern classification problem. To overcome this
fundamental challenge in part, this paper exposes the fundamental testing
performance limits of DL-based binary classifiers trained with hinge loss. For
binary classifiers that are based on deep rectified linear unit (ReLU)
feedforward neural networks (FNNs) and ones that are based on deep FNNs with
ReLU and Tanh activation, we derive their respective novel asymptotic testing
performance limits. The derived testing performance limits are validated by
extensive computer experiments.
less
By: Hao Mei, Junxian Li, Zhiming Liang, Guanjie Zheng, Bin Shi, Hua Wei
Traffic prediction is a crucial topic because of its broad scope of
applications in the transportation domain. Recently, various studies have
achieved promising results. However, most studies assume the prediction
locations have complete or at least partial historical records and cannot be
extended to non-historical recorded locations. In real-life scenarios, the
deployment of sensors could be limited due to budget limitations and
installat... more
Traffic prediction is a crucial topic because of its broad scope of
applications in the transportation domain. Recently, various studies have
achieved promising results. However, most studies assume the prediction
locations have complete or at least partial historical records and cannot be
extended to non-historical recorded locations. In real-life scenarios, the
deployment of sensors could be limited due to budget limitations and
installation availability, which makes most current models not applicable.
Though few pieces of literature tried to impute traffic states at the missing
locations, these methods need the data simultaneously observed at the locations
with sensors, making them not applicable to prediction tasks. Another drawback
is the lack of measurement of uncertainty in prediction, making prior works
unsuitable for risk-sensitive tasks or involving decision-making. To fill the
gap, inspired by the previous inductive graph neural network, this work
proposed an uncertainty-aware framework with the ability to 1) extend
prediction to missing locations with no historical records and significantly
extend spatial coverage of prediction locations while reducing deployment of
sensors and 2) generate probabilistic prediction with uncertainty
quantification to help the management of risk and decision making in the
down-stream tasks. Through extensive experiments on real-life datasets, the
result shows our method achieved promising results on prediction tasks, and the
uncertainty quantification gives consistent results which highly correlated
with the locations with and without historical data. We also show that our
model could help support sensor deployment tasks in the transportation field to
achieve higher accuracy with a limited sensor deployment budget.
less
By: Qianyu Long, Christos Anagnostopoulos, Shameem Puthiya Parambath, Daning Bi
Federated Learning (FL) has been successfully adopted for distributed
training and inference of large-scale Deep Neural Networks (DNNs). However,
DNNs are characterized by an extremely large number of parameters, thus,
yielding significant challenges in exchanging these parameters among
distributed nodes and managing the memory. Although recent DNN compression
methods (e.g., sparsification, pruning) tackle such challenges, they do not
holis... more
Federated Learning (FL) has been successfully adopted for distributed
training and inference of large-scale Deep Neural Networks (DNNs). However,
DNNs are characterized by an extremely large number of parameters, thus,
yielding significant challenges in exchanging these parameters among
distributed nodes and managing the memory. Although recent DNN compression
methods (e.g., sparsification, pruning) tackle such challenges, they do not
holistically consider an adaptively controlled reduction of parameter exchange
while maintaining high accuracy levels. We, therefore, contribute with a novel
FL framework (coined FedDIP), which combines (i) dynamic model pruning with
error feedback to eliminate redundant information exchange, which contributes
to significant performance improvement, with (ii) incremental regularization
that can achieve \textit{extreme} sparsity of models. We provide convergence
analysis of FedDIP and report on a comprehensive performance and comparative
assessment against state-of-the-art methods using benchmark data sets and DNN
models. Our results showcase that FedDIP not only controls the model sparsity
but efficiently achieves similar or better performance compared to other model
pruning methods adopting incremental regularization during distributed model
training. The code is available at: https://github.com/EricLoong/feddip.
less
By: Kateřina Macková, Martin Pilát
The goal of product mapping is to decide, whether two listings from two
different e-shops describe the same products. Existing datasets of matching and
non-matching pairs of products, however, often suffer from incomplete product
information or contain only very distant non-matching products. Therefore,
while predictive models trained on these datasets achieve good results on them,
in practice, they are unusable as they cannot distinguish v... more
The goal of product mapping is to decide, whether two listings from two
different e-shops describe the same products. Existing datasets of matching and
non-matching pairs of products, however, often suffer from incomplete product
information or contain only very distant non-matching products. Therefore,
while predictive models trained on these datasets achieve good results on them,
in practice, they are unusable as they cannot distinguish very similar but
non-matching pairs of products. This paper introduces two new datasets for
product mapping: ProMapCz consisting of 1,495 Czech product pairs and ProMapEn
consisting of 1,555 English product pairs of matching and non-matching products
manually scraped from two pairs of e-shops. The datasets contain both images
and textual descriptions of the products, including their specifications,
making them one of the most complete datasets for product mapping.
Additionally, the non-matching products were selected in two phases, creating
two types of non-matches -- close non-matches and medium non-matches. Even the
medium non-matches are pairs of products that are much more similar than
non-matches in other datasets -- for example, they still need to have the same
brand and similar name and price. After simple data preprocessing, several
machine learning algorithms were trained on these and two the other datasets to
demonstrate the complexity and completeness of ProMap datasets. ProMap datasets
are presented as a golden standard for further research of product mapping
filling the gaps in existing ones.
less
By: Joao Fonseca, Andrew Bell, Carlo Abrate, Francesco Bonchi, Julia Stoyanovich
Algorithmic systems are often called upon to assist in high-stakes decision
making. In light of this, algorithmic recourse, the principle wherein
individuals should be able to take action against an undesirable outcome made
by an algorithmic system, is receiving growing attention. The bulk of the
literature on algorithmic recourse to-date focuses primarily on how to provide
recourse to a single individual, overlooking a critical element: th... more
Algorithmic systems are often called upon to assist in high-stakes decision
making. In light of this, algorithmic recourse, the principle wherein
individuals should be able to take action against an undesirable outcome made
by an algorithmic system, is receiving growing attention. The bulk of the
literature on algorithmic recourse to-date focuses primarily on how to provide
recourse to a single individual, overlooking a critical element: the effects of
a continuously changing context. Disregarding these effects on recourse is a
significant oversight, since, in almost all cases, recourse consists of an
individual making a first, unfavorable attempt, and then being given an
opportunity to make one or several attempts at a later date - when the context
might have changed. This can create false expectations, as initial recourse
recommendations may become less reliable over time due to model drift and
competition for access to the favorable outcome between individuals.
In this work we propose an agent-based simulation framework for studying the
effects of a continuously changing environment on algorithmic recourse. In
particular, we identify two main effects that can alter the reliability of
recourse for individuals represented by the agents: (1) competition with other
agents acting upon recourse, and (2) competition with new agents entering the
environment. Our findings highlight that only a small set of specific
parameterizations result in algorithmic recourse that is reliable for agents
over time. Consequently, we argue that substantial additional work is needed to
understand recourse reliability over time, and to develop recourse methods that
reward agents' effort.
less
By: Bailey J. Eccles, Philip Rodgers, Peter Kilpatrick, Ivor Spence, Blesson Varghese
Deep neural networks (DNNs) underpin many machine learning applications.
Production quality DNN models achieve high inference accuracy by training
millions of DNN parameters which has a significant resource footprint. This
presents a challenge for resources operating at the extreme edge of the
network, such as mobile and embedded devices that have limited computational
and memory resources. To address this, models are pruned to create light... more
Deep neural networks (DNNs) underpin many machine learning applications.
Production quality DNN models achieve high inference accuracy by training
millions of DNN parameters which has a significant resource footprint. This
presents a challenge for resources operating at the extreme edge of the
network, such as mobile and embedded devices that have limited computational
and memory resources. To address this, models are pruned to create lightweight,
more suitable variants for these devices. Existing pruning methods are unable
to provide similar quality models compared to their unpruned counterparts
without significant time costs and overheads or are limited to offline use
cases. Our work rapidly derives suitable model variants while maintaining the
accuracy of the original model. The model variants can be swapped quickly when
system and network conditions change to match workload demand. This paper
presents DNNShifter, an end-to-end DNN training, spatial pruning, and model
switching system that addresses the challenges mentioned above. At the heart of
DNNShifter is a novel methodology that prunes sparse models using structured
pruning. The pruned model variants generated by DNNShifter are smaller in size
and thus faster than dense and sparse model predecessors, making them suitable
for inference at the edge while retaining near similar accuracy as of the
original dense model. DNNShifter generates a portfolio of model variants that
can be swiftly interchanged depending on operational conditions. DNNShifter
produces pruned model variants up to 93x faster than conventional training
methods. Compared to sparse models, the pruned model variants are up to 5.14x
smaller and have a 1.67x inference latency speedup, with no compromise to
sparse model accuracy. In addition, DNNShifter has up to 11.9x lower overhead
for switching models and up to 3.8x lower memory utilisation than existing
approaches.
less
By: Niklas Stoehr, Pengxiang Cheng, Jing Wang, Daniel Preotiuc-Pietro, Rajarshi Bhowmik
Language models contain ranking-based knowledge and are powerful solvers of
in-context ranking tasks. For instance, they may have parametric knowledge
about the ordering of countries by size or may be able to rank reviews by
sentiment. Recent work focuses on pairwise, pointwise, and listwise prompting
techniques to elicit a language model's ranking knowledge. However, we find
that even with careful calibration and constrained decoding, prom... more
Language models contain ranking-based knowledge and are powerful solvers of
in-context ranking tasks. For instance, they may have parametric knowledge
about the ordering of countries by size or may be able to rank reviews by
sentiment. Recent work focuses on pairwise, pointwise, and listwise prompting
techniques to elicit a language model's ranking knowledge. However, we find
that even with careful calibration and constrained decoding, prompting-based
techniques may not always be self-consistent in the rankings they produce. This
motivates us to explore an alternative approach that is inspired by an
unsupervised probing method called Contrast-Consistent Search (CCS). The idea
is to train a probing model guided by a logical constraint: a model's
representation of a statement and its negation must be mapped to contrastive
true-false poles consistently across multiple statements. We hypothesize that
similar constraints apply to ranking tasks where all items are related via
consistent pairwise or listwise comparisons. To this end, we extend the binary
CCS method to Contrast-Consistent Ranking (CCR) by adapting existing ranking
methods such as the Max-Margin Loss, Triplet Loss, and Ordinal Regression
objective. Our results confirm that, for the same language model, CCR probing
outperforms prompting and even performs on a par with prompting much larger
language models.
less
By: Khotso Selialia, Yasra Chandio, Fatima M. Anwar
Federated Learning is emerging as a privacy-preserving model training
approach in distributed edge applications. As such, most edge deployments are
heterogeneous in nature i.e., their sensing capabilities and environments vary
across deployments. This edge heterogeneity violates the independence and
identical distribution (IID) property of local data across clients and produces
biased global models i.e. models that contribute to unfair deci... more
Federated Learning is emerging as a privacy-preserving model training
approach in distributed edge applications. As such, most edge deployments are
heterogeneous in nature i.e., their sensing capabilities and environments vary
across deployments. This edge heterogeneity violates the independence and
identical distribution (IID) property of local data across clients and produces
biased global models i.e. models that contribute to unfair decision-making and
discrimination against a particular community or a group. Existing bias
mitigation techniques only focus on bias generated from label heterogeneity in
non-IID data without accounting for domain variations due to feature
heterogeneity and do not address global group-fairness property.
Our work proposes a group-fair FL framework that minimizes group-bias while
preserving privacy and without resource utilization overhead. Our main idea is
to leverage average conditional probabilities to compute a cross-domain group
\textit{importance weights} derived from heterogeneous training data to
optimize the performance of the worst-performing group using a modified
multiplicative weights update method. Additionally, we propose regularization
techniques to minimize the difference between the worst and best-performing
groups while making sure through our thresholding mechanism to strike a balance
between bias reduction and group performance degradation. Our evaluation of
human emotion recognition and image classification benchmarks assesses the fair
decision-making of our framework in real-world heterogeneous settings.
less
By: Samuel Wiggins, Yuan Meng, Rajgopal Kannan, Viktor Prasanna
Multi-Agent Reinforcement Learning (MARL) has achieved significant success in
large-scale AI systems and big-data applications such as smart grids,
surveillance, etc. Existing advancements in MARL algorithms focus on improving
the rewards obtained by introducing various mechanisms for inter-agent
cooperation. However, these optimizations are usually compute- and
memory-intensive, thus leading to suboptimal speed performance in end-to-end
tr... more
Multi-Agent Reinforcement Learning (MARL) has achieved significant success in
large-scale AI systems and big-data applications such as smart grids,
surveillance, etc. Existing advancements in MARL algorithms focus on improving
the rewards obtained by introducing various mechanisms for inter-agent
cooperation. However, these optimizations are usually compute- and
memory-intensive, thus leading to suboptimal speed performance in end-to-end
training time. In this work, we analyze the speed performance (i.e.,
latency-bounded throughput) as the key metric in MARL implementations.
Specifically, we first introduce a taxonomy of MARL algorithms from an
acceleration perspective categorized by (1) training scheme and (2)
communication method. Using our taxonomy, we identify three state-of-the-art
MARL algorithms - Multi-Agent Deep Deterministic Policy Gradient (MADDPG),
Target-oriented Multi-agent Communication and Cooperation (ToM2C), and
Networked Multi-Agent RL (NeurComm) - as target benchmark algorithms, and
provide a systematic analysis of their performance bottlenecks on a homogeneous
multi-core CPU platform. We justify the need for MARL latency-bounded
throughput to be a key performance metric in future literature while also
addressing opportunities for parallelization and acceleration.
less
By: Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan
While traditional machine learning can effectively tackle a wide range of
problems, it primarily operates within a closed-world setting, which presents
limitations when dealing with streaming data. As a solution, incremental
learning emerges to address real-world scenarios involving new data's arrival.
Recently, pre-training has made significant advancements and garnered the
attention of numerous researchers. The strong performance of these... more
While traditional machine learning can effectively tackle a wide range of
problems, it primarily operates within a closed-world setting, which presents
limitations when dealing with streaming data. As a solution, incremental
learning emerges to address real-world scenarios involving new data's arrival.
Recently, pre-training has made significant advancements and garnered the
attention of numerous researchers. The strong performance of these pre-trained
models (PTMs) presents a promising avenue for developing continual learning
algorithms that can effectively adapt to real-world scenarios. Consequently,
exploring the utilization of PTMs in incremental learning has become essential.
This paper introduces a pre-trained model-based continual learning toolbox
known as PILOT. On the one hand, PILOT implements some state-of-the-art
class-incremental learning algorithms based on pre-trained models, such as L2P,
DualPrompt, and CODA-Prompt. On the other hand, PILOT also fits typical
class-incremental learning algorithms (e.g., DER, FOSTER, and MEMO) within the
context of pre-trained models to evaluate their effectiveness.
less
By: Yun Bai, Simon Camal, Andrea Michiorri
Electricity demand forecasting is a well established research field. Usually
this task is performed considering historical loads, weather forecasts,
calendar information and known major events. Recently attention has been given
on the possible use of new sources of information from textual news in order to
improve the performance of these predictions. This paper proposes a Long and
Short-Term Memory (LSTM) network incorporating textual news... more
Electricity demand forecasting is a well established research field. Usually
this task is performed considering historical loads, weather forecasts,
calendar information and known major events. Recently attention has been given
on the possible use of new sources of information from textual news in order to
improve the performance of these predictions. This paper proposes a Long and
Short-Term Memory (LSTM) network incorporating textual news features that
successfully predicts the deterministic and probabilistic tasks of the UK
national electricity demand. The study finds that public sentiment and word
vector representations related to transport and geopolitics have
time-continuity effects on electricity demand. The experimental results show
that the LSTM with textual features improves by more than 3% compared to the
pure LSTM benchmark and by close to 10% over the official benchmark.
Furthermore, the proposed model effectively reduces forecasting uncertainty by
narrowing the confidence interval and bringing the forecast distribution closer
to the truth.
less
By: Eran Malach
Large language models display remarkable capabilities in logical and
mathematical reasoning, allowing them to solve complex tasks. Interestingly,
these abilities emerge in networks trained on the simple task of next-token
prediction. In this work, we present a theoretical framework for studying
auto-regressive next-token predictors. We demonstrate that even simple models
such as linear next-token predictors, trained on Chain-of-Thought (CoT... more
Large language models display remarkable capabilities in logical and
mathematical reasoning, allowing them to solve complex tasks. Interestingly,
these abilities emerge in networks trained on the simple task of next-token
prediction. In this work, we present a theoretical framework for studying
auto-regressive next-token predictors. We demonstrate that even simple models
such as linear next-token predictors, trained on Chain-of-Thought (CoT) data,
can approximate any function efficiently computed by a Turing machine. We
introduce a new complexity measure -- length complexity -- which measures the
number of intermediate tokens in a CoT sequence required to approximate some
target function, and analyze the interplay between length complexity and other
notions of complexity. Finally, we show experimentally that simple next-token
predictors, such as linear networks and shallow Multi-Layer Perceptrons (MLPs),
display non-trivial performance on text generation and arithmetic tasks. Our
results demonstrate that the power of language models can be attributed, to a
great extent, to the auto-regressive next-token training scheme, and not
necessarily to a particular choice of architecture.
less
By: Fei Dou, Jin Ye, Geng Yuan, Qin Lu, Wei Niu, Haijian Sun, Le Guan, Guoyu Lu, Gengchen Mai, Ninghao Liu, Jin Lu, Zhengliang Liu, Zihao Wu, Chenjiao Tan, Shaochen Xu, Xianqiao Wang, Guoming Li, Lilong Chai, Sheng Li, Jin Sun, Hongyue Sun, Yunli Shao, Changying Li, Tianming Liu, Wenzhan Song
Artificial General Intelligence (AGI), possessing the capacity to comprehend,
learn, and execute tasks with human cognitive abilities, engenders significant
anticipation and intrigue across scientific, commercial, and societal arenas.
This fascination extends particularly to the Internet of Things (IoT), a
landscape characterized by the interconnection of countless devices, sensors,
and systems, collectively gathering and sharing data to en... more
Artificial General Intelligence (AGI), possessing the capacity to comprehend,
learn, and execute tasks with human cognitive abilities, engenders significant
anticipation and intrigue across scientific, commercial, and societal arenas.
This fascination extends particularly to the Internet of Things (IoT), a
landscape characterized by the interconnection of countless devices, sensors,
and systems, collectively gathering and sharing data to enable intelligent
decision-making and automation. This research embarks on an exploration of the
opportunities and challenges towards achieving AGI in the context of the IoT.
Specifically, it starts by outlining the fundamental principles of IoT and the
critical role of Artificial Intelligence (AI) in IoT systems. Subsequently, it
delves into AGI fundamentals, culminating in the formulation of a conceptual
framework for AGI's seamless integration within IoT. The application spectrum
for AGI-infused IoT is broad, encompassing domains ranging from smart grids,
residential environments, manufacturing, and transportation to environmental
monitoring, agriculture, healthcare, and education. However, adapting AGI to
resource-constrained IoT settings necessitates dedicated research efforts.
Furthermore, the paper addresses constraints imposed by limited computing
resources, intricacies associated with large-scale IoT communication, as well
as the critical concerns pertaining to security and privacy.
less
By: Ann Speed
Generative AI models garnered a large amount of public attention and
speculation with the release of OpenAIs chatbot, ChatGPT. At least two opinion
camps exist: one excited about possibilities these models offer for fundamental
changes to human tasks, and another highly concerned about power these models
seem to have. To address these concerns, we assessed GPT3.5 using standard,
normed, and validated cognitive and personality measures. For ... more
Generative AI models garnered a large amount of public attention and
speculation with the release of OpenAIs chatbot, ChatGPT. At least two opinion
camps exist: one excited about possibilities these models offer for fundamental
changes to human tasks, and another highly concerned about power these models
seem to have. To address these concerns, we assessed GPT3.5 using standard,
normed, and validated cognitive and personality measures. For this seedling
project, we developed a battery of tests that allowed us to estimate the
boundaries of some of these models capabilities, how stable those capabilities
are over a short period of time, and how they compare to humans.
Our results indicate that GPT 3.5 is unlikely to have developed sentience,
although its ability to respond to personality inventories is interesting. It
did display large variability in both cognitive and personality measures over
repeated observations, which is not expected if it had a human-like
personality. Variability notwithstanding, GPT3.5 displays what in a human would
be considered poor mental health, including low self-esteem and marked
dissociation from reality despite upbeat and helpful responses.
less
By: Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Qin Liu, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huan, Tao Gui
For a long time, humanity has pursued artificial intelligence (AI) equivalent
to or surpassing the human level, with AI agents considered a promising vehicle
for this pursuit. AI agents are artificial entities that sense their
environment, make decisions, and take actions. Many efforts have been made to
develop intelligent AI agents since the mid-20th century. However, these
efforts have mainly focused on advancement in algorithms or traini... more
For a long time, humanity has pursued artificial intelligence (AI) equivalent
to or surpassing the human level, with AI agents considered a promising vehicle
for this pursuit. AI agents are artificial entities that sense their
environment, make decisions, and take actions. Many efforts have been made to
develop intelligent AI agents since the mid-20th century. However, these
efforts have mainly focused on advancement in algorithms or training strategies
to enhance specific capabilities or performance on particular tasks. Actually,
what the community lacks is a sufficiently general and powerful model to serve
as a starting point for designing AI agents that can adapt to diverse
scenarios. Due to the versatile and remarkable capabilities they demonstrate,
large language models (LLMs) are regarded as potential sparks for Artificial
General Intelligence (AGI), offering hope for building general AI agents. Many
research efforts have leveraged LLMs as the foundation to build AI agents and
have achieved significant progress. We start by tracing the concept of agents
from its philosophical origins to its development in AI, and explain why LLMs
are suitable foundations for AI agents. Building upon this, we present a
conceptual framework for LLM-based agents, comprising three main components:
brain, perception, and action, and the framework can be tailored to suit
different applications. Subsequently, we explore the extensive applications of
LLM-based agents in three aspects: single-agent scenarios, multi-agent
scenarios, and human-agent cooperation. Following this, we delve into agent
societies, exploring the behavior and personality of LLM-based agents, the
social phenomena that emerge when they form societies, and the insights they
offer for human society. Finally, we discuss a range of key topics and open
problems within the field.
less
When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence System
0upvotes
By: Hao Zhang, Jin-Jian Xu
Geoscience foundation models represent a revolutionary approach in the field
of Earth sciences by integrating massive cross-disciplinary data to simulate
and understand the Earth systems dynamics. As a data-centric artificial
intelligence (AI) paradigm, they uncover insights from petabytes of structured
and unstructured data. Flexible task specification, diverse inputs and outputs
and multi-modal knowledge representation enable comprehensiv... more
Geoscience foundation models represent a revolutionary approach in the field
of Earth sciences by integrating massive cross-disciplinary data to simulate
and understand the Earth systems dynamics. As a data-centric artificial
intelligence (AI) paradigm, they uncover insights from petabytes of structured
and unstructured data. Flexible task specification, diverse inputs and outputs
and multi-modal knowledge representation enable comprehensive analysis
infeasible with individual data sources. Critically, the scalability and
generalizability of geoscience models allow for tackling diverse prediction,
simulation, and decision challenges related to Earth systems interactions.
Collaboration between domain experts and computer scientists leads to
innovations in these invaluable tools for understanding the past, present, and
future of our planet. However, challenges remain in validation and
verification, scale, interpretability, knowledge representation, and social
bias. Going forward, enhancing model integration, resolution, accuracy, and
equity through cross-disciplinary teamwork is key. Despite current limitations,
geoscience foundation models show promise for providing critical insights into
pressing issues including climate change, natural hazards, and sustainability
through their ability to probe scenarios and quantify uncertainties. Their
continued evolution toward integrated, data-driven modeling holds
paradigm-shifting potential for Earth science.
less
By: Konrad Abicht
In a systematic literature and software review over 100 OWL reasoners/systems
were analyzed to see if they would still be usable in 2023. This has never been
done in this capacity. OWL reasoners still play an important role in knowledge
organisation and management, but the last comprehensive surveys/studies are
more than 8 years old. The result of this work is a comprehensive list of 95
standalone OWL reasoners and systems using an OWL reas... more
In a systematic literature and software review over 100 OWL reasoners/systems
were analyzed to see if they would still be usable in 2023. This has never been
done in this capacity. OWL reasoners still play an important role in knowledge
organisation and management, but the last comprehensive surveys/studies are
more than 8 years old. The result of this work is a comprehensive list of 95
standalone OWL reasoners and systems using an OWL reasoner. For each item,
information on project pages, source code repositories and related
documentation was gathered. The raw research data is provided in a Github
repository for anyone to use.
less
By: Marco Gori, Stefano Melacci
By and large, the professional handling of huge data collections is regarded
as a fundamental ingredient of the progress of machine learning and of its
spectacular results in related disciplines, with a growing agreement on risks
connected to the centralization of such data collections. This paper sustains
the position that the time has come for thinking of new learning protocols
where machines conquer cognitive skills in a truly human-like... more
By and large, the professional handling of huge data collections is regarded
as a fundamental ingredient of the progress of machine learning and of its
spectacular results in related disciplines, with a growing agreement on risks
connected to the centralization of such data collections. This paper sustains
the position that the time has come for thinking of new learning protocols
where machines conquer cognitive skills in a truly human-like context centered
on environmental interactions. This comes with specific restrictions on the
learning protocol according to the collectionless principle, which states that,
at each time instant, data acquired from the environment is processed with the
purpose of contributing to update the current internal representation of the
environment, and that the agent is not given the privilege of recording the
temporal stream. Basically, there is neither permission to store the temporal
information coming from the sensors, thus promoting the development of
self-organized memorization skills at a more abstract level, instead of relying
on bare storage to simulate learning dynamics that are typical of offline
learning algorithms. This purposely extreme position is intended to stimulate
the development of machines that learn to dynamically organize the information
by following human-based schemes. The proposition of this challenge suggests
developing new foundations on computational processes of learning and reasoning
that might open the doors to a truly orthogonal competitive track on AI
technologies that avoid data accumulation by design, thus offering a framework
which is better suited concerning privacy issues, control and customizability.
Finally, pushing towards massively distributed computation, the collectionless
approach to AI will likely reduce the concentration of power in companies and
governments, thus better facing geopolitical issues.
less
By: Tetsuya Abe, Ryusuke Sagawa, Ko Ayusawa, Wataru Takano
The present paper proposes an encoder-decoder model for extracting the
structures of human motions represented by frame-wise discrete features in a
self-supervised manner. In the proposed method, features are extracted as codes
in a motion codebook without the use of human knowledge, and the relationship
between these codes can be visualized on a graph. Since the codes are expected
to be temporally sparse compared to the captured frame rate... more
The present paper proposes an encoder-decoder model for extracting the
structures of human motions represented by frame-wise discrete features in a
self-supervised manner. In the proposed method, features are extracted as codes
in a motion codebook without the use of human knowledge, and the relationship
between these codes can be visualized on a graph. Since the codes are expected
to be temporally sparse compared to the captured frame rate and can be shared
by multiple sequences, the proposed network model also addresses the need for
training constraints. Specifically, the model consists of self-attention layers
and a vector clustering block. The attention layers contribute to finding
sparse keyframes and discrete features as motion codes, which are then
extracted by vector clustering. The constraints are realized as training losses
so that the same motion codes can be as contiguous as possible and can be
shared by multiple sequences. In addition, we propose the use of causal
self-attention as a method by which to calculate attention for long sequences
consisting of numerous frames. In our experiments, the sparse structures of
motion codes were used to compile a graph that facilitates visualization of the
relationship between the codes and the differences between sequences. We then
evaluated the effectiveness of the extracted motion codes by applying them to
multiple recognition tasks and found that performance levels comparable to
task-optimized methods could be achieved by linear probing.
less
FLDNet: A Foreground-Aware Network for Polyp Segmentation Leveraging Long-Distance Dependencies
0upvotes
By: Xuefeng Wei, Xuan Zhou
Given the close association between colorectal cancer and polyps, the
diagnosis and identification of colorectal polyps play a critical role in the
detection and surgical intervention of colorectal cancer. In this context, the
automatic detection and segmentation of polyps from various colonoscopy images
has emerged as a significant problem that has attracted broad attention.
Current polyp segmentation techniques face several challenges: fi... more
Given the close association between colorectal cancer and polyps, the
diagnosis and identification of colorectal polyps play a critical role in the
detection and surgical intervention of colorectal cancer. In this context, the
automatic detection and segmentation of polyps from various colonoscopy images
has emerged as a significant problem that has attracted broad attention.
Current polyp segmentation techniques face several challenges: firstly, polyps
vary in size, texture, color, and pattern; secondly, the boundaries between
polyps and mucosa are usually blurred, existing studies have focused on
learning the local features of polyps while ignoring the long-range
dependencies of the features, and also ignoring the local context and global
contextual information of the combined features. To address these challenges,
we propose FLDNet (Foreground-Long-Distance Network), a Transformer-based
neural network that captures long-distance dependencies for accurate polyp
segmentation. Specifically, the proposed model consists of three main modules:
a pyramid-based Transformer encoder, a local context module, and a
foreground-Aware module. Multilevel features with long-distance dependency
information are first captured by the pyramid-based transformer encoder. On the
high-level features, the local context module obtains the local characteristics
related to the polyps by constructing different local context information. The
coarse map obtained by decoding the reconstructed highest-level features guides
the feature fusion process in the foreground-Aware module of the high-level
features to achieve foreground enhancement of the polyps. Our proposed method,
FLDNet, was evaluated using seven metrics on common datasets and demonstrated
superiority over state-of-the-art methods on widely-used evaluation measures.
less
By: Haibo Chen, Lei Zhao, Jun Li, Jian Yang
Artistic style transfer aims to create new artistic images by rendering a
given photograph with the target artistic style. Existing methods learn styles
simply based on global statistics or local patches, lacking careful
consideration of the drawing process in practice. Consequently, the stylization
results either fail to capture abundant and diversified local style patterns,
or contain undesired semantic information of the style image and ... more
Artistic style transfer aims to create new artistic images by rendering a
given photograph with the target artistic style. Existing methods learn styles
simply based on global statistics or local patches, lacking careful
consideration of the drawing process in practice. Consequently, the stylization
results either fail to capture abundant and diversified local style patterns,
or contain undesired semantic information of the style image and deviate from
the global style distribution. To address this issue, we imitate the drawing
process of humans and propose a Two-Stage Statistics-Aware Transformation
(TSSAT) module, which first builds the global style foundation by aligning the
global statistics of content and style features and then further enriches local
style details by swapping the local statistics (instead of local features) in a
patch-wise manner, significantly improving the stylization effects. Moreover,
to further enhance both content and style representations, we introduce two
novel losses: an attention-based content loss and a patch-based style loss,
where the former enables better content preservation by enforcing the semantic
relation in the content image to be retained during stylization, and the latter
focuses on increasing the local style similarity between the style and stylized
images. Extensive qualitative and quantitative experiments verify the
effectiveness of our method.
less
Feature Aggregation Network for Building Extraction from High-resolution Remote Sensing Images
0upvotes
By: Xuan Zhou, Xuefeng Wei
The rapid advancement in high-resolution satellite remote sensing data
acquisition, particularly those achieving submeter precision, has uncovered the
potential for detailed extraction of surface architectural features. However,
the diversity and complexity of surface distributions frequently lead to
current methods focusing exclusively on localized information of surface
features. This often results in significant intraclass variability in... more
The rapid advancement in high-resolution satellite remote sensing data
acquisition, particularly those achieving submeter precision, has uncovered the
potential for detailed extraction of surface architectural features. However,
the diversity and complexity of surface distributions frequently lead to
current methods focusing exclusively on localized information of surface
features. This often results in significant intraclass variability in boundary
recognition and between buildings. Therefore, the task of fine-grained
extraction of surface features from high-resolution satellite imagery has
emerged as a critical challenge in remote sensing image processing. In this
work, we propose the Feature Aggregation Network (FANet), concentrating on
extracting both global and local features, thereby enabling the refined
extraction of landmark buildings from high-resolution satellite remote sensing
imagery. The Pyramid Vision Transformer captures these global features, which
are subsequently refined by the Feature Aggregation Module and merged into a
cohesive representation by the Difference Elimination Module. In addition, to
ensure a comprehensive feature map, we have incorporated the Receptive Field
Block and Dual Attention Module, expanding the receptive field and intensifying
attention across spatial and channel dimensions. Extensive experiments on
multiple datasets have validated the outstanding capability of FANet in
extracting features from high-resolution satellite images. This signifies a
major breakthrough in the field of remote sensing image processing. We will
release our code soon.
less
By: Gang Wu, Junjun Jiang, Kui Jiang, Xianming Liu
Contrastive learning has emerged as a prevailing paradigm for high-level
vision tasks, which, by introducing properly negative samples, has also been
exploited for low-level vision tasks to achieve a compact optimization space to
account for their ill-posed nature. However, existing methods rely on manually
predefined, task-oriented negatives, which often exhibit pronounced
task-specific biases. In this paper, we propose a innovative approa... more
Contrastive learning has emerged as a prevailing paradigm for high-level
vision tasks, which, by introducing properly negative samples, has also been
exploited for low-level vision tasks to achieve a compact optimization space to
account for their ill-posed nature. However, existing methods rely on manually
predefined, task-oriented negatives, which often exhibit pronounced
task-specific biases. In this paper, we propose a innovative approach for the
adaptive generation of negative samples directly from the target model itself,
called ``learning from history``. We introduce the Self-Prior guided Negative
loss for image restoration (SPNIR) to enable this approach. Our approach is
task-agnostic and generic, making it compatible with any existing image
restoration method or task. We demonstrate the effectiveness of our approach by
retraining existing models with SPNIR. The results show significant
improvements in image restoration across various tasks and architectures. For
example, models retrained with SPNIR outperform the original FFANet and
DehazeFormer by 3.41 dB and 0.57 dB on the RESIDE indoor dataset for image
dehazing. Similarly, they achieve notable improvements of 0.47 dB on SPA-Data
over IDT for image deraining and 0.12 dB on Manga109 for a 4x scale
super-resolution over lightweight SwinIR, respectively. Code and retrained
models are available at
https://github.com/Aitical/Task-agnostic_Model_Contrastive_Learning_Image_Restoration.
less
By: Teppei Suzuki
We envision a system to continuously build and maintain a map based on
earth-scale neural radiance fields (NeRF) using data collected from vehicles
and drones in a lifelong learning manner. However, existing large-scale
modeling by NeRF has problems in terms of scalability and maintainability when
modeling earth-scale environments. Therefore, to address these problems, we
propose a federated learning pipeline for large-scale modeling with N... more
We envision a system to continuously build and maintain a map based on
earth-scale neural radiance fields (NeRF) using data collected from vehicles
and drones in a lifelong learning manner. However, existing large-scale
modeling by NeRF has problems in terms of scalability and maintainability when
modeling earth-scale environments. Therefore, to address these problems, we
propose a federated learning pipeline for large-scale modeling with NeRF. We
tailor the model aggregation pipeline in federated learning for NeRF, thereby
allowing local updates of NeRF. In the aggregation step, the accuracy of the
clients' global pose is critical. Thus, we also propose global pose alignment
to align the noisy global pose of clients before the aggregation step. In
experiments, we show the effectiveness of the proposed pose alignment and the
federated learning pipeline on the large-scale scene dataset, Mill19.
less
By: Clifford Broni-Bediako, Junshi Xia, Naoto Yokoya
Real-time semantic segmentation of remote sensing imagery is a challenging
task that requires a trade-off between effectiveness and efficiency. It has
many applications including tracking forest fires, detecting changes in land
use and land cover, crop health monitoring, and so on. With the success of
efficient deep learning methods (i.e., efficient deep neural networks) for
real-time semantic segmentation in computer vision, researchers ha... more
Real-time semantic segmentation of remote sensing imagery is a challenging
task that requires a trade-off between effectiveness and efficiency. It has
many applications including tracking forest fires, detecting changes in land
use and land cover, crop health monitoring, and so on. With the success of
efficient deep learning methods (i.e., efficient deep neural networks) for
real-time semantic segmentation in computer vision, researchers have adopted
these efficient deep neural networks in remote sensing image analysis. This
paper begins with a summary of the fundamental compression methods for
designing efficient deep neural networks and provides a brief but comprehensive
survey, outlining the recent developments in real-time semantic segmentation of
remote sensing imagery. We examine several seminal efficient deep learning
methods, placing them in a taxonomy based on the network architecture design
approach. Furthermore, we evaluate the quality and efficiency of some existing
efficient deep neural networks on a publicly available remote sensing semantic
segmentation benchmark dataset, the OpenEarthMap. The experimental results of
an extensive comparative study demonstrate that most of the existing efficient
deep neural networks have good segmentation quality, but they suffer low
inference speed (i.e., high latency rate), which may limit their capability of
deployment in real-time applications of remote sensing image segmentation. We
provide some insights into the current trend and future research directions for
real-time semantic segmentation of remote sensing imagery.
less
By: Alessandro Duico, Ombretta Strafforello, Jan van Gemert
Predicting which specific parts of a video users will replay is important for
several applications, including targeted advertisement placement on video
platforms and assisting video creators. In this work, we explore whether it is
possible to predict the Most Replayed (MR) data from YouTube videos. To this
end, we curate a large video benchmark, the YTMR500 dataset, which comprises
500 YouTube videos with MR data annotations. We evaluate De... more
Predicting which specific parts of a video users will replay is important for
several applications, including targeted advertisement placement on video
platforms and assisting video creators. In this work, we explore whether it is
possible to predict the Most Replayed (MR) data from YouTube videos. To this
end, we curate a large video benchmark, the YTMR500 dataset, which comprises
500 YouTube videos with MR data annotations. We evaluate Deep Learning (DL)
models of varying complexity on our dataset and perform an extensive ablation
study. In addition, we conduct a user study to estimate the human performance
on MR data prediction. Our results show that, although by a narrow margin, all
the evaluated DL models outperform random predictions. Additionally, they
exceed human-level accuracy. This suggests that predicting the MR data is a
difficult task that can be enhanced through the assistance of DL. Finally, we
believe that DL performance on MR data prediction can be further improved, for
example, by using multi-modal learning. We encourage the research community to
use our benchmark dataset to further investigate automatic MR data prediction.
less
By: Yafei Zhang, Keying Du, Huafeng Li, Zhengtao Yu, Yu Liu
Infrared and visible image fusion (IVIF) aims to extract and integrate the
complementary information in two different modalities to generate high-quality
fused images with salient targets and abundant texture details. However,
current image fusion methods go to great lengths to excavate complementary
features, which is generally achieved through two efforts. On the one hand, the
feature extraction network is expected to have excellent perfo... more
Infrared and visible image fusion (IVIF) aims to extract and integrate the
complementary information in two different modalities to generate high-quality
fused images with salient targets and abundant texture details. However,
current image fusion methods go to great lengths to excavate complementary
features, which is generally achieved through two efforts. On the one hand, the
feature extraction network is expected to have excellent performance in
extracting complementary information. On the other hand, complex fusion
strategies are often designed to aggregate the complementary information. In
other words, enabling the network to perceive and extract complementary
information is extremely challenging. Complicated fusion strategies, while
effective, still run the risk of losing weak edge details. To this end, this
paper rethinks the IVIF outside the box, proposing a complementary-redundant
information transfer network (C-RITNet). It reasonably transfers complementary
information into redundant one, which integrates both the shared and
complementary features from two modalities. Hence, the proposed method is able
to alleviate the challenges posed by the complementary information extraction
and reduce the reliance on sophisticated fusion strategies. Specifically, to
skillfully sidestep aggregating complementary information in IVIF, we first
design the mutual information transfer (MIT) module to mutually represent
features from two modalities, roughly transferring complementary information
into redundant one. Then, a redundant information acquisition supervised by
source image (RIASSI) module is devised to further ensure the
complementary-redundant information transfer after MIT. Meanwhile, we also
propose a structure information preservation (SIP) module to guarantee that the
edge structure information of the source images can be transferred to the
fusion results.
less
LEyes: A Lightweight Framework for Deep Learning-Based Eye Tracking using Synthetic Eye Images
0upvotes
By: sean anthony byrne, virmarie maquiling, marcus nyström, enkelejda kasneci, diederick c. niehorster
Deep learning has bolstered gaze estimation techniques, but real-world
deployment has been impeded by inadequate training datasets. This problem is
exacerbated by both hardware-induced variations in eye images and inherent
biological differences across the recorded participants, leading to both
feature and pixel-level variance that hinders the generalizability of models
trained on specific datasets. While synthetic datasets can be a solutio... more
Deep learning has bolstered gaze estimation techniques, but real-world
deployment has been impeded by inadequate training datasets. This problem is
exacerbated by both hardware-induced variations in eye images and inherent
biological differences across the recorded participants, leading to both
feature and pixel-level variance that hinders the generalizability of models
trained on specific datasets. While synthetic datasets can be a solution, their
creation is both time and resource-intensive. To address this problem, we
present a framework called Light Eyes or "LEyes" which, unlike conventional
photorealistic methods, only models key image features required for video-based
eye tracking using simple light distributions. LEyes facilitates easy
configuration for training neural networks across diverse gaze-estimation
tasks. We demonstrate that models trained using LEyes outperform other
state-of-the-art algorithms in terms of pupil and CR localization across
well-known datasets. In addition, a LEyes trained model outperforms the
industry standard eye tracker using significantly more cost-effective hardware.
Going forward, we are confident that LEyes will revolutionize synthetic data
generation for gaze estimation models, and lead to significant improvements of
the next generation video-based eye trackers.
less
By: Mohammed Guermal, Francois Bremond, Rui Dai, Abid Ali
Action anticipation involves forecasting future actions by connecting past
events to future ones. However, this reasoning ignores the real-life hierarchy
of events which is considered to be composed of three main parts: past,
present, and future. We argue that considering these three main parts and their
dependencies could improve performance. On the other hand, online action
detection is the task of predicting actions in a streaming manner... more
Action anticipation involves forecasting future actions by connecting past
events to future ones. However, this reasoning ignores the real-life hierarchy
of events which is considered to be composed of three main parts: past,
present, and future. We argue that considering these three main parts and their
dependencies could improve performance. On the other hand, online action
detection is the task of predicting actions in a streaming manner. In this
case, one has access only to the past and present information. Therefore, in
online action detection (OAD) the existing approaches miss semantics or future
information which limits their performance. To sum up, for both of these tasks,
the complete set of knowledge (past-present-future) is missing, which makes it
challenging to infer action dependencies, therefore having low performances. To
address this limitation, we propose to fuse both tasks into a single uniform
architecture. By combining action anticipation and online action detection, our
approach can cover the missing dependencies of future information in online
action detection. This method referred to as JOADAA, presents a uniform model
that jointly performs action anticipation and online action detection. We
validate our proposed model on three challenging datasets: THUMOS'14, which is
a sparsely annotated dataset with one action per time step, CHARADES, and
Multi-THUMOS, two densely annotated datasets with more complex scenarios.
JOADAA achieves SOTA results on these benchmarks for both tasks.
less
By: Jiao Zhang, Xu-Yao Zhang, Cheng-Lin Liu
There are ubiquitous distribution shifts in the real world. However, deep
neural networks (DNNs) are easily biased towards the training set, which causes
severe performance degradation when they receive out-of-distribution data. Many
methods are studied to train models that generalize under various distribution
shifts in the literature of domain generalization (DG). However, the recent
DomainBed and WILDS benchmarks challenged the effective... more
There are ubiquitous distribution shifts in the real world. However, deep
neural networks (DNNs) are easily biased towards the training set, which causes
severe performance degradation when they receive out-of-distribution data. Many
methods are studied to train models that generalize under various distribution
shifts in the literature of domain generalization (DG). However, the recent
DomainBed and WILDS benchmarks challenged the effectiveness of these methods.
Aiming at the problems in the existing research, we propose a new domain
generalization task for handwritten Chinese character recognition (HCCR) to
enrich the application scenarios of DG method research. We evaluate eighteen DG
methods on the proposed PaHCC (Printed and Handwritten Chinese Characters)
dataset and show that the performance of existing methods on this dataset is
still unsatisfactory. Besides, under a designed dynamic DG setting, we reveal
more properties of DG methods and argue that only the leave-one-domain-out
protocol is unreliable. We advocate that researchers in the DG community refer
to dynamic performance of methods for more comprehensive and reliable
evaluation. Our dataset and evaluations bring new perspectives to the community
for more substantial progress. We will make our dataset public with the article
published to facilitate the study of domain generalization.
less
By: Tuan Pham Minh, Jayan Wijesingha, Daniel Kottke, Marek Herde, Denis Huseljic, Bernhard Sick, Michael Wachendorf, Thomas Esch
Remote sensing through semantic segmentation of satellite images contributes
to the understanding and utilisation of the earth's surface. For this purpose,
semantic segmentation networks are typically trained on large sets of labelled
satellite images. However, obtaining expert labels for these images is costly.
Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or
pretrained networks, to label the images in the first ... more
Remote sensing through semantic segmentation of satellite images contributes
to the understanding and utilisation of the earth's surface. For this purpose,
semantic segmentation networks are typically trained on large sets of labelled
satellite images. However, obtaining expert labels for these images is costly.
Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or
pretrained networks, to label the images in the first step. Since these initial
labels are partially erroneous, we use active learning strategies to
cost-efficiently refine the labels in the second step. We evaluate the active
learning strategies using satellite images of Bengaluru in India, labelled with
land cover and land use labels. Our experimental results suggest that an active
label refinement to improve the semantic segmentation network's performance is
beneficial.
less
By: Jiaxiu Li, Kun Li, Jia Li, Guoliang Chen, Dan Guo, Meng Wang
Make-up temporal video grounding (MTVG) aims to localize the target video
segment which is semantically related to a sentence describing a make-up
activity, given a long video. Compared with the general video grounding task,
MTVG focuses on meticulous actions and changes on the face. The make-up
instruction step, usually involving detailed differences in products and facial
areas, is more fine-grained than general activities (e.g, cooking a... more
Make-up temporal video grounding (MTVG) aims to localize the target video
segment which is semantically related to a sentence describing a make-up
activity, given a long video. Compared with the general video grounding task,
MTVG focuses on meticulous actions and changes on the face. The make-up
instruction step, usually involving detailed differences in products and facial
areas, is more fine-grained than general activities (e.g, cooking activity and
furniture assembly). Thus, existing general approaches cannot locate the target
activity effectually. More specifically, existing proposal generation modules
are not yet fully developed in providing semantic cues for the more
fine-grained make-up semantic comprehension. To tackle this issue, we propose
an effective proposal-based framework named Dual-Path Temporal Map Optimization
Network (DPTMO) to capture fine-grained multimodal semantic details of make-up
activities. DPTMO extracts both query-agnostic and query-guided features to
construct two proposal sets and uses specific evaluation methods for the two
sets. Different from the commonly used single structure in previous methods,
our dual-path structure can mine more semantic information in make-up videos
and distinguish fine-grained actions well. These two candidate sets represent
the cross-modal makeup video-text similarity and multi-modal fusion
relationship, complementing each other. Each set corresponds to its respective
optimization perspective, and their joint prediction enhances the accuracy of
video timestamp prediction. Comprehensive experiments on the YouMakeup dataset
demonstrate our proposed dual structure excels in fine-grained semantic
comprehension.
less
By: Mazvydas Gudelis, Michal Mackiewicz, Julie Bremner, Sophie Fielding
British Antarctic Survey (BAS) researchers launch annual expeditions to the
Antarctic in order to estimate Antarctic Krill biomass and assess the change
from previous years. These comparisons provide insight into the effects of the
current environment on this key component of the marine food chain. In this
work we have developed tools for automating the data collection and analysis
process, using web-based image annotation tools and deep le... more
British Antarctic Survey (BAS) researchers launch annual expeditions to the
Antarctic in order to estimate Antarctic Krill biomass and assess the change
from previous years. These comparisons provide insight into the effects of the
current environment on this key component of the marine food chain. In this
work we have developed tools for automating the data collection and analysis
process, using web-based image annotation tools and deep learning image
classification and regression models. We achieve highly accurate krill instance
segmentation results with an average 77.28% AP score, as well as separate
maturity stage and length estimation of krill specimens with 62.99% accuracy
and a 1.96 mm length error respectively.
less
By: Laurenz Reichardt, Nikolas Ebert, Oliver Wasenmüller
Deep learning applications on LiDAR data suffer from a strong domain gap when
applied to different sensors or tasks. In order for these methods to obtain
similar accuracy on different data in comparison to values reported on public
benchmarks, a large scale annotated dataset is necessary. However, in practical
applications labeled data is costly and time consuming to obtain. Such factors
have triggered various research in label-efficient me... more
Deep learning applications on LiDAR data suffer from a strong domain gap when
applied to different sensors or tasks. In order for these methods to obtain
similar accuracy on different data in comparison to values reported on public
benchmarks, a large scale annotated dataset is necessary. However, in practical
applications labeled data is costly and time consuming to obtain. Such factors
have triggered various research in label-efficient methods, but a large gap
remains to their fully-supervised counterparts. Thus, we propose ImageTo360, an
effective and streamlined few-shot approach to label-efficient LiDAR
segmentation. Our method utilizes an image teacher network to generate semantic
predictions for LiDAR data within a single camera view. The teacher is used to
pretrain the LiDAR segmentation student network, prior to optional fine-tuning
on 360$^\circ$ data. Our method is implemented in a modular manner on the point
level and as such is generalizable to different architectures. We improve over
the current state-of-the-art results for label-efficient methods and even
surpass some traditional fully-supervised segmentation networks.
less
By: Yiming Shan, Yan Xia, Yuhong Chen, Daniel Cremers
3D object detection using LiDAR point clouds is a fundamental task in the
fields of computer vision, robotics, and autonomous driving. However, existing
3D detectors heavily rely on annotated datasets, which are both time-consuming
and prone to errors during the process of labeling 3D bounding boxes. In this
paper, we propose a Scene Completion Pre-training (SCP) method to enhance the
performance of 3D object detectors with less labeled dat... more
3D object detection using LiDAR point clouds is a fundamental task in the
fields of computer vision, robotics, and autonomous driving. However, existing
3D detectors heavily rely on annotated datasets, which are both time-consuming
and prone to errors during the process of labeling 3D bounding boxes. In this
paper, we propose a Scene Completion Pre-training (SCP) method to enhance the
performance of 3D object detectors with less labeled data. SCP offers three key
advantages: (1) Improved initialization of the point cloud model. By completing
the scene point clouds, SCP effectively captures the spatial and semantic
relationships among objects within urban environments. (2) Elimination of the
need for additional datasets. SCP serves as a valuable auxiliary network that
does not impose any additional efforts or data requirements on the 3D
detectors. (3) Reduction of the amount of labeled data for detection. With the
help of SCP, the existing state-of-the-art 3D detectors can achieve comparable
performance while only relying on 20% labeled data.
less
By: Junjing Zheng, Xinyu Zhang, Yongxiang Liu, Weidong Jiang, Kai Huo, Li Liu
In the field of unsupervised feature selection, sparse principal component
analysis (SPCA) methods have attracted more and more attention recently.
Compared to spectral-based methods, SPCA methods don't rely on the construction
of a similarity matrix and show better feature selection ability on real-world
data. The original SPCA formulates a nonconvex optimization problem. Existing
convex SPCA methods reformulate SPCA as a convex model by r... more
In the field of unsupervised feature selection, sparse principal component
analysis (SPCA) methods have attracted more and more attention recently.
Compared to spectral-based methods, SPCA methods don't rely on the construction
of a similarity matrix and show better feature selection ability on real-world
data. The original SPCA formulates a nonconvex optimization problem. Existing
convex SPCA methods reformulate SPCA as a convex model by regarding the
reconstruction matrix as an optimization variable. However, they are lack of
constraints equivalent to the orthogonality restriction in SPCA, leading to
larger solution space. In this paper, it's proved that the optimal solution to
a convex SPCA model falls onto the Positive Semidefinite (PSD) cone. A standard
convex SPCA-based model with PSD constraint for unsupervised feature selection
is proposed. Further, a two-step fast optimization algorithm via PSD projection
is presented to solve the proposed model. Two other existing convex SPCA-based
models are also proven to have their solutions optimized on the PSD cone in
this paper. Therefore, the PSD versions of these two models are proposed to
accelerate their convergence as well. We also provide a regularization
parameter setting strategy for our proposed method. Experiments on synthetic
and real-world datasets demonstrate the effectiveness and efficiency of the
proposed methods.
less
By: Qianliang Wu, Yaqing Ding, Lei Luo, Chuanwei Zhou, Jin Xie, Jian Yang
Point Cloud Registration (PCR) is a critical and challenging task in computer
vision. One of the primary difficulties in PCR is identifying salient and
meaningful points that exhibit consistent semantic and geometric properties
across different scans. Previous methods have encountered challenges with
ambiguous matching due to the similarity among patch blocks throughout the
entire point cloud and the lack of consideration for efficient glob... more
Point Cloud Registration (PCR) is a critical and challenging task in computer
vision. One of the primary difficulties in PCR is identifying salient and
meaningful points that exhibit consistent semantic and geometric properties
across different scans. Previous methods have encountered challenges with
ambiguous matching due to the similarity among patch blocks throughout the
entire point cloud and the lack of consideration for efficient global geometric
consistency. To address these issues, we propose a new framework that includes
several novel techniques. Firstly, we introduce a semantic-aware geometric
encoder that combines object-level and patch-level semantic information. This
encoder significantly improves registration recall by reducing ambiguity in
patch-level superpoint matching. Additionally, we incorporate a prior knowledge
approach that utilizes an intrinsic shape signature to identify salient points.
This enables us to extract the most salient super points and meaningful dense
points in the scene. Secondly, we introduce an innovative transformer that
encodes High-Order (HO) geometric features. These features are crucial for
identifying salient points within initial overlap regions while considering
global high-order geometric consistency. To optimize this high-order
transformer further, we introduce an anchor node selection strategy. By
encoding inter-frame triangle or polyhedron consistency features based on these
anchor nodes, we can effectively learn high-order geometric features of salient
super points. These high-order features are then propagated to dense points and
utilized by a Sinkhorn matching module to identify key correspondences for
successful registration. In our experiments conducted on well-known datasets
such as 3DMatch/3DLoMatch and KITTI, our approach has shown promising results,
highlighting the effectiveness of our novel method.
less
Self-Training and Multi-Task Learning for Limited Data: Evaluation Study on Object Detection
0upvotes
By: Hoàng-Ân Lê, Minh-Tan Pham
Self-training allows a network to learn from the predictions of a more
complicated model, thus often requires well-trained teacher models and mixture
of teacher-student data while multi-task learning jointly optimizes different
targets to learn salient interrelationship and requires multi-task annotations
for each training example. These frameworks, despite being particularly data
demanding have potentials for data exploitation if such assu... more
Self-training allows a network to learn from the predictions of a more
complicated model, thus often requires well-trained teacher models and mixture
of teacher-student data while multi-task learning jointly optimizes different
targets to learn salient interrelationship and requires multi-task annotations
for each training example. These frameworks, despite being particularly data
demanding have potentials for data exploitation if such assumptions can be
relaxed. In this paper, we compare self-training object detection under the
deficiency of teacher training data where students are trained on unseen
examples by the teacher, and multi-task learning with partially annotated data,
i.e. single-task annotation per training example. Both scenarios have their own
limitation but potentially helpful with limited annotated data. Experimental
results show the improvement of performance when using a weak teacher with
unseen data for training a multi-task student. Despite the limited setup we
believe the experimental results show the potential of multi-task knowledge
distillation and self-training, which could be beneficial for future study.
Source code is at https://lhoangan.github.io/multas.
less
By: Oana Ignat, Santiago Castro, Weiji Li, Rada Mihalcea
We introduce the task of automatic human action co-occurrence identification,
i.e., determine whether two human actions can co-occur in the same interval of
time. We create and make publicly available the ACE (Action Co-occurrencE)
dataset, consisting of a large graph of ~12k co-occurring pairs of visual
actions and their corresponding video clips. We describe graph link prediction
models that leverage visual and textual information to auto... more
We introduce the task of automatic human action co-occurrence identification,
i.e., determine whether two human actions can co-occur in the same interval of
time. We create and make publicly available the ACE (Action Co-occurrencE)
dataset, consisting of a large graph of ~12k co-occurring pairs of visual
actions and their corresponding video clips. We describe graph link prediction
models that leverage visual and textual information to automatically infer if
two actions are co-occurring. We show that graphs are particularly well suited
to capture relations between human actions, and the learned graph
representations are effective for our task and capture novel and relevant
information across different data domains. The ACE dataset and the code
introduced in this paper are publicly available at
https://github.com/MichiganNLP/vlog_action_co-occurrence.
less
By: JiaJun Zhu, Zichuan Yang, Binjie Hong, Jiacheng Song, Jiwei Wang, Tianhao Chen, Shuilan Yang, Zixun Lan, Fei Ma
Correcting students' multiple-choice answers is a repetitive and mechanical
task that can be considered an image multi-classification task. Assuming
possible options are 'abcd' and the correct option is one of the four, some
students may write incorrect symbols or options that do not exist. In this
paper, five classifications were set up - four for possible correct options and
one for other incorrect writing. This approach takes into accoun... more
Correcting students' multiple-choice answers is a repetitive and mechanical
task that can be considered an image multi-classification task. Assuming
possible options are 'abcd' and the correct option is one of the four, some
students may write incorrect symbols or options that do not exist. In this
paper, five classifications were set up - four for possible correct options and
one for other incorrect writing. This approach takes into account the
possibility of non-standard writing options.
less
By: Yake Wei, Ruoxuan Feng, Zihe Wang, Di Hu
One primary topic of multi-modal learning is to jointly incorporate
heterogeneous information from different modalities. However, most models often
suffer from unsatisfactory multi-modal cooperation, which could not jointly
utilize all modalities well. Some methods are proposed to identify and enhance
the worse learnt modality, but are often hard to provide the fine-grained
observation of multi-modal cooperation at sample-level with theoret... more
One primary topic of multi-modal learning is to jointly incorporate
heterogeneous information from different modalities. However, most models often
suffer from unsatisfactory multi-modal cooperation, which could not jointly
utilize all modalities well. Some methods are proposed to identify and enhance
the worse learnt modality, but are often hard to provide the fine-grained
observation of multi-modal cooperation at sample-level with theoretical
support. Hence, it is essential to reasonably observe and improve the
fine-grained cooperation between modalities, especially when facing realistic
scenarios where the modality discrepancy could vary across different samples.
To this end, we introduce a fine-grained modality valuation metric to evaluate
the contribution of each modality at sample-level. Via modality valuation, we
regretfully observe that the multi-modal model tends to rely on one specific
modality, resulting in other modalities being low-contributing. We further
analyze this issue and improve cooperation between modalities by enhancing the
discriminative ability of low-contributing modalities in a targeted manner.
Overall, our methods reasonably observe the fine-grained uni-modal contribution
at sample-level and achieve considerable improvement on different multi-modal
models.
less
By: Hao Yu, Xu Cheng, Wei Peng, Weihao Liu, Guoying Zhao
Visible-infrared person re-identification (VI-ReID) is a challenging task due
to large cross-modality discrepancies and intra-class variations. Existing
methods mainly focus on learning modality-shared representations by embedding
different modalities into the same feature space. As a result, the learned
feature emphasizes the common patterns across modalities while suppressing
modality-specific and identity-aware information that is valuab... more
Visible-infrared person re-identification (VI-ReID) is a challenging task due
to large cross-modality discrepancies and intra-class variations. Existing
methods mainly focus on learning modality-shared representations by embedding
different modalities into the same feature space. As a result, the learned
feature emphasizes the common patterns across modalities while suppressing
modality-specific and identity-aware information that is valuable for Re-ID. To
address these issues, we propose a novel Modality Unifying Network (MUN) to
explore a robust auxiliary modality for VI-ReID. First, the auxiliary modality
is generated by combining the proposed cross-modality learner and
intra-modality learner, which can dynamically model the modality-specific and
modality-shared representations to alleviate both cross-modality and
intra-modality variations. Second, by aligning identity centres across the
three modalities, an identity alignment loss function is proposed to discover
the discriminative feature representations. Third, a modality alignment loss is
introduced to consistently reduce the distribution distance of visible and
infrared images by modality prototype modeling. Extensive experiments on
multiple public datasets demonstrate that the proposed method surpasses the
current state-of-the-art methods by a significant margin.
less
By: Qiyu Sun, Huilin Chen, Meng Zheng, Ziyan Wu, Michael Felsberg, Yang Tang
Domain generalized semantic segmentation (DGSS) is a critical yet challenging
task, where the model is trained only on source data without access to any
target data. Despite the proposal of numerous DGSS strategies, the
generalization capability remains limited in CNN architectures. Though some
Transformer-based segmentation models show promising performance, they
primarily focus on capturing intra-sample attentive relationships, disregardi... more
Domain generalized semantic segmentation (DGSS) is a critical yet challenging
task, where the model is trained only on source data without access to any
target data. Despite the proposal of numerous DGSS strategies, the
generalization capability remains limited in CNN architectures. Though some
Transformer-based segmentation models show promising performance, they
primarily focus on capturing intra-sample attentive relationships, disregarding
inter-sample correlations which can potentially benefit DGSS. To this end, we
enhance the attention modules in Transformer networks for improving DGSS by
incorporating information from other independent samples in the same batch,
enriching contextual information, and diversifying the training data for each
attention block. Specifically, we propose two alternative intra-batch attention
mechanisms, namely mean-based intra-batch attention (MIBA) and element-wise
intra-batch attention (EIBA), to capture correlations between different
samples, enhancing feature representation and generalization capabilities.
Building upon intra-batch attention, we introduce IBAFormer, which integrates
self-attention modules with the proposed intra-batch attention for DGSS.
Extensive experiments demonstrate that IBAFormer achieves SOTA performance in
DGSS, and ablation studies further confirm the effectiveness of each introduced
component.
less
By: Yin Wang, Zhiying Leng, Frederick W. B. Li, Shun-Cheng Wu, Xiaohui Liang
Text-driven human motion generation in computer vision is both significant
and challenging. However, current methods are limited to producing either
deterministic or imprecise motion sequences, failing to effectively control the
temporal and spatial relationships required to conform to a given text
description. In this work, we propose a fine-grained method for generating
high-quality, conditional human motion sequences supporting precise t... more
Text-driven human motion generation in computer vision is both significant
and challenging. However, current methods are limited to producing either
deterministic or imprecise motion sequences, failing to effectively control the
temporal and spatial relationships required to conform to a given text
description. In this work, we propose a fine-grained method for generating
high-quality, conditional human motion sequences supporting precise text
description. Our approach consists of two key components: 1) a
linguistics-structure assisted module that constructs accurate and complete
language feature to fully utilize text information; and 2) a context-aware
progressive reasoning module that learns neighborhood and overall semantic
linguistics features from shallow and deep graph neural networks to achieve a
multi-step inference. Experiments show that our approach outperforms
text-driven motion generation methods on HumanML3D and KIT test sets and
generates better visually confirmed motion to the text conditions.
less
Jersey Number Recognition using Keyframe Identification from Low-Resolution Broadcast Videos
0upvotes
By: Bavesh Balaji, Jerrin Bright, Harish Prakash, Yuhao Chen, David A Clausi, John Zelek
Player identification is a crucial component in vision-driven soccer
analytics, enabling various downstream tasks such as player assessment, in-game
analysis, and broadcast production. However, automatically detecting jersey
numbers from player tracklets in videos presents challenges due to motion blur,
low resolution, distortions, and occlusions. Existing methods, utilizing
Spatial Transformer Networks, CNNs, and Vision Transformers, have ... more
Player identification is a crucial component in vision-driven soccer
analytics, enabling various downstream tasks such as player assessment, in-game
analysis, and broadcast production. However, automatically detecting jersey
numbers from player tracklets in videos presents challenges due to motion blur,
low resolution, distortions, and occlusions. Existing methods, utilizing
Spatial Transformer Networks, CNNs, and Vision Transformers, have shown success
in image data but struggle with real-world video data, where jersey numbers are
not visible in most of the frames. Hence, identifying frames that contain the
jersey number is a key sub-problem to tackle. To address these issues, we
propose a robust keyframe identification module that extracts frames containing
essential high-level information about the jersey number. A spatio-temporal
network is then employed to model spatial and temporal context and predict the
probabilities of jersey numbers in the video. Additionally, we adopt a
multi-task loss function to predict the probability distribution of each digit
separately. Extensive evaluations on the SoccerNet dataset demonstrate that
incorporating our proposed keyframe identification module results in a
significant 37.81% and 37.70% increase in the accuracies of 2 different test
sets with domain gaps. These results highlight the effectiveness and importance
of our approach in tackling the challenges of automatic jersey number detection
in sports videos.
less
AI4Food-NutritionFW: A Novel Framework for the Automatic Synthesis and Analysis of Eating Behaviours
0upvotes
By: Sergio Romero-Tapiador, Ruben Tolosana, Aythami Morales, Isabel Espinosa-Salinas, Gala Freixer, Julian Fierrez, Ruben Vera-Rodriguez, Enrique Carrillo de Santa Pau, Ana Ramírez de Molina, Javier Ortega-Garcia
Nowadays millions of images are shared on social media and web platforms. In
particular, many of them are food images taken from a smartphone over time,
providing information related to the individual's diet. On the other hand,
eating behaviours are directly related to some of the most prevalent diseases
in the world. Exploiting recent advances in image processing and Artificial
Intelligence (AI), this scenario represents an excellent oppor... more
Nowadays millions of images are shared on social media and web platforms. In
particular, many of them are food images taken from a smartphone over time,
providing information related to the individual's diet. On the other hand,
eating behaviours are directly related to some of the most prevalent diseases
in the world. Exploiting recent advances in image processing and Artificial
Intelligence (AI), this scenario represents an excellent opportunity to: i)
create new methods that analyse the individuals' health from what they eat, and
ii) develop personalised recommendations to improve nutrition and diet under
specific circumstances (e.g., obesity or COVID). Having tunable tools for
creating food image datasets that facilitate research in both lines is very
much needed.
This paper proposes AI4Food-NutritionFW, a framework for the creation of food
image datasets according to configurable eating behaviours. AI4Food-NutritionFW
simulates a user-friendly and widespread scenario where images are taken using
a smartphone. In addition to the framework, we also provide and describe a
unique food image dataset that includes 4,800 different weekly eating
behaviours from 15 different profiles and 1,200 subjects. Specifically, we
consider profiles that comply with actual lifestyles from healthy eating
behaviours (according to established knowledge), variable profiles (e.g.,
eating out, holidays), to unhealthy ones (e.g., excess of fast food or sweets).
Finally, we automatically evaluate a healthy index of the subject's eating
behaviours using multidimensional metrics based on guidelines for healthy diets
proposed by international organisations, achieving promising results (99.53%
and 99.60% accuracy and sensitivity, respectively). We also release to the
research community a software implementation of our proposed
AI4Food-NutritionFW and the mentioned food image dataset created with it.
less
By: Maria Priisalu
It is difficult to perform 3D reconstruction from on-vehicle gathered video
due to the large forward motion of the vehicle. Even object detection and human
sensing models perform significantly worse on onboard videos when compared to
standard benchmarks because objects often appear far away from the camera
compared to the standard object detection benchmarks, image quality is often
decreased by motion blur and occlusions occur often. This h... more
It is difficult to perform 3D reconstruction from on-vehicle gathered video
due to the large forward motion of the vehicle. Even object detection and human
sensing models perform significantly worse on onboard videos when compared to
standard benchmarks because objects often appear far away from the camera
compared to the standard object detection benchmarks, image quality is often
decreased by motion blur and occlusions occur often. This has led to the
popularisation of traffic data-specific benchmarks. Recently Light Detection
And Ranging (LiDAR) sensors have become popular to directly estimate depths
without the need to perform 3D reconstructions. However, LiDAR-based methods
still lack in articulated human detection at a distance when compared to
image-based methods. We hypothesize that benchmarks targeted at articulated
human sensing from LiDAR data could bring about increased research in human
sensing and prediction in traffic and could lead to improved traffic safety for
pedestrians.
less
By: James Robert Kubricht, Zhaoyuan Yang, Jianwei Qiu, Peter Henry Tu
Deep learning approaches to natural language processing have made great
strides in recent years. While these models produce symbols that convey vast
amounts of diverse knowledge, it is unclear how such symbols are grounded in
data from the world. In this paper, we explore the development of a private
language for visual data representation by training emergent language (EL)
encoders/decoders in both i) a traditional referential game environ... more
Deep learning approaches to natural language processing have made great
strides in recent years. While these models produce symbols that convey vast
amounts of diverse knowledge, it is unclear how such symbols are grounded in
data from the world. In this paper, we explore the development of a private
language for visual data representation by training emergent language (EL)
encoders/decoders in both i) a traditional referential game environment and ii)
a contrastive learning environment utilizing a within-class matching training
paradigm. An additional classification layer utilizing neural machine
translation and random forest classification was used to transform symbolic
representations (sequences of integer symbols) to class labels. These methods
were applied in two experiments focusing on object recognition and action
recognition. For object recognition, a set of sketches produced by human
participants from real imagery was used (Sketchy dataset) and for action
recognition, 2D trajectories were generated from 3D motion capture systems
(MOVI dataset). In order to interpret the symbols produced for data in each
experiment, gradient-weighted class activation mapping (Grad-CAM) methods were
used to identify pixel regions indicating semantic features which contribute
evidence towards symbols in learned languages. Additionally, a t-distributed
stochastic neighbor embedding (t-SNE) method was used to investigate embeddings
learned by CNN feature extractors.
less
By: Kuangdai Leng, Jeyan Thiyagalingam
Convolution is a fundamental operation in image processing and machine
learning. Aimed primarily at maintaining image size, padding is a key
ingredient of convolution, which, however, can introduce undesirable boundary
effects. We present a non-padding-based method for size-keeping convolution
based on the preservation of differential characteristics of kernels. The main
idea is to make convolution over an incomplete sliding window "collaps... more
Convolution is a fundamental operation in image processing and machine
learning. Aimed primarily at maintaining image size, padding is a key
ingredient of convolution, which, however, can introduce undesirable boundary
effects. We present a non-padding-based method for size-keeping convolution
based on the preservation of differential characteristics of kernels. The main
idea is to make convolution over an incomplete sliding window "collapse" to a
linear differential operator evaluated locally at its central pixel, which no
longer requires information from the neighbouring missing pixels. While the
underlying theory is rigorous, our final formula turns out to be simple: the
convolution over an incomplete window is achieved by convolving its nearest
complete window with a transformed kernel. This formula is computationally
lightweight, involving neither interpolation or extrapolation nor restrictions
on image and kernel sizes. Our method favours data with smooth boundaries, such
as high-resolution images and fields from physics. Our experiments include: i)
filtering analytical and non-analytical fields from computational physics and,
ii) training convolutional neural networks (CNNs) for the tasks of image
classification, semantic segmentation and super-resolution reconstruction. In
all these experiments, our method has exhibited visible superiority over the
compared ones.
less
By: Jindong Gu, Fangyun Wei, Philip Torr, Han Hu
Deep Neural Networks can be easily fooled by small and imperceptible
perturbations. The query-based black-box attack (QBBA) is able to create the
perturbations using model output probabilities of image queries requiring no
access to the underlying models. QBBA poses realistic threats to real-world
applications. Recently, various types of robustness have been explored to
defend against QBBA. In this work, we first taxonomize the stochastic d... more
Deep Neural Networks can be easily fooled by small and imperceptible
perturbations. The query-based black-box attack (QBBA) is able to create the
perturbations using model output probabilities of image queries requiring no
access to the underlying models. QBBA poses realistic threats to real-world
applications. Recently, various types of robustness have been explored to
defend against QBBA. In this work, we first taxonomize the stochastic defense
strategies against QBBA. Following our taxonomy, we propose to explore
non-additive randomness in models to defend against QBBA. Specifically, we
focus on underexplored Vision Transformers based on their flexible
architectures. Extensive experiments show that the proposed defense approach
achieves effective defense, without much sacrifice in performance.
less
Attention De-sparsification Matters: Inducing Diversity in Digital Pathology Representation Learning
0upvotes
By: Saarthak Kapse, Srijan Das, Jingwei Zhang, Rajarsi R. Gupta, Joel Saltz, Dimitris Samaras, Prateek Prasanna
We propose DiRL, a Diversity-inducing Representation Learning technique for
histopathology imaging. Self-supervised learning techniques, such as
contrastive and non-contrastive approaches, have been shown to learn rich and
effective representations of digitized tissue samples with limited pathologist
supervision. Our analysis of vanilla SSL-pretrained models' attention
distribution reveals an insightful observation: sparsity in attention, i... more
We propose DiRL, a Diversity-inducing Representation Learning technique for
histopathology imaging. Self-supervised learning techniques, such as
contrastive and non-contrastive approaches, have been shown to learn rich and
effective representations of digitized tissue samples with limited pathologist
supervision. Our analysis of vanilla SSL-pretrained models' attention
distribution reveals an insightful observation: sparsity in attention, i.e,
models tends to localize most of their attention to some prominent patterns in
the image. Although attention sparsity can be beneficial in natural images due
to these prominent patterns being the object of interest itself, this can be
sub-optimal in digital pathology; this is because, unlike natural images,
digital pathology scans are not object-centric, but rather a complex phenotype
of various spatially intermixed biological components. Inadequate
diversification of attention in these complex images could result in crucial
information loss. To address this, we leverage cell segmentation to densely
extract multiple histopathology-specific representations, and then propose a
prior-guided dense pretext task for SSL, designed to match the multiple
corresponding representations between the views. Through this, the model learns
to attend to various components more closely and evenly, thus inducing adequate
diversification in attention for capturing context rich representations.
Through quantitative and qualitative analysis on multiple tasks across cancer
types, we demonstrate the efficacy of our method and observe that the attention
is more globally distributed.
less
By: Gang Fu, Qing Zhang, Lei Zhu, Chunxia Xiao, Ping Li
This paper aims to remove specular highlights from a single object-level
image. Although previous methods have made some progresses, their performance
remains somewhat limited, particularly for real images with complex specular
highlights. To this end, we propose a three-stage network to address them.
Specifically, given an input image, we first decompose it into the albedo,
shading, and specular residue components to estimate a coarse spec... more
This paper aims to remove specular highlights from a single object-level
image. Although previous methods have made some progresses, their performance
remains somewhat limited, particularly for real images with complex specular
highlights. To this end, we propose a three-stage network to address them.
Specifically, given an input image, we first decompose it into the albedo,
shading, and specular residue components to estimate a coarse specular-free
image. Then, we further refine the coarse result to alleviate its visual
artifacts such as color distortion. Finally, we adjust the tone of the refined
result to match that of the input as closely as possible. In addition, to
facilitate network training and quantitative evaluation, we present a
large-scale synthetic dataset of object-level images, covering diverse objects
and illumination conditions. Extensive experiments illustrate that our network
is able to generalize well to unseen real object-level images, and even produce
good results for scene-level images with multiple background objects and
complex lighting.
less
By: Chunqing Ruan, Hongjian Wang
Parameter efficient transfer learning (PETL) is an emerging research spot
that aims to adapt large-scale pre-trained models to downstream tasks. Recent
advances have achieved great success in saving storage and computation costs.
However, these methods do not take into account instance-specific visual clues
for visual tasks. In this paper, we propose a Dynamic Visual Prompt Tuning
framework (DVPT), which can generate a dynamic instance-wise... more
Parameter efficient transfer learning (PETL) is an emerging research spot
that aims to adapt large-scale pre-trained models to downstream tasks. Recent
advances have achieved great success in saving storage and computation costs.
However, these methods do not take into account instance-specific visual clues
for visual tasks. In this paper, we propose a Dynamic Visual Prompt Tuning
framework (DVPT), which can generate a dynamic instance-wise token for each
image. In this way, it can capture the unique visual feature of each image,
which can be more suitable for downstream visual tasks. We designed a Meta-Net
module that can generate learnable prompts based on each image, thereby
capturing dynamic instance-wise visual features. Extensive experiments on a
wide range of downstream recognition tasks show that DVPT achieves superior
performance than other PETL methods. More importantly, DVPT even outperforms
full fine-tuning on 17 out of 19 downstream tasks while maintaining high
parameter efficiency. Our code will be released soon.
less
By: Zhitong Gao, Shipeng Yan, Xuming He
Recent advancements in dense out-of-distribution (OOD) detection have
primarily focused on scenarios where the training and testing datasets share a
similar domain, with the assumption that no domain shift exists between them.
However, in real-world situations, domain shift often exits and significantly
affects the accuracy of existing out-of-distribution (OOD) detection models. In
this work, we propose a dual-level OOD detection framework ... more
Recent advancements in dense out-of-distribution (OOD) detection have
primarily focused on scenarios where the training and testing datasets share a
similar domain, with the assumption that no domain shift exists between them.
However, in real-world situations, domain shift often exits and significantly
affects the accuracy of existing out-of-distribution (OOD) detection models. In
this work, we propose a dual-level OOD detection framework to handle domain
shift and semantic shift jointly. The first level distinguishes whether domain
shift exists in the image by leveraging global low-level features, while the
second level identifies pixels with semantic shift by utilizing dense
high-level feature maps. In this way, we can selectively adapt the model to
unseen domains as well as enhance model's capacity in detecting novel classes.
We validate the efficacy of our proposed method on several OOD segmentation
benchmarks, including those with significant domain shifts and those without,
observing consistent performance improvements across various baseline models.
less
By: Stefan Ainetter, Sinisa Stekovic, Friedrich Fraundorfer, Vincent Lepetit
We present an automated and efficient approach for retrieving high-quality
CAD models of objects and their poses in a scene captured by a moving RGB-D
camera. We first investigate various objective functions to measure similarity
between a candidate CAD object model and the available data, and the best
objective function appears to be a "render-and-compare" method comparing depth
and mask rendering. We thus introduce a fast-search method th... more
We present an automated and efficient approach for retrieving high-quality
CAD models of objects and their poses in a scene captured by a moving RGB-D
camera. We first investigate various objective functions to measure similarity
between a candidate CAD object model and the available data, and the best
objective function appears to be a "render-and-compare" method comparing depth
and mask rendering. We thus introduce a fast-search method that approximates an
exhaustive search based on this objective function for simultaneously
retrieving the object category, a CAD model, and the pose of an object given an
approximate 3D bounding box. This method involves a search tree that organizes
the CAD models and object properties including object category and pose for
fast retrieval and an algorithm inspired by Monte Carlo Tree Search, that
efficiently searches this tree. We show that this method retrieves CAD models
that fit the real objects very well, with a speed-up factor of 10x to 120x
compared to exhaustive search.
less
By: Xiangyu Chen, Xintao Wang, Wenlong Zhang, Xiangtao Kong, Yu Qiao, Jiantao Zhou, Chao Dong
Transformer-based methods have shown impressive performance in image
restoration tasks, such as image super-resolution and denoising. However, we
find that these networks can only utilize a limited spatial range of input
information through attribution analysis. This implies that the potential of
Transformer is still not fully exploited in existing networks. In order to
activate more input pixels for better restoration, we propose a new Hyb... more
Transformer-based methods have shown impressive performance in image
restoration tasks, such as image super-resolution and denoising. However, we
find that these networks can only utilize a limited spatial range of input
information through attribution analysis. This implies that the potential of
Transformer is still not fully exploited in existing networks. In order to
activate more input pixels for better restoration, we propose a new Hybrid
Attention Transformer (HAT). It combines both channel attention and
window-based self-attention schemes, thus making use of their complementary
advantages. Moreover, to better aggregate the cross-window information, we
introduce an overlapping cross-attention module to enhance the interaction
between neighboring window features. In the training stage, we additionally
adopt a same-task pre-training strategy to further exploit the potential of the
model for further improvement. Extensive experiments have demonstrated the
effectiveness of the proposed modules. We further scale up the model to show
that the performance of the SR task can be greatly improved. Besides, we extend
HAT to more image restoration applications, including real-world image
super-resolution, Gaussian image denoising and image compression artifacts
reduction. Experiments on benchmark and real-world datasets demonstrate that
our HAT achieves state-of-the-art performance both quantitatively and
qualitatively. Codes and models are publicly available at
https://github.com/XPixelGroup/HAT.
less
By: Yiming Zhang, ZeMing Gong, Angel X. Chang
We introduce the task of localizing a flexible number of objects in
real-world 3D scenes using natural language descriptions. Existing 3D visual
grounding tasks focus on localizing a unique object given a text description.
However, such a strict setting is unnatural as localizing potentially multiple
objects is a common need in real-world scenarios and robotic tasks (e.g.,
visual navigation and object rearrangement). To address this setting... more
We introduce the task of localizing a flexible number of objects in
real-world 3D scenes using natural language descriptions. Existing 3D visual
grounding tasks focus on localizing a unique object given a text description.
However, such a strict setting is unnatural as localizing potentially multiple
objects is a common need in real-world scenarios and robotic tasks (e.g.,
visual navigation and object rearrangement). To address this setting we propose
Multi3DRefer, generalizing the ScanRefer dataset and task. Our dataset contains
61926 descriptions of 11609 objects, where zero, single or multiple target
objects are referenced by each description. We also introduce a new evaluation
metric and benchmark methods from prior work to enable further investigation of
multi-modal 3D scene understanding. Furthermore, we develop a better baseline
leveraging 2D features from CLIP by rendering object proposals online with
contrastive learning, which outperforms the state of the art on the ScanRefer
benchmark.
less
By: Jinfeng Liu, Lingtong Kong, Jie Yang, Wei Liu
Depth estimation plays an important role in the robotic perception system.
Self-supervised monocular paradigm has gained significant attention since it
can free training from the reliance on depth annotations. Despite recent
advancements, existing self-supervised methods still underutilize the available
training data, limiting their generalization ability. In this paper, we take
two data augmentation techniques, namely Resizing-Cropping and... more
Depth estimation plays an important role in the robotic perception system.
Self-supervised monocular paradigm has gained significant attention since it
can free training from the reliance on depth annotations. Despite recent
advancements, existing self-supervised methods still underutilize the available
training data, limiting their generalization ability. In this paper, we take
two data augmentation techniques, namely Resizing-Cropping and
Splitting-Permuting, to fully exploit the potential of training datasets.
Specifically, the original image and the generated two augmented images are fed
into the training pipeline simultaneously and we leverage them to conduct
self-distillation. Additionally, we introduce the detail-enhanced DepthNet with
an extra full-scale branch in the encoder and a grid decoder to enhance the
restoration of fine details in depth maps. Experimental results demonstrate our
method can achieve state-of-the-art performance on the KITTI benchmark, with
both raw ground truth and improved ground truth. Moreover, our models also show
superior generalization performance when transferring to Make3D and NYUv2
datasets. Our codes are available at https://github.com/Sauf4896/BDEdepth.
less
By: Chunyong Hu, Hang Zheng, Kun Li, Jianyun Xu, Weibo Mao, Maochun Luo, Lingxuan Wang, Mingxia Chen, Kaixuan Liu, Yiru Zhao, Peihan Hao, Minzhe Liu, Kaicheng Yu
Multi-sensor modal fusion has demonstrated strong advantages in 3D object
detection tasks. However, existing methods that fuse multi-modal features
through a simple channel concatenation require transformation features into
bird's eye view space and may lose the information on Z-axis thus leads to
inferior performance. To this end, we propose FusionFormer, an end-to-end
multi-modal fusion framework that leverages transformers to fuse multi-... more
Multi-sensor modal fusion has demonstrated strong advantages in 3D object
detection tasks. However, existing methods that fuse multi-modal features
through a simple channel concatenation require transformation features into
bird's eye view space and may lose the information on Z-axis thus leads to
inferior performance. To this end, we propose FusionFormer, an end-to-end
multi-modal fusion framework that leverages transformers to fuse multi-modal
features and obtain fused BEV features. And based on the flexible adaptability
of FusionFormer to the input modality representation, we propose a depth
prediction branch that can be added to the framework to improve detection
performance in camera-based detection tasks. In addition, we propose a
plug-and-play temporal fusion module based on transformers that can fuse
historical frame BEV features for more stable and reliable detection results.
We evaluate our method on the nuScenes dataset and achieve 72.6% mAP and 75.1%
NDS for 3D object detection tasks, outperforming state-of-the-art methods.
less
By: Jiaxin Gao, Ziyu Yue, Yaohua Liu, Sihan Xie, Xin Fan, Risheng Liu
Super-resolution tasks oriented to images captured in ultra-dark environments
is a practical yet challenging problem that has received little attention. Due
to uneven illumination and low signal-to-noise ratio in dark environments, a
multitude of problems such as lack of detail and color distortion may be
magnified in the super-resolution process compared to normal-lighting
environments. Consequently, conventional low-light enhancement or
s... more
Super-resolution tasks oriented to images captured in ultra-dark environments
is a practical yet challenging problem that has received little attention. Due
to uneven illumination and low signal-to-noise ratio in dark environments, a
multitude of problems such as lack of detail and color distortion may be
magnified in the super-resolution process compared to normal-lighting
environments. Consequently, conventional low-light enhancement or
super-resolution methods, whether applied individually or in a cascaded manner
for such problem, often encounter limitations in recovering luminance, color
fidelity, and intricate details. To conquer these issues, this paper proposes a
specialized dual-modulated learning framework that, for the first time,
attempts to deeply dissect the nature of the low-light super-resolution task.
Leveraging natural image color characteristics, we introduce a self-regularized
luminance constraint as a prior for addressing uneven lighting. Expanding on
this, we develop Illuminance-Semantic Dual Modulation (ISDM) components to
enhance feature-level preservation of illumination and color details. Besides,
instead of deploying naive up-sampling strategies, we design the
Resolution-Sensitive Merging Up-sampler (RSMU) module that brings together
different sampling modalities as substrates, effectively mitigating the
presence of artifacts and halos. Comprehensive experiments showcases the
applicability and generalizability of our approach to diverse and challenging
ultra-low-light conditions, outperforming state-of-the-art methods with a
notable improvement (i.e., $\uparrow$5\% in PSNR, and $\uparrow$43\% in LPIPS).
Especially noteworthy is the 19-fold increase in the RMSE score, underscoring
our method's exceptional generalization across different darkness levels. The
code will be available online upon publication of the paper.
less
By: Shentong Mo, Weiguo Pian, Yapeng Tian
Continual learning is a challenging problem in which models need to be
trained on non-stationary data across sequential tasks for class-incremental
learning. While previous methods have focused on using either regularization or
rehearsal-based frameworks to alleviate catastrophic forgetting in image
classification, they are limited to a single modality and cannot learn compact
class-aware cross-modal representations for continual audio-visu... more
Continual learning is a challenging problem in which models need to be
trained on non-stationary data across sequential tasks for class-incremental
learning. While previous methods have focused on using either regularization or
rehearsal-based frameworks to alleviate catastrophic forgetting in image
classification, they are limited to a single modality and cannot learn compact
class-aware cross-modal representations for continual audio-visual learning. To
address this gap, we propose a novel class-incremental grouping network (CIGN)
that can learn category-wise semantic features to achieve continual
audio-visual learning. Our CIGN leverages learnable audio-visual class tokens
and audio-visual grouping to continually aggregate class-aware features.
Additionally, it utilizes class tokens distillation and continual grouping to
prevent forgetting parameters learned from previous tasks, thereby improving
the model's ability to capture discriminative audio-visual categories. We
conduct extensive experiments on VGGSound-Instruments, VGGSound-100, and
VGG-Sound Sources benchmarks. Our experimental results demonstrate that the
CIGN achieves state-of-the-art audio-visual class-incremental learning
performance. Code is available at https://github.com/stoneMo/CIGN.
less
By: Mihir Kulkarni, Kostas Alexis
This paper contributes a novel learning-based method for aggressive
task-driven compression of depth images and their encoding as images tailored
to collision prediction for robotic systems. A novel 3D image processing
methodology is proposed that accounts for the robot's size in order to
appropriately "inflate" the obstacles represented in the depth image and thus
obtain the distance that can be traversed by the robot in a collision-free
m... more
This paper contributes a novel learning-based method for aggressive
task-driven compression of depth images and their encoding as images tailored
to collision prediction for robotic systems. A novel 3D image processing
methodology is proposed that accounts for the robot's size in order to
appropriately "inflate" the obstacles represented in the depth image and thus
obtain the distance that can be traversed by the robot in a collision-free
manner along any given ray within the camera frustum. Such depth-and-collision
image pairs are used to train a neural network that follows the architecture of
Variational Autoencoders to compress-and-transform the information in the
original depth image to derive a latent representation that encodes the
collision information for the given depth image. We compare our proposed
task-driven encoding method with classical task-agnostic methods and
demonstrate superior performance for the task of collision image prediction
from extremely low-dimensional latent spaces. A set of comparative studies show
that the proposed approach is capable of encoding depth image-and-collision
image tuples from complex scenes with thin obstacles at long distances better
than the classical methods at compression ratios as high as 4050:1.
less
By: Eden Belouadah, Arnaud Dapogny, Kevin Bailly
Class-Incremental learning (CIL) is the ability of artificial agents to
accommodate new classes as they appear in a stream. It is particularly
interesting in evolving environments where agents have limited access to memory
and computational resources. The main challenge of class-incremental learning
is catastrophic forgetting, the inability of neural networks to retain past
knowledge when learning a new one. Unfortunately, most existing
cla... more
Class-Incremental learning (CIL) is the ability of artificial agents to
accommodate new classes as they appear in a stream. It is particularly
interesting in evolving environments where agents have limited access to memory
and computational resources. The main challenge of class-incremental learning
is catastrophic forgetting, the inability of neural networks to retain past
knowledge when learning a new one. Unfortunately, most existing
class-incremental object detectors are applied to two-stage algorithms such as
Faster-RCNN and rely on rehearsal memory to retain past knowledge. We believe
that the current benchmarks are not realistic, and more effort should be
dedicated to anchor-free and rehearsal-free object detection. In this context,
we propose MultIOD, a class-incremental object detector based on CenterNet. Our
main contributions are: (1) we propose a multihead feature pyramid and
multihead detection architecture to efficiently separate class representations,
(2) we employ transfer learning between classes learned initially and those
learned incrementally to tackle catastrophic forgetting, and (3) we use a
class-wise non-max-suppression as a post-processing technique to remove
redundant boxes. Without bells and whistles, our method outperforms a range of
state-of-the-art methods on two Pascal VOC datasets.
less
By: Xiao He, Mingrui Zhu, Dongxin Chen, Nannan Wang, Xinbo Gao
Privacy protection has become a top priority as the proliferation of AI
techniques has led to widespread collection and misuse of personal data.
Anonymization and visual identity information hiding are two important facial
privacy protection tasks that aim to remove identification characteristics from
facial images at the human perception level. However, they have a significant
difference in that the former aims to prevent the machine from ... more
Privacy protection has become a top priority as the proliferation of AI
techniques has led to widespread collection and misuse of personal data.
Anonymization and visual identity information hiding are two important facial
privacy protection tasks that aim to remove identification characteristics from
facial images at the human perception level. However, they have a significant
difference in that the former aims to prevent the machine from recognizing
correctly, while the latter needs to ensure the accuracy of machine
recognition. Therefore, it is difficult to train a model to complete these two
tasks simultaneously. In this paper, we unify the task of anonymization and
visual identity information hiding and propose a novel face privacy protection
method based on diffusion models, dubbed Diff-Privacy. Specifically, we train
our proposed multi-scale image inversion module (MSI) to obtain a set of SDM
format conditional embeddings of the original image. Based on the conditional
embeddings, we design corresponding embedding scheduling strategies and
construct different energy functions during the denoising process to achieve
anonymization and visual identity information hiding. Extensive experiments
have been conducted to validate the effectiveness of our proposed framework in
protecting facial privacy.
less
By: Chenghao Li, Chaoning Zhang
The success of Vision Transformer (ViT) has been widely reported on a wide
range of image recognition tasks. The merit of ViT over CNN has been largely
attributed to large training datasets or auxiliary pre-training. Without
pre-training, the performance of ViT on small datasets is limited because the
global self-attention has limited capacity in local modeling. Towards boosting
ViT on small datasets without pre-training, this work improves... more
The success of Vision Transformer (ViT) has been widely reported on a wide
range of image recognition tasks. The merit of ViT over CNN has been largely
attributed to large training datasets or auxiliary pre-training. Without
pre-training, the performance of ViT on small datasets is limited because the
global self-attention has limited capacity in local modeling. Towards boosting
ViT on small datasets without pre-training, this work improves its local
modeling by applying a weight mask on the original self-attention matrix. A
straightforward way to locally adapt the self-attention matrix can be realized
by an element-wise learnable weight mask (ELM), for which our preliminary
results show promising results. However, the element-wise simple learnable
weight mask not only induces a non-trivial additional parameter overhead but
also increases the optimization complexity. To this end, this work proposes a
novel Gaussian mixture mask (GMM) in which one mask only has two learnable
parameters and it can be conveniently used in any ViT variants whose attention
mechanism allows the use of masks. Experimental results on multiple small
datasets demonstrate that the effectiveness of our proposed Gaussian mask for
boosting ViTs for free (almost zero additional parameter or computation cost).
Our code will be publicly available at
\href{https://github.com/CatworldLee/Gaussian-Mixture-Mask-Attention}{https://github.com/CatworldLee/Gaussian-Mixture-Mask-Attention}.
less
FlowIBR: Leveraging Pre-Training for Efficient Neural Image-Based Rendering of Dynamic Scenes
0upvotes
By: Marcel Büsching, Josef Bengtson, David Nilsson, Mårten Björkman
We introduce a novel approach for monocular novel view synthesis of dynamic
scenes. Existing techniques already show impressive rendering quality but tend
to focus on optimization within a single scene without leveraging prior
knowledge. This limitation has been primarily attributed to the lack of
datasets of dynamic scenes available for training and the diversity of scene
dynamics. Our method FlowIBR circumvents these issues by integrating... more
We introduce a novel approach for monocular novel view synthesis of dynamic
scenes. Existing techniques already show impressive rendering quality but tend
to focus on optimization within a single scene without leveraging prior
knowledge. This limitation has been primarily attributed to the lack of
datasets of dynamic scenes available for training and the diversity of scene
dynamics. Our method FlowIBR circumvents these issues by integrating a neural
image-based rendering method, pre-trained on a large corpus of widely available
static scenes, with a per-scene optimized scene flow field. Utilizing this flow
field, we bend the camera rays to counteract the scene dynamics, thereby
presenting the dynamic scene as if it were static to the rendering network. The
proposed method reduces per-scene optimization time by an order of magnitude,
achieving comparable results to existing methods - all on a single
consumer-grade GPU.
less
By: Haoran Chen, Kenneth Blomqvist, Francesco Milano, Roland Siegwart
Recently, methods have been proposed for 3D open-vocabulary semantic
segmentation. Such methods are able to segment scenes into arbitrary classes
given at run-time using their text description. In this paper, we propose to
our knowledge the first algorithm for open-vocabulary panoptic segmentation,
simultaneously performing both semantic and instance segmentation. Our
algorithm, Panoptic Vision-Language Feature Fields (PVLFF) learns a featu... more
Recently, methods have been proposed for 3D open-vocabulary semantic
segmentation. Such methods are able to segment scenes into arbitrary classes
given at run-time using their text description. In this paper, we propose to
our knowledge the first algorithm for open-vocabulary panoptic segmentation,
simultaneously performing both semantic and instance segmentation. Our
algorithm, Panoptic Vision-Language Feature Fields (PVLFF) learns a feature
field of the scene, jointly learning vision-language features and hierarchical
instance features through a contrastive loss function from 2D instance segment
proposals on input frames. Our method achieves comparable performance against
the state-of-the-art close-set 3D panoptic systems on the HyperSim, ScanNet and
Replica dataset and outperforms current 3D open-vocabulary systems in terms of
semantic segmentation. We additionally ablate our method to demonstrate the
effectiveness of our model architecture. Our code will be available at
https://github.com/ethz-asl/autolabel.
less
By: Sven Teufel, Jörg Gamerdinger, Georg Volk, Oliver Bringmann
Comprehensive perception of the environment is crucial for the safe operation
of autonomous vehicles. However, the perception capabilities of autonomous
vehicles are limited due to occlusions, limited sensor ranges, or environmental
influences. Collective Perception (CP) aims to mitigate these problems by
enabling the exchange of information between vehicles. A major challenge in CP
is the fusion of the exchanged information. Due to the eno... more
Comprehensive perception of the environment is crucial for the safe operation
of autonomous vehicles. However, the perception capabilities of autonomous
vehicles are limited due to occlusions, limited sensor ranges, or environmental
influences. Collective Perception (CP) aims to mitigate these problems by
enabling the exchange of information between vehicles. A major challenge in CP
is the fusion of the exchanged information. Due to the enormous bandwidth
requirement of early fusion approaches and the interchangeability issues of
intermediate fusion approaches, only the late fusion of shared detections is
practical. Current late fusion approaches neglect valuable information for
local detection, this is why we propose a novel fusion method to fuse the
detections of cooperative vehicles within the local LiDAR-based detection
pipeline. Therefore, we present Collective PV-RCNN (CPV-RCNN), which extends
the PV-RCNN++ framework to fuse collective detections. Code is available at
https://github.com/ekut-es
less
By: Santiago Rivier, Carlos Hinojosa, Silvio Giancola, Bernard Ghanem
Semantic segmentation is crucial in remote sensing, where high-resolution
satellite images are segmented into meaningful regions. Recent advancements in
deep learning have significantly improved satellite image segmentation.
However, most of these methods are typically trained in fully supervised
settings that require high-quality pixel-level annotations, which are expensive
and time-consuming to obtain. In this work, we present a weakly su... more
Semantic segmentation is crucial in remote sensing, where high-resolution
satellite images are segmented into meaningful regions. Recent advancements in
deep learning have significantly improved satellite image segmentation.
However, most of these methods are typically trained in fully supervised
settings that require high-quality pixel-level annotations, which are expensive
and time-consuming to obtain. In this work, we present a weakly supervised
learning algorithm to train semantic segmentation algorithms that only rely on
query point annotations instead of full mask labels. Our proposed approach
performs accurate semantic segmentation and improves efficiency by
significantly reducing the cost and time required for manual annotation.
Specifically, we generate superpixels and extend the query point labels into
those superpixels that group similar meaningful semantics. Then, we train
semantic segmentation models, supervised with images partially labeled with the
superpixels pseudo-labels. We benchmark our weakly supervised training approach
on an aerial image dataset and different semantic segmentation architectures,
showing that we can reach competitive performance compared to fully supervised
training while reducing the annotation effort.
less
By: Haoke Xiao, Lv Tang, Bo Li, Zhiming Luo, Shaozi Li
Co-salient Object Detection (CoSOD) endeavors to replicate the human visual
system's capacity to recognize common and salient objects within a collection
of images. Despite recent advancements in deep learning models, these models
still rely on training with well-annotated CoSOD datasets. The exploration of
training-free zero-shot CoSOD frameworks has been limited. In this paper,
taking inspiration from the zero-shot transfer capabilities o... more
Co-salient Object Detection (CoSOD) endeavors to replicate the human visual
system's capacity to recognize common and salient objects within a collection
of images. Despite recent advancements in deep learning models, these models
still rely on training with well-annotated CoSOD datasets. The exploration of
training-free zero-shot CoSOD frameworks has been limited. In this paper,
taking inspiration from the zero-shot transfer capabilities of foundational
computer vision models, we introduce the first zero-shot CoSOD framework that
harnesses these models without any training process. To achieve this, we
introduce two novel components in our proposed framework: the group prompt
generation (GPG) module and the co-saliency map generation (CMP) module. We
evaluate the framework's performance on widely-used datasets and observe
impressive results. Our approach surpasses existing unsupervised methods and
even outperforms fully supervised methods developed before 2020, while
remaining competitive with some fully supervised methods developed before 2022.
less
By: Sebastian Schmidt BMW and TUM, Stephan Günnemann TUM
Active learning (AL) reduces the amount of labeled data needed to train a
machine learning model by intelligently choosing which instances to label.
Classic pool-based AL requires all data to be present in a datacenter, which
can be challenging with the increasing amounts of data needed in deep learning.
However, AL on mobile devices and robots, like autonomous cars, can filter the
data from perception sensor streams before reaching the dat... more
Active learning (AL) reduces the amount of labeled data needed to train a
machine learning model by intelligently choosing which instances to label.
Classic pool-based AL requires all data to be present in a datacenter, which
can be challenging with the increasing amounts of data needed in deep learning.
However, AL on mobile devices and robots, like autonomous cars, can filter the
data from perception sensor streams before reaching the datacenter. We
exploited the temporal properties for such image streams in our work and
proposed the novel temporal predicted loss (TPL) method. To evaluate the
stream-based setting properly, we introduced the GTA V streets and the A2D2
streets dataset and made both publicly available. Our experiments showed that
our approach significantly improves the diversity of the selection while being
an uncertainty-based method. As pool-based approaches are more common in
perception applications, we derived a concept for comparing pool-based and
stream-based AL, where TPL out-performed state-of-the-art pool- or stream-based
approaches for different models. TPL demonstrated a gain of 2.5 precept points
(pp) less required data while being significantly faster than pool-based
methods.
less
By: Loïc Le Bescond, Maria Vakalopoulou, Stergios Christodoulidis, Fabrice André, Hugues Talbot
The deployment of machine learning solutions in real-world scenarios often
involves addressing the challenge of out-of-distribution (OOD) detection. While
significant efforts have been devoted to OOD detection in classical supervised
settings, the context of weakly supervised learning, particularly the Multiple
Instance Learning (MIL) framework, remains under-explored. In this study, we
tackle this challenge by adapting post-hoc OOD detecti... more
The deployment of machine learning solutions in real-world scenarios often
involves addressing the challenge of out-of-distribution (OOD) detection. While
significant efforts have been devoted to OOD detection in classical supervised
settings, the context of weakly supervised learning, particularly the Multiple
Instance Learning (MIL) framework, remains under-explored. In this study, we
tackle this challenge by adapting post-hoc OOD detection methods to the MIL
setting while introducing a novel benchmark specifically designed to assess OOD
detection performance in weakly supervised scenarios. Extensive experiments
based on diverse public datasets do not reveal a single method with a clear
advantage over the others. Although DICE emerges as the best-performing method
overall, it exhibits significant shortcomings on some datasets, emphasizing the
complexity of this under-explored and challenging topic. Our findings shed
light on the complex nature of OOD detection under the MIL framework,
emphasizing the importance of developing novel, robust, and reliable methods
that can generalize effectively in a weakly supervised context. The code for
the paper is available here: https://github.com/loic-lb/OOD_MIL.
less
By: Misgina Tsighe Hagos, Niamh Belton, Kathleen M. Curran, Brian Mac Namee
eXplanation Based Learning (XBL) is an interactive learning approach that
provides a transparent method of training deep learning models by interacting
with their explanations. XBL augments loss functions to penalize a model based
on deviation of its explanations from user annotation of image features. The
literature on XBL mostly depends on the intersection of visual model
explanations and image feature annotations. We present a method to ... more
eXplanation Based Learning (XBL) is an interactive learning approach that
provides a transparent method of training deep learning models by interacting
with their explanations. XBL augments loss functions to penalize a model based
on deviation of its explanations from user annotation of image features. The
literature on XBL mostly depends on the intersection of visual model
explanations and image feature annotations. We present a method to add a
distance-aware explanation loss to categorical losses that trains a learner to
focus on important regions of a training dataset. Distance is an appropriate
approach for calculating explanation loss since visual model explanations such
as Gradient-weighted Class Activation Mapping (Grad-CAMs) are not strictly
bounded as annotations and their intersections may not provide complete
information on the deviation of a model's focus from relevant image regions. In
addition to assessing our model using existing metrics, we propose an
interpretability metric for evaluating visual feature-attribution based model
explanations that is more informative of the model's performance than existing
metrics. We demonstrate performance of our proposed method on three image
classification tasks.
less
By: Dingfeng Shi, Qiong Cao, Yujie Zhong, Shan An, Jian Cheng, Haogang Zhu, Dacheng Tao
Temporal action detection (TAD) aims to detect all action boundaries and
their corresponding categories in an untrimmed video. The unclear boundaries of
actions in videos often result in imprecise predictions of action boundaries by
existing methods. To resolve this issue, we propose a one-stage framework named
TriDet. First, we propose a Trident-head to model the action boundary via an
estimated relative probability distribution around the... more
Temporal action detection (TAD) aims to detect all action boundaries and
their corresponding categories in an untrimmed video. The unclear boundaries of
actions in videos often result in imprecise predictions of action boundaries by
existing methods. To resolve this issue, we propose a one-stage framework named
TriDet. First, we propose a Trident-head to model the action boundary via an
estimated relative probability distribution around the boundary. Then, we
analyze the rank-loss problem (i.e. instant discriminability deterioration) in
transformer-based methods and propose an efficient scalable-granularity
perception (SGP) layer to mitigate this issue. To further push the limit of
instant discriminability in the video backbone, we leverage the strong
representation capability of pretrained large models and investigate their
performance on TAD. Last, considering the adequate spatial-temporal context for
classification, we design a decoupled feature pyramid network with separate
feature pyramids to incorporate rich spatial context from the large model for
localization. Experimental results demonstrate the robustness of TriDet and its
state-of-the-art performance on multiple TAD datasets, including hierarchical
(multilabel) TAD datasets.
less
By: Bo Pang, Zhongtian Zheng, Guoping Wang, Peng-Shuai Wang
We present GeGnn, a learning-based method for computing the approximate
geodesic distance between two arbitrary points on discrete polyhedra surfaces
with constant time complexity after fast precomputation. Previous relevant
methods either focus on computing the geodesic distance between a single source
and all destinations, which has linear complexity at least or require a long
precomputation time. Our key idea is to train a graph neural n... more
We present GeGnn, a learning-based method for computing the approximate
geodesic distance between two arbitrary points on discrete polyhedra surfaces
with constant time complexity after fast precomputation. Previous relevant
methods either focus on computing the geodesic distance between a single source
and all destinations, which has linear complexity at least or require a long
precomputation time. Our key idea is to train a graph neural network to embed
an input mesh into a high-dimensional embedding space and compute the geodesic
distance between a pair of points using the corresponding embedding vectors and
a lightweight decoding function. To facilitate the learning of the embedding,
we propose novel graph convolution and graph pooling modules that incorporate
local geodesic information and are verified to be much more effective than
previous designs. After training, our method requires only one forward pass of
the network per mesh as precomputation. Then, we can compute the geodesic
distance between a pair of points using our decoding function, which requires
only several matrix multiplications and can be massively parallelized on GPUs.
We verify the efficiency and effectiveness of our method on ShapeNet and
demonstrate that our method is faster than existing methods by orders of
magnitude while achieving comparable or better accuracy. Additionally, our
method exhibits robustness on noisy and incomplete meshes and strong
generalization ability on out-of-distribution meshes. The code and pretrained
model can be found on https://github.com/IntelligentGeometry/GeGnn.
less
By: Jordan A. James, Heather K. Manching, Matthew R. Mattia, Kim D. Bowman, Amanda M. Hulse-Kemp, William J. Beksi
In this letter, we present a new dataset to advance the state of the art in
detecting citrus fruit and accurately estimate yield on trees affected by the
Huanglongbing (HLB) disease in orchard environments via imaging. Despite the
fact that significant progress has been made in solving the fruit detection
problem, the lack of publicly available datasets has complicated direct
comparison of results. For instance, citrus detection has long be... more
In this letter, we present a new dataset to advance the state of the art in
detecting citrus fruit and accurately estimate yield on trees affected by the
Huanglongbing (HLB) disease in orchard environments via imaging. Despite the
fact that significant progress has been made in solving the fruit detection
problem, the lack of publicly available datasets has complicated direct
comparison of results. For instance, citrus detection has long been of interest
in the agricultural research community, yet there is an absence of work,
particularly involving public datasets of citrus affected by HLB. To address
this issue, we enhance state-of-the-art object detection methods for use in
typical orchard settings. Concretely, we provide high-resolution images of
citrus trees located in an area known to be highly affected by HLB, along with
high-quality bounding box annotations of citrus fruit. Fruit on both the trees
and the ground are labeled to allow for identification of fruit location, which
contributes to advancements in yield estimation and potential measure of HLB
impact via fruit drop. The dataset consists of over 32,000 bounding box
annotations for fruit instances contained in 579 high-resolution images. In
summary, our contributions are the following: (i) we introduce a novel dataset
along with baseline performance benchmarks on multiple contemporary object
detection algorithms, (ii) we show the ability to accurately capture fruit
location on tree or on ground, and finally (ii) we present a correlation of our
results with yield estimations.
less
By: Zheng Wang, Dong Xie, Hanzhi Wang, Jiang Tian
Learning from the limited amount of labeled data to the pre-train model has
always been viewed as a challenging task. In this report, an effective and
robust solution, the two-stage training paradigm YOLOv8 detector (TP-YOLOv8),
is designed for the object detection track in VIPriors Challenge 2023. First,
the backbone of YOLOv8 is pre-trained as the encoder using the masked image
modeling technique. Then the detector is fine-tuned with elab... more
Learning from the limited amount of labeled data to the pre-train model has
always been viewed as a challenging task. In this report, an effective and
robust solution, the two-stage training paradigm YOLOv8 detector (TP-YOLOv8),
is designed for the object detection track in VIPriors Challenge 2023. First,
the backbone of YOLOv8 is pre-trained as the encoder using the masked image
modeling technique. Then the detector is fine-tuned with elaborate
augmentations. During the test stage, test-time augmentation (TTA) is used to
enhance each model, and weighted box fusion (WBF) is implemented to further
boost the performance. With the well-designed structure, our approach has
achieved 30.4% average precision from 0.50 to 0.95 on the DelftBikes test set,
ranking 4th on the leaderboard.
less
By: Seong Hun Lee, Javier Civera
In this work, we propose a novel method for robust single rotation averaging
that can efficiently handle an extremely large fraction of outliers. Our
approach is to minimize the total truncated least unsquared deviations (TLUD)
cost of geodesic distances. The proposed algorithm consists of three steps:
First, we consider each input rotation as a potential initial solution and
choose the one that yields the least sum of truncated chordal dev... more
In this work, we propose a novel method for robust single rotation averaging
that can efficiently handle an extremely large fraction of outliers. Our
approach is to minimize the total truncated least unsquared deviations (TLUD)
cost of geodesic distances. The proposed algorithm consists of three steps:
First, we consider each input rotation as a potential initial solution and
choose the one that yields the least sum of truncated chordal deviations. Next,
we obtain the inlier set using the initial solution and compute its chordal
$L_2$-mean. Finally, starting from this estimate, we iteratively compute the
geodesic $L_1$-mean of the inliers using the Weiszfeld algorithm on $SO(3)$. An
extensive evaluation shows that our method is robust against up to 99% outliers
given a sufficient number of accurate inliers, outperforming the current state
of the art.
less
By: Soumen Basu, Ashish Papanai, Mayank Gupta, Pankaj Gupta, Chetan Arora
Automated detection of Gallbladder Cancer (GBC) from Ultrasound (US) images
is an important problem, which has drawn increased interest from researchers.
However, most of these works use difficult-to-acquire information such as
bounding box annotations or additional US videos. In this paper, we focus on
GBC detection using only image-level labels. Such annotation is usually
available based on the diagnostic report of a patient, and do not r... more
Automated detection of Gallbladder Cancer (GBC) from Ultrasound (US) images
is an important problem, which has drawn increased interest from researchers.
However, most of these works use difficult-to-acquire information such as
bounding box annotations or additional US videos. In this paper, we focus on
GBC detection using only image-level labels. Such annotation is usually
available based on the diagnostic report of a patient, and do not require
additional annotation effort from the physicians. However, our analysis reveals
that it is difficult to train a standard image classification model for GBC
detection. This is due to the low inter-class variance (a malignant region
usually occupies only a small portion of a US image), high intra-class variance
(due to the US sensor capturing a 2D slice of a 3D object leading to large
viewpoint variations), and low training data availability. We posit that even
when we have only the image level label, still formulating the problem as
object detection (with bounding box output) helps a deep neural network (DNN)
model focus on the relevant region of interest. Since no bounding box
annotations is available for training, we pose the problem as weakly supervised
object detection (WSOD). Motivated by the recent success of transformer models
in object detection, we train one such model, DETR, using
multi-instance-learning (MIL) with self-supervised instance selection to suit
the WSOD task. Our proposed method demonstrates an improvement of AP and
detection sensitivity over the SOTA transformer-based and CNN-based WSOD
methods. Project page is at https://gbc-iitd.github.io/wsod-gbc
less
By: Yufei Ye, Poorvi Hebbar, Abhinav Gupta, Shubham Tulsiani
We tackle the task of reconstructing hand-object interactions from short
video clips. Given an input video, our approach casts 3D inference as a
per-video optimization and recovers a neural 3D representation of the object
shape, as well as the time-varying motion and hand articulation. While the
input video naturally provides some multi-view cues to guide 3D inference,
these are insufficient on their own due to occlusions and limited viewpo... more
We tackle the task of reconstructing hand-object interactions from short
video clips. Given an input video, our approach casts 3D inference as a
per-video optimization and recovers a neural 3D representation of the object
shape, as well as the time-varying motion and hand articulation. While the
input video naturally provides some multi-view cues to guide 3D inference,
these are insufficient on their own due to occlusions and limited viewpoint
variations. To obtain accurate 3D, we augment the multi-view signals with
generic data-driven priors to guide reconstruction. Specifically, we learn a
diffusion network to model the conditional distribution of (geometric)
renderings of objects conditioned on hand configuration and category label, and
leverage it as a prior to guide the novel-view renderings of the reconstructed
scene. We empirically evaluate our approach on egocentric videos across 6
object categories, and observe significant improvements over prior single-view
and multi-view methods. Finally, we demonstrate our system's ability to
reconstruct arbitrary clips from YouTube, showing both 1st and 3rd person
interactions.
less
By: Weijian Xie, Guanyi Chu, Quanhao Qian, Yihao Yu, Hai Li, Danpeng Chen, Shangjin Zhai, Nan Wang, Hujun Bao, Guofeng Zhang
Dense SLAM based on monocular cameras does indeed have immense application
value in the field of AR/VR, especially when it is performed on a mobile
device. In this paper, we propose a novel method that integrates a light-weight
depth completion network into a sparse SLAM system using a multi-basis depth
representation, so that dense mapping can be performed online even on a mobile
phone. Specifically, we present a specifically optimized mul... more
Dense SLAM based on monocular cameras does indeed have immense application
value in the field of AR/VR, especially when it is performed on a mobile
device. In this paper, we propose a novel method that integrates a light-weight
depth completion network into a sparse SLAM system using a multi-basis depth
representation, so that dense mapping can be performed online even on a mobile
phone. Specifically, we present a specifically optimized multi-basis depth
completion network, called BBC-Net, tailored to the characteristics of
traditional sparse SLAM systems. BBC-Net can predict multiple balanced bases
and a confidence map from a monocular image with sparse points generated by
off-the-shelf keypoint-based SLAM systems. The final depth is a linear
combination of predicted depth bases that can be optimized by tuning the
corresponding weights. To seamlessly incorporate the weights into traditional
SLAM optimization and ensure efficiency and robustness, we design a set of
depth weight factors, which makes our network a versatile plug-in module,
facilitating easy integration into various existing sparse SLAM systems and
significantly enhancing global depth consistency through bundle adjustment. To
verify the portability of our method, we integrate BBC-Net into two
representative SLAM systems. The experimental results on various datasets show
that the proposed method achieves better performance in monocular dense mapping
than the state-of-the-art methods. We provide an online demo running on a
mobile phone, which verifies the efficiency and mapping quality of the proposed
method in real-world scenarios.
less
By: Akankshya Kar, Sajal Maheshwari, Shamit Lal, Vinay Sameer Raja Kad
Visual odometry (VO) and SLAM have been using multi-view geometry via local
structure from motion for decades. These methods have a slight disadvantage in
challenging scenarios such as low-texture images, dynamic scenarios, etc.
Meanwhile, use of deep neural networks to extract high level features is
ubiquitous in computer vision. For VO, we can use these deep networks to
extract depth and pose estimates using these high level features. The... more
Visual odometry (VO) and SLAM have been using multi-view geometry via local
structure from motion for decades. These methods have a slight disadvantage in
challenging scenarios such as low-texture images, dynamic scenarios, etc.
Meanwhile, use of deep neural networks to extract high level features is
ubiquitous in computer vision. For VO, we can use these deep networks to
extract depth and pose estimates using these high level features. The visual
odometry task then can be modeled as an image generation task where the pose
estimation is the by-product. This can also be achieved in a self-supervised
manner, thereby eliminating the data (supervised) intensive nature of training
deep neural networks. Although some works tried the similar approach [1], the
depth and pose estimation in the previous works are vague sometimes resulting
in accumulation of error (drift) along the trajectory. The goal of this work is
to tackle these limitations of past approaches and to develop a method that can
provide better depths and pose estimates. To address this, a couple of
approaches are explored: 1) Modeling: Using optical flow and recurrent neural
networks (RNN) in order to exploit spatio-temporal correlations which can
provide more information to estimate depth. 2) Loss function: Generative
adversarial network (GAN) [2] is deployed to improve the depth estimation (and
thereby pose too), as shown in Figure 1. This additional loss term improves the
realism in generated images and reduces artifacts.
less
Representation Synthesis by Probabilistic Many-Valued Logic Operation in Self-Supervised Learning
0upvotes
By: Hiroki Nakamura, Masashi Okada, Tadahiro Taniguchi
Self-supervised learning (SSL) using mixed images has been studied to learn
various image representations. Existing methods using mixed images learn a
representation by maximizing the similarity between the representation of the
mixed image and the synthesized representation of the original images. However,
few methods consider the synthesis of representations from the perspective of
mathematical logic. In this study, we focused on a synthe... more
Self-supervised learning (SSL) using mixed images has been studied to learn
various image representations. Existing methods using mixed images learn a
representation by maximizing the similarity between the representation of the
mixed image and the synthesized representation of the original images. However,
few methods consider the synthesis of representations from the perspective of
mathematical logic. In this study, we focused on a synthesis method of
representations. We proposed a new SSL with mixed images and a new
representation format based on many-valued logic. This format can indicate the
feature-possession degree, that is, how much of each image feature is possessed
by a representation. This representation format and representation synthesis by
logic operation realize that the synthesized representation preserves the
remarkable characteristics of the original representations. Our method
performed competitively with previous representation synthesis methods for
image classification tasks. We also examined the relationship between the
feature-possession degree and the number of classes of images in the multilabel
image classification dataset to verify that the intended learning was achieved.
In addition, we discussed image retrieval, which is an application of our
proposed representation format using many-valued logic.
less
By: Aoxu Liu, Xiaohong Fan, Yin Yang, Jianping Zhang
The problem of phase retrieval (PR) involves recovering an unknown image from
limited amplitude measurement data and is a challenge nonlinear inverse problem
in computational imaging and image processing. However, many of the PR methods
are based on black-box network models that lack interpretability and
plug-and-play (PnP) frameworks that are computationally complex and require
careful parameter tuning. To address this, we have developed P... more
The problem of phase retrieval (PR) involves recovering an unknown image from
limited amplitude measurement data and is a challenge nonlinear inverse problem
in computational imaging and image processing. However, many of the PR methods
are based on black-box network models that lack interpretability and
plug-and-play (PnP) frameworks that are computationally complex and require
careful parameter tuning. To address this, we have developed PRISTA-Net, a deep
unfolding network (DUN) based on the first-order iterative shrinkage
thresholding algorithm (ISTA). This network utilizes a learnable nonlinear
transformation to address the proximal-point mapping sub-problem associated
with the sparse priors, and an attention mechanism to focus on phase
information containing image edges, textures, and structures. Additionally, the
fast Fourier transform (FFT) is used to learn global features to enhance local
information, and the designed logarithmic-based loss function leads to
significant improvements when the noise level is low. All parameters in the
proposed PRISTA-Net framework, including the nonlinear transformation,
threshold parameters, and step size, are learned end-to-end instead of being
manually set. This method combines the interpretability of traditional methods
with the fast inference ability of deep learning and is able to handle noise at
each iteration during the unfolding stage, thus improving recovery quality.
Experiments on Coded Diffraction Patterns (CDPs) measurements demonstrate that
our approach outperforms the existing state-of-the-art methods in terms of
qualitative and quantitative evaluations. Our source codes are available at
\emph{https://github.com/liuaxou/PRISTA-Net}.
less
By: Yeonghwan Song, Seokwoo Jang, Dina Katabi, Jeany Son
We propose a novel unsupervised object localization method that allows us to
explain the predictions of the model by utilizing self-supervised pre-trained
models without additional finetuning. Existing unsupervised and self-supervised
object localization methods often utilize class-agnostic activation maps or
self-similarity maps of a pre-trained model. Although these maps can offer
valuable information for localization, their limited abili... more
We propose a novel unsupervised object localization method that allows us to
explain the predictions of the model by utilizing self-supervised pre-trained
models without additional finetuning. Existing unsupervised and self-supervised
object localization methods often utilize class-agnostic activation maps or
self-similarity maps of a pre-trained model. Although these maps can offer
valuable information for localization, their limited ability to explain how the
model makes predictions remains challenging. In this paper, we propose a simple
yet effective unsupervised object localization method based on representer
point selection, where the predictions of the model can be represented as a
linear combination of representer values of training points. By selecting
representer points, which are the most important examples for the model
predictions, our model can provide insights into how the model predicts the
foreground object by providing relevant examples as well as their importance.
Our method outperforms the state-of-the-art unsupervised and self-supervised
object localization methods on various datasets with significant margins and
even outperforms recent weakly supervised and few-shot methods.
less
By: Ziang Cheng, Jiayu Yang, Hongdong Li
Real-time Stereo Matching is a cornerstone algorithm for many Extended
Reality (XR) applications, such as indoor 3D understanding, video pass-through,
and mixed-reality games. Despite significant advancements in deep stereo
methods, achieving real-time depth inference with high accuracy on a low-power
device remains a major challenge. One of the major difficulties is the lack of
high-quality indoor video stereo training datasets captured by... more
Real-time Stereo Matching is a cornerstone algorithm for many Extended
Reality (XR) applications, such as indoor 3D understanding, video pass-through,
and mixed-reality games. Despite significant advancements in deep stereo
methods, achieving real-time depth inference with high accuracy on a low-power
device remains a major challenge. One of the major difficulties is the lack of
high-quality indoor video stereo training datasets captured by head-mounted
VR/AR glasses. To address this issue, we introduce a novel video stereo
synthetic dataset that comprises photorealistic renderings of various indoor
scenes and realistic camera motion captured by a 6-DoF moving VR/AR
head-mounted display (HMD). This facilitates the evaluation of existing
approaches and promotes further research on indoor augmented reality scenarios.
Our newly proposed dataset enables us to develop a novel framework for
continuous video-rate stereo matching.
As another contribution, our dataset enables us to proposed a new video-based
stereo matching approach tailored for XR applications, which achieves real-time
inference at an impressive 134fps on a standard desktop computer, or 30fps on a
battery-powered HMD. Our key insight is that disparity and contextual
information are highly correlated and redundant between consecutive stereo
frames. By unrolling an iterative cost aggregation in time (i.e. in the
temporal dimension), we are able to distribute and reuse the aggregated
features over time. This approach leads to a substantial reduction in
computation without sacrificing accuracy. We conducted extensive evaluations
and comparisons and demonstrated that our method achieves superior performance
compared to the current state-of-the-art, making it a strong contender for
real-time stereo matching in VR/AR applications.
less
By: Junfeng Cheng, Mingdong Wu, Ruiyuan Zhang, Guanqi Zhan, Chao Wu, Hao Dong
Autonomous 3D part assembly is a challenging task in the areas of robotics
and 3D computer vision. This task aims to assemble individual components into a
complete shape without relying on predefined instructions. In this paper, we
formulate this task from a novel generative perspective, introducing the
Score-based 3D Part Assembly framework (Score-PA) for 3D part assembly. Knowing
that score-based methods are typically time-consuming durin... more
Autonomous 3D part assembly is a challenging task in the areas of robotics
and 3D computer vision. This task aims to assemble individual components into a
complete shape without relying on predefined instructions. In this paper, we
formulate this task from a novel generative perspective, introducing the
Score-based 3D Part Assembly framework (Score-PA) for 3D part assembly. Knowing
that score-based methods are typically time-consuming during the inference
stage. To address this issue, we introduce a novel algorithm called the Fast
Predictor-Corrector Sampler (FPC) that accelerates the sampling process within
the framework. We employ various metrics to assess assembly quality and
diversity, and our evaluation results demonstrate that our algorithm
outperforms existing state-of-the-art approaches. We release our code at
https://github.com/J-F-Cheng/Score-PA_Score-based-3D-Part-Assembly.
less
By: Dawen Yu, Shunping Ji
Long-range dependency modeling has been widely considered in modern deep
learning based semantic segmentation methods, especially those designed for
large-size remote sensing images, to compensate the intrinsic locality of
standard convolutions. However, in previous studies, the long-range dependency,
modeled with an attention mechanism or transformer model, has been based on
unsupervised learning, instead of explicit supervision from the o... more
Long-range dependency modeling has been widely considered in modern deep
learning based semantic segmentation methods, especially those designed for
large-size remote sensing images, to compensate the intrinsic locality of
standard convolutions. However, in previous studies, the long-range dependency,
modeled with an attention mechanism or transformer model, has been based on
unsupervised learning, instead of explicit supervision from the objective
ground truth. In this paper, we propose a novel supervised long-range
correlation method for land-cover classification, called the supervised
long-range correlation network (SLCNet), which is shown to be superior to the
currently used unsupervised strategies. In SLCNet, pixels sharing the same
category are considered highly correlated and those having different categories
are less relevant, which can be easily supervised by the category consistency
information available in the ground truth semantic segmentation map. Under such
supervision, the recalibrated features are more consistent for pixels of the
same category and more discriminative for pixels of other categories,
regardless of their proximity. To complement the detailed information lacking
in the global long-range correlation, we introduce an auxiliary adaptive
receptive field feature extraction module, parallel to the long-range
correlation module in the encoder, to capture finely detailed feature
representations for multi-size objects in multi-scale remote sensing images. In
addition, we apply multi-scale side-output supervision and a hybrid loss
function as local and global constraints to further boost the segmentation
accuracy. Experiments were conducted on three remote sensing datasets. Compared
with the advanced segmentation methods from the computer vision, medicine, and
remote sensing communities, the SLCNet achieved a state-of-the-art performance
on all the datasets.
less
By: Xiangtao Wang, Ruizhi Wang, Jie Zhou, Thomas Lukasiewicz, Zhenghua Xu
Self-supervised masked image modeling has shown promising results on natural
images. However, directly applying such methods to medical images remains
challenging. This difficulty stems from the complexity and distinct
characteristics of lesions compared to natural images, which impedes effective
representation learning. Additionally, conventional high fixed masking ratios
restrict reconstructing fine lesion details, limiting the scope of l... more
Self-supervised masked image modeling has shown promising results on natural
images. However, directly applying such methods to medical images remains
challenging. This difficulty stems from the complexity and distinct
characteristics of lesions compared to natural images, which impedes effective
representation learning. Additionally, conventional high fixed masking ratios
restrict reconstructing fine lesion details, limiting the scope of learnable
information. To tackle these limitations, we propose a novel self-supervised
medical image segmentation framework, Adaptive Masking Lesion Patches (AMLP).
Specifically, we design a Masked Patch Selection (MPS) strategy to identify and
focus learning on patches containing lesions. Lesion regions are scarce yet
critical, making their precise reconstruction vital. To reduce
misclassification of lesion and background patches caused by unsupervised
clustering in MPS, we introduce an Attention Reconstruction Loss (ARL) to focus
on hard-to-reconstruct patches likely depicting lesions. We further propose a
Category Consistency Loss (CCL) to refine patch categorization based on
reconstruction difficulty, strengthening distinction between lesions and
background. Moreover, we develop an Adaptive Masking Ratio (AMR) strategy that
gradually increases the masking ratio to expand reconstructible information and
improve learning. Extensive experiments on two medical segmentation datasets
demonstrate AMLP's superior performance compared to existing self-supervised
approaches. The proposed strategies effectively address limitations in applying
masked modeling to medical images, tailored to capturing fine lesion details
vital for segmentation tasks.
less
By: Rayson Laroca, Luiz A. Zanlorensi, Valter Estevam, Rodrigo Minetto, David Menotti
License Plate Recognition (LPR) plays a critical role in various
applications, such as toll collection, parking management, and traffic law
enforcement. Although LPR has witnessed significant advancements through the
development of deep learning, there has been a noticeable lack of studies
exploring the potential improvements in results by fusing the outputs from
multiple recognition models. This research aims to fill this gap by
investigat... more
License Plate Recognition (LPR) plays a critical role in various
applications, such as toll collection, parking management, and traffic law
enforcement. Although LPR has witnessed significant advancements through the
development of deep learning, there has been a noticeable lack of studies
exploring the potential improvements in results by fusing the outputs from
multiple recognition models. This research aims to fill this gap by
investigating the combination of up to 12 different models using
straightforward approaches, such as selecting the most confident prediction or
employing majority vote-based strategies. Our experiments encompass a wide
range of datasets, revealing substantial benefits of fusion approaches in both
intra- and cross-dataset setups. Essentially, fusing multiple models reduces
considerably the likelihood of obtaining subpar performance on a particular
dataset/scenario. We also found that combining models based on their speed is
an appealing approach. Specifically, for applications where the recognition
task can tolerate some additional time, though not excessively, an effective
strategy is to combine 4-6 models. These models may not be the most accurate
individually, but their fusion strikes an optimal balance between accuracy and
speed.
less
By: Erik Daxberger, Floris Weers, Bowen Zhang, Tom Gunter, Ruoming Pang, Marcin Eichner, Michael Emmersberger, Yinfei Yang, Alexander Toshev, Xianzhi Du
Sparse Mixture-of-Experts models (MoEs) have recently gained popularity due
to their ability to decouple model size from inference efficiency by only
activating a small subset of the model parameters for any given input token. As
such, sparse MoEs have enabled unprecedented scalability, resulting in
tremendous successes across domains such as natural language processing and
computer vision. In this work, we instead explore the use of sparse... more
Sparse Mixture-of-Experts models (MoEs) have recently gained popularity due
to their ability to decouple model size from inference efficiency by only
activating a small subset of the model parameters for any given input token. As
such, sparse MoEs have enabled unprecedented scalability, resulting in
tremendous successes across domains such as natural language processing and
computer vision. In this work, we instead explore the use of sparse MoEs to
scale-down Vision Transformers (ViTs) to make them more attractive for
resource-constrained vision applications. To this end, we propose a simplified
and mobile-friendly MoE design where entire images rather than individual
patches are routed to the experts. We also propose a stable MoE training
procedure that uses super-class information to guide the router. We empirically
show that our sparse Mobile Vision MoEs (V-MoEs) can achieve a better trade-off
between performance and efficiency than the corresponding dense ViTs. For
example, for the ViT-Tiny model, our Mobile V-MoE outperforms its dense
counterpart by 3.39% on ImageNet-1k. For an even smaller ViT variant with only
54M FLOPs inference cost, our MoE achieves an improvement of 4.66%.
less
By: Casper van Engelenburg, Seyran Khademi, Jan van Gemert
We propose a simple yet effective metric that measures structural similarity
between visual instances of architectural floor plans, without the need for
learning. Qualitatively, our experiments show that the retrieval results are
similar to deeply learned methods. Effectively comparing instances of floor
plan data is paramount to the success of machine understanding of floor plan
data, including the assessment of floor plan generative model... more
We propose a simple yet effective metric that measures structural similarity
between visual instances of architectural floor plans, without the need for
learning. Qualitatively, our experiments show that the retrieval results are
similar to deeply learned methods. Effectively comparing instances of floor
plan data is paramount to the success of machine understanding of floor plan
data, including the assessment of floor plan generative models and floor plan
recommendation systems. Comparing visual floor plan images goes beyond a sole
pixel-wise visual examination and is crucially about similarities and
differences in the shapes and relations between subdivisions that compose the
layout. Currently, deep metric learning approaches are used to learn a
pair-wise vector representation space that closely mimics the structural
similarity, in which the models are trained on similarity labels that are
obtained by Intersection-over-Union (IoU). To compensate for the lack of
structural awareness in IoU, graph-based approaches such as Graph Matching
Networks (GMNs) are used, which require pairwise inference for comparing data
instances, making GMNs less practical for retrieval applications. In this
paper, an effective evaluation metric for judging the structural similarity of
floor plans, coined SSIG (Structural Similarity by IoU and GED), is proposed
based on both image and graph distances. In addition, an efficient algorithm is
developed that uses SSIG to rank a large-scale floor plan database. Code will
be openly available.
less
By: Shuning Xu, Xiangyu Chen, Binbin Song, Jiantao Zhou
Capturing images with incorrect exposure settings fails to deliver a
satisfactory visual experience. Only when the exposure is properly set, can the
color and details of the images be appropriately preserved. Previous exposure
correction methods based on convolutions often produce exposure deviation in
images as a consequence of the restricted receptive field of convolutional
kernels. This issue arises because convolutions are not capable o... more
Capturing images with incorrect exposure settings fails to deliver a
satisfactory visual experience. Only when the exposure is properly set, can the
color and details of the images be appropriately preserved. Previous exposure
correction methods based on convolutions often produce exposure deviation in
images as a consequence of the restricted receptive field of convolutional
kernels. This issue arises because convolutions are not capable of capturing
long-range dependencies in images accurately. To overcome this challenge, we
can apply the Transformer to address the exposure correction problem,
leveraging its capability in modeling long-range dependencies to capture global
representation. However, solely relying on the window-based Transformer leads
to visually disturbing blocking artifacts due to the application of
self-attention in small patches. In this paper, we propose a CNN Injected
Transformer (CIT) to harness the individual strengths of CNN and Transformer
simultaneously. Specifically, we construct the CIT by utilizing a window-based
Transformer to exploit the long-range interactions among different regions in
the entire image. Within each CIT block, we incorporate a channel attention
block (CAB) and a half-instance normalization block (HINB) to assist the
window-based self-attention to acquire the global statistics and refine local
features. In addition to the hybrid architecture design for exposure
correction, we apply a set of carefully formulated loss functions to improve
the spatial coherence and rectify potential color deviations. Extensive
experiments demonstrate that our image exposure correction method outperforms
state-of-the-art approaches in terms of both quantitative and qualitative
metrics.
less
MoEController: Instruction-based Arbitrary Image Manipulation with Mixture-of-Expert Controllers
0upvotes
By: Sijia Li, Chen Chen, Haonan Lu
Diffusion-model-based text-guided image generation has recently made
astounding progress, producing fascinating results in open-domain image
manipulation tasks. Few models, however, currently have complete zero-shot
capabilities for both global and local image editing due to the complexity and
diversity of image manipulation tasks. In this work, we propose a method with a
mixture-of-expert (MOE) controllers to align the text-guided capacity... more
Diffusion-model-based text-guided image generation has recently made
astounding progress, producing fascinating results in open-domain image
manipulation tasks. Few models, however, currently have complete zero-shot
capabilities for both global and local image editing due to the complexity and
diversity of image manipulation tasks. In this work, we propose a method with a
mixture-of-expert (MOE) controllers to align the text-guided capacity of
diffusion models with different kinds of human instructions, enabling our model
to handle various open-domain image manipulation tasks with natural language
instructions. First, we use large language models (ChatGPT) and conditional
image synthesis models (ControlNet) to generate a large number of global image
transfer dataset in addition to the instruction-based local image editing
dataset. Then, using an MOE technique and task-specific adaptation training on
a large-scale dataset, our conditional diffusion model can edit images globally
and locally. Extensive experiments demonstrate that our approach performs
surprisingly well on various image manipulation tasks when dealing with
open-domain images and arbitrary human instructions. Please refer to our
project page: [https://oppo-mente-lab.github.io/moe_controller/]
less
By: Dongming Wu, Wencheng Han, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, Jianbing Shen
A new trend in the computer vision community is to capture objects of
interest following flexible human command represented by a natural language
prompt. However, the progress of using language prompts in driving scenarios is
stuck in a bottleneck due to the scarcity of paired prompt-instance data. To
address this challenge, we propose the first object-centric language prompt set
for driving scenes within 3D, multi-view, and multi-frame spa... more
A new trend in the computer vision community is to capture objects of
interest following flexible human command represented by a natural language
prompt. However, the progress of using language prompts in driving scenarios is
stuck in a bottleneck due to the scarcity of paired prompt-instance data. To
address this challenge, we propose the first object-centric language prompt set
for driving scenes within 3D, multi-view, and multi-frame space, named
NuPrompt. It expands Nuscenes dataset by constructing a total of 35,367
language descriptions, each referring to an average of 5.3 object tracks. Based
on the object-text pairs from the new benchmark, we formulate a new
prompt-based driving task, \ie, employing a language prompt to predict the
described object trajectory across views and frames. Furthermore, we provide a
simple end-to-end baseline model based on Transformer, named PromptTrack.
Experiments show that our PromptTrack achieves impressive performance on
NuPrompt. We hope this work can provide more new insights for the autonomous
driving community. Dataset and Code will be made public at
\href{https://github.com/wudongming97/Prompt4Driving}{https://github.com/wudongming97/Prompt4Driving}.
less
By: Amr Gomaa, Robin Zitt, Guillermo Reyes, Antonio Krüger
Creating a diverse and comprehensive dataset of hand gestures for dynamic
human-machine interfaces in the automotive domain can be challenging and
time-consuming. To overcome this challenge, we propose using synthetic gesture
datasets generated by virtual 3D models. Our framework utilizes Unreal Engine
to synthesize realistic hand gestures, offering customization options and
reducing the risk of overfitting. Multiple variants, including ges... more
Creating a diverse and comprehensive dataset of hand gestures for dynamic
human-machine interfaces in the automotive domain can be challenging and
time-consuming. To overcome this challenge, we propose using synthetic gesture
datasets generated by virtual 3D models. Our framework utilizes Unreal Engine
to synthesize realistic hand gestures, offering customization options and
reducing the risk of overfitting. Multiple variants, including gesture speed,
performance, and hand shape, are generated to improve generalizability. In
addition, we simulate different camera locations and types, such as RGB,
infrared, and depth cameras, without incurring additional time and cost to
obtain these cameras. Experimental results demonstrate that our proposed
framework,
SynthoGestures\footnote{\url{https://github.com/amrgomaaelhady/SynthoGestures}},
improves gesture recognition accuracy and can replace or augment real-hand
datasets. By saving time and effort in the creation of the data set, our tool
accelerates the development of gesture recognition systems for automotive
applications.
less
By: Gan Sun, Wenqi Liang, Jiahua Dong, Jun Li, Zhengming Ding, Yang Cong
Text-to-image generative models can produce diverse high-quality images of
concepts with a text prompt, which have demonstrated excellent ability in image
generation, image translation, etc. We in this work study the problem of
synthesizing instantiations of a use's own concepts in a never-ending manner,
i.e., create your world, where the new concepts from user are quickly learned
with a few examples. To achieve this goal, we propose a Life... more
Text-to-image generative models can produce diverse high-quality images of
concepts with a text prompt, which have demonstrated excellent ability in image
generation, image translation, etc. We in this work study the problem of
synthesizing instantiations of a use's own concepts in a never-ending manner,
i.e., create your world, where the new concepts from user are quickly learned
with a few examples. To achieve this goal, we propose a Lifelong text-to-image
Diffusion Model (L2DM), which intends to overcome knowledge "catastrophic
forgetting" for the past encountered concepts, and semantic "catastrophic
neglecting" for one or more concepts in the text prompt. In respect of
knowledge "catastrophic forgetting", our L2DM framework devises a task-aware
memory enhancement module and a elastic-concept distillation module, which
could respectively safeguard the knowledge of both prior concepts and each past
personalized concept. When generating images with a user text prompt, the
solution to semantic "catastrophic neglecting" is that a concept attention
artist module can alleviate the semantic neglecting from concept aspect, and an
orthogonal attention module can reduce the semantic binding from attribute
aspect. To the end, our model can generate more faithful image across a range
of continual text prompts in terms of both qualitative and quantitative
metrics, when comparing with the related state-of-the-art models. The code will
be released at https://wenqiliang.github.io/.
less
By: Brandon Zhao, Aviad Levis, Liam Connor, Pratul P. Srinivasan, Katherine L. Bouman
Refractive Index Tomography is an inverse problem in which we seek to
reconstruct a scene's 3D refractive field from 2D projected image measurements.
The refractive field is not visible itself, but instead affects how the path of
a light ray is continuously curved as it travels through space. Refractive
fields appear across a wide variety of scientific applications, from
translucent cell samples in microscopy to fields of dark matter bendin... more
Refractive Index Tomography is an inverse problem in which we seek to
reconstruct a scene's 3D refractive field from 2D projected image measurements.
The refractive field is not visible itself, but instead affects how the path of
a light ray is continuously curved as it travels through space. Refractive
fields appear across a wide variety of scientific applications, from
translucent cell samples in microscopy to fields of dark matter bending light
from faraway galaxies. This problem poses a unique challenge because the
refractive field directly affects the path that light takes, making its
recovery a non-linear problem. In addition, in contrast with traditional
tomography, we seek to recover the refractive field using a projected image
from only a single viewpoint by leveraging knowledge of light sources scattered
throughout the medium. In this work, we introduce a method that uses a
coordinate-based neural network to model the underlying continuous refractive
field in a scene. We then use explicit modeling of rays' 3D spatial curvature
to optimize the parameters of this network, reconstructing refractive fields
with an analysis-by-synthesis approach. The efficacy of our approach is
demonstrated by recovering refractive fields in simulation, and analyzing how
recovery is affected by the light source distribution. We then test our method
on a simulated dark matter mapping problem, where we recover the refractive
field underlying a realistic simulated dark matter distribution.
less
By: Aman Bhatta, Gabriella Pangelinan, Micheal C. King, Kevin W. Bowyer
Most studies to date that have examined demographic variations in face
recognition accuracy have analyzed 1-to-1 matching accuracy, using images that
could be described as "government ID quality". This paper analyzes the accuracy
of 1-to-many facial identification across demographic groups, and in the
presence of blur and reduced resolution in the probe image as might occur in
"surveillance camera quality" images. Cumulative match character... more
Most studies to date that have examined demographic variations in face
recognition accuracy have analyzed 1-to-1 matching accuracy, using images that
could be described as "government ID quality". This paper analyzes the accuracy
of 1-to-many facial identification across demographic groups, and in the
presence of blur and reduced resolution in the probe image as might occur in
"surveillance camera quality" images. Cumulative match characteristic
curves(CMC) are not appropriate for comparing propensity for rank-one
recognition errors across demographics, and so we introduce three metrics for
this: (1) d' metric between mated and non-mated score distributions, (2)
absolute score difference between thresholds in the high-similarity tail of the
non-mated and the low-similarity tail of the mated distribution, and (3)
distribution of (mated - non-mated rank one scores) across the set of probe
images. We find that demographic variation in 1-to-many accuracy does not
entirely follow what has been observed in 1-to-1 matching accuracy. Also,
different from 1-to-1 accuracy, demographic comparison of 1-to-many accuracy
can be affected by different numbers of identities and images across
demographics. Finally, we show that increased blur in the probe image, or
reduced resolution of the face in the probe image, can significantly increase
the false positive identification rate. And we show that the demographic
variation in these high blur or low resolution conditions is much larger for
male/ female than for African-American / Caucasian. The point that 1-to-many
accuracy can potentially collapse in the context of processing "surveillance
camera quality" probe images against a "government ID quality" gallery is an
important one.
less
By: Felix Rosberg, Eren Erdal Aksoy, Cristofer Englund, Fernando Alonso-Fernandez
In this paper, we present a new approach for facial anonymization in images
and videos, abbreviated as FIVA. Our proposed method is able to maintain the
same face anonymization consistently over frames with our suggested
identity-tracking and guarantees a strong difference from the original face.
FIVA allows for 0 true positives for a false acceptance rate of 0.001. Our work
considers the important security issue of reconstruction attacks a... more
In this paper, we present a new approach for facial anonymization in images
and videos, abbreviated as FIVA. Our proposed method is able to maintain the
same face anonymization consistently over frames with our suggested
identity-tracking and guarantees a strong difference from the original face.
FIVA allows for 0 true positives for a false acceptance rate of 0.001. Our work
considers the important security issue of reconstruction attacks and
investigates adversarial noise, uniform noise, and parameter noise to disrupt
reconstruction attacks. In this regard, we apply different defense and
protection methods against these privacy threats to demonstrate the scalability
of FIVA. On top of this, we also show that reconstruction attack models can be
used for detection of deep fakes. Last but not least, we provide experimental
results showing how FIVA can even enable face swapping, which is purely trained
on a single target image.
less
By: Maximilian Li, Xander Davies, Max Nadeau
Language models often exhibit behaviors that improve performance on a
pre-training objective but harm performance on downstream tasks. We propose a
novel approach to removing undesirable behaviors by ablating a small number of
causal pathways between model components, with the intention of disabling the
computational circuit responsible for the bad behavior. Given a small dataset
of inputs where the model behaves poorly, we learn to ablate ... more
Language models often exhibit behaviors that improve performance on a
pre-training objective but harm performance on downstream tasks. We propose a
novel approach to removing undesirable behaviors by ablating a small number of
causal pathways between model components, with the intention of disabling the
computational circuit responsible for the bad behavior. Given a small dataset
of inputs where the model behaves poorly, we learn to ablate a small number of
important causal pathways. In the setting of reducing GPT-2 toxic language
generation, we find ablating just 12 of the 11.6K causal edges mitigates toxic
generation with minimal degradation of performance on other inputs.
less
By: Shaoxiong Ji, Wei Sun, Pekka Marttinen
Many computational linguistic methods have been proposed to study the
information content of languages. We consider two interesting research
questions: 1) how is information distributed over long documents, and 2) how
does content reduction, such as token selection and text summarization, affect
the information density in long documents. We present four criteria for
information density estimation for long documents, including surprisal,
ent... more
Many computational linguistic methods have been proposed to study the
information content of languages. We consider two interesting research
questions: 1) how is information distributed over long documents, and 2) how
does content reduction, such as token selection and text summarization, affect
the information density in long documents. We present four criteria for
information density estimation for long documents, including surprisal,
entropy, uniform information density, and lexical density. Among those
criteria, the first three adopt the measures from information theory. We
propose an attention-based word selection method for clinical notes and study
machine summarization for multiple-domain documents. Our findings reveal the
systematic difference in information density of long text in various domains.
Empirical results on automated medical coding from long clinical notes show the
effectiveness of the attention-based word selection method.
less
BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models
0upvotes
By: Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, William Chandra Tjhi
The rapid development of Large Language Models (LLMs) and the emergence of
novel abilities with scale have necessitated the construction of holistic,
diverse and challenging benchmarks such as HELM and BIG-bench. However, at the
moment, most of these benchmarks focus only on performance in English and
evaluations that include Southeast Asian (SEA) languages are few in number. We
therefore propose BHASA, a holistic linguistic and cultural ev... more
The rapid development of Large Language Models (LLMs) and the emergence of
novel abilities with scale have necessitated the construction of holistic,
diverse and challenging benchmarks such as HELM and BIG-bench. However, at the
moment, most of these benchmarks focus only on performance in English and
evaluations that include Southeast Asian (SEA) languages are few in number. We
therefore propose BHASA, a holistic linguistic and cultural evaluation suite
for LLMs in SEA languages. It comprises three components: (1) a NLP benchmark
covering eight tasks across Natural Language Understanding (NLU), Generation
(NLG) and Reasoning (NLR) tasks, (2) LINDSEA, a linguistic diagnostic toolkit
that spans the gamut of linguistic phenomena including syntax, semantics and
pragmatics, and (3) a cultural diagnostics dataset that probes for both
cultural representation and sensitivity. For this preliminary effort, we
implement the NLP benchmark only for Indonesian, Vietnamese, Thai and Tamil,
and we only include Indonesian and Tamil for LINDSEA and the cultural
diagnostics dataset. As GPT-4 is purportedly one of the best-performing
multilingual LLMs at the moment, we use it as a yardstick to gauge the
capabilities of LLMs in the context of SEA languages. Our initial experiments
on GPT-4 with BHASA find it lacking in various aspects of linguistic
capabilities, cultural representation and sensitivity in the targeted SEA
languages. BHASA is a work in progress and will continue to be improved and
expanded in the future.
less
By: Sharath Srivatsa, Srinath Srinivasa
Media houses reporting on public figures, often come with their own biases
stemming from their respective worldviews. A characterization of these
underlying patterns helps us in better understanding and interpreting news
stories. For this, we need diverse or subjective summarizations, which may not
be amenable for classifying into predefined class labels. This work proposes a
zero-shot approach for non-extractive or generative characterizat... more
Media houses reporting on public figures, often come with their own biases
stemming from their respective worldviews. A characterization of these
underlying patterns helps us in better understanding and interpreting news
stories. For this, we need diverse or subjective summarizations, which may not
be amenable for classifying into predefined class labels. This work proposes a
zero-shot approach for non-extractive or generative characterizations of person
entities from a corpus using GPT-2. We use well-articulated articles from
several well-known news media houses as a corpus to build a sound argument for
this approach. First, we fine-tune a GPT-2 pre-trained language model with a
corpus where specific person entities are characterized. Second, we further
fine-tune this with demonstrations of person entity characterizations, created
from a corpus of programmatically constructed characterizations. This twice
fine-tuned model is primed with manual prompts consisting of entity names that
were not previously encountered in the second fine-tuning, to generate a simple
sentence about the entity. The results were encouraging, when compared against
actual characterizations from the corpus.
less
Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts
0upvotes
By: Zhi-Yi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen, Wei-Chen Chiu
Text-to-image diffusion models, e.g. Stable Diffusion (SD), lately have shown
remarkable ability in high-quality content generation, and become one of the
representatives for the recent wave of transformative AI. Nevertheless, such
advance comes with an intensifying concern about the misuse of this generative
technology, especially for producing copyrighted or NSFW (i.e. not safe for
work) images. Although efforts have been made to filter i... more
Text-to-image diffusion models, e.g. Stable Diffusion (SD), lately have shown
remarkable ability in high-quality content generation, and become one of the
representatives for the recent wave of transformative AI. Nevertheless, such
advance comes with an intensifying concern about the misuse of this generative
technology, especially for producing copyrighted or NSFW (i.e. not safe for
work) images. Although efforts have been made to filter inappropriate
images/prompts or remove undesirable concepts/styles via model fine-tuning, the
reliability of these safety mechanisms against diversified problematic prompts
remains largely unexplored. In this work, we propose Prompting4Debugging (P4D)
as a debugging and red-teaming tool that automatically finds problematic
prompts for diffusion models to test the reliability of a deployed safety
mechanism. We demonstrate the efficacy of our P4D tool in uncovering new
vulnerabilities of SD models with safety mechanisms. Particularly, our result
shows that around half of prompts in existing safe prompting benchmarks which
were originally considered "safe" can actually be manipulated to bypass many
deployed safety mechanisms, including concept removal, negative prompt, and
safety guidance. Our findings suggest that, without comprehensive testing, the
evaluations on limited safe prompting benchmarks can lead to a false sense of
safety for text-to-image models.
less
By: Alessandra Polimeno, Myrthe Reuver, Sanne Vrijenhoek, Antske Fokkens
News recommender systems play an increasingly influential role in shaping
information access within democratic societies. However, tailoring
recommendations to users' specific interests can result in the divergence of
information streams. Fragmented access to information poses challenges to the
integrity of the public sphere, thereby influencing democracy and public
discourse. The Fragmentation metric quantifies the degree of fragmentation ... more
News recommender systems play an increasingly influential role in shaping
information access within democratic societies. However, tailoring
recommendations to users' specific interests can result in the divergence of
information streams. Fragmented access to information poses challenges to the
integrity of the public sphere, thereby influencing democracy and public
discourse. The Fragmentation metric quantifies the degree of fragmentation of
information streams in news recommendations. Accurate measurement of this
metric requires the application of Natural Language Processing (NLP) to
identify distinct news events, stories, or timelines. This paper presents an
extensive investigation of various approaches for quantifying Fragmentation in
news recommendations. These approaches are evaluated both intrinsically, by
measuring performance on news story clustering, and extrinsically, by assessing
the Fragmentation scores of different simulated news recommender scenarios. Our
findings demonstrate that agglomerative hierarchical clustering coupled with
SentenceBERT text representation is substantially better at detecting
Fragmentation than earlier implementations. Additionally, the analysis of
simulated scenarios yields valuable insights and recommendations for
stakeholders concerning the measurement and interpretation of Fragmentation.
less
AKEM: Aligning Knowledge Base to Queries with Ensemble Model for Entity Recognition and Linking
0upvotes
By: Di Lu, Zhongping Liang, Caixia Yuan, Xiaojie Wang
This paper presents a novel approach to address the Entity Recognition and
Linking Challenge at NLPCC 2015. The task involves extracting named entity
mentions from short search queries and linking them to entities within a
reference Chinese knowledge base. To tackle this problem, we first expand the
existing knowledge base and utilize external knowledge to identify candidate
entities, thereby improving the recall rate. Next, we extract feat... more
This paper presents a novel approach to address the Entity Recognition and
Linking Challenge at NLPCC 2015. The task involves extracting named entity
mentions from short search queries and linking them to entities within a
reference Chinese knowledge base. To tackle this problem, we first expand the
existing knowledge base and utilize external knowledge to identify candidate
entities, thereby improving the recall rate. Next, we extract features from the
candidate entities and utilize Support Vector Regression and Multiple Additive
Regression Tree as scoring functions to filter the results. Additionally, we
apply rules to further refine the results and enhance precision. Our method is
computationally efficient and achieves an F1 score of 0.535.
less
By: Xiangci Li, Yi-Hui Lee, Jessica Ouyang
Automatic related work generation must ground their outputs to the content of
the cited papers to avoid non-factual hallucinations, but due to the length of
scientific documents, existing abstractive approaches have conditioned only on
the cited paper \textit{abstracts}. We demonstrate that the abstract is not
always the most appropriate input for citation generation and that models
trained in this way learn to hallucinate. We propose to co... more
Automatic related work generation must ground their outputs to the content of
the cited papers to avoid non-factual hallucinations, but due to the length of
scientific documents, existing abstractive approaches have conditioned only on
the cited paper \textit{abstracts}. We demonstrate that the abstract is not
always the most appropriate input for citation generation and that models
trained in this way learn to hallucinate. We propose to condition instead on
the \textit{cited text span} (CTS) as an alternative to the abstract. Because
manual CTS annotation is extremely time- and labor-intensive, we experiment
with automatic, ROUGE-based labeling of candidate CTS sentences, achieving
sufficiently strong performance to substitute for expensive human annotations,
and we propose a human-in-the-loop, keyword-based CTS retrieval approach that
makes generating citation texts grounded in the full text of cited papers both
promising and practical.
less
By: Mohsin Ali, Kandukuri Sai Teja, Neeharika Gupta, Parth Patwa, Anubhab Chatterjee, Vinija Jain, Aman Chadha, Amitava Das
The mixing of two or more languages is called Code-Mixing (CM). CM is a
social norm in multilingual societies. Neural Language Models (NLMs) like
transformers have been very effective on many NLP tasks. However, NLM for CM is
an under-explored area. Though transformers are capable and powerful, they
cannot always encode positional/sequential information since they are
non-recurrent. Therefore, to enrich word information and incorporate posi... more
The mixing of two or more languages is called Code-Mixing (CM). CM is a
social norm in multilingual societies. Neural Language Models (NLMs) like
transformers have been very effective on many NLP tasks. However, NLM for CM is
an under-explored area. Though transformers are capable and powerful, they
cannot always encode positional/sequential information since they are
non-recurrent. Therefore, to enrich word information and incorporate positional
information, positional encoding is defined. We hypothesize that Switching
Points (SPs), i.e., junctions in the text where the language switches (L1 -> L2
or L2-> L1), pose a challenge for CM Language Models (LMs), and hence give
special emphasis to switching points in the modeling process. We experiment
with several positional encoding mechanisms and show that rotatory positional
encodings along with switching point information yield the best results.
We introduce CONFLATOR: a neural language modeling approach for code-mixed
languages. CONFLATOR tries to learn to emphasize switching points using smarter
positional encoding, both at unigram and bigram levels. CONFLATOR outperforms
the state-of-the-art on two tasks based on code-mixed Hindi and English
(Hinglish): (i) sentiment analysis and (ii) machine translation.
less
By: František Kmječ, Ondřej Bojar
Many meetings require creating a meeting summary to keep everyone up to date.
Creating minutes of sufficient quality is however very cognitively demanding.
Although we currently possess capable models for both audio speech recognition
(ASR) and summarization, their fully automatic use is still problematic. ASR
models frequently commit errors when transcribing named entities while the
summarization models tend to hallucinate and misinterpret... more
Many meetings require creating a meeting summary to keep everyone up to date.
Creating minutes of sufficient quality is however very cognitively demanding.
Although we currently possess capable models for both audio speech recognition
(ASR) and summarization, their fully automatic use is still problematic. ASR
models frequently commit errors when transcribing named entities while the
summarization models tend to hallucinate and misinterpret the transcript. We
propose a novel tool -- Minuteman -- to enable efficient semi-automatic meeting
minuting. The tool provides a live transcript and a live meeting summary to the
users, who can edit them in a collaborative manner, enabling correction of ASR
errors and imperfect summary points in real time. The resulting application
eases the cognitive load of the notetakers and allows them to easily catch up
if they missed a part of the meeting due to absence or a lack of focus. We
conduct several tests of the application in varied settings, exploring the
worthiness of the concept and the possible user strategies.
less
By: Michael Beukman, Manuel Fokam
Transfer learning has led to large gains in performance for nearly all NLP
tasks while making downstream models easier and faster to train. This has also
been extended to low-resourced languages, with some success. We investigate the
properties of cross-lingual transfer learning between ten low-resourced
languages, from the perspective of a named entity recognition task. We
specifically investigate how much adaptive fine-tuning and the choi... more
Transfer learning has led to large gains in performance for nearly all NLP
tasks while making downstream models easier and faster to train. This has also
been extended to low-resourced languages, with some success. We investigate the
properties of cross-lingual transfer learning between ten low-resourced
languages, from the perspective of a named entity recognition task. We
specifically investigate how much adaptive fine-tuning and the choice of
transfer language affect zero-shot transfer performance. We find that models
that perform well on a single language often do so at the expense of
generalising to others, while models with the best generalisation to other
languages suffer in individual language performance. Furthermore, the amount of
data overlap between the source and target datasets is a better predictor of
transfer performance than either the geographical or genetic distance between
the languages.
less
By: Olga Kellert, Mahmud Uz Zaman, Nicholas Hill Matlis, Carlos Gómez-Rodríguez
This paper summarizes the results of experimenting with Universal
Dependencies (UD) adaptation of an Unsupervised, Compositional and Recursive
(UCR) rule-based approach for Sentiment Analysis (SA) submitted to the Shared
Task at Rest-Mex 2023 (Team Olga/LyS-SALSA) (within the IberLEF 2023
conference). By using basic syntactic rules such as rules of modification and
negation applied on words from sentiment dictionaries, our approach exploits... more
This paper summarizes the results of experimenting with Universal
Dependencies (UD) adaptation of an Unsupervised, Compositional and Recursive
(UCR) rule-based approach for Sentiment Analysis (SA) submitted to the Shared
Task at Rest-Mex 2023 (Team Olga/LyS-SALSA) (within the IberLEF 2023
conference). By using basic syntactic rules such as rules of modification and
negation applied on words from sentiment dictionaries, our approach exploits
some advantages of an unsupervised method for SA: (1) interpretability and
explainability of SA, (2) robustness across datasets, languages and domains and
(3) usability by non-experts in NLP. We compare our approach with other
unsupervised approaches of SA that in contrast to our UCR rule-based approach
use simple heuristic rules to deal with negation and modification. Our results
show a considerable improvement over these approaches. We discuss future
improvements of our results by using modality features as another shifting rule
of polarity and word disambiguation techniques to identify the right sentiment
words.
less
Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
0upvotes
By: Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, Sara Hooker
The Mixture of Experts (MoE) is a widely known neural architecture where an
ensemble of specialized sub-models optimizes overall performance with a
constant computational cost. However, conventional MoEs pose challenges at
scale due to the need to store all experts in memory. In this paper, we push
MoE to the limit. We propose extremely parameter-efficient MoE by uniquely
combining MoE architecture with lightweight experts.Our MoE architect... more
The Mixture of Experts (MoE) is a widely known neural architecture where an
ensemble of specialized sub-models optimizes overall performance with a
constant computational cost. However, conventional MoEs pose challenges at
scale due to the need to store all experts in memory. In this paper, we push
MoE to the limit. We propose extremely parameter-efficient MoE by uniquely
combining MoE architecture with lightweight experts.Our MoE architecture
outperforms standard parameter-efficient fine-tuning (PEFT) methods and is on
par with full fine-tuning by only updating the lightweight experts -- less than
1% of an 11B parameters model. Furthermore, our method generalizes to unseen
tasks as it does not depend on any prior task knowledge. Our research
underscores the versatility of the mixture of experts architecture, showcasing
its ability to deliver robust performance even when subjected to rigorous
parameter constraints. Our code used in all the experiments is publicly
available here: https://github.com/for-ai/parameter-efficient-moe.
less
By: Yongrui Chen, Haiyun Jiang, Xinting Huang, Shuming Shi, Guilin Qi
High-quality instruction-tuning data is critical to improving LLM
capabilities. Existing data collection methods are limited by unrealistic
manual labeling costs or by the hallucination of relying solely on LLM
generation. To address the problems, this paper presents a scalable method to
automatically collect high-quality instructional adaptation data by training
language models to automatically design tasks based on human-written texts.
In... more
High-quality instruction-tuning data is critical to improving LLM
capabilities. Existing data collection methods are limited by unrealistic
manual labeling costs or by the hallucination of relying solely on LLM
generation. To address the problems, this paper presents a scalable method to
automatically collect high-quality instructional adaptation data by training
language models to automatically design tasks based on human-written texts.
Intuitively, human-written text helps to help the model attenuate illusions
during the generation of tasks. Unlike instruction back-translation-based
methods that directly take the given text as a response, we require the model
to generate the \textit{instruction}, \textit{input}, and \textit{output}
simultaneously to filter the noise. The results of the automated and manual
evaluation experiments demonstrate the quality of our dataset.
less
By: Neel Jitesh Bhate, Ansh Mittal, Zhe He, Xiao Luo
Demographics, Social determinants of health, and family history documented in
the unstructured text within the electronic health records are increasingly
being studied to understand how this information can be utilized with the
structured data to improve healthcare outcomes. After the GPT models were
released, many studies have applied GPT models to extract this information from
the narrative clinical notes. Different from the existing work... more
Demographics, Social determinants of health, and family history documented in
the unstructured text within the electronic health records are increasingly
being studied to understand how this information can be utilized with the
structured data to improve healthcare outcomes. After the GPT models were
released, many studies have applied GPT models to extract this information from
the narrative clinical notes. Different from the existing work, our research
focuses on investigating the zero-shot learning on extracting this information
together by providing minimum information to the GPT model. We utilize
de-identified real-world clinical notes annotated for demographics, various
social determinants, and family history information. Given that the GPT model
might provide text different from the text in the original data, we explore two
sets of evaluation metrics, including the traditional NER evaluation metrics
and semantic similarity evaluation metrics, to completely understand the
performance. Our results show that the GPT-3.5 method achieved an average of
0.975 F1 on demographics extraction, 0.615 F1 on social determinants
extraction, and 0.722 F1 on family history extraction. We believe these results
can be further improved through model fine-tuning or few-shots learning.
Through the case studies, we also identified the limitations of the GPT models,
which need to be addressed in future research.
less
By: Hai-Long Nguyen, Dieu-Quynh Nguyen, Hoang-Trung Nguyen, Thu-Trang Pham, Huu-Dong Nguyen, Thach-Anh Nguyen, Thi-Hai-Yen Vuong, Ha-Thanh Nguyen
In recent years, natural language processing has gained significant
popularity in various sectors, including the legal domain. This paper presents
NeCo Team's solutions to the Vietnamese text processing tasks provided in the
Automated Legal Question Answering Competition 2023 (ALQAC 2023), focusing on
legal domain knowledge acquisition for low-resource languages through data
enrichment. Our methods for the legal document retrieval task empl... more
In recent years, natural language processing has gained significant
popularity in various sectors, including the legal domain. This paper presents
NeCo Team's solutions to the Vietnamese text processing tasks provided in the
Automated Legal Question Answering Competition 2023 (ALQAC 2023), focusing on
legal domain knowledge acquisition for low-resource languages through data
enrichment. Our methods for the legal document retrieval task employ a
combination of similarity ranking and deep learning models, while for the
second task, which requires extracting an answer from a relevant legal article
in response to a question, we propose a range of adaptive techniques to handle
different question types. Our approaches achieve outstanding results on both
tasks of the competition, demonstrating the potential benefits and
effectiveness of question answering systems in the legal field, particularly
for low-resource languages.
less
By: Ha-Thanh Nguyen, Randy Goebel, Francesca Toni, Kostas Stathis, Ken Satoh
The evolution of Generative Pre-trained Transformer (GPT) models has led to
significant advancements in various natural language processing applications,
particularly in legal textual entailment. We present an analysis of GPT-3.5
(ChatGPT) and GPT-4 performances on COLIEE Task 4 dataset, a prominent
benchmark in this domain. The study encompasses data from Heisei 18 (2006) to
Reiwa 3 (2021), exploring the models' abilities to discern entail... more
The evolution of Generative Pre-trained Transformer (GPT) models has led to
significant advancements in various natural language processing applications,
particularly in legal textual entailment. We present an analysis of GPT-3.5
(ChatGPT) and GPT-4 performances on COLIEE Task 4 dataset, a prominent
benchmark in this domain. The study encompasses data from Heisei 18 (2006) to
Reiwa 3 (2021), exploring the models' abilities to discern entailment
relationships within Japanese statute law across different periods. Our
preliminary experimental results unveil intriguing insights into the models'
strengths and weaknesses in handling legal textual entailment tasks, as well as
the patterns observed in model performance. In the context of proprietary
models with undisclosed architectures and weights, black-box analysis becomes
crucial for evaluating their capabilities. We discuss the influence of training
data distribution and the implications on the models' generalizability. This
analysis serves as a foundation for future research, aiming to optimize
GPT-based models and enable their successful adoption in legal information
extraction and entailment applications.
less
By: Thibault Douzon, Stefan Duffner, Christophe Garcia, Jérémy Espinas
Since their release, Transformers have revolutionized many fields from
Natural Language Understanding to Computer Vision. Document Understanding (DU)
was not left behind with first Transformer based models for DU dating from late
2019. However, the computational complexity of the self-attention operation
limits their capabilities to small sequences. In this paper we explore multiple
strategies to apply Transformer based models to long multi... more
Since their release, Transformers have revolutionized many fields from
Natural Language Understanding to Computer Vision. Document Understanding (DU)
was not left behind with first Transformer based models for DU dating from late
2019. However, the computational complexity of the self-attention operation
limits their capabilities to small sequences. In this paper we explore multiple
strategies to apply Transformer based models to long multi-page documents. We
introduce 2 new multi-modal (text + layout) long-range models for DU. They are
based on efficient implementations of Transformers for long sequences.
Long-range models can process whole documents at once effectively and are less
impaired by the document's length. We compare them to LayoutLM, a classical
Transformer adapted for DU and pre-trained on millions of documents. We further
propose 2D relative attention bias to guide self-attention towards relevant
tokens without harming model efficiency. We observe improvements on multi-page
business documents on Information Retrieval for a small performance cost on
smaller sequences. Relative 2D attention revealed to be effective on dense text
for both normal and long-range models.
less
By: Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, Kaokao Lv
Large Language Models (LLMs) have proven their exceptional capabilities in
performing language-related tasks. However, their deployment poses significant
challenges due to their considerable memory and storage requirements. In
response to this issue, weight-only quantization, particularly 3 and 4-bit
weight-only quantization, has emerged as one of the most viable solutions. As
the number of bits decreases, the quantization grid broadens, th... more
Large Language Models (LLMs) have proven their exceptional capabilities in
performing language-related tasks. However, their deployment poses significant
challenges due to their considerable memory and storage requirements. In
response to this issue, weight-only quantization, particularly 3 and 4-bit
weight-only quantization, has emerged as one of the most viable solutions. As
the number of bits decreases, the quantization grid broadens, thus emphasizing
the importance of up and down rounding. While previous studies have
demonstrated that fine-tuning up and down rounding with the addition of
perturbations can enhance accuracy in some scenarios, our study is driven by
the precise and limited boundary of these perturbations, where only the
threshold for altering the rounding value is of significance. Consequently, we
propose a concise and highly effective approach for optimizing the weight
rounding task. Our method, named SignRound, involves lightweight block-wise
tuning using signed gradient descent, enabling us to achieve outstanding
results within 400 steps. SignRound outperforms the established baseline of
rounding-to-nearest (RTN) and competes impressively against recent methods,
without introducing additional inference overhead. The source code will be
publicly available at https://github.com/intel/neural-compressor soon.
less
By: Chengyu Wang, Zhongjie Duan, Bingyan Liu, Xinyi Zou, Cen Chen, Kui Jia, Jun Huang
Text-to-image synthesis for the Chinese language poses unique challenges due
to its large vocabulary size, and intricate character relationships. While
existing diffusion models have shown promise in generating images from textual
descriptions, they often neglect domain-specific contexts and lack robustness
in handling the Chinese language. This paper introduces PAI-Diffusion, a
comprehensive framework that addresses these limitations. PAI-... more
Text-to-image synthesis for the Chinese language poses unique challenges due
to its large vocabulary size, and intricate character relationships. While
existing diffusion models have shown promise in generating images from textual
descriptions, they often neglect domain-specific contexts and lack robustness
in handling the Chinese language. This paper introduces PAI-Diffusion, a
comprehensive framework that addresses these limitations. PAI-Diffusion
incorporates both general and domain-specific Chinese diffusion models,
enabling the generation of contextually relevant images. It explores the
potential of using LoRA and ControlNet for fine-grained image style transfer
and image editing, empowering users with enhanced control over image
generation. Moreover, PAI-Diffusion seamlessly integrates with Alibaba Cloud's
Machine Learning Platform for AI, providing accessible and scalable solutions.
All the Chinese diffusion model checkpoints, LoRAs, and ControlNets, including
domain-specific ones, are publicly available. A user-friendly Chinese WebUI and
the diffusers-api elastic inference toolkit, also open-sourced, further
facilitate the easy deployment of PAI-Diffusion models in various environments,
making it a valuable resource for Chinese text-to-image synthesis.
less
By: Ruibo Chen, Zhiyuan Zhang, Yi Liu, Ruihan Bao, Keiko Harimoto, Xu Sun
Multimodal stock trading volume movement prediction with stock-related news
is one of the fundamental problems in the financial area. Existing multimodal
works that train models from scratch face the problem of lacking universal
knowledge when modeling financial news. In addition, the models ability may be
limited by the lack of domain-related knowledge due to insufficient data in the
datasets. To handle this issue, we propose the Prompt-ba... more
Multimodal stock trading volume movement prediction with stock-related news
is one of the fundamental problems in the financial area. Existing multimodal
works that train models from scratch face the problem of lacking universal
knowledge when modeling financial news. In addition, the models ability may be
limited by the lack of domain-related knowledge due to insufficient data in the
datasets. To handle this issue, we propose the Prompt-based MUltimodal Stock
volumE prediction model (ProMUSE) to process text and time series modalities.
We use pre-trained language models for better comprehension of financial news
and adopt prompt learning methods to leverage their capability in universal
knowledge to model textual information. Besides, simply fusing two modalities
can cause harm to the unimodal representations. Thus, we propose a novel
cross-modality contrastive alignment while reserving the unimodal heads beside
the fusion head to mitigate this problem. Extensive experiments demonstrate
that our proposed ProMUSE outperforms existing baselines. Comprehensive
analyses further validate the effectiveness of our architecture compared to
potential variants and learning mechanisms.
less
By: Wei Du, Laksh Advani, Yashmeet Gambhir, Daniel J Perry, Prashant Shiralkar, Zhengzheng Xing, Aaron Colak
Large language models (LLMs) have demonstrated significant capability to
generalize across a large number of NLP tasks. For industry applications, it is
imperative to assess the performance of the LLM on unlabeled production data
from time to time to validate for a real-world setting. Human labeling to
assess model error requires considerable expense and time delay. Here we
demonstrate that ensemble disagreement scores work well as a proxy ... more
Large language models (LLMs) have demonstrated significant capability to
generalize across a large number of NLP tasks. For industry applications, it is
imperative to assess the performance of the LLM on unlabeled production data
from time to time to validate for a real-world setting. Human labeling to
assess model error requires considerable expense and time delay. Here we
demonstrate that ensemble disagreement scores work well as a proxy for human
labeling for language models in zero-shot, few-shot, and fine-tuned settings,
per our evaluation on keyphrase extraction (KPE) task. We measure fidelity of
the results by comparing to true error measured from human labeled ground
truth. We contrast with the alternative of using another LLM as a source of
machine labels, or silver labels. Results across various languages and domains
show disagreement scores provide a better estimation of model performance with
mean average error (MAE) as low as 0.4% and on average 13.8% better than using
silver labels.
less
By: Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, Wenhu Chen
We introduce MAmmoTH, a series of open-source large language models (LLMs)
specifically tailored for general math problem-solving. The MAmmoTH models are
trained on MathInstruct, our meticulously curated instruction tuning dataset.
MathInstruct is compiled from 13 math datasets with intermediate rationales,
six of which have rationales newly curated by us. It presents a unique hybrid
of chain-of-thought (CoT) and program-of-thought (PoT) ra... more
We introduce MAmmoTH, a series of open-source large language models (LLMs)
specifically tailored for general math problem-solving. The MAmmoTH models are
trained on MathInstruct, our meticulously curated instruction tuning dataset.
MathInstruct is compiled from 13 math datasets with intermediate rationales,
six of which have rationales newly curated by us. It presents a unique hybrid
of chain-of-thought (CoT) and program-of-thought (PoT) rationales, and also
ensures extensive coverage of diverse fields in math. The hybrid of CoT and PoT
not only unleashes the potential of tool use but also allows different thought
processes for different math problems. As a result, the MAmmoTH series
substantially outperform existing open-source models on nine mathematical
reasoning datasets across all scales with an average accuracy gain between 13%
and 29%. Remarkably, our MAmmoTH-7B model reaches 35% on MATH (a
competition-level dataset), which exceeds the best open-source 7B model
(WizardMath) by 25%, and the MAmmoTH-34B model achieves 46% accuracy on MATH,
even surpassing GPT-4's CoT result. Our work underscores the importance of
diverse problem coverage and the use of hybrid rationales in developing
superior math generalist models.
less
Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models
0upvotes
By: Joseph Marvin Imperial, Harish Tayyar Madabushi
Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL)
and the Common European Framework of Reference for Languages (CEFR) exist to
guide teachers and educators to properly assess the complexity of educational
materials before administering them for classroom use. In this study, we select
a diverse set of open and closed-source instruction-tuned language models and
investigate their performances in writing story complet... more
Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL)
and the Common European Framework of Reference for Languages (CEFR) exist to
guide teachers and educators to properly assess the complexity of educational
materials before administering them for classroom use. In this study, we select
a diverse set of open and closed-source instruction-tuned language models and
investigate their performances in writing story completions and simplifying
narratives$-$tasks that teachers perform$-$using standard-guided prompts
controlling text readability. Our extensive findings provide empirical proof of
how globally recognized models like ChatGPT may be considered less effective
and may require more refined prompts for these generative tasks compared to
other open-sourced models such as BLOOMZ and FlanT5$-$which have shown
promising results.
less
By: Rabindra Lamsal, Maria Rodriguez Read, Shanika Karunasekera
Social media platforms play an essential role in crisis communication, but
analyzing crisis-related social media texts is challenging due to their
informal nature. Transformer-based pre-trained models like BERT and RoBERTa
have shown success in various NLP tasks, but they are not tailored for
crisis-related texts. Furthermore, general-purpose sentence encoders are used
to generate sentence embeddings, regardless of the textual complexities ... more
Social media platforms play an essential role in crisis communication, but
analyzing crisis-related social media texts is challenging due to their
informal nature. Transformer-based pre-trained models like BERT and RoBERTa
have shown success in various NLP tasks, but they are not tailored for
crisis-related texts. Furthermore, general-purpose sentence encoders are used
to generate sentence embeddings, regardless of the textual complexities in
crisis-related texts. Advances in applications like text classification,
semantic search, and clustering contribute to effective processing of
crisis-related texts, which is essential for emergency responders to gain a
comprehensive view of a crisis event, whether historical or real-time. To
address these gaps in crisis informatics literature, this study introduces
CrisisTransformers, an ensemble of pre-trained language models and sentence
encoders trained on an extensive corpus of over 15 billion word tokens from
tweets associated with more than 30 crisis events, including disease outbreaks,
natural disasters, conflicts, and other critical incidents. We evaluate
existing models and CrisisTransformers on 18 crisis-specific public datasets.
Our pre-trained models outperform strong baselines across all datasets in
classification tasks, and our best-performing sentence encoder improves the
state-of-the-art by 17.43% in sentence encoding tasks. Additionally, we
investigate the impact of model initialization on convergence and evaluate the
significance of domain-specific models in generating semantically meaningful
sentence embeddings. All models are publicly released
(https://huggingface.co/crisistransformers), with the anticipation that they
will serve as a robust baseline for tasks involving the analysis of
crisis-related social media texts.
less
By: Abhilash Datta, Souvic Chakraborty, Animesh Mukherjee
Personality types are important in various fields as they hold relevant
information about the characteristics of a human being in an explainable
format. They are often good predictors of a person's behaviors in a particular
environment and have applications ranging from candidate selection to marketing
and mental health. Recently automatic detection of personality traits from
texts has gained significant attention in computational linguisti... more
Personality types are important in various fields as they hold relevant
information about the characteristics of a human being in an explainable
format. They are often good predictors of a person's behaviors in a particular
environment and have applications ranging from candidate selection to marketing
and mental health. Recently automatic detection of personality traits from
texts has gained significant attention in computational linguistics. Most
personality detection and analysis methods have focused on small datasets
making their experimental observations often limited. To bridge this gap, we
focus on collecting and releasing the largest automatically curated dataset for
the research community which has 152 million tweets and 56 thousand data points
for the Myers-Briggs personality type (MBTI) prediction task. We perform a
series of extensive qualitative and quantitative studies on our dataset to
analyze the data patterns in a better way and infer conclusions. We show how
our intriguing analysis results often follow natural intuition. We also perform
a series of ablation studies to show how the baselines perform for our dataset.
less
By: Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, Ajay Divakaran
Vision-language models (VLMs) have recently demonstrated strong efficacy as
visual assistants that can parse natural queries about the visual content and
generate human-like outputs. In this work, we explore the ability of these
models to demonstrate human-like reasoning based on the perceived information.
To address a crucial concern regarding the extent to which their reasoning
capabilities are fully consistent and grounded, we also measu... more
Vision-language models (VLMs) have recently demonstrated strong efficacy as
visual assistants that can parse natural queries about the visual content and
generate human-like outputs. In this work, we explore the ability of these
models to demonstrate human-like reasoning based on the perceived information.
To address a crucial concern regarding the extent to which their reasoning
capabilities are fully consistent and grounded, we also measure the reasoning
consistency of these models. We achieve this by proposing a chain-of-thought
(CoT) based consistency measure. However, such an evaluation requires a
benchmark that encompasses both high-level inference and detailed reasoning
chains, which is costly. We tackle this challenge by proposing a
LLM-Human-in-the-Loop pipeline, which notably reduces cost while simultaneously
ensuring the generation of a high-quality dataset. Based on this pipeline and
the existing coarse-grained annotated dataset, we build the CURE benchmark to
measure both the zero-shot reasoning performance and consistency of VLMs. We
evaluate existing state-of-the-art VLMs, and find that even the best-performing
model is unable to demonstrate strong visual reasoning capabilities and
consistency, indicating that substantial efforts are required to enable VLMs to
perform visual reasoning as systematically and consistently as humans. As an
early step, we propose a two-stage training framework aimed at improving both
the reasoning performance and consistency of VLMs. The first stage involves
employing supervised fine-tuning of VLMs using step-by-step reasoning samples
automatically generated by LLMs. In the second stage, we further augment the
training process by incorporating feedback provided by LLMs to produce
reasoning chains that are highly consistent and grounded. We empirically
highlight the effectiveness of our framework in both reasoning performance and
consistency.
less
By: Ronald Seoh, Haw-Shiuan Chang, Andrew McCallum
Many useful tasks on scientific documents, such as topic classification and
citation prediction, involve corpora that span multiple scientific domains.
Typically, such tasks are accomplished by representing the text with a vector
embedding obtained from a Transformer's single CLS token. In this paper, we
argue that using multiple CLS tokens could make a Transformer better specialize
to multiple scientific domains. We present Multi2SPE: it e... more
Many useful tasks on scientific documents, such as topic classification and
citation prediction, involve corpora that span multiple scientific domains.
Typically, such tasks are accomplished by representing the text with a vector
embedding obtained from a Transformer's single CLS token. In this paper, we
argue that using multiple CLS tokens could make a Transformer better specialize
to multiple scientific domains. We present Multi2SPE: it encourages each of
multiple CLS tokens to learn diverse ways of aggregating token embeddings, then
sums them up together to create a single vector representation. We also propose
our new multi-domain benchmark, Multi-SciDocs, to test scientific paper vector
encoders under multi-domain settings. We show that Multi2SPE reduces error by
up to 25 percent in multi-domain citation prediction, while requiring only a
negligible amount of computation in addition to one BERT forward pass.
less
By: Haochun Wang, Sendong Zhao, Zewen Qiang, Zijian Li, Nuwa Xi, Yanrui Du, MuZhen Cai, Haoqiang Guo, Yuhan Chen, Haoming Xu, Bing Qin, Ting Liu
Large Language Models (LLMs) have demonstrated remarkable success in diverse
natural language processing (NLP) tasks in general domains. However, LLMs
sometimes generate responses with the hallucination about medical facts due to
limited domain knowledge. Such shortcomings pose potential risks in the
utilization of LLMs within medical contexts. To address this challenge, we
propose knowledge-tuning, which leverages structured medical knowle... more
Large Language Models (LLMs) have demonstrated remarkable success in diverse
natural language processing (NLP) tasks in general domains. However, LLMs
sometimes generate responses with the hallucination about medical facts due to
limited domain knowledge. Such shortcomings pose potential risks in the
utilization of LLMs within medical contexts. To address this challenge, we
propose knowledge-tuning, which leverages structured medical knowledge bases
for the LLMs to grasp domain knowledge efficiently and facilitate reliable
response generation. We also release cMedKnowQA, a Chinese medical knowledge
question-answering dataset constructed from medical knowledge bases to assess
the medical knowledge proficiency of LLMs. Experimental results show that the
LLMs which are knowledge-tuned with cMedKnowQA, can exhibit higher levels of
accuracy in response generation compared with vanilla instruction-tuning and
offer a new reliable way for the domain adaptation of LLMs.
less
Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity
0upvotes
By: Ng Shyh-Chang, A-Li Luo, Bo Qiu
We prove the converse of the universal approximation theorem, i.e. a neural
network (NN) encoding theorem which shows that for every stably converged NN of
continuous activation functions, its weight matrix actually encodes a
continuous function that approximates its training dataset to within a finite
margin of error over a bounded domain. We further show that using the
Eckart-Young theorem for truncated singular value decomposition of the... more
We prove the converse of the universal approximation theorem, i.e. a neural
network (NN) encoding theorem which shows that for every stably converged NN of
continuous activation functions, its weight matrix actually encodes a
continuous function that approximates its training dataset to within a finite
margin of error over a bounded domain. We further show that using the
Eckart-Young theorem for truncated singular value decomposition of the weight
matrix for every NN layer, we can illuminate the nature of the latent space
manifold of the training dataset encoded and represented by every NN layer, and
the geometric nature of the mathematical operations performed by each NN layer.
Our results have implications for understanding how NNs break the curse of
dimensionality by harnessing memory capacity for expressivity, and that the two
are complementary. This Layer Matrix Decomposition (LMD) further suggests a
close relationship between eigen-decomposition of NN layers and the latest
advances in conceptualizations of Hopfield networks and Transformer NN models.
less
By: Zhifeng Kong, Wei Ping, Ambrish Dantrey, Bryan Catanzaro
In this work, we present CleanUNet 2, a speech denoising model that combines
the advantages of waveform denoiser and spectrogram denoiser and achieves the
best of both worlds. CleanUNet 2 uses a two-stage framework inspired by popular
speech synthesis methods that consist of a waveform model and a spectrogram
model. Specifically, CleanUNet 2 builds upon CleanUNet, the state-of-the-art
waveform denoiser, and further boosts its performance by... more
In this work, we present CleanUNet 2, a speech denoising model that combines
the advantages of waveform denoiser and spectrogram denoiser and achieves the
best of both worlds. CleanUNet 2 uses a two-stage framework inspired by popular
speech synthesis methods that consist of a waveform model and a spectrogram
model. Specifically, CleanUNet 2 builds upon CleanUNet, the state-of-the-art
waveform denoiser, and further boosts its performance by taking predicted
spectrograms from a spectrogram denoiser as the input. We demonstrate that
CleanUNet 2 outperforms previous methods in terms of various objective and
subjective evaluations.
less
By: Sadia Kamal, Jimmy Hartford, Jeremy Willis, Arunkumar Bagavathi
Quantification of the political leaning of online news articles can aid in
understanding the dynamics of political ideology in social groups and measures
to mitigating them. However, predicting the accurate political leaning of a
news article with machine learning models is a challenging task. This is due to
(i) the political ideology of a news article is defined by several factors, and
(ii) the innate nature of existing learning models to ... more
Quantification of the political leaning of online news articles can aid in
understanding the dynamics of political ideology in social groups and measures
to mitigating them. However, predicting the accurate political leaning of a
news article with machine learning models is a challenging task. This is due to
(i) the political ideology of a news article is defined by several factors, and
(ii) the innate nature of existing learning models to be biased with the
political bias of the news publisher during the model training. There is only a
limited number of methods to study the political leaning of news articles which
also do not consider the algorithmic political bias which lowers the
generalization of machine learning models to predict the political leaning of
news articles published by any new news publishers. In this work, we propose a
knowledge-infused deep learning model that utilizes relatively reliable
external data resources to learn unbiased representations of news articles
using their global and local contexts. We evaluate the proposed model by
setting the data in such a way that news domains or news publishers in the test
set are completely unseen during the training phase. With this setup we show
that the proposed model mitigates algorithmic political bias and outperforms
baseline methods to predict the political leaning of news articles with up to
73% accuracy.
less
By: Jingpu Cheng, Qianxiao Li, Ting Lin, Zuowei Shen
We investigate the expressive power of deep residual neural networks
idealized as continuous dynamical systems through control theory. Specifically,
we consider two properties that arise from supervised learning, namely
universal interpolation - the ability to match arbitrary input and target
training samples - and the closely related notion of universal approximation -
the ability to approximate input-target functional relationships via fl... more
We investigate the expressive power of deep residual neural networks
idealized as continuous dynamical systems through control theory. Specifically,
we consider two properties that arise from supervised learning, namely
universal interpolation - the ability to match arbitrary input and target
training samples - and the closely related notion of universal approximation -
the ability to approximate input-target functional relationships via flow maps.
Under the assumption of affine invariance of the control family, we give a
characterisation of universal interpolation, showing that it holds for
essentially any architecture with non-linearity. Furthermore, we elucidate the
relationship between universal interpolation and universal approximation in the
context of general control systems, showing that the two properties cannot be
deduced from each other. At the same time, we identify conditions on the
control family and the target function that ensures the equivalence of the two
notions.
less
By: Marwa Chafii, Salmane Naoumi, Reda Alami, Ebtesam Almazrouei, Mehdi Bennis, Merouane Debbah
In different wireless network scenarios, multiple network entities need to
cooperate in order to achieve a common task with minimum delay and energy
consumption. Future wireless networks mandate exchanging high dimensional data
in dynamic and uncertain environments, therefore implementing communication
control tasks becomes challenging and highly complex. Multi-agent reinforcement
learning with emergent communication (EC-MARL) is a promisin... more
In different wireless network scenarios, multiple network entities need to
cooperate in order to achieve a common task with minimum delay and energy
consumption. Future wireless networks mandate exchanging high dimensional data
in dynamic and uncertain environments, therefore implementing communication
control tasks becomes challenging and highly complex. Multi-agent reinforcement
learning with emergent communication (EC-MARL) is a promising solution to
address high dimensional continuous control problems with partially observable
states in a cooperative fashion where agents build an emergent communication
protocol to solve complex tasks. This paper articulates the importance of
EC-MARL within the context of future 6G wireless networks, which imbues
autonomous decision-making capabilities into network entities to solve complex
tasks such as autonomous driving, robot navigation, flying base stations
network planning, and smart city applications. An overview of EC-MARL
algorithms and their design criteria are provided while presenting use cases
and research opportunities on this emerging topic.
less
By: Jingcan Duan, Pei Zhang, Siwei Wang, Jingtao Hu, Hu Jin, Jiaxin Zhang, Haifang Zhou, Haifang Zhou
Graph anomaly detection (GAD) has attracted increasing attention in machine
learning and data mining. Recent works have mainly focused on how to capture
richer information to improve the quality of node embeddings for GAD. Despite
their significant advances in detection performance, there is still a relative
dearth of research on the properties of the task. GAD aims to discern the
anomalies that deviate from most nodes. However, the model i... more
Graph anomaly detection (GAD) has attracted increasing attention in machine
learning and data mining. Recent works have mainly focused on how to capture
richer information to improve the quality of node embeddings for GAD. Despite
their significant advances in detection performance, there is still a relative
dearth of research on the properties of the task. GAD aims to discern the
anomalies that deviate from most nodes. However, the model is prone to learn
the pattern of normal samples which make up the majority of samples. Meanwhile,
anomalies can be easily detected when their behaviors differ from normality.
Therefore, the performance can be further improved by enhancing the ability to
learn the normal pattern. To this end, we propose a normality learning-based
GAD framework via multi-scale contrastive learning networks (NLGAD for
abbreviation). Specifically, we first initialize the model with the contrastive
networks on different scales. To provide sufficient and reliable normal nodes
for normality learning, we design an effective hybrid strategy for normality
selection. Finally, the model is refined with the only input of reliable normal
nodes and learns a more accurate estimate of normality so that anomalous nodes
can be more easily distinguished. Eventually, extensive experiments on six
benchmark graph datasets demonstrate the effectiveness of our normality
learning-based scheme on GAD. Notably, the proposed algorithm improves the
detection performance (up to 5.89% AUC gain) compared with the state-of-the-art
methods. The source code is released at https://github.com/FelixDJC/NLGAD.
less
By: Ákos Hajnal
This paper presents a novel online learning method that aims at finding a
separator hyperplane between data points labelled as either positive or
negative. Since weights and biases of artificial neurons can directly be
related to hyperplanes in high-dimensional spaces, the technique is applicable
to train perceptron-based binary classifiers in machine learning. In case of
large or imbalanced data sets, use of analytical or gradient-based so... more
This paper presents a novel online learning method that aims at finding a
separator hyperplane between data points labelled as either positive or
negative. Since weights and biases of artificial neurons can directly be
related to hyperplanes in high-dimensional spaces, the technique is applicable
to train perceptron-based binary classifiers in machine learning. In case of
large or imbalanced data sets, use of analytical or gradient-based solutions
can become prohibitive and impractical, where heuristics and approximation
techniques are still applicable. The proposed method is based on the Perceptron
algorithm, however, it tunes neuron weights in just the necessary extent during
searching the separator hyperplane. Due to an appropriate transformation of the
initial data set we need not to consider data labels, neither the bias term.
respectively, reducing separability to a one-class classification problem. The
presented method has proven converge; empirical results show that it can be
more efficient than the Perceptron algorithm, especially, when the size of the
data set exceeds data dimensionality.
less
By: Jingwen Fu, Tao Yang, Yuwang Wang, Yan Lu, Nanning Zheng
In-context learning, i.e., learning from in-context samples, is an impressive
ability of Transformer. However, the mechanism driving the in-context learning
is not yet fully understood. In this study, we aim to investigate from an
underexplored perspective of representation learning. The representation is
more complex for in-context learning senario, where the representation can be
impacted by both model weights and in-context samples. We r... more
In-context learning, i.e., learning from in-context samples, is an impressive
ability of Transformer. However, the mechanism driving the in-context learning
is not yet fully understood. In this study, we aim to investigate from an
underexplored perspective of representation learning. The representation is
more complex for in-context learning senario, where the representation can be
impacted by both model weights and in-context samples. We refer the above two
conceptually aspects of representation as in-weight component and in-context
component, respectively. To study how the two components affect in-context
learning capabilities, we construct a novel synthetic task, making it possible
to device two probes, in-weights probe and in-context probe, to evaluate the
two components, respectively. We demonstrate that the goodness of in-context
component is highly related to the in-context learning performance, which
indicates the entanglement between in-context learning and representation
learning. Furthermore, we find that a good in-weights component can actually
benefit the learning of the in-context component, indicating that in-weights
learning should be the foundation of in-context learning. To further understand
the the in-context learning mechanism and importance of the in-weights
component, we proof by construction that a simple Transformer, which uses
pattern matching and copy-past mechanism to perform in-context learning, can
match the in-context learning performance with more complex, best tuned
Transformer under the perfect in-weights component assumption. In short, those
discoveries from representation learning perspective shed light on new
approaches to improve the in-context capacity.
less
By: Jeroen M. Galjaard, Robert Birke, Juan Perez, Lydia Y. Chen
The negative impact of label noise is well studied in classical supervised
learning yet remains an open research question in meta-learning. Meta-learners
aim to adapt to unseen learning tasks by learning a good initial model in
meta-training and consecutively fine-tuning it according to new tasks during
meta-testing. In this paper, we present the first extensive analysis of the
impact of varying levels of label noise on the performance of s... more
The negative impact of label noise is well studied in classical supervised
learning yet remains an open research question in meta-learning. Meta-learners
aim to adapt to unseen learning tasks by learning a good initial model in
meta-training and consecutively fine-tuning it according to new tasks during
meta-testing. In this paper, we present the first extensive analysis of the
impact of varying levels of label noise on the performance of state-of-the-art
meta-learners, specifically gradient-based $N$-way $K$-shot learners. We show
that the accuracy of Reptile, iMAML, and foMAML drops by up to 42% on the
Omniglot and CifarFS datasets when meta-training is affected by label noise. To
strengthen the resilience against label noise, we propose two sampling
techniques, namely manifold (Man) and batch manifold (BatMan), which transform
the noisy supervised learners into semi-supervised ones to increase the utility
of noisy labels. We first construct manifold samples of $N$-way
$2$-contrastive-shot tasks through augmentation, learning the embedding via a
contrastive loss in meta-training, and then perform classification through
zeroing on the embedding in meta-testing. We show that our approach can
effectively mitigate the impact of meta-training label noise. Even with 60%
wrong labels \batman and \man can limit the meta-testing accuracy drop to
${2.5}$, ${9.4}$, ${1.1}$ percent points, respectively, with existing
meta-learners across the Omniglot, CifarFS, and MiniImagenet datasets.
less
By: Cheng Chen, Lei Fan
Landslides are a common natural disaster that can cause casualties, property
safety threats and economic losses. Therefore, it is important to understand or
predict the probability of landslide occurrence at potentially risky sites. A
commonly used means is to carry out a landslide susceptibility assessment based
on a landslide inventory and a set of landslide contributing factors. This can
be readily achieved using machine learning (ML) mo... more
Landslides are a common natural disaster that can cause casualties, property
safety threats and economic losses. Therefore, it is important to understand or
predict the probability of landslide occurrence at potentially risky sites. A
commonly used means is to carry out a landslide susceptibility assessment based
on a landslide inventory and a set of landslide contributing factors. This can
be readily achieved using machine learning (ML) models such as logistic
regression (LR), support vector machine (SVM), random forest (RF), extreme
gradient boosting (Xgboost), or deep learning (DL) models such as convolutional
neural network (CNN) and long short time memory (LSTM). As the input data for
these models, landslide contributing factors have varying influences on
landslide occurrence. Therefore, it is logically feasible to select more
important contributing factors and eliminate less relevant ones, with the aim
of increasing the prediction accuracy of these models. However, selecting more
important factors is still a challenging task and there is no generally
accepted method. Furthermore, the effects of factor selection using various
methods on the prediction accuracy of ML and DL models are unclear. In this
study, the impact of the selection of contributing factors on the accuracy of
landslide susceptibility predictions using ML and DL models was investigated.
Four methods for selecting contributing factors were considered for all the
aforementioned ML and DL models, which included Information Gain Ratio (IGR),
Recursive Feature Elimination (RFE), Particle Swarm Optimization (PSO), Least
Absolute Shrinkage and Selection Operators (LASSO) and Harris Hawk Optimization
(HHO). In addition, autoencoder-based factor selection methods for DL models
were also investigated. To assess their performances, an exhaustive approach
was adopted,...
less
A Machine Learning Framework to Deconstruct the Primary Drivers for Electricity Market Price Events
0upvotes
By: Milan Jain, Xueqing Sun, Sohom Datta, Abhishek Somani
Power grids are moving towards 100% renewable energy source bulk power grids,
and the overall dynamics of power system operations and electricity markets are
changing. The electricity markets are not only dispatching resources
economically but also taking into account various controllable actions like
renewable curtailment, transmission congestion mitigation, and energy storage
optimization to ensure grid reliability. As a result, price for... more
Power grids are moving towards 100% renewable energy source bulk power grids,
and the overall dynamics of power system operations and electricity markets are
changing. The electricity markets are not only dispatching resources
economically but also taking into account various controllable actions like
renewable curtailment, transmission congestion mitigation, and energy storage
optimization to ensure grid reliability. As a result, price formations in
electricity markets have become quite complex. Traditional root cause analysis
and statistical approaches are rendered inapplicable to analyze and infer the
main drivers behind price formation in the modern grid and markets with
variable renewable energy (VRE). In this paper, we propose a machine
learning-based analysis framework to deconstruct the primary drivers for price
spike events in modern electricity markets with high renewable energy. The
outcomes can be utilized for various critical aspects of market design,
renewable dispatch and curtailment, operations, and cyber-security
applications. The framework can be applied to any ISO or market data; however,
in this paper, it is applied to open-source publicly available datasets from
California Independent System Operator (CAISO) and ISO New England (ISO-NE).
less
By: Alex Gomez-Villa, Bartlomiej Twardowski, Kai Wang, Joost van de Weijer
Continuous unsupervised representation learning (CURL) research has greatly
benefited from improvements in self-supervised learning (SSL) techniques. As a
result, existing CURL methods using SSL can learn high-quality representations
without any labels, but with a notable performance drop when learning on a
many-tasks data stream. We hypothesize that this is caused by the
regularization losses that are imposed to prevent forgetting, leading... more
Continuous unsupervised representation learning (CURL) research has greatly
benefited from improvements in self-supervised learning (SSL) techniques. As a
result, existing CURL methods using SSL can learn high-quality representations
without any labels, but with a notable performance drop when learning on a
many-tasks data stream. We hypothesize that this is caused by the
regularization losses that are imposed to prevent forgetting, leading to a
suboptimal plasticity-stability trade-off: they either do not adapt fully to
the incoming data (low plasticity), or incur significant forgetting when
allowed to fully adapt to a new SSL pretext-task (low stability). In this work,
we propose to train an expert network that is relieved of the duty of keeping
the previous knowledge and can focus on performing optimally on the new tasks
(optimizing plasticity). In the second phase, we combine this new knowledge
with the previous network in an adaptation-retrospection phase to avoid
forgetting and initialize a new expert with the knowledge of the old network.
We perform several experiments showing that our proposed approach outperforms
other CURL exemplar-free methods in few- and many-task split settings.
Furthermore, we show how to adapt our approach to semi-supervised continual
learning (Semi-SCL) and show that we surpass the accuracy of other
exemplar-free Semi-SCL methods and reach the results of some others that use
exemplars.
less
By: Khoa Tran, Hai-Canh Vu, Lam Pham, Nassim Boudaoud
In this paper, a Robust Multi-branch Deep learning-based system for remaining
useful life (RUL) prediction and condition operations (CO) identification of
rotating machines is proposed. In particular, the proposed system comprises
main components: (1) an LSTM-Autoencoder to denoise the vibration data; (2) a
feature extraction to generate time-domain, frequency-domain, and
time-frequency based features from the denoised data; (3) a novel and... more
In this paper, a Robust Multi-branch Deep learning-based system for remaining
useful life (RUL) prediction and condition operations (CO) identification of
rotating machines is proposed. In particular, the proposed system comprises
main components: (1) an LSTM-Autoencoder to denoise the vibration data; (2) a
feature extraction to generate time-domain, frequency-domain, and
time-frequency based features from the denoised data; (3) a novel and robust
multi-branch deep learning network architecture to exploit the multiple
features. The performance of our proposed system was evaluated and compared to
the state-of-the-art systems on two benchmark datasets of XJTU-SY and
PRONOSTIA. The experimental results prove that our proposed system outperforms
the state-of-the-art systems and presents potential for real-life applications
on bearing machines.
less
By: Max Losch, David Stutz, Bernt Schiele, Mario Fritz
Despite recent success, state-of-the-art learning-based models remain highly
vulnerable to input changes such as adversarial examples. In order to obtain
certifiable robustness against such perturbations, recent work considers
Lipschitz-based regularizers or constraints while at the same time increasing
prediction margin. Unfortunately, this comes at the cost of significantly
decreased accuracy. In this paper, we propose a Calibrated Lipsch... more
Despite recent success, state-of-the-art learning-based models remain highly
vulnerable to input changes such as adversarial examples. In order to obtain
certifiable robustness against such perturbations, recent work considers
Lipschitz-based regularizers or constraints while at the same time increasing
prediction margin. Unfortunately, this comes at the cost of significantly
decreased accuracy. In this paper, we propose a Calibrated Lipschitz-Margin
Loss (CLL) that addresses this issue and improves certified robustness by
tackling two problems: Firstly, commonly used margin losses do not adjust the
penalties to the shrinking output distribution; caused by minimizing the
Lipschitz constant $K$. Secondly, and most importantly, we observe that
minimization of $K$ can lead to overly smooth decision functions. This limits
the model's complexity and thus reduces accuracy. Our CLL addresses these
issues by explicitly calibrating the loss w.r.t. margin and Lipschitz constant,
thereby establishing full control over slack and improving robustness
certificates even with larger Lipschitz constants. On CIFAR-10, CIFAR-100 and
Tiny-ImageNet, our models consistently outperform losses that leave the
constant unattended. On CIFAR-100 and Tiny-ImageNet, CLL improves upon
state-of-the-art deterministic $L_2$ robust accuracies. In contrast to current
trends, we unlock potential of much smaller models without $K=1$ constraints.
less
By: Dimitris Spathis, Fahim Kawsar
Large Language Models (LLMs) have demonstrated remarkable generalization
across diverse tasks, leading individuals to increasingly use them as personal
assistants and universal computing engines. Nevertheless, a notable obstacle
emerges when feeding numerical/temporal data into these models, such as data
sourced from wearables or electronic health records. LLMs employ tokenizers in
their input that break down text into smaller units. Howeve... more
Large Language Models (LLMs) have demonstrated remarkable generalization
across diverse tasks, leading individuals to increasingly use them as personal
assistants and universal computing engines. Nevertheless, a notable obstacle
emerges when feeding numerical/temporal data into these models, such as data
sourced from wearables or electronic health records. LLMs employ tokenizers in
their input that break down text into smaller units. However, tokenizers are
not designed to represent numerical values and might struggle to understand
repetitive patterns and context, treating consecutive values as separate tokens
and disregarding their temporal relationships. Here, we discuss recent works
that employ LLMs for human-centric tasks such as in mobile health sensing and
present a case study showing that popular LLMs tokenize temporal data
incorrectly. To address that, we highlight potential solutions such as prompt
tuning with lightweight embedding layers as well as multimodal adapters, that
can help bridge this "modality gap". While the capability of language models to
generalize to other modalities with minimal or no finetuning is exciting, this
paper underscores the fact that their outputs cannot be meaningful if they
stumble over input nuances.
less
By: Ali Baheri
In the dynamic and uncertain environments where reinforcement learning (RL)
operates, risk management becomes a crucial factor in ensuring reliable
decision-making. Traditional RL approaches, while effective in reward
optimization, often overlook the landscape of potential risks. In response,
this paper pioneers the integration of Optimal Transport (OT) theory with RL to
create a risk-aware framework. Our approach modifies the objective fun... more
In the dynamic and uncertain environments where reinforcement learning (RL)
operates, risk management becomes a crucial factor in ensuring reliable
decision-making. Traditional RL approaches, while effective in reward
optimization, often overlook the landscape of potential risks. In response,
this paper pioneers the integration of Optimal Transport (OT) theory with RL to
create a risk-aware framework. Our approach modifies the objective function,
ensuring that the resulting policy not only maximizes expected rewards but also
respects risk constraints dictated by OT distances between state visitation
distributions and the desired risk profiles. By leveraging the mathematical
precision of OT, we offer a formulation that elevates risk considerations
alongside conventional RL objectives. Our contributions are substantiated with
a series of theorems, mapping the relationships between risk distributions,
optimal value functions, and policy behaviors. Through the lens of OT, this
work illuminates a promising direction for RL, ensuring a balanced fusion of
reward pursuit and risk awareness.
less
By: Euihyeon Choi, Jooyoung Kim, Wonkyung Lee
Probability estimation models play an important role in various fields, such
as weather forecasting, recommendation systems, and sports analysis. Among
several models estimating probabilities, it is difficult to evaluate which
model gives reliable probabilities since the ground-truth probabilities are not
available. The win probability estimation model for esports, which calculates
the win probability under a certain game state, is also one... more
Probability estimation models play an important role in various fields, such
as weather forecasting, recommendation systems, and sports analysis. Among
several models estimating probabilities, it is difficult to evaluate which
model gives reliable probabilities since the ground-truth probabilities are not
available. The win probability estimation model for esports, which calculates
the win probability under a certain game state, is also one of the fields being
actively studied in probability estimation. However, most of the previous works
evaluated their models using accuracy, a metric that only can measure the
performance of discrimination. In this work, we firstly investigate the Brier
score and the Expected Calibration Error (ECE) as a replacement of accuracy
used as a performance evaluation metric for win probability estimation models
in esports field. Based on the analysis, we propose a novel metric called
Balance score which is a simple yet effective metric in terms of six good
properties that probability estimation metric should have. Under the general
condition, we also found that the Balance score can be an effective
approximation of the true expected calibration error which has been imperfectly
approximated by ECE using the binning technique. Extensive evaluations using
simulation studies and real game snapshot data demonstrate the promising
potential to adopt the proposed metric not only for the win probability
estimation model for esports but also for evaluating general probability
estimation models.
less
By: Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, Tong Zhang
Foundation models, including Vision Language Models (VLMs) and Large Language
Models (LLMs), possess the $generality$ to handle diverse distributions and
tasks, which stems from their extensive pre-training datasets. The fine-tuning
of foundation models is a common practice to enhance task performance or align
the model's behavior with human expectations, allowing them to gain
$speciality$. However, the small datasets used for fine-tuning m... more
Foundation models, including Vision Language Models (VLMs) and Large Language
Models (LLMs), possess the $generality$ to handle diverse distributions and
tasks, which stems from their extensive pre-training datasets. The fine-tuning
of foundation models is a common practice to enhance task performance or align
the model's behavior with human expectations, allowing them to gain
$speciality$. However, the small datasets used for fine-tuning may not
adequately cover the diverse distributions and tasks encountered during
pre-training. Consequently, the pursuit of speciality during fine-tuning can
lead to a loss of {generality} in the model, which is related to catastrophic
forgetting (CF) in deep learning. In this study, we demonstrate this phenomenon
in both VLMs and LLMs. For instance, fine-tuning VLMs like CLIP on ImageNet
results in a loss of generality in handling diverse distributions, and
fine-tuning LLMs like Galactica in the medical domain leads to a loss in
following instructions and common sense.
To address the trade-off between the speciality and generality, we
investigate multiple regularization methods from continual learning, the weight
averaging method (Wise-FT) from out-of-distributional (OOD) generalization,
which interpolates parameters between pre-trained and fine-tuned models, and
parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA). Our
findings show that both continual learning and Wise-ft methods effectively
mitigate the loss of generality, with Wise-FT exhibiting the strongest
performance in balancing speciality and generality.
less
By: Alexander Kleinsorge, Stefan Kupper, Alexander Fauck, Felix Rothe
We present a novel, fast (exponential rate adaption), ab initio
(hyper-parameter-free) gradient based optimizer algorithm. The main idea of the
method is to adapt the learning rate $\alpha$ by situational awareness, mainly
striving for orthogonal neighboring gradients. The method has a high success
and fast convergence rate and does not rely on hand-tuned parameters giving it
greater universality. It can be applied to problems of any dimens... more
We present a novel, fast (exponential rate adaption), ab initio
(hyper-parameter-free) gradient based optimizer algorithm. The main idea of the
method is to adapt the learning rate $\alpha$ by situational awareness, mainly
striving for orthogonal neighboring gradients. The method has a high success
and fast convergence rate and does not rely on hand-tuned parameters giving it
greater universality. It can be applied to problems of any dimensions n and
scales only linearly (of order O(n)) with the dimension of the problem. It
optimizes convex and non-convex continuous landscapes providing some kind of
gradient. In contrast to the Ada-family (AdaGrad, AdaMax, AdaDelta, Adam, etc.)
the method is rotation invariant: optimization path and performance are
independent of coordinate choices. The impressive performance is demonstrated
by extensive experiments on the MNIST benchmark data-set against
state-of-the-art optimizers. We name this new class of optimizers after its
core idea Exponential Learning Rate Adaption - ELRA. We present it in two
variants c2min and p2min with slightly different control. The authors strongly
believe that ELRA will open a completely new research direction for gradient
descent optimize.
less
By: Ole-Christoffer Granmo, Per-Arne Andersen, Lei Jiao, Xuan Zhang, Christian Blakely, Tor Tveit
A set of variables is the Markov blanket of a random variable if it contains
all the information needed for predicting the variable. If the blanket cannot
be reduced without losing useful information, it is called a Markov boundary.
Identifying the Markov boundary of a random variable is advantageous because
all variables outside the boundary are superfluous. Hence, the Markov boundary
provides an optimal feature set. However, learning the ... more
A set of variables is the Markov blanket of a random variable if it contains
all the information needed for predicting the variable. If the blanket cannot
be reduced without losing useful information, it is called a Markov boundary.
Identifying the Markov boundary of a random variable is advantageous because
all variables outside the boundary are superfluous. Hence, the Markov boundary
provides an optimal feature set. However, learning the Markov boundary from
data is challenging for two reasons. If one or more variables are removed from
the Markov boundary, variables outside the boundary may start providing
information. Conversely, variables within the boundary may stop providing
information. The true role of each candidate variable is only manifesting when
the Markov boundary has been identified. In this paper, we propose a new
Tsetlin Machine (TM) feedback scheme that supplements Type I and Type II
feedback. The scheme introduces a novel Finite State Automaton - a
Context-Specific Independence Automaton. The automaton learns which features
are outside the Markov boundary of the target, allowing them to be pruned from
the TM during learning. We investigate the new scheme empirically, showing how
it is capable of exploiting context-specific independence to find Markov
boundaries. Further, we provide a theoretical analysis of convergence. Our
approach thus connects the field of Bayesian networks (BN) with TMs,
potentially opening up for synergies when it comes to inference and learning,
including TM-produced Bayesian knowledge bases and TM-based Bayesian inference.
less
By: Daniel Fentham University of Birmingham, David Parker University of Oxford, Mark Ryan University of Birmingham
When deploying classifiers in the real world, users expect them to respond to
inputs appropriately. However, traditional classifiers are not equipped to
handle inputs which lie far from the distribution they were trained on.
Malicious actors can exploit this defect by making adversarial perturbations
designed to cause the classifier to give an incorrect output.
Classification-with-rejection methods attempt to solve this problem by allowing
... more
When deploying classifiers in the real world, users expect them to respond to
inputs appropriately. However, traditional classifiers are not equipped to
handle inputs which lie far from the distribution they were trained on.
Malicious actors can exploit this defect by making adversarial perturbations
designed to cause the classifier to give an incorrect output.
Classification-with-rejection methods attempt to solve this problem by allowing
networks to refuse to classify an input in which they have low confidence. This
works well for strongly adversarial examples, but also leads to the rejection
of weakly perturbed images, which intuitively could be correctly classified. To
address these issues, we propose Reed-Muller Aggregation Networks (RMAggNet), a
classifier inspired by Reed-Muller error-correction codes which can correct and
reject inputs. This paper shows that RMAggNet can minimise incorrectness while
maintaining good correctness over multiple adversarial attacks at different
perturbation budgets by leveraging the ability to correct errors in the
classification process. This provides an alternative
classification-with-rejection method which can reduce the amount of additional
processing in situations where a small number of incorrect classifications are
permissible.
less
By: Abhin Shah, Devavrat Shah, Gregory W. Wornell
We consider the classical problem of learning, with arbitrary accuracy, the
natural parameters of a $k$-parameter truncated \textit{minimal} exponential
family from i.i.d. samples in a computationally and statistically efficient
manner. We focus on the setting where the support as well as the natural
parameters are appropriately bounded. While the traditional maximum likelihood
estimator for this class of exponential family is consistent, a... more
We consider the classical problem of learning, with arbitrary accuracy, the
natural parameters of a $k$-parameter truncated \textit{minimal} exponential
family from i.i.d. samples in a computationally and statistically efficient
manner. We focus on the setting where the support as well as the natural
parameters are appropriately bounded. While the traditional maximum likelihood
estimator for this class of exponential family is consistent, asymptotically
normal, and asymptotically efficient, evaluating it is computationally hard. In
this work, we propose a novel loss function and a computationally efficient
estimator that is consistent as well as asymptotically normal under mild
conditions. We show that, at the population level, our method can be viewed as
the maximum likelihood estimation of a re-parameterized distribution belonging
to the same class of exponential family. Further, we show that our estimator
can be interpreted as a solution to minimizing a particular Bregman score as
well as an instance of minimizing the \textit{surrogate} likelihood. We also
provide finite sample guarantees to achieve an error (in $\ell_2$-norm) of
$\alpha$ in the parameter estimation with sample complexity $O({\sf
poly}(k)/\alpha^2)$. Our method achives the order-optimal sample complexity of
$O({\sf log}(k)/\alpha^2)$ when tailored for node-wise-sparse Markov random
fields. Finally, we demonstrate the performance of our estimator via numerical
experiments.
less
By: Miranda Bihler, Hala Nelson, Erin Okey, Noe Reyes Rivas, John Webb, Anna White
The Harrisonburg Department of Public Transportation (HDPT) aims to leverage
their data to improve the efficiency and effectiveness of their operations. We
construct two supply and demand models that help the department identify gaps
in their service. The models take many variables into account, including the
way that the HDPT reports to the federal government and the areas with the most
vulnerable populations in Harrisonburg City. We emplo... more
The Harrisonburg Department of Public Transportation (HDPT) aims to leverage
their data to improve the efficiency and effectiveness of their operations. We
construct two supply and demand models that help the department identify gaps
in their service. The models take many variables into account, including the
way that the HDPT reports to the federal government and the areas with the most
vulnerable populations in Harrisonburg City. We employ data analysis and
machine learning techniques to make our predictions.
less
By: Jiashu Pu, Shiwei Zhao, Ling Cheng, Yongzhu Chang, Runze Wu, Tangjie Lv, Rongsheng Zhang
Although the pre-training followed by fine-tuning paradigm is used
extensively in many fields, there is still some controversy surrounding the
impact of pre-training on the fine-tuning process. Currently, experimental
findings based on text and image data lack consensus. To delve deeper into the
unsupervised pre-training followed by fine-tuning paradigm, we have extended
previous research to a new modality: time series. In this study, we co... more
Although the pre-training followed by fine-tuning paradigm is used
extensively in many fields, there is still some controversy surrounding the
impact of pre-training on the fine-tuning process. Currently, experimental
findings based on text and image data lack consensus. To delve deeper into the
unsupervised pre-training followed by fine-tuning paradigm, we have extended
previous research to a new modality: time series. In this study, we conducted a
thorough examination of 150 classification datasets derived from the Univariate
Time Series (UTS) and Multivariate Time Series (MTS) benchmarks. Our analysis
reveals several key conclusions. (i) Pre-training can only help improve the
optimization process for models that fit the data poorly, rather than those
that fit the data well. (ii) Pre-training does not exhibit the effect of
regularization when given sufficient training time. (iii) Pre-training can only
speed up convergence if the model has sufficient ability to fit the data. (iv)
Adding more pre-training data does not improve generalization, but it can
strengthen the advantage of pre-training on the original data volume, such as
faster convergence. (v) While both the pre-training task and the model
structure determine the effectiveness of the paradigm on a given dataset, the
model structure plays a more significant role.
less
By: Haohao Qu, Haoxuan Kuang, Jun Li, Linlin You
Along with the proliferation of electric vehicles (EVs), optimizing the use
of EV charging space can significantly alleviate the growing load on
intelligent transportation systems. As the foundation to achieve such an
optimization, a spatiotemporal method for EV charging demand prediction in
urban areas is required. Although several solutions have been proposed by using
data-driven deep learning methods, it can be found that these
performan... more
Along with the proliferation of electric vehicles (EVs), optimizing the use
of EV charging space can significantly alleviate the growing load on
intelligent transportation systems. As the foundation to achieve such an
optimization, a spatiotemporal method for EV charging demand prediction in
urban areas is required. Although several solutions have been proposed by using
data-driven deep learning methods, it can be found that these
performance-oriented methods may suffer from misinterpretations to correctly
handle the reverse relationship between charging demands and prices. To tackle
the emerging challenges of training an accurate and interpretable prediction
model, this paper proposes a novel approach that enables the integration of
graph and temporal attention mechanisms for feature extraction and the usage of
physic-informed meta-learning in the model pre-training step for knowledge
transfer. Evaluation results on a dataset of 18,013 EV charging piles in
Shenzhen, China, show that the proposed approach, named PAG, can achieve
state-of-the-art forecasting performance and the ability in understanding the
adaptive changes in charging demands caused by price fluctuations.
less
By: Yucheng Wang, Yuecong Xu, Jianfei Yang, Min Wu, Xiaoli Li, Lihua Xie, Zhenghua Chen
Multivariate Time-Series (MTS) data is crucial in various application fields.
With its sequential and multi-source (multiple sensors) properties, MTS data
inherently exhibits Spatial-Temporal (ST) dependencies, involving temporal
correlations between timestamps and spatial correlations between sensors in
each timestamp. To effectively leverage this information, Graph Neural
Network-based methods (GNNs) have been widely adopted. However, exi... more
Multivariate Time-Series (MTS) data is crucial in various application fields.
With its sequential and multi-source (multiple sensors) properties, MTS data
inherently exhibits Spatial-Temporal (ST) dependencies, involving temporal
correlations between timestamps and spatial correlations between sensors in
each timestamp. To effectively leverage this information, Graph Neural
Network-based methods (GNNs) have been widely adopted. However, existing
approaches separately capture spatial dependency and temporal dependency and
fail to capture the correlations between Different sEnsors at Different
Timestamps (DEDT). Overlooking such correlations hinders the comprehensive
modelling of ST dependencies within MTS data, thus restricting existing GNNs
from learning effective representations. To address this limitation, we propose
a novel method called Fully-Connected Spatial-Temporal Graph Neural Network
(FC-STGNN), including two key components namely FC graph construction and FC
graph convolution. For graph construction, we design a decay graph to connect
sensors across all timestamps based on their temporal distances, enabling us to
fully model the ST dependencies by considering the correlations between DEDT.
Further, we devise FC graph convolution with a moving-pooling GNN layer to
effectively capture the ST dependencies for learning effective representations.
Extensive experiments show the effectiveness of FC-STGNN on multiple MTS
datasets compared to SOTA methods.
less
By: Alfredo Reichlin, Giovanni Luca Marchetti, Hang Yin, Anastasiia Varava, Danica Kragic
We address the problem of learning representations from observations of a
scene involving an agent and an external object the agent interacts with. To
this end, we propose a representation learning framework extracting the
location in physical space of both the agent and the object from unstructured
observations of arbitrary nature. Our framework relies on the actions performed
by the agent as the only source of supervision, while assuming ... more
We address the problem of learning representations from observations of a
scene involving an agent and an external object the agent interacts with. To
this end, we propose a representation learning framework extracting the
location in physical space of both the agent and the object from unstructured
observations of arbitrary nature. Our framework relies on the actions performed
by the agent as the only source of supervision, while assuming that the object
is displaced by the agent via unknown dynamics. We provide a theoretical
foundation and formally prove that an ideal learner is guaranteed to infer an
isometric representation, disentangling the agent from the object and correctly
extracting their locations. We evaluate empirically our framework on a variety
of scenarios, showing that it outperforms vision-based approaches such as a
state-of-the-art keypoint extractor. We moreover demonstrate how the extracted
representations enable the agent to solve downstream tasks via reinforcement
learning in an efficient manner.
less
Career Path Recommendations for Long-term Income Maximization: A Reinforcement Learning Approach
0upvotes
By: Spyros Avlonitis, Dor Lavi, Masoud Mansoury, David Graus
This study explores the potential of reinforcement learning algorithms to
enhance career planning processes. Leveraging data from Randstad The
Netherlands, the study simulates the Dutch job market and develops strategies
to optimize employees' long-term income. By formulating career planning as a
Markov Decision Process (MDP) and utilizing machine learning algorithms such as
Sarsa, Q-Learning, and A2C, we learn optimal policies that recomme... more
This study explores the potential of reinforcement learning algorithms to
enhance career planning processes. Leveraging data from Randstad The
Netherlands, the study simulates the Dutch job market and develops strategies
to optimize employees' long-term income. By formulating career planning as a
Markov Decision Process (MDP) and utilizing machine learning algorithms such as
Sarsa, Q-Learning, and A2C, we learn optimal policies that recommend career
paths with high-income occupations and industries. The results demonstrate
significant improvements in employees' income trajectories, with RL models,
particularly Q-Learning and Sarsa, achieving an average increase of 5% compared
to observed career paths. The study acknowledges limitations, including narrow
job filtering, simplifications in the environment formulation, and assumptions
regarding employment continuity and zero application costs. Future research can
explore additional objectives beyond income optimization and address these
limitations to further enhance career planning processes.
less
By: Antoine Choffrut, Rachid Guerraoui, Rafael Pinot, Renaud Sirdey, John Stephan, Martin Zuber
Due to the large-scale availability of data, machine learning (ML) algorithms
are being deployed in distributed topologies, where different nodes collaborate
to train ML models over their individual data by exchanging model-related
information (e.g., gradients) with a central server. However, distributed
learning schemes are notably vulnerable to two threats. First, Byzantine nodes
can single-handedly corrupt the learning by sending incorre... more
Due to the large-scale availability of data, machine learning (ML) algorithms
are being deployed in distributed topologies, where different nodes collaborate
to train ML models over their individual data by exchanging model-related
information (e.g., gradients) with a central server. However, distributed
learning schemes are notably vulnerable to two threats. First, Byzantine nodes
can single-handedly corrupt the learning by sending incorrect information to
the server, e.g., erroneous gradients. The standard approach to mitigate such
behavior is to use a non-linear robust aggregation method at the server.
Second, the server can violate the privacy of the nodes. Recent attacks have
shown that exchanging (unencrypted) gradients enables a curious server to
recover the totality of the nodes' data. The use of homomorphic encryption
(HE), a gold standard security primitive, has extensively been studied as a
privacy-preserving solution to distributed learning in non-Byzantine scenarios.
However, due to HE's large computational demand especially for high-dimensional
ML models, there has not yet been any attempt to design purely homomorphic
operators for non-linear robust aggregators. In this work, we present SABLE,
the first completely homomorphic and Byzantine robust distributed learning
algorithm. SABLE essentially relies on a novel plaintext encoding method that
enables us to implement the robust aggregator over batching-friendly BGV.
Moreover, this encoding scheme also accelerates state-of-the-art homomorphic
sorting with larger security margins and smaller ciphertext size. We perform
extensive experiments on image classification tasks and show that our algorithm
achieves practical execution times while matching the ML performance of its
non-private counterpart.
less
By: Nat Wannawas, A. Aldo Faisal
Reinforcement learning of real-world tasks is very data inefficient, and
extensive simulation-based modelling has become the dominant approach for
training systems. However, in human-robot interaction and many other real-world
settings, there is no appropriate one-model-for-all due to differences in
individual instances of the system (e.g. different people) or necessary
oversimplifications in the simulation models. This requires two approac... more
Reinforcement learning of real-world tasks is very data inefficient, and
extensive simulation-based modelling has become the dominant approach for
training systems. However, in human-robot interaction and many other real-world
settings, there is no appropriate one-model-for-all due to differences in
individual instances of the system (e.g. different people) or necessary
oversimplifications in the simulation models. This requires two approaches: 1.
either learning the individual system's dynamics approximately from data which
requires data-intensive training or 2. using a complete digital twin of the
instances, which may not be realisable in many cases. We introduce two
approaches: co-kriging adjustments (CKA) and ridge regression adjustment (RRA)
as novel ways to combine the advantages of both approaches. Our adjustment
methods are based on an auto-regressive AR1 co-kriging model that we integrate
with GP priors. This yield a data- and simulation-efficient way of using
simplistic simulation models (e.g., simple two-link model) and rapidly adapting
them to individual instances (e.g., biomechanics of individual people). Using
CKA and RRA, we obtain more accurate uncertainty quantification of the entire
system's dynamics than pure GP-based and AR1 methods. We demonstrate the
efficiency of co-kriging adjustment with an interpretable reinforcement
learning control example, learning to control a biomechanical human arm using
only a two-link arm simulation model (offline part) and CKA derived from a
small amount of interaction data (on-the-fly online). Our method unlocks an
efficient and uncertainty-aware way to implement reinforcement learning methods
in real world complex systems for which only imperfect simulation models exist.
less
By: Haohui Lu, Shahadat Uddin
Link prediction is a key aspect of graph machine learning, with applications
as diverse as disease prediction, social network recommendations, and drug
discovery. It involves predicting new links that may form between network
nodes. Despite the clear importance of link prediction, existing models have
significant shortcomings. Graph Convolutional Networks, for instance, have been
proven to be highly efficient for link prediction on a variet... more
Link prediction is a key aspect of graph machine learning, with applications
as diverse as disease prediction, social network recommendations, and drug
discovery. It involves predicting new links that may form between network
nodes. Despite the clear importance of link prediction, existing models have
significant shortcomings. Graph Convolutional Networks, for instance, have been
proven to be highly efficient for link prediction on a variety of datasets.
However, they encounter severe limitations when applied to short-path networks
and ego networks, resulting in poor performance. This presents a critical
problem space that this work aims to address. In this paper, we present the
Node Centrality and Similarity Based Parameterised Model (NCSM), a novel method
for link prediction tasks. NCSM uniquely integrates node centrality and
similarity measures as edge features in a customised Graph Neural Network (GNN)
layer, effectively leveraging the topological information of large networks.
This model represents the first parameterised GNN-based link prediction model
that considers topological information. The proposed model was evaluated on
five benchmark graph datasets, each comprising thousands of nodes and edges.
Experimental results highlight NCSM's superiority over existing
state-of-the-art models like Graph Convolutional Networks and Variational Graph
Autoencoder, as it outperforms them across various metrics and datasets. This
exceptional performance can be attributed to NCSM's innovative integration of
node centrality, similarity measures, and its efficient use of topological
information.
less
By: Frederiek Wesel, Kim Batselier
In the context of kernel machines, polynomial and Fourier features are
commonly used to provide a nonlinear extension to linear models by mapping the
data to a higher-dimensional space. Unless one considers the dual formulation
of the learning problem, which renders exact large-scale learning unfeasible,
the exponential increase of model parameters in the dimensionality of the data
caused by their tensor-product structure prohibits to tackl... more
In the context of kernel machines, polynomial and Fourier features are
commonly used to provide a nonlinear extension to linear models by mapping the
data to a higher-dimensional space. Unless one considers the dual formulation
of the learning problem, which renders exact large-scale learning unfeasible,
the exponential increase of model parameters in the dimensionality of the data
caused by their tensor-product structure prohibits to tackle high-dimensional
problems. One of the possible approaches to circumvent this exponential scaling
is to exploit the tensor structure present in the features by constraining the
model weights to be an underparametrized tensor network. In this paper we
quantize, i.e. further tensorize, polynomial and Fourier features. Based on
this feature quantization we propose to quantize the associated model weights,
yielding quantized models. We show that, for the same number of model
parameters, the resulting quantized models have a higher bound on the
VC-dimension as opposed to their non-quantized counterparts, at no additional
computational cost while learning from identical features. We verify
experimentally how this additional tensorization regularizes the learning
problem by prioritizing the most salient features in the data and how it
provides models with increased generalization capabilities. We finally
benchmark our approach on large regression task, achieving state-of-the-art
results on a laptop computer.
less
By: Tim Bakker, Herke van Hoof, Max Welling
Pool-based active learning (AL) is a promising technology for increasing
data-efficiency of machine learning models. However, surveys show that
performance of recent AL methods is very sensitive to the choice of dataset and
training setting, making them unsuitable for general application. In order to
tackle this problem, the field Learning Active Learning (LAL) suggests to learn
the active learning strategy itself, allowing it to adapt to t... more
Pool-based active learning (AL) is a promising technology for increasing
data-efficiency of machine learning models. However, surveys show that
performance of recent AL methods is very sensitive to the choice of dataset and
training setting, making them unsuitable for general application. In order to
tackle this problem, the field Learning Active Learning (LAL) suggests to learn
the active learning strategy itself, allowing it to adapt to the given setting.
In this work, we propose a novel LAL method for classification that exploits
symmetry and independence properties of the active learning problem with an
Attentive Conditional Neural Process model. Our approach is based on learning
from a myopic oracle, which gives our model the ability to adapt to
non-standard objectives, such as those that do not equally weight the error on
all data points. We experimentally verify that our Neural Process model
outperforms a variety of baselines in these settings. Finally, our experiments
show that our model exhibits a tendency towards improved stability to changing
datasets. However, performance is sensitive to choice of classifier and more
work is necessary to reduce the performance the gap with the myopic oracle and
to improve scalability. We present our work as a proof-of-concept for LAL on
nonstandard objectives and hope our analysis and modelling considerations
inspire future LAL work.
less
By: Zebang Shen, Jiayuan Ye, Anmin Kang, Hamed Hassani, Reza Shokri
Repeated parameter sharing in federated learning causes significant
information leakage about private data, thus defeating its main purpose: data
privacy. Mitigating the risk of this information leakage, using state of the
art differentially private algorithms, also does not come for free. Randomized
mechanisms can prevent convergence of models on learning even the useful
representation functions, especially if there is more disagreement be... more
Repeated parameter sharing in federated learning causes significant
information leakage about private data, thus defeating its main purpose: data
privacy. Mitigating the risk of this information leakage, using state of the
art differentially private algorithms, also does not come for free. Randomized
mechanisms can prevent convergence of models on learning even the useful
representation functions, especially if there is more disagreement between
local models on the classification functions (due to data heterogeneity). In
this paper, we consider a representation federated learning objective that
encourages various parties to collaboratively refine the consensus part of the
model, with differential privacy guarantees, while separately allowing
sufficient freedom for local personalization (without releasing it). We prove
that in the linear representation setting, while the objective is non-convex,
our proposed new algorithm \DPFEDREP\ converges to a ball centered around the
\emph{global optimal} solution at a linear rate, and the radius of the ball is
proportional to the reciprocal of the privacy budget. With this novel utility
analysis, we improve the SOTA utility-privacy trade-off for this problem by a
factor of $\sqrt{d}$, where $d$ is the input dimension. We empirically evaluate
our method with the image classification task on CIFAR10, CIFAR100, and EMNIST,
and observe a significant performance improvement over the prior work under the
same small privacy budget. The code can be found in this link:
https://github.com/shenzebang/CENTAUR-Privacy-Federated-Representation-Learning.
less
By: Marin Vlastelica, Sebastian Blaes, Cristina Pineri, Georg Martius
We introduce a simple but effective method for managing risk in model-based
reinforcement learning with trajectory sampling that involves probabilistic
safety constraints and balancing of optimism in the face of epistemic
uncertainty and pessimism in the face of aleatoric uncertainty of an ensemble
of stochastic neural networks.Various experiments indicate that the separation
of uncertainties is essential to performing well with data-driven... more
We introduce a simple but effective method for managing risk in model-based
reinforcement learning with trajectory sampling that involves probabilistic
safety constraints and balancing of optimism in the face of epistemic
uncertainty and pessimism in the face of aleatoric uncertainty of an ensemble
of stochastic neural networks.Various experiments indicate that the separation
of uncertainties is essential to performing well with data-driven MPC
approaches in uncertain and safety-critical control environments.
less
By: Konstantinos Pitas, Julyan Arbel
We conduct a detailed investigation of tempered posteriors and uncover a
number of crucial and previously undiscussed points. Contrary to previous
results, we first show that for realistic models and datasets and the tightly
controlled case of the Laplace approximation to the posterior, stochasticity
does not in general improve test accuracy. The coldest temperature is often
optimal. One might think that Bayesian models with some stochastic... more
We conduct a detailed investigation of tempered posteriors and uncover a
number of crucial and previously undiscussed points. Contrary to previous
results, we first show that for realistic models and datasets and the tightly
controlled case of the Laplace approximation to the posterior, stochasticity
does not in general improve test accuracy. The coldest temperature is often
optimal. One might think that Bayesian models with some stochasticity can at
least obtain improvements in terms of calibration. However, we show empirically
that when gains are obtained this comes at the cost of degradation in test
accuracy. We then discuss how targeting Frequentist metrics using Bayesian
models provides a simple explanation of the need for a temperature parameter
$\lambda$ in the optimization objective. Contrary to prior works, we finally
show through a PAC-Bayesian analysis that the temperature $\lambda$ cannot be
seen as simply fixing a misspecified prior or likelihood.
less
By: Karlis Freivalds, Emils Ozolins, Guntis Barzdins
Integer factorization is a famous computational problem unknown whether being
solvable in the polynomial time. With the rise of deep neural networks, it is
interesting whether they can facilitate faster factorization. We present an
approach to factorization utilizing deep neural networks and discrete denoising
diffusion that works by iteratively correcting errors in a partially-correct
solution. To this end, we develop a new seq2seq neural ... more
Integer factorization is a famous computational problem unknown whether being
solvable in the polynomial time. With the rise of deep neural networks, it is
interesting whether they can facilitate faster factorization. We present an
approach to factorization utilizing deep neural networks and discrete denoising
diffusion that works by iteratively correcting errors in a partially-correct
solution. To this end, we develop a new seq2seq neural network architecture,
employ relaxed categorical distribution and adapt the reverse diffusion process
to cope better with inaccuracies in the denoising step. The approach is able to
find factors for integers of up to 56 bits long. Our analysis indicates that
investment in training leads to an exponential decrease of sampling steps
required at inference to achieve a given success rate, thus counteracting an
exponential run-time increase depending on the bit-length.
less
By: Yinghao Zhu, Zixiang Wang, Long He, Shiyun Xie, Zixi Chen, Jingkun An, Liantao Ma, Chengwei Pan
Electronic Health Record (EHR) data frequently exhibits sparse
characteristics, posing challenges for predictive modeling. Current direct
imputation such as matrix imputation approaches hinge on referencing analogous
rows or columns to complete raw missing data and do not differentiate between
imputed and actual values. As a result, models may inadvertently incorporate
irrelevant or deceptive information with respect to the prediction objec... more
Electronic Health Record (EHR) data frequently exhibits sparse
characteristics, posing challenges for predictive modeling. Current direct
imputation such as matrix imputation approaches hinge on referencing analogous
rows or columns to complete raw missing data and do not differentiate between
imputed and actual values. As a result, models may inadvertently incorporate
irrelevant or deceptive information with respect to the prediction objective,
thereby compromising the efficacy of downstream performance. While some methods
strive to recalibrate or augment EHR embeddings after direct imputation, they
often mistakenly prioritize imputed features. This misprioritization can
introduce biases or inaccuracies into the model. To tackle these issues, our
work resorts to indirect imputation, where we leverage prototype
representations from similar patients to obtain a denser embedding. Recognizing
the limitation that missing features are typically treated the same as present
ones when measuring similar patients, our approach designs a feature confidence
learner module. This module is sensitive to the missing feature status,
enabling the model to better judge the reliability of each feature. Moreover,
we propose a novel patient similarity metric that takes feature confidence into
account, ensuring that evaluations are not based merely on potentially
inaccurate imputed values. Consequently, our work captures dense prototype
patient representations with feature-missing-aware calibration process.
Comprehensive experiments demonstrate that designed model surpasses established
EHR-focused models with a statistically significant improvement on MIMIC-III
and MIMIC-IV datasets in-hospital mortality outcome prediction task. The code
is publicly available at \url{https://anonymous.4open.science/r/SparseEHR} to
assure the reproducibility.
less
By: Edward A. Small, Jeffrey N. Clark, Christopher J. McWilliams, Kacper Sokol, Jeffrey Chan, Flora D. Salim, Raul Santos-Rodriguez
Counterfactuals operationalised through algorithmic recourse have become a
powerful tool to make artificial intelligence systems explainable.
Conceptually, given an individual classified as y -- the factual -- we seek
actions such that their prediction becomes the desired class y' -- the
counterfactual. This process offers algorithmic recourse that is (1) easy to
customise and interpret, and (2) directly aligned with the goals of each
indiv... more
Counterfactuals operationalised through algorithmic recourse have become a
powerful tool to make artificial intelligence systems explainable.
Conceptually, given an individual classified as y -- the factual -- we seek
actions such that their prediction becomes the desired class y' -- the
counterfactual. This process offers algorithmic recourse that is (1) easy to
customise and interpret, and (2) directly aligned with the goals of each
individual. However, the properties of a "good" counterfactual are still
largely debated; it remains an open challenge to effectively locate a
counterfactual along with its corresponding recourse. Some strategies use
gradient-driven methods, but these offer no guarantees on the feasibility of
the recourse and are open to adversarial attacks on carefully created
manifolds. This can lead to unfairness and lack of robustness. Other methods
are data-driven, which mostly addresses the feasibility problem at the expense
of privacy, security and secrecy as they require access to the entire training
data set. Here, we introduce LocalFACE, a model-agnostic technique that
composes feasible and actionable counterfactual explanations using
locally-acquired information at each step of the algorithmic recourse. Our
explainer preserves the privacy of users by only leveraging data that it
specifically requires to construct actionable algorithmic recourse, and
protects the model by offering transparency solely in the regions deemed
necessary for the intervention.
less
Adaptive Distributed Kernel Ridge Regression: A Feasible Distributed Learning Scheme for Data Silos
0upvotes
By: Di Wang, Xiaotong Liu, Shao-Bo Lin, Ding-Xuan Zhou
Data silos, mainly caused by privacy and interoperability, significantly
constrain collaborations among different organizations with similar data for
the same purpose. Distributed learning based on divide-and-conquer provides a
promising way to settle the data silos, but it suffers from several challenges,
including autonomy, privacy guarantees, and the necessity of collaborations.
This paper focuses on developing an adaptive distributed ke... more
Data silos, mainly caused by privacy and interoperability, significantly
constrain collaborations among different organizations with similar data for
the same purpose. Distributed learning based on divide-and-conquer provides a
promising way to settle the data silos, but it suffers from several challenges,
including autonomy, privacy guarantees, and the necessity of collaborations.
This paper focuses on developing an adaptive distributed kernel ridge
regression (AdaDKRR) by taking autonomy in parameter selection, privacy in
communicating non-sensitive information, and the necessity of collaborations in
performance improvement into account. We provide both solid theoretical
verification and comprehensive experiments for AdaDKRR to demonstrate its
feasibility and effectiveness. Theoretically, we prove that under some mild
conditions, AdaDKRR performs similarly to running the optimal learning
algorithms on the whole data, verifying the necessity of collaborations and
showing that no other distributed learning scheme can essentially beat AdaDKRR
under the same conditions. Numerically, we test AdaDKRR on both toy simulations
and two real-world applications to show that AdaDKRR is superior to other
existing distributed learning schemes. All these results show that AdaDKRR is a
feasible scheme to defend against data silos, which are highly desired in
numerous application regions such as intelligent decision-making, pricing
forecasting, and performance prediction for products.
less
By: Arian Prabowo, Kaixuan Chen, Hao Xue, Subbu Sethuvenkatraman, Flora D. Salim
In traditional deep learning algorithms, one of the key assumptions is that
the data distribution remains constant during both training and deployment.
However, this assumption becomes problematic when faced with
Out-of-Distribution periods, such as the COVID-19 lockdowns, where the data
distribution significantly deviates from what the model has seen during
training. This paper employs a two-fold strategy: utilizing continual learning
tech... more
In traditional deep learning algorithms, one of the key assumptions is that
the data distribution remains constant during both training and deployment.
However, this assumption becomes problematic when faced with
Out-of-Distribution periods, such as the COVID-19 lockdowns, where the data
distribution significantly deviates from what the model has seen during
training. This paper employs a two-fold strategy: utilizing continual learning
techniques to update models with new data and harnessing human mobility data
collected from privacy-preserving pedestrian counters located outside
buildings. In contrast to online learning, which suffers from 'catastrophic
forgetting' as newly acquired knowledge often erases prior information,
continual learning offers a holistic approach by preserving past insights while
integrating new data. This research applies FSNet, a powerful continual
learning algorithm, to real-world data from 13 building complexes in Melbourne,
Australia, a city which had the second longest total lockdown duration globally
during the pandemic. Results underscore the crucial role of continual learning
in accurate energy forecasting, particularly during Out-of-Distribution
periods. Secondary data such as mobility and temperature provided ancillary
support to the primary forecasting model. More importantly, while traditional
methods struggled to adapt during lockdowns, models featuring at least online
learning demonstrated resilience, with lockdown periods posing fewer challenges
once armed with adaptive learning techniques. This study contributes valuable
methodologies and insights to the ongoing effort to improve energy load
forecasting during future Out-of-Distribution periods.
less
By: Christos Chrysanthos Nikolaidis, Vasileios Perifanis, Nikolaos Pavlidis, Pavlos S. Efraimidis
The provision of social care applications is crucial for elderly people to
improve their quality of life and enables operators to provide early
interventions. Accurate predictions of user dropouts in healthy ageing
applications are essential since they are directly related to individual health
statuses. Machine Learning (ML) algorithms have enabled highly accurate
predictions, outperforming traditional statistical methods that struggle to
c... more
The provision of social care applications is crucial for elderly people to
improve their quality of life and enables operators to provide early
interventions. Accurate predictions of user dropouts in healthy ageing
applications are essential since they are directly related to individual health
statuses. Machine Learning (ML) algorithms have enabled highly accurate
predictions, outperforming traditional statistical methods that struggle to
cope with individual patterns. However, ML requires a substantial amount of
data for training, which is challenging due to the presence of personal
identifiable information (PII) and the fragmentation posed by regulations. In
this paper, we present a federated machine learning (FML) approach that
minimizes privacy concerns and enables distributed training, without
transferring individual data. We employ collaborative training by considering
individuals and organizations under FML, which models both cross-device and
cross-silo learning scenarios. Our approach is evaluated on a real-world
dataset with non-independent and identically distributed (non-iid) data among
clients, class imbalance and label ambiguity. Our results show that data
selection and class imbalance handling techniques significantly improve the
predictive accuracy of models trained under FML, demonstrating comparable or
superior predictive performance than traditional ML models.
less
By: Sjoerd de Vries, Dirk Thierens
Most real-world classification tasks suffer from label noise to some extent.
Such noise in the data adversely affects the generalization error of learned
models and complicates the evaluation of noise-handling methods, as their
performance cannot be accurately measured without clean labels. In label noise
research, typically either noisy or incomplex simulated data are accepted as a
baseline, into which additional noise with known propertie... more
Most real-world classification tasks suffer from label noise to some extent.
Such noise in the data adversely affects the generalization error of learned
models and complicates the evaluation of noise-handling methods, as their
performance cannot be accurately measured without clean labels. In label noise
research, typically either noisy or incomplex simulated data are accepted as a
baseline, into which additional noise with known properties is injected. In
this paper, we propose SYNLABEL, a framework that aims to improve upon the
aforementioned methodologies. It allows for creating a noiseless dataset
informed by real data, by either pre-specifying or learning a function and
defining it as the ground truth function from which labels are generated.
Furthermore, by resampling a number of values for selected features in the
function domain, evaluating the function and aggregating the resulting labels,
each data point can be assigned a soft label or label distribution. Such
distributions allow for direct injection and quantification of label noise. The
generated datasets serve as a clean baseline of adjustable complexity into
which different types of noise may be introduced. We illustrate how the
framework can be applied, how it enables quantification of label noise and how
it improves over existing methodologies.
less
By: Jonathan D. Boyd, Joshua H. Tyler, Anthony M. Murphy, Donald R. Reising
As power quality becomes a higher priority in the electric utility industry,
the amount of disturbance event data continues to grow. Utilities do not have
the required personnel to analyze each event by hand. This work presents an
automated approach for analyzing power quality events recorded by digital fault
recorders and power quality monitors operating within a power transmission
system. The automated approach leverages rule-based analyt... more
As power quality becomes a higher priority in the electric utility industry,
the amount of disturbance event data continues to grow. Utilities do not have
the required personnel to analyze each event by hand. This work presents an
automated approach for analyzing power quality events recorded by digital fault
recorders and power quality monitors operating within a power transmission
system. The automated approach leverages rule-based analytics to examine the
time and frequency domain characteristics of the voltage and current signals.
Customizable thresholds are set to categorize each disturbance event. The
events analyzed within this work include various faults, motor starting, and
incipient instrument transformer failure. Analytics for fourteen different
event types have been developed. The analytics were tested on 160 signal files
and yielded an accuracy of ninety-nine percent. Continuous, nominal signal data
analysis is performed using an approach coined as the cyclic histogram. The
cyclic histogram process will be integrated into the digital fault recorders
themselves to facilitate the detection of subtle signal variations that are too
small to trigger a disturbance event and that can occur over hours or days. In
addition to reducing memory requirements by a factor of 320, it is anticipated
that cyclic histogram processing will aid in identifying incipient events and
identifiers. This project is expected to save engineers time by automating the
classification of disturbance events and increase the reliability of the
transmission system by providing near real time detection and identification of
disturbances as well as prevention of problems before they occur.
less
By: Mahsa Bazzaz, Seth Cooper
Determining the completability of levels generated by procedural generators
such as machine learning models can be challenging, as it can involve the use
of solver agents that often require a significant amount of time to analyze and
solve levels. Active learning is not yet widely adopted in game evaluations,
although it has been used successfully in natural language processing, image
and speech recognition, and computer vision, where the a... more
Determining the completability of levels generated by procedural generators
such as machine learning models can be challenging, as it can involve the use
of solver agents that often require a significant amount of time to analyze and
solve levels. Active learning is not yet widely adopted in game evaluations,
although it has been used successfully in natural language processing, image
and speech recognition, and computer vision, where the availability of labeled
data is limited or expensive. In this paper, we propose the use of active
learning for learning level completability classification. Through an active
learning approach, we train deep-learning models to classify the completability
of generated levels for Super Mario Bros., Kid Icarus, and a Zelda-like game.
We compare active learning for querying levels to label with completability
against random queries. Our results show using an active learning approach to
label levels results in better classifier performance with the same amount of
labeled data.
less
By: Fredrik Hellström, Giuseppe Durisi, Benjamin Guedj, Maxim Raginsky
A fundamental question in theoretical machine learning is generalization.
Over the past decades, the PAC-Bayesian approach has been established as a
flexible framework to address the generalization capabilities of machine
learning algorithms, and design new ones. Recently, it has garnered increased
interest due to its potential applicability for a variety of learning
algorithms, including deep neural networks. In parallel, an
information-th... more
A fundamental question in theoretical machine learning is generalization.
Over the past decades, the PAC-Bayesian approach has been established as a
flexible framework to address the generalization capabilities of machine
learning algorithms, and design new ones. Recently, it has garnered increased
interest due to its potential applicability for a variety of learning
algorithms, including deep neural networks. In parallel, an
information-theoretic view of generalization has developed, wherein the
relation between generalization and various information measures has been
established. This framework is intimately connected to the PAC-Bayesian
approach, and a number of results have been independently discovered in both
strands. In this monograph, we highlight this strong connection and present a
unified treatment of generalization. We present techniques and results that the
two perspectives have in common, and discuss the approaches and interpretations
that differ. In particular, we demonstrate how many proofs in the area share a
modular structure, through which the underlying ideas can be intuited. We pay
special attention to the conditional mutual information (CMI) framework;
analytical studies of the information complexity of learning algorithms; and
the application of the proposed methods to deep learning. This monograph is
intended to provide a comprehensive introduction to information-theoretic
generalization bounds and their connection to PAC-Bayes, serving as a
foundation from which the most recent developments are accessible. It is aimed
broadly towards researchers with an interest in generalization and theoretical
machine learning.
less
By: Sofiane Ouaari, Ali Burak Ünal, Mete Akgün, Nico Pfeifer
Several domains increasingly rely on machine learning in their applications.
The resulting heavy dependence on data has led to the emergence of various laws
and regulations around data ethics and privacy and growing awareness of the
need for privacy-preserving machine learning (ppML). Current ppML techniques
utilize methods that are either purely based on cryptography, such as
homomorphic encryption, or that introduce noise into the input, ... more
Several domains increasingly rely on machine learning in their applications.
The resulting heavy dependence on data has led to the emergence of various laws
and regulations around data ethics and privacy and growing awareness of the
need for privacy-preserving machine learning (ppML). Current ppML techniques
utilize methods that are either purely based on cryptography, such as
homomorphic encryption, or that introduce noise into the input, such as
differential privacy. The main criticism given to those techniques is the fact
that they either are too slow or they trade off a model s performance for
improved confidentiality. To address this performance reduction, we aim to
leverage robust representation learning as a way of encoding our data while
optimizing the privacy-utility trade-off. Our method centers on training
autoencoders in a multi-objective manner and then concatenating the latent and
learned features from the encoding part as the encoded form of our data. Such a
deep learning-powered encoding can then safely be sent to a third party for
intensive training and hyperparameter tuning. With our proposed framework, we
can share our data and use third party tools without being under the threat of
revealing its original form. We empirically validate our results on unimodal
and multimodal settings, the latter following a vertical splitting system and
show improved performance over state-of-the-art.
less
By: Keegan Kelly, Lorena Piedras, Sukrit Rao, David Roth
Normalizing Flows (NFs) describe a class of models that express a complex
target distribution as the composition of a series of bijective transformations
over a simpler base distribution. By limiting the space of candidate
transformations to diffeomorphisms, NFs enjoy efficient, exact sampling and
density evaluation, enabling NFs to flexibly behave as both discriminative and
generative models. Their restriction to diffeomorphisms, however, ... more
Normalizing Flows (NFs) describe a class of models that express a complex
target distribution as the composition of a series of bijective transformations
over a simpler base distribution. By limiting the space of candidate
transformations to diffeomorphisms, NFs enjoy efficient, exact sampling and
density evaluation, enabling NFs to flexibly behave as both discriminative and
generative models. Their restriction to diffeomorphisms, however, enforces that
input, output and all intermediary spaces share the same dimension, limiting
their ability to effectively represent target distributions with complex
topologies. Additionally, in cases where the prior and target distributions are
not homeomorphic, Normalizing Flows can leak mass outside of the support of the
target. This survey covers a selection of recent works that combine aspects of
other generative model classes, such as VAEs and score-based diffusion, and in
doing so loosen the strict bijectivity constraints of NFs to achieve a balance
of expressivity, training speed, sample efficiency and likelihood tractability.
less
By: David Yunis, Justin Jung, Falcon Dai, Matthew Walter
Exploration in sparse-reward reinforcement learning is difficult due to the
requirement of long, coordinated sequences of actions in order to achieve any
reward. Moreover, in continuous action spaces there are an infinite number of
possible actions, which only increases the difficulty of exploration. One class
of methods designed to address these issues forms temporally extended actions,
often called skills, from interaction data collected ... more
Exploration in sparse-reward reinforcement learning is difficult due to the
requirement of long, coordinated sequences of actions in order to achieve any
reward. Moreover, in continuous action spaces there are an infinite number of
possible actions, which only increases the difficulty of exploration. One class
of methods designed to address these issues forms temporally extended actions,
often called skills, from interaction data collected in the same domain, and
optimizes a policy on top of this new action space. Typically such methods
require a lengthy pretraining phase, especially in continuous action spaces, in
order to form the skills before reinforcement learning can begin. Given prior
evidence that the full range of the continuous action space is not required in
such tasks, we propose a novel approach to skill-generation with two
components. First we discretize the action space through clustering, and second
we leverage a tokenization technique borrowed from natural language processing
to generate temporally extended actions. Such a method outperforms baselines
for skill-generation in several challenging sparse-reward domains, and requires
orders-of-magnitude less computation in skill-generation and online rollouts.
less
By: Lydia T. Liu, Solon Barocas, Jon Kleinberg, Karen Levy
Predicting future outcomes is a prevalent application of machine learning in
social impact domains. Examples range from predicting student success in
education to predicting disease risk in healthcare. Practitioners recognize
that the ultimate goal is not just to predict but to act effectively.
Increasing evidence suggests that relying on outcome predictions for downstream
interventions may not have desired results.
In most domains there ... more
Predicting future outcomes is a prevalent application of machine learning in
social impact domains. Examples range from predicting student success in
education to predicting disease risk in healthcare. Practitioners recognize
that the ultimate goal is not just to predict but to act effectively.
Increasing evidence suggests that relying on outcome predictions for downstream
interventions may not have desired results.
In most domains there exists a multitude of possible interventions for each
individual, making the challenge of taking effective action more acute. Even
when causal mechanisms connecting the individual's latent states to outcomes is
well understood, in any given instance (a specific student or patient),
practitioners still need to infer -- from budgeted measurements of latent
states -- which of many possible interventions will be most effective for this
individual. With this in mind, we ask: when are accurate predictors of outcomes
helpful for identifying the most suitable intervention?
Through a simple model encompassing actions, latent states, and measurements,
we demonstrate that pure outcome prediction rarely results in the most
effective policy for taking actions, even when combined with other
measurements. We find that except in cases where there is a single decisive
action for improving the outcome, outcome prediction never maximizes "action
value", the utility of taking actions. Making measurements of actionable latent
states, where specific actions lead to desired outcomes, considerably enhances
the action value compared to outcome prediction, and the degree of improvement
depends on action costs and the outcome model. This analysis emphasizes the
need to go beyond generic outcome prediction in interventional settings by
incorporating knowledge of plausible actions and latent states.
less
By: Xuyang Zhong, Chen Liu
Dataset distillation methods have demonstrated remarkable performance for
neural networks trained with very limited training data. However, a significant
challenge arises in the form of architecture overfitting: the distilled
training data synthesized by a specific network architecture (i.e., training
network) generates poor performance when trained by other network architectures
(i.e., test networks). This paper addresses this issue and pr... more
Dataset distillation methods have demonstrated remarkable performance for
neural networks trained with very limited training data. However, a significant
challenge arises in the form of architecture overfitting: the distilled
training data synthesized by a specific network architecture (i.e., training
network) generates poor performance when trained by other network architectures
(i.e., test networks). This paper addresses this issue and proposes a series of
approaches in both architecture designs and training schemes which can be
adopted together to boost the generalization performance across different
network architectures on the distilled training data. We conduct extensive
experiments to demonstrate the effectiveness and generality of our methods.
Particularly, across various scenarios involving different sizes of distilled
data, our approaches achieve comparable or superior performance to existing
methods when training on the distilled data using networks with larger
capacities.
less
By: Fu-Yao Ko, Katsuyuki Suzuki, Kazuo Yonekura
Sizing optimization of truss structures is a complex computational problem,
and the reinforcement learning (RL) is suitable for dealing with multimodal
problems without gradient computations. In this paper, a new efficient
optimization algorithm called update Monte Carlo tree search (UMCTS) is
developed to obtain the appropriate design for truss structures. UMCTS is an
RL-based method that combines the novel update process and Monte Carlo t... more
Sizing optimization of truss structures is a complex computational problem,
and the reinforcement learning (RL) is suitable for dealing with multimodal
problems without gradient computations. In this paper, a new efficient
optimization algorithm called update Monte Carlo tree search (UMCTS) is
developed to obtain the appropriate design for truss structures. UMCTS is an
RL-based method that combines the novel update process and Monte Carlo tree
search (MCTS) with the upper confidence bound (UCB). Update process means that
in each round, the optimal cross-sectional area of each member is determined by
search tree, and its initial state is the final state in the previous round. In
the UMCTS algorithm, an accelerator for the number of selections for member
area and iteration number is introduced to reduce the computation time.
Moreover, for each state, the average reward is replaced by the best reward
collected on the simulation process to determine the optimal solution. The
proposed optimization method is examined on some benchmark problems of planar
and spatial trusses with discrete sizing variables to demonstrate the
efficiency and validity. It is shown that the computation time for the proposed
approach is at least ten times faster than the branch and bound (BB) method.
The numerical results indicate that the proposed method stably achieves better
solution than other conventional methods.
less
By: Sungwoo Lee, Younghyun Oh, Hyunhoe An, Hyebhin Yoon, Karl J. Friston, Seok Jun Hong, Choong-Wan Woo
Building autonomous --- i.e., choosing goals based on one's needs -- and
adaptive -- i.e., surviving in ever-changing environments -- agents has been a
holy grail of artificial intelligence (AI). A living organism is a prime
example of such an agent, offering important lessons about adaptive autonomy.
Here, we focus on interoception, a process of monitoring one's internal
environment to keep it within certain bounds, which underwrites the s... more
Building autonomous --- i.e., choosing goals based on one's needs -- and
adaptive -- i.e., surviving in ever-changing environments -- agents has been a
holy grail of artificial intelligence (AI). A living organism is a prime
example of such an agent, offering important lessons about adaptive autonomy.
Here, we focus on interoception, a process of monitoring one's internal
environment to keep it within certain bounds, which underwrites the survival of
an organism. To develop AI with interoception, we need to factorize the state
variables representing internal environments from external environments and
adopt life-inspired mathematical properties of internal environment states.
This paper offers a new perspective on how interoception can help build
autonomous and adaptive agents by integrating the legacy of cybernetics with
recent advances in theories of life, reinforcement learning, and neuroscience.
less
By: Yide Qiu, Shaoxiang Ling, Tong Zhang, Bo Huang, Zhen Cui
Irregular data in real-world are usually organized as heterogeneous graphs
(HGs) consisting of multiple types of nodes and edges. To explore useful
knowledge from real-world data, both the large-scale encyclopedic HG datasets
and corresponding effective learning methods are crucial, but haven't been well
investigated. In this paper, we construct a large-scale HG benchmark dataset
named UniKG from Wikidata to facilitate knowledge mining and ... more
Irregular data in real-world are usually organized as heterogeneous graphs
(HGs) consisting of multiple types of nodes and edges. To explore useful
knowledge from real-world data, both the large-scale encyclopedic HG datasets
and corresponding effective learning methods are crucial, but haven't been well
investigated. In this paper, we construct a large-scale HG benchmark dataset
named UniKG from Wikidata to facilitate knowledge mining and heterogeneous
graph representation learning. Overall, UniKG contains more than 77 million
multi-attribute entities and 2000 diverse association types, which
significantly surpasses the scale of existing HG datasets. To perform effective
learning on the large-scale UniKG, two key measures are taken, including (i)
the semantic alignment strategy for multi-attribute entities, which projects
the feature description of multi-attribute nodes into a common embedding space
to facilitate node aggregation in a large receptive field; (ii) proposing a
novel plug-and-play anisotropy propagation module (APM) to learn effective
multi-hop anisotropy propagation kernels, which extends methods of large-scale
homogeneous graphs to heterogeneous graphs. These two strategies enable
efficient information propagation among a tremendous number of multi-attribute
entities and meantimes adaptively mine multi-attribute association through the
multi-hop aggregation in large-scale HGs. We set up a node classification task
on our UniKG dataset, and evaluate multiple baseline methods which are
constructed by embedding our APM into large-scale homogenous graph learning
methods. Our UniKG dataset and the baseline codes have been released at
https://github.com/Yide-Qiu/UniKG.
less
By: Jean-Baptiste Hervé, Oliver Withington, Marion Hervé, Laurissa Tokarchuk, Christoph Salge
With growing interest in Procedural Content Generation (PCG) it becomes
increasingly important to develop methods and tools for evaluating and
comparing alternative systems. There is a particular lack regarding the
evaluation of generative pipelines, where a set of generative systems work in
series to make iterative changes to an artifact. We introduce a novel method
called Generative Shift for evaluating the impact of individual stages in ... more
With growing interest in Procedural Content Generation (PCG) it becomes
increasingly important to develop methods and tools for evaluating and
comparing alternative systems. There is a particular lack regarding the
evaluation of generative pipelines, where a set of generative systems work in
series to make iterative changes to an artifact. We introduce a novel method
called Generative Shift for evaluating the impact of individual stages in a PCG
pipeline by quantifying the impact that a generative process has when it is
applied to a pre-existing artifact. We explore this technique by applying it to
a very rich dataset of Minecraft game maps produced by a set of alternative
settlement generators developed as part of the Generative Design in Minecraft
Competition (GDMC), all of which are designed to produce appropriate
settlements for a pre-existing map. While this is an early exploration of this
technique we find it to be a promising lens to apply to PCG evaluation, and we
are optimistic about the potential of Generative Shift to be a domain-agnostic
method for evaluating generative pipelines.
less
By: Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, Tat-Seng Chua
While recently Multimodal Large Language Models (MM-LLMs) have made exciting
strides, they mostly fall prey to the limitation of only input-side multimodal
understanding, without the ability to produce content in multiple modalities.
As we humans always perceive the world and communicate with people through
various modalities, developing any-to-any MM-LLMs capable of accepting and
delivering content in any modality becomes essential to huma... more
While recently Multimodal Large Language Models (MM-LLMs) have made exciting
strides, they mostly fall prey to the limitation of only input-side multimodal
understanding, without the ability to produce content in multiple modalities.
As we humans always perceive the world and communicate with people through
various modalities, developing any-to-any MM-LLMs capable of accepting and
delivering content in any modality becomes essential to human-level AI. To fill
the gap, we present an end-to-end general-purpose any-to-any MM-LLM system,
NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion
decoders, enabling NExT-GPT to perceive inputs and generate outputs in
arbitrary combinations of text, images, videos, and audio. By leveraging the
existing well-trained highly-performing encoders and decoders, NExT-GPT is
tuned with only a small amount of parameter (1%) of certain projection layers,
which not only benefits low-cost training and also facilitates convenient
expansion to more potential modalities. Moreover, we introduce a
modality-switching instruction tuning (MosIT) and manually curate a
high-quality dataset for MosIT, based on which NExT-GPT is empowered with
complex cross-modal semantic understanding and content generation. Overall, our
research showcases the promising possibility of building an AI agent capable of
modeling universal modalities, paving the way for more human-like AI research
in the community.
less
By: Cassandra Bird University of Exeter, Daniel Williamson University of Exeter, Sabina Leonelli University of Exeter
Whether and how data scientists, statisticians and modellers should be
accountable for the AI systems they develop remains a controversial and highly
debated topic, especially given the complexity of AI systems and the
difficulties in comparing and synthesising competing claims arising from their
deployment for data analysis. This paper proposes to address this issue by
decreasing the opacity and heightening the accountability of decision m... more
Whether and how data scientists, statisticians and modellers should be
accountable for the AI systems they develop remains a controversial and highly
debated topic, especially given the complexity of AI systems and the
difficulties in comparing and synthesising competing claims arising from their
deployment for data analysis. This paper proposes to address this issue by
decreasing the opacity and heightening the accountability of decision making
using AI systems, through the explicit acknowledgement of the statistical
foundations that underpin their development and the ways in which these dictate
how their results should be interpreted and acted upon by users. In turn, this
enhances (1) the responsiveness of the models to feedback, (2) the quality and
meaning of uncertainty on their outputs and (3) their transparency to
evaluation. To exemplify this approach, we extend Posterior Belief Assessment
to offer a route to belief ownership from complex and competing AI structures.
We argue that this is a significant way to bring ethical considerations into
mathematical reasoning, and to implement ethical AI in statistical practice. We
demonstrate these ideas within the context of competing models used to advise
the UK government on the spread of the Omicron variant of COVID-19 during
December 2021.
less
By: Anna Brandenberger, Cassandra Marcussen, Elchanan Mossel, Madhu Sudan
We analyze Cumulative Knowledge Processes, introduced by Ben-Eliezer,
Mikulincer, Mossel, and Sudan (ITCS 2023), in the setting of "directed acyclic
graphs", i.e., when new units of knowledge may be derived by combining multiple
previous units of knowledge. The main considerations in this model are the role
of errors (when new units may be erroneous) and local checking (where a few
antecedent units of knowledge are checked when a new unit o... more
We analyze Cumulative Knowledge Processes, introduced by Ben-Eliezer,
Mikulincer, Mossel, and Sudan (ITCS 2023), in the setting of "directed acyclic
graphs", i.e., when new units of knowledge may be derived by combining multiple
previous units of knowledge. The main considerations in this model are the role
of errors (when new units may be erroneous) and local checking (where a few
antecedent units of knowledge are checked when a new unit of knowledge is
discovered). The aforementioned work defined this model but only analyzed an
idealized and simplified "tree-like" setting, i.e., a setting where new units
of knowledge only depended directly on one previously generated unit of
knowledge.
The main goal of our work is to understand when the general process is safe,
i.e., when the effect of errors remains under control. We provide some
necessary and some sufficient conditions for safety. As in the earlier work, we
demonstrate that the frequency of checking as well as the depth of the checks
play a crucial role in determining safety. A key new parameter in the current
work is the $\textit{combination factor}$ which is the distribution of the
number of units $M$ of old knowledge that a new unit of knowledge depends on.
Our results indicate that a large combination factor can compensate for a small
depth of checking. The dependency of the safety on the combination factor is
far from trivial. Indeed some of our main results are stated in terms of
$\mathbb{E}\{1/M\}$ while others depend on $\mathbb{E}\{M\}$.
less
By: Edmund R. Hunt, Chris Baber, Mehdi Sobhani, Sanja Milivojevic, Sagir Yusuf, Mirco Musolesi, Patrick Waterson, Sally Maynard
Defining and measuring trust in dynamic, multiagent teams is important in a
range of contexts, particularly in defense and security domains. Team members
should be trusted to work towards agreed goals and in accordance with shared
values. In this paper, our concern is with the definition of goals and values
such that it is possible to define 'trust' in a way that is interpretable, and
hence usable, by both humans and robots. We argue that t... more
Defining and measuring trust in dynamic, multiagent teams is important in a
range of contexts, particularly in defense and security domains. Team members
should be trusted to work towards agreed goals and in accordance with shared
values. In this paper, our concern is with the definition of goals and values
such that it is possible to define 'trust' in a way that is interpretable, and
hence usable, by both humans and robots. We argue that the outcome of team
activity can be considered in terms of 'goal', 'individual/team values', and
'legal principles'. We question whether alignment is possible at the level of
'individual/team values', or only at the 'goal' and 'legal principles' levels.
We argue for a set of metrics to define trust in human-robot teams that are
interpretable by human or robot team members, and consider an experiment that
could demonstrate the notion of 'satisficing trust' over the course of a
simulated mission.
less
By: Chengwu Liu, Jianhao Shen, Huajian Xin, Zhengying Liu, Ye Yuan, Haiming Wang, Wei Ju, Chuanyang Zheng, Yichun Yin, Lin Li, Ming Zhang, Qun Liu
We present FIMO, an innovative dataset comprising formal mathematical problem
statements sourced from the International Mathematical Olympiad (IMO)
Shortlisted Problems. Designed to facilitate advanced automated theorem proving
at the IMO level, FIMO is currently tailored for the Lean formal language. It
comprises 149 formal problem statements, accompanied by both informal problem
descriptions and their corresponding LaTeX-based informal pr... more
We present FIMO, an innovative dataset comprising formal mathematical problem
statements sourced from the International Mathematical Olympiad (IMO)
Shortlisted Problems. Designed to facilitate advanced automated theorem proving
at the IMO level, FIMO is currently tailored for the Lean formal language. It
comprises 149 formal problem statements, accompanied by both informal problem
descriptions and their corresponding LaTeX-based informal proofs. Through
initial experiments involving GPT-4, our findings underscore the existing
limitations in current methodologies, indicating a substantial journey ahead
before achieving satisfactory IMO-level automated theorem proving outcomes.
less
By: Qiao Jin, Yifan Yang, Qingyu Chen, Zhiyong Lu
While large language models (LLMs) have been successfully applied to various
tasks, they still face challenges with hallucinations. Augmenting LLMs with
domain-specific tools such as database utilities can facilitate easier and more
precise access to specialized knowledge. In this paper, we present GeneGPT, a
novel method for teaching LLMs to use the Web APIs of the National Center for
Biotechnology Information (NCBI) for answering genomics... more
While large language models (LLMs) have been successfully applied to various
tasks, they still face challenges with hallucinations. Augmenting LLMs with
domain-specific tools such as database utilities can facilitate easier and more
precise access to specialized knowledge. In this paper, we present GeneGPT, a
novel method for teaching LLMs to use the Web APIs of the National Center for
Biotechnology Information (NCBI) for answering genomics questions.
Specifically, we prompt Codex to solve the GeneTuring tests with NCBI Web APIs
by in-context learning and an augmented decoding algorithm that can detect and
execute API calls. Experimental results show that GeneGPT achieves
state-of-the-art performance on eight tasks in the GeneTuring benchmark with an
average score of 0.83, largely surpassing retrieval-augmented LLMs such as the
new Bing (0.44), biomedical LLMs such as BioMedLM (0.08) and BioGPT (0.04), as
well as GPT-3 (0.16) and ChatGPT (0.12). Our further analyses suggest that: (1)
API demonstrations have good cross-task generalizability and are more useful
than documentations for in-context learning; (2) GeneGPT can generalize to
longer chains of API calls and answer multi-hop questions in GeneHop, a novel
dataset introduced in this work; (3) Different types of errors are enriched in
different tasks, providing valuable insights for future improvements.
less
By: Jeongwhan Choi, Seoyoung Hong, Noseong Park, Sung-Bae Cho
Graph neural networks (GNNs) are one of the most popular research topics for
deep learning. GNN methods typically have been designed on top of the graph
signal processing theory. In particular, diffusion equations have been widely
used for designing the core processing layer of GNNs, and therefore they are
inevitably vulnerable to the notorious oversmoothing problem. Recently, a
couple of papers paid attention to reaction equations in conju... more
Graph neural networks (GNNs) are one of the most popular research topics for
deep learning. GNN methods typically have been designed on top of the graph
signal processing theory. In particular, diffusion equations have been widely
used for designing the core processing layer of GNNs, and therefore they are
inevitably vulnerable to the notorious oversmoothing problem. Recently, a
couple of papers paid attention to reaction equations in conjunctions with
diffusion equations. However, they all consider limited forms of reaction
equations. To this end, we present a reaction-diffusion equation-based GNN
method that considers all popular types of reaction equations in addition to
one special reaction equation designed by us. To our knowledge, our paper is
one of the most comprehensive studies on reaction-diffusion equation-based
GNNs. In our experiments with 9 datasets and 28 baselines, our method, called
GREAD, outperforms them in a majority of cases. Further synthetic data
experiments show that it mitigates the oversmoothing problem and works well for
various homophily rates.
less
By: Naishan Zheng, Man Zhou, Yanmeng Dong, Xiangyu Rui, Jie Huang, Chongyi Li, Feng Zhao
Deep neural networks have achieved remarkable progress in enhancing low-light
images by improving their brightness and eliminating noise. However, most
existing methods construct end-to-end mapping networks heuristically,
neglecting the intrinsic prior of image enhancement task and lacking
transparency and interpretability. Although some unfolding solutions have been
proposed to relieve these issues, they rely on proximal operator networks ... more
Deep neural networks have achieved remarkable progress in enhancing low-light
images by improving their brightness and eliminating noise. However, most
existing methods construct end-to-end mapping networks heuristically,
neglecting the intrinsic prior of image enhancement task and lacking
transparency and interpretability. Although some unfolding solutions have been
proposed to relieve these issues, they rely on proximal operator networks that
deliver ambiguous and implicit priors. In this work, we propose a paradigm for
low-light image enhancement that explores the potential of customized learnable
priors to improve the transparency of the deep unfolding paradigm. Motivated by
the powerful feature representation capability of Masked Autoencoder (MAE), we
customize MAE-based illumination and noise priors and redevelop them from two
perspectives: 1) \textbf{structure flow}: we train the MAE from a normal-light
image to its illumination properties and then embed it into the proximal
operator design of the unfolding architecture; and m2) \textbf{optimization
flow}: we train MAE from a normal-light image to its gradient representation
and then employ it as a regularization term to constrain noise in the model
output. These designs improve the interpretability and representation
capability of the model.Extensive experiments on multiple low-light image
enhancement datasets demonstrate the superiority of our proposed paradigm over
state-of-the-art methods. Code is available at
https://github.com/zheng980629/CUE.
less
Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples
0upvotes
By: Guanghui Li, Mingqi Gao, Heng Liu, Xiantong Zhen, Feng Zheng
Referring video object segmentation (RVOS), as a supervised learning task,
relies on sufficient annotated data for a given scene. However, in more
realistic scenarios, only minimal annotations are available for a new scene,
which poses significant challenges to existing RVOS methods. With this in mind,
we propose a simple yet effective model with a newly designed cross-modal
affinity (CMA) module based on a Transformer architecture. The CMA... more
Referring video object segmentation (RVOS), as a supervised learning task,
relies on sufficient annotated data for a given scene. However, in more
realistic scenarios, only minimal annotations are available for a new scene,
which poses significant challenges to existing RVOS methods. With this in mind,
we propose a simple yet effective model with a newly designed cross-modal
affinity (CMA) module based on a Transformer architecture. The CMA module
builds multimodal affinity with a few samples, thus quickly learning new
semantic information, and enabling the model to adapt to different scenarios.
Since the proposed method targets limited samples for new scenes, we generalize
the problem as - few-shot referring video object segmentation (FS-RVOS). To
foster research in this direction, we build up a new FS-RVOS benchmark based on
currently available datasets. The benchmark covers a wide range and includes
multiple situations, which can maximally simulate real-world scenarios.
Extensive experiments show that our model adapts well to different scenarios
with only a few samples, reaching state-of-the-art performance on the
benchmark. On Mini-Ref-YouTube-VOS, our model achieves an average performance
of 53.1 J and 54.8 F, which are 10% better than the baselines. Furthermore, we
show impressive results of 77.7 J and 74.8 F on Mini-Ref-SAIL-VOS, which are
significantly better than the baselines. Code is publicly available at
https://github.com/hengliusky/Few_shot_RVOS.
less
By: Yufei Wang, Yuxin Mao, Qi Liu, Yuchao Dai
RGB-guided depth completion aims at predicting dense depth maps from sparse
depth measurements and corresponding RGB images, where how to effectively and
efficiently exploit the multi-modal information is a key issue. Guided dynamic
filters, which generate spatially-variant depth-wise separable convolutional
filters from RGB features to guide depth features, have been proven to be
effective in this task. However, the dynamically generated f... more
RGB-guided depth completion aims at predicting dense depth maps from sparse
depth measurements and corresponding RGB images, where how to effectively and
efficiently exploit the multi-modal information is a key issue. Guided dynamic
filters, which generate spatially-variant depth-wise separable convolutional
filters from RGB features to guide depth features, have been proven to be
effective in this task. However, the dynamically generated filters require
massive model parameters, computational costs and memory footprints when the
number of feature channels is large. In this paper, we propose to decompose the
guided dynamic filters into a spatially-shared component multiplied by
content-adaptive adaptors at each spatial location. Based on the proposed idea,
we introduce two decomposition schemes A and B, which decompose the filters by
splitting the filter structure and using spatial-wise attention, respectively.
The decomposed filters not only maintain the favorable properties of guided
dynamic filters as being content-dependent and spatially-variant, but also
reduce model parameters and hardware costs, as the learned adaptors are
decoupled with the number of feature channels. Extensive experimental results
demonstrate that the methods using our schemes outperform state-of-the-art
methods on the KITTI dataset, and rank 1st and 2nd on the KITTI benchmark at
the time of submission. Meanwhile, they also achieve comparable performance on
the NYUv2 dataset. In addition, our proposed methods are general and could be
employed as plug-and-play feature fusion blocks in other multi-modal fusion
tasks such as RGB-D salient object detection.
less
By: Anand Sharma MIET, Meerut, A. G. Ramakrishnan IISc, Bengaluru
A set of features independent of character stroke direction and order
variations is proposed for online handwritten character recognition. A method
is developed that maps features like co-ordinates of points, orientations of
strokes at points, and dynamics of orientations of strokes at points spatially
as a function of co-ordinate values of the points and computes histograms of
these features from different regions in the spatial map.
Dif... more
A set of features independent of character stroke direction and order
variations is proposed for online handwritten character recognition. A method
is developed that maps features like co-ordinates of points, orientations of
strokes at points, and dynamics of orientations of strokes at points spatially
as a function of co-ordinate values of the points and computes histograms of
these features from different regions in the spatial map.
Different features like spatio-temporal, discrete Fourier transform, discrete
cosine transform, discrete wavelet transform, spatial, and histograms of
oriented gradients used in other studies for training classifiers for character
recognition are considered. The classifier chosen for classification
performance comparison, when trained with different features, is support vector
machines (SVM).
The character datasets used for training and testing the classifiers consist
of online handwritten samples of 96 different Hindi characters. There are 12832
and 2821 samples in training and testing datasets, respectively.
SVM classifiers trained with the proposed features has the highest
classification accuracy of 92.9\% when compared to the performances of SVM
classifiers trained with the other features and tested on the same testing
dataset. Therefore, the proposed features have better character discriminative
capability than the other features considered for comparison.
less
By: Siyang Jiang, Rui Fang, Hsi-Wen Chen, Wei Ding, Ming-Syan Chen
Support-query shift few-shot learning aims to classify unseen examples (query
set) to labeled data (support set) based on the learned embedding in a
low-dimensional space under a distribution shift between the support set and
the query set. However, in real-world scenarios the shifts are usually unknown
and varied, making it difficult to estimate in advance. Therefore, in this
paper, we propose a novel but more difficult challenge, RSQS, fo... more
Support-query shift few-shot learning aims to classify unseen examples (query
set) to labeled data (support set) based on the learned embedding in a
low-dimensional space under a distribution shift between the support set and
the query set. However, in real-world scenarios the shifts are usually unknown
and varied, making it difficult to estimate in advance. Therefore, in this
paper, we propose a novel but more difficult challenge, RSQS, focusing on
Realistic Support-Query Shift few-shot learning. The key feature of RSQS is
that the individual samples in a meta-task are subjected to multiple
distribution shifts in each meta-task. In addition, we propose a unified
adversarial feature alignment method called DUal adversarial ALignment
framework (DuaL) to relieve RSQS from two aspects, i.e., inter-domain bias and
intra-domain variance. On the one hand, for the inter-domain bias, we corrupt
the original data in advance and use the synthesized perturbed inputs to train
the repairer network by minimizing distance in the feature level. On the other
hand, for intra-domain variance, we proposed a generator network to synthesize
hard, i.e., less similar, examples from the support set in a self-supervised
manner and introduce regularized optimal transportation to derive a smooth
optimal transportation plan. Lastly, a benchmark of RSQS is built with several
state-of-the-art baselines among three datasets (CIFAR100, mini-ImageNet, and
Tiered-Imagenet). Experiment results show that DuaL significantly outperforms
the state-of-the-art methods in our benchmark.
less
By: Sander Riisøen Jyhne, Per-Arne Andersen, Morten Goodwin
This paper presents Deep Networks for Improved Segmentation Edges (DeNISE), a
novel data enhancement technique using edge detection and segmentation models
to improve the boundary quality of segmentation masks. DeNISE utilizes the
inherent differences in two sequential deep neural architectures to improve the
accuracy of the predicted segmentation edge. DeNISE applies to all types of
neural networks and is not trained end-to-end, allowing r... more
This paper presents Deep Networks for Improved Segmentation Edges (DeNISE), a
novel data enhancement technique using edge detection and segmentation models
to improve the boundary quality of segmentation masks. DeNISE utilizes the
inherent differences in two sequential deep neural architectures to improve the
accuracy of the predicted segmentation edge. DeNISE applies to all types of
neural networks and is not trained end-to-end, allowing rapid experiments to
discover which models complement each other. We test and apply DeNISE for
building segmentation in aerial images. Aerial images are known for difficult
conditions as they have a low resolution with optical noise, such as
reflections, shadows, and visual obstructions. Overall the paper demonstrates
the potential for DeNISE. Using the technique, we improve the baseline results
with a building IoU of 78.9%.
less
By: Andrew Du, Anh-Dzung Doan, Yee Wei Law, Tat-Jun Chin
The advent of satellite-borne machine learning hardware accelerators has
enabled the on-board processing of payload data using machine learning
techniques such as convolutional neural networks (CNN). A notable example is
using a CNN to detect the presence of clouds in hyperspectral data captured on
Earth observation (EO) missions, whereby only clear sky data is downlinked to
conserve bandwidth. However, prior to deployment, new missions tha... more
The advent of satellite-borne machine learning hardware accelerators has
enabled the on-board processing of payload data using machine learning
techniques such as convolutional neural networks (CNN). A notable example is
using a CNN to detect the presence of clouds in hyperspectral data captured on
Earth observation (EO) missions, whereby only clear sky data is downlinked to
conserve bandwidth. However, prior to deployment, new missions that employ new
sensors will not have enough representative datasets to train a CNN model,
while a model trained solely on data from previous missions will underperform
when deployed to process the data on the new missions. This underperformance
stems from the domain gap, i.e., differences in the underlying distributions of
the data generated by the different sensors in previous and future missions. In
this paper, we address the domain gap problem in the context of on-board
hyperspectral cloud detection. Our main contributions lie in formulating new
domain adaptation tasks that are motivated by a concrete EO mission, developing
a novel algorithm for bandwidth-efficient supervised domain adaptation, and
demonstrating test-time adaptation algorithms on space deployable neural
network accelerators. Our contributions enable minimal data transmission to be
invoked (e.g., only 1% of the weights in ResNet50) to achieve domain
adaptation, thereby allowing more sophisticated CNN models to be deployed and
updated on satellites without being hampered by domain gap and bandwidth
limitations.
less
By: Yuxiang Yang, Yingqi Deng, Jiahao Nie, Jing Zhang
3D single object tracking (SOT) in point clouds is still a challenging
problem due to appearance variation, distractors, and high sparsity of point
clouds. Notably, in autonomous driving scenarios, the target object typically
maintains spatial adjacency across consecutive frames, predominantly moving
horizontally. This spatial continuity offers valuable prior knowledge for
target localization. However, existing trackers, which often employ ... more
3D single object tracking (SOT) in point clouds is still a challenging
problem due to appearance variation, distractors, and high sparsity of point
clouds. Notably, in autonomous driving scenarios, the target object typically
maintains spatial adjacency across consecutive frames, predominantly moving
horizontally. This spatial continuity offers valuable prior knowledge for
target localization. However, existing trackers, which often employ point-wise
representations, struggle to efficiently utilize this knowledge owing to the
irregular format of such representations. Consequently, they require elaborate
designs and solving multiple subtasks to establish spatial correspondence. In
this paper, we introduce BEVTrack, a simple yet strong baseline framework for
3D SOT. After converting consecutive point clouds into the common
Bird's-Eye-View representation, BEVTrack inherently encodes spatial proximity
and adeptly captures motion cues for tracking via a simple element-wise
operation and convolutional layers. Additionally, to better deal with objects
having diverse sizes and moving patterns, BEVTrack directly learns the
underlying motion distribution rather than making a fixed Laplacian or Gaussian
assumption as in previous works. Without bells and whistles, BEVTrack achieves
state-of-the-art performance on KITTI and NuScenes datasets while maintaining a
high inference speed of 122 FPS. The code will be released at
https://github.com/xmm-prio/BEVTrack.
less
By: Renyu Zhu, Chengcheng Han, Yong Qian, Qiushi Sun, Xiang Li, Ming Gao, Xuezhi Cao, Yunsen Xian
We study the problem of multimodal fusion in this paper. Recent
exchanging-based methods have been proposed for vision-vision fusion, which aim
to exchange embeddings learned from one modality to the other. However, most of
them project inputs of multimodalities into different low-dimensional spaces
and cannot be applied to the sequential input data. To solve these issues, in
this paper, we propose a novel exchanging-based multimodal fusion... more
We study the problem of multimodal fusion in this paper. Recent
exchanging-based methods have been proposed for vision-vision fusion, which aim
to exchange embeddings learned from one modality to the other. However, most of
them project inputs of multimodalities into different low-dimensional spaces
and cannot be applied to the sequential input data. To solve these issues, in
this paper, we propose a novel exchanging-based multimodal fusion model MuSE
for text-vision fusion based on Transformer. We first use two encoders to
separately map multimodal inputs into different low-dimensional spaces. Then we
employ two decoders to regularize the embeddings and pull them into the same
space. The two decoders capture the correlations between texts and images with
the image captioning task and the text-to-image generation task, respectively.
Further, based on the regularized embeddings, we present CrossTransformer,
which uses two Transformer encoders with shared parameters as the backbone
model to exchange knowledge between multimodalities. Specifically,
CrossTransformer first learns the global contextual information of the inputs
in the shallow layers. After that, it performs inter-modal exchange by
selecting a proportion of tokens in one modality and replacing their embeddings
with the average of embeddings in the other modality. We conduct extensive
experiments to evaluate the performance of MuSE on the Multimodal Named Entity
Recognition task and the Multimodal Sentiment Analysis task. Our results show
the superiority of MuSE against other competitors. Our code and data are
provided at https://github.com/RecklessRonan/MuSE.
less
By: Paolo Cudrano, Matteo Bellusci, Giuseppe Macino, Matteo Matteucci
Accurate road surface classification is crucial for autonomous vehicles (AVs)
to optimize driving conditions, enhance safety, and enable advanced road
mapping. However, deep learning models for road surface classification suffer
from poor generalization when tested on unseen datasets. To update these models
with new information, also the original training dataset must be taken into
account, in order to avoid catastrophic forgetting. This is... more
Accurate road surface classification is crucial for autonomous vehicles (AVs)
to optimize driving conditions, enhance safety, and enable advanced road
mapping. However, deep learning models for road surface classification suffer
from poor generalization when tested on unseen datasets. To update these models
with new information, also the original training dataset must be taken into
account, in order to avoid catastrophic forgetting. This is, however,
inefficient if not impossible, e.g., when the data is collected in streams or
large amounts. To overcome this limitation and enable fast and efficient
cross-dataset adaptation, we propose to employ continual learning finetuning
methods designed to retain past knowledge while adapting to new data, thus
effectively avoiding forgetting. Experimental results demonstrate the
superiority of this approach over naive finetuning, achieving performance close
to fresh retraining. While solving this known problem, we also provide a
general description of how the same technique can be adopted in other AV
scenarios. We highlight the potential computational and economic benefits that
a continual-based adaptation can bring to the AV industry, while also reducing
greenhouse emissions due to unnecessary joint retraining.
less
By: Yifan Song, Mengkun She, Kevin Köser
Underwater image restoration has been a challenging problem for decades since
the advent of underwater photography. Most solutions focus on shallow water
scenarios, where the scene is uniformly illuminated by the sunlight. However,
the vast majority of uncharted underwater terrain is located beyond 200 meters
depth where natural light is scarce and artificial illumination is needed. In
such cases, light sources co-moving with the camera, dy... more
Underwater image restoration has been a challenging problem for decades since
the advent of underwater photography. Most solutions focus on shallow water
scenarios, where the scene is uniformly illuminated by the sunlight. However,
the vast majority of uncharted underwater terrain is located beyond 200 meters
depth where natural light is scarce and artificial illumination is needed. In
such cases, light sources co-moving with the camera, dynamically change the
scene appearance, which make shallow water restoration methods inadequate. In
particular for multi-light source systems (composed of dozens of LEDs
nowadays), calibrating each light is time-consuming, error-prone and tedious,
and we observe that only the integrated illumination within the viewing volume
of the camera is critical, rather than the individual light sources. The key
idea of this paper is therefore to exploit the appearance changes of objects or
the seafloor, when traversing the viewing frustum of the camera. Through new
constraints assuming Lambertian surfaces, corresponding image pixels constrain
the light field in front of the camera, and for each voxel a signal factor and
a backscatter value are stored in a volumetric grid that can be used for very
efficient image restoration of camera-light platforms, which facilitates
consistently texturing large 3D models and maps that would otherwise be
dominated by lighting and medium artifacts. To validate the effectiveness of
our approach, we conducted extensive experiments on simulated and real-world
datasets. The results of these experiments demonstrate the robustness of our
approach in restoring the true albedo of objects, while mitigating the
influence of lighting and medium effects. Furthermore, we demonstrate our
approach can be readily extended to other scenarios, including in-air imaging
with artificial illumination or other similar cases.
less
By: Haixu Song, Shiyu Huang, Yinpeng Dong, Wei-Wei Tu
The rise of deepfake images, especially of well-known personalities, poses a
serious threat to the dissemination of authentic information. To tackle this,
we present a thorough investigation into how deepfakes are produced and how
they can be identified. The cornerstone of our research is a rich collection of
artificial celebrity faces, titled DeepFakeFace (DFF). We crafted the DFF
dataset using advanced diffusion models and have shared it ... more
The rise of deepfake images, especially of well-known personalities, poses a
serious threat to the dissemination of authentic information. To tackle this,
we present a thorough investigation into how deepfakes are produced and how
they can be identified. The cornerstone of our research is a rich collection of
artificial celebrity faces, titled DeepFakeFace (DFF). We crafted the DFF
dataset using advanced diffusion models and have shared it with the community
through online platforms. This data serves as a robust foundation to train and
test algorithms designed to spot deepfakes. We carried out a thorough review of
the DFF dataset and suggest two evaluation methods to gauge the strength and
adaptability of deepfake recognition tools. The first method tests whether an
algorithm trained on one type of fake images can recognize those produced by
other methods. The second evaluates the algorithm's performance with imperfect
images, like those that are blurry, of low quality, or compressed. Given varied
results across deepfake methods and image changes, our findings stress the need
for better deepfake detectors. Our DFF dataset and tests aim to boost the
development of more effective tools against deepfakes.
less
By: Cathrine Damgaard, Trine Naja Eriksen, Dovile Juodelyte, Veronika Cheplygina, Amelia Jiménez-Sánchez
The advancement of machine learning algorithms in medical image analysis
requires the expansion of training datasets. A popular and cost-effective
approach is automated annotation extraction from free-text medical reports,
primarily due to the high costs associated with expert clinicians annotating
chest X-ray images. However, it has been shown that the resulting datasets are
susceptible to biases and shortcuts. Another strategy to increase... more
The advancement of machine learning algorithms in medical image analysis
requires the expansion of training datasets. A popular and cost-effective
approach is automated annotation extraction from free-text medical reports,
primarily due to the high costs associated with expert clinicians annotating
chest X-ray images. However, it has been shown that the resulting datasets are
susceptible to biases and shortcuts. Another strategy to increase the size of a
dataset is crowdsourcing, a widely adopted practice in general computer vision
with some success in medical image analysis. In a similar vein to
crowdsourcing, we enhance two publicly available chest X-ray datasets by
incorporating non-expert annotations. However, instead of using diagnostic
labels, we annotate shortcuts in the form of tubes. We collect 3.5k chest drain
annotations for CXR14, and 1k annotations for 4 different tube types in
PadChest. We train a chest drain detector with the non-expert annotations that
generalizes well to expert labels. Moreover, we compare our annotations to
those provided by experts and show "moderate" to "almost perfect" agreement.
Finally, we present a pathology agreement study to raise awareness about ground
truth annotations. We make our annotations and code available.
less
By: Hongyu Hu, Jiyuan Zhang, Minyi Zhao, Zhenbang Sun
Nowadays, the research on Large Vision-Language Models (LVLMs) has been
significantly promoted thanks to the success of Large Language Models (LLM).
Nevertheless, these Vision-Language Models (VLMs) are suffering from the
drawback of hallucination -- due to insufficient understanding of vision and
language modalities, VLMs may generate incorrect perception information when
doing downstream applications, for example, captioning a non-existen... more
Nowadays, the research on Large Vision-Language Models (LVLMs) has been
significantly promoted thanks to the success of Large Language Models (LLM).
Nevertheless, these Vision-Language Models (VLMs) are suffering from the
drawback of hallucination -- due to insufficient understanding of vision and
language modalities, VLMs may generate incorrect perception information when
doing downstream applications, for example, captioning a non-existent entity.
To address the hallucination phenomenon, on the one hand, we introduce a
Contrastive Instruction Evaluation Method (CIEM), which is an automatic
pipeline that leverages an annotated image-text dataset coupled with an LLM to
generate factual/contrastive question-answer pairs for the evaluation of the
hallucination of VLMs. On the other hand, based on CIEM, we further propose a
new instruction tuning method called CIT (the abbreviation of Contrastive
Instruction Tuning) to alleviate the hallucination of VLMs by automatically
producing high-quality factual/contrastive question-answer pairs and
corresponding justifications for model tuning. Through extensive experiments on
CIEM and CIT, we pinpoint the hallucination issues commonly present in existing
VLMs, the disability of the current instruction-tuning dataset to handle the
hallucination phenomenon and the superiority of CIT-tuned VLMs over both CIEM
and public datasets.
less
By: Zhenghong Zhou, Huangxuan Zhao, Jiemin Fang, Dongqiao Xiang, Lei Chen, Lingxia Wu, Feihong Wu, Wenyu Liu, Chuansheng Zheng, Xinggang Wang
Four-dimensional Digital Subtraction Angiography (4D DSA) plays a critical
role in the diagnosis of many medical diseases, such as Arteriovenous
Malformations (AVM) and Arteriovenous Fistulas (AVF). Despite its significant
application value, the reconstruction of 4D DSA demands numerous views to
effectively model the intricate vessels and radiocontrast flow, thereby
implying a significant radiation dose. To address this high radiation issue... more
Four-dimensional Digital Subtraction Angiography (4D DSA) plays a critical
role in the diagnosis of many medical diseases, such as Arteriovenous
Malformations (AVM) and Arteriovenous Fistulas (AVF). Despite its significant
application value, the reconstruction of 4D DSA demands numerous views to
effectively model the intricate vessels and radiocontrast flow, thereby
implying a significant radiation dose. To address this high radiation issue, we
propose a Time-aware Attenuation Voxel (TiAVox) approach for sparse-view 4D DSA
reconstruction, which paves the way for high-quality 4D imaging. Additionally,
2D and 3D DSA imaging results can be generated from the reconstructed 4D DSA
images. TiAVox introduces 4D attenuation voxel grids, which reflect attenuation
properties from both spatial and temporal dimensions. It is optimized by
minimizing discrepancies between the rendered images and sparse 2D DSA images.
Without any neural network involved, TiAVox enjoys specific physical
interpretability. The parameters of each learnable voxel represent the
attenuation coefficients. We validated the TiAVox approach on both clinical and
simulated datasets, achieving a 31.23 Peak Signal-to-Noise Ratio (PSNR) for
novel view synthesis using only 30 views on the clinically sourced dataset,
whereas traditional Feldkamp-Davis-Kress methods required 133 views. Similarly,
with merely 10 views from the synthetic dataset, TiAVox yielded a PSNR of 34.32
for novel view synthesis and 41.40 for 3D reconstruction. We also executed
ablation studies to corroborate the essential components of TiAVox. The code
will be publically available.
less
By: Sergi Garcia-Bordils, Dimosthenis Karatzas, Marçal Rusiñol
We introduce the structured scene-text spotting task, which requires a
scene-text OCR system to spot text in the wild according to a query regular
expression. Contrary to generic scene text OCR, structured scene-text spotting
seeks to dynamically condition both scene text detection and recognition on
user-provided regular expressions. To tackle this task, we propose the
Structured TExt sPotter (STEP), a model that exploits the provided text... more
We introduce the structured scene-text spotting task, which requires a
scene-text OCR system to spot text in the wild according to a query regular
expression. Contrary to generic scene text OCR, structured scene-text spotting
seeks to dynamically condition both scene text detection and recognition on
user-provided regular expressions. To tackle this task, we propose the
Structured TExt sPotter (STEP), a model that exploits the provided text
structure to guide the OCR process. STEP is able to deal with regular
expressions that contain spaces and it is not bound to detection at the
word-level granularity. Our approach enables accurate zero-shot structured text
spotting in a wide variety of real-world reading scenarios and is solely
trained on publicly available data. To demonstrate the effectiveness of our
approach, we introduce a new challenging test dataset that contains several
types of out-of-vocabulary structured text, reflecting important reading
applications of fields such as prices, dates, serial numbers, license plates
etc. We demonstrate that STEP can provide specialised OCR performance on demand
in all tested scenarios.
less
By: Yue Xu, Yong-Lu Li, Zhemin Huang, Michael Xu Liu, Cewu Lu, Yu-Wing Tai, Chi-Keung Tang
With the surge in attention to Egocentric Hand-Object Interaction (Ego-HOI),
large-scale datasets such as Ego4D and EPIC-KITCHENS have been proposed.
However, most current research is built on resources derived from third-person
video action recognition. This inherent domain gap between first- and
third-person action videos, which have not been adequately addressed before,
makes current Ego-HOI suboptimal. This paper rethinks and proposes a... more
With the surge in attention to Egocentric Hand-Object Interaction (Ego-HOI),
large-scale datasets such as Ego4D and EPIC-KITCHENS have been proposed.
However, most current research is built on resources derived from third-person
video action recognition. This inherent domain gap between first- and
third-person action videos, which have not been adequately addressed before,
makes current Ego-HOI suboptimal. This paper rethinks and proposes a new
framework as an infrastructure to advance Ego-HOI recognition by Probing,
Curation and Adaption (EgoPCA). We contribute comprehensive pre-train sets,
balanced test sets and a new baseline, which are complete with a
training-finetuning strategy. With our new framework, we not only achieve
state-of-the-art performance on Ego-HOI benchmarks but also build several new
and effective mechanisms and settings to advance further research. We believe
our data and the findings will pave a new way for Ego-HOI understanding. Code
and data are available at https://mvig-rhos.com/ego_pca
less
By: Vimal K B, Saketh Bachu, Tanmay Garg, Niveditha Lakshmi Narasimhan, Raghavan Konuru, Vineeth N Balasubramanian
Estimating the transferability of publicly available pretrained models to a
target task has assumed an important place for transfer learning tasks in
recent years. Existing efforts propose metrics that allow a user to choose one
model from a pool of pre-trained models without having to fine-tune each model
individually and identify one explicitly. With the growth in the number of
available pre-trained models and the popularity of model ense... more
Estimating the transferability of publicly available pretrained models to a
target task has assumed an important place for transfer learning tasks in
recent years. Existing efforts propose metrics that allow a user to choose one
model from a pool of pre-trained models without having to fine-tune each model
individually and identify one explicitly. With the growth in the number of
available pre-trained models and the popularity of model ensembles, it also
becomes essential to study the transferability of multiple-source models for a
given target task. The few existing efforts study transferability in such
multi-source ensemble settings using just the outputs of the classification
layer and neglect possible domain or task mismatch. Moreover, they overlook the
most important factor while selecting the source models, viz., the cohesiveness
factor between them, which can impact the performance and confidence in the
prediction of the ensemble. To address these gaps, we propose a novel Optimal
tranSport-based suBmOdular tRaNsferability metric (OSBORN) to estimate the
transferability of an ensemble of models to a downstream task. OSBORN
collectively accounts for image domain difference, task difference, and
cohesiveness of models in the ensemble to provide reliable estimates of
transferability. We gauge the performance of OSBORN on both image
classification and semantic segmentation tasks. Our setup includes 28 source
datasets, 11 target datasets, 5 model architectures, and 2 pre-training
methods. We benchmark our method against current state-of-the-art metrics
MS-LEEP and E-LEEP, and outperform them consistently using the proposed
approach.
less
By: Yiwei Bao, Feng Lu
Although recent deep learning based gaze estimation approaches have achieved
much improvement, we still know little about how gaze features are connected to
the physics of gaze. In this paper, we try to answer this question by analyzing
the gaze feature manifold. Our analysis revealed the insight that the geodesic
distance between gaze features is consistent with the gaze differences between
samples. According to this finding, we construct ... more
Although recent deep learning based gaze estimation approaches have achieved
much improvement, we still know little about how gaze features are connected to
the physics of gaze. In this paper, we try to answer this question by analyzing
the gaze feature manifold. Our analysis revealed the insight that the geodesic
distance between gaze features is consistent with the gaze differences between
samples. According to this finding, we construct the Physics- Consistent
Feature (PCF) in an analytical way, which connects gaze feature to the physical
definition of gaze. We further propose the PCFGaze framework that directly
optimizes gaze feature space by the guidance of PCF. Experimental results
demonstrate that the proposed framework alleviates the overfitting problem and
significantly improves cross-domain gaze estimation accuracy without extra
training data. The insight of gaze feature has the potential to benefit other
regression tasks with physical meanings.
less
By: Haonan Qiu, Zhaoxi Chen, Yuming Jiang, Hang Zhou, Xiangyu Fan, Lei Yang, Wayne Wu, Ziwei Liu
Recent years have witnessed great progress in creating vivid audio-driven
portraits from monocular videos. However, how to seamlessly adapt the created
video avatars to other scenarios with different backgrounds and lighting
conditions remains unsolved. On the other hand, existing relighting studies
mostly rely on dynamically lighted or multi-view data, which are too expensive
for creating video portraits. To bridge this gap, we propose Rel... more
Recent years have witnessed great progress in creating vivid audio-driven
portraits from monocular videos. However, how to seamlessly adapt the created
video avatars to other scenarios with different backgrounds and lighting
conditions remains unsolved. On the other hand, existing relighting studies
mostly rely on dynamically lighted or multi-view data, which are too expensive
for creating video portraits. To bridge this gap, we propose ReliTalk, a novel
framework for relightable audio-driven talking portrait generation from
monocular videos. Our key insight is to decompose the portrait's reflectance
from implicitly learned audio-driven facial normals and images. Specifically,
we involve 3D facial priors derived from audio features to predict delicate
normal maps through implicit functions. These initially predicted normals then
take a crucial part in reflectance decomposition by dynamically estimating the
lighting condition of the given video. Moreover, the stereoscopic face
representation is refined using the identity-consistent loss under simulated
multiple lighting conditions, addressing the ill-posed problem caused by
limited views available from a single monocular video. Extensive experiments
validate the superiority of our proposed framework on both real and synthetic
datasets. Our code is released in https://github.com/arthur-qiu/ReliTalk.
less
By: Svetlana Pavlitska, Nico Lambing, Ashok Kumar Bangaru, J. Marius Zöllner
Real-time traffic light recognition is essential for autonomous driving. Yet,
a cohesive overview of the underlying model architectures for this task is
currently missing. In this work, we conduct a comprehensive survey and analysis
of traffic light recognition methods that use convolutional neural networks
(CNNs). We focus on two essential aspects: datasets and CNN architectures.
Based on an underlying architecture, we cluster methods into... more
Real-time traffic light recognition is essential for autonomous driving. Yet,
a cohesive overview of the underlying model architectures for this task is
currently missing. In this work, we conduct a comprehensive survey and analysis
of traffic light recognition methods that use convolutional neural networks
(CNNs). We focus on two essential aspects: datasets and CNN architectures.
Based on an underlying architecture, we cluster methods into three major
groups: (1) modifications of generic object detectors which compensate for
specific task characteristics, (2) multi-stage approaches involving both
rule-based and CNN components, and (3) task-specific single-stage methods. We
describe the most important works in each cluster, discuss the usage of the
datasets, and identify research gaps.
less
By: Yongkang Zhao, Chuang Zhu, Yuan Li, Shuaishuai Wang, Zihan Lan, Yuanyuan Qiao
Detecting small moving targets accurately in infrared (IR) image sequences is
a significant challenge. To address this problem, we propose a novel method
called spatial-temporal local feature difference (STLFD) with adaptive
background suppression (ABS). Our approach utilizes filters in the spatial and
temporal domains and performs pixel-level ABS on the output to enhance the
contrast between the target and the background. The proposed meth... more
Detecting small moving targets accurately in infrared (IR) image sequences is
a significant challenge. To address this problem, we propose a novel method
called spatial-temporal local feature difference (STLFD) with adaptive
background suppression (ABS). Our approach utilizes filters in the spatial and
temporal domains and performs pixel-level ABS on the output to enhance the
contrast between the target and the background. The proposed method comprises
three steps. First, we obtain three temporal frame images based on the current
frame image and extract two feature maps using the designed spatial domain and
temporal domain filters. Next, we fuse the information of the spatial domain
and temporal domain to produce the spatial-temporal feature maps and suppress
noise using our pixel-level ABS module. Finally, we obtain the segmented binary
map by applying a threshold. Our experimental results demonstrate that the
proposed method outperforms existing state-of-the-art methods for infrared
small-moving target detection.
less
By: Fanda Fan, Chaoxu Guo, Litong Gong, Biao Wang, Tiezheng Ge, Yuning Jiang, Chunjie Luo, Jianfeng Zhan
Video outpainting aims to adequately complete missing areas at the edges of
video frames. Compared to image outpainting, it presents an additional
challenge as the model should maintain the temporal consistency of the filled
area. In this paper, we introduce a masked 3D diffusion model for video
outpainting. We use the technique of mask modeling to train the 3D diffusion
model. This allows us to use multiple guide frames to connect the resu... more
Video outpainting aims to adequately complete missing areas at the edges of
video frames. Compared to image outpainting, it presents an additional
challenge as the model should maintain the temporal consistency of the filled
area. In this paper, we introduce a masked 3D diffusion model for video
outpainting. We use the technique of mask modeling to train the 3D diffusion
model. This allows us to use multiple guide frames to connect the results of
multiple video clip inferences, thus ensuring temporal consistency and reducing
jitter between adjacent frames. Meanwhile, we extract the global frames of the
video as prompts and guide the model to obtain information other than the
current video clip using cross-attention. We also introduce a hybrid
coarse-to-fine inference pipeline to alleviate the artifact accumulation
problem. The existing coarse-to-fine pipeline only uses the infilling strategy,
which brings degradation because the time interval of the sparse frames is too
large. Our pipeline benefits from bidirectional learning of the mask modeling
and thus can employ a hybrid strategy of infilling and interpolation when
generating sparse frames. Experiments show that our method achieves
state-of-the-art results in video outpainting tasks. More results are provided
at our https://fanfanda.github.io/M3DDM/.
less
By: Lorenzo Mur-Labadia, Jose J. Guerrero, Ruben Martinez-Cantin
Accurate affordance detection and segmentation with pixel precision is an
important piece in many complex systems based on interactions, such as robots
and assitive devices. We present a new approach to affordance perception which
enables accurate multi-label segmentation. Our approach can be used to
automatically extract grounded affordances from first person videos of
interactions using a 3D map of the environment providing pixel level pr... more
Accurate affordance detection and segmentation with pixel precision is an
important piece in many complex systems based on interactions, such as robots
and assitive devices. We present a new approach to affordance perception which
enables accurate multi-label segmentation. Our approach can be used to
automatically extract grounded affordances from first person videos of
interactions using a 3D map of the environment providing pixel level precision
for the affordance location. We use this method to build the largest and most
complete dataset on affordances based on the EPIC-Kitchen dataset, EPIC-Aff,
which provides interaction-grounded, multi-label, metric and spatial affordance
annotations. Then, we propose a new approach to affordance segmentation based
on multi-label detection which enables multiple affordances to co-exists in the
same space, for example if they are associated with the same object. We present
several strategies of multi-label detection using several segmentation
architectures. The experimental results highlight the importance of the
multi-label detection. Finally, we show how our metric representation can be
exploited for build a map of interaction hotspots in spatial action-centric
zones and use that representation to perform a task-oriented navigation.
less
By: Wei Suo, Mengyang Sun, Weisong Liu, Yiqi Gao, Peng Wang, Yanning Zhang, Qi Wu
VQA Natural Language Explanation (VQA-NLE) task aims to explain the
decision-making process of VQA models in natural language. Unlike traditional
attention or gradient analysis, free-text rationales can be easier to
understand and gain users' trust. Existing methods mostly use post-hoc or
self-rationalization models to obtain a plausible explanation. However, these
frameworks are bottlenecked by the following challenges: 1) the reasoning
pr... more
VQA Natural Language Explanation (VQA-NLE) task aims to explain the
decision-making process of VQA models in natural language. Unlike traditional
attention or gradient analysis, free-text rationales can be easier to
understand and gain users' trust. Existing methods mostly use post-hoc or
self-rationalization models to obtain a plausible explanation. However, these
frameworks are bottlenecked by the following challenges: 1) the reasoning
process cannot be faithfully responded to and suffer from the problem of
logical inconsistency. 2) Human-annotated explanations are expensive and
time-consuming to collect. In this paper, we propose a new Semi-Supervised
VQA-NLE via Self-Critical Learning (S3C), which evaluates the candidate
explanations by answering rewards to improve the logical consistency between
answers and rationales. With a semi-supervised learning framework, the S3C can
benefit from a tremendous amount of samples without human-annotated
explanations. A large number of automatic measures and human evaluations all
show the effectiveness of our method. Meanwhile, the framework achieves a new
state-of-the-art performance on the two VQA-NLE datasets.
less
DCP-Net: A Distributed Collaborative Perception Network for Remote Sensing Semantic Segmentation
0upvotes
By: Zhechao Wang, Peirui Cheng, Shujing Duan, Kaiqiang Chen, Zhirui Wang, Xinming Li, Xian Sun
Onboard intelligent processing is widely applied in emergency tasks in the
field of remote sensing. However, it is predominantly confined to an individual
platform with a limited observation range as well as susceptibility to
interference, resulting in limited accuracy. Considering the current state of
multi-platform collaborative observation, this article innovatively presents a
distributed collaborative perception network called DCP-Net. ... more
Onboard intelligent processing is widely applied in emergency tasks in the
field of remote sensing. However, it is predominantly confined to an individual
platform with a limited observation range as well as susceptibility to
interference, resulting in limited accuracy. Considering the current state of
multi-platform collaborative observation, this article innovatively presents a
distributed collaborative perception network called DCP-Net. Firstly, the
proposed DCP-Net helps members to enhance perception performance by integrating
features from other platforms. Secondly, a self-mutual information match module
is proposed to identify collaboration opportunities and select suitable
partners, prioritizing critical collaborative features and reducing redundant
transmission cost. Thirdly, a related feature fusion module is designed to
address the misalignment between local and collaborative features, improving
the quality of fused features for the downstream task. We conduct extensive
experiments and visualization analyses using three semantic segmentation
datasets, including Potsdam, iSAID and DFC23. The results demonstrate that
DCP-Net outperforms the existing methods comprehensively, improving mIoU by
2.61%~16.89% at the highest collaboration efficiency, which promotes the
performance to a state-of-the-art level.
less
By: Julian Lorenz, Florian Barthel, Daniel Kienzle, Rainer Lienhart
Current scene graph datasets suffer from strong long-tail distributions of
their predicate classes. Due to a very low number of some predicate classes in
the test sets, no reliable metrics can be retrieved for the rarest classes. We
construct a new panoptic scene graph dataset and a set of metrics that are
designed as a benchmark for the predictive performance especially on rare
predicate classes. To construct the new dataset, we propose a ... more
Current scene graph datasets suffer from strong long-tail distributions of
their predicate classes. Due to a very low number of some predicate classes in
the test sets, no reliable metrics can be retrieved for the rarest classes. We
construct a new panoptic scene graph dataset and a set of metrics that are
designed as a benchmark for the predictive performance especially on rare
predicate classes. To construct the new dataset, we propose a model-assisted
annotation pipeline that efficiently finds rare predicate classes that are
hidden in a large set of images like needles in a haystack.
Contrary to prior scene graph datasets, Haystack contains explicit negative
annotations, i.e. annotations that a given relation does not have a certain
predicate class. Negative annotations are helpful especially in the field of
scene graph generation and open up a whole new set of possibilities to improve
current scene graph generation models.
Haystack is 100% compatible with existing panoptic scene graph datasets and
can easily be integrated with existing evaluation pipelines. Our dataset and
code can be found here: https://lorjul.github.io/haystack/. It includes
annotation files and simple to use scripts and utilities, to help with
integrating our dataset in existing work.
less
By: Nanne van Noord
Dataset summarisation is a fruitful approach to dataset inspection. However,
when applied to a single dataset the discovery of visual concepts is restricted
to those most prominent. We argue that a comparative approach can expand upon
this paradigm to enable richer forms of dataset inspection that go beyond the
most prominent concepts. To enable dataset comparison we present a module that
learns concept-level prototypes across datasets. We ... more
Dataset summarisation is a fruitful approach to dataset inspection. However,
when applied to a single dataset the discovery of visual concepts is restricted
to those most prominent. We argue that a comparative approach can expand upon
this paradigm to enable richer forms of dataset inspection that go beyond the
most prominent concepts. To enable dataset comparison we present a module that
learns concept-level prototypes across datasets. We leverage self-supervised
learning to discover these prototypes without supervision, and we demonstrate
the benefits of our approach in two case-studies. Our findings show that
dataset comparison extends dataset inspection and we hope to encourage more
works in this direction. Code and usage instructions available at
https://github.com/Nanne/ProtoSim
less
By: Taegyeong Lee, Jeonghun Kang, Hyeonyu Kim, Taehwan Kim
Representing wild sounds as images is an important but challenging task due
to the lack of paired datasets between sound and images and the significant
differences in the characteristics of these two modalities. Previous studies
have focused on generating images from sound in limited categories or music. In
this paper, we propose a novel approach to generate images from in-the-wild
sounds. First, we convert sound into text using audio capti... more
Representing wild sounds as images is an important but challenging task due
to the lack of paired datasets between sound and images and the significant
differences in the characteristics of these two modalities. Previous studies
have focused on generating images from sound in limited categories or music. In
this paper, we propose a novel approach to generate images from in-the-wild
sounds. First, we convert sound into text using audio captioning. Second, we
propose audio attention and sentence attention to represent the rich
characteristics of sound and visualize the sound. Lastly, we propose a direct
sound optimization with CLIPscore and AudioCLIP and generate images with a
diffusion-based model. In experiments, it shows that our model is able to
generate high quality images from wild sounds and outperforms baselines in both
quantitative and qualitative evaluations on wild audio datasets.
less
By: George Stoica, Mihaela Breaban, Vlad Barbu
Using additional training data is known to improve the results, especially
for medical image 3D segmentation where there is a lack of training material
and the model needs to generalize well from few available data. However, the
new data could have been acquired using other instruments and preprocessed such
its distribution is significantly different from the original training data.
Therefore, we study techniques which ameliorate domain shi... more
Using additional training data is known to improve the results, especially
for medical image 3D segmentation where there is a lack of training material
and the model needs to generalize well from few available data. However, the
new data could have been acquired using other instruments and preprocessed such
its distribution is significantly different from the original training data.
Therefore, we study techniques which ameliorate domain shift during training so
that the additional data becomes better usable for preprocessing and training
together with the original data. Our results show that transforming the
additional data using histogram matching has better results than using simple
normalization.
less
By: Xin Zhou, Jinghua Hou, Tingting Yao, Dingkang Liang, Zhe Liu, Zhikang Zou, Xiaoqing Ye, Jianwei Cheng, Xiang Bai
3D object detection is an essential task for achieving autonomous driving.
Existing anchor-based detection methods rely on empirical heuristics setting of
anchors, which makes the algorithms lack elegance. In recent years, we have
witnessed the rise of several generative models, among which diffusion models
show great potential for learning the transformation of two distributions. Our
proposed Diff3Det migrates the diffusion model to propos... more
3D object detection is an essential task for achieving autonomous driving.
Existing anchor-based detection methods rely on empirical heuristics setting of
anchors, which makes the algorithms lack elegance. In recent years, we have
witnessed the rise of several generative models, among which diffusion models
show great potential for learning the transformation of two distributions. Our
proposed Diff3Det migrates the diffusion model to proposal generation for 3D
object detection by considering the detection boxes as generative targets.
During training, the object boxes diffuse from the ground truth boxes to the
Gaussian distribution, and the decoder learns to reverse this noise process. In
the inference stage, the model progressively refines a set of random boxes to
the prediction results. We provide detailed experiments on the KITTI benchmark
and achieve promising performance compared to classical anchor-based 3D
detection methods.
less
By: Roman Klinger
The term emotion analysis in text subsumes various natural language
processing tasks which have in common the goal to enable computers to
understand emotions. Most popular is emotion classification in which one or
multiple emotions are assigned to a predefined textual unit. While such setting
is appropriate to identify the reader's or author's emotion, emotion role
labeling adds the perspective of mentioned entities and extracts text spans
... more
The term emotion analysis in text subsumes various natural language
processing tasks which have in common the goal to enable computers to
understand emotions. Most popular is emotion classification in which one or
multiple emotions are assigned to a predefined textual unit. While such setting
is appropriate to identify the reader's or author's emotion, emotion role
labeling adds the perspective of mentioned entities and extracts text spans
that correspond to the emotion cause. The underlying emotion theories agree on
one important point; that an emotion is caused by some internal or external
event and comprises several subcomponents, including the subjective feeling and
a cognitive evaluation. We therefore argue that emotions and events are related
in two ways. (1) Emotions are events; and this perspective is the fundament in
NLP for emotion role labeling. (2) Emotions are caused by events; a perspective
that is made explicit with research how to incorporate psychological appraisal
theories in NLP models to interpret events. These two research directions, role
labeling and (event-focused) emotion classification, have by and large been
tackled separately. We contributed to both directions with the projects SEAT
(Structured Multi-Domain Emotion Analysis from Text) and CEAT (Computational
Event Evaluation based on Appraisal Theories for Emotion Analysis), both funded
by the German Research Foundation. In this paper, we consolidate the findings
and point out open research questions.
less
By: Tiezheng Yu, Ziwei Ji, Pascale Fung
Query-Focused Meeting Summarization (QFMS) aims to generate a summary of a
given meeting transcript conditioned upon a query. The main challenges for QFMS
are the long input text length and sparse query-relevant information in the
meeting transcript. In this paper, we propose a knowledge-enhanced two-stage
framework called Knowledge-Aware Summarizer (KAS) to tackle the challenges. In
the first stage, we introduce knowledge-aware scores to i... more
Query-Focused Meeting Summarization (QFMS) aims to generate a summary of a
given meeting transcript conditioned upon a query. The main challenges for QFMS
are the long input text length and sparse query-relevant information in the
meeting transcript. In this paper, we propose a knowledge-enhanced two-stage
framework called Knowledge-Aware Summarizer (KAS) to tackle the challenges. In
the first stage, we introduce knowledge-aware scores to improve the
query-relevant segment extraction. In the second stage, we incorporate
query-relevant knowledge in the summary generation. Experimental results on the
QMSum dataset show that our approach achieves state-of-the-art performance.
Further analysis proves the competency of our methods in generating relevant
and faithful summaries.
less
By: Peiying Wang, Sunlu Zeng, Junqing Chen, Lu Fan, Meng Chen, Youzheng Wu, Xiaodong He
Multimodal emotion recognition (MER) aims to detect the emotional status of a
given expression by combining the speech and text information. Intuitively,
label information should be capable of helping the model locate the salient
tokens/frames relevant to the specific emotion, which finally facilitates the
MER task. Inspired by this, we propose a novel approach for MER by leveraging
label information. Specifically, we first obtain the repre... more
Multimodal emotion recognition (MER) aims to detect the emotional status of a
given expression by combining the speech and text information. Intuitively,
label information should be capable of helping the model locate the salient
tokens/frames relevant to the specific emotion, which finally facilitates the
MER task. Inspired by this, we propose a novel approach for MER by leveraging
label information. Specifically, we first obtain the representative label
embeddings for both text and speech modalities, then learn the label-enhanced
text/speech representations for each utterance via label-token and label-frame
interactions. Finally, we devise a novel label-guided attentive fusion module
to fuse the label-aware text and speech representations for emotion
classification. Extensive experiments were conducted on the public IEMOCAP
dataset, and experimental results demonstrate that our proposed approach
outperforms existing baselines and achieves new state-of-the-art performance.
less
By: Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, Zhifang Sui
Reasoning is a cognitive process of using evidence to reach a sound
conclusion. The reasoning capability is essential for large language models
(LLMs) to serve as the brain of the artificial general intelligence agent.
Recent studies reveal that fine-tuning LLMs on data with the chain of thought
(COT) reasoning process can significantly enhance their reasoning capabilities.
However, we find that the fine-tuned LLMs suffer from an \textit{As... more
Reasoning is a cognitive process of using evidence to reach a sound
conclusion. The reasoning capability is essential for large language models
(LLMs) to serve as the brain of the artificial general intelligence agent.
Recent studies reveal that fine-tuning LLMs on data with the chain of thought
(COT) reasoning process can significantly enhance their reasoning capabilities.
However, we find that the fine-tuned LLMs suffer from an \textit{Assessment
Misalignment} problem, i.e., they frequently assign higher scores to subpar
COTs, leading to potential limitations in their reasoning abilities. To address
this problem, we introduce an \textit{Alignment Fine-Tuning (AFT)} paradigm,
which involves three steps: 1) fine-tuning LLMs with COT training data; 2)
generating multiple COT responses for each question, and categorizing them into
positive and negative ones based on whether they achieve the correct answer; 3)
calibrating the scores of positive and negative responses given by LLMs with a
novel constraint alignment loss. Specifically, the constraint alignment loss
has two objectives: a) Alignment, which guarantees that positive scores surpass
negative scores to encourage answers with high-quality COTs; b) Constraint,
which keeps the negative scores confined to a reasonable range to prevent the
model degradation. Beyond just the binary positive and negative feedback, the
constraint alignment loss can be seamlessly adapted to the ranking situations
when ranking feedback is accessible. Furthermore, we also delve deeply into
recent ranking-based alignment methods, such as DPO, RRHF, and PRO, and
discover that the constraint, which has been overlooked by these approaches, is
also crucial for their performance. Extensive experiments on four reasoning
benchmarks with both binary and ranking feedback demonstrate the effectiveness
of AFT.
less
By: Patrick Eickhoff, Matthias Möller, Theresa Pekarek Rosin, Johannes Twiefel, Stefan Wermter
In recent research, in the domain of speech processing, large End-to-End
(E2E) systems for Automatic Speech Recognition (ASR) have reported
state-of-the-art performance on various benchmarks. These systems intrinsically
learn how to handle and remove noise conditions from speech. Previous research
has shown, that it is possible to extract the denoising capabilities of these
models into a preprocessor network, which can be used as a frontend... more
In recent research, in the domain of speech processing, large End-to-End
(E2E) systems for Automatic Speech Recognition (ASR) have reported
state-of-the-art performance on various benchmarks. These systems intrinsically
learn how to handle and remove noise conditions from speech. Previous research
has shown, that it is possible to extract the denoising capabilities of these
models into a preprocessor network, which can be used as a frontend for
downstream ASR models. However, the proposed methods were limited to specific
fully convolutional architectures. In this work, we propose a novel method to
extract the denoising capabilities, that can be applied to any encoder-decoder
architecture. We propose the Cleancoder preprocessor architecture that extracts
hidden activations from the Conformer ASR model and feeds them to a decoder to
predict denoised spectrograms. We train our pre-processor on the Noisy Speech
Database (NSD) to reconstruct denoised spectrograms from noisy inputs. Then, we
evaluate our model as a frontend to a pretrained Conformer ASR model as well as
a frontend to train smaller Conformer ASR models from scratch. We show that the
Cleancoder is able to filter noise from speech and that it improves the total
Word Error Rate (WER) of the downstream model in noisy conditions for both
applications.
less
By: Abul Hasan, Mark Levene, David Weston
We investigate the potential benefit of incorporating dictionary information
into a neural network architecture for natural language processing. In
particular, we make use of this architecture to extract several concepts
related to COVID-19 from an on-line medical forum. We use a sample from the
forum to manually curate one dictionary for each concept. In addition, we use
MetaMap, which is a tool for extracting biomedical concepts, to ident... more
We investigate the potential benefit of incorporating dictionary information
into a neural network architecture for natural language processing. In
particular, we make use of this architecture to extract several concepts
related to COVID-19 from an on-line medical forum. We use a sample from the
forum to manually curate one dictionary for each concept. In addition, we use
MetaMap, which is a tool for extracting biomedical concepts, to identify a
small number of semantic concepts. For a supervised concept extraction task on
the forum data, our best model achieved a macro $F_1$ score of 90\%. A major
difficulty in medical concept extraction is obtaining labelled data from which
to build supervised models. We investigate the utility of our models to
transfer to data derived from a different source in two ways. First for
producing labels via weak learning and second to perform concept extraction.
The dataset we use in this case comprises COVID-19 related tweets and we
achieve an $F_1$ score 81\% for symptom concept extraction trained on weakly
labelled data. The utility of our dictionaries is compared with a COVID-19
symptom dictionary that was constructed directly from Twitter. Further
experiments that incorporate BERT and a COVID-19 version of BERTweet
demonstrate that the dictionaries provide a commensurate result. Our results
show that incorporating small domain dictionaries to deep learning models can
improve concept extraction tasks. Moreover, models built using dictionaries
generalize well and are transferable to different datasets on a similar task.
less
By: Helena Bonaldi, Giuseppe Attanasio, Debora Nozza, Marco Guerini
Recent computational approaches for combating online hate speech involve the
automatic generation of counter narratives by adapting Pretrained
Transformer-based Language Models (PLMs) with human-curated data. This process,
however, can produce in-domain overfitting, resulting in models generating
acceptable narratives only for hatred similar to training data, with little
portability to other targets or to real-world toxic language. This pap... more
Recent computational approaches for combating online hate speech involve the
automatic generation of counter narratives by adapting Pretrained
Transformer-based Language Models (PLMs) with human-curated data. This process,
however, can produce in-domain overfitting, resulting in models generating
acceptable narratives only for hatred similar to training data, with little
portability to other targets or to real-world toxic language. This paper
introduces novel attention regularization methodologies to improve the
generalization capabilities of PLMs for counter narratives generation.
Overfitting to training-specific terms is then discouraged, resulting in more
diverse and richer narratives. We experiment with two attention-based
regularization techniques on a benchmark English dataset. Regularized models
produce better counter narratives than state-of-the-art approaches in most
cases, both in terms of automatic metrics and human evaluation, especially when
hateful targets are not present in the training data. This work paves the way
for better and more flexible counter-speech generation models, a task for which
datasets are highly challenging to produce.
less
nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resources
0upvotes
By: Piotr Nawrot
State-of-the-art language models like T5 have revolutionized the NLP
landscape, but their computational demands hinder a large portion of the
research community. To address this challenge, we present nanoT5, a
specially-optimized PyTorch framework for efficient pre-training and
fine-tuning of T5 models. Drawing on insights from optimizer differences and
prioritizing efficiency, nanoT5 allows a T5-Base model to be pre-trained on a
single GPU... more
State-of-the-art language models like T5 have revolutionized the NLP
landscape, but their computational demands hinder a large portion of the
research community. To address this challenge, we present nanoT5, a
specially-optimized PyTorch framework for efficient pre-training and
fine-tuning of T5 models. Drawing on insights from optimizer differences and
prioritizing efficiency, nanoT5 allows a T5-Base model to be pre-trained on a
single GPU in just 16 hours, without any loss in performance. With the
introduction of this open-source framework, we hope to widen the accessibility
to language modelling research and cater to the community's demand for more
user-friendly T5 (Encoder-Decoder) implementations. Our contributions,
including configurations, codebase, software/hardware insights, and pre-trained
models, are available to the public, aiming to strike a balance between
research accessibility and resource constraints in NLP.
less
Advancing Text-to-GLOSS Neural Translation Using a Novel Hyper-parameter Optimization Technique
0upvotes
By: Younes Ouargani, Noussaima El Khattabi
In this paper, we investigate the use of transformers for Neural Machine
Translation of text-to-GLOSS for Deaf and Hard-of-Hearing communication. Due to
the scarcity of available data and limited resources for text-to-GLOSS
translation, we treat the problem as a low-resource language task. We use our
novel hyper-parameter exploration technique to explore a variety of
architectural parameters and build an optimal transformer-based architectu... more
In this paper, we investigate the use of transformers for Neural Machine
Translation of text-to-GLOSS for Deaf and Hard-of-Hearing communication. Due to
the scarcity of available data and limited resources for text-to-GLOSS
translation, we treat the problem as a low-resource language task. We use our
novel hyper-parameter exploration technique to explore a variety of
architectural parameters and build an optimal transformer-based architecture
specifically tailored for text-to-GLOSS translation. The study aims to improve
the accuracy and fluency of Neural Machine Translation generated GLOSS. This is
achieved by examining various architectural parameters including layer count,
attention heads, embedding dimension, dropout, and label smoothing to identify
the optimal architecture for improving text-to-GLOSS translation performance.
The experiments conducted on the PHOENIX14T dataset reveal that the optimal
transformer architecture outperforms previous work on the same dataset. The
best model reaches a ROUGE (Recall-Oriented Understudy for Gisting Evaluation)
score of 55.18% and a BLEU-1 (BiLingual Evaluation Understudy 1) score of
63.6%, outperforming state-of-the-art results on the BLEU1 and ROUGE score by
8.42 and 0.63 respectively.
less
By: Huimin Wang, Wai-Chung Kwan, Kam-Fai Wong
Recent works usually address Dialog policy learning DPL by training a
reinforcement learning (RL) agent to determine the best dialog action. However,
existing works on deep RL require a large volume of agent-user interactions to
achieve acceptable performance. In this paper, we propose to make full use of
the plain text knowledge from the pre-trained language model to accelerate the
RL agent's learning speed. Specifically, we design a dialo... more
Recent works usually address Dialog policy learning DPL by training a
reinforcement learning (RL) agent to determine the best dialog action. However,
existing works on deep RL require a large volume of agent-user interactions to
achieve acceptable performance. In this paper, we propose to make full use of
the plain text knowledge from the pre-trained language model to accelerate the
RL agent's learning speed. Specifically, we design a dialog action-aware
transformer encoder (DaTrans), which integrates a new fine-tuning procedure
named masked last action task to encourag