By: Yue Xiang, Bo Jiang, Haifeng Dai
The degradation process of lithium-ion batteries is intricately linked to
their entire lifecycle as power sources and energy storage devices,
encompassing aspects such as performance delivery and cycling utilization.
Consequently, the accurate and expedient estimation or prediction of the aging
state of lithium-ion batteries has garnered extensive attention. Nonetheless,
prevailing research predominantly concentrates on either aging estimat... more
The degradation process of lithium-ion batteries is intricately linked to
their entire lifecycle as power sources and energy storage devices,
encompassing aspects such as performance delivery and cycling utilization.
Consequently, the accurate and expedient estimation or prediction of the aging
state of lithium-ion batteries has garnered extensive attention. Nonetheless,
prevailing research predominantly concentrates on either aging estimation or
prediction, neglecting the dynamic fusion of both facets. This paper proposes a
hybrid model for capacity aging estimation and prediction based on deep
learning, wherein salient features highly pertinent to aging are extracted from
charge and discharge relaxation processes. By amalgamating historical capacity
decay data, the model dynamically furnishes estimations of the present capacity
and forecasts of future capacity for lithium-ion batteries. Our approach is
validated against a novel dataset involving charge and discharge cycles at
varying rates. Specifically, under a charging condition of 0.25C, a mean
absolute percentage error (MAPE) of 0.29% is achieved. This outcome underscores
the model's adeptness in harnessing relaxation processes commonly encountered
in the real world and synergizing with historical capacity records within
battery management systems (BMS), thereby affording estimations and
prognostications of capacity decline with heightened precision.
less
By: Bhaskar Dhariyal, Thach Le Nguyen, Georgiana Ifrim
The state-of-the-art in time series classification has come a long way, from
the 1NN-DTW algorithm to the ROCKET family of classifiers. However, in the
current fast-paced development of new classifiers, taking a step back and
performing simple baseline checks is essential. These checks are often
overlooked, as researchers are focused on establishing new state-of-the-art
results, developing scalable algorithms, and making models explainable.... more
The state-of-the-art in time series classification has come a long way, from
the 1NN-DTW algorithm to the ROCKET family of classifiers. However, in the
current fast-paced development of new classifiers, taking a step back and
performing simple baseline checks is essential. These checks are often
overlooked, as researchers are focused on establishing new state-of-the-art
results, developing scalable algorithms, and making models explainable.
Nevertheless, there are many datasets that look like time series at first
glance, but classic algorithms such as tabular methods with no time ordering
may perform better on such problems. For example, for spectroscopy datasets,
tabular methods tend to significantly outperform recent time series methods. In
this study, we compare the performance of tabular models using classic machine
learning approaches (e.g., Ridge, LDA, RandomForest) with the ROCKET family of
classifiers (e.g., Rocket, MiniRocket, MultiRocket). Tabular models are simple
and very efficient, while the ROCKET family of classifiers are more complex and
have state-of-the-art accuracy and efficiency among recent time series
classifiers. We find that tabular models outperform the ROCKET family of
classifiers on approximately 19% of univariate and 28% of multivariate datasets
in the UCR/UEA benchmark and achieve accuracy within 10 percentage points on
about 50% of datasets. Our results suggest that it is important to consider
simple tabular models as baselines when developing time series classifiers.
These models are very fast, can be as effective as more complex methods and may
be easier to understand and deploy.
less
By: Felicia Ruppel, Florian Faion, Claudius Gläser, Klaus Dietmayer
Group regression is commonly used in 3D object detection to predict box
parameters of similar classes in a joint head, aiming to benefit from
similarities while separating highly dissimilar classes. For query-based
perception methods, this has, so far, not been feasible. We close this gap and
present a method to incorporate multi-class group regression, especially
designed for the 3D domain in the context of autonomous driving, into existin... more
Group regression is commonly used in 3D object detection to predict box
parameters of similar classes in a joint head, aiming to benefit from
similarities while separating highly dissimilar classes. For query-based
perception methods, this has, so far, not been feasible. We close this gap and
present a method to incorporate multi-class group regression, especially
designed for the 3D domain in the context of autonomous driving, into existing
attention and query-based perception approaches. We enhance a transformer based
joint object detection and tracking model with this approach, and thoroughly
evaluate its behavior and performance. For group regression, the classes of the
nuScenes dataset are divided into six groups of similar shape and prevalence,
each being regressed by a dedicated head. We show that the proposed method is
applicable to many existing transformer based perception approaches and can
bring potential benefits. The behavior of query group regression is thoroughly
analyzed in comparison to a unified regression head, e.g. in terms of
class-switching behavior and distribution of the output parameters. The
proposed method offers many possibilities for further research, such as in the
direction of deep multi-hypotheses tracking.
less
Fast Feedforward Networks
0upvotes
By: Peter Belcak, Roger Wattenhofer
We break the linear link between the layer size and its inference cost by
introducing the fast feedforward (FFF) architecture, a logarithmic-time
alternative to feedforward networks.
We show that FFFs give comparable performance to feedforward networks at an
exponential fraction of their inference cost, are quicker to deliver
performance compared to mixture-of-expert networks, and can readily take the
place of either in transformers.
Pu... more
We break the linear link between the layer size and its inference cost by
introducing the fast feedforward (FFF) architecture, a logarithmic-time
alternative to feedforward networks.
We show that FFFs give comparable performance to feedforward networks at an
exponential fraction of their inference cost, are quicker to deliver
performance compared to mixture-of-expert networks, and can readily take the
place of either in transformers.
Pushing FFFs to the absolute limit, we train a vision transformer to perform
single-neuron inferences at the cost of only 5.8% performance decrease against
the full-width variant.
Our implementation is available as a Python package; just use "pip install
fastfeedforward".
less
WePaMaDM-Outlier Detection: Weighted Outlier Detection using Pattern Approaches for Mass Data Mining
3upvotes
By: Ravindrakumar Purohit, Jai Prakash Verma, Rachna Jain, Madhuri Bhavsar
Weighted Outlier Detection is a method for identifying unusual or anomalous
data points in a dataset, which can be caused by various factors like human
error, fraud, or equipment malfunctions. Detecting outliers can reveal vital
information about system faults, fraudulent activities, and patterns in the
data, assisting experts in addressing the root causes of these anomalies.
However,creating a model of normal data patterns to identify outl... more
Weighted Outlier Detection is a method for identifying unusual or anomalous
data points in a dataset, which can be caused by various factors like human
error, fraud, or equipment malfunctions. Detecting outliers can reveal vital
information about system faults, fraudulent activities, and patterns in the
data, assisting experts in addressing the root causes of these anomalies.
However,creating a model of normal data patterns to identify outliers can be
challenging due to the nature of input data, labeled data availability, and
specific requirements of the problem. This article proposed the
WePaMaDM-Outlier Detection with distinct mass data mining domain, demonstrating
that such techniques are domain-dependent and usually developed for specific
problem formulations. Nevertheless, similar domains can adapt solutions with
modifications. This work also investigates the significance of data modeling in
outlier detection techniques in surveillance, fault detection, and trend
analysis, also referred to as novelty detection, a semisupervised task where
the algorithm learns to recognize abnormality while being taught the normal
class.
less
By: Haochong Xia, Shuo Sun, Xinrun Wang, Bo An
Financial simulators play an important role in enhancing forecasting
accuracy, managing risks, and fostering strategic financial decision-making.
Despite the development of financial market simulation methodologies, existing
frameworks often struggle with adapting to specialized simulation context. We
pinpoint the challenges as i) current financial datasets do not contain context
labels; ii) current techniques are not designed to generate f... more
Financial simulators play an important role in enhancing forecasting
accuracy, managing risks, and fostering strategic financial decision-making.
Despite the development of financial market simulation methodologies, existing
frameworks often struggle with adapting to specialized simulation context. We
pinpoint the challenges as i) current financial datasets do not contain context
labels; ii) current techniques are not designed to generate financial data with
context as control, which demands greater precision compared to other
modalities; iii) the inherent difficulties in generating context-aligned,
high-fidelity data given the non-stationary, noisy nature of financial data. To
address these challenges, our contributions are: i) we proposed the Contextual
Market Dataset with market dynamics, stock ticker, and history state as
context, leveraging a market dynamics modeling method that combines linear
regression and Dynamic Time Warping clustering to extract market dynamics; ii)
we present Market-GAN, a novel architecture incorporating a Generative
Adversarial Networks (GAN) for the controllable generation with context, an
autoencoder for learning low-dimension features, and supervisors for knowledge
transfer; iii) we introduce a two-stage training scheme to ensure that
Market-GAN captures the intrinsic market distribution with multiple objectives.
In the pertaining stage, with the use of the autoencoder and supervisors, we
prepare the generator with a better initialization for the adversarial training
stage. We propose a set of holistic evaluation metrics that consider alignment,
fidelity, data usability on downstream tasks, and market facts. We evaluate
Market-GAN with the Dow Jones Industrial Average data from 2000 to 2023 and
showcase superior performance in comparison to 4 state-of-the-art time-series
generative models.
less
By: Lekang Jiang, Caiqi Zhang, Farimah Poursafaei, Shenyang Huang
Recently, Graph Neural Networks (GNNs) have shown promising performance in
tasks on dynamic graphs such as node classification, link prediction and graph
regression. However, few work has studied the temporal edge regression task
which has important real-world applications. In this paper, we explore the
application of GNNs to edge regression tasks in both static and dynamic
settings, focusing on predicting food and agriculture trade values ... more
Recently, Graph Neural Networks (GNNs) have shown promising performance in
tasks on dynamic graphs such as node classification, link prediction and graph
regression. However, few work has studied the temporal edge regression task
which has important real-world applications. In this paper, we explore the
application of GNNs to edge regression tasks in both static and dynamic
settings, focusing on predicting food and agriculture trade values between
nations. We introduce three simple yet strong baselines and comprehensively
evaluate one static and three dynamic GNN models using the UN Trade dataset.
Our experimental results reveal that the baselines exhibit remarkably strong
performance across various settings, highlighting the inadequacy of existing
GNNs. We also find that TGN outperforms other GNN models, suggesting TGN is a
more appropriate choice for edge regression tasks. Moreover, we note that the
proportion of negative edges in the training samples significantly affects the
test performance. The companion source code can be found at:
https://github.com/scylj1/GNN_Edge_Regression.
less
By: Hong Zhu, Runpeng Yu, Xing Tang, Yifei Wang, Yuan Fang, Yisen Wang
Data in the real-world classification problems are always imbalanced or
long-tailed, wherein the majority classes have the most of the samples that
dominate the model training. In such setting, the naive model tends to have
poor performance on the minority classes. Previously, a variety of loss
modifications have been proposed to address the long-tailed leaning problem,
while these methods either treat the samples in the same class
indiscri... more
Data in the real-world classification problems are always imbalanced or
long-tailed, wherein the majority classes have the most of the samples that
dominate the model training. In such setting, the naive model tends to have
poor performance on the minority classes. Previously, a variety of loss
modifications have been proposed to address the long-tailed leaning problem,
while these methods either treat the samples in the same class
indiscriminatingly or lack a theoretical guarantee. In this paper, we propose
two novel approaches based on CVaR (Conditional Value at Risk) to improve the
performance of long-tailed learning with a solid theoretical ground.
Specifically, we firstly introduce a Label-Aware Bounded CVaR (LAB-CVaR) loss
to overcome the pessimistic result of the original CVaR, and further design the
optimal weight bounds for LAB-CVaR theoretically. Based on LAB-CVaR, we
additionally propose a LAB-CVaR with logit adjustment (LAB-CVaR-logit) loss to
stabilize the optimization process, where we also offer the theoretical
support. Extensive experiments on real-world datasets with long-tailed label
distributions verify the superiority of our proposed methods.
less
Integrating LLMs and Decision Transformers for Language Grounded Generative Quality-Diversity
0upvotes
By: Achkan Salehi, Stephane Doncieux
Quality-Diversity is a branch of stochastic optimization that is often
applied to problems from the Reinforcement Learning and control domains in
order to construct repertoires of well-performing policies/skills that exhibit
diversity with respect to a behavior space. Such archives are usually composed
of a finite number of reactive agents which are each associated to a unique
behavior descriptor, and instantiating behavior descriptors outs... more
Quality-Diversity is a branch of stochastic optimization that is often
applied to problems from the Reinforcement Learning and control domains in
order to construct repertoires of well-performing policies/skills that exhibit
diversity with respect to a behavior space. Such archives are usually composed
of a finite number of reactive agents which are each associated to a unique
behavior descriptor, and instantiating behavior descriptors outside of that
coarsely discretized space is not straight-forward. While a few recent works
suggest solutions to that issue, the trajectory that is generated is not easily
customizable beyond the specification of a target behavior descriptor. We
propose to jointly solve those problems in environments where semantic
information about static scene elements is available by leveraging a Large
Language Model to augment the repertoire with natural language descriptions of
trajectories, and training a policy conditioned on those descriptions. Thus,
our method allows a user to not only specify an arbitrary target behavior
descriptor, but also provide the model with a high-level textual prompt to
shape the generated trajectory. We also propose an LLM-based approach to
evaluating the performance of such generative agents. Furthermore, we develop a
benchmark based on simulated robot navigation in a 2d maze that we use for
experimental validation.
less
By: Xuyang Zhong, Chen Liu
Dataset distillation methods have demonstrated remarkable performance for
neural networks trained with very limited training data. However, a significant
challenge arises in the form of architecture overfitting: the distilled
training data synthesized by a specific network architecture (i.e., training
network) generates poor performance when trained by other network architectures
(i.e., test networks). This paper addresses this issue and pr... more
Dataset distillation methods have demonstrated remarkable performance for
neural networks trained with very limited training data. However, a significant
challenge arises in the form of architecture overfitting: the distilled
training data synthesized by a specific network architecture (i.e., training
network) generates poor performance when trained by other network architectures
(i.e., test networks). This paper addresses this issue and proposes a series of
approaches in both architecture designs and training schemes which can be
adopted together to boost the generalization performance across different
network architectures on the distilled training data. We conduct extensive
experiments to demonstrate the effectiveness and generality of our methods.
Particularly, across various scenarios involving different sizes of distilled
data, our approaches achieve comparable or superior performance to existing
methods when training on the distilled data using networks with larger
capacities.
less
By: I. J. Arnquist, F. T. Avignone III, A. S. Barabash, C. J. Barton, K. H. Bhimani, E. Blalock, B. Bos, M. Busch, M. Buuck, T. S. Caldwell, Y. -D. Chan, C. D. Christofferson, P. -H. Chu, M. L. Clark, C. Cuesta, J. A. Detwiler, Yu. Efremenko, H. Ejiri, S. R. Elliott, N. Fuad, G. K. Giovanetti, M. P. Green, J. Gruszko, I. S. Guinn, V. E. Guiseppe, C. R. Haufe, R. Henning, D. Hervas Aguilar, E. W. Hoppe, A. Hostiuc, M. F. Kidd, I. Kim, R. T. Kouzes, T. E. Lannen V, A. Li, J. M. Lopez-Castano, R. D. Martin, R. Massarczyk, S. J. Meijer, S. Mertens, T. K. Oli, L. S. Paudel, W. Pettus, A. W. P. Poon, B. Quenallata, D. C. Radford, A. L. Reine, K. Rielage, N. W. Ruof, D. C. Schaper, S. J. Schleich, D. Tedeschi, R. L. Varner, S. Vasilyev, S. L. Watkins, J. F. Wilkerson, C. Wiseman, W. Xu, C. -H. Yu, B. X. Zhu
The enclosed data release consists of a subset of the calibration data from
the Majorana Demonstrator experiment. Each Majorana event is accompanied by raw
Germanium detector waveforms, pulse shape discrimination cuts, and calibrated
final energies, all shared in an HDF5 file format along with relevant metadata.
This release is specifically designed to support the training and testing of
Artificial Intelligence (AI) and Machine Learning (ML... more
The enclosed data release consists of a subset of the calibration data from
the Majorana Demonstrator experiment. Each Majorana event is accompanied by raw
Germanium detector waveforms, pulse shape discrimination cuts, and calibrated
final energies, all shared in an HDF5 file format along with relevant metadata.
This release is specifically designed to support the training and testing of
Artificial Intelligence (AI) and Machine Learning (ML) algorithms upon our
data. This document is structured as follows. Section I provides an overview of
the dataset's content and format; Section II outlines the location of this
dataset and the method for accessing it; Section III presents the NPML Machine
Learning Challenge associated with this dataset; Section IV contains a
disclaimer from the Majorana collaboration regarding the use of this dataset;
Appendix A contains technical details of this data release. Please direct
questions about the material provided within this release to [email protected]
(A. Li).
less
By: Niklas Stoehr, Pengxiang Cheng, Jing Wang, Daniel Preotiuc-Pietro, Rajarshi Bhowmik
Language models contain ranking-based knowledge and are powerful solvers of
in-context ranking tasks. For instance, they may have parametric knowledge
about the ordering of countries by size or may be able to rank reviews by
sentiment. Recent work focuses on pairwise, pointwise, and listwise prompting
techniques to elicit a language model's ranking knowledge. However, we find
that even with careful calibration and constrained decoding, prom... more
Language models contain ranking-based knowledge and are powerful solvers of
in-context ranking tasks. For instance, they may have parametric knowledge
about the ordering of countries by size or may be able to rank reviews by
sentiment. Recent work focuses on pairwise, pointwise, and listwise prompting
techniques to elicit a language model's ranking knowledge. However, we find
that even with careful calibration and constrained decoding, prompting-based
techniques may not always be self-consistent in the rankings they produce. This
motivates us to explore an alternative approach that is inspired by an
unsupervised probing method called Contrast-Consistent Search (CCS). The idea
is to train a probing model guided by a logical constraint: a model's
representation of a statement and its negation must be mapped to contrastive
true-false poles consistently across multiple statements. We hypothesize that
similar constraints apply to ranking tasks where all items are related via
consistent pairwise or listwise comparisons. To this end, we extend the binary
CCS method to Contrast-Consistent Ranking (CCR) by adapting existing ranking
methods such as the Max-Margin Loss, Triplet Loss, and Ordinal Regression
objective. Our results confirm that, for the same language model, CCR probing
outperforms prompting and even performs on a par with prompting much larger
language models.
less
By: Xin Zhou, Ling Chen, Houming Wu
As the size of models and datasets grows, it has become increasingly common
to train models in parallel. However, existing distributed stochastic gradient
descent (SGD) algorithms suffer from insufficient utilization of computational
resources and poor convergence in heterogeneous clusters. In this paper, we
propose a delayed synchronous SGD algorithm with adaptive batch size (ABS-SGD)
for heterogeneous GPU clusters. In ABS-SGD, workers per... more
As the size of models and datasets grows, it has become increasingly common
to train models in parallel. However, existing distributed stochastic gradient
descent (SGD) algorithms suffer from insufficient utilization of computational
resources and poor convergence in heterogeneous clusters. In this paper, we
propose a delayed synchronous SGD algorithm with adaptive batch size (ABS-SGD)
for heterogeneous GPU clusters. In ABS-SGD, workers perform global
synchronization to accumulate delayed gradients and use the accumulated delayed
gradients to update parameters. While workers are performing global
synchronization for delayed gradients, they perform the computation of the next
batch without specifying batch size in advance, which lasts until the next
global synchronization starts, realizing the full utilization of computational
resources. Since the gradient delay is only one iteration, the stale gradient
problem can be alleviated. We theoretically prove the convergence of ABS-SGD in
heterogeneous clusters. Extensive experiments in three types of heterogeneous
clusters demonstrate that ABS-SGD can make full use of computational resources
and accelerate model convergence: When training ResNet18 network with 4
workers, ABS-SGD increases the convergence speed by 1.30x on average compared
with the best baseline algorithm.
less
By: Rishabh Agrawal, Nathan Dahlin, Rahul Jain, Ashutosh Nayyar
Imitation Learning (IL) is an important paradigm within the broader
reinforcement learning (RL) methodology. Unlike most of RL, it does not assume
availability of reward-feedback. Reward inference and shaping are known to be
difficult and error-prone methods particularly when the demonstration data
comes from human experts. Classical methods such as behavioral cloning and
inverse reinforcement learning are highly sensitive to estimation err... more
Imitation Learning (IL) is an important paradigm within the broader
reinforcement learning (RL) methodology. Unlike most of RL, it does not assume
availability of reward-feedback. Reward inference and shaping are known to be
difficult and error-prone methods particularly when the demonstration data
comes from human experts. Classical methods such as behavioral cloning and
inverse reinforcement learning are highly sensitive to estimation errors, a
problem that is particularly acute in continuous state space problems.
Meanwhile, state-of-the-art IL algorithms convert behavioral policy learning
problems into distribution-matching problems which often require additional
online interaction data to be effective. In this paper, we consider the problem
of imitation learning in continuous state space environments based solely on
observed behavior, without access to transition dynamics information, reward
structure, or, most importantly, any additional interactions with the
environment. Our approach is based on the Markov balance equation and
introduces a novel conditional kernel density estimation-based imitation
learning framework. It involves estimating the environment's transition
dynamics using conditional kernel density estimators and seeks to satisfy the
probabilistic balance equations for the environment. We establish that our
estimators satisfy basic asymptotic consistency requirements. Through a series
of numerical experiments on continuous state benchmark environments, we show
consistently superior empirical performance over many state-of-the-art IL
algorithms.
less
By: Minjie Chen, Yao Cheng, Ye Wang, Xiang Li, Ming Gao
Graph contrastive learning (GCL) has recently emerged as a promising approach
for graph representation learning. Some existing methods adopt the 1-vs-K
scheme to construct one positive and K negative samples for each graph, but it
is difficult to set K. For those methods that do not use negative samples, it
is often necessary to add additional strategies to avoid model collapse, which
could only alleviate the problem to some extent. All the... more
Graph contrastive learning (GCL) has recently emerged as a promising approach
for graph representation learning. Some existing methods adopt the 1-vs-K
scheme to construct one positive and K negative samples for each graph, but it
is difficult to set K. For those methods that do not use negative samples, it
is often necessary to add additional strategies to avoid model collapse, which
could only alleviate the problem to some extent. All these drawbacks will
undoubtedly have an adverse impact on the generalizability and efficiency of
the model. In this paper, to address these issues, we propose a novel graph
self-contrast framework GraphSC, which only uses one positive and one negative
sample, and chooses triplet loss as the objective. Specifically, self-contrast
has two implications. First, GraphSC generates both positive and negative views
of a graph sample from the graph itself via graph augmentation functions of
various intensities, and use them for self-contrast. Second, GraphSC uses
Hilbert-Schmidt Independence Criterion (HSIC) to factorize the representations
into multiple factors and proposes a masked self-contrast mechanism to better
separate positive and negative samples. Further, Since the triplet loss only
optimizes the relative distance between the anchor and its positive/negative
samples, it is difficult to ensure the absolute distance between the anchor and
positive sample. Therefore, we explicitly reduced the absolute distance between
the anchor and positive sample to accelerate convergence. Finally, we conduct
extensive experiments to evaluate the performance of GraphSC against 19 other
state-of-the-art methods in both unsupervised and transfer learning settings.
less
By: Ryan-Rhys Griffiths
In many areas of the observational and experimental sciences data is scarce.
Data observation in high-energy astrophysics is disrupted by celestial
occlusions and limited telescope time while data derived from laboratory
experiments in synthetic chemistry and materials science is time and
cost-intensive to collect. On the other hand, knowledge about the
data-generation mechanism is often available in the sciences, such as the
measurement er... more
In many areas of the observational and experimental sciences data is scarce.
Data observation in high-energy astrophysics is disrupted by celestial
occlusions and limited telescope time while data derived from laboratory
experiments in synthetic chemistry and materials science is time and
cost-intensive to collect. On the other hand, knowledge about the
data-generation mechanism is often available in the sciences, such as the
measurement error of a piece of laboratory apparatus. Both characteristics,
small data and knowledge of the underlying physics, make Gaussian processes
(GPs) ideal candidates for fitting such datasets. GPs can make predictions with
consideration of uncertainty, for example in the virtual screening of molecules
and materials, and can also make inferences about incomplete data such as the
latent emission signature from a black hole accretion disc. Furthermore, GPs
are currently the workhorse model for Bayesian optimisation, a methodology
foreseen to be a guide for laboratory experiments in scientific discovery
campaigns. The first contribution of this thesis is to use GP modelling to
reason about the latent emission signature from the Seyfert galaxy Markarian
335, and by extension, to reason about the applicability of various theoretical
models of black hole accretion discs. The second contribution is to extend the
GP framework to molecular and chemical reaction representations and to provide
an open-source software library to enable the framework to be used by
scientists. The third contribution is to leverage GPs to discover novel and
performant photoswitch molecules. The fourth contribution is to introduce a
Bayesian optimisation scheme capable of modelling aleatoric uncertainty to
facilitate the identification of material compositions that possess intrinsic
robustness to large scale fabrication processes.
less
By: Euihyeon Choi, Jooyoung Kim, Wonkyung Lee
Probability estimation models play an important role in various fields, such
as weather forecasting, recommendation systems, and sports analysis. Among
several models estimating probabilities, it is difficult to evaluate which
model gives reliable probabilities since the ground-truth probabilities are not
available. The win probability estimation model for esports, which calculates
the win probability under a certain game state, is also one... more
Probability estimation models play an important role in various fields, such
as weather forecasting, recommendation systems, and sports analysis. Among
several models estimating probabilities, it is difficult to evaluate which
model gives reliable probabilities since the ground-truth probabilities are not
available. The win probability estimation model for esports, which calculates
the win probability under a certain game state, is also one of the fields being
actively studied in probability estimation. However, most of the previous works
evaluated their models using accuracy, a metric that only can measure the
performance of discrimination. In this work, we firstly investigate the Brier
score and the Expected Calibration Error (ECE) as a replacement of accuracy
used as a performance evaluation metric for win probability estimation models
in esports field. Based on the analysis, we propose a novel metric called
Balance score which is a simple yet effective metric in terms of six good
properties that probability estimation metric should have. Under the general
condition, we also found that the Balance score can be an effective
approximation of the true expected calibration error which has been imperfectly
approximated by ECE using the binning technique. Extensive evaluations using
simulation studies and real game snapshot data demonstrate the promising
potential to adopt the proposed metric not only for the win probability
estimation model for esports but also for evaluating general probability
estimation models.
less
By: Yi Wen, Siwei Wang, Ke Liang, Weixuan Liang, Xinhang Wan, Xinwang Liu, Suyuan Liu, Jiyuan Liu, En Zhu
The success of existing multi-view clustering (MVC) relies on the assumption
that all views are complete. However, samples are usually partially available
due to data corruption or sensor malfunction, which raises the research of
incomplete multi-view clustering (IMVC). Although several anchor-based IMVC
methods have been proposed to process the large-scale incomplete data, they
still suffer from the following drawbacks: i) Most existing ap... more
The success of existing multi-view clustering (MVC) relies on the assumption
that all views are complete. However, samples are usually partially available
due to data corruption or sensor malfunction, which raises the research of
incomplete multi-view clustering (IMVC). Although several anchor-based IMVC
methods have been proposed to process the large-scale incomplete data, they
still suffer from the following drawbacks: i) Most existing approaches neglect
the inter-view discrepancy and enforce cross-view representation to be
consistent, which would corrupt the representation capability of the model; ii)
Due to the samples disparity between different views, the learned anchor might
be misaligned, which we referred as the Anchor-Unaligned Problem for Incomplete
data (AUP-ID). Such the AUP-ID would cause inaccurate graph fusion and degrades
clustering performance. To tackle these issues, we propose a novel incomplete
anchor graph learning framework termed Scalable Incomplete Multi-View
Clustering with Structure Alignment (SIMVC-SA). Specially, we construct the
view-specific anchor graph to capture the complementary information from
different views. In order to solve the AUP-ID, we propose a novel structure
alignment module to refine the cross-view anchor correspondence. Meanwhile, the
anchor graph construction and alignment are jointly optimized in our unified
framework to enhance clustering quality. Through anchor graph construction
instead of full graphs, the time and space complexity of the proposed SIMVC-SA
is proven to be linearly correlated with the number of samples. Extensive
experiments on seven incomplete benchmark datasets demonstrate the
effectiveness and efficiency of our proposed method. Our code is publicly
available at https://github.com/wy1019/SIMVC-SA.
less
By: Tianchao Li, Yulong Pei
This paper is to introduce an asynchronous and local learning framework for
neural networks, named Modular Learning Framework (MOLE). This framework
modularizes neural networks by layers, defines the training objective via
mutual information for each module, and sequentially trains each module by
mutual information maximization. MOLE makes the training become local
optimization with gradient-isolated across modules, and this scheme is more
... more
This paper is to introduce an asynchronous and local learning framework for
neural networks, named Modular Learning Framework (MOLE). This framework
modularizes neural networks by layers, defines the training objective via
mutual information for each module, and sequentially trains each module by
mutual information maximization. MOLE makes the training become local
optimization with gradient-isolated across modules, and this scheme is more
biologically plausible than BP. We run experiments on vector-, grid- and
graph-type data. In particular, this framework is capable of solving both
graph- and node-level tasks for graph-type data. Therefore, MOLE has been
experimentally proven to be universally applicable to different types of data.
less
By: David Yunis, Justin Jung, Falcon Dai, Matthew Walter
Exploration in sparse-reward reinforcement learning is difficult due to the
requirement of long, coordinated sequences of actions in order to achieve any
reward. Moreover, in continuous action spaces there are an infinite number of
possible actions, which only increases the difficulty of exploration. One class
of methods designed to address these issues forms temporally extended actions,
often called skills, from interaction data collected ... more
Exploration in sparse-reward reinforcement learning is difficult due to the
requirement of long, coordinated sequences of actions in order to achieve any
reward. Moreover, in continuous action spaces there are an infinite number of
possible actions, which only increases the difficulty of exploration. One class
of methods designed to address these issues forms temporally extended actions,
often called skills, from interaction data collected in the same domain, and
optimizes a policy on top of this new action space. Typically such methods
require a lengthy pretraining phase, especially in continuous action spaces, in
order to form the skills before reinforcement learning can begin. Given prior
evidence that the full range of the continuous action space is not required in
such tasks, we propose a novel approach to skill-generation with two
components. First we discretize the action space through clustering, and second
we leverage a tokenization technique borrowed from natural language processing
to generate temporally extended actions. Such a method outperforms baselines
for skill-generation in several challenging sparse-reward domains, and requires
orders-of-magnitude less computation in skill-generation and online rollouts.
less
By: William Ahlberg, Alessandro Sestini, Konrad Tollmar, Linus Gisslén
Reinforcement learning has been widely successful in producing agents capable
of playing games at a human level. However, this requires complex reward
engineering, and the agent's resulting policy is often unpredictable. Going
beyond reinforcement learning is necessary to model a wide range of human
playstyles, which can be difficult to represent with a reward function. This
paper presents a novel imitation learning approach to generate mul... more
Reinforcement learning has been widely successful in producing agents capable
of playing games at a human level. However, this requires complex reward
engineering, and the agent's resulting policy is often unpredictable. Going
beyond reinforcement learning is necessary to model a wide range of human
playstyles, which can be difficult to represent with a reward function. This
paper presents a novel imitation learning approach to generate multiple persona
policies for playtesting. Multimodal Generative Adversarial Imitation Learning
(MultiGAIL) uses an auxiliary input parameter to learn distinct personas using
a single-agent model. MultiGAIL is based on generative adversarial imitation
learning and uses multiple discriminators as reward models, inferring the
environment reward by comparing the agent and distinct expert policies. The
reward from each discriminator is weighted according to the auxiliary input.
Our experimental analysis demonstrates the effectiveness of our technique in
two environments with continuous and discrete action spaces.
less
By: Mike Nguyen, Nicole Mücke
Random feature approximation is arguably one of the most popular techniques
to speed up kernel methods in large scale algorithms and provides a theoretical
approach to the analysis of deep neural networks. We analyze generalization
properties for a large class of spectral regularization methods combined with
random features, containing kernel methods with implicit regularization such as
gradient descent or explicit methods like Tikhonov reg... more
Random feature approximation is arguably one of the most popular techniques
to speed up kernel methods in large scale algorithms and provides a theoretical
approach to the analysis of deep neural networks. We analyze generalization
properties for a large class of spectral regularization methods combined with
random features, containing kernel methods with implicit regularization such as
gradient descent or explicit methods like Tikhonov regularization. For our
estimators we obtain optimal learning rates over regularity classes (even for
classes that are not included in the reproducing kernel Hilbert space), which
are defined through appropriate source conditions. This improves or completes
previous results obtained in related settings for specific kernel algorithms.
less
By: Sydney Pugh, Ivan Ruchkin, Insup Lee, James Weimer
Deep learning models have shown promising predictive accuracy for time-series
healthcare applications. However, ensuring the robustness of these models is
vital for building trustworthy AI systems. Existing research predominantly
focuses on robustness to synthetic adversarial examples, crafted by adding
imperceptible perturbations to clean input data. However, these synthetic
adversarial examples do not accurately reflect the most challengi... more
Deep learning models have shown promising predictive accuracy for time-series
healthcare applications. However, ensuring the robustness of these models is
vital for building trustworthy AI systems. Existing research predominantly
focuses on robustness to synthetic adversarial examples, crafted by adding
imperceptible perturbations to clean input data. However, these synthetic
adversarial examples do not accurately reflect the most challenging real-world
scenarios, especially in the context of healthcare data. Consequently,
robustness to synthetic adversarial examples may not necessarily translate to
robustness against naturally occurring adversarial examples, which is highly
desirable for trustworthy AI. We propose a method to curate datasets comprised
of natural adversarial examples to evaluate model robustness. The method relies
on probabilistic labels obtained from automated weakly-supervised labeling that
combines noisy and cheap-to-obtain labeling heuristics. Based on these labels,
our method adversarially orders the input data and uses this ordering to
construct a sequence of increasingly adversarial datasets. Our evaluation on
six medical case studies and three non-medical case studies demonstrates the
efficacy and statistical validity of our approach to generating naturally
adversarial datasets
less
By: Hinrikus Wolf, Luca Oeljeklaus, Pascal Kühner, Martin Grohe
Graph homomorphism counts, first explored by Lov\'asz in 1967, have recently
garnered interest as a powerful tool in graph-based machine learning. Grohe
(PODS 2020) proposed the theoretical foundations for using homomorphism counts
in machine learning on graph level as well as node level tasks. By their very
nature, these capture local structural information, which enables the creation
of robust structural embeddings. While a first approach... more
Graph homomorphism counts, first explored by Lov\'asz in 1967, have recently
garnered interest as a powerful tool in graph-based machine learning. Grohe
(PODS 2020) proposed the theoretical foundations for using homomorphism counts
in machine learning on graph level as well as node level tasks. By their very
nature, these capture local structural information, which enables the creation
of robust structural embeddings. While a first approach for graph level tasks
has been made by Nguyen and Maehara (ICML 2020), we experimentally show the
effectiveness of homomorphism count based node embeddings. Enriched with node
labels, node weights, and edge weights, these offer an interpretable
representation of graph data, allowing for enhanced explainability of machine
learning models.
We propose a theoretical framework for isomorphism-invariant homomorphism
count based embeddings which lend themselves to a wide variety of downstream
tasks. Our approach capitalises on the efficient computability of graph
homomorphism counts for bounded treewidth graph classes, rendering it a
practical solution for real-world applications. We demonstrate their
expressivity through experiments on benchmark datasets. Although our results do
not match the accuracy of state-of-the-art neural architectures, they are
comparable to other advanced graph learning models. Remarkably, our approach
demarcates itself by ensuring explainability for each individual feature. By
integrating interpretable machine learning algorithms like SVMs or Random
Forests, we establish a seamless, end-to-end explainable pipeline. Our study
contributes to the advancement of graph-based techniques that offer both
performance and interpretability.
less
By: Junming Yang, Xingguo Chen, Shengyuan Wang, Bolei Zhang
Model-based offline reinforcement learning (RL), which builds a supervised
transition model with logging dataset to avoid costly interactions with the
online environment, has been a promising approach for offline policy
optimization. As the discrepancy between the logging data and online
environment may result in a distributional shift problem, many prior works have
studied how to build robust transition models conservatively and estimate t... more
Model-based offline reinforcement learning (RL), which builds a supervised
transition model with logging dataset to avoid costly interactions with the
online environment, has been a promising approach for offline policy
optimization. As the discrepancy between the logging data and online
environment may result in a distributional shift problem, many prior works have
studied how to build robust transition models conservatively and estimate the
model uncertainty accurately. However, the over-conservatism can limit the
exploration of the agent, and the uncertainty estimates may be unreliable. In
this work, we propose a novel Model-based Offline policy optimization framework
with Adversarial Network (MOAN). The key idea is to use adversarial learning to
build a transition model with better generalization, where an adversary is
introduced to distinguish between in-distribution and out-of-distribution
samples. Moreover, the adversary can naturally provide a quantification of the
model's uncertainty with theoretical guarantees. Extensive experiments showed
that our approach outperforms existing state-of-the-art baselines on widely
studied offline RL benchmarks. It can also generate diverse in-distribution
samples, and quantify the uncertainty more accurately.
less
By: Wangkun Xu, Fei Teng
Data privacy and security have become a non-negligible factor in load
forecasting. Previous researches mainly focus on training stage enhancement.
However, once the model is trained and deployed, it may need to `forget' (i.e.,
remove the impact of) part of training data if the data is found to be
malicious or as requested by the data owner. This paper introduces machine
unlearning algorithm which is specifically designed to remove the influ... more
Data privacy and security have become a non-negligible factor in load
forecasting. Previous researches mainly focus on training stage enhancement.
However, once the model is trained and deployed, it may need to `forget' (i.e.,
remove the impact of) part of training data if the data is found to be
malicious or as requested by the data owner. This paper introduces machine
unlearning algorithm which is specifically designed to remove the influence of
part of the original dataset on an already trained forecaster. However, direct
unlearning inevitably degrades the model generalization ability. To balance
between unlearning completeness and performance degradation, a
performance-aware algorithm is proposed by evaluating the sensitivity of local
model parameter change using influence function and sample re-weighting.
Moreover, we observe that the statistic criterion cannot fully reflect the
operation cost of down-stream tasks. Therefore, a task-aware machine unlearning
is proposed whose objective is a tri-level optimization with dispatch and
redispatch problems considered. We theoretically prove the existence of the
gradient of such objective, which is key to re-weighting the remaining samples.
We test the unlearning algorithms on linear and neural network load forecasters
with realistic load dataset. The simulation demonstrates the balance on
unlearning completeness and operational cost. All codes can be found at
https://github.com/xuwkk/task_aware_machine_unlearning.
less
By: Hadar Schreiber Galler, Tom Zahavy, Guillaume Desjardins, Alon Cohen
We study diverse skill discovery in reward-free environments, aiming to
discover all possible skills in simple grid-world environments where prior
methods have struggled to succeed. This problem is formulated as mutual
training of skills using an intrinsic reward and a discriminator trained to
predict a skill given its trajectory. Our initial solution replaces the
standard one-vs-all (softmax) discriminator with a one-vs-one (all pairs)
dis... more
We study diverse skill discovery in reward-free environments, aiming to
discover all possible skills in simple grid-world environments where prior
methods have struggled to succeed. This problem is formulated as mutual
training of skills using an intrinsic reward and a discriminator trained to
predict a skill given its trajectory. Our initial solution replaces the
standard one-vs-all (softmax) discriminator with a one-vs-one (all pairs)
discriminator and combines it with a novel intrinsic reward function and a
dropout regularization technique. The combined approach is named APART: Diverse
Skill Discovery using All Pairs with Ascending Reward and Dropout. We
demonstrate that APART discovers all the possible skills in grid worlds with
remarkably fewer samples than previous works. Motivated by the empirical
success of APART, we further investigate an even simpler algorithm that
achieves maximum skills by altering VIC, rescaling its intrinsic reward, and
tuning the temperature of its softmax discriminator. We believe our findings
shed light on the crucial factors underlying success of skill discovery
algorithms in reinforcement learning.
less
By: Manal Helal
As the size and complexity of data continue to increase, the need for
efficient and effective analysis methods becomes ever more crucial.
Tensorization, the process of converting 2-dimensional datasets into
multidimensional structures, has emerged as a promising approach for multiway
analysis methods. This paper explores the steps involved in tensorization,
multidimensional data sources, various multiway analysis methods employed, and
the b... more
As the size and complexity of data continue to increase, the need for
efficient and effective analysis methods becomes ever more crucial.
Tensorization, the process of converting 2-dimensional datasets into
multidimensional structures, has emerged as a promising approach for multiway
analysis methods. This paper explores the steps involved in tensorization,
multidimensional data sources, various multiway analysis methods employed, and
the benefits of these approaches. A small example of Blind Source Separation
(BSS) is presented comparing 2-dimensional algorithms and a multiway algorithm
in Python. Results indicate that multiway analysis is more expressive.
Additionally, tensorization techniques aid in compressing deep learning models
by reducing the number of required parameters while enhancing the expression of
relationships across dimensions. A survey of the multi-away analysis methods
and integration with various Deep Neural Networks models is presented using
case studies in different domains.
less
By: Claudio Battiloro, Lucia Testa, Lorenzo Giusti, Stefania Sardellitti, Paolo Di Lorenzo, Sergio Barbarossa
The aim of this work is to introduce Generalized Simplicial Attention Neural
Networks (GSANs), i.e., novel neural architectures designed to process data
defined on simplicial complexes using masked self-attentional layers. Hinging
on topological signal processing principles, we devise a series of
self-attention schemes capable of processing data components defined at
different simplicial orders, such as nodes, edges, triangles, and beyond. ... more
The aim of this work is to introduce Generalized Simplicial Attention Neural
Networks (GSANs), i.e., novel neural architectures designed to process data
defined on simplicial complexes using masked self-attentional layers. Hinging
on topological signal processing principles, we devise a series of
self-attention schemes capable of processing data components defined at
different simplicial orders, such as nodes, edges, triangles, and beyond. These
schemes learn how to weight the neighborhoods of the given topological domain
in a task-oriented fashion, leveraging the interplay among simplices of
different orders through the Dirac operator and its Dirac decomposition. We
also theoretically establish that GSANs are permutation equivariant and
simplicial-aware. Finally, we illustrate how our approach compares favorably
with other methods when applied to several (inductive and transductive) tasks
such as trajectory prediction, missing data imputation, graph classification,
and simplex prediction.
less
By: Abhishek Chandra, Taniya Kapoor, Bram Daniels, Mitrofan Curti, Koen Tiels, Daniel M. Tartakovsky, Elena A. Lomonova
Hysteresis is a ubiquitous phenomenon in science and engineering; its
modeling and identification are crucial for understanding and optimizing the
behavior of various systems. We develop an ordinary differential equation-based
recurrent neural network (RNN) approach to model and quantify the hysteresis,
which manifests itself in sequentiality and history-dependence. Our neural
oscillator, HystRNN, draws inspiration from coupled-oscillatory ... more
Hysteresis is a ubiquitous phenomenon in science and engineering; its
modeling and identification are crucial for understanding and optimizing the
behavior of various systems. We develop an ordinary differential equation-based
recurrent neural network (RNN) approach to model and quantify the hysteresis,
which manifests itself in sequentiality and history-dependence. Our neural
oscillator, HystRNN, draws inspiration from coupled-oscillatory RNN and
phenomenological hysteresis models to update the hidden states. The performance
of HystRNN is evaluated to predict generalized scenarios, involving first-order
reversal curves and minor loops. The findings show the ability of HystRNN to
generalize its behavior to previously untrained regions, an essential feature
that hysteresis models must have. This research highlights the advantage of
neural oscillators over the traditional RNN-based methods in capturing complex
hysteresis patterns in magnetic materials, where traditional rate-dependent
methods are inadequate to capture intrinsic nonlinearity.
less
By: Wei-Chao Cheng, Tan-Ha Mai, Hsuan-Tien Lin
Given imbalanced data, it is hard to train a good classifier using deep
learning because of the poor generalization of minority classes. Traditionally,
the well-known synthetic minority oversampling technique (SMOTE) for data
augmentation, a data mining approach for imbalanced learning, has been used to
improve this generalization. However, it is unclear whether SMOTE also benefits
deep learning. In this work, we study why the original SMOT... more
Given imbalanced data, it is hard to train a good classifier using deep
learning because of the poor generalization of minority classes. Traditionally,
the well-known synthetic minority oversampling technique (SMOTE) for data
augmentation, a data mining approach for imbalanced learning, has been used to
improve this generalization. However, it is unclear whether SMOTE also benefits
deep learning. In this work, we study why the original SMOTE is insufficient
for deep learning, and enhance SMOTE using soft labels. Connecting the
resulting soft SMOTE with Mixup, a modern data augmentation technique, leads to
a unified framework that puts traditional and modern data augmentation
techniques under the same umbrella. A careful study within this framework shows
that Mixup improves generalization by implicitly achieving uneven margins
between majority and minority classes. We then propose a novel margin-aware
Mixup technique that more explicitly achieves uneven margins. Extensive
experimental results demonstrate that our proposed technique yields
state-of-the-art performance on deep imbalanced classification while achieving
superior performance on extremely imbalanced data. The code is open-sourced in
our developed package https://github.com/ntucllab/imbalanced-DL to foster
future research in this direction.
less
By: Saurabh Kumar, Henrik Marklund, Benjamin Van Roy
In continual learning, plasticity refers to the ability of an agent to
quickly adapt to new information. Neural networks are known to lose plasticity
when processing non-stationary data streams. In this paper, we propose L2 Init,
a very simple approach for maintaining plasticity by incorporating in the loss
function L2 regularization toward initial parameters. This is very similar to
standard L2 regularization (L2), the only difference bein... more
In continual learning, plasticity refers to the ability of an agent to
quickly adapt to new information. Neural networks are known to lose plasticity
when processing non-stationary data streams. In this paper, we propose L2 Init,
a very simple approach for maintaining plasticity by incorporating in the loss
function L2 regularization toward initial parameters. This is very similar to
standard L2 regularization (L2), the only difference being that L2 regularizes
toward the origin. L2 Init is simple to implement and requires selecting only a
single hyper-parameter. The motivation for this method is the same as that of
methods that reset neurons or parameter values. Intuitively, when recent losses
are insensitive to particular parameters, these parameters drift toward their
initial values. This prepares parameters to adapt quickly to new tasks. On
simple problems representative of different types of nonstationarity in
continual learning, we demonstrate that L2 Init consistently mitigates
plasticity loss. We additionally find that our regularization term reduces
parameter magnitudes and maintains a high effective feature rank.
less
By: Konstantinos P. Panousis, Dino Ienco, Diego Marcos
The recent mass adoption of DNNs, even in safety-critical scenarios, has
shifted the focus of the research community towards the creation of inherently
intrepretable models. Concept Bottleneck Models (CBMs) constitute a popular
approach where hidden layers are tied to human understandable concepts allowing
for investigation and correction of the network's decisions. However, CBMs
usually suffer from: (i) performance degradation and (ii) low... more
The recent mass adoption of DNNs, even in safety-critical scenarios, has
shifted the focus of the research community towards the creation of inherently
intrepretable models. Concept Bottleneck Models (CBMs) constitute a popular
approach where hidden layers are tied to human understandable concepts allowing
for investigation and correction of the network's decisions. However, CBMs
usually suffer from: (i) performance degradation and (ii) lower
interpretability than intended due to the sheer amount of concepts contributing
to each decision. In this work, we propose a simple yet highly intuitive
interpretable framework based on Contrastive Language Image models and a single
sparse linear layer. In stark contrast to related approaches, the sparsity in
our framework is achieved via principled Bayesian arguments by inferring
concept presence via a data-driven Bernoulli distribution. As we experimentally
show, our framework not only outperforms recent CBM approaches accuracy-wise,
but it also yields high per example concept sparsity, facilitating the
individual investigation of the emerging concepts.
less
By: David Steinmann, Wolfgang Stammer, Felix Friedrich, Kristian Kersting
While traditional deep learning models often lack interpretability, concept
bottleneck models (CBMs) provide inherent explanations via their concept
representations. Specifically, they allow users to perform interventional
interactions on these concepts by updating the concept values and thus
correcting the predictive output of the model. Traditionally, however, these
interventions are applied to the model only once and discarded afterward.... more
While traditional deep learning models often lack interpretability, concept
bottleneck models (CBMs) provide inherent explanations via their concept
representations. Specifically, they allow users to perform interventional
interactions on these concepts by updating the concept values and thus
correcting the predictive output of the model. Traditionally, however, these
interventions are applied to the model only once and discarded afterward. To
rectify this, we present concept bottleneck memory models (CB2M), an extension
to CBMs. Specifically, a CB2M learns to generalize interventions to appropriate
novel situations via a two-fold memory with which it can learn to detect
mistakes and to reapply previous interventions. In this way, a CB2M learns to
automatically improve model performance from a few initially obtained
interventions. If no prior human interventions are available, a CB2M can detect
potential mistakes of the CBM bottleneck and request targeted interventions. In
our experimental evaluations on challenging scenarios like handling
distribution shifts and confounded training data, we illustrate that CB2M are
able to successfully generalize interventions to unseen data and can indeed
identify wrongly inferred concepts. Overall, our results show that CB2M is a
great tool for users to provide interactive feedback on CBMs, e.g., by guiding
a user's interaction and requiring fewer interventions.
less
By: Yifan Zhang, Rui Wu, Sergiu M. Dascalu, Frederick C. Harris Jr
Multivariate Time Series (MTS) forecasting involves modeling temporal
dependencies within historical records. Transformers have demonstrated
remarkable performance in MTS forecasting due to their capability to capture
long-term dependencies. However, prior work has been confined to modeling
temporal dependencies at either a fixed scale or multiple scales that
exponentially increase (most with base 2). This limitation hinders their
effective... more
Multivariate Time Series (MTS) forecasting involves modeling temporal
dependencies within historical records. Transformers have demonstrated
remarkable performance in MTS forecasting due to their capability to capture
long-term dependencies. However, prior work has been confined to modeling
temporal dependencies at either a fixed scale or multiple scales that
exponentially increase (most with base 2). This limitation hinders their
effectiveness in capturing diverse seasonalities, such as hourly and daily
patterns. In this paper, we introduce a dimension invariant embedding technique
that captures short-term temporal dependencies and projects MTS data into a
higher-dimensional space, while preserving the dimensions of time steps and
variables in MTS data. Furthermore, we present a novel Multi-scale Transformer
Pyramid Network (MTPNet), specifically designed to effectively capture temporal
dependencies at multiple unconstrained scales. The predictions are inferred
from multi-scale latent representations obtained from transformers at various
scales. Extensive experiments on nine benchmark datasets demonstrate that the
proposed MTPNet outperforms recent state-of-the-art methods.
less
By: Tim Bakker, Herke van Hoof, Max Welling
Pool-based active learning (AL) is a promising technology for increasing
data-efficiency of machine learning models. However, surveys show that
performance of recent AL methods is very sensitive to the choice of dataset and
training setting, making them unsuitable for general application. In order to
tackle this problem, the field Learning Active Learning (LAL) suggests to learn
the active learning strategy itself, allowing it to adapt to t... more
Pool-based active learning (AL) is a promising technology for increasing
data-efficiency of machine learning models. However, surveys show that
performance of recent AL methods is very sensitive to the choice of dataset and
training setting, making them unsuitable for general application. In order to
tackle this problem, the field Learning Active Learning (LAL) suggests to learn
the active learning strategy itself, allowing it to adapt to the given setting.
In this work, we propose a novel LAL method for classification that exploits
symmetry and independence properties of the active learning problem with an
Attentive Conditional Neural Process model. Our approach is based on learning
from a myopic oracle, which gives our model the ability to adapt to
non-standard objectives, such as those that do not equally weight the error on
all data points. We experimentally verify that our Neural Process model
outperforms a variety of baselines in these settings. Finally, our experiments
show that our model exhibits a tendency towards improved stability to changing
datasets. However, performance is sensitive to choice of classifier and more
work is necessary to reduce the performance the gap with the myopic oracle and
to improve scalability. We present our work as a proof-of-concept for LAL on
nonstandard objectives and hope our analysis and modelling considerations
inspire future LAL work.
less
By: Keegan Kelly, Lorena Piedras, Sukrit Rao, David Roth
Normalizing Flows (NFs) describe a class of models that express a complex
target distribution as the composition of a series of bijective transformations
over a simpler base distribution. By limiting the space of candidate
transformations to diffeomorphisms, NFs enjoy efficient, exact sampling and
density evaluation, enabling NFs to flexibly behave as both discriminative and
generative models. Their restriction to diffeomorphisms, however, ... more
Normalizing Flows (NFs) describe a class of models that express a complex
target distribution as the composition of a series of bijective transformations
over a simpler base distribution. By limiting the space of candidate
transformations to diffeomorphisms, NFs enjoy efficient, exact sampling and
density evaluation, enabling NFs to flexibly behave as both discriminative and
generative models. Their restriction to diffeomorphisms, however, enforces that
input, output and all intermediary spaces share the same dimension, limiting
their ability to effectively represent target distributions with complex
topologies. Additionally, in cases where the prior and target distributions are
not homeomorphic, Normalizing Flows can leak mass outside of the support of the
target. This survey covers a selection of recent works that combine aspects of
other generative model classes, such as VAEs and score-based diffusion, and in
doing so loosen the strict bijectivity constraints of NFs to achieve a balance
of expressivity, training speed, sample efficiency and likelihood tractability.
less
By: Sven Buechel, Udo Hahn
Human emotion is expressed in many communication modalities and media formats
and so their computational study is equally diversified into natural language
processing, audio signal analysis, computer vision, etc. Similarly, the large
variety of representation formats used in previous research to describe
emotions (polarity scales, basic emotion categories, dimensional approaches,
appraisal theory, etc.) have led to an ever proliferating div... more
Human emotion is expressed in many communication modalities and media formats
and so their computational study is equally diversified into natural language
processing, audio signal analysis, computer vision, etc. Similarly, the large
variety of representation formats used in previous research to describe
emotions (polarity scales, basic emotion categories, dimensional approaches,
appraisal theory, etc.) have led to an ever proliferating diversity of
datasets, predictive models, and software tools for emotion analysis. Because
of these two distinct types of heterogeneity, at the expressional and
representational level, there is a dire need to unify previous work on
increasingly diverging data and label types. This article presents such a
unifying computational model. We propose a training procedure that learns a
shared latent representation for emotions, so-called emotion embeddings,
independent of different natural languages, communication modalities, media or
representation label formats, and even disparate model architectures.
Experiments on a wide range of heterogeneous affective datasets indicate that
this approach yields the desired interoperability for the sake of reusability,
interpretability and flexibility, without penalizing prediction quality. Code
and data are archived under https://doi.org/10.5281/zenodo.7405327 .
less
By: Hangyu Zhu, Yuxiang Fan, Zhenping Xie
Federated learning has gained significant attention due to its groundbreaking
ability to enable distributed learning while maintaining privacy constraints.
However, as a consequence of data heterogeneity among decentralized devices, it
inherently experiences significant learning degradation and slow convergence
speed. Therefore, it is natural to employ the concept of clustering homogeneous
clients into the same group, allowing only the mode... more
Federated learning has gained significant attention due to its groundbreaking
ability to enable distributed learning while maintaining privacy constraints.
However, as a consequence of data heterogeneity among decentralized devices, it
inherently experiences significant learning degradation and slow convergence
speed. Therefore, it is natural to employ the concept of clustering homogeneous
clients into the same group, allowing only the model weights within each group
to be aggregated. While most existing clustered federated learning methods
employ either model gradients or inference outputs as metrics for client
partitioning, with the goal of grouping similar devices together, may still
have heterogeneity within each cluster. Moreover, there is a scarcity of
research exploring the underlying reasons for determining the appropriate
timing for clustering, resulting in the common practice of assigning each
client to its own individual cluster, particularly in the context of highly non
independent and identically distributed (Non-IID) data. In this paper, we
introduce a two-stage decoupling federated learning algorithm with adaptive
personalization layers named FedTSDP, where client clustering is performed
twice according to inference outputs and model weights, respectively. Hopkins
amended sampling is adopted to determine the appropriate timing for clustering
and the sampling weight of public unlabeled data. In addition, a simple yet
effective approach is developed to adaptively adjust the personalization layers
based on varying degrees of data skew. Experimental results show that our
proposed method has reliable performance on both IID and non-IID scenarios.
less
By: Yuxi Liu, Zhenhao Zhang, Shaowen Qin, Flora D. Salim, Antonio Jimeno Yepes, Jun Shen
The Intensive Care Unit (ICU) is one of the most important parts of a
hospital, which admits critically ill patients and provides continuous
monitoring and treatment. Various patient outcome prediction methods have been
attempted to assist healthcare professionals in clinical decision-making.
Existing methods focus on measuring the similarity between patients using deep
neural networks to capture the hidden feature structures. However, the
... more
The Intensive Care Unit (ICU) is one of the most important parts of a
hospital, which admits critically ill patients and provides continuous
monitoring and treatment. Various patient outcome prediction methods have been
attempted to assist healthcare professionals in clinical decision-making.
Existing methods focus on measuring the similarity between patients using deep
neural networks to capture the hidden feature structures. However, the
higher-order relationships are ignored, such as patient characteristics (e.g.,
diagnosis codes) and their causal effects on downstream clinical predictions.
In this paper, we propose a novel Hypergraph Convolutional Network that
allows the representation of non-pairwise relationships among diagnosis codes
in a hypergraph to capture the hidden feature structures so that fine-grained
patient similarity can be calculated for personalized mortality risk
prediction. Evaluation using a publicly available eICU Collaborative Research
Database indicates that our method achieves superior performance over the
state-of-the-art models on mortality risk prediction. Moreover, the results of
several case studies demonstrated the effectiveness of constructing graph
networks in providing good transparency and robustness in decision-making.
less
By: Shenglong Zhou, Kaidi Xu, Geoffrey Ye Li
Decentralized federated learning (DFL) has gained popularity due to its
practicality across various applications. Compared to the centralized version,
training a shared model among a large number of nodes in DFL is more
challenging, as there is no central server to coordinate the training process.
Especially when distributed nodes suffer from limitations in communication or
computational resources, DFL will experience extremely inefficient ... more
Decentralized federated learning (DFL) has gained popularity due to its
practicality across various applications. Compared to the centralized version,
training a shared model among a large number of nodes in DFL is more
challenging, as there is no central server to coordinate the training process.
Especially when distributed nodes suffer from limitations in communication or
computational resources, DFL will experience extremely inefficient and unstable
training. Motivated by these challenges, in this paper, we develop a novel
algorithm based on the framework of the inexact alternating direction method
(iADM). On one hand, our goal is to train a shared model with a sparsity
constraint. This constraint enables us to leverage one-bit compressive sensing
(1BCS), allowing transmission of one-bit information among neighbour nodes. On
the other hand, communication between neighbour nodes occurs only at certain
steps, reducing the number of communication rounds. Therefore, the algorithm
exhibits notable communication efficiency. Additionally, as each node selects
only a subset of neighbours to participate in the training, the algorithm is
robust against stragglers. Additionally, complex items are computed only once
for several consecutive steps and subproblems are solved inexactly using
closed-form solutions, resulting in high computational efficiency. Finally,
numerical experiments showcase the algorithm's effectiveness in both
communication and computation.
less
By: Yinghao Zhu, Zixiang Wang, Long He, Shiyun Xie, Zixi Chen, Jingkun An, Liantao Ma, Chengwei Pan
Electronic Health Record (EHR) data frequently exhibits sparse
characteristics, posing challenges for predictive modeling. Current direct
imputation such as matrix imputation approaches hinge on referencing analogous
rows or columns to complete raw missing data and do not differentiate between
imputed and actual values. As a result, models may inadvertently incorporate
irrelevant or deceptive information with respect to the prediction objec... more
Electronic Health Record (EHR) data frequently exhibits sparse
characteristics, posing challenges for predictive modeling. Current direct
imputation such as matrix imputation approaches hinge on referencing analogous
rows or columns to complete raw missing data and do not differentiate between
imputed and actual values. As a result, models may inadvertently incorporate
irrelevant or deceptive information with respect to the prediction objective,
thereby compromising the efficacy of downstream performance. While some methods
strive to recalibrate or augment EHR embeddings after direct imputation, they
often mistakenly prioritize imputed features. This misprioritization can
introduce biases or inaccuracies into the model. To tackle these issues, our
work resorts to indirect imputation, where we leverage prototype
representations from similar patients to obtain a denser embedding. Recognizing
the limitation that missing features are typically treated the same as present
ones when measuring similar patients, our approach designs a feature confidence
learner module. This module is sensitive to the missing feature status,
enabling the model to better judge the reliability of each feature. Moreover,
we propose a novel patient similarity metric that takes feature confidence into
account, ensuring that evaluations are not based merely on potentially
inaccurate imputed values. Consequently, our work captures dense prototype
patient representations with feature-missing-aware calibration process.
Comprehensive experiments demonstrate that designed model surpasses established
EHR-focused models with a statistically significant improvement on MIMIC-III
and MIMIC-IV datasets in-hospital mortality outcome prediction task. The code
is publicly available at \url{https://anonymous.4open.science/r/SparseEHR} to
assure the reproducibility.
less
By: Léo Grinsztajn SODA, Edouard Oyallon ISIR, CNRS, Gaël Varoquaux SODA
While deep learning has enabled tremendous progress on text and image
datasets, its superiority on tabular data is not clear. We contribute extensive
benchmarks of standard and novel deep learning methods as well as tree-based
models such as XGBoost and Random Forests, across a large number of datasets
and hyperparameter combinations. We define a standard set of 45 datasets from
varied domains with clear characteristics of tabular data and ... more
While deep learning has enabled tremendous progress on text and image
datasets, its superiority on tabular data is not clear. We contribute extensive
benchmarks of standard and novel deep learning methods as well as tree-based
models such as XGBoost and Random Forests, across a large number of datasets
and hyperparameter combinations. We define a standard set of 45 datasets from
varied domains with clear characteristics of tabular data and a benchmarking
methodology accounting for both fitting models and finding good
hyperparameters. Results show that tree-based models remain state-of-the-art on
medium-sized data ($\sim$10K samples) even without accounting for their
superior speed. To understand this gap, we conduct an empirical investigation
into the differing inductive biases of tree-based models and Neural Networks
(NNs). This leads to a series of challenges which should guide researchers
aiming to build tabular-specific NNs: 1. be robust to uninformative features,
2. preserve the orientation of the data, and 3. be able to easily learn
irregular functions. To stimulate research on tabular architectures, we
contribute a standard benchmark and raw data for baselines: every point of a 20
000 compute hours hyperparameter search for each learner.
less
By: S. S. Hotegni, S. Peitz, M. Berkemeier
Different conflicting optimization criteria arise naturally in various Deep
Learning scenarios. These can address different main tasks (i.e., in the
setting of Multi-Task Learning), but also main and secondary tasks such as loss
minimization versus sparsity. The usual approach is a simple weighting of the
criteria, which formally only works in the convex setting. In this paper, we
present a Multi-Objective Optimization algorithm using a mod... more
Different conflicting optimization criteria arise naturally in various Deep
Learning scenarios. These can address different main tasks (i.e., in the
setting of Multi-Task Learning), but also main and secondary tasks such as loss
minimization versus sparsity. The usual approach is a simple weighting of the
criteria, which formally only works in the convex setting. In this paper, we
present a Multi-Objective Optimization algorithm using a modified Weighted
Chebyshev scalarization for training Deep Neural Networks (DNNs) with respect
to several tasks. By employing this scalarization technique, the algorithm can
identify all optimal solutions of the original problem while reducing its
complexity to a sequence of single-objective problems. The simplified problems
are then solved using an Augmented Lagrangian method, enabling the use of
popular optimization techniques such as Adam and Stochastic Gradient Descent,
while efficaciously handling constraints. Our work aims to address the
(economical and also ecological) sustainability issue of DNN models, with a
particular focus on Deep Multi-Task models, which are typically designed with a
very large number of weights to perform equally well on multiple tasks. Through
experiments conducted on two Machine Learning datasets, we demonstrate the
possibility of adaptively sparsifying the model during training without
significantly impacting its performance, if we are willing to apply
task-specific adaptations to the network weights. Code is available at
https://github.com/salomonhotegni/MDMTN.
less
By: Jeroen M. Galjaard, Robert Birke, Juan Perez, Lydia Y. Chen
The negative impact of label noise is well studied in classical supervised
learning yet remains an open research question in meta-learning. Meta-learners
aim to adapt to unseen learning tasks by learning a good initial model in
meta-training and consecutively fine-tuning it according to new tasks during
meta-testing. In this paper, we present the first extensive analysis of the
impact of varying levels of label noise on the performance of s... more
The negative impact of label noise is well studied in classical supervised
learning yet remains an open research question in meta-learning. Meta-learners
aim to adapt to unseen learning tasks by learning a good initial model in
meta-training and consecutively fine-tuning it according to new tasks during
meta-testing. In this paper, we present the first extensive analysis of the
impact of varying levels of label noise on the performance of state-of-the-art
meta-learners, specifically gradient-based $N$-way $K$-shot learners. We show
that the accuracy of Reptile, iMAML, and foMAML drops by up to 42% on the
Omniglot and CifarFS datasets when meta-training is affected by label noise. To
strengthen the resilience against label noise, we propose two sampling
techniques, namely manifold (Man) and batch manifold (BatMan), which transform
the noisy supervised learners into semi-supervised ones to increase the utility
of noisy labels. We first construct manifold samples of $N$-way
$2$-contrastive-shot tasks through augmentation, learning the embedding via a
contrastive loss in meta-training, and then perform classification through
zeroing on the embedding in meta-testing. We show that our approach can
effectively mitigate the impact of meta-training label noise. Even with 60%
wrong labels \batman and \man can limit the meta-testing accuracy drop to
${2.5}$, ${9.4}$, ${1.1}$ percent points, respectively, with existing
meta-learners across the Omniglot, CifarFS, and MiniImagenet datasets.
less
By: Kateřina Macková, Martin Pilát
The goal of product mapping is to decide, whether two listings from two
different e-shops describe the same products. Existing datasets of matching and
non-matching pairs of products, however, often suffer from incomplete product
information or contain only very distant non-matching products. Therefore,
while predictive models trained on these datasets achieve good results on them,
in practice, they are unusable as they cannot distinguish v... more
The goal of product mapping is to decide, whether two listings from two
different e-shops describe the same products. Existing datasets of matching and
non-matching pairs of products, however, often suffer from incomplete product
information or contain only very distant non-matching products. Therefore,
while predictive models trained on these datasets achieve good results on them,
in practice, they are unusable as they cannot distinguish very similar but
non-matching pairs of products. This paper introduces two new datasets for
product mapping: ProMapCz consisting of 1,495 Czech product pairs and ProMapEn
consisting of 1,555 English product pairs of matching and non-matching products
manually scraped from two pairs of e-shops. The datasets contain both images
and textual descriptions of the products, including their specifications,
making them one of the most complete datasets for product mapping.
Additionally, the non-matching products were selected in two phases, creating
two types of non-matches -- close non-matches and medium non-matches. Even the
medium non-matches are pairs of products that are much more similar than
non-matches in other datasets -- for example, they still need to have the same
brand and similar name and price. After simple data preprocessing, several
machine learning algorithms were trained on these and two the other datasets to
demonstrate the complexity and completeness of ProMap datasets. ProMap datasets
are presented as a golden standard for further research of product mapping
filling the gaps in existing ones.
less
By: Guangji Bai, Ziyang Yu, Zheng Chai, Yue Cheng, Liang Zhao
Despite the recent success of Graph Neural Networks (GNNs), it remains
challenging to train GNNs on large-scale graphs due to neighbor explosions. As
a remedy, distributed computing becomes a promising solution by leveraging
abundant computing resources (e.g., GPU). However, the node dependency of graph
data increases the difficulty of achieving high concurrency in distributed GNN
training, which suffers from the massive communication overh... more
Despite the recent success of Graph Neural Networks (GNNs), it remains
challenging to train GNNs on large-scale graphs due to neighbor explosions. As
a remedy, distributed computing becomes a promising solution by leveraging
abundant computing resources (e.g., GPU). However, the node dependency of graph
data increases the difficulty of achieving high concurrency in distributed GNN
training, which suffers from the massive communication overhead. To address it,
Historical value approximation is deemed a promising class of distributed
training techniques. It utilizes an offline memory to cache historical
information (e.g., node embedding) as an affordable approximation of the exact
value and achieves high concurrency. However, such benefits come at the cost of
involving dated training information, leading to staleness, imprecision, and
convergence issues. To overcome these challenges, this paper proposes SAT
(Staleness-Alleviated Training), a novel and scalable distributed GNN training
framework that reduces the embedding staleness adaptively. The key idea of SAT
is to model the GNN's embedding evolution as a temporal graph and build a model
upon it to predict future embedding, which effectively alleviates the staleness
of the cached historical embedding. We propose an online algorithm to train the
embedding predictor and the distributed GNN alternatively and further provide a
convergence analysis. Empirically, we demonstrate that SAT can effectively
reduce embedding staleness and thus achieve better performance and convergence
speed on multiple large-scale graph datasets.
less
By: Hannah Dröge, Zorah Lähner, Yuval Bahat, Onofre Martorell, Felix Heide, Michael Möller
Permutation matrices play a key role in matching and assignment problems
across the fields, especially in computer vision and robotics. However, memory
for explicitly representing permutation matrices grows quadratically with the
size of the problem, prohibiting large problem instances. In this work, we
propose to tackle the curse of dimensionality of large permutation matrices by
approximating them using low-rank matrix factorization, foll... more
Permutation matrices play a key role in matching and assignment problems
across the fields, especially in computer vision and robotics. However, memory
for explicitly representing permutation matrices grows quadratically with the
size of the problem, prohibiting large problem instances. In this work, we
propose to tackle the curse of dimensionality of large permutation matrices by
approximating them using low-rank matrix factorization, followed by a
nonlinearity. To this end, we rely on the Kissing number theory to infer the
minimal rank required for representing a permutation matrix of a given size,
which is significantly smaller than the problem size. This leads to a drastic
reduction in computation and memory costs, e.g., up to $3$ orders of magnitude
less memory for a problem of size $n=20000$, represented using $8.4\times10^5$
elements in two small matrices instead of using a single huge matrix with
$4\times 10^8$ elements. The proposed representation allows for accurate
representations of large permutation matrices, which in turn enables handling
large problems that would have been infeasible otherwise. We demonstrate the
applicability and merits of the proposed approach through a series of
experiments on a range of problems that involve predicting permutation
matrices, from linear and quadratic assignment to shape matching problems.
less
By: Lucas Berry, David Meger
This work introduces a novel approach for epistemic uncertainty estimation
for ensemble models using pairwise-distance estimators (PaiDEs). These
estimators utilize the pairwise-distance between model components to establish
bounds on entropy and uses said bounds as estimates for information-based
criterion. Unlike recent deep learning methods for epistemic uncertainty
estimation, which rely on sample-based Monte Carlo estimators, PaiDEs ar... more
This work introduces a novel approach for epistemic uncertainty estimation
for ensemble models using pairwise-distance estimators (PaiDEs). These
estimators utilize the pairwise-distance between model components to establish
bounds on entropy and uses said bounds as estimates for information-based
criterion. Unlike recent deep learning methods for epistemic uncertainty
estimation, which rely on sample-based Monte Carlo estimators, PaiDEs are able
to estimate epistemic uncertainty up to 100$\times$ faster, over a larger space
(up to 100$\times$) and perform more accurately in higher dimensions. To
validate our approach, we conducted a series of experiments commonly used to
evaluate epistemic uncertainty estimation: 1D sinusoidal data, Pendulum-v0,
Hopper-v2, Ant-v2 and Humanoid-v2. For each experimental setting, an Active
Learning framework was applied to demonstrate the advantages of PaiDEs for
epistemic uncertainty estimation.
less
By: Mahsa Bazzaz, Seth Cooper
Determining the completability of levels generated by procedural generators
such as machine learning models can be challenging, as it can involve the use
of solver agents that often require a significant amount of time to analyze and
solve levels. Active learning is not yet widely adopted in game evaluations,
although it has been used successfully in natural language processing, image
and speech recognition, and computer vision, where the a... more
Determining the completability of levels generated by procedural generators
such as machine learning models can be challenging, as it can involve the use
of solver agents that often require a significant amount of time to analyze and
solve levels. Active learning is not yet widely adopted in game evaluations,
although it has been used successfully in natural language processing, image
and speech recognition, and computer vision, where the availability of labeled
data is limited or expensive. In this paper, we propose the use of active
learning for learning level completability classification. Through an active
learning approach, we train deep-learning models to classify the completability
of generated levels for Super Mario Bros., Kid Icarus, and a Zelda-like game.
We compare active learning for querying levels to label with completability
against random queries. Our results show using an active learning approach to
label levels results in better classifier performance with the same amount of
labeled data.
less
By: Cheng Chen, Lei Fan
Landslides are a common natural disaster that can cause casualties, property
safety threats and economic losses. Therefore, it is important to understand or
predict the probability of landslide occurrence at potentially risky sites. A
commonly used means is to carry out a landslide susceptibility assessment based
on a landslide inventory and a set of landslide contributing factors. This can
be readily achieved using machine learning (ML) mo... more
Landslides are a common natural disaster that can cause casualties, property
safety threats and economic losses. Therefore, it is important to understand or
predict the probability of landslide occurrence at potentially risky sites. A
commonly used means is to carry out a landslide susceptibility assessment based
on a landslide inventory and a set of landslide contributing factors. This can
be readily achieved using machine learning (ML) models such as logistic
regression (LR), support vector machine (SVM), random forest (RF), extreme
gradient boosting (Xgboost), or deep learning (DL) models such as convolutional
neural network (CNN) and long short time memory (LSTM). As the input data for
these models, landslide contributing factors have varying influences on
landslide occurrence. Therefore, it is logically feasible to select more
important contributing factors and eliminate less relevant ones, with the aim
of increasing the prediction accuracy of these models. However, selecting more
important factors is still a challenging task and there is no generally
accepted method. Furthermore, the effects of factor selection using various
methods on the prediction accuracy of ML and DL models are unclear. In this
study, the impact of the selection of contributing factors on the accuracy of
landslide susceptibility predictions using ML and DL models was investigated.
Four methods for selecting contributing factors were considered for all the
aforementioned ML and DL models, which included Information Gain Ratio (IGR),
Recursive Feature Elimination (RFE), Particle Swarm Optimization (PSO), Least
Absolute Shrinkage and Selection Operators (LASSO) and Harris Hawk Optimization
(HHO). In addition, autoencoder-based factor selection methods for DL models
were also investigated. To assess their performances, an exhaustive approach
was adopted,...
less
By: Dimitris Spathis, Fahim Kawsar
Large Language Models (LLMs) have demonstrated remarkable generalization
across diverse tasks, leading individuals to increasingly use them as personal
assistants and universal computing engines. Nevertheless, a notable obstacle
emerges when feeding numerical/temporal data into these models, such as data
sourced from wearables or electronic health records. LLMs employ tokenizers in
their input that break down text into smaller units. Howeve... more
Large Language Models (LLMs) have demonstrated remarkable generalization
across diverse tasks, leading individuals to increasingly use them as personal
assistants and universal computing engines. Nevertheless, a notable obstacle
emerges when feeding numerical/temporal data into these models, such as data
sourced from wearables or electronic health records. LLMs employ tokenizers in
their input that break down text into smaller units. However, tokenizers are
not designed to represent numerical values and might struggle to understand
repetitive patterns and context, treating consecutive values as separate tokens
and disregarding their temporal relationships. Here, we discuss recent works
that employ LLMs for human-centric tasks such as in mobile health sensing and
present a case study showing that popular LLMs tokenize temporal data
incorrectly. To address that, we highlight potential solutions such as prompt
tuning with lightweight embedding layers as well as multimodal adapters, that
can help bridge this "modality gap". While the capability of language models to
generalize to other modalities with minimal or no finetuning is exciting, this
paper underscores the fact that their outputs cannot be meaningful if they
stumble over input nuances.
less
By: Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, Jitao Sang, Haoyu Tang
Large Vision-Language Models (LVLMs) have recently achieved remarkable
success. However, LVLMs are still plagued by the hallucination problem, which
limits the practicality in many scenarios. Hallucination refers to the
information of LVLMs' responses that does not exist in the visual input, which
poses potential risks of substantial consequences. There has been limited work
studying hallucination evaluation in LVLMs. In this paper, we prop... more
Large Vision-Language Models (LVLMs) have recently achieved remarkable
success. However, LVLMs are still plagued by the hallucination problem, which
limits the practicality in many scenarios. Hallucination refers to the
information of LVLMs' responses that does not exist in the visual input, which
poses potential risks of substantial consequences. There has been limited work
studying hallucination evaluation in LVLMs. In this paper, we propose
Hallucination Evaluation based on Large Language Models (HaELM), an LLM-based
hallucination evaluation framework. HaELM achieves an approximate 95%
performance comparable to ChatGPT and has additional advantages including low
cost, reproducibility, privacy preservation and local deployment. Leveraging
the HaELM, we evaluate the hallucination in current LVLMs. Furthermore, we
analyze the factors contributing to hallucination in LVLMs and offer helpful
suggestions to mitigate the hallucination problem. Our training data and human
annotation hallucination data will be made public soon.
less
By: Fredrik Hellström, Giuseppe Durisi, Benjamin Guedj, Maxim Raginsky
A fundamental question in theoretical machine learning is generalization.
Over the past decades, the PAC-Bayesian approach has been established as a
flexible framework to address the generalization capabilities of machine
learning algorithms, and design new ones. Recently, it has garnered increased
interest due to its potential applicability for a variety of learning
algorithms, including deep neural networks. In parallel, an
information-th... more
A fundamental question in theoretical machine learning is generalization.
Over the past decades, the PAC-Bayesian approach has been established as a
flexible framework to address the generalization capabilities of machine
learning algorithms, and design new ones. Recently, it has garnered increased
interest due to its potential applicability for a variety of learning
algorithms, including deep neural networks. In parallel, an
information-theoretic view of generalization has developed, wherein the
relation between generalization and various information measures has been
established. This framework is intimately connected to the PAC-Bayesian
approach, and a number of results have been independently discovered in both
strands. In this monograph, we highlight this strong connection and present a
unified treatment of generalization. We present techniques and results that the
two perspectives have in common, and discuss the approaches and interpretations
that differ. In particular, we demonstrate how many proofs in the area share a
modular structure, through which the underlying ideas can be intuited. We pay
special attention to the conditional mutual information (CMI) framework;
analytical studies of the information complexity of learning algorithms; and
the application of the proposed methods to deep learning. This monograph is
intended to provide a comprehensive introduction to information-theoretic
generalization bounds and their connection to PAC-Bayes, serving as a
foundation from which the most recent developments are accessible. It is aimed
broadly towards researchers with an interest in generalization and theoretical
machine learning.
less
By: Arian Prabowo, Kaixuan Chen, Hao Xue, Subbu Sethuvenkatraman, Flora D. Salim
In traditional deep learning algorithms, one of the key assumptions is that
the data distribution remains constant during both training and deployment.
However, this assumption becomes problematic when faced with
Out-of-Distribution periods, such as the COVID-19 lockdowns, where the data
distribution significantly deviates from what the model has seen during
training. This paper employs a two-fold strategy: utilizing continual learning
tech... more
In traditional deep learning algorithms, one of the key assumptions is that
the data distribution remains constant during both training and deployment.
However, this assumption becomes problematic when faced with
Out-of-Distribution periods, such as the COVID-19 lockdowns, where the data
distribution significantly deviates from what the model has seen during
training. This paper employs a two-fold strategy: utilizing continual learning
techniques to update models with new data and harnessing human mobility data
collected from privacy-preserving pedestrian counters located outside
buildings. In contrast to online learning, which suffers from 'catastrophic
forgetting' as newly acquired knowledge often erases prior information,
continual learning offers a holistic approach by preserving past insights while
integrating new data. This research applies FSNet, a powerful continual
learning algorithm, to real-world data from 13 building complexes in Melbourne,
Australia, a city which had the second longest total lockdown duration globally
during the pandemic. Results underscore the crucial role of continual learning
in accurate energy forecasting, particularly during Out-of-Distribution
periods. Secondary data such as mobility and temperature provided ancillary
support to the primary forecasting model. More importantly, while traditional
methods struggled to adapt during lockdowns, models featuring at least online
learning demonstrated resilience, with lockdown periods posing fewer challenges
once armed with adaptive learning techniques. This study contributes valuable
methodologies and insights to the ongoing effort to improve energy load
forecasting during future Out-of-Distribution periods.
less
By: Vishnu Pandi Chellapandi, Antesh Upadhyay, Abolfazl Hashemi, Stanislaw H /. Zak
Decentralized learning and optimization is a central problem in control that
encompasses several existing and emerging applications, such as federated
learning. While there exists a vast literature on this topic and most methods
centered around the celebrated average-consensus paradigm, less attention has
been devoted to scenarios where the communication between the agents may be
imperfect. To this end, this paper presents three different a... more
Decentralized learning and optimization is a central problem in control that
encompasses several existing and emerging applications, such as federated
learning. While there exists a vast literature on this topic and most methods
centered around the celebrated average-consensus paradigm, less attention has
been devoted to scenarios where the communication between the agents may be
imperfect. To this end, this paper presents three different algorithms of
Decentralized Federated Learning (DFL) in the presence of imperfect information
sharing modeled as noisy communication channels. The first algorithm, Federated
Noisy Decentralized Learning (FedNDL1), comes from the literature, where the
noise is added to their parameters to simulate the scenario of the presence of
noisy communication channels. This algorithm shares parameters to form a
consensus with the clients based on a communication graph topology through a
noisy communication channel. The proposed second algorithm (FedNDL2) is similar
to the first algorithm but with added noise to the parameters, and it performs
the gossip averaging before the gradient optimization. The proposed third
algorithm
(FedNDL3), on the other hand, shares the gradients through noisy
communication channels instead of the parameters. Theoretical and experimental
results demonstrate that under imperfect information sharing, the third scheme
that mixes gradients is more robust in the presence of a noisy channel compared
with the algorithms from the literature that mix the parameters.
less
By: Pascal Esser, Satyaki Mukherjee, Debarghya Ghoshdastidar
Self-Supervised Learning (SSL) is an important paradigm for learning
representations from unlabelled data, and SSL with neural networks has been
highly successful in practice. However current theoretical analysis of SSL is
mostly restricted to generalisation error bounds. In contrast, learning
dynamics often provide a precise characterisation of the behaviour of neural
networks based models but, so far, are mainly known in supervised settin... more
Self-Supervised Learning (SSL) is an important paradigm for learning
representations from unlabelled data, and SSL with neural networks has been
highly successful in practice. However current theoretical analysis of SSL is
mostly restricted to generalisation error bounds. In contrast, learning
dynamics often provide a precise characterisation of the behaviour of neural
networks based models but, so far, are mainly known in supervised settings. In
this paper, we study the learning dynamics of SSL models, specifically
representations obtained by minimising contrastive and non-contrastive losses.
We show that a naive extension of the dymanics of multivariate regression to
SSL leads to learning trivial scalar representations that demonstrates
dimension collapse in SSL. Consequently, we formulate SSL objectives with
orthogonality constraints on the weights, and derive the exact (network width
independent) learning dynamics of the SSL models trained using gradient descent
on the Grassmannian manifold. We also argue that the infinite width
approximation of SSL models significantly deviate from the neural tangent
kernel approximations of supervised models. We numerically illustrate the
validity of our theoretical findings, and discuss how the presented results
provide a framework for further theoretical analysis of contrastive and
non-contrastive SSL.
less
By: Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, Tom Goldstein
As Large Language Models quickly become ubiquitous, their security
vulnerabilities are critical to understand. Recent work shows that text
optimizers can produce jailbreaking prompts that bypass moderation and
alignment. Drawing from the rich body of work on adversarial machine learning,
we approach these attacks with three questions: What threat models are
practically useful in this domain? How do baseline defense techniques perform
in thi... more
As Large Language Models quickly become ubiquitous, their security
vulnerabilities are critical to understand. Recent work shows that text
optimizers can produce jailbreaking prompts that bypass moderation and
alignment. Drawing from the rich body of work on adversarial machine learning,
we approach these attacks with three questions: What threat models are
practically useful in this domain? How do baseline defense techniques perform
in this new domain? How does LLM security differ from computer vision?
We evaluate several baseline defense strategies against leading adversarial
attacks on LLMs, discussing the various settings in which each is feasible and
effective. Particularly, we look at three types of defenses: detection
(perplexity based), input preprocessing (paraphrase and retokenization), and
adversarial training. We discuss white-box and gray-box settings and discuss
the robustness-performance trade-off for each of the defenses considered.
Surprisingly, we find much more success with filtering and preprocessing than
we would expect from other domains, such as vision, providing a first
indication that the relative strengths of these defenses may be weighed
differently in these domains.
less
By: Daniel Freund, Thodoris Lykouris, Wentao Weng
Queueing systems are widely applicable stochastic models with use cases in
communication networks, healthcare, service systems, etc. Although their
optimal control has been extensively studied, most existing approaches assume
perfect knowledge of system parameters. Of course, this assumption rarely holds
in practice where there is parameter uncertainty, thus motivating a recent line
of work on bandit learning for queueing systems. This nasc... more
Queueing systems are widely applicable stochastic models with use cases in
communication networks, healthcare, service systems, etc. Although their
optimal control has been extensively studied, most existing approaches assume
perfect knowledge of system parameters. Of course, this assumption rarely holds
in practice where there is parameter uncertainty, thus motivating a recent line
of work on bandit learning for queueing systems. This nascent stream of
research focuses on the asymptotic performance of the proposed algorithms.
In this paper, we argue that an asymptotic metric, which focuses on
late-stage performance, is insufficient to capture the intrinsic statistical
complexity of learning in queueing systems which typically occurs in the early
stage. Instead, we propose the Cost of Learning in Queueing (CLQ), a new metric
that quantifies the maximum increase in time-averaged queue length caused by
parameter uncertainty. We characterize the CLQ of a single-queue multi-server
system, and then extend these results to multi-queue multi-server systems and
networks of queues. In establishing our results, we propose a unified analysis
framework for CLQ that bridges Lyapunov and bandit analysis, which could be of
independent interest.
less
By: Fahad Alrasheedi, Xin Zhong
Although Deep Neural Networks (DNNs), such as the convolutional neural
networks (CNN) and Vision Transformers (ViTs), have been successfully applied
in the field of computer vision, they are demonstrated to be vulnerable to
well-sought Adversarial Examples (AEs) that can easily fool the DNNs. The
research in AEs has been active, and many adversarial attacks and explanations
have been proposed since they were discovered in 2014. The mystery ... more
Although Deep Neural Networks (DNNs), such as the convolutional neural
networks (CNN) and Vision Transformers (ViTs), have been successfully applied
in the field of computer vision, they are demonstrated to be vulnerable to
well-sought Adversarial Examples (AEs) that can easily fool the DNNs. The
research in AEs has been active, and many adversarial attacks and explanations
have been proposed since they were discovered in 2014. The mystery of the AE's
existence is still an open question, and many studies suggest that DNN training
algorithms have blind spots. The salient objects usually do not overlap with
boundaries; hence, the boundaries are not the DNN model's attention.
Nevertheless, recent studies show that the boundaries can dominate the behavior
of the DNN models. Hence, this study aims to look at the AEs from a different
perspective and proposes an imperceptible adversarial attack that systemically
attacks the input image boundary for finding the AEs. The experimental results
have shown that the proposed boundary attacking method effectively attacks six
CNN models and the ViT using only 32% of the input image content (from the
boundaries) with an average success rate (SR) of 95.2% and an average peak
signal-to-noise ratio of 41.37 dB. Correlation analyses are conducted,
including the relation between the adversarial boundary's width and the SR and
how the adversarial boundary changes the DNN model's attention. This paper's
discoveries can potentially advance the understanding of AEs and provide a
different perspective on how AEs can be constructed.
less
By: Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, Samet Oymak
Since its inception in "Attention Is All You Need", transformer architecture
has led to revolutionary advancements in NLP. The attention layer within the
transformer admits a sequence of input tokens $X$ and makes them interact
through pairwise similarities computed as softmax$(XQK^\top X^\top)$, where
$(K,Q)$ are the trainable key-query parameters. In this work, we establish a
formal equivalence between the optimization geometry of self-at... more
Since its inception in "Attention Is All You Need", transformer architecture
has led to revolutionary advancements in NLP. The attention layer within the
transformer admits a sequence of input tokens $X$ and makes them interact
through pairwise similarities computed as softmax$(XQK^\top X^\top)$, where
$(K,Q)$ are the trainable key-query parameters. In this work, we establish a
formal equivalence between the optimization geometry of self-attention and a
hard-margin SVM problem that separates optimal input tokens from non-optimal
tokens using linear constraints on the outer-products of token pairs. This
formalism allows us to characterize the implicit bias of 1-layer transformers
optimized with gradient descent: (1) Optimizing the attention layer with
vanishing regularization, parameterized by $(K,Q)$, converges in direction to
an SVM solution minimizing the nuclear norm of the combined parameter
$W=KQ^\top$. Instead, directly parameterizing by $W$ minimizes a Frobenius norm
objective. We characterize this convergence, highlighting that it can occur
toward locally-optimal directions rather than global ones. (2) Complementing
this, we prove the local/global directional convergence of gradient descent
under suitable geometric conditions. Importantly, we show that
over-parameterization catalyzes global convergence by ensuring the feasibility
of the SVM problem and by guaranteeing a benign optimization landscape devoid
of stationary points. (3) While our theory applies primarily to linear
prediction heads, we propose a more general SVM equivalence that predicts the
implicit bias with nonlinear heads. Our findings are applicable to arbitrary
datasets and their validity is verified via experiments. We also introduce
several open problems and research directions. We believe these findings
inspire the interpretation of transformers as a hierarchy of SVMs that
separates and selects optimal tokens.
less
Multi-Source Domain Adaptation meets Dataset Distillation through Dataset Dictionary Learning
0upvotes
By: Eduardo Fernandes Montesuma, Fred Ngolè Mboula, Antoine Souloumiac
In this paper, we consider the intersection of two problems in machine
learning: Multi-Source Domain Adaptation (MSDA) and Dataset Distillation (DD).
On the one hand, the first considers adapting multiple heterogeneous labeled
source domains to an unlabeled target domain. On the other hand, the second
attacks the problem of synthesizing a small summary containing all the
information about the datasets. We thus consider a new problem called ... more
In this paper, we consider the intersection of two problems in machine
learning: Multi-Source Domain Adaptation (MSDA) and Dataset Distillation (DD).
On the one hand, the first considers adapting multiple heterogeneous labeled
source domains to an unlabeled target domain. On the other hand, the second
attacks the problem of synthesizing a small summary containing all the
information about the datasets. We thus consider a new problem called MSDA-DD.
To solve it, we adapt previous works in the MSDA literature, such as
Wasserstein Barycenter Transport and Dataset Dictionary Learning, as well as DD
method Distribution Matching. We thoroughly experiment with this novel problem
on four benchmarks (Caltech-Office 10, Tennessee-Eastman Process, Continuous
Stirred Tank Reactor, and Case Western Reserve University), where we show that,
even with as little as 1 sample per class, one achieves state-of-the-art
adaptation performance.
less
By: Lukas Blecher, Guillem Cucurull, Thomas Scialom, Robert Stojnic
Scientific knowledge is predominantly stored in books and scientific
journals, often in the form of PDFs. However, the PDF format leads to a loss of
semantic information, particularly for mathematical expressions. We propose
Nougat (Neural Optical Understanding for Academic Documents), a Visual
Transformer model that performs an Optical Character Recognition (OCR) task for
processing scientific documents into a markup language, and demonstr... more
Scientific knowledge is predominantly stored in books and scientific
journals, often in the form of PDFs. However, the PDF format leads to a loss of
semantic information, particularly for mathematical expressions. We propose
Nougat (Neural Optical Understanding for Academic Documents), a Visual
Transformer model that performs an Optical Character Recognition (OCR) task for
processing scientific documents into a markup language, and demonstrate the
effectiveness of our model on a new dataset of scientific documents. The
proposed approach offers a promising solution to enhance the accessibility of
scientific knowledge in the digital age, by bridging the gap between
human-readable documents and machine-readable text. We release the models and
code to accelerate future work on scientific text recognition.
less
By: Ahmad Chamma Inria Universite Paris Saclay CEA, Denis A. Engemann Roche Pharma Research and Early Development, Neuroscience and Rare Diseases, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland, Bertrand Thirion Inria Universite Paris Saclay CEA
Variable importance assessment has become a crucial step in machine-learning
applications when using complex learners, such as deep neural networks, on
large-scale data. Removal-based importance assessment is currently the
reference approach, particularly when statistical guarantees are sought to
justify variable inclusion. It is often implemented with variable permutation
schemes. On the flip side, these approaches risk misidentifying unim... more
Variable importance assessment has become a crucial step in machine-learning
applications when using complex learners, such as deep neural networks, on
large-scale data. Removal-based importance assessment is currently the
reference approach, particularly when statistical guarantees are sought to
justify variable inclusion. It is often implemented with variable permutation
schemes. On the flip side, these approaches risk misidentifying unimportant
variables as important in the presence of correlations among covariates. Here
we develop a systematic approach for studying Conditional Permutation
Importance (CPI) that is model agnostic and computationally lean, as well as
reusable benchmarks of state-of-the-art variable importance estimators. We show
theoretically and empirically that $\textit{CPI}$ overcomes the limitations of
standard permutation importance by providing accurate type-I error control.
When used with a deep neural network, $\textit{CPI}$ consistently showed top
accuracy across benchmarks. An empirical benchmark on real-world data analysis
in a large-scale medical dataset showed that $\textit{CPI}$ provides a more
parsimonious selection of statistically significant variables. Our results
suggest that $\textit{CPI}$ can be readily used as drop-in replacement for
permutation-based methods.
less
deepFDEnet: A Novel Neural Network Architecture for Solving Fractional Differential Equations
0upvotes
By: Ali Nosrati Firoozsalari, Hassan Dana Mazraeh, Alireza Afzal Aghaei, Kourosh Parand
The primary goal of this research is to propose a novel architecture for a
deep neural network that can solve fractional differential equations
accurately. A Gaussian integration rule and a $L_1$ discretization technique
are used in the proposed design. In each equation, a deep neural network is
used to approximate the unknown function. Three forms of fractional
differential equations have been examined to highlight the method's
versatility... more
The primary goal of this research is to propose a novel architecture for a
deep neural network that can solve fractional differential equations
accurately. A Gaussian integration rule and a $L_1$ discretization technique
are used in the proposed design. In each equation, a deep neural network is
used to approximate the unknown function. Three forms of fractional
differential equations have been examined to highlight the method's
versatility: a fractional ordinary differential equation, a fractional order
integrodifferential equation, and a fractional order partial differential
equation. The results show that the proposed architecture solves different
forms of fractional differential equations with excellent precision.
less
By: Md Sabbirul Haque, Md Shahedul Amin, Jonayet Miah
Accurate demand forecasting in the retail industry is a critical determinant
of financial performance and supply chain efficiency. As global markets become
increasingly interconnected, businesses are turning towards advanced prediction
models to gain a competitive edge. However, existing literature mostly focuses
on historical sales data and ignores the vital influence of macroeconomic
conditions on consumer spending behavior. In this study... more
Accurate demand forecasting in the retail industry is a critical determinant
of financial performance and supply chain efficiency. As global markets become
increasingly interconnected, businesses are turning towards advanced prediction
models to gain a competitive edge. However, existing literature mostly focuses
on historical sales data and ignores the vital influence of macroeconomic
conditions on consumer spending behavior. In this study, we bridge this gap by
enriching time series data of customer demand with macroeconomic variables,
such as the Consumer Price Index (CPI), Index of Consumer Sentiment (ICS), and
unemployment rates. Leveraging this comprehensive dataset, we develop and
compare various regression and machine learning models to predict retail demand
accurately.
less
By: Jiaoyan Chen, Yuxia Geng, Zhuo Chen, Jeff Z. Pan, Yuan He, Wen Zhang, Ian Horrocks, Huajun Chen
Machine learning especially deep neural networks have achieved great success
but many of them often rely on a number of labeled samples for supervision. As
sufficient labeled training data are not always ready due to e.g., continuously
emerging prediction targets and costly sample annotation in real world
applications, machine learning with sample shortage is now being widely
investigated. Among all these studies, many prefer to utilize aux... more
Machine learning especially deep neural networks have achieved great success
but many of them often rely on a number of labeled samples for supervision. As
sufficient labeled training data are not always ready due to e.g., continuously
emerging prediction targets and costly sample annotation in real world
applications, machine learning with sample shortage is now being widely
investigated. Among all these studies, many prefer to utilize auxiliary
information including those in the form of Knowledge Graph (KG) to reduce the
reliance on labeled samples. In this survey, we have comprehensively reviewed
over 90 papers about KG-aware research for two major sample shortage settings
-- zero-shot learning (ZSL) where some classes to be predicted have no labeled
samples, and few-shot learning (FSL) where some classes to be predicted have
only a small number of labeled samples that are available. We first introduce
KGs used in ZSL and FSL as well as their construction methods, and then
systematically categorize and summarize KG-aware ZSL and FSL methods, dividing
them into different paradigms such as the mapping-based, the data augmentation,
the propagation-based and the optimization-based. We next present different
applications, including not only KG augmented prediction tasks such as image
classification, question answering, text classification and knowledge
extraction, but also KG completion tasks, and some typical evaluation resources
for each task. We eventually discuss some challenges and open problems from
different perspectives.
less
By: Sopan Sarkar, Mohammad Hossein Manshaei, Marwan Krunz
Radio-frequency coverage maps (RF maps) are extensively utilized in wireless
networks for capacity planning, placement of access points and base stations,
localization, and coverage estimation. Conducting site surveys to obtain RF
maps is labor-intensive and sometimes not feasible. In this paper, we propose
radio-frequency adversarial deep-learning inference for automated network
coverage estimation (RADIANCE), a generative adversarial netw... more
Radio-frequency coverage maps (RF maps) are extensively utilized in wireless
networks for capacity planning, placement of access points and base stations,
localization, and coverage estimation. Conducting site surveys to obtain RF
maps is labor-intensive and sometimes not feasible. In this paper, we propose
radio-frequency adversarial deep-learning inference for automated network
coverage estimation (RADIANCE), a generative adversarial network (GAN) based
approach for synthesizing RF maps in indoor scenarios. RADIANCE utilizes a
semantic map, a high-level representation of the indoor environment to encode
spatial relationships and attributes of objects within the environment and
guide the RF map generation process. We introduce a new gradient-based loss
function that computes the magnitude and direction of change in received signal
strength (RSS) values from a point within the environment. RADIANCE
incorporates this loss function along with the antenna pattern to capture
signal propagation within a given indoor configuration and generate new
patterns under new configuration, antenna (beam) pattern, and center frequency.
Extensive simulations are conducted to compare RADIANCE with ray-tracing
simulations of RF maps. Our results show that RADIANCE achieves a mean average
error (MAE) of 0.09, root-mean-squared error (RMSE) of 0.29, peak
signal-to-noise ratio (PSNR) of 10.78, and multi-scale structural similarity
index (MS-SSIM) of 0.80.
less
By: Seokhyeon Ha, Sunbeom Jung, Jungwoo Lee
Fine-tuning pre-trained neural network models has become a widely adopted
approach across various domains. However, it can lead to the distortion of
pre-trained feature extractors that already possess strong generalization
capabilities. Mitigating feature distortion during adaptation to new target
domains is crucial. Recent studies have shown promising results in handling
feature distortion by aligning the head layer on in-distribution data... more
Fine-tuning pre-trained neural network models has become a widely adopted
approach across various domains. However, it can lead to the distortion of
pre-trained feature extractors that already possess strong generalization
capabilities. Mitigating feature distortion during adaptation to new target
domains is crucial. Recent studies have shown promising results in handling
feature distortion by aligning the head layer on in-distribution datasets
before performing fine-tuning. Nonetheless, a significant limitation arises
from the treatment of batch normalization layers during fine-tuning, leading to
suboptimal performance. In this paper, we propose Domain-Aware Fine-Tuning
(DAFT), a novel approach that incorporates batch normalization conversion and
the integration of linear probing and fine-tuning. Our batch normalization
conversion method effectively mitigates feature distortion by reducing
modifications to the neural network during fine-tuning. Additionally, we
introduce the integration of linear probing and fine-tuning to optimize the
head layer with gradual adaptation of the feature extractor. By leveraging
batch normalization layers and integrating linear probing and fine-tuning, our
DAFT significantly mitigates feature distortion and achieves improved model
performance on both in-distribution and out-of-distribution datasets. Extensive
experiments demonstrate that our method outperforms other baseline methods,
demonstrating its effectiveness in not only improving performance but also
mitigating feature distortion.
less
By: Mehak Arora, Hassan Mortagy, Nathan Dwarshius, Swati Gupta, Andre L. Holder, Rishikesan Kamaleswaran
Machine learning (ML) models are increasingly pivotal in automating clinical
decisions. Yet, a glaring oversight in prior research has been the lack of
proper processing of Electronic Medical Record (EMR) data in the clinical
context for errors and outliers. Addressing this oversight, we introduce an
innovative projections-based method that seamlessly integrates clinical
expertise as domain constraints, generating important meta-data that c... more
Machine learning (ML) models are increasingly pivotal in automating clinical
decisions. Yet, a glaring oversight in prior research has been the lack of
proper processing of Electronic Medical Record (EMR) data in the clinical
context for errors and outliers. Addressing this oversight, we introduce an
innovative projections-based method that seamlessly integrates clinical
expertise as domain constraints, generating important meta-data that can be
used in ML workflows. In particular, by using high-dimensional mixed-integer
programs that capture physiological and biological constraints on patient
vitals and lab values, we can harness the power of mathematical "projections"
for the EMR data to correct patient data. Consequently, we measure the distance
of corrected data from the constraints defining a healthy range of patient
data, resulting in a unique predictive metric we term as "trust-scores". These
scores provide insight into the patient's health status and significantly boost
the performance of ML classifiers in real-life clinical settings. We validate
the impact of our framework in the context of early detection of sepsis using
ML. We show an AUROC of 0.865 and a precision of 0.922, that surpasses
conventional ML models without such projections.
less
By: Leonard Berrada, Soham De, Judy Hanwen Shen, Jamie Hayes, Robert Stanforth, David Stutz, Pushmeet Kohli, Samuel L. Smith, Borja Balle
Privacy-preserving machine learning aims to train models on private data
without leaking sensitive information. Differential privacy (DP) is considered
the gold standard framework for privacy-preserving training, as it provides
formal privacy guarantees. However, compared to their non-private counterparts,
models trained with DP often have significantly reduced accuracy. Private
classifiers are also believed to exhibit larger performance di... more
Privacy-preserving machine learning aims to train models on private data
without leaking sensitive information. Differential privacy (DP) is considered
the gold standard framework for privacy-preserving training, as it provides
formal privacy guarantees. However, compared to their non-private counterparts,
models trained with DP often have significantly reduced accuracy. Private
classifiers are also believed to exhibit larger performance disparities across
subpopulations, raising fairness concerns. The poor performance of classifiers
trained with DP has prevented the widespread adoption of privacy preserving
machine learning in industry. Here we show that pre-trained foundation models
fine-tuned with DP can achieve similar accuracy to non-private classifiers,
even in the presence of significant distribution shifts between pre-training
data and downstream tasks. We achieve private accuracies within a few percent
of the non-private state of the art across four datasets, including two medical
imaging benchmarks. Furthermore, our private medical classifiers do not exhibit
larger performance disparities across demographic groups than non-private
models. This milestone to make DP training a practical and reliable technology
has the potential to widely enable machine learning practitioners to train
safely on sensitive datasets while protecting individuals' privacy.
less
By: Pierre Gaillard Thoth, Sébastien Gerchinovitz IMT, Étienne de Montbrun TSE-R
We study the classical problem of approximating a non-decreasing function $f:
\mathcal{X} \to \mathcal{Y}$ in $L^p(\mu)$ norm by sequentially querying its
values, for known compact real intervals $\mathcal{X}$, $\mathcal{Y}$ and a
known probability measure $\mu$ on $\cX$. For any function~$f$ we characterize
the minimum number of evaluations of $f$ that algorithms need to guarantee an
approximation $\hat{f}$ with an $L^p(\mu)$ error below $... more
We study the classical problem of approximating a non-decreasing function $f:
\mathcal{X} \to \mathcal{Y}$ in $L^p(\mu)$ norm by sequentially querying its
values, for known compact real intervals $\mathcal{X}$, $\mathcal{Y}$ and a
known probability measure $\mu$ on $\cX$. For any function~$f$ we characterize
the minimum number of evaluations of $f$ that algorithms need to guarantee an
approximation $\hat{f}$ with an $L^p(\mu)$ error below $\epsilon$ after
stopping. Unlike worst-case results that hold uniformly over all $f$, our
complexity measure is dependent on each specific function $f$. To address this
problem, we introduce GreedyBox, a generalization of an algorithm originally
proposed by Novak (1992) for numerical integration. We prove that GreedyBox
achieves an optimal sample complexity for any function $f$, up to logarithmic
factors. Additionally, we uncover results regarding piecewise-smooth functions.
Perhaps as expected, the $L^p(\mu)$ error of GreedyBox decreases much faster
for piecewise-$C^2$ functions than predicted by the algorithm (without any
knowledge on the smoothness of $f$). A simple modification even achieves
optimal minimax approximation rates for such functions, which we compute
explicitly. In particular, our findings highlight multiple performance gaps
between adaptive and non-adaptive algorithms, smooth and piecewise-smooth
functions, as well as monotone or non-monotone functions. Finally, we provide
numerical experiments to support our theoretical results.
less
By: Md Abrar Jahin, Md Sakib Hossain Shovon, Md. Saiful Islam, Jungpil Shin, M. F. Mridha, Yuichi Okuyama
Supply chain management relies on accurate backorder prediction for optimizing inventory control, reducing costs, and enhancing customer satisfaction. However, traditional machine-learning models struggle with large-scale datasets and complex relationships, hindering real-world data collection. This research introduces a novel methodological framework for supply chain backorder prediction, addressing the challenge of handling large datasets. ... more
Supply chain management relies on accurate backorder prediction for optimizing inventory control, reducing costs, and enhancing customer satisfaction. However, traditional machine-learning models struggle with large-scale datasets and complex relationships, hindering real-world data collection. This research introduces a novel methodological framework for supply chain backorder prediction, addressing the challenge of handling large datasets. Our proposed model, QAmplifyNet, employs quantum-inspired techniques within a quantum-classical neural network to predict backorders effectively on short and imbalanced datasets. Experimental evaluations on a benchmark dataset demonstrate QAmplifyNet's superiority over classical models, quantum ensembles, quantum neural networks, and deep reinforcement learning. Its proficiency in handling short, imbalanced datasets makes it an ideal solution for supply chain management. To enhance model interpretability, we use Explainable Artificial Intelligence techniques. Practical implications include improved inventory control, reduced backorders, and enhanced operational efficiency. QAmplifyNet seamlessly integrates into real-world supply chain management systems, enabling proactive decision-making and efficient resource allocation. Future work involves exploring additional quantum-inspired techniques, expanding the dataset, and investigating other supply chain applications. This research unlocks the potential of quantum computing in supply chain optimization and paves the way for further exploration of quantum-inspired machine learning models in supply chain management. Our framework and QAmplifyNet model offer a breakthrough approach to supply chain backorder prediction, providing superior performance and opening new avenues for leveraging quantum-inspired techniques in supply chain management. less
By: Timothy Chu, Zhao Song, Chiwun Yang
Large language models (LLMs) and generative AI have played a transformative
role in computer research and applications. Controversy has arisen as to
whether these models output copyrighted data, which can occur if the data the
models are trained on is copyrighted. LLMs are built on the transformer neural
network architecture, which in turn relies on a mathematical computation called
Attention that uses the softmax function.
In this paper,... more
Large language models (LLMs) and generative AI have played a transformative
role in computer research and applications. Controversy has arisen as to
whether these models output copyrighted data, which can occur if the data the
models are trained on is copyrighted. LLMs are built on the transformer neural
network architecture, which in turn relies on a mathematical computation called
Attention that uses the softmax function.
In this paper, we show that large language model training and optimization
can be seen as a softmax regression problem. We then establish a method of
efficiently performing softmax regression, in a way that prevents the
regression function from generating copyright data. This establishes a
theoretical method of training large language models in a way that avoids
generating copyright data.
less
By: Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, Tong Zhang
Foundation models, including Vision Language Models (VLMs) and Large Language
Models (LLMs), possess the $generality$ to handle diverse distributions and
tasks, which stems from their extensive pre-training datasets. The fine-tuning
of foundation models is a common practice to enhance task performance or align
the model's behavior with human expectations, allowing them to gain
$speciality$. However, the small datasets used for fine-tuning m... more
Foundation models, including Vision Language Models (VLMs) and Large Language
Models (LLMs), possess the $generality$ to handle diverse distributions and
tasks, which stems from their extensive pre-training datasets. The fine-tuning
of foundation models is a common practice to enhance task performance or align
the model's behavior with human expectations, allowing them to gain
$speciality$. However, the small datasets used for fine-tuning may not
adequately cover the diverse distributions and tasks encountered during
pre-training. Consequently, the pursuit of speciality during fine-tuning can
lead to a loss of {generality} in the model, which is related to catastrophic
forgetting (CF) in deep learning. In this study, we demonstrate this phenomenon
in both VLMs and LLMs. For instance, fine-tuning VLMs like CLIP on ImageNet
results in a loss of generality in handling diverse distributions, and
fine-tuning LLMs like Galactica in the medical domain leads to a loss in
following instructions and common sense.
To address the trade-off between the speciality and generality, we
investigate multiple regularization methods from continual learning, the weight
averaging method (Wise-FT) from out-of-distributional (OOD) generalization,
which interpolates parameters between pre-trained and fine-tuned models, and
parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA). Our
findings show that both continual learning and Wise-ft methods effectively
mitigate the loss of generality, with Wise-FT exhibiting the strongest
performance in balancing speciality and generality.
less
By: Sjoerd de Vries, Dirk Thierens
Most real-world classification tasks suffer from label noise to some extent.
Such noise in the data adversely affects the generalization error of learned
models and complicates the evaluation of noise-handling methods, as their
performance cannot be accurately measured without clean labels. In label noise
research, typically either noisy or incomplex simulated data are accepted as a
baseline, into which additional noise with known propertie... more
Most real-world classification tasks suffer from label noise to some extent.
Such noise in the data adversely affects the generalization error of learned
models and complicates the evaluation of noise-handling methods, as their
performance cannot be accurately measured without clean labels. In label noise
research, typically either noisy or incomplex simulated data are accepted as a
baseline, into which additional noise with known properties is injected. In
this paper, we propose SYNLABEL, a framework that aims to improve upon the
aforementioned methodologies. It allows for creating a noiseless dataset
informed by real data, by either pre-specifying or learning a function and
defining it as the ground truth function from which labels are generated.
Furthermore, by resampling a number of values for selected features in the
function domain, evaluating the function and aggregating the resulting labels,
each data point can be assigned a soft label or label distribution. Such
distributions allow for direct injection and quantification of label noise. The
generated datasets serve as a clean baseline of adjustable complexity into
which different types of noise may be introduced. We illustrate how the
framework can be applied, how it enables quantification of label noise and how
it improves over existing methodologies.
less
By: Yucheng Wang, Yuecong Xu, Jianfei Yang, Min Wu, Xiaoli Li, Lihua Xie, Zhenghua Chen
Multivariate Time-Series (MTS) data is crucial in various application fields.
With its sequential and multi-source (multiple sensors) properties, MTS data
inherently exhibits Spatial-Temporal (ST) dependencies, involving temporal
correlations between timestamps and spatial correlations between sensors in
each timestamp. To effectively leverage this information, Graph Neural
Network-based methods (GNNs) have been widely adopted. However, exi... more
Multivariate Time-Series (MTS) data is crucial in various application fields.
With its sequential and multi-source (multiple sensors) properties, MTS data
inherently exhibits Spatial-Temporal (ST) dependencies, involving temporal
correlations between timestamps and spatial correlations between sensors in
each timestamp. To effectively leverage this information, Graph Neural
Network-based methods (GNNs) have been widely adopted. However, existing
approaches separately capture spatial dependency and temporal dependency and
fail to capture the correlations between Different sEnsors at Different
Timestamps (DEDT). Overlooking such correlations hinders the comprehensive
modelling of ST dependencies within MTS data, thus restricting existing GNNs
from learning effective representations. To address this limitation, we propose
a novel method called Fully-Connected Spatial-Temporal Graph Neural Network
(FC-STGNN), including two key components namely FC graph construction and FC
graph convolution. For graph construction, we design a decay graph to connect
sensors across all timestamps based on their temporal distances, enabling us to
fully model the ST dependencies by considering the correlations between DEDT.
Further, we devise FC graph convolution with a moving-pooling GNN layer to
effectively capture the ST dependencies for learning effective representations.
Extensive experiments show the effectiveness of FC-STGNN on multiple MTS
datasets compared to SOTA methods.
less
By: Ian Colbert, Alessandro Pappalardo, Jakoba Petri-Koenig
We present accumulator-aware quantization (A2Q), a novel weight quantization
method designed to train quantized neural networks (QNNs) to avoid overflow
when using low-precision accumulators during inference. A2Q introduces a unique
formulation inspired by weight normalization that constrains the L1-norm of
model weights according to accumulator bit width bounds that we derive. Thus,
in training QNNs for low-precision accumulation, A2Q also... more
We present accumulator-aware quantization (A2Q), a novel weight quantization
method designed to train quantized neural networks (QNNs) to avoid overflow
when using low-precision accumulators during inference. A2Q introduces a unique
formulation inspired by weight normalization that constrains the L1-norm of
model weights according to accumulator bit width bounds that we derive. Thus,
in training QNNs for low-precision accumulation, A2Q also inherently promotes
unstructured weight sparsity to guarantee overflow avoidance. We apply our
method to deep learning-based computer vision tasks to show that A2Q can train
QNNs for low-precision accumulators while maintaining model accuracy
competitive with a floating-point baseline. In our evaluations, we consider the
impact of A2Q on both general-purpose platforms and programmable hardware.
However, we primarily target model deployment on FPGAs because they can be
programmed to fully exploit custom accumulator bit widths. Our experimentation
shows accumulator bit width significantly impacts the resource efficiency of
FPGA-based accelerators. On average across our benchmarks, A2Q offers up to a
2.3x reduction in resource utilization over 32-bit accumulator counterparts
with 99.2% of the floating-point model accuracy.
less
By: Lianke Qin, Zhao Song, Baocheng Sun
A rising trend in theoretical deep learning is to understand why deep
learning works through Neural Tangent Kernel (NTK) [jgh18], a kernel method
that is equivalent to using gradient descent to train a multi-layer
infinitely-wide neural network. NTK is a major step forward in the theoretical
deep learning because it allows researchers to use traditional mathematical
tools to analyze properties of deep neural networks and to explain various
... more
A rising trend in theoretical deep learning is to understand why deep
learning works through Neural Tangent Kernel (NTK) [jgh18], a kernel method
that is equivalent to using gradient descent to train a multi-layer
infinitely-wide neural network. NTK is a major step forward in the theoretical
deep learning because it allows researchers to use traditional mathematical
tools to analyze properties of deep neural networks and to explain various
neural network techniques from a theoretical view. A natural extension of NTK
on graph learning is \textit{Graph Neural Tangent Kernel (GNTK)}, and
researchers have already provide GNTK formulation for graph-level regression
and show empirically that this kernel method can achieve similar accuracy as
GNNs on various bioinformatics datasets [dhs+19]. The remaining question now is
whether solving GNTK regression is equivalent to training an infinite-wide
multi-layer GNN using gradient descent. In this paper, we provide three new
theoretical results. First, we formally prove this equivalence for graph-level
regression. Second, we present the first GNTK formulation for node-level
regression. Finally, we prove the equivalence for node-level regression.
less
By: Sruthi Gorantla, Eshaan Bhansali, Amit Deshpande, Anand Louis
In learning-to-rank (LTR), optimizing only the relevance (or the expected
ranking utility) can cause representational harm to certain categories of
items. Moreover, if there is implicit bias in the relevance scores, LTR models
may fail to optimize for true relevance. Previous works have proposed efficient
algorithms to train stochastic ranking models that achieve fairness of exposure
to the groups ex-ante (or, in expectation), which may not... more
In learning-to-rank (LTR), optimizing only the relevance (or the expected
ranking utility) can cause representational harm to certain categories of
items. Moreover, if there is implicit bias in the relevance scores, LTR models
may fail to optimize for true relevance. Previous works have proposed efficient
algorithms to train stochastic ranking models that achieve fairness of exposure
to the groups ex-ante (or, in expectation), which may not guarantee
representation fairness to the groups ex-post, that is, after realizing a
ranking from the stochastic ranking model. Typically, ex-post fairness is
achieved by post-processing, but previous work does not train stochastic
ranking models that are aware of this post-processing.
In this paper, we propose a novel objective that maximizes expected relevance
only over those rankings that satisfy given representation constraints to
ensure ex-post fairness. Building upon recent work on an efficient sampler for
ex-post group-fair rankings, we propose a group-fair Plackett-Luce model and
show that it can be efficiently optimized for our objective in the LTR
framework.
Experiments on three real-world datasets show that our group-fair algorithm
guarantees fairness alongside usually having better relevance compared to the
LTR baselines. In addition, our algorithm also achieves better relevance than
post-processing baselines, which also ensures ex-post fairness. Further, when
implicit bias is injected into the training data, our algorithm typically
outperforms existing LTR baselines in relevance.
less
By: Hadi Esmaeilzadeh, Soroush Ghodrati, Andrew B. Kahng, Joon Kyung Kim, Sean Kinzer, Sayak Kundu, Rohan Mahapatra, Susmita Dey Manasi, Sachin Sapatnekar, Zhiang Wang, Ziqing Zeng
Parameterizable machine learning (ML) accelerators are the product of recent
breakthroughs in ML. To fully enable their design space exploration (DSE), we
propose a physical-design-driven, learning-based prediction framework for
hardware-accelerated deep neural network (DNN) and non-DNN ML algorithms. It
adopts a unified approach that combines backend power, performance, and area
(PPA) analysis with frontend performance simulation, thereby ... more
Parameterizable machine learning (ML) accelerators are the product of recent
breakthroughs in ML. To fully enable their design space exploration (DSE), we
propose a physical-design-driven, learning-based prediction framework for
hardware-accelerated deep neural network (DNN) and non-DNN ML algorithms. It
adopts a unified approach that combines backend power, performance, and area
(PPA) analysis with frontend performance simulation, thereby achieving a
realistic estimation of both backend PPA and system metrics such as runtime and
energy. In addition, our framework includes a fully automated DSE technique,
which optimizes backend and system metrics through an automated search of
architectural and backend parameters. Experimental studies show that our
approach consistently predicts backend PPA and system metrics with an average
7% or less prediction error for the ASIC implementation of two deep learning
accelerator platforms, VTA and VeriGOOD-ML, in both a commercial 12 nm process
and a research-oriented 45 nm process.
less
By: Kwan Ho Ryan Chan, Aditya Chattopadhyay, Benjamin David Haeffele, Rene Vidal
Variational Information Pursuit (V-IP) is a framework for making
interpretable predictions by design by sequentially selecting a short chain of
task-relevant, user-defined and interpretable queries about the data that are
most informative for the task. While this allows for built-in interpretability
in predictive models, applying V-IP to any task requires data samples with
dense concept-labeling by domain experts, limiting the application o... more
Variational Information Pursuit (V-IP) is a framework for making
interpretable predictions by design by sequentially selecting a short chain of
task-relevant, user-defined and interpretable queries about the data that are
most informative for the task. While this allows for built-in interpretability
in predictive models, applying V-IP to any task requires data samples with
dense concept-labeling by domain experts, limiting the application of V-IP to
small-scale tasks where manual data annotation is feasible. In this work, we
extend the V-IP framework with Foundational Models (FMs) to address this
limitation. More specifically, we use a two-step process, by first leveraging
Large Language Models (LLMs) to generate a sufficiently large candidate set of
task-relevant interpretable concepts, then using Large Multimodal Models to
annotate each data sample by semantic similarity with each concept in the
generated concept set. While other interpretable-by-design frameworks such as
Concept Bottleneck Models (CBMs) require an additional step of removing
repetitive and non-discriminative concepts to have good interpretability and
test performance, we mathematically and empirically justify that, with a
sufficiently informative and task-relevant query (concept) set, the proposed
FM+V-IP method does not require any type of concept filtering. In addition, we
show that FM+V-IP with LLM generated concepts can achieve better test
performance than V-IP with human annotated concepts, demonstrating the
effectiveness of LLMs at generating efficient query sets. Finally, when
compared to other interpretable-by-design frameworks such as CBMs, FM+V-IP can
achieve competitive test performance using fewer number of concepts/queries in
both cases with filtered or unfiltered concept sets.
less
By: Karlis Freivalds, Emils Ozolins, Guntis Barzdins
Integer factorization is a famous computational problem unknown whether being
solvable in the polynomial time. With the rise of deep neural networks, it is
interesting whether they can facilitate faster factorization. We present an
approach to factorization utilizing deep neural networks and discrete denoising
diffusion that works by iteratively correcting errors in a partially-correct
solution. To this end, we develop a new seq2seq neural ... more
Integer factorization is a famous computational problem unknown whether being
solvable in the polynomial time. With the rise of deep neural networks, it is
interesting whether they can facilitate faster factorization. We present an
approach to factorization utilizing deep neural networks and discrete denoising
diffusion that works by iteratively correcting errors in a partially-correct
solution. To this end, we develop a new seq2seq neural network architecture,
employ relaxed categorical distribution and adapt the reverse diffusion process
to cope better with inaccuracies in the denoising step. The approach is able to
find factors for integers of up to 56 bits long. Our analysis indicates that
investment in training leads to an exponential decrease of sampling steps
required at inference to achieve a given success rate, thus counteracting an
exponential run-time increase depending on the bit-length.
less
By: Paul M Baggenstoss
In this paper, we exploit the unique properties of a deterministic projected
belief network (D-PBN) to take full advantage of trainable compound activation
functions (TCAs). A D-PBN is a type of auto-encoder that operates by "backing
up" through a feed-forward neural network. TCAs are activation functions with
complex monotonic-increasing shapes that change the distribution of the data so
that the linear transformation that follows is more ... more
In this paper, we exploit the unique properties of a deterministic projected
belief network (D-PBN) to take full advantage of trainable compound activation
functions (TCAs). A D-PBN is a type of auto-encoder that operates by "backing
up" through a feed-forward neural network. TCAs are activation functions with
complex monotonic-increasing shapes that change the distribution of the data so
that the linear transformation that follows is more effective. Because a D-PBN
operates by "backing up", the TCAs are inverted in the reconstruction process,
restoring the original distribution of the data, thus taking advantage of a
given TCA in both analysis and reconstruction. In this paper, we show that a
D-PBN auto-encoder with TCAs can significantly out-perform standard
auto-encoders including variational auto-encoders.
less
By: Lorenzo Vorabbi, Davide Maltoni, Stefano Santi
Existing Continual Learning (CL) solutions only partially address the
constraints on power, memory and computation of the deep learning models when
deployed on low-power embedded CPUs. In this paper, we propose a CL solution
that embraces the recent advancements in CL field and the efficiency of the
Binary Neural Networks (BNN), that use 1-bit for weights and activations to
efficiently execute deep learning models. We propose a hybrid quant... more
Existing Continual Learning (CL) solutions only partially address the
constraints on power, memory and computation of the deep learning models when
deployed on low-power embedded CPUs. In this paper, we propose a CL solution
that embraces the recent advancements in CL field and the efficiency of the
Binary Neural Networks (BNN), that use 1-bit for weights and activations to
efficiently execute deep learning models. We propose a hybrid quantization of
CWR* (an effective CL approach) that considers differently forward and backward
pass in order to retain more precision during gradient update step and at the
same time minimizing the latency overhead. The choice of a binary network as
backbone is essential to meet the constraints of low power devices and, to the
best of authors' knowledge, this is the first attempt to prove on-device
learning with BNN. The experimental validation carried out confirms the
validity and the suitability of the proposed method.
less
Development and external validation of a lung cancer risk estimation tool using gradient-boosting
0upvotes
By: Pierre-Louis Benveniste, Julie Alberge, Lei Xing, Jean-Emmanuel Bibault
Lung cancer is a significant cause of mortality worldwide, emphasizing the
importance of early detection for improved survival rates. In this study, we
propose a machine learning (ML) tool trained on data from the PLCO Cancer
Screening Trial and validated on the NLST to estimate the likelihood of lung
cancer occurrence within five years. The study utilized two datasets, the PLCO
(n=55,161) and NLST (n=48,595), consisting of comprehensive in... more
Lung cancer is a significant cause of mortality worldwide, emphasizing the
importance of early detection for improved survival rates. In this study, we
propose a machine learning (ML) tool trained on data from the PLCO Cancer
Screening Trial and validated on the NLST to estimate the likelihood of lung
cancer occurrence within five years. The study utilized two datasets, the PLCO
(n=55,161) and NLST (n=48,595), consisting of comprehensive information on risk
factors, clinical measurements, and outcomes related to lung cancer. Data
preprocessing involved removing patients who were not current or former smokers
and those who had died of causes unrelated to lung cancer. Additionally, a
focus was placed on mitigating bias caused by censored data. Feature selection,
hyper-parameter optimization, and model calibration were performed using
XGBoost, an ensemble learning algorithm that combines gradient boosting and
decision trees. The ML model was trained on the pre-processed PLCO dataset and
tested on the NLST dataset. The model incorporated features such as age,
gender, smoking history, medical diagnoses, and family history of lung cancer.
The model was well-calibrated (Brier score=0.044). ROC-AUC was 82% on the PLCO
dataset and 70% on the NLST dataset. PR-AUC was 29% and 11% respectively. When
compared to the USPSTF guidelines for lung cancer screening, our model provided
the same recall with a precision of 13.1% vs. 9.3% on the PLCO dataset and 3.2%
vs. 3.1% on the NLST dataset. The developed ML tool provides a freely available
web application for estimating the likelihood of developing lung cancer within
five years. By utilizing risk factors and clinical data, individuals can assess
their risk and make informed decisions regarding lung cancer screening. This
research contributes to the efforts in early detection and prevention
strategies, aiming to reduce lung cancer-related mortality rates.
less
By: Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, Jimmy Ba
By conditioning on natural language instructions, large language models
(LLMs) have displayed impressive capabilities as general-purpose computers.
However, task performance depends significantly on the quality of the prompt
used to steer the model, and most effective prompts have been handcrafted by
humans. Inspired by classical program synthesis and the human approach to
prompt engineering, we propose Automatic Prompt Engineer (APE) for a... more
By conditioning on natural language instructions, large language models
(LLMs) have displayed impressive capabilities as general-purpose computers.
However, task performance depends significantly on the quality of the prompt
used to steer the model, and most effective prompts have been handcrafted by
humans. Inspired by classical program synthesis and the human approach to
prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic
instruction generation and selection. In our method, we treat the instruction
as the "program," optimized by searching over a pool of instruction candidates
proposed by an LLM in order to maximize a chosen score function. To evaluate
the quality of the selected instruction, we evaluate the zero-shot performance
of another LLM following the selected instruction. Experiments on 24 NLP tasks
show that our automatically generated instructions outperform the prior LLM
baseline by a large margin and achieve better or comparable performance to the
instructions generated by human annotators on 19/24 tasks. We conduct extensive
qualitative and quantitative analyses to explore the performance of APE. We
show that APE-engineered prompts can be applied to steer models toward
truthfulness and/or informativeness, as well as to improve few-shot learning
performance by simply prepending them to standard in-context learning prompts.
Please check out our webpage at
https://sites.google.com/view/automatic-prompt-engineer.
less
By: Rahul Goswami, Arabin Kr. Dey
The paper explores a different variation of combined regression strategy to
calculate the conditional survival function. We use regression based weak
learners to create the proposed ensemble technique. The proposed combined
regression strategy uses proximity measure as area between two survival curves.
The proposed model shows a construction which ensures that it performs better
than the Random Survival Forest. The paper discusses a novel t... more
The paper explores a different variation of combined regression strategy to
calculate the conditional survival function. We use regression based weak
learners to create the proposed ensemble technique. The proposed combined
regression strategy uses proximity measure as area between two survival curves.
The proposed model shows a construction which ensures that it performs better
than the Random Survival Forest. The paper discusses a novel technique to
select the most important variable in the combined regression setup. We perform
a simulation study to show that our proposition for finding relevance of the
variables works quite well. We also use three real-life datasets to illustrate
the model.
less
By: Yin Fang, Qiang Zhang, Haihong Yang, Xiang Zhuang, Shumin Deng, Wen Zhang, Ming Qin, Zhuo Chen, Xiaohui Fan, Huajun Chen
Molecular representation learning contributes to multiple downstream tasks
such as molecular property prediction and drug design. To properly represent
molecules, graph contrastive learning is a promising paradigm as it utilizes
self-supervision signals and has no requirements for human annotations.
However, prior works fail to incorporate fundamental domain knowledge into
graph semantics and thus ignore the correlations between atoms that ... more
Molecular representation learning contributes to multiple downstream tasks
such as molecular property prediction and drug design. To properly represent
molecules, graph contrastive learning is a promising paradigm as it utilizes
self-supervision signals and has no requirements for human annotations.
However, prior works fail to incorporate fundamental domain knowledge into
graph semantics and thus ignore the correlations between atoms that have common
attributes but are not directly connected by bonds. To address these issues, we
construct a Chemical Element Knowledge Graph (KG) to summarize microscopic
associations between elements and propose a novel Knowledge-enhanced
Contrastive Learning (KCL) framework for molecular representation learning. KCL
framework consists of three modules. The first module, knowledge-guided graph
augmentation, augments the original molecular graph based on the Chemical
Element KG. The second module, knowledge-aware graph representation, extracts
molecular representations with a common graph encoder for the original
molecular graph and a Knowledge-aware Message Passing Neural Network (KMPNN) to
encode complex information in the augmented molecular graph. The final module
is a contrastive objective, where we maximize agreement between these two views
of molecular graphs. Extensive experiments demonstrated that KCL obtained
superior performances against state-of-the-art baselines on eight molecular
datasets. Visualization experiments properly interpret what KCL has learned
from atoms and attributes in the augmented molecular graphs. Our codes and data
are available at https://github.com/ZJU-Fangyin/KCL.
less
By: Siyu Yi, Zhengyang Mao, Wei Ju, Yongdao Zhou, Luchen Liu, Xiao Luo, Ming Zhang
Graph classification, aiming at learning the graph-level representations for
effective class assignments, has received outstanding achievements, which
heavily relies on high-quality datasets that have balanced class distribution.
In fact, most real-world graph data naturally presents a long-tailed form,
where the head classes occupy much more samples than the tail classes, it thus
is essential to study the graph-level classification over lo... more
Graph classification, aiming at learning the graph-level representations for
effective class assignments, has received outstanding achievements, which
heavily relies on high-quality datasets that have balanced class distribution.
In fact, most real-world graph data naturally presents a long-tailed form,
where the head classes occupy much more samples than the tail classes, it thus
is essential to study the graph-level classification over long-tailed data
while still remaining largely unexplored. However, most existing long-tailed
learning methods in visions fail to jointly optimize the representation
learning and classifier training, as well as neglect the mining of the
hard-to-classify classes. Directly applying existing methods to graphs may lead
to sub-optimal performance, since the model trained on graphs would be more
sensitive to the long-tailed distribution due to the complex topological
characteristics. Hence, in this paper, we propose a novel long-tailed
graph-level classification framework via Collaborative Multi-expert Learning
(CoMe) to tackle the problem. To equilibrate the contributions of head and tail
classes, we first develop balanced contrastive learning from the view of
representation learning, and then design an individual-expert classifier
training based on hard class mining. In addition, we execute gated fusion and
disentangled knowledge distillation among the multiple experts to promote the
collaboration in a multi-expert framework. Comprehensive experiments are
performed on seven widely-used benchmark datasets to demonstrate the
superiority of our method CoMe over state-of-the-art baselines.
less
By: Zahra Taghiyarrenani, Abdallah Abdallah, Slawomir Nowaczyk, Sepideh Pashami
Federated Learning (FL) allows several clients to construct a common global
machine-learning model without having to share their data. FL, however, faces
the challenge of statistical heterogeneity between the client's data, which
degrades performance and slows down the convergence toward the global model. In
this paper, we provide theoretical proof that minimizing heterogeneity between
clients facilitates the convergence of a global model f... more
Federated Learning (FL) allows several clients to construct a common global
machine-learning model without having to share their data. FL, however, faces
the challenge of statistical heterogeneity between the client's data, which
degrades performance and slows down the convergence toward the global model. In
this paper, we provide theoretical proof that minimizing heterogeneity between
clients facilitates the convergence of a global model for every single client.
This becomes particularly important under empirical concept shifts among
clients, rather than merely considering imbalanced classes, which have been
studied until now. Therefore, we propose a method for knowledge transfer
between clients where the server trains client-specific generators. Each
generator generates samples for the corresponding client to remove the conflict
with other clients' models. Experiments conducted on synthetic and real data,
along with a theoretical study, support the effectiveness of our method in
constructing a well-generalizable global model by reducing the conflict between
local models.
less
By: Reza Mirzaeifard, Naveen K. D. Venkategowda, Stefan Werner
This paper addresses the problem of localization, which is inherently
non-convex and non-smooth in a federated setting where the data is distributed
across a multitude of devices. Due to the decentralized nature of federated
environments, distributed learning becomes essential for scalability and
adaptability. Moreover, these environments are often plagued by outlier data,
which presents substantial challenges to conventional methods, parti... more
This paper addresses the problem of localization, which is inherently
non-convex and non-smooth in a federated setting where the data is distributed
across a multitude of devices. Due to the decentralized nature of federated
environments, distributed learning becomes essential for scalability and
adaptability. Moreover, these environments are often plagued by outlier data,
which presents substantial challenges to conventional methods, particularly in
maintaining estimation accuracy and ensuring algorithm convergence. To mitigate
these challenges, we propose a method that adopts an $L_1$-norm robust
formulation within a distributed sub-gradient framework, explicitly designed to
handle these obstacles. Our approach addresses the problem in its original
form, without resorting to iterative simplifications or approximations,
resulting in enhanced computational efficiency and improved estimation
accuracy. We demonstrate that our method converges to a stationary point,
highlighting its effectiveness and reliability. Through numerical simulations,
we confirm the superior performance of our approach, notably in outlier-rich
environments, which surpasses existing state-of-the-art localization methods.
less
By: Sophie A. Neubauer née Gruenbacher, Radu Grosu
This paper presents, in a unified fashion, deterministic as well as
statistical Lagrangian-verification techniques. They formally quantify the
behavioral robustness of any time-continuous process, formulated as a
continuous-depth model. To this end, we review LRT-NG, SLR, and GoTube,
algorithms for constructing a tight reachtube, that is, an over-approximation
of the set of states reachable within a given time-horizon, and provide
guarantee... more
This paper presents, in a unified fashion, deterministic as well as
statistical Lagrangian-verification techniques. They formally quantify the
behavioral robustness of any time-continuous process, formulated as a
continuous-depth model. To this end, we review LRT-NG, SLR, and GoTube,
algorithms for constructing a tight reachtube, that is, an over-approximation
of the set of states reachable within a given time-horizon, and provide
guarantees for the reachtube bounds. We compare the usage of the variational
equations, associated to the system equations, the mean value theorem, and the
Lipschitz constants, in achieving deterministic and statistical guarantees. In
LRT-NG, the Lipschitz constant is used as a bloating factor of the initial
perturbation, to compute the radius of an ellipsoid in an optimal metric, which
over-approximates the set of reachable states. In SLR and GoTube, we get
statistical guarantees, by using the Lipschitz constants to compute local balls
around samples. These are needed to calculate the probability of having found
an upper bound, of the true maximum perturbation at every timestep. Our
experiments demonstrate the superior performance of Lagrangian techniques, when
compared to LRT, Flow*, and CAPD, and illustrate their use in the robustness
analysis of various continuous-depth models.
less
By: Lingbing Guo, Weiqing Wang, Zhuo Chen, Ningyu Zhang, Zequn Sun, Yixuan Lai, Qiang Zhang, Huajun Chen
Reasoning system dynamics is one of the most important analytical approaches
for many scientific studies. With the initial state of a system as input, the
recent graph neural networks (GNNs)-based methods are capable of predicting the
future state distant in time with high accuracy. Although these methods have
diverse designs in modeling the coordinates and interacting forces of the
system, we show that they actually share a common paradigm... more
Reasoning system dynamics is one of the most important analytical approaches
for many scientific studies. With the initial state of a system as input, the
recent graph neural networks (GNNs)-based methods are capable of predicting the
future state distant in time with high accuracy. Although these methods have
diverse designs in modeling the coordinates and interacting forces of the
system, we show that they actually share a common paradigm that learns the
integration of the velocity over the interval between the initial and terminal
coordinates. However, their integrand is constant w.r.t. time. Inspired by this
observation, we propose a new approach to predict the integration based on
several velocity estimations with Newton-Cotes formulas and prove its
effectiveness theoretically. Extensive experiments on several benchmarks
empirically demonstrate consistent and significant improvement compared with
the state-of-the-art methods.
less
By: Sayash Kapoor, Emily Cantrell, Kenny Peng, Thanh Hien Pham, Christopher A. Bail, Odd Erik Gundersen, Jake M. Hofman, Jessica Hullman, Michael A. Lones, Momin M. Malik, Priyanka Nanayakkara, Russell A. Poldrack, Inioluwa Deborah Raji, Michael Roberts, Matthew J. Salganik, Marta Serra-Garcia, Brandon M. Stewart, Gilles Vandewiele, Arvind Narayanan
Machine learning (ML) methods are proliferating in scientific research.
However, the adoption of these methods has been accompanied by failures of
validity, reproducibility, and generalizability. These failures can hinder
scientific progress, lead to false consensus around invalid claims, and
undermine the credibility of ML-based science. ML methods are often applied and
fail in similar ways across disciplines. Motivated by this observation... more
Machine learning (ML) methods are proliferating in scientific research.
However, the adoption of these methods has been accompanied by failures of
validity, reproducibility, and generalizability. These failures can hinder
scientific progress, lead to false consensus around invalid claims, and
undermine the credibility of ML-based science. ML methods are often applied and
fail in similar ways across disciplines. Motivated by this observation, our
goal is to provide clear reporting standards for ML-based science. Drawing from
an extensive review of past literature, we present the REFORMS checklist
($\textbf{Re}$porting Standards $\textbf{For}$ $\textbf{M}$achine Learning
Based $\textbf{S}$cience). It consists of 32 questions and a paired set of
guidelines. REFORMS was developed based on a consensus of 19 researchers across
computer science, data science, mathematics, social sciences, and biomedical
sciences. REFORMS can serve as a resource for researchers when designing and
implementing a study, for referees when reviewing papers, and for journals when
enforcing standards for transparency and reproducibility.
less
By: Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, Marc G. Bellemare
By: Hua Wang, Yuqiong Wu, Yushun Zhang, Fuqiang Lai, Zhou Feng, Bing Xie, Ailin Zhao
Logs are valuable information for oil and gas fields as they help to
determine the lithology of the formations surrounding the borehole and the
location and reserves of subsurface oil and gas reservoirs. However, important
logs are often missing in horizontal or old wells, which poses a challenge in
field applications. In this paper, we utilize data from the 2020 machine
learning competition of the SPWLA, which aims to predict the missing
c... more
Logs are valuable information for oil and gas fields as they help to
determine the lithology of the formations surrounding the borehole and the
location and reserves of subsurface oil and gas reservoirs. However, important
logs are often missing in horizontal or old wells, which poses a challenge in
field applications. In this paper, we utilize data from the 2020 machine
learning competition of the SPWLA, which aims to predict the missing
compressional wave slowness and shear wave slowness logs using other logs in
the same borehole. We employ the NGBoost algorithm to construct an Ensemble
Learning model that can predicate the results as well as their uncertainty.
Furthermore, we combine the SHAP method to investigate the interpretability of
the machine learning model. We compare the performance of the NGBosst model
with four other commonly used Ensemble Learning methods, including Random
Forest, GBDT, XGBoost, LightGBM. The results show that the NGBoost model
performs well in the testing set and can provide a probability distribution for
the prediction results. In addition, the variance of the probability
distribution of the predicted log can be used to justify the quality of the
constructed log. Using the SHAP explainable machine learning model, we
calculate the importance of each input log to the predicted results as well as
the coupling relationship among input logs. Our findings reveal that the
NGBoost model tends to provide greater slowness prediction results when the
neutron porosity and gamma ray are large, which is consistent with the
cognition of petrophysical models. Furthermore, the machine learning model can
capture the influence of the changing borehole caliper on slowness, where the
influence of borehole caliper on slowness is complex and not easy to establish
a direct relationship. These findings are in line with the physical principle
of borehole acoustics.
less
By: John Harshith, Mantej Singh Gill, Madhan Jothimani
There have been recent adversarial attacks that are difficult to find. These
new adversarial attacks methods may pose challenges to current deep learning
cyber defense systems and could influence the future defense of cyberattacks.
The authors focus on this domain in this research paper. They explore the
consequences of vulnerabilities in AI systems. This includes discussing how
they might arise, differences between randomized and adversari... more
There have been recent adversarial attacks that are difficult to find. These
new adversarial attacks methods may pose challenges to current deep learning
cyber defense systems and could influence the future defense of cyberattacks.
The authors focus on this domain in this research paper. They explore the
consequences of vulnerabilities in AI systems. This includes discussing how
they might arise, differences between randomized and adversarial examples and
also potential ethical implications of vulnerabilities. Moreover, it is
important to train the AI systems appropriately when they are in testing phase
and getting them ready for broader use.
less
By: Alfredo Reichlin, Giovanni Luca Marchetti, Hang Yin, Anastasiia Varava, Danica Kragic
We address the problem of learning representations from observations of a
scene involving an agent and an external object the agent interacts with. To
this end, we propose a representation learning framework extracting the
location in physical space of both the agent and the object from unstructured
observations of arbitrary nature. Our framework relies on the actions performed
by the agent as the only source of supervision, while assuming ... more
We address the problem of learning representations from observations of a
scene involving an agent and an external object the agent interacts with. To
this end, we propose a representation learning framework extracting the
location in physical space of both the agent and the object from unstructured
observations of arbitrary nature. Our framework relies on the actions performed
by the agent as the only source of supervision, while assuming that the object
is displaced by the agent via unknown dynamics. We provide a theoretical
foundation and formally prove that an ideal learner is guaranteed to infer an
isometric representation, disentangling the agent from the object and correctly
extracting their locations. We evaluate empirically our framework on a variety
of scenarios, showing that it outperforms vision-based approaches such as a
state-of-the-art keypoint extractor. We moreover demonstrate how the extracted
representations enable the agent to solve downstream tasks via reinforcement
learning in an efficient manner.
less
By: M. Andrecut
Predicting the dynamics of chaotic systems is one of the most challenging
tasks for neural networks, and machine learning in general. Here we aim to
predict the spatiotemporal chaotic dynamics of a high-dimensional non-linear
system. In our attempt we use the TensorFlow library, representing the state of
the art for deep neural networks training and prediction. While our results are
encouraging, and show that the dynamics of the considered ... more
Predicting the dynamics of chaotic systems is one of the most challenging
tasks for neural networks, and machine learning in general. Here we aim to
predict the spatiotemporal chaotic dynamics of a high-dimensional non-linear
system. In our attempt we use the TensorFlow library, representing the state of
the art for deep neural networks training and prediction. While our results are
encouraging, and show that the dynamics of the considered system can be
predicted for short time, we also indirectly discovered an unexpected and
undesirable behavior of the TensorFlow library. More specifically, the longer
term prediction of the system's chaotic behavior quickly deteriorates and blows
up due to the nondeterministic behavior of the TensorFlow library. Here we
provide numerical evidence of the short time prediction ability, and of the
longer term predictability blow up.
less
By: Mingxi Tan, Andong Tian, Ludovic Denoyer
Standard cooperative multi-agent reinforcement learning (MARL) methods aim to
find the optimal team cooperative policy to complete a task. However there may
exist multiple different ways of cooperating, which usually are very needed by
domain experts. Therefore, identifying a set of significantly different
policies can alleviate the task complexity for them. Unfortunately, there is a
general lack of effective policy diversity approaches spe... more
Standard cooperative multi-agent reinforcement learning (MARL) methods aim to
find the optimal team cooperative policy to complete a task. However there may
exist multiple different ways of cooperating, which usually are very needed by
domain experts. Therefore, identifying a set of significantly different
policies can alleviate the task complexity for them. Unfortunately, there is a
general lack of effective policy diversity approaches specifically designed for
the multi-agent domain. In this work, we propose a method called
Moment-Matching Policy Diversity to alleviate this problem. This method can
generate different team policies to varying degrees by formalizing the
difference between team policies as the difference in actions of selected
agents in different policies. Theoretically, we show that our method is a
simple way to implement a constrained optimization problem that regularizes the
difference between two trajectory distributions by using the maximum mean
discrepancy. The effectiveness of our approach is demonstrated on a challenging
team-based shooter.
less
By: Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, Haoxing Ren
The increasing popularity of large language models (LLMs) has paved the way
for their application in diverse domains. This paper proposes a benchmarking
framework tailored specifically for evaluating LLM performance in the context
of Verilog code generation for hardware design and verification. We present a
comprehensive evaluation dataset consisting of 156 problems from the Verilog
instructional website HDLBits. The evaluation set consists... more
The increasing popularity of large language models (LLMs) has paved the way
for their application in diverse domains. This paper proposes a benchmarking
framework tailored specifically for evaluating LLM performance in the context
of Verilog code generation for hardware design and verification. We present a
comprehensive evaluation dataset consisting of 156 problems from the Verilog
instructional website HDLBits. The evaluation set consists of a diverse set of
Verilog code generation tasks, ranging from simple combinational circuits to
complex finite state machines. The Verilog code completions can be
automatically tested for functional correctness by comparing the transient
simulation outputs of the generated design with a golden solution. We also
demonstrate that the Verilog code generation capability of pretrained language
models could be improved with supervised fine-tuning by bootstrapping with LLM
generated synthetic problem-code pairs.
less
By: Davide Scassola, Sebastiano Saccani, Ginevra Carbone, Luca Bortolussi
Score-based and diffusion models have emerged as effective approaches for
both conditional and unconditional generation. Still conditional generation is
based on either a specific training of a conditional model or classifier
guidance, which requires training a noise-dependent classifier, even when the
classifier for uncorrupted data is given. We propose an approach to sample from
unconditional score-based generative models enforcing arbitr... more
Score-based and diffusion models have emerged as effective approaches for
both conditional and unconditional generation. Still conditional generation is
based on either a specific training of a conditional model or classifier
guidance, which requires training a noise-dependent classifier, even when the
classifier for uncorrupted data is given. We propose an approach to sample from
unconditional score-based generative models enforcing arbitrary logical
constraints, without any additional training. Firstly, we show how to
manipulate the learned score in order to sample from an un-normalized
distribution conditional on a user-defined constraint. Then, we define a
flexible and numerically stable neuro-symbolic framework for encoding soft
logical constraints. Combining these two ingredients we obtain a general, but
approximate, conditional sampling algorithm. We further developed effective
heuristics aimed at improving the approximation. Finally, we show the
effectiveness of our approach for various types of constraints and data:
tabular data, images and time series.
less
By: Alexander Kleinsorge, Stefan Kupper, Alexander Fauck, Felix Rothe
We present a novel, fast (exponential rate adaption), ab initio
(hyper-parameter-free) gradient based optimizer algorithm. The main idea of the
method is to adapt the learning rate $\alpha$ by situational awareness, mainly
striving for orthogonal neighboring gradients. The method has a high success
and fast convergence rate and does not rely on hand-tuned parameters giving it
greater universality. It can be applied to problems of any dimens... more
We present a novel, fast (exponential rate adaption), ab initio
(hyper-parameter-free) gradient based optimizer algorithm. The main idea of the
method is to adapt the learning rate $\alpha$ by situational awareness, mainly
striving for orthogonal neighboring gradients. The method has a high success
and fast convergence rate and does not rely on hand-tuned parameters giving it
greater universality. It can be applied to problems of any dimensions n and
scales only linearly (of order O(n)) with the dimension of the problem. It
optimizes convex and non-convex continuous landscapes providing some kind of
gradient. In contrast to the Ada-family (AdaGrad, AdaMax, AdaDelta, Adam, etc.)
the method is rotation invariant: optimization path and performance are
independent of coordinate choices. The impressive performance is demonstrated
by extensive experiments on the MNIST benchmark data-set against
state-of-the-art optimizers. We name this new class of optimizers after its
core idea Exponential Learning Rate Adaption - ELRA. We present it in two
variants c2min and p2min with slightly different control. The authors strongly
believe that ELRA will open a completely new research direction for gradient
descent optimize.
less
By: Juan Lu, Mohammed Bennamoun, Jonathon Stewart, JasonK. Eshraghian, Yanbin Liu, Benjamin Chow, Frank M. Sanfilippo, Girish Dwivedi
Diagnostic investigation has an important role in risk stratification and
clinical decision making of patients with suspected and documented Coronary
Artery Disease (CAD). However, the majority of existing tools are primarily
focused on the selection of gatekeeper tests, whereas only a handful of systems
contain information regarding the downstream testing or treatment. We propose a
multi-task deep learning model to support risk stratificat... more
Diagnostic investigation has an important role in risk stratification and
clinical decision making of patients with suspected and documented Coronary
Artery Disease (CAD). However, the majority of existing tools are primarily
focused on the selection of gatekeeper tests, whereas only a handful of systems
contain information regarding the downstream testing or treatment. We propose a
multi-task deep learning model to support risk stratification and down-stream
test selection for patients undergoing Coronary Computed Tomography Angiography
(CCTA). The analysis included 14,021 patients who underwent CCTA between 2006
and 2017. Our novel multitask deep learning framework extends the state-of-the
art Perceiver model to deal with real-world CCTA report data. Our model
achieved an Area Under the receiver operating characteristic Curve (AUC) of
0.76 in CAD risk stratification, and 0.72 AUC in predicting downstream tests.
Our proposed deep learning model can accurately estimate the likelihood of CAD
and provide recommended downstream tests based on prior CCTA data. In clinical
practice, the utilization of such an approach could bring a paradigm shift in
risk stratification and downstream management. Despite significant progress
using deep learning models for tabular data, they do not outperform gradient
boosting decision trees, and further research is required in this area.
However, neural networks appear to benefit more readily from multi-task
learning than tree-based models. This could offset the shortcomings of using
single task learning approach when working with tabular data.
less
By: Zhijie Deng, Peng Cui, Jun Zhu
Mislabeled, duplicated, or biased data in real-world scenarios can lead to
prolonged training and even hinder model convergence. Traditional solutions
prioritizing easy or hard samples lack the flexibility to handle such a variety
simultaneously. Recent work has proposed a more reasonable data selection
principle by examining the data's impact on the model's generalization loss.
However, its practical adoption relies on less principled appr... more
Mislabeled, duplicated, or biased data in real-world scenarios can lead to
prolonged training and even hinder model convergence. Traditional solutions
prioritizing easy or hard samples lack the flexibility to handle such a variety
simultaneously. Recent work has proposed a more reasonable data selection
principle by examining the data's impact on the model's generalization loss.
However, its practical adoption relies on less principled approximations and
additional clean holdout data. This work solves these problems by leveraging a
lightweight Bayesian treatment and incorporating off-the-shelf zero-shot
predictors built on large-scale pre-trained models. The resulting algorithm is
efficient and easy-to-implement. We perform extensive empirical studies on
challenging benchmarks with considerable data noise and imbalance in the online
batch selection scenario, and observe superior training efficiency over
competitive baselines. Notably, on the challenging WebVision benchmark, our
method can achieve similar predictive performance with significantly fewer
training iterations than leading data selection methods.
less
By: Sadia Kamal, Jimmy Hartford, Jeremy Willis, Arunkumar Bagavathi
Quantification of the political leaning of online news articles can aid in
understanding the dynamics of political ideology in social groups and measures
to mitigating them. However, predicting the accurate political leaning of a
news article with machine learning models is a challenging task. This is due to
(i) the political ideology of a news article is defined by several factors, and
(ii) the innate nature of existing learning models to ... more
Quantification of the political leaning of online news articles can aid in
understanding the dynamics of political ideology in social groups and measures
to mitigating them. However, predicting the accurate political leaning of a
news article with machine learning models is a challenging task. This is due to
(i) the political ideology of a news article is defined by several factors, and
(ii) the innate nature of existing learning models to be biased with the
political bias of the news publisher during the model training. There is only a
limited number of methods to study the political leaning of news articles which
also do not consider the algorithmic political bias which lowers the
generalization of machine learning models to predict the political leaning of
news articles published by any new news publishers. In this work, we propose a
knowledge-infused deep learning model that utilizes relatively reliable
external data resources to learn unbiased representations of news articles
using their global and local contexts. We evaluate the proposed model by
setting the data in such a way that news domains or news publishers in the test
set are completely unseen during the training phase. With this setup we show
that the proposed model mitigates algorithmic political bias and outperforms
baseline methods to predict the political leaning of news articles with up to
73% accuracy.
less
By: Edouard Yvinec, Arnaud Dapogny, Kevin Bailly
Quantization has become a crucial step for the efficient deployment of deep
neural networks, where floating point operations are converted to simpler fixed
point operations. In its most naive form, it simply consists in a combination
of scaling and rounding transformations, leading to either a limited
compression rate or a significant accuracy drop. Recently, Gradient-based
post-training quantization (GPTQ) methods appears to be constitute ... more
Quantization has become a crucial step for the efficient deployment of deep
neural networks, where floating point operations are converted to simpler fixed
point operations. In its most naive form, it simply consists in a combination
of scaling and rounding transformations, leading to either a limited
compression rate or a significant accuracy drop. Recently, Gradient-based
post-training quantization (GPTQ) methods appears to be constitute a suitable
trade-off between such simple methods and more powerful, yet expensive
Quantization-Aware Training (QAT) approaches, particularly when attempting to
quantize LLMs, where scalability of the quantization process is of paramount
importance. GPTQ essentially consists in learning the rounding operation using
a small calibration set. In this work, we challenge common choices in GPTQ
methods. In particular, we show that the process is, to a certain extent,
robust to a number of variables (weight selection, feature augmentation, choice
of calibration set). More importantly, we derive a number of best practices for
designing more efficient and scalable GPTQ methods, regarding the problem
formulation (loss, degrees of freedom, use of non-uniform quantization schemes)
or optimization process (choice of variable and optimizer). Lastly, we propose
a novel importance-based mixed-precision technique. Those guidelines lead to
significant performance improvements on all the tested state-of-the-art GPTQ
methods and networks (e.g. +6.819 points on ViT for 4-bit quantization), paving
the way for the design of scalable, yet effective quantization methods.
less
By: Binh Duong Nguyen, Pavlo Potapenko, Aytekin Dermici, Kishan Govinda, Stefan Sandfeld
Determining, understanding, and predicting the so-called structure-property
relation is an important task in many scientific disciplines, such as
chemistry, biology, meteorology, physics, engineering, and materials science.
Structure refers to the spatial distribution of, e.g., substances, material, or
matter in general, while property is a resulting characteristic that usually
depends in a non-trivial way on spatial details of the structur... more
Determining, understanding, and predicting the so-called structure-property
relation is an important task in many scientific disciplines, such as
chemistry, biology, meteorology, physics, engineering, and materials science.
Structure refers to the spatial distribution of, e.g., substances, material, or
matter in general, while property is a resulting characteristic that usually
depends in a non-trivial way on spatial details of the structure.
Traditionally, forward simulations models have been used for such tasks.
Recently, several machine learning algorithms have been applied in these
scientific fields to enhance and accelerate simulation models or as surrogate
models. In this work, we develop and investigate the applications of six
machine learning techniques based on two different datasets from the domain of
materials science: data from a two-dimensional Ising model for predicting the
formation of magnetic domains and data representing the evolution of dual-phase
microstructures from the Cahn-Hilliard model. We analyze the accuracy and
robustness of all models and elucidate the reasons for the differences in their
performances. The impact of including domain knowledge through tailored
features is studied, and general recommendations based on the availability and
quality of training data are derived from this.
less
By: David Levin, Gonen Singer
This paper introduces a novel graph-based filter method for automatic feature
selection (abbreviated as GB-AFS) for multi-class classification tasks. The
method determines the minimum combination of features required to sustain
prediction performance while maintaining complementary discriminating abilities
between different classes. It does not require any user-defined parameters such
as the number of features to select. The methodology emp... more
This paper introduces a novel graph-based filter method for automatic feature
selection (abbreviated as GB-AFS) for multi-class classification tasks. The
method determines the minimum combination of features required to sustain
prediction performance while maintaining complementary discriminating abilities
between different classes. It does not require any user-defined parameters such
as the number of features to select. The methodology employs the
Jeffries-Matusita (JM) distance in conjunction with t-distributed Stochastic
Neighbor Embedding (t-SNE) to generate a low-dimensional space reflecting how
effectively each feature can differentiate between each pair of classes. The
minimum number of features is selected using our newly developed Mean
Simplified Silhouette (abbreviated as MSS) index, designed to evaluate the
clustering results for the feature selection task. Experimental results on
public data sets demonstrate the superior performance of the proposed GB-AFS
over other filter-based techniques and automatic feature selection approaches.
Moreover, the proposed algorithm maintained the accuracy achieved when
utilizing all features, while using only $7\%$ to $30\%$ of the features.
Consequently, this resulted in a reduction of the time needed for
classifications, from $15\%$ to $70\%$.
less
By: Zhenjiang Mao, Carson Sobolewski, Ivan Ruchkin
End-to-end learning has emerged as a major paradigm for developing autonomous
systems. Unfortunately, with its performance and convenience comes an even
greater challenge of safety assurance. A key factor of this challenge is the
absence of the notion of a low-dimensional and interpretable dynamical state,
around which traditional assurance methods revolve. Focusing on the online
safety prediction problem, this paper proposes a configurable... more
End-to-end learning has emerged as a major paradigm for developing autonomous
systems. Unfortunately, with its performance and convenience comes an even
greater challenge of safety assurance. A key factor of this challenge is the
absence of the notion of a low-dimensional and interpretable dynamical state,
around which traditional assurance methods revolve. Focusing on the online
safety prediction problem, this paper proposes a configurable family of
learning pipelines based on generative world models, which do not require
low-dimensional states. To implement these pipelines, we overcome the
challenges of learning safety-informed latent representations and missing
safety labels under prediction-induced distribution shift. These pipelines come
with statistical calibration guarantees on their safety chance predictions
based on conformal prediction. We perform an extensive evaluation of the
proposed learning pipelines on two case studies of image-controlled systems: a
racing car and a cartpole.
less
By: Milan Ganai, Haichen Li, Theodore Enns, Yida Wang, Randy Huang
An important challenge in Machine Learning compilers like XLA is multi-pass
optimization and analysis. There has been recent interest chiefly in XLA
target-dependent optimization on the graph-level, subgraph-level, and
kernel-level phases. We specifically focus on target-independent optimization
XLA HLO pass ordering: our approach aims at finding the optimal sequence of
compiler optimization passes, which is decoupled from target-dependent
... more
An important challenge in Machine Learning compilers like XLA is multi-pass
optimization and analysis. There has been recent interest chiefly in XLA
target-dependent optimization on the graph-level, subgraph-level, and
kernel-level phases. We specifically focus on target-independent optimization
XLA HLO pass ordering: our approach aims at finding the optimal sequence of
compiler optimization passes, which is decoupled from target-dependent
optimization. However, there is little domain specific study in pass ordering
for XLA HLO. To this end, we propose introducing deep Reinforcement Learning
(RL) based search for optimal XLA HLO pass ordering. We also propose
enhancements to the deep RL algorithms to further improve optimal search
performance and open the research direction for domain-specific guidance for
RL. We create an XLA Gym experimentation framework as a tool to enable RL
algorithms to interact with the compiler for passing optimizations and thereby
train agents. Overall, in our experimentation we observe an average of $13.3\%$
improvement in operation count reduction on a benchmark of GPT-2 training
graphs and $10.4\%$ improvement on a diverse benchmark including GPT-2, BERT,
and ResNet graphs using the proposed approach over the compiler's default phase
ordering.
less
By: Ziwei Zhang, Haoyang Li, Zeyang Zhang, Yijian Qin, Xin Wang, Wenwu Zhu
Large models have emerged as the most recent groundbreaking achievements in
artificial intelligence, and particularly machine learning. However, when it
comes to graphs, large models have not achieved the same level of success as in
other fields, such as natural language processing and computer vision. In order
to promote applying large models for graphs forward, we present a perspective
paper to discuss the challenges and opportunities ass... more
Large models have emerged as the most recent groundbreaking achievements in
artificial intelligence, and particularly machine learning. However, when it
comes to graphs, large models have not achieved the same level of success as in
other fields, such as natural language processing and computer vision. In order
to promote applying large models for graphs forward, we present a perspective
paper to discuss the challenges and opportunities associated with developing
large graph models. First, we discuss the desired characteristics of large
graph models. Then, we present detailed discussions from three key
perspectives: representation basis, graph data, and graph models. In each
category, we provide a brief overview of recent advances and highlight the
remaining challenges together with our visions. Finally, we discuss valuable
applications of large graph models. We believe this perspective paper is able
to encourage further investigations into large graph models, ultimately pushing
us one step closer towards artificial general intelligence (AGI).
less
By: Michael Kleinman, Alessandro Achille, Stefano Soatto
Critical learning periods are periods early in development where temporary
sensory deficits can have a permanent effect on behavior and learned
representations. Despite the radical differences between biological and
artificial networks, critical learning periods have been empirically observed
in both systems. This suggests that critical periods may be fundamental to
learning and not an accident of biology. Yet, why exactly critical periods
... more
Critical learning periods are periods early in development where temporary
sensory deficits can have a permanent effect on behavior and learned
representations. Despite the radical differences between biological and
artificial networks, critical learning periods have been empirically observed
in both systems. This suggests that critical periods may be fundamental to
learning and not an accident of biology. Yet, why exactly critical periods
emerge in deep networks is still an open question, and in particular it is
unclear whether the critical periods observed in both systems depend on
particular architectural or optimization details. To isolate the key underlying
factors, we focus on deep linear network models, and show that, surprisingly,
such networks also display much of the behavior seen in biology and artificial
networks, while being amenable to analytical treatment. We show that critical
periods depend on the depth of the model and structure of the data
distribution. We also show analytically and in simulations that the learning of
features is tied to competition between sources. Finally, we extend our
analysis to multi-task learning to show that pre-training on certain tasks can
damage the transfer performance on new tasks, and show how this depends on the
relationship between tasks and the duration of the pre-training stage. To the
best of our knowledge, our work provides the first analytically tractable model
that sheds light into why critical learning periods emerge in biological and
artificial networks.
less
By: Miguel Folgado, Veronica Sanz, Johannes Hirn, Edgar Lorenzo-Saez, Javier Urchueguia
Urban traffic emissions represent a significant concern due to their
detrimental impacts on both public health and the environment. Consequently,
decision-makers have flagged their reduction as a crucial goal. In this study,
we first analyze the correlation between traffic flux and pollution in the city
of Valencia, Spain. Our results demonstrate that traffic has a significant
impact on the levels of certain pollutants (especially $\text{NO... more
Urban traffic emissions represent a significant concern due to their
detrimental impacts on both public health and the environment. Consequently,
decision-makers have flagged their reduction as a crucial goal. In this study,
we first analyze the correlation between traffic flux and pollution in the city
of Valencia, Spain. Our results demonstrate that traffic has a significant
impact on the levels of certain pollutants (especially $\text{NO}_\text{x}$).
Secondly, we develop an alarm system to predict if a street is likely to
experience unusually high traffic in the next 30 minutes, using an independent
three-tier level for each street. To make the predictions, we use traffic data
updated every 10 minutes and Long Short-Term Memory (LSTM) neural networks. We
trained the LSTM using traffic data from 2018, and tested it using traffic data
from 2019.
less
By: Mingyuan Zhang, Ambuj Tewari
In online ranking, a learning algorithm sequentially ranks a set of items and
receives feedback on its ranking in the form of relevance scores. Since
obtaining relevance scores typically involves human annotation, it is of great
interest to consider a partial feedback setting where feedback is restricted to
the top-$k$ items in the rankings. Chaudhuri and Tewari [2017] developed a
framework to analyze online ranking algorithms with top $k$ ... more
In online ranking, a learning algorithm sequentially ranks a set of items and
receives feedback on its ranking in the form of relevance scores. Since
obtaining relevance scores typically involves human annotation, it is of great
interest to consider a partial feedback setting where feedback is restricted to
the top-$k$ items in the rankings. Chaudhuri and Tewari [2017] developed a
framework to analyze online ranking algorithms with top $k$ feedback. A key
element in their work was the use of techniques from partial monitoring. In
this paper, we further investigate online ranking with top $k$ feedback and
solve some open problems posed by Chaudhuri and Tewari [2017]. We provide a
full characterization of minimax regret rates with the top $k$ feedback model
for all $k$ and for the following ranking performance measures: Pairwise Loss,
Discounted Cumulative Gain, and Precision@n. In addition, we give an efficient
algorithm that achieves the minimax regret rate for Precision@n.
less
By: Jingpu Cheng, Qianxiao Li, Ting Lin, Zuowei Shen
We investigate the expressive power of deep residual neural networks
idealized as continuous dynamical systems through control theory. Specifically,
we consider two properties that arise from supervised learning, namely
universal interpolation - the ability to match arbitrary input and target
training samples - and the closely related notion of universal approximation -
the ability to approximate input-target functional relationships via fl... more
We investigate the expressive power of deep residual neural networks
idealized as continuous dynamical systems through control theory. Specifically,
we consider two properties that arise from supervised learning, namely
universal interpolation - the ability to match arbitrary input and target
training samples - and the closely related notion of universal approximation -
the ability to approximate input-target functional relationships via flow maps.
Under the assumption of affine invariance of the control family, we give a
characterisation of universal interpolation, showing that it holds for
essentially any architecture with non-linearity. Furthermore, we elucidate the
relationship between universal interpolation and universal approximation in the
context of general control systems, showing that the two properties cannot be
deduced from each other. At the same time, we identify conditions on the
control family and the target function that ensures the equivalence of the two
notions.
less
By: Dongjin Lee, Juho Lee, Kijung Shin
Real-world graphs are dynamic, constantly evolving with new interactions,
such as financial transactions in financial networks. Temporal Graph Neural
Networks (TGNNs) have been developed to effectively capture the evolving
patterns in dynamic graphs. While these models have demonstrated their
superiority, being widely adopted in various important fields, their
vulnerabilities against adversarial attacks remain largely unexplored. In this
pa... more
Real-world graphs are dynamic, constantly evolving with new interactions,
such as financial transactions in financial networks. Temporal Graph Neural
Networks (TGNNs) have been developed to effectively capture the evolving
patterns in dynamic graphs. While these models have demonstrated their
superiority, being widely adopted in various important fields, their
vulnerabilities against adversarial attacks remain largely unexplored. In this
paper, we propose T-SPEAR, a simple and effective adversarial attack method for
link prediction on continuous-time dynamic graphs, focusing on investigating
the vulnerabilities of TGNNs. Specifically, before the training procedure of a
victim model, which is a TGNN for link prediction, we inject edge perturbations
to the data that are unnoticeable in terms of the four constraints we propose,
and yet effective enough to cause malfunction of the victim model. Moreover, we
propose a robust training approach T-SHIELD to mitigate the impact of
adversarial attacks. By using edge filtering and enforcing temporal smoothness
to node embeddings, we enhance the robustness of the victim model. Our
experimental study shows that T-SPEAR significantly degrades the victim model's
performance on link prediction tasks, and even more, our attacks are
transferable to other TGNNs, which differ from the victim model assumed by the
attacker. Moreover, we demonstrate that T-SHIELD effectively filters out
adversarial edges and exhibits robustness against adversarial attacks,
surpassing the link prediction performance of the naive TGNN by up to 11.2%
under T-SPEAR.
less
By: Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, Christopher Ré
Recent advances in deep learning have relied heavily on the use of large
Transformers due to their ability to learn at scale. However, the core building
block of Transformers, the attention operator, exhibits quadratic cost in
sequence length, limiting the amount of context accessible. Existing
subquadratic methods based on low-rank and sparse approximations need to be
combined with dense attention layers to match Transformers, indicating a... more
Recent advances in deep learning have relied heavily on the use of large
Transformers due to their ability to learn at scale. However, the core building
block of Transformers, the attention operator, exhibits quadratic cost in
sequence length, limiting the amount of context accessible. Existing
subquadratic methods based on low-rank and sparse approximations need to be
combined with dense attention layers to match Transformers, indicating a gap in
capability. In this work, we propose Hyena, a subquadratic drop-in replacement
for attention constructed by interleaving implicitly parametrized long
convolutions and data-controlled gating. In recall and reasoning tasks on
sequences of thousands to hundreds of thousands of tokens, Hyena improves
accuracy by more than 50 points over operators relying on state-spaces and
other implicit and explicit methods, matching attention-based models. We set a
new state-of-the-art for dense-attention-free architectures on language
modeling in standard datasets (WikiText103 and The Pile), reaching Transformer
quality with a 20% reduction in training compute required at sequence length
2K. Hyena operators are twice as fast as highly optimized attention at sequence
length 8K, and 100x faster at sequence length 64K.
less
By: Tianyi Zhao, Hui Hu, Lu Cheng
Graph Neural Networks (GNNs) are powerful tools for learning representations
on graphs, such as social networks. However, their vulnerability to privacy
inference attacks restricts their practicality, especially in high-stake
domains. To address this issue, privacy-preserving GNNs have been proposed,
focusing on preserving node and/or link privacy. This work takes a step back
and investigates how GNNs contribute to privacy leakage. Through ... more
Graph Neural Networks (GNNs) are powerful tools for learning representations
on graphs, such as social networks. However, their vulnerability to privacy
inference attacks restricts their practicality, especially in high-stake
domains. To address this issue, privacy-preserving GNNs have been proposed,
focusing on preserving node and/or link privacy. This work takes a step back
and investigates how GNNs contribute to privacy leakage. Through theoretical
analysis and simulations, we identify message passing under structural bias as
the core component that allows GNNs to \textit{propagate} and \textit{amplify}
privacy leakage. Building upon these findings, we propose a principled
privacy-preserving GNN framework that effectively safeguards both node and link
privacy, referred to as dual-privacy preservation. The framework comprises
three major modules: a Sensitive Information Obfuscation Module that removes
sensitive information from node embeddings, a Dynamic Structure Debiasing
Module that dynamically corrects the structural bias, and an Adversarial
Learning Module that optimizes the privacy-utility trade-off. Experimental
results on four benchmark datasets validate the effectiveness of the proposed
model in protecting both node and link privacy while preserving high utility
for downstream tasks, such as node classification.
less
By: Guanhua Ye, Guanhua Ye, Quoc Viet Hung Nguyen, Hongzhi Yin
As some recent information security legislation endowed users with
unconditional rights to be forgotten by any trained machine learning model,
personalized IoT service providers have to put unlearning functionality into
their consideration. The most straightforward method to unlearn users'
contribution is to retrain the model from the initial state, which is not
realistic in high throughput applications with frequent unlearning requests.
Th... more
As some recent information security legislation endowed users with
unconditional rights to be forgotten by any trained machine learning model,
personalized IoT service providers have to put unlearning functionality into
their consideration. The most straightforward method to unlearn users'
contribution is to retrain the model from the initial state, which is not
realistic in high throughput applications with frequent unlearning requests.
Though some machine unlearning frameworks have been proposed to speed up the
retraining process, they fail to match decentralized learning scenarios. In
this paper, we design a decentralized unlearning framework called HDUS, which
uses distilled seed models to construct erasable ensembles for all clients.
Moreover, the framework is compatible with heterogeneous on-device models,
representing stronger scalability in real-world applications. Extensive
experiments on three real-world datasets show that our HDUS achieves
state-of-the-art performance.
less
By: Rüdiger Brecht, Dmytro R. Popovych, Alex Bihlo, Roman O. Popovych
Current physics-informed (standard or operator) neural networks still rely on
accurately learning the initial conditions of the system they are solving. In
contrast, standard numerical methods evolve such initial conditions without
needing to learn these. In this study, we propose to improve current
physics-informed deep learning strategies such that initial conditions do not
need to be learned and are represented exactly in the predicted s... more
Current physics-informed (standard or operator) neural networks still rely on
accurately learning the initial conditions of the system they are solving. In
contrast, standard numerical methods evolve such initial conditions without
needing to learn these. In this study, we propose to improve current
physics-informed deep learning strategies such that initial conditions do not
need to be learned and are represented exactly in the predicted solution.
Moreover, this method guarantees that when a DeepONet is applied multiple times
to time step a solution, the resulting function is continuous.
less
By: Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, Ramana Kumar
One of the most surprising puzzles in neural network generalisation is
grokking: a network with perfect training accuracy but poor generalisation
will, upon further training, transition to perfect generalisation. We propose
that grokking occurs when the task admits a generalising solution and a
memorising solution, where the generalising solution is slower to learn but
more efficient, producing larger logits with the same parameter norm. We... more
One of the most surprising puzzles in neural network generalisation is
grokking: a network with perfect training accuracy but poor generalisation
will, upon further training, transition to perfect generalisation. We propose
that grokking occurs when the task admits a generalising solution and a
memorising solution, where the generalising solution is slower to learn but
more efficient, producing larger logits with the same parameter norm. We
hypothesise that memorising circuits become more inefficient with larger
training datasets while generalising circuits do not, suggesting there is a
critical dataset size at which memorisation and generalisation are equally
efficient. We make and confirm four novel predictions about grokking, providing
significant evidence in favour of our explanation. Most strikingly, we
demonstrate two novel and surprising behaviours: ungrokking, in which a network
regresses from perfect to low test accuracy, and semi-grokking, in which a
network shows delayed generalisation to partial rather than perfect test
accuracy.
less
By: Emmanuel Balogun, Robert Buechler, Ram Rajagopal, Arun Majumdar
Stochastic generators are useful for estimating climate impacts on various sectors. Projecting climate risk in various sectors, e.g. energy systems, requires generators that are accurate (statistical resemblance to ground-truth), reliable (do not produce erroneous examples), and efficient. Leveraging data from the North American Land Data Assimilation System, we introduce TemperatureGAN, a Generative Adversarial Network conditioned on month... more
Stochastic generators are useful for estimating climate impacts on various sectors. Projecting climate risk in various sectors, e.g. energy systems, requires generators that are accurate (statistical resemblance to ground-truth), reliable (do not produce erroneous examples), and efficient. Leveraging data from the North American Land Data Assimilation System, we introduce TemperatureGAN, a Generative Adversarial Network conditioned on months, locations, and time periods, to generate 2m above ground atmospheric temperatures at an hourly resolution. We propose evaluation methods and metrics to measure the quality of generated samples. We show that TemperatureGAN produces high-fidelity examples with good spatial representation and temporal dynamics consistent with known diurnal cycles.
less
By: Baijiong Lin, Weisen Jiang, Feiyang Ye, Yu Zhang, Pengguang Chen, Ying-Cong Chen, Shu Liu
Multi-task learning (MTL), a learning paradigm to learn multiple related
tasks simultaneously, has achieved great success in various fields. However,
task-balancing remains a significant challenge in MTL, with the disparity in
loss/gradient scales often leading to performance compromises. In this paper,
we propose a Scale-Invariant Multi-Task Learning (SI-MTL) method to alleviate
the task-balancing problem from both loss and gradient perspe... more
Multi-task learning (MTL), a learning paradigm to learn multiple related
tasks simultaneously, has achieved great success in various fields. However,
task-balancing remains a significant challenge in MTL, with the disparity in
loss/gradient scales often leading to performance compromises. In this paper,
we propose a Scale-Invariant Multi-Task Learning (SI-MTL) method to alleviate
the task-balancing problem from both loss and gradient perspectives.
Specifically, SI-MTL contains a logarithm transformation which is performed on
all task losses to ensure scale-invariant at the loss level, and a gradient
balancing method, SI-G, which normalizes all task gradients to the same
magnitude as the maximum gradient norm. Extensive experiments conducted on
several benchmark datasets consistently demonstrate the effectiveness of SI-G
and the state-of-the-art performance of SI-MTL.
less
By: Eran Malach
Large language models display remarkable capabilities in logical and
mathematical reasoning, allowing them to solve complex tasks. Interestingly,
these abilities emerge in networks trained on the simple task of next-token
prediction. In this work, we present a theoretical framework for studying
auto-regressive next-token predictors. We demonstrate that even simple models
such as linear next-token predictors, trained on Chain-of-Thought (CoT... more
Large language models display remarkable capabilities in logical and
mathematical reasoning, allowing them to solve complex tasks. Interestingly,
these abilities emerge in networks trained on the simple task of next-token
prediction. In this work, we present a theoretical framework for studying
auto-regressive next-token predictors. We demonstrate that even simple models
such as linear next-token predictors, trained on Chain-of-Thought (CoT) data,
can approximate any function efficiently computed by a Turing machine. We
introduce a new complexity measure -- length complexity -- which measures the
number of intermediate tokens in a CoT sequence required to approximate some
target function, and analyze the interplay between length complexity and other
notions of complexity. Finally, we show experimentally that simple next-token
predictors, such as linear networks and shallow Multi-Layer Perceptrons (MLPs),
display non-trivial performance on text generation and arithmetic tasks. Our
results demonstrate that the power of language models can be attributed, to a
great extent, to the auto-regressive next-token training scheme, and not
necessarily to a particular choice of architecture.
less
By: Marin Vlastelica, Sebastian Blaes, Cristina Pineri, Georg Martius
We introduce a simple but effective method for managing risk in model-based
reinforcement learning with trajectory sampling that involves probabilistic
safety constraints and balancing of optimism in the face of epistemic
uncertainty and pessimism in the face of aleatoric uncertainty of an ensemble
of stochastic neural networks.Various experiments indicate that the separation
of uncertainties is essential to performing well with data-driven... more
We introduce a simple but effective method for managing risk in model-based
reinforcement learning with trajectory sampling that involves probabilistic
safety constraints and balancing of optimism in the face of epistemic
uncertainty and pessimism in the face of aleatoric uncertainty of an ensemble
of stochastic neural networks.Various experiments indicate that the separation
of uncertainties is essential to performing well with data-driven MPC
approaches in uncertain and safety-critical control environments.
less
By: Zebang Shen, Jiayuan Ye, Anmin Kang, Hamed Hassani, Reza Shokri
Repeated parameter sharing in federated learning causes significant
information leakage about private data, thus defeating its main purpose: data
privacy. Mitigating the risk of this information leakage, using state of the
art differentially private algorithms, also does not come for free. Randomized
mechanisms can prevent convergence of models on learning even the useful
representation functions, especially if there is more disagreement be... more
Repeated parameter sharing in federated learning causes significant
information leakage about private data, thus defeating its main purpose: data
privacy. Mitigating the risk of this information leakage, using state of the
art differentially private algorithms, also does not come for free. Randomized
mechanisms can prevent convergence of models on learning even the useful
representation functions, especially if there is more disagreement between
local models on the classification functions (due to data heterogeneity). In
this paper, we consider a representation federated learning objective that
encourages various parties to collaboratively refine the consensus part of the
model, with differential privacy guarantees, while separately allowing
sufficient freedom for local personalization (without releasing it). We prove
that in the linear representation setting, while the objective is non-convex,
our proposed new algorithm \DPFEDREP\ converges to a ball centered around the
\emph{global optimal} solution at a linear rate, and the radius of the ball is
proportional to the reciprocal of the privacy budget. With this novel utility
analysis, we improve the SOTA utility-privacy trade-off for this problem by a
factor of $\sqrt{d}$, where $d$ is the input dimension. We empirically evaluate
our method with the image classification task on CIFAR10, CIFAR100, and EMNIST,
and observe a significant performance improvement over the prior work under the
same small privacy budget. The code can be found in this link:
https://github.com/shenzebang/CENTAUR-Privacy-Federated-Representation-Learning.
less
By: Chunchao Ma, Arthur Leroy, Mauricio Alvarez
Multi-output Gaussian processes (MOGPs) have been introduced to deal with
multiple tasks by exploiting the correlations between different outputs.
Generally, MOGPs models assume a flat correlation structure between the
outputs. However, such a formulation does not account for more elaborate
relationships, for instance, if several replicates were observed for each
output (which is a typical setting in biological experiments). This paper
prop... more
Multi-output Gaussian processes (MOGPs) have been introduced to deal with
multiple tasks by exploiting the correlations between different outputs.
Generally, MOGPs models assume a flat correlation structure between the
outputs. However, such a formulation does not account for more elaborate
relationships, for instance, if several replicates were observed for each
output (which is a typical setting in biological experiments). This paper
proposes an extension of MOGPs for hierarchical datasets (i.e. datasets for
which the relationships between observations can be represented within a tree
structure). Our model defines a tailored kernel function accounting for
hierarchical structures in the data to capture different levels of correlations
while leveraging the introduction of latent variables to express the underlying
dependencies between outputs through a dedicated kernel. This latter feature is
expected to significantly improve scalability as the number of tasks increases.
An extensive experimental study involving both synthetic and real-world data
from genomics and motion capture is proposed to support our claims.
less
Geodesic Mode Connectivity
0upvotes
By: Charlie Tan, Theodore Long, Sarah Zhao, Rudolf Laine
Mode connectivity is a phenomenon where trained models are connected by a
path of low loss. We reframe this in the context of Information Geometry, where
neural networks are studied as spaces of parameterized distributions with
curved geometry. We hypothesize that shortest paths in these spaces, known as
geodesics, correspond to mode-connecting paths in the loss landscape. We
propose an algorithm to approximate geodesics and demonstrate tha... more
Mode connectivity is a phenomenon where trained models are connected by a
path of low loss. We reframe this in the context of Information Geometry, where
neural networks are studied as spaces of parameterized distributions with
curved geometry. We hypothesize that shortest paths in these spaces, known as
geodesics, correspond to mode-connecting paths in the loss landscape. We
propose an algorithm to approximate geodesics and demonstrate that they achieve
mode connectivity.
less
By: Ishita Mediratta, Minqi Jiang, Jack Parker-Holder, Michael Dennis, Eugene Vinitsky, Tim Rocktäschel
A key challenge in training generally-capable agents is the design of
training tasks that facilitate broad generalization and robustness to
environment variations. This challenge motivates the problem setting of
Unsupervised Environment Design (UED), whereby a student agent trains on an
adaptive distribution of tasks proposed by a teacher agent. A pioneering
approach for UED is PAIRED, which uses reinforcement learning (RL) to train a
teach... more
A key challenge in training generally-capable agents is the design of
training tasks that facilitate broad generalization and robustness to
environment variations. This challenge motivates the problem setting of
Unsupervised Environment Design (UED), whereby a student agent trains on an
adaptive distribution of tasks proposed by a teacher agent. A pioneering
approach for UED is PAIRED, which uses reinforcement learning (RL) to train a
teacher policy to design tasks from scratch, making it possible to directly
generate tasks that are adapted to the agent's current capabilities. Despite
its strong theoretical backing, PAIRED suffers from a variety of challenges
that hinder its practical performance. Thus, state-of-the-art methods currently
rely on curation and mutation rather than generation of new tasks. In this
work, we investigate several key shortcomings of PAIRED and propose solutions
for each shortcoming. As a result, we make it possible for PAIRED to match or
exceed state-of-the-art methods, producing robust agents in several established
challenging procedurally-generated environments, including a partially-observed
maze navigation task and a continuous-control car racing environment. We
believe this work motivates a renewed emphasis on UED methods based on learned
models that directly generate challenging environments, potentially unlocking
more open-ended RL training and, as a result, more general agents.
less
By: Mojtaba Valizadeh, Philip John Gorinski, Ignacio Iacobacci, Martin Berger
We propose \emph{regular expression inference (REI)} as a challenge for
code/language modelling, and the wider machine learning community. REI is a
supervised machine learning (ML) and program synthesis task, and poses the
problem of finding minimal regular expressions from examples: Given two finite
sets of strings $P$ and $N$ and a cost function $\text{cost}(\cdot)$, the task
is to generate an expression $r$ that accepts all strings in $P... more
We propose \emph{regular expression inference (REI)} as a challenge for
code/language modelling, and the wider machine learning community. REI is a
supervised machine learning (ML) and program synthesis task, and poses the
problem of finding minimal regular expressions from examples: Given two finite
sets of strings $P$ and $N$ and a cost function $\text{cost}(\cdot)$, the task
is to generate an expression $r$ that accepts all strings in $P$ and rejects
all strings in $N$, while no other such expression $r'$ exists with
$\text{cost}(r')<\text{cost}(r)$.
REI has advantages as a challenge problem: (i) regular expressions are
well-known, widely used, and a natural idealisation of code; (ii) REI's
asymptotic worst-case complexity is well understood; (iii) REI has a small
number of easy to understand parameters (e.g.~$P$ or $N$ cardinality, string
lengths of examples, or the cost function); this lets us easily finetune
REI-hardness; (iv) REI is an unsolved problem for deep learning based ML.
Recently, an REI solver was implemented on GPUs, using program synthesis
techniques. This enabled, for the first time, fast generation of minimal
expressions for complex REI instances. Building on this advance, we generate
and publish the first large-scale datasets for REI, and devise and evaluate
several initial heuristic and machine learning baselines.
We invite the community to participate and explore ML methods that learn to
solve REI problems. We believe that progress in REI directly translates to
code/language modelling.
less
By: Zhe Chen
In recent years, the popular Transformer architecture has achieved great
success in many application areas, including natural language processing and
computer vision. Many existing works aim to reduce the computational and memory
complexity of the self-attention mechanism in the Transformer by trading off
performance. However, performance is key for the continuing success of the
Transformer. In this paper, a drop-in replacement for the self... more
In recent years, the popular Transformer architecture has achieved great
success in many application areas, including natural language processing and
computer vision. Many existing works aim to reduce the computational and memory
complexity of the self-attention mechanism in the Transformer by trading off
performance. However, performance is key for the continuing success of the
Transformer. In this paper, a drop-in replacement for the self-attention
mechanism in the Transformer, called the Extractor, is proposed. Experimental
results show that replacing the self-attention mechanism with the Extractor
improves the performance of the Transformer. Furthermore, the proposed
Extractor has the potential to run faster than the self-attention since it has
a much shorter critical path of computation. Additionally, the sequence
prediction problem in the context of text generation is formulated using
variable-length discrete-time Markov chains, and the Transformer is reviewed
based on our understanding.
less
By: Peizhen Bai, Xianyuan Liu, Haiping Lu
Molecular property prediction with deep learning has gained much attention
over the past years. Owing to the scarcity of labeled molecules, there has been
growing interest in self-supervised learning methods that learn generalizable
molecular representations from unlabeled data. Molecules are typically treated
as 2D topological graphs in modeling, but it has been discovered that their 3D
geometry is of great importance in determining molecu... more
Molecular property prediction with deep learning has gained much attention
over the past years. Owing to the scarcity of labeled molecules, there has been
growing interest in self-supervised learning methods that learn generalizable
molecular representations from unlabeled data. Molecules are typically treated
as 2D topological graphs in modeling, but it has been discovered that their 3D
geometry is of great importance in determining molecular functionalities. In
this paper, we propose the Geometry-aware line graph transformer (Galformer)
pre-training, a novel self-supervised learning framework that aims to enhance
molecular representation learning with 2D and 3D modalities. Specifically, we
first design a dual-modality line graph transformer backbone to encode the
topological and geometric information of a molecule. The designed backbone
incorporates effective structural encodings to capture graph structures from
both modalities. Then we devise two complementary pre-training tasks at the
inter and intra-modality levels. These tasks provide properly supervised
information and extract discriminative 2D and 3D knowledge from unlabeled
molecules. Finally, we evaluate Galformer against six state-of-the-art
baselines on twelve property prediction benchmarks via downstream fine-tuning.
Experimental results show that Galformer consistently outperforms all baselines
on both classification and regression tasks, demonstrating its effectiveness.
less
By: Jeongwhan Choi, Seoyoung Hong, Noseong Park, Sung-Bae Cho
Graph neural networks (GNNs) are one of the most popular research topics for
deep learning. GNN methods typically have been designed on top of the graph
signal processing theory. In particular, diffusion equations have been widely
used for designing the core processing layer of GNNs, and therefore they are
inevitably vulnerable to the notorious oversmoothing problem. Recently, a
couple of papers paid attention to reaction equations in conju... more
Graph neural networks (GNNs) are one of the most popular research topics for
deep learning. GNN methods typically have been designed on top of the graph
signal processing theory. In particular, diffusion equations have been widely
used for designing the core processing layer of GNNs, and therefore they are
inevitably vulnerable to the notorious oversmoothing problem. Recently, a
couple of papers paid attention to reaction equations in conjunctions with
diffusion equations. However, they all consider limited forms of reaction
equations. To this end, we present a reaction-diffusion equation-based GNN
method that considers all popular types of reaction equations in addition to
one special reaction equation designed by us. To our knowledge, our paper is
one of the most comprehensive studies on reaction-diffusion equation-based
GNNs. In our experiments with 9 datasets and 28 baselines, our method, called
GREAD, outperforms them in a majority of cases. Further synthetic data
experiments show that it mitigates the oversmoothing problem and works well for
various homophily rates.
less
By: Peiyan Zhang, Yuchen Yan, Chaozhuo Li, Senzhang Wang, Xing Xie, Sunghun Kim
Although Transformer has achieved great success in natural language process
and computer vision, it has difficulty generalizing to medium and large-scale
graph data for two important reasons: (i) High complexity. (ii) Failing to
capture the complex and entangled structure information. In graph
representation learning, Graph Neural Networks(GNNs) can fuse the graph
structure and node attributes but have limited receptive fields. Therefore, w... more
Although Transformer has achieved great success in natural language process
and computer vision, it has difficulty generalizing to medium and large-scale
graph data for two important reasons: (i) High complexity. (ii) Failing to
capture the complex and entangled structure information. In graph
representation learning, Graph Neural Networks(GNNs) can fuse the graph
structure and node attributes but have limited receptive fields. Therefore, we
question whether can we combine Transformers and GNNs to help each other. In
this paper, we propose a new model named TransGNN where the Transformer layer
and GNN layer are used alternately to improve each other. Specifically, to
expand the receptive field and disentangle the information aggregation from
edges, we propose using Transformer to aggregate more relevant nodes'
information to improve the message passing of GNNs. Besides, to capture the
graph structure information, we utilize positional encoding and make use of the
GNN layer to fuse the structure into node attributes, which improves the
Transformer in graph data. We also propose to sample the most relevant nodes
for Transformer and two efficient samples update strategies to lower the
complexity. At last, we theoretically prove that TransGNN is more expressive
than GNNs only with extra linear complexity. The experiments on eight datasets
corroborate the effectiveness of TransGNN on node and graph classification
tasks.
less
Sampling From Autoencoders' Latent Space via Quantization And Probability Mass Function Concepts
0upvotes
By: Aymene Mohammed Bouayed, Adrian Iaccovelli, David Naccache
In this study, we focus on sampling from the latent space of generative
models built upon autoencoders so as the reconstructed samples are lifelike
images. To do to, we introduce a novel post-training sampling algorithm rooted
in the concept of probability mass functions, coupled with a quantization
process. Our proposed algorithm establishes a vicinity around each latent
vector from the input data and then proceeds to draw samples from the... more
In this study, we focus on sampling from the latent space of generative
models built upon autoencoders so as the reconstructed samples are lifelike
images. To do to, we introduce a novel post-training sampling algorithm rooted
in the concept of probability mass functions, coupled with a quantization
process. Our proposed algorithm establishes a vicinity around each latent
vector from the input data and then proceeds to draw samples from these defined
neighborhoods. This strategic approach ensures that the sampled latent vectors
predominantly inhabit high-probability regions, which, in turn, can be
effectively transformed into authentic real-world images. A noteworthy point of
comparison for our sampling algorithm is the sampling technique based on
Gaussian mixture models (GMM), owing to its inherent capability to represent
clusters. Remarkably, we manage to improve the time complexity from the
previous $\mathcal{O}(n\times d \times k \times i)$ associated with GMM
sampling to a much more streamlined $\mathcal{O}(n\times d)$, thereby resulting
in substantial speedup during runtime. Moreover, our experimental results,
gauged through the Fr\'echet inception distance (FID) for image generation,
underscore the superior performance of our sampling algorithm across a diverse
range of models and datasets. On the MNIST benchmark dataset, our approach
outperforms GMM sampling by yielding a noteworthy improvement of up to $0.89$
in FID value. Furthermore, when it comes to generating images of faces and
ocular images, our approach showcases substantial enhancements with FID
improvements of $1.69$ and $0.87$ respectively, as compared to GMM sampling, as
evidenced on the CelebA and MOBIUS datasets. Lastly, we substantiate our
methodology's efficacy in estimating latent space distributions in contrast to
GMM sampling, particularly through the lens of the Wasserstein distance.
less
By: Collin Drent, Melvin Drent, Geert-Jan van Houtum
This paper addresses the benefits of pooling data for shared learning in
maintenance operations. We consider a set of systems subject to Poisson
degradation that are coupled through an a-priori unknown rate. Decision
problems involving these systems are high-dimensional Markov decision processes
(MDPs). We present a decomposition result that reduces such an MDP to
two-dimensional MDPs, enabling structural analyses and computations. We
lever... more
This paper addresses the benefits of pooling data for shared learning in
maintenance operations. We consider a set of systems subject to Poisson
degradation that are coupled through an a-priori unknown rate. Decision
problems involving these systems are high-dimensional Markov decision processes
(MDPs). We present a decomposition result that reduces such an MDP to
two-dimensional MDPs, enabling structural analyses and computations. We
leverage this decomposition to demonstrate that pooling data can lead to
significant cost reductions compared to not pooling.
less
Some notes concerning a generalized KMM-type optimization method for density ratio estimation
0upvotes
By: Cristian Daniel Alecsa
In the present paper we introduce new optimization algorithms for the task of
density ratio estimation. More precisely, we consider extending the well-known
KMM method using the construction of a suitable loss function, in order to
encompass more general situations involving the estimation of density ratio
with respect to subsets of the training data and test data, respectively. The
associated codes can be found at https://github.com/CDAlec... more
In the present paper we introduce new optimization algorithms for the task of
density ratio estimation. More precisely, we consider extending the well-known
KMM method using the construction of a suitable loss function, in order to
encompass more general situations involving the estimation of density ratio
with respect to subsets of the training data and test data, respectively. The
associated codes can be found at https://github.com/CDAlecsa/Generalized-KMM.
less
By: Xiao Shen, Shirui Pan, Kup-Sze Choi, Xi Zhou
Cross-network node classification (CNNC), which aims to classify nodes in a
label-deficient target network by transferring the knowledge from a source
network with abundant labels, draws increasing attention recently. To address
CNNC, we propose a domain-adaptive message passing graph neural network
(DM-GNN), which integrates graph neural network (GNN) with conditional
adversarial domain adaptation. DM-GNN is capable of learning informative... more
Cross-network node classification (CNNC), which aims to classify nodes in a
label-deficient target network by transferring the knowledge from a source
network with abundant labels, draws increasing attention recently. To address
CNNC, we propose a domain-adaptive message passing graph neural network
(DM-GNN), which integrates graph neural network (GNN) with conditional
adversarial domain adaptation. DM-GNN is capable of learning informative
representations for node classification that are also transferrable across
networks. Firstly, a GNN encoder is constructed by dual feature extractors to
separate ego-embedding learning from neighbor-embedding learning so as to
jointly capture commonality and discrimination between connected nodes.
Secondly, a label propagation node classifier is proposed to refine each node's
label prediction by combining its own prediction and its neighbors' prediction.
In addition, a label-aware propagation scheme is devised for the labeled source
network to promote intra-class propagation while avoiding inter-class
propagation, thus yielding label-discriminative source embeddings. Thirdly,
conditional adversarial domain adaptation is performed to take the
neighborhood-refined class-label information into account during adversarial
domain adaptation, so that the class-conditional distributions across networks
can be better matched. Comparisons with eleven state-of-the-art methods
demonstrate the effectiveness of the proposed DM-GNN.
less
By: Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, Itir Onal Ertugrul
Diffusion models have demonstrated impressive generative capabilities, but
their 'exposure bias' problem, described as the input mismatch between training
and sampling, lacks in-depth exploration. In this paper, we systematically
investigate the exposure bias problem in diffusion models by first analytically
modelling the sampling distribution, based on which we then attribute the
prediction error at each sampling step as the root cause of ... more
Diffusion models have demonstrated impressive generative capabilities, but
their 'exposure bias' problem, described as the input mismatch between training
and sampling, lacks in-depth exploration. In this paper, we systematically
investigate the exposure bias problem in diffusion models by first analytically
modelling the sampling distribution, based on which we then attribute the
prediction error at each sampling step as the root cause of the exposure bias
issue. Furthermore, we discuss potential solutions to this issue and propose an
intuitive metric for it. Along with the elucidation of exposure bias, we
propose a simple, yet effective, training-free method called Epsilon Scaling to
alleviate the exposure bias. We show that Epsilon Scaling explicitly moves the
sampling trajectory closer to the vector field learned in the training phase by
scaling down the network output (Epsilon), mitigating the input mismatch
between training and sampling. Experiments on various diffusion frameworks
(ADM, DDPM/DDIM, LDM), unconditional and conditional settings, and
deterministic vs. stochastic sampling verify the effectiveness of our method.
less
By: Jingwen Fu, Tao Yang, Yuwang Wang, Yan Lu, Nanning Zheng
In-context learning, i.e., learning from in-context samples, is an impressive
ability of Transformer. However, the mechanism driving the in-context learning
is not yet fully understood. In this study, we aim to investigate from an
underexplored perspective of representation learning. The representation is
more complex for in-context learning senario, where the representation can be
impacted by both model weights and in-context samples. We r... more
In-context learning, i.e., learning from in-context samples, is an impressive
ability of Transformer. However, the mechanism driving the in-context learning
is not yet fully understood. In this study, we aim to investigate from an
underexplored perspective of representation learning. The representation is
more complex for in-context learning senario, where the representation can be
impacted by both model weights and in-context samples. We refer the above two
conceptually aspects of representation as in-weight component and in-context
component, respectively. To study how the two components affect in-context
learning capabilities, we construct a novel synthetic task, making it possible
to device two probes, in-weights probe and in-context probe, to evaluate the
two components, respectively. We demonstrate that the goodness of in-context
component is highly related to the in-context learning performance, which
indicates the entanglement between in-context learning and representation
learning. Furthermore, we find that a good in-weights component can actually
benefit the learning of the in-context component, indicating that in-weights
learning should be the foundation of in-context learning. To further understand
the the in-context learning mechanism and importance of the in-weights
component, we proof by construction that a simple Transformer, which uses
pattern matching and copy-past mechanism to perform in-context learning, can
match the in-context learning performance with more complex, best tuned
Transformer under the perfect in-weights component assumption. In short, those
discoveries from representation learning perspective shed light on new
approaches to improve the in-context capacity.
less
By: Yu Shi, Dong-Dong Wu, Xin Geng, Min-Ling Zhang
Partial Label Learning (PLL) is a type of weakly supervised learning where
each training instance is assigned a set of candidate labels, but only one
label is the ground-truth. However, this idealistic assumption may not always
hold due to potential annotation inaccuracies, meaning the ground-truth may not
be present in the candidate label set. This is known as Unreliable Partial
Label Learning (UPLL) that introduces an additional complexit... more
Partial Label Learning (PLL) is a type of weakly supervised learning where
each training instance is assigned a set of candidate labels, but only one
label is the ground-truth. However, this idealistic assumption may not always
hold due to potential annotation inaccuracies, meaning the ground-truth may not
be present in the candidate label set. This is known as Unreliable Partial
Label Learning (UPLL) that introduces an additional complexity due to the
inherent unreliability and ambiguity of partial labels, often resulting in a
sub-optimal performance with existing methods. To address this challenge, we
propose the Unreliability-Robust Representation Learning framework (URRL) that
leverages unreliability-robust contrastive learning to help the model fortify
against unreliable partial labels effectively. Concurrently, we propose a dual
strategy that combines KNN-based candidate label set correction and
consistency-regularization-based label disambiguation to refine label quality
and enhance the ability of representation learning within the URRL framework.
Extensive experiments demonstrate that the proposed method outperforms
state-of-the-art PLL methods on various datasets with diverse degrees of
unreliability and ambiguity. Furthermore, we provide a theoretical analysis of
our approach from the perspective of the expectation maximization (EM)
algorithm. Upon acceptance, we pledge to make the code publicly accessible.
less
By: Preben M. Ness, Dusica Marijan, Sunanda Bose
Causal Neural Network models have shown high levels of robustness to
adversarial attacks as well as an increased capacity for generalisation tasks
such as few-shot learning and rare-context classification compared to
traditional Neural Networks. This robustness is argued to stem from the
disentanglement of causal and confounder input signals. However, no
quantitative study has yet measured the level of disentanglement achieved by
these type... more
Causal Neural Network models have shown high levels of robustness to
adversarial attacks as well as an increased capacity for generalisation tasks
such as few-shot learning and rare-context classification compared to
traditional Neural Networks. This robustness is argued to stem from the
disentanglement of causal and confounder input signals. However, no
quantitative study has yet measured the level of disentanglement achieved by
these types of causal models or assessed how this relates to their adversarial
robustness.
Existing causal disentanglement metrics are not applicable to deterministic
models trained on real-world datasets. We, therefore, utilise metrics of
content/style disentanglement from the field of Computer Vision to measure
different aspects of the causal disentanglement for four state-of-the-art
causal Neural Network models. By re-implementing these models with a common
ResNet18 architecture we are able to fairly measure their adversarial
robustness on three standard image classification benchmarking datasets under
seven common white-box attacks. We find a strong association (r=0.820, p=0.001)
between the degree to which models decorrelate causal and confounder signals
and their adversarial robustness. Additionally, we find a moderate negative
association between the pixel-level information content of the confounder
signal and adversarial robustness (r=-0.597, p=0.040).
less
By: Hadrien Hendrikx, Paul Mangold, Aurélien Bellet
The Gaussian Mechanism (GM), which consists in adding Gaussian noise to a
vector-valued query before releasing it, is a standard privacy protection
mechanism. In particular, given that the query respects some L2 sensitivity
property (the L2 distance between outputs on any two neighboring inputs is
bounded), GM guarantees R\'enyi Differential Privacy (RDP). Unfortunately,
precisely bounding the L2 sensitivity can be hard, thus leading to loo... more
The Gaussian Mechanism (GM), which consists in adding Gaussian noise to a
vector-valued query before releasing it, is a standard privacy protection
mechanism. In particular, given that the query respects some L2 sensitivity
property (the L2 distance between outputs on any two neighboring inputs is
bounded), GM guarantees R\'enyi Differential Privacy (RDP). Unfortunately,
precisely bounding the L2 sensitivity can be hard, thus leading to loose
privacy bounds. In this work, we consider a Relative L2 sensitivity assumption,
in which the bound on the distance between two query outputs may also depend on
their norm. Leveraging this assumption, we introduce the Relative Gaussian
Mechanism (RGM), in which the variance of the noise depends on the norm of the
output. We prove tight bounds on the RDP parameters under relative L2
sensitivity, and characterize the privacy loss incurred by using
output-dependent noise. In particular, we show that RGM naturally adapts to a
latent variable that would control the norm of the output. Finally, we
instantiate our framework to show tight guarantees for Private Gradient
Descent, a problem that naturally fits our relative L2 sensitivity assumption.
less
By: Zixuan Liu, Liu Liu, Xueqian Wang, Peilin Zhao
Differentiable optimization has received a significant amount of attention
due to its foundational role in the domain of machine learning based on neural
networks. The existing methods leverages the optimality conditions and implicit
function theorem to obtain the Jacobian matrix of the output, which increases
the computational cost and limits the application of differentiable
optimization. In addition, some non-differentiable constraints l... more
Differentiable optimization has received a significant amount of attention
due to its foundational role in the domain of machine learning based on neural
networks. The existing methods leverages the optimality conditions and implicit
function theorem to obtain the Jacobian matrix of the output, which increases
the computational cost and limits the application of differentiable
optimization. In addition, some non-differentiable constraints lead to more
challenges when using prior differentiable optimization layers. This paper
proposes a differentiable layer, named Differentiable Frank-Wolfe Layer
(DFWLayer), by rolling out the Frank-Wolfe method, a well-known optimization
algorithm which can solve constrained optimization problems without projections
and Hessian matrix computations, thus leading to a efficient way of dealing
with large-scale problems. Theoretically, we establish a bound on the
suboptimality gap of the DFWLayer in the context of l1-norm constraints.
Experimental assessments demonstrate that the DFWLayer not only attains
competitive accuracy in solutions and gradients but also consistently adheres
to constraints. Moreover, it surpasses the baselines in both forward and
backward computational speeds.
less
By: Bikram Adhikari
Risky drivers account for 70% of fatal accidents in the United States. With
recent advances in sensors and intelligent vehicular systems, there has been
significant research on assessing driver behavior to improve driving
experiences and road safety. This paper examines the various techniques used to
analyze driver behavior using visual and vehicular data, providing an overview
of the latest research in this field. The paper also discusses ... more
Risky drivers account for 70% of fatal accidents in the United States. With
recent advances in sensors and intelligent vehicular systems, there has been
significant research on assessing driver behavior to improve driving
experiences and road safety. This paper examines the various techniques used to
analyze driver behavior using visual and vehicular data, providing an overview
of the latest research in this field. The paper also discusses the challenges
and open problems in the field and offers potential recommendations for future
research. The survey concludes that integrating vision and vehicular
information can significantly enhance the accuracy and effectiveness of driver
behavior analysis, leading to improved safety measures and reduced traffic
accidents.
less
By: Jiali Wang, Yuning Jiang, Xin Liu, Ting Wang, Yuanming Shi
In this paper, we investigate federated contextual linear bandit learning
within a wireless system that comprises a server and multiple devices. Each
device interacts with the environment, selects an action based on the received
reward, and sends model updates to the server. The primary objective is to
minimize cumulative regret across all devices within a finite time horizon. To
reduce the communication overhead, devices communicate with t... more
In this paper, we investigate federated contextual linear bandit learning
within a wireless system that comprises a server and multiple devices. Each
device interacts with the environment, selects an action based on the received
reward, and sends model updates to the server. The primary objective is to
minimize cumulative regret across all devices within a finite time horizon. To
reduce the communication overhead, devices communicate with the server via
over-the-air computation (AirComp) over noisy fading channels, where the
channel noise may distort the signals. In this context, we propose a customized
federated linear bandits scheme, where each device transmits an analog signal,
and the server receives a superposition of these signals distorted by channel
noise. A rigorous mathematical analysis is conducted to determine the regret
bound of the proposed scheme. Both theoretical analysis and numerical
experiments demonstrate the competitive performance of our proposed scheme in
terms of regret bounds in various settings.
less
Layer-wise Feedback Propagation
0upvotes
By: Leander Weber, Jim Berend, Alexander Binder, Thomas Wiegand, Wojciech Samek, Sebastian Lapuschkin
In this paper, we present Layer-wise Feedback Propagation (LFP), a novel
training approach for neural-network-like predictors that utilizes
explainability, specifically Layer-wise Relevance Propagation(LRP), to assign
rewards to individual connections based on their respective contributions to
solving a given task. This differs from traditional gradient descent, which
updates parameters towards anestimated loss minimum. LFP distributes a re... more
In this paper, we present Layer-wise Feedback Propagation (LFP), a novel
training approach for neural-network-like predictors that utilizes
explainability, specifically Layer-wise Relevance Propagation(LRP), to assign
rewards to individual connections based on their respective contributions to
solving a given task. This differs from traditional gradient descent, which
updates parameters towards anestimated loss minimum. LFP distributes a reward
signal throughout the model without the need for gradient computations. It then
strengthens structures that receive positive feedback while reducingthe
influence of structures that receive negative feedback. We establish the
convergence of LFP theoretically and empirically, and demonstrate its
effectiveness in achieving comparable performance to gradient descent on
various models and datasets. Notably, LFP overcomes certain limitations
associated with gradient-based methods, such as reliance on meaningful
derivatives. We further investigate how the different LRP-rules can be extended
to LFP, what their effects are on training, as well as potential applications,
such as training models with no meaningful derivatives, e.g., step-function
activated Spiking Neural Networks (SNNs), or for transfer learning, to
efficiently utilize existing knowledge.
less
By: Haksoo Lim, Sewon Park, Minjung Kim, Jaehoon Lee, Seonkyu Lim, Noseong Park
The time-series anomaly detection is one of the most fundamental tasks for
time-series. Unlike the time-series forecasting and classification, the
time-series anomaly detection typically requires unsupervised (or
self-supervised) training since collecting and labeling anomalous observations
are difficult. In addition, most existing methods resort to limited forms of
anomaly measurements and therefore, it is not clear whether they are optima... more
The time-series anomaly detection is one of the most fundamental tasks for
time-series. Unlike the time-series forecasting and classification, the
time-series anomaly detection typically requires unsupervised (or
self-supervised) training since collecting and labeling anomalous observations
are difficult. In addition, most existing methods resort to limited forms of
anomaly measurements and therefore, it is not clear whether they are optimal in
all circumstances. To this end, we present a multivariate time-series anomaly
detector based on score-based generative models, called MadSGM, which considers
the broadest ever set of anomaly measurement factors: i) reconstruction-based,
ii) density-based, and iii) gradient-based anomaly measurements. We also design
a conditional score network and its denoising score matching loss for the
time-series anomaly detection. Experiments on five real-world benchmark
datasets illustrate that MadSGM achieves the most robust and accurate
predictions.
less
By: George Trimponias, Thomas G. Dietterich
Exogenous state variables and rewards can slow reinforcement learning by injecting uncontrolled variation into the reward signal. This paper formalizes exogenous state variables and rewards and shows that if the reward function decomposes additively into endogenous and exogenous components, the MDP can be decomposed into an exogenous Markov Reward Process (based on the exogenous
reward) and an endogenous Markov Decision Process (optimizing ... more
Exogenous state variables and rewards can slow reinforcement learning by injecting uncontrolled variation into the reward signal. This paper formalizes exogenous state variables and rewards and shows that if the reward function decomposes additively into endogenous and exogenous components, the MDP can be decomposed into an exogenous Markov Reward Process (based on the exogenous
reward) and an endogenous Markov Decision Process (optimizing the endogenous reward). Any optimal policy for the endogenous MDP is also an optimal policy for the original MDP, but because the endogenous reward typically has reduced variance, the endogenous MDP is easier to solve. We study settings where the decomposition of the state space into exogenous and endogenous state spaces is
not given but must be discovered. The paper introduces and proves correctness of algorithms for discovering the exogenous and endogenous subspaces of the state space when they are mixed through linear combination. These algorithms can be applied during reinforcement learning to discover the exogenous space,
remove the exogenous reward, and focus reinforcement learning on the endogenous MDP. Experiments on a variety of challenging synthetic MDPs show that these methods, applied online, discover large exogenous state spaces and produce substantial speedups in reinforcement learning.
less
By: Puning Zhao, Fei Yu, Zhiguo Wan
Federated learning systems are susceptible to adversarial attacks. To combat
this, we introduce a novel aggregator based on Huber loss minimization, and
provide a comprehensive theoretical analysis. Under independent and identically
distributed (i.i.d) assumption, our approach has several advantages compared to
existing methods. Firstly, it has optimal dependence on $\epsilon$, which
stands for the ratio of attacked clients. Secondly, our a... more
Federated learning systems are susceptible to adversarial attacks. To combat
this, we introduce a novel aggregator based on Huber loss minimization, and
provide a comprehensive theoretical analysis. Under independent and identically
distributed (i.i.d) assumption, our approach has several advantages compared to
existing methods. Firstly, it has optimal dependence on $\epsilon$, which
stands for the ratio of attacked clients. Secondly, our approach does not need
precise knowledge of $\epsilon$. Thirdly, it allows different clients to have
unequal data sizes. We then broaden our analysis to include non-i.i.d data,
such that clients have slightly different distributions.
less
Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis
0upvotes
By: Mehdi Zadem, Sergio Mover, Sao Mai Nguyen
Open-ended learning benefits immensely from the use of symbolic methods for
goal representation as they offer ways to structure knowledge for efficient and
transferable learning. However, the existing Hierarchical Reinforcement
Learning (HRL) approaches relying on symbolic reasoning are often limited as
they require a manual goal representation. The challenge in autonomously
discovering a symbolic goal representation is that it must preserv... more
Open-ended learning benefits immensely from the use of symbolic methods for
goal representation as they offer ways to structure knowledge for efficient and
transferable learning. However, the existing Hierarchical Reinforcement
Learning (HRL) approaches relying on symbolic reasoning are often limited as
they require a manual goal representation. The challenge in autonomously
discovering a symbolic goal representation is that it must preserve critical
information, such as the environment dynamics. In this paper, we propose a
developmental mechanism for goal discovery via an emergent representation that
abstracts (i.e., groups together) sets of environment states that have similar
roles in the task. We introduce a Feudal HRL algorithm that concurrently learns
both the goal representation and a hierarchical policy. The algorithm uses
symbolic reachability analysis for neural networks to approximate the
transition relation among sets of states and to refine the goal representation.
We evaluate our approach on complex navigation tasks, showing the learned
representation is interpretable, transferrable and results in data efficient
learning.
less
By: Qinghe Gao, Artur M. Schweidtmann
The transformation towards renewable energy and feedstock supply in the
chemical industry requires new conceptual process design approaches. Recently,
breakthroughs in artificial intelligence offer opportunities to accelerate this
transition. Specifically, deep reinforcement learning, a subclass of machine
learning, has shown the potential to solve complex decision-making problems and
aid sustainable process design. We survey state-of-the-a... more
The transformation towards renewable energy and feedstock supply in the
chemical industry requires new conceptual process design approaches. Recently,
breakthroughs in artificial intelligence offer opportunities to accelerate this
transition. Specifically, deep reinforcement learning, a subclass of machine
learning, has shown the potential to solve complex decision-making problems and
aid sustainable process design. We survey state-of-the-art research in
reinforcement learning for process design through three major elements: (i)
information representation, (ii) agent architecture, and (iii) environment and
reward. Moreover, we discuss perspectives on underlying challenges and
promising future works to unfold the full potential of reinforcement learning
for process design in chemical engineering.
less
By: Mengge Du, Longfeng Nie, Siyu Lou, Yuntian Chenc, Dongxiao Zhang
Unveiling the underlying governing equations of nonlinear dynamic systems
remains a significant challenge, especially when encountering noisy
observations and no prior knowledge available. This study proposes R-DISCOVER,
a framework designed to robustly uncover open-form partial differential
equations (PDEs) from limited and noisy data. The framework operates through
two alternating update processes: discovering and embedding. The discoveri... more
Unveiling the underlying governing equations of nonlinear dynamic systems
remains a significant challenge, especially when encountering noisy
observations and no prior knowledge available. This study proposes R-DISCOVER,
a framework designed to robustly uncover open-form partial differential
equations (PDEs) from limited and noisy data. The framework operates through
two alternating update processes: discovering and embedding. The discovering
phase employs symbolic representation and a reinforcement learning (RL)-guided
hybrid PDE generator to efficiently produce diverse open-form PDEs with tree
structures. A neural network-based predictive model fits the system response
and serves as the reward evaluator for the generated PDEs. PDEs with superior
fits are utilized to iteratively optimize the generator via the RL method and
the best-performing PDE is selected by a parameter-free stability metric. The
embedding phase integrates the initially identified PDE from the discovering
process as a physical constraint into the predictive model for robust training.
The traversal of PDE trees automates the construction of the computational
graph and the embedding process without human intervention. Numerical
experiments demonstrate our framework's capability to uncover governing
equations from nonlinear dynamic systems with limited and highly noisy data and
outperform other physics-informed neural network-based discovery methods. This
work opens new potential for exploring real-world systems with limited
understanding.
less
By: Andrea Corsini, Shanchieh Jay Yang
Machine learning (ML) has become increasingly popular in network intrusion
detection. However, ML-based solutions always respond regardless of whether the
input data reflects known patterns, a common issue across safety-critical
applications. While several proposals exist for detecting Out-Of-Distribution
(OOD) in other fields, it remains unclear whether these approaches can
effectively identify new forms of intrusions for network security.... more
Machine learning (ML) has become increasingly popular in network intrusion
detection. However, ML-based solutions always respond regardless of whether the
input data reflects known patterns, a common issue across safety-critical
applications. While several proposals exist for detecting Out-Of-Distribution
(OOD) in other fields, it remains unclear whether these approaches can
effectively identify new forms of intrusions for network security. New attacks,
not necessarily affecting overall distributions, are not guaranteed to be
clearly OOD as instead, images depicting new classes are in computer vision. In
this work, we investigate whether existing OOD detectors from other fields
allow the identification of unknown malicious traffic. We also explore whether
more discriminative and semantically richer embedding spaces within models,
such as those created with contrastive learning and multi-class tasks, benefit
detection. Our investigation covers a set of six OOD techniques that employ
different detection strategies. These techniques are applied to models trained
in various ways and subsequently exposed to unknown malicious traffic from the
same and different datasets (network environments). Our findings suggest that
existing detectors can identify a consistent portion of new malicious traffic,
and that improved embedding spaces enhance detection. We also demonstrate that
simple combinations of certain detectors can identify almost 100% of malicious
traffic in our tested scenarios.
less
By: Andy Gray, Alma Rahat, Tom Crick, Stephen Lindsay, Darren Wallace
Assessment is a crucial part of education. Traditional marking is a source of
inconsistencies and unconscious bias, placing a high cognitive load on the
assessors. An approach to address these issues is comparative judgement (CJ).
In CJ, the assessor is presented with a pair of items and is asked to select
the better one. Following a series of comparisons, a rank is derived using a
ranking model, for example, the BTM, based on the results. ... more
Assessment is a crucial part of education. Traditional marking is a source of
inconsistencies and unconscious bias, placing a high cognitive load on the
assessors. An approach to address these issues is comparative judgement (CJ).
In CJ, the assessor is presented with a pair of items and is asked to select
the better one. Following a series of comparisons, a rank is derived using a
ranking model, for example, the BTM, based on the results. While CJ is
considered a reliable method for marking, there are concerns around
transparency, and the ideal number of pairwise comparisons to generate a
reliable estimation of the rank order is not known. Additionally, there have
been attempts to generate a method of selecting pairs that should be compared
next in an informative manner, but some existing methods are known to have
created their own bias within results inflating the reliability metric used. As
a result, a random selection approach is usually deployed.
We propose a novel Bayesian approach to CJ (BCJ) for determining the ranks of
compared items alongside a new way to select the pairs to present to the
marker(s) using active learning (AL), addressing the key shortcomings of
traditional CJ. Furthermore, we demonstrate how the entire approach may provide
transparency by providing the user insights into how it is making its decisions
and, at the same time, being more efficient. Results from our experiments
confirm that the proposed BCJ combined with entropy-driven AL pair-selection
method is superior to other alternatives. We also find that the more
comparisons done, the more accurate BCJ becomes, which solves the issue the
current method has of the model deteriorating if too many comparisons are
performed. As our approach can generate the complete predicted rank
distribution for an item, we also show how this can be utilised in devising a
predicted grade, guided by the assessor.
less
By: Vahid Ghafouri, Vibhor Agarwal, Yong Zhang, Nishanth Sastry, Jose Such, Guillermo Suarez-Tangil
The introduction of ChatGPT and the subsequent improvement of Large Language
Models (LLMs) have prompted more and more individuals to turn to the use of
ChatBots, both for information and assistance with decision-making. However,
the information the user is after is often not formulated by these ChatBots
objectively enough to be provided with a definite, globally accepted answer.
Controversial topics, such as "religion", "gender identity"... more
The introduction of ChatGPT and the subsequent improvement of Large Language
Models (LLMs) have prompted more and more individuals to turn to the use of
ChatBots, both for information and assistance with decision-making. However,
the information the user is after is often not formulated by these ChatBots
objectively enough to be provided with a definite, globally accepted answer.
Controversial topics, such as "religion", "gender identity", "freedom of
speech", and "equality", among others, can be a source of conflict as partisan
or biased answers can reinforce preconceived notions or promote disinformation.
By exposing ChatGPT to such debatable questions, we aim to understand its level
of awareness and if existing models are subject to socio-political and/or
economic biases. We also aim to explore how AI-generated answers compare to
human ones. For exploring this, we use a dataset of a social media platform
created for the purpose of debating human-generated claims on polemic subjects
among users, dubbed Kialo.
Our results show that while previous versions of ChatGPT have had important
issues with controversial topics, more recent versions of ChatGPT
(gpt-3.5-turbo) are no longer manifesting significant explicit biases in
several knowledge areas. In particular, it is well-moderated regarding economic
aspects. However, it still maintains degrees of implicit libertarian leaning
toward right-winged ideals which suggest the need for increased moderation from
the socio-political point of view. In terms of domain knowledge on
controversial topics, with the exception of the "Philosophical" category,
ChatGPT is performing well in keeping up with the collective human level of
knowledge. Finally, we see that sources of Bing AI have slightly more tendency
to the center when compared to human answers. All the analyses we make are
generalizable to other types of biases and domains.
less
By: Wenjie Fu Huazhong University of Science and Technology, Huandong Wang Tsinghua University, Chen Gao Tsinghua University, Guanghua Liu Huazhong University of Science and Technology, Yong Li Tsinghua University, Tao Jiang Huazhong University of Science and Technology
Membership Inference Attack (MIA) identifies whether a record exists in a
machine learning model's training set by querying the model. MIAs on the
classic classification models have been well-studied, and recent works have
started to explore how to transplant MIA onto generative models. Our
investigation indicates that existing MIAs designed for generative models
mainly depend on the overfitting in target models. However, overfitting can be... more
Membership Inference Attack (MIA) identifies whether a record exists in a
machine learning model's training set by querying the model. MIAs on the
classic classification models have been well-studied, and recent works have
started to explore how to transplant MIA onto generative models. Our
investigation indicates that existing MIAs designed for generative models
mainly depend on the overfitting in target models. However, overfitting can be
avoided by employing various regularization techniques, whereas existing MIAs
demonstrate poor performance in practice. Unlike overfitting, memorization is
essential for deep learning models to attain optimal performance, making it a
more prevalent phenomenon. Memorization in generative models leads to an
increasing trend in the probability distribution of generating records around
the member record. Therefore, we propose a Probabilistic Fluctuation Assessing
Membership Inference Attack (PFAMI), a black-box MIA that infers memberships by
detecting these trends via analyzing the overall probabilistic fluctuations
around given records. We conduct extensive experiments across multiple
generative models and datasets, which demonstrate PFAMI can improve the attack
success rate (ASR) by about 27.9% when compared with the best baseline.
less
By: Joao Fonseca, Andrew Bell, Carlo Abrate, Francesco Bonchi, Julia Stoyanovich
Algorithmic systems are often called upon to assist in high-stakes decision
making. In light of this, algorithmic recourse, the principle wherein
individuals should be able to take action against an undesirable outcome made
by an algorithmic system, is receiving growing attention. The bulk of the
literature on algorithmic recourse to-date focuses primarily on how to provide
recourse to a single individual, overlooking a critical element: th... more
Algorithmic systems are often called upon to assist in high-stakes decision
making. In light of this, algorithmic recourse, the principle wherein
individuals should be able to take action against an undesirable outcome made
by an algorithmic system, is receiving growing attention. The bulk of the
literature on algorithmic recourse to-date focuses primarily on how to provide
recourse to a single individual, overlooking a critical element: the effects of
a continuously changing context. Disregarding these effects on recourse is a
significant oversight, since, in almost all cases, recourse consists of an
individual making a first, unfavorable attempt, and then being given an
opportunity to make one or several attempts at a later date - when the context
might have changed. This can create false expectations, as initial recourse
recommendations may become less reliable over time due to model drift and
competition for access to the favorable outcome between individuals.
In this work we propose an agent-based simulation framework for studying the
effects of a continuously changing environment on algorithmic recourse. In
particular, we identify two main effects that can alter the reliability of
recourse for individuals represented by the agents: (1) competition with other
agents acting upon recourse, and (2) competition with new agents entering the
environment. Our findings highlight that only a small set of specific
parameterizations result in algorithmic recourse that is reliable for agents
over time. Consequently, we argue that substantial additional work is needed to
understand recourse reliability over time, and to develop recourse methods that
reward agents' effort.
less
By: Yiqun Diao, Yutong Yang, Qinbin Li, Bingsheng He, Mian Lu
Relational datasets are widespread in real-world scenarios and are usually
delivered in a streaming fashion. This type of data stream can present unique
challenges, such as distribution drifts, outliers, emerging classes, and
changing features, which have recently been described as open environment
challenges for machine learning. While some work has been done on incremental
learning for data streams, their evaluations are mostly conducted ... more
Relational datasets are widespread in real-world scenarios and are usually
delivered in a streaming fashion. This type of data stream can present unique
challenges, such as distribution drifts, outliers, emerging classes, and
changing features, which have recently been described as open environment
challenges for machine learning. While some work has been done on incremental
learning for data streams, their evaluations are mostly conducted with manually
partitioned datasets. Moreover, while several real-world streaming datasets are
available, it is uncertain whether these open environment challenges are
prevalent and how existing incremental learning algorithms perform on real
datasets. To fill this gap, we develop an Open Environment Benchmark named
OEBench to evaluate open environment challenges in relational data streams.
Specifically, we investigate 55 real-world streaming datasets and establish
that open environment scenarios are indeed widespread in real-world datasets,
which presents significant challenges for stream learning algorithms. Through
benchmarks, we find that increased data quantity may not consistently enhance
the model accuracy when applied in open environment scenarios, where machine
learning models can be significantly compromised by distribution shifts,
anomalies, or untrustworthy data within real-world data streams. The current
techniques are insufficient in effectively mitigating these challenges posed by
open environments. Thus, it is promising to conduct more researches to address
real-world new challenges of open environment scenarios.
less
By: Yuanjiang Cao, Lina Yao, Julian McAuley, Quan Z. Sheng
Deep Generative AI has been a long-standing essential topic in the machine
learning community, which can impact a number of application areas like text
generation and computer vision. The major paradigm to train a generative model
is maximum likelihood estimation, which pushes the learner to capture and
approximate the target data distribution by decreasing the divergence between
the model distribution and the target distribution. This form... more
Deep Generative AI has been a long-standing essential topic in the machine
learning community, which can impact a number of application areas like text
generation and computer vision. The major paradigm to train a generative model
is maximum likelihood estimation, which pushes the learner to capture and
approximate the target data distribution by decreasing the divergence between
the model distribution and the target distribution. This formulation
successfully establishes the objective of generative tasks, while it is
incapable of satisfying all the requirements that a user might expect from a
generative model. Reinforcement learning, serving as a competitive option to
inject new training signals by creating new objectives that exploit novel
signals, has demonstrated its power and flexibility to incorporate human
inductive bias from multiple angles, such as adversarial learning,
hand-designed rules and learned reward model to build a performant model.
Thereby, reinforcement learning has become a trending research field and has
stretched the limits of generative AI in both model design and application. It
is reasonable to summarize and conclude advances in recent years with a
comprehensive review. Although there are surveys in different application areas
recently, this survey aims to shed light on a high-level review that spans a
range of application areas. We provide a rigorous taxonomy in this area and
make sufficient coverage on various models and applications. Notably, we also
surveyed the fast-developing large language model area. We conclude this survey
by showing the potential directions that might tackle the limit of current
models and expand the frontiers for generative AI.
less
By: Hado van Hasselt, Arthur Guez, David Silver
The popular Q-learning algorithm is known to overestimate action values under
certain conditions. It was not previously known whether, in practice, such
overestimations are common, whether they harm performance, and whether they can
generally be prevented. In this paper, we answer all these questions
affirmatively. In particular, we first show that the recent DQN algorithm,
which combines Q-learning with a deep neural network, suffers from ... more
The popular Q-learning algorithm is known to overestimate action values under
certain conditions. It was not previously known whether, in practice, such
overestimations are common, whether they harm performance, and whether they can
generally be prevented. In this paper, we answer all these questions
affirmatively. In particular, we first show that the recent DQN algorithm,
which combines Q-learning with a deep neural network, suffers from substantial
overestimations in some games in the Atari 2600 domain. We then show that the
idea behind the Double Q-learning algorithm, which was introduced in a tabular
setting, can be generalized to work with large-scale function approximation. We
propose a specific adaptation to the DQN algorithm and show that the resulting
algorithm not only reduces the observed overestimations, as hypothesized, but
that this also leads to much better performance on several games.
less
By: Boyu Chen, Hanxuan Chen, Jiao He, Fengyu Sun, Shangling Jui
We present a simple yet novel parameterized form of linear mapping to
achieves remarkable network compression performance: a pseudo SVD called
Ternary SVD (TSVD).
Unlike vanilla SVD, TSVD limits the $U$ and $V$ matrices in SVD to ternary
matrices form in $\{\pm 1, 0\}$. This means that instead of using the expensive
multiplication instructions, TSVD only requires addition instructions when
computing $U(\cdot)$ and $V(\cdot)$.
We provide... more
We present a simple yet novel parameterized form of linear mapping to
achieves remarkable network compression performance: a pseudo SVD called
Ternary SVD (TSVD).
Unlike vanilla SVD, TSVD limits the $U$ and $V$ matrices in SVD to ternary
matrices form in $\{\pm 1, 0\}$. This means that instead of using the expensive
multiplication instructions, TSVD only requires addition instructions when
computing $U(\cdot)$ and $V(\cdot)$.
We provide direct and training transition algorithms for TSVD like Post
Training Quantization and Quantization Aware Training respectively.
Additionally, we analyze the convergence of the direct transition algorithms in
theory.
In experiments, we demonstrate that TSVD can achieve state-of-the-art network
compression performance in various types of networks and tasks, including
current baseline models such as ConvNext, Swim, BERT, and large language model
like OPT.
less
By: Lai Wei, Zihao Jiang, Weiran Huang, Lichao Sun
Multimodal large language models acquire their instruction-following
capabilities through a two-stage training process: pre-training on image-text
pairs and fine-tuning on supervised vision-language instruction data. Recent
studies have shown that large language models can achieve satisfactory results
even with a limited amount of high-quality instruction-following data. In this
paper, we introduce InstructionGPT-4, which is fine-tuned on a... more
Multimodal large language models acquire their instruction-following
capabilities through a two-stage training process: pre-training on image-text
pairs and fine-tuning on supervised vision-language instruction data. Recent
studies have shown that large language models can achieve satisfactory results
even with a limited amount of high-quality instruction-following data. In this
paper, we introduce InstructionGPT-4, which is fine-tuned on a small dataset
comprising only 200 examples, amounting to approximately 6% of the
instruction-following data used in the alignment dataset for MiniGPT-4. We
first propose several metrics to access the quality of multimodal instruction
data. Based on these metrics, we present a simple and effective data selector
to automatically identify and filter low-quality vision-language data. By
employing this method, InstructionGPT-4 outperforms the original MiniGPT-4 on
various evaluations (e.g., visual question answering, GPT-4 preference).
Overall, our findings demonstrate that less but high-quality instruction tuning
data is efficient to enable multimodal large language models to generate better
output.
less
By: Cedric Sanders, Andreas Roth, Thomas Liebig
Over-squashing and over-smoothing are two critical issues, that limit the
capabilities of graph neural networks (GNNs). While over-smoothing eliminates
the differences between nodes making them indistinguishable, over-squashing
refers to the inability of GNNs to propagate information over long distances,
as exponentially many node states are squashed into fixed-size representations.
Both phenomena share similar causes, as both are largely i... more
Over-squashing and over-smoothing are two critical issues, that limit the
capabilities of graph neural networks (GNNs). While over-smoothing eliminates
the differences between nodes making them indistinguishable, over-squashing
refers to the inability of GNNs to propagate information over long distances,
as exponentially many node states are squashed into fixed-size representations.
Both phenomena share similar causes, as both are largely induced by the graph
topology. To mitigate these problems in graph classification tasks, we propose
CurvPool, a novel pooling method. CurvPool exploits the notion of curvature of
a graph to adaptively identify structures responsible for both over-smoothing
and over-squashing. By clustering nodes based on the Balanced Forman curvature,
CurvPool constructs a graph with a more suitable structure, allowing deeper
models and the combination of distant information. We compare it to other
state-of-the-art pooling approaches and establish its competitiveness in terms
of classification accuracy, computational complexity, and flexibility. CurvPool
outperforms several comparable methods across all considered tasks. The most
consistent results are achieved by pooling densely connected clusters using the
sum aggregation, as this allows additional information about the size of each
pool.
less
By: Marcos Eduardo Valle
Despite the many successful applications of deep learning models for
multidimensional signal and image processing, most traditional neural networks
process data represented by (multidimensional) arrays of real numbers. The
intercorrelation between feature channels is usually expected to be learned
from the training data, requiring numerous parameters and careful training. In
contrast, vector-valued neural networks are conceived to process a... more
Despite the many successful applications of deep learning models for
multidimensional signal and image processing, most traditional neural networks
process data represented by (multidimensional) arrays of real numbers. The
intercorrelation between feature channels is usually expected to be learned
from the training data, requiring numerous parameters and careful training. In
contrast, vector-valued neural networks are conceived to process arrays of
vectors and naturally consider the intercorrelation between feature channels.
Consequently, they usually have fewer parameters and often undergo more robust
training than traditional neural networks. This paper aims to present a broad
framework for vector-valued neural networks, referred to as V-nets. In this
context, hypercomplex-valued neural networks are regarded as vector-valued
models with additional algebraic properties. Furthermore, this paper explains
the relationship between vector-valued and traditional neural networks.
Precisely, a vector-valued neural network can be obtained by placing
restrictions on a real-valued model to consider the intercorrelation between
feature channels. Finally, we show how V-nets, including hypercomplex-valued
neural networks, can be implemented in current deep-learning libraries as
real-valued networks.
less
By: Lucas Correia, Jan-Christoph Goos, Philipp Klein, Thomas Bäck, Anna V. Kononova
A clear need for automatic anomaly detection applied to automotive testing
has emerged as more and more attention is paid to the data recorded and manual
evaluation by humans reaches its capacity. Such real-world data is massive,
diverse, multivariate and temporal in nature, therefore requiring modelling of
the testee behaviour. We propose a variational autoencoder with multi-head
attention (MA-VAE), which, when trained on unlabelled data, ... more
A clear need for automatic anomaly detection applied to automotive testing
has emerged as more and more attention is paid to the data recorded and manual
evaluation by humans reaches its capacity. Such real-world data is massive,
diverse, multivariate and temporal in nature, therefore requiring modelling of
the testee behaviour. We propose a variational autoencoder with multi-head
attention (MA-VAE), which, when trained on unlabelled data, not only provides
very few false positives but also manages to detect the majority of the
anomalies presented. In addition to that, the approach offers a novel way to
avoid the bypass phenomenon, an undesirable behaviour investigated in
literature. Lastly, the approach also introduces a new method to remap
individual windows to a continuous time series. The results are presented in
the context of a real-world industrial data set and several experiments are
undertaken to further investigate certain aspects of the proposed model. When
configured properly, it is 9% of the time wrong when an anomaly is flagged and
discovers 67% of the anomalies present. Also, MA-VAE has the potential to
perform well with only a fraction of the training and validation subset,
however, to extract it, a more sophisticated threshold estimation method is
required.
less
By: Hanchi Huang, Li Shen, Deheng Ye, Wei Liu
We propose a novel master-slave architecture to solve the top-$K$
combinatorial multi-armed bandits problem with non-linear bandit feedback and
diversity constraints, which, to the best of our knowledge, is the first
combinatorial bandits setting considering diversity constraints under bandit
feedback. Specifically, to efficiently explore the combinatorial and
constrained action space, we introduce six slave models with distinguished
merits... more
We propose a novel master-slave architecture to solve the top-$K$
combinatorial multi-armed bandits problem with non-linear bandit feedback and
diversity constraints, which, to the best of our knowledge, is the first
combinatorial bandits setting considering diversity constraints under bandit
feedback. Specifically, to efficiently explore the combinatorial and
constrained action space, we introduce six slave models with distinguished
merits to generate diversified samples well balancing rewards and constraints
as well as efficiency. Moreover, we propose teacher learning based optimization
and the policy co-training technique to boost the performance of the multiple
slave models. The master model then collects the elite samples provided by the
slave models and selects the best sample estimated by a neural contextual
UCB-based network to make a decision with a trade-off between exploration and
exploitation. Thanks to the elaborate design of slave models, the co-training
mechanism among slave models, and the novel interactions between the master and
slave models, our approach significantly surpasses existing state-of-the-art
algorithms in both synthetic and real datasets for recommendation tasks. The
code is available at:
\url{https://github.com/huanghanchi/Master-slave-Algorithm-for-Top-K-Bandits}.
less
By: Alex Gomez-Villa, Bartlomiej Twardowski, Kai Wang, Joost van de Weijer
Continuous unsupervised representation learning (CURL) research has greatly
benefited from improvements in self-supervised learning (SSL) techniques. As a
result, existing CURL methods using SSL can learn high-quality representations
without any labels, but with a notable performance drop when learning on a
many-tasks data stream. We hypothesize that this is caused by the
regularization losses that are imposed to prevent forgetting, leading... more
Continuous unsupervised representation learning (CURL) research has greatly
benefited from improvements in self-supervised learning (SSL) techniques. As a
result, existing CURL methods using SSL can learn high-quality representations
without any labels, but with a notable performance drop when learning on a
many-tasks data stream. We hypothesize that this is caused by the
regularization losses that are imposed to prevent forgetting, leading to a
suboptimal plasticity-stability trade-off: they either do not adapt fully to
the incoming data (low plasticity), or incur significant forgetting when
allowed to fully adapt to a new SSL pretext-task (low stability). In this work,
we propose to train an expert network that is relieved of the duty of keeping
the previous knowledge and can focus on performing optimally on the new tasks
(optimizing plasticity). In the second phase, we combine this new knowledge
with the previous network in an adaptation-retrospection phase to avoid
forgetting and initialize a new expert with the knowledge of the old network.
We perform several experiments showing that our proposed approach outperforms
other CURL exemplar-free methods in few- and many-task split settings.
Furthermore, we show how to adapt our approach to semi-supervised continual
learning (Semi-SCL) and show that we surpass the accuracy of other
exemplar-free Semi-SCL methods and reach the results of some others that use
exemplars.
less
By: Guanghui Zhu, Mengyu Chen, Chunfeng Yuan, Yihua Huang
As the study of graph neural networks becomes more intensive and
comprehensive, their robustness and security have received great research
interest. The existing global attack methods treat all nodes in the graph as
their attack targets. Although existing methods have achieved excellent
results, there is still considerable space for improvement. The key problem is
that the current approaches rigidly follow the definition of global attacks.
... more
As the study of graph neural networks becomes more intensive and
comprehensive, their robustness and security have received great research
interest. The existing global attack methods treat all nodes in the graph as
their attack targets. Although existing methods have achieved excellent
results, there is still considerable space for improvement. The key problem is
that the current approaches rigidly follow the definition of global attacks.
They ignore an important issue, i.e., different nodes have different robustness
and are not equally resilient to attacks. From a global attacker's view, we
should arrange the attack budget wisely, rather than wasting them on highly
robust nodes. To this end, we propose a totally new method named partial graph
attack (PGA), which selects the vulnerable nodes as attack targets. First, to
select the vulnerable items, we propose a hierarchical target selection policy,
which allows attackers to only focus on easy-to-attack nodes. Then, we propose
a cost-effective anchor-picking policy to pick the most promising anchors for
adding or removing edges, and a more aggressive iterative greedy-based attack
method to perform more efficient attacks. Extensive experimental results
demonstrate that PGA can achieve significant improvements in both attack effect
and attack efficiency compared to other existing graph global attack methods.
less
By: Dirk van der Hoeven, Nikita Zhivotovskiy, Nicolò Cesa-Bianchi
Online learning methods yield sequential regret bounds under minimal
assumptions and provide in-expectation risk bounds for statistical learning.
However, despite the apparent advantage of online guarantees over their
statistical counterparts, recent findings indicate that in many important
cases, regret bounds may not guarantee tight high-probability risk bounds in
the statistical setting. In this work we show that online to batch conversi... more
Online learning methods yield sequential regret bounds under minimal
assumptions and provide in-expectation risk bounds for statistical learning.
However, despite the apparent advantage of online guarantees over their
statistical counterparts, recent findings indicate that in many important
cases, regret bounds may not guarantee tight high-probability risk bounds in
the statistical setting. In this work we show that online to batch conversions
applied to general online learning algorithms can bypass this limitation. Via a
general second-order correction to the loss function defining the regret, we
obtain nearly optimal high-probability risk bounds for several classical
statistical estimation problems, such as discrete distribution estimation,
linear regression, logistic regression, and conditional density estimation. Our
analysis relies on the fact that many online learning algorithms are improper,
as they are not restricted to use predictors from a given reference class. The
improper nature of our estimators enables significant improvements in the
dependencies on various problem parameters. Finally, we discuss some
computational advantages of our sequential algorithms over their existing batch
counterparts.
less
By: Luca Herranz-Celotti, Jean Rouat
Numerous theories of learning suggest to prevent the gradient variance from
exponential growth with depth or time, to stabilize and improve training.
Typically, these analyses are conducted on feed-forward fully-connected neural
networks or single-layer recurrent neural networks, given their mathematical
tractability. In contrast, this study demonstrates that pre-training the
network to local stability can be effective whenever the architec... more
Numerous theories of learning suggest to prevent the gradient variance from
exponential growth with depth or time, to stabilize and improve training.
Typically, these analyses are conducted on feed-forward fully-connected neural
networks or single-layer recurrent neural networks, given their mathematical
tractability. In contrast, this study demonstrates that pre-training the
network to local stability can be effective whenever the architectures are too
complex for an analytical initialization. Furthermore, we extend known
stability theories to encompass a broader family of deep recurrent networks,
requiring minimal assumptions on data and parameter distribution, a theory that
we refer to as the Local Stability Condition (LSC). Our investigation reveals
that the classical Glorot, He, and Orthogonal initialization schemes satisfy
the LSC when applied to feed-forward fully-connected neural networks. However,
analysing deep recurrent networks, we identify a new additive source of
exponential explosion that emerges from counting gradient paths in a
rectangular grid in depth and time. We propose a new approach to mitigate this
issue, that consists on giving a weight of a half to the time and depth
contributions to the gradient, instead of the classical weight of one. Our
empirical results confirm that pre-training both feed-forward and recurrent
networks to fulfill the LSC often results in improved final performance across
models. This study contributes to the field by providing a means to stabilize
networks of any complexity. Our approach can be implemented as an additional
step before pre-training on large augmented datasets, and as an alternative to
finding stable initializations analytically.
less
By: Jaya Chaturvedi, Diana Shamsutdinova, Felix Zimmer, Sumithra Velupillai, Daniel Stahl, Robert Stewart, Angus Roberts
Sample size calculation is an essential step in most data-based disciplines.
Large enough samples ensure representativeness of the population and determine
the precision of estimates. This is true for most quantitative studies,
including those that employ machine learning methods, such as natural language
processing, where free-text is used to generate predictions and classify
instances of text. Within the healthcare domain, the lack of suf... more
Sample size calculation is an essential step in most data-based disciplines.
Large enough samples ensure representativeness of the population and determine
the precision of estimates. This is true for most quantitative studies,
including those that employ machine learning methods, such as natural language
processing, where free-text is used to generate predictions and classify
instances of text. Within the healthcare domain, the lack of sufficient corpora
of previously collected data can be a limiting factor when determining sample
sizes for new studies. This paper tries to address the issue by making
recommendations on sample sizes for text classification tasks in the healthcare
domain.
Models trained on the MIMIC-III database of critical care records from Beth
Israel Deaconess Medical Center were used to classify documents as having or
not having Unspecified Essential Hypertension, the most common diagnosis code
in the database. Simulations were performed using various classifiers on
different sample sizes and class proportions. This was repeated for a
comparatively less common diagnosis code within the database of diabetes
mellitus without mention of complication.
Smaller sample sizes resulted in better results when using a K-nearest
neighbours classifier, whereas larger sample sizes provided better results with
support vector machines and BERT models. Overall, a sample size larger than
1000 was sufficient to provide decent performance metrics.
The simulations conducted within this study provide guidelines that can be
used as recommendations for selecting appropriate sample sizes and class
proportions, and for predicting expected performance, when building classifiers
for textual healthcare data. The methodology used here can be modified for
sample size estimates calculations with other datasets.
less
By: Feng Zhou, Quyu Kong, Zhijie Deng, Fengxiang He, Peng Cui, Jun Zhu
This paper presents a novel extension of multi-task Gaussian Cox processes
for modeling multiple heterogeneous correlated tasks jointly, e.g.,
classification and regression, via multi-output Gaussian processes (MOGP). A
MOGP prior over the parameters of the dedicated likelihoods for classification,
regression and point process tasks can facilitate sharing of information
between heterogeneous tasks, while allowing for nonparametric parameter... more
This paper presents a novel extension of multi-task Gaussian Cox processes
for modeling multiple heterogeneous correlated tasks jointly, e.g.,
classification and regression, via multi-output Gaussian processes (MOGP). A
MOGP prior over the parameters of the dedicated likelihoods for classification,
regression and point process tasks can facilitate sharing of information
between heterogeneous tasks, while allowing for nonparametric parameter
estimation. To circumvent the non-conjugate Bayesian inference in the MOGP
modulated heterogeneous multi-task framework, we employ the data augmentation
technique and derive a mean-field approximation to realize closed-form
iterative updates for estimating model parameters. We demonstrate the
performance and inference on both 1D synthetic data as well as 2D urban data of
Vancouver.
less
By: Sajal Khandelwal, Pawan Kumar, Syed Azeemuddin
Radio frequency (RF) biosensors, in particular those based on inter-digitated
capacitors (IDCs), are pivotal in areas like biomedical diagnosis, remote
sensing, and wireless communication. Despite their advantages of low cost and
easy fabrication, their sensitivity can be hindered by design imperfections,
environmental factors, and circuit noise. This paper investigates enhancing the
sensitivity of IDC-based RF sensors using novel reinforce... more
Radio frequency (RF) biosensors, in particular those based on inter-digitated
capacitors (IDCs), are pivotal in areas like biomedical diagnosis, remote
sensing, and wireless communication. Despite their advantages of low cost and
easy fabrication, their sensitivity can be hindered by design imperfections,
environmental factors, and circuit noise. This paper investigates enhancing the
sensitivity of IDC-based RF sensors using novel reinforcement learning based
Binary Particle Swarm Optimization (RLBPSO), and it is compared to Ant Colony
Optimization (ACO), and other state-of-the-art methods. By focusing on
optimizing design parameters like electrode design and finger width, the
proposed study found notable improvements in sensor sensitivity. The proposed
RLBPSO method shows best optimized design for various frequency ranges when
compared to current state-of-the-art methods.
less
By: Nat Wannawas, A. Aldo Faisal
Reinforcement learning of real-world tasks is very data inefficient, and
extensive simulation-based modelling has become the dominant approach for
training systems. However, in human-robot interaction and many other real-world
settings, there is no appropriate one-model-for-all due to differences in
individual instances of the system (e.g. different people) or necessary
oversimplifications in the simulation models. This requires two approac... more
Reinforcement learning of real-world tasks is very data inefficient, and
extensive simulation-based modelling has become the dominant approach for
training systems. However, in human-robot interaction and many other real-world
settings, there is no appropriate one-model-for-all due to differences in
individual instances of the system (e.g. different people) or necessary
oversimplifications in the simulation models. This requires two approaches: 1.
either learning the individual system's dynamics approximately from data which
requires data-intensive training or 2. using a complete digital twin of the
instances, which may not be realisable in many cases. We introduce two
approaches: co-kriging adjustments (CKA) and ridge regression adjustment (RRA)
as novel ways to combine the advantages of both approaches. Our adjustment
methods are based on an auto-regressive AR1 co-kriging model that we integrate
with GP priors. This yield a data- and simulation-efficient way of using
simplistic simulation models (e.g., simple two-link model) and rapidly adapting
them to individual instances (e.g., biomechanics of individual people). Using
CKA and RRA, we obtain more accurate uncertainty quantification of the entire
system's dynamics than pure GP-based and AR1 methods. We demonstrate the
efficiency of co-kriging adjustment with an interpretable reinforcement
learning control example, learning to control a biomechanical human arm using
only a two-link arm simulation model (offline part) and CKA derived from a
small amount of interaction data (on-the-fly online). Our method unlocks an
efficient and uncertainty-aware way to implement reinforcement learning methods
in real world complex systems for which only imperfect simulation models exist.
less
By: Mahsa Mesgaran, A. Ben Hamza
A key component of many graph neural networks (GNNs) is the pooling
operation, which seeks to reduce the size of a graph while preserving important
structural information. However, most existing graph pooling strategies rely on
an assignment matrix obtained by employing a GNN layer, which is characterized
by trainable parameters, often leading to significant computational complexity
and a lack of interpretability in the pooling process. In ... more
A key component of many graph neural networks (GNNs) is the pooling
operation, which seeks to reduce the size of a graph while preserving important
structural information. However, most existing graph pooling strategies rely on
an assignment matrix obtained by employing a GNN layer, which is characterized
by trainable parameters, often leading to significant computational complexity
and a lack of interpretability in the pooling process. In this paper, we
propose an unsupervised graph encoder-decoder model to detect abnormal nodes
from graphs by learning an anomaly scoring function to rank nodes based on
their degree of abnormality. In the encoding stage, we design a novel pooling
mechanism, named LCPool, which leverages locality-constrained linear coding for
feature encoding to find a cluster assignment matrix by solving a least-squares
optimization problem with a locality regularization term. By enforcing locality
constraints during the coding process, LCPool is designed to be free from
learnable parameters, capable of efficiently handling large graphs, and can
effectively generate a coarser graph representation while retaining the most
significant structural characteristics of the graph. In the decoding stage, we
propose an unpooling operation, called LCUnpool, to reconstruct both the
structure and nodal features of the original graph. We conduct empirical
evaluations of our method on six benchmark datasets using several evaluation
metrics, and the results demonstrate its superiority over state-of-the-art
anomaly detection approaches.
less
By: Jingcan Duan, Pei Zhang, Siwei Wang, Jingtao Hu, Hu Jin, Jiaxin Zhang, Haifang Zhou, Haifang Zhou
Graph anomaly detection (GAD) has attracted increasing attention in machine
learning and data mining. Recent works have mainly focused on how to capture
richer information to improve the quality of node embeddings for GAD. Despite
their significant advances in detection performance, there is still a relative
dearth of research on the properties of the task. GAD aims to discern the
anomalies that deviate from most nodes. However, the model i... more
Graph anomaly detection (GAD) has attracted increasing attention in machine
learning and data mining. Recent works have mainly focused on how to capture
richer information to improve the quality of node embeddings for GAD. Despite
their significant advances in detection performance, there is still a relative
dearth of research on the properties of the task. GAD aims to discern the
anomalies that deviate from most nodes. However, the model is prone to learn
the pattern of normal samples which make up the majority of samples. Meanwhile,
anomalies can be easily detected when their behaviors differ from normality.
Therefore, the performance can be further improved by enhancing the ability to
learn the normal pattern. To this end, we propose a normality learning-based
GAD framework via multi-scale contrastive learning networks (NLGAD for
abbreviation). Specifically, we first initialize the model with the contrastive
networks on different scales. To provide sufficient and reliable normal nodes
for normality learning, we design an effective hybrid strategy for normality
selection. Finally, the model is refined with the only input of reliable normal
nodes and learns a more accurate estimate of normality so that anomalous nodes
can be more easily distinguished. Eventually, extensive experiments on six
benchmark graph datasets demonstrate the effectiveness of our normality
learning-based scheme on GAD. Notably, the proposed algorithm improves the
detection performance (up to 5.89% AUC gain) compared with the state-of-the-art
methods. The source code is released at https://github.com/FelixDJC/NLGAD.
less
By: Jiani Liu, Qinghua Tao, Ce Zhu, Yipeng Liu, Xiaolin Huang, Johan A. K. Suykens
Multitask learning (MTL) leverages task-relatedness to enhance performance.
With the emergence of multimodal data, tasks can now be referenced by multiple
indices. In this paper, we employ high-order tensors, with each mode
corresponding to a task index, to naturally represent tasks referenced by
multiple indices and preserve their structural relations. Based on this
representation, we propose a general framework of low-rank MTL methods wit... more
Multitask learning (MTL) leverages task-relatedness to enhance performance.
With the emergence of multimodal data, tasks can now be referenced by multiple
indices. In this paper, we employ high-order tensors, with each mode
corresponding to a task index, to naturally represent tasks referenced by
multiple indices and preserve their structural relations. Based on this
representation, we propose a general framework of low-rank MTL methods with
tensorized support vector machines (SVMs) and least square support vector
machines (LSSVMs), where the CP factorization is deployed over the coefficient
tensor. Our approach allows to model the task relation through a linear
combination of shared factors weighted by task-specific factors and is
generalized to both classification and regression problems. Through the
alternating optimization scheme and the Lagrangian function, each subproblem is
transformed into a convex problem, formulated as a quadratic programming or
linear system in the dual form. In contrast to previous MTL frameworks, our
decision function in the dual induces a weighted kernel function with a
task-coupling term characterized by the similarities of the task-specific
factors, better revealing the explicit relations across tasks in MTL.
Experimental results validate the effectiveness and superiority of our proposed
methods compared to existing state-of-the-art approaches in MTL. The code of
implementation will be available at https://github.com/liujiani0216/TSVM-MTL.
less
By: Lei Guan
This paper proposes an efficient optimizer called AdaPlus which integrates
Nesterov momentum and precise stepsize adjustment on AdamW basis. AdaPlus
combines the advantages of AdamW, Nadam, and AdaBelief and, in particular, does
not introduce any extra hyper-parameters. We perform extensive experimental
evaluations on three machine learning tasks to validate the effectiveness of
AdaPlus. The experiment results validate that AdaPlus (i) is t... more
This paper proposes an efficient optimizer called AdaPlus which integrates
Nesterov momentum and precise stepsize adjustment on AdamW basis. AdaPlus
combines the advantages of AdamW, Nadam, and AdaBelief and, in particular, does
not introduce any extra hyper-parameters. We perform extensive experimental
evaluations on three machine learning tasks to validate the effectiveness of
AdaPlus. The experiment results validate that AdaPlus (i) is the best adaptive
method which performs most comparable with (even slightly better than) SGD with
momentum on image classification tasks and (ii) outperforms other
state-of-the-art optimizers on language modeling tasks and illustrates the
highest stability when training GANs. The experiment code of AdaPlus is
available at: https://github.com/guanleics/AdaPlus.
less
DIVA: A Dirichlet Process Based Incremental Deep Clustering Algorithm via Variational Auto-Encoder
0upvotes
By: Zhenshan Bing, Yuan Meng, Yuqi Yun, Hang Su, Xiaojie Su, Kai Huang, Alois Knoll
Generative model-based deep clustering frameworks excel in classifying
complex data, but are limited in handling dynamic and complex features because
they require prior knowledge of the number of clusters. In this paper, we
propose a nonparametric deep clustering framework that employs an infinite
mixture of Gaussians as a prior. Our framework utilizes a memoized online
variational inference method that enables the "birth" and "merge" moves... more
Generative model-based deep clustering frameworks excel in classifying
complex data, but are limited in handling dynamic and complex features because
they require prior knowledge of the number of clusters. In this paper, we
propose a nonparametric deep clustering framework that employs an infinite
mixture of Gaussians as a prior. Our framework utilizes a memoized online
variational inference method that enables the "birth" and "merge" moves of
clusters, allowing our framework to cluster data in a "dynamic-adaptive"
manner, without requiring prior knowledge of the number of features. We name
the framework as DIVA, a Dirichlet Process-based Incremental deep clustering
framework via Variational Auto-Encoder. Our framework, which outperforms
state-of-the-art baselines, exhibits superior performance in classifying
complex data with dynamically changing features, particularly in the case of
incremental features. The code can be found at: https://github.com/Ghiara/diva
less
By: Filippo Galli, Kangsoo Jung, Sayan Biswas, Catuscia Palamidessi, Tommaso Cucinotta
Federated learning (FL) is a framework for training machine learning models
in a distributed and collaborative manner. During training, a set of
participating clients process their data stored locally, sharing only the model
updates obtained by minimizing a cost function over their local inputs. FL was
proposed as a stepping-stone towards privacy-preserving machine learning, but
it has been shown vulnerable to issues such as leakage of priv... more
Federated learning (FL) is a framework for training machine learning models
in a distributed and collaborative manner. During training, a set of
participating clients process their data stored locally, sharing only the model
updates obtained by minimizing a cost function over their local inputs. FL was
proposed as a stepping-stone towards privacy-preserving machine learning, but
it has been shown vulnerable to issues such as leakage of private information,
lack of personalization of the model, and the possibility of having a trained
model that is fairer to some groups than to others. In this paper, we address
the triadic interaction among personalization, privacy guarantees, and fairness
attained by models trained within the FL framework. Differential privacy and
its variants have been studied and applied as cutting-edge standards for
providing formal privacy guarantees. However, clients in FL often hold very
diverse datasets representing heterogeneous communities, making it important to
protect their sensitive information while still ensuring that the trained model
upholds the aspect of fairness for the users. To attain this objective, a
method is put forth that introduces group privacy assurances through the
utilization of $d$-privacy (aka metric privacy). $d$-privacy represents a
localized form of differential privacy that relies on a metric-oriented
obfuscation approach to maintain the original data's topological distribution.
This method, besides enabling personalized model training in a federated
approach and providing formal privacy guarantees, possesses significantly
better group fairness measured under a variety of standard metrics than a
global model trained within a classical FL template. Theoretical justifications
for the applicability are provided, as well as experimental validation on
real-world datasets to illustrate the working of the proposed method.
less
By: Nicolas Michel, Giovanni Chierchia, Romain Negrel, Jean-François Bercher, Toshihiko Yamasaki
Deep neural networks have shown remarkable performance when trained on
independent and identically distributed data from a fixed set of classes.
However, in real-world scenarios, it can be desirable to train models on a
continuous stream of data where multiple classification tasks are presented
sequentially. This scenario, known as Continual Learning (CL) poses challenges
to standard learning algorithms which struggle to maintain knowledge ... more
Deep neural networks have shown remarkable performance when trained on
independent and identically distributed data from a fixed set of classes.
However, in real-world scenarios, it can be desirable to train models on a
continuous stream of data where multiple classification tasks are presented
sequentially. This scenario, known as Continual Learning (CL) poses challenges
to standard learning algorithms which struggle to maintain knowledge of old
tasks while learning new ones. This stability-plasticity dilemma remains
central to CL and multiple metrics have been proposed to adequately measure
stability and plasticity separately. However, none considers the increasing
difficulty of the classification task, which inherently results in performance
loss for any model. In that sense, we analyze some limitations of current
metrics and identify the presence of setup-induced forgetting. Therefore, we
propose new metrics that account for the task's increasing difficulty. Through
experiments on benchmark datasets, we demonstrate that our proposed metrics can
provide new insights into the stability-plasticity trade-off achieved by models
in the continual learning environment.
less
By: Mingjie Qiu, Zhiyi Tan, Bing-kun Bao
Infectious disease forecasting has been a key focus and proved to be crucial
in controlling epidemic. A recent trend is to develop forecast-ing models based
on graph neural networks (GNNs). However, existing GNN-based methods suffer
from two key limitations: (1) Current models broaden receptive fields by
scaling the depth of GNNs, which is insuffi-cient to preserve the semantics of
long-range connectivity between distant but epidemic relate... more
Infectious disease forecasting has been a key focus and proved to be crucial
in controlling epidemic. A recent trend is to develop forecast-ing models based
on graph neural networks (GNNs). However, existing GNN-based methods suffer
from two key limitations: (1) Current models broaden receptive fields by
scaling the depth of GNNs, which is insuffi-cient to preserve the semantics of
long-range connectivity between distant but epidemic related areas. (2)
Previous approaches model epidemics within single spatial scale, while ignoring
the multi-scale epidemic pat-terns derived from different scales. To address
these deficiencies, we devise the Multi-scale Spatio-temporal Graph Neural
Network (MSGNN) based on an innovative multi-scale view. To be specific, in the
proposed MSGNN model, we first devise a novel graph learning module, which
directly captures long-range connectivity from trans-regional epidemic signals
and integrates them into a multi-scale graph. Based on the learned multi-scale
graph, we utilize a newly designed graph convolution module to exploit
multi-scale epidemic patterns. This module allows us to facilitate multi-scale
epidemic modeling by mining both scale-shared and scale-specific pat-terns.
Experimental results on forecasting new cases of COVID-19 in United State
demonstrate the superiority of our method over state-of-arts. Further analyses
and visualization also show that MSGNN offers not only accurate, but also
robust and interpretable forecasting result.
less
By: Hongyan Chang, Reza Shokri
We show that participating in federated learning can be detrimental to group
fairness. In fact, the bias of a few parties against under-represented groups
(identified by sensitive attributes such as gender or race) can propagate
through the network to all the parties in the network. We analyze and explain
bias propagation in federated learning on naturally partitioned real-world
datasets. Our analysis reveals that biased parties unintention... more
We show that participating in federated learning can be detrimental to group
fairness. In fact, the bias of a few parties against under-represented groups
(identified by sensitive attributes such as gender or race) can propagate
through the network to all the parties in the network. We analyze and explain
bias propagation in federated learning on naturally partitioned real-world
datasets. Our analysis reveals that biased parties unintentionally yet
stealthily encode their bias in a small number of model parameters, and
throughout the training, they steadily increase the dependence of the global
model on sensitive attributes. What is important to highlight is that the
experienced bias in federated learning is higher than what parties would
otherwise encounter in centralized training with a model trained on the union
of all their data. This indicates that the bias is due to the algorithm. Our
work calls for auditing group fairness in federated learning and designing
learning algorithms that are robust to bias propagation.
less
By: Hojoon Lee, Hawon Jeong, Byungkun Lee, Kyungyup Lee, Jaegul Choo
In this paper, we introduce ST-RAP, a novel Spatio-Temporal framework for
Real estate APpraisal. ST-RAP employs a hierarchical architecture with a
heterogeneous graph neural network to encapsulate temporal dynamics and spatial
relationships simultaneously. Through comprehensive experiments on a
large-scale real estate dataset, ST-RAP outperforms previous methods,
demonstrating the significant benefits of integrating spatial and temporal
asp... more
In this paper, we introduce ST-RAP, a novel Spatio-Temporal framework for
Real estate APpraisal. ST-RAP employs a hierarchical architecture with a
heterogeneous graph neural network to encapsulate temporal dynamics and spatial
relationships simultaneously. Through comprehensive experiments on a
large-scale real estate dataset, ST-RAP outperforms previous methods,
demonstrating the significant benefits of integrating spatial and temporal
aspects in real estate appraisal. Our code and dataset are available at
https://github.com/dojeon-ai/STRAP.
less
By: Davide Italo Serramazza, Thu Trang Nguyen, Thach Le Nguyen, Georgiana Ifrim
Multivariate time series classification is an important computational task
arising in applications where data is recorded over time and over multiple
channels. For example, a smartwatch can record the acceleration and orientation
of a person's motion, and these signals are recorded as multivariate time
series. We can classify this data to understand and predict human movement and
various properties such as fitness levels. In many applicatio... more
Multivariate time series classification is an important computational task
arising in applications where data is recorded over time and over multiple
channels. For example, a smartwatch can record the acceleration and orientation
of a person's motion, and these signals are recorded as multivariate time
series. We can classify this data to understand and predict human movement and
various properties such as fitness levels. In many applications classification
alone is not enough, we often need to classify but also understand what the
model learns (e.g., why was a prediction given, based on what information in
the data). The main focus of this paper is on analysing and evaluating
explanation methods tailored to Multivariate Time Series Classification (MTSC).
We focus on saliency-based explanation methods that can point out the most
relevant channels and time series points for the classification decision. We
analyse two popular and accurate multivariate time series classifiers, ROCKET
and dResNet, as well as two popular explanation methods, SHAP and dCAM. We
study these methods on 3 synthetic datasets and 2 real-world datasets and
provide a quantitative and qualitative analysis of the explanations provided.
We find that flattening the multivariate datasets by concatenating the channels
works as well as using multivariate classifiers directly and adaptations of
SHAP for MTSC work quite well. Additionally, we also find that the popular
synthetic datasets we used are not suitable for time series analysis.
less
By: Yujin Hwang, Won Jo, Juyoung Hong, Yukyung Choi
It is not an exaggeration to say that the recent progress in artificial
intelligence technology depends on large-scale and high-quality data.
Simultaneously, a prevalent issue exists everywhere: the budget for data
labeling is constrained. Active learning is a prominent approach for addressing
this issue, where valuable data for labeling is selected through a model and
utilized to iteratively adjust the model. However, due to the limited am... more
It is not an exaggeration to say that the recent progress in artificial
intelligence technology depends on large-scale and high-quality data.
Simultaneously, a prevalent issue exists everywhere: the budget for data
labeling is constrained. Active learning is a prominent approach for addressing
this issue, where valuable data for labeling is selected through a model and
utilized to iteratively adjust the model. However, due to the limited amount of
data in each iteration, the model is vulnerable to bias; thus, it is more
likely to yield overconfident predictions. In this paper, we present two novel
methods to address the problem of overconfidence that arises in the active
learning scenario. The first is an augmentation strategy named
Cross-Mix-and-Mix (CMaM), which aims to calibrate the model by expanding the
limited training distribution. The second is a selection strategy named Ranked
Margin Sampling (RankedMS), which prevents choosing data that leads to overly
confident predictions. Through various experiments and analyses, we are able to
demonstrate that our proposals facilitate efficient data selection by
alleviating overconfidence, even though they are readily applicable.
less
Why Data Science Projects Fail
0upvotes
By: Balaram Panda The University of Auckland
Data Science is a modern Data Intelligence practice, which is the core of
many businesses and helps businesses build smart strategies around to deal with
businesses challenges more efficiently. Data Science practice also helps in
automating business processes using the algorithm, and it has several other
benefits, which also deliver in a non-profitable framework. In regards to data
science, three key components primarily influence the effec... more
Data Science is a modern Data Intelligence practice, which is the core of
many businesses and helps businesses build smart strategies around to deal with
businesses challenges more efficiently. Data Science practice also helps in
automating business processes using the algorithm, and it has several other
benefits, which also deliver in a non-profitable framework. In regards to data
science, three key components primarily influence the effective outcome of a
data science project. Those are 1.Availability of Data 2.Algorithm 3.Processing
power or infrastructure
less
By: Mengnan Zhao, Lihe Zhang, Yuqiu Kong, Baocai Yin
Fast adversarial training (FAT) is beneficial for improving the adversarial
robustness of neural networks. However, previous FAT work has encountered a
significant issue known as catastrophic overfitting when dealing with large
perturbation budgets, \ie the adversarial robustness of models declines to near
zero during training.
To address this, we analyze the training process of prior FAT work and
observe that catastrophic overfitting is ... more
Fast adversarial training (FAT) is beneficial for improving the adversarial
robustness of neural networks. However, previous FAT work has encountered a
significant issue known as catastrophic overfitting when dealing with large
perturbation budgets, \ie the adversarial robustness of models declines to near
zero during training.
To address this, we analyze the training process of prior FAT work and
observe that catastrophic overfitting is accompanied by the appearance of loss
convergence outliers.
Therefore, we argue a moderately smooth loss convergence process will be a
stable FAT process that solves catastrophic overfitting.
To obtain a smooth loss convergence process, we propose a novel oscillatory
constraint (dubbed ConvergeSmooth) to limit the loss difference between
adjacent epochs. The convergence stride of ConvergeSmooth is introduced to
balance convergence and smoothing. Likewise, we design weight centralization
without introducing additional hyperparameters other than the loss balance
coefficient.
Our proposed methods are attack-agnostic and thus can improve the training
stability of various FAT techniques.
Extensive experiments on popular datasets show that the proposed methods
efficiently avoid catastrophic overfitting and outperform all previous FAT
methods. Code is available at \url{https://github.com/FAT-CS/ConvergeSmooth}.
less
By: Bhaskar Dhariyal, Thach Le Nguyen, Georgiana Ifrim
Accuracy is a key focus of current work in time series classification.
However, speed and data reduction in many applications is equally important,
especially when the data scale and storage requirements increase rapidly.
Current MTSC algorithms need hundreds of compute hours to complete training and
prediction. This is due to the nature of multivariate time series data, which
grows with the number of time series, their length and the numbe... more
Accuracy is a key focus of current work in time series classification.
However, speed and data reduction in many applications is equally important,
especially when the data scale and storage requirements increase rapidly.
Current MTSC algorithms need hundreds of compute hours to complete training and
prediction. This is due to the nature of multivariate time series data, which
grows with the number of time series, their length and the number of channels.
In many applications, not all the channels are useful for the classification
task; hence we require methods that can efficiently select useful channels and
thus save computational resources. We propose and evaluate two methods for
channel selection. Our techniques work by representing each class by a
prototype time series and performing channel selection based on the prototype
distance between classes. The main hypothesis is that useful channels enable
better separation between classes; hence, channels with the higher distance
between class prototypes are more useful. On the UEA Multivariate Time Series
Classification (MTSC) benchmark, we show that these techniques achieve
significant data reduction and classifier speedup for similar levels of
classification accuracy. Channel selection is applied as a pre-processing step
before training state-of-the-art MTSC algorithms and saves about 70\% of
computation time and data storage, with preserved accuracy. Furthermore, our
methods enable even efficient classifiers, such as ROCKET, to achieve better
accuracy than using no channel selection or forward channel selection. To
further study the impact of our techniques, we present experiments on
classifying synthetic multivariate time series datasets with more than 100
channels, as well as a real-world case study on a dataset with 50 channels. Our
channel selection methods lead to significant data reduction with preserved or
improved accuracy.
less
By: Sehwan Moon, Hyunju Lee
Multimodal learning often outperforms its unimodal counterparts by exploiting
unimodal contributions and cross-modal interactions. However, focusing only on
integrating multimodal features into a unified comprehensive representation
overlooks the unimodal characteristics. In real data, the contributions of
modalities can vary from instance to instance, and they often reinforce or
conflict with each other. In this study, we introduce a novel... more
Multimodal learning often outperforms its unimodal counterparts by exploiting
unimodal contributions and cross-modal interactions. However, focusing only on
integrating multimodal features into a unified comprehensive representation
overlooks the unimodal characteristics. In real data, the contributions of
modalities can vary from instance to instance, and they often reinforce or
conflict with each other. In this study, we introduce a novel \text{MultiModal}
loss paradigm for multimodal learning, which subgroups instances according to
their unimodal contributions. \text{MultiModal} loss can prevent inefficient
learning caused by overfitting and efficiently optimize multimodal models. On
synthetic data, \text{MultiModal} loss demonstrates improved classification
performance by subgrouping difficult instances within certain modalities. On
four real multimodal datasets, our loss is empirically shown to improve the
performance of recent models. Ablation studies verify the effectiveness of our
loss. Additionally, we show that our loss generates a reliable prediction score
for each modality, which is essential for subgrouping. Our \text{MultiModal}
loss is a novel loss function to subgroup instances according to the
contribution of modalities in multimodal learning and is applicable to a
variety of multimodal models with unimodal decisions. Our code is available at
https://github.com/SehwanMoon/MultiModalLoss.
less
By: Ali Baheri
In the dynamic and uncertain environments where reinforcement learning (RL)
operates, risk management becomes a crucial factor in ensuring reliable
decision-making. Traditional RL approaches, while effective in reward
optimization, often overlook the landscape of potential risks. In response,
this paper pioneers the integration of Optimal Transport (OT) theory with RL to
create a risk-aware framework. Our approach modifies the objective fun... more
In the dynamic and uncertain environments where reinforcement learning (RL)
operates, risk management becomes a crucial factor in ensuring reliable
decision-making. Traditional RL approaches, while effective in reward
optimization, often overlook the landscape of potential risks. In response,
this paper pioneers the integration of Optimal Transport (OT) theory with RL to
create a risk-aware framework. Our approach modifies the objective function,
ensuring that the resulting policy not only maximizes expected rewards but also
respects risk constraints dictated by OT distances between state visitation
distributions and the desired risk profiles. By leveraging the mathematical
precision of OT, we offer a formulation that elevates risk considerations
alongside conventional RL objectives. Our contributions are substantiated with
a series of theorems, mapping the relationships between risk distributions,
optimal value functions, and policy behaviors. Through the lens of OT, this
work illuminates a promising direction for RL, ensuring a balanced fusion of
reward pursuit and risk awareness.
less
Model-free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systems
0upvotes
By: Tianchi Cai, Shenliao Bao, Jiyan Jiang, Shiji Zhou, Wenpeng Zhang, Lihong Gu, Jinjie Gu, Guannan Zhang
Model-free RL-based recommender systems have recently received increasing
research attention due to their capability to handle partial feedback and
long-term rewards. However, most existing research has ignored a critical
feature in recommender systems: one user's feedback on the same item at
different times is random. The stochastic rewards property essentially differs
from that in classic RL scenarios with deterministic rewards, which mak... more
Model-free RL-based recommender systems have recently received increasing
research attention due to their capability to handle partial feedback and
long-term rewards. However, most existing research has ignored a critical
feature in recommender systems: one user's feedback on the same item at
different times is random. The stochastic rewards property essentially differs
from that in classic RL scenarios with deterministic rewards, which makes
RL-based recommender systems much more challenging. In this paper, we first
demonstrate in a simulator environment where using direct stochastic feedback
results in a significant drop in performance. Then to handle the stochastic
feedback more efficiently, we design two stochastic reward stabilization
frameworks that replace the direct stochastic feedback with that learned by a
supervised model. Both frameworks are model-agnostic, i.e., they can
effectively utilize various supervised models. We demonstrate the superiority
of the proposed frameworks over different RL-based recommendation baselines
with extensive experiments on a recommendation simulator as well as an
industrial-level recommender system.
less
Relational Concept Based Models
0upvotes
By: Pietro Barbiero, Francesco Giannini, Gabriele Ciravegna, Michelangelo Diligenti, Giuseppe Marra
The design of interpretable deep learning models working in relational
domains poses an open challenge: interpretable deep learning methods, such as
Concept-Based Models (CBMs), are not designed to solve relational problems,
while relational models are not as interpretable as CBMs. To address this
problem, we propose Relational Concept-Based Models, a family of relational
deep learning methods providing interpretable task predictions. Our
e... more
The design of interpretable deep learning models working in relational
domains poses an open challenge: interpretable deep learning methods, such as
Concept-Based Models (CBMs), are not designed to solve relational problems,
while relational models are not as interpretable as CBMs. To address this
problem, we propose Relational Concept-Based Models, a family of relational
deep learning methods providing interpretable task predictions. Our
experiments, ranging from image classification to link prediction in knowledge
graphs, show that relational CBMs (i) match generalization performance of
existing relational black-boxes (as opposed to non-relational CBMs), (ii)
support the generation of quantified concept-based explanations, (iii)
effectively respond to test-time interventions, and (iv) withstand demanding
settings including out-of-distribution scenarios, limited training data
regimes, and scarce concept supervisions.
less
By: Abhin Shah, Devavrat Shah, Gregory W. Wornell
We consider the classical problem of learning, with arbitrary accuracy, the
natural parameters of a $k$-parameter truncated \textit{minimal} exponential
family from i.i.d. samples in a computationally and statistically efficient
manner. We focus on the setting where the support as well as the natural
parameters are appropriately bounded. While the traditional maximum likelihood
estimator for this class of exponential family is consistent, a... more
We consider the classical problem of learning, with arbitrary accuracy, the
natural parameters of a $k$-parameter truncated \textit{minimal} exponential
family from i.i.d. samples in a computationally and statistically efficient
manner. We focus on the setting where the support as well as the natural
parameters are appropriately bounded. While the traditional maximum likelihood
estimator for this class of exponential family is consistent, asymptotically
normal, and asymptotically efficient, evaluating it is computationally hard. In
this work, we propose a novel loss function and a computationally efficient
estimator that is consistent as well as asymptotically normal under mild
conditions. We show that, at the population level, our method can be viewed as
the maximum likelihood estimation of a re-parameterized distribution belonging
to the same class of exponential family. Further, we show that our estimator
can be interpreted as a solution to minimizing a particular Bregman score as
well as an instance of minimizing the \textit{surrogate} likelihood. We also
provide finite sample guarantees to achieve an error (in $\ell_2$-norm) of
$\alpha$ in the parameter estimation with sample complexity $O({\sf
poly}(k)/\alpha^2)$. Our method achives the order-optimal sample complexity of
$O({\sf log}(k)/\alpha^2)$ when tailored for node-wise-sparse Markov random
fields. Finally, we demonstrate the performance of our estimator via numerical
experiments.
less
By: Samuel Wiggins, Yuan Meng, Rajgopal Kannan, Viktor Prasanna
Multi-Agent Reinforcement Learning (MARL) has achieved significant success in
large-scale AI systems and big-data applications such as smart grids,
surveillance, etc. Existing advancements in MARL algorithms focus on improving
the rewards obtained by introducing various mechanisms for inter-agent
cooperation. However, these optimizations are usually compute- and
memory-intensive, thus leading to suboptimal speed performance in end-to-end
tr... more
Multi-Agent Reinforcement Learning (MARL) has achieved significant success in
large-scale AI systems and big-data applications such as smart grids,
surveillance, etc. Existing advancements in MARL algorithms focus on improving
the rewards obtained by introducing various mechanisms for inter-agent
cooperation. However, these optimizations are usually compute- and
memory-intensive, thus leading to suboptimal speed performance in end-to-end
training time. In this work, we analyze the speed performance (i.e.,
latency-bounded throughput) as the key metric in MARL implementations.
Specifically, we first introduce a taxonomy of MARL algorithms from an
acceleration perspective categorized by (1) training scheme and (2)
communication method. Using our taxonomy, we identify three state-of-the-art
MARL algorithms - Multi-Agent Deep Deterministic Policy Gradient (MADDPG),
Target-oriented Multi-agent Communication and Cooperation (ToM2C), and
Networked Multi-Agent RL (NeurComm) - as target benchmark algorithms, and
provide a systematic analysis of their performance bottlenecks on a homogeneous
multi-core CPU platform. We justify the need for MARL latency-bounded
throughput to be a key performance metric in future literature while also
addressing opportunities for parallelization and acceleration.
less
By: Amr Alkhatib, Henrik Boström, Sofiane Ennadir, Ulf Johansson
Score-based explainable machine-learning techniques are often used to
understand the logic behind black-box models. However, such explanation
techniques are often computationally expensive, which limits their application
in time-critical contexts. Therefore, we propose and investigate the use of
computationally less costly regression models for approximating the output of
score-based explanation techniques, such as SHAP. Moreover, validity ... more
Score-based explainable machine-learning techniques are often used to
understand the logic behind black-box models. However, such explanation
techniques are often computationally expensive, which limits their application
in time-critical contexts. Therefore, we propose and investigate the use of
computationally less costly regression models for approximating the output of
score-based explanation techniques, such as SHAP. Moreover, validity guarantees
for the approximated values are provided by the employed inductive conformal
prediction framework. We propose several non-conformity measures designed to
take the difficulty of approximating the explanations into account while
keeping the computational cost low. We present results from a large-scale
empirical investigation, in which the approximate explanations generated by our
proposed models are evaluated with respect to efficiency (interval size). The
results indicate that the proposed method can significantly improve execution
time compared to the fast version of SHAP, TreeSHAP. The results also suggest
that the proposed method can produce tight intervals, while providing validity
guarantees. Moreover, the proposed approach allows for comparing explanations
of different approximation methods and selecting a method based on how
informative (tight) are the predicted intervals.
less
Minimum Width for Deep, Narrow MLP: A Diffeomorphism and the Whitney Embedding Theorem Approach
0upvotes
By: Geonho Hwang
Recently, there has been significant attention on determining the minimum
width for the universal approximation property of deep, narrow MLPs. Among
these challenges, approximating a continuous function under the uniform norm is
important and challenging, with the gap between its lower and upper bound being
hard to narrow. In this regard, we propose a novel upper bound for the minimum
width, given by $\operatorname{max}(2d_x+1, d_y) + \alph... more
Recently, there has been significant attention on determining the minimum
width for the universal approximation property of deep, narrow MLPs. Among
these challenges, approximating a continuous function under the uniform norm is
important and challenging, with the gap between its lower and upper bound being
hard to narrow. In this regard, we propose a novel upper bound for the minimum
width, given by $\operatorname{max}(2d_x+1, d_y) + \alpha(\sigma)$, to achieve
uniform approximation in deep narrow MLPs, where $0\leq \alpha(\sigma)\leq 2$
represents the constant depending on the activation function. We demonstrate
this bound through two key proofs. First, we establish that deep, narrow MLPs
with little additional width can approximate diffeomorphisms. Secondly, we
utilize the Whitney embedding theorem to show that any continuous function can
be approximated by embeddings, further decomposed into linear transformations
and diffeomorphisms.
less
Extended Linear Regression: A Kalman Filter Approach for Minimizing Loss via Area Under the Curve
0upvotes
By: Gokulprasath R
This research enhances linear regression models by integrating a Kalman
filter and analysing curve areas to minimize loss. The goal is to develop an
optimal linear regression equation using stochastic gradient descent (SGD) for
weight updating. Our approach involves a stepwise process, starting with
user-defined parameters. The linear regression model is trained using SGD,
tracking weights and loss separately and zipping them finally. A Kal... more
This research enhances linear regression models by integrating a Kalman
filter and analysing curve areas to minimize loss. The goal is to develop an
optimal linear regression equation using stochastic gradient descent (SGD) for
weight updating. Our approach involves a stepwise process, starting with
user-defined parameters. The linear regression model is trained using SGD,
tracking weights and loss separately and zipping them finally. A Kalman filter
is then trained based on weight and loss arrays to predict the next
consolidated weights. Predictions result from multiplying input averages with
weights, evaluated for loss to form a weight-versus-loss curve. The curve's
equation is derived using the two-point formula, and area under the curve is
calculated via integration. The linear regression equation with minimum area
becomes the optimal curve for prediction. Benefits include avoiding constant
weight updates via gradient descent and working with partial datasets, unlike
methods needing the entire set. However, computational complexity should be
considered. The Kalman filter's accuracy might diminish beyond a certain
prediction range.
less
By: Stefan Sylvius Wagner, Peter Arndt, Jan Robine, Stefan Harmeling
In environments with sparse rewards, finding a good inductive bias for
exploration is crucial to the agent's success. However, there are two competing
goals: novelty search and systematic exploration. While existing approaches
such as curiosity-driven exploration find novelty, they sometimes do not
systematically explore the whole state space, akin to depth-first-search vs
breadth-first-search. In this paper, we propose a new intrinsic rewa... more
In environments with sparse rewards, finding a good inductive bias for
exploration is crucial to the agent's success. However, there are two competing
goals: novelty search and systematic exploration. While existing approaches
such as curiosity-driven exploration find novelty, they sometimes do not
systematically explore the whole state space, akin to depth-first-search vs
breadth-first-search. In this paper, we propose a new intrinsic reward that is
cyclophobic, i.e., it does not reward novelty, but punishes redundancy by
avoiding cycles. Augmenting the cyclophobic intrinsic reward with a sequence of
hierarchical representations based on the agent's cropped observations we are
able to achieve excellent results in the MiniGrid and MiniHack environments.
Both are particularly hard, as they require complex interactions with different
objects in order to be solved. Detailed comparisons with previous approaches
and thorough ablation studies show that our newly proposed cyclophobic
reinforcement learning is more sample efficient than other state of the art
methods in a variety of tasks.
less
By: Jiawei Zhang, Zhongzhu Chen, Huan Zhang, Chaowei Xiao, Bo Li
Diffusion models have been leveraged to perform adversarial purification and
thus provide both empirical and certified robustness for a standard model. On
the other hand, different robustly trained smoothed models have been studied to
improve the certified robustness. Thus, it raises a natural question: Can
diffusion model be used to achieve improved certified robustness on those
robustly trained smoothed models? In this work, we first theo... more
Diffusion models have been leveraged to perform adversarial purification and
thus provide both empirical and certified robustness for a standard model. On
the other hand, different robustly trained smoothed models have been studied to
improve the certified robustness. Thus, it raises a natural question: Can
diffusion model be used to achieve improved certified robustness on those
robustly trained smoothed models? In this work, we first theoretically show
that recovered instances by diffusion models are in the bounded neighborhood of
the original instance with high probability; and the "one-shot" denoising
diffusion probabilistic models (DDPM) can approximate the mean of the generated
distribution of a continuous-time diffusion model, which approximates the
original instance under mild conditions. Inspired by our analysis, we propose a
certifiably robust pipeline DiffSmooth, which first performs adversarial
purification via diffusion models and then maps the purified instances to a
common region via a simple yet effective local smoothing strategy. We conduct
extensive experiments on different datasets and show that DiffSmooth achieves
SOTA-certified robustness compared with eight baselines. For instance,
DiffSmooth improves the SOTA-certified accuracy from $36.0\%$ to $53.0\%$ under
$\ell_2$ radius $1.5$ on ImageNet. The code is available at
[https://github.com/javyduck/DiffSmooth].
less
Graph Neural Bandits
0upvotes
By: Yunzhe Qi, Yikun Ban, Jingrui He
Contextual bandits algorithms aim to choose the optimal arm with the highest
reward out of a set of candidates based on the contextual information. Various
bandit algorithms have been applied to real-world applications due to their
ability of tackling the exploitation-exploration dilemma. Motivated by online
recommendation scenarios, in this paper, we propose a framework named Graph
Neural Bandits (GNB) to leverage the collaborative nature ... more
Contextual bandits algorithms aim to choose the optimal arm with the highest
reward out of a set of candidates based on the contextual information. Various
bandit algorithms have been applied to real-world applications due to their
ability of tackling the exploitation-exploration dilemma. Motivated by online
recommendation scenarios, in this paper, we propose a framework named Graph
Neural Bandits (GNB) to leverage the collaborative nature among users empowered
by graph neural networks (GNNs). Instead of estimating rigid user clusters as
in existing works, we model the "fine-grained" collaborative effects through
estimated user graphs in terms of exploitation and exploration respectively.
Then, to refine the recommendation strategy, we utilize separate GNN-based
models on estimated user graphs for exploitation and adaptive exploration.
Theoretical analysis and experimental results on multiple real data sets in
comparison with state-of-the-art baselines are provided to demonstrate the
effectiveness of our proposed framework.
less
By: Tianyu Liang, Yida Mu, Soonho Kim, Darline Larissa Kengne Kuate, Julie Lang, Rob Vos, Xingyi Song
A large number of conflict events are affecting the world all the time. In
order to analyse such conflict events effectively, this paper presents a
Classification-Aware Neural Topic Model (CANTM-IA) for Conflict Information
Classification and Topic Discovery. The model provides a reliable
interpretation of classification results and discovered topics by introducing
interpretability analysis. At the same time, interpretation is introduced in... more
A large number of conflict events are affecting the world all the time. In
order to analyse such conflict events effectively, this paper presents a
Classification-Aware Neural Topic Model (CANTM-IA) for Conflict Information
Classification and Topic Discovery. The model provides a reliable
interpretation of classification results and discovered topics by introducing
interpretability analysis. At the same time, interpretation is introduced into
the model architecture to improve the classification performance of the model
and to allow interpretation to focus further on the details of the data.
Finally, the model architecture is optimised to reduce the complexity of the
model.
less
By: Arman Rahbar, Niklas Åkerblom, Morteza Haghir Chehreghani
Online decision making plays a crucial role in numerous real-world
applications. In many scenarios, the decision is made based on performing a
sequence of tests on the incoming data points. However, performing all tests
can be expensive and is not always possible. In this paper, we provide a novel
formulation of the online decision making problem based on combinatorial
multi-armed bandits and take the cost of performing tests into account. ... more
Online decision making plays a crucial role in numerous real-world
applications. In many scenarios, the decision is made based on performing a
sequence of tests on the incoming data points. However, performing all tests
can be expensive and is not always possible. In this paper, we provide a novel
formulation of the online decision making problem based on combinatorial
multi-armed bandits and take the cost of performing tests into account. Based
on this formulation, we provide a new framework for cost-efficient online
decision making which can utilize posterior sampling or BayesUCB for
exploration. We provide a rigorous theoretical analysis for our framework and
present various experimental results that demonstrate its applicability to
real-world problems.
less
By: Kai Brach, Beate Sick, Oliver Dürr
Deep neural networks (NNs) are known for their high-prediction performances.
However, NNs are prone to yield unreliable predictions when encountering
completely new situations without indicating their uncertainty. Bayesian
variants of NNs (BNNs), such as Monte Carlo (MC) dropout BNNs, do provide
uncertainty measures and simultaneously increase the prediction performance.
The only disadvantage of BNNs is their higher computation time during ... more
Deep neural networks (NNs) are known for their high-prediction performances.
However, NNs are prone to yield unreliable predictions when encountering
completely new situations without indicating their uncertainty. Bayesian
variants of NNs (BNNs), such as Monte Carlo (MC) dropout BNNs, do provide
uncertainty measures and simultaneously increase the prediction performance.
The only disadvantage of BNNs is their higher computation time during test time
because they rely on a sampling approach. Here we present a single-shot MC
dropout approximation that preserves the advantages of BNNs while being as fast
as NNs. Our approach is based on moment propagation (MP) and allows to
analytically approximate the expected value and the variance of the MC dropout
signal for commonly used layers in NNs, i.e. convolution, max pooling, dense,
softmax, and dropout layers. The MP approach can convert an NN into a BNN
without re-training given the NN has been trained with standard dropout. We
evaluate our approach on different benchmark datasets and a simulated toy
example in a classification and regression setting. We demonstrate that our
single-shot MC dropout approximation resembles the point estimate and the
uncertainty estimate of the predictive distribution that is achieved with an MC
approach, while being fast enough for real-time deployments of BNNs. We show
that using part of the saved time to combine our MP approach with deep ensemble
techniques does further improve the uncertainty measures.
less
By: Antoine Choffrut, Rachid Guerraoui, Rafael Pinot, Renaud Sirdey, John Stephan, Martin Zuber
Due to the large-scale availability of data, machine learning (ML) algorithms
are being deployed in distributed topologies, where different nodes collaborate
to train ML models over their individual data by exchanging model-related
information (e.g., gradients) with a central server. However, distributed
learning schemes are notably vulnerable to two threats. First, Byzantine nodes
can single-handedly corrupt the learning by sending incorre... more
Due to the large-scale availability of data, machine learning (ML) algorithms
are being deployed in distributed topologies, where different nodes collaborate
to train ML models over their individual data by exchanging model-related
information (e.g., gradients) with a central server. However, distributed
learning schemes are notably vulnerable to two threats. First, Byzantine nodes
can single-handedly corrupt the learning by sending incorrect information to
the server, e.g., erroneous gradients. The standard approach to mitigate such
behavior is to use a non-linear robust aggregation method at the server.
Second, the server can violate the privacy of the nodes. Recent attacks have
shown that exchanging (unencrypted) gradients enables a curious server to
recover the totality of the nodes' data. The use of homomorphic encryption
(HE), a gold standard security primitive, has extensively been studied as a
privacy-preserving solution to distributed learning in non-Byzantine scenarios.
However, due to HE's large computational demand especially for high-dimensional
ML models, there has not yet been any attempt to design purely homomorphic
operators for non-linear robust aggregators. In this work, we present SABLE,
the first completely homomorphic and Byzantine robust distributed learning
algorithm. SABLE essentially relies on a novel plaintext encoding method that
enables us to implement the robust aggregator over batching-friendly BGV.
Moreover, this encoding scheme also accelerates state-of-the-art homomorphic
sorting with larger security margins and smaller ciphertext size. We perform
extensive experiments on image classification tasks and show that our algorithm
achieves practical execution times while matching the ML performance of its
non-private counterpart.
less
Why do universal adversarial attacks work on large language models?: Geometry might be the answer
0upvotes
By: Varshini Subhash, Anna Bialas, Weiwei Pan, Finale Doshi-Velez
Transformer based large language models with emergent capabilities are
becoming increasingly ubiquitous in society. However, the task of understanding
and interpreting their internal workings, in the context of adversarial
attacks, remains largely unsolved. Gradient-based universal adversarial attacks
have been shown to be highly effective on large language models and potentially
dangerous due to their input-agnostic nature. This work prese... more
Transformer based large language models with emergent capabilities are
becoming increasingly ubiquitous in society. However, the task of understanding
and interpreting their internal workings, in the context of adversarial
attacks, remains largely unsolved. Gradient-based universal adversarial attacks
have been shown to be highly effective on large language models and potentially
dangerous due to their input-agnostic nature. This work presents a novel
geometric perspective explaining universal adversarial attacks on large
language models. By attacking the 117M parameter GPT-2 model, we find evidence
indicating that universal adversarial triggers could be embedding vectors which
merely approximate the semantic information in their adversarial training
region. This hypothesis is supported by white-box model analysis comprising
dimensionality reduction and similarity measurement of hidden representations.
We believe this new geometric perspective on the underlying mechanism driving
universal attacks could help us gain deeper insight into the internal workings
and failure modes of LLMs, thus enabling their mitigation.
less
By: Shiro Takagi
We empirically investigate how pre-training on data of different modalities,
such as language and vision, affects fine-tuning of Transformer-based models to
Mujoco offline reinforcement learning tasks. Analysis of the internal
representation reveals that the pre-trained Transformers acquire largely
different representations before and after pre-training, but acquire less
information of data in fine-tuning than the randomly initialized one. ... more
We empirically investigate how pre-training on data of different modalities,
such as language and vision, affects fine-tuning of Transformer-based models to
Mujoco offline reinforcement learning tasks. Analysis of the internal
representation reveals that the pre-trained Transformers acquire largely
different representations before and after pre-training, but acquire less
information of data in fine-tuning than the randomly initialized one. A closer
look at the parameter changes of the pre-trained Transformers reveals that
their parameters do not change that much and that the bad performance of the
model pre-trained with image data could partially come from large gradients and
gradient clipping. To study what information the Transformer pre-trained with
language data utilizes, we fine-tune this model with no context provided,
finding that the model learns efficiently even without context information.
Subsequent follow-up analysis supports the hypothesis that pre-training with
language data is likely to make the Transformer get context-like information
and utilize it to solve the downstream task.
less
By: Bohan Lyu, Jianzhong Li
This paper introduces a new data analysis method for big data using a newly
defined regression model named multiple model linear regression(MMLR), which
separates input datasets into subsets and construct local linear regression
models of them. The proposed data analysis method is shown to be more efficient
and flexible than other regression based methods. This paper also proposes an
approximate algorithm to construct MMLR models based on
$... more
This paper introduces a new data analysis method for big data using a newly
defined regression model named multiple model linear regression(MMLR), which
separates input datasets into subsets and construct local linear regression
models of them. The proposed data analysis method is shown to be more efficient
and flexible than other regression based methods. This paper also proposes an
approximate algorithm to construct MMLR models based on
$(\epsilon,\delta)$-estimator, and gives mathematical proofs of the correctness
and efficiency of MMLR algorithm, of which the time complexity is linear with
respect to the size of input datasets. This paper also empirically implements
the method on both synthetic and real-world datasets, the algorithm shows to
have comparable performance to existing regression methods in many cases, while
it takes almost the shortest time to provide a high prediction accuracy.
less
By: Khoa Tran, Hai-Canh Vu, Lam Pham, Nassim Boudaoud
In this paper, a Robust Multi-branch Deep learning-based system for remaining
useful life (RUL) prediction and condition operations (CO) identification of
rotating machines is proposed. In particular, the proposed system comprises
main components: (1) an LSTM-Autoencoder to denoise the vibration data; (2) a
feature extraction to generate time-domain, frequency-domain, and
time-frequency based features from the denoised data; (3) a novel and... more
In this paper, a Robust Multi-branch Deep learning-based system for remaining
useful life (RUL) prediction and condition operations (CO) identification of
rotating machines is proposed. In particular, the proposed system comprises
main components: (1) an LSTM-Autoencoder to denoise the vibration data; (2) a
feature extraction to generate time-domain, frequency-domain, and
time-frequency based features from the denoised data; (3) a novel and robust
multi-branch deep learning network architecture to exploit the multiple
features. The performance of our proposed system was evaluated and compared to
the state-of-the-art systems on two benchmark datasets of XJTU-SY and
PRONOSTIA. The experimental results prove that our proposed system outperforms
the state-of-the-art systems and presents potential for real-life applications
on bearing machines.
less
By: Theo Delemazure LAMSADE, François Durand CREM, LINCS, Fabien Mathieu LINCS
Many decision problems cannot be solved exactly and use several estimation
algorithms that assign scores to the different available options. The
estimation errors can have various correlations, from low (e.g. between two
very different approaches) to high (e.g. when using a given algorithm with
different hyperparameters). Most aggregation rules would suffer from this
diversity of correlations. In this article, we propose different aggregati... more
Many decision problems cannot be solved exactly and use several estimation
algorithms that assign scores to the different available options. The
estimation errors can have various correlations, from low (e.g. between two
very different approaches) to high (e.g. when using a given algorithm with
different hyperparameters). Most aggregation rules would suffer from this
diversity of correlations. In this article, we propose different aggregation
rules that take correlations into account, and we compare them to naive rules
in various experiments based on synthetic data. Our results show that when
sufficient information is known about the correlations between errors, a
maximum likelihood aggregation should be preferred. Otherwise, typically with
limited training data, we recommend a method that we call Embedded Voting (EV).
less
By: Honggu Kang, Seohyeon Cha, Jinwoo Shin, Jongmyeong Lee, Joonhyuk Kang
Federated learning (FL) is a promising approach in distributed learning
keeping privacy. However, during the training pipeline of FL, slow or incapable
clients (i.e., stragglers) slow down the total training time and degrade
performance. System heterogeneity, including heterogeneous computing and
network bandwidth, has been addressed to mitigate the impact of stragglers.
Previous studies split models to tackle the issue, but with less
degre... more
Federated learning (FL) is a promising approach in distributed learning
keeping privacy. However, during the training pipeline of FL, slow or incapable
clients (i.e., stragglers) slow down the total training time and degrade
performance. System heterogeneity, including heterogeneous computing and
network bandwidth, has been addressed to mitigate the impact of stragglers.
Previous studies split models to tackle the issue, but with less
degree-of-freedom in terms of model architecture. We propose nested federated
learning (NeFL), a generalized framework that efficiently divides a model into
submodels using both depthwise and widthwise scaling. NeFL is implemented by
interpreting models as solving ordinary differential equations (ODEs) with
adaptive step sizes. To address the inconsistency that arises when training
multiple submodels with different architecture, we decouple a few parameters.
NeFL enables resource-constrained clients to effectively join the FL pipeline
and the model to be trained with a larger amount of data. Through a series of
experiments, we demonstrate that NeFL leads to significant gains, especially
for the worst-case submodel (e.g., 8.33 improvement on CIFAR-10). Furthermore,
we demonstrate NeFL aligns with recent studies in FL.
less
By: L. Féret, A. Gepperth, S. Lambeck
This article investigates synthetic model-predictive control (MPC) problems
to demonstrate that an increased precision of the internal prediction model
(PM) automatially entails an improvement of the controller as a whole. In
contrast to reinforcement learning (RL), MPC uses the PM to predict subsequent
states of the controlled system (CS), instead of directly recommending suitable
actions. To assess how the precision of the PM translates i... more
This article investigates synthetic model-predictive control (MPC) problems
to demonstrate that an increased precision of the internal prediction model
(PM) automatially entails an improvement of the controller as a whole. In
contrast to reinforcement learning (RL), MPC uses the PM to predict subsequent
states of the controlled system (CS), instead of directly recommending suitable
actions. To assess how the precision of the PM translates into the quality of
the model-predictive controller, we compare a DNN-based PM to the optimal
baseline PM for three well-known control problems of varying complexity. The
baseline PM achieves perfect accuracy by accessing the simulation of the CS
itself. Based on the obtained results, we argue that an improvement of the PM
will always improve the controller as a whole, without considering the impact
of other components such as action selection (which, in this article, relies on
evolutionary optimization).
less
By: Rui Feng, Huan Tran, Aubrey Toland, Binghong Chen, Qi Zhu, Rampi Ramprasad, Chao Zhang
Polymer simulation with both accuracy and efficiency is a challenging task.
Machine learning (ML) forcefields have been developed to achieve both the
accuracy of ab initio methods and the efficiency of empirical force fields.
However, existing ML force fields are usually limited to single-molecule
settings, and their simulations are not robust enough. In this paper, we
present PolyGET, a new framework for Polymer Forcefields with Generaliza... more
Polymer simulation with both accuracy and efficiency is a challenging task.
Machine learning (ML) forcefields have been developed to achieve both the
accuracy of ab initio methods and the efficiency of empirical force fields.
However, existing ML force fields are usually limited to single-molecule
settings, and their simulations are not robust enough. In this paper, we
present PolyGET, a new framework for Polymer Forcefields with Generalizable
Equivariant Transformers. PolyGET is designed to capture complex quantum
interactions between atoms and generalize across various polymer families,
using a deep learning model called Equivariant Transformers. We propose a new
training paradigm that focuses exclusively on optimizing forces, which is
different from existing methods that jointly optimize forces and energy. This
simple force-centric objective function avoids competing objectives between
energy and forces, thereby allowing for learning a unified forcefield ML model
over different polymer families. We evaluated PolyGET on a large-scale dataset
of 24 distinct polymer types and demonstrated state-of-the-art performance in
force accuracy and robust MD simulations. Furthermore, PolyGET can simulate
large polymers with high fidelity to the reference ab initio DFT method while
being able to generalize to unseen polymers.
less
By: Steve Vott, Adam M. Lehavi
The Ramsey number is the minimum number of nodes, $n = R(s, t)$, such that
all undirected simple graphs of order $n$, contain a clique of order $s$, or an
independent set of order $t$. This paper explores the application of a best
first search algorithm and reinforcement learning (RL) techniques to find
counterexamples to specific Ramsey numbers. We incrementally improve over prior
search methods such as random search by introducing a graph... more
The Ramsey number is the minimum number of nodes, $n = R(s, t)$, such that
all undirected simple graphs of order $n$, contain a clique of order $s$, or an
independent set of order $t$. This paper explores the application of a best
first search algorithm and reinforcement learning (RL) techniques to find
counterexamples to specific Ramsey numbers. We incrementally improve over prior
search methods such as random search by introducing a graph vectorization and
deep neural network (DNN)-based heuristic, which gauge the likelihood of a
graph being a counterexample. The paper also proposes algorithmic optimizations
to confine a polynomial search runtime. This paper does not aim to present new
counterexamples but rather introduces and evaluates a framework supporting
Ramsey counterexample exploration using other heuristics. Code and methods are
made available through a PyPI package and GitHub repository.
less
By: Le Thi Khanh Hien, Sukanya Patra, Souhaib Ben Taieb
A significant limitation of one-class classification anomaly detection
methods is their reliance on the assumption that unlabeled training data only
contains normal instances. To overcome this impractical assumption, we propose
two novel classification-based anomaly detection methods. Firstly, we introduce
a semi-supervised shallow anomaly detection method based on an unbiased risk
estimator. Secondly, we present a semi-supervised deep anom... more
A significant limitation of one-class classification anomaly detection
methods is their reliance on the assumption that unlabeled training data only
contains normal instances. To overcome this impractical assumption, we propose
two novel classification-based anomaly detection methods. Firstly, we introduce
a semi-supervised shallow anomaly detection method based on an unbiased risk
estimator. Secondly, we present a semi-supervised deep anomaly detection method
utilizing a nonnegative (biased) risk estimator. We establish estimation error
bounds and excess risk bounds for both risk minimizers. Additionally, we
propose techniques to select appropriate regularization parameters that ensure
the nonnegativity of the empirical risk in the shallow model under specific
loss functions. Our extensive experiments provide strong evidence of the
effectiveness of the risk-based anomaly detection methods.
less
By: Coenraad Mouton, Marthinus W. Theunissen, Marelie H. Davel
Understanding generalization in deep neural networks is an active area of
research. A promising avenue of exploration has been that of margin
measurements: the shortest distance to the decision boundary for a given sample
or its representation internal to the network. While margins have been shown to
be correlated with the generalization ability of a model when measured at its
hidden representations (hidden margins), no such link between la... more
Understanding generalization in deep neural networks is an active area of
research. A promising avenue of exploration has been that of margin
measurements: the shortest distance to the decision boundary for a given sample
or its representation internal to the network. While margins have been shown to
be correlated with the generalization ability of a model when measured at its
hidden representations (hidden margins), no such link between large margins and
generalization has been established for input margins. We show that while input
margins are not generally predictive of generalization, they can be if the
search space is appropriately constrained. We develop such a measure based on
input margins, which we refer to as `constrained margins'. The predictive power
of this new measure is demonstrated on the 'Predicting Generalization in Deep
Learning' (PGDL) dataset and contrasted with hidden representation margins. We
find that constrained margins achieve highly competitive scores and outperform
other margin measurements in general. This provides a novel insight on the
relationship between generalization and classification margins, and highlights
the importance of considering the data manifold for investigations of
generalization in DNNs.
less
By: Nikolaos Koursioumpas, Lina Magoula, Nikolaos Petropouleas, Alexandros-Ioannis Thanopoulos, Theodora Panagea, Nancy Alonistioti, M. A. Gutierrez-Estevez, Ramin Khalili
Progressing towards a new era of Artificial Intelligence (AI) - enabled
wireless networks, concerns regarding the environmental impact of AI have been
raised both in industry and academia. Federated Learning (FL) has emerged as a
key privacy preserving decentralized AI technique. Despite efforts currently
being made in FL, its environmental impact is still an open problem. Targeting
the minimization of the overall energy consumption of an F... more
Progressing towards a new era of Artificial Intelligence (AI) - enabled
wireless networks, concerns regarding the environmental impact of AI have been
raised both in industry and academia. Federated Learning (FL) has emerged as a
key privacy preserving decentralized AI technique. Despite efforts currently
being made in FL, its environmental impact is still an open problem. Targeting
the minimization of the overall energy consumption of an FL process, we propose
the orchestration of computational and communication resources of the involved
devices to minimize the total energy required, while guaranteeing a certain
performance of the model. To this end, we propose a Soft Actor Critic Deep
Reinforcement Learning (DRL) solution, where a penalty function is introduced
during training, penalizing the strategies that violate the constraints of the
environment, and ensuring a safe RL process. A device level synchronization
method, along with a computationally cost effective FL environment are
proposed, with the goal of further reducing the energy consumption and
communication overhead. Evaluation results show the effectiveness of the
proposed scheme compared to four state-of-the-art baseline solutions in both
static and dynamic environments, achieving a decrease of up to 94% in the total
energy consumption.
less
By: Marc Christiansen, Lea Villadsen, Zhiqiang Zhong, Stefano Teso, Davide Mottin
Self-explainable deep neural networks are a recent class of models that can
output ante-hoc local explanations that are faithful to the model's reasoning,
and as such represent a step forward toward filling the gap between
expressiveness and interpretability. Self-explainable graph neural networks
(GNNs) aim at achieving the same in the context of graph data. This begs the
question: do these models fulfill their implicit guarantees in terms... more
Self-explainable deep neural networks are a recent class of models that can
output ante-hoc local explanations that are faithful to the model's reasoning,
and as such represent a step forward toward filling the gap between
expressiveness and interpretability. Self-explainable graph neural networks
(GNNs) aim at achieving the same in the context of graph data. This begs the
question: do these models fulfill their implicit guarantees in terms of
faithfulness? In this extended abstract, we analyze the faithfulness of several
self-explainable GNNs using different measures of faithfulness, identify
several limitations -- both in the models themselves and in the evaluation
metrics -- and outline possible ways forward.
less
By: Michele Guerra, Simone Scardapane, Filippo Maria Bianchi
Some applications of deep learning require not only to provide accurate
results but also to quantify the amount of confidence in their prediction. The
management of an electric power grid is one of these cases: to avoid risky
scenarios, decision-makers need both precise and reliable forecasts of, for
example, power loads. For this reason, point forecasts are not enough hence it
is necessary to adopt methods that provide an uncertainty quant... more
Some applications of deep learning require not only to provide accurate
results but also to quantify the amount of confidence in their prediction. The
management of an electric power grid is one of these cases: to avoid risky
scenarios, decision-makers need both precise and reliable forecasts of, for
example, power loads. For this reason, point forecasts are not enough hence it
is necessary to adopt methods that provide an uncertainty quantification.
This work focuses on reservoir computing as the core time series forecasting
method, due to its computational efficiency and effectiveness in predicting
time series. While the RC literature mostly focused on point forecasting, this
work explores the compatibility of some popular uncertainty quantification
methods with the reservoir setting. Both Bayesian and deterministic approaches
to uncertainty assessment are evaluated and compared in terms of their
prediction accuracy, computational resource efficiency and reliability of the
estimated uncertainty, based on a set of carefully chosen performance metrics.
less
By: Yuxiao Luo, Ziyu Lyu, Xingyu Huang
Long-term time series forecasting is a vital task and has a wide range of
real applications. Recent methods focus on capturing the underlying patterns
from one single domain (e.g. the time domain or the frequency domain), and have
not taken a holistic view to process long-term time series from the
time-frequency domains. In this paper, we propose a Time-Frequency Enhanced
Decomposed Network (TFDNet) to capture both the long-term underlying ... more
Long-term time series forecasting is a vital task and has a wide range of
real applications. Recent methods focus on capturing the underlying patterns
from one single domain (e.g. the time domain or the frequency domain), and have
not taken a holistic view to process long-term time series from the
time-frequency domains. In this paper, we propose a Time-Frequency Enhanced
Decomposed Network (TFDNet) to capture both the long-term underlying patterns
and temporal periodicity from the time-frequency domain. In TFDNet, we devise a
multi-scale time-frequency enhanced encoder backbone and develop two separate
trend and seasonal time-frequency blocks to capture the distinct patterns
within the decomposed trend and seasonal components in multi-resolutions.
Diverse kernel learning strategies of the kernel operations in time-frequency
blocks have been explored, by investigating and incorporating the potential
different channel-wise correlation patterns of multivariate time series.
Experimental evaluation of eight datasets from five benchmark domains
demonstrated that TFDNet is superior to state-of-the-art approaches in both
effectiveness and efficiency.
less
By: Ghazal Afroozi Milani University of Surrey, Daniel Cyrus University of Surrey, Alireza Tamaddoni-Nezhad University of Surrey
With the ever-increasing potential of AI to perform personalised tasks, it is
becoming essential to develop new machine learning techniques which are
data-efficient and do not require hundreds or thousands of training data. In
this paper, we explore an Inductive Logic Programming approach for one-shot
text classification. In particular, we explore the framework of
Meta-Interpretive Learning (MIL), along with using common-sense background
kn... more
With the ever-increasing potential of AI to perform personalised tasks, it is
becoming essential to develop new machine learning techniques which are
data-efficient and do not require hundreds or thousands of training data. In
this paper, we explore an Inductive Logic Programming approach for one-shot
text classification. In particular, we explore the framework of
Meta-Interpretive Learning (MIL), along with using common-sense background
knowledge extracted from ConceptNet. Results indicate that MIL can learn text
classification rules from a small number of training examples. Moreover, the
higher complexity of chosen examples, the higher accuracy of the outcome.
less
By: Jonathan D. Boyd, Joshua H. Tyler, Anthony M. Murphy, Donald R. Reising
As power quality becomes a higher priority in the electric utility industry,
the amount of disturbance event data continues to grow. Utilities do not have
the required personnel to analyze each event by hand. This work presents an
automated approach for analyzing power quality events recorded by digital fault
recorders and power quality monitors operating within a power transmission
system. The automated approach leverages rule-based analyt... more
As power quality becomes a higher priority in the electric utility industry,
the amount of disturbance event data continues to grow. Utilities do not have
the required personnel to analyze each event by hand. This work presents an
automated approach for analyzing power quality events recorded by digital fault
recorders and power quality monitors operating within a power transmission
system. The automated approach leverages rule-based analytics to examine the
time and frequency domain characteristics of the voltage and current signals.
Customizable thresholds are set to categorize each disturbance event. The
events analyzed within this work include various faults, motor starting, and
incipient instrument transformer failure. Analytics for fourteen different
event types have been developed. The analytics were tested on 160 signal files
and yielded an accuracy of ninety-nine percent. Continuous, nominal signal data
analysis is performed using an approach coined as the cyclic histogram. The
cyclic histogram process will be integrated into the digital fault recorders
themselves to facilitate the detection of subtle signal variations that are too
small to trigger a disturbance event and that can occur over hours or days. In
addition to reducing memory requirements by a factor of 320, it is anticipated
that cyclic histogram processing will aid in identifying incipient events and
identifiers. This project is expected to save engineers time by automating the
classification of disturbance events and increase the reliability of the
transmission system by providing near real time detection and identification of
disturbances as well as prevention of problems before they occur.
less
By: Qianyu Long, Christos Anagnostopoulos, Shameem Puthiya Parambath, Daning Bi
Federated Learning (FL) has been successfully adopted for distributed
training and inference of large-scale Deep Neural Networks (DNNs). However,
DNNs are characterized by an extremely large number of parameters, thus,
yielding significant challenges in exchanging these parameters among
distributed nodes and managing the memory. Although recent DNN compression
methods (e.g., sparsification, pruning) tackle such challenges, they do not
holis... more
Federated Learning (FL) has been successfully adopted for distributed
training and inference of large-scale Deep Neural Networks (DNNs). However,
DNNs are characterized by an extremely large number of parameters, thus,
yielding significant challenges in exchanging these parameters among
distributed nodes and managing the memory. Although recent DNN compression
methods (e.g., sparsification, pruning) tackle such challenges, they do not
holistically consider an adaptively controlled reduction of parameter exchange
while maintaining high accuracy levels. We, therefore, contribute with a novel
FL framework (coined FedDIP), which combines (i) dynamic model pruning with
error feedback to eliminate redundant information exchange, which contributes
to significant performance improvement, with (ii) incremental regularization
that can achieve \textit{extreme} sparsity of models. We provide convergence
analysis of FedDIP and report on a comprehensive performance and comparative
assessment against state-of-the-art methods using benchmark data sets and DNN
models. Our results showcase that FedDIP not only controls the model sparsity
but efficiently achieves similar or better performance compared to other model
pruning methods adopting incremental regularization during distributed model
training. The code is available at: https://github.com/EricLoong/feddip.
less
By: Khotso Selialia, Yasra Chandio, Fatima M. Anwar
Federated Learning is emerging as a privacy-preserving model training
approach in distributed edge applications. As such, most edge deployments are
heterogeneous in nature i.e., their sensing capabilities and environments vary
across deployments. This edge heterogeneity violates the independence and
identical distribution (IID) property of local data across clients and produces
biased global models i.e. models that contribute to unfair deci... more
Federated Learning is emerging as a privacy-preserving model training
approach in distributed edge applications. As such, most edge deployments are
heterogeneous in nature i.e., their sensing capabilities and environments vary
across deployments. This edge heterogeneity violates the independence and
identical distribution (IID) property of local data across clients and produces
biased global models i.e. models that contribute to unfair decision-making and
discrimination against a particular community or a group. Existing bias
mitigation techniques only focus on bias generated from label heterogeneity in
non-IID data without accounting for domain variations due to feature
heterogeneity and do not address global group-fairness property.
Our work proposes a group-fair FL framework that minimizes group-bias while
preserving privacy and without resource utilization overhead. Our main idea is
to leverage average conditional probabilities to compute a cross-domain group
\textit{importance weights} derived from heterogeneous training data to
optimize the performance of the worst-performing group using a modified
multiplicative weights update method. Additionally, we propose regularization
techniques to minimize the difference between the worst and best-performing
groups while making sure through our thresholding mechanism to strike a balance
between bias reduction and group performance degradation. Our evaluation of
human emotion recognition and image classification benchmarks assesses the fair
decision-making of our framework in real-world heterogeneous settings.
less
By: Haohui Lu, Shahadat Uddin
Link prediction is a key aspect of graph machine learning, with applications
as diverse as disease prediction, social network recommendations, and drug
discovery. It involves predicting new links that may form between network
nodes. Despite the clear importance of link prediction, existing models have
significant shortcomings. Graph Convolutional Networks, for instance, have been
proven to be highly efficient for link prediction on a variet... more
Link prediction is a key aspect of graph machine learning, with applications
as diverse as disease prediction, social network recommendations, and drug
discovery. It involves predicting new links that may form between network
nodes. Despite the clear importance of link prediction, existing models have
significant shortcomings. Graph Convolutional Networks, for instance, have been
proven to be highly efficient for link prediction on a variety of datasets.
However, they encounter severe limitations when applied to short-path networks
and ego networks, resulting in poor performance. This presents a critical
problem space that this work aims to address. In this paper, we present the
Node Centrality and Similarity Based Parameterised Model (NCSM), a novel method
for link prediction tasks. NCSM uniquely integrates node centrality and
similarity measures as edge features in a customised Graph Neural Network (GNN)
layer, effectively leveraging the topological information of large networks.
This model represents the first parameterised GNN-based link prediction model
that considers topological information. The proposed model was evaluated on
five benchmark graph datasets, each comprising thousands of nodes and edges.
Experimental results highlight NCSM's superiority over existing
state-of-the-art models like Graph Convolutional Networks and Variational Graph
Autoencoder, as it outperforms them across various metrics and datasets. This
exceptional performance can be attributed to NCSM's innovative integration of
node centrality, similarity measures, and its efficient use of topological
information.
less
By: Haohao Qu, Haoxuan Kuang, Jun Li, Linlin You
Along with the proliferation of electric vehicles (EVs), optimizing the use
of EV charging space can significantly alleviate the growing load on
intelligent transportation systems. As the foundation to achieve such an
optimization, a spatiotemporal method for EV charging demand prediction in
urban areas is required. Although several solutions have been proposed by using
data-driven deep learning methods, it can be found that these
performan... more
Along with the proliferation of electric vehicles (EVs), optimizing the use
of EV charging space can significantly alleviate the growing load on
intelligent transportation systems. As the foundation to achieve such an
optimization, a spatiotemporal method for EV charging demand prediction in
urban areas is required. Although several solutions have been proposed by using
data-driven deep learning methods, it can be found that these
performance-oriented methods may suffer from misinterpretations to correctly
handle the reverse relationship between charging demands and prices. To tackle
the emerging challenges of training an accurate and interpretable prediction
model, this paper proposes a novel approach that enables the integration of
graph and temporal attention mechanisms for feature extraction and the usage of
physic-informed meta-learning in the model pre-training step for knowledge
transfer. Evaluation results on a dataset of 18,013 EV charging piles in
Shenzhen, China, show that the proposed approach, named PAG, can achieve
state-of-the-art forecasting performance and the ability in understanding the
adaptive changes in charging demands caused by price fluctuations.
less
Hyperbolic Random Forests
0upvotes
By: Lars Doorenbos, Pablo Márquez-Neila, Raphael Sznitman, Pascal Mettes
Hyperbolic space is becoming a popular choice for representing data due to
the hierarchical structure - whether implicit or explicit - of many real-world
datasets. Along with it comes a need for algorithms capable of solving
fundamental tasks, such as classification, in hyperbolic space. Recently,
multiple papers have investigated hyperbolic alternatives to hyperplane-based
classifiers, such as logistic regression and SVMs. While effective,... more
Hyperbolic space is becoming a popular choice for representing data due to
the hierarchical structure - whether implicit or explicit - of many real-world
datasets. Along with it comes a need for algorithms capable of solving
fundamental tasks, such as classification, in hyperbolic space. Recently,
multiple papers have investigated hyperbolic alternatives to hyperplane-based
classifiers, such as logistic regression and SVMs. While effective, these
approaches struggle with more complex hierarchical data. We, therefore, propose
to generalize the well-known random forests to hyperbolic space. We do this by
redefining the notion of a split using horospheres. Since finding the globally
optimal split is computationally intractable, we find candidate horospheres
through a large-margin classifier. To make hyperbolic random forests work on
multi-class data and imbalanced experiments, we furthermore outline a new
method for combining classes based on their lowest common ancestor and a
class-balanced version of the large-margin loss. Experiments on standard and
new benchmarks show that our approach outperforms both conventional random
forest algorithms and recent hyperbolic classifiers.
less
By: Christos Chrysanthos Nikolaidis, Vasileios Perifanis, Nikolaos Pavlidis, Pavlos S. Efraimidis
The provision of social care applications is crucial for elderly people to
improve their quality of life and enables operators to provide early
interventions. Accurate predictions of user dropouts in healthy ageing
applications are essential since they are directly related to individual health
statuses. Machine Learning (ML) algorithms have enabled highly accurate
predictions, outperforming traditional statistical methods that struggle to
c... more
The provision of social care applications is crucial for elderly people to
improve their quality of life and enables operators to provide early
interventions. Accurate predictions of user dropouts in healthy ageing
applications are essential since they are directly related to individual health
statuses. Machine Learning (ML) algorithms have enabled highly accurate
predictions, outperforming traditional statistical methods that struggle to
cope with individual patterns. However, ML requires a substantial amount of
data for training, which is challenging due to the presence of personal
identifiable information (PII) and the fragmentation posed by regulations. In
this paper, we present a federated machine learning (FML) approach that
minimizes privacy concerns and enables distributed training, without
transferring individual data. We employ collaborative training by considering
individuals and organizations under FML, which models both cross-device and
cross-silo learning scenarios. Our approach is evaluated on a real-world
dataset with non-independent and identically distributed (non-iid) data among
clients, class imbalance and label ambiguity. Our results show that data
selection and class imbalance handling techniques significantly improve the
predictive accuracy of models trained under FML, demonstrating comparable or
superior predictive performance than traditional ML models.
less
Lie-Poisson Neural Networks (LPNets): Data-Based Computing of Hamiltonian Systems with Symmetries
0upvotes
By: Christopher Eldred, François Gay-Balmaz, Sofiia Huraka, Vakhtang Putkaradze
An accurate data-based prediction of the long-term evolution of Hamiltonian
systems requires a network that preserves the appropriate structure under each
time step. Every Hamiltonian system contains two essential ingredients: the
Poisson bracket and the Hamiltonian. Hamiltonian systems with symmetries, whose
paradigm examples are the Lie-Poisson systems, have been shown to describe a
broad category of physical phenomena, from satellite mot... more
An accurate data-based prediction of the long-term evolution of Hamiltonian
systems requires a network that preserves the appropriate structure under each
time step. Every Hamiltonian system contains two essential ingredients: the
Poisson bracket and the Hamiltonian. Hamiltonian systems with symmetries, whose
paradigm examples are the Lie-Poisson systems, have been shown to describe a
broad category of physical phenomena, from satellite motion to underwater
vehicles, fluids, geophysical applications, complex fluids, and plasma physics.
The Poisson bracket in these systems comes from the symmetries, while the
Hamiltonian comes from the underlying physics. We view the symmetry of the
system as primary, hence the Lie-Poisson bracket is known exactly, whereas the
Hamiltonian is regarded as coming from physics and is considered not known, or
known approximately. Using this approach, we develop a network based on
transformations that exactly preserve the Poisson bracket and the special
functions of the Lie-Poisson systems (Casimirs) to machine precision. We
present two flavors of such systems: one, where the parameters of
transformations are computed from data using a dense neural network (LPNets),
and another, where the composition of transformations is used as building
blocks (G-LPNets). We also show how to adapt these methods to a larger class of
Poisson brackets. We apply the resulting methods to several examples, such as
rigid body (satellite) motion, underwater vehicles, a particle in a magnetic
field, and others. The methods developed in this paper are important for the
construction of accurate data-based methods for simulating the long-term
dynamics of physical systems.
less
By: Emanuele Marconato, Andrea Passerini, Stefano Teso
Focus in Explainable AI is shifting from explanations defined in terms of
low-level elements, such as input features, to explanations encoded in terms of
interpretable concepts learned from data. How to reliably acquire such concepts
is, however, still fundamentally unclear. An agreed-upon notion of concept
interpretability is missing, with the result that concepts used by both
post-hoc explainers and concept-based neural networks are acqui... more
Focus in Explainable AI is shifting from explanations defined in terms of
low-level elements, such as input features, to explanations encoded in terms of
interpretable concepts learned from data. How to reliably acquire such concepts
is, however, still fundamentally unclear. An agreed-upon notion of concept
interpretability is missing, with the result that concepts used by both
post-hoc explainers and concept-based neural networks are acquired through a
variety of mutually incompatible strategies. Critically, most of these neglect
the human side of the problem: a representation is understandable only insofar
as it can be understood by the human at the receiving end. The key challenge in
Human-interpretable Representation Learning (HRL) is how to model and
operationalize this human element. In this work, we propose a mathematical
framework for acquiring interpretable representations suitable for both
post-hoc explainers and concept-based neural networks. Our formalization of HRL
builds on recent advances in causal representation learning and explicitly
models a human stakeholder as an external observer. This allows us to derive a
principled notion of alignment between the machine representation and the
vocabulary of concepts understood by the human. In doing so, we link alignment
and interpretability through a simple and intuitive name transfer game, and
clarify the relationship between alignment and a well-known property of
representations, namely disentanglment. We also show that alignment is linked
to the issue of undesirable correlations among concepts, also known as concept
leakage, and to content-style separation, all through a general
information-theoretic reformulation of these properties. Our conceptualization
aims to bridge the gap between the human and algorithmic sides of
interpretability and establish a stepping stone for new research on
human-interpretable representations.
less
Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity
0upvotes
By: Ng Shyh-Chang, A-Li Luo, Bo Qiu
We prove the converse of the universal approximation theorem, i.e. a neural
network (NN) encoding theorem which shows that for every stably converged NN of
continuous activation functions, its weight matrix actually encodes a
continuous function that approximates its training dataset to within a finite
margin of error over a bounded domain. We further show that using the
Eckart-Young theorem for truncated singular value decomposition of the... more
We prove the converse of the universal approximation theorem, i.e. a neural
network (NN) encoding theorem which shows that for every stably converged NN of
continuous activation functions, its weight matrix actually encodes a
continuous function that approximates its training dataset to within a finite
margin of error over a bounded domain. We further show that using the
Eckart-Young theorem for truncated singular value decomposition of the weight
matrix for every NN layer, we can illuminate the nature of the latent space
manifold of the training dataset encoded and represented by every NN layer, and
the geometric nature of the mathematical operations performed by each NN layer.
Our results have implications for understanding how NNs break the curse of
dimensionality by harnessing memory capacity for expressivity, and that the two
are complementary. This Layer Matrix Decomposition (LMD) further suggests a
close relationship between eigen-decomposition of NN layers and the latest
advances in conceptualizations of Hopfield networks and Transformer NN models.
less
By: Peifeng Ma, Li Chen, Chang Yu, Qing Zhu, Yulin Ding
Landslide susceptibility assessment (LSA) is of paramount importance in
mitigating landslide risks. Recently, there has been a surge in the utilization
of data-driven methods for predicting landslide susceptibility due to the
growing availability of aerial and satellite data. Nonetheless, the rapid
oscillations within the landslide-inducing environment (LIE), primarily due to
significant changes in external triggers such as rainfall, pose d... more
Landslide susceptibility assessment (LSA) is of paramount importance in
mitigating landslide risks. Recently, there has been a surge in the utilization
of data-driven methods for predicting landslide susceptibility due to the
growing availability of aerial and satellite data. Nonetheless, the rapid
oscillations within the landslide-inducing environment (LIE), primarily due to
significant changes in external triggers such as rainfall, pose difficulties
for contemporary data-driven LSA methodologies to accommodate LIEs over diverse
timespans. This study presents dynamic landslide susceptibility mapping that
simply employs multiple predictive models for annual LSA. In practice, this
will inevitably encounter small sample problems due to the limited number of
landslide samples in certain years. Another concern arises owing to the
majority of the existing LSA approaches train black-box models to fit distinct
datasets, yet often failing in generalization and providing comprehensive
explanations concerning the interactions between input features and
predictions. Accordingly, we proposed to meta-learn representations with fast
adaptation ability using a few samples and gradient updates; and apply SHAP for
each model interpretation and landslide feature permutation. Additionally, we
applied MT-InSAR for LSA result enhancement and validation. The chosen study
area is Lantau Island, Hong Kong, where we conducted a comprehensive dynamic
LSA spanning from 1992 to 2019. The model interpretation results demonstrate
that the primary factors responsible for triggering landslides in Lantau Island
are terrain slope and extreme rainfall. The results also indicate that the
variation in landslide causes can be primarily attributed to extreme rainfall
events, which result from global climate change, and the implementation of the
Landslip Prevention and Mitigation Programme (LPMitP) by the Hong Kong
government.
less
By: Yin Fang, Qiang Zhang, Zhuo Chen, Xiaohui Fan, Huajun Chen
Machine learning, especially deep learning, has greatly advanced molecular
studies in the biochemical domain. Most typically, modeling for most molecular
tasks have converged to several paradigms. For example, we usually adopt the
prediction paradigm to solve tasks of molecular property prediction. To improve
the generation and interpretability of purely data-driven models, researchers
have incorporated biochemical domain knowledge into the... more
Machine learning, especially deep learning, has greatly advanced molecular
studies in the biochemical domain. Most typically, modeling for most molecular
tasks have converged to several paradigms. For example, we usually adopt the
prediction paradigm to solve tasks of molecular property prediction. To improve
the generation and interpretability of purely data-driven models, researchers
have incorporated biochemical domain knowledge into these models for molecular
studies. This knowledge incorporation has led to a rising trend of paradigm
transfer, which is solving one molecular learning task by reformulating it as
another one. In this paper, we present a literature review towards
knowledge-informed molecular learning in perspective of paradigm transfer,
where we categorize the paradigms, review their methods and analyze how domain
knowledge contributes. Furthermore, we summarize the trends and point out
interesting future directions for molecular learning.
less
By: Hao Mei, Junxian Li, Zhiming Liang, Guanjie Zheng, Bin Shi, Hua Wei
Traffic prediction is a crucial topic because of its broad scope of
applications in the transportation domain. Recently, various studies have
achieved promising results. However, most studies assume the prediction
locations have complete or at least partial historical records and cannot be
extended to non-historical recorded locations. In real-life scenarios, the
deployment of sensors could be limited due to budget limitations and
installat... more
Traffic prediction is a crucial topic because of its broad scope of
applications in the transportation domain. Recently, various studies have
achieved promising results. However, most studies assume the prediction
locations have complete or at least partial historical records and cannot be
extended to non-historical recorded locations. In real-life scenarios, the
deployment of sensors could be limited due to budget limitations and
installation availability, which makes most current models not applicable.
Though few pieces of literature tried to impute traffic states at the missing
locations, these methods need the data simultaneously observed at the locations
with sensors, making them not applicable to prediction tasks. Another drawback
is the lack of measurement of uncertainty in prediction, making prior works
unsuitable for risk-sensitive tasks or involving decision-making. To fill the
gap, inspired by the previous inductive graph neural network, this work
proposed an uncertainty-aware framework with the ability to 1) extend
prediction to missing locations with no historical records and significantly
extend spatial coverage of prediction locations while reducing deployment of
sensors and 2) generate probabilistic prediction with uncertainty
quantification to help the management of risk and decision making in the
down-stream tasks. Through extensive experiments on real-life datasets, the
result shows our method achieved promising results on prediction tasks, and the
uncertainty quantification gives consistent results which highly correlated
with the locations with and without historical data. We also show that our
model could help support sensor deployment tasks in the transportation field to
achieve higher accuracy with a limited sensor deployment budget.
less
By: Can Jiang, Xiong Liang, Yu-Cheng Zhou, Yong Tian, Shengli Xu, Jia-Rui Lin, Zhiliang Ma, Shiji Yang, Hao Zhou
In Chinese building codes, it is required that residential buildings receive
a minimum number of hours of natural, direct sunlight on a specified winter
day, which represents the worst sunlight condition in a year. This requirement
is a prerequisite for obtaining a building permit during the conceptual design
of a residential project. Thus, officially sanctioned software is usually used
to assess the sunlight performance of buildings. These... more
In Chinese building codes, it is required that residential buildings receive
a minimum number of hours of natural, direct sunlight on a specified winter
day, which represents the worst sunlight condition in a year. This requirement
is a prerequisite for obtaining a building permit during the conceptual design
of a residential project. Thus, officially sanctioned software is usually used
to assess the sunlight performance of buildings. These software programs
predict sunlight hours based on repeated shading calculations, which is
time-consuming. This paper proposed a multilayer perceptron-based method, a
one-stage prediction approach, which outputs a shading time interval caused by
the inputted cuboid-form building. The sunlight hours of a site can be obtained
by calculating the union of the sunlight time intervals (complement of shading
time interval) of all the buildings. Three numerical experiments, i.e.,
horizontal level and slope analysis, and simulation-based optimization are
carried out; the results show that the method reduces the computation time to
1/84~1/50 with 96.5%~98% accuracies. A residential neighborhood layout planning
plug-in for Rhino 7/Grasshopper is also developed based on the proposed model.
This paper indicates that deep learning techniques can be adopted to accelerate
sunlight hour simulations at the conceptual design phase.
less
By: Andrew M. Dai, Quoc V. Le
We present two approaches that use unlabeled data to improve sequence
learning with recurrent networks. The first approach is to predict what comes
next in a sequence, which is a conventional language model in natural language
processing. The second approach is to use a sequence autoencoder, which reads
the input sequence into a vector and predicts the input sequence again. These
two algorithms can be used as a "pretraining" step for a late... more
We present two approaches that use unlabeled data to improve sequence
learning with recurrent networks. The first approach is to predict what comes
next in a sequence, which is a conventional language model in natural language
processing. The second approach is to use a sequence autoencoder, which reads
the input sequence into a vector and predicts the input sequence again. These
two algorithms can be used as a "pretraining" step for a later supervised
sequence learning algorithm. In other words, the parameters obtained from the
unsupervised step can be used as a starting point for other supervised training
models. In our experiments, we find that long short term memory recurrent
networks after being pretrained with the two approaches are more stable and
generalize better. With pretraining, we are able to train long short term
memory recurrent networks up to a few hundred timesteps, thereby achieving
strong performance in many text classification tasks, such as IMDB, DBpedia and
20 Newsgroups.
less
By: Jack Foster, Stefan Schoepf, Alexandra Brintrup
Machine unlearning, the ability for a machine learning model to forget, is
becoming increasingly important to comply with data privacy regulations, as
well as to remove harmful, manipulated, or outdated information. The key
challenge lies in forgetting specific information while protecting model
performance on the remaining data. While current state-of-the-art methods
perform well, they typically require some level of retraining over the re... more
Machine unlearning, the ability for a machine learning model to forget, is
becoming increasingly important to comply with data privacy regulations, as
well as to remove harmful, manipulated, or outdated information. The key
challenge lies in forgetting specific information while protecting model
performance on the remaining data. While current state-of-the-art methods
perform well, they typically require some level of retraining over the retained
data, in order to protect or restore model performance. This adds computational
overhead and mandates that the training data remain available and accessible,
which may not be feasible. In contrast, other methods employ a retrain-free
paradigm, however, these approaches are prohibitively computationally expensive
and do not perform on par with their retrain-based counterparts. We present
Selective Synaptic Dampening (SSD), a novel two-step, post hoc, retrain-free
approach to machine unlearning which is fast, performant, and does not require
long-term storage of the training data. First, SSD uses the Fisher information
matrix of the training and forgetting data to select parameters that are
disproportionately important to the forget set. Second, SSD induces forgetting
by dampening these parameters proportional to their relative importance to the
forget set with respect to the wider training data. We evaluate our method
against several existing unlearning methods in a range of experiments using
ResNet18 and Vision Transformer. Results show that the performance of SSD is
competitive with retrain-based post hoc methods, demonstrating the viability of
retrain-free post hoc unlearning approaches.
less
A multiobjective continuation method to compute the regularization path of deep neural networks
0upvotes
By: Augustina C. Amakor, Konstantin Sontag, Sebastian Peitz
Sparsity is a highly desired feature in deep neural networks (DNNs) since it
ensures numerical efficiency, improves the interpretability of models (due to
the smaller number of relevant features), and robustness. In machine learning
approaches based on linear models, it is well known that there exists a
connecting path between the sparsest solution in terms of the $\ell^1$ norm
(i.e., zero weights) and the non-regularized solution, which is... more
Sparsity is a highly desired feature in deep neural networks (DNNs) since it
ensures numerical efficiency, improves the interpretability of models (due to
the smaller number of relevant features), and robustness. In machine learning
approaches based on linear models, it is well known that there exists a
connecting path between the sparsest solution in terms of the $\ell^1$ norm
(i.e., zero weights) and the non-regularized solution, which is called the
regularization path. Very recently, there was a first attempt to extend the
concept of regularization paths to DNNs by means of treating the empirical loss
and sparsity ($\ell^1$ norm) as two conflicting criteria and solving the
resulting multiobjective optimization problem. However, due to the
non-smoothness of the $\ell^1$ norm and the high number of parameters, this
approach is not very efficient from a computational perspective. To overcome
this limitation, we present an algorithm that allows for the approximation of
the entire Pareto front for the above-mentioned objectives in a very efficient
manner. We present numerical examples using both deterministic and stochastic
gradients. We furthermore demonstrate that knowledge of the regularization path
allows for a well-generalizing network parametrization.
less
By: Miranda Bihler, Hala Nelson, Erin Okey, Noe Reyes Rivas, John Webb, Anna White
The Harrisonburg Department of Public Transportation (HDPT) aims to leverage
their data to improve the efficiency and effectiveness of their operations. We
construct two supply and demand models that help the department identify gaps
in their service. The models take many variables into account, including the
way that the HDPT reports to the federal government and the areas with the most
vulnerable populations in Harrisonburg City. We emplo... more
The Harrisonburg Department of Public Transportation (HDPT) aims to leverage
their data to improve the efficiency and effectiveness of their operations. We
construct two supply and demand models that help the department identify gaps
in their service. The models take many variables into account, including the
way that the HDPT reports to the federal government and the areas with the most
vulnerable populations in Harrisonburg City. We employ data analysis and
machine learning techniques to make our predictions.
less
By: Luca Gherardini, Varun Ravi Varma, Karol Capala, Roger Woods, Jose Sousa
The availability of large data sets is providing an impetus for driving
current artificial intelligent developments. There are, however, challenges for
developing solutions with small data sets due to practical and cost-effective
deployment and the opacity of deep learning models. The Comprehensive
Abstraction and Classification Tool for Uncovering Structures called CACTUS is
presented for improved secure analytics by effectively employing ... more
The availability of large data sets is providing an impetus for driving
current artificial intelligent developments. There are, however, challenges for
developing solutions with small data sets due to practical and cost-effective
deployment and the opacity of deep learning models. The Comprehensive
Abstraction and Classification Tool for Uncovering Structures called CACTUS is
presented for improved secure analytics by effectively employing explainable
artificial intelligence. It provides additional support for categorical
attributes, preserving their original meaning, optimising memory usage, and
speeding up the computation through parallelisation. It shows to the user the
frequency of the attributes in each class and ranks them by their
discriminative power. Its performance is assessed by application to the
Wisconsin diagnostic breast cancer and Thyroid0387 data sets.
less
Federated Learning in UAV-Enhanced Networks: Joint Coverage and Convergence Time Optimization
0upvotes
By: Mariam Yahya, Setareh Maghsudi, Slawomir Stanczak
Federated learning (FL) involves several devices that collaboratively train a
shared model without transferring their local data. FL reduces the
communication overhead, making it a promising learning method in UAV-enhanced
wireless networks with scarce energy resources. Despite the potential,
implementing FL in UAV-enhanced networks is challenging, as conventional UAV
placement methods that maximize coverage increase the FL delay significan... more
Federated learning (FL) involves several devices that collaboratively train a
shared model without transferring their local data. FL reduces the
communication overhead, making it a promising learning method in UAV-enhanced
wireless networks with scarce energy resources. Despite the potential,
implementing FL in UAV-enhanced networks is challenging, as conventional UAV
placement methods that maximize coverage increase the FL delay significantly.
Moreover, the uncertainty and lack of a priori information about crucial
variables, such as channel quality, exacerbate the problem. In this paper, we
first analyze the statistical characteristics of a UAV-enhanced wireless sensor
network (WSN) with energy harvesting. We then develop a model and solution
based on the multi-objective multi-armed bandit theory to maximize the network
coverage while minimizing the FL delay. Besides, we propose another solution
that is particularly useful with large action sets and strict energy
constraints at the UAVs. Our proposal uses a scalarized best-arm identification
algorithm to find the optimal arms that maximize the ratio of the expected
reward to the expected energy cost by sequentially eliminating one or more arms
in each round. Then, we derive the upper bound on the error probability of our
multi-objective and cost-aware algorithm. Numerical results show the
effectiveness of our approach.
less
By: Daniel LeJeune, Sina Alemohammad
In order to better understand feature learning in neural networks, we propose
a framework for understanding linear models in tangent feature space where the
features are allowed to be transformed during training. We consider linear
transformations of features, resulting in a joint optimization over parameters
and transformations with a bilinear interpolation constraint. We show that this
optimization problem has an equivalent linearly const... more
In order to better understand feature learning in neural networks, we propose
a framework for understanding linear models in tangent feature space where the
features are allowed to be transformed during training. We consider linear
transformations of features, resulting in a joint optimization over parameters
and transformations with a bilinear interpolation constraint. We show that this
optimization problem has an equivalent linearly constrained optimization with
structured regularization that encourages approximately low rank solutions.
Specializing to neural network structure, we gain insights into how the
features and thus the kernel function change, providing additional nuance to
the phenomenon of kernel alignment when the target function is poorly
represented using tangent features. In addition to verifying our theoretical
observations in real neural networks on a simple regression problem, we
empirically show that an adaptive feature implementation of tangent feature
classification has an order of magnitude lower sample complexity than the fixed
tangent feature model on MNIST and CIFAR-10.
less
By: Amir Mohammad Abouei, Ehsan Mokhtarian, Negar Kiyavash
Causal inference in a sub-population involves identifying the causal effect
of an intervention on a specific subgroup within a larger population. However,
ignoring the subtleties introduced by sub-populations can either lead to
erroneous inference or limit the applicability of existing methods. We
introduce and advocate for a causal inference problem in sub-populations
(henceforth called s-ID), in which we merely have access to observationa... more
Causal inference in a sub-population involves identifying the causal effect
of an intervention on a specific subgroup within a larger population. However,
ignoring the subtleties introduced by sub-populations can either lead to
erroneous inference or limit the applicability of existing methods. We
introduce and advocate for a causal inference problem in sub-populations
(henceforth called s-ID), in which we merely have access to observational data
of the targeted sub-population (as opposed to the entire population). Existing
inference problems in sub-populations operate on the premise that the given
data distributions originate from the entire population, thus, cannot tackle
the s-ID problem. To address this gap, we provide necessary and sufficient
conditions that must hold in the causal graph for a causal effect in a
sub-population to be identifiable from the observational distribution of that
sub-population. Given these conditions, we present a sound and complete
algorithm for the s-ID problem.
less
By: Marwa Chafii, Salmane Naoumi, Reda Alami, Ebtesam Almazrouei, Mehdi Bennis, Merouane Debbah
In different wireless network scenarios, multiple network entities need to
cooperate in order to achieve a common task with minimum delay and energy
consumption. Future wireless networks mandate exchanging high dimensional data
in dynamic and uncertain environments, therefore implementing communication
control tasks becomes challenging and highly complex. Multi-agent reinforcement
learning with emergent communication (EC-MARL) is a promisin... more
In different wireless network scenarios, multiple network entities need to
cooperate in order to achieve a common task with minimum delay and energy
consumption. Future wireless networks mandate exchanging high dimensional data
in dynamic and uncertain environments, therefore implementing communication
control tasks becomes challenging and highly complex. Multi-agent reinforcement
learning with emergent communication (EC-MARL) is a promising solution to
address high dimensional continuous control problems with partially observable
states in a cooperative fashion where agents build an emergent communication
protocol to solve complex tasks. This paper articulates the importance of
EC-MARL within the context of future 6G wireless networks, which imbues
autonomous decision-making capabilities into network entities to solve complex
tasks such as autonomous driving, robot navigation, flying base stations
network planning, and smart city applications. An overview of EC-MARL
algorithms and their design criteria are provided while presenting use cases
and research opportunities on this emerging topic.
less
By: Jishnu Mukhoti, Yarin Gal, Philip H. S. Torr, Puneet K. Dokania
Pre-trained foundation models, owing primarily to their enormous capacity and
exposure to vast amount of training data scraped from the internet, enjoy the
advantage of storing knowledge about plenty of real-world concepts. Such models
are typically fine-tuned on downstream datasets to produce remarkable
state-of-the-art performances. While various fine-tuning methods have been
devised and are shown to be highly effective, we observe that a... more
Pre-trained foundation models, owing primarily to their enormous capacity and
exposure to vast amount of training data scraped from the internet, enjoy the
advantage of storing knowledge about plenty of real-world concepts. Such models
are typically fine-tuned on downstream datasets to produce remarkable
state-of-the-art performances. While various fine-tuning methods have been
devised and are shown to be highly effective, we observe that a fine-tuned
model's ability to recognize concepts on tasks $\textit{different}$ from the
downstream one is reduced significantly compared to its pre-trained
counterpart. This is clearly undesirable as a huge amount of time and money
went into learning those very concepts in the first place. We call this
undesirable phenomenon "concept forgetting" and via experiments show that most
end-to-end fine-tuning approaches suffer heavily from this side effect. To this
end, we also propose a rather simple fix to this problem by designing a method
called LDIFS (short for $\ell_2$ distance in feature space) that simply
preserves the features of the original foundation model during fine-tuning. We
show that LDIFS significantly reduces concept forgetting without having
noticeable impact on the downstream task performance.
less
By: Konstantinos Pitas, Julyan Arbel
We conduct a detailed investigation of tempered posteriors and uncover a
number of crucial and previously undiscussed points. Contrary to previous
results, we first show that for realistic models and datasets and the tightly
controlled case of the Laplace approximation to the posterior, stochasticity
does not in general improve test accuracy. The coldest temperature is often
optimal. One might think that Bayesian models with some stochastic... more
We conduct a detailed investigation of tempered posteriors and uncover a
number of crucial and previously undiscussed points. Contrary to previous
results, we first show that for realistic models and datasets and the tightly
controlled case of the Laplace approximation to the posterior, stochasticity
does not in general improve test accuracy. The coldest temperature is often
optimal. One might think that Bayesian models with some stochasticity can at
least obtain improvements in terms of calibration. However, we show empirically
that when gains are obtained this comes at the cost of degradation in test
accuracy. We then discuss how targeting Frequentist metrics using Bayesian
models provides a simple explanation of the need for a temperature parameter
$\lambda$ in the optimization objective. Contrary to prior works, we finally
show through a PAC-Bayesian analysis that the temperature $\lambda$ cannot be
seen as simply fixing a misspecified prior or likelihood.
less
By: Marin Vlastelica, Tatiana López-Guevara, Kelsey Allen, Peter Battaglia, Arnaud Doucet, Kimberley Stachenfeld
Inverse design refers to the problem of optimizing the input of an objective
function in order to enact a target outcome. For many real-world engineering
problems, the objective function takes the form of a simulator that predicts
how the system state will evolve over time, and the design challenge is to
optimize the initial conditions that lead to a target outcome. Recent
developments in learned simulation have shown that graph neural netw... more
Inverse design refers to the problem of optimizing the input of an objective
function in order to enact a target outcome. For many real-world engineering
problems, the objective function takes the form of a simulator that predicts
how the system state will evolve over time, and the design challenge is to
optimize the initial conditions that lead to a target outcome. Recent
developments in learned simulation have shown that graph neural networks (GNNs)
can be used for accurate, efficient, differentiable estimation of simulator
dynamics, and support high-quality design optimization with gradient- or
sampling-based optimization procedures. However, optimizing designs from
scratch requires many expensive model queries, and these procedures exhibit
basic failures on either non-convex or high-dimensional problems.In this work,
we show how denoising diffusion models (DDMs) can be used to solve inverse
design problems efficiently and propose a particle sampling algorithm for
further improving their efficiency. We perform experiments on a number of fluid
dynamics design challenges, and find that our approach substantially reduces
the number of calls to the simulator compared to standard techniques.
less
By: Ákos Hajnal
This paper presents a novel online learning method that aims at finding a
separator hyperplane between data points labelled as either positive or
negative. Since weights and biases of artificial neurons can directly be
related to hyperplanes in high-dimensional spaces, the technique is applicable
to train perceptron-based binary classifiers in machine learning. In case of
large or imbalanced data sets, use of analytical or gradient-based so... more
This paper presents a novel online learning method that aims at finding a
separator hyperplane between data points labelled as either positive or
negative. Since weights and biases of artificial neurons can directly be
related to hyperplanes in high-dimensional spaces, the technique is applicable
to train perceptron-based binary classifiers in machine learning. In case of
large or imbalanced data sets, use of analytical or gradient-based solutions
can become prohibitive and impractical, where heuristics and approximation
techniques are still applicable. The proposed method is based on the Perceptron
algorithm, however, it tunes neuron weights in just the necessary extent during
searching the separator hyperplane. Due to an appropriate transformation of the
initial data set we need not to consider data labels, neither the bias term.
respectively, reducing separability to a one-class classification problem. The
presented method has proven converge; empirical results show that it can be
more efficient than the Perceptron algorithm, especially, when the size of the
data set exceeds data dimensionality.
less
A Machine Learning Framework to Deconstruct the Primary Drivers for Electricity Market Price Events
0upvotes
By: Milan Jain, Xueqing Sun, Sohom Datta, Abhishek Somani
Power grids are moving towards 100% renewable energy source bulk power grids,
and the overall dynamics of power system operations and electricity markets are
changing. The electricity markets are not only dispatching resources
economically but also taking into account various controllable actions like
renewable curtailment, transmission congestion mitigation, and energy storage
optimization to ensure grid reliability. As a result, price for... more
Power grids are moving towards 100% renewable energy source bulk power grids,
and the overall dynamics of power system operations and electricity markets are
changing. The electricity markets are not only dispatching resources
economically but also taking into account various controllable actions like
renewable curtailment, transmission congestion mitigation, and energy storage
optimization to ensure grid reliability. As a result, price formations in
electricity markets have become quite complex. Traditional root cause analysis
and statistical approaches are rendered inapplicable to analyze and infer the
main drivers behind price formation in the modern grid and markets with
variable renewable energy (VRE). In this paper, we propose a machine
learning-based analysis framework to deconstruct the primary drivers for price
spike events in modern electricity markets with high renewable energy. The
outcomes can be utilized for various critical aspects of market design,
renewable dispatch and curtailment, operations, and cyber-security
applications. The framework can be applied to any ISO or market data; however,
in this paper, it is applied to open-source publicly available datasets from
California Independent System Operator (CAISO) and ISO New England (ISO-NE).
less
By: Jan-Philipp Roche, Oliver Niggemann, Jens Friebe
Existing black box modeling approaches in machine learning suffer from a
fixed input and output feature combination. In this paper, a new approach to
reconstruct missing variables in a set of time series is presented. An
autoencoder is trained as usual with every feature on both sides and the neural
network parameters are fixed after this training. Then, the searched variables
are defined as missing variables at the autoencoder input and op... more
Existing black box modeling approaches in machine learning suffer from a
fixed input and output feature combination. In this paper, a new approach to
reconstruct missing variables in a set of time series is presented. An
autoencoder is trained as usual with every feature on both sides and the neural
network parameters are fixed after this training. Then, the searched variables
are defined as missing variables at the autoencoder input and optimized via
automatic differentiation. This optimization is performed with respect to the
available features loss calculation. With this method, different input and
output feature combinations of the trained model can be realized by defining
the searched variables as missing variables and reconstructing them. The
combination can be changed without training the autoencoder again. The approach
is evaluated on the base of a strongly nonlinear electrical component. It is
working well for one of four variables missing and generally even for multiple
missing variables.
less
By: Frederik Boe Hüttel, Filipe Rodrigues, Francisco Câmara Pereira
It is desirable to have accurate uncertainty estimation from a single
deterministic forward-pass model, as traditional methods for uncertainty
quantification are computationally expensive. However, this is difficult
because single forward-pass models do not sample weights during inference and
often make assumptions about the target distribution, such as assuming it is
Gaussian. This can be restrictive in regression tasks, where the mean and... more
It is desirable to have accurate uncertainty estimation from a single
deterministic forward-pass model, as traditional methods for uncertainty
quantification are computationally expensive. However, this is difficult
because single forward-pass models do not sample weights during inference and
often make assumptions about the target distribution, such as assuming it is
Gaussian. This can be restrictive in regression tasks, where the mean and
standard deviation are inadequate to model the target distribution accurately.
This paper proposes a deep Bayesian quantile regression model that can estimate
the quantiles of a continuous target distribution without the Gaussian
assumption. The proposed method is based on evidential learning, which allows
the model to capture aleatoric and epistemic uncertainty with a single
deterministic forward-pass model. This makes the method efficient and scalable
to large models and datasets. We demonstrate that the proposed method achieves
calibrated uncertainties on non-Gaussian distributions, disentanglement of
aleatoric and epistemic uncertainty, and robustness to out-of-distribution
samples.
less
Career Path Recommendations for Long-term Income Maximization: A Reinforcement Learning Approach
0upvotes
By: Spyros Avlonitis, Dor Lavi, Masoud Mansoury, David Graus
This study explores the potential of reinforcement learning algorithms to
enhance career planning processes. Leveraging data from Randstad The
Netherlands, the study simulates the Dutch job market and develops strategies
to optimize employees' long-term income. By formulating career planning as a
Markov Decision Process (MDP) and utilizing machine learning algorithms such as
Sarsa, Q-Learning, and A2C, we learn optimal policies that recomme... more
This study explores the potential of reinforcement learning algorithms to
enhance career planning processes. Leveraging data from Randstad The
Netherlands, the study simulates the Dutch job market and develops strategies
to optimize employees' long-term income. By formulating career planning as a
Markov Decision Process (MDP) and utilizing machine learning algorithms such as
Sarsa, Q-Learning, and A2C, we learn optimal policies that recommend career
paths with high-income occupations and industries. The results demonstrate
significant improvements in employees' income trajectories, with RL models,
particularly Q-Learning and Sarsa, achieving an average increase of 5% compared
to observed career paths. The study acknowledges limitations, including narrow
job filtering, simplifications in the environment formulation, and assumptions
regarding employment continuity and zero application costs. Future research can
explore additional objectives beyond income optimization and address these
limitations to further enhance career planning processes.
less
By: Furkan Cantürk, Reyhan Aydoğan
Gaining insights into the preferences of new users and subsequently
personalizing recommendations necessitate managing user interactions
intelligently, namely, posing pertinent questions to elicit valuable
information effectively. In this study, our focus is on a specific scenario of
the cold-start problem, where the recommendation system lacks adequate user
presence or access to other users' data is restricted, obstructing employing
user p... more
Gaining insights into the preferences of new users and subsequently
personalizing recommendations necessitate managing user interactions
intelligently, namely, posing pertinent questions to elicit valuable
information effectively. In this study, our focus is on a specific scenario of
the cold-start problem, where the recommendation system lacks adequate user
presence or access to other users' data is restricted, obstructing employing
user profiling methods utilizing existing data in the system. We employ Active
Learning (AL) to solve the addressed problem with the objective of maximizing
information acquisition with minimal user effort. AL operates for selecting
informative data from a large unlabeled set to inquire an oracle to label them
and eventually updating a machine learning (ML) model. We operate AL in an
integrated process of unsupervised, semi-supervised, and supervised ML within
an explanatory preference elicitation process. It harvests user feedback (given
for the system's explanations on the presented items) over informative samples
to update an underlying ML model estimating user preferences. The designed user
interaction facilitates personalizing the system by incorporating user feedback
into the ML model and also enhances user trust by refining the system's
explanations on recommendations. We implement the proposed preference
elicitation methodology for food recommendation. We conducted human experiments
to assess its efficacy in the short term and also experimented with several AL
strategies over synthetic user profiles that we created for two food datasets,
aiming for long-term performance analysis. The experimental results demonstrate
the efficiency of the proposed preference elicitation with limited user-labeled
data while also enhancing user trust through accurate explanations.
less
Simple Modification of the Upper Confidence Bound Algorithm by Generalized Weighted Averages
0upvotes
By: Nobuhito Manome, Shuji Shinohara, Ung-il Chung
The multi-armed bandit (MAB) problem is a classical problem that models
sequential decision-making under uncertainty in reinforcement learning. In this
study, we propose a new generalized upper confidence bound (UCB) algorithm
(GWA-UCB1) by extending UCB1, which is a representative algorithm for MAB
problems, using generalized weighted averages, and present an effective
algorithm for various problem settings. GWA-UCB1 is a two-parameter
gen... more
The multi-armed bandit (MAB) problem is a classical problem that models
sequential decision-making under uncertainty in reinforcement learning. In this
study, we propose a new generalized upper confidence bound (UCB) algorithm
(GWA-UCB1) by extending UCB1, which is a representative algorithm for MAB
problems, using generalized weighted averages, and present an effective
algorithm for various problem settings. GWA-UCB1 is a two-parameter
generalization of the balance between exploration and exploitation in UCB1 and
can be implemented with a simple modification of the UCB1 formula. Therefore,
this algorithm can be easily applied to UCB-based reinforcement learning
models. In preliminary experiments, we investigated the optimal parameters of a
simple generalized UCB1 (G-UCB1), prepared for comparison and GWA-UCB1, in a
stochastic MAB problem with two arms. Subsequently, we confirmed the
performance of the algorithms with the investigated parameters on stochastic
MAB problems when arm reward probabilities were sampled from uniform or normal
distributions and on survival MAB problems assuming more realistic situations.
GWA-UCB1 outperformed G-UCB1, UCB1-Tuned, and Thompson sampling in most problem
settings and can be useful in many situations. The code is available at
https://github.com/manome/python-mab.
less
By: Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, Lei Zhang
In this paper, we present Delta-LoRA, which is a novel parameter-efficient
approach to fine-tune large language models (LLMs). In contrast to LoRA and
other low-rank adaptation methods such as AdaLoRA, Delta-LoRA not only updates
the low-rank matrices $\bA$ and $\bB$, but also propagate the learning to the
pre-trained weights $\bW$ via updates utilizing the delta of the product of two
low-rank matrices ($\bA^{(t+1)}\bB^{(t+1)} - \bA^{(t)}\b... more
In this paper, we present Delta-LoRA, which is a novel parameter-efficient
approach to fine-tune large language models (LLMs). In contrast to LoRA and
other low-rank adaptation methods such as AdaLoRA, Delta-LoRA not only updates
the low-rank matrices $\bA$ and $\bB$, but also propagate the learning to the
pre-trained weights $\bW$ via updates utilizing the delta of the product of two
low-rank matrices ($\bA^{(t+1)}\bB^{(t+1)} - \bA^{(t)}\bB^{(t)}$). Such a
strategy effectively addresses the limitation that the incremental update of
low-rank matrices is inadequate for learning representations capable for
downstream tasks. Moreover, as the update of $\bW$ does not need to compute the
gradients of $\bW$ and store their momentums, Delta-LoRA shares comparable
memory requirements and computational costs with LoRA. Extensive experiments
show that Delta-LoRA significantly outperforms existing low-rank adaptation
methods. We further support these results with comprehensive analyses that
underscore the effectiveness of Delta-LoRA.
less
By: Phitchaya Mangpo Phothilimthana, Sami Abu-El-Haija, Kaidi Cao, Bahare Fatemi, Charith Mendis, Bryan Perozzi
Precise hardware performance models play a crucial role in code
optimizations. They can assist compilers in making heuristic decisions or aid
autotuners in identifying the optimal configuration for a given program. For
example, the autotuner for XLA, a machine learning compiler, discovered 10-20%
speedup on state-of-the-art models serving substantial production traffic at
Google. Although there exist a few datasets for program performance p... more
Precise hardware performance models play a crucial role in code
optimizations. They can assist compilers in making heuristic decisions or aid
autotuners in identifying the optimal configuration for a given program. For
example, the autotuner for XLA, a machine learning compiler, discovered 10-20%
speedup on state-of-the-art models serving substantial production traffic at
Google. Although there exist a few datasets for program performance prediction,
they target small sub-programs such as basic blocks or kernels. This paper
introduces TpuGraphs, a performance prediction dataset on full tensor pro