By: Julian Lorenz, Florian Barthel, Daniel Kienzle, Rainer Lienhart
Current scene graph datasets suffer from strong long-tail distributions of
their predicate classes. Due to a very low number of some predicate classes in
the test sets, no reliable metrics can be retrieved for the rarest classes. We
construct a new panoptic scene graph dataset and a set of metrics that are
designed as a benchmark for the predictive performance especially on rare
predicate classes. To construct the new dataset, we propose a ... more
Current scene graph datasets suffer from strong long-tail distributions of
their predicate classes. Due to a very low number of some predicate classes in
the test sets, no reliable metrics can be retrieved for the rarest classes. We
construct a new panoptic scene graph dataset and a set of metrics that are
designed as a benchmark for the predictive performance especially on rare
predicate classes. To construct the new dataset, we propose a model-assisted
annotation pipeline that efficiently finds rare predicate classes that are
hidden in a large set of images like needles in a haystack.
Contrary to prior scene graph datasets, Haystack contains explicit negative
annotations, i.e. annotations that a given relation does not have a certain
predicate class. Negative annotations are helpful especially in the field of
scene graph generation and open up a whole new set of possibilities to improve
current scene graph generation models.
Haystack is 100% compatible with existing panoptic scene graph datasets and
can easily be integrated with existing evaluation pipelines. Our dataset and
code can be found here: https://lorjul.github.io/haystack/. It includes
annotation files and simple to use scripts and utilities, to help with
integrating our dataset in existing work.
less
By: Zengqun Zhao, Ioannis Patras
This paper presents a novel visual-language model called DFER-CLIP, which is
based on the CLIP model and designed for in-the-wild Dynamic Facial Expression
Recognition (DFER). Specifically, the proposed DFER-CLIP consists of a visual
part and a textual part. For the visual part, based on the CLIP image encoder,
a temporal model consisting of several Transformer encoders is introduced for
extracting temporal facial expression features, and t... more
This paper presents a novel visual-language model called DFER-CLIP, which is
based on the CLIP model and designed for in-the-wild Dynamic Facial Expression
Recognition (DFER). Specifically, the proposed DFER-CLIP consists of a visual
part and a textual part. For the visual part, based on the CLIP image encoder,
a temporal model consisting of several Transformer encoders is introduced for
extracting temporal facial expression features, and the final feature embedding
is obtained as a learnable "class" token. For the textual part, we use as
inputs textual descriptions of the facial behaviour that is related to the
classes (facial expressions) that we are interested in recognising -- those
descriptions are generated using large language models, like ChatGPT. This, in
contrast to works that use only the class names and more accurately captures
the relationship between them. Alongside the textual description, we introduce
a learnable token which helps the model learn relevant context information for
each expression during training. Extensive experiments demonstrate the
effectiveness of the proposed method and show that our DFER-CLIP also achieves
state-of-the-art results compared with the current supervised DFER methods on
the DFEW, FERV39k, and MAFW benchmarks. Code is publicly available at
https://github.com/zengqunzhao/DFER-CLIP.
less
By: Haibo Chen, Lei Zhao, Jun Li, Jian Yang
Artistic style transfer aims to create new artistic images by rendering a
given photograph with the target artistic style. Existing methods learn styles
simply based on global statistics or local patches, lacking careful
consideration of the drawing process in practice. Consequently, the stylization
results either fail to capture abundant and diversified local style patterns,
or contain undesired semantic information of the style image and ... more
Artistic style transfer aims to create new artistic images by rendering a
given photograph with the target artistic style. Existing methods learn styles
simply based on global statistics or local patches, lacking careful
consideration of the drawing process in practice. Consequently, the stylization
results either fail to capture abundant and diversified local style patterns,
or contain undesired semantic information of the style image and deviate from
the global style distribution. To address this issue, we imitate the drawing
process of humans and propose a Two-Stage Statistics-Aware Transformation
(TSSAT) module, which first builds the global style foundation by aligning the
global statistics of content and style features and then further enriches local
style details by swapping the local statistics (instead of local features) in a
patch-wise manner, significantly improving the stylization effects. Moreover,
to further enhance both content and style representations, we introduce two
novel losses: an attention-based content loss and a patch-based style loss,
where the former enables better content preservation by enforcing the semantic
relation in the content image to be retained during stylization, and the latter
focuses on increasing the local style similarity between the style and stylized
images. Extensive qualitative and quantitative experiments verify the
effectiveness of our method.
less
By: YeongHyeon Park, Sungho Kang, Myung Jin Kim, Hyeonho Jeong, Hyunkyu Park, Hyeong Seok Kim, Juneho Yi
Unsupervised anomaly detection (UAD) is a widely adopted approach in industry
due to rare anomaly occurrences and data imbalance. A desirable characteristic
of an UAD model is contained generalization ability which excels in the
reconstruction of seen normal patterns but struggles with unseen anomalies.
Recent studies have pursued to contain the generalization capability of their
UAD models in reconstruction from different perspectives, suc... more
Unsupervised anomaly detection (UAD) is a widely adopted approach in industry
due to rare anomaly occurrences and data imbalance. A desirable characteristic
of an UAD model is contained generalization ability which excels in the
reconstruction of seen normal patterns but struggles with unseen anomalies.
Recent studies have pursued to contain the generalization capability of their
UAD models in reconstruction from different perspectives, such as design of
neural network (NN) structure and training strategy. In contrast, we note that
containing of generalization ability in reconstruction can also be obtained
simply from steep-shaped loss landscape. Motivated by this, we propose a loss
landscape sharpening method by amplifying the reconstruction loss, dubbed Loss
AMPlification (LAMP). LAMP deforms the loss landscape into a steep shape so the
reconstruction error on unseen anomalies becomes greater. Accordingly, the
anomaly detection performance is improved without any change of the NN
architecture. Our findings suggest that LAMP can be easily applied to any
reconstruction error metrics in UAD settings where the reconstruction model is
trained with anomaly-free samples only.
less
By: Zhuoqi Cheng, Simon Lyck Bjært Sørensen, Mikkel Werge Olsen, René Lynge Eriksen, Thiusius Rajeeth Savarimuthu
Deep needle insertion to a target often poses a huge challenge, requiring a
combination of specialized skills, assistive technology, and extensive
training. One of the frequently encountered medical scenarios demanding such
expertise includes the needle insertion into a femoral vessel in the groin.
After the access to the femoral vessel, various medical procedures, such as
cardiac catheterization and extracorporeal membrane oxygenation (ECM... more
Deep needle insertion to a target often poses a huge challenge, requiring a
combination of specialized skills, assistive technology, and extensive
training. One of the frequently encountered medical scenarios demanding such
expertise includes the needle insertion into a femoral vessel in the groin.
After the access to the femoral vessel, various medical procedures, such as
cardiac catheterization and extracorporeal membrane oxygenation (ECMO) can be
performed. However, even with the aid of Ultrasound imaging, achieving
successful insertion can necessitate multiple attempts due to the complexities
of anatomy and tissue deformation. To address this challenge, this paper
presents an innovative technology for needle tip real-time tracking, aiming for
enhanced needle insertion guidance. Specifically, our approach revolves around
the creation of scattering imaging using an optical fiber-equipped needle, and
uses Convolutional Neural Network (CNN) based algorithms to enable real-time
estimation of the needle tip's position and orientation during insertion
procedures. The efficacy of the proposed technology was rigorously evaluated
through three experiments. The first two experiments involved rubber and bacon
phantoms to simulate groin anatomy. The positional errors averaging 2.3+1.5mm
and 2.0+1.2mm, and the orientation errors averaging 0.2+0.11rad and
0.16+0.1rad. Furthermore, the system's capabilities were validated through
experiments conducted on fresh porcine phantom mimicking more complex
anatomical structures, yielding positional accuracy results of 3.2+3.1mm and
orientational accuracy of 0.19+0.1rad. Given the average femoral arterial
radius of 4 to 5mm, the proposed system is demonstrated with a great potential
for precise needle guidance in femoral artery insertion procedures. In
addition, the findings highlight the broader potential applications of the
system in the medical field.
less
Joint Modeling of Feature, Correspondence, and a Compressed Memory for Video Object Segmentation
0upvotes
By: Jiaming Zhang, Yutao Cui, Gangshan Wu, Limin Wang
Current prevailing Video Object Segmentation (VOS) methods usually perform
dense matching between the current and reference frames after extracting their
features. One on hand, the decoupled modeling restricts the targets information
propagation only at high-level feature space. On the other hand, the pixel-wise
matching leads to a lack of holistic understanding of the targets. To overcome
these issues, we propose a unified VOS framework, c... more
Current prevailing Video Object Segmentation (VOS) methods usually perform
dense matching between the current and reference frames after extracting their
features. One on hand, the decoupled modeling restricts the targets information
propagation only at high-level feature space. On the other hand, the pixel-wise
matching leads to a lack of holistic understanding of the targets. To overcome
these issues, we propose a unified VOS framework, coined as JointFormer, for
joint modeling the three elements of feature, correspondence, and a compressed
memory. The core design is the Joint Block, utilizing the flexibility of
attention to simultaneously extract feature and propagate the targets
information to the current tokens and the compressed memory token. This scheme
allows to perform extensive information propagation and discriminative feature
learning. To incorporate the long-term temporal targets information, we also
devise a customized online updating mechanism for the compressed memory token,
which can prompt the information flow along the temporal dimension and thus
improve the global modeling capability. Under the design, our method achieves a
new state-of-art performance on DAVIS 2017 val/test-dev (89.7% and 87.6%) and
YouTube-VOS 2018/2019 val (87.0% and 87.0%) benchmarks, outperforming existing
works by a large margin.
less
CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation
0upvotes
By: Devaansh Gupta, Siddhant Kharbanda, Jiawei Zhou, Wanhua Li, Hanspeter Pfister, Donglai Wei
There has been a growing interest in developing multimodal machine
translation (MMT) systems that enhance neural machine translation (NMT) with
visual knowledge. This problem setup involves using images as auxiliary
information during training, and more recently, eliminating their use during
inference. Towards this end, previous works face a challenge in training
powerful MMT models from scratch due to the scarcity of annotated multilingual... more
There has been a growing interest in developing multimodal machine
translation (MMT) systems that enhance neural machine translation (NMT) with
visual knowledge. This problem setup involves using images as auxiliary
information during training, and more recently, eliminating their use during
inference. Towards this end, previous works face a challenge in training
powerful MMT models from scratch due to the scarcity of annotated multilingual
vision-language data, especially for low-resource languages. Simultaneously,
there has been an influx of multilingual pre-trained models for NMT and
multimodal pre-trained models for vision-language tasks, primarily in English,
which have shown exceptional generalisation ability. However, these are not
directly applicable to MMT since they do not provide aligned multimodal
multilingual features for generative tasks. To alleviate this issue, instead of
designing complex modules for MMT, we propose CLIPTrans, which simply adapts
the independently pre-trained multimodal M-CLIP and the multilingual mBART. In
order to align their embedding spaces, mBART is conditioned on the M-CLIP
features by a prefix sequence generated through a lightweight mapping network.
We train this in a two-stage pipeline which warms up the model with image
captioning before the actual translation task. Through experiments, we
demonstrate the merits of this framework and consequently push forward the
state-of-the-art across standard benchmarks by an average of +2.67 BLEU. The
code can be found at www.github.com/devaansh100/CLIPTrans.
less
By: Qiyu Sun, Huilin Chen, Meng Zheng, Ziyan Wu, Michael Felsberg, Yang Tang
Domain generalized semantic segmentation (DGSS) is a critical yet challenging
task, where the model is trained only on source data without access to any
target data. Despite the proposal of numerous DGSS strategies, the
generalization capability remains limited in CNN architectures. Though some
Transformer-based segmentation models show promising performance, they
primarily focus on capturing intra-sample attentive relationships, disregardi... more
Domain generalized semantic segmentation (DGSS) is a critical yet challenging
task, where the model is trained only on source data without access to any
target data. Despite the proposal of numerous DGSS strategies, the
generalization capability remains limited in CNN architectures. Though some
Transformer-based segmentation models show promising performance, they
primarily focus on capturing intra-sample attentive relationships, disregarding
inter-sample correlations which can potentially benefit DGSS. To this end, we
enhance the attention modules in Transformer networks for improving DGSS by
incorporating information from other independent samples in the same batch,
enriching contextual information, and diversifying the training data for each
attention block. Specifically, we propose two alternative intra-batch attention
mechanisms, namely mean-based intra-batch attention (MIBA) and element-wise
intra-batch attention (EIBA), to capture correlations between different
samples, enhancing feature representation and generalization capabilities.
Building upon intra-batch attention, we introduce IBAFormer, which integrates
self-attention modules with the proposed intra-batch attention for DGSS.
Extensive experiments demonstrate that IBAFormer achieves SOTA performance in
DGSS, and ablation studies further confirm the effectiveness of each introduced
component.
less
By: Svetlana Pavlitska, Nico Lambing, Ashok Kumar Bangaru, J. Marius Zöllner
Real-time traffic light recognition is essential for autonomous driving. Yet,
a cohesive overview of the underlying model architectures for this task is
currently missing. In this work, we conduct a comprehensive survey and analysis
of traffic light recognition methods that use convolutional neural networks
(CNNs). We focus on two essential aspects: datasets and CNN architectures.
Based on an underlying architecture, we cluster methods into... more
Real-time traffic light recognition is essential for autonomous driving. Yet,
a cohesive overview of the underlying model architectures for this task is
currently missing. In this work, we conduct a comprehensive survey and analysis
of traffic light recognition methods that use convolutional neural networks
(CNNs). We focus on two essential aspects: datasets and CNN architectures.
Based on an underlying architecture, we cluster methods into three major
groups: (1) modifications of generic object detectors which compensate for
specific task characteristics, (2) multi-stage approaches involving both
rule-based and CNN components, and (3) task-specific single-stage methods. We
describe the most important works in each cluster, discuss the usage of the
datasets, and identify research gaps.
less
By: Taotao Jing, Lichen Wang, Naji Khosravan, Zhiqiang Wan, Zachary Bessinger, Zhengming Ding, Sing Bing Kang
Room layout estimation predicts layouts from a single panorama. It requires
datasets with large-scale and diverse room shapes to train the models. However,
there are significant imbalances in real-world datasets including the
dimensions of layout complexity, camera locations, and variation in scene
appearance. These issues considerably influence the model training performance.
In this work, we propose the imBalance-Aware Room Layout Estimat... more
Room layout estimation predicts layouts from a single panorama. It requires
datasets with large-scale and diverse room shapes to train the models. However,
there are significant imbalances in real-world datasets including the
dimensions of layout complexity, camera locations, and variation in scene
appearance. These issues considerably influence the model training performance.
In this work, we propose the imBalance-Aware Room Layout Estimation (iBARLE)
framework to address these issues. iBARLE consists of (1) Appearance Variation
Generation (AVG) module, which promotes visual appearance domain
generalization, (2) Complex Structure Mix-up (CSMix) module, which enhances
generalizability w.r.t. room structure, and (3) a gradient-based layout
objective function, which allows more effective accounting for occlusions in
complex layouts. All modules are jointly trained and help each other to achieve
the best performance. Experiments and ablation studies based on
ZInD~\cite{cruz2021zillow} dataset illustrate that iBARLE has state-of-the-art
performance compared with other layout estimation baselines.
less
EVREAL: Towards a Comprehensive Benchmark and Analysis Suite for Event-based Video Reconstruction
0upvotes
By: Burak Ercan, Onur Eker, Aykut Erdem, Erkut Erdem
Event cameras are a new type of vision sensor that incorporates asynchronous
and independent pixels, offering advantages over traditional frame-based
cameras such as high dynamic range and minimal motion blur. However, their
output is not easily understandable by humans, making the reconstruction of
intensity images from event streams a fundamental task in event-based vision.
While recent deep learning-based methods have shown promise in vi... more
Event cameras are a new type of vision sensor that incorporates asynchronous
and independent pixels, offering advantages over traditional frame-based
cameras such as high dynamic range and minimal motion blur. However, their
output is not easily understandable by humans, making the reconstruction of
intensity images from event streams a fundamental task in event-based vision.
While recent deep learning-based methods have shown promise in video
reconstruction from events, this problem is not completely solved yet. To
facilitate comparison between different approaches, standardized evaluation
protocols and diverse test datasets are essential. This paper proposes a
unified evaluation methodology and introduces an open-source framework called
EVREAL to comprehensively benchmark and analyze various event-based video
reconstruction methods from the literature. Using EVREAL, we give a detailed
analysis of the state-of-the-art methods for event-based video reconstruction,
and provide valuable insights into the performance of these methods under
varying settings, challenging scenarios, and downstream tasks.
less
By: Yeonghwan Song, Seokwoo Jang, Dina Katabi, Jeany Son
We propose a novel unsupervised object localization method that allows us to
explain the predictions of the model by utilizing self-supervised pre-trained
models without additional finetuning. Existing unsupervised and self-supervised
object localization methods often utilize class-agnostic activation maps or
self-similarity maps of a pre-trained model. Although these maps can offer
valuable information for localization, their limited abili... more
We propose a novel unsupervised object localization method that allows us to
explain the predictions of the model by utilizing self-supervised pre-trained
models without additional finetuning. Existing unsupervised and self-supervised
object localization methods often utilize class-agnostic activation maps or
self-similarity maps of a pre-trained model. Although these maps can offer
valuable information for localization, their limited ability to explain how the
model makes predictions remains challenging. In this paper, we propose a simple
yet effective unsupervised object localization method based on representer
point selection, where the predictions of the model can be represented as a
linear combination of representer values of training points. By selecting
representer points, which are the most important examples for the model
predictions, our model can provide insights into how the model predicts the
foreground object by providing relevant examples as well as their importance.
Our method outperforms the state-of-the-art unsupervised and self-supervised
object localization methods on various datasets with significant margins and
even outperforms recent weakly supervised and few-shot methods.
less
Integrating Boxes and Masks: A Multi-Object Framework for Unified Visual Tracking and Segmentation
0upvotes
By: Yuanyou Xu, Zongxin Yang, Yi Yang
Tracking any given object(s) spatially and temporally is a common purpose in
Visual Object Tracking (VOT) and Video Object Segmentation (VOS). Joint
tracking and segmentation have been attempted in some studies but they often
lack full compatibility of both box and mask in initialization and prediction,
and mainly focus on single-object scenarios. To address these limitations, this
paper proposes a Multi-object Mask-box Integrated framework... more
Tracking any given object(s) spatially and temporally is a common purpose in
Visual Object Tracking (VOT) and Video Object Segmentation (VOS). Joint
tracking and segmentation have been attempted in some studies but they often
lack full compatibility of both box and mask in initialization and prediction,
and mainly focus on single-object scenarios. To address these limitations, this
paper proposes a Multi-object Mask-box Integrated framework for unified
Tracking and Segmentation, dubbed MITS. Firstly, the unified identification
module is proposed to support both box and mask reference for initialization,
where detailed object information is inferred from boxes or directly retained
from masks. Additionally, a novel pinpoint box predictor is proposed for
accurate multi-object box prediction, facilitating target-oriented
representation learning. All target objects are processed simultaneously from
encoding to propagation and decoding, as a unified pipeline for VOT and VOS.
Experimental results show MITS achieves state-of-the-art performance on both
VOT and VOS benchmarks. Notably, MITS surpasses the best prior VOT competitor
by around 6% on the GOT-10k test set, and significantly improves the
performance of box initialization on VOS benchmarks. The code is available at
https://github.com/yoxu515/MITS.
less
By: Gan Sun, Wenqi Liang, Jiahua Dong, Jun Li, Zhengming Ding, Yang Cong
Text-to-image generative models can produce diverse high-quality images of
concepts with a text prompt, which have demonstrated excellent ability in image
generation, image translation, etc. We in this work study the problem of
synthesizing instantiations of a use's own concepts in a never-ending manner,
i.e., create your world, where the new concepts from user are quickly learned
with a few examples. To achieve this goal, we propose a Life... more
Text-to-image generative models can produce diverse high-quality images of
concepts with a text prompt, which have demonstrated excellent ability in image
generation, image translation, etc. We in this work study the problem of
synthesizing instantiations of a use's own concepts in a never-ending manner,
i.e., create your world, where the new concepts from user are quickly learned
with a few examples. To achieve this goal, we propose a Lifelong text-to-image
Diffusion Model (L2DM), which intends to overcome knowledge "catastrophic
forgetting" for the past encountered concepts, and semantic "catastrophic
neglecting" for one or more concepts in the text prompt. In respect of
knowledge "catastrophic forgetting", our L2DM framework devises a task-aware
memory enhancement module and a elastic-concept distillation module, which
could respectively safeguard the knowledge of both prior concepts and each past
personalized concept. When generating images with a user text prompt, the
solution to semantic "catastrophic neglecting" is that a concept attention
artist module can alleviate the semantic neglecting from concept aspect, and an
orthogonal attention module can reduce the semantic binding from attribute
aspect. To the end, our model can generate more faithful image across a range
of continual text prompts in terms of both qualitative and quantitative
metrics, when comparing with the related state-of-the-art models. The code will
be released at https://wenqiliang.github.io/.
less
By: Masoud Mokhtari, Neda Ahmadi, Teresa S. M. Tsang, Purang Abolmaesumi, Renjie Liao
Echocardiography (echo) is an ultrasound imaging modality that is widely used
for various cardiovascular diagnosis tasks. Due to inter-observer variability
in echo-based diagnosis, which arises from the variability in echo image
acquisition and the interpretation of echo images based on clinical experience,
vision-based machine learning (ML) methods have gained popularity to act as
secondary layers of verification. For such safety-critical ... more
Echocardiography (echo) is an ultrasound imaging modality that is widely used
for various cardiovascular diagnosis tasks. Due to inter-observer variability
in echo-based diagnosis, which arises from the variability in echo image
acquisition and the interpretation of echo images based on clinical experience,
vision-based machine learning (ML) methods have gained popularity to act as
secondary layers of verification. For such safety-critical applications, it is
essential for any proposed ML method to present a level of explainability along
with good accuracy. In addition, such methods must be able to process several
echo videos obtained from various heart views and the interactions among them
to properly produce predictions for a variety of cardiovascular measurements or
interpretation tasks. Prior work lacks explainability or is limited in scope by
focusing on a single cardiovascular task. To remedy this, we propose a General,
Echo-based, Multi-Level Transformer (GEMTrans) framework that provides
explainability, while simultaneously enabling multi-video training where the
inter-play among echo image patches in the same frame, all frames in the same
video, and inter-video relationships are captured based on a downstream task.
We show the flexibility of our framework by considering two critical tasks
including ejection fraction (EF) and aortic stenosis (AS) severity detection.
Our model achieves mean absolute errors of 4.15 and 4.84 for single and
dual-video EF estimation and an accuracy of 96.5 % for AS detection, while
providing informative task-specific attention maps and prototypical
explainability.
less
By: Jan-Aike Termöhlen, Timo Bartels, Tim Fingscheidt
The task of semantic segmentation requires a model to assign semantic labels
to each pixel of an image. However, the performance of such models degrades
when deployed in an unseen domain with different data distributions compared to
the training domain. We present a new augmentation-driven approach to domain
generalization for semantic segmentation using a re-parameterized vision
transformer (ReVT) with weight averaging of multiple models a... more
The task of semantic segmentation requires a model to assign semantic labels
to each pixel of an image. However, the performance of such models degrades
when deployed in an unseen domain with different data distributions compared to
the training domain. We present a new augmentation-driven approach to domain
generalization for semantic segmentation using a re-parameterized vision
transformer (ReVT) with weight averaging of multiple models after training. We
evaluate our approach on several benchmark datasets and achieve
state-of-the-art mIoU performance of 47.3% (prior art: 46.3%) for small models
and of 50.1% (prior art: 47.8%) for midsized models on commonly used benchmark
datasets. At the same time, our method requires fewer parameters and reaches a
higher frame rate than the best prior art. It is also easy to implement and,
unlike network ensembles, does not add any computational complexity during
inference.
less
By: Youngrae Kim, Younggeol Cho, Thanh-Tung Nguyen, Dongman Lee
Real-world vision tasks frequently suffer from the appearance of adverse
weather conditions including rain, fog, snow, and raindrops in captured images.
Recently, several generic methods for restoring weather-degraded images have
been proposed, aiming to remove multiple types of adverse weather effects
present in the images. However, these methods have considered weather as
discrete and mutually exclusive variables, leading to failure in ge... more
Real-world vision tasks frequently suffer from the appearance of adverse
weather conditions including rain, fog, snow, and raindrops in captured images.
Recently, several generic methods for restoring weather-degraded images have
been proposed, aiming to remove multiple types of adverse weather effects
present in the images. However, these methods have considered weather as
discrete and mutually exclusive variables, leading to failure in generalizing
to unforeseen weather conditions beyond the scope of the training data, such as
the co-occurrence of rain, fog, and raindrops. To this end, weather-degraded
image restoration models should have flexible adaptability to the current
unknown weather condition to ensure reliable and optimal performance. The
adaptation method should also be able to cope with data scarcity for real-world
adaptation. This paper proposes MetaWeather, a few-shot weather-degraded image
restoration method for arbitrary weather conditions. For this, we devise the
core piece of MetaWeather, coined Degradation Pattern Matching Module (DPMM),
which leverages representations from a few-shot support set by matching
features between input and sample images under new weather conditions. In
addition, we build meta-knowledge with episodic meta-learning on top of our
MetaWeather architecture to provide flexible adaptability. In the meta-testing
phase, we adopt a parameter-efficient fine-tuning method to preserve the
prebuilt knowledge and avoid the overfitting problem. Experiments on the BID
Task II.A dataset show our method achieves the best performance on PSNR and
SSIM compared to state-of-the-art image restoration methods. Code is available
at (TBA).
less
By: Yicheng Chen, Shengxiang Ji, Yuelin Xin, Kun Han, Xiaohui Xie
Previously, in the field of medical image registration, there are primarily
two paradigms, the traditional optimization-based methods, and the
deep-learning-based methods. Each of these paradigms has its advantages, and in
this work, we aim to take the best of both worlds. Instead of developing a new
deep learning model, we designed a robust training architecture that is simple
and generalizable. We present Optron, a general training archit... more
Previously, in the field of medical image registration, there are primarily
two paradigms, the traditional optimization-based methods, and the
deep-learning-based methods. Each of these paradigms has its advantages, and in
this work, we aim to take the best of both worlds. Instead of developing a new
deep learning model, we designed a robust training architecture that is simple
and generalizable. We present Optron, a general training architecture
incorporating the idea of training-in-the-loop. By iteratively optimizing the
prediction result of a deep learning model through a plug-and-play optimizer
module in the training loop, Optron introduces pseudo ground truth to an
unsupervised training process. And by bringing the training process closer to
that of supervised training, Optron can consistently improve the models'
performance and convergence speed. We evaluated our method on various
combinations of models and datasets, and we have achieved state-of-the-art
performance on the IXI dataset, improving the previous state-of-the-art method
TransMorph by a significant margin of +1.6% DSC. Moreover, Optron also
consistently achieved positive results with other models and datasets. It
increases the validation DSC for VoxelMorph and ViT-V-Net by +2.3% and +2.2%
respectively on IXI, demonstrating our method's generalizability. Our
implementation is publicly available at
https://github.com/miraclefactory/optron
less
By: Dawen Yu, Shunping Ji
Long-range dependency modeling has been widely considered in modern deep
learning based semantic segmentation methods, especially those designed for
large-size remote sensing images, to compensate the intrinsic locality of
standard convolutions. However, in previous studies, the long-range dependency,
modeled with an attention mechanism or transformer model, has been based on
unsupervised learning, instead of explicit supervision from the o... more
Long-range dependency modeling has been widely considered in modern deep
learning based semantic segmentation methods, especially those designed for
large-size remote sensing images, to compensate the intrinsic locality of
standard convolutions. However, in previous studies, the long-range dependency,
modeled with an attention mechanism or transformer model, has been based on
unsupervised learning, instead of explicit supervision from the objective
ground truth. In this paper, we propose a novel supervised long-range
correlation method for land-cover classification, called the supervised
long-range correlation network (SLCNet), which is shown to be superior to the
currently used unsupervised strategies. In SLCNet, pixels sharing the same
category are considered highly correlated and those having different categories
are less relevant, which can be easily supervised by the category consistency
information available in the ground truth semantic segmentation map. Under such
supervision, the recalibrated features are more consistent for pixels of the
same category and more discriminative for pixels of other categories,
regardless of their proximity. To complement the detailed information lacking
in the global long-range correlation, we introduce an auxiliary adaptive
receptive field feature extraction module, parallel to the long-range
correlation module in the encoder, to capture finely detailed feature
representations for multi-size objects in multi-scale remote sensing images. In
addition, we apply multi-scale side-output supervision and a hybrid loss
function as local and global constraints to further boost the segmentation
accuracy. Experiments were conducted on three remote sensing datasets. Compared
with the advanced segmentation methods from the computer vision, medicine, and
remote sensing communities, the SLCNet achieved a state-of-the-art performance
on all the datasets.
less
By: Yue Xu, Yong-Lu Li, Zhemin Huang, Michael Xu Liu, Cewu Lu, Yu-Wing Tai, Chi-Keung Tang
With the surge in attention to Egocentric Hand-Object Interaction (Ego-HOI),
large-scale datasets such as Ego4D and EPIC-KITCHENS have been proposed.
However, most current research is built on resources derived from third-person
video action recognition. This inherent domain gap between first- and
third-person action videos, which have not been adequately addressed before,
makes current Ego-HOI suboptimal. This paper rethinks and proposes a... more
With the surge in attention to Egocentric Hand-Object Interaction (Ego-HOI),
large-scale datasets such as Ego4D and EPIC-KITCHENS have been proposed.
However, most current research is built on resources derived from third-person
video action recognition. This inherent domain gap between first- and
third-person action videos, which have not been adequately addressed before,
makes current Ego-HOI suboptimal. This paper rethinks and proposes a new
framework as an infrastructure to advance Ego-HOI recognition by Probing,
Curation and Adaption (EgoPCA). We contribute comprehensive pre-train sets,
balanced test sets and a new baseline, which are complete with a
training-finetuning strategy. With our new framework, we not only achieve
state-of-the-art performance on Ego-HOI benchmarks but also build several new
and effective mechanisms and settings to advance further research. We believe
our data and the findings will pave a new way for Ego-HOI understanding. Code
and data are available at https://mvig-rhos.com/ego_pca
less
By: Paolo Cudrano, Matteo Bellusci, Giuseppe Macino, Matteo Matteucci
Accurate road surface classification is crucial for autonomous vehicles (AVs)
to optimize driving conditions, enhance safety, and enable advanced road
mapping. However, deep learning models for road surface classification suffer
from poor generalization when tested on unseen datasets. To update these models
with new information, also the original training dataset must be taken into
account, in order to avoid catastrophic forgetting. This is... more
Accurate road surface classification is crucial for autonomous vehicles (AVs)
to optimize driving conditions, enhance safety, and enable advanced road
mapping. However, deep learning models for road surface classification suffer
from poor generalization when tested on unseen datasets. To update these models
with new information, also the original training dataset must be taken into
account, in order to avoid catastrophic forgetting. This is, however,
inefficient if not impossible, e.g., when the data is collected in streams or
large amounts. To overcome this limitation and enable fast and efficient
cross-dataset adaptation, we propose to employ continual learning finetuning
methods designed to retain past knowledge while adapting to new data, thus
effectively avoiding forgetting. Experimental results demonstrate the
superiority of this approach over naive finetuning, achieving performance close
to fresh retraining. While solving this known problem, we also provide a
general description of how the same technique can be adopted in other AV
scenarios. We highlight the potential computational and economic benefits that
a continual-based adaptation can bring to the AV industry, while also reducing
greenhouse emissions due to unnecessary joint retraining.
less
By: JiaJun Zhu, Zichuan Yang, Binjie Hong, Jiacheng Song, Jiwei Wang, Tianhao Chen, Shuilan Yang, Zixun Lan, Fei Ma
Correcting students' multiple-choice answers is a repetitive and mechanical
task that can be considered an image multi-classification task. Assuming
possible options are 'abcd' and the correct option is one of the four, some
students may write incorrect symbols or options that do not exist. In this
paper, five classifications were set up - four for possible correct options and
one for other incorrect writing. This approach takes into accoun... more
Correcting students' multiple-choice answers is a repetitive and mechanical
task that can be considered an image multi-classification task. Assuming
possible options are 'abcd' and the correct option is one of the four, some
students may write incorrect symbols or options that do not exist. In this
paper, five classifications were set up - four for possible correct options and
one for other incorrect writing. This approach takes into account the
possibility of non-standard writing options.
less
By: Urs Waldmann, Alex Hoi Hang Chan, Hemal Naik, Máté Nagy, Iain D. Couzin, Oliver Deussen, Bastian Goldluecke, Fumihiro Kano
Markerless methods for animal posture tracking have been developing recently,
but frameworks and benchmarks for tracking large animal groups in 3D are still
lacking. To overcome this gap in the literature, we present 3D-MuPPET, a
framework to estimate and track 3D poses of up to 10 pigeons at interactive
speed using multiple-views. We train a pose estimator to infer 2D keypoints and
bounding boxes of multiple pigeons, then triangulate the k... more
Markerless methods for animal posture tracking have been developing recently,
but frameworks and benchmarks for tracking large animal groups in 3D are still
lacking. To overcome this gap in the literature, we present 3D-MuPPET, a
framework to estimate and track 3D poses of up to 10 pigeons at interactive
speed using multiple-views. We train a pose estimator to infer 2D keypoints and
bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For
correspondence matching, we first dynamically match 2D detections to global
identities in the first frame, then use a 2D tracker to maintain
correspondences accross views in subsequent frames. We achieve comparable
accuracy to a state of the art 3D pose estimator for Root Mean Square Error
(RMSE) and Percentage of Correct Keypoints (PCK). We also showcase a novel use
case where our model trained with data of single pigeons provides comparable
results on data containing multiple pigeons. This can simplify the domain shift
to new species because annotating single animal data is less labour intensive
than multi-animal data. Additionally, we benchmark the inference speed of
3D-MuPPET, with up to 10 fps in 2D and 1.5 fps in 3D, and perform quantitative
tracking evaluation, which yields encouraging results. Finally, we show that
3D-MuPPET also works in natural environments without model fine-tuning on
additional annotations. To the best of our knowledge we are the first to
present a framework for 2D/3D posture and trajectory tracking that works in
both indoor and outdoor environments.
less
By: Tetsuya Abe, Ryusuke Sagawa, Ko Ayusawa, Wataru Takano
The present paper proposes an encoder-decoder model for extracting the
structures of human motions represented by frame-wise discrete features in a
self-supervised manner. In the proposed method, features are extracted as codes
in a motion codebook without the use of human knowledge, and the relationship
between these codes can be visualized on a graph. Since the codes are expected
to be temporally sparse compared to the captured frame rate... more
The present paper proposes an encoder-decoder model for extracting the
structures of human motions represented by frame-wise discrete features in a
self-supervised manner. In the proposed method, features are extracted as codes
in a motion codebook without the use of human knowledge, and the relationship
between these codes can be visualized on a graph. Since the codes are expected
to be temporally sparse compared to the captured frame rate and can be shared
by multiple sequences, the proposed network model also addresses the need for
training constraints. Specifically, the model consists of self-attention layers
and a vector clustering block. The attention layers contribute to finding
sparse keyframes and discrete features as motion codes, which are then
extracted by vector clustering. The constraints are realized as training losses
so that the same motion codes can be as contiguous as possible and can be
shared by multiple sequences. In addition, we propose the use of causal
self-attention as a method by which to calculate attention for long sequences
consisting of numerous frames. In our experiments, the sparse structures of
motion codes were used to compile a graph that facilitates visualization of the
relationship between the codes and the differences between sequences. We then
evaluated the effectiveness of the extracted motion codes by applying them to
multiple recognition tasks and found that performance levels comparable to
task-optimized methods could be achieved by linear probing.
less
By: Jingbo Zeng
Technologies of human action recognition in the dark are gaining more and
more attention as huge demand in surveillance, motion control and
human-computer interaction. However, because of limitation in image enhancement
method and low-lighting video datasets, e.g. labeling cost, existing methods
meet some problems. Some video-based approached are effect and efficient in
specific datasets but cannot generalize to most cases while others meth... more
Technologies of human action recognition in the dark are gaining more and
more attention as huge demand in surveillance, motion control and
human-computer interaction. However, because of limitation in image enhancement
method and low-lighting video datasets, e.g. labeling cost, existing methods
meet some problems. Some video-based approached are effect and efficient in
specific datasets but cannot generalize to most cases while others methods
using multiple sensors rely heavily to prior knowledge to deal with noisy
nature from video stream. In this paper, we proposes action recognition method
using deep multi-input network. Furthermore, we proposed a Independent Gamma
Intensity Corretion (Ind-GIC) to enhance poor-illumination video, generating
one gamma for one frame to increase enhancement performance. To prove our
method is effective, there is some evaluation and comparison between our method
and existing methods. Experimental results show that our model achieves high
accuracy in on ARID dataset.
less
By: Haoke Xiao, Lv Tang, Bo Li, Zhiming Luo, Shaozi Li
Co-salient Object Detection (CoSOD) endeavors to replicate the human visual
system's capacity to recognize common and salient objects within a collection
of images. Despite recent advancements in deep learning models, these models
still rely on training with well-annotated CoSOD datasets. The exploration of
training-free zero-shot CoSOD frameworks has been limited. In this paper,
taking inspiration from the zero-shot transfer capabilities o... more
Co-salient Object Detection (CoSOD) endeavors to replicate the human visual
system's capacity to recognize common and salient objects within a collection
of images. Despite recent advancements in deep learning models, these models
still rely on training with well-annotated CoSOD datasets. The exploration of
training-free zero-shot CoSOD frameworks has been limited. In this paper,
taking inspiration from the zero-shot transfer capabilities of foundational
computer vision models, we introduce the first zero-shot CoSOD framework that
harnesses these models without any training process. To achieve this, we
introduce two novel components in our proposed framework: the group prompt
generation (GPG) module and the co-saliency map generation (CMP) module. We
evaluate the framework's performance on widely-used datasets and observe
impressive results. Our approach surpasses existing unsupervised methods and
even outperforms fully supervised methods developed before 2020, while
remaining competitive with some fully supervised methods developed before 2022.
less
By: Fu-En Yang, Chien-Yi Wang, Yu-Chiang Frank Wang
Federated learning (FL) emerges as a decentralized learning framework which
trains models from multiple distributed clients without sharing their data to
preserve privacy. Recently, large-scale pre-trained models (e.g., Vision
Transformer) have shown a strong capability of deriving robust representations.
However, the data heterogeneity among clients, the limited computation
resources, and the communication bandwidth restrict the deployment... more
Federated learning (FL) emerges as a decentralized learning framework which
trains models from multiple distributed clients without sharing their data to
preserve privacy. Recently, large-scale pre-trained models (e.g., Vision
Transformer) have shown a strong capability of deriving robust representations.
However, the data heterogeneity among clients, the limited computation
resources, and the communication bandwidth restrict the deployment of
large-scale models in FL frameworks. To leverage robust representations from
large-scale models while enabling efficient model personalization for
heterogeneous clients, we propose a novel personalized FL framework of
client-specific Prompt Generation (pFedPG), which learns to deploy a
personalized prompt generator at the server for producing client-specific
visual prompts that efficiently adapts frozen backbones to local data
distributions. Our proposed framework jointly optimizes the stages of
personalized prompt adaptation locally and personalized prompt generation
globally. The former aims to train visual prompts that adapt foundation models
to each client, while the latter observes local optimization directions to
generate personalized prompts for all clients. Through extensive experiments on
benchmark datasets, we show that our pFedPG is favorable against
state-of-the-art personalized FL methods under various types of data
heterogeneity, allowing computation and communication efficient model
personalization.
less
By: Yubiao Yue, Xiaoqiang Shi, Li Qin, Xinyue Zhang, Yanmei Chen, Jialong Xu, Zipei Zheng, Yujun Cao, Di Liu, Zhenzhang Li, Yang Li
Due to the lack of more efficient diagnostic tools for monkeypox, its spread
remains unchecked, presenting a formidable challenge to global health. While
the high efficacy of deep learning models for monkeypox diagnosis has been
demonstrated in related studies, the overlook of inference speed, the parameter
size and diagnosis performance for early-stage monkeypox renders the models
inapplicable in real-world settings. To address these chall... more
Due to the lack of more efficient diagnostic tools for monkeypox, its spread
remains unchecked, presenting a formidable challenge to global health. While
the high efficacy of deep learning models for monkeypox diagnosis has been
demonstrated in related studies, the overlook of inference speed, the parameter
size and diagnosis performance for early-stage monkeypox renders the models
inapplicable in real-world settings. To address these challenges, we proposed
an ultrafast and ultralight network named Fast-MpoxNet. Fast-MpoxNet possesses
only 0.27M parameters and can process input images at 68 frames per second
(FPS) on the CPU. To counteract the diagnostic performance limitation brought
about by the small model capacity, it integrates the attention-based feature
fusion module and the multiple auxiliary losses enhancement strategy for better
detecting subtle image changes and optimizing weights. Using transfer learning
and five-fold cross-validation, Fast-MpoxNet achieves 94.26% Accuracy on the
Mpox dataset. Notably, its recall for early-stage monkeypox achieves 93.65%. By
adopting data augmentation, our model's Accuracy rises to 98.40% and attains a
Practicality Score (A new metric for measuring model practicality in real-time
diagnosis application) of 0.80. We also developed an application system named
Mpox-AISM V2 for both personal computers and mobile phones. Mpox-AISM V2
features ultrafast responses, offline functionality, and easy deployment,
enabling accurate and real-time diagnosis for both the public and individuals
in various real-world settings, especially in populous settings during the
outbreak. Our work could potentially mitigate future monkeypox outbreak and
illuminate a fresh paradigm for developing real-time diagnostic tools in the
healthcare field.
less
By: Brad Larson, Bishal Upadhyaya, Luke McDermott, Siddha Ganju
Deep learning harnesses massive parallel floating-point processing to train
and evaluate large neural networks. Trends indicate that deeper and larger
neural networks with an increasing number of parameters achieve higher accuracy
than smaller neural networks. This performance improvement, which often
requires heavy compute for both training and evaluation, eventually needs to
translate well to resource-constrained hardware for practical va... more
Deep learning harnesses massive parallel floating-point processing to train
and evaluate large neural networks. Trends indicate that deeper and larger
neural networks with an increasing number of parameters achieve higher accuracy
than smaller neural networks. This performance improvement, which often
requires heavy compute for both training and evaluation, eventually needs to
translate well to resource-constrained hardware for practical value. Structured
pruning asserts that while large networks enable us to find solutions to
complex computer vision problems, a smaller, computationally efficient
sub-network can be derived from the large neural network that retains model
accuracy but significantly improves computational efficiency.
We generalize structured pruning with algorithms for network augmentation,
pruning, sub-network collapse and removal. In addition, we demonstrate
efficient and stable convergence up to 93% sparsity and 95% FLOPs reduction
without loss of inference accuracy using with continuous relaxation matching or
exceeding the state of the art for all structured pruning methods. The
resulting CNN executes efficiently on GPU hardware without computationally
expensive sparse matrix operations. We achieve this with routine automatable
operations on classification and segmentation problems using CIFAR-10,
ImageNet, and CityScapes datasets with the ResNet and U-NET network
architectures.
less
Neural Progressive Meshes
0upvotes
By: Yun-Chun Chen, Vladimir G. Kim, Noam Aigerman, Alec Jacobson
The recent proliferation of 3D content that can be consumed on hand-held
devices necessitates efficient tools for transmitting large geometric data,
e.g., 3D meshes, over the Internet. Detailed high-resolution assets can pose a
challenge to storage as well as transmission bandwidth, and level-of-detail
techniques are often used to transmit an asset using an appropriate bandwidth
budget. It is especially desirable for these methods to transm... more
The recent proliferation of 3D content that can be consumed on hand-held
devices necessitates efficient tools for transmitting large geometric data,
e.g., 3D meshes, over the Internet. Detailed high-resolution assets can pose a
challenge to storage as well as transmission bandwidth, and level-of-detail
techniques are often used to transmit an asset using an appropriate bandwidth
budget. It is especially desirable for these methods to transmit data
progressively, improving the quality of the geometry with more data. Our key
insight is that the geometric details of 3D meshes often exhibit similar local
patterns even across different shapes, and thus can be effectively represented
with a shared learned generative space. We learn this space using a
subdivision-based encoder-decoder architecture trained in advance on a large
collection of surfaces. We further observe that additional residual features
can be transmitted progressively between intermediate levels of subdivision
that enable the client to control the tradeoff between bandwidth cost and
quality of reconstruction, providing a neural progressive mesh representation.
We evaluate our method on a diverse set of complex 3D shapes and demonstrate
that it outperforms baselines in terms of compression ratio and reconstruction
quality.
less
By: Shuning Xu, Binbin Song, Xiangyu Chen, Jiantao Zhou
Moire patterns occur when capturing images or videos on screens, severely
degrading the quality of the captured images or videos. Despite the recent
progresses, existing video demoireing methods neglect the physical
characteristics and formation process of moire patterns, significantly limiting
the effectiveness of video recovery. This paper presents a unified framework,
DTNet, a direction-aware and temporal-guided bilateral learning networ... more
Moire patterns occur when capturing images or videos on screens, severely
degrading the quality of the captured images or videos. Despite the recent
progresses, existing video demoireing methods neglect the physical
characteristics and formation process of moire patterns, significantly limiting
the effectiveness of video recovery. This paper presents a unified framework,
DTNet, a direction-aware and temporal-guided bilateral learning network for
video demoireing. DTNet effectively incorporates the process of moire pattern
removal, alignment, color correction, and detail refinement. Our proposed DTNet
comprises two primary stages: Frame-level Direction-aware Demoireing and
Alignment (FDDA) and Tone and Detail Refinement (TDR). In FDDA, we employ
multiple directional DCT modes to perform the moire pattern removal process in
the frequency domain, effectively detecting the prominent moire edges. Then,
the coarse and fine-grained alignment is applied on the demoired features for
facilitating the utilization of neighboring information. In TDR, we propose a
temporal-guided bilateral learning pipeline to mitigate the degradation of
color and details caused by the moire patterns while preserving the restored
frequency information in FDDA. Guided by the aligned temporal features from
FDDA, the affine transformations for the recovery of the ultimate clean frames
are learned in TDR. Extensive experiments demonstrate that our video demoireing
method outperforms state-of-the-art approaches by 2.3 dB in PSNR, and also
delivers a superior visual experience.
less
A Consumer-tier based Visual-Brain Machine Interface for Augmented Reality Glasses Interactions
0upvotes
By: Yuying Jiang, Fan Bai, Zicheng Zhang, Xiaochen Ye, Zheng Liu, Zhiping Shi, Jianwei Yao, Xiaojun Liu, Fangkun Zhu, Junling Li Qian Guo, Xiaoan Wang, Junwen Luo
Objective.Visual-Brain Machine Interface(V-BMI) has provide a novel
interaction technique for Augmented Reality (AR) industries. Several
state-of-arts work has demonstates its high accuracy and real-time interaction
capbilities. However, most of the studies employ EEGs devices that are rigid
and difficult to apply in real-life AR glasseses application sceniraros. Here
we develop a consumer-tier Visual-Brain Machine Inteface(V-BMI) system
sp... more
Objective.Visual-Brain Machine Interface(V-BMI) has provide a novel
interaction technique for Augmented Reality (AR) industries. Several
state-of-arts work has demonstates its high accuracy and real-time interaction
capbilities. However, most of the studies employ EEGs devices that are rigid
and difficult to apply in real-life AR glasseses application sceniraros. Here
we develop a consumer-tier Visual-Brain Machine Inteface(V-BMI) system
specialized for Augmented Reality(AR) glasses interactions. Approach. The
developed system consists of a wearable hardware which takes advantages of fast
set-up, reliable recording and comfortable wearable experience that
specificized for AR glasses applications. Complementing this hardware, we have
devised a software framework that facilitates real-time interactions within the
system while accommodating a modular configuration to enhance scalability. Main
results. The developed hardware is only 110g and 120x85x23 mm, which with 1
Tohm and peak to peak voltage is less than 1.5 uV, and a V-BMI based angry bird
game and an Internet of Thing (IoT) AR applications are deisgned, we
demonstrated such technology merits of intuitive experience and efficiency
interaction. The real-time interaction accuracy is between 85 and 96
percentages in a commercial AR glasses (DTI is 2.24s and ITR 65 bits-min ).
Significance. Our study indicates the developed system can provide an essential
hardware-software framework for consumer based V-BMI AR glasses. Also, we
derive several pivotal design factors for a consumer-grade V-BMI-based AR
system: 1) Dynamic adaptation of stimulation patterns-classification methods
via computer vision algorithms is necessary for AR glasses applications; and 2)
Algorithmic localization to foster system stability and latency reduction.
less
AI4Food-NutritionFW: A Novel Framework for the Automatic Synthesis and Analysis of Eating Behaviours
0upvotes
By: Sergio Romero-Tapiador, Ruben Tolosana, Aythami Morales, Isabel Espinosa-Salinas, Gala Freixer, Julian Fierrez, Ruben Vera-Rodriguez, Enrique Carrillo de Santa Pau, Ana Ramírez de Molina, Javier Ortega-Garcia
Nowadays millions of images are shared on social media and web platforms. In
particular, many of them are food images taken from a smartphone over time,
providing information related to the individual's diet. On the other hand,
eating behaviours are directly related to some of the most prevalent diseases
in the world. Exploiting recent advances in image processing and Artificial
Intelligence (AI), this scenario represents an excellent oppor... more
Nowadays millions of images are shared on social media and web platforms. In
particular, many of them are food images taken from a smartphone over time,
providing information related to the individual's diet. On the other hand,
eating behaviours are directly related to some of the most prevalent diseases
in the world. Exploiting recent advances in image processing and Artificial
Intelligence (AI), this scenario represents an excellent opportunity to: i)
create new methods that analyse the individuals' health from what they eat, and
ii) develop personalised recommendations to improve nutrition and diet under
specific circumstances (e.g., obesity or COVID). Having tunable tools for
creating food image datasets that facilitate research in both lines is very
much needed.
This paper proposes AI4Food-NutritionFW, a framework for the creation of food
image datasets according to configurable eating behaviours. AI4Food-NutritionFW
simulates a user-friendly and widespread scenario where images are taken using
a smartphone. In addition to the framework, we also provide and describe a
unique food image dataset that includes 4,800 different weekly eating
behaviours from 15 different profiles and 1,200 subjects. Specifically, we
consider profiles that comply with actual lifestyles from healthy eating
behaviours (according to established knowledge), variable profiles (e.g.,
eating out, holidays), to unhealthy ones (e.g., excess of fast food or sweets).
Finally, we automatically evaluate a healthy index of the subject's eating
behaviours using multidimensional metrics based on guidelines for healthy diets
proposed by international organisations, achieving promising results (99.53%
and 99.60% accuracy and sensitivity, respectively). We also release to the
research community a software implementation of our proposed
AI4Food-NutritionFW and the mentioned food image dataset created with it.
less
Semi-Supervised Semantic Depth Estimation using Symbiotic Transformer and NearFarMix Augmentation
0upvotes
By: Md Awsafur Rahman, Shaikh Anowarul Fattah
In computer vision, depth estimation is crucial for domains like robotics,
autonomous vehicles, augmented reality, and virtual reality. Integrating
semantics with depth enhances scene understanding through reciprocal
information sharing. However, the scarcity of semantic information in datasets
poses challenges. Existing convolutional approaches with limited local
receptive fields hinder the full utilization of the symbiotic potential betwe... more
In computer vision, depth estimation is crucial for domains like robotics,
autonomous vehicles, augmented reality, and virtual reality. Integrating
semantics with depth enhances scene understanding through reciprocal
information sharing. However, the scarcity of semantic information in datasets
poses challenges. Existing convolutional approaches with limited local
receptive fields hinder the full utilization of the symbiotic potential between
depth and semantics. This paper introduces a dataset-invariant semi-supervised
strategy to address the scarcity of semantic information. It proposes the Depth
Semantics Symbiosis module, leveraging the Symbiotic Transformer for achieving
comprehensive mutual awareness by information exchange within both local and
global contexts. Additionally, a novel augmentation, NearFarMix is introduced
to combat overfitting and compensate both depth-semantic tasks by strategically
merging regions from two images, generating diverse and structurally consistent
samples with enhanced control. Extensive experiments on NYU-Depth-V2 and KITTI
datasets demonstrate the superiority of our proposed techniques in indoor and
outdoor environments.
less
Face Presentation Attack Detection by Excavating Causal Clues and Adapting Embedding Statistics
0upvotes
By: Meiling Fang, Naser Damer
Recent face presentation attack detection (PAD) leverages domain adaptation
(DA) and domain generalization (DG) techniques to address performance
degradation on unknown domains. However, DA-based PAD methods require access to
unlabeled target data, while most DG-based PAD solutions rely on a priori,
i.e., known domain labels. Moreover, most DA-/DG-based methods are
computationally intensive, demanding complex model architectures and/or
mult... more
Recent face presentation attack detection (PAD) leverages domain adaptation
(DA) and domain generalization (DG) techniques to address performance
degradation on unknown domains. However, DA-based PAD methods require access to
unlabeled target data, while most DG-based PAD solutions rely on a priori,
i.e., known domain labels. Moreover, most DA-/DG-based methods are
computationally intensive, demanding complex model architectures and/or
multi-stage training processes. This paper proposes to model face PAD as a
compound DG task from a causal perspective, linking it to model optimization.
We excavate the causal factors hidden in the high-level representation via
counterfactual intervention. Moreover, we introduce a class-guided MixStyle to
enrich feature-level data distribution within classes instead of focusing on
domain information. Both class-guided MixStyle and counterfactual intervention
components introduce no extra trainable parameters and negligible computational
resources. Extensive cross-dataset and analytic experiments demonstrate the
effectiveness and efficiency of our method compared to state-of-the-art PADs.
The implementation and the trained weights are publicly available.
less
Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples
0upvotes
By: Guanghui Li, Mingqi Gao, Heng Liu, Xiantong Zhen, Feng Zheng
Referring video object segmentation (RVOS), as a supervised learning task,
relies on sufficient annotated data for a given scene. However, in more
realistic scenarios, only minimal annotations are available for a new scene,
which poses significant challenges to existing RVOS methods. With this in mind,
we propose a simple yet effective model with a newly designed cross-modal
affinity (CMA) module based on a Transformer architecture. The CMA... more
Referring video object segmentation (RVOS), as a supervised learning task,
relies on sufficient annotated data for a given scene. However, in more
realistic scenarios, only minimal annotations are available for a new scene,
which poses significant challenges to existing RVOS methods. With this in mind,
we propose a simple yet effective model with a newly designed cross-modal
affinity (CMA) module based on a Transformer architecture. The CMA module
builds multimodal affinity with a few samples, thus quickly learning new
semantic information, and enabling the model to adapt to different scenarios.
Since the proposed method targets limited samples for new scenes, we generalize
the problem as - few-shot referring video object segmentation (FS-RVOS). To
foster research in this direction, we build up a new FS-RVOS benchmark based on
currently available datasets. The benchmark covers a wide range and includes
multiple situations, which can maximally simulate real-world scenarios.
Extensive experiments show that our model adapts well to different scenarios
with only a few samples, reaching state-of-the-art performance on the
benchmark. On Mini-Ref-YouTube-VOS, our model achieves an average performance
of 53.1 J and 54.8 F, which are 10% better than the baselines. Furthermore, we
show impressive results of 77.7 J and 74.8 F on Mini-Ref-SAIL-VOS, which are
significantly better than the baselines. Code is publicly available at
https://github.com/hengliusky/Few_shot_RVOS.
less
By: Mohammed Guermal, Francois Bremond, Rui Dai, Abid Ali
Action anticipation involves forecasting future actions by connecting past
events to future ones. However, this reasoning ignores the real-life hierarchy
of events which is considered to be composed of three main parts: past,
present, and future. We argue that considering these three main parts and their
dependencies could improve performance. On the other hand, online action
detection is the task of predicting actions in a streaming manner... more
Action anticipation involves forecasting future actions by connecting past
events to future ones. However, this reasoning ignores the real-life hierarchy
of events which is considered to be composed of three main parts: past,
present, and future. We argue that considering these three main parts and their
dependencies could improve performance. On the other hand, online action
detection is the task of predicting actions in a streaming manner. In this
case, one has access only to the past and present information. Therefore, in
online action detection (OAD) the existing approaches miss semantics or future
information which limits their performance. To sum up, for both of these tasks,
the complete set of knowledge (past-present-future) is missing, which makes it
challenging to infer action dependencies, therefore having low performances. To
address this limitation, we propose to fuse both tasks into a single uniform
architecture. By combining action anticipation and online action detection, our
approach can cover the missing dependencies of future information in online
action detection. This method referred to as JOADAA, presents a uniform model
that jointly performs action anticipation and online action detection. We
validate our proposed model on three challenging datasets: THUMOS'14, which is
a sparsely annotated dataset with one action per time step, CHARADES, and
Multi-THUMOS, two densely annotated datasets with more complex scenarios.
JOADAA achieves SOTA results on these benchmarks for both tasks.
less
By: George Retsinas, Giorgos Sfikas, Christophoros Nikou
Recent advances in segmentation-free keyword spotting treat this problem
w.r.t. an object detection paradigm and borrow from state-of-the-art detection
systems to simultaneously propose a word bounding box proposal mechanism and
compute a corresponding representation. Contrary to the norm of such methods
that rely on complex and large DNN models, we propose a novel segmentation-free
system that efficiently scans a document image to find rec... more
Recent advances in segmentation-free keyword spotting treat this problem
w.r.t. an object detection paradigm and borrow from state-of-the-art detection
systems to simultaneously propose a word bounding box proposal mechanism and
compute a corresponding representation. Contrary to the norm of such methods
that rely on complex and large DNN models, we propose a novel segmentation-free
system that efficiently scans a document image to find rectangular areas that
include the query information. The underlying model is simple and compact,
predicting character occurrences over rectangular areas through an implicitly
learned scale map, trained on word-level annotated images. The proposed
document scanning is then performed using this character counting in a
cost-effective manner via integral images and binary search. Finally, the
retrieval similarity by character counting is refined by a pyramidal
representation and a CTC-based re-scoring algorithm, fully utilizing the
trained CNN model. Experimental validation on two widely-used datasets shows
that our method achieves state-of-the-art results outperforming the more
complex alternatives, despite the simplicity of the underlying model.
less
By: Ziang Cheng, Jiayu Yang, Hongdong Li
Real-time Stereo Matching is a cornerstone algorithm for many Extended
Reality (XR) applications, such as indoor 3D understanding, video pass-through,
and mixed-reality games. Despite significant advancements in deep stereo
methods, achieving real-time depth inference with high accuracy on a low-power
device remains a major challenge. One of the major difficulties is the lack of
high-quality indoor video stereo training datasets captured by... more
Real-time Stereo Matching is a cornerstone algorithm for many Extended
Reality (XR) applications, such as indoor 3D understanding, video pass-through,
and mixed-reality games. Despite significant advancements in deep stereo
methods, achieving real-time depth inference with high accuracy on a low-power
device remains a major challenge. One of the major difficulties is the lack of
high-quality indoor video stereo training datasets captured by head-mounted
VR/AR glasses. To address this issue, we introduce a novel video stereo
synthetic dataset that comprises photorealistic renderings of various indoor
scenes and realistic camera motion captured by a 6-DoF moving VR/AR
head-mounted display (HMD). This facilitates the evaluation of existing
approaches and promotes further research on indoor augmented reality scenarios.
Our newly proposed dataset enables us to develop a novel framework for
continuous video-rate stereo matching.
As another contribution, our dataset enables us to proposed a new video-based
stereo matching approach tailored for XR applications, which achieves real-time
inference at an impressive 134fps on a standard desktop computer, or 30fps on a
battery-powered HMD. Our key insight is that disparity and contextual
information are highly correlated and redundant between consecutive stereo
frames. By unrolling an iterative cost aggregation in time (i.e. in the
temporal dimension), we are able to distribute and reuse the aggregated
features over time. This approach leads to a substantial reduction in
computation without sacrificing accuracy. We conducted extensive evaluations
and comparisons and demonstrated that our method achieves superior performance
compared to the current state-of-the-art, making it a strong contender for
real-time stereo matching in VR/AR applications.
less
By: Prateek Chhikara, Dhiraj Chaurasia, Yifan Jiang, Omkar Masur, Filip Ilievski
Food computing has emerged as a prominent multidisciplinary field of research
in recent years. An ambitious goal of food computing is to develop end-to-end
intelligent systems capable of autonomously producing recipe information for a
food image. Current image-to-recipe methods are retrieval-based and their
success depends heavily on the dataset size and diversity, as well as the
quality of learned embeddings. Meanwhile, the emergence of po... more
Food computing has emerged as a prominent multidisciplinary field of research
in recent years. An ambitious goal of food computing is to develop end-to-end
intelligent systems capable of autonomously producing recipe information for a
food image. Current image-to-recipe methods are retrieval-based and their
success depends heavily on the dataset size and diversity, as well as the
quality of learned embeddings. Meanwhile, the emergence of powerful
attention-based vision and language models presents a promising avenue for
accurate and generalizable recipe generation, which has yet to be extensively
explored. This paper proposes FIRE, a novel multimodal methodology tailored to
recipe generation in the food computing domain, which generates the food title,
ingredients, and cooking instructions based on input food images. FIRE
leverages the BLIP model to generate titles, utilizes a Vision Transformer with
a decoder for ingredient extraction, and employs the T5 model to generate
recipes incorporating titles and ingredients as inputs. We showcase two
practical applications that can benefit from integrating FIRE with large
language model prompting: recipe customization to fit recipes to user
preferences and recipe-to-code transformation to enable automated cooking
processes. Our experimental findings validate the efficacy of our proposed
approach, underscoring its potential for future advancements and widespread
adoption in food computing.
less
By: Ashish Singh, Antonio Bevilacqua, Timilehin B. Aderinola, Thach Le Nguyen, Darragh Whelan, Martin O'Reilly, Brian Caulfield, Georgiana Ifrim
Wearable sensors such as Inertial Measurement Units (IMUs) are often used to
assess the performance of human exercise. Common approaches use handcrafted
features based on domain expertise or automatically extracted features using
time series analysis. Multiple sensors are required to achieve high
classification accuracy, which is not very practical. These sensors require
calibration and synchronization and may lead to discomfort over longer... more
Wearable sensors such as Inertial Measurement Units (IMUs) are often used to
assess the performance of human exercise. Common approaches use handcrafted
features based on domain expertise or automatically extracted features using
time series analysis. Multiple sensors are required to achieve high
classification accuracy, which is not very practical. These sensors require
calibration and synchronization and may lead to discomfort over longer time
periods. Recent work utilizing computer vision techniques has shown similar
performance using video, without the need for manual feature engineering, and
avoiding some pitfalls such as sensor calibration and placement on the body. In
this paper, we compare the performance of IMUs to a video-based approach for
human exercise classification on two real-world datasets consisting of Military
Press and Rowing exercises. We compare the performance using a single camera
that captures video in the frontal view versus using 5 IMUs placed on different
parts of the body. We observe that an approach based on a single camera can
outperform a single IMU by 10 percentage points on average. Additionally, a
minimum of 3 IMUs are required to outperform a single camera. We observe that
working with the raw data using multivariate time series classifiers
outperforms traditional approaches based on handcrafted or automatically
extracted features. Finally, we show that an ensemble model combining the data
from a single camera with a single IMU outperforms either data modality. Our
work opens up new and more realistic avenues for this application, where a
video captured using a readily available smartphone camera, combined with a
single sensor, can be used for effective human exercise classification.
less
By: Xiao He, Mingrui Zhu, Dongxin Chen, Nannan Wang, Xinbo Gao
Privacy protection has become a top priority as the proliferation of AI
techniques has led to widespread collection and misuse of personal data.
Anonymization and visual identity information hiding are two important facial
privacy protection tasks that aim to remove identification characteristics from
facial images at the human perception level. However, they have a significant
difference in that the former aims to prevent the machine from ... more
Privacy protection has become a top priority as the proliferation of AI
techniques has led to widespread collection and misuse of personal data.
Anonymization and visual identity information hiding are two important facial
privacy protection tasks that aim to remove identification characteristics from
facial images at the human perception level. However, they have a significant
difference in that the former aims to prevent the machine from recognizing
correctly, while the latter needs to ensure the accuracy of machine
recognition. Therefore, it is difficult to train a model to complete these two
tasks simultaneously. In this paper, we unify the task of anonymization and
visual identity information hiding and propose a novel face privacy protection
method based on diffusion models, dubbed Diff-Privacy. Specifically, we train
our proposed multi-scale image inversion module (MSI) to obtain a set of SDM
format conditional embeddings of the original image. Based on the conditional
embeddings, we design corresponding embedding scheduling strategies and
construct different energy functions during the denoising process to achieve
anonymization and visual identity information hiding. Extensive experiments
have been conducted to validate the effectiveness of our proposed framework in
protecting facial privacy.
less
Attention De-sparsification Matters: Inducing Diversity in Digital Pathology Representation Learning
0upvotes
By: Saarthak Kapse, Srijan Das, Jingwei Zhang, Rajarsi R. Gupta, Joel Saltz, Dimitris Samaras, Prateek Prasanna
We propose DiRL, a Diversity-inducing Representation Learning technique for
histopathology imaging. Self-supervised learning techniques, such as
contrastive and non-contrastive approaches, have been shown to learn rich and
effective representations of digitized tissue samples with limited pathologist
supervision. Our analysis of vanilla SSL-pretrained models' attention
distribution reveals an insightful observation: sparsity in attention, i... more
We propose DiRL, a Diversity-inducing Representation Learning technique for
histopathology imaging. Self-supervised learning techniques, such as
contrastive and non-contrastive approaches, have been shown to learn rich and
effective representations of digitized tissue samples with limited pathologist
supervision. Our analysis of vanilla SSL-pretrained models' attention
distribution reveals an insightful observation: sparsity in attention, i.e,
models tends to localize most of their attention to some prominent patterns in
the image. Although attention sparsity can be beneficial in natural images due
to these prominent patterns being the object of interest itself, this can be
sub-optimal in digital pathology; this is because, unlike natural images,
digital pathology scans are not object-centric, but rather a complex phenotype
of various spatially intermixed biological components. Inadequate
diversification of attention in these complex images could result in crucial
information loss. To address this, we leverage cell segmentation to densely
extract multiple histopathology-specific representations, and then propose a
prior-guided dense pretext task for SSL, designed to match the multiple
corresponding representations between the views. Through this, the model learns
to attend to various components more closely and evenly, thus inducing adequate
diversification in attention for capturing context rich representations.
Through quantitative and qualitative analysis on multiple tasks across cancer
types, we demonstrate the efficacy of our method and observe that the attention
is more globally distributed.
less
By: Yufeng Huang, Zhuo Chen, Jiaoyan Chen, Jeff Z. Pan, Zhen Yao, Wen Zhang
Multi-modal aspect-based sentiment classification (MABSC) is task of
classifying the sentiment of a target entity mentioned in a sentence and an
image. However, previous methods failed to account for the fine-grained
semantic association between the image and the text, which resulted in limited
identification of fine-grained image aspects and opinions. To address these
limitations, in this paper we propose a new approach called SeqCSG, whic... more
Multi-modal aspect-based sentiment classification (MABSC) is task of
classifying the sentiment of a target entity mentioned in a sentence and an
image. However, previous methods failed to account for the fine-grained
semantic association between the image and the text, which resulted in limited
identification of fine-grained image aspects and opinions. To address these
limitations, in this paper we propose a new approach called SeqCSG, which
enhances the encoder-decoder sentiment classification framework using
sequential cross-modal semantic graphs. SeqCSG utilizes image captions and
scene graphs to extract both global and local fine-grained image information
and considers them as elements of the cross-modal semantic graph along with
tokens from tweets. The sequential cross-modal semantic graph is represented as
a sequence with a multi-modal adjacency matrix indicating relationships between
elements. Experimental results show that the approach outperforms existing
methods and achieves state-of-the-art performance on two standard datasets.
Further analysis has demonstrated that the model can implicitly learn the
correlation between fine-grained information of the image and the text with the
given target. Our code is available at https://github.com/zjukg/SeqCSG.
less
Attending Generalizability in Course of Deep Fake Detection by Exploring Multi-task Learning
0upvotes
By: Pranav Balaji, Abhijit Das, Srijan Das, Antitza Dantcheva
This work explores various ways of exploring multi-task learning (MTL)
techniques aimed at classifying videos as original or manipulated in
cross-manipulation scenario to attend generalizability in deep fake scenario.
The dataset used in our evaluation is FaceForensics++, which features 1000
original videos manipulated by four different techniques, with a total of 5000
videos. We conduct extensive experiments on multi-task learning and cont... more
This work explores various ways of exploring multi-task learning (MTL)
techniques aimed at classifying videos as original or manipulated in
cross-manipulation scenario to attend generalizability in deep fake scenario.
The dataset used in our evaluation is FaceForensics++, which features 1000
original videos manipulated by four different techniques, with a total of 5000
videos. We conduct extensive experiments on multi-task learning and contrastive
techniques, which are well studied in literature for their generalization
benefits. It can be concluded that the proposed detection model is quite
generalized, i.e., accurately detects manipulation methods not encountered
during training as compared to the state-of-the-art.
less
By: Haibo Jin, Haoxuan Che, Hao Chen
Recently, anatomical landmark detection has achieved great progresses on
single-domain data, which usually assumes training and test sets are from the
same domain. However, such an assumption is not always true in practice, which
can cause significant performance drop due to domain shift. To tackle this
problem, we propose a novel framework for anatomical landmark detection under
the setting of unsupervised domain adaptation (UDA), which ai... more
Recently, anatomical landmark detection has achieved great progresses on
single-domain data, which usually assumes training and test sets are from the
same domain. However, such an assumption is not always true in practice, which
can cause significant performance drop due to domain shift. To tackle this
problem, we propose a novel framework for anatomical landmark detection under
the setting of unsupervised domain adaptation (UDA), which aims to transfer the
knowledge from labeled source domain to unlabeled target domain. The framework
leverages self-training and domain adversarial learning to address the domain
gap during adaptation. Specifically, a self-training strategy is proposed to
select reliable landmark-level pseudo-labels of target domain data with dynamic
thresholds, which makes the adaptation more effective. Furthermore, a domain
adversarial learning module is designed to handle the unaligned data
distributions of two domains by learning domain-invariant features via
adversarial training. Our experiments on cephalometric and lung landmark
detection show the effectiveness of the method, which reduces the domain gap by
a large margin and outperforms other UDA methods consistently. The code is
available at https://github.com/jhb86253817/UDA_Med_Landmark.
less
By: Akankshya Kar, Sajal Maheshwari, Shamit Lal, Vinay Sameer Raja Kad
Visual odometry (VO) and SLAM have been using multi-view geometry via local
structure from motion for decades. These methods have a slight disadvantage in
challenging scenarios such as low-texture images, dynamic scenarios, etc.
Meanwhile, use of deep neural networks to extract high level features is
ubiquitous in computer vision. For VO, we can use these deep networks to
extract depth and pose estimates using these high level features. The... more
Visual odometry (VO) and SLAM have been using multi-view geometry via local
structure from motion for decades. These methods have a slight disadvantage in
challenging scenarios such as low-texture images, dynamic scenarios, etc.
Meanwhile, use of deep neural networks to extract high level features is
ubiquitous in computer vision. For VO, we can use these deep networks to
extract depth and pose estimates using these high level features. The visual
odometry task then can be modeled as an image generation task where the pose
estimation is the by-product. This can also be achieved in a self-supervised
manner, thereby eliminating the data (supervised) intensive nature of training
deep neural networks. Although some works tried the similar approach [1], the
depth and pose estimation in the previous works are vague sometimes resulting
in accumulation of error (drift) along the trajectory. The goal of this work is
to tackle these limitations of past approaches and to develop a method that can
provide better depths and pose estimates. To address this, a couple of
approaches are explored: 1) Modeling: Using optical flow and recurrent neural
networks (RNN) in order to exploit spatio-temporal correlations which can
provide more information to estimate depth. 2) Loss function: Generative
adversarial network (GAN) [2] is deployed to improve the depth estimation (and
thereby pose too), as shown in Figure 1. This additional loss term improves the
realism in generated images and reduces artifacts.
less
By: Wei Suo, Mengyang Sun, Weisong Liu, Yiqi Gao, Peng Wang, Yanning Zhang, Qi Wu
VQA Natural Language Explanation (VQA-NLE) task aims to explain the
decision-making process of VQA models in natural language. Unlike traditional
attention or gradient analysis, free-text rationales can be easier to
understand and gain users' trust. Existing methods mostly use post-hoc or
self-rationalization models to obtain a plausible explanation. However, these
frameworks are bottlenecked by the following challenges: 1) the reasoning
pr... more
VQA Natural Language Explanation (VQA-NLE) task aims to explain the
decision-making process of VQA models in natural language. Unlike traditional
attention or gradient analysis, free-text rationales can be easier to
understand and gain users' trust. Existing methods mostly use post-hoc or
self-rationalization models to obtain a plausible explanation. However, these
frameworks are bottlenecked by the following challenges: 1) the reasoning
process cannot be faithfully responded to and suffer from the problem of
logical inconsistency. 2) Human-annotated explanations are expensive and
time-consuming to collect. In this paper, we propose a new Semi-Supervised
VQA-NLE via Self-Critical Learning (S3C), which evaluates the candidate
explanations by answering rewards to improve the logical consistency between
answers and rationales. With a semi-supervised learning framework, the S3C can
benefit from a tremendous amount of samples without human-annotated
explanations. A large number of automatic measures and human evaluations all
show the effectiveness of our method. Meanwhile, the framework achieves a new
state-of-the-art performance on the two VQA-NLE datasets.
less
Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization
0upvotes
By: Tao Yang, Peiran Ren, Xuansong Xie, Lei Zhang
Realistic image super-resolution (Real-ISR) aims to reproduce perceptually
realistic image details from a low-quality input. The commonly used adversarial
training based Real-ISR methods often introduce unnatural visual artifacts and
fail to generate realistic textures for natural scene images. The recently
developed generative stable diffusion models provide a potential solution to
Real-ISR with pre-learned strong image priors. However, th... more
Realistic image super-resolution (Real-ISR) aims to reproduce perceptually
realistic image details from a low-quality input. The commonly used adversarial
training based Real-ISR methods often introduce unnatural visual artifacts and
fail to generate realistic textures for natural scene images. The recently
developed generative stable diffusion models provide a potential solution to
Real-ISR with pre-learned strong image priors. However, the existing methods
along this line either fail to keep faithful pixel-wise image structures or
resort to extra skipped connections to reproduce details, which requires
additional training in image space and limits their extension to other related
tasks in latent space such as image stylization. In this work, we propose a
pixel-aware stable diffusion (PASD) network to achieve robust Real-ISR as well
as personalized stylization. In specific, a pixel-aware cross attention module
is introduced to enable diffusion models perceiving image local structures in
pixel-wise level, while a degradation removal module is used to extract
degradation insensitive features to guide the diffusion process together with
image high level information. By simply replacing the base diffusion model with
a personalized one, our method can generate diverse stylized images without the
need to collect pairwise training data. PASD can be easily integrated into
existing diffusion models such as Stable Diffusion. Experiments on Real-ISR and
personalized stylization demonstrate the effectiveness of our proposed
approach. The source code and models can be found at
\url{https://github.com/yangxy/PASD}.
less
By: Tendai Mapungwana Chikake, Boris Goldengorin, Alexey Samosyuk
We introduce a deterministic approach to edge detection and image
segmentation by formulating pseudo-Boolean polynomials on image patches. The
approach works by applying a binary classification of blob and edge regions in
an image based on the degrees of pseudo-Boolean polynomials calculated on
patches extracted from the provided image. We test our method on simple images
containing primitive shapes of constant and contrasting colour and es... more
We introduce a deterministic approach to edge detection and image
segmentation by formulating pseudo-Boolean polynomials on image patches. The
approach works by applying a binary classification of blob and edge regions in
an image based on the degrees of pseudo-Boolean polynomials calculated on
patches extracted from the provided image. We test our method on simple images
containing primitive shapes of constant and contrasting colour and establish
the feasibility before applying it to complex instances like aerial landscape
images. The proposed method is based on the exploitation of the reduction,
polynomial degree, and equivalence properties of penalty-based pseudo-Boolean
polynomials.
less
By: Jiao Zhang, Xu-Yao Zhang, Cheng-Lin Liu
There are ubiquitous distribution shifts in the real world. However, deep
neural networks (DNNs) are easily biased towards the training set, which causes
severe performance degradation when they receive out-of-distribution data. Many
methods are studied to train models that generalize under various distribution
shifts in the literature of domain generalization (DG). However, the recent
DomainBed and WILDS benchmarks challenged the effective... more
There are ubiquitous distribution shifts in the real world. However, deep
neural networks (DNNs) are easily biased towards the training set, which causes
severe performance degradation when they receive out-of-distribution data. Many
methods are studied to train models that generalize under various distribution
shifts in the literature of domain generalization (DG). However, the recent
DomainBed and WILDS benchmarks challenged the effectiveness of these methods.
Aiming at the problems in the existing research, we propose a new domain
generalization task for handwritten Chinese character recognition (HCCR) to
enrich the application scenarios of DG method research. We evaluate eighteen DG
methods on the proposed PaHCC (Printed and Handwritten Chinese Characters)
dataset and show that the performance of existing methods on this dataset is
still unsatisfactory. Besides, under a designed dynamic DG setting, we reveal
more properties of DG methods and argue that only the leave-one-domain-out
protocol is unreliable. We advocate that researchers in the DG community refer
to dynamic performance of methods for more comprehensive and reliable
evaluation. Our dataset and evaluations bring new perspectives to the community
for more substantial progress. We will make our dataset public with the article
published to facilitate the study of domain generalization.
less
By: Casper van Engelenburg, Seyran Khademi, Jan van Gemert
We propose a simple yet effective metric that measures structural similarity
between visual instances of architectural floor plans, without the need for
learning. Qualitatively, our experiments show that the retrieval results are
similar to deeply learned methods. Effectively comparing instances of floor
plan data is paramount to the success of machine understanding of floor plan
data, including the assessment of floor plan generative model... more
We propose a simple yet effective metric that measures structural similarity
between visual instances of architectural floor plans, without the need for
learning. Qualitatively, our experiments show that the retrieval results are
similar to deeply learned methods. Effectively comparing instances of floor
plan data is paramount to the success of machine understanding of floor plan
data, including the assessment of floor plan generative models and floor plan
recommendation systems. Comparing visual floor plan images goes beyond a sole
pixel-wise visual examination and is crucially about similarities and
differences in the shapes and relations between subdivisions that compose the
layout. Currently, deep metric learning approaches are used to learn a
pair-wise vector representation space that closely mimics the structural
similarity, in which the models are trained on similarity labels that are
obtained by Intersection-over-Union (IoU). To compensate for the lack of
structural awareness in IoU, graph-based approaches such as Graph Matching
Networks (GMNs) are used, which require pairwise inference for comparing data
instances, making GMNs less practical for retrieval applications. In this
paper, an effective evaluation metric for judging the structural similarity of
floor plans, coined SSIG (Structural Similarity by IoU and GED), is proposed
based on both image and graph distances. In addition, an efficient algorithm is
developed that uses SSIG to rank a large-scale floor plan database. Code will
be openly available.
less
By: Vimal K B, Saketh Bachu, Tanmay Garg, Niveditha Lakshmi Narasimhan, Raghavan Konuru, Vineeth N Balasubramanian
Estimating the transferability of publicly available pretrained models to a
target task has assumed an important place for transfer learning tasks in
recent years. Existing efforts propose metrics that allow a user to choose one
model from a pool of pre-trained models without having to fine-tune each model
individually and identify one explicitly. With the growth in the number of
available pre-trained models and the popularity of model ense... more
Estimating the transferability of publicly available pretrained models to a
target task has assumed an important place for transfer learning tasks in
recent years. Existing efforts propose metrics that allow a user to choose one
model from a pool of pre-trained models without having to fine-tune each model
individually and identify one explicitly. With the growth in the number of
available pre-trained models and the popularity of model ensembles, it also
becomes essential to study the transferability of multiple-source models for a
given target task. The few existing efforts study transferability in such
multi-source ensemble settings using just the outputs of the classification
layer and neglect possible domain or task mismatch. Moreover, they overlook the
most important factor while selecting the source models, viz., the cohesiveness
factor between them, which can impact the performance and confidence in the
prediction of the ensemble. To address these gaps, we propose a novel Optimal
tranSport-based suBmOdular tRaNsferability metric (OSBORN) to estimate the
transferability of an ensemble of models to a downstream task. OSBORN
collectively accounts for image domain difference, task difference, and
cohesiveness of models in the ensemble to provide reliable estimates of
transferability. We gauge the performance of OSBORN on both image
classification and semantic segmentation tasks. Our setup includes 28 source
datasets, 11 target datasets, 5 model architectures, and 2 pre-training
methods. We benchmark our method against current state-of-the-art metrics
MS-LEEP and E-LEEP, and outperform them consistently using the proposed
approach.
less
By: Paul Melki IMS, Lionel Bombrun IMS, Boubacar Diallo IMS, Jérôme Dias IMS, Jean-Pierre da Costa IMS
As deep learning predictive models become an integral part of a large
spectrum of precision agricultural systems, a barrier to the adoption of such
automated solutions is the lack of user trust in these highly complex, opaque
and uncertain models. Indeed, deep neural networks are not equipped with any
explicit guarantees that can be used to certify the system's performance,
especially in highly varying uncontrolled environments such as the ... more
As deep learning predictive models become an integral part of a large
spectrum of precision agricultural systems, a barrier to the adoption of such
automated solutions is the lack of user trust in these highly complex, opaque
and uncertain models. Indeed, deep neural networks are not equipped with any
explicit guarantees that can be used to certify the system's performance,
especially in highly varying uncontrolled environments such as the ones
typically faced in computer vision for agriculture.Fortunately, certain methods
developed in other communities can prove to be important for agricultural
applications. This article presents the conformal prediction framework that
provides valid statistical guarantees on the predictive performance of any
black box prediction machine, with almost no assumptions, applied to the
problem of deep visual classification of weeds and crops in real-world
conditions. The framework is exposed with a focus on its practical aspects and
special attention accorded to the Adaptive Prediction Sets (APS) approach that
delivers marginal guarantees on the model's coverage. Marginal results are then
shown to be insufficient to guarantee performance on all groups of individuals
in the population as characterized by their environmental and pedo-climatic
auxiliary data gathered during image acquisition.To tackle this shortcoming,
group-conditional conformal approaches are presented: the ''classical'' method
that consists of iteratively applying the APS procedure on all groups, and a
proposed elegant reformulation and implementation of the procedure using
quantile regression on group membership indicators. Empirical results showing
the validity of the proposed approach are presented and compared to the
marginal APS then discussed.
less
Steerable Conditional Diffusion for Out-of-Distribution Adaptation in Imaging Inverse Problems
0upvotes
By: Riccardo Barbano, Alexander Denker, Hyungjin Chung, Tae Hoon Roh, Simon Arrdige, Peter Maass, Bangti Jin, Jong Chul Ye
Denoising diffusion models have emerged as the go-to framework for solving
inverse problems in imaging. A critical concern regarding these models is their
performance on out-of-distribution (OOD) tasks, which remains an under-explored
challenge. Realistic reconstructions inconsistent with the measured data can be
generated, hallucinating image features that are uniquely present in the
training dataset. To simultaneously enforce data-consist... more
Denoising diffusion models have emerged as the go-to framework for solving
inverse problems in imaging. A critical concern regarding these models is their
performance on out-of-distribution (OOD) tasks, which remains an under-explored
challenge. Realistic reconstructions inconsistent with the measured data can be
generated, hallucinating image features that are uniquely present in the
training dataset. To simultaneously enforce data-consistency and leverage
data-driven priors, we introduce a novel sampling framework called Steerable
Conditional Diffusion. This framework adapts the denoising network specifically
to the available measured data. Utilising our proposed method, we achieve
substantial enhancements in OOD performance across diverse imaging modalities,
advancing the robust deployment of denoising diffusion models in real-world
applications.
less
By: James Robert Kubricht, Zhaoyuan Yang, Jianwei Qiu, Peter Henry Tu
Deep learning approaches to natural language processing have made great
strides in recent years. While these models produce symbols that convey vast
amounts of diverse knowledge, it is unclear how such symbols are grounded in
data from the world. In this paper, we explore the development of a private
language for visual data representation by training emergent language (EL)
encoders/decoders in both i) a traditional referential game environ... more
Deep learning approaches to natural language processing have made great
strides in recent years. While these models produce symbols that convey vast
amounts of diverse knowledge, it is unclear how such symbols are grounded in
data from the world. In this paper, we explore the development of a private
language for visual data representation by training emergent language (EL)
encoders/decoders in both i) a traditional referential game environment and ii)
a contrastive learning environment utilizing a within-class matching training
paradigm. An additional classification layer utilizing neural machine
translation and random forest classification was used to transform symbolic
representations (sequences of integer symbols) to class labels. These methods
were applied in two experiments focusing on object recognition and action
recognition. For object recognition, a set of sketches produced by human
participants from real imagery was used (Sketchy dataset) and for action
recognition, 2D trajectories were generated from 3D motion capture systems
(MOVI dataset). In order to interpret the symbols produced for data in each
experiment, gradient-weighted class activation mapping (Grad-CAM) methods were
used to identify pixel regions indicating semantic features which contribute
evidence towards symbols in learned languages. Additionally, a t-distributed
stochastic neighbor embedding (t-SNE) method was used to investigate embeddings
learned by CNN feature extractors.
less
By: Amir Belder, Refael Vivanti, Ayellet Tal
Bundle adjustment is the common way to solve localization and mapping. It is
an iterative process in which a system of non-linear equations is solved using
two optimization methods, weighted by a damping factor. In the classic
approach, the latter is chosen heuristically by the Levenberg-Marquardt
algorithm on each iteration. This might take many iterations, making the
process computationally expensive, which might be harmful to real-time
a... more
Bundle adjustment is the common way to solve localization and mapping. It is
an iterative process in which a system of non-linear equations is solved using
two optimization methods, weighted by a damping factor. In the classic
approach, the latter is chosen heuristically by the Levenberg-Marquardt
algorithm on each iteration. This might take many iterations, making the
process computationally expensive, which might be harmful to real-time
applications. We propose to replace this heuristic by viewing the problem in a
holistic manner, as a game, and formulating it as a reinforcement-learning
task. We set an environment which solves the non-linear equations and train an
agent to choose the damping factor in a learned manner. We demonstrate that our
approach considerably reduces the number of iterations required to reach the
bundle adjustment's convergence, on both synthetic and real-life scenarios. We
show that this reduction benefits the classic approach and can be integrated
with other bundle adjustment acceleration methods.
less
By: Jia Li, Wei Qian, Kun Li, Qi Li, Dan Guo, Meng Wang
In this paper, we present our solution to the MuSe-Personalisation
sub-challenge in the MuSe 2023 Multimodal Sentiment Analysis Challenge. The
task of MuSe-Personalisation aims to predict the continuous arousal and valence
values of a participant based on their audio-visual, language, and
physiological signal modalities data. Considering different people have
personal characteristics, the main challenge of this task is how to build
robustne... more
In this paper, we present our solution to the MuSe-Personalisation
sub-challenge in the MuSe 2023 Multimodal Sentiment Analysis Challenge. The
task of MuSe-Personalisation aims to predict the continuous arousal and valence
values of a participant based on their audio-visual, language, and
physiological signal modalities data. Considering different people have
personal characteristics, the main challenge of this task is how to build
robustness feature presentation for sentiment prediction. To address this
issue, we propose exploiting diverse features. Specifically, we proposed a
series of feature extraction methods to build a robust representation and model
ensemble. We empirically evaluate the performance of the utilized method on the
officially provided dataset. \textbf{As a result, we achieved 3rd place in the
MuSe-Personalisation sub-challenge.} Specifically, we achieve the results of
0.8492 and 0.8439 for MuSe-Personalisation in terms of arousal and valence CCC.
less
By: Chunqing Ruan, Hongjian Wang
Parameter efficient transfer learning (PETL) is an emerging research spot
that aims to adapt large-scale pre-trained models to downstream tasks. Recent
advances have achieved great success in saving storage and computation costs.
However, these methods do not take into account instance-specific visual clues
for visual tasks. In this paper, we propose a Dynamic Visual Prompt Tuning
framework (DVPT), which can generate a dynamic instance-wise... more
Parameter efficient transfer learning (PETL) is an emerging research spot
that aims to adapt large-scale pre-trained models to downstream tasks. Recent
advances have achieved great success in saving storage and computation costs.
However, these methods do not take into account instance-specific visual clues
for visual tasks. In this paper, we propose a Dynamic Visual Prompt Tuning
framework (DVPT), which can generate a dynamic instance-wise token for each
image. In this way, it can capture the unique visual feature of each image,
which can be more suitable for downstream visual tasks. We designed a Meta-Net
module that can generate learnable prompts based on each image, thereby
capturing dynamic instance-wise visual features. Extensive experiments on a
wide range of downstream recognition tasks show that DVPT achieves superior
performance than other PETL methods. More importantly, DVPT even outperforms
full fine-tuning on 17 out of 19 downstream tasks while maintaining high
parameter efficiency. Our code will be released soon.
less
Self-Training and Multi-Task Learning for Limited Data: Evaluation Study on Object Detection
0upvotes
By: Hoàng-Ân Lê, Minh-Tan Pham
Self-training allows a network to learn from the predictions of a more
complicated model, thus often requires well-trained teacher models and mixture
of teacher-student data while multi-task learning jointly optimizes different
targets to learn salient interrelationship and requires multi-task annotations
for each training example. These frameworks, despite being particularly data
demanding have potentials for data exploitation if such assu... more
Self-training allows a network to learn from the predictions of a more
complicated model, thus often requires well-trained teacher models and mixture
of teacher-student data while multi-task learning jointly optimizes different
targets to learn salient interrelationship and requires multi-task annotations
for each training example. These frameworks, despite being particularly data
demanding have potentials for data exploitation if such assumptions can be
relaxed. In this paper, we compare self-training object detection under the
deficiency of teacher training data where students are trained on unseen
examples by the teacher, and multi-task learning with partially annotated data,
i.e. single-task annotation per training example. Both scenarios have their own
limitation but potentially helpful with limited annotated data. Experimental
results show the improvement of performance when using a weak teacher with
unseen data for training a multi-task student. Despite the limited setup we
believe the experimental results show the potential of multi-task knowledge
distillation and self-training, which could be beneficial for future study.
Source code is at https://lhoangan.github.io/multas.
less
By: Zhuo Zhou, Wenxuan Liu, Danni Xu, Zheng Wang, Jian Zhao
This paper introduces a new and challenging Hidden Intention Discovery (HID)
task. Unlike existing intention recognition tasks, which are based on obvious
visual representations to identify common intentions for normal behavior, HID
focuses on discovering hidden intentions when humans try to hide their
intentions for abnormal behavior. HID presents a unique challenge in that
hidden intentions lack the obvious visual representations to disti... more
This paper introduces a new and challenging Hidden Intention Discovery (HID)
task. Unlike existing intention recognition tasks, which are based on obvious
visual representations to identify common intentions for normal behavior, HID
focuses on discovering hidden intentions when humans try to hide their
intentions for abnormal behavior. HID presents a unique challenge in that
hidden intentions lack the obvious visual representations to distinguish them
from normal intentions. Fortunately, from a sociological and psychological
perspective, we find that the difference between hidden and normal intentions
can be reasoned from multiple micro-behaviors, such as gaze, attention, and
facial expressions. Therefore, we first discover the relationship between
micro-behavior and hidden intentions and use graph structure to reason about
hidden intentions. To facilitate research in the field of HID, we also
constructed a seminal dataset containing a hidden intention annotation of a
typical theft scenario for HID. Extensive experiments show that the proposed
network improves performance on the HID task by 9.9\% over the state-of-the-art
method SBP.
less
By: Xiangyu Chen, Xintao Wang, Wenlong Zhang, Xiangtao Kong, Yu Qiao, Jiantao Zhou, Chao Dong
Transformer-based methods have shown impressive performance in image
restoration tasks, such as image super-resolution and denoising. However, we
find that these networks can only utilize a limited spatial range of input
information through attribution analysis. This implies that the potential of
Transformer is still not fully exploited in existing networks. In order to
activate more input pixels for better restoration, we propose a new Hyb... more
Transformer-based methods have shown impressive performance in image
restoration tasks, such as image super-resolution and denoising. However, we
find that these networks can only utilize a limited spatial range of input
information through attribution analysis. This implies that the potential of
Transformer is still not fully exploited in existing networks. In order to
activate more input pixels for better restoration, we propose a new Hybrid
Attention Transformer (HAT). It combines both channel attention and
window-based self-attention schemes, thus making use of their complementary
advantages. Moreover, to better aggregate the cross-window information, we
introduce an overlapping cross-attention module to enhance the interaction
between neighboring window features. In the training stage, we additionally
adopt a same-task pre-training strategy to further exploit the potential of the
model for further improvement. Extensive experiments have demonstrated the
effectiveness of the proposed modules. We further scale up the model to show
that the performance of the SR task can be greatly improved. Besides, we extend
HAT to more image restoration applications, including real-world image
super-resolution, Gaussian image denoising and image compression artifacts
reduction. Experiments on benchmark and real-world datasets demonstrate that
our HAT achieves state-of-the-art performance both quantitatively and
qualitatively. Codes and models are publicly available at
https://github.com/XPixelGroup/HAT.
less
By: Heewon Lee, Sangtae Ahn
Object detection is a task that performs position identification and label
classification of objects in images or videos. The information obtained through
this process plays an essential role in various tasks in the field of computer
vision. In object detection, the data utilized for training and validation
typically originate from public datasets that are well-balanced in terms of the
number of objects ascribed to each class in an image. H... more
Object detection is a task that performs position identification and label
classification of objects in images or videos. The information obtained through
this process plays an essential role in various tasks in the field of computer
vision. In object detection, the data utilized for training and validation
typically originate from public datasets that are well-balanced in terms of the
number of objects ascribed to each class in an image. However, in real-world
scenarios, handling datasets with much greater class imbalance, i.e., very
different numbers of objects for each class , is much more common, and this
imbalance may reduce the performance of object detection when predicting unseen
test images. In our study, thus, we propose a method that evenly distributes
the classes in an image for training and validation, solving the class
imbalance problem in object detection. Our proposed method aims to maintain a
uniform class distribution through multi-label stratification. We tested our
proposed method not only on public datasets that typically exhibit balanced
class distribution but also on custom datasets that may have imbalanced class
distribution. We found that our proposed method was more effective on datasets
containing severe imbalance and less data. Our findings indicate that the
proposed method can be effectively used on datasets with substantially
imbalanced class distribution.
less
By: Yafei Zhang, Keying Du, Huafeng Li, Zhengtao Yu, Yu Liu
Infrared and visible image fusion (IVIF) aims to extract and integrate the
complementary information in two different modalities to generate high-quality
fused images with salient targets and abundant texture details. However,
current image fusion methods go to great lengths to excavate complementary
features, which is generally achieved through two efforts. On the one hand, the
feature extraction network is expected to have excellent perfo... more
Infrared and visible image fusion (IVIF) aims to extract and integrate the
complementary information in two different modalities to generate high-quality
fused images with salient targets and abundant texture details. However,
current image fusion methods go to great lengths to excavate complementary
features, which is generally achieved through two efforts. On the one hand, the
feature extraction network is expected to have excellent performance in
extracting complementary information. On the other hand, complex fusion
strategies are often designed to aggregate the complementary information. In
other words, enabling the network to perceive and extract complementary
information is extremely challenging. Complicated fusion strategies, while
effective, still run the risk of losing weak edge details. To this end, this
paper rethinks the IVIF outside the box, proposing a complementary-redundant
information transfer network (C-RITNet). It reasonably transfers complementary
information into redundant one, which integrates both the shared and
complementary features from two modalities. Hence, the proposed method is able
to alleviate the challenges posed by the complementary information extraction
and reduce the reliance on sophisticated fusion strategies. Specifically, to
skillfully sidestep aggregating complementary information in IVIF, we first
design the mutual information transfer (MIT) module to mutually represent
features from two modalities, roughly transferring complementary information
into redundant one. Then, a redundant information acquisition supervised by
source image (RIASSI) module is devised to further ensure the
complementary-redundant information transfer after MIT. Meanwhile, we also
propose a structure information preservation (SIP) module to guarantee that the
edge structure information of the source images can be transferred to the
fusion results.
less
By: István Sárándi, Alexander Hermans, Bastian Leibe
Deep learning-based 3D human pose estimation performs best when trained on large amounts of labeled data, making combined learning from many datasets an important research direction. One obstacle to this endeavor are the different skeleton formats provided by different datasets, i.e., they do not label the same set of anatomical landmarks. There is little prior research on how to best supervise one model with such discrepant labels. We show t... more
Deep learning-based 3D human pose estimation performs best when trained on large amounts of labeled data, making combined learning from many datasets an important research direction. One obstacle to this endeavor are the different skeleton formats provided by different datasets, i.e., they do not label the same set of anatomical landmarks. There is little prior research on how to best supervise one model with such discrepant labels. We show that simply using separate output heads for different skeletons results in inconsistent depth estimates and insufficient information sharing across skeletons. As a remedy, we propose a novel affine-combining autoencoder (ACAE) method to perform dimensionality reduction on the number of landmarks. The discovered latent 3D points capture the redundancy among skeletons, enabling enhanced information sharing when used for consistency regularization. Our approach scales to an extreme multi-dataset regime, where we use 28 3D human pose datasets to supervise one model, which outperforms prior work on a range of benchmarks, including the challenging 3D Poses in the Wild (3DPW) dataset. Our code and models are available for research purposes. less
By: Haixu Song, Shiyu Huang, Yinpeng Dong, Wei-Wei Tu
The rise of deepfake images, especially of well-known personalities, poses a
serious threat to the dissemination of authentic information. To tackle this,
we present a thorough investigation into how deepfakes are produced and how
they can be identified. The cornerstone of our research is a rich collection of
artificial celebrity faces, titled DeepFakeFace (DFF). We crafted the DFF
dataset using advanced diffusion models and have shared it ... more
The rise of deepfake images, especially of well-known personalities, poses a
serious threat to the dissemination of authentic information. To tackle this,
we present a thorough investigation into how deepfakes are produced and how
they can be identified. The cornerstone of our research is a rich collection of
artificial celebrity faces, titled DeepFakeFace (DFF). We crafted the DFF
dataset using advanced diffusion models and have shared it with the community
through online platforms. This data serves as a robust foundation to train and
test algorithms designed to spot deepfakes. We carried out a thorough review of
the DFF dataset and suggest two evaluation methods to gauge the strength and
adaptability of deepfake recognition tools. The first method tests whether an
algorithm trained on one type of fake images can recognize those produced by
other methods. The second evaluates the algorithm's performance with imperfect
images, like those that are blurry, of low quality, or compressed. Given varied
results across deepfake methods and image changes, our findings stress the need
for better deepfake detectors. Our DFF dataset and tests aim to boost the
development of more effective tools against deepfakes.
less
By: Aman Bhatta, Gabriella Pangelinan, Micheal C. King, Kevin W. Bowyer
Most studies to date that have examined demographic variations in face
recognition accuracy have analyzed 1-to-1 matching accuracy, using images that
could be described as "government ID quality". This paper analyzes the accuracy
of 1-to-many facial identification across demographic groups, and in the
presence of blur and reduced resolution in the probe image as might occur in
"surveillance camera quality" images. Cumulative match character... more
Most studies to date that have examined demographic variations in face
recognition accuracy have analyzed 1-to-1 matching accuracy, using images that
could be described as "government ID quality". This paper analyzes the accuracy
of 1-to-many facial identification across demographic groups, and in the
presence of blur and reduced resolution in the probe image as might occur in
"surveillance camera quality" images. Cumulative match characteristic
curves(CMC) are not appropriate for comparing propensity for rank-one
recognition errors across demographics, and so we introduce three metrics for
this: (1) d' metric between mated and non-mated score distributions, (2)
absolute score difference between thresholds in the high-similarity tail of the
non-mated and the low-similarity tail of the mated distribution, and (3)
distribution of (mated - non-mated rank one scores) across the set of probe
images. We find that demographic variation in 1-to-many accuracy does not
entirely follow what has been observed in 1-to-1 matching accuracy. Also,
different from 1-to-1 accuracy, demographic comparison of 1-to-many accuracy
can be affected by different numbers of identities and images across
demographics. Finally, we show that increased blur in the probe image, or
reduced resolution of the face in the probe image, can significantly increase
the false positive identification rate. And we show that the demographic
variation in these high blur or low resolution conditions is much larger for
male/ female than for African-American / Caucasian. The point that 1-to-many
accuracy can potentially collapse in the context of processing "surveillance
camera quality" probe images against a "government ID quality" gallery is an
important one.
less
By: Zhenghong Zhou, Huangxuan Zhao, Jiemin Fang, Dongqiao Xiang, Lei Chen, Lingxia Wu, Feihong Wu, Wenyu Liu, Chuansheng Zheng, Xinggang Wang
Four-dimensional Digital Subtraction Angiography (4D DSA) plays a critical
role in the diagnosis of many medical diseases, such as Arteriovenous
Malformations (AVM) and Arteriovenous Fistulas (AVF). Despite its significant
application value, the reconstruction of 4D DSA demands numerous views to
effectively model the intricate vessels and radiocontrast flow, thereby
implying a significant radiation dose. To address this high radiation issue... more
Four-dimensional Digital Subtraction Angiography (4D DSA) plays a critical
role in the diagnosis of many medical diseases, such as Arteriovenous
Malformations (AVM) and Arteriovenous Fistulas (AVF). Despite its significant
application value, the reconstruction of 4D DSA demands numerous views to
effectively model the intricate vessels and radiocontrast flow, thereby
implying a significant radiation dose. To address this high radiation issue, we
propose a Time-aware Attenuation Voxel (TiAVox) approach for sparse-view 4D DSA
reconstruction, which paves the way for high-quality 4D imaging. Additionally,
2D and 3D DSA imaging results can be generated from the reconstructed 4D DSA
images. TiAVox introduces 4D attenuation voxel grids, which reflect attenuation
properties from both spatial and temporal dimensions. It is optimized by
minimizing discrepancies between the rendered images and sparse 2D DSA images.
Without any neural network involved, TiAVox enjoys specific physical
interpretability. The parameters of each learnable voxel represent the
attenuation coefficients. We validated the TiAVox approach on both clinical and
simulated datasets, achieving a 31.23 Peak Signal-to-Noise Ratio (PSNR) for
novel view synthesis using only 30 views on the clinically sourced dataset,
whereas traditional Feldkamp-Davis-Kress methods required 133 views. Similarly,
with merely 10 views from the synthetic dataset, TiAVox yielded a PSNR of 34.32
for novel view synthesis and 41.40 for 3D reconstruction. We also executed
ablation studies to corroborate the essential components of TiAVox. The code
will be publically available.
less
By: Zhuo Chen, Yufeng Huang, Jiaoyan Chen, Yuxia Geng, Wen Zhang, Yin Fang, Jeff Z. Pan, Huajun Chen
Zero-shot learning (ZSL) aims to predict unseen classes whose samples have
never appeared during training. One of the most effective and widely used
semantic information for zero-shot image classification are attributes which
are annotations for class-level visual characteristics. However, the current
methods often fail to discriminate those subtle visual distinctions between
images due to not only the shortage of fine-grained annotations, ... more
Zero-shot learning (ZSL) aims to predict unseen classes whose samples have
never appeared during training. One of the most effective and widely used
semantic information for zero-shot image classification are attributes which
are annotations for class-level visual characteristics. However, the current
methods often fail to discriminate those subtle visual distinctions between
images due to not only the shortage of fine-grained annotations, but also the
attribute imbalance and co-occurrence. In this paper, we present a
transformer-based end-to-end ZSL method named DUET, which integrates latent
semantic knowledge from the pre-trained language models (PLMs) via a
self-supervised multi-modal learning paradigm. Specifically, we (1) developed a
cross-modal semantic grounding network to investigate the model's capability of
disentangling semantic attributes from the images; (2) applied an
attribute-level contrastive learning strategy to further enhance the model's
discrimination on fine-grained visual characteristics against the attribute
co-occurrence and imbalance; (3) proposed a multi-task learning policy for
considering multi-model objectives. We find that our DUET can achieve
state-of-the-art performance on three standard ZSL benchmarks and a knowledge
graph equipped ZSL benchmark. Its components are effective and its predictions
are interpretable.
less
By: Pakizar Shamoi, Atsushi Inoue, Hiroharu Kawanaka
Color is the most important intrinsic sensory feature that has a powerful
impact on product sales. Color is even responsible for raising the aesthetic
senses in our brains. Account for individual differences is crucial in color
aesthetics. It requires user-driven mechanisms for various e-commerce
applications. We propose a method for quantitative evaluation of all types of
perceptual responses to color(s): distinct color preference, color h... more
Color is the most important intrinsic sensory feature that has a powerful
impact on product sales. Color is even responsible for raising the aesthetic
senses in our brains. Account for individual differences is crucial in color
aesthetics. It requires user-driven mechanisms for various e-commerce
applications. We propose a method for quantitative evaluation of all types of
perceptual responses to color(s): distinct color preference, color harmony, and
color combination preference. Preference for color schemes can be predicted by
combining preferences for the basic colors and ratings of color harmony.
Harmonious pallets are extracted from big data set using comparison algorithms
based on fuzzy similarity and grouping. The proposed model results in useful
predictions of harmony and preference of multicolored images. For example, in
the context of apparel coordination, it allows predicting a preference for a
look based on clothing colors. Our approach differs from standard aesthetic
models, since in accounts for a personal variation. In addition, it can process
not only lower-order color pairs, but also groups of several colors.
less
By: Chenghao Li, Chaoning Zhang
The success of Vision Transformer (ViT) has been widely reported on a wide
range of image recognition tasks. The merit of ViT over CNN has been largely
attributed to large training datasets or auxiliary pre-training. Without
pre-training, the performance of ViT on small datasets is limited because the
global self-attention has limited capacity in local modeling. Towards boosting
ViT on small datasets without pre-training, this work improves... more
The success of Vision Transformer (ViT) has been widely reported on a wide
range of image recognition tasks. The merit of ViT over CNN has been largely
attributed to large training datasets or auxiliary pre-training. Without
pre-training, the performance of ViT on small datasets is limited because the
global self-attention has limited capacity in local modeling. Towards boosting
ViT on small datasets without pre-training, this work improves its local
modeling by applying a weight mask on the original self-attention matrix. A
straightforward way to locally adapt the self-attention matrix can be realized
by an element-wise learnable weight mask (ELM), for which our preliminary
results show promising results. However, the element-wise simple learnable
weight mask not only induces a non-trivial additional parameter overhead but
also increases the optimization complexity. To this end, this work proposes a
novel Gaussian mixture mask (GMM) in which one mask only has two learnable
parameters and it can be conveniently used in any ViT variants whose attention
mechanism allows the use of masks. Experimental results on multiple small
datasets demonstrate that the effectiveness of our proposed Gaussian mask for
boosting ViTs for free (almost zero additional parameter or computation cost).
Our code will be publicly available at
\href{https://github.com/CatworldLee/Gaussian-Mixture-Mask-Attention}{https://github.com/CatworldLee/Gaussian-Mixture-Mask-Attention}.
less
Jersey Number Recognition using Keyframe Identification from Low-Resolution Broadcast Videos
0upvotes
By: Bavesh Balaji, Jerrin Bright, Harish Prakash, Yuhao Chen, David A Clausi, John Zelek
Player identification is a crucial component in vision-driven soccer
analytics, enabling various downstream tasks such as player assessment, in-game
analysis, and broadcast production. However, automatically detecting jersey
numbers from player tracklets in videos presents challenges due to motion blur,
low resolution, distortions, and occlusions. Existing methods, utilizing
Spatial Transformer Networks, CNNs, and Vision Transformers, have ... more
Player identification is a crucial component in vision-driven soccer
analytics, enabling various downstream tasks such as player assessment, in-game
analysis, and broadcast production. However, automatically detecting jersey
numbers from player tracklets in videos presents challenges due to motion blur,
low resolution, distortions, and occlusions. Existing methods, utilizing
Spatial Transformer Networks, CNNs, and Vision Transformers, have shown success
in image data but struggle with real-world video data, where jersey numbers are
not visible in most of the frames. Hence, identifying frames that contain the
jersey number is a key sub-problem to tackle. To address these issues, we
propose a robust keyframe identification module that extracts frames containing
essential high-level information about the jersey number. A spatio-temporal
network is then employed to model spatial and temporal context and predict the
probabilities of jersey numbers in the video. Additionally, we adopt a
multi-task loss function to predict the probability distribution of each digit
separately. Extensive evaluations on the SoccerNet dataset demonstrate that
incorporating our proposed keyframe identification module results in a
significant 37.81% and 37.70% increase in the accuracies of 2 different test
sets with domain gaps. These results highlight the effectiveness and importance
of our approach in tackling the challenges of automatic jersey number detection
in sports videos.
less
By: Jiaxiu Li, Kun Li, Jia Li, Guoliang Chen, Dan Guo, Meng Wang
Make-up temporal video grounding (MTVG) aims to localize the target video
segment which is semantically related to a sentence describing a make-up
activity, given a long video. Compared with the general video grounding task,
MTVG focuses on meticulous actions and changes on the face. The make-up
instruction step, usually involving detailed differences in products and facial
areas, is more fine-grained than general activities (e.g, cooking a... more
Make-up temporal video grounding (MTVG) aims to localize the target video
segment which is semantically related to a sentence describing a make-up
activity, given a long video. Compared with the general video grounding task,
MTVG focuses on meticulous actions and changes on the face. The make-up
instruction step, usually involving detailed differences in products and facial
areas, is more fine-grained than general activities (e.g, cooking activity and
furniture assembly). Thus, existing general approaches cannot locate the target
activity effectually. More specifically, existing proposal generation modules
are not yet fully developed in providing semantic cues for the more
fine-grained make-up semantic comprehension. To tackle this issue, we propose
an effective proposal-based framework named Dual-Path Temporal Map Optimization
Network (DPTMO) to capture fine-grained multimodal semantic details of make-up
activities. DPTMO extracts both query-agnostic and query-guided features to
construct two proposal sets and uses specific evaluation methods for the two
sets. Different from the commonly used single structure in previous methods,
our dual-path structure can mine more semantic information in make-up videos
and distinguish fine-grained actions well. These two candidate sets represent
the cross-modal makeup video-text similarity and multi-modal fusion
relationship, complementing each other. Each set corresponds to its respective
optimization perspective, and their joint prediction enhances the accuracy of
video timestamp prediction. Comprehensive experiments on the YouMakeup dataset
demonstrate our proposed dual structure excels in fine-grained semantic
comprehension.
less
By: Hiroshi Fukui, Taiki Miyagawa, Yusuke Morishita
We propose a conceptually simple and thus fast multi-object tracking (MOT) model that does not require any attached modules, such as the Kalman filter, Hungarian algorithm, transformer blocks, or graph networks. Conventional MOT models are built upon the multi-step modules listed above, and thus the computational cost is high. Our proposed end-to-end MOT model, TicrossNet, is composed of a base detector and a cross-attention module only. As... more
We propose a conceptually simple and thus fast multi-object tracking (MOT) model that does not require any attached modules, such as the Kalman filter, Hungarian algorithm, transformer blocks, or graph networks. Conventional MOT models are built upon the multi-step modules listed above, and thus the computational cost is high. Our proposed end-to-end MOT model, TicrossNet, is composed of a base detector and a cross-attention module only. As a result, the overhead of tracking does not increase
significantly even when the number of instances ($N_t$) increases. We show that TicrossNet runs in real-time; specifically, it achieves 32.6 FPS on MOT17 and 31.0 FPS on MOT20 (Tesla V100), which includes as many as >100 instances per frame. We also demonstrate that TicrossNet is robust to $N_t$; thus, it does not have to change the size of the base detector, depending on $N_t$, as is often done by other models for real-time processing.
less
By: Qingyao Shuai, Chi Zhang, Kaizhi Yang, Xuejin Chen
Unsupervised methods for reconstructing structures face significant
challenges in capturing the geometric details with consistent structures among
diverse shapes of the same category. To address this issue, we present a novel
unsupervised structural reconstruction method, named DPF-Net, based on a new
Deformable Primitive Field (DPF) representation, which allows for high-quality
shape reconstruction using parameterized geometric primitives.... more
Unsupervised methods for reconstructing structures face significant
challenges in capturing the geometric details with consistent structures among
diverse shapes of the same category. To address this issue, we present a novel
unsupervised structural reconstruction method, named DPF-Net, based on a new
Deformable Primitive Field (DPF) representation, which allows for high-quality
shape reconstruction using parameterized geometric primitives. We design a
two-stage shape reconstruction pipeline which consists of a primitive
generation module and a primitive deformation module to approximate the target
shape of each part progressively. The primitive generation module estimates the
explicit orientation, position, and size parameters of parameterized geometric
primitives, while the primitive deformation module predicts a dense deformation
field based on a parameterized primitive field to recover shape details. The
strong shape prior encoded in parameterized geometric primitives enables our
DPF-Net to extract high-level structures and recover fine-grained shape details
consistently. The experimental results on three categories of objects in
diverse shapes demonstrate the effectiveness and generalization ability of our
DPF-Net on structural reconstruction and shape segmentation.
less
By: Seong Hun Lee, Javier Civera
In this work, we propose a novel method for robust single rotation averaging
that can efficiently handle an extremely large fraction of outliers. Our
approach is to minimize the total truncated least unsquared deviations (TLUD)
cost of geodesic distances. The proposed algorithm consists of three steps:
First, we consider each input rotation as a potential initial solution and
choose the one that yields the least sum of truncated chordal dev... more
In this work, we propose a novel method for robust single rotation averaging
that can efficiently handle an extremely large fraction of outliers. Our
approach is to minimize the total truncated least unsquared deviations (TLUD)
cost of geodesic distances. The proposed algorithm consists of three steps:
First, we consider each input rotation as a potential initial solution and
choose the one that yields the least sum of truncated chordal deviations. Next,
we obtain the inlier set using the initial solution and compute its chordal
$L_2$-mean. Finally, starting from this estimate, we iteratively compute the
geodesic $L_1$-mean of the inliers using the Weiszfeld algorithm on $SO(3)$. An
extensive evaluation shows that our method is robust against up to 99% outliers
given a sufficient number of accurate inliers, outperforming the current state
of the art.
less
By: Oscar David Rafael Narvaez Luces, Minh-Tan Pham, Quentin Poterek, Rémi Braun
Wildfire detection using satellite images is a widely studied task in remote
sensing with many applications to fire delineation and mapping. Recently, deep
learning methods have become a scalable solution to automate this task,
especially in the field of unsupervised learning where no training data is
available. This is particularly important in the context of emergency risk
monitoring where fast and effective detection is needed, generally... more
Wildfire detection using satellite images is a widely studied task in remote
sensing with many applications to fire delineation and mapping. Recently, deep
learning methods have become a scalable solution to automate this task,
especially in the field of unsupervised learning where no training data is
available. This is particularly important in the context of emergency risk
monitoring where fast and effective detection is needed, generally based on
high-resolution satellite data. Among various approaches, Anomaly Detection
(AD) appears to be highly potential thanks to its broad applications in
computer vision, medical imaging, as well as remote sensing. In this work, we
build upon the framework of Vector Quantized Variational Autoencoder (VQ-VAE),
a popular reconstruction-based AD method with discrete latent spaces, to
perform unsupervised burnt area extraction. We integrate VQ-VAE into an
end-to-end framework with an intensive post-processing step using dedicated
vegetation, water and brightness indexes. Our experiments conducted on
high-resolution SPOT-6/7 images provide promising results of the proposed
technique, showing its high potential in future research on unsupervised burnt
area extraction.
less
By: Teppei Suzuki
We envision a system to continuously build and maintain a map based on
earth-scale neural radiance fields (NeRF) using data collected from vehicles
and drones in a lifelong learning manner. However, existing large-scale
modeling by NeRF has problems in terms of scalability and maintainability when
modeling earth-scale environments. Therefore, to address these problems, we
propose a federated learning pipeline for large-scale modeling with N... more
We envision a system to continuously build and maintain a map based on
earth-scale neural radiance fields (NeRF) using data collected from vehicles
and drones in a lifelong learning manner. However, existing large-scale
modeling by NeRF has problems in terms of scalability and maintainability when
modeling earth-scale environments. Therefore, to address these problems, we
propose a federated learning pipeline for large-scale modeling with NeRF. We
tailor the model aggregation pipeline in federated learning for NeRF, thereby
allowing local updates of NeRF. In the aggregation step, the accuracy of the
clients' global pose is critical. Thus, we also propose global pose alignment
to align the noisy global pose of clients before the aggregation step. In
experiments, we show the effectiveness of the proposed pose alignment and the
federated learning pipeline on the large-scale scene dataset, Mill19.
less
By: Shuyi Zhou, Shuxiang Xie, Ryoichi Ishikawa, Ken Sakurada, Masaki Onishi, Takeshi Oishi
Sensor fusion has become a popular topic in robotics. However, conventional
fusion methods encounter many difficulties, such as data representation
differences, sensor variations, and extrinsic calibration. For example, the
calibration methods used for LiDAR-camera fusion often require manual operation
and auxiliary calibration targets. Implicit neural representations (INRs) have
been developed for 3D scenes, and the volume density distribu... more
Sensor fusion has become a popular topic in robotics. However, conventional
fusion methods encounter many difficulties, such as data representation
differences, sensor variations, and extrinsic calibration. For example, the
calibration methods used for LiDAR-camera fusion often require manual operation
and auxiliary calibration targets. Implicit neural representations (INRs) have
been developed for 3D scenes, and the volume density distribution involved in
an INR unifies the scene information obtained by different types of sensors.
Therefore, we propose implicit neural fusion (INF) for LiDAR and camera. INF
first trains a neural density field of the target scene using LiDAR frames.
Then, a separate neural color field is trained using camera images and the
trained neural density field. Along with the training process, INF both
estimates LiDAR poses and optimizes extrinsic parameters. Our experiments
demonstrate the high accuracy and stable performance of the proposed method.
less
By: Bang Yang, Fenglin Liu, Xian Wu, Yaowei Wang, Xu Sun, Yuexian Zou
Supervised visual captioning models typically require a large scale of images
or videos paired with descriptions in a specific language (i.e., the
vision-caption pairs) for training. However, collecting and labeling
large-scale datasets is time-consuming and expensive for many scenarios and
languages. Therefore, sufficient labeled pairs are usually not available. To
deal with the label shortage problem, we present a simple yet effective
zer... more
Supervised visual captioning models typically require a large scale of images
or videos paired with descriptions in a specific language (i.e., the
vision-caption pairs) for training. However, collecting and labeling
large-scale datasets is time-consuming and expensive for many scenarios and
languages. Therefore, sufficient labeled pairs are usually not available. To
deal with the label shortage problem, we present a simple yet effective
zero-shot approach MultiCapCLIP that can generate visual captions for different
scenarios and languages without any labeled vision-caption pairs of downstream
datasets. In the training stage, MultiCapCLIP only requires text data for
input. Then it conducts two main steps: 1) retrieving concept prompts that
preserve the corresponding domain knowledge of new scenarios; 2) auto-encoding
the prompts to learn writing styles to output captions in a desired language.
In the testing stage, MultiCapCLIP instead takes visual data as input directly
to retrieve the concept prompts to generate the final visual descriptions. The
extensive experiments on image and video captioning across four benchmarks and
four languages (i.e., English, Chinese, German, and French) confirm the
effectiveness of our approach. Compared with state-of-the-art zero-shot and
weakly-supervised methods, our method achieves 4.8% and 21.5% absolute
improvements in terms of BLEU@4 and CIDEr metrics. Our code is available at
https://github.com/yangbang18/MultiCapCLIP.
less
By: Yanyan Huang, Weiqin Zhao, Shujun Wang, Yu Fu, Yuming Jiang, Lequan Yu
Whole slide image (WSI) analysis has become increasingly important in the
medical imaging community, enabling automated and objective diagnosis,
prognosis, and therapeutic-response prediction. However, in clinical practice,
the ever-evolving environment hamper the utility of WSI analysis models. In
this paper, we propose the FIRST continual learning framework for WSI analysis,
named ConSlide, to tackle the challenges of enormous image size,... more
Whole slide image (WSI) analysis has become increasingly important in the
medical imaging community, enabling automated and objective diagnosis,
prognosis, and therapeutic-response prediction. However, in clinical practice,
the ever-evolving environment hamper the utility of WSI analysis models. In
this paper, we propose the FIRST continual learning framework for WSI analysis,
named ConSlide, to tackle the challenges of enormous image size, utilization of
hierarchical structure, and catastrophic forgetting by progressive model
updating on multiple sequential datasets. Our framework contains three key
components. The Hierarchical Interaction Transformer (HIT) is proposed to model
and utilize the hierarchical structural knowledge of WSI. The
Breakup-Reorganize (BuRo) rehearsal method is developed for WSI data replay
with efficient region storing buffer and WSI reorganizing operation. The
asynchronous updating mechanism is devised to encourage the network to learn
generic and specific knowledge respectively during the replay stage, based on a
nested cross-scale similarity learning (CSSL) module. We evaluated the proposed
ConSlide on four public WSI datasets from TCGA projects. It performs best over
other state-of-the-art methods with a fair WSI-based continual learning setting
and achieves a better trade-off of the overall performance and forgetting on
previous task
less
Representation Synthesis by Probabilistic Many-Valued Logic Operation in Self-Supervised Learning
0upvotes
By: Hiroki Nakamura, Masashi Okada, Tadahiro Taniguchi
Self-supervised learning (SSL) using mixed images has been studied to learn
various image representations. Existing methods using mixed images learn a
representation by maximizing the similarity between the representation of the
mixed image and the synthesized representation of the original images. However,
few methods consider the synthesis of representations from the perspective of
mathematical logic. In this study, we focused on a synthe... more
Self-supervised learning (SSL) using mixed images has been studied to learn
various image representations. Existing methods using mixed images learn a
representation by maximizing the similarity between the representation of the
mixed image and the synthesized representation of the original images. However,
few methods consider the synthesis of representations from the perspective of
mathematical logic. In this study, we focused on a synthesis method of
representations. We proposed a new SSL with mixed images and a new
representation format based on many-valued logic. This format can indicate the
feature-possession degree, that is, how much of each image feature is possessed
by a representation. This representation format and representation synthesis by
logic operation realize that the synthesized representation preserves the
remarkable characteristics of the original representations. Our method
performed competitively with previous representation synthesis methods for
image classification tasks. We also examined the relationship between the
feature-possession degree and the number of classes of images in the multilabel
image classification dataset to verify that the intended learning was achieved.
In addition, we discussed image retrieval, which is an application of our
proposed representation format using many-valued logic.
less
By: Dingfeng Shi, Qiong Cao, Yujie Zhong, Shan An, Jian Cheng, Haogang Zhu, Dacheng Tao
Temporal action detection (TAD) aims to detect all action boundaries and
their corresponding categories in an untrimmed video. The unclear boundaries of
actions in videos often result in imprecise predictions of action boundaries by
existing methods. To resolve this issue, we propose a one-stage framework named
TriDet. First, we propose a Trident-head to model the action boundary via an
estimated relative probability distribution around the... more
Temporal action detection (TAD) aims to detect all action boundaries and
their corresponding categories in an untrimmed video. The unclear boundaries of
actions in videos often result in imprecise predictions of action boundaries by
existing methods. To resolve this issue, we propose a one-stage framework named
TriDet. First, we propose a Trident-head to model the action boundary via an
estimated relative probability distribution around the boundary. Then, we
analyze the rank-loss problem (i.e. instant discriminability deterioration) in
transformer-based methods and propose an efficient scalable-granularity
perception (SGP) layer to mitigate this issue. To further push the limit of
instant discriminability in the video backbone, we leverage the strong
representation capability of pretrained large models and investigate their
performance on TAD. Last, considering the adequate spatial-temporal context for
classification, we design a decoupled feature pyramid network with separate
feature pyramids to incorporate rich spatial context from the large model for
localization. Experimental results demonstrate the robustness of TriDet and its
state-of-the-art performance on multiple TAD datasets, including hierarchical
(multilabel) TAD datasets.
less
Video Diffusion Models
1upvote
By: Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, David J. Fleet
Generating temporally coherent high fidelity video is an important milestone
in generative modeling research. We make progress towards this milestone by
proposing a diffusion model for video generation that shows very promising
initial results. Our model is a natural extension of the standard image
diffusion architecture, and it enables jointly training from image and video
data, which we find to reduce the variance of minibatch gradients a... more
Generating temporally coherent high fidelity video is an important milestone
in generative modeling research. We make progress towards this milestone by
proposing a diffusion model for video generation that shows very promising
initial results. Our model is a natural extension of the standard image
diffusion architecture, and it enables jointly training from image and video
data, which we find to reduce the variance of minibatch gradients and speed up
optimization. To generate long and higher resolution videos we introduce a new
conditional sampling technique for spatial and temporal video extension that
performs better than previously proposed methods. We present the first results
on a large text-conditioned video generation task, as well as state-of-the-art
results on established benchmarks for video prediction and unconditional video
generation. Supplementary material is available at
https://video-diffusion.github.io/
less
By: Yuxia Geng, Jiaoyan Chen, Zhuo Chen, Zhiquan Ye, Zonggang Yuan, Yantao Jia, Huajun Chen
Zero-shot learning (ZSL) is to handle the prediction of those unseen classes
that have no labeled training data. Recently, generative methods like
Generative Adversarial Networks (GANs) are being widely investigated for ZSL
due to their high accuracy, generalization capability and so on. However, the
side information of classes used now is limited to text descriptions and
attribute annotations, which are in short of semantics of the classes... more
Zero-shot learning (ZSL) is to handle the prediction of those unseen classes
that have no labeled training data. Recently, generative methods like
Generative Adversarial Networks (GANs) are being widely investigated for ZSL
due to their high accuracy, generalization capability and so on. However, the
side information of classes used now is limited to text descriptions and
attribute annotations, which are in short of semantics of the classes. In this
paper, we introduce a new generative ZSL method named KG-GAN by incorporating
rich semantics in a knowledge graph (KG) into GANs. Specifically, we build upon
Graph Neural Networks and encode KG from two views: class view and attribute
view considering the different semantics of KG. With well-learned semantic
embeddings for each node (representing a visual category), we leverage GANs to
synthesize compelling visual features for unseen classes. According to our
evaluation with multiple image classification datasets, KG-GAN can achieve
better performance than the state-of-the-art baselines.
less
By: Laurenz Reichardt, Nikolas Ebert, Oliver Wasenmüller
Deep learning applications on LiDAR data suffer from a strong domain gap when
applied to different sensors or tasks. In order for these methods to obtain
similar accuracy on different data in comparison to values reported on public
benchmarks, a large scale annotated dataset is necessary. However, in practical
applications labeled data is costly and time consuming to obtain. Such factors
have triggered various research in label-efficient me... more
Deep learning applications on LiDAR data suffer from a strong domain gap when
applied to different sensors or tasks. In order for these methods to obtain
similar accuracy on different data in comparison to values reported on public
benchmarks, a large scale annotated dataset is necessary. However, in practical
applications labeled data is costly and time consuming to obtain. Such factors
have triggered various research in label-efficient methods, but a large gap
remains to their fully-supervised counterparts. Thus, we propose ImageTo360, an
effective and streamlined few-shot approach to label-efficient LiDAR
segmentation. Our method utilizes an image teacher network to generate semantic
predictions for LiDAR data within a single camera view. The teacher is used to
pretrain the LiDAR segmentation student network, prior to optional fine-tuning
on 360$^\circ$ data. Our method is implemented in a modular manner on the point
level and as such is generalizable to different architectures. We improve over
the current state-of-the-art results for label-efficient methods and even
surpass some traditional fully-supervised segmentation networks.
less
By: Yiwei Bao, Feng Lu
Although recent deep learning based gaze estimation approaches have achieved
much improvement, we still know little about how gaze features are connected to
the physics of gaze. In this paper, we try to answer this question by analyzing
the gaze feature manifold. Our analysis revealed the insight that the geodesic
distance between gaze features is consistent with the gaze differences between
samples. According to this finding, we construct ... more
Although recent deep learning based gaze estimation approaches have achieved
much improvement, we still know little about how gaze features are connected to
the physics of gaze. In this paper, we try to answer this question by analyzing
the gaze feature manifold. Our analysis revealed the insight that the geodesic
distance between gaze features is consistent with the gaze differences between
samples. According to this finding, we construct the Physics- Consistent
Feature (PCF) in an analytical way, which connects gaze feature to the physical
definition of gaze. We further propose the PCFGaze framework that directly
optimizes gaze feature space by the guidance of PCF. Experimental results
demonstrate that the proposed framework alleviates the overfitting problem and
significantly improves cross-domain gaze estimation accuracy without extra
training data. The insight of gaze feature has the potential to benefit other
regression tasks with physical meanings.
less
By: Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang, Yi Liu
Recently, end-to-end transformer-based detectors (DETRs) have achieved
remarkable performance. However, the issue of the high computational cost of
DETRs has not been effectively addressed, limiting their practical application
and preventing them from fully exploiting the benefits of no post-processing,
such as non-maximum suppression (NMS). In this paper, we first analyze the
influence of NMS in modern real-time object detectors on inferen... more
Recently, end-to-end transformer-based detectors (DETRs) have achieved
remarkable performance. However, the issue of the high computational cost of
DETRs has not been effectively addressed, limiting their practical application
and preventing them from fully exploiting the benefits of no post-processing,
such as non-maximum suppression (NMS). In this paper, we first analyze the
influence of NMS in modern real-time object detectors on inference speed, and
establish an end-to-end speed benchmark. To avoid the inference delay caused by
NMS, we propose a Real-Time DEtection TRansformer (RT-DETR), the first
real-time end-to-end object detector to our best knowledge. Specifically, we
design an efficient hybrid encoder to efficiently process multi-scale features
by decoupling the intra-scale interaction and cross-scale fusion, and propose
IoU-aware query selection to improve the initialization of object queries. In
addition, our proposed detector supports flexibly adjustment of the inference
speed by using different decoder layers without the need for retraining, which
facilitates the practical application of real-time object detectors. Our
RT-DETR-L achieves 53.0% AP on COCO val2017 and 114 FPS on T4 GPU, while
RT-DETR-X achieves 54.8% AP and 74 FPS, outperforming all YOLO detectors of the
same scale in both speed and accuracy. Furthermore, our RT-DETR-R50 achieves
53.1% AP and 108 FPS, outperforming DINO-Deformable-DETR-R50 by 2.2% AP in
accuracy and by about 21 times in FPS. Source code and pretrained models will
be available at PaddleDetection.
less
By: Xin Zhou, Jinghua Hou, Tingting Yao, Dingkang Liang, Zhe Liu, Zhikang Zou, Xiaoqing Ye, Jianwei Cheng, Xiang Bai
3D object detection is an essential task for achieving autonomous driving.
Existing anchor-based detection methods rely on empirical heuristics setting of
anchors, which makes the algorithms lack elegance. In recent years, we have
witnessed the rise of several generative models, among which diffusion models
show great potential for learning the transformation of two distributions. Our
proposed Diff3Det migrates the diffusion model to propos... more
3D object detection is an essential task for achieving autonomous driving.
Existing anchor-based detection methods rely on empirical heuristics setting of
anchors, which makes the algorithms lack elegance. In recent years, we have
witnessed the rise of several generative models, among which diffusion models
show great potential for learning the transformation of two distributions. Our
proposed Diff3Det migrates the diffusion model to proposal generation for 3D
object detection by considering the detection boxes as generative targets.
During training, the object boxes diffuse from the ground truth boxes to the
Gaussian distribution, and the decoder learns to reverse this noise process. In
the inference stage, the model progressively refines a set of random boxes to
the prediction results. We provide detailed experiments on the KITTI benchmark
and achieve promising performance compared to classical anchor-based 3D
detection methods.
less
By: Cathrine Damgaard, Trine Naja Eriksen, Dovile Juodelyte, Veronika Cheplygina, Amelia Jiménez-Sánchez
The advancement of machine learning algorithms in medical image analysis
requires the expansion of training datasets. A popular and cost-effective
approach is automated annotation extraction from free-text medical reports,
primarily due to the high costs associated with expert clinicians annotating
chest X-ray images. However, it has been shown that the resulting datasets are
susceptible to biases and shortcuts. Another strategy to increase... more
The advancement of machine learning algorithms in medical image analysis
requires the expansion of training datasets. A popular and cost-effective
approach is automated annotation extraction from free-text medical reports,
primarily due to the high costs associated with expert clinicians annotating
chest X-ray images. However, it has been shown that the resulting datasets are
susceptible to biases and shortcuts. Another strategy to increase the size of a
dataset is crowdsourcing, a widely adopted practice in general computer vision
with some success in medical image analysis. In a similar vein to
crowdsourcing, we enhance two publicly available chest X-ray datasets by
incorporating non-expert annotations. However, instead of using diagnostic
labels, we annotate shortcuts in the form of tubes. We collect 3.5k chest drain
annotations for CXR14, and 1k annotations for 4 different tube types in
PadChest. We train a chest drain detector with the non-expert annotations that
generalizes well to expert labels. Moreover, we compare our annotations to
those provided by experts and show "moderate" to "almost perfect" agreement.
Finally, we present a pathology agreement study to raise awareness about ground
truth annotations. We make our annotations and code available.
less
By: Dongming Wu, Wencheng Han, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, Jianbing Shen
A new trend in the computer vision community is to capture objects of
interest following flexible human command represented by a natural language
prompt. However, the progress of using language prompts in driving scenarios is
stuck in a bottleneck due to the scarcity of paired prompt-instance data. To
address this challenge, we propose the first object-centric language prompt set
for driving scenes within 3D, multi-view, and multi-frame spa... more
A new trend in the computer vision community is to capture objects of
interest following flexible human command represented by a natural language
prompt. However, the progress of using language prompts in driving scenarios is
stuck in a bottleneck due to the scarcity of paired prompt-instance data. To
address this challenge, we propose the first object-centric language prompt set
for driving scenes within 3D, multi-view, and multi-frame space, named
NuPrompt. It expands Nuscenes dataset by constructing a total of 35,367
language descriptions, each referring to an average of 5.3 object tracks. Based
on the object-text pairs from the new benchmark, we formulate a new
prompt-based driving task, \ie, employing a language prompt to predict the
described object trajectory across views and frames. Furthermore, we provide a
simple end-to-end baseline model based on Transformer, named PromptTrack.
Experiments show that our PromptTrack achieves impressive performance on
NuPrompt. We hope this work can provide more new insights for the autonomous
driving community. Dataset and Code will be made public at
\href{https://github.com/wudongming97/Prompt4Driving}{https://github.com/wudongming97/Prompt4Driving}.
less
By: Jordan A. James, Heather K. Manching, Matthew R. Mattia, Kim D. Bowman, Amanda M. Hulse-Kemp, William J. Beksi
In this letter, we present a new dataset to advance the state of the art in
detecting citrus fruit and accurately estimate yield on trees affected by the
Huanglongbing (HLB) disease in orchard environments via imaging. Despite the
fact that significant progress has been made in solving the fruit detection
problem, the lack of publicly available datasets has complicated direct
comparison of results. For instance, citrus detection has long be... more
In this letter, we present a new dataset to advance the state of the art in
detecting citrus fruit and accurately estimate yield on trees affected by the
Huanglongbing (HLB) disease in orchard environments via imaging. Despite the
fact that significant progress has been made in solving the fruit detection
problem, the lack of publicly available datasets has complicated direct
comparison of results. For instance, citrus detection has long been of interest
in the agricultural research community, yet there is an absence of work,
particularly involving public datasets of citrus affected by HLB. To address
this issue, we enhance state-of-the-art object detection methods for use in
typical orchard settings. Concretely, we provide high-resolution images of
citrus trees located in an area known to be highly affected by HLB, along with
high-quality bounding box annotations of citrus fruit. Fruit on both the trees
and the ground are labeled to allow for identification of fruit location, which
contributes to advancements in yield estimation and potential measure of HLB
impact via fruit drop. The dataset consists of over 32,000 bounding box
annotations for fruit instances contained in 579 high-resolution images. In
summary, our contributions are the following: (i) we introduce a novel dataset
along with baseline performance benchmarks on multiple contemporary object
detection algorithms, (ii) we show the ability to accurately capture fruit
location on tree or on ground, and finally (ii) we present a correlation of our
results with yield estimations.
less
FlowIBR: Leveraging Pre-Training for Efficient Neural Image-Based Rendering of Dynamic Scenes
0upvotes
By: Marcel Büsching, Josef Bengtson, David Nilsson, Mårten Björkman
We introduce a novel approach for monocular novel view synthesis of dynamic
scenes. Existing techniques already show impressive rendering quality but tend
to focus on optimization within a single scene without leveraging prior
knowledge. This limitation has been primarily attributed to the lack of
datasets of dynamic scenes available for training and the diversity of scene
dynamics. Our method FlowIBR circumvents these issues by integrating... more
We introduce a novel approach for monocular novel view synthesis of dynamic
scenes. Existing techniques already show impressive rendering quality but tend
to focus on optimization within a single scene without leveraging prior
knowledge. This limitation has been primarily attributed to the lack of
datasets of dynamic scenes available for training and the diversity of scene
dynamics. Our method FlowIBR circumvents these issues by integrating a neural
image-based rendering method, pre-trained on a large corpus of widely available
static scenes, with a per-scene optimized scene flow field. Utilizing this flow
field, we bend the camera rays to counteract the scene dynamics, thereby
presenting the dynamic scene as if it were static to the rendering network. The
proposed method reduces per-scene optimization time by an order of magnitude,
achieving comparable results to existing methods - all on a single
consumer-grade GPU.
less
By: Sander Riisøen Jyhne, Per-Arne Andersen, Morten Goodwin
This paper presents Deep Networks for Improved Segmentation Edges (DeNISE), a
novel data enhancement technique using edge detection and segmentation models
to improve the boundary quality of segmentation masks. DeNISE utilizes the
inherent differences in two sequential deep neural architectures to improve the
accuracy of the predicted segmentation edge. DeNISE applies to all types of
neural networks and is not trained end-to-end, allowing r... more
This paper presents Deep Networks for Improved Segmentation Edges (DeNISE), a
novel data enhancement technique using edge detection and segmentation models
to improve the boundary quality of segmentation masks. DeNISE utilizes the
inherent differences in two sequential deep neural architectures to improve the
accuracy of the predicted segmentation edge. DeNISE applies to all types of
neural networks and is not trained end-to-end, allowing rapid experiments to
discover which models complement each other. We test and apply DeNISE for
building segmentation in aerial images. Aerial images are known for difficult
conditions as they have a low resolution with optical noise, such as
reflections, shadows, and visual obstructions. Overall the paper demonstrates
the potential for DeNISE. Using the technique, we improve the baseline results
with a building IoU of 78.9%.
less
Multimodal Contrastive Learning and Tabular Attention for Automated Alzheimer's Disease Prediction
0upvotes
By: Weichen Huang
Alongside neuroimaging such as MRI scans and PET, Alzheimer's disease (AD)
datasets contain valuable tabular data including AD biomarkers and clinical
assessments. Existing computer vision approaches struggle to utilize this
additional information. To address these needs, we propose a generalizable
framework for multimodal contrastive learning of image data and tabular data, a
novel tabular attention module for amplifying and ranking salien... more
Alongside neuroimaging such as MRI scans and PET, Alzheimer's disease (AD)
datasets contain valuable tabular data including AD biomarkers and clinical
assessments. Existing computer vision approaches struggle to utilize this
additional information. To address these needs, we propose a generalizable
framework for multimodal contrastive learning of image data and tabular data, a
novel tabular attention module for amplifying and ranking salient features in
tables, and the application of these techniques onto Alzheimer's disease
prediction. Experimental evaulations demonstrate the strength of our framework
by detecting Alzheimer's disease (AD) from over 882 MR image slices from the
ADNI database. We take advantage of the high interpretability of tabular data
and our novel tabular attention approach and through attribution of the
attention scores for each row of the table, we note and rank the most
predominant features. Results show that the model is capable of an accuracy of
over 83.8%, almost a 10% increase from previous state of the art.
less
By: Oana Ignat, Santiago Castro, Weiji Li, Rada Mihalcea
We introduce the task of automatic human action co-occurrence identification,
i.e., determine whether two human actions can co-occur in the same interval of
time. We create and make publicly available the ACE (Action Co-occurrencE)
dataset, consisting of a large graph of ~12k co-occurring pairs of visual
actions and their corresponding video clips. We describe graph link prediction
models that leverage visual and textual information to auto... more
We introduce the task of automatic human action co-occurrence identification,
i.e., determine whether two human actions can co-occur in the same interval of
time. We create and make publicly available the ACE (Action Co-occurrencE)
dataset, consisting of a large graph of ~12k co-occurring pairs of visual
actions and their corresponding video clips. We describe graph link prediction
models that leverage visual and textual information to automatically infer if
two actions are co-occurring. We show that graphs are particularly well suited
to capture relations between human actions, and the learned graph
representations are effective for our task and capture novel and relevant
information across different data domains. The ACE dataset and the code
introduced in this paper are publicly available at
https://github.com/MichiganNLP/vlog_action_co-occurrence.
less
By: Mahendran N
Convolutional neural networks have spatial representations which read
patterns in the vision tasks. Squeeze and excitation links the channel wise
representations by explicitly modeling on channel level. Multi layer
perceptrons learn global representations and in most of the models it is used
often at the end after all convolutional layers to gather all the information
learned before classification. We propose a method of inducing the global... more
Convolutional neural networks have spatial representations which read
patterns in the vision tasks. Squeeze and excitation links the channel wise
representations by explicitly modeling on channel level. Multi layer
perceptrons learn global representations and in most of the models it is used
often at the end after all convolutional layers to gather all the information
learned before classification. We propose a method of inducing the global
representations within channels to have better performance of the model. We
propose SaEnet, Squeeze aggregated excitation network, for learning global
channelwise representation in between layers. The proposed module takes
advantage of passing important information after squeeze by having aggregated
excitation before regaining its shape. We also introduce a new idea of having a
multibranch linear(dense) layer in the network. This learns global
representations from the condensed information which enhances the
representational power of the network. The proposed module have undergone
extensive experiments by using Imagenet and CIFAR100 datasets and compared with
closely related architectures. The analyzes results that proposed models
outputs are comparable and in some cases better than existing state of the art
architectures.
less
By: Zhitong Gao, Shipeng Yan, Xuming He
Recent advancements in dense out-of-distribution (OOD) detection have
primarily focused on scenarios where the training and testing datasets share a
similar domain, with the assumption that no domain shift exists between them.
However, in real-world situations, domain shift often exits and significantly
affects the accuracy of existing out-of-distribution (OOD) detection models. In
this work, we propose a dual-level OOD detection framework ... more
Recent advancements in dense out-of-distribution (OOD) detection have
primarily focused on scenarios where the training and testing datasets share a
similar domain, with the assumption that no domain shift exists between them.
However, in real-world situations, domain shift often exits and significantly
affects the accuracy of existing out-of-distribution (OOD) detection models. In
this work, we propose a dual-level OOD detection framework to handle domain
shift and semantic shift jointly. The first level distinguishes whether domain
shift exists in the image by leveraging global low-level features, while the
second level identifies pixels with semantic shift by utilizing dense
high-level feature maps. In this way, we can selectively adapt the model to
unseen domains as well as enhance model's capacity in detecting novel classes.
We validate the efficacy of our proposed method on several OOD segmentation
benchmarks, including those with significant domain shifts and those without,
observing consistent performance improvements across various baseline models.
less
By: Sebastian Schmidt BMW and TUM, Stephan Günnemann TUM
Active learning (AL) reduces the amount of labeled data needed to train a
machine learning model by intelligently choosing which instances to label.
Classic pool-based AL requires all data to be present in a datacenter, which
can be challenging with the increasing amounts of data needed in deep learning.
However, AL on mobile devices and robots, like autonomous cars, can filter the
data from perception sensor streams before reaching the dat... more
Active learning (AL) reduces the amount of labeled data needed to train a
machine learning model by intelligently choosing which instances to label.
Classic pool-based AL requires all data to be present in a datacenter, which
can be challenging with the increasing amounts of data needed in deep learning.
However, AL on mobile devices and robots, like autonomous cars, can filter the
data from perception sensor streams before reaching the datacenter. We
exploited the temporal properties for such image streams in our work and
proposed the novel temporal predicted loss (TPL) method. To evaluate the
stream-based setting properly, we introduced the GTA V streets and the A2D2
streets dataset and made both publicly available. Our experiments showed that
our approach significantly improves the diversity of the selection while being
an uncertainty-based method. As pool-based approaches are more common in
perception applications, we derived a concept for comparing pool-based and
stream-based AL, where TPL out-performed state-of-the-art pool- or stream-based
approaches for different models. TPL demonstrated a gain of 2.5 precept points
(pp) less required data while being significantly faster than pool-based
methods.
less
By: Fang Liu, Yuhao Liu, Yuqiu Kong, Ke Xu, Lihe Zhang, Baocai Yin, Gerhard Hancke, Rynson Lau
Existing Referring Image Segmentation (RIS) methods typically require
expensive pixel-level or box-level annotations for supervision. In this paper,
we observe that the referring texts used in RIS already provide sufficient
information to localize the target object. Hence, we propose a novel
weakly-supervised RIS framework to formulate the target localization problem as
a classification process to differentiate between positive and negative... more
Existing Referring Image Segmentation (RIS) methods typically require
expensive pixel-level or box-level annotations for supervision. In this paper,
we observe that the referring texts used in RIS already provide sufficient
information to localize the target object. Hence, we propose a novel
weakly-supervised RIS framework to formulate the target localization problem as
a classification process to differentiate between positive and negative text
expressions. While the referring text expressions for an image are used as
positive expressions, the referring text expressions from other images can be
used as negative expressions for this image. Our framework has three main
novelties. First, we propose a bilateral prompt method to facilitate the
classification process, by harmonizing the domain discrepancy between visual
and linguistic features. Second, we propose a calibration method to reduce
noisy background information and improve the correctness of the response maps
for target object localization. Third, we propose a positive response map
selection strategy to generate high-quality pseudo-labels from the enhanced
response maps, for training a segmentation network for RIS inference. For
evaluation, we propose a new metric to measure localization accuracy.
Experiments on four benchmarks show that our framework achieves promising
performances to existing fully-supervised RIS methods while outperforming
state-of-the-art weakly-supervised methods adapted from related areas. Code is
available at https://github.com/fawnliu/TRIS.
less
By: Wenze Liu, Hao Lu, Hongtao Fu, Zhiguo Cao
We present DySample, an ultra-lightweight and effective dynamic upsampler.
While impressive performance gains have been witnessed from recent kernel-based
dynamic upsamplers such as CARAFE, FADE, and SAPA, they introduce much
workload, mostly due to the time-consuming dynamic convolution and the
additional sub-network used to generate dynamic kernels. Further, the need for
high-res feature guidance of FADE and SAPA somehow limits their appl... more
We present DySample, an ultra-lightweight and effective dynamic upsampler.
While impressive performance gains have been witnessed from recent kernel-based
dynamic upsamplers such as CARAFE, FADE, and SAPA, they introduce much
workload, mostly due to the time-consuming dynamic convolution and the
additional sub-network used to generate dynamic kernels. Further, the need for
high-res feature guidance of FADE and SAPA somehow limits their application
scenarios. To address these concerns, we bypass dynamic convolution and
formulate upsampling from the perspective of point sampling, which is more
resource-efficient and can be easily implemented with the standard built-in
function in PyTorch. We first showcase a naive design, and then demonstrate how
to strengthen its upsampling behavior step by step towards our new upsampler,
DySample. Compared with former kernel-based dynamic upsamplers, DySample
requires no customized CUDA package and has much fewer parameters, FLOPs, GPU
memory, and latency. Besides the light-weight characteristics, DySample
outperforms other upsamplers across five dense prediction tasks, including
semantic segmentation, object detection, instance segmentation, panoptic
segmentation, and monocular depth estimation. Code is available at
https://github.com/tiny-smart/dysample.
less
By: Eklavya Sarkar, Pavel Korshunov, Laurent Colbois, Sébastien Marcel
Morphing attacks are a threat to biometric systems where the biometric reference in an identity document can be altered. This form of attack presents an important issue in applications relying on identity documents such as border security or access control. Research in generation of face morphs and their detection is developing rapidly, however very few datasets with morphing attacks and open-source detection toolkits are publicly available. ... more
Morphing attacks are a threat to biometric systems where the biometric reference in an identity document can be altered. This form of attack presents an important issue in applications relying on identity documents such as border security or access control. Research in generation of face morphs and their detection is developing rapidly, however very few datasets with morphing attacks and open-source detection toolkits are publicly available. This paper bridges this gap by providing two datasets and the corresponding code for four types of morphing attacks: two that rely on facial landmarks based on OpenCV and FaceMorpher, and two that use StyleGAN 2 to generate synthetic morphs. We also conduct extensive experiments to assess the vulnerability of four state-of-the-art face recognition systems, including FaceNet, VGG-Face, ArcFace, and ISV. Surprisingly, the experiments demonstrate that, although visually more appealing, morphs based on StyleGAN 2 do not pose a significant threat to the state to face recognition systems, as these morphs were outmatched by the simple morphs that are based facial landmarks. less
By: Yuxiang Yang, Yingqi Deng, Jiahao Nie, Jing Zhang
3D single object tracking (SOT) in point clouds is still a challenging
problem due to appearance variation, distractors, and high sparsity of point
clouds. Notably, in autonomous driving scenarios, the target object typically
maintains spatial adjacency across consecutive frames, predominantly moving
horizontally. This spatial continuity offers valuable prior knowledge for
target localization. However, existing trackers, which often employ ... more
3D single object tracking (SOT) in point clouds is still a challenging
problem due to appearance variation, distractors, and high sparsity of point
clouds. Notably, in autonomous driving scenarios, the target object typically
maintains spatial adjacency across consecutive frames, predominantly moving
horizontally. This spatial continuity offers valuable prior knowledge for
target localization. However, existing trackers, which often employ point-wise
representations, struggle to efficiently utilize this knowledge owing to the
irregular format of such representations. Consequently, they require elaborate
designs and solving multiple subtasks to establish spatial correspondence. In
this paper, we introduce BEVTrack, a simple yet strong baseline framework for
3D SOT. After converting consecutive point clouds into the common
Bird's-Eye-View representation, BEVTrack inherently encodes spatial proximity
and adeptly captures motion cues for tracking via a simple element-wise
operation and convolutional layers. Additionally, to better deal with objects
having diverse sizes and moving patterns, BEVTrack directly learns the
underlying motion distribution rather than making a fixed Laplacian or Gaussian
assumption as in previous works. Without bells and whistles, BEVTrack achieves
state-of-the-art performance on KITTI and NuScenes datasets while maintaining a
high inference speed of 122 FPS. The code will be released at
https://github.com/xmm-prio/BEVTrack.
less
By: Anand Sharma MIET, Meerut, A. G. Ramakrishnan IISc, Bengaluru
A set of features independent of character stroke direction and order
variations is proposed for online handwritten character recognition. A method
is developed that maps features like co-ordinates of points, orientations of
strokes at points, and dynamics of orientations of strokes at points spatially
as a function of co-ordinate values of the points and computes histograms of
these features from different regions in the spatial map.
Dif... more
A set of features independent of character stroke direction and order
variations is proposed for online handwritten character recognition. A method
is developed that maps features like co-ordinates of points, orientations of
strokes at points, and dynamics of orientations of strokes at points spatially
as a function of co-ordinate values of the points and computes histograms of
these features from different regions in the spatial map.
Different features like spatio-temporal, discrete Fourier transform, discrete
cosine transform, discrete wavelet transform, spatial, and histograms of
oriented gradients used in other studies for training classifiers for character
recognition are considered. The classifier chosen for classification
performance comparison, when trained with different features, is support vector
machines (SVM).
The character datasets used for training and testing the classifiers consist
of online handwritten samples of 96 different Hindi characters. There are 12832
and 2821 samples in training and testing datasets, respectively.
SVM classifiers trained with the proposed features has the highest
classification accuracy of 92.9\% when compared to the performances of SVM
classifiers trained with the other features and tested on the same testing
dataset. Therefore, the proposed features have better character discriminative
capability than the other features considered for comparison.
less
By: Fanda Fan, Chaoxu Guo, Litong Gong, Biao Wang, Tiezheng Ge, Yuning Jiang, Chunjie Luo, Jianfeng Zhan
Video outpainting aims to adequately complete missing areas at the edges of
video frames. Compared to image outpainting, it presents an additional
challenge as the model should maintain the temporal consistency of the filled
area. In this paper, we introduce a masked 3D diffusion model for video
outpainting. We use the technique of mask modeling to train the 3D diffusion
model. This allows us to use multiple guide frames to connect the resu... more
Video outpainting aims to adequately complete missing areas at the edges of
video frames. Compared to image outpainting, it presents an additional
challenge as the model should maintain the temporal consistency of the filled
area. In this paper, we introduce a masked 3D diffusion model for video
outpainting. We use the technique of mask modeling to train the 3D diffusion
model. This allows us to use multiple guide frames to connect the results of
multiple video clip inferences, thus ensuring temporal consistency and reducing
jitter between adjacent frames. Meanwhile, we extract the global frames of the
video as prompts and guide the model to obtain information other than the
current video clip using cross-attention. We also introduce a hybrid
coarse-to-fine inference pipeline to alleviate the artifact accumulation
problem. The existing coarse-to-fine pipeline only uses the infilling strategy,
which brings degradation because the time interval of the sparse frames is too
large. Our pipeline benefits from bidirectional learning of the mask modeling
and thus can employ a hybrid strategy of infilling and interpolation when
generating sparse frames. Experiments show that our method achieves
state-of-the-art results in video outpainting tasks. More results are provided
at our https://fanfanda.github.io/M3DDM/.
less
By: Mohamed L. Mekhalfi, Davide Boscaini, Fabio Poiesi
Unsupervised domain adaptation (UDA) plays a crucial role in object detection
when adapting a source-trained detector to a target domain without annotated
data. In this paper, we propose a novel and effective four-step UDA approach
that leverages self-supervision and trains source and target data concurrently.
We harness self-supervised learning to mitigate the lack of ground truth in the
target domain. Our method consists of the following ... more
Unsupervised domain adaptation (UDA) plays a crucial role in object detection
when adapting a source-trained detector to a target domain without annotated
data. In this paper, we propose a novel and effective four-step UDA approach
that leverages self-supervision and trains source and target data concurrently.
We harness self-supervised learning to mitigate the lack of ground truth in the
target domain. Our method consists of the following steps: (1) identify the
region with the highest-confidence set of detections in each target image,
which serve as our pseudo-labels; (2) crop the identified region and generate a
collection of its augmented versions; (3) combine these latter into a composite
image; (4) adapt the network to the target domain using the composed image.
Through extensive experiments under cross-camera, cross-weather, and
synthetic-to-real scenarios, our approach achieves state-of-the-art
performance, improving upon the nearest competitor by more than 2% in terms of
mean Average Precision (mAP). The code is available at
https://github.com/MohamedTEV/DACA.
less
By: Xiangtao Wang, Ruizhi Wang, Jie Zhou, Thomas Lukasiewicz, Zhenghua Xu
Self-supervised masked image modeling has shown promising results on natural
images. However, directly applying such methods to medical images remains
challenging. This difficulty stems from the complexity and distinct
characteristics of lesions compared to natural images, which impedes effective
representation learning. Additionally, conventional high fixed masking ratios
restrict reconstructing fine lesion details, limiting the scope of l... more
Self-supervised masked image modeling has shown promising results on natural
images. However, directly applying such methods to medical images remains
challenging. This difficulty stems from the complexity and distinct
characteristics of lesions compared to natural images, which impedes effective
representation learning. Additionally, conventional high fixed masking ratios
restrict reconstructing fine lesion details, limiting the scope of learnable
information. To tackle these limitations, we propose a novel self-supervised
medical image segmentation framework, Adaptive Masking Lesion Patches (AMLP).
Specifically, we design a Masked Patch Selection (MPS) strategy to identify and
focus learning on patches containing lesions. Lesion regions are scarce yet
critical, making their precise reconstruction vital. To reduce
misclassification of lesion and background patches caused by unsupervised
clustering in MPS, we introduce an Attention Reconstruction Loss (ARL) to focus
on hard-to-reconstruct patches likely depicting lesions. We further propose a
Category Consistency Loss (CCL) to refine patch categorization based on
reconstruction difficulty, strengthening distinction between lesions and
background. Moreover, we develop an Adaptive Masking Ratio (AMR) strategy that
gradually increases the masking ratio to expand reconstructible information and
improve learning. Extensive experiments on two medical segmentation datasets
demonstrate AMLP's superior performance compared to existing self-supervised
approaches. The proposed strategies effectively address limitations in applying
masked modeling to medical images, tailored to capturing fine lesion details
vital for segmentation tasks.
less
By: Lucas Ventura, Antoine Yang, Cordelia Schmid, Gül Varol
Composed Image Retrieval (CoIR) has recently gained popularity as a task that
considers both text and image queries together, to search for relevant images
in a database. Most CoIR approaches require manually annotated datasets,
comprising image-text-image triplets, where the text describes a modification
from the query image to the target image. However, manual curation of CoIR
triplets is expensive and prevents scalability. In this work, ... more
Composed Image Retrieval (CoIR) has recently gained popularity as a task that
considers both text and image queries together, to search for relevant images
in a database. Most CoIR approaches require manually annotated datasets,
comprising image-text-image triplets, where the text describes a modification
from the query image to the target image. However, manual curation of CoIR
triplets is expensive and prevents scalability. In this work, we instead
propose a scalable automatic dataset creation methodology that generates
triplets given video-caption pairs, while also expanding the scope of the task
to include composed video retrieval (CoVR). To this end, we mine paired videos
with a similar caption from a large database, and leverage a large language
model to generate the corresponding modification text. Applying this
methodology to the extensive WebVid2M collection, we automatically construct
our WebVid-CoVR dataset, resulting in 1.6 million triplets. Moreover, we
introduce a new benchmark for CoVR with a manually annotated evaluation set,
along with baseline results. Our experiments further demonstrate that training
a CoVR model on our dataset effectively transfers to CoIR, leading to improved
state-of-the-art performance in the zero-shot setup on both the CIRR and
FashionIQ benchmarks. Our code, datasets, and models are publicly available at
https://imagine.enpc.fr/~ventural/covr.
less
By: Abhijay Ghildyal, Feng Liu
Existing perceptual similarity metrics assume an image and its reference are
well aligned. As a result, these metrics are often sensitive to a small
alignment error that is imperceptible to the human eyes. This paper studies the
effect of small misalignment, specifically a small shift between the input and
reference image, on existing metrics, and accordingly develops a shift-tolerant
similarity metric. This paper builds upon LPIPS, a widel... more
Existing perceptual similarity metrics assume an image and its reference are
well aligned. As a result, these metrics are often sensitive to a small
alignment error that is imperceptible to the human eyes. This paper studies the
effect of small misalignment, specifically a small shift between the input and
reference image, on existing metrics, and accordingly develops a shift-tolerant
similarity metric. This paper builds upon LPIPS, a widely used learned
perceptual similarity metric, and explores architectural design considerations
to make it robust against imperceptible misalignment. Specifically, we study a
wide spectrum of neural network elements, such as anti-aliasing filtering,
pooling, striding, padding, and skip connection, and discuss their roles in
making a robust metric. Based on our studies, we develop a new deep neural
network-based perceptual similarity metric. Our experiments show that our
metric is tolerant to imperceptible shifts while being consistent with the
human similarity judgment.
less
Unlocking Fine-Grained Details with Wavelet-based High-Frequency Enhancement in Transformers
0upvotes
By: Reza Azad, Amirhossein Kazerouni, Alaa Sulaiman, Afshin Bozorgpour, Ehsan Khodapanah Aghdam, Abin Jose, Dorit Merhof
Medical image segmentation is a critical task that plays a vital role in
diagnosis, treatment planning, and disease monitoring. Accurate segmentation of
anatomical structures and abnormalities from medical images can aid in the
early detection and treatment of various diseases. In this paper, we address
the local feature deficiency of the Transformer model by carefully re-designing
the self-attention map to produce accurate dense prediction... more
Medical image segmentation is a critical task that plays a vital role in
diagnosis, treatment planning, and disease monitoring. Accurate segmentation of
anatomical structures and abnormalities from medical images can aid in the
early detection and treatment of various diseases. In this paper, we address
the local feature deficiency of the Transformer model by carefully re-designing
the self-attention map to produce accurate dense prediction in medical images.
To this end, we first apply the wavelet transformation to decompose the input
feature map into low-frequency (LF) and high-frequency (HF) subbands. The LF
segment is associated with coarse-grained features while the HF components
preserve fine-grained features such as texture and edge information. Next, we
reformulate the self-attention operation using the efficient Transformer to
perform both spatial and context attention on top of the frequency
representation. Furthermore, to intensify the importance of the boundary
information, we impose an additional attention map by creating a Gaussian
pyramid on top of the HF components. Moreover, we propose a multi-scale context
enhancement block within skip connections to adaptively model inter-scale
dependencies to overcome the semantic gap among stages of the encoder and
decoder modules. Throughout comprehensive experiments, we demonstrate the
effectiveness of our strategy on multi-organ and skin lesion segmentation
benchmarks. The implementation code will be available upon acceptance.
\href{https://github.com/mindflow-institue/WaveFormer}{GitHub}.
less
By: Shuxiao Ma, Linyuan Wang, Bin Yan
Biological research has revealed that the verbal semantic information in the
brain cortex, as an additional source, participates in nonverbal semantic
tasks, such as visual encoding. However, previous visual encoding models did
not incorporate verbal semantic information, contradicting this biological
finding. This paper proposes a multimodal visual information encoding network
model based on stimulus images and associated textual informati... more
Biological research has revealed that the verbal semantic information in the
brain cortex, as an additional source, participates in nonverbal semantic
tasks, such as visual encoding. However, previous visual encoding models did
not incorporate verbal semantic information, contradicting this biological
finding. This paper proposes a multimodal visual information encoding network
model based on stimulus images and associated textual information in response
to this issue. Our visual information encoding network model takes stimulus
images as input and leverages textual information generated by a text-image
generation model as verbal semantic information. This approach injects new
information into the visual encoding model. Subsequently, a Transformer network
aligns image and text feature information, creating a multimodal feature space.
A convolutional network then maps from this multimodal feature space to voxel
space, constructing the multimodal visual information encoding network model.
Experimental results demonstrate that the proposed multimodal visual
information encoding network model outperforms previous models under the exact
training cost. In voxel prediction of the left hemisphere of subject 1's brain,
the performance improves by approximately 15.87%, while in the right
hemisphere, the performance improves by about 4.6%. The multimodal visual
encoding network model exhibits superior encoding performance. Additionally,
ablation experiments indicate that our proposed model better simulates the
brain's visual information processing.
less
By: Lin Geng Foo, Jia Gong, Hossein Rahmani, Jun Liu
Recovering a 3D human mesh from a single RGB image is a challenging task due
to depth ambiguity and self-occlusion, resulting in a high degree of
uncertainty. Meanwhile, diffusion models have recently seen much success in
generating high-quality outputs by progressively denoising noisy inputs.
Inspired by their capability, we explore a diffusion-based approach for human
mesh recovery, and propose a Human Mesh Diffusion (HMDiff) framework wh... more
Recovering a 3D human mesh from a single RGB image is a challenging task due
to depth ambiguity and self-occlusion, resulting in a high degree of
uncertainty. Meanwhile, diffusion models have recently seen much success in
generating high-quality outputs by progressively denoising noisy inputs.
Inspired by their capability, we explore a diffusion-based approach for human
mesh recovery, and propose a Human Mesh Diffusion (HMDiff) framework which
frames mesh recovery as a reverse diffusion process. We also propose a
Distribution Alignment Technique (DAT) that injects input-specific distribution
information into the diffusion process, and provides useful prior knowledge to
simplify the mesh recovery task. Our method achieves state-of-the-art
performance on three widely used datasets. Project page:
https://gongjia0208.github.io/HMDiff/.
less
By: Yixuan Zhou, Xing Xu, Jingkuan Song, Fumin Shen, Heng Tao Shen
Unsupervised anomaly detection (UAD) attracts a lot of research interest and
drives widespread applications, where only anomaly-free samples are available
for training. Some UAD applications intend to further locate the anomalous
regions without any anomaly information.
Although the absence of anomalous samples and annotations deteriorates the
UAD performance, an inconspicuous yet powerful statistics model, the
normalizing flows, is appro... more
Unsupervised anomaly detection (UAD) attracts a lot of research interest and
drives widespread applications, where only anomaly-free samples are available
for training. Some UAD applications intend to further locate the anomalous
regions without any anomaly information.
Although the absence of anomalous samples and annotations deteriorates the
UAD performance, an inconspicuous yet powerful statistics model, the
normalizing flows, is appropriate for anomaly detection and localization in an
unsupervised fashion. The flow-based probabilistic models, only trained on
anomaly-free data, can efficiently distinguish unpredictable anomalies by
assigning them much lower likelihoods than normal data.
Nevertheless, the size variation of unpredictable anomalies introduces
another inconvenience to the flow-based methods for high-precision anomaly
detection and localization. To generalize the anomaly size variation, we
propose a novel Multi-Scale Flow-based framework dubbed MSFlow composed of
asymmetrical parallel flows followed by a fusion flow to exchange multi-scale
perceptions. Moreover, different multi-scale aggregation strategies are adopted
for image-wise anomaly detection and pixel-wise anomaly localization according
to the discrepancy between them. The proposed MSFlow is evaluated on three
anomaly detection datasets, significantly outperforming existing methods.
Notably, on the challenging MVTec AD benchmark, our MSFlow achieves a new
state-of-the-art with a detection AUORC score of up to 99.7%, localization
AUCROC score of 98.8%, and PRO score of 97.1%. The reproducible code is
available at https://github.com/cool-xuan/msflow.
less
By: Brandon Zhao, Aviad Levis, Liam Connor, Pratul P. Srinivasan, Katherine L. Bouman
Refractive Index Tomography is an inverse problem in which we seek to
reconstruct a scene's 3D refractive field from 2D projected image measurements.
The refractive field is not visible itself, but instead affects how the path of
a light ray is continuously curved as it travels through space. Refractive
fields appear across a wide variety of scientific applications, from
translucent cell samples in microscopy to fields of dark matter bendin... more
Refractive Index Tomography is an inverse problem in which we seek to
reconstruct a scene's 3D refractive field from 2D projected image measurements.
The refractive field is not visible itself, but instead affects how the path of
a light ray is continuously curved as it travels through space. Refractive
fields appear across a wide variety of scientific applications, from
translucent cell samples in microscopy to fields of dark matter bending light
from faraway galaxies. This problem poses a unique challenge because the
refractive field directly affects the path that light takes, making its
recovery a non-linear problem. In addition, in contrast with traditional
tomography, we seek to recover the refractive field using a projected image
from only a single viewpoint by leveraging knowledge of light sources scattered
throughout the medium. In this work, we introduce a method that uses a
coordinate-based neural network to model the underlying continuous refractive
field in a scene. We then use explicit modeling of rays' 3D spatial curvature
to optimize the parameters of this network, reconstructing refractive fields
with an analysis-by-synthesis approach. The efficacy of our approach is
demonstrated by recovering refractive fields in simulation, and analyzing how
recovery is affected by the light source distribution. We then test our method
on a simulated dark matter mapping problem, where we recover the refractive
field underlying a realistic simulated dark matter distribution.
less
Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations
0upvotes
By: Yu-Hui Chen, Raman Sarokin, Juhyun Lee, Jiuqiang Tang, Chuo-Ling Chang, Andrei Kulik, Matthias Grundmann
The rapid development and application of foundation models have
revolutionized the field of artificial intelligence. Large diffusion models
have gained significant attention for their ability to generate photorealistic
images and support various tasks. On-device deployment of these models provides
benefits such as lower server costs, offline functionality, and improved user
privacy. However, common large diffusion models have over 1 billion... more
The rapid development and application of foundation models have
revolutionized the field of artificial intelligence. Large diffusion models
have gained significant attention for their ability to generate photorealistic
images and support various tasks. On-device deployment of these models provides
benefits such as lower server costs, offline functionality, and improved user
privacy. However, common large diffusion models have over 1 billion parameters
and pose challenges due to restricted computational and memory resources on
devices. We present a series of implementation optimizations for large
diffusion models that achieve the fastest reported inference latency to-date
(under 12 seconds for Stable Diffusion 1.4 without int8 quantization on Samsung
S23 Ultra for a 512x512 image with 20 iterations) on GPU-equipped mobile
devices. These enhancements broaden the applicability of generative AI and
improve the overall user experience across a wide range of devices.
less
By: Leonid Keselman, Martial Hebert
Fast, reliable shape reconstruction is an essential ingredient in many
computer vision applications. Neural Radiance Fields demonstrated that
photorealistic novel view synthesis is within reach, but was gated by
performance requirements for fast reconstruction of real scenes and objects.
Several recent approaches have built on alternative shape representations, in
particular, 3D Gaussians. We develop extensions to these renderers, such as
i... more
Fast, reliable shape reconstruction is an essential ingredient in many
computer vision applications. Neural Radiance Fields demonstrated that
photorealistic novel view synthesis is within reach, but was gated by
performance requirements for fast reconstruction of real scenes and objects.
Several recent approaches have built on alternative shape representations, in
particular, 3D Gaussians. We develop extensions to these renderers, such as
integrating differentiable optical flow, exporting watertight meshes and
rendering per-ray normals. Additionally, we show how two of the recent methods
are interoperable with each other. These reconstructions are quick, robust, and
easily performed on GPU or CPU. For code and visual examples, see
https://leonidk.github.io/fmb-plus
less
By: Haoran Chen, Kenneth Blomqvist, Francesco Milano, Roland Siegwart
Recently, methods have been proposed for 3D open-vocabulary semantic
segmentation. Such methods are able to segment scenes into arbitrary classes
given at run-time using their text description. In this paper, we propose to
our knowledge the first algorithm for open-vocabulary panoptic segmentation,
simultaneously performing both semantic and instance segmentation. Our
algorithm, Panoptic Vision-Language Feature Fields (PVLFF) learns a featu... more
Recently, methods have been proposed for 3D open-vocabulary semantic
segmentation. Such methods are able to segment scenes into arbitrary classes
given at run-time using their text description. In this paper, we propose to
our knowledge the first algorithm for open-vocabulary panoptic segmentation,
simultaneously performing both semantic and instance segmentation. Our
algorithm, Panoptic Vision-Language Feature Fields (PVLFF) learns a feature
field of the scene, jointly learning vision-language features and hierarchical
instance features through a contrastive loss function from 2D instance segment
proposals on input frames. Our method achieves comparable performance against
the state-of-the-art close-set 3D panoptic systems on the HyperSim, ScanNet and
Replica dataset and outperforms current 3D open-vocabulary systems in terms of
semantic segmentation. We additionally ablate our method to demonstrate the
effectiveness of our model architecture. Our code will be available at
https://github.com/ethz-asl/autolabel.
less
By: Yake Wei, Ruoxuan Feng, Zihe Wang, Di Hu
One primary topic of multi-modal learning is to jointly incorporate
heterogeneous information from different modalities. However, most models often
suffer from unsatisfactory multi-modal cooperation, which could not jointly
utilize all modalities well. Some methods are proposed to identify and enhance
the worse learnt modality, but are often hard to provide the fine-grained
observation of multi-modal cooperation at sample-level with theoret... more
One primary topic of multi-modal learning is to jointly incorporate
heterogeneous information from different modalities. However, most models often
suffer from unsatisfactory multi-modal cooperation, which could not jointly
utilize all modalities well. Some methods are proposed to identify and enhance
the worse learnt modality, but are often hard to provide the fine-grained
observation of multi-modal cooperation at sample-level with theoretical
support. Hence, it is essential to reasonably observe and improve the
fine-grained cooperation between modalities, especially when facing realistic
scenarios where the modality discrepancy could vary across different samples.
To this end, we introduce a fine-grained modality valuation metric to evaluate
the contribution of each modality at sample-level. Via modality valuation, we
regretfully observe that the multi-modal model tends to rely on one specific
modality, resulting in other modalities being low-contributing. We further
analyze this issue and improve cooperation between modalities by enhancing the
discriminative ability of low-contributing modalities in a targeted manner.
Overall, our methods reasonably observe the fine-grained uni-modal contribution
at sample-level and achieve considerable improvement on different multi-modal
models.
less
By: Nissim Maruani, Roman Klokov, Maks Ovsjanikov, Pierre Alliez, Mathieu Desbrun
In stark contrast to the case of images, finding a concise, learnable
discrete representation of 3D surfaces remains a challenge. In particular,
while polygon meshes are arguably the most common surface representation used
in geometry processing, their irregular and combinatorial structure often make
them unsuitable for learning-based applications. In this work, we present
VoroMesh, a novel and differentiable Voronoi-based representation of... more
In stark contrast to the case of images, finding a concise, learnable
discrete representation of 3D surfaces remains a challenge. In particular,
while polygon meshes are arguably the most common surface representation used
in geometry processing, their irregular and combinatorial structure often make
them unsuitable for learning-based applications. In this work, we present
VoroMesh, a novel and differentiable Voronoi-based representation of watertight
3D shape surfaces. From a set of 3D points (called generators) and their
associated occupancy, we define our boundary representation through the Voronoi
diagram of the generators as the subset of Voronoi faces whose two associated
(equidistant) generators are of opposite occupancy: the resulting polygon mesh
forms a watertight approximation of the target shape's boundary. To learn the
position of the generators, we propose a novel loss function, dubbed VoroLoss,
that minimizes the distance from ground truth surface samples to the closest
faces of the Voronoi diagram which does not require an explicit construction of
the entire Voronoi diagram. A direct optimization of the Voroloss to obtain
generators on the Thingi32 dataset demonstrates the geometric efficiency of our
representation compared to axiomatic meshing algorithms and recent
learning-based mesh representations. We further use VoroMesh in a
learning-based mesh prediction task from input SDF grids on the ABC dataset,
and show comparable performance to state-of-the-art methods while guaranteeing
closed output surfaces free of self-intersections.
less
By: Yufei Ye, Poorvi Hebbar, Abhinav Gupta, Shubham Tulsiani
We tackle the task of reconstructing hand-object interactions from short
video clips. Given an input video, our approach casts 3D inference as a
per-video optimization and recovers a neural 3D representation of the object
shape, as well as the time-varying motion and hand articulation. While the
input video naturally provides some multi-view cues to guide 3D inference,
these are insufficient on their own due to occlusions and limited viewpo... more
We tackle the task of reconstructing hand-object interactions from short
video clips. Given an input video, our approach casts 3D inference as a
per-video optimization and recovers a neural 3D representation of the object
shape, as well as the time-varying motion and hand articulation. While the
input video naturally provides some multi-view cues to guide 3D inference,
these are insufficient on their own due to occlusions and limited viewpoint
variations. To obtain accurate 3D, we augment the multi-view signals with
generic data-driven priors to guide reconstruction. Specifically, we learn a
diffusion network to model the conditional distribution of (geometric)
renderings of objects conditioned on hand configuration and category label, and
leverage it as a prior to guide the novel-view renderings of the reconstructed
scene. We empirically evaluate our approach on egocentric videos across 6
object categories, and observe significant improvements over prior single-view
and multi-view methods. Finally, we demonstrate our system's ability to
reconstruct arbitrary clips from YouTube, showing both 1st and 3rd person
interactions.
less
By: Hengwei Zhao, Xinyu Wang, Jingtao Li, Yanfei Zhong
Positive-unlabeled learning (PU learning) in hyperspectral remote sensing
imagery (HSI) is aimed at learning a binary classifier from positive and
unlabeled data, which has broad prospects in various earth vision applications.
However, when PU learning meets limited labeled HSI, the unlabeled data may
dominate the optimization process, which makes the neural networks overfit the
unlabeled data. In this paper, a Taylor variational loss is pr... more
Positive-unlabeled learning (PU learning) in hyperspectral remote sensing
imagery (HSI) is aimed at learning a binary classifier from positive and
unlabeled data, which has broad prospects in various earth vision applications.
However, when PU learning meets limited labeled HSI, the unlabeled data may
dominate the optimization process, which makes the neural networks overfit the
unlabeled data. In this paper, a Taylor variational loss is proposed for HSI PU
learning, which reduces the weight of the gradient of the unlabeled data by
Taylor series expansion to enable the network to find a balance between
overfitting and underfitting. In addition, the self-calibrated optimization
strategy is designed to stabilize the training process. Experiments on 7
benchmark datasets (21 tasks in total) validate the effectiveness of the
proposed method. Code is at: https://github.com/Hengwei-Zhao96/T-HOneCls.
less
By: Tim Meinhardt, Matt Feiszli, Yuchen Fan, Laura Leal-Taixe, Rakesh Ranjan
Until recently, the Video Instance Segmentation (VIS) community operated
under the common belief that offline methods are generally superior to a frame
by frame online processing. However, the recent success of online methods
questions this belief, in particular, for challenging and long video sequences.
We understand this work as a rebuttal of those recent observations and an
appeal to the community to focus on dedicated near-online VIS ap... more
Until recently, the Video Instance Segmentation (VIS) community operated
under the common belief that offline methods are generally superior to a frame
by frame online processing. However, the recent success of online methods
questions this belief, in particular, for challenging and long video sequences.
We understand this work as a rebuttal of those recent observations and an
appeal to the community to focus on dedicated near-online VIS approaches. To
support our argument, we present a detailed analysis on different processing
paradigms and the new end-to-end trainable NOVIS (Near-Online Video Instance
Segmentation) method. Our transformer-based model directly predicts
spatio-temporal mask volumes for clips of frames and performs instance tracking
between clips via overlap embeddings. NOVIS represents the first near-online
VIS approach which avoids any handcrafted tracking heuristics. We outperform
all existing VIS methods by large margins and provide new state-of-the-art
results on both YouTube-VIS (2019/2021) and the OVIS benchmarks.
less
By: Jonathan Cui, David A. Araujo, Suman Saha, Md. Faisal Kabir
Despite their simpler information fusion designs compared with Vision
Transformers and Convolutional Neural Networks, Vision MLP architectures have
demonstrated strong performance and high data efficiency in recent research.
However, existing works such as CycleMLP and Vision Permutator typically model
spatial information in equal-size spatial regions and do not consider
cross-scale spatial interactions. Further, their token mixers only mod... more
Despite their simpler information fusion designs compared with Vision
Transformers and Convolutional Neural Networks, Vision MLP architectures have
demonstrated strong performance and high data efficiency in recent research.
However, existing works such as CycleMLP and Vision Permutator typically model
spatial information in equal-size spatial regions and do not consider
cross-scale spatial interactions. Further, their token mixers only model 1- or
2-axis correlations, avoiding 3-axis spatial-channel mixing due to its
computational demands. We therefore propose CS-Mixer, a hierarchical Vision MLP
that learns dynamic low-rank transformations for spatial-channel mixing through
cross-scale local and global aggregation. The proposed methodology achieves
competitive results on popular image recognition benchmarks without incurring
substantially more compute. Our largest model, CS-Mixer-L, reaches 83.2% top-1
accuracy on ImageNet-1k with 13.7 GFLOPs and 94 M parameters.
less
By: Yiming Shan, Yan Xia, Yuhong Chen, Daniel Cremers
3D object detection using LiDAR point clouds is a fundamental task in the
fields of computer vision, robotics, and autonomous driving. However, existing
3D detectors heavily rely on annotated datasets, which are both time-consuming
and prone to errors during the process of labeling 3D bounding boxes. In this
paper, we propose a Scene Completion Pre-training (SCP) method to enhance the
performance of 3D object detectors with less labeled dat... more
3D object detection using LiDAR point clouds is a fundamental task in the
fields of computer vision, robotics, and autonomous driving. However, existing
3D detectors heavily rely on annotated datasets, which are both time-consuming
and prone to errors during the process of labeling 3D bounding boxes. In this
paper, we propose a Scene Completion Pre-training (SCP) method to enhance the
performance of 3D object detectors with less labeled data. SCP offers three key
advantages: (1) Improved initialization of the point cloud model. By completing
the scene point clouds, SCP effectively captures the spatial and semantic
relationships among objects within urban environments. (2) Elimination of the
need for additional datasets. SCP serves as a valuable auxiliary network that
does not impose any additional efforts or data requirements on the 3D
detectors. (3) Reduction of the amount of labeled data for detection. With the
help of SCP, the existing state-of-the-art 3D detectors can achieve comparable
performance while only relying on 20% labeled data.
less
By: Yuta Sato
Population aging is one of the most serious problems in certain countries. In
order to implement its countermeasures, understanding its rapid progress is of
urgency with a granular resolution. However, a detailed and rigorous survey
with high frequency is not feasible due to the constraints of financial and
human resources. Nowadays, Deep Learning is prevalent for pattern recognition
with significant accuracy, with its application to remote... more
Population aging is one of the most serious problems in certain countries. In
order to implement its countermeasures, understanding its rapid progress is of
urgency with a granular resolution. However, a detailed and rigorous survey
with high frequency is not feasible due to the constraints of financial and
human resources. Nowadays, Deep Learning is prevalent for pattern recognition
with significant accuracy, with its application to remote sensing. This paper
proposes a multi-head Convolutional Neural Network model with transfer learning
from pre-trained ResNet50 for estimating mesh-wise demographics of Japan as one
of the most aged countries in the world, with satellite images from
Landsat-8/OLI and Suomi NPP/VIIRS-DNS as inputs and census demographics as
labels. The trained model was performed on a testing dataset with a test score
of at least 0.8914 in $\text{R}^2$ for all the demographic composition groups,
and the estimated demographic composition was generated and visualised for 2022
as a non-census year.
less
By: Bowei Chen, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz
We present a method to generate full-body selfies -- photos that you take of
yourself, but capturing your whole body as if someone else took the photo of
you from a few feet away. Our approach takes as input a pre-captured video of
your body, a target pose photo, and a selfie + background pair for each
location. We introduce a novel diffusion-based approach to combine all of this
information into high quality, well-composed photos of you wi... more
We present a method to generate full-body selfies -- photos that you take of
yourself, but capturing your whole body as if someone else took the photo of
you from a few feet away. Our approach takes as input a pre-captured video of
your body, a target pose photo, and a selfie + background pair for each
location. We introduce a novel diffusion-based approach to combine all of this
information into high quality, well-composed photos of you with the desired
pose and background.
less
MoEController: Instruction-based Arbitrary Image Manipulation with Mixture-of-Expert Controllers
0upvotes
By: Sijia Li, Chen Chen, Haonan Lu
Diffusion-model-based text-guided image generation has recently made
astounding progress, producing fascinating results in open-domain image
manipulation tasks. Few models, however, currently have complete zero-shot
capabilities for both global and local image editing due to the complexity and
diversity of image manipulation tasks. In this work, we propose a method with a
mixture-of-expert (MOE) controllers to align the text-guided capacity... more
Diffusion-model-based text-guided image generation has recently made
astounding progress, producing fascinating results in open-domain image
manipulation tasks. Few models, however, currently have complete zero-shot
capabilities for both global and local image editing due to the complexity and
diversity of image manipulation tasks. In this work, we propose a method with a
mixture-of-expert (MOE) controllers to align the text-guided capacity of
diffusion models with different kinds of human instructions, enabling our model
to handle various open-domain image manipulation tasks with natural language
instructions. First, we use large language models (ChatGPT) and conditional
image synthesis models (ControlNet) to generate a large number of global image
transfer dataset in addition to the instruction-based local image editing
dataset. Then, using an MOE technique and task-specific adaptation training on
a large-scale dataset, our conditional diffusion model can edit images globally
and locally. Extensive experiments demonstrate that our approach performs
surprisingly well on various image manipulation tasks when dealing with
open-domain images and arbitrary human instructions. Please refer to our
project page: [https://oppo-mente-lab.github.io/moe_controller/]
less
By: Soumen Basu, Ashish Papanai, Mayank Gupta, Pankaj Gupta, Chetan Arora
Automated detection of Gallbladder Cancer (GBC) from Ultrasound (US) images
is an important problem, which has drawn increased interest from researchers.
However, most of these works use difficult-to-acquire information such as
bounding box annotations or additional US videos. In this paper, we focus on
GBC detection using only image-level labels. Such annotation is usually
available based on the diagnostic report of a patient, and do not r... more
Automated detection of Gallbladder Cancer (GBC) from Ultrasound (US) images
is an important problem, which has drawn increased interest from researchers.
However, most of these works use difficult-to-acquire information such as
bounding box annotations or additional US videos. In this paper, we focus on
GBC detection using only image-level labels. Such annotation is usually
available based on the diagnostic report of a patient, and do not require
additional annotation effort from the physicians. However, our analysis reveals
that it is difficult to train a standard image classification model for GBC
detection. This is due to the low inter-class variance (a malignant region
usually occupies only a small portion of a US image), high intra-class variance
(due to the US sensor capturing a 2D slice of a 3D object leading to large
viewpoint variations), and low training data availability. We posit that even
when we have only the image level label, still formulating the problem as
object detection (with bounding box output) helps a deep neural network (DNN)
model focus on the relevant region of interest. Since no bounding box
annotations is available for training, we pose the problem as weakly supervised
object detection (WSOD). Motivated by the recent success of transformer models
in object detection, we train one such model, DETR, using
multi-instance-learning (MIL) with self-supervised instance selection to suit
the WSOD task. Our proposed method demonstrates an improvement of AP and
detection sensitivity over the SOTA transformer-based and CNN-based WSOD
methods. Project page is at https://gbc-iitd.github.io/wsod-gbc
less
By: Zhuo Chen, Yufeng Huang, Jiaoyan Chen, Yuxia Geng, Yin Fang, Jeff Pan, Ningyu Zhang, Wen Zhang
Visual question answering (VQA) often requires an understanding of visual
concepts and language semantics, which relies on external knowledge. Most
existing methods exploit pre-trained language models or/and unstructured text,
but the knowledge in these resources are often incomplete and noisy. Some other
methods prefer to use knowledge graphs (KGs) which often have intensive
structured knowledge, but the research is still quite preliminary... more
Visual question answering (VQA) often requires an understanding of visual
concepts and language semantics, which relies on external knowledge. Most
existing methods exploit pre-trained language models or/and unstructured text,
but the knowledge in these resources are often incomplete and noisy. Some other
methods prefer to use knowledge graphs (KGs) which often have intensive
structured knowledge, but the research is still quite preliminary. In this
paper, we propose LaKo, a knowledge-driven VQA method via Late
Knowledge-to-text Injection. To effectively incorporate an external KG, we
transfer triples into textual format and propose a late injection mechanism for
knowledge fusion. Finally we address VQA as a text generation task with an
effective encoder-decoder paradigm, which achieves state-of-the-art results on
OKVQA dataset.
less
By: Shentong Mo, Weiguo Pian, Yapeng Tian
Continual learning is a challenging problem in which models need to be
trained on non-stationary data across sequential tasks for class-incremental
learning. While previous methods have focused on using either regularization or
rehearsal-based frameworks to alleviate catastrophic forgetting in image
classification, they are limited to a single modality and cannot learn compact
class-aware cross-modal representations for continual audio-visu... more
Continual learning is a challenging problem in which models need to be
trained on non-stationary data across sequential tasks for class-incremental
learning. While previous methods have focused on using either regularization or
rehearsal-based frameworks to alleviate catastrophic forgetting in image
classification, they are limited to a single modality and cannot learn compact
class-aware cross-modal representations for continual audio-visual learning. To
address this gap, we propose a novel class-incremental grouping network (CIGN)
that can learn category-wise semantic features to achieve continual
audio-visual learning. Our CIGN leverages learnable audio-visual class tokens
and audio-visual grouping to continually aggregate class-aware features.
Additionally, it utilizes class tokens distillation and continual grouping to
prevent forgetting parameters learned from previous tasks, thereby improving
the model's ability to capture discriminative audio-visual categories. We
conduct extensive experiments on VGGSound-Instruments, VGGSound-100, and
VGG-Sound Sources benchmarks. Our experimental results demonstrate that the
CIGN achieves state-of-the-art audio-visual class-incremental learning
performance. Code is available at https://github.com/stoneMo/CIGN.
less
By: Sushmanth reddy Mereddy, Jyothi Swaroop Reddy, Somnath Sharma
Eye tracking has been a pivotal tool in diverse fields such as vision
research, language analysis, and usability assessment. The majority of prior
investigations, however, have concentrated on expansive desktop displays
employing specialized, costly eye tracking hardware that lacks scalability.
Remarkably little insight exists into ocular movement patterns on smartphones,
despite their widespread adoption and significant usage. In this manu... more
Eye tracking has been a pivotal tool in diverse fields such as vision
research, language analysis, and usability assessment. The majority of prior
investigations, however, have concentrated on expansive desktop displays
employing specialized, costly eye tracking hardware that lacks scalability.
Remarkably little insight exists into ocular movement patterns on smartphones,
despite their widespread adoption and significant usage. In this manuscript, we
present an open-source implementation of a smartphone-based gaze tracker that
emulates the methodology proposed by a GooglePaper (whose source code remains
proprietary). Our focus is on attaining accuracy comparable to that attained
through the GooglePaper's methodology, without the necessity for supplementary
hardware. Through the integration of machine learning techniques, we unveil an
accurate eye tracking solution that is native to smartphones. Our approach
demonstrates precision akin to the state-of-the-art mobile eye trackers, which
are characterized by a cost that is two orders of magnitude higher. Leveraging
the vast MIT GazeCapture dataset, which is available through registration on
the dataset's website, we successfully replicate crucial findings from previous
studies concerning ocular motion behavior in oculomotor tasks and saliency
analyses during natural image observation. Furthermore, we emphasize the
applicability of smartphone-based gaze tracking in discerning reading
comprehension challenges. Our findings exhibit the inherent potential to
amplify eye movement research by significant proportions, accommodating
participation from thousands of subjects with explicit consent. This
scalability not only fosters advancements in vision research, but also extends
its benefits to domains such as accessibility enhancement and healthcare
applications.
less
By: Loïc Le Bescond, Maria Vakalopoulou, Stergios Christodoulidis, Fabrice André, Hugues Talbot
The deployment of machine learning solutions in real-world scenarios often
involves addressing the challenge of out-of-distribution (OOD) detection. While
significant efforts have been devoted to OOD detection in classical supervised
settings, the context of weakly supervised learning, particularly the Multiple
Instance Learning (MIL) framework, remains under-explored. In this study, we
tackle this challenge by adapting post-hoc OOD detecti... more
The deployment of machine learning solutions in real-world scenarios often
involves addressing the challenge of out-of-distribution (OOD) detection. While
significant efforts have been devoted to OOD detection in classical supervised
settings, the context of weakly supervised learning, particularly the Multiple
Instance Learning (MIL) framework, remains under-explored. In this study, we
tackle this challenge by adapting post-hoc OOD detection methods to the MIL
setting while introducing a novel benchmark specifically designed to assess OOD
detection performance in weakly supervised scenarios. Extensive experiments
based on diverse public datasets do not reveal a single method with a clear
advantage over the others. Although DICE emerges as the best-performing method
overall, it exhibits significant shortcomings on some datasets, emphasizing the
complexity of this under-explored and challenging topic. Our findings shed
light on the complex nature of OOD detection under the MIL framework,
emphasizing the importance of developing novel, robust, and reliable methods
that can generalize effectively in a weakly supervised context. The code for
the paper is available here: https://github.com/loic-lb/OOD_MIL.
less
By: Jindong Gu, Fangyun Wei, Philip Torr, Han Hu
Deep Neural Networks can be easily fooled by small and imperceptible
perturbations. The query-based black-box attack (QBBA) is able to create the
perturbations using model output probabilities of image queries requiring no
access to the underlying models. QBBA poses realistic threats to real-world
applications. Recently, various types of robustness have been explored to
defend against QBBA. In this work, we first taxonomize the stochastic d... more
Deep Neural Networks can be easily fooled by small and imperceptible
perturbations. The query-based black-box attack (QBBA) is able to create the
perturbations using model output probabilities of image queries requiring no
access to the underlying models. QBBA poses realistic threats to real-world
applications. Recently, various types of robustness have been explored to
defend against QBBA. In this work, we first taxonomize the stochastic defense
strategies against QBBA. Following our taxonomy, we propose to explore
non-additive randomness in models to defend against QBBA. Specifically, we
focus on underexplored Vision Transformers based on their flexible
architectures. Extensive experiments show that the proposed defense approach
achieves effective defense, without much sacrifice in performance.
less
By: Weijian Xie, Guanyi Chu, Quanhao Qian, Yihao Yu, Hai Li, Danpeng Chen, Shangjin Zhai, Nan Wang, Hujun Bao, Guofeng Zhang
Dense SLAM based on monocular cameras does indeed have immense application
value in the field of AR/VR, especially when it is performed on a mobile
device. In this paper, we propose a novel method that integrates a light-weight
depth completion network into a sparse SLAM system using a multi-basis depth
representation, so that dense mapping can be performed online even on a mobile
phone. Specifically, we present a specifically optimized mul... more
Dense SLAM based on monocular cameras does indeed have immense application
value in the field of AR/VR, especially when it is performed on a mobile
device. In this paper, we propose a novel method that integrates a light-weight
depth completion network into a sparse SLAM system using a multi-basis depth
representation, so that dense mapping can be performed online even on a mobile
phone. Specifically, we present a specifically optimized multi-basis depth
completion network, called BBC-Net, tailored to the characteristics of
traditional sparse SLAM systems. BBC-Net can predict multiple balanced bases
and a confidence map from a monocular image with sparse points generated by
off-the-shelf keypoint-based SLAM systems. The final depth is a linear
combination of predicted depth bases that can be optimized by tuning the
corresponding weights. To seamlessly incorporate the weights into traditional
SLAM optimization and ensure efficiency and robustness, we design a set of
depth weight factors, which makes our network a versatile plug-in module,
facilitating easy integration into various existing sparse SLAM systems and
significantly enhancing global depth consistency through bundle adjustment. To
verify the portability of our method, we integrate BBC-Net into two
representative SLAM systems. The experimental results on various datasets show
that the proposed method achieves better performance in monocular dense mapping
than the state-of-the-art methods. We provide an online demo running on a
mobile phone, which verifies the efficiency and mapping quality of the proposed
method in real-world scenarios.
less
By: Alessandro Duico, Ombretta Strafforello, Jan van Gemert
Predicting which specific parts of a video users will replay is important for
several applications, including targeted advertisement placement on video
platforms and assisting video creators. In this work, we explore whether it is
possible to predict the Most Replayed (MR) data from YouTube videos. To this
end, we curate a large video benchmark, the YTMR500 dataset, which comprises
500 YouTube videos with MR data annotations. We evaluate De... more
Predicting which specific parts of a video users will replay is important for
several applications, including targeted advertisement placement on video
platforms and assisting video creators. In this work, we explore whether it is
possible to predict the Most Replayed (MR) data from YouTube videos. To this
end, we curate a large video benchmark, the YTMR500 dataset, which comprises
500 YouTube videos with MR data annotations. We evaluate Deep Learning (DL)
models of varying complexity on our dataset and perform an extensive ablation
study. In addition, we conduct a user study to estimate the human performance
on MR data prediction. Our results show that, although by a narrow margin, all
the evaluated DL models outperform random predictions. Additionally, they
exceed human-level accuracy. This suggests that predicting the MR data is a
difficult task that can be enhanced through the assistance of DL. Finally, we
believe that DL performance on MR data prediction can be further improved, for
example, by using multi-modal learning. We encourage the research community to
use our benchmark dataset to further investigate automatic MR data prediction.
less
By: Yijin Li, Han Zhou, Bangbang Yang, Ye Zhang, Zhaopeng Cui, Hujun Bao, Guofeng Zhang
Different from traditional video cameras, event cameras capture asynchronous
events stream in which each event encodes pixel location, trigger time, and the
polarity of the brightness changes. In this paper, we introduce a novel
graph-based framework for event cameras, namely SlideGCN. Unlike some recent
graph-based methods that use groups of events as input, our approach can
efficiently process data event-by-event, unlock the low latency n... more
Different from traditional video cameras, event cameras capture asynchronous
events stream in which each event encodes pixel location, trigger time, and the
polarity of the brightness changes. In this paper, we introduce a novel
graph-based framework for event cameras, namely SlideGCN. Unlike some recent
graph-based methods that use groups of events as input, our approach can
efficiently process data event-by-event, unlock the low latency nature of
events data while still maintaining the graph's structure internally. For fast
graph construction, we develop a radius search algorithm, which better exploits
the partial regular structure of event cloud against k-d tree based generic
methods. Experiments show that our method reduces the computational complexity
up to 100 times with respect to current graph-based methods while keeping
state-of-the-art performance on object recognition. Moreover, we verify the
superiority of event-wise processing with our method. When the state becomes
stable, we can give a prediction with high confidence, thus making an early
recognition. Project page: \url{https://zju3dv.github.io/slide_gcn/}.
less
By: Sergi Garcia-Bordils, Dimosthenis Karatzas, Marçal Rusiñol
We introduce the structured scene-text spotting task, which requires a
scene-text OCR system to spot text in the wild according to a query regular
expression. Contrary to generic scene text OCR, structured scene-text spotting
seeks to dynamically condition both scene text detection and recognition on
user-provided regular expressions. To tackle this task, we propose the
Structured TExt sPotter (STEP), a model that exploits the provided text... more
We introduce the structured scene-text spotting task, which requires a
scene-text OCR system to spot text in the wild according to a query regular
expression. Contrary to generic scene text OCR, structured scene-text spotting
seeks to dynamically condition both scene text detection and recognition on
user-provided regular expressions. To tackle this task, we propose the
Structured TExt sPotter (STEP), a model that exploits the provided text
structure to guide the OCR process. STEP is able to deal with regular
expressions that contain spaces and it is not bound to detection at the
word-level granularity. Our approach enables accurate zero-shot structured text
spotting in a wide variety of real-world reading scenarios and is solely
trained on publicly available data. To demonstrate the effectiveness of our
approach, we introduce a new challenging test dataset that contains several
types of out-of-vocabulary structured text, reflecting important reading
applications of fields such as prices, dates, serial numbers, license plates
etc. We demonstrate that STEP can provide specialised OCR performance on demand
in all tested scenarios.
less
By: Gang Wu, Junjun Jiang, Kui Jiang, Xianming Liu
Contrastive learning has emerged as a prevailing paradigm for high-level
vision tasks, which, by introducing properly negative samples, has also been
exploited for low-level vision tasks to achieve a compact optimization space to
account for their ill-posed nature. However, existing methods rely on manually
predefined, task-oriented negatives, which often exhibit pronounced
task-specific biases. In this paper, we propose a innovative approa... more
Contrastive learning has emerged as a prevailing paradigm for high-level
vision tasks, which, by introducing properly negative samples, has also been
exploited for low-level vision tasks to achieve a compact optimization space to
account for their ill-posed nature. However, existing methods rely on manually
predefined, task-oriented negatives, which often exhibit pronounced
task-specific biases. In this paper, we propose a innovative approach for the
adaptive generation of negative samples directly from the target model itself,
called ``learning from history``. We introduce the Self-Prior guided Negative
loss for image restoration (SPNIR) to enable this approach. Our approach is
task-agnostic and generic, making it compatible with any existing image
restoration method or task. We demonstrate the effectiveness of our approach by
retraining existing models with SPNIR. The results show significant
improvements in image restoration across various tasks and architectures. For
example, models retrained with SPNIR outperform the original FFANet and
DehazeFormer by 3.41 dB and 0.57 dB on the RESIDE indoor dataset for image
dehazing. Similarly, they achieve notable improvements of 0.47 dB on SPA-Data
over IDT for image deraining and 0.12 dB on Manga109 for a 4x scale
super-resolution over lightweight SwinIR, respectively. Code and retrained
models are available at
https://github.com/Aitical/Task-agnostic_Model_Contrastive_Learning_Image_Restoration.
less
By: Leo Fillioux, Emilie Gontran, Jérôme Cartry, Jacques RR Mathieu, Sabrina Bedja, Alice Boilève, Paul-Henry Cournède, Fanny Jaulin, Stergios Christodoulidis, Maria Vakalopoulou
Over the last ten years, Patient-Derived Organoids (PDOs) emerged as the most
reliable technology to generate ex-vivo tumor avatars. PDOs retain the main
characteristics of their original tumor, making them a system of choice for
pre-clinical and clinical studies. In particular, PDOs are attracting interest
in the field of Functional Precision Medicine (FPM), which is based upon an
ex-vivo drug test in which living tumor cells (such as PDOs... more
Over the last ten years, Patient-Derived Organoids (PDOs) emerged as the most
reliable technology to generate ex-vivo tumor avatars. PDOs retain the main
characteristics of their original tumor, making them a system of choice for
pre-clinical and clinical studies. In particular, PDOs are attracting interest
in the field of Functional Precision Medicine (FPM), which is based upon an
ex-vivo drug test in which living tumor cells (such as PDOs) from a specific
patient are exposed to a panel of anti-cancer drugs. Currently, the Adenosine
Triphosphate (ATP) based cell viability assay is the gold standard test to
assess the sensitivity of PDOs to drugs. The readout is measured at the end of
the assay from a global PDO population and therefore does not capture single
PDO responses and does not provide time resolution of drug effect. To this end,
in this study, we explore for the first time the use of powerful large
foundation models for the automatic processing of PDO data. In particular, we
propose a novel imaging-based high-throughput screening method to assess
real-time drug efficacy from a time-lapse microscopy video of PDOs. The
recently proposed SAM algorithm for segmentation and DINOv2 model are adapted
in a comprehensive pipeline for processing PDO microscopy frames. Moreover, an
attention mechanism is proposed for fusing temporal and spatial features in a
multiple instance learning setting to predict ATP. We report better results
than other non-time-resolved methods, indicating that the temporality of data
is an important factor for the prediction of ATP. Extensive ablations shed
light on optimizing the experimental setting and automating the prediction both
in real-time and for forecasting.
less
By: Kuangdai Leng, Jeyan Thiyagalingam
Convolution is a fundamental operation in image processing and machine
learning. Aimed primarily at maintaining image size, padding is a key
ingredient of convolution, which, however, can introduce undesirable boundary
effects. We present a non-padding-based method for size-keeping convolution
based on the preservation of differential characteristics of kernels. The main
idea is to make convolution over an incomplete sliding window "collaps... more
Convolution is a fundamental operation in image processing and machine
learning. Aimed primarily at maintaining image size, padding is a key
ingredient of convolution, which, however, can introduce undesirable boundary
effects. We present a non-padding-based method for size-keeping convolution
based on the preservation of differential characteristics of kernels. The main
idea is to make convolution over an incomplete sliding window "collapse" to a
linear differential operator evaluated locally at its central pixel, which no
longer requires information from the neighbouring missing pixels. While the
underlying theory is rigorous, our final formula turns out to be simple: the
convolution over an incomplete window is achieved by convolving its nearest
complete window with a transformed kernel. This formula is computationally
lightweight, involving neither interpolation or extrapolation nor restrictions
on image and kernel sizes. Our method favours data with smooth boundaries, such
as high-resolution images and fields from physics. Our experiments include: i)
filtering analytical and non-analytical fields from computational physics and,
ii) training convolutional neural networks (CNNs) for the tasks of image
classification, semantic segmentation and super-resolution reconstruction. In
all these experiments, our method has exhibited visible superiority over the
compared ones.
less
By: Yemin li, Zhongcheng Liu, Xiaoying Lou, Mirigual Kurban, Miao Li, Jie Yang, Kaiwei Che, Jiankun Wang, Max Q. -H Meng, Yan Huang, Qin Guo, Pinjin Hu
BACKGROUND & AIMS: Histological remission (HR) is advocated and considered as
a new therapeutic target in ulcerative colitis (UC). Diagnosis of histologic
remission currently relies on biopsy; during this process, patients are at risk
for bleeding, infection, and post-biopsy fibrosis. In addition, histologic
response scoring is complex and time-consuming, and there is heterogeneity
among pathologists. Endocytoscopy (EC) is a novel ultra-hig... more
BACKGROUND & AIMS: Histological remission (HR) is advocated and considered as
a new therapeutic target in ulcerative colitis (UC). Diagnosis of histologic
remission currently relies on biopsy; during this process, patients are at risk
for bleeding, infection, and post-biopsy fibrosis. In addition, histologic
response scoring is complex and time-consuming, and there is heterogeneity
among pathologists. Endocytoscopy (EC) is a novel ultra-high magnification
endoscopic technique that can provide excellent in vivo assessment of glands.
Based on the EC technique, we propose a neural network model that can assess
histological disease activity in UC using EC images to address the above
issues. The experiment results demonstrate that the proposed method can assist
patients in precise treatment and prognostic assessment.
METHODS: We construct a neural network model for UC evaluation. A total of
5105 images of 154 intestinal segments from 87 patients undergoing EC treatment
at a center in China between March 2022 and March 2023 are scored according to
the Geboes score. Subsequently, 103 intestinal segments are used as the
training set, 16 intestinal segments are used as the validation set for neural
network training, and the remaining 35 intestinal segments are used as the test
set to measure the model performance together with the validation set.
RESULTS: By treating HR as a negative category and histologic activity as a
positive category, the proposed neural network model can achieve an accuracy of
0.9, a specificity of 0.95, a sensitivity of 0.75, and an area under the curve
(AUC) of 0.81.
CONCLUSION: We develop a specific neural network model that can distinguish
histologic remission/activity in EC images of UC, which helps to accelerate
clinical histological diagnosis.
keywords: ulcerative colitis; Endocytoscopy; Geboes score; neural network.
less
By: Maria Priisalu
It is difficult to perform 3D reconstruction from on-vehicle gathered video
due to the large forward motion of the vehicle. Even object detection and human
sensing models perform significantly worse on onboard videos when compared to
standard benchmarks because objects often appear far away from the camera
compared to the standard object detection benchmarks, image quality is often
decreased by motion blur and occlusions occur often. This h... more
It is difficult to perform 3D reconstruction from on-vehicle gathered video
due to the large forward motion of the vehicle. Even object detection and human
sensing models perform significantly worse on onboard videos when compared to
standard benchmarks because objects often appear far away from the camera
compared to the standard object detection benchmarks, image quality is often
decreased by motion blur and occlusions occur often. This has led to the
popularisation of traffic data-specific benchmarks. Recently Light Detection
And Ranging (LiDAR) sensors have become popular to directly estimate depths
without the need to perform 3D reconstructions. However, LiDAR-based methods
still lack in articulated human detection at a distance when compared to
image-based methods. We hypothesize that benchmarks targeted at articulated
human sensing from LiDAR data could bring about increased research in human
sensing and prediction in traffic and could lead to improved traffic safety for
pedestrians.
less
By: Cristiano Saltori, Fabio Galasso, Giuseppe Fiameni, Nicu Sebe, Fabio Poiesi, Elisa Ricci
Deep-learning models for 3D point cloud semantic segmentation exhibit limited
generalization capabilities when trained and tested on data captured with
different sensors or in varying environments due to domain shift. Domain
adaptation methods can be employed to mitigate this domain shift, for instance,
by simulating sensor noise, developing domain-agnostic generators, or training
point cloud completion networks. Often, these methods are ta... more
Deep-learning models for 3D point cloud semantic segmentation exhibit limited
generalization capabilities when trained and tested on data captured with
different sensors or in varying environments due to domain shift. Domain
adaptation methods can be employed to mitigate this domain shift, for instance,
by simulating sensor noise, developing domain-agnostic generators, or training
point cloud completion networks. Often, these methods are tailored for range
view maps or necessitate multi-modal input. In contrast, domain adaptation in
the image domain can be executed through sample mixing, which emphasizes input
data manipulation rather than employing distinct adaptation modules. In this
study, we introduce compositional semantic mixing for point cloud domain
adaptation, representing the first unsupervised domain adaptation technique for
point cloud segmentation based on semantic and geometric sample mixing. We
present a two-branch symmetric network architecture capable of concurrently
processing point clouds from a source domain (e.g. synthetic) and point clouds
from a target domain (e.g. real-world). Each branch operates within one domain
by integrating selected data fragments from the other domain and utilizing
semantic information derived from source labels and target (pseudo) labels.
Additionally, our method can leverage a limited number of human point-level
annotations (semi-supervised) to further enhance performance. We assess our
approach in both synthetic-to-real and real-to-real scenarios using LiDAR
datasets and demonstrate that it significantly outperforms state-of-the-art
methods in both unsupervised and semi-supervised settings.
less
By: Hao Yu, Xu Cheng, Wei Peng, Weihao Liu, Guoying Zhao
Visible-infrared person re-identification (VI-ReID) is a challenging task due
to large cross-modality discrepancies and intra-class variations. Existing
methods mainly focus on learning modality-shared representations by embedding
different modalities into the same feature space. As a result, the learned
feature emphasizes the common patterns across modalities while suppressing
modality-specific and identity-aware information that is valuab... more
Visible-infrared person re-identification (VI-ReID) is a challenging task due
to large cross-modality discrepancies and intra-class variations. Existing
methods mainly focus on learning modality-shared representations by embedding
different modalities into the same feature space. As a result, the learned
feature emphasizes the common patterns across modalities while suppressing
modality-specific and identity-aware information that is valuable for Re-ID. To
address these issues, we propose a novel Modality Unifying Network (MUN) to
explore a robust auxiliary modality for VI-ReID. First, the auxiliary modality
is generated by combining the proposed cross-modality learner and
intra-modality learner, which can dynamically model the modality-specific and
modality-shared representations to alleviate both cross-modality and
intra-modality variations. Second, by aligning identity centres across the
three modalities, an identity alignment loss function is proposed to discover
the discriminative feature representations. Third, a modality alignment loss is
introduced to consistently reduce the distribution distance of visible and
infrared images by modality prototype modeling. Extensive experiments on
multiple public datasets demonstrate that the proposed method surpasses the
current state-of-the-art methods by a significant margin.
less
By: A. Emin Orhan
This paper asks whether current self-supervised learning methods, if
sufficiently scaled up, would be able to reach human-level visual object
recognition capabilities with the same type and amount of visual experience
humans learn from. Previous work on this question only considered the scaling
of data size. Here, we consider the simultaneous scaling of data size, model
size, and image resolution. We perform a scaling experiment with vision... more
This paper asks whether current self-supervised learning methods, if
sufficiently scaled up, would be able to reach human-level visual object
recognition capabilities with the same type and amount of visual experience
humans learn from. Previous work on this question only considered the scaling
of data size. Here, we consider the simultaneous scaling of data size, model
size, and image resolution. We perform a scaling experiment with vision
transformers up to 633M parameters in size (ViT-H/14) trained with up to 5K
hours of human-like video data (long, continuous, mostly egocentric videos)
with image resolutions of up to 476x476 pixels. The efficiency of masked
autoencoders (MAEs) as a self-supervised learning algorithm makes it possible
to run this scaling experiment on an unassuming academic budget. We find that
it is feasible to reach human-level object recognition capacity at sub-human
scales of model size, data size, and image size, if these factors are scaled up
simultaneously. To give a concrete example, we estimate that a 2.5B parameter
ViT model trained with 20K hours (2.3 years) of human-like video data with a
spatial resolution of 952x952 pixels should be able to reach human-level
accuracy on ImageNet. Human-level competence is thus achievable for a
fundamental perceptual capability from human-like perceptual experience
(human-like in both amount and type) with extremely generic learning algorithms
and architectures and without any substantive inductive biases.
less
By: Markus Knoche, István Sárándi, Bastian Leibe
We address the problem of reposing an image of a human into any desired novel pose. This conditional image-generation task requires reasoning about the 3D structure of the human, including self-occluded body parts. Most prior works are either based on 2D representations or require fitting and manipulating an explicit 3D body mesh. Based on the recent success in deep learning-based volumetric representations, we propose to implicitly learn a d... more
We address the problem of reposing an image of a human into any desired novel pose. This conditional image-generation task requires reasoning about the 3D structure of the human, including self-occluded body parts. Most prior works are either based on 2D representations or require fitting and manipulating an explicit 3D body mesh. Based on the recent success in deep learning-based volumetric representations, we propose to implicitly learn a dense feature volume from human images, which lends itself to simple and intuitive manipulation through explicit geometric warping. Once the latent feature volume is warped according to the desired pose change, the volume is mapped back to RGB space by a convolutional decoder. Our state-of-the-art results on the DeepFashion and the iPER benchmarks indicate that dense volumetric human representations are worth investigating in more detail. less
By: Jionghao Wang, Ziyu Chen, Jun Ling, Rong Xie, Li Song
360$^\circ$ panoramas are extensively utilized as environmental light sources
in computer graphics. However, capturing a 360$^\circ$ $\times$ 180$^\circ$
panorama poses challenges due to the necessity of specialized and costly
equipment, and additional human resources. Prior studies develop various
learning-based generative methods to synthesize panoramas from a single Narrow
Field-of-View (NFoV) image, but they are limited in alterable inp... more
360$^\circ$ panoramas are extensively utilized as environmental light sources
in computer graphics. However, capturing a 360$^\circ$ $\times$ 180$^\circ$
panorama poses challenges due to the necessity of specialized and costly
equipment, and additional human resources. Prior studies develop various
learning-based generative methods to synthesize panoramas from a single Narrow
Field-of-View (NFoV) image, but they are limited in alterable input patterns,
generation quality, and controllability. To address these issues, we propose a
novel pipeline called PanoDiff, which efficiently generates complete
360$^\circ$ panoramas using one or more unregistered NFoV images captured from
arbitrary angles. Our approach has two primary components to overcome the
limitations. Firstly, a two-stage angle prediction module to handle various
numbers of NFoV inputs. Secondly, a novel latent diffusion-based panorama
generation model uses incomplete panorama and text prompts as control signals
and utilizes several geometric augmentation schemes to ensure geometric
properties in generated panoramas. Experiments show that PanoDiff achieves
state-of-the-art panoramic generation quality and high controllability, making
it suitable for applications such as content editing.
less
By: Taegyeong Lee, Jeonghun Kang, Hyeonyu Kim, Taehwan Kim
Representing wild sounds as images is an important but challenging task due
to the lack of paired datasets between sound and images and the significant
differences in the characteristics of these two modalities. Previous studies
have focused on generating images from sound in limited categories or music. In
this paper, we propose a novel approach to generate images from in-the-wild
sounds. First, we convert sound into text using audio capti... more
Representing wild sounds as images is an important but challenging task due
to the lack of paired datasets between sound and images and the significant
differences in the characteristics of these two modalities. Previous studies
have focused on generating images from sound in limited categories or music. In
this paper, we propose a novel approach to generate images from in-the-wild
sounds. First, we convert sound into text using audio captioning. Second, we
propose audio attention and sentence attention to represent the rich
characteristics of sound and visualize the sound. Lastly, we propose a direct
sound optimization with CLIPscore and AudioCLIP and generate images with a
diffusion-based model. In experiments, it shows that our model is able to
generate high quality images from wild sounds and outperforms baselines in both
quantitative and qualitative evaluations on wild audio datasets.
less
Video-Based Hand Pose Estimation for Remote Assessment of Bradykinesia in Parkinson's Disease
0upvotes
By: Gabriela T. Acevedo Trebbau, Andrea Bandini, Diego L. Guarin
There is a growing interest in using pose estimation algorithms for
video-based assessment of Bradykinesia in Parkinson's Disease (PD) to
facilitate remote disease assessment and monitoring. However, the accuracy of
pose estimation algorithms in videos from video streaming services during
Telehealth appointments has not been studied. In this study, we used seven
off-the-shelf hand pose estimation models to estimate the movement of the thumb... more
There is a growing interest in using pose estimation algorithms for
video-based assessment of Bradykinesia in Parkinson's Disease (PD) to
facilitate remote disease assessment and monitoring. However, the accuracy of
pose estimation algorithms in videos from video streaming services during
Telehealth appointments has not been studied. In this study, we used seven
off-the-shelf hand pose estimation models to estimate the movement of the thumb
and index fingers in videos of the finger-tapping (FT) test recorded from
Healthy Controls (HC) and participants with PD and under two different
conditions: streaming (videos recorded during a live Zoom meeting) and
on-device (videos recorded locally with high-quality cameras). The accuracy and
reliability of the models were estimated by comparing the models' output with
manual results. Three of the seven models demonstrated good accuracy for
on-device recordings, and the accuracy decreased significantly for streaming
recordings. We observed a negative correlation between movement speed and the
model's accuracy for the streaming recordings. Additionally, we evaluated the
reliability of ten movement features related to bradykinesia extracted from
video recordings of PD patients performing the FT test. While most of the
features demonstrated excellent reliability for on-device recordings, most of
the features demonstrated poor to moderate reliability for streaming
recordings. Our findings highlight the limitations of pose estimation
algorithms when applied to video recordings obtained during Telehealth visits,
and demonstrate that on-device recordings can be used for automatic
video-assessment of bradykinesia in PD.
less
By: Aoxu Liu, Xiaohong Fan, Yin Yang, Jianping Zhang
The problem of phase retrieval (PR) involves recovering an unknown image from
limited amplitude measurement data and is a challenge nonlinear inverse problem
in computational imaging and image processing. However, many of the PR methods
are based on black-box network models that lack interpretability and
plug-and-play (PnP) frameworks that are computationally complex and require
careful parameter tuning. To address this, we have developed P... more
The problem of phase retrieval (PR) involves recovering an unknown image from
limited amplitude measurement data and is a challenge nonlinear inverse problem
in computational imaging and image processing. However, many of the PR methods
are based on black-box network models that lack interpretability and
plug-and-play (PnP) frameworks that are computationally complex and require
careful parameter tuning. To address this, we have developed PRISTA-Net, a deep
unfolding network (DUN) based on the first-order iterative shrinkage
thresholding algorithm (ISTA). This network utilizes a learnable nonlinear
transformation to address the proximal-point mapping sub-problem associated
with the sparse priors, and an attention mechanism to focus on phase
information containing image edges, textures, and structures. Additionally, the
fast Fourier transform (FFT) is used to learn global features to enhance local
information, and the designed logarithmic-based loss function leads to
significant improvements when the noise level is low. All parameters in the
proposed PRISTA-Net framework, including the nonlinear transformation,
threshold parameters, and step size, are learned end-to-end instead of being
manually set. This method combines the interpretability of traditional methods
with the fast inference ability of deep learning and is able to handle noise at
each iteration during the unfolding stage, thus improving recovery quality.
Experiments on Coded Diffraction Patterns (CDPs) measurements demonstrate that
our approach outperforms the existing state-of-the-art methods in terms of
qualitative and quantitative evaluations. Our source codes are available at
\emph{https://github.com/liuaxou/PRISTA-Net}.
less
By: Yan Luo, Min Shi, Yu Tian, Tobias Elze, Mengyu Wang
Glaucoma is the number one cause of irreversible blindness globally. A major
challenge for accurate glaucoma detection and progression forecasting is the
bottleneck of limited labeled patients with the state-of-the-art (SOTA) 3D
retinal imaging data of optical coherence tomography (OCT). To address the data
scarcity issue, this paper proposes two solutions. First, we develop a novel
generalization-reinforced semi-supervised learning (SSL) m... more
Glaucoma is the number one cause of irreversible blindness globally. A major
challenge for accurate glaucoma detection and progression forecasting is the
bottleneck of limited labeled patients with the state-of-the-art (SOTA) 3D
retinal imaging data of optical coherence tomography (OCT). To address the data
scarcity issue, this paper proposes two solutions. First, we develop a novel
generalization-reinforced semi-supervised learning (SSL) model called pseudo
supervisor to optimally utilize unlabeled data. Compared with SOTA models, the
proposed pseudo supervisor optimizes the policy of predicting pseudo labels
with unlabeled samples to improve empirical generalization. Our pseudo
supervisor model is evaluated with two clinical tasks consisting of glaucoma
detection and progression forecasting. The progression forecasting task is
evaluated both unimodally and multimodally. Our pseudo supervisor model
demonstrates superior performance than SOTA SSL comparison models. Moreover,
our model also achieves the best results on the publicly available LAG fundus
dataset. Second, we introduce the Harvard Glaucoma Detection and Progression
(Harvard-GDP) Dataset, a multimodal multitask dataset that includes data from
1,000 patients with OCT imaging data, as well as labels for glaucoma detection
and progression. This is the largest glaucoma detection dataset with 3D OCT
imaging data and the first glaucoma progression forecasting dataset that is
publicly available. Detailed sex and racial analysis are provided, which can be
used by interested researchers for fairness learning studies. Our released
dataset is benchmarked with several SOTA supervised CNN and transformer deep
learning models. The dataset and code are made publicly available via
\url{https://ophai.hms.harvard.edu/datasets/harvard-gdp1000}.
less
By: Weiwen Wu, Yanyang Wang
Score-based generative models (SGMs) have gained prominence in sparse-view CT
reconstruction for their precise sampling of complex distributions. In
SGM-based reconstruction, data consistency in the score-based diffusion model
ensures close adherence of generated samples to observed data distribution,
crucial for improving image quality. Shortcomings in data consistency
characterization manifest in three aspects. Firstly, data from the opti... more
Score-based generative models (SGMs) have gained prominence in sparse-view CT
reconstruction for their precise sampling of complex distributions. In
SGM-based reconstruction, data consistency in the score-based diffusion model
ensures close adherence of generated samples to observed data distribution,
crucial for improving image quality. Shortcomings in data consistency
characterization manifest in three aspects. Firstly, data from the optimization
process can lead to artifacts in reconstructed images. Secondly, it often
neglects that the generation model and original data constraints are
independently completed, fragmenting unity. Thirdly, it predominantly focuses
on constraining intermediate results in the inverse sampling process, rather
than ideal real images. Thus, we propose an iterative optimization data scoring
model. This paper introduces the data-iterative optimization score-based model
(DOSM), integrating innovative data consistency into the Stochastic
Differential Equation, a valuable constraint for ultra-sparse-view CT
reconstruction. The novelty of this data consistency element lies in its sole
reliance on original measurement data to confine generation outcomes,
effectively balancing measurement data and generative model constraints.
Additionally, we pioneer an inference strategy that traces back from current
iteration results to ideal truth, enhancing reconstruction stability. We
leverage conventional iteration techniques to optimize DOSM updates.
Quantitative and qualitative results from 23 views of numerical and clinical
cardiac datasets demonstrate DOSM's superiority over other methods. Remarkably,
even with 10 views, our method achieves excellent performance.
less
By: Haonan Qiu, Zhaoxi Chen, Yuming Jiang, Hang Zhou, Xiangyu Fan, Lei Yang, Wayne Wu, Ziwei Liu
Recent years have witnessed great progress in creating vivid audio-driven
portraits from monocular videos. However, how to seamlessly adapt the created
video avatars to other scenarios with different backgrounds and lighting
conditions remains unsolved. On the other hand, existing relighting studies
mostly rely on dynamically lighted or multi-view data, which are too expensive
for creating video portraits. To bridge this gap, we propose Rel... more
Recent years have witnessed great progress in creating vivid audio-driven
portraits from monocular videos. However, how to seamlessly adapt the created
video avatars to other scenarios with different backgrounds and lighting
conditions remains unsolved. On the other hand, existing relighting studies
mostly rely on dynamically lighted or multi-view data, which are too expensive
for creating video portraits. To bridge this gap, we propose ReliTalk, a novel
framework for relightable audio-driven talking portrait generation from
monocular videos. Our key insight is to decompose the portrait's reflectance
from implicitly learned audio-driven facial normals and images. Specifically,
we involve 3D facial priors derived from audio features to predict delicate
normal maps through implicit functions. These initially predicted normals then
take a crucial part in reflectance decomposition by dynamically estimating the
lighting condition of the given video. Moreover, the stereoscopic face
representation is refined using the identity-consistent loss under simulated
multiple lighting conditions, addressing the ill-posed problem caused by
limited views available from a single monocular video. Extensive experiments
validate the superiority of our proposed framework on both real and synthetic
datasets. Our code is released in https://github.com/arthur-qiu/ReliTalk.
less
By: Renyu Zhu, Chengcheng Han, Yong Qian, Qiushi Sun, Xiang Li, Ming Gao, Xuezhi Cao, Yunsen Xian
We study the problem of multimodal fusion in this paper. Recent
exchanging-based methods have been proposed for vision-vision fusion, which aim
to exchange embeddings learned from one modality to the other. However, most of
them project inputs of multimodalities into different low-dimensional spaces
and cannot be applied to the sequential input data. To solve these issues, in
this paper, we propose a novel exchanging-based multimodal fusion... more
We study the problem of multimodal fusion in this paper. Recent
exchanging-based methods have been proposed for vision-vision fusion, which aim
to exchange embeddings learned from one modality to the other. However, most of
them project inputs of multimodalities into different low-dimensional spaces
and cannot be applied to the sequential input data. To solve these issues, in
this paper, we propose a novel exchanging-based multimodal fusion model MuSE
for text-vision fusion based on Transformer. We first use two encoders to
separately map multimodal inputs into different low-dimensional spaces. Then we
employ two decoders to regularize the embeddings and pull them into the same
space. The two decoders capture the correlations between texts and images with
the image captioning task and the text-to-image generation task, respectively.
Further, based on the regularized embeddings, we present CrossTransformer,
which uses two Transformer encoders with shared parameters as the backbone
model to exchange knowledge between multimodalities. Specifically,
CrossTransformer first learns the global contextual information of the inputs
in the shallow layers. After that, it performs inter-modal exchange by
selecting a proportion of tokens in one modality and replacing their embeddings
with the average of embeddings in the other modality. We conduct extensive
experiments to evaluate the performance of MuSE on the Multimodal Named Entity
Recognition task and the Multimodal Sentiment Analysis task. Our results show
the superiority of MuSE against other competitors. Our code and data are
provided at https://github.com/RecklessRonan/MuSE.
less
Enhancing OCR Performance through Post-OCR Models: Adopting Glyph Embedding for Improved Correction
0upvotes
By: Yung-Hsin Chen, Yuli Zhou
The study investigates the potential of post-OCR models to overcome
limitations in OCR models and explores the impact of incorporating glyph
embedding on post-OCR correction performance. In this study, we have developed
our own post-OCR correction model. The novelty of our approach lies in
embedding the OCR output using CharBERT and our unique embedding technique,
capturing the visual characteristics of characters. Our findings show that
po... more
The study investigates the potential of post-OCR models to overcome
limitations in OCR models and explores the impact of incorporating glyph
embedding on post-OCR correction performance. In this study, we have developed
our own post-OCR correction model. The novelty of our approach lies in
embedding the OCR output using CharBERT and our unique embedding technique,
capturing the visual characteristics of characters. Our findings show that
post-OCR correction effectively addresses deficiencies in inferior OCR models,
and glyph embedding enables the model to achieve superior results, including
the ability to correct individual words.
less
By: Di Yang, Yaohui Wang, Antitza Dantcheva, Quan Kong, Lorenzo Garattoni, Gianpiero Francesca, Francois Bremond
Skeleton-based action segmentation requires recognizing composable actions in
untrimmed videos. Current approaches decouple this problem by first extracting
local visual features from skeleton sequences and then processing them by a
temporal model to classify frame-wise actions. However, their performances
remain limited as the visual features cannot sufficiently express composable
actions. In this context, we propose Latent Action Composit... more
Skeleton-based action segmentation requires recognizing composable actions in
untrimmed videos. Current approaches decouple this problem by first extracting
local visual features from skeleton sequences and then processing them by a
temporal model to classify frame-wise actions. However, their performances
remain limited as the visual features cannot sufficiently express composable
actions. In this context, we propose Latent Action Composition (LAC), a novel
self-supervised framework aiming at learning from synthesized composable
motions for skeleton-based action segmentation. LAC is composed of a novel
generation module towards synthesizing new sequences. Specifically, we design a
linear latent space in the generator to represent primitive motion. New
composed motions can be synthesized by simply performing arithmetic operations
on latent representations of multiple input skeleton sequences. LAC leverages
such synthesized sequences, which have large diversity and complexity, for
learning visual representations of skeletons in both sequence and frame spaces
via contrastive learning. The resulting visual encoder has a high expressive
power and can be effectively transferred onto action segmentation tasks by
end-to-end fine-tuning without the need for additional temporal models. We
conduct a study focusing on transfer-learning and we show that representations
learned from pre-trained LAC outperform the state-of-the-art by a large margin
on TSU, Charades, PKU-MMD datasets.
less
Structural Cycle GAN for Virtual Immunohistochemistry Staining of Gland Markers in the Colon
0upvotes
By: Shikha Dubey, Tushar Kataria, Beatrice Knudsen, Shireen Y. Elhabian
With the advent of digital scanners and deep learning, diagnostic operations
may move from a microscope to a desktop. Hematoxylin and Eosin (H&E) staining
is one of the most frequently used stains for disease analysis, diagnosis, and
grading, but pathologists do need different immunohistochemical (IHC) stains to
analyze specific structures or cells. Obtaining all of these stains (H&E and
different IHCs) on a single specimen is a tedious and... more
With the advent of digital scanners and deep learning, diagnostic operations
may move from a microscope to a desktop. Hematoxylin and Eosin (H&E) staining
is one of the most frequently used stains for disease analysis, diagnosis, and
grading, but pathologists do need different immunohistochemical (IHC) stains to
analyze specific structures or cells. Obtaining all of these stains (H&E and
different IHCs) on a single specimen is a tedious and time-consuming task.
Consequently, virtual staining has emerged as an essential research direction.
Here, we propose a novel generative model, Structural Cycle-GAN (SC-GAN), for
synthesizing IHC stains from H&E images, and vice versa. Our method expressly
incorporates structural information in the form of edges (in addition to color
data) and employs attention modules exclusively in the decoder of the proposed
generator model. This integration enhances feature localization and preserves
contextual information during the generation process. In addition, a structural
loss is incorporated to ensure accurate structure alignment between the
generated and input markers. To demonstrate the efficacy of the proposed model,
experiments are conducted with two IHC markers emphasizing distinct structures
of glands in the colon: the nucleus of epithelial cells (CDX2) and the
cytoplasm (CK818). Quantitative metrics such as FID and SSIM are frequently
used for the analysis of generative models, but they do not correlate
explicitly with higher-quality virtual staining results. Therefore, we propose
two new quantitative metrics that correlate directly with the virtual staining
specificity of IHC markers.
less
SAAN: Similarity-aware attention flow network for change detection with VHR remote sensing images
0upvotes
By: Haonan Guo, Xin Su, Chen Wu, Bo Du, Liangpei Zhang
Change detection (CD) is a fundamental and important task for monitoring the
land surface dynamics in the earth observation field. Existing deep
learning-based CD methods typically extract bi-temporal image features using a
weight-sharing Siamese encoder network and identify change regions using a
decoder network. These CD methods, however, still perform far from
satisfactorily as we observe that 1) deep encoder layers focus on irrelevant
b... more
Change detection (CD) is a fundamental and important task for monitoring the
land surface dynamics in the earth observation field. Existing deep
learning-based CD methods typically extract bi-temporal image features using a
weight-sharing Siamese encoder network and identify change regions using a
decoder network. These CD methods, however, still perform far from
satisfactorily as we observe that 1) deep encoder layers focus on irrelevant
background regions and 2) the models' confidence in the change regions is
inconsistent at different decoder stages. The first problem is because deep
encoder layers cannot effectively learn from imbalanced change categories using
the sole output supervision, while the second problem is attributed to the lack
of explicit semantic consistency preservation. To address these issues, we
design a novel similarity-aware attention flow network (SAAN). SAAN
incorporates a similarity-guided attention flow module with deeply supervised
similarity optimization to achieve effective change detection. Specifically, we
counter the first issue by explicitly guiding deep encoder layers to discover
semantic relations from bi-temporal input images using deeply supervised
similarity optimization. The extracted features are optimized to be
semantically similar in the unchanged regions and dissimilar in the changing
regions. The second drawback can be alleviated by the proposed
similarity-guided attention flow module, which incorporates similarity-guided
attention modules and attention flow mechanisms to guide the model to focus on
discriminative channels and regions. We evaluated the effectiveness and
generalization ability of the proposed method by conducting experiments on a
wide range of CD tasks. The experimental results demonstrate that our method
achieves excellent performance on several CD tasks, with discriminative
features and semantic consistency preserved.
less
By: Matthew Dutson, Yin Li, Mohit Gupta
Vision Transformers achieve impressive accuracy across a range of visual
recognition tasks. Unfortunately, their accuracy frequently comes with high
computational costs. This is a particular issue in video recognition, where
models are often applied repeatedly across frames or temporal chunks. In this
work, we exploit temporal redundancy between subsequent inputs to reduce the
cost of Transformers for video processing. We describe a method ... more
Vision Transformers achieve impressive accuracy across a range of visual
recognition tasks. Unfortunately, their accuracy frequently comes with high
computational costs. This is a particular issue in video recognition, where
models are often applied repeatedly across frames or temporal chunks. In this
work, we exploit temporal redundancy between subsequent inputs to reduce the
cost of Transformers for video processing. We describe a method for identifying
and re-processing only those tokens that have changed significantly over time.
Our proposed family of models, Eventful Transformers, can be converted from
existing Transformers (often without any re-training) and give adaptive control
over the compute cost at runtime. We evaluate our method on large-scale
datasets for video object detection (ImageNet VID) and action recognition
(EPIC-Kitchens 100). Our approach leads to significant computational savings
(on the order of 2-4x) with only minor reductions in accuracy.
less
By: U Mong Sain Chak, Md. Asib Rahman
In this research paper, we present an innovative system designed for the
purpose of segmenting the layout of Bangla documents. Our methodology involves
utilizing a sophisticated collection of YOLOv8 models, meticulously adapted for
the DL Sprint 2.0 - BUET CSE Fest 2023 Competition that centers around Bangla
document layout segmentation. Our primary focus lies in elevating various
elements of the task, including techniques like image augmen... more
In this research paper, we present an innovative system designed for the
purpose of segmenting the layout of Bangla documents. Our methodology involves
utilizing a sophisticated collection of YOLOv8 models, meticulously adapted for
the DL Sprint 2.0 - BUET CSE Fest 2023 Competition that centers around Bangla
document layout segmentation. Our primary focus lies in elevating various
elements of the task, including techniques like image augmentation, model
architecture, and the use of model ensembles. We intentionally lower the
quality of a subset of document images to enhance the resilience of model
training, consequently leading to an improvement in our cross-validation score.
Employing Bayesian optimization, we determine the optimal confidence and IoU
thresholds for our model ensemble. Through our approach, we successfully
showcase the effectiveness of amalgamating anchor-free models to achieve robust
layout segmentation in Bangla documents.
less
By: Yufei Wang, Yuxin Mao, Qi Liu, Yuchao Dai
RGB-guided depth completion aims at predicting dense depth maps from sparse
depth measurements and corresponding RGB images, where how to effectively and
efficiently exploit the multi-modal information is a key issue. Guided dynamic
filters, which generate spatially-variant depth-wise separable convolutional
filters from RGB features to guide depth features, have been proven to be
effective in this task. However, the dynamically generated f... more
RGB-guided depth completion aims at predicting dense depth maps from sparse
depth measurements and corresponding RGB images, where how to effectively and
efficiently exploit the multi-modal information is a key issue. Guided dynamic
filters, which generate spatially-variant depth-wise separable convolutional
filters from RGB features to guide depth features, have been proven to be
effective in this task. However, the dynamically generated filters require
massive model parameters, computational costs and memory footprints when the
number of feature channels is large. In this paper, we propose to decompose the
guided dynamic filters into a spatially-shared component multiplied by
content-adaptive adaptors at each spatial location. Based on the proposed idea,
we introduce two decomposition schemes A and B, which decompose the filters by
splitting the filter structure and using spatial-wise attention, respectively.
The decomposed filters not only maintain the favorable properties of guided
dynamic filters as being content-dependent and spatially-variant, but also
reduce model parameters and hardware costs, as the learned adaptors are
decoupled with the number of feature channels. Extensive experimental results
demonstrate that the methods using our schemes outperform state-of-the-art
methods on the KITTI dataset, and rank 1st and 2nd on the KITTI benchmark at
the time of submission. Meanwhile, they also achieve comparable performance on
the NYUv2 dataset. In addition, our proposed methods are general and could be
employed as plug-and-play feature fusion blocks in other multi-modal fusion
tasks such as RGB-D salient object detection.
less
By: Misgina Tsighe Hagos, Niamh Belton, Kathleen M. Curran, Brian Mac Namee
eXplanation Based Learning (XBL) is an interactive learning approach that
provides a transparent method of training deep learning models by interacting
with their explanations. XBL augments loss functions to penalize a model based
on deviation of its explanations from user annotation of image features. The
literature on XBL mostly depends on the intersection of visual model
explanations and image feature annotations. We present a method to ... more
eXplanation Based Learning (XBL) is an interactive learning approach that
provides a transparent method of training deep learning models by interacting
with their explanations. XBL augments loss functions to penalize a model based
on deviation of its explanations from user annotation of image features. The
literature on XBL mostly depends on the intersection of visual model
explanations and image feature annotations. We present a method to add a
distance-aware explanation loss to categorical losses that trains a learner to
focus on important regions of a training dataset. Distance is an appropriate
approach for calculating explanation loss since visual model explanations such
as Gradient-weighted Class Activation Mapping (Grad-CAMs) are not strictly
bounded as annotations and their intersections may not provide complete
information on the deviation of a model's focus from relevant image regions. In
addition to assessing our model using existing metrics, we propose an
interpretability metric for evaluating visual feature-attribution based model
explanations that is more informative of the model's performance than existing
metrics. We demonstrate performance of our proposed method on three image
classification tasks.
less
By: Shuning Xu, Xiangyu Chen, Binbin Song, Jiantao Zhou
Capturing images with incorrect exposure settings fails to deliver a
satisfactory visual experience. Only when the exposure is properly set, can the
color and details of the images be appropriately preserved. Previous exposure
correction methods based on convolutions often produce exposure deviation in
images as a consequence of the restricted receptive field of convolutional
kernels. This issue arises because convolutions are not capable o... more
Capturing images with incorrect exposure settings fails to deliver a
satisfactory visual experience. Only when the exposure is properly set, can the
color and details of the images be appropriately preserved. Previous exposure
correction methods based on convolutions often produce exposure deviation in
images as a consequence of the restricted receptive field of convolutional
kernels. This issue arises because convolutions are not capable of capturing
long-range dependencies in images accurately. To overcome this challenge, we
can apply the Transformer to address the exposure correction problem,
leveraging its capability in modeling long-range dependencies to capture global
representation. However, solely relying on the window-based Transformer leads
to visually disturbing blocking artifacts due to the application of
self-attention in small patches. In this paper, we propose a CNN Injected
Transformer (CIT) to harness the individual strengths of CNN and Transformer
simultaneously. Specifically, we construct the CIT by utilizing a window-based
Transformer to exploit the long-range interactions among different regions in
the entire image. Within each CIT block, we incorporate a channel attention
block (CAB) and a half-instance normalization block (HINB) to assist the
window-based self-attention to acquire the global statistics and refine local
features. In addition to the hybrid architecture design for exposure
correction, we apply a set of carefully formulated loss functions to improve
the spatial coherence and rectify potential color deviations. Extensive
experiments demonstrate that our image exposure correction method outperforms
state-of-the-art approaches in terms of both quantitative and qualitative
metrics.
less
Attention-Guided Lidar Segmentation and Odometry Using Image-to-Point Cloud Saliency Transfer
0upvotes
By: Guanqun Ding, Nevrez Imamoglu, Ali Caglayan, Masahiro Murakawa, Ryosuke Nakamura
LiDAR odometry estimation and 3D semantic segmentation are crucial for
autonomous driving, which has achieved remarkable advances recently. However,
these tasks are challenging due to the imbalance of points in different
semantic categories for 3D semantic segmentation and the influence of dynamic
objects for LiDAR odometry estimation, which increases the importance of using
representative/salient landmarks as reference points for robust fe... more
LiDAR odometry estimation and 3D semantic segmentation are crucial for
autonomous driving, which has achieved remarkable advances recently. However,
these tasks are challenging due to the imbalance of points in different
semantic categories for 3D semantic segmentation and the influence of dynamic
objects for LiDAR odometry estimation, which increases the importance of using
representative/salient landmarks as reference points for robust feature
learning. To address these challenges, we propose a saliency-guided approach
that leverages attention information to improve the performance of LiDAR
odometry estimation and semantic segmentation models. Unlike in the image
domain, only a few studies have addressed point cloud saliency information due
to the lack of annotated training data. To alleviate this, we first present a
universal framework to transfer saliency distribution knowledge from color
images to point clouds, and use this to construct a pseudo-saliency dataset
(i.e. FordSaliency) for point clouds. Then, we adopt point cloud-based
backbones to learn saliency distribution from pseudo-saliency labels, which is
followed by our proposed SalLiDAR module. SalLiDAR is a saliency-guided 3D
semantic segmentation model that integrates saliency information to improve
segmentation performance. Finally, we introduce SalLONet, a self-supervised
saliency-guided LiDAR odometry network that uses the semantic and saliency
predictions of SalLiDAR to achieve better odometry estimation. Our extensive
experiments on benchmark datasets demonstrate that the proposed SalLiDAR and
SalLONet models achieve state-of-the-art performance against existing methods,
highlighting the effectiveness of image-to-LiDAR saliency knowledge transfer.
Source code will be available at https://github.com/nevrez/SalLONet.
less
FLDNet: A Foreground-Aware Network for Polyp Segmentation Leveraging Long-Distance Dependencies
0upvotes
By: Xuefeng Wei, Xuan Zhou
Given the close association between colorectal cancer and polyps, the
diagnosis and identification of colorectal polyps play a critical role in the
detection and surgical intervention of colorectal cancer. In this context, the
automatic detection and segmentation of polyps from various colonoscopy images
has emerged as a significant problem that has attracted broad attention.
Current polyp segmentation techniques face several challenges: fi... more
Given the close association between colorectal cancer and polyps, the
diagnosis and identification of colorectal polyps play a critical role in the
detection and surgical intervention of colorectal cancer. In this context, the
automatic detection and segmentation of polyps from various colonoscopy images
has emerged as a significant problem that has attracted broad attention.
Current polyp segmentation techniques face several challenges: firstly, polyps
vary in size, texture, color, and pattern; secondly, the boundaries between
polyps and mucosa are usually blurred, existing studies have focused on
learning the local features of polyps while ignoring the long-range
dependencies of the features, and also ignoring the local context and global
contextual information of the combined features. To address these challenges,
we propose FLDNet (Foreground-Long-Distance Network), a Transformer-based
neural network that captures long-distance dependencies for accurate polyp
segmentation. Specifically, the proposed model consists of three main modules:
a pyramid-based Transformer encoder, a local context module, and a
foreground-Aware module. Multilevel features with long-distance dependency
information are first captured by the pyramid-based transformer encoder. On the
high-level features, the local context module obtains the local characteristics
related to the polyps by constructing different local context information. The
coarse map obtained by decoding the reconstructed highest-level features guides
the feature fusion process in the foreground-Aware module of the high-level
features to achieve foreground enhancement of the polyps. Our proposed method,
FLDNet, was evaluated using seven metrics on common datasets and demonstrated
superiority over state-of-the-art methods on widely-used evaluation measures.
less
By: Bilel Benjdira, Anis Koubaa, Ahmad Taher Azar, Zahid Khan, Adel Ammar, Wadii Boulila
Smart traffic engineering and intelligent transportation services are in increasing demand from governmental
authorities to optimize traffic performance and thus reduce energy costs, increase the drivers’ safety and
comfort, ensure traffic laws enforcement, and detect traffic violations. In this paper, we address this challenge, and we leverage the use of Artificial Intelligence (AI) and Unmanned Aerial Vehicles (UAVs) to develop an AI-integr... more
Smart traffic engineering and intelligent transportation services are in increasing demand from governmental
authorities to optimize traffic performance and thus reduce energy costs, increase the drivers’ safety and
comfort, ensure traffic laws enforcement, and detect traffic violations. In this paper, we address this challenge, and we leverage the use of Artificial Intelligence (AI) and Unmanned Aerial Vehicles (UAVs) to develop an AI-integrated video analytics framework, called TAU (Traffic Analysis from UAVs), for automated traffic analytics and understanding. Unlike previous works on traffic video analytics, we propose an automated object detection and tracking pipeline from video processing to advanced traffic understanding using high-resolution UAV images. TAU combines six main contributions. First, it proposes a pre-processing algorithm to adapt the high- resolution UAV image as input to the object detector without lowering the resolution. This ensures an excellent detection accuracy from high-quality features, particularly the small size of detected objects from UAV images. Second, it introduces an algorithm for recalibrating the vehicle coordinates to ensure that vehicles are uniquely identified and tracked across the multiple crops of the same frame. Third, it presents a speed calculation algorithm based on accumulating information from successive frames. Fourth, TAU counts the number of vehicles per traffic zone based on the Ray Tracing algorithm. Fifth, TAU has a fully independent algorithm for crossroad arbitration based on the data gathered from the different zones surrounding it. Sixth, TAU introduces a set of algorithms for extracting twenty-four types of insights from the raw data collected. These insights facilitate the traffic understanding using curves, histograms, heatmaps, and animations. The present work presents a valuable added value for academic researchers and transportation engineers to automate the traffic video analytics process and extract useful insights to optimize traffic performance. TAU is a ready- to-use framework for any Transportation Engineer to better understand and manage daily road traffic (video demonstrations are provided here: https://youtu.be/wXJV0H7LviU and here: https://youtu.be/kGv0gmtVEbI). The source code is available at: https://github.com/bilel-bj/TAU. less
By: Yan Sun, Xueling Feng, Liyan Ma, Long Hu, Mark Nixon
Gait recognition is a promising biometric technology for identification due
to its non-invasiveness and long-distance. However, external variations such as
clothing changes and viewpoint differences pose significant challenges to gait
recognition. Silhouette-based methods preserve body shape but neglect internal
structure information, while skeleton-based methods preserve structure
information but omit appearance. To fully exploit the compl... more
Gait recognition is a promising biometric technology for identification due
to its non-invasiveness and long-distance. However, external variations such as
clothing changes and viewpoint differences pose significant challenges to gait
recognition. Silhouette-based methods preserve body shape but neglect internal
structure information, while skeleton-based methods preserve structure
information but omit appearance. To fully exploit the complementary nature of
the two modalities, a novel triple branch gait recognition framework, TriGait,
is proposed in this paper. It effectively integrates features from the skeleton
and silhouette data in a hybrid fusion manner, including a two-stream network
to extract static and motion features from appearance, a simple yet effective
module named JSA-TC to capture dependencies between all joints, and a third
branch for cross-modal learning by aligning and fusing low-level features of
two modalities. Experimental results demonstrate the superiority and
effectiveness of TriGait for gait recognition. The proposed method achieves a
mean rank-1 accuracy of 96.0% over all conditions on CASIA-B dataset and 94.3%
accuracy for CL, significantly outperforming all the state-of-the-art methods.
The source code will be available at https://github.com/feng-xueling/TriGait/.
less
DCP-Net: A Distributed Collaborative Perception Network for Remote Sensing Semantic Segmentation
0upvotes
By: Zhechao Wang, Peirui Cheng, Shujing Duan, Kaiqiang Chen, Zhirui Wang, Xinming Li, Xian Sun
Onboard intelligent processing is widely applied in emergency tasks in the
field of remote sensing. However, it is predominantly confined to an individual
platform with a limited observation range as well as susceptibility to
interference, resulting in limited accuracy. Considering the current state of
multi-platform collaborative observation, this article innovatively presents a
distributed collaborative perception network called DCP-Net. ... more
Onboard intelligent processing is widely applied in emergency tasks in the
field of remote sensing. However, it is predominantly confined to an individual
platform with a limited observation range as well as susceptibility to
interference, resulting in limited accuracy. Considering the current state of
multi-platform collaborative observation, this article innovatively presents a
distributed collaborative perception network called DCP-Net. Firstly, the
proposed DCP-Net helps members to enhance perception performance by integrating
features from other platforms. Secondly, a self-mutual information match module
is proposed to identify collaboration opportunities and select suitable
partners, prioritizing critical collaborative features and reducing redundant
transmission cost. Thirdly, a related feature fusion module is designed to
address the misalignment between local and collaborative features, improving
the quality of fused features for the downstream task. We conduct extensive
experiments and visualization analyses using three semantic segmentation
datasets, including Potsdam, iSAID and DFC23. The results demonstrate that
DCP-Net outperforms the existing methods comprehensively, improving mIoU by
2.61%~16.89% at the highest collaboration efficiency, which promotes the
performance to a state-of-the-art level.
less
By: Emanuele Ballarin, Alessio Ansuini, Luca Bortolussi
In this paper, we propose a novel adversarial defence mechanism for image
classification -- CARSO -- inspired by cues from cognitive neuroscience. The
method is synergistically complementary to adversarial training and relies on
knowledge of the internal representation of the attacked classifier. Exploiting
a generative model for adversarial purification, conditioned on such
representation, it samples reconstructions of inputs to be finally... more
In this paper, we propose a novel adversarial defence mechanism for image
classification -- CARSO -- inspired by cues from cognitive neuroscience. The
method is synergistically complementary to adversarial training and relies on
knowledge of the internal representation of the attacked classifier. Exploiting
a generative model for adversarial purification, conditioned on such
representation, it samples reconstructions of inputs to be finally classified.
Experimental evaluation by a well-established benchmark of varied, strong
adaptive attacks, across diverse image datasets and classifier architectures,
shows that CARSO is able to defend the classifier significantly better than
state-of-the-art adversarial training alone -- with a tolerable clean accuracy
toll. Furthermore, the defensive architecture succeeds in effectively shielding
itself from unforeseen threats, and end-to-end attacks adapted to fool
stochastic defences. Code and pre-trained models are available at
https://github.com/emaballarin/CARSO .
less
By: Jingyi Zhang, Jiaxing Huang, Xueying Jiang, Shijian Lu
Black-box unsupervised domain adaptation (UDA) learns with source predictions
of target data without accessing either source data or source models during
training, and it has clear superiority in data privacy and flexibility in
target network selection. However, the source predictions of target data are
often noisy and training with them is prone to learning collapses. We propose
BiMem, a bi-directional memorization mechanism that learns to... more
Black-box unsupervised domain adaptation (UDA) learns with source predictions
of target data without accessing either source data or source models during
training, and it has clear superiority in data privacy and flexibility in
target network selection. However, the source predictions of target data are
often noisy and training with them is prone to learning collapses. We propose
BiMem, a bi-directional memorization mechanism that learns to remember useful
and representative information to correct noisy pseudo labels on the fly,
leading to robust black-box UDA that can generalize across different visual
recognition tasks. BiMem constructs three types of memory, including sensory
memory, short-term memory, and long-term memory, which interact in a
bi-directional manner for comprehensive and robust memorization of learnt
features. It includes a forward memorization flow that identifies and stores
useful features and a backward calibration flow that rectifies features' pseudo
labels progressively. Extensive experiments show that BiMem achieves superior
domain adaptation performance consistently across various visual recognition
tasks such as image classification, semantic segmentation and object detection.
less
By: Minda Zhao, Chaoyi Zhao, Xinyue Liang, Lincheng Li, Zeng Zhao, Zhipeng Hu, Changjie Fan, Xin Yu
While the image diffusion model has made significant strides in text-driven
3D content creation, it often falls short in accurately capturing the intended
meaning of the text prompt, particularly with respect to direction information.
This shortcoming gives rise to the Janus problem, where multi-faced 3D models
are produced with the guidance of such diffusion models. In this paper, we
present a robust pipeline for generating high-fidelity 3... more
While the image diffusion model has made significant strides in text-driven
3D content creation, it often falls short in accurately capturing the intended
meaning of the text prompt, particularly with respect to direction information.
This shortcoming gives rise to the Janus problem, where multi-faced 3D models
are produced with the guidance of such diffusion models. In this paper, we
present a robust pipeline for generating high-fidelity 3D content with
orthogonal-view image guidance. Specifically, we introduce a novel 2D diffusion
model that generates an image consisting of four orthogonal-view sub-images for
the given text prompt. The 3D content is then created with this diffusion
model, which enhances 3D consistency and provides strong structured semantic
priors. This addresses the infamous Janus problem and significantly promotes
generation efficiency. Additionally, we employ a progressive 3D synthesis
strategy that results in substantial improvement in the quality of the created
3D contents. Both quantitative and qualitative evaluations show that our method
demonstrates a significant improvement over previous text-to-3D techniques.
less
By: Jaehyun So, Youngjoon Han
The ability of humans to infer head poses from face shapes, and vice versa,
indicates a strong correlation between the two. Accordingly, recent studies on
face alignment have employed head pose information to predict facial landmarks
in computer vision tasks. In this study, we propose a novel method that employs
head pose information to improve face alignment performance by fusing said
information with the feature maps of a face alignment n... more
The ability of humans to infer head poses from face shapes, and vice versa,
indicates a strong correlation between the two. Accordingly, recent studies on
face alignment have employed head pose information to predict facial landmarks
in computer vision tasks. In this study, we propose a novel method that employs
head pose information to improve face alignment performance by fusing said
information with the feature maps of a face alignment network, rather than
simply using it to initialize facial landmarks. Furthermore, the proposed
network structure performs robust face alignment through a dual-dimensional
network using multidimensional features represented by 2D feature maps and a 3D
heatmap. For effective dense face alignment, we also propose a prediction
method for facial geometric landmarks through training based on knowledge
distillation using predicted keypoints. We experimentally assessed the
correlation between the predicted facial landmarks and head pose information,
as well as variations in the accuracy of facial landmarks with respect to the
quality of head pose information. In addition, we demonstrated the
effectiveness of the proposed method through a competitive performance
comparison with state-of-the-art methods on the AFLW2000-3D, AFLW, and BIWI
datasets.
less
By: Amr Gomaa, Robin Zitt, Guillermo Reyes, Antonio Krüger
Creating a diverse and comprehensive dataset of hand gestures for dynamic
human-machine interfaces in the automotive domain can be challenging and
time-consuming. To overcome this challenge, we propose using synthetic gesture
datasets generated by virtual 3D models. Our framework utilizes Unreal Engine
to synthesize realistic hand gestures, offering customization options and
reducing the risk of overfitting. Multiple variants, including ges... more
Creating a diverse and comprehensive dataset of hand gestures for dynamic
human-machine interfaces in the automotive domain can be challenging and
time-consuming. To overcome this challenge, we propose using synthetic gesture
datasets generated by virtual 3D models. Our framework utilizes Unreal Engine
to synthesize realistic hand gestures, offering customization options and
reducing the risk of overfitting. Multiple variants, including gesture speed,
performance, and hand shape, are generated to improve generalizability. In
addition, we simulate different camera locations and types, such as RGB,
infrared, and depth cameras, without incurring additional time and cost to
obtain these cameras. Experimental results demonstrate that our proposed
framework,
SynthoGestures\footnote{\url{https://github.com/amrgomaaelhady/SynthoGestures}},
improves gesture recognition accuracy and can replace or augment real-hand
datasets. By saving time and effort in the creation of the data set, our tool
accelerates the development of gesture recognition systems for automotive
applications.
less
By: Junfeng Cheng, Mingdong Wu, Ruiyuan Zhang, Guanqi Zhan, Chao Wu, Hao Dong
Autonomous 3D part assembly is a challenging task in the areas of robotics
and 3D computer vision. This task aims to assemble individual components into a
complete shape without relying on predefined instructions. In this paper, we
formulate this task from a novel generative perspective, introducing the
Score-based 3D Part Assembly framework (Score-PA) for 3D part assembly. Knowing
that score-based methods are typically time-consuming durin... more
Autonomous 3D part assembly is a challenging task in the areas of robotics
and 3D computer vision. This task aims to assemble individual components into a
complete shape without relying on predefined instructions. In this paper, we
formulate this task from a novel generative perspective, introducing the
Score-based 3D Part Assembly framework (Score-PA) for 3D part assembly. Knowing
that score-based methods are typically time-consuming during the inference
stage. To address this issue, we introduce a novel algorithm called the Fast
Predictor-Corrector Sampler (FPC) that accelerates the sampling process within
the framework. We employ various metrics to assess assembly quality and
diversity, and our evaluation results demonstrate that our algorithm
outperforms existing state-of-the-art approaches. We release our code at
https://github.com/J-F-Cheng/Score-PA_Score-based-3D-Part-Assembly.
less
By: Xiaotong Li, Zixuan Hu, Yixiao Ge, Ying Shan, Ling-Yu Duan
Transfer learning has become crucial in computer vision tasks due to the vast
availability of pre-trained deep learning models. However, selecting the
optimal pre-trained model from a diverse pool for a specific downstream task
remains a challenge. Existing methods for measuring the transferability of
pre-trained models rely on statistical correlations between encoded static
features and task labels, but they overlook the impact of underlyi... more
Transfer learning has become crucial in computer vision tasks due to the vast
availability of pre-trained deep learning models. However, selecting the
optimal pre-trained model from a diverse pool for a specific downstream task
remains a challenge. Existing methods for measuring the transferability of
pre-trained models rely on statistical correlations between encoded static
features and task labels, but they overlook the impact of underlying
representation dynamics during fine-tuning, leading to unreliable results,
especially for self-supervised models. In this paper, we present an insightful
physics-inspired approach named PED to address these challenges. We reframe the
challenge of model selection through the lens of potential energy and directly
model the interaction forces that influence fine-tuning dynamics. By capturing
the motion of dynamic representations to decline the potential energy within a
force-driven physical model, we can acquire an enhanced and more stable
observation for estimating transferability. The experimental results on 10
downstream tasks and 12 self-supervised models demonstrate that our approach
can seamlessly integrate into existing ranking techniques and enhance their
performances, revealing its effectiveness for the model selection task and its
potential for understanding the mechanism in transfer learning. Code will be
available at https://github.com/lixiaotong97/PED.
less
By: Pengcheng Dong, Xiaojin Mao, Lixia Fan, Wenbo Wan, Jiande Sun
In recent years, the assessment of fundamental movement skills integrated
with physical education has focused on both teaching practice and the
feasibility of assessment. The object of assessment has shifted from multiple
ages to subdivided ages, while the content of assessment has changed from
complex and time-consuming to concise and efficient. Therefore, we apply deep
learning to physical fitness evaluation, we propose a system based on ... more
In recent years, the assessment of fundamental movement skills integrated
with physical education has focused on both teaching practice and the
feasibility of assessment. The object of assessment has shifted from multiple
ages to subdivided ages, while the content of assessment has changed from
complex and time-consuming to concise and efficient. Therefore, we apply deep
learning to physical fitness evaluation, we propose a system based on the
Canadian Agility and Movement Skill Assessment (CAMSA) Physical Fitness
Evaluation System (CPFES), which evaluates children's physical fitness based on
CAMSA, and gives recommendations based on the scores obtained by CPFES to help
children grow. We have designed a landmark detection module and a pose
estimation module, and we have also designed a pose evaluation module for the
CAMSA criteria that can effectively evaluate the actions of the child being
tested. Our experimental results demonstrate the high accuracy of the proposed
system.
less
By: Nanne van Noord
Dataset summarisation is a fruitful approach to dataset inspection. However,
when applied to a single dataset the discovery of visual concepts is restricted
to those most prominent. We argue that a comparative approach can expand upon
this paradigm to enable richer forms of dataset inspection that go beyond the
most prominent concepts. To enable dataset comparison we present a module that
learns concept-level prototypes across datasets. We ... more
Dataset summarisation is a fruitful approach to dataset inspection. However,
when applied to a single dataset the discovery of visual concepts is restricted
to those most prominent. We argue that a comparative approach can expand upon
this paradigm to enable richer forms of dataset inspection that go beyond the
most prominent concepts. To enable dataset comparison we present a module that
learns concept-level prototypes across datasets. We leverage self-supervised
learning to discover these prototypes without supervision, and we demonstrate
the benefits of our approach in two case-studies. Our findings show that
dataset comparison extends dataset inspection and we hope to encourage more
works in this direction. Code and usage instructions available at
https://github.com/Nanne/ProtoSim
less
By: Sven Teufel, Jörg Gamerdinger, Georg Volk, Oliver Bringmann
Comprehensive perception of the environment is crucial for the safe operation
of autonomous vehicles. However, the perception capabilities of autonomous
vehicles are limited due to occlusions, limited sensor ranges, or environmental
influences. Collective Perception (CP) aims to mitigate these problems by
enabling the exchange of information between vehicles. A major challenge in CP
is the fusion of the exchanged information. Due to the eno... more
Comprehensive perception of the environment is crucial for the safe operation
of autonomous vehicles. However, the perception capabilities of autonomous
vehicles are limited due to occlusions, limited sensor ranges, or environmental
influences. Collective Perception (CP) aims to mitigate these problems by
enabling the exchange of information between vehicles. A major challenge in CP
is the fusion of the exchanged information. Due to the enormous bandwidth
requirement of early fusion approaches and the interchangeability issues of
intermediate fusion approaches, only the late fusion of shared detections is
practical. Current late fusion approaches neglect valuable information for
local detection, this is why we propose a novel fusion method to fuse the
detections of cooperative vehicles within the local LiDAR-based detection
pipeline. Therefore, we present Collective PV-RCNN (CPV-RCNN), which extends
the PV-RCNN++ framework to fuse collective detections. Code is available at
https://github.com/ekut-es
less
By: Zheng Wang, Dong Xie, Hanzhi Wang, Jiang Tian
Learning from the limited amount of labeled data to the pre-train model has
always been viewed as a challenging task. In this report, an effective and
robust solution, the two-stage training paradigm YOLOv8 detector (TP-YOLOv8),
is designed for the object detection track in VIPriors Challenge 2023. First,
the backbone of YOLOv8 is pre-trained as the encoder using the masked image
modeling technique. Then the detector is fine-tuned with elab... more
Learning from the limited amount of labeled data to the pre-train model has
always been viewed as a challenging task. In this report, an effective and
robust solution, the two-stage training paradigm YOLOv8 detector (TP-YOLOv8),
is designed for the object detection track in VIPriors Challenge 2023. First,
the backbone of YOLOv8 is pre-trained as the encoder using the masked image
modeling technique. Then the detector is fine-tuned with elaborate
augmentations. During the test stage, test-time augmentation (TTA) is used to
enhance each model, and weighted box fusion (WBF) is implemented to further
boost the performance. With the well-designed structure, our approach has
achieved 30.4% average precision from 0.50 to 0.95 on the DelftBikes test set,
ranking 4th on the leaderboard.
less
By: Eden Belouadah, Arnaud Dapogny, Kevin Bailly
Class-Incremental learning (CIL) is the ability of artificial agents to
accommodate new classes as they appear in a stream. It is particularly
interesting in evolving environments where agents have limited access to memory
and computational resources. The main challenge of class-incremental learning
is catastrophic forgetting, the inability of neural networks to retain past
knowledge when learning a new one. Unfortunately, most existing
cla... more
Class-Incremental learning (CIL) is the ability of artificial agents to
accommodate new classes as they appear in a stream. It is particularly
interesting in evolving environments where agents have limited access to memory
and computational resources. The main challenge of class-incremental learning
is catastrophic forgetting, the inability of neural networks to retain past
knowledge when learning a new one. Unfortunately, most existing
class-incremental object detectors are applied to two-stage algorithms such as
Faster-RCNN and rely on rehearsal memory to retain past knowledge. We believe
that the current benchmarks are not realistic, and more effort should be
dedicated to anchor-free and rehearsal-free object detection. In this context,
we propose MultIOD, a class-incremental object detector based on CenterNet. Our
main contributions are: (1) we propose a multihead feature pyramid and
multihead detection architecture to efficiently separate class representations,
(2) we employ transfer learning between classes learned initially and those
learned incrementally to tackle catastrophic forgetting, and (3) we use a
class-wise non-max-suppression as a post-processing technique to remove
redundant boxes. Without bells and whistles, our method outperforms a range of
state-of-the-art methods on two Pascal VOC datasets.
less
By: Alexander Lehner, Stefano Gasperini, Alvaro Marcos-Ramiro, Michael Schmidt, Nassir Navab, Benjamin Busam, Federico Tombari
Since real-world training datasets cannot properly sample the long tail of
the underlying data distribution, corner cases and rare out-of-domain samples
can severely hinder the performance of state-of-the-art models. This problem
becomes even more severe for dense tasks, such as 3D semantic segmentation,
where points of non-standard objects can be confidently associated to the wrong
class. In this work, we focus on improving the generalizat... more
Since real-world training datasets cannot properly sample the long tail of
the underlying data distribution, corner cases and rare out-of-domain samples
can severely hinder the performance of state-of-the-art models. This problem
becomes even more severe for dense tasks, such as 3D semantic segmentation,
where points of non-standard objects can be confidently associated to the wrong
class. In this work, we focus on improving the generalization to out-of-domain
data. We achieve this by augmenting the training set with adversarial examples.
First, we learn a set of vectors that deform the objects in an adversarial
fashion. To prevent the adversarial examples from being too far from the
existing data distribution, we preserve their plausibility through a series of
constraints, ensuring sensor-awareness and shapes smoothness. Then, we perform
adversarial augmentation by applying the learned sample-independent vectors to
the available objects when training a model. We conduct extensive experiments
across a variety of scenarios on data from KITTI, Waymo, and CrashD for 3D
object detection, and on data from SemanticKITTI, Waymo, and nuScenes for 3D
semantic segmentation. Despite training on a standard single dataset, our
approach substantially improves the robustness and generalization of both 3D
object detection and 3D semantic segmentation methods to out-of-domain data.
less
By: Chong Zeng, Guojun Chen, Yue Dong, Pieter Peers, Hongzhi Wu, Xin Tong
This paper presents a novel neural implicit radiance representation for free
viewpoint relighting from a small set of unstructured photographs of an object
lit by a moving point light source different from the view position. We express
the shape as a signed distance function modeled by a multi layer perceptron. In
contrast to prior relightable implicit neural representations, we do not
disentangle the different reflectance components, but m... more
This paper presents a novel neural implicit radiance representation for free
viewpoint relighting from a small set of unstructured photographs of an object
lit by a moving point light source different from the view position. We express
the shape as a signed distance function modeled by a multi layer perceptron. In
contrast to prior relightable implicit neural representations, we do not
disentangle the different reflectance components, but model both the local and
global reflectance at each point by a second multi layer perceptron that, in
addition, to density features, the current position, the normal (from the
signed distace function), view direction, and light position, also takes shadow
and highlight hints to aid the network in modeling the corresponding high
frequency light transport effects. These hints are provided as a suggestion,
and we leave it up to the network to decide how to incorporate these in the
final relit result. We demonstrate and validate our neural implicit
representation on synthetic and real scenes exhibiting a wide variety of
shapes, material properties, and global illumination light transport.
less
By: Cheng-Che Cheng, Min-Xuan Qiu, Chen-Kuo Chiang, Shang-Hong Lai
Multi-Camera Multi-Object Tracking (MC-MOT) utilizes information from
multiple views to better handle problems with occlusion and crowded scenes.
Recently, the use of graph-based approaches to solve tracking problems has
become very popular. However, many current graph-based methods do not
effectively utilize information regarding spatial and temporal consistency.
Instead, they rely on single-camera trackers as input, which are prone to
fra... more
Multi-Camera Multi-Object Tracking (MC-MOT) utilizes information from
multiple views to better handle problems with occlusion and crowded scenes.
Recently, the use of graph-based approaches to solve tracking problems has
become very popular. However, many current graph-based methods do not
effectively utilize information regarding spatial and temporal consistency.
Instead, they rely on single-camera trackers as input, which are prone to
fragmentation and ID switch errors. In this paper, we propose a novel
reconfigurable graph model that first associates all detected objects across
cameras spatially before reconfiguring it into a temporal graph for Temporal
Association. This two-stage association approach enables us to extract robust
spatial and temporal-aware features and address the problem with fragmented
tracklets. Furthermore, our model is designed for online tracking, making it
suitable for real-world applications. Experimental results show that the
proposed graph model is able to extract more discriminating features for object
tracking, and our model achieves state-of-the-art performance on several public
datasets.
less
By: Erik Daxberger, Floris Weers, Bowen Zhang, Tom Gunter, Ruoming Pang, Marcin Eichner, Michael Emmersberger, Yinfei Yang, Alexander Toshev, Xianzhi Du
Sparse Mixture-of-Experts models (MoEs) have recently gained popularity due
to their ability to decouple model size from inference efficiency by only
activating a small subset of the model parameters for any given input token. As
such, sparse MoEs have enabled unprecedented scalability, resulting in
tremendous successes across domains such as natural language processing and
computer vision. In this work, we instead explore the use of sparse... more
Sparse Mixture-of-Experts models (MoEs) have recently gained popularity due
to their ability to decouple model size from inference efficiency by only
activating a small subset of the model parameters for any given input token. As
such, sparse MoEs have enabled unprecedented scalability, resulting in
tremendous successes across domains such as natural language processing and
computer vision. In this work, we instead explore the use of sparse MoEs to
scale-down Vision Transformers (ViTs) to make them more attractive for
resource-constrained vision applications. To this end, we propose a simplified
and mobile-friendly MoE design where entire images rather than individual
patches are routed to the experts. We also propose a stable MoE training
procedure that uses super-class information to guide the router. We empirically
show that our sparse Mobile Vision MoEs (V-MoEs) can achieve a better trade-off
between performance and efficiency than the corresponding dense ViTs. For
example, for the ViT-Tiny model, our Mobile V-MoE outperforms its dense
counterpart by 3.39% on ImageNet-1k. For an even smaller ViT variant with only
54M FLOPs inference cost, our MoE achieves an improvement of 4.66%.
less
By: Yibo Wang, Yunhu Ye, Yuanpeng Mao, Yanwei Yu, Yuanping Song
Text segmentation tasks have a very wide range of application values, such as
image editing, style transfer, watermark removal, etc.However, existing public
datasets are of poor quality of pixel-level labels that have been shown to be
notoriously costly to acquire, both in terms of money and time. At the same
time, when pretraining is performed on synthetic datasets, the data
distribution of the synthetic datasets is far from the data distr... more
Text segmentation tasks have a very wide range of application values, such as
image editing, style transfer, watermark removal, etc.However, existing public
datasets are of poor quality of pixel-level labels that have been shown to be
notoriously costly to acquire, both in terms of money and time. At the same
time, when pretraining is performed on synthetic datasets, the data
distribution of the synthetic datasets is far from the data distribution in the
real scene. These all pose a huge challenge to the current pixel-level text
segmentation algorithms.To alleviate the above problems, we propose a
self-supervised scene text segmentation algorithm with layered decoupling of
representations derived from the object-centric manner to segment images into
texts and background. In our method, we propose two novel designs which include
Region Query Module and Representation Consistency Constraints adapting to the
unique properties of text as complements to Auto Encoder, which improves the
network's sensitivity to texts.For this unique design, we treat the
polygon-level masks predicted by the text localization model as extra input
information, and neither utilize any pixel-level mask annotations for training
stage nor pretrain on synthetic datasets.Extensive experiments show the
effectiveness of the method proposed. On several public scene text datasets,
our method outperforms the state-of-the-art unsupervised segmentation
algorithms.
less
By: Clifford Broni-Bediako, Junshi Xia, Naoto Yokoya
Real-time semantic segmentation of remote sensing imagery is a challenging
task that requires a trade-off between effectiveness and efficiency. It has
many applications including tracking forest fires, detecting changes in land
use and land cover, crop health monitoring, and so on. With the success of
efficient deep learning methods (i.e., efficient deep neural networks) for
real-time semantic segmentation in computer vision, researchers ha... more
Real-time semantic segmentation of remote sensing imagery is a challenging
task that requires a trade-off between effectiveness and efficiency. It has
many applications including tracking forest fires, detecting changes in land
use and land cover, crop health monitoring, and so on. With the success of
efficient deep learning methods (i.e., efficient deep neural networks) for
real-time semantic segmentation in computer vision, researchers have adopted
these efficient deep neural networks in remote sensing image analysis. This
paper begins with a summary of the fundamental compression methods for
designing efficient deep neural networks and provides a brief but comprehensive
survey, outlining the recent developments in real-time semantic segmentation of
remote sensing imagery. We examine several seminal efficient deep learning
methods, placing them in a taxonomy based on the network architecture design
approach. Furthermore, we evaluate the quality and efficiency of some existing
efficient deep neural networks on a publicly available remote sensing semantic
segmentation benchmark dataset, the OpenEarthMap. The experimental results of
an extensive comparative study demonstrate that most of the existing efficient
deep neural networks have good segmentation quality, but they suffer low
inference speed (i.e., high latency rate), which may limit their capability of
deployment in real-time applications of remote sensing image segmentation. We
provide some insights into the current trend and future research directions for
real-time semantic segmentation of remote sensing imagery.
less
By: Tai D. Nguyen, Shengbang Fang, Matthew C. Stamm
Fake videos represent an important misinformation threat. While existing
forensic networks have demonstrated strong performance on image forgeries,
recent results reported on the Adobe VideoSham dataset show that these networks
fail to identify fake content in videos. In this paper, we show that this is
due to video coding, which introduces local variation into forensic traces. In
response, we propose VideoFACT - a new network that is able ... more
Fake videos represent an important misinformation threat. While existing
forensic networks have demonstrated strong performance on image forgeries,
recent results reported on the Adobe VideoSham dataset show that these networks
fail to identify fake content in videos. In this paper, we show that this is
due to video coding, which introduces local variation into forensic traces. In
response, we propose VideoFACT - a new network that is able to detect and
localize a wide variety of video forgeries and manipulations. To overcome
challenges that existing networks face when analyzing videos, our network
utilizes both forensic embeddings to capture traces left by manipulation,
context embeddings to control for variation in forensic traces introduced by
video coding, and a deep self-attention mechanism to estimate the quality and
relative importance of local forensic embeddings. We create several new video
forgery datasets and use these, along with publicly available data, to
experimentally evaluate our network's performance. These results show that our
proposed network is able to identify a diverse set of video forgeries,
including those not encountered during training. Furthermore, we show that our
network can be fine-tuned to achieve even stronger performance on challenging
AI-based manipulations.
less
LEyes: A Lightweight Framework for Deep Learning-Based Eye Tracking using Synthetic Eye Images
0upvotes
By: sean anthony byrne, virmarie maquiling, marcus nyström, enkelejda kasneci, diederick c. niehorster
Deep learning has bolstered gaze estimation techniques, but real-world
deployment has been impeded by inadequate training datasets. This problem is
exacerbated by both hardware-induced variations in eye images and inherent
biological differences across the recorded participants, leading to both
feature and pixel-level variance that hinders the generalizability of models
trained on specific datasets. While synthetic datasets can be a solutio... more
Deep learning has bolstered gaze estimation techniques, but real-world
deployment has been impeded by inadequate training datasets. This problem is
exacerbated by both hardware-induced variations in eye images and inherent
biological differences across the recorded participants, leading to both
feature and pixel-level variance that hinders the generalizability of models
trained on specific datasets. While synthetic datasets can be a solution, their
creation is both time and resource-intensive. To address this problem, we
present a framework called Light Eyes or "LEyes" which, unlike conventional
photorealistic methods, only models key image features required for video-based
eye tracking using simple light distributions. LEyes facilitates easy
configuration for training neural networks across diverse gaze-estimation
tasks. We demonstrate that models trained using LEyes outperform other
state-of-the-art algorithms in terms of pupil and CR localization across
well-known datasets. In addition, a LEyes trained model outperforms the
industry standard eye tracker using significantly more cost-effective hardware.
Going forward, we are confident that LEyes will revolutionize synthetic data
generation for gaze estimation models, and lead to significant improvements of
the next generation video-based eye trackers.
less
By: Yifan Song, Mengkun She, Kevin Köser
Underwater image restoration has been a challenging problem for decades since
the advent of underwater photography. Most solutions focus on shallow water
scenarios, where the scene is uniformly illuminated by the sunlight. However,
the vast majority of uncharted underwater terrain is located beyond 200 meters
depth where natural light is scarce and artificial illumination is needed. In
such cases, light sources co-moving with the camera, dy... more
Underwater image restoration has been a challenging problem for decades since
the advent of underwater photography. Most solutions focus on shallow water
scenarios, where the scene is uniformly illuminated by the sunlight. However,
the vast majority of uncharted underwater terrain is located beyond 200 meters
depth where natural light is scarce and artificial illumination is needed. In
such cases, light sources co-moving with the camera, dynamically change the
scene appearance, which make shallow water restoration methods inadequate. In
particular for multi-light source systems (composed of dozens of LEDs
nowadays), calibrating each light is time-consuming, error-prone and tedious,
and we observe that only the integrated illumination within the viewing volume
of the camera is critical, rather than the individual light sources. The key
idea of this paper is therefore to exploit the appearance changes of objects or
the seafloor, when traversing the viewing frustum of the camera. Through new
constraints assuming Lambertian surfaces, corresponding image pixels constrain
the light field in front of the camera, and for each voxel a signal factor and
a backscatter value are stored in a volumetric grid that can be used for very
efficient image restoration of camera-light platforms, which facilitates
consistently texturing large 3D models and maps that would otherwise be
dominated by lighting and medium artifacts. To validate the effectiveness of
our approach, we conducted extensive experiments on simulated and real-world
datasets. The results of these experiments demonstrate the robustness of our
approach in restoring the true albedo of objects, while mitigating the
influence of lighting and medium effects. Furthermore, we demonstrate our
approach can be readily extended to other scenarios, including in-air imaging
with artificial illumination or other similar cases.
less
By: Zhixin Ling, Zhen Xing, Xiangdong Zhou, Manliang Cao, Guichun Zhou
In panorama understanding, the widely used equirectangular projection (ERP)
entails boundary discontinuity and spatial distortion. It severely deteriorates
the conventional CNNs and vision Transformers on panoramas. In this paper, we
propose a simple yet effective architecture named PanoSwin to learn panorama
representations with ERP. To deal with the challenges brought by
equirectangular projection, we explore a pano-style shift windowing ... more
In panorama understanding, the widely used equirectangular projection (ERP)
entails boundary discontinuity and spatial distortion. It severely deteriorates
the conventional CNNs and vision Transformers on panoramas. In this paper, we
propose a simple yet effective architecture named PanoSwin to learn panorama
representations with ERP. To deal with the challenges brought by
equirectangular projection, we explore a pano-style shift windowing scheme and
novel pitch attention to address the boundary discontinuity and the spatial
distortion, respectively. Besides, based on spherical distance and Cartesian
coordinates, we adapt absolute positional embeddings and relative positional
biases for panoramas to enhance panoramic geometry information. Realizing that
planar image understanding might share some common knowledge with panorama
understanding, we devise a novel two-stage learning framework to facilitate
knowledge transfer from the planar images to panoramas. We conduct experiments
against the state-of-the-art on various panoramic tasks, i.e., panoramic object
detection, panoramic classification, and panoramic layout estimation. The
experimental results demonstrate the effectiveness of PanoSwin in panorama
understanding.
less
By: Jiaxin Gao, Ziyu Yue, Yaohua Liu, Sihan Xie, Xin Fan, Risheng Liu
Super-resolution tasks oriented to images captured in ultra-dark environments
is a practical yet challenging problem that has received little attention. Due
to uneven illumination and low signal-to-noise ratio in dark environments, a
multitude of problems such as lack of detail and color distortion may be
magnified in the super-resolution process compared to normal-lighting
environments. Consequently, conventional low-light enhancement or
s... more
Super-resolution tasks oriented to images captured in ultra-dark environments
is a practical yet challenging problem that has received little attention. Due
to uneven illumination and low signal-to-noise ratio in dark environments, a
multitude of problems such as lack of detail and color distortion may be
magnified in the super-resolution process compared to normal-lighting
environments. Consequently, conventional low-light enhancement or
super-resolution methods, whether applied individually or in a cascaded manner
for such problem, often encounter limitations in recovering luminance, color
fidelity, and intricate details. To conquer these issues, this paper proposes a
specialized dual-modulated learning framework that, for the first time,
attempts to deeply dissect the nature of the low-light super-resolution task.
Leveraging natural image color characteristics, we introduce a self-regularized
luminance constraint as a prior for addressing uneven lighting. Expanding on
this, we develop Illuminance-Semantic Dual Modulation (ISDM) components to
enhance feature-level preservation of illumination and color details. Besides,
instead of deploying naive up-sampling strategies, we design the
Resolution-Sensitive Merging Up-sampler (RSMU) module that brings together
different sampling modalities as substrates, effectively mitigating the
presence of artifacts and halos. Comprehensive experiments showcases the
applicability and generalizability of our approach to diverse and challenging
ultra-low-light conditions, outperforming state-of-the-art methods with a
notable improvement (i.e., $\uparrow$5\% in PSNR, and $\uparrow$43\% in LPIPS).
Especially noteworthy is the 19-fold increase in the RMSE score, underscoring
our method's exceptional generalization across different darkness levels. The
code will be available online upon publication of the paper.
less
By: Hanyang Kong, Kehong Gong, Dongze Lian, Michael Bi Mi, Xinchao Wang
Text-to-motion generation is a formidable task, aiming to produce human
motions that align with the input text while also adhering to human
capabilities and physical laws. While there have been advancements in diffusion
models, their application in discrete spaces remains underexplored. Current
methods often overlook the varying significance of different motions, treating
them uniformly. It is essential to recognize that not all motions hol... more
Text-to-motion generation is a formidable task, aiming to produce human
motions that align with the input text while also adhering to human
capabilities and physical laws. While there have been advancements in diffusion
models, their application in discrete spaces remains underexplored. Current
methods often overlook the varying significance of different motions, treating
them uniformly. It is essential to recognize that not all motions hold the same
relevance to a particular textual description. Some motions, being more salient
and informative, should be given precedence during generation. In response, we
introduce a Priority-Centric Motion Discrete Diffusion Model (M2DM), which
utilizes a Transformer-based VQ-VAE to derive a concise, discrete motion
representation, incorporating a global self-attention mechanism and a
regularization term to counteract code collapse. We also present a motion
discrete diffusion model that employs an innovative noise schedule, determined
by the significance of each motion token within the entire motion sequence.
This approach retains the most salient motions during the reverse diffusion
process, leading to more semantically rich and varied motions. Additionally, we
formulate two strategies to gauge the importance of motion tokens, drawing from
both textual and visual indicators. Comprehensive experiments on the HumanML3D
and KIT-ML datasets confirm that our model surpasses existing techniques in
fidelity and diversity, particularly for intricate textual descriptions.
less
By: Mihir Kulkarni, Kostas Alexis
This paper contributes a novel learning-based method for aggressive
task-driven compression of depth images and their encoding as images tailored
to collision prediction for robotic systems. A novel 3D image processing
methodology is proposed that accounts for the robot's size in order to
appropriately "inflate" the obstacles represented in the depth image and thus
obtain the distance that can be traversed by the robot in a collision-free
m... more
This paper contributes a novel learning-based method for aggressive
task-driven compression of depth images and their encoding as images tailored
to collision prediction for robotic systems. A novel 3D image processing
methodology is proposed that accounts for the robot's size in order to
appropriately "inflate" the obstacles represented in the depth image and thus
obtain the distance that can be traversed by the robot in a collision-free
manner along any given ray within the camera frustum. Such depth-and-collision
image pairs are used to train a neural network that follows the architecture of
Variational Autoencoders to compress-and-transform the information in the
original depth image to derive a latent representation that encodes the
collision information for the given depth image. We compare our proposed
task-driven encoding method with classical task-agnostic methods and
demonstrate superior performance for the task of collision image prediction
from extremely low-dimensional latent spaces. A set of comparative studies show
that the proposed approach is capable of encoding depth image-and-collision
image tuples from complex scenes with thin obstacles at long distances better
than the classical methods at compression ratios as high as 4050:1.
less
By: Felix Rosberg, Eren Erdal Aksoy, Cristofer Englund, Fernando Alonso-Fernandez
In this paper, we present a new approach for facial anonymization in images
and videos, abbreviated as FIVA. Our proposed method is able to maintain the
same face anonymization consistently over frames with our suggested
identity-tracking and guarantees a strong difference from the original face.
FIVA allows for 0 true positives for a false acceptance rate of 0.001. Our work
considers the important security issue of reconstruction attacks a... more
In this paper, we present a new approach for facial anonymization in images
and videos, abbreviated as FIVA. Our proposed method is able to maintain the
same face anonymization consistently over frames with our suggested
identity-tracking and guarantees a strong difference from the original face.
FIVA allows for 0 true positives for a false acceptance rate of 0.001. Our work
considers the important security issue of reconstruction attacks and
investigates adversarial noise, uniform noise, and parameter noise to disrupt
reconstruction attacks. In this regard, we apply different defense and
protection methods against these privacy threats to demonstrate the scalability
of FIVA. On top of this, we also show that reconstruction attack models can be
used for detection of deep fakes. Last but not least, we provide experimental
results showing how FIVA can even enable face swapping, which is purely trained
on a single target image.
less
By: Santiago Rivier, Carlos Hinojosa, Silvio Giancola, Bernard Ghanem
Semantic segmentation is crucial in remote sensing, where high-resolution
satellite images are segmented into meaningful regions. Recent advancements in
deep learning have significantly improved satellite image segmentation.
However, most of these methods are typically trained in fully supervised
settings that require high-quality pixel-level annotations, which are expensive
and time-consuming to obtain. In this work, we present a weakly su... more
Semantic segmentation is crucial in remote sensing, where high-resolution
satellite images are segmented into meaningful regions. Recent advancements in
deep learning have significantly improved satellite image segmentation.
However, most of these methods are typically trained in fully supervised
settings that require high-quality pixel-level annotations, which are expensive
and time-consuming to obtain. In this work, we present a weakly supervised
learning algorithm to train semantic segmentation algorithms that only rely on
query point annotations instead of full mask labels. Our proposed approach
performs accurate semantic segmentation and improves efficiency by
significantly reducing the cost and time required for manual annotation.
Specifically, we generate superpixels and extend the query point labels into
those superpixels that group similar meaningful semantics. Then, we train
semantic segmentation models, supervised with images partially labeled with the
superpixels pseudo-labels. We benchmark our weakly supervised training approach
on an aerial image dataset and different semantic segmentation architectures,
showing that we can reach competitive performance compared to fully supervised
training while reducing the annotation effort.
less
By: Gang Fu, Qing Zhang, Lei Zhu, Chunxia Xiao, Ping Li
This paper aims to remove specular highlights from a single object-level
image. Although previous methods have made some progresses, their performance
remains somewhat limited, particularly for real images with complex specular
highlights. To this end, we propose a three-stage network to address them.
Specifically, given an input image, we first decompose it into the albedo,
shading, and specular residue components to estimate a coarse spec... more
This paper aims to remove specular highlights from a single object-level
image. Although previous methods have made some progresses, their performance
remains somewhat limited, particularly for real images with complex specular
highlights. To this end, we propose a three-stage network to address them.
Specifically, given an input image, we first decompose it into the albedo,
shading, and specular residue components to estimate a coarse specular-free
image. Then, we further refine the coarse result to alleviate its visual
artifacts such as color distortion. Finally, we adjust the tone of the refined
result to match that of the input as closely as possible. In addition, to
facilitate network training and quantitative evaluation, we present a
large-scale synthetic dataset of object-level images, covering diverse objects
and illumination conditions. Extensive experiments illustrate that our network
is able to generalize well to unseen real object-level images, and even produce
good results for scene-level images with multiple background objects and
complex lighting.
less
By: Xiuwei Chen, Xiaobin Chang
The rehearsal strategy is widely used to alleviate the catastrophic
forgetting problem in class incremental learning (CIL) by preserving limited
exemplars from previous tasks. With imbalanced sample numbers between old and
new classes, the classifier learning can be biased. Existing CIL methods
exploit the long-tailed (LT) recognition techniques, e.g., the adjusted losses
and the data re-sampling methods, to handle the data imbalance issue ... more
The rehearsal strategy is widely used to alleviate the catastrophic
forgetting problem in class incremental learning (CIL) by preserving limited
exemplars from previous tasks. With imbalanced sample numbers between old and
new classes, the classifier learning can be biased. Existing CIL methods
exploit the long-tailed (LT) recognition techniques, e.g., the adjusted losses
and the data re-sampling methods, to handle the data imbalance issue within
each increment task. In this work, the dynamic nature of data imbalance in CIL
is shown and a novel Dynamic Residual Classifier (DRC) is proposed to handle
this challenging scenario. Specifically, DRC is built upon a recent advance
residual classifier with the branch layer merging to handle the model-growing
problem. Moreover, DRC is compatible with different CIL pipelines and
substantially improves them. Combining DRC with the model adaptation and fusion
(MAF) pipeline, this method achieves state-of-the-art results on both the
conventional CIL and the LT-CIL benchmarks. Extensive experiments are also
conducted for a detailed analysis. The code is publicly available.
less
By: Jinfeng Liu, Lingtong Kong, Jie Yang, Wei Liu
Depth estimation plays an important role in the robotic perception system.
Self-supervised monocular paradigm has gained significant attention since it
can free training from the reliance on depth annotations. Despite recent
advancements, existing self-supervised methods still underutilize the available
training data, limiting their generalization ability. In this paper, we take
two data augmentation techniques, namely Resizing-Cropping and... more
Depth estimation plays an important role in the robotic perception system.
Self-supervised monocular paradigm has gained significant attention since it
can free training from the reliance on depth annotations. Despite recent
advancements, existing self-supervised methods still underutilize the available
training data, limiting their generalization ability. In this paper, we take
two data augmentation techniques, namely Resizing-Cropping and
Splitting-Permuting, to fully exploit the potential of training datasets.
Specifically, the original image and the generated two augmented images are fed
into the training pipeline simultaneously and we leverage them to conduct
self-distillation. Additionally, we introduce the detail-enhanced DepthNet with
an extra full-scale branch in the encoder and a grid decoder to enhance the
restoration of fine details in depth maps. Experimental results demonstrate our
method can achieve state-of-the-art performance on the KITTI benchmark, with
both raw ground truth and improved ground truth. Moreover, our models also show
superior generalization performance when transferring to Make3D and NYUv2
datasets. Our codes are available at https://github.com/Sauf4896/BDEdepth.
less
By: Yin Wang, Zhiying Leng, Frederick W. B. Li, Shun-Cheng Wu, Xiaohui Liang
Text-driven human motion generation in computer vision is both significant
and challenging. However, current methods are limited to producing either
deterministic or imprecise motion sequences, failing to effectively control the
temporal and spatial relationships required to conform to a given text
description. In this work, we propose a fine-grained method for generating
high-quality, conditional human motion sequences supporting precise t... more
Text-driven human motion generation in computer vision is both significant
and challenging. However, current methods are limited to producing either
deterministic or imprecise motion sequences, failing to effectively control the
temporal and spatial relationships required to conform to a given text
description. In this work, we propose a fine-grained method for generating
high-quality, conditional human motion sequences supporting precise text
description. Our approach consists of two key components: 1) a
linguistics-structure assisted module that constructs accurate and complete
language feature to fully utilize text information; and 2) a context-aware
progressive reasoning module that learns neighborhood and overall semantic
linguistics features from shallow and deep graph neural networks to achieve a
multi-step inference. Experiments show that our approach outperforms
text-driven motion generation methods on HumanML3D and KIT test sets and
generates better visually confirmed motion to the text conditions.
less
By: Haiwen Diao, Bo Wan, Ying Zhang, Xu Jia, Huchuan Lu, Long Chen
Fine-tuning pre-trained models has emerged as a powerful technique in
numerous domains, owing to its ability to leverage enormous pre-existing
knowledge and achieve remarkable performance on downstream tasks. However,
updating the parameters of entire networks is computationally intensive.
Although state-of-the-art parameter-efficient transfer learning (PETL) methods
significantly reduce the trainable parameters and storage demand, almost a... more
Fine-tuning pre-trained models has emerged as a powerful technique in
numerous domains, owing to its ability to leverage enormous pre-existing
knowledge and achieve remarkable performance on downstream tasks. However,
updating the parameters of entire networks is computationally intensive.
Although state-of-the-art parameter-efficient transfer learning (PETL) methods
significantly reduce the trainable parameters and storage demand, almost all of
them still need to back-propagate the gradients through large pre-trained
networks. This memory-extensive characteristic extremely limits the
applicability of PETL methods in real-world scenarios. To this end, we propose
a new memory-efficient PETL strategy, dubbed Universal Parallel Tuning (UniPT).
Specifically, we facilitate the transfer process via a lightweight learnable
parallel network, which consists of two modules: 1) A parallel interaction
module that decouples the inherently sequential connections and processes the
intermediate activations detachedly of the pre-trained network. 2) A confidence
aggregation module that learns optimal strategies adaptively for integrating
cross-layer features. We evaluate UniPT with different backbones (e.g.,
VSE$\infty$, CLIP4Clip, Clip-ViL, and MDETR) on five challenging
vision-and-language tasks (i.e., image-text retrieval, video-text retrieval,
visual question answering, compositional question answering, and visual
grounding). Extensive ablations on ten datasets have validated that our UniPT
can not only dramatically reduce memory consumption and outperform the best
memory-efficient competitor, but also achieve higher performance than existing
PETL methods in a low-memory scenario on different architectures. Our code is
publicly available at: https://github.com/Paranioar/UniPT.
less
By: István Sárándi, Timm Linder, Kai O. Arras, Bastian Leibe
Heatmap representations have formed the basis of human pose estimation systems for many years, and their extension to 3D has been a fruitful line of recent research. This includes 2.5D volumetric heatmaps, whose X and Y axes correspond to image space and Z to metric depth around the subject. To obtain metric-scale predictions, 2.5D methods need a separate post-processing step to resolve scale ambiguity. Further, they cannot localize body join... more
Heatmap representations have formed the basis of human pose estimation systems for many years, and their extension to 3D has been a fruitful line of recent research. This includes 2.5D volumetric heatmaps, whose X and Y axes correspond to image space and Z to metric depth around the subject. To obtain metric-scale predictions, 2.5D methods need a separate post-processing step to resolve scale ambiguity. Further, they cannot localize body joints outside the image boundaries, leading to incomplete estimates for truncated images. To address these limitations, we propose metric-scale truncation-robust (MeTRo) volumetric heatmaps, whose dimensions are all defined in metric 3D space, instead of being aligned with image space. This reinterpretation of heatmap dimensions allows us to directly estimate complete, metric-scale poses without test-time knowledge of distance or relying on anthropometric heuristics, such as bone lengths. To further demonstrate the utility our representation, we present a differentiable combination of our 3D metric-scale heatmaps with 2D image-space ones to estimate absolute 3D pose (our MeTRAbs architecture). We find that supervision via absolute pose loss is crucial for accurate non-root-relative localization. Using a ResNet-50 backbone without further learned layers, we obtain state-of-the-art results on Human3.6M, MPI-INF-3DHP and MuPoTS-3D. Our code will be made publicly available to facilitate further research. less
By: Zheng Gao, Chen Feng, Ioannis Patras
Whilst contrastive learning yields powerful representations by matching
different augmented views of the same instance, it lacks the ability to capture
the similarities between different instances. One popular way to address this
limitation is by learning global features (after the global pooling) to capture
inter-instance relationships based on knowledge distillation, where the global
features of the teacher are used to guide the learning ... more
Whilst contrastive learning yields powerful representations by matching
different augmented views of the same instance, it lacks the ability to capture
the similarities between different instances. One popular way to address this
limitation is by learning global features (after the global pooling) to capture
inter-instance relationships based on knowledge distillation, where the global
features of the teacher are used to guide the learning of the global features
of the student. Inspired by cross-modality learning, we extend this existing
framework that only learns from global features by encouraging the global
features and intermediate layer features to learn from each other. This leads
to our novel self-supervised framework: cross-context learning between global
and hypercolumn features (CGH), that enforces the consistency of instance
relations between low- and high-level semantics. Specifically, we stack the
intermediate feature maps to construct a hypercolumn representation so that we
can measure instance relations using two contexts (hypercolumn and global
feature) separately, and then use the relations of one context to guide the
learning of the other. This cross-context learning allows the model to learn
from the differences between the two contexts. The experimental results on
linear classification and downstream tasks show that our method outperforms the
state-of-the-art methods.
less
By: Stefan Ainetter, Sinisa Stekovic, Friedrich Fraundorfer, Vincent Lepetit
We present an automated and efficient approach for retrieving high-quality
CAD models of objects and their poses in a scene captured by a moving RGB-D
camera. We first investigate various objective functions to measure similarity
between a candidate CAD object model and the available data, and the best
objective function appears to be a "render-and-compare" method comparing depth
and mask rendering. We thus introduce a fast-search method th... more
We present an automated and efficient approach for retrieving high-quality
CAD models of objects and their poses in a scene captured by a moving RGB-D
camera. We first investigate various objective functions to measure similarity
between a candidate CAD object model and the available data, and the best
objective function appears to be a "render-and-compare" method comparing depth
and mask rendering. We thus introduce a fast-search method that approximates an
exhaustive search based on this objective function for simultaneously
retrieving the object category, a CAD model, and the pose of an object given an
approximate 3D bounding box. This method involves a search tree that organizes
the CAD models and object properties including object category and pose for
fast retrieval and an algorithm inspired by Monte Carlo Tree Search, that
efficiently searches this tree. We show that this method retrieves CAD models
that fit the real objects very well, with a speed-up factor of 10x to 120x
compared to exhaustive search.
less
By: Yuhan Song, Armagan Elibol, Nak Young Chong
As recent advances in AI are causing the decline of conventional diagnostic
methods, the realization of end-to-end diagnosis is fast approaching.
Ultrasound image segmentation is an important step in the diagnostic process.
An accurate and robust segmentation model accelerates the process and reduces
the burden of sonographers. In contrast to previous research, we take two
inherent features of ultrasound images into consideration: (1) diffe... more
As recent advances in AI are causing the decline of conventional diagnostic
methods, the realization of end-to-end diagnosis is fast approaching.
Ultrasound image segmentation is an important step in the diagnostic process.
An accurate and robust segmentation model accelerates the process and reduces
the burden of sonographers. In contrast to previous research, we take two
inherent features of ultrasound images into consideration: (1) different organs
and tissues vary in spatial sizes, (2) the anatomical structures inside human
body form a relatively constant spatial relationship. Based on those two ideas,
we propose a new image segmentation model combining Feature Pyramid Network
(FPN) and Spatial Recurrent Neural Network (SRNN). We discuss why we use FPN to
extract anatomical structures of different scales and how SRNN is implemented
to extract the spatial context features in abdominal ultrasound images.
less
By: Junyu Zhu, Lina Liu, Yu Tang, Feng Wen, Wanlong Li, Yong Liu
Visual bird's eye view (BEV) semantic segmentation helps autonomous vehicles
understand the surrounding environment only from images, including static
elements (e.g., roads) and dynamic elements (e.g., vehicles, pedestrians).
However, the high cost of annotation procedures of full-supervised methods
limits the capability of the visual BEV semantic segmentation, which usually
needs HD maps, 3D object bounding boxes, and camera extrinsic matr... more
Visual bird's eye view (BEV) semantic segmentation helps autonomous vehicles
understand the surrounding environment only from images, including static
elements (e.g., roads) and dynamic elements (e.g., vehicles, pedestrians).
However, the high cost of annotation procedures of full-supervised methods
limits the capability of the visual BEV semantic segmentation, which usually
needs HD maps, 3D object bounding boxes, and camera extrinsic matrixes. In this
paper, we present a novel semi-supervised framework for visual BEV semantic
segmentation to boost performance by exploiting unlabeled images during the
training. A consistency loss that makes full use of unlabeled data is then
proposed to constrain the model on not only semantic prediction but also the
BEV feature. Furthermore, we propose a novel and effective data augmentation
method named conjoint rotation which reasonably augments the dataset while
maintaining the geometric relationship between the front-view images and the
BEV semantic segmentation. Extensive experiments on the nuScenes and Argoverse
datasets show that our semi-supervised framework can effectively improve
prediction accuracy. To the best of our knowledge, this is the first work that
explores improving visual BEV semantic segmentation performance using unlabeled
data. The code will be publicly available.
less
By: Hongyu Hu, Jiyuan Zhang, Minyi Zhao, Zhenbang Sun
Nowadays, the research on Large Vision-Language Models (LVLMs) has been
significantly promoted thanks to the success of Large Language Models (LLM).
Nevertheless, these Vision-Language Models (VLMs) are suffering from the
drawback of hallucination -- due to insufficient understanding of vision and
language modalities, VLMs may generate incorrect perception information when
doing downstream applications, for example, captioning a non-existen... more
Nowadays, the research on Large Vision-Language Models (LVLMs) has been
significantly promoted thanks to the success of Large Language Models (LLM).
Nevertheless, these Vision-Language Models (VLMs) are suffering from the
drawback of hallucination -- due to insufficient understanding of vision and
language modalities, VLMs may generate incorrect perception information when
doing downstream applications, for example, captioning a non-existent entity.
To address the hallucination phenomenon, on the one hand, we introduce a
Contrastive Instruction Evaluation Method (CIEM), which is an automatic
pipeline that leverages an annotated image-text dataset coupled with an LLM to
generate factual/contrastive question-answer pairs for the evaluation of the
hallucination of VLMs. On the other hand, based on CIEM, we further propose a
new instruction tuning method called CIT (the abbreviation of Contrastive
Instruction Tuning) to alleviate the hallucination of VLMs by automatically
producing high-quality factual/contrastive question-answer pairs and
corresponding justifications for model tuning. Through extensive experiments on
CIEM and CIT, we pinpoint the hallucination issues commonly present in existing
VLMs, the disability of the current instruction-tuning dataset to handle the
hallucination phenomenon and the superiority of CIT-tuned VLMs over both CIEM
and public datasets.
less
By: Yongkang Zhao, Chuang Zhu, Yuan Li, Shuaishuai Wang, Zihan Lan, Yuanyuan Qiao
Detecting small moving targets accurately in infrared (IR) image sequences is
a significant challenge. To address this problem, we propose a novel method
called spatial-temporal local feature difference (STLFD) with adaptive
background suppression (ABS). Our approach utilizes filters in the spatial and
temporal domains and performs pixel-level ABS on the output to enhance the
contrast between the target and the background. The proposed meth... more
Detecting small moving targets accurately in infrared (IR) image sequences is
a significant challenge. To address this problem, we propose a novel method
called spatial-temporal local feature difference (STLFD) with adaptive
background suppression (ABS). Our approach utilizes filters in the spatial and
temporal domains and performs pixel-level ABS on the output to enhance the
contrast between the target and the background. The proposed method comprises
three steps. First, we obtain three temporal frame images based on the current
frame image and extract two feature maps using the designed spatial domain and
temporal domain filters. Next, we fuse the information of the spatial domain
and temporal domain to produce the spatial-temporal feature maps and suppress
noise using our pixel-level ABS module. Finally, we obtain the segmented binary
map by applying a threshold. Our experimental results demonstrate that the
proposed method outperforms existing state-of-the-art methods for infrared
small-moving target detection.
less
Cross-Modal Retrieval Meets Inference:Improving Zero-Shot Classification with Cross-Modal Retrieval
0upvotes
By: Seongha Eom, Namgyu Ho, Jaehoon Oh, Se-Young Yun
Contrastive language-image pre-training (CLIP) has demonstrated remarkable
zero-shot classification ability, namely image classification using novel text
labels. Existing works have attempted to enhance CLIP by fine-tuning on
downstream tasks, but these have inadvertently led to performance degradation
on unseen classes, thus harming zero-shot generalization. This paper aims to
address this challenge by leveraging readily available image-te... more
Contrastive language-image pre-training (CLIP) has demonstrated remarkable
zero-shot classification ability, namely image classification using novel text
labels. Existing works have attempted to enhance CLIP by fine-tuning on
downstream tasks, but these have inadvertently led to performance degradation
on unseen classes, thus harming zero-shot generalization. This paper aims to
address this challenge by leveraging readily available image-text pairs from an
external dataset for cross-modal guidance during inference. To this end, we
propose X-MoRe, a novel inference method comprising two key steps: (1)
cross-modal retrieval and (2) modal-confidence-based ensemble. Given a query
image, we harness the power of CLIP's cross-modal representations to retrieve
relevant textual information from an external image-text pair dataset. Then, we
assign higher weights to the more reliable modality between the original query
image and retrieved text, contributing to the final prediction. X-MoRe
demonstrates robust performance across a diverse set of tasks without the need
for additional training, showcasing the effectiveness of utilizing cross-modal
features to maximize CLIP's zero-shot ability.
less
By: Lorenzo Mur-Labadia, Jose J. Guerrero, Ruben Martinez-Cantin
Accurate affordance detection and segmentation with pixel precision is an
important piece in many complex systems based on interactions, such as robots
and assitive devices. We present a new approach to affordance perception which
enables accurate multi-label segmentation. Our approach can be used to
automatically extract grounded affordances from first person videos of
interactions using a 3D map of the environment providing pixel level pr... more
Accurate affordance detection and segmentation with pixel precision is an
important piece in many complex systems based on interactions, such as robots
and assitive devices. We present a new approach to affordance perception which
enables accurate multi-label segmentation. Our approach can be used to
automatically extract grounded affordances from first person videos of
interactions using a 3D map of the environment providing pixel level precision
for the affordance location. We use this method to build the largest and most
complete dataset on affordances based on the EPIC-Kitchen dataset, EPIC-Aff,
which provides interaction-grounded, multi-label, metric and spatial affordance
annotations. Then, we propose a new approach to affordance segmentation based
on multi-label detection which enables multiple affordances to co-exists in the
same space, for example if they are associated with the same object. We present
several strategies of multi-label detection using several segmentation
architectures. The experimental results highlight the importance of the
multi-label detection. Finally, we show how our metric representation can be
exploited for build a map of interaction hotspots in spatial action-centric
zones and use that representation to perform a task-oriented navigation.
less
By: Yiming Zhang, ZeMing Gong, Angel X. Chang
We introduce the task of localizing a flexible number of objects in
real-world 3D scenes using natural language descriptions. Existing 3D visual
grounding tasks focus on localizing a unique object given a text description.
However, such a strict setting is unnatural as localizing potentially multiple
objects is a common need in real-world scenarios and robotic tasks (e.g.,
visual navigation and object rearrangement). To address this setting... more
We introduce the task of localizing a flexible number of objects in
real-world 3D scenes using natural language descriptions. Existing 3D visual
grounding tasks focus on localizing a unique object given a text description.
However, such a strict setting is unnatural as localizing potentially multiple
objects is a common need in real-world scenarios and robotic tasks (e.g.,
visual navigation and object rearrangement). To address this setting we propose
Multi3DRefer, generalizing the ScanRefer dataset and task. Our dataset contains
61926 descriptions of 11609 objects, where zero, single or multiple target
objects are referenced by each description. We also introduce a new evaluation
metric and benchmark methods from prior work to enable further investigation of
multi-modal 3D scene understanding. Furthermore, we develop a better baseline
leveraging 2D features from CLIP by rendering object proposals online with
contrastive learning, which outperforms the state of the art on the ScanRefer
benchmark.
less
By: Chonghyuk Song, Gengshan Yang, Kangle Deng, Jun-Yan Zhu, Deva Ramanan
We explore the task of embodied view synthesis from monocular videos of
deformable scenes. Given a minute-long RGBD video of people interacting with
their pets, we render the scene from novel camera trajectories derived from
in-scene motion of actors: (1) egocentric cameras that simulate the point of
view of a target actor and (2) 3rd-person cameras that follow the actor.
Building such a system requires reconstructing the root-body and arti... more
We explore the task of embodied view synthesis from monocular videos of
deformable scenes. Given a minute-long RGBD video of people interacting with
their pets, we render the scene from novel camera trajectories derived from
in-scene motion of actors: (1) egocentric cameras that simulate the point of
view of a target actor and (2) 3rd-person cameras that follow the actor.
Building such a system requires reconstructing the root-body and articulated
motion of each actor in the scene, as well as a scene representation that
supports free-viewpoint synthesis. Longer videos are more likely to capture the
scene from diverse viewpoints (which helps reconstruction) but are also more
likely to contain larger motions (which complicates reconstruction). To address
these challenges, we present Total-Recon, the first method to
photorealistically reconstruct deformable scenes from long monocular RGBD
videos. Crucially, to scale to long videos, our method hierarchically
decomposes the scene motion into the motion of each object, which itself is
decomposed into global root-body motion and local articulations. To quantify
such "in-the-wild" reconstruction and view synthesis, we collect ground-truth
data from a specialized stereo RGBD capture rig for 11 challenging videos,
significantly outperforming prior art. Code, videos, and data can be found at
https://andrewsonga.github.io/totalrecon .
less
Local-Global Pseudo-label Correction for Source-free Domain Adaptive Medical Image Segmentation
0upvotes
By: Yanyu Ye, Zhengxi Zhang, Chunna Tianb, Wei wei
Domain shift is a commonly encountered issue in medical imaging solutions,
primarily caused by variations in imaging devices and data sources. To mitigate
this problem, unsupervised domain adaptation techniques have been employed.
However, concerns regarding patient privacy and potential degradation of image
quality have led to an increased focus on source-free domain adaptation. In
this study, we address the issue of false labels in self-t... more
Domain shift is a commonly encountered issue in medical imaging solutions,
primarily caused by variations in imaging devices and data sources. To mitigate
this problem, unsupervised domain adaptation techniques have been employed.
However, concerns regarding patient privacy and potential degradation of image
quality have led to an increased focus on source-free domain adaptation. In
this study, we address the issue of false labels in self-training based
source-free domain adaptive medical image segmentation methods. To correct
erroneous pseudo-labels, we propose a novel approach called the local-global
pseudo-label correction (LGDA) method for source-free domain adaptive medical
image segmentation. Our method consists of two components: An offline local
context-based pseudo-label correction method that utilizes local context
similarity in image space. And an online global pseudo-label correction method
based on class prototypes, which corrects erroneously predicted pseudo-labels
by considering the relative distance between pixel-wise feature vectors and
prototype vectors. We evaluate the performance of our method on three benchmark
fundus image datasets for optic disc and cup segmentation. Our method achieves
superior performance compared to the state-of-the-art approaches, even without
using of any source data.
less
By: Bo Pang, Zhongtian Zheng, Guoping Wang, Peng-Shuai Wang
We present GeGnn, a learning-based method for computing the approximate
geodesic distance between two arbitrary points on discrete polyhedra surfaces
with constant time complexity after fast precomputation. Previous relevant
methods either focus on computing the geodesic distance between a single source
and all destinations, which has linear complexity at least or require a long
precomputation time. Our key idea is to train a graph neural n... more
We present GeGnn, a learning-based method for computing the approximate
geodesic distance between two arbitrary points on discrete polyhedra surfaces
with constant time complexity after fast precomputation. Previous relevant
methods either focus on computing the geodesic distance between a single source
and all destinations, which has linear complexity at least or require a long
precomputation time. Our key idea is to train a graph neural network to embed
an input mesh into a high-dimensional embedding space and compute the geodesic
distance between a pair of points using the corresponding embedding vectors and
a lightweight decoding function. To facilitate the learning of the embedding,
we propose novel graph convolution and graph pooling modules that incorporate
local geodesic information and are verified to be much more effective than
previous designs. After training, our method requires only one forward pass of
the network per mesh as precomputation. Then, we can compute the geodesic
distance between a pair of points using our decoding function, which requires
only several matrix multiplications and can be massively parallelized on GPUs.
We verify the efficiency and effectiveness of our method on ShapeNet and
demonstrate that our method is faster than existing methods by orders of
magnitude while achieving comparable or better accuracy. Additionally, our
method exhibits robustness on noisy and incomplete meshes and strong
generalization ability on out-of-distribution meshes. The code and pretrained
model can be found on https://github.com/IntelligentGeometry/GeGnn.
less
By: Aristotelis Ballas, Christos Diou
During the past decade, deep neural networks have led to fast-paced progress
and significant achievements in computer vision problems, for both academia and
industry. Yet despite their success, state-of-the-art image classification
approaches fail to generalize well in previously unseen visual contexts, as
required by many real-world applications. In this paper, we focus on this
domain generalization (DG) problem and argue that the generali... more
During the past decade, deep neural networks have led to fast-paced progress
and significant achievements in computer vision problems, for both academia and
industry. Yet despite their success, state-of-the-art image classification
approaches fail to generalize well in previously unseen visual contexts, as
required by many real-world applications. In this paper, we focus on this
domain generalization (DG) problem and argue that the generalization ability of
deep convolutional neural networks can be improved by taking advantage of
multi-layer and multi-scaled representations of the network. We introduce a
framework that aims at improving domain generalization of image classifiers by
combining both low-level and high-level features at multiple scales, enabling
the network to implicitly disentangle representations in its latent space and
learn domain-invariant attributes of the depicted objects. Additionally, to
further facilitate robust representation learning, we propose a novel objective
function, inspired by contrastive learning, which aims at constraining the
extracted representations to remain invariant under distribution shifts. We
demonstrate the effectiveness of our method by evaluating on the domain
generalization datasets of PACS, VLCS, Office-Home and NICO. Through extensive
experimentation, we show that our model is able to surpass the performance of
previous DG methods and consistently produce competitive and state-of-the-art
results in all datasets.
less
By: Chunyong Hu, Hang Zheng, Kun Li, Jianyun Xu, Weibo Mao, Maochun Luo, Lingxuan Wang, Mingxia Chen, Kaixuan Liu, Yiru Zhao, Peihan Hao, Minzhe Liu, Kaicheng Yu
Multi-sensor modal fusion has demonstrated strong advantages in 3D object
detection tasks. However, existing methods that fuse multi-modal features
through a simple channel concatenation require transformation features into
bird's eye view space and may lose the information on Z-axis thus leads to
inferior performance. To this end, we propose FusionFormer, an end-to-end
multi-modal fusion framework that leverages transformers to fuse multi-... more
Multi-sensor modal fusion has demonstrated strong advantages in 3D object
detection tasks. However, existing methods that fuse multi-modal features
through a simple channel concatenation require transformation features into
bird's eye view space and may lose the information on Z-axis thus leads to
inferior performance. To this end, we propose FusionFormer, an end-to-end
multi-modal fusion framework that leverages transformers to fuse multi-modal
features and obtain fused BEV features. And based on the flexible adaptability
of FusionFormer to the input modality representation, we propose a depth
prediction branch that can be added to the framework to improve detection
performance in camera-based detection tasks. In addition, we propose a
plug-and-play temporal fusion module based on transformers that can fuse
historical frame BEV features for more stable and reliable detection results.
We evaluate our method on the nuScenes dataset and achieve 72.6% mAP and 75.1%
NDS for 3D object detection tasks, outperforming state-of-the-art methods.
less
Feature Aggregation Network for Building Extraction from High-resolution Remote Sensing Images
0upvotes
By: Xuan Zhou, Xuefeng Wei
The rapid advancement in high-resolution satellite remote sensing data
acquisition, particularly those achieving submeter precision, has uncovered the
potential for detailed extraction of surface architectural features. However,
the diversity and complexity of surface distributions frequently lead to
current methods focusing exclusively on localized information of surface
features. This often results in significant intraclass variability in... more
The rapid advancement in high-resolution satellite remote sensing data
acquisition, particularly those achieving submeter precision, has uncovered the
potential for detailed extraction of surface architectural features. However,
the diversity and complexity of surface distributions frequently lead to
current methods focusing exclusively on localized information of surface
features. This often results in significant intraclass variability in boundary
recognition and between buildings. Therefore, the task of fine-grained
extraction of surface features from high-resolution satellite imagery has
emerged as a critical challenge in remote sensing image processing. In this
work, we propose the Feature Aggregation Network (FANet), concentrating on
extracting both global and local features, thereby enabling the refined
extraction of landmark buildings from high-resolution satellite remote sensing
imagery. The Pyramid Vision Transformer captures these global features, which
are subsequently refined by the Feature Aggregation Module and merged into a
cohesive representation by the Difference Elimination Module. In addition, to
ensure a comprehensive feature map, we have incorporated the Receptive Field
Block and Dual Attention Module, expanding the receptive field and intensifying
attention across spatial and channel dimensions. Extensive experiments on
multiple datasets have validated the outstanding capability of FANet in
extracting features from high-resolution satellite images. This signifies a
major breakthrough in the field of remote sensing image processing. We will
release our code soon.
less
By: Naishan Zheng, Man Zhou, Yanmeng Dong, Xiangyu Rui, Jie Huang, Chongyi Li, Feng Zhao
Deep neural networks have achieved remarkable progress in enhancing low-light
images by improving their brightness and eliminating noise. However, most
existing methods construct end-to-end mapping networks heuristically,
neglecting the intrinsic prior of image enhancement task and lacking
transparency and interpretability. Although some unfolding solutions have been
proposed to relieve these issues, they rely on proximal operator networks ... more
Deep neural networks have achieved remarkable progress in enhancing low-light
images by improving their brightness and eliminating noise. However, most
existing methods construct end-to-end mapping networks heuristically,
neglecting the intrinsic prior of image enhancement task and lacking
transparency and interpretability. Although some unfolding solutions have been
proposed to relieve these issues, they rely on proximal operator networks that
deliver ambiguous and implicit priors. In this work, we propose a paradigm for
low-light image enhancement that explores the potential of customized learnable
priors to improve the transparency of the deep unfolding paradigm. Motivated by
the powerful feature representation capability of Masked Autoencoder (MAE), we
customize MAE-based illumination and noise priors and redevelop them from two
perspectives: 1) \textbf{structure flow}: we train the MAE from a normal-light
image to its illumination properties and then embed it into the proximal
operator design of the unfolding architecture; and m2) \textbf{optimization
flow}: we train MAE from a normal-light image to its gradient representation
and then employ it as a regularization term to constrain noise in the model
output. These designs improve the interpretability and representation
capability of the model.Extensive experiments on multiple low-light image
enhancement datasets demonstrate the superiority of our proposed paradigm over
state-of-the-art methods. Code is available at
https://github.com/zheng980629/CUE.
less
By: Haichao Shi, Mandi Luo, Xiao-Yu Zhang, Ran He
Visible-Infrared person re-identification (VI-ReID) is an important and
challenging task in intelligent video surveillance. Existing methods mainly
focus on learning a shared feature space to reduce the modality discrepancy
between visible and infrared modalities, which still leave two problems
underexplored: information redundancy and modality complementarity. To this
end, properly eliminating the identity-irrelevant information as well as... more
Visible-Infrared person re-identification (VI-ReID) is an important and
challenging task in intelligent video surveillance. Existing methods mainly
focus on learning a shared feature space to reduce the modality discrepancy
between visible and infrared modalities, which still leave two problems
underexplored: information redundancy and modality complementarity. To this
end, properly eliminating the identity-irrelevant information as well as making
up for the modality-specific information are critical and remains a challenging
endeavor. To tackle the above problems, we present a novel mutual information
and modality consensus network, namely CMInfoNet, to extract modality-invariant
identity features with the most representative information and reduce the
redundancies. The key insight of our method is to find an optimal
representation to capture more identity-relevant information and compress the
irrelevant parts by optimizing a mutual information bottleneck trade-off.
Besides, we propose an automatically search strategy to find the most prominent
parts that identify the pedestrians. To eliminate the cross- and intra-modality
variations, we also devise a modality consensus module to align the visible and
infrared modalities for task-specific guidance. Moreover, the global-local
feature representations can also be acquired for key parts discrimination.
Experimental results on four benchmarks, i.e., SYSU-MM01, RegDB,
Occluded-DukeMTMC, Occluded-REID, Partial-REID and Partial\_iLIDS dataset, have
demonstrated the effectiveness of CMInfoNet.
less
By: Mazvydas Gudelis, Michal Mackiewicz, Julie Bremner, Sophie Fielding
British Antarctic Survey (BAS) researchers launch annual expeditions to the
Antarctic in order to estimate Antarctic Krill biomass and assess the change
from previous years. These comparisons provide insight into the effects of the
current environment on this key component of the marine food chain. In this
work we have developed tools for automating the data collection and analysis
process, using web-based image annotation tools and deep le... more
British Antarctic Survey (BAS) researchers launch annual expeditions to the
Antarctic in order to estimate Antarctic Krill biomass and assess the change
from previous years. These comparisons provide insight into the effects of the
current environment on this key component of the marine food chain. In this
work we have developed tools for automating the data collection and analysis
process, using web-based image annotation tools and deep learning image
classification and regression models. We achieve highly accurate krill instance
segmentation results with an average 77.28% AP score, as well as separate
maturity stage and length estimation of krill specimens with 62.99% accuracy
and a 1.96 mm length error respectively.
less
By: Rayson Laroca, Luiz A. Zanlorensi, Valter Estevam, Rodrigo Minetto, David Menotti
License Plate Recognition (LPR) plays a critical role in various
applications, such as toll collection, parking management, and traffic law
enforcement. Although LPR has witnessed significant advancements through the
development of deep learning, there has been a noticeable lack of studies
exploring the potential improvements in results by fusing the outputs from
multiple recognition models. This research aims to fill this gap by
investigat... more
License Plate Recognition (LPR) plays a critical role in various
applications, such as toll collection, parking management, and traffic law
enforcement. Although LPR has witnessed significant advancements through the
development of deep learning, there has been a noticeable lack of studies
exploring the potential improvements in results by fusing the outputs from
multiple recognition models. This research aims to fill this gap by
investigating the combination of up to 12 different models using
straightforward approaches, such as selecting the most confident prediction or
employing majority vote-based strategies. Our experiments encompass a wide
range of datasets, revealing substantial benefits of fusion approaches in both
intra- and cross-dataset setups. Essentially, fusing multiple models reduces
considerably the likelihood of obtaining subpar performance on a particular
dataset/scenario. We also found that combining models based on their speed is
an appealing approach. Specifically, for applications where the recognition
task can tolerate some additional time, though not excessively, an effective
strategy is to combine 4-6 models. These models may not be the most accurate
individually, but their fusion strikes an optimal balance between accuracy and
speed.
less
By: Qianliang Wu, Yaqing Ding, Lei Luo, Chuanwei Zhou, Jin Xie, Jian Yang
Point Cloud Registration (PCR) is a critical and challenging task in computer
vision. One of the primary difficulties in PCR is identifying salient and
meaningful points that exhibit consistent semantic and geometric properties
across different scans. Previous methods have encountered challenges with
ambiguous matching due to the similarity among patch blocks throughout the
entire point cloud and the lack of consideration for efficient glob... more
Point Cloud Registration (PCR) is a critical and challenging task in computer
vision. One of the primary difficulties in PCR is identifying salient and
meaningful points that exhibit consistent semantic and geometric properties
across different scans. Previous methods have encountered challenges with
ambiguous matching due to the similarity among patch blocks throughout the
entire point cloud and the lack of consideration for efficient global geometric
consistency. To address these issues, we propose a new framework that includes
several novel techniques. Firstly, we introduce a semantic-aware geometric
encoder that combines object-level and patch-level semantic information. This
encoder significantly improves registration recall by reducing ambiguity in
patch-level superpoint matching. Additionally, we incorporate a prior knowledge
approach that utilizes an intrinsic shape signature to identify salient points.
This enables us to extract the most salient super points and meaningful dense
points in the scene. Secondly, we introduce an innovative transformer that
encodes High-Order (HO) geometric features. These features are crucial for
identifying salient points within initial overlap regions while considering
global high-order geometric consistency. To optimize this high-order
transformer further, we introduce an anchor node selection strategy. By
encoding inter-frame triangle or polyhedron consistency features based on these
anchor nodes, we can effectively learn high-order geometric features of salient
super points. These high-order features are then propagated to dense points and
utilized by a Sinkhorn matching module to identify key correspondences for
successful registration. In our experiments conducted on well-known datasets
such as 3DMatch/3DLoMatch and KITTI, our approach has shown promising results,
highlighting the effectiveness of our novel method.
less
By: George Stoica, Mihaela Breaban, Vlad Barbu
Using additional training data is known to improve the results, especially
for medical image 3D segmentation where there is a lack of training material
and the model needs to generalize well from few available data. However, the
new data could have been acquired using other instruments and preprocessed such
its distribution is significantly different from the original training data.
Therefore, we study techniques which ameliorate domain shi... more
Using additional training data is known to improve the results, especially
for medical image 3D segmentation where there is a lack of training material
and the model needs to generalize well from few available data. However, the
new data could have been acquired using other instruments and preprocessed such
its distribution is significantly different from the original training data.
Therefore, we study techniques which ameliorate domain shift during training so
that the additional data becomes better usable for preprocessing and training
together with the original data. Our results show that transforming the
additional data using histogram matching has better results than using simple
normalization.
less
By: Junjing Zheng, Xinyu Zhang, Yongxiang Liu, Weidong Jiang, Kai Huo, Li Liu
In the field of unsupervised feature selection, sparse principal component
analysis (SPCA) methods have attracted more and more attention recently.
Compared to spectral-based methods, SPCA methods don't rely on the construction
of a similarity matrix and show better feature selection ability on real-world
data. The original SPCA formulates a nonconvex optimization problem. Existing
convex SPCA methods reformulate SPCA as a convex model by r... more
In the field of unsupervised feature selection, sparse principal component
analysis (SPCA) methods have attracted more and more attention recently.
Compared to spectral-based methods, SPCA methods don't rely on the construction
of a similarity matrix and show better feature selection ability on real-world
data. The original SPCA formulates a nonconvex optimization problem. Existing
convex SPCA methods reformulate SPCA as a convex model by regarding the
reconstruction matrix as an optimization variable. However, they are lack of
constraints equivalent to the orthogonality restriction in SPCA, leading to
larger solution space. In this paper, it's proved that the optimal solution to
a convex SPCA model falls onto the Positive Semidefinite (PSD) cone. A standard
convex SPCA-based model with PSD constraint for unsupervised feature selection
is proposed. Further, a two-step fast optimization algorithm via PSD projection
is presented to solve the proposed model. Two other existing convex SPCA-based
models are also proven to have their solutions optimized on the PSD cone in
this paper. Therefore, the PSD versions of these two models are proposed to
accelerate their convergence as well. We also provide a regularization
parameter setting strategy for our proposed method. Experiments on synthetic
and real-world datasets demonstrate the effectiveness and efficiency of the
proposed methods.
less
By: Richard J. Chen, Tong Ding, Ming Y. Lu, Drew F. K. Williamson, Guillaume Jaume, Bowen Chen, Andrew Zhang, Daniel Shao, Andrew H. Song, Muhammad Shaban, Mane Williams, Anurag Vaidya, Sharifa Sahai, Lukas Oldenburg, Luca L. Weishaupt, Judy J. Wang, Walt Williams, Long Phi Le, Georg Gerber, Faisal Mahmood
Tissue phenotyping is a fundamental computational pathology (CPath) task in
learning objective characterizations of histopathologic biomarkers in anatomic
pathology. However, whole-slide imaging (WSI) poses a complex computer vision
problem in which the large-scale image resolutions of WSIs and the enormous
diversity of morphological phenotypes preclude large-scale data annotation.
Current efforts have proposed using pretrained image encode... more
Tissue phenotyping is a fundamental computational pathology (CPath) task in
learning objective characterizations of histopathologic biomarkers in anatomic
pathology. However, whole-slide imaging (WSI) poses a complex computer vision
problem in which the large-scale image resolutions of WSIs and the enormous
diversity of morphological phenotypes preclude large-scale data annotation.
Current efforts have proposed using pretrained image encoders with either
transfer learning from natural image datasets or self-supervised pretraining on
publicly-available histopathology datasets, but have not been extensively
developed and evaluated across diverse tissue types at scale. We introduce UNI,
a general-purpose self-supervised model for pathology, pretrained using over
100 million tissue patches from over 100,000 diagnostic haematoxylin and
eosin-stained WSIs across 20 major tissue types, and evaluated on 33
representative CPath clinical tasks in CPath of varying diagnostic
difficulties. In addition to outperforming previous state-of-the-art models, we
demonstrate new modeling capabilities in CPath such as resolution-agnostic
tissue classification, slide classification using few-shot class prototypes,
and disease subtyping generalization in classifying up to 108 cancer types in
the OncoTree code classification system. UNI advances unsupervised
representation learning at scale in CPath in terms of both pretraining data and
downstream evaluation, enabling data-efficient AI models that can generalize
and transfer to a gamut of diagnostically-challenging tasks and clinical
workflows in anatomic pathology.
less
By: Andrew Du, Anh-Dzung Doan, Yee Wei Law, Tat-Jun Chin
The advent of satellite-borne machine learning hardware accelerators has
enabled the on-board processing of payload data using machine learning
techniques such as convolutional neural networks (CNN). A notable example is
using a CNN to detect the presence of clouds in hyperspectral data captured on
Earth observation (EO) missions, whereby only clear sky data is downlinked to
conserve bandwidth. However, prior to deployment, new missions tha... more
The advent of satellite-borne machine learning hardware accelerators has
enabled the on-board processing of payload data using machine learning
techniques such as convolutional neural networks (CNN). A notable example is
using a CNN to detect the presence of clouds in hyperspectral data captured on
Earth observation (EO) missions, whereby only clear sky data is downlinked to
conserve bandwidth. However, prior to deployment, new missions that employ new
sensors will not have enough representative datasets to train a CNN model,
while a model trained solely on data from previous missions will underperform
when deployed to process the data on the new missions. This underperformance
stems from the domain gap, i.e., differences in the underlying distributions of
the data generated by the different sensors in previous and future missions. In
this paper, we address the domain gap problem in the context of on-board
hyperspectral cloud detection. Our main contributions lie in formulating new
domain adaptation tasks that are motivated by a concrete EO mission, developing
a novel algorithm for bandwidth-efficient supervised domain adaptation, and
demonstrating test-time adaptation algorithms on space deployable neural
network accelerators. Our contributions enable minimal data transmission to be
invoked (e.g., only 1% of the weights in ResNet50) to achieve domain
adaptation, thereby allowing more sophisticated CNN models to be deployed and
updated on satellites without being hampered by domain gap and bandwidth
limitations.
less
By: Haodong He, Jian Ding, Gui-Song Xia
The robustness of object detection models is a major concern when applied to
real-world scenarios. However, the performance of most object detection models
degrades when applied to images subjected to corruptions, since they are
usually trained and evaluated on clean datasets. Enhancing the robustness of
object detection models is of utmost importance, especially for those designed
for aerial images, which feature complex backgrounds, subst... more
The robustness of object detection models is a major concern when applied to
real-world scenarios. However, the performance of most object detection models
degrades when applied to images subjected to corruptions, since they are
usually trained and evaluated on clean datasets. Enhancing the robustness of
object detection models is of utmost importance, especially for those designed
for aerial images, which feature complex backgrounds, substantial variations in
scales and orientations of objects. This paper addresses the challenge of
assessing the robustness of object detection models in aerial images, with a
specific emphasis on scenarios where images are affected by clouds. In this
study, we introduce two novel benchmarks based on DOTA-v1.0. The first
benchmark encompasses 19 prevalent corruptions, while the second focuses on
cloud-corrupted images-a phenomenon uncommon in natural pictures yet frequent
in aerial photography. We systematically evaluate the robustness of mainstream
object detection models and perform numerous ablation experiments. Through our
investigations, we find that enhanced model architectures, larger networks,
well-crafted modules, and judicious data augmentation strategies collectively
enhance the robustness of aerial object detection models. The benchmarks we
propose and our comprehensive experimental analyses can facilitate research on
robust object detection in aerial images. Codes and datasets are available at:
(https://github.com/hehaodong530/DOTA-C)
less
By: Javier Fumanal-Idocin, Javier Andreu-Perez, Oscar Cordón, Hani Hagras, Humberto Bustince
Automatic art analysis employs different image processing techniques to
classify and categorize works of art. When working with artistic images, we
need to take into account further considerations compared to classical image
processing. This is because such artistic paintings change drastically
depending on the author, the scene depicted, and their artistic style. This can
result in features that perform very well in a given task but do not... more
Automatic art analysis employs different image processing techniques to
classify and categorize works of art. When working with artistic images, we
need to take into account further considerations compared to classical image
processing. This is because such artistic paintings change drastically
depending on the author, the scene depicted, and their artistic style. This can
result in features that perform very well in a given task but do not grasp the
whole of the visual and symbolic information contained in a painting. In this
paper, we show how the features obtained from different tasks in artistic image
classification are suitable to solve other ones of similar nature. We present
different methods to improve the generalization capabilities and performance of
artistic classification systems. Furthermore, we propose an explainable
artificial intelligence method to map known visual traits of an image with the
features used by the deep learning model considering fuzzy rules. These rules
show the patterns and variables that are relevant to solve each task and how
effective is each of the patterns found. Our results show that our proposed
context-aware features can achieve up to $6\%$ and $26\%$ more accurate results
than other context- and non-context-aware solutions, respectively, depending on
the specific task. We also show that some of the features used by these models
can be more clearly correlated to visual traits in the original image than
others.
less
By: Xinyang Huang, Chuang Zhu, Wenkai Chen
Cross-domain few-shot segmentation (CD-FSS) aims to achieve semantic
segmentation in previously unseen domains with a limited number of annotated
samples. Although existing CD-FSS models focus on cross-domain feature
transformation, relying exclusively on inter-domain knowledge transfer may lead
to the loss of critical intra-domain information. To this end, we propose a
novel residual transformation network (RestNet) that facilitates knowle... more
Cross-domain few-shot segmentation (CD-FSS) aims to achieve semantic
segmentation in previously unseen domains with a limited number of annotated
samples. Although existing CD-FSS models focus on cross-domain feature
transformation, relying exclusively on inter-domain knowledge transfer may lead
to the loss of critical intra-domain information. To this end, we propose a
novel residual transformation network (RestNet) that facilitates knowledge
transfer while retaining the intra-domain support-query feature information.
Specifically, we propose a Semantic Enhanced Anchor Transform (SEAT) module
that maps features to a stable domain-agnostic space using advanced semantics.
Additionally, an Intra-domain Residual Enhancement (IRE) module is designed to
maintain the intra-domain representation of the original discriminant space in
the new space. We also propose a mask prediction strategy based on prototype
fusion to help the model gradually learn how to segment. Our RestNet can
transfer cross-domain knowledge from both inter-domain and intra-domain without
requiring additional fine-tuning. Extensive experiments on ISIC, Chest X-ray,
and FSS-1000 show that our RestNet achieves state-of-the-art performance. Our
code will be available soon.
less
By: Shreyas Goyal, Jagath C. Rajapakse
Hotspot detection using thermal imaging has recently become essential in
several industrial applications, such as security applications, health
applications, and equipment monitoring applications. Hotspot detection is of
utmost importance in industrial safety where equipment can develop anomalies.
Hotspots are early indicators of such anomalies. We address the problem of
hotspot detection in thermal images by proposing a self-supervised lea... more
Hotspot detection using thermal imaging has recently become essential in
several industrial applications, such as security applications, health
applications, and equipment monitoring applications. Hotspot detection is of
utmost importance in industrial safety where equipment can develop anomalies.
Hotspots are early indicators of such anomalies. We address the problem of
hotspot detection in thermal images by proposing a self-supervised learning
approach. Self-supervised learning has shown potential as a competitive
alternative to their supervised learning counterparts but their application to
thermography has been limited. This has been due to lack of diverse data
availability, domain specific pre-trained models, standardized benchmarks, etc.
We propose a self-supervised representation learning approach followed by
fine-tuning that improves detection of hotspots by classification. The SimSiam
network based ensemble classifier decides whether an image contains hotspots or
not. Detection of hotspots is followed by precise hotspot isolation. By doing
so, we are able to provide a highly accurate and precise hotspot
identification, applicable to a wide range of applications. We created a novel
large thermal image dataset to address the issue of paucity of easily
accessible thermal images. Our experiments with the dataset created by us and a
publicly available segmentation dataset show the potential of our approach for
hotspot detection and its ability to isolate hotspots with high accuracy. We
achieve a Dice Coefficient of 0.736, the highest when compared with existing
hotspot identification techniques. Our experiments also show self-supervised
learning as a strong contender of supervised learning, providing competitive
metrics for hotspot detection, with the highest accuracy of our approach being
97%.
less
By: Siyang Jiang, Rui Fang, Hsi-Wen Chen, Wei Ding, Ming-Syan Chen
Support-query shift few-shot learning aims to classify unseen examples (query
set) to labeled data (support set) based on the learned embedding in a
low-dimensional space under a distribution shift between the support set and
the query set. However, in real-world scenarios the shifts are usually unknown
and varied, making it difficult to estimate in advance. Therefore, in this
paper, we propose a novel but more difficult challenge, RSQS, fo... more
Support-query shift few-shot learning aims to classify unseen examples (query
set) to labeled data (support set) based on the learned embedding in a
low-dimensional space under a distribution shift between the support set and
the query set. However, in real-world scenarios the shifts are usually unknown
and varied, making it difficult to estimate in advance. Therefore, in this
paper, we propose a novel but more difficult challenge, RSQS, focusing on
Realistic Support-Query Shift few-shot learning. The key feature of RSQS is
that the individual samples in a meta-task are subjected to multiple
distribution shifts in each meta-task. In addition, we propose a unified
adversarial feature alignment method called DUal adversarial ALignment
framework (DuaL) to relieve RSQS from two aspects, i.e., inter-domain bias and
intra-domain variance. On the one hand, for the inter-domain bias, we corrupt
the original data in advance and use the synthesized perturbed inputs to train
the repairer network by minimizing distance in the feature level. On the other
hand, for intra-domain variance, we proposed a generator network to synthesize
hard, i.e., less similar, examples from the support set in a self-supervised
manner and introduce regularized optimal transportation to derive a smooth
optimal transportation plan. Lastly, a benchmark of RSQS is built with several
state-of-the-art baselines among three datasets (CIFAR100, mini-ImageNet, and
Tiered-Imagenet). Experiment results show that DuaL significantly outperforms
the state-of-the-art methods in our benchmark.
less
By: Tanvir Hassan, A. Ben Hamza
Graph convolutional networks and their variants have shown significant
promise in 3D human pose estimation. Despite their success, most of these
methods only consider spatial correlations between body joints and do not take
into account temporal correlations, thereby limiting their ability to capture
relationships in the presence of occlusions and inherent ambiguity. To address
this potential weakness, we propose a spatio-temporal network a... more
Graph convolutional networks and their variants have shown significant
promise in 3D human pose estimation. Despite their success, most of these
methods only consider spatial correlations between body joints and do not take
into account temporal correlations, thereby limiting their ability to capture
relationships in the presence of occlusions and inherent ambiguity. To address
this potential weakness, we propose a spatio-temporal network architecture
composed of a joint-mixing multi-layer perceptron block that facilitates
communication among different joints and a graph weighted Jacobi network block
that enables communication among various feature channels. The major novelty of
our approach lies in a new weighted Jacobi feature propagation rule obtained
through graph filtering with implicit fairing. We leverage temporal information
from the 2D pose sequences, and integrate weight modulation into the model to
enable untangling of the feature transformations of distinct nodes. We also
employ adjacency modulation with the aim of learning meaningful correlations
beyond defined linkages between body joints by altering the graph topology
through a learnable modulation matrix. Extensive experiments on two benchmark
datasets demonstrate the effectiveness of our model, outperforming recent
state-of-the-art methods for 3D human pose estimation.
less
By: Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, Jinqiao Wang
Large Vision-Language Models (LVLMs) such as MiniGPT-4 and LLaVA have
demonstrated the capability of understanding images and achieved remarkable
performance in various visual tasks. Despite their strong abilities in
recognizing common objects due to extensive training datasets, they lack
specific domain knowledge and have a weaker understanding of localized details
within objects, which hinders their effectiveness in the Industrial Anomaly... more
Large Vision-Language Models (LVLMs) such as MiniGPT-4 and LLaVA have
demonstrated the capability of understanding images and achieved remarkable
performance in various visual tasks. Despite their strong abilities in
recognizing common objects due to extensive training datasets, they lack
specific domain knowledge and have a weaker understanding of localized details
within objects, which hinders their effectiveness in the Industrial Anomaly
Detection (IAD) task. On the other hand, most existing IAD methods only provide
anomaly scores and necessitate the manual setting of thresholds to distinguish
between normal and abnormal samples, which restricts their practical
implementation. In this paper, we explore the utilization of LVLM to address
the IAD problem and propose AnomalyGPT, a novel IAD approach based on LVLM. We
generate training data by simulating anomalous images and producing
corresponding textual descriptions for each image. We also employ an image
decoder to provide fine-grained semantic and design a prompt learner to
fine-tune the LVLM using prompt embeddings. Our AnomalyGPT eliminates the need
for manual threshold adjustments, thus directly assesses the presence and
locations of anomalies. Additionally, AnomalyGPT supports multi-turn dialogues
and exhibits impressive few-shot in-context learning capabilities. With only
one normal shot, AnomalyGPT achieves the state-of-the-art performance with an
accuracy of 86.1%, an image-level AUC of 94.1%, and a pixel-level AUC of 95.3%
on the MVTec-AD dataset. Code is available at
https://github.com/CASIA-IVA-Lab/AnomalyGPT.
less
By: Tuan Pham Minh, Jayan Wijesingha, Daniel Kottke, Marek Herde, Denis Huseljic, Bernhard Sick, Michael Wachendorf, Thomas Esch
Remote sensing through semantic segmentation of satellite images contributes
to the understanding and utilisation of the earth's surface. For this purpose,
semantic segmentation networks are typically trained on large sets of labelled
satellite images. However, obtaining expert labels for these images is costly.
Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or
pretrained networks, to label the images in the first ... more
Remote sensing through semantic segmentation of satellite images contributes
to the understanding and utilisation of the earth's surface. For this purpose,
semantic segmentation networks are typically trained on large sets of labelled
satellite images. However, obtaining expert labels for these images is costly.
Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or
pretrained networks, to label the images in the first step. Since these initial
labels are partially erroneous, we use active learning strategies to
cost-efficiently refine the labels in the second step. We evaluate the active
learning strategies using satellite images of Bengaluru in India, labelled with
land cover and land use labels. Our experimental results suggest that an active
label refinement to improve the semantic segmentation network's performance is
beneficial.
less
By: Zhenfeng Fan, Zhiheng Zhang, Shuang Yang, Chongyang Zhong, Min Cao, Shihong Xia
While impressive progress has recently been made in image-oriented facial
attribute translation, shape-oriented 3D facial attribute translation remains
an unsolved issue. This is primarily limited by the lack of 3D generative
models and ineffective usage of 3D facial data. We propose a learning framework
for 3D facial attribute translation to relieve these limitations. Firstly, we
customize a novel geometric map for 3D shape representation ... more
While impressive progress has recently been made in image-oriented facial
attribute translation, shape-oriented 3D facial attribute translation remains
an unsolved issue. This is primarily limited by the lack of 3D generative
models and ineffective usage of 3D facial data. We propose a learning framework
for 3D facial attribute translation to relieve these limitations. Firstly, we
customize a novel geometric map for 3D shape representation and embed it in an
end-to-end generative adversarial network. The geometric map represents 3D
shapes symmetrically on a square image grid, while preserving the neighboring
relationship of 3D vertices in a local least-square sense. This enables
effective learning for the latent representation of data with different
attributes. Secondly, we employ a unified and unpaired learning framework for
multi-domain attribute translation. It not only makes effective usage of data
correlation from multiple domains, but also mitigates the constraint for hardly
accessible paired data. Finally, we propose a hierarchical architecture for the
discriminator to guarantee robust results against both global and local
artifacts. We conduct extensive experiments to demonstrate the advantage of the
proposed framework over the state-of-the-art in generating high-fidelity facial
shapes. Given an input 3D facial shape, the proposed framework is able to
synthesize novel shapes of different attributes, which covers some downstream
applications, such as expression transfer, gender translation, and aging. Code
at https://github.com/NaughtyZZ/3D_facial_shape_attribute_translation_ssgmap.
less