Showing 1–50 of 11,296 results for author: Zhang, X

Guidelines for Augmentation Selection in Contrastive Learning for Time Series Classification

Authors: Ziyu Liu, Azadeh Alavi, Minyi Li, Xiang Zhang

Abstract: Self-supervised contrastive learning has become a key technique in deep learning, particularly in time series analysis, due to its ability to learn meaningful representations without explicit supervision. Augmentation is a critical component in contrastive learning, where different augmentations can dramatically impact performance, sometimes influencing accuracy by over 30%. However, the selection… ▽ More Self-supervised contrastive learning has become a key technique in deep learning, particularly in time series analysis, due to its ability to learn meaningful representations without explicit supervision. Augmentation is a critical component in contrastive learning, where different augmentations can dramatically impact performance, sometimes influencing accuracy by over 30%. However, the selection of augmentations is predominantly empirical which can be suboptimal, or grid searching that is time-consuming. In this paper, we establish a principled framework for selecting augmentations based on dataset characteristics such as trend and seasonality. Specifically, we construct 12 synthetic datasets incorporating trend, seasonality, and integration weights. We then evaluate the effectiveness of 8 different augmentations across these synthetic datasets, thereby inducing generalizable associations between time series characteristics and augmentation efficiency. Additionally, we evaluated the induced associations across 6 real-world datasets encompassing domains such as activity recognition, disease diagnosis, traffic monitoring, electricity usage, mechanical fault prognosis, and finance. These real-world datasets are diverse, covering a range from 1 to 12 channels, 2 to 10 classes, sequence lengths of 14 to 1280, and data frequencies from 250 Hz to daily intervals. The experimental results show that our proposed trend-seasonality-based augmentation recommendation algorithm can accurately identify the effective augmentations for a given time series dataset, achieving an average Recall@3 of 0.667, outperforming baselines. Our work provides guidance for studies employing contrastive learning in time series analysis, with wide-ranging applications. All the code, datasets, and analysis results will be released at https://github.com/DL4mHealth/TS-Contrastive-Augmentation-Recommendation. △ Less

Submitted 12 July, 2024; originally announced July 2024.

Comments: 20 pages, 11 figures

Suppression of quantum dissipation: A cooperative effect of quantum squeezing and quantum measurement

Authors: Yi-Ming Xia, Yi-Fei Wang, Xiao-Yun Zhang, Hai-Chao Li, Wei Xiong

Abstract: The ability to isolate a quantum system from its environment is of fundamental interest and importance in optical quantum science and technology. Here we propose an experimentally feasible scheme for beating environment-induced dissipation in an open two-level system coupled to a parametrically driven cavity. The mechanism relies on a novel cooperation between light-matter coupling enhancement and… ▽ More The ability to isolate a quantum system from its environment is of fundamental interest and importance in optical quantum science and technology. Here we propose an experimentally feasible scheme for beating environment-induced dissipation in an open two-level system coupled to a parametrically driven cavity. The mechanism relies on a novel cooperation between light-matter coupling enhancement and frequent measurements. We demonstrate that, in the presence of the cooperation, the system dynamics can be completely dominated by the effective system-cavity interaction and the dissipative effects from the system-environment coupling can be surprisingly ignored. This work provides a generic method of dissipation suppression in a variety of quantum mechanical platforms, including natural atoms and superconducting circuits. △ Less

Submitted 12 July, 2024; originally announced July 2024.

Unifying Sequences, Structures, and Descriptions for Any-to-Any Protein Generation with the Large Multimodal Model HelixProtX

Authors: Zhiyuan Chen, Tianhao Chen, Chenggang Xie, Yang Xue, Xiaonan Zhang, Jingbo Zhou, Xiaomin Fang

Abstract: Proteins are fundamental components of biological systems and can be represented through various modalities, including sequences, structures, and textual descriptions. Despite the advances in deep learning and scientific large language models (LLMs) for protein research, current methodologies predominantly focus on limited specialized tasks -- often predicting one protein modality from another. Th… ▽ More Proteins are fundamental components of biological systems and can be represented through various modalities, including sequences, structures, and textual descriptions. Despite the advances in deep learning and scientific large language models (LLMs) for protein research, current methodologies predominantly focus on limited specialized tasks -- often predicting one protein modality from another. These approaches restrict the understanding and generation of multimodal protein data. In contrast, large multimodal models have demonstrated potential capabilities in generating any-to-any content like text, images, and videos, thus enriching user interactions across various domains. Integrating these multimodal model technologies into protein research offers significant promise by potentially transforming how proteins are studied. To this end, we introduce HelixProtX, a system built upon the large multimodal model, aiming to offer a comprehensive solution to protein research by supporting any-to-any protein modality generation. Unlike existing methods, it allows for the transformation of any input protein modality into any desired protein modality. The experimental results affirm the advanced capabilities of HelixProtX, not only in generating functional descriptions from amino acid sequences but also in executing critical tasks such as designing protein sequences and structures from textual descriptions. Preliminary findings indicate that HelixProtX consistently achieves superior accuracy across a range of protein-related tasks, outperforming existing state-of-the-art models. By integrating multimodal large models into protein research, HelixProtX opens new avenues for understanding protein biology, thereby promising to accelerate scientific discovery. △ Less

Submitted 12 July, 2024; originally announced July 2024.

HPC: Hierarchical Progressive Coding Framework for Volumetric Video

Authors: Zihan Zheng, Houqiang Zhong, Qiang Hu, Xiaoyun Zhang, Li Song, Ya Zhang, Yanfeng Wang

Abstract: Volumetric video based on Neural Radiance Field (NeRF) holds vast potential for various 3D applications, but its substantial data volume poses significant challenges for compression and transmission. Current NeRF compression lacks the flexibility to adjust video quality and bitrate within a single model for various network and device capacities. To address these issues, we propose HPC, a novel hie… ▽ More Volumetric video based on Neural Radiance Field (NeRF) holds vast potential for various 3D applications, but its substantial data volume poses significant challenges for compression and transmission. Current NeRF compression lacks the flexibility to adjust video quality and bitrate within a single model for various network and device capacities. To address these issues, we propose HPC, a novel hierarchical progressive volumetric video coding framework achieving variable bitrate using a single model. Specifically, HPC introduces a hierarchical representation with a multi-resolution residual radiance field to reduce temporal redundancy in long-duration sequences while simultaneously generating various levels of detail. Then, we propose an end-to-end progressive learning approach with a multi-rate-distortion loss function to jointly optimize both hierarchical representation and compression. Our HPC trained only once can realize multiple compression levels, while the current methods need to train multiple fixed-bitrate models for different rate-distortion (RD) tradeoffs. Extensive experiments demonstrate that HPC achieves flexible quality levels with variable bitrate by a single model and exhibits competitive RD performance, even outperforming fixed-bitrate models across various datasets. △ Less

Submitted 12 July, 2024; originally announced July 2024.

Comments: 11 pages, 7 figures

Probing Cold-to-Temperate Exoplanetary Atmospheres: The Role of Water Condensation on Surface Identification with JWST

Authors: Ziyu Huang, Xinting Yu, Shang-Min Tsai, Julianne I. Moses, Kazumasa Ohno, Joshua Krissansen-Totton, Xi Zhang, Jonathan Fortney

Abstract: Understanding the surface temperature and interior structure of cold-to-temperate sub-Neptunes is critical for assessing their habitability, yet direct observations are challenging. In this study, we investigate the impact of water condensation on the atmospheric compositions of sub-Neptunes, focusing on the implications for JWST spectroscopic observations. By modeling the atmospheric photochemist… ▽ More Understanding the surface temperature and interior structure of cold-to-temperate sub-Neptunes is critical for assessing their habitability, yet direct observations are challenging. In this study, we investigate the impact of water condensation on the atmospheric compositions of sub-Neptunes, focusing on the implications for JWST spectroscopic observations. By modeling the atmospheric photochemistry of two canonical sub-Neptunes, K2-18 b and LHS 1140 b, both with and without water condensation and with and without thick atmospheres, we demonstrate that water condensation can significantly affect the predicted atmospheric compositions. This effect is driven by oxygen depletion from the condensation of water vapor and primarily manifests as an increase in the C/O ratio within the photochemically active regions of the atmosphere. This change in composition particularly affects planets with thin H$_2$-dominated atmospheres, leading to a transition in dominant nitrogen and carbon carriers from N$_2$ and oxygen-rich species like CO/CO$_2$ towards heavier hydrocarbons and nitriles. While our models do not fully account for the loss mechanisms of these higher-order species, such molecules can go on to form more refractory molecules or hazes. Planets with thin H2-rich atmospheres undergoing significant water condensation are thus likely to exhibit very hazy atmospheres. The relatively flat JWST spectra observed for LHS 1140 b could be consistent with such a scenario, suggesting a shallow surface with extensive water condensation or a high atmospheric C/O ratio. Conversely, the JWST observations of K2-18 b are better aligned with a volatile-rich mini-Neptune with a thick atmosphere. △ Less

Submitted 12 July, 2024; originally announced July 2024.

Comments: 21 pages, 7 figures

Dynamic neural network with memristive CIM and CAM for 2D and 3D vision

Authors: Yue Zhang, Woyu Zhang, Shaocong Wang, Ning Lin, Yifei Yu, Yangu He, Bo Wang, Hao Jiang, Peng Lin, Xiaoxin Xu, Xiaojuan Qi, Zhongrui Wang, Xumeng Zhang, Dashan Shang, Qi Liu, Kwang-Ting Cheng, Ming Liu

Abstract: The brain is dynamic, associative and efficient. It reconfigures by associating the inputs with past experiences, with fused memory and processing. In contrast, AI models are static, unable to associate inputs with past experiences, and run on digital computers with physically separated memory and processing. We propose a hardware-software co-design, a semantic memory-based dynamic neural network… ▽ More The brain is dynamic, associative and efficient. It reconfigures by associating the inputs with past experiences, with fused memory and processing. In contrast, AI models are static, unable to associate inputs with past experiences, and run on digital computers with physically separated memory and processing. We propose a hardware-software co-design, a semantic memory-based dynamic neural network (DNN) using memristor. The network associates incoming data with the past experience stored as semantic vectors. The network and the semantic memory are physically implemented on noise-robust ternary memristor-based Computing-In-Memory (CIM) and Content-Addressable Memory (CAM) circuits, respectively. We validate our co-designs, using a 40nm memristor macro, on ResNet and PointNet++ for classifying images and 3D points from the MNIST and ModelNet datasets, which not only achieves accuracy on par with software but also a 48.1% and 15.9% reduction in computational budget. Moreover, it delivers a 77.6% and 93.3% reduction in energy consumption. △ Less

Submitted 12 July, 2024; originally announced July 2024.

Comments: In press

DeCE: Deceptive Cross-Entropy Loss Designed for Defending Backdoor Attacks

Authors: Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Terry Yue Zhuo, David Lo, Taolue Chen

Abstract: Code Language Models (CLMs), particularly those leveraging deep learning, have achieved significant success in code intelligence domain. However, the issue of security, particularly backdoor attacks, is often overlooked in this process. The previous research has focused on designing backdoor attacks for CLMs, but effective defenses have not been adequately addressed. In particular, existing defens… ▽ More Code Language Models (CLMs), particularly those leveraging deep learning, have achieved significant success in code intelligence domain. However, the issue of security, particularly backdoor attacks, is often overlooked in this process. The previous research has focused on designing backdoor attacks for CLMs, but effective defenses have not been adequately addressed. In particular, existing defense methods from natural language processing, when directly applied to CLMs, are not effective enough and lack generality, working well in some models and scenarios but failing in others, thus fall short in consistently mitigating backdoor attacks. To bridge this gap, we first confirm the phenomenon of ``early learning" as a general occurrence during the training of CLMs. This phenomenon refers to that a model initially focuses on the main features of training data but may become more sensitive to backdoor triggers over time, leading to overfitting and susceptibility to backdoor attacks. We then analyze that overfitting to backdoor triggers results from the use of the cross-entropy loss function, where the unboundedness of cross-entropy leads the model to increasingly concentrate on the features of the poisoned data. Based on this insight, we propose a general and effective loss function DeCE (Deceptive Cross-Entropy) by blending deceptive distributions and applying label smoothing to limit the gradient to be bounded, which prevents the model from overfitting to backdoor triggers and then enhances the security of CLMs against backdoor attacks. To verify the effectiveness of our defense method, we select code synthesis tasks as our experimental scenarios. Our experiments across various code synthesis datasets, models, and poisoning ratios demonstrate the applicability and effectiveness of DeCE in enhancing the security of CLMs. △ Less

Submitted 11 July, 2024; originally announced July 2024.

Comments: Under Review; Waiting for updates

Quantitative Evaluation of the Saliency Map for Alzheimer's Disease Classifier with Anatomical Segmentation

Authors: Yihan Zhang, Xuanshuo Zhang, Wei Wu, Haohan Wang

Abstract: Saliency maps have been widely used to interpret deep learning classifiers for Alzheimer's disease (AD). However, since AD is heterogeneous and has multiple subtypes, the pathological mechanism of AD remains not fully understood and may vary from patient to patient. Due to the lack of such understanding, it is difficult to comprehensively and effectively assess the saliency map of AD classifier. I… ▽ More Saliency maps have been widely used to interpret deep learning classifiers for Alzheimer's disease (AD). However, since AD is heterogeneous and has multiple subtypes, the pathological mechanism of AD remains not fully understood and may vary from patient to patient. Due to the lack of such understanding, it is difficult to comprehensively and effectively assess the saliency map of AD classifier. In this paper, we utilize the anatomical segmentation to allocate saliency values into different brain regions. By plotting the distributions of saliency maps corresponding to AD and NC (Normal Control), we can gain a comprehensive view of the model's decisions process. In order to leverage the fact that the brain volume shrinkage happens in AD patients during disease progression, we define a new evaluation metric, brain volume change score (VCS), by computing the average Pearson correlation of the brain volume changes and the saliency values of a model in different brain regions for each patient. Thus, the VCS metric can help us gain some knowledge of how saliency maps resulting from different models relate to the changes of the volumes across different regions in the whole brain. We trained candidate models on the ADNI dataset and tested on three different datasets. Our results indicate: (i) models with higher VCSs tend to demonstrate saliency maps with more details relevant to the AD pathology, (ii) using gradient-based adversarial training strategies such as FGSM and stochastic masking can improve the VCSs of the models. △ Less

Submitted 11 July, 2024; originally announced July 2024.

Beyond Instruction Following: Evaluating Rule Following of Large Language Models

Authors: Wangtao Sun, Chenxiang Zhang, Xueyou Zhang, Ziyang Huang, Haotian Xu, Pei Chen, Shizhu He, Jun Zhao, Kang Liu

Abstract: Although Large Language Models (LLMs) have demonstrated strong instruction-following ability to be helpful, they are further supposed to be controlled and guided by rules in real-world scenarios to be safe, and accurate in responses. This demands the possession of rule-following capability of LLMs. However, few works have made a clear evaluation of the rule-following capability of LLMs. Previous s… ▽ More Although Large Language Models (LLMs) have demonstrated strong instruction-following ability to be helpful, they are further supposed to be controlled and guided by rules in real-world scenarios to be safe, and accurate in responses. This demands the possession of rule-following capability of LLMs. However, few works have made a clear evaluation of the rule-following capability of LLMs. Previous studies that try to evaluate the rule-following capability of LLMs fail to distinguish the rule-following scenarios from the instruction-following scenarios. Therefore, this paper first makes a clarification of the concept of rule-following, and curates a comprehensive benchmark, RuleBench, to evaluate a diversified range of rule-following abilities. Our experimental results on a variety of LLMs show that they are still limited in following rules. Our further analysis provides insights into the improvements for LLMs toward a better rule-following intelligent agent. The data and code can be found at: https://anonymous.4open.science/r/llm-rule-following-B3E3/ △ Less

Submitted 11 July, 2024; originally announced July 2024.

Thermodynamic bounce effect in quantum BTZ black hole

Authors: Zhen-Ming Xu, Pan-Pan Zhang, Bin Wu, Xing Zhang

Abstract: A novel thermodynamic phenomenon has been observed in the quantum Bañados-Teitelboim-Zanelli (qBTZ) black hole, utilizing generalized free energy and Kramer escape rate. This phenomenon also reveals the unique property of the quantum black hole. The stochastic thermal motion of various thermodynamic states within the black hole system induces phase transitions, under the influence of generalized f… ▽ More A novel thermodynamic phenomenon has been observed in the quantum Bañados-Teitelboim-Zanelli (qBTZ) black hole, utilizing generalized free energy and Kramer escape rate. This phenomenon also reveals the unique property of the quantum black hole. The stochastic thermal motion of various thermodynamic states within the black hole system induces phase transitions, under the influence of generalized free energy which obtained by extending Maxwell's construction. Through the analysis of Kramer escape rate, it is discovered that the qBTZ black hole thermodynamic system exhibits a bounce effect. Furthermore, the overall thermodynamic picture of the qBTZ black hole has been obtained under different quantum backreactions. △ Less

Submitted 11 July, 2024; originally announced July 2024.

Comments: 8 pages, 3 figures

An Unsupervised Domain Adaptation Method for Locating Manipulated Region in partially fake Audio

Authors: Siding Zeng, Jiangyan Yi, Jianhua Tao, Yujie Chen, Shan Liang, Yong Ren, Xiaohui Zhang

Abstract: When the task of locating manipulation regions in partially-fake audio (PFA) involves cross-domain datasets, the performance of deep learning models drops significantly due to the shift between the source and target domains. To address this issue, existing approaches often employ data augmentation before training. However, they overlook the characteristics in target domain that are absent in sourc… ▽ More When the task of locating manipulation regions in partially-fake audio (PFA) involves cross-domain datasets, the performance of deep learning models drops significantly due to the shift between the source and target domains. To address this issue, existing approaches often employ data augmentation before training. However, they overlook the characteristics in target domain that are absent in source domain. Inspired by the mixture-of-experts model, we propose an unsupervised method named Samples mining with Diversity and Entropy (SDE). Our method first learns from a collection of diverse experts that achieve great performance from different perspectives in the source domain, but with ambiguity on target samples. We leverage these diverse experts to select the most informative samples by calculating their entropy. Furthermore, we introduced a label generation method tailored for these selected samples that are incorporated in the training process in source domain integrating the target domain information. We applied our method to a cross-domain partially fake audio detection dataset, ADD2023Track2. By introducing 10% of unknown samples from the target domain, we achieved an F1 score of 43.84%, which represents a relative increase of 77.2% compared to the second-best method. △ Less

Submitted 11 July, 2024; originally announced July 2024.

Sulphur dioxide in the mid-infrared transmission spectrum of WASP-39b

Authors: Diana Powell, Adina D. Feinstein, Elspeth K. H. Lee, Michael Zhang, Shang-Min Tsai, Jake Taylor, James Kirk, Taylor Bell, Joanna K. Barstow, Peter Gao, Jacob L. Bean, Jasmina Blecic, Katy L. Chubb, Ian J. M. Crossfield, Sean Jordan, Daniel Kitzmann, Sarah E. Moran, Giuseppe Morello, Julianne I. Moses, Luis Welbanks, Jeehyun Yang, Xi Zhang, Eva-Maria Ahrer, Aaron Bello-Arufe, Jonathan Brande , et al. (48 additional authors not shown)

Abstract: The recent inference of sulphur dioxide (SO$_2$) in the atmosphere of the hot ($\sim$1100 K), Saturn-mass exoplanet WASP-39b from near-infrared JWST observations suggests that photochemistry is a key process in high temperature exoplanet atmospheres. This is due to the low ($<$1 ppb) abundance of SO$_2$ under thermochemical equilibrium, compared to that produced from the photochemistry of H$_2$O a… ▽ More The recent inference of sulphur dioxide (SO$_2$) in the atmosphere of the hot ($\sim$1100 K), Saturn-mass exoplanet WASP-39b from near-infrared JWST observations suggests that photochemistry is a key process in high temperature exoplanet atmospheres. This is due to the low ($<$1 ppb) abundance of SO$_2$ under thermochemical equilibrium, compared to that produced from the photochemistry of H$_2$O and H$_2$S (1-10 ppm). However, the SO$_2$ inference was made from a single, small molecular feature in the transmission spectrum of WASP-39b at 4.05 $μ$m, and therefore the detection of other SO$_2$ absorption bands at different wavelengths is needed to better constrain the SO$_2$ abundance. Here we report the detection of SO$_2$ spectral features at 7.7 and 8.5 $μ$m in the 5-12 $μ$m transmission spectrum of WASP-39b measured by the JWST Mid-Infrared Instrument (MIRI) Low Resolution Spectrometer (LRS). Our observations suggest an abundance of SO$_2$ of 0.5-25 ppm (1$σ$ range), consistent with previous findings. In addition to SO$_2$, we find broad water vapour absorption features, as well as an unexplained decrease in the transit depth at wavelengths longer than 10 $μ$m. Fitting the spectrum with a grid of atmospheric forward models, we derive an atmospheric heavy element content (metallicity) for WASP-39b of $\sim$7.1-8.0 $\times$ solar and demonstrate that photochemistry shapes the spectra of WASP-39b across a broad wavelength range. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Comments: Published in Nature

Journal ref: Nature 626, 979-983 (2024)

Search, Examine and Early-Termination: Fake News Detection with Annotation-Free Evidences

Authors: Yuzhou Yang, Yangming Zhou, Qichao Ying, Zhenxing Qian, Xinpeng Zhang

Abstract: Pioneer researches recognize evidences as crucial elements in fake news detection apart from patterns. Existing evidence-aware methods either require laborious pre-processing procedures to assure relevant and high-quality evidence data, or incorporate the entire spectrum of available evidences in all news cases, regardless of the quality and quantity of the retrieved data. In this paper, we propos… ▽ More Pioneer researches recognize evidences as crucial elements in fake news detection apart from patterns. Existing evidence-aware methods either require laborious pre-processing procedures to assure relevant and high-quality evidence data, or incorporate the entire spectrum of available evidences in all news cases, regardless of the quality and quantity of the retrieved data. In this paper, we propose an approach named \textbf{SEE} that retrieves useful information from web-searched annotation-free evidences with an early-termination mechanism. The proposed SEE is constructed by three main phases: \textbf{S}earching online materials using the news as a query and directly using their titles as evidences without any annotating or filtering procedure, sequentially \textbf{E}xamining the news alongside with each piece of evidence via attention mechanisms to produce new hidden states with retrieved information, and allowing \textbf{E}arly-termination within the examining loop by assessing whether there is adequate confidence for producing a correct prediction. We have conducted extensive experiments on datasets with unprocessed evidences, i.e., Weibo21, GossipCop, and pre-processed evidences, namely Snopes and PolitiFact. The experimental results demonstrate that the proposed method outperforms state-of-the-art approaches. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Comments: ECAI 2024 paper. Fudan University & NVIDIA. To appear

Token-Mol 1.0: Tokenized drug design with large language model

Authors: Jike Wang, Rui Qin, Mingyang Wang, Meijing Fang, Yangyang Zhang, Yuchen Zhu, Qun Su, Qiaolin Gou, Chao Shen, Odin Zhang, Zhenxing Wu, Dejun Jiang, Xujun Zhang, Huifeng Zhao, Xiaozhe Wan, Zhourui Wu, Liwei Liu, Yu Kang, Chang-Yu Hsieh, Tingjun Hou

Abstract: Significant interests have recently risen in leveraging sequence-based large language models (LLMs) for drug design. However, most current applications of LLMs in drug discovery lack the ability to comprehend three-dimensional (3D) structures, thereby limiting their effectiveness in tasks that explicitly involve molecular conformations. In this study, we introduced Token-Mol, a token-only 3D drug… ▽ More Significant interests have recently risen in leveraging sequence-based large language models (LLMs) for drug design. However, most current applications of LLMs in drug discovery lack the ability to comprehend three-dimensional (3D) structures, thereby limiting their effectiveness in tasks that explicitly involve molecular conformations. In this study, we introduced Token-Mol, a token-only 3D drug design model. This model encodes all molecular information, including 2D and 3D structures, as well as molecular property data, into tokens, which transforms classification and regression tasks in drug discovery into probabilistic prediction problems, thereby enabling learning through a unified paradigm. Token-Mol is built on the transformer decoder architecture and trained using random causal masking techniques. Additionally, we proposed the Gaussian cross-entropy (GCE) loss function to overcome the challenges in regression tasks, significantly enhancing the capacity of LLMs to learn continuous numerical values. Through a combination of fine-tuning and reinforcement learning (RL), Token-Mol achieves performance comparable to or surpassing existing task-specific methods across various downstream tasks, including pocket-based molecular generation, conformation generation, and molecular property prediction. Compared to existing molecular pre-trained models, Token-Mol exhibits superior proficiency in handling a wider range of downstream tasks essential for drug design. Notably, our approach improves regression task accuracy by approximately 30% compared to similar token-only methods. Token-Mol overcomes the precision limitations of token-only models and has the potential to integrate seamlessly with general models such as ChatGPT, paving the way for the development of a universal artificial intelligence drug design model that facilitates rapid and high-quality drug design by experts. △ Less

Submitted 10 July, 2024; originally announced July 2024.

StoryDiffusion: How to Support UX Storyboarding With Generative-AI

Authors: Zhaohui Liang, Xiaoyu Zhang, Kevin Ma, Zhao Liu, Xipei Ren, Kosa Goucher-Lambert, Can Liu

Abstract: Storyboarding is an established method for designing user experiences. Generative AI can support this process by helping designers quickly create visual narratives. However, existing tools only focus on accurate text-to-image generation. Currently, it is not clear how to effectively support the entire creative process of storyboarding and how to develop AI-powered tools to support designers' indiv… ▽ More Storyboarding is an established method for designing user experiences. Generative AI can support this process by helping designers quickly create visual narratives. However, existing tools only focus on accurate text-to-image generation. Currently, it is not clear how to effectively support the entire creative process of storyboarding and how to develop AI-powered tools to support designers' individual workflows. In this work, we iteratively developed and implemented StoryDiffusion, a system that integrates text-to-text and text-to-image models, to support the generation of narratives and images in a single pipeline. With a user study, we observed 12 UX designers using the system for both concept ideation and illustration tasks. Our findings identified AI-directed vs. user-directed creative strategies in both tasks and revealed the importance of supporting the interchange between narrative iteration and image generation. We also found effects of the design tasks on their strategies and preferences, providing insights for future development. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Study of the decay and production properties of $D_{s1}(2536)$ and $D_{s2}^*(2573)$

Authors: M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere, A. Brueggemann , et al. (645 additional authors not shown)

Abstract: The $e^+e^-\rightarrow D_s^+D_{s1}(2536)^-$ and $e^+e^-\rightarrow D_s^+D^*_{s2}(2573)^-$ processes are studied using data samples collected with the BESIII detector at center-of-mass energies from 4.530 to 4.946~GeV. The absolute branching fractions of $D_{s1}(2536)^- \rightarrow \bar{D}^{*0}K^-$ and $D_{s2}^*(2573)^- \rightarrow \bar{D}^0K^-$ are measured for the first time to be… ▽ More The $e^+e^-\rightarrow D_s^+D_{s1}(2536)^-$ and $e^+e^-\rightarrow D_s^+D^*_{s2}(2573)^-$ processes are studied using data samples collected with the BESIII detector at center-of-mass energies from 4.530 to 4.946~GeV. The absolute branching fractions of $D_{s1}(2536)^- \rightarrow \bar{D}^{*0}K^-$ and $D_{s2}^*(2573)^- \rightarrow \bar{D}^0K^-$ are measured for the first time to be $(35.9\pm 4.8\pm 3.5)\%$ and $(37.4\pm 3.1\pm 4.6)\%$, respectively. The measurements are in tension with predictions based on the assumption that the $D_{s1}(2536)$ and $D_{s2}^*(2573)$ are dominated by a bare $c\bar{s}$ component. The $e^+e^-\rightarrow D_s^+D_{s1}(2536)^-$ and $e^+e^-\rightarrow D_s^+D^*_{s2}(2573)^-$ cross sections are measured, and a resonant structure at around 4.6~GeV with a width of 50~MeV is observed for the first time with a statistical significance of $15σ$ in the $e^+e^-\rightarrow D_s^+D^*_{s2}(2573)^-$ process. It could be the $Y(4626)$ found by the Belle collaboration in the $D_s^+D_{s1}(2536)^{-}$ final state, since they have similar masses and widths. There is also evidence for a structure at around 4.75~GeV in both processes. △ Less

Submitted 10 July, 2024; originally announced July 2024.

InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Prior

Authors: Chenguo Lin, Yuchen Lin, Panwang Pan, Xuanyang Zhang, Yadong Mu

Abstract: Comprehending natural language instructions is a charming property for both 2D and 3D layout synthesis systems. Existing methods implicitly model object joint distributions and express object relations, hindering generation's controllability. We introduce InstructLayout, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity… ▽ More Comprehending natural language instructions is a charming property for both 2D and 3D layout synthesis systems. Existing methods implicitly model object joint distributions and express object relations, hindering generation's controllability. We introduce InstructLayout, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity for 2D and 3D layout synthesis. The proposed semantic graph prior learns layout appearances and object distributions simultaneously, demonstrating versatility across various downstream tasks in a zero-shot manner. To facilitate the benchmarking for text-driven 2D and 3D scene synthesis, we respectively curate two high-quality datasets of layout-instruction pairs from public Internet resources with large language and multimodal models. Extensive experimental results reveal that the proposed method outperforms existing state-of-the-art approaches by a large margin in both 2D and 3D layout synthesis tasks. Thorough ablation studies confirm the efficacy of crucial design components. △ Less

Submitted 10 July, 2024; v1 submitted 10 July, 2024; originally announced July 2024.

Comments: This paper is an extension of ICLR 2024 "InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior". arXiv admin note: substantial text overlap with arXiv:2402.04717

Waveguide Superlattices with Artificial Gauge Field Towards Colorless and Crosstalkless Ultrahigh-Density Photonic Integration

Authors: Xuelin Zhang, Jiangbing Du, Ke Xu, Zuyuan He

Abstract: Dense waveguide arrays with low crosstalk and ultra-broadband remain a vital issue for chip-scale integrated photonics. However, the sub-wavelength regime of such devices has not been adequately explored in practice. Herein, we propose the advanced waveguide superlattices leveraging the artificial gauge field mechanism. This approach achieves remarkable -24 dB crosstalk suppression with an ultra-b… ▽ More Dense waveguide arrays with low crosstalk and ultra-broadband remain a vital issue for chip-scale integrated photonics. However, the sub-wavelength regime of such devices has not been adequately explored in practice. Herein, we propose the advanced waveguide superlattices leveraging the artificial gauge field mechanism. This approach achieves remarkable -24 dB crosstalk suppression with an ultra-broadband bandwidth, experimentally demonstrated over 500 nm, in silicon nitride waveguides. Moreover, the 112 Gbit/s signal encoded per channel of ultra-compact circuits with a bit error rate below the 7% forward error correction limit verified the capability for high-speed on-chip transmission. This design is compatible with metal back end-of-the-line (BEOL) processes and can be readily transferred to other platforms. Thus it holds great promise for significant reduction in on-chip footprint and cost in large-scale integrated photonics, and salient enhancement in the performance of a wide range of active and passive photonic devices and systems. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation

Authors: Hao Fang, Peng Wu, Yawei Li, Xinxin Zhang, Xiankai Lu

Abstract: Open-Vocabulary Video Instance Segmentation (VIS) is attracting increasing attention due to its ability to segment and track arbitrary objects. However, the recent Open-Vocabulary VIS attempts obtained unsatisfactory results, especially in terms of generalization ability of novel categories. We discover that the domain gap between the VLM features (e.g., CLIP) and the instance queries and the unde… ▽ More Open-Vocabulary Video Instance Segmentation (VIS) is attracting increasing attention due to its ability to segment and track arbitrary objects. However, the recent Open-Vocabulary VIS attempts obtained unsatisfactory results, especially in terms of generalization ability of novel categories. We discover that the domain gap between the VLM features (e.g., CLIP) and the instance queries and the underutilization of temporal consistency are two central causes. To mitigate these issues, we design and train a novel Open-Vocabulary VIS baseline called OVFormer. OVFormer utilizes a lightweight module for unified embedding alignment between query embeddings and CLIP image embeddings to remedy the domain gap. Unlike previous image-based training methods, we conduct video-based model training and deploy a semi-online inference scheme to fully mine the temporal consistency in the video. Without bells and whistles, OVFormer achieves 21.9 mAP with a ResNet-50 backbone on LV-VIS, exceeding the previous state-of-the-art performance by 7.7. Extensive experiments on some Close-Vocabulary VIS datasets also demonstrate the strong zero-shot generalization ability of OVFormer (+ 7.6 mAP on YouTube-VIS 2019, + 3.9 mAP on OVIS). Code is available at https://github.com/fanghaook/OVFormer. △ Less

Submitted 11 July, 2024; v1 submitted 10 July, 2024; originally announced July 2024.

Comments: ECCV 2024

Trustworthy Contrast-enhanced Brain MRI Synthesis

Authors: Jiyao Liu, Yuxin Li, Shangqi Gao, Yuncheng Zhou, Xin Gao, Ningsheng Xu, Xiao-Yong Zhang, Xiahai Zhuang

Abstract: Contrast-enhanced brain MRI (CE-MRI) is a valuable diagnostic technique but may pose health risks and incur high costs. To create safer alternatives, multi-modality medical image translation aims to synthesize CE-MRI images from other available modalities. Although existing methods can generate promising predictions, they still face two challenges, i.e., exhibiting over-confidence and lacking inte… ▽ More Contrast-enhanced brain MRI (CE-MRI) is a valuable diagnostic technique but may pose health risks and incur high costs. To create safer alternatives, multi-modality medical image translation aims to synthesize CE-MRI images from other available modalities. Although existing methods can generate promising predictions, they still face two challenges, i.e., exhibiting over-confidence and lacking interpretability on predictions. To address the above challenges, this paper introduces TrustI2I, a novel trustworthy method that reformulates multi-to-one medical image translation problem as a multimodal regression problem, aiming to build an uncertainty-aware and reliable system. Specifically, our method leverages deep evidential regression to estimate prediction uncertainties and employs an explicit intermediate and late fusion strategy based on the Mixture of Normal Inverse Gamma (MoNIG) distribution, enhancing both synthesis quality and interpretability. Additionally, we incorporate uncertainty calibration to improve the reliability of uncertainty. Validation on the BraTS2018 dataset demonstrates that our approach surpasses current methods, producing higher-quality images with rational uncertainty estimation. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Comments: 11 pages, 3 figures

Electrically Tuning Quasi-Bound States in the Continuum with Hybrid Graphene-Silicon Metasurfaces

Authors: Ziqiang Cai, Xianzhe Zhang, Tushar Sanjay Karnik, Yihao Xu, Tae Yoon Kim, Juejun Hu, Yongmin Liu

Abstract: Metasurfaces have become one of the most prominent research topics in the field of optics owing to their unprecedented properties and novel applications on an ultrathin platform. By combining graphene with metasurfaces, electrical tunable functions can be achieved with fast tuning speed, large modulation depth and broad tuning range. However, the tuning efficiency of hybrid graphene metasurfaces w… ▽ More Metasurfaces have become one of the most prominent research topics in the field of optics owing to their unprecedented properties and novel applications on an ultrathin platform. By combining graphene with metasurfaces, electrical tunable functions can be achieved with fast tuning speed, large modulation depth and broad tuning range. However, the tuning efficiency of hybrid graphene metasurfaces within the short-wavelength infrared (SWIR) spectrum is typically low because of the small resonance wavelength shift in this wavelength range. In this work, through the integration of graphene and silicon metasurfaces that support quasi-bound states in the continuum (quasi-BIC), we experimentally demonstrate significant transmittance tuning even with less than 30 nm resonance wavelength shift thanks to the high quality-factor of quasi-BIC metasurfaces. The tunable transmittance spectrum was measured using Fourier Transform Infrared Spectroscopy (FTIR) with a modified reflective lens to improve the accuracy, and the electrical tuning was realized utilizing the cut-and-stick method of ion gel. At the wavelength of 3.0 um, the measured change of transmittance T_max-T_min and modulation depth (T_max-T_min)/T_max can reach 22.2% and 28.9%, respectively, under a small bias voltage ranging from -2 V to +2 V. To the best of our knowledge, this work is the first experimental demonstration of tunable graphene/quasi-BIC metasurfaces, which have potential applications in optical modulation, reconfigurable photonic devices, and optical communications. △ Less

Submitted 9 July, 2024; originally announced July 2024.

Comments: 14 pages, 5 figures

Testing the cosmic distance duality relation using strong gravitational lensing time delays and Type Ia supernovae

Authors: Jing-Zhao Qi, Yi-Fan Jiang, Wan-Ting Hou, Xin Zhang

Abstract: We present a comprehensive test of the cosmic distance duality relation (DDR) using a combination of strong gravitational lensing (SGL) time delay measurements and Type Ia supernovae (SNe Ia) data. We investigate three different parameterizations of potential DDR violations. To bridge the gap between SGL and SNe Ia datasets, we implement an artificial neural network (ANN) approach to reconstruct t… ▽ More We present a comprehensive test of the cosmic distance duality relation (DDR) using a combination of strong gravitational lensing (SGL) time delay measurements and Type Ia supernovae (SNe Ia) data. We investigate three different parameterizations of potential DDR violations. To bridge the gap between SGL and SNe Ia datasets, we implement an artificial neural network (ANN) approach to reconstruct the distance modulus of SNe Ia. Our analysis uniquely considers both scenarios where the absolute magnitude of SNe Ia ($M_B$) is treated as a free parameter and where it is fixed to a Cepheid-calibrated value. Using a sample of six SGL systems and the Pantheon+ SNe Ia dataset, we find no statistically significant evidence for DDR violations across all parameterizations. The consistency of our findings across different parameterizations not only reinforces confidence in the standard DDR but also demonstrates the robustness of our analytical approach. △ Less

Submitted 9 July, 2024; originally announced July 2024.

Comments: 11 pages, 3 figures

Bimerons create bimerons: proliferation and aggregation induced by currents and magnetic fields

Authors: Xichao Zhang, Yan Zhou, Xiuzhen Yu, Masahito Mochizuki

Abstract: The aggregation of topological spin textures at nano and micro scales has practical applications in spintronic technologies. Here, the authors report the in-plane current-induced proliferation and aggregation of bimerons in a bulk chiral magnet. It is found that the spin-transfer torques can induce the proliferation and aggregation of bimerons only in the presence of an appropriate out-of-plane ma… ▽ More The aggregation of topological spin textures at nano and micro scales has practical applications in spintronic technologies. Here, the authors report the in-plane current-induced proliferation and aggregation of bimerons in a bulk chiral magnet. It is found that the spin-transfer torques can induce the proliferation and aggregation of bimerons only in the presence of an appropriate out-of-plane magnetic field. It is also found that a relatively small damping and a relatively large non-adiabatic spin-transfer torque could lead to more pronounced bimeron proliferation and aggregation. Particularly, the current density should be larger than a certain threshold in order to trigger the proliferation; namely, the bimerons may only be driven into translational motion under weak current injection. Besides, the authors find that the aggregate bimerons could relax into a deformed honeycomb bimeron lattice with a few lattice structure defects after the current injection. The results are promising for the development of bio-inspired spintronic devices that use a large number of aggregate bimerons. The findings also provide a platform for studying aggregation-induced effects in spintronic systems, such as the aggregation-induced lattice phase transitions. △ Less

Submitted 9 July, 2024; originally announced July 2024.

Comments: 13 pages, 8 figures

Transformation of a cellular skyrmion to polyomino-like structures

Authors: Jing Xia, Xichao Zhang, Yan Zhou, Xiaoxi Liu, Guoping Zhao, Masahito Mochizuki

Abstract: Topological spin structures with transformable shapes may have potential implications on data storage and computation. Here, we demonstrate that a square cellular skyrmion on an artificial grid pinning pattern can be manipulated by programmed current pulses. We find that parallel short pulses could result in the elongation of the skyrmion mainly in the current direction, while parallel long pulses… ▽ More Topological spin structures with transformable shapes may have potential implications on data storage and computation. Here, we demonstrate that a square cellular skyrmion on an artificial grid pinning pattern can be manipulated by programmed current pulses. We find that parallel short pulses could result in the elongation of the skyrmion mainly in the current direction, while parallel long pulses are able to induce the elongation in the direction perpendicular to the current due to the intrinsic skyrmion Hall effect. Consequently, a programmed sequence of parallel pulses could lead to the transformation of the skyrmion to I-, L-, and Z-shaped polyomino-like structures without affecting the topological charge. In addition, we find that orthogonal pulses could lead to the transformation to more complex polyomino-like structures, including the T-shaped and irregular ones. Particularly, when a small T-shaped structure is formed, the topological charge of the system is found to be non-integer due to incomplete compensation of local topological charge densities; however, the T-shaped structure is stable on the attractive pinning pattern. Our results offer an effective way to create polyomino-like spin structures toward functional applications. △ Less

Submitted 9 July, 2024; originally announced July 2024.

Comments: 7 pages, 5 figures

Electrical switching of spin-polarized light-emitting diodes based on a 2D CrI3/hBN/WSe2 heterostructure

Authors: Jianchen Dang, Tongyao Wu, Shuohua Yan, Kenji Watanabe, Takashi Taniguchi, Hechang Lei, Xiao-Xiao Zhang

Abstract: Spin-polarized light-emitting diodes (spin-LEDs) convert the electronic spin information to photon circular polarization, offering potential applications including spin amplification, optical communications, and advanced imaging. The conventional control of the emitted light's circular polarization requires a change in the external magnetic field, limiting the operation conditions of spin-LEDs. He… ▽ More Spin-polarized light-emitting diodes (spin-LEDs) convert the electronic spin information to photon circular polarization, offering potential applications including spin amplification, optical communications, and advanced imaging. The conventional control of the emitted light's circular polarization requires a change in the external magnetic field, limiting the operation conditions of spin-LEDs. Here, we demonstrate an atomically thin spin-LED device based on a heterostructure of a monolayer WSe2 and a few-layer antiferromagnetic CrI3, separated by a thin hBN tunneling barrier. The CrI3 and hBN layers polarize the spin of the injected carriers into the WSe2. With the valley optical selection rule in the monolayer WSe2, the electroluminescence exhibits a high degree of circular polarization that follows the CrI3 magnetic states. Importantly, we show an efficient electrical tuning, including a sign reversal, of the electroluminescent circular polarization by applying an electrostatic field due to the electrical tunability of the few-layer CrI3 magnetization. Our results establish a new platform to achieve on-demand operation of nanoscale spin-LED and electrical control of helicity for device applications. △ Less

Submitted 9 July, 2024; originally announced July 2024.

Limiting Over-Smoothing and Over-Squashing of Graph Message Passing by Deep Scattering Transforms

Authors: Yuanhong Jiang, Dongmian Zou, Xiaoqun Zhang, Yu Guang Wang

Abstract: Graph neural networks (GNNs) have become pivotal tools for processing graph-structured data, leveraging the message passing scheme as their core mechanism. However, traditional GNNs often grapple with issues such as instability, over-smoothing, and over-squashing, which can degrade performance and create a trade-off dilemma. In this paper, we introduce a discriminatively trained, multi-layer Deep… ▽ More Graph neural networks (GNNs) have become pivotal tools for processing graph-structured data, leveraging the message passing scheme as their core mechanism. However, traditional GNNs often grapple with issues such as instability, over-smoothing, and over-squashing, which can degrade performance and create a trade-off dilemma. In this paper, we introduce a discriminatively trained, multi-layer Deep Scattering Message Passing (DSMP) neural network designed to overcome these challenges. By harnessing spectral transformation, the DSMP model aggregates neighboring nodes with global information, thereby enhancing the precision and accuracy of graph signal processing. We provide theoretical proofs demonstrating the DSMP's effectiveness in mitigating these issues under specific conditions. Additionally, we support our claims with empirical evidence and thorough frequency analysis, showcasing the DSMP's superior ability to address instability, over-smoothing, and over-squashing. △ Less

Submitted 9 July, 2024; originally announced July 2024.

Comments: 35 pages, 6 figures

Detection-Triggered Recursive Impact Mitigation against Secondary False Data Injection Attacks in Microgrids

Authors: Mengxiang Liu, Xin Zhang, Rui Zhang, Zhuoran Zhou, Zhenyong Zhang, Ruilong Deng

Abstract: The cybersecurity of microgrid has received widespread attentions due to the frequently reported attack accidents against distributed energy resource (DER) manufactures. Numerous impact mitigation schemes have been proposed to reduce or eliminate the impacts of false data injection attacks (FDIAs). Nevertheless, the existing methods either requires at least one neighboring trustworthy agent or may… ▽ More The cybersecurity of microgrid has received widespread attentions due to the frequently reported attack accidents against distributed energy resource (DER) manufactures. Numerous impact mitigation schemes have been proposed to reduce or eliminate the impacts of false data injection attacks (FDIAs). Nevertheless, the existing methods either requires at least one neighboring trustworthy agent or may bring in unacceptable cost burdens. This paper aims to propose a detection-triggered recursive impact mitigation scheme that can timely and precisely counter the secondary FDIAs (SFDIAs) against the communication links among DERs. Once triggering attack alarms, the power line current readings will be utilised to observe the voltage bias injections through the physical interconnections among DERs, based on which the current bias injections can be recursively reconstructed from the residuals generated by unknown input observers (UIOs). The attack impacts are eliminated by subtracting the reconstructed bias from the incoming compromised data. The proposed mitigation method can work even in the worst case where all communication links are under SFDIAs and only require extra current sensors. The bias reconstruction performance under initial errors and system noises is theoretically analysed and the reconstruction error is proved to be bounded regardless of the electrical parameters. To avoid deploying current sensors on all power lines, a cost-effective deployment strategy is presented to secure a spanning tree set of communication links that can guarantee the secondary control performance. Extensive validation studies are conducted in MATLAB/SIMULINK and hardware-in-the-loop (HIL) testbeds to validate the proposed method's effectiveness against single/multiple and continuous/discontinuous SFDIAs. △ Less

Submitted 9 July, 2024; originally announced July 2024.

Comments: Submitted to IEEE Transactions on Smart Grid

Preference-Guided Reinforcement Learning for Efficient Exploration

Authors: Guojian Wang, Faguo Wu, Xiao Zhang, Tianyuan Chen, Xuyang Chen, Lin Zhao

Abstract: In this paper, we investigate preference-based reinforcement learning (PbRL) that allows reinforcement learning (RL) agents to learn from human feedback. This is particularly valuable when defining a fine-grain reward function is not feasible. However, this approach is inefficient and impractical for promoting deep exploration in hard-exploration tasks with long horizons and sparse rewards. To tac… ▽ More In this paper, we investigate preference-based reinforcement learning (PbRL) that allows reinforcement learning (RL) agents to learn from human feedback. This is particularly valuable when defining a fine-grain reward function is not feasible. However, this approach is inefficient and impractical for promoting deep exploration in hard-exploration tasks with long horizons and sparse rewards. To tackle this issue, we introduce LOPE: Learning Online with trajectory Preference guidancE, an end-to-end preference-guided RL framework that enhances exploration efficiency in hard-exploration tasks. Our intuition is that LOPE directly adjusts the focus of online exploration by considering human feedback as guidance, avoiding learning a separate reward model from preferences. Specifically, LOPE includes a two-step sequential policy optimization process consisting of trust-region-based policy improvement and preference guidance steps. We reformulate preference guidance as a novel trajectory-wise state marginal matching problem that minimizes the maximum mean discrepancy distance between the preferred trajectories and the learned policy. Furthermore, we provide a theoretical analysis to characterize the performance improvement bound and evaluate the LOPE's effectiveness. When assessed in various challenging hard-exploration environments, LOPE outperforms several state-of-the-art methods regarding convergence rate and overall performance. The code used in this study is available at \url{https://github.com/buaawgj/LOPE}. △ Less

Submitted 8 July, 2024; originally announced July 2024.

Comments: 13 pages, 17 figures

Multi-Object Hallucination in Vision-Language Models

Authors: Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David F. Fouhey, Joyce Chai

Abstract: Large vision language models (LVLMs) often suffer from object hallucination, producing objects not present in the given images. While current benchmarks for object hallucination primarily concentrate on the presence of a single object class rather than individual entities, this work systematically investigates multi-object hallucination, examining how models misperceive (e.g., invent nonexistent o… ▽ More Large vision language models (LVLMs) often suffer from object hallucination, producing objects not present in the given images. While current benchmarks for object hallucination primarily concentrate on the presence of a single object class rather than individual entities, this work systematically investigates multi-object hallucination, examining how models misperceive (e.g., invent nonexistent objects or become distracted) when tasked with focusing on multiple objects simultaneously. We introduce Recognition-based Object Probing Evaluation (ROPE), an automated evaluation protocol that considers the distribution of object classes within a single image during testing and uses visual referring prompts to eliminate ambiguity. With comprehensive empirical studies and analysis of potential factors leading to multi-object hallucination, we found that (1) LVLMs suffer more hallucinations when focusing on multiple objects compared to a single object. (2) The tested object class distribution affects hallucination behaviors, indicating that LVLMs may follow shortcuts and spurious correlations.(3) Hallucinatory behaviors are influenced by data-specific factors, salience and frequency, and model intrinsic behaviors. We hope to enable LVLMs to recognize and reason about multiple objects that often occur in realistic visual scenes, provide insights, and quantify our progress towards mitigating the issues. △ Less

Submitted 8 July, 2024; originally announced July 2024.

Comments: Accepted to ALVR @ ACL 2024 | Project page: https://multi-object-hallucination.github.io/

A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion

Authors: Xiaoli Zhang, Liying Wang, Libo Zhao, Xiongfei Li, Siwei Ma

Abstract: Multi-modality image fusion aims at fusing specific-modality and shared-modality information from two source images. To tackle the problem of insufficient feature extraction and lack of semantic awareness for complex scenes, this paper focuses on how to model correlation-driven decomposing features and reason high-level graph representation by efficiently extracting complementary features and mult… ▽ More Multi-modality image fusion aims at fusing specific-modality and shared-modality information from two source images. To tackle the problem of insufficient feature extraction and lack of semantic awareness for complex scenes, this paper focuses on how to model correlation-driven decomposing features and reason high-level graph representation by efficiently extracting complementary features and multi-guided feature aggregation. We propose a three-branch encoder-decoder architecture along with corresponding fusion layers as the fusion strategy. The transformer with Multi-Dconv Transposed Attention and Local-enhanced Feed Forward network is used to extract shallow features after the depthwise convolution. In the three parallel branches encoder, Cross Attention and Invertible Block (CAI) enables to extract local features and preserve high-frequency texture details. Base feature extraction module (BFE) with residual connections can capture long-range dependency and enhance shared-modality expression capabilities. Graph Reasoning Module (GR) is introduced to reason high-level cross-modality relations and extract low-level details features as CAI's specific-modality complementary information simultaneously. Experiments demonstrate that our method has obtained competitive results compared with state-of-the-art methods in visible/infrared image fusion and medical image fusion tasks. Moreover, we surpass other fusion methods in terms of subsequent tasks, averagely scoring 9.78% mAP@.5 higher in object detection and 6.46% mIoU higher in semantic segmentation. △ Less

Submitted 11 June, 2024; originally announced July 2024.

A Survey of Controllable Learning: Methods and Applications in Information Retrieval

Authors: Chenglei Shen, Xiao Zhang, Teng Shi, Changshuo Zhang, Guofu Xie, Jun Xu

Abstract: Controllable learning (CL) emerges as a critical component in trustworthy machine learning, ensuring that learners meet predefined targets and can adaptively adjust without retraining according to the changes in those targets. We provide a formal definition of CL, and discuss its applications in information retrieval (IR) where information needs are often complex and dynamic. The survey categorize… ▽ More Controllable learning (CL) emerges as a critical component in trustworthy machine learning, ensuring that learners meet predefined targets and can adaptively adjust without retraining according to the changes in those targets. We provide a formal definition of CL, and discuss its applications in information retrieval (IR) where information needs are often complex and dynamic. The survey categorizes CL according to who controls (users or platforms), what is controllable (e.g., retrieval objectives, users' historical behaviors, controllable environmental adaptation), how control is implemented (e.g., rule-based method, Pareto optimization, Hypernetwork), and where to implement control (e.g.,pre-processing, in-processing, post-processing methods). Then, we identify challenges faced by CL across training, evaluation, task setting, and deployment in online environments. Additionally, we outline promising directions for CL in theoretical analysis, efficient computation, empowering large language models, application scenarios and evaluation frameworks in IR. △ Less

Submitted 4 July, 2024; originally announced July 2024.

Causality-driven Sequence Segmentation for Enhancing Multiphase Industrial Process Data Analysis and Soft Sensing

Authors: Yimeng He, Le Yao, Xinmin Zhang, Xiangyin Kong, Zhihuan Song

Abstract: The dynamic characteristics of multiphase industrial processes present significant challenges in the field of industrial big data modeling. Traditional soft sensing models frequently neglect the process dynamics and have difficulty in capturing transient phenomena like phase transitions. To address this issue, this article introduces a causality-driven sequence segmentation (CDSS) model. This mode… ▽ More The dynamic characteristics of multiphase industrial processes present significant challenges in the field of industrial big data modeling. Traditional soft sensing models frequently neglect the process dynamics and have difficulty in capturing transient phenomena like phase transitions. To address this issue, this article introduces a causality-driven sequence segmentation (CDSS) model. This model first identifies the local dynamic properties of the causal relationships between variables, which are also referred to as causal mechanisms. It then segments the sequence into different phases based on the sudden shifts in causal mechanisms that occur during phase transitions. Additionally, a novel metric, similarity distance, is designed to evaluate the temporal consistency of causal mechanisms, which includes both causal similarity distance and stable similarity distance. The discovered causal relationships in each phase are represented as a temporal causal graph (TCG). Furthermore, a soft sensing model called temporal-causal graph convolutional network (TC-GCN) is trained for each phase, by using the time-extended data and the adjacency matrix of TCG. The numerical examples are utilized to validate the proposed CDSS model, and the segmentation results demonstrate that CDSS has excellent performance on segmenting both stable and unstable multiphase series. Especially, it has higher accuracy in separating non-stationary time series compared to other methods. The effectiveness of the proposed CDSS model and the TC-GCN model is also verified through a penicillin fermentation process. Experimental results indicate that the breakpoints discovered by CDSS align well with the reaction mechanisms and TC-GCN significantly has excellent predictive accuracy. △ Less

Submitted 30 June, 2024; originally announced July 2024.

HiT-SR: Hierarchical Transformer for Efficient Image Super-Resolution

Authors: Xiang Zhang, Yulun Zhang, Fisher Yu

Abstract: Transformers have exhibited promising performance in computer vision tasks including image super-resolution (SR). However, popular transformer-based SR methods often employ window self-attention with quadratic computational complexity to window sizes, resulting in fixed small windows with limited receptive fields. In this paper, we present a general strategy to convert transformer-based SR network… ▽ More Transformers have exhibited promising performance in computer vision tasks including image super-resolution (SR). However, popular transformer-based SR methods often employ window self-attention with quadratic computational complexity to window sizes, resulting in fixed small windows with limited receptive fields. In this paper, we present a general strategy to convert transformer-based SR networks to hierarchical transformers (HiT-SR), boosting SR performance with multi-scale features while maintaining an efficient design. Specifically, we first replace the commonly used fixed small windows with expanding hierarchical windows to aggregate features at different scales and establish long-range dependencies. Considering the intensive computation required for large windows, we further design a spatial-channel correlation method with linear complexity to window sizes, efficiently gathering spatial and channel information from hierarchical windows. Extensive experiments verify the effectiveness and efficiency of our HiT-SR, and our improved versions of SwinIR-Light, SwinIR-NG, and SRFormer-Light yield state-of-the-art SR results with fewer parameters, FLOPs, and faster speeds ($\sim7\times$). △ Less

Submitted 8 July, 2024; originally announced July 2024.

Comments: ECCV 2024

Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports

Authors: Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zhengliang Liu, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang

Abstract: Medical images and radiology reports are crucial for diagnosing medical conditions, highlighting the importance of quantitative analysis for clinical decision-making. However, the diversity and cross-source heterogeneity of these data challenge the generalizability of current data-mining methods. Multimodal large language models (MLLMs) have recently transformed many domains, significantly affecti… ▽ More Medical images and radiology reports are crucial for diagnosing medical conditions, highlighting the importance of quantitative analysis for clinical decision-making. However, the diversity and cross-source heterogeneity of these data challenge the generalizability of current data-mining methods. Multimodal large language models (MLLMs) have recently transformed many domains, significantly affecting the medical field. Notably, Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models have epitomized a paradigm shift in Artificial General Intelligence (AGI) for computer vision, showcasing their potential in the biomedical domain. In this study, we evaluated the performance of the Gemini, GPT-4, and 4 popular large models for an exhaustive evaluation across 14 medical imaging datasets, including 5 medical imaging categories (dermatology, radiology, dentistry, ophthalmology, and endoscopy), and 3 radiology report datasets. The investigated tasks encompass disease classification, lesion segmentation, anatomical localization, disease diagnosis, report generation, and lesion detection. Our experimental results demonstrated that Gemini-series models excelled in report generation and lesion detection but faces challenges in disease classification and anatomical localization. Conversely, GPT-series models exhibited proficiency in lesion segmentation and anatomical localization but encountered difficulties in disease diagnosis and lesion detection. Additionally, both the Gemini series and GPT series contain models that have demonstrated commendable generation efficiency. While both models hold promise in reducing physician workload, alleviating pressure on limited healthcare resources, and fostering collaboration between clinical practitioners and artificial intelligence technologies, substantial enhancements and comprehensive validations remain imperative before clinical deployment. △ Less

Submitted 8 July, 2024; originally announced July 2024.

InverseCoder: Unleashing the Power of Instruction-Tuned Code LLMs with Inverse-Instruct

Authors: Yutong Wu, Di Huang, Wenxuan Shi, Wei Wang, Lingzhe Gao, Shihao Liu, Ziyuan Nan, Kaizhao Yuan, Rui Zhang, Xishan Zhang, Zidong Du, Qi Guo, Yewen Pu, Dawei Yin, Xing Hu, Yunji Chen

Abstract: Recent advancements in open-source code large language models (LLMs) have demonstrated remarkable coding abilities by fine-tuning on the data generated from powerful closed-source LLMs such as GPT-3.5 and GPT-4 for instruction tuning. This paper explores how to further improve an instruction-tuned code LLM by generating data from itself rather than querying closed-source LLMs. Our key observation… ▽ More Recent advancements in open-source code large language models (LLMs) have demonstrated remarkable coding abilities by fine-tuning on the data generated from powerful closed-source LLMs such as GPT-3.5 and GPT-4 for instruction tuning. This paper explores how to further improve an instruction-tuned code LLM by generating data from itself rather than querying closed-source LLMs. Our key observation is the misalignment between the translation of formal and informal languages: translating formal language (i.e., code) to informal language (i.e., natural language) is more straightforward than the reverse. Based on this observation, we propose INVERSE-INSTRUCT, which summarizes instructions from code snippets instead of the reverse. Specifically, given an instruction tuning corpus for code and the resulting instruction-tuned code LLM, we ask the code LLM to generate additional high-quality instructions for the original corpus through code summarization and self-evaluation. Then, we fine-tune the base LLM on the combination of the original corpus and the self-generated one, which yields a stronger instruction-tuned LLM. We present a series of code LLMs named InverseCoder, which surpasses the performance of the original code LLMs on a wide range of benchmarks, including Python text-to-code generation, multilingual coding, and data-science code generation. △ Less

Submitted 8 July, 2024; originally announced July 2024.

On the Power of Convolution Augmented Transformer

Authors: Mingchen Li, Xuechen Zhang, Yixiao Huang, Samet Oymak

Abstract: The transformer architecture has catalyzed revolutionary advances in language modeling. However, recent architectural recipes, such as state-space models, have bridged the performance gap. Motivated by this, we examine the benefits of Convolution-Augmented Transformer (CAT) for recall, copying, and length generalization tasks. CAT incorporates convolutional filters in the K/Q/V embeddings of an at… ▽ More The transformer architecture has catalyzed revolutionary advances in language modeling. However, recent architectural recipes, such as state-space models, have bridged the performance gap. Motivated by this, we examine the benefits of Convolution-Augmented Transformer (CAT) for recall, copying, and length generalization tasks. CAT incorporates convolutional filters in the K/Q/V embeddings of an attention layer. Through CAT, we show that the locality of the convolution synergizes with the global view of the attention. Unlike comparable architectures, such as Mamba or transformer, CAT can provably solve the associative recall (AR) and copying tasks using a single layer while also enjoying guaranteed length generalization. We also establish computational tradeoffs between convolution and attention by characterizing how convolution can mitigate the need for full attention by summarizing the context window and creating salient summary tokens to attend. Evaluations on real datasets corroborate our findings and demonstrate that CAT and its variations indeed enhance the language modeling performance. △ Less

Submitted 8 July, 2024; originally announced July 2024.

A modified Korteweg-de Vries equation soliton gas under the nonzero background

Authors: Xiaoen Zhang, Liming Ling

Abstract: In this paper, we consider a soliton gas of the focusing modified Korteweg-de Vries generated from the $N$-soliton solutions under the nonzero background. The spectral soliton density is chosen on the pure imaginary axis, excluding the branch cut $Σ_{c}=\left[-i, i\right]$. In the limit $N\to\infty$, we establish the Riemann-Hilbert problem of the soliton gas. Using the Deift-Zhou nonlinear steepe… ▽ More In this paper, we consider a soliton gas of the focusing modified Korteweg-de Vries generated from the $N$-soliton solutions under the nonzero background. The spectral soliton density is chosen on the pure imaginary axis, excluding the branch cut $Σ_{c}=\left[-i, i\right]$. In the limit $N\to\infty$, we establish the Riemann-Hilbert problem of the soliton gas. Using the Deift-Zhou nonlinear steepest-descent method, this soliton gas under the nonzero background will decay to a constant background as $x\to+\infty$, while its asymptotics as $x\to-\infty$ can be expressed with a Riemann-Theta function, attached to a Riemann surface with genus-two. We also analyze the large $t$ asymptotics over the entire spatial domain, which is divided into three distinct asymptotic regions depending on the ratio $ξ=\frac{x}{t}$. Using the similar method, we provide the leading-order asymptotic behaviors for these three regions and exhibit the dynamics of large $t$ asymptotics. △ Less

Submitted 7 July, 2024; originally announced July 2024.

Comments: 29pages,6 figures

MSC Class: 35Q55; 35Q51; 35Q15; 37K40; 37K15; 37K10

Advancing Prompt Recovery in NLP: A Deep Dive into the Integration of Gemma-2b-it and Phi2 Models

Authors: Jianlong Chen, Wei Xu, Zhicheng Ding, Jinxin Xu, Hao Yan, Xinyu Zhang

Abstract: Prompt recovery, a crucial task in natural language processing, entails the reconstruction of prompts or instructions that language models use to convert input text into a specific output. Although pivotal, the design and effectiveness of prompts represent a challenging and relatively untapped field within NLP research. This paper delves into an exhaustive investigation of prompt recovery methodol… ▽ More Prompt recovery, a crucial task in natural language processing, entails the reconstruction of prompts or instructions that language models use to convert input text into a specific output. Although pivotal, the design and effectiveness of prompts represent a challenging and relatively untapped field within NLP research. This paper delves into an exhaustive investigation of prompt recovery methodologies, employing a spectrum of pre-trained language models and strategies. Our study is a comparative analysis aimed at gauging the efficacy of various models on a benchmark dataset, with the goal of pinpointing the most proficient approach for prompt recovery. Through meticulous experimentation and detailed analysis, we elucidate the outstanding performance of the Gemma-2b-it + Phi2 model + Pretrain. This model surpasses its counterparts, showcasing its exceptional capability in accurately reconstructing prompts for text transformation tasks. Our findings offer a significant contribution to the existing knowledge on prompt recovery, shedding light on the intricacies of prompt design and offering insightful perspectives for future innovations in text rewriting and the broader field of natural language processing. △ Less

Submitted 6 July, 2024; originally announced July 2024.

HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning

Authors: Liyuan Wang, Jingyi Xie, Xingxing Zhang, Hang Su, Jun Zhu

Abstract: The deployment of pre-trained models (PTMs) has greatly advanced the field of continual learning (CL), enabling positive knowledge transfer and resilience to catastrophic forgetting. To sustain these advantages for sequentially arriving tasks, a promising direction involves keeping the pre-trained backbone frozen while employing parameter-efficient tuning (PET) techniques to instruct representatio… ▽ More The deployment of pre-trained models (PTMs) has greatly advanced the field of continual learning (CL), enabling positive knowledge transfer and resilience to catastrophic forgetting. To sustain these advantages for sequentially arriving tasks, a promising direction involves keeping the pre-trained backbone frozen while employing parameter-efficient tuning (PET) techniques to instruct representation learning. Despite the popularity of Prompt-based PET for CL, its empirical design often leads to sub-optimal performance in our evaluation of different PTMs and target tasks. To this end, we propose a unified framework for CL with PTMs and PET that provides both theoretical and empirical advancements. We first perform an in-depth theoretical analysis of the CL objective in a pre-training context, decomposing it into hierarchical components namely within-task prediction, task-identity inference and task-adaptive prediction. We then present Hierarchical Decomposition PET (HiDe-PET), an innovative approach that explicitly optimizes the decomposed objective through incorporating task-specific and task-shared knowledge via mainstream PET techniques along with efficient recovery of pre-trained representations. Leveraging this framework, we delve into the distinct impacts of implementation strategy, PET technique and PET architecture, as well as adaptive knowledge accumulation amidst pronounced distribution changes. Finally, across various CL scenarios, our approach demonstrates remarkably superior performance over a broad spectrum of recent strong baselines. △ Less

Submitted 6 July, 2024; originally announced July 2024.

Comments: This is a generalized version of our HiDe-Prompt (NeurIPS 2023, Spotlight)

Decoherence without einselection

Authors: Xiao Zhang

Abstract: Decoherence in a quantum measurement is typically explained as an interaction with the environment that destroys coherence between the system's eigenstates, a phenomenon known as environment-induced superselection (einselection). In this work, we demonstrate that einselection and the associated envariance are actually artifacts resulting from neglecting the non-equilibrium dynamics of the apparatu… ▽ More Decoherence in a quantum measurement is typically explained as an interaction with the environment that destroys coherence between the system's eigenstates, a phenomenon known as environment-induced superselection (einselection). In this work, we demonstrate that einselection and the associated envariance are actually artifacts resulting from neglecting the non-equilibrium dynamics of the apparatus. We propose a new formalism of operator dressing, which we call the stochastic matrix integral (SMI), inspired by recent developments in quantum gravity algebras. This approach naturally arises from a modified Page-Wootters (PW) formula and describes decoherence as an interacting non-equilibrium process. It leads to the reduction of the Hilbert space and the emergence of an intrinsic non-unitary process as well as Born's rule. These outcomes are achieved without relying on the assumptions of einselection and pointer basis. △ Less

Submitted 9 July, 2024; v1 submitted 6 July, 2024; originally announced July 2024.

Leveraging Data Mining, Active Learning, and Domain Adaptation in a Multi-Stage, Machine Learning-Driven Approach for the Efficient Discovery of Advanced Acidic Oxygen Evolution Electrocatalysts

Authors: Rui Ding, Jianguo Liu, Kang Hua, Xuebin Wang, Xiaoben Zhang, Minhua Shao, Yuxin Chen, Junhong Chen

Abstract: Developing advanced catalysts for acidic oxygen evolution reaction (OER) is crucial for sustainable hydrogen production. This study introduces a novel, multi-stage machine learning (ML) approach to streamline the discovery and optimization of complex multi-metallic catalysts. Our method integrates data mining, active learning, and domain adaptation throughout the materials discovery process. Unlik… ▽ More Developing advanced catalysts for acidic oxygen evolution reaction (OER) is crucial for sustainable hydrogen production. This study introduces a novel, multi-stage machine learning (ML) approach to streamline the discovery and optimization of complex multi-metallic catalysts. Our method integrates data mining, active learning, and domain adaptation throughout the materials discovery process. Unlike traditional trial-and-error methods, this approach systematically narrows the exploration space using domain knowledge with minimized reliance on subjective intuition. Then the active learning module efficiently refines element composition and synthesis conditions through iterative experimental feedback. The process culminated in the discovery of a promising Ru-Mn-Ca-Pr oxide catalyst. Our workflow also enhances theoretical simulations with domain adaptation strategy, providing deeper mechanistic insights aligned with experimental findings. By leveraging diverse data sources and multiple ML strategies, we establish an efficient pathway for electrocatalyst discovery and optimization. This comprehensive, data-driven approach represents a paradigm shift and potentially new benchmark in electrocatalysts research. △ Less

Submitted 5 July, 2024; originally announced July 2024.

Comments: 95 pages (main text 37 pages; supplementary materials 58 pages); 38 figures (main text 6 figures; supplementary materials 32 figures)

A Unified Learn-to-Distort-Data Framework for Privacy-Utility Trade-off in Trustworthy Federated Learning

Authors: Xiaojin Zhang, Mingcong Xu, Wei Chen

Abstract: In this paper, we first give an introduction to the theoretical basis of the privacy-utility equilibrium in federated learning based on Bayesian privacy definitions and total variation distance privacy definitions. We then present the \textit{Learn-to-Distort-Data} framework, which provides a principled approach to navigate the privacy-utility equilibrium by explicitly modeling the distortion intr… ▽ More In this paper, we first give an introduction to the theoretical basis of the privacy-utility equilibrium in federated learning based on Bayesian privacy definitions and total variation distance privacy definitions. We then present the \textit{Learn-to-Distort-Data} framework, which provides a principled approach to navigate the privacy-utility equilibrium by explicitly modeling the distortion introduced by the privacy-preserving mechanism as a learnable variable and optimizing it jointly with the model parameters. We demonstrate the applicability of our framework to a variety of privacy-preserving mechanisms on the basis of data distortion and highlight its connections to related areas such as adversarial training, input robustness, and unlearnable examples. These connections enable leveraging techniques from these areas to design effective algorithms for privacy-utility equilibrium in federated learning under the \textit{Learn-to-Distort-Data} framework. △ Less

Submitted 9 July, 2024; v1 submitted 5 July, 2024; originally announced July 2024.

Poisson stability of solutions for stochastic evolution equations driven by fractional Brownian motion

Authors: Xinze Zhang, Xue Yang

Abstract: In this paper, we study the problem of Poisson stability of solutions for stochastic semi-linear evolution equation driven by fractional Brownian motion $$\mathrm{d} X(t)= \left( AX(t) + f(t,X(t)) \right) \mathrm{d}t + g\left(t,X(t)\right)\mathrm{d}B^H_{Q}(t),$$ where $A$ is an exponentially stable linear operator acting on a separable Hilbert space $\mathbb{H}$, coefficients $f$ and $g$ are Poiss… ▽ More In this paper, we study the problem of Poisson stability of solutions for stochastic semi-linear evolution equation driven by fractional Brownian motion $$\mathrm{d} X(t)= \left( AX(t) + f(t,X(t)) \right) \mathrm{d}t + g\left(t,X(t)\right)\mathrm{d}B^H_{Q}(t),$$ where $A$ is an exponentially stable linear operator acting on a separable Hilbert space $\mathbb{H}$, coefficients $f$ and $g$ are Poisson stable in time, and $B^H_Q (t)$ is a $Q$-cylindrical fBm with Hurst index $H$. First, we establish the existence and uniqueness of the solution for this equation. Then, we prove that under the condition where the functions $f$ and $g$ are sufficiently "small", the equation admits a solution that exhibits the same character of recurrence as $f$ and $g$. The discussion is further extended to the asymptotic stability of these Poisson stable solutions. Finally, we include an example to validate our results. △ Less

Submitted 5 July, 2024; originally announced July 2024.

Comments: 25 pages, 1 figures. arXiv admin note: substantial text overlap with arXiv:1702.02718 by other authors

MSC Class: 60G22; 34C25; 34C27; 37B20; 60H10; 34D20

Onsager-Machlup functional for stochastic differential equations with time-varying noise

Authors: Xinze Zhang, Yong Li

Abstract: This paper is devoted to studying the Onsager-Machlup functional for stochastic differential equations with time-varying noise of the α-Hölder, 0<α<1/4, dXt =f(t,Xt)dt+g(t)dWt. Our study focuses on scenarios where the diffusion coefficient g(t) exhibits temporal variability, starkly contrasting the conventional assumption of a constant diffusion coefficient in the existing literature. This var… ▽ More This paper is devoted to studying the Onsager-Machlup functional for stochastic differential equations with time-varying noise of the α-Hölder, 0<α<1/4, dXt =f(t,Xt)dt+g(t)dWt. Our study focuses on scenarios where the diffusion coefficient g(t) exhibits temporal variability, starkly contrasting the conventional assumption of a constant diffusion coefficient in the existing literature. This variance brings some complexity to the analysis. Through this investigation, we derive the Onsager-Machlup functional, which acts as the Lagrangian for mapping the most probable transition path between metastable states in stochastic processes affected by time-varying noise. This is done by introducing new measurable norms and applying an appropriate version of the Girsanov transformation. To illustrate our theoretical advancements, we provide numerical simulations, including cases of a one-dimensional SDE and a fast-slow SDE system, which demonstrate the application to multiscale stochastic volatility models, thereby highlighting the significant impact of time-varying diffusion coefficients. △ Less

Submitted 5 July, 2024; originally announced July 2024.

Comments: 17 pages, 4 figures

MSC Class: 82C35; 60H10; 37H10

PA-LOCO: Learning Perturbation-Adaptive Locomotion for Quadruped Robots

Authors: Zhiyuan Xiao, Xinyu Zhang, Xiang Zhou, Qingrui Zhang

Abstract: Numerous locomotion controllers have been designed based on Reinforcement Learning (RL) to facilitate blind quadrupedal locomotion traversing challenging terrains. Nevertheless, locomotion control is still a challenging task for quadruped robots traversing diverse terrains amidst unforeseen disturbances. Recently, privileged learning has been employed to learn reliable and robust quadrupedal locom… ▽ More Numerous locomotion controllers have been designed based on Reinforcement Learning (RL) to facilitate blind quadrupedal locomotion traversing challenging terrains. Nevertheless, locomotion control is still a challenging task for quadruped robots traversing diverse terrains amidst unforeseen disturbances. Recently, privileged learning has been employed to learn reliable and robust quadrupedal locomotion over various terrains based on a teacher-student architecture. However, its one-encoder structure is not adequate in addressing external force perturbations. The student policy would experience inevitable performance degradation due to the feature embedding discrepancy between the feature encoder of the teacher policy and the one of the student policy. Hence, this paper presents a privileged learning framework with multiple feature encoders and a residual policy network for robust and reliable quadruped locomotion subject to various external perturbations. The multi-encoder structure can decouple latent features from different privileged information, ultimately leading to enhanced performance of the learned policy in terms of robustness, stability, and reliability. The efficiency of the proposed feature encoding module is analyzed in depth using extensive simulation data. The introduction of the residual policy network helps mitigate the performance degradation experienced by the student policy that attempts to clone the behaviors of a teacher policy. The proposed framework is evaluated on a Unitree GO1 robot, showcasing its performance enhancement over the state-of-the-art privileged learning algorithm through extensive experiments conducted on diverse terrains. Ablation studies are conducted to illustrate the efficiency of the residual policy network. △ Less

Submitted 4 July, 2024; originally announced July 2024.

Comments: 8 pages, Accepted by IROS 2024

Stephanie: Step-by-Step Dialogues for Mimicking Human Interactions in Social Conversations

Authors: Hao Yang, Hongyuan Lu, Xinhua Zeng, Yang Liu, Xiang Zhang, Haoran Yang, Yumeng Zhang, Shan Huang, Yiran Wei, Wai Lam

Abstract: In the rapidly evolving field of natural language processing, dialogue systems primarily employ a single-step dialogue paradigm. Although this paradigm is efficient, it lacks the depth and fluidity of human interactions and does not appear natural. We introduce a novel \textbf{Step}-by-Step Dialogue Paradigm (Stephanie), designed to mimic the ongoing dynamic nature of human conversations. By emplo… ▽ More In the rapidly evolving field of natural language processing, dialogue systems primarily employ a single-step dialogue paradigm. Although this paradigm is efficient, it lacks the depth and fluidity of human interactions and does not appear natural. We introduce a novel \textbf{Step}-by-Step Dialogue Paradigm (Stephanie), designed to mimic the ongoing dynamic nature of human conversations. By employing a dual learning strategy and a further-split post-editing method, we generated and utilized a high-quality step-by-step dialogue dataset to fine-tune existing large language models, enabling them to perform step-by-step dialogues. We thoroughly present Stephanie. Tailored automatic and human evaluations are conducted to assess its effectiveness compared to the traditional single-step dialogue paradigm. We will release code, Stephanie datasets, and Stephanie LLMs to facilitate the future of chatbot eras. △ Less

Submitted 12 July, 2024; v1 submitted 4 July, 2024; originally announced July 2024.

Benchmark on Drug Target Interaction Modeling from a Structure Perspective

Authors: Xinnan Zhang, Jialin Wu, Junyi Xie, Tianlong Chen, Kaixiong Zhou

Abstract: The prediction modeling of drug-target interactions is crucial to drug discovery and design, which has seen rapid advancements owing to deep learning technologies. Recently developed methods, such as those based on graph neural networks (GNNs) and Transformers, demonstrate exceptional performance across various datasets by effectively extracting structural information. However, the benchmarking of… ▽ More The prediction modeling of drug-target interactions is crucial to drug discovery and design, which has seen rapid advancements owing to deep learning technologies. Recently developed methods, such as those based on graph neural networks (GNNs) and Transformers, demonstrate exceptional performance across various datasets by effectively extracting structural information. However, the benchmarking of these novel methods often varies significantly in terms of hyperparameter settings and datasets, which limits algorithmic progress. In view of these, we conduct a comprehensive survey and benchmark for drug-target interaction modeling from a structure perspective, via integrating tens of explicit (i.e., GNN-based) and implicit (i.e., Transformer-based) structure learning algorithms. To this end, we first unify the hyperparameter setting within each class of structure learning methods. Moreover, we conduct a macroscopical comparison between these two classes of encoding strategies as well as the different featurization techniques that inform molecules' chemical and physical properties. We then carry out the microscopical comparison between all the integrated models across the six datasets, via comprehensively benchmarking their effectiveness and efficiency. Remarkably, the summarized insights from the benchmark studies lead to the design of model combos. We demonstrate that our combos can achieve new state-of-the-art performance on various datasets associated with cost-effective memory and computation. Our code is available at \hyperlink{https://github.com/justinwjl/GTB-DTI/tree/main}{https://github.com/justinwjl/GTB-DTI/tree/main}. △ Less

Submitted 4 July, 2024; originally announced July 2024.

Comments: Submitted to NIPS 2024 Dataset and Benchmark

MineNetCD: A Benchmark for Global Mining Change Detection on Remote Sensing Imagery

Authors: Weikang Yu, Xiaokang Zhang, Xiao Xiang Zhu, Richard Gloaguen, Pedram Ghamisi

Abstract: Monitoring changes triggered by mining activities is crucial for industrial controlling, environmental management and regulatory compliance, yet it poses significant challenges due to the vast and often remote locations of mining sites. Remote sensing technologies have increasingly become indispensable to detect and analyze these changes over time. We thus introduce MineNetCD, a comprehensive benc… ▽ More Monitoring changes triggered by mining activities is crucial for industrial controlling, environmental management and regulatory compliance, yet it poses significant challenges due to the vast and often remote locations of mining sites. Remote sensing technologies have increasingly become indispensable to detect and analyze these changes over time. We thus introduce MineNetCD, a comprehensive benchmark designed for global mining change detection using remote sensing imagery. The benchmark comprises three key contributions. First, we establish a global mining change detection dataset featuring more than 70k paired patches of bi-temporal high-resolution remote sensing images and pixel-level annotations from 100 mining sites worldwide. Second, we develop a novel baseline model based on a change-aware Fast Fourier Transform (ChangeFFT) module, which enhances various backbones by leveraging essential spectrum components within features in the frequency domain and capturing the channel-wise correlation of bi-temporal feature differences to learn change-aware representations. Third, we construct a unified change detection (UCD) framework that integrates over 13 advanced change detection models. This framework is designed for streamlined and efficient processing, utilizing the cloud platform hosted by HuggingFace. Extensive experiments have been conducted to demonstrate the superiority of the proposed baseline model compared with 12 state-of-the-art change detection approaches. Empirical studies on modularized backbones comprehensively confirm the efficacy of different representation learners on change detection. This contribution represents significant advancements in the field of remote sensing and change detection, providing a robust resource for future research and applications in global mining monitoring. Dataset and Codes are available via the link. △ Less

Submitted 4 July, 2024; originally announced July 2024.

Protecting Deep Learning Model Copyrights with Adversarial Example-Free Reuse Detection

Authors: Xiaokun Luan, Xiyue Zhang, Jingyi Wang, Meng Sun

Abstract: Model reuse techniques can reduce the resource requirements for training high-performance deep neural networks (DNNs) by leveraging existing models. However, unauthorized reuse and replication of DNNs can lead to copyright infringement and economic loss to the model owner. This underscores the need to analyze the reuse relation between DNNs and develop copyright protection techniques to safeguard… ▽ More Model reuse techniques can reduce the resource requirements for training high-performance deep neural networks (DNNs) by leveraging existing models. However, unauthorized reuse and replication of DNNs can lead to copyright infringement and economic loss to the model owner. This underscores the need to analyze the reuse relation between DNNs and develop copyright protection techniques to safeguard intellectual property rights. Existing white-box testing-based approaches cannot address the common heterogeneous reuse case where the model architecture is changed, and DNN fingerprinting approaches heavily rely on generating adversarial examples with good transferability, which is known to be challenging in the black-box setting. To bridge the gap, we propose NFARD, a Neuron Functionality Analysis-based Reuse Detector, which only requires normal test samples to detect reuse relations by measuring the models' differences on a newly proposed model characterization, i.e., neuron functionality (NF). A set of NF-based distance metrics is designed to make NFARD applicable to both white-box and black-box settings. Moreover, we devise a linear transformation method to handle heterogeneous reuse cases by constructing the optimal projection matrix for dimension consistency, significantly extending the application scope of NFARD. To the best of our knowledge, this is the first adversarial example-free method that exploits neuron functionality for DNN copyright protection. As a side contribution, we constructed a reuse detection benchmark named Reuse Zoo that covers various practical reuse techniques and popular datasets. Extensive evaluations on this comprehensive benchmark show that NFARD achieves F1 scores of 0.984 and 1.0 for detecting reuse relationships in black-box and white-box settings, respectively, while generating test suites 2 ~ 99 times faster than previous methods. △ Less

Submitted 4 July, 2024; originally announced July 2024.

Comments: 12 pages, 5 figures

Neural Probabilistic Logic Learning for Knowledge Graph Reasoning

Authors: Fengsong Sun, Jinyu Wang, Zhiqing Wei, Xianchao Zhang

Abstract: Knowledge graph (KG) reasoning is a task that aims to predict unknown facts based on known factual samples. Reasoning methods can be divided into two categories: rule-based methods and KG-embedding based methods. The former possesses precise reasoning capabilities but finds it challenging to reason efficiently over large-scale knowledge graphs. While gaining the ability to reason over large-scale… ▽ More Knowledge graph (KG) reasoning is a task that aims to predict unknown facts based on known factual samples. Reasoning methods can be divided into two categories: rule-based methods and KG-embedding based methods. The former possesses precise reasoning capabilities but finds it challenging to reason efficiently over large-scale knowledge graphs. While gaining the ability to reason over large-scale knowledge graphs, the latter sacrifices reasoning accuracy. This paper aims to design a reasoning framework called Neural Probabilistic Logic Learning(NPLL) that achieves accurate reasoning on knowledge graphs. Our approach introduces a scoring module that effectively enhances the expressive power of embedding networks, striking a balance between model simplicity and reasoning capabilities. We improve the interpretability of the model by incorporating a Markov Logic Network based on variational inference. We empirically evaluate our approach on several benchmark datasets, and the experimental results validate that our method substantially enhances the accuracy and quality of the reasoning results. △ Less

Submitted 4 July, 2024; originally announced July 2024.