Showing 1–50 of 453 results for author: Cheng, G

MSTF: Multiscale Transformer for Incomplete Trajectory Prediction

Authors: Zhanwen Liu, Chao Li, Nan Yang, Yang Wang, Jiaqi Ma, Guangliang Cheng, Xiangmo Zhao

Abstract: Motion forecasting plays a pivotal role in autonomous driving systems, enabling vehicles to execute collision warnings and rational local-path planning based on predictions of the surrounding vehicles. However, prevalent methods often assume complete observed trajectories, neglecting the potential impact of missing values induced by object occlusion, scope limitation, and sensor failures. Such ove… ▽ More Motion forecasting plays a pivotal role in autonomous driving systems, enabling vehicles to execute collision warnings and rational local-path planning based on predictions of the surrounding vehicles. However, prevalent methods often assume complete observed trajectories, neglecting the potential impact of missing values induced by object occlusion, scope limitation, and sensor failures. Such oversights inevitably compromise the accuracy of trajectory predictions. To tackle this challenge, we propose an end-to-end framework, termed Multiscale Transformer (MSTF), meticulously crafted for incomplete trajectory prediction. MSTF integrates a Multiscale Attention Head (MAH) and an Information Increment-based Pattern Adaptive (IIPA) module. Specifically, the MAH component concurrently captures multiscale motion representation of trajectory sequence from various temporal granularities, utilizing a multi-head attention mechanism. This approach facilitates the modeling of global dependencies in motion across different scales, thereby mitigating the adverse effects of missing values. Additionally, the IIPA module adaptively extracts continuity representation of motion across time steps by analyzing missing patterns in the data. The continuity representation delineates motion trend at a higher level, guiding MSTF to generate predictions consistent with motion continuity. We evaluate our proposed MSTF model using two large-scale real-world datasets. Experimental results demonstrate that MSTF surpasses state-of-the-art (SOTA) models in the task of incomplete trajectory prediction, showcasing its efficacy in addressing the challenges posed by missing values in motion forecasting for autonomous driving systems. △ Less

Submitted 8 July, 2024; originally announced July 2024.

EXCGEC: A Benchmark of Edit-wise Explainable Chinese Grammatical Error Correction

Authors: Jingheng Ye, Shang Qin, Yinghui Li, Xuxin Cheng, Libo Qin, Hai-Tao Zheng, Peng Xing, Zishan Xu, Guo Cheng, Zhao Wei

Abstract: Existing studies explore the explainability of Grammatical Error Correction (GEC) in a limited scenario, where they ignore the interaction between corrections and explanations. To bridge the gap, this paper introduces the task of EXplainable GEC (EXGEC), which focuses on the integral role of both correction and explanation tasks. To facilitate the task, we propose EXCGEC, a tailored benchmark for… ▽ More Existing studies explore the explainability of Grammatical Error Correction (GEC) in a limited scenario, where they ignore the interaction between corrections and explanations. To bridge the gap, this paper introduces the task of EXplainable GEC (EXGEC), which focuses on the integral role of both correction and explanation tasks. To facilitate the task, we propose EXCGEC, a tailored benchmark for Chinese EXGEC consisting of 8,216 explanation-augmented samples featuring the design of hybrid edit-wise explanations. We benchmark several series of LLMs in multiple settings, covering post-explaining and pre-explaining. To promote the development of the task, we introduce a comprehensive suite of automatic metrics and conduct human evaluation experiments to demonstrate the human consistency of the automatic metrics for free-text explanations. All the codes and data will be released after the review. △ Less

Submitted 30 June, 2024; originally announced July 2024.

Comments: 22 pages, 10 tables, 9 figures. Under review

Networked ISAC for Low-Altitude Economy: Coordinated Transmit Beamforming and UAV Trajectory Design

Authors: Gaoyuan Cheng, Xianxin Song, Zhonghao Lyu, Jie Xu

Abstract: This paper exploits the networked integrated sensing and communications (ISAC) to support low-altitude economy (LAE), in which a set of networked ground base stations (GBSs) cooperatively transmit joint information and sensing signals to communicate with multiple authorized unmanned aerial vehicles (UAVs) and concurrently detect unauthorized objects over the interested region in the three-dimensio… ▽ More This paper exploits the networked integrated sensing and communications (ISAC) to support low-altitude economy (LAE), in which a set of networked ground base stations (GBSs) cooperatively transmit joint information and sensing signals to communicate with multiple authorized unmanned aerial vehicles (UAVs) and concurrently detect unauthorized objects over the interested region in the three-dimensional (3D) space. We assume that each GBS is equipped with uniform linear array (ULA) antennas, which are deployed either horizontally or vertically to the ground. We also consider two types of UAV receivers, which have and do not have the capability of canceling the interference caused by dedicated sensing signals, respectively. Under each setup, we jointly design the coordinated transmit beamforming at multiple GBSs together with the authorized UAVs' trajectory control and their GBS associations, for enhancing the authorized UAVs' communication performance while ensuring the sensing requirements. In particular, we aim to maximize the average sum rate of authorized UAVs over a given flight period, subject to the minimum illumination power constraints toward the interested 3D sensing region, the maximum transmit power constraints at individual GBSs, and the flight constraints of UAVs. These problems are highly non-convex and challenging to solve, due to the involvement of binary UAV-GBS association variables as well as the coupling of beamforming and trajectory variables. To solve these non-convex problems, we propose efficient algorithms by using the techniques of alternating optimization, successive convex approximation, and semi-definite relaxation. Numerical results show that the proposed joint coordinated transmit beamforming and UAV trajectory designs efficiently balance the sensing-communication performance tradeoffs and significantly outperform various benchmarks. △ Less

Submitted 19 June, 2024; originally announced June 2024.

Comments: arXiv admin note: text overlap with arXiv:2405.07568

TimeAutoDiff: Combining Autoencoder and Diffusion model for time series tabular data synthesizing

Authors: Namjoon Suh, Yuning Yang, Din-Yin Hsieh, Qitong Luan, Shirong Xu, Shixiang Zhu, Guang Cheng

Abstract: In this paper, we leverage the power of latent diffusion models to generate synthetic time series tabular data. Along with the temporal and feature correlations, the heterogeneous nature of the feature in the table has been one of the main obstacles in time series tabular data modeling. We tackle this problem by combining the ideas of the variational auto-encoder (VAE) and the denoising diffusion… ▽ More In this paper, we leverage the power of latent diffusion models to generate synthetic time series tabular data. Along with the temporal and feature correlations, the heterogeneous nature of the feature in the table has been one of the main obstacles in time series tabular data modeling. We tackle this problem by combining the ideas of the variational auto-encoder (VAE) and the denoising diffusion probabilistic model (DDPM). Our model named as \texttt{TimeAutoDiff} has several key advantages including (1) Generality: the ability to handle the broad spectrum of time series tabular data from single to multi-sequence datasets; (2) Good fidelity and utility guarantees: numerical experiments on six publicly available datasets demonstrating significant improvements over state-of-the-art models in generating time series tabular data, across four metrics measuring fidelity and utility; (3) Fast sampling speed: entire time series data generation as opposed to the sequential data sampling schemes implemented in the existing diffusion-based models, eventually leading to significant improvements in sampling speed, (4) Entity conditional generation: the first implementation of conditional generation of multi-sequence time series tabular data with heterogenous features in the literature, enabling scenario exploration across multiple scientific and engineering domains. Codes are in preparation for release to the public, but available upon request. △ Less

Submitted 23 June, 2024; originally announced June 2024.

Advancing Retail Data Science: Comprehensive Evaluation of Synthetic Data

Authors: Yu Xia, Chi-Hua Wang, Joshua Mabry, Guang Cheng

Abstract: The evaluation of synthetic data generation is crucial, especially in the retail sector where data accuracy is paramount. This paper introduces a comprehensive framework for assessing synthetic retail data, focusing on fidelity, utility, and privacy. Our approach differentiates between continuous and discrete data attributes, providing precise evaluation criteria. Fidelity is measured through stab… ▽ More The evaluation of synthetic data generation is crucial, especially in the retail sector where data accuracy is paramount. This paper introduces a comprehensive framework for assessing synthetic retail data, focusing on fidelity, utility, and privacy. Our approach differentiates between continuous and discrete data attributes, providing precise evaluation criteria. Fidelity is measured through stability and generalizability. Stability ensures synthetic data accurately replicates known data distributions, while generalizability confirms its robustness in novel scenarios. Utility is demonstrated through the synthetic data's effectiveness in critical retail tasks such as demand forecasting and dynamic pricing, proving its value in predictive analytics and strategic planning. Privacy is safeguarded using Differential Privacy, ensuring synthetic data maintains a perfect balance between resembling training and holdout datasets without compromising security. Our findings validate that this framework provides reliable and scalable evaluation for synthetic retail data. It ensures high fidelity, utility, and privacy, making it an essential tool for advancing retail data science. This framework meets the evolving needs of the retail industry with precision and confidence, paving the way for future advancements in synthetic data methodologies. △ Less

Submitted 18 June, 2024; originally announced June 2024.

Data Plagiarism Index: Characterizing the Privacy Risk of Data-Copying in Tabular Generative Models

Authors: Joshua Ward, Chi-Hua Wang, Guang Cheng

Abstract: The promise of tabular generative models is to produce realistic synthetic data that can be shared and safely used without dangerous leakage of information from the training set. In evaluating these models, a variety of methods have been proposed to measure the tendency to copy data from the training dataset when generating a sample. However, these methods suffer from either not considering data-c… ▽ More The promise of tabular generative models is to produce realistic synthetic data that can be shared and safely used without dangerous leakage of information from the training set. In evaluating these models, a variety of methods have been proposed to measure the tendency to copy data from the training dataset when generating a sample. However, these methods suffer from either not considering data-copying from a privacy threat perspective, not being motivated by recent results in the data-copying literature or being difficult to make compatible with the high dimensional, mixed type nature of tabular data. This paper proposes a new similarity metric and Membership Inference Attack called Data Plagiarism Index (DPI) for tabular data. We show that DPI evaluates a new intuitive definition of data-copying and characterizes the corresponding privacy risk. We show that the data-copying identified by DPI poses both privacy and fairness threats to common, high performing architectures; underscoring the necessity for more sophisticated generative modeling techniques to mitigate this issue. △ Less

Submitted 18 June, 2024; originally announced June 2024.

SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answering

Authors: Zhe Yang, Wenrui Li, Guanghui Cheng

Abstract: The Audio-Visual Question Answering (AVQA) task holds significant potential for applications. Compared to traditional unimodal approaches, the multi-modal input of AVQA makes feature extraction and fusion processes more challenging. Euclidean space is difficult to effectively represent multi-dimensional relationships of data. Especially when extracting and processing data with a tree structure or… ▽ More The Audio-Visual Question Answering (AVQA) task holds significant potential for applications. Compared to traditional unimodal approaches, the multi-modal input of AVQA makes feature extraction and fusion processes more challenging. Euclidean space is difficult to effectively represent multi-dimensional relationships of data. Especially when extracting and processing data with a tree structure or hierarchical structure, Euclidean space is not suitable as an embedding space. Additionally, the self-attention mechanism in Transformers is effective in capturing the dynamic relationships between elements in a sequence. However, the self-attention mechanism's limitations in window modeling and quadratic computational complexity reduce its effectiveness in modeling long sequences. To address these limitations, we propose SHMamba: Structured Hyperbolic State Space Model to integrate the advantages of hyperbolic geometry and state space models. Specifically, SHMamba leverages the intrinsic properties of hyperbolic space to represent hierarchical structures and complex relationships in audio-visual data. Meanwhile, the state space model captures dynamic changes over time by globally modeling the entire sequence. Furthermore, we introduce an adaptive curvature hyperbolic alignment module and a cross fusion block to enhance the understanding of hierarchical structures and the dynamic exchange of cross-modal information, respectively. Extensive experiments demonstrate that SHMamba outperforms previous methods with fewer parameters and computational costs. Our learnable parameters are reduced by 78.12\%, while the average performance improves by 2.53\%. Experiments show that our method demonstrates superiority among all current major methods and is more suitable for practical application scenarios. △ Less

Submitted 20 June, 2024; v1 submitted 14 June, 2024; originally announced June 2024.

CTSyn: A Foundational Model for Cross Tabular Data Generation

Authors: Xiaofeng Lin, Chenheng Xu, Matthew Yang, Guang Cheng

Abstract: Generative Foundation Models (GFMs) have produced synthetic data with remarkable quality in modalities such as images and text. However, applying GFMs to tabular data poses significant challenges due to the inherent heterogeneity of table features. Existing cross-table learning frameworks are hindered by the absence of both a generative model backbone and a decoding mechanism for heterogeneous fea… ▽ More Generative Foundation Models (GFMs) have produced synthetic data with remarkable quality in modalities such as images and text. However, applying GFMs to tabular data poses significant challenges due to the inherent heterogeneity of table features. Existing cross-table learning frameworks are hindered by the absence of both a generative model backbone and a decoding mechanism for heterogeneous feature values. To overcome these limitations, we introduce the Cross-Table Synthesizer (CTSyn), a diffusion-based foundational model tailored for tabular data generation. CTSyn introduces three major components: an aggregator that consolidates heterogeneous tables into a unified latent space; a conditional latent diffusion model for sampling from this space; and type-specific decoders that reconstruct values of varied data types from sampled latent vectors. Extensive testing on real-world datasets reveals that CTSyn not only significantly outperforms existing table synthesizers in utility and diversity, but also uniquely enhances performances of downstream machine learning beyond what is achievable with real data, thus establishing a new paradigm for synthetic data generation. △ Less

Submitted 7 June, 2024; originally announced June 2024.

Dynamic Online Recommendation for Two-Sided Market with Bayesian Incentive Compatibility

Authors: Yuantong Li, Guang Cheng, Xiaowu Dai

Abstract: Recommender systems play a crucial role in internet economies by connecting users with relevant products or services. However, designing effective recommender systems faces two key challenges: (1) the exploration-exploitation tradeoff in balancing new product exploration against exploiting known preferences, and (2) dynamic incentive compatibility in accounting for users' self-interested behaviors… ▽ More Recommender systems play a crucial role in internet economies by connecting users with relevant products or services. However, designing effective recommender systems faces two key challenges: (1) the exploration-exploitation tradeoff in balancing new product exploration against exploiting known preferences, and (2) dynamic incentive compatibility in accounting for users' self-interested behaviors and heterogeneous preferences. This paper formalizes these challenges into a Dynamic Bayesian Incentive-Compatible Recommendation Protocol (DBICRP). To address the DBICRP, we propose a two-stage algorithm (RCB) that integrates incentivized exploration with an efficient offline learning component for exploitation. In the first stage, our algorithm explores available products while maintaining dynamic incentive compatibility to determine sufficient sample sizes. The second stage employs inverse proportional gap sampling integrated with an arbitrary machine learning method to ensure sublinear regret. Theoretically, we prove that RCB achieves $O(\sqrt{KdT})$ regret and satisfies Bayesian incentive compatibility (BIC) under a Gaussian prior assumption. Empirically, we validate RCB's strong incentive gain, sublinear regret, and robustness through simulations and a real-world application on personalized warfarin dosing. Our work provides a principled approach for incentive-aware recommendation in online preference learning settings. △ Less

Submitted 4 June, 2024; originally announced June 2024.

BACON: Bayesian Optimal Condensation Framework for Dataset Distillation

Authors: Zheng Zhou, Hongbo Zhao, Guangliang Cheng, Xiangtai Li, Shuchang Lyu, Wenquan Feng, Qi Zhao

Abstract: Dataset Distillation (DD) aims to distill knowledge from extensive datasets into more compact ones while preserving performance on the test set, thereby reducing storage costs and training expenses. However, existing methods often suffer from computational intensity, particularly exhibiting suboptimal performance with large dataset sizes due to the lack of a robust theoretical framework for analyz… ▽ More Dataset Distillation (DD) aims to distill knowledge from extensive datasets into more compact ones while preserving performance on the test set, thereby reducing storage costs and training expenses. However, existing methods often suffer from computational intensity, particularly exhibiting suboptimal performance with large dataset sizes due to the lack of a robust theoretical framework for analyzing the DD problem. To address these challenges, we propose the BAyesian optimal CONdensation framework (BACON), which is the first work to introduce the Bayesian theoretical framework to the literature of DD. This framework provides theoretical support for enhancing the performance of DD. Furthermore, BACON formulates the DD problem as the minimization of the expected risk function in joint probability distributions using the Bayesian framework. Additionally, by analyzing the expected risk function for optimal condensation, we derive a numerically feasible lower bound based on specific assumptions, providing an approximate solution for BACON. We validate BACON across several datasets, demonstrating its superior performance compared to existing state-of-the-art methods. For instance, under the IPC-10 setting, BACON achieves a 3.46% accuracy gain over the IDM method on the CIFAR-10 dataset and a 3.10% gain on the TinyImageNet dataset. Our extensive experiments confirm the effectiveness of BACON and its seamless integration with existing methods, thereby enhancing their performance for the DD task. Code and distilled datasets are available at BACON. △ Less

Submitted 3 June, 2024; originally announced June 2024.

Comments: 22 pages, 10 figures

Auto-selected Knowledge Adapters for Lifelong Person Re-identification

Authors: Xuelin Qian, Ruiqi Wu, Gong Cheng, Junwei Han

Abstract: Lifelong Person Re-Identification (LReID) extends traditional ReID by requiring systems to continually learn from non-overlapping datasets across different times and locations, adapting to new identities while preserving knowledge of previous ones. Existing approaches, either rehearsal-free or rehearsal-based, still suffer from the problem of catastrophic forgetting since they try to cram diverse… ▽ More Lifelong Person Re-Identification (LReID) extends traditional ReID by requiring systems to continually learn from non-overlapping datasets across different times and locations, adapting to new identities while preserving knowledge of previous ones. Existing approaches, either rehearsal-free or rehearsal-based, still suffer from the problem of catastrophic forgetting since they try to cram diverse knowledge into one fixed model. To overcome this limitation, we introduce a novel framework AdalReID, that adopts knowledge adapters and a parameter-free auto-selection mechanism for lifelong learning. Concretely, we incrementally build distinct adapters to learn domain-specific knowledge at each step, which can effectively learn and preserve knowledge across different datasets. Meanwhile, the proposed auto-selection strategy adaptively calculates the knowledge similarity between the input set and the adapters. On the one hand, the appropriate adapters are selected for the inputs to process ReID, and on the other hand, the knowledge interaction and fusion between adapters are enhanced to improve the generalization ability of the model. Extensive experiments are conducted to demonstrate the superiority of our AdalReID, which significantly outperforms SOTAs by about 10$\sim$20\% mAP on both seen and unseen domains. △ Less

Submitted 30 May, 2024; v1 submitted 29 May, 2024; originally announced May 2024.

MindMerger: Efficient Boosting LLM Reasoning in non-English Languages

Authors: Zixian Huang, Wenhao Zhu, Gong Cheng, Lei Li, Fei Yuan

Abstract: Reasoning capabilities are crucial for Large Language Models (LLMs), yet a notable gap exists between English and non-English languages. To bridge this disparity, some works fine-tune LLMs to relearn reasoning capabilities in non-English languages, while others replace non-English inputs with an external model's outputs such as English translation text to circumvent the challenge of LLM understand… ▽ More Reasoning capabilities are crucial for Large Language Models (LLMs), yet a notable gap exists between English and non-English languages. To bridge this disparity, some works fine-tune LLMs to relearn reasoning capabilities in non-English languages, while others replace non-English inputs with an external model's outputs such as English translation text to circumvent the challenge of LLM understanding non-English. Unfortunately, these methods often underutilize the built-in skilled reasoning and useful language understanding capabilities of LLMs. In order to better utilize the minds of reasoning and language understanding in LLMs, we propose a new method, namely MindMerger, which merges LLMs with the external language understanding capabilities from multilingual models to boost the multilingual reasoning performance. Furthermore, a two-step training scheme is introduced to first train to embeded the external capabilities into LLMs and then train the collaborative utilization of the external capabilities and the built-in capabilities in LLMs. Experiments on three multilingual reasoning datasets and a language understanding dataset demonstrate that MindMerger consistently outperforms all baselines, especially in low-resource languages. Without updating the parameters of LLMs, the average accuracy improved by 6.7% and 8.0% across all languages and low-resource languages on the MGSM dataset, respectively. △ Less

Submitted 27 May, 2024; originally announced May 2024.

Transfer Learning for Diffusion Models

Authors: Yidong Ouyang, Liyan Xie, Hongyuan Zha, Guang Cheng

Abstract: Diffusion models, a specific type of generative model, have achieved unprecedented performance in recent years and consistently produce high-quality synthetic samples. A critical prerequisite for their notable success lies in the presence of a substantial number of training samples, which can be impractical in real-world applications due to high collection costs or associated risks. Consequently,… ▽ More Diffusion models, a specific type of generative model, have achieved unprecedented performance in recent years and consistently produce high-quality synthetic samples. A critical prerequisite for their notable success lies in the presence of a substantial number of training samples, which can be impractical in real-world applications due to high collection costs or associated risks. Consequently, various finetuning and regularization approaches have been proposed to transfer knowledge from existing pre-trained models to specific target domains with limited data. This paper introduces the Transfer Guided Diffusion Process (TGDP), a novel approach distinct from conventional finetuning and regularization methods. We prove that the optimal diffusion model for the target domain integrates pre-trained diffusion models on the source domain with additional guidance from a domain classifier. We further extend TGDP to a conditional version for modeling the joint distribution of data and its corresponding labels, together with two additional regularization terms to enhance the model performance. We validate the effectiveness of TGDP on Gaussian mixture simulations and on real electrocardiogram (ECG) datasets. △ Less

Submitted 27 May, 2024; v1 submitted 27 May, 2024; originally announced May 2024.

Comments: 24 pages

Latent Energy-Based Odyssey: Black-Box Optimization via Expanded Exploration in the Energy-Based Latent Space

Authors: Peiyu Yu, Dinghuai Zhang, Hengzhi He, Xiaojian Ma, Ruiyao Miao, Yifan Lu, Yasi Zhang, Deqian Kong, Ruiqi Gao, Jianwen Xie, Guang Cheng, Ying Nian Wu

Abstract: Offline Black-Box Optimization (BBO) aims at optimizing a black-box function using the knowledge from a pre-collected offline dataset of function values and corresponding input designs. However, the high-dimensional and highly-multimodal input design space of black-box function pose inherent challenges for most existing methods that model and operate directly upon input designs. These issues inclu… ▽ More Offline Black-Box Optimization (BBO) aims at optimizing a black-box function using the knowledge from a pre-collected offline dataset of function values and corresponding input designs. However, the high-dimensional and highly-multimodal input design space of black-box function pose inherent challenges for most existing methods that model and operate directly upon input designs. These issues include but are not limited to high sample complexity, which relates to inaccurate approximation of black-box function; and insufficient coverage and exploration of input design modes, which leads to suboptimal proposal of new input designs. In this work, we consider finding a latent space that serves as a compressed yet accurate representation of the design-value joint space, enabling effective latent exploration of high-value input design modes. To this end, we formulate an learnable energy-based latent space, and propose Noise-intensified Telescoping density-Ratio Estimation (NTRE) scheme for variational learning of an accurate latent space model without costly Markov Chain Monte Carlo. The optimization process is then exploration of high-value designs guided by the learned energy-based model in the latent space, formulated as gradient-based sampling from a latent-variable-parameterized inverse model. We show that our particular parameterization encourages expanded exploration around high-value design modes, motivated by inversion thinking of a fundamental result of conditional covariance matrix typically used for variance reduction. We observe that our method, backed by an accurately learned informative latent space and an expanding-exploration model design, yields significant improvements over strong previous methods on both synthetic and real world datasets such as the design-bench suite. △ Less

Submitted 26 May, 2024; originally announced May 2024.

BadGD: A unified data-centric framework to identify gradient descent vulnerabilities

Authors: Chi-Hua Wang, Guang Cheng

Abstract: We present BadGD, a unified theoretical framework that exposes the vulnerabilities of gradient descent algorithms through strategic backdoor attacks. Backdoor attacks involve embedding malicious triggers into a training dataset to disrupt the model's learning process. Our framework introduces three novel constructs: Max RiskWarp Trigger, Max GradWarp Trigger, and Max GradDistWarp Trigger, each des… ▽ More We present BadGD, a unified theoretical framework that exposes the vulnerabilities of gradient descent algorithms through strategic backdoor attacks. Backdoor attacks involve embedding malicious triggers into a training dataset to disrupt the model's learning process. Our framework introduces three novel constructs: Max RiskWarp Trigger, Max GradWarp Trigger, and Max GradDistWarp Trigger, each designed to exploit specific aspects of gradient descent by distorting empirical risk, deterministic gradients, and stochastic gradients respectively. We rigorously define clean and backdoored datasets and provide mathematical formulations for assessing the distortions caused by these malicious backdoor triggers. By measuring the impact of these triggers on the model training procedure, our framework bridges existing empirical findings with theoretical insights, demonstrating how a malicious party can exploit gradient descent hyperparameters to maximize attack effectiveness. In particular, we show that these exploitations can significantly alter the loss landscape and gradient calculations, leading to compromised model integrity and performance. This research underscores the severe threats posed by such data-centric attacks and highlights the urgent need for robust defenses in machine learning. BadGD sets a new standard for understanding and mitigating adversarial manipulations, ensuring the reliability and security of AI systems. △ Less

Submitted 24 May, 2024; originally announced May 2024.

Comments: 25 pages, 1 figure

Discriminative Estimation of Total Variation Distance: A Fidelity Auditor for Generative Data

Authors: Lan Tao, Shirong Xu, Chi-Hua Wang, Namjoon Suh, Guang Cheng

Abstract: With the proliferation of generative AI and the increasing volume of generative data (also called as synthetic data), assessing the fidelity of generative data has become a critical concern. In this paper, we propose a discriminative approach to estimate the total variation (TV) distance between two distributions as an effective measure of generative data fidelity. Our method quantitatively charac… ▽ More With the proliferation of generative AI and the increasing volume of generative data (also called as synthetic data), assessing the fidelity of generative data has become a critical concern. In this paper, we propose a discriminative approach to estimate the total variation (TV) distance between two distributions as an effective measure of generative data fidelity. Our method quantitatively characterizes the relation between the Bayes risk in classifying two distributions and their TV distance. Therefore, the estimation of total variation distance reduces to that of the Bayes risk. In particular, this paper establishes theoretical results regarding the convergence rate of the estimation error of TV distance between two Gaussian distributions. We demonstrate that, with a specific choice of hypothesis class in classification, a fast convergence rate in estimating the TV distance can be achieved. Specifically, the estimation accuracy of the TV distance is proven to inherently depend on the separation of two Gaussian distributions: smaller estimation errors are achieved when the two Gaussian distributions are farther apart. This phenomenon is also validated empirically through extensive simulations. In the end, we apply this discriminative estimation method to rank fidelity of synthetic image data using the MNIST dataset. △ Less

Submitted 24 May, 2024; originally announced May 2024.

Online robust estimation and bootstrap inference for function-on-scalar regression

Authors: Guanghui Cheng, Wenjuan Hu, Ruitao Lin, Chen Wang

Abstract: We propose a novel and robust online function-on-scalar regression technique via geometric median to learn associations between functional responses and scalar covariates based on massive or streaming datasets. The online estimation procedure, developed using the average stochastic gradient descent algorithm, offers an efficient and cost-effective method for analyzing sequentially augmented datase… ▽ More We propose a novel and robust online function-on-scalar regression technique via geometric median to learn associations between functional responses and scalar covariates based on massive or streaming datasets. The online estimation procedure, developed using the average stochastic gradient descent algorithm, offers an efficient and cost-effective method for analyzing sequentially augmented datasets, eliminating the need to store large volumes of data in memory. We establish the almost sure consistency, $L_p$ convergence, and asymptotic normality of the online estimator. To enable efficient and fast inference of the parameters of interest, including the derivation of confidence intervals, we also develop an innovative two-step online bootstrap procedure to approximate the limiting error distribution of the robust online estimator. Numerical studies under a variety of scenarios demonstrate the effectiveness and efficiency of the proposed online learning method. A real application analyzing PM$_{2.5}$ air-quality data is also included to exemplify the proposed online approach. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Watermarking Generative Tabular Data

Authors: Hengzhi He, Peiyu Yu, Junpeng Ren, Ying Nian Wu, Guang Cheng

Abstract: In this paper, we introduce a simple yet effective tabular data watermarking mechanism with statistical guarantees. We show theoretically that the proposed watermark can be effectively detected, while faithfully preserving the data fidelity, and also demonstrates appealing robustness against additive noise attack. The general idea is to achieve the watermarking through a strategic embedding based… ▽ More In this paper, we introduce a simple yet effective tabular data watermarking mechanism with statistical guarantees. We show theoretically that the proposed watermark can be effectively detected, while faithfully preserving the data fidelity, and also demonstrates appealing robustness against additive noise attack. The general idea is to achieve the watermarking through a strategic embedding based on simple data binning. Specifically, it divides the feature's value range into finely segmented intervals and embeds watermarks into selected ``green list" intervals. To detect the watermarks, we develop a principled statistical hypothesis-testing framework with minimal assumptions: it remains valid as long as the underlying data distribution has a continuous density function. The watermarking efficacy is demonstrated through rigorous theoretical analysis and empirical validation, highlighting its utility in enhancing the security of synthetic and real-world datasets. △ Less

Submitted 22 May, 2024; originally announced May 2024.

Networked ISAC for Low-Altitude Economy: Transmit Beamforming and UAV Trajectory Design

Authors: Gaoyuan Cheng, Xianxin Song, Zhonghao Lyu, Jie Xu

Abstract: This paper studies the exploitation of networked integrated sensing and communications (ISAC) to support low-altitude economy (LAE), in which a set of networked ground base stations (GBSs) transmit wireless signals to cooperatively communicate with multiple authorized unmanned aerial vehicles (UAVs) and concurrently use the echo signals to detect the invasion of unauthorized objects in interested… ▽ More This paper studies the exploitation of networked integrated sensing and communications (ISAC) to support low-altitude economy (LAE), in which a set of networked ground base stations (GBSs) transmit wireless signals to cooperatively communicate with multiple authorized unmanned aerial vehicles (UAVs) and concurrently use the echo signals to detect the invasion of unauthorized objects in interested airspace. Under this setup, we jointly design the cooperative transmit beamforming at multiple GBSs together with the trajectory control of authorized UAVs and their GBS associations, for enhancing the authorized UAVs' communication performance while ensuring the sensing requirements for airspace monitoring. In particular, our objective is to maximize the average sum rate of authorized UAVs over a particular flight period, subject to the minimum illumination power constraints for sensing over the interested airspace, the maximum transmit power constraints at individual GBSs, and the flight constraints at UAVs. This problem is non-convex and challenging to solve, due to the involvement of integer variables and the coupling of optimization variables. To solve this non-convex problem, we propose an efficient algorithm by using the techniques of alternating optimization (AO), successive convex approximation (SCA), and semi-definite relaxation (SDR). Numerical results show that the obtained transmit beamforming and UAV trajectory designs in the proposed algorithm efficiently balance the tradeoff between the sensing and communication performances, thus significantly outperforming various benchmarks. △ Less

Submitted 13 May, 2024; originally announced May 2024.

Tree-based Ensemble Learning for Out-of-distribution Detection

Authors: Zhaiming Shen, Menglun Wang, Guang Cheng, Ming-Jun Lai, Lin Mu, Ruihao Huang, Qi Liu, Hao Zhu

Abstract: Being able to successfully determine whether the testing samples has similar distribution as the training samples is a fundamental question to address before we can safely deploy most of the machine learning models into practice. In this paper, we propose TOOD detection, a simple yet effective tree-based out-of-distribution (TOOD) detection mechanism to determine if a set of unseen samples will ha… ▽ More Being able to successfully determine whether the testing samples has similar distribution as the training samples is a fundamental question to address before we can safely deploy most of the machine learning models into practice. In this paper, we propose TOOD detection, a simple yet effective tree-based out-of-distribution (TOOD) detection mechanism to determine if a set of unseen samples will have similar distribution as of the training samples. The TOOD detection mechanism is based on computing pairwise hamming distance of testing samples' tree embeddings, which are obtained by fitting a tree-based ensemble model through in-distribution training samples. Our approach is interpretable and robust for its tree-based nature. Furthermore, our approach is efficient, flexible to various machine learning tasks, and can be easily generalized to unsupervised setting. Extensive experiments are conducted to show the proposed method outperforms other state-of-the-art out-of-distribution detection methods in distinguishing the in-distribution from out-of-distribution on various tabular, image, and text data. △ Less

Submitted 5 May, 2024; originally announced May 2024.

Inferring State Machine from the Protocol Implementation via Large Language Model

Authors: Haiyang Wei, Zhengjie Du, Haohui Huang, Yue Liu, Guang Cheng, Linzhang Wang, Bing Mao

Abstract: State machines play a pivotal role in augmenting the efficacy of protocol analyzing to unveil more vulnerabilities. However, the task of inferring state machines from network protocol implementations presents significant challenges. Traditional methods based on dynamic analysis often overlook crucial state transitions due to limited coverage, while static analysis faces difficulties with complex c… ▽ More State machines play a pivotal role in augmenting the efficacy of protocol analyzing to unveil more vulnerabilities. However, the task of inferring state machines from network protocol implementations presents significant challenges. Traditional methods based on dynamic analysis often overlook crucial state transitions due to limited coverage, while static analysis faces difficulties with complex code structures and behaviors. To address these limitations, we propose an innovative state machine inference approach powered by Large Language Models (LLMs). Utilizing text-embedding technology, this method allows LLMs to dissect and analyze the intricacies of protocol implementation code. Through targeted prompt engineering, we systematically identify and infer the underlying state machines. Our evaluation across six protocol implementations demonstrates the method's high efficacy, achieving an accuracy rate exceeding 90% and successfully delineating differences on state machines among various implementations of the same protocol. Importantly, integrating this approach with protocol fuzzing has notably enhanced AFLNet's code coverage by 10% over RFCNLP, showcasing the considerable potential of LLMs in advancing network protocol security analysis. Our proposed method not only marks a significant step forward in accurate state machine inference but also opens new avenues for improving the security and reliability of protocol implementations. △ Less

Submitted 14 June, 2024; v1 submitted 1 May, 2024; originally announced May 2024.

Evidence for Plasmoid-mediated Magnetic Reconnection during a Small-scale Flare in the Partially Ionized Low Solar Atmosphere

Authors: Guanchong Cheng, Lei Ni, Zehao Tang, Yajie Chen, Yuhao Chen, Jialiang Hu, Jun Lin

Abstract: Magnetic reconnection plays a crucial role in the energy release process for different kinds of solar eruptions and activities. The rapid solar eruption requires a fast reconnection model. Plasmoid instability in the reconnecting current sheets is one of the most acceptable fast reconnection mechanisms for explaining the explosive events in the magnetohydrodynamics (MHD) scale, which is also a pot… ▽ More Magnetic reconnection plays a crucial role in the energy release process for different kinds of solar eruptions and activities. The rapid solar eruption requires a fast reconnection model. Plasmoid instability in the reconnecting current sheets is one of the most acceptable fast reconnection mechanisms for explaining the explosive events in the magnetohydrodynamics (MHD) scale, which is also a potential bridge between the macroscopic MHD reconnection process and microscale dissipations. Plenty of high resolution observations indicate that the plasmoid-like structures exist in the high temperature solar corona, but such evidences are very rare in the lower solar atmosphere with partially ionized plasmas. Utilizing joint observations from the Goode Solar Telescope (GST) and the Solar Dynamics Observatory (SDO), we discovered a small-scale eruptive phenomenon in NOAA AR 13085, characterized by clear reconnection cusp structures, supported by Nonlinear Force-Free Field (NLFFF) extrapolation results. The plasmoid-like structures with a size about 150 km were observed to be ejected downward from the current sheet at a maximum velocity of 24 km$\cdot$s$^{-1}$ in the H$α$ line wing images, followed by enhanced emissions at around the post flare loop region in multiple wave lengths. Our 2.5D high-resolution MHD simulations further reproduced such a phenomenon and revealed reconnection fine structures. These results provide comprehensive evidences for the plasmoid mediated reconnection in partially ionized plasmas, and suggest an unified reconnection model for solar flares with different length scales from the lower chromosphere to corona. △ Less

Submitted 29 April, 2024; originally announced April 2024.

The Jive Verification System and its Transformative Impact on Weather Forecasting Operations

Authors: Nicholas Loveday, Deryn Griffiths, Tennessee Leeuwenburg, Robert Taggart, Thomas C. Pagano, George Cheng, Kevin Plastow, Elizabeth Ebert, Cassandra Templeton, Maree Carroll, Mohammadreza Khanarmuei, Isha Nagpal

Abstract: Forecast verification is critical for continuous improvement in meteorological organizations. The Jive verification system was originally developed to assess the accuracy of public weather forecasts issued by the Australian Bureau of Meteorology. It started as a research project in 2015 and gradually evolved to be a Bureau operational verification system in 2022. The system includes daily verifica… ▽ More Forecast verification is critical for continuous improvement in meteorological organizations. The Jive verification system was originally developed to assess the accuracy of public weather forecasts issued by the Australian Bureau of Meteorology. It started as a research project in 2015 and gradually evolved to be a Bureau operational verification system in 2022. The system includes daily verification dashboards for forecasters to visualize recent forecast performance and "Evidence Targeted Automation" dashboards for exploring the performance of competing forecast systems. Additionally, there is a Jupyter Notebook server with the Jive Python library which supports research experiments, case studies, and the development of new verification metrics and tools. This paper describes the Jive verification system and how it helped bring verification to the forefront at the Bureau of Meteorology, leading to more accurate, streamlined forecasts. Jive has been used to provide evidence to support forecast automation decisions and has helped to understand the evolving role of meteorologists in the forecast process. It has given operational meteorologists tools for evaluating forecast processes, including identifying when and how manual interventions lead to superior predictions. Work on Jive led to new verification science, including novel metrics that are decision-focused, including for extreme conditions. Jive also provided the Bureau with an enterprise-wide data analysis environment and has prompted a clarification of forecast definitions. These collective impacts have resulted in more accurate forecasts, ultimately benefiting society, and building trust with forecast users. These positive outcomes highlight the importance of meteorological organizations investing in verification science and technology. △ Less

Submitted 9 July, 2024; v1 submitted 29 April, 2024; originally announced April 2024.

Angle-Resolved Magneto-Chiral Anisotropy in a Non-Centrosymmetric Atomic Layer Superlattice

Authors: Long Cheng, Mingrui Bao, Jingxian Zhang, Xue Zhang, Qun Yang, Qiang Li, Hui Cao, Dawei Qiu, Jia Liu, Fei Ye, Qing Wang, Genhao Liang, Hui Li, Guanglei Cheng, Hua Zhou, Jian-Min Zuo, Xiaodong Zhou, Jian Shen, Zhifeng Zhu, Sai Mu, Wenbo Wang, Xiaofang Zhai

Abstract: Chirality in solid-state materials has sparked significant interest due to potential applications of topologically-protected chiral states in next-generation information technology. The electrical magneto-chiral effect (eMChE), arising from relativistic spin-orbit interactions, shows great promise for developing chiral materials and devices for electronic integration. Here we demonstrate an angle-… ▽ More Chirality in solid-state materials has sparked significant interest due to potential applications of topologically-protected chiral states in next-generation information technology. The electrical magneto-chiral effect (eMChE), arising from relativistic spin-orbit interactions, shows great promise for developing chiral materials and devices for electronic integration. Here we demonstrate an angle-resolved eMChE in an A-B-C-C type atomic-layer superlattice lacking time and space inversion symmetry. We observe non-superimposable enantiomers of left-handed and right-handed tilted uniaxial magnetic anisotropy as the sample rotates under static fields, with the tilting angle reaching a striking 45 degree. Magnetic force microscopy and atomistic simulations correlate the tilt to the emergence and evolution of chiral spin textures. The Dzyaloshinskii-Moriya interaction lock effect in competition with Zeeman effect is demonstrated to be responsible for the angle-resolved eMChE. Our findings open up a new horizon for engineering angle-resolved magneto-chiral anisotropy, shedding light on the development of novel angle-resolved sensing or writing techniques in chiral spintronics. △ Less

Submitted 20 April, 2024; originally announced April 2024.

Exploring Key Point Analysis with Pairwise Generation and Graph Partitioning

Authors: Xiao Li, Yong Jiang, Shen Huang, Pengjun Xie, Gong Cheng, Fei Huang

Abstract: Key Point Analysis (KPA), the summarization of multiple arguments into a concise collection of key points, continues to be a significant and unresolved issue within the field of argument mining. Existing models adapt a two-stage pipeline of clustering arguments or generating key points for argument clusters. This approach rely on semantic similarity instead of measuring the existence of shared key… ▽ More Key Point Analysis (KPA), the summarization of multiple arguments into a concise collection of key points, continues to be a significant and unresolved issue within the field of argument mining. Existing models adapt a two-stage pipeline of clustering arguments or generating key points for argument clusters. This approach rely on semantic similarity instead of measuring the existence of shared key points among arguments. Additionally, it only models the intra-cluster relationship among arguments, disregarding the inter-cluster relationship between arguments that do not share key points. To address these limitations, we propose a novel approach for KPA with pairwise generation and graph partitioning. Our objective is to train a generative model that can simultaneously provide a score indicating the presence of shared key point between a pair of arguments and generate the shared key point. Subsequently, to map generated redundant key points to a concise set of key points, we proceed to construct an arguments graph by considering the arguments as vertices, the generated key points as edges, and the scores as edge weights. We then propose a graph partitioning algorithm to partition all arguments sharing the same key points to the same subgraph. Notably, our experimental findings demonstrate that our proposed model surpasses previous models when evaluated on both the ArgKP and QAM datasets. △ Less

Submitted 17 April, 2024; originally announced April 2024.

Comments: 11 pages, 4 figures, 4 tables. Accepted to NAACL 2024

Empowering Large Language Models on Robotic Manipulation with Affordance Prompting

Authors: Guangran Cheng, Chuheng Zhang, Wenzhe Cai, Li Zhao, Changyin Sun, Jiang Bian

Abstract: While large language models (LLMs) are successful in completing various language processing tasks, they easily fail to interact with the physical world by generating control sequences properly. We find that the main reason is that LLMs are not grounded in the physical world. Existing LLM-based approaches circumvent this problem by relying on additional pre-defined skills or pre-trained sub-policie… ▽ More While large language models (LLMs) are successful in completing various language processing tasks, they easily fail to interact with the physical world by generating control sequences properly. We find that the main reason is that LLMs are not grounded in the physical world. Existing LLM-based approaches circumvent this problem by relying on additional pre-defined skills or pre-trained sub-policies, making it hard to adapt to new tasks. In contrast, we aim to address this problem and explore the possibility to prompt pre-trained LLMs to accomplish a series of robotic manipulation tasks in a training-free paradigm. Accordingly, we propose a framework called LLM+A(ffordance) where the LLM serves as both the sub-task planner (that generates high-level plans) and the motion controller (that generates low-level control sequences). To ground these plans and control sequences on the physical world, we develop the affordance prompting technique that stimulates the LLM to 1) predict the consequences of generated plans and 2) generate affordance values for relevant objects. Empirically, we evaluate the effectiveness of LLM+A in various language-conditioned robotic manipulation tasks, which show that our approach substantially improves performance by enhancing the feasibility of generated plans and control and can easily generalize to different environments. △ Less

Submitted 16 April, 2024; originally announced April 2024.

StreakNet-Arch: An Anti-scattering Network-based Architecture for Underwater Carrier LiDAR-Radar Imaging

Authors: Xuelong Li, Hongjun An, Guangying Li, Xing Wang, Guanghua Cheng, Zhe Sun

Abstract: In this paper, we introduce StreakNet-Arch, a novel signal processing architecture designed for Underwater Carrier LiDAR-Radar (UCLR) imaging systems, to address the limitations in scatter suppression and real-time imaging. StreakNet-Arch formulates the signal processing as a real-time, end-to-end binary classification task, enabling real-time image acquisition. To achieve this, we leverage Self-A… ▽ More In this paper, we introduce StreakNet-Arch, a novel signal processing architecture designed for Underwater Carrier LiDAR-Radar (UCLR) imaging systems, to address the limitations in scatter suppression and real-time imaging. StreakNet-Arch formulates the signal processing as a real-time, end-to-end binary classification task, enabling real-time image acquisition. To achieve this, we leverage Self-Attention networks and propose a novel Double Branch Cross Attention (DBC-Attention) mechanism that surpasses the performance of traditional methods. Furthermore, we present a method for embedding streak-tube camera images into attention networks, effectively acting as a learned bandpass filter. To facilitate further research, we contribute a publicly available streak-tube camera image dataset. The dataset contains 2,695,168 real-world underwater 3D point cloud data. These advancements significantly improve UCLR capabilities, enhancing its performance and applicability in underwater imaging tasks. The source code and dataset can be found at https://github.com/BestAnHongjun/StreakNet . △ Less

Submitted 23 April, 2024; v1 submitted 14 April, 2024; originally announced April 2024.

Comments: Reduce the number of pages to 13

SAM-dPCR: Real-Time and High-throughput Absolute Quantification of Biological Samples Using Zero-Shot Segment Anything Model

Authors: Yuanyuan Wei, Shanhang Luo, Changran Xu, Yingqi Fu, Qingyue Dong, Yi Zhang, Fuyang Qu, Guangyao Cheng, Yi-Ping Ho, Ho-Pui Ho, Wu Yuan

Abstract: Digital PCR (dPCR) has revolutionized nucleic acid diagnostics by enabling absolute quantification of rare mutations and target sequences. However, current detection methodologies face challenges, as flow cytometers are costly and complex, while fluorescence imaging methods, relying on software or manual counting, are time-consuming and prone to errors. To address these limitations, we present SAM… ▽ More Digital PCR (dPCR) has revolutionized nucleic acid diagnostics by enabling absolute quantification of rare mutations and target sequences. However, current detection methodologies face challenges, as flow cytometers are costly and complex, while fluorescence imaging methods, relying on software or manual counting, are time-consuming and prone to errors. To address these limitations, we present SAM-dPCR, a novel self-supervised learning-based pipeline that enables real-time and high-throughput absolute quantification of biological samples. Leveraging the zero-shot SAM model, SAM-dPCR efficiently analyzes diverse microreactors with over 97.7% accuracy within a rapid processing time of 3.16 seconds. By utilizing commonly available lab fluorescence microscopes, SAM-dPCR facilitates the quantification of sample concentrations. The accuracy of SAM-dPCR is validated by the strong linear relationship observed between known and inferred sample concentrations. Additionally, SAM-dPCR demonstrates versatility through comprehensive verification using various samples and reactor morphologies. This accessible, cost-effective tool transcends the limitations of traditional detection methods or fully supervised AI models, marking the first application of SAM in nucleic acid detection or molecular diagnostics. By eliminating the need for annotated training data, SAM-dPCR holds great application potential for nucleic acid quantification in resource-limited settings. △ Less

Submitted 22 January, 2024; originally announced March 2024.

Comments: 23 pages, 6 figures

Minimax Optimal Fair Classification with Bounded Demographic Disparity

Authors: Xianli Zeng, Guang Cheng, Edgar Dobriban

Abstract: Mitigating the disparate impact of statistical machine learning methods is crucial for ensuring fairness. While extensive research aims to reduce disparity, the effect of using a \emph{finite dataset} -- as opposed to the entire population -- remains unclear. This paper explores the statistical foundations of fair binary classification with two protected groups, focusing on controlling demographic… ▽ More Mitigating the disparate impact of statistical machine learning methods is crucial for ensuring fairness. While extensive research aims to reduce disparity, the effect of using a \emph{finite dataset} -- as opposed to the entire population -- remains unclear. This paper explores the statistical foundations of fair binary classification with two protected groups, focusing on controlling demographic disparity, defined as the difference in acceptance rates between the groups. Although fairness may come at the cost of accuracy even with infinite data, we show that using a finite sample incurs additional costs due to the need to estimate group-specific acceptance thresholds. We study the minimax optimal classification error while constraining demographic disparity to a user-specified threshold. To quantify the impact of fairness constraints, we introduce a novel measure called \emph{fairness-aware excess risk} and derive a minimax lower bound on this measure that all classifiers must satisfy. Furthermore, we propose FairBayes-DDP+, a group-wise thresholding method with an offset that we show attains the minimax lower bound. Our lower bound proofs involve several innovations. Experiments support that FairBayes-DDP+ controls disparity at the user-specified level, while being faster and having a more favorable fairness-accuracy tradeoff than several baselines. △ Less

Submitted 26 March, 2024; originally announced March 2024.

Modeling Analog Dynamic Range Compressors using Deep Learning and State-space Models

Authors: Hanzhi Yin, Gang Cheng, Christian J. Steinmetz, Ruibin Yuan, Richard M. Stern, Roger B. Dannenberg

Abstract: We describe a novel approach for developing realistic digital models of dynamic range compressors for digital audio production by analyzing their analog prototypes. While realistic digital dynamic compressors are potentially useful for many applications, the design process is challenging because the compressors operate nonlinearly over long time scales. Our approach is based on the structured stat… ▽ More We describe a novel approach for developing realistic digital models of dynamic range compressors for digital audio production by analyzing their analog prototypes. While realistic digital dynamic compressors are potentially useful for many applications, the design process is challenging because the compressors operate nonlinearly over long time scales. Our approach is based on the structured state space sequence model (S4), as implementing the state-space model (SSM) has proven to be efficient at learning long-range dependencies and is promising for modeling dynamic range compressors. We present in this paper a deep learning model with S4 layers to model the Teletronix LA-2A analog dynamic range compressor. The model is causal, executes efficiently in real time, and achieves roughly the same quality as previous deep-learning models but with fewer parameters. △ Less

Submitted 24 March, 2024; originally announced March 2024.

Approximation of RKHS Functionals by Neural Networks

Authors: Tian-Yi Zhou, Namjoon Suh, Guang Cheng, Xiaoming Huo

Abstract: Motivated by the abundance of functional data such as time series and images, there has been a growing interest in integrating such data into neural networks and learning maps from function spaces to R (i.e., functionals). In this paper, we study the approximation of functionals on reproducing kernel Hilbert spaces (RKHS's) using neural networks. We establish the universality of the approximation… ▽ More Motivated by the abundance of functional data such as time series and images, there has been a growing interest in integrating such data into neural networks and learning maps from function spaces to R (i.e., functionals). In this paper, we study the approximation of functionals on reproducing kernel Hilbert spaces (RKHS's) using neural networks. We establish the universality of the approximation of functionals on the RKHS's. Specifically, we derive explicit error bounds for those induced by inverse multiquadric, Gaussian, and Sobolev kernels. Moreover, we apply our findings to functional regression, proving that neural networks can accurately approximate the regression maps in generalized functional linear models. Existing works on functional learning require integration-type basis function expansions with a set of pre-specified basis functions. By leveraging the interpolating orthogonal projections in RKHS's, our proposed network is much simpler in that we use point evaluations to replace basis function expansions. △ Less

Submitted 18 March, 2024; originally announced March 2024.

FairRR: Pre-Processing for Group Fairness through Randomized Response

Authors: Xianli Zeng, Joshua Ward, Guang Cheng

Abstract: The increasing usage of machine learning models in consequential decision-making processes has spurred research into the fairness of these systems. While significant work has been done to study group fairness in the in-processing and post-processing setting, there has been little that theoretically connects these results to the pre-processing domain. This paper proposes that achieving group fairne… ▽ More The increasing usage of machine learning models in consequential decision-making processes has spurred research into the fairness of these systems. While significant work has been done to study group fairness in the in-processing and post-processing setting, there has been little that theoretically connects these results to the pre-processing domain. This paper proposes that achieving group fairness in downstream models can be formulated as finding the optimal design matrix in which to modify a response variable in a Randomized Response framework. We show that measures of group fairness can be directly controlled for with optimal model utility, proposing a pre-processing algorithm called FairRR that yields excellent downstream model utility and fairness. △ Less

Submitted 12 March, 2024; originally announced March 2024.

Gravitational back-reaction is magical

Authors: ChunJun Cao, Gong Cheng, Alioscia Hamma, Lorenzo Leone, William Munizzi, Savatore F. E. Oliviero

Abstract: We study the interplay between magic and entanglement in quantum many-body systems. We show that non-local magic, which is supported by the quantum correlations is lower bounded by the non-flatness of entanglement spectrum and upper bounded by the amount of entanglement in the system. We then argue that a smoothed version of non-local magic bounds the hardness of classical simulations for incompre… ▽ More We study the interplay between magic and entanglement in quantum many-body systems. We show that non-local magic, which is supported by the quantum correlations is lower bounded by the non-flatness of entanglement spectrum and upper bounded by the amount of entanglement in the system. We then argue that a smoothed version of non-local magic bounds the hardness of classical simulations for incompressible states. In conformal field theories, we conjecture that the non-local magic should scale linearly with entanglement entropy but sublinearly when an approximation of the state is allowed. We support the conjectures using both analytical arguments based on unitary distillation and numerical data from an Ising CFT. If the CFT has a holographic dual, then we prove that the non-local magic vanishes if and only if there is no gravitational back-reaction. Furthermore, we show that non-local magic is approximately equal to the rate of change of the minimal surface area in response to the change of cosmic brane tension in the bulk. △ Less

Submitted 16 May, 2024; v1 submitted 11 March, 2024; originally announced March 2024.

Comments: 62 pages, 20 figures; title changed, Theorem 1 and 2 refined, references added

TRAWL: External Knowledge-Enhanced Recommendation with LLM Assistance

Authors: Weiqing Luo, Chonggang Song, Lingling Yi, Gong Cheng

Abstract: Combining semantic information with behavioral data is a crucial research area in recommender systems. A promising approach involves leveraging external knowledge to enrich behavioral-based recommender systems with abundant semantic information. However, this approach faces two primary challenges: denoising raw external knowledge and adapting semantic representations. To address these challenges,… ▽ More Combining semantic information with behavioral data is a crucial research area in recommender systems. A promising approach involves leveraging external knowledge to enrich behavioral-based recommender systems with abundant semantic information. However, this approach faces two primary challenges: denoising raw external knowledge and adapting semantic representations. To address these challenges, we propose an External Knowledge-Enhanced Recommendation method with LLM Assistance (TRAWL). This method utilizes large language models (LLMs) to extract relevant recommendation knowledge from raw external data and employs a contrastive learning strategy for adapter training. Experiments on public datasets and real-world online recommender systems validate the effectiveness of our approach. △ Less

Submitted 24 May, 2024; v1 submitted 11 March, 2024; originally announced March 2024.

Comments: 8 pages, 3 figures

Data Nuggets: A Method for Reducing Big Data While Preserving Data Structure

Authors: Traymon E. Beavers, Ge Cheng, Yajie Duan, Javier Cabrera, Mariusz Lubomirski, Dhammika Amaratunga, Jeffrey E. Teigler

Abstract: Big data, with NxP dimension where N is extremely large, has created new challenges for data analysis, particularly in the realm of creating meaningful clusters of data. Clustering techniques, such as K-means or hierarchical clustering are popular methods for performing exploratory analysis on large datasets. Unfortunately, these methods are not always possible to apply to big data due to memory o… ▽ More Big data, with NxP dimension where N is extremely large, has created new challenges for data analysis, particularly in the realm of creating meaningful clusters of data. Clustering techniques, such as K-means or hierarchical clustering are popular methods for performing exploratory analysis on large datasets. Unfortunately, these methods are not always possible to apply to big data due to memory or time constraints generated by calculations of order PxN(N-1). To circumvent this problem, typically, the clustering technique is applied to a random sample drawn from the dataset: however, a weakness is that the structure of the dataset, particularly at the edges, is not necessarily maintained. We propose a new solution through the concept of "data nuggets", which reduce a large dataset into a small collection of nuggets of data, each containing a center, weight, and scale parameter. The data nuggets are then input into algorithms that compute methods such as principal components analysis and clustering in a more computationally efficient manner. We show the consistency of the data nuggets-based covariance estimator and apply the methodology of data nuggets to perform exploratory analysis of a flow cytometry dataset containing over one million observations using PCA and K-means clustering for weighted observations. Supplementary materials for this article are available online. △ Less

Submitted 5 March, 2024; originally announced March 2024.

Comments: 31 pages and 8 figures

MSC Class: 62:0

Rate-Optimal Rank Aggregation with Private Pairwise Rankings

Authors: Shirong Xu, Will Wei Sun, Guang Cheng

Abstract: In various real-world scenarios like recommender systems and political surveys, pairwise rankings are commonly collected and utilized for rank aggregation to obtain an overall ranking of items. However, preference rankings can reveal individuals' personal preferences, underscoring the need to protect them before releasing for downstream analysis. In this paper, we address the challenge of preservi… ▽ More In various real-world scenarios like recommender systems and political surveys, pairwise rankings are commonly collected and utilized for rank aggregation to obtain an overall ranking of items. However, preference rankings can reveal individuals' personal preferences, underscoring the need to protect them before releasing for downstream analysis. In this paper, we address the challenge of preserving privacy while ensuring the utility of rank aggregation based on pairwise rankings generated from the Bradley-Terry-Luce (BTL) model. Using the randomized response mechanism to perturb raw pairwise rankings is a common privacy protection strategy used in practice, but a critical challenge arises because the privatized rankings no longer adhere to the BTL model, resulting in significant bias in downstream rank aggregation tasks. Motivated from this, we propose a debiased randomized response mechanism to protect the raw pairwise rankings, ensuring consistent estimation of true preferences and rankings in downstream rank aggregation. Theoretically, we offer insights into the relationship between overall privacy guarantees and estimation errors from private ranking data, and establish minimax rates for estimation errors. This enables the determination of optimal privacy guarantees that balance consistency in rank aggregation with robust privacy protection. We also investigate convergence rates of expected ranking errors for partial and full ranking recovery, quantifying how privacy protection influences the specification of top-$K$ item sets and complete rankings. Our findings are validated through extensive simulations and a real application. △ Less

Submitted 26 February, 2024; originally announced February 2024.

FormulaReasoning: A Dataset for Formula-Based Numerical Reasoning

Authors: Xiao Li, Bolin Zhu, Sichen Liu, Yin Zhu, Yiwei Liu, Gong Cheng

Abstract: The application of formulas is a fundamental ability of humans when addressing numerical reasoning problems. However, existing numerical reasoning datasets seldom explicitly indicate the formulas employed during the reasoning steps. To bridge this gap, we construct a dataset for formula-based numerical reasoning called FormulaReasoning, which consists of 5,420 reasoning-based questions. We employ… ▽ More The application of formulas is a fundamental ability of humans when addressing numerical reasoning problems. However, existing numerical reasoning datasets seldom explicitly indicate the formulas employed during the reasoning steps. To bridge this gap, we construct a dataset for formula-based numerical reasoning called FormulaReasoning, which consists of 5,420 reasoning-based questions. We employ it to conduct evaluations of LLMs with size ranging from 7B to over 100B parameters utilizing zero-shot and few-shot chain-of-thought methods, and we further explore using retrieval-augmented LLMs provided with an external formula database associated with our dataset. We also experiment with supervised methods where we divide the reasoning process into formula generation, parameter extraction, and numerical calculation, and perform data augmentation. Our empirical findings underscore the significant potential for improvement in existing models when applied to our complex, formula-driven FormulaReasoning. △ Less

Submitted 12 June, 2024; v1 submitted 19 February, 2024; originally announced February 2024.

A Survey on Extractive Knowledge Graph Summarization: Applications, Approaches, Evaluation, and Future Directions

Authors: Xiaxia Wang, Gong Cheng

Abstract: With the continuous growth of large Knowledge Graphs (KGs), extractive KG summarization becomes a trending task. Aiming at distilling a compact subgraph with condensed information, it facilitates various downstream KG-based tasks. In this survey paper, we are among the first to provide a systematic overview of its applications and define a taxonomy for existing methods from its interdisciplinary s… ▽ More With the continuous growth of large Knowledge Graphs (KGs), extractive KG summarization becomes a trending task. Aiming at distilling a compact subgraph with condensed information, it facilitates various downstream KG-based tasks. In this survey paper, we are among the first to provide a systematic overview of its applications and define a taxonomy for existing methods from its interdisciplinary studies. Future directions are also laid out based on our extensive and comparative review. △ Less

Submitted 19 February, 2024; originally announced February 2024.

Comments: 9 pages, 13 figures, submitted to the IJCAI 2024 Survey Track

Holographic phenomenology via overlapping degrees of freedom

Authors: Oliver Friedrich, ChunJun Cao, Sean M. Carroll, Gong Cheng, Ashmeet Singh

Abstract: The holographic principle suggests that regions of space contain fewer physical degrees of freedom than would be implied by conventional quantum field theory. Meanwhile, in Hilbert spaces of large dimension $2^n$, it is possible to define $N \gg n$ Pauli algebras that are nearly anti-commuting (but not quite) and which can be thought of as "overlapping degrees of freedom". We propose to model the… ▽ More The holographic principle suggests that regions of space contain fewer physical degrees of freedom than would be implied by conventional quantum field theory. Meanwhile, in Hilbert spaces of large dimension $2^n$, it is possible to define $N \gg n$ Pauli algebras that are nearly anti-commuting (but not quite) and which can be thought of as "overlapping degrees of freedom". We propose to model the phenomenology of holographic theories by allowing field-theory modes to be overlapping, and derive potential observational consequences. In particular, we build a Fermionic quantum field whose effective degrees of freedom approximately obey area scaling and satisfy a cosmic Bekenstein bound, and compare predictions of that model to cosmic neutrino observations. Our implementation of holography implies a finite lifetime of plane waves, which depends on the overall UV cutoff of the theory. To allow for neutrino flux from blazar TXS 0506+056 to be observable, our model needs to have a cutoff $k_{\mathrm{UV}} \lesssim 500\, k_{\mathrm{LHC}}\,$. This is broadly consistent with current bounds on the energy spectrum of cosmic neutrinos from IceCube, but high energy neutrinos are a potential challenge for our model of holography. We motivate our construction via quantum mereology, i.e. using the idea that EFT degrees of freedom should emerge from an abstract theory of quantum gravity by finding quasi-classical Hilbert space decompositions. We also discuss how to extend the framework to Bosons. Finally, using results from random matrix theory we derive an analytical understanding of the energy spectrum of our theory. The numerical tools used in this work are publicly available within the GPUniverse package, https://github.com/OliverFHD/GPUniverse . △ Less

Submitted 5 March, 2024; v1 submitted 16 February, 2024; originally announced February 2024.

Comments: 46 pages + appendix; code and data available at https://github.com/OliverFHD/GPUniverse

A magnetic reconnection model for the hot explosion with both ultraviolet and Hα wing emissions

Authors: Guanchong Cheng, Lei Ni, Yajie Chen, Jun Lin

Abstract: Ellerman bombs (EBs) with significant H$α$ wing emissions and ultraviolet bursts (UV bursts) with strong Si IV emissions are two kinds of small transient brightening events that occur in the low solar atmosphere.We numerically investigated the magnetic reconnection process between the emerging arch magnetic field and the lower atmospheric background magnetic field. We aim to find out if the hot UV… ▽ More Ellerman bombs (EBs) with significant H$α$ wing emissions and ultraviolet bursts (UV bursts) with strong Si IV emissions are two kinds of small transient brightening events that occur in the low solar atmosphere.We numerically investigated the magnetic reconnection process between the emerging arch magnetic field and the lower atmospheric background magnetic field. We aim to find out if the hot UV emissions and much colder H$α$ wing emissions can both appear in the same reconnection process and how they are located in the reconnection region. The open-source code NIRVANA was applied to perform the 2.5D magnetohydrodynamic (MHD) simulation. We developed the related sub-codes to include the more realistic radiative cooling process for the photosphere and chromosphere and the time-dependent ionization degree of hydrogen. The initial background magnetic field is 600 G, and the emerged magnetic field in the solar atmosphere is of the same magnitude, meaning that it results in a low- $β$ magnetic reconnection environment. We also used the radiative transfer code RH1.5D to synthesize the Si IV and H$α$ spectral line profiles based on the MHD simulation results. Magnetic reconnection between emerged and background magnetic fields creates a thin, curved current sheet, which then leads to the formation of plasmoid instability and the nonuniform density distributions. The mix of hot tenuous and much cooler dense plasmas in the turbulent reconnection region can appear at about the same height, or even in the same plasmoid. The turbulent current sheet is always in a dense plasma environment with an optical depth larger than 6.5$\times$10$^{-5}$ due to the emerged magnetic field pushing high-density plasmas upward. △ Less

Submitted 20 February, 2024; v1 submitted 11 February, 2024; originally announced February 2024.

DeMarking: A Defense for Network Flow Watermarking in Real-Time

Authors: Yali Yuan, Jian Ge, Guang Cheng

Abstract: The network flow watermarking technique associates the two communicating parties by actively modifying certain characteristics of the stream generated by the sender so that it covertly carries some special marking information. Some curious users communicating with the hidden server as a Tor client may attempt de-anonymization attacks to uncover the real identity of the hidden server by using this… ▽ More The network flow watermarking technique associates the two communicating parties by actively modifying certain characteristics of the stream generated by the sender so that it covertly carries some special marking information. Some curious users communicating with the hidden server as a Tor client may attempt de-anonymization attacks to uncover the real identity of the hidden server by using this technique. This compromises the privacy of the anonymized communication system. Therefore, we propose a defense scheme against flow watermarking. The scheme is based on deep neural networks and utilizes generative adversarial networks to convert the original Inter-Packet Delays (IPD) into new IPDs generated by the model. We also adopt the concept of adversarial attacks to ensure that the detector will produce an incorrect classification when detecting these new IPDs. This approach ensures that these IPDs are considered "clean", effectively covering the potential watermarks. This scheme is effective against time-based flow watermarking techniques. △ Less

Submitted 6 February, 2024; v1 submitted 6 February, 2024; originally announced February 2024.

Bayes-Optimal Fair Classification with Linear Disparity Constraints via Pre-, In-, and Post-processing

Authors: Xianli Zeng, Guang Cheng, Edgar Dobriban

Abstract: Machine learning algorithms may have disparate impacts on protected groups. To address this, we develop methods for Bayes-optimal fair classification, aiming to minimize classification error subject to given group fairness constraints. We introduce the notion of \emph{linear disparity measures}, which are linear functions of a probabilistic classifier; and \emph{bilinear disparity measures}, which… ▽ More Machine learning algorithms may have disparate impacts on protected groups. To address this, we develop methods for Bayes-optimal fair classification, aiming to minimize classification error subject to given group fairness constraints. We introduce the notion of \emph{linear disparity measures}, which are linear functions of a probabilistic classifier; and \emph{bilinear disparity measures}, which are also linear in the group-wise regression functions. We show that several popular disparity measures -- the deviations from demographic parity, equality of opportunity, and predictive equality -- are bilinear. We find the form of Bayes-optimal fair classifiers under a single linear disparity measure, by uncovering a connection with the Neyman-Pearson lemma. For bilinear disparity measures, Bayes-optimal fair classifiers become group-wise thresholding rules. Our approach can also handle multiple fairness constraints (such as equalized odds), and the common scenario when the protected attribute cannot be used at the prediction phase. Leveraging our theoretical results, we design methods that learn fair Bayes-optimal classifiers under bilinear disparity constraints. Our methods cover three popular approaches to fairness-aware classification, via pre-processing (Fair Up- and Down-Sampling), in-processing (Fair Cost-Sensitive Classification) and post-processing (a Fair Plug-In Rule). Our methods control disparity directly while achieving near-optimal fairness-accuracy tradeoffs. We show empirically that our methods compare favorably to existing algorithms. △ Less

Submitted 6 February, 2024; v1 submitted 5 February, 2024; originally announced February 2024.

Comments: This paper replaces the preprint "Bayes-optimal classifiers under group fairness" by Xianli Zeng, Edgar Dobriban, and Guang Cheng (arXiv:2202.09724)

Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Data

Authors: Yue Xing, Xiaofeng Lin, Chenheng Xu, Namjoon Suh, Qifan Song, Guang Cheng

Abstract: Large language models (LLMs) are powerful models that can learn concepts at the inference stage via in-context learning (ICL). While theoretical studies, e.g., \cite{zhang2023trained}, attempt to explain the mechanism of ICL, they assume the input $x_i$ and the output $y_i$ of each demonstration example are in the same token (i.e., structured data). However, in real practice, the examples are usua… ▽ More Large language models (LLMs) are powerful models that can learn concepts at the inference stage via in-context learning (ICL). While theoretical studies, e.g., \cite{zhang2023trained}, attempt to explain the mechanism of ICL, they assume the input $x_i$ and the output $y_i$ of each demonstration example are in the same token (i.e., structured data). However, in real practice, the examples are usually text input, and all words, regardless of their logic relationship, are stored in different tokens (i.e., unstructured data \cite{wibisono2023role}). To understand how LLMs learn from the unstructured data in ICL, this paper studies the role of each component in the transformer architecture and provides a theoretical understanding to explain the success of the architecture. In particular, we consider a simple transformer with one/two attention layers and linear regression tasks for the ICL prediction. We observe that (1) a transformer with two layers of (self-)attentions with a look-ahead attention mask can learn from the prompt in the unstructured data, and (2) positional encoding can match the $x_i$ and $y_i$ tokens to achieve a better ICL performance. △ Less

Submitted 18 June, 2024; v1 submitted 1 February, 2024; originally announced February 2024.

Better Representations via Adversarial Training in Pre-Training: A Theoretical Perspective

Authors: Yue Xing, Xiaofeng Lin, Qifan Song, Yi Xu, Belinda Zeng, Guang Cheng

Abstract: Pre-training is known to generate universal representations for downstream tasks in large-scale deep learning such as large language models. Existing literature, e.g., \cite{kim2020adversarial}, empirically observe that the downstream tasks can inherit the adversarial robustness of the pre-trained model. We provide theoretical justifications for this robustness inheritance phenomenon. Our theoreti… ▽ More Pre-training is known to generate universal representations for downstream tasks in large-scale deep learning such as large language models. Existing literature, e.g., \cite{kim2020adversarial}, empirically observe that the downstream tasks can inherit the adversarial robustness of the pre-trained model. We provide theoretical justifications for this robustness inheritance phenomenon. Our theoretical results reveal that feature purification plays an important role in connecting the adversarial robustness of the pre-trained model and the downstream tasks in two-layer neural networks. Specifically, we show that (i) with adversarial training, each hidden node tends to pick only one (or a few) feature; (ii) without adversarial training, the hidden nodes can be vulnerable to attacks. This observation is valid for both supervised pre-training and contrastive learning. With purified nodes, it turns out that clean training is enough to achieve adversarial robustness in downstream tasks. △ Less

Submitted 26 January, 2024; originally announced January 2024.

Comments: To appear in AISTATS2024

Discovery of a Topological Charge Density Wave

Authors: Maksim Litskevich, Md Shafayat Hossain, Songbo Zhang, Zi-Jia Cheng, Satya N. Guin, Nitesh Kumar, Chandra Shekhar, Zhiwei Wang, Yongkai Li, Guoqing Chang, Jia-Xin Yin, Qi Zhang, Guangming Cheng, Yu-Xiao Jiang, Tyler A. Cochran, Nana Shumiya, Xian P. Yang, Daniel Multer, Xiaoxiong Liu, Nan Yao, Yugui Yao, Claudia Felser, Titus Neupert, M. Zahid Hasan

Abstract: Charge density waves (CDWs) appear in numerous condensed matter platforms, ranging from high-Tc superconductors to quantum Hall systems. Despite such ubiquity, there has been a lack of direct experimental study on boundary states that can uniquely stem from the charge order. Here, using scanning tunneling microscopy, we directly visualize the bulk and boundary phenomenology of CDW in a topological… ▽ More Charge density waves (CDWs) appear in numerous condensed matter platforms, ranging from high-Tc superconductors to quantum Hall systems. Despite such ubiquity, there has been a lack of direct experimental study on boundary states that can uniquely stem from the charge order. Here, using scanning tunneling microscopy, we directly visualize the bulk and boundary phenomenology of CDW in a topological material, Ta2Se8I. Below the transition temperature (TCDW = 260 K), tunneling spectra on an atomically resolved lattice reveal a large insulating gap in the bulk and on the surface, exceeding 500 meV, surpassing predictions from standard weakly-coupled mean-field theory. Spectroscopic imaging confirms the presence of CDW, with LDOS maxima at the conduction band corresponding to the LDOS minima at the valence band, thus revealing a π phase difference in the respective CDW order. Concomitantly, at a monolayer step edge, we detect an in-gap boundary mode with modulations along the edge that match the CDW wavevector along the edge. Intriguingly, the phase of the edge state modulation shifts by π within the charge order gap, connecting the fully gapped bulk (and surface) conduction and valence bands via a smooth energy-phase relation. This bears similarity to the topological spectral flow of edge modes, where the boundary modes bridge the gapped bulk modes in energy and momentum magnitude but in Ta2Se8I, the connectivity distinctly occurs in energy and momentum phase. Notably, our temperature-dependent measurements indicate a vanishing of the insulating gap and the in-gap edge state above TCDW, suggesting their direct relation to CDW. The theoretical analysis also indicates that the observed boundary mode is topological and linked to CDW. △ Less

Submitted 25 January, 2024; originally announced January 2024.

Comments: Nature Physics (2024); in press

On the impact of robot personalization on human-robot interaction: A review

Authors: Jinyu Yang, Camille Vindolet, Julio Rogelio Guadarrama Olvera, Gordon Cheng

Abstract: This study reviews the impact of personalization on human-robot interaction. Firstly, the various strategies used to achieve personalization are briefly described. Secondly, the effects of personalization known to date are discussed. They are presented along with the personalized parameters, personalized features, used technology, and use case they relate to. It is observed that various positive e… ▽ More This study reviews the impact of personalization on human-robot interaction. Firstly, the various strategies used to achieve personalization are briefly described. Secondly, the effects of personalization known to date are discussed. They are presented along with the personalized parameters, personalized features, used technology, and use case they relate to. It is observed that various positive effects have been discussed in the literature while possible negative effects seem to require further investigation. △ Less

Submitted 22 January, 2024; originally announced January 2024.

Report number: CONCATENATE/2023/14 ACM Class: I.2.9

A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models

Authors: Namjoon Suh, Guang Cheng

Abstract: In this article, we review the literature on statistical theories of neural networks from three perspectives. In the first part, results on excess risks for neural networks are reviewed in the nonparametric framework of regression or classification. These results rely on explicit constructions of neural networks, leading to fast convergence rates of excess risks, in that tools from the approximati… ▽ More In this article, we review the literature on statistical theories of neural networks from three perspectives. In the first part, results on excess risks for neural networks are reviewed in the nonparametric framework of regression or classification. These results rely on explicit constructions of neural networks, leading to fast convergence rates of excess risks, in that tools from the approximation theory are adopted. Through these constructions, the width and depth of the networks can be expressed in terms of sample size, data dimension, and function smoothness. Nonetheless, their underlying analysis only applies to the global minimizer in the highly non-convex landscape of deep neural networks. This motivates us to review the training dynamics of neural networks in the second part. Specifically, we review papers that attempt to answer ``how the neural network trained via gradient-based methods finds the solution that can generalize well on unseen data.'' In particular, two well-known paradigms are reviewed: the Neural Tangent Kernel (NTK) paradigm, and Mean-Field (MF) paradigm. In the last part, we review the most recent theoretical advancements in generative models including Generative Adversarial Networks (GANs), diffusion models, and in-context learning (ICL) in the Large Language Models (LLMs). The former two models are known to be the main pillars of the modern generative AI era, while ICL is a strong capability of LLMs in learning from a few examples in the context. Finally, we conclude the paper by suggesting several promising directions for deep learning theory. △ Less

Submitted 4 July, 2024; v1 submitted 13 January, 2024; originally announced January 2024.

Comments: 33 pages, no figures,Invited for review in Annual Review of Statistics and Its Application (In review)

Legendre-Moment Transform for Linear Ensemble Control and Computation

Authors: Xin Ning, Gong Cheng, Wei Zhang, Jr-Shin Li

Abstract: Ensemble systems, pervasive in diverse scientific and engineering domains, pose challenges to existing control methods due to their massive scale and underactuated nature. This paper presents a dynamic moment approach to addressing theoretical and computational challenges in systems-theoretic analysis and control design for linear ensemble systems. We introduce the Legendre-moments and Legendre-mo… ▽ More Ensemble systems, pervasive in diverse scientific and engineering domains, pose challenges to existing control methods due to their massive scale and underactuated nature. This paper presents a dynamic moment approach to addressing theoretical and computational challenges in systems-theoretic analysis and control design for linear ensemble systems. We introduce the Legendre-moments and Legendre-moment transform, which maps an ensemble system defined on the $L^2$-space to a Legendre-moment system defined on the $\ell^2$-space. We show that this pair of systems is of one-to-one correspondence and shares the same controllability property. This equivalence admits the control of an ensemble system through the control of the corresponding Legendre-moment system and inspires a unified control design scheme for linear ensemble systems using structured truncated moment systems. In particular, we develop a sampling-free ensemble control design algorithm, then conduct error analysis for control design using truncated moment systems and derive error bounds with respect to the truncation orders, which are illustrated with numerical examples. △ Less

Submitted 3 January, 2024; originally announced January 2024.

MSC Class: 93B05; 93B28; 93B51

Graph Elimination Networks

Authors: Shuo Wang, Ge Cheng, Yun Zhang

Abstract: Graph Neural Networks (GNNs) are widely applied across various domains, yet they perform poorly in deep layers. Existing research typically attributes this problem to node over-smoothing, where node representations become indistinguishable after multiple rounds of propagation. In this paper, we delve into the neighborhood propagation mechanism of GNNs and discover that the real root cause of GNNs'… ▽ More Graph Neural Networks (GNNs) are widely applied across various domains, yet they perform poorly in deep layers. Existing research typically attributes this problem to node over-smoothing, where node representations become indistinguishable after multiple rounds of propagation. In this paper, we delve into the neighborhood propagation mechanism of GNNs and discover that the real root cause of GNNs' performance degradation in deep layers lies in ineffective neighborhood feature propagation. This propagation leads to an exponential growth of a node's current representation at every propagation step, making it extremely challenging to capture valuable dependencies between long-distance nodes. To address this issue, we introduce Graph Elimination Networks (GENs), which employ a specific algorithm to eliminate redundancies during neighborhood propagation. We demonstrate that GENs can enhance nodes' perception of distant neighborhoods and extend the depth of network propagation. Extensive experiments show that GENs outperform the state-of-the-art methods on various graph-level and node-level datasets. △ Less

Submitted 2 January, 2024; originally announced January 2024.

Comments: Includes 8 pages of main text and 4 pages of appendices

Downstream Task-Oriented Generative Model Selections on Synthetic Data Training for Fraud Detection Models

Authors: Yinan Cheng, Chi-Hua Wang, Vamsi K. Potluru, Tucker Balch, Guang Cheng

Abstract: Devising procedures for downstream task-oriented generative model selections is an unresolved problem of practical importance. Existing studies focused on the utility of a single family of generative models. They provided limited insights on how synthetic data practitioners select the best family generative models for synthetic training tasks given a specific combination of machine learning model… ▽ More Devising procedures for downstream task-oriented generative model selections is an unresolved problem of practical importance. Existing studies focused on the utility of a single family of generative models. They provided limited insights on how synthetic data practitioners select the best family generative models for synthetic training tasks given a specific combination of machine learning model class and performance metric. In this paper, we approach the downstream task-oriented generative model selections problem in the case of training fraud detection models and investigate the best practice given different combinations of model interpretability and model performance constraints. Our investigation supports that, while both Neural Network(NN)-based and Bayesian Network(BN)-based generative models are both good to complete synthetic training task under loose model interpretability constrain, the BN-based generative models is better than NN-based when synthetic training fraud detection model under strict model interpretability constrain. Our results provides practical guidance for machine learning practitioner who is interested in replacing their training dataset from real to synthetic, and shed lights on more general downstream task-oriented generative model selection problems. △ Less

Submitted 1 January, 2024; originally announced January 2024.

Comments: The following article has been accepted by ICAIF22, Synthetic Data for AI in Finance; see https://sites.google.com/view/icaif-synthetic-2022/program