Searcharxiv⌕ Search

arXiv subjects

Kai Wang

Publications and source records attributed to Kai Wang.

At least 505 records · Page 28Linked to original sources
arXiv2023

Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion

Automated creation of synthetic traffic scenarios is a key part of validating the safety of autonomous vehicles (AVs). In this paper, we propose Scenario Diffusion, a novel diffusion-based architecture for generating traffic scenarios that enables controllable scenario generation. We combine latent diffusion, object detection and trajectory regression to generate distributions of synthetic agent poses, orientations and trajectories simultaneously. To provide additional control over the generated scenario, this distribution is conditioned on a map and sets of tokens describing the desired scenario. We show that our approach has sufficient expressive capacity to model diverse traffic patterns and generalizes to different geographical regions.

cs.LG↗
arXiv2023

Towards Long-term Annotators: A Supervised Label Aggregation Baseline

Relying on crowdsourced workers, data crowdsourcing platforms are able to efficiently provide vast amounts of labeled data. Due to the variability in the annotation quality of crowd workers, modern techniques resort to redundant annotations and subsequent label aggregation to infer true labels. However, these methods require model updating during the inference, posing challenges in real-world implementation. Meanwhile, in recent years, many data labeling tasks have begun to require skilled and experienced annotators, leading to an increasing demand for long-term annotators. These annotators could leave substantial historical annotation records on the crowdsourcing platforms, which can benefit label aggregation, but are ignored by previous works. Hereby, in this paper, we propose a novel label aggregation technique, which does not need any model updating during inference and can extensively explore the historical annotation records. We call it SuperLA, a Supervised Label Aggregation method. Inside this model, we design three types of input features and a straightforward neural network structure to merge all the information together and subsequently produce aggregated labels. Based on comparison experiments conducted on 22 public datasets and 11 baseline methods, we find that SuperLA not only outperforms all those baselines in inference performance but also offers significant advantages in terms of efficiency.

cs.HC↗
arXiv2023

PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning

Classical federated learning (FL) enables training machine learning models without sharing data for privacy preservation, but heterogeneous data characteristic degrades the performance of the localized model. Personalized FL (PFL) addresses this by synthesizing personalized models from a global model via training on local data. Such a global model may overlook the specific information that the clients have been sampled. In this paper, we propose a novel scheme to inject personalized prior knowledge into the global model in each client, which attempts to mitigate the introduced incomplete information problem in PFL. At the heart of our proposed approach is a framework, the PFL with Bregman Divergence (pFedBreD), decoupling the personalized prior from the local objective function regularized by Bregman divergence for greater adaptability in personalized scenarios. We also relax the mirror descent (RMD) to extract the prior explicitly to provide optional strategies. Additionally, our pFedBreD is backed up by a convergence analysis. Sufficient experiments demonstrate that our method reaches the state-of-the-art performances on 5 datasets and outperforms other methods by up to 3.5% across 8 benchmarks. Extensive analyses verify the robustness and necessity of proposed designs.

cs.LG↗
arXiv2023

Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT

We hypothesize that large language models (LLMs) based on the transformer architecture can enable automated detection of clinical phenotype terms, including terms not documented in the HPO. In this study, we developed two types of models: PhenoBCBERT, a BERT-based model, utilizing Bio+Clinical BERT as its pre-trained model, and PhenoGPT, a GPT-based model that can be initialized from diverse GPT models, including open-source versions such as GPT-J, Falcon, and LLaMA, as well as closed-source versions such as GPT-3 and GPT-3.5. We compared our methods with PhenoTagger, a recently developed HPO recognition tool that combines rule-based and deep learning methods. We found that our methods can extract more phenotype concepts, including novel ones not characterized by HPO. We also performed case studies on biomedical literature to illustrate how new phenotype information can be recognized and extracted. We compared current BERT-based versus GPT-based models for phenotype tagging, in multiple aspects including model architecture, memory usage, speed, accuracy, and privacy protection. We also discussed the addition of a negation step and an HPO normalization layer to the transformer models for improved HPO term tagging. In conclusion, PhenoBCBERT and PhenoGPT enable the automated discovery of phenotype terms from clinical notes and biomedical literature, facilitating automated downstream tasks to derive new biological insights on human diseases.

q-bio.QM↗
arXiv2023

High-energy Neutrino Emission Associated with GWs from Binary Black Hole Mergers in AGN Accretion Discs

The search for multi-messenger signals of binary black hole (BBH) mergers is crucial to understanding the merger process of BBH and the relative astrophysical environment. Considering BBH mergers occurring in the active galactic nuclei (AGN) accretion disks, we focus on the accompanying high-energy neutrino production from the interaction between the jet launched by the post-merger remnant BH and disk materials. Particles can be accelerated by the shocks generated from the jet-disk interaction and subsequently interact with the disk gas and radiations to produce high-energy neutrinos through hadronic processes. We demonstrate that the identification of the high-energy neutrino signal from BBH merger in AGN disks is feasible. In addition, the joint BBH gravitational wave (GW) and neutrino detection rate is derived, which can be used to constrain the BBH merger rate and the accretion rate of the remnant BH based on the future associated detections of GWs and neutrinos. To date, an upper limit of BBH merger rate density in AGN disks of $R_0 \lesssim 3\,\rm Gpc^{-3} yr^{-1}$ is derived for the fiducial parameter values based on the current null association of GWs and neutrinos.

astro-ph.HE↗
arXiv2023

Branch-and-Price for Prescriptive Contagion Analytics

Predictive contagion models are ubiquitous in epidemiology, social sciences, engineering, and management. This paper formulates a prescriptive contagion analytics model where a decision-maker allocates shared resources across multiple segments of a population, each governed by continuous-time dynamics. We define four real-world problems under this umbrella: vaccine distribution, vaccination centers deployment, content promotion, and congestion mitigation. These problems feature a large-scale mixed-integer non-convex optimization structure with constraints governed by ordinary differential equations, combining the challenges of discrete optimization, non-linear optimization, and continuous-time system dynamics. This paper develops a branch-and-price methodology for prescriptive contagion analytics based on: (i) a set partitioning reformulation; (ii) a column generation decomposition; (iii) a state-clustering algorithm for discrete-decision continuous-state dynamic programming; and (iv) a tri-partite branching scheme to circumvent non-linearities. Extensive experiments show that the algorithm scales to very large and otherwise-intractable instances, outperforming state-of-the-art benchmarks. Our methodology provides practical benefits in contagion systems; in particular, it can increase the effectiveness of a vaccination campaign by an estimated 12-70%, resulting in 7,000 to 12,000 extra saved lives over a three-month horizon mirroring the COVID-19 pandemic. We provide an open-source implementation of the methodology in an online repository to enable replication.

math.OC↗
arXiv2023

DREAM+: Efficient Dataset Distillation by Bidirectional Representative Matching

Dataset distillation plays a crucial role in creating compact datasets with similar training performance compared with original large-scale ones. This is essential for addressing the challenges of data storage and training costs. Prevalent methods facilitate knowledge transfer by matching the gradients, embedding distributions, or training trajectories of synthetic images with those of the sampled original images. Although there are various matching objectives, currently the strategy for selecting original images is limited to naive random sampling. We argue that random sampling overlooks the evenness of the selected sample distribution, which may result in noisy or biased matching targets. Besides, the sample diversity is also not constrained by random sampling. Additionally, current methods predominantly focus on single-dimensional matching, where information is not fully utilized. To address these challenges, we propose a novel matching strategy called Dataset Distillation by Bidirectional REpresentAtive Matching (DREAM+), which selects representative original images for bidirectional matching. DREAM+ is applicable to a variety of mainstream dataset distillation frameworks and significantly reduces the number of distillation iterations by more than 15 times without affecting performance. Given sufficient training time, DREAM+ can further improve the performance and achieve state-of-the-art results. We have released the code at github.com/NUS-HPC-AI-Lab/DREAM+.

cs.CV↗
arXiv2023

InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning

Data pruning aims to obtain lossless performances with less overall cost. A common approach is to filter out samples that make less contribution to the training. This could lead to gradient expectation bias compared to the original data. To solve this problem, we propose \textbf{InfoBatch}, a novel framework aiming to achieve lossless training acceleration by unbiased dynamic data pruning. Specifically, InfoBatch randomly prunes a portion of less informative samples based on the loss distribution and rescales the gradients of the remaining samples to approximate the original gradient. As a plug-and-play and architecture-agnostic framework, InfoBatch consistently obtains lossless training results on classification, semantic segmentation, vision pertaining, and instruction fine-tuning tasks. On CIFAR10/100, ImageNet-1K, and ADE20K, InfoBatch losslessly saves 40\% overall cost. For pertaining MAE and diffusion model, InfoBatch can respectively save 24.8\% and 27\% cost. For LLaMA instruction fine-tuning, InfoBatch is also able to save 20\% cost and is compatible with coreset selection methods. The code is publicly available at \href{https://github.com/henryqin1997/InfoBatch}{github.com/NUS-HPC-AI-Lab/InfoBatch}.

cs.CV↗
arXiv2023

A Conditional Abundance Matching Method of Extending Simulated Halo Merger Trees to Resolve Low-Mass Progenitors and Sub-halos

We present an algorithm to extend subhalo merger trees in a low-resolution dark-matter-only simulation by conditionally matching them to those in a high-resolution simulation. The algorithm is general and can be applied to simulation data with different resolutions using different target variables. We instantiate the algorithm by a case in which trees from ELUCID, a constrained simulation of $(500h^{-1}{\rm Mpc})^3$ volume of the local universe, are extended by matching trees from TNGDark, a simulation with much higher resolution. Our tests show that the extended trees are statistically equivalent to the high-resolution trees in the joint distribution of subhalo quantities and in important summary statistics relevant to modeling galaxy formation and evolution in halos. The extended trees preserve certain information of individual systems in the target simulation, including properties of resolved satellite subhalos, and shapes and orientations of their host halos. With the extension, subhalo merger trees in a cosmological scale simulation are extrapolated to a mass resolution comparable to that in a higher-resolution simulation carried out in a smaller volume, which can be used as the input for (sub)halo-based models of galaxy formation. The source code of the algorithm, and halo merger trees extended to a mass resolution of $\sim 2 \times 10^8 h^{-1}M_\odot$ in the entire ELUCID simulation, are available.

astro-ph.GA↗
arXiv2023

Massive Dark Matter Halos at High Redshift: Implications for Observations in the JWST Era

The presence of massive galaxies at high $z$ as recently observed by JWST appears to contradict the current $Λ$CDM cosmology. Here we aim to alleviate this tension by incorporating uncertainties from three sources in counting galaxies: cosmic variance, error in stellar mass estimation, and backsplash enhancement. Each of these factors significantly increases the cumulative stellar mass density $ρ_*(>M_*)$ at the high-mass end, and their combined effect can boost the density by more than one order of magnitude. Assuming a star formation efficiency of $ε_* \sim 0.5$, cosmic variance alone reduces the tension to a $2σ$ level, except for the most massive galaxy at $z=8$. Additionally, incorporating a 0.3 dex lognormal dispersion in the stellar mass estimation brings the observed $ρ_*(>M_*)$ at $z \sim 7 - 10$ within $2σ$. The tension is completely eliminated when we account for the gas stripped from backsplash halos. These results highlight the importance of fully modeling uncertainties when interpreting observational data of rare objects. We use the constrained simulation, ELUCID, to investigate the descendants of high-$z$ massive galaxies. Our findings reveal that a significant portion of these galaxies ultimately reside in massive halos at $z=0$ with $M_{\rm halo} > 10^{13} h^{-1}M_\odot$. Moreover, a large fraction of local central galaxies in $M_{\rm halo} \geqslant 10^{14.5} h^{-1}M_\odot$ halos are predicted to contain substantial amounts of ancient stars formed in massive galaxies at $z \sim 8$. This prediction can be tested by studying the structure and stellar population of central galaxies in present-day massive clusters.

astro-ph.GA↗
arXiv2023

Evidence of Hadronic Emission from the brightest-of-all-time GRB 221009A

Acceleration of hadrons in relativistic shocks has been long expected and invoked to model GRB high-energy photon and neutrino emissions. However, so far there has been no direct observational evidence of hadronic emission from GRBs. The B.O.A.T. ("brightest of all time") gamma-ray burst (GRB) 221009A had extreme energies (with an isotropic energy exceeding $10^{55}$ erg) and was detected in broad-band including the very-high-energy (VHE, $>100\,\rm GeV$) band up to $>10$ TeV. Here we perform a comprehensive spectral analysis of the GRB from keV to TeV energy range and perform detailed spectral and light curve modelings considering both the traditional synchrotron self-Compton process and the electromagnetic (EM) cascade process initiated by hadronic interactions by accelerated cosmic rays in the external shock. We find that the leptonic scenario alone is not adequate to account for the observations, whereas the proposed scenario with the combination of hadronic and leptonic components can well reproduce the multi-wavelength spectra and the light curve. This result reveals the existence of the accelerated hadronic component in the early afterglow of this extreme burst. According to this scenario, the observed TeV light curve should contain imprints of the prompt MeV emission.

astro-ph.HE↗
arXiv2023

From Halos to Galaxies. VII. The Connections Between Stellar Mass Growth History, Quenching History and Halo Assembly History for Central Galaxies

The assembly of galaxies over cosmic time is tightly connected to the assembly of their host dark matter halos. We investigate the stellar mass growth history and the chemical enrichment history of central galaxies in SDSS-MaNGA. We find that the derived stellar metallicity of passive central galaxies is always higher than that of the star-forming ones. This stellar metallicity enhancement becomes progressively larger towards low-mass galaxies (at a given epoch) and earlier epochs (at a given stellar mass), which suggests strangulation as the primary mechanism for star formation quenching in central galaxies not only in the local universe, but also very likely at higher redshifts up to $z\sim3$. We show that at the same present-day stellar mass, passive central galaxies assembled half of their final stellar mass $\sim 2$ Gyr earlier than star-forming central galaxies, which agrees well with semi-analytic model. Exploring semi-analytic model, we find that this is because passive central galaxies reside in, on average, more massive halos with a higher halo mass increase rate across cosmic time. As a consequence, passive central galaxies are assembled faster and also quenched earlier than their star-forming counterparts. While at the same present-day halo mass, different halo assembly history also produces very different final stellar mass of the central galaxy within, and halos assembled earlier host more massive centrals with a higher quenched fraction, in particular around the "golden halo mass" at $10^{12}\mathrm{M_\odot}$. Our results call attention back to the dark matter halo as a key driver of galaxy evolution.

astro-ph.GA↗
arXiv2023

Does Graph Distillation See Like Vision Dataset Counterpart?

Training on large-scale graphs has achieved remarkable results in graph representation learning, but its cost and storage have attracted increasing concerns. Existing graph condensation methods primarily focus on optimizing the feature matrices of condensed graphs while overlooking the impact of the structure information from the original graphs. To investigate the impact of the structure information, we conduct analysis from the spectral domain and empirically identify substantial Laplacian Energy Distribution (LED) shifts in previous works. Such shifts lead to poor performance in cross-architecture generalization and specific tasks, including anomaly detection and link prediction. In this paper, we propose a novel Structure-broadcasting Graph Dataset Distillation (SGDD) scheme for broadcasting the original structure information to the generation of the synthetic one, which explicitly prevents overlooking the original structure information. Theoretically, the synthetic graphs by SGDD are expected to have smaller LED shifts than previous works, leading to superior performance in both cross-architecture settings and specific tasks. We validate the proposed SGDD across 9 datasets and achieve state-of-the-art results on all of them: for example, on the YelpChi dataset, our approach maintains 98.6% test accuracy of training on the original graph dataset with 1,000 times saving on the scale of the graph. Moreover, we empirically evaluate there exist 17.6% ~ 31.4% reductions in LED shift crossing 9 datasets. Extensive experiments and analysis verify the effectiveness and necessity of the proposed designs. The code is available in the GitHub repository: https://github.com/RingBDStack/SGDD.

cs.LG↗
arXiv2023

DiM: Distilling Dataset into Generative Model

Dataset distillation reduces the network training cost by synthesizing small and informative datasets from large-scale ones. Despite the success of the recent dataset distillation algorithms, three drawbacks still limit their wider application: i). the synthetic images perform poorly on large architectures; ii). they need to be re-optimized when the distillation ratio changes; iii). the limited diversity restricts the performance when the distillation ratio is large. In this paper, we propose a novel distillation scheme to \textbf{D}istill information of large train sets \textbf{i}nto generative \textbf{M}odels, named DiM. Specifically, DiM learns to use a generative model to store the information of the target dataset. During the distillation phase, we minimize the differences in logits predicted by a models pool between real and generated images. At the deployment stage, the generative model synthesizes various training samples from random noises on the fly. Due to the simple yet effective designs, the trained DiM can be directly applied to different distillation ratios and large architectures without extra cost. We validate the proposed DiM across 4 datasets and achieve state-of-the-art results on all of them. To the best of our knowledge, we are the first to achieve higher accuracy on complex architectures than simple ones, such as 75.1\% with ResNet-18 and 72.6\% with ConvNet-3 on ten images per class of CIFAR-10. Besides, DiM outperforms previous methods with 10\% $\sim$ 22\% when images per class are 1 and 10 on the SVHN dataset.

cs.CV↗
arXiv2023

Expanding Small-Scale Datasets with Guided Imagination

The power of DNNs relies heavily on the quantity and quality of training data. However, collecting and annotating data on a large scale is often expensive and time-consuming. To address this issue, we explore a new task, termed dataset expansion, aimed at expanding a ready-to-use small dataset by automatically creating new labeled samples. To this end, we present a Guided Imagination Framework (GIF) that leverages cutting-edge generative models like DALL-E2 and Stable Diffusion (SD) to "imagine" and create informative new data from the input seed data. Specifically, GIF conducts data imagination by optimizing the latent features of the seed data in the semantically meaningful space of the prior model, resulting in the creation of photo-realistic images with new content. To guide the imagination towards creating informative samples for model training, we introduce two key criteria, i.e., class-maintained information boosting and sample diversity promotion. These criteria are verified to be essential for effective dataset expansion: GIF-SD obtains 13.5% higher model accuracy on natural image datasets than unguided expansion with SD. With these essential criteria, GIF successfully expands small datasets in various scenarios, boosting model accuracy by 36.9% on average over six natural image datasets and by 13.5% on average over three medical datasets. The source code is available at https://github.com/Vanint/DatasetExpansion.

cs.CV↗
arXiv2023

Can pre-trained models assist in dataset distillation?

Dataset Distillation (DD) is a prominent technique that encapsulates knowledge from a large-scale original dataset into a small synthetic dataset for efficient training. Meanwhile, Pre-trained Models (PTMs) function as knowledge repositories, containing extensive information from the original dataset. This naturally raises a question: Can PTMs effectively transfer knowledge to synthetic datasets, guiding DD accurately? To this end, we conduct preliminary experiments, confirming the contribution of PTMs to DD. Afterwards, we systematically study different options in PTMs, including initialization parameters, model architecture, training epoch and domain knowledge, revealing that: 1) Increasing model diversity enhances the performance of synthetic datasets; 2) Sub-optimal models can also assist in DD and outperform well-trained ones in certain cases; 3) Domain-specific PTMs are not mandatory for DD, but a reasonable domain match is crucial. Finally, by selecting optimal options, we significantly improve the cross-architecture generalization over baseline DD methods. We hope our work will facilitate researchers to develop better DD techniques. Our code is available at https://github.com/yaolu-zjut/DDInterpreter.

cs.CV↗
arXiv2023

Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing

Large-scale text-to-image generative models have been a ground-breaking development in generative AI, with diffusion models showing their astounding ability to synthesize convincing images following an input text prompt. The goal of image editing research is to give users control over the generated images by modifying the text prompt. Current image editing techniques are susceptible to unintended modifications of regions outside the targeted area, such as on the background or on distractor objects which have some semantic or visual relationship with the targeted object. According to our experimental findings, inaccurate cross-attention maps are at the root of this problem. Based on this observation, we propose Dynamic Prompt Learning (DPL) to force cross-attention maps to focus on correct noun words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, we achieve fine-grained image editing over particular objects while preventing undesired changes to other image regions. Our method DPL, based on the publicly available Stable Diffusion, is extensively evaluated on a wide range of images, and consistently obtains superior results both quantitatively (CLIP score, Structure-Dist) and qualitatively (on user-evaluation). We show improved prompt editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.

cs.CV↗
arXiv2023

Computational Design of Wiring Layout on Tight Suits with Minimal Motion Resistance

An increasing number of electronics are directly embedded on the clothing to monitor human status (e.g., skeletal motion) or provide haptic feedback. A specific challenge to prototype and fabricate such a clothing is to design the wiring layout, while minimizing the intervention to human motion. We address this challenge by formulating the topological optimization problem on the clothing surface as a deformation-weighted Steiner tree problem on a 3D clothing mesh. Our method proposed an energy function for minimizing strain energy in the wiring area under different motions, regularized by its total length. We built the physical prototype to verify the effectiveness of our method and conducted user study with participants of both design experts and smart cloth users. On three types of commercial products of smart clothing, the optimized layout design reduced wire strain energy by an average of 77% among 248 actions compared to baseline design, and 18% over the expert design.

cs.HC↗