arXiv subjects

Tong Wu

Publications and source records attributed to Tong Wu.

At least 163 records · Page 9Linked to original sources

arXiv2024

FiVA: Fine-grained Visual Attribute Dataset for Text-to-Image Diffusion Models

Recent advances in text-to-image generation have enabled the creation of high-quality images with diverse applications. However, accurately describing desired visual attributes can be challenging, especially for non-experts in art and photography. An intuitive solution involves adopting favorable attributes from the source images. Current methods attempt to distill identity and style from source images. However, "style" is a broad concept that includes texture, color, and artistic elements, but does not cover other important attributes such as lighting and dynamics. Additionally, a simplified "style" adaptation prevents combining multiple attributes from different sources into one generated image. In this work, we formulate a more effective approach to decompose the aesthetics of a picture into specific visual attributes, allowing users to apply characteristics such as lighting, texture, and dynamics from different images. To achieve this goal, we constructed the first fine-grained visual attributes dataset (FiVA) to the best of our knowledge. This FiVA dataset features a well-organized taxonomy for visual attributes and includes around 1 M high-quality generated images with visual attribute annotations. Leveraging this dataset, we propose a fine-grained visual attribute adaptation framework (FiVA-Adapter), which decouples and adapts visual attributes from one or more source images into a generated one. This approach enhances user-friendly customization, allowing users to selectively apply desired attributes to create images that meet their unique preferences and specific content requirements.

cs.CV↗

arXiv2024

On the Exact Maxwell evolution equation of resonator dynamics

In a recent publication [Opt. Express 32, 20904 (2024)], the accuracy of the main evolution equation that governs resonator dynamics in the coupled-mode theory (CMT) was questioned. The study concluded that the driving force is proportional to the temporal derivative of the excitation field rather than the excitation field itself. This conclusion was reached with a derivation of an "exact" Maxwell evolution (EME) equation obtained directly from Maxwell's equations, which was further supported by extensive numerical tests. Hereafter, we argue that the original derivation lacks mathematical rigor. We present a direct and rigorous derivation that establishes a solid mathematical foundation for the EME equation. This new approach clarifies the origin of the temporal derivative in the excitation term of CMT and elucidates the approximations present in the classical CMT evolution equation through a straightforward argument.

physics.optics↗

arXiv2024

SimC3D: A Simple Contrastive 3D Pretraining Framework Using RGB Images

The 3D contrastive learning paradigm has demonstrated remarkable performance in downstream tasks through pretraining on point cloud data. Recent advances involve additional 2D image priors associated with 3D point clouds for further improvement. Nonetheless, these existing frameworks are constrained by the restricted range of available point cloud datasets, primarily due to the high costs of obtaining point cloud data. To this end, we propose SimC3D, a simple but effective 3D contrastive learning framework, for the first time, pretraining 3D backbones from pure RGB image data. SimC3D performs contrastive 3D pretraining with three appealing properties. (1) Pure image data: SimC3D simplifies the dependency of costly 3D point clouds and pretrains 3D backbones using solely RBG images. By employing depth estimation and suitable data processing, the monocular synthesized point cloud shows great potential for 3D pretraining. (2) Simple framework: Traditional multi-modal frameworks facilitate 3D pretraining with 2D priors by utilizing an additional 2D backbone, thereby increasing computational expense. In this paper, we empirically demonstrate that the primary benefit of the 2D modality stems from the incorporation of locality information. Inspired by this insightful observation, SimC3D directly employs 2D positional embeddings as a stronger contrastive objective, eliminating the necessity for 2D backbones and leading to considerable performance improvements. (3) Strong performance: SimC3D outperforms previous approaches that leverage ground-truth point cloud data for pretraining in various downstream tasks. Furthermore, the performance of SimC3D can be further enhanced by combining multiple image datasets, showcasing its significant potential for scalability. The code will be available at https://github.com/Dongjiahua/SimC3D.

cs.CV↗

arXiv2024

Imagine360: Immersive 360 Video Generation from Perspective Anchor

$360^\circ$ videos offer a hyper-immersive experience that allows the viewers to explore a dynamic scene from full 360 degrees. To achieve more user-friendly and personalized content creation in $360^\circ$ video format, we seek to lift standard perspective videos into $360^\circ$ equirectangular videos. To this end, we introduce Imagine360, the first perspective-to-$360^\circ$ video generation framework that creates high-quality $360^\circ$ videos with rich and diverse motion patterns from video anchors. Imagine360 learns fine-grained spherical visual and motion patterns from limited $360^\circ$ video data with several key designs. 1) Firstly we adopt the dual-branch design, including a perspective and a panorama video denoising branch to provide local and global constraints for $360^\circ$ video generation, with motion module and spatial LoRA layers fine-tuned on extended web $360^\circ$ videos. 2) Additionally, an antipodal mask is devised to capture long-range motion dependencies, enhancing the reversed camera motion between antipodal pixels across hemispheres. 3) To handle diverse perspective video inputs, we propose elevation-aware designs that adapt to varying video masking due to changing elevations across frames. Extensive experiments show Imagine360 achieves superior graphics quality and motion coherence among state-of-the-art $360^\circ$ video generation methods. We believe Imagine360 holds promise for advancing personalized, immersive $360^\circ$ video creation.

cs.CV↗

arXiv2024

Discussions on the spatial exponential growth of electromagnetic quasinormal modes

The temporal response of open systems is marked by damped oscillations. These oscillations, often referred to as ringings, are the signature of the decay of quasinormal modes (QNMs). A major research objective across various fields is to represent the response of open systems using QNM expansions, akin to the treatment of normal modes in closed systems. In electromagnetism, it is widely acknowledged that QNM expansions provide a relevant representation of the modal physics within the interior of compact resonators in free space, where QNMs form a complete set of the resonator. However, challenges emerge in the exterior of the resonator, where QNM fields exhibit exponential divergence, rendering QNM expansions incomplete. The divergence poses delicate mathematical issues that often lead to misinterpretations on the physics side. Hereafter, we analyze foundational concepts such as cavity perturbation theory and dissipative coupling between resonators. By studying model problems, we show that the exponential growth is physical and meaningful for understanding the interaction between remote electromagnetic bodies. The analysis consistently reveals that the coupling coefficients between QNMs of two distant bodies strengthen as the separation distance increases, therein challenging the intuition that distant bodies behave independently. These insights that shed light on the origin and implications of the divergence hold significant implications for understanding the ability of contemporary electromagnetic QNM theories in offering a modal representation of the physics in the open space surrounding resonator bodies. Our critical examination of these theories reveals the existence of two distinct perspectives and elucidates the significance of their contrasting viewpoints.

physics.optics↗

arXiv2024

Multistage spatial model for informing release of Wolbachia-infected mosquitoes as disease control

Wolbachia is a naturally occurring bacterium that can infect Aedes mosquitoes and reduce the transmission of mosquito-borne diseases, including dengue fever, Zika, and chikungunya. Field trials have been conducted worldwide to suppress local epidemics. We introduce a novel partial differential equation model to simulate the spread of Wolbachia infection in mosquito populations. Our model incorporates the intricate Wolbachia maternal transmission cycle and detailed mosquito life stages, while also accounting for the spatial heterogeneity induced by mosquito dispersion across a two-dimensional domain. Prior modeling studies and field data indicate that a critical threshold of Wolbachia-infected mosquitoes is necessary for infection to persist among the mosquito population. Through our spatial model, we identify a threshold condition, termed the ``critical bubble'', for having a self-sustainable Wolbachia infection in the field. When releasing beyond this threshold, the model predicts a spatial wave of Wolbachia infection. We further quantify how this threshold and infection wave velocity depend on the diffusion process and other parameters. We numerically study various intervention scenarios to inform efficient Wolbachia release strategies. Our findings suggest that: (1) integrating Wolbachia release with pre-release mitigations targeting the adult mosquitoes, rather than the aquatic stages, better reduces the threshold for Wolbachia establishment; Habitats modification before the release may increase the threshold; (2) releases in the dry regions lower the threshold, though the infection waves may slow down or stall at the dry-wet interfaces due to the difference in carrying capacities; and (3) initiating releases just before the wet season further reduces the release threshold.

q-bio.PE↗

arXiv2024

DepthSSC: Monocular 3D Semantic Scene Completion via Depth-Spatial Alignment and Voxel Adaptation

The task of 3D semantic scene completion using monocular cameras is gaining significant attention in the field of autonomous driving. This task aims to predict the occupancy status and semantic labels of each voxel in a 3D scene from partial image inputs. Despite numerous existing methods, many face challenges such as inaccurately predicting object shapes and misclassifying object boundaries. To address these issues, we propose DepthSSC, an advanced method for semantic scene completion using only monocular cameras. DepthSSC integrates the Spatial Transformation Graph Fusion (ST-GF) module with Geometric-Aware Voxelization (GAV), enabling dynamic adjustment of voxel resolution to accommodate the geometric complexity of 3D space. This ensures precise alignment between spatial and depth information, effectively mitigating issues such as object boundary distortion and incorrect depth perception found in previous methods. Evaluations on the SemanticKITTI and SSCBench-KITTI-360 dataset demonstrate that DepthSSC not only captures intricate 3D structural details effectively but also achieves state-of-the-art performance.

cs.CV↗

arXiv2024

Robust Optimal Power Flow Against Adversarial Attacks: A Tri-Level Optimization Approach

In power systems, unpredictable events like extreme weather, equipment failures, and cyberattacks present significant challenges to ensuring safety and reliability. Ensuring resilience in the face of these uncertainties is crucial for reliable and efficient operations. This paper presents a tri-level optimization approach for robust power system operations that effectively address worst-case attacks. The first stage focuses on optimizing economic dispatch under normal operating conditions, aiming to minimize generation costs while maintaining the supply-demand balance. The second stage introduces an adversarial attack model, identifying worst-case scenarios that maximize the system's vulnerability by targeting distributed generation (DG). In the third stage, mitigation strategies are developed using fast-response energy storage systems (ESS) to minimize disruptions caused by these attacks. By integrating economic dispatch, vulnerability assessment, and mitigation into a unified framework, this approach provides a robust solution for enhancing power system resilience and safety against evolving adversarial threats. The approach is validated using the IEEE-33 node distribution system to demonstrate its effectiveness in achieving both cost efficiency and system resilience.

eess.SY↗

arXiv2024

PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications

Developing intelligent pediatric consultation systems offers promising prospects for improving diagnostic efficiency, especially in China, where healthcare resources are scarce. Despite recent advances in Large Language Models (LLMs) for Chinese medicine, their performance is sub-optimal in pediatric applications due to inadequate instruction data and vulnerable training procedures. To address the above issues, this paper builds PedCorpus, a high-quality dataset of over 300,000 multi-task instructions from pediatric textbooks, guidelines, and knowledge graph resources to fulfil diverse diagnostic demands. Upon well-designed PedCorpus, we propose PediatricsGPT, the first Chinese pediatric LLM assistant built on a systematic and robust training pipeline. In the continuous pre-training phase, we introduce a hybrid instruction pre-training mechanism to mitigate the internal-injected knowledge inconsistency of LLMs for medical domain adaptation. Immediately, the full-parameter Supervised Fine-Tuning (SFT) is utilized to incorporate the general medical knowledge schema into the models. After that, we devise a direct following preference optimization to enhance the generation of pediatrician-like humanistic responses. In the parameter-efficient secondary SFT phase, a mixture of universal-specific experts strategy is presented to resolve the competency conflict between medical generalist and pediatric expertise mastery. Extensive results based on the metrics, GPT-4, and doctor evaluations on distinct doctor downstream tasks show that PediatricsGPT consistently outperforms previous Chinese medical LLMs. Our model and dataset will be open-source for community development.

cs.CL↗

arXiv2024

MotionClone: Training-Free Motion Cloning for Controllable Video Generation

Motion-based controllable video generation offers the potential for creating captivating visual content. Existing methods typically necessitate model training to encode particular motion cues or incorporate fine-tuning to inject certain motion patterns, resulting in limited flexibility and generalization. In this work, we propose MotionClone, a training-free framework that enables motion cloning from reference videos to versatile motion-controlled video generation, including text-to-video and image-to-video. Based on the observation that the dominant components in temporal-attention maps drive motion synthesis, while the rest mainly capture noisy or very subtle motions, MotionClone utilizes sparse temporal attention weights as motion representations for motion guidance, facilitating diverse motion transfer across varying scenarios. Meanwhile, MotionClone allows for the direct extraction of motion representation through a single denoising step, bypassing the cumbersome inversion processes and thus promoting both efficiency and flexibility. Extensive experiments demonstrate that MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.

cs.CV↗

arXiv2024

Two Birds with One Stone: Multi-Task Semantic Communications Systems over Relay Channel

In this paper, we propose a novel multi-task, multi-link relay semantic communications (MTML-RSC) scheme that enables the destination node to simultaneously perform image reconstruction and classification with one transmission from the source node. In the MTML-RSC scheme, the source node broadcasts a signal using semantic communications, and the relay node forwards the signal to the destination. We analyze the coupling relationship between the two tasks and the two links (source-to-relay and source-to-destination) and design a semantic-focused forward method for the relay node, where it selectively forwards only the semantics of the relevant class while ignoring others. At the destination, the node combines signals from both the source node and the relay node to perform classification, and then uses the classification result to assist in decoding the signal from the relay node for image reconstructing. Experimental results demonstrate that the proposed MTML-RSC scheme achieves significant performance gains, e.g., $1.73$ dB improvement in peak-signal-to-noise ratio (PSNR) for image reconstruction and increasing the accuracy from $64.89\%$ to $70.31\%$ for classification.

cs.IT↗

arXiv2024

An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding

Recently, many methods have been developed to extend the context length of pre-trained large language models (LLMs), but they often require fine-tuning at the target length ($\gg4K$) and struggle to effectively utilize information from the middle part of the context. To address these issues, we propose $\textbf{C}$ontinuity-$\textbf{R}$elativity ind$\textbf{E}$xing with g$\textbf{A}$ussian $\textbf{M}$iddle ($\texttt{CREAM}$), which interpolates positional encodings by manipulating position indices. Apart from being simple, $\texttt{CREAM}$ is training-efficient: it only requires fine-tuning at the pre-trained context window (e.g., Llama 2-4K) and can extend LLMs to a much longer target context length (e.g., 256K). To ensure that the model focuses more on the information in the middle, we introduce a truncated Gaussian to encourage sampling from the middle part of the context during fine-tuning, thus alleviating the "Lost-in-the-Middle" problem faced by long-context LLMs. Experimental results show that $\texttt{CREAM}$ successfully extends LLMs to the target length for both Base and Chat versions of $\texttt{Llama2-7B}$ with "Never Miss A Beat". Our code is publicly available at https://github.com/bigai-nlco/cream.

cs.CL↗

arXiv2024

Field-free spin-orbit switching of canted magnetization in Pt/Co/Ru/RuO2(101) multilayers

Enabling field-free current-induced switching of perpendicular magnetization is essential for advancing spin-orbit-torque magnetic random access memory technology. Our research on the Pt/Co/Ru/RuO2(101) system has successfully demonstrated field-free switching through current injection along the RuO2[010] axis. We discovered that the system exhibits a tilted easy axis, inclined from the out-of-plane towards the RuO2[-101] direction. The application of current perpendicular to this tilted axis generates a substantial out-of-plane effective field, which facilitates field-free magnetization switching. Our results also indicate that adjusting the thickness of the Ru layer to optimize the tilt angle can significantly reduce the critical switching current density. This work provides a viable strategy for controlling the tilting magnetization, essential for the development of RuO2-based magnetic devices.

physics.app-ph↗

arXiv2024

Position: Towards Resilience Against Adversarial Examples

Current research on defending against adversarial examples focuses primarily on achieving robustness against a single attack type such as $\ell_2$ or $\ell_{\infty}$-bounded attacks. However, the space of possible perturbations is much larger than considered by many existing defenses and is difficult to mathematically model, so the attacker can easily bypass the defense by using a type of attack that is not covered by the defense. In this position paper, we argue that in addition to robustness, we should also aim to develop defense algorithms that are adversarially resilient -- defense algorithms should specify a means to quickly adapt the defended model to be robust against new attacks. We provide a definition of adversarial resilience and outline considerations of designing an adversarially resilient defense. We then introduce a subproblem of adversarial resilience which we call continual adaptive robustness, in which the defender gains knowledge of the formulation of possible perturbation spaces over time and can then update their model based on this information. Additionally, we demonstrate the connection between continual adaptive robustness and previously studied problems of multiattack robustness and unforeseen attack robustness and outline open directions within these fields which can contribute to improving continual adaptive robustness and adversarial resilience.

cs.LG↗

arXiv2024

Generative Co-Learners: Enhancing Cognitive and Social Presence of Students in Asynchronous Learning with Generative AI

Cognitive presence and social presence are crucial for a comprehensive learning experience. Despite the flexibility of asynchronous learning environments to accommodate individual schedules, the inherent constraints of asynchronous environments make augmenting cognitive and social presence particularly challenging. Students often face challenges such as a lack of timely feedback and support, a lack of non-verbal cues in communication, and a sense of isolation. To address this challenge, this paper introduces Generative Co-Learners, a system designed to leverage generative AI-powered agents, simulating co-learners supporting multimodal interactions, to improve cognitive and social presence in asynchronous learning environments. We conducted a study involving 12 student participants who used our system to engage with online programming tutorials to assess the system's effectiveness. The results show that by implementing features to support textual and visual communication and simulate an interactive learning environment with generative agents, our system enhances the cognitive and social presence in the asynchronous learning environment. These results suggest the potential to use generative AI to support students learning at scale and transform asynchronous learning into a more inclusive, engaging, and efficacious educational approach.

cs.HC↗

arXiv2024

Bootstrap3D: Improving Multi-view Diffusion Model with Synthetic Data

Recent years have witnessed remarkable progress in multi-view diffusion models for 3D content creation. However, there remains a significant gap in image quality and prompt-following ability compared to 2D diffusion models. A critical bottleneck is the scarcity of high-quality 3D objects with detailed captions. To address this challenge, we propose Bootstrap3D, a novel framework that automatically generates an arbitrary quantity of multi-view images to assist in training multi-view diffusion models. Specifically, we introduce a data generation pipeline that employs (1) 2D and video diffusion models to generate multi-view images based on constructed text prompts, and (2) our fine-tuned 3D-aware MV-LLaVA for filtering high-quality data and rewriting inaccurate captions. Leveraging this pipeline, we have generated 1 million high-quality synthetic multi-view images with dense descriptive captions to address the shortage of high-quality 3D data. Furthermore, we present a Training Timestep Reschedule (TTR) strategy that leverages the denoising process to learn multi-view consistency while maintaining the original 2D diffusion prior. Extensive experiments demonstrate that Bootstrap3D can generate high-quality multi-view images with superior aesthetic quality, image-text alignment, and maintained view consistency.

cs.CV↗

arXiv2024

The pseudo-differential perturbations of the Dirac operator and the Kastler-Kalau-Walze type theorems

We define two types of pseudo-differential perturbations of the Dirac operator within the framework of the noncommutative geometry. And we obtain the noncommutative residue of the inverse square of these perturbations on 4-dimensional compact manifolds without boundary. As a generalization of the result of noncommutative residue on closed manifolds, we prove the Kastler-Kalau-Walze type theorems for these perturbations on 4-dimensional compact manifolds with boundary. Finally, several examples in which we can consider the corresponding pseudo-differential operators and the Kastler-Kalau-Walze type theorems are listed.

math.DG↗

arXiv2024

MambaJSCC: Adaptive Deep Joint Source-Channel Coding with Generalized State Space Model

Lightweight and efficient neural network models for deep joint source-channel coding (JSCC) are crucial for semantic communications. In this paper, we propose a novel JSCC architecture, named MambaJSCC, that achieves state-of-the-art performance with low computational and parameter overhead. MambaJSCC utilizes the visual state space model with channel adaptation (VSSM-CA) blocks as its backbone for transmitting images over wireless channels, where the VSSM-CA primarily consists of the generalized state space models (GSSM) and the zero-parameter, zero-computational channel adaptation method (CSI-ReST). We design the GSSM module, leveraging reversible matrix transformations to express generalized scan expanding operations, and theoretically prove that two GSSM modules can effectively capture global information. We discover that GSSM inherently possesses the ability to adapt to channels, a form of endogenous intelligence. Based on this, we design the CSI-ReST method, which injects channel state information (CSI) into the initial state of GSSM to utilize its native response, and into the residual state to mitigate CSI forgetting, enabling effective channel adaptation without introducing additional computational and parameter overhead. Experimental results show that MambaJSCC not only outperforms existing JSCC methods (e.g., SwinJSCC) across various scenarios but also significantly reduces parameter size, computational overhead, and inference delay.

cs.IT↗