arXiv subjects

Tianyu Yang

Publications and source records attributed to Tianyu Yang.

At least 91 records · Page 5Linked to original sources

arXiv2023

Consistent123: Improve Consistency for One Image to 3D Object Synthesis

Large image diffusion models enable novel view synthesis with high quality and excellent zero-shot capability. However, such models based on image-to-image translation have no guarantee of view consistency, limiting the performance for downstream tasks like 3D reconstruction and image-to-3D generation. To empower consistency, we propose Consistent123 to synthesize novel views simultaneously by incorporating additional cross-view attention layers and the shared self-attention mechanism. The proposed attention mechanism improves the interaction across all synthesized views, as well as the alignment between the condition view and novel views. In the sampling stage, such architecture supports simultaneously generating an arbitrary number of views while training at a fixed length. We also introduce a progressive classifier-free guidance strategy to achieve the trade-off between texture and geometry for synthesized object views. Qualitative and quantitative experiments show that Consistent123 outperforms baselines in view consistency by a large margin. Furthermore, we demonstrate a significant improvement of Consistent123 on varying downstream tasks, showing its great potential in the 3D generation field. The project page is available at consistent-123.github.io.

cs.CV↗

arXiv2023

Scalable Video Object Segmentation with Simplified Framework

The current popular methods for video object segmentation (VOS) implement feature matching through several hand-crafted modules that separately perform feature extraction and matching. However, the above hand-crafted designs empirically cause insufficient target interaction, thus limiting the dynamic target-aware feature learning in VOS. To tackle these limitations, this paper presents a scalable Simplified VOS (SimVOS) framework to perform joint feature extraction and matching by leveraging a single transformer backbone. Specifically, SimVOS employs a scalable ViT backbone for simultaneous feature extraction and matching between query and reference features. This design enables SimVOS to learn better target-ware features for accurate mask prediction. More importantly, SimVOS could directly apply well-pretrained ViT backbones (e.g., MAE) for VOS, which bridges the gap between VOS and large-scale self-supervised pre-training. To achieve a better performance-speed trade-off, we further explore within-frame attention and propose a new token refinement module to improve the running speed and save computational cost. Experimentally, our SimVOS achieves state-of-the-art results on popular video object segmentation benchmarks, i.e., DAVIS-2017 (88.0% J&F), DAVIS-2016 (92.9% J&F) and YouTube-VOS 2019 (84.2% J&F), without applying any synthetic video or BL30K pre-training used in previous VOS approaches.

cs.CV↗

arXiv2023

Latent Video Diffusion Models for High-Fidelity Long Video Generation

AI-generated content has attracted lots of attention recently, but photo-realistic video synthesis is still challenging. Although many attempts using GANs and autoregressive models have been made in this area, the visual quality and length of generated videos are far from satisfactory. Diffusion models have shown remarkable results recently but require significant computational resources. To address this, we introduce lightweight video diffusion models by leveraging a low-dimensional 3D latent space, significantly outperforming previous pixel-space video diffusion models under a limited computational budget. In addition, we propose hierarchical diffusion in the latent space such that longer videos with more than one thousand frames can be produced. To further overcome the performance degradation issue for long video generation, we propose conditional latent perturbation and unconditional guidance that effectively mitigate the accumulated errors during the extension of video length. Extensive experiments on small domain datasets of different categories suggest that our framework generates more realistic and longer videos than previous strong baselines. We additionally provide an extension to large-scale text-to-video generation to demonstrate the superiority of our work. Our code and models will be made publicly available.

cs.CV↗

arXiv2023

Deep-Learning Aided Channel Training and Precoding in FDD Massive MIMO with Channel Statistics Knowledge

We propose a method for channel training and precoding in FDD massive MIMO based on deep neural networks (DNNs), exploiting Downlink (DL) channel covariance knowledge. The DNN is optimized to maximize the DL multi-user sum-rate, by producing a pre-beamforming matrix based on user channel covariances that maps the original channel vectors to effective channels. Measurements of these effective channels are received at the users via common pilot transmission and sent back to the base station (BS) through analog feedback without further processing. The BS estimates the effective channels from received feedback and constructs a linear precoder by concatenating the optimized pre-beamforming matrix with a zero-forcing precoder over the effective channels. We show that the proposed method yields significantly higher sum-rates than the state-of-the-art DNN-based channel training and precoding scheme, especially in scenarios with small pilot and feedback size relative to the channel coherence block length. Unlike many works in the literature, our proposition does not involve deployment of a DNN at the user side, which typically comes at a high computational cost and parameter-transmission overhead on the system, and is therefore considerably more practical.

cs.IT↗

arXiv2022

SimVTP: Simple Video Text Pre-training with Masked Autoencoders

This paper presents SimVTP: a Simple Video-Text Pretraining framework via masked autoencoders. We randomly mask out the spatial-temporal tubes of input video and the word tokens of input text and then feed them into a unified autencoder to reconstruct the missing pixels and words. Our SimVTP has several properties: 1) Thanks to the unified autoencoder, SimVTP reconstructs the masked signal of one modality with the help from another modality, which implicitly learns the cross-modal alignment between video tubes and text tokens. 2) SimVTP not only benefits from a high video masking ratio (e.g. 90%) due to the temporal redundancy of video, but also needs a high text masking ratio (e.g. 75%), which is much higher than BERT (e.g. 15%), to achieve optimal performance. This is because the aid of video modality makes text reconstruction less challenging, which thus needs a higher mask ratio to make the pretext harder for useful feature learning. 3) Equipping SimVTP with video-text contrastive learning (VTC) and video-text matching (VTM), which are two commonly used cross-modal training strategies, could further improve the transferable performance significantly. 4) SimVTP is dataefficent, e.g., pre-training only on 10% data of WebVid-2M, SimVTP achieves surprisingly good results (43.8 R@1) on MSRVTT, which is far above recent state-of-the-art methods pre-trained on both CC3M and WebVid-2M. We transfer our pre-trained model to various downstream tasks and achieve superior performance. The codes and models will be released at https://github.com/mayuelala/SimVTP.

cs.CV↗

arXiv2022

SWEM: Towards Real-Time Video Object Segmentation with Sequential Weighted Expectation-Maximization

Matching-based methods, especially those based on space-time memory, are significantly ahead of other solutions in semi-supervised video object segmentation (VOS). However, continuously growing and redundant template features lead to an inefficient inference. To alleviate this, we propose a novel Sequential Weighted Expectation-Maximization (SWEM) network to greatly reduce the redundancy of memory features. Different from the previous methods which only detect feature redundancy between frames, SWEM merges both intra-frame and inter-frame similar features by leveraging the sequential weighted EM algorithm. Further, adaptive weights for frame features endow SWEM with the flexibility to represent hard samples, improving the discrimination of templates. Besides, the proposed method maintains a fixed number of template features in memory, which ensures the stable inference complexity of the VOS system. Extensive experiments on commonly used DAVIS and YouTube-VOS datasets verify the high efficiency (36 FPS) and high performance (84.3\% $\mathcal{J}\&\mathcal{F}$ on DAVIS 2017 validation dataset) of SWEM. Code is available at: https://github.com/lmm077/SWEM.

cs.CV↗

arXiv2022

LocVTP: Video-Text Pre-training for Temporal Localization

Video-Text Pre-training (VTP) aims to learn transferable representations for various downstream tasks from large-scale web videos. To date, almost all existing VTP methods are limited to retrieval-based downstream tasks, e.g., video retrieval, whereas their transfer potentials on localization-based tasks, e.g., temporal grounding, are under-explored. In this paper, we experimentally analyze and demonstrate the incompatibility of current VTP methods with localization tasks, and propose a novel Localization-oriented Video-Text Pre-training framework, dubbed as LocVTP. Specifically, we perform the fine-grained contrastive alignment as a complement to the coarse-grained one by a clip-word correspondence discovery scheme. To further enhance the temporal reasoning ability of the learned feature, we propose a context projection head and a temporal aware contrastive loss to perceive the contextual relationships. Extensive experiments on four downstream tasks across six datasets demonstrate that our LocVTP achieves state-of-the-art performance on both retrieval-based and localization-based tasks. Furthermore, we conduct comprehensive ablation studies and thorough analyses to explore the optimum model designs and training strategies.

cs.CV↗

arXiv2022

FDD Massive MIMO Channel Training Optimal Rate Distortion Bounds and the Efficiency of one-shot Schemes

We study the problem of providing channel state information (CSI) at the transmitter in multi-user massive MIMO systems operating in frequency division duplexing (FDD). The wideband MIMO channel is a vector-valued random process correlated in time, space (antennas), and frequency (subcarriers). The base station (BS) broadcasts periodically beta_tr pilot symbols from its M antenna ports to K single-antenna users (UEs). Correspondingly, the K UEs send feedback messages about their channel state using beta_fb symbols in the uplink (UL). Using results from remote rate-distortion theory, we show that, as snr reaches infty, the optimal feedback strategy achieves a channel state estimation mean squared error (MSE) that behaves as Theta(1) if beta_tr < r and as Theta(snr^(-alpha)) when beta_tr >=r, where alpha = min(beta_fb/r, 1), where r is the rank of the channel covariance matrix. The MSE-optimal rate-distortion strategy implies encoding of long sequences of channel states, which would yield completely stale CSI and therefore poor multiuser precoding performance. Hence, we consider three practical one-shot CSI strategies with minimum one-slot delay and analyze their large-SNR channel estimation MSE behavior. These are: (1) digital feedback via entropy-coded scalar quantization (ECSQ), (2) analog feedback (AF), and (3) local channel estimation at the UEs and digital feedback. These schemes have different requirements in terms of knowledge of the channel statistics at the UE and at the BS. In particular, the latter strategy requires no statistical knowledge and is closely inspired by a CSI feedback scheme currently proposed in 3GPP standardization.

cs.IT↗

arXiv2022

Structured Channel Covariance Estimation from Limited Samples for Large Antenna Arrays

In massive multiple-input multiple-output (MIMO) systems, the knowledge of the users' channel covariance matrix is crucial for minimum mean square error (MMSE) channel estimation in the uplink as well as it plays an important role in several multiuser beamforming schemes in the downlink. Due to the large number of base station antennas in massive MIMO, accurate covariance estimation is challenging especially in the case where the number of samples is limited and thus comparable to the channel vector dimension. As a result, the standard sample covariance estimator may yield a too large estimation error which in turn may yield significant system performance degradation with respect to the ideal channel covariance knowledge case. To address such problem, we propose a method based on a parametric representation of the channel angular scattering function. The proposed parametric representation includes a discrete specular component which is addressed using the well-known MUltiple SIgnal Classification (MUSIC) method, and a diffuse scattering component, which is modeled as the superposition of suitable dictionary functions. To obtain the representation parameters we propose two methods, where the first solves a non-negative least-squares problem and the second maximizes the likelihood function using expectation-maximization. Our simulation results show that the proposed methods outperform the state of the art with respect to various estimation quality metrics and different sample sizes.

cs.IT↗

arXiv2022

Unsupervised Pre-training for Temporal Action Localization Tasks

Unsupervised video representation learning has made remarkable achievements in recent years. However, most existing methods are designed and optimized for video classification. These pre-trained models can be sub-optimal for temporal localization tasks due to the inherent discrepancy between video-level classification and clip-level localization. To bridge this gap, we make the first attempt to propose a self-supervised pretext task, coined as Pseudo Action Localization (PAL) to Unsupervisedly Pre-train feature encoders for Temporal Action Localization tasks (UP-TAL). Specifically, we first randomly select temporal regions, each of which contains multiple clips, from one video as pseudo actions and then paste them onto different temporal positions of the other two videos. The pretext task is to align the features of pasted pseudo action regions from two synthetic videos and maximize the agreement between them. Compared to the existing unsupervised video representation learning approaches, our PAL adapts better to downstream TAL tasks by introducing a temporal equivariant contrastive learning paradigm in a temporally dense and scale-aware manner. Extensive experiments show that PAL can utilize large-scale unlabeled video data to significantly boost the performance of existing TAL methods. Our codes and models will be made publicly available at https://github.com/zhang-can/UP-TAL.

cs.CV↗

arXiv2022

Motion-aware Contrastive Video Representation Learning via Foreground-background Merging

In light of the success of contrastive learning in the image domain, current self-supervised video representation learning methods usually employ contrastive loss to facilitate video representation learning. When naively pulling two augmented views of a video closer, the model however tends to learn the common static background as a shortcut but fails to capture the motion information, a phenomenon dubbed as background bias. Such bias makes the model suffer from weak generalization ability, leading to worse performance on downstream tasks such as action recognition. To alleviate such bias, we propose \textbf{F}oreground-b\textbf{a}ckground \textbf{Me}rging (FAME) to deliberately compose the moving foreground region of the selected video onto the static background of others. Specifically, without any off-the-shelf detector, we extract the moving foreground out of background regions via the frame difference and color statistics, and shuffle the background regions among the videos. By leveraging the semantic consistency between the original clips and the fused ones, the model focuses more on the motion patterns and is debiased from the background shortcut. Extensive experiments demonstrate that FAME can effectively resist background cheating and thus achieve the state-of-the-art performance on downstream tasks across UCF101, HMDB51, and Diving48 datasets. The code and configurations are released at https://github.com/Mark12Ding/FAME.

cs.CV↗

arXiv2022

Semantic-Preserving Linguistic Steganography by Pivot Translation and Semantic-Aware Bins Coding

Linguistic steganography (LS) aims to embed secret information into a highly encoded text for covert communication. It can be roughly divided to two main categories, i.e., modification based LS (MLS) and generation based LS (GLS). Unlike MLS that hides secret data by slightly modifying a given text without impairing the meaning of the text, GLS uses a trained language model to directly generate a text carrying secret data. A common disadvantage for MLS methods is that the embedding payload is very low, whose return is well preserving the semantic quality of the text. In contrast, GLS allows the data hider to embed a high payload, which has to pay the high price of uncontrollable semantics. In this paper, we propose a novel LS method to modify a given text by pivoting it between two different languages and embed secret data by applying a GLS-like information encoding strategy. Our purpose is to alter the expression of the given text, enabling a high payload to be embedded while keeping the semantic information unchanged. Experimental results have shown that the proposed work not only achieves a high embedding payload, but also shows superior performance in maintaining the semantic consistency and resisting linguistic steganalysis.

cs.CR↗

arXiv2022

Bone tumor suppression in rabbits by hyperthermia below the clinical safety limit using aligned magnetic bone cement

Demonstrating highly efficient alternating current (AC) magnetic field heating of nanoparticles in physiological environments under clinically safe field parameters has remained a great challenge, hindering clinical applications of magnetic hyperthermia. In this work, we report exceptionally high loss power of magnetic bone cement under clinical safety limit of AC field parameters, incorporating DC field-aligned soft magnetic Zn0.3Fe2.7O4 nanoparticles with low concentration. Under an AC field of 4 kA/m at 430 kHz, the aligned bone cement with 0.2 wt% nanoparticles achieved a temperature increase of 30 C in 180 s. This amounts to a specific loss power value of 327 W/gmetal and an intrinsic loss power of 47 nHm^2/kg, which is enhanced by 50-fold compared to randomly oriented samples. The high-performance magnetic bone cement allows for the demonstration of effective hyperthermia suppression of tumor growth in the bone marrow cavity of New Zealand White Rabbits subjecting to rapid cooling due to blood circulation, and significant enhancement of survival rate.

physics.med-ph↗

arXiv2022

Channel State Acquisition in FDD Massive MIMO: Rate-Distortion Bound and Effectiveness of "Analog" Feedback

We consider the problem of estimating channel fading coefficients (modeled as a correlated Gaussian vector) via Downlink (DL) training and Uplink (UL) feedback in wideband FDD massive MIMO systems. Using rate-distortion theory, we derive optimal bounds on the achievable channel state estimation error in terms of the number of training pilots in DL ($β_{tr}$) and feedback dimension in UL ($β_{fb}$), with random, spatially isotropic pilots. It is shown that when the number of training pilots exceeds the channel covariance rank ($r$), the optimal rate-distortion feedback strategy achieves an estimation error decay of $Θ(SNR^{-α})$ in estimating the channel state, where $α= min (β_{fb}/r , 1)$ is the so-called quality scaling exponent. We also discuss an "analog" feedback strategy, showing that it can achieve the optimal quality scaling exponent for a wide range of training and feedback dimensions with no channel covariance knowledge and simple signal processing at the user side. Our findings are supported by numerical simulations comparing various strategies in terms of channel state mean squared error and achievable ergodic sum-rate in DL with zero-forcing precoding.

cs.IT↗

arXiv2021

A Linearized Boundary Control Method for the Acoustic Inverse Boundary Value Problem

We develop a linearized boundary control method for the inverse boundary value problem of determining a potential in the acoustic wave equation from the Neumann-to-Dirichlet map. When the linearization is at the zero potential, we derive a reconstruction formula based on the boundary control method and prove that it is of Lipschitz-type stability. When the linearization is at a nonzero potential, we prove that the problem is of Hölder-type stability. The proposed reconstruction formula is implemented and evaluated using several numerical experiments to validate its feasibility.

math.AP↗

arXiv2021

Achievable Regions and Precoder Designs for the Multiple Access Wiretap Channels with Confidential and Open Messages

This paper investigates the secrecy capacity region of multiple access wiretap (MAC-WT) channels where, besides confidential messages, the users have also open messages to transmit. All these messages are intended for the legitimate receiver (or Bob for brevity) but only the confidential messages need to be protected from the eavesdropper (Eve). We first consider a discrete memoryless (DM) MAC-WT channel where both Bob and Eve jointly decode their interested messages. By using random coding, we find an achievable rate region, within which perfect secrecy can be realized, i.e., all users can communicate with Bob with arbitrarily small probability of error, while the confidential information leaked to Eve tends to zero. Due to the high implementation complexity of joint decoding, we also consider the DM MAC-WT channel where Bob simply decodes messages independently while Eve still applies joint decoding. We then extend the results in the DM case to a Gaussian vector (GV) MAC-WT channel. Based on the information theoretic results, we further maximize the sum secrecy rate of the GV MAC-WT system by designing precoders for all users. Since the problems are non-convex, we provide iterative algorithms to obtain suboptimal solutions. Simulation results show that compared with existing schemes, secure communication can be greatly enhanced by the proposed algorithms, and in contrast to the works which only focus on the network secrecy performance, the system spectrum efficiency can be effectively improved since open messages can be simultaneously transmitted.

cs.IT↗

arXiv2021

Quantization-Aided Secrecy: FD C-RAN Communications with Untrusted Radios

In this work, we study a full-duplex (FD) cloud radio access network (C-RAN) from the aspects of infrastructure sharing and information secrecy, where the central unit utilizes FD remote radio units (RU)s belonging to the same operator, i.e., the trusted RUs, as well as the RUs belonging to other operators or private owners, i.e., the untrusted RUs. Furthermore, the communication takes place in the presence of untrusted external receivers, i.e., eavesdropper nodes. The communicated uplink (UL) and downlink (DL) waveforms are quantized in order to comply with the limited capacity of the fronthaul links. In order to provide information secrecy, we propose a novel utilization of the quantization noise shaping in the DL, such that it is simultaneously used to comply with the limited capacity of the fronthaul links, as well as to degrade decoding capability of the individual eavesdropper and the untrusted RUs for both the UL and DL communications. In this regard, expressions describing the achievable secrecy rates are obtained. An optimization problem for jointly designing the DL and UL quantization and precoding strategies are then formulated, with the purpose of maximizing the overall system weighted sum secrecy rate. Due to the intractability of the formulated problem, an iterative solution is proposed, following the successive inner approximation and semi-definite relaxation frameworks, with convergence to a stationary point. Numerical evaluations indicate a promising gain of the proposed approaches for providing information secrecy against the untrusted infrastructure nodes and/or external eavesdroppers in the context of FD C-RAN communications.

cs.IT↗

arXiv2021

New challenges in covariance estimation: multiple structures and coarse quantization

In this self-contained chapter, we revisit a fundamental problem of multivariate statistics: estimating covariance matrices from finitely many independent samples. Based on massive Multiple-Input Multiple-Output (MIMO) systems we illustrate the necessity of leveraging structure and considering quantization of samples when estimating covariance matrices in practice. We then provide a selective survey of theoretical advances of the last decade focusing on the estimation of structured covariance matrices. This review is spiced up by some yet unpublished insights on how to benefit from combined structural constraints. Finally, we summarize the findings of our recently published preprint "Covariance estimation under one-bit quantization" to show how guaranteed covariance estimation is possible even under coarse quantization of the samples.

math.ST↗