arXiv subjects

Sheng Li

Publications and source records attributed to Sheng Li.

At least 217 records · Page 12Linked to original sources

arXiv2022

Image Generation Network for Covert Transmission in Online Social Network

Online social networks have stimulated communications over the Internet more than ever, making it possible for secret message transmission over such noisy channels. In this paper, we propose a Coverless Image Steganography Network, called CIS-Net, that synthesizes a high-quality image directly conditioned on the secret message to transfer. CIS-Net is composed of four modules, namely, the Generation, Adversarial, Extraction, and Noise Module. The receiver can extract the hidden message without any loss even the images have been distorted by JPEG compression attacks. To disguise the behaviour of steganography, we collected images in the context of profile photos and stickers and train our network accordingly. As such, the generated images are more inclined to escape from malicious detection and attack. The distinctions from previous image steganography methods are majorly the robustness and losslessness against diverse attacks. Experiments over diverse public datasets have manifested the superior ability of anti-steganalysis.

cs.CV↗

arXiv2022

Robust Watermarking for Video Forgery Detection with Improved Imperceptibility and Robustness

Videos are prone to tampering attacks that alter the meaning and deceive the audience. Previous video forgery detection schemes find tiny clues to locate the tampered areas. However, attackers can successfully evade supervision by destroying such clues using video compression or blurring. This paper proposes a video watermarking network for tampering localization. We jointly train a 3D-UNet-based watermark embedding network and a decoder that predicts the tampering mask. The perturbation made by watermark embedding is close to imperceptible. Considering that there is no off-the-shelf differentiable video codec simulator, we propose to mimic video compression by ensembling simulation results of other typical attacks, e.g., JPEG compression and blurring, as an approximation. Experimental results demonstrate that our method generates watermarked videos with good imperceptibility and robustly and accurately locates tampered areas within the attacked version.

cs.CV↗

arXiv2022

Sustainable AI Processing at the Edge

Edge computing is a popular target for accelerating machine learning algorithms supporting mobile devices without requiring the communication latencies to handle them in the cloud. Edge deployments of machine learning primarily consider traditional concerns such as SWaP constraints (Size, Weight, and Power) for their installations. However, such metrics are not entirely sufficient to consider environmental impacts from computing given the significant contributions from embodied energy and carbon. In this paper we explore the tradeoffs of convolutional neural network acceleration engines for both inference and on-line training. In particular, we explore the use of processing-in-memory (PIM) approaches, mobile GPU accelerators, and recently released FPGAs, and compare them with novel Racetrack memory PIM. Replacing PIM-enabled DDR3 with Racetrack memory PIM can recover its embodied energy as quickly as 1 year. For high activity ratios, mobile GPUs can be more sustainable but have higher embodied energy to overcome compared to PIM-enabled Racetrack memory.

cs.AR↗

arXiv2022

Coupling Visual Semantics of Artificial Neural Networks and Human Brain Function via Synchronized Activations

Artificial neural networks (ANNs), originally inspired by biological neural networks (BNNs), have achieved remarkable successes in many tasks such as visual representation learning. However, whether there exists semantic correlations/connections between the visual representations in ANNs and those in BNNs remains largely unexplored due to both the lack of an effective tool to link and couple two different domains, and the lack of a general and effective framework of representing the visual semantics in BNNs such as human functional brain networks (FBNs). To answer this question, we propose a novel computational framework, Synchronized Activations (Sync-ACT), to couple the visual representation spaces and semantics between ANNs and BNNs in human brain based on naturalistic functional magnetic resonance imaging (nfMRI) data. With this approach, we are able to semantically annotate the neurons in ANNs with biologically meaningful description derived from human brain imaging for the first time. We evaluated the Sync-ACT framework on two publicly available movie-watching nfMRI datasets. The experiments demonstrate a) the significant correlation and similarity of the semantics between the visual representations in FBNs and those in a variety of convolutional neural networks (CNNs) models; b) the close relationship between CNN's visual representation similarity to BNNs and its performance in image classification tasks. Overall, our study introduces a general and effective paradigm to couple the ANNs and BNNs and provides novel insights for future studies such as brain-inspired artificial intelligence.

cs.CV↗

arXiv2022

A DTCWT-SVD Based Video Watermarking resistant to frame rate conversion

Videos can be easily tampered, copied and redistributed by attackers for illegal and monetary usage. Such behaviors severely jeopardize the interest of content owners. Despite huge efforts made in digital video watermarking for copyright protection, typical distortions in video transmission including signal attacks, geometric attacks and temporal synchronization attacks can still easily erase the embedded signal. Among them, temporal synchronization attacks which include frame dropping, frame insertion and frame rate conversion is one of the most prevalent attacks. To address this issue, we present a new video watermarking based on joint Dual-Tree Cosine Wavelet Transformation (DTCWT) and Singular Value Decomposition (SVD), which is resistant to frame rate conversion. We first extract a set of candidate coefficient by applying SVD decomposition after DTCWT transform. Then, we simulate the watermark embedding by adjusting the shape of candidate coefficient. Finally, we perform group-level watermarking that includes moderate temporal redundancy to resist temporal desynchronization attacks. Extensive experimental results show that the proposed scheme is more resilient to temporal desynchronization attacks and performs better than the existing blind video watermarking schemes.

cs.MM↗

arXiv2022

NeuralSound: Learning-based Modal Sound Synthesis With Acoustic Transfer

We present a novel learning-based modal sound synthesis approach that includes a mixed vibration solver for modal analysis and an end-to-end sound radiation network for acoustic transfer. Our mixed vibration solver consists of a 3D sparse convolution network and a Locally Optimal Block Preconditioned Conjugate Gradient module (LOBPCG) for iterative optimization. Moreover, we highlight the correlation between a standard modal vibration solver and our network architecture. Our radiation network predicts the Far-Field Acoustic Transfer maps (FFAT Maps) from the surface vibration of the object. The overall running time of our learning method for any new object is less than one second on a GTX 3080 Ti GPU while maintaining a high sound quality close to the ground truth that is computed using standard numerical methods. We also evaluate the numerical accuracy and perceptual accuracy of our sound synthesis approach on different objects corresponding to various materials.

cs.SD↗

arXiv2022

Multimodal Fake News Detection via CLIP-Guided Learning

Multimodal fake news detection has attracted many research interests in social forensics. Many existing approaches introduce tailored attention mechanisms to guide the fusion of unimodal features. However, how the similarity of these features is calculated and how it will affect the decision-making process in FND are still open questions. Besides, the potential of pretrained multi-modal feature learning models in fake news detection has not been well exploited. This paper proposes a FND-CLIP framework, i.e., a multimodal Fake News Detection network based on Contrastive Language-Image Pretraining (CLIP). Given a targeted multimodal news, we extract the deep representations from the image and text using a ResNet-based encoder, a BERT-based encoder and two pair-wise CLIP encoders. The multimodal feature is a concatenation of the CLIP-generated features weighted by the standardized cross-modal similarity of the two modalities. The extracted features are further processed for redundancy reduction before feeding them into the final classifier. We introduce a modality-wise attention module to adaptively reweight and aggregate the features. We have conducted extensive experiments on typical fake news datasets. The results indicate that the proposed framework has a better capability in mining crucial features for fake news detection. The proposed FND-CLIP can achieve better performances than previous works, i.e., 0.7\%, 6.8\% and 1.3\% improvements in overall accuracy on Weibo, Politifact and Gossipcop, respectively. Besides, we justify that CLIP-based learning can allow better flexibility on multimodal feature selection.

cs.CV↗

arXiv2022

Fusion of Self-supervised Learned Models for MOS Prediction

We participated in the mean opinion score (MOS) prediction challenge, 2022. This challenge aims to predict MOS scores of synthetic speech on two tracks, the main track and a more challenging sub-track: out-of-domain (OOD). To improve the accuracy of the predicted scores, we have explored several model fusion-related strategies and proposed a fused framework in which seven pretrained self-supervised learned (SSL) models have been engaged. These pretrained SSL models are derived from three ASR frameworks, including Wav2Vec, Hubert, and WavLM. For the OOD track, we followed the 7 SSL models selected on the main track and adopted a semi-supervised learning method to exploit the unlabeled data. According to the official analysis results, our system has achieved 1st rank in 6 out of 16 metrics and is one of the top 3 systems for 13 out of 16 metrics. Specifically, we have achieved the highest LCC, SRCC, and KTAU scores at the system level on main track, as well as the best performance on the LCC, SRCC, and KTAU evaluation metrics at the utterance level on OOD track. Compared with the basic SSL models, the prediction accuracy of the fused system has been largely improved, especially on OOD sub-track.

cs.SD↗

arXiv2022

Multi-Task Adversarial Learning for Treatment Effect Estimation in Basket Trials

Estimating treatment effects from observational data provides insights about causality guiding many real-world applications such as different clinical study designs, which are the formulations of trials, experiments, and observational studies in medical, clinical, and other types of research. In this paper, we describe causal inference for application in a novel clinical design called basket trial that tests how well a new drug works in patients who have different types of cancer that all have the same mutation. We propose a multi-task adversarial learning (MTAL) method, which incorporates feature selection multi-task representation learning and adversarial learning to estimate potential outcomes across different tumor types for patients sharing the same genetic mutation but having different tumor types. In our paper, the basket trial is employed as an intuitive example to present this new causal inference setting. This new causal inference setting includes, but is not limited to basket trials. This setting has the same challenges as the traditional causal inference problem, i.e., missing counterfactual outcomes under different subgroups and treatment selection bias due to confounders. We present the practical advantages of our MTAL method for the analysis of synthetic basket trial data and evaluate the proposed estimator on two benchmarks, IHDP and News. The results demonstrate the superiority of our MTAL method over the competing state-of-the-art methods.

cs.LG↗

arXiv2022

Emergence of insulating ferrimagnetism and perpendicular magnetic anisotropy in 3d-5d perovskite oxide composite films for insulator spintronic

Magnetic insulators with strong perpendicular magnetic anisotropy (PMA) play a key role in exploring pure spin current phenomena and developing ultralow-dissipation spintronic devices, thereby it is highly desirable to develop new material platforms. Here we report epitaxial growth of La2/3Sr1/3MnO3 (LSMO)-SrIrO3 (SIO) composite oxide films (LSMIO) with different crystalline orientations fabricated by sequential two-target ablation process using pulsed laser deposition. The LSMIO films exhibit high crystalline quality with homogeneous mixture of LSMO and SIO at atomic level. Ferrimagnetic and insulating transport characteristics are observed, with the temperature-dependent electric resistivity well fitted by Mott variable-range-hopping model. Moreover, the LSMIO films show strong PMA. Through further constructing all perovskite oxide heterostructures of the ferrimagnetic insulator LSMIO and a strong spin-orbital coupled SIO layer, pronounced spin Hall magnetoresistance (SMR) and spin Hall-like anomalous Hall effect (SH-AHE) were observed. These results illustrate the potential application of the ferrimagnetic insulator LSMIO in developing all-oxide ultralow-dissipation spintronic devices.

cond-mat.mtrl-sci↗

arXiv2022

Learning Infomax and Domain-Independent Representations for Causal Effect Inference with Real-World Data

The foremost challenge to causal inference with real-world data is to handle the imbalance in the covariates with respect to different treatment options, caused by treatment selection bias. To address this issue, recent literature has explored domain-invariant representation learning based on different domain divergence metrics (e.g., Wasserstein distance, maximum mean discrepancy, position-dependent metric, and domain overlap). In this paper, we reveal the weaknesses of these strategies, i.e., they lead to the loss of predictive information when enforcing the domain invariance; and the treatment effect estimation performance is unstable, which heavily relies on the characteristics of the domain distributions and the choice of domain divergence metrics. Motivated by information theory, we propose to learn the Infomax and Domain-Independent Representations to solve the above puzzles. Our method utilizes the mutual information between the global feature representations and individual feature representations, and the mutual information between feature representations and treatment assignment predictions, in order to maximally capture the common predictive information for both treatment and control groups. Moreover, our method filters out the influence of instrumental and irrelevant variables, and thus it effectively increases the predictive ability of potential outcomes. Experimental results on both the synthetic and real-world datasets show that our method achieves state-of-the-art performance on causal effect inference. Moreover, our method exhibits reliable prediction performances when facing data with different characteristics of data distributions, complicated variable types, and severe covariate imbalance.

stat.ML↗

arXiv2022

High-Capacity Framework for Reversible Data Hiding in Encrypted Image Using Pixel Predictions and Entropy Encoding

While the existing vacating room before encryption (VRBE) based schemes can achieve decent embedding rate, the payloads of the existing vacating room after encryption (VRAE) based schemes are relatively low. To address this issue, this paper proposes a generalized framework for high-capacity RDHEI for both VRBE and VRAE cases. First, an efficient embedding room generation algorithm (ERGA) is designed to produce large embedding room by using pixel prediction and entropy encoding. Then, we propose two RDHEI schemes, one for VRBE, another for VRAE. In the VRBE scenario, the image owner generates the embedding room with ERGA and encrypts the preprocessed image by using the stream cipher with two encryption keys. Then, the data hider locates the embedding room and embeds the encrypted additional data. In the VRAE scenario, the cover image is encrypted by an improved block modulation and permutation encryption algorithm, where the spatial redundancy in the plain-text image is largely preserved. Then, the data hider applies ERGA on the encrypted image to generate the embedding room and conducts data embedding. For both schemes, the receivers with different authentication keys can respectively conduct error-free data extraction and/or error-free image recovery. The experimental results show that the two proposed schemes outperform many state-of-the-art RDHEI arts. Besides, the schemes can ensure high security level, where the original image can be hardly discovered from the encrypted version before and after data hiding by the unauthorized user.

cs.MM↗

arXiv2021

Invertible Image Dataset Protection

Deep learning has achieved enormous success in various industrial applications. Companies do not want their valuable data to be stolen by malicious employees to train pirated models. Nor do they wish the data analyzed by the competitors after using them online. We propose a novel solution for dataset protection in this scenario by robustly and reversibly transform the images into adversarial images. We develop a reversible adversarial example generator (RAEG) that introduces slight changes to the images to fool traditional classification models. Even though malicious attacks train pirated models based on the defensed versions of the protected images, RAEG can significantly weaken the functionality of these models. Meanwhile, the reversibility of RAEG ensures the performance of authorized models. Extensive experiments demonstrate that RAEG can better protect the data with slight distortion against adversarial defense than previous methods.

cs.CV↗

arXiv2021

Adversarial Memory Networks for Action Prediction

Action prediction aims to infer the forthcoming human action with partially-observed videos, which is a challenging task due to the limited information underlying early observations. Existing methods mainly adopt a reconstruction strategy to handle this task, expecting to learn a single mapping function from partial observations to full videos to facilitate the prediction process. In this study, we propose adversarial memory networks (AMemNet) to generate the "full video" feature conditioning on a partial video query from two new aspects. Firstly, a key-value structured memory generator is designed to memorize different partial videos as key memories and dynamically write full videos in value memories with gating mechanism and querying attention. Secondly, we develop a class-aware discriminator to guide the memory generator to deliver not only realistic but also discriminative full video features upon adversarial training. The final prediction result of AMemNet is given by late fusion over RGB and optical flow streams. Extensive experimental results on two benchmark video datasets, UCF-101 and HMDB51, are provided to demonstrate the effectiveness of the proposed AMemNet model over state-of-the-art methods.

cs.CV↗

arXiv2021

GenReg: Deep Generative Method for Fast Point Cloud Registration

Accurate and efficient point cloud registration is a challenge because the noise and a large number of points impact the correspondence search. This challenge is still a remaining research problem since most of the existing methods rely on correspondence search. To solve this challenge, we propose a new data-driven registration algorithm by investigating deep generative neural networks to point cloud registration. Given two point clouds, the motivation is to generate the aligned point clouds directly, which is very useful in many applications like 3D matching and search. We design an end-to-end generative neural network for aligned point clouds generation to achieve this motivation, containing three novel components. Firstly, a point multi-perception layer (MLP) mixer (PointMixer) network is proposed to efficiently maintain both the global and local structure information at multiple levels from the self point clouds. Secondly, a feature interaction module is proposed to fuse information from cross point clouds. Thirdly, a parallel and differential sample consensus method is proposed to calculate the transformation matrix of the input point clouds based on the generated registration results. The proposed generative neural network is trained in a GAN framework by maintaining the data distribution and structure similarity. The experiments on both ModelNet40 and 7Scene datasets demonstrate that the proposed algorithm achieves state-of-the-art accuracy and efficiency. Notably, our method reduces $2\times$ in registration error (CD) and $12\times$ running time compared to the state-of-the-art correspondence-based algorithm.

cs.CV↗

arXiv2021

Transport anomalies in the layered compound BaPt4Se6

We report a layered ternary selenide BaPt4Se6 featuring sesqui-selenide Pt2Se3 layers sandwiched by Ba atoms. The Pt2Se3 layers in this compound can be derived from the Dirac-semimetal PtSe2 phase with Se vacancies that form a honeycomb structure. This structure results in a Pt (VI) and Pt (II) mixed-valence compound with both PtSe6 octahedra and PtSe4 square net coordination configurations. Temperature dependent electrical transport measurements suggest two distinct anomalies: a resistivity crossover, mimic to the metal-insulator (M-I) transition at ~150K, and a resistivity plateau at temperatures below 10K. The resistivity crossover is not associated with any structural, magnetic or charge order modulated phase transitions. Magnetoresistivity, Hall and heat capacity measurements concurrently suggest an existing hidden state below 5K in this system. Angle-resolved photoemission spectroscopy measurements reveal a metallic state and no dramatic reconstruction of the electronic structure up to 200K.

cond-mat.str-el↗

arXiv2021

Circuit QED simulator of two-dimensional Su-Schrieffer-Hegger model: magnetic field induced topological phase transition in high-order topological insulators

High-order topological insulator (HOTI) occupies an important position in topological band theory due to its exotic bulk-edge correspondence. Recently, it has been predicted that external magnetic field can introduce rich physics into two-dimensional (2D) HOTIs. However, up to now the theoretical description is still incomplete and the experimental realization is still lacking. Here we investigate the influence of continuously varying magnetic field on 2D Su-Schriffer-Heeger lattice, which is one of the most celebrated HOTI models, and proposed a corresponding circuit quantum electrodynamics (cQED) simulator. Our numerical calculation shows that the zero energy corner modes (ZECMs), which can serve as evidence of the high order topology of the lattice, exhibit exotic and rich dependence on the imposed magnetic field and the inhomogeneous hopping strength. Moreover, by exploiting the parametric conversion method, we can establish time- and site-resolved tunable hopping constants in the proposed cQED simulator, thus providing an ideal platform for simulating the magnetic field induced topological phase transitions in 2D HOTIs. Since the high-order topological phases of the proposed model can be characterized by the existence of the ZECMs on the lattice, we further investigate the corner site excitation of the lattice in the steady state limit. Our numerical results imply that the predicted topological phase transitions can be unambiguously identified by the steady-state photon number measurement of the corner sites and their few neighbors. Requiring only current level of technology, our scheme can be readily tested in experiment and may pave an alternative way towards the future investigation of HOTIs in the presence of magnetic field, disorder, and strong correlation.

quant-ph↗

arXiv2021

Lateral modulation of magnetic anisotropy in tricolor 3d-5d oxide superlattices

Manipulating magnetic anisotropy (MA) purposefully in transition metal oxides (TMOs) enables the development of oxide-based spintronic devices with practical applications. Here, we report a pathway to reversibly switch the lateral magnetic easy-axis via interfacial oxygen octahedral coupling (OOC) effects in 3d-5d tricolor superlattices, i.e. [SrIrO3,mRTiO3,SrIrO3,2La0.67Sr0.33MnO3]10 (RTiO3: SrTiO3 and CaTiO3). In the heterostructures, the anisotropy energy (MAE) is enhanced over one magnitude to ~106 erg/cm3 compared to La0.67Sr0.33MnO3 films. Moreover, the magnetic easy-axis is reversibly reoriented between (100)- and (110)-directions by changing the RTiO3. Using first-principles density functional theory calculations, we find that the SrIrO3 owns a large single-ion anisotropy due to its strong spin-orbit interaction. This anisotropy can be reversibly controlled by the OOC, then reorient the easy-axis of the superlattices. Additionally, it enlarges the MAE of the films via the cooperation with a robust orbital hybridization between the Ir and Mn atoms. Our results indicate that the tricolor superlattices consisting of 3d and 5d oxides provide a powerful platform to study the MA and develop oxide-based spintronic devices.

cond-mat.str-el↗