arXiv subjects

Di Liang

Publications and source records attributed to Di Liang.

50 records · Page 3Linked to original sources

arXiv2024

Anything in Any Scene: Photorealistic Video Object Insertion

Realistic video simulation has shown significant potential across diverse applications, from virtual reality to film production. This is particularly true for scenarios where capturing videos in real-world settings is either impractical or expensive. Existing approaches in video simulation often fail to accurately model the lighting environment, represent the object geometry, or achieve high levels of photorealism. In this paper, we propose Anything in Any Scene, a novel and generic framework for realistic video simulation that seamlessly inserts any object into an existing dynamic video with a strong emphasis on physical realism. Our proposed general framework encompasses three key processes: 1) integrating a realistic object into a given scene video with proper placement to ensure geometric realism; 2) estimating the sky and environmental lighting distribution and simulating realistic shadows to enhance the light realism; 3) employing a style transfer network that refines the final video output to maximize photorealism. We experimentally demonstrate that Anything in Any Scene framework produces simulated videos of great geometric realism, lighting realism, and photorealism. By significantly mitigating the challenges associated with video data generation, our framework offers an efficient and cost-effective solution for acquiring high-quality videos. Furthermore, its applications extend well beyond video data augmentation, showing promising potential in virtual reality, video editing, and various other video-centric applications. Please check our project website https://anythinginanyscene.github.io for access to our project code and more high-resolution video results.

cs.CV↗

arXiv2024

Silicon Optical Memory: Non-Volatile Optoelectronic Devices via Si-SiO$_2$ Hysteresis Effect

Implementing on-chip non-volatile optical memories has long been an actively pursued goal, promising significant enhancements in the capability and energy efficiency of photonic integrated circuits. Here, a novel optical memory has been demonstrated exclusively using the semiconductor primary material, silicon. By manipulating the optoelectronic effect of this device, we introduce a hysteresis effect at the silicon-silicon oxide interface, which in turn demonstrates multi-level, non-volatile optical data storage with robust retention and endurance. This new silicon optical memory provides a distinctively simple and accessible route to realize optical data storage in standard silicon foundry processes.

physics.app-ph↗

arXiv2023

Frame-wise streaming end-to-end speaker diarization with non-autoregressive self-attention-based attractors

This work proposes a frame-wise online/streaming end-to-end neural diarization (FS-EEND) method in a frame-in-frame-out fashion. To frame-wisely detect a flexible number of speakers and extract/update their corresponding attractors, we propose to leverage a causal speaker embedding encoder and an online non-autoregressive self-attention-based attractor decoder. A look-ahead mechanism is adopted to allow leveraging some future frames for effectively detecting new speakers in real time and adaptively updating speaker attractors. The proposed method processes the audio stream frame by frame, and has a low inference latency caused by the look-ahead frames. Experiments show that, compared with the recently proposed block-wise online methods, our method FS-EEND achieves state-of-the-art diarization results, with a low inference latency and computational cost.

eess.AS↗

arXiv2023

Unleashing Potential of Evidence in Knowledge-Intensive Dialogue Generation

Incorporating external knowledge into dialogue generation (KIDG) is crucial for improving the correctness of response, where evidence fragments serve as knowledgeable snippets supporting the factual dialogue replies. However, introducing irrelevant content often adversely impacts reply quality and easily leads to hallucinated responses. Prior work on evidence retrieval and integration in dialogue systems falls short of fully leveraging existing evidence since the model fails to locate useful fragments accurately and overlooks hidden evidence labels within the KIDG dataset. To fully Unleash the potential of evidence, we propose a framework to effectively incorporate Evidence in knowledge-Intensive Dialogue Generation (u-EIDG). Specifically, we introduce an automatic evidence generation framework that harnesses the power of Large Language Models (LLMs) to mine reliable evidence veracity labels from unlabeled data. By utilizing these evidence labels, we train a reliable evidence indicator to effectively identify relevant evidence from retrieved passages. Furthermore, we propose an evidence-augmented generator with an evidence-focused attention mechanism, which allows the model to concentrate on evidenced segments. Experimental results on MultiDoc2Dial demonstrate the efficacy of evidential label augmentation and refined attention mechanisms in improving model performance. Further analysis confirms that the proposed method outperforms other baselines (+3~+5 points) regarding coherence and factual consistency.

cs.CL↗

arXiv2023

Improving Semantic Matching through Dependency-Enhanced Pre-trained Model with Adaptive Fusion

Transformer-based pre-trained models like BERT have achieved great progress on Semantic Sentence Matching. Meanwhile, dependency prior knowledge has also shown general benefits in multiple NLP tasks. However, how to efficiently integrate dependency prior structure into pre-trained models to better model complex semantic matching relations is still unsettled. In this paper, we propose the \textbf{D}ependency-Enhanced \textbf{A}daptive \textbf{F}usion \textbf{A}ttention (\textbf{DAFA}), which explicitly introduces dependency structure into pre-trained models and adaptively fuses it with semantic information. Specifically, \textbf{\emph{(i)}} DAFA first proposes a structure-sensitive paradigm to construct a dependency matrix for calibrating attention weights. It adopts an adaptive fusion module to integrate the obtained dependency information and the original semantic signals. Moreover, DAFA reconstructs the attention calculation flow and provides better interpretability. By applying it on BERT, our method achieves state-of-the-art or competitive performance on 10 public datasets, demonstrating the benefits of adaptively fusing dependency structure in semantic matching task.

cs.CL↗

arXiv2023

Energy-Efficient Photonic Memory Based on Electrically Programmable Embedded III-V/Si Memristors: Switches and Filters

We demonstrate non-volatile optical functionality by embedding multi-layer $HfO_2/Al_2O_3$ memristors with III-V/Si photonics. The wafer-bonded III-V/Si memristor facilitates non-volatile optical functionality for a variety of devices such as Mach-Zehnder Interferometers (MZIs), and (de-)interleaver filters. The MZI optical memristor exhibits non-volatile optical phase shifts $> π(Δn_{g} > 2.70 \times 10^{-3}$) with ~ 30 dB extinction ratio while consuming 0 electrical power consumption in a true "set-and-forget" operation. We demonstrate 6 non-volatile states with each state capable of 4 Gbps modulation. III-V/Si (de-)interleavers were also demonstrated to exhibit memristive non-volatile passband transformation with full set/reset states. Time duration tests were performed on all devices and indicated non-volatility up to 24 hours and most likely beyond. To the best of our knowledge, we have demonstrated for the first time, non-volatile III-V/Si optical memristors with the largest electric-field driven phase shifts and reconfigurable filters with the lowest power consumption.

physics.optics↗

arXiv2023

Non-volatile heterogeneous III-V/Si photonics via optical charge-trap memory

We demonstrate, for the first time, non-volatile charge-trap flash memory (CTM) co-located with heterogeneous III-V/Si photonics. The wafer-bonded III-V/Si CTM cell facilitates non-volatile optical functionality for a variety of devices such as Mach-Zehnder Interferometers (MZIs), asymmetric MZI lattice filters, and ring resonator filters. The MZI CTM exhibits full write/erase operation (100 cycles with 500 states) with wavelength shifts of $Δλ_{non-volatile} = 1.16 nm$ ($Δn_{eff,non-volatile} ~ 2.5 \times 10^{-4}$) and a dynamic power consumption $<$ 20 pW (limited by measurement). Multi-bit write operation (2 bits) is also demonstrated and verified over a time duration of 24 hours and most likely beyond. The cascaded 2nd order ring resonator CTM filter exhibited an improved ER of ~ 7.11 dB compared to the MZI and wavelength shifts of $Δλ_{non-volatile} = 0.041 nm$ ($Δn_{eff, non-volatile} = 1.5 \times 10^{-4}$) with similar pW-level dynamic power consumption as the MZI CTM. The ability to co-locate photonic computing elements and non-volatile memory provides an attractive path towards eliminating the von-Neumann bottleneck.

physics.optics↗

arXiv2023

High-Speed and Energy-Efficient Non-Volatile Silicon Photonic Memory Based on Heterogeneously Integrated Memresonator

Recently, interest in programmable photonics integrated circuits has grown as a potential hardware framework for deep neural networks, quantum computing, and field programmable arrays (FPGAs). However, these circuits are constrained by the limited tuning speed and large power consumption of the phase shifters used. In this paper, introduced for the first time are memresonators, or memristors heterogeneously integrated with silicon photonic microring resonators, as phase shifters with non-volatile memory. These devices are capable of retention times of 12 hours, switching voltages lower than 5 V, an endurance of 1,000 switching cycles. Also, these memresonators have been switched using voltage pulses as short as 300 ps with a record low switching energy of 0.15 pJ. Furthermore, these memresonators are fabricated on a heterogeneous III-V/Si platform capable of integrating a rich family of active, passive, and non-linear optoelectronic devices, such as lasers and detectors, directly on-chip to enable in-memory photonic computing and further advance the scalability of integrated photonic processor circuits.

physics.optics↗

arXiv2023

Fast and energy-efficient non-volatile III-V-on-silicon photonic phase shifter based on memristors

Silicon photonics has evolved from lab research to commercial products in the past decade as it plays an increasingly crucial role in data communication for next-generation data centers and high performance computing1. Recently, programmable silicon photonics has also found new applications in quantum2 and classical 3 information processing. A key component of programmable silicon photonic integrated circuits (PICs) is the phase shifter, traditionally realized via the thermo-optic or plasma dispersion effect which are weak, volatile, and power hungry. A non-volatile phase shifter can circumvent these limitations by requiring zero power to maintain the switched phases. Previously non-volatile phase modulation was achieved via phase-change4 or ferroelectric materials5, but the switching energy remains high (pico to nano joules) and the speed is slow (micro to milli seconds). Here, we report a non-volatile III-V-on-silicon photonic phase shifter based on HfO2 memristor with sub-pJ switching energy (~400fJ), representing over an order of magnitude improvement in energy efficiency compared to the state of the art. The non-volatile phase shifter can be switched reversibly using a single 100ns pulse and exhibits an excellent endurance over 800 cycles. This technology can enable future energy-efficient programmable PICs for data centers, optical neural networks, and quantum information processing.

physics.optics↗

arXiv2023

DABERT: Dual Attention Enhanced BERT for Semantic Matching

Transformer-based pre-trained language models such as BERT have achieved remarkable results in Semantic Sentence Matching. However, existing models still suffer from insufficient ability to capture subtle differences. Minor noise like word addition, deletion, and modification of sentences may cause flipped predictions. To alleviate this problem, we propose a novel Dual Attention Enhanced BERT (DABERT) to enhance the ability of BERT to capture fine-grained differences in sentence pairs. DABERT comprises (1) Dual Attention module, which measures soft word matches by introducing a new dual channel alignment mechanism to model affinity and difference attention. (2) Adaptive Fusion module, this module uses attention to learn the aggregation of difference and affinity features, and generates a vector describing the matching details of sentence pairs. We conduct extensive experiments on well-studied semantic matching and robustness test datasets, and the experimental results show the effectiveness of our proposed method.

cs.CL↗

arXiv2023

Time-aware Multiway Adaptive Fusion Network for Temporal Knowledge Graph Question Answering

Knowledge graphs (KGs) have received increasing attention due to its wide applications on natural language processing. However, its use case on temporal question answering (QA) has not been well-explored. Most of existing methods are developed based on pre-trained language models, which might not be capable to learn \emph{temporal-specific} presentations of entities in terms of temporal KGQA task. To alleviate this problem, we propose a novel \textbf{T}ime-aware \textbf{M}ultiway \textbf{A}daptive (\textbf{TMA}) fusion network. Inspired by the step-by-step reasoning behavior of humans. For each given question, TMA first extracts the relevant concepts from the KG, and then feeds them into a multiway adaptive module to produce a \emph{temporal-specific} representation of the question. This representation can be incorporated with the pre-trained KG embedding to generate the final prediction. Empirical results verify that the proposed model achieves better performance than the state-of-the-art models in the benchmark dataset. Notably, the Hits@1 and Hits@10 results of TMA on the CronQuestions dataset's complex questions are absolutely improved by 24\% and 10\% compared to the best-performing baseline. Furthermore, we also show that TMA employing an adaptive fusion mechanism can provide interpretability by analyzing the proportion of information in question representations.

cs.CL↗

arXiv2023

Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflicts

Transformer-based pre-trained models have achieved great improvements in semantic matching. However, existing models still suffer from insufficient ability to capture subtle differences. The modification, addition and deletion of words in sentence pairs may make it difficult for the model to predict their relationship. To alleviate this problem, we propose a novel Dual Path Modeling Framework to enhance the model's ability to perceive subtle differences in sentence pairs by separately modeling affinity and difference semantics. Based on dual-path modeling framework we design the Dual Path Modeling Network (DPM-Net) to recognize semantic relations. And we conduct extensive experiments on 10 well-studied semantic matching and robustness test datasets, and the experimental results show that our proposed method achieves consistent improvements over baselines.

cs.CL↗

arXiv2022

Robust Lottery Tickets for Pre-trained Language Models

Recent works on Lottery Ticket Hypothesis have shown that pre-trained language models (PLMs) contain smaller matching subnetworks(winning tickets) which are capable of reaching accuracy comparable to the original models. However, these tickets are proved to be notrobust to adversarial examples, and even worse than their PLM counterparts. To address this problem, we propose a novel method based on learning binary weight masks to identify robust tickets hidden in the original PLMs. Since the loss is not differentiable for the binary mask, we assign the hard concrete distribution to the masks and encourage their sparsity using a smoothing approximation of L0 regularization.Furthermore, we design an adversarial loss objective to guide the search for robust tickets and ensure that the tickets perform well bothin accuracy and robustness. Experimental results show the significant improvement of the proposed method over previous work on adversarial robustness evaluation.

cs.CL↗

arXiv2022

Searching for Optimal Subword Tokenization in Cross-domain NER

Input distribution shift is one of the vital problems in unsupervised domain adaptation (UDA). The most popular UDA approaches focus on domain-invariant representation learning, trying to align the features from different domains into similar feature distributions. However, these approaches ignore the direct alignment of input word distributions between domains, which is a vital factor in word-level classification tasks such as cross-domain NER. In this work, we shed new light on cross-domain NER by introducing a subword-level solution, X-Piece, for input word-level distribution shift in NER. Specifically, we re-tokenize the input words of the source domain to approach the target subword distribution, which is formulated and solved as an optimal transport problem. As this approach focuses on the input level, it can also be combined with previous DIRL methods for further improvement. Experimental results show the effectiveness of the proposed method based on BERT-tagger on four benchmark NER datasets. Also, the proposed method is proved to benefit DIRL methods such as DANN.

cs.CL↗