arXiv subjects

Tao Jin

Publications and source records attributed to Tao Jin.

At least 91 records · Page 5Linked to original sources

arXiv2023

High Resolution Point Clouds from mmWave Radar

This paper explores a machine learning approach for generating high resolution point clouds from a single-chip mmWave radar. Unlike lidar and vision-based systems, mmWave radar can operate in harsh environments and see through occlusions like smoke, fog, and dust. Unfortunately, current mmWave processing techniques offer poor spatial resolution compared to lidar point clouds. This paper presents RadarHD, an end-to-end neural network that constructs lidar-like point clouds from low resolution radar input. Enhancing radar images is challenging due to the presence of specular and spurious reflections. Radar data also doesn't map well to traditional image processing techniques due to the signal's sinc-like spreading pattern. We overcome these challenges by training RadarHD on a large volume of raw I/Q radar data paired with lidar point clouds across diverse indoor settings. Our experiments show the ability to generate rich point clouds even in scenes unobserved during training and in the presence of heavy smoke occlusion. Further, RadarHD's point clouds are high-quality enough to work with existing lidar odometry and mapping workflows.

cs.RO↗

arXiv2023

Gloss Attention for Gloss-free Sign Language Translation

Most sign language translation (SLT) methods to date require the use of gloss annotations to provide additional supervision information, however, the acquisition of gloss is not easy. To solve this problem, we first perform an analysis of existing models to confirm how gloss annotations make SLT easier. We find that it can provide two aspects of information for the model, 1) it can help the model implicitly learn the location of semantic boundaries in continuous sign language videos, 2) it can help the model understand the sign language video globally. We then propose \emph{gloss attention}, which enables the model to keep its attention within video segments that have the same semantics locally, just as gloss helps existing models do. Furthermore, we transfer the knowledge of sentence-to-sentence similarity from the natural language model to our gloss attention SLT network (GASLT) to help it understand sign language videos at the sentence level. Experimental results on multiple large-scale sign language datasets show that our proposed GASLT model significantly outperforms existing methods. Our code is provided in \url{https://github.com/YinAoXiong/GASLT}.

cs.CV↗

arXiv2023

Chromium Self-Traceable Length Standard: Investigating Geometry and Diffraction for Length Traceability Chain

Natural constant-based metrology methods offer an effective approach to achieving traceability in nanometric measurements. The Cr grating, fabricated by atom lithography and featuring a pitch of $d=212.7705\pm0.0049~{\rm nm}$ traceable to the Cr transition frequency $^{7}S_{3}$ $\rightarrow$ $^{7}P_{4}^{0}$, demonstrates potential as a self-traceable length standard in nano-length metrology by grating interferometer. This research aims to analyze and engineer the diffraction characteristics that enhance the Cr grating as a self-traceable length standard within the length traceability chain based on the Cr transition frequency. Accordingly, we investigate the geometric morphology and diffraction characteristics of the Cr grating, analyzes the influence of the grating's polarization-sensitive characteristics on the Littrow configuration grating interferometer, and establishes the criteria for Cr grating fabrication. Experimentally, we fabricate an expanded Cr grating by scanning atom lithography, characterize its diffraction performance, and conduct preliminary verification of length measurement in a self-traceable grating interferometer. This work adheres to the international trend of flattened metrology development, offering a valuable reference for advancing subsequent metrological technologies throughout the new traceability chain.

physics.ins-det↗

arXiv2023

Nanometer displacement measurement based on metrological self-mixing grating interferometer traceable to the pitch standard of one-dimension chromium self-traceable grating

Traceability of precision instrument and measuring method is the core issue in metrology science. In the field of nanometer length measurement, the laser interferometers are usually used to trace the measurement value to the laser wavelength, but the laser wavelength is sensitive to the environment disturbance. Chromium self-traceable grating is an ideal nanometer length reference grating with pitch traceability, fabricated by the atomic lithography technique. The new nanometer length traceability chain can be established based on the pitch traceability of chromium self-traceable grating, which is often used to calibrate the systematic error of the atomic force microscope. In this paper, the metrological self-mixing grating interferometer based on the chromium self-traceable grating (SMGI-Cr) is firstly established, whose interfere phase is traceable to the pitch of the chromium self-traceable grating directly and traceable to the chromium atomic transition frequency of energy level 7 S 3 to 7 P 4 indirectly. The nanometer displacement measurement is also achieved by the SMGI-Cr. The measurement error is no more than 0.2366%, compared to a commercial interferometer.

physics.app-ph↗

arXiv2023

OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment

Speech Recognition builds a bridge between the multimedia streaming (audio-only, visual-only or audio-visual) and the corresponding text transcription. However, when training the specific model of new domain, it often gets stuck in the lack of new-domain utterances, especially the labeled visual utterances. To break through this restriction, we attempt to achieve zero-shot modality transfer by maintaining the multi-modality alignment in phoneme space learned with unlabeled multimedia utterances in the high resource domain during the pre-training \cite{shi2022learning}, and propose a training system Open-modality Speech Recognition (\textbf{OpenSR}) that enables the models trained on a single modality (e.g., audio-only) applicable to more modalities (e.g., visual-only and audio-visual). Furthermore, we employ a cluster-based prompt tuning strategy to handle the domain shift for the scenarios with only common words in the new domain utterances. We demonstrate that OpenSR enables modality transfer from one to any in three different settings (zero-, few- and full-shot), and achieves highly competitive zero-shot performance compared to the existing few-shot and full-shot lip-reading methods. To the best of our knowledge, OpenSR achieves the state-of-the-art performance of word error rate in LRS2 on audio-visual speech recognition and lip-reading with 2.7\% and 25.0\%, respectively. The code and demo are available at https://github.com/Exgc/OpenSR.

cs.CL↗

arXiv2023

DATE: Domain Adaptive Product Seeker for E-commerce

Product Retrieval (PR) and Grounding (PG), aiming to seek image and object-level products respectively according to a textual query, have attracted great interest recently for better shopping experience. Owing to the lack of relevant datasets, we collect two large-scale benchmark datasets from Taobao Mall and Live domains with about 474k and 101k image-query pairs for PR, and manually annotate the object bounding boxes in each image for PG. As annotating boxes is expensive and time-consuming, we attempt to transfer knowledge from annotated domain to unannotated for PG to achieve un-supervised Domain Adaptation (PG-DA). We propose a {\bf D}omain {\bf A}daptive Produc{\bf t} S{\bf e}eker ({\bf DATE}) framework, regarding PR and PG as Product Seeking problem at different levels, to assist the query {\bf date} the product. Concretely, we first design a semantics-aggregated feature extractor for each modality to obtain concentrated and comprehensive features for following efficient retrieval and fine-grained grounding tasks. Then, we present two cooperative seekers to simultaneously search the image for PR and localize the product for PG. Besides, we devise a domain aligner for PG-DA to alleviate uni-modal marginal and multi-modal conditional distribution shift between source and target domains, and design a pseudo box generator to dynamically select reliable instances and generate bounding boxes for further knowledge transfer. Extensive experiments show that our DATE achieves satisfactory performance in fully-supervised PR, PG and un-supervised PG-DA. Our desensitized datasets will be publicly available here\footnote{\url{https://github.com/Taobao-live/Product-Seeking}}.

cs.CV↗

arXiv2023

MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition

Multi-media communications facilitate global interaction among people. However, despite researchers exploring cross-lingual translation techniques such as machine translation and audio speech translation to overcome language barriers, there is still a shortage of cross-lingual studies on visual speech. This lack of research is mainly due to the absence of datasets containing visual speech and translated text pairs. In this paper, we present \textbf{AVMuST-TED}, the first dataset for \textbf{A}udio-\textbf{V}isual \textbf{Mu}ltilingual \textbf{S}peech \textbf{T}ranslation, derived from \textbf{TED} talks. Nonetheless, visual speech is not as distinguishable as audio speech, making it difficult to develop a mapping from source speech phonemes to the target language text. To address this issue, we propose MixSpeech, a cross-modality self-learning framework that utilizes audio speech to regularize the training of visual speech tasks. To further minimize the cross-modality gap and its impact on knowledge transfer, we suggest adopting mixed speech, which is created by interpolating audio and visual streams, along with a curriculum learning strategy to adjust the mixing ratio as needed. MixSpeech enhances speech translation in noisy environments, improving BLEU scores for four languages on AVMuST-TED by +1.4 to +4.2. Moreover, it achieves state-of-the-art performance in lip reading on CMLR (11.1\%), LRS2 (25.5\%), and LRS3 (28.0\%).

cs.CV↗

arXiv2023

Length traceability chain based on chromium atom transition frequency

Precise positioning measurement plays an important role in in today advanced manufacturing industry, and length traceability chain has been optimizing and enriching to fulfill the developing and various precise positioning requirement. In this paper, we propose a new length traceability chain based on chromium atom transition frequency, which is a combining utilization of fundamental physical constant accuracy and grating interferometer environmental robustness. The selftraceable grating pitch standard, the selftraceable angle standard and the selftraceable grating interferometer are promising to improve the measurement accuracy, consistency and selfcalibration ability in situ for precise positioning.

physics.ins-det↗

arXiv2022

Recent experimental progress on probing Lorentz Violation in pure gravity for d=6

Short-range experiment with striped design of the test and source masses offers uniquely sensitive probes for Lorentz Violation. In our previous work, we proposed to combine the horizontally and vertically striped experiments to constrain the violating parameters. Here, we further point out adopting the vertically striped structure with the experimental setup placed at two azimuth angles, which has more experimental operability. For the experiment, we have completed the assembly and measurement of the torsion pendulum.

hep-ph↗

arXiv2022

A Multi-Resolution Approach to Hydraulic Fracture Simulation

We present a multi-resolution approach for constructing model-based simulations of hydraulic fracturing, wherein flow through porous media is coupled with fluid-driven fracture. The approach consists of a hybrid scheme that couples a discrete crack representation in a global domain to a phase-field representation in a local subdomain near the crack tip. The multi-resolution approach addresses issues such as the computational expense of accurate hydraulic fracture simulations and the difficulties associated with reconstructing crack apertures from diffuse fracture representations. In the global domain, a coupled system of equations for displacements and pressures is considered. The crack geometry is assumed to be fixed and the displacement field is enriched with discontinuous functions. Around the crack tips in the local subdomains, phase-field sub-problems are instantiated on the fly to propagate fractures in arbitrary, mesh independent directions. The governing equations and fields in the global and local domains are approximated using a combination of finite-volume and finite element discretizations. The efficacy of the method is illustrated through various benchmark problems in hydraulic fracturing, as well as a new study of fluid-driven crack growth around a stiff inclusion.

cs.CE↗

arXiv2022

On Schur Algebras and Derivations of Free Lie Algebras

We investigate the action of Schur algebra on the Lie algebras of derivations of free Lie algebras and operad structures constructed from it. We also show that the Lie algebra of derivations is generated by quadratic derivations together with the action of the Schur operad. Applications to certain subgroups of the automorphism group of a finitely generated free group are given as well.

math.GR↗

arXiv2021

Adaptive Sampling for Heterogeneous Rank Aggregation from Noisy Pairwise Comparisons

In heterogeneous rank aggregation problems, users often exhibit various accuracy levels when comparing pairs of items. Thus a uniform querying strategy over users may not be optimal. To address this issue, we propose an elimination-based active sampling strategy, which estimates the ranking of items via noisy pairwise comparisons from users and improves the users' average accuracy by maintaining an active set of users. We prove that our algorithm can return the true ranking of items with high probability. We also provide a sample complexity bound for the proposed algorithm which is better than that of non-active strategies in the literature. Experiments are provided to show the empirical advantage of the proposed methods over the state-of-the-art baselines.

cs.LG↗

arXiv2020

SBAT: Video Captioning with Sparse Boundary-Aware Transformer

In this paper, we focus on the problem of applying the transformer structure to video captioning effectively. The vanilla transformer is proposed for uni-modal language generation task such as machine translation. However, video captioning is a multimodal learning problem, and the video features have much redundancy between different time steps. Based on these concerns, we propose a novel method called sparse boundary-aware transformer (SBAT) to reduce the redundancy in video representation. SBAT employs boundary-aware pooling operation for scores from multihead attention and selects diverse features from different scenarios. Also, SBAT includes a local correlation scheme to compensate for the local information loss brought by sparse operation. Based on SBAT, we further propose an aligned cross-modal encoding scheme to boost the multimodal interaction. Experimental results on two benchmark datasets show that SBAT outperforms the state-of-the-art methods under most of the metrics.

cs.CV↗

arXiv2019

Rank Aggregation via Heterogeneous Thurstone Preference Models

We propose the Heterogeneous Thurstone Model (HTM) for aggregating ranked data, which can take the accuracy levels of different users into account. By allowing different noise distributions, the proposed HTM model maintains the generality of Thurstone's original framework, and as such, also extends the Bradley-Terry-Luce (BTL) model for pairwise comparisons to heterogeneous populations of users. Under this framework, we also propose a rank aggregation algorithm based on alternating gradient descent to estimate the underlying item scores and accuracy levels of different users simultaneously from noisy pairwise comparisons. We theoretically prove that the proposed algorithm converges linearly up to a statistical error which matches that of the state-of-the-art method for the single-user BTL model. We evaluate the proposed HTM model and algorithm on both synthetic and real data, demonstrating that it outperforms existing methods.

cs.LG↗

arXiv2019

Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning

This paper addresses the challenging task of video captioning which aims to generate descriptions for video data. Recently, the attention-based encoder-decoder structures have been widely used in video captioning. In existing literature, the attention weights are often built from the information of an individual modality, while, the association relationships between multiple modalities are neglected. Motivated by this observation, we propose a video captioning model with High-Order Cross-Modal Attention (HOCA) where the attention weights are calculated based on the high-order correlation tensor to capture the frame-level cross-modal interaction of different modalities sufficiently. Furthermore, we novelly introduce Low-Rank HOCA which adopts tensor decomposition to reduce the extremely large space requirement of HOCA, leading to a practical and efficient implementation in real-world applications. Experimental results on two benchmark datasets, MSVD and MSR-VTT, show that Low-rank HOCA establishes a new state-of-the-art.

cs.LG↗

arXiv2019

Inpatient2Vec: Medical Representation Learning for Inpatients

Representation learning (RL) plays an important role in extracting proper representations from complex medical data for various analyzing tasks, such as patient grouping, clinical endpoint prediction and medication recommendation. Medical data can be divided into two typical categories, outpatient and inpatient, that have different data characteristics. However, few of existing RL methods are specially designed for inpatients data, which have strong temporal relations and consistent diagnosis. In addition, for unordered medical activity set, existing medical RL methods utilize a simple pooling strategy, which would result in indistinguishable contributions among the activities for learning. In this work, weproposeInpatient2Vec, anovelmodel for learning three kinds of representations for inpatient, including medical activity, hospital day and diagnosis. A multi-layer self-attention mechanism with two training tasks is designed to capture the inpatient data characteristics and process the unordered set. Using a real-world dataset, we demonstrate that the proposed approach outperforms the competitive baselines on semantic similarity measurement and clinical events prediction tasks.

cs.LG↗

arXiv2016

Precision requirements and innovative manufacturing for ultrahigh precision laser interferometry of gravitational-wave astronomy

With the LIGO announcement of the first direct detection of gravitational waves (GWs), the GW Astronomy was formally ushered into our age. After one-hundred years of theoretical investigation and fifty years of experimental endeavor, this is a historical landmark not just for physics and astronomy, but also for industry and manufacturing. The challenge and opportunity for industry is precision and innovative manufacturing in large size - production of large and homogeneous optical components, optical diagnosis of large components, high reflectance dielectric coating on large mirrors, manufacturing of components for ultrahigh vacuum of large volume, manufacturing of high attenuating vibration isolation system, production of high-power high-stability single-frequency lasers, production of high-resolution positioning systems etc. In this talk, we address the requirements and methods to satisfy these requirements. Optical diagnosis of large optical components requires large phase-shifting interferometer; the 1.06 μm Phase Shifting Interferometer for testing LIGO optics is an example. High quality mirrors are crucial for laser interferometric GW detection, so as for ring laser gyroscope, high precision laser stabilization via optical cavities, quantum optomechanics, cavity quantum electrodynamics and vacuum birefringence measurement. There are stringent requirements on the substrate materials and coating methods. For cryogenic GW interferometer, appropriate coating on sapphire or silicon are required for good thermal and homogeneity properties. Large ultrahigh vacuum components and high attenuating vibration system together with an efficient metrology system are required and will be addressed. For space interferometry, drag-free technology is well-developed; weak-light phase locking is demonstrated in the laboratories while weak-light manipulation technology still needs developments.

astro-ph.IM↗

arXiv2016

Online Bayesian Collaborative Topic Regression

Collaborative Topic Regression (CTR) combines ideas of probabilistic matrix factorization (PMF) and topic modeling (e.g., LDA) for recommender systems, which has gained increasing successes in many applications. Despite enjoying many advantages, the existing CTR algorithms have some critical limitations. First of all, they are often designed to work in a batch learning manner, making them unsuitable to deal with streaming data or big data in real-world recommender systems. Second, the document-specific topic proportions of LDA are fed to the downstream PMF, but not reverse, which is sub-optimal as the rating information is not exploited in discovering the low-dimensional representation of documents and thus can result in a sub-optimal representation for prediction. In this paper, we propose a novel scheme of Online Bayesian Collaborative Topic Regression (OBCTR) which is efficient and scalable for learning from data streams. Particularly, we {\it jointly} optimize the combined objective function of both PMF and LDA in an online learning fashion, in which both PMF and LDA tasks can be reinforced each other during the online learning process. Our encouraging experimental results on real-world data validate the effectiveness of the proposed method.

cs.LG↗