arXiv subjects

Tianyu Huang

Publications and source records attributed to Tianyu Huang.

61 records · Page 4Linked to original sources

arXiv2023

CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training

Pre-training across 3D vision and language remains under development because of limited training data. Recent works attempt to transfer vision-language pre-training models to 3D vision. PointCLIP converts point cloud data to multi-view depth maps, adopting CLIP for shape classification. However, its performance is restricted by the domain gap between rendered depth maps and images, as well as the diversity of depth distributions. To address this issue, we propose CLIP2Point, an image-depth pre-training method by contrastive learning to transfer CLIP to the 3D domain, and adapt it to point cloud classification. We introduce a new depth rendering setting that forms a better visual effect, and then render 52,460 pairs of images and depth maps from ShapeNet for pre-training. The pre-training scheme of CLIP2Point combines cross-modality learning to enforce the depth features for capturing expressive visual and textual features and intra-modality learning to enhance the invariance of depth aggregation. Additionally, we propose a novel Dual-Path Adapter (DPA) module, i.e., a dual-path structure with simplified adapters for few-shot learning. The dual-path structure allows the joint use of CLIP and CLIP2Point, and the simplified adapter can well fit few-shot tasks without post-search. Experimental results show that CLIP2Point is effective in transferring CLIP knowledge to 3D vision. Our CLIP2Point outperforms PointCLIP and other self-supervised 3D networks, achieving state-of-the-art results on zero-shot and few-shot classification.

cs.CV↗

arXiv2023

Machine Learning Based Alignment For LCLS-II-HE Optics

The hard X-ray instruments at the Linac Coherent Light Source are in the design phase for upgrades that will take full advantage of the high repetition rates that will become available with LCLS-II-HE. The current X-ray Correlation Spectroscopy instrument will be converted to the Dynamic X-ray Scattering instrument, and will feature a meV-scale high-resolution monochromator at its front end with unprecedented coherent flux. With the new capability come many engineering and design challenges, not least of which is the sensitivity to long-term drift of the optics. With this in mind, we have estimated the system tolerance to angular drift and vibration for all the relevant optics ($\sim$10 components) in terms of how the central energy out of the monochromator will be affected to inform the mechanical design. Additionally, we have started planning for methods to correct for such drifts using available (both invasive and non-invasive) X-ray beam diagnostics. In simulations, we have demonstrated the ability of trained Machine Learning models to correct misalignments to maintain the desired central energy and optical axis within the necessary tolerances. Additionally, we exhibit the use of Bayesian Optimization to minimize the impact of thermal deformations of crystals as well as beam alignment from scratch. The initial results are very promising and efforts to further extend this work are ongoing.

physics.acc-ph↗

arXiv2023

Record Deduplication for Entity Distribution Modeling in ASR Transcripts

Voice digital assistants must keep up with trending search queries. We rely on a speech recognition model using contextual biasing with a rapidly updated set of entities, instead of frequent model retraining, to keep up with trends. There are several challenges with this approach: (1) the entity set must be frequently reconstructed, (2) the entity set is of limited size due to latency and accuracy trade-offs, and (3) finding the true entity distribution for biasing is complicated by ASR misrecognition. We address these challenges and define an entity set by modeling customers true requested entity distribution from ASR output in production using record deduplication, a technique from the field of entity resolution. Record deduplication resolves or deduplicates coreferences, including misrecognitions, of the same latent entity. Our method successfully retrieves 95% of misrecognized entities and when used for contextual biasing shows an estimated 5% relative word error rate reduction.

cs.CL↗

arXiv2022

Symmetry-Aware Transformer-based Mirror Detection

Mirror detection aims to identify the mirror regions in the given input image. Existing works mainly focus on integrating the semantic features and structural features to mine specific relations between mirror and non-mirror regions, or introducing mirror properties like depth or chirality to help analyze the existence of mirrors. In this work, we observe that a real object typically forms a loose symmetry relationship with its corresponding reflection in the mirror, which is beneficial in distinguishing mirrors from real objects. Based on this observation, we propose a dual-path Symmetry-Aware Transformer-based mirror detection Network (SATNet), which includes two novel modules: Symmetry-Aware Attention Module (SAAM) and Contrast and Fusion Decoder Module (CFDM). Specifically, we first adopt a transformer backbone to model global information aggregation in images, extracting multi-scale features in two paths. We then feed the high-level dual-path features to SAAMs to capture the symmetry relations. Finally, we fuse the dual-path features and refine our prediction maps progressively with CFDMs to obtain the final mirror mask. Experimental results show that SATNet outperforms both RGB and RGB-D mirror detection methods on all available mirror detection datasets. Codes and trained models are available at: https://github.com/tyhuang0428/SATNet.

cs.CV↗

arXiv2022

AIMusicGuru: Music Assisted Human Pose Correction

Pose Estimation techniques rely on visual cues available through observations represented in the form of pixels. But the performance is bounded by the frame rate of the video and struggles from motion blur, occlusions, and temporal coherence. This issue is magnified when people are interacting with objects and instruments, for example playing the violin. Standard approaches for postprocessing use interpolation and smoothing functions to filter noise and fill gaps, but they cannot model highly non-linear motion. We present a method that leverages our understanding of the high degree of a causal relationship between the sound produced and the motion that produces them. We use the audio signature to refine and predict accurate human body pose motion models. We propose MAPnet (Music Assisted Pose network) for generating a fine grain motion model from sparse input pose sequences but continuous audio. To accelerate further research in this domain, we also open-source MAPdat, a new multi-modal dataset of 3D violin playing motion with music. We perform a comparison of different standard machine learning models and perform analysis on input modalities, sampling techniques, and audio and motion features. Experiments on MAPdat suggest multi-modal approaches like ours as a promising direction for tasks previously approached with visual methods only. Our results show both qualitatively and quantitatively how audio can be combined with visual observation to help improve any pose estimation methods.

cs.CV↗

arXiv2020

Intelligent multiscale simulation based on process-guided composite database

In the paper, we present an integrated data-driven modeling framework based on process modeling, material homogenization, mechanistic machine learning, and concurrent multiscale simulation. We are interested in the injection-molded short fiber reinforced composites, which have been identified as key material systems in automotive, aerospace, and electronics industries. The molding process induces spatially varying microstructures across various length scales, while the resulting strongly anisotropic and nonlinear material properties are still challenging to be captured by conventional modeling approaches. To prepare the linear elastic training data for our machine learning tasks, Representative Volume Elements (RVE) with different fiber orientations and volume fractions are generated through stochastic reconstruction. More importantly, we utilize the recently proposed Deep Material Network (DMN) to learn the hidden microscale morphologies from data. With essential physics embedded in its building blocks, this data-driven material model can be extrapolated to predict nonlinear material behaviors efficiently and accurately. Through the transfer learning of DMN, we create a unified process-guided material database that covers a full range of geometric descriptors for short fiber reinforced composites. Finally, this unified DMN database is implemented and coupled with macroscale finite element model to enable concurrent multiscale simulations. From our perspective, the proposed framework is also promising in many other emergent multiscale engineering systems, such as additive manufacturing and compressive molding.

cs.CE↗

arXiv2017

To Bond or not to Bond: An Optimal Channel Allocation Algorithm For Flexible Dynamic Channel Bonding in WLANs

IEEE 802.11 has evolved from 802.11a/b/g/n to 802.11ac to meet rapidly increasing data rate requirements in WLANs. One important technique adopted in 802.11ac is the channel bonding (CB) scheme that combines multiple 20MHz channels for a single transmission in 5GHz band. In order to effectively access channel after a series of contention operations, 802.11ac specifies two different CB operations: Dynamic Channel Bonding (DCB) and Static Channel Bonding (SCB). This paper proposes an optimal channel allocation algorithm to achieve maximal throughputs in DCB WLANs. Specifically, we first adopt a continuous-time Markov Chain (CTMC) model to analyze the equilibrium throughputs. Based on the throughput analysis, we then construct an integer nonlinear programming (INLP) model with the target of maximizing system throughputs. By solving the INLP model, we then propose an optimal channel allocation algorithm based on the Branch-and-Bound Method (BBM). It turns out that the maximal throughput performance can be achieved under the channel allocation scheme with the least overlapped channels among WLANs. Simulations show that the proposed channel allocation algorithm can achieve the maximal system throughput under various network settings. Our analysis on the optimal channel allocation schemes brings new insights into the design and optimization of future WLANs, especially for those adopting channel bonding technique.

cs.NI↗