arXiv subjects

Zeyu Huang

Publications and source records attributed to Zeyu Huang.

46 records · Page 3Linked to original sources

arXiv2024

Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers

The feed-forward networks (FFNs) in transformers are recognized as a group of key-value neural memories to restore abstract high-level knowledge. In this work, we conduct an empirical ablation study on updating keys (the 1st layer in the FFNs layer) or values (the 2nd layer in the FFNs layer). We compare those two methods in various knowledge editing and fine-tuning tasks of large language models to draw insights to understand FFNs further. Code is available at $\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\,repo}$.

cs.CL↗

arXiv2024

Testing space-time non-commutativity with TianQin

The direct detection of gravitational waves offers a powerful tool to explore the nature of gravity and the structure of space-time. This paper focuses on the capabilities of space-based gravitational wave detectors in testing space-time non-commutativity. Our findings indicate that TianQin has the potential to impose constraints on the non-commutative scale at a sub-Planckian level using massive black hole binaries. Additionally, we have developed a pipeline tailored to this specific topic.

gr-qc↗

arXiv2023

Neutron Activation Background in the NvDEx Experiment

An extremely low-background environment is a crucial requirement for any neutrinoless double beta decay experiment. Neutrons are very difficult to stop, because they can pass through the shields and activate nuclei in the detector, even inside the fiducial volume itself. Using Geant4 simulations we have studied the neutron background for N$ν$DEx-100 and the most efficient way to reduce it. Using a 60 cm thick external HDPE shield the neutron background can be reduced down to $0.24\pm 0.06$ events/year, lower than the background rate due to natural radioactivity (0.42 events/year), which was used as a benchmark for these calculations. The amount of shielding material needed can be significantly reduced by placing HDPE in the empty space between the lead shield and the steel vessel; in this way, it is sufficient to add 20 cm external HDPE shield to reduce the neutron background down to $0.15\pm0.05$ events/year.

physics.ins-det↗

arXiv2023

ARO-Net: Learning Implicit Fields from Anchored Radial Observations

We introduce anchored radial observations (ARO), a novel shape encoding for learning implicit field representation of 3D shapes that is category-agnostic and generalizable amid significant shape variations. The main idea behind our work is to reason about shapes through partial observations from a set of viewpoints, called anchors. We develop a general and unified shape representation by employing a fixed set of anchors, via Fibonacci sampling, and designing a coordinate-based deep neural network to predict the occupancy value of a query point in space. Differently from prior neural implicit models that use global shape feature, our shape encoder operates on contextual, query-specific features. To predict point occupancy, locally observed shape information from the perspective of the anchors surrounding the input query point are encoded and aggregated through an attention module, before implicit decoding is performed. We demonstrate the quality and generality of our network, coined ARO-Net, on surface reconstruction from sparse point clouds, with tests on novel and unseen object categories, "one-shape" training, and comparisons to state-of-the-art neural and classical methods for reconstruction and tessellation.

cs.CV↗

arXiv2023

NIFT: Neural Interaction Field and Template for Object Manipulation

We introduce NIFT, Neural Interaction Field and Template, a descriptive and robust interaction representation of object manipulations to facilitate imitation learning. Given a few object manipulation demos, NIFT guides the generation of the interaction imitation for a new object instance by matching the Neural Interaction Template (NIT) extracted from the demos in the target Neural Interaction Field (NIF) defined for the new object. Specifically, the NIF is a neural field that encodes the relationship between each spatial point and a given object, where the relative position is defined by a spherical distance function rather than occupancies or signed distances, which are commonly adopted by conventional neural fields but less informative. For a given demo interaction, the corresponding NIT is defined by a set of spatial points sampled in the demo NIF with associated neural features. To better capture the interaction, the points are sampled on the Interaction Bisector Surface (IBS), which consists of points that are equidistant to the two interacting objects and has been used extensively for interaction representation. With both point selection and pointwise features defined for better interaction encoding, NIT effectively guides the feature matching in the NIFs of the new object instances such that the relative poses are optimized to realize the manipulation while imitating the demo interactions. Experiments show that our NIFT solution outperforms state-of-the-art imitation learning methods for object manipulation and generalizes better to objects from new categories.

cs.RO↗

arXiv2023

Transformer-Patcher: One Mistake worth One Neuron

Large Transformer-based Pretrained Language Models (PLMs) dominate almost all Natural Language Processing (NLP) tasks. Nevertheless, they still make mistakes from time to time. For a model deployed in an industrial environment, fixing these mistakes quickly and robustly is vital to improve user experiences. Previous works formalize such problems as Model Editing (ME) and mostly focus on fixing one mistake. However, the one-mistake-fixing scenario is not an accurate abstraction of the real-world challenge. In the deployment of AI services, there are ever-emerging mistakes, and the same mistake may recur if not corrected in time. Thus a preferable solution is to rectify the mistakes as soon as they appear nonstop. Therefore, we extend the existing ME into Sequential Model Editing (SME) to help develop more practical editing methods. Our study shows that most current ME methods could yield unsatisfying results in this scenario. We then introduce Transformer-Patcher, a novel model editor that can shift the behavior of transformer-based models by simply adding and training a few neurons in the last Feed-Forward Network layer. Experimental results on both classification and generation tasks show that Transformer-Patcher can successively correct up to thousands of errors (Reliability) and generalize to their equivalent inputs (Generality) while retaining the model's accuracy on irrelevant inputs (Locality). Our method outperforms previous fine-tuning and HyperNetwork-based methods and achieves state-of-the-art performance for Sequential Model Editing (SME). The code is available at https://github.com/ZeroYuHuang/Transformer-Patcher.

cs.CL↗

arXiv2022

Mixture of Attention Heads: Selecting Attention Heads Per Token

Mixture-of-Experts (MoE) networks have been proposed as an efficient way to scale up model capacity and implement conditional computing. However, the study of MoE components mostly focused on the feedforward layer in Transformer architecture. This paper proposes the Mixture of Attention Heads (MoA), a new architecture that combines multi-head attention with the MoE mechanism. MoA includes a set of attention heads that each has its own set of parameters. Given an input, a router dynamically selects a subset of $k$ attention heads per token. This conditional computation schema allows MoA to achieve stronger performance than the standard multi-head attention layer. Furthermore, the sparsely gated MoA can easily scale up the number of attention heads and the number of parameters while preserving computational efficiency. In addition to the performance improvements, MoA also automatically differentiates heads' utilities, providing a new perspective to discuss the model's interpretability. We conducted experiments on several important tasks, including Machine Translation and Masked Language Modeling. Experiments have shown promising results on several tasks against strong baselines that involve large and very deep models.

cs.CL↗

arXiv2020

Air-to-Ground Channel Characterization for Low-Height UAVs in Realistic Network Deployments

Due to the decrease in cost, size and weight, \acp{UAV} are becoming more and more popular for general-purpose civil and commercial applications. Provision of communication services to \acp{UAV} both for user data and control messaging by using off-the-shelf terrestrial cellular deployments introduces several technical challenges. In this paper, an approach to the air-to-ground channel characterization for low-height \acp{UAV} based on an extensive measurement campaign is proposed, giving special attention to the comparison of the results when a typical directional antenna for network deployments is used and when a quasi-omnidirectional one is considered. Channel characteristics like path loss, shadow fading, root mean square delay and Doppler frequency spreads and the K-factor are statistically characterized for different suburban scenarios.

eess.SP↗

arXiv2020

Graph2Plan: Learning Floorplan Generation from Layout Graphs

We introduce a learning framework for automated floorplan generation which combines generative modeling using deep neural networks and user-in-the-loop designs to enable human users to provide sparse design constraints. Such constraints are represented by a layout graph. The core component of our learning framework is a deep neural network, Graph2Plan, which converts a layout graph, along with a building boundary, into a floorplan that fulfills both the layout and boundary constraints. Given an input building boundary, we allow a user to specify room counts and other layout constraints, which are used to retrieve a set of floorplans, with their associated layout graphs, from a database. For each retrieved layout graph, along with the input boundary, Graph2Plan first generates a corresponding raster floorplan image, and then a refined set of boxes representing the rooms. Graph2Plan is trained on RPLAN, a large-scale dataset consisting of 80K annotated floorplans. The network is mainly based on convolutional processing over both the layout graph, via a graph neural network (GNN), and the input building boundary, as well as the raster floorplan images, via conventional image convolution.

cs.CV↗

arXiv2019

Entity Extraction with Knowledge from Web Scale Corpora

Entity extraction is an important task in text mining and natural language processing. A popular method for entity extraction is by comparing substrings from free text against a dictionary of entities. In this paper, we present several techniques as a post-processing step for improving the effectiveness of the existing entity extraction technique. These techniques utilise models trained with the web-scale corpora which makes our techniques robust and versatile. Experiments show that our techniques bring a notable improvement on efficiency and effectiveness.

cs.CL↗