arXiv subjects

Ting Huang

Publications and source records attributed to Ting Huang.

53 records · Page 3Linked to original sources

arXiv2024

InternLM2 Technical Report

The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advancements in open-source models has been challenging. This paper introduces InternLM2, an open-source LLM that outperforms its predecessors in comprehensive evaluations across 6 dimensions and 30 benchmarks, long-context modeling, and open-ended subjective evaluations through innovative pre-training and optimization techniques. The pre-training process of InternLM2 is meticulously detailed, highlighting the preparation of diverse data types including text, code, and long-context data. InternLM2 efficiently captures long-term dependencies, initially trained on 4k tokens before advancing to 32k tokens in pre-training and fine-tuning stages, exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test. InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF) strategy that addresses conflicting human preferences and reward hacking. By releasing InternLM2 models in different training stages and model sizes, we provide the community with insights into the model's evolution.

cs.CL↗

arXiv2024

AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training

Training large language models (LLMs) encounters challenges in GPU memory consumption due to the high memory requirements of model states. The widely used Zero Redundancy Optimizer (ZeRO) addresses this issue through strategic sharding but introduces communication challenges at scale. To tackle this problem, we propose AMSP, a system designed to optimize ZeRO for scalable LLM training. AMSP incorporates three flexible sharding strategies: Full-Replica, Full-Sharding, and Partial-Sharding, and allows each component within the model states (Parameters, Gradients, Optimizer States) to independently choose a sharding strategy as well as the device mesh. We conduct a thorough analysis of communication costs, formulating an optimization problem to discover the optimal sharding strategy. Additionally, AMSP optimizes distributed LLM training by efficiently overlapping communication with computation. Evaluations demonstrate up to 52\% Model FLOPs Utilization (MFU) when training the LLaMA-based model on 1024 GPUs, resulting in a 1.56 times improvement in training throughput compared to newly proposed systems like MiCS and ZeRO++.

cs.DC↗

arXiv2024

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding

Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are neither efficient nor compatible with commonly-used training algorithms such as FlashAttention. We design InternEvo to address these issues. InternEvo decouples all of the sharding dimensions into a new hierarchical space, and systematically analyzes the memory and communication cost of LLM training. Then, it generates an effective hybrid parallelism strategy. We design a new selective overlap mechanism to mitigate the communication overhead introduced by the hybrid parallelism. We also implement memory management techniques to reduce GPU memory fragmentation. Evaluation results show that InternEvo generates parallelization strategies that match or outperform existing methods in model FLOPs utilization.

cs.DC↗

arXiv2023

A quantitative pairwise comparison-based constraint handling technique for constrained optimization

This study proposes a new constraint handling technique for assisting metaheuristic optimization algorithms to solve constrained optimization problems more effectively and efficiently. Given any two solutions of any constrained optimization problems, they are first mapped into a two-dimensional Cartesian coordinate system with their objective function value differences and constraint violation differences as the two axes. To the best of our knowledge, we are the first to deal with constraints by building such a Cartesian coordinate system. Then, the Cartesian coordinate system is divided into a series of grids by assigning ranks to different intervals of differences. In this way, a pairwise comparison criterion is derived with the use of the fused ranks, which achieves non-hierarchical comparison neither preferring objective function values nor constraint violations, resulting in more accurate evaluation compared with existing techniques. Moreover, an evaluation function that is equivalent to the pairwise comparison criterion is proposed, which further improves computational efficiency. The effectiveness and efficiency of the proposed constraint handling technique are verified on two well-known public datasets, that is, CEC 2006 and CEC 2017. The results demonstrate that metaheuristic optimization algorithms with using the proposed constraint handling technique can converge to a feasible optimal solution faster and more reliably. Experimental analysis on the parameters involved reveal guidance for their optimal settings.

math.OC↗

arXiv2023

On-chip arrayed waveguide grating fabricated on thin film lithium niobate

We design an on-chip 8-channel TFLN AWG and fabricate the device using photolithography assisted chemo-mechanical etching (PLACE) technique. We experimentally measure the transmission of the fabricated TFLN AWG near the central wavelength of 1550 nm. We obtain an on-chip loss as low as 3.32 dB, a single-channel bandwidth of 1.6 nm and a total-channel bandwidth of 12.8 nm. The crosstalk between adjacent channels was measured to be below -7.01 dB within the wavelength range from 1543 nm to 1558 nm, and the crosstalk between non-adjacent channels was below -15 dB.

physics.optics↗

arXiv2022

Monolithically integrated active passive waveguide array fabricated on thin film lithium niobate using a single continuous photolithography process

We demonstrate a robust low-loss optical interface by tiling passive (i.e., without doping of active ions) thin film lithium niobate (TFLN) and active (i.e., doped with rare earth ions) TFLN substrates for monolithic integration of passive/active lithium niobate photonics. The tiled substrates composed of both active and passive areas allow to pattern the mask of the integrated active passive photonic device at once using a single continuous photolithography process. The interface loss of tiled substrate is measured as low as 0.26 dB. Thanks to the stability provided by this approach, a four-channel waveguide amplifier is realized in a straightforward manner, which shows a net gain of ~5 dB at 1550-nm wavelength and that of ~8 dB at 1530-nm wavelength for each channel. The robust low-loss optical interface for passive/active photonic integration will facilitate large-scale high performance photonic devices which require on-chip light sources and amplifiers.

physics.optics↗

arXiv2022

Electrically driven compact hybrid lithium niobate microring laser

We demonstrate an electrically driven compact hybrid lithium niobate microring laser by butt coupling a commercial 980-nm pump laser diode chip with a high quality Er3+-doped lithium niobate microring chip. Single mode lasing emission at 1531 nm wavelength from the Er3+-doped lithium niobate microring can be observed with the integrated 980-nm laser pumping. The compact hybrid lithium niobate microring laser occupies the chip size of 3 mmx4mmx0.5 mm. The threshold pumping laser power is 6 mW and the threshold current is 0.5 A (operating voltage 1.64 V) in the atmospheric temperature. The spectrum featuring single mode lasing with small linewidth of 0.05 nm is observed. This work explores a robust hybrid lithium niobate microring laser source which has potential applications in coherent optical communication and precision metrology.

physics.optics↗

arXiv2022

On-chip integrated Yb3+-doped waveguide amplifiers on thin film lithium niobate

We report the fabrication and optical characterization of Yb3+-doped waveguide amplifiers (YDWA) on the thin film lithium niobate fabricated by photolithography assisted chemo-mechanical etching. The fabricated Yb3+-doped lithium niobate waveguides demonstrates low propagation loss of 0.13 dB/cm at 1030 nm and 0.1 dB/cm at 1060 nm. The internal net gain of 5 dB at 1030 nm and 8 dB at 1060 nm are measured on a 4.0 cm long waveguide pumped by 976nm laser diodes, indicating the gain per unit length of 1.25 dB/cm at 1030 nm and 2 dB/cm at 1060 nm, respectively. The integrated Yb3+-doped lithium niobate waveguide amplifiers will benefit the development of a powerful gain platform and are expected to contribute to the high-density integration of thin film lithium niobate based photonic chip.

physics.optics↗

arXiv2021

On-chip multi-color microdisk laser on Yb3+-doped thin-film lithium niobate

We demonstrate an on-chip Yb3+-doped lithium niobate (LN) microdisk laser. The intrinsic quality factors of the fabricated Yb3+-doped LN microdisk resonator are measured up to 3.79x10^5 at 976 nm wavelength and 1.1x10^6 at 1514 nm wavelength. The multi-mode laser emissions are obtained in a band from 1020 nm to 1070 nm pumped by 984 nm laser and with the low threshold of 103 μW, resulting in a slope efficiency of 0.53% at room temperature. Furthermore, the second-harmonic frequency of pump light and the sum-frequency of the pump light and laser emissions are both generated in the on-chip Yb3+-doped LN microdisk benefited from the strong \c{hi}(2) nonlinearity of LN. These microdisk lasers are expected to contribute to the high-density integration of LNOI-based photonic chip.

physics.optics↗

arXiv2019

Utilizing Players' Playtime Records for Churn Prediction: Mining Playtime Regularity

In the free online game industry, churn prediction is an important research topic. Reducing the churn rate of a game significantly helps with the success of the game. Churn prediction helps a game operator identify possible churning players and keep them engaged in the game via appropriate operational strategies, marketing strategies, and/or incentives. Playtime related features are some of the widely used universal features for most churn prediction models. In this paper, we consider developing new universal features for churn predictions for long-term players based on players' playtime.

cs.HC↗

arXiv2019

MIN: Co-Governing Multi-Identifier Network Architecture and its Prototype on Operator's Network

IP protocol is the core of TCP/IP network layer. However, since IP address and its Domain Name are allocated and managed by a single agency, there are risks of centralization. The semantic overload of IP address also reduces its scalability and mobility, which further hinders the security. This paper proposes a co-governing Multi-Identifier Network (MIN) architecture that constructs a network layer with parallel coexistence of multiple identifiers, including identity, content, geographic information, and IP address. On the management plane, we develop an efficient management system using consortium blockchain with voting consensus, so the network can simultaneously manage and support by hundreds or thousands of nodes with high throughput. On the data plane, we propose an algorithm merging hash table and prefix tree (HTP) for FIB, which avoids the false-negative error and can inter-translate different identifiers with tens of billions of entries. Further, we propose a scheme to transport IP packets using CCN as a tunnel for supporting progressive deployment. We deployed the prototype of MIN to the largest operators' network in Mainland China, Hongkong and Macao, and demonstrated that the network can register identifier under co-governing consensus algorithm, support VoD service very well.

cs.NI↗

arXiv2019

Leap-LSTM: Enhancing Long Short-Term Memory for Text Categorization

Recurrent Neural Networks (RNNs) are widely used in the field of natural language processing (NLP), ranging from text categorization to question answering and machine translation. However, RNNs generally read the whole text from beginning to end or vice versa sometimes, which makes it inefficient to process long texts. When reading a long document for a categorization task, such as topic categorization, large quantities of words are irrelevant and can be skipped. To this end, we propose Leap-LSTM, an LSTM-enhanced model which dynamically leaps between words while reading texts. At each step, we utilize several feature encoders to extract messages from preceding texts, following texts and the current word, and then determine whether to skip the current word. We evaluate Leap-LSTM on several text categorization tasks: sentiment analysis, news categorization, ontology classification and topic classification, with five benchmark data sets. The experimental results show that our model reads faster and predicts better than standard LSTM. Compared to previous models which can also skip words, our model achieves better trade-offs between performance and efficiency.

cs.CL↗

arXiv2018

Learning to Compose over Tree Structures via POS Tags

Recursive Neural Network (RecNN), a type of models which compose words or phrases recursively over syntactic tree structures, has been proven to have superior ability to obtain sentence representation for a variety of NLP tasks. However, RecNN is born with a thorny problem that a shared compositional function for each node of trees can't capture the complex semantic compositionality so that the expressive power of model is limited. In this paper, in order to address this problem, we propose Tag-Guided HyperRecNN/TreeLSTM (TG-HRecNN/TreeLSTM), which introduces hypernetwork into RecNNs to take as inputs Part-of-Speech (POS) tags of word/phrase and generate the semantic composition parameters dynamically. Experimental results on five datasets for two typical NLP tasks show proposed models both obtain significant improvement compared with RecNN and TreeLSTM consistently. Our TG-HTreeLSTM outperforms all existing RecNN-based models and achieves or is competitive with state-of-the-art on four sentence classification benchmarks. The effectiveness of our models is also demonstrated by qualitative analysis.

cs.CL↗

arXiv2018

A Novel Framework for Recurrent Neural Networks with Enhancing Information Processing and Transmission between Units

This paper proposes a novel framework for recurrent neural networks (RNNs) inspired by the human memory models in the field of cognitive neuroscience to enhance information processing and transmission between adjacent RNNs' units. The proposed framework for RNNs consists of three stages that is working memory, forget, and long-term store. The first stage includes taking input data into sensory memory and transferring it to working memory for preliminary treatment. And the second stage mainly focuses on proactively forgetting the secondary information rather than the primary in the working memory. And finally, we get the long-term store normally using some kind of RNN's unit. Our framework, which is generalized and simple, is evaluated on 6 datasets which fall into 3 different tasks, corresponding to text classification, image classification and language modelling. Experiments reveal that our framework can obviously improve the performance of traditional recurrent neural networks. And exploratory task shows the ability of our framework of correctly forgetting the secondary information.

cs.NE↗

arXiv2012

On unbiased performance evaluation for protein inference

This letter is a response to the comments of Serang (2012) on Huang and He (2012) in Bioinformatics. Serang (2012) claimed that the parameters for the Fido algorithm should be specified using the grid search method in Serang et al. (2010) so as to generate a deserved accuracy in performance comparison. It seems that it is an argument on parameter tuning. However, it is indeed the issue of how to conduct an unbiased performance evaluation for comparing different protein inference algorithms. In this letter, we would explain why we don't use the grid search for parameter selection in Huang and He (2012) and show that this procedure may result in an over-estimated performance that is unfair to competing algorithms. In fact, this issue has also been pointed out by Li and Radivojac (2012).

stat.AP↗

arXiv2012

Protein Inference and Protein Quantification: Two Sides of the Same Coin

Motivation: In mass spectrometry-based shotgun proteomics, protein quantification and protein identification are two major computational problems. To quantify the protein abundance, a list of proteins must be firstly inferred from the sample. Then the relative or absolute protein abundance is estimated with quantification methods, such as spectral counting. Until now, researchers have been dealing with these two processes separately. In fact, they are two sides of same coin in the sense that truly present proteins are those proteins with non-zero abundances. Then, one interesting question is if we regard the protein inference problem as a special protein quantification problem, is it possible to achieve better protein inference performance? Contribution: In this paper, we investigate the feasibility of using protein quantification methods to solve the protein inference problem. Protein inference is to determine whether each candidate protein is present in the sample or not. Protein quantification is to calculate the abundance of each protein. Naturally, the absent proteins should have zero abundances. Thus, we argue that the protein inference problem can be viewed as a special case of protein quantification problem: present proteins are those proteins with non-zero abundances. Based on this idea, our paper tries to use three very simple protein quantification methods to solve the protein inference problem effectively. Results: The experimental results on six datasets show that these three methods are competitive with previous protein inference algorithms. This demonstrates that it is plausible to take the protein inference problem as a special case of protein quantification, which opens the door of devising more effective protein inference algorithms from a quantification perspective.

cs.CE↗

arXiv2010

Efficient K-Nearest Neighbor Join Algorithms for High Dimensional Sparse Data

The K-Nearest Neighbor (KNN) join is an expensive but important operation in many data mining algorithms. Several recent applications need to perform KNN join for high dimensional sparse data. Unfortunately, all existing KNN join algorithms are designed for low dimensional data. To fulfill this void, we investigate the KNN join problem for high dimensional sparse data. In this paper, we propose three KNN join algorithms: a brute force (BF) algorithm, an inverted index-based(IIB) algorithm and an improved inverted index-based(IIIB) algorithm. Extensive experiments on both synthetic and real-world datasets were conducted to demonstrate the effectiveness of our algorithms for high dimensional sparse data.

cs.DB↗