arXiv subjects

Yongxiang Li

Publications and source records attributed to Yongxiang Li.

49 records · Page 3Linked to original sources

arXiv2024

52B to 1T: Lessons Learned via Tele-FLM Series

Large Language Models (LLMs) represent a significant stride toward Artificial General Intelligence. As scaling laws underscore the potential of increasing model sizes, the academic community has intensified its investigations into LLMs with capacities exceeding 50 billion parameters. This technical report builds on our prior work with Tele-FLM (also known as FLM-2), a publicly available 52-billion-parameter model. We delve into two primary areas: we first discuss our observation of Supervised Fine-tuning (SFT) on Tele-FLM-52B, which supports the "less is more" approach for SFT data construction; second, we demonstrate our experiments and analyses on the best practices for progressively growing a model from 52 billion to 102 billion, and subsequently to 1 trillion parameters. We will open-source a 1T model checkpoint, namely Tele-FLM-1T, to advance further training and research.

cs.CL↗

arXiv2024

Tele-FLM Technical Report

Large language models (LLMs) have showcased profound capabilities in language understanding and generation, facilitating a wide array of applications. However, there is a notable paucity of detailed, open-sourced methodologies on efficiently scaling LLMs beyond 50 billion parameters with minimum trial-and-error cost and computational resources. In this report, we introduce Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that features a stable, efficient pre-training paradigm and enhanced factual judgment capabilities. Tele-FLM demonstrates superior multilingual language modeling abilities, measured by BPB on textual corpus. Besides, in both English and Chinese foundation model evaluation, it is comparable to strong open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B and DeepSeek-67B. In addition to the model weights, we share the core designs, engineering practices, and training details, which we expect to benefit both the academic and industrial communities.

cs.CL↗

arXiv2024

ProTA: Probabilistic Token Aggregation for Text-Video Retrieval

Text-video retrieval aims to find the most relevant cross-modal samples for a given query. Recent methods focus on modeling the whole spatial-temporal relations. However, since video clips contain more diverse content than captions, the model aligning these asymmetric video-text pairs has a high risk of retrieving many false positive results. In this paper, we propose Probabilistic Token Aggregation (ProTA) to handle cross-modal interaction with content asymmetry. Specifically, we propose dual partial-related aggregation to disentangle and re-aggregate token representations in both low-dimension and high-dimension spaces. We propose token-based probabilistic alignment to generate token-level probabilistic representation and maintain the feature representation diversity. In addition, an adaptive contrastive loss is proposed to learn compact cross-modal distribution space. Based on extensive experiments, ProTA achieves significant improvements on MSR-VTT (50.9%), LSMDC (25.8%), and DiDeMo (47.2%).

cs.CV↗

arXiv2024

TeleChat Technical Report

In this technical report, we present TeleChat, a collection of large language models (LLMs) with parameters of 3 billion, 7 billion and 12 billion. It includes pretrained language models as well as fine-tuned chat models that is aligned with human preferences. TeleChat is initially pretrained on an extensive corpus containing a diverse collection of texts from both English and Chinese languages, including trillions of tokens. Subsequently, the model undergoes fine-tuning to align with human preferences, following a detailed methodology that we describe. We evaluate the performance of TeleChat on various tasks, including language understanding, mathematics, reasoning, code generation, and knowledge-based question answering. Our findings indicate that TeleChat achieves comparable performance to other open-source models of similar size across a wide range of public benchmarks. To support future research and applications utilizing LLMs, we release the fine-tuned model checkpoints of TeleChat's 7B and 12B variant, along with code and a portion of our pretraining data, to the public community.

cs.CL↗

arXiv2024

Towards Robustness and Diversity: Continual Learning in Dialog Generation with Text-Mixup and Batch Nuclear-Norm Maximization

In our dynamic world where data arrives in a continuous stream, continual learning enables us to incrementally add new tasks/domains without the need to retrain from scratch. A major challenge in continual learning of language model is catastrophic forgetting, the tendency of models to forget knowledge from previously trained tasks/domains when training on new ones. This paper studies dialog generation under the continual learning setting. We propose a novel method that 1) uses \textit{Text-Mixup} as data augmentation to avoid model overfitting on replay memory and 2) leverages Batch-Nuclear Norm Maximization (BNNM) to alleviate the problem of mode collapse. Experiments on a $37$-domain task-oriented dialog dataset and DailyDialog (a $10$-domain chitchat dataset) demonstrate that our proposed approach outperforms the state-of-the-art in continual learning.

cs.CL↗

arXiv2023

Intrinsic defect engineering of CVD grown monolayer MoS$_2$ for tuneable functional nanodevices

Defects in atomically thin materials can drive new functionalities and expand applications to multifunctional systems that are monolithically integrated. An ability to control formation of defects during the synthesis process is an important capability to create practical deployment opportunities. Molybdenum disulfide (MoS$_2$), a two-dimensional (2D) semiconducting material harbors intrinsic defects that can be harnessed to achieve tuneable electronic, optoelectronic, and electrochemical devices. However, achieving precise control over defect formation within monolayer MoS$_2$, while maintaining the structural integrity of the crystals remains a notable challenge. Here, we present a one-step, in-situ defect engineering approach for monolayer MoS$_2$ using a pressure dependent chemical vapour deposition (CVD) process. Monolayer MoS$_2$ grown in low-pressure CVD conditions (LP-MoS$_2$) produces sulfur vacancy (Vs) induced defect rich crystals primarily attributed to the kinetics of the growth conditions. Conversely, atmospheric pressure CVD grown MoS$_2$ (AP-MoS$_2$) passivates these Vs defects with oxygen. This disparity in defect profiles profoundly impacts crucial functional properties and device performance. AP-MoS$_2$ shows a drastically enhanced photoluminescence, which is significantly quenched in LP-MoS$_2$ attributed to in-gap electron donor states induced by the Vs defects. However, the n-doping induced by the Vs defects in LP-MoS$_2$ generates enhanced photoresponsivity and detectivity in our fabricated photodetectors compared to the AP-MoS$_2$ based devices. Defect-rich LP-MoS$_2$ outperforms AP-MoS$_2$ as channel layers of field-effect transistors (FETs), as well as electrocatalytic material for hydrogen evolution reaction (HER). This work presents a single-step CVD approach for in-situ defect engineering in monolayer MoS$_2$ and presents a pathway to control defects in other monolayer material systems.

physics.app-ph↗

arXiv2023

Generalized Time Warping Invariant Dictionary Learning for Time Series Classification and Clustering

Dictionary learning is an effective tool for pattern recognition and classification of time series data. Among various dictionary learning techniques, the dynamic time warping (DTW) is commonly used for dealing with temporal delays, scaling, transformation, and many other kinds of temporal misalignments issues. However, the DTW suffers overfitting or information loss due to its discrete nature in aligning time series data. To address this issue, we propose a generalized time warping invariant dictionary learning algorithm in this paper. Our approach features a generalized time warping operator, which consists of linear combinations of continuous basis functions for facilitating continuous temporal warping. The integration of the proposed operator and the dictionary learning is formulated as an optimization problem, where the block coordinate descent method is employed to jointly optimize warping paths, dictionaries, and sparseness coefficients. The optimized results are then used as hyperspace distance measures to feed classification and clustering algorithms. The superiority of the proposed method in terms of dictionary learning, classification, and clustering is validated through ten sets of public datasets in comparing with various benchmark methods.

stat.ML↗

arXiv2023

A Scalable Gaussian Process for Large-Scale Periodic Data

The periodic Gaussian process (PGP) has been increasingly used to model periodic data due to its high accuracy. Yet, computing the likelihood of PGP has a high computational complexity of $\mathcal{O}\left(n^{3}\right)$ ($n$ is the data size), which hinders its wide application. To address this issue, we propose a novel circulant PGP (CPGP) model for large-scale periodic data collected at grids that are commonly seen in signal processing applications. The proposed CPGP decomposes the log-likelihood of PGP into the sum of two computationally scalable composite log-likelihoods, which do not involve any approximations. Computing the likelihood of CPGP requires only $\mathcal{O}\left(p^{2}\right)$ (or $\mathcal{O}\left(p\log p\right)$ in some special cases) time for grid observations, where the segment length $p$ is independent of and much smaller than $n$. Simulations and real case studies are presented to show the superiority of CPGP over some state-of-the-art methods, especially for applications requiring periodicity estimation. This new modeling technique can greatly advance the applicability of PGP in many areas and allow the modeling of many previously intractable problems.

stat.ME↗

arXiv2022

Online Structural Change-point Detection of High-dimensional Streaming Data via Dynamic Sparse Subspace Learning

High-dimensional streaming data are becoming increasingly ubiquitous in many fields. They often lie in multiple low-dimensional subspaces, and the manifold structures may change abruptly on the time scale due to pattern shift or occurrence of anomalies. However, the problem of detecting the structural changes in a real-time manner has not been well studied. To fill this gap, we propose a dynamic sparse subspace learning approach for online structural change-point detection of high-dimensional streaming data. A novel multiple structural change-point model is proposed and the asymptotic properties of the estimators are investigated. A tuning method based on Bayesian information criterion and change-point detection accuracy is proposed for penalty coefficients selection. An efficient Pruned Exact Linear Time based algorithm is proposed for online optimization and change-point detection. The effectiveness of the proposed method is demonstrated through several simulation studies and a real case study on gesture data for motion tracking.

stat.ML↗

arXiv2019

Cauchy problem for fractional non-autonomous evolution equations

This paper deals with the following Cauchy problem to nonlinear time fractional non-autonomous integro-differential evolution equation of mixed type via measure of noncompactness $$ \left\{\begin{array}{ll} ^CD^α_tu(t)+A(t)u(t)= f(t,u(t),(Tu)(t), (Su)(t)),\quad t\in [0,a], \\[12pt] u(0)=A^{-1}(0)u_0 \end{array} \right. $$ in infinite-dimensional Banach space $E$, where $ ^CD^α_t$ is the standard Caputo's fractional time derivative of order $0<α\leq 1$, $A(t)$ is a family of closed linear operators defined on a dense domain $D(A)$ in Banach space $E$ into $E$ such that $D(A)$ is independent of $t$, $a>0$ is a constant, $f:[0,a]\times E\times E\times E\rightarrow E$ is a Carathéodory type function, $u_0\in E$, $T$ and $S$ are Volterra and Fredholm integral operators, respectively. Combining the theory of fractional calculus and evolution families, the fixed point theorem with respect to convex-power condensing operator and a new estimation technique of the measure of noncompactness, we obtained the existence of mild solutions under the situation that the nonlinear function satisfy some appropriate local growth condition and a noncompactness measure condition. Our results generalize and improve some previous results on this topic, since the condition of uniformly continuity of the nonlinearity is not required, and also the strong restriction on the constants in the condition of noncompactness measure is completely deleted. As samples of applications, we consider the initial value problem to a class of time fractional non-autonomous partial differential equation with homogeneous Dirichlet boundary condition at the end of this paper.

math.FA↗

arXiv2018

Composite Inference for Gaussian Processes

Large-scale Gaussian process models are becoming increasingly important and widely used in many areas, such as, computer experiments, stochastic optimization via simulation, and machine learning using Gaussian processes. The standard methods, such as maximum likelihood estimation (MLE) for parameter estimation and the best linear unbiased predictor (BLUP) for prediction, are generally the primary choices in many applications. In spite of their merits, those methods are not feasible due to intractable computation when the sample size is huge. A novel method for the purposes of parameter estimation and prediction is proposed to solve the computational problems of large-scale Gaussian process based models, by separating the original dataset into tractable subsets. This method consistently combines parameter estimation and prediction by making full use of the dependence among conditional densities: a statistically efficient composite likelihood based on joint distributions of some well selected conditional densities is developed to estimate parameters and then "composite inference" is coined to make prediction for an unknown input point, based on its distributions conditional on each block subset. The proposed method transforms the intractable BLUP into a tractable convex optimization problem. It is also shown that the prediction given by the proposed method, called the best linear unbiased block predictor, has a minimum variance for a given separation of the dataset. Keywords: Large scale, Parallel computing, Composite likelihood, Spatial process

stat.ME↗

arXiv2018

Periodic solutions for neutral evolution equations with delays

The aim is to study the periodic solution problem for neutral evolution equation $$(u(t)-G(t,u(t-ξ)))'+Au(t)=F(t,u(t),u(t-τ)),\ \ \ \ t\in\R$$in Banach space $X$, where $A:D(A)\subset X\rightarrow X$ is a closed linear operator, and $-A$ generates a compact analytic operator semigroup $T(t)(t\geq0)$. With the aid of the analytic operator semigroup theories and some fixed point theorems, we obtain the existence and uniqueness of periodic mild solution for neutral evolution equations. The regularity of periodic mild solution for evolution equation with delay is studied, and some the existence results of the classical and strong solutions are obtained. In the end, we give an example to illustrate the applicability of abstract results. Our works greatly improve and generalize the relevant results of existing literatures.

math.FA↗

arXiv2018

Positive periodic solutions for abstract evolution equations with delay

In this paper, we discuss the existence and asymptotic stability of the positive periodic mild solutions for the abstract evolution equation with delay in an ordered Banach space $E$, $$u'(t)+Au(t)=F(t,u(t),u(t-τ)),\ \ \ \ t\in\R,$$ where $A:D(A)\subset E\rightarrow E$ is a closed linear operator and $-A$ generates a positive $C_{0}$-semigroup $T(t)(t\geq0)$, $F:\R\times E\times E\rightarrow E$ is a continuous mapping which is $ω$-periodic in $t$. Under order conditions on the nonlinearity $F$ concerning the growth exponent of the semigroup $T(t)(t\geq0)$ or the first eigenvalue of the operator $A$, we obtain the existence and asymptotic stability results of the positive $ω$-periodic mild solutions by applying operator semigroup theory. In the end, an example is given to illustrate the applicability of our abstract results.

math.FA↗