arXiv subjects

Shiming Yang

Publications and source records attributed to Shiming Yang.

7 recordsLinked to original sources

arXiv2024

Yi-Lightning Technical Report

This technical report presents Yi-Lightning, our latest flagship large language model (LLM). It achieves exceptional performance, ranking 6th overall on Chatbot Arena, with particularly strong results (2nd to 4th place) in specialized categories including Chinese, Math, Coding, and Hard Prompts. Yi-Lightning leverages an enhanced Mixture-of-Experts (MoE) architecture, featuring advanced expert segmentation and routing mechanisms coupled with optimized KV-caching techniques. Our development process encompasses comprehensive pre-training, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF), where we devise deliberate strategies for multi-stage training, synthetic data construction, and reward modeling. Furthermore, we implement RAISE (Responsible AI Safety Engine), a four-component framework to address safety issues across pre-training, post-training, and serving phases. Empowered by our scalable super-computing infrastructure, all these innovations substantially reduce training, deployment and inference costs while maintaining high-performance standards. With further evaluations on public academic benchmarks, Yi-Lightning demonstrates competitive performance against top-tier LLMs, while we observe a notable disparity between traditional, static benchmark results and real-world, dynamic human preferences. This observation prompts a critical reassessment of conventional benchmarks' utility in guiding the development of more intelligent and powerful AI systems for practical applications. Yi-Lightning is now available through our developer platform at https://platform.lingyiwanwu.com.

cs.CL↗

arXiv2024

Yi: Open Foundation Models by 01.AI

We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.

cs.CL↗

arXiv2023

ESGReveal: An LLM-based approach for extracting structured data from ESG reports

ESGReveal is an innovative method proposed for efficiently extracting and analyzing Environmental, Social, and Governance (ESG) data from corporate reports, catering to the critical need for reliable ESG information retrieval. This approach utilizes Large Language Models (LLM) enhanced with Retrieval Augmented Generation (RAG) techniques. The ESGReveal system includes an ESG metadata module for targeted queries, a preprocessing module for assembling databases, and an LLM agent for data extraction. Its efficacy was appraised using ESG reports from 166 companies across various sectors listed on the Hong Kong Stock Exchange in 2022, ensuring comprehensive industry and market capitalization representation. Utilizing ESGReveal unearthed significant insights into ESG reporting with GPT-4, demonstrating an accuracy of 76.9% in data extraction and 83.7% in disclosure analysis, which is an improvement over baseline models. This highlights the framework's capacity to refine ESG data analysis precision. Moreover, it revealed a demand for reinforced ESG disclosures, with environmental and social data disclosures standing at 69.5% and 57.2%, respectively, suggesting a pursuit for more corporate transparency. While current iterations of ESGReveal do not process pictorial information, a functionality intended for future enhancement, the study calls for continued research to further develop and compare the analytical capabilities of various LLMs. In summary, ESGReveal is a stride forward in ESG data processing, offering stakeholders a sophisticated tool to better evaluate and advance corporate sustainability efforts. Its evolution is promising in promoting transparency in corporate reporting and aligning with broader sustainable development aims.

cs.CL↗

arXiv2022

Exploration of Differentiability in a Proton Computed Tomography Simulation Framework

Objective. Algorithmic differentiation (AD) can be a useful technique to numerically optimize design and algorithmic parameters by, and quantify uncertainties in, computer simulations. However, the effectiveness of AD depends on how "well-linearizable" the software is. In this study, we assess how promising derivative information of a typical proton computed tomography (pCT) scan computer simulation is for the aforementioned applications. Approach. This study is mainly based on numerical experiments, in which we repeatedly evaluate three representative computational steps with perturbed input values. We support our observations with a review of the algorithmic steps and arithmetic operations performed by the software, using debugging techniques. Main results. The model-based iterative reconstruction (MBIR) subprocedure (at the end of the software pipeline) and the Monte Carlo (MC) simulation (at the beginning) were piecewise differentiable. Jumps in the MBIR function arose from the discrete computation of the set of voxels intersected by a proton path. Jumps in the MC function likely arose from changes in the control flow that affect the amount of consumed random numbers. The tracking algorithm solves an inherently non-differentiable problem. Significance. The MC and MBIR codes are ready for the integration of AD, and further research on surrogate models for the tracking subprocedure is necessary.

physics.med-ph↗

arXiv2019

The Modular X- and Gamma-Ray Sensor (MXGS)of the ASIM Payload on the International Space Station

The Modular X- and Gamma-ray Sensor (MXGS) is an imaging and spectral X- and Gamma-ray instrument mounted on the starboard side of the Columbus module on the International Space Station. Together with the Modular Multi-Spectral Imaging Assembly (MMIA) (Chanrion et al. this issue) MXGS constitutes the instruments of the Atmosphere-Space Interactions Monitor (ASIM) (Neubert et al. this issue). The main objectives of MXGS are to image and measure the spectrum of X- and $\gamma$-rays from lightning discharges, known as Terrestrial Gamma-ray Flashes (TGFs), and for MMIA to image and perform high speed photometry of Transient Luminous Events (TLEs) and lightning discharges. With these two instruments specifically designed to explore the relation between electrical discharges, TLEs and TGFs, ASIM is the first mission of its kind.

physics.ins-det↗

arXiv2018

Effects of Proton Irradiation on 60 GHz CMOS Transceiver Chip for Multi-Gbps Communication in High-Energy Physics Experiments

This paper presents the experimental results of $17~MeV$ proton irradiation on a $60~GHz$ low power, half-duplex transceiver (TRX) chip implemented in $65~nm$ CMOS technology. It supports short range point-to-point data rate up to $6~Gbps$ by employing on-off keying (OOK). To investigate the irradiation hardness for high energy physics applications, two TRX chips were irradiated with total ionizing doses (TID) of $74~kGy$ and $42~kGy$ and fluence of $1.4~\times$10$^{14}~ N_{eq}/cm^2$ and $0.8~\times$10$^{14}~N_{eq}/cm^2$ for RX and TX modes, respectively. The chips were characterized by pre- and post-irradiation analogue voltage measurements on different circuit blocks as well as through the analysis of wireless transmission parameters like bit error rate (BER), eye diagram, jitter etc. Post-irradiation measurements have shown certain reduction in performance but both TRX chips have been found operational through over the air measurements at $5~Gbps$. Moreover, very small shift in the carrier frequency was observed after the irradiation.

physics.ins-det↗

arXiv2017

Proton tracking in a high-granularity Digital Tracking Calorimeter for proton CT purposes

Radiation therapy with protons as of today utilizes information from x-ray CT in order to estimate the proton stopping power of the traversed tissue in a patient. The conversion from x-ray attenuation to proton stopping power in tissue introduces range uncertainties of the order of 2-3% of the range, uncertainties that are contributing to an increase of the necessary planning margins added to the target volume in a patient. Imaging methods and modalities, such as Dual Energy CT and proton CT, have come into consideration in the pursuit of obtaining an as good as possible estimate of the proton stopping power. In this study, a Digital Tracking Calorimeter is benchmarked for proof-of-concept for proton CT purposes. The Digital Tracking Calorimeteris applied for reconstruction of the tracks and energies of individual high energy protons. The presented prototype forms the basis for a proton CT system using a single technology for tracking and calorimetry. This advantage simplifies the setup and reduces the cost of a proton CT system assembly, and it is a unique feature of the Digital Tracking Calorimeter. Data from the AGORFIRM beamline at KVI-CART in Groningen in the Netherlands and Monte Carlo simulation results are used to in order to develop a tracking algorithm for the estimation of the residual ranges of a high number of concurrent proton tracks. The range of the individual protons can at present be estimated with a resolution of 4%. The readout system for this prototype is able to handle an effective proton frequency of 1 MHz by using 500 concurrent proton tracks in each readout frame, which is at the high end range of present similar prototypes. A future further optimized prototype will enable a high-speed and more accurate determination of the ranges of individual protons in a therapeutic beam.

physics.med-ph↗