arXiv subjects

Zhenda Xie

Publications and source records attributed to Zhenda Xie.

At least 37 records · Page 2Linked to original sources

arXiv2024

DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.

cs.SE↗

arXiv2024

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-$K$ out of $N$ experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into $mN$ ones and activating $mK$ from them, allowing for a more flexible combination of activated experts; (2) isolating $K_s$ experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.

cs.CL↗

arXiv2024

DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.

cs.CL↗

arXiv2023

Passively stable 0.7-octave microcombs in thin-film lithium niobate microresonators

Optical frequency comb based on microresonator (microcomb) is an integrated coherent light source and has the potential to promise a high-precision frequency standard, and self-reference and long-term stable microcomb is the key to this realization. Here, we demonstrated a 0.7-octave spectrum Kerr comb via dispersion engineering in a thin film lithium niobate microresonator, and the single soliton state can be accessed passively with long-term stability over 3 hours. With such a robust broadband coherent comb source using thin film lithium niobate, fully stabilized microcomb can be expected for massive practical applications.

physics.optics↗

arXiv2023

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior

We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation. Code available at https://github.com/deepseek-ai/DreamCraft3D.

cs.CV↗

arXiv2023

A chip-scale polarization-spatial-momentum quantum SWAP gate in silicon nanophotonics

Recent progress in quantum computing and networking enables high-performance large-scale quantum processors by connecting different quantum modules. Optical quantum systems show advantages in both computing and communications, and integrated quantum photonics further increases the level of scaling and complexity. Here we demonstrate an efficient SWAP gate that deterministically swaps a photon's polarization qubit with its spatial-momentum qubit on a nanofabricated two-level silicon-photonics chip containing three cascaded gates. The on-chip SWAP gate is comprehensively characterized by tomographic measurements with high fidelity for both single-qubit and two-qubit operation. The coherence preservation of the SWAP gate process is verified by single-photon and two-photon quantum interference. The coherent reversible conversion of our SWAP gate facilitates a quantum interconnect between different photonic subsystems with different degrees of freedom, demonstrated by distributing four Bell states between two chips. We also elucidate the source of decoherence in the SWAP operation in pursuit of near-unity fidelity. Our deterministic SWAP gate in the silicon platform provides a pathway towards integrated quantum information processing for interconnected modular systems.

quant-ph↗

arXiv2023

High speed free-space optical communication using standard fiber communication component without optical amplification

Free-space optical communication (FSO) can achieve fast, secure and license-free communication without need for physical cables, making it a cost-effective, energy-efficient and flexible solution when the fiber connection is unavailable. To establish FSO connection on-demand, it is essential to build portable FSO devices with compact structure and light weight. Here, we develop a miniaturized FSO system and realize 9.16 Gbps FSO between two nodes that is 1 km apart, using a commercial single-mode-fiber-coupled optical transceiver module without optical amplification. Using our 4-stage acquisition, pointing and tracking (APT) systems, the tracking error is within 3 μrad and results an average link loss of 13.7 dB, which is the key for this high-bandwidth FSO demonstration without optical amplification. Our FSO link has been tested up to 4 km, with link loss of 18 dB that is limited by the foggy weather during the test. Longer FSO distances can be expected with better weather condition and optical amplification. With single FSO device weight of only 9.5 kg, this result arouses massive applications of field-deployable high-speed wireless communication.

eess.SP↗

arXiv2022

Turnkey photonic flywheel in a Chimera cavity

Dissipative Kerr soliton (DKS) microcomb has emerged as an enabling technology that revolutionizes a wide range of applications in both basic science and technological innovation. Reliable turnkey operation with sub-opticalcycle and sub-femtosecond timing jitter is key to the success of many intriguing microcomb applications at the intersection of ultrafast optics and microwave electronics. Here we propose a novel approach to demonstrate the first turnkey Brillouin-DKS frequency comb. Our approach with a Chimera cavity offers essential benefits that are not attainable previously, including phase insensitivity, self-healing capability, deterministic selection of DKS state, and access to the ultralow noise comb state. The demonstrated turnkey Brillouin-DKS frequency comb achieves a fundamental comb linewidth of 100 mHz and DKS timing jitter of 1 femtosecond for averaging times up to 56 μs. The approach is universal and generalizable to various device platforms for user-friendly and field-deployable comb devices.

physics.optics↗

arXiv2022

Noise-squeezed forward Brillouin lasers in multimode fiber microresonators

Stimulated Brillouin scattering (SBS) in low-power and compact microresonators has created a new field in cavity nonlinear photonics due to the marriage between acoustic and optical signal processing. Considering the fundamental differences between backward SBS and forward SBS processes, it is challenging to observe the coexistence of both processes in the same microresonator, as well as the photon noise suppression for the forward stimulated Brillouin laser (FSBL). In this paper, we demonstrate the first 20-dB-noise-squeezed FSBL generation excited by the coexisting backward SBL (BSBL) in an ultrahigh-quality-factor Fabry-Perot (FP) microresonator based on multimode fiber (MMF). Multiple FSBLs and BSBLs are cascaded by multiple intermodal SBS processes in the multimode microresonator, where the cascaded process between backward SBS and forward SBS process (pump-BSBL-FSBL) provides a route towards additional noise squeezing, rendering the FSBL phase noise to be -120 dBc/Hz at 1 MHz offset frequency. Furthermore, we demonstrate the first Brillouin-Kerr soliton from a high-order BSBL, which also coexists with FSBLs. Our experimental results show the potential of MMF FP microresonator as an ideal testbed for high-dimensional nonlinear cavity dynamics and laser source with ultrahigh coherence.

physics.optics↗

arXiv2022

Automated turnkey microcomb for low-noise microwave synthesis

Microresonator-based optical frequency comb (microcomb) has the potential to revolutionize the accuracy of frequency synthesizer in radar and communication applications. However, fundamental limit exists for low noise microcomb generation, especially in low size, weight, power and cost (SWaP-C) package. Here we resolve this limit, by the demonstration of an automated turnkey microcomb, operating close to its low quantum-limited phase noise, within a compact setup size of 85 mm * 90 mm * 25 mm. High quality factor fiber Fabry-Perot resonator (FFPR), with Q up to 4.0 * 10^9, is the key for both low quantum noise and pump noise limit, in the diode-pump case in a self-injection locking scheme. Low phase noise of -80 and -105 dBc/Hz at 100 Hz, -106 and -125 dBc/Hz at 1 kHz, -133 and -148 dBc/Hz at 10 kHz is achieved at 10.1 GHz and 1.7 GHz repetition frequencies, respectively. With the simultaneous automated turnkey, low-noise and direct-diode-pump capability, our microcomb is ready to be used as a low-noise frequency synthesizer with low SWaP-C and thus field deployability.

physics.optics↗

arXiv2022

Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation

Masked image modeling (MIM) learns representations with remarkably good fine-tuning performances, overshadowing previous prevalent pre-training approaches such as image classification, instance contrastive learning, and image-text alignment. In this paper, we show that the inferior fine-tuning performance of these pre-training approaches can be significantly improved by a simple post-processing in the form of feature distillation (FD). The feature distillation converts the old representations to new representations that have a few desirable properties just like those representations produced by MIM. These properties, which we aggregately refer to as optimization friendliness, are identified and analyzed by a set of attention- and optimization-related diagnosis tools. With these properties, the new representations show strong fine-tuning performance. Specifically, the contrastive self-supervised learning methods are made as competitive in fine-tuning as the state-of-the-art masked image modeling (MIM) algorithms. The CLIP models' fine-tuning performance is also significantly improved, with a CLIP ViT-L model reaching 89.0% top-1 accuracy on ImageNet-1K classification. On the 3-billion-parameter SwinV2-G model, the fine-tuning accuracy is improved by +1.5 mIoU / +1.1 mAP to 61.4 mIoU / 64.2 mAP on ADE20K semantic segmentation and COCO object detection, respectively, creating new records on both benchmarks. More importantly, our work provides a way for the future research to focus more effort on the generality and scalability of the learnt representations without being pre-occupied with optimization friendliness since it can be enhanced rather easily. The code will be available at https://github.com/SwinTransformer/Feature-Distillation.

cs.CV↗

arXiv2022

Dissipative soliton generation and real-time dynamics in microresonator-filtered fiber lasers

Optical frequency combs in microresonators (microcombs) have a wide range of applications in science and technology, due to its compact size and access to considerably larger comb spacing. Despite recent successes, the problems of self-starting, high mode efficiency as well as high output power have not been fully addressed for conventional soliton microcombs. Recent demonstration of laser cavity soliton microcombs by nesting a microresonator into a fiber cavity, shows great potential to solve the problems. Here we comprehensively study the dissipative soliton generation and interaction dynamics in a microresonator-filtered fiber laser in both theory and experiment. We first bring theoretical insight into the mode-locking principle, discuss the parameters effect on soliton properties and provide experimental guidelines for broadband soliton generation. We predict chirped bright dissipative soliton with flat-top spectral envelope in microresonators with normal dispersion, which is fundamentally infeasible for externally driven case. Furthermore, we experimentally achieve soliton microcombs with large bandwidth of ~10 nm and high mode efficiency of 90.7%. Finally, by taking advantage of an ultrahigh-speed time magnifier, we study the real-time soliton formation and interaction dynamics and experimentally observe soliton Newton's cradle. Our study will benefit the design of the novel, high-efficiency and self-starting microcombs for real-world applications.

physics.optics↗

arXiv2022

On Data Scaling in Masked Image Modeling

An important goal of self-supervised learning is to enable model pre-training to benefit from almost unlimited data. However, one method that has recently become popular, namely masked image modeling (MIM), is suspected to be unable to benefit from larger data. In this work, we break this misconception through extensive experiments, with data scales ranging from 10\% of ImageNet-1K to full ImageNet-22K, model sizes ranging from 49 million to 1 billion, and training lengths ranging from 125K iterations to 500K iterations. Our study reveals that: (i) Masked image modeling is also demanding on larger data. We observed that very large models got over-fitted with relatively small data; (ii) The length of training matters. Large models trained with masked image modeling can benefit from more data with longer training; (iii) The validation loss in pre-training is a good indicator to measure how well the model performs for fine-tuning on multiple tasks. This observation allows us to pre-evaluate pre-trained models in advance without having to make costly trial-and-error assessments of downstream tasks. We hope that our findings will advance the understanding of masked image modeling in terms of scaling ability.

cs.CV↗

arXiv2022

A Scheme for Deterministic N-photon State Generation Using Lithium Niobate on Insulator Device

Large-photon-number quantum state is a fundamental but non-resolved request for practical quantum information applications. Here we propose an N-photon state generation scheme that is feasible and scalable, using lithium niobate on insulator circuits. Such scheme is based on the integration of a common building block called photon-number doubling unit (PDU), for deterministic single-photon parametric down-conversion and up-conversion. The PDU relies on 10^7-optical-quality-factor resonator and mW-level on-chip power, which is within the current fabrication and experiment limits. N-photon state generation schemes, with cluster and GHZ state as examples, are shown for different quantum tasks.

quant-ph↗

arXiv2022

Revealing the Dark Secrets of Masked Image Modeling

Masked image modeling (MIM) as pre-training is shown to be effective for numerous vision downstream tasks, but how and where MIM works remain unclear. In this paper, we compare MIM with the long-dominant supervised pre-trained models from two perspectives, the visualizations and the experiments, to uncover their key representational differences. From the visualizations, we find that MIM brings locality inductive bias to all layers of the trained models, but supervised models tend to focus locally at lower layers but more globally at higher layers. That may be the reason why MIM helps Vision Transformers that have a very large receptive field to optimize. Using MIM, the model can maintain a large diversity on attention heads in all layers. But for supervised models, the diversity on attention heads almost disappears from the last three layers and less diversity harms the fine-tuning performance. From the experiments, we find that MIM models can perform significantly better on geometric and motion tasks with weak semantics or fine-grained classification tasks, than their supervised counterparts. Without bells and whistles, a standard MIM pre-trained SwinV2-L could achieve state-of-the-art performance on pose estimation (78.9 AP on COCO test-dev and 78.0 AP on CrowdPose), depth estimation (0.287 RMSE on NYUv2 and 1.966 RMSE on KITTI), and video object tracking (70.7 SUC on LaSOT). For the semantic understanding datasets where the categories are sufficiently covered by the supervised pre-training, MIM models can still achieve highly competitive transfer performance. With a deeper understanding of MIM, we hope that our work can inspire new and solid research in this direction.

cs.CV↗

arXiv2022

iCAR: Bridging Image Classification and Image-text Alignment for Visual Recognition

Image classification, which classifies images by pre-defined categories, has been the dominant approach to visual representation learning over the last decade. Visual learning through image-text alignment, however, has emerged to show promising performance, especially for zero-shot recognition. We believe that these two learning tasks are complementary, and suggest combining them for better visual learning. We propose a deep fusion method with three adaptations that effectively bridge two learning tasks, rather than shallow fusion through naive multi-task learning. First, we modify the previous common practice in image classification, a linear classifier, with a cosine classifier which shows comparable performance. Second, we convert the image classification problem from learning parametric category classifier weights to learning a text encoder as a meta network to generate category classifier weights. The learnt text encoder is shared between image classification and image-text alignment. Third, we enrich each class name with a description to avoid confusion between classes and make the classification method closer to the image-text alignment. We prove that this deep fusion approach performs better on a variety of visual recognition tasks and setups than the individual learning or shallow fusion approach, from zero-shot/few-shot image classification, such as the Kornblith 12-dataset benchmark, to downstream tasks of action recognition, semantic segmentation, and object detection in fine-tuning and open-vocabulary settings. The code will be available at https://github.com/weiyx16/iCAR.

cs.CV↗

arXiv2022

SimMIM: A Simple Framework for Masked Image Modeling

This paper presents SimMIM, a simple framework for masked image modeling. We simplify recently proposed related approaches without special designs such as block-wise masking and tokenization via discrete VAE or clustering. To study what let the masked image modeling task learn good representations, we systematically study the major components in our framework, and find that simple designs of each component have revealed very strong representation learning performance: 1) random masking of the input image with a moderately large masked patch size (e.g., 32) makes a strong pre-text task; 2) predicting raw pixels of RGB values by direct regression performs no worse than the patch classification approaches with complex designs; 3) the prediction head can be as light as a linear layer, with no worse performance than heavier ones. Using ViT-B, our approach achieves 83.8% top-1 fine-tuning accuracy on ImageNet-1K by pre-training also on this dataset, surpassing previous best approach by +0.6%. When applied on a larger model of about 650 million parameters, SwinV2-H, it achieves 87.1% top-1 accuracy on ImageNet-1K using only ImageNet-1K data. We also leverage this approach to facilitate the training of a 3B model (SwinV2-G), that by $40\times$ less data than that in previous practice, we achieve the state-of-the-art on four representative vision benchmarks. The code and models will be publicly available at https://github.com/microsoft/SimMIM.

cs.CV↗

arXiv2022

Swin Transformer V2: Scaling Up Capacity and Resolution

Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536$\times$1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time. Code is available at \url{https://github.com/microsoft/Swin-Transformer}.

cs.CV↗