arXiv subjects

Jie Jiang

Publications and source records attributed to Jie Jiang.

At least 163 records · Page 9Linked to original sources

arXiv2024

Ads Recommendation in a Collapsed and Entangled World

We present Tencent's ads recommendation system and examine the challenges and practices of learning appropriate recommendation representations. Our study begins by showcasing our approaches to preserving prior knowledge when encoding features of diverse types into embedding representations. We specifically address sequence features, numeric features, and pre-trained embedding features. Subsequently, we delve into two crucial challenges related to feature representation: the dimensional collapse of embeddings and the interest entanglement across different tasks or scenarios. We propose several practical approaches to address these challenges that result in robust and disentangled recommendation representations. We then explore several training techniques to facilitate model optimization, reduce bias, and enhance exploration. Additionally, we introduce three analysis tools that enable us to study feature correlation, dimensional collapse, and interest entanglement. This work builds upon the continuous efforts of Tencent's ads recommendation team over the past decade. It summarizes general design principles and presents a series of readily applicable solutions and analysis tools. The reported performance is based on our online advertising platform, which handles hundreds of billions of requests daily and serves millions of ads to billions of users.

cs.IR↗

arXiv2024

Temporal Interest Network for User Response Prediction

User response prediction is essential in industrial recommendation systems, such as online display advertising. Among all the features in recommendation models, user behaviors are among the most critical. Many works have revealed that a user's behavior reflects her interest in the candidate item, owing to the semantic or temporal correlation between behaviors and the candidate. While the literature has individually examined each of these correlations, researchers have yet to analyze them in combination, that is, the semantic-temporal correlation. We empirically measure this correlation and observe intuitive yet robust patterns. We then examine several popular user interest models and find that, surprisingly, none of them learn such correlation well. To fill this gap, we propose a Temporal Interest Network (TIN) to capture the semantic-temporal correlation simultaneously between behaviors and the target. We achieve this by incorporating target-aware temporal encoding, in addition to semantic encoding, to represent behaviors and the target. Furthermore, we conduct explicit 4-way interaction by deploying target-aware attention and target-aware representation to capture both semantic and temporal correlation. We conduct comprehensive evaluations on two popular public datasets, and our proposed TIN outperforms the best-performing baselines by 0.43% and 0.29% on GAUC, respectively. During online A/B testing in Tencent's advertising platform, TIN achieves 1.65% cost lift and 1.93% GMV lift over the base model. It has been successfully deployed in production since October 2023, serving the WeChat Moments traffic. We have released our code at https://github.com/zhouxy1003/TIN.

cs.IR↗

arXiv2024

Retrieval-Augmented Generation for AI-Generated Content: A Survey

Advancements in model algorithms, the growth of foundational models, and access to high-quality datasets have propelled the evolution of Artificial Intelligence Generated Content (AIGC). Despite its notable successes, AIGC still faces hurdles such as updating knowledge, handling long-tail data, mitigating data leakage, and managing high training and inference costs. Retrieval-Augmented Generation (RAG) has recently emerged as a paradigm to address such challenges. In particular, RAG introduces the information retrieval process, which enhances the generation process by retrieving relevant objects from available data stores, leading to higher accuracy and better robustness. In this paper, we comprehensively review existing efforts that integrate RAG technique into AIGC scenarios. We first classify RAG foundations according to how the retriever augments the generator, distilling the fundamental abstractions of the augmentation methodologies for various retrievers and generators. This unified perspective encompasses all RAG scenarios, illuminating advancements and pivotal technologies that help with potential future progress. We also summarize additional enhancements methods for RAG, facilitating effective engineering and implementation of RAG systems. Then from another view, we survey on practical applications of RAG across different modalities and tasks, offering valuable references for researchers and practitioners. Furthermore, we introduce the benchmarks for RAG, discuss the limitations of current RAG systems, and suggest potential directions for future research. Github: https://github.com/PKU-DAIR/RAG-Survey.

cs.CV↗

arXiv2024

ADSNet: Cross-Domain LTV Prediction with an Adaptive Siamese Network in Advertising

Advertising platforms have evolved in estimating Lifetime Value (LTV) to better align with advertisers' true performance metric. However, the sparsity of real-world LTV data presents a significant challenge to LTV predictive model(i.e., pLTV), severely limiting the their capabilities. Therefore, we propose to utilize external data, in addition to the internal data of advertising platform, to expand the size of purchase samples and enhance the LTV prediction model of the advertising platform. To tackle the issue of data distribution shift between internal and external platforms, we introduce an Adaptive Difference Siamese Network (ADSNet), which employs cross-domain transfer learning to prevent negative transfer. Specifically, ADSNet is designed to learn information that is beneficial to the target domain. We introduce a gain evaluation strategy to calculate information gain, aiding the model in learning helpful information for the target domain and providing the ability to reject noisy samples, thus avoiding negative transfer. Additionally, we also design a Domain Adaptation Module as a bridge to connect different domains, reduce the distribution distance between them, and enhance the consistency of representation space distribution. We conduct extensive offline experiments and online A/B tests on a real advertising platform. Our proposed ADSNet method outperforms other methods, improving GINI by 2$\%$. The ablation study highlights the importance of the gain evaluation strategy in negative gain sample rejection and improving model performance. Additionally, ADSNet significantly improves long-tail prediction. The online A/B tests confirm ADSNet's efficacy, increasing online LTV by 3.47$\%$ and GMV by 3.89$\%$.

cs.IR↗

arXiv2024

The Sun's Magnetic Power Spectra over Two Solar Cycles. \uppercase\expandafter{\romannumeral2}. Cycle Dependence of Active Region, Magnetic Network, and Their Relation

The multi-scaled solar magnetic field consists of two major components: active regions (ARs) and magnetic network. Unraveling the cycle-dependent properties and interrelations of these components is crucial for understanding the evolution of the solar magnetic field. In this study, we investigate these components using magnetic power spectra derived from high-resolution and continuous synoptic magnetograms since cycle 23 onwards. Our results show that the size of the magnetic network ranges from 26 Mm to 41 Mm without dependence on the solar cycle. The power of the network field ($P_{NW}$) accounts for approximately 20\% of the total power during any phase of solar cycles. In contrast to the AR power ($P_{AR}$), $P_{NW}$ displays a weaker cycle dependence, as described by the relationship $P_{NW}$ $\approx$ 0.6* $P_{AR}$ + 40. The power-law index between AR sizes and magnetic network sizes presents a strong anti-correlation with the activity level. Additionally, our study indicates that in the absence of sunspots on the solar disc, the magnetic power spectra remain time-independent, consistently exhibiting similarity in both shape and power. This study introduces a new method to investigate the properties of the magnetic network and provides magnetic power spectra for high-resolution simulations of the solar magnetic field at the surface at various phases of solar cycles.

astro-ph.SR↗

arXiv2024

On the Embedding Collapse when Scaling up Recommendation Models

Recent advances in foundation models have led to a promising trend of developing large recommendation models to leverage vast amounts of available data. Still, mainstream models remain embarrassingly small in size and naïve enlarging does not lead to sufficient performance gain, suggesting a deficiency in the model scalability. In this paper, we identify the embedding collapse phenomenon as the inhibition of scalability, wherein the embedding matrix tends to occupy a low-dimensional subspace. Through empirical and theoretical analysis, we demonstrate a \emph{two-sided effect} of feature interaction specific to recommendation models. On the one hand, interacting with collapsed embeddings restricts embedding learning and exacerbates the collapse issue. On the other hand, interaction is crucial in mitigating the fitting of spurious features as a scalability guarantee. Based on our analysis, we propose a simple yet effective multi-embedding design incorporating embedding-set-specific interaction modules to learn embedding sets with large diversity and thus reduce collapse. Extensive experiments demonstrate that this proposed design provides consistent scalability and effective collapse mitigation for various recommendation models. Code is available at this repository: https://github.com/thuml/Multi-Embedding.

cs.LG↗

arXiv2024

Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding

We present Hunyuan-DiT, a text-to-image diffusion transformer with fine-grained understanding of both English and Chinese. To construct Hunyuan-DiT, we carefully design the transformer structure, text encoder, and positional encoding. We also build from scratch a whole data pipeline to update and evaluate data for iterative model optimization. For fine-grained language understanding, we train a Multimodal Large Language Model to refine the captions of the images. Finally, Hunyuan-DiT can perform multi-turn multimodal dialogue with users, generating and refining images according to the context. Through our holistic human evaluation protocol with more than 50 professional human evaluators, Hunyuan-DiT sets a new state-of-the-art in Chinese-to-image generation compared with other open-source models. Code and pretrained models are publicly available at github.com/Tencent/HunyuanDiT

cs.CV↗

arXiv2024

Toward a live homogeneous database of solar active regions based on SOHO/MDI and SDO/HMI synoptic magnetograms.II.parameters for solar cycle variability

Solar active regions (ARs) determine solar polar fields and cause solar cycle variability within the framework of the Babcock-Leighton (BL) dynamo. The contribution of an AR to the polar field is measured by its dipole field, which results from flux emergence and subsequent flux transport over the solar surface. The dipole fields contributed by an AR before and after the flux transport are referred to as the initial and final dipole fields, respectively. For a better understanding and prediction of solar cycles, in this paper, we provide a database including AR's initial and final dipole fields and the corresponding results of their bipolar magnetic region (BMR) approximation from 1996 onwards. We also identify the repeated ARs and provide the optimized transport parameters. Based on our database, we find that although the commonly used BMR approximation performs well for the initial dipole field, it exhibits a significant deviation for the final dipole field. To accurately assess an AR's contribution to the polar field, the final dipole field with its real configuration should be applied. Despite the notable contributions of a few rogue ARs, approximately the top 500 ARs ordered by their final dipole fields are necessary to derive the polar field at the cycle minimum. While flux transport may increase or decrease the dipole field for an individual AR, its collective impact over all ARs in a cycle is a reduction in their total dipole field.

astro-ph.SR↗

arXiv2024

Deep Event-based Object Detection in Autonomous Driving: A Survey

Object detection plays a critical role in autonomous driving, where accurately and efficiently detecting objects in fast-moving scenes is crucial. Traditional frame-based cameras face challenges in balancing latency and bandwidth, necessitating the need for innovative solutions. Event cameras have emerged as promising sensors for autonomous driving due to their low latency, high dynamic range, and low power consumption. However, effectively utilizing the asynchronous and sparse event data presents challenges, particularly in maintaining low latency and lightweight architectures for object detection. This paper provides an overview of object detection using event data in autonomous driving, showcasing the competitive benefits of event cameras.

cs.CV↗

arXiv2024

Deep Pattern Network for Click-Through Rate Prediction

Click-through rate (CTR) prediction tasks play a pivotal role in real-world applications, particularly in recommendation systems and online advertising. A significant research branch in this domain focuses on user behavior modeling. Current research predominantly centers on modeling co-occurrence relationships between the target item and items previously interacted with by users in their historical data. However, this focus neglects the intricate modeling of user behavior patterns. In reality, the abundance of user interaction records encompasses diverse behavior patterns, indicative of a spectrum of habitual paradigms. These patterns harbor substantial potential to significantly enhance CTR prediction performance. To harness the informational potential within user behavior patterns, we extend Target Attention (TA) to Target Pattern Attention (TPA) to model pattern-level dependencies. Furthermore, three critical challenges demand attention: the inclusion of unrelated items within behavior patterns, data sparsity in behavior patterns, and computational complexity arising from numerous patterns. To address these challenges, we introduce the Deep Pattern Network (DPN), designed to comprehensively leverage information from user behavior patterns. DPN efficiently retrieves target-related user behavior patterns using a target-aware attention mechanism. Additionally, it contributes to refining user behavior patterns through a pre-training paradigm based on self-supervised learning while promoting dependency learning within sparse patterns. Our comprehensive experiments, conducted across three public datasets, substantiate the superior performance and broad compatibility of DPN.

cs.IR↗

arXiv2024

A Quasi-Newton Subspace Trust Region Algorithm for nonmonotone variational inequalities in adversarial learning over box constraints

The first-order optimality condition of convexly constrained nonconvex nonconcave min-max optimization problems with box constraints formulates a nonmonotone variational inequality (VI), which is equivalent to a system of nonsmooth equations. In this paper, we propose a quasi-Newton subspace trust region (QNSTR) algorithm for the least squares problems defined by the smoothing approximation of nonsmooth equations. Based on the structure of the nonmonotone VI, we use an adaptive quasi-Newton formula to approximate the Hessian matrix and solve a low-dimensional strongly convex quadratic program with ellipse constraints in a subspace at each step of the QNSTR algorithm efficiently. We prove the global convergence of the QNSTR algorithm to an $ε$-first-order stationary point of the min-max optimization problem. Moreover, we present numerical results based on the QNSTR algorithm with different subspaces for a mixed generative adversarial networks in eye image segmentation using real data to show the efficiency and effectiveness of the QNSTR algorithm for solving large-scale min-max optimization problems.

math.OC↗

arXiv2024

Numerical Investigation of Two-Dimensional Fokker-Planck Equation in Inflationary Models: Importance of Slow-Roll Parameters

In this study, we generalize the Fokker-Planck equation to two-dimensional cases, including potential functions with periodic boundary conditions and piecewise-defined structures, to analyze the probability distribution in multi-field inflationary models. We employ the spectral method for spatial derivatives and the Crank-Nicolson method for the time evolution to solve the equation numerically for the slow-roll inflation. We find that the distribution in the Fokker-Planck equation was determined by the two-dimensional potential combined slow-roll parameters. And the volume weighting effect makes the distribution in the Fokker-Planck Volume equation is determined by the potential.

gr-qc↗

arXiv2024

Notes on emergent conformal symmetry for black holes

We examine the motion of the massless scalar field and nearly bound null geodesics in the near-ring region of a black hole, which may possess either acceleration or a gravitomagnetic mass. Around such black holes, the photon ring deviates from the equatorial plane. In the large angular momentum limit, we demonstrate that the massless scalar field exhibits an emergent conformal symmetry in this near-ring region. Additionally, in the nearly bound limit, we observe the emergence of a conformal symmetry for the null geodesics that constitute the photon ring in the black hole image. These findings suggest that the hidden conformal symmetry, associated with the Lie algebra $\mathfrak{s l}(2, \mathbb{R})$, persists even for black holes lacking north-south reflection symmetry, thereby broadening the foundation of photon ring holography. Finally, we show that the conformal symmetry also emerges for nearly bound timelike geodesics and scalar fields in proximity to the particle ring, and with specific mass around a Schwarzschild black hole.

hep-th↗

arXiv2024

Towards Training A Chinese Large Language Model for Anesthesiology

Medical large language models (LLMs) have gained popularity recently due to their significant practical utility. However, most existing research focuses on general medicine, and there is a need for in-depth study of LLMs in specific fields like anesthesiology. To fill the gap, we introduce Hypnos, a Chinese Anesthesia model built upon existing LLMs, e.g., Llama. Hypnos' contributions have three aspects: 1) The data, such as utilizing Self-Instruct, acquired from current LLMs likely includes inaccuracies. Hypnos implements a cross-filtering strategy to improve the data quality. This strategy involves using one LLM to assess the quality of the generated data from another LLM and filtering out the data with low quality. 2) Hypnos employs a general-to-specific training strategy that starts by fine-tuning LLMs using the general medicine data and subsequently improving the fine-tuned LLMs using data specifically from Anesthesiology. The general medical data supplement the medical expertise in Anesthesiology and enhance the effectiveness of Hypnos' generation. 3) We introduce a standardized benchmark for evaluating medical LLM in Anesthesiology. Our benchmark includes both publicly available instances from the Internet and privately obtained cases from the Hospital. Hypnos outperforms other medical LLMs in anesthesiology in metrics, GPT-4, and human evaluation on the benchmark dataset.

cs.CL↗

arXiv2024

Modeling effects of starspots on stellar magnetic cycles

Observations show that faster-rotating stars tend to have stronger magnetic activity and shorter magnetic cycles. The cyclical magnetic activity of the Sun and stars is believed to be driven by the dynamo process. The success of the Babcock-Leighton (BL) dynamo in understanding the solar cycle suggests an important role that starspots could play in stellar magnetic cycles. We aim at extending the BL mechanism to solar-mass stars with various rotation rates and explore the effects of emergence properties of starspots in latitudes and tilt angles on stellar magnetic cycles. We adopt a kinematic BL-type dynamo model operating in the bulk of the convection zone. The profiles of the large-scale flow fields are from the mean-field hydrodynamical model for various rotators. The BL source term in the model is constructed based on the rotation dependence of starspots emergence. That is, faster rotators have starspots at higher latitudes with larger tilt angles.Faster rotators have poloidal flux appearing closer to about $\pm55^\circ$ latitudes, where the toroidal field generation efficiency is the strongest because of the strongest latitudinal differential rotation there. It takes a shorter time for faster rotators to transport the surface poloidal field from their emergence latitude to the $\pm 55^\circ$ latitudes of efficient $Ω$-effect thus shortening their magnetic cycles. The faster rotators operate in a more supercritical regime due to a stronger BL $α$-effect relating to the tilt angles, which leads to stronger saturated magnetic fields and a coupling of the poloidal field between two hemispheres more difficult. Thus the magnetic field parity shifts from the hemispherically asymmetric mixed mode to quadrupole, and further to dipole when a star spins down. The emergence of starspots plays an essential role in the large-scale stellar dynamo.

astro-ph.SR↗

arXiv2024

Decoupled Training: Return of Frustratingly Easy Multi-Domain Learning

Multi-domain learning (MDL) aims to train a model with minimal average risk across multiple overlapping but non-identical domains. To tackle the challenges of dataset bias and domain domination, numerous MDL approaches have been proposed from the perspectives of seeking commonalities by aligning distributions to reduce domain gap or reserving differences by implementing domain-specific towers, gates, and even experts. MDL models are becoming more and more complex with sophisticated network architectures or loss functions, introducing extra parameters and enlarging computation costs. In this paper, we propose a frustratingly easy and hyperparameter-free multi-domain learning method named Decoupled Training (D-Train). D-Train is a tri-phase general-to-specific training strategy that first pre-trains on all domains to warm up a root model, then post-trains on each domain by splitting into multi-heads, and finally fine-tunes the heads by fixing the backbone, enabling decouple training to achieve domain independence. Despite its extraordinary simplicity and efficiency, D-Train performs remarkably well in extensive evaluations of various datasets from standard benchmarks to applications of satellite imagery and recommender systems.

cs.LG↗

arXiv2024

A narrow-band parameterization for the stochastic gravitational wave background

In light of the non-perturbative resonance effects that may occur during inflation, we introduce a parametrization for the power spectrum of the stochastic gravitational wave background (SGWB) characterized by narrow-band amplification. We utilize the universal $Ω_\text{GW}\propto k^3$ infrared limit, applicable to a wide array of gravitational wave sources, to devise a robust yet straightforward parameterization optimized for Markov Chain Monte Carlo (MCMC) analyses. This parameterization is demonstrated through select examples where its application is pertinent, and we discuss the advantages of this approach over traditional parametrizations for narrow-band scenarios. To evaluate the sensitivity of our proposed model parameters, we apply a mock likelihood based on the CMB-Stage4 data. Furthermore, we explicate the computational process for the mapping relationship between the foundational model parameters and our parameterized framework, using a two-field inflation model that resonantly amplifies gravitational waves (GWs) as an example.

astro-ph.CO↗

arXiv2024

Pareto-based Multi-Objective Recommender System with Forgetting Curve

Recommender systems with cascading architecture play an increasingly significant role in online recommendation platforms, where the approach to dealing with negative feedback is a vital issue. For instance, in short video platforms, users tend to quickly slip away from candidates that they feel aversive, and recommender systems are expected to receive these explicit negative feedbacks and make adjustments to avoid these recommendations. Considering recency effect in memories, we propose a forgetting model based on Ebbinghaus Forgetting Curve to cope with negative feedback. In addition, we introduce a Pareto optimization solver to guarantee a better trade-off between recency and model performance. In conclusion, we propose Pareto-based Multi-Objective Recommender System with forgetting curve (PMORS), which can be applied to any multi-objective recommendation and show sufficiently superiority when facing explicit negative feedback. We have conducted evaluations of PMORS and achieved favorable outcomes in short-video scenarios on both public dataset and industrial dataset. After being deployed on an online short video platform named WeChat Channels in May, 2023, PMORS has not only demonstrated promising results for both consistency and recency but also achieved an improvement of up to +1.45% GMV.

cs.IR↗