SearcharxivSearch

arXiv subjects

Idris Malik

Publications and source records attributed to Idris Malik.

2 recordsLinked to original sources

A hierarchy tree data structure for behavior-based user segment representation

User attributes are essential in multiple stages of modern recommendation systems and are particularly important for mitigating the cold-start problem and improving the experience of new or infrequent users. We propose Behavior-based User Segmentation (BUS), a novel tree-based data structure that hierarchically segments the user universe with various users' categorical attributes based on the users' product-specific engagement behaviors. During the BUS tree construction, we use Normalized Discounted Cumulative Gain (NDCG) as the objective function to maximize the behavioral representativeness of marginal users relative to active users in the same segment. The constructed BUS tree undergoes further processing and aggregation across the leaf nodes and internal nodes, allowing the generation of popular social content and behavioral patterns for each node in the tree. To further mitigate bias and improve fairness, we use the social graph to derive the user's connection-based BUS segments, enabling the combination of behavioral patterns extracted from both the user's own segment and connection-based segments as the connection aware BUS-based recommendation. Our offline analysis shows that the BUS-based retrieval significantly outperforms traditional user cohort-based aggregation on ranking quality. We have successfully deployed our data structure and machine learning algorithm and tested it with various production traffic serving billions of users daily, achieving statistically significant improvements in the online product metrics, including music ranking and email notifications. To the best of our knowledge, our study represents the first list-wise learning-to-rank framework for tree-based recommendation that effectively integrates diverse user categorical attributes while preserving real-world semantic interpretability at a large industrial scale.

cs.LG

Personalized Execution Time Optimization for Billion-Scale Scheduled Jobs

Scheduled batch jobs are widely used on asynchronous computing platforms to execute enterprise applications such as promotional notifications and candidate pre-computation for recommender systems. Delivering or updating information at the right time is important for user experience and execution impact, yet providing a versatile, personalized execution time optimization solution across diverse product scenarios while maintaining reasonable infrastructure costs remains challenging. In this paper, we present a deployed system that serves billions of users daily, combining learning-to-rank with a "best time policy" for execution time selection. We describe the four-stage evolution of our approach: from heuristic peak-hour rules, to pointwise ML-based activity pattern predictions, to a linear signal assembler with globally fixed weights, and finally to a contextual ensemble learner that produces per-user adaptive fusion weights via a neural policy network trained with listwise learning-to-rank objectives. We further report the discovery of cross-use-case cannibalization effects and introduce a coordination system to mitigate the problem. Our production experiments demonstrate measurable improvements in both execution efficiency and downstream product impact. We share deployment lessons including failure analyses and design decisions accumulated over four years of operating this system at scale. To our knowledge, this represents the first ML-based multi-tenant execution time optimization system deployed across different product domains at industrial scale.

cs.LG