arXiv subjects

Bo Zheng

Publications and source records attributed to Bo Zheng.

At least 397 records · Page 22Linked to original sources

arXiv2021

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment

The cross-lingual language models are typically pretrained with masked language modeling on multilingual text or parallel sentences. In this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. Specifically, the model first self-labels word alignments for parallel sentences. Then we randomly mask tokens in a bitext pair. Given a masked token, the model uses a pointer network to predict the aligned token in the other language. We alternately perform the above two steps in an expectation-maximization manner. Experimental results show that our method improves cross-lingual transferability on various datasets, especially on the token-level tasks, such as question answering, and structured prediction. Moreover, the model can serve as a pretrained word aligner, which achieves reasonably low error rates on the alignment benchmarks. The code and pretrained parameters are available at https://github.com/CZWin32768/XLM-Align.

cs.CL↗

arXiv2021

Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer

Embedding learning for categorical features is crucial for the deep learning-based recommendation models (DLRMs). Each feature value is mapped to an embedding vector via an embedding learning process. Conventional methods configure a fixed and uniform embedding size to all feature values from the same feature field. However, such a configuration is not only sub-optimal for embedding learning but also memory costly. Existing methods that attempt to resolve these problems, either rule-based or neural architecture search (NAS)-based, need extensive efforts on the human design or network training. They are also not flexible in embedding size selection or in warm-start-based applications. In this paper, we propose a novel and effective embedding size selection scheme. Specifically, we design an Adaptively-Masked Twins-based Layer (AMTL) behind the standard embedding layer. AMTL generates a mask vector to mask the undesired dimensions for each embedding vector. The mask vector brings flexibility in selecting the dimensions and the proposed layer can be easily added to either untrained or trained DLRMs. Extensive experimental evaluations show that the proposed scheme outperforms competitive baselines on all the benchmark tasks, and is also memory-efficient, saving 60\% memory usage without compromising any performance metrics.

cs.LG↗

arXiv2021

Binary Code based Hash Embedding for Web-scale Applications

Nowadays, deep learning models are widely adopted in web-scale applications such as recommender systems, and online advertising. In these applications, embedding learning of categorical features is crucial to the success of deep learning models. In these models, a standard method is that each categorical feature value is assigned a unique embedding vector which can be learned and optimized. Although this method can well capture the characteristics of the categorical features and promise good performance, it can incur a huge memory cost to store the embedding table, especially for those web-scale applications. Such a huge memory cost significantly holds back the effectiveness and usability of EDRMs. In this paper, we propose a binary code based hash embedding method which allows the size of the embedding table to be reduced in arbitrary scale without compromising too much performance. Experimental evaluation results show that one can still achieve 99\% performance even if the embedding table size is reduced 1000$\times$ smaller than the original one with our proposed method.

cs.IR↗

arXiv2021

Consistency Regularization for Cross-Lingual Fine-Tuning

Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others. In this work, we propose to improve cross-lingual fine-tuning with consistency regularization. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations, i.e., subword sampling, Gaussian noise, code-switch substitution, and machine translation. In addition, we employ model consistency to regularize the models trained with two augmented versions of the same training set. Experimental results on the XTREME benchmark show that our method significantly improves cross-lingual fine-tuning across various tasks, including text classification, question answering, and sequence labeling.

cs.CL↗

arXiv2021

Multi-Agent Cooperative Bidding Games for Multi-Objective Optimization in e-Commercial Sponsored Search

Bid optimization for online advertising from single advertiser's perspective has been thoroughly investigated in both academic research and industrial practice. However, existing work typically assume competitors do not change their bids, i.e., the wining price is fixed, leading to poor performance of the derived solution. Although a few studies use multi-agent reinforcement learning to set up a cooperative game, they still suffer the following drawbacks: (1) They fail to avoid collusion solutions where all the advertisers involved in an auction collude to bid an extremely low price on purpose. (2) Previous works cannot well handle the underlying complex bidding environment, leading to poor model convergence. This problem could be amplified when handling multiple objectives of advertisers which are practical demands but not considered by previous work. In this paper, we propose a novel multi-objective cooperative bid optimization formulation called Multi-Agent Cooperative bidding Games (MACG). MACG sets up a carefully designed multi-objective optimization framework where different objectives of advertisers are incorporated. A global objective to maximize the overall profit of all advertisements is added in order to encourage better cooperation and also to protect self-bidding advertisers. To avoid collusion, we also introduce an extra platform revenue constraint. We analyze the optimal functional form of the bidding formula theoretically and design a policy network accordingly to generate auction-level bids. Then we design an efficient multi-agent evolutionary strategy for model optimization. Offline experiments and online A/B tests conducted on the Taobao platform indicate both single advertiser's objective and global profit have been significantly improved compared to state-of-art methods.

cs.AI↗

arXiv2021

Towards a Better Tradeoff between Effectiveness and Efficiency in Pre-Ranking: A Learnable Feature Selection based Approach

In real-world search, recommendation, and advertising systems, the multi-stage ranking architecture is commonly adopted. Such architecture usually consists of matching, pre-ranking, ranking, and re-ranking stages. In the pre-ranking stage, vector-product based models with representation-focused architecture are commonly adopted to account for system efficiency. However, it brings a significant loss to the effectiveness of the system. In this paper, a novel pre-ranking approach is proposed which supports complicated models with interaction-focused architecture. It achieves a better tradeoff between effectiveness and efficiency by utilizing the proposed learnable Feature Selection method based on feature Complexity and variational Dropout (FSCD). Evaluations in a real-world e-commerce sponsored search system for a search engine demonstrate that utilizing the proposed pre-ranking, the effectiveness of the system is significantly improved. Moreover, compared to the systems with conventional pre-ranking models, an identical amount of computational resource is consumed.

cs.IR↗

arXiv2021

Explicit Semantic Cross Feature Learning via Pre-trained Graph Neural Networks for CTR Prediction

Cross features play an important role in click-through rate (CTR) prediction. Most of the existing methods adopt a DNN-based model to capture the cross features in an implicit manner. These implicit methods may lead to a sub-optimized performance due to the limitation in explicit semantic modeling. Although traditional statistical explicit semantic cross features can address the problem in these implicit methods, it still suffers from some challenges, including lack of generalization and expensive memory cost. Few works focus on tackling these challenges. In this paper, we take the first step in learning the explicit semantic cross features and propose Pre-trained Cross Feature learning Graph Neural Networks (PCF-GNN), a GNN based pre-trained model aiming at generating cross features in an explicit fashion. Extensive experiments are conducted on both public and industrial datasets, where PCF-GNN shows competence in both performance and memory-efficiency in various tasks.

cs.AI↗

arXiv2021

4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface

Tracking non-rigidly deforming scenes using range sensors has numerous applications including computer vision, AR/VR, and robotics. However, due to occlusions and physical limitations of range sensors, existing methods only handle the visible surface, thus causing discontinuities and incompleteness in the motion field. To this end, we introduce 4DComplete, a novel data-driven approach that estimates the non-rigid motion for the unobserved geometry. 4DComplete takes as input a partial shape and motion observation, extracts 4D time-space embedding, and jointly infers the missing geometry and motion field using a sparse fully-convolutional network. For network training, we constructed a large-scale synthetic dataset called DeformingThings4D, which consists of 1972 animation sequences spanning 31 different animals or humanoid categories with dense 4D annotation. Experiments show that 4DComplete 1) reconstructs high-resolution volumetric shape and motion field from a partial observation, 2) learns an entangled 4D feature representation that benefits both shape and motion estimation, 3) yields more accurate and natural deformation than classic non-rigid priors such as As-Rigid-As-Possible (ARAP) deformation, and 4) generalizes well to unseen objects in real-world sequences.

cs.CV↗

arXiv2021

Tracking Air Pollution in China: Near Real-Time PM2.5 Retrievals from Multiple Data Sources

Air pollution has altered the Earth radiation balance, disturbed the ecosystem and increased human morbidity and mortality. Accordingly, a full-coverage high-resolution air pollutant dataset with timely updates and historical long-term records is essential to support both research and environmental management. Here, for the first time, we develop a near real-time air pollutant database known as Tracking Air Pollution in China (TAP, tapdata.org) that combines information from multiple data sources, including ground measurements, satellite retrievals, dynamically updated emission inventories, operational chemical transport model simulations and other ancillary data. Daily full-coverage PM2.5 data at a spatial resolution of 10 km is our first near real-time product. The TAP PM2.5 is estimated based on a two-stage machine learning model coupled with the synthetic minority oversampling technique and a tree-based gap-filling method. Our model has an averaged out-of-bag cross-validation R2 of 0.83 for different years, which is comparable to those of other studies, but improves its performance at high pollution levels and fills the gaps in missing AOD on daily scale. The full coverage and near real-time updates of the daily PM2.5 data allow us to track the day-to-day variations in PM2.5 concentrations over China in a timely manner. The long-term records of PM2.5 data since 2000 will also support policy assessments and health impact studies. The TAP PM2.5 data are publicly available through our website for sharing with the research and policy communities.

physics.ao-ph↗

arXiv2021

Global Daily CO$_2$ emissions for the year 2020

The diurnal cycle CO$_2$ emissions from fossil fuel combustion and cement production reflect seasonality, weather conditions, working days, and more recently the impact of the COVID-19 pandemic. Here, for the first time we provide a daily CO$_2$ emission dataset for the whole year of 2020 calculated from inventory and near-real-time activity data (called Carbon Monitor project: https://carbonmonitor.org). It was previously suggested from preliminary estimates that did not cover the entire year of 2020 that the pandemics may have caused more than 8% annual decline of global CO$_2$ emissions. Here we show from detailed estimates of the full year data that the global reduction was only 5.4% (-1,901 MtCO$_2$, ). This decrease is 5 times larger than the annual emission drop at the peak of the 2008 Global Financial Crisis. However, global CO$_2$ emissions gradually recovered towards 2019 levels from late April with global partial re-opening. More importantly, global CO$_2$ emissions even increased slightly by +0.9% in December 2020 compared with 2019, indicating the trends of rebound of global emissions. Later waves of COVID-19 infections in late 2020 and corresponding lockdowns have caused further CO$_2$ emissions reductions particularly in western countries, but to a much smaller extent than the declines in the first wave. That even substantial world-wide lockdowns of activity led to a one-time decline in global CO$_2$ emissions of only 5.4% in one year highlights the significant challenges for climate change mitigation that we face in the post-COVID era. These declines are significant, but will be quickly overtaken with new emissions unless the COVID-19 crisis is utilized as a break-point with our fossil-fuel trajectory, notably through policies that make the COVID-19 recovery an opportunity to green national energy and development plans.

physics.ao-ph↗

arXiv2021

De-carbonization of global energy use during the COVID-19 pandemic

The COVID-19 pandemic has disrupted human activities, leading to unprecedented decreases in both global energy demand and GHG emissions. Yet a little known that there is also a low carbon shift of the global energy system in 2020. Here, using the near-real-time data on energy-related GHG emissions from 30 countries (about 70% of global power generation), we show that the pandemic caused an unprecedented de-carbonization of global power system, representing by a dramatic decrease in the carbon intensity of power sector that reached a historical low of 414.9 tCO2eq/GWh in 2020. Moreover, the share of energy derived from renewable and low-carbon sources (nuclear, hydro-energy, wind, solar, geothermal, and biomass) exceeded that from coal and oil for the first time in history in May of 2020. The decrease in global net energy demand (-1.3% in the first half of 2020 relative to the average of the period in 2016-2019) masks a large down-regulation of fossil-fuel-burning power plants supply (-6.1%) coincident with a surge of low-carbon sources (+6.2%). Concomitant changes in the diurnal cycle of electricity demand also favored low-carbon generators, including a flattening of the morning ramp, a lower midday peak, and delays in both the morning and midday load peaks in most countries. However, emission intensities in the power sector have since rebounded in many countries, and a key question for climate mitigation is thus to what extent countries can achieve and maintain lower, pandemic-level carbon intensities of electricity as part of a green recovery.

physics.ao-ph↗

arXiv2020

Satellite-based estimates of decline and rebound in China's CO$_2$ emissions during COVID-19 pandemic

Changes in CO$_2$ emissions during the COVID-19 pandemic have been estimated from indicators on activities like transportation and electricity generation. Here, we instead use satellite observations together with bottom-up information to track the daily dynamics of CO$_2$ emissions during the pandemic. Unlike activity data, our observation-based analysis can be independently evaluated and can provide more detailed insights into spatially-explicit changes. Specifically, we use TROPOMI observations of NO$_2$ to deduce ten-day moving averages of NO$_x$ and CO$_2$ emissions over China, differentiating emissions by sector and province. Between January and April 2020, China's CO$_2$ emissions fell by 11.5% compared to the same period in 2019, but emissions have since rebounded to pre-pandemic levels owing to the fast economic recovery in provinces where industrial activity is concentrated.

physics.ao-ph↗

arXiv2020

COVID-19 causes record decline in global CO2 emissions

The considerable cessation of human activities during the COVID-19 pandemic has affected global energy use and CO2 emissions. Here we show the unprecedented decrease in global fossil CO2 emissions from January to April 2020 was of 7.8% (938 Mt CO2 with a +6.8% of 2-σ uncertainty) when compared with the period last year. In addition other emerging estimates of COVID impacts based on monthly energy supply or estimated parameters, this study contributes to another step that constructed the near-real-time daily CO2 emission inventories based on activity from power generation (for 29 countries), industry (for 73 countries), road transportation (for 406 cities), aviation and maritime transportation and commercial and residential sectors emissions (for 206 countries). The estimates distinguished the decline of CO2 due to COVID-19 from the daily, weekly and seasonal variations as well as the holiday events. The COVID-related decreases in CO2 emissions in road transportation (340.4 Mt CO2, -15.5%), power (292.5 Mt CO2, -6.4% compared to 2019), industry (136.2 Mt CO2, -4.4%), aviation (92.8 Mt CO2, -28.9%), residential (43.4 Mt CO2, -2.7%), and international shipping (35.9Mt CO2, -15%). Regionally, decreases in China were the largest and earliest (234.5 Mt CO2,-6.9%), followed by Europe (EU-27 & UK) (138.3 Mt CO2, -12.0%) and the U.S. (162.4 Mt CO2, -9.5%). The declines of CO2 are consistent with regional nitrogen oxides concentrations observed by satellites and ground-based networks, but the calculated signal of emissions decreases (about 1Gt CO2) will have little impacts (less than 0.13ppm by April 30, 2020) on the overserved global CO2 concertation. However, with observed fast CO2 recovery in China and partial re-opening globally, our findings suggest the longer-term effects on CO2 emissions are unknown and should be carefully monitored using multiple measures.

econ.GN↗

arXiv2020

Carbon Monitor: a near-real-time daily dataset of global CO2 emission from fossil fuel and cement production

We constructed a near-real-time daily CO2 emission dataset, namely the Carbon Monitor, to monitor the variations of CO2 emissions from fossil fuel combustion and cement production since January 1st 2019 at national level with near-global coverage on a daily basis, with the potential to be frequently updated. Daily CO2 emissions are estimated from a diverse range of activity data, including: hourly to daily electrical power generation data of 29 countries, monthly production data and production indices of industry processes of 62 countries/regions, daily mobility data and mobility indices of road transportation of 416 cities worldwide. Individual flight location data and monthly data were utilised for aviation and maritime transportation sectors estimates. In addition, monthly fuel consumption data that corrected for daily air temperature of 206 countries were used for estimating the emissions from commercial and residential buildings. This Carbon Monitor dataset manifests the dynamic nature of CO2 emissions through daily, weekly and seasonal variations as influenced by workdays and holidays, as well as the unfolding impacts of the COVID-19 pandemic. The Carbon Monitor near-real-time CO2 emission dataset shows a 7.8% decline of CO2 emission globally from Jan 1st to Apr 30th in 2020 when compared with the same period in 2019, and detects a re-growth of CO2 emissions by late April which are mainly attributed to the recovery of economy activities in China and partial easing of lockdowns in other countries. Further, this daily updated CO2 emission dataset could offer a range of opportunities for related scientific research and policy making.

physics.soc-ph↗

arXiv2020

Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension

Natural Questions is a new challenging machine reading comprehension benchmark with two-grained answers, which are a long answer (typically a paragraph) and a short answer (one or more entities inside the long answer). Despite the effectiveness of existing methods on this benchmark, they treat these two sub-tasks individually during training while ignoring their dependencies. To address this issue, we present a novel multi-grained machine reading comprehension framework that focuses on modeling documents at their hierarchical nature, which are different levels of granularity: documents, paragraphs, sentences, and tokens. We utilize graph attention networks to obtain different levels of representations so that they can be learned simultaneously. The long and short answers can be extracted from paragraph-level representation and token-level representation, respectively. In this way, we can model the dependencies between the two-grained answers to provide evidence for each other. We jointly train the two sub-tasks, and our experiments show that our approach significantly outperforms previous systems at both long and short answer criteria.

cs.CL↗

arXiv2020

Natural gas shortages during the "coal-to-gas" transition in China have caused a large redistribution of air pollution

The Chinese "coal-to-gas" strategy aims at reducing coal consumption and related air pollution by promoting the use of clean and low carbon fuels in northern China. Here we show that on top of meteorological influences, these measures achieved an average decrease of fine particulate matter (PM2.5) concentrations of ~14% during winter 2017 in Beijing and surrounding areas (the "2+26" pilot cities). However, the localized air quality improvement was accompanied by a contemporaneous ~15% upsurge of PM2.5 concentrations over large areas in southern China. We find that the pollution transfer that resulted from a shift in emissions was caused by a natural gas shortage in the south due to the "coal-to-gas" transition in the north. The overall shortage of natural gas greatly jeopardized the air quality benefits of the "coal-to-gas" strategy in winter 2017 and reflects structural challenges and potential threats in China's clean energy transition.

physics.ao-ph↗

arXiv2020

Abrupt declines in tropospheric nitrogen dioxide over China after the outbreak of COVID-19

China's policy interventions to reduce the spread of the coronavirus disease 2019 have environmental and economic impacts. Tropospheric nitrogen dioxide indicates economic activities, as nitrogen dioxide is primarily emitted from fossil fuel consumption. Satellite measurements show a 48% drop in tropospheric nitrogen dioxide vertical column densities from the 20 days averaged before the 2020 Lunar New Year to the 20 days averaged after. This is 20% larger than that from recent years. We relate to this reduction to two of the government's actions: the announcement of the first report in each province and the date of a province's lockdown. Both actions are associated with nearly the same magnitude of reductions. Our analysis offers insights into the unintended environmental and economic consequences through reduced economic activities.

physics.ao-ph↗

arXiv2020

Adversarial Multimodal Representation Learning for Click-Through Rate Prediction

For better user experience and business effectiveness, Click-Through Rate (CTR) prediction has been one of the most important tasks in E-commerce. Although extensive CTR prediction models have been proposed, learning good representation of items from multimodal features is still less investigated, considering an item in E-commerce usually contains multiple heterogeneous modalities. Previous works either concatenate the multiple modality features, that is equivalent to giving a fixed importance weight to each modality; or learn dynamic weights of different modalities for different items through technique like attention mechanism. However, a problem is that there usually exists common redundant information across multiple modalities. The dynamic weights of different modalities computed by using the redundant information may not correctly reflect the different importance of each modality. To address this, we explore the complementarity and redundancy of modalities by considering modality-specific and modality-invariant features differently. We propose a novel Multimodal Adversarial Representation Network (MARN) for the CTR prediction task. A multimodal attention network first calculates the weights of multiple modalities for each item according to its modality-specific features. Then a multimodal adversarial network learns modality-invariant representations where a double-discriminators strategy is introduced. Finally, we achieve the multimodal item representations by combining both modality-specific and modality-invariant representations. We conduct extensive experiments on both public and industrial datasets, and the proposed method consistently achieves remarkable improvements to the state-of-the-art methods. Moreover, the approach has been deployed in an operational E-commerce system and online A/B testing further demonstrates the effectiveness.

cs.IR↗