arXiv subjects

Tong Chen

Publications and source records attributed to Tong Chen.

At least 235 records · Page 13Linked to original sources

arXiv2021

Learning Elastic Embeddings for Customizing On-Device Recommenders

In today's context, deploying data-driven services like recommendation on edge devices instead of cloud servers becomes increasingly attractive due to privacy and network latency concerns. A common practice in building compact on-device recommender systems is to compress their embeddings which are normally the cause of excessive parameterization. However, despite the vast variety of devices and their associated memory constraints, existing memory-efficient recommender systems are only specialized for a fixed memory budget in every design and training life cycle, where a new model has to be retrained to obtain the optimal performance while adapting to a smaller/larger memory budget. In this paper, we present a novel lightweight recommendation paradigm that allows a well-trained recommender to be customized for arbitrary device-specific memory constraints without retraining. The core idea is to compose elastic embeddings for each item, where an elastic embedding is the concatenation of a set of embedding blocks that are carefully chosen by an automated search function. Correspondingly, we propose an innovative approach, namely recommendation with universally learned elastic embeddings (RULE). To ensure the expressiveness of all candidate embedding blocks, RULE enforces a diversity-driven regularization when learning different embedding blocks. Then, a performance estimator-based evolutionary search function is designed, allowing for efficient specialization of elastic embeddings under any memory constraint for on-device recommendation. Extensive experiments on real-world datasets reveal the superior performance of RULE under tight memory budgets.

cs.IR↗

arXiv2021

Semialgebraic Representation of Monotone Deep Equilibrium Models and Applications to Certification

Deep equilibrium models are based on implicitly defined functional relations and have shown competitive performance compared with the traditional deep networks. Monotone operator equilibrium networks (monDEQ) retain interesting performance with additional theoretical guaranties. Existing certification tools for classical deep networks cannot directly be applied to monDEQs for which much fewer tools exist. We introduce a semialgebraic representation for ReLU based monDEQs which allows to approximate the corresponding input output relation by semidefinite programming (SDP). We present several applications to network certification and obtain SDP models for the following problems : robustness certification, Lipschitz constant estimation, ellipsoidal uncertainty propagation. We use these models to certify robustness of monDEQs w.r.t. a general $L_q$ norm. Experimental results show that the proposed models outperform existing approaches for monDEQ certification. Furthermore, our investigations suggest that monDEQs are much more robust to $L_2$ perturbations than $L_{\infty}$ perturbations.

math.OC↗

arXiv2021

Learning to Ask Appropriate Questions in Conversational Recommendation

Conversational recommender systems (CRSs) have revolutionized the conventional recommendation paradigm by embracing dialogue agents to dynamically capture the fine-grained user preference. In a typical conversational recommendation scenario, a CRS firstly generates questions to let the user clarify her/his demands and then makes suitable recommendations. Hence, the ability to generate suitable clarifying questions is the key to timely tracing users' dynamic preferences and achieving successful recommendations. However, existing CRSs fall short in asking high-quality questions because: (1) system-generated responses heavily depends on the performance of the dialogue policy agent, which has to be trained with huge conversation corpus to cover all circumstances; and (2) current CRSs cannot fully utilize the learned latent user profiles for generating appropriate and personalized responses. To mitigate these issues, we propose the Knowledge-Based Question Generation System (KBQG), a novel framework for conversational recommendation. Distinct from previous conversational recommender systems, KBQG models a user's preference in a finer granularity by identifying the most relevant relations from a structured knowledge graph (KG). Conditioned on the varied importance of different relations, the generated clarifying questions could perform better in impelling users to provide more details on their preferences. Finially, accurate recommendations can be generated in fewer conversational turns. Furthermore, the proposed KBQG outperforms all baselines in our experiments on two real-world datasets.

cs.AI↗

arXiv2021

DA-GCN: A Domain-aware Attentive Graph Convolution Network for Shared-account Cross-domain Sequential Recommendation

Shared-account Cross-domain Sequential recommendation (SCSR) is the task of recommending the next item based on a sequence of recorded user behaviors, where multiple users share a single account, and their behaviours are available in multiple domains. Existing work on solving SCSR mainly relies on mining sequential patterns via RNN-based models, which are not expressive enough to capture the relationships among multiple entities. Moreover, all existing algorithms try to bridge two domains via knowledge transfer in the latent space, and the explicit cross-domain graph structure is unexploited. In this work, we propose a novel graph-based solution, namely DA-GCN, to address the above challenges. Specifically, we first link users and items in each domain as a graph. Then, we devise a domain-aware graph convolution network to learn user-specific node representations. To fully account for users' domain-specific preferences on items, two novel attention mechanisms are further developed to selectively guide the message passing process. Extensive experiments on two real-world datasets are conducted to demonstrate the superiority of our DA-GCN method.

cs.IR↗

arXiv2021

Pressure-induced high-temperature superconductivity retained at ambient

To raise the superconducting-transition temperature (Tc) has been the driving force for the long, sustained effort in superconductivity research. Recent progress in hydrides with Tcs up to 287 K under 267 GPa has heralded a new era of room-temperature superconductivity (RTS) with immense technological promise. Indeed, RTS has lifted the temperature barrier for the ubiquitous application of superconductivity. Unfortunately, formidable pressure is required to attain such high Tcs. The most effective relief to this impasse is to remove the pressure needed while retaining the pressure-induced Tc without pressure. Here we show such a possibility in the pure and doped high-temperature superconductor (HTS) FeSe by retaining, at ambient via pressure-quenching (PQ), its Tc up to 37 K (quadrupling that of a pristine FeSe) and other pressure-induced phases. We have also observed that some phases remain stable without pressure at up to 300 K and for at least 7 days. The observations are in qualitative agreement with our ab initio simulations using the solid-state nudged elastic band (SSNEB) method. We strongly believe that the PQ technique developed here can be adapted to the RTS hydrides and other materials of value with minimal effort.

cond-mat.supr-con↗

arXiv2021

Uniting Heterogeneity, Inductiveness, and Efficiency for Graph Representation Learning

With the ubiquitous graph-structured data in various applications, models that can learn compact but expressive vector representations of nodes have become highly desirable. Recently, bearing the message passing paradigm, graph neural networks (GNNs) have greatly advanced the performance of node representation learning on graphs. However, a majority class of GNNs are only designed for homogeneous graphs, leading to inferior adaptivity to the more informative heterogeneous graphs with various types of nodes and edges. Also, despite the necessity of inductively producing representations for completely new nodes (e.g., in streaming scenarios), few heterogeneous GNNs can bypass the transductive learning scheme where all nodes must be known during training. Furthermore, the training efficiency of most heterogeneous GNNs has been hindered by their sophisticated designs for extracting the semantics associated with each meta path or relation. In this paper, we propose WIde and DEep message passing Network (WIDEN) to cope with the aforementioned problems about heterogeneity, inductiveness, and efficiency that are rarely investigated together in graph representation learning. In WIDEN, we propose a novel inductive, meta path-free message passing scheme that packs up heterogeneous node features with their associated edges from both low- and high-order neighbor nodes. To further improve the training efficiency, we innovatively present an active downsampling strategy that drops unimportant neighbor nodes to facilitate faster information propagation. Experiments on three real-world heterogeneous graphs have further validated the efficacy of WIDEN on both transductive and inductive node representation learning, as well as the superior training efficiency against state-of-the-art baselines.

cs.SI↗

arXiv2021

Quaternion Factorization Machines: A Lightweight Solution to Intricate Feature Interaction Modelling

As a well-established approach, factorization machine (FM) is capable of automatically learning high-order interactions among features to make predictions without the need for manual feature engineering. With the prominent development of deep neural networks (DNNs), there is a recent and ongoing trend of enhancing the expressiveness of FM-based models with DNNs. However, though better results are obtained with DNN-based FM variants, such performance gain is paid off by an enormous amount (usually millions) of excessive model parameters on top of the plain FM. Consequently, the heavy parameterization impedes the real-life practicality of those deep models, especially efficient deployment on resource-constrained IoT and edge devices. In this paper, we move beyond the traditional real space where most deep FM-based models are defined, and seek solutions from quaternion representations within the hypercomplex space. Specifically, we propose the quaternion factorization machine (QFM) and quaternion neural factorization machine (QNFM), which are two novel lightweight and memory-efficient quaternion-valued models for sparse predictive analytics. By introducing a brand new take on FM-based models with the notion of quaternion algebra, our models not only enable expressive inter-component feature interactions, but also significantly reduce the parameter size due to lower degrees of freedom in the hypercomplex Hamilton product compared with real-valued matrix multiplication. Extensive experimental results on three large-scale datasets demonstrate that QFM achieves 4.36% performance improvement over the plain FM without introducing any extra parameters, while QNFM outperforms all baselines with up to two magnitudes' parameter size reduction in comparison to state-of-the-art peer methods.

cs.IR↗

arXiv2021

Hierarchical Hyperedge Embedding-based Representation Learning for Group Recommendation

In this work, we study group recommendation in a particular scenario, namely Occasional Group Recommendation (OGR). Most existing works have addressed OGR by aggregating group members' personal preferences to learn the group representation. However, the representation learning for a group is most complex beyond the fusion of group member representation, as the personal preferences and group preferences may be in different spaces. In addition, the learned user representation is not accurate due to the sparsity of users' interaction data. Moreover, the group similarity in terms of common group members has been overlooked, which however has the great potential to improve the group representation learning. In this work, we focus on addressing the above challenges in group representation learning task, and devise a hierarchical hyperedge embedding-based group recommender, namely HyperGroup. Specifically, we propose to leverage the user-user interactions to alleviate the sparsity issue of user-item interactions, and design a GNN-based representation learning network to enhance the learning of individuals' preferences from their friends' preferences, which provides a solid foundation for learning groups' preferences. To exploit the group similarity to learn a more accurate group representation from highly limited group-item interactions, we connect all groups as a network of overlapping sets, and treat the task of group preference learning as embedding hyperedges in a hypergraph, where an inductive hyperedge embedding method is proposed. To further enhance the group-level preference modeling, we develop a joint training strategy to learn both user-item and group-item interactions in the same process. We conduct extensive experiments on two real-world datasets and the experimental results demonstrate the superiority of our proposed HyperGroup in comparison to the state-of-the-art baselines.

cs.IR↗

arXiv2021

Recognizing Micro-Expression in Video Clip with Adaptive Key-Frame Mining

As a spontaneous expression of emotion on face, micro-expression reveals the underlying emotion that cannot be controlled by human. In micro-expression, facial movement is transient and sparsely localized through time. However, the existing representation based on various deep learning techniques learned from a full video clip is usually redundant. In addition, methods utilizing the single apex frame of each video clip require expert annotations and sacrifice the temporal dynamics. To simultaneously localize and recognize such fleeting facial movements, we propose a novel end-to-end deep learning architecture, referred to as adaptive key-frame mining network (AKMNet). Operating on the video clip of micro-expression, AKMNet is able to learn discriminative spatio-temporal representation by combining spatial features of self-learned local key frames and their global-temporal dynamics. Theoretical analysis and empirical evaluation show that the proposed approach improved recognition accuracy in comparison with state-of-the-art methods on multiple benchmark datasets.

cs.CV↗

arXiv2021

Non-Thermal Emergence of an Orbital-Selective Mott Phase in FeTe$_{1-x}$Se$_x$

Electronic correlation is of fundamental importance to high temperature superconductivity. Iron-based superconductors are believed to possess moderate correlation strength, which combined with their multi-orbital nature makes them a fascinating platform for the emergence of exotic phenomena. A particularly striking form is the emergence of an orbital selective Mott phase, where the localization of a subset of orbitals leads to a drastically reconstructed Fermi surface. Here, we report spectroscopic evidence of the reorganization of the Fermi surface from FeSe to FeTe as Se is substituted by Te. We uncover a particularly transparent way to visualize the localization of the $d_{xy}$ electron orbital through the suppression of its hybridization with the more coherent $d$ electron orbitals, which leads to a redistribution of the orbital-dependent spectral weight near the Fermi level. These noteworthy features of the Fermi surface are accompanied by a divergent behavior of a band renormalization in the $d_{xy}$ orbital. All of our observations are further supported by our theoretical calculations to be salient spectroscopic signatures of such a non-thermal evolution from a strongly correlated metallic phase towards an orbital-selective Mott phase in FeTe$_{1-x}$Se$_x$ as Se concentration is reduced.

cond-mat.supr-con↗

arXiv2021

Graph Embedding for Recommendation against Attribute Inference Attacks

In recent years, recommender systems play a pivotal role in helping users identify the most suitable items that satisfy personal preferences. As user-item interactions can be naturally modelled as graph-structured data, variants of graph convolutional networks (GCNs) have become a well-established building block in the latest recommenders. Due to the wide utilization of sensitive user profile data, existing recommendation paradigms are likely to expose users to the threat of privacy breach, and GCN-based recommenders are no exception. Apart from the leakage of raw user data, the fragility of current recommenders under inference attacks offers malicious attackers a backdoor to estimate users' private attributes via their behavioral footprints and the recommendation results. However, little attention has been paid to developing recommender systems that can defend such attribute inference attacks, and existing works achieve attack resistance by either sacrificing considerable recommendation accuracy or only covering specific attack models or protected information. In our paper, we propose GERAI, a novel differentially private graph convolutional network to address such limitations. Specifically, in GERAI, we bind the information perturbation mechanism in differential privacy with the recommendation capability of graph convolutional networks. Furthermore, based on local differential privacy and functional mechanism, we innovatively devise a dual-stage encryption paradigm to simultaneously enforce privacy guarantee on users' sensitive features and the model optimization process. Extensive experiments show the superiority of GERAI in terms of its resistance to attribute inference attacks and recommendation effectiveness.

cs.IR↗

arXiv2021

Knowledge Graph Completion with Text-aided Regularization

Knowledge Graph Completion is a task of expanding the knowledge graph/base through estimating possible entities, or proper nouns, that can be connected using a set of predefined relations, or verb/predicates describing interconnections of two things. Generally, we describe this problem as adding new edges to a current network of vertices and edges. Traditional approaches mainly focus on using the existing graphical information that is intrinsic of the graph and train the corresponding embeddings to describe the information; however, we think that the corpus that are related to the entities should also contain information that can positively influence the embeddings to better make predictions. In our project, we try numerous ways of using extracted or raw textual information to help existing KG embedding frameworks reach better prediction results, in the means of adding a similarity function to the regularization part in the loss function. Results have shown that we have made decent improvements over baseline KG embedding methods.

cs.CL↗

arXiv2021

A Sublevel Moment-SOS Hierarchy for Polynomial Optimization

We introduce a sublevel Moment-SOS hierarchy where each SDP relaxation can be viewed as an intermediate (or interpolation) between the d-th and (d+1)-th order SDP relaxations of the Moment-SOS hierarchy (dense or sparse version). With the flexible choice of determining the size (level) and number (depth) of subsets in the SDP relaxation, one is able to obtain different improvements compared to the d-th order relaxation, based on the machine memory capacity. In particular, we provide numerical experiments for d=1 and various types of problems both in combinatorial optimization (Max-Cut, Mixed Integer Programming) and deep learning (robustness certification, Lipschitz constant of neural networks), where the standard Lasserre's relaxation (or its sparse variant) is computationally intractable. In our numerical results, the lower bounds from the sublevel relaxations improve the bound from Shor's relaxation (first order Lasserre's relaxation) and are significantly closer to the optimal value or to the best-known lower/upper bounds.

math.OC↗

arXiv2021

Van Hove Singularity Arising from Mexican-Hat-Shaped Inverted Bands in the Topological Insulator Sn-doped Bi$_{1.1}$Sb$_{0.9}$Te$_{2}$S

The optical properties of Sn-doped Bi$_{1.1}$Sb$_{0.9}$Te$_{2}$S, the most bulk-insulating topological insulator thus far, have been examined at different temperatures over a broad frequency range. No Drude response is detected in the low-frequency range down to 30~cm$^{-1}$, corroborating the excellent bulk-insulating property of this material. Intriguingly, we observe a sharp peak at about 2\,200~cm$^{-1}$ in the optical conductivity at 5~K. Further quantitative analyses of the line shape and temperature dependence of this sharp peak, in combination with first-principles calculations, suggest that it corresponds to a van Hove singularity arising from Mexican-hat-shaped inverted bands. Such a van Hove singularity is a pivotal ingredient of various strongly correlated phases.

cond-mat.str-el↗

arXiv2021

FENet: A Frequency Extraction Network for Obstructive Sleep Apnea Detection

Obstructive Sleep Apnea (OSA) is a highly prevalent but inconspicuous disease that seriously jeopardizes the health of human beings. Polysomnography (PSG), the gold standard of detecting OSA, requires multiple specialized sensors for signal collection, hence patients have to physically visit hospitals and bear the costly treatment for a single detection. Recently, many single-sensor alternatives have been proposed to improve the cost efficiency and convenience. Among these methods, solutions based on RR-interval (i.e., the interval between two consecutive pulses) signals reach a satisfactory balance among comfort, portability and detection accuracy. In this paper, we advance RR-interval based OSA detection by considering its real-world practicality from energy perspectives. As photoplethysmogram (PPG) pulse sensors are commonly equipped on smart wrist-worn wearable devices (e.g., smart watches and wristbands), the energy efficiency of the detection model is crucial to fully support an overnight observation on patients. This creates challenges as the PPG sensors are unable to keep collecting continuous signals due to the limited battery capacity on smart wrist-worn devices. Therefore, we propose a novel Frequency Extraction Network (FENet), which can extract features from different frequency bands of the input RR-interval signals and generate continuous detection results with downsampled, discontinuous RR-interval signals. With the help of the one-to-multiple structure, FENet requires only one-third of the operation time of the PPG sensor, thus sharply cutting down the energy consumption and enabling overnight diagnosis. Experimental results on real OSA datasets reveal the state-of-the-art performance of FENet.

eess.SP↗

arXiv2021

Passenger Mobility Prediction via Representation Learning for Dynamic Directed and Weighted Graph

In recent years, ride-hailing services have been increasingly prevalent as they provide huge convenience for passengers. As a fundamental problem, the timely prediction of passenger demands in different regions is vital for effective traffic flow control and route planning. As both spatial and temporal patterns are indispensable passenger demand prediction, relevant research has evolved from pure time series to graph-structured data for modeling historical passenger demand data, where a snapshot graph is constructed for each time slot by connecting region nodes via different relational edges (e.g., origin-destination relationship, geographical distance, etc.). Consequently, the spatiotemporal passenger demand records naturally carry dynamic patterns in the constructed graphs, where the edges also encode important information about the directions and volume (i.e., weights) of passenger demands between two connected regions. However, existing graph-based solutions fail to simultaneously consider those three crucial aspects of dynamic, directed, and weighted (DDW) graphs, leading to limited expressiveness when learning graph representations for passenger demand prediction. Therefore, we propose a novel spatiotemporal graph attention network, namely Gallat (Graph prediction with all attention) as a solution. In Gallat, by comprehensively incorporating those three intrinsic properties of DDW graphs, we build three attention layers to fully capture the spatiotemporal dependencies among different regions across all historical time slots. Moreover, the model employs a subtask to conduct pretraining so that it can obtain accurate results more quickly. We evaluate the proposed model on real-world datasets, and our experimental results demonstrate that Gallat outperforms the state-of-the-art approaches.

cs.LG↗

arXiv2020

Semialgebraic Optimization for Lipschitz Constants of ReLU Networks

The Lipschitz constant of a network plays an important role in many applications of deep learning, such as robustness certification and Wasserstein Generative Adversarial Network. We introduce a semidefinite programming hierarchy to estimate the global and local Lipschitz constant of a multiple layer deep neural network. The novelty is to combine a polynomial lifting for ReLU functions derivatives with a weak generalization of Putinar's positivity certificate. This idea could also apply to other, nearly sparse, polynomial optimization problems in machine learning. We empirically demonstrate that our method provides a trade-off with respect to state of the art linear programming approach, and in some cases we obtain better bounds in less time.

math.OC↗

arXiv2020

Compiling ONNX Neural Network Models Using MLIR

Deep neural network models are becoming increasingly popular and have been used in various tasks such as computer vision, speech recognition, and natural language processing. Machine learning models are commonly trained in a resource-rich environment and then deployed in a distinct environment such as high availability machines or edge devices. To assist the portability of models, the open-source community has proposed the Open Neural Network Exchange (ONNX) standard. In this paper, we present a high-level, preliminary report on our onnx-mlir compiler, which generates code for the inference of deep neural network models described in the ONNX format. Onnx-mlir is an open-source compiler implemented using the Multi-Level Intermediate Representation (MLIR) infrastructure recently integrated in the LLVM project. Onnx-mlir relies on the MLIR concept of dialects to implement its functionality. We propose here two new dialects: (1) an ONNX specific dialect that encodes the ONNX standard semantics, and (2) a loop-based dialect to provide for a common lowering point for all ONNX dialect operations. Each intermediate representation facilitates its own characteristic set of graph-level and loop-based optimizations respectively. We illustrate our approach by following several models through the proposed representations and we include some early optimization work and performance results.

cs.PL↗