arXiv subjects

Ming Zhang

Publications and source records attributed to Ming Zhang.

At least 217 records · Page 12Linked to original sources

arXiv2024

MouSi: Poly-Visual-Expert Vision-Language Models

Current large vision-language models (VLMs) often encounter challenges such as insufficient capabilities of a single visual component and excessively long visual tokens. These issues can limit the model's effectiveness in accurately interpreting complex visual information and over-lengthy contextual information. Addressing these challenges is crucial for enhancing the performance and applicability of VLMs. This paper proposes the use of ensemble experts technique to synergizes the capabilities of individual visual encoders, including those skilled in image-text matching, OCR, image segmentation, etc. This technique introduces a fusion network to unify the processing of outputs from different visual experts, while bridging the gap between image encoders and pre-trained LLMs. In addition, we explore different positional encoding schemes to alleviate the waste of positional encoding caused by lengthy image feature sequences, effectively addressing the issue of position overflow and length limitations. For instance, in our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient and manageable 64 or even down to 1. Experimental results demonstrate that VLMs with multiple experts exhibit consistently superior performance over isolated visual encoders and mark a significant performance boost as more experts are integrated. We have open-sourced the training code used in this report. All of these resources can be found on our project website.

cs.CV↗

arXiv2024

PolyCF: Towards the Optimal Spectral Graph Filters for Collaborative Filtering

Collaborative Filtering (CF) is a pivotal research area in recommender systems that capitalizes on collaborative similarities between users and items to provide personalized recommendations. With the remarkable achievements of node embedding-based Graph Neural Networks (GNNs), we explore the upper bounds of expressiveness inherent to embedding-based methodologies and tackle the challenges by reframing the CF task as a graph signal processing problem. To this end, we propose PolyCF, a flexible graph signal filter that leverages polynomial graph filters to process interaction signals. PolyCF exhibits the capability to capture spectral features across multiple eigenspaces through a series of Generalized Gram filters and is able to approximate the optimal polynomial response function for recovering missing interactions. A graph optimization objective and a pair-wise ranking objective are jointly used to optimize the parameters of the convolution kernel. Experiments on three widely adopted datasets demonstrate the superiority of PolyCF over current state-of-the-art CF methods. Moreover, comprehensive studies empirically validate each component's efficacy in the proposed PolyCF.

cs.IR↗

arXiv2024

From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities

Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents. However, there is still a wide gap between the performance of recent MLLM-based applications and the expectation of the broad public, even though the most powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper strives to enhance understanding of the gap through the lens of a qualitative study on the generalizability, trustworthiness, and causal reasoning capabilities of recent proprietary and open-source MLLMs across four modalities: ie, text, code, image, and video, ultimately aiming to improve the transparency of MLLMs. We believe these properties are several representative factors that define the reliability of MLLMs, in supporting various downstream applications. To be specific, we evaluate the closed-source GPT-4 and Gemini and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed cases, where the qualitative results are then summarized into 12 scores (ie, 4 modalities times 3 properties). In total, we uncover 14 empirical findings that are useful to understand the capabilities and limitations of both proprietary and open-source MLLMs, towards more reliable downstream multi-modal applications.

cs.CV↗

arXiv2024

GPS: Graph Contrastive Learning via Multi-scale Augmented Views from Adversarial Pooling

Self-supervised graph representation learning has recently shown considerable promise in a range of fields, including bioinformatics and social networks. A large number of graph contrastive learning approaches have shown promising performance for representation learning on graphs, which train models by maximizing agreement between original graphs and their augmented views (i.e., positive views). Unfortunately, these methods usually involve pre-defined augmentation strategies based on the knowledge of human experts. Moreover, these strategies may fail to generate challenging positive views to provide sufficient supervision signals. In this paper, we present a novel approach named Graph Pooling ContraSt (GPS) to address these issues. Motivated by the fact that graph pooling can adaptively coarsen the graph with the removal of redundancy, we rethink graph pooling and leverage it to automatically generate multi-scale positive views with varying emphasis on providing challenging positives and preserving semantics, i.e., strongly-augmented view and weakly-augmented view. Then, we incorporate both views into a joint contrastive learning framework with similarity learning and consistency learning, where our pooling module is adversarially trained with respect to the encoder for adversarial robustness. Experiments on twelve datasets on both graph classification and transfer learning tasks verify the superiority of the proposed method over its counterparts.

cs.LG↗

arXiv2024

A Survey of Reasoning with Foundation Models

Reasoning, a crucial ability for complex problem-solving, plays a pivotal role in various real-world settings such as negotiation, medical diagnosis, and criminal investigation. It serves as a fundamental methodology in the field of Artificial General Intelligence (AGI). With the ongoing development of foundation models, e.g., Large Language Models (LLMs), there is a growing interest in exploring their abilities in reasoning tasks. In this paper, we introduce seminal foundation models proposed or adaptable for reasoning, highlighting the latest advancements in various reasoning tasks, methods, and benchmarks. We then delve into the potential future directions behind the emergence of reasoning abilities within foundation models. We also discuss the relevance of multimodal learning, autonomous agents, and super alignment in the context of reasoning. By discussing these future research directions, we hope to inspire researchers in their exploration of this field, stimulate further advancements in reasoning with foundation models, and contribute to the development of AGI.

cs.AI↗

arXiv2024

Learning Graph ODE for Continuous-Time Sequential Recommendation

Sequential recommendation aims at understanding user preference by capturing successive behavior correlations, which are usually represented as the item purchasing sequences based on their past interactions. Existing efforts generally predict the next item via modeling the sequential patterns. Despite effectiveness, there exist two natural deficiencies: (i) user preference is dynamic in nature, and the evolution of collaborative signals is often ignored; and (ii) the observed interactions are often irregularly-sampled, while existing methods model item transitions assuming uniform intervals. Thus, how to effectively model and predict the underlying dynamics for user preference becomes a critical research problem. To tackle the above challenges, in this paper, we focus on continuous-time sequential recommendation and propose a principled graph ordinary differential equation framework named GDERec. Technically, GDERec is characterized by an autoregressive graph ordinary differential equation consisting of two components, which are parameterized by two tailored graph neural networks (GNNs) respectively to capture user preference from the perspective of hybrid dynamical systems. The two customized GNNs are trained alternately in an autoregressive manner to track the evolution of the underlying system from irregular observations, and thus learn effective representations of users and items beneficial to the sequential recommendation. Extensive experiments on five benchmark datasets demonstrate the superiority of our model over various state-of-the-art recommendation methods.

cs.IR↗

arXiv2024

Reframing Tax Law Entailment as Analogical Reasoning

Statutory reasoning refers to the application of legislative provisions to a series of case facts described in natural language. We re-frame statutory reasoning as an analogy task, where each instance of the analogy task involves a combination of two instances of statutory reasoning. This increases the dataset size by two orders of magnitude, and introduces an element of interpretability. We show that this task is roughly as difficult to Natural Language Processing models as the original task. Finally, we come back to statutory reasoning, solving it with a combination of a retrieval mechanism and analogy models, and showing some progress on prior comparable work.

cs.CL↗

arXiv2024

RJUA-QA: A Comprehensive QA Dataset for Urology

We introduce RJUA-QA, a novel medical dataset for question answering (QA) and reasoning with clinical evidence, contributing to bridge the gap between general large language models (LLMs) and medical-specific LLM applications. RJUA-QA is derived from realistic clinical scenarios and aims to facilitate LLMs in generating reliable diagnostic and advice. The dataset contains 2,132 curated Question-Context-Answer pairs, corresponding about 25,000 diagnostic records and clinical cases. The dataset covers 67 common urological disease categories, where the disease coverage exceeds 97.6\% of the population seeking medical services in urology. Each data instance in RJUA-QA comprises: (1) a question mirroring real patient to inquiry about clinical symptoms and medical conditions, (2) a context including comprehensive expert knowledge, serving as a reference for medical examination and diagnosis, (3) a doctor response offering the diagnostic conclusion and suggested examination guidance, (4) a diagnosed clinical disease as the recommended diagnostic outcome, and (5) clinical advice providing recommendations for medical examination. RJUA-QA is the first medical QA dataset for clinical reasoning over the patient inquiries, where expert-level knowledge and experience are required for yielding diagnostic conclusions and medical examination advice. A comprehensive evaluation is conducted to evaluate the performance of both medical-specific and general LLMs on the RJUA-QA dataset. Our data is are publicly available at \url{https://github.com/alipay/RJU_Ant_QA}.

cs.CL↗

arXiv2024

Cross-Tokamak Deployment Study of Plasma Disruption Predictors Based on Convolutional Autoencoder

In the initial stages of operation for future tokamak, facing limited data availability, deploying data-driven disruption predictors requires optimal performance with minimal use of new device data. This paper studies the issue of data utilization in data-driven disruption predictor during cross tokamak deployment. Current predictors primarily employ supervised learning methods and require a large number of disruption and non-disruption shots for training. However, the scarcity and high cost of obtaining disruption shots for future tokamaks result in imbalanced training datasets, reducing the performance of supervised learning predictors. To solve this problem, we propose the Enhanced Convolutional Autoencoder Anomaly Detection (E-CAAD) predictor. E-CAAD can be only trained by normal samples from non-disruption shots and can also be trained by disruption precursor samples when disruption shots occur. This model not only overcomes the sample imbalance in supervised learning predictors, but also overcomes the inefficient dataset utilization faced by traditional anomaly detection predictors that cannot use disruption precursor samples for training, making it more suitable for the unpredictable datasets of future tokamaks. Compared to traditional anomaly detection predictor, the E-CAAD predictor performs better in disruption prediction and is deployed faster on new devices. Additionally, we explore strategies to accelerate deployment of E-CAAD predictor on the new device by using data from existing devices. Two deployment strategies are presented: mixing data from existing devices and fine-tuning the predictor trained on existing devices. Our comparisons indicate that the data from existing device can accelerate the deployment of predictor on new device. Notably, the fine-tuning strategy yields the fastest deployment on new device among the designed strategies.

physics.plasm-ph↗

arXiv2024

TBDD: A New Trust-based, DRL-driven Framework for Blockchain Sharding in IoT

Integrating sharded blockchain with IoT presents a solution for trust issues and optimized data flow. Sharding boosts blockchain scalability by dividing its nodes into parallel shards, yet it's vulnerable to the $1\%$ attacks where dishonest nodes target a shard to corrupt the entire blockchain. Balancing security with scalability is pivotal for such systems. Deep Reinforcement Learning (DRL) adeptly handles dynamic, complex systems and multi-dimensional optimization. This paper introduces a Trust-based and DRL-driven (\textsc{TbDd}) framework, crafted to counter shard collusion risks and dynamically adjust node allocation, enhancing throughput while maintaining network security. With a comprehensive trust evaluation mechanism, \textsc{TbDd} discerns node types and performs targeted resharding against potential threats. The model maximizes tolerance for dishonest nodes, optimizes node movement frequency, ensures even node distribution in shards, and balances sharding risks. Rigorous evaluations prove \textsc{TbDd}'s superiority over conventional random-, community-, and trust-based sharding methods in shard risk equilibrium and reducing cross-shard transactions.

cs.CR↗

arXiv2023

Ultrafast X-ray Diffraction Probe of Coherent Spin-state Dynamics in Molecules

We propose an approach to probe coherent spin-state dynamics of molecules using circularly polarized hard x-ray pulses. For the dynamically aligned nitric oxide molecules in a coherent superposition spin-orbit coupled electronic state that can be prepared through stimulated Raman scattering, we demonstrate the capability of ultrafast x-ray diffraction to not only reveal the quantum beating of the coherent spin-state wave packet, but also image the spatial spin density of the molecule. With circularly polarized ultrafast x-ray diffraction signal, we show that the electronic density matrix can be retrieved. The spatio-temporal resolving power of ultrafast x-ray diffraction paves the way for tracking transient spatial wave function in molecular dynamics involving spin degree of freedom.

physics.optics↗

arXiv2023

Overcharging an accelerating Reissner-Nordström-Anti-de Sitter black hole with test field and particle

Accelerating black holes have been widely studied in the context of black hole thermodynamics, holographic gravity theories, and in the description of black holes at the center of galaxies. As a fundamental assumption to ensure spacetime causality, we investigated the weak cosmic censorship conjecture (WCCC) in the accelerating Reissner-Nordström-Anti-de Sitter (RN-AdS) spacetime through the scattering of a charged field and the absorption of a charged particle. For the scattering of a charged scalar field, both near-extremal and extremal accelerating RN-AdS black holes cannot be overcharged, thereby upholding the validity of the WCCC. In the case of the absorption of a test charged particle, the results demonstrate that the event horizon of the extremal accelerating RN-AdS black hole cannot be destroyed, while the event horizon of the near-extremal black hole can be overcharged if the test particle satisfies certain conditions. The above results suggest that, in the case of test particles, second-order effects like self-force and self-energy should be further considered.

gr-qc↗

arXiv2023

Hydrogen Doping Induced $p_x\pm ip_y$ Triplet Superconductivity in Quasi-One-Dimensional K$_2$Cr$_3$As$_3$

Quasi-one-dimensional (Q1D) Cr-based pnictide K$_2$Cr$_3$As$_3$ has aroused great research interest due to its possible triplet superconducting pairing symmetry. Recent experiments have shown that incorporating hydrogen atoms into K$_2$Cr$_3$As$_3$ would significantly change its electronic and magnetic properties. Hence, it's necessary to investigate the impact of hydrogen doping in superconducting pairing symmetry of this material. Employing the hydrogen as an non-trivial electron-doping, our calculates show that, different from the $p_z$-wave obtained without hydrogen, the system exhibits $p_x\pm ip_y$ pairing superconductivity under specific hydrogen doping. Specifically, we adopt the random-phase-approximation approach based on a six-band tight-binding model equipped with multi-orbital Hubbard interactions to study the hydrogen-doping dependence of the pairing symmetry and superconducting $T_c$. Under the rigid-band approximation, our pairing phase diagram shows the spin-triplet pairing states is dominated through out the hydrogen-doping regime $x\in (0,0.7)$. Particularly, the $T_c\sim x$ curve shows a peak at the 3D-quasi-1D Lifshitz transition point, and the pairing symmetry around this doping level is $p_x\pm ip_y$. The physical origin of this pairing symmetry is that the density of states is mainly concentrated at $k_x(k_y)$ with large momentum. Due to the three-dimensional character of the real material, this $p_x\pm ip_y$-wave superconducting state possesses point gap nodes. We further provide experiment prediction to identify this triplet $p_x\pm ip_y$-wave superconductivity.

cond-mat.supr-con↗

arXiv2023

LLMEval: A Preliminary Study on How to Evaluate Large Language Models

Recently, the evaluation of Large Language Models has emerged as a popular area of research. The three crucial questions for LLM evaluation are ``what, where, and how to evaluate''. However, the existing research mainly focuses on the first two questions, which are basically what tasks to give the LLM during testing and what kind of knowledge it should deal with. As for the third question, which is about what standards to use, the types of evaluators, how to score, and how to rank, there hasn't been much discussion. In this paper, we analyze evaluation methods by comparing various criteria with both manual and automatic evaluation, utilizing onsite, crowd-sourcing, public annotators and GPT-4, with different scoring methods and ranking systems. We propose a new dataset, LLMEval and conduct evaluations on 20 LLMs. A total of 2,186 individuals participated, leading to the generation of 243,337 manual annotations and 57,511 automatic evaluation results. We perform comparisons and analyses of different settings and conduct 10 conclusions that can provide some insights for evaluating LLM in the future. The dataset and the results are publicly available at https://github.com/llmeval .

cs.AI↗

arXiv2023

FIMO: A Challenge Formal Dataset for Automated Theorem Proving

We present FIMO, an innovative dataset comprising formal mathematical problem statements sourced from the International Mathematical Olympiad (IMO) Shortlisted Problems. Designed to facilitate advanced automated theorem proving at the IMO level, FIMO is currently tailored for the Lean formal language. It comprises 149 formal problem statements, accompanied by both informal problem descriptions and their corresponding LaTeX-based informal proofs. Through initial experiments involving GPT-4, our findings underscore the existing limitations in current methodologies, indicating a substantial journey ahead before achieving satisfactory IMO-level automated theorem proving outcomes.

cs.AI↗

arXiv2023

A*Net: A Scalable Path-based Reasoning Approach for Knowledge Graphs

Reasoning on large-scale knowledge graphs has been long dominated by embedding methods. While path-based methods possess the inductive capacity that embeddings lack, their scalability is limited by the exponential number of paths. Here we present A*Net, a scalable path-based method for knowledge graph reasoning. Inspired by the A* algorithm for shortest path problems, our A*Net learns a priority function to select important nodes and edges at each iteration, to reduce time and memory footprint for both training and inference. The ratio of selected nodes and edges can be specified to trade off between performance and efficiency. Experiments on both transductive and inductive knowledge graph reasoning benchmarks show that A*Net achieves competitive performance with existing state-of-the-art path-based methods, while merely visiting 10% nodes and 10% edges at each iteration. On a million-scale dataset ogbl-wikikg2, A*Net not only achieves a new state-of-the-art result, but also converges faster than embedding methods. A*Net is the first path-based method for knowledge graph reasoning at such scale.

cs.AI↗

arXiv2023

Analytical approximate solutions of AdS black holes in Einstein-Weyl-scalar gravity

We consider Einstein-Weyl gravity with a minimally coupled scalar field in four dimensional spacetime. By using the Minimal Geometric Deformation (MGD) approach, we split the highly nonlinear coupled field equations into two subsystems that describing the background geometry and scalar field source, respectively. Regarding the Schwarzschild-AdS metric as a background geometry, we derive analytical approximate solutions of scalar field and deformation metric functions with Homotopy Analysis Method (HAM), providing their analytical approximations to fourth order. Moreover, we discuss the accuracy of the analytical approximations, showing they are sufficiently accurate throughout the exterior spacetime.

gr-qc↗

arXiv2023

A Diffusion model for POI recommendation

Next Point-of-Interest (POI) recommendation is a critical task in location-based services that aim to provide personalized suggestions for the user's next destination. Previous works on POI recommendation have laid focused on modeling the user's spatial preference. However, existing works that leverage spatial information are only based on the aggregation of users' previous visited positions, which discourages the model from recommending POIs in novel areas. This trait of position-based methods will harm the model's performance in many situations. Additionally, incorporating sequential information into the user's spatial preference remains a challenge. In this paper, we propose Diff-POI: a Diffusion-based model that samples the user's spatial preference for the next POI recommendation. Inspired by the wide application of diffusion algorithm in sampling from distributions, Diff-POI encodes the user's visiting sequence and spatial character with two tailor-designed graph encoding modules, followed by a diffusion-based sampling strategy to explore the user's spatial visiting trends. We leverage the diffusion process and its reversed form to sample from the posterior distribution and optimized the corresponding score function. We design a joint training and inference framework to optimize and evaluate the proposed Diff-POI. Extensive experiments on four real-world POI recommendation datasets demonstrate the superiority of our Diff-POI over state-of-the-art baseline methods. Further ablation and parameter studies on Diff-POI reveal the functionality and effectiveness of the proposed diffusion-based sampling strategy for addressing the limitations of existing methods.

cs.IR↗