arXiv subjects

Zijian Zhou

Publications and source records attributed to Zijian Zhou.

At least 91 records · Page 5Linked to original sources

arXiv2024

OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Models

Panoptic Scene Graph Generation (PSG) aims to segment objects and recognize their relations, enabling the structured understanding of an image. Previous methods focus on predicting predefined object and relation categories, hence limiting their applications in the open world scenarios. With the rapid development of large multimodal models (LMMs), significant progress has been made in open-set object detection and segmentation, yet open-set relation prediction in PSG remains unexplored. In this paper, we focus on the task of open-set relation prediction integrated with a pretrained open-set panoptic segmentation model to achieve true open-set panoptic scene graph generation (OpenPSG). Our OpenPSG leverages LMMs to achieve open-set relation prediction in an autoregressive manner. We introduce a relation query transformer to efficiently extract visual features of object pairs and estimate the existence of relations between them. The latter can enhance the prediction efficiency by filtering irrelevant pairs. Finally, we design the generation and judgement instructions to perform open-set relation prediction in PSG autoregressively. To our knowledge, we are the first to propose the open-set PSG task. Extensive experiments demonstrate that our method achieves state-of-the-art performance in open-set relation prediction and panoptic scene graph generation. Code is available at \url{https://github.com/franciszzj/OpenPSG}.

cs.CV↗

arXiv2024

Data-Centric AI in the Age of Large Language Models

This position paper proposes a data-centric viewpoint of AI research, focusing on large language models (LLMs). We start by making the key observation that data is instrumental in the developmental (e.g., pretraining and fine-tuning) and inferential stages (e.g., in-context learning) of LLMs, and yet it receives disproportionally low attention from the research community. We identify four specific scenarios centered around data, covering data-centric benchmarks and data curation, data attribution, knowledge transfer, and inference contextualization. In each scenario, we underscore the importance of data, highlight promising research directions, and articulate the potential impacts on the research community and, where applicable, the society as a whole. For instance, we advocate for a suite of data-centric benchmarks tailored to the scale and complexity of data for LLMs. These benchmarks can be used to develop new data curation methods and document research efforts and results, which can help promote openness and transparency in AI and LLM research.

cs.LG↗

arXiv2024

VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generation

Panoptic Scene Graph Generation (PSG) aims at achieving a comprehensive image understanding by simultaneously segmenting objects and predicting relations among objects. However, the long-tail problem among relations leads to unsatisfactory results in real-world applications. Prior methods predominantly rely on vision information or utilize limited language information, such as object or relation names, thereby overlooking the utility of language information. Leveraging the recent progress in Large Language Models (LLMs), we propose to use language information to assist relation prediction, particularly for rare relations. To this end, we propose the Vision-Language Prompting (VLPrompt) model, which acquires vision information from images and language information from LLMs. Then, through a prompter network based on attention mechanism, it achieves precise relation prediction. Our extensive experiments show that VLPrompt significantly outperforms previous state-of-the-art methods on the PSG dataset, proving the effectiveness of incorporating language information and alleviating the long-tail problem of relations. Code is available at \url{https://github.com/franciszzj/TP-SIS}.

cs.CV↗

arXiv2024

The Endomorphism Rings of Supersingular Elliptic Curves over $\mathbb{F}_p$ and the Binary Quadratic Forms

It is well known that there is a one-to-one correspondence between supersingular $j$-invariants up to the action of $\text{Gal}(\mathbb{F}_{p^2}/\mathbb{F}_p)$ and type classes of maximal orders in $B_{p,\infty}$ by Deuring's theorem. Interestingly, we establish a one-to-one correspondence between $\mathbb{F}_p$-isomorphism classes of supersingular elliptic curves and primitive reduced binary quadratic forms with discriminant $-p$ or $-16p$. Due to this correspondence and the fact that $\mathbb{F}_p$-isogenies between elliptic curves could be represented by quadratic forms, we show that operations of these isogenies on supersingular elliptic curves over $\mathbb{F}_p$ are compatible with the composition of quadratic forms. Based on these results, we could reduce the security of CSIDH cryptosystem to computing this correspondence explicitly.

math.NT↗

arXiv2024

Haina Storage: A Decentralized Secure Storage Framework Based on Improved Blockchain Structure

Although the decentralized storage technology based on the blockchain can effectively realize secure data storage on cloud services. However, there are still some problems in the existing schemes, such as low storage capacity and low efficiency. To address related issues, we propose a novel decentralized storage framework, which mainly includes four aspects: (1) we proposed a Bi-direction Circular Linked Chain Structure (BCLCS), which improves data's storage capacity and applicability in decentralized storage. (2) A Proof of Resources (PoR) decision model is proposed. By introducing the network environment as an essential evaluation parameter of storage right decision, the energy and time consumption of decision-making are reduced, and the fairness of decision-making is improved. (3) A chain structure dynamic locking mechanism (CSDLM) is designed to realize anti-traverse and access control. (4) A Bi-directional data Access Mechanism (BDAM) is proposed, which improves the efficiency of data access and acquisition in decentralized storage mode. The experimental results show that the framework has significantly improved the shortcomings of the current decentralized storage.

cs.CR↗

arXiv2024

Neighborhood of vertices in the isogeny graph of principally polarized superspecial abelian surfaces

For two supersingular elliptic curves $E$ and $E'$ defined over $\mathbb{F}_{p^2}$, let $[E \times E']$ be the superspecial abelian surface with the principal polarization $\{0\} \times E' + E \times \{0\}$. We determine local structure of the vertices $[E \times E']$ in the $(\ell, \ell)$-isogeny graph of principally polarized superspecial abelian surfaces where either $E$ or $E'$ is defined over $\mathbb{F}_p$. We also present a simple new proof of the main theorem in \cite{LOX20}.

math.NT↗

arXiv2024

Auxiliary Factor Method to Remove ISI of Nyquist Filters

As has been known, the Nyquist first condition promises no intersymbol interference (ISI) as derived in the frequency domain. However, the practical implementation using the FIR filter truncates the Fourier transform by its window and prevents the mathematical calculation from reaching the ideal solution at zero-ISI. For obtaining better results, an increase in the window's length is required in general. To address this problem, a new approach is presented by using auxiliary factors (AFs) to compensate shortcomings of the truncated Fourier transform and remove the ISI completely, regardless of the window's length. In addition, the performance in the presence of the timing jitter is also improved significantly. The closed-form solution of the AFs is derived and the effectiveness is confirmed by the simulation results. Finally, the problems of the transmission delay and additional calculation complexity are analysed.

eess.SP↗

arXiv2023

Nano-Imaging of Landau-Phonon Polaritons in Dirac Heterostructures

Polaritons are light-matter quasiparticles that govern the optical response of quantum materials and enable their nanophotonic applications. We have studied a new type of polaritons arising in magnetized graphene encapsulated in hexagonal boron nitride (hBN). These polaritons stem from hybridization of Dirac magnetoexciton modes of graphene with waveguide phonon modes of hBN crystals. We refer to these quasiparticles as the Landau-phonon polaritons (LPPs). Using infrared magneto nanoscopy, we imaged LPPs and controlled their real-space propagation by varying the magnetic field. These LLPs have large in-plane momenta and are not bound by the conventional optical selection rules, granting us access to the "forbidden" inter-Landau level transitions (ILTs). We observed avoided crossings in the LPP dispersion - a hallmark of the strong coupling regime - occurring when the magnetoexciton and hBN phonon frequencies matched. Our LPP-based nanoscopy also enabled us to resolve two fundamental many-body effects: the graphene Fermi velocity renormalization and ILT-dependent magnetoexciton binding energies. These results indicate that magnetic-field-tuned Dirac heterostructures are promising platforms for precise nanoscale control and sensing of light-matter interaction.

physics.optics↗

arXiv2023

A Contradiction to the Law of Energy Conservation by Waves Interference in Symmetric/Asymmetric mode

It can be agreed that the linear superposition and energy conservation are two independent physics laws in general. The former allows the energy to be re-distributed over space and the latter restricts the energy in the total amount. However, Levine shows the contradiction of the two laws mentioned above by creating a cleaver model that demonstrates the energy "doubling"- and "missing" phenomenon with the constrictive- and destructive interference at every point of whole space, respectively. While, he presented a wrong explanation by using one of the radiating sources to compare with an isolated source by the compensation of the impedance, where the mistake is simply analyzed in this paper. By setting up a spatial symmetric- and asymmetric-mode, we work upon Poynting theorem from the sources to the waves with the considerations of the superposition. The theoretical results reveal the invalidity of the energy conservation. Moreover, the experiments performed in the microwave anechoic chamber confirm the theoretical conclusion.

physics.gen-ph↗

arXiv2023

DeepMetricEye: Metric Depth Estimation in Periocular VR Imagery

Despite the enhanced realism and immersion provided by VR headsets, users frequently encounter adverse effects such as digital eye strain (DES), dry eye, and potential long-term visual impairment due to excessive eye stimulation from VR displays and pressure from the mask. Recent VR headsets are increasingly equipped with eye-oriented monocular cameras to segment ocular feature maps. Yet, to compute the incident light stimulus and observe periocular condition alterations, it is imperative to transform these relative measurements into metric dimensions. To bridge this gap, we propose a lightweight framework derived from the U-Net 3+ deep learning backbone that we re-optimised, to estimate measurable periocular depth maps. Compatible with any VR headset equipped with an eye-oriented monocular camera, our method reconstructs three-dimensional periocular regions, providing a metric basis for related light stimulus calculation protocols and medical guidelines. Navigating the complexities of data collection, we introduce a Dynamic Periocular Data Generation (DPDG) environment based on UE MetaHuman, which synthesises thousands of training images from a small quantity of human facial scan data. Evaluated on a sample of 36 participants, our method exhibited notable efficacy in the periocular global precision evaluation experiment, and the pupil diameter measurement.

eess.IV↗

arXiv2023

Text Promptable Surgical Instrument Segmentation with Vision-Language Models

In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. We redefine the task as text promptable, thereby enabling a more nuanced comprehension of surgical instruments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders as our model backbone and design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. Our model leverages multiple text prompts for each surgical instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision. Extensive experiments on several surgical instrument segmentation datasets demonstrate our model's superior performance and promising generalization capability. To our knowledge, this is the first implementation of a promptable approach to surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery. Code is available at https://github.com/franciszzj/TP-SIS.

cs.CV↗

arXiv2023

Data-driven localized waves and parameter discovery in the massive Thirring model via extended physics-informed neural networks with interface zones

In this paper, we study data-driven localized wave solutions and parameter discovery in the massive Thirring (MT) model via the deep learning in the framework of physics-informed neural networks (PINNs) algorithm. Abundant data-driven solutions including soliton of bright/dark type, breather and rogue wave are simulated accurately and analyzed contrastively with relative and absolute errors. For higher-order localized wave solutions, we employ the extended PINNs (XPINNs) with domain decomposition to capture the complete pictures of dynamic behaviors such as soliton collisions, breather oscillations and rogue-wave superposition. In particular, we modify the interface line in domain decomposition of XPINNs into a small interface zone and introduce the pseudo initial, residual and gradient conditions as interface conditions linked adjacently with individual neural networks. Then this modified approach is applied successfully to various solutions ranging from bright-bright soliton, dark-dark soliton, dark-antidark soliton, general breather, Kuznetsov-Ma breather and second-order rogue wave. Experimental results show that this improved version of XPINNs reduce the complexity of computation with faster convergence rate and keep the quality of learned solutions with smoother stitching performance as well. For the inverse problems, the unknown coefficient parameters of linear and nonlinear terms in the MT model are identified accurately with and without noise by using the classical PINNs algorithm.

nlin.PS↗

arXiv2023

HiLo: Exploiting High Low Frequency Relations for Unbiased Panoptic Scene Graph Generation

Panoptic Scene Graph generation (PSG) is a recently proposed task in image scene understanding that aims to segment the image and extract triplets of subjects, objects and their relations to build a scene graph. This task is particularly challenging for two reasons. First, it suffers from a long-tail problem in its relation categories, making naive biased methods more inclined to high-frequency relations. Existing unbiased methods tackle the long-tail problem by data/loss rebalancing to favor low-frequency relations. Second, a subject-object pair can have two or more semantically overlapping relations. While existing methods favor one over the other, our proposed HiLo framework lets different network branches specialize on low and high frequency relations, enforce their consistency and fuse the results. To the best of our knowledge we are the first to propose an explicitly unbiased PSG method. In extensive experiments we show that our HiLo framework achieves state-of-the-art results on the PSG task. We also apply our method to the Scene Graph Generation task that predicts boxes instead of masks and see improvements over all baseline methods. Code is available at https://github.com/franciszzj/HiLo.

cs.CV↗

arXiv2022

Probably Approximate Shapley Fairness with Applications in Machine Learning

The Shapley value (SV) is adopted in various scenarios in machine learning (ML), including data valuation, agent valuation, and feature attribution, as it satisfies their fairness requirements. However, as exact SVs are infeasible to compute in practice, SV estimates are approximated instead. This approximation step raises an important question: do the SV estimates preserve the fairness guarantees of exact SVs? We observe that the fairness guarantees of exact SVs are too restrictive for SV estimates. Thus, we generalise Shapley fairness to probably approximate Shapley fairness and propose fidelity score, a metric to measure the variation of SV estimates, that determines how probable the fairness guarantees hold. Our last theoretical contribution is a novel greedy active estimation (GAE) algorithm that will maximise the lowest fidelity score and achieve a better fairness guarantee than the de facto Monte-Carlo estimation. We empirically verify GAE outperforms several existing methods in guaranteeing fairness while remaining competitive in estimation accuracy in various ML scenarios using real-world datasets.

cs.LG↗

arXiv2022

Data-driven discoveries of Bäcklund transforms and soliton evolution equations via deep neural network learning schemes

We introduce a deep neural network learning scheme to learn the Bäcklund transforms (BTs) of soliton evolution equations and an enhanced deep learning scheme for data-driven soliton equation discovery based on the known BTs, respectively. The first scheme takes advantage of some solution (or soliton equation) information to study the data-driven BT of sine-Gordon equation, and complex and real Miura transforms between the defocusing (focusing) mKdV equation and KdV equation, as well as the data-driven mKdV equation discovery via the Miura transforms. The second deep learning scheme uses the explicit/implicit BTs generating the higher-order solitons to train the data-driven discovery of mKdV and sine-Gordon equations, in which the high-order solution informations are more powerful for the enhanced leaning soliton equations with higher accurates.

cs.LG↗

arXiv2021

Deep neural networks for solving forward and inverse problems of (2+1)-dimensional nonlinear wave equations with rational solitons

In this paper, we investigate the forward problems on the data-driven rational solitons for the (2+1)-dimensional KP-I equation and spin-nonlinear Schrödinger (spin-NLS) equation via the deep neural networks leaning. Moreover, the inverse problems of the (2+1)-dimensional KP-I equation and spin-NLS equation are studied via deep learning. The main idea of the data-driven forward and inverse problems is to use the deep neural networks with the activation function to approximate the solutions of the considered (2+1)-dimensional nonlinear wave equations by optimizing the chosen loss functions related to the considered nonlinear wave equations.

nlin.PS↗

arXiv2021

Deep learning neural networks for the third-order nonlinear Schrodinger equation: Solitons, breathers, and rogue waves

The third-order nonlinear Schrodinger equation (alias the Hirota equation) is investigated via deep leaning neural networks, which describes the strongly dispersive ion-acoustic wave in plasma and the wave propagation of ultrashort light pulses in optical fibers, as well as broader-banded waves on deep water. In this paper, we use the physics-informed neural networks (PINNs) deep learning method to explore the data-driven solutions (e.g., soliton, breather, and rogue waves) of the Hirota equation when the two types of the unperturbated and unperturbated (a 2% noise) training data are considered. Moreover, we use the PINNs deep learning to study the data-driven discovery of parameters appearing in the Hirota equation with the aid of solitons.

nlin.PS↗

arXiv2021

Randomized Ensembled Double Q-Learning: Learning Fast Without a Model

Using a high Update-To-Data (UTD) ratio, model-based methods have recently achieved much higher sample efficiency than previous model-free methods for continuous-action DRL benchmarks. In this paper, we introduce a simple model-free algorithm, Randomized Ensembled Double Q-Learning (REDQ), and show that its performance is just as good as, if not better than, a state-of-the-art model-based algorithm for the MuJoCo benchmark. Moreover, REDQ can achieve this performance using fewer parameters than the model-based method, and with less wall-clock run time. REDQ has three carefully integrated ingredients which allow it to achieve its high performance: (i) a UTD ratio >> 1; (ii) an ensemble of Q functions; (iii) in-target minimization across a random subset of Q functions from the ensemble. Through carefully designed experiments, we provide a detailed analysis of REDQ and related model-free algorithms. To our knowledge, REDQ is the first successful model-free DRL algorithm for continuous-action spaces using a UTD ratio >> 1.

cs.LG↗