arXiv subjects

Zixu Zhao

Publications and source records attributed to Zixu Zhao.

36 records · Page 2Linked to original sources

arXiv2022

TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery

Surgical instrument segmentation -- in general a pixel classification task -- is fundamentally crucial for promoting cognitive intelligence in robot-assisted surgery (RAS). However, previous methods are struggling with discriminating instrument types and instances. To address the above issues, we explore a mask classification paradigm that produces per-segment predictions. We propose TraSeTR, a novel Track-to-Segment Transformer that wisely exploits tracking cues to assist surgical instrument segmentation. TraSeTR jointly reasons about the instrument type, location, and identity with instance-level predictions i.e., a set of class-bbox-mask pairs, by decoding query embeddings. Specifically, we introduce the prior query that encoded with previous temporal knowledge, to transfer tracking signals to current instances via identity matching. A contrastive query learning strategy is further applied to reshape the query feature space, which greatly alleviates the tracking difficulty caused by large temporal variations. The effectiveness of our method is demonstrated with state-of-the-art instrument type segmentation results on three public datasets, including two RAS benchmarks from EndoVis Challenges and one cataract surgery dataset CaDIS.

cs.CV↗

arXiv2021

Modelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence Learning

This paper presents a self-supervised method for learning reliable visual correspondence from unlabeled videos. We formulate the correspondence as finding paths in a joint space-time graph, where nodes are grid patches sampled from frames, and are linked by two types of edges: (i) neighbor relations that determine the aggregation strength from intra-frame neighbors in space, and (ii) similarity relations that indicate the transition probability of inter-frame paths across time. Leveraging the cycle-consistency in videos, our contrastive learning objective discriminates dynamic objects from both their neighboring views and temporal views. Compared with prior works, our approach actively explores the neighbor relations of central instances to learn a latent association between center-neighbor pairs (e.g., "hand -- arm") across time, thus improving the instance discrimination. Without fine-tuning, our learned representation outperforms the state-of-the-art self-supervised methods on a variety of visual tasks including video object propagation, part propagation, and pose keypoint tracking. Our self-supervised method also surpasses some fully supervised algorithms designed for the specific tasks.

cs.CV↗

arXiv2021

Estimation precision of parameter associated with Unruh-like effect

We study the quantum Fisher information (QFI) of acceleration, in the open quantum systems, for a two-level atom with the circular motion coupled to a massless scalar field in the Minkowski vacuum without and with a reflecting boundary in the ultra-relativistic limit. As we amplify $a$, the saturation time decreases for $θ\neqπ$, but first increases and then decreases for $θ=π$. Without a boundary, there exists a peak value of QFI with a certain time. The QFI varies with the initial state parameter $θ$, and firstly takes peak value in the ground state of the atom. The variation of QFI with respect to $θ$ gradually fades away with the evolution of time. With a boundary, the detection range of acceleration has been expanded. The QFI firstly takes the maximum in the excited state of the atom. In addition, we study the QFI of temperature for a static atom immersed in a thermal bath without and with a boundary. The relation between the saturation time and $T$ is similar to $a$. Without a boundary, the QFI of temperature is similar to that of acceleration. With a boundary, the QFI firstly takes peak value in the ground state of the atom, which is different from the behavior of acceleration. The results provide references for the detection of Unruh-like effect.

gr-qc↗

arXiv2021

Temporal Memory Relation Network for Workflow Recognition from Surgical Video

Automatic surgical workflow recognition is a key component for developing context-aware computer-assisted systems in the operating theatre. Previous works either jointly modeled the spatial features with short fixed-range temporal information, or separately learned visual and long temporal cues. In this paper, we propose a novel end-to-end temporal memory relation network (TMRNet) for relating long-range and multi-scale temporal patterns to augment the present features. We establish a long-range memory bank to serve as a memory cell storing the rich supportive information. Through our designed temporal variation layer, the supportive cues are further enhanced by multi-scale temporal-only convolutions. To effectively incorporate the two types of cues without disturbing the joint learning of spatio-temporal features, we introduce a non-local bank operator to attentively relate the past to the present. In this regard, our TMRNet enables the current feature to view the long-range temporal dependency, as well as tolerate complex temporal extents. We have extensively validated our approach on two benchmark surgical video datasets, M2CAI challenge dataset and Cholec80 dataset. Experimental results demonstrate the outstanding performance of our method, consistently exceeding the state-of-the-art methods by a large margin (e.g., 67.0% v.s. 78.9% Jaccard on Cholec80 dataset).

cs.CV↗

arXiv2021

One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video

Surgical instrument segmentation in robot-assisted surgery (RAS) - especially that using learning-based models - relies on the assumption that training and testing videos are sampled from the same domain. However, it is impractical and expensive to collect and annotate sufficient data from every new domain. To greatly increase the label efficiency, we explore a new problem, i.e., adaptive instrument segmentation, which is to effectively adapt one source model to new robotic surgical videos from multiple target domains, only given the annotated instruments in the first frame. We propose MDAL, a meta-learning based dynamic online adaptive learning scheme with a two-stage framework to fast adapt the model parameters on the first frame and partial subsequent frames while predicting the results. MDAL learns the general knowledge of instruments and the fast adaptation ability through the video-specific meta-learning paradigm. The added gradient gate excludes the noisy supervision from pseudo masks for dynamic online adaptation on target videos. We demonstrate empirically that MDAL outperforms other state-of-the-art methods on two datasets (including a real-world RAS dataset). The promising performance on ex-vivo scenes also benefits the downstream tasks such as robot-assisted suturing and camera control.

cs.CV↗

arXiv2021

Future Frame Prediction for Robot-assisted Surgery

Predicting future frames for robotic surgical video is an interesting, important yet extremely challenging problem, given that the operative tasks may have complex dynamics. Existing approaches on future prediction of natural videos were based on either deterministic models or stochastic models, including deep recurrent neural networks, optical flow, and latent space modeling. However, the potential in predicting meaningful movements of robots with dual arms in surgical scenarios has not been tapped so far, which is typically more challenging than forecasting independent motions of one arm robots in natural scenarios. In this paper, we propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools. Furthermore, we add the invariant prior information from the gesture class into the generation process to constrain the latent space of our model. To our best knowledge, this is the first time that the future frames of dual arm robots are predicted considering their unique characteristics relative to general robotic videos. Experiments demonstrate that our model gains more stable and realistic future frame prediction scenes with the suturing task on the public JIGSAWS dataset.

cs.CV↗

arXiv2020

Quantum estimation of acceleration and temperature in open quantum system

In an open quantum system, we study the evolution of a two-level atom as a detector which interacts with given environments. For a uniformly accelerated two-level atom coupled to a massless scalar field in the Minkowski vacuum, when it evolves for a certain time, we find that there exists a peak value for the quantum Fisher information (QFI) of acceleration, which indicates that the optimal precision of estimation is achieved when choosing an appropriate acceleration $a$. QFI has different behaviors for different initial state parameters $θ$ of the atom, displaying periodicity. However, the periodicity fades away with the evolution of time, which means that the initial state cannot affect the later stable quantum state. Furthermore, adding a boundary, we observe that the peak value of QFI increases when the atom is close to the boundary, which shows that QFI is protected by the boundary. Here, QFI fluctuates, and there may exist two peak values with a certain moment, which expands the detection range of the acceleration. Therefore, we can enhance the estimation precision of acceleration by choosing an appropriate position and acceleration $a$. The periodicity of QFI with respect to the initial state parameter $θ$ lasts a longer time than the previous unbounded case, which indicates that the initial state is protected by the boundary. Finally, for a thermal bath with a boundary, QFI of temperature has no more than one peak value with a certain moment, which is different from QFI of acceleration with a boundary. The periodicity also lasts a longer time than unbounded case, which shows the initial quantum state of the atom is protected by the boundary for two cases.

quant-ph↗

arXiv2020

Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video

Performing low hertz labeling for surgical videos at intervals can greatly releases the burden of surgeons. In this paper, we study the semi-supervised instrument segmentation from robotic surgical videos with sparse annotations. Unlike most previous methods using unlabeled frames individually, we propose a dual motion based method to wisely learn motion flows for segmentation enhancement by leveraging temporal dynamics. We firstly design a flow predictor to derive the motion for jointly propagating the frame-label pairs given the current labeled frame. Considering the fast instrument motion, we further introduce a flow compensator to estimate intermediate motion within continuous frames, with a novel cycle learning strategy. By exploiting generated data pairs, our framework can recover and even enhance temporal consistency of training sequences to benefit segmentation. We validate our framework with binary, part, and type tasks on 2017 MICCAI EndoVis Robotic Instrument Segmentation Challenge dataset. Results show that our method outperforms the state-of-the-art semi-supervised methods by a large margin, and even exceeds fully supervised training on two tasks.

cs.CV↗

arXiv2019

PFA-ScanNet: Pyramidal Feature Aggregation with Synergistic Learning for Breast Cancer Metastasis Analysis

Automatic detection of cancer metastasis from whole slide images (WSIs) is a crucial step for following patient staging and prognosis. Recent convolutional neural network based approaches are struggling with the trade-off between accuracy and computational efficiency due to the difficulty in processing large-scale gigapixel WSIs. To meet this challenge, we propose a novel Pyramidal Feature Aggregation ScanNet (PFA-ScanNet) for robust and fast analysis of breast cancer metastasis. Our method mainly benefits from the aggregation of extracted local-to-global features with diverse receptive fields, as well as the proposed synergistic learning for training the main detector and extra decoder with semantic guidance. Furthermore, a high-efficiency inference mechanism is designed with dense pooling layers, which allows dense and fast scanning for gigapixel WSI analysis. As a result, the proposed PFA-ScanNet achieved the state-of-the-art FROC of 90.2% on the Camelyon16 dataset, as well as competitive kappa score of 0.905 on the Camelyon17 leaderboard. In addition, our method shows leading speed advantage over other methods, about 7.2 min per WSI with a single GPU, making automatic analysis of breast cancer metastasis more applicable in the clinical usage.

cs.CV↗

arXiv2018

MRI Cross-Modality NeuroImage-to-NeuroImage Translation

We present a cross-modality generation framework that learns to generate translated modalities from given modalities in MR images without real acquisition. Our proposed method performs NeuroImage-to-NeuroImage translation (abbreviated as N2N) by means of a deep learning model that leverages conditional generative adversarial networks (cGANs). Our framework jointly exploits the low-level features (pixel-wise information) and high-level representations (e.g. brain tumors, brain structure like gray matter, etc.) between cross modalities which are important for resolving the challenging complexity in brain structures. Our framework can serve as an auxiliary method in clinical diagnosis and has great application potential. Based on our proposed framework, we first propose a method for cross-modality registration by fusing the deformation fields to adopt the cross-modality information from translated modalities. Second, we propose an approach for MRI segmentation, translated multichannel segmentation (TMS), where given modalities, along with translated modalities, are segmented by fully convolutional networks (FCN) in a multichannel manner. Both of these two methods successfully adopt the cross-modality information to improve the performance without adding any extra data. Experiments demonstrate that our proposed framework advances the state-of-the-art on five brain MRI datasets. We also observe encouraging results in cross-modality registration and segmentation on some widely adopted brain datasets. Overall, our work can serve as an auxiliary method in clinical diagnosis and be applied to various tasks in medical fields. Keywords: image-to-image, cross-modality, registration, segmentation, brain MRI

cs.CV↗

arXiv2018

Matlab Implementation of Machine Vision Algorithm on Ballast Degradation Evaluation

America has a massive railway system. As of 2006, U.S. freight railroads have 140,490 route- miles of standard gauge, but maintaining such a huge system and eliminating any dangers, like reduced track stability and poor drainage, caused by railway ballast degradation require huge amount of labor. The traditional way to quantify the degradation of ballast is to use an index called Fouling Index (FI) through ballast sampling and sieve analysis. However, determining the FI values in lab is very time-consuming and laborious, but with the help of recent development in the field of computer vision, a novel method for a potential machine-vison based ballast inspection system can be employed that can hopefully replace the traditional mechanical method. The new machine-vision approach analyses the images of the in-service ballasts, and then utilizes image segmentation algorithm to get ballast segments. By comparing the segment results and their corresponding FI values, this novel method produces a machine-vision-based index that has the best-fit relation with FI. The implementation details of how this algorithm works are discussed in this report.

cs.CV↗

arXiv2018

Constructing Locally Dense Point Clouds Using OpenSfM and ORB-SLAM2

This paper aims at finding a method to register two different point clouds constructed by ORB-SLAM2 and OpenSfM. To do this, we post some tags with unique textures in the scene and take videos and photos of that area. Then we take short videos of only the tags to extract their features. By matching the ORB feature of the tags with their corresponding features in the scene, it is then possible to localize the position of these tags both in point clouds constructed by ORB-SLAM2 and OpenSfM. Thus, the best transformation matrix between two point clouds can be calculated, and the two point clouds can be aligned.

cs.CV↗

arXiv2016

Generalized thermodynamic identity and new Maxwell's law for charged AdS black hole

We study the thermodynamic properties of the RN-AdS black hole in full phase space and propose a generalized thermodynamic identity. As an example, we use it to find relations of thermodynamical coefficients between the grand canonical and canonical ensembles. We also show, for the first order phase transition, that the usual Maxwell's equal area law should be extended to a new form for the RN-AdS black hole.

gr-qc↗

arXiv2014

Notes on analytical study of holographic superconductors with Lifshitz scaling in external magnetic field

We employ the matching method to analytically investigate the holographic superconductors with Lifshitz scaling in an external magnetic field. We discuss systematically the restricted conditions for the matching method and find that this analytic method is not always powerful to explore the effect of external magnetic field on the holographic superconductors unless the matching point is chosen in an appropriate range and the dynamical exponent $z$ satisfies the relation $z=d-1$ or $z=d-2$. From the analytic treatment, we observe that Lifshitz scaling can hinder the condensation to be formed, which can be used to back up the numerical results. Moreover, we study the effect of Lifshitz scaling on the upper critical magnetic field and reproduce the well-known relation obtained from Ginzburg-Landau theory.

hep-th↗

arXiv2014

Ehrenfest scheme for complex thermodynamic systems in full phase space

For a thermodynamic system with multiple pairs of intensive/extensive variables and the thermodynamical coefficients attain finite or infinite values on the phase boundary, we obtain the two classes of Ehrenfest equations in the full phase space, and find that the rank of the matrix for these equations can tell us the dimensions of the phase boundary. We also apply this treatment to the RN-AdS black hole.

gr-qc↗

arXiv2013

Notes on holographic superconductor models with the nonlinear electrodynamics

We investigate systematically the effect of the nonlinear correction to the usual Maxwell electrodynamics on the holographic dual models in the backgrounds of AdS black hole and AdS soliton. Considering three types of typical nonlinear electrodynamics, we observe that in the black hole background the higher nonlinear electrodynamics correction makes the condensation harder to form and changes the expected relation in the gap frequency, which is similar to that caused by the curvature correction. However, in strong contrast to the influence of the curvature correction, we find that in the AdS soliton background the nonlinear electrodynamics correction will not affect the properties of the holographic superconductor and insulator phase transitions, which may be a quite general feature for the s-wave holographic superconductor/insulator system.

hep-th↗

arXiv2013

Holographic insulator/superconductor phase transition with Weyl corrections

We analytically investigate the phase transition between the holographic insulator and superconductor with Weyl corrections by using the variational method for the Sturm-Liouville eigenvalue problem. We find that similar to the curvature corrections, in p-wave model, the higher Weyl couplings make the insulator/superconductor phase transition harder to occur. However, in s-wave case the Weyl corrections do not influence the critical chemical potential, which is in contrast to the effect caused by the curvature corrections. Moreover, we observe that the Weyl corrections will not affect the critical phenomena and the critical exponent of the system always takes the mean-field value in both models. Our analytic results are found to be in good agreement with the numerical findings.

hep-th↗

arXiv2013

Holographic superconductor models with $RF^{2}$ corrections

We investigate the effect of the $RF^2$ correction on the holographic superconductor model in the background of AdS black hole. We find that, similar to the effect caused by the Weyl correction, the higher $RF^2$ correction term can make it easier for the scalar operator to condense and result in the larger deviations from the expected relation in the gap frequency. However, if a non-trivial hair for the black hole has been triggered, we observe that the $RF^2$ correction and the Weyl correction do play different roles in the behavior of the condensation, which can be used to support the existing findings.

gr-qc↗