arXiv subjects

Zejian Li

Publications and source records attributed to Zejian Li.

32 records · Page 2Linked to original sources

arXiv2024

MindScratch: A Visual Programming Support Tool for Classroom Learning Based on Multimodal Generative AI

Programming has become an essential component of K-12 education and serves as a pathway for developing computational thinking skills. Given the complexity of programming and the advanced skills it requires, previous research has introduced user-friendly tools to support young learners. However, our interviews with six programming educators revealed that current tools often fail to reflect classroom learning objectives, offer flexible, high-quality guidance, and foster student creativity. This highlights the need for more adaptive and reflective tools. Therefore, we introduced MindScratch, a multimodal generative AI (GAI) powered visual programming support tool. MindScratch aims to balance structured classroom activities with free programming creation, supporting students in completing creative programming projects based on teacher-set learning objectives while also providing programming scaffolding. Our user study results indicate that, compared to the baseline, MindScratch more effectively helps students achieve high-quality projects aligned with learning objectives. It also enhances students' computational thinking skills and creative thinking. Overall, we believe that GAI-driven educational tools like MindScratch offer students a focused and engaging learning experience.

cs.HC↗

arXiv2024

Measurement-induced phase transitions in monitored infinite-range interacting systems

A key challenge in observing measurement-induced phase transitions is the mitigation of the post-selection barrier, which causes the reproducibility of specific sequences of measurement readouts--the trajectory--to be exponentially small in system size. Recent studies suggest that some classes of monitored infinite-range systems alleviate this problem by exhibiting a fast saturation of entanglement, resulting in only a polynomial post-selection overhead. This paper answers whether this feature is inherent in infinite-range systems, due to their underlying semiclassical dynamics. We consider three experimentally relevant monitored models: a Tavis-Cummings model, a Superradiance model, and a Bose-Hubbard dimer, each exhibiting non-trivial monitored dynamics. We unveil the occurrence of entanglement phase transitions in these models, showing how the saturation time is strongly affected by bistability regions, which also prevent the mitigation of the post-selection barrier. Finally, we propose experimental realizations of these models, providing a discussion of post selection from an experimental perspective.

quant-ph↗

arXiv2024

Img2CAD: Conditioned 3D CAD Model Generation from Single Image with Structured Visual Geometry

In this paper, we propose Img2CAD, the first approach to our knowledge that uses 2D image inputs to generate CAD models with editable parameters. Unlike existing AI methods for 3D model generation using text or image inputs often rely on mesh-based representations, which are incompatible with CAD tools and lack editability and fine control, Img2CAD enables seamless integration between AI-based 3D reconstruction and CAD software. We have identified an innovative intermediate representation called Structured Visual Geometry (SVG), characterized by vectorized wireframes extracted from objects. This representation significantly enhances the performance of generating conditioned CAD models. Additionally, we introduce two new datasets to further support research in this area: ABC-mono, the largest known dataset comprising over 200,000 3D CAD models with rendered images, and KOCAD, the first dataset featuring real-world captured objects alongside their ground truth CAD models, supporting further research in conditioned CAD model generation.

cs.CV↗

arXiv2024

SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More

The advent of large models, also known as foundation models, has significantly transformed the AI research landscape, with models like Segment Anything (SAM) achieving notable success in diverse image segmentation scenarios. Despite its advancements, SAM encountered limitations in handling some complex low-level segmentation tasks like camouflaged object and medical imaging. In response, in 2023, we introduced SAM-Adapter, which demonstrated improved performance on these challenging tasks. Now, with the release of Segment Anything 2 (SAM2), a successor with enhanced architecture and a larger training corpus, we reassess these challenges. This paper introduces SAM2-Adapter, the first adapter designed to overcome the persistent limitations observed in SAM2 and achieve new state-of-the-art (SOTA) results in specific downstream tasks including medical image segmentation, camouflaged (concealed) object detection, and shadow detection. SAM2-Adapter builds on the SAM-Adapter's strengths, offering enhanced generalizability and composability for diverse applications. We present extensive experimental results demonstrating SAM2-Adapter's effectiveness. We show the potential and encourage the research community to leverage the SAM2 model with our SAM2-Adapter for achieving superior segmentation outcomes. Code, pre-trained models, and data processing protocols are available at http://tianrun-chen.github.io/SAM-Adaptor/

cs.CV↗

arXiv2024

xLSTM-UNet can be an Effective 2D & 3D Medical Image Segmentation Backbone with Vision-LSTM (ViL) better than its Mamba Counterpart

Convolutional Neural Networks (CNNs) and Vision Transformers (ViT) have been pivotal in biomedical image segmentation, yet their ability to manage long-range dependencies remains constrained by inherent locality and computational overhead. To overcome these challenges, in this technical report, we first propose xLSTM-UNet, a UNet structured deep learning neural network that leverages Vision-LSTM (xLSTM) as its backbone for medical image segmentation. xLSTM is a recently proposed as the successor of Long Short-Term Memory (LSTM) networks and have demonstrated superior performance compared to Transformers and State Space Models (SSMs) like Mamba in Neural Language Processing (NLP) and image classification (as demonstrated in Vision-LSTM, or ViL implementation). Here, xLSTM-UNet we designed extend the success in biomedical image segmentation domain. By integrating the local feature extraction strengths of convolutional layers with the long-range dependency capturing abilities of xLSTM, xLSTM-UNet offers a robust solution for comprehensive image analysis. We validate the efficacy of xLSTM-UNet through experiments. Our findings demonstrate that xLSTM-UNet consistently surpasses the performance of leading CNN-based, Transformer-based, and Mamba-based segmentation networks in multiple datasets in biomedical segmentation including organs in abdomen MRI, instruments in endoscopic images, and cells in microscopic images. With comprehensive experiments performed, this technical report highlights the potential of xLSTM-based architectures in advancing biomedical image analysis in both 2D and 3D. The code, models, and datasets are publicly available at http://tianrun-chen.github.io/xLSTM-UNet/

eess.IV↗

arXiv2024

Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models

In this paper, we introduce a new task: Zero-Shot 3D Reasoning Segmentation for parts searching and localization for objects, which is a new paradigm to 3D segmentation that transcends limitations for previous category-specific 3D semantic segmentation, 3D instance segmentation, and open-vocabulary 3D segmentation. We design a simple baseline method, Reasoning3D, with the capability to understand and execute complex commands for (fine-grained) segmenting specific parts for 3D meshes with contextual awareness and reasoned answers for interactive segmentation. Specifically, Reasoning3D leverages an off-the-shelf pre-trained 2D segmentation network, powered by Large Language Models (LLMs), to interpret user input queries in a zero-shot manner. Previous research have shown that extensive pre-training endows foundation models with prior world knowledge, enabling them to comprehend complex commands, a capability we can harness to "segment anything" in 3D with limited 3D datasets (source efficient). Experimentation reveals that our approach is generalizable and can effectively localize and highlight parts of 3D objects (in 3D mesh) based on implicit textual queries, including these articulated 3d objects and real-world scanned data. Our method can also generate natural language explanations corresponding to these 3D models and the decomposition. Moreover, our training-free approach allows rapid deployment and serves as a viable universal baseline for future research of part-level 3d (semantic) object understanding in various fields including robotics, object manipulation, part assembly, autonomous driving applications, augment reality and virtual reality (AR/VR), and medical applications. The code, the model weight, the deployment guide, and the evaluation protocol are: http://tianrun-chen.github.io/Reason3D/

cs.CV↗

arXiv2023

Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches

The emerging trend of AR/VR places great demands on 3D content. However, most existing software requires expertise and is difficult for novice users to use. In this paper, we aim to create sketch-based modeling tools for user-friendly 3D modeling. We introduce Reality3DSketch with a novel application of an immersive 3D modeling experience, in which a user can capture the surrounding scene using a monocular RGB camera and can draw a single sketch of an object in the real-time reconstructed 3D scene. A 3D object is generated and placed in the desired location, enabled by our novel neural network with the input of a single sketch. Our neural network can predict the pose of a drawing and can turn a single sketch into a 3D model with view and structural awareness, which addresses the challenge of sparse sketch input and view ambiguity. We conducted extensive experiments synthetic and real-world datasets and achieved state-of-the-art (SOTA) results in both sketch view estimation and 3D modeling performance. According to our user study, our method of performing 3D modeling in a scene is $>$5x faster than conventional methods. Users are also more satisfied with the generated 3D model than the results of existing methods.

cs.HC↗

arXiv2023

Efficient estimation of trainability for variational quantum circuits

Parameterized quantum circuits used as variational ansätze are emerging as promising tools to tackle complex problems ranging from quantum chemistry to combinatorial optimization. These variational quantum circuits can suffer from the well-known curse of barren plateaus, which is characterized by an exponential vanishing of the cost-function gradient with the system size, making training unfeasible for practical applications. Since a generic quantum circuit cannot be simulated efficiently, the determination of its trainability is an important problem. Here we find an efficient method to compute the gradient of the cost function and its variance for a wide class of variational quantum circuits. Our scheme relies on our proof of an exact mapping from randomly initialized circuits to a set of Clifford circuits that can be efficiently simulated on a classical computer by virtue of the celebrated Gottesmann-Knill theorem. This method is scalable and can be used to certify trainability for variational quantum circuits and explore design strategies that can overcome the barren plateau problem. As illustrative examples, we show results with up to 100 qubits.

quant-ph↗

arXiv2023

SAM Fails to Segment Anything? -- SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More

The emergence of large models, also known as foundation models, has brought significant advancements to AI research. One such model is Segment Anything (SAM), which is designed for image segmentation tasks. However, as with other foundation models, our experimental findings suggest that SAM may fail or perform poorly in certain segmentation tasks, such as shadow detection and camouflaged object detection (concealed object detection). This study first paves the way for applying the large pre-trained image segmentation model SAM to these downstream tasks, even in situations where SAM performs poorly. Rather than fine-tuning the SAM network, we propose \textbf{SAM-Adapter}, which incorporates domain-specific information or visual prompts into the segmentation network by using simple yet effective adapters. By integrating task-specific knowledge with general knowledge learnt by the large model, SAM-Adapter can significantly elevate the performance of SAM in challenging tasks as shown in extensive experiments. We can even outperform task-specific network models and achieve state-of-the-art performance in the task we tested: camouflaged object detection, shadow detection. We also tested polyp segmentation (medical image segmentation) and achieves better results. We believe our work opens up opportunities for utilizing SAM in downstream tasks, with potential applications in various fields, including medical image processing, agriculture, remote sensing, and more.

cs.CV↗

arXiv2022

Machine learning via relativity-inspired quantum dynamics

We present a machine-learning scheme based on the relativistic dynamics of a quantum system, namely a quantum detector inside a cavity resonator. An equivalent analog model can be realized for example in a circuit QED platform subject to properly modulated driving fields. We consider a reservoir-computing scheme where the input data are embedded in the modulation of the system (equivalent to the acceleration of the relativistic object) and the output data are obtained by linear combinations of measured observables. As an illustrative example, we have simulated such a relativistic quantum machine for a challenging classification task, showing a very large enhancement of the accuracy in the relativistic regime. Using kernel-machine theory, we show that in the relativistic regime the task-independent expressivity is dramatically magnified with respect to the Newtonian regime.

quant-ph↗

arXiv2022

Noisy Quantum Kernel Machines

In the noisy intermediate-scale quantum era, an important goal is the conception of implementable algorithms that exploit the rich dynamics of quantum systems and the high dimensionality of the underlying Hilbert spaces to perform tasks while prescinding from noise-proof physical systems. An emerging class of quantum learning machines is that based on the paradigm of quantum kernels. Here, we study how dissipation and decoherence affect their performance. We address this issue by investigating the expressivity and the generalization capacity of these models within the framework of kernel theory. We introduce and study the effective kernel rank, a figure of merit that quantifies the number of independent features a noisy quantum kernel is able to extract from input data. Moreover, we derive an upper bound on the generalization error of the model that involves the average purity of the encoded states. Thereby we show that decoherence and dissipation can be seen as an implicit regularization for the quantum kernel machines. As an illustrative example, we report exact finite-size simulations of machines based on chains of driven-dissipative quantum spins to perform a classification task, where the input data are encoded into the driving fields and the quantum physical system is fixed. We determine how the performance of noisy kernel machines scales with the number of nodes (chain sites) as a function of decoherence and examine the effect of imperfect measurements.

quant-ph↗

arXiv2021

Effective bath model for arrays of coupled non-Hermitian nanoresonators

Nanophotonics systems have recently been studied under the perspective of non-Hermitian physics. Given their potential for wavefront control, nonlinear optics and quantum optics, it is crucial to develop predictive tools to assist their design. We present here a simple model relying on the coupling to an effective bath consisting of a continuum of modes to describe systems of coupled resonators, and test it on dielectric nanocylinder chains accessible to experiments. The effective coupling constants, which depend non-trivially on the distance between resonators, are extracted from numerical simulations in the case of just two coupled elements. The model predicts successfully the dispersive and reactive nature of modes for configurations with multiple resonators, as validated by numerical solutions. It can be applied to larger systems, which are hardly solvable with finite-element approaches.

physics.optics↗

arXiv2020

Dissipation-induced antiferromagnetic-like frustration in coupled photonic resonators

We propose a photonic quantum simulator for anti-ferromagnetic spin systems based on reservoir engineering. We consider a scheme where quadratically driven dissipative Kerr cavities are indirectly coupled via lossy ancillary cavities. We show that the ancillary cavities can produce an effective dissipative and Hamiltonian anti-ferromagnetic-like coupling between the cavities. By solving the master equation for a triangular cavity configuration, we demonstrate that the non-equilibrium steady state of the system bears full analogy with the ground state of an antiferromagnetic Ising model, exhibiting key signatures of frustration. We show that when the effective photon hopping amplitude is zero, the engineered non-local dissipation alone is capable of inducing antiferromagnetic interaction and frustration. This simple scheme can be generalised to arbitrary lattice geometries, providing a fully controllable recipe for simulating antiferromagnetism and frustration on a controlled quantum optical platform.

quant-ph↗

arXiv2018

Unsupervised Disentangled Representation Learning with Analogical Relations

Learning the disentangled representation of interpretable generative factors of data is one of the foundations to allow artificial intelligence to think like people. In this paper, we propose the analogical training strategy for the unsupervised disentangled representation learning in generative models. The analogy is one of the typical cognitive processes, and our proposed strategy is based on the observation that sample pairs in which one is different from the other in one specific generative factor show the same analogical relation. Thus, the generator is trained to generate sample pairs from which a designed classifier can identify the underlying analogical relation. In addition, we propose a disentanglement metric called the subspace score, which is inspired by subspace learning methods and does not require supervised information. Experiments show that our proposed training strategy allows the generative models to find the disentangled factors, and that our methods can give competitive performances as compared with the state-of-the-art methods.

cs.LG↗