arXiv subjects

Zeng Zhao

Publications and source records attributed to Zeng Zhao.

17 recordsLinked to original sources

arXiv2025

ComfyGPT: A Self-Optimizing Multi-Agent System for Comprehensive ComfyUI Workflow Generation

ComfyUI is a popular workflow-based interface that allows users to customize image generation tasks through an intuitive node-based system. However, the complexity of managing node connections and diverse modules can be challenging for users. In this paper, we introduce ComfyGPT, a self-optimizing multi-agent system designed to generate ComfyUI workflows based on task descriptions automatically. The key innovations of ComfyGPT include: (1) consisting of four specialized agents to build a multi-agent workflow generation system: ReformatAgent, FlowAgent, RefineAgent, and ExecuteAgent; (2) focusing on generating precise node connections instead of entire workflows, improving generation accuracy; and (3) enhancing workflow generation through reinforcement learning. Moreover, we introduce FlowDataset, a large-scale dataset containing 13,571 workflow-description pairs, and FlowBench, a comprehensive benchmark for evaluating workflow generation systems. Additionally, we propose four novel evaluation metrics: Format Validation (FV), Pass Accuracy (PA), Pass Instruct Alignment (PIA), and Pass Node Diversity (PND). Experimental results demonstrate that ComfyGPT significantly outperforms existing LLM-based methods in workflow generation, making it a significant step forward in this field. Code is avaliable at https://github.com/comfygpt/comfygpt.

cs.MA↗

arXiv2024

Storynizor: Consistent Story Generation via Inter-Frame Synchronized and Shuffled ID Injection

Recent advances in text-to-image diffusion models have spurred significant interest in continuous story image generation. In this paper, we introduce Storynizor, a model capable of generating coherent stories with strong inter-frame character consistency, effective foreground-background separation, and diverse pose variation. The core innovation of Storynizor lies in its key modules: ID-Synchronizer and ID-Injector. The ID-Synchronizer employs an auto-mask self-attention module and a mask perceptual loss across inter-frame images to improve the consistency of character generation, vividly representing their postures and backgrounds. The ID-Injector utilize a Shuffling Reference Strategy (SRS) to integrate ID features into specific locations, enhancing ID-based consistent character generation. Additionally, to facilitate the training of Storynizor, we have curated a novel dataset called StoryDB comprising 100, 000 images. This dataset contains single and multiple-character sets in diverse environments, layouts, and gestures with detailed descriptions. Experimental results indicate that Storynizor demonstrates superior coherent story generation with high-fidelity character consistency, flexible postures, and vivid backgrounds compared to other character-specific methods.

cs.CV↗

arXiv2024

LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation

Diffusion models have exhibited substantial success in text-to-image generation. However, they often encounter challenges when dealing with complex and dense prompts involving multiple objects, attribute binding, and long descriptions. In this paper, we propose a novel framework called \textbf{LLM4GEN}, which enhances the semantic understanding of text-to-image diffusion models by leveraging the representation of Large Language Models (LLMs). It can be seamlessly incorporated into various diffusion models as a plug-and-play component. A specially designed Cross-Adapter Module (CAM) integrates the original text features of text-to-image models with LLM features, thereby enhancing text-to-image generation. Additionally, to facilitate and correct entity-attribute relationships in text prompts, we develop an entity-guided regularization loss to further improve generation performance. We also introduce DensePrompts, which contains $7,000$ dense prompts to provide a comprehensive evaluation for the text-to-image generation task. Experiments indicate that LLM4GEN significantly improves the semantic alignment of SD1.5 and SDXL, demonstrating increases of 9.69\% and 12.90\% in color on T2I-CompBench, respectively. Moreover, it surpasses existing models in terms of sample quality, image-text alignment, and human evaluation.

cs.CV↗

arXiv2024

Character-Adapter: Prompt-Guided Region Control for High-Fidelity Character Customization

Customized image generation, which seeks to synthesize images with consistent characters, holds significant relevance for applications such as storytelling, portrait generation, and character design. However, previous approaches have encountered challenges in preserving characters with high-fidelity consistency due to inadequate feature extraction and concept confusion of reference characters. Therefore, we propose Character-Adapter, a plug-and-play framework designed to generate images that preserve the details of reference characters, ensuring high-fidelity consistency. Character-Adapter employs prompt-guided segmentation to ensure fine-grained regional features of reference characters and dynamic region-level adapters to mitigate concept confusion. Extensive experiments are conducted to validate the effectiveness of Character-Adapter. Both quantitative and qualitative results demonstrate that Character-Adapter achieves the state-of-the-art performance of consistent character generation, with an improvement of 24.8% compared with other methods. Our code will be released at https://github.com/Character-Adapter/Character-Adapter.

cs.CV↗

arXiv2024

Let Storytelling Tell Vivid Stories: An Expressive and Fluent Multimodal Storyteller

Storytelling aims to generate reasonable and vivid narratives based on an ordered image stream. The fidelity to the image story theme and the divergence of story plots attract readers to keep reading. Previous works iteratively improved the alignment of multiple modalities but ultimately resulted in the generation of simplistic storylines for image streams. In this work, we propose a new pipeline, termed LLaMS, to generate multimodal human-level stories that are embodied in expressiveness and consistency. Specifically, by fully exploiting the commonsense knowledge within the LLM, we first employ a sequence data auto-enhancement strategy to enhance factual content expression and leverage a textual reasoning architecture for expressive story generation and prediction. Secondly, we propose SQ-Adatpter module for story illustration generation which can maintain sequence consistency. Numerical results are conducted through human evaluation to verify the superiority of proposed LLaMS. Evaluations show that LLaMS achieves state-of-the-art storytelling performance and 86% correlation and 100% consistency win rate as compared with previous SOTA methods. Furthermore, ablation experiments are conducted to verify the effectiveness of proposed sequence data enhancement and SQ-Adapter.

cs.CV↗

arXiv2023

EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior

While image diffusion models have made significant progress in text-driven 3D content creation, they often fail to accurately capture the intended meaning of text prompts, especially for view information. This limitation leads to the Janus problem, where multi-faced 3D models are generated under the guidance of such diffusion models. In this paper, we propose a robust high-quality 3D content generation pipeline by exploiting orthogonal-view image guidance. First, we introduce a novel 2D diffusion model that generates an image consisting of four orthogonal-view sub-images based on the given text prompt. Then, the 3D content is created using this diffusion model. Notably, the generated orthogonal-view image provides strong geometric structure priors and thus improves 3D consistency. As a result, it effectively resolves the Janus problem and significantly enhances the quality of 3D content creation. Additionally, we present a 3D synthesis fusion network that can further improve the details of the generated 3D contents. Both quantitative and qualitative evaluations demonstrate that our method surpasses previous text-to-3D techniques. Project page: https://efficientdreamer.github.io.

cs.CV↗

arXiv2023

Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations

Large-scale vision-language pre-training has achieved significant performance in multi-modal understanding and generation tasks. However, existing methods often perform poorly on image-text matching tasks that require structured representations, i.e., representations of objects, attributes, and relations. As illustrated in Fig.~reffig:case (a), the models cannot make a distinction between ``An astronaut rides a horse" and ``A horse rides an astronaut". This is because they fail to fully leverage structured knowledge when learning representations in multi-modal scenarios. In this paper, we present an end-to-end framework Structure-CLIP, which integrates Scene Graph Knowledge (SGK) to enhance multi-modal structured representations. Firstly, we use scene graphs to guide the construction of semantic negative examples, which results in an increased emphasis on learning structured representations. Moreover, a Knowledge-Enhance Encoder (KEE) is proposed to leverage SGK as input to further enhance structured representations. To verify the effectiveness of the proposed framework, we pre-train our model with the aforementioned approaches and conduct experiments on downstream tasks. Experimental results demonstrate that Structure-CLIP achieves state-of-the-art (SOTA) performance on VG-Attribution and VG-Relation datasets, with 12.5% and 4.1% ahead of the multi-modal SOTA model respectively. Meanwhile, the results on MSCOCO indicate that Structure-CLIP significantly enhances the structured representations while maintaining the ability of general representations. Our code is available at https://github.com/zjukg/Structure-CLIP.

cs.CL↗

arXiv2022

Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation

As a fine-grained and local expression behavior measurement, facial action unit (FAU) analysis (e.g., detection and intensity estimation) has been documented for its time-consuming, labor-intensive, and error-prone annotation. Thus a long-standing challenge of FAU analysis arises from the data scarcity of manual annotations, limiting the generalization ability of trained models to a large extent. Amounts of previous works have made efforts to alleviate this issue via semi/weakly supervised methods and extra auxiliary information. However, these methods still require domain knowledge and have not yet avoided the high dependency on data annotation. This paper introduces a robust facial representation model MAE-Face for AU analysis. Using masked autoencoding as the self-supervised pre-training approach, MAE-Face first learns a high-capacity model from a feasible collection of face images without additional data annotations. Then after being fine-tuned on AU datasets, MAE-Face exhibits convincing performance for both AU detection and AU intensity estimation, achieving a new state-of-the-art on nearly all the evaluation results. Further investigation shows that MAE-Face achieves decent performance even when fine-tuned on only 1\% of the AU training set, strongly proving its robustness and generalization performance.

cs.CV↗

arXiv2022

Easy and Efficient Transformer : Scalable Inference Solution For large NLP model

Recently, large-scale transformer-based models have been proven to be effective over various tasks across many domains. Nevertheless, applying them in industrial production requires tedious and heavy works to reduce inference costs. To fill such a gap, we introduce a scalable inference solution: Easy and Efficient Transformer (EET), including a series of transformer inference optimization at the algorithm and implementation levels. First, we design highly optimized kernels for long inputs and large hidden sizes. Second, we propose a flexible CUDA memory manager to reduce the memory footprint when deploying a large model. Compared with the state-of-the-art transformer inference library (Faster Transformer v4.0), EET can achieve an average of 1.40-4.20x speedup on the transformer decoder layer with an A100 GPU

cs.CL↗

arXiv2020

A Note on Graph-Based Nearest Neighbor Search

Nearest neighbor search has found numerous applications in machine learning, data mining and massive data processing systems. The past few years have witnessed the popularity of the graph-based nearest neighbor search paradigm because of its superiority over the space-partitioning algorithms. While a lot of empirical studies demonstrate the efficiency of graph-based algorithms, not much attention has been paid to a more fundamental question: why graph-based algorithms work so well in practice? And which data property affects the efficiency and how? In this paper, we try to answer these questions. Our insight is that "the probability that the neighbors of a point o tends to be neighbors in the KNN graph" is a crucial data property for query efficiency. For a given dataset, such a property can be qualitatively measured by clustering coefficient of the KNN graph. To show how clustering coefficient affects the performance, we identify that, instead of the global connectivity, the local connectivity around some given query q has more direct impact on recall. Specifically, we observed that high clustering coefficient makes most of the k nearest neighbors of q sit in a maximum strongly connected component (SCC) in the graph. From the algorithmic point of view, we show that the search procedure is actually composed of two phases - the one outside the maximum SCC and the other one in it, which is different from the widely accepted single or multiple paths search models. We proved that the commonly used graph-based search algorithm is guaranteed to traverse the maximum SCC once visiting any point in it. Our analysis reveals that high clustering coefficient leads to large size of the maximum SCC, and thus provides good answer quality with the help of the two-phase search procedure. Extensive empirical results over a comprehensive collection of datasets validate our findings.

cs.LG↗

arXiv2014

Ultrafast and Nanoscale Plasmonic Phenomena in Exfoliated Graphene Revealed by Infrared Pump-Probe Nanoscopy

Pump-probe spectroscopy is central for exploring ultrafast dynamics of fundamental excitations, collective modes and energy transfer processes. Typically carried out using conventional diffraction-limited optics, pump-probe experiments inherently average over local chemical, compositional, and electronic inhomogeneities. Here we circumvent this deficiency and introduce pump-probe infrared spectroscopy with ~20 nm spatial resolution, far below the diffraction limit, which is accomplished using a scattering scanning near-field optical microscope (s-SNOM). This technique allows us to investigate exfoliated graphene single-layers on SiO2 at technologically significant mid-infrared (MIR) frequencies where the local optical conductivity becomes experimentally accessible through the excitation of surface plasmons via the s-SNOM tip. Optical pumping at near-infrared (NIR) frequencies prompts distinct changes in the plasmonic behavior on 200 femtosecond (fs) time scales. The origin of the pump-induced, enhanced plasmonic response is identified as an increase in the effective electron temperature up to several thousand Kelvin, as deduced directly from the Drude weight associated with the plasmonic resonances.

cond-mat.mes-hall↗

arXiv2013

Transport in Suspended Monolayer and Bilayer Graphene Under Strain: A New Platform for Material Studies

We develop two types of graphene devices based on nanoelectromechanical systems (NEMS), that allows transport measurement in the presence of in situ strain modulation. Different mobility and conductance responses to strain were observed for single layer and bilayer samples. These types of devices can be extended to other 2D membranes such as MoS2, providing transport, optical or other measurements with in situ strain.

cond-mat.mtrl-sci↗

arXiv2012

Visualizing Electrical Breakdown and ON/OFF States in Electrically Switchable Suspended Graphene Break Junctions

Narrow gaps are formed in suspended single to few layer graphene devices using a pulsed electrical breakdown technique. The conductance of the resulting devices can be programmed by the application of voltage pulses, with a voltage of 2.5V~4.5V corresponding to an ON pulse and voltages ~8V corresponding to OFF pulses. Electron microscope imaging of the devices shows that the graphene sheets typically remain suspended and that the device conductance tends to zero when the observed gap is large. The switching rate is strongly temperature dependent, which rules out a purely electromechanical switching mechanism. This observed switching in suspended graphene devices strongly suggests a switching mechanism via atomic movement and/or chemical rearrangement, and underscores the potential of all-carbon devices for integration with graphene electronics.

cond-mat.mes-hall↗

arXiv2012

Aryl Functionalization as a Route to Band Gap Engineering in Single Layer Graphene Devices

Chemical functionalization is a promising route to band gap engineering of graphene. We chemically grafted nitrophenyl groups onto exfoliated single-layer graphene sheets in the form of substrate-supported or free-standing films. Our transport measurements demonstrate that non-suspended functionalized graphene behaves as a granular metal, with variable range hopping transport and a mobility gap ~ 0.1 eV at low temperature. For suspended graphene that allows functionalization on both surfaces, we demonstrate tuning of its electronic properties from a granular metal to a gapped semiconductor, in which charge transport occurs via thermal activation over a gap ~ 80 meV. This non-invasive and scalable functionalization technique paves the way for CMOS-compatible band gap engineering of graphene electronic devices.

cond-mat.mes-hall↗

arXiv2011

Infrared nanoscopy of Dirac plasmons at the graphene-SiO2 interface

We report on infrared (IR) nanoscopy of 2D plasmon excitations of Dirac fermions in graphene. This is achieved by confining mid-IR radiation at the apex of a nanoscale tip: an approach yielding two orders of magnitude increase in the value of in-plane component of incident wavevector q compared to free space propagation. At these high wavevectors, the Dirac plasmon is found to dramatically enhance the near-field interaction with mid-IR surface phonons of SiO2 substrate. Our data augmented by detailed modeling establish graphene as a new medium supporting plasmonic effects that can be controlled by gate voltage.

cond-mat.mes-hall↗

arXiv2010

Magnetoconductance Oscillations in High-Mobility Suspended Bilayer and Trilayer Graphene

We report pronounced magnetoconductance oscillations observed on suspended bilayer and trilayer graphene devices with mobilities up to 270,000 cm2/Vs. For bilayer devices, we observe conductance minima at all integer filling factors nu between 0 and -8, as well as a small plateau at ν=1/3. For trilayer devices, we observe features at nu=-1, -2, -3 and -4, and at ν~0.5 that persist to 4.5K at B=8T. All of these features persist for all accessible values of Vg and B, and could suggest the onset of symmetry breaking of the first few Landau (LL) levels and fractional quantum Hall states.

cond-mat.mes-hall↗

arXiv2010

Suspension and Measurement of Graphene and Bi2Se3 Atomic Membranes

Coupling high quality, suspended atomic membranes to specialized electrodes enables investigation of many novel phenomena, such as spin or Cooper pair transport in these two dimensional systems. However, many electrode materials are not stable in acids that are used to dissolve underlying substrates. Here we present a versatile and powerful multi-level lithographical technique to suspend atomic membranes, which can be applied to the vast majority of substrate, membrane and electrode materials. Using this technique, we fabricated suspended graphene devices with Al electrodes and mobility of 5500 cm^2/Vs. We also demonstrate, for the first time, fabrication and measurement of a free-standing thin Bi2Se3 membrane, which has low contact resistance to electrodes and a mobility of >~500 cm^2/Vs.

cond-mat.mes-hall↗