arXiv subjects

Zhen Zeng

Publications and source records attributed to Zhen Zeng.

At least 37 records · Page 2Linked to original sources

arXiv2024

LAW: Legal Agentic Workflows for Custody and Fund Services Contracts

Legal contracts in the custody and fund services domain govern critical aspects such as key provider responsibilities, fee schedules, and indemnification rights. However, it is challenging for an off-the-shelf Large Language Model (LLM) to ingest these contracts due to the lengthy unstructured streams of text, limited LLM context windows, and complex legal jargon. To address these challenges, we introduce LAW (Legal Agentic Workflows for Custody and Fund Services Contracts). LAW features a modular design that responds to user queries by orchestrating a suite of domain-specific tools and text agents. Our experiments demonstrate that LAW, by integrating multiple specialized agents and tools, significantly outperforms the baseline. LAW excels particularly in complex tasks such as calculating a contract's termination date, surpassing the baseline by 92.9% points. Furthermore, LAW offers a cost-effective alternative to traditional fine-tuned legal LLMs by leveraging reusable, domain-specific tools.

cs.AI↗

arXiv2024

AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

State-of-the-art multimodal web agents, powered by Multimodal Large Language Models (MLLMs), can autonomously execute many web tasks by processing user instructions and interacting with graphical user interfaces (GUIs). Current strategies for building web agents rely on (i) the generalizability of underlying MLLMs and their steerability via prompting, and (ii) large-scale fine-tuning of MLLMs on web-related tasks. However, web agents still struggle to automate tasks on unseen websites and domains, limiting their applicability to enterprise-specific and proprietary platforms. Beyond generalization from large-scale pre-training and fine-tuning, we propose building agents for few-shot adaptability using human demonstrations. We introduce the AdaptAgent framework that enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations (up to 2). Our experiments on two popular benchmarks -- Mind2Web & VisualWebArena -- show that using in-context demonstrations (for proprietary models) or meta-adaptation demonstrations (for meta-learned open-weights models) boosts task success rate by 3.36% to 7.21% over non-adapted state-of-the-art models, corresponding to a relative increase of 21.03% to 65.75%. Furthermore, our additional analyses (a) show the effectiveness of multimodal demonstrations over text-only ones, (b) shed light on the influence of different data selection strategies during meta-learning on the generalization of the agent, and (c) demonstrate the effect of number of few-shot examples on the web agent's success rate. Overall, our results unlock a complementary axis for developing widely applicable multimodal web agents beyond large-scale pre-training and fine-tuning, emphasizing few-shot adaptability.

cs.AI↗

arXiv2024

Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Models

Knowledge editing aims to efficiently and cost-effectively correct inaccuracies and update outdated information. Recently, there has been growing interest in extending knowledge editing from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs), which integrate both textual and visual information, introducing additional editing complexities. Existing multimodal knowledge editing works primarily focus on text-oriented, coarse-grained scenarios, failing to address the unique challenges posed by multimodal contexts. In this paper, we propose a visual-oriented, fine-grained multimodal knowledge editing task that targets precise editing in images with multiple interacting entities. We introduce the Fine-Grained Visual Knowledge Editing (FGVEdit) benchmark to evaluate this task. Moreover, we propose a Multimodal Scope Classifier-based Knowledge Editor (MSCKE) framework. MSCKE leverages a multimodal scope classifier that integrates both visual and textual information to accurately identify and update knowledge related to specific entities within images. This approach ensures precise editing while preserving irrelevant information, overcoming the limitations of traditional text-only editing methods. Extensive experiments on the FGVEdit benchmark demonstrate that MSCKE outperforms existing methods, showcasing its effectiveness in solving the complex challenges of multimodal knowledge editing.

cs.CV↗

arXiv2024

Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark

Large Language Models (LLMs) offer the potential for automatic time series analysis and reporting, which is a critical task across many domains, spanning healthcare, finance, climate, energy, and many more. In this paper, we propose a framework for rigorously evaluating the capabilities of LLMs on time series understanding, encompassing both univariate and multivariate forms. We introduce a comprehensive taxonomy of time series features, a critical framework that delineates various characteristics inherent in time series data. Leveraging this taxonomy, we have systematically designed and synthesized a diverse dataset of time series, embodying the different outlined features, each accompanied by textual descriptions. This dataset acts as a solid foundation for assessing the proficiency of LLMs in comprehending time series. Our experiments shed light on the strengths and limitations of state-of-the-art LLMs in time series understanding, revealing which features these models readily comprehend effectively and where they falter. In addition, we uncover the sensitivity of LLMs to factors including the formatting of the data, the position of points queried within a series and the overall time series length.

cs.CL↗

arXiv2024

TRIP-PAL: Travel Planning with Guarantees by Combining Large Language Models and Automated Planners

Travel planning is a complex task that involves generating a sequence of actions related to visiting places subject to constraints and maximizing some user satisfaction criteria. Traditional approaches rely on problem formulation in a given formal language, extracting relevant travel information from web sources, and use an adequate problem solver to generate a valid solution. As an alternative, recent Large Language Model (LLM) based approaches directly output plans from user requests using language. Although LLMs possess extensive travel domain knowledge and provide high-level information like points of interest and potential routes, current state-of-the-art models often generate plans that lack coherence, fail to satisfy constraints fully, and do not guarantee the generation of high-quality solutions. We propose TRIP-PAL, a hybrid method that combines the strengths of LLMs and automated planners, where (i) LLMs get and translate travel information and user information into data structures that can be fed into planners; and (ii) automated planners generate travel plans that guarantee constraint satisfaction and optimize for users' utility. Our experiments across various travel scenarios show that TRIP-PAL outperforms an LLM when generating travel plans.

cs.AI↗

arXiv2024

Rating Multi-Modal Time-Series Forecasting Models (MM-TSFM) for Robustness Through a Causal Lens

AI systems are notorious for their fragility; minor input changes can potentially cause major output swings. When such systems are deployed in critical areas like finance, the consequences of their uncertain behavior could be severe. In this paper, we focus on multi-modal time-series forecasting, where imprecision due to noisy or incorrect data can lead to erroneous predictions, impacting stakeholders such as analysts, investors, and traders. Recently, it has been shown that beyond numeric data, graphical transformations can be used with advanced visual models to achieve better performance. In this context, we introduce a rating methodology to assess the robustness of Multi-Modal Time-Series Forecasting Models (MM-TSFM) through causal analysis, which helps us understand and quantify the isolated impact of various attributes on the forecasting accuracy of MM-TSFM. We apply our novel rating method on a variety of numeric and multi-modal forecasting models in a large experimental setup (six input settings of control and perturbations, ten data distributions, time series from six leading stocks in three industries over a year of data, and five time-series forecasters) to draw insights on robust forecasting models and the context of their strengths. Within the scope of our study, our main result is that multi-modal (numeric + visual) forecasting, which was found to be more accurate than numeric forecasting in previous studies, can also be more robust in diverse settings. Our work will help different stakeholders of time-series forecasting understand the models` behaviors along trust (robustness) and accuracy dimensions to select an appropriate model for forecasting using our rating method, leading to improved decision-making.

cs.LG↗

arXiv2024

OVAL-Prompt: Open-Vocabulary Affordance Localization for Robot Manipulation through LLM Affordance-Grounding

In order for robots to interact with objects effectively, they must understand the form and function of each object they encounter. Essentially, robots need to understand which actions each object affords, and where those affordances can be acted on. Robots are ultimately expected to operate in unstructured human environments, where the set of objects and affordances is not known to the robot before deployment (i.e. the open-vocabulary setting). In this work, we introduce OVAL-Prompt, a prompt-based approach for open-vocabulary affordance localization in RGB-D images. By leveraging a Vision Language Model (VLM) for open-vocabulary object part segmentation and a Large Language Model (LLM) to ground each part-segment-affordance, OVAL-Prompt demonstrates generalizability to novel object instances, categories, and affordances without domain-specific finetuning. Quantitative experiments demonstrate that without any finetuning, OVAL-Prompt achieves localization accuracy that is competitive with supervised baseline models. Moreover, qualitative experiments show that OVAL-Prompt enables affordance-based robot manipulation of open-vocabulary object instances and categories. Project Page: https://ekjt.github.io/OVAL-Prompt/

cs.RO↗

arXiv2024

Broadband microwave waveform generation with programmable chirp shapes via recirculating phase-modulated optical fiber loop controlled by low-speed electronics

Broadband microwave waveforms with programmable chirp shapes are captivating in numerous practical applications. Compared with electronic technology, photonic-assisted solutions exhibit excellent performance in bandwidth and flexibility, but still suffer from complex architecture and requirement of high-speed electronics. Besides, rapid manipulation of chirp shape is still a challenge in the scientific community. In this paper, we propose and demonstrate a novel concept for generating broadband microwave waveforms with programmable chirp shapes. This concept is realized on a simple fiber-optic platform involving a continuous-wave laser source, a recirculating phase-modulated optical fiber loop, and low-speed electronics with a sampling rate at the level of MS/s. Based on this method, chirped microwave waveforms with a bandwidth up to tens of GHz can be generated, where the chirp shape is identical to the low-frequency driving waveform of the recirculating phase-modulated optical fiber loop. In addition, all the parameters of the generated chirped microwave waveforms can be easily reconfigured in real time, including the bandwidth, the central frequency, and the temporal duration. In the experiment, broadband microwave waveforms with customized chirp shapes are generated, where the center frequency and bandwidth tuning ranges exceed 21 GHz, the temporal duration is tuned in the range of 9 ns to 180 ns, and the coherent time of the generated microwave waveform is larger than 100 μs. This simple fiber-optic platform paves a way to generate broadband microwave waveforms with user-definable chirp shapes, which can find applications in broadband radar systems, electronic warfare and wireless communications.

physics.optics↗

arXiv2024

From Pixels to Predictions: Spectrogram and Vision Transformer for Better Time Series Forecasting

Time series forecasting plays a crucial role in decision-making across various domains, but it presents significant challenges. Recent studies have explored image-driven approaches using computer vision models to address these challenges, often employing lineplots as the visual representation of time series data. In this paper, we propose a novel approach that uses time-frequency spectrograms as the visual representation of time series data. We introduce the use of a vision transformer for multimodal learning, showcasing the advantages of our approach across diverse datasets from different domains. To evaluate its effectiveness, we compare our method against statistical baselines (EMA and ARIMA), a state-of-the-art deep learning-based approach (DeepAR), other visual representations of time series data (lineplot images), and an ablation study on using only the time series as input. Our experiments demonstrate the benefits of utilizing spectrograms as a visual representation for time series data, along with the advantages of employing a vision transformer for simultaneous learning in both the time and frequency domains.

cs.CV↗

arXiv2024

FlowMind: Automatic Workflow Generation with LLMs

The rapidly evolving field of Robotic Process Automation (RPA) has made significant strides in automating repetitive processes, yet its effectiveness diminishes in scenarios requiring spontaneous or unpredictable tasks demanded by users. This paper introduces a novel approach, FlowMind, leveraging the capabilities of Large Language Models (LLMs) such as Generative Pretrained Transformer (GPT), to address this limitation and create an automatic workflow generation system. In FlowMind, we propose a generic prompt recipe for a lecture that helps ground LLM reasoning with reliable Application Programming Interfaces (APIs). With this, FlowMind not only mitigates the common issue of hallucinations in LLMs, but also eliminates direct interaction between LLMs and proprietary data or code, thus ensuring the integrity and confidentiality of information - a cornerstone in financial services. FlowMind further simplifies user interaction by presenting high-level descriptions of auto-generated workflows, enabling users to inspect and provide feedback effectively. We also introduce NCEN-QA, a new dataset in finance for benchmarking question-answering tasks from N-CEN reports on funds. We used NCEN-QA to evaluate the performance of workflows generated by FlowMind against baseline and ablation variants of FlowMind. We demonstrate the success of FlowMind, the importance of each component in the proposed lecture recipe, and the effectiveness of user interaction and feedback in FlowMind.

cs.CL↗

arXiv2024

Ionomer structure and component transport in the cathode catalyst layer of PEM fuel cells: A molecular dynamics study

The transport of water and protons in the cathode catalyst layer (CCL) of proton exchange membrane (PEM) fuel cells is critical for cell performance, but the underlying mechanism is still unclear. Herein, the ionomer structure and the distribution/transport characteristics of water and protons in CCLs are investigated via all-atom molecular dynamics simulations. The results show that at low water contents, isolated water clusters form in ionomer pores, while proton transport is mainly via the charged sites of the ionomer side chains and the Grotthuss mechanism. Moreover, with increasing water content, water clusters are interconnected to form continuous water channels, which provide effective paths for proton transfer via the vehicular and Grotthuss mechanisms. Increasing the ionomer mass content can enhance the dense arrangement of the ionomer, which in turn increases the density of charge sites and improves the proton transport efficiency. When the ionomer mass content is high, the clustering effect reduces the space for water diffusion, increases the proton transport path, and finally decreases the proton transport efficiency. By providing physics insights into the proton transport mechanism, this study is helpful for the structural design and performance improvement of CCLs of PEM fuel cells.

physics.chem-ph↗

arXiv2023

Variation of Critical Crystallization Pressure for the Formation of Square Ice in Graphene Nanocapillaries

Two-dimensional square ice in graphene nanocapillaries at room temperature is a fascinating phenomenon and has been confirmed experimentally. Instead of temperature for bulk ice, the high van der Waals pressure becomes an all-important factor to induce the formation of square ice and needs to be studied further. By all-atom molecular dynamics simulations of water confined between two parallel graphene sheets, which are changed in size (the length and the width of the graphene sheets) over a wide range, we find that the critical crystallization pressure for the formation of square ice in nanocapillary strongly depends on the size of the graphene sheet. The critical crystallization pressure slowly decreases as the graphene size increases, converging to approximately macroscopic crystallization pressure. The unfreezable threshold for graphene size is obtained by estimating the actual pressure and it is difficult to form the square ice spontaneously in practice when the graphene sheet is smaller than the threshold. Moreover, the critical crystallization pressure fluctuates when the graphene size is minuscule, and the range of oscillation narrows as the sheet size increases, converging to the macroscopic behavior of a single critical icing pressure for large sheets. The graphene size also affects the stability and crystallization time of square ice.

physics.flu-dyn↗

arXiv2023

Financial Time Series Forecasting using CNN and Transformer

Time series forecasting is important across various domains for decision-making. In particular, financial time series such as stock prices can be hard to predict as it is difficult to model short-term and long-term temporal dependencies between data points. Convolutional Neural Networks (CNN) are good at capturing local patterns for modeling short-term dependencies. However, CNNs cannot learn long-term dependencies due to the limited receptive field. Transformers on the other hand are capable of learning global context and long-term dependencies. In this paper, we propose to harness the power of CNNs and Transformers to model both short-term and long-term dependencies within a time series, and forecast if the price would go up, down or remain the same (flat) in the future. In our experiments, we demonstrated the success of the proposed method in comparison to commonly adopted statistical and deep learning methods on forecasting intraday stock price change of S&P 500 constituents.

cs.LG↗

arXiv2022

Self-supervised Graph Learning for Long-tailed Cognitive Diagnosis

Cognitive diagnosis is a fundamental yet critical research task in the field of intelligent education, which aims to discover the proficiency level of different students on specific knowledge concepts. Despite the effectiveness of existing efforts, previous methods always considered the mastery level on the whole students, so they still suffer from the Long Tail Effect. A large number of students who have sparse data are performed poorly in the model. To relieve the situation, we proposed a Self-supervised Cognitive Diagnosis (SCD) framework which leverages the self-supervised manner to assist the graph-based cognitive diagnosis, then the performance on those students with sparse data can be improved. Specifically, we came up with a graph confusion method that drops edges under some special rules to generate different sparse views of the graph. By maximizing the consistency of the representation on the same node under different views, the model could be more focused on long-tailed students. Additionally, we proposed an importance-based view generation rule to improve the influence of long-tailed students. Extensive experiments on real-world datasets show the effectiveness of our approach, especially on the students with sparse data.

cs.CY↗

arXiv2022

Manipulation-Oriented Object Perception in Clutter through Affordance Coordinate Frames

In order to enable robust operation in unstructured environments, robots should be able to generalize manipulation actions to novel object instances. For example, to pour and serve a drink, a robot should be able to recognize novel containers which afford the task. Most importantly, robots should be able to manipulate these novel containers to fulfill the task. To achieve this, we aim to provide robust and generalized perception of object affordances and their associated manipulation poses for reliable manipulation. In this work, we combine the notions of affordance and category-level pose, and introduce the Affordance Coordinate Frame (ACF). With ACF, we represent each object class in terms of individual affordance parts and the compatibility between them, where each part is associated with a part category-level pose for robot manipulation. In our experiments, we demonstrate that ACF outperforms state-of-the-art methods for object detection, as well as category-level pose estimation for object parts. We further demonstrate the applicability of ACF to robot manipulation tasks through experiments in a simulated environment.

cs.RO↗

arXiv2022

SeanNet: Semantic Understanding Network for Localization Under Object Dynamics

We aim for domestic robots to perform long-term indoor service. Under the object-level scene dynamics induced by daily human activities, a robot needs to robustly localize itself in the environment subject to scene uncertainties. Previous works have addressed visual-based localization in static environments, yet the object-level scene dynamics challenge existing methods for the long-term deployment of the robot. This paper proposes a SEmantic understANding Network (SeanNet) architecture that enables an effective learning process with coupled visual and semantic inputs. With a dataset that contains object dynamics, we propose a cascaded contrastive learning scheme to train the SeanNet for learning a vector scene embedding. Subsequently, we can measure the similarity between the current observed scene and the target scene, whereby enables robust localization under object-level dynamics. In our experiments, we benchmark SeanNet against state-of-the-art image-encoding networks (baselines) on scene similarity measures. The SeanNet architecture with the proposed training method can achieve an 85.02\% accuracy which is higher than baselines. We further integrate the SeanNet and the other networks as the localizers into a visual navigation application. We demonstrate that SeanNet achieves higher success rates compared to the baselines.

cs.RO↗

arXiv2022

TGAVC: Improving Autoencoder Voice Conversion with Text-Guided and Adversarial Training

Non-parallel many-to-many voice conversion remains an interesting but challenging speech processing task. Recently, AutoVC, a conditional autoencoder based method, achieved excellent conversion results by disentangling the speaker identity and the speech content using information-constraining bottlenecks. However, due to the pure autoencoder training method, it is difficult to evaluate the separation effect of content and speaker identity. In this paper, a novel voice conversion framework, named $\boldsymbol T$ext $\boldsymbol G$uided $\boldsymbol A$utoVC(TGAVC), is proposed to more effectively separate content and timbre from speech, where an expected content embedding produced based on the text transcriptions is designed to guide the extraction of voice content. In addition, the adversarial training is applied to eliminate the speaker identity information in the estimated content embedding extracted from speech. Under the guidance of the expected content embedding and the adversarial training, the content encoder is trained to extract speaker-independent content embedding from speech. Experiments on AIShell-3 dataset show that the proposed model outperforms AutoVC in terms of naturalness and similarity of converted speech.

cs.SD↗

arXiv2022

Two-dimensional partitioned square ice confined in graphene/graphite nanocapillaries

As one of the most fascinating confined water/ice phenomena, two-dimensional square ice has been extensively studied and experimentally confirmed in recent years. Apart from the unidirectional homogeneous square icing patterns considered in previous studies, the multidirectional partitioned square icing patterns are discovered in this study and characterized by molecular dynamics (MD) simulations. Square icing parameters are proposed to quantitatively distinguish the partitioned patterns from the homogeneous patterns and the liquid water. The number of graphene monolayers n is varied in this study, and the results show that it is more energetically favorable to form partitioned square icing patterns when the water molecules are confined between graphite sheets (n >= 2) compared to graphene (n = 1). This phenomenon is insensitive to n as long as n >= 2, because of the short-range nature of the interaction between water molecules and the carbon substrate. Moreover, it is energetically unfavorable to form partitioned square icing patterns for a single layer of water molecules even for n >= 2, verifying that the interaction between layers of water molecules is another dominant factor in the formation of partitioned structures. The conversion from partitioned structure to homogenous square patterns is investigated by changing the pressure and the temperature. Based on the comprehensive MD simulations, this study unveils the formation mechanism of the partitioned square icing patterns.

physics.flu-dyn↗