arXiv subjects

Siwei Dong

Publications and source records attributed to Siwei Dong.

14 recordsLinked to original sources

arXiv2026

Lect\=uraAgents: A Multi-Agent Framework for Adaptive Personalized AI-Assisted Learning and Embodied Teaching

Effective personalized AI-assisted learning demands systems that can not only generate accurate learner-specific educational materials, but also dynamically adapt their instruction to diverse learners. However, existing educational agents have primarily focused on lecture content automation and simulations, which often fall short of modelling multimodal and embodied instructional methods tailored for the individual learner. To this end, we propose Lect\=uraAgents - a multi-agent framework that enables personalized learning through end-to-end adaptive embodied teaching. At its core, Lect\=uraAgents mirrors a professor-student relationship, in which a ProfessorAgent leads a collaborative team of specialized subordinate agents through research, planning, review, and embodied delivery of lecture contents that adapt to a learner's needs. The framework offers three main contributions: (1) a hierarchical multi-agent architecture for end-to-end personalized learning; (2) an adaptive embodied teaching mechanism, wherein the ProfessorAgent executes visible and pedagogically motivated teaching actions (e.g., handwrite, highlight, underline, etc.) over contents in a teaching environment; and (3) a Teaching Action-Speech Alignment (TASA) algorithm that employs salience-based heuristics and temporal semantic segmentation to generate coherent teaching action sequences aligned with learner profiles. We evaluate Lect\=uraAgents on diverse courses at high school, undergraduate, and graduate levels using sample-specific rubric-based analysis; with generated lecture materials and teaching actions assessed and validated by expert educators. Experimental results show consistent gains in lecture content quality, embodied teaching quality, assessment, and personalization over existing approaches, positioning Lect\=uraAgents as a pedagogically well-grounded framework for personalized learning at scale.

cs.CL↗

arXiv2025

Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play

A voice AI agent that blends seamlessly into daily life would interact with humans in an autonomous, real-time, and emotionally expressive manner. Rather than merely reacting to commands, it would continuously listen, reason, and respond proactively, fostering fluid, dynamic, and emotionally resonant interactions. We introduce Voila, a family of large voice-language foundation models that make a step towards this vision. Voila moves beyond traditional pipeline systems by adopting a new end-to-end architecture that enables full-duplex, low-latency conversations while preserving rich vocal nuances such as tone, rhythm, and emotion. It achieves a response latency of just 195 milliseconds, surpassing the average human response time. Its hierarchical multi-scale Transformer integrates the reasoning capabilities of large language models (LLMs) with powerful acoustic modeling, enabling natural, persona-aware voice generation -- where users can simply write text instructions to define the speaker's identity, tone, and other characteristics. Moreover, Voila supports over one million pre-built voices and efficient customization of new ones from brief audio samples as short as 10 seconds. Beyond spoken dialogue, Voila is designed as a unified model for a wide range of voice-based applications, including automatic speech recognition (ASR), Text-to-Speech (TTS), and, with minimal adaptation, multilingual speech translation. Voila is fully open-sourced to support open research and accelerate progress toward next-generation human-machine interactions.

cs.AI↗

arXiv2024

Momentum and kinetic energy transport in supersonic particle-laden turbulent boundary layers

In the present study, we conduct direct numerical simulations of two-way force-coupled particle-laden compressible turbulent boundary layers at the free-stream Mach number of 2.0 for the purpose of examining the effects of particles on the transport of momentum and kinetic energy. By analyzing turbulent databases with various particle Stokes numbers and mass loadings, we observe that the presence of particles suppresses turbulent fluctuations and can even laminarize flow under high mass loading conditions. This is reflected by the wider and more coherent near-wall velocity streaks, reduced Reynolds stresses, and diminished contributions to skin friction and turbulent kinetic energy production. Additionally, the particle feedback force becomes more dominant in turbulent production near the wall and at small scales as mass loadings increase, which is found to be caused by the residual velocity fluctuations from particles swept down from the outer region. Furthermore, we identify that particle dissipation, resulting from the relative velocity between the fluid and particles, accounts for less than 1% of mean kinetic energy viscous dissipation and less than 10% of turbulent kinetic energy dissipation in the case with the highest mass loading. This suggests a modest impact on the internal energy variation of the fluid if two-way heat coupling is introduced. The elevated mean temperature is found in the near-wall region and is ascribed to the influence of the particle feedback force and reduced turbulent diffusion in high mass loading cases.

physics.flu-dyn↗

arXiv2024

AutoAgents: A Framework for Automatic Agent Generation

Large language models (LLMs) have enabled remarkable advances in automated task-solving with multi-agent systems. However, most existing LLM-based multi-agent approaches rely on predefined agents to handle simple tasks, limiting the adaptability of multi-agent collaboration to different scenarios. Therefore, we introduce AutoAgents, an innovative framework that adaptively generates and coordinates multiple specialized agents to build an AI team according to different tasks. Specifically, AutoAgents couples the relationship between tasks and roles by dynamically generating multiple required agents based on task content and planning solutions for the current task based on the generated expert agents. Multiple specialized agents collaborate with each other to efficiently accomplish tasks. Concurrently, an observer role is incorporated into the framework to reflect on the designated plans and agents' responses and improve upon them. Our experiments on various benchmarks demonstrate that AutoAgents generates more coherent and accurate solutions than the existing multi-agent methods. This underscores the significance of assigning different roles to different tasks and of team cooperation, offering new perspectives for tackling complex tasks. The repository of this project is available at https://github.com/Link-AGI/AutoAgents.

cs.AI↗

arXiv2023

LLaSM: Large Language and Speech Model

Multi-modal large language models have garnered significant interest recently. Though, most of the works focus on vision-language multi-modal models providing strong capabilities in following vision-and-language instructions. However, we claim that speech is also an important modality through which humans interact with the world. Hence, it is crucial for a general-purpose assistant to be able to follow multi-modal speech-and-language instructions. In this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an end-to-end trained large multi-modal speech-language model with cross-modal conversational abilities, capable of following speech-and-language instructions. Our early experiments show that LLaSM demonstrates a more convenient and natural way for humans to interact with artificial intelligence. Specifically, we also release a large Speech Instruction Following dataset LLaSM-Audio-Instructions. Code and demo are available at https://github.com/LinkSoul-AI/LLaSM and https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions dataset is available at https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.

cs.CL↗

arXiv2023

Chinese Open Instruction Generalist: A Preliminary Release

Instruction tuning is widely recognized as a key technique for building generalist language models, which has attracted the attention of researchers and the public with the release of InstructGPT~\citep{ouyang2022training} and ChatGPT\footnote{\url{https://chat.openai.com/}}. Despite impressive progress in English-oriented large-scale language models (LLMs), it is still under-explored whether English-based foundation LLMs can perform similarly on multilingual tasks compared to English tasks with well-designed instruction tuning and how we can construct the corpora needed for the tuning. To remedy this gap, we propose the project as an attempt to create a Chinese instruction dataset by various methods adapted to the intrinsic characteristics of 4 sub-tasks. We collect around 200k Chinese instruction tuning samples, which have been manually checked to guarantee high quality. We also summarize the existing English and Chinese instruction corpora and briefly describe some potential applications of the newly constructed Chinese instruction corpora. The resulting \textbf{C}hinese \textbf{O}pen \textbf{I}nstruction \textbf{G}eneralist (\textbf{COIG}) corpora are available in Huggingface\footnote{\url{https://huggingface.co/datasets/BAAI/COIG}} and Github\footnote{\url{https://github.com/BAAI-Zlab/COIG}}, and will be continuously updated.

cs.CL↗

arXiv2023

Picking Up Quantization Steps for Compressed Image Classification

The sensitivity of deep neural networks to compressed images hinders their usage in many real applications, which means classification networks may fail just after taking a screenshot and saving it as a compressed file. In this paper, we argue that neglected disposable coding parameters stored in compressed files could be picked up to reduce the sensitivity of deep neural networks to compressed images. Specifically, we resort to using one of the representative parameters, quantization steps, to facilitate image classification. Firstly, based on quantization steps, we propose a novel quantization aware confidence (QAC), which is utilized as sample weights to reduce the influence of quantization on network training. Secondly, we utilize quantization steps to alleviate the variance of feature distributions, where a quantization aware batch normalization (QABN) is proposed to replace batch normalization of classification networks. Extensive experiments show that the proposed method significantly improves the performance of classification networks on CIFAR-10, CIFAR-100, and ImageNet. The code is released on https://github.com/LiMaPKU/QSAM.git

cs.CV↗

arXiv2022

1000x Faster Camera and Machine Vision with Ordinary Devices

In digital cameras, we find a major limitation: the image and video form inherited from a film camera obstructs it from capturing the rapidly changing photonic world. Here, we present vidar, a bit sequence array where each bit represents whether the accumulation of photons has reached a threshold, to record and reconstruct the scene radiance at any moment. By employing only consumer-level CMOS sensors and integrated circuits, we have developed a vidar camera that is 1,000x faster than conventional cameras. By treating vidar as spike trains in biological vision, we have further developed a spiking neural network-based machine vision system that combines the speed of the machine and the mechanism of biological vision, achieving high-speed object detection and tracking 1,000x faster than human vision. We demonstrate the utility of the vidar camera and the super vision system in an assistant referee and target pointing system. Our study is expected to fundamentally revolutionize the image and video concepts and related industries, including photography, movies, and visual media, and to unseal a new spiking neural network-enabled speed-free machine vision era.

cs.CV↗

arXiv2021

Spike Camera and Its Coding Methods

This paper introduces a spike camera with a distinct video capture scheme and proposes two methods of decoding the spike stream for texture reconstruction. The spike camera captures light and accumulates the converted luminance intensity at each pixel. A spike is fired when the accumulated intensity exceeds the dispatch threshold. The spike stream generated by the camera indicates the luminance variation. Analyzing the patterns of the spike stream makes it possible to reconstruct the picture of any moment which enables the playback of high speed movement.

cs.MM↗

arXiv2020

The coherent structure of the kinetic energy transfer in shear turbulence

The cascade of energy in turbulent flows, i.e., the transfer of kinetic energy from large to small flow scales or vice versa (backward cascade), is the cornerstone of most theories and models of turbulence since the 1940s. Yet, understanding the spatial organisation of kinetic energy transfer remains an outstanding challenge in fluid mechanics. Here, we unveil the three-dimensional structure of the energy cascade across the shear-dominated scales using numerical data of homogeneous shear turbulence. We show that the characteristic flow structure associated with the energy transfer is a vortex shaped as an inverted hairpin followed by an upright hairpin. The asymmetry between the forward and backward cascade arises from the opposite flow circulation within the hairpins, which triggers reversed patterns in the flow.

physics.flu-dyn↗

arXiv2019

An Efficient Coding Method for Spike Camera using Inter-Spike Intervals

Recently, a novel bio-inspired spike camera has been proposed, which continuously accumulates luminance intensity and fires spikes while the dispatch threshold is reached. Compared to the conventional frame-based cameras and the emerging dynamic vision sensors, the spike camera has shown great advantages in capturing fast-moving scene in a frame-free manner with full texture reconstruction capabilities. However, it is difficult to transmit or store the large amount of spike data. To address this problem, we first investigate the spatiotemporal distribution of inter-spike intervals and propose an intensity-based measurement of spike train distance. Then, we design an efficient spike coding method, which integrates the techniques of adaptive temporal partitioning, intra-/inter-pixel prediction, quantization and entropy coding into a unified lossy coding framework. Finally, we construct a PKU-Spike dataset captured by the spike camera to evaluate the compression performance. The experimental results on the dataset demonstrate that the proposed approach is effective in compressing such spike data while maintaining the fidelity.

cs.MM↗

arXiv2019

A Retina-inspired Sampling Method for Visual Texture Reconstruction

Conventional frame-based camera is not able to meet the demand of rapid reaction for real-time applications, while the emerging dynamic vision sensor (DVS) can realize high speed capturing for moving objects. However, to achieve visual texture reconstruction, DVS need extra information apart from the output spikes. This paper introduces a fovea-like sampling method inspired by the neuron signal processing in retina, which aims at visual texture reconstruction only taking advantage of the properties of spikes. In the proposed method, the pixels independently respond to the luminance changes with temporal asynchronous spikes. Analyzing the arrivals of spikes makes it possible to restore the luminance information, enabling reconstructing the natural scene for visualization. Three decoding methods of spike stream for texture reconstruction are proposed for high-speed motion and stationary scenes. Compared to conventional frame-based camera and DVS, our model can achieve better image quality and higher flexibility, which is capable of changing the way that demanding machine vision applications are built.

eess.IV↗

arXiv2016

Direct numerical simulation of statistically stationary and homogeneous shear turbulence and its relation to other shear flows

Statistically stationary and homogeneous shear turbulence (SS-HST) is investigated by means of a new direct numerical simulation code, spectral in the two horizontal directions and compact-finite-differences in the direction of the shear. No remeshing is used to impose the shear-periodic boundary condition. The influence of the geometry of the computational box is explored. Since HST has no characteristic outer length scale and tends to fill the computational domain, long-term simulations of HST are `minimal' in the sense of containing on average only a few large-scale structures. It is found that the main limit is the spanwise box width, $L_z$, which sets the length and velocity scales of the turbulence, and that the two other box dimensions should be sufficiently large $(L_x\gtrsim 2L_z$, $L_y \gtrsim L_z$) to prevent other directions to be constrained as well. It is also found that very long boxes, $L_x \gtrsim 2 L_y$, couple with the passing period of the shear-periodic boundary condition, and develop strong unphysical linearized bursts. Within those limits, the flow shows interesting similarities and differences with other shear flows, and in particular with the logarithmic layer of wall-bounded turbulence. They are explored in some detail. They include a self-sustaining process for large-scale streaks and quasi-periodic bursting. The bursting time scale is approximately universal, $\sim 20 S^{-1}$, and the availability of two different bursting systems allows the growth of the bursts to be related with some confidence to the shearing of initially isotropic turbulence. It is concluded that SS-HST, conducted within the proper computational parameters, is a very promising system to study shear turbulence in general.

physics.flu-dyn↗

arXiv2015

The temporal evolution of the energy flux across scales in homogeneous turbulence

A temporal study of energy transfer across length scales is performed in 3D numerical simulations of homogeneous shear flow and isotropic turbulence. The average time taken by perturbations in the energy flux to travel between scales is measured and shown to be additive. Our data suggests that the propagation of disturbances in the energy flux is independent of the forcing and that it defines a `velocity' that determines the energy flux itself. These results support that the cascade is, on average, a scale-local process where energy is continuously transmitted from one scale to the next in order of decreasing size.

physics.flu-dyn↗