Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · physics/0401007

The CAVES Project - Exploring Virtual Data Concepts for Data Analysis

Dimitri Bourilkov
View original source ↗Read source document ↗Export metadata ↓

Abstract

The Collaborative Analysis Versioning Environment System (CAVES) project concentrates on the interactions between users performing data and/or computing intensive analyses on large data sets, as encountered in many contemporary scientific disciplines. In modern science increasingly larger groups of researchers collaborate on a given topic over extended periods of time. The logging and sharing of knowledge about how analyses are performed or how results are obtained is important throughout the lifetime of a project. Here is where virtual data concepts play a major role. The ability to seamlessly log, exchange and reproduce results and the methods, algorithms and computer programs used in obtaining them enhances in a qualitative way the level of collaboration in a group or between groups in larger organizations. The CAVES project takes a pragmatic approach in assessing the needs of a community of scientists by building series of prototypes with increasing sophistication. In extending the functionality of existing data analysis packages with virtual data capabilities these prototypes provide an easy and habitual entry point for researchers to explore virtual data concepts in real life applications and to provide valuable feedback for refining the system design. The architecture is modular based on Web, Grid and other services which can be plugged in as desired. As a proof of principle we build a first system by extending the very popular data analysis framework ROOT, widely used in high energy physics and other fields, making it virtual data enabled.

Explore related subjects

physics.data-anhep-ex

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Dimitri Bourilkov. 2004-01-04. The CAVES Project - Exploring Virtual Data Concepts for Data Analysis. https://arxiv.org/abs/physics/0401007

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A Probability Model for Pentagonal Prism Dice Rolls

In the realm of probability theory and game design, the study of dice probabilities plays a crucial role. The featured dice are usually fair, so their probabilities can be predicted without rolling the dice. Pentagonal prismatic dice are not fair. Hence, the probabilities are more complicated. Our mathematical model is naturally derived from both the geometric properties of the dice and empirical evidence. Dice of varying sizes but with constant volume were designed. Using a 3D printer with 100 percent infill, the dice were printed and prepared. A machine that simulates dropping dice through a dice tower onto a hard surface was used to obtain about 41,000 results. From these data, a statistical model was constructed that gives probabilities of dice with varying height-to-radius ratios. The formula for the model is based on the centroid solid angle, but with two adjustment parameters due to varying energy and other requirements to transition between different resting aspects. Within the range of height-to-radius ratios tested, the data are within reasonable statistical error of the model.

physics.data-an↗
arXiv2026

Accelerating fits in high-energy physics and beyond

Data analysis in particle physics often involves optimizing a loss function that quantifies how well a theoretical model describes the observed data, in order to estimate the parameters of the model. This is referred to as fitting. As datasets grow larger and more expensive analysis methods are used, the time required to perform this parameter estimation can be a limiting factor. This work explores techniques for accelerating the fitting process, and measures their performance on a set of models representative of a variety of physics fits. We compare the speed and precision of variants of well-known optimization algorithms, namely Newton's method, BFGS, and the Migrad algorithm (commonly used in particle physics fits). We observe significant speedups when using some algorithms: most notably, some variants of BFGS are about ten times faster than Migrad, while achieving the same precision. For quasi-Newtonian algorithms like BFGS and Migrad, we find that the initial value of the Hessian matrix has a large impact on the runtime. We additionally discuss methods for accelerating the evaluation of the loss function and its derivatives, using algebraic simplifications, automatic differentiation, and modern hardware features. We explore techniques for achieving good precision with 32-bit operations, which can achieve significantly higher throughput on the GPU. Lastly we explore efficient parallelization strategies for the CPU and GPU. All of the techniques and algorithms in this paper are implemented in our new FastFit library, which was used for all of our benchmarks.

physics.data-an↗
arXiv2026

Elementwise Adaptive Random Search for High-Dimensional Constrained Continuous Global Optimization

We propose a new stochastic method for high-dimensional constrained continuous global optimization without convexity assumptions. The algorithm combines global exploration and local refinement through coordinate-wise adaptive perturbations, whose scale is proportional to the current iterate. We show that global exploration reaches any feasible region with positive probability, while local search benefits from strong concentration properties, enabling efficient decline near optima. Their combination yields convergence to a global minimizer under mild regularity conditions. Applications to LASSO sparsity improvement as well as similar BPDN (Basis Pursuit DeNoising) problems, and to neural network pruning, illustrate the practical effectiveness of the approach.

physics.data-an↗