arXiv subjects

Staas de Jong

Publications and source records attributed to Staas de Jong.

12 recordsLinked to original sources

arXiv2025

Computational techniques enabling the perception of virtual images exclusive to the retinal afterimage

The retinal afterimage is a widely known effect in the human visual system, which has been studied and used in the context of a number of major art movements. Therefore, when considering the general role of computation in the visual arts, this begs the question whether this effect, too, may be induced using partly automated techniques. If so, it may become a computationally controllable ingredient of (interactive) visual art, and thus take its place among the many other aspects of visual perception which already have preceded it in this sense. The present moment provides additional inspiration to lay the groundwork for extending computer graphics in general with the retinal afterimage: Historically, we are in a phase where some head-mounted stereoscopic AR/VR technologies are now providing eye tracking by default, thereby allowing realtime monitoring of the processes of visual fixation that can induce the retinal afterimage. A logical starting point for general investigation is then shape display via the retinal afterimage, since shape recognition lends itself well to unambiguous reporting. Shape recognition, however, may also occur due to normal vision, which happens simultaneously. Carefully and rigorously excluding this possibility, we develop computational techniques enabling shape display exclusive to the retinal afterimage.

cs.HC↗

arXiv2025

Balancing physical modeling and musical requirements: Algorithmically simulating the calls of Hyalessa maculaticollis for real-time instrumental control

This paper presents an algorithm that simulates the calls of the Hyalessa maculaticollis cicada for musical use. Written in SuperCollider, its input parameters enable real-time control of the insect call phase, loudness, and perceived musical pitch. To this end, the anatomical mechanics of the tymbal muscles, tymbal apodeme, tymbal ribs, tymbal plate, abdominal air sac, tympana, and opercula are physically modeled. This also includes decoherence, following the hypothesis that it, in H. maculaticollis, might explain the change in timbre apparent during the final phase of a call sequence. Overall, the algorithm seems to illustrate three main points regarding the trade-offs encountered when modeling bioacoustics for tonal use: that it may be necessary to prioritize musical requirements over realistic physical modeling at many stages of design and implementation; that the resulting adjustments may revolve around having physical modeling perceptually yield sonic events that are well-pitched, single-attack, single-source, and timbrally expressive; that the pitch-adjusted simulation of resonating bodies may fail musically precisely when it succeeds physically, by inducing the perception of different sound sources for different pitches. Audio examples are included, and the source code is structured and documented so as to support the further development of cicada bioacoustics for musical use.

cs.SD↗

arXiv2025

Computed fingertip touch for the instrumental control of musical sound with an excursion on the computed retinal afterimage

In this thesis, we present an articulated, empirical view on what human music making is, and on how this fundamentally relates to computation. The experimental evidence which we obtained seems to indicate that this view can be used as a tool, to systematically generate models, hypotheses and new technologies that enable an ever more complete answer to the fundamental question as to what forms of instrumental control of musical sound are possible to implement. This also entails the development of two novel transducer technologies for computed fingertip touch: The cyclotactor (CT) system, which provides fingerpad-orthogonal force output while tracking surface-orthogonal fingertip movement; and the kinetic surface friction transducer (KSFT) system, which provides fingerpad-parallel force output while tracking surface-parallel fingertip movement. In addition to the main research, the thesis also contains two research excursions, which are due to the nature of the Ph.D. position. The first excursion shows how repeated and varying pressing movements on the already held-down key of a computer keyboard can be used both to simplify existing user interactions and to implement new ones, that allow the rapid yet detailed navigation of multiple possible interaction outcomes. The second excursion shows that automated computational techniques can display shape specifically in the retinal afterimage, a well-known effect in the human visual system.

cs.HC↗

arXiv2025

Human noise at the fingertip: Positional (non)control under varying haptic $\times$ musical conditions (Appendices included)

As technologies and interfaces for the instrumental control of musical sound get ever better at tracking aspects of human position and motion in space, a fundamental problem emerges: Unintended or even counter-intentional control may result when humans themselves become a source of positional noise. A clear case of what is meant by this is the "stillness movement" of a body part occurring despite the simultaneous explicit intention for that body part to remain still. In this paper, we present the results of a randomized, controlled experiment investigating this phenomenon along a vertical axis relative to the human fingertip. The results include characterizations of both the spatial distribution and frequency distribution of the stillness movement observed. Also included are results indicating a possible role for constant forces and viscosities in reducing stillness movement amplitude, thereby potentially enabling the implementation of more positional control of musical sound within the same available spatial range. Importantly, the above is summarized in a form that is directly interpretable for anyone designing technologies, interactions, or performances that involve fingertip control of musical sound. Also, a complete data set of the experimental results is included in the separate Appendices to this paper, again in a format that is directly interpretable.

cs.HC↗

arXiv2021

One-press control: a tactile input method for pressure-sensitive computer keyboards

This work presents One-press control, a tactile input method for pressure-sensitive keyboards based on the detection and classification of pressing movements on the already held-down key. To seamlessly integrate the added control input with existing practices for ordinary computer keyboards, the redefined notion of virtual modifier keys is introduced. A number of application examples are given, especially to point out a potential for simplifying existing interactions by replacing modifier key combinations with single key presses. Also, a new class of interaction scenarios employing the technique is proposed, based on an interaction model named "What You Touch Is What You Get (WYTIWYG)". Here, the proposed tactile input method is used to navigate interaction options, get full previews of potential outcomes, and then either commit to one or abort altogether - all in the space of one key depress / release cycle. The results of user testing indicate some remaining implementation issues, as well as that the technique can be learned within about a quarter of an hour of hands-on operating practice time.

cs.HC↗

arXiv2021

Presenting the cyclotactor project

The cyclotactor is a novel platform for finger-based tactile interaction research. The operating principle is to track vertical fingerpad position above a freely approachable surface aperture, while directly projecting a force on the same fingerpad. The projected force can be specified in Newtons, with high temporal resolution. In combination with a relatively low overall latency between tactile input and output, this is used to work towards the ideal of instant programmable haptic feedback. This enables support for output across the continuum between static force levels and vibrotactile feedback, targeting both the kinesthetic and cutaneous senses of touch. The current state of the technology is described, and an overview of the research goals of the cyclotactor project is given.

cs.HC↗

arXiv2021

Kinetic surface friction rendering for interactive sonification: an initial exploration

Inspired by the role sound and friction play in interactions with everyday objects, this work aims to identify some of the ways in which kinetic surface friction rendering can complement interactive sonification controlled by movable objects. In order to do this, a tactile system is presented which implements a movable physical object with programmable friction. Important aspects of this system include the capacity to display high-resolution kinetic friction patterns, the ability to algorithmically define interactions directly in terms of physical units, and the complete integration of audio and tactile synthesis. A prototype interaction spatially mapping arbitrary 1D signal data on a surface and directly converting these to sound and friction during movements across the surface is described. The results of a pilot evaluation of this interaction indicate how kinetic surface friction rendering can be a means for giving dynamically created virtual objects for sonification a tangible presence. Some specific possible roles for movement input and friction output are identified, as well as issues to be considered when applying and further developing this type of haptic feedback in the context of interactive sonification.

cs.HC↗

arXiv2021

Ghostfinger: a novel platform for fully computational fingertip controllers

We present Ghostfinger, a technology for highly dynamic up/down fingertip haptics and control. The overall user experience offered by the technology can be described as that of tangibly and audibly interacting with a small hologram. More specifically, Ghostfinger implements automatic visualization of the dynamic instantiation/parametrization of algorithmic primitives that together determine the current haptic conditions for fingertip action. Some aspects of this visualization are visuospatial: A floating see-through cursor provides real-time, to-scale display of the fingerpad transducer, as it is being moved by the user. Simultaneously, each haptic primitive instance is represented by a floating block shape, type-colored, variably transparent, and possibly overlapping with other such block shapes. Further aspects of visualization are symbolic: Each instance is also represented by a type symbol, lighting up within a grid if the instance is providing output to the user. We discuss the system's user interface, programming interface, and potential applications. This is done from a general perspective that articulates and emphasizes the uniquely enabling role of the principle of computation in the implementation of new forms of instrumental control of musical sound. Beyond the currently presented technology, this also reflects more broadly on the role of Digital Musical Instruments (DMIs) in NIME.

cs.HC↗

arXiv2021

A tactile closed-loop device for musical interaction

This paper presents a device implementing a closed tactile loop for musical interaction, based on a small freely held magnet which serves as the medium for both input and output. The component parts as well as an example of its programmable behaviour are described.

cs.HC↗

arXiv2021

Developing the cyclotactor

This paper presents developments in the technology underlying the cyclotactor, a finger-based tactile I/O device for musical interaction. These include significant improvements both in the basic characteristics of tactile interaction and in the related (vibro)tactile sample rates, latencies, and timing precision. After presenting the new prototype's tactile output force landscape, some of the new possibilities for interaction are discussed, especially those for musical interaction with zero audio/tactile latency.

cs.HC↗

arXiv2021

Making grains tangible: microtouch for microsound

This paper proposes a new research direction for the large family of instrumental musical interfaces where sound is generated using digital granular synthesis, and where interaction and control involve the (fine) operation of stiff, flat contact surfaces. First, within a historical context, a general absence of, and clear need for, tangible output that is dynamically instantiated by the grain-generating process itself is identified. Second, to fill this gap, a concrete general approach is proposed based on the careful construction of non-vibratory and vibratory force pulses, in a one-to-one relationship with sonic grains. An informal pilot psychophysics experiment initiating the approach was conducted, which took into account the two main cases for applying forces to the human skin: perpendicular, and lateral. Initial results indicate that the force pulse approach can enable perceivably multidimensional, tangible display of the ongoing grain-generating process. Moreover, it was found that this can be made to meaningfully happen (in real time) in the same timescale of basic sonic grain generation. This is not a trivial property, and provides an important and positive fundament for further developing this type of enhanced display. It also leads to the exciting prospect of making arbitrary sonic grains actual physical manipulanda.

cs.HC↗

arXiv2021

The cyclotactor: towards a tactile platform for musical interaction

This paper reports on work in progress on a finger-based tactile I/O device for musical interaction. Central to the device is the ability to set up cyclical relationships between tactile input and output. A direct practical application of this to musical interaction is given, using the idea to multiplex two degrees of freedom on a single tactile loop.

cs.HC↗