umbra: In the Digital Shadow

About

umbra: In the Digital Shadow is an audiovisual performance built as a real-time feedback loop between a performer with the sword-looking multimedia instrument Type-0《零式》 and the improvisation machine Aiii. Movement is captured, interpreted and responded — turning choreography and gestures into live sonic and visual improvisation.

In this performance, multiple depth cameras observe the performer’s body to form a haunting digital double made of point clouds. The double is reprojected onto a wide backdrop, and shifts between trace, echo and counterpart. The performance continuously records itself: the “twin” encounters its own previous instances, building an accumulating archive of embodied dialogue. Memory becomes an instrument: past gestures return as material for the next decisions.

Skeleton tracking and sensor data from Type-0《零式》feed into Aiii — an improvisation machine that breathes, learns, samples and responds. Real-time AI image synthesis is driven by the same embodied data stream, shaping the moment as a co-improviser. The work asks where agency sits when your gestures are answered by your own computational double. We challenge the roles of an instrument and the one that operates, speculating on the interdependency, expectations, reliability, and ownership.

Composition Structure

umbra is composed of 7 parts. Inspired by forms and structures of western classical music and opera, the composed timeline helps to narrate a progression of dialogue and interaction between the human performer and the improvisation machine while keeping the openness of improvisation, expression, and elaboration at the moment.

Act 1: Prelude — An introduction of space & bodies

Act 2: Development — A process from creation to overtake of the digital shadows

Act 3: Body as “Truth” — Revealing of the "true" physical entity

Act 4: Aiiiria - An Aria of the digital

Act 5: Encounter — An improvisational dialogue and conflict between the multitude of the reimagined twins

Act 6: A.i.r.A — Separation towards autonomy and trapping the shadow back into synergy

Act 7: Postlude — A resolution between the twin