-
Shared Minds Dimensionality (performance) Thesis Video in Dumbo New Portraits Synthetic Architectures Reconstructing Lost Spaces Web Art as Site
yilin ye
About
Portfolio
GitHub
yilin ye
About
Portfolio
GitHub
Week 7
Final Documentation—Virtual Cypher
Virtual Cypher
is an interactive installation that bring the essence of street & club dance into a virtual space, highlighting its core values of exchange and connection. Participants are invited to step into a one-on-one cypher with a virtual dancer, captured through Depth kit in Unity. Using real-time depth and body tracking from an Azure Kinect camera, the installation responds to participants’ movements, creating a dynamic dialogue of motion. Through this shared, interactive experience, Virtual Cypher explores how technology can carry the essence of street dance—communication, energy, and human connection—into new digital dimensions.
Club dance, especially house dance has deeply shaped both of us—our personalities, our outlook on life, and the ways we connect with others. It has been a source of joy and belonging, helping us feel part of a community, even as foreigners in a new country.
The idea for Virtual Cypher began with my original dream of creating a virtual dance platform. When I was physically away from my best friend who used to dance a lot with me, I wanted to virtually dance and cypher with her— that’s how we hang out. I wondered whether the essence of dance—the energy, emotion, and exchange—could still be felt in a virtual environment. Inspired by the work of Alan Winslow, we became curious about whether Depthkit could be the right tool to carry these intangible qualities into digital space. This curiosity led us to explore how interactive technology can preserve the core values of street dance and create new ways for people to connect.
workflow Recap:
1. Pre-production
- Brainstorm & Ideation, Storyboarding
- User/ Participant Expereince Design
2. Production & Post-production
- Playing around lighting
- Shooting Setup
- Import to Unity
- Integrate Azure Kinect to interact
- Chroma key, lighting and post-processing, in Unity
3. Prototype Presentation
- Projection & Kinect setup
Week 5
Continue Testing— interaction, chrome-key, etc.
We added lighting and post-processing in Unity. Originally, we wanted players to experience different levels. If they could follow along with the virtual dancer or interact with them, they would unlock new levels or dance sequences. However, we decided not to gamify the experience because it was difficult to seamlessly loop the video if a player failed to unlock the next level.
Therefore, we decided to focus on making the interaction feel natural, as if you were dancing with another person.
As shown below, the virtual dancer can move closer and further from you, turn to left or right, or even turn 360 degree, as you move in different direction.
In Unity, we detect shoulders and other different body parts, to achieve the interaction.
private void TransformPositionAndRotation() {
float targetCameraPosX = GetTargetPosX();
float targetCameraPosZ = GetTargetPosZ();
Vector3 targetPosition = new Vector3(targetCameraPosX, this.gameObject.transform.position.y, targetCameraPosZ);
this.gameObject.transform.position = Vector3.Lerp(this.gameObject.transform.position, targetPosition, moveSpeed * Time.deltaTime);
Quaternion targetRotation = Quaternion.Euler(0, this.GetTargetRotation(), 0);
this.gameObject.transform.rotation = Quaternion.Slerp(transform.rotation, targetRotation, moveSpeed * Time.deltaTime);
}
private void GetUserPosAndUpdateCameraPosition(ulong userId)
{
int headJoint = (int)KinectInterop.JointType.Head;
int shoulderLeft = (int)KinectInterop.JointType.ShoulderLeft;
int shoulderRight = (int)KinectInterop.JointType.ShoulderRight;
if (kinectManager.IsJointTracked(userId, headJoint))
{
Vector3 posHeadRaw = kinectManager.GetJointKinectPosition(userId, headJoint, false);
Vector3 posShoulderLeft = kinectManager.GetJointKinectPosition(userId, shoulderLeft, false);
Vector3 posShoulderRight = kinectManager.GetJointKinectPosition(userId, shoulderRight, false);
if (posHeadRaw != Vector3.zero && posShoulderLeft != Vector3.zero && posShoulderRight != Vector3.zero)
{
Vector2 posDepthHead = kinectManager.MapSpacePointToDepthCoords(sensorIndex, posHeadRaw);
ushort depthHead = kinectManager.GetDepthForPixel(sensorIndex, (int)posDepthHead.x, (int)posDepthHead.y);
float angleRad = Mathf.Atan2(posShoulderRight.y - posShoulderLeft.y, posShoulderRight.x - posShoulderLeft.x);
this.bodyRotation = angleRad * Mathf.Rad2Deg;
// For calibration
// if (depthHead < closestDistance) {
// closestDistance = depthHead;
// }
// if (depthHead > futherestDistance) {
// futherestDistance = depthHead;
// }
// if (posHeadRaw.x > mostLeft) {
// mostLeft = posHeadRaw.x;
// }
// if (posHeadRaw.x < mostRight) {
// mostRight = posHeadRaw.x;
// }
// Debug.Log($"closest distance: ${closestDistance}, futherest distance: ${futherestDistance}, most left: ${mostLeft}, most right: ${mostRight}");
// Debug.Log($"posHead: {posHeadRaw}, dPosHead: {posDepthHead}, depthHead: {depthHead}");
this.bodyDistance = depthHead;
this.bodyPosX = posHeadRaw.x;
this.TransformPositionAndRotation();
}
}
}
Because of baggy clothing, there are certain part from the depthkit scan that is not clean, and would show the background. To prepare for next shoot, in which we used greenscreen to cover the background, we also figured the chromekey effect, with (depthkit’s) shadergraph.
Week 4
Testing for Final Project:
We set up the blue and pink/ purple lighting to mock the set-up in the club: two blue lights in the front, and two pink/purple light in the back.
We picked a track, and recorded three clips that we intend to use:
1 minute of grooving, from just vibing to actually grooving; 1 minute of two steps; 1 minute of freestyle.
(track: Fox Tooth— Atjazz Galaxy Aart Dub)
We’re able to put the 3D scan from depth kit into unity on Haoren’s game engine.
Then we tried connecting kinect azure to Unity, so we could sense human body as well as distance betwee human and the object.
Response for the article, Ripping reality: Blind spots and wrecked data in 3D by Hito Steyerl
I found Steyerl’s persepctive very interesting — with the start of 3D scanning technology, how our interpretation of “surface”, “depth”, and even “image” could change. I also liked her discussion of how 3D scanning are fundamentally changing our relationship with images, reality, and (documentary) truth.
this part of the article seems particular interesting:
Georg Simmel introduced the idea that surfaces were not only an integral part of modern urban life but in a sense it´s condensation. This was in opposition to more traditional views of surfaces which connected them to mere appearance, inauthenticity and shallowness….
...Depth, in this evocation, acts as a synonym for cultural truth, authenticity, or as the locus for a better interpretation of events, as in many psychoanalytical accounts of the everyday. Equally, the metaphor of society as a smooth, flat surface is one that can give rise to a rather stultifying geography where space amounts to little more than a setting in which events take place, rather than as a source of animation and experience in and of itself
... 3D scanning thus does highlight the idea of the surface: by blending in actants, props, actions and forces indiscriminately
I also found the comparison that she drew interesting —— how image used to be mainly “representation” but with the uprising of images, it became its own architecture. The last paragraph calls back to the questions in the beginning:
What if [images] transformed into the objects they claim to represent?
... How would one create an indexical material replica of a situation? How does an image turn into dead stone?
while also being able to try volumetric capture ourselves for the first time this week, I found it also fascinating to see myself moving in a 3d way— being able to drag myself 360 degree when I was dancing was definitely a whole new experience. It’s also such a new perspective to see my dance. It even felt different than with motion capture technology, which is mainly capturing the movement data while not having the “surface”, meaning my appernace attahced. There is only the “depth” of movement in Mocap. Not even mention to the traditional 2D video, which is the most accesible technology for me to “document” myself dancing.
Seeing my 3D scan dancing on the screen almost feel like watching myself from an absolute third person persepctive, where I can ven manipulate my perspective all the time. (It feels absolutely different than moving perspective video too, since in this case my “image”// “surface” is attached to my body with the “depth” of movement).
I wonder, how does the concept of images having "bodies" change our relationship with traditional 2D representation?
How do we experience and feel about Benjamin's concept of aura, when we can create not only make 2D visual reproductions but also 3D material replicas from scanning and printing? How might the ability to physically reproduce objects change our understanding of truth in representation?
Week 2
Storyboard for Final Project:
Response for the article, In Defense of the Poor Image by Hito Steyerl
Mainly in defense, Steyerl presented the duality of "poor images" — images that are visually degraded, can be massively copied and circulated, and can be examined from several perspectives.
In the first section regarding "resolution", I was already intrigued, especially when she stated, "Resolution was fetishized as if its lack amounted to castration of the author." Especially from a commercial perspective in photography and film production, the higher the resolution, the more value the work naturally contains. As she continued to state, in contrast to the commercialization and commodification of cinema under neoliberalism and capitalism, poor images, often found in film essays and independent films, disappear into surfaces, becoming harder to be seen.
- "The poor image reveals the decline and degradation of the film essay, or indeed any experimental and non-commercial cinema, which in many places was made possible because the production of culture was considered a task of the state. Privatization of media production gradually grew more important than state-controlled/sponsored media production. But, on the other hand, the rampant privatization of intellectual content, along with online marketing and commodification, also enable piracy and appropriation; it gives rise to the circulation of poor images."
As poor images could be massively circulated through piracy, they became popular images.
- "While it enables the users' active participation in the creation and distribution of content, it also drafts them into production. Users become the editors, critics, translators, and (co-)authors of poor images."
This is where I found her argument most interesting: the poor images are not only about the content of the images themselves, but also the medium through which users contribute to the poor images; even simply by receiving the images and redistributing them, they become part of this circulation, changing the resolution of the images, etc., and creating bonds between "comrades".
- "The circulation of poor images thus initiates another chapter in the historical genealogy of nonconformist information circuits: Vertov's 'visual bonds,' the internationalist workers' pedagogies that Peter Weiss described in The Aesthetics of Resistance, the circuits of Third Cinema and Tricontinentalism, of non-aligned filmmaking and thinking."
For the most part, Steyerl argued that poor images have the power of decentralizing commodified film, democratizing the process of circulation, etc. While this essay was published in 2009, the poor images referenced by Steyerl, such as VHS tapes, are less prevalent nowadays than 14 years ago. Social media platforms gather users together in several massive platforms that seem more centralized than ever. When reading and imagining about low-quality images today, two examples I thought about immediately are memes and film cameras.
(Analog) Film photography became more popular in recent years, as more young people return to the nostalgic feeling of using second-hand film cameras. This somehow even became a "trend," and people are obsessed with not knowing how the picture would look like at the moment of taking the picture. The longer time span between taking a picture and seeing the final result seemingly slows down the capitalistic drive for productivity and immediate outcomes; the graininess of the picture, the lowered resolution, is also contrary to the "fetishized" high resolution. The film texture of the picture represents the artistic side of the author. It is an interesting phenomenon to witness the "comeback" of the "poor images".
Week 1
polycam scan