April 1st,

Storytelling Through Data And VR


Interviewing Creative Technologists around the Globe

Storytelling Through Data And VR

Aaron Koblin / アーロン・コブリン

Storytelling with Data and VR

Ask a creative technologist

The theme of this interview series is about “creative output” and “technology.” By creative output, we define this as by someone transcribing one’s thought or emotion into something else which others can perceive. In that sense, TV spots and documentary movies, for example, are creative outputs. Technology is a tool for creative output. For example, technology such as filming equipment were used to describe what is like to fall in love as an creative output. In the same manner, technology such as musical instruments were used to describe the music that emerged within one’s head, as an creative output. At Dentsu Lab Tokyo, we call those who aim to provide creative output through the use of technology as “creative technologists.” Through the series of interviews, we will interview creative technologists worldwide and understand what kind of creative output comes forth through the use of technology. (This interview was conducted through online correspondence.)

As our first interview, we have Aaron Koblin the pioneer of data art. He is a media artist from the United States, and also an entrepreneur. At UCLA Design | Media Arts where he completed his MFA, he was making media art using data. Togo Kida, who is the member of Dentsu Lab Tokyo went to school with him. Aaron has lead Google Data Arts team from 2008 to 2015 as an creative director, and last year, he has become the CTO of VRSE, a VR company.

Using Data for Art.

TogoHello. Thanks for taking this interview.
Aaron: No problem. Hope you are well.
Togo I remember you had a strong interest during your MFA, and you made “The Sheep Market” as your thesis project.
Aaron: The reason for making “The Sheep Market” was that I came to question the value of human work after seeing thousands of people working like a machine on Mecanical Turk by Amazon.

The Sheep Market
Through the use of crowdsourcing, this art work gathers total of 10,000 sheep being drawn manually.

Ten Thousand Cents
This art work critically questions the meaning of work. A 100 dollar bill is divided into 10,000 pieces. Then, each of the pieces were put on Mechanical Turk to be drawn by 1 cent. The collective 10,000 cents amount of work, or 100 dollars, will replicate the original 100 dollar bill, but is not exactly the same as the original, and thus questions the inequality of the work itself and the compensation. Within the movie, one can also view the process of pieces being drawn. Some are meticulous, and others simply give up the work, and the viewer can see each of them.

Togo After joining Google, you lead the Google Data Arts team. Why were you interested in “data” to begin with? Why did you choose “data” as to make your artistic expression?

Wilderness Downtown
The place you were born becomes a music video. Once you enter your address at the website, a personalized music video will be generated which uses the street view data on Google Map.

Unnumbered Sparks
The color and the form of the net structure hung in mid-air changes when the users interact through their smartphones. Hundreds of viewers can cooperate and interact with the art work. Aaron mentions that he likes this artwork particularly because of the fact that people look at each other’s delighted faces, and not looking at their smartphones. This is another great example of which an art work using data interacts with people.

Aaron: We live in the age of data – so it’s natural for us to reflect on that, and to use data for cultural pursuits. We’ve always made art and found ways to communicate with each other — it never seemed strange to me that we would use data for these purposes.I’ve been very interested in ways that we can make sense of this world we live in, where so much of our lives are driven by things we can’t see or hear. I’m interested in ways that technology can enhance our humanity. We can extend our natural abilities and push our intelligence and emotions into other systems.

Amsterdam SMS
The total amount of SMS in Amsterdam gets visualized. By looking from the position axis, the viewer can understand which part of the city is populated, and by looking from the time axis, the viewer can understand how certain messages, “Happy New Year” for example, is being corresponded. Through the use of data, the viewer obtains a way to understand the city thoroughly.

New York Talk Exchange
Information that is exchanged among New York and other cities is visualized. For example, the viewer can instantly understand the amount of telephone being called increases from daytime to night. Being at Google Data Arts team, Aaron must have dealt with nation-wide amount of data, and is the very reason why he is an expert in imagining invididual’s life behind the data.

Flight Patterns
An artwork that visualizes the pathway of airplanes in the sky. By coloring over 10,000 pathways, the viewer can see when the sun rises in the west coast and how people swarm into New York. Through the use of data and technology, something that was not visible previously becomes evident. For Aaron, perhaps technology is one way to perceive the world.

Hacking One’s Senses.

TogoWith “data” being your expertise, why did you shift to “VR” and joined VRSE? What did this mean to you? What was the reason for you to make this transition?


Aaron: I’m very interested right now in the new technology supporting virtual reality — specifically the technologies which enable us to hack our senses — to convince ourselves that we’re experiencing something we’re not.For a very long time we’ve been hacking our ears. Closing our eyes we could almost convince ourselves that we’re places we’re not with sound and music. Now with VR we can combine audio with vision and sensors to sufficiently trick our minds into thinking we’re somewhere else – at least for a moment. For instance, there is “Waves of Grace” as an example.

Waves Of Grace
This is a monologue of Davis, who survived Ebola in 2013. By viewing Davis, the storyteller, one can feel as if one is viewing from Davis’s viewpoint. As opposed to a typical motion picture experience, where one is only able to view from a single point, this work successfully replicates the feeling of actually standing there.

The Displaced
This work is a story of children who had to evacuate their homes due to a war. The viewer can experience scenes such as food being dropped from the sky. By moving the viewer’s device, the viewer can see the sunlight of South Sudan, and even the building top of Ukraine, and makes the viewer feel as if viewer is standing at the actual place. Over 60 million people are being forced to leave their houses, and half of them are children.

I think this is a very powerful thing. It’s something we’ve been trying to do since the dawn of man — that is, allow one to walk in another’s shoes. Traditionally we’ve used our imaginations guided by words, pictures, etc.We can now eliminate this intermediate part of communication and convey an actual experience. I believe this will revolutionize how we communicate, and even how we think.

I’m very interested in the effects of VR on storytelling, I’m also thinking about interaction paradigms, and how this technology will check how we think about computers, data, and information.

Technology connects Art and People.

Togo: Can you briefly talk about what you do at Vrse?
Aaron: As founder and CTO my job is primarily about vision setting, team building, and product/engineering.
TogoWhat specifically are you doing?
Aaron: We’re doing a lot of testing and learning right now. Trying to best understand what’s needed to tell great stories in VR. Much of this is about understanding the structure of story, and how VR is different from any other medium.We’re thinking about things like interactivity and 3D space, but also basic things about how you communicate and what the role of the viewer should be from a story perspective.

TogoI see. I think when it comes to VR, it’s also important to realize from what viewpoint one captures the image. If the protagonist is a child, then the height of the viewpoint must be considered accordingly. I believe there is a decision whether the narrator comes into the experience or not.

NYT Mag VR: Walking New York
This work attempts to represent New York from the viewpoint of immigrants. This picture is taken from a helicopter above, and became the cover picture for Walking New York.

TogoYou, or VRSE, have worked with The New York Times, UN, and U2. What kind of people do you want to work together with?
Aaron: We want to make great immersive storytelling experiences in VR. To do so, we partner with people who are passionate artists and creators. Our aim is to make things that lots of people are excited by and connected with. We start with this vision and work towards ways to reach this goal.

U2 – Song for Someone – 360 Version
This is a music video where U2 performs together with musician all over the world.

TogoI think that your attempt to connect the viewer and the output through the use of technology has never changed, before or after the use of VR.
Aaron: For many years I’ve worked with Chris Milk. We’ve done many projects from crowdsourcing animations in The Johnny Cash Project, to interactive music videos in Arcade Fire’s Wilderness Downtown, and many more.

Johnny Cash Project
172カ国、25万人のファンが描いたミュージックビデオです。亡きJohnny Cashによる“Ain’t No Grave”のビデオの1370フレームを誰もがトレースできるように公開。集まった画をビデオにしました。ファンの持つJohnny Cashへの追悼の意が、ビデオ制作の参加という形で表現できます。トレースは、今もサイトを通じて可能です。

Building the Future of VR.

TogoDuring your time at Google Data Arts team, what was the process of creation like? Especially, in comparison to Vrse, is there any difference in process? If so, what is different?
Aaron: Yes, at DAT we were very free to experiment. There was a lot of trial and error and eventually we would release smaller experiments and a few bigger projects.

Much of what we’ve been doing so far at Vrse is experiment — but also building a real product and process. Our aim is to make a system for great VR production and distribution, so we’re definitely taking a different approach than I’m used to as an artist and hacker.

A collaborative work between Aaron and Chris Milk. Travel from the middle of a lake to the palm of a baby.

A horror experiment on VR. The viewer is fixed from a point, and explores a dark building. The viewer witnesses people strapped on a table screaming, and gets surrounded by ghosts.

Togo By looking at some of your latest and successful projects, it seems that there is a lot of collaborative work (i.e. Chris Milk). What is something you think it’s crucial upon collaborating with others? In what kind of role do you perform as one of the collaborators? How do you draw line upon you and your collaborator?
Aaron: Chris and I compliment each other well. We have some overlap in expertise and a large overlap in interests. Ultimately though we have a fair amount of unique skills that each of us can bring to the table and own as our own domain.Choosing a great collaborator is crucial in my opinion. It’s difficult to find someone with whom you can communicate openly with – that’s critical. Radical candor with an understanding that it’s coming from a good place. Working as a team.

TogoI see. If there is anything you want to mention about Vrse, or your work, to the Japanese audience, here is your moment!
Aaron: We’re incredibly excited about the future of Vrse. We have some grand ambitions and hopefully we’ll be able to help shape the future of VR.

I would encourage everyone to give VR a try and continue to check back as things evolve. It’s taken other mediums quite a while to mature, I think we’re seeing VR turn into something powerful very quickly.

TogoThank you very much! I’m looking really forward to see your future work.

[Post Note: Redefining the usage of technology]
When you talk about VR, many people only look at the technological side, but when you are talking with Aaron, having technology as a basic expertise, it was interesting to see how he kept on talking about storytelling, in other words, how the communication should be done. Aaron attempts to narrate a story that exists behind the data. Therefore, what’s probably important is not the technology but also what you want to convey using that.

In that sense, Aaron’s attitude is consistent. What he has been doing up to now is not about demonstrating technology, but to narrate by understanding the humanly aspect that lies within. Perhaps, that could be rephrased as “connecting with people through technology.” For example, “Ten Thousand Cents” and “The Sheep Market” would never have been realized unless there were 10,000 people who actually took part in the project. Therefore, Aaron does not think the finished product is the mere output, but rather understands that the process is equally important. In “The Sheep Market” the process was even sold.

Connecting with people through the use of technology also means to be interested in people who were involved in making the output. I think that still stays the same even by using VR. “Walking New York,” “The Displaced,” and “Clouds Over Sidra” were about narrating the world from different viewpoints of people.

Having storytelling as the center will eventually redefine the meaning and the usage of the technology which enables to expand the technology itself even more.

(Interview by Togo Kida / Written by Naruki Higashi)


Dentsu Lab Tokyo