🕹️They needed a virtual world, so they built one

Go deep on the design process behind this public hub, makerspace, and presentation hall

This week, we’re giving the entire edition of the newsletter over to a group of artists and researchers from New York University: Aidan Nelson, August Luhrs, Billy Bennett, Lydia Jessup and Yuguang Zhang. What follows is their collective account of how they recreated their university facility as a live, 3D virtual space when the pandemic forced them into physical isolation. Things got strange quickly. Read on for their truly fascinating story about this one-of-a-kind digital space, and what they learned from trying to live in it. 

– Josh Kramer

NYU researchers created a virtual space for classes, hangouts, and much more
They took inspiration from one of the strangest public-access call-in shows ever
These zooming video cubes have yielded many insights for community and design

What happens when a global pandemic sends home a community of creative technologists? Well, in our case, at NYU’s Interactive Telecommunications Program (ITP), a new media graduate program (and its corresponding Interactive Media Arts (IMA) and IMA Low Res Programs), we created an interactive 3D world in virtual space to replace not only the social connections we lost, but the physical structures as well. 

What became clear in late March of 2020 to our group of ITP researchers was that despite the numerous tools the students, faculty, researchers, and staff were using to communicate with one another on a daily basis (Zoom, Slack, Discord, WhatsApp, Signal, SMS, etc.), our community felt farther apart than ever before. We found our digital selves splintering — Zoom fatigue, burnout, and finally, languishing. In response — and while under lockdown in our respective NYC apartments — we created a virtual place we called YORB

The name “YORB” (and its visual aesthetic) comes from a live interactive public-access television show run out of ITP in the mid 1990s called “Y(our) Orb,” started by Dan O’Sullivan, the current Chair of ITP. During this show, audience members called in (on a landline phone) to control the show’s camera as it looked out on a rendered 3D environment, revealing new areas to explore and things to do each week. This lineage was important to us in creating YORB 2020, since there is often a lack of context in the adoption of new technologies. While the materials of YORB have changed from a combination of computer graphics, broadcast television, and telephony to a suite of modern web technologies, the ethos of experimentation, learning, and remote connection is the same (for more about the original YORB, see this archive of videos).

For a school where the mediums range from electronics, to code, to physical materials, with YORB we wanted to explore the opportunities that emerge when we collaboratively and openly hand-build a digital community space. To make sure YORB would be an effort everyone could contribute to, we made the YORB code open-source and created a club devoted to its ongoing development. Located within a dynamic academic environment, we knew that for YORB to be a software for our specific community, it couldn’t simply be a tool to communicate remotely – it also had to be a place for exchange of both code and ideas. 

With the first prototype, we found ourselves back on a digital replica of our floor at 370 Jay St in Brooklyn, each of us represented as video cubes zooming around in three dimensions in a web browser. We could “walk around” and visit the (empty) classrooms, but most importantly, we could serendipitously run into friends, classmates and professors we hadn’t seen in weeks. 

Since its creation last year, YORB has become an invitation, a challenge, and an opportunity for our community to connect in new ways. We’ll share here how we have worked (often in a messy, circuitous way) to understand and address the gap between current video communications technology and the more natural and engaging feeling of sharing physical space with our community at NYU.

Making our community 3D 

ITP/IMA is a place where coding, circuit making, wood fabrication, 3D printing, virtual reality development and laser cutting happen all on the same floor of an NYU building. Learning here happens not only in classrooms but also through attending weekly peer-to-peer workshops, dreaming up collaborations over drinks at local bars, and helping to locate a bug in the code of the person sitting at the next table. Pre-pandemic, this rapid and interdisciplinary exchange happened at all times of day. Early in the morning over jazz and coffee, as well as over late-night snacks well past midnight, students could be found on the 4th floor, working huddled together.

How could we possibly move this community online? YORB allowed us to answer this question for ourselves, instead of working with an existing platform or service that at some level would determine, limit, and streamline our interactions. With YORB, we made a persistent space that could act as a public hub at any time of day (in any time zone), just like our physical counterpart did. It felt so powerful to not only decide what kind of space we wanted to have, but to be able to choose which parts of our virtual lives to bring into this new space. We took inspiration from other digital experiences (incorporating a “run” mode, for moving quickly around the space, as many video games do) and in some cases, we directly integrated features from other digital platforms (pulling in video streams from Twitch or photos from Instagram).

In a virtual 3D environment, almost anything is possible when real-world physics doesn't exist and materials don’t have the same monetary costs. The flipside of this is that even small design decisions can feel overwhelming and hugely consequential. When you enter YORB you are given a simple avatar – a 3D cube displaying your live video feed and broadcasting your microphone. You can move about the floor in any direction.

Gravity had to be included in the code, leading to otherworldly questions like, “how strong should gravity be?” We decided to embrace this by building in a “double jump” allowing you to jump higher than possible on earth.

As a shortcut to many of these difficult world-building questions, we let the familiar guide our design decisions. To decorate the 3D model of the physical floor, we chose digital textures that mimicked the industrial wood floors in our building. We made the default speed a walking pace, to mirror the social norms of our academic environment. And we chose logarithmic calculations that increased the volume of the sounds coming from other users the closer you were to them. This imparted a sense of closeness — even intimacy — to the experience, which many visitors cited as crucial.

These decisions made YORB familiar to all users, whether they were gamers or casual internet surfers. This design had worked for our small community over the course of the semester, but after the Winter Show at the end of 2020, which welcomed hundreds of attendees over two days, we realized this familiar visual language alone wasn’t enough. To make YORB a familiar and welcoming place, we created an optional onboarding tutorial that introduced users to the space and allowed them to practice the keyboard/mouse controls for navigating and interacting. We had to assume that some of the parents or grandparents of students showcasing their work in YORB had never played a first-person video game or ever navigated a virtual environment with an avatar.

To assist with the onboarding, we brought in Yorbie, a 3D model of a little yellow french bulldog puppy. Yorbie accompanied the pop-up instructions and gave users a cute target to look at, move to, and interact with throughout the tutorial — inspired by “Clippy” from the early days of Windows (and our collective childhood). 

Even with this effort, we face an ongoing technical and design challenge to allow many people who are unfamiliar with their computer’s settings, are not using the Chrome browser, or are behind a VPN (as was the case for many students in China), to fully participate in the space.

Designing a space to reflect all the ways our community connects, online and IRL

Through our community events, we began to discover what aspects of our IRL experience could be replicated online. One early event had only one attendee. We had decided to host a “Pizza Friday,” which previously had been a pre-pandemic weekly hangout that entailed standing around and eating pizza.

Despite changing the YORB floor tiles to look like pizza slices, we discovered that it wasn’t as fun without real pizza. We realized we needed to turn the blank space we had created into a real place by giving people roles to fill or activities to participate in.

Over the course of the following semester we set out to do so through a series of creative infrastructure experiments.

When it came time for the end of semester student shows, we wrestled with questions like: How might we display work online, outside of a typical 2D webpage or video conference? Could crowds browse collections of digital work online as they do in galleries, moving from work to work and chatting in response? The alternative was to have students present their work in a webinar or video call — an interaction everyone was tired of and would risk low attendance. But the stakes of building a DIY alternative for showcasing projects that students had spent weeks on, as well as welcoming guests, felt extremely high.

As an answer, we built “projection screens” throughout the space. Students could take over a projection screen with a click and present their own computer screen in the 3D space. Positioning their avatar next to their work while explaining their process was similar to the familiar experience of ITP student shows. In later developments, the size of the screen was greatly increased and moved out into the open “outdoor” field space where students hosted live performances, DJ sets, and movie nights, in lieu of the pre-pandemic weekly social events that took place at various bars and spaces throughout the Brooklyn area.

These designs caused familiar behavior to emerge in this new digital setting as students positioned their avatars in groups, shuffled to see the screen and get close to the music, or even find an out of the way place to get away from the crowd, which we were delighted to hear one student describe as “going out for a ciggy.” 

With this setup and other mini experiments throughout the school year, we aimed to create a basic infrastructure that would best support sharing and learning all in one space. Red Burns, the late founder of ITP, said, “The program’s focus on interactivity is asking us to reimagine the new technologies as the verb, not the noun.” What does it mean ‘to YORB?’

Beyond YORB

Red Burns also said: “Technology is not value-free. It takes on the values of its designers.” Throughout a year’s worth of building and rebuilding YORB for student showcases, performances, and spontaneous hangouts (as well as the accompanying server crashes, empty events, lost connections and broken video feeds), we’ve sought to address the needs and desires of our community with creativity, curiosity and humanity. As the novelty of our initial prototype wore off, some students became frustrated with the limitations of the experience, or the lack of particular features (i.e. text chat). We started to reimagine what role YORB might play in an ongoing hybrid community experience.

Ultimately, YORB has never been one thing — it’s a constant active process of building, maintaining, fixing, experimenting. As a platform, YORB was and is an incomplete solution for our robust IRL community. Even in the span of a year, design decisions made early on, (such as the creation of a virtual double of our IRL space), became out of touch with an evolving remote student community, many of whom had never stepped foot inside the physical building. As a process, however, YORB represents a commitment to finding more creative and human approaches to connecting with our community online.

What we’ve learned is that we’d like to see more multi-modal places of learning and teaching online that counter the engagement defaults of social online spaces today. The barrier between the physical and the virtual is thinner than ever.

As we design digital social spaces, we must leverage the affordances of these futuristic networks while also keeping in mind the frameworks that have supported human connection for centuries, interactions that have existed since we became "social" millennia ago — things like eye contact, mingling in groups, being able to feel yourself in a crowd of friends, being able to hear snippets of a conversation as you pass by. The future of digital social spaces is so exciting, because the further we go towards these new technological developments, the more we can weave back in the timeless behaviors and environments that make social connection possible in the first place.

This week’s newsletter written by:

Aidan Nelson, August Luhrs, Billy Bennett, Lydia Jessup and Yuguang Zhang, with editing from Josh Kramer

Images courtesy of the authors

New_ Public is a partnership between the Center for Media Engagement at the University of Texas, Austin, and the National Conference on Citizenship, and was incubated by New America.