Game On: Columbia’s Interactive Arts and Media Department Is Creating Innovative Games “to Get Stuff Done”

This feature was published in the Winter/Spring issue of DEMO Magazine (PDF link)

“This is like the quarter-million-dollar table,” says Columbia professor David Gerding, sitting in front of a modest conference table outfitted with six laptops that a re connected to optical sensors. As Gerding speaks, an avatar of his face appears on the laptop screen before him. A click of a button, and beams of light appear to radiate from the eyes of his digital self. “It’s detecting the iris. Can you see the laser beams coming out of my eyes? It’s like Blade Runner!”

In a modest conference room in Columbia College’s Department of Interactive Arts and Media (IAM), we’ve stepped into what could pass as a set from the ’80s sci-fi classic.
It’s a futuristic space where individuals interact on screen as small sensors track their eye movements, pupil dilation, and even facial expressions, all in the service of teaching machines to understand how people communicate and collaborate with one another.*

Gerding is one of several visionaries in Columbia’s IAM department who are leveraging video game technology to create what he terms “constructive games,” or “games to get stuff done.” Gelding heads the project CONsTRUcT, an effort to merge video gaming with behavioral and computer science to quantify the ways in which individuals communicate–such as speech, body language, and facial movements-in order to facilitate more constructive interactions.

In a few months, the system will be used to analyze the speech and eye movements of four Columbia students as they interact in a meeting-like setting in Columbia’s Media Production Center.

Screen Shot 2014-07-17 at 4.37.00 PMBuilding on the success of the department’s High-Rise Evacuation Project—a 2007 collaboration with the City of Chicago to develop a video game that helps train first responders how to evacuate a building in a disaster–Gerding began work on CONSTRUCT. a two-phase project to teach machines to analyze the interactions among humans as they collaborate. The project is funded by a cooperative agreement between Columbia College and the U.S. Army Research Laboratory concerning Conglomerated Networked Telemetry to Raise Understanding of Collaborative Teams, or CONSTRUCT. Gerding explains that what this basically entails is conducting “human factor studies–how do people work together, how can they interact with each other?. The idea was they could deploy what we make and use it in their lab. We’re creating research tools for the broader research community.”

The first phase of the research took place largely within a video game space.
“We made the game in about a year and a half,” Gerding says. “We built a lab with five game-playing workstations set up, and five other workstations that were doing speech recognition in real time on the players. Everyone was wearing wireless headsets, and as they played the game. all the data was streaming to this database.”

The game is deceptively simple in its execution. Built on a software engine often used in first-person-shooter games, it places five players in a confined virtual room. But instead of shooting at one another, players must collaborate to succeed. The center of the virtual room contains a stack of colored blocks. A simple image is displayed on one wall. The players are given two rounds to reproduce the image using the blocks, working within 10- and 20-minute time limits to complete the task more quickly than other groups.

Gerding’s project team was able to pull reams of data from the avatars“ movements and points of visual focus, as well as from the conversations between players, which were captured using speech-recognition software. The data was then harvested into a custom piece of software Gerding has dubbed Timeline, which visualized the separate data streams side by side. Doctoral students from Chicago’s DePaul University used the software to label patterns of data as distinct forms of human behavior. These patterns, once identified by humans, can be taught to the software so that it can automatically recognize types of behavior in future trials. As Gerding explains. ”Instead of having a human being mark up the data. the machine can make those assertions based on the probability that a human would have marked that scope of data that way."

Once the Timeline system had been trained to recognize certain types of behavior, Gerding and his team began to introduce new forms of input and study interactions in the real world.
“In the new round of research.” Gerding says, “instead of only having five people playing a game, we’re going to have some number of those teams in other labs and four people sitting around a table of screens in the real world and acting in some kind of supervisory role.” While one group plays the game, another group will sit in the Blade Runner meeting room, advising the players over headsets while their own real-world interactions as a team are tracked.

“The gaze trackers will look at them and gather data,” Gerding explains. “There is useful data for humans in things like eye contact. and body language. The trick is getting machines to recognize those patterns. When the gaze trackers are all running together, we’ll get the data back as 3-D heads in a 3-D space. It will say, ‘Dave was looking at subject one. or subject two,’ and you can get that as data.” By re-creating these real-world interactions, the software can then apply the same algorithms already established in the video game world to determine which types of interaction best achieved the shared goal of the individuals.

Applying the Research

Gerding notes that his ultimate goal is to be able to integrate this technology and research into Columbia’s own programs, beginning with a special topics class slated for next fall. “We’re going to try and find what you would do if you were going to make a smart classroom.” he says. “What kinds of data would you try and collect?”

When Gerding speaks of smart classrooms, he’s not talking about rooms outfitted with computers and a projector. “Imagine if we had a traditional classroom learning environment where everyone’s facing a board,” he says. “But we have gazetracking sensors on the students, and I. as the professor, get the sensor data in my glasses, so when I go up to the beard and am drawing. I can see data showing which students are looking at which part of the board. What can we do to make the environments literally facilitate the goal of that room. which in this case would be learning?. Gaze tracking could wake up the student who is falling asleep before the professor has to. Or the room will connect to the learning management system, and as the professor is lecturing, related documents are automatically being pulled up on a screen based on keywords that the professor is saying.”*

Gerding is clearly energized by the prospect of opening up these high-end technologies and theoretical concepts to Columbia students, providing them with opportunities to think about and experience media and sensory-laden environments. The equipment involved with CONSTRUCT and the earlier HighRise Evaculation Project provided the foundation for the motion capture studio in Columbia’s new Media Production Center, which goes online this semester. “The students will get the chance to try these different, cool toys,” notes Gerding, who also sees possibilities to apply this research to media criticism and analysis, presenting students with a method of quantifying types of human behavior to better understand how players react to student-developed game worlds, or how directors respond to film shot on the Media Production Center soundstages.

The CONSTRUCT project exemplifies the possibilities that Doreen Bartoni.
dean of Columbia’s School of Media Arts. envisions for students as a result of successful efforts to bring increased research dollars to the college. “Dave has taken his role as a researcher very seriously,” she says. “His mind is amazingly facile. He is innovative. He is a wonderful combination of being a creative individual who also has the technical expertise to be able to create a product. He’s a total package in terms of bringing his department and the school to the next level. I feel very confident when he’s involved in a project that he will be able to have deliverables that not only satisfy the project but also have other applications.”

Creating Games, Creating Change

Screen Shot 2014-07-17 at 4.37.13 PMAt Columbia, this sort of odginal thought about gaming and collaboration is not confined to Gerding’s work with CONSTRUCT, however. Work being done by Mindy Faber, another innovative thinker, demonstrates that these ideas and approaches are part of the ethos of the IAM department. Faber is the founder of the nonprofit Open Youth Networks, which she launched in 2007 to explore how video games and social media could empower women and urban youth.

“1 was interested in figuring out ways to train youth in participatory culture and media, and to introduce them to Web 2.0 tools that could empower them to better their own lives and communities,” Faber says.

Faber knew the chair of the IAM department, Annette Barbier. and made her aware of the project suggesting a partnership between the nonprofit and the college. When Faber joined the IAM department as academic manager in 2009. she and Barbier developed a strategy for turning Open Youth Networks into a program within the department working on projects with urban youth based on new and emerging technologies and games.

One of Faber’s first projects at Columbia was Games for Change, a High School Summer Institute course she taught in the summer of 2009 with Emily Kuehn, an adjunct professor. “We introduced youth to the idea of creating games that have a social-issue message.” she explains. "We showed them examples and led them through a fascinating curriculum showing bow values can be built into the mechanics of a game itself–not so much through the content of the game, but what you’re doing inside the game. For example, are you cooperating with others, or are you trying to undermine them? Are you trying to build collaboration among teams of people, or are you trying to just go all out for yourself?

“In that class, our students came up with an idea for a game called Surviving High School,” Faber continues. “]hey broke up into four teams, and each team worked on one year of high school. Then they combined that into one game about how to survive. What are the moral and ethical issues high school kids deal with every day as they navigate their adolescent years? It was great. I never would have been able to come up with that idea. and in the process they’re basically computer programming, which is a really high-level skill.”

]This semester. Faber will involve Columbia students in community outreach through a course she’s working on with Andrew Hicks, an IAM professor, called Interactive Arts Media Team. “The class has a client, and the client comes to them with a problem that requires an interactive media-arts solution,” she explains.

“The class is charged with trying to figure out how to solve this problem using emerging technologies and interactive media.”

The first client chosen for the course is the Chicago Youth Voices Network, a nonprofit that is seeking assistance with an effort called the Youth 2.0 Recovery Reporting Project. Faber’s students will collaborate with 12 Chicago youth media organizations to design an interactive solution to the problem the client has posed, which involves creating Facebook apps, quizzes, and social media tools to aggregate polling data about how youth are faring during the economic recovery. “They’re aplying their technology skills to address real community issues.” says Faber.

Screen Shot 2014-07-17 at 4.37.22 PM“We’re trying to figure out how we can create opportunities for our college students in Interactive Arts and Media and Game Arts in a mentoring/ tutoring/service-learning capacity, in which they’re applying their skills to collaborate in a community-based setting with high school students around social issues or ideas of using technology to create a better world.”

Barbier believes Faber’s work is serving an important role in Columbia’s community outreach efforts and in the development of the culture of the department. “What we’re finding is that there’s not a whole lot of diversity in the kind of students who are coming to us for an education in game design,” Barbier says. “We would like to diversify the character of the program by bringing in a broader range of voices and backgrounds. One of the reasons that we wanted to do this was as a recruitment and development effort-recruitment in the sense that we would like to bring young people from various backgrounds into the game program, particularly young women, don’t generally feel empowered to make games.”

Faber is particularly concerned with issues of women in gaming. Despite recent Nielsen and BBC studies suggesting that more than half of gamers are women, the stereotype that women don’t play games has led to a stunning gender divide in the industry. “We hope to prototype and build games that girls are actually in control of–online games. Facebook games, Wii games that teach selfdefense for women,” Faber says.

“Our goal is to form collaborative relationships with high schools through which we can create an incubator where girls can come in and be trained as game designers, and be asked, ’What do you want to play and build?” And then our IAM students can build and prototype these games, put them in the marketplace, and begin changing things up." To that end, Faber is currently organizing the 3G Summit: The Future of Girls.

Gaming and Gender, an exposition planned for August 2010 at Columbia, in which 50 urban teenage girls will collaborate with faculty and women game designers over three intensive days to build gender-inclusive game prototypes for the future.

As Barbier notes, the current conception of what gaming is and can be is very limited. “There’s a narrow bandwidth for what’s considered appropriate for games, so that’s one of the areas we’d like to develop into–to give voice to other reasons for games to exist, other strategies and stories to tell.”

While Faber’s and Gerding’s work differs widely in execution, both are committed to reimagining video games and social media to tum these technologies into tools that engender fruitful collaborations–ensuring that Columbia will play an important role in the ongoing evolution of games and electronic media.

Photos by Drew Reynolds, a photographer based in Chicago and Los Angeles.