The reelection of Barack Obama was won by people, not by software. But in a contest as close as last week's election, software may have given the Obama for America organization's people a tiny edge—making them by some measures more efficient, better connected, and more engaged than the competition.
That edge was provided by the work of a group of people unique in the history of presidential politics: Team Tech, a dedicated internal team of technology professionals who operated like an Internet startup, leveraging a combination of open source software, Web services, and cloud computing power. The result was the sort of numbers any startup would consider a success. As Scott VanDenPlas, the head of the Obama technology team's DevOps group, put it in a tweet:
4Gb/s, 10k requests per second, 2,000 nodes, 3 datacenters, 180TB and 8.5 billion requests. Design, deploy, dismantle in 583 days to elect the President. #madops
But the tech had not always worked so well. Four years ago, the Obama camp was in much the same position as Romney found himself in 2012, coming out of the primaries with only a short amount of time in which to pull together a national campaign. Despite having some people with tech expertise, the 2008 Obama campaign lacked an internal IT team, relying on vendors and field volunteers to pull much of the weight.
"One of the biggest problems in the last campaign was that you had all these people who are out in the field who are volunteering who start building their own versions of these rogue tools to do the same thing over and over again," said Clint Ecker, senior engineer for Obama for America (and an Ars Technica alum). Every field office assembled its own patchwork of tools using spreadsheets or a hacked Web application to track operations. They communicated over Google groups or simple e-mail lists. "It made it hard to keep everyone on the same page," he added.
Then there was the much-vaunted secret weapon, Project Houdini—a get-out-the-vote system that was supposed to revolutionize the Election Day ground game. Each voter in each swing-state voting precinct was assigned a numeric code; when poll watchers recorded the voters arriving, the watchers were supposed to dial in the code to Houdini's automated hotline. But the load on the hotline brought it down, and the campaign had to fail over either to texting or to calling the codes back into local field offices, where the data was re-entered into a webpage manually.
Houdini's database stayed up, and it still played a role in the Obama campaign's efforts on Election Day 2008. But it was clear that the system hadn't been ready for the wave of data it was supposed to handle. "2008 was the 'Jaws' moment," said Obama for America's Chief Technology Officer Harper Reed. "It was, 'Oh my God, we're going to need a bigger boat."
So they began to build one.
The Narwhal
To pull it off, the Obama team relied almost exclusively on Amazon's cloud computing services for computing and storage power. At its peak, the IT infrastructure for the Obama campaign took up "a significant amount of resources in AWS's Northern Virginia data center," said Ecker. "We actually had to start using beefier servers, because for a period of time we were buying up most of the available smaller Elastic Compute Cloud instance types in the East data center."
"Reed wanted to wire the campaign with its own application programming interface."
Atop Amazon's services, the Obama team built Narwhal—a set of services that acted as an interface to a single shared data store for all of the campaign's applications, making it possible to quickly develop new applications and to integrate existing ones into the campaign's system. Those apps include sophisticated analytics programs like Dreamcatcher, a tool developed to "microtarget" voters based on sentiments within text. And there's Dashboard, the "virtual field office" application that helped volunteers communicate and collaborate.
"Being able to decouple all the apps from each other [by using Narwhal] has such power," Harper Reed, the chief technology officer for the Obama campaign, told Ars. "It allowed us to scale each app individually and to share a lot of data between the apps, and it really saved us a lot of time." The resulting platform gave Obama for America tools that helped "force-multiply" volunteers, giving them organizational and communication tools that made the Obama "ground game" even more effective.
When Reed was brought onboard by Obama campaign Chief Integration and Innovation Officer Michael Slaby in June 2011, the choices Reed made were informed by the 2008 campaign—but also by the mentality of Internet startups.
"We knew we were going to go big," Reed said. "We needed to architect [the campaign's IT infrastructure] in such a way that it actually worked. And we also knew we were going to be resource-constrained—whether it was money or people, we knew we weren't going to have everything we wanted."
They did have the incumbent's advantage, though: time to prepare. "I don't think we would have been able to [build Narwhal] if we had to deal with the primaries," Reed said.
The architecture Reed envisioned had to scale up rapidly. It needed flexibility, allowing developers with any level of experience who joined the campaign to work in and be productive with whatever language they preferred, and it had to integrate with the systems of vendors (such as Blue State Digital and NGP VAN). It needed to handle the needs of volunteers in the field and in the campaign's "Team Digital" (responsible for the campaign's Web presence, social, and other digital media presence), while also feeding the "big data" machine of Team Data (the campaign's analytics department).
In other words, Reed wanted to wire the campaign with its own application programming interface (API). And, like many Internet startups, he looked to the cloud to make it scale.
"I looked at a lot of architectures," Reed said. "Coming out of [previous employer] Threadless, and having watched what Amazon and other large organizations had done—and how powerful these API platforms have become—it was obvious that the best path forward was to follow services oriented architecture (SOA) principles and build a services architecture that allowed for all of our apps to connect together and share one common data store."
But to build Reed's "bigger boat" meant adopting a startup-like strategy in more ways than just architecture. He needed to build a team within the Obama campaign that behaved like an Internet startup, a dedicated staff of engineers willing to work long hours for a rapid ramp-up to the ultimate payout: reelecting the president.
The team
Reed went out and recruited people who already knew the territory, snapping up both local talent (such as Ecker) and people from out of town with Internet bona fides—veterans from companies like Google, Facebook, Twitter, and TripIt.
"All these guys have had experience working in startups and experience in scaling apps from nothing to huge in really tight situations like we were in the campaign," Ecker said.
"When you put down the constraints you have, it's pretty easy to figure out who you're hiring," Reed added. "You're looking for engineers who understand APIs—engineers that spend a lot of time on the Internet building platforms. When you talk about SOA, they're not just going to laugh at you and say you're talking about Java."
Rather than focusing on creating something significantly new, Reed said, the team focused on taking what they already knew worked and fitting the pieces together.
"We aggressively stood on the shoulders of giants like Amazon, and used technology that was built by other people," he said. "We had a pretty good culture of using not-invented-here technologies. And we weren't scared about it."
In some cases, he added, it was just the opposite—if something came up as a solution and had never been used at that scale by someone else, it was "super scary" and usually avoided. The Democrats were going to play it conservative.
The cloud
The first recruits started rolling in soon after Reed joined, giving the team 583 days to do their work.
"It really became apparent that there needed to be a centralized place for all this data to flow into from all our vendors and for the new applications," Ecker said. "We needed a standard RESTful API that any project could talk to using any programming language and any Web framework, whatever people are most comfortable in so that they can hit the ground running."
"We aggressively stood on the shoulders of giants and used technology that was built by other people."
That meant Narwhal. Written in Python, the API side of Narwhal exposes data elements through standard HTTP requests. While it was designed to work on top of any data store, the Obama tech team relied on Amazon's MySQL-based Relational Database Service (RDS). The "snapshot" capability of RDS allowed images of databases to be dumped into Simple Storage Service (S3) instances without having to run backups.
Even with the rapidly growing sets of shared data, the Obama tech team was able to stick with RDS for the entire campaign—though it required some finesse.
"We definitely bumped up against some limitations with RDS but they were largely self-inflicted," Ecker said. "We were able to work around those and stretch how far we were able to take RDS. If the campaign had been longer, we would have definitely had to migrate to big EC2 boxes with MySQL on them instead." Not having to switch off RDS meant that the Obama campaign saved "a shitload of money" on hiring additional database administrators, Ecker added.
Ecker says the team also tested Amazon's DynamoDB "NoSQL" database when it was introduced. While it didn't replace the SQL-based RDS service as Narwhal's data store, it was pressed into service for some of the other parts of the campaign's infrastructure. In particular, it was used in conjunction with the campaign's social networking "get-out-the-vote" efforts.
The integration element of Narwhal was built largely using programs that run off Amazon's Simple Queue Service (SQS). It pulled in streams of data from NGP VAN's and Blue State Digital's applications, polling data providers, and many more, and handed them off to worker applications—which in turn stuffed the data into SQS queues for processing and conversion from the vendors' APIs. Another element of Narwhal that used SQS was its e-mail infrastructure for applications, using worker applications to process e-mails, storing them in S3 to pass them in bulk from one stage of handling to another.
Initially, Narwhal development was shared across all the engineers. As the team grew near the beginning of 2012, however, Narwhal development was broken into two groups—an API team that developed the interfaces required for the applications being developed in-house by the campaign, and an integration team that handled connecting the data streams from vendors' applications.
reader comments
189