Iowa: A Lesson In How Not To Write Mission Critical Software

As I write this the country is still waiting on the results of the Iowa Democratic Caucuses -- the kick-off event for the Democratic Primary Process and the 2020 elections. The reason we still don't have a winner? A faulty app.

HuffPo has an in-depth article running down the "Buzzy Democratic Firm That Botched The Iowa Caucuses" but there are a couple of highlights from their analysis which stand out to me, as a software development professional.

State campaign finance records indicate the Iowa Democratic Party paid Shadow, a tech company that joined with ACRONYM last year, more than $60,000 for “website development” over two installments in November and December of last year. A Democratic source with knowledge of the process said those payments were for the app that caucus site leaders were supposed to use to upload the results at their locales. 

$60,000 sounds like a lot of money for an app but it's not. Iowa Democrats paid $30,000 a month for two months of work. For a development team consisting of a front-end engineer, a back-end engineer, a product/project manager, and a test-engineer that works out to annual salaries for the team of around $60,000 (assuming a 1.5x multiplier for full burden etc).

Shadow is based in New York City, Denver, and Seattle. $60,000 does not go far in those cities. At best, financially, that means junior level tallent was tasked with grinding out a mission critical application on a shoestring budget. That's a recipe for disaster.

This next snippet is from the New York Times "App Used To Tabulate Votes Is Said to Have Been Inadequately Tested."

The app that the Iowa Democratic Party commissioned to tabulate and report results from the caucuses on Monday was not properly tested at a statewide scale, said people who were briefed on the app by the state party.

Given a two month timeline it's no wonder the app was never tested at scale across the state. This is more of a process criticism than a development criticism but the importance of frequent feedback and real-world use really can't be overstated. A dry-run test of the application, even a few days before the caucus, would have allowed officials to familiarize themselves with the process. Moreover, these would have highlighted the app's difficulties with network unavailability and/or congestion.

Those turned out to be a major problem.

There were concerns that the app would malfunction in areas with poor connectivity, or because of high bandwidth use, such as when many people tried to use it at the same time.

Those concerns were born out last night. Multiple parts of the software development lifecycle exist to prevent exactly this problem and, by the look of it, none of them were followed here. Good user stories would have called out spotty cell service in rural areas. Early testing and user feedback should have shown that the application struggled to perform at scale. Load testing should have allowed the development team to model and address these concerns in the safety of a development environment.

Instead, an ill considered veil of secrecy was whipped away in the middle of the most important event in the Iowa political calendar.

As HuffPo noted:

The Iowa Democratic Party had refused to reveal details about the app, including the company behind it and what security measures were being taken to safeguard the results, arguing that it made the technology more vulnerable to hackers. 

Underfunded and under-wraps, it's no wonder this application failed. Without the money to build, test, and deploy a robust application and without the benefit of rapid, repeated feedback, errors and issues could do nothing but pile up. With those obstacles in the way the team at Shadow should be congratulated on getting the software out the door at all.

And to think, they could have just called the results in.