By Kenneth Cukier and Viktor Mayer-Schönberger | 26 June 2014
Big-data analytics has been making a major impact on cities as far apart as New York and Baghdad, saving lives and money in the process. Now the latest stage in the information revolution looks set to transform the way we commission and deliver public services
Mike Flowers was a lawyer in the Manhattan district attorney’s office in the early 2000s, prosecuting everything from homicides to Wall Street crimes, when he made the shift to a plush corporate law firm. After a year of being bored behind a desk, he left that job, too.
Looking for something more meaningful, he decided to help rebuild war-torn Iraq. A friendly partner at his law firm made a few calls to people in high places, and the next thing Flowers knew, he was heading into the ‘Green Zone’ – the secure area for US troops in the centre of Baghdad – as part of the legal team for the trial of Saddam Hussein.
Most of his work turned out to be logistical, not legal. He was required to identify areas of suspected mass graves to know where to send investigators digging. He had to ferry witnesses into the Green Zone without getting them blown up by the many improvised explosive device (IED) attacks that were a grim daily reality.
Flowers noticed that the military treated these tasks as information problems, and that it was data that always came to the rescue. Intelligence analysts would combine field reports with details about the location, time and casualties of past IED attacks to predict the safest route for that day.
On his return to New York City a few years later, Flowers realised that the same data-driven methods that he’d seen used in Iraq demonstrated a more powerful way to combat crime than he’d ever had at his disposal as a prosecutor. He then joined a special task force that was given the job of crunching the numbers that might unmask the villains of the subprime mortgage scandal in 2009.
Their unit was so successful that Flowers became the city’s first ‘director of analytics’. His mission: to build a team of the best data scientists available, and then harness the city’s untapped troves of information to reap efficiencies covering everything and anything.
Among the first challenges his team tackled was ‘illegal conversions’ – the practice of cutting up a single dwelling into smaller units so that it can house ten times the number of people it was originally designed for. Such residences are not only major fire hazards, but also tend to be cauldrons of crime, drugs, disease and pest infestation. With tangled masses of extension cords snaking across the walls and hotplates sitting perilously on top of bedspreads, it is no wonder people packed this tightly into buildings regularly die in out-of-control blazes.
New York City gets roughly 25,000 complaints a year about building overcrowding, but it has only 200 inspectors to handle them. There seemed to be no good way to distinguish cases that were simply nuisances from ones that were poised to burst into flames. To Flowers and his ‘kids’ however, this looked like a problem that could best be solved with lots of data.
They started with a list of every property lot in the city – all 900,000 of them. Next, they poured in data sets from 19 different agencies indicating, for example, whether the building owner was delinquent in paying property taxes; if there had been foreclosure proceedings; and if anomalies in utilities usage or missed payments had led to any service cuts. They also fed in information about the type of building and when it was built, plus numbers of ambulance visits, crime rates and rodent complaints.
Then they compared this against five years of fire data ranked by severity and looked for correlations to generate a system that could predict which complaints should be investigated most urgently. When the first results came back in 2011, everyone was stunned.
Prior to the big-data analysis, inspectors followed up the complaints they deemed most dire, but they only found conditions severe enough to warrant a vacate order in 13% of cases. Now they were issuing vacate orders on more than 70% of the buildings they inspected. By indicating which buildings most needed their attention, big data improved their efficiency fivefold. Fires in overcrowded buildings are 15 times more likely than other fires to result in injury or death for firefighters, so the fire department loved it. Flowers and his kids looked like savants.
The experience of New York City’s analytical alchemists highlights many of the themes of big data. They used a gargantuan quantity of data, not just a tiny sample. The data was messy, such as location information or ambulance records, but that didn’t deter them.
In fact, the benefits of using more data outweighed the drawbacks of less pristine information. And the inklings of experts had to take a back seat to the data-driven approach. The programme dispensed with a reliance on causation in favour of correlation.
Big data is poised to reshape the way we live, work and think. Companies are adopting these techniques at breakneck speeds. But for public sector employees, the changes will be just as immense. It requires new skills and a new mindset on how problems can be tackled in novel ways by harnessing data. Accountants in particular have a role to play, since many of the most impressive techniques are ones that accountants and auditors have long used, since the profession is already steeped in data.
But the changes we face are even greater than those sparked by earlier epochal innovations that dramatically expanded the scope and scale of information and how it is used in society. Now the ground beneath our feet is shifting, and old certainties are being questioned. Big data requires fresh discussion of the nature of decision-making, destiny and justice.
A worldview we thought was made of causes is being challenged by a preponderance of correlations. And the possession of knowledge – which once meant an understanding of the past – is coming to mean an ability to predict the future.
Issues such as these are far more significant than the ones that first presented themselves when we prepared to exploit e-commerce, live with the internet, enter the computer age or take up the abacus. The idea that our quest to understand causes may be overrated – and that, in many cases, it may be more advantageous to eschew why in favour of what – suggests that the matters are fundamental to our society and our existence.
The challenges posed by big data may not have set answers, either. Rather, they are part of the timeless debate over our place in the universe and our search for meaning amid the hurly-burly of a chaotic, incomprehensible world.
Ultimately, big data marks the moment when the ‘information society’ finally fulfils the promise implied by its name – when data takes centre stage as the ruling force of our society. All those digital bits that we have gathered can now be harnessed in novel ways to serve new purposes and unlock new forms of value. But this requires a different way of thinking, and will challenge our institutions and even our sense of ourselves.
Kenneth Cukier is data editor of The Economist. Viktor Mayer-Schönberger is professor of internet governance and regulation at the Oxford Internet Institute in the UK. This article is adapted from their book Big data: a revolution that will transform how we live, work, and think (Houghton Mifflin Harcourt, 2013). Kenneth Cukier is speaking at the CIPFA annual conference on 1–3 July
