Throughout the developed world, and at all levels of government, the latest explosive innovation linked to government use of IT is the rush to make government data open. Particularly in larger cities in the U.S. and Europe, state and local governments are finding more and more datasets never heretofore made public and are publishing these new sources of information that may be relevant to solving important social problems.
Cities such as New York and Chicago have already created open data portals where anyone can review and download data sets related to crime, public health, transportation, utilities and so on. At a recent conference on shaping our digital future held in Trieste, Italy, one of the plenary sessions was completely refocused at the last minute to cover open data and its revolutionary potential. And all of this heated exploration follows the Health Datapalooza organized by HHS in Washington last spring. There, over 1,600 developers and 240 companies showcased innovative new products related to health care that were derived from the much more extensive publication of federal data sets. In January, 2014, Stewards of Change Institute will host the first California Health and Human Services Open DataFest, in partnership with the Health Data Consortium, the California HealthCare Foundation.
What appears to be driving all of this activity is the hope that by publishing data collected by government but never before released to the public, innovators and entrepreneurs will create new products and services including applications that directly address significant problems that government has been unable to solve with conventional processes. One of the most notable results of all this energy is that organizations such as Code for America have teamed up with cities to organize hackathons that result in apps being built in record time. In these self-organizing sessions, cities have benefited from technologists finding ways to make open source products widely available, and built largely by volunteer labor.
There is little doubt that one of the key drivers of this new phenomenon has been the transformative efforts introduce by President Obama. As he entered the White House, President Obama was adamant about increasing transparency in government, and his commitment lead directly to the creation of the U.S. publication of www.data.gov. The site now provides access to thousands of datasets collected and cataloged by the U.S. federal government. The appearance of data.gov spurred similar efforts in states and cities, and was exciting enough to generate interest in the U.K. whichactually offers a “portal to go” for use by any government entity that wishes to create such a capability. The City of Buenos Aires has posted its code on-line for others to use. Rich and robust open data portals have been put on-line by the City of Chicago and New York City.
The White House focus on transparency and open data was a lot more than simple rhetoric. President Obama brought Beth Noveck to the administration as the Deputy CTO responsible for opening up data to the public and in effect she was in charge of open data initiatives. She took a can opener to the sealed mounds of government data and brought the data into the light. She remains a passionate defender of the potential of open data to improve government, as you can see in this presentation.
From a technology perspective, it is quite likely that impetus toward open data came from Tim Berners-Lee, inventor of the World Wide Web, undoubtedly helped fire up the world about open data and its potential in his proposals about linked data. He gave a very impressive Ted presentation in 2010 that shows the way to build mash-ups involving open data.
There are several leading options available to cities and states wanting to open up a portal to reveal data that might lead to innovative solutions. The apparent leader in an open data portal product is Socrata, selling a fully packaged and supported product to kickstart an open data portal with little or no development and lots of customization options. The most popular open source version of a portal for open data is CKAN. Both of these options have been deployed and are operating in a number of cities, states, and nations.
One reason that open data and corresponding portals appear to be so interesting is the tendency to believe that this kind of potential partnership between government and industry actually does create jobs and therefore does what many believe to be the most important work of government in these times. While it is hard to find any strong scientific evidence of the contribution of open data initiatives to creating jobs, we do know that growth of companies happens when new products are created, and the outcome of many of the open data initiatives has been to do just that. So if this revolution continues to expand, there should be a causal correlation with job growth and the improvement of economic conditions all over the globe.
Any time there is such a tsunami of innovation as has been evident in the fast spread of this force, there are shortcomings and issues with achieving a more rational methodology to guide such explosions. Observers have noted that open data offerings have taken hold so quickly that there are no useful ways to compare the data sets from city to city, or to gain any national insight from the publications of individual sites. In effect, there are no standards on which cities might find a common ground to publish data sets that could be aggregated. There are certainly advantages to building such standards for comparison purposes, but the stage of our current development is exciting just as it is unfolding, and the potential in this effort for much more productive partnerships between government and industry is alone worth our working toward a new way of doing business.