NextGov reports on the challenges of turning raw government data into commercial products:

• Turning government data into private sector products is complicated business, By Joseph Marks, NextGov (02/09/2012).

  "The theory behind Data.gov was, let's move forward when it comes to sharing data," says Josh Green, chief executive officer of Panjiva, a company that crunches customs data for U.S. businesses that import some of their raw materials. "I think that's right in terms of what would be good for entrepreneurship, but realistically I don't think that has filtered down to the agency level." While Panjiva relies on some Census data, which it downloads directly from the Census Bureau, the company uses mostly Customs and Border Protection data on CD-ROMs that it pays to have delivered every day by FedEx.

  ...Data.gov is laudable, Rossmeissl says, but developers' biggest hurdle with government data isn't finding it, but getting it quickly and in a form they can use. "That wasn't the focus of Data.gov and, in general, it isn't the focus of agencies producing data," he says. "That's not because their intentions aren't great, but they have a history of producing data in a very specific way that goes back to the Federal Register and quarterly releases."

  ...The Data.gov team also meets regularly with about 400 agency "data stewards" to change the way government data is initially created so that it requires less translation and reformatting on the back end.

See also:

• The Federal Government Must Reimagine Its Role As An Information Provider.
• Private Sector Supports Public Data.

White House Begins Open Sourcing Data.gov, By J. Nicholas Hoover, Information Week (December 05, 2011).

The Obama administration has begun to open source pieces of the Data.gov platform and plans to launch a full-scale open source project early next year. This open data platform--called Data.gov-in-a-box--will allow other governments to easily stand up their own versions of Data.gov.

Data.gov developer and General Services Administration software architect Chris Musialek last Wednesday posted to open source development site Github some early test code for what appears to be a database management system and Web app that will serve as key pieces of Data.gov-in-a-box....

Launched just two years ago as a site for users to find and download "bulk" data sets from federal government agencies, Data.Gov is re-launching a "next generation" version with datasets that are "interactive" meaning you can search, filter, and explore data on the fly with your web browser. "You can also create charts and maps, and APIs are available for developers."

• http://explore.data.gov/
• Next Generation Datasets
• Introducing the New Data.Gov Platform. Socrata [a Seattle-based startup that has quietly been helping cities and states around the country to get their data online.] "Discover The New Cloud-Based Data.Gov Platform In This 10-Minute Video Tour."
• Data.gov relaunches as a cloud-based open data platform, by Alex Howard, Gov20.govfresh (May 16, 2011)

I'm in 2 minds about yesterday's announcement by Congressman Darrell Issa (R-Calif.), chairman of the Oversight and Government Reform Committee. On the one hand, I'm heartened by the fact that he stood up for the importance of the e-government fund and sites like data.gov and USAspending.gov and understood the importance of "governmentwide collection and governmentwide disclosure." On the other hand, I really worry about commercialized aggregations of data replacing free govt aggregations (statistical abstract, etc.) if the "pretty part" is not fully funded. If "governmentwide collection and disclosure" is not fully funded, what non-govt entity/company will be able to do that? Remember, according to the Census 2012 budget request, base funding for the Statistical Abstract = 24 FTE staff and $2.9 million (p.79). Data collection, aggregation, analysis, access and preservation cost money and these things won't be done by the "invisible hand."

And with that cheery thought, I'd again exhort our readers to help save the Statistical Abstract!

Issa pledges to keep transparency sites online
Federal News Radio
April 13, 2011

Congressman Darrell Issa (R-Calif.), chairman of the Oversight and Government Reform Committee, said Wednesday the $27 million cut to the E-Government Fund doesn't have to be the end of the sites such as Data.gov, USAspending.gov and many others.

"It was one of the things I begged my partners on the other side of the aisle to come back and say there is a price to pay to keep this up," Issa said during a panel discussion hosted by the Association of Government Accountants. "We will find a way, and this is a personal pledge, to make sure they are not shutdown. The specific funding goes away but reprogramming authority would still be available. Our view is on a case-by-case basis we will be able to keep them open."

Sunlight reports on the House bill that will slash funding for major government data sharing and transparency projects. Noting that the funding for these programs is only a few million dollars, Daniel Shuman says, "The returns from these e-government initiatives in terms of transparency are priceless."

• Budget Technopocalypse: Proposed Congressional Budgets Slash Funding for Data Transparency, by Daniel Schuman, Sunlight Foundation Blog (March 23, 2011).

  Data.gov, USASpending.gov, and other Obama tech innovations face virtual extinction if the FY 2011 budget bill passed by the House of Representatives in February or considered by the Senate in March becomes law.

It is not just the Statistical Abstract and related compilations that we are in danger of losing due to budget cuts (see The demise of the Statistical Abstract and other critical Census titles). Budget cuts are aimed at some of the most basic government information programs.

In such a climate, how can we rely on GPO, FDsys, NARA, and government agencies as our sole source of the government information that is released? We need this information in our digital depository libraries so that our communities can decide what is essential for long term preservation and access!

Bringing government up to data. by Abby Phillip and Kim Hart. Politico (7/20/10)

Their goal: Create government websites that are more like an Apple app store than the Department of Motor Vehicles. And for Vivek Kundra, Jeffrey Zients and Aneesh Chopra, that means trying to turn Obama’s vision of data-driven and digital government into reality.

Data.gov To House New APIs, By Dawn Lim, TechInsider (06/21/10).

A series of new application programming interfaces - tools that facilitate interaction between datasets and other software programs - will make it easier for developers to play and interact with the content on Data.gov, the online repository of federal information and a cornerstone of the open government initiative.

But those are just the preliminary steps to establishing a self-running ecosystem that will convert raw government data into valuable content and interesting applications, a White House technology expert said last week at a government IT forum.

Data.gov's next big thing: Mashing up federal stats with maps, By Aliya Sternstein, NextGov (06/18/2010).

Within the next month, data.gov will offer the public a chance to preview a so-called viewer that will let them combine many of the 270,000 data sets posted on Data.gov with maps.

White House to tie together mapping and data sites, By Aliya Sternstein NextGov, 05/28/2010.

"The White House has contracted with a major developer of mapping software to merge a federal website that publishes geospatial information with Data.gov, the government's depot for downloadable data sets, the company's president said on Thursday.

"California-based ESRI began last summer tying Data.gov to Geodata.gov, the geospatial information gateway, said company President Jack Dangermond in an interview with Nextgov."

How researchers enhanced Data.gov using semantic technology, by Sean Gallagher, GCN, May 18, 2010.

A three-person team at Rensselaer Polytechnic Institute, however, has demonstrated how one approach can make greater use of the massive sets of data available on Data.gov, using the power of the semantic web. The conversion project has shown how quickly and inexpensively visualization and mash-up applications can be built from government data when it’s put into a web-friendly form.