As part of the fruit of the ALA GODORT State and Local Documents Task Force's State Agency Databases Across the Fifty States project, I used the project blog to create a listing of state-level campaign finance databases.

So far I've got nine states: Alaska, Alabama, Colorado, District of Columbia, Hawaii, Illinois, Missouri, Nevada and Ohio. Do you know of other state campaign finance databases? Either leave a comment below of drop me a line at dnlcornwall AT alaska DOT net.

And if you use any of the databases listed above, I'd really love to see your comments on the project blog entry for that database.

At the Alaska State Library, we recently completed a barcoding project which finally let us put all of our manual shelflist items into our catalog for our patrons to find. This also meant that our holdings went onto Open WorldCat for others to find.

I'm happy to report that we've had a 7% increase in checkouts of federal documents compared to the previous fiscal year. I'm sure the cataloging project was responsible because the rate of increase for documents checkouts outperformed other parts of the collection.

Since the project was only completed in the fiscal year that ended on June 30th, I expect to see more growth in documents checkouts in the coming year.

There are many ways to make open a tangible collection to the world. Good cataloging is a start!

Here is a pre-print (not-final version) of a paper with fascinating ideas about distribution of government information:

• Robinson, David, Yu, Harlan, Zeller, William P and Felten, Edward W, "Government Data and the Invisible Hand" (2008). Yale Journal of Law & Technology, Vol. 11, 2008.

They say that "the federal government must reimagine its role as an information provider" and more specifically, that the next administration should...

...reduce the federal role in presenting important government information to citizens. Today, government bodies consider their own websites to be a higher priority than technical infrastructures that open up their data for others to use. We argue that this understanding is a mistake. It would be preferable for government to understand providing reusable data, rather than providing websites, as the core of its online publishing responsibility.

While the paper does not address preservation and long term access explicitly, it does suggest that the government should provide a "permanent location" with a permanent URL for "each piece of government data." It also implies (I think) that something like LOCKSS will ensure authenticity and permanent access ("As long as there is vigorous competition between third party sites, we expect most citizens will be able to ?nd a site provider they trust.") I believe that oversimplifies the problem and relies too much on hope and not enough on a social commitment to preservation through public funding of memory organizations.

Thanks and a tip of the hat to Joshua Taubere (GovTrack.us) for pointing to this article. He describes and comments on the paper in a post on the Open House Project blog: (Government Data and the Invisible Hand June 6th, 2008 by Joshua Tauberer).

Today, some mixed good news/bad news about the availability of free public data in the UK. As we've noted here before (e.g., Privatized Data Woes in Britain and News from abroad: UK open statutes & RFID in Canadian coins and The Semantic Web + Government Information = Serendipitous Reuse) the British government sells limited-use licences to its GIS data on a cost recovery basis. Now, as part of a proposed national geoportal, the UK would "create a single point of entry on the web to data held by public bodies such as local councils, Ordnance Survey (OS), the British Geological Survey and the Environment Agency." But, as the story says, "A new system will make geospatial information available without charge - yet we'll still have to pay."

• An Inspired debate on access, by Michael Cross, The Guardian, May 22 2008.

First, some very good news. Civil servants revealed last week that the British government has begun work on a system to make all the geospatial data it holds on the natural environment available for free inspection and re-use. Now the bad news. In this context, "free" means we will still have to pay to download much key data, especially if it is to be published or otherwise used commercially.

On May 9, this past Friday, the White House issued a memorandum doing away with the old category of "Sensitive But Unclassified" and replacing it with "Controlled Unclassified Information."

The memo is titled, "Memorandum For The Heads Of Executive Departments And Agencies - SUBJECT: Designation and Sharing of Controlled Unclassified Information (CUI)".

Here is how the memo defines CUI (bolding mine):

a. "Controlled Unclassified Information" is a categorical designation that refers to unclassified information that does not meet the standards for National Security Classification under Executive Order 12958, as amended, but is (i) pertinent to the national interests of the United States or to the important interests of entities outside the Federal Government, and (ii) under law or policy requires protection from unauthorized disclosure, special handling safeguards, or prescribed limits on exchange or dissemination. Henceforth, the designation CUI replaces "Sensitive But Unclassified" (SBU).

This is such a broad brush that it could be used to exempt anything from public disclosure. By definition, isn't anything the government produces at least somewhat pertinent to the national interests of the United States? Otherwise, why waste taxpayer dollars and government time creating the product?

Then there's the allowance for "policy." I'm sure that DoD's 2002 domestic propaganda program probably "required protection" from "unauthorized disclosure" for "policy" reasons.

This is a definition that appears to be designed to give the executive branch total control over what information is released by the US Government regardless of existing law. It is an order that should be challenged by Congress today with no funding permitted for its implementation. Assuming the President doesn't back down, it should be one of the first items revoked by the incoming President.

Government information items should be properly classified and protected, safeguarded when containing individuals personal data, or public record. No twilight zone of "It's unclassified, by I can't give it to you for reasons of government policy."

Erik Ringmar, professor of social and cultural studies at the National Chiao Tung University, Hsinchu, Taiwan, wants others to join him in putting restricted government documents on the web.

I say this is awesome! There's certainly precedent for this kind of activism: Jared Benedict liberated a bunch of USGS maps and just last week, I uploaded the Iraqi Perspectives Report to the Internet Archive. Anyone else out there set free a government document? Leave us a comment.

So, I've taken it upon myself to start an organisation called MLOP, the "Movement for the Liberation of Old Papers". What I do is hack into restricted websites, download the documents I'm interested in, and then use my favourite open-source paint program to remove the copyright statements from each page. Next I assemble the pages into one single pdf file and upload it to the Internet Archive, where it will become universally available to both researchers and citizens. Yes, it does take a bit of time, but it's a very worthy cause (and I have a hardworking research assistant to help me).

I feel strongly about this, and I'm prepared to live with the legal consequences of my actions. This, after all, is the new frontier of civil rights - the right of access to information. How else can corruption be stopped and falsehoods exposed? How else can people in power be held accountable? I'd go to prison for the old parliamentary papers if I had to. Ever after I would proudly brag about having liberated an old House of Commons report from the clutches of market capitalism.

While FGI normally focuses on US government information policy issues, there is a conflict going on in the UK that mirrors some of the recent stories about public data being used by private companies in a privileged way, forcing the taxpayer to pay twice for their data.

An April 17, 2008 Guardian article titled A costly 2008 Domesday Book details how not one, but two British agencies contracted with commercial companies to post government compiled data. The result:

After seven years of legal wrangling, an official, complete and constantly updated list of addresses in England and Wales is about to become available for commercial use. The National Land and Property Gazetteer (NLPG), compiled from data supplied by local councils, is being promoted as the best list of property addresses since the Domesday Book.

Free data it is not. Although prices have yet to be finalised, the commercial firm hosting the service said this week it will cost between £15,000 and £20,000 a year. Profits will be shared among local authorities to help them keep data up to date.

The gazetteer is not the only address database on the market. The state-owned Ordnance Survey also offers addresses as part of its MasterMap digital geographical database of Britain.

Most of the article is about campaigns to free the data. In analyzing the roadblocks, they talk about issues that will be familiar to US readers:

"We would like to give it away free," says Nicholson. However, he says, local authorities are not going to give their work away when they have to pay for the use of postcodes from the Royal Mail's Postcode Address File. Neither can Ordnance Survey, which is required by the Treasury to show a return on its activities, and regards MasterMap as a key part of its revenue-generating portfolio.

We wish the Free Our Data campaign well.

This is probably a good to time to mention that what FGI objects to isn't the selling of data per se, but the selling of data that has already been compiled at taxpayer expense. If a private company wanted to raise its own venture capital, compile its own address list completely independent of government sources, we'd be all for it charging whatever the market could bear. But a private entity should not be allowed to be the sole, fee-based dispenser of information that has been compiled by government agencies using money confiscated through taxation. THAT's what we're against.

Well...this is a good sign. The EPA wants to know "what kind of environmental information you need, and how you want to get it". It's part of the National Dialogue on Access to Environmental Information and the EPA wants your input. You can contribute to their discussion board or submit a comment. I'm forwarding this link along to the Environmental Sciences Dept. faculty here where I work, among others. Spread the word!

A few weeks ago, Daniel had a great post, "GAO/Thomson-West Contract Raises Questions" in which he expanded on a Boing Boing post "Did the US gov't sell exclusive access to its legislative history to Thomson West?" and analyzed the Thompson-West contract with the GAO for digitizing 20,597 legislative histories of most public laws from 1915-1995. Today, Carl Malamud got an answer to his FOIA request to the GAO seeking access to the digitized images of those legislative histories. I'll let Carl tell it in his own words:

Well, the answer is now a definitive yes, that data has been sold down the river and is out to sea.
Public.Resource.Org sent in a FOIA request to GAO on this topic seeking access to the scanned data. Today's letter answering our FOIA request spells out the bad news. Turns out the GAO doesn't even get the data, they simply are given an account on Thomson's service. The rest of the government doesn't get access to this data, and the public is invited to stop by the GAO headquarters and pay 20 cents per page to copy paper.

This is one of those deals where the public domain got sold off ... GAO gets a bit of convenience by having their stuff scanned for them, but they gave up way more than they got in the deal, and the public (including government workers and public interest groups who need to consult this data) lost big-time.

Carl has put up his paper trail explaining the story. Here's the link to the Scribd group with the full paper trail on this issue, and here's the link to last week's response from the GAO.

This perfectly exemplifies the problems we see with government agencies entering into contracts with private companies to digitize public domain materials (see for example "NARA/TGN contract as a bad precedent"). We have no problem with government agencies contracting with private companies to digitize government information. The problem as we see it is that so many agencies seem ignorant of the fact that privatizing access to said digitized public domain information actually limits access in the long run.

Thanks to an alert from a dedicated but shy reader, our attention has been focused on a story on boing-boing titled Did the US gov't sell exclusive access to its legislative history to Thomson West? This story has links to documents relating to this deal requested by the redoubtable Carl Malamud. I took the time to read/skim through the contract documents and found this interesting section:

Taken from "Attachment A, Statement of Work" from the contract between Thomson West and GAO, posted at http://www.scribd.com/doc/2299358/Contract-Between-Thomson-West-and-GAO

Background: Since its inception in 1921, the US Government Accountability Office has compiled 20,597 legislative histories of most public laws from 1915-1995. These histories, spanning the 64th-104th Congresses, are currently being used onsite in the GAO headquarters Law Library in paper or microfiche format by GAO staff. On rare occasions other federal government employees are allowed onsite access to the paper or microfiche copies of these histories. Because of its historical and research value the legislative history collection shall be digitized to preserve the integrity of the files and improve the searchability of this valuable information resource.

Two years ago, GAO began a pilot project to convert a small number of GAO legislative histories from paper and microfiche formats to digital format. Since then 243 histories have been digitized using in-house resources and will be made accessible to GAO staff only through a web-based database on the GAO Intranet. The 243 histories consisting of 1,214,438 pages were randomly selected and include some of the largest histories in the collection. These histories shall also be re-scanned as part of this digitization contract.

This sounds like a major goldmine of information that really hasn't been shared with other parts of the government, let alone the public. It also sounds like GAO tried to do some of this work on its own but found it unviable. So left to itself, the information wouldn't contained in the paper files wouldn't be available to anybody. So I'm not surprised it went looking for a partner. But I am surprised and concerned that they went with a commercial partner when the GAO office is within driving distance of a number of major universities and when public-spirited organizations like the Internet Archive and Public Resource might have been happy to come up with a solution to provide this taxpayer-funded information at zero cost to the taxpayers and either zero or minimal costs to GAO. Conceivably there might have been some way for the Government Printing Office to incorporate this into GPO Access, although that certainly would have been at some cost to GAO unless Congress was willing to make an appropriation for this purpose. But any Congress that claims to be committed to strong public access should be willing.

Were alternatives to in-house digitization or wholesale privatization pursued? If not, why not?

Long time readers of FGI know that most government information is considered public domain and also subject to Freedom of Information Act requests. So what's to stop Carl, Internet Archive, or some other public minded group from exposing this rich trove of legislative histories to the public which were taxpayer funded to begin with? According to the GAO, plenty:

Taken from "Attachment A, Statement of Work" from the contract between Thomson West and GAO, posted at http://www.scribd.com/doc/2299358/Contract-Between-Thomson-West-and-GAO

FOIA Requirements: While GAO is not subject to the Freedom of Information Act (FOIA), GAO has regulations (4 CFR Part 81) that follow the spirit of FOIA. The paper or microfiche copies of the legislative histories (and possibly the PDF copies of the "GAO Materials" section) would be available for public inspection and copying. However, under GAO's public disclosure regulations, GAO charges a per page copy fee. Accordingly, any extensive copying would be expensive and the quality of the copies, for many of the histories would be poor.

I assume this was put into the contract to assure Thomson-West their investment would be secure from public-access zealots who have the idea that the American people should only be charged once instead of twice for government information. But the paragraph raises two important questions that I hope someone in Congress will ask GAO:

1) On what rational basis would you charge a per-page fee on the 1,214,438 pages that have already been digitized? Running a backup tape isn't the same as hand copying files. GAO should be directed to immediately release that database at zero cost unless they can carefully and believably document actual copying expenses including staff time. But a per page copy for PDF files isn't credible.

2) When GAO says "quality of the copies, for many of the histories would be poor", are they saying that the quality of copies would be poor just for the public or for Thomson-West as well? The first reading suggests a deliberate effort to sabotage no-fee public access, while the second reading suggests that Thomson-West customers will be paying a high price for lousy duplication. Neither option seems particularly fair.

I think I speak for all of us at FGI when I say that while digitization for greater access is a laudable goal, wholesale privatization without a careful, public examination of other, more citizen-friendly, alternatives is not acceptable. If you agree, please ask your Members of Congress to direct GAO to take a second look at this contract and facilitate no-fee access to this valuable set of legal materials.

Lawrence Lessig gave a lecture at the National Press Club on Thursday and introduced a plan designed to increase congressional transparency via the launch of Change-Congress.org.

The website provides a venue for data on earmarks, campaign financing, etc. and advocates an end to corporate and private interests. It organizes citizens to push candidates to make four commitments: No money from lobbyists or PACs, vote to end earmarks, support publicly-financed campaigns, and support reform to increase Congressional transparency.

Change-Congress.org has ambitious goals, that's for sure. Will it work? Greater transparency is needed, and this initiative will certainly help in that effort. But can "big money" every truly go away? Nevertheless, it is a noble effort, and I urge you to join their cause, take the pledge, and volunteer your efforts in contacting candidates to take the pledge!

Monday, March 17: American University's Washington College of Law's Collaboration on Government Secrecy hosted its first Freedom of Information Day, featuring panels and speakers addressing new FOIA legislation, the state secrets privilege, and transparency. They also presented the first "Robert Vaughn FOIA Legend Award" to Thomas M. Susman, a government openness advocate and initial drafter of the 1974 FOIA Amendments.

Tuesday, March 18: Associated Press President and CEO Tom Curley discussed Freedom of Information and other open government issues during a Sunshine Week dinner event at The National Press Club. The speech looked ahead to priorities in the new administration, an update to Curley's 2004 Hays-Enterprise Lecture, which many view as a defining moment in moving forward efforts to preserve and protect access to information. The full text of Curley's speech can be found at the Sunshine Week website.

Wednesday, March 19: OpenTheGovernment.org, among others, sponsored the 3rd annual Sunshine Week National Dialogue on Open Government and Secrecy. This year's panel discussions focused on "Government Secrecy: Censoring Your Right to Know". The webcast is archived and will also be available soon at OpenTheGovernment.org and the event will be on a DVD available for purchase. They also compiled a list of legislation and resources about government secrecy and related issues.

The First Amendment Center posted the full-text of the 2008 National FOI Day Conference's keynote speech, "A New Balancing Test: How Excessive Classification Undermines National Security" by J. William Leonard, former chief of the Information Security Oversight Office.

Leonard quipped that his remarks on government secrecy would be his most candid, "a sort of ‘Leonard Unplugged’ if you will for those of you into the MTV scene". He discussed instances of excessive secrecy that produced serious consequences, including the decision to go to war in Iraq, stating, "Secrecy comes at a price - sometimes a deadly price - often through its impact upon the decision-making process".

He also proposed a new way for government officials to determine whether information needs to be classified in the interest of national security; what he calls the "New Balancing Test":

"We are long familiar with what many regard as the “traditional” balancing test of national security versus openness – of secrecy versus transparency. Instead, the balancing test of which I talk is more along the lines of national security versus national security; i.e. what will cause greater damage to national security, the disclosing or withholding of specific information".

We hadn't added much to our Best. Titles. Ever. humor page lately. I'm happy to end that dry spell with a document that is both humorously titled and useful:

Hills Bros. coffee can chronology : field guide, published by U.S. Dept. of the Interior, Bureau of Land Management in 2006.

Why does the Bureau of Land Management care about what coffee cans looked like in the 1910s? For a very good reason. According to the document's introduction, Hill's Brothers Coffee cans are a great way to date digs dating back to the late 1800s because of the tendency of Hills Brothers to change their can designs every so often.

I'd love to copy and paste their explanation into this post, but I can't. The BLM authors chose to lock their PDF into a form that cannot be copied from. You can make copies of the entire file and you can print pages from it, but you can't copy and paste the text nor can you extract the pictures from it. Yet as a public domain government document, there is no legal reason to impose these kind of restrictions. This is part of the future we fear, one of crippled electronic documents that aren't as reusable as they could be. Today BLM has decided we can't copy and paste from a public domain document. Maybe another agency will decide tomorrow that we shouldn't be able to print their document. That's what faces us unless the federal government has a consistent policy that renounces Digital Rights Management (DRM).

**Addition by James: I've attached a PDF of the document from which I was able to copy and paste. Please download this copy and leave a comment if you're *not* able to copy and paste.
**Addition by Daniel: Thanks for the demonstrating the power of a polite request. It's nice to see responsive and helpful gov't agencies.
**Further addition by James: While I believe in the power of a polite request, this one was Jim working his magic to subvert the copy-blocking. He saved the original pdf, printed/saved as pdf (macs let you convert to pdf from the print menu!), jiggered a few things and then the DRM was foiled. That's the PDF doc that is attached to this discussion :-)

Today I was searching the shelves of my depository for a report related to the Catalog of Federal Domestic Assistance and instead my eyes fell on this publication from the CIA:

Donovan and the CIA : a history of the establishment of the Central Intelligence Agency
by Thomas F Troy
Publisher: [Langley, Va.?] : Central Intelligence Agency, Center for the Study of Intelligence, 1981.
SuDoc: PREX 3.10:D71

This is the official story of how the CIA came to be. According to the preface, it was written in 1975 as an account of the agency's origin for new employees. It was promptly stamped SECRET. In 1981, the agency released a declassified edition which the preface hails as being "released for leisurely reading outside the office, and printed in one volume, this history should better serve it's original purpose."

What made the CIA decide it could let its secret origin story out of the shadows? According to the preface, it was the excision of no more than six typewritten pages. Obviously, I haven't seen the missing six pages, but doesn't this case make it seem like the classification process is a little whimsical?

The book is divided into three sections: "The COI (Coordinator of Information) Story" which covers from about 1929 to World War II, Wartime--The OSS Story and Postwar--The CIA Story, which ends with the establishment of the CIA. If anyone's read this volume, let us know what you think.

As far as I can tell, the book has no electronic version. Google Books has a commercial version with no preview. Perhaps one day the government document version, which is in the public domain, will be posted. Might be interesting to run a tag cloud on the text.

The electronic world has given us many gifts, but I'm not sure it could deliver an experience of serendipity like the printed shelf did for me today. From welfare to spies.

On a personal note, thanks to everyone who has sent words of appreciation to me regarding my being named as a Library Journal 2008 Mover and Shaker. I really think this is an award that in a real sense belongs to the entire documents community and especially to my co-volunteers here at FGI and the dozens of folks working on State Agency Databases Across the Fifty States. Thanks to each and every one of you who contributes to finding, describing, and making government information available to people in a way that respects their privacy. Without you I'd just be a mouth on legs.