In Mother Jones, Will Potter profiles Ryan Shapiro, a punk rocker-turned-PhD student who wanted to study how the FBI monitors animal-rights activists. Through trial and error, and a lot of digging, he devised a perfectly legal, highly effective strategy to unearth sensitive documents from the bureau's 'byzantine' filing system. So now the FBI is petitioning the United States District Court in Washington, DC, to prevent the release of 350,000 pages of documents he's after. If the court buys the FBI's argument here, it could make it harder for scholars and journalists to keep tabs on federal agencies.
Meet the Punk Rocker Who Can Liberate Your FBI File. By Will Potter. Mother Jones. Wed Nov. 13, 2013
According to the Justice Department, this tattooed activist-turned-academic is the FBI's "most prolific" Freedom of Information Act requester—filing, during one period in 2011, upward of two documents requests a day. In the course of his doctoral work, which examines how the FBI monitors and investigates protesters, Shapiro has developed a novel, legal, and highly effective approach to mining the agency's records. Which is why the government is petitioning the United States District Court in Washington, DC, to prevent the release of 350,000 pages of documents he's after.
According to the American Assn of Law Libraries (AALL) "blawg:"
During last week’s Open Government Partnership (OGP) meeting in London, the Obama administration released a preview of its U.S. Open Government National Action Plan 2.0 (NAP). While the second NAP will not be finalized until December 2013, six new commitments to further advance the goals of transparency and accountability in the federal government were announced:
- Expand Open Data
- Modernize the Freedom of Information Act (FOIA)
- Increase Fiscal Transparency
- Increase Corporate Transparency
- Advance Citizen Engagement and Empowerment
- More Effectively Manage Public Resources
This is great news for open government (though it's still troubling how the administration is walking a very thin, troubling line in re to the NSA and their attacks on whistleblowers). I hope the administration and policy makers on open government will take some cues from our 2010 Letter to Deputy CTO Noveck: "Open Government Publications".
Back in May, we posted about the Obama Administration's executive order on open government data and cited Govtrack.us's Josh Tauberer's analysis of the Executive Order as missing the mark and being confused -- if not downright misleading -- about "open licensing."
The White House has issued a new Executive Order on open data:
- Making Open and Machine Readable the New Default for Government Information. EXECUTIVE ORDER, May 09, 2013.
To promote continued job growth, Government efficiency, and the social good that can be gained from opening Government data to the public, the default state of new and modernized Government information resources shall be open and machine readable. Government information shall be managed as an asset throughout its life cycle to promote interoperability and openness, and, wherever possible and legally permissible, to ensure that data are released to the public in ways that make the data easy to find, accessible, and usable. In making this the new default state, executive departments and agencies (agencies) shall ensure that they safeguard individual privacy, confidentiality, and national security. [emphasis added]
- Open Data Policy-Managing Information as an Asset. Memorandum For The Heads Of Executive Departments And Agencies M-13-13, Office of Management and Budget (May 9, 2013). [pdf. 12 pages]
- Landmark Steps to Liberate Open Data. by Todd Park and Steve VanRoekel White House Blog (May 09, 2013)
John Wonderlich at the Sunlight Foundation has an excellent analysis and commentary:
- Open Data Executive Order Shows Path Forward, by John Wonderlich, Sunlight Foundation Blog (May 9, 2013).
[T]he new policies take on one of the most important, trickiest questions that these policies face -- how can we reset the default to openness when there is so much data? How can we take on managing and releasing all the government's data, or as much as possible, without negotiating over every dataset the government has?
How can the public (or policymakers) request what they don't know exists? How can CIOs manage what they haven't surveyed?
...Today's Executive Order demonstrates a new approach to open data, moving beyond rhetoric and aspiration, requiring agencies to publicly report on what data can be made public, building a new backbone for federal open data policy, and setting an example for other governments to follow. [emphasis added]
- New Open Data Memorandum almost defines open data, misses mark with open licenses. by Joshua Tauberer (May 9th, 2013).
- President Obama’s New E.O.: Open Data, Not Government Transparency by Jim Harper, Cato Institute (May 9, 2013).
Sunlight Foundation's OpenGov Champion of the month is Sandra Moscoso. Sandra is a mom of two public school students in Washington DC, and a member of the Capitol Hill Public School Parent Organization (CHPSPO) -- oh and she just happens to manage an open data portal at the World Bank’s financial sector.
...she and other CHPSPO members were able to collect data to show how the schools that had a full time librarian had better test score results than those who had lost theirs due to budget cuts. The group was able to use that figure as an effective basis for their request to the city to restore funding for librarians.
Man, this week is Sunshine-week-alicious! The Sunlight Foundation has long advocated for -- along with FGI, library- and open govt organizations -- the free public access to Congressional Research Service (CRS) reports. CRS Reports are commonly not available to the public as CRS has this arcane and outdated rule that CRS reports are privileged communication between Congress and CRS. But CRS reports ARE available randomly online and Proquest, Penny Hill Press and other commercial publishers have long published them for a fee (I've even heard that CRS subscribes to Proquest to get access to their own reports historically!).
But this all may change. According to the Sunlight Blog, Representatives Leonard Lance (R-NJ) and Mike Quigley (D-IL) have reintroduced the bipartisan House Resolution 110 "Public Access to Congressional Research Service Reports Resolution of 2013" (text not received by GPO yet so not publicly available on Thomas). The Resolution would direct the Clerk of the House of Representatives to provide members of the public with Internet access to certain Congressional Research Service publications. Easy-peasy right?!
More than 30 organizations -- including Sunlight Foundation and FGI -- have signed on to a letter supporting the resolution. Please consider contacting your Representative and ask them to support H.Res. 110!
[UPDATE 4/2/13: We've had some questions about the meaning of "ALL." Please read the comment thread for clarification. We don't mean "records" (which fall under FOIA) and we don't mean classified information. We mean public domain documents, publications, reports, data, statistics and the like. JRJ]
A convergence of several things -- the White House's new policy on Open Access to federally funded scientific information, the NAPA Report on the GPO, the CASSANDRA Letter to the Public Printer, and Sunshine Week among them -- has led us to create a petition on the White House's We the People petition site. If you believe in free permanent public access to authentic government information, we hope you'll sign the petition and forward on to all your friends and social networks to help us reach our goal of 100,000 signatures by April 11, 2013! Thanks in advance!!
WE PETITION THE OBAMA ADMINISTRATION TO:
Require free online permanent public access to ALL federal government information and publications.
1. Assure that GPO has the funds to continue to maintain and develop the Federal Digital System (FDsys).
2. Raise ALL Congressional, Executive & Judicial branch information, publications & data to the level of federally funded scientific information & publish ALL government information as "Open Access."
3. Mandate the free permanent public access to other Federal information currently maintained in fee-based databases - including the Public Access to Court Electronic Records (PACER), the National Technical Reports Library (NTRL), & USA Trade Online.
4. Establish an interagency, govt-wide strategy to manage the entire lifecycle of digital government information w/ FDLP Libraries - publication, access, usability, bulk download, long-term preservation, standards & metadata.
The National Academy of Public Administration (NAPA) completed an operational review of the Government Printing Office (GPO) mandated under the 2012 Consolidated Appropriations Act (Public Law 112-74). The NAPA report, “Rebooting the Government Printing Office: Keeping America Informed in the Digital Age,” acknowledged the obligation Congress has to establish an interagency government-wide strategy to manage the lifecycle of digital government information. The report also acknowledged the vital role GPO plays in providing free permanent public access to authentic government information in tangible formats through its Federal Depository Library Program (FDLP) and to authentic government information in electronic formats via GPO’s Federal Digital System (FDSys).
However, Recommendation 4 states: “GPO and Congress should explore alternative funding models for the Federal Digital System in order to ensure a stable and sufficient funding source.” Among the models recommended are “…reimbursement for services; fees for end users; dedicated appropriations; and/or an automatic charge to agencies, depending on size, to encourage agencies to take advantage of GPO’s existing infrastructure and cover the cost of the services being provided by GPO.”
Just as the Obama Administration supports the public’s right to “free access over the Internet to scientific journal articles arising from taxpayer-funded research,” the Administration must support the creation of “stable and sufficient funding” to ensure free permanent public access to authentic government information arising from the work of taxpayer-funded Executive, Congressional, and Judicial Branch agencies.
- NAPA report, “Rebooting the Government Printing Office: Keeping America Informed in the Digital Age.”
- CASSANDRA Letter to US Public Printer in response to the NAPA Report.
- Expanding Public Access to the Results of Federally Funded Research. John P. Holdren, Director of the White House Office of Science and Technology Policy (OSTP).
- White House response to "We The People" petition "Increasing Public Access to the Results of Scientific Research"
- Government Accountability Office (GAO), Information Management: National Technical Information Service's Dissemination of Technical Reports Needs Congressional Attention. GAO-13-99, November 19, 2012. Context on the GAO report from FGI.
- GPO's Federal Digital System (FDsys): http://fdsys.gov
- PACER: http://www.pacer.gov
- National Technical Reports Library (NTRL): http://ntrl.ntis.gov
- USAtrade: https://www.usatradeonline.gov
- Federal Depository Library Program (FDLP). http://fdlp.gov
The Sunlight Foundation just put out their Open Legislative Data Report Card. California received a D grade :-| Find out how your state is doing. Below is the methodology that they used to grade state legislatures.
Each state was evaluated in six categories based largely on the Ten Principles For Opening Up Government Information. Each score is based on at least two members of staff and a volunteer during our state survey. Additionally, state legislatures were contacted (unless noted in their score) to ensure that our information on bulk data availability and timeliness was as accurate as possible.
The specific criteria for each category are as follows:
We evaluated each state on the data collected by Open States: bills, legislators, committees, votes and events. We also took note if a state went above and beyond to provide this information and other relevant contextual information such as supporting documents, legislative journals and schedules. Points were deducted for missing data, often roll call votes.
- 0 State provides full breadth of legislative artifacts Open States collects: bills, legislators, votes, and committees.
- -1 State does not provide stand-alone roll call votes.
Legislative information is most relevant when it happens, and many states are publishing information in real time. Unfortunately, there are also states where updates are more infrequent and showing up days after a legislative action took place. States were dinged if data took more than 48 hours to go online.
- 1 Multiple updates throughout the day, real time or as close to it as systems will allow.
- 0 Site updates once or twice daily, typically at the end of the legislative day.
- -1 Updates take longer than 24 hours to appear on the site, often up to a week.
EASE OF ACCESS
For many sites, the Open States team wrote scrapers to collect legislative information from the website code—a slow, tedious and error prone process. We collected data faster and more reliably when data was provided in a machine-readable format such as XML, JSON, CSV or via bulk downloads. If a state posted PDF image files or scanned documents, it received the lowest score possible.
- 2 Essentially all data can be found in machine-readable formats.
- 1 Lots of data in machine readable format but substantial portions that still required scraping HTML.
- 0 No machine readable data but standard screen scraping techniques applied.
- -2 Site had information that was unaccessible to Open States due to use of scanned PDFs.
USE OF COMMONLY OWNED STANDARDS
Because our ability to access most of a state’s data is represented by the above “Machine Readability” metric, we decided to use this provision to measure how a state made their bill text available. Making text available in HTML or PDF is the norm, and was considered an acceptable commonly owned standard (PDFs are a commonly owned standard, but it would be certainly nice to see alternative options where bill text is only available via PDF). States that only make documents available in Microsoft Word or Wordperfect formats require an individual to purchase expensive software or rely on free alternatives that may not preserve the correct formatting. It is worth noting, all states except for two met the common criteria of providing HTML and/or PDF only, one state (Kansas) went above and beyond and another (Kentucky) did not even meet this threshold.
- 1 State made an effort to go above and beyond.
- 0 State provided bills in PDF and/or HTML format and nothing better (plaintext, ODT, etc.).
- -1 State only provided bills in a proprietary format.
Many states move or remove information when a new session starts, much to the dismay of citizens seeking information on old proposals and researchers that may have cited a link (e.g. http://somelegislature.gov/HB1 vs http://somelegislature.gov/2011/HB1) only to see it point to a different bill in the following session. Tim Berners-Lee, inventor of the World Wide Web, wrote an article declaring Cool URIs Don’t Change and we agree.
This poses a particular challenge to us since every page on OpenStates.org points to the page we collected data from, but if a state changes their site then users lose the ability to check us against the original source. Most (but not all) states are good about at least preserving bill information, but few were equally as good about preserving information about out-of-office legislators and historical committees, equally important parts of the legislative process.
- 2 All information is avaialble in a permanent location and data goes back a reasonable amount of time (a decade or so).
- 1 Almost all information has a permanent location but a single data set doesn't. (Or a recent change to the site has wiped out historical links but information appears to be preservable going forward.)
- 0 Legislator & committee information lacks a permanent location (such as committees and legislators) but most is acceptable.
- -1 Ability to link to old information is badly damaged and and/or there is less than a decade of historical information.
- -2 Vital information like bills or versions lack a permanent location.
The great technology publisher O'Reilly is making its Open Government book files available for free for anyone to download, read and share. The files are posted on the O’Reilly Media GitHub account as PDF, Mobi, and EPUB files for now.
- We're releasing the files for O'Reilly's Open Government book: A #PDFtribute to Aaron Swartz (announcement) by Laurel Ruma, O'Reilly Radar (January 18, 2013).
- Open Government: Collaboration, Transparency, and Participation in Practice, Edited By Daniel Lathrop, Laurel Ruma, Foreward by Don Tapscott. O'Reilly Media (February 2010).
Be sure to check out Chapter 25, "When Is Transparency Useful?" by Aaron Swartz.
Shinjoung and I were stunned when we heard the news early yesterday morning that our friend -- and supreme friend of libraries and the Internet! -- Aaron Swartz left this world late friday evening. Aaron was deeply committed to and passionate about internet freedom and making information and knowledge as available as possible. To those ends, he worked on many projects large and small in his short but influential life. He was 26.
The *many* heartfelt remembrances from communities as diverse as journalism, law and open source tech -- witness Rick Perlstein, Lawrence Lessig, Glenn Greenwald, Karl Fogel -- attest to Aaron's supreme impact on the world at large (and that's no hyperbole!).
Before I had even heard of his tragic demise, a few colleagues and I were in the midst of writing letters of support for Aaron's nomination for this year's James Madison award from the American Library Association (ALA). This award, named in honor of President James Madison, was established by the ALA in 1986 to honor individuals or groups who have championed, protected and promoted public access to government information and the public’s “right to know” on the national level. I hope now that ALA will award Aaron posthumously!
We're helping Archive-it staff harvest a Web archive of Aaron's work, writings, images, videos, and remembrances. If you've got a URI that you'd like to be included in the archive, please paste it to this Google Doc.
Remembrances of Aaron, as well as donations in his memory, can be submitted at http://rememberaaronsw.com
The world will miss you Aaron. Be at peace my friend!