Coverity, in collaboration with Stanford University and under contract from the Department of Homeland Security (DHS), has just released their Open Source Report 2008 (PDF). Their environmental scan of major open source projects found that the number of defects in open source code is dramatically dropping! More detail is available on ArsTechnica.

Now that we have definitive data that shows that open source software is strong on security, how can we get libraries to participate more readily on collaborative open source projects (like citation management, ILSs, CMSs...)? I'm reminded of a thought experiment posted by Joe Lucia, University Librarian @ Villanova University, in November 2007 on the NewGenCatalog list. In his post, Mr. Lucia called for a "shift of those investments from commercial software support (and staff technical support for commercial products) to a collaborative support environment for open source applications." Come on folks, let's make this shift happen!

In 2006, Coverity's scan detected an average of 0.30 defects per 1,000 lines of code, or, put differently, one code defects per every 3,333 lines. The lower boundary, in this case, was 0.02 (one defect per 50,000 lines) and the upper boundary was 1.22 defects per thousand lines of code.

Two years later, the average defect density has fallen to 0.25, or one error per 4,000 lines of code. The upper boundary remains unchanged at 1.22, but the lower boundary has shrunk to 0, implying that repeated scanning has eliminated the errors from at least one program—at least all the errors that Coverity's 2006 static analysis program was able to detect.

A 16 percent reduction in defect density over two years is a notable gain, and Coverity singled out certain participating projects as having an exceptionally low defect density.

• Postfix
• Perl
• PHP
• Python
• Samba

For some reason I just went back to review the keynote given by Karen Schneider -- aka Free Range Librarian -- at the 2007 Code4Lib conference. I found some really great practical tidbits for talking about open source in libraries as well as some food for thought. So please take the 50 minutes to sit back and enjoy Karen's most interesting talk. You'll be glad you did!

Below are some highlights:

• @ the 19min mark...seizing control of the tools we know we need to have and that we can create oursleves...we're really in a renaissance of librarian-built software for the first time it's like we're shaking ourselves awake and really grabbing hold, seizing the day. Librarian-built software begins to restore the balance of power in our profession...
• @27min mark (slide 16), there's an interesting exchange about open source including stereotypes of open source and how to talk to directors about open source at @ 36min.
• Slide 32 = every library needs a developer
• 48min mark = Q&A from Dan Chudnov discussing free software and Schneider's over-simplification that there's no free software. Dan points out that there's a higher level of conversation about free software that needs to happen.

| View | Upload your own

We at FGI are big proponents of free and open source software and open document formats for government information because we believe that using such software and formats is the best way to ensure long-term viability, usability, re-usability, and preservability of government information. But two FGI volunteers, James R. Jacobs and Shinjoung Yeo, are actively and successfully promoting open source where they work -- at the Stanford Libraries.

• Open source on campus: The Stanford Open Source Lab by Ruth Suehle, Red Hat Magazine (2/12/08).

Over the last few months, open source has gained momentum at Stanford University in the form of the Stanford Open Source Lab. Inspired by groups like the Free Software Foundation, Oregon State University's Open Source Lab, Drupal, Openflows Community Technology Lab, and MIT's Open Course Ware, a few people at Stanford decided to band together and dedicate their time and energies to the development of free/open/libre learning and knowledge resources. The vision of the Open Source Lab is to be a nexus on campus for the discussion, advocacy, and technical support of community-based technologies and information systems.

As James notes in the article, their promotion of open source is philosophical as well as technological. As he puts it, "The ideals of the Library intersect closely with those of the open source community. That is, the free flow of and access to information, support by and of a community of interest, open standards, and the necessity for a growing and vibrant public domain to further the goals and interests of the community. Those ideals as well as the example of OSU's Open Source Lab, led me to the idea of supporting open source at Stanford."

Happy Anniversary to Open Source! W00t!! Bruce Perens has just published "State of Open Source Message: A New Decade For Open Source."

The concept of open source has also hit the mainstream of the library world (although there have been many long-time proponents in the oss4lib, web4lib, code4lib communities!) with vendors like LibLime on the exhibits floor and talks about Drupal, Koha, Evergreen, VuFind etc and at ALA's 2008 midwinter conference. You owe it to yourselves, your libraries and your communities to find out about open source!

On February 9, 1998, I published the Open Source Definition and the public announcement of the Open Source Initiative that Eric Raymond and I were starting. This was the first time that the general public heard what Open Source was about. Friday, February 8 is the last day of Decade Zero of Open Source. Saturday, February 9 is the anniversary of Open Source and the start of Decade One. It's a computer scientist thing. We always start counting from zero :-) ...

...Most Open Source today is software being produced by its users, for its users ... By participating in Open Source development, users distribute the cost and risk of the development of enabling technology and infrastructure for their businesses. Their profit centers are not tied to software sales, but to some other business. To find them, look to the communities rather than the companies.

Check out the new search engine wikia search!  It is not only the newest search engine on the block, but also new (and I believe unique) in its values (see Wikia Search Alpha Launched January 7, 2008):

1. Transparency - Openness in how the systems and algorithms operate, both in the form of open source licenses and open content + APIs.
2. Community - Everyone is able to contribute in some way (as individuals or entire organizations), strong social and community focus.
3. Quality - Significantly improve the relevancy and accuracy of search results and the searching experience.
4. Privacy - Must be protected, do not store or transmit any identifying data.

Plus, building on the approach that has made Wikipedia so successful, it allows users to enhance the search engine by contributing to "mini articles" on search terms that will provide short definitions of terms,  help disambiguate similar or identical terms, and provide photos and "see also" references.

Here is the NYT coverage: Wiki Citizens Taking on a New Area: Searching By MIGUEL HELFT Published: January 7, 2008.

There was a great post to NGC4Lib list (ngc4lib = "Next Generation Catalogs for libraries") yesterday by Joe Lucia, the University Librarian
at Villanova, entitled a "thought experiment." In it, Lucia describes how to create next generation library systems via an open source collaborative commons. WOW!, a university librarian suggesting that a bunch of libraries get together to build an open source development system?! Thanks Joe Lucia for starting this conversation. I really hope it becomes more than simply a "thought experiment." Here's one of the juicier bits:

If we look beyond money to personnel, the option looks even better. Let me suggest some numbers. What if, in the U.S., 50 ARL libraries, 20 large public libraries, 20 medium-sized academic libraries, and 20 Oberlin group libraries anted up one full-time technology position for collaborative open source development. That's 110 developers working on library applications with robust, quickly-implemented current Web technology -- not legacy stuff. There is not a company in the industry that I know of which has put that much technical effort into product development. With such a cohort of developers working in libraries on library technology needs -- and in light of the creativity and thoughtfulness evident on forums like this one -- I think we would quickly see radical change in the library technology arena. Instead of being technology followers, I venture to say that libraries might once again become leaders. Let's add to the pool some talent from beyond the U.S. -- say 20 libraries in Canada, 10 in Australia, and 10 in the U.K. put staff into the pool. We've now got 150 developers in this little start-up. Then we begin pouring our current software support funds into regional collaboratives. Within a year or two, we could be re-directing 10s of millions of dollars into regional technology development partnerships sponsored by and housed within the regional consortia, supporting and extending the work of libraries. The potential for innovation and rapid deployment of new tools boggles the mind. The resources at our disposal in this scenario dwarf what any software vendor in our small application space is ever going to support. And, as is implicit in all I've said, the NGC is just the tip of the iceberg.

[Thanks OSS4lib list!]

Have you ever tried to open a WordPerfect document when all you have is Microsoft Word? Or maybe you've received a Microsoft Works document and found that your version of Word won't open it. If you've been around documents for a while, perhaps you've tried to open some of the spreadsheets that agencies distributed in Lotus format and found that you couldn't open the files If so, you've experienced first-hand the problem that Aliya Sternstein describes in an article about the importance of open formats for government information:

• The Risks Of Software Choice In E-Government, by Aliya Sternstein, National Journal's Technology Daily, (Sept. 20, 2007 pm edition)

One of the big technological battles going on now is between the truly open ODF format and Microsoft's so-called open format, OOXML. Sternstein writes "Microsoft and its supporters maintain that having a choice between any and all open file formats would be advantageous for governments" but that "[g]iving U.S. agencies a choice in file formats could be bad for record-keeping because down the road, records might be saved in different, non-compatible formats or agencies might be held hostage by one company's product line."

Will Rodger, public policy director with the Computer and Communications Industry Association, says:

"It is hugely ironic that promoters of OOXML and critics of ODF say you need to look at what their technologies do. As far as we can tell, the greatest impetus for the development of OOXML is to create technologies that perpetuate the proprietary lock-in [that] governments were trying to eliminate in the first place."

(See also:   Government Information in Legacy Formats: Scaling a Pilot Project to Enable Long-Term Access, by Gretchen Gano and Julie Linden, D-Lib Magazine (July/August 2007) Volume 13 Number 7/8, and a project a colleague of mine, Doug Tower, worked on several years ago, the UCSD GPO Data Migration Project and the page that describes some of the processing for that project, Processing and Quality Control. Also see: Microsoft vs. Open Formats.)

• Towards an Open Source Repository and Preservation System: Recommendations on the Implementation of an Open Source Digital Archival and Preservation System and on Related Software Development, Kevin Bradley, Junran Lei, and Chris Blackall. Paris: UNESCO, 2007.

This is an interesting and worthwhile whitepaper. It examines existing open source tools and evaluates them for constructing a digital archival and preservation system compatible with the  Reference Model for an Open Archival Information System (OAIS). Their aim it to develop a single package open source repository system based on existing open source platforms. Its model organization is a small- to medium-size cultural heritage organization with a low, but not non-existent, capital investment budget aiming to manage between 1TB and 20TB of data.

One of the recommendations of the whitepaper is to "Encourage the development of federated and cooperative approaches through the adoption of standard data packages."

Although the report does not examine LOCKSS because its goal was to be able to support standalone preservation repositories that are not dependant on remote storage facilities or systems, it does have as another key finding that the system should "Make multiple copies, and check and verify them regularly" -- just as LOCKSS does

This from the executive summary:

This Document defines the requirements for a digital archival and preservation system using standard hardware and describes a set of open source software which can be used to implement it. This report defines the requirements for a digital archival and preservation system using standard hardware and describes a set of open source software which could used to implement it. There are two aspects of this report that distinguish it from other approaches. One is the complete or holistic approach to digital preservation. The report recognises that a functioning preservation system must consider all aspects of a digital repositories; Ingest, Access, Administration, Data Management, Preservation Planning and Archival Storage, including storage media and management software. Secondly, the report argues that, for simple digital objects, the solution to digital preservation is relatively well understood, and that what is needed are affordable tools, technology and training in using those systems.

As Daniel pointed out on Tuesday (Another Example of Access Bad, Ownership Good), when Google shut down its premium video service on August 15, it was able to prevent customers who had bought and paid for videos from Google from ever watching those videos again. But there is a bit more to the story -- and it has implications for how government information is distributed.

In this, Part One of a two part look at the issues, we examine how Digital Rights Management (DRM) technologies and proprietary software allowed Google to make it impossible for users to watch videos that they had bought from Google even though they had downloaded the video files onto their own computers. In Part Two we examine why this story is important for government information specialists and Federal Depository Library Program (FDLP) librarians.

While it wasn't obvious from the initial news stories about this, Google used DRM and proprietary software to deny access to files users had paid for and downloaded. Although the Government Printing Office (GPO) has not used identical techniques, it has experimented with similar ones and has never explicitly rejected use of techniques that could provide government a way to deny access to information even if users have copies of files on their own computers. More on that below.

Google's premium video-purchase-and-download service overlapped with with Google's YouTube-like service, Google Video, which offers streaming video without charge and some free downloads. But in early 2006, Google announced a service as part of Google Video that would allow users to pay to rent or buy certain videos (such as NBA basketball games and TV shows such as CSI) and it is that service that changed this week and those videos that people purchased that they no longer can watch.

YouTube and Google Video use "streaming" video technology so that you watch the stream of video as it comes to you. It is also possible to download videos in some cases. But the Google premium video service allowed users to pay for videos, download them, keep the video files, and watch them without streaming or re-streaming the content. Customers had the files on their own computers and could copy them and put them on different machines as if they really did "own" them. But there was a catch.

Actually, there were three catches. First, users of this service had to download and install the proprietary "Google Player" software. (The software was originally downloadable from http://video.google.com/playerdownload but even the Google cache of that page disappeared this week.) It served a similar function to Windows Media Player or Quicktime or other media players, but it used its own proprietary format (".gvi"). Only the Google Player could play Google Videos.

Second, you could watch the proprietary format using the proprietary player only if you were connected to the internet and authenticated yourself as the purchaser.

Since the Google Player was the only player that could read the files one purchased, users were locked-in to the DRM of authentication-over-the-internet (sometimes called the "phone home" feature). So, even if you paid for a video and "owned" the file you downloaded, you couldn't watch it unless Google allowed you to do so -- every time you watched it. This week Google simply turned off the ability for users to authenticate. Presumably, this is the way the Google rental service operated from the start: after 24 hours, you no longer had permission to view the file you downloaded. It turned out that the "purchase" program was just a temporary service as well.

This is why The Guardian described the situation this way:

Google handed opponents of digital rights management (DRM) a huge weapon this week when it announced that DRM-protected videos bought from its online video store will no longer work, and that customers will not be reimbursed.
  -- Kiss goodbye to your DRM-protected Google Video clips, by Charles Arthur The Guardian, August 16 2007

And that brings us to the third catch: the Digital Millennium Copyright Act (DMCA). Although there are hacks, work-arounds, and other technical tricks that allow one to circumvent the Google phone-home DRM, they are against the law. Again, The Guardian saw the implication of this for libraries:

But the fact that thousands of purchased files will cease working will give pause to organisations charged with creating public archives of published information - such as the British Library and, in the US, the Library of Congress. The latter in particular was anyway considering whether any redrafting is needed on the Digital Millennium Copyright Act (DMCA): the idea of offering a loophole to circumvent DRM on products that no longer work properly was rejected in its last consideration. Google's decision might lead to a reversal in thinking.

There is more about this story and its implications for FDLP libraries in Part Two.

More information:

• Google Video robs customers of the videos they "own", BoingBoing August 10, 2007
• Google Video service to go black by Dawn C. Chmielewski and Alex Pham Los Angeles Times, August 11, 2007
• Google's permanent video sales less-than-permanent by Cade Metz, The Register, August 11, 2007
• Google Video DRM: Why is Hollywood more important than users? by Cory Doctorow, BoingBoing, February 14, 2006
• More On Google Copy Protection by Mike Masnick, TechDirt, Mon, Jan 9th 2006
• Google's Copy Protection: Supplying The Tools For Others To Be Evil by Mike Masnick, TechDirt, Fri, Jan 6th 2006
• Google Video (beta) by Troy Dreier, CNet, 2/7/06
• Google Video Player Terms and Conditions Google Video
• Good Uses for DRM Jimmy Palmer, DRM Blog (September 2. 2005) [good description of "phone home" DRM]

What a great idea! Karl Fogel, an open-source software developer, writer and activist for copyright reform had this to say in a recent New York Times article:

Karl Fogel, president of the Subversion Corporation, which produces open-source version control software, He sees its power to shape public behavior.

Think of what version control software could mean for the Congress, he was quoted as saying recently at Tim O’Reilly’s blog . If bills were created under a system where strike-throughs and additions were carefully tracked, the public would know which legislator made which change to a proposed piece of legislation as it made its way through the Capitol.

At last, there would be transparency in the legislative process. Best-case scenario, it would shame legislators from inserting language against the public interest and only meant to reward political contributors; at worst, it would make such insertions public and allow the voters to punish the politicians who made them.

"Crossing Out, for Emphasis" by Noam Cohen. NYT 7/23/07.

Well, this is another Lunchtime Watch rather than listen. But you do listen while you watch, don't you?

Meredith Farkas was asked to give her top tech trends at LITA's famous trends forum at the American Library Association annual conference. Unfortunately for attendees but fortunately for the rest of us, Meredith couldn't be there in person and is giving her presentation via screencast:

Click To Play

The whole segment is worth watching, but I really want to highlight her first trend - "Open Source for the rest of us". Meredith explains why it's not just for programmers anymore. For those who'd rather read than watch, but miss great screenshots of programs she talks about, she put her script into a blog posting.

When you're doing watching Meredith's trends, check out our digital library technologies and remixes pages and start thinking about how YOU can add value to government information and help it keep it safe, accessible and DRM free for our children and our grandchildren.

Thought you had a handle on the concept of copyright? Think again! Last week there was a post on Slashdot entitled, "Should Copyright Be Abolished?" by Greg Bulmash (full article posted on his blog, "Brainhandles"). I think this discussion has relevance to government documents and libraries in general, since we are steadily moving away from a copyright information world and into a licensing information world. I'm trying to get my head around this shift and so welcome the reading material. The ideas of attribution, distribution, DRM, fair use, licensing, public domain... all feature prominently in this discussion.

Bulmash waded into the copyright debate, taking on those in the tech community that seek to abolish copyright. The gist of Bulmash's argument was that "you can't oppose copyright and support open source." Bulmash opines that the GNU Public License, the license under which much open-source software is distributed (there are several flavors of open source licenses, but I won't get into that here), depends on copyright to be enforceable. Therefore, you can't have the GPL without copyright. Bulmash argues for reforming copyright, not abolishing it -- "surgery, not euthanasia."

These members of the anti-copyright crowd cite the GPL (GNU Public License) as an alternative to copyright without any sense of the ironic fact that the GPL can't exist without copyright. They're proposing a solution while simultaneously advocating the destruction of the thing that makes their solution workable. While the GPL is less restrictive than other licensing methods, it's a license and it does impose some restrictions on or conditions for use of the work. It is a method of controlling your work. But without copyright, the GPL could not be enforced.

Bulmash was answered the next day by Karl Fogel of Question Copyright in his essay, "Supporting Open Source While Opposing Copyright." Fogel makes a very compelling argument that the abolition of copyright doesn't necessarily go against the spirit of the GPL, nor does the GPL need to rely on copyright in order to forward the cause of open source or free software (two different, but conflated ideas!). He suggested that Bulmash, "mixes up two completely different concepts: the right to be credited for a work, and the right to control distribution of that work." Fogel goes on to state that copyright is simply the current enforcement tool du jour, but is not a natural and uncontroversial "right."

The basic argument of copyright abolitionists is that people should be free to share when sharing does not result in any diminution of supply. The GPL simply uses copyright law in a jiujitsu-like manner to enforce this principle, in a legal environment where sharing is prohibited by default and must be explicitly permitted to be legal. All the GPL does is create a space where permission to share is enforced. Take his exercise in imagination all the way: imagine if we had laws that did away with most prohibitions against sharing, but that enforced crediting and permitted authors to enforce GPL-like provisions requiring sharing.

and...

Put bluntly: a future law that merely allows authors to enforce sharing need have little in common with today's laws that allow the restriction of sharing. Since these two things are more opposite than alike, calling them both "copyright" doesn't make much sense. But that is what Bulmash does, when he implies that the current copyright regime (or something structurally similar to it) is the only way the GPL could be enforced.

There are some great comments in both threads so if you have the time, brew a pot of tea, sit down and wade through them. You'll be glad you did because this debate definitely has import to what librarians do!

This one's going to take a while to completely absorb, but I'd highly recommend that everyone check out the new, enormous (287 pages!) and wide-ranging study of Free/Libre Open Source Software (FLOSS). The report, financed by the European Commission’s Directorate General for Enterprise and Industry, was written by Rishab Ayer Gosh and an international, interdisciplinary team of researchers. Based on this report, The European Commission has issued an endorsement of open-source software. CNet's got the news on the report. Here's the PDF of the document, also attached below.

The Economic impact of open source software on innovation and the competitiveness of the Information and Communication Technologies (ICT) sector in the EU exhaustively documents the way that Free/Open technologies dominate information technology and describes who actually writes Free/Open software. It also talks about what it would cost to replicate the benefits of Free/Open software through proprietary development (EU12 billion!), how many person years that would take (131,000!), and projects the total size of the Free/Open market in the years to come.

This is the most authoritative study on Free/Libre Open Source Software (FLOSS) that I've seen to date. If you know of others, please post them to the comments.

Here at FGI, we're really interested in open source -- also called FLOSS for Free/Libre Open Source Software. Open source aligns positively with the philosophy underscoring librarianship in general and documents librarianship specifically -- that is, free and open access to and widespread distribution of information, use and reuse of information, and the leveraging of community resources for the betterment of the community as a whole. For more on open source in libraries, read Dan Chudnov's instructive article, "Open Source Library Systems: Getting Started" (which originally appeared in Library Journal on August 1, 2999) on oss4lib (don't be confused by their look, oss4lib is using Drupal content management system and the same template theme as FGI!).

On the lighter side of things and in the spirit of the season, check out the open source gift guide from Make Magazine. The guide lists lots of cool techie gifts for those geeks on your list -- for those keeping score, I'd love a year of ubuntu support, some open source beer, and USB AA rechargeable batteries!)

Here's a twofer to give you some more reading matter over the long weekend:

A friend sent me this internet campaign to shed public light on the secret Smithsonian/Showtime contract that would give Showtime a 30 year, non-competitive stranglehold on Smithsonian (i.e. public domain!) archives. Background on the story can be found at boingboing. If you want to be added as a signatory, please send email to Carl Malamud (carl@media.org) no later than Sunday, November 25, 2006.

After, I signed the letter, I was looking around the public.resource site and came across another campaign (perhaps dated but still viable!) that Carl had put together calling for a hacker tax credit! The basic idea is that open source software, because it is the driving force behind our new information culture, should be supported publically so that more growth can happen. Check out the text of the letter below that Malamud suggests you send to your Congresspeople. This campaign, as I said earlier, may be dated (he lists Vice President Al Gore as a suggested addressee), but open source software (sometimes called FLOSS) is still something for which we should all be advocating!

Pablo Picasso once said that good art is created, but great art is stolen. On the Internet, the same holds true. Good code is created, but great code is copied over and over.

The Internet was created from open source software, code that people can freely use to build new code, to run their networks, to create a new business, or to build a service that people can use.

Take for example the work of Paul Vixie, who has placed in the public domain the software that the Domain Name System runs on. This software has been used by every major Internet Service Provider and has been bundled into the operating system products of IBM, DEC, Silicon Graphics, and Sun.

Open source software created the Internet, and created the economic boom we now see in Silicon Valley. Most of the large web sites in the world run on the open source Apache web server. The $4 billion Netscape Corporation was built from the open source Mosaic. The PERL programming language was created as open source, but now fuels over $100 million in book sales for publishers like O'Reilly & Associates.

But, we are eating our seed corn. There is no systematic national effort to create open source software and it is increasingly difficult to keep this infrastructure alive. For every success story like Apache, there are dozens of projects that languish because of the lack of formal support for open source projects.

In the global village, open source software is not an alternative to commercial software, just as in our real cities public parks are not an alternative to our commercial districts. The parks make our cities thrive, and thriving cities are a good place to do business.

It is a happy accident that we have open source software, but there are simple steps that the federal government can take to provide even more fuel for the growth of our information economy. Here is a simple algorithm for a Hacker Tax Credit that could be added to the U.S. Code:

      #/us/usc/irs
      if {
            You produce software that is in the public domain ; 

      } andif {
            That software is used by at least 1000 people ; 

      } then {
            You may deduct your development and operational costs from your gross income for tax purposes ; } 

If the U.S. Congress could compile this simple subroutine into the U.S. Code, this simple step would have a greater effect than any cuts in capital gain taxes. I urge you to consider steps that the U.S. Congress can take to insure a strategic national reserve of open source software.

Sincerely,

Carl Malamud
media.org