You are here

Web archiving

Yesterday the U.S. Senate Select Committee on Intelligence released its "Study of the CIA's Detention and Interrogation Program - Foreword, Findings, and Conclusions, and Executive Summary." (BIG PDF!) The report is 525 pages, heavily redacted, and includes graphic details about the torture techniques used by the CIA. The study found that American torture was not confined to a handful of aberrational cases or techniques, nor was it the work of rogue CIA agents. It was an officially sanctioned, worldwide (over 1/4 of the world's countries participated in some way!) regime of torture that had the acquiescence, if not explicit approval, of the top members of both political parties in Congress.

SLAC Early website "We had no idea that we were making history and were just trying to get the job done in our 'spare' time',”  Louise Addis, one of the WWWizards team who developed the SLAC website from 1991, said during our conversation about the restoration of SLAC's earliest website. Last May, Nicholas Taylor, web archiving service manager, told me, "SLAC has a historical collection of webpages that may be the first website in the US. Can we help them to find a home for this archive?” As Web archivist, I felt that I found a treasure. I replied, "Of course, Stanford Web Archive Portal should be the home."

One of the major use cases for the Web Archiving Service is preserving Stanford University web content. The earliest SLAC website represent the oldest such content we could find; it is the first website in the US dated to 1991, so we started there. Stanford Web Archiving Service launched its portal this week which featured SLAC's earliest website that was kept on SLAC servers for many years. This Halloween, it comes back to life. Our task was to convert the original list of scattered files into an accessible, browsable website with temporal navigation. In this post, I will discuss the technical challenges of and lessons learned from restoration process.

logo graphic appearing on the "SLAC Home Page" 1994-1995

In the course of creating a browsable archive of the SLAC earliest websites, we discovered a number of interesting facts and features that might not be readily apparent on casual browsing. While surely not an exhaustive catalog, we hope that these observations will help you to quickly get into the archive and discover some of what it has to offer.

logo graphic appearing on the "SLAC Home Page" 1994-1995

At a microscopic level, web archives document the evolution of individual websites. At a macroscopic level, they document the evolution of the Web itself. In the case of web archives for the period when the entire Web consisted of only a handful of individual websites, changes to even a single website reflect changes to the Web itself. We are pleased to announce the availability of such an archive, notably featuring the oldest U.S. website, dating to December 21, 1991.

'The Dish (HDR)' (under CC BY-NC-SA 2.0)

A couple of weeks ago, Stanford University Libraries hosted Dame Wendy Hall, Jim Hendler, and other web scientists affiliated with the Web Science Trust for a briefing on the Web Observatory initiative and a follow-on workshop organized by Lisa Green from Common Crawl. The notion of a Web Observatory implies a center proferring scientific instruments, but for the analysis of web data rather than natural phenomena. Indeed, the group's vision is that Web Observatories provide access to web datasets, projects, and tools. Eventually, a network of Web Observatories might offer both an interoperable architecture and distributed infrastructures for sharing and analysis of web datasets. The initiative touches on several areas of interest and investment by Stanford University Libraries, including data curation, web archiving, and supporting social science research.

'Step 7' (under CC BY-NC-ND 2.0)

A major challenge for web archivists is the low visibility that downstream archiving has on upstream web content creation. And, yet, deliberate and inadvertent architectural decisions made by web content creators strongly impact the ease or difficulty with which their websites can be captured and faithfully re-presented. A non-trivial byproduct of webmasters helping to ensure their content is archived for their own later use is that the Web itself becomes more archivable, to everyone's benefit.

logo of the International Internet Preservation Consortium

Web archivists Ahmed AlSum and Nicholas Taylor and LOCKSS Chief Scientist David Rosenthal recently attended the International Internet Preservation Consortium (IIPC) General Assembly, an annual meeting of national libraries, research universities, non-profits, and service providers engaged in web archiving. This was the first General Assembly we all attended since Stanford University Libraries (SUL) joined the IIPC, though we had all previously attended meetings under the auspices of other organizations.

'Material' (under CC BY-NC 2.0)

Congressional campaign websites are valuable primary source material for historians, social scientists, and the public to better understand the evolution of political communication in the Web era. Campaign websites also afford unique opportunities for the mass collection of materials that would have been previously difficult to acquire outside of the candidate's district. While it is a truism that the Web is constantly changing and broken links are an inevitable outcome, campaign websites are predictably ephemeral given their time-limited purpose.