The UK Web Archive, Born Digital Sources and Rethinking the Future of Research

The following post is derived from a short talk I gave at a doctoral training event at the British Library in May 2015, focused on using the UK Web Archive.  It was written with PhD students in mind, but really forms a meditation on the opportunities created when we are working with web sites rather than print.  While lightly edited, the text retains the ticks and repetitions of public presentation.

My office c.1984

I normally work on properly dead people of the sort that do not really appear in the UK Web Archive – most of them eighteenth-century beggars and criminals.  And in many respects the object of study for people like me – interlocutors of the long dead -  has not changed that much in the last twenty years.  For most of us, the ‘object of study’ remains text.  Of course the ‘digital’ and the online has changed the nature of that text.  How we find things – the conundrums of search – shape the questions we ask.  And a series of new conundrums have been added to all the old ones – does, for instance, ‘big data’ and new forms of visualisation, imply a new ‘open eyed’ interrogation of data?  Are we being subtly encouraged to abandon older social science ‘models’, for something new?   And if we are, should these new approaches take the form of ‘scientific’ interrogation, looking for ‘natural’ patterns – following the lead of the Culturomics movement; or perhaps take the form of a re-engagement with the longue durée– in answer to the pleas of the History Manifesto.   Or perhaps we should be seeking a return to ‘close reading’ combined with a radical contextualisation - looking at the individual word, person, and thing – in its wider context, preserving focus across the spectrum.

And of course, the online and the digital also raises issues about history writing as a genre and form of publication.   Open access, linked data, open data, the 'crisis' of the monograph, and the opportunities of multi-modal forms of publication, all challenge us to think again about the kind of writing we do, as a  literary form.  Why not do your PhD as a graphic novel? Why not insist on publishing the research data with your literary over-lay?  Why not do something different?  Why not self-publish?

These are conundrums all – but conundrums largely of the ‘textual humanities’.  

Ironically, all these conundrums have not had much effect on the academy and the kind of scholarship the academy values.  The world of academic writing is largely, and boringly, the same as it was thirty years ago.  How we do it has changed, but what it looks like feels very familiar.

But the born digital is different.  Arguably, the sorts of things I do, history writing focused on the  properly dead, looks ‘conservative’ because it necessarily engages with the categories of knowing that dominated the nineteenth and twentieth centuries – these were centuries of text, organised into libraries of books, and commentated on by cadres of increasingly professional historians.  The born digital – and most importantly the UK web archive – is just different.  It sings to a different tune, and demands different questions – and if anywhere is going to change practise, it should be here. 

Somewhat to my frustration, I don’t work on the web as an ‘object of study’ –  and therefore feel uncertain about what it can answer and how its form is shaping the conversation; but I did want to suggest that the web itself and more particularly the UK Web Archive provides an opportunity to re-think what is possible, and to rethink what it is we are asking; how we might ask it, and to what purpose.

And I suppose the way I want to frame this is to suggest that the web itself brings on to a single screen, a series of forms of data that can be subject to lots of different forms of analysis.  A few years ago, when APIs were first being advocated as a component of web design, the comment that really struck me, was that the web itself is a form of API, and that by extension the Web Archive is subject to the same kind of ‘re-imagination’ and re-purposing that an API allows for a single site or source.  

As a result, you can – if you want – treat a web page as simple text – and apply all the tools of distant reading of text - that wonderful sense that millions of words can be consumed in a single gulp.   You can apply ‘topic modelling’, and Latent Semantic Analysis; or Word Frequency/Inverse Document Frequency measures.  Or, even more simply; you can count words, and look for outliers – stare hard at the word on the web!

But you can also go well beyond this.  In performance art, in geography and archaeology, in music and linguistics, new forms of reading are emerging with each passing year that seem to me to significantly challenge our sense of the ‘object of study’ – both traditional text and web page.  In part, this is simply a reflection of the fact that all our senses and measures are suddenly open to new forms of analysis and representation.  When everything is digital – when all forms of stuff come to us down a single pipeline -  everything can be read in a new way.  

 Consider for a moment the ‘LIVE’ project from the Royal Veterinary College in London, and their ‘haptic simulator’.  In this instance they have developed a full scale ‘haptic’ representation of a cow in labour, facing a difficult birth, which allows students to physically engage and experience the process of manipulating a calf in situ.  I haven’t had a chance to try this, but I am told that it is a mind altering experience.  It suggests that reading can be different; and should include the haptic - the feel and heft of a thing in your hand.  This is being coded for millions of objects through 3d scanning; but we do not yet have an effective way of incorporating that 3d text into how we read the past. 

 The same could be said of the aural - that weird world of sound on which we continually impose the order of language, music and meaning; but which is in fact a stream of sensations filtered through place and culture.  

Projects like the Virtual St Paul's Cross, which allows you to ‘hear’ John Donne’s sermons from the 1620s, from different vantage points around the yard, changes how we imagine them, and moves from ‘text’ to something much more complex and powerful.  And begins to navigate that normally unbridgeable space between text and the material world.  And if you think about this in relation to music and speech online – you end up with something different on a massive scale.

One of my current projects is to create a sound scape of the courtroom at the Old Bailey - to re-create the aural experience of the defendant - what it felt like to speak to power, and what it felt like to have power spoken at you from the bench. And in turn, to use that knowledge to assess who was more effective in their dealings with the court, and whether, having a bit of shirt to you, for instance, effected your experience of transportation or imprisonment.  And the point of the project is to simply add a few more variables to the ones we can securely derive from text.

It is an attempt to add just a couple of more columns to a spreadsheet of almost infinite categories of knowing.  And you could keep going – weather, sunlight, temperature, the presence of the smells and reeks of other bodies.  Ever more layers to the sense of place.  In part, this is what the gaming industries have been doing from the beginning, but it also becomes possible to turn that creativity on its head, and make it serve a different purpose.

In the work of people such as Ian Gregory, we can see the beginnings of new ways of reading both the landscape, and the textual leavings of dead.  Bob Shoemaker, Matthew Davies and I (with a lot of other people) tried to do something similar with Old Bailey material, and the geography of London in the Locating London’s Past project.

This map is simply colours blue, red and yellow mapped against brown and green.  I have absolutely no idea what this mapping actually means, but it did force me to think differently about the feel and experience of the city.  And I want to be able to do the same for all the text captured in the UK domain name. 

All of which is to state the obvious.  There are lots of new readings that change how we connect with historical evidence – whether that is text, or something more interesting.    In creating new digital forms of inherited culture - the stuff of the dead - we naturally innovate, and naturally enough, discover ever changing readings.  But the Web Archive, challenges us to do a lot more; and to begin to unpick what you might start pulling together from this near infinite archive. 

In other words, the tools of text are there, and arguably moving in the right direction, but there are several more dimensions we can exploit when the object of study is itself an encoding.

Each web page, for instance, embodies a dozen different forms.  Text is obvious, but it is important to remember that each component of the text – each word and letter, on a web page - is itself a complex composite.  What happens when you divide text by font or font size; weight, colour, kerning, formatting etc.  By location - in the header, or the body, or wherever the CSS sends it; or more subtly by where it appears to a users’ eye - in the middle of a line – or at the end.

Suddenly, to all the forms of analysis we have associated with ‘distant reading’ there are five or six further columns in the spread sheet – five or six new variables to investigate in that ‘big data’ eye-opened sort of way.

And that is just the text.  The page itself is both a single image, and a collection of them – each with their own properties.  And one of the great things that is coming out of image research is that we can begin to automate the process of analysing those screens as ‘images’.  Colour, layout, face recognition etc.  Each page, is suddenly ten images in one – all available as a new variable; a new column in the spreadsheet of analysis.  And, of course, the same could be said of embedded audio and video.

And all of that is before we even look under the bonnet.  The code, the links, the meta data for each page – in part we can think of these as just another iteration of the text; but more imaginatively, we can think about it as more variables in the mix.

But, of course, that in itself miss-understands the web and the Web Archive.  The commonplace metaphor I have been using up till now is of a ‘page’ – and is the intellectual equivalent of skeumorphism - relying on material world metaphors to understand the online.

But these aren’t pages at all, they are collections of code and data that generate in to an experience in real time.  They do not exist until they are used - if a website in the forest is never accessed, it does not exists.  The web archive therefore is not an archive of ‘objects’ in the traditional sense, but a snapshot from a moving film of possibilities.  At its most abstract, what the UK Web Archive has done, is spirit in to being the very object it seeks to capture – and of course, we all know that in doing so, the capturing itself changes the object.  Schrödinger's cat may be alive or dead, but its box is definitely open, and we have visited our observations upon its content.

So to add to all the layers of stuff that can fill your spreadsheet, there also needs to be columns for time and use; re-use and republication.  And all this is before we seek to change the metaphor and talk about networks of connections, instead of pages on a website.

Where I end up is seriously jealous of the possibilities; and seriously wondering what the ‘object of study’ might be.  In the nature of an archives, the UK Web Archive imagines itself as an ‘object of study’; created in the service of an imaginary scholar.  The question it raises is how do we turn something we really can’t understand, cannot really capture as an object of study, to serious purpose?  How do we think at one and the same time of the web as alive and dead, as code, text, and image – all in dynamic conversation one with the other.  And even if we can hold all that at once, what is it are we asking?

Judging a book by its URLs

It will sound odd, but I have recently had a great time editing URLs.  Robert Shoemaker and I have have just finished a book for CUP, derived from the London Lives project, and called - London Lives: Poverty, Crime and the Making of a Modern City, 1690-1800. It is a long book (170,000 words) and each quote and reference in it is linked via a URL to the original document or article, book or web-resource used as evidence or to contextualize the argument.  It will be published as both an ebook and in hard copy, and the links need to be robust, and secure.  My estimate is that there are in the region of 4,000 URLs included in the manuscript (which was written collaboratively in PMWiki).  In the end, I found that I could identify an appropriate link for 98% of all footnote references, but then had to eliminate around 10% of these, as the relevant URL was just not useable.  The book took some nine years, and I am glad it is finished.

One of my final jobs was editing those 4000 URLs.   It took about three months work, spread over the last year, and I have just finished spending a week or so confirming what I hope will be their final form.  When I have told people about this work many have looked incredulous and suggested that this is the sort of technical implementation process that should be left to others.  A couple of otherwise nice people have suggested I dump this job on the shoulders of the nearest PhD student.  But for myself, it is precisely the kind of thing that an author should do for themselves.  And in doing it, two things kept coming to mind.  First was how the role of the scholar in creating a rigorous academic apparatus is a central part of the intellectual journey that academic writing involves - and that we should see the implementation of the online version of this in the light of the precise writing of footnotes and references that mark out good scholarship.  And second, that URLs encode a system of design and intent, online architecture and system of access, that signal the quality and permanence (the academic credibility and perceived audience) of historical materials online.  And that just as we have always sorted and judged scholarship by its form, we should think a bit harder about how the form of a URL can let us interrogate online materials.

On the first point, I do not know of much discussion of the joys of this kind of academic slog.  There is a lot of good writing on research and archives (by Carolyn Steedman and Arlette Farge among many others), on writing and thinking, but no-one talks much about the painstaking labour that goes in to turning a rough draft in to a final finished piece of scholarship.  And here I am really talking about generating accurate and fully comprehensive footnotes that reflect both the material cited, and the research journey that resulted in the main text.   This has become much easier with online catalogs and citation management packages, but nevertheless remains laborious and a reflection of our collective and individual commitment to a particular kind of evidenced discussion.  But for me it also represents my favourite compromise.  The writing of history is a wonderfully imaginative and creative process.  And in some respects we wish to judge the product of history writing as art.  Is it enjoyable to read? Is it convincing?  Does it do the job of good writing in liberating the readers' imagination?  In making these judgements we tend to appeal to a notion of 'value' that is cultural and that privileges dominant forms of authority.  This aspect of judgement is essentially romantic; with all the implications for western and elite hegemony embedded in that idea.  At the same time history writing is the result of simple hard work of a more technical kind - in the archives, in collating and collecting, re-ordering and interrogating data.  And it is valuable because it encompasses that hard work.  The beauty of the academic apparatus is that it evidences this and in the process generates a different measure of value.  In other words it is where quality is tied to a 'labour theory of value'.  I love the academic slog because it is where un-moored judgement is tied down to hard labour; and where value can be universalized in a common human experience (work).  In other words I really enjoyed editing 4000 URLs precisely because in them and their associated footnotes lies a claim to and evidence of the hard labour that underpins the book itself.

 At the same time, the process also taught me to read URLs differently.  Clearly coders and web designers do this as a matter of course.  But I am a historian and want to read URLs as a scholar, rather than as a programmer or designer.  And for me, the important thing is that URLs embed the structure of a site, making it plain to see for anyone willing to look hard; and that they are made up of both the character of a library reference, and a command directed at the new technology of discovery - the Internet .  There are just lots of different types of URL.

There are 'Search URLs' that include all the elements that  take the user past a collection to a specific object, but don't let you go directly there without the query.  And there are URLs that encode a cataloging hierarchy.  There are URLs that sift data, or work in your browser to change the data delivered, highlighting phrases or sifting material.  And there are URLs that encode licensing, passwords, and access information.  It is easy enough to find that the whole search journey that took you from a library catalog to an individual item is encoded directly in the URL, and even personalized to you, the machine you are using, or the forms of access you can deploy.  It is easy to find URLs that run on for hundreds of characters, each element divided by a '&' or a '%', or such.

But in creating robust reproducible links to credible historical materials most of these URLs are at least problematic if not useless.  If they include details for institutional access, or session information, they cannot be re-used by someone else.  These URLs are friable and fragile things and not fit for scholarly purposes.  And as a result, for the London Lives book we have been forced to eliminate all the links we originally hoped to include to forty or fifty different sites.  To take a single example, most archives structure their online collections with search in mind, making it difficult to link to a single item.  I spent a lot of time finding the catalog entry for every manuscript we cited in the London Metropolitan Archives, and Westminster Archives Centre, only to regretfully strip out the links when confronted by a complex URL that just did not look credible as a long term citation of the item itself.

Even in its simplest, and in the form recommended by the site for sharing a link, a London Metropolitan Archives URL looks like this:

http://search.lma.gov.uk/scripts/mwimain.dll/144/LMA_OPAC/web_detail/REFD+P69~2FBRI~2FB~2F001~2FMS06554~2F004?SESSIONSEARCH


Since we had consulted these items in their physical form in any case, it did not seem too problematic to leave out these links, but a shame nevertheless.  And likewise, with paywall material there seemed little point in dangling real access, and the promise of credible evidence, before the eyes of readers who would not be able to go beyond the login screen.  It seemed better to cite a specific item in combination with a general (unlinked) URL and date of consultation as reflecting our own research journey, rather than to promise access when we could not deliver it.

With few exceptions the URLs that have been retained (and there are still 4000 of them) address specific items with a specific ID, and usually run to 20 to 40 characters.  DOIs are not bad once you figure out their structure and reformulate them as they should be, rather than the way they are normally cited on journal web pages.

dx.doi.org/10.1353/sec.2010.0268

And Google Books creates a very nice URL once you strip out all the complex formatting instructions that are normally generated as part of a search and inserted after the main ID.  This is what a Google Books' URL looks like if you were to use the 'search' version:

 http://books.google.co.uk/books?id=1sMJGt7_rTAC&printsec=frontcover&dq=%22Prosecution+and+Punishment:+Petty+Crime+and+the+Law%22&hl=en&sa=X&ei=rrzGUq_aDsSy7Aa_9YGQCg&redir_esc=y#v=onepage&q=%22Prosecution%20and%20Punishment%3A%20Petty%20Crime%20and%20the%20Law%22&f=false

And this URL will take to the same book:

 books.google.co.uk/books?id=1sMJGt7_rTAC

 And the Eighteenth-century Short Title Catalog generates some of the most elegant URLs I have found:

estc.bl.uk/T174945

And to a lesser extent, so does the Ethos collection of doctoral theses at the British Library.

ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.354762

And London Lives and the Old Bailey Online do pretty well on this score:

www.londonlives.org/browse.jsp?div=LMSMPS501980014
http://www.oldbaileyonline.org/browse.jsp?ref=t17910413-19


In part, I suspect that these issues would all disappear if I had a better sense of the layer of structure that lies beneath the WWW.  But for the moment I am keen to have a short, human-readable URL that looks like it will last longer than the session I am currently logged on for.   All of which simply takes me back to the joy of academic slogging and the importance of the academic apparatus as something that evidences hard work and opens up scholarship to credible criticism that goes beyond simple romantic appreciation and prejudice.

I know all too well that one of the skills of an academic is the ability to judge a book by its cover and the form of the text it contains.   For the online we need to embed URLs into precisely this process - and the joy of all that editing was that at the end of it, I feel I have learned to do just that.

The Old Bailey on Steroids

The text of a talk delivered at the Humanities Research Institute at the University of Sheffield on 21 May 2008, as a part of the formal launch event of the Old Bailey Online, 1674-1913 (www.oldbaileyonline.org)

When in 1999, Bob Shoemaker and I sat in what was then the new British Library and first discussed the possibility of creating an online edition of the Proceedings, it seemed to us a good idea. From conception, to getting the money, to the hard slog of digitisation, to the equally hard slog of tagging and preparation, to the second tranche, in partnership with Clive Emsley and the Open University, to the ever inventive crises of implementation, to the whirlwind of publicity, and the equally chaotic whirlwind of online delivery, I still think it was a good idea.

But, along the way it has become something rather better than the idea we started with. In some ways, even the present on-line edition of the Proceedings the people in this room have made a reality, is an extension of an essentially Enlightenment notion of knowledge and publication. It is a massive serial body of text, indexed and structured, huge in its compass, but essentially recognisable to any historian who cut their research teeth on the old Times indexes, or who built upper body strength hefting the catalogue in the old Round Reading Room. As a result, in its current form, the Proceedings are a beautiful culmination of an old fashioned idea. Perhaps ironically, they are just so last century.

But they form the starting point for something else. The promise of online, searchable, mash-upable, analysable, deracinated text is that it offers the opportunity to do something, not just better, but different. We have yet to see the first volume of twenty-first century history; of internet enabled history. We have yet to see the first major piece of historical writing that could only be conceived as a result of the existence of online resources; but it is the promise that such a new historical form might be created that is now driving the evolution of the Old Bailey.

Through the ESRC funded Plebeian Lives and the Making of Modern London project, which we have spent the last three years helping to make happen, and will spend the next two bringing to some kind of fruition, we are taking the scripts of criminality - the events of a trial, or the retrospective biography of an Ordinary’s Account, the things we have – and adding to them 40 million manuscript words of the scripts of both poverty, and just plain normality.

This new Old Bailey on steroids will encompass eight coherent manuscript archives – the records of St Thomas’s Hospital, the Coroners’ Inquests for Westminster, the parish records of St Clement Dane, St Dionis Backchurch, and St Botolph Aldgate, and the records of the Middlesex Bench - Pauper examinations and depositions, account books, and a massive archive of rotting, quotidian paper. And to this, it will add the electronic leavings deposited by a generation historians in the AHDS; the Cantebury Wills, the voting records of Westminster, the Hearth Tax Returns. And to this too, the death and marriage and birth registers created by shoals of family historians and published as CDs.

No one has ever attempted to digitise manuscript materials of this complexity before on this scale; and more importantly no one has ever managed to order this vast complex legacy of ill-digested record keeping, into a single searchable thing. In undertaking this project, we are building on the skills created in the long process of making the Old Bailey happen. Without the world beating experience built up by the Higher Education Digitisation Service in double entry rekeying, in creating usable transcripts of every kind of eighteenth-century hand imaginable; we couldn’t do Plebeian Lives. Without the expertise of Jamie McLaughlin, without his experience of designing search facilities and making them work, we couldn’t do this project. Without Sharon Howard’s management and technical skills, and simple commitment; we couldn’t do this project. Without Ed McKenzie and Kathy Rogers, without Philippa Hardman and the HRI, we couldn’t do this project. In other words, we have taken a body of expertise built up across a decade between three Universities, re-imagined what could be done and turned it to a new and complex purpose.

The really new thing about Plebeian lives is that it relies for its very sense on the ability of search engines to find a single word in a sea of text. If we could not sift these sands of time for historical meaning at the click of a mouse, there would be no point in digitising these archives – better to leave them in their original form. But, if we can search them, analyse them, link them across archival spaces that have hitherto been unbridgeable; we as historians can move beyond scripts that retell limited stories of admittedly gripping and dramatic events created essentially for an eighteenth-century audience; to telling life scripts, to building life lines that snake through the archives, to evidence each interaction between a beggar or a criminal, or just a person, and the systems on which they relied, or feared, or avoided - to map a collection of lives.

The Old Bailey Proceedings were a good idea because they brought into the digital age the single most important and classic source for history from below, for that humane tradition of writing from the bottom up, and from the single individual’s perspective.

The point is now to move beyond that tradition, to what is only now conceivable. In my own mind, this amounts to adding a collective rigour to the humane narratives of History from Below; to – if you will excuse the historiographical contrast – forcing Raphael Samuel to sit down with Lewis Namier; and then figuring out what they together might have to say to Karl Marx. In the process, what the project seeks to achieve is a new way of understanding how individual decisions, made by non-elite men and women, contributed to the evolution of the institutions of modernity – of hospitals and workhouses, prisons, courtrooms and local government. And it will do so by tracing people’s lives through this massive archive; and in the process will argue that the agency deployed by working people, the decisions they made in difficult circumstances, help to explain why and how things changed, both in the past and by extension in the present.

But more than a new intellectual aspiration, a new explanation of historical change; this project is also about recognising that the audience we need to reach is different than it once was. The point about building upwards from individual lives, is that it allows us to connect in ways that most historians cannot, to the greatest body of readers and historical researchers ever – to the family historians. By making this project about lives, we generate something they want to read; at the same time as we analyse something we want to explain.

And as the tools of social software gain a new maturity; as Sharon Howard adds a wiki to the Old Bailey site; as we create facilities for our users to provide new content; to comment on, correct and add links to the resources we create; and more tentatively, as data mining and the searching of distant distributed resources becomes more achievable; a growing and organic intellectual project is gradually being brought into existence. It will have its monographs, articles, and authors; but it will also have its blogs and mash-ups, and its communities of users. In the process it will tie the unbounded historical interest in the everyday, to the intellectual exercise of academic history; and create the infrastructure that allows us to communicate beyond the academy.

Essentially, and in other words, what began as an attempt to make a humane, individual history from below easier to research and write; has become an attempt to add a new layer to that tradition; to update it in light of the full promise of digital resources, and to make it fit for the politics of today. The project is now about rediscovering the magical combination of narrative, and individuals, and politics that characterised the best progressive histories of the 1960s and 1970s; and adding to those the magic of digitisation and the internet, and turning the whole lot to a new purpose.

One of the great arts of academic life is to trust that an interesting, important conclusion will emerge at the end of a project, will present itself at the end of five or ten or twenty years of hard slog – and that it will be important even if it turns out to bear little relationship to the question with which you began. I still like the idea with which we started, but I have to admit that it has become a shed load more interesting along the way.