Judging a book by its URLs

It will sound odd, but I have recently had a great time editing URLs.  Robert Shoemaker and I have have just finished a book for CUP, derived from the London Lives project, and called - London Lives: Poverty, Crime and the Making of a Modern City, 1690-1800. It is a long book (170,000 words) and each quote and reference in it is linked via a URL to the original document or article, book or web-resource used as evidence or to contextualize the argument.  It will be published as both an ebook and in hard copy, and the links need to be robust, and secure.  My estimate is that there are in the region of 4,000 URLs included in the manuscript (which was written collaboratively in PMWiki).  In the end, I found that I could identify an appropriate link for 98% of all footnote references, but then had to eliminate around 10% of these, as the relevant URL was just not useable.  The book took some nine years, and I am glad it is finished.

One of my final jobs was editing those 4000 URLs.   It took about three months work, spread over the last year, and I have just finished spending a week or so confirming what I hope will be their final form.  When I have told people about this work many have looked incredulous and suggested that this is the sort of technical implementation process that should be left to others.  A couple of otherwise nice people have suggested I dump this job on the shoulders of the nearest PhD student.  But for myself, it is precisely the kind of thing that an author should do for themselves.  And in doing it, two things kept coming to mind.  First was how the role of the scholar in creating a rigorous academic apparatus is a central part of the intellectual journey that academic writing involves - and that we should see the implementation of the online version of this in the light of the precise writing of footnotes and references that mark out good scholarship.  And second, that URLs encode a system of design and intent, online architecture and system of access, that signal the quality and permanence (the academic credibility and perceived audience) of historical materials online.  And that just as we have always sorted and judged scholarship by its form, we should think a bit harder about how the form of a URL can let us interrogate online materials.

On the first point, I do not know of much discussion of the joys of this kind of academic slog.  There is a lot of good writing on research and archives (by Carolyn Steedman and Arlette Farge among many others), on writing and thinking, but no-one talks much about the painstaking labour that goes in to turning a rough draft in to a final finished piece of scholarship.  And here I am really talking about generating accurate and fully comprehensive footnotes that reflect both the material cited, and the research journey that resulted in the main text.   This has become much easier with online catalogs and citation management packages, but nevertheless remains laborious and a reflection of our collective and individual commitment to a particular kind of evidenced discussion.  But for me it also represents my favourite compromise.  The writing of history is a wonderfully imaginative and creative process.  And in some respects we wish to judge the product of history writing as art.  Is it enjoyable to read? Is it convincing?  Does it do the job of good writing in liberating the readers' imagination?  In making these judgements we tend to appeal to a notion of 'value' that is cultural and that privileges dominant forms of authority.  This aspect of judgement is essentially romantic; with all the implications for western and elite hegemony embedded in that idea.  At the same time history writing is the result of simple hard work of a more technical kind - in the archives, in collating and collecting, re-ordering and interrogating data.  And it is valuable because it encompasses that hard work.  The beauty of the academic apparatus is that it evidences this and in the process generates a different measure of value.  In other words it is where quality is tied to a 'labour theory of value'.  I love the academic slog because it is where un-moored judgement is tied down to hard labour; and where value can be universalized in a common human experience (work).  In other words I really enjoyed editing 4000 URLs precisely because in them and their associated footnotes lies a claim to and evidence of the hard labour that underpins the book itself.

 At the same time, the process also taught me to read URLs differently.  Clearly coders and web designers do this as a matter of course.  But I am a historian and want to read URLs as a scholar, rather than as a programmer or designer.  And for me, the important thing is that URLs embed the structure of a site, making it plain to see for anyone willing to look hard; and that they are made up of both the character of a library reference, and a command directed at the new technology of discovery - the Internet .  There are just lots of different types of URL.

There are 'Search URLs' that include all the elements that  take the user past a collection to a specific object, but don't let you go directly there without the query.  And there are URLs that encode a cataloging hierarchy.  There are URLs that sift data, or work in your browser to change the data delivered, highlighting phrases or sifting material.  And there are URLs that encode licensing, passwords, and access information.  It is easy enough to find that the whole search journey that took you from a library catalog to an individual item is encoded directly in the URL, and even personalized to you, the machine you are using, or the forms of access you can deploy.  It is easy to find URLs that run on for hundreds of characters, each element divided by a '&' or a '%', or such.

But in creating robust reproducible links to credible historical materials most of these URLs are at least problematic if not useless.  If they include details for institutional access, or session information, they cannot be re-used by someone else.  These URLs are friable and fragile things and not fit for scholarly purposes.  And as a result, for the London Lives book we have been forced to eliminate all the links we originally hoped to include to forty or fifty different sites.  To take a single example, most archives structure their online collections with search in mind, making it difficult to link to a single item.  I spent a lot of time finding the catalog entry for every manuscript we cited in the London Metropolitan Archives, and Westminster Archives Centre, only to regretfully strip out the links when confronted by a complex URL that just did not look credible as a long term citation of the item itself.

Even in its simplest, and in the form recommended by the site for sharing a link, a London Metropolitan Archives URL looks like this:

http://search.lma.gov.uk/scripts/mwimain.dll/144/LMA_OPAC/web_detail/REFD+P69~2FBRI~2FB~2F001~2FMS06554~2F004?SESSIONSEARCH


Since we had consulted these items in their physical form in any case, it did not seem too problematic to leave out these links, but a shame nevertheless.  And likewise, with paywall material there seemed little point in dangling real access, and the promise of credible evidence, before the eyes of readers who would not be able to go beyond the login screen.  It seemed better to cite a specific item in combination with a general (unlinked) URL and date of consultation as reflecting our own research journey, rather than to promise access when we could not deliver it.

With few exceptions the URLs that have been retained (and there are still 4000 of them) address specific items with a specific ID, and usually run to 20 to 40 characters.  DOIs are not bad once you figure out their structure and reformulate them as they should be, rather than the way they are normally cited on journal web pages.

dx.doi.org/10.1353/sec.2010.0268

And Google Books creates a very nice URL once you strip out all the complex formatting instructions that are normally generated as part of a search and inserted after the main ID.  This is what a Google Books' URL looks like if you were to use the 'search' version:

 http://books.google.co.uk/books?id=1sMJGt7_rTAC&printsec=frontcover&dq=%22Prosecution+and+Punishment:+Petty+Crime+and+the+Law%22&hl=en&sa=X&ei=rrzGUq_aDsSy7Aa_9YGQCg&redir_esc=y#v=onepage&q=%22Prosecution%20and%20Punishment%3A%20Petty%20Crime%20and%20the%20Law%22&f=false

And this URL will take to the same book:

 books.google.co.uk/books?id=1sMJGt7_rTAC

 And the Eighteenth-century Short Title Catalog generates some of the most elegant URLs I have found:

estc.bl.uk/T174945

And to a lesser extent, so does the Ethos collection of doctoral theses at the British Library.

ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.354762

And London Lives and the Old Bailey Online do pretty well on this score:

www.londonlives.org/browse.jsp?div=LMSMPS501980014
http://www.oldbaileyonline.org/browse.jsp?ref=t17910413-19


In part, I suspect that these issues would all disappear if I had a better sense of the layer of structure that lies beneath the WWW.  But for the moment I am keen to have a short, human-readable URL that looks like it will last longer than the session I am currently logged on for.   All of which simply takes me back to the joy of academic slogging and the importance of the academic apparatus as something that evidences hard work and opens up scholarship to credible criticism that goes beyond simple romantic appreciation and prejudice.

I know all too well that one of the skills of an academic is the ability to judge a book by its cover and the form of the text it contains.   For the online we need to embed URLs into precisely this process - and the joy of all that editing was that at the end of it, I feel I have learned to do just that.

OA in the UK

Between the Finch Report in the UK, and more direct moves towards a comprehensive Open Access policy for government funded research in the US, it seems we are confronted by an important moment in the evolution of the ecology of scholarship within which academic history writing is produced and published.     

A recent one-day colloquium sponsored by the Institute of Historical Research and the Royal Historical Society was called precisely to bring together major institutional players (Scholarly Societies, Journals, and Publishers) for a conversation about the best ways forward (the Tweet stream is here).  The general feeling seems to be that while every well-meaning historian is keen to promote Open Access (a show of hands at the conference confirmed this), the Gold Route, whereby authors and institutions are asked to shoulder the cost of peer review and publishing, is just not workable in the humanities.   

There is also the beginnings of what many feel is an apparent solution to the problem.  Both the past and present presidents of the Royal Historical Society, and some 21 editors of major humanities journals, signed a letter proposing the imposition of an increased embargo period on the articles in their journals - essentially suggesting they be allowed to have three years in which to make money on their publications before being forced to make them available through Open Access.   They also proposed maintaining a two-track system to ensure that overseas and non-academic authors are excluded from the government led requirement for Open Access.

To me this feels like Saint Augustine's plaint: "Grant me chastity and continence, but not yet." (Confessions 8:17).

Let me be clear, though.  I understand completely the anxieties motivating these institutions and commentators.  A narrowly defined Gold Route process of the sort privileged by the Finch Report is not workable in the humanities.  The 'author pays' model is predicated on the direct funding of research by government, and on the assumption that the consumers of research outputs are the same as the producers.  In the case of history this is not true. 

 The vast majority of historical research and publication is not funded by project grants; and while a higher proportion is funded through the Universities, and through QR, there is still a large body of excellent work that is undertaken by independent scholars, or as part of a self-funded PhD, or by staff in institutions which do not receive QR funding or participate in the REF.  And similarly, all historians seek to reach a wider audience than most scientists, and imagine their work in the light of a successful 'trade monograph'; which itself forms a recognised academic achievement. 

In other words, I largely agree with the diagnosis that the main thrust of the Finch Report is unworkable.  Though, of course, the Report does not restrict academics to the single route to Open Access, and makes it clear that other types of OA (Green Route) are entirely consistent with the objective of making publicly funded research available to the public.  Following on from this, I also believe the RCUK policy to cover the new costs entailed through grants is also largely unworkable, and if poorly executed in pursuit of a narrow Gold Route form of publication will create issues of fair access, with institutional meddling in academic decision making and serious problems for post-graduates and early career academics. 

What is missing in all this is any positive model of Open Access publishing that takes seriously the fundamental interests and values of history as a discipline, as opposed to the interests of the collection of institutions and journals that purport to speak for it.  For myself I have a clear sense of what I would like Open Access in history (and more broadly in the academy) to look like in ten years' time; and it would have the following characteristics:

•   It would be built on the deposit of articles and research data (including notes) in institutional repositories, linked to APIs that allow their content to be re-'published', mashed up and re-used (with acknowledgment).
•  The route to 'publication' would include the initial deposit of research materials, followed by the posting of a 'rough draft' for comment and revision, leading to a post-publication peer review system.  The author would then be allowed to specify at what revision the 'article' is complete, with perhaps a six month norm for revisions.  See for instance the History Working Papers Project.
•  Metrics for downloads, re-use and citations for the now online article will be used to generate a measure of scholarly importance.  These metrics can include the kind of complex systems for assessing 'authority' (i.e. whose post peer review assessments are worth most) implicit in the Altmetrics movement.
•  'Journals' will be made up of adopted 'articles' that fit their theme and which are moulded to a particular house style in the open peer review process.  In this ecology of scholarship journals will take on a new intellectual role in shaping debate and argument, and in defining academic communities, and will have a 'promoting' role rather than a 'publishing' role.
•  Academic monographs will be seen as a simple extension of article publication - i.e. either in the form of long articles, or perhaps as a collection of pieces created as 'articles'.
•  Genuine 'Trade History' will continue to be sold in the generic forms of biography and narrative etc., but the underlying academic historical content will be available in institutional repositories, while the revisions and adaptions for a popular audience are dealt with separately.
•  The co-archiving of secondary writing and notes and research materials will allow for the creation of an increasingly vertically integrated form of writing, in which source material and commentary are connected.
•  The costs of maintaining, curating and archiving the system will be borne by the Universities, with savings from the journal and book purchasing costs.   A separate tNA or British Library repository will support and archive the work of otherwise unaffiliated scholars.

Getting to this point is not straightforward.  But the universities are fully empowered to use the RCUK funding to beef up their repositories, rather than paying journal fees.   The repositories could also take a first step towards a more rigorous ecology of scholarship by archiving and making available the underlying research data we all endlessly collect (and jealously guard as the capital of an ego driven system of professional advancement). 

At the moment most public debate and effort seems to be devoted to preserving the current business model that underpins the public/private partnership that lies at the heart of academic publishing. The journals worry that their main income stream (allowing them to provide studentships etc) will be eliminated; while the publishers worry that their privileged position between subsidised creators of content and subsidised buyers of content will be squeezed.  Both these anxieties are justified.  

 But, we need to ask ourselves whether we really want to use the roundabout and expensive route of generating income from University Library budgets via the publication of materials produced by academic staff, to take money from the providers of education - the Universities - in order to give it to the journals and scholarly societies, in order to allow them, to in turn purchase education from the Universities.  It is ridiculous.

As for the academic presses, they have spent thirty years squeezing the 'added value' from their operation.  In-house copy-editing and proof reading for the most part went ages ago.  And many presses now demand what amounts to 'camera ready' copy.  If the presses do not want to serve their traditional role in an ecology of scholarship (sifting and polishing its products), then it is not clear what their profits are based on.  At the moment, the greatest input on the part of the presses lies in advertising and licensing content, policing its re-use and in producing hard copy versions of books and articles that are largely unwanted (ask any librarian).  A thoroughgoing Open Access model eliminates the need for selling, licensing, and policing, while time will take care of the romantic attachment to wood pulp.

Current debate seems most fully motivated by a reactionary and defensive fear that a change in the nature of academic publication will unravel the systems of authority and organisational finance that used to deliver public debate.  But, if we have faith in the importance of the academy and of scholarship, then we need to continually re-invent the process.  Open Access provides a perfect opportunity to reconnect with the founding principles of the academy.

The Conundrums of Assessment

To my chagrin I recently realised that I have been assessing research proposals and grant applications for some twenty years, and have done so for most of the major humanities and social science funders in the UK, Europe and North America.  Over the last ten years I have also sat on numerous grant awarding panels, and helped to design the odd funding programme.  I have form in this particular small area of academic life.  So when I was asked for advice about how to write an assessment by a colleague faced with their first request of this sort, I felt obliged to offer some.  And since there is no independent body of advice about how the system works, or how the panels who arbitrate on the final decision use the reports drawn up by assessors, it seemed worthwhile posting that advice:

In the UK, funding bodies have made great efforts to train assessors, and brief them on what is expected; and to all intents and purposes have created a system that seems transparent and clear, with apparently precise criteria laid out in straightforward prose.   A great deal of effort has also been put into the supporting documentation in an attempt to ensure that assessments can be compared against one another, and that the process of decision making undertaken by the panels is speedy and uncontentious.  A real attempt has been made to eliminate special pleading, conflicts of interest, and the administratively perverse.  But, of course, all bureaucratic systems are also cultural systems, and there remain many unstated realities that effectively determine how a grant application and the assessments written in response to it, are read by the people charged with eventually sifting the funded from the unfunded. 

The single most important and determining factor that every assessor needs to keep in mind is that  most funding programmes have a success rate of around 20-25% (some as low as 10%).  From a panel's perspective, four out of five applications must be rejected.  As a result even small issues and problems in an otherwise exemplary application will be used to make a determination.  After reading perhaps thirty solid projects (and these days few applications are less than solid) and when faced with the need to find just five or six to fund, any panel, however well meaning or intelligent its individual members, will begin to reach for the smallest weakness.  

As an assessor you need to be aware of this problem.  This does not mean that you simply laud your favoured project to the skies.  If you do, your assessment will be judged to be insufficiently critical, and therefore worthless.  Instead, it means that if you seriously think a project is likely to be better than 80% of the others, you need to act as a critical advocate, and to place yourself at the heart of the debate that the application you are assessing will inevitably generate.

There will always be one or two applications that sit on the top of the pile, and if you are assessing one of these you can give yourself the freedom to engage with the underlying ideas, and to simply discuss the project's importance for a wider field.  Even in this instance, you might want to suggest where small problems might exist, but have been effectively addressed.

But these few, intellectually exciting and beautifully realised projects are rare and are consistently funded.  Where all the debate will be focussed is over the next tranche of projects.  This usually comprises some 40% of the total.  In a panel meeting (and regardless of how they are organised) the grading schema inevitably breaks down, and this 40% of applications start to bunch around the boundary grades.  Most panels give up on whole number grading of the sort the funders recommend, and end up using some form of 4.257, or 4-+(?) (if they are dominated by older academics from the Russell Group).  If you are asked to assess an application of this sort, the first decision you need to make is whether you think it should be funded; and having made that decision (assuming it is positive) you need to act as an advocate.

In other words, the first thing that any assessor needs to do is make an informed, over-arching judgement about the quality and importance of the application in front of them.  If you think the application you are assessing is compelling, although not so strong as to sit on the top of the pile, then you need to say so, and say why.  Alternatively,  if it is in the bottom 50%, for either technical or intellectual reasons (i.e. not exciting, or not practical, or just not well written) there is little point in expending your time struggling to find something good to say.  It will not affect the outcome, and it is likely to be fed back to the applicant in a way that just encourages them to resubmit something similar, instead of something better.

For myself, I start of with a basic question in my mind:  Am really excited by the project?  Has it caught my imagination, and left me thinking that I would actually want to know the outcome?  Would I want to read the book?  Or in a Knowledge Transfer  context, would it make a significant social difference? 

A minority of  academic projects get past this hurdle.  As a result, the real problem comes with the next stage, which is that while the intellectual case needs assessing (and if you are excited by the project, this is the easy bit) it is essentially all the things around it that will be used to exclude marginal projects.  If the project plan (with methodology and budgeting etc.) looks less than professional and doable, the application will be excluded on these grounds.  But if the fundamental idea is exciting and you decide you want to support it, regardless of its minor faults, you will need to deal with these issues directly and explicitly.  If you don't, the curmudgeon in the corner (and there is always one) will use these small issues to denigrate the project - generally as a stratagem to promote their own discipline or methodology or favoured project.  In this context, if you see a weakness, you need to address it directly and explain why it is not important to the success of the project.

My feeling is that you need to exercise an abstract academic and professional judgement, and in the round (regardless of the hoops and bureaucratic forms the funders want you to jump through) come to a conclusion about the worth of the project.  Once you do this, you are duty bound to do everything you can to ensure that the result is positive, in full recognition that most projects, including innumerable worthy ones, wont be funded.   This can lead to a rather instrumental and manipulative approach (which is a problem), but it at least has the advantage of allowing us to exercise the kind of judgement that is implied in peer review even when the process feels like bureaucratic games playing.