The Conundrums of Assessment

To my chagrin I recently realised that I have been assessing research proposals and grant applications for some twenty years, and have done so for most of the major humanities and social science funders in the UK, Europe and North America.  Over the last ten years I have also sat on numerous grant awarding panels, and helped to design the odd funding programme.  I have form in this particular small area of academic life.  So when I was asked for advice about how to write an assessment by a colleague faced with their first request of this sort, I felt obliged to offer some.  And since there is no independent body of advice about how the system works, or how the panels who arbitrate on the final decision use the reports drawn up by assessors, it seemed worthwhile posting that advice:

In the UK, funding bodies have made great efforts to train assessors, and brief them on what is expected; and to all intents and purposes have created a system that seems transparent and clear, with apparently precise criteria laid out in straightforward prose.   A great deal of effort has also been put into the supporting documentation in an attempt to ensure that assessments can be compared against one another, and that the process of decision making undertaken by the panels is speedy and uncontentious.  A real attempt has been made to eliminate special pleading, conflicts of interest, and the administratively perverse.  But, of course, all bureaucratic systems are also cultural systems, and there remain many unstated realities that effectively determine how a grant application and the assessments written in response to it, are read by the people charged with eventually sifting the funded from the unfunded. 

The single most important and determining factor that every assessor needs to keep in mind is that  most funding programmes have a success rate of around 20-25% (some as low as 10%).  From a panel's perspective, four out of five applications must be rejected.  As a result even small issues and problems in an otherwise exemplary application will be used to make a determination.  After reading perhaps thirty solid projects (and these days few applications are less than solid) and when faced with the need to find just five or six to fund, any panel, however well meaning or intelligent its individual members, will begin to reach for the smallest weakness.  

As an assessor you need to be aware of this problem.  This does not mean that you simply laud your favoured project to the skies.  If you do, your assessment will be judged to be insufficiently critical, and therefore worthless.  Instead, it means that if you seriously think a project is likely to be better than 80% of the others, you need to act as a critical advocate, and to place yourself at the heart of the debate that the application you are assessing will inevitably generate.

There will always be one or two applications that sit on the top of the pile, and if you are assessing one of these you can give yourself the freedom to engage with the underlying ideas, and to simply discuss the project's importance for a wider field.  Even in this instance, you might want to suggest where small problems might exist, but have been effectively addressed.

But these few, intellectually exciting and beautifully realised projects are rare and are consistently funded.  Where all the debate will be focussed is over the next tranche of projects.  This usually comprises some 40% of the total.  In a panel meeting (and regardless of how they are organised) the grading schema inevitably breaks down, and this 40% of applications start to bunch around the boundary grades.  Most panels give up on whole number grading of the sort the funders recommend, and end up using some form of 4.257, or 4-+(?) (if they are dominated by older academics from the Russell Group).  If you are asked to assess an application of this sort, the first decision you need to make is whether you think it should be funded; and having made that decision (assuming it is positive) you need to act as an advocate.

In other words, the first thing that any assessor needs to do is make an informed, over-arching judgement about the quality and importance of the application in front of them.  If you think the application you are assessing is compelling, although not so strong as to sit on the top of the pile, then you need to say so, and say why.  Alternatively,  if it is in the bottom 50%, for either technical or intellectual reasons (i.e. not exciting, or not practical, or just not well written) there is little point in expending your time struggling to find something good to say.  It will not affect the outcome, and it is likely to be fed back to the applicant in a way that just encourages them to resubmit something similar, instead of something better.

For myself, I start of with a basic question in my mind:  Am really excited by the project?  Has it caught my imagination, and left me thinking that I would actually want to know the outcome?  Would I want to read the book?  Or in a Knowledge Transfer  context, would it make a significant social difference? 

A minority of  academic projects get past this hurdle.  As a result, the real problem comes with the next stage, which is that while the intellectual case needs assessing (and if you are excited by the project, this is the easy bit) it is essentially all the things around it that will be used to exclude marginal projects.  If the project plan (with methodology and budgeting etc.) looks less than professional and doable, the application will be excluded on these grounds.  But if the fundamental idea is exciting and you decide you want to support it, regardless of its minor faults, you will need to deal with these issues directly and explicitly.  If you don't, the curmudgeon in the corner (and there is always one) will use these small issues to denigrate the project - generally as a stratagem to promote their own discipline or methodology or favoured project.  In this context, if you see a weakness, you need to address it directly and explain why it is not important to the success of the project.

My feeling is that you need to exercise an abstract academic and professional judgement, and in the round (regardless of the hoops and bureaucratic forms the funders want you to jump through) come to a conclusion about the worth of the project.  Once you do this, you are duty bound to do everything you can to ensure that the result is positive, in full recognition that most projects, including innumerable worthy ones, wont be funded.   This can lead to a rather instrumental and manipulative approach (which is a problem), but it at least has the advantage of allowing us to exercise the kind of judgement that is implied in peer review even when the process feels like bureaucratic games playing.

A Review to be published in the Economic History Review

Phil Withington, Society in early modern England: the vernacular origins of some powerful ideas. (Cambridge: Polity Press, 2010. Pp. xi + 298. 23 figs. 14 illus.  ISBN 9780745641300 Pbk. £16.99)

10129

This volume does something new, remarkable and important.  It uses a quantitative approach to the evolution of sixteenth- and seventeenth-century language and word use as the basis for a re-analysis of the significance of corporatism and sociability in the creation of a ‘modern’, and more specifically ‘early modern’ society.  In the process, it attempts to re-integrate the economic and the cultural, the linguistic and the material.

Following an extended and sophisticated account of the development of the profession of economic and social history in Britain since the nineteenth century, organised around the evolution of the phrase ‘early modern’, Withington dives into an entirely innovative form of analysis.  The core of this study is a new mapping of the appearance of a series of key words in the titles of all the books that appear in the English Short Title Catalogue for the period up to 1700.  ‘Modern’, ‘Society’, ‘Company’, ‘Wit’, ‘Civil’, ‘Commonwealth’, among a host of related terms, have been trawled from the full title fields of the ESTC, and transformed into frequency graphs.  These graphs have then been used to illustrate, first, that terms signifying and labelling a specific kind of ‘modernity’ (based in a notion of what Withington terms the ‘sociable self’), became prominent in the latter half of the sixteenth century, and in particular during the 1570s and 80s.   And second, that terms like ‘commonwealth’, which referenced an older form of social ordering, went into relative decline (particularly after the Civil Wars).

Withington’s conclusions essentially re-enforce a growing consensus among historians about the importance of sociability and forms of corporatism in creating a transitional Res Publica, (in this context a kind of beer and skittles ‘public sphere’), that contributed to and resulted from both a newly decentralised but bureaucratic state, and the development of corporate capitalism (with a remarkably sociable scientific revolution thrown in for good measure).  In many respects, and in company with Keith Wrightson, Mike Braddick, Steve Hindle and Andy Wood, Withington is pushing back the origins of Jürgen Habermas’s ‘authentic public sphere’ from the 1690s to the 1570s, and attempting to articulate the relationship between ‘modernity’ (in both its statist and possessive individualist forms) and civic humanism.

As a description of early modern English and British culture, and the evolution of the state and the economy, this is entirely compelling.  Habermas’ chronology, based on coffee and newsprint, has always been suspect even if his overarching analysis of the role of public debate in the history of the nation state remains compelling.  More problematic is the methodology Withington uses to illustrate this new chronology.

As historians we are faced with an entirely new kind of evidence – mass digitised text - billions of words, retrievable through keyword searches.   To make sense of these new resources we need new tools; and this book is a laudable first attempt at creating precisely these.  Unfortunately the methodology used here is essentially unconvincing.  What appears on a title page of a book, on the colophon, and end papers, changed dramatically between the late fifteenth and the eighteenth centuries – the very nature of a book changed.  For all the heroic efforts of cataloguers and bibliographers to force early modern print objects into a single format, to tame their ‘slipperiness’, it is not credible to read the title of a book published in 1520 in the same way as one produced in 1690.  Nor is it necessary to take this aggressively reductionist approach.  Withington could, for instance, have translated the text of his titles in to a formal corpus, and used the tools of quantitative linguistics to chart the rise and fall of his ‘keywords’ against more robust measures of textual density, variety and proximity.   By restricting himself to only the most basic statistical techniques we are left with a series of graphs measuring change in a way that at first sight seems intuitively reasonable, indeed common sensical, but which belie all the subtle complexity that historians have found in language through the many decades of the linguistic turn.  In essence, what Withington has produced is a clear and compelling narrative of the evolution of the ‘sociable self’, and an equally clear series of measures charting the development of the language of title pages, but has not effectively related one to the other.

This will seem a harsh criticism, but it is not meant to be.  As a profession we are confronted with both the real challenge of dealing with massive electronic texts (produced in half a dozen different ways, and of hugely varying quality), and the need to create usable and intellectually credible tools that can both deal with words in their billions, and at the same reflect our new understanding of the complexity hidden in a single phrase.  Dr Withington has taken an all-important first step in the direction of a new form of historical scholarship and we should all look forward to the next.

University of Hertfordshire                                                                 TIM HITCHCOCK

A Couple of Eighteenth-Century Ballads

I have recently been involved in a couple of projects that allowed me to engage with the eighteenth century in a very different way than I am used to. I was involved - in a non-musical capacity! - in helping to create recordings of a couple of eighteenth-century ballads.

The first is a ballad the only copy of which (at least as for as I know) is in the British Library, and is titled The Workhouse Cruelty. I first came across this piece in the early 1980s, and haven't really done anything with it since. But, when I was asked to provide a ballad that would help illustrate a case about a murder in a workhouse for Voices from the Old Bailey on Radio 4, it immediately sprang to mind. The nice thing is that the producer, Elizabeth Burke, then went out and had a recording of it made. The result is here:




This particular recording seems a little sweet to me, and lacking in the political grit of the original rough printed version. And I suspect that it was originally sung by a man - of the sort known as a 'chanting' ballad singer (they normally specialised in durge-like religious songs). But it nevertheless made me want to think harder about how one interprets the words, and how one re-constructs the soundscape in which it was performed.

This then encouraged me to have a go at commissioning a recording myself. Francis Place's papers contain the words of a dozen or so, primarily bawdy, ballads he recalled from his youth in the 1780s. The really nice thing about Place's notes, is that he described where he heard them sung - behind St Clements church, etc - and by whom - two young women - and when - in the evening. The ballad I was particularly interested in was Jack Chance, which Place describes as being sung just after the Gordon Riots. As I was giving a lecture on the Riots, it seemed a natural thing to accompany it. I was also familiar with a printed version of this particular song, mis-dated at 1795, and retitled as Just the thing, among the digital collection at ECCO. I integrated the two versions to eliminate some of the blanks in Place's version and asked a friend of my son's, Henry Skewes, to write the music. Unlike most 18th century ballads, no tune was mentioned as being used with this one. Henry wrote the music, and asked another friend Stephanie Waldheim to sing it. The upshot is:

Jack Chance: Or Just the Thing

Music and arrangement by Henry Skewes;
performed by Stephanie Waldheim and Henry Skewes
copyright: 2010, Henry Skewes, Stephanie Waldheim and Tim Hitchcock.

This version has been translated from its original AAC Audio format to a mp3 format, and has developed a few oddities, but you get the point.

One way or another, this experience has taken me back to Bruce Smith's wonderful, but seldom cited, The Acoustic World of Early Modern England: Attending to the O-Factor. It has also reminded me that there is a lot to do to recreate a neighbourhood soundscape, but that if one could, it would help; and that perhaps it is time to have a go.

Can we make a thesaurus of meanings for digital humanities?

In an idle moment I have been re-reading the introduction to Roget's Thesaurus, and have been struck by what a thesaurus actually does. It breaks up all words in to one of eight categories and assigns a number to each category. "Affections", for instance, is class eight. It then subdivides these categories in to more sub-categories, with "cheerfulness" falling under "Pleasure and Pleasurableness", and being assigned a number of 868 - after discontent (867) and before solemnity (869). If you then look up 868 - cheerfulness, it is divided again into 20 further subcategories, with around 15 words in each category. So, "gladden" falls under 868.6, and as it is the second word in the category, could be expressed as 868.6.2. The second half of the thesaurus simply lists all these words in alphabetical order to allow the user access to the hierarchy of meaning in the first half.

In other words, a thesaurus assigns a numerical value, that equates to one or more category of meaning for every word it contains.

If you took a large body of text - ten years of the Times, or everything published in 1840 - and broke it up into a set of words, and assigned each word a number from the thesaurus's hierarchy, you would end up with a unique numerical representation of the collected meaning of the words that make up the text.

If you take the sentence 'the sunlight warmed his forehead' and convert it into its thesaurus equivalents, you end with 334.10.7; 328.17.3; 239.5

It can be equated with: "The midday sun cooked his brow", which is 334.10.11; 328.17.20; 239.5

This would allow a kind of semantic search that was not dependent on direct context, and would not be universal, but would work perfectly well for historical text in English (all the better suited to historical material, as Roget was a late Enlightenment figure, and his categories map well on to historical text). There would remain an issue of disambiguation (making a distinction between "clip" as a noun, versus "clip" as a verb); but this could be mitigated either though mathematical approximations (you could create a third unique number that essentially averaged the two or more meanings assigned to any single word), or you could simply live with the errors generated, on the assumption that the historians are used to filtering their own reading. You could also apply the chronological data contained in the Oxford Historical Thesaurus to map how close (or distant) an individual text is to standard usage for a particular period; or how different genre relate to a standard evolving language (how literature vs law treatises map onto accepted usage in the decade they were written).

As we are confronted by massive text objects (I think the notion from linguistics of a corpora is less useful for historians who are seeking to find information, rather than define bodies of text), the ability to locate related or similar text across genre and texts is important. It would also be another way of approaching the measurement of "distance" between texts.

Alternatively, and this is closer to Roget's original scheme you could use this numerical labelling as a basic form of computer aided reading. You could, for instance, assign a colour to each broad category, and a shade of that colour to each sub-category; allowing you to identify the work that different parts of the text is doing, through a simple visual examination. When skimming through a large body of text, at perhaps 40 pages on a single screen, the colour coding would allow you to identify areas in which "affections" are directly discussed, or any of the other thesaurus categories - "Space", or "Physics" or "Matter".

Have I missed something - Does anyone know why we don't use a thesaurus based numerical hierarchy to code meaning in large texts? It would give you a "word" = "a set of numbers" (an unique number for each major word) in a paragraph or sentence or text division, which could then be compared statistically, or colour coded to reflect the breadth of meaning found. Or you could colour code for types of words and locate relevant sections in a large text in a particular colour. It just seems dead obvious as a way of moving towards the ideal at the heart of the semantic web, while avoiding the creation of 'triples', and the ever retreating promise of universality. My guess is either that librarians having been doing this for the last thirty years and not telling me (librarians are cruel that way), or that the rest of the world forgot to read the introduction to Roget's Thesaurus, which is also possible. Of course, the final possibility is that I don't understand the semantic web; and that ontologies in aggregate are already a form of thesaurus.

A New History From Below

A few years ago I argued in a review essay in History Workshop Journal for the need to rethink the history from below tradition, to take into account both the changing nature of how we find information on the internet - what the creation of new resources makes possible - and also how Western European progressive politics have changed over the last twenty years. Coming on the heels of the posting of the Old Bailey Online I suspect this argument struck some as a bit hubristic. Several historians certainly expressed the view that there was nothing wrong with the original version, and that I was, in any case, not in a position to change it. A couple of years later, in a remarkably sneering (and ill-informed) review of two books based on the Old Bailey, books that were preliminary experiments in trying to create a new history from below, Nicholas Rogers lambasted and mocked that work and intellectual direction. This response made me feel rather wary of publishing much of anything, for fear of offending a generation of historians whose work I respect, but who don't seem to warm to much of anything new. Instead, I just got on with the task of posting large bodies of historical manuscripts on the web, on the assumption that democratic access to primary sources was an unproblematic good thing, and that perhaps I should leave intellectual innovation to people like Nicholas Rogers.

But, I was recently invited to Brussels by Hugo Soly, to give a lecture on my work to his MA class, and as a part of this, had to revisit a 'new history from below' and attempt to explain what I had meant, in terms that would make sense to an audience more interested historiography than technology. Where I ended up was re-convincing myself that we do need a new history from below, and that there is an opportunity to create a form of history that engages with the present, makes proper use of online resources, and which moves with both the technology and the politics of now.

What I said to Hugo Soly's students was that a new history from below is new, and needs to be new, for precisely three reasons.

First, because the relationship between the individual and the state has changed. It seems to me that the Western European mixed welfare state is as good as it is going to get for the moment, and so we need a history from below that is not focussed solely on raising political consciousness in strictly idealogical terms; but instead takes as its object how the individual forces the state to deliver the goods of a good life. As a result, this ‘New’ history from below, should be precisely focussed on how paupers and prisoners, the poor and the benighted, navigate the emerging institutions of the modern state – and how their behaviour (and agency) shape those institutions. Everyone knows that a school, or a prison, or a hospital, or a university, is a subtle compact – a conspiracy – between guard and prisoner, doctor and patient, teacher and student. This new history from below should be about how to write about the prisoner, the patient and the student when they speak to power.

Second, it is new because the one lesson that the interminable ‘linguist turn’ should have taught us was that language is a technology. If we now know the subtle stratagems of textuality, then we ought to be able to use those strategies in a more self-conscious way, to self-consciously manipulative the reader. In other words, what is new, is the recognition that it is not enough to write truth, or to do good research; and that what is needed is writing that makes the best use of the technology of emotions and representation – how you use words and pictures and a story to impact, not just on what people think, but on what they see in their mind’s eye. I always come back to the notion of a simulcra – the idea that literary representation is made up of a few fragments of information that are used to represent the whole. In history writing, in which the details about a single poor person might add up to nothing but a few lines in an account book, that notion, that idea that you can use a single detail to represent the whole, becomes even more important than in fiction. When you add, for instance, the colour of a jacket, or the weather on a given afternoon, to a bald list drawn from the boring records of administration – it brings someone to life. In some ways this is just about good writing of a sort historians used to value, but given the mountains of poor writing that are published every year, by academics who think their sheer brilliance makes up for their deadly prose, this remains new.

And finally, the most obvious newness, is the internet. In the last ten years the nature of what can be found has changed out of all recognition. In the London Lives project, and The Old Bailey Online, in the Burney Collection and Parliamentary Papers Online, we have a remarkable haystack, and a powerful magnet with which to search out its needles. And if you add to that Google Books, The Times Digital Archive and all the rest, suddenly for eighteenth century London at least, 80% of every word published, and 10% or 20% of every pen stroke, every manuscript, can be searched electronically. This means the archive, the search, can be thoroughly re-configured around groups of individuals, rather than archives themselves. London is uniquely well-served in this regard, but the infinite archive of electronic texts is building. And it is being used by people in their millions. The internet, in other words, both creates a new audience, and is a way of re-configuring research from the archive and the institution to the individual – to the pauper, the prisoner and patient.
Revisiting this material has made me more optimistic about history writing, and what is possible, and it seemed a good moment to set out a stall, and see if anyone is interested.