Wednesday, 15 October 2008

Peer Review Smells.

Most academics who have thought about it will agree that peer review is just rubbish. It is ridiculously hard work - work volunteered by academics as a mulct on their lives. And it is subject to the politics of the playground. Most subjects and academic communities are simply too small to allow true anonymity, and as a result the outcome of peer review is frequently informed by nothing more than hubris and hurt feelings.

My personal bugbears regarding peer review are that, first, it is inherently conservative - encouraging a narrow definition of scholarship; and second, that it forms a comprehensive subsidy on publishers that in turn shifts money that should be spent on either research or dissemination into the coffers of private companies. Universities pay for the privilege of buying journals that are written and edited by their own staff, and reviewed at their own expense. In an online environment, where the costs of printing is decentralised, this makes no sense whatsoever.

At the same time, there are aspects of peer review I like. I want to know that someone with some expertise thought an article was worth publishing. It saves me the time (the seconds it takes to skim a few hundred words) otherwise spent on determining whether the material being discussed is worth the effort of reading.

The traditional answer to these complaints and observations is the care-worn observation that peer review is the worst system ever, except for all the others.

But, with the advent of new models of online sharing and community interaction, it seems to me the moment to reinvent this particularly irritating wheel. Why not create an open 'academic journal space' for all articles that anyone cares to post, with a tagging regime for subject definition. In other words, anyone could post under the tags 'eighteenth-century British history', or indeed 'past & present'. If editors, or foundations, chose to shepherd particular collections of tags and to create an 'identity' from this association, this could certainly be accommodated. Moves in this direction were commonplace in many science subjects in the 1990s, but it hasn't resulted in a reform of the world of journal publication.

The point, however, would be to ask all contributors (all people - even academics - who choose to post a contribution) to also undertake a series of peer review assessments. These would be post-publication, but since online material could be continuously edited, it would nevertheless act in the way current pre-publication peer review acts in relation to polishing an article.

The point, however, would be to rate the assessors. I am more interested in what Natalie Zemon Davis thinks about an article on 16th century France, than I am in the opinion of a third year undergraduate (at least in the first instance). By allowing assessors to themselves build up an online profile you could incorporate a continuum of quality that is made up of the opinions of all assessors, weighted according to authority (i.e. the opinion of three senior figures in a field, could balance out that of 12 graduate students, or 18 unrated individuals, or whatever). The rating of individual assessors, could be determined by what they have contributed in the past, or based in a metrics of cross posting of the sort being used by the Australian and UK higher education systems to distribute research income. There are real problems with metrics, but they do form a consistent measure of community regard (at least in most subjects).

The result would be an academic commons that nevertheless preserves the 'authority' that is the single valuable aspect of peer review. It would also open up academic publishing to a broader community. There is no reason for 'outsiders' not to post material. Much of it might be denigrated by 'high authority' reviewers, making it possible to still select out articles of a certain 'standard', but it would also create a way in of a sort that doesn't frequently exist in the current system. Groups of reviewers who choose to generate a 'journal' identity, could still do so - with the 'journal' being made up of those articles that a defined group of 'assessors' chose to assessn positively.

As an evolving community of assessment and contribution, the system would be self-validating, and a lot cheaper to run than the plethora of journals currently clogging our libraries and lives.

1 comment:

Vinoth Kumar said...

Wiztech Automation is a Chennai based one-stop Training Centre/Institute for the Students Looking for Practically Oriented Training in Industrial Automation PLC, SCADA, DCS, HMI, VFD,VLSI, Embedded, and others – IT Software, Web Designing and SEO.

PLC Training in Chennai
Embedded Training in Chennai
VLSI Training in Chennai
DCS Training in Chennai
IT Training Institutes in Chennai
Web Designing Training in Chennai