Peer Reviewing

Incredible Generosity

December is supposedly the “holiday” month of gift-giving, but recent experience as a track chair for the 2012 Academy of International Business (“AIB”) annual meeting tells me that the real gift-giving time, at least professionally, is from mid-January to mid-February.  It was during those weeks that so many of my AIB peers gave their time and attention to read and review more than 150 papers submitted to my track for competitive or interactive presentation at this summer’s annual meeting in Washington, DC.  My goal was to get multiple reviews for every submission and get them in about four weeks. Virtually every AIB member submitting a paper got a paper.  Several got more than one paper to review.  More than a few got several papers from me, from other AIB track chairs, and from our counterparts over at the Academy of Management.  Four weeks later, practically everyone had submitted their reviews, whether it was just one or several from those “overfished” reviewers.  It was amazing to observe.  And it’s critically important to preserve and nurture.  So that’s my topic for the month:  peer review and its importance in our professional lives; peer review and the motivations of peer reviewers; and how we might do a little less  “overfishing” of some especially good and generous peer reviewers. 

Peer Review in Publication and Professional Life

Our profession is shot through with peer review.[1]  Think about the publication process.  There is “developmental” peer review –the comments on a first-draft paper you solicit from departmental colleagues down the hall or from former grad school classmates across the globe.  There is “pre-publication” peer review –papers given at departmental seminars, invited talks at other institutions, symposia or conferences like the AIB annual meeting.  Then there is “publication” peer review –the multiple rounds of single- or double-blinded review that more often (in my case) lead to editorial rejection but occasionally lead to publication in journals, conference proceedings, authored books and edited volumes.

 

But wait, there’s more.  “Post-publication” peer review means that the book quarried from your dissertation may get another round of review and write-up in prominent professional journal or newspaper.  The book or journal may be judged for purposes of awarding various “best” prizes:  best of the conference, the year, the decade.  There will be total citation counts as well as citation counts in this, that and yet another citation count database that I’ve never heard of but apparently should have (because my citation counts there are a little “low”).  In other fields, this sort of post-publication peer review applies to databases, films, websites, and software.

 

And then there’s, perhaps, the most important peer review:  “institutional” peer review in tenure and promotion cases.  Department, college and university committees are formed.  Letters are solicited from outside academics.  Stacks of papers, working papers, books, book manuscripts, reviews of the same, works-in-progress, and candidate “statements” get submitted.  That stack gets read, re-read, counted and weighed (metaphorically and maybe literally).  This peer review process usually takes months and multiple meetings.  By and large, our peers do their best to read it all and read between the lines to figure out what the quality of the candidate’s overall contributions to the profession and institution really are, and whether they merit the recommendation of appointment “without time limit” (tenure) or promotion to “full professor” or “distinguished professor” or some other position of extraordinary distinction in the field.  The candidate’s job in all of this is to not let the peer review process consume him or her.  Relax.  Just let it happen.  In the end, they usually do the right thing…usually.

 

Peer Reviewers:  Democratic, Representative and Motivated By…

 

I’m not writing today to attack or defend the system of peer review. (If I were, I might defend the system like Churchill defended democracy, as the “the worst form of government except all the others that have been tried.”)  Instead, let me talk about people behind the peer review system:  the reviewers.  Without them, the whole system of scholarly publication and career development would quickly grind to a halt.  So who are these people?  Why do they do it?  And why do a few of them do so much of it?  The quick answer to the first question is nearly everyone.  To get my track papers reviewed, I drew on a mix of old and new AIB members, located in the US and overseas, from large research universities, small colleges, government-funded research institutes, and even a few businesses –yes, businesses.  It’s a similar story over at the Academy of Management.  The rank and file play an important role in figuring out what is presented in what forum at annual meetings.

 

Other smaller professional meetings may rely less on the vox populi and more on a review committee drawn from the broader membership.  It’s similar with many scholarly journals where experienced and well-published representatives of the profession are invited to join editorial review boards.  The boards are smaller if editors prefer to stay up all night reviewing manuscripts themselves.  The boards are larger if editors prefer to stay up all night keeping track of the reviews board members agreed to complete but haven’t yet.  These journal boards are not democracies –reviewers and their reviews are merely advisory to an editor’s decision—but they are nonetheless representative.  Journal boards are comprised of members from across our global profession.   It’s similar with the choice of outside reviewers in tenure and promotion processes.  The process is far from democratic –committees don’t solicit outside letters from anyone in the profession.  Those tenure and promotion reviewers are leaders in their respective fields, an elite.  There might have been a time when that elite resided at just a few research universities in North America and Western Europe.  Increasingly, that elite is dispersed across many institutions and regions of the world.  So even the pool of tenure and promotion reviewers is becoming more representative.

 

It may be a weekend spent reviewing a candidate package and writing a tenure letter.  It may be a day spent reading and writing a review for a journal.  Maybe it’s 30 minutes on the phone with a grad student talking about the first draft of a paper for conference presentation.  In these and other contexts, it’s useful to ask what motivates these peer reviewers?  I thought about my AIB track reviewers.  They did the reviews so quickly and helpfully, and not merely because I asked.  I’m pretty sure that they saw it as part of their job, as a chance to learn, as their opportunity to “matter” in the process of scholarly discovery.  Maybe they also saw it as a shot at harpooning some big cigar under the cover of double-blind review.  It certainly wasn’t about money.  I think there is little, if any, homo economicus, that is, narrow self interest, motivating most peer review work.  I think there’s much more homo sociologicus at work here.  It’s part of how we define our career, our sense of a professional community.  We’re lucky.

 

Peer Reviewer Overfishing

 

There are a few peer reviewers who do an awful lot of work for the rest of us.  I sit on the AMJ editorial review board and get a new or revised manuscript to read and assess about once every 6-8 weeks.  Other members do it more frequently, much more frequently.  Almost every summer or early fall, a few of those elite professors get multiple requests to write outside letters for tenure review and promotion cases.  And then there’s the AIB anecdote.  A few members got a disproportionate share of reviews.  It isn’t random.  It’s overfishing.  And I’ll admit.  I’m one of the guilty anglers.

 

As our careers develop and we interact with colleagues at different institutions, often through peer review processes, we gain insight on who can and will do the reviewing, do it carefully, do it constructively and candidly, and do it on (or about) time.  What’s their reward?  They get more of it, a lot more of it.  I tend to request more from them at AIB time.  Others ask for their reviews for Academy of Management meeting paper submissions.  Editors seek their reviews for journal submissions.  Boards seek their reviews on grant requests. And then there are the tenure and promotion letters.  A few really great peer reviewers get requests from too many of us.  We’re overfishing.  And the danger is similar what we get when we overfish any pool.  The “stock” collapses.  Overfished reviewers drop out from the process, in part to protect their time for research, maybe also in part because it’s just taxing to judge and judge and judge again.  There’s a burn-out factor that matters over time and intensity of review work.  Nobody wants that to happen.

 

Two Ideas:  Less Rains, More Recognition

 

How can we do a little less of that overfishing?  Here are two ideas.  First, try to be less like Claude Rains, I mean, Claude Rains playing the corrupt but still loveable Vichy Captain Louis Renault in the 1942 classic movie, Casablanca.  More than once in the movie, Rains’ character responds to news of a crime with the cynical command:  “Round up the usual suspects.”  Those of us with a tendency to overfish might do well to think about his words.  We go back to the same generous, reliable peer reviewers perhaps too reflexively.  We’re rounding up the usual suspects.  Don’t, or at least don’t so much.  The AIB review process introduced me to a dozen new scholars providing great reviews.  They’re now on my expanded list of reviewers for future work, in part so that I can move a few of the usual suspects down the list.  Maybe we could all benefit from checking our written or mental list of “go-to” reviewers.  Try adding a few more names so that the “go-to” reviewers you’ve relied on in the past don’t become the “go-to-too-much” reviewers of the future.

 

Second, try to give more recognition for this important work.  I said earlier that self interest probably explains little, if any, motivation for so much of the peer review work we do.  Even so, we like to know that it’s appreciated.  I can’t tell you how much I enjoy getting a note from an associate editor after submitting a review, or getting a letter from the editor-in-chief at the end of the year.  It seems so little but it means so much.  The plaque for best conference reviewer is just a plaque, but it means a lot to new and not-so-new members of an association or institution.  These tokens remind us of what we do to build the profession bigger than any one career within it.  You can’t send too many notes.  You can’t hand out too many plaques.

 

At the end of my work as an AIB track chair, I had the privilege of nominating a few papers for awards.  That was tough.  There were so many I found interesting and worthy of nomination.  I did my best and made my nominations.  There was another nomination I got to make:  best reviewer.  Again, it was difficult as there were many worthy candidates, and they came from institutions around the world.  Again, I did my best and made my nomination.  Congratulations to whoever gets that best reviewer award this summer.  I really appreciated all that you did to help me do my work, help several peers with their scholarly research, and help our profession grow and thrive.  Remember, though, you’re also representing a broader group of tremendously generous people.  In recognizing you with the best reviewer award, we’re thanking them, too.

 

Please address your comments to:

 

Paul M. Vaaler

Department of Strategic Management

Carlson School of Management

University of Minnesota

3-424 CarlSMgmt

321 19th Avenue South

Minneapolis, MN 55455

Tel (612) 625-4951

Fax (612) 626-1316

E-mail: vaal0001@umn.edu

 


 

[1] In working on this blog post, I drew liberally from a collection of terrific working papers by Diane Harley, Sophia Krzys Acord and Sarah Earl-Novell.  In April 2010, they ran a workshop at UC-Berkeley sponsored by the Andrew W. Mellon Foundation and entitled:  Peer Review in Academic Publishing and Promotion:  Its Meaning, Locus and Future.  These working papers can be accessed at this address:   http://cshe.berkeley.edu/research/scholarlycommunication.

 

Author: Paul Vaaler

Professional Life

1 thought on “Peer Reviewing”

  1. Hi Paul,

    I read your thoughts on peer reviewing with interest. I’d like to add another suggestion to the mix – as well as the two excellent ideas you have proposed.

    The reason some reviewers are relied upon more than others is their skill. As you phrase it, they “do it carefully, do it constructively and candidly, and do it on (or about) time”. These people weren’t born with these skills – although part of it will undoubtedly be down to their helpful nature. I suspect this craft has been carefully honed over the years through a mix of support, instruction and experience. In my work with Editors I’ve discovered a real need for some kind of reviewer training or development. Reviewing is something academics are expected to do, but with no real incentive and no real guidance. No wonder it’s tough to find (and keep) good ones! I’d welcome an initiative to teach reviewing skills in institutions, and perhaps link participation to some form of accreditation or qualification. Not only will the skills learned be invaluable, the participation can be recorded on CVs and therefore be an aid for those at the beginning of their careers. I’d be interested in people’s thoughts on the need (or perhaps not?!) of training for reviewers – is this something that would be welcomed by academia? Or are reviewing skills just acquired over time and exposure to the process? I suspect it’s a mix of both.

    As far as incentives are concerned, many journals give an outstanding reviewer award each year to acknowledge and thank these people (see http://www.emeraldinsight.com/authors/literati/reviewer_2011.htm for examples). Some Publishers offer special discount codes to reviewers, or even open up their databases for a limited period. But is this effective – when reviewers will most likely have access through their institutions anyway? Personal and anecdotal recognition has a big role to play, but is there anything else we can do to incentivize and reward reviewers?

    Thanks,

    Kim

    P.S. Scholar One Manuscripts allows Editors to rate reviewers for timeliness and quality – so even reviewers are being reviewed!

Comments are closed.