Peer(less) Review: Whither Art Thou?

Peer review, especially peer review of chemical safety/risk assessments, is under assault.  Despite the fact that government agencies, industries, universities, NGOs and consulting groups recognize the added credibility that peer review brings to their chemical safety assessment in the eyes of each other, and despite the fact that engaging outside experts to review chemical safety assessments and assessment methods invariably helps insure high quality and scientifically-defensible work products, the process is often challenged. 

Why is this?

Many governments and industries conduct scientific research on the toxicity of chemicals as part of a necessary process to bring products to market.  This toxicity information is used to develop safety/risk positions for individual chemicals and often their mixtures in order to protect human and environmental health.  This research is often done in a transparent and collaborative fashion, or at least is obtainable after government deliberative process is complete and confidential information is redacted.  Universities, NGOs and consulting groups are valuable assistants in this government and industry endeavor. Notwithstanding the fact that many toxicology studies have scientific uncertainty, a clear exposition of the relevant facts allows informed debate and reasoned judgment over a chemical’s safe or virtually safe dose.

This informed debate and judgment is part of any scientific endeavor, of course, and is a constant feature of the scientific process.  Part of this process is the recognition that scientific issues can be complex and not any one person, or often even any one group, can corner the market on relevant information or analysis.  This is especially true with toxicology and epidemiology studies and the safety assessment of individual chemicals.  One only has to look at the length of time it takes for some safety assessments to be completed to get a sense of the complexity of potential issues.  A peer review broadens the scientific involvement in these assessments and offers the potential for a higher quality and more scientifically defensible result.

Or does it?

The first response lurking behind this simple question depends on what folks mean by the use of the word “peer.”  Naturally, several definitions are available.  For example, the Cambridge dictionary states a peer to be:  a person who is the same age or has the same social position or the same abilities as other people in a group.  For chemical safety/risk assessment in particular, the definition of peer has been explored by Patterson et al. (2007) who states peers to be:

“…experts who collectively are of equal standing, that is, those who have at least the same level of training and experience as the authors of the report. Therefore, they are qualified to evaluate and judge the adequacy of the authors’ science and conclusions.”

It follows from either definition that persons who do not otherwise “have at least the same level of training and experience as the authors of ” the specific chemical risk assessment under review are not included in the definition of “peer,” and that opinions from such experts are peer(less).  In brief, these peer(less) opinions should not be afforded equal standing with those offered by peers.  Part of the questioning one often hears over peer review meetings is that certain experts are not qualified for the review.  In short, they do not have equal standing as peers.  When any particular peer review panel has few if any “peers” compared with the authors of the chemical risk assessment, the results can lead to a less scientifically appropriate result.

Patterson et al. (2007) go on to define differing types of “peer” meetings, the confusion among which also leads to peer review being questioned.  For example, one might hear that a peer “review” meeting engaged stakeholders, followed by disparaging comments.  Well, if it was a peer input meeting to develop appropriate focus, data acquisition and identification of issues, then this would be entirely appropriate and the disparaging remarks are uninformed.  A peer consultation on the other hand, is most helpful when a chemical risk assessment is complete enough to benefit from a review, but the analysis may still be in flux, allowing the experts’ comments to be readily considered and influence future direction.  Good examples of peer consultations are found with the Voluntary Children’s Chemical Evaluation Program of U.S. Environmental Protection Agency and Toxicology Excellence for Risk Assessment.   Peer review is a more formal, external, and independent review of an intended final work product.  Such reviews are often the focus of questioning since the intended final work product often has societal impact.  A primary focus in this questioning should be whether the reviewers have sufficient erudition in chemical risk assessment to be considered peers. 

See the table for a summary of these differing types of peer engagement.

Three types of peer involvement (Patterson et al., 2007)

A second response to the question of whether a peer review garners a higher quality and more scientifically-defensible result is that the review also depends in part on the underlying political process, which in turn depends in part on the press’s reporting of the process.  In an effort to be balanced, the press often gives activist environmental working groups equal standing to toxicologists and epidemiologists whose careers span decades of safety/risk assessment work.  But such groups routinely release information that can be readily shown to be inaccurate, leading these toxicologists and epidemiologists to lament the tyranny of misinformation in public discussions of their work.  The tactics of these activist groups and the environmental defense fund are deplorable, despite the fact that the occasional effort of such organizations can be seen to hit the nail squarely on the head.  Fortunately, several groups, such as ours, are available to counter this misinformation. The press, and the political process would be wise to rely more on these latter groups.  Good peer reviews, after all, depend on credible information from all stakeholders. 

A third response to the question of whether a peer review ends up with a higher quality and more scientifically defensible result is one of avoiding conflicts of interest and balancing reviewer biases.  In fact, the most common criticisms leveled against some peer reviews is that the panel contained one or more scientists from a sector that some other sector did not think was appropriate, or that the panel was not balance in the necessary expertise.  Both criticisms should be carefully checked.

Managing conflicts of interest, usually financial, is relatively easy.  Organizations developing peer review panels often set a financial limit in dollars to the potential conflicting interest.  Potential panelists above this limit are excluded, with occasional exceptions because the expert may bring to the table knowledge critical for the peer review.  Folks below this financial limit are normally included in the pool of potential panelists. 

In contrast, managing biases in peer review is hard.  This is because all of us have one or more biases.  Fortunately, several criteria have been developed to do this management.  The first and foremost criterion is that of necessary scientific expertise.  For chemical safety assessment reviews, the necessary expertise normally falls into three categories: the chemistry to be reviewed, the specific toxicity that has been observed, and the experience of conducting similar assessments.  The most successful chemical risk assessment peer review meetings balance these three types of expertise by roughly thirds of the panel, with the panel chair usually being a scientist with experience of conducting similar assessments.  (Click here for a compilation of meetings run in this fashion).  A second balancing criterion is sector representation where government (both state or provincial and federal or tribal), industry, university, NGO and consulting groups are represented.  If the meeting is open to the public, then time should also be allotted for public comments, including the comments of stakeholders, usually near the start of the meeting to allow for panel consideration.  A third balancing criterion is personal characteristics.  This includes gender, race, ethnicity and disability.  Finally, peer review meetings should never be held over any group’s major religious holiday. 

The U.S. Environmental Protection Agency Inspector General has compared six organizations’ peer review effort along some of these criteria. 

So is something wrong with the peer review of chemical safety assessments in this country?  Well, no and yes.  No, because many groups aspire to follow established guidelines in this area and conduct successful peer inputs, consultations and reviews.  Successful meetings include panels of peers without conflicts of interest that are balance as to the chemistry to be reviewed, the specific toxicity that has been observed, and the experience of conducting similar assessments.  These panels are also balanced in biases, sectors, and personal characteristics.  

But yes, peer review has some problems.  Some panels do not have a sufficient number of peers to offer credible comments, or whose peer(less) panelists offer comments that would otherwise degrade the assessment if accepted.  Also some panels are not balanced in biases, sectors and/or personal characteristics, and some panels contain scientists with known conflicts of interest. 

Public health is best served when peer reviews are conducted in a credible, balanced and, as much as possible, open fashion.  We would all be wise to insist on this, but in conducting these reviews to also remember the words of our scientific predecessors…

It is the mark of an instructed mind to rest satisfied with the degree of precision which the nature of the subject permits and not to seek an exactness where only an approximation of the truth is possible.  --Aristotle

And…

The right to search for the truth implies also a duty; one must not conceal any part of what one has recognized to be true.  --Albert Einstein.