lidated or markedly weakened the conclusions of the study." It is interesting to note that one of the minor errors included in the manuscript was a misspelling of the drug's name. Out of the 203 reviewers, 30 were convinced in the correctness of the misspelled name and used it throughout their interview. The authors of the study said about the results (with the usual scientific undertone): "the small number of errors identified bA study (Baxt WG, Waeckerle JF, Berlin JA, Callaham ML. Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Ann Emerg Med. 1998 Sep;32(3 Pt 1):310-7) introduced 10 major and 13 minor errors in a fictitious scientific manuscript. The manuscript was sent to all reviewers of the Annals of Emergency Medicine, the official publication of the American College of Emergency Physicians. The Annals has been in print for more than 25 years, and is the most widely read journal in emergency medicine. The work described in the manuscript was a standard double-blind, placebo control study of the effect of the propranolol drug on migraine headaches. The manuscript was reviewed by 203 reviewers. Eighty percent of the reviewers were professors at academic emergency medicine departments, and twenty percent were physicians in private practice.

The analysis of the reviewers' comments produced the following results. Fifteen reviewers recommended publication. The reviewers in this group missed 82.7% of the major errors and 88.2% of the minor errors. Sixty seven reviewers recommended revisions. The reviewers in this group missed 70.4% of the major errors and 78.0% of the minor errors. One hundred and seventeen reviewers recommended rejection. The reviewers in this group missed 60.9% of the major errors and 74.8% of the minor errors.

According to these results, the 15 professors who recommended publication, on average, missed 82.7% of the major errors, and 88.2% of the minor errors. In other words, the professors missed at least 4 out of 5 errors inserted in the manuscript. These errors were defined by the authors as "nonremediable errors that invalidated or markedly weakened the conclusions of the study." It is interesting to note that one of the minor errors included in the manuscript was a misspelling of the drug's name. Out of the 203 reviewers, 30 were convinced in the correctness of the misspelled name and used it throughout their interview. The authors of the study said about the results (with the usual scientific undertone): "the small number of errors identified by

ans. The Annals has been in print for more than 25 years, and is the most widely read journal in emergency medicine. The work described in the manuscript was a standard double-blind, placebo control study of the effect of the propranolol drug on migraine headaches. The manuscript was reviewed by 203 reviewers. Eighty percent of the reviewers were professors at academic emergency medicine departments, and twenty percent were physicians in private practice.

The analysis of the reviewers' comments produced the following results. Fifteen reviewers recommended publication. The reviewers in this group missed 82.7% of the major errors and 88.2% of the minor errors. Sixty seven reviewers recommended revisions. The reviewers in this group missed 70.4% of the major errors and 78.0% of the minor errors. One hundred and seventeen reviewers recommended rejection. The reviewers in this group missed 60.9% of the major errors and 74.8% of the minor errors.

According to these results, the 15 professors who recommended publication, on average, missed 82.7% of the major errors, and 88.2% of the minor errors. In other words, the professors missed at least 4 out of 5 errors inserted in the manuscript. These errors were defined by the authors as "nonremediable errors that invalidated or markedly weakened the conclusions of the study." It is interesting to note that one of the minor errors included in the manuscript was a misspelling of the drug's name. Out of the 203 reviewers, 30 were convinced in the correctness of the misspelled name and used it throughout their interview. The authors of the study said about the results (with the usual scientific undertone): "the small number of errors identified b

cians in private practice.

The analysis of the reviewers' comments produced the following results. Fifteen reviewers recommended publication. The reviewers in this group missed 82.7% of the major errors and 88.2% of the minor errors. Sixty seven reviewers recommended revisions. The reviewers in this group missed 70.4% of the major errors and 78.0% of the minor errors. One hundred and seventeen reviewers recommended rejection. The reviewers in this group missed 60.9% of the major errors and 74.8% of the minor errors.

According to these results, the 15 professors who recommended publication, on average, missed 82.7% of the major errors, and 88.2% of the minor errors. In other words, the professors missed at least 4 out of 5 errors inserted in the manuscript. These errors were defined by the authors as "nonremediable errors that invalidated or markedly weakened the conclusions of the study." It is interesting to note that one of the minor errors included in the manuscript was a misspelling of the drug's name. Out of the 203 reviewers, 30 were convinced in the correctness of the misspelled name and used it throughout their interview. The authors of the study said about the results (with the usual scientific undertone): "the small number of errors identified b

ejection. The reviewers in this group missed 60.9% of the major errors and 74.8% of the minor errors.

According to these results, the 15 professors who recommended publication, on average, missed 82.7% of the major errors, and 88.2% of the minor errors. In other words, the professors missed at least 4 out of 5 errors inserted in the manuscript. These errors were defined by the authors as "nonremediable errors that invalidated or markedly weakened the conclusions of the study." It is interesting to note that one of the minor errors included in the manuscript was a misspelling of the drug's name. Out of the 203 reviewers, 30 were convinced in the correctness of the misspelled name and used it throughout their interview. The authors of the study said about the results (with the usual scientific undertone): "the small number of errors identified b

lidated or markedly weakened the conclusions of the study." It is interesting to note that one of the minor errors included in the manuscript was a misspelling of the drug's name. Out of the 203 reviewers, 30 were convinced in the correctness of the misspelled name and used it throughout their interview. The authors of the study said about the results (with the usual scientific undertone): "the small number of errors identified by the reviewers in this study was surprising. The major errors placed in the manuscript invalidated or undermined each of the major methodologic steps of the study … The identification of even a fraction of these errors should have indicated that the study was unsalvageable, yet the reviewers identified only 34% of these errors, and only 59% of the reviewers rejected the work."

Points to consider:

1. In this study, the reviewers were professors and private practice physicians with an average of 3 years experience as reviewers for the Annals and additional years of experience reviewing scientific manuscripts for 2 other scientific journals, and with 10 years of experience practicing emergency medicine. These reviewers possess a much higher level of expertise in the subject of the tested manuscript relative to even the most experienced market researchers analyzing qualitative customer data, the most experienced human resource managers analyzing candidate data, the lawyers analyzing patents, or the investment analysts and consultants analyzing business data. So, if professors and physicians failed to recognize major errors in a standard scientific manuscript, what are the chances that the less trained professionals will identify gaps and inconsistencies in non-standard qualitative business data?

2. In this study, the professors were expected to identify the technical errors found in the manuscript. The identification and elimination of this type of errors is the objective of the years of training undergone by every scientist. Unlike this study, the great majority of qualitative studies in business include psychological gaps and inconsistencies, and unlike scientists, most other professionals