How Data Become Science

Credit: shutterstock

Scientists know better than to say that they believe the data. The reason is, as Neil deGrasse Tyson once said, "The good thing about science is that it's true whether or not you believe in it."

How can he be so sure?

There is so much scientific sounding information out there, how can the pseudoscience be separated from the truth?

Here is an insider's look on how to know when data should be taken seriously and when they should be ignored.

Hint ... it's all about where you find it.

A bar graph that appears in a peer reviewed journal is (most likely) legitimate science. But, that same bar graph found on a website is junk. The difference is the peer review process. This process allows the data to be analyzed by experts in the field and eventually, either accepted or rejected. It's like "tinder" for scientists, but, with far less attractive people involved. Peer review is what validates data, defines science and allows us to call anything else pseudoscience.

How does the peer review process work? Here are the main steps, followed by a more detailed description.

1. Scientists craft their data into a story and write a manuscript.
2. The manuscript is submitted to a journal.
3. The article, if deemed acceptable by an editor, is sent out to scientists (in the same field) for peer-review.
4. The reviewers' comments are sent to the authors. There may be back and forth dialog.
5. The article is then either accepted for publication or rejected.

Let's look more closely at each step.

1. Scientists craft their data into a story and write a manuscript.

A scientist is also a storyteller. They do not throw data together haphazardly, rather, they link connected pieces of data together to tell a story. That story is presented as a five part article. Regardless of the journal, scientific articles generally have five sections. They are;

• Abstract - A summary of the entire paper.
• Introduction - The background that lends context to the study. What has been done in this field up until this point? What is already known and what are the questions that remain unanswered?
• Materials and Methods - One of the hallmarks of a scientific article. The authors must describe, in painstaking detail, the methods they used in order to perform the experiments that are presented in the article. This is very important, because a staple of scientific integrity is reproducibility. But, an experiment can only be reproduced if someone else knows exactly how it was done the first time around. Therefore, although this section consists of the minutiae of procedural detail, it is quite important.
• Results - Data are presented and the authors' interpretation is given.
• Discussion - Here, the authors have space to offer a larger view of the significance of their results, suggest what contribution they have made to the field (and the larger scientific community) and opine on the next steps to take on the pathway to discovery.

2. The manuscript is submitted to a journal.

Once the manuscript it written, it is submitted to a journal. The choice of which journal is important because some journals are more rigorous than others or select only the most cutting edge science. (2) Journals are rated by a scoring system called an "impact factor," which indicates how many times that journal has been cited within the last year. This is the measure of the significance of the journal (and, thus, the paper) within the scientific community.

3. The article, if deemed acceptable by an editor, is sent out to scientists (in the same field) for peer-review.

Upon submission, the article first comes across the desk of an editor for initial review.  The editor determines if the paper will proceed to the next step in the review process, or be "editorially rejected." If the editor decides that the paper is good enough to be reviewed by experts in the field, three are chosen to be reviewers and asked to give their opinions on the paper. (1) The reviewers' job is to carefully evaluate, and poke holes in the paper. They examine every detail, searching for inaccuracies, mistakes, sloppy experimentation, or simply poor writing. They can offer criticism or praise on everything from the grammar to the methods used to the manner in which the data are presented. The reviewers look for science that is well done (with appropriate controls), correct statistics, uses acceptable techniques, and is presented clearly. They will criticize anything that falls short.

4. The reviewers' comments are sent to the authors. There may be back and forth dialog. 

The reviewers have a number of choices when they are finished the review. They can choose to reject it, accept it, or something in between, which is called "reject with the possibility of resubmission." In this case, the authors receive the reviewers' comments and have a finite amount of time to address them, either through a written discussion, or by conducting additional experiments and providing more data. They, then resubmit their paper with the changes made, new experiments added (if necessary), and their comments to the authors. Then, the entire package goes back to the reviewers. If they are satisfied, they will change their decision to "accept." If they still have concerns, the entire process will repeat itself. This can happen multiple times, and take months (sometimes years.) Regardless of the number of times it happens and how long it takes, the reviewer ultimately checks an "accept" or "reject" box at the end of the process. This determines the fate of the paper. In the event of a split decision between an even number of reviewers, the editor will make the final decision.

5. The article is then either accepted for publication or rejected.

If the paper is accepted, it will be published in that journal. If the paper is rejected, the authors may try to submit it to another journal, probably one with a lower impact factor. If the paper is rejected multiple times from multiple different journals, it is most likely fatally flawed and the peer review process will have done what it is supposed to do.

Some scientific results are worthless, such as a bar graph on any website associated with a weight loss supplement. Some can change the world, for example, the data presented in the 1953 papers by Jonas Salk "Studies in Human Subjects on Active Immunization Against Poliomyelitis." The peer review process is what allows us to separate these two and to turn meaningful data into science.

NOTES:

(1) Different journals have different criteria for reviewers

(2) It is the responsibility of the journal to uphold the high standards of scientific integrity, to keep this process at its best.