BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

Solving The Problem Of Scientific Reproducibility With Peer-Reviewed Video

Following
This article is more than 8 years old.

In a story published last week in the Boston Globe, Carolyn Johnson covered one of the quiet crises facing scientific research today: the fact that a lot of published experiments simply can't be reproduced by other scientists .

"But talk to a scientist long enough, and you’ll probably hear a story like this: An intriguing new discovery was reported in a research journal," wrote Carolyn Johnson in an excellent article on this topic in the Boston Globe. "Maybe it was a biologist describing a new Achilles’ heel in cancer cells, a psychologist’s profound insight into human behavior, or an astronomer’s finding about the first moments of the universe. The scientist read about the finding and tried to confirm it in her own lab, but the experiment just didn’t come out the same."

Although in some instances the cause of this is outright fraud, far more often the causes are more proasic and mundane. Not being able to replicate an experiment may just mean there was something wrong with the instruments in the initial experiment. (This quite famously happened to the OPERA collaboration, when they infamously announced that they had measured neutrinos moving faster than light. After attempts to reproduce the experiment failed, it was later revealed that the measurements had been erroneous due to a bad data connection in their instruments.)

Sometimes, though, the reason why an experiment doesn't work is because important steps were left out of the original paper, making it impossible for other scientists to replicate the experiment. Sometimes this is oversight, and sometimes it's just that a particular lab has common practices that don't carryover to the wider community. In these cases, the experimenters didn't find a false result - it's just hard for other people to demonstrate the same thing.

Journal of Visualized Experiments cofounder Moshe Pritsker (Credit: JoVE)

This particular aspect of day to day aspect of scientific research was deeply frustrating to Dr. Moshe Pritsker when he was engaged in research a towards his PhD a little over a decade ago. He would find himself unable to complete experiments working just off papers, but when he was able to meet with an experimenter or visit their lab, he was able to then replicate the experiment. But this experience made him frustrated about the way science was being done in the 21st century.

"Why doesn't it work?" he told me. "It's text. It's not good for transfer of knowledge about complex experiments. When you see people do it, you'll get small details you can't get from text."

Faced with the time consuming process of visiting original labs to see how experiments were performed led Pritsker to thinking that there had to be a better way for scientists to share information than the centuries-old process of publishing papers.

"I asked myself: is this really how science should work in 21st century?"  he said. "Biology is still practiced in this same way - where you have to see a grandmaster in order to do what they do. That's when I decided on a more systematic solution - a type of science publishing which focused on video."

Pritsker completed his PhD in 2005 and went to Harvard Medical School for his postdoctoral work. There he discovered the same reproducibility problem that had plagued him as a PhD student. So he began to solve it on his own by filming experiments at Harvard himself so that other people could watch video to see how an experiment was performed. Eventually, he and his colleague Nikita Bernstein took this initiative along with an investor to cofound the Journal of Visualized Experiments (JoVE) in 2007.

For scientists, the submission process to JoVE is mostly typical - researchers produce a paper similar to what you'd send to other journals describing their experiment and results. But when they want to be considered, JoVE will send a filmographer from its network to film one of their experiments. The paper and video are then peer-reviewed by other scientists in the field as is typical for other scientific publications.

Once a paper is accepted to JoVE, the next step is video production. A script based on the accepted paper is written, and a local videographer will be sent to spend the day filming the experiment according to script. The video is then edited and cut for the most important aspect for reproducibility - the methods of the experiment itself. Once completed, a video is typically about 10-15 minutes in length.

The response to this type of paper, says Pritsker, has been overwhelmingly positive from the scientific community.

"Usually people ask, 'Why didn't this happen before?'," he said. "Everyone is suffering from this problem in science. It's much more valuable to see than read. We usually get 50 queries a day from scientists asking for access to video."

In addition to their primary business of publishing peer-reviewed research papers and video, the company also added a new line to their business, where they produce videos showing basic, step-by-step experimental methods for use in undergraduate education and other institutions that will be using similar methods.

Although they initially tried different business models, in 2009 JoVE adopted a business model fairly typical for scientific publishing. Its primary source of revenue is through institutional subscriptions - currently nearly 800 large institutions in 43 countries subscribe to the journal, and its sales increased 37% year over year in 2014. Its science education database of videos showing basic experiments currently has over 200 subscribers. JoVE broke even in 2010 and has since then been profitable.

The business will continue to succeed, Pritsker contends, as more and more attention is being paid to the problem of replicating experiments.

"This was an anecdote for a long time, but now it's systematic and quantified," he said.  "If we can't reproduce science, it's not science anymore."

Follow me on Twitter or LinkedInCheck out my websiteSend me a secure tip