-
Notifications
You must be signed in to change notification settings - Fork 0
/
killing-animals-for-nothing.html
153 lines (144 loc) · 9.82 KB
/
killing-animals-for-nothing.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>Yet Another Instance of a Broken Publishing System at Work</title>
<link rel="stylesheet" href="http://federicov.github.io/theme/css/main.css" />
<link href="http://federicov.github.io/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Federico's Blog Atom Feed" />
<!--[if IE]>
<script src="https://html5shiv.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
</head>
<body id="index" class="home">
<header id="banner" class="body">
<h1><a href="http://federicov.github.io/">Federico's Blog </a></h1>
<nav><ul>
<li><a href="http://federicov.github.io/pages/about.html">About</a></li>
<li><a href="http://federicov.github.io/category/machine-learning.html">Machine Learning</a></li>
<li><a href="http://federicov.github.io/category/misc.html">misc</a></li>
<li><a href="http://federicov.github.io/category/optimization.html">Optimization</a></li>
<li><a href="http://federicov.github.io/category/programming.html">Programming</a></li>
<li class="active"><a href="http://federicov.github.io/category/science.html">Science</a></li>
</ul></nav>
</header><!-- /#banner -->
<section id="content" class="body">
<article>
<header>
<h1 class="entry-title">
<a href="http://federicov.github.io/killing-animals-for-nothing.html" rel="bookmark"
title="Permalink to Yet Another Instance of a Broken Publishing System at Work">Yet Another Instance of a Broken Publishing System at Work</a></h1>
<a href="https://twitter.com/share" class="twitter-share-button" data-count="horizontal" data-via="f_vaggi">Tweet</a><script type="text/javascript" src="https://platform.twitter.com/widgets.js"></script>
</header>
<div class="entry-content">
<footer class="post-info">
<abbr class="published" title="2015-10-15T00:00:00+02:00">
Published: Thu 15 October 2015
</abbr>
<address class="vcard author">
By <a class="url fn" href="http://federicov.github.io/author/federico-vaggi.html">Federico Vaggi</a>
</address>
<p>In <a href="http://federicov.github.io/category/science.html">Science</a>.</p>
<p>tags: <a href="http://federicov.github.io/tag/scientific-publishing.html">scientific publishing</a> <a href="http://federicov.github.io/tag/incentives.html">incentives</a> <a href="http://federicov.github.io/tag/research.html">research</a> </p>
</footer><!-- /.post-info --> <h2>Reproducibility in the Spotlight:</h2>
<p>Psychology is still reeling from the results of a <a href="https://www.sciencemag.org/content/349/6251/aac4716.abstract">massive study of
reproducibility</a>
which found that less than half of statistically significant findings recently
published in the top psychology journals remain significant when independently
reproduced. If this is new to you, Jesse Singal wrote a great summary about it <a href="http://nymag.com/scienceofus/2015/08/many-psychology-research-findings-may-be-false.html">here</a>.</p>
<p>Biology is not exactly doing that much better. Ioannidis first highlighted
shoddy research practices in biology with his aptly named paper
<a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124">Why Most Published Research Findings Are False</a>.</p>
<p>The latest entry in this saga is discussed in <a href="http://www.nature.com/news/poorly-designed-animal-experiments-in-the-spotlight-1.18559">this</a>
excellent article from Nature. In papers that rely on animal models, the
results are too fragile and unreliable to be used as a basis for further
research because the statistical design of the experiments is often
critically flawed. This is unfortunately not a huge surprise: scientists
at Bayer had already warned that published data had become <a href="http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html">progressively less reliable</a> as
basis for drug research.</p>
<p>In <a href="http://www.nature.com/news/2011/110928/full/477511a.html">a previous editorial</a>,
Malcolm Macleod points out that:</p>
<blockquote>
<p>The most reliable animal studies are those that: use randomization to eliminate
systematic differences between treatment groups; induce the condition under
investigation without knowledge of whether or not the animal will get the drug of
interest; and assess the outcome in a blinded fashion. Studies that do not
report these measures are much more likely to overstate the efficacy of
interventions. <strong>Unfortunately, at best one in three publications follows
these basic protections against bias</strong>. This suggests that authors,
reviewers and editors accord them little importance.</p>
</blockquote>
<p>For those that aren't familiar with experimental design - these aren't hyper
advanced techniques. Those are all things which are taught in a 1st year graduate
experimental design course and that every senior scientist should be very
familiar with.</p>
<h2>Animal Models In Science:</h2>
<p>What makes this particularly frustrating is that in this case, the scientists
aren't just wasting public money, but they are killing loads of animals for
absolutely no public good. Until computational models massively improve,
research with animal models is absolutely necessary, especially in the later
stages, when we wish to validate promising drug targets or test drug safety.</p>
<p>In theory, the use of animal models in research in the US is
<a href="https://grants.nih.gov/grants/olaw/references/phspol.htm">strictly regulated</a>
by the NIH. In practice, while charges of animal cruelty are taken seriously,
poor protocol design leading to a waste of model animals is very rarely
sanctioned.</p>
<h2>You Always Get What You Measure:</h2>
<p>Why do really smart scientists make such stupid mistakes?</p>
<ul>
<li>
<p><em>Arrogance</em>: <a href="http://retractionwatch.com/2014/05/30/braggadacio-information-control-and-fear-life-inside-a-brigham-stem-cell-lab-under-investigation/">Some oldschool PIs</a> are so confident in their own theories that they see
statistics not as a critical way to evaluate their data, but as a simple
threshold (p < 0.05) they gotta cross to be able to publish. This is thankfully
not very common, but if you go to any bioinformatics conference and talk over drinks
to some of the junior people there, you'll hear all sorts of horror stories about
being the only bioinformatician in a group and having to somehow come up with a
statistical test that gives p<0.05.</p>
</li>
<li>
<p><em>Misaligned Incentives</em>: This is unfortunately very common. The currency of
scientific careers are publications in high impact journals, and the easiest way
to publish in those journals is to produce splashy results. Unfortunately,
results are the one thing that scientists have no actual control over - if you
execute a well designed experiment to test a reasonable hypothesis, the
actual outcome is up to the Universe. By dis-proportionally rewarding splashy
results, we punish the scientists that actually do research properly, since they
will have a much lower fraction of positive results.</p>
</li>
</ul>
<p>The irony that Nature, one of the most prestigious scientific journals in the world,
acknowledges that scientists push splashy but unreliable findings (like say,
<a href="http://www.nature.com/nature/journal/v505/n7485/full/nature12968.html">these</a>)
instead of solid but boring ones is not lost on me. It reminds me of the time
a Facebook executive complained <a href="https://www.facebook.com/mhudack/posts/10152148792566194">that the Internet was drowning in shitty click-baity articles</a>, when the
product he was responsible for was the main driver of that behaviour.</p>
<p>It's important to add that the quest for high impact publications is not
just motivated by a selfish desire for success. Postdocs are facing a very
difficult job market and are desperate to have a high impact paper before
applying for tenure track positions. PIs are under an <a href="http://www.dcscience.net/2014/12/01/publish-and-perish-at-imperial-college-london-the-death-of-stefan-grimm/">immense amount of
pressure</a>
by their institutions to obtain grants, and it's getting harder and harder.</p>
<p><img src="http://nexus.od.nih.gov/all/wp-content/uploads/2014/03/FundingAwardSuccessRate_RPG.jpg " alt="NIH_Funding" style="width: 700px;"/></p>
<p>Further, missing grants doesn't just hurt the career of the PI. Very few people
in science have the luxury of fixed positions - even lab technicians and staff
scientists are often funded by grants, and not getting a grant renewed can mean
having to fire people people you've worked with for many years.</p>
</div><!-- /.entry-content -->
</article>
</section>
<section id="extras" class="body">
<div class="social">
<h2>social</h2>
<ul>
<li><a href="http://federicov.github.io/feeds/all.atom.xml" type="application/atom+xml" rel="alternate">atom feed</a></li>
<li><a href="http://twitter.com/f_vaggi">@f_vaggi</a></li>
</ul>
</div><!-- /.social -->
</section><!-- /#extras -->
<footer id="contentinfo" class="body">
<address id="about" class="vcard body">
Proudly powered by <a href="http://getpelican.com/">Pelican</a>, which takes great advantage of <a href="http://python.org">Python</a>.
</address><!-- /#about -->
<p>The theme is by <a href="http://coding.smashingmagazine.com/2009/08/04/designing-a-html-5-layout-from-scratch/">Smashing Magazine</a>, thanks!</p>
</footer><!-- /#contentinfo -->
</body>
</html>