-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
287 lines (251 loc) · 12.8 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
<title>Technical Support for Climate Assessment Data Services</title>
<link rel="icon" type="image/png" href="img/tsu-logo.png" />
<meta name="description"
content="To meet NOAA Information Quality Act requirements and manage the complexity of a large, distributed effort, the TSU adopted a number of technologies and best practices. With so many contributors to the NCA and iterations of code to process scientific datasets and produce figures, version control has been a great asset. ">
<meta name="author" content="Andrew Buddenberg">
<link rel="stylesheet" href="css/reveal.css">
<link rel="stylesheet" href="css/theme/black.css">
<!-- Theme used for syntax highlighting of code -->
<link rel="stylesheet" href="lib/css/zenburn.css">
<style>
.reveal section img { background:none; border:none; box-shadow:none;}
#nca4 {
width: 302px;
height: 390px;
float: right;
border: 5px solid gray;
position: relative;
top: 10px;
}
#nca4 span {
text-align: center;
line-height: 390px;
}
</style>
<!-- Printing and PDF exports -->
<script>
var link = document.createElement( 'link' );
link.rel = 'stylesheet';
link.type = 'text/css';
link.href = window.location.search.match( /print-pdf/gi ) ? 'css/print/pdf.css' : 'css/print/paper.css';
document.getElementsByTagName( 'head' )[0].appendChild( link );
</script>
</head>
<body>
<div class="reveal">
<div class="slides">
<section>
<h2>Andrew Buddenberg</h2>
<br>
<p>Software Engineer @ <a href="http://www.cicsnc.org/people/andrew-buddenberg/">CICS-NC</a></p>
<p><a href="https://github.com/abuddenb">github.com/abuddenb</a></p>
<div style="font-size: 50%; padding-top: 10px">
<p>Paula Ann Hennon, PhD @ <a href="http://www.cicsnc.org/people/paula-hennon/">CICS-NC</a></p>
<p>Lou Vasquez @ <a href="http://www.cicsnc.org/people/lou-vasquez/">CICS-NC</a></p>
</div>
<footer>
<img src="img/noaa_white_200.png" height="96px" width="96px">
<img src="img/brick4x1.jpg" height="20%" width="20%" style="position: relative; top: -7px; padding: 20px">
<img src="img/cicsnc-logo.png" height="96px" width="96px">
</footer>
</section>
<section>
<h3>Technical Support for Climate Assessment Data Services</h3>
<!--<p><img src="Paper_36097_abstract_31278_0.png" height="20%" width="20%"</p>-->
<p><img src="img/nca2014.jpg" height="50%" width="50%"></p>
<!--<p><img src="img/thumb-nca3-cover.png"></p>-->
<p><span class="insert fragment">We do what we can...</span></p>
<aside class="notes">
The <b>National Climate Assessment</b> summarizes the impacts of climate change on the United States,
now and in the future. A team of more than 300 experts guided by a 60-member Federal Advisory
Committee produced the report, which was extensively reviewed by the public and experts,
including federal agencies and a panel of the National Academy of Sciences. It is classified as
a <b>Highly Influential Scientific Assessment</b>.<br><br>
A <b>scientific assessment</b> is an evaluation of a body of scientific or technical knowledge
that typically synthesizes multiple factual inputs, data, models, assumptions, and/or
applies best professional judgment to bridge uncertainties in the available information.
</aside>
</section>
<section>
<p><a href="http://www.cio.noaa.gov/services_programs/pdfs/OMB_Peer_Review_Bulletin_m05-03.pdf" target="_blank">...because we must</a></p>
<p><img src="img/NOAA%20IQA.png"></p>
<aside class="notes">
<aside class="notes">
The classification <b>HISA</b> mandates a higher standard of Scientific and Process integrity.
"Scientific integrity," in the context of peer review, refers to such issues as
"expertise and balance of the panel members; the identification of the scientific issues
and clarity of the charge to the panel; the quality, focus and depth of the discussion of the
issues by the panel; the rationale and supportability of the panel’s findings; and the
accuracy and clarity of the panel report." <b>Process integrity</b> includes such issues as
“transparency and openness, avoidance of real or perceived conflicts of interest, a
workable process for public comment and involvement,” and adherence to defined
procedures.
</aside>
</aside>
</section>
<section>
<section>
<h3>Software Engineering Best Practices</h3>
<p>Made extensive use of Python, NumPy, SciPy, and matplotlib as our preferred platform since they're tested, supported, free/open source, and accessible to the general public</p>
<div>
<img src="img/python-logo-master-v3-TM.png" height="61px" width="175px" style="padding: 10px; ">
<img src="img/NumPy_logo.png" style="padding: 10px; position: relative; top: 15px">
<img src="img/scipy_logo.png" style="padding: 10px; position: relative; top: 25px">
<br>
<img src="img/matplotlib_logo2.png" height="25%" width="25%" style="position: relative; left: -150px">
</div>
<small class="fragment">Though ArcGIS, IDL, GrADS, and Fortran 95 also made appearances where appropriate.</small>
</section>
<section>
<h3>Source code version control with git</h3>
<p><img src="img/gitlab_k3.png" height="50%" width="50%"></p>
<ul>
<li>Facilitates collaboration</li>
<li>Catalog of programs for easy recall</li>
<li>Easy way of undoing experimental changes</li>
<li>Provides traceability</li>
</ul>
</section>
<section>
<h3>Dataset version control with git annex</h3>
<div style="position: relative; left: 0; top: 0;">
<img src="img/git_annex_dir.png" style="position: relative; top: 0; left: 0;">
<img src="img/git_annex_logo_small.png" style="position: absolute; top: -30px; left: 30px; background-color: rgba(0, 0, 0, 0.5)">
</div>
<ul>
<li>Tracks file metadata and SHA-1 hash of file contents</li>
<li>One can seamlessly checkout any version of a dataset in-place</li>
<li>Ensures integrity of datasets; file contents are frozen unless changes explicitly checked in</li>
</ul>
</section>
<section>
<h3>Automated testing</h3>
<p><img src="img/pytest.png" height="75%" width="75%"></p>
<p>Data processing jobs automatically run tests to validate integrity</p>
<small class="fragment">So we don't make the same mistake twice!</small>
</section>
</section>
<section>
<section>
<h3>Unprecedented<sup>*</sup> collection of metadata</h3>
<p><img src="img/metadata_survey.png" height="50%" width="50%"></p>
<p>
Over 200 of the 290 figures in the assessment are fully traced. For figures original to the NCA,
detailed information about sub-figures, datasets, and the processes used to create the figure is
available.
</p>
<small><sup>*</sup>as far as we know</small>
<aside class="notes">
Collection began mid-way through the production time of NCA, hence the incompletion. We have
tools and processes in place now to collection metadata from the outset.
</aside>
</section>
<section>
<h3>Global Change Information System</h3>
<h4>It's like the Internet Movie Database for climate data</h4>
<p><a href="http://data.globalchange.gov/report/nca3/chapter/our-changing-climate/figure/emissions-levels-determine-temperature-rises" target="_blank"><img src="img/gcis.png" height="45%" width="45%"></a></p>
<ul>
<li>Searchable <a href="http://data.globalchange.gov" target="_blank">database</a> of publications, citations, datasets, people, organizations, and more.</li>
<li>Web API provides easy way to automatically move metadata in and embed on <a href="http://nca2014.globalchange.gov/report/our-changing-climate/observed-change" target="_blank">other websites</a>.</li>
</ul>
</section>
</section>
<section>
<section>
<h3>High Performance Computing</h3>
<p><img src="img/cics_cluster.png" height="50%" width="50%"></p>
<p>At some point, the datasets got bigger while the deadlines got shorter, so we needed to take data processing to the next level.</p>
</section>
<section>
<h4>Computing multi-model averages from CMIP3 Downscaled Dataset</h4>
<p><a href="http://data.globalchange.gov/report/nca3/chapter/agriculture/figure/projected-changes-in-key-climate-variables-affecting-agricultural-productivity" target="_blank"><img src="img/thumb-658ab2c7-43a6-45c5-9e1d-eb4d2c3bafe9.png"></a></p>
<table>
<tr><th>Processing Mode</th><th>Single Variable</th><th>All variables</th></tr>
<tr><td>Single process</td><td>94 minutes</td><td>114 hours</td></tr>
<tr><td>MPI (old cluster)</td><td>5 minutes</td><td>6 hours</td></tr>
<tr><td>MPI (new cluster)</td><td>50 seconds</td><td>61 minutes</td></tr>
</table>
<small class="fragment">Keep in mind that we went through many iterations of this dataset. Being able to respond to change quickly was invaluable.</small>
</section>
</section>
<section>
<h2>The End</h2>
<div class="fragment">
<img src="img/thumb-nca2-cover.png" height="32%" width="32%">
<img src="img/thumb-nca3-cover.png" height="32%" width="32%">
<div id="nca4"><span>NCA4</span></div>
<br><br><small>...for now</small>
</div>
</section>
<!--<section>-->
<!--<section id="fragments">-->
<!--<h2>Fragments</h2>-->
<!--<p>Hit the next arrow...</p>-->
<!--<p class="fragment">... to step through ...</p>-->
<!--<p><span class="fragment">... a</span> <span class="fragment">fragmented</span> <span class="fragment">slide.</span></p>-->
<!--<aside class="notes">-->
<!--This slide has fragments which are also stepped through in the notes window.-->
<!--</aside>-->
<!--</section>-->
<!--<section>-->
<!--<h2>Fragment Styles</h2>-->
<!--<p>There's different types of fragments, like:</p>-->
<!--<p class="fragment grow">grow</p>-->
<!--<p class="fragment shrink">shrink</p>-->
<!--<p class="fragment roll-in">roll-in</p>-->
<!--<p class="fragment fade-out">fade-out</p>-->
<!--<p class="fragment current-visible">current-visible</p>-->
<!--<p class="fragment highlight-red">highlight-red</p>-->
<!--<p class="fragment highlight-blue">highlight-blue</p>-->
<!--</section>-->
<!--</section>-->
<!--<section data-transition="slide" data-background="#b5533c" data-background-transition="zoom">-->
<!--<h2>Background Transitions</h2>-->
<!--<p>-->
<!--You can override background transitions per-slide.-->
<!--</p>-->
<!--<pre><code style="word-wrap: break-word;"><section data-background-transition="zoom"></code></pre>-->
<!--</section>-->
<!--<section>-->
<!--<h2>Pretty Code</h2>-->
<!--<pre><code data-trim contenteditable>-->
<!--function linkify( selector ) {-->
<!--if( supports3DTransforms ) {-->
<!--var nodes = document.querySelectorAll( selector );-->
<!--for( var i = 0, len = nodes.length; i < len; i++ ) {-->
<!--var node = nodes[i];-->
<!--if( !node.className ) {-->
<!--node.className += ' roll';-->
<!--}-->
<!--}-->
<!--}-->
<!--}-->
<!--</code></pre>-->
<!--<p>Code syntax highlighting courtesy of <a href="http://softwaremaniacs.org/soft/highlight/en/description/">highlight.js</a>.</p>-->
<!--</section>-->
</div>
</div>
<script src="lib/js/head.min.js"></script>
<script src="js/reveal.js"></script>
<script>
// More info about config & dependencies:
// - https://github.com/hakimel/reveal.js#configuration
// - https://github.com/hakimel/reveal.js#dependencies
Reveal.initialize({
dependencies: [
{ src: 'plugin/markdown/marked.js' },
{ src: 'plugin/markdown/markdown.js' },
{ src: 'plugin/notes/notes.js', async: true },
{ src: 'plugin/highlight/highlight.js', async: true, callback: function() { hljs.initHighlightingOnLoad(); }},
{ src: 'plugin/zoom-js/zoom.js', async: true },
{ src: 'plugin/notes/notes.js', async: true }
]
});
</script>
</body>
</html>