forked from nerfies/nerfies.github.io
-
Notifications
You must be signed in to change notification settings - Fork 1
/
index.html
415 lines (376 loc) · 18.5 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="description"
content="Binding Language Models in Symbolic Languages">
<meta name="keywords" content="Language Model, Symbolic Language, In-context Learning">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Binding Language Models in Symbolic Languages</title>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-PYVRSFMDRL"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag() {
dataLayer.push(arguments);
}
gtag('js', new Date());
gtag('config', 'G-PYVRSFMDRL');
</script>
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
rel="stylesheet">
<link rel="stylesheet" href="./static/css/bulma.min.css">
<link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="./static/css/bulma-slider.min.css">
<link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="./static/css/index.css">
<link rel="icon" href="./static/images/favicon.svg">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script defer src="./static/js/fontawesome.all.min.js"></script>
<script src="./static/js/bulma-carousel.min.js"></script>
<script src="./static/js/bulma-slider.min.js"></script>
<script src="./static/js/index.js"></script>
</head>
<body>
<nav class="navbar" role="navigation" aria-label="main navigation">
<div class="navbar-brand">
<a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false">
<span aria-hidden="true"></span>
<span aria-hidden="true"></span>
<span aria-hidden="true"></span>
</a>
</div>
<div class="navbar-menu">
<div class="navbar-start" style="flex-grow: 1; justify-content: center;">
<a class="navbar-item" href="https://www.xlang.ai/">
<span class="icon">
<i class="fas fa-home"></i>
</span>
</a>
<div class="navbar-item has-dropdown is-hoverable">
<a class="navbar-link">
More Research
</a>
<div class="navbar-dropdown">
<a class="navbar-item" href="https://yale-lily.github.io/spider">
Spider
</a>
<a class="navbar-item" href="https://github.com/HKUNLP/UnifiedSKG">
UnifiedSKG
</a>
<a class="navbar-item" href="https://github.com/Yushi-Hu/IC-DST">
IC-DST
</a>
<a class="navbar-item" href="https://github.com/HKUNLP/icl-selective-annotation">
Selective Annotation
</a>
<a class="navbar-item" href="https://ds1000-code-gen.github.io/">
DS-1000
</a>
<a class="navbar-item" href="https://instructor-embedding.github.io/">
Instructor
</a>
<a class="navbar-item" href="https://text-to-reward.github.io/">
Text2Reward
</a>
<a class="navbar-item" href="https://github.com/xlang-ai/OpenAgents">
OpenAgents
</a>
<a class="navbar-item" href="https://github.com/OpenLemur/lemur">
Lemur-70B
</a>
<a class="navbar-item" href="https://arks-codegen.github.io/">
ARKS
</a>
<a class="navbar-item" href="https://os-world.github.io/">
OSWorld
</a>
<a class="navbar-item" href="https://brightbenchmark.github.io/">
BRIGHT
</a>
<a class="navbar-item" href="https://spider2-v.github.io/">
Spider2-V
</a>
</div>
</div>
</div>
</div>
</nav>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">Binding Language Models in Symbolic Languages</h1>
<div class="is-size-5 publication-authors">
<span class="author-block">
<a href="https://blankcheng.github.io/">Zhoujun Cheng*</a><sup>1,2</sup>,</span>
<span class="author-block">
<a href="https://tianbaoxie.com/">Tianbao Xie*</a><sup>1</sup>,</span>
<span class="author-block">
<a href="https://scholar.google.com/citations?user=XTbDLrkAAAAJ&hl=en">Peng Shi</a><sup>5</sup>,
</span>
<span class="author-block">
<a href="https://scholar.google.com/citations?user=t_Bwt70AAAAJ&hl=en">Chengzu Li</a><sup>1</sup>,
</span>
<span class="author-block">
<a href="https://homes.cs.washington.edu/~rahuln/">Rahul Nadkarni</a><sup>3</sup>,
</span>
<span class="author-block">
<a href="https://yushi-hu.github.io/">Yushi Hu</a><sup>3</sup>,
</span>
<span class="author-block">
<a href="http://cmxiong.com/">Caiming Xiong</a><sup>6</sup>,
</span>
<span class="author-block">
<a href="http://www.cs.yale.edu/homes/radev/">Dragomir Radev</a><sup>7</sup>,
</span>
<span class="author-block">
<a href="https://people.ece.uw.edu/ostendorf/">Mari Ostendorf</a><sup>3</sup>,
</span>
<span class="author-block">
<a href="https://www.cs.washington.edu/people/faculty/lsz">Luke Zettlemoyer</a><sup>3,8</sup>,
</span>
<span class="author-block">
<a href="https://nasmith.github.io/">Noah A. Smith</a><sup>3,4</sup>,
</span>
<span class="author-block">
<a href="https://taoyds.github.io/">Tao Yu</a><sup>1,3</sup>
</span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block"><sup>1</sup>The University of Hong Kong,</span>
<span class="author-block"><sup>2</sup>Shanghai Jiao Tong University,</span>
<span class="author-block"><sup>3</sup>University of Washington,</span>
<span class="author-block"><sup>4</sup>Allen Institute for AI,</span>
<span class="author-block"><sup>5</sup>University of Waterloo,</span>
<span class="author-block"><sup>6</sup>Salesforce Research,</span>
<span class="author-block"><sup>7</sup>Yale University,</span>
<span class="author-block"><sup>8</sup>Meta AI</span>
</div>
<div class="column has-text-centered">
<div class="publication-links">
<!-- PDF Link. -->
<span class="link-block">
<a href="https://arxiv.org/abs/2210.02875"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>Paper</span>
</a>
</span>
<!-- Code Link. -->
<span class="link-block">
<a href="https://github.com/HKUNLP/binder"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code</span>
</a>
</span>
<!-- Demo Link. -->
<span class="link-block">
<a href="https://huggingface.co/spaces/hkunlp/Binder"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-coffee"></i>
</span>
<span>Demo</span>
</a>
</span>
<!-- Twitter Link. -->
<span class="link-block">
<a href="https://twitter.com/taoyds/status/1580987466974396417"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-twitter"></i>
</span>
<span>Twitter</span>
</a>
</span>
<!-- Video Link. -->
<span class="link-block">
<a href="https://vimeo.com/760302884"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-youtube"></i>
</span>
<span>Video</span>
</a>
</span>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body">
<h2 class="subtitle has-text-centered">
<span class="dnerf">Binder</span> is a training-free neural-symbolic framework that maps the task input to an executable <span class="dnerf">Binder</span> program that (1) allows binding API calls to GPT-3 Codex into SQL/Python, (2) is executed with SQL/Python Interpreter + GPT-3 Codex to derive the answer.
</h2>
<img src="static/images/main.jpg"> <!-- To .gif format-->
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<!-- Abstract. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
Though end-to-end neural approaches have recently been dominating NLP tasks in both performance and ease-of-use, they lack interpretability and robustness. We propose <span class="dnerf">Binder</span>, a training-free neural-symbolic framework that maps the task input to a program, which (1) allows binding a unified API of language model (LM) functionalities to a programming language (e.g., SQL, Python) to extend its grammar coverage and thus tackle more diverse questions, (2) adopts an LM as both the program parser and the underlying model called by the API during execution, and (3) requires only a few in-context exemplar annotations. Specifically, we employ GPT-3 Codex as the LM. In the parsing stage, with only a few in-context exemplars, Codex is able to identify the part of the task input that cannot be answerable by the original programming language, correctly generate API calls to prompt Codex to solve the unanswerable part, and identify where to place the API calls while being compatible with the original grammar. In the execution stage, Codex can perform versatile functionalities (e.g., commonsense QA, information extraction) given proper prompts in the API calls. <span class="dnerf">Binder</span> achieves state-of-the-art results on WikiTableQuestions and TabFact datasets, with explicit output programs that benefit human debugging. Note that previous best systems are all finetuned on tens of thousands of task-specific samples, while <span class="dnerf">Binder</span> only uses dozens of annotations as in-context exemplars without any training.
</p>
Compare <span class="dnerf">Binder</span> with:
<ul>
<li><strong>End2end & Chain-of-Thought:</strong>   More interpretable and robust!
<br><span class="dnerf">Binder</span> program is deterministically executed to entail the prediction/answer. </li>
<li><strong>Semantic Parsing & Code Generation:</strong>   Higher grammar coverage!
<br><span class="dnerf">Binder</span> injects Codex functionalities into SQL/Python to handle more diverse questions. </li>
<li><strong>Neural-symbolic Approach:</strong>   More general (domains), training-free, few annotations!
<br><span class="dnerf">Binder</span> is based on general programming languages (SQL/Python), and requires only a dozens in-context exemplar annotations. </li>
</ul>
</div>
</div>
</div>
</div>
<!--/ Abstract. -->
<!-- Paper video. -->
<div class="columns is-centered has-text-centered">
<div class="column is-three-fifths">
<h2 class="title is-3">Video</h2>
<div class="publication-video">
<iframe src="https://player.vimeo.com/video/760302884?h=61a3fa5973" width="640" height="400" frameborder="0" allow="autoplay; fullscreen; picture-in-picture" allowfullscreen></iframe>
</div>
</div>
</div>
<!--/ Paper video. -->
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<!-- Overview. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Overview</h2>
<div class="content has-text-justified">
<p>
<span class="dnerf">Binder</span> first prompts Codex to parse a question input into a <span class="dnerf">Binder</span> program, in which Codex has to decide (1) which parts in the input can be converted to the target programming language (<span style="color:dimgrey;">grey clause</span> in figure), (2) the corresponding task API calls (<span style="color: royalblue">blue clause</span> in figure) to prompt Codex to resolve the other parts, and (3) where to insert the API calls in the <span class="dnerf">Binder</span> program.
Next, <span class="dnerf">Binder</span> prompts Codex again to generate answers to the task API calls (given the generated task prompts), integrates the generated results back to the programming language, and executes the resulting programming language expression to derive the final answer.
In this way, <span class="dnerf">Binder</span> enables flexible functionality integration to the programming language to improve its grammar coverage while requiring only few-shot exemplar annotations.
</p>
</div>
</div>
</div>
<img src="static/images/intro.png"> <!-- To .gif format-->
</div>
<!--/ Overview. -->
</section>
<section class="section">
<div class="container is-max-desktop">
<!-- Results. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Results</h2>
<div class="content has-text-justified">
<p>
<span class="dnerf">Binder</span> framework achieves <strong>SOTA</strong> or comparable performance with only <strong>dozens of</strong> program annotations <strong>without training</strong> on three benchmarks!
</p>
<ul>
<li>SOTA on <a href="https://ppasupat.github.io/WikiTableQuestions/">WikiTableQuestions</a>.</li>
<li>SOTA on <a href="https://tabfact.github.io/">TabFact</a>.</li>
<li>Comparable to SOTA on <a href="https://allenai.github.io/multimodalqa/">MultiModalQA</a>. </li>
</ul>
<img src="static/images/exp_results.png">
<p>
Further, <span class="dnerf">Binder</span> is much more robust to large or noisy inputs than the end-to-end manner.
</p>
<img src="static/images/analysis.png">
</div>
</div>
</div>
</div>
<!--/ Results. -->
</section>
<section class="section">
<div class="container is-max-desktop">
<!-- Examples. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">More <span class="dnerf">Binder</span> Program Examples</h2>
</div>
</div>
<img src="static/images/examples.png">
</div>
<!--/ Examples. -->
</section>
<section class="section">
<div class="container is-max-desktop">
<!-- Prompt play. -->
Copy our prompt from <a href="https://github.com/HKUNLP/Binder/blob/main/templates/prompts/wikitq_binder.txt">here</a> with your input (table, question etc.) on <a href="https://beta.openai.com/playground">OpenAI Playground</a> to generate Binder program by yourself!
</div>
<!--/ Prompt play. -->
</section>
<!-- <iframe
src="https://hkunlp-binder.hf.space"
frameborder="0"
width="850"
height="450"
/> -->
<section class="section">
<div class="container is-max-desktop content">
<h2 class="title">Acknowledgement</h2>
We thank <a href="https://dpfried.github.io/">Daniel Fried</a>, <a href="https://hongjin-su.github.io/">Hongjin Su</a>, <a href="https://github.com/ChenWu98">Chen Henry Wu</a>, <a href="https://ryanzhumich.github.io/">Rui Zhang</a>, <a href="https://berlino.github.io/">Bailin Wang</a>, <a href="https://pcyin.me/">Pengcheng Yin</a>, <a href="https://homes.cs.washington.edu/~yizhongw/">Yizhong Wang</a>, <a href="https://weijia-shi.netlify.app/">Weijia Shi</a> for their helpful feedback on this work.
</div>
</section>
<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>@article{Binder,
title={Binding Language Models in Symbolic Languages},
author={Zhoujun Cheng and Tianbao Xie and Peng Shi and Chengzu Li and Rahul Nadkarni and Yushi Hu and Caiming Xiong and Dragomir Radev and Mari Ostendorf and Luke Zettlemoyer and Noah A. Smith and Tao Yu},
journal={ICLR},
year={2023}
}</code></pre>
</div>
</section>
<footer class="footer">
<div class="container">
<div class="content has-text-centered">
<a class="icon-link"
href="https://arxiv.org/pdf/2210.02875.pdf">
<i class="fas fa-file-pdf"></i>
</a>
<a class="icon-link" href="https://github.com/HKUNLP/binder" class="external-link" disabled>
<i class="fab fa-github"></i>
</a>
</div>
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This website is licensed under a <a rel="license"
href="http://creativecommons.org/licenses/by-sa/4.0/">Creative
Commons Attribution-ShareAlike 4.0 International License</a>.
</p>
<p>
This means you are free to borrow the <a
href="https://github.com/nerfies/nerfies.github.io">source code</a> of this website,
we just ask that you link back to this page in the footer.
Please remember to remove the analytics code included in the header of the website which
you do not want on your website.
</p>
</div>
</div>
</div>
</div>
</footer>
</body>
</html>