-
Notifications
You must be signed in to change notification settings - Fork 7
/
retargeting.html
188 lines (171 loc) · 15.4 KB
/
retargeting.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
<!DOCTYPE html><!-- Last Published: Fri Mar 27 2020 21:28:31 GMT+0000 (Coordinated Universal Time) -->
<html data-wf-domain="www.matthewtancik.com" data-wf-page="5e6fb768456f961381500a5f" data-wf-site="51e0d73d83d06baa7a00000f">
<head>
<meta charset="utf-8"/>
<title>Skeleton-Aware Networks</title>
<meta content="A motion processing framework for cross-strutcural motion retargeting." name="description"/>
<meta content="Skeleton-Aware Networks" property="og:title"/>
<meta content="An end-to-end method for cross-structural motion retargeting" property="og:description"/>
<meta content="summary" name="twitter:card"/><meta content="width=device-width, initial-scale=1" name="viewport"/>
<link href="./motion_editing.css" rel="stylesheet" type="text/css"/>
<script src="https://ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js" type="text/javascript"></script>
<script type="text/javascript">WebFont.load({ google: { families: ["Lato:100,100italic,300,300italic,400,400italic,700,700italic,900,900italic","Montserrat:100,100italic,200,200italic,300,300italic,400,400italic,500,500italic,600,600italic,700,700italic,800,800italic,900,900italic","Ubuntu:300,300italic,400,400italic,500,500italic,700,700italic","Changa One:400,400italic","Open Sans:300,300italic,400,400italic,600,600italic,700,700italic,800,800italic","Varela Round:400","Bungee Shade:regular","Roboto:300,regular,500"] }});</script>
<!--[if lt IE 9]><script src="https://cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv.min.js" type="text/javascript"></script><![endif]-->
<script type="text/javascript">!function(o,c){var n=c.documentElement,t=" w-mod-";n.className+=t+"js",("ontouchstart"in o||o.DocumentTouch&&c instanceof DocumentTouch)&&(n.className+=t+"touch")}(window,document);</script>
<link href="images/logo.png" rel="shortcut icon" type="image/x-icon"/>
<link href="images/logo.png" rel="apple-touch-icon"/>
<style>
.wf-loading * {
opacity: 0;
}
</style></head>
<body>
<div class="section hero nerf-_v2">
<div class="container-2 nerf_header_v2 w-container">
<h1 class="nerf_title_v2">Skeleton-Aware Networks for Deep Motion Retargeting</h1>
<h1 class="nerf_subheader_v2">SIGGRAPH 2020</h1>
<div class="nerf_authors_list_single w-row">
<div class="w-col w-col-2 w-col-small-4 w-col-tiny-6"><a href="http://kfiraberman.github.io/" target="_blank" class="nerf_authors_v2">Kfir Aberman<span class="text-span_nerf_star">*</span><span class="superscript text-span_nerf">1,5*</span></a></div>
<div class="w-col w-col-2 w-col-small-4 w-col-tiny-6"><a href="https://peizhuoli.github.io/" target="_blank" class="nerf_authors_v2">Peizhuo Li<span class="text-span_nerf_star">*</span><span class="superscript text-span_nerf">2,1*</span></a></div>
<div class="w-col w-col-2 w-col-small-4 w-col-tiny-6"><a href="https://www.cse.huji.ac.il/~danix/" target="_blank" class="nerf_authors_v2">Dani Lischinski<span class="text-span_nerf">3,1</span><span class="superscript"></span></a></div>
<div class="w-col w-col-2 w-col-small-4 w-col-tiny-6"><a href="https://igl.ethz.ch/people/sorkine/" target="_blank" class="nerf_authors_v2">Olga Sorkine-Hornung<span class="text-span_nerf">4,1</span><span class="superscript"></span></a></div>
<div class="w-col w-col-2 w-col-small-4 w-col-tiny-6"><a href="https://www.cs.tau.ac.il/~dcor/" target="_blank" class="nerf_authors_v2">Daniel Cohen-Or<span class="text-span_nerf">5,1</span></a></div>
<div class="w-col w-col-2 w-col-small-4 w-col-tiny-6"><a href="https://cfcs.pku.edu.cn/baoquan" target="_blank" class="nerf_authors_v2">Baoquan Chen<span class="text-span_nerf">2,1</span></a></div>
</div>
<div class="columns-6 w-row"><div class="nerf_mobile_col_inst w-col w-col-4 w-col-small-4 w-col-tiny-4">
<div class="nerf_mobile_inst"><span class="text-span_nerf">1 </span>Beijing Film Academy</div></div>
<div class="nerf_mobile_col_inst w-col w-col-4 w-col-small-4 w-col-tiny-4"><div class="nerf_mobile_inst"><span class="text-span_nerf">2</span>Peking University</div></div>
<div class="nerf_mobile_col_inst w-col w-col-4 w-col-small-4 w-col-tiny-4"><div class="nerf_mobile_inst"><span class="text-span_nerf">3</span>Hebrew University</div></div>
<div class="nerf_mobile_col_inst w-col w-col-4 w-col-small-4 w-col-tiny-4"><div class="nerf_mobile_inst"><span class="text-span_nerf">4</span>ETH Zurich</div></div>
<div class="nerf_mobile_col_inst w-col w-col-4 w-col-small-4 w-col-tiny-4"><div class="nerf_mobile_inst"><span class="text-span_nerf">5</span>Tel-Aviv University</div></div>
</div>
<div class="nerf_authors_list_single nerf_authors_affiliation w-row">
<div class="w-col w-col-2"><h1 class="nerf_affiliation_v2">Beijing Film Academy</h1></div>
<div class="column w-col w-col-2"><h1 class="nerf_affiliation_v2">Peking University</h1></div>
<div class="w-col w-col-2"><h1 class="nerf_affiliation_v2">Hebrew University</h1></div>
<div class="w-col w-col-2"><h1 class="nerf_affiliation_v2">ETH Zurich</h1></div>
<div class="w-col w-col-2"><h1 class="nerf_affiliation_v2">Tel-Aviv University</h1></div>
<div class="w-col w-col-2"><h1 class="nerf_affiliation_v2">Peking University</h1></div>
</div>
<div class="div-block-10"><div class="nerf_equal_v2"><span class="text-span_nerf">*</span><span class="text-span_nerf_star">*</span>Denotes Equal Contribution</div></div>
<!-- <img src="./images/teaser.png" alt=""/> -->
<div>
<span class="center"><img src="images/video_teaser.gif"></span>
<!-- <div class="w-col w-col-4"><a class="examples_header">Input</a></div>
<div class="w-col w-col-4"><a class="examples_header">Output 1</a></div>
<div class="w-col w-col-4"><a class="examples_header">Output 2</a></div> -->
</div>
<div class="link_column_nerf_v2 w-row">
<div class="w-col w-col-4 w-col-small-4 w-col-tiny-4">
<a href="./papers/skeleton-aware-camera-ready.pdf" target="_blank" class="link-block w-inline-block">
<img src="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5cab99df4998decfbf9e218e_paper-01.png" alt="paper" class="paper_img image-8 github_icon_nerf_v2"/></a>
<!-- <img src="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5cab99df4998decfbf9e218e_paper-01.png" alt="paper" srcset="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5cab99df4998decfbf9e218e_paper-01-p-500.png 500w, https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5cab99df4998decfbf9e218e_paper-01.png 672w" sizes="(max-width: 479px) 12vw, (max-width: 767px) 7vw, (max-width: 991px) 41.84375px, 56.6875px" class="paper_img image-8_nerf"/></a> -->
</div>
<div class="w-col w-col-4 w-col-small-4 w-col-tiny-4">
<a href="https://github.com/DeepMotionEditing/deep-motion-editing" target="_blank" class="link-block w-inline-block">
<img src="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5cae3b53b42ebb3dd4175a82_68747470733a2f2f7777772e69636f6e66696e6465722e636f6d2f646174612f69636f6e732f6f637469636f6e732f313032342f6d61726b2d6769746875622d3235362e706e67.png" alt="paper" class="paper_img image-8 github_icon_nerf_v2"/></a>
</div>
<div class="column-2 w-col w-col-4 w-col-small-4 w-col-tiny-4"><a href="https://docs.google.com/uc?export=download&id=1_849LvuT3WBEHktBT97P2oMBzeJz7-UP" target="_blank" class="link-block w-inline-block">
<img src="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5e7136849ee3b0a0c6a95151_database.svg" alt="paper" class="paper_img image-8_nerf nerf_db_icon"/></a>
</div>
</div>
<div class="paper_code_nerf w-row">
<div class="w-col w-col-4 w-col-small-4 w-col-tiny-4">
<div class="text-block-2"><strong class="bold-text-nerf_v2">Paper</strong></div>
</div>
<div class="w-col w-col-4 w-col-small-4 w-col-tiny-4"><div class="text-block-2">
<strong class="bold-text-nerf_v2">Code</strong></div>
</div>
<div class="w-col w-col-4 w-col-small-4 w-col-tiny-4">
<div class="text-block-2"><strong class="bold-text-nerf_v2">Data</strong></div>
</div>
</div>
<div class="nerf_slide_nav w-slider-nav w-slider-nav-invert w-round"></div></div></div>
<div data-anchor="slide1" class="section nerf_section">
<div class="w-container"><h2 class="grey-heading_nerf">Overview Video</h2>
<div style="padding-top:56.17021276595745%" id="w-node-e5e45b1d55ac-81500a5f" class="w-embed-youtubevideo stega_movie youtube">
<iframe src="https://www.youtube.com/embed/ym8Tnmiz5N8?rel=1&controls=1&autoplay=0&mute=0&start=0" frameBorder="0" style="position:absolute;left:0;top:0;width:100%;height:100%;pointer-events:auto" allow="autoplay; encrypted-media" allowfullscreen=""></iframe>
</div></div>
</div>
<div data-anchor="slide1" class="section nerf_section">
<div class="grey_container w-container">
<h2 class="grey-heading_nerf">Abstract & Method</h2>
<p class="paragraph-3 nerf_text">We introduce a novel deep learning framework for data-driven motion retargeting between skeletons, which may have different structure, yet corresponding to homeomorphic graphs.
Importantly, our approach learns how to retarget without requiring any explicit pairing between the motions in the training set.
We leverage the fact that different homeomorphic skeletons may be reduced to a common primal skeleton by a sequence of edge merging operations, which we refer to as skeletal pooling.</p>
<img src="./images/pool_to_primal.png" alt="" class="nerf_network"/>
<p class="paragraph-3 nerf_text">Our main technical contribution is the introduction of novel differentiable convolution, pooling, and unpooling operators, which are skeleton-aware, meaning that they explicitly account for the skeleton's hierarchical structure and joint adjacency.</p>
<img src="./images/conv_and_pool.png" alt="" class="nerf_network"/>
<p class="paragraph-3 nerf_text">Our operators form the building blocks of a new deep motion processing framework that embeds the motion into a common latent space, shared by a collection of homeomorphic skeletons.
Thus, retargeting can be achieved simply by encoding to, and decoding from this latent space.</p>
<img src="./images/common_latent.png" alt="" class="nerf_network"/>
</div></div>
<div class="white_section_nerf">
<div class="w-container">
<h2 class="grey-heading_nerf">Cross-Structural Motion Retargeting</h2>
<p class="paragraph-3 nerf_text nerf_results_text">Our method can retarget motion to asymmetric characters or character with missing/extra bones:</p>
<span class="center">
<img class="skeleton_example" src="images/assymetric.png">
<img class="skeleton_example" src="images/extra_limb.png">
</span>
<div class="center">
<span class="center">
<img class="special_characters" src="images/assymetric.gif">
<img class="special_characters" src="images/extra_limb.gif">
</span>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Input</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Retargeted Output</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Input</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Retargeted Output</a></div>
</div>
</div></div>
<div class="white_section_nerf">
<div class="grey_container w-container">
<h2 class="grey-heading_nerf">Intra-Structural Retargeting - Comparison</h2>
<p class="paragraph-3 nerf_text nerf_results_text">Our framework can be used also for retargeting of skeletons with
the same structure, but different proportions. Here our method is compared to a naive adaptation of CycleGAN [Zhu et al. 2017] to the motion domain and to NKN
of Villegas et al. [2018]. The outputs are overlaid with the ground truth (green skeleton):</p>
<span class="center"><img class="skeleton_example" src="images/intra_structural.png"></span>
<div class="center">
<span class="center"><img src="images/intra_structural.gif"></span>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Input</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">CycleGAN [2016] (Adaptation)</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">NKN [2018]</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Ours</a></div>
</div>
</div></div>
<div data-anchor="slide1" class="section nerf_section"></div>
<div class="white_section_nerf">
<div class="w-container">
<h2 class="grey-heading_nerf">Cross-Structural Retargeting - Comparison</h2>
<p class="paragraph-3 nerf_text nerf_results_text">Our method is compared to a naive adaptation of CycleGAN [Zhu et al. 2017] to the motion domain and to a
cross-structural version of NKN of Villegas et al. [2018]. The outputs are overlaid with the ground truth (green skeleton):</p>
<span class="center"><img class="skeleton_example" src="images/cross_structural.png"></span>
<div class="center">
<span class="center"><img src="images/cross_structural.gif"></span>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Input</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">CycleGAN [2016] (Adaptation)</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">NKN [2018] (Adaptation)</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Ours</a></div>
</div>
</div></div>
<div class="white_section_nerf">
<div class="w-container">
<h2 class="grey-heading_nerf">BibTeX</h2>
<div class="grey_container w-container">
<div class="bibtex">
<pre><code>@article{aberman2020skeleton,
author = {Aberman, Kfir and Li, Peizhuo and Lischinski, Dani and Sorkine-Hornung, Olga and Cohen-Or, Daniel and Chen, Baoquan},
title = {Skeleton-Aware Networks for Deep Motion Retargeting},
journal = {ACM Transactions on Graphics (TOG)},
volume = {39},
number = {4},
pages = {62},
year = {2020},
publisher = {ACM}
}</code></pre>
</div>
</div>
</div>
</div>
<script src="https://d3e54v103j8qbb.cloudfront.net/js/jquery-3.4.1.min.220afd743d.js?site=51e0d73d83d06baa7a00000f" type="text/javascript" integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script><script src="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/js/webflow.3057c11af.js" type="text/javascript"></script><!--[if lte IE 9]><script src="//cdnjs.cloudflare.com/ajax/libs/placeholders/3.0.2/placeholders.min.js"></script><![endif]--></body></html>