-
Notifications
You must be signed in to change notification settings - Fork 7
/
style_transfer.html
209 lines (187 loc) · 16.2 KB
/
style_transfer.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
<!DOCTYPE html><!-- Last Published: Fri Mar 27 2020 21:28:31 GMT+0000 (Coordinated Universal Time) -->
<html data-wf-domain="www.matthewtancik.com" data-wf-page="5e6fb768456f961381500a5f" data-wf-site="51e0d73d83d06baa7a00000f">
<head>
<meta charset="utf-8"/>
<title>Motion Style Transfer</title>
<meta content="A motion processing framework for cross-strutcural motion retargeting." name="description"/>
<meta content="Skeleton-Aware Networks" property="og:title"/>
<meta content="An end-to-end method for cross-structural motion retargeting" property="og:description"/>
<meta content="summary" name="twitter:card"/><meta content="width=device-width, initial-scale=1" name="viewport"/>
<link href="./motion_editing.css" rel="stylesheet" type="text/css"/>
<script src="https://ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js" type="text/javascript"></script>
<script type="text/javascript">WebFont.load({ google: { families: ["Lato:100,100italic,300,300italic,400,400italic,700,700italic,900,900italic","Montserrat:100,100italic,200,200italic,300,300italic,400,400italic,500,500italic,600,600italic,700,700italic,800,800italic,900,900italic","Ubuntu:300,300italic,400,400italic,500,500italic,700,700italic","Changa One:400,400italic","Open Sans:300,300italic,400,400italic,600,600italic,700,700italic,800,800italic","Varela Round:400","Bungee Shade:regular","Roboto:300,regular,500"] }});</script>
<!--[if lt IE 9]><script src="https://cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv.min.js" type="text/javascript"></script><![endif]-->
<script type="text/javascript">!function(o,c){var n=c.documentElement,t=" w-mod-";n.className+=t+"js",("ontouchstart"in o||o.DocumentTouch&&c instanceof DocumentTouch)&&(n.className+=t+"touch")}(window,document);</script>
<link href="images/logo.png" rel="shortcut icon" type="image/x-icon"/>
<link href="images/logo.png" rel="apple-touch-icon"/>
<style>
.wf-loading * {
opacity: 0;
}
</style></head>
<body>
<div class="section hero nerf-_v2">
<div class="container-2 nerf_header_v2 w-container">
<h1 class="nerf_title_v2">Unpaired Motion Style Transfer from Video to Animation</h1>
<h1 class="nerf_subheader_v2">SIGGRAPH 2020</h1>
<div class="nerf_authors_list_single w-row">
<div class="w-col w-col-five w-col-small-4 w-col-tiny-6"><a href="http://kfiraberman.github.io/" target="_blank" class="nerf_authors_v2">Kfir Aberman<span class="text-span_nerf_star">*</span><span class="superscript text-span_nerf">1,4*</span></a></div>
<div class="w-col w-col-five w-col-small-4 w-col-tiny-6"><a href="https://halfsummer11.github.io/" target="_blank" class="nerf_authors_v2">Yijia Weng<span class="text-span_nerf_star">*</span><span class="superscript text-span_nerf">2,1*</span></a></div>
<div class="w-col w-col-five w-col-small-4 w-col-tiny-6"><a href="https://www.cse.huji.ac.il/~danix/" target="_blank" class="nerf_authors_v2">Dani Lischinski<span class="text-span_nerf">3,1</span></a></div>
<div class="w-col w-col-five w-col-small-4 w-col-tiny-6"><a href="https://www.cs.tau.ac.il/~dcor/" target="_blank" class="nerf_authors_v2">Daniel Cohen-Or<span class="text-span_nerf">4,1</span></a></div>
<div class="w-col w-col-five w-col-small-4 w-col-tiny-6"><a href="https://cfcs.pku.edu.cn/baoquan" target="_blank" class="nerf_authors_v2">Baoquan Chen<span class="text-span_nerf">2,1</span></a></div>
</div>
<div class="columns-6 w-row">
<div class="nerf_mobile_col_inst w-col w-col-4 w-col-small-4 w-col-tiny-4"><div class="nerf_mobile_inst"><span class="text-span_nerf">1 </span>Beijing Film Academy</div></div>
<div class="nerf_mobile_col_inst w-col w-col-4 w-col-small-4 w-col-tiny-4"><div class="nerf_mobile_inst"><span class="text-span_nerf">2</span>Peking University</div></div>
<div class="nerf_mobile_col_inst w-col w-col-4 w-col-small-4 w-col-tiny-4"><div class="nerf_mobile_inst"><span class="text-span_nerf">3</span>Hebrew University</div></div>
<div class="nerf_mobile_col_inst w-col w-col-4 w-col-small-4 w-col-tiny-4"><div class="nerf_mobile_inst"><span class="text-span_nerf">4</span>Tel-Aviv University</div></div>
</div>
<div class="nerf_authors_list_single nerf_authors_affiliation w-row">
<div class="w-col w-col-five"><h1 class="nerf_affiliation_v2">Beijing Film Academy</h1></div>
<div class="w-col w-col-five"><h1 class="nerf_affiliation_v2">Peking University</h1></div>
<div class="w-col w-col-five"><h1 class="nerf_affiliation_v2">Hebrew University</h1></div>
<div class="w-col w-col-five"><h1 class="nerf_affiliation_v2">Tel-Aviv University</h1></div>
<div class="w-col w-col-five"><h1 class="nerf_affiliation_v2">Peking University</h1></div></div>
<div class="div-block-10"><div class="nerf_equal_v2"><span class="text-span_nerf">*</span><span class="text-span_nerf_star">*</span>Denotes Equal Contribution</div></div>
<div>
<span class="center"><img src="images/style_transfer/teaser.gif"></span>
<!-- <div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Input Content</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Input Style</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header"></a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Output</a></div> -->
<!-- <div class="w-col w-col-4"><a class="examples_header">Input</a></div>
<div class="w-col w-col-4"><a class="examples_header">Output 1</a></div>
<div class="w-col w-col-4"><a class="examples_header">Output 2</a></div> -->
</div>
<div class="link_column_nerf_v2 w-row">
<div class="w-col w-col-4 w-col-small-4 w-col-tiny-4">
<a href="./papers/Motion_Style_Transfer-camera-ready.pdf" target="_blank" class="link-block w-inline-block">
<img src="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5cab99df4998decfbf9e218e_paper-01.png" alt="paper" class="paper_img image-8 github_icon_nerf_v2"/></a>
<!-- <img src="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5cab99df4998decfbf9e218e_paper-01.png" alt="paper" srcset="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5cab99df4998decfbf9e218e_paper-01-p-500.png 500w, https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5cab99df4998decfbf9e218e_paper-01.png 672w" sizes="(max-width: 479px) 12vw, (max-width: 767px) 7vw, (max-width: 991px) 41.84375px, 56.6875px" class="paper_img image-8_nerf"/></a> -->
</div>
<div class="w-col w-col-4 w-col-small-4 w-col-tiny-4">
<a href="https://github.com/DeepMotionEditing/deep-motion-editing" target="_blank" class="link-block w-inline-block">
<img src="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5cae3b53b42ebb3dd4175a82_68747470733a2f2f7777772e69636f6e66696e6465722e636f6d2f646174612f69636f6e732f6f637469636f6e732f313032342f6d61726b2d6769746875622d3235362e706e67.png" alt="paper" class="paper_img image-8 github_icon_nerf_v2"/></a>
</div>
<div class="column-2 w-col w-col-4 w-col-small-4 w-col-tiny-4"><a href="https://drive.google.com/drive/folders/1C-_iZJj-PSUWZwh25yAsQe1tLpPm9EZ5" target="_blank" class="link-block w-inline-block">
<img src="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/5e7136849ee3b0a0c6a95151_database.svg" alt="paper" class="paper_img image-8_nerf nerf_db_icon"/></a>
</div>
</div>
<div class="paper_code_nerf w-row">
<div class="w-col w-col-4 w-col-small-4 w-col-tiny-4">
<div class="text-block-2"><strong class="bold-text-nerf_v2">Paper</strong></div>
</div>
<div class="w-col w-col-4 w-col-small-4 w-col-tiny-4">
<div class="text-block-2">
<strong class="bold-text-nerf_v2">Code</strong>
</div>
</div>
<div class="w-col w-col-4 w-col-small-4 w-col-tiny-4">
<div class="text-block-2"><strong class="bold-text-nerf_v2">Data</strong></div>
</div>
</div>
<div class="nerf_slide_nav w-slider-nav w-slider-nav-invert w-round"></div>
</div>
</div>
<div data-anchor="slide1" class="section nerf_section">
<div class="w-container"><h2 class="grey-heading_nerf">Overview Video</h2>
<div style="padding-top:56.17021276595745%" id="w-node-e5e45b1d55ac-81500a5f" class="w-embed-youtubevideo stega_movie youtube">
<iframe src="https://www.youtube.com/embed/m04zuBSdGrc?rel=1&controls=1&autoplay=0&mute=0&start=0" frameBorder="0" style="position:absolute;left:0;top:0;width:100%;height:100%;pointer-events:auto" allow="autoplay; encrypted-media" allowfullscreen=""></iframe>
</div></div>
</div>
<div data-anchor="slide1" class="section nerf_section">
<div class="grey_container w-container">
<h2 class="grey-heading_nerf">Abstract & Method</h2>
<p class="paragraph-3 nerf_text">We present a novel data-driven framework for motion style transfer, which learns from an unpaired collection of motions with style labels, and enables transferring motion styles not observed during training.
Furthermore, our framework is able to extract motion styles directly from videos, bypassing 3D reconstruction - an ability which enables one to extend the set of style examples far beyond motions captured by MoCap systems.</p>
<img src="./images/style_transfer/teaser.png" alt="" class="nerf_network teaser"/>
<p class="paragraph-3 nerf_text">Our style transfer network encodes motions into two latent codes, for content and for style, each of which plays a different role in the decoding (synthesis) process. While the content code is decoded into the output motion by several temporal convolutional layers, the style code modifies deep features via temporally invariant adaptive instance normalization (AdaIN).</p>
<img src="./images/style_transfer/architecture.png" alt="" class="nerf_network architecture"/>
<p class="paragraph-3 nerf_text">While the content code is encoded from 3D joint rotations, we learn a common embedding for style from either 3D or 2D joint positions, enabling style extraction from videos.</p>
<img src="./images/style_transfer/clusters.png" alt="" class="nerf_network"/>
</div></div>
<div class="white_section_nerf">
<div class="w-container">
<h2 class="grey-heading_nerf">Motion Style Transfer</h2>
<p class="paragraph-3 nerf_text nerf_results_text">Given content and style input motions our network can synthesize novel motions that exhibit the content of one input and the style of the other.</p>
<div class="center">
<span class="center">
<img class="special_characters" src="images/style_transfer/3D_example_1.gif">
<img class="special_characters" src="images/style_transfer/3D_example_2.gif">
</span>
<!-- <div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Input</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Retargeted Output</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Input</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Retargeted Output</a></div> -->
</div>
</div></div>
<div class="white_section_nerf">
<div class="grey_container w-container">
<h2 class="grey-heading_nerf">Motion Style Transfer from Video</h2>
<p class="paragraph-3 nerf_text nerf_results_text">Our network can infer style from 2D key-points extracted from video and apply it to 3D animations.</p>
<div class="center">
<span class="center">
<img class="special_characters" src="images/style_transfer/2D_example_1.gif">
<img class="special_characters" src="images/style_transfer/2D_example_2.gif">
</span>
<!-- <span class="center"><img src="images/intra_structural.gif"></span>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Input</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">CycleGAN [2016] (Adaptation)</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">NKN [2018]</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Ours</a></div> -->
</div>
</div></div>
<!-- <div class="white_section_nerf">
<div class="grey_container w-container">
<h2 class="grey-heading_nerf">Unseen Styles</h2>
<p class="paragraph-3 nerf_text nerf_results_text">TBC</p>
<div class="center">
</div>
</div></div> -->
<div data-anchor="slide1" class="section nerf_section"></div>
<div class="white_section_nerf">
<div class="w-container">
<h2 class="grey-heading_nerf">Motion Style Transfer - Comparison</h2>
<p class="paragraph-3 nerf_text nerf_results_text">Qualitative comparison of our method to the approach of Holden et al. [2016].</p>
<div class="center">
<span class="center"><img src="images/style_transfer/comparison_3d.gif"></span>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Content Input</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Style Input</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Holden et al. [2016]</a></div>
<div class="w-col w-col-3 w-col-small-3 w-col-tiny-3"><a class="examples_header">Ours</a></div>
</div>
</div></div>
<div class="white_section_nerf">
<div class="grey_container w-container">
<h2 class="grey-heading_nerf">Style Interpolation</h2>
<p class="paragraph-3 nerf_text nerf_results_text">Our learned continuous style code space can be used to interpolate between styles. Style interpolation can be achieved by linearly interpolating between style code, and then decoding the results through our decoder.</p>
<div class="center">
<span class="center">
<img class="special_characters" src="images/style_transfer/interpolation_1.gif">
<img class="special_characters" src="images/style_transfer/interpolation_2.gif">
</span>
<div class="w-col w-col-6 w-col-small-6 w-col-tiny-6"><a class="examples_header">Old to Neutral</a></div>
<div class="w-col w-col-6 w-col-small-6 w-col-tiny-6"><a class="examples_header">Proud to Depressed</a></div>
</div>
</div></div>
<div class="white_section_nerf">
<div class="w-container">
<h2 class="grey-heading_nerf">BibTeX</h2>
<div class="grey_container w-container">
<div class="bibtex">
<pre><code>@article{aberman2020unpaired,
author = {Aberman, Kfir and Weng, Yijia and Lischinski, Dani and Cohen-Or, Daniel and Chen, Baoquan},
title = {Unpaired Motion Style Transfer from Video to Animation},
journal = {ACM Transactions on Graphics (TOG)},
volume = {39},
number = {4},
pages = {64},
year = {2020},
publisher = {ACM}
}</code></pre>
</div>
</div>
</div>
</div>
<script src="https://d3e54v103j8qbb.cloudfront.net/js/jquery-3.4.1.min.220afd743d.js?site=51e0d73d83d06baa7a00000f" type="text/javascript" integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script><script src="https://uploads-ssl.webflow.com/51e0d73d83d06baa7a00000f/js/webflow.3057c11af.js" type="text/javascript"></script><!--[if lte IE 9]><script src="//cdnjs.cloudflare.com/ajax/libs/placeholders/3.0.2/placeholders.min.js"></script><![endif]--></body></html>