diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..f2d5de1 --- /dev/null +++ b/.gitignore @@ -0,0 +1,10 @@ +*.gz +*.dvi +*.out +*.bst +*.xlsx +*.aux +*.blg +*.glsdefs +*.log +*.bbl diff --git a/Back/Acronyms.tex b/Back/Acronyms.tex new file mode 100644 index 0000000..741f327 --- /dev/null +++ b/Back/Acronyms.tex @@ -0,0 +1,6 @@ +\chapter{Acronyms} + \textbf{3DMM} - 3D Morphable Model\\ + \textbf{MCMC} - Markov Chain Monte Carlo\\ + \textbf{ICP} - Iterative Closest Point\\ + \textbf{DDMCMC} - Data Driven Markov Chain Monte Carlo + diff --git a/Back/AppendixA.tex b/Back/AppendixA.tex new file mode 100755 index 0000000..478ae8b --- /dev/null +++ b/Back/AppendixA.tex @@ -0,0 +1,9 @@ +\chapter{Thrift API Scheme} + +\begin{figure} + \centering + \includegraphics[angle=-90, origin=c, width=0.40\textwidth]{Figures/thrift/Rs-ex.pdf} + \caption{The API scheme we have implemented to obtain the required data.} + \label{fA.1} +\end{figure} + diff --git a/Back/DeclarationOfAuthorship.tex b/Back/DeclarationOfAuthorship.tex new file mode 100755 index 0000000..0626192 --- /dev/null +++ b/Back/DeclarationOfAuthorship.tex @@ -0,0 +1,38 @@ +\chapter[Declaration on Scientific Integrity]{Declaration on Scientific Integrity\\Erklärung zur wissenschaftlichen Redlichkeit} +\label{DeclarationOfAuthorship} + +includes Declaration on Plagiarism and Fraud \\ +beinhaltet Erklärung zu Plagiat und Betrug \vspace{1cm} + +\formlabel{Author}{Autor} +\authorsint + +\formlabel{Matriculation number}{Matrikelnummer} +\immatriculnrint + +\formlabel{Title of work}{Titel der Arbeit} +\titleint + +\formlabel{Type of work}{Typ der Arbeit} +\thesistypeint + +\formlabel{Declaration}{Erklärung} +I hereby declare that this submission is my own work and that I have fully acknowledged the assistance received in completing this work and that it contains no material that has not been formally acknowledged. +I have mentioned all source materials used and have cited these in accordance with recognised scientific rules. + +\vspace{0.3cm} + +Hiermit erkläre ich, dass mir bei der Abfassung dieser Arbeit nur die darin angegebene +Hilfe zuteil wurde und dass ich sie nur mit den in der Arbeit angegebenen Hilfsmitteln +verfasst habe. Ich habe sämtliche verwendeten Quellen erwähnt und gemäss anerkannten wissenschaftlichen Regeln zitiert. + + +\vspace*{0.5cm} + +Basel, \dateint +\vspace*{0.25cm} + +\begin{flushright} +\rule{75mm}{0.4pt} \\ +\formlabel{Signature}{Unterschrift} +\end{flushright} diff --git a/Chapters/Chapter1.tex b/Chapters/Chapter1.tex new file mode 100755 index 0000000..af6ca7f --- /dev/null +++ b/Chapters/Chapter1.tex @@ -0,0 +1,51 @@ +\chapter{Introduction}\label{ch1} + +Depth camera technologies have gotten a lot of attention lately. The key reason behind its substantial popularity growth and technological advancements is the fact that a lot of computer vision applications rely on its 3D sensing capabilities. As the demand for Machine Learning (ML) systems grow, we observe a rapid growth of the technologies using 3D sensing systems across different technology areas, be it a consumer electronics, self-driving cars or drones. The one such market which has gotten noticeably better and cheaper lately is the consumer depth camera market. Embedded into the latest generation of smartphones, depth camera modules found its way to scale of consumers. Standalone depth cameras also got better, cheaper and smaller while maintaining its qualitative advantages. While the former are designed to enhance overall image quality, stability and introduce industry's first dedicated sensor based Augmented Reality (AR) features inside the mobile operating system, the latter are designed to deal with a wide range of computationally heavy and accuracy focused applications. Such applications include object detection, motion tracking, face scanning, object reconstruction, etc. \bigskip + +In this thesis, we focus on the 3D reconstruction of the human face. A dense 3D face reconstruction from a single 2D image is a long-standing and widely researched topic in computer vision. Approaches include, model based probabilistic approaches\cite{Romdhani3DM, Schoenborn2017, 10.1007/978-3-642-40602-7_11} or nowadays the more popular convolution neural networks (CNN)\cite{Guo_2019, tu2019joint}. The aim of this thesis is to incorporate consumer depth camera capabilities into the existing model based 3D face reconstruction pipeline hereafter referred to as a face fitting pipeline using 3D Morphable Model (3DMM)\cite{Blanz:1999:MMS:311535.311556, Romdhani3DM}. We also attempt to replicate and potentially improve the result of the original 2D face fitting pipeline \cite{Schoenborn2017} as well as augmented depth fitting \cite{betschard2016} pipeline, which proved to improve the original approach. The original 2D fitting approach was performing the reconstruction based solely on a single image, while its improved version was utilizing an additional depth image information. Our main goal is to design and implement a stable client-server based system which robustly and efficiently takes care of all the sensitive details of both the image capturing procedure and later face fitting pipeline. In addition to that, we will investigate the possibility to speed up the fitting process so that the final pipeline could be used for live demos and could seamlessly be integrated into the existing Scalismo Face Morpher web service (Figure \ref{f1.1}). + +\begin{figure} + \centering + \includegraphics[width=\textwidth]{Figures/site.PNG} + \caption{Scalismo Face Morpher website — \url{https://face-morpher.scalismo.org}.} + \label{f1.1} +\end{figure} + +\section{Motivation \& Previous Work} +Reconstructing a 3D face from a single 2D image is a challenging task. The major issue the fitting process has to overcome is to efficiently recover the 3D shape information and overall appearance of the face in the image. The appearance of the face in the 3DMM environment is being controlled by a set of model parameters that control faces pose, shape, expression, color, and illumination. Face fitting, therefore, is a pipeline that takes a 2D color image with a human face in it (hereafter referred to as a target image) as an input and attempts to find the best possible set of parameters mentioned above by using Analysis-by-Synthesis paradigm\cite{Schoenborn2014,10.1007/978-3-642-40602-7_11}. The key idea of the Analysis-by-Synthesis is to generate a synthetic face image which is close to an observed target image. By doing so, the fitting process fits 3DMM face model to the face appearing in the image such that, the fitted model instance (hereafter referred to as the best fit or simply fit) represents the target face as accurately as possible. Thus, the result of the face fitting pipeline is a synthetic face matching to the target face in the image and replicating all the important characteristics of it.\bigskip + +The complexity of inferring a face appearance from a single 2D image using the Analysis-by-Synthesis approach was investigated by Sch{\"o}nborn et al. \cite{Schoenborn2014, Schoenborn2017, 10.1007/978-3-642-40602-7_11}. Considering the fact that the project was solely relying on a single 2D image that only holds information about the color intensities, quite good results were achieved. However, since the amount of information that can be utilized from a 2D image is limited, the authors proposed an idea of improving the standard 2D face fitting pipeline by combining the RGB data with the depth information obtained from a dedicated depth camera. Betschard et al. \cite{betschard2016} investigated this idea by using Intel® RealSense™ F200 depth camera and demonstrated that it is possible to improve the reconstruction quality of the initial method by up to 35\%. The authors of the latter research also proposed an idea of further improving the pipeline by relying on an additional point cloud information, which we thoroughly investigate in this thesis, and at the same time attempt to improve the efficiency and robustness of the fitting pipeline as a whole.\bigskip + +The main difference between a 3D depth camera and a normal 2D image capturing camera is that a depth camera instead of taking just a 2D RGB-color image also has an ability to produce a point cloud of the scene. A point cloud is a set of data points produced by the camera based on the combination of information obtained from the depth sensors and cameras' intrinsic parameters. Each point is located in the 3D coordinate system representing a direct mapping of pixels in the color image. This means that the number of points in a point cloud object directly correlates to the number of pixels in the corresponding color image. Therefore, each point in a point cloud represents an $(x, y, z)$ tuple (coordinates) with the coordinate $(0, 0, 0)$ referring to the center of the physical camera sensor\cite{rs-projection}, the $z$ axes in this tuple is called depth (or $z$-Buffer). It corresponds to a distance, measured from the camera to each corresponding pixel in the 2D color image. It is clear that having $(x, y, z)$ coordinates will represent the pixel location in 3D coordinate space more accurately than having just a $z$-Buffer which was used by \cite{betschard2016}; and obviously, this would give us significantly more information (especially, regarding the shape information of the face), than standard fitting can extract from a 2D image. Therefore, the performance, efficiency, and quality of majority applications, that only make use of the information extracted from a single 2D image could be potentially improved by employing a 3D depth information obtained from the depth camera. \bigskip + +\begin{figure} + \centering + \includegraphics[width=\textwidth]{Figures/stereo_DT_d415_front-crop1a-1.png} + \caption{Intel® RealSense™ D415 camera equipped with an array of sensors (left to right) depth sensor 1, infrared (IR) projector, depth sensor 2, color sensor. Source: Intel} + \label{f1.2} +\end{figure} + +We will be leveraging the latest generation Intel® RealSense™ D400 series depth camera shown in Figure \ref{f1.2} to obtain all necessary data for the fitting pipeline - including an RGB color image, a mesh constructed from point cloud, and a set of facial landmarks. The camera retails under 200CHF and it is one of the best affordable 3D depth cameras currently available on the market. Designed specificity for 3D scanning, portable D415 offers a good accuracy (Z-Accuracy $\leq$2\% also known as the absolute error)\footnote{The accuracy depends on the camera calibration procedure.} and dedicated powerful computer vision processor for on-the-fly calculations. + + +\section{Contributions} +In this work, we have made the following contributions. 1) We have implemented a new fitting pipeline, which in addition to color image also integrates a 3D depth information into the fitting process. We propose a new modular approach for solving this problem which utilizes point cloud information that is being captured from the camera. 2) We have implemented a client-server based modular system, which splits camera configuration, data acquisition, shape fitting, and color fitting procedures from each other. Thanks to its modularity, the system introduces additional flexibility and ease of use for any potential future applications. 3) We have successfully integrated our system into already existing Scaismo Face Morpher service (Figure \ref{f1.1}), which provides the possibility to use our modules for demo purposes. 4) We evaluated our proposed implementation based on a dataset that consists of a series of sample shots that we have made using our data capturing procedure, and present results at the end of the thesis. +% We propose a new modular approach for solving the problem of human face reconstruction based on a single image with additional point cloud information obtained from the depth camera. Splitting data acquisition, shape fitting, and color fitting procedures from each other our system introduces additional flexibility and ease of use for any potential future projects. Besides the complete client-server based system, our main contributions include the data acquisition and the shape fitting procedures. + +\section{Thesis Structure} +We structure this thesis as follows. The details of all the above mentioned procedures combined with background material necessary to follow the flow of events and concepts we employ throughout this thesis will be elaborated in Chapter \ref{ch2}. We introduce the model that we will be using for our application, as well as, describe the idea and implementation of a novel fitting pipeline. We also give a quick overview of depth camera technologies and how they work. Including how pixel-to-point and point-to-pixel de-projection is performed based on a pinhole camera model. We then proceed with describing the methods (Chapter \ref{ch3}) we used to implement a reliable system. Chapter starts with describing our module based system, including the description of each of its key modules and components. We first introduce our camera of choice, its calibration procedure, and parameter configuration together with a software development kit (SDK) that allows us to seamlessly communicate with the camera and utilize its depth sensing capabilities. We further introduce a client-server based architecture that successfully delivers the data from the camera to the Scala client. The data client receives from the server is then being used to perform our proposed shape fitting pipeline and later slightly modified color fitting pipeline. Chapter \ref{ch4} contains the results and the evaluation of our method and its performance comparison to a novel fitting method. We also provide intermediate comparison results between the shape and color fitting modules to ensure that the final output is within the limits of our desired result and it actually improves previous achievements. An example result of our proposed method can be seen in Figure \ref{f1.3}. + +\begin{figure} + \centering + \begin{minipage}{0.49\textwidth} + \includegraphics[width=0.999\textwidth]{Figures/dataset/target/1.png} + \end{minipage} + \begin{minipage}{0.49\textwidth} + \includegraphics[width=0.999\textwidth]{Figures/dataset/our/1blended.png} + \end{minipage} + \caption{Example fitting result with target image (left) and synthetic fit produced by our method (right).} + \label{f1.3} +\end{figure} + +In Chapter \ref{ch5} we summarize our work and provide conclusions, followed by Chapter \ref{ch6} where we propose a couple of potential improvements that can be done to further improve the stability and performance of our system. + diff --git a/Chapters/Chapter2.tex b/Chapters/Chapter2.tex new file mode 100644 index 0000000..7960dc6 --- /dev/null +++ b/Chapters/Chapter2.tex @@ -0,0 +1,235 @@ +\chapter{Background}\label{ch2} + +In this chapter, the foundations of the thesis are elaborated. We formally define methods, concepts and tools used. Starting with the description of the 3D Morphable Model (3DMM) \cite{Blanz:1999:MMS:311535.311556, Romdhani3DM} that we utilize, followed by an explanation of a novel fully probabilistic method +to interpret a single face image with the 3DMM\cite{Schoenborn2017}. As well as, the important characteristics of the depth cameras. + +\section{3D Morphable Model} +The 3DMM is a fully parametric generative face model constructed from 200 high quality face scans. The model contains probabilistic PCA (PPCA) models for shape, color and expression \cite{8373814, EGGER2017115}. In a traditional PCA-based settings the shape S, color C and facial expression E models are constructed through the parameter set $\theta=\{\theta_S, \theta_C, \theta_E\}$ as follows: +\begin{equation} + \begin{split} + S(\theta) &= \mu_S + U_S D_S \theta_S + \mu_E + U_E D_E \theta_E \\ + C(\theta) &= \mu_C + U_C D_C + \end{split} +\end{equation} +Where $\mu$ denotes the mean of the corresponding model, $U$ is a matrix that contains the principal components, and $D$ denotes a diagonal matrix consisting of the variances along the principal directions. The expression model here, is modeled as a deformation of the neutral (mean) face shape. In a probabilistic setting the shape and color models are transformed into a distribution of shape $P(S \mid \theta)$ and color $P(C \mid \theta)$ components with the additive Gaussian noise\cite{ALBRECHT2013959, EGGER2017115}: + +\begin{equation} + \begin{split} + P(S \mid \theta) &= \mathcal{N}(S \mid \mu_S + U_S D_S \theta_S + \mu_E + U_E D_E \theta_E, \sigma_S^2 I) \\ + P(C \mid \theta) &= \mathcal{N}(C \mid \mu_C + U_C D_C, \sigma_C^2 I) + \end{split} +\end{equation} + +The parameter set $\theta$ follow a standard normal distribution in latent vector space\cite{EGGER2017115} which is also defined outside the linear span of 200 scans: + +\begin{equation} + P(\theta) = \mathcal{N}(\theta \mid 0, I) +\end{equation} +To generate realistic 3DMM instances (Figure \ref{f2.1}) and render synthetic face images $\mathcal{I}$, parameter set $\theta$ contains an additional camera $\theta_P$ and illumination $\theta_L$ parameters that are modeled separately from the other three. + +\begin{figure} + \centering + \captionsetup{labelformat=empty} + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/rs1_t.png} + \caption*{Sample} + \end{minipage} + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/mean_t.png} + \caption*{Mean} + \end{minipage} + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/rs2_expr_t.png} + \caption*{Sample with expression} + \end{minipage} + \captionsetup{labelformat=default} + \caption{3DMM face model instances.} + \label{f2.1} +\end{figure} + +This allows us to synthesize the real world appearance of the face. By tuning all the mentioned parameters, we can approximate the appearance of the synthetic face to the target face. Besides parameter space, the model is also able to take care of the image rendering process $\Re$ which renders an image $\mathcal{I}(\theta)$ based on given parameters through +\begin{equation} + \mathcal{I}(\theta) = \Re(\mathcal{M}(\theta_S, \theta_C, \theta_E); \theta_P, \theta_L). +\end{equation} +The 3DMM model also has a set of modified model versions that are either restricted to a certain region or are down-scaled, low-resolution versions\footnote{Down-scaled versions usually have a lesser number of points and triangles hence are less flexible, but they perform the mesh operations faster} of the original model. The model restricted to a certain region is important for domain specific applications that only utilize specific parts of the model. For example, we use the model (Figure \ref{f2.2}) from \cite{Schoenborn2017} which only covers the face region and discards other parts like ears, neck, and part of the forehead. + +\begin{figure} + \centering + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/sideA_t.png} + \end{minipage} + \begin{minipage}{.33\textwidth} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/face_bfm_close_t.png} + \end{minipage} + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/sideB_t.png} + \end{minipage} + \caption{Restricted face model displayed over the full model(dark)} + \label{f2.2} +\end{figure} + +Beneficially to our application, this restriction also ignores parts of the model that are too complex, noisy or low quality, for example ears, that could potentially cause some problems during the fitting. +\section{Fitting Pipeline} +Having the flexible Parametric Appearance Model (PAM) as a basis of the fitting framework allows us to model a variety of real world scenes with challenging illumination conditions. In this section, we explain how standard fitting pipeline and its augmented variant utilizes Markov Chain Monte Carlo sampling to perform approximate inference.\bigskip + +\subsection{Markov Chain Monte Carlo (MCMC)}\label{s2.2.1} +To generate parametric face instances, the Analysis-by-Synthesis method employs a probabilistic image analysis using Bayesian inference. Formally, the fitting pipeline is a procedure that for any given face image returns a model fit which best corresponds to the face in the input image. It is important to note that commonly applications aim to obtain a single solution as a result, however, this is not the case for fitting pipeline. Instead of aiming for a single solution the fitting pipeline produces a posterior distribution of possible solutions. Since the true posterior distribution is unknown, the fitting process needs to approximate it. To perform approximate inference in this setting the Metropolis-Hastings (MH) algorithm is used. The algorithm is a method of Markov Chain Monte Carlo sampling and it is especially useful when sampling from distributions from which direct sampling is not feasible. The algorithm is an iterative process that draws random samples $\theta'$ from a proposal distribution $Q(\theta'\mid\theta)$ that are being evaluated based on their likelihood value. In the figure bellow single step of MH algorithm is shown.\bigskip + +\begin{algorithm}[H] + \SetAlgoLined + $\bullet$ Initialize sample $\theta \sim Q(\theta)$\; + $\bullet$ Generate the next sample with current sample $\theta$:\\ + {\addtolength{\leftskip}{5mm} + 1. Propose a sample $\theta'$ based on current sample: $\theta'\sim Q(\theta'\mid\theta)$\; + 2. With probability: $\alpha = min\Big\lbrace\frac{P(\theta')}{P(\theta)}\frac{Q(\theta\mid\theta')}{Q(\theta'\mid\theta)}, 1\Big\rbrace$ accept or reject samples\; + + 3. If sample $\theta'$ is accepted set $\theta'$ to be a new state $\theta$, otherwise keep $\theta$ unchanged\; + } + \caption{Metropolis-Hastings algorithm} + \label{a1} +\end{algorithm}\bigskip + +The two principal parts of the algorithm are the proposal distribution $Q$ from which samples are drawn and the likelihood estimation $P(\theta)$. Let us describe both of these key components in detail. \bigskip + +The proposal distribution should be simple enough to draw random samples from it. In the settings of the fitting pipeline commonly used proposal distribution is a simple Gaussian random walk proposal distribution: + +\begin{equation} + Q(\theta'\mid\theta) = \mathcal{N}(\theta'\mid\theta, \sigma^2 I_d). + \label{2.5} +\end{equation} + +Where $d$ stands for the dimension and $\sigma$ controls the step size the algorithm makes at each sampling iteration. In the fitting pipeline we usually have more than one proposal distributions for each 3DMM parameter update $\theta = \{\theta_S, \theta_C, \theta_E\, \theta_P, \theta_L\}$. Therefore, in practice, $Q(\theta'\mid\theta)$ consists of multiple proposal distributions $Q_i$ combined into one mixture proposal distribution: + +\begin{equation} + Q(\theta' | \theta) = \sum_i c_i Q_i(\theta'|\theta), \sum_i c_i = 1. + \label{eq2.3} +\end{equation} + +Where $\theta$ is a current sample, $\theta'$ is a newly proposed sample drawn from a proposal distribution $Q_i$, and $c_i$ coefficient controls how often samples are being drawn from the specific $Q_i$ distribution\cite{Schoenborn2017}.\bigskip + +The second key part of the MH algorithm is a likelihood estimation term $P(\theta)$ commonly written as $P(\theta\mid\mathcal{I})$. It determines a posterior belief of the current sample parameter set $\theta$ conditioned on the target image $\mathcal{I}$. Computing the posterior directly is not possible, therefore, we employ the classical Bayes' theorem\footnote{Bayes' theorem — \url{https://en.wikipedia.org/wiki/Bayes'\_theorem}} to approximate it based on the prior knowledge $P(\theta)$ and an image likelihood $P(\mathcal{I}\mid\theta)$: + +\begin{equation} + P(\theta \mid \mathcal{I}) = \frac{P(\theta)P(\mathcal{I}\mid \theta)}{\int P(\mathcal{I}\mid \theta)P(\theta)d\theta} +\end{equation} + +In this setting finding a single solution then becomes a problem of a maximum-a-posteriori (MAP) inference, which is finding the parameters with the highest posterior probability\cite{10.1007/978-3-642-40602-7_11}. Since we are only interested in the ratio of two different likelihoods $P(\theta\mid\mathcal{I})$ and $P(\theta'\mid\mathcal{I})$ the normalization term can be ignored, and we can rewrite the above formulation as: + +\begin{equation} + P(\theta | \mathcal{I}) \propto P(\theta)P(\mathcal{I} | \theta) + \label{eq2.5} +\end{equation} + +The prior probability $P(\theta)$ of the model estimate $\theta$ is usually defined as a normal distribution: +\begin{equation} + \label{eq3} + P(\theta) = \mathcal N(\theta | 0, \mathcal I). +\end{equation} + +The only missing part of the Equation \ref{eq2.5} than is an image likelihood $P(\mathcal{I}\mid\theta)$ which can be reformulated as $P(\mathcal{I}\mid\theta) = \mathcal{L}(\theta;\mathcal{I})$ with equation \ref{eq2.5} transforming to: + +\begin{equation} + \label{eq2.5} + P(\theta | \tilde{\mathcal I}) \propto \mathcal L(\theta;\tilde{\mathcal I})P(\theta) +\end{equation} +An image $\mathcal{I}$ that appears in likelihood term means that during the fitting target image is treated as an observation of the 3DMM model instance. +Depending on the phase the fitting pipeline is in, a combination of likelihood functions is used. We briefly discuss the most important likelihood functions used in the fitting pipeline alongside with the phase they are used in. + +\subsection{Landmark Fitting} +During the starting phase of the fitting, it is important to have good pose estimation parameters since the pipeline is crucially dependent on it. If the pose is wrong, the rest of the model parameters cannot be inferred reliably. To deal with this problem, the standard fitting pipeline relies on the landmark fitting phase. During this phase fitting only modifies 3DMMs $\theta_P$ and $\theta_S$ parameters. It does so by evaluating a set of landmark point locations $x_i$ detected (by detection algorithm or manually) onto the target image and projected to the model instance using computer graphics. The projection method uses a pinhole camera model to align model instance to the target face by applying a rotation and translation parameters and then render individual landmarks onto the image plane. By treating those points as an observation of corresponding model points, the algorithm evaluates them using isotropic Gaussian likelihood function: + +\begin{equation} +\mathcal L(\theta; x_1, x_2,\dots, x_N) = \prod_{i=1}^{N} \mathcal N(x_i\mid y_i(\theta), \sigma^2_{LM} I_2) \text{\cite{Schoenborn2017}}. +\label{lmeval} +\end{equation} + +Where $x_i$ are observed landmark positions and $y_i(\theta)$ are model landmark positions based on $\theta$. Proposed samples with new camera and shape parameters are getting accepted or rejected based on its likelihood value according to the procedure discussed earlier in Algorithm \ref{a1}. Pose parameters that are part of $\theta_P$ are a combination of Euler rotation angles \textit{\{yaw, nick, roll\}}, translation vector $\vec{t} = \{t_x, t_y, t_z\}$ with $t_z$ indicating the distance from the camera, and scaling parameter to control the focal length of the camera. The proposal distribution (Equation +\ref{eq2.3}) for landmark fitting consists of a mixture distribution over shape and camera (pose) parameter update proposals. As discussed previously, shape parameters $\theta_S$ are PCA coefficients of the low-rank expansion of the Gaussian Process\cite{8010438} model that are proposed by a weak isotropic Gaussian perturbation proposal\cite{Schoenborn2017}. The described process adjusts the pose and at the same time makes initial shape correction. Landmark fitting is usually a very fast process since the only few landmark points (usually less than 10) are getting evaluated at a time. + +\subsection{Color Fitting} + +After the landmark fitting phase the pose of the model instance usually fits well with the target face in the image and the model instance is ready to be utilized for color, illumination and optionally expression proposals as well as shape update proposals. Thus, during the color fitting phase the mixture proposal $Q(\theta'\mid\theta)$ will be a combination of all the above mentioned parameter proposals with various step size and drawing probability as of Equation \ref{eq2.3}. To evaluate proposed samples in this phase the standard fitting pipeline together with landmark evaluation (Equation \ref{lmeval}) relies on independent pixel evaluator likelihood which distinguishes foreground and background pixels\cite{Schonborn:2015:BMG:2798342.2798359} and is formulated as follows: + +\begin{equation} + \mathcal L \left (\theta; \tilde{\mathcal I} \right ) += \prod_{i \in \mathcal F} \mathcal N \left( \tilde{\mathcal I}_i \,\middle |\, \mathcal I_i(\theta), \sigma^2 I_3 \right ) \prod_{i \in \mathcal B} \mathcal L_{\text{BG}}\left ( \tilde{\mathcal I}_i \right ). +\label{eq2.7} +\end{equation} + +Where the first product evaluates foreground pixels $i\in\mathcal F$ and second product evaluates background pixels $i\in\mathcal B$. The motivation behind incorporating background pixels into this likelihood and not ignoring them is that, when they are ignored, their likelihood value is assumed to be 1, which is not necessarily true in most cases. \bigskip + +So called propose-and-verify flow of the initial architecture of the Metropolis-Hastings algorithm used by \cite{Schoenborn2017, Schoenborn2014} is shown in Figure \ref{f2.3}. As we have mentioned previously, the architecture only uses color image and 2D landmarks as an input. + +\begin{figure} + \centering + \includegraphics[width=0.85\textwidth]{Figures/flow1.PNG} + \caption{The architecture of the MH algorithm used by \cite{Schoenborn2017} taken from \cite{betschard2016}}. + \label{f2.3} +\end{figure} + +An augmented version of this flow by \cite{betschard2016} can be seen in Figure \ref{f2.4}, where authors introduced additional depth image and 3D landmark input sources, with respective evaluators. \bigskip + +\begin{figure} + \centering + \includegraphics[width=0.85\textwidth]{Figures/flow2.PNG} + \caption{Augmented MH architecture proposed by \cite{betschard2016}, changes made in the architecture are in bold.} + \label{f2.4} +\end{figure} + +We introduce an alternative way of dealing with this architecture. Instead of using just a depth image (only z-Buffer) we construct a triangle mesh from a point cloud obtained using the depth camera. Details of this process are described in Section \ref{s3.3.3}. We were also forced to find a work-around way to obtain 3D landmarks (Section \ref{s3.3.2}), since they are not provided by the camera SDK anymore. We split the standard fitting pipeline into two sub-fitting pipelines, one that deals with the shape and pose parameter estimation hereafter referred to as \textit{shape fitting module} and the other with color, illumination, and expression with slight shape and pose parameter updates hereafter referred as \textit{color fitting module}. Both of these modules are described in Section \ref{s3.4}. + +\section{Depth Camera} + +Inferring the exact size of an object when analyzing color images is a challenging task, there is no effective way to reliably recover this information based on color intensities. The depth camera helps us to resolve this issue by providing a distance measure for each pixel (if it is available). There are a few different ways distance is obtained by the depth cameras. One of the algorithms commonly used (and the one our camera uses) to calculate distance is the depth from stereo algorithm better known as \textbf{Stereoscopic Vision}\footnote{\url{https://en.wikipedia.org/wiki/Computer_stereo_vision}} which is a realization of a natural Binocular vision. The basic idea behind Stereoscopic vision is to estimate the distance from the camera to each point by calculating disparities between two parallel view-ports\cite{serg, DBLP:journals/corr/KeselmanWGB17} (Figure \ref{f2.5}). View-port parallelism is achieved with the traditional Image Rectification\footnote{\url{https://en.wikipedia.org/wiki/Image_rectification}} approach, which transforms images onto a common virtual image plane and makes sure that two view-port points are matching. + +\begin{figure} + \centering + \includegraphics[width=0.55\textwidth]{Figures/Pictures/stereo-ssd-1.png} + \caption{Depth from stereo. Visualizing two separate view-ports that are being used to calculate disparities. Source: Intel} + \label{f2.5} +\end{figure} + +\subsection{De-projection} +Pixel-to-Point and Point-to-Pixel de-projection is a common use-case in computer vision. When needed this feature offers pixel to point mapping and vice-versa by relying on cameras' depth information and intrinsic parameters. Successful de-projection is based on the traditional pinhole camera model \cite{Hartley:2003:MVG:861369} shown in Figure \ref{f2.6}. + +\begin{figure}[h] + \centering + \includegraphics[width=\textwidth]{Figures/pcm.png} + \caption{Pinhole camera model. $\mathcal F_c$ is the center of the camera, (X, Y, Z) are coordinates of a 3D point in the world coordinate system, ($u$,$v$) are coordinates of a respective point projection in pixels, ($c_x, c_y$) indicates principal points usually located at the image center. Source: \url{https://docs.opencv.org}} + \label{f2.6} +\end{figure} + + +The problem of point to pixel projection is formulated as follows. Given a 3D point coordinates $P_{3D}(X, Y, Z)$ with camera intrinsic parameters\footnote{\url{https://github.com/IntelRealSense/librealsense/blob/master/include/librealsense2/h/rs\_types.h\#L55}} containing ($width$, $height$, $ppx$, $ppy$, $f_x$, $f_y$), calculate the respective pixel coordinates $P(u, v)$ with no distortion introduced. Where in camera parameters, $width$ and $height$ are image dimensions in pixels, $ppx$ and $ppy$ are the horizontal and vertical coordinate of the principal point of the image given as an offset from the left and top edge respectively, and the $f_x$ and $f_y$ are the focal lengths of the image as a multiple of pixel width and height. The focal length is usually the distance from the center of the camera to the image plane also known as the focal plane. It is shown as gray square in Figure \ref{f2.6}. Once we have all these variables, then the $u$ and $v$ pixel coordinates of $P(u, v)$ are calculated as follows: + +\begin{equation} + \begin{split} + &x' = \frac{X}{Z}\\ + &y' = \frac{Y}{Z}\\ + &u = x' \cdot f_x + ppx\\ + &v = y' \cdot f_y + ppy\\ + \end{split} +\end{equation} + +On the other hand, the problem of de-projecting pixel coordinates into the 3D coordinate space to obtain a 3D point is formulated as follows. Given a pixel coordinates $P(u, v)$ and depth information $d = Z$ alongside the camera parameter set mentioned previously with no distortion introduced, compute the corresponding $P_{3D}(X, Y, Z)$ point in the 3D space. Then $P_{3D}(X, Y, Z)$ coordinates are being calculated as follows: + +\begin{equation} + \begin{split} + &x' = \frac{u - ppx}{fx}\\ + &y' = \frac{v - ppy}{fy}\\ + &X = d \cdot x'\\ + &Y = d \cdot y'\\ + &Z = d + \end{split} +\end{equation} + +Both techniques we have discussed are already implemented in the camera SDK for us, therefore, throughout the project, we make use of those default methods provided by the SDK. + + + diff --git a/Chapters/Chapter3.tex b/Chapters/Chapter3.tex new file mode 100644 index 0000000..03c0dfc --- /dev/null +++ b/Chapters/Chapter3.tex @@ -0,0 +1,462 @@ +\chapter{Methods}\label{ch3} + +In this chapter, our main contributions are elaborated. We explain in detail our proposed method of the fitting pipeline, including its key components that we have put together to build the module based coherent and reliable system. The overview diagram of the system we have implemented can be seen in Figure \ref{flow}. The system makes sure that the data from the camera is seamlessly delivered to the fitting pipeline in the correct format and being utilized efficiently. Further, in this chapter, we describe each of the key components of the system that could be split into three distinct modules: +\begin{enumerate}[$\bullet$] + \item Module 1: takes care of the camera configuration, data capturing and preprocessing procedure (Figure \ref{flow} \textbf{Server} block). Among others, the data capturing procedure includes the acquisition of 3D landmarks, which introduces an additional technical challenge since they are not directly obtainable from the camera. + \item Module 2: receives the data from the server, processes it, and performs the shape fitting (Figure \ref{flow} \textbf{Client \& Shape Fitting} blocks\footnote{In practice, these two modules are merged into one, we split them here to make procedures clear.}). Besides the more important shape fitting procedure, this module also takes care of minor but nevertheless important aspects of the module, such as data conversion and cleaning. + \item Module 3: receives the best shape fit produced by the shape fitting module, alongside other necessary data, and performs the color fitting (Figure \ref{flow} \textbf{Color Fitting} block). The color fitting performed in this module is very much like the standard fitting approach, except small modifications as we discuss them in the following chapters. +\end{enumerate} + +In this way, it is very easy to control and use those modules for specific applications, like only producing the data and using it for other purposes, or only obtaining the shape fit where the color fit is not required. \bigskip + +We first explain in detail the camera calibration and configuration procedure. Then we introduce the software development kit that allows us to communicate with the camera and perform the data acquisition procedure. Further, the client-server API is introduced. Based on this API data is seamlessly being delivered to the module that utilizes it. We also describe the data capturing procedure alongside the client-server interaction. Finally, and most importantly both of our fitting modules are elaborated, including the description of the proposals and evaluators used. + + +\begin{figure} + \centering + \includegraphics[width=\textwidth]{Figures/FittingFlowNew.pdf} + \caption{An overview of our system with its key modules and procedures.} + \label{flow} +\end{figure} + +\section{Camera} +Before we start working on our project is it essential to configure and calibrate the camera in order to maximize its performance. As we have mentioned previously, our camera of choice is the Intel® RealSense™ D415 depth camera. Thanks to its rolling shutter and narrow field of view, the camera is focused on accuracy and is especially well suited for near and still object scanning applications. + +\subsection{Camera Calibration} +To calibrate the camera we use Intel's Dynamic Calibrator tool. The camera can be calibrated by either displaying a calibration texture pattern on a mobile device screen or printing it on a white paper and following the Calibrator tool instructions (Figure \ref{f3.1}). Dynamic calibration tool optimizes the cameras' extrinsic parameters with regard to the main axis, consisting of rotation and translation parameters\cite{cal}. + +\begin{figure} + \begin{minipage}{.49\textwidth} + \includegraphics[width=0.99\textwidth]{Figures/Pictures/cal1.png} + \end{minipage} + \begin{minipage}{.49\textwidth} + \includegraphics[width=0.99\textwidth]{Figures/Pictures/cal2.png} + \end{minipage} + \caption{Left: Rectification phase, Right: Scale phase. Source: Intel} + \label{f3.1} +\end{figure} + +To evaluate the performance of our camera after the calibration procedure we use Depth Quality tool\footnote{\text{Depth Quality Tool} — \url{https://github.com/IntelRealSense/librealsense/tree/master/tools/depth-quality}} (Figure \ref{f3.2}). + +\begin{figure} + \begin{minipage}{.49\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/dc.png} + \end{minipage} + \begin{minipage}{.49\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/ic.png} + \end{minipage} + \caption{Depth quality tool using Intel's recommended textured wall test method to assess performance, displaying depth (left) and infrared (right) stream view, alongside with an ROI area marked with a yellow rectangle.} + \label{f3.2} +\end{figure} + +By running a series of recommended tests we obtain a set of important quality measurements presented in Table \ref{t3.1}. \textbf{Fill rate} measures the percentage of pixels with valid depth value within the Region-Of-Interest (ROI). \textbf{Z-accuracy} indicates the depth accuracy given the Ground Truth (GT) as a percentage of the range, where positive value means that the depth plane fit is behind the ground truth (overshot), and a negative value indicates that the depth plane fit is in front of ground truth (undershot). \textbf{Plane Fit RMSE} calculates the Root-Mean-Squared (RMS) error of Z-error (Spatial noise). \textbf{Sub-pixel RMSE} provides sub-pixel RMS error.\footnote{This information is taken from the depth quality tool itself, refer to footnote link above for more details.} + + +\begin{table}[h] + \centering + \begin{tabular}{c|c|c|c} + Measurement & Result & ROI & Resolution\\ + \hline + Fill Rate & 100\% &40\% & 1280x720 \\ \hline + Z-Accuracy & 0.24\% &40\% & 1280x720 \\ \hline + Plane Fit RMSE & 0.24\% & 40\% & 1280x720 \\ \hline + Sub-pixel RMSE & 0.13\% & 40\% & 1280x720 \\ + \hline + \end{tabular} + \caption{Calibration test results.} + \label{t3.1} +\end{table} + +\subsection{Camera Configuration Parameters} +After the camera calibration, we proceed with configuring the camera parameters including resolution, stream profiles (Color and Depth stream separately), measurement distance, filters and a long set of other depth configuration parameters. If not stated otherwise we do not modify any sensitive parameter and use Intel's recommended settings as of \cite{bestcal}. Although Intel recommends to use 1280x720 image resolution for an optimal image quality and depth accuracy, we have decided to use 640x480 resolution to right away reduce the number of calculations needed to do any sort pixel-wise or point-wise operations and yet maintain a decent image quality. 1280x720 resolution yields ~900k pixels/points as opposed to ~300k in the case of 640x480. When it comes to camera stream formats we use traditional \textit{z16} format for the depth stream and \textit{rgb8} format for the color stream. We enable auto exposure setting which allows the camera to dynamically adjust exposure for the various light and illumination environments. Since our application only requires a still relatively close-up distance head shots we set the minimum depth distance to 0.2 meter and maximum distance to 1 meter as well as the ROI parameter to 40\%. This is because we are not interested in the entire scene captured by the camera but rather the face area. As of filters, we did not make use of them because we opted to work with raw data without various filters influencing its quality. The final note we would like to make regarding the camera configuration is that we align color and depth stream frames. The reason is straightforward, we would like to have the data that have the same view-point — color and depth frames should be in correspondence. If we do not align them, they will be viewed from slightly different angles, this is due to spatial differences between physical camera sensors. This process will become more obvious in Section \ref{s3.3.2} were we describe the process of capturing 3D landmarks. + +\subsection{SDK}\label{s3.1.3} +To take advantage of the camera hardware and to fit the capture procedure to our needs we have to use Intel's \textit{librealsense}\footnote{Intel® RealSense™ SDK — \url{https://github.com/IntelRealSense/librealsense}} SDK. The reason why it is necessary to communicate with the camera through the API instead of using Intel's RealSense Viewer (Figure \ref{f3r}) is that firstly, it does not provide all the required data out-of-the-box and secondly, the data cannot be captured all at once. As we are designing our system to work in a variety of application scenarios including a web environment, it is crucial to automate all parts and run the system with minimal to no human interaction. \bigskip + +\begin{figure}[ht!] + \centering + \includegraphics[width=0.98\textwidth]{Figures/Pictures/rsviewer.png} + \caption{Intel RealSense Viewer — 3D Depth View with camera configuration parameters on the left side. Blue color indicates near points, red — far.} + \label{f3r} +\end{figure} +While being written in C++ the SDK provides wrappers for the majority of the popular programming languages\footnote{Not all wrappers support full features offered by the native SDK}. Unfortunately, Java Virtual Machine (JVM) based languages (one of which, namely Scala we are going to use in the main part of the project) are not officially supported. Therefore, we had to choose an alternative lightweight and fast language, flexible enough to seamlessly run on any machine and operating system. We have decided to use Python because of its powerful companion frameworks, ease of use and also because of \textit{pyrealsense2} — librealsense's Python wrapper that is stable and relatively fully featured comparing to others. We hereby note that upon investigating the camera SDK we have discovered\footnote{GitHub Issue — \url{https://github.com/IntelRealSense/librealsense/issues/3716}} that it does not have a dedicated computer vision middleware module it used to have in the previous generation \cite{betschard2016} used. The module used to offer relevant features for our application such as face scanning and landmark detection. Not having a direct, native access to these features introduces additional technical challenge. Since the performance of or application crucially depends on a set of facial landmarks we had to find a way to detect them. We present our solution to this problem in Section \ref{s3.3} where we describe a complete data acquisition procedure. The one last important notice we would like to make regarding the choice we have made in this section is that obviously, we cannot simply communicate between Python and Scala code base. To deal with cross-language modules we rely on interface definition language framework — Apache Thrift\footnote{Apache Thrift™ — \url{https://thrift.apache.org}}. + +\section{Cross-Language API} +\begin{figure} + \centering + \includegraphics[width=\textwidth]{Figures/thrift/Rs-ex.pdf} + \caption{Diagram of our API. *For better resolution please refer to Appendix \ref{fA.1}} + \label{f3.3} +\end{figure} + +Thrift is an interface definition language designed to solve the problem of sharing the code base of cross-language modules. Specifying the signature of all the important types, methods and exceptions in the thrift interface — the framework generates an identical API for any major programming languages. By doing so, Thrift takes care of all the language specific details such as syntax, type variations, exceptions, etc. and makes sure that the code generated for the different languages are usability-wise identical from an API consumer's perspective. The diagram of our thrift interface can be seen in Figure \ref{f3.3}. We define a set of relevant data structures such as \textit{Pixel, Point, Vector, Image, Landmark, Triangle} and \textit{Color} with respective lists that utilize some of those values \textit{PointList, LandmarkList, ColorList}, and \textit{TriangleList}. We also define a mesh structure with signature \textit{TriangleMesh(PointList, ColorList, TriangleList)}, as well as, the main data holder object \textit{CaptureResult(Image, Landmark2D, Landmark3D, TriangleMesh)}. \textit{CaptureResult} is the object that is being passed from the server-side to the client-side. Every mentioned data type has its respective converter method implemented in both Python and Scala code base. We did not make use of 2D landmarks in this project, so we do not pay attention to it any further, however, if needed — its functionality is fully implemented in both Python and Scala code. + + +\section{Capture Procedure (Server)}\label{s3.3} + +\begin{figure} + \centering + \includegraphics[width=0.5\textwidth]{Figures/Server.pdf} + \caption{The server diagram.} + \label{server} +\end{figure} + +Upon receiving a request our server (Figure \ref{server}) launches a pre-configured camera stream into the OpenCV\footnote{Open Source Computer Vision Library — \url{https://opencv.org/}} window (Figure \ref{f3.5}) that is split into two sub-windows, one displaying an RGB color stream and the other depth stream. Red color indicates pixels that are closer to the camera and blue that are farther. Note that the server does not execute capturing once it receives the request, instead it runs the camera stream and waits for additional user input (for example pressing the “D” key on the keyboard). This is because otherwise, the user will not be able to adjust their position. Obviously, it is possible to add timeout before capturing but having a visual tool is still preferred. Once the server receives a call to run the capturing script, it captures the required raw data and starts processing it. For the successful pipeline run, as we have mentioned previously, our modified fitting pipeline requires the following data: \textbf{RGB Color Image}, \textbf{A Set of 3D Landmarks} and \textbf{Point Cloud Mesh}. + +\begin{figure} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/no-glasses.png} + \caption{The OpenCV window displaying an RGB color image (left) and the depth image (right) visualized using JET color map. } + \label{f3.5} +\end{figure} + +\subsection{RGB Color Image} +Capturing a color image is not complicated, and we do not do any post-processing to it. After converting it to the Thrift image type using the corresponding converter method we directly pass it to the \textit{CaptureResult} object described above. + +\subsection{3D Landmarks}\label{s3.3.2} +Obtaining 3D landmarks is one of the important data requirements for this project. The problem arose due to the lack of 3D landmark availability in the SDK. The solution to this problem we have implemented is to detect 2D (pixel) landmarks on the color image using Dlib\footnote{Dlib C++ library — \url{http://Dlib.net}} landmark detection library (Figure \ref{f3.6}). These landmarks are then de-projected into 3D landmark points using the camera SDKs de-projection capabilities we discussed previously. The de-projection is based on camera settings for the specific session (shot). + +\begin{figure}[h] + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/lmoverlay.png} + \caption{Facial landmark detection using OpenCV and Dlib libraries visualized over D415's color stream.} + \label{f3.6} +\end{figure} + +Dlib landmarks usually have corresponding landmark ID assigned to it, based on those IDs we match each 2D landmark to the fitting pipeline landmarks by the name. This way, after performing de-projection we create the landmark structure that corresponds to the structure defined in the fitting pipeline \textit{\{"landmarkID": [$LM_x, LM_y, LM_z$]\}}. Sometimes detected pixel landmarks do not have corresponding 3D point due to missing depth information, in situations like this 3D landmark points will be filled with 0.0 values. We will deal with these landmarks on the client-side where we determine which landmarks to use. During landmark de-projection, it is essential to have color and depth frames in correspondence (aligned). Otherwise, when examining point cloud, landmarks will not appear in locations where they supposed to, because point cloud mesh will be captured based on depth stream and landmarks will be obtained via de-projecting color stream pixels to 3D points. We have already emphasized the importance of 3D landmarks in our application, therefore, we place a hard constraint over this block of information. If Dlib for some reason fails to detect the face in the image, the capturing process throws an exception. Furthermore, when there is more than one face detected in the image, we also break the process, because the pipeline would not be able to figure out which face we are trying to reconstruct. + + +\subsection{Point Cloud}\label{s3.3.3} +The camera SDK does provide raw point cloud access out-of-the-box, however, the operations that are defined on top of point cloud objects are limited to accessing the point information, saving them as a structured mesh file, obtaining texture coordinates and getting the size of a point cloud. Saving the point cloud as a mesh file is the feature we need, however, we have to modify it, because we do not save the mesh file anywhere, instead serialize and sent it through the network. We have implemented an overridden version of the mesh formation in order to satisfy this requirement. + +\begin{figure} + \centering + \begin{minipage}{.5\textwidth} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/pts1.png} + \end{minipage}% + \begin{minipage}{.5\textwidth} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/pts.png} + \end{minipage} + \caption{Obtained point cloud clipped at face region.} +\end{figure} + +The method takes point cloud provided from the SDK and creates triangles between neighboring points. Triangle formation itself is based on the same algorithm Intel uses to save the mesh as a file. That is, we only modify the usability of the method but the algorithm is unchanged. + +\begin{figure} + \centering + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/full_pc.png} + \caption*{(a)} + \end{minipage} + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/full_with_overlay.png} + \caption*{(b)} + \end{minipage} + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/clip.png} + \caption*{(c)} + \end{minipage} + \caption{(a) Full scene point cloud mesh. (b) The same scene as (a) but with region of interest highlighted in green. (c) Clipped face region (target mesh)} + \label{f3.8} +\end{figure} + +The point cloud obtained from the SDK contains points which are describing the entire scene most of which we are not interested in, and more importantly having unnecessary points introduces additional computational overhead during the fitting. Therefore, we consider clipping the point cloud in the face region and discard the rest of the points. This process happens on the client-side which we yet to describe. The basic flow of mesh clipping operation is shown in Figure \ref{f3.8}. We will describe this process further in Section \ref{s3.4.1}. + + +\section{Fitting Pipeline (Client)}\label{s3.4} + +\begin{figure} + \centering + \includegraphics[width=0.5\textwidth]{Figures/Client.pdf} + \caption{The client diagram.} + \label{client} +\end{figure} + +As described previously, the fitting pipeline is split into two parts. The first part (Figure \ref{client}) is responsible for receiving the data from the server, converting it to respective Scalismo\footnote{Scalismo — Scalable Image Analysis and Shape Modelling \url{https://scalismo.org/}} objects (\textit{PixelImage, Landmark3D, TriangleMesh}), performing the necessary data cleansing procedure, and finally starting the shape fitting process. The shape fitting utilizes additional point cloud information by fitting the 3DMM onto the point cloud to obtain better pose and shape model parameters. We describe this process in Section \ref{s3.4.2}. The second part of our proposed fitting pipeline is the color fitting stage. It is a continuation of the shape fitting in the sense that it takes the result of the shape fitting as an input, and runs the slightly modified standard color fitting. This process is described in Section \ref{s3.4.3}. + +\subsection{Data Processing}\label{s3.4.1} +Before shape fitting starts it is necessary to process and clean the data received from the server. The more important data cleansing happening with \textit{Landmark3D} and \textit{TriangleMesh} structures. Before we do that, we augment the current model with additional chin landmarks that we are getting from the Dlib, and remove some which we do not have in Dlib. To make this process better understandable we visualize landmark differences in Figure \ref{f3.9}. Where the figure on the left shows the original landmarks that are available in the model, and the figure on the right shows our version of it. It should be noted that the pose variation in the image tends to affect obtained chin landmarks significantly, and the pixel-to-point de-projection we perform further increases the uncertainty of those landmarks. Therefore, to deal with this problem we assign higher uncertainty to chin landmarks that are being received from the server. + +\begin{figure} + \centering + \begin{minipage}{.49\textwidth} + \centering + \includegraphics[width=0.98\textwidth]{Figures/Pictures/origGreen.png} + \caption*{Original model landmarks} + \end{minipage} + \begin{minipage}{.49\textwidth} + \centering + \includegraphics[width=0.98\textwidth]{Figures/Pictures/pipelineGreen.png} + \caption*{Landmarks we use} + \end{minipage} + \caption{Visualization of the differences between the original model landmarks and the landmarks we use.} + \label{f3.9} +\end{figure} + +Once we have our model ready we discard landmarks that contain 0.0 values, which were produced by de-projection due to unavailable depth information. We also discard landmark points that are being de-projected wrongly due to the occlusion or frame misalignment. This is done by evaluating each landmarks' $z$ coordinate which indicates its distance from the camera and discarding landmarks that are farther than predetermined capture distance. Additionally, we remove model landmarks that are not part of our target landmarks. Based on cleaned up target landmarks we perform mesh clipping procedure. + +\begin{figure} + \centering + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/mesh_r.png} + \end{minipage} + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/mesh_f.png} + \end{minipage} + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/mesh_l.png} + \end{minipage} + \caption{Clipped sample target mesh from different angles.} + \label{f3.10} +\end{figure} + +The basic idea is to create a box region covering the face by determining the outer most landmark positions for each dimension and ignoring all the points that lie outside this region. The result of this procedure can be seen in Figure \ref{f3.10}. The only issue with this approach is that determining outer most landmarks and slicing based on their value will slice up some part of the face if the pose is too extreme. This is because slicing operation is happening parallel to basis axes. Hence, we recommend to work with no more than $\ang{30}$ degree yaw rotations and for optimal performance frontal shots where occlusion and frame misalignment are not affecting to the landmarks. The final result of the mesh slicing and landmark filtering is shown in Figure \ref{f3.11}. + +\begin{figure} + \centering + \includegraphics[width=0.85\textwidth]{Figures/Pictures/targetMeshWithLandmarks_t.png} + \caption{Sample target mesh with filtered landmarks.} + \label{f3.11} +\end{figure} + +Note that there are fewer chin landmarks available on the target mesh due to the landmark filtering process described previously. This can be explained by taking into account what we have described in Section \ref{s3.3.2}. The pixel-to-point de-projection is successful, if and only if, the camera detects the respective depth information for the specific pixel we are trying to de-project. Since chin pixels commonly suffer from occlusion, camera often is not able to detect the depth information. Hence, we have some 3D landmarks that are missing from the target mesh. + +\subsection{Shape Fitting}\label{s3.4.2} + +\begin{figure}[h!] + \centering + \includegraphics[width=0.8\textwidth]{Figures/ShapeFitting.pdf} + \caption{The shape fitting diagram.} + \label{shapeF} +\end{figure} +\FloatBarrier + +Before we start the shape fitting process (Figure \ref{shapeF}) itself, we first align the model and the target mesh rigidly based on landmark correspondences using rigid 3D landmark registration method. The method, which is already implemented in Scalismo, performs generalized Procrustes analysis\cite{Lorusso:1995:CFA:236190.236213} to obtain the translation and rotation parameters, which we then use to give the model a good starting position parameters. This step is required because the model is centered at the origin of the 3D coordinate system, whereas the target mesh location is determined by how far the face was from the camera when the shot was made\footnote{The camera center is located at (0, 0, 0)}. A visual example of this scenario can be seen in Figure \ref{f3.12}. + +\begin{figure} + \centering + \includegraphics[width=0.95\textwidth]{Figures/Pictures/modelTargetDist_t.png} + \caption{Initial location of the model (right) and the target (left).} + \label{f3.12} +\end{figure} + +The process alternatively can be performed during the fitting process itself, but we opted to obtain necessary parameters using the rigid alignment because it is usually very fast and leads the model to a correct initial position. \bigskip + +Shape fitting boils down to optimizing the set of parameters $\theta = \{\theta_S, \theta_R, \theta_T\}$ referring to the model's \textit{shape}, \textit{rotation}, and \textit{translation} parameters respectively. The shape parameter $\theta_S$ is a vector of model coefficients $\theta_S = (\alpha_0,\dots,\alpha_{199})$. The rotation parameter $\theta_R$ consists of Euler rotation angles $\theta_R = (r_\phi, r_\psi, r_\omega)$ and the translation parameter $\theta_T$ is a vector holding 3D translation information $\theta_T = (t_x, t_y, t_z)$. As we have elucidated previously, the initial rotation and translation parameters are set to be an output of rigid landmark registration procedure, which leads to the state shown in Figure \ref{f3.13}. + +\begin{figure} + \centering + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=0.98\textwidth]{Figures/Pictures/initLAlt_t.png} + \end{minipage} + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/initFAlt_t.png} + \end{minipage} + \begin{minipage}{.32\textwidth} + \centering + \includegraphics[width=0.98\textwidth]{Figures/Pictures/initRAlt_t.png} + \end{minipage} + \caption{The model instance and the target mesh alignment after applying the initial $\theta$.} + \label{f3.13} +\end{figure} + +\section*{Proposal Architecture} +The proposal distribution that will be used in the Metropolis-Hastings algorithm for the shape fitting is constructed similarly to the mixture proposal distribution described previously according to Equation \ref{eq2.3}. We create a mixture of three separate proposal distributions each responsible for sampling shape, translation and rotation parameters. Separating proposals from each other not only gives us better control over the individual proposals, but it also simplifies the analysis of the process as a whole. For example, this allows us to specify the step size (standard deviation) of each proposal separately. The rotation and translation proposals both are simple Random Walk proposals with isotropic Gaussian perturbation distribution. For the shape proposal though, instead of using just Random Walk proposal we use a combination of Random Walk and Iterative Closest Point (ICP) proposals. Due to fundamental characteristics of the Random Walk proposals, that is, proposing samples without having any specific target direction, they tend to require more iterations, and therefore more time for convergence. Hence, by plugging in the ICP proposal that always moves towards the target, we reduce the number of iterations and accordingly the execution time by an order of magnitude. The final mixture proposal distribution then is as follows: + +\begin{equation} + Q(\theta' | \theta) = \frac{2}{10}Q_R(\theta' | \theta) + \frac{2}{10}Q_T(\theta' | \theta) + \frac{3}{10}Q_{S(RW)}(\theta' | \theta) + \frac{3}{10} Q_{S(ICP)}(\theta' | \theta). +\end{equation} + +Where $Q_R$ is the rotation proposal, $Q_T$ translation proposal, $Q_{S(RW)}$ random walk based shape proposal and $Q_{S(ICP)}$ ICP based shape proposal. The coefficients of the proposals are $c_i$ from Equation \ref{eq2.3} indicating the probability of drawing samples from the specific proposals. + +\section*{Evaluators} +The main purpose of the evaluators is to evaluate the proposed sample $\theta$ and assess whether it should be accepted or rejected based on its posterior value $P(\theta | D) \propto P(\theta) P(D | \theta)$ as described in Section \ref{s2.2.1}. We introduce two distinct likelihood evaluators under the assumption that they are conditionally independent. Based on this assumption we combine conditionally independent likelihoods using product likelihood alongside with prior evaluator. The first evaluator we model is for the landmark correspondences. Its purpose is to measure how plausible observed $n$ landmark locations are under the influence of normal isotropic Gaussian noise assumption at every point with $\sigma = 3mm$ standard deviation: +\begin{equation} + P(D|\theta) = \mathcal L(\theta; \ell_1, \dots, \ell_n) = \prod_{i}\mathcal N(\ell_i|\ell'_i(\theta), \sigma^2_{LM3D}). + \label{eq3.1} +\end{equation} +Where $\ell_i$ are observed target landmarks and $\ell'_i(\theta)$ are i-th landmarks corresponding landmark on the model instance rendered based on $\theta$ parameters. \bigskip + +The second evaluator we introduce is an evaluator for uniformly sampled $n \leq model_{pts}$ points on the model instance under $\theta$, which evaluates the distance between sampled points and the closest point on the target mesh surface. The Gaussian noise assumption is known to encounter problems when dealing with outliers. We observe quite a few outliers on our target mesh, therefore, we choose to use known to be more robust Cauchy\footnote{\url{https://en.wikipedia.org/wiki/Cauchy_distribution}} distribution\cite{Schoenborn2014} noise model with scale $\gamma = 0.5$: + +\begin{equation} + \mathcal L(\theta; p'_1(\theta), \dots, p'_n(\theta)) = \prod_i Cauchy(p'_i(\theta) |p_i, \gamma_p) + \label{eq3.3} +\end{equation} +where $p'_i(\theta)$ are uniformly sampled points onto the model instance under $\theta$ parameters and $p_i$ are its corresponding closest points on the target surface.\bigskip + +When it comes to a prior evaluator, we use zero-mean normal distribution for $\theta_T$ and $\theta_R$. As for $\theta_S$ prior, we treat the model itself as a prior observation. \cite{scalismoTut} + +\subsection*{Model Decimation} +Before we put everything together and construct a Markov Chain through Metropolis-Hastings algorithm and do the inference, we have to make a small but important computationally beneficial adjustments. The model we are working on contains ~28k points and ~56k triangles. Every time we apply a parameter set $\theta$ to the model to obtain a model instance it has to calculate the new position for ~28k points. This is a computationally rather heavy task, especially when considering the fact that we do not need to execute transformation over the full set of model points. The way to overcome this problem is to introduce the standard model decimation technique, which reduces the number of points without model losing its expressive shape characteristics. We will not go into details how the decimation process works, but rather report important differences between the full model and the decimated model. In Table \ref{t3.2} various decimation rates and the corresponding number of points and triangles are shown. Figure \ref{f3.14} shows how much the visual appearance changes after decimation. + +\begin{table}[h] + \centering + \begin{tabular}{c|c|c} + Decimation Rate & Number of Points & Number of Triangles\\ + \hline + 0 & 28588 & 56572\\ + \hline + 0.1 & 25731 & 50913 \\ + \hline + 0.3 & 20017 & 39600 \\ + \hline + 0.6 & 11467 & 22628 \\ + \hline + 0.9 & 2919 & 5657 \\ + \hline + \end{tabular} + \caption{The number of points and triangles after decimation.} + \label{t3.2} +\end{table} + +\begin{figure}[h] + \centering + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/d01a_t.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/d06a_t.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/d09a_t.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/d01_t.png} + \caption*{0.1} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/d06_t.png} + \caption*{0.6} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/d09_t.png} + \caption*{0.9} + \end{minipage} + \caption{Various rates of model decimation and how they influence the appearance of the model.} + \label{f3.14} +\end{figure} + +We use the decimated model with 0.9 rate for both ICP shape proposal described previously and for the closest point evaluator (Equation \ref{eq3.3}). + +\subsection*{Metropolis-Hastings} +To recall, the Metropolis-Hastings algorithm is a method of the family of Markov Chain Monte Carlo algorithms, which is based on the acceptance-rejection sampling technique. As described in Section \ref{s2.2.1} it constructs a Markov Chain by accepting or rejecting samples drawn from a proposal distribution\cite{Schoenborn2017}. We now have defined both a proposal distribution $Q$ (Equation \ref{eq3.1}) and a target distribution $P$ which is a product of three previously mentioned evaluators(\textit{Landmarks Evaluator}, \textit{Closet Point Evaluator}, and \textit{Prior Evaluator}). Having these two distributions is all that is needed for the Metropolis-Hastings algorithm to start the propose-and-verify cycle. Therefore, we simply pass our proposal and evaluator distributions to it. In Table \ref{t3.3} proposal acceptance ratios are shown after a test shape fitting run with 1000 iterations. As expected, our ICP proposal has higher acceptance ratios than any other proposal. + +\begin{table}[h] + \centering + \begin{tabular}{c|c|c} + Proposal & Std. Deviation & Acceptance Ratio \\ + \hline + \textit{ShapeProposalICP} & 0.1 & 0.6292 \\ + \hline + \textit{ShapeProposalRW} & 0.1 & 0.1909 \\ + \hline + \textit{RotationProposal} & 0.01 & 0.1428 \\ + \hline + \textit{TranslationProposal} & 1.0 & 0.0754 \\ + \hline + \end{tabular} + \caption{Proposal acceptance ratios after 1000 sampling iterations.} + \label{t3.3} +\end{table} + +The runtime of this specific run was \textit{84s} on a standard laptop computer.\footnote{Intel® Core™ - i7-8550U, 32GB RAM, Intel® UHD Graphics 620} On a better performing machine, the execution time easily falls under \textit{60s}. Decreasing the number of iterations from 1000 to 500 further reduces the time on earlier mentioned machine from \textit{84s} to \textit{56s} without sacrificing fit quality too much.\footnote{Mentioned run times are only shape fitting times without client-server protocol} We will report detailed results in the next chapter but to make the above mentioned test more clear, we provide quantitative and qualitative assessments for that specific run. + +\begin{figure} + \centering + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/sfX.png} + \caption*{X Slice} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/sfY.png} + \caption*{Y Slice} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/sfZ.png} + \caption*{Z Slice} + \end{minipage} + \caption{The white line is our target mesh, colored line is the best fit converted into the scalar mesh field with scalar distance range \textit{0.0 mm} (blue) to \textit{3.0 mm+} (red).} + \label{f3.15} +\end{figure} + +\begin{figure} + \centering + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/gtX.png} + \caption*{X Slice} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/gtY.png} + \caption*{Y Slice} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/Pictures/gtZ.png} + \caption*{Z Slice} + \end{minipage} + \caption{The white line is the ground truth, colored line is the best fit converted into the scalar mesh field with scalar distance range \textit{0.0 mm}(blue) to \textit{3.0 mm+}(red).} + \label{f3.16} +\end{figure} +The average mesh distance calculated point-wise between the best model fit and the target mesh shown in Figure \ref{f3.15} was \textbf{\textit{1.26 mm}}. The average mesh distance calculated between the same best model fit and the ground truth of the target person yielded \textbf{\textit{0.82 mm}}. Figure \ref{f3.16} shows the result of ground truth and best fit alignment with the same scalar distance range (\textit{0.0 – 3.0 mm}). We hereby note that the red line visible on X and Z slices (better seen on Figure \ref{f3.17} right) that indicates \textit{3.0 mm+} distance between the points are due to the ground truth scan having a gap in the mandible region as well as in the nose, cheek and eyebrow regions. + +\begin{figure} + \centering + \begin{minipage}{.49\textwidth} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/bf_big1_tnew1.png} + \caption*{Target mesh vs. Best shape fit (\textit{1.26 mm})} + \end{minipage} + \begin{minipage}{.49\textwidth} + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/gtBig_tnew.png} + \caption*{Best shape fit vs. Ground Truth (\textit{0.82 mm})} + \end{minipage} + \caption{Mesh distances colored via scalar mesh field. } + \label{f3.17} +\end{figure} + +\subsection{Color Fitting}\label{s3.4.3} + +\begin{figure}[h!] + \centering + \includegraphics[width=0.7\textwidth]{Figures/ColorFitting.pdf} + \caption{The color fitting diagram.} + \label{colorF} +\end{figure} +After the successful run of the shape fitting module, we collect the resulted data and pass it to the color fitting module (Figure \ref{colorF}). We do not make too many changes in the color fitting part. If not stated otherwise we use standard color fitting settings from \cite{Schoenborn2017}. The most important data structure that is very important throughout the entire fitting pipeline is so called \textit{RenderParameter}. It is an object which holds all the necessary parameters for the shape model to render reconstructions. It combines parameters of the model such as shape, color, expression, illumination, and camera parameters. Each of these parameters are updated via dedicated proposals. We go through proposals that we modify and describe what kind of changes, if any, we introduce.\bigskip + +Generally, standard image fitting proposals are constructed as a mixture of proposals with varying scales. This is because, when proposals are defined on a single scale, the samples they propose are commonly very similar in terms of their step size. Mixing varying scales introduces an additional feature of the proposals to make larger steps when possible or smaller in other cases. This process is especially important during the initial phase when it is possible to make bigger steps and by doing so move towards the target distribution faster. + + +\subsection*{Camera Proposals} +Since we know an actual camera parameters for our D415 camera, which are fixed unless we change resolution, there is no need to infer them from the image. Therefore, we fix the camera parameters and make sure to omit proposals that try to modify this parameter. Such proposals are \textit{DistanceProposal} and \textit{ScalingProposal} both of which are part of \textit{PoseProposal}. + +\subsection*{Shape \& Pose Proposals} +Even though we already have a decent shape and pose parameters after the shape fitting, we still have to include these two proposals in the color fitting module. The change we make though is that we drop the highest step size proposals inside the mixture proposal and lower the rate of how often they propose. This is because during the shape fitting it is not possible to determine the key facial characteristics like eyebrows, lip borders, eye lines, etc. and if we do not allow slight shape and pose changes we will encounter anomalies like double eyebrows. The rest of the details are kept the same as standard fitting. + +\subsection*{Evaluators} +Besides default image evaluators proposed by \cite{Schoenborn2017}, we make use of point cloud evaluator used during the shape fitting part (Equation \ref{eq3.3}), in addition to 3D landmark evaluator to filter our samples that are not making through it. \ No newline at end of file diff --git a/Chapters/Chapter4.tex b/Chapters/Chapter4.tex new file mode 100644 index 0000000..70714eb --- /dev/null +++ b/Chapters/Chapter4.tex @@ -0,0 +1,385 @@ +\chapter{Results}\label{ch4} + +To understand the process better, we have evaluated the shape fitting and the color fitting module separately. The dataset we evaluate is exclusively based on captures made by the D415 camera, it includes images with varying faces, poses and illumination conditions. Around 80\% of the dataset consists of frontal shots, the rest of the images are varying between $\pm 30^{\circ}$ yaw and roll rotation angles. To replicate the real world scenarios as close as possible, we did not set up a specific camera rig for capturing procedure, rather, photos are taken arbitrarily. To compare our results to the previous results we have also executed a novel 2D fitting \cite{Schoenborn2017} over the same dataset. Unfortunately, we were unable to run \cite{betschard2016} project and therefore the values that we list are directly taken from the report — those values are not tested on our dataset. Before we present our results, we shall define the evaluation metrics that we employ to assess the performance and quality of the final reconstructions. + +\section{Evaluation Metrics} +The most obvious and straightforward way to evaluate the shape quality of 3D object reconstruction is to measure the distance between the target object and the reconstructed one. Thanks to the target mesh we have obtained from the camera and the reconstructed mesh produced after the 3DMM fitting, we have a possibility to work with 3D mesh objects and evaluate the distance between them. The more robust and meaningful distance measure is the average mesh distance. It measures the distance from every point on one mesh to the closest point on another mesh and averages them. This kind of evaluation metrics are known to work well when two objects that are being evaluated are symmetric. Unfortunately, in our case, the target mesh and the reconstructed meshes are far from symmetric and quite a lot of outlier points are observed. Therefore, to make the returned value reasonable we have made a small adjustment. We employ the simple boundary point ignorance technique, which ignores the distance entry for every point on one mesh that has its closest point located on the boundary of another mesh. This will ignore obvious outliers and give us a quantitative measure of how far our shape reconstruction is from the real shape. The method is robust enough and works well even when the mesh points are distributed non-evenly. When it comes to image evaluation, the best way to evaluate it is using human perception. Therefore, we make sure to provide a set of sample reconstructions alongside numerical measurements. + +\section{Shape Fitting Module Evaluation} +We first start listing results of shape fitting module and evaluating how well reconstructed shape fits to the target mesh. Table \ref{t4.1} shows the average mesh reconstruction accuracy in millimeters after 1000 shape fitting iterations. + +\begin{table}[h] + \centering + \begin{tabular}{ c|c|c } + Evaluated Against & \#Iterations & Avg. Mesh Distance [mm]\\ + \hline + Target Mesh & 1000 & \textbf{1.18} \\ + Ground Truth & 1000 & \textbf{1.47} \\ + \hline + \end{tabular} + \caption{Shape fitting accuracy in mm.} + \label{t4.1} +\end{table} + +The value for the ground truth alignment is obtained by aligning the best shape reconstruction over the ground truth using the method described previously, however, we do not perform any explicit alignment procedure for the shape fit and the target mesh because the shape fitting itself is an alignment problem. The latter also explains the difference between those two evaluations - shape fit is constantly getting evaluated against the target mesh so it is expected that the average value would be lower between the target mesh and the reconstructed mesh than the average value between the ground truth and the reconstructed mesh. + +\begin{figure}[h] + \centering + \includegraphics[width=\textwidth]{Figures/shapeFittingPlot.pdf} + \caption{The first column of this box plot shows the average mesh distance between the best shape fit and the target mesh, the second column shows the distance between the best shape fit and the ground truth. The red line indicates the median (Table \ref{t4.1}).} + \label{f4.1} +\end{figure} + +Figure \ref{f4.1} shows the distribution of the entire dataset. Outlier points in the first column are produced by $\pm 30^\circ$ yaw rotated shots. As we have emphasized previously our method performs noticeably better in the frontal shots and the worst in shots where the yaw rotation is introduced. The reason behind this goes back to the landmark detection procedure. Recall, we detect landmarks onto a color image and then de-project landmark pixels to points. The problem occurs due to the fact that whenever the Dlib landmark detection library detects a face, it provides the full set of landmarks without respecting the pose. If the part of the face is not visible, landmarks that are located onto the occluded side of the face are still present. Those landmarks are first not very accurate and secondly, they are getting grouped into one area and the distance between landmark points does not correspond to the actual distance. This either breaks our shape fitting part or produces bad reconstructions, because of this constraint majority of our dataset samples are frontal shots. In Figure \ref{f4.2} we show two sample shape reconstruction after 1000 iterations. + +\begin{figure} + \centering + \begin{minipage}{.325\textwidth} + \centering + \caption*{X Slice} + \includegraphics[width=0.99\textwidth]{Figures/eval/our/3/x.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \caption*{Y Slice} + \includegraphics[width=0.99\textwidth]{Figures/eval/our/3/y.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \caption*{Z Slice} + \includegraphics[width=0.99\textwidth]{Figures/eval/our/3/z.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/eval/our/3/x-gt.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/eval/our/3/y-gt.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/eval/our/3/z-gt.png} + \end{minipage} + + Sample 1 + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/x.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/y.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/z.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/x-gt.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/y-gt.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/z-gt.png} + \end{minipage} + + Sample 2 + + % \begin{minipage}{.325\textwidth} + % \centering + % \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/x.png} + % \end{minipage} + % \begin{minipage}{.325\textwidth} + % \centering + % \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/y.png} + % \end{minipage} + % \begin{minipage}{.325\textwidth} + % \centering + % \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/z.png} + % \end{minipage} + + % \begin{minipage}{.325\textwidth} + % \centering + % \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/x-gt.png} + % \end{minipage} + % \begin{minipage}{.325\textwidth} + % \centering + % \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/y-gt.png} + % \end{minipage} + % \begin{minipage}{.325\textwidth} + % \centering + % \includegraphics[width=0.99\textwidth]{Figures/eval/our/1/z-gt.png} + % \end{minipage} + % Sample 3 + \caption{The first row of these two sample fits shows the target and the best shape fit alignment, the second ground truth and the best shape fit. In both cases, the colored line represents the best shape fit instance, where the blue color indicates 0.0 mm distance between the points and the red indicates 3.0 mm+ distance. Both shots are frontal shots to make the slice visualization easier to understand.} + \label{f4.2} +\end{figure} + +As we can clearly see from the sample slices above, the highest errors (colored in red) are being accumulated in places where there are a lot of curvature or acute angles. For example the lip details or nose angle. This is because the point cloud itself does not capture this kind of details, what we have instead is a rather smooth and generic shape without detailed facial features. + +\section{Full Fitting Evaluation (Shape Fitting + Color Fitting)} +Once the shape fitting module ends its cycle the final reconstruction and other necessary parameters are then being passed to the color fitting module. It starts another proposal-and-verify cycle to approximate the color values from the target image. Figure \ref{f4.3} shows the comparison of the shape reconstruction quality after the shape fitting and after the full fitting. + +\begin{figure} + \centering + \includegraphics[width=\textwidth]{Figures/our.pdf} + \caption{The average mesh distances after the shape fitting only (1st and 2nd column) and after the full fitting (3rd and 4th column). The first and third column shows the average mesh distances between the reconstructed mesh and the target mesh, the second and fourth column shows the average mesh distances between the reconstructed mesh and the ground truth.} + \label{f4.3} +\end{figure} + +As stated previously, if we do not add shape or pose proposals to the color fitting module, obviously the average distance will not change. While numerically this would be perfect, visually we would not get satisfactory results. Because of this, we had to make a compromise and include shape and pose parameter proposals into the color fitting module. Even though we evaluate landmark positions and mirror the point cloud evaluator from the shape fitting module, shape still gets worst in comparison to the shape fitting results.\bigskip + +Alongside our fitting module, we have evaluated a novel 2D fitting approach\footnote{Basel Face Registration Pipeline — \url{https://github.com/unibas-gravis/basel-face-pipeline/}}. To do so, we took our entire dataset images and obtained landmarks for each target image using the \textit{landmarks-clicker}\footnote{Landmarks Clicker +— \url{https://github.com/unibas-gravis/landmarks-clicker/}} tool. We did not make any changes to the standard fitting module, we simply pass our dataset images with corresponding landmarks to the script, and in the end, evaluate the final reconstructions. Table \ref{t4.2} shows the result of the final average shape reconstruction accuracy calculated against the ground truth.\bigskip + +\begin{table}[h] + \centering + \begin{tabular}{ c|c|c } + Landmarks & Shape Information & Avg. Mesh Distance [mm]\\ + \hline + 2D & No & 2.21 \cite{Schoenborn2017}\\ + 3D & Yes(Depth) & 1.94 \cite{betschard2016}*\\ + 3D & \textbf{Yes(Point Cloud)} & \textbf{1.61} \\ + \hline + \end{tabular} + \caption{The final shape fitting accuracy in mm obtained by aligning the final 3D reconstructions over the ground truth. *BFM dataset evaluation value; taken directly from the paper to have a rough idea how does our method compare to it — we did not manage to run the module to test it on our dataset.} + \label{t4.2} +\end{table} + +Table \ref{t4.3} shows the average distance between the final shape fit and the target mesh. This value is obtained by aligning the final reconstruction and the corresponding target mesh. \bigskip + +\begin{table}[h] + \centering + \begin{tabular}{ c|c|c } + Landmarks & Shape Information & Avg. Mesh Distance [mm]\\ + \hline + 2D & No & 2.94 \cite{Schoenborn2017}\\ + 3D & \textbf{Yes(Point Cloud)} & \textbf{1.56} \\ + \hline + \end{tabular} + \caption{Final shape fitting accuracy in mm obtained by aligning the final 3D reconstructions over the target mesh.} + \label{t4.3} +\end{table} + +Figure \ref{f4.5} shows the visual comparison of the target image, rendered model fit of our method and standard fitting method. Figure \ref{f4.6} shows the same results as Figure \ref{f4.5} but the final fit is overlaid to the target image to make the comparison easier. We argue that the reconstructions that are produced by our method are capturing a bit more details and facial characteristics, whereas the standard fitting results are more generic while still maintaining close-to-target state. We think that this is because our shape reconstruction quality is noticeably better and hence after applying the texture and illumination we get a bit more close to the target results. We once again emphasize and more closely observe(rows 3 and 4) the difficulties our method experiences when working with poses that have yaw rotation applied.\bigskip + +The box plot in Figure \ref{f4.4} shows the distribution of the average mesh distances of the entire dataset. Where the first and second columns indicate our final shape reconstruction accuracies, and the third and fourth column indicate the performance of the standard fitting method. Unlike results that we have listed for the shape fitting evaluation, this time, the final fits are being aligned both to the target mesh and the ground truth mesh. + +\begin{figure}[h] + \centering + \begin{minipage}{.325\textwidth} + \centering + \caption*{Target Image} + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/1.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \caption*{\textbf{Our Fit}} + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/1.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \caption*{Standard Fit} + \includegraphics[width=0.99\textwidth]{Figures/dataset/2D/1.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/3.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/3.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/2D/3.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/4.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/4.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/2D/4.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/5.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/5old.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/2D/5.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/6.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/6.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/2D/6.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/7.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/7.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/2D/7.png} + \end{minipage} + \caption{A visual comparison of our method and the standard 2D fitting results.} + \label{f4.5} +\end{figure} + +% overly +\begin{figure}[h] + \centering + \begin{minipage}{.325\textwidth} + \centering + \caption*{Target} + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/1.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \caption*{Our fit overlaid} + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/1blended.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \caption*{Standard fit overlaid} + \includegraphics[width=0.99\textwidth]{Figures/dataset/blended/1.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/3.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/3blended.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/blended/3.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/4.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/4blended.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/blended/4.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/5.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/5overlay.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/blended/5.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/6.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/6blended.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/blended/6.png} + \end{minipage} + + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/target/7.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/our/7blended.png} + \end{minipage} + \begin{minipage}{.325\textwidth} + \centering + \includegraphics[width=0.99\textwidth]{Figures/dataset/blended/7.png} + \end{minipage} + \caption{A visual comparison of the same fits from Figure \ref{f4.5} overlaid to the target image.} + \label{f4.6} +\end{figure} +\clearpage +\begin{figure} + \centering + \includegraphics[width=0.99\textwidth]{Figures/standard.pdf} + \caption{The average mesh distance comparison of our method(1st and 2nd column) and the standard fitting method(3rd and 4th column).} + \label{f4.4} +\end{figure} + +\section{Execution Performance} +In order for our system to be useful for demo applications the execution time of at least one module needs to be reduced down to seconds. While it was not possible for us to achieve a decent full fitting performance time-wise. We were able to execute shape fitting module within the reasonable time limits (bellow 30 seconds). We show average execution times for each module in Table \ref{t4.4}. + +\begin{table}[h] + \centering + \begin{tabular}{ c|c|c } + Module & \#Iterations & Avg. Execution time \\ + \hline + Shape Fitting & 500 & $\thicksim$ 25 seconds\\ + Shape Fitting & 1000 & $\thicksim$ 45 seconds\\\hline + Full Fitting (Shape + Color Fitting) & 5000 & $\thicksim$ 10 minute \\ + Full Fitting (shape + Color Fitting) & 10000 & $\thicksim$ 18 minute \\ + \hline + \end{tabular} + \caption{Timing the execution times for shape fitting and color fitting module.} + \label{t4.4} +\end{table} + +\section*{Web-Service Integration} + +Based on shown results, the shape fitting does have the potential of being used for demonstration purposes and it is possible to integrate into the Scalismo Face Morpher web-service. Obviously, listed times are directly dependent on sampling iterations, increasing them will increase the execution time and decreasing will reduce time alongside with quality of the reconstruction. To be able to integrate our proposed shape fitting procedure into the current web client we have implemented a single intermediate Thrift server which will connect the already existing Face Morpher web module and proposed shape fitting module to each other. The flow of this system can be seen in Figure \ref{f4.7}. + +\begin{figure} + \centering + \includegraphics[width=\textwidth]{Figures/web-fitting.pdf} + \caption{Connecting web client to our proposed fitting modules using the single intermediate server. Everything that is inside the orange box is part of our fitting pipeline.} + \label{f4.7} +\end{figure} + +Whenever a user triggers the starting sequence, the web client sends the request to the intermediate server. The server, that has our fitting pipeline embedded into it as a library, then calls a method from our pipeline which triggers the fitting client to send a request to the camera module in order to obtain the data and start the shape fitting procedure. Due to these multiple client-server interactions, the overall execution time will increase slightly but it will still stay in an acceptable region. \ No newline at end of file diff --git a/Chapters/Chapter5.tex b/Chapters/Chapter5.tex new file mode 100644 index 0000000..44ebe62 --- /dev/null +++ b/Chapters/Chapter5.tex @@ -0,0 +1,13 @@ +\chapter{Conclusion}\label{ch5} + +We have successfully implemented the complete, reliable and coherent system for more efficient face fitting pipeline. We have taken a modularized approach to tackle this problem. By separating data acquisition (including the module which handles all the camera related processes), shape fitting and color fitting modules from each other, we have made the system useful for any future application that will either make use of one of the modules or all of them together. We have introduced an alternative, more efficient way of a standard 3DMM face fitting by relying on a point cloud information obtained from the consumer depth camera. We have shown that our method produces especially good shape reconstructions in comparison to previous approaches. This was achieved by fitting the 3D Morphable model to the point cloud first to obtain an accurate shape and pose parameters. Those parameter sets are very important prerequisites for the color fitting module and directly influence the final reconstructions. The parameter set that is produced by the shape fitting module, is getting utilized by the color fitting module to obtain final face reconstruction. Based on more accurate shape reconstruction, our method also attempts to capture more specific details of the target face in the image. \bigskip + +We have presented an evaluation of our proposed shape and color fitting modules, as well as, comparison of our final results and the previous approaches. Our final reconstructions are noticeably better than previous methods. Especially stands out the shape reconstruction quality, which we measure based on average mesh distances. This was expected since, out method has an actual shape information to rely on during the fitting process. When comparing our shape and color fitting modules we have observed a slight decrease in the shape quality. This is because of the color fitting module requiring additional adjustments throughout the process to correctly infer the color information from the image. The decrease in shape quality also is due to increased difficulty of the fitting problem during the color fitting phase. Despite this decrease, final reconstructions that our method produces are still more accurate than previous method results. Our method performs noticeably better in full-frontal shots and suffers mainly when yaw rotation is introduced. This is due to the landmark detection algorithm onto the image, which provides all landmarks whether the landmark point is visible or not. This is not necessarily optimal for our application because pixels that get de-projected incorrectly might cause problems for shape fitting.\bigskip + +We have also presented a demo implementation of our shape fitting module into the already existing Scalismo Face Morpher web service. +Since our system is module based, this kind of implementations are straightforward and can be easily preformed for any future applications. + + + + + diff --git a/Chapters/Chapter6.tex b/Chapters/Chapter6.tex new file mode 100644 index 0000000..d7e2683 --- /dev/null +++ b/Chapters/Chapter6.tex @@ -0,0 +1,24 @@ +\chapter{Future Work}\label{ch6} + +In this chapter, we will be discussing potential improvements to our approach that were not possible to execute in our project due to time constraints. + +\section{Using Multiple Camera Setup} +\begin{figure}[h] + \centering + \includegraphics[width=\textwidth]{Figures/Pictures/multicam.jpg} + \caption{Multi-camera configuration examples. Source: Intel} + \label{f6.1} +\end{figure} +As we have seen from the previous images we are only working on the face region instead of full head region (including ears). One of the reasons we opted to do so is that one single camera view is not able to capture some parts of the face that are occluded. However, if multiple camera angles will be used this problem will no longer persist and it would be possible to work with a full face model instead of the reduced model of only the face region. This would also improve the pose estimation process and will most likely produce more accurate shape reconstructions. Thus, we think that it is worthwhile to investigate a multiple camera setup (Figure \ref{f6.1}) and obtain a more accurate and well-structured mesh from a point cloud. The Intel RealSense SDK does provide support for multi-camera setup out of the box\cite{multicam}. Especially useful would be, the inward facing camera configuration from Figure \ref{f6.1}. It will capture the target object from four (or more) different angles and will likely produce full $360^{\circ}$ scan.\bigskip + +Additionally, tweaking camera parameters and performing post-processing steps as per \cite{bestcal} could also improve the quality of the target mesh constructed from a point cloud. As we have mentioned in Section \ref{ch3}, there are a lot of configuration parameters available which we did not investigate in detail due to time limitations. Denoising the depth data is one of the obvious optimization steps that would be beneficial for the fitting pipeline\cite{icpram19}. As discussed previously, a point cloud that we obtain from the camera by default is very noisy and lacks the important details and facial characteristics. The higher quality point cloud would most likely produce final reconstructions that are capturing more details of the target face. + +\section{Better Landmark Filtering} +The main reason for our method being very sensitive to the yaw rotation angle is due to Dlib landmarks not being filtered according to the pose. One possible fix of this issue would be to control the Dlib face box during the landmark detection procedure and by doing so only take into account landmark points that are actually visible. Recall, Dlib estimates a location of the landmarks that are not visible in the image and returns a full set of landmarks that are then being de-projected to obtain 3D landmark points. \bigskip + +Another work-around would be to assign a higher uncertainty to the landmarks that are being estimated. In our implementation, we have used higher landmark uncertainty for chin landmarks than for the rest of the landmarks. However, we have no way of dynamically detecting the pose variation in the image to ignore altogether or assign higher uncertainty to a subset of landmarks that latter cause problems for the fitting pipeline. We think this is a very important feature update for the entire module since both shape fitting and color fitting module crucially depend on those landmarks. \bigskip + +An alternative way to deal with this issue would be to implement a native landmark detection method instead of relying on 3rd party pixel landmark detection method and then de-projection which introduces an additional error. + +\section{Multi-Resolution Fitting} +Unfortunately, due to time constraints, we were not able to deeply research the color fitting module and its speed up capabilities. Because of this, the web service integration was only possible for our shape fitting module. Therefore, we state this challenge as a future work, one possible solution of which could be a multi-resolution fitting. As the name suggests, the basic idea is to perform the color fitting in multiple stages, starting from a low-resolution image and model, and gradually increase the resolution and the complexity of the problem \cite{10.1007/3-540-47967-8_46, surrey809478, visapp16}. \bigskip \ No newline at end of file diff --git a/Figures/Capture.PNG b/Figures/Capture.PNG new file mode 100644 index 0000000..0c283f1 Binary files /dev/null and b/Figures/Capture.PNG differ diff --git a/Figures/Client.drawio b/Figures/Client.drawio new file mode 100644 index 0000000..6c6231b --- /dev/null +++ b/Figures/Client.drawio @@ -0,0 +1 @@ +7VrZcuI4FP0aHpOyLa+PAZLMTK+VTM1MnrqELYwSYbllEaC/viVbxptISGNDqmuoCrGuFpt7zr06sjQCk+XmlsF08YlGiIwsI9qMwHRkWYFhi29p2BYG37AKQ8xwVJjMynCPfyBlNJR1hSOUNRpySgnHadMY0iRBIW/YIGN03Ww2p6R51xTGqGO4DyHpWv/FEV8oq2kYVcUfCMcLdWvfURUzGD7FjK4Sdb+RBeb5p6hewnIs1T5bwIiuayZwPQITRikvrpabCSLStaXbin43e2p3z81Qwg/p8PmfaYYfvzxcTD+waXyTzsbAvwgC9XB8WzoERcI/qkgZX9CYJpBcV9Zx/qORHNYQparNR0pTYTSF8RFxvlVgwxWnwrTgS6JqxROz7X+y/6VTFh/qddONGrwobXelkkCm7JdxRp92qNnC0vVJSSnIYsRfcISnHCF/fa2ncuUtoksknkQ0YIhAjp+b7IGKhPGunep6xRjc1hqkFCc8q438VRpEAxVOPgDFiCqaXNAAVVwUI+p7u7ah7b27fUZXLESqV2PY2m+sTDmD3sAm0z4LmzaY18gkSg8lX8R1RSVZOJ5JhQtfcIL1PohkNangH8Mjv0WjIpoGo5Hy4DMkK1Tm1hatKtJIBqwXmKP7FOa4rMU01STIHBMyoYSyvC8wjMnEMHao12qsqefmNXOa8Jpd5XUwjhmMMNLX7WXMM2IcbV7kgqp13KbfzTJ+19XM5JTALmqTku0Y+/nTgOfNIW10wLhD31co4x1MxI/kHa8mNEEtCJSpDpBwtwpHyy7LamBZLz2IxZR9RXCcCNsSR1GeO6AyhMLniPUDgmfYIpHo6F9DYScPGigMBoKpASFLaZKh3xeFJgZOcG4QrAPSktB4qbxMGQ1Rlr2emioR+WXFCU5KX7dS1twPURjqUtbMd2Tw9+Nzp+lz29X4vGxT97lvD+Vz54w+n88tvc8jd+Y67jA+B8ahPn9BMRzlc8/W+NwleVJJYdJwvvt9JZcveaq4yPLccSUaACPdVJXiKpb/J0ROneVY4tmK4YrKd5nHGOVColFpuwh6CjLHb87xoOT4KRLbh+Q58MbofnLzF7rbcvQZXK0vgNlx/uuqPVyx553+OpOE/7VFZFPrW0dpfU9h9+ry0ndHPa8KjotxXV4t4lJGyyExbrq6GP97wfCc/7mUL12qQC/G3BPox0j5MEIzf6bL0cAFAYgGmhed4LAcXYZ2//i5g+L3CWWLU8AnV06+Pzx8bUFv+2cH0BsUwI8wiZaQPWWnQTEMg+D0KDrm2VEMjpZKpq9D8Q6FSE4Cb9NKg2ggjdbqA03faqNpn3CBp9dB3bXF/zroMB3kgwN1UND729GjArh87t7T8Fe8QaSQQc747nY8cqa/rx6yAu9Ss4g5aSr2B1O0DMMkJijXRM74mxjgNFieShy13j9Y7vmxHErdlrLoxDieRh61cHTOjWKZ6ntHcQo5FJUTgmCCk/hQDAmcIfI6enhZHBjASzHyTUiwgEUMenOLIPtmWv5G/F2m4rbagO1rJ8izvQacpmPpwtIzu4B6QwFqGjpEX1BMiMzoui6WcoOoWFCGfwisIMlVSXQlz3FI0UlgluGwCUhXp+wUz07iPDTlz/7jAzeYVIcQGlvApfQqZVhtyL3S69cF08E6qL7zpxHDpe1tm8mdDd/AaG08Bi0S7Tk40BnINFzn0rQt3/GK7+ZxBiAWcUFVaTpu8zZ7NpbfuvkdWC2B4gcqerT7333tW/vHvznXLwdbeTPc5ZyqITDyj+atO01k2pHinIovefJJZk86e0Qh3/9+4F0uPPWbAXv26vvIwqD1rl6zCWmBgdao+MfzBnAWTpwtmUSreGqD9KK7EXwMN/ph51eRtCW9Fkh8R/nkXAw7Y2WTnHpFg2K7znLhUk68ySxLaw9S9ZhjzvPJ3VjSaEVQt8Vejh4mB94/dTs81bB5L3UD02+mQNvuygcXdKnbzvwHUFcUq2OMRTKtjoqC658= \ No newline at end of file diff --git a/Figures/Client.pdf b/Figures/Client.pdf new file mode 100644 index 0000000..0c4cd5d Binary files /dev/null and b/Figures/Client.pdf differ diff --git a/Figures/ColorBarBig.png b/Figures/ColorBarBig.png new file mode 100644 index 0000000..98f3711 Binary files /dev/null and b/Figures/ColorBarBig.png differ diff --git a/Figures/ColorFitting.drawio b/Figures/ColorFitting.drawio new file mode 100644 index 0000000..9bc11a3 --- /dev/null +++ b/Figures/ColorFitting.drawio @@ -0,0 +1 @@ +5VpbV9s4EP41eYRjW77EjyQBtktpOctptzyKWHFUHMsry7nsr1/JluKbEgyxy0J5AGukkZVvZr4ZTRiB6Wp7TWGyvCUBikaWEWxHYDayLM91+W8h2CmBWQhCioNCVBHc43+RFBpSmuEApbWFjJCI4aQunJM4RnNWk0FKyaa+bEGi+lsTGKKW4H4Oo7b0bxywpZSahlFO/IFwuJSvHjty4hHOn0JKsli+b2SBRf5TTK+g2kuuT5cwIJuKCFyOwJQSwoqn1XaKIgGtgq3Quzowuz83RTHrovDl+yzFP78+nM1u6Cy8Sh4nYHwGil3WMMqQ+hj5YdlOAZR/RCQ2MUdgsllihu4TOBezG+4RXLZkq0hOL3AUTUlEaK4LDGM65UCCScooeUKVGWvmufnMgsSsIpcQgklIYYDR8bkZptwlMIn5fEyoAHzShkUitUaUoW1FJGG6RmSFGN3xJWp2XGhIj3bHEqVN6R+uK4FaVlwD+J70S+mT4X7r0i78QZrmBWYy1ZEqhiqAsYwrzBiOw5bd+IdlLeRjEqOGmaSoakRuEhmolq3GcmMxL5DEPIIuIhwK6Fc4CMTkBErBnGOPaD/GAIZ97tTt4Tkte+zDtWoP2zjdHNfJ9ia4+7a6H19Nd4Z9ffstoCpqKmCjgJOKHApHJCGJYXRZSidlGAmvL9d8JiSRuP5EjO0k8DBjpG4VtMXsh1DncBSjB7mZeJ5tq4OdGsT84/6oDh7KHcSwVMtHSq9wGsWHgEvmGV3vSSCA6XI/KLAQABwkJilKSUbnctVNvPa9CbqfXv2J/tox9AVcbM5MCSyDNETs2EJH7zMURZDhdf0kp5j/6DFrrOlGItoCvOaPIctxKURpAuOas7j/ZILz87A6S3NzX/AF5jjZlpPNXR5pU8KPXmzdEmsOkUWnHyHCSqKo59NKJNfyxeWKts4dwdwdLGMakSzoqCOOanyGcbCC9CntqDRBqXgPmN3e8j/3S5ggcdQ4ZTCeHz8tFwqglLDHRLhPXYoiI7Rgx6i0jzTm12hzrAqRCm1ahiaNmaCHwNHypv8K3qwRT48kavhuhUbNc8NQ889R6bnheTU6fYZM89EdopiDKLJiLqxRaJtuxaubKn0SrdOVaP03JVpn2PJ0H5U9RJsF3Fq4eUATbratCTerh6pRD5/7juDzm2WeZ2rKPEtT5oEeyjw9ft7BPC8S5+sz6h0lCUmruajY70Di+TCVvN8ycTtGfm0l7/xvMpJ6rhToXXLRvpJ/UHscyUTPJJ0+E4zfMcFY7i9KMFrzK/97S/uXtvT5Lb92TfN97xmLvqa26G7m4e9hUlVdDiRT2C6o8YTvNrYo/E9qNXxgf4wTiN9/R4nTNhtY6eoOT1N3GPZAeVMl6cHhCyr9P+4Sef+PEgal6Mw3+sEYNDAe+955uzgB2uLOGQpkUwNyH8XJd84mi91vX5v4ttWtNhms/FRRo7Fwv+2ceqdK03rhj5fiFJCJts9z7acXdIG02x7Tlx2njnrH+zmvd10dXR10V8JnFlH+zc+Sy1DcF+97TY/12h67r5yrHjsezGN1F84+OKne09P08o5yVC+5pvUNlpH/9GNMRxGLSjCO5mI01pjS8QcyJdC1XobsJRd3Yhh1JYIoylY4LlL9i7VVV/iFanf5pb2b1sfmnno95AG/zTxAW3OC0/1Vf5PTlUNvXHO2ANXAfhBjv/4ltOe2IQYaQjANdyiED5cjp5H7pzjJ2PurN0+ybuOCOzY6fqXdR7Gpt679weLH8dxmt1F7a7O0NDUYzIfT6mlB9DVjv18UOXajbFLGHCCM+LD8X62i0VT+Pxy4/A8= \ No newline at end of file diff --git a/Figures/ColorFitting.pdf b/Figures/ColorFitting.pdf new file mode 100644 index 0000000..0fa764d Binary files /dev/null and b/Figures/ColorFitting.pdf differ diff --git a/Figures/D415spec.png b/Figures/D415spec.png new file mode 100644 index 0000000..edc6d1c Binary files /dev/null and b/Figures/D415spec.png differ diff --git a/Figures/FittingFlowNew.drawio b/Figures/FittingFlowNew.drawio new file mode 100644 index 0000000..ebd6840 --- /dev/null +++ b/Figures/FittingFlowNew.drawio @@ -0,0 +1 @@ +7V1td5u4Ev41OWfvB+cgiRfxsUmattt2m5vs2W7vlx5iyzYtAS/GTbK//gqDMIjBxraEsZOesxsjBMYzo5lnXjSckcuHp3exN5t+jkYsOMPG6OmMXJ1hjDA1+J905DkfQcTJRiaxP8rHVgN3/r8sH8wvnCz8EZtXJiZRFCT+rDo4jMKQDZPKmBfH0WN12jgKqt868yasNnA39IL66Fd/lEzFzzCM1Yn3zJ9M86+mVn7i3hv+nMTRIsy/7wyT8fJfdvrBE/fK58+n3ih6LA2Rt2fkMo6iJPv08HTJgpS4gmzZddcNZ4vnjlmYtLngj7+u5v6PL98GVx/jq8n17P6C0IHr5g+XPAuCsBGnT34Yxck0mkShF7xdjV4sfzRLb2vwo9WcT1E044OID/5gSfKcM9tbJBEfmiYPQX6WPfnJ3+nl51Z+9K105uopv/Py4FkchEn8XLooPSyuSg9Wly2PVtcJoUPpdfMkjn4WnDb5SJ2OOWnn0SIesjXEE+LqxROWrJnn5EROKVv6hpxN71j0wPgT8wkxC7zE/1WVTC8X8EkxL7/0TRx7z6UJs8gPk3npzjfpAJ+QL1Zsm9kd86VqE0lgpPmUkHXz+YfsCcRR6aeshpZCuIVA5j/6lxcsmFhXkoCuxC/l/ePUT9jdzFty6pErqaqojf0guIyCKF5eSwzj8tIwCjkoncFXjr08M47CpDSer2lyMYm9kc/gc40y9IvFCXtay3Rx1qgQGwm18bhSSpaYMy3pI9M2muWkxKsdWGH2RzcYO+kG43C6AbfUDagXqoHiqvTR9ZoB29aa6ZoUA7IPKY6oLIyFaG4QR1QRxpVsNovjmxTR8MNh4M3n/jAbvPaD4nF6ILHI6oXMIiQJLbGstVJbu4BSol9u8Us1aJYtWTQCWDQMWTRLl0UjR8kLce7Kj7kD5EchPx+mqkwL8LDzRbEReBDX0cQmG9XY0ndNrxJ44L3UuNlSjZukF1ocO1Xxw8Z6JW6StfP16HAT0Bt2wKl7MauIqf3PIvXmLwI/ZAOxVt7wKemPtFbn+adJ/nd5l/tYHuGPOVuNSYthPvVm6cdZHA3ZfL5ZT61CFV8WSfpwsP4a0yEbDiH9dU+tVCur0TeWZBdo3S4g0C5oc3SMGn9v2T8LNk9qpOe/MakRKIxSilaomQ+VbQXX/Pmax6Y4zm+cnk8J6A+94E3gT1IN/+CPRkvF5uUDQ05yFiuyzRISMgEWGJ2yoO7337L5LArn7KXwwHIPzQTzoB7WdsFAbfZTcGWjAcXKXfe9mIfrSqw7wzEeY9hwjOx727L1GA4MLRjIcDi6FgxuEa3URfORxejIhGhO8T2xFdFcwKvCUBD30DRv4VAfN0ByLInmNiDnYk6Z5tTURXPr1HWLTHNitKW5Ln1ObIDmmXOQwhbQ70hPDOZLw5h6HciePdWdjg8PaUZ05Wdkd2twNfYJhwxH7J7eQ5wjNnHJSJM74VjtNBTWpaFEAlw555au8GUQLUZdsC+NPlHaNfswtPA6ZZ99WBS8S/Rp+zyDtuhTHlXaHH6iZ31Cz2aL0LA+Czccuq7uhUYkC4ctyMIBC63wQdUTXZeiPHET19rx0aYjxep9NXE7sI+gQyMU09XEvk9eOHrw4p/zbrjXhd6scQ+KlXbKPQuK9GzJPQpx79KbJYu4teLsQxx2BORE4yjx8iFVKQspSYUAGaCACGiL1FrOQTGqgrLNvTKkaC+MKoDQRoxq2b3CqBYUhVGhta9YklZWY+PGf1qWd3erxt2xk17aU/irbw3rCvBcsQF3UX5kLOVrmPM1T8KfKjMJxucApoLYSbSxUxck/nMa++PkxFyanfMK+lCVLkyc8e8zm09Px6WpsU9sLeiCfR/DX65zwe4ur39nt88J+4O8eSyCrlshouEi/lXQ+ajgkcqYnUgvbcRDFPcKD9kK/KA1K/YEfdnasqWH1rp2c2ndnr5sFKZ0EQgoYyn/8OU+BUateZr7uFp8V8BHVsBjZDjnEjoC8scFDuoE6toKPBeQy2KN8gmZE5O6/euKKnsf0FCBjuWtZA7g6nRaWOZA2ZXt+E8MiP93jBvw+Jj4WwpODVxl4Sl5xWPAG+qW4816fT7zwt05fhmkWyFKHM9u98I4blHjcAyH8fcueza6wN+77RzdtKtcJf522sYjab/ikY6ueORJRjxqVX0WgL2hCjOxtNXzT1cA8hQjHo5lnrdwnrploK76iZN1gOtctNDBudgceGwLlWDn6JYNWWoEtsNKR+TqOhTL3Dz4RiYRI1POzTsWjk6XlRhJjITScxAjVeRzYECLXwHtjoCWti0CdfsVUKb7hykaqtrSvHqGZ62L23cXZ9bV6QJb7DpgMrZLm0q1uSax74WTgC3BrXXxnd+gG152hXLlQJJ9eF7qclOK4HG3fOwG50p8tA7NRaHq1Ve7eInHT14GzAv9cNKWh4F3z4LN3PMfsvaR/gO/8/Uw8Dlb+E2v3zEv/o4wfeL/nc/414ILVlUrHUdUyouIvgXWuziozlBHF0ORAXF0DWJiwX30WAZLywF+YhrF/r+cV15w1tADq8SQOk7ZYxdMc2OtXZrB7A6YWuOgck8eAAyLse1attR6rLiGlEByJSHKflB+1UqO6h23DNs6RyamlpP9v9pQknBv3F2dRKIiTnxNhh9rX7NtixkXSwCFuucdNI0p+qS9tpta1xbMdmhNk8FtwZAunx2heiDtbrnZCxvXfpIszQrsYh97rpgDBSmEYjsgfO+4G0y9mDIj6MkzBLlGHxkCdrrIkNYi2B28iXsEvjxSDAi+y+mg2hV8rH6b1YgwCvLmsnXXpL8HLIavX8QHU0Ko8BiEWAVsnKgRKleqCKEEQo/g3k9t8NGE3DrFDrMq/G1TWqGfQ+o9EjG0IPXtnUXg5tme0s/BEv0QqNIgq6+tXh+p2MMKRupv4mgWzY9qG50KHptVHeMYPTBbVosGTZs8zpE3nxZrqOp+krM9g/TbuqzbJSAaubr5zQZi5uZu0GbbopSOfFe3ip6oXLrc3nc1q3dy5JZWqtxT6YmdLrpSI3AzYU/Nh+3YEibmNAOUC9ckdbmyqS7lYmtuJ12QsLzJmYuvvMlZWVWh4yCZzgSiMwHIjLRtEUDa9gj8xWJ//PzS7LTYaSdY3Lo6HO/PYDDTjhvZq8e1LJdOVDw8/vFt+hReknqcDaUWuzib4G239SF3lz9I7TTKXMTPjIMsMM/HWKhIh9uupFu4kEH7UqAchi7JgyqvFKPDVTkGcqtgzzCcTXCvgJZo2eew2FJ4bqAWncFuuHrjdEoVRzFDHtsDIgrgtxkiCqzRE4hIRY2W0IAW2Q0i2o50I7cdQtwBxIHiC/QtVy6/hQjSsnsz4OJLnPUiqFba+iZDpmRFXbSbDFG7mtotQnPqZej639skCf/7HP2Yfnv/9ZM7ev74NAAk5siaHRrmJjUKKcO6qLcSTngdtnWWlb9yYy/rB/TcV1OScbF8c4JBrj5/5n/yTJq8KfNDOE+8cNjJDo/Ck6plSo3lPzUQR85sUgtynoreURWtpCDMDvMYAtdbmoi9VuuWZagVc7RluGvHJb3e3pTX9NpV1BPDJL8fB2E5bd7WMrnyjeTXculGN83VuCP/F+zU7essgnvGG/w/4CH0+6tN2VDoGcEfc8AU6WoENBAGYBG05lmbjISUf230lBWYDELkxGy9GAZBnQWR2Lak3mIcUWAYYSIFFRxCoMCwttwsTMIjSm0jyf/oODUL00/BvsbXzGypPYTE4m4zszCLoeS7rtDFcSZm14KizbC0X/ES01CUljWtbrKyJpKzsqYk9PtlZWGmtQhI98ZwGFROFjouZHsFIVW/qQjOJbUIiR5VThYbtdw3rqdNtGVkYSIrCBy9JmRX5lkGsa7YCHEw8yxWzWtOdjWSu9yvuVygxaBLEaj6CaD6tclsi3IcZZDScMs7w9C5YYjzirNhxwDuEKmW91IivRm+dc2dqDFZiVUrcKcKf4n6sW4Sqk51c6GBu5Qgt1cSRJ1zlxqrf1UxoCY9lzcitBUp7NqSbLbLrzb7C8qFrkXA7YWnaNaK8EZXGPerQnmAbFUpmoEl79TtOkmDdTVnqKYITi9ZLO/Igt7xCwVdVWSK4TYb+3dshZ06SR6GBT1XEwVlN3XxvuMgN232EG1o4t3LLlkw2G/YoK1Awhwpt0SAtwZqawMOt9Rq7sqj1qsUWdIs3u8FbT28IFg8+GEWw9n6apHG3PKym2VCot1VJ+1UYnnjswPsUm3I5BFdEtts3tRKbF14jN9uv/5nZ9EzfvtwedP28lsRt9xaeP+MvXAebHXxKcuwXWtI6ZB6KK9BhhWU+b39OHD/Nt+HH9ybr39d//n0dvD3H4PmYO3h7PoN92FSoz5NpXa07OAk16wsDX42YQ5XEI5FJwjjIRotAmDGK0BYJ61FIqF4JV7rt6ip6CrFjDdPH8h773+zwafv7Nu9S37/CAqrDmzfQCuAos3kIw0pzY0v01GwIxEkHhS07ynxLOFuFCEeVCOe6IpVoZ2CxCFIOwWtP0Et+SGcLZJGPaTM+qnOaO23MqSd0xQBK0NXPgvkrq4dpF8Wyctjr/zyEUpaspfoUnzdlizuRTziyBCR4nrNpwvUTIgfpZx63VYr7kU9LNecULNOPNFQrhPi6SpVfJGapfYiKwgWgKpFG6bS1SToJeIC4kiGA0EFY6r4yw/jKKV5ce4dV1XTz9Eobfz79v8= \ No newline at end of file diff --git a/Figures/FittingFlowNew.pdf b/Figures/FittingFlowNew.pdf new file mode 100644 index 0000000..1b17185 Binary files /dev/null and b/Figures/FittingFlowNew.pdf differ diff --git a/Figures/Pictures/Points-Mesh.png b/Figures/Pictures/Points-Mesh.png new file mode 100644 index 0000000..e17c049 Binary files /dev/null and b/Figures/Pictures/Points-Mesh.png differ diff --git a/Figures/Pictures/Screenshot 2019-10-03 11.02.42.png b/Figures/Pictures/Screenshot 2019-10-03 11.02.42.png new file mode 100644 index 0000000..9874e96 Binary files /dev/null and b/Figures/Pictures/Screenshot 2019-10-03 11.02.42.png differ diff --git a/Figures/Pictures/Screenshot 2019-10-03 12.17.07.png b/Figures/Pictures/Screenshot 2019-10-03 12.17.07.png new file mode 100644 index 0000000..7b78355 Binary files /dev/null and b/Figures/Pictures/Screenshot 2019-10-03 12.17.07.png differ diff --git a/Figures/Pictures/Screenshot 2019-10-03 12.17.41.png b/Figures/Pictures/Screenshot 2019-10-03 12.17.41.png new file mode 100644 index 0000000..07bb679 Binary files /dev/null and b/Figures/Pictures/Screenshot 2019-10-03 12.17.41.png differ diff --git a/Figures/Pictures/Screenshot 2019-10-03 12.21.14.png b/Figures/Pictures/Screenshot 2019-10-03 12.21.14.png new file mode 100644 index 0000000..269926e Binary files /dev/null and b/Figures/Pictures/Screenshot 2019-10-03 12.21.14.png differ diff --git a/Figures/Pictures/Screenshot from 2019-10-14 19-14-49.png b/Figures/Pictures/Screenshot from 2019-10-14 19-14-49.png new file mode 100644 index 0000000..12f6aa7 Binary files /dev/null and b/Figures/Pictures/Screenshot from 2019-10-14 19-14-49.png differ diff --git a/Figures/Pictures/TvsBF_contour.png b/Figures/Pictures/TvsBF_contour.png new file mode 100644 index 0000000..3bc7477 Binary files /dev/null and b/Figures/Pictures/TvsBF_contour.png differ diff --git a/Figures/Pictures/augModelLms.png b/Figures/Pictures/augModelLms.png new file mode 100644 index 0000000..4bdfc31 Binary files /dev/null and b/Figures/Pictures/augModelLms.png differ diff --git a/Figures/Pictures/augModelLmsBlur.png b/Figures/Pictures/augModelLmsBlur.png new file mode 100644 index 0000000..6c06cca Binary files /dev/null and b/Figures/Pictures/augModelLmsBlur.png differ diff --git a/Figures/Pictures/augModelLmsBlur_t.png b/Figures/Pictures/augModelLmsBlur_t.png new file mode 100644 index 0000000..e26fb24 Binary files /dev/null and b/Figures/Pictures/augModelLmsBlur_t.png differ diff --git a/Figures/Pictures/augModelLms_t.png b/Figures/Pictures/augModelLms_t.png new file mode 100644 index 0000000..6654631 Binary files /dev/null and b/Figures/Pictures/augModelLms_t.png differ diff --git a/Figures/Pictures/bf_big.png b/Figures/Pictures/bf_big.png new file mode 100644 index 0000000..53e7f1b Binary files /dev/null and b/Figures/Pictures/bf_big.png differ diff --git a/Figures/Pictures/bf_big1.png b/Figures/Pictures/bf_big1.png new file mode 100644 index 0000000..d2915bf Binary files /dev/null and b/Figures/Pictures/bf_big1.png differ diff --git a/Figures/Pictures/bf_big1_t.png b/Figures/Pictures/bf_big1_t.png new file mode 100644 index 0000000..99da43b Binary files /dev/null and b/Figures/Pictures/bf_big1_t.png differ diff --git a/Figures/Pictures/bf_big1_t.psd b/Figures/Pictures/bf_big1_t.psd new file mode 100644 index 0000000..4f6b3aa Binary files /dev/null and b/Figures/Pictures/bf_big1_t.psd differ diff --git a/Figures/Pictures/bf_big1_tnew.png b/Figures/Pictures/bf_big1_tnew.png new file mode 100644 index 0000000..36c9ddb Binary files /dev/null and b/Figures/Pictures/bf_big1_tnew.png differ diff --git a/Figures/Pictures/bf_big1_tnew1.png b/Figures/Pictures/bf_big1_tnew1.png new file mode 100644 index 0000000..89c3a9e Binary files /dev/null and b/Figures/Pictures/bf_big1_tnew1.png differ diff --git a/Figures/Pictures/bf_big_t.png b/Figures/Pictures/bf_big_t.png new file mode 100644 index 0000000..b9a5feb Binary files /dev/null and b/Figures/Pictures/bf_big_t.png differ diff --git a/Figures/Pictures/cal1.png b/Figures/Pictures/cal1.png new file mode 100644 index 0000000..48e278f Binary files /dev/null and b/Figures/Pictures/cal1.png differ diff --git a/Figures/Pictures/cal2.png b/Figures/Pictures/cal2.png new file mode 100644 index 0000000..d033c61 Binary files /dev/null and b/Figures/Pictures/cal2.png differ diff --git a/Figures/Pictures/clip.png b/Figures/Pictures/clip.png new file mode 100644 index 0000000..2ac80ed Binary files /dev/null and b/Figures/Pictures/clip.png differ diff --git a/Figures/Pictures/clip_t.png b/Figures/Pictures/clip_t.png new file mode 100644 index 0000000..73d62cc Binary files /dev/null and b/Figures/Pictures/clip_t.png differ diff --git a/Figures/Pictures/d01.png b/Figures/Pictures/d01.png new file mode 100644 index 0000000..16b6cff Binary files /dev/null and b/Figures/Pictures/d01.png differ diff --git a/Figures/Pictures/d01_t.png b/Figures/Pictures/d01_t.png new file mode 100644 index 0000000..be59035 Binary files /dev/null and b/Figures/Pictures/d01_t.png differ diff --git a/Figures/Pictures/d01a.png b/Figures/Pictures/d01a.png new file mode 100644 index 0000000..4be647f Binary files /dev/null and b/Figures/Pictures/d01a.png differ diff --git a/Figures/Pictures/d01a_t.png b/Figures/Pictures/d01a_t.png new file mode 100644 index 0000000..1f09919 Binary files /dev/null and b/Figures/Pictures/d01a_t.png differ diff --git a/Figures/Pictures/d06.png b/Figures/Pictures/d06.png new file mode 100644 index 0000000..cf0305f Binary files /dev/null and b/Figures/Pictures/d06.png differ diff --git a/Figures/Pictures/d06_t.png b/Figures/Pictures/d06_t.png new file mode 100644 index 0000000..0a5d742 Binary files /dev/null and b/Figures/Pictures/d06_t.png differ diff --git a/Figures/Pictures/d06a.png b/Figures/Pictures/d06a.png new file mode 100644 index 0000000..4397275 Binary files /dev/null and b/Figures/Pictures/d06a.png differ diff --git a/Figures/Pictures/d06a_t.png b/Figures/Pictures/d06a_t.png new file mode 100644 index 0000000..75524db Binary files /dev/null and b/Figures/Pictures/d06a_t.png differ diff --git a/Figures/Pictures/d09.png b/Figures/Pictures/d09.png new file mode 100644 index 0000000..df46253 Binary files /dev/null and b/Figures/Pictures/d09.png differ diff --git a/Figures/Pictures/d09_t.png b/Figures/Pictures/d09_t.png new file mode 100644 index 0000000..5605e5d Binary files /dev/null and b/Figures/Pictures/d09_t.png differ diff --git a/Figures/Pictures/d09a.png b/Figures/Pictures/d09a.png new file mode 100644 index 0000000..19ce43a Binary files /dev/null and b/Figures/Pictures/d09a.png differ diff --git a/Figures/Pictures/d09a_t.png b/Figures/Pictures/d09a_t.png new file mode 100644 index 0000000..4aa3e97 Binary files /dev/null and b/Figures/Pictures/d09a_t.png differ diff --git a/Figures/Pictures/dc.png b/Figures/Pictures/dc.png new file mode 100644 index 0000000..9eb4be2 Binary files /dev/null and b/Figures/Pictures/dc.png differ diff --git a/Figures/Pictures/depthquali.png b/Figures/Pictures/depthquali.png new file mode 100644 index 0000000..cf40cb9 Binary files /dev/null and b/Figures/Pictures/depthquali.png differ diff --git a/Figures/Pictures/face_bfm.png b/Figures/Pictures/face_bfm.png new file mode 100644 index 0000000..b041610 Binary files /dev/null and b/Figures/Pictures/face_bfm.png differ diff --git a/Figures/Pictures/face_bfm_close.png b/Figures/Pictures/face_bfm_close.png new file mode 100644 index 0000000..c9b2562 Binary files /dev/null and b/Figures/Pictures/face_bfm_close.png differ diff --git a/Figures/Pictures/face_bfm_close_t.png b/Figures/Pictures/face_bfm_close_t.png new file mode 100644 index 0000000..7962d1d Binary files /dev/null and b/Figures/Pictures/face_bfm_close_t.png differ diff --git a/Figures/Pictures/face_bfm_t.png b/Figures/Pictures/face_bfm_t.png new file mode 100644 index 0000000..aaeb959 Binary files /dev/null and b/Figures/Pictures/face_bfm_t.png differ diff --git a/Figures/Pictures/fit1.png b/Figures/Pictures/fit1.png new file mode 100644 index 0000000..5dcbfe0 Binary files /dev/null and b/Figures/Pictures/fit1.png differ diff --git a/Figures/Pictures/full_pc.png b/Figures/Pictures/full_pc.png new file mode 100644 index 0000000..a3334b1 Binary files /dev/null and b/Figures/Pictures/full_pc.png differ diff --git a/Figures/Pictures/full_pc_t.png b/Figures/Pictures/full_pc_t.png new file mode 100644 index 0000000..2d6eb8f Binary files /dev/null and b/Figures/Pictures/full_pc_t.png differ diff --git a/Figures/Pictures/full_with_overlay.png b/Figures/Pictures/full_with_overlay.png new file mode 100644 index 0000000..c484170 Binary files /dev/null and b/Figures/Pictures/full_with_overlay.png differ diff --git a/Figures/Pictures/full_with_overlay_t.png b/Figures/Pictures/full_with_overlay_t.png new file mode 100644 index 0000000..43475e1 Binary files /dev/null and b/Figures/Pictures/full_with_overlay_t.png differ diff --git a/Figures/Pictures/gtBig.png b/Figures/Pictures/gtBig.png new file mode 100644 index 0000000..be73e09 Binary files /dev/null and b/Figures/Pictures/gtBig.png differ diff --git a/Figures/Pictures/gtBig_t.png b/Figures/Pictures/gtBig_t.png new file mode 100644 index 0000000..e86893f Binary files /dev/null and b/Figures/Pictures/gtBig_t.png differ diff --git a/Figures/Pictures/gtBig_t.psd b/Figures/Pictures/gtBig_t.psd new file mode 100644 index 0000000..352b60b Binary files /dev/null and b/Figures/Pictures/gtBig_t.psd differ diff --git a/Figures/Pictures/gtBig_tnew.png b/Figures/Pictures/gtBig_tnew.png new file mode 100644 index 0000000..9d124eb Binary files /dev/null and b/Figures/Pictures/gtBig_tnew.png differ diff --git a/Figures/Pictures/gtX.png b/Figures/Pictures/gtX.png new file mode 100644 index 0000000..243aa96 Binary files /dev/null and b/Figures/Pictures/gtX.png differ diff --git a/Figures/Pictures/gtY.png b/Figures/Pictures/gtY.png new file mode 100644 index 0000000..7e14834 Binary files /dev/null and b/Figures/Pictures/gtY.png differ diff --git a/Figures/Pictures/gtZ.png b/Figures/Pictures/gtZ.png new file mode 100644 index 0000000..f745487 Binary files /dev/null and b/Figures/Pictures/gtZ.png differ diff --git a/Figures/Pictures/gyY.png b/Figures/Pictures/gyY.png new file mode 100644 index 0000000..505d1d3 Binary files /dev/null and b/Figures/Pictures/gyY.png differ diff --git a/Figures/Pictures/ic.png b/Figures/Pictures/ic.png new file mode 100644 index 0000000..9421945 Binary files /dev/null and b/Figures/Pictures/ic.png differ diff --git a/Figures/Pictures/initF.png b/Figures/Pictures/initF.png new file mode 100644 index 0000000..19c6ea8 Binary files /dev/null and b/Figures/Pictures/initF.png differ diff --git a/Figures/Pictures/initFAlt.png b/Figures/Pictures/initFAlt.png new file mode 100644 index 0000000..d7314ea Binary files /dev/null and b/Figures/Pictures/initFAlt.png differ diff --git a/Figures/Pictures/initFAlt_t.png b/Figures/Pictures/initFAlt_t.png new file mode 100644 index 0000000..78e8415 Binary files /dev/null and b/Figures/Pictures/initFAlt_t.png differ diff --git a/Figures/Pictures/initF_t.png b/Figures/Pictures/initF_t.png new file mode 100644 index 0000000..cb4d9cf Binary files /dev/null and b/Figures/Pictures/initF_t.png differ diff --git a/Figures/Pictures/initL.png b/Figures/Pictures/initL.png new file mode 100644 index 0000000..6183bfd Binary files /dev/null and b/Figures/Pictures/initL.png differ diff --git a/Figures/Pictures/initLAlt.png b/Figures/Pictures/initLAlt.png new file mode 100644 index 0000000..0ae90ca Binary files /dev/null and b/Figures/Pictures/initLAlt.png differ diff --git a/Figures/Pictures/initLAlt_t.png b/Figures/Pictures/initLAlt_t.png new file mode 100644 index 0000000..e15267a Binary files /dev/null and b/Figures/Pictures/initLAlt_t.png differ diff --git a/Figures/Pictures/initL_t.png b/Figures/Pictures/initL_t.png new file mode 100644 index 0000000..d3ca52a Binary files /dev/null and b/Figures/Pictures/initL_t.png differ diff --git a/Figures/Pictures/initR.png b/Figures/Pictures/initR.png new file mode 100644 index 0000000..0dfe86b Binary files /dev/null and b/Figures/Pictures/initR.png differ diff --git a/Figures/Pictures/initRAlt.png b/Figures/Pictures/initRAlt.png new file mode 100644 index 0000000..0c1361a Binary files /dev/null and b/Figures/Pictures/initRAlt.png differ diff --git a/Figures/Pictures/initRAlt_t.png b/Figures/Pictures/initRAlt_t.png new file mode 100644 index 0000000..3d94a98 Binary files /dev/null and b/Figures/Pictures/initRAlt_t.png differ diff --git a/Figures/Pictures/initR_t.png b/Figures/Pictures/initR_t.png new file mode 100644 index 0000000..84a4546 Binary files /dev/null and b/Figures/Pictures/initR_t.png differ diff --git a/Figures/Pictures/lmdetect.png b/Figures/Pictures/lmdetect.png new file mode 100644 index 0000000..9f2119c Binary files /dev/null and b/Figures/Pictures/lmdetect.png differ diff --git a/Figures/Pictures/lmoverlay.png b/Figures/Pictures/lmoverlay.png new file mode 100644 index 0000000..d47d74a Binary files /dev/null and b/Figures/Pictures/lmoverlay.png differ diff --git a/Figures/Pictures/lms.png b/Figures/Pictures/lms.png new file mode 100644 index 0000000..8a5e494 Binary files /dev/null and b/Figures/Pictures/lms.png differ diff --git a/Figures/Pictures/mean.png b/Figures/Pictures/mean.png new file mode 100644 index 0000000..29c6112 Binary files /dev/null and b/Figures/Pictures/mean.png differ diff --git a/Figures/Pictures/mean_t.png b/Figures/Pictures/mean_t.png new file mode 100644 index 0000000..ff203e8 Binary files /dev/null and b/Figures/Pictures/mean_t.png differ diff --git a/Figures/Pictures/mesh.png b/Figures/Pictures/mesh.png new file mode 100644 index 0000000..5d0f549 Binary files /dev/null and b/Figures/Pictures/mesh.png differ diff --git a/Figures/Pictures/mesh1.png b/Figures/Pictures/mesh1.png new file mode 100644 index 0000000..22c54b1 Binary files /dev/null and b/Figures/Pictures/mesh1.png differ diff --git a/Figures/Pictures/mesh_f.png b/Figures/Pictures/mesh_f.png new file mode 100644 index 0000000..fc9857a Binary files /dev/null and b/Figures/Pictures/mesh_f.png differ diff --git a/Figures/Pictures/mesh_f_t.png b/Figures/Pictures/mesh_f_t.png new file mode 100644 index 0000000..4ae73b7 Binary files /dev/null and b/Figures/Pictures/mesh_f_t.png differ diff --git a/Figures/Pictures/mesh_l.png b/Figures/Pictures/mesh_l.png new file mode 100644 index 0000000..abf29b0 Binary files /dev/null and b/Figures/Pictures/mesh_l.png differ diff --git a/Figures/Pictures/mesh_l_t.png b/Figures/Pictures/mesh_l_t.png new file mode 100644 index 0000000..fbbc842 Binary files /dev/null and b/Figures/Pictures/mesh_l_t.png differ diff --git a/Figures/Pictures/mesh_r.png b/Figures/Pictures/mesh_r.png new file mode 100644 index 0000000..9bc0884 Binary files /dev/null and b/Figures/Pictures/mesh_r.png differ diff --git a/Figures/Pictures/mesh_r_t.png b/Figures/Pictures/mesh_r_t.png new file mode 100644 index 0000000..af08b5b Binary files /dev/null and b/Figures/Pictures/mesh_r_t.png differ diff --git a/Figures/Pictures/modelTargetDist.png b/Figures/Pictures/modelTargetDist.png new file mode 100644 index 0000000..6baadd2 Binary files /dev/null and b/Figures/Pictures/modelTargetDist.png differ diff --git a/Figures/Pictures/modelTargetDist_t.png b/Figures/Pictures/modelTargetDist_t.png new file mode 100644 index 0000000..5d1863a Binary files /dev/null and b/Figures/Pictures/modelTargetDist_t.png differ diff --git a/Figures/Pictures/multicam.jpg b/Figures/Pictures/multicam.jpg new file mode 100644 index 0000000..eafa98b Binary files /dev/null and b/Figures/Pictures/multicam.jpg differ diff --git a/Figures/Pictures/no-glasses.png b/Figures/Pictures/no-glasses.png new file mode 100644 index 0000000..88039f6 Binary files /dev/null and b/Figures/Pictures/no-glasses.png differ diff --git a/Figures/Pictures/notgoodpic.png b/Figures/Pictures/notgoodpic.png new file mode 100644 index 0000000..a7084a3 Binary files /dev/null and b/Figures/Pictures/notgoodpic.png differ diff --git a/Figures/Pictures/origGreen.png b/Figures/Pictures/origGreen.png new file mode 100644 index 0000000..81a446e Binary files /dev/null and b/Figures/Pictures/origGreen.png differ diff --git a/Figures/Pictures/origGreen_t.png b/Figures/Pictures/origGreen_t.png new file mode 100644 index 0000000..a625e2d Binary files /dev/null and b/Figures/Pictures/origGreen_t.png differ diff --git a/Figures/Pictures/origModelLms.png b/Figures/Pictures/origModelLms.png new file mode 100644 index 0000000..a497823 Binary files /dev/null and b/Figures/Pictures/origModelLms.png differ diff --git a/Figures/Pictures/origModelLmsBlur.png b/Figures/Pictures/origModelLmsBlur.png new file mode 100644 index 0000000..227cc46 Binary files /dev/null and b/Figures/Pictures/origModelLmsBlur.png differ diff --git a/Figures/Pictures/origModelLmsBlur_t.png b/Figures/Pictures/origModelLmsBlur_t.png new file mode 100644 index 0000000..9895b1c Binary files /dev/null and b/Figures/Pictures/origModelLmsBlur_t.png differ diff --git a/Figures/Pictures/origModelLms_t.png b/Figures/Pictures/origModelLms_t.png new file mode 100644 index 0000000..9ff4237 Binary files /dev/null and b/Figures/Pictures/origModelLms_t.png differ diff --git a/Figures/Pictures/pinhole_model.png b/Figures/Pictures/pinhole_model.png new file mode 100644 index 0000000..f5a355c Binary files /dev/null and b/Figures/Pictures/pinhole_model.png differ diff --git a/Figures/Pictures/pipelineGreen.png b/Figures/Pictures/pipelineGreen.png new file mode 100644 index 0000000..2f0e4ee Binary files /dev/null and b/Figures/Pictures/pipelineGreen.png differ diff --git a/Figures/Pictures/pipelineGreen_t.png b/Figures/Pictures/pipelineGreen_t.png new file mode 100644 index 0000000..19e9b79 Binary files /dev/null and b/Figures/Pictures/pipelineGreen_t.png differ diff --git a/Figures/Pictures/pipelineLms.png b/Figures/Pictures/pipelineLms.png new file mode 100644 index 0000000..2620291 Binary files /dev/null and b/Figures/Pictures/pipelineLms.png differ diff --git a/Figures/Pictures/pipelineLmsBlured.png b/Figures/Pictures/pipelineLmsBlured.png new file mode 100644 index 0000000..aca1017 Binary files /dev/null and b/Figures/Pictures/pipelineLmsBlured.png differ diff --git a/Figures/Pictures/pipelineLmsBlured_t.png b/Figures/Pictures/pipelineLmsBlured_t.png new file mode 100644 index 0000000..611d073 Binary files /dev/null and b/Figures/Pictures/pipelineLmsBlured_t.png differ diff --git a/Figures/Pictures/pipelineLms_t.png b/Figures/Pictures/pipelineLms_t.png new file mode 100644 index 0000000..3e1d2c4 Binary files /dev/null and b/Figures/Pictures/pipelineLms_t.png differ diff --git a/Figures/Pictures/pts.png b/Figures/Pictures/pts.png new file mode 100644 index 0000000..1a37c07 Binary files /dev/null and b/Figures/Pictures/pts.png differ diff --git a/Figures/Pictures/pts1.png b/Figures/Pictures/pts1.png new file mode 100644 index 0000000..0b74160 Binary files /dev/null and b/Figures/Pictures/pts1.png differ diff --git a/Figures/Pictures/rs1.png b/Figures/Pictures/rs1.png new file mode 100644 index 0000000..1fe9f0d Binary files /dev/null and b/Figures/Pictures/rs1.png differ diff --git a/Figures/Pictures/rs1_t.png b/Figures/Pictures/rs1_t.png new file mode 100644 index 0000000..af1ced5 Binary files /dev/null and b/Figures/Pictures/rs1_t.png differ diff --git a/Figures/Pictures/rs2_expr.png b/Figures/Pictures/rs2_expr.png new file mode 100644 index 0000000..84ec340 Binary files /dev/null and b/Figures/Pictures/rs2_expr.png differ diff --git a/Figures/Pictures/rs2_expr_t.png b/Figures/Pictures/rs2_expr_t.png new file mode 100644 index 0000000..69d019b Binary files /dev/null and b/Figures/Pictures/rs2_expr_t.png differ diff --git a/Figures/Pictures/rsviewer.png b/Figures/Pictures/rsviewer.png new file mode 100644 index 0000000..882cd69 Binary files /dev/null and b/Figures/Pictures/rsviewer.png differ diff --git a/Figures/Pictures/sfX.png b/Figures/Pictures/sfX.png new file mode 100644 index 0000000..3330bde Binary files /dev/null and b/Figures/Pictures/sfX.png differ diff --git a/Figures/Pictures/sfY.png b/Figures/Pictures/sfY.png new file mode 100644 index 0000000..96a9824 Binary files /dev/null and b/Figures/Pictures/sfY.png differ diff --git a/Figures/Pictures/sfZ.png b/Figures/Pictures/sfZ.png new file mode 100644 index 0000000..a8b634f Binary files /dev/null and b/Figures/Pictures/sfZ.png differ diff --git a/Figures/Pictures/sff.png b/Figures/Pictures/sff.png new file mode 100644 index 0000000..e9ad320 Binary files /dev/null and b/Figures/Pictures/sff.png differ diff --git a/Figures/Pictures/sff_t.png b/Figures/Pictures/sff_t.png new file mode 100644 index 0000000..d7a2d10 Binary files /dev/null and b/Figures/Pictures/sff_t.png differ diff --git a/Figures/Pictures/sideA.png b/Figures/Pictures/sideA.png new file mode 100644 index 0000000..6ce2c70 Binary files /dev/null and b/Figures/Pictures/sideA.png differ diff --git a/Figures/Pictures/sideAA.png b/Figures/Pictures/sideAA.png new file mode 100644 index 0000000..4fc80f6 Binary files /dev/null and b/Figures/Pictures/sideAA.png differ diff --git a/Figures/Pictures/sideA_t.png b/Figures/Pictures/sideA_t.png new file mode 100644 index 0000000..282ea5c Binary files /dev/null and b/Figures/Pictures/sideA_t.png differ diff --git a/Figures/Pictures/sideB.png b/Figures/Pictures/sideB.png new file mode 100644 index 0000000..8b489da Binary files /dev/null and b/Figures/Pictures/sideB.png differ diff --git a/Figures/Pictures/sideB_t.png b/Figures/Pictures/sideB_t.png new file mode 100644 index 0000000..4a4c230 Binary files /dev/null and b/Figures/Pictures/sideB_t.png differ diff --git a/Figures/Pictures/stereo-ssd-1.png b/Figures/Pictures/stereo-ssd-1.png new file mode 100644 index 0000000..6db952a Binary files /dev/null and b/Figures/Pictures/stereo-ssd-1.png differ diff --git a/Figures/Pictures/targetMeshWithLandmarks.png b/Figures/Pictures/targetMeshWithLandmarks.png new file mode 100644 index 0000000..b4069bf Binary files /dev/null and b/Figures/Pictures/targetMeshWithLandmarks.png differ diff --git a/Figures/Pictures/targetMeshWithLandmarks_t.png b/Figures/Pictures/targetMeshWithLandmarks_t.png new file mode 100644 index 0000000..3ab933b Binary files /dev/null and b/Figures/Pictures/targetMeshWithLandmarks_t.png differ diff --git a/Figures/Pictures/web-fitting (1).pdf b/Figures/Pictures/web-fitting (1).pdf new file mode 100644 index 0000000..bea8ccd Binary files /dev/null and b/Figures/Pictures/web-fitting (1).pdf differ diff --git a/Figures/Server.drawio b/Figures/Server.drawio new file mode 100644 index 0000000..8e1d7ca --- /dev/null +++ b/Figures/Server.drawio @@ -0,0 +1 @@ +7Vtbd6o4FP41PtoFCSA+Vm3PzJxLz6qzZqaPCFFpkXBCrDq/fhJIuMZWK4h1jS+SnQuwv335dow9OF5tvxAnWn7HHgp6QPO2PTjpAaBrtsm+uGQnJRpMJQvie0KWC6b+v0gOFNK176G4NJBiHFA/KgtdHIbIpSWZQwjelIfNcVC+a+QsUE0wdZ2gLv3b9+hSSNl75B2/IX+xFLe2TdExc9yXBcHrUNyvB+A8+aTdK0euJcbHS8fDm4II3vXgmGBM06vVdowCrlyptnTe/Z7e7LkJCukhE378NYn954en/uQrmSzuo9kI2v3hUDwc3UmFII/pRzQxoUu8wKET3OXSUfLSiC+rsVY+5hvGERPqTPiMKN0JsJ01xUy0pKtA9KKtT//h029M0Xoq9Ey2YuWksZONUBqPzifFlOCXDDGDSer6ECqK8Zq46A0liHH8xQsThRa/ILxClOzYAIICh/qvZcNxhP0tsnFi6i0hzq4wIMJ+SOPCyj+5gA0QvgQs4UrCkyxYwpNdpCuqZxuGppyd3Z46ZIGomFVatvCOuSgxniMMSWjj1QnWSPpDxbBys+FIb5Y+RdPISZDZsOBSNpG5HwRjHGCSzIWaNh5rWoZ7oQdMBlbSM8chLciFL8LRgjiej9R9e23mFRGKtm9aQxYCS3rXpbtv8mBiyjHLQhwxLG2/AZXQORoKoxOfDtmzF5yaN58y32WN3K2T1ul+nVr0p/Nr+xS3titGk8a21rza0jsxJZEe9EJyyFPFvvTQgvmBk9KKcaCZGvAy7HRQNjWgmWU+UTVN+Ob4smE3ZZCGIs1YAdPuKCqZqfVrzXnVKPBD1Jdh95YN4S9p5v3saiG+k1VmpCphjxnlsoozME4X8cuIYBfF8ftpLSeND2vKH06d7ua2i1xXle5mtmmYWkOpy6zkLrueu7J8VspdraUurYbvI/q1RjGtqZ69I60pKMRcoyVtClGRWjCiIHweGLItFub9XIE+KxBuA38RMtnK97wksDlC4DKVI9IMBlBTB/giBNpZIagzuUcURziM0bVikCldYGAOuwbBsLrMu42VZaflTxl53k2goHGidxJ4oB7Ezpc45nOgThyeNbNMq53EAVQOo0ocg7YcBhxQf7alc89EtmeodG6DGbQa0rmkV1LnBhx2rHNo7SVjPJwr+Rjv6MdJwOBsTLeibZ2M/b7ie3Y5/0pX20PBTtlVcD00s2cq5KAFh9BriWYNzMOQA60hN2gJuaREGAd47Z0DPr6JY9vnhg9oBwa71uCzumUHH6nKsx2iwqyOqnJ4IKsw7ItiFfK5u2EVrjsctu1o0CxnOGAqHM1UOFrGzZtXeluB8spT3MGEsLUYKb33/xT3Afig3jVDMYYtwffNCb2VQ17i86B3jrhZQ8/o2vlMVQV8JHq2Cr2xE9E1OThwXsL+lOcT5FIfh8l9CAeEcTHqCFFTW7mVzXtdYQO2wgRa28EyB51y1KN2sFr45Ug/iaNKIvQuRzWti+Ko8rkbj9oTRPnZH6D99LfJAaTzhvHhfMCnXij9bc+H29rgmaA+K1GeU0iZDzNcxY+T1womBOBGwalUcFYP7jQHZ1uU+M8l8ef0ykqaD++3tseq2uLEKX7fUby8npKmBp9lnA++r+HrcDBC0/H9H+hxR9EPeLvJNl2PYkTumrxmev5U9KjJPTt5VO5dPmSDi+JDVgN10Bsee4W1bM1t7a6jrrX/yNGJtSwOuV4kA0ohZRcPM06MDsZU1Lit1K6KGrkBjHVtcFNhR1ad6mY86CxU12qgclGiLH2UDUiLGF72v3XY7OI3NJpgx9VD0wNFqXPWAzcD1a8rx+EPNRX+U8QSOPlM+BY2p/rDxranqh4PFNXQWRGXVEGBeBw54cc9fopCr4B3utjVBG+gl4E0VbsUKiA/UNayZv73qPQEcf4nNHj3Hw== \ No newline at end of file diff --git a/Figures/Server.pdf b/Figures/Server.pdf new file mode 100644 index 0000000..e273946 Binary files /dev/null and b/Figures/Server.pdf differ diff --git a/Figures/ShapeFitting.pdf b/Figures/ShapeFitting.pdf new file mode 100644 index 0000000..2bbff42 Binary files /dev/null and b/Figures/ShapeFitting.pdf differ diff --git a/Figures/cam.png b/Figures/cam.png new file mode 100644 index 0000000..66e1bfc Binary files /dev/null and b/Figures/cam.png differ diff --git a/Figures/dataset/2D/1.png b/Figures/dataset/2D/1.png new file mode 100644 index 0000000..358e853 Binary files /dev/null and b/Figures/dataset/2D/1.png differ diff --git a/Figures/dataset/2D/2.png b/Figures/dataset/2D/2.png new file mode 100644 index 0000000..a859f75 Binary files /dev/null and b/Figures/dataset/2D/2.png differ diff --git a/Figures/dataset/2D/3.png b/Figures/dataset/2D/3.png new file mode 100644 index 0000000..f78c136 Binary files /dev/null and b/Figures/dataset/2D/3.png differ diff --git a/Figures/dataset/2D/4.png b/Figures/dataset/2D/4.png new file mode 100644 index 0000000..21c5812 Binary files /dev/null and b/Figures/dataset/2D/4.png differ diff --git a/Figures/dataset/2D/5.png b/Figures/dataset/2D/5.png new file mode 100644 index 0000000..13bbfdc Binary files /dev/null and b/Figures/dataset/2D/5.png differ diff --git a/Figures/dataset/2D/6.png b/Figures/dataset/2D/6.png new file mode 100644 index 0000000..5721711 Binary files /dev/null and b/Figures/dataset/2D/6.png differ diff --git a/Figures/dataset/2D/7.png b/Figures/dataset/2D/7.png new file mode 100644 index 0000000..af7df09 Binary files /dev/null and b/Figures/dataset/2D/7.png differ diff --git a/Figures/dataset/blended/1.png b/Figures/dataset/blended/1.png new file mode 100644 index 0000000..4359fd0 Binary files /dev/null and b/Figures/dataset/blended/1.png differ diff --git a/Figures/dataset/blended/2.png b/Figures/dataset/blended/2.png new file mode 100644 index 0000000..e8753b8 Binary files /dev/null and b/Figures/dataset/blended/2.png differ diff --git a/Figures/dataset/blended/3.png b/Figures/dataset/blended/3.png new file mode 100644 index 0000000..b9b5c38 Binary files /dev/null and b/Figures/dataset/blended/3.png differ diff --git a/Figures/dataset/blended/4.png b/Figures/dataset/blended/4.png new file mode 100644 index 0000000..ebd64b0 Binary files /dev/null and b/Figures/dataset/blended/4.png differ diff --git a/Figures/dataset/blended/5.png b/Figures/dataset/blended/5.png new file mode 100644 index 0000000..e22fc91 Binary files /dev/null and b/Figures/dataset/blended/5.png differ diff --git a/Figures/dataset/blended/6.png b/Figures/dataset/blended/6.png new file mode 100644 index 0000000..9676c1d Binary files /dev/null and b/Figures/dataset/blended/6.png differ diff --git a/Figures/dataset/blended/7.png b/Figures/dataset/blended/7.png new file mode 100644 index 0000000..7e8f707 Binary files /dev/null and b/Figures/dataset/blended/7.png differ diff --git a/Figures/dataset/our/1.png b/Figures/dataset/our/1.png new file mode 100644 index 0000000..c75db34 Binary files /dev/null and b/Figures/dataset/our/1.png differ diff --git a/Figures/dataset/our/1blended.png b/Figures/dataset/our/1blended.png new file mode 100644 index 0000000..3691f3e Binary files /dev/null and b/Figures/dataset/our/1blended.png differ diff --git a/Figures/dataset/our/1old.png b/Figures/dataset/our/1old.png new file mode 100644 index 0000000..74c6a4c Binary files /dev/null and b/Figures/dataset/our/1old.png differ diff --git a/Figures/dataset/our/3.png b/Figures/dataset/our/3.png new file mode 100644 index 0000000..4633aa2 Binary files /dev/null and b/Figures/dataset/our/3.png differ diff --git a/Figures/dataset/our/3blended.png b/Figures/dataset/our/3blended.png new file mode 100644 index 0000000..118128b Binary files /dev/null and b/Figures/dataset/our/3blended.png differ diff --git a/Figures/dataset/our/3old.png b/Figures/dataset/our/3old.png new file mode 100644 index 0000000..35a48e5 Binary files /dev/null and b/Figures/dataset/our/3old.png differ diff --git a/Figures/dataset/our/4.png b/Figures/dataset/our/4.png new file mode 100644 index 0000000..5d925b2 Binary files /dev/null and b/Figures/dataset/our/4.png differ diff --git a/Figures/dataset/our/4alt.png b/Figures/dataset/our/4alt.png new file mode 100644 index 0000000..5f07696 Binary files /dev/null and b/Figures/dataset/our/4alt.png differ diff --git a/Figures/dataset/our/4blended.png b/Figures/dataset/our/4blended.png new file mode 100644 index 0000000..04ddfe9 Binary files /dev/null and b/Figures/dataset/our/4blended.png differ diff --git a/Figures/dataset/our/4old.png b/Figures/dataset/our/4old.png new file mode 100644 index 0000000..6b04bb7 Binary files /dev/null and b/Figures/dataset/our/4old.png differ diff --git a/Figures/dataset/our/5.png b/Figures/dataset/our/5.png new file mode 100644 index 0000000..d013d21 Binary files /dev/null and b/Figures/dataset/our/5.png differ diff --git a/Figures/dataset/our/5blended.png b/Figures/dataset/our/5blended.png new file mode 100644 index 0000000..d1bbdfa Binary files /dev/null and b/Figures/dataset/our/5blended.png differ diff --git a/Figures/dataset/our/5old.png b/Figures/dataset/our/5old.png new file mode 100644 index 0000000..dc31e21 Binary files /dev/null and b/Figures/dataset/our/5old.png differ diff --git a/Figures/dataset/our/5overlay.png b/Figures/dataset/our/5overlay.png new file mode 100644 index 0000000..4387c47 Binary files /dev/null and b/Figures/dataset/our/5overlay.png differ diff --git a/Figures/dataset/our/6.png b/Figures/dataset/our/6.png new file mode 100644 index 0000000..b98678a Binary files /dev/null and b/Figures/dataset/our/6.png differ diff --git a/Figures/dataset/our/6blended.png b/Figures/dataset/our/6blended.png new file mode 100644 index 0000000..829b554 Binary files /dev/null and b/Figures/dataset/our/6blended.png differ diff --git a/Figures/dataset/our/7.png b/Figures/dataset/our/7.png new file mode 100644 index 0000000..674966c Binary files /dev/null and b/Figures/dataset/our/7.png differ diff --git a/Figures/dataset/our/7blended.png b/Figures/dataset/our/7blended.png new file mode 100644 index 0000000..bcf9bfc Binary files /dev/null and b/Figures/dataset/our/7blended.png differ diff --git a/Figures/dataset/target/1.png b/Figures/dataset/target/1.png new file mode 100644 index 0000000..61c65ef Binary files /dev/null and b/Figures/dataset/target/1.png differ diff --git a/Figures/dataset/target/2.png b/Figures/dataset/target/2.png new file mode 100644 index 0000000..a012a4a Binary files /dev/null and b/Figures/dataset/target/2.png differ diff --git a/Figures/dataset/target/3.png b/Figures/dataset/target/3.png new file mode 100644 index 0000000..4cba842 Binary files /dev/null and b/Figures/dataset/target/3.png differ diff --git a/Figures/dataset/target/4.png b/Figures/dataset/target/4.png new file mode 100644 index 0000000..19c9f54 Binary files /dev/null and b/Figures/dataset/target/4.png differ diff --git a/Figures/dataset/target/5.png b/Figures/dataset/target/5.png new file mode 100644 index 0000000..cbf4adf Binary files /dev/null and b/Figures/dataset/target/5.png differ diff --git a/Figures/dataset/target/6.png b/Figures/dataset/target/6.png new file mode 100644 index 0000000..d489925 Binary files /dev/null and b/Figures/dataset/target/6.png differ diff --git a/Figures/dataset/target/7.png b/Figures/dataset/target/7.png new file mode 100644 index 0000000..d02280b Binary files /dev/null and b/Figures/dataset/target/7.png differ diff --git a/Figures/eval/2d/19-52-06/front-box.png b/Figures/eval/2d/19-52-06/front-box.png new file mode 100644 index 0000000..5d43320 Binary files /dev/null and b/Figures/eval/2d/19-52-06/front-box.png differ diff --git a/Figures/eval/2d/19-52-06/front-gt-box.png b/Figures/eval/2d/19-52-06/front-gt-box.png new file mode 100644 index 0000000..73be53a Binary files /dev/null and b/Figures/eval/2d/19-52-06/front-gt-box.png differ diff --git a/Figures/eval/2d/19-52-06/front-gt.png b/Figures/eval/2d/19-52-06/front-gt.png new file mode 100644 index 0000000..64af052 Binary files /dev/null and b/Figures/eval/2d/19-52-06/front-gt.png differ diff --git a/Figures/eval/2d/19-52-06/front.png b/Figures/eval/2d/19-52-06/front.png new file mode 100644 index 0000000..f5c2ac6 Binary files /dev/null and b/Figures/eval/2d/19-52-06/front.png differ diff --git a/Figures/eval/2d/19-52-06/x-gt.png b/Figures/eval/2d/19-52-06/x-gt.png new file mode 100644 index 0000000..113980a Binary files /dev/null and b/Figures/eval/2d/19-52-06/x-gt.png differ diff --git a/Figures/eval/2d/19-52-06/x.png b/Figures/eval/2d/19-52-06/x.png new file mode 100644 index 0000000..7ada812 Binary files /dev/null and b/Figures/eval/2d/19-52-06/x.png differ diff --git a/Figures/eval/2d/19-52-06/y-gt.png b/Figures/eval/2d/19-52-06/y-gt.png new file mode 100644 index 0000000..0cf4e89 Binary files /dev/null and b/Figures/eval/2d/19-52-06/y-gt.png differ diff --git a/Figures/eval/2d/19-52-06/y.png b/Figures/eval/2d/19-52-06/y.png new file mode 100644 index 0000000..303a930 Binary files /dev/null and b/Figures/eval/2d/19-52-06/y.png differ diff --git a/Figures/eval/2d/19-52-06/z-gt.png b/Figures/eval/2d/19-52-06/z-gt.png new file mode 100644 index 0000000..cc28d21 Binary files /dev/null and b/Figures/eval/2d/19-52-06/z-gt.png differ diff --git a/Figures/eval/2d/19-52-06/z.png b/Figures/eval/2d/19-52-06/z.png new file mode 100644 index 0000000..2c2365f Binary files /dev/null and b/Figures/eval/2d/19-52-06/z.png differ diff --git a/Figures/eval/2d/20-05-23/front-box-gt.png b/Figures/eval/2d/20-05-23/front-box-gt.png new file mode 100644 index 0000000..bd07ada Binary files /dev/null and b/Figures/eval/2d/20-05-23/front-box-gt.png differ diff --git a/Figures/eval/2d/20-05-23/front-box.png b/Figures/eval/2d/20-05-23/front-box.png new file mode 100644 index 0000000..be8d802 Binary files /dev/null and b/Figures/eval/2d/20-05-23/front-box.png differ diff --git a/Figures/eval/2d/20-05-23/front-gt.png b/Figures/eval/2d/20-05-23/front-gt.png new file mode 100644 index 0000000..a6b8cb1 Binary files /dev/null and b/Figures/eval/2d/20-05-23/front-gt.png differ diff --git a/Figures/eval/2d/20-05-23/front.png b/Figures/eval/2d/20-05-23/front.png new file mode 100644 index 0000000..1346efd Binary files /dev/null and b/Figures/eval/2d/20-05-23/front.png differ diff --git a/Figures/eval/2d/20-05-23/x-gt.png b/Figures/eval/2d/20-05-23/x-gt.png new file mode 100644 index 0000000..63d8206 Binary files /dev/null and b/Figures/eval/2d/20-05-23/x-gt.png differ diff --git a/Figures/eval/2d/20-05-23/x.png b/Figures/eval/2d/20-05-23/x.png new file mode 100644 index 0000000..aa95cff Binary files /dev/null and b/Figures/eval/2d/20-05-23/x.png differ diff --git a/Figures/eval/2d/20-05-23/y-gt.png b/Figures/eval/2d/20-05-23/y-gt.png new file mode 100644 index 0000000..932b426 Binary files /dev/null and b/Figures/eval/2d/20-05-23/y-gt.png differ diff --git a/Figures/eval/2d/20-05-23/y.png b/Figures/eval/2d/20-05-23/y.png new file mode 100644 index 0000000..1ad86da Binary files /dev/null and b/Figures/eval/2d/20-05-23/y.png differ diff --git a/Figures/eval/2d/20-05-23/z.png b/Figures/eval/2d/20-05-23/z.png new file mode 100644 index 0000000..01839dd Binary files /dev/null and b/Figures/eval/2d/20-05-23/z.png differ diff --git a/Figures/eval/2d/20-14-11/front-box-gt.png b/Figures/eval/2d/20-14-11/front-box-gt.png new file mode 100644 index 0000000..4de1ca7 Binary files /dev/null and b/Figures/eval/2d/20-14-11/front-box-gt.png differ diff --git a/Figures/eval/2d/20-14-11/front-box.png b/Figures/eval/2d/20-14-11/front-box.png new file mode 100644 index 0000000..f6c7fda Binary files /dev/null and b/Figures/eval/2d/20-14-11/front-box.png differ diff --git a/Figures/eval/2d/20-14-11/front-gt.png b/Figures/eval/2d/20-14-11/front-gt.png new file mode 100644 index 0000000..f32fb1c Binary files /dev/null and b/Figures/eval/2d/20-14-11/front-gt.png differ diff --git a/Figures/eval/2d/20-14-11/front.png b/Figures/eval/2d/20-14-11/front.png new file mode 100644 index 0000000..2c5bfb4 Binary files /dev/null and b/Figures/eval/2d/20-14-11/front.png differ diff --git a/Figures/eval/2d/20-14-11/x-gt.png b/Figures/eval/2d/20-14-11/x-gt.png new file mode 100644 index 0000000..2ef8fb3 Binary files /dev/null and b/Figures/eval/2d/20-14-11/x-gt.png differ diff --git a/Figures/eval/2d/20-14-11/x.png b/Figures/eval/2d/20-14-11/x.png new file mode 100644 index 0000000..1e040cc Binary files /dev/null and b/Figures/eval/2d/20-14-11/x.png differ diff --git a/Figures/eval/2d/20-14-11/y-gt.png b/Figures/eval/2d/20-14-11/y-gt.png new file mode 100644 index 0000000..a092000 Binary files /dev/null and b/Figures/eval/2d/20-14-11/y-gt.png differ diff --git a/Figures/eval/2d/20-14-11/y.png b/Figures/eval/2d/20-14-11/y.png new file mode 100644 index 0000000..da5fbbf Binary files /dev/null and b/Figures/eval/2d/20-14-11/y.png differ diff --git a/Figures/eval/2d/20-14-11/z-gt.png b/Figures/eval/2d/20-14-11/z-gt.png new file mode 100644 index 0000000..fe16d11 Binary files /dev/null and b/Figures/eval/2d/20-14-11/z-gt.png differ diff --git a/Figures/eval/2d/20-14-11/z.png b/Figures/eval/2d/20-14-11/z.png new file mode 100644 index 0000000..4b63c73 Binary files /dev/null and b/Figures/eval/2d/20-14-11/z.png differ diff --git a/Figures/eval/2d/20-19-03/front-box-gt.png b/Figures/eval/2d/20-19-03/front-box-gt.png new file mode 100644 index 0000000..2b0567a Binary files /dev/null and b/Figures/eval/2d/20-19-03/front-box-gt.png differ diff --git a/Figures/eval/2d/20-19-03/front-gt.png b/Figures/eval/2d/20-19-03/front-gt.png new file mode 100644 index 0000000..21d2a52 Binary files /dev/null and b/Figures/eval/2d/20-19-03/front-gt.png differ diff --git a/Figures/eval/2d/20-19-03/front-nobox.png b/Figures/eval/2d/20-19-03/front-nobox.png new file mode 100644 index 0000000..77338bb Binary files /dev/null and b/Figures/eval/2d/20-19-03/front-nobox.png differ diff --git a/Figures/eval/2d/20-19-03/front.png b/Figures/eval/2d/20-19-03/front.png new file mode 100644 index 0000000..667a673 Binary files /dev/null and b/Figures/eval/2d/20-19-03/front.png differ diff --git a/Figures/eval/2d/20-19-03/x-gt.png b/Figures/eval/2d/20-19-03/x-gt.png new file mode 100644 index 0000000..bc68246 Binary files /dev/null and b/Figures/eval/2d/20-19-03/x-gt.png differ diff --git a/Figures/eval/2d/20-19-03/x.png b/Figures/eval/2d/20-19-03/x.png new file mode 100644 index 0000000..c1f854a Binary files /dev/null and b/Figures/eval/2d/20-19-03/x.png differ diff --git a/Figures/eval/2d/20-19-03/y-gt.png b/Figures/eval/2d/20-19-03/y-gt.png new file mode 100644 index 0000000..ff6ec13 Binary files /dev/null and b/Figures/eval/2d/20-19-03/y-gt.png differ diff --git a/Figures/eval/2d/20-19-03/y.png b/Figures/eval/2d/20-19-03/y.png new file mode 100644 index 0000000..146f0ac Binary files /dev/null and b/Figures/eval/2d/20-19-03/y.png differ diff --git a/Figures/eval/2d/20-19-03/z-gt.png b/Figures/eval/2d/20-19-03/z-gt.png new file mode 100644 index 0000000..2e113d2 Binary files /dev/null and b/Figures/eval/2d/20-19-03/z-gt.png differ diff --git a/Figures/eval/2d/20-19-03/z.png b/Figures/eval/2d/20-19-03/z.png new file mode 100644 index 0000000..646be24 Binary files /dev/null and b/Figures/eval/2d/20-19-03/z.png differ diff --git a/Figures/eval/2d/20-21-18/front-box-gt.png b/Figures/eval/2d/20-21-18/front-box-gt.png new file mode 100644 index 0000000..c6ff809 Binary files /dev/null and b/Figures/eval/2d/20-21-18/front-box-gt.png differ diff --git a/Figures/eval/2d/20-21-18/front-box.png b/Figures/eval/2d/20-21-18/front-box.png new file mode 100644 index 0000000..2edf2b6 Binary files /dev/null and b/Figures/eval/2d/20-21-18/front-box.png differ diff --git a/Figures/eval/2d/20-21-18/front-gt.png b/Figures/eval/2d/20-21-18/front-gt.png new file mode 100644 index 0000000..4eda1b4 Binary files /dev/null and b/Figures/eval/2d/20-21-18/front-gt.png differ diff --git a/Figures/eval/2d/20-21-18/front.png b/Figures/eval/2d/20-21-18/front.png new file mode 100644 index 0000000..1b5aa67 Binary files /dev/null and b/Figures/eval/2d/20-21-18/front.png differ diff --git a/Figures/eval/2d/20-21-18/x-gt.png b/Figures/eval/2d/20-21-18/x-gt.png new file mode 100644 index 0000000..6e2c4ee Binary files /dev/null and b/Figures/eval/2d/20-21-18/x-gt.png differ diff --git a/Figures/eval/2d/20-21-18/x.png b/Figures/eval/2d/20-21-18/x.png new file mode 100644 index 0000000..017858d Binary files /dev/null and b/Figures/eval/2d/20-21-18/x.png differ diff --git a/Figures/eval/2d/20-21-18/y-gt.png b/Figures/eval/2d/20-21-18/y-gt.png new file mode 100644 index 0000000..26cf1bb Binary files /dev/null and b/Figures/eval/2d/20-21-18/y-gt.png differ diff --git a/Figures/eval/2d/20-21-18/y.png b/Figures/eval/2d/20-21-18/y.png new file mode 100644 index 0000000..60ba4d0 Binary files /dev/null and b/Figures/eval/2d/20-21-18/y.png differ diff --git a/Figures/eval/2d/20-21-18/z-gt.png b/Figures/eval/2d/20-21-18/z-gt.png new file mode 100644 index 0000000..143af9f Binary files /dev/null and b/Figures/eval/2d/20-21-18/z-gt.png differ diff --git a/Figures/eval/2d/20-21-18/z.png b/Figures/eval/2d/20-21-18/z.png new file mode 100644 index 0000000..a6a8796 Binary files /dev/null and b/Figures/eval/2d/20-21-18/z.png differ diff --git a/Figures/eval/2d/20-26-28/front-box-gt.png b/Figures/eval/2d/20-26-28/front-box-gt.png new file mode 100644 index 0000000..58bbe77 Binary files /dev/null and b/Figures/eval/2d/20-26-28/front-box-gt.png differ diff --git a/Figures/eval/2d/20-26-28/front-box.png b/Figures/eval/2d/20-26-28/front-box.png new file mode 100644 index 0000000..cf4b51f Binary files /dev/null and b/Figures/eval/2d/20-26-28/front-box.png differ diff --git a/Figures/eval/2d/20-26-28/front-gt.png b/Figures/eval/2d/20-26-28/front-gt.png new file mode 100644 index 0000000..5353aed Binary files /dev/null and b/Figures/eval/2d/20-26-28/front-gt.png differ diff --git a/Figures/eval/2d/20-26-28/front.png b/Figures/eval/2d/20-26-28/front.png new file mode 100644 index 0000000..c8d19b4 Binary files /dev/null and b/Figures/eval/2d/20-26-28/front.png differ diff --git a/Figures/eval/2d/20-26-28/x-gt.png b/Figures/eval/2d/20-26-28/x-gt.png new file mode 100644 index 0000000..88f787b Binary files /dev/null and b/Figures/eval/2d/20-26-28/x-gt.png differ diff --git a/Figures/eval/2d/20-26-28/x.png b/Figures/eval/2d/20-26-28/x.png new file mode 100644 index 0000000..f28fa14 Binary files /dev/null and b/Figures/eval/2d/20-26-28/x.png differ diff --git a/Figures/eval/2d/20-26-28/y-gt.png b/Figures/eval/2d/20-26-28/y-gt.png new file mode 100644 index 0000000..867b93b Binary files /dev/null and b/Figures/eval/2d/20-26-28/y-gt.png differ diff --git a/Figures/eval/2d/20-26-28/y.png b/Figures/eval/2d/20-26-28/y.png new file mode 100644 index 0000000..514775d Binary files /dev/null and b/Figures/eval/2d/20-26-28/y.png differ diff --git a/Figures/eval/2d/20-26-28/z-gt.png b/Figures/eval/2d/20-26-28/z-gt.png new file mode 100644 index 0000000..f42dc99 Binary files /dev/null and b/Figures/eval/2d/20-26-28/z-gt.png differ diff --git a/Figures/eval/2d/20-26-28/z.png b/Figures/eval/2d/20-26-28/z.png new file mode 100644 index 0000000..c6cb547 Binary files /dev/null and b/Figures/eval/2d/20-26-28/z.png differ diff --git a/Figures/eval/2d/20-28-45/front-box-gt.png b/Figures/eval/2d/20-28-45/front-box-gt.png new file mode 100644 index 0000000..26401bd Binary files /dev/null and b/Figures/eval/2d/20-28-45/front-box-gt.png differ diff --git a/Figures/eval/2d/20-28-45/front-box.png b/Figures/eval/2d/20-28-45/front-box.png new file mode 100644 index 0000000..8a9bb3f Binary files /dev/null and b/Figures/eval/2d/20-28-45/front-box.png differ diff --git a/Figures/eval/2d/20-28-45/front-gt.png b/Figures/eval/2d/20-28-45/front-gt.png new file mode 100644 index 0000000..b91dc25 Binary files /dev/null and b/Figures/eval/2d/20-28-45/front-gt.png differ diff --git a/Figures/eval/2d/20-28-45/front.png b/Figures/eval/2d/20-28-45/front.png new file mode 100644 index 0000000..6f2082f Binary files /dev/null and b/Figures/eval/2d/20-28-45/front.png differ diff --git a/Figures/eval/2d/20-28-45/x-gt.png b/Figures/eval/2d/20-28-45/x-gt.png new file mode 100644 index 0000000..ce9d492 Binary files /dev/null and b/Figures/eval/2d/20-28-45/x-gt.png differ diff --git a/Figures/eval/2d/20-28-45/x.png b/Figures/eval/2d/20-28-45/x.png new file mode 100644 index 0000000..860aa60 Binary files /dev/null and b/Figures/eval/2d/20-28-45/x.png differ diff --git a/Figures/eval/2d/20-28-45/y-gt.png b/Figures/eval/2d/20-28-45/y-gt.png new file mode 100644 index 0000000..fe10eab Binary files /dev/null and b/Figures/eval/2d/20-28-45/y-gt.png differ diff --git a/Figures/eval/2d/20-28-45/y.png b/Figures/eval/2d/20-28-45/y.png new file mode 100644 index 0000000..f4e009d Binary files /dev/null and b/Figures/eval/2d/20-28-45/y.png differ diff --git a/Figures/eval/2d/20-28-45/z-gt.png b/Figures/eval/2d/20-28-45/z-gt.png new file mode 100644 index 0000000..024b002 Binary files /dev/null and b/Figures/eval/2d/20-28-45/z-gt.png differ diff --git a/Figures/eval/2d/20-28-45/z.png b/Figures/eval/2d/20-28-45/z.png new file mode 100644 index 0000000..012d8bd Binary files /dev/null and b/Figures/eval/2d/20-28-45/z.png differ diff --git a/Figures/eval/our/1/x-gt.png b/Figures/eval/our/1/x-gt.png new file mode 100644 index 0000000..9c8ecf7 Binary files /dev/null and b/Figures/eval/our/1/x-gt.png differ diff --git a/Figures/eval/our/1/x.png b/Figures/eval/our/1/x.png new file mode 100644 index 0000000..925616e Binary files /dev/null and b/Figures/eval/our/1/x.png differ diff --git a/Figures/eval/our/1/y-gt.png b/Figures/eval/our/1/y-gt.png new file mode 100644 index 0000000..f847406 Binary files /dev/null and b/Figures/eval/our/1/y-gt.png differ diff --git a/Figures/eval/our/1/y.png b/Figures/eval/our/1/y.png new file mode 100644 index 0000000..32d3b08 Binary files /dev/null and b/Figures/eval/our/1/y.png differ diff --git a/Figures/eval/our/1/z-gt.png b/Figures/eval/our/1/z-gt.png new file mode 100644 index 0000000..c1edd51 Binary files /dev/null and b/Figures/eval/our/1/z-gt.png differ diff --git a/Figures/eval/our/1/z.png b/Figures/eval/our/1/z.png new file mode 100644 index 0000000..3afc739 Binary files /dev/null and b/Figures/eval/our/1/z.png differ diff --git a/Figures/eval/our/3/x-gt.png b/Figures/eval/our/3/x-gt.png new file mode 100644 index 0000000..44e7668 Binary files /dev/null and b/Figures/eval/our/3/x-gt.png differ diff --git a/Figures/eval/our/3/x.png b/Figures/eval/our/3/x.png new file mode 100644 index 0000000..426c006 Binary files /dev/null and b/Figures/eval/our/3/x.png differ diff --git a/Figures/eval/our/3/y-gt.png b/Figures/eval/our/3/y-gt.png new file mode 100644 index 0000000..bdeb152 Binary files /dev/null and b/Figures/eval/our/3/y-gt.png differ diff --git a/Figures/eval/our/3/y.png b/Figures/eval/our/3/y.png new file mode 100644 index 0000000..5f78a4c Binary files /dev/null and b/Figures/eval/our/3/y.png differ diff --git a/Figures/eval/our/3/z-gt.png b/Figures/eval/our/3/z-gt.png new file mode 100644 index 0000000..31b5691 Binary files /dev/null and b/Figures/eval/our/3/z-gt.png differ diff --git a/Figures/eval/our/3/z.png b/Figures/eval/our/3/z.png new file mode 100644 index 0000000..b7bfb79 Binary files /dev/null and b/Figures/eval/our/3/z.png differ diff --git a/Figures/flow1.PNG b/Figures/flow1.PNG new file mode 100644 index 0000000..b7a7660 Binary files /dev/null and b/Figures/flow1.PNG differ diff --git a/Figures/flow2.PNG b/Figures/flow2.PNG new file mode 100644 index 0000000..4fd0dbd Binary files /dev/null and b/Figures/flow2.PNG differ diff --git a/Figures/landmarkdet.png b/Figures/landmarkdet.png new file mode 100644 index 0000000..ae86621 Binary files /dev/null and b/Figures/landmarkdet.png differ diff --git a/Figures/myplot.png b/Figures/myplot.png new file mode 100644 index 0000000..0b181e6 Binary files /dev/null and b/Figures/myplot.png differ diff --git a/Figures/old/ColorFitting.pdf b/Figures/old/ColorFitting.pdf new file mode 100644 index 0000000..27ce23c Binary files /dev/null and b/Figures/old/ColorFitting.pdf differ diff --git a/Figures/old/FittingFlow.drawio b/Figures/old/FittingFlow.drawio new file mode 100644 index 0000000..eee4cae --- /dev/null +++ b/Figures/old/FittingFlow.drawio @@ -0,0 +1 @@ +7V1bc9u2Ev41nul5sIcAeAEfYzlOc5I0qd1pTp46tARLTGhSpajYzq8/oEjwAi4lSgIoSnZmWosgQEn7LRbf7gKrMzJ6eHoXe/PZp2jCgjNsTJ7OyNUZxogYmP9JW57zFmyYWcs09id5W9lw6/9ieaORty79CVvUOiZRFCT+vN44jsKQjZNamxfH0WO9230U1N917k1Zo+F27AXN1q/+JJnlrcgwyhu/M386y9+aWvmNO2/8YxpHyzB/vzNM7lf/stsPnnhW3n8x8ybRY6WJvD0joziKkuzVw9OIBalwhdiycdctd4vPHbMw6TLgj7+vFv73z9/Orz7EV9Pr+d0loeeum3+45FkIhE24fPLLKE5m0TQKveBt2Xq5+tIsfazBr8o+H6NozhsRb/zOkuQ5B9tbJhFvmiUPQX6XPfnJ/9LhF1Z+9a1y5+opf/Lq4llchEn8XBmUXhaj0oty2OqqHCeUDqXjFkkc/SiQNnlLU465aBfRMh6zNcIT6urFU5as6efkQk4lW3mHHKZ3LHpg/BPzDjELvMT/WddML1fwadEvH/omjr3nSod55IfJovLkL2kD75BPVmznMzOfqjaRFEbqTwlZ15+/yD6BuKp8lbJppYRbKGT+pX96wZKJeSUpaKl+KfaPMz9ht3NvhdQjN1J1Vbv3g2AUBVG8GksMYzQyjEIPKnfwlWOv7txHYVJpz+c0uZzG3sRn8L1WHfrJ4oQ9rQVd3DVqwkbCbDyWRskSfWYVe2TaRrueVLDaAQpzOLbB2Mk2GIezDbijbUCDMA0U17WPrrcM2LbWdNdkGJB9SHVEVWUsVHODOqKaMpa62a6Ob1JGwy/HgbdY+OOs8doPio8zAI1F1iB0FiFJaYllrdXaxgBKiX69xS91QbNsaUUjwIqGoRXN0rWikaPEQty78mPuAPlRyO+HqSnTQjzsfFJsJB7EdTTBZKMGLEO39CqJB97LjJsdzbhJBmHFsVNXP2ysN+ImWdtfjw03AbthB1y6l/Oamtr/LlNv/jLwQ3Yu5sob3iX9klZ5n7+a5n9XT7mL5Rb+MedlmzQZFjNvnr6cx9GYLRab7VQZqvi8TNIPB9uvezpm4zFkv+6olVplNfbGktYF2lwXELguaHN0jAa+N+zfJVskDdHz75g0BBRGqURr0sybqmsFt/z5nMemuM4fnN5PBeiPveBN4E9TC//gTyYrw+blDWMuchYrWpslJmQCEBi9QtD0+2/YYh6FC/ZSMLDcQ4NgHtTD2i4YqG39FKhsXECxctd9L/Bw04j1t3Dc32N44ZjYd7Zl61k4MDRhoIXD0TVhcIdopS6ZTyxGJyYkc4rviK1I5oJeFQsFcQ8t8w4O9XETJMeSZG4Dei76VGVOTV0yt07dtsgyJ0ZXmeuy58QGZJ45ByltAf2O9Mb5YrUwpl4HsudPTafj/UOaES39jOxpLa7GPuGQ8YTd0TsIOWITl0w0uROO1c1CYV0WijiakFu5wqMgWk76gC+NPlHaN3wYmni9wmcflgXvEn3aPs+gLfqUR5U2h5/o2ZDYs9khNKxvhRuPXVf3RCPSCoctaIUDJlrhg6oXui5DeeJLXGfHR5uNFLP3dYnbAT6CDs1QTFcTfB+9cPLgxT8W/aDXh91soAfFSntFz4IiPVuiRyH0Rt48WcadDecQ4rATICcaR4mXN6lKWUhJKgToAAVUQFuk1nIOylEVbNvcK0OK9uKogght5KiWPSiOakFRGBVW+4ol6c5qbHzxn1bbu/s14+69kw4dKP3VN4d1BXiu2Dl3Ub5nkPI5zHHNk/CnCibB+ALgVBCcRBucuijxX7PYv09OzKXZOa+gj1Xp4sQZfp/YYnY6Lk0DPnG0oA/4PoQ/XeeS3Y6u/8tunhP2B3nzWARdt2JE42X8s5DzUdEjlTE7kV7ayIcoHhQfshX4QWtm7An6so1pSw9tde32rXV7+rJRmMpFMKAMUv7i811KjDpjmvu4WnxXwEdWgDEynAuJHQH544IH9UJ1bQWeC4iymKO8Q+bEpG7/uk2Vgw9oqGDH8lEyB3B1et1Y5kDZle3wJwaE/y3jC3h8TPhWglPnrrLwlDzjMeAN9Yt4u11fzL1wd8RHQXoUooJ49rgXhrhFjcMBDvPvXc5s9MG/dzs5uulUuUr+7XSNR9JhxSMdXfHIk4x4NHb1WQD3hnaYiamtHj9dAchTjHg4lnnRwXnqF0Bd+ydO1gFuomihg6PYHnjsSpVg5+iGjVm6CGzHlY7I1XUoltE8+EEmESNTjuYtCyenCyVGEpBQeg4CUkU+Bya0+JXQ7khoaddNoO6wAsp0/zBFy662NK+e8Vnr8ubd5Zl1dbrEFrsOmIztc02l2lyT2PfCacBW5Na6/Ic/oB8s+2K5ciDJPjyWutyUInjcL4798FwJR+vQKApTr363i5d4/OYoYF7oh9OuGAbeHQs2o+c/ZOUj/Qf+5Otx4HNY+EOv3zEv/gdh+sT/u5jztwUnrKpSOo7YKS8i+ha438VBTUAdXYAiA0J0DWNiwV30WCVLqwZ+YxbF/i+OlRectdTAqgDS5Cl7nIJpL6y1SzGY3QlTZx5UrckDkGHRtl3JlkaNFdeQEkiupETZF8pHlXrUrLhl2NYFMjG1nOz/9YKShHvjbnkTiR1x4m0y/th4m21LzLhYIijUveihaExRJ+213NS6smC2QxuWDC4LhnT57Ag1A2m3q8Ne2Lj2k2S1rMAu9rHnijlRkEIotgPS956rwTQ3U2YCPXlAkGsMERCw0kXGtJbB7uRNPCPwy307Gcxy9qfs0Rwj7L18bmzdmPSjgvvcm4N4Y/odVTgDQmMCdp+o0RdX2uxBCUQMwWOd2pihCXlsin1hVdTaprQmP4c0yx9iaK7pOxaLwHOxA5WfgyX5IdBaQQu6tq34SMXxVDAI/yWO5tHiqE7IqcDYrNsYxxjAimR1qL20yZmceItZMYfqniU52zP+vq03ul1uoRXVzT9aIHpuLvRsdt1v0pNb6taJEZV3JXd3S836kxy5WpUqz1P6xE5eoUmv3wmeExzo8mE7tkR3ucwA48ItSVOvbKrLuNiaK0UXIqyeX+bqK59fVrZh0HGQLGcCyZkAYkbadv8jbdv//2axf//80tZpQmWQS9gPtjMUt0Ks1nOsb7MAvDz+8m36Kbwk9TBbdlLs4nCCj93Wj9xdByHT06p3Eb9zH2Rxd97GQkV23HYl1eNKBh07gVIUWJPmQRurFDPEcrcFcuuEzzCcTZSvoJdoVcawODF4YaAOhb++cBPH5ZQaj6KH3LYHTRTkbzNNFHxjIDSRii1YwgZaZDeaaDvSg9xuLHEHIgeqL1CWXLn+FipIqy7OOVdf4qxXQbXaNjQdEhWSCujRbjpE7XrmtgjPqdeh6183SRL++Rx9n337/etHd/L84ekc0Jgjq2VomJvMKGQMm6oO6tdGbYUnZlcPmsA6rHw329pPqXzHxeXqhxEMcvXpE/+TJ8rkM5fvw0XiheNeDnAU3pTWUthy4pJakANVlIaqWSUFoXYYY4hcb7lE7DVbt9xlWluOtgx5tU3pvdab6hReO4sGsjDJP3+DsJwV77oyufKD5F/d0s1u2jfbnnxGsWwBbakBGE+tack2eyqlK1udSgXWlUiKTU3UtK3a8piwhh5RHBVhIvnfDiFQHFVbKhMW4RFlgpFE1XvOZMLyU3DC7zWRWSmUIEHcbyIThhjKVevy8o8zj7mWP2xmcMMKLZiGoiymafWTxDSR3iQmDFqH2O1gFg6jmXZxobVXCFL1b/bAaZcO0cOjSmFio5Eqxs0Mg7YEJixkBTGW1/xluTzLJNYVRwIOl72EIiwvMn1ZtuSe9GvaEyi251Io444FW6mZfm0622H3ijJKabjVM1LowjDEfcWJo2Mgd4jUd8NSIv1GeuctaiaW1aoTuVPFv8R2q35yj079mJ2B+9Qgd1AaRJ0Llxrlv7oaUJNeyPGuriqFXVvSzW6pyHZ/QbnSdQi4vfBsxloV3ugK42Ft6D1Htqpsxrkln1ntO5+BdZUpyHMDp5dQlU8uQT9zC0VbVWRT4UoT+xcthb05SRHGhTzLjiQz982xUiHrW85u03oH0YY61oMsFAWz/JYzygo0zJGSSgT44TxtlbDhqlLthWn0ZD2zQL8XdHXtgmD54IdZ8Gbr0SJ/ueWwL6tMRLdRJ+1NYvnsrwOc5mxJ4RFdGtu+rqnV2KbyGL/dfP3Pzqpn/PZ+9KXr8BsRsNxaef+KvXARbDX4lHXYbtRkdEgzhteiw9tvheOXcZQqVknl+PeefYomaXWft/8H \ No newline at end of file diff --git a/Figures/old/FittingFlow.pdf b/Figures/old/FittingFlow.pdf new file mode 100644 index 0000000..cac0462 Binary files /dev/null and b/Figures/old/FittingFlow.pdf differ diff --git a/Figures/old/FittingFlowNew.pdf b/Figures/old/FittingFlowNew.pdf new file mode 100644 index 0000000..3f1b01a Binary files /dev/null and b/Figures/old/FittingFlowNew.pdf differ diff --git a/Figures/our.pdf b/Figures/our.pdf new file mode 100644 index 0000000..e7b4d9b Binary files /dev/null and b/Figures/our.pdf differ diff --git a/Figures/our.png b/Figures/our.png new file mode 100644 index 0000000..c1ed12f Binary files /dev/null and b/Figures/our.png differ diff --git a/Figures/pcm.png b/Figures/pcm.png new file mode 100644 index 0000000..3594de9 Binary files /dev/null and b/Figures/pcm.png differ diff --git a/Figures/pointcloud.png b/Figures/pointcloud.png new file mode 100644 index 0000000..309caac Binary files /dev/null and b/Figures/pointcloud.png differ diff --git a/Figures/px/our1.png b/Figures/px/our1.png new file mode 100644 index 0000000..0d96738 Binary files /dev/null and b/Figures/px/our1.png differ diff --git a/Figures/px/our3.png b/Figures/px/our3.png new file mode 100644 index 0000000..6d93984 Binary files /dev/null and b/Figures/px/our3.png differ diff --git a/Figures/px/our4.png b/Figures/px/our4.png new file mode 100644 index 0000000..916cca3 Binary files /dev/null and b/Figures/px/our4.png differ diff --git a/Figures/px/our7.png b/Figures/px/our7.png new file mode 100644 index 0000000..e185a02 Binary files /dev/null and b/Figures/px/our7.png differ diff --git a/Figures/px/out5.png b/Figures/px/out5.png new file mode 100644 index 0000000..fe9ab4f Binary files /dev/null and b/Figures/px/out5.png differ diff --git a/Figures/px/out6.png b/Figures/px/out6.png new file mode 100644 index 0000000..f4a0188 Binary files /dev/null and b/Figures/px/out6.png differ diff --git a/Figures/px/sd1.png b/Figures/px/sd1.png new file mode 100644 index 0000000..5561f76 Binary files /dev/null and b/Figures/px/sd1.png differ diff --git a/Figures/px/sd3.png b/Figures/px/sd3.png new file mode 100644 index 0000000..f44dc2c Binary files /dev/null and b/Figures/px/sd3.png differ diff --git a/Figures/px/sd4.png b/Figures/px/sd4.png new file mode 100644 index 0000000..f62537f Binary files /dev/null and b/Figures/px/sd4.png differ diff --git a/Figures/px/sd5.png b/Figures/px/sd5.png new file mode 100644 index 0000000..840f8ce Binary files /dev/null and b/Figures/px/sd5.png differ diff --git a/Figures/px/sd6.png b/Figures/px/sd6.png new file mode 100644 index 0000000..82d8450 Binary files /dev/null and b/Figures/px/sd6.png differ diff --git a/Figures/px/sd7.png b/Figures/px/sd7.png new file mode 100644 index 0000000..8582008 Binary files /dev/null and b/Figures/px/sd7.png differ diff --git a/Figures/rs3d.png b/Figures/rs3d.png new file mode 100644 index 0000000..64830f8 Binary files /dev/null and b/Figures/rs3d.png differ diff --git a/Figures/schedule1.png b/Figures/schedule1.png new file mode 100644 index 0000000..2b38fe6 Binary files /dev/null and b/Figures/schedule1.png differ diff --git a/Figures/schedule2.png b/Figures/schedule2.png new file mode 100644 index 0000000..514715e Binary files /dev/null and b/Figures/schedule2.png differ diff --git a/Figures/shapeFittingPlot.pdf b/Figures/shapeFittingPlot.pdf new file mode 100644 index 0000000..d3a2694 Binary files /dev/null and b/Figures/shapeFittingPlot.pdf differ diff --git a/Figures/shapeFittingPlot.png b/Figures/shapeFittingPlot.png new file mode 100644 index 0000000..e6c9e63 Binary files /dev/null and b/Figures/shapeFittingPlot.png differ diff --git a/Figures/site.PNG b/Figures/site.PNG new file mode 100644 index 0000000..5744f9b Binary files /dev/null and b/Figures/site.PNG differ diff --git a/Figures/standard.pdf b/Figures/standard.pdf new file mode 100644 index 0000000..cc8f509 Binary files /dev/null and b/Figures/standard.pdf differ diff --git a/Figures/standard.png b/Figures/standard.png new file mode 100644 index 0000000..0028516 Binary files /dev/null and b/Figures/standard.png differ diff --git a/Figures/stereo_DT_d415_front-crop1a-1.png b/Figures/stereo_DT_d415_front-crop1a-1.png new file mode 100644 index 0000000..abff440 Binary files /dev/null and b/Figures/stereo_DT_d415_front-crop1a-1.png differ diff --git a/Figures/thrift/Rs-ex.pdf b/Figures/thrift/Rs-ex.pdf new file mode 100644 index 0000000..df26db0 Binary files /dev/null and b/Figures/thrift/Rs-ex.pdf differ diff --git a/Figures/thrift/Rs-ex.svg b/Figures/thrift/Rs-ex.svg new file mode 100644 index 0000000..8d3d0e9 --- /dev/null +++ b/Figures/thrift/Rs-ex.svg @@ -0,0 +1,324 @@ + + + + + + +Rs + + +cluster_RealSenseService + +RealSenseService service + + + +ThriftPointList + +ThriftPointList :: list<ThriftPoint3D> + + + +ThriftPoint3D + +struct ThriftPoint3D + +x :: double + +y :: double + +z :: double + + + +ThriftPointList->ThriftPoint3D + + + + + +ThriftLandmark2DList + +ThriftLandmark2DList :: list<ThriftLandmark2D> + + + +ThriftLandmark2D + +struct ThriftLandmark2D + +name :: string + +pixels :: ThriftPixel + +uncertainty :: double + + + +ThriftLandmark2DList->ThriftLandmark2D + + + + + +ThriftLandmarkList + +ThriftLandmarkList :: list<ThriftLandmark> + + + +ThriftLandmark + +struct ThriftLandmark + +name :: string + +point :: ThriftPoint3D + +uncertainty :: ThriftUncertaintyCovariance + + + +ThriftLandmarkList->ThriftLandmark + + + + + +ThriftVertexColorList + +ThriftVertexColorList :: list<ThriftColor> + + + +ThriftColor + +struct ThriftColor + +r :: i16 + +g :: i16 + +b :: i16 + + + +ThriftVertexColorList->ThriftColor + + + + + +ThriftTriangleCellList + +ThriftTriangleCellList :: list<ThriftTriangleCell> + + + +ThriftTriangleCell + +struct ThriftTriangleCell + +id1 :: i32 + +id2 :: i32 + +id3 :: i32 + + + +ThriftTriangleCellList->ThriftTriangleCell + + + + + +ThriftImage + +struct ThriftImage + +data :: string + +width :: i32 + +height :: i32 + + + +ThriftPixel + +struct ThriftPixel + +width :: i32 + +height :: i32 + + + +ThriftVector3D + +struct ThriftVector3D + +x :: double + +y :: double + +z :: double + + + +ThriftUncertaintyCovariance + +struct ThriftUncertaintyCovariance + +variances :: ThriftVector3D + +principalAxis1 :: ThriftVector3D + +principalAxis2 :: ThriftVector3D + +principalAxis3 :: ThriftVector3D + + + +ThriftUncertaintyCovariance:field_variances->ThriftVector3D + + + + + +ThriftUncertaintyCovariance:field_principalAxis1->ThriftVector3D + + + + + +ThriftUncertaintyCovariance:field_principalAxis2->ThriftVector3D + + + + + +ThriftUncertaintyCovariance:field_principalAxis3->ThriftVector3D + + + + + +ThriftLandmark2D:field_pixels->ThriftPixel + + + + + +ThriftLandmark:field_point->ThriftPoint3D + + + + + +ThriftLandmark:field_uncertainty->ThriftUncertaintyCovariance + + + + + +ThriftVertexColorMesh + +struct ThriftVertexColorMesh + +vertices :: ThriftPointList + +color :: ThriftVertexColorList + +faces :: ThriftTriangleCellList + + + +ThriftVertexColorMesh:field_vertices->ThriftPointList + + + + + +ThriftVertexColorMesh:field_color->ThriftVertexColorList + + + + + +ThriftVertexColorMesh:field_faces->ThriftTriangleCellList + + + + + +CaptureResult + +struct CaptureResult + +image :: ThriftImage + +landmarks2d :: ThriftLandmark2DList + +landmarks :: ThriftLandmarkList + +mesh :: ThriftVertexColorMesh + + + +CaptureResult:field_landmarks2d->ThriftLandmark2DList + + + + + +CaptureResult:field_landmarks->ThriftLandmarkList + + + + + +CaptureResult:field_image->ThriftImage + + + + + +CaptureResult:field_mesh->ThriftVertexColorMesh + + + + + +ThriftServerError + +exception ThriftServerError + +message :: string + +stackTrace :: string + + + +function_RealSenseServicecapture + +function capture :: CaptureResult + + + +function_RealSenseServicecapture:return_type->CaptureResult + + + + + +function_RealSenseServicecapture->ThriftServerError + + + + + diff --git a/Figures/web-fitting.pdf b/Figures/web-fitting.pdf new file mode 100644 index 0000000..dbb9606 Binary files /dev/null and b/Figures/web-fitting.pdf differ diff --git a/Front/Abstract.tex b/Front/Abstract.tex new file mode 100755 index 0000000..85a7c2d --- /dev/null +++ b/Front/Abstract.tex @@ -0,0 +1,3 @@ +\chapter{Abstract} + +Human face analysis is an important branch of Computer Vision. One of the interesting topics in this area of research is the analysis of a human face based on a single image. The complexity of this problem is rather high due to restricted information that is available in a single image. Especially challenging is a task of the 3D shape reconstruction of the face based on a single image. The reconstruction process has little information to reliably recover the shape information (depth), which is the third dimension that is completely missing from an image. Using a single image does not always produce an accurate 3D reconstruction, especially in settings where lighting conditions cannot be controlled. This thesis aims to build an efficient and stable system, which adds, in addition to a photo of a face, also a 3D depth information to the reconstruction process. By incorporating a 3D depth information into this process, we aim to improve the overall quality of the final shape reconstruction, as well as, the visual appearance of the reconstructed face. We build the complete \textit{fitting} pipeline that consists of separate modules dealing with data acquisition, shape reconstruction, and analysis of color and illumination. We show that including the complete 3D depth information into the framework in the form of a point cloud indeed improves the final reconstruction quality. To obtain a 3D depth information alongside the other input requirements, we use an affordable consumer depth camera technology offered by Intel® RealSense™ platform. The pipeline is flexible enough that it can be seamlessly integrated into the existing demo-framework (the face-fitting web service\footnote{Scalismo Face Morpher — \url{https://face-morpher.scalismo.org}}) for various use-cases. The applications of this work include but not limited to photo-realistic face manipulation\footnote{Photo-realistic Face Manipulation — \url{https://gravis.dmi.unibas.ch/PMM/demo/face-manipulation/}}, 3D face modelling and analysis. \ No newline at end of file diff --git a/Front/Acknowledgment.tex b/Front/Acknowledgment.tex new file mode 100755 index 0000000..bcd80ff --- /dev/null +++ b/Front/Acknowledgment.tex @@ -0,0 +1,3 @@ +\chapter{Acknowledgments} + +First, I would like to thank my supervisor Dr. Marcel L{\"u}thi for his continuous support and guidance over the course of this thesis. Further, I would like to thank Professor Thomas Vetter for giving me the opportunity to work in the Graphics and Vision Research Group. I also thank Dr. Andreas-Morel Foster for his help and valuable input, as well as, PhD students Patrick Kahr, Dana Rahbani and Dennis Madsen for their involvement in this project. Last but not least, I would like to thank my friends (especially those who proofread my work) and family for their support. \ No newline at end of file diff --git a/LICENSE.md b/LICENSE.md new file mode 100644 index 0000000..44939e6 --- /dev/null +++ b/LICENSE.md @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2014 Ivan Giangreco + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..7d64608 --- /dev/null +++ b/Makefile @@ -0,0 +1,27 @@ +# +# Thanks to Kevin Urban for providing this makefile. +# Jul 2012 +# + +# choose which latex compiler you want to use (xelatex or pdflatex) +TEX = pdflatex + + +################ +# stop editing # +################ +latexfile = Thesis + +# rerun pdf generation until it doesn't say rerun anymore +$(latexfile).pdf: $(latexfile).bbl + while ($(TEX) $(latexfile); grep -q "Rerun to get cross" $(latexfile).log); do true; done + +$(latexfile).bbl: $(latexfile).aux + bibtex $(latexfile) + +$(latexfile).aux: $(latexfile).tex + $(TEX) $(latexfile) + +clean: + rm *.aux + rm *.bbl diff --git a/README.md b/README.md new file mode 100644 index 0000000..ada5977 --- /dev/null +++ b/README.md @@ -0,0 +1 @@ +Latex Template for Bachelor and Master thesis (University of Basel, Department of Mathematics and Computer Science, DBIS) diff --git a/Template/logo-de.pdf b/Template/logo-de.pdf new file mode 100644 index 0000000..348ad7b Binary files /dev/null and b/Template/logo-de.pdf differ diff --git a/Template/logo-en.pdf b/Template/logo-en.pdf new file mode 100644 index 0000000..8b5efb9 Binary files /dev/null and b/Template/logo-en.pdf differ diff --git a/Thesis.pdf b/Thesis.pdf new file mode 100644 index 0000000..71a7986 Binary files /dev/null and b/Thesis.pdf differ diff --git a/Thesis.tex b/Thesis.tex new file mode 100755 index 0000000..32a152f --- /dev/null +++ b/Thesis.tex @@ -0,0 +1,74 @@ +\documentclass[a4paper, 10pt]{memoir} +\usepackage{basilea} +% \usepackage{tgpagella} +\usepackage{mathpazo} +\usepackage{lipsum} +\usepackage{eurosym} +\usepackage{graphicx} +\usepackage[center]{caption} +\usepackage[colorinlistoftodos]{todonotes} +\usepackage{siunitx} +\usepackage[ruled,vlined]{algorithm2e} +\usepackage{placeins} + +\title{Efficient 3D Morphable Model Face Fitting using Depth Sensing Technologies} +% automated image analysis +\thesistype{Master's Thesis } +\department{Department of Mathematics and Computer Science} +\faculty{Natural Science Faculty of the University of Basel} +\research{Graphics and Vision Research Group \\ \url{https://gravis.dmi.unibas.ch}} +\examiner{Prof. Dr. Thomas Vetter} +\supervisor{Dr. Marcel L{\"u}thi} + +\authors{Giorgi Grigalashvili} +\email{g.grigalashvili@unibas.ch} +\immatriculnr {12-059-754} + +\date {13.11.2019} + +% switch here for the german logo to logo-de +\ulogo {Template/logo-en} + +\makeatletter + \def\blank{\clearpage% + \vspace*{\fill} + \hfill + \begin{center} + This page intentionally left blank. + \end{center} + \vspace{\fill} + \thispagestyle{empty} + } +\makeatother + +\begin{document} +\selectlanguage{english} +\thesisfront +\maketitle + +\pagestyle{thesis} +\blank +\input{./Front/Acknowledgment} +\blank +\input{./Front/Abstract} +\blank +\thesistoc + +\thesismain +\input{./Chapters/Chapter1} +\input{./Chapters/Chapter2} +\input{./Chapters/Chapter3} +\input{./Chapters/Chapter4} +\input{./Chapters/Chapter5} +\input{./Chapters/Chapter6} + +\thesisappendix +\thesisbib +\begin{appendices} + \input{./Back/AppendixA} +\end{appendices} +\thesisback +% \input{./Back/Acronyms} +\input{./Back/DeclarationOfAuthorship} + +\end{document} \ No newline at end of file diff --git a/Thesis.toc b/Thesis.toc new file mode 100644 index 0000000..8991c2f --- /dev/null +++ b/Thesis.toc @@ -0,0 +1,46 @@ +\babel@toc {english}{} +\babel@toc {english}{} +\contentsline {chapter}{Acknowledgments}{iii}{chapter*.1}% +\contentsline {chapter}{Abstract}{v}{chapter*.2}% +\vspace {2em} +\contentsline {chapter}{\chapternumberline {1}Introduction}{1}{chapter.5}% +\contentsline {section}{\numberline {1.1}Motivation \& Previous Work}{2}{section.7}% +\contentsline {section}{\numberline {1.2}Contributions}{4}{section.10}% +\contentsline {section}{\numberline {1.3}Thesis Structure}{4}{section.11}% +\contentsline {chapter}{\chapternumberline {2}Background}{7}{chapter.13}% +\contentsline {section}{\numberline {2.1}3D Morphable Model}{7}{section.14}% +\contentsline {section}{\numberline {2.2}Fitting Pipeline}{9}{section.22}% +\contentsline {subsection}{\numberline {2.2.1}Markov Chain Monte Carlo (MCMC)}{9}{subsection.23}% +\contentsline {subsection}{\numberline {2.2.2}Landmark Fitting}{11}{subsection.33}% +\contentsline {subsection}{\numberline {2.2.3}Color Fitting}{11}{subsection.35}% +\contentsline {section}{\numberline {2.3}Depth Camera}{13}{section.39}% +\contentsline {subsection}{\numberline {2.3.1}De-projection}{14}{subsection.43}% +\contentsline {chapter}{\chapternumberline {3}Methods}{17}{chapter.48}% +\contentsline {section}{\numberline {3.1}Camera}{18}{section.54}% +\contentsline {subsection}{\numberline {3.1.1}Camera Calibration}{19}{subsection.55}% +\contentsline {subsection}{\numberline {3.1.2}Camera Configuration Parameters}{20}{subsection.61}% +\contentsline {subsection}{\numberline {3.1.3}SDK}{20}{subsection.62}% +\contentsline {section}{\numberline {3.2}Cross-Language API}{22}{section.68}% +\contentsline {section}{\numberline {3.3}Capture Procedure (Server)}{22}{section.70}% +\contentsline {subsection}{\numberline {3.3.1}RGB Color Image}{23}{subsection.74}% +\contentsline {subsection}{\numberline {3.3.2}3D Landmarks}{24}{subsection.75}% +\contentsline {subsection}{\numberline {3.3.3}Point Cloud}{25}{subsection.78}% +\contentsline {section}{\numberline {3.4}Fitting Pipeline (Client)}{26}{section.81}% +\contentsline {subsection}{\numberline {3.4.1}Data Processing}{26}{subsection.84}% +\contentsline {subsection}{\numberline {3.4.2}Shape Fitting}{29}{subsection.88}% +\contentsline {subsection}{\numberline {3.4.3}Color Fitting}{34}{subsection.109}% +\contentsline {chapter}{\chapternumberline {4}Results}{37}{chapter.114}% +\contentsline {section}{\numberline {4.1}Evaluation Metrics}{37}{section.115}% +\contentsline {section}{\numberline {4.2}Shape Fitting Module Evaluation}{38}{section.116}% +\contentsline {section}{\numberline {4.3}Full Fitting Evaluation (Shape Fitting + Color Fitting)}{40}{section.120}% +\contentsline {section}{\numberline {4.4}Execution Performance}{44}{section.129}% +\contentsline {chapter}{\chapternumberline {5}Conclusion}{47}{chapter.133}% +\contentsline {chapter}{\chapternumberline {6}Future Work}{49}{chapter.134}% +\contentsline {section}{\numberline {6.1}Using Multiple Camera Setup}{49}{section.135}% +\contentsline {section}{\numberline {6.2}Better Landmark Filtering}{50}{section.137}% +\contentsline {section}{\numberline {6.3}Multi-Resolution Fitting}{50}{section.138}% +\vspace {2em} +\contentsline {chapter}{Bibliography}{53}{appendix*.139}% +\contentsline {appendix}{\chapternumberline {A}Thrift API Scheme}{57}{appendix.140}% +\vspace {2em} +\contentsline {chapter}{Declaration on Scientific Integrity}{59}{appendix*.142}% diff --git a/basilea.sty b/basilea.sty new file mode 100644 index 0000000..077ba51 --- /dev/null +++ b/basilea.sty @@ -0,0 +1,251 @@ +%% +%% This file is originally based on 'ECSthesis.cls' by Steve R. Gunn +%% and 'phdthesis.sty' by Jamie Stevens. In this new version many parts +%% have been replaced by calls to the memoir package. Many existing +%% templates using the memoir served as model. +%% +%% v.2.0 +%% +%% 2014, Ivan Giangreco +%% + +\NeedsTeXFormat{LaTeX2e} +\ProvidesPackage{basilea}[2014] + +\newcommand*{\basileaBibStyle}{thesis} + +%% +%% Provides the option to use \citet to get the authors name +%% Internally, this changes the bibliographystyle from thesis to plainnat +%% +\DeclareOption{citeauthor}{ + \renewcommand*{\basileaBibStyle}{plainnat} +} +%% +%% Provides the default option to not use \citet to get the authors name +%% Internally, this sets the bibliographystyle to thesis +%% +\DeclareOption{citenumber}{ + \renewcommand*{\basileaBibStyle}{thesis} +} + +\ExecuteOptions{citenumber} + +\ProcessOptions\relax + +\usepackage[ngerman, english]{babel} +\usepackage[utf8]{inputenc} + +% text to be set for frontpage +\newcommand*{\supervisor}[1]{\def\supervisorint{#1}} +% \newcommand*{\cosupervisor}[1]{\def\cosupervisorint{#1}} +\newcommand*{\examiner}[1]{\def\examinerint{#1}} +\newcommand*{\authors}[1]{\def\authorsint{#1}} +\newcommand*{\email}[1]{\def\emailint{#1}} +\newcommand*{\ulogo}[1]{\def\universitylogoint{#1}} +\newcommand*{\department}[1]{\def\departmentint{#1}} +\newcommand*{\faculty}[1]{\def\facultyint{#1}} +\newcommand*{\research}[1]{\def\researchgroupint{#1}} +\newcommand*{\thesistype}[1]{\def\thesistypeint{#1}} +\newcommand*{\immatriculnr}[1]{\def\immatriculnrint{#1}} + +\def\titleint{\@title} +\def\dateint{\@date} + +% settings for nomenclature (i.e. the abbreviations) +% - for using in text use e.g. \nomenclature{LED}{Light-Emitting Diode} +% this will print the text directly as Light-Emitting Diode (LED) +% - the abbreviations for the whole document can be printed with the command \thesisnomencl +% - invoke MakeIndex to create a *.nlo file (see also documentation of nomenclature package) +\usepackage{nomencl} +\newcommand*{\nomenclint}{} +\let\nomenclint\nomenclature +\renewcommand*{\nomenclature}[2]{\textit{#2} (#1) \nomenclint{#1}{#2}} + +\newcommand*{\thesisnomencl}{\renewcommand{\nomname}{\iflanguage{english}{Abbreviations}{Abkürzungsverzeichnis}}\printnomenclature\addcontentsline{toc}{chapter}{\iflanguage{english}{Abbreviations}{Abkürzungsverzeichnis}}} + +% graphics +% (see also floats section in this document for more options) +\usepackage{eso-pic} +\usepackage{everyshi} +\usepackage{ifthen} +\usepackage{calc} +\usepackage{pdfpages} +\graphicspath{{Figures/}} +\newsubfloat{figure} + +% definition of fonts (phv is Helvetica, pcr is Courrier) +\renewcommand{\sfdefault}{phv} +\renewcommand{\ttdefault}{pcr} + +% definition of colors +\usepackage{color,graphicx} +\definecolor{chaptercolor}{rgb}{.7,.7,.7} +\definecolor{emphasizecolor}{rgb}{.7,.7,.7} + +% margins +\setlrmarginsandblock{3.5cm}{3.5cm}{*} +\setulmarginsandblock{3.5cm}{3.5cm}{*} +\checkandfixthelayout + +% document organization +\setsecnumdepth{subsubsection} +\setcounter{tocdepth}{3} +\newcounter{dummy} +\newcommand\addtotoc[1]{ +\refstepcounter{dummy} +\addcontentsline{toc}{chapter}{#1}} + +% definition of chapter style (i.e. large number and text) +\setlength{\beforechapskip}{50pt} +\setlength{\afterchapskip}{1pt} +\newcommand*{\colorchapnumber}{\color{chaptercolor}} +\newcommand*{\fontchapnumber}{\usefont{T1}{phv}{b}{n}\fontsize{100}{130}\selectfont} +\newcommand*{\fontchaptitle}{\bfseries\huge} +\renewcommand*{\afterchapternum}{} +\renewcommand*{\printchaptername}{} +\setlength{\midchapskip}{20mm} +\renewcommand*{\chapternamenum}{} +\renewcommand*{\printchapternum}{\raggedleft{\colorchapnumber\fontchapnumber\thechapter}\par} +\renewcommand*{\printchaptertitle}[1]{\raggedleft\fontchaptitle{#1}\par \vspace{30pt} \nobreak} + +% definition of (subsub)section styles +\setsecheadstyle{\bfseries \Large} +\setbeforesecskip{25pt} +\setaftersecskip{1pt} + +\setsubsecheadstyle{\bfseries\large} +\setbeforesubsecskip{20pt} +\setaftersubsecskip{1pt} + +\setsubsubsecheadstyle{\bfseries\normalsize} +\setbeforesubsubsecskip{20pt} +\setaftersubsubsecskip{1pt} + +% definition of line spacing, indent of paragraph, etc. +\OnehalfSpacing +\setlength{\parindent}{0pt} +\setlength{\parskip}{0pt} +\raggedbottom + +% header, footer +\nouppercaseheads +\renewcommand{\chaptermark}[1]{\markboth{#1}{}} +\renewcommand{\sectionmark}[1]{\markright{#1}{}} + +\makepagestyle{thesis} +\makeevenhead{thesis}{\leftmark}{}{\thepage} +\makeoddhead{thesis}{\leftmark}{}{\thepage} +\makeheadrule{thesis}{\textwidth}{0.2pt} +\makeevenfoot{thesis}{}{}{} +\makeoddfoot{thesis}{}{}{} +\copypagestyle{chapter}{empty} + +% footnotes +\setlength{\skip\footins}{20pt} +\usepackage[hang]{footmisc} +\setlength{\footnotemargin}{10pt} +\usepackage{chngcntr} +\counterwithout{footnote}{chapter} + +% add highlighting possibilities +% use \hlt to highlight parts of text, e.g. \hlt{this is important} +\usepackage{soul} +\newcommand{\hlt}[1]{\hl{#1}} + +% various imports +\usepackage{amsmath,amsfonts,amssymb,amscd,amsthm} + +% clearing pages +\def\cleardoublepage{\clearpage\if@twoside \ifodd\c@page\else\hbox{}\thispagestyle{empty}\newpage\if@twocolumn\hbox{}\newpage\fi\fi\fi} +\newcommand*{\tmpcleardoublepage}{} +\let\tmpcleardoublepage\cleardoublepage + +% floats +\captionstyle{\raggedright} +\setfloatlocations{figure}{ht!} +\setlength{\intextsep}{10pt} +\setlength{\textfloatsep}{10pt} + +% definition of table of contents (i.e. title, header text, appearance in pdf-TOC, etc.) +\newcommand*{\thesistoc}{\cleardoublepage\renewcommand*{\contentsname}{\iflanguage{english}{Table of Contents}{Inhaltsverzeichnis}} \createplainmark{toc}{both}{\iflanguage{english}{Table of Contents}{Inhaltsverzeichnis}} \currentpdfbookmark{\iflanguage{english}{Table of Contents}{Inhaltsverzeichnis}}{toc} \tableofcontents*} +% adds the word "Appendix" in front of chapters in the appendix section +\renewcommand*{\cftappendixname}{\iflanguage{english}{Appendix}{Anhang}\space} + +% definition of bibliography +\newcommand*{\thesisbib}{\cleardoublepage\renewcommand*{\contentsname}{\iflanguage{english}{Bibliography}{Literaturverzeichnis}} \createplainmark{bib}{both}{\iflanguage{english}{Bibliography}{Literaturverzeichnis}}\bibliographystyle{\basileaBibStyle}\bibliography{thesis}} +\usepackage[square, numbers, comma, sort&compress]{natbib} + +% document structure +\newcommand*{\thesisfront}{\frontmatter \let\cleardoublepage\clearpage } +\newcommand*{\thesismain}{ \addtocontents{toc}{\vspace{2em}} \mainmatter \let\cleardoublepage\tmpcleardoublepage } +\newcommand*{\thesisappendix}{ \addtocontents{toc}{\vspace{2em}} \appendix \let\cleardoublepage\clearpage} + +\newcommand*{\thesisback}{ \addtocontents{toc}{\vspace{2em}} \backmatter \let\cleardoublepage\clearpage} + +\renewcommand\maketitle{ + \hypersetup{pdftitle={\@title}} + \hypersetup{pdfauthor=\authorsint} + \thispagestyle{empty} + {\raggedright\includegraphics{\universitylogoint}}\par + \begin{center} + \vspace{3cm} + {\Huge \bfseries \@title \par} + \vspace{0.5cm} + {\thesistypeint \par} + \vspace{3cm} + {\facultyint \par} + {\departmentint \par} + {\researchgroupint \par} + \vspace{1.5cm} + {\iflanguage{english}{Examiner}{Beurteiler}: \examinerint \par} + {\iflanguage{english}{Supervisor}{Zweitbeurteiler}: \supervisorint \par} + % {\iflanguage{english}{Co-Supervisor}{Co-Zweitbeurteiler}: \cosupervisorint \par} + \vspace{1.5cm} + {\authorsint \par} + {\emailint \par} + {\immatriculnrint \par} + \vfill + {\@date \par} + \end{center} +} + +% declaration +\newcommand{\formlabel}[2]{\vspace{0.25cm}\textbf{\footnotesize #1 --- #2}\\} + +% urls +\usepackage{url} +\urlstyle{same} + +% footnotes +\usepackage{chngcntr} +\counterwithout{footnote}{chapter} + +% math +\usepackage{amsmath} +\usepackage{amssymb} +\usepackage{amsfonts} + +% reference settings +\usepackage[pdfpagemode={UseOutlines},bookmarks=true,bookmarksopen=true, + bookmarksopenlevel=0,bookmarksnumbered=true,hypertexnames=false, + colorlinks,linkcolor={black},citecolor={black},urlcolor={black}, filecolor={black}, + pdfstartview={FitV}, breaklinks=true, unicode]{hyperref} +\usepackage[capitalize]{cleveref} + + +% todos +\usepackage[colorinlistoftodos, textwidth=4cm]{todonotes} % add ",disable" in [] to remove all todos, missing figures and the todo list +\newcommand{\todoEmpty}[2][]{\todo[fancyline, #1]{#2}} +\newcommand{\todoMissing}[2][]{\todoEmpty[color=magenta!80, linecolor=magenta!80, #1]{Missing: #2}} +\newcommand{\todoCheck}[2][]{\todoEmpty[color=red!80, linecolor=red!80, #1]{Check: #2}} +\newcommand{\todoRevise}[2][]{\todoEmpty[color=orange!80, linecolor=orange!80, #1]{Revise: #2}} +\newcommand{\todoCitation}[2][]{\todoEmpty[color=yellow!80, linecolor=yellow!80, #1]{Citation: #2}} +\newcommand{\todoLanguage}[2][]{\todoEmpty[color=blue!40!white, linecolor=blue!40!white, #1]{Language: #2}} +\newcommand{\todoQuestion}[2][]{\todoEmpty[color=green!80!white, linecolor=green!80!white, #1]{Question: #2}} +\newcommand{\todoNote}[2][]{\todoEmpty[color=black!20!white, linecolor=black!20!white, #1]{Note: #2}} +\newcommand{\todoFigure}[5]{\begin{figure}[#1]\centering\missingfigure[figwidth=#2]{#3}\caption{#4}\label{#5}\end{figure}} + +\endinput + +%% End of file 'basilea.sty' diff --git a/lstpatch.sty b/lstpatch.sty new file mode 100755 index 0000000..e53ec08 --- /dev/null +++ b/lstpatch.sty @@ -0,0 +1 @@ +%% %% This is file `lstpatch.sty', generated manually. %% %% (w)(c) 2004 Carsten Heinz %% %% This file may be distributed under the terms of the LaTeX Project Public %% License from CTAN archives in directory macros/latex/base/lppl.txt. %% Either version 1.0 or, at your option, any later version. %% %% Send comments and ideas on the package, error reports and additional %% programming languages to . %% %% This patch file will remove the following bugs from the listings package. %% Each item contains the bug finder with date of report and first bug fix %% version, a short description of the problem, and the reason for the bug %% in parenthesis. %% %% 1) Frank Atanassow, 2004/10/07, 1.3b %% %% space after mathescape is not preserved %% (\lst@newlines>0) %% %% 2) Benjamin Lings, 2004/10/15, 1.3b (2004/10/17) %% %% \usepackage{xy,listings} yields: %% "Forbidden control sequence found while scanning use of \lst@lExtend" %% (xy-pic correctly resets catcode of ^^L (to active), which is \outer) %% %% %% The following features are added to the base package. %% %% 1.3a (2004/09/07) %% %% a) H I G H L Y E X P E R I M E N T A L %% %% Use the options %% rangeprefix= %% rangesuffix= %% %% rangebeginprefix= %% rangebeginsuffix= %% %% rangeendprefix= %% rangeendsuffix= %% %% includerangemarker=true|false %% together with %% firstline= %% lastline= %% or %% linerange={-, %% -, ...} %% The according markers in the source code are %% %% for begin respectively end of range. Moreover, one can use %% includerangemarker=true|false %% to show or hide the range markers in the output. %% %% 1.3b (2004/10/17) %% %% b) multicols= (requires loaded multicol package) %% %% \lst@CheckVersion{1.3} {\typeout{^^J% ***^^J% *** This is a patch for listings 1.3, but you're using^^J% *** version \lst@version.^^J% ***^^J *** Patch file not loaded.^^J% ***^^J}% \endinput } \def\fileversion{1.3b} \def\filedate{2004/10/17} \ProvidesFile{lstpatch.sty}[\filedate\space\fileversion\space (Carsten Heinz)] % % 0) Insert % after #1. \def\@@xbitor #1{\@tempcntb \count#1% \ifnum \@tempcnta =\z@ \else \divide\@tempcntb\@tempcnta \ifodd\@tempcntb \@testtrue\fi \fi} % % 1) Reset \lst@newlines at end of escape. \def\lstpatch@escape{% \gdef\lst@Escape##1##2##3##4{% \lst@CArgX ##1\relax\lst@CDefX {}% {\lst@ifdropinput\else \lst@TrackNewLines\lst@OutputLostSpace \lst@XPrintToken \lst@InterruptModes \lst@EnterMode{\lst@TeXmode}{\lst@modetrue}% \ifx\^^M##2% \lst@CArg ##2\relax\lst@ActiveCDefX {}% {\lst@escapeend ##4\lst@LeaveAllModes\lst@ReenterModes}% {\lst@MProcessListing}% \else \lst@CArg ##2\relax\lst@ActiveCDefX {}% {\lst@escapeend ##4\lst@LeaveAllModes\lst@ReenterModes \lst@newlines\z@ \lst@whitespacefalse}% {}% \fi ##3\lst@escapebegin \fi}% {}}% } % % 2) Deactivate \outer definition of ^^L temporarily (inside and outside % of \lst@ScanChars) and restore \catcode at end of package. \begingroup \catcode12=\active\let^^L\@empty \gdef\lst@ScanChars{% \let\lsts@ssL^^L% \def^^L{\par}% \lst@GetChars\lst@RestoreOrigCatcodes\@ne {128}% \let^^L\lsts@ssL \lst@GetChars\lst@RestoreOrigExtendedCatcodes{128}{256}} \endgroup \lst@lAddTo\lst@RestoreCatcodes{\catcode12\active} % % a) Let's start with the options: \lst@Key{rangeprefix}\relax{\def\lst@rangebeginprefix{#1}% \def\lst@rangeendprefix{#1}} \lst@Key{rangesuffix}\relax{\def\lst@rangebeginsuffix{#1}% \def\lst@rangeendsuffix{#1}} \lst@Key{rangebeginprefix}{}{\def\lst@rangebeginprefix{#1}} \lst@Key{rangebeginsuffix}{}{\def\lst@rangebeginsuffix{#1}} \lst@Key{rangeendprefix}{}{\def\lst@rangeendprefix{#1}} \lst@Key{rangeendsuffix}{}{\def\lst@rangeendsuffix{#1}} \lst@Key{includerangemarker}{true}[t]{\lstKV@SetIf{#1}\lst@ifincluderangemarker} % % The key is a redefinition of \lst@GLI@ checking for numbers. \def\lst@GLI@#1-#2-#3\@nil{% \lst@IfNumber{#1}% {\ifx\@empty#1\@empty \let\lst@firstline\@ne \else \def\lst@firstline{#1\relax}% \fi \ifx\@empty#3\@empty \def\lst@lastline{9999999\relax}% \else \ifx\@empty#2\@empty \let\lst@lastline\lst@firstline \else \def\lst@lastline{#2\relax}% \fi \fi}% % % If we've found a general marker, we set firstline and lastline to 9999999. % This prevents (almost) anything to be printed for now. {\def\lst@firstline{9999999\relax}% \let\lst@lastline\lst@firstline % % We add the prefixes and suffixes to the markers. \let\lst@rangebegin\lst@rangebeginprefix \lst@AddTo\lst@rangebegin{#1}\lst@Extend\lst@rangebegin\lst@rangebeginsuffix \ifx\@empty#3\@empty \let\lst@rangeend\lst@rangeendprefix \lst@AddTo\lst@rangeend{#1}\lst@Extend\lst@rangeend\lst@rangeendsuffix \else \ifx\@empty#2\@empty \let\lst@rangeend\@empty \else \let\lst@rangeend\lst@rangeendprefix \lst@AddTo\lst@rangeend{#2}\lst@Extend\lst@rangeend\lst@rangeendsuffix \fi \fi % The following definition will be executed in the SelectCharTable hook % and here right now if we are already processing a listing. \global\def\lst@DefRange{\expandafter\lst@CArgX\lst@rangebegin\relax\lst@DefRangeB}% \ifnum\lst@mode=\lst@Pmode \expandafter\lst@DefRange \fi}} % \lst@DefRange is not inserted via a hook anymore. Instead it is now called % directly from \lst@SelectCharTable. This was necessary to get rid of an % interference with the escape-to-LaTeX-feature. The bug was reported by % \lsthelper{Michael~Bachmann}{2004/07/21}{Keine label-Referenzierung % m\"oglich...}. Another chance is due to the same bug: \lst@DefRange is % redefined globally when the begin of code is found, see below. The bug was % reported by \lsthelper{Tobias~Rapp}{2004/04/06}{undetected end of range if % listing crosses page break} \lsthelper{Markus~Luisser}{2004/08/13}{Bug mit % 'linerangemarker' in umgebrochenen listings} %\lst@AddToHook{SelectCharTable}{\lst@DefRange} \lst@AddToHookExe{DeInit}{\global\let\lst@DefRange\@empty} % % Actually defining the marker (via \lst@GLI@, \lst@DefRange, \lst@CArgX as % seen above) is similar to \lst@DefDelimB---except that we unfold the first % parameter and use different , 
, and  statements.
\def\lst@DefRangeB#1#2{\lst@DefRangeB@#1#2}
\def\lst@DefRangeB@#1#2#3#4{%
    \lst@CDef{#1{#2}{#3}}#4{}%
    {\lst@ifincluderangemarker
         \lst@LeaveMode
         \let#1#4%
         \lst@DefRangeEnd
         \lst@InitLstNumber
     \else
         \@tempcnta\lst@lineno \advance\@tempcnta\@ne
         \edef\lst@firstline{\the\@tempcnta\relax}%
         \gdef\lst@OnceAtEOL{\let#1#4\lst@DefRangeEnd}%
         \lst@InitLstNumber
     \fi
	 \global\let\lst@DefRange\lst@DefRangeEnd
     \lst@CArgEmpty}%
    \@empty}
%
% Modify labels and define |\lst@InitLstNumber| used above.
% \lsthelper{Omair-Inam~Abdul-Matin}{2004/05/10}{experimental linerange
% feature does not work with firstnumber}
\def\lstpatch@labels{%
\gdef\lst@SetFirstNumber{%
    \ifx\lst@firstnumber\@undefined
        \@tempcnta 0\csname\@lst no@\lst@intname\endcsname\relax
        \ifnum\@tempcnta=\z@ \else
            \lst@nololtrue
            \advance\@tempcnta\lst@advancenumber
            \edef\lst@firstnumber{\the\@tempcnta\relax}%
        \fi
    \fi}%
}
\lst@AddToAtTop\lsthk@PreInit
    {\ifx\lst@firstnumber\@undefined
         \def\lst@firstnumber{\lst@lineno}%
     \fi}
\def\lst@InitLstNumber{%
     \global\c@lstnumber\lst@firstnumber
     \global\advance\c@lstnumber\lst@advancenumber
     \global\advance\c@lstnumber-\lst@advancelstnum
     \ifx \lst@firstnumber\c@lstnumber
         \global\advance\c@lstnumber-\lst@advancelstnum
     \fi}
%
%    The end-marker is defined if and only if it's not empty. The definition is
%    similar to \lst@DefDelimE---with the above exceptions and except that we
%    define the re-entry point \lst@DefRangeE@@ as it is defined in the new
%    version of \lst@MProcessListing above.
\def\lst@DefRangeEnd{%
    \ifx\lst@rangeend\@empty\else
        \expandafter\lst@CArgX\lst@rangeend\relax\lst@DefRangeE
    \fi}
\def\lst@DefRangeE#1#2{\lst@DefRangeE@#1#2}
\def\lst@DefRangeE@#1#2#3#4{%
    \lst@CDef{#1#2{#3}}#4{}%
    {\let#1#4%
     \edef\lst@lastline{\the\lst@lineno\relax}%
     \lst@DefRangeE@@}%
    \@empty}
\def\lst@DefRangeE@@#1\@empty{%
    \lst@ifincluderangemarker
        #1\lst@XPrintToken
    \fi
    \lst@LeaveModeToPmode
    \lst@BeginDropInput{\lst@Pmode}}
%
\def\lst@LeaveModeToPmode{%
    \ifnum\lst@mode=\lst@Pmode
        \expandafter\lsthk@EndGroup
    \else
        \expandafter\egroup\expandafter\lst@LeaveModeToPmode
    \fi}
%
%    Eventually we shouldn't forget to install \lst@OnceAtEOL, which must
%    also be called in \lst@MSkipToFirst.
\lst@AddToHook{EOL}{\lst@OnceAtEOL\global\let\lst@OnceAtEOL\@empty}
\gdef\lst@OnceAtEOL{}% Init
\def\lst@MSkipToFirst{%
    \global\advance\lst@lineno\@ne
    \ifnum \lst@lineno=\lst@firstline
        \def\lst@next{\lst@LeaveMode \global\lst@newlines\z@
        \lst@OnceAtEOL \global\let\lst@OnceAtEOL\@empty
        \lst@InitLstNumber % Added to work with modified \lsthk@PreInit.
        \lsthk@InitVarsBOL
        \lst@BOLGobble}%
        \expandafter\lst@next
    \fi}
\def\lst@SkipToFirst{%
    \ifnum \lst@lineno<\lst@firstline
        \def\lst@next{\lst@BeginDropInput\lst@Pmode
        \lst@Let{13}\lst@MSkipToFirst
        \lst@Let{10}\lst@MSkipToFirst}%
        \expandafter\lst@next
    \else
        \expandafter\lst@BOLGobble
    \fi}
%
%    Finally the service macro \lst@IfNumber:
\def\lst@IfNumber#1{%
    \ifx\@empty#1\@empty
        \let\lst@next\@firstoftwo
    \else
        \lst@IfNumber@#1\@nil
    \fi
    \lst@next}
\def\lst@IfNumber@#1#2\@nil{%
    \let\lst@next\@secondoftwo
    \ifnum`#1>47\relax \ifnum`#1>57\relax\else
        \let\lst@next\@firstoftwo
    \fi\fi}
%
% b) The following is known to fail with some keys.
\lst@Key{multicols}{}{\@tempcnta=0#1\relax\def\lst@multicols{#1}}
\def\lst@Init#1{%
    \begingroup
    \ifx\lst@float\relax\else
        \edef\@tempa{\noexpand\lst@beginfloat{lstlisting}[\lst@float]}%
        \expandafter\@tempa
    \fi
% chmod begin
    \ifx\lst@multicols\@empty\else
        \edef\lst@next{\noexpand\multicols{\lst@multicols}}
        \expandafter\lst@next
    \fi
% chmod end
    \ifhmode\ifinner \lst@boxtrue \fi\fi
    \lst@ifbox
        \lsthk@BoxUnsafe
        \hbox to\z@\bgroup
             $\if t\lst@boxpos \vtop
        \else \if b\lst@boxpos \vbox
        \else \vcenter \fi\fi
        \bgroup \par\noindent
    \else
        \lst@ifdisplaystyle
            \lst@EveryDisplay
            \par\penalty-50\relax
            \vspace\lst@aboveskip
        \fi
    \fi
    \normalbaselines
    \abovecaptionskip\lst@abovecaption\relax
    \belowcaptionskip\lst@belowcaption\relax
    \lst@MakeCaption t%
    \lsthk@PreInit \lsthk@Init
    \lst@ifdisplaystyle
        \global\let\lst@ltxlabel\@empty
        \if@inlabel
            \lst@ifresetmargins
                \leavevmode
            \else
                \xdef\lst@ltxlabel{\the\everypar}%
                \lst@AddTo\lst@ltxlabel{%
                    \global\let\lst@ltxlabel\@empty
                    \everypar{\lsthk@EveryLine\lsthk@EveryPar}}%
            \fi
        \fi
        \everypar\expandafter{\lst@ltxlabel
                              \lsthk@EveryLine\lsthk@EveryPar}%
    \else
        \everypar{}\let\lst@NewLine\@empty
    \fi
    \lsthk@InitVars \lsthk@InitVarsBOL
    \lst@Let{13}\lst@MProcessListing
    \let\lst@Backslash#1%
    \lst@EnterMode{\lst@Pmode}{\lst@SelectCharTable}%
    \lst@InitFinalize}
\def\lst@DeInit{%
    \lst@XPrintToken \lst@EOLUpdate
    \global\advance\lst@newlines\m@ne
    \lst@ifshowlines
        \lst@DoNewLines
    \else
        \setbox\@tempboxa\vbox{\lst@DoNewLines}%
    \fi
    \lst@ifdisplaystyle \par\removelastskip \fi
    \lsthk@ExitVars\everypar{}\lsthk@DeInit\normalbaselines\normalcolor
    \lst@MakeCaption b%
    \lst@ifbox
        \egroup $\hss \egroup
        \vrule\@width\lst@maxwidth\@height\z@\@depth\z@
    \else
        \lst@ifdisplaystyle
            \par\penalty-50\vspace\lst@belowskip
        \fi
    \fi
% chmod begin
    \ifx\lst@multicols\@empty\else
        \def\lst@next{\global\let\@checkend\@gobble
                      \endmulticols
                      \global\let\@checkend\lst@@checkend}
        \expandafter\lst@next
    \fi
% chmod end
    \ifx\lst@float\relax\else
        \expandafter\lst@endfloat
    \fi
    \endgroup}
\let\lst@@checkend\@checkend
%%
\endinput
%%
%% End of file `lstpatch.sty'.
\ No newline at end of file
diff --git a/thesis.bib b/thesis.bib
new file mode 100644
index 0000000..ba92091
--- /dev/null
+++ b/thesis.bib
@@ -0,0 +1,561 @@
+@article{tu2019joint,
+  title={Joint 3D Face Reconstruction and Dense Face Alignment from A Single Image with 2D-Assisted Self-Supervised Learning},
+  author={Tu, Xiaoguang and Zhao, Jian and Jiang, Zihang and Luo, Yao and Xie, Mei and Zhao, Yang and He, Linxiao and Ma, Zheng and Feng, Jiashi},
+  journal={arXiv preprint arXiv:1903.09359},
+  year={2019}
+}
+
+@article{Guo_2019,
+   title={CNN-Based Real-Time Dense Face Reconstruction with Inverse-Rendered Photo-Realistic Face Images},
+   volume={41},
+   ISSN={1939-3539},
+   url={http://dx.doi.org/10.1109/tpami.2018.2837742},
+   DOI={10.1109/tpami.2018.2837742},
+   number={6},
+   journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
+   publisher={Institute of Electrical and Electronics Engineers (IEEE)},
+   author={Guo, Yudong and Zhang, Juyong and Cai, Jianfei and Jiang, Boyi and Zheng, Jianmin},
+   year={2019},
+   month={Jun},
+   pages={1294–1307}
+}
+
+
+@Article{betschard2016,
+	author = {Christoph Betschard},
+	title = {Fitting a 3D Morphable Face Model to Captures from Consumer Depth Cameras},
+	journal = {University of Basel, Department of Mathematics and Computer Science, Master's Thesis},
+	institution = {University of Basel, Department of Mathematics and Computer Science},
+	year = {2016}
+}
+
+@Article{Schoenborn2017,
+	author="Sch{\"o}nborn, Sandro
+	and Egger, Bernhard
+	and Morel-Forster, Andreas
+	and Vetter, Thomas",
+	title="Markov Chain Monte Carlo for Automated Face Image Analysis",
+	journal="International Journal of Computer Vision",
+	year="2017",
+	month="Jun",
+	day="01",
+	volume="123",
+	number="2",
+	pages="160--183",
+	abstract="We present a novel fully probabilistic method to interpret a single face image with the 3D Morphable Model. The new method is based on Bayesian inference and makes use of unreliable image-based information. Rather than searching a single optimal solution, we infer the posterior distribution of the model parameters given the target image. The method is a stochastic sampling algorithm with a propose-and-verify architecture based on the Metropolis--Hastings algorithm. The stochastic method can robustly integrate unreliable information and therefore does not rely on feed-forward initialization. The integrative concept is based on two ideas, a separation of proposal moves and their verification with the model (Data-Driven Markov Chain Monte Carlo), and filtering with the Metropolis acceptance rule. It does not need gradients and is less prone to local optima than standard fitters. We also introduce a new collective likelihood which models the average difference between the model and the target image rather than individual pixel differences. The average value shows a natural tendency towards a normal distribution, even when the individual pixel-wise difference is not Gaussian. We employ the new fitting method to calculate posterior models of 3D face reconstructions from single real-world images. A direct application of the algorithm with the 3D Morphable Model leads us to a fully automatic face recognition system with competitive performance on the Multi-PIE database without any database adaptation.",
+	issn="1573-1405",
+	doi="10.1007/s11263-016-0967-5",
+	url="https://doi.org/10.1007/s11263-016-0967-5"
+}
+
+@Article{Schoenborn2014,
+	author = {Sch{\"o}nborn, Sandro},
+	title = {Markov Chain Monte Carlo for
+Integrated Face Image Analysis},
+	journal = {University of Basel, Department of Mathematics and Computer Science, PhD Thesis},
+	institution = {University of Basel, Department of Mathematics and Computer Science},
+	year = {2014},
+}
+
+@InProceedings{10.1007/978-3-642-40602-7_11,
+	author="Sch{\"o}nborn, Sandro
+	and Forster, Andreas
+	and Egger, Bernhard
+	and Vetter, Thomas",
+	editor="Weickert, Joachim
+	and Hein, Matthias
+	and Schiele, Bernt",
+	title="A Monte Carlo Strategy to Integrate Detection and Model-Based Face Analysis",
+	booktitle="Pattern Recognition",
+	year="2013",
+	publisher="Springer Berlin Heidelberg",
+	address="Berlin, Heidelberg",
+	pages="101--110",
+	abstract="We present a novel probabilistic approach for fitting a statistical model to an image. A 3D Morphable Model (3DMM) of faces is interpreted as a generative (Top-Down) Bayesian model. Random Forests are used as noisy detectors (Bottom-Up) for the face and facial landmark positions. The Top-Down and Bottom-Up parts are then combined using a Data-Driven Markov Chain Monte Carlo Method (DDMCMC). As core of the integration, we use the Metropolis-Hastings algorithm which has two main advantages. First, the algorithm can handle unreliable detections and therefore does not need the detectors to take an early and possible wrong hard decision before fitting. Second, it is open for integration of various cues to guide the fitting process. Based on the proposed approach, we implemented a completely automatic, pose and illumination invariant face recognition application. We are able to train and test the building blocks of our application on different databases. The system is evaluated on the Multi-PIE database and reaches state of the art performance.",
+	isbn="978-3-642-40602-7"
+}
+
+
+@book{vision2013,
+	author = {Andrea Fossati and Juergen Gall and Helmut Grabner and Xiaofeng Ren and Kurt Konolige},
+	title = {Consumer Depth Cameras for Computer Vision},
+	year = {2013},
+	publisher = {Springer-Verlag London},
+	isbn = {978-1-4471-4640-7},
+	doi = {10.1007/978-1-4471-4640-7},
+	url = {https://doi.org/10.1007/978-1-4471-4640-7},
+}
+
+@Inbook{Henry2014,
+	author="Henry, Peter
+	and Krainin, Michael
+	and Herbst, Evan
+	and Ren, Xiaofeng
+	and Fox, Dieter",
+	editor="Khatib, Oussama
+	and Kumar, Vijay
+	and Sukhatme, Gaurav",
+	title="RGB-D Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments",
+	bookTitle="Experimental Robotics: The 12th International Symposium on Experimental Robotics",
+	year="2014",
+	publisher="Springer Berlin Heidelberg",
+	address="Berlin, Heidelberg",
+	pages="477--491",
+	abstract="RGB-D cameras are novel sensing systems that capture RGB images along with per-pixel depth information. In this paper we investigate how such cameras can be used in the context of robotics, specifically for building dense 3D maps of indoor environments. Such maps have applications in robot navigation, manipulation, semantic mapping, and telepresence. We present RGB-D Mapping, a full 3D mapping system that utilizes a novel joint optimization algorithm combining visual features and shape-based alignment. Visual and depth information are also combined for view-based loop closure detection, followed by pose optimization to achieve globally consistent maps.We evaluate RGB-D Mapping on two large indoor environments, and show that it effectively combines the visual and shape information available from RGB-D cameras.",
+	isbn="978-3-642-28572-1",
+	doi="10.1007/978-3-642-28572-1_33",
+	url="https://doi.org/10.1007/978-3-642-28572-1_33"
+}
+
+@Inbook{Smisek2013,
+	author="Smisek, Jan
+	and Jancosek, Michal
+	and Pajdla, Tomas",
+	editor="Fossati, Andrea
+	and Gall, Juergen
+	and Grabner, Helmut
+	and Ren, Xiaofeng
+	and Konolige, Kurt",
+	title="3D with Kinect",
+	bookTitle="Consumer Depth Cameras for Computer Vision: Research Topics and Applications",
+	year="2013",
+	publisher="Springer London",
+	address="London",
+	pages="3--25",
+	abstract="We analyze Kinect as a 3D measuring device, experimentally investigate depth measurement resolution and error properties, and make a quantitative comparison of Kinect accuracy with stereo reconstruction from SLR cameras and a 3D-TOF camera. We propose a Kinect geometrical model and its calibration procedure providing an accurate calibration of Kinect 3D measurement and Kinect cameras. We compare our Kinect calibration procedure with its alternatives available on Internet, and integrate it into an SfM pipeline where 3D measurements from a moving Kinect are transformed into a common coordinate system, by computing relative poses from matches in its color camera.",
+	isbn="978-1-4471-4640-7",
+	doi="10.1007/978-1-4471-4640-7_1",
+	url="https://doi.org/10.1007/978-1-4471-4640-7_1"
+}
+
+@Inbook{Weiss2013,
+	author="Weiss, Alexander
+	and Hirshberg, David
+	and Black, Michael J.",
+	editor="Fossati, Andrea
+	and Gall, Juergen
+	and Grabner, Helmut
+	and Ren, Xiaofeng
+	and Konolige, Kurt",
+	title="Home 3D Body Scans from a Single Kinect",
+	bookTitle="Consumer Depth Cameras for Computer Vision: Research Topics and Applications",
+	year="2013",
+	publisher="Springer London",
+	address="London",
+	pages="99--117",
+	abstract="The 3D shape of the human body is useful for applications in fitness, games, and apparel. Accurate body scanners, however, are expensive, limiting the availability of 3D body models. Although there has been a great deal of interest recently in the use of active depth sensing cameras, such as the Microsoft Kinect, for human pose tracking, little has been said about the related problem of human shape estimation. We present a method for human shape reconstruction from noisy monocular image and range data using a single inexpensive commodity sensor. The approach combines low-resolution image silhouettes with coarse range data to estimate a parametric model of the body. Accurate 3D shape estimates are obtained by combining multiple monocular views of a person moving in front of the sensor. To cope with varying body pose, we use a SCAPE body model which factors 3D body shape and pose variations. This enables the estimation of a single consistent shape, while allowing pose to vary. Additionally, we describe a novel method to minimize the distance between the projected 3D body contour and the image silhouette that uses analytic derivatives of the objective function. We use a simple method to estimate standard body measurements from the recovered SCAPE model and show that the accuracy of our method is competitive with commercial body scanning systems costing orders of magnitude more.",
+	isbn="978-1-4471-4640-7",
+	doi="10.1007/978-1-4471-4640-7_6",
+	url="https://doi.org/10.1007/978-1-4471-4640-7_6"
+}
+
+@inproceedings{Blanz:1999:MMS:311535.311556,
+ author = {Blanz, Volker and Vetter, Thomas},
+ title = {A Morphable Model for the Synthesis of 3D Faces},
+ booktitle = {Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques},
+ series = {SIGGRAPH '99},
+ year = {1999},
+ isbn = {0-201-48560-5},
+ pages = {187--194},
+ numpages = {8},
+ url = {http://dx.doi.org/10.1145/311535.311556},
+ doi = {10.1145/311535.311556},
+ acmid = {311556},
+ publisher = {ACM Press/Addison-Wesley Publishing Co.},
+ address = {New York, NY, USA},
+ keywords = {computer vision, facial animation, facial modeling, morphing, photogrammetry, registration},
+} 
+
+@incollection{Romdhani3DM,
+title = "CHAPTER 4 - 3D MORPHABLE FACE MODEL, A UNIFIED APPROACH FOR ANALYSIS AND SYNTHESIS OF IMAGES",
+editor = "Wenyi Zhao and Rama Chellappa",
+booktitle = "Face Processing",
+publisher = "Academic Press",
+address = "Burlington",
+pages = "127 - 158",
+year = "2006",
+isbn = "978-0-12-088452-0",
+doi = "https://doi.org/10.1016/B978-012088452-0/50005-4",
+url = "http://www.sciencedirect.com/science/article/pii/B9780120884520500054"
+}
+
+@misc{D415,
+  title = {Intel® RealSense™ D415 Description},
+  howpublished = {\url{https://www.intelrealsense.com}},
+  note = {Accessed: 2019-04-12}
+}
+
+@misc{D415_SDK,
+  title = {Intel® RealSense™ SDK 2.0},
+  howpublished = {\url{https://github.com/IntelRealSense/librealsense
+}},
+  note = {Accessed: 2019-04-12}
+}
+
+@article{oliverwisler18,
+	title = {3D-Morphable Model Fitting incorporating contour information},
+	author = {Oliver Wisler},
+	year = {2018},
+	journal = {University of Basel, Department of Mathematics and Computer Science, Master's Thesis},
+	institution = {University of Basel, Department of Mathematics and Computer Science}
+}
+
+@misc{lmdetect,
+  title = {Facial mapping (landmarks) with Dlib + Python},
+  howpublished = {\url{https://towardsdatascience.com/facial-mapping-landmarks-with-dlib-python-160abcf7d672}},
+  note = {Accessed: 2019-04-16}
+
+}
+@misc{scalismo,
+  title = {Scalismo - Scalable Image Analysis and Shape Modelling},
+  howpublished = {\url{https://github.com/unibas-gravis/scalismo
+}},
+  note = {Accessed: 2019-04-16}
+
+}
+
+@misc{dlib, 
+	title = {dlib - A toolkit for making real world machine learning and data analysis applications in C++},
+	howpublished = {\url{http://dlib.net}},
+	note = {Accessed: 2019-04-19}
+}
+
+@misc{scalismofaces,
+  title = {Scalable Image Analysis and Shape Modelling: Module to work with 2d images, with a focus on face images},
+  howpublished = {\url{https://github.com/unibas-gravis/scalismo-faces
+}},
+  note = {Accessed: 2019-04-16}
+
+}
+
+@misc{opencv,
+  title = {Open Source Computer Vision Library(OpenCV)},
+  howpublished = {\url{https://opencv.org/
+}},
+  note = {Accessed: 2019-04-16}
+
+}
+@misc{thrift,
+  title = {The Apache Thrift software framework},
+  howpublished = {\url{https://thrift.apache.org/},\url{https://en.wikipedia.org/wiki/Apache_Thrift
+
+}},
+  note = {Accessed: 2019-04-20}
+
+}
+
+@misc{rs-projection,
+	title={Projection in RealSense SDK 2.0},
+	howpublished = {\url{https://dev.intelrealsense.com/docs/projection-in-intel-realsense-sdk-20#section-point-coordinates}},
+	note={Access Date: 2019-10-01}
+}
+
+
+
+@ARTICLE{121791, 
+author={P. J. {Besl} and N. D. {McKay}}, 
+journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
+title={A method for registration of 3-D shapes}, 
+year={1992}, 
+volume={14}, 
+number={2}, 
+pages={239-256}, 
+keywords={computational geometry;convergence of numerical methods;iterative methods;optimisation;pattern recognition;picture processing;3D shape registration;pattern recognition;point set registration;iterative closest point;geometric entity;mean-square distance metric;convergence;geometric model;Solid modeling;Motion estimation;Iterative closest point algorithm;Iterative algorithms;Testing;Inspection;Shape measurement;Iterative methods;Convergence;Quaternions}, 
+doi={10.1109/34.121791}, 
+ISSN={0162-8828}, 
+month={Feb},
+}
+
+@INPROCEEDINGS{6618317, 
+author={G. P. {Meyer} and M. N. {Do}}, 
+booktitle={2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW)}, 
+title={Real-time 3D face modeling with a commodity depth camera}, 
+year={2013}, 
+volume={}, 
+number={}, 
+pages={1-4}, 
+keywords={cameras;face recognition;image registration;image segmentation;interactive devices;solid modelling;real-time 3D face modeling;commodity depth camera;Kinect sensor;user head depth image registration;high-quality 3D face models;user head depth image integration;fixed depth camera;KinectFusion system;depth image segmentation;Three-dimensional displays;Solid modeling;Face;Cameras;Image segmentation;Computational modeling;3D Face Modeling;Kinect;Real-time}, 
+doi={10.1109/ICMEW.2013.6618317}, 
+ISSN={}, 
+month={July},}
+
+@INPROCEEDINGS{6334272, 
+author={M. {Hernandez} and J. {Choi} and G. {Medioni}}, 
+booktitle={2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO)}, 
+title={Laser scan quality 3-D face modeling using a low-cost depth camera}, 
+year={2012}, 
+volume={}, 
+number={}, 
+pages={1995-1999}, 
+keywords={cameras;face recognition;graphics processing units;image registration;image representation;iterative methods;object detection;optical scanners;smoothing methods;spatiotemporal phenomena;laser scan quality;3D face model;depth camera;geometric model;star shaped object;object representation;cylindrical coordinate;filtering operation;depth image;3D points cloud;image registration;GPU;graphics processing unit;ICP;iterative closest point;ICP algorithm;facial expression;occlusion;temporal smoothing;spatial smoothing;Face;Computational modeling;Solid modeling;Estimation;Image reconstruction;Cameras;Laser modes;Kinect;face modeling;graphics}, 
+doi={}, 
+ISSN={2076-1465}, 
+month={Aug},}
+
+@Inproceedings{kinectfusion-real-time-dense-surface-mapping-and-tracking,
+	author = {Newcombe, Richard A. and  and  and  and Davison, Andrew J. and  and Shotton, Jamie and Hodges, Steve and Fitzgibbon, Andrew},
+	title = {KinectFusion: Real-Time Dense Surface Mapping and Tracking},
+	booktitle = {IEEE ISMAR},
+	year = {2011},
+	month = {October},
+	abstract = {
+	We present a system for accurate real-time mapping of complex and arbitrary indoor scenes in variable lighting conditions, using only a moving low-cost depth camera and commodity graphics hardware. We fuse all of the depth data streamed from a Kinect sensor into a single global implicit surface model of the observed scene in real-time. The current sensor pose is simultaneously obtained by tracking the live depth frame relative to the global model using a coarse-to-fine iterative closest point (ICP) algorithm, which uses all of the observed depth data available. We demonstrate the advantages of tracking against the growing full surface model compared with frame-to-frame tracking, obtaining tracking and mapping results in constant time within room sized scenes with limited drift and high accuracy. We also show both qualitative and quantitative results relating to various aspects of our tracking and mapping system. Modelling of natural scenes, in real-time with only commodity sensor and GPU hardware, promises an exciting step forward in augmented reality (AR), in particular, it allows dense surfaces to be reconstructed in real-time, with a level of detail and robustness beyond any solution yet presented using passive computer vision.
+	},
+	publisher = {IEEE},
+	url = {https://www.microsoft.com/en-us/research/publication/kinectfusion-real-time-dense-surface-mapping-and-tracking/},
+	edition = {IEEE ISMAR},
+}
+
+@INPROCEEDINGS{924423, 
+author={S. {Rusinkiewicz} and M. {Levoy}}, 
+booktitle={Proceedings Third International Conference on 3-D Digital Imaging and Modeling}, 
+title={Efficient variants of the ICP algorithm}, 
+year={2001}, 
+volume={}, 
+number={}, 
+pages={145-152}, 
+keywords={distance measurement;image processing;minimisation;real-time systems;iterative closest point algorithm;geometric alignment;three-dimensional models;minimization strategy;nearly-flat meshes;inscribed surfaces;uniform sampling;range images;real-time 3D model acquisition;model-based tracking;Iterative closest point algorithm;Iterative algorithms;Convergence;Solid modeling;Iterative methods;Geometry;Layout;Minimization methods;Image sampling;Rough surfaces}, 
+doi={10.1109/IM.2001.924423}, 
+ISSN={}, 
+month={May},}
+
+@ARTICLE{1227983, 
+author={V. {Blanz} and T. {Vetter}}, 
+journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
+title={Face recognition based on fitting a 3D morphable model}, 
+year={2003}, 
+volume={25}, 
+number={9}, 
+pages={1063-1074}, 
+keywords={face recognition;solid modelling;image morphing;lighting;image texture;image representation;visual databases;face recognition;3D morphable model fitting;pose variations;profile view;frontal views;illuminations;3D shape;CMU-PIE database;FERET database;image database;shadows;specular reflections;image texture;statistical morphable model;face identification;image formation;3D space;computer graphics;Face recognition;Lighting;Shape;Head;Deformable models;Computational modeling;Computer simulation;Computer graphics;Image databases;Image recognition}, 
+doi={10.1109/TPAMI.2003.1227983}, 
+ISSN={0162-8828}, 
+month={Sep.},}
+
+@article{Schonborn:2015:BMG:2798342.2798359,
+ author = {Sch\"{o}nborn, Sandro and Egger, Bernhard and Forster, Andreas and Vetter, Thomas},
+ title = {Background Modeling for Generative Image Models},
+ journal = {Comput. Vis. Image Underst.},
+ issue_date = {July 2015},
+ volume = {136},
+ number = {C},
+ month = jul,
+ year = {2015},
+ issn = {1077-3142},
+ pages = {117--127},
+ numpages = {11},
+ url = {http://dx.doi.org/10.1016/j.cviu.2015.01.008},
+ doi = {10.1016/j.cviu.2015.01.008},
+ acmid = {2798359},
+ publisher = {Elsevier Science Inc.},
+ address = {New York, NY, USA},
+ keywords = {Bayesian model, Face analysis, Face model, Generative models, Implicit background models, Morphable Model},
+} 
+@ARTICLE{1000239, 
+author={ Zhuowen Tu and Song-Chun Zhu }, 
+journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
+title={Image segmentation by data-driven Markov chain Monte Carlo}, 
+year={2002}, 
+volume={24}, 
+number={5}, 
+pages={657-673}, 
+keywords={image segmentation;Markov processes;Monte Carlo methods;Data-Driven Markov Chain Monte Carlo;image segmentation;Bayesian statistical framework;computational paradigm;split-merge;snake/balloon;region competition;data clustering;Markov random field;Markov chain sequence;clustering;edge detection;Image segmentation;Monte Carlo methods}, 
+doi={10.1109/34.1000239}, 
+ISSN={0162-8828}, 
+month={May},}
+
+@misc{multicam,
+	title={Using the Intel® RealSense™ Depth cameras D4xx in Multi-Camera Configurations.},
+	author={Anders Grunnet-Jepsen and Paul Winer and Aki Takagi and John Sweetser and Kevin Zhao and Tri Khuong and Dan Nie and John Woodfill},
+	journal={Intel},
+	url ={https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration},
+	note={Access Date: 2019-10-01}
+}
+
+@misc{cal, 
+	title={Calibration Tools User Guide for Intel® RealSense™ D400 Series},
+	journal={Intel},
+	url = {https://dev.intelrealsense.com/docs/intel-realsensetm-d400-series-calibration-tools-user-guide},
+	note={Access Date: 2019-10-01}
+}
+
+@misc{bestcal,
+	title={Best-Known-Methods for Tuning Intel® RealSense™ D400 Depth Cameras for
+Best Performance},
+	author={Anders Grunnet-Jepsen and John N. Sweetser and John Woodfill},
+	journal={Intel},
+	revision={Revision 1.9},
+	url ={https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance},
+	note={Access Date: 2019-10-01}
+}
+
+@incollection{EGGER2017115,
+title = "Chapter 5 - Probabilistic Morphable Models",
+editor = "Guoyan Zheng and Shuo Li and Gabor Székely",
+booktitle = "Statistical Shape and Deformation Analysis",
+publisher = "Academic Press",
+pages = "115 - 135",
+year = "2017",
+isbn = "978-0-12-810493-4",
+doi = "https://doi.org/10.1016/B978-0-12-810493-4.00006-7",
+url = "http://www.sciencedirect.com/science/article/pii/B9780128104934000067",
+author = "Bernhard Egger and Sandro Schönborn and Clemens Blumer and Thomas Vetter",
+keywords = "Face image analysis, Markov chain Monte Carlo, Morphable model, Generative models, Top-down and bottom-up integration, Occlusion-aware model fitting, Facial expressions",
+abstract = "3D Morphable Face Models have been introduced for the analysis of 2D face photographs. The analysis is performed by actively reconstructing the three-dimensional face from the image in an Analysis-by-Synthesis loop, exploring statistical models for shape and appearance. Here we follow a probabilistic approach to acquire a robust and automatic model adaptation. The probabilistic formulation helps to overcome two main limitations of the classical approach. First, Morphable Model adaptation is highly depending on a good initialization. The initial position of landmark points and face pose was given by manual annotation in previous approaches. Our fully probabilistic formulation allows us to integrate unreliable Bottom-Up cues from face and feature point detectors. This integration is superior to the classical feed-forward approach, which is prone to early and possibly wrong decisions. The integration of uncertain Bottom-Up detectors leads to a fully automatic model adaptation process. Second, the probabilistic framework gives us a natural way to handle outliers and occlusions. Face images are recorded in highly unconstrained settings. Often parts of the face are occluded by various objects. Unhandled occlusions can mislead the model adaptation process. The probabilistic interpretation of our model makes possible to detect and segment occluded parts of the image and leads to robust model adaptation. Throughout this chapter we develop a fully probabilistic framework for image interpretation. We start by reformulating the Morphable Model as a probabilistic model in a fully Bayesian framework. Given an image, we search for a posterior distribution of possible image explanations. The integration of Bottom-Up information and the model parameters adaptation is performed using a Data Driven Markov Chain Monte Carlo approach. The face model is extended to be occlusion-aware and explicitly segments the image into face and non-face regions during the model adaptation process. The segmentation and model adaptation is performed in an Expectation-Maximization-style algorithm utilizing a robust illumination estimation method. The presented fully automatic face model adaptation can be used in a wide range of applications like face analysis, face recognition or face image manipulation. Our framework is able to handle images containing strong outliers, occlusions and facial expressions under arbitrary poses and illuminations. Furthermore, the fully probabilistic embedding has the additional advantage that it also delivers the uncertainty of the resulting image interpretation."
+}
+
+@INPROCEEDINGS{8373814,
+author={T. {Gerig} and A. {Morel-Forster} and C. {Blumer} and B. {Egger} and M. {Luthi} and S. {Schoenborn} and T. {Vetter}},
+booktitle={2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018)},
+title={Morphable Face Models - An Open Framework},
+year={2018},
+volume={},
+number={},
+pages={75-82},
+keywords={computer vision;face recognition;Gaussian processes;image registration;principal component analysis;public domain software;splines (mathematics);face registration;neutral faces;open-source software framework;registration model-building;BU3D-FE database;2D face images;open framework;Gaussian processes;nonrigid registration;Morphable face models;nonrigid deformation models;PCA models;GPMM separate problem specific requirements;registration algorithm;prior model;modeling technique;facial expression model;open-source pipeline;Gaussian process morphable models;analysis-by-synthesis model adaption;Basel face model;morphable face models;Face;Adaptation models;Strain;Kernel;Shape;Deformable models;Gaussian processes;Morphable Model;Gaussian processes;Gaussian Process Morphable Model;Registration;Basel Face Model;Face Reconstruction},
+doi={10.1109/FG.2018.00021},
+ISSN={},
+month={May},}
+
+@ARTICLE{8010438,
+author={M. {Lüthi} and T. {Gerig} and C. {Jud} and T. {Vetter}},
+journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
+title={Gaussian Process Morphable Models},
+year={2018},
+volume={40},
+number={8},
+pages={1860-1873},
+keywords={Gaussian processes;image registration;image segmentation;Karhunen-Loeve transforms;principal component analysis;splines (mathematics);Gaussian Process Morphable Models;shape variation;PDMs;shape models;classical spline models;Gaussian processes;spline model;shape characteristics;model-based segmentation;statistical model;point distribution models;point variations;principal component analysis;Karhunen-Loeve expansion;registration schemes;3D forearm images;Shape;Computational modeling;Mathematical model;Gaussian processes;Analytical models;Deformable models;Kernel;Statistical shape modeling;Gaussian processes;image analysis;non-rigid registration},
+doi={10.1109/TPAMI.2017.2739743},
+ISSN={},
+month={Aug},}
+
+@article{DBLP:journals/corr/KeselmanWGB17,
+  author    = {Leonid Keselman and
+               John Iselin Woodfill and
+               Anders Grunnet{-}Jepsen and
+               Achintya Bhowmik},
+  title     = {Intel RealSense Stereoscopic Depth Cameras},
+  journal   = {CoRR},
+  volume    = {abs/1705.05548},
+  year      = {2017},
+  url       = {http://arxiv.org/abs/1705.05548},
+  archivePrefix = {arXiv},
+  eprint    = {1705.05548},
+  timestamp = {Mon, 13 Aug 2018 16:47:02 +0200},
+  biburl    = {https://dblp.org/rec/bib/journals/corr/KeselmanWGB17},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+
+@misc{serg,
+title={The basics of stereo depth vision},
+author={Sergey Dorodnicov},
+url={https://www.intelrealsense.com/stereo-depth-vision-basics/},
+note={Access Date: 2019-10-01}
+}
+
+@misc{pinhole,
+title={Turning any surface into a touch surface},
+author={Natalie Polansky and Yaniv Hasbani},
+}
+@inproceedings{Lorusso:1995:CFA:236190.236213,
+ author = {Lorusso, A. and Eggert, D. W. and Fisher, R. B.},
+ title = {A Comparison of Four Algorithms for Estimating 3-D Rigid Transformations},
+ booktitle = {Proceedings of the 1995 British Conference on Machine Vision (Vol. 1)},
+ series = {BMVC '95},
+ year = {1995},
+ isbn = {0-9521898-2-8},
+ location = {Birmingham, United Kingdom},
+ pages = {237--246},
+ numpages = {10},
+ url = {http://dl.acm.org/citation.cfm?id=236190.236213},
+ acmid = {236213},
+ publisher = {BMVA Press},
+ address = {Surrey, UK, UK},
+} 
+
+@misc{scalismoTut,
+title={Scalismo Tutorials},
+url={https://scalismo.org/tutorials},
+note={Access Date: 2019-10-01}
+}
+
+@book{Hartley:2003:MVG:861369,
+ author = {Hartley, Richard and Zisserman, Andrew},
+ title = {Multiple View Geometry in Computer Vision},
+ year = {2003},
+ isbn = {0521540518},
+ edition = {2},
+ publisher = {Cambridge University Press},
+ address = {New York, NY, USA},
+}
+
+@article{ALBRECHT2013959,
+title = "Posterior shape models",
+journal = "Medical Image Analysis",
+volume = "17",
+number = "8",
+pages = "959 - 973",
+year = "2013",
+issn = "1361-8415",
+doi = "https://doi.org/10.1016/j.media.2013.05.010",
+url = "http://www.sciencedirect.com/science/article/pii/S1361841513000844",
+author = "Thomas Albrecht and Marcel Lüthi and Thomas Gerig and Thomas Vetter",
+keywords = "Statistical shape model, Conditional shape model, Posterior shape model, Image segmentation, Trochlear dysplasia",
+abstract = "We present a method to compute the conditional distribution of a statistical shape model given partial data. The result is a “posterior shape model”, which is again a statistical shape model of the same form as the original model. This allows its direct use in the variety of algorithms that include prior knowledge about the variability of a class of shapes with a statistical shape model. Posterior shape models then provide a statistically sound yet easy method to integrate partial data into these algorithms. Usually, shape models represent a complete organ, for instance in our experiments the femur bone, modeled by a multivariate normal distribution. But because in many application certain parts of the shape are known a priori, it is of great interest to model the posterior distribution of the whole shape given the known parts. These could be isolated landmark points or larger portions of the shape, like the healthy part of a pathological or damaged organ. However, because for most shape models the dimensionality of the data is much higher than the number of examples, the normal distribution is singular, and the conditional distribution not readily available. In this paper, we present two main contributions: First, we show how the posterior model can be efficiently computed as a statistical shape model in standard form and used in any shape model algorithm. We complement this paper with a freely available implementation of our algorithms. Second, we show that most common approaches put forth in the literature to overcome this are equivalent to probabilistic principal component analysis (PPCA), and Gaussian Process regression. To illustrate the use of posterior shape models, we apply them on two problems from medical image analysis: model-based image segmentation incorporating prior knowledge from landmarks, and the prediction of anatomically correct knee shapes for trochlear dysplasia patients, which constitutes a novel medical application. Our experiments confirm that the use of conditional shape models for image segmentation improves the overall segmentation accuracy and robustness."
+}
+
+@conference{icpram19,
+author={Claudio Ferrari. and Stefano Berretti. and Pietro Pala. and Alberto Del Bimbo.},
+title={3D Face Reconstruction from RGB-D Data by Morphable Model to Point Cloud Dense Fitting},
+booktitle={Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
+year={2019},
+pages={728-735},
+publisher={SciTePress},
+organization={INSTICC},
+doi={10.5220/0007521007280735},
+isbn={978-989-758-351-3},
+}
+
+@conference{visapp16,
+author={Patrik Huber. and Guosheng Hu. and Rafael Tena. and Pouria Mortazavian. and Willem P. Koppen. and William J. Christmas. and Matthias Rätsch. and Josef Kittler.},
+title={A Multiresolution 3D Morphable Face Model and Fitting Framework},
+booktitle={Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, (VISIGRAPP 2016)},
+year={2016},
+pages={79-86},
+publisher={SciTePress},
+organization={INSTICC},
+doi={10.5220/0005669500790086},
+isbn={978-989-758-175-5},
+}
+
+@InProceedings{10.1007/3-540-47967-8_46,
+author="Liu, Ce
+and Shum, Heung-Yeung
+and Zhang, Changshui",
+editor="Heyden, Anders
+and Sparr, Gunnar
+and Nielsen, Mads
+and Johansen, Peter",
+title="Hierarchical Shape Modeling for Automatic Face Localization",
+booktitle="Computer Vision --- ECCV 2002",
+year="2002",
+publisher="Springer Berlin Heidelberg",
+address="Berlin, Heidelberg",
+pages="687--703",
+abstract="Many approaches have been proposed to locate faces in an image. There are, however, two problems in previous facial shape models using feature points. First, the dimension of the solution space is too big since a large number of key points are needed to model a face. Second, the local features associated with the key points are assumed to be independent. Therefore, previous approaches require good initialization (which is often done manually), and may generate inaccurate localization. To automatically locate faces, we propose a novel hierarchical shape model (HSM) or multi-resolution shape models corresponding to a Gaussian pyramid of the face image. The coarsest shape model can be quickly located in the lowest resolution image. The located coarse model is then used to guide the search for a finer face model in the higher resolution image. Moreover, we devise a Global and Local (GL) distribution to learn the likelihood of the joint distribution of facial features. A novel hierarchical data-driven Markov chain Monte Carlo (HDDMCMC) approach is proposed to achieve the global optimum of face localization. Experimental results demonstrate that our algorithm produces accurate localization results quickly, bypassing the need for good initialization.",
+isbn="978-3-540-47967-3"
+}
+
+@inproceedings{surrey809478,
+           month = {February},
+          author = {P Huber and G Hu and R Tena and P Mortazavian and P Koppen and WJ Christmas and M Ratsch and J Kittler},
+            note = {Paper accepted for presentation at 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 27-29 February 2016.  Full text may be available at a later date.},
+       booktitle = {11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications},
+           title = {A Multiresolution 3D Morphable Face Model and Fitting Framework},
+         journal = {Proceedings of the 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications},
+            year = {2016},
+             doi = {10.5220/0005669500790086},
+             url = {http://epubs.surrey.ac.uk/809478/},
+        abstract = {3D Morphable Face Models are a powerful tool in computer vision. They consist of a PCA model of face shape and colour information and allow to reconstruct a 3D face from a single 2D image. 3D Morphable Face Models are used for 3D head pose estimation, face analysis, face recognition, and, more recently, facial landmark detection and tracking. However, they are not as widely used as 2D methods - the process of building and using a 3D model is much more involved. In this paper, we present the Surrey Face Model, a multi-resolution 3D Morphable Model that we make available to the public for non-commercial purposes. The model contains different mesh resolution levels and landmark point annotations as well as metadata for texture remapping. Accompanying the model is a lightweight open-source C++ library designed with simplicity and ease of integration as its foremost goals. In addition to basic functionality, it contains pose estimation and face frontalisation algorithms. With the tools presented in this paper, we aim to close two gaps. First, by offering different model resolution levels and fast fitting functionality, we enable the use of a 3D Morphable Model in time-critical applications like tracking. Second, the software library makes it easy for the community to adopt the 3D Morphable Face Model in their research, and it offers a public place for collaboration.}
+}
\ No newline at end of file