# General Automatic Human Shape and Motion Capture Using Volumetric Contour Cues

- 27 Citations
- 8.3k Downloads

## Abstract

Markerless motion capture algorithms require a 3D body with properly personalized skeleton dimension and/or body shape and appearance to successfully track a person. Unfortunately, many tracking methods consider model personalization a different problem and use manual or semi-automatic model initialization, which greatly reduces applicability. In this paper, we propose a fully automatic algorithm that jointly creates a rigged actor model commonly used for animation – skeleton, volumetric shape, appearance, and optionally a body surface – and estimates the actor’s motion from multi-view video input only. The approach is rigorously designed to work on footage of general outdoor scenes recorded with very few cameras and without background subtraction. Our method uses a new image formation model with analytic visibility and analytically differentiable alignment energy. For reconstruction, 3D body shape is approximated as a Gaussian density field. For pose and shape estimation, we minimize a new edge-based alignment energy inspired by volume ray casting in an absorbing medium. We further propose a new statistical human body model that represents the body surface, volumetric Gaussian density, and variability in skeleton shape. Given any multi-view sequence, our method jointly optimizes the pose and shape parameters of this model fully automatically in a spatiotemporal way.

## Keywords

Motion Capture Body Model Bone Length Mesh Vertex Joint Location## 1 Introduction

Markerless full-body motion capture techniques refrain from markers used in most commercial solutions, and promise to be an important enabling technique in computer animation and visual effects production, in sports and biomechanics research, and the growing fields of virtual and augmented reality. While early markerless methods were confined to indoor use in more controlled scenes and backgrounds recorded with eight or more cameras [1], recent methods succeed in general outdoor scenes with much fewer cameras [2, 3].

Before motion capture commences, the 3D body model for tracking needs to be personalized to the captured human. This includes personalization of the bone lengths, but often also of biomechanical shape and surface, including appearance. This essential initialization is, unfortunately, neglected by many methods and solved with an entirely different approach, or with specific and complex manual or semi-automatic initialization steps. For instance, some methods for motion capture in studios with controlled backgrounds rely on static full-body scans [4, 5, 6], or personalization on manually segmented initialization poses [7]. Recent outdoor motion capture methods use entirely manual model initialization [3]. When using depth cameras, automatic model initialization was shown [8, 9, 10, 11, 12], but RGB-D cameras are less accessible and not usable outdoors. Simultaneous pose and shape estimation from in-studio multi-view footage with background subtraction was also shown [13, 14, 15], but not on footage of less constrained setups such as outdoor scenes filmed with very few cameras.

First, we introduce a body representation that extends a scene model inspired by light transport in absorbing transparent media [16]. We represent the volumetric body shape by Gaussian density functions attached to a kinematic skeleton. We further define a novel 2D contour-based energy that measures contour alignment with image gradients on the raw RGB images using a new volume raycasting image formation model. We define contour direction and magnitude for each image position, which form a ridge at the model outline, see Fig. 1. No explicit background segmentation is needed. Importantly, our energy features analytic derivatives, including fully-differentiable visibility everywhere.

The second contribution is a new data-driven body model that represents human surface variation, the space of skeleton dimensions, and the space of volumetric density distributions optimally for reconstruction using a low-dimensional parameter space.

Finally, we propose a space-time optimization approach that fully automatically computes both the shape and the 3D skeletal pose of the actor using both contour and ConvNet-based joint detection cues. The final outputs are (1) a rigged character, as commonly used in animation, comprising a personalized skeleton and attached surface, along with the (optionally colored) volumetric human shape representation, and (2) the joint angles for each video frame. We tested our method on eleven sequences, indoor and outdoor, showing reconstructions with fewer cameras and less manual effort compared to the state of the art.

## 2 Related Work

Our goal is to fully automatically capture a personalized, rigged surface model, as used in animation, *together with* its sequence of skeletal poses from sparse multi-view video of general scenes where background segmentation is hard. Many multi-view markerless motion capture approaches consider model initialization and tracking separate problems [2]. Even in recent methods working outdoors, shape and skeleton dimensions of the tracked model are either initialized manually prior to tracking [3], or estimated from manually segmented initialization poses [7]. In controlled studios, static shape [17, 18] or dimensions and pose of simple parametric human models [13, 14] can be optimized by matching against chroma-keyed multi-view image silhouettes. Many multi-view performance capture methods [19] deform a static full-body shape template obtained with a full-body scanner [4, 5, 20, 21], or through fitting against the visual hull [22, 23, 24, 25] to match scene motion. Again, all these require controlled in-studio footage, an off-line scan, or both. Shape estimation of a naked parametric model in single images using shading and edge cues [26], or monocular pose and shape estimation from video is also feasible [27, 28, 29], but require substantial manual intervention (joint labeling, feature/pose correction, background subtraction etc.). For multi-view in-studio setups (3–4 views), where background subtraction works, Bălan et al. [15] estimate shape and pose of the SCAPE parametric body model. Optimization is independent for each frame and requires initialization by a coarse cylindrical shape model. Implicit surface representations yield beneficial properties for pose [30] and surface [31] reconstruction, but do not avoid the dependency on explicit silhouette input. In contrast to all previously mentioned methods, our approach requires no manual interaction, succeeds even with only two camera views, and on scenes recorded outdoors without any background segmentation.

Recently, several methods to capture both shape and pose of a parametric human body model with depth cameras were proposed [9, 11, 32]; these special cameras are not as easily available and often do not work outdoors. We also build up on the success of parametric body models for surface representation, e.g. [29, 33, 34, 35], but extend these models to represent the space of volumetric shape models needed for tracking, along with a rigged surface and skeleton.

Our approach is designed to work without explicit background subtraction. In outdoor settings with moving backgrounds and uncontrolled illumination, such segmentation is hard, but progress has been made by multi-view segmentation [36, 37, 38], joint segmentation and reconstruction [39, 40, 41, 42], and also aided by propagation of a manual initialization [20, 43]. However, the obtained segmentations are still noisy, enabling only rather coarse 3D reconstructions [42], and many methods would not work with only two cameras.

Edge cues have been widely used in human shape and motion estimation [1, 2, 44, 45, 46, 47], but we provide a new formulation for their use and make edges in general scenes the primary cue. In contrast, existing shape estimation works use edges are supplemental information, for example to find self-occluding edges in silhouette-based methods and to correct rough silhouette borders [26]. Our new formulation is inspired by the work of Nagel et al., where model contours are directly matched to image edges for rigid object [48] and human pose tracking [49]. Contour edges on tracked meshes are found by a visibility test, and are convolved with a Gaussian kernel. This approach forms piecewise-smooth and differentiable model contours which are optimized to maximize overlap with image gradients. We advance this idea in several ways: our model is volumetric, analytic visibility is incorporated in the model and optimization, occlusion changes are differentiable, the human is represented as a deformable object, allowing for shape estimation, and contour direction is handled separately from contour magnitude.

Our approach follows the generative analysis-by-synthesis approach: contours are formed by a 3D volumetric model and image formation is an extension of the volumetric ray-tracing model proposed by Rhodin et al. [16]. Many discriminative methods for 2D pose estimation were proposed [50, 51, 52]; multi-view extensions were also investigated [46, 53, 54]. Their goal is different to ours, as they find single-shot 2D/3D joint locations, but no 3D rigged body shape and no temporally stable joint angles needed for animation. We thus use a discriminative detector only for initialization. Our work has links to non-rigid structure-from-motion that finds sparse 3D point trajectories (e.g. on the body) from single-view images of a non-rigidly moving scene [55]. Articulation constraints [56] can help to find the sparse scene structure, but the goal is different from our estimation of a fully dense, rigged 3D character and stable skeleton motion.

## 3 Notation and Overview

Input to our algorithm are RGB image sequences \(\mathcal {I}_{c,t}\), recorded with calibrated cameras \(c \!=\! 1,\ldots ,C\) and synchronized to within a frame (see list of datasets in supplemental document). The output of our approach is the configuration of a virtual actor model \(\mathcal {K}(\mathbf {p}_t,\mathbf {b},\varvec{\gamma })\) for each frame \(t \!=\! 1,\ldots ,T\), comprising the per-frame joint angles \(\mathbf {p}_t\), the personalized bone lengths \(\mathbf {b}\), as well as the personalized volumetric Gaussian representation \(\varvec{\gamma }\), including color, of the actor.

In the following, we first explain the basis of our new image formation model, the Gaussian density scene representation, and our new parametric human shape model building on it (Sect. 4). Subsequently, we detail our space-time optimization approach (Sect. 5) in two stages: (I) using ConvNet-based joint detection constraints (Sect. 5.1); and (II) using a new ray-casting-based volumetric image formation model and a new contour-based alignment energy (Sect. 5.2).

## 4 Volumetric Statistical Body Shape Model

To model the human in 3D for reconstruction, we build on sum-of-Gaussians representations [7, 16] and model the volumetric extent of the actor using a set of 91 isotropic Gaussian density functions distributed in 3D space. Each Gaussian \(G_q\) is parametrized by its standard deviation \({\sigma }_q\), mean location \(\varvec{\mu }_q\) in 3D, and density \(c_q\), which define the Gaussian shape parameters \(\varvec{\gamma }\!=\! \{\varvec{\mu }_q, {\sigma }_q, c_q\}_q\). The combined density field of the Gaussians, \(\sum _q c_q G_q\), smoothly describes the volumetric occupancy of the human in 3D space, see Fig. 1. Each Gaussian is rigidly attached to one of the bones of an articulated skeleton with bone lengths \(\mathbf {b}\) and 16 joints, whose pose is parameterized with 43 twist pose parameters, i.e. the Gaussian position \(\varvec{\mu }_q\) is relative to the attached bone. This representation allows us to formulate a new alignment energy tailored to pose fitting in general scenes, featuring analytic derivatives and fully-differentiable visibility (Sect. 5).

Traditional statistical human body models represent variations in body surface only across individuals, as well as pose-dependent surface deformations using linear [57, 58] or non-linear [28, 33] subspaces of the mesh vertex positions. For our task, we build an enriched statistical body model that parameterizes, in addition to the body surface, the optimal volumetric Gaussian density distribution \(\varvec{\gamma }\) for tracking, and the space of skeleton dimensions \(\mathbf {b}\), through linear functions \(\varvec{\gamma }(\mathbf {s})\), \(\mathbf {b}(\mathbf {s})\) of a low-dimensional shape vector \(\mathbf {s}\). To build our model, we use an existing database of 228 registered scanned meshes of human bodies in neutral pose [59]. We take one of the scans as *reference* mesh, and place the articulated skeleton inside. The 91 Gaussians are attached to bones, their position is set to uniformly fill the mesh volume, and their standard deviation and density is set such that a density gradient forms at the mesh surface, see Fig. 2 (left). This manual step has to be done only once to obtain Gaussian parameters \(\varvec{\gamma }_\text {ref}\) for the database reference, and can also be automated by silhouette alignment [16].

The best positions \(\{\varvec{\mu }_q\}_q\) and scales \(\{{\sigma }_q\}_q\) of Gaussians \(\varvec{\gamma }_i\) for each remaining database *instance* mesh *i* are automatically derived by weighted Procrustes alignment. Each Gaussian \(G_q\) in the reference has a set of neighboring surface mesh vertices. The set is inferred by weighting vertices proportional to the density of \(G_q\) at their position in the reference mesh, see Fig. 2 (right). For each Gaussian \(G_q\), vertices are optimally translated, rotated and scaled to align to the corresponding instance mesh vertices. These similarity transforms are applied on \(\varvec{\gamma }_\text {ref}\) to obtain \(\varvec{\gamma }_i\), where scaling multiplies \({\sigma }_q\) and translation shifts \(\varvec{\mu }_q\).

To infer the adapted skeleton dimensions \(\mathbf {b}_i\) for each instance mesh, we follow a similar strategy: we place Gaussians of standard deviation 10 cm at each joint in the reference mesh, which are then scaled and repositioned to fit the target mesh using the same Procrustes strategy as before. This yields properly scaled bone lengths for each target mesh.

Having estimates of volume \(\varvec{\gamma }_i\) and bone lengths \(\mathbf {b}_i\) for each database entry *i*, we now learn a joint body model. We build a PCA model on the data matrix \(\left[ (\varvec{\gamma }_1;\mathbf {b}_1), (\varvec{\gamma }_2;\mathbf {b}_2), \ldots \right] \), where each column vector \((\varvec{\gamma }_i;\mathbf {b}_i)\) is the stack of estimates for entry *i*. The mean is the average human shape \((\bar{\varvec{\gamma }};\bar{\mathbf {b}})\), and the PCA basis vectors span the principal shape variations of the database. The PCA coefficients are the elements of our shape model \(\mathbf {s}\), and hence define the volume \(\varvec{\gamma }(\mathbf {s})\) and bone lengths \(\mathbf {b}(\mathbf {s})\). Due to the joined model, bone length and Gaussian parameters are correlated, and optimizing \(\mathbf {s}\) for bone length during pose estimation (stage I) thus moves and scales the attached Gaussians accordingly. To reduce dimensionality, we use only the first 50 coefficients in our experiments.

To infer the actor body surface, we introduce a volumetric skinning approach. The reference surface mesh is deformed in a free-form manner along with the Gaussian set under new pose and shape parameters. Similar to linear blend skinning [60], each surface vertex is deformed with the set of 3D transforms of nearby Gaussians, weighted by the density weights used earlier for Procrustes alignment. This coupling of body surfaces to volumetric model is as computationally efficient as using a linear PCA space on mesh vertices [34], while yielding comparable shape generalization and extrapolation qualities to methods using more expensive non-linear reconstruction [33], see supplemental document. Isosurface reconstruction using Marching Cubes would also be more costly [30].

## 5 Pose and Shape Estimation

*T*pose parameters \(\mathbf {P}\!=\! \{\mathbf {p}_1,\ldots ,\mathbf {p}_T\}\) as a combined space-time optimization problem over all frames \(\mathcal {I}_{c,t}\) and camera viewpoints

*c*of the input sequence of length

*T*:

### 5.1 Stage I – Initial Estimation

We employ the discriminative ConvNet-based body-part detector by Tompson et al. [50] to estimate the approximate 2D skeletal joint positions. The detector is independently applied to each input frame \(\mathcal {I}_{c,t}\), and outputs heat maps of joint location probability \(\mathcal {D}_{c,t,j}\) for each joint *j* in frame *t* seen from camera *c*. Importantly, the detector discriminates the joints on the left and right side of the body (see Fig. 1). The detections exhibit noticeable spatial and temporal uncertainty, but are nonetheless a valuable cue for an initial space-time optimization solve. The output heat maps are in general multi-modal due to detection ambiguities, but also in the presence of multiple people, e.g. in the background.

To infer the poses \(\mathbf {P}\) and an initial guess for the body shape of the subject, we optimize Eq. 1 with data term \(E_\text {detection}\). It measures the overlap of the heat maps \(\mathcal {D}\) with the projected skeleton joints. Each joint in the model skeleton has an attached joint Gaussian (Sect. 4), and the overlap with the corresponding heat map is maximized using the visibility model of Rhodin et al. [16]. We use a hierarchical approach by first optimizing the torso joints, followed by optimizing the limbs; please see the supplemental document for details. The optimization is initialized with the average human shape \((\bar{\varvec{\gamma }},\bar{\mathbf {b}})\) in T-pose, at the center of the capture volume. We assume a single person in the capture volume; people in the background are implicitly ignored, as they are typically not visible from all cameras and are dominated by the foreground actor.

Please note that bone lengths \(\mathbf {b}(\mathbf {s})\) and volume \(\varvec{\gamma }(\mathbf {s})\) are determined through \(\mathbf {s}\), hence, Stage I yields a rough estimate of \(\varvec{\gamma }\). In Stage II, we use more informative image constraints than pure joint locations to better estimate volumetric extent.

### 5.2 Stage II – Contour-Based Refinement

The pose \(\mathbf {P}\) and shape \(\mathbf {s}\) found in the previous stage are now refined by using a new density-based contour model in the alignment energy. This model explains the spatial image gradients formed at the edge of the projected model, between actor and background, and thus bypasses the need for silhouette extraction, which is difficult for general scenes. To this end, we extend the ray-casting image formation model of Rhodin et al. [16], as summarized in the following paragraph, and subsequently explain how to use it in the contour data term \(E_\text {contour}\).

**Ray-casting image formation model.**Each image pixel spawns a ray that starts at the camera center \(\mathbf {o}\) and points in direction \(\mathbf {n}\). The visibility of a particular model Gaussian \(G_q\) along the ray \((\mathbf {o},\mathbf {n})\) is defined as

*u*,

*v*):

*u*,

*v*) by \(\mathbf {n}\!=\! \frac{(u,v,1)^{\!\top }}{\vert \vert {(u,v,1)}\vert \vert _2}\) and \(\nabla \mathbf {n}\!=\! (\frac{\partial \mathbf {n}}{\partial u}, \frac{\partial \mathbf {n}}{\partial v})^{\!\top }\).

**Contour energy.**To refine the initial pose and shape estimates from Stage I (Sect. 5.1), we optimize Eq. 1 with a new contour data term \(E_\text {contour}\), to estimate the per-pixel similarity of model and image gradients:

**Appearance estimation.**Our method is versatile: given the shape and pose estimates from Stage II, we can also estimate a color for each Gaussian. This is needed by earlier tracking methods that use similar volume models, but color appearance-based alignment energies [7, 16] – we compare against them in our experiments. Direct back-projection of the image color onto the model suffers from occasional reconstruction errors in Stages I and II. Instead, we compute the weighted mean color \(\bar{a}_{q,c}\) over all pixels separately for each Gaussian \(G_q\) and view

*c*, where the contribution of each pixel is weighted by the Gaussian’s visibility \(\mathcal {V}_q\) (Eq. 3). Colors \(\bar{a}_{q,c}\) are taken as candidates from which outliers are removed by iteratively computing the mean color and removing the largest outlier (in Euclidean distance). In our experiments, removing 50 % of the candidates leads to consistently clean color estimates, as shown in Figs. 4 and 9.

## 6 Evaluation

We evaluate our method on 11 sequences of publicly available datasets with large variety, both indoor and outdoor, and show comparisons to state-of-the-art methods (see supplementary document for dataset details). The quality of pose and shape reconstruction is best assessed in the supplemental video, where we also apply and compare our reconstructions to tracking with the volumetric Gaussian representations of Rhodin et al. [16] and Stoll et al. [7].

**Robustness in general scenes.**We validate the robustness of our method on three outdoor sequences. On the Walk dataset [3], people move in the background, and background and foreground color are very similar. Our method is nevertheless able to accurately estimate shape and pose across 100 frames from 6 views, see Fig. 1. We also qualitatively compare against the recent the model-free method of Mustafa et al. [42]. On the Cathedral sequence of Kim et al. [62], they achieve rough surface reconstruction using 8 cameras without the explicit need for silhouettes; in contrast, 4 views and 20 frames are sufficient for us to reconstruct shape and pose of a quick outdoor run, see Fig. 4 (top) and supplementary material. Furthermore, we demonstrate reconstruction of complex motions on Subject3 during a two-person volleyball play from only 3 views and 100 frames, see Fig. 4 (bottom). The second player was segmented out during Stage I, but Stage II was executed automatically. Fully automatic model and pose estimation are even possible from only two views as we demonstrate on the Marker sequence [3], see Fig. 6.

Quantitative evaluation of estimated shapes in different stages and comparison to Guan et al.’s results [26]. We use three body measures (chest, waist and hips, as shown on the right) to evaluate predicted body shapes against the ground truth (GT) captured using a laser scan.

**Shape estimation accuracy.** To assess the accuracy of the estimated actor models, we tested our method on a variety of subjects performing general motions such as walking, kicking and gymnastics. Evaluation of estimated shape is performed in two ways: (1) the estimated body shape is compared against ground-truth measurements, and (2) the 3D mesh derived from Stage II is projected from the captured camera viewpoints to compute the overlap with a manually segmented foreground. We introduce two datasets Subject1 and Subject2, in addition to Subject3, with pronounced body proportions and ground-truth laser scans for quantitative evaluation. Please note that shape estimates are constant across the sequence and can be evaluated at sparse frames, while pose varies and is separately evaluated per frame in the sequel.

The shape accuracy is evaluated by measurements of chest, waist and hip circumference. Subject1 and Subject2 are captured indoor and are processed using 6 cameras and 40 frames uniformly sampled over 200 frames. Subject3 is an outdoor sequence and only 3 camera views are used, see Fig. 4. All subjects are reconstructed with high quality in shape, skeleton dimensions and color, despite inaccurately estimated poses in Stage I for some frames. We only observed little variation dependent on the performed motions, i.e. a simple walking motion is sufficient, but bone length estimation degrades if joints are not sufficiently articulated during performance. All estimates are presented quantitatively in Table 1 and qualitatively in Fig. 5. In addition, we compare against Guan et al. [26] on their single-camera and single-frame datasets Pose1, Pose2 and Pose3. Stage I requires multi-view input and was not used; instead, we manually initialized the pose roughly, as shown in Fig. 8, and body height is normalized to 185 cm [26]. Our reconstructions are within the same error range, demonstrating that Stage II is well suited even for monocular shape and pose refinement. Our reconstruction is accurate overall, with a mean error of only \(2.3\pm 1.9\) cm, measured across all sequences with known ground truth.

*precision*) and how much of the foreground is overlapped by the model (

*recall*). Despite the low number of cameras, low-quality images, and without requiring background subtraction, our reconstructions are accurate with 95 % precision and 85 % recall, and improve slightly on the results of Bălan et al. [15]. Results are presented in Fig. 7 and Table 2. Note that Stage II significantly improves shape estimation. The temporal consistency and benefit of the model components are shown in the supplemental video on multiple frames evenly spread along the

*HumanEva-I*Box sequence.

**Pose estimation accuracy.** Pose estimation accuracy is quantitatively evaluated on the public HumanEva-I dataset, where ground-truth data is available, see Table 3. We tested the method on the designated validation sequences Walk and Box of subject S1. Reconstruction quality is measured as the average Euclidean distance of estimated and ground-truth joint locations, frames with ground truth inaccuracies are excluded by the provided scripts.

Our pose estimation results are on par with state-of-the-art methods with 6–7 cm average accuracy [3, 46, 53, 54]. In particular, we obtain comparable results to Elhayek et al. [3], which however requires a separately initialized actor model. Please note that Amin et al. [53] specifically trained their model on manually annotated sequences of the same subject in the same room. For best tracking performance, the ideal joint placement and bone lengths of the virtual skeleton may deviate from the real human anatomy, and may generally vary for different tracking approaches. To compensate differences in the skeleton structure, we also report results where the offset between ground truth and estimated joint locations is estimated in the first frame and compensated in the remaining frames, reducing the reconstruction error to 3–5 cm. Datasets without ground-truth data cannot be quantitatively evaluated; however, our shape overlap evaluation results suggest that pose estimation is generally accurate. In summary, pose estimation is reliable, with only occasional failures in Stage I, although the main focus of our work is on the combination with shape estimation.

**Runtime.**In our experiments, runtime scaled linearly with the number of cameras and frames. Contour-based shape optimization is efficient: it only takes 3 s per view, totaling 15 min for 50 frames and 6 views on a standard desktop machine. Skeleton pose estimation is not the main focus of this work and is not optimized; it takes 10 s per frame and view, totaling 50 min.

Pose estimation accuracy measured in mm on the HumanEva-I dataset. The standard deviation is reported in parentheses.

**Limitations and discussion.** Even though the body model was learned from tight clothing scans, our approach handles general apparel well, correctly reconstructing the overall shape and body dimensions. We demonstrate that even if not all assumptions are fulfilled, our method produces acceptable results, such as for the dance performance Skirt of Gall et al. [5] in Fig. 9 (top left) that features a skirt. However, our method was not designed to accurately reconstruct fine wrinkles, facial details, hand articulation, or highly non-rigid clothing.

We demonstrate fully automatic reconstructions from as few as two cameras and semi-automatic shape estimation using a single image. Fully automatic pose and shape estimates from a single image remains difficult.

## 7 Conclusion

We proposed a fully automatic approach for estimating the shape and pose of a rigged actor model from general multi-view video input with just a few cameras. It is the first approach that reasons about contours within sum-of-Gaussians representations and which transfers their beneficial properties, such as analytic form and smoothness, and differentiable visibility [16], to the domain of edge- and silhouette-based shape estimation. This results in an analytic volumetric contour alignment energy that efficiently and fully automatically optimizes the pose and shape parameters. Based on a new statistical body model, our approach reconstructs a personalized kinematic skeleton, a volumetric Gaussian density representation with appearance modeling, a surface mesh, and the time-varying poses of an actor. We demonstrated shape estimation and motion capture results on challenging datasets, indoors and outdoors, captured with very few cameras. This is an important step towards making motion capture more practical.

## Notes

### Acknowledgements

We thank PerceptiveCode, in particular Arjun Jain and Jonathan Tompson, for providing and installing the ConvNet detector, Ahmed Elhayek, Jürgen Gall, Peng Guan, Hansung Kim, Armin Mustafa and Leonid Sigal for providing their data and test sequences, The Foundry for license support, and all our actors. This research was funded by the ERC Starting Grant project CapReal (335545).

## Supplementary material

## References

- 1.Moeslund, T.B., Hilton, A., Krüger, V.: A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Underst.
**104**(2), 90–126 (2006)CrossRefGoogle Scholar - 2.Holte, M.B., Tran, C., Trivedi, M.M., Moeslund, T.B.: Human pose estimation and activity recognition from multi-view videos: comparative explorations of recent developments. IEEE J. Sel. Top. Sign. Proces.
**6**(5), 538–552 (2012)CrossRefGoogle Scholar - 3.Elhayek, A., de Aguiar, E., Jain, A., Tompson, J., Pishchulin, L., Andriluka, M., Bregler, C., Schiele, B., Theobalt, C.: Efficient ConvNet-based marker-less motion capture in general scenes with a low number of cameras. In: CVPR, pp. 3810–3818 (2015)Google Scholar
- 4.de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.P., Thrun, S.: Performance capture from sparse multi-view video. ACM Trans. Graph.
**27**(3), 98 (2008)CrossRefGoogle Scholar - 5.Gall, J., Stoll, C., de Aguiar, E., Theobalt, C., Rosenhahn, B., Seidel, H.P.: Motion capture using joint skeleton tracking and surface estimation. In: CVPR, pp. 1746–1753 (2009)Google Scholar
- 6.Zollhöfer, M., Nießner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., Stamminger, M.: Real-time non-rigid reconstruction using an RGB-D camera. ACM Trans. Graph.
**33**(4), 156 (2014)CrossRefGoogle Scholar - 7.Stoll, C., Hasler, N., Gall, J., Seidel, H.P., Theobalt, C.: Fast articulated motion tracking using a sums of Gaussians body model. In: ICCV, pp. 951–958 (2011)Google Scholar
- 8.Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., Moore, R.: Real-time human pose recognition in parts from single depth images. Commun. ACM
**56**(1), 116–124 (2013)CrossRefGoogle Scholar - 9.Bogo, F., Black, M.J., Loper, M., Romero, J.: Detailed full-body reconstructions of moving people from monocular RGB-D sequences. In: ICCV, pp. 2300–2308 (2015)Google Scholar
- 10.Tong, J., Zhou, J., Liu, L., Pan, Z., Yan, H.: Scanning 3D full human bodies using Kinects. IEEE Trans. Vis. Comput. Graph.
**18**(4), 643–650 (2012)CrossRefGoogle Scholar - 11.Helten, T., Baak, A., Bharaj, G., Müller, M., Seidel, H.P., Theobalt, C.: Personalization and evaluation of a real-time depth-based full body tracker. In: 3DV, pp. 279–286 (2013)Google Scholar
- 12.Newcombe, R.A., Fox, D., Seitz, S.M.: DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In: CVPR, pp. 343–352 (2015)Google Scholar
- 13.Kakadiaris, I.A., Metaxas, D.: Three-dimensional human body model acquisition from multiple views. Int. J. Comput. Vis.
**30**(3), 191–218 (1998)CrossRefGoogle Scholar - 14.Ahmed, N., de Aguiar, E., Theobalt, C., Magnor, M., Seidel, H.P.: Automatic generation of personalized human avatars from multi-view video. In: ACM Symposium on Virtual Reality Software and Technology, pp. 257–260 (2005)Google Scholar
- 15.Bălan, A.O., Sigal, L., Black, M.J., Davis, J.E., Haussecker, H.W.: Detailed human shape and pose from images. In: CVPR (2007)Google Scholar
- 16.Rhodin, H., Robertini, N., Richardt, C., Seidel, H.P., Theobalt, C.: A versatile scene model with differentiable visibility applied to generative pose estimation. In: ICCV (2015)Google Scholar
- 17.Hilton, A., Beresford, D., Gentils, T., Smith, R., Sun, W.: Virtual people: capturing human models to populate virtual worlds. In: Computer Animation, pp. 174–185 (1999)Google Scholar
- 18.Bălan, A.O., Black, M.J.: The naked truth: estimating body shape under clothing. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 15–29. Springer, Heidelberg (2008)CrossRefGoogle Scholar
- 19.Theobalt, C., de Aguiar, E., Stoll, C., Seidel, H.P., Thrun, S.: Performance capture from multi-view video. In: Ronfard, R., Taubin, G. (eds.) Image and Geometry Processing for 3-D Cinematography. Geometry and Computing, pp. 127–149. Springer, Heidelberg (2010)CrossRefGoogle Scholar
- 20.Wu, C., Stoll, C., Valgaerts, L., Theobalt, C.: On-set performance capture of multiple actors with a stereo camera. ACM Trans. Graph.
**32**(6), 161 (2013)Google Scholar - 21.Wu, C., Varanasi, K., Theobalt, C.: Full body performance capture under uncontrolled and varying illumination: a shading-based approach. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part IV. LNCS, vol. 7575, pp. 757–770. Springer, Heidelberg (2012)Google Scholar
- 22.Vlasic, D., Baran, I., Matusik, W., Popović, J.: Articulated mesh animation from multi-view silhouettes. ACM Trans. Graph.
**27**(3), 97 (2008)CrossRefGoogle Scholar - 23.Starck, J., Hilton, A.: Model-based multiple view reconstruction of people. In: ICCV, pp. 915–922 (2003)Google Scholar
- 24.Ballan, L., Cortelazzo, G.M.: Marker-less motion capture of skinned models in a four camera set-up using optical flow and silhouettes. In: 3DPVT (2008)Google Scholar
- 25.Allain, B., Franco, J.S., Boyer, E.: An efficient volumetric framework for shape tracking. In: CVPR, pp. 268–276 (2015)Google Scholar
- 26.Guan, P., Weiss, A., Bălan, A.O., Black, M.J.: Estimating human shape and pose from a single image. In: ICCV, pp. 1381–1388 (2009)Google Scholar
- 27.Guo, Y., Chen, X., Zhou, B., Zhao, Q.: Clothed and naked human shapes estimation from a single image. In: Hu, S.-M., Martin, R.R. (eds.) CVM 2012. LNCS, vol. 7633, pp. 43–50. Springer, Heidelberg (2012)CrossRefGoogle Scholar
- 28.Hasler, N., Ackermann, H., Rosenhahn, B., Thormählen, T., Seidel, H.P.: Multilinear pose and body shape estimation of dressed subjects from image sets. In: CVPR, pp. 1823–1830 (2010)Google Scholar
- 29.Jain, A., Thormählen, T., Seidel, H.P., Theobalt, C.: MovieReshape: Tracking and reshaping of humans in videos. ACM Trans. Graph.
**29**(5) (2010)Google Scholar - 30.Plankers, R., Fua, P.: Articulated soft objects for multi-view shape and motion capture. IEEE Trans. Pattern Anal. Mach. Intell.
**25**(9), 63–83 (2003)CrossRefGoogle Scholar - 31.Ilic, S., Fua, P.: Implicit meshes for surface reconstruction. IEEE Trans. Pattern Anal. Mach. Intell.
**28**(2), 328–333 (2006)CrossRefGoogle Scholar - 32.Cui, Y., Chang, W., Nöll, T., Stricker, D.: KinectAvatar: fully automatic body capture using a single Kinect. In: ACCV Workshops, pp. 133–147 (2012)Google Scholar
- 33.Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., Davis, J.: SCAPE: shape completion and animation of people. ACM Trans. Graph.
**24**(3), 408–416 (2005)CrossRefGoogle Scholar - 34.Pishchulin, L., Wuhrer, S., Helten, T., Theobalt, C., Schiele, B.: Building statistical shape spaces for 3D human modeling. arXiv:1503.05860 (2015)
- 35.Loper, M., Mahmood, N., Black, M.J.: MoSh: Motion and shape capture from sparse markers. ACM Trans. Graph.
**33**(6), 220 (2014)CrossRefGoogle Scholar - 36.Campbell, N.D.F., Vogiatzis, G., Hernández, C., Cipolla, R.: Automatic 3D object segmentation in multiple views using volumetric graph-cuts. In: BMVC, pp. 530–539 (2007)Google Scholar
- 37.Wang, T., Collomosse, J., Hilton, A.: Wide baseline multi-view video matting using a hybrid Markov random field. In: ICPR, pp. 136–141 (2014)Google Scholar
- 38.Djelouah, A., Franco, J.S., Boyer, E., Le Clerc, F., Pérez, P.: Sparse multi-view consistency for object segmentation. IEEE Trans. Pattern Anal. Mach. Intell.
**37**(9), 1890–1903 (2015)CrossRefGoogle Scholar - 39.Szeliski, R., Golland, P.: Stereo matching with transparency and matting. In: ICCV, pp. 517–524 (1998)Google Scholar
- 40.Guillemaut, J.Y., Hilton, A.: Joint multi-layer segmentation and reconstruction for free-viewpoint video applications. Int. J. Comput. Vis.
**93**(1), 73–100 (2011)CrossRefGoogle Scholar - 41.Bray, M., Kohli, P., Torr, P.H.S.: PoseCut: simultaneous segmentation and 3D pose estimation of humans using dynamic graph-cuts. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 642–655. Springer, Heidelberg (2006). doi: 10.1007/11744047_49 CrossRefGoogle Scholar
- 42.Mustafa, A., Kim, H., Guillemaut, J.Y., Hilton, A.: General dynamic scene reconstruction from multiple view video. In: ICCV (2015)Google Scholar
- 43.Hasler, N., Rosenhahn, B., Thormahlen, T., Wand, M., Gall, J., Seidel, H.P.: Markerless motion capture with unsynchronized moving cameras. In: CVPR, pp. 224–231 (2009)Google Scholar
- 44.Deutscher, J., Blake, A., Reid, I.: Articulated body motion capture by annealed particle filtering. In: CVPR, pp. 126–133 (2000)Google Scholar
- 45.Sidenbladh, H., Black, M.J.: Learning the statistics of people in images and video. Int. J. Comput. Vis.
**54**(1–3), 183–209 (2003)zbMATHGoogle Scholar - 46.Sigal, L., Isard, M., Haussecker, H., Black, M.J.: Loose-limbed people: estimating 3D human pose and motion using non-parametric belief propagation. Int. J. Comput. Vis.
**98**(1), 15–48 (2012)MathSciNetCrossRefzbMATHGoogle Scholar - 47.Kehl, R., Bray, M., Van Gool, L.: Markerless full body tracking by integrating multiple cues. In: ICCV Workshop on Modeling People and Human Interaction (2005)Google Scholar
- 48.Kollnig, H., Nagel, H.H.: 3D pose estimation by fitting image gradients directly to polyhedral models. In: ICCV, pp. 569–574 (1995)Google Scholar
- 49.Wachter, S., Nagel, H.H.: Tracking of persons in monocular image sequences. In: Nonrigid and Articulated Motion Workshop, pp. 2–9 (1997)Google Scholar
- 50.Tompson, J.J., Jain, A., LeCun, Y., Bregler, C.: Joint training of a convolutional network and a graphical model for human pose estimation. In: NIPS, pp. 1799–1807 (2014)Google Scholar
- 51.Felzenszwalb, P.F., Huttenlocher, D.P.: Pictorial structures for object recognition. Int. J. Comput. Vis.
**61**(1), 55–79 (2005)CrossRefGoogle Scholar - 52.Andriluka, M., Roth, S., Schiele, B.: Pictorial structures revisited: people detection and articulated pose estimation. In: CVPR, pp. 1014–1021 (2009)Google Scholar
- 53.Amin, S., Andriluka, M., Rohrbach, M., Schiele, B.: Multi-view pictorial structures for 3D human pose estimation. In: BMVC (2013)Google Scholar
- 54.Belagiannis, V., Amin, S., Andriluka, M., Schiele, B., Navab, N., Ilic, S.: 3D pictorial structures for multiple human pose estimation. In: CVPR, pp. 1669–1676 (2014)Google Scholar
- 55.Park, H.S., Shiratori, T., Matthews, I., Sheikh, Y.: 3D trajectory reconstruction under perspective projection. Int. J. Comput. Vis.
**115**(2), 115–135 (2015)MathSciNetCrossRefGoogle Scholar - 56.Fayad, J., Russell, C., Agapito, L.: Automated articulated structure and 3D shape recovery from point correspondences. In: ICCV, pp. 431–438 (2011)Google Scholar
- 57.Allen, B., Curless, B., Popović, Z.: The space of human body shapes: reconstruction and parameterization from range scans. ACM Trans. Graph.
**22**(3), 587–594 (2003)CrossRefGoogle Scholar - 58.Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graph.
**34**(6), 248 (2015)CrossRefGoogle Scholar - 59.Hasler, N., Stoll, C., Sunkel, M., Rosenhahn, B., Seidel, H.P.: A statistical model of human pose and body shape. Comput. Graph. Forum
**28**(2), 337–346 (2009)CrossRefGoogle Scholar - 60.Lewis, J.P., Cordner, M., Fong, N.: Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In: SIGGRAPH, pp. 165–172 (2000)Google Scholar
- 61.Cerezo, E., Pérez, F., Pueyo, X., Seron, F.J., Sillion, F.X.: A survey on participating media rendering techniques. Vis. Comput.
**21**(5), 303–328 (2005)CrossRefGoogle Scholar - 62.Kim, H., Hilton, A.: Influence of colour and feature geometry on multi-modal 3D point clouds data registration. In: 3DV, pp. 202–209 (2014)Google Scholar
- 63.Sigal, L., Bălan, A.O., Black, M.J.: HumanEva: synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. Int. J. Comput. Vis.
**87**, 4–27 (2010)CrossRefGoogle Scholar