Temporal Interpolation of human point clouds using neural networks and body part segmentation

Abstract

In the context of social VR, one of the media formats that is gaining popularity is that of a point cloud. Point clouds are unstructured volumetric representations of individual points that represent a 3D shape. They are easy to render but are voluminous in size, and thus they require high bandwidth to be transmitted, so concessions have to be made either in spatial or temporal resolution. In this thesis we explore the state-of-the-art solutions for temporal interpolation of dynamic point clouds, with a focus on human bodies. We see that the current solutions work well predicting rigid motions but not deformations, which is the case of the human bodies. We hypothesize that the performance of these architectures can be boosted by segmenting the body in different body parts and predicting the interpolation for each body part individually. Due to the lack of dynamic human point clouds datasets, we generate our own point cloud dataset based on a publicly available image dataset, being that the first contribution of this thesis. It consists of a total of 248.080 point cloud frames representing 40 avatars (20 males and 20 females) performing 70 actions each. We adapt a current state of the art neural network architecture to fit our data, changing the loss function, tuning some parameters from its feature's extraction layers, and adding an extra layer to obtain the desired output. We obtain an architecture capable of performing temporal interpolation, which is the second contribution of this thesis. We design a set of experiments in order to validate our hypothesis. These consist on a series of models trained to interpolate individual body parts, and one model trained to interpolate the full body. We observe performance gains in all the models trained with individual body parts, so we conclude with the hypothesis that applying body part segmentation and predicting the interpolation of individual body parts can improve the accuracy of point cloud temporal interpolation systems.