Is it possible to train a high-quality model that predicts the 3D coordinates of the animal’s body surface from a photo, without the corresponding DensePose markup? The answer is yes!
Researchers from Facebook AI introduced this issue at the CVPR 2020 conference.
About DensePose task
Earlier, these researchers presented the scientific community with a new DensePose-COCO dataset and a neural network architecture that works with this data (article). This dataset consists of specially assembled markup of people in images from COCO 2014.
More about the dataset
Data includes:
- bounding boxes of people in the photo,
- pixel-perfect foreground-background masks,
- segmentation of 32x body parts, inside the above masks,
- and a large set of triplets (c, u, v) for each photo, where c is the index of the body part, u , v are geodesic coordinates within the body part.
They were assembled as follows:
The task was to determine the correspondence between randomly thrown points on the human image (within the segmentation mask) and points on six pre-rendered images of the 3D human model, i.e. on six 2D projections of the SMPL model at different angles. After getting this markup, the researchers restored (c, u, v) coordinates for these points on the human surface.
According to this principle, 5 million points were collected for 50 thousand images of people inside the COCO 2014 dataset.
This kind of data allowed us to build a Mask-RCNN like model for predicting masks with 3D coordinates from images.
This decision was notable for the dress-up followups. Since the neural network allows you to get a model of a person, you can put a model of clothing on top of this model. The examples below:
Visualizations in video format can be found here.
Now about the monkeys
In thenew paper, researchers suggest using markup from DensePose-COCO and COCO Dataset to solve a similar problem of predicting 3D coordinates of animal surfaces. However, it is suggested not to mark up images of animals according to the algorithm described above. It is argued that you can use the existing markup not only for chimpanzees, but also for any other animals that are less similar to humans in terms of anatomy.
In this way, the researchers managed to achieve quality by the metric DensePose Average Precision equals to 34.9. Having marked up data for a class of human, the model from the first article showed the result = 46.8, taking into account the fact that the metric varies from 0 to 100. Good result of knowledge transferring?
To measure the quality of the new model, it was necessary to mark up a certain amount of photos of chimpanzees (the same way how it was done for humans). To do this, a method was proposed to restore the correspondence between the SMPL points of the human model and a very detailed artistic 3D model of a chimpanzee.
This made it possible not only to collect the necessary markup for evaluation, but also not to significantly change the code that measures the quality of the model.
Show me the code!
As is usually the case with articles published by famous scientific groups such as the FAIR, they are accompanied by a code. For both articles, it is available inside the official detectron2 repository on GitHub.
The early code from the first article, written using the first version of detectron, which is based on Caffe2, it can also be found on GitHub.