Today we are going to discuss a method proposed by researchers from four institutions one of which is ByteDance AI Lab (known for their TikTok App). They give us a new method termed Sparse R-CNN (not confuse with Sparse R-CNN that works with sparse convolutions on 3D computer vision tasks such that) that achieve near state-of-the-art performance in object detection and uses completely sparse and learnable bounding boxes generation
Related work
Let’s start with short overview of existing detecting methods.
Dense method
One of widely used pipelines is one-stage detector, which directly predicts the label and location of anchor boxes densely covering spatial positions, scales, and aspect ratios in a single-shot way. For example SSD or YOLO.
Let’s consider YOLO algorithm. Ultimately, it aims to predict a class of an object on the image and the bounding box specifying object location. Each bounding box can be described using four descriptors:
- center of a bounding box (bx, by)
- width (bw)
- height (bh)
- value c is corresponding to a class of an object (such as: car, traffic lights, etc.).
In addition, we have to predict the pc value, which is the probability that there is an object in the bounding box.
It is a dense method because it is not searching for interesting regions in the given image that could potentially contain an object. Instead, YOLO is splitting the image into cells, using 19×19 grid. But in general one-stage detector could produce W×H cells, one per pixel. Each cell is responsible for predicting k bounding boxes (for this example k is chosen as 5). Therefore, we arrive at a large number of W×H×k bounding boxes for one image.
For each cell in the grid and each bbox mentioned above values are produced (source)
Dense-to-sparse method
There are two-stage detectors that piggy-backs on dense proposals generated using RPN like Faster R-CNN paper proposed. These detectors has dominated modern object detection for years.
Using RPN it obtains a sparse set of foreground proposal boxes from dense region candidates, and then refines location of each proposal and predicts its specific category.
(source)
Proposals are obtained the similar way as in one-stage detectors, but instead of predicting class of object directly it predicts objectness probability. After that the second stage predict classes on filtered by objectness and overlap score bounding boxes.
Sparse method
Authors of this paper categorize a paradigm of their new Sparse R-CNN as the extension of existing object detector paradigm which includes from thoroughly dense to dense-to-sparse with new step which leads to thoroughly sparse.
Model architectures comparison (source)
In the reviewed paper using RPN is avoided and replaced with a small set of proposal boxes (e.g. 100 per image). These boxes are obtained using learnable proposal boxes part and proposal features part of the network. The formal predicts 4 values (x,y,h,w) per proposal and the latter predicts latent representation vector of length 256 of each bbox contents. Learned proposal boxes is used as a reasonable statistic to perform refining step afterwards and learned proposal features used to introduce attention mechanism. This mechanism is very similar to one that is used in DETR paper. This manipulations are performed inside dynamic instance interactive head that we will cover in the following section.
Proposed Model Features
As the name of paper implies this model is end-to-end. The architecture is elegant. It consists of FPN based backbone that acquire features from images, mentioned above learnable proposal boxes and proposal features and dynamic instance interactive head which is the main contribution to neural nets architecture of this very paper.
Dynamic instance interactive head
Given N proposal boxes, Sparse R-CNN first utilizes the RoIAlign operation to extract features from backbone for each region defined with proposal bounding boxes. Each RoI feature is fed into its own exclusive head for object location and classification, where each head is conditioned on specific learnable proposal feature.
Dynamic module (source)
Proposal features are used as weights for convolutions, on the image above they are mentioned as “Params.” The RoI feature is processed by this generated convolution to obtain the final feature. In this way, those bins with most foreground information make effect on final object location and classification. Self-attention module is embedded into dynamic head to reason about the relations between objects and affects predictions by this convolution.
Main Result
Authors provide several comparison tables that shows performance of a new method. Sparse R-CNN is compared to RetinaNet, Faster R-CNN and DETR in two variations with ResNet50 and ResNet100.
Model performance (source)
Here we can see that Sparse R-CNN outperform RetinaNet and Faster R-CNN in both R50 and R100, but it perform quite similar to DETR based architectures.
According to authors DETR model is in practice dense-to-sparse model because it utilizes a sparse set of object queries, to interact with global(dense) image feature. Hence the novelty of the article is arise comparing to DETR.
Qualitative analysis (source)
On that image you can see qualitative result of model inference on COCO Dataset. In the first column learned proposal boxes are shown, they are predicted for any new image. In the next columns you can see final bboxes that were refined from proposals. They different depending on stage in iterative learning process.
Show me the code!
In conclusion I would like to say that now in 2020 we see a lot of papers that applies transformers to images. Transformers has proved their worth in fields of NLP and now they gradually enter the scene of image processing. This paper shows us that using transformers it is possible to create fast one-stage detectors that perform comparable in terms of quality to the best for now two-stage ones.
All details on implementation you can find in the author’s code that based on FAIR’s DETR and detectron2 codebases: https://github.com/PeizeSun/SparseR-CNN
References
[1] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks https://arxiv.org/abs/1506.01497
[2] YOLO Algorithm and YOLO Object Detection: An Introduction https://appsilon.com/object-detection-yolo-algorithm/
[3] Sparse R-CNN: End-to-End Object Detection with Learnable Proposals https://arxiv.org/abs/2011.12450
Originally published on Medium.