2.1

Local feature extraction

As in Figure 1, we adopt a mask-based manner for local feature extraction. Because most mask-based methods 9, 12, 16] have similar structures for local feature extraction, we would emphasise the difference of our method. In our work, the

Figure 1.

The Proposed Part-Aware Network. The feature extraction module is a backbone network (e.g., ResNet50) and the mask generator is a 1×1 convolution layer. “WP” is weighted pooling, “GMP” is global maximum pooling. “FC” is for dimension reduction and global-local balance, while “Shared FC” is a share-weight fully connection employed to all local features. ⊗ means element-wise matrix multiply, © denotes “concat”.

proposed Part-Aware Network (PAN) transforms pedestrian images into an embedded feature f_cat for discrimination, and this embedded feature f_cat is a concat of one global embedded feature and p local embedded features.

Given a pedestrian image, we first resize and input it to PAN. Through a feature extraction module (a backbone network, e.g., ResNet 17] or OSNet 18]), PAN outputs 3 global features:

c, h, w are channel number, height and width of the feature respectively. Compared to VPM where these three tensors are the same tensor, we separate them to discuss whether an additional buffer layer helps to improve performance.

Then a mask generator is employed to , which is formulated by

where masks M ∈ ℝ ^p×h×w, G(·) is the mask generator, which is a simple l×l convolution layer in our work. Though most mask-based works^9,12,19 employ a following softmax to masks for a probability explanation, our opinion is that this operation affects the independence of masks and causes gradient vanishing. So we remove the softmax and develop a new weighted pooling method.

Based on the masks M, we extract local features from , which is formulated by

where ⊗ denotes element-wise multiply operation, the i^th mask m_i ∈ ℝ^l×h×w, and the i^th output local feature is F_l,i ∈ ℝ^c₂×h×w.

To squeeze the space information of local features, a proposed mask-based weighted pooling is employed to the local features, as in

the output local embedded feature f_l,i ∈ ℝ^c₂. As elements x_j of masks with a softmax satisfy x_j ∈ [0,1], the elements of our masks are x_j ∈ (–∞, ∞), thus the sum of a mask may equal to zero, so we sum up the absolution of the mask. Our weighted pooling is a general form of the weighted pooling in VPM 9] and PAT 16], which gets rid of probability explanation and improves gradient transfer.

2.2

Global-local balance strategy

As we combine global feature and local features to a discriminative embedded feature, there is a balance between the global feature and the local features. Previous methods balance global and local branches on loss weight rather than on embedded feature dimensions, thus the range of global features and local features may be different, which means redundancy on the final embedded feature. We adopt fully connection layers to reduce dimension of embedded feature and achieve global-local balance. Concretely, we apply a fully connection layer for the global feature (dimension c₃ → N_g) and a share-weight fully connection layer (“Shared FC” in Figure 1) for local features (dimension c₂ → n_l). The shared weight manner on all local branches restricts that the difference of local features is only from mask-based local feature extraction.

We define the balance of global and local branches with the ratio of local features versus global γ

where n_l is the dimension of each local embedded feature (256 as default). Here we exploit a weight shared fully connection layer to adjust local feature size, thus change the ratio γ. Since γ means the importance ratio of local features vs. global feature, we expect our model reach the best performance when 1 < γ < n_l. Because local features are more important than the global feature, while the global feature contains more information than a single local feature.

2.3

Training loss

We utilize identity loss and triplet loss for our model:

while identity loss L_id is cross entropy loss with label smoothing¹⁹, and triplet loss L_tri is Soft Margin Hard Mining Triplet Loss²⁰.

3.1

Experiment setting

All models are trained and tested on Ubuntu 18.04 with 2 GTX 1080Ti GPUs. And we employ random flip and random erasing 16] as our data augmentation strategy. Samples are re-scaled to 256×128. We employ Adam optimizer and the base learning rate is 3.5×10^–5. On the training, there are 20 linear warm-up epochs for learning rate going to 3.5 × 10^-4, then on milestones 60 and 90 epoch, learning rate multiply 0.1 respectively.

3.2

Datasets

Market1501¹ and DukeMTMC-reID² are 2 common holistic ReID datasets. Occluded Duke¹² is made from DukeMTMC-reID. Its training, query, and gallery set contain 9%/100%/10% occluded images (14%/15%/10% for DukeMTMC-reID). In other words, the training set of Occluded Duke contains fewer occluded images than DukeMTMC-reID, and only occluded images are in its query set.

3.3

Evaluation protocol

For performance evaluation, we employ the common metrics of person re-identification, Cumulative Matching Characteristics (CMC) and mean Average Precision (mAP).

5.1

Share-weight fully connection

Compared shared and not shared in Table 2, shared fully connection has little influence on holistic datasets, but shows consistent improvement on occluded dataset in spite of backbones (ResNet50 +1.0% R1/+0.9% mAP, OSNet+4.5% R1/+3.4% mAP). So the restriction on generating local features is helpful for occluded cases, perhaps leads to more precise local feature alignment.

Table 2.

Comparison of shared and not shared fully connection for local features.

5.2

Global-local balance

We vary local-global feature ratio γ in equation (4) by changing shared FC output channels in Figure 2. There are two peaks on the curve, the first peak is at γ = 1.75, the second peak is at γ = 7. When γ increase from 0.875 to 1.75, performance increase 1.4% Rank-1 and 1.4% mAP scores, thus indicates local branch is necessary for discrimination. When γ increases from 1.75 to 7, a valley is at γ = 3.5, so there may be a competition between global and local branches. And when γ increases from 7 to 28, performance continuously drops down, which means global feature cannot be replaced by local features. Finally, we choose γ = 7 (n_l = 256) for our model.

Figure 2.

Global-local balance.

5.3

The usage of mask generator

In Table 3, we discuss the usage of mask generator, including using which tensor as to generate masks and employing masks to which tensor as . Compared paired experiments (1, 2), (3, 4), and (5, 6), using the output of the separated buffer layer as can improve performance, which indicates there is a conflict between global branch and local branches, and global branch and local branches should not share the same global feature for feature extraction. And compared experiments (1, 3) and (2, 4), using the output of the separated buffer layer “Conv41” as performs worse. Compared paired experiments (2, 6) and (1, 5), it is better to use the final output of feature extraction module for mask generation. Finally, we choose the output of “Conv50”, the final output of global branch, as for mask generation, and the output of “Conv51”, the output of the separated buffer layer, as to extract local features.

Table 3.

The usage of mask generator.

Note: “Conv40” and “Conv50” are the last two ResNet blocks; “Conv41” is a copied block of “Conv40”; “Conv51” is a copied block of “Conv50”.

5.4

Mask number

We further study the influence of mask number (or parts), as shown in Figure 3. In our model, best mask number is 14 on Occluded Duke, and 16 on Market1501, which means fine-grained masks is helpful not only for occluded cases but also for holistic cases. The observation is different from PAT²¹ on holistic cases, because PAT employs a softmax to masks and adopts a loss to supervise the diversity of local features. However, when we employ softmax to masks, Rank-1/mAP for Occluded Duke decreases from 62.5%/52.6% to 60.6%/51.5%, thus the probability explanation (softmax operation) is not necessary for masks.

Figure 3.

The influence of mask number: (a) is experiments on occluded dataset Occluded Duke, (b) is experiments on holistic dataset Market1501.

5.5

Weighted pooling

In Table 4, we compare commonly used weighted pooling (use softmax to provide probability explanation to masks) and the proposed weighted pooling without softmax. Our proposed method shows consistent improvement on most datasets, especially on Occluded Duke (Rank-1+1.9%/mAP+1.1%). The proposed weighted pooling without softmax improves the gradient transfer, so performance commonly increase. For occlusion cases, our opinion is that without softmax, different local features freely combine to discriminative features (Figure 4), thus improve occlusion performance.

Figure 4.

Visualization of masks. The 1st row is original images, the 2nd row is fusion masks of all masks, the 3rd, 4th, and 5th row are masks 1, 2, 3 of each person.

Table 4.

The comparison of weighted pooling.

5.6

Visualization of masks

12 different persons’ masks are shown on the visualization of Figure 4. The 1st row is original images; the 2nd row is the fusion of all masks, they are all concerned on bodies; the rows from 4th to 6th are the 1st to 3rd masks, they are combination of different parts, as mentioned on Section 5.5. Visualization of masks shows that our method achieves good performance for concerning on human bodies, but mis-alignment also exists (e.g., the 1st mask of the last person), this might be caused by the lack of restriction of masks.