Something about mAP of COCO API about yolov3

[glenn-jocher]

@phinoo hi, thanks for the feedback. yolov3.weights is not produced by this repo, these are the official darknet weights. This repo will save checkpoints as latest.pt and best.pt, and also every 5 epochs i.e. backup_5.pt. If I understand, you trained to 80 epochs, so you should have backup_80.pt saved. Did you get COCO mAP with this weightfile?

You say you changed the resize function in pytorch, can you show your changes? Thanks!

[phinoo]

@glenn-jocher thanks for reply. The resize function just the same as the darknet file image.c->resize_image(...) function, my deal rate is slow,which cost about 1 second to resize the image -,Now I'll test the COCO mAP with the backup.pt .There is another question,I find that your module don't support mutil_scale training,I have get it,just change something a little.I don't know if this can help rise the performance.

[glenn-jocher]

I just added multi_scale support in the latest commit. This will randomly resize each batch from 320 - 608 pixels. Use smaller batches to avoid running out of memory.

yolov3/train.py

Line 14 in 40b536a

parser.add_argument('-multi_scale', default=False, help='random image sizes per batch 320 - 608')

I don't understand about resize. Can you post 3 pictures please: original image, resized using darknet, resized using this repo?

[okanlv]

@glenn-jocher Darknet resizes the images for every 10 batches. I have added the code snippet from darknet below as a reference.

    if(l.random && count++%10 == 0){
        printf("Resizing\n");
        int dim = (rand() % 10 + 10) * 32;
        if (get_current_batch(net)+200 > net->max_batches) dim = 608;
        printf("%d\n", dim);
        args.w = dim;
        args.h = dim;

[glenn-jocher]

Ah yes you are correct, thank you @okanlv. I currently have it changing every batch when -multi_scale = True, I'll try and change that to every 10 batches in the next commit.

[phinoo]

@glenn-jocher It's difficult to tell the difference between resized image using darknet and using this repo.but you can compare the pix value in the same index after resize.Here is a example of this:

we can see that although the picture is the only one,but pix value is much different

[phinoo]

@glenn-jocher this is the darknet resize code:
image resize_image(image im, int w, int h)
{
image resized = make_image(w, h, im.c);
image part = make_image(w, im.h, im.c);
int r, c, k;
float w_scale = (float)(im.w - 1) / (w - 1);
float h_scale = (float)(im.h - 1) / (h - 1);
for(k = 0; k < im.c; ++k){
for(r = 0; r < im.h; ++r){
for(c = 0; c < w; ++c){
float val = 0;
if(c == w-1 || im.w == 1){
val = get_pixel(im, im.w-1, r, k);
} else {
float sx = cw_scale;
int ix = (int) sx;
float dx = sx - ix;
val = (1 - dx) * get_pixel(im, ix, r, k) + dx * get_pixel(im, ix+1, r, k);
}
set_pixel(part, c, r, k, val);
}
}
}
for(k = 0; k < im.c; ++k){
for(r = 0; r < h; ++r){
float sy = rh_scale;
int iy = (int) sy;
float dy = sy - iy;
for(c = 0; c < w; ++c){
float val = (1-dy) * get_pixel(part, c, iy, k);
set_pixel(resized, c, r, k, val);
}
if(r == h-1 || im.h == 1) continue;
for(c = 0; c < w; ++c){
float val = dy * get_pixel(part, c, iy+1, k);
add_pixel(resized, c, r, k, val);
}
}
}

free_image(part);
return resized;

}

and this is my(just to test the difference ):
def resize_img(img,img_w,img_h,resized_w,resized_h,padding):
img=np.transpose(img,(2,0,1))/255.0
if img_w>img_h:
new_w=int(resized_w)
new_h=int(img_h/(img_w/resized_w))
else:
new_h=int(resized_h)
new_w=int(img_w/(img_h/resized_h))
img_resized=np.zeros((3,resized_w,resized_h))+padding
img_inter=np.zeros((3,img_h,new_w))
img_shaped=np.zeros((3,new_h,new_w))
w_scale=(img_w-1)/(new_w-1)
h_scale=(img_h-1)/(new_h-1)
for c in range(3):
for h in range(img_h):
for w in range(new_w):
if w==new_w-1 or img_w==1:
pix=img[c][h,img_w-1]
else:
sx=ww_scale
ix=int(sx)
dx=sx-ix
pix=(1-dx)img[c][h,ix]+dximg[c][h,ix+1]
#pdb.set_trace()
img_inter[c][h,w]=pix
for c in range(3):
for h in range(new_h):
sy=hh_scale
iy=int(sy)
dy=sy-iy
for w in range(new_w):
pix=(1-dy)img_inter[c][iy,w]
img_shaped[2-c][h,w]=pix
if h==new_h-1 or new_h==1:
continue
for w in range(new_w):
pix=dyimg_inter[c][iy+1,w]
img_shaped[2-c][h,w]+=pix
#pdb.set_trace()
img_resized[0][91:325,:]=img_shaped[0]
img_resized[1][91:325,:]=img_shaped[1]
img_resized[2][91:325,:]=img_shaped[2]
return img_resized

at last ,if I use my resize function to replace the pytorch's resize function ,I'll get the perfect same detect result with darknet's detect result(All of them use the weights from original darknet)

[phinoo]

@glenn-jocher I changed my image size every 100 steps like this

[glenn-jocher]

@phinoo Ok thanks for the info. This repo does not use pytorch for resizing, it uses cv2.resize(), computes padding and pads using cv2.copyMakeBorder().

1. Perhaps the interpolation scheme is different (try in interpolation=cv.2INTER_LINEAR instead)
2. Perhaps the padding scheme is different. Play around with the function and see if you can get closer to darknet.

yolov3/utils/datasets.py

Lines 202 to 211 in 45ee668

	def resize_square(img, height=416, color=(0, 0, 0)): # resize a rectangular image to a padded square
	shape = img.shape[:2] # shape = [height, width]
	ratio = float(height) / max(shape) # ratio = old / new
	new_shape = [round(shape[0] * ratio), round(shape[1] * ratio)]
	dw = height - new_shape[1] # width padding
	dh = height - new_shape[0] # height padding
	top, bottom = dh // 2, dh - (dh // 2)
	left, right = dw // 2, dw - (dw // 2)
	img = cv2.resize(img, (new_shape[1], new_shape[0]), interpolation=cv2.INTER_AREA) # resized, no border
	return cv2.copyMakeBorder(img, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color), ratio, dw // 2, dh // 2

[muye5]

@phinoo Did you achieve 55.9% on COCO with yourself project or this repo?
I think 55.9 is very close to the result reported by the author. Could you share your implementation, I trained for a long time but cannot achieve a close mAP.

[glenn-jocher]

Yes, that would be interesting if @phinoo achieved .559 mAP, as the default settings only achieve .52. What settings did you use @phinoo?

[nirbenz]

I got the 55.9 mAP reported in the original paper. I reported on this quite a while ago (on original weights).

[glenn-jocher]

@nirbenz yeah thanks. Is there any way you could PR or show us your mAP code?

[phinoo]

@glenn-jocher yeal,But I just train it with the initial weights by yolov3.weights from darknet.After then ,I Train my model with sparsity which can achive mAP 53.2 but it's overfit

[phinoo]

@glenn-jocher There is something strange that if I train yolov3-pytorch epoch by epoch(Restart training after an epoch),It will keep convergence, or it will hard to convergence.

[ydixon]

@nirbenz I'm only getting around 0.43 mAP with COCO tools. Would really appreciated it if you can share the code so I can see what I did different.

[glenn-jocher]

@phinoo @muye5 this issue is now resolved. pycocotools mAP is 0.550 (416) and 0.579 (608) with yolov3.weights in the latest commit. See #71 (comment) for more info.

sudo rm -rf yolov3 && git clone https://github.com/ultralytics/yolov3
sudo rm -rf cocoapi && git clone https://github.com/cocodataset/cocoapi && cd cocoapi/PythonAPI && make && cd ../.. && cp -r cocoapi/PythonAPI/pycocotools yolov3
cd yolov3
...
python3 test.py --save-json --conf-thres 0.005
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.308
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.550
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.313
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.143
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.339
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.448
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.266
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.398
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.417
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.226
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.456
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.572
...
python3 test.py --save-json --conf-thres 0.005 --img-size 608 --batch-size 16
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.328
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.579
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.341
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.196
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.359
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.425
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.279
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.423
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.444
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.293
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.472
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.557