giddyyupp / coco-minitrain Goto Github PK

View Code? Open in Web Editor NEW

217.0 217.0 30.0 4.74 MB

a subset of coco dataset for faster experimentation

Python 100.00%

coco coco-dataset dataset object-detection

coco-minitrain's People

Contributors

Stargazers

Watchers

coco-minitrain's Issues

Every run is a failed run

When starting sample_run.py as python sample_coco.py --coco_path "coco/" --save_format "json" --sample_image_count 1000 --run_count 100, every run is a failed run.

Did I get something wrong or?

If I add print('failed_run) inside failed run check, this is the output:

loading annotations into memory...
Done (t=12.92s)
creating index...
index created!
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:06<00:00, 15.12it/s]
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
failed_run
{"info": {"description": "COCO 2017 Dataset", "url": "http://cocodataset.org", "version": "1.0", "year": 2017, "contributor": "COCO Consortium", "date_created": "2017/09/01"}, "licenses": [{"url": "http://creativecommons.org/licenses/by-nc-sa/2.0/", "id": 1, "name": "Attribution-NonCommercial-ShareAlike License"}, {"url": "http://creativecommons.org/licenses/by-nc/2.0/", "id": 2, "name": "Attribution-NonCommercial License"}, {"url": "http://creativecommons.org/licenses/by-nc-nd/2.0/", "id": 3, "name": "Attribution-NonCommercial-NoDerivs License"}, {"url": "http://creativecommons.org/licenses/by/2.0/", "id": 4, "name": "Attribution License"}, {"url": "http://creativecommons.org/licenses/by-sa/2.0/", "id": 5, "name": "Attribution-ShareAlike License"}, {"url": "http://creativecommons.org/licenses/by-nd/2.0/", "id": 6, "name": "Attribution-NoDerivs License"}, {"url": "http://flickr.com/commons/usage/", "id": 7, "name": "No known copyright restrictions"}, {"url": "http://www.usa.gov/copyright.shtml", "id": 8, "name": "United States Government Work"}], "categories": [{"supercategory": "person", "id": 1, "name": "person"}, {"supercategory": "vehicle", "id": 2, "name": "bicycle"}, {"supercategory": "vehicle", "id": 3, "name": "car"}, {"supercategory": "vehicle", "id": 4, "name": "motorcycle"}, {"supercategory": "vehicle", "id": 5, "name": "airplane"}, {"supercategory": "vehicle", "id": 6, "name": "bus"}, {"supercategory": "vehicle", "id": 7, "name": "train"}, {"supercategory": "vehicle", "id": 8, "name": "truck"}, {"supercategory": "vehicle", "id": 9, "name": "boat"}, {"supercategory": "outdoor", "id": 10, "name": "traffic light"}, {"supercategory": "outdoor", "id": 11, "name": "fire hydrant"}, {"supercategory": "outdoor", "id": 13, "name": "stop sign"}, {"supercategory": "outdoor", "id": 14, "name": "parking meter"}, {"supercategory": "outdoor", "id": 15, "name": "bench"}, {"supercategory": "animal", "id": 16, "name": "bird"}, {"supercategory": "animal", "id": 17, "name": "cat"}, {"supercategory": "animal", "id": 18, "name": "dog"}, {"supercategory": "animal", "id": 19, "name": "horse"}, {"supercategory": "animal", "id": 20, "name": "sheep"}, {"supercategory": "animal", "id": 21, "name": "cow"}, {"supercategory": "animal", "id": 22, "name": "elephant"}, {"supercategory": "animal", "id": 23, "name": "bear"}, {"supercategory": "animal", "id": 24, "name": "zebra"}, {"supercategory": "animal", "id": 25, "name": "giraffe"}, {"supercategory": "accessory", "id": 27, "name": "backpack"}, {"supercategory": "accessory", "id": 28, "name": "umbrella"}, {"supercategory": "accessory", "id": 31, "name": "handbag"}, {"supercategory": "accessory", "id": 32, "name": "tie"}, {"supercategory": "accessory", "id": 33, "name": "suitcase"}, {"supercategory": "sports", "id": 34, "name": "frisbee"}, {"supercategory": "sports", "id": 35, "name": "skis"}, {"supercategory": "sports", "id": 36, "name": "snowboard"}, {"supercategory": "sports", "id": 37, "name": "sports ball"}, {"supercategory": "sports", "id": 38, "name": "kite"}, {"supercategory": "sports", "id": 39, "name": "baseball bat"}, {"supercategory": "sports", "id": 40, "name": "baseball glove"}, {"supercategory": "sports", "id": 41, "name": "skateboard"}, {"supercategory": "sports", "id": 42, "name": "surfboard"}, {"supercategory": "sports", "id": 43, "name": "tennis racket"}, {"supercategory": "kitchen", "id": 44, "name": "bottle"}, {"supercategory": "kitchen", "id": 46, "name": "wine glass"}, {"supercategory": "kitchen", "id": 47, "name": "cup"}, {"supercategory": "kitchen", "id": 48, "name": "fork"}, {"supercategory": "kitchen", "id": 49, "name": "knife"}, {"supercategory": "kitchen", "id": 50, "name": "spoon"}, {"supercategory": "kitchen", "id": 51, "name": "bowl"}, {"supercategory": "food", "id": 52, "name": "banana"}, {"supercategory": "food", "id": 53, "name": "apple"}, {"supercategory": "food", "id": 54, "name": "sandwich"}, {"supercategory": "food", "id": 55, "name": "orange"}, {"supercategory": "food", "id": 56, "name": "broccoli"}, {"supercategory": "food", "id": 57, "name": "carrot"}, {"supercategory": "food", "id": 58, "name": "hot dog"}, {"supercategory": "food", "id": 59, "name": "pizza"}, {"supercategory": "food", "id": 60, "name": "donut"}, {"supercategory": "food", "id": 61, "name": "cake"}, {"supercategory": "furniture", "id": 62, "name": "chair"}, {"supercategory": "furniture", "id": 63, "name": "couch"}, {"supercategory": "furniture", "id": 64, "name": "potted plant"}, {"supercategory": "furniture", "id": 65, "name": "bed"}, {"supercategory": "furniture", "id": 67, "name": "dining table"}, {"supercategory": "furniture", "id": 70, "name": "toilet"}, {"supercategory": "electronic", "id": 72, "name": "tv"}, {"supercategory": "electronic", "id": 73, "name": "laptop"}, {"supercategory": "electronic", "id": 74, "name": "mouse"}, {"supercategory": "electronic", "id": 75, "name": "remote"}, {"supercategory": "electronic", "id": 76, "name": "keyboard"}, {"supercategory": "electronic", "id": 77, "name": "cell phone"}, {"supercategory": "appliance", "id": 78, "name": "microwave"}, {"supercategory": "appliance", "id": 79, "name": "oven"}, {"supercategory": "appliance", "id": 80, "name": "toaster"}, {"supercategory": "appliance", "id": 81, "name": "sink"}, {"supercategory": "appliance", "id": 82, "name": "refrigerator"}, {"supercategory": "indoor", "id": 84, "name": "book"}, {"supercategory": "indoor", "id": 85, "name": "clock"}, {"supercategory": "indoor", "id": 86, "name": "vase"}, {"supercategory": "indoor", "id": 87, "name": "scissors"}, {"supercategory": "indoor", "id": 88, "name": "teddy bear"}, {"supercategory": "indoor", "id": 89, "name": "hair drier"}, {"supercategory": "indoor", "id": 90, "name": "toothbrush"}], "images": [], "annotations": []}

Missing coco_class_labels.csv

Hi,

Thanks for the contribution. When running the csv_to_coco_json.py, the 'coco_class_labels.csv' is missing.

FileNotFoundError: [Errno 2] No such file or directory: 'coco_class_labels.csv''

Could pls add this file? Thanks!

I like this minitrain so much

i use sample_coco.py to sample my own mini-coco dataset for my experiment, it helps me save lots of time. Thanks for your work!
Starred and wanted to Starred more than 1k stars

Result table for keypoint detection

Thank you for your excellent project!

I'm training the keypoint RCNN network implemented in torchvision by scaling the image to the size denoted in object detection result tables.

I know the size of input image between simplebaseline2D and networks of RCNN family(faster, mask, etc.) is different.

I'd like to know how to scale the input image size for minicoco dataset in the keypoint detection experiment using a simplebaseline2D network.

Despite using different networks, is the same size between the object detection task and the keypoint detection task?

Thanks in advance!

Regarding instances_minitrain2017.json

One clarification question: is the link of
instances_minitrain2017.json you shared is the 25k training samples that has been sampled by sample_coco.py for 10M times? Then I don't need to run it for 25M times #17.

So by using this instances_minitrain2017.json, we can essentially download these 25k training images as the same as yours by

python3 coco_download.py --annotation <path_to_instances_minitrain2017.json> --output train2017_mini_25k

?
Thanks in advance @giddyyupp

Broken Link

Thank you for your great work, after clicking the link I get "8026 This link has been deleted by the owner" error.

Thanks

Are segmentations preserved in JSON format?

Hi,

Thank you for the paper and dataset, it is really useful for running ablations faster.
I wanted to ask if the segmentations provided in the JSON files are changed in any way from the originaltrain2017 JSON.

Also, do you have any metrics for segmentation?

Thanks a lot.

How to use sample_coco.py

Hi, thanks for your contribution.
When i run sample_coco.py, the following lines appear:

loading annotations into memory...
Done (t=14.37s)
creating index...
index created!

Then it's stuck here all the time without generating the final json file.

Could you help me? @giddyyupp @nerminsamet

not valid dir

Error when running

python3 coco_download.py --annotation train2017_mini_25k.json --output train2017_mini_25k

Traceback (most recent call last):
  File "coco_download.py", line 23, in <module>
    assert pathlib.Path(args.output_dir).is_dir(), "not valid dir"
AssertionError: not valid dir

Broken download link

Thanks for curating this. I can see the download link is not working. Can you please update the link if possible?

Thanks

Where can I find your json and images file?

I want to benchmark my model via coco-minitrain. As you know, training whole coco data is too slow to develop models.
So I decided to use coco-minitrain.

And you submitted your paper to ECCV with this coco-minitrain as a minor contribution.
But I downloaded your json file and downloaded via coco_download.py.

But this does NOT match at all.

Is there something I did wrong???????
I think not.
Simply to say, Could you share your json and image files when you used for making the benchmark table?

DOUBT : What is the format of label ? are BBOX in some kind of ratio

Example
img_id : 000000007278 have label as [0 ,0.391727 ,0.221691 ,0.190391 ,0.196867] as far as i can understand 0 stands for class 2nd and 3rd element represent x_min and y_min respectively and 4th and 5th are width and height

But i can't understand how to convert the BBOX ratio to original coordinates

It doen't match json and images

In annotation json files,
There is image 000000449108.jpg

but no where in image 000000449108.jpg

Download link for coco-minitrain

Could you give the download link for minitrain images ?

About training settings

Hi. Thank you for the great project!

I'm curious about the training settings for each model epoch, learning rate, etc.

Can I get information about each model? (Faster RCNN, RetinaNet, etc)

Effect of run_count on stats/distribution?

The default value of run_count is 10M and it takes a long time for me to sample. Does it only affect to what extent the stats/distribution of sampled train dataset matches the original train2017? If I set run_count to be 10, should I assume that the stats/distribution would not be affected too much? Thanks!

Code

Can you release the code which can generate the coco-minitrain set? I want to test some models on different proportions of COCO, but coco-minitrain only contain 20% images of COCO

Is there any reference data on the average precision (AP) achieved by various networks in instance segmentation tasks?

Keypoints data in the dataset

Hey!
Amazing dataset!
I was hoping to get a small data set for my pose detection model but I couldnt find keypoints in the annotations file. Are the keypoints not included in this dataset?

Thanks!

Are there any adjustments for training on "minitrain" compared to "train2017"?

Hello, are the models mentioned on the github page trained with the exact same methods, params, epochs,... to their original dataset? The only difference is the training set itself right? (from "train2017" to "minitrain")

I'm asking this because I'm currently training Deformable DETR on "minitrain" for my university final project, 26 epochs in (maximum is 50 epochs) and all the APs are still at 0. The model seems to peform quite well on the original dataset (I even checked with "val2017"). I kept all the config the same to the original model (I should mention that I divided "minitrain" into 5 folders, 5000 images each since google colab encounter error with drive if there are too many files in 1 folder, I think this shouldn't affect the result).

So did you guys encounter this issue as well in your training? Is this normal and I should wait until 50 epochs, or is there something that I'm supposed to change that I'm not aware of? ("minitrain" and annotations are from the shared file and val set is just "val2017")

read_data_sets

while running the sample_coco.py there's an error "cannot import name 'read_data_sets' from 'dataloader' "

image not found error

Hi,
how are we setting the path for this database?
I mean where is the instances_train2017.json with all the 25k images. My model requires instance_train2017.json file path and train2017 image path to start the training. I downloaded this dataset and can see only 25k images not the json file. How do we get that?

Thanks

CenterNet code requires person_keypoints_val2017.json

hello there! I am currently attempting to train COCO minitrain on the CenterNet-better algorithm. However, as shown in the image below, I am required to have the person_keypoints_val_2017.json AND person_keypoints_train_2017.json file. I am only able to find the person_keypoints_train2017.json file but not the val one. May I ask where would it be? Thank you!

How to use this coco-minitrain?

Hi...
How to use this coco-minitrain?

I want to get APsmall, APmedium, APlarge?
I confuse no example code.

Thank's alot

Download link is broken

Hi folks,

Thank you for curating the useful subset. It seems that the google drive link on your github readme is broken, could you please look into it?

Thanks

TypeError: slice indices must be integers or None or have an index method

Traceback (most recent call last):
  File "sample_coco.py", line 156, in <module>
    main()
  File "sample_coco.py", line 62, in main
    for i in keys[:parser.sample_image_count]:
TypeError: slice indices must be integers or None or have an __index__ method

when running sample_coco.py script.

split 10% instead of 20%

Hi @giddyyupp ,
what should I modify to obtain a 10% dataset of COCO? 😄
Thanks a lot!

The download exception is Remote end closed connection without response

由于网络限制，不能从coco官网抽样下载，所以
我更改了下代码，可以直接从完整的train数据中，根据作者提哦的instances_minitrain2017.json 将图片抽取出来

import os
from pycocotools.coco import COCO
import wget
import concurrent.futures
import argparse
import pathlib
from shutil import copy

parser = argparse.ArgumentParser(description="Download COCO images")
parser.add_argument(
    "--annotation",
    type=str,
    default="",
    help="Json file containing annotations",
)
parser.add_argument(
    "--output_dir", type=str, default="", help="Output file to save images to"
)
parser.add_argument(
    "--start_dir", type=str, default="", help="all train data(20G) dir"
)
args = parser.parse_args()
annotation = args.annotation
start_dir= args.start_dir
root = pathlib.Path().absolute()
ann_file = root / annotation
assert pathlib.Path(args.output_dir).is_dir(), "not valid dir"
if not os.fsdecode(ann_file).endswith(".json"):
    assert "Only support COCO style JSON file"
try:
    coco = COCO(os.fsdecode(ann_file))
    img_ids = list(coco.imgs.keys())
except FileNotFoundError:
    raise
def download_images(id):
    try:
        filename = "{0:0>12d}".format(id)
        filename = filename + ".jpg"
        source = f"{start_dir}/{filename}"
        # wget.download(full_url, out=args.output_dir)
        copy(source,args.output_dir)
    except Exception as e:
        print(f"The download exception is {e}", flush=True)
with concurrent.futures.ThreadPoolExecutor() as executor:
    executor.map(download_images, img_ids)

Downloading images using coco_download.py

There is a "/" missing at the end of line 37 in the scripts coco_download.py. I could not initiate download without it.

Val2017 only has 4952 labels

It should be 5000.

giddyyupp / coco-minitrain Goto Github PK

coco-minitrain's People

Contributors

Stargazers

Watchers

Forkers

coco-minitrain's Issues

But i can't understand how to convert the BBOX ratio to original coordinates

Recommend Projects

Recommend Topics

Recommend Org