yewsiang / conceptbottleneck

Concept Bottleneck Models, ICML 2020

👨‍🦱 About Me

Knowledge problems excite me!

Currently thinking about Feeds - current feeds are noisy and full of spam, and users have very little control over their daily information diet. How can we give users more meaningful control over their feeds? Chat with me on @Jason_Ys on Twitter if you're interested.

Currently Senior SWE at You.com - working on A/B tests, YouChat and ML. Previously CTO & Co-founder at Hypotenuse.ai (YC S20) and an ML researcher at Stanford AI Labs, NUS CVRP Lab and DSO Laboratories (Defence).

🚀 Recent Projects

Personamo - Giving users more meaningful control over their Feeds.

Crowdwise - Relevant discussions from HN & Reddit as you browse. Chrome extension that adds to your browsing experience by showing you backlinks and relevant discussions about your current web page from Hacker News and Reddit.

Branchminds - AI Research Assistant to find Web links related to your Notion Content. A widget that is embedded right within your Notion that will read from your Notion content to help you to research and find relevant Web links in the topics that you care about.

🤖 ML Research

Concept Bottleneck Models - Learning models that we can interact with using high-level concepts: would the model predict severe arthritis if it thinks there is a bone spur in the x-ray?

Transferable Semi-supervised 3D Object Detection from RGB-D Data - Training 3D object detectors for new object classes from only 2D bounding box labels of these new classes, while simultaneously transferring information from 3D bounding box labels of the existing classes.

conceptbottleneck's Issues

en-core-web-sm dependency download issue

pip install -r requirements.txt causes the following error on Mac OS Catalina 10.15:

ERROR: Could not find a version that satisfies the requirement en-core-web-sm (from -r requirements.txt (line 9)) (from versions: none)
ERROR: No matching distribution found for en-core-web-sm (from -r requirements.txt (line 9))

I get a similar error for the version 2.1.0 or for any other version also.

Can I install this module via spaCy? (python -m spacy download en_core_web_sm) Do you have any other suggestion/alternatives for downloading the en-core-web-sm module?

Model choice for CUB

Hello,

thanks for your interesting work. I was wondering why you chose InceptionV3 for the experiments on CUB rather than Resnet18/50 which seems to be more common in the literature?

I am grateful for some help.
Greetings

Dataset link is dead?

Thanks for sharing the code of your work.

I've been trying to access the dataset from worksheets.codalab.org, but it seems to be down, at least at the time of posting this. I was wondering if it would be possible for you to provide an alternative link? It would help a lot!

Adding license

First of all, thank you for your great work.

Would you please add an open-source license to this repository? Thanks!

Unable to reproduce CUB_processed

Hi, thanks for the great work! I am trying to use this function to process the CUB dataset, but it results in only 109 attribute labels. Specifically, I first ran data_processing.py to process the data in to CUB_200_2011, and then I called

get_class_attributes_data(10, CUB_processed, modify_data_dir=CUB_200_2011, keep_instance_data=False)

and there were only 109 attribute labels in the new train.pkl. Attached is the first instance in train.pkl. May I know how to fix this? Thanks!

>>> a[0]
{'id': 4032, 'img_path': '/home/***/CUB_200_2011/images/070.Green_Violetear/Green_Violetear_0089_795698.jpg', 'class_label': 69, 'attribute_label': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1], 'attribute_certainty': [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 1, 1, 1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3], 'uncertain_attribute_label': [0.25, 0.25, 0.25, 0.75, 0.25, 0.25, 0.25, 0.25, 0.25, 0.75, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.75, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.75, 0.25, 0.25, 0.25, 0.75, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.75, 0.25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.75, 0.25, 0.25, 0.75, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0.25, 0.75, 0.25, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.75, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.75, 0.25, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.75, 0.25, 0.25, 0.25, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.75, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.75, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.75]}
>>> len(a[0]['attribute_label'])
109

preprocessing concepts

Hi,

thank you for your fantastic work. May I ask you how you processed the concepts? There is a script?

Concept Labels

Hi @yewsiang,
While I am trying to apply the method with other datasets, I feel difficult to find the ground truths of the concepts (blue wings etc. ) for CUB data in your implementation. Could you please indicate me how to get the concept labels? It will be good if you can point out the funcs or the variable names. I couldn't find it out with a key word search as "concept". I am not sure if attribute labels refer to the concepts.

Thank you in advance

Questions about N_ATTRIBUTES for inception

Hi guys, I'm looking at the code and I'm just curious why n_attribute is set to be 112 for the inception model. The CUB_processed uploaded on Codalab only contains 112 elements in attribute_label. However, the CUB 200_2011 dataset contains 312 attributes. Looking at CUB/data_processing.py, there seems no attribute selection process. Could you please explain the reason for the difference between the number of attributes used in the inception model and the dataset? Thanks a lot!

Hyperparameters

Hello,

What are the final hyperparameters used for the joint CUB model?
https://github.com/yewsiang/ConceptBottleneck/blob/master/CUB/scripts/experiments.sh contains different values per seed, with the step size not in the reported range of the paper.

Also:
Do you by any chance have tried resnet50 and have some hyperparameters/results for them?
I am trying to compare my work to yours on the basis of resnet50 too, as a lot of the competitors have used that (and my method seems to be benefit from resnet).

Best regards

Traveling Birds Dataset

Hello,

did you use all images in the train and test folders for training and testing with the Traveling Birds dataset or did you only use those assigned to train/test by CUB200?

Greetings

Missing information for CUB TTI data_dir2

Hello,

To reproduce the TTI experiment an input parameter for data_dar2 is required and the commands provided have the value of CUB. Looking at the function this command calls I have assumed this is for the complete CUB dataset in the same format as the processed version of CUB. I cannot find details on this however and am receiving the error message AssertionError: len(instance_attr_labels): 631546, len(b_attr_labels): 648928 which I am assuming is due to the mask list generated having the following following value: [ 1 4 6 7 10 14 15 20 21 23 25 29 30 35 36 38 40 44 45 50 51 53 54 56 57 59 63 64 69 70 72 75 80 84 90 91 93 101 104 106 110 111 116 117 119 126 131 132 134 145 149 151 153 157 158 163 164 168 172 173 178 179 181 182 183 187 188 193 194 196 202 203 208 209 211 212 213 218 220 221 235 236 238 239 240 242 243 244 249 253 254 259 260 262 268 274 277 283 289 292 293 294 298 299 304 305 308 310 311]. Comparing this to the concept in processed cub it is missing the concepts 99, 125, 152, 198, 225 and 309 while gaining the concepts 104, 173 and 182.

The attributes of each entry in train.pkl is as follows:

id: image ID
img_path: path to image file
class_label: label of image classification
attribute_label: list of length 312 containing 1 if concept at index is present and 0 if not
attribute_certainty: list of length 312 containing value of concept certainty where the index is the concept number

Is the AssertionError being thrown due to the mask value (the assert is defined on line 27 of tti.py)? And is it possible to have a copy of train.pkl and test.pkl for data_dir2 provided to reproduce the results in the paper?

Edit: I have also generated train and test files with data_processing.py with a similar outcome

Order of concepts in attribute_label

Hello,

I am trying to reproduce the model but I am having an issue translating the concept vector to concept labels. For instance, with the classification class 9 (Brewer_Blackbird) the concept vector in the dataset is [0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0] but that translates to the following concepts when using the concepts commented here:

has_bill_shape::spatulate
has_wing_color::orange
has_upperparts_color::orange
has_underparts_color::orange
has_underparts_color::buff
has_back_color::orange
has_upper_tail_color::orange
has_breast_color::orange
has_throat_color::orange
has_eye_color::orange
has_bill_length::longer_than_head
has_forehead_color::orange
has_under_tail_color::orange
has_nape_color::orange
has_belly_color::orange
has_belly_color::buff
has_size::large_(16_-_32_in)
has_shape::perching-like
has_back_pattern::multi-colored
has_tail_pattern::multi-colored
has_primary_color::orange
has_leg_color::orange
has_bill_color::orange
has_crown_color::orange
has_crown_color::buff

It looks as if the concepts are off by one or in a different order to the order in the comment? Would it be possible to know if the concepts are correct and if the order is correct? If the concepts are not correct could the correct ones be listed?

I have tried to re-run dataset processing on my machine but this throws errors

no definition of 'val_files' in the data_processing.py

I have used python to run the preprocessing file: data_processing.py, which is in the CUB folder. But it fails in a strange issue, below is the code and error message.

Code:

python3 data_processing.py -data_dir datasets/CUB_200_2011 -save_dir CUB_processed

Error:

Number of train images from official train test split: 5994
Traceback (most recent call last):
File "/home/chiayilai/Desktop/ConceptBottleneck-master/CUB/data_processing.py", line 85, in
train_data, val_data, test_data = extract_data(args.data_dir)
File "/home/chiayilai/Desktop/ConceptBottleneck-master/CUB/data_processing.py", line 64, in extract_data
if val_files is not None:
NameError: name 'val_files' is not defined

The strange thing is that I can’t find the definition of “val_files” in the data_processing.py. And I can’t find any keywords related to it in the readme file. I tried to add a new folder named “val_files”, but it was still no use. Then I altered the “val_files” to “val_data”. It successfully produced three .pkl files, but I don’t know if this way is correct or not.