ABS not detecting Badnet about trojanzoo

[ain-soph]

I’ll finish the ABS docs soon. Hope that will help you.

[ain-soph]

Sorry for not replying in time. In hospital for a few days. Will take a look tonight.

From provided commands and logs, I find several suspicious points, that might be helpful.

1. You are using old version trojanzoo. --pretrain is replaced by --pretrained in new version to keep consistent with torchvision. And new version trojanzoo supports python 3.10 only. You may use the last release version that supports 3.9 if you don’t want to change.

   I’ve reimplemented abs in recent version and tested. It should be working fine. I’ll later add a log file.

2. In defense log, the clean acc listed in attack and defense summary is 76, which is strange. It should be consistent with training result 96.

[ain-soph]

Here's the result.
I just updated the print sentences to make it prettier.

Note the current implementation should be consistent with original author's code.
While TrojanZoo paper's ABS result is actually my own implementation after reading the paper (Because the original authors didn't release their codes at that time). That implementation is more similar to Neural Cleanse and therefore its performance is good. But it should NOT be.

From the result, we can observe that class 0 (target class) has a 30% ASR watermark, while class 2 has a 40%.
That 40% backdoor does exist in the poisoned model, but Neural Cleanse can't detect that interestingly.

[programehr]

Hi, I hope you're fine now.
Thanks a lot for your time. I'm investigating the problem and will tell you the result.

[programehr]

2. In defense log, the clean acc listed in attack and defense summary is 76, which is strange. It should be consistent with training result 96.

I realized that the defense code first prints the summary, then the detect method of the defense class loads the trojaned model.

[ain-soph]

I realized that the defense code first prints the summary, then the detect method of the defense class loads the trojaned model.

In your case, you call --pretrained, so the model will load clean pretrained weights in model.__init__(). And clean_acc of attack and defense should be set as this value as well. After you call defense.detect(), It will load poisoned model weights.

---

I've moved the model loading from detect() to __init__()

And btw, you don't need to set --pretrained in command line

trojanzoo/trojanvision/defenses/abstract.py

Lines 58 to 61 in 2698c22

	def __init__(self, attack: BadNet, original: bool = False, **kwargs):
	self.original: bool = original
	if not self.original:
	attack.load(**kwargs)

[programehr]

Now it's OK for me, but just FYI, I ran it without --pretrain and faced a similar issue. I think that in this case, it uses a randomly initialized model at first and prints its summary.

Anyway, thank you for investigating the issue.

[ain-soph]

I'll close this issue if there is no further question.