Cifar10 Witches' Brew Baseline Evaluation

Results obtained using Armory 0.15.X (July 2022)

Note: the baseline CIFAR resnet was updated in Armory 0.16.1.

Undefended

Accuracy on Non-trigger Images |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|--------------| |0% |0.746446|0.747848|0.748448|0.7475806667|0.0008388886829| |10% |0.743744|0.743544|0.739139|0.7421423333|0.002125246391| |20% |0.735035|0.731532|0.74004|0.7355356667|0.003491371873| |30% |0.750651|0.741842|0.736537|0.74301 |0.740463 |

Accuracy on Trigger Images |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.7 |0.7 |0.8 |0.7333333333|0.04714045208| |10% |0.2 |0.3 |0.4 |0.3 |0.08164965809| |20% |0.1 |0.1 |0.1 |0.1 |0 | |30% |0.3 |0.4 |0.3 |0.3333333333|0.04714045208|

Attack Success Rate |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0 |0 |0 |0 |0 | |10% |0.7 |0.5 |0.6 |0.6 |0.08164965809| |20% |0.7 |0.8 |0.6 |0.7 |0.08164965809| |30% |0.7 |0.6 |0.4 |0.5666666667|0.1247219129 |

Perfect Filter

Accuracy on Non-trigger Images |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.743944|0.738238|0.733934|0.7387053333|0.004099904498| |10% |0.741041|0.741842|0.739339|0.7407406667|0.001043680134| |20% |0.732232|0.733534|0.729229|0.731665 |0.001802659702| |30% |0.727928|0.720921|0.726527|0.7251253333|0.003027431731|

Accuracy on Trigger Images |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.5 |0.5 |0.5 |0.5 |0 | |10% |0.7 |0.6 |0.5 |0.6 |0.08164965809| |20% |0.6 |0.6 |0.6 |0.6 |0 | |30% |0.5 |0.6 |0.5 |0.5333333333|0.04714045208|

Attack Success Rate |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0 |0 |0.1 |0.03333333333|0.04714045208| |10% |0.1 |0 |0 |0.03333333333|0.04714045208| |20% |0 |0 |0 |0 |0 | |30% |0 |0 |0 |0 |0 |

Random Filter

Accuracy on Non-trigger Images |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.728128|0.728128|0.708609|0.722523 |0.009201344841| |10% |0.715716|0.714214|0.716717|0.715549 |0.001028646036| |20% |0.702503|0.717417|0.700901|0.7069403333|0.007436935271| |30% |0.703203|0.688488|0.696597|0.696096 |0.006017810067|

Accuracy on Trigger Images |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.6 |0.6 |0.6 |0.6 |0 | |10% |0.1 |0.2 |0.2 |0.1666666667|0.04714045208| |20% |0 |0.1 |0 |0.03333333333|0.04714045208| |30% |0.1 |0.1 |0.1 |0.1 |0 |

Attack Success Rate |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.1 |0.1 |0.1 |0.1 |0 | |10% |0.9 |0.7 |0.6 |0.7333333333|0.1247219129 | |20% |0.8 |0.5 |0.8 |0.7 |0.1414213562 | |30% |0.7 |0.7 |0.9 |0.7666666667|0.09428090416|

Activation Clustering

Accuracy on Non-trigger Images |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.664665|0.658559|0.657257|0.6601603333|0.003229325764| |10% |0.65966|0.660561|0.651451|0.657224 |0.004098666206| |20% |0.667067|0.64004|0.643544|0.650217 |0.01200031608| |30% |0.637037|0.630731|0.645846|0.6378713333|0.006198811194|

Accuracy on Trigger Images |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.3 |0.4 |0.3 |0.3333333333|0.04714045208| |10% |0.4 |0.2 |0.4 |0.3333333333|0.09428090416| |20% |0.3 |0.1 |0 |0.1333333333|0.1247219129 | |30% |0.1 |0.2 |0.2 |0.1666666667|0.04714045208|

Attack Success Rate |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.1 |0 |0 |0.03333333333|0.04714045208| |10% |0.6 |0.3 |0.5 |0.4666666667|0.1247219129 | |20% |0.6 |0.7 |0.6 |0.6333333333|0.04714045208| |30% |0.8 |0.8 |0.5 |0.7 |0.1414213562 |

Spectral Signatures

Accuracy on Non-trigger Images |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.687487|0.683483|0.67958|0.6835166667|0.003228107013| |10% |0.664765|0.667067|0.667568|0.6664666667|0.001220519653| |20% |0.673373|0.655656|0.664064|0.6643643333|0.007236051978| |30% |0.63994|0.657257|0.664064|0.6537536667|0.01015535499|

Accuracy on Trigger Images |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0.7 |0.6 |0.6 |0.6333333333|0.04714045208| |10% |0.2 |0.2 |0.2 |0.2 |0 | |20% |0 |0.1 |0.1 |0.06666666667|0.04714045208| |30% |0 |0 |0.1 |0.03333333333|0.04714045208|

Attack Success Rate |Poison Ratio |Run 1|Run 2|Run 3|Mean |Std | |-------------------|-----|-----|-----|------------|-------------| |0% |0 |0.1 |0 |0.03333333333|0.04714045208| |10% |0.7 |0.5 |0.8 |0.6666666667|0.1247219129 | |20% |0.6 |0.6 |0.6 |0.6 |0 | |30% |0.6 |0.8 |0.3 |0.5666666667|0.2054804668 |