GTSRB Clean-label Backdoor Baseline Evaluation

Results obtained using Armory ~0.14.X (March 2022)

Bullet Hole Trigger

Undefended

Accuracy on Benign Test Data Source Class | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.967 | 0.961 | 0.963 | 0.963 | 0.002 | | 20 | 0.971 | 0.965 | 0.975 | 0.970 | 0.004 | | 50 | 0.967 | 0.964 | 0.961 | 0.964 | 0.002 | | 80 | 0.967 | 0.971 | 0.958 | 0.965 | 0.005 |

Accuracy on Benign Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.928 | 0.930 | 0.925 | 0.928 | 0.002 | | 20 | 0.930 | 0.926 | 0.932 | 0.929 | 0.002 | | 50 | 0.925 | 0.927 | 0.928 | 0.927 | 0.001 | | 80 | 0.927 | 0.925 | 0.924 | 0.925 | 0.001 |

Attack Success Rate | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.194 | 0.222 | 0.257 | 0.225 | 0.026 | | 50 | 0.237 | 0.237 | 0.233 | 0.236 | 0.002 | | 80 | 0.267 | 0.276 | 0.261 | 0.268 | 0.006 |

Accuracy on Poisoned Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.920 | 0.914 | 0.917 | 0.917 | 0.002 | | 50 | 0.913 | 0.915 | 0.916 | 0.914 | 0.001 | | 80 | 0.913 | 0.909 | 0.910 | 0.911 | 0.001 |

Random Filter

Accuracy on Benign Test Data Source Class | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.961 | 0.950 | 0.947 | 0.953 | 0.006 | | 20 | 0.968 | 0.961 | 0.967 | 0.965 | 0.003 | | 50 | 0.969 | 0.971 | 0.971 | 0.970 | 0.001 | | 80 | 0.981 | 0.954 | 0.972 | 0.969 | 0.011 |

Accuracy on Benign Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.925 | 0.923 | 0.923 | 0.924 | 0.001 | | 20 | 0.925 | 0.924 | 0.924 | 0.924 | 0.000 | | 50 | 0.926 | 0.929 | 0.925 | 0.927 | 0.002 | | 80 | 0.928 | 0.924 | 0.929 | 0.927 | 0.002 |

Attack Success Rate | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.236 | 0.207 | 0.237 | 0.227 | 0.014 | | 50 | 0.253 | 0.250 | 0.239 | 0.247 | 0.006 | | 80 | 0.246 | 0.279 | 0.275 | 0.267 | 0.015 |

Accuracy on Poisoned Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.912 | 0.913 | 0.911 | 0.912 | 0.001 | | 50 | 0.912 | 0.916 | 0.912 | 0.913 | 0.002 | | 80 | 0.914 | 0.910 | 0.914 | 0.913 | 0.002 |

Activation Clustering

Accuracy on Benign Test Data Source Class | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.915 | 0.886 | 0.842 | 0.881 | 0.030 | | 20 | 0.906 | 0.897 | 0.912 | 0.905 | 0.006 | | 50 | 0.894 | 0.849 | 0.861 | 0.868 | 0.019 | | 80 | 0.904 | 0.910 | 0.907 | 0.907 | 0.002 |

Accuracy on Benign Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.871 | 0.866 | 0.861 | 0.866 | 0.004 | | 20 | 0.873 | 0.879 | 0.874 | 0.875 | 0.003 | | 50 | 0.854 | 0.858 | 0.865 | 0.859 | 0.004 | | 80 | 0.866 | 0.861 | 0.854 | 0.860 | 0.005 |

Attack Success Rate | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.197 | 0.292 | 0.256 | 0.248 | 0.039 | | 50 | 0.303 | 0.265 | 0.282 | 0.283 | 0.015 | | 80 | 0.250 | 0.260 | 0.272 | 0.261 | 0.009 |

Accuracy on Poisoned Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.866 | 0.867 | 0.862 | 0.865 | 0.002 | | 50 | 0.841 | 0.849 | 0.855 | 0.848 | 0.005 | | 80 | 0.854 | 0.849 | 0.842 | 0.848 | 0.005 |

Perfect Filter

Accuracy on Benign Test Data Source Class | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.965 | 0.963 | 0.963 | 0.963 | 0.001 | | 20 | 0.968 | 0.964 | 0.958 | 0.963 | 0.004 | | 50 | 0.963 | 0.969 | 0.961 | 0.964 | 0.004 | | 80 | 0.971 | 0.971 | 0.969 | 0.970 | 0.001 |

Accuracy on Benign Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.928 | 0.929 | 0.925 | 0.928 | 0.002 | | 20 | 0.927 | 0.929 | 0.926 | 0.927 | 0.001 | | 50 | 0.926 | 0.929 | 0.927 | 0.927 | 0.001 | | 80 | 0.929 | 0.929 | 0.927 | 0.928 | 0.001 |

Attack Success Rate | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.126 | 0.097 | 0.133 | 0.119 | 0.016 | | 50 | 0.092 | 0.082 | 0.147 | 0.107 | 0.029 | | 80 | 0.126 | 0.118 | 0.100 | 0.115 | 0.011 |

Accuracy on Poisoned Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.920 | 0.923 | 0.918 | 0.920 | 0.002 | | 50 | 0.919 | 0.921 | 0.919 | 0.920 | 0.001 | | 80 | 0.922 | 0.919 | 0.920 | 0.920 | 0.001 |

Peace Sign Trigger

Undefended

Accuracy on Benign Test Data Source Class | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.968 | 0.964 | 0.972 | 0.968 | 0.003 | | 20 | 0.971 | 0.964 | 0.975 | 0.970 | 0.005 | | 50 | 0.964 | 0.956 | 0.969 | 0.963 | 0.006 | | 80 | 0.964 | 0.972 | 0.965 | 0.967 | 0.004 |

Accuracy on Benign Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.930 | 0.927 | 0.926 | 0.928 | 0.002 | | 20 | 0.932 | 0.934 | 0.932 | 0.933 | 0.001 | | 50 | 0.926 | 0.926 | 0.929 | 0.927 | 0.002 | | 80 | 0.929 | 0.928 | 0.928 | 0.928 | 0.000 |

Attack Success Rate | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.676 | 0.560 | 0.597 | 0.611 | 0.049 | | 50 | 0.547 | 0.533 | 0.660 | 0.580 | 0.057 | | 80 | 0.643 | 0.649 | 0.679 | 0.657 | 0.016 |

Accuracy on Poisoned Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.893 | 0.900 | 0.898 | 0.897 | 0.003 | | 50 | 0.895 | 0.896 | 0.892 | 0.895 | 0.002 | | 80 | 0.893 | 0.891 | 0.888 | 0.891 | 0.002 |

Random Filter

Accuracy on Benign Test Data Source Class | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.964 | 0.958 | 0.956 | 0.959 | 0.003 | | 20 | 0.960 | 0.963 | 0.965 | 0.963 | 0.002 | | 50 | 0.957 | 0.958 | 0.967 | 0.961 | 0.004 | | 80 | 0.969 | 0.961 | 0.965 | 0.965 | 0.003 |

Accuracy on Benign Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.919 | 0.924 | 0.924 | 0.922 | 0.002 | | 20 | 0.924 | 0.926 | 0.924 | 0.925 | 0.001 | | 50 | 0.924 | 0.926 | 0.922 | 0.924 | 0.002 | | 80 | 0.927 | 0.922 | 0.924 | 0.924 | 0.002 |

Attack Success Rate | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.637 | 0.653 | 0.624 | 0.638 | 0.012 | | 50 | 0.611 | 0.581 | 0.656 | 0.616 | 0.031 | | 80 | 0.622 | 0.646 | 0.710 | 0.659 | 0.037 |

Accuracy on Poisoned Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.885 | 0.887 | 0.887 | 0.886 | 0.001 | | 50 | 0.890 | 0.894 | 0.885 | 0.889 | 0.004 | | 80 | 0.891 | 0.885 | 0.885 | 0.887 | 0.003 |

Activation Clustering

Accuracy on Benign Test Data Source Class | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.907 | 0.929 | 0.914 | 0.917 | 0.009 | | 20 | 0.858 | 0.897 | 0.889 | 0.881 | 0.017 | | 50 | 0.914 | 0.910 | 0.929 | 0.918 | 0.008 | | 80 | 0.858 | 0.926 | 0.892 | 0.892 | 0.028 |

Accuracy on Benign Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.861 | 0.871 | 0.872 | 0.868 | 0.005 | | 20 | 0.865 | 0.852 | 0.870 | 0.862 | 0.008 | | 50 | 0.871 | 0.865 | 0.871 | 0.869 | 0.003 | | 80 | 0.856 | 0.861 | 0.860 | 0.859 | 0.002 |

Attack Success Rate | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.579 | 0.685 | 0.621 | 0.628 | 0.043 | | 50 | 0.660 | 0.607 | 0.675 | 0.647 | 0.029 | | 80 | 0.671 | 0.722 | 0.700 | 0.698 | 0.021 |

Accuracy on Poisoned Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.838 | 0.816 | 0.838 | 0.831 | 0.011 | | 50 | 0.838 | 0.833 | 0.835 | 0.836 | 0.002 | | 80 | 0.823 | 0.823 | 0.825 | 0.824 | 0.001 |

Perfect Filter

Accuracy on Benign Test Data Source Class | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.965 | 0.965 | 0.965 | 0.965 | 0.000 | | 20 | 0.958 | 0.960 | 0.958 | 0.959 | 0.001 | | 50 | 0.974 | 0.972 | 0.963 | 0.969 | 0.005 | | 80 | 0.965 | 0.963 | 0.972 | 0.967 | 0.004 |

Accuracy on Benign Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | 0.929 | 0.928 | 0.929 | 0.929 | 0.000 | | 20 | 0.926 | 0.925 | 0.929 | 0.927 | 0.002 | | 50 | 0.926 | 0.926 | 0.928 | 0.927 | 0.001 | | 80 | 0.926 | 0.927 | 0.927 | 0.927 | 0.000 |

Attack Success Rate | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.222 | 0.214 | 0.151 | 0.196 | 0.032 | | 50 | 0.047 | 0.064 | 0.057 | 0.056 | 0.007 | | 80 | 0.253 | 0.272 | 0.118 | 0.214 | 0.069 |

Accuracy on Poisoned Test Data All Classes | Poison Ratio | run1 | run2 | run3 | mean | std | |--------------|------|------|------|------|-----| | 0 | - |- |- |- |- | | 20 | 0.895 | 0.899 | 0.903 | 0.899 | 0.003 | | 50 | 0.902 | 0.905 | 0.910 | 0.905 | 0.003 | | 80 | 0.900 | 0.899 | 0.903 | 0.901 | 0.001 |