Adversarial Datasets

The armory.data.adversarial_datasets module implements functionality to return adversarial datasets of various data modalities. By default, this is a NumPy ArmoryDataGenerator which implements the methods needed by the ART framework.

For most adversarial datasets, get_batch() returns a tuple of ((data_clean, data_adversarial), label_clean) for a specified batch size in numpy format, where data_clean and label_clean represent a clean example and its true label, and data_adversarial represents the corresponding adversarially attacked example. The APRICOT and DAPRICOT datasets differ in that get_batch() returns a tuple of (data_adversarial, label_adversarial).

Currently, datasets are loaded using TensorFlow Datasets from cached tfrecord files. These tfrecord files will be pulled from S3 if not available on your dataset_dir directory.

Refer to datasets.md for descriptions of the original datasets from which the adversarial datasets are created.

Usage

To use an adversarial dataset for evaluation, specify the desired values for the name and adversarial_key keywords in the attack module of a scenario configuration. Valid values for each keyword are given in the table below.

Example attack module for image classification scenario:

"attack": {
    "knowledge": "white",
    "kwargs": {
        "adversarial_key": "adversarial_univpatch",
        "batch_size": 1,
        "description": "'adversarial_key' can be 'adversarial_univperturbation' or 'adversarial_univpatch'"
    },
    "module": "armory.data.adversarial_datasets",
    "name": "resisc45_adversarial_224x224",
    "type": "preloaded"
}

Note: the APRICOT dataset contains splits for ["frcnn", "ssd", "retinanet"] rather than adversarial keys. See example below:

"attack": {
    "knowledge": "white",
    "kwargs": {
        "batch_size": 1,
        "split": "frcnn"
    },
    "module": "armory.data.adversarial_datasets",
    "name": "apricot_dev_adversarial",
    "type": "preloaded",

Image Datasets

name adversarial_key Description Attack Source Split x_shape x_type y_shape y_type Size
"apricot_dev_adversarial" ["adversarial", frcnn", "ssd", "retinanet"] * Physical Adversarial Attacks on Object Detection Targeted, universal patch dev (nb, variable_height, variable_width, 3) uint8 n/a dict 138 images
"apricot_test_adversarial" ["adversarial", frcnn", "ssd", "retinanet"] * Physical Adversarial Attacks on Object Detection Targeted, universal patch test (nb, variable_height, variable_width, 3) uint8 n/a dict 873 images
"dapricot_dev_adversarial" ["small", medium", "large"] ** Physical Adversarial Attacks on Object Detection Targeted patch dev (nb, 3, 1008, 756, 3) uint8 n/a 2-tuple 81 examples (3 images per example)
"dapricot_test_adversarial" ["small", medium", "large"] ** Physical Adversarial Attacks on Object Detection Targeted patch test (nb, 3, 1008, 756, 3) uint8 n/a 2-tuple 324 examples (3 images per example)
"imagenet_adversarial" "adversarial" ILSVRC12 adversarial image dataset for ResNet50 Targeted, universal perturbation test (nb, 224, 224, 3) uint8 (N,) int64 1000 images
"resisc45_adversarial_224x224" "adversarial_univpatch" REmote Sensing Image Scene Classification Targeted, universal patch test (nb, 224, 224, 3) uint8 (N,) int64 5 images/class
"resisc45_adversarial_224x224" "adversarial_univperturbation" REmote Sensing Image Scene Classification Untargeted, universal perturbation test (nb, 224, 224, 3) uint8 (N,) int64 5 images/class

* allowed values for apricot_dev and apricot_test dataset splits. See example APRICOT config above. The "frcnn" split, for example, is the subset of images containing a patch that was generated by attacking a Faster-RCNN model. Using the "adversarial" split returns the entire dataset. ** allowed values for dapricot_dev dataset splits. The "small" split, for example, is the subset of images containing small patch green-screens. Using the "adversarial" split returns the entire dataset.

Note: the APRICOT dataset contains labels and bounding boxes for both COCO objects and physical adversarial patches. The label used to signify the patch is the ADV_PATCH_MAGIC_NUMBER_LABEL_ID defined in armory/data/adversarial_datasets.py. Each image contains one adversarial patch and a varying number of COCO objects (in some cases zero). COCO object class labels are one-indexed (start from 1) in Armory <= 0.13.1 and zero-indexed in Armory > 0.13.1.

The D-APRICOT dataset does NOT contain labels/bounding boxes for COCO objects, which may occasionally appear in the background (e.g. car). Each image contains one green screen intended for patch insertion. The green screen shapes vary between diamond, rectangle, and octagon. A dataset example consists of three images, each of a different camera angle of the same scene and green screen.

Audio Datasets

name adversarial_key Description Attack Source Split x_shape x_type y_shape y_type sampling_rate Size
"librispeech_adversarial" "adversarial_perturbation Librispeech dev dataset for speaker identification Targeted, universal perturbation test (N, variable_length) int64 (N,) int64 16 kHz ~5 sec/speaker
"librispeech_adversarial" "adversarial_univperturbation" Librispeech dev dataset for speaker identification Untargeted, universal perturbation test (N, variable_length) int64 (N,) int64 16 kHz ~5 sec/speaker

Video Datasets

name adversarial_key Description Attack Source Split x_shape x_type y_shape y_type Size
"ucf101_adversarial_112x112" "adversarial_patch" UCF 101 Action Recognition Untargeted, universal perturbation test (N, variable_frames, 112, 112, 3) uint8 (N,) int64 5 videos/class
"ucf101_adversarial_112x112" "adversarial_perturbation" UCF 101 Action Recognition Untargeted, universal perturbation test (N, variable_frames, 112, 112, 3) uint8 (N,) int64 5 videos/class

Poison Datasets

name split_type Description Attack Source Split x_shape x_type y_shape y_type Size
"gtsrb_poison" poison German Traffic Sign Poison Dataset Data poisoning train (N, 48, 48, 3) float32 (N,) int64 2220 images
"gtsrb_poison" poison_test German Traffic Sign Poison Dataset Data poisoning test (N, 48, 48, 3) float32 (N,) int64 750 images