Armory can be installed from PyPi:
pip install armory-testbed
When a user runs a given configuration file, the necessary docker image, datasets and model weights will be pulled as needed. We do have convenience functions to download all images, datasets and model weights for a scenario set release. This can take a while, so you may want to run it overnight:
git clone https://github.com/twosixlabs/armory-example.git cd armory-example # First set of examples: armory download scenario_download_configs/scenarios-set1.json # Second set of scenarios: armory download scenario_download_configs/scenarios-set2.json
If you are not using Docker, then add
armory download scenario_download_configs/scenarios-set*.json --no-docker
If you want to download with a specific image, use:
armory download --docker-image <image tag> scenario_download_configs/scenarios-set*.json
The armory package contains several framework specific baseline models that can be used during evaluation. Please see our documentation on baseline models for more information about what is available and what pretrained weights can be pulled from S3:
Running an evaluation
Evaluations are typically run though the use of configuration files. See the config file documentation for information regarding the schema and what the fields refer to.
To run a configuration:
git clone https://github.com/twosixlabs/armory-example.git cd armory-example armory run official_scenario_configs/cifar10_baseline.json
You may want to include code from an external repository that is outside of your current working directory project. This is fully supported by Armory and more information can be found in the external repo documentation.