Running Armory Tests

Tests have to download a bunch of code (external repos,etc.) and model weights the first time around so that one can take a while.

You will need to have the ARMORY_GITHUB_TOKEN env variable set (which may be done by armory configure...but will need to make sure)

Can use pytest -s to run all tests:

pytest -s ./tests/`

To only run a single file:

pytest -s ./tests/test_file.py

or to run only a single tests

pytest -s ./tests/test_file.py::test_name

If a test is parameterized to see how to only run one of the parameters sets use:

pytest --collect-only -q 

Then run the one you want (for example):

 pytest -s tests/test_models.py::test_model_creation[armory.baseline_models.pytorch.cifar-get_art_model-None-cifar10-500-1-1-100-1-numpy-0.25]

Running pytest in Docker

When running pytest with docker, you have two choices.

First, you can rebuild the docker container and then run pytest with the container:

python docker/build.py -f pytorch --no-pull
armory exec pytorch -- pytest -s ./tests/

The armory exec pytorch is equivalent to launching an interactive container with armory launch pytorch, bashing into the container, and running pytest -s ./tests/.

Or, if rebuilding the container is onerous, you can just do the armory exec command, but you need to make sure that pytest is invoked with python -m. Otherwise, tests will import armory installed in the container, not your locally modified dev version.

armory exec pytorch -- python -m pytest -s ./tests/