logging facilities and options in armory

Because the primary communication between armory and the user happens by way of logs, the system and configuration options are broad, with reasonable defaults so that it does the right thing to start.

the armory logging api

In order to use the armory logger, all you need to do is import it:

from armory.logs import log

The log object is the primary interface to the module and supports the following standard functions:

log.debug
log.info
log.warning
log.error
log.critical

The armory logger also adds two new levels

log.trace - even more verbose than debug
log.success - report that something completed ok

All these functions take one string argument

log.success(f'uploaded {file} to {server}')

There are two additional log levels without a standard function:

`"PROGRESS"` - for determining whether to log progress for downloads/uploads
`"METRIC"` - for logging metric results

The explicit ordering of these log levels are:

TRACE = 5
DEBUG = 10
PROGRESS = 15
INFO = 20
METRIC = 24
SUCCESS = 25
WARNING = 30
ERROR = 40
CRITICAL = 50

The armory logger upon import is pre-initialized and requires no configuration. So as a user of the library, that's all you need to know.

logger destinations

The armory.logs system always logs to the console, this is initialized at import time. The module also writes to two files called

colored-log.txt
armory-log.txt

where the first is a direct duplicate of the console, and the second is identical but without ansi color escapes for easier parsing. These files want to be placed in the armory output_dir in the same timestamped directory that holds the output file. However, that directory name isn't known until at armory start time so the log files are created in the configured output_dir at the start.

When armory knows the name of that timestamped directory it calls

armory.logs.make_logfiles(timestamp_directory)

TODO: The output directory needs to be created earlier in the armory initialization so that the logfile can start near the top of run.

logging level specification

As with the standard logging module, armory log messages are conditionally emitted based on their level. Messages sent by armory will be logged at the INFO level by default.

But armory.logs also handles messages sent by libraries that we call such as: art, tensorflow, boto, etc. The armory.logs module has a filter that is applied before emitting messages and is configured as a dictionary mapping the originating module name to level such as:

default_message_filters = {
    "": "TRACE"
    "armory": "INFO",
    "art": "INFO",
    "docker": "INFO",
    "botocore": "WARNING",
    "s3transfer": "INFO",
    "urllib3": "INFO",
    "absl": False,
    "h5py": False,
    "avro": False,
}

which is how the logger is configured at the start. The "" module is special, and covers all cases which aren't otherwise specified. So we start with printing all messages at TRACE and higher (which means all). For messages from armory, art, and docker, INFO is fine. The botocore and s3transfer modules are crazy chatty so we raise the threshold for them. I don't know that I've ever heard mention of debug tracing in absl, h5py, or avro so I've disabled any messages from them.

I'm not sure these are good defaults, so I am hoping the other armory devs will give opinions.

armory --log-level option

The command

armory run --log-level armory:debug --log-level art:debug ...

overrides the default log-levels for armory and art. Because argparse allows unique option substrings to be used, this command can be written more briefly as

armory run --log debug --log art:debug ...

as a convenience omitting the module: part of the level assumes armory:. The --debug option become a deprecated alias for --log-level armory:debug.

The module names are done with a simple text match, so if you see too many messages like

2022-02-24 15:31:19  4s INFO     art.estimators.classification.pytorch:get_layers:986 ...

you can adjust that down with --log art.estimators:warning or the hyper-specific --log art.estimators.classification:warning.