reliability-tool 201
This section walks through an example of how to do customize reliability tests. This tutorial/ability requires the understanding of all .yaml files within the reliability-checklist.
Remove any specific functionality:
Quick solution is to add ~ before the functionality name:
recheck task=mnli ~<functionality-name>.<name-of-the-feature>
For example, our task is to remove inv_augmentation augmentation defined at inv_pass.yaml then
recheck task=mnli ~augmentation.inv_augmentation
Another example is to remove calibration_callback evaluation defined at calibration.yaml then
recheck task=mnli ~callbacks.calibration_callback
Pass different value to specific parameters:
The simple solution is to pass the value via command line argument:
recheck task=mnli callbacks.calibration_callback.num_bins=20