Welcome to reliability-checklist’s documentation!

reliability-checklist attempts to overcome the limitations of Language-Models on robustness.

This tool is developed in such a way that you don’t need to do any coding as far as your pre-trained model is available on huggingface space or developed using transformers library.

If you are new to the reliability-checklist, please use following links for installation guidance and example tutorial:

installation

reliability-checklist 101

reliability-checklist 201

If you have already setup the tool, then feel free to refer following pages for BYoM (Bring Your own Model) based evaluations:

BYoM (Bring Your own Model)

BYoD (Bring Your own Data)

Contents:

• Installation
  • Develop installation guidelines
• reliability-tool 101
  • Run a sample reliability tests on MNLI dataset:
  • Using on different devices:
  • Saving the output of the reliability tests:
  • Going beyond user and configuring each experiments:
• reliability-tool 201
  • Remove any specific functionality:
  • Pass different value to specific parameters:
• BYoM (Bring Your own Model)
  • How to specify your model to reliability-checklist for tests?
  • Pre-defined list of templates:
  • How to create template from scratch?
  • Where to store new templates?
• BYoD (Bring Your own Data)
  • How to specify your dataset to reliability-checklist for tests?
  • Pre-defined list of templates:
  • How to create template from scratch?
  • Where to store new templates?

Indices and tables

• Index

• Module Index

• Search Page