Usage

Installation

To use CountergenTorch, first install it using pip:

(.venv) $ pip install countergen

Diagram of How Evaluation Works

Abstract Workflow

Outline of the evaluation process

Data Augmentation

First, you produce a list of AugmentedSample, either by loading an existing one, or using tools the library provides to build you it from raw data, or by creating your own from scratch.

Where a Variation is defined as follows.

Model Loading

Second, you load your model and turn it into a ModelEvaluator, which is just a callable that returns the performance of a model given an input and expected outputs.

Model Evaluation

Third, you pass your list of AugmentedSample and your ModelEvaluator into the following function:

By default, it will return the average performance on each kind of data. You can specify other ways to aggregate the performance on each variation by passing another Aggregator.

Alternatively, if you just want to print or save the results, directly evaluate_and_print() or evaluate_and_save()

And that’s it! countergen and countergenedit provide powerful ways to generate variations, load models easily, and edit them to decrease performance gaps between different categories. Click on Next to know more.