Welcome to CounterGen’s documentation!

CounterGen is a framework for generating counterfactual datasets, evaluating NLP models, and editing models to reduce bias. It provides powerful defaults, while offering simple ways to use your own data, data augmentation techniques, models, and evaluation metrics.

The project is free and open source. The code can be found here: https://github.com/FabienRoger/Countergen.

This project is split in two parts:

• countergen is a lightweight Python module which helps you generate counterfactual datasets and evaluate model bias. It provides useful default datasets, lightweight augmentation techniques, and evaluation of models available through an API.

• countergenedit is a Python module which adds methods to easily evaluate PyTorch text generation models as well as text classifiers. It provides tools to analyze model activation and quickly edit model to reduce bias.

Check out the countergen usage and the countergenedit usage for more information about installation and usage of these modules.

Note

This project is under development. Please report issues in the appropriate section of the Github repository.

Note

This module only works for Python version 3.7 and above

Contents

• Demos
  • Minimal examples
    • Model evaluation
    • Data augmentation
    • Model editing
  • Colab notebook
• Work We Use

countergen

• Usage
  • Installation
  • Diagram of How Evaluation Works
  • Abstract Workflow
  • Outline of the evaluation process
• Data Loading and Augmentation
  • A. Loading default augmented datasets
  • B. Loading your data and augmenting it
• Model Loading
  • What is a ModelEvaluator
  • Create ModelEvaluator from an API
  • Create ModelEvaluator from a local model
  • Examples of ModelEvaluator
• Model Evaluation
  • Aggregator Selection
  • Model Evaluation

countergenedit

• Usage
• PyTorch Model Loading
• Measuring Internal Bias
• Finding the Relevant Directions
• Editing the Model