fairlearn

A Python package to assess and improve fairness of machine learning models.

1,992

430

1,992

168

View on GitHub

Top Related Projects

responsible-ai-toolbox

1,441

Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.

AIF360

2,503

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.

causalml

5,295

Uplift modeling and causal inference with machine learning algorithms

Quick Overview

Fairlearn is an open-source Python library that empowers developers to assess and improve the fairness of their machine learning models. It provides a set of tools for measuring and mitigating unfairness in classification and regression problems, helping to address bias and discrimination in AI systems.

Pros

Comprehensive set of fairness metrics and mitigation algorithms
Easy integration with popular machine learning libraries like scikit-learn
Extensive documentation and examples for various use cases
Active community and regular updates

Cons

Limited support for deep learning frameworks
Some advanced fairness concepts may require additional background knowledge
Performance overhead when applying certain mitigation techniques
Primarily focused on binary classification scenarios

Code Examples

Measuring fairness metrics:

from fairlearn.metrics import demographic_parity_difference

y_true = [0, 1, 1, 0, 1]
y_pred = [0, 1, 1, 1, 1]
sensitive_features = ['A', 'A', 'B', 'B', 'A']

dpd = demographic_parity_difference(y_true, y_pred, sensitive_features=sensitive_features)
print(f"Demographic Parity Difference: {dpd}")

Applying a fairness constraint:

from fairlearn.reductions import ExponentiatedGradient, DemographicParity
from sklearn.linear_model import LogisticRegression

constraint = DemographicParity()
mitigator = ExponentiatedGradient(LogisticRegression(), constraint)
mitigator.fit(X_train, y_train, sensitive_features=sensitive_features_train)

y_pred = mitigator.predict(X_test)

Visualizing fairness metrics:

from fairlearn.metrics import MetricFrame
from fairlearn.metrics import false_positive_rate
from fairlearn.metrics import true_positive_rate
import matplotlib.pyplot as plt

metric_frame = MetricFrame(metrics={'fpr': false_positive_rate, 'tpr': true_positive_rate},
                           y_true=y_true,
                           y_pred=y_pred,
                           sensitive_features=sensitive_features)

metric_frame.by_group.plot.bar(
    subplots=True,
    layout=[1, 2],
    figsize=[12, 4],
    legend=False,
    title='False and True Positive Rates by Group')

plt.show()

Getting Started

To get started with Fairlearn, follow these steps:

Install the library:

pip install fairlearn

Import necessary modules:

from fairlearn.metrics import demographic_parity_difference
from fairlearn.reductions import ExponentiatedGradient, DemographicParity
from sklearn.linear_model import LogisticRegression

Load your data and split it into training and testing sets.
Apply fairness metrics and mitigation techniques:

# Measure fairness
dpd = demographic_parity_difference(y_true, y_pred, sensitive_features=sensitive_features)

# Apply fairness constraint
constraint = DemographicParity()
mitigator = ExponentiatedGradient(LogisticRegression(), constraint)
mitigator.fit(X_train, y_train, sensitive_features=sensitive_features_train)

# Make predictions
y_pred_mitigated = mitigator.predict(X_test)

Analyze and visualize results using Fairlearn's built-in tools.

Competitor Comparisons

responsible-ai-toolbox

1,441

Pros of Responsible AI Toolbox

Broader scope, covering explainability, fairness, and error analysis
More comprehensive visualization tools and interactive dashboards
Integration with Azure Machine Learning for enterprise-level deployment

Cons of Responsible AI Toolbox

Steeper learning curve due to more complex features
Less focus on specific fairness metrics compared to Fairlearn
Requires more computational resources for some advanced features

Code Comparison

Fairlearn example:

from fairlearn.metrics import demographic_parity_difference
from fairlearn.reductions import DemographicParity

dp = DemographicParity()
mitigator = dp.fit(X, y, sensitive_features=A)
y_pred = mitigator.predict(X)

Responsible AI Toolbox example:

from raiwidgets import FairnessDashboard
from responsibleai import RAIInsights

rai_insights = RAIInsights(model, train_data, test_data, target_column, task_type='classification')
rai_insights.compute()
FairnessDashboard(rai_insights)

Both libraries offer tools for assessing and mitigating fairness in machine learning models. Fairlearn focuses specifically on fairness metrics and mitigation algorithms, while Responsible AI Toolbox provides a more comprehensive suite of tools for responsible AI practices, including fairness, explainability, and error analysis. The choice between them depends on the specific needs of the project and the desired level of depth in fairness analysis versus broader responsible AI considerations.

AIF360

2,503

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.

Pros of AIF360

More comprehensive set of fairness metrics and algorithms
Supports multiple programming languages (Python, R, and JavaScript)
Includes educational resources and tutorials for understanding fairness concepts

Cons of AIF360

Steeper learning curve due to its extensive feature set
Less integration with popular machine learning frameworks
Slower development and update cycle compared to Fairlearn

Code Comparison

Fairlearn example:

from fairlearn.metrics import demographic_parity_difference
from fairlearn.reductions import DemographicParity

dp = DemographicParity()
mitigator = dp.fit(X, y, sensitive_features=A)
y_pred = mitigator.predict(X)

AIF360 example:

from aif360.datasets import BinaryLabelDataset
from aif360.metrics import BinaryLabelDatasetMetric
from aif360.algorithms.preprocessing import Reweighing

dataset = BinaryLabelDataset(df=df, label_name='label', protected_attribute_names=['race'])
metric = BinaryLabelDatasetMetric(dataset, unprivileged_groups=unprivileged_groups, privileged_groups=privileged_groups)
reweighing = Reweighing(unprivileged_groups=unprivileged_groups, privileged_groups=privileged_groups)
dataset_transformed = reweighing.fit_transform(dataset)

Both libraries offer tools for assessing and mitigating bias in machine learning models, but they differ in their approach and complexity. Fairlearn focuses on simplicity and integration with popular ML frameworks, while AIF360 provides a more comprehensive set of fairness-related tools and educational resources.

causalml

5,295

Uplift modeling and causal inference with machine learning algorithms

Pros of CausalML

Focuses on causal inference and uplift modeling, offering a broader range of causal analysis tools
Provides advanced techniques for heterogeneous treatment effect estimation
Includes visualization tools for causal analysis results

Cons of CausalML

Less emphasis on fairness-specific metrics and algorithms
May have a steeper learning curve for users not familiar with causal inference concepts
Smaller community and fewer contributors compared to Fairlearn

Code Comparison

Fairlearn (fairness assessment):

from fairlearn.metrics import demographic_parity_difference
demographic_parity_difference(y_true, y_pred, sensitive_features=A)

CausalML (uplift modeling):

from causalml.inference.meta import XGBTRegressor
uplift_model = XGBTRegressor()
uplift_model.fit(X, treatment, y)

Both libraries offer valuable tools for responsible machine learning, but they focus on different aspects. Fairlearn specializes in fairness assessment and mitigation, while CausalML emphasizes causal inference and uplift modeling. The choice between them depends on the specific needs of your project, whether it's addressing fairness concerns or understanding causal relationships in your data.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Fairlearn

Fairlearn is a Python package that empowers developers of artificial intelligence (AI) systems to assess their system's fairness and mitigate any observed unfairness issues. Fairlearn contains mitigation algorithms as well as metrics for model assessment. Besides the source code, this repository also contains Jupyter notebooks with examples of Fairlearn usage.

Website: https://fairlearn.org/

What we mean by *fairness* <#what-we-mean-by-fairness>__
Overview of Fairlearn <#overview-of-fairlearn>__
Getting started with fairlearn <#getting-started-with-fairlearn>__
Maintainers <#maintainers>__
Code of conduct <#code-of-conduct>__
Issues <#issues>__

What we mean by fairness

An AI system can behave unfairly for a variety of reasons. In Fairlearn, we define whether an AI system is behaving unfairly in terms of its impact on people â i.e., in terms of harms. We focus on two kinds of harms:

Allocation harms. These harms can occur when AI systems extend or withhold opportunities, resources, or information. Some of the key applications are in hiring, school admissions, and lending.
Quality-of-service harms. Quality of service refers to whether a system works as well for one person as it does for another, even if no opportunities, resources, or information are extended or withheld.

We follow the approach known as group fairness, which asks: Which groups of individuals are at risk for experiencing harms? The relevant groups need to be specified by the data scientist and are application specific.

Group fairness is formalized by a set of constraints, which require that some aspect (or aspects) of the AI system's behavior be comparable across the groups. The Fairlearn package enables assessment and mitigation of unfairness under several common definitions. To learn more about our definitions of fairness, please visit our user guide on Fairness of AI Systems <https://fairlearn.org/main/user_guide/fairness_in_machine_learning.html#fairness-of-ai-systems>__.

*Note*: Fairness is fundamentally a sociotechnical challenge. Many
aspects of fairness, such as justice and due process, are not
captured by quantitative fairness metrics. Furthermore, there are
many quantitative fairness metrics which cannot all be satisfied
simultaneously. Our goal is to enable humans to assess different
mitigation strategies and then make trade-offs appropriate to their
scenario.

Overview of Fairlearn

The Fairlearn Python package has two components:

Metrics for assessing which groups are negatively impacted by a model, and for comparing multiple models in terms of various fairness and accuracy metrics.
Algorithms for mitigating unfairness in a variety of AI tasks and along a variety of fairness definitions.

Fairlearn metrics


Check out our in-depth `guide on the Fairlearn metrics <https://fairlearn.org/main/user_guide/assessment>`__.

Fairlearn algorithms

For an overview of our algorithms please refer to our website <https://fairlearn.org/main/user_guide/mitigation/index.html>__.

Getting Started with Fairlearn

First steps


- Install via pip: ``pip install fairlearn``

- Visit the `Quickstart guide <https://fairlearn.org/main/quickstart.html>`__.

- **Learning Resources**:

  - Read the comprehensive `user guide <https://fairlearn.org/main/user_guide/index.html>`__.

  - Look through the `example notebooks <https://fairlearn.org/main/auto_examples/index.html>`__.


For Users & Practitioners

Browse the example gallery <https://fairlearn.org/main/auto_examples/index.html>__. Please note that notebooks downloaded from main may not be compatible with pip-installed versions.
Check the API reference <https://fairlearn.org/main/api_reference/index.html>__.
Get Help:
- Ask questions on Stack Overflow <https://stackoverflow.com/questions/tagged/fairlearn>__ with tag fairlearn.
- Join the Discord community <https://discord.gg/R22yCfgsRn>__ for discussions.

For Contributors


- Read the `contributor guide <https://fairlearn.org/main/contributor_guide/index.html>`__.

- Check out the `good first issues <https://github.com/fairlearn/fairlearn/labels/good%20first%20issue>`__.

- Follow the `development process <https://fairlearn.org/main/contributor_guide/development_process.html>`__.

- Join the `Discord <https://discord.gg/R22yCfgsRn>`__ for contributor discussions. Please use the ``#development`` channel.


Maintainers
-----------

A list of current maintainers is
`on our website <https://fairlearn.org/main/about/index.html>`__.

Code of conduct
---------------
Fairlearn follows the `Fairlearn Organization's Code of Conduct <https://github.com/fairlearn/governance/blob/main/code-of-conduct.md>`__.

Issues
------

Regular (non-security) issues

Issues are meant for bugs, feature requests, and documentation improvements. Please submit a report through GitHub issues <https://github.com/fairlearn/fairlearn/issues>__. A maintainer will respond promptly as appropriate.

Maintainers will try to link duplicate issues when possible.

Reporting security issues


To report security issues please send an email to
``fairlearn-internal@python.org``.

.. |MIT license| image:: https://img.shields.io/badge/License-MIT-blue.svg
   :target: https://github.com/fairlearn/fairlearn/blob/main/LICENSE
.. |PyPI| image:: https://img.shields.io/pypi/v/fairlearn?color=blue
   :target: https://pypi.org/project/fairlearn/
.. |Discord| image:: https://img.shields.io/discord/840099830160031744
   :target: https://discord.gg/R22yCfgsRn
.. |StackOverflow| image:: https://img.shields.io/badge/StackOverflow-questions-blueviolet
   :target: https://stackoverflow.com/questions/tagged/fairlearn

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot