Flexible tool for bias detection, visualization, and mitigation. Use
models explained with DALEX and calculate
fairness classification metrics based on confusion matrices using
fairness_check() or try newly developed module for
regression models using fairness_check_regression(). R
package fairmodels allows to compare and gain information about various
machine learning models. Mitigate bias with various pre-processing and
post-processing techniques. Make sure your models are classifying
protected groups similarly.
Install it from CRAN:
install.packages("fairmodels")or developer version from GitHub:
devtools::install_github("ModelOriented/fairmodels")Checking fairness is easy!
library(fairmodels)
library(ranger)
library(DALEX)
data("german")
# ------------ step 1 - create model(s) -----------------
lm_model <- glm(Risk~.,
data = german,
family=binomial(link="logit"))
rf_model <- ranger(Risk ~.,
data = german,
probability = TRUE,
num.trees = 200)
# ------------ step 2 - create explainer(s) ------------
# numeric y for explain function
y_numeric <- as.numeric(german$Risk) -1
explainer_lm <- explain(lm_model, data = german[,-1], y = y_numeric)
explainer_rf <- explain(rf_model, data = german[,-1], y = y_numeric)
# ------------ step 3 - fairness check -----------------
fobject <- fairness_check(explainer_lm, explainer_rf,
protected = german$Sex,
privileged = "male")
print(fobject)
plot(fobject)
Compas recidivism data use case: Basic
tutorial
Bias mitigation techniques on Adult data: Advanced
tutorial
fairness_check parameters are
* x, … - explainers and fairness_objects
(products of fairness_check).
* protected - factor with different subgroups as levels. Usually
specific race, sex etc…
* privileged - subgroup, base on which to calculate parity loss
metrics.
* cutoff - custom cutoff, might be single value - cutoff same for all
subgroups or vector - for each subgroup individually. Affecting only
explainers.
* label - character vector for every explainer.
Models might be trained on different data, even without protected
variable. May have different cutoffs which gives different values of
metrics. fairness_check() is place where
explainers and fairness_objects are checked
for compatibility and then glued together.
So it is possible to to something like this:
fairness_object <- fairness_check(explainer1, explainer2, ...)
fairness_object <- fairness_check(explainer3, explainer4, fairness_object, ...)even with more fairness_objects!
If one is even more keen to know how fairmodels works
and what are relations between objects, please look at this diagram class
diagram
There are 12 metrics based on confusion matrix :
| Metric | Formula | Full name | fairness names while checking among subgroups |
|---|---|---|---|
| TPR |  |
true positive rate | equal opportunity |
| TNR |  |
true negative rate | |
| PPV |  |
positive predictive value | predictive parity |
| NPV |  |
negative predictive value | |
| FNR |  |
false negative rate | |
| FPR |  |
false positive rate | predictive equality |
| FDR |  |
false discovery rate | |
| FOR |  |
false omission rate | |
| TS |  |
threat score | |
| STP |  |
statistical parity | statistical parity |
| ACC |  |
accuracy | Overall accuracy equality |
| F1 |  |
F1 score |
and their parity loss.
How is parity loss calculated?
Where i denotes the membership to unique subgroup from
protected variable. Unprivileged subgroups are represented by small
letters and privileged by simply “privileged”.
some fairness metrics like Equalized odds are satisfied if parity loss in both TPR and FPR is low
It is relatively easy! Check it out here
R package fairmodels has support for regression models. Check
fairness using fairness_check_regression() to approximate
classification fairness metrics in regression setting. Plot object with
plot() to visualize fairness check or with
plot_density() to see model’s output.
Zafar, Valera, Rodriguez, Gummadi (2017) https://arxiv.org/pdf/1610.08452.pdf
Barocas, Hardt, Narayanan (2019) https://fairmlbook.org/
Steinberg, Daniel & Reid, Alistair & O’Callaghan, Simon. (2020). Fairness Measures for Regression via Probabilistic Classification. - https://arxiv.org/pdf/2001.06089.pdf
