utils.metrics

Evaluation metrics utilities for BRIDGE experiments.

This module provides a small collection of metric functions and a lightweight accumulator class for tracking classification/regression-style metrics across training/evaluation steps. It is primarily used by training loops to compute:

• scalar summary metrics (accuracy, ROC-AUC, PR-AUC, F1, MCC)

• confusion-matrix counts (TP, TN, FP, FN)

• correlation/fit metrics for regression-like objectives (Pearson r, R^2-like score, slope)

Who this is for

• Users running BRIDGE training / validation scripts who need consistent metric reporting.

• Developers extending objectives or adding new tracked scalars (e.g., loss) via the accumulator.

This module assumes NumPy arrays as inputs and relies on scikit-learn for curve metrics.

Key dependencies

• numpy

• scikit-learn: roc_curve, auc, precision_recall_curve, accuracy_score, confusion_matrix, f1_score, matthews_corrcoef

• scipy.stats for Pearson correlation

Public API

The module exports (see __all__):

• pearsonr(label, prediction)

• rsquare(label, prediction)

• accuracy(label, prediction)

• roc(label, prediction)

• pr(label, prediction)

• calculate_metrics(label, prediction, objective)

It also defines an accumulator class:

• MLMetrics: stores per-step metric vectors and provides running averages/sums.

Input conventions

Labels and predictions

Most functions accept:

• binary targets: label shape (N,) or (N, 1) or (N, K)

• predictions: same shape as labels (probabilities/scores in [0, 1] for binary)

Multi-label behavior

For 2D inputs ((N, K)), metrics are computed per column and then aggregated with np.nanmean / np.nanstd where applicable.

Objectives in calculate_metrics

calculate_metrics(label, prediction, objective) supports:

• "binary" and "hinge"

  Treats inputs as binary (or multi-label) classification.

  Returns: mean: [acc, auc_roc, auc_pr, f1, mcc, tp, tn, fp, fn] std : [acc_std, auc_roc_std, auc_pr_std, f1_std, mcc_std]

• "categorical"

  Treats input as multi-class with one-hot labels and predicted class probabilities.

  Returns:

  mean: begins with [acc, auc_roc, auc_pr] (macro over columns),

  then appends per-class ROC-AUC for each column.

  stdbegins with [acc_std, auc_roc_std, auc_pr_std],

  then appends the corresponding per-class ROC-AUC standard deviations.

  Note:

  The current implementation appends per-class ROC-AUC only (not per-class PR-AUC).

• "squared_error", "kl_divergence", "cdf"

  Treated as regression-like objectives, but labels are thresholded to binary (0/1) first.

  Returns:

  mean: [acc, auc_roc, auc_pr, tp, tn, fp, fn, pearsonr_mean, rsquare_mean, slope_mean]

  std : [acc_std, auc_roc_std, auc_pr_std, pearsonr_std, rsquare_std, slope_std]

  Note:

  pearsonr, rsquare, and slope are computed after label thresholding.

Return value (important)

calculate_metrics returns a two-element list:

• [mean, std]

So typical usage is:

mean, std = calculate_metrics(y_true, y_pred, objective="binary")

Note that the current MLMetrics.update implementation does:

met, _ = calculate_metrics(...)

which will set met to the mean list and ignore std.

Notes on individual helpers

pearsonr

• For 1D input, returns [stats.pearsonr(label, prediction)] (a tuple inside a list).

• For 2D input, returns a list of coefficients (floats).

This asymmetry is preserved for backward compatibility.

rsquare

• Computes an R^2-like score using a no-intercept fit (regression through the origin). This differs from sklearn’s default R^2 which fits an intercept.

roc / pr

• Return both metric values and the underlying curve points for plotting.

• For multi-label inputs, curves are computed independently per column.

MLMetrics

MLMetrics stores metric vectors returned by calculate_metrics and maintains running average and sum. For objective "binary" / "hinge", it also exposes:

• acc, auc, prc, f1, mcc from the running average

• tp, tn, fp, fn from the running sum

You may append additional scalars (e.g., loss) by passing them as other_lst to update.

How to use

Compute metrics for one evaluation run:

from metrics import calculate_metrics
mean, std = calculate_metrics(y_true, y_prob, objective="binary")
acc, auc_roc, auc_pr, f1, mcc, tp, tn, fp, fn = mean

Accumulate metrics across batches:

from metrics import MLMetrics
meter = MLMetrics(objective="binary")

for y_true, y_prob in dataloader:
    meter.update(y_true, y_prob, other_lst=[loss_value])

print(meter.acc, meter.auc, meter.prc, meter.f1, meter.mcc)

Functions

accuracy(label, prediction)	Compute accuracy using np.round(prediction) as the classifier.
calculate_metrics(label, prediction, objective)	Unified metric computation for different learning objectives.
f1_sc(label, prediction)	Compute F1 score(s) using a 0.5 threshold via np.round(prediction).
mcc_sc(label, prediction)	Compute Matthews correlation coefficient (MCC) using np.round(prediction) as the classifier.
pearsonr(label, prediction)	Compute Pearson correlation(s) between labels and predictions.
pr(label, prediction)	Compute PR-AUC and precision-recall curves.
roc(label, prediction)	Compute ROC-AUC and ROC curves.
rsquare(label, prediction)	Compute an R^2-like metric and slope for a simple linear fit y ≈ m * x (no intercept).
tfnp(label, prediction)	Compute confusion-matrix counts (TP, TN, FP, FN) for binary classification.

Classes

MLMetrics([objective])

Accumulator for per-step metrics with running average and sums.

utils.metrics.pearsonr(label, prediction)

Compute Pearson correlation(s) between labels and predictions.

Parameters:

• label (np.ndarray) – Ground-truth values. Shape (N,) or (N, K).

• prediction (np.ndarray) – Predicted values. Same shape as label.

Returns:

list[float] – - If input is 1D: a single-element list containing the Pearson correlation coefficient. - If input is 2D: a list of length K containing per-column Pearson correlations.

Notes

• For 1D input, this function currently returns [stats.pearsonr(…)] (a tuple), while for 2D it returns only the coefficient (float). This is preserved as-is. If you want strict consistency, convert the 1D case to stats.pearsonr(…)[0].

utils.metrics.rsquare(label, prediction)

Compute an R^2-like metric and slope for a simple linear fit y ≈ m * x (no intercept).

For each target dimension, this fits:

m = (x · y) / (x · x)

and reports:

R^2 = 1 - ||y - m x||^2 / ||y - mean(y)||^2

Parameters:

• label (np.ndarray) – Ground-truth values. Shape (N,) or (N, K).

• prediction (np.ndarray) – Predicted values. Same shape as label.

Returns:

Tuple[list[float], list[float]] –

metric:

List of R^2 values (length 1 for 1D input, else length K).

slope:

List of slopes m (same length as metric).

Notes

• This is not the standard sklearn R^2 with intercept; it forces the regression through origin.

utils.metrics.accuracy(label, prediction)

Compute accuracy using np.round(prediction) as the classifier.

Parameters:

• label (np.ndarray) – Binary labels. Shape (N,) or (N, K).

• prediction (np.ndarray) – Predicted probabilities/scores. Same shape as label.

Returns:

np.ndarray – - Scalar array for 1D input. - Shape (K,) array for 2D input.

utils.metrics.roc(label, prediction)

Compute ROC-AUC and ROC curves.

Parameters:

• label (np.ndarray) – Binary labels. Shape (N,) or (N, K).

• prediction (np.ndarray) – Predicted scores/probabilities. Same shape as label.

Returns:

Tuple[np.ndarray, list[tuple[np.ndarray, np.ndarray]]] –

metric:

ROC-AUC value(s). Scalar array for 1D, or shape (K,) for 2D.

curves:

List of (fpr, tpr) arrays, one per label dimension.

Notes

• Uses sklearn.metrics.roc_curve and auc.

• For multi-label (2D), ROC is computed independently per label column.

utils.metrics.pr(label, prediction)

Compute PR-AUC and precision-recall curves.

Parameters:

• label (np.ndarray) – Binary labels. Shape (N,) or (N, K).

• prediction (np.ndarray) – Predicted scores/probabilities. Same shape as label.

Returns:

Tuple[np.ndarray, list[tuple[np.ndarray, np.ndarray]]] –

metric:

PR-AUC value(s), computed as AUC(recall, precision).

curves:

List of (precision, recall) arrays, one per label dimension.

utils.metrics.calculate_metrics(label, prediction, objective)

Unified metric computation for different learning objectives.

Depending on objective, this function computes a set of metrics and returns:

mean: list of aggregated metrics (nanmean over label dimensions where applicable) std: list of metric standard deviations (nanstd over label dimensions)

Parameters:

• label (np.ndarray) – Ground-truth labels/targets. - binary/hinge: shape (N,) or (N,1) or (N,K) for multi-label. - categorical: shape (N,K), typically one-hot. - squared_error/kl_divergence/cdf: numeric targets; internally thresholded at 0.5.

• prediction (np.ndarray) – Model outputs. - binary/hinge: probabilities/scores in [0,1], same shape as label. - categorical: probabilities/logits post-processed to probabilities, shape (N,K). - squared_error/kl_divergence/cdf: numeric predictions aligned to label.

• objective (str) –

  One of:

  • “binary” or “hinge”

  • “categorical”

  • “squared_error”, “kl_divergence”, or “cdf”

  Other values return (0, 0).

Returns:

Tuple[list[float], list[float]] –

mean, std:

For objective == “binary” or “hinge”:

mean = [acc, auc_roc, auc_pr, f1, mcc, tp, tn, fp, fn] std = [acc_std, auc_roc_std, auc_pr_std, f1_std, mcc_std]

For objective == “categorical”:

mean starts as [acc, auc_roc, auc_pr] and then appends per-class ROC-AUC:

mean = [acc, auc_roc_macro, auc_pr_macro, auc_roc_class0, …, auc_roc_class(K-1)]

std similarly starts as [acc_std, auc_roc_std, auc_pr_std] and appends per-class std.

For objective in {“squared_error”,”kl_divergence”,”cdf”}:

The labels are thresholded into {0,1} before classification metrics. mean = [acc, auc_roc, auc_pr, tp, tn, fp, fn, pearsonr_mean, rsquare_mean, slope_mean] std = [acc_std, auc_roc_std, auc_pr_std, pearsonr_std, rsquare_std, slope_std]

Notes

• For binary/hinge and regression-like objectives, confusion counts are computed using:

  pred_class = prediction > 0.5

  while accuracy/F1/MCC use np.round(prediction).

• If label is 2D with shape (N,1), the function flattens to 1D before confusion counts.

• Multi-label (2D) metrics are computed per column and aggregated with nanmean/nanstd.