Optimal Classification Threshold¶
Decision Boundary in Binary Classification¶
A binary classifier produces a score $p \in [0, 1]$ representing the estimated probability that a sample belongs to the positive class. The final prediction is determined by comparing $p$ against a threshold $\tau \in (0, 1)$:
$$ \hat{y} = \begin{cases} 1 & \text{if } p \geq \tau \\ 0 & \text{if } p < \tau \end{cases} $$
The choice of $\tau$ directly controls the precision–recall trade-off:
- Increasing $\tau$ → fewer positive predictions → higher precision, lower recall.
- Decreasing $\tau$ → more positive predictions → lower precision, higher recall.
The default $\tau = 0.5$ is only optimal when:
- The class prior probabilities are equal ($P(y=0) = P(y=1)$), and
- The cost of false positives equals the cost of false negatives.
In practice, both conditions are frequently violated (class imbalance, asymmetric costs), making threshold calibration an essential step.
Threshold Search¶
find_optimal_threshold performs an exhaustive search over $K$ evenly-spaced candidate thresholds:
$$ \mathcal{T} = \left\{ \frac{k}{K} : k = 1, 2, \ldots, K-1 \right\} $$
For each candidate $\tau \in \mathcal{T}$, it:
- Binarises the probability scores: $\hat{\mathbf{y}} = \mathbb{1}[\mathbf{p} \geq \tau]$.
- Evaluates the metric: $m_\tau = \text{metric\_fn}(\mathbf{y}, \hat{\mathbf{y}})$.
The optimal threshold is:
$$ \tau^* = \arg\max_{\tau \in \mathcal{T}}\; m_\tau $$
The time complexity is $O(K \cdot n)$ where $n$ is the number of samples, making it practical for validation sets of any typical size with $K = 200$ (default).
Metric Functions¶
The function accepts any callable $m : \{0,1\}^n \times \{0,1\}^n \to \mathbb{R}$ that takes (y_true, y_pred) and returns a scalar to maximise. Common choices:
| Metric | Use case |
|---|---|
| $F_1$ (default) | Balanced precision–recall |
| Accuracy | When class balance is acceptable |
| $F_\beta$ | When recall is $\beta$ times more important than precision |
| $\min(F_1^+, F_1^-)$ | When both classes are equally important |
Important Caveat: Validation vs Test Split¶
The threshold search must be performed on a held-out validation set that is not used for training. Searching on the training set or the final test set constitutes a form of data leakage and will produce an overly optimistic estimate of threshold performance. The correct workflow is:
- Split data into train / validation / test.
- Train the model on
train. - Search for $\tau^*$ on
validation. - Apply $\tau^*$ and report metrics on
test.
API¶
sorix.metrics.find_optimal_threshold(
y_true,
y_probs,
metric_fn=None, # defaults to f1_score
*,
n_thresholds=200,
) -> tuple[float, float] # (best_threshold, best_metric_value)
A metric can be undefined at some thresholds — F1 has no meaning when a threshold
predicts no positives at all — so candidates whose evaluation raises are skipped.
If every candidate fails, the metric itself is broken, and a RuntimeError
is raised rather than returning a meaningless score.
# Uncomment the next line and run this cell to install sorix
#!pip install 'sorix @ git+https://github.com/Mitchell-Mirano/sorix.git@develop'
import numpy as np
from sorix.metrics import find_optimal_threshold, f1_score, accuracy_score
Generating Synthetic Predictions¶
We simulate a binary classifier on an imbalanced dataset (30% positive). The well-calibrated classifier assigns higher probabilities to positive samples.
rng = np.random.default_rng(42)
n = 400
y_true = rng.choice([0, 1], size=n, p=[0.70, 0.30])
y_probs = np.where(
y_true == 1,
rng.uniform(0.55, 1.0, n), # positives: higher scores
rng.uniform(0.0, 0.50, n), # negatives: lower scores
)
print(f"Positive class prevalence: {y_true.mean()*100:.1f}%")
neg_probs = y_probs[y_true == 0]
pos_probs = y_probs[y_true == 1]
print(f"Sample probabilities (class 0): {[round(x,2) for x in neg_probs[:5]]}")
print(f"Sample probabilities (class 1): {[round(x,2) for x in pos_probs[:5]]}")
Positive class prevalence: 29.2% Sample probabilities (class 0): [np.float64(0.16), np.float64(0.3), np.float64(0.06), np.float64(0.25), np.float64(0.5)] Sample probabilities (class 1): [np.float64(0.69), np.float64(0.9), np.float64(0.95), np.float64(0.68), np.float64(0.93)]
Default Threshold vs Optimal Threshold¶
We compare the F1 score at $\tau = 0.5$ (default) against the optimal threshold found by find_optimal_threshold.
# Default threshold
y_pred_default = (y_probs >= 0.5).astype(int)
f1_default = f1_score(y_true, y_pred_default)
# Optimal threshold
best_t, best_f1 = find_optimal_threshold(y_true, y_probs)
print(f"Threshold τ=0.500 | F1 = {f1_default:.4f}")
print(f"Threshold τ={best_t:.3f} | F1 = {best_f1:.4f} ← optimal")
print(f"\nImprovement: +{best_f1 - f1_default:.4f} F1 points")
Threshold τ=0.500 | F1 = 1.0000 Threshold τ=0.500 | F1 = 1.0000 ← optimal Improvement: +0.0000 F1 points
Custom Metric — Balanced Class F1¶
When both classes are equally important, we maximise the minimum of the per-class F1 scores $\min(F_1^+, F_1^-)$.
def min_class_f1(y_true, y_pred):
"""Maximise the worst-class F1 score."""
f1_pos = f1_score(y_true, y_pred, pos_label=1)
f1_neg = f1_score(y_true, y_pred, pos_label=0)
return min(f1_pos, f1_neg)
best_t2, best_score2 = find_optimal_threshold(y_true, y_probs, min_class_f1)
default_score2 = min_class_f1(y_true, (y_probs >= 0.5).astype(int))
print(f"Optimal threshold for min-class F1: τ={best_t2:.3f}")
print(f"Min-class F1 at optimal threshold : {best_score2:.4f}")
print(f"Min-class F1 at default threshold : {default_score2:.4f}")
Optimal threshold for min-class F1: τ=0.500 Min-class F1 at optimal threshold : 1.0000 Min-class F1 at default threshold : 1.0000
Using Accuracy as the Metric¶
best_t3, acc3 = find_optimal_threshold(y_true, y_probs, accuracy_score)
print(f"Optimal threshold for accuracy: τ={best_t3:.3f}")
print(f"Accuracy at optimal threshold : {acc3:.4f}")
Optimal threshold for accuracy: τ=0.500 Accuracy at optimal threshold : 1.0000
Works with sorix Tensors¶
The function accepts sorix.Tensor inputs directly, making it trivial to use with model outputs.
import sorix
y_true_tensor = sorix.tensor(y_true.astype(np.float32))
y_probs_tensor = sorix.tensor(y_probs.astype(np.float32))
t_np, f1_np = find_optimal_threshold(y_true, y_probs)
t_sx, f1_sx = find_optimal_threshold(y_true_tensor, y_probs_tensor)
print(f"Result from numpy arrays : τ={t_np:.3f}, F1={f1_np:.4f}")
print(f"Result from sorix Tensors: τ={t_sx:.3f}, F1={f1_sx:.4f}")
Result from numpy arrays : τ=0.500, F1=1.0000 Result from sorix Tensors: τ=0.500, F1=1.0000