Walk-Forward Split¶
The Problem with Standard Cross-Validation on Temporal Data¶
In standard $k$-fold cross-validation, the dataset is randomly shuffled before splitting. When data has a temporal order — financial time series, sensor readings, user event logs — this shuffling introduces future data leakage: a validation fold may contain samples from the past, and the corresponding training fold may include samples from the future. The model therefore trains on information it could not have had at prediction time, producing optimistic validation scores that do not generalise.
Walk-Forward (Chronological) Validation¶
WalkForwardSplit enforces the constraint that training data always precedes validation data in time:
$$ \forall (x_i, t_i) \in \mathcal{D}_{\text{train}},\; \forall (x_j, t_j) \in \mathcal{D}_{\text{val}}: t_i < t_j $$
where $t_i$ is the timestamp (or ordering index) of sample $i$.
Two Window Modes¶
Expanding Window (default)¶
The training set grows with each fold — all data prior to the current validation window is used:
$$ \mathcal{D}_{\text{train}}^{(k)} = \{x_1, x_2, \ldots, x_{s_k - g - 1}\} $$
$$ \mathcal{D}_{\text{val}}^{(k)} = \{x_{s_k}, x_{s_k+1}, \ldots, x_{s_k + v - 1}\} $$
where $s_k$ is the start of the $k$-th validation window, $v$ is val_size, and $g$ is the gap.
Fold 1: [==train===========][gap][ val ]
Fold 2: [==train==================][gap][ val ]
Fold 3: [==train======================][gap][ val ]
When to use: when more historical data is beneficial (e.g., stationary distributions, long-term seasonality).
Rolling (Fixed-Size) Window¶
The training set has a fixed size train_size, sliding forward with each fold:
$$ \mathcal{D}_{\text{train}}^{(k)} = \{x_{s_k - g - w}, \ldots, x_{s_k - g - 1}\} $$
where $w$ is train_size.
Fold 1: [=train=][gap][ val ]
Fold 2: [=train=][gap][ val ]
Fold 3: [=train=][gap][ val ]
When to use: when recent data is more informative than old data (concept drift, regime changes).
The Gap Parameter¶
The gap parameter drops $g$ samples between the end of the training window and the start of the validation window. This simulates a prediction lag — e.g., if stock prices are published at market close and predictions must be made one trading day in advance, we set gap=1 to ensure the model never trains on information unavailable at prediction time.
API¶
sorix.utils.data.WalkForwardSplit(
n_splits=5,
train_size=None, # rolling-window size; passing it selects rolling mode
val_size=None, # None = auto (len(X) // (n_splits+1))
gap=0,
expanding=None, # None = infer from train_size
)
Calling .split(X, y) yields (train_X, train_y, val_X, val_y) tuples. If y is omitted, yields (train_X, val_X).
Two things to keep in mind:
train_sizeimplies a rolling window. It has no meaning for an expanding window, so supplying it switches modes automatically; combining it with an explicitexpanding=TrueraisesValueErrorinstead of being silently ignored.- The fold count is guaranteed.
split()always yields exactlyn_splitsfolds. If the data cannot hold them — it needsn_splits * val_size + gap + 1samples — it raisesValueErrorrather than quietly returning fewer folds.
Unlike scikit-learn's TimeSeriesSplit, split() yields slices of the data rather than index arrays. For a pandas object, pass df.to_numpy().
# Uncomment the next line and run this cell to install sorix
#!pip install 'sorix @ git+https://github.com/Mitchell-Mirano/sorix.git@develop'
import numpy as np
from sorix.utils.data import WalkForwardSplit
Verifying Temporal Ordering¶
We create a dataset where the feature is simply the sample index, making it easy to verify that training data always precedes validation data.
n = 160
X = np.arange(n).reshape(n, 1).astype(float)
y = np.arange(n).astype(float)
splitter = WalkForwardSplit(n_splits=5, val_size=20)
for k, (train_X, train_y, val_X, val_y) in enumerate(splitter.split(X, y), 1):
t_start, t_end = int(train_X[0, 0]), int(train_X[-1, 0])
v_start, v_end = int(val_X[0, 0]), int(val_X[-1, 0])
no_overlap = len(set(range(t_start, t_end+1)) & set(range(v_start, v_end+1))) == 0
train_before_val = t_end < v_start
print(f"Fold {k} | train: [{t_start}–{t_end}] val: [{v_start}–{v_end}]"
f"| no overlap: {no_overlap} | train < val: {train_before_val}")
Fold 1 | train: [0–59] val: [60–79]| no overlap: True | train < val: True Fold 2 | train: [0–79] val: [80–99]| no overlap: True | train < val: True Fold 3 | train: [0–99] val: [100–119]| no overlap: True | train < val: True Fold 4 | train: [0–119] val: [120–139]| no overlap: True | train < val: True Fold 5 | train: [0–139] val: [140–159]| no overlap: True | train < val: True
Expanding vs Rolling Window¶
We compare how the training set size evolves across folds for both modes.
X = np.arange(160).reshape(160, 1).astype(float)
y = np.arange(160).astype(float)
expanding = WalkForwardSplit(n_splits=5, val_size=20, expanding=True)
rolling = WalkForwardSplit(n_splits=5, val_size=20, train_size=50, expanding=False)
print("EXPANDING WINDOW:")
for k, (tx, ty, vx, vy) in enumerate(expanding.split(X, y), 1):
print(f" Fold {k} | train_size={len(tx):<5} val_size={len(vx)}")
print("\nROLLING WINDOW (train_size=50):")
for k, (tx, ty, vx, vy) in enumerate(rolling.split(X, y), 1):
print(f" Fold {k} | train_size={len(tx):<5} val_size={len(vx)}")
EXPANDING WINDOW: Fold 1 | train_size=60 val_size=20 Fold 2 | train_size=80 val_size=20 Fold 3 | train_size=100 val_size=20 Fold 4 | train_size=120 val_size=20 Fold 5 | train_size=140 val_size=20 ROLLING WINDOW (train_size=50): Fold 1 | train_size=50 val_size=20 Fold 2 | train_size=50 val_size=20 Fold 3 | train_size=50 val_size=20 Fold 4 | train_size=50 val_size=20 Fold 5 | train_size=50 val_size=20
Gap — Simulating a Prediction Lag¶
Setting gap > 0 excludes samples between the end of the training window and the start of validation. This is essential when there is a natural lag between the end of the observation window and the time the prediction is needed.
X = np.arange(160).reshape(160, 1).astype(float)
y = np.arange(160).astype(float)
for gap in [0, 5, 10]:
spl = WalkForwardSplit(n_splits=5, val_size=20, gap=gap)
tx, ty, vx, vy = next(iter(spl.split(X, y)))
t_end = int(tx[-1, 0])
v_start = int(vx[0, 0])
print(f"gap={gap:<2} | Fold 1: train ends at {t_end}, val starts at {v_start}, gap between = {v_start - t_end - 1}")
gap=0 | Fold 1: train ends at 59, val starts at 60, gap between = 0 gap=5 | Fold 1: train ends at 54, val starts at 60, gap between = 5 gap=10 | Fold 1: train ends at 49, val starts at 60, gap between = 10
Practical Usage in a Training Loop¶
The splitter integrates directly with sorix's training workflow. Arrays are returned as plain numpy slices — no index gymnastics required.
import sorix
from sorix import tensor
sorix.manual_seed(0)
# Synthetic linear regression dataset (N=550 temporal samples)
N = 550
X_data = np.random.randn(N, 8).astype(np.float32)
y_data = (X_data @ np.random.randn(8).astype(np.float32) + 0.1 * np.random.randn(N).astype(np.float32))
splitter = WalkForwardSplit(n_splits=5, val_size=50)
val_losses = []
for k, (train_X, train_y, val_X, val_y) in enumerate(splitter.split(X_data, y_data), 1):
# Build and train a small MLP on this fold
model = sorix.nn.Linear(8, 1, bias=True)
opt = sorix.optim.Adam(model.parameters(), lr=1e-2)
X_tr = tensor(train_X)
y_tr = tensor(train_y.reshape(-1, 1))
for _ in range(50):
opt.zero_grad()
pred = model(X_tr)
loss = ((pred - y_tr) ** 2).mean()
loss.backward()
opt.step()
# Evaluate on validation fold
with sorix.no_grad():
X_val = tensor(val_X)
y_val = tensor(val_y.reshape(-1, 1))
val_loss = float(((model(X_val) - y_val) ** 2).mean().data)
val_losses.append(val_loss)
print(f"Fold {k} | train={len(train_X)} | val={len(val_X)} | val_loss={val_loss:.4f}")
print(f"\nMean val loss: {np.mean(val_losses):.4f}")
Fold 1 | train=300 | val=50 | val_loss=1.4473 Fold 2 | train=350 | val=50 | val_loss=0.6709 Fold 3 | train=400 | val=50 | val_loss=2.0869 Fold 4 | train=450 | val=50 | val_loss=3.5512
Fold 5 | train=500 | val=50 | val_loss=0.4483 Mean val loss: 1.6409
Guardrails¶
Silently returning fewer folds than requested is a common way to get misleading cross-validation results, so the splitter refuses configurations that do not fit.
X = np.arange(100).reshape(100, 1).astype(float)
y = np.arange(100).astype(float)
# 5 folds x 50 validation samples need 251 samples, but we only have 100
try:
list(WalkForwardSplit(n_splits=5, val_size=50).split(X, y))
except ValueError as e:
print(f"ValueError: {e}")
# train_size alone means "rolling", so expanding=True is a contradiction
try:
WalkForwardSplit(n_splits=5, train_size=20, expanding=True)
except ValueError as e:
print(f"\nValueError: {e}")
# Passing train_size selects rolling mode automatically
spl = WalkForwardSplit(n_splits=5, train_size=20, val_size=10)
print(f"\nexpanding inferred as: {spl.expanding}")
print(f"train sizes: {[len(tx) for tx, _, _, _ in spl.split(X, y)]}")
ValueError: Not enough samples for 5 chronological splits: len(X)=100 but 251 are required (n_splits * val_size + gap + 1 = 5 * 50 + 0 + 1). Reduce n_splits, val_size or gap. ValueError: train_size only applies to a rolling window, but expanding=True was requested. Pass expanding=False for a fixed-size rolling window, or drop train_size to use an expanding window. expanding inferred as: False train sizes: [20, 20, 20, 20, 20]