Fundamentals of Machine Learning

Question: can information available this month help predict next month's return, or does a model only fit observations it has already seen?

We will work with twelve invented monthly observations. First, use four observations to calculate an average prediction and fit a straight line. Next, compare that line with more flexible curves on four later observations. Finally, evaluate the selected method on the last four. The sample is deliberately small so you can check the arithmetic; it is not evidence of a profitable forecasting strategy.

You need averages, percentages, and basic Python. The Course 3 overview explains the broader learning sequence. Run this chapter's cells in order. The widget also works independently.

import json
import numpy as np
import pandas as pd
import PortfolioOptimizationKit as pok
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error

1. Define What Is Known and What Is Predicted

A feature is an input available when a prediction is made. A target is the quantity we want to predict. Here, the feature $x_t$ is a hypothetical market return observed by the end of month $t$. The target $y_t$ is a hypothetical asset's return during the following month.

A row labeled January therefore contains January's market return and February's asset return. At the end of January, the feature is known but the target is not. The row becomes a complete training example only after February ends.

Decision month	Feature $x_t$: this month's market return (%)	Target $y_t$: next month's asset return (%)
January	−1	−1
February	0	1
March	1	1
April	2	3

For easy arithmetic, numbers in this chapter are percentage values: 1 means 1%, not a decimal return of 1. An actual return of 3% minus a prediction of 1% is an error of 2 percentage points (pp). Convert to decimals by dividing by 100 before using portfolio helpers that expect decimal returns.

Construct the Teaching Data

We choose the features and noise explicitly; no random draws or seed are needed. The invented relationship is:

$$y_t=0.5+x_t+{\epsilon }_t.$$

The symbol ${\epsilon }_t$ is the part not explained by the feature, called noise. For January, $0.5+(-1)+(-0.5)=-1$. Without noise, its target would be −0.5%; with this chosen noise, it is −1%. We know the relationship because we constructed the example. The fitted models will only receive the observed feature–target pairs, not this formula or the noise values.

decision_months = pd.period_range("2020-01", periods=12, freq="M")
features = np.array([-1, 0, 1, 2, -0.5, 0.5, 1.5, 1, -0.75, 0.25, 1.25, 1.75])
noise = np.array([-0.5, 0.5, -0.5, 0.5, 0.1, -0.1, 0.1, -0.1, -0.15, 0.15, -0.15, 0.15])
toy = pd.DataFrame({
    "Feature (%)": features,
    "Target (%)": 0.5 + features + noise,
    "Target month": decision_months + 1,
}, index=decision_months)
toy.index.name = "Decision month"
train = toy.iloc[:4].copy()
validation = toy.iloc[4:8].copy()
test = toy.iloc[8:].copy()
print(train.to_string(float_format=lambda value: f"{value:.2f}"))

Machine learning here means choosing the parameters of a prediction rule from examples, rather than setting every prediction by hand. Training is the process of choosing those parameters; prediction applies the fitted rule to an input.

This is supervised learning because training rows include known targets. Predicting a number, such as a return, is regression. Predicting a category, such as whether a return is negative, is classification. In unsupervised learning, there is no supplied target: grouping assets by similarity is one example. We begin with regression so predictions and errors are easy to calculate.

2. Start with a Baseline Prediction

Before using the feature, ask how well a simple rule performs. A baseline is a reference method against which to compare a more elaborate one.

Use the mean of the four training targets for every prediction:

$${\overline{y}}_{\mathrm{train}}=\frac{-1+1+1+3}{4}=1.$$

This rule predicts 1% regardless of the feature. A prediction is written $\hat{y}$, read “y hat.” Define the error as actual minus predicted: $e_i=y_i-{\hat{y}}_i$. Positive error means the prediction was too low.

Target (%)	Prediction (%)	Error (pp)	Squared error (pp²)
−1	1	−2	4
1	1	0	0
1	1	0	0
3	1	2	4

The errors sum to zero, although two predictions miss by two percentage points. Squaring prevents opposite signs from canceling. The mean squared error, or MSE, averages these squared errors:

$$\mathrm{MSE}=\frac{1}{n}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2=\frac{4+0+0+4}{4}=2{\mathrm{pp}}^2.$$

Here $n=4$ is the number of observations. The summation symbol means “add the squared error for each observation.” Squaring also gives larger errors more weight: a 2-pp error contributes four times as much as a 1-pp error. MSE is in squared units; it is not a percentage return or a risk-adjusted performance measure.

The chart below compares the same targets and predictions. Its horizontal axis shows decision months; each bar refers to the following month's asset return.

baseline_mean = train["Target (%)"].mean()
baseline_predictions = np.full(len(train), baseline_mean)
baseline_errors = train["Target (%)"].to_numpy() - baseline_predictions
baseline_mse = np.mean(baseline_errors ** 2)
baseline_table = pd.DataFrame({
    "Actual (%)": train["Target (%)"], "Predicted (%)": baseline_predictions,
    "Error (pp)": baseline_errors, "Squared error (pp²)": baseline_errors ** 2,
})
print(baseline_table.to_string(float_format=lambda value: f"{value:.3f}"))
print(f"Training MSE: {baseline_mse:.4f} pp²")
print("\n<ECHARTS_DATA>" + json.dumps({
    "mlBaseline": {
        "type": "bar", "title": "The Training Mean Predicts 1% for Every Row",
        "xAxis": {"type": "category", "name": "Decision month", "data": train.index.astype(str).tolist()},
        "yAxisName": "Next-month asset return (%)",
        "series": {"Actual": train["Target (%)"].tolist(), "Training mean": baseline_predictions.tolist()},
    },
}, allow_nan=False))

The January and April bars show the two misses. The middle two happen to match. A sensible baseline need not be accurate on every row; it gives us a defined comparison using information from training only.

Check your understanding: if the first prediction were −0.5% instead of 1%, what would its squared error be? The error would be $-1-(-0.5)=-0.5$ pp, giving $(-0.5{)}^2=0.25$ pp².

3. Let the Prediction Depend on the Feature

A straight-line model predicts:

$$\hat{y}=a+bx.$$

The intercept $a$ is the prediction when the feature is zero. The slope $b$ is the change in the predicted target for a one-unit change in the feature. Here both returns use percentage values, so increasing the feature by one percentage point changes the prediction by $b$ percentage points.

We choose $a$ and $b$ to minimize training MSE. This is ordinary least squares: fitting the line whose squared vertical errors have the smallest sum. For one varying feature, its solution is:

$$b=\frac{\sum _i(x_i-\overline{x})(y_i-\overline{y})}{\sum _i(x_i-\overline{x}{)}^2},a=\overline{y}-b\overline{x}.$$

The bars over $x$ and $y$ mean training averages. Our averages are $\overline{x}=0.5$ and $\overline{y}=1$. The centered calculations are:

$x_i-0.5$	$y_i-1$	Product	Squared feature difference
−1.5	−2	3	2.25
−0.5	0	0	0.25
0.5	0	0	0.25
1.5	2	3	2.25
Sum		6	5

Therefore $b=6/5=1.2$ and $a=1-1.2(0.5)=0.4$. The fitted rule is $\hat{y}=0.4+1.2x$. For a feature of 1.5%, it predicts $0.4+1.2(1.5)=2.2\mathrm{\%}$.

At the four training features, predictions are −0.8%, 0.4%, 1.6%, and 2.8%. Errors are −0.2, 0.6, −0.6, and 0.2 pp, so MSE is $(0.04+0.36+0.36+0.04)/4=0.2$ pp².

The graph places the feature on the horizontal axis and the target on the vertical axis. A point's vertical distance from the fitted line is its prediction error. This horizontal axis is not time; the table's dates determine which observations are eligible for training.

x_train = train["Feature (%)"].to_numpy()
y_train = train["Target (%)"].to_numpy()
slope = np.sum((x_train - x_train.mean()) * (y_train - y_train.mean())) / np.sum((x_train - x_train.mean()) ** 2)
intercept = y_train.mean() - slope * x_train.mean()
line_train_predictions = intercept + slope * x_train
line_train_mse = np.mean((y_train - line_train_predictions) ** 2)
line_grid = np.linspace(-1, 2, 61)
print(f"Intercept: {intercept:.3f}; slope: {slope:.3f}")
print(f"Prediction at x = 1.5%: {intercept + slope * 1.5:.3f}%")
print(f"Training MSE: {line_train_mse:.4f} pp²")
print("\n<ECHARTS_DATA>" + json.dumps({
    "mlFittedLine": {
        "type": "line", "title": "A Fitted Line and the Constant Baseline",
        "xAxis": {"type": "value", "name": "Feature (%)"},
        "yAxisName": "Next-month asset return (%)",
        "series": {
            "Training observations": np.column_stack([x_train, y_train]).tolist(),
            "Fitted line": np.column_stack([line_grid, intercept + slope * line_grid]).tolist(),
            "Training mean": [[-1, float(baseline_mean)], [2, float(baseline_mean)]],
        },
        "seriesOptions": {
            "Training observations": {"type": "scatter"},
            "Training mean": {"lineStyle": {"type": "dashed"}},
        },
    },
}, allow_nan=False))

The line reduces training MSE from 2 to 0.2 pp². That establishes a better fit to these four observations. It does not yet establish better prediction on new observations. Its fitted coefficients also differ from the relationship used to generate the data: noise affects the estimates.

Reproduce the Calculation with scikit-learn

LinearRegression performs the same least-squares calculation. fit estimates the coefficients; predict applies the fitted model. scikit-learn expects a two-dimensional feature table: rows are observations and columns are features. train[["Feature (%)"]] keeps that table shape, while the target is a one-dimensional vector.

line_model = LinearRegression()
line_model.fit(train[["Feature (%)"]], train["Target (%)"])
library_predictions = line_model.predict(train[["Feature (%)"]])
np.testing.assert_allclose(library_predictions, line_train_predictions, atol=1e-12)
print(f"Library intercept: {line_model.intercept_:.3f}")
print(f"Library slope: {line_model.coef_[0]:.3f}")
print(f"Library training MSE: {mean_squared_error(y_train, library_predictions):.4f} pp²")

The library gives the same intercept, slope, and errors. The code automates a calculation you can already check.

4. Evaluate on Later Observations

Set the split before comparing models:

Block	Decision months	Target months	Use
Training	January–April 2020	February–May 2020	Estimate coefficients
Validation	May–August 2020	June–September 2020	Choose model complexity
Final test	September–December 2020	October 2020–January 2021	Evaluate the selected procedure

Validation uses observations excluded from coefficient fitting to compare modeling choices. The test block is reserved until those choices are fixed. The data are visible for teaching, but the selection code below uses only training and validation rows.

Assume returns become available at month-end, before the next prediction. We can fit the first model at the end of May, when all four training targets are known, and predict June's return using May's feature. Freeze its coefficients throughout validation. After September ends, the validation targets are known and the October forecast can use a refitted model. Actual financial releases may have delays or revisions; their availability dates must replace this simplified assumption.

Keep the original baseline at 1% during validation. Recalculating it from validation targets would give it information unavailable at the original fit date.

For May's feature of −0.5%, the line predicts $0.4+1.2(-0.5)=-0.2\mathrm{\%}$. Its June target is 0.1%, giving an error of 0.3 pp. Across the validation block, line errors are 0.3, −0.1, −0.1, and −0.2 pp: MSE is 0.0375 pp², compared with 0.5475 pp² for the frozen baseline.

validation_predictions = line_model.predict(validation[["Feature (%)"]])
validation_baseline = np.full(len(validation), baseline_mean)
validation_table = pd.DataFrame({
    "Target month": validation["Target month"],
    "Actual (%)": validation["Target (%)"],
    "Line (%)": validation_predictions,
    "Baseline (%)": validation_baseline,
    "Line error (pp)": validation["Target (%)"].to_numpy() - validation_predictions,
}, index=validation.index)
print(validation_table.to_string(float_format=lambda value: f"{value:.3f}"))
print(f"Line validation MSE: {mean_squared_error(validation['Target (%)'], validation_predictions):.4f} pp²")
print(f"Baseline validation MSE: {mean_squared_error(validation['Target (%)'], validation_baseline):.4f} pp²")
print("\n<ECHARTS_DATA>" + json.dumps({
    "mlValidation": {
        "type": "bar", "title": "Predictions on the Validation Months",
        "xAxis": {"type": "category", "name": "Decision month", "data": validation.index.astype(str).tolist()},
        "yAxisName": "Next-month asset return (%)",
        "series": {"Actual": validation["Target (%)"].tolist(), "Fitted line": validation_predictions.tolist(),
                   "Training mean": validation_baseline.tolist()},
    },
}, allow_nan=False))

The fitted line is closer to the targets than the baseline in this block. Validation error also happens to be lower than training error because we chose smaller noise for these later observations. There is no rule that validation error must exceed training error; the relevant comparison is between models evaluated on the same block.

5. More Flexibility Can Fit the Noise

Could a curved prediction rule fit the training points better? A polynomial adds powers of the feature:

$$\hat{y}=a+b_{1}x+b_2{x}^2+b_3{x}^3.$$

The degree is the highest included power. Degree 1 is a line; degree 2 adds $x^2$; degree 3 also adds $x^3$. At $x=1.5$, these input columns are $1.5$, $2.25$, and $3.375$. We still fit coefficients by least squares. There are more coefficients, so the model can express more shapes.

For these four distinct training features, a cubic has enough coefficients to pass through all four points. Its fitted rule is:

$$\hat{y}=1+\frac{x}{3}-x^2+\frac{2x^3}{3}.$$

At $x=-0.5$, it predicts $1-1/6-1/4-1/12=0.5\mathrm{\%}$. The validation target is 0.1%, so its squared error is $(0.1-0.5{)}^2=0.16$ pp². The line's squared error on that same row was 0.09 pp². Passing exactly through the training points did not make the cubic better on this new point.

Overfitting occurs when a model learns sample-specific variation that does not generalize well to new observations. Underfitting occurs when a model is too restrictive to represent a useful relationship. Our constant baseline ignores the feature; the cubic follows the four training points exactly. The validation comparison helps distinguish these choices.

PolynomialFeatures constructs the powers; include_bias=False leaves the intercept to LinearRegression. make_pipeline joins those steps so fitting and prediction apply the same transformation. This transformation does not estimate a mean or scale, but any learned preprocessing introduced later must also be fitted on training rows only.

candidate_models = {0: None}  # Degree 0 is the constant training-mean baseline.
score_rows = [{"Degree": 0, "Training": baseline_mse,
               "Validation": mean_squared_error(validation["Target (%)"], validation_baseline)}]
for degree in (1, 2, 3):
    candidate = make_pipeline(PolynomialFeatures(degree, include_bias=False), LinearRegression())
    candidate.fit(train[["Feature (%)"]], train["Target (%)"])
    candidate_models[degree] = candidate
    score_rows.append({
        "Degree": degree,
        "Training": mean_squared_error(train["Target (%)"], candidate.predict(train[["Feature (%)"]])),
        "Validation": mean_squared_error(validation["Target (%)"], candidate.predict(validation[["Feature (%)"]])),
    })
scores = pd.DataFrame(score_rows).set_index("Degree")
best_validation = scores["Validation"].min()
# Prefer the simpler model when validation errors tie to numerical precision.
selected_degree = int(scores.index[np.isclose(scores["Validation"], best_validation, rtol=0, atol=1e-12)][0])
print("Mean squared errors (pp²):")
print(scores.to_string(float_format=lambda value: f"{value:.4f}"))
print(f"Selected degree: {selected_degree}")
print("\n<ECHARTS_DATA>" + json.dumps({
    "mlModelSelection": {
        "type": "bar", "title": "Choose Complexity Using Validation Error",
        "xAxis": {"type": "category", "name": "Polynomial degree", "data": scores.index.astype(str).tolist()},
        "yAxisName": "Mean squared error (pp²)",
        "series": {name: scores[name].tolist() for name in scores},
    },
}, allow_nan=False))

The line has training/validation MSE of 0.2000/0.0375 pp². The cubic has approximately 0/0.1725 pp². Select the line on validation error. Degree 2 ties with the line here because its fitted quadratic coefficient is zero up to rounding; added flexibility need not change the fitted shape.

The chart compares errors within each block. The near-zero training bar for degree 3 is not the goal: the validation bar is higher than for degree 1. Choosing degree is a hyperparameter choice—a setting decided outside the coefficient-fitting step. The fitted intercept and slope are parameters learned within that step.

Explore the Same Comparison

Predict what will happen when you move from degree 1 to degree 3. Then use the slider. Circles are training observations; diamonds are validation observations. The dashed line is the unchanged baseline. The second chart reports errors for the selected curve and that baseline. The helper repeats the same fitting calculation on the same first eight observations; test observations are excluded.

import ipywidgets as widgets
import PortfolioOptimizationKit as pok

widget_instance = widgets.interactive(
    pok.show_ml_complexity_echart,
    degree=widgets.IntSlider(value=1, min=1, max=3, step=1, description="Degree", continuous_update=False),
)
widget_instance

Try it: compare degrees 1, 2, and 3. Explain why degrees 1 and 2 overlap, and why the cubic's perfect training fit is insufficient reason to choose it. The plotted range is restricted to the observed feature range; behavior outside it is extrapolation, for which this exercise supplies no validation evidence.

6. Refit, Then Open the Final Test Block

Once the degree is selected, refit that model using the eight training and validation pairs. This uses more available observations to estimate its coefficients; it does not change the selected degree. Update the baseline using those same eight targets: their mean is $8.5/8=1.0625\mathrm{\%}$.

Keep both refitted rules fixed throughout the last four decision months. The selected line becomes approximately $\hat{y}=0.425974+1.131602x$. For September's feature of −0.75%, the prediction is about −0.4227%, compared with an October target of −0.4%.

The chart uses target months this time, so October 2020 corresponds to the forecast made at the end of September. It compares actual outcomes with both frozen prediction rules, not cumulative portfolio wealth.

development = pd.concat([train, validation])
final_baseline_mean = development["Target (%)"].mean()
test_baseline = np.full(len(test), final_baseline_mean)
if selected_degree == 0:
    final_model = None
    test_predictions = test_baseline.copy()
else:
    final_model = make_pipeline(PolynomialFeatures(selected_degree, include_bias=False), LinearRegression())
    final_model.fit(development[["Feature (%)"]], development["Target (%)"])
    test_predictions = final_model.predict(test[["Feature (%)"]])
test_results = pd.DataFrame({
    "Decision month": test.index.astype(str),
    "Actual (%)": test["Target (%)"].to_numpy(),
    "Selected model (%)": test_predictions,
    "Baseline (%)": test_baseline,
    "Model error (pp)": test["Target (%)"].to_numpy() - test_predictions,
}, index=pd.PeriodIndex(test["Target month"]))
test_results.index.name = "Target month"
test_mse = mean_squared_error(test["Target (%)"], test_predictions)
test_baseline_mse = mean_squared_error(test["Target (%)"], test_baseline)
print(test_results.to_string(float_format=lambda value: f"{value:.4f}"))
print(f"Selected-model test MSE: {test_mse:.6f} pp²")
print(f"Baseline test MSE: {test_baseline_mse:.6f} pp²")
print("\n<ECHARTS_DATA>" + json.dumps({
    "mlFinalTest": {
        "type": "bar", "title": "Final Test: Predictions Fixed Before Observing Outcomes",
        "xAxis": {"type": "category", "name": "Target month", "data": test_results.index.astype(str).tolist()},
        "yAxisName": "Asset return (%)",
        "series": {"Actual": test_results["Actual (%)"].tolist(),
                   "Selected model": test_predictions.tolist(), "Development mean": test_baseline.tolist()},
    },
}, allow_nan=False))

Test MSE is approximately 0.023728 pp² for the selected model and 1.060781 pp² for the baseline. These numbers describe four invented outcomes, not the reliability of a market forecast. In a larger study, examine performance across multiple chronological validation windows and retain a final evaluation period.

Do not try another degree because you dislike its test result and still call the same block an untouched test. Once used to choose a model, those observations have become part of model development. Likewise, random shuffling can allow later observations into a model used to evaluate earlier predictions. Forecast horizons with overlapping targets may require gaps between blocks.

7. Check Your Understanding

Using the original four-row line, a new feature is 0.5% and the eventual target is 0.7%.

1. Calculate the line's prediction, error, and squared error.
2. Compare them with the original 1% baseline.
3. Is this one observation enough to declare either method better?

Worked answer

The line predicts $0.4+1.2(0.5)=1\mathrm{\%}$. Its error is $0.7-1=-0.3$ pp and its squared error is $0.09$ pp². The baseline also predicts 1%, so both tie on this observation. A single observation cannot establish general forecasting performance; compare a defined set of later observations using the same metric and information rules.

exercise_feature, exercise_actual = 0.5, 0.7
exercise_prediction = intercept + slope * exercise_feature
exercise_error = exercise_actual - exercise_prediction
print(f"Prediction: {exercise_prediction:.2f}%")
print(f"Error: {exercise_error:.2f} pp; squared error: {exercise_error ** 2:.4f} pp²")

Chapter Summary

• A feature must be available when the prediction is made; a training target must already be observed when fitting begins.
• A baseline gives a defined comparison. MSE measures squared prediction errors, not portfolio performance.
• Fitting estimates parameters. Validation chooses modeling settings; the final test evaluates the selected procedure.
• More flexibility can reduce training error while worsening later predictions. A perfect historical fit is insufficient evidence of a useful forecast.
• A financial strategy also needs an allocation rule, constraints, trading timing, and costs. A lower prediction error alone does not establish higher net returns.

Next in the Course 3 learning sequence is regularized factor estimation. It will connect these fitting and validation ideas to the exposures studied in Chapter 2.1.