research
Prediction Accuracy Is Not the Same as Trading Profitability
Forecast error, directional accuracy, and trading profit measure different things. A useful trading evaluation needs to explain how a prediction becomes an executable decision after costs and risk.
A forecast can be accurate but economically useless
Suppose a model predicts the next price very closely most of the time, but its errors become largest during the few moves that matter most. The average forecast error may look excellent while the resulting trades lose money. Another model may have larger numerical error but consistently identify moves large enough to cover costs. The usefulness of a prediction therefore depends on the decision rule built around it, not only the statistical distance between forecast and outcome.
This is why mean absolute error, signed error, directional accuracy, calibration, and realized strategy return should not be treated as interchangeable scoreboards. Each exposes a different failure mode. Magnitude error asks how far the forecast missed, signed error reveals systematic bias, direction asks whether the sign was right, and trading outcomes include thresholds, sizing, costs, and execution.
Thresholds transform continuous forecasts into actions
Many systems do not trade every prediction. They act only when expected movement is large enough to justify risk and friction. That means a model can improve its average error by getting many small, irrelevant moves right without improving the subset of decisions that actually trigger trades. Evaluation should therefore inspect performance around the action boundary rather than assume that better aggregate forecasting automatically creates better orders.
Threshold behavior also creates asymmetry. A small forecast change can move one observation from no-trade to trade while leaving another unchanged. If the research metric ignores that discontinuity, the model may appear smoothly better while the executed policy changes sharply. Reporting both prediction quality and downstream decision behavior makes that transformation visible.
Costs and sizing can reverse the ranking
Two models with similar forecast quality can produce very different trading outcomes if one trades more often, concentrates risk, or tends to act when spread is wide. The economically relevant comparison therefore needs realistic fees, slippage assumptions, and position rules. A model that generates fewer but more robust opportunities can outperform a more statistically accurate model once implementation friction is included.
Sizing introduces another layer. Forecast confidence is not automatically position size, and poorly calibrated confidence can magnify the worst errors. A research process should test the mapping from prediction to exposure separately from the prediction itself. This helps identify whether a disappointing result comes from forecasting, decision thresholds, or risk allocation.
Keep capability metrics and outcome metrics separate
A good evaluation report can say that a model improved forecast calibration without claiming the trading system improved. It can also say that trading results improved while the evidence is not yet strong enough to attribute the change to better forecasting. Separating these statements prevents a favorable downstream outcome from being used as retroactive proof that every upstream component became smarter.
For self-improving systems, this separation is especially important. Promotion rules should specify whether they are testing prediction capability, decision quality, execution quality, or full-system economic outcomes. When each layer has its own evidence, the system can learn where improvement actually occurred instead of compressing every success into one vague claim that the AI learned to trade.