Building Reliable Prediction Systems in Production

Most machine-learning projects die in the gap between a promising notebook and a system people actually rely on. The notebook is where you prove an idea is possible; production is where you prove it is dependable. Those are different problems, and the second one is harder, slower, and far less glamorous than the first. Yet it is the only one that creates business value.

At S.C.G.A. we build prediction systems for Hong Kong clients — from horse-racing odds models for betting syndicates to intraday signal models for local brokerages and churn predictors for membership businesses. The lesson is always the same: the model is the smallest part of the work. The surrounding system — data, monitoring, evaluation, and the discipline to operate it — is what separates a toy from a tool your finance team will stake real money on.

This article is a field guide to that system. It is written for engineers and founders who already know how to train a model and now need to make it survive contact with reality.

From notebook to production: the real handoff

The notebook ends with model.fit() and a confusion matrix you are proud of. Production begins long before that cell runs again. The first thing to decide is whether your “model” is actually a training pipeline or an inference service — because the two have completely different failure modes.

A training pipeline fails silently and slowly. A bad feature join can quietly halve your AUC, and nothing crashes. An inference service fails loudly and immediately. A missing column throws a 500, and a trader is staring at a blank dashboard at 9:32am.

We deliberately split these concerns. The training pipeline is a batch job — often an Airflow DAG or a cron-triggered script on an EC2 box in the ap-east-1 (Hong Kong) region — that produces a versioned model artifact and writes its evaluation report somewhere humans will read it. The inference service is a small FastAPI app behind a load balancer that loads that artifact and answers POST /predict with a p95 latency budget, typically under 80ms for our low-latency signal clients.

A concrete handoff checklist

Before any model leaves the notebook, it must clear these:

The feature code is identical at train and serve time. No reimplementing a rolling mean in SQL for training and in Python for serving. One function, imported by both. This single rule prevents more silent bugs than any other.
Every input has a schema and a validation gate. We use Pydantic models at the API boundary and Great Expectations or simple asserts inside the pipeline. If volume is negative or date is in the future, the request is rejected with a clear error, not silently coerced.
The model is versioned, not overwritten. Each training run writes model_v47.pkl plus a metrics_v47.json. We can roll back to v46 in one config change.
There is a reproducible entrypoint. python train.py --config configs/hk_racing.yaml produces the same artifact given the same data and seed. No “run the third cell twice” tribal knowledge.

Data pipelines: the contract is the product

A prediction system is only as stable as the pipeline that feeds it. We treat the data contract as a first-class deliverable — not an afterthought scribbled on a whiteboard. A data contract states, in machine-enforceable form: the columns, their types, their allowed ranges, their freshness SLA, and who owns the upstream source.

Consider a horse-racing odds model. Its inputs include past performance, track condition, jockey history, and — critically — the closing odds from the tote. Those odds arrive from a third-party feed that, in practice, is occasionally late, occasionally malformed, and occasionally just wrong (a decimal point in the wrong place turns 2.50 into 25.0). If you ingest that feed without a contract, your model trains on garbage for three days before anyone notices. If you ingest it with a contract — odds BETWEEN 1.01 AND 1000, updated_at within 5 minutes of post time — the bad record is quarantined and an alert fires.

Building a pipeline that survives HK operating hours

Hong Kong markets and events run on Hong Kong time, but your data sources might be in Singapore, London, or the US. We always pin pipelines to HKT and log timestamps in UTC internally to avoid the daylight-saving ambiguity that bites cross-border setups. A pipeline that “runs every hour” without a timezone is a pipeline that will surprise you at 3am during a US holiday.

For most of our clients the pipeline looks like this:

Ingest raw feeds into a staging bucket (S3 in ap-east-1, or a local NAS for data-residency-sensitive clients).
Validate against the contract; quarantine failures.
Transform into features with pure, tested functions.
Store features in a feature store or at minimum a versioned Parquet dataset.
Train/evaluate and publish the artifact + report.

The whole thing is idempotent. Re-running it for yesterday does not duplicate or corrupt anything. Idempotency is what lets you sleep when a job fails at 2am — you just re-run it in the morning.

Monitoring for drift, not just errors

Traditional monitoring watches for crashes. Prediction systems need to watch for quiet failure: a slow shift in input distribution that quietly erodes accuracy long before anyone files a ticket. This is data drift and concept drift, and it is the number-one reason models decay.

Data drift is when the inputs change. Imagine your churn model was trained on pre-COVID member behaviour; post-pandemic, members interact differently and the feature distributions shift. Concept drift is when the relationship between inputs and the outcome changes — the model’s assumptions no longer hold even if the inputs look normal. Both are invisible to a “is the server up?” dashboard.

A drift monitoring stack we actually use

Input distribution tracking. For each numeric feature we log the rolling mean and standard deviation and alert when they deviate beyond, say, 3 standard deviations from the training baseline. For categoricals, we track the category-frequency distribution and alert on new or vanishing categories.
Prediction distribution tracking. If your model suddenly starts predicting “default” 40% of the time when it historically predicted 8%, something upstream broke — a null feature, a mis-scaled input, or genuine regime change.
Calibration checks. A well-calibrated model’s predicted probability matches the real frequency. We log a rolling calibration curve and alert on degradation, because for a betting or trading model, confidence is itself a tradable signal.
Ground-truth feedback loops. The hardest part. For a horse-racing model the ground truth (did it win?) is known within minutes. For a churn model the truth might take 90 days. Design the pipeline to ingest the label when it arrives, so you can compute live accuracy, not just hope.

We surface all of this in a Grafana dashboard the client’s ops team can read without a data scientist present. If only the PhD can tell whether the system is healthy, the system is not production-grade.

Evaluation: what “good” actually means

Accuracy is a comforting number and a dangerous one. A model that predicts “not fraud” 99.4% of the time is 99.4% accurate and completely useless if fraud is the 0.6% you care about. Evaluation has to be tied to the decision the model supports.

Metrics by use case

Ranking / odds models (horse racing, recommendations): use AUC, log loss, and top-k precision. For racing specifically, we care whether the model’s implied probabilities beat the market’s — a model that merely mirrors the tote has no edge.
Imbalanced classification (fraud, default, churn): precision, recall, F1, and especially the PR curve and lift at the operating threshold. Show the business the expected true-positive rate at the score cutoff they will actually use.
Regression / forecasting (demand, signal magnitude): MAE, RMSE, and sMAPE, but also directional accuracy — for a trading signal, being right about direction matters more than being exactly right about magnitude.

Backtesting the honest way

For any model touched by time — which is most of them — never evaluate on a random train/test split. Random splits leak future information into the past and inflate your numbers. Use time-ordered splits: train on Jan–Mar, validate on April, test on May. Better still, walk-forward validation that rolls the window forward month by month. We have watched a model look brilliant under random splitting and fall apart under walk-forward — and only walk-forward reflects what deployment will actually experience.

For Hong Kong trading and racing clients we go further: we backtest in event time, replaying real market or race conditions, including the spreads, commissions, and the fact that you cannot always fill at the quoted price. A model that “wins” on clean historical prices but loses after costs is not a model you ship.

MLOps: the discipline that keeps it alive

The teams that succeed treat ML like any other production system: code review, tests, observability, and rollback. The model is the smallest part of the work, as we said, but it is also the part most likely to be updated, so the process around updating it matters more than the model itself.

CI tests your data and features, not just code. A unit test that asserts compute_rolling_mean returns the right value on a fixture is worth more than ten tests of the HTTP layer. We also add a data smoke test that runs the pipeline on a tiny sample and asserts the contract holds.
Promote, don’t overwrite. Staging → canary → production. A canary model serves 5% of traffic for 48 hours while we watch its live metrics against the incumbent before full rollout.
Keep a rollback that takes minutes, not hours. Because artifacts are versioned, reverting is a config flip. We have used this at 11pm when a retrain shipped a model that, for reasons we found the next morning, had learned to predict the majority class. The rollback took four minutes.
Document the model card. What it predicts, what it was trained on, known limitations, and — important for Hong Kong — what personal data it touches and how that interacts with the PDPO (Personal Data Privacy Ordinance). If a model uses personal data of Hong Kong residents, the ordinance governs collection, use, and retention. “We didn’t think about that” is not a defensible position with the Privacy Commissioner.

Deployment patterns: pick the right shape

How you deploy depends on the latency, cost, and update frequency the business needs.

Batch scoring. Run nightly, score all customers, write results to a table the dashboard reads. Cheapest, simplest, fine for “who is likely to churn this month.” No real-time infrastructure required.
Real-time inference (online). A small API that scores on demand. Needed when the decision is per-request — a live odds overlay, a real-time credit check at point of sale. Costs more and demands the latency budget and monitoring we described.
Edge / on-device. Occasionally a client needs inference inside a building with poor connectivity, or wants to keep raw data off external networks for residency reasons. We quantize the model (e.g. ONNX or a distilled variant) and run it locally. Hong Kong’s cross-border data sensitivity — particularly for clients with mainland operations — makes this pattern more common than you would expect.
Shadow mode. Ship the new model in parallel with the old, log both predictions, compare, but serve only the old. This is how we build the confidence (and the backtest evidence) to flip the switch without drama.

Hong Kong-specific realities

Building for Hong Kong adds wrinkles that generic ML tutorials ignore:

Data residency and cross-border flow. A client with operations in both Hong Kong and Shenzhen may face constraints on moving personal data across the border. We design pipelines that can train and infer within a single jurisdiction when needed, and we document the data flow for compliance.
PDPO compliance. Any model touching personal data needs a defined purpose, minimal collection, and a retention policy. Baking this into the data contract — not bolting it on later — is far cheaper.
Business hours and timezone. “Real-time” in Hong Kong means being live and healthy during Hong Kong trading and racing hours, which may overlap with a US night when your on-call engineer is asleep. Monitoring that pages the right person in their waking hours is part of the system, not a nice-to-have.
Cost discipline. Hong Kong commercial rents are famous; cloud bills are less famous but equally real. We right-size — batch where possible, reserve instances for steady training, and only pay for real-time when the use case demands it.

A closing rule of thumb

If you cannot reproduce a prediction from last month, you do not have a system — you have a lucky accident. Everything in this article serves that one principle: make the prediction reproducible, observable, and reversible. Reproducible through versioned data and code. Observable through drift and calibration monitoring. Reversible through versioned artifacts and fast rollback.

Do those three well and the model — the part you enjoyed building — finally becomes something your business can lean on. That is the difference between a notebook that impressed your colleagues and a prediction system that impressed your CFO.

If you are shipping a model into production and want a second set of eyes on the pipeline, monitoring, or evaluation design, that is exactly the kind of system we build at S.C.G.A. The model is the easy part. We help with the rest.