End-to-End ML Overview¶
Before diving into the individual building blocks, it helps to see the whole picture at once. This page is a map of the entire machine learning workflow: the parts that make up an ML solution and how they connect, from framing a business problem all the way to monitoring a model running in production.
Use it as a reference you can return to. Each stage below links to the deep-dive module that covers it in full.
The big picture¶
Note - Read the diagram as a loop, not a line: Real ML projects are iterative. Results from evaluation and monitoring feed back into data and model work, so the pipeline is a cycle that keeps improving rather than a one-way path that finishes.
The workflow has six working stages, all resting on two foundations (mathematics and neural networks) that show up in every stage.
The stages at a glance¶
| # | Stage | What happens here | Where to learn it |
|---|---|---|---|
| 1 | Problem framing | Turn a business goal into a precise prediction target: the decision, the unit, the label, the success KPI, and the cost of each error. | Introduction and Lifecycle |
| 2 | Data preparation | Collect source data, clean and encode it, identify the right datasets, and split into train and test sets without leakage. | Data Preparation |
| 3 | Model development | Set up the workspace and environment, choose a model family, configure AutoML, and run training and hyperparameter search. | Model Types, Training and AutoML |
| 4 | Evaluation | Compute the right metrics, analyze where the model wins and loses, and explain its predictions. | Performance Metrics, Results and Explainability |
| 5 | Deployment | Register the model and serve it as an online or batch endpoint, including on Kubernetes. | Deployment |
| 6 | Monitoring | Track data and model drift, debug live endpoints, and trigger retraining when quality drops. | Deployment Debugging |
How the parts connect¶
1. Problem framing. Everything starts with a clear question. If you cannot state the decision the prediction will change, the unit you predict on, and how success is measured as a number, the rest of the pipeline has no target to aim at. This is covered in Introduction and Lifecycle.
2. Data preparation. Models are only as good as the data behind them. This stage gathers the source data, cleans and encodes it, identifies which datasets matter, and carves out an honest train and test split. See Data Preparation, with workspace and tooling in Azure ML Environment and Environment Setup.
3. Model development. With clean data ready, you choose an algorithm family, configure the training job (often through AutoML), and search for the best model. The theory behind the choices lives in ML Foundations and Neural Networks and Deep Learning; the practice lives in Model Types and Training and AutoML.
4. Evaluation. A trained model is not done until you have measured it with the right metrics, understood its mistakes, and explained its behavior. See Performance Metrics and Results and Explainability.
5. Deployment. A good model only creates value once it serves predictions. You register the model and expose it as a real-time or batch endpoint, including Kubernetes-based serving. See Deployment.
6. Monitoring. Production is where the work continues. You watch for drift, debug endpoint and infrastructure issues, and retrain when quality slips. This closes the loop back to data and model development. See Deployment Debugging.
The foundations under every stage¶
Two areas appear again and again, no matter which stage you are in:
- Mathematics (Math Prerequisites): probability, linear algebra, calculus, and statistics power every loss function, metric, and optimization step.
- Neural networks and deep learning (Neural Networks and Deep Learning): the dominant model family for images, text, and large-scale problems.
Note - Why an overview first: Knowing where each topic fits in the larger flow makes the detailed modules easier to absorb. When you study data splitting or a specific metric later, you will already know which stage it belongs to and what comes before and after it.