Exam program: Machine learning and MLOps
IAE-EXAM-ML · version 1.0
Sections, format, sample tasks and scoring rubric of the IAE written exam in the field "Machine learning and MLOps".
01. Who is the exam for?
For ML engineers, data scientists who train and deploy models, MLOps engineers, applied machine learning researchers in companies and heads of ML teams. Admission requires a bachelor's degree or higher and at least 3 years of experience in the field (without a degree, at least 5 years). The exam tests calculations and decisions based on data and model results: problem framing and data splits, model choice and training, quality metrics, the decision threshold and model comparison, deployment, monitoring and retraining. Knowledge of a particular library or platform is not tested: code in answers may be written in any widely used language or in pseudocode, and the thresholds, compute prices and latency requirements needed for the solution are given in the task. System design and code quality are covered by the "Software and data" program, business metrics and product A/B tests by the "Data and business analysis" program, and AI system inventories, personal data and human oversight by the "AI governance and data protection" program.
02. Format
- 5 questions, one per section, 10 minutes each.
- 2 cases of 25 minutes from two different sections: problem framing and data, models and training, model evaluation and experiments, deployment and operations. The cases require calculations with the candidate's own data: quality metrics and the cost of errors at different thresholds, sample sizes for splits by time and by group, the compute cost of hyperparameter search, the population stability index (PSI) and the shift in the shares of a feature, response latency and the number of servers needed to serve a model.
- 1 question about the project from the candidate's questionnaire, 10 minutes.
- Up to 2 hours in total, in one sitting. Pasting is disabled, and each candidate gets their own case data.
03. Sections and weights
- Problem framing and data - 20 %: the target variable and its link to the business decision, the unit and horizon of the prediction, data leakage through features and through the split, splits by time and by group, label quality and agreement between annotators, class imbalance, class weights and sampling.
- Models and training - 20 %: choosing a model family for the volume and type of data and for latency requirements, a baseline model for comparison, regularization and overfitting, learning curves, hyperparameter search and the compute budget, early stopping, interpretability as a requirement of the task and explanations of individual predictions.
- Model evaluation and experiments - 20 %: the confusion matrix, precision (the share of correct cases among the model's positive predictions), recall (the share of all positive cases that the model finds), F1 (the harmonic mean of precision and recall), the area under the ROC curve (ROC-AUC) and under the precision-recall curve (PR-AUC) with imbalanced classes, calibration (agreement between the model's scores and the observed share of positive cases), choosing the threshold by the cost of errors, offline evaluation and testing the model on live traffic (shadow mode, a share of traffic), comparing two models on one sample and the error margin of the estimate.
- MLOps: deployment and operations - 20 %: versions of data, code and models, reproducible training, training pipelines and checks before deployment, latency and throughput in serving, batch and online predictions, monitoring of data drift and quality when labels arrive late, the population stability index (PSI), the decision to retrain or roll back.
- Professional practice - 20 %: model documentation (purpose, data, metrics, limitations), measuring quality differences between groups as a technical task, explaining results and uncertainty to the client, the limits of the model's use, handing the model over to operations, honest reporting, the candidate's own project.
04. Sample tasks
Question. A model selects payments for a fraud check. The validation sample has 20,000 payments, and 400 of them are fraudulent. At a threshold of 0.5 the model flagged 1,000 payments, 320 of them fraudulent. At a threshold of 0.3 the model flagged 2,400 payments, 372 of them fraudulent. A false positive (checking an honest payment and the delay for the customer) costs the company 6, and a missed fraudulent payment costs 180. Build the confusion matrix for each threshold, calculate precision and recall, choose the threshold by the total cost of errors and explain why the share of correct answers (accuracy) does not work for this choice.
Case. A bank issues loans through a partner's website. Version 2 of the credit scoring model went live 6 weeks ago in place of version 1, and version 1 now runs in shadow mode: it scores the same applications but makes no decisions. During serving, version 2 requests the feature "partner-reported income" from the partner. On the validation sample before deployment, the ROC-AUC for first payment default was 0.79 for version 2 and 0.76 for version 1. On loans issued after deployment whose first payment date has passed, the ROC-AUC is 0.71 for version 2 and 0.75 for version 1. The shares of applications by interval of the feature (income in thousands a year: below 20, 20-40, 40-80, 80 and above) were 15, 35, 35 and 15 % in the training sample and 41, 33, 19 and 7 % over the last 4 weeks. PSI is the sum over the intervals of (a minus e) x ln(a / e), where a and e are the shares of the interval now and in training. The PSI of the other 23 features is no higher than 0.04, and under the team's rules a PSI of 0.25 or more means a strong shift. The partner cancels an application if no response arrives within 400 ms: version 2 loses 2.1 % of applications this way, while version 1 lost 0.3 %. The bank receives 50,000 applications a month and approves 40 % of them. An approved loan brings the bank 120 on average. Calculate the PSI of the feature and compare it with the threshold. Estimate the monthly loss from cancelled applications under version 2 compared with version 1. Decide whether to roll back to version 1, retrain version 2 on the data of the last weeks or keep it, and justify the decision. Propose checks in the data and serving pipelines that would find such a shift before quality drops.
The figures and circumstances of the cases are different for each candidate.
05. Scoring rubric
Each task is scored on four criteria, and the total is converted to 100 points taking the section weights into account.
- Correctness of the solution - 40 %.
- Reasoning behind decisions - 25 %.
- Risks and constraints - 20 %.
- Clarity of the answer - 15 %: structure and precise wording. Grammar and style do not reduce the score if the meaning is clear.
06. Result
- Pass mark - 70 of 100.
- Result in 5 business days. Those who pass receive an IAE Certified Professional certificate in the field "Machine learning and MLOps", valid for 3 years; scores by section are in a separate report.
- The certificate confirms that the candidate passed the written exam under this program. It is not a license or an accreditation and does not replace the certificates of cloud providers or of online machine learning courses. Employers and authorities make hiring and other decisions about the candidate under their own rules.
- Retake 30 days after the first failed attempt and 60 days after the second, with no more than 3 attempts in 12 months.
- Full terms: the Exam terms page.
- The Credentialing Council reviews the program at least once a year.
07. What is not covered?
- The IAE exam is not a machine learning exam of a cloud provider or of an online course, is not affiliated with them and does not replace their exams. The topics and tasks of the IAE exam do not repeat the syllabi of those exams. The names of certifications and organizations belong to their owners.
- Work with particular machine learning libraries, frameworks and platforms, their functions and parameters. Code in answers may be written in any widely used language or in pseudocode, and calculations are done by hand or on a calculator.
- Software system design, code quality, databases and SQL: this is the "Software and data" program.
- Business metrics, business analysis and A/B tests of product changes: this is the "Data and business analysis" program.
- AI system inventories, assessment before launch from a governance point of view, legal requirements for personal data and human oversight: this is the "AI governance and data protection" program. In this exam, quality differences between groups are measured as a technical task, without a legal assessment.
- Cloud infrastructure, containers and software delivery pipelines: this is the "Cloud and DevOps" program.
- Research and training of foundation models from scratch, including the pretraining of large language models and image models.
08. Preparation
- The practice exam runs in the same interface: 2 questions from different sections, a 15-minute case and a project question.
- Review the calculations: the sizes of the training, validation and test samples for splits by time and by group, the share of the positive class and class weights; the number of runs and compute hours for grid and random hyperparameter search, the gap between training and validation error; the confusion matrix, precision, recall and F1, the cost of errors at different thresholds, the threshold set by the ratio of error costs for a calibrated model, the predicted and observed share of positive cases in calibration bins, the error margin of a share on the validation sample; PSI by feature interval, the difference in shares, latency percentiles, requests per second per server and the number of servers, the cost of serving and retraining; model metrics by group and the gap between them.
- Formulas are given in the task when needed, as are the team's thresholds, compute prices and latency requirements. A calculator is enough for the calculations, and reference materials may be used.
- Answer by the points of the task: the method or formula, the calculation on your own figures, the decision, risks and what needs to be checked. Pasting is disabled, so the answer is typed.
- Prepare the project description for the questionnaire: your role, the task and the target variable, the data, the model, its metrics before and after, the decisions on deployment and monitoring, the result in figures. Do not include confidential employer data (source code, internal datasets, figures the company does not publish) or personal data: the result can be given as a percentage of the starting level. The project question is based on this description.
- Text from AI services and help from other people are not allowed.
Exam
Book the exam
The questionnaire takes 15 minutes. Admission decision in 1 business day.

