Evaluating a quantum machine-learning model requires more than reporting its best accuracy. This checklist shows how to design fair data splits, establish classical baselines, measure performance and calibration, test trainability, account for shot noise, and compare simulators with hardware so that a result is reproducible and meaningful.
Overview
A quantum machine-learning experiment can appear promising for the wrong reasons. A small or unbalanced test set may inflate accuracy, an underpowered classical baseline may make a quantum model look competitive, and a simulator result with ideal measurements may not survive execution on a real device. The purpose of evaluation is not to prove that a quantum model wins every comparison. It is to establish what the model does, under which conditions, and at what computational cost.
Start by defining the task precisely. Record the dataset, target variable, number of features, number of classes, preprocessing steps, encoding method, circuit architecture, optimiser, stopping rule, random seeds, and execution environment. Decide in advance which metric is primary and which results would count as a useful outcome. “Useful” might mean higher recall for a minority class, similar predictive quality with a smaller feature representation, or a clear research finding about trainability. It should not mean selecting the most favourable metric after seeing the results.
For a fair comparison, keep the data preparation and evaluation protocol consistent across models wherever possible. A quantum model should be compared with classical methods that receive the same training information and are tuned with reasonable care. It is also important to separate predictive quality from quantum resource use. A model may produce an interesting result while still requiring too many circuit evaluations, excessive depth, or costly data encoding for the intended application.
Checklist by scenario
When starting with a new dataset
- Inspect the target: Check class counts, missing labels, duplicate records, suspected leakage, and whether the prediction task matches the intended use case.
- Define the split before training: Use training, validation, and test data where tuning is required. For time-ordered or grouped data, use a split that respects time or group boundaries instead of a random split.
- Fit preprocessing on training data only: Scaling, imputation, feature selection, dimensionality reduction, and any learned transformation must not use information from validation or test records.
- Record the data budget: State the total number of samples, features retained, class balance, and the number of samples in each split.
- Choose an encoding deliberately: Basis, angle, and amplitude encoding make different assumptions and can add different circuit or preprocessing costs. Explain why the chosen method suits the feature count and task. See our comparison of quantum data encoding methods.
When comparing a quantum model with a classical baseline
- Include simple baselines: A majority-class predictor, linear model, or other transparent reference can reveal whether the task is difficult and whether accuracy is being driven by class imbalance.
- Add a capable baseline: Depending on the data, compare against at least one well-established nonlinear or ensemble approach. Tune it using the same validation principle applied to the quantum model.
- Match the information available: Do not give the classical model engineered features or augmentation that the quantum model does not receive, and do not handicap the baseline through arbitrary hyperparameter restrictions.
- Report variability: Repeat training across multiple seeds or resampled splits when practical. Report the mean and spread, not only the single best run.
- Separate accuracy from advantage: If the quantum model matches a baseline, that may still be a valid result. However, do not call it an advantage without a defined benefit in quality, resources, robustness, or another relevant dimension.
When testing trainability and circuit design
- Use a small pilot first: Confirm that the loss changes, gradients or parameter updates are non-trivial, and the model can overfit a deliberately tiny subset. Failure on this diagnostic often points to an implementation or optimisation problem.
- Vary architecture components systematically: Compare feature maps, ansatz depth, entanglement patterns, parameter initialisation, and measurement choices one factor at a time where possible.
- Track optimisation behaviour: Save training and validation loss, metric values, parameter norms where useful, number of iterations, and early-stopping events.
- Check parameter sensitivity: Run several initialisations. A model that succeeds only for one narrow initialisation may be fragile even if its best score is high.
- Measure circuit cost: Record qubit count, logical depth, transpiled depth when relevant, gate counts, measurement observables, and the number of circuit evaluations used during training and inference.
When moving from simulation to hardware
- Freeze the experiment: Use the same trained parameters, preprocessing, circuit definition, and post-processing for the simulator and hardware comparison.
- Match shot settings: Record the number of shots per circuit and repeat the comparison at more than one shot budget if shot noise is part of the question.
- Use a noise-aware reference: Where supported by the workflow, compare ideal simulation, noisy simulation, and hardware execution. This helps distinguish device effects from ordinary sampling variation.
- Inspect execution details: Record compilation or transpilation settings, connectivity-related changes, measurement-mitigation steps, queue or batching choices if they affect the experiment, and the date or device configuration used.
- Compare distributions, not only labels: Examine probabilities, expectation values, confidence intervals, and prediction changes. A small accuracy difference may conceal a large shift in probability calibration.
| Comparison | Question it answers | Minimum record |
|---|---|---|
| Quantum model vs simple baseline | Does the model learn beyond a trivial rule? | Primary metric, class distribution, split method |
| Quantum model vs tuned classical model | Is the result competitive under a fair comparison? | Model settings, tuning budget, repeated-run results |
| Shallow vs deeper circuit | Does added expressivity justify added cost? | Depth, gates, evaluations, validation results |
| Ideal simulator vs noisy simulator | How sensitive is the method to modelled noise? | Noise assumptions, shots, uncertainty |
| Simulator vs hardware | Does the result persist under execution conditions? | Device, compilation, shots, mitigation, raw results |
What to double-check
Metrics and uncertainty
Accuracy is easy to interpret but can be misleading when classes are uneven. For classification, consider precision, recall, F1 score, balanced accuracy, and a confusion matrix. For probabilistic predictions, add log loss or another proper scoring rule and inspect calibration. A model that predicts the correct class often but assigns extreme probabilities may be less useful than a slightly less accurate model with better-calibrated confidence.
For regression, report an error metric such as mean absolute error or root mean squared error alongside a scale-aware comparison to a simple baseline. Choose metrics that reflect the actual cost of errors. State whether averages are macro, micro, weighted, or computed over samples, and explain how undefined cases are handled.
Use confidence intervals or repeated-run summaries where feasible. With finite test sets and shot-based measurements, a small difference may not be practically or statistically persuasive. Avoid treating one test-set score as a permanent property of the algorithm.
Shot noise and measurement handling
Quantum predictions are estimated from repeated measurements. Fewer shots can make probabilities and expectation values fluctuate, while more shots increase execution cost. Report shots separately for training and evaluation, because the two stages may use different budgets. If measurement mitigation, post-selection, smoothing, thresholding, or calibration is applied, describe the procedure and apply it consistently.
For details on counts, probabilities, expectation values, and uncertainty, use this guide to interpreting quantum measurement results. Also check that the classical post-processing does not accidentally use test labels or aggregate information across splits.
Resource reporting
A useful experiment report connects model quality to resources. Include the number of qubits, circuit depth before and after compilation where available, two-qubit gate count, shots, number of training iterations, number of circuit evaluations, wall-clock time if measured consistently, and whether execution used an ideal simulator, noisy simulator, or hardware. These values make later comparisons possible even when accuracy is similar.
Hardware context matters. Connectivity, gate fidelity, readout error, relaxation, and dephasing can affect the executed circuit. They should be treated as experimental conditions rather than hidden implementation details. The quantum hardware metrics guide provides useful terminology for documenting these checks.
Common mistakes
- Testing repeatedly on the final test set: If decisions are made after every test result, the test set becomes part of tuning. Keep a final holdout or use a documented resampling strategy.
- Reporting the best seed: A best-case run hides instability. Show all planned runs or a summary with the selection rule stated in advance.
- Ignoring preprocessing leakage: Scaling or feature selection performed before splitting can transfer information from evaluation data into training.
- Using an unrealistic baseline: A poorly tuned or mismatched classical model does not establish quantum value.
- Changing several variables at once: If encoding, depth, optimiser, shots, and data split all change together, you cannot identify the cause of an improvement.
- Confusing simulator precision with evidence: An ideal simulator can validate logic and provide a reference, but it does not demonstrate hardware robustness.
- Optimising only for accuracy: A model with slightly better accuracy may require far more circuit evaluations or produce poorly calibrated decisions.
- Skipping implementation checks: Validate circuit construction, parameter binding, transpilation, measurement mapping, and regression behaviour. The quantum program testing checklist can help structure this stage.
A practical results template is: “Using [dataset and split], [quantum model] achieved [metric] across [number] runs, compared with [baseline]. Results varied by [spread or interval]. The circuit used [qubits], [depth or gate count], [shots], and [evaluation count]. Ideal simulation produced [result], noisy simulation produced [result], and hardware produced [result] under [execution conditions]. The main limitation was [limitation], so the finding should be interpreted as [bounded conclusion].”
When to revisit
Revisit this evaluation checklist whenever the dataset, feature preprocessing, encoding, ansatz, optimiser, shot budget, simulator noise model, hardware target, or framework training API changes. A change that appears minor can alter the effective hypothesis class, resource cost, or measurement variance. Recheck the experiment before comparing new results with old ones.
It is also worth reviewing the protocol before a seasonal planning cycle, a research milestone, a publication, or a decision to move from simulation to hardware. Confirm that baseline implementations still run, metrics still match the use case, and the reporting template captures the resources now relevant to the workflow.
For a final pre-publication or pre-deployment pass, complete these actions:
- Freeze the dataset version, split logic, preprocessing pipeline, and random seeds.
- Run the agreed classical and trivial baselines.
- Execute the quantum model across planned seeds and record failed runs rather than silently excluding them.
- Report primary and secondary metrics with uncertainty or run-to-run variation.
- Compare ideal, noisy, and hardware conditions when hardware claims are part of the question.
- Archive circuit definitions, configuration files, environment versions, raw counts, and processed predictions.
- Write a conclusion that states the observed result, its resource cost, and its limits without claiming more than the experiment supports.
This process turns a quantum machine-learning experiment into an auditable comparison. It does not guarantee a quantum advantage, but it makes it much easier to see whether an apparent improvement comes from the model, the data, the evaluation design, or the execution conditions.