Code Review & Testing Checklist for AI & Machine Learning

Add Great Expectations or Pandera checks for nulls, ranges, dtypes, and categorical domains, and fail the pull request if contracts are violated. Include row count sanity checks and primary key uniqueness to catch silent pipeline defects.

Use DVC or LakeFS to version raw and processed datasets, and require hash pins in code and configs. Review diffs of data manifests in pull requests to ensure immutable, reproducible training and evaluation.

Run Evidently or Alibi Detect in CI to compare feature and label distributions against a locked baseline, with thresholds that gate merges. Flag upstream data shifts early to prevent silent performance drops.

Add tests that compute a small batch of features through both offline and online feature store paths (eg Feast) and assert equality within tolerance. Include time travel tests to prevent training serving skew.

Add automated checks for suspicious correlations between features and targets, and enforce time based splitting with clear cutoff dates. Include PR comments that link to leakage audits and remediation steps.

Verify class distributions, annotation agreement, and label noise rates on each dataset change using small suite tests. Fail fast when label proportions shift beyond configured tolerances.

Run the same pipeline twice in a sandbox orchestrator (Prefect, Dagster, or Airflow) and assert identical artifact checksums when inputs are unchanged. Log seed values and component versions for traceability.

Set seeds for Python, NumPy, and framework libraries, and enable deterministic algorithms (eg torch.use_deterministic_algorithms and cudnn.deterministic). Include a test that verifies repeatable metrics within set tolerance.

Pin dependencies via Poetry or pip-tools, lock CUDA and cuDNN versions, and reference a base Docker image tag in CI. Add a smoke test that builds the container and runs a minimal training step.

Use Hydra or OmegaConf with Pydantic or attrs to validate training configs and snapshot them as MLflow or W and B artifacts. Fail merges if required fields are missing or unexpected overrides appear.

Add finite difference gradient checks and compare outputs to small reference implementations. Include shape inference and mixed precision compatibility tests in the pull request.

Apply numpy.testing or torch.allclose with explicit rtol and atol across CPU and GPU runners. Document platform specific tolerances to avoid flaky tests due to kernel or BLAS differences.

Add an integration test that logs params, metrics, and artifacts to MLflow or Weights and Biases, and asserts run completeness. Enforce run naming and tagging conventions via a review checklist.

Resume from checkpoints mid epoch and ensure metrics and states (optimizer, scheduler, AMP) are restored. Gate merges if resumed training deviates beyond tolerance from a fresh run.

Compare custom metrics to scikit learn or standard references on randomized fixtures using Hypothesis. Add boundary case tests for empty classes, large logits, and ties in ranking metrics.

Run Fairlearn or Aequitas to compute group metrics and raise PR comments when parity gaps exceed policy thresholds. Require reviewers to sign off on mitigation plans or data changes.

Evaluate small PGD or FGSM attacks for vision, or TextAttack perturbations for NLP, and track the relative drop in accuracy. Set a budget that prevents major regressions before deployment.

Compute metrics on key data slices (geography, device type, rare classes) and track worst case slices in CI. Flag regressions even if overall performance looks stable.

Generate reliability diagrams and Brier scores, and ensure post training calibration steps like temperature scaling are applied. Review operating point selection and alert if drift changes optimal thresholds.

Verify that performance, data sources, intended use, and ethical considerations are filled, and attach evaluation artifacts. Fail merges if the model card is missing or out of date.

Pin a small, representative reference set and compare predictions and metrics to a stored baseline within tolerance. Block merges when regressions exceed agreed deltas.

Run load tests with Locust or k6 and capture P50, P95, and P99 latencies for representative payloads. Record SLOs in CI and fail if degradations exceed thresholds.

Test max batch sizes on target hardware, monitor peak GPU and CPU memory, and assert no out of memory or fragmentation issues. Include a stress test with dynamic shapes if applicable.

Exercise inference under concurrent requests using FastAPI, TorchServe, or Triton with multiple workers and assert reproducible outputs. Verify that model objects are not mutated across requests.

Export models to ONNX or TorchScript and run parity checks against native frameworks with numeric tolerances. Validate opset compatibility and fallback paths in CI.

Compare FP32 vs FP16 and INT8 accuracy drops with calibration data and enforce a maximum allowed delta. Include tests for dynamic and static quantization workflows where relevant.

Track model load time, CUDA context initialization, and cache warmup costs for containerized deployments. Gate merges if new dependencies inflate startup beyond agreed budgets.

Expose Prometheus metrics, structured logs, and traces via OpenTelemetry, and assert presence in CI with a quick scrape test. Include request IDs and model version tags in every log line.

Enforce ruff or flake8, black, isort, and mypy in pre commit and CI to maintain readability and type safety. Add Bandit and Semgrep rules for risky patterns like insecure pickle loads.

Use pip audit or Safety and Snyk to block known vulnerabilities and output SBOMs with Syft for traceability. Track CUDA and cuDNN CVEs and set patch policies.

Enable git secrets or Trufflehog in CI, and add tests that assert synthetic datasets are de identified before storage. Restrict access to raw data via role based controls and audit logs.

Forbid unsafe pickle loading paths and prefer safe formats like safetensors, ONNX, or TorchScript. Add tests that ensure unsafe loaders raise exceptions and document approved formats.

Sign artifacts and containers with Cosign, verify checksums, and enforce registry access controls. Add CI attestations for training data versions and config hashes.

Use pytest xdist, reuse cached datasets and model weights across jobs, and split long running tests by duration. Keep seed sets stable to reduce flakiness while covering parallel paths.

Include prompt injection and jailbreak suites, PII red teaming, and content filter checks using frameworks like Guardrails or LangSmith evals. Block merges if safety scorecards slip below policy thresholds.