Documentation & Knowledge Base Checklist for AI & Machine Learning

Catalog each dataset's owner, schema, upstream systems, and retention policy in a central registry like OpenMetadata or DataHub. Link to storage locations and backup procedures so dataset versions are discoverable and compliant.

Document field-level expectations, update cadence, nullability, and value ranges, then enforce with Great Expectations or Pandera. Break builds on contract violations to prevent silent training-serving skew.

Record dataset digests, DVC tags, LakeFS commits, or Delta Lake version IDs in README files and tracking tools. Include checksum and query filters to recreate the exact training and evaluation splits.

Document feature semantics, units, allowed transforms, and leakage risks. If using a feature store like Feast, store feature views, owners, and backfill windows alongside usage examples.

Write clear annotation guidelines, edge cases, and escalation paths, and track inter-annotator agreement. Reference tools such as Label Studio or Prodigy with scripts for audit sampling and class balance checks.

List PII fields, masking or tokenization strategy, and approved de-identification methods. Capture DPIA outcomes and access controls so teams can align model training with privacy requirements.

Include objective, dataset versions, seed settings, environment export, and expected runtime. Keep the template in your repo and auto-populate with commit hash and branch via a pre-commit hook.

Log hyperparameters, metrics, artifacts, and code reference. Attach dataset digests and environment snapshots so a run can be replayed without guessing.

Document Dockerfiles, base images, CUDA drivers, and Python lock files. Record random seeds and flags like cudnn.deterministic to reduce nondeterministic variance across GPUs.

Store SQL queries, feature store commit IDs, and preprocessing code hash. Persist a pipeline fingerprint to detect accidental data drift between experiments.

Ship Makefile or Invoke tasks like make repro RUN_ID=... that fetch artifacts and run inference against the saved environment. Document expected hardware and fallback behaviors.

Keep a canonical baseline with frozen metrics and seed, plus guidelines for single-factor ablations. Include a small ablation matrix template to avoid confounded conclusions.

Adopt a model card template that covers intended use, limitations, training data summary, metrics by segment, and ethical considerations. Store alongside the model in the registry and link to the run ID.

Define fixed test sets, seed handling, and acceptable variance bounds. Use paired tests like McNemar or bootstrap CIs and document decision thresholds to avoid metric roulette.

List online and batch metrics, population stability indices, and data quality rules with Evidently or custom checks. Document alert channels and the immediate rollback or degrade path.

Define protected attributes, business-relevant subgroups, and fairness metrics. Capture mitigation strategies evaluated with tools like Fairlearn or Aequitas and their tradeoffs.

Record go or no-go criteria, stakeholder approvals, and risk sign-offs. Include a reproducible artifact digest, rollback image tag, and compatibility matrix with client services.

List dataset and model weight licenses, allowed commercial use, and attribution requirements. Attach a software bill of materials generated with Syft for compliance audits.

Document request shapes, response fields, error codes, rate limits, and timeouts. Include curl and Python examples, plus Postman collections for quick smoke tests.

Capture SLOs, SLIs, dashboards, common failure modes, and escalation paths. Include steps for canary promotion, blue-green rollback, and cache or feature flag toggles.

Describe Airflow, Prefect, or Dagster graphs, dependencies, retries, and idempotency. Add backfill and replay procedures with data validation gates to avoid duplicate writes.

Use Keep a Changelog format and conventional commits to auto-generate release notes. Flag breaking changes explicitly and provide migration steps for downstream teams.

Link Terraform or Pulumi modules, environment variables, and secret rotation policies. Include least-privilege IAM roles and drift detection procedures.

Document GPU utilization targets, batch sizes, quantization or distillation options, and autoscaling policies. Include profiling steps with Nsight or PyTorch profiler and expected gains.

Store prompts with IDs, variants, and A or B outcomes, plus guardrail behavior. Use promptfoo or TruLens to keep regression tests and tie each change to a model or dataset hash.

Specify sources, deduping, toxicity filters, and PII scrubbing steps. Include regex and embedding-based filters with evaluation of false positives and negatives.

Record task-specific metrics, human review protocols, and judge model versions. Use RAGAS or custom rubric scoring and capture inter-rater agreement to stabilize launch decisions.

Explain chunking, overlap, embedding models, vector DB choice, and recall targets. Include negative sampling strategy, filter clauses, and cache invalidation rules.

List injection patterns, jailbreak defenses, and content moderation flows. Provide allow and block lists, sandboxed tool execution policies, and audit logging schema.

Include repo structure, naming conventions, pre-commit hooks, and codeowners. Provide a quickstart path that runs tests, spins a local API, and executes a smoke RAG query.