Ml Pipeline Workflow
Unverified●31/40Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor◐PartialPlain prose you can paste in — but no Cursor rules file
Codex◐PartialPlain prose you can paste in — but no AGENTS.md
Gemini CLI◐PartialPlain prose you can paste in
Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file
npx agentalley add ml-pipeline-workflowWho is stuck, and on what
Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows.
The whole source
Frontmatter — 2 properties
| name | ml-pipeline-workflow |
|---|---|
| description | Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows. |
| 1 | --- |
| 2 | name: ml-pipeline-workflow |
| 3 | description: Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows. |
| 4 | ---A5 — No allowed-tools declared — no way to tell what this skill may touch |
| 5 | |
| 6 | # ML Pipeline Workflow |
| 7 | |
| 8 | Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment. |
| 9 | |
| 10 | ## Overview |
| 11 | |
| 12 | This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring. |
| 13 | |
| 14 | ## When to Use This Skill |
| 15 | |
| 16 | - Building new ML pipelines from scratch |
| 17 | - Designing workflow orchestration for ML systems |
| 18 | - Implementing data → model → deployment automation |
| 19 | - Setting up reproducible training workflows |
| 20 | - Creating DAG-based ML orchestration |
| 21 | - Integrating ML components into production systems |
| 22 | |
| 23 | ## What This Skill Provides |
| 24 | |
| 25 | ### Core Capabilities |
| 26 | |
| 27 | 1. **Pipeline Architecture** |
| 28 | - End-to-end workflow design |
| 29 | - DAG orchestration patterns (Airflow, Dagster, Kubeflow) |
| 30 | - Component dependencies and data flow |
| 31 | - Error handling and retry strategies |
| 32 | |
| 33 | 2. **Data Preparation** |
| 34 | - Data validation and quality checks |
| 35 | - Feature engineering pipelines |
| 36 | - Data versioning and lineage |
| 37 | - Train/validation/test splitting strategies |
| 38 | |
| 39 | 3. **Model Training** |
| 40 | - Training job orchestration |
| 41 | - Hyperparameter management |
| 42 | - Experiment tracking integration |
| 43 | - Distributed training patterns |
| 44 | |
| 45 | 4. **Model Validation** |
| 46 | - Validation frameworks and metrics |
| 47 | - A/B testing infrastructure |
| 48 | - Performance regression detection |
| 49 | - Model comparison workflows |
| 50 | |
| 51 | 5. **Deployment Automation** |
| 52 | - Model serving patterns |
| 53 | - Canary deployments |
| 54 | - Blue-green deployment strategies |
| 55 | - Rollback mechanisms |
| 56 | |
| 57 | ### Reference Documentation |
| 58 | |
| 59 | See the `references/` directory for detailed guides: |
| 60 | |
| 61 | - **data-preparation.md** - Data cleaning, validation, and feature engineering |
| 62 | - **model-training.md** - Training workflows and best practices |
| 63 | - **model-validation.md** - Validation strategies and metrics |
| 64 | - **model-deployment.md** - Deployment patterns and serving architectures |
| 65 | |
| 66 | ### Assets and Templates |
| 67 | |
| 68 | The `assets/` directory contains: |
| 69 | |
| 70 | - **pipeline-dag.yaml.template** - DAG template for workflow orchestration |
| 71 | - **training-config.yaml** - Training configuration template |
| 72 | - **validation-checklist.md** - Pre-deployment validation checklist |
| 73 | |
| 74 | ## Usage Patterns |
| 75 | |
| 76 | ### Basic Pipeline Setup |
| 77 | |
| 78 | ```python |
| 79 | # 1. Define pipeline stages |
| 80 | stages = [ |
| 81 | "data_ingestion", |
| 82 | "data_validation", |
| 83 | "feature_engineering", |
| 84 | "model_training", |
| 85 | "model_validation", |
| 86 | "model_deployment" |
| 87 | ] |
| 88 | |
| 89 | # 2. Configure dependencies |
| 90 | # See assets/pipeline-dag.yaml.template for full example |
| 91 | ``` |
| 92 | |
| 93 | ### Production Workflow |
| 94 | |
| 95 | 1. **Data Preparation Phase** |
| 96 | - Ingest raw data from sources |
| 97 | - Run data quality checks |
| 98 | - Apply feature transformations |
| 99 | - Version processed datasets |
| 100 | |
| 101 | 2. **Training Phase** |
| 102 | - Load versioned training data |
| 103 | - Execute training jobs |
| 104 | - Track experiments and metrics |
| 105 | - Save trained models |
| 106 | |
| 107 | 3. **Validation Phase** |
| 108 | - Run validation test suite |
| 109 | - Compare against baseline |
| 110 | - Generate performance reports |
| 111 | - Approve for deployment |
| 112 | |
| 113 | 4. **Deployment Phase** |
| 114 | - Package model artifacts |
| 115 | - Deploy to serving infrastructure |
| 116 | - Configure monitoring |
| 117 | - Validate production traffic |
| 118 | |
| 119 | ## Best Practices |
| 120 | |
| 121 | ### Pipeline Design |
| 122 | |
| 123 | - **Modularity**: Each stage should be independently testable |
| 124 | - **Idempotency**: Re-running stages should be safe |
| 125 | - **Observability**: Log metrics at every stage |
| 126 | - **Versioning**: Track data, code, and model versions |
| 127 | - **Failure Handling**: Implement retry logic and alerting |
| 128 | |
| 129 | ### Data Management |
| 130 | |
| 131 | - Use data validation libraries (Great Expectations, TFX) |
| 132 | - Version datasets with DVC or similar tools |
| 133 | - Document feature engineering transformations |
| 134 | - Maintain data lineage tracking |
| 135 | |
| 136 | ### Model Operations |
| 137 | |
| 138 | - Separate training and serving infrastructure |
| 139 | - Use model registries (MLflow, Weights & Biases) |
| 140 | - Implement gradual rollouts for new models |
| 141 | - Monitor model performance drift |
| 142 | - Maintain rollback capabilities |
| 143 | |
| 144 | ### Deployment Strategies |
| 145 | |
| 146 | - Start with shadow deployments |
| 147 | - Use canary releases for validation |
| 148 | - Implement A/B testing infrastructure |
| 149 | - Set up automated rollback triggers |
| 150 | - Monitor latency and throughput |
| 151 | |
| 152 | ## Integration Points |
| 153 | |
| 154 | ### Orchestration Tools |
| 155 | |
| 156 | - **Apache Airflow**: DAG-based workflow orchestration |
| 157 | - **Dagster**: Asset-based pipeline orchestration |
| 158 | - **Kubeflow Pipelines**: Kubernetes-native ML workflows |
| 159 | - **Prefect**: Modern dataflow automation |
| 160 | |
| 161 | ### Experiment Tracking |
| 162 | |
| 163 | - MLflow for experiment tracking and model registry |
| 164 | - Weights & Biases for visualization and collaboration |
| 165 | - TensorBoard for training metrics |
| 166 | |
| 167 | ### Deployment Platforms |
| 168 | |
| 169 | - AWS SageMaker for managed ML infrastructure |
| 170 | - Google Vertex AI for GCP deployments |
| 171 | - Azure ML for Azure cloud |
| 172 | - OCI Data Science for Oracle Cloud Infrastructure deployments |
| 173 | - Kubernetes + KServe for cloud-agnostic serving |
| 174 | |
| 175 | ## Progressive Disclosure |
| 176 | |
| 177 | Start with the basics and gradually add complexity: |
| 178 | |
| 179 | 1. **Level 1**: Simple linear pipeline (data → train → deploy) |
| 180 | 2. **Level 2**: Add validation and monitoring stages |
| 181 | 3. **Level 3**: Implement hyperparameter tuning |
| 182 | 4. **Level 4**: Add A/B testing and gradual rollouts |
| 183 | 5. **Level 5**: Multi-model pipelines with ensemble strategies |
| 184 | |
| 185 | ## Common Patterns |
| 186 | |
| 187 | ### Batch Training Pipeline |
| 188 | |
| 189 | ```yaml |
| 190 | # See assets/pipeline-dag.yaml.template |
| 191 | stages: |
| 192 | - name: data_preparation |
| 193 | dependencies: [] |
| 194 | - name: model_training |
| 195 | dependencies: [data_preparation] |
| 196 | - name: model_evaluation |
| 197 | dependencies: [model_training] |
| 198 | - name: model_deployment |
| 199 | dependencies: [model_evaluation] |
| 200 | ``` |
| 201 | |
| 202 | ### Real-time Feature Pipeline |
| 203 | |
| 204 | ```python |
| 205 | # Stream processing for real-time features |
| 206 | # Combined with batch training |
| 207 | # See references/data-preparation.md |
| 208 | ``` |
| 209 | |
| 210 | ### Continuous Training |
| 211 | |
| 212 | ```python |
| 213 | # Automated retraining on schedule |
| 214 | # Triggered by data drift detection |
| 215 | # See references/model-training.md |
| 216 | ``` |
| 217 | |
| 218 | ## Troubleshooting |
| 219 | |
| 220 | ### Common Issues |
| 221 | |
| 222 | - **Pipeline failures**: Check dependencies and data availability |
| 223 | - **Training instability**: Review hyperparameters and data quality |
| 224 | - **Deployment issues**: Validate model artifacts and serving config |
| 225 | - **Performance degradation**: Monitor data drift and model metrics |
| 226 | |
| 227 | ### Debugging Steps |
| 228 | |
| 229 | 1. Check pipeline logs for each stage |
| 230 | 2. Validate input/output data at boundaries |
| 231 | 3. Test components in isolation |
| 232 | 4. Review experiment tracking metrics |
| 233 | 5. Inspect model artifacts and metadata |
| 234 | |
| 235 | ## Next Steps |
| 236 | |
| 237 | After setting up your pipeline: |
| 238 | |
| 239 | 1. Explore **hyperparameter-tuning** skill for optimization |
| 240 | 2. Learn **experiment-tracking-setup** for MLflow/W&B |
| 241 | 3. Review **model-deployment-patterns** for serving strategies |
| 242 | 4. Implement monitoring with observability tools |
| 243 | |
| 244 | ## Related Skills |
| 245 | |
| 246 | - **experiment-tracking-setup**: MLflow and Weights & Biases integration |
| 247 | - **hyperparameter-tuning**: Automated hyperparameter optimization |
| 248 | - **model-deployment-patterns**: Advanced deployment strategies |
| 249 |
Reviews
Installed this one?Write the first review and take the Trailblazer badge.
Alternatives
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.◐◐◐◐◐●35/40Ebay Seller Tools·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.●····●32/40