Machine Learning Primer: Building Robust Training Pipelines

Learn common requirements and patterns in building training pipelines.

We will cover the following

• Training pipeline
• Data partitioning
• Handle imbalance class distribution
• Choose the right loss function
• Retraining requirements

1. Training pipeline

• A training pipeline needs to handle a large volume of data with low costs. One common solution is to store data in a column-oriented format like Parquet or ORC.
• These data formats enable high throughput for ML and analytics use cases.
• In other use cases, the tfrecord(TensorFlow format for storing a sequence of binary records) data format is widely used in the TensorFlow ecosystem.

2.Data partitioning