With huge data volumes to manage but only one source of shipping and delivery data models, we aimed at converting key functional requirements into ETL code. To achieve reliable results and to meet set KPIs, we needed to prepare the data properly for use on dashboards, with each data flow completing within 24 hours to ensure real-time results.
We evaluated three different ETL options, two different encoding options and one ELT graphic UI. From here, we used extensive testing to identify the optimal configuration overall.
The advanced features we implemented led to better execution performance for indexes, partitioning and stored procedure. We also implemented sequential or automatic activation when the input source data is updated, and automated alerts for incomplete data.