• Dr I Morgan

Data Pipeline Development

Updated: May 26

Automatic data injestion and analytics on Azure

The customer required a way of injesting multiple data points from different sources including vessels, and laboratories before generating output reporting and conclusions. We developed an Azure web app in C# and Vue.js, with Azure functions running a number of Python analytic models, and blob storage to manage the custom analytics and file management pipeline.

As data is sparse and data load is generally low, we use a Snowflake instance on the consumption plan to store the generated output which can be consumed by a variety of OLEDB dashboarding tools with the aim of keeping processing and storage costs low yet allowing the analytics to scale when needed.

This allowed the customer to process many more reports and provided an audit trail so that results are repeatable and saved for future reference.

5 views0 comments

Recent Posts

See All