FMCG Data Integration
As part of a client's migration to Microsoft Dynamics 365 an integration to their Warehouse Management System was required. The designed solution was capable of handling millions of incoming files, transforming & staging data, before applying business logic and pushing into Microsoft Dynamics. By utilising Azure data factory and leveraging cloud computing, the solution reduced the processing time from minutes down to seconds per file.
Final Solution
The final solution was a fully event driven pipeline that would trigger when files landed in blob storage. Ingesting and transforming the data before pushing it into staging tables. This in turn would trigger stored procedures to run business logic on the new data before inserting into the final tables ready to be pushed into Dynmaics365.
This project required a wide range of technical and business skills used in conjunction together to ensure the project was delivered successfully.
Work with business analysts to understand the problem and gather requirements
Design and build the database used by the solution from scratch
Write stored procedures in T-SQL used by the database
Design the end to end data flow of the solution​
Build Data Factory pipelines
Consider error handling from technical and business perspective
Consider and challenge companies integration policy to obtain best technical solution
Write complete technical documentation
Regular status updates and feedback sessions for continuous improvement
Project Manager
"His unique ability to work under pressure and break down complex problems into manageable chunks, coupled with effective communication with all stakeholders, ensured alignment among individuals from different backgrounds."
Head of Data Integration
"Rob’s ability to provide clarity of progress and solution to large (and at times difficult) client audiences was fantastic. The solution Rob was tasked with was quite complex, and required strong collaboration with multiple teams to achieve the right end state, which Rob again did again very well and set a high benchmark."
Head of Data Integration
"Rob was able to take on board feedback, propose a design and iterate this design down to the final state. Based on this experience, gave confidence to his competency and the solutions quality."