Description
Consulting, BI
Technologies
Snowflake, SQL, Sigma Computing, PHP
Problem
I had to set up a business intelligence project for a company that wanted to start using its data from multiple sources, such as the ticketing system, the core app, the KYC software, and other minor projects.
Solution
Based on what the company was already using, I designed a solution based on AWS, Snowflake, and Sigma Computing.
I designed the data warehouse on Snowflake, to which new components were gradually added as the integration with the different data sources was completed.
Then I provided the development teams working on the different software applications with the JSON schema for the data that needed to be sent to AWS Kinesis.
Stack
This project is based on AWS, Snowflake, and Sigma Computing
Design
It was necessary to handle the data stream using a queue manager like Beanstalk to prevent the risk of losing data while pushing it to AWS Kinesis.
We therefore created the data lake from which data is imported into Snowflake and transformed to populate the data warehouse.
I was fully responsible for implementing the ELTs using tasks and streams, as well as for designing the different data marts.
Finally, I worked with Sigma Computing to create the reports that gave the different teams access to the data.