Transforming Cloud Data Infrastructure: A Cloud Modernization Initiative

Background

Our client is largest independent processor of flat-rolled steel and manufacturer of industrial cylinders and pressure vessels in the United States.

Heavily customized BI stack (Informatica mappings, Data Warehouse, RPD Models, and Dashboards) hosted on Oracle Cloud. Cloudera Data lake on AWS to capture detailed operational data from machines, sensors, and shop floor applications to help improve production efficiency and quality.

However, both the Oracle DW and Cloudera Data lake were unstable and unable to integrate with BI tools, hence there was a strong need to modernize the cloud data and analytics platform.

Current Challenges

The client faced several challenges with their existing data infrastructure:

Stability of the Cloudera data lake – many outages and incidents
Data redundancy across data warehouse, data lake, and tableau extracts
Tableau connectivity issue to CDP Impala
Tableau extracts required to hit query performance targets
Unable to reliably hit data latency SLA’s
Complexity of environment requires additional resources
Lacking sufficient visibility over data pipeline health and data quality

Solution

To address these challenges, we proposed the following solution:

Recommend and prove out the target state data and analytics architecture
Evaluate and shortlist technology and tools (Matillion, Talend, DBT, Streamsets, Airflow, Astronomer)
Pilot two uses on the recommended architecture on Data Warehouse (ETL process) and Data Lake (streaming data) to meet success criteria including acceptable performance
Help setup Infrastructure of (AWS, Snowflake, Streamsets, environment
Extend the pilot implementation to migrate DW tables (54 Dim and 60 Fact) and Datalake artifacts (274 base tables and 73 views) for about 800 users
Setup and automate CI / CD pipeline with Github actions and branches for Streamsets + Snowflake with scheduling and monitoring tools

The Results

30% performance improvement by converting Oracle Cloud and Cloudera to AWS-Snowflake stack.
20% SLA improvement by modernizing Oracle and Cloudera workloads on AWS-Snowflake.
50% cost and time savings when transforming Informatica workflows and Oracle EDW to AWS.
Reduction in operational expenses, over $350k annual savings from consolidating Data Warehouse and Data Lakes.

Migrating data from on-premises servers to the AWS cloud significantly improved reporting capabilities. The unified and scalable platform facilitated self-service BI, reducing manual reporting efforts, enhancing data availability, and enabling data-driven decision-making.

Comments (5)

Ismael
Apr 22, 2024

After looking into a handful of the articles on your web site, I seriously like your technique of blogging.
I saved it to my bookmark site list and will be checking back in the near future.

Please check out my website as well and let me know how you feel.

- Ganesh Anabattula
  May 17, 2024
  
  Thank you for your feedback and for bookmarking our site! We’re glad you enjoy our blogs. We’ll definitely check out your website. If you need any support with AI, Data, or Cloud services, feel free to reach out.
  Keep exploring our insights (Blogs, Success Stories, Use Cases, and News) here: https://narwalinc.com/insights/
  contact us: https://narwalinc.com/contact-us/
  
Reynaldo
May 08, 2024

What’s Happening i’m new to this, I stumbled upon this I’ve discovered It absolutely helpful and it
has aided me out loads. I hope to contribute &
aid different customers like its helped me. Good job.

- Ganesh Anabattula
  May 17, 2024
  
  Thank you! We’re glad you found our content helpful. At Narwal, we specialize in AI, Data, Cloud, and Quality Engineering services. If you’re looking to contribute and help others, we’d love to connect with you. Explore our services and see how we can work together to drive innovation and success.
  
  Check our services here: https://narwalinc.com/services/
  contact us: https://narwalinc.com/contact-us/
  
Concetta
May 28, 2024

Greetings from Ohio! I’m bored at work so I decided to browse your
site on my iphone during lunch break. I enjoy the knowledge you provide here and can’t wait to take a look
when I get home. I’m shocked at how quick your blog loaded on my mobile ..

I’m not even using WIFI, just 3G .. Anyways, superb site!