On Prem to Cloud Data Migration

Background

We partnered with an American Payment Processor operating nationwide in North America. They provide online payment processing, as well as products for face-to-face and telephone payments. They wanted to create a cloud-based unified platform for enterprise reporting, known as the Enterprise Data Platform, using the AWS technology stack.

Current Challenges

They faced several challenges with their existing data infrastructure:

Data Silos: Each line of business had its own reporting platform, leading to complexity and time-consuming reconciliation efforts.

In-House Data Processing: Most of the data was stored on in-house servers, increasing the maintenance burden.

Data Availability & Self-Service BI: Reporting and consolidation were predominantly manual processes, limiting business users’ ability to access and analyze data independently.

Scalability: The existing infrastructure was not designed to scale up or down based on varying data volumes and business needs.

Solution

To address these challenges, we proposed the following solution:

Data Migration: We used AWS Database Migration Service (DMS) to transfer data from their local data center to the AWS cloud.

Data Storage: The migrated data was stored in the Raw Layer, ensuring its availability for further processing.

Data Processing: Standard data quality rules were applied to transform and curate the data, moving it to the Curated Layer / Lake.

Harmonized Data: Business rules were applied to populate the harmonized layer in Amazon Redshift, enabling standardized reporting and analysis.

Reporting Platform: The unified platform leveraged Amazon QuickSight to access pre-calculated data from Redshift, enabling self-service Business Intelligence (BI).

The Results

We achieved the following results for them:

Unified Platform: All the previously disparate data sources were successfully migrated and consolidated into a single data repository in Redshift. This created a one-stop solution for accessing any data request within the organization.

Cloud-Based Scalability: By utilizing DMS and Redshift, they achieved a serverless and scalable solution. They could easily scale up or down based on varying data volumes and business needs.

Self-Service BI: Business calculations were pre-calculated and stored in Redshift, making it accessible to business users. This empowered them to consume the data as needed and perform independent data analysis quickly and efficiently.

By migrating their data from on-premises servers to the AWS cloud, they achieved a significant improvement in their reporting capabilities. They now had a unified and scalable platform that allowed for self-service BI, reducing the manual effort required for reporting and analysis. Overall, the cloud-based data migration solution increased efficiency, improved data availability, and empowered the organization to make data-driven decisions.