Client story Major Insurance Provider Saves Big With Big Data Migration
How Insight improved scalability and dramatically reduced cloud costs by spearheading the first successful shift from SAP Big Data Service to Azure HDInsight
By Insight Editor / 27 Oct 2021 / Topics: Artificial Intelligence (AI) Microsoft Azure
Facts at a glance
Client industry:
Finance
Size of company:
Fortune 500, managing more than $248 billion in total assets for two million customers
Challenge:
The organization needed to migrate its complex, legacy SAP® Big Data Service (BDS) data environment by Dec. 31, 2020, to avoid millions of dollars in post-contract fees and fines.
Solution:
Insight conducted a comprehensive assessment of the existing Hadoop clusters and executed the industry’s first successful big data migration from SAP BDS to Azure® HDInsight®.
Results:
- Project completed by end-of-year deadline, avoiding millions in fines
- Reduced monthly cloud expenses by nearly 80% with ongoing savings of more than $750K per month
- Dramatically improved scalability, visibility and productivity for data teams
- Ongoing optimization efforts leveraging Azure Databricks®
In 2020, one of the largest providers of annuities and life insurance in the U.S. reached out to Insight with a unique big data challenge.
As part of their commitment to helping people plan, save and invest for the future, the Fortune 500 company manages $248 billion in total assets for more than two million customers. To help keep the business running, the organization relies on big data to enable data scientists and data engineers to capture key metrics and inform decision-making. The company’s actuaries also depend on this data to inform routine risk analysis and management.
High costs, hard deadlines
At the time, the organization was in the process of severing ties to its parent holding company, which necessitated a migration away from the legacy SAP Big Data Services (BDS) data environment. The company was rapidly approaching the end of its contract with SAP on Dec. 31, 2020. Failure to complete the data migration by this date would put them at risk of fines up to $3 million, with an additional cost of up to $700K for every month beyond their contract.
Even prior to the branch off, the legacy system had grown outdated and inefficient, running Hadoop clusters at high capacity 24/7 and costing nearly $1 million per month, so there was significant opportunity for cost optimization. But, adding to the complexity, there was limited visibility into the size of existing workloads, which made it difficult to determine the best method or even the feasibility of the initiative.
The company needed to migrate away from its legacy SAP Big Data Services environment by Dec. 31, 2020, or risk fines up to $3 million with an additional cost of $700K per month.
Recognizing the scale of this challenge, one of the organization’s data specialists began seeking external support.
Insight had actively been working with the financial services team at Microsoft to explore opportunities to support the client’s former parent company. When Microsoft heard about the subsidiary’s migration project, they recommended Insight for the job.
A groundbreaking shift to HDInsight
In July of 2020, our team began working with the client’s VP of enterprise data and analytics. A discovery session was conducted to assess the legacy environment and the size of the clusters to be migrated. Over the next few months, Insight analysts and program managers worked around the clock to establish a comprehensive understanding of the organization’s needs and build a strong foundation for success.
With the initial assessment finalized, a complete analysis of the BDS data environment was presented back to the client, along with a project plan tailored to the requisite timeline.
Based on the need for rapid execution and the desire to maximize the scalability of the data environment, Insight partnered with Microsoft to propose a two-phase migration approach. Initially, to meet the required timeline, we recommended migrating from SAP to Azure HDInsight — a feat which, to date, had never been accomplished. This would require minimal changes to the work previously completed by the client’s IT team and would allow Insight’s team to focus on the migration and security of the SAP data, as well as the cluster setup and administration. A second phase was also planned to finalize the migration to Databricks, which offered a simplified architecture that would further decrease both the cost and complexity of the existing Hadoop environment.
Architects and engineers began executing the migration in mid-October. Due to the financial nature of the organization’s data, rigorous protocols were implemented and thoroughly tested to ensure the information remained secure from source to destination. Once the data had been successfully transferred to the new environment, the team began standing up two Hadoop clusters to serve the needs of both the client’s actuarial and data science teams — one Hive LLAP cluster and one Spark cluster.
Data was migrated from SAP storage to Azure for consumption through the new Hadoop clusters. Both were designed to maximize the scalability of the HDInsight environment, automatically scaling down to a small number of nodes during times of low demand and scaling up when workloads hit. This elasticity improved performance while dramatically reducing costs compared to the legacy system.
Delivering greater value for less
The project came to a successful close on schedule by the end of the year, enabling the client to avoid hefty fines — and making Insight’s team the first to successfully complete an SAP BDS to HDInsight migration.
The shift has reduced monthly cloud expenses by nearly 80% and saved the client more than $750K per month. In addition to reduced costs, the environment has dramatically improved visibility and productivity for data teams.
The success of this project has led to an ongoing relationship and created new opportunities for our team to assist with the organization’s greater transformation goals, including new initiatives with DevOps, data analytics and SharePoint®.
The shift has reduced monthly cloud expenses by nearly 80% and saved the client more than $750K per month while enabling greater visibility for data teams.
Insight has continued to provide operational support beyond the initial data migration to ensure the new environment continues to deliver value to the organization. Now, in 2022, preparations are underway for the second phase of the project, intended to continue migration and optimization efforts for Azure Databricks. This will enable the client to build upon the scalability and performance improvements, while enabling advanced analytics and machine learning to further improve upon the operating costs of the environment.
