From Information to Locations: How Skyscanner Optimizes Traveler Experiences with Databricks Unity Catalog

At Skyscanner, we’re greater than only a flight search engine. We’re a worldwide chief in journey in serving greater than 110 million customers every month to plan and guide their journeys with confidence and ease. Working in over 30 languages, our platform connects vacationers with a variety of flights, motels, and automotive rental choices from over 1,200 journey companions throughout 180 international locations.  

We use information and AI to boost the traveler expertise in addition to assist inside decision-making. For our vacationers, we use machine studying (ML) fashions to verify over 80 billion costs on daily basis, rating and recommending motels, flights, and automotive leases, aiming to offer the perfect choices primarily based on journey time and prices. Databricks Information Intelligence Platform powers a few of these journey insights. On this weblog, we talk about our journey with Databricks and the way Unity Catalog helped us streamline our information administration and governance.

To study extra, attend the Information + AI Summit 2024 for our session titled  Skyscanner’s Journey of Enabling Sensible Information and AI Governance. 

Understanding Our Information Panorama and Challenges

Information has at all times been central to Skyscanner’s operations. On daily basis, our platform handles 35 million searches, producing over 30 to 35 billion analytical occasions. The sheer quantity of knowledge—roughly 15 to twenty petabytes saved at any given time—poses vital challenges in information administration and utilization.  Our information is essential for each consumer-facing options and inside decision-making processes, making its efficient administration a high precedence for our engineering groups. This scale of knowledge operations presents a number of challenges:

• Quantity and Velocity: Dealing with billions of occasions generated day by day requires strong infrastructure and environment friendly information processing capabilities.
• Scalability and Efficiency Points: As Skyscanner grew, the information infrastructure struggled to maintain tempo with the growing demand. Our legacy methods couldn’t scale effectively, resulting in delays in information processing and an lack of ability to deal with large-scale information workloads successfully.
• Complexity and Value: Earlier than transitioning to extra streamlined options, our information administration concerned a number of methods, which regularly led to inefficiencies and elevated operational prices.
• Information Silos and Inconsistency: The disparate methods led to information being siloed, which hindered information accessibility and high quality, affecting decision-making processes.
• Compliance and Safety Dangers: With information unfold throughout numerous methods, making certain complete safety and compliance with worldwide information safety rules (like GDPR) was more and more difficult. This danger was compounded by the dearth of centralized management over information entry and processing.

Databricks: A Recreation-Changer for Skyscanner

At Skyscanner, our dedication to leveraging cutting-edge know-how is obvious in our strategic partnership with Databricks. Databricks has been instrumental in reworking our strategy to information administration, enabling us to streamline operations and improve the traveler expertise. 

All our information pipelines are constructed on high of the Databricks Information Intelligence Platform. we have established a strong information ingestion framework that captures information from a wide range of sources, incorporating each batch and real-time streams. We make the most of AWS Kinesis for streaming and Fivetran for batch information ingestion, making certain that every one incoming information is collected effectively into our preliminary staging space, which we check with because the ‘bronze layer’ of our medallion structure. This stage is essential because it handles the uncooked information collected from our numerous channels, together with direct interactions from our net and cellular platforms.

As soon as within the bronze layer, the information undergoes a sequence of transformations and enrichments to organize it for deeper analytical duties. It then strikes to the ‘silver layer,’ the place it’s cleaned, consolidated, and structured, prepared for analytical consumption. On this part, Databricks’ highly effective Spark engine performs an important position, enabling quick and scalable information transformations.

Advancing the information to the ‘gold layer,’ our information is optimized for consumption by numerous enterprise models the place it’s modeled and aggregated into metrics that instantly assist decision-making throughout the corporate. We leverage MLflow, to handle the whole machine studying lifecycle. This contains every thing from experimentation and reproducibility to the deployment of ML fashions, permitting us to trace experiments, package deal code into reproducible runs, and deploy fashions instantly into manufacturing seamlessly. Whereas we’re at the moment serving these fashions into manufacturing utilizing our personal model-serving structure, we’re within the technique of evaluating Databricks’ model-serving capabilities which can be a part of the Databricks Mosaic AI providing.

Past processing and machine studying, we make the most of Databricks for operational reporting and analytics. Databricks SQL permits our groups to carry out SQL queries instantly towards our information lake, create dashboards, and execute advanced analytical operations at scale. Integration with BI instruments like Tableau Cloud enhances our capabilities, enabling us to visualise information and extract actionable insights effectively.

Our Migration Journey to Unity Catalog

Information governance is a important element of Skyscanner’s structure. It underpins our skill to handle information securely and effectively, making certain that we will belief our information for making enterprise choices and sustaining compliance with world information safety rules, together with GDPR. As a subsidiary of an organization listed on NASDAQ, adhering to strict regulatory requirements such because the Sarbanes-Oxley Act is paramount for making certain transparency and accountability in our operations. Databricks Unity Catalog, being constructed into the platform, helped us streamline these necessities.

Earlier than implementing Unity Catalog, we confronted a number of vital challenges

• Low Ranges of Information Possession: One of many extra vital challenges we confronted was the low stage of possession over datasets throughout the corporate. This usually led to accountability points, the place no particular crew or particular person was chargeable for the accuracy, privateness, and safety of explicit datasets.
• Lack of Centralized Oversight: Managing information throughout disparate methods made it tough to implement constant information governance insurance policies. This lack of centralized management led to inefficiencies and elevated the chance of non-compliance with information rules resembling GDPR.
• Entry Management Difficulties: And not using a unified system, managing who had entry to what information was cumbersome and sometimes insecure. Dealing with IAM insurance policies was significantly difficult, requiring substantial guide effort and being vulnerable to errors. Guaranteeing the correct stage of entry for numerous groups concerned navigating advanced IAM roles, which regularly led to both overly permissive entry or overly restrictive practices, each of which might impede operational effectivity.
• Insufficient Information Lineage and Auditing: We lacked automated instruments for monitoring information lineage and auditing modifications, that are important for troubleshooting and understanding the influence of knowledge modifications. Because of this, lineage graphs needed to be ready manually.

Recognizing these challenges, we developed a strategic strategy emigrate to Unity Catalog. Our technique included:

• Prioritizing Enterprise-Vital Tables: We performed a complete overview of all information belongings to categorise them in line with their significance to enterprise operations, sensitivity, and compliance necessities. Though we had 30,000 tables in complete, our energetic tables numbered solely about 1,500, and of these, solely about 350 had been business-critical. That discovery was a recreation changer for us as this simplified our migration course of.
• Leveraging Automation: Initially, our groups manually migrated tables into Unity Catalog and tailored them to suit our area mannequin, which was a sluggish and time-consuming course of. By leveraging Databricks’ automation instruments, we considerably accelerated the migration with no need to rewrite our pipelines. To expedite the mixing of all our information into Unity Catalog, we grew to become much less inflexible about adhering strictly to the Medallion structure, which requires all information to be categorised into bronze, silver, and gold layers. As an alternative, we adopted a extra versatile strategy: “We’ll meet you the place your information is.” This technique allowed us to make information seen within the Unity Catalog instantly, with the intention of aligning it with the bronze, silver, and gold definitions over time.

Enhancing information visibility and governance with Unity Catalog

Unity Catalog has change into a pivotal aspect in our information governance framework at Skyscanner. it now manages and governs a major quantity, roughly 15 to twenty petabytes, of our information. This information contains every thing from uncooked information in our ‘bronze’ layer to processed information in our ‘silver’ and ‘gold’ layers, that are used extensively throughout numerous enterprise capabilities for analytical and operational functions. 

The implementation of Unity Catalog has introduced substantial enhancements to our information administration and governance capabilities, yielding a number of key advantages:

• Enhanced Information Safety and Compliance: Unity Catalog has enabled us to centralize our information governance, offering strong security measures and streamlined compliance processes. This centralization decreased the complexities related to managing permissions throughout disparate methods and helped make sure that solely licensed personnel had entry to delicate information and is essential for adhering to stringent information safety legal guidelines, together with GDPR. 
• Value Optimization: The streamlined information administration course of enabled by Unity Catalog has led to extra environment friendly use of our information storage and computing sources.
• Scalability and Future-Proofing: Unity Catalog has supplied a scalable structure that accommodates our rising information wants. As Skyscanner continues to increase and evolve, Unity Catalog helps this progress by enabling us to handle growing volumes of knowledge with out compromising on efficiency or safety.
• Enhanced Information Lineage:  With Unity Catalog, we have considerably enhanced our information lineage capabilities. This implies we now have a transparent and detailed view of the place our information originates, the way it’s processed alongside the best way, and the place it finally ends up. This stage of transparency is essential not only for day-to-day operations but in addition for our compliance efforts, significantly with GDPR. With the ability to hint the whole journey of our information helps us make sure that we’re dealing with it appropriately and staying compliant with all vital rules. It additionally simplifies the audit course of, as we will readily present detailed mappings of our information flows.
• Information Observability:  Constructing on our information in Unity Catalog, we’ve got built-in Monte Carlo to enhance information reliability throughout our energetic datasets. We now have launched a wholesome information framework in order that we will measure the adoption of knowledge governance throughout Skyscanner. 

Planning for the longer term: Capitalizing on new alternatives 

As we glance forward, I believe the worth in generative AI will come from the distinctive, precious information we’ve got at Skyscanner. There’s loads of potential, however a key step for us is ensuring we’ve got every thing, together with ML fashions, managed and ruled with Unity Catalog to capitalize on any alternatives.

At present we’re evaluating utilizing Databricks’ Mannequin Serving functionality. We’re enabling Unity Catalog in a number of areas utilizing Delta Sharing to maneuver information between areas. We’re additionally excited about utilizing this for exterior information sharing – we’ve got some information merchandise the place we share information with third celebration firms.

Sooner or later, we wish our information groups to concentrate on issues distinctive to Skyscanner. Databricks does loads of the heavy lifting relating to mannequin serving and supplies an excellent framework for excited about the AI journey—from immediate engineering to constructing your personal mannequin. We now have confidence in our skill to appreciate the alternatives we’re figuring out utilizing the Databricks ecosystem.

Be taught extra about Skyscanner’s journey on the Information + AI  2024 Summit by becoming a member of Michael’s session, Skyscanner’s Journey of Enabling Sensible Information and AI Governance.