Transport for London (TfL) set out to modernise its Road User Charging system, which involved consolidating and migrating vast amounts of historical data from its legacy infrastructure. We designed a custom data migration strategy and system to efficiently transfer petabytes of structured and unstructured data, ensuring a seamless transition with parallel-run support.
1. Requirement Analysis and Planning:
to identify systems, establish compliance, and define performance metrics;
2. Data Profiling and Assessment:
to analyse and cleanse source data;
3. Data Mapping and Transformation Design:
to define mappings, transformation rules, and error-handling mechanisms;
4. Migration Strategy and Execution Plan:
to select the migration approach, design cloud architecture, and create a detailed execution plan and timeline.
Our seasoned data engineering team carefully evaluated the project’s needs to select the most suitable tooling. While Microsoft Fabric and Data Factory were the original preferences, our assessment demonstrated that Databricks was the best fit for the job. Its cloud-independent nature provided flexibility, while its ability to handle massive-scale workloads ensured a seamless migration of TfL’s vast data assets. Additionally, Databricks offers superior data governance and cost control, making it the optimal choice for maintaining efficiency, security, and scalability.
