A product will inevitably yield miscellaneous data from different aspects, which is veracious reflection on the product’s characteristic and could provide insight regarding improvement of product itself or detection of miscellaneous users, etc. Consequently, the concept of ETL emerges.

ETL represents a standard way of architecting the data pipeline. More specifically, data will be loaded from multiple sources with heterogeneous formats (raw text records, XML, JSON, Image, etc.), some transformation will be made on top of the raw data and persists to the underlying data warehouse. After this procedure, data could be used to trace back for the history and status quo of the product via being queried to generate report or feeding to monitoring system on batch/real-time computing process frameworks like MapReduce, Hive or Storm, etc.

In the past, there’s relatively small amount of data which could be stored, organized and handled by limited PC machines, this is more common in traditional companies like banks, hospitals or supermarkets, etc. However, the topic for data processing is changed dramatically in terms of size, variety, heterogeneity and business requirements as the Internet becomes faster, at the same time, PC and mobile phone become more prevalent nowadays. Previous works have already been done to migrate from storing and computing on a single supercomputer to distributed storage system and parallel computing thanks to the emergence of Hadoop Ecosystem 

Hadoop framework helps solve an essential problem, that the supercomputer will eventually suffer from hardware bottleneck as the data size and computation complexity is growing day by day. Moreover, company doesn’t have to buy this kind of costly server machine anymore, instead, commodity machine will suffice with the dedicate-designed distributed computing algorithms [2]. Data will be stored on each server and the computation procedure is moved to where data resides, not reverse, which could reduce the I/O shuffle remarkably and provide lots of valuable features like fault tolerance, replications, consistency, etc. which are all transparent to front-end users

At the same time, Spark is one of the most popular in-memory parallel computing frameworks which yields better performance in most scenarios when comparing to Hive or MapReduce [5]. Moreover, Spark provides unified integral packages for different computation models which are all benefit from Spark’s underlying design and structure, namely, graph computing, machine learning, stream processing and batch processing 

Being in a car transportation company, there are tens of millions of users all around the world calling on private cars serving as taxi, which will generate over billions of user behavior events every day. For instance, user and driver information, service area, trip and booking related tables. Some of the metrics will be massive in amount, like driver pings, which will yield more than billions of records per day by sending driver’s location to the backend data center; also, like trip and booking related tables, every status (booking a ride, order is taken, arrival of driver, customer in the car and destination is made) will be logged in the backend as well for the purpose of tracing back later on, so the amount of data for trip and booking will be up to hundreds of millions per day.

Consequently, great values could be extracted and the company has already built data platform based on on-premises Hive cluster to consume data for usages like guiding decision making, fraud detection, A/B testing and monitoring systems, etc.