Spatial Big Data Management and Analytics - Taxi Trajectory Analysis for Finding Pick-up Hotspots

Loading...

來自 Yonsei University 的課程

Spatial Data Science and Applications

64 個評分

Yonsei University

Spatial Data Science and Applications

64 個評分

Spatial (map) is considered as a core infrastructure of modern IT world, which is substantiated by business transactions of major IT companies such as Apple, Google, Microsoft, Amazon, Intel, and Uber, and even motor companies such as Audi, BMW, and Mercedes. Consequently, they are bound to hire more and more spatial data scientists. Based on such business trend, this course is designed to present a firm understanding of spatial data science to the learners, who would have a basic knowledge of data science and data analysis, and eventually to make their expertise differentiated from other nominal data scientists and data analysts. Additionally, this course could make learners realize the value of spatial big data and the power of open source software's to deal with spatial data science problems. This course will start with defining spatial data science and answering why spatial is special from three different perspectives - business, technology, and data in the first week. In the second week, four disciplines related to spatial data science - GIS, DBMS, Data Analytics, and Big Data Systems, and the related open source software's - QGIS, PostgreSQL, PostGIS, R, and Hadoop tools are introduced together. During the third, fourth, and fifth weeks, you will learn the four disciplines one by one from the principle to applications. In the final week, five real world problems and the corresponding solutions are presented with step-by-step procedures in environment of open source software's.

從本節課中

Practical Applications of Spatial Data Science

The sixth module is entitled to "Practical Applications of Spatial Data Science", in which five real-world problems are introduced and corresponding solutions are presented with step-by-step procedures in the solution structures and related open source software's, discussed in Module 2. The first lecture presents an example of Desktop GIS, in which only QGIS is used, to find the top 5 counties for timberland investment in the southeastern states of the U.S, in which simple differencing of demand and supply is applied to figure out counties of large deficit of timber supply in comparison with timber demand. In the second lecture, an example of sever GIS, in which QGIS and PostgreSQL/PostGIS are used, will be presented as a solution for a given problem of NYC spatial data center, which required multiple user access and different levels of privileges. The third lecture presents an example of spatial data analytics, in which QGIS and R are used, to find out any regional factors which contribute to higher or lower disease prevalence in administrative districts, for which spatial autocorrelation analysis is conducted and decision tree analysis is applied. The fourth lecture is another example of spatial data analytics, to find optimal infiltration routing with network analysis, in which cost surface is produced and Dijkstra's algorithm is used. The fifth lecture is an example of spatial big data management and analytics, in which QGIS, PostGIS, R, and Hadoop MapReduce are all used, to provide a solution of "Passenger Finder", which can guide to the places where more passengers are waiting for taxi cabs. For the solution, spatial big data, taxi trajectory, are collected, and noise removal and map matching are conducted in Hadoop environment. Then, a series of spatial data processing and analysis such as spatial join in PostGIS, hotspot analysis in R are conducted in order to provide the solution. All in all, learners will realize the value of spatial big data and power of the solution structure with combination of four disciplines.

Spatial Big Data Management and Analytics - Taxi Trajectory Analysis for Finding Pick-up Hotspots14:36

與講師見面

Joon Heo
Professor
School of Civil and Environmental Engineering

Welcome back. This is the last lecture of the whole course,

Spatial Data Science and Applications.

In this lecture, you will study

spatial big data management and analytics using taxi trajectory data,

which is a typical example of spatial big data.

Assume that you have a client,

who is a taxi call-service company,

and they want to provide a new service to taxi driver named "Passenger Finder",

which can guide to the places where

more passengers are waiting for taxi cabs.

The solution would have

a private and public value, with which each taxi drivers would expect more incomes,

and the municipal government would be able to provide a better taxi service to citizens.

Some issues of the development include the size and the noise of taxi trajectory data,

and appropriate design of spatial data analysis to solve the problem.

We will take care of each issue one by one throughout this lecture.

For the given problem,

a proposed solution is summarized in flow chart.

The noisy taxi trajectory should be preprocessed,

for filtering outliers and map-matching to road network.

The drop-off/pick-up locations are spatially matched to the regions,

and time information should be properly saved.

Next, polygon-based hotspot analysis conducted for every hour of the week,

and finally deliver hotspot with respect to any given time of the week.

The solution would require the functionalities

of all the disciplines related to spatial data science.

So, it would be fit in a solutions structure of spatial big data management and analytics.

All the four disciplines would be used for implementation of the proposed solution.

Big data system will be cut out for filtering outliers and

map-matching of taxi trajectory data.

Spatial DBMS could save all the refined datasets of pick-ups and take-offs,

and basic spatial operations,

such as spatial join, can be conducted.

Data analysis tool can be used for hotspot analysis, and finally

the outcome of the solution can be visualized in GIS with a series of data processing.

Now, let's take a step-by-step procedures to

implement the proposed solution for "Passenger Finder" service.

The first step is to prepare the datasets for the problem.

Taxi trajectory, road network layer, and administrative district of project site,

which is Seoul in Korea.

The specification of taxi trajectory data is summarized in the slide,

which is given by company name Navicall,

which has more than one year time span collectively from 6,500 cabs,

and data size per day is about 400 megabytes and so on.

The attribute of taxi trajectory is described in the table,

which has 11 attributes including x, y

and time tag as a trajectory,

taxi status of empty/pick-up/drop-off

and hire, car ID, driver ID,

company ID and so on.

The road network layer around city of Seoul is presented here

and the administrative district of Seoul is presented.

The region polygon layer will be used for spatial unit of hotspot analysis.

Now, datasets are all ready and start data processing.

The first data processing is to filter out

outliers and to keep only trajectory data around Seoul.

The second preprocessing is to conduct map-matching the filtered trajectory.

Fortunately, the filtering and map-matching can be implemented in the distributed manner,

and the data size is rather big,

so, Hadoop MapReduce could be a good solution for the two preprocessing.

Taxi trajectory data is subject to noises and outliers as figure illustrates.

That would happen to most of big data,

so preprocessing is an inevitable steps for big data analysis.

For the given problem,

you will analyze the taxi trajectory only inside of Seoul

because "Passenger Finder" service will be presented to taxi drivers of Seoul.

In Korea, taxi drivers are registered to only one municipal district,

and they can run taxi businesses only within the registered district.

For the reason, trajectory outside of Seoul

would be meaningless to the target taxi drivers.

So, bounding box filtering is applied to the taxi trajectory data,

which you learned in big data systems in the fourth week.

The next preprocessing is map-matching,

which you learned in network analysis in the fifth week.

One quick question is what the value of map-matching for the problem would be?

Basically, it can improve the data accuracy

with locating every point to an appropriate map link,

and eventually, it can present a more analytic power.

For example, it can reveal driving directions of a trajectory,

traffic volume estimation of each link, and so on.

Map-matching algorithm were already discussed in the fifth week.

The difference here is that we will use Hadoop MapReduce.

For that, it would require spreading the whole trajectory dataset

into multiple subsets using Key-value pairs of MapReduce.

Then with respect to each subset,

a geometric map-matching will be conducted.

MapReduce code is given here for implementation of map-matching algorithm.

In the Map function, bounding box filtering and splitting

the entire dataset based on car ID, key, are conducted.

In the Reduce function,

point of curve geometric map-matching algorithm is conducted,

and the output of the map-matching is written to HDFS.

You are looking at the outcome of map-matching algorithm.

The first 11 columns are identical to

the original attributes of input trajectory data and 2 columns are added,

which are line segment ID,

which is result of map-matching

and distance between trajectory point and corresponding line segment.

Also note that pick-ups and drop-offs points are only selected because

pick-ups and drop-offs will be considered in hotspot analysis.

The outcome of two preprocessing in Hadoop MapReduce

will be converted and stored to spatial DBMS,

and spatial join will be conducted between the

drop-off/pick-up locations and administrative district regions.

The slide present a list of step-by-step procedures to conduct the data processing.

Please note that SQOOP is used for transferring

HDFS to PostgreSQL which you learned in Hadoop Ecosystem in the fourth week.

You are looking at the query of

group drop-off/pick-up by region in the time of interest.

From the query the number of drop-offs and pick-ups in the morning of November 26,

year 2016 are retrieved and exported to text file.

You are looking at the output text file of the query.

The first column denotes region ID

and the final column represent the number of drop-offs.

You are looking at a R code of hotspot analysis using Getis-Ord Gi* which is

a polygon-based hotspot analysis that you already learn in the fourth week.

For the visualization of the final result, QGIS is used.

After hotspot analysis completed in R, the output of Gi*, text file,

is joined to region shapefile and then it can be visualized in QGIS.

You are looking at outcome after GI* value and region shapefiles are joined by region ID.

Now, let's take a look at the visualization of

hotspot analysis with respect to specific period of time.

The first set of visualizations are from a regular day,

November 24, year 2016 Thursday.

The first outcome of hotspot analysis is pick-up hotspots during 06:00 am to 09:00 am.

The second outcome of hotspot analysis is drop-off hotspots and coldspot

during the same time,

06:00 am to 09:00 am.

The next outcome of hotspot analysis is pickup hotspots during 09:00 pm to midnight.

This is interesting outcome that drop-off in

the morning and pick-up in the night have very similar pattern.

The next outcome of a hotspot analysis it drop-off hotspots during the same time,

09:00 pm to midnight.

Many hotspots of drop-offs are located outskirt of Seoul.

Now, this time let's take a look at hotspot analysis on the day of Mass Protest Event,

which is November 26, year 2016 Saturday.

The outcome is pick-up hotspots and coldspots during 06:00 am to 09:00 am,

which is very different from regular day.

A lot of pick-ups took place at the downtown in red color,

where the mass protest was planned to occur in the evening.

The next outcome is drop-off hotspots and

coldspots during the same time 06:00 am to 09:00 am.

The downtown is coldspot drop-offs,

which collectively means that people are getting out of the town by taxi in the morning.

The next outcome is pick-up hotspots and coldspots during

03:00 pm to 06:00 pm just before the event starts.

The little pick-up took place at the downtown.

Likewise very little drop-off activity took place at the downtown as well.

Those makes sense, because most of the participant of

the event used public transit to access to downtown for protest.

These outcomes can be averaged with respect to the time of

the week and a few other parameters such as weather conditions.

The result could be compiled and then delivered to

taxi drivers through "Passenger Finder".

All right, you and I took a step-by-step

procedure to come up with a solution to the question,

where are more passengers waiting for taxi cab?

We have a solution structure and tangible outcomes to the question.

However, the proposed solution has some issues and limitations.

First of all, it would not fit in big data system because the data size is not that big.

Secondly, region-based hotspot analysis do not represent the best solution.

Probably the solution is somewhat basic.

For a realistic solution,

we should consider demand and supply together and competitions as well.

In this lecture, you studied the framework of

spatial big data management and analytics for taxi trajectory analysis,

in which we utilize Hadoop MapReduce, SQOOP, PostgreSQL,

PostGIS, R, and QGIS.

You could see the real value of

spatial big data to deal with real world problems.

All right, this is the end of this lecture at

the same time this is the end of this course,

Spatial Data Science and Applications.

I would like to congratulate you on your completion of this long course.

As an instructor, thank for your hard working to get here.

At the same time hope you could enjoy this course and

realize the value of spatial data and spatial big data,

not only public but also private purpose.

Again, I appreciate watching this course and wish you all the best. Bye now.

探索我們的目錄

免費加入並獲得個性化推薦、更新和優惠。

Coursera 致力於普及全世界最好的教育，它與全球一流大學和機構合作提供在線課程。

© 2018 Coursera Inc. 保留所有權利。

通過 App Store 下載

通過 Google Play 獲取