Technologies for Data Analytics Platform

Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /home/zhenxiangba/zhenxiangba.com/public_html/phproxy-improved-master/index.php on line 456
Technologies for Data Analytics Platform

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Technologies for
Data Analytics Platform
YAPC::Asia Tokyo 2015 - Aug 22, 2015

Who are you?
• Masahiro Nakagawa
• github: @repeatedly
• Treasure Data Inc.
• Fluentd / td-agent developer
• https://jobs....

Why do we analyze data?

Reporting
Monitoring
Exploratory data analysis
Conﬁrmatory data analysis
etc…

Need data, data, data!

It means we need
data analysis platform
for own requirements

Data Analytics Flow
Collect Store Process Visualize
Data source
Reporting
Monitoring

Let’s launch platform!

• Easy to use and maintain
• Single server
• RDBMS is popular and has huge ecosystem 
 
 
 
 
RDBMS
ETL Query
Extract + Tr...

×
Oops! RDBMS is not good for data
analytics against large data volume.
We need more speed and scalability!

Let’s consider
Parallel RDBMS instead!

Parallel RDBMS
• Optimized for OLAP workload
• Columnar storage, Shared nothing, etc…
• Netezza, Teradata, Vertica, Greenp...

time code method
2015-12-01 10:02:36 200 GET
2015-12-01 10:22:09 404 GET
2015-12-01 10:36:45 200 GET
2015-12-01 10:49:21 2...

Okay, query is now
processed normally. 
L
C C C

No silver bullet
• Performance depends on data modeling and query
• distkey and sortkey are important
• should reduce data...

Performance is good :)
But we often want to change schema 
for new workloads. Now, 
hard to maintain schema and its data…
...

Okay, let’s separate
data sources into multiple
layers for reliable platform

Schema on Write(RDBMS)
• Writing data using schema 
for improving query performance
• Pros:
• minimum query overhead
• Con...

Schema on Read(Hadoop)
• Writing data without schema and 
map schema at query time
• Pros:
• Robust over schema and worklo...

Data Lake
• Schema management is hard
• Volume is increasing and format is often changed
• There are lots of log types
• F...

Data Lake Patterns
• Use DFS, e.g. HDFS, for log storage
• ETL or data processing by Hadoop ecosystem
• Can convert logs v...

Apache Hadoop
• Distributed computing framework
• First implementation based on Google MapReduce 
 
 
 
 
 
http://hortonw...

HDFS
http://nosqlessentials.com/

MapReduce

Cool!
Data load becomes robust!
EL
T
Raw data Transformed data

Apache Tez
• Low level framework for YARN Applications
• Hive, Pig, new query engine and more
• Task and DAG based process...

MapReduce vs Tez
MapReduce Tez
M
HDFS
R
R
M M
HDFS HDFS
R
M M
R
M M
R
M
R
M MM
M M
R
R
R
SELECT g1.x, g2.avg, g2.cnt 
FROM...

Superstition
• HDFS and YARN have SPOF
• Recent version doesn’t have SPOF on both
MapReduce 1 and MapReduce 2
• Can’t buil...

Which Hadoop package 
should we use?
• Distribution by Hadoop distributor is better
• CDH by Cloudera
• HDP by Hortonworks...

Good :)
In addition, we want to
collect data in efﬁcient way!

Ingestion tools
• There are two execution model!
• Bulk load:
• For high-throughput
• Almost tools transfer data in batch ...

Bulk load tools
• Embulk
• Pluggable bulk data loader for 
various inputs and outputs
• Write plugins using Java and JRuby...

Streaming load tools
• Fluentd
• Pluggable and json based streaming collector
• Lots of plugins in rubygems
• Flume
• Main...

Data ingestion also 
becomes robust and efﬁcient!
Raw data Transformed data

It works! but… 
we want to issue ad-hoc query to entire
data.
We can’t wait loading data into database.

You can use MPP query
engine for data stores.

MPP query engine
• It doesn’t have own storage unlike parallel RDBMS
• Follow “Schema on Read” approach
• data distributio...

• Distributed Query Engine for interactive queries 
against various data sources and large data.
• Pluggable connector for...

HDFS
Hive
PostgreSQL, etc.
Daily/Hourly Batch
Interactive query
Commercial 
BI Tools
Batch analysis platform Visualization...

HDFS
Hive
Daily/Hourly Batch
Interactive query
✓ Less scalable
✓ Extra cost
Commercial 
BI Tools
Dashboard
✓ More work to ...

HDFS
Hive Dashboard
Presto
PostgreSQL, etc.
Daily/Hourly Batch
HDFS
Hive
Dashboard
Daily/Hourly Batch
Interactive query
In...

Presto
HDFS
Hive
Dashboard
Daily/Hourly Batch
Interactive query
Cassandra MySQL Commertial DBs
SQL on any data sets Commer...

Client
Coordinator Connector 
Plugin
Worker
Worker
Worker
Storage / Metadata
Discovery Service

Execution Model
All stages are pipe-lined
✓ No wait time
✓ No fault-tolerance
MapReduce Presto
map map
reduce reduce
task ...

Okay, we have now low latency
and batch combination.
Raw data

Resolved our concern! But…
we also need quick estimation.

Currently, we have several
stream processing softwares.
Let’s try!!

Apache Storm
• Distributed realtime processing framework
• Low latency: tuple at a time
• Trident mode uses micro batch 
 ...

Norikra
• Schema-less CEP engine for stream processing
• Use SQL like Esper EPL
• Not distributed unlike Storm for now 
 
...

Great! We can get insight by
streaming and batch way :)

One more. We can make data
transfer more reliable for multiple
data streams with distributed queue

• Distributed messaging system
• Producer - Broker - Consumer pattern
• Pull model, replication, etc… 
 
 
 
 
Apache Kafk...

Push vs Pull
• Push:
• Easy to transfer data to multiple destinations
• Hard to control stream ratio in multiple streams
•...

This is a modern analytics platform

Seems complex and hard to
maintain?
Let’s use useful services!

Amazon Redshift
• Parallel RDBMS on AWS
• Re-use traditional Parallel RDMBS know-how
• Scale is easier than traditional sy...

Using AWS Services

Google BigQuery
• Distributed query engine and scalable storage
• Tree model, Columnar storage, etc…
• Separate storage fr...

BigQuery architecture

Using GCP Services

Treasure Data
• Cloud based end-to-end data analytics service
• Hive, Presto, Pig and Hivemall for one big repository
• Lo...

Using Treasure Data Service

Resource Model Trade-off
Pros Cons
Fully Guaranteed
Stable execution
Easy to control resource
Non boost mechanizm
Guarante...

MS Azure also has useful services:
DataHub, SQL DWH, DataLake,
Stream Analytics, HDInsight…

Use service or build a platform?
• Should consider using service ﬁrst
• AWS, GCP, MS Azure, Treasure Data, etc…
• Importan...

Conclusion
• Many softwares and services for data analytics
• Lots of trade-off, performance, complexity,
connectivity, ex...

Cloud service for
entire data pipeline!

Appendix

Apache Spark
• Another Distributed computing framework
• Mainly for in-memory computing with DAG
• RDD and DataFrame based...

Apache Flink
• Streaming based execution engine
• Support batch and pipelined processing
• Hadoop and Spark are batch base...

Batch vs Pipelined
All stages are pipe-lined
✓ No wait time
✓ fault-tolerance with 
check pointing
Batch(Staged) Pipelined...

Visualization
• Tableau
• Popular BI tool in many area
• Awesome GUI, easy to use, lots of charts, etc
• Metric Insights
•...

How to manage job dependency?
We want to issue Job X
after Job A and Job B are ﬁnished.

Data pipeline tool
• There are some important features
• Manage job dependency
• Handle job failure and retry
• Easy to de...

Luigi
• Python module for building job pipeline
• Write python code and run it.
• task is deﬁned as Python class
• Easy to...

Airflow
• Python and DAG based workflow
• Write python code but it is for defining ADAG
• Task is defined by Operator
• There ...