Treasure Data and AWS - Developers.io 2015

Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /home/zhenxiangba/zhenxiangba.com/public_html/phproxy-improved-master/index.php on line 456
Treasure Data and AWS - Developers.io 2015

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Masahiro Nakagawa
Senior Software Engineer
Treasure Data, inc.
Treasure Data & AWS
The light and dark side of the Cloud

Who am I
> Masahiro Nakagawa
> github: repeatedly
> Treasure Data, Inc.
> Senior Software Engineer
> Fluentd / td-agent de...

TD Service Architecture
Time to Value
Send query result
Result Push
Acquire
Analyze
Store
Plazma DB
Flexible, Scalable,
...

Treasure Data System Overview
Frontend
Job Queue
Worker
Hadoop
Presto
Fluentd
Applications push
metrics to Fluentd 
(via l...

Plazma - Treasure Data’s distributed
analytical database

Plazma by the numbers
> Data import
> 500,000 records / sec
> 43 billion records / day
> Hive Query
> 2 trillion records /...

Used AWS components
> EC2
> Hadoop / Presto Clusters
> API Servers
> S3
> MessagePack Columnar Storage
> RDS
> MySQL for s...

Used AWS components
> CloudWatch
> Monitor AWS service metrics
> ELB
> Endpoint for APIs
> Endpoint for Heroku drains
> El...

Why not use HDFS for storage?
> To separate machine resource and storage
> Easy to add or replace workers
> Import load do...

Data Importing

Import
Queue
td-agent
/ ﬂuentd
Import
Worker
✓ Buffering for 
5 minute
✓ Retrying 
(at-least once)
✓ On-disk buffering
on ...

Import
Queue
td-agent
/ ﬂuentd
Import
Worker
✓ Buffering for 
1 minute
✓ Retrying 
(at-least once)
✓ On-disk buffering
on ...

Import
Queue
td-agent
/ ﬂuentd
Import
Worker
✓ Buffering for 
5 minute
✓ Retrying 
(at-least once)
✓ On-disk buffering
on ...

Import
Queue
Import
Worker
Import
Worker
Import
Worker
✓ HA
✓ Load balancing

Realtime
Storage
PostgreSQL
Amazon S3 /
Basho Riak CS
Metadata
Import
Queue
Import
Worker
Import
Worker
Import
Worker
Arch...

Realtime
Storage
PostgreSQL
Amazon S3 /
Basho Riak CS
Metadata
Import
Queue
Import
Worker
Import
Worker
Import
Worker
uplo...

Amazon S3 /
Basho Riak CS
Metadata
Merge Worker 
(MapReduce)
uploaded time ﬁle index range records
2015-03-08 10:47
[2015-...

Amazon S3 /
Basho Riak CS
Metadata
uploaded time ﬁle index range records
2015-03-08 10:47
[2015-12-01 10:47:11, 
2015-12-0...

Why not use LIST API?
> LIST API is slow
> It causes slow query on large dataset
> Riak CS’s LIST is also toooo slow!
> LI...

Why not MySQL? - benchmark
0
45
90
135
180
INSERT 50,000 rows SELECT sum(id) SELECT sum(ﬁle_size) WHERE index range
0.656....

Data Importing
> Scalable & Reliable importing
> Fluentd buffers data on a disk
> Import queue deduplicates uploaded chunk...

Data processing

time code method
2015-12-01 10:02:36 200 GET
2015-12-01 10:22:09 404 GET
2015-12-01 10:36:45 200 GET
2015-12-01 10:49:21 2...

time code method
2015-12-01 10:02:36 200 GET
2015-12-01 10:22:09 404 GET
2015-12-01 10:36:45 200 GET
2015-12-01 10:49:21 2...

time code method
2015-12-01 10:02:36 200 GET
2015-12-01 10:22:09 404 GET
2015-12-01 10:36:45 200 GET
2015-12-01 10:49:21 2...

Handling Eventual Consistency
1. Writing data / metadata ﬁrst
> At this time, data is not visible
2. Check S3 data is avai...

Hide network cost
> Open a lot of connections to S3
> Using range feature with columnar offset
> Improve scan performance ...

buffer
Optimizing Scan Performance
•  Fully utilize the network bandwidth from S3
•  TD Presto becomes CPU bottleneck
8
Ta...

Recoverable errors
> Error types
> User error
> Syntax error, Semantic error
> Insufﬁcient resource
> Exceeded task memory...

Recoverable errors
> Error types
> User error
> Syntax error, Semantic error
> Insufﬁcient resource
> Exceeded task memory...

Presto retry on Internal Errors
> Query succeed eventually 
 
 
 
 
 
 
 
log scale

time code method
2015-12-01 10:02:36 200 GET
2015-12-01 10:22:09 404 GET
2015-12-01 10:36:45 200 GET
2015-12-01 10:49:21 2...

time code method
2015-12-01 10:02:36 200 GET
2015-12-01 10:22:09 404 GET
2015-12-01 10:36:45 200 GET
2015-12-01 10:49:21 2...

Realtime
Storage
Query Engine 
Hive, Pig, Presto
Archive
Storage
{“user”:54, “name”:”plazma”, “value”:”120”, “host”:”local...

Realtime
Storage
Query Engine 
Hive, Pig, Presto
Archive
Storage
Schema-full
Schema-less
Schema
{“user”:54, “name”:”plazma...

Realtime
Storage
Query Engine 
Hive, Pig, Presto
Archive
Storage
{“user”:54, “name”:”plazma”, “value”:”120”, “host”:”local...

Monitoring

Datadog based monitoring
> dd-agent for system metrics
> Send application metrics using Fluentd
> Hadoop / Presto usage
> ...

CloudTrail example

Presto example

Pitfall of PostgreSQL on RDS
> PostgreSQL on RDS has TCP Proxy
> “DB connections” metrics shows TCP connections, 
not exec...

Conclusion
> Build scalable data analytics platform on Cloud
> Separate resource and storage
> loosely-coupled components
...

Check: treasuredata.com 
treasure-data.hateblo.jp/ (Japan blog)
Cloud service for the entire data pipeline