NoSQL at Twitter (NoSQL EU 2010)

Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /home/zhenxiangba/zhenxiangba.com/public_html/phproxy-improved-master/index.php on line 456
NoSQL at Twitter (NoSQL EU 2010)

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

NoSQL at Twitter
Kevin Weil -- @kevinweil
Analytics Lead, Twitter

April 21, 2010

TM

Introduction
‣ How We Arrived at NoSQL: A Crash Course
‣ Collecting Data (Scribe)
‣ Storing and Analyzing Data (...

My Background
‣ Studied Mathematics and Physics at Harvard, Physics at
Stanford
‣ Tropos Networks (city-wide wirel...

Introduction
‣ How We Arrived at NoSQL: A Crash Course
‣ Collecting Data (Scribe)
‣ Storing and Analyzing Data (...

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?
‣ 7 TB/day (2+ PB/yr)

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?
‣ 7 TB/day (2+ PB/yr)
‣ ...

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?
‣ 7 TB/day (2+ PB/yr)
‣ ...

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?
‣ 7 TB/day (2+ PB/yr)
‣ ...

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?
‣ 7 TB/day (2+ PB/yr)
‣ ...

Syslog?
‣ Started with syslog-ng
‣ As our volume grew, it didn’t scale

Syslog?
‣ Started with syslog-ng
‣ As our volume grew, it didn’t scale
‣ Resources overwhelmed
‣ Lost data

Scribe
‣ Surprise! FB had same problem, built and open-sourced Scribe
‣ Log collection framework over Thrift
‣ You w...

Scribe
‣ Runs locally; reliable in network outage

FE FE FE

Scribe
‣ Runs locally; reliable in network outage
‣ Nodes only know downstream
...

Scribe
‣ Runs locally; reliable in network outage
‣ Nodes only know downstream
...

Scribe at Twitter
‣ Solved our problem, opened new vistas
‣ Currently 30 different categories logged from multiple sou...

Scribe at Twitter
‣ We’ve contributed to it as we’ve used it
‣ Improved logging, monitoring, writing to HDFS, comp...

Introduction
‣ How We Arrived at NoSQL: A Crash Course
‣ Collecting Data (Scribe)
‣ Storing and Analyzing Data (...

How do you store 7TB/day?
‣ Single machine?
‣ What’s HD write speed?

How do you store 7TB/day?
‣ Single machine?
‣ What’s HD write speed?
‣ ~80 MB/s

How do you store 7TB/day?
‣ Single machine?
‣ What’s HD write speed?
‣ ~80 MB/s
‣ 24.3 hours to write 7 TB

How do you store 7TB/day?
‣ Single machine?
‣ What’s HD write speed?
‣ ~80 MB/s
‣ 24.3 hours to write 7 TB
‣ U...

Where do I put 7TB/day?
‣ Need a cluster of machines

Where do I put 7TB/day?
‣ Need a cluster of machines

‣ ... which adds new layers of complexity

Hadoop
‣ Distributed file system
‣ Automatic replication, fault tolerance

Hadoop
‣ Distributed file system
‣ Automatic replication, fault tolerance
‣ MapReduce-based parallel computation
‣...

Hadoop
‣ Open source: top-level Apache project
‣ Scalable: Y! has a 4000 node cluster
‣ Powerful: sorted 1TB of rand...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

Two Analysis Challenges
‣ 1. Compute friendships in Twitter’s social graph
‣ grep, awk? No way.
‣ Data is in MyS...

Two Analysis Challenges
‣ 1. Compute friendships in Twitter’s social graph
‣ grep, awk? No way.
‣ Data is in MyS...

Two Analysis Challenges
‣ 2. Large-scale grouping and counting?
‣ select count(*) from users? Maybe...
‣ select co...

Back to Hadoop
‣ Didn’t we have a cluster of machines?

Back to Hadoop
‣ Didn’t we have a cluster of machines?

Back to Hadoop
‣ Didn’t we have a cluster of machines?
‣ Hadoop makes it easy to distribute the
calculation
‣ Pu...

Back to Hadoop
‣ Didn’t we have a cluster of machines?
‣ Hadoop makes it easy to distribute the
calculation
‣ Pu...

Analysis at scale
‣ Now we’re rolling
‣ Count all tweets: 12 billion, 5 minutes
‣ Hit FlockDB in parallel to assembl...

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”
‣ Single-input, two-stage data flow...

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”
‣ Single-input, two-stage data flow...

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”
‣ Single-input, two-stage data flow...

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”
‣ Single-input, two-stage data flow...

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”
‣ Single-input, two-stage data flow...

Introduction
‣ How We Arrived at NoSQL: A Crash Course
‣ Collecting Data (Scribe)
‣ Storing and Analyzing Data (...

Pig
‣ High-level language
‣ Transformations on sets of records
‣ Process data one step at a time
‣ Easier than SQL?

Why Pig?
‣ Because I bet you can read the following script.

A Real Pig Script

A Real Pig Script

‣ Now, just for fun... the same calculation in vanilla Hadoop MapReduce.

No, seriously.

Pig Democratizes Large-scale Data
Analysis
‣ The Pig version is:
‣ 5% of the code

Pig Democratizes Large-scale Data
Analysis
‣ The Pig version is:
‣ 5% of the code
‣ 5% of the time

Pig Democratizes Large-scale Data
Analysis
‣ The Pig version is:
‣ 5% of the code
‣ 5% of the time
‣ Within ...

One Thing I’ve Learned
‣ It’s easy to answer questions
‣ It’s hard to ask the right questions

One Thing I’ve Learned
‣ It’s easy to answer questions
‣ It’s hard to ask the right questions

‣ Value the system t...

One Thing I’ve Learned
‣ It’s easy to answer questions
‣ It’s hard to ask the right questions

‣ Value the system t...

The Hadoop Ecosystem at Twitter
‣ Running Cloudera’s free distro, CDH2 and Hadoop 0.20.1

The Hadoop Ecosystem at Twitter
‣ Running Cloudera’s free distro, CDH2 and Hadoop 0.20.1
‣ Heavily modified Scribe wri...

The Hadoop Ecosystem at Twitter
‣ Running Cloudera’s free distro, CDH2 and Hadoop 0.20.1
‣ Heavily modified Scribe wri...

The Hadoop Ecosystem at Twitter
‣ Running Cloudera’s free distro, CDH2 and Hadoop 0.20.1
‣ Heavily modified Scribe wri...

The Hadoop Ecosystem at Twitter
‣ Running Cloudera’s free distro, CDH2 and Hadoop 0.20.1
‣ Heavily modified Scribe wri...

Data?
‣ Semi-structured: apache logs
(search, .com, mobile), search query
logs, RoR logs, mysql query logs, A/B
...

Data?
‣ Semi-structured: apache logs
(search, .com, mobile), search query
logs, RoR logs, mysql query logs, A/B
...

Data?
‣ Semi-structured: apache logs
(search, .com, mobile), search query
logs, RoR logs, mysql query logs, A/B
...

So what do we do with it?

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?
‣ What ...

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?
‣ What ...

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?
‣ What ...

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?
‣ What ...

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?
‣ What...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ What is the correlation between users with registered phones
and users that tweet?

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features get ...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features get ...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features get ...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features get ...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features get ...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features ge...

Research on Big Data
‣ How well can we detect bots and other non-human tweeters?

Introduction
‣ How We Arrived at NoSQL: A Crash Course
‣ Collecting Data (Scribe)
‣ Storing and Analyzing Data (...

HBase
‣ BigTable clone on top of HDFS
‣ Distributed, column-oriented, no datatypes
‣ Unlike the rest of HDFS, design...

HBase at Twitter
‣ We began building real products based on Hadoop
‣ People search

HBase at Twitter
‣ We began building real products based on Hadoop
‣ People search
‣ Old version: offline process ...

HBase at Twitter
‣ We began building real products based on Hadoop
‣ People search
‣ Old version: offline process ...

HBase at Twitter
‣ We began building real products based on Hadoop
‣ People search
‣ Old version: offline process ...

People Search
‣ Import user data into HBase

People Search
‣ Import user data into HBase
‣ Periodic MapReduce job reading from HBase
‣ Hits FlockDB, multiple o...

People Search
‣ Import user data into HBase
‣ Periodic MapReduce job reading from HBase
‣ Hits FlockDB, multiple o...

People Search
‣ Import user data into HBase
‣ Periodic MapReduce job reading from HBase
‣ Hits FlockDB, multiple o...

HBase
‣ More products now being built on top of it
‣ Flexible, easy to connect to MapReduce/Pig

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”
‣ HBase built on top of batch-oriented syst...

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”
‣ HBase built on top of batch-oriented syst...

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”
‣ HBase built on top of batch-oriented syst...

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”
‣ HBase built on top of batch-oriented syst...

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”
‣ HBase built on top of batch-oriented syst...

HBase vs Cassandra
‣ Your mileage may vary
‣ At Twitter: HBase for analytics, analysis, dataset generation
‣ Cas...

HBase vs Cassandra
‣ Your mileage may vary
‣ At Twitter: HBase for analytics, analysis, dataset generation
‣ Cas...

FlockDB
‣ Realtime, distributed
social graph store
‣ NOT optimized for data mining

‣ Note: the following slide...

FlockDB
‣ Realtime, distributed Intersection
Temporal

soci...

Set operations?
‣ This tweet needs to
be delivered to people
who follow both
@aplusk (4.7M
followers) an...

Original solution
‣ MySQL table source_id destination-id

‣ Indices on source_id
...

Next Try
‣ MySQL still
‣ Denormalized
‣ Byte-packed
‣ Chunked
‣ Still temporally ordered

Next Try
‣ Problems
‣ O(n) deletes
‣ Data consistency challenges
‣ Inefficient intersections
‣ All of thes...

FlockDB
‣ MySQL underneath still (like PNUTS from Y!)
‣ Partitioned by user_id, gizzard handles sharding/partitioning
...

FlockDB Timings
‣ Counts: 1ms

FlockDB Timings
‣ Counts: 1ms
‣ Temporal Query: 2ms

FlockDB Timings
‣ Counts: 1ms
‣ Temporal Query: 2ms
‣ Writes: 1ms for journal, 16ms for durability

FlockDB Timings
‣ Counts: 1ms
‣ Temporal Query: 2ms
‣ Writes: 1ms for journal, 16ms for durability
‣ Full walks: 1...

FlockDB is Open Source
‣ We will maintain a community at
‣ http://www.github.com/twitter/flockdb
‣ http://...

Cassandra
‣ Why Cassandra, for Twitter?

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL
‣ All kinds of...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL
‣ All kinds of...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL
‣ All kinds of...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL
‣ All kinds of...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL
‣ All kinds of...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Decentralized, fault-tolerant
‣ All kinds of caching layers, all appli...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Decentralized, fault-tolerant
‣ All kinds of caching layers, all appli...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Decentralized, fault-tolerant
‣ All kinds of caching layers, all appli...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Decentralized, fault-tolerant
‣ All kinds of caching layers, all appli...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Decentralized, fault-tolerant
‣ All kinds of caching layers, all appli...

Eventually Consistent?
‣ Twitter is already eventually consistent

Eventually Consistent?
‣ Twitter is already eventually consistent
‣ Your system may be even worse

Eventually Consistent?
‣ Twitter is already eventually consistent
‣ Your system may be even worse
‣ Ryan’s new ter...

Eventually Consistent?
‣ Twitter is already eventually consistent
‣ Your system may be even worse
‣ Ryan’s new ter...

Eventually Consistent?
‣ Twitter is already eventually consistent
‣ Your system may be even worse
‣ Ryan’s new ter...

Rolling out Cassandra
‣ 1. Integrate Cassandra alongside MySQL
‣ 100% reads/writes to MySQL
‣ Dynamic switches f...

Rolling out Cassandra
‣ 1. Integrate Cassandra alongside MySQL
‣ 100% reads/writes to MySQL
‣ Dynamic switches f...

Rolling out Cassandra
‣ 1. Integrate Cassandra alongside MySQL
‣ 100% reads/writes to MySQL
‣ Dynamic switches f...

Rolling out Cassandra
‣ 1. Integrate Cassandra alongside MySQL
‣ 100% reads/writes to MySQL
‣ Dynamic switches...

Rolling out Cassandra
‣ 1. Integrate Cassandra alongside MySQL
‣ 100% reads/writes to MySQL
‣ Dynamic switches...

Cassandra for Realtime Analytics
‣ Starting a project around realtime analytics
‣ Cassandra as the backing store
‣ ...

That was a lot of slides
‣ Thanks for sticking with me.

Questions? Follow me at
twitter.com/kevinweil

TM