122 lines
4.7 KiB
HTML
122 lines
4.7 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<title>Big Data With Hadoop</title>
|
|
<meta charset="utf-8">
|
|
<meta name="generator" content="S5">
|
|
<meta name="version" content="S5 1.1">
|
|
<meta name="presdate" content="20140418">
|
|
<meta name="author" content="Zack Zatkin-Gold">
|
|
<meta name="company" content="Rochester Institute of Technology">
|
|
<!-- configuration parameters -->
|
|
<meta name="defaultView" content="slideshow">
|
|
<meta name="controlVis" content="hidden">
|
|
<!-- style sheet links -->
|
|
<link rel="stylesheet" href="ui/hadoop/slides.css" type="text/css" media="projection" id="slideProj">
|
|
<link rel="stylesheet" href="ui/hadoop/outline.css" type="text/css" media="screen" id="outlineStyle">
|
|
<link rel="stylesheet" href="ui/hadoop/print.css" type="text/css" media="print" id="slidePrint">
|
|
<link rel="stylesheet" href="ui/hadoop/opera.css" type="text/css" media="projection" id="operaFix">
|
|
<!-- S5 JS -->
|
|
<script src="ui/hadoop/slides.js" type="text/javascript"></script>
|
|
</head>
|
|
<body>
|
|
|
|
<div class="layout">
|
|
<div id="controls"><!-- DO NOT EDIT --></div>
|
|
<div id="currentSlide"><!-- DO NOT EDIT --></div>
|
|
<div id="header"></div>
|
|
<div id="footer">
|
|
<h1>Rochester Institute of Technology / 2014-04-18</h1>
|
|
<h2>Big Data With Hadoop</h2>
|
|
</div>
|
|
</div>
|
|
<div class="presentation">
|
|
<div class="slide">
|
|
<h1>Big Data With Hadoop</h1>
|
|
<h3>Zack Zatkin-Gold</h3>
|
|
<h4>Rochester Institute of Technology</h4>
|
|
</div>
|
|
<div class="slide">
|
|
<h1>What is Big Data?</h1>
|
|
<ul class="incremental">
|
|
<li>"... used to describe a massive volume of both structured and unstructured data that is so large that it's difficult to process using traditional database and software techniques."</li>
|
|
<li>Data sets grow to this size from various sources: mobile device location, meta data from photos, social network accounts, software logs, and more.</li>
|
|
<li>Historically, before the 1990s, nothing could handle Big Data efficiently.</li>
|
|
</ul>
|
|
<div class="handout">
|
|
Sources: Wikipedia
|
|
</div>
|
|
</div>
|
|
<div class="slide">
|
|
<h1>What about RDBMS?</h1>
|
|
<ul class="incremental">
|
|
<li>Relational database management systems (MySQL, PostgreSQL, Oracle, etc.) are great, but not for Big Data</li>
|
|
<li>Sacrificing performance, <!--loosening ACID restrictions, -->making distribution more difficult, reliance on a primary master copy
|
|
<!--<ul>
|
|
<li>Atomicity - "All or nothing"</li>
|
|
<li>Consistency - Ensuring to bring one state to the next</li>
|
|
<li>Isolation - One transaction after another</li>
|
|
<li>Durability - Ensure all transactions occur</li>
|
|
</ul>-->
|
|
</li>
|
|
<li>Distributed data is heavy on bandwidth (through syncing)</li>
|
|
<li>New features results in bigger queries or table restructuring</li>
|
|
<li>Any form of slave replication or external caching introduces weak consistency</li>
|
|
</ul>
|
|
<div class="handout">
|
|
</div>
|
|
</div>
|
|
<div class="slide">
|
|
<h1>In comes Hadoop</h1>
|
|
<ul class="incremental">
|
|
<li>Hadoop is a collection of tools that allows for the distributed processing of large data sets across clusters of commodity hardware using simple programming models.</li>
|
|
<li>This means that Hadoop can work with Big Data on clusters of cheap hardware.</li>
|
|
<li>Some of the technologies within Hadoop consist of
|
|
<ul>
|
|
<li>MapReduce - A distributed data processing model and execution environment</li>
|
|
<li>Hadoop Distributed File System (HDFS) - A distributed filesystem</li>
|
|
<li>HBase - A distributed, column-oriented database that uses HDFS for its underlying storage</li>
|
|
<li>Hive - A distributed data warehouse. Manages data stored in HDFS and provides a query language (based on SQL)</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<div class="handout">
|
|
</div>
|
|
</div>
|
|
<div class="slide">
|
|
<h1>What're the secrets?</h1>
|
|
<ul class="incremental">
|
|
<li>WORM (write once, read many) - Once it's written, it cannot be modified</li>
|
|
<li>Unstructured data - no schema for how the data is organized</li>
|
|
<li>Google MapReduce - a programming model for processing large data sets with a parallel, distributed algorithm on a cluster</li>
|
|
<li>Automatic partitioning - As your tables grow, they're distributed evenly</li>
|
|
<li>Fault tolerance - Lots of nodes means each is relatively insignificant</li>
|
|
<li>Scale linearly - Add a node and RegionServer will automatically rebalance, distributing the load evenly</li>
|
|
</ul>
|
|
<div class="handout">
|
|
</div>
|
|
</div>
|
|
<div class="slide">
|
|
<h1>Some companies using Hadoop</h1>
|
|
<ul>
|
|
<li>eBay - ~5.3 petabytes (~5,300 TB)</li>
|
|
<li>Facebook - ~12 petabytes</li>
|
|
<li>Last.fm - ~800 terabytes</li>
|
|
<li>LinkedIn - ~16.8 petabytes</li>
|
|
<li>New York Times</li>
|
|
<li>Rackspace</li>
|
|
<li>Spotify - ~28 petabytes</li>
|
|
<li>...</li>
|
|
</ul>
|
|
<div class="handout">
|
|
</div>
|
|
</div>
|
|
<div class="slide">
|
|
<h1>Questions?</h1>
|
|
<ul>
|
|
<div class="handout">
|
|
</div>
|
|
</div>
|
|
</body>
|
|
</html>
|