Big Data

• Apache
  • Zeppelin Docker
    • Docker
    • Create new note
    • Screenshots
    • References
  • Flink Docker
    • Docker
    • Docker Compose
    • Screenshots
    • References
• Cloudera
  • Cloudera Manager Docker
    • Docker
    • HDFS
    • Hive
    • Screenshots
    • References
• Flume
  • Flume Introduction
    • What is Flume?
    • Architecture
    • Data Flow
    • References
  • Flume Environment
    • Installing Flume
    • Configuring Flume
    • Verifying the Installation
    • References
  • Flume Configuration
    • Naming the Components
    • Describing the Source
    • Describing the Sink
    • Describing the Channel
    • Binding the Source and the Sink to the Channel
    • Starting a Flume Agent
    • References
  • Flume Sequence Generator Source
    • Prerequisites
    • Configuring Flume
    • Example – Configuration File
    • Execution
    • Verifying the HDFS
    • Verifying the Contents of the File
    • Results
    • References
  • Flume NetCat Source
    • Prerequisites
    • Configuring Flume
    • Example Configuration File
    • Execution
    • References
  • Flume Fetching Twitter Data
    • Creating a Twitter Application
    • Starting HDFS
    • Configuring Flume
    • Example – Configuration File
    • Execution
    • Verifying HDFS
    • Tips
    • References
• Hadoop
  • Enviornment Setup
    • Creating a User
    • SSH Setup and Key Generation
    • Downloading Hadoop
    • Installing Hadoop in Standalone Mode
    • Installing Hadoop in Pseudo Distributed Mode
    • Verifying Hadoop Installation
    • References
  • HDFS
    • Features of HDFS
    • Where to use HDFS
    • Where not to use HDFS
    • HDFS Architecture
    • Starting HDFS
    • Listing Files in HDFS
    • Inserting Data into HDFS
    • Retrieving Data from HDFS
    • Shutting Down the HDFS
    • HDFS Basic File Operations
    • HDFS Other commands
    • References
  • Command Reference
    • References
  • MapReduce
    • The Algorithm
    • Inputs and Outputs (Java Perspective)
    • Terminology
    • Example Scenario
    • Compilation and Execution of Process Units Program
    • Important Commands
    • How to Interact with MapReduce Jobs
    • Word Count Example
    • MapReduce Char Count Example
    • References
  • Streaming
    • Hadoop Streaming
    • Streaming Command Options
    • References
  • Multi-Node Cluster
    • Installing Java
    • Creating User Account
    • Mapping the nodes
    • Configuring Key Based Login
    • Installing Hadoop
    • Configuring Hadoop
    • Installing Hadoop on Slave Servers
    • Configuring Hadoop on Master Server
    • Starting Hadoop Services
    • Results
    • Adding a New DataNode in the Hadoop Cluster
    • Adding User and SSH Access
    • Set Hostname of New Node
    • Start the DataNode on New Node
    • Removing a DataNode from the Hadoop Cluster
    • References
  • YARN
    • Architecture
    • Components Of YARN
    • Benefits of YARN
    • References
  • Azkaban Docker
    • Build
    • Installing the Solo Server
    • Installing Azkaban Executor Server
    • Installing Azkaban Web Server
    • Docker Compose
    • Screenshots
    • References
• HBase
  • HBase Overview
    • Why HBase
    • HBase Data Model
    • HBase Commands
    • HBase and HDFS
    • HBase and RDBMS
    • References
  • HBase Architecture
    • Region server
    • References
  • HBase Installation
    • Pre-Installation Setup
    • Installing HBase
    • References
  • HBase Shell
    • General Commands
    • Data Definition Language
    • Data Manipulation Language
    • Starting HBase Shell
    • Results
    • References
  • HBase Admin API
    • Class HBaseAdmin
    • Class Descriptor
    • Shutting Down
    • References
  • HBase Table
    • Create Table
    • Listing Table
    • Disabling a Table
    • Enabling a Table
    • Describe & Alter
    • Exists
    • Drop a Table
    • References
  • HBase Client API
    • Class HBase Configuration
    • Class HTable
    • Class Put
    • Class Get
    • Class Delete
    • Class Result
  • HBase Data
    • Create Data
    • Update Data
    • Read Data
    • Delete Data
    • Scan
    • Count & Truncate
    • Security
    • References
• Hive
  • Hive Introduction
    • What is Hive
    • Features of Hive
    • Architecture of Hive
    • Working of Hive
    • References
  • Hive Installation
    • Pre-Installation Setup
    • Installing Hive
    • Configuring Hive
    • Downloading and Installing Apache Derby
    • Configuring Metastore of Hive
    • Verifying Hive Installation
    • Tips
    • References
  • Hive Data Types
    • Column Types
    • Literals
    • Null Value
    • Complex Types
    • References
  • Hive Database
    • Create Database
    • Drop Database
    • References
  • Hive Table
    • Create Table
    • Alter Table
    • Tips
    • Drop Table
    • References
  • Hive Partitioning
    • Adding a Partition
    • Renaming a Partition
    • Dropping a Partition
    • References
  • Hive Built-in Operators
    • Relational Operators
    • Arithmetic Operators
    • Logical Operators
    • Complex Operators
    • References
  • Hive Built-in Functions
    • Built-In Functions
    • Aggregate Functions
    • References
  • Hive View and Indexes
    • Creating a View
    • Dropping a View
    • Creating an Index
    • Dropping an Index
    • References
  • HiveQL
    • Select-Where
    • Select-Order By
    • Select-Group By
    • Select-Joins
    • References
  • Hive vs. HBase
    • Summary
    • NoSQL and Big Data Processing
    • References
• Impala
  • Impala Overview
    • What is Impala?
    • Relational Databases and Impala
    • Hive, Hbase, and Impala
    • References
  • Impala Environment
    • Tips
    • References
• Pig
  • Pig Overview
    • Why Do We Need Apache Pig?
    • Features of Pig
    • Pig Vs MapReduce
    • Pig Vs SQL
    • Pig Vs Hive
    • References
  • Pig Architecture
    • Pig Components
    • Pig Latin Data Model
    • References
  • Pig Installation
    • Prerequisites
    • Install Pig
    • Configure Pig
    • Verifying the Installation
    • References
  • Pig Execution
    • Execution Modes
    • Pig Execution Mechanisms
    • Invoking the Grunt Shell
    • Batch Mode
    • Tips
    • References
  • Pig Grunt Shell
    • Shell Commands
    • Utility Commands
    • References
  • Pig Latin Basics
    • Pig Latin – Data Model
    • Pig Latin – Statemets
    • Pig Latin – Data types
    • Pig Latin – Arithmetic Operators
    • Pig Latin – Comparison Operators
    • Pig Latin – Type Construction Operators
    • Pig Latin – Relational Operations
    • References
  • Pig Load & Store Operators
    • Preparing HDFS
    • Reading Data
    • Storing Data
    • References
  • Pig Diagnostic Operators
    • Dump Operator
    • Describe Operator
    • Explain Operator
    • Illustrate Operator
    • References
  • Pig Group Operator
    • Syntax
    • Example
    • Tips
    • References
  • Pig Cogroup Operator
    • Grouping Two Relations using Cogroup
    • References
  • Pig Join Operator
    • Self - join
    • Inner Join
    • Left Outer Join
    • Right Outer Join
    • Full Outer Join
    • Using Multiple Keys
    • References
  • Pig Cross Operator
    • Syntax
    • Example
    • References
  • Combining & Splitting
    • Union Operator
    • Split Operator
    • References
  • Filtering
    • Filter Operator
    • Distinct Operator
    • Foreach Operator
    • References
  • Sorting
    • Order By
    • Limit Operator
    • References
  • Pig Eval Functions
    • Eval Functions
    • References
  • Pig Load & Store Functions
    • PigStorage()
    • TextLoader()
    • BinStorage()
    • Handling Compression
    • References
  • Pig Bag & Tuple Functions
    • TOBAG()
    • TOP()
    • TOTUPLE()
    • TOMAP()
    • References
  • Pig String Functions
    • ENDSWITH()
    • References
  • Pig Date-time Functions
    • ToDate()
    • References
  • Pig Math Functions
    • ABS()
    • References
  • Pig User Defined Functions
    • Writing UDF’s using Java
    • Using the UDF
    • References
  • Pig Running Scripts
    • Comments in Pig Script
    • Executing Pig Script in Batch mode
    • References
• Kafka
  • Kafka Quickstart
    • 1.Download the code
    • 2.Start the server
    • 3.Create a topic
    • 4.Send some messages
    • 5.Start a consumer
    • 6.Setting up a multi-broker cluster
    • 7.Use Kafka Connect to import/export data
    • Windows
    • References
  • Kafka Manager
    • Configuration
    • Starting the service
    • References
• Solr
  • Solr笔记
    • 1.1 全文检索技术简介
    • 1.2 Solr简介
    • 1.3 Solr示意图
    • 1.4 Solr与Lucene区别
    • 1.5 Solr下载与安装
    • 1.6 简单操作Solr服务器页面
    • 2.1 Java操作Solr服务器HelloWorld
    • 2.2 全文检索基础
    • 2.3 全文检索基础（IK Analyzer）
    • 2.4 Solr基础
    • 2.5 Solr索引操作
    • 2.6 Solr搜索
    • 2.7 Solr模式
    • 2.8 Solr添加信息对象
    • 2.9 Solr查询详细使用
    • 3.1 Solr管理员命令
    • 3.2 Solr集群搭建
    • 4.x 案例实战
    • References
  • Solr网摘笔记
    • Spring Boot整合Solr
    • Results
    • References
• Spark
  • Spark Introduction
    • Features of Apache Spark
    • Spark Built on Hadoop
    • Components of Spark
    • References
  • Spark RDD
    • Resilient Distributed Datasets
    • Iterative Operations on MapReduce
    • Interactive Operations on MapReduce
    • Iterative Operations on Spark RDD
    • Interactive Operations on Spark RDD
    • References
  • Spark Installation
    • Verifying Java Installation
    • Downloading Scala
    • Installing Scala
    • Downloading Apache Spark
    • Installing Spark
    • Verifying the Spark Installation
    • References
  • Spark SQL Introduction
    • Features of Spark SQL
    • Spark SQL Architecture
    • References
  • Spark SQL DataFrames
    • SQLContext
    • DataFrame Operations
    • Running SQL Queries Programmatically
    • Inferring the Schema using Reflection
    • Programmatically Specifying the Schema
    • References
  • Spark SQL Data Sources
    • Spark SQL JSON Datasets
    • Spark SQL Hive Tables
    • Spark SQL Parquet Files
    • References
• Storm
  • Storm笔记
    • 1.1 Storm简介
    • 1.2 Storm特性
    • 2.1 Storm体系结构
    • 2.2 Storm架构图
    • 3.1 Storm集群环境搭建
    • 4.1 Storm Hello World
    • 5.1 Storm API
    • 5.2 Storm拓扑
    • 5.3 Storm拓扑配置
    • 5.4 Storm流分组
    • 5.5 Storm WordCount
    • 5.6 Storm Spout的可靠性
    • 6.1 RPC介绍
    • 6.2 Storm DRPC介绍
    • 6.3 Storm DRPC配置和示例
    • 6.4 Storm DRPC实例场景
    • 7.1 Storm Trident介绍
    • 7.2 Storm Trident functions
    • 7.3 Storm Trident filters
    • 7.4 Storm Trident projections
    • 7.5 Trident repartitioning operations
    • 7.6 Trident Partition aggregate
    • 7.7 Batch和Spout与Transactional
    • 8.1 Storm与Kafka
    • 8.2 Kafka相关术语介绍
    • 8.3 Kafka安装与使用
    • 8.4 Kafka-Manager-Console安装
    • 9.1 Storm与Redis
    • References
  • Storm官方文档
    • Examples
    • References
  • Storm网摘笔记
    • Storm集群安装配置
    • Tips
    • References