Practical Real-time Data Processing and Analytics电子书

Title Page

Copyright

Practical Real-Time Data Processing and Analytics

Credits

About the Authors

About the Reviewers

www.PacktPub.com

Why subscribe?

Customer Feedback

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

Introducing Real-Time Analytics

What is big data?

Big data infrastructure

Real–time analytics – the myth and the reality

Near real–time solution – an architecture that works

NRT – The Storm solution

NRT – The Spark solution

Lambda architecture – analytics possibilities

IOT – thoughts and possibilities

Edge analytics

Cloud – considerations for NRT and IOT

Summary

Real Time Applications – The Basic Ingredients

The NRT system and its building blocks

Data collection

Stream processing

Analytical layer – serve it to the end user

NRT – high-level system view

NRT – technology view

Event producer

Collection

Broker

Transformation and processing

Storage

Summary

Understanding and Tailing Data Streams

Understanding data streams

Setting up infrastructure for data ingestion

Apache Kafka

Apache NiFi

Logstash

Fluentd

Flume

Taping data from source to the processor - expectations and caveats

Comparing and choosing what works best for your use case

Do it yourself

Setting up Elasticsearch

Summary

Setting up the Infrastructure for Storm

Overview of Storm

Storm architecture and its components

Characteristics

Components

Stream grouping

Setting up and configuring Storm

Setting up Zookeeper

Installing

Configuring

Standalone

Cluster

Running

Setting up Apache Storm

Installing

Configuring

Running

Real-time processing job on Storm

Running job

Local

Cluster

Summary

Configuring Apache Spark and Flink

Setting up and a quick execution of Spark

Building from source

Downloading Spark

Running an example

Setting up and a quick execution of Flink

Build Flink source

Download Flink

Running example

Setting up and a quick execution of Apache Beam

Beam model

Running example

MinimalWordCount example walk through

Balancing in Apache Beam

Summary

Integrating Storm with a Data Source

RabbitMQ – messaging that works

RabbitMQ exchanges

Direct exchanges

Fanout exchanges

Topic exchanges

Headers exchanges

RabbitMQ setup

RabbitMQ — publish and subscribe

RabbitMQ – integration with Storm

AMQPSpout

PubNub data stream publisher

String together Storm-RMQ-PubNub sensor data topology

Summary

From Storm to Sink

Setting up and configuring Cassandra

Setting up Cassandra

Configuring Cassandra

Storm and Cassandra topology

Storm and IMDB integration for dimensional data

Integrating the presentation layer with Storm

Setting up Grafana with the Elasticsearch plugin

Downloading Grafana

Configuring Grafana

Installing the Elasticsearch plugin in Grafana

Running Grafana

Adding the Elasticsearch datasource in Grafana

Writing code

Executing code

Visualizing the output on Grafana

Do It Yourself

Summary

Storm Trident

State retention and the need for Trident

Transactional spout

Opaque transactional Spout

Basic Storm Trident topology

Trident internals

Trident operations

Functions

map and flatMap

peek

Filters

Windowing

Tumbling window

Sliding window

Aggregation

Aggregate

Partition aggregate

Persistence aggregate

Combiner aggregator

Reducer aggregator

Aggregator

Grouping

Merge and joins

DRPC

Do It Yourself

Summary

Working with Spark

Spark overview

Spark framework and schedulers

Distinct advantages of Spark

When to avoid using Spark

Spark – use cases

Spark architecture - working inside the engine

Spark pragmatic concepts

RDD – the name says it all

Spark 2.x – advent of data frames and datasets

Summary

Working with Spark Operations

Spark – packaging and API

RDD pragmatic exploration

Transformations

Actions

Shared variables – broadcast variables and accumulators

Broadcast variables

Accumulators

Summary

Spark Streaming

Spark Streaming concepts

Spark Streaming - introduction and architecture

Packaging structure of Spark Streaming

Spark Streaming APIs

Spark Streaming operations

Connecting Kafka to Spark Streaming

Summary

Working with Apache Flink

Flink architecture and execution engine

Flink basic components and processes

Integration of source stream to Flink

Integration with Apache Kafka

Example

Integration with RabbitMQ

Running example

Flink processing and computation

DataStream API

DataSet API

Flink persistence

Integration with Cassandra

Running example

FlinkCEP

Pattern API

Detecting pattern

Selecting from patterns

Example

Gelly

Gelly API

Graph representation

Graph creation

Graph transformations

DIY

Summary

Case Study

Introduction

Data modeling

Tools and frameworks

Setting up the infrastructure

Implementing the case study

Building the data simulator

Hazelcast loader

Building Storm topology

Parser bolt

Check distance and alert bolt

Generate alert Bolt

Elasticsearch Bolt

Complete Topology

Running the case study

Load Hazelcast

Generate Vehicle static value

Deploy topology

Start simulator

Visualization using Kibana

Summary