Generic placeholder image

OnCourse

Please use OnCourse for submission of some assignments and checking your grades.

Go to OnCourse

Generic placeholder image

Use of Google+

The class will interact with postings on a Google community group. The online section will also interact with Google Hangout or equivalent. Information about Google+ Hangout times will be posted for the data science ONLINE only section.

Go to Community

Generic placeholder image

Office Mix

Office Mix technology is being used for creating the lessons. The lessons are available in Office Mix, which supports combined voice, video, and slides.

Check out Office Mix

Generic placeholder image

Grading

Grading will be based on participation (10%), ABDS deployment (30%) and Project (60%).

Check your grades on OnCourse

Generic placeholder image

Technology Used

We will use FutureSystems (previously FutureGrid) facilities and cloud computing experience is helpful but not essential. Good working experience with Java is required.

Go to FutureSystems

FutureSystems Tutorial

Free Online Tutorials

Generic placeholder image

Instructor

Geoffrey Charles Fox
Senior Associate Dean for Research
Distinguished Professor of Physics,
Computer Science and Informatics
gcf@indiana.edu

More Details

Course description

This course studies software used in many commercial activities to study Big Data. The backdrop for course is the ~120 software subsystems illustrated here. We will describe the software architecture represented by this collection which we term HPC-ABDS (High Performance Computing enhanced Apache Big Data Stack).

A paper discussing this can be found here and presentations here and here. Copies of this material may be found in resources. You can download the syllabus (PDF).

Course Overview

Lessons

Unit 1: Introduction

  1. Part A: Introduction
  2. Part B: Real World Big Data
  3. Part C: Basic Trends and Jobs

Unit 2: Data Access Patterns and Introduction to using HPC-ABDS

  1. Part A: Introduction to HPC-ABDS
  2. Part B: Science Examples (Data Access Patterns)
  3. Part C: Remaining General Access Patterns
  4. Part D: Using the HPC-ABDS Stack

Unit 3: Big Data Applications Structure

  1. Part A: NIST Big Data Sub Groups
  2. Part B: 51 Use Cases and how they are parallelized
  3. Part C: First and Second Set of Features
  4. Part D: Machine Learning Aspect of Second Feature Set and the Third Set

Unit 4: Aspects of Big Data Applications

  1. Part A: Other sources of use cases and Classical Databases/SQL Solutions
  2. Part B: NOSQL Solutions and MapReduce
  3. Part C: Clouds vs HPC, BigData vs. Simulations

Unit 5: Big Data Applications and Generalizing their Structure

  1. Part A: NIST UseCases and 13 Image Based Applications
  2. Part B: Image Based Applications II
  3. Part C: Internet of Things Based Applications
  4. Part D: Big Data Ogres
  5. Part E: Facets of the Big Data Ogres

Unit 6: More on Software Stack (only one part)

  1. Part A: More of Software Stack.

Unit 7: Introduction to Cloudmesh

  1. Part A: Account Creation on FutureSystems - Summary (optional details in next 4 videos) More Info
  2. Part A.1: Account Creation on FutureSystems - Creating a Portal Account More Info
  3. Part A.2: Account Creation on FutureSystems - Upload an OpenID More Info
  4. Part A.3: Account Creation on FutureSystems - Upload a SSh key More Info
  5. Part A.4: Account Creation on FutureSystems - Join a Project More Info
  6. Part B: Cloudmesh for Big Data Analytics More Info
  7. Part C: Cloudmesh - Introduction More Info
  8. Part D.1: Cloudmesh - Quick Install on Desktop (D.1 and D.2 are alternatives) More Info
  9. Part D.2: Cloudmesh - Quick Install on VM (D.1 and D.2 are alternatives) More Info
  10. Part E: Cloudmesh - Web Interface More Info
  11. Part F: Cloudmesh - IPython Learning Server More Info
  12. Part G: Cloudmesh - Cloudmesh Command Shell interface More Info
  13. Part H: Cloudmesh - Command Shell API More Info
  14. Part I: Cloudmesh - Command Python API More Info

Unit 8: Introductory Tutorial: OpenStack, Chef, Hadoop, Hbase, Pig

  1. Part A: Create our Virtual Machine Instance
  2. Part B: Deploy Software with DevOps Tools
  3. Part C: Using the Apache Big Data Stack
  4. Part D: Installing a Hadoop Cluster on India

FutureSystems Access

Getting started with hands-on access:

  1. Create an account on the FutureSystems Portal.
  2. Request to be added to project FG-452.
  3. Upload a public SSH key to FutureSystems portal in order to access FutureSystems systems. Initial steps described in videos 1) Get a Portal Account, and 2) Upload an SSH key
  4. Explore the OpenStack Tutorial.
  5. Instructions for account creation, joining a project and uploading an SSH key are all available here.
  6. If you are using Windows, the simplest solution for using SSH keys is the Putty SSH client, and it’s SSH authentication agent Pageant. Putty and its associated programs are available here

Free Online Training via Lynda.com

More Information

The course covers the following material

  1. The cloud computing architecture underlying ABDS and contrast of this with HPC.
  2. The software architecture with its different layers at HPC-ABDS Kaleidoscope covering broad functionality and rationale for each layer.
  3. We will give application examples
  4. Then we will go through selected software systems – about 10% of those in the Kaleidoscope which have been already deployed on FutureSystems systems using OpenStack and Chef recipes.
  5. Students will chose one other open source member of Kaleidoscope each and deploy as in 4).
  6. The main activity of the course will be building a significant project using multiple HPC-ABDS subsystems combined with user code and data.
  7. Teams of up to 3 students can be formed with corresponding increase in scope in activities 5), 6)