Data Center Tech Blog

San Diego Super Computing Center demonstrates the growing convergence between Big Data Analytics and High Performance Computing using “Big Data Flash”

Last week at Super Computing 2015 (SC15) show in Austin, TX, the University of San Diego (UCSD) and San Diego Super Computer Center (SDSC) presented the “Pacific Research Platform” (PRP). This is a virtual highway, capable of delivering on the promise of Big Data by leveraging fiber optic networks to transmit data at speeds of 10 gigabits to 100 gigabits per second.

The challenge PRP faced was to have an endpoint capable of terminating this 1,000x times faster optical network. One of the innovations PRP leveraged was the Flash I/O Network Appliance (FIONA) which used commodity PCs and turned them into Big Data appliances using Mellanox adapters talking to a 100G Arista TOR switch and SanDisk® InfiniFlash as the flash storage system. This enabled PRP to participate successfully in the “Petabyte Challenge” where a Lustre-based data oasis was transferred to the remotely mounted FIONA running at the SC2015 booth.

1PB Flash Storage, 1400 Watts

FIONA rack with 1PB flash storage (2 InfiniFlash units)

FIONA rack with 1PB flash storage (2 InfiniFlash units)

The stacked FIONA rack with 1PB flash storage (2 InfiniFlash units) consumed only 1400 watts. Just imagine designing the same rack with conventional flash arrays or SAS HDDs. We’d need to fill up the hall with storage systems!

High Performance Data Analytics (HPDA)

The Big Data Superhighway is a great example of High Performance Data Analytics (HPDA) a new market category established by IDC which brings High Performance Computing and Big Data Analytics together —a market that is predicted to grow rapidly.

Recent IDC presentations show how several “data driven” businesses like PayPal, Geico, the US Postal service, and various medical organizations and hospitals, are already using HPDA for use cases such as:

  • Fraud Analysis,
  • Internet of Things/Everything
  • Smart Grids
  • Healthcare
  • Genome Sequencing/Analysis

HPDA is enabled by new trends and hardware technology advancements, notably:

  • InfiniBand and 100 Gbit/s Ethernet fabrics
  • High performance POSIX-compliant distributed parallel file systems, with Hadoop connectors such as
  • Lustre, IBM Spectrum Scale (GPFS), Red Hat Gluster
  • Disaggregated, denser flash storage
  • Compute-driven HPC architectures and HDFS-driven Big Data architectures, leveraging the best of both worlds
  • The need to process both batch-oriented large capacity historical data and live streaming data, which, for example, is popularizing Lambda Architecture.

Enabling HPDA with Big Data Flash

HPDA needs real-time ingest and performance while providing fab economics for lowering CapEx and OpEx. The Big Data Superhighway is just one of the use cases demonstrating this new Big Data convergence. Customers are increasingly using HPDA for emerging use cases like Machine Learning, Natural Language Processing (NLP), Graph Analytics, Video, Sound, and Image Analysis.

HPDA is already contributing to solving science, healthcare, security and data challenges and will continue to have a major role in finding new solutions and opportunities. InfiniFlash , which IDC defined as “Big Data Flash,” is an essential building block to enable the HPDA market, delivering massive capacity with extreme performance and breakthrough economics.

Learn more about InfiniFlash on SanDisk.com

 

 

A Big Data Thanksgiving

SanDisk® at Discover 2015 with Hewlett Packard Enterprise

subscribe blog