File System Evaluation Project

Next-gen HPC file system evaluation and rollout

File storage is a core piece of any HPC deployment, one that nearly every other piece depends upon.  We are taking the opportunity with the creation of a new cluster to evaluate and deploy a parallel file system that will be more suited to HPC use, easier to manage, and provide us a path forward as we scale up our infrastructure.

This is a complex task with many requirements and a very high need for reliability.  The major phases are as follows:

Phase 1: Information gathering

We have approximately 20 petabytes of storage serving thousands of cores of compute capacity.  Understanding what our storage is doing is crucial.

Our primary approach for file serving uses NFS v4 as a network protocol, and ZFS as an on-disk filesystem.  These are well understood and capable but have become harder to maintain and less flexible as we have grown.  Filesystem and protocol features in current use include:

Three clusters were selected from which to gather representative load and usage information from: Farm, LSSC0, and HPC2.  Data was extracted from approximately 90 days of operation to generate the following average figures:

From this we can conclude:

• Our average IO volume is roughly 10TB per day per petabyte (1%)
• Our average IO operations are roughly 700 per second per petabyte
• Average IO size is moderate to large (>100KB)
• Effective compression is mandatory for our data

Our storage hardware purchase includes:

• 6 storage nodes
• 6x 792TB (44 spindles) of disk

Adding in a 100% safety factor, we project our new cluster to be:

The limiting factor here for performance is very likely to be network IOPS per spindle.

Previous evaluations have been focused on Quobyte as a parallel filesystem solution. BeeGFS was researched as an alternative, and two widely used competitors in the HPC space (Lustre and Ceph) have been added to create a comprehensive list for consideration.

Phase 2: Feature evaluation

Our requirements can be divided into features which can be evaluated ahead of install time, and metrics which require a testbed to examine behavior.  We first evaluate the features, then top candidates are chosen.

From this feature grid, prioritize the best choices for operational testing.  These are ranked here:

Phase 3: Operational testing

Operational testing at this point is based on these assumptions:

• Initial systems will be delivered in the near future (late August)
• Storage and network equipment delivery will take an additional 4-8 weeks

In order to deliver a solution as soon as possible once all hardware is in place, we must perform as much testing as possible using available resources.

For each software candidate we must:

• Identify an area for install/test
• Plan our storage topology
• Install software
• Validate features listed in the Phase 2 table
• Evaluate, score and add notes for each category

Based on this, identify our primary choice for deployment, and document structure and rollout plan.

Phase 4: Final cluster setup

Hardware arrives and we have our primary software choice ready.  Setup includes:

• Install and setup of network hardware
• Install and setup of hardware and OS on storage nodes
• Ensure host monitoring is in place
• Baseline performance testing to quantify hardware
• Install and configuration of storage software
• Setup of monitoring to take storage alerts and to collect statistics
• Perform feature testing of critical features from Phase 2
• Repeat failure testing at all levels (disk, enclosure, cabling, server, network).  Ensure that any downtime is as expected and that alerts are operating and documented.
• Ensure documentation of architecture and operation is correct and up to date.

Phase 5: Acceptance testing

This is our final phase.  We must ensure these three things before rollout:

• Reliability is as expected
• Performance is sufficient with our load testing
• Responsiveness and overall user experience is good

Reliability is primarily addressed in Phase 4, but we must keep an eye out for anything actionable or unusual during Phase 5.

Performance testing can be done using a synthetic background load and then running A/B tests on the new cluster and an old cluster.  The background load should match the worst parts of the IO envelope experienced by Farm and HPC2.  This can be produced by using two instances of fio per test client, averaging:

• 2700/s 120KB read ops
• 3500/s 90KB write ops

This will produce roughly 320MB/s each of read and write traffic.

A/B testing can be done either with user applications or with synthetic load.  Synthetic load should include:

• Large (512KB) asynchronous reads
• Medium (128KB) asynchronous reads and writes
• Small (16KB) synchronous writes, to represent MPI traffic

User experience is best addressed by finding “power users” who can stress the storage effectively, and then having them run real-world processing both in an existing cluster and the new cluster.  This processing must be repeatable; in the event of a significant difference we will want to re-run it and gather detailed information.