Hammerspace leverages shipshape metadata handling for AI/ML workloads

Digital Author March 15, 2024

0 0 3 minutes read

Plot-outlined storage maker separates metadata from files to give glimpse-from-anyplace file machine visibility. It has now leveraged that for AI/ML workloads in Hyperscale NAS

Antony Adshead

Antony Adshead,
Storage Editor

Published: 12 Mar 2024 10: 41

Plot-outlined storage maker Hammerspace claims its Hyperscale NAS functionality will offer a international file machine that’s built for man made intelligence/machine studying (AI/ML) workloads and for the excessive demands of GPU-driven processing. It offers efficiency generally most effective supplied by dedicated excessive efficiency computing (HPC) storage merchandise, but for recordsdata resident in any on-set or cloud insist.

That’s a expend for purchasers that wish to expend recordsdata of many utterly different kinds, doubtlessly positioned in a pair of datacentres or clouds, but that would possibly perhaps well invent training datasets for analytics workloads.

Hammerspace if truth be told permits clients to glimpse, access and arrange recordsdata wherever it’s held and no matter storage it’s held on.

It’s a core component of Hammerspace’s expertise stack that it parses out the metadata from the file at an earlier stage than opponents. In diversified phrases, the Linux kernel that Hammerspace is built on separates out the metadata on the client and earlier than it’s written to storage.

This lightens the load on storage, but additionally draw metadata is offloaded when transmitted for processing akin to in AI/ML workloads and GPU farms. It has been leveraged to give the core offer of Hammerspace’s Hyperscale NAS.

Hammerspace is currently lawful for file recordsdata and competes with scale-out NAS makers akin to NetApp, Isilon and Qumulo. With Hyperscale NAS, it makes an strive to give storage targeted at HPC and AI/ML workloads.

“Historically, HPC file machine merchandise, worship DDN, own been a fancy sell into enterprises that already jog file machine merchandise,” talked about Molly Presley, senior vice-president of promoting at Hammerspace. “We’ve been ready to steal metadata out of the recordsdata course and build a parallel file machine that eliminates this overhead. Primitive file machine merchandise don’t attain that, and it’s no longer perfect in an HPC ambiance.

“In the AI world, most organisations don’t know which models they would possibly be able to wish to expend. So, what we offer offers flexibility, with recordsdata that resides in datacentres or within the cloud or is unstructured and that they then mediate they wish to access for AI training.”

Presley cited one buyer on the US Los Alamos Nationwide Laboratory, the set, she talked about, the organisation runs several diversified file methods and had struggled with being ready to distribute recordsdata to collaborators.

One other buyer cited by Presley runs Isilon scale-out NAS and had hit bottlenecks in how many GPUs it would possibly perhaps well probably feed, so the 32-node NAS machine had most effective been ready to give a 300-node render processing farm. The usage of Hyperscale NAS – since it took metadata out of the I/O course – the client became once ready to double the selection of rendering nodes to 600.

Hammerspace is amongst a neighborhood of merchandise that goal to give international file access and collaboration with access to the most modern version of files from any insist. Opponents include Ctera, Nasuni, Panzura and Watch Plot.

Hyperscale NAS is straight away available now for all Hammerspace clients at no extra rate. Hammerspace licencing relies completely mostly on the total quantity of recordsdata below administration.

In Can also final twelve months, Hammerspace obtained Rozo Programs for its RozoFS, and notably its developed erasure coding capabilities that enable for sharding of files all the scheme through a pair of locations.