By Greg Whynott -
I've been bouncing the idea around about using luster in M&E for some time now. For awhile the thinking was I'd be a pioneer in our segment if I did deploy it, but some research indicates there are a few places using it already with success. so no pioneer badge. 8(
the below is an edited version of the emails received.
==============================
Hi Greg,
We've been using lustre for a few years in production now and I would suggest you do some testing with your workload.
Apps like Houdini & 3Delight can grind the system to a halt! In saying that, it does make a great 'scratch' filesystem if you can separate your workloads out. The main contention is that these renderers require all your rendernodes to hit single texture files, ifds etc, which can sometimes just be on one disk [depending on size], creating hotspots and blocking writes to the storage nodes.
==============================
Hi Greg,
One of our customers had a similar requirement and we deployed Lustre 2.0.0.1 for them. This was in July 2011. Though there were a lots of problems initially, all of them were sorted out over time. They are quite happy with it now.
Environment:
Its a 150 Artist studio with around 60 Render nodes. The studio mainly uses Mocha, After Effects, Silhouette, Synth Eye, Maya, and Nuke among others. They mainly work on 3D Effects and Stereoscopy Conversions.
Around 45% of Artists and Render Nodes are on Linux and use native Lustre Client. All others access it through Samba.
Lustre Setup:
It consists of 2 x Dell R610 as MDS Nodes, and 4 x Dell R710 as OSS Nodes. 2 x Dell MD3200 with 12x1TB SAS Nearline Disks are used for storage. Each Dell MD3200s are shared among 2 OSS nodes for H/A.
Since the original plan (which didn't happen) was to move to a 100% Linux environment, we didn't allocate separate Samba Gateways and use the OSS nodes with CTDB for it. Thankfully, we haven't had any issues with that yet.
Performance:
We get a good THROUGHPUT of 800 - 1000MB/s with Lustre Caching. The disks it self provide much lesser speeds. But that is fine, as caching is in effect most of the time.
Challenge:
The challenge for us was to tune the storage for small files 10 - 50MB totalling to 10s of GBs. An average shot would consist of 2000 - 4000 .dpx images. Some Scenes / Shots also had millions of <1MB Maya Cache files. This did tax the storage, especially the MDS. Fixed it to an extent by adding more RAM to MDS.
Suggestions:
1. Get the real number of small files (I mean <1MB ones) created / used by all software. These are the ones that could give you the most trouble. Do not assume anything.
2. Get the file - sizes, numbers and access patterns absolutely correct. This is the key.
Its easier to design and tune Lustre for large files and I/O.
3. Network tuning is as important and storage tuning. Tune Switches, each Workstation, Render Nodes, Samba / NFS Gateways, OSS Nodes, MDS Nodes, everything.
4. Similarly do not undermine Samba / NFS Gateway. Size and tune them correctly too.
5. Use High Speed Switching like QDR Infiniband or 40GigE, especially for backend connectivity between Samba/NFS Gateway and Lustre MDS/OSS Nodes.
6. As far as possible, have fixed directory pattern for all projects. Separate working files (Maya, Nuke, etc.) from the data, i.e. frames / images, videos, etc. at the top directory level it self. This will help you tune / manage the storage better. Different directory tree for different file sizes or file access types.
If designed and tuned right, I think Lustre is best storage currently available for your kind of work.
Hope this helps.
==========================================================================
The new release of luster (2.x) is around the corner, I was told that it could be the first release which could be considered "business" ready, meaning it could be trusted to perform well with many types of work loads (outside of the typical IO profile HPC environments introduce) and highly available. It was correctly pointed out to me that Luster itself is very stable / available, its the choice of hardware and the method of setup (HA etc..) which determines more than anything how well it'll perform and its availability to the business.
Using various vendors (from highly managed / built for luster to the very typical) array pricing we determined it would cost between 140k to 400k for 600TB of storage, depending on your comfort level with the various vendors / value adds. These prices include the machines required to serve the storage. As many of you know, getting a similar amount of storage from one of the popular vendors typically is a $700+ investment. Yes there are advantages (and disadvantages) going with one of these vendors. I guess one of the questions become is it worth the extra $250,000 - $500,000 and nearly $100k every YEAR in support for a small-mid sized (even large) studio?
I suspect if a few shops in North America started going down this path, we may start to see a decline in the prices high end storage vendors are demanding. Especially when the advantages and disadvantages start to become harder to define/justify between the two solutions.
we need to start a M&E file system consortium group! 8)
thought it would be useful / interesting to some if I re-posted some responses I received while poking around the lists.have a great day,
greg
greg
==============================
Hi Greg,
We've been using lustre for a few years in production now and I would suggest you do some testing with your workload.
Apps like Houdini & 3Delight can grind the system to a halt! In saying that, it does make a great 'scratch' filesystem if you can separate your workloads out. The main contention is that these renderers require all your rendernodes to hit single texture files, ifds etc, which can sometimes just be on one disk [depending on size], creating hotspots and blocking writes to the storage nodes.
==============================
Hi Greg,
One of our customers had a similar requirement and we deployed Lustre 2.0.0.1 for them. This was in July 2011. Though there were a lots of problems initially, all of them were sorted out over time. They are quite happy with it now.
Environment:
Its a 150 Artist studio with around 60 Render nodes. The studio mainly uses Mocha, After Effects, Silhouette, Synth Eye, Maya, and Nuke among others. They mainly work on 3D Effects and Stereoscopy Conversions.
Around 45% of Artists and Render Nodes are on Linux and use native Lustre Client. All others access it through Samba.
Lustre Setup:
It consists of 2 x Dell R610 as MDS Nodes, and 4 x Dell R710 as OSS Nodes. 2 x Dell MD3200 with 12x1TB SAS Nearline Disks are used for storage. Each Dell MD3200s are shared among 2 OSS nodes for H/A.
Since the original plan (which didn't happen) was to move to a 100% Linux environment, we didn't allocate separate Samba Gateways and use the OSS nodes with CTDB for it. Thankfully, we haven't had any issues with that yet.
Performance:
We get a good THROUGHPUT of 800 - 1000MB/s with Lustre Caching. The disks it self provide much lesser speeds. But that is fine, as caching is in effect most of the time.
Challenge:
The challenge for us was to tune the storage for small files 10 - 50MB totalling to 10s of GBs. An average shot would consist of 2000 - 4000 .dpx images. Some Scenes / Shots also had millions of <1MB Maya Cache files. This did tax the storage, especially the MDS. Fixed it to an extent by adding more RAM to MDS.
Suggestions:
1. Get the real number of small files (I mean <1MB ones) created / used by all software. These are the ones that could give you the most trouble. Do not assume anything.
2. Get the file - sizes, numbers and access patterns absolutely correct. This is the key.
Its easier to design and tune Lustre for large files and I/O.
3. Network tuning is as important and storage tuning. Tune Switches, each Workstation, Render Nodes, Samba / NFS Gateways, OSS Nodes, MDS Nodes, everything.
4. Similarly do not undermine Samba / NFS Gateway. Size and tune them correctly too.
5. Use High Speed Switching like QDR Infiniband or 40GigE, especially for backend connectivity between Samba/NFS Gateway and Lustre MDS/OSS Nodes.
6. As far as possible, have fixed directory pattern for all projects. Separate working files (Maya, Nuke, etc.) from the data, i.e. frames / images, videos, etc. at the top directory level it self. This will help you tune / manage the storage better. Different directory tree for different file sizes or file access types.
If designed and tuned right, I think Lustre is best storage currently available for your kind of work.
Hope this helps.
==========================================================================
-g
