OK - story time.
I've got a client here in NYC, who had a single flame, and bought a new one from my company. Basically, it's a pretty rippin' spec'd box - z820, 64gb, brand new peripherals (CG246's are FIRE!) - but didn't buy a stone for favor of an NFS-based framestore.?
Nothing out of the ordinary there, tons of people are doing it these days.?
So, these guys have an Arista 7148 (awesome switch, used to have one myself back at MPC NY) and a pretty legit set of Isilon clusters. In short, the one hosting both the framestore for the new and old flame is a 6-node S200, which, if you are familiar, absolutely RIPS.?
We get this new machine built, and we fail to get any good performance out of NFS. We're hitting maybe, 2-500mbps, on a crazy?roller coaster?of inconsistent data. The only other client to this cluster (which is already capable of 2.5gbps/out, by the way) is the other Flame.?
Project-wise, they're often 10/12 bit 2k, and typically run 6-800mbps for playback requirements to their stones. Pretty beefy requirements, but the cluster can totally handle it.?
So, naturally, with the numbers were hitting on the new flame, performance-wise, we're not on the mark of delivering what we need from the NFS to satisfy this Flame.?
Much deliberating, tweaking, switch config looking at, Isilon config looking at, and we come down to the fact that when the first one was installed, they had purchased a Myricom card. Now, the story goes (before my time) that when the first one was installed, they saw the same results from the Myricom card - and couldn't figure it out.?
The solution at the time the first flame was bought, was to purchase a Mellanox ConnectX-2, which at the time was a 2500$ card with a 2000$ QSFP to SFP+ adapter to get it on their 10g network. Kind of shitty, as the Mellanox is about 600$... I thought to myself, but I was wondering why.
We wound up having to use a ConnectX-3 (the 2's are disco'd, etc) - which was thankfully only 750'ish, capable of the same stuff (QDR, etc) as it's older brother, and the adapter was 40$. Wow, they have really come down in price.?
The second we put that card in, using the SAME AVAGO/Finnistar cheapie 10g SFP+s - we are getting CONSTANT 700-900mbps from this, AND from both at the same time.?
So, what I'm wondering is - optics aside, is the QDR in that Mellanox card actually worth the extra 100$? Or is it the cacheing on the card? Because AFAIK, some other 10g nerd can confirm this, I'm hoping - the Myricom cards DO NOT cache anything, or attempt to buffer - where the Mellanox cards do? Is that just a bit of extra ROM on the card that you're paying for?
Either way, in my books - if you can spend an extra 100-150$ to get a ConnectX-3, get both IB and 10g in one card, it's a freakin' no brainer over the Myricom (which blows my mind because I LOVE(d?) Myricom cards) - but I really really really can't figure out why, and vastly would like to know.
Also, I don't think I've ever seen NFS get rates like that over 10g, goddamn.
TL;DR - What is the data rate of a Myricom 10g card and why is it so much less sufficient than a QDR Mellanox dual-personality?card?!
--
----------
Dan Young
CineSys-Oceana New York
347-334-0132