This past year I had the opportunity to be involved in a number of storage projects. One project that is somewhat memorable was a customer replacing their now aging FC SAN with 10GbE based iSCSI. We worked through a number of design issues and in the end the customer was very satisfied with the results. Here are some random pitfalls when migrating to iSCSI SAN.
Underwhelm the switching requirements.
I have heard this a number of times, and seems like there is a bit of misunderstanding around the requirements for iSCSI switching hardware. One engineer boldly proclaimed, that after suffering performance issues, they went to Best Buy and bought an off-the-shelf Netgear just for their iSCSI network! The fact that iSCSI can use commodity Ethernet switches does not mean you should. The hardware used for iSCSI should be “enterprise” grade. It should be wired speed (or not heavily oversubscribed) and if at all possible dedicated to the iSCSI SAN. Usually aggregation grade data center switches (Cisco Nexus 5000/7000) are great candidates.
Forget about QoS
I was recently working with a customer reporting a very small and random amount of packet loss. When asked if they have a QoS policy, the response was, “Nope!” I pointed out to the customer that if there is no QoS in use, all traffic on their network should be considered best effort. Things like 802.1p, Flow-Control and queuing policy need to be carefully implemented on the network, especially when there are diverse traffic flows. Storage traffic sits somewhere below control-plane traffic in terms of priority. Use QoS!
Routing SAN Traffic
It is not a good practice to place endpoints of an iSCSI SAN on a network in such a way that traffic must be routed. If you look at the data, there is substantial delays added, even on high-end multilayer switches when the transit path involves routing. Keep it on the same subnet/broadcast domain.
Hardware HBA
I strongly recommend using hardware HBA’ s. Data shows that software initiators place a substantially higher load on the server CPU. You don’t need this added burden since it limits the compute resources available for server workloads. In a virtualized environment it becomes the “wildcard”.
Network Engineer interested in many areas including switches/routers/firewalls, SAN, and virtualization. I am currently employed by Cisco Systems. While I like to think that everything I write is well reasoned and insightful, the opinions expressed are solely mine and do not represent my employer.
area51d 
Leave a comment