VMware vSphere Cluster Design

In today's ever-evolving digital landscape, the foundation of any virtual infrastructure is crucial for meeting the dynamic needs of businesses.
Let's delve into the nuts and bolts of crafting a high-performing vSphere cluster, focusing on utilising 10Gbe networking technology with VMware vCenter 5.5 U2A and ESXi v5.5 U2a

The clear goal we had from a customer was 'consolidate our physical and virtual servers onto new hardware, running on a private vmware cloud platform, within our existing datacentre.

We need to qualify the assets in-scope; and itemise their workloads. In order to identity the requirements in terms of raw RAM, CPU and DISK needs.
Evaluating those items, helps you derrive a capacity plan.

Being  more specific with that objective:

• Always make calculations on memory-footprint, from maximum assigned values.
  Why? If an  SQL Server might be given 32GB of RAM, but only actively uses 22GB on average. You would not be asking the customer to reduce the assigned RAM on this VM.

• Research the old hardware specification on each server, and extrapolate the CPU Ghz each server/vm that it is "actively consuming" (mode).
  If you are consolidating dozens of servers; onto 3-4x new hosts. The new CPU architecture, and core-count and possibly the FSB rate and multipliers all make a HUGE difference.. what if it's not enough...
  The combined CPU capabilities of your new kit, needs to be sufficient to satisfy the CPU-demand (extrapolated CPU Ghz calculation from old kit).. and then add some room for expansion.. say +60% (this is before any N+1 modification)

• Ask yourself; will the storage arrays be sufficient for the next year or two? Even after planning for growth, with an increased I/O?

• Your customer will likely agreed to buying cheaper dedicated backup arrays; higher capacity, slower performance arrays.
  Promising a client they will follow an industry leading best-practice "3-2-1 recovery point policy", is a real selling point! 

• Disk consumption is not always an accurate representation of the data-footprint.  Some DBs have clones/replicas or prod/test/dev copies, DBs have whitespace; logs get rotated/erased on interval,file-servers.
  Group disk statistics into categories; understand how many duplicates exist (corporate file-shares, or SQL Database replicas). You might be surprised. Again add 30-40% on the initial investment; but include expansion capabilities for projected growth. 

• Understanding the Network Uplink utilisation of your current servers, may be a complicated endevour. Ask for NetFlow statistics. Few clients can provide them; but they save a lot of time here!
  Consider looking at switch throughput, from each host uplink, over peak times.  Maybe reset counters and capture them again 1x week later.. and take a crude average. There are many ways to skin a cat; but you do need to get some broad estimates.
  If you are above 2Gbps on average, then quote for 10Gbe. 

RVTools by RobWare is increadibly useful in this qualification exercise.

• Calculate your RAM requirements, and divide it equally amongst a set number of servers. Find the sweet spot on the market cost per performance. Then add another host; effectively over provisioning (giving you maintenance capabilities).
  EG: 700GB RAM required > spec 3x hosts with 384GB of RAM, to give you 3x384GB = 1152GB.
  This is called N+1.

• ALWAYS use ECC DIMMs (Error Code Correction). There is just no excuse not-to these days.

• Configure your BIOS settings to support Fault Redundant memory configuration.

• iSCSI/NFS:  No two storage arrays are the same.  Some vendors prefer a single subnet for accessing the SAN (green), others recommend a Red/Blue fault tolerant domain. >>>> Check the vendors documentation!!!!!
  Then go and find some SAN xxx best practices and performance enhancement documentation too - this won't be time wasted!

• FibreChannel always uses a dedicated HBA. or TWO, for redundancy. These can operate at 4, 8, 16, or higher Gbps of bandwidth, and removes the TCP/IP layer, reducing latencies even further. Compared to iSCSI, FibreChannel can often out-perform LUN access times by 50-80% when under load.
  If you have a 3PAR array, or even MSA, look  at the SPOCK website. EMC VMAX or smaller VNX series are almost identical to Dell PS series from a configuration standpoint. HP MSA (although i've only seen fibre channel, does support iSCSI), again these all operate on a single fault-tolerant domain.

• VAAI for performance! This can be increadibly beneficial for Backup Mechanisms!
  By asking the array to do X work, rather than downloading, performning the task, then uploading the changes to the SAN.

Lets tackle the top-down architecture.

• A Router will terminate all VLANs as sub-interfaces, configured with 802.1Q encapsulation, with a specific VLAN ID.

• The Router needs to support LAG with LACP, to enable support for higher bandwidth.  Depending on the vendor, you may also find support for 'interface failover (active/passive LAG)', but doesn't require any advanced switch port configuration.

• My experience has shown almost all clients use a dedicated customer Firewall functioning in-place of the Layer 3 Router. Major benefit for this is simply the ability to terminate traffic that doesn't conform to the approved ACLs. some Switches may be capable of doing this L3 ACL work; but that remains out of scope of this article.

• If you need LAG and LACP down to the hosts, your switches should to be stacked, or support a VLT-domain configuration. Then an MLAG configuration can be used between switch member interfaces, for each of your hosts. This requires some more advanced switch configurations that most VMware engineers are simply not used to performing.
  
  My experience again shows, clients Adopt Load-Based-Teaming (LBT) on port-groups. This feature has been around since v4.1, but today in v5.5, offers several hotfixes incorporating over half a dozen load balancing mechanisms. I highly recommend considering LBT instead of LAG+LACP.
  If you are backing up over the network: a LBT configuraiton isn't going to make any single-backup-job performance improvements; but the aggregate of all jobs should be considerably faster!
  
  

• If you have a Fibre Channel SAN, configure Zoning, or any SAN configure Raid Groups, Volumes etc ahead. 

Notes

• Limiting traffic beyond the above basic (service-based) ACLs remains out-of-scope of this article. Example SAN-ACL may need to support replication between arrays on different sites. You may have Syslog services to consider, or Veeam...  Each 'component' requires a review of the Firewall Rules.

• Most SANs have a dedicated Management interface/subnet, and separate iSCSI/NFS interfaces. With a recommended fault-tolerant domain design, supporting a RED and BLUE configuration.. Which is just two subnets, one per-vlan.  

Don't forget the VLANs are TRUNKED (tagged) on host uplinks.
Older SANs will likely be using a NATIVE (untagged) config; but some of the newer SANs (VMAX, VNX, PS-Series and Compellent) all now support Datacentre Bridging, with a encapsulated VLAN, Session Prioritisation and queue-depth control. Again consult the vendors documentation to be 100% sure.

NOTE: Some SAN Vendors require you to use their own MultiPathing Extension, or just MultiPathing Rules, others require half a dozen adapter advanced properties and a host reboot before you can use them.  Please check the vendor documentation to be absolutely sure.

I began by outlining the essential steps, from ensuring hardware compatibility to configuring multipathing for redundancy and load balancing. Zoning on Fibre Channel switches was highlighted as a crucial aspect of SAN configuration, providing security by restricting communication between devices.

I would like emphasise the importance of thorough testing to validate connectivity and failover mechanisms, ensuring the robustness of the setup. Additionally, we discussed the significance of ongoing monitoring and maintenance to sustain optimal performance and reliability.  DVS supports a health check now, that relies on CDP/LLDP. So configure that too! It will save you time with testing, and give you confidence you did it right.

Feel free to follow my outlined steps and with your own research and other best practices, other engineers should be easily able to establish a highly-available, highly-redundant and highly-performance VMware vSphere environment.

Hope this helps someone.