UCS Deployment Guide – Cabling and Cluster Creation – Part 2

2.1 UCS Cabling

One of the most important topics in UCS, if not the most important one; UCS Cabling. When it comes to UCS cabling, and starting from the chassis, passing through the FI, and going up to the ToR switch, three questions come to mind, that need to be discussed:

1. How to cable the UCS chassis to the FI?

2. How to cable both FIs between them?

3. How to cable the FI to the uplink ToR/EoR switch?

UCS Chassis to FI Cabling

With regard to Chassis-FI connectivity, no way to think Virtual Port Channel (vPC) or Virtual Link Aggregration (vLAG) for redundancy between the IOMs and the FIs. Actually, the correct cabling is to do it the old classical way. In other words, connect first FEX to first FI, and the second FEX to the second FI. Any other cabling will not be supported by Cisco. Have a look on the screenshot below to get a better overview.

Assuming we have one chassis, the cabling would like this

Chassis-To-FI-Connectivity-FEX

Assuming we have two chassis, it would look like this

ucs-cabling

 

Fabric Interconnects Cabling

As they are part of the same cluster, Fabric Interconnects need to be connected to each other, to be able to communicate and exchange heartbeats. These heartbeat flow through two ports named L1 and L2, if you look closer to a 6332 Fabric Interconnect , you’ll see these two ports.

UCS-L1L2

For a proper cabling, we’ll have to connect the the L1 port of Fabric Interconnect-A to the L1 port of Fabric Interconnect-B. Likewise, L2 of Fabric Interconnect-A will be connected to L2 of Fabric Interconnect-B. You’ll also need to connect the Mgmt port from each FI to the managment network, in order to be able to manage the UCS cluster after the cluster configuration is done.

UCS-L1L2Cabling

FI to uplink switch

The Fabric Interconnect will be used as an intermediary to connect our server farm to the uplink Ethernet and Storage networks, and some ports will be configured specifically for this. The uplink cabling from the FI to the upstream networks will look like this.

fi-to-uplinknetworks

 

2.2 UCS Initialisation

First, we need to decide which FI will be the Primary one, and which one will be the Subordinate. Once done, we need to make sure, to avoid any configuration mistake, that the Subordinate FI is shutdown, as we we’ll do most of the configuration on the Primary one to create the cluster, then power on the Subordinate FI, and join it to the cluster. Basically, these are the steps in order, after making sure the cabling has been done properly:

  1. Power on FI-A
  2. Connect through console port
  3. Create the cluster by following a step-by-step CLI guide.
  4. Power on FI-B
  5. Connect through console port
  6. Join the FI-B to the cluster

Once the FI-A is on, connect to the console port of Fabric Interconnect (FI) “A”, which will be the primary member of the cluster. Verify that the console port parameters on the attached computer are as follows “9600 baud”, “8 data bits”, “No parity”, “1 stop bit”.

UCS-ConsolePort

2.3 Creating the UCS cluster

You will then be presented with the following menu items.

Enter the configuration method. (console/gui) ? console
Enter the setup mode; setup newly or restore from backup. (setup/restore) ? setup
You have chosen to setup a new Fabric interconnect. Continue? (y/n): y
Enter the password for “admin”: Pa$$w0rd
Confirm the password for “admin”: Pa$$w0rd
Is this Fabric interconnect part of a cluster(select ‘no’ for standalone)? (yes/no) [n]: yes
Enter the switch fabric (A/B) []: A
Enter the system name: NameOfSystem (NOTE: “-A” will be appended to the end of the name)
Physical Switch Mgmt0 IPv4 address : X.X.X.X
Physical Switch Mgmt0 IPv4 netmask : X.X.X.X
IPv4 address of the default gateway : X.X.X.X
Cluster IPv4 address : X.X.X.X (NOTE: This IP address will be used for Management)
Configure the DNS Server IPv4 address? (yes/no) [n]: y
DNS IPv4 address : X.X.X.X
Configure the default domain name? (yes/no) [n]: y
Default domain name: UCSDomain.com
Apply and save the configuration (select ‘no’ if you want to re-enter)? (yes/no): yes

Now connect to the console port of the secondary FI and power it on. FI-B will detect the first FI through the heartbeat network, and will ask to joined to the cluster already created. Once again, you will be presented with the following menu items:

Enter the configuration method. (console/gui) ? console
Installer has detected the presence of a peer Fabric interconnect. This Fabric interconnect will be added to the cluster. Continue (y/n) ? y
Enter the admin password of the peer Fabric interconnect: Pa$$w0rd
Physical Switch Mgmt0 IPv4 address : X.X.X.X
Apply and save the configuration (select ‘no’ if you want to re-enter)? (yes/no): yes

On the real world, the configuration on the putty console will look like this for the FI-A

UCS-FI-Setup-1

and like this for the FI-B

UCS-FI-Setup-1
2.4 Connecting to UCS Manager

Once the UCS cluster is created, we should be able to connect it through a supported browser, or through SSH. we’ll just need to put the cluster’s IP address. When using the browser, we’ll be presented with two ways to launch UCS Manager, the old Java applet way, and in the new HTML based console.

UCS-ConnectToUCSM

Type the credentials and Log In. The password is the same configured during the cluster creation.

UCS-ConnectToUCSM

We’ll choose if we want to authorize Cisco Smart CallHome or not.

UCS-ConnectToUCSM

There you have it!. We’ve connected successfully to the UCSM console.

UCS-ConnectToUCSM

 

2.5 Configuring Port Types: Server Ports, Network Ports and Storage Ports

To understand what needs to be done in this step, we need to have a closer look of what is unified port. In a nutshell, Unified Ports are ports which are capable to handle either Ethernet or Fibre Channel traffic. In previous version of Fabric Interconnect, like in Cisco 6100 series, ports in the fixed module (the module located on the left of the FI) were not unified, and this basically means that ports were used only for Server or Ethernet Uplinks, but not for Storage Uplinks.

Starting from Cisco 6200 series, ports are unified. Therefore, ports can be configured as Server Ports, or to carry either Ethernet or Fibre Channel traffic. Using UCS manager , you can configure the port type as Ethernet or Fibre.  In this section, we will see how to set the port mode and discussing about various port types, and we’ll use the following information as a reference to configure the ports needed.

1/1 to 1/6 used for Uplink Storage Ports

1/17 to 1/20 used for Server Ports

1/33 to 1/34 used for Uplink Network Ports

2.5.1 Configure Unified Ports

As mentioned earlier, we need to tell the UCSM which ports will be used for storage, and which ones will be used for network. For this, we’ll use the Configure Unified Ports tool.

In this deployment, we’ll use the first six ports, from 1/1 to 1/6 as FC ports. To do so, follow the the steps.

Go UCSM > Equipment > Fabric Interconnect A (primary) > On the central pane, click on General tab, then on Configure Unified Ports

UCS-ConfigUnifiedPorts

Be careful with this change in a production and active environment, as any change in the fixed module will cause a reboot of the FI, and any change in an expansion module will cause a reboot of the module itself. Click on Yes to continue

UCS-ConfigUnifiedPorts

Click on the horizontal scroll bar to decide about how many ports will be used for FC Storage.

UCS-ConfigUnifiedPorts

We’ll use use 1/1 to 1/6 as FC ports. The Desired If Role will change as we scroll from left to right. Once done, we’ll click on OK, then twice on Yes. Be aware the the FI will be rebooted, as mentionned by the Warning message.

UCS-ConfigUnifiedPorts

UCS-ConfigUnifiedPorts

2016-10-31-11_13_51-dr-ucs1-dmz01-unified-computing-system

2.5.2 Configure Server Ports, Network Ports, and Storage Ports

At this stage, we have to identify the ports types in Fabric Interconnects. In other words, we need to tell to our two FIs which ports are used to connect the chassis, and which ports are used to connect to the uplinks network and storage switches, otherwise, our FI will not have any idea what is connected to them in both downside and upside. Before the actual configuration, let’s have a look on the UCS port types:

Server Port: Port is configured for southbound connection to an IOM Fabric Extender (FEX) module in a blade chassis. In other words, These ports are used to connect the chassis to the FI.

Uplink Port: Port is configured for northbound connection to the upstream Ethernet switch. They are used to connect the FI to the uplink core switches.

FCoE Uplink Port: same as Uplink Ports, but used in case the port is unified.

FCoE Storage Port: used in case the port is unified and will be used to connect to a for storage fabric.

Appliance Port: will be used to connect directly to a storage array for SAN or NAS storage.

Basically, and assuming to have one chassis that need (actually, it has to!) to be cabled to both FI (Fabric-Interconnect), and then from the FIs to the uplink Fabric and Ethernet switches, we’ll connect

-Two cables from each FEX to the Fabric-Interconnect. We’ll need for this two Server Ports.

– Three cables from each the FI to the Fabric switch. We’ll need for this two Storage Ports. This was already done by configuring the 2.5.1 Configure Unified Ports Section

-Two cables from each the FI to the Ethernet switch. We’ll need for this two FCoE Uplink Ports.

 

Ports 1/16 to 1/20 will be configured as Server Ports. The following steps need to be done on both FI. to To do so, we go to Equipment > Fabric Interconnect A (primary), and we right click the port we want to turn to a Server Port.

UCS-ConfigPortType

UCS-ConfigPortType UCS-ConfigPortType UCS-ConfigPortTypeWe’ll use the same steps to configure ports 1/33 and 1/34 as FCoEUplink Ports, that will be used for upstream Ethernet connectivity

UCS-ConfigPortType

 

2.6 Configure Chassis Discovery Policy

The Chassis Discovery Policy determines how the system reacts when you add a new chassis. It determines the number of cables that need to connected from the chassis to the FI, so that USCM can detect it. Therefore, UCSM uses the settings in the Chassis Discovery Policy to determine the minimum threshold for the number of links between the chassis and the fabric interconnect.

In this implementation, UCS will be configured to discover the chassis with 2 Uplink mode per Fabric Extender Module. To configure the UCS Chassis Discovery Policy, go to:
Equipment > Policies > Global Policies

UCS-CDP

 

After configuring our Discovery Policy successfully, we should be able to see our detected chassis. We have two that have appeared in the UCSM.

 

UCS-CDP

 

While clicking on Chassis 1, you can see that is already populated with 8 servers

 

UCS-CDP

 

All the servers visible right now are just a piece of hardware, with no identity. No one of them can be exploited for the time being, until they get assgned a configuration file that contains all the information needed to boot, to configure their RAID, to enumerate the number of NIC card and HBA card they could evntually have, …etc. These configuration files are called Service Profiles, and they are used to make our servers live and operative, but that’s not all. The Service Profiles to give the servers a stateless identity, and it basically means that all the configuration information, like BIOS config, UUID, WWNN, WWPN, MAC addresses, and so on are registered on them, which will allow for easier troubleshooting and hardware replacement in case there is an issue in one of our servers, as all the information are not stored on the server itself, but in the Service Profile, that can be unssigned from a failed server, and reassigned to any new server installed in the UCS domain.

2.7 Summary

In this post, we have seen how to correctly cable a UCS envioronment from the chassis to the FIs, and also from the FIs to the uplink network switches and storage fabrics. We have also initiated and created our UCS cluster, then have connected to the FI virtual IP through UCS Manager (UCSM), and have checked that connectivity to the UCS cluster is working fine.

We then configured our Server, Network and Storge ports in order to allow to UCSM to discover our servers and detect any existing uplink connectivity.

Finally, we have configured the Chassis Discovery Policy to determine how the system reacts when you add a new chassis, or when the cabling in any existing chassis is changed.

The next post will be quite long, as we will go through how to prepare the building bricks to create our Service Profiles, assign then to our UCS server and make our servers fully functional.

 

Find this post interesting. Share it!

5 Comments

  1. Pingback: Cisco UCS Deployment Guide – vAdmin-Land

  2. Maverick

    One of the most detailed tutorial on UCSM.

    Reply
    1. Mourad NAKIB (Post author)

      Thanks Maverick. Happy you liked it!

      Reply
  3. Giuseppe Mautone

    Thank a lot the good guide

    Reply
    1. Mourad NAKIB (Post author)

      Most welcome Giuseppe! Happy it helped.

      Reply

Leave a Reply to Maverick Cancel reply

Your email address will not be published. Required fields are marked *