Cisco HyperFlex Hyper Converged Infrastructure Instructions
cisco HyperFlex Hyper-Converged Infrastructure
Recommended Upgrade Method
For both combined upgrade and split upgrade, Cisco recommends upgrading the HyperFlex components in the following order for optimizing the upgrade time:
Note
Make sure to upgrade the vCenter to the desired version based on the ESXi version and recommendation from VMware before upgrading ESXi.
As part of the Server Firmware upgrade operation initiated from HX Connect, some of the UCS policies may be updated to be compatible with the new HXDP version. These changes are applied only to the nodes that are part of the cluster is upgraded. It is highly recommended to use HX Connect to initiate the Server Firmware upgrade to avoid any policy drift.
- Upgrade Cisco UCS Infrastructure
- Upgrade Cisco HX Data Platform
- Upgrade Cisco customized VMware ESXi
- Upgrade Cisco UCS firmware
Upgrading Your HyperFlex Cluster Using the HX Connect UI
Note
Hypercheck Health Check Utility— Cisco recommends running this proactive health check utility on your HyperFlex cluster prior to upgrade. These checks provide early visibility into any areas that may need attention and will help ensure a seamless upgrade experience. For more information see the Hyperflex Health & Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck.
Important
Use the HX Connect UI when upgrading from a current HX Data Platform version of 2.5(1a) or later releases.
Procedure
Step 1
If UCSM (A-bundle) or UCS Server Firmware (C-bundle) upgrade is required, download Cisco UCS Infrastructure A, blade bundle B, and rack bundle C. See Downloading Software for more details.
Step 2
Ensure that the hx-storage-data and vMotion upstream switches are configured for full network failover capability before proceeding forward. Otherwise, the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts. See Test Upstream Network Connectivity for more details.
Step 3
Upgrade Cisco UCS Infrastructure bundle as required. See Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager for more details.
Note
It is important that you manually upgrade the UCS infrastructure first before initiating the upgrade sequence of the HyperFlex components as outlined in Recommended Upgrade Method, on page 1. The upgrade feature of the HX Platform software will not upgrade the UCS infrastructure bundle. This upgrade is a manual process.
Step 4
Bootstrap to upgrade Cisco HX Data Platform.
Note
If you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI (Auto Bootstrap Upgrade Process from HX Connect UI). However, if you are running a HyperFlex release that is earlier than release 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform (Manual Bootstrap Upgrade Process).
Step 5
Log in to HX Connect.
- Enter the HX Storage Cluster management IP address in a browser. Navigate to https://storage-cluster-management-ip.
- Enter the administrative username and password.
- Click Login.
Step 6
In the Navigation pane, select Upgrade.
Step 7
Choose the type of upgrade from the Select Upgrade Type page.
Caution
After manual bootstrap, validation will fail if you perform UCS only, ESXi only, or UCS and ESXi combined upgrade. For a successful upgrade, Cisco recommends the following upgrade types:
- HX Data Platform only upgrade, followed by UCS Firmware and/or Hypervisor Software upgrade
- HX Data Platform and UCS Firmware
- HX Data Platform and Hypervisor Software
- HX Data Platform, UCS Firmware, and Hypervisor Software.
Step 8
Depending on the type of upgrade you want to perform, complete the following fields on the Enter Credentials tab.
UCS Server Firmware
| Field | Essential Information |
| UCS Manager Hostname field | Enter the Cisco UCS Manager FQDN or IP address. Example: 10.193.211.120. |
| User Name field | Enter the Cisco UCS Manager <admin> username. |
| Admin Password field | Enter the Cisco UCS Manager <admin> password. |
| Discover button | Click Discover to view the current UCS firmware package version, in the Current Version field. |
HX Data Platform
| UI Element | Essential Information |
| Drag the HX file here or click to browse | Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Download Software – HyperFlex HX Data Platform. Sample file name format: storfs-packages-3.5.2a-31601.tgz. |
| Current version | Displays the current HyperFlex Data Platform version. |
| Current cluster details | Lists the HyperFlex cluster details like the HyperFlex version and Cluster upgrade state. |
| Bundle version | Displays the HyperFlex Data Platform version of the uploaded bundle. |
| (Optional) Checksum field | The MD5 Checksum number is stored in a separate text file at the /tmp directory where the upgrade package was downloaded. This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle. |
ESXi
Note
The ESXi upgrade option is supported in the HyperFlex Connect UI for HyperFlex release 3.5(1a) or later.
| UI Element | Essential Information |
| Drag the ESXi file here or click to browse field | Upload the latest Cisco HyperFlex Custom Image Offline Bundle for upgrading existing ESXi hosts from Download Software – HyperFlex HX Data Platform. Example: HX-ESXi-6.5U2-10884925-Cisco-Custom-6.5.2.4-upgrade-bundle.zip. |
| Current version field | Displays the current ESXi version. |
| Current hypervisor details field | Lists the HyperFlex cluster details like the Hypervisor version and Cluster upgrade state. |
| Bundle details field | Displays the ESXi version of the uploaded bundle. |
vCenter Credentials
| UI Element | Essential Information |
| User Name field | Enter the vCenter <admin> username. |
| Admin Password field | Enter the vCenter <admin> password. |
Step 9
Click Upgrade to begin the cluster upgrade process.
Step 10
The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete. When the upgrade is in progress, you may see an error message, ‘Websocket connection failed. Automatic refresh disabled’. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message. During cluster upgrade, if the orchestration node reboots or power cycles due to a power issue, the cluster upgrade will be stuck. Once the node is up, restart the cluster upgrade process after cleaning the cluster system using the following command:
stcli cluster upgrades –components hxdp –clean.
If the clean-up command fails, restart the stMgr service on all control VMs (ctrlVM) by running the following command:
restart stMgr
Then, clean the cluster system by rerunning the following command:
stcli cluster upgrade –components hxdp –clean
Here is a sample code:
- :~# stcli cluster upgrade –clean –components hxdp
- Forcefully cleaned up upgrade progress
- :~# stcli cluster upgrade –status
- No active upgrade was found. Upgrade progress is available after triggering an upgrade.
Post vCenter Upgrade Tasks
If the extension is not working, and HyperFLex and vCenter are upgraded to compatible versions, perform the following steps:
Note
If you have more than one HyperFlex cluster, then you must first update all HX clusters to compatible HX versions for the corresponding vCenter versions prior to attempting to reregister. Do not unregister
com.springpath.sysmgmt unless all clusters are removed from the vCenter.
Before you begin
Verify the extension is working. If so, then there is no need to perform any post-upgrade tasks.
Procedure
Step 1
Try to re-register the extension. If the extension still does not work, then continue with the next steps.
Step 1
Unregister the extension.
Example:
com.spring path.sysmgmt.domain-<id>
com.spring path.sysmgmt Use the mob browser https://<vCenter IP or FQDN>/mob (content > extension manager path and Invoke UnregisterExtension method).
Note
We recommend removing the cluster before unregistering extensions.
Step 3
Re-register the Springpath plug-in using:
Example:
stcli cluster reregister
Note
You can use stcli cluster reregister –-for help and then continue with the reregistration.
Online Upgrade Process Workflow
Attention
If you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI (Auto Bootstrap Upgrade Process from HX Connect UI). However, if you are running a HyperFlex release that is earlier than release 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform (Manual Bootstrap Upgrade Process).
When using the online upgrade process workflow, consider the following:
- First, upgrade Cisco UCS infrastructure to the latest version and then use the automated upgrade workflow for a combined upgrade of Cisco UCS firmware and Cisco HX Data Platform. The online upgrade uses host firmware packages to upgrade all server endpoints.
- During the online upgrade, as one node is being upgraded (placed into maintenance mode), the number of tolerated node failures is reduced based on the Data Replication Factor and the Access Policy settings. See Entering Cisco HyperFlex Maintenance Mode for the procedures on how to access Cisco HyperFlex Maintenance Mode.
- If upgrading both HXDP and UCS firmware, the combined upgrade can be selected through HX Connect depending on the length of the maintenance window.
- Do not use the Firefox browser. It is not supported due to an outdated version of flash that is bundled with the browser.
Note
Cisco UCS Manager infrastructure upgrade is only supported using AutoInstall and the direct server firmware upgrade should be performed only through the upgrade orchestration framework provided by the HX Data Platform Plug-in.
Note
During the online upgrade process, do not acknowledge server reboot pending activities from UCS Manager. Doing so will interrupt the upgrade process and can cause a storage outage. HyperFlex will automatically reboot each node.
The following table summarizes the online upgrade workflow:
| Step | Description | Reference |
| 1. | If UCSM (A-bundle) or UCS Server Firmware (C-bundle) the upgrade is required, download Cisco UCS Infrastructure A, blade bundle B, and rack bundle C. |
Downloading Software |
| 2. | Ensure that the hx-storage-data and vMotion upstream switches are configured for full network failover capability before proceeding forward. Otherwise, the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts. |
Test Upstream Network Connectivity |
| 3. | Upgrade Cisco UCS Infrastructure bundle as required. Note It is important that you manually upgrade the UCS infrastructure first before initiating the upgrade sequence of the HyperFlex components as outlined in Recommended Upgrade Method, on page 1 . The upgrade feature of the HX Platform software will not upgrade the UCS infrastructure bundle. This upgrade is a manual process. |
Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager |
| Step | Description | Reference |
| 4. | Bootstrap to upgrade Cisco HX Data Platform. | HX Release 3.5(1a) and later: • Auto bootstrap is supported beginning with HX release 3.5(1a). Auto Bootstrap Upgrade Process from HX Connect UI
HX Release earlier than 3.5(1a): • Manual cluster bootstrap is required HX releases earlier than 3.5(1a). Manual Bootstrap Upgrade Process |
| 5. | Disable the snapshot schedule, on the bootstrapped storage controller VM. Note It is enough to run this script on one of the controller nodes. |
Run the command stcli snapshot-schedule –disable. |
| 6. | Log in to HX Connect with administrator credentials. | |
| 7. | Start combined upgrade of: • HX Data Platform and UCS Firmware • HX Data Platform and Hypervisor Software |
Upgrading Your HyperFlex Cluster Using the HX Connect UI, on page 2 |
| Attention To perform a split upgrade, you must upgrade HX Data Platform first. After HX Data Platform is upgraded to 3.5(1x), you can perform a split upgrade of UCSM only and/or ESXi only. When only UCS firmware is being upgraded, you may see the upgrade process pause at the validation screen after the fabric interconnect discovery. It may be a network connectivity failure issue, however, in most cases, it just requires waiting for the process to finish. |
Upgrading Cisco HyperFlex Data Platform Using HX Connect Upgrading Cisco UCS Server Firmware Using the HX Connect UI | |
| 8. | Confirm that the upgrade task is complete. | Post Upgrade Tasks |
| 9. | Dynamic certificate creation. | Starting with 4.0(2a) release, Dynamic self-signed certificate are generated rather than static certificates. |
| 10. | On the same controller VM, enable snapshot schedule. | Run the command stcli snapshot-schedule -–enable. |
Offline Upgrade Process Workflow
The following table summarizes the offline upgrade workflow:
| Step | Description | Reference |
| 1. | If UCSM (A-bundle) or UCS Server Firmware (C-bundle) upgrade is required, download Cisco UCS Infrastructure A, blade bundle B, and rack bundle C. | Downloading Software |
| 2. | Ensure that the hx-storage-data and vMotion upstream switches are configured for full network failover capability before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts. |
Test Upstream Network Connectivity |
| 3. | Upgrade Cisco UCS Infrastructure bundle as required. Note It is important that you manually upgrade the UCS infrastructure first before initiating the upgrade sequence of the HyperFlex components as outlined in Recommended Upgrade Method, on page 1. The upgrade feature of the HX Platform software will not upgrade the UCS infrastructure bundle. This upgrade is a manual process. |
Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager |
| Step | Description | Reference |
| 4. | Launch the vSphere Web Client and power down all user VMs residing on HX servers and all user VMs running on HX datastores. This includes VMs running on computer-only nodes. After the VMs have been shut down, verify the health state of the cluster and perform a graceful shutdown. Important HyperFlex controller VMs (stCtlVMs) must remain powered on. |
Graceful Shutdown of a HX Cluster |
| 5. | Manually stage the correct firmware version before starting the upgrade process. | Modifying Host Firmware Package Using Cisco UCS Manager |
| 6. | Shutdown the HyperFlex Controller VMs (stCtlVMs). | In vCenter, right-click on each HX Controller VM (stCtlVM) and select Power > Shut Down Guest OS. |
| 7. | Once the Controller VMs are shutdown, place the ESXi hosts into Maintenance Mode. | In vCenter, right-click on each ESXi host select Maintenance Mode > Enter Maintenance Mode. |
| 8. | Acknowledge the pending reboot on the servers that comprise your HX cluster nodes, including both converged nodes and computer-only nodes connected to the cluster. Wait until all nodes are upgraded. Confirm that correct firmware packages have been installed before proceeding. |
|
| 9. | Once the ESXi hosts have booted, take them out of Maintenance Mode. Now the controller VM should come back online. | In vCenter, right-click on each ESXi host select Maintenance
Mode > Exit Maintenance Mode. |
| Step | Description | Reference |
| 10. | Bootstrap to upgrade the Cisco HX Data Platform Plug-in. Important • Be sure to copy the bootstrap file to the controller VM /tmp directory. • Ensure that you confirm the version of the plug-in in the vCenter Administration > Client Plug-Ins page. |
Manual Bootstrap Upgrade Process |
| 11. | Disable snapshot schedule, on the bootstrapped storage controller VM. Note It is enough to run this script on one of the controller nodes. |
Run the command stcli snapshot-schedule –disable. |
| 12. | From the same controller VM, begin the upgrade. | Upgrading Your HyperFlex Cluster Using the HX Connect UI, on page 2 |
| 13. | Confirm that upgrade is complete. | Post Upgrade Tasks |
| 14. | After the upgrade is complete, start the cluster and power on VMs. | Start Cluster and Power On VMs , on page 12 |
| 15. | On the same controller VM, enable snapshot schedule. | Run the command stcli snapshot-schedule -–enable. |
Offline Upgrade Guidelines
Important
ucsm-host and ucsm-user parameters are required when you are upgrading from 1.7x to 1.8x. These parameters must not be used when moving up from 1.8(1a)/1.8(1b) to 2.0(1a) as we are not changing the Cisco UCS server firmware version. Cisco recommends using HX Connect UI to upgrade HyperFlex clusters over CLI for ease of use and better reporting. You can safely ignore Cluster Invalid State alerts in HX Connect.
Before you proceed, with either a combined or split upgrade consider the following guidelines:
- The package name must match the file that you uploaded to the controller VM.
- Enter passwords when prompted.
- Nodes are upgraded with the new version of the Cisco HX Data Platform software and rebooted one at a time.
- Offline cluster upgrades with nested vCenter is not supported.
Offline Upgrade Using CLI
Important
If you need to perform a split upgrade, you must upgrade HX Data Platform first. After HX Data Platform is upgraded to Release 3.5(1x), you can perform a split upgrade of UCSM only and/or ESXi only.
Combined Upgrade of Cisco HX Data Platform, ESXi and Cisco UCS Firmware
M5 Servers
stcli cluster upgrade –components ucs-fw, hxdp, hypervisor –location/tmp/ <storfs package name,ESXi package name> –ucsm-host <IP/FQDN of UCSM> ucsm-user <UCSM User> –ucsm5-fw-version <UCSM Firmware Version>
Example for M5 Servers:
~# stcli cluster upgrade –components ucs-fw, hxdp, hypervisor –location /tmp/storfs-packages-3.5.1a-19712.tgz –ucsm-host eng-fi16.eng.storvisor.com –ucsm-user admin –ucs5fw-version ‘3.1(2g)’
M4 Servers
# stcli cluster upgrade –components ucs-fw, hxdp, hypervisor –location/tmp/ <storfs package name, ESXi package name> –ucsm-host <IP/FQDN of UCSM> –ucsm-user <UCSM User> –ucsfw-version <UCSM Firmware Version>
Example for M4 Servers:
~# stcli cluster upgrade –components ucs-fw, hxdp, hypervisor –location
/tmp/storfs-packages-3.5.1a-19712.tgz
–ucsm-host eng-fi16.eng.storvisor.com –ucsm-user admin –ucsfw-version ‘3.1(2g)’
Combined Upgrade of Cisco HX Data Platform and ESXi
M5 Servers
stcli cluster upgrade –components hxdp,hypervisor –location /tmp/hxupgrade_bundle.tgz –hypervisor-bundle /tmp/esxiupgrade_bundle.zip
Example for M5 Servers:
stcli cluster upgrade –components hxdp,hypervisor –location /tmp/hxupgrade_bundle.tgz
- hypervisor-bundle /tmp/esxiupgrade_bundle.zip
M4 Servers
# stcli cluster upgrade –components hxdp,hypervisor –location /tmp/hxupgrade_bundle.tgz –hypervisor-bundle /tmp/esxiupgrade_bundle.zip
Example for M4 Servers:
~# stcli cluster upgrade –components hxdp,hypervisor –location /tmp/hxupgrade_bundle.tgz
- hypervisor-bundle /tmp/esxiupgrade_bundle.zip
Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware
M5 Servers
# stcli cluster upgrade –components hxdp,ucs-fw –location/tmp/ <storfs package name> –vcenter-user <vcuser> –ucsm-host <IP/FQDN of UCSM> –ucsm-user <UCSM User> –ucsm5-fw-version <UCSM Firmware Version>
M4 Servers
# stcli cluster upgrade –components hxdp,ucs-fw –location/tmp/ <storfs package name> –vcenter-user <vcuser> –ucsm-host <IP/FQDN of UCSM> –ucsm-user <UCSM User> –ucsfw-version <UCSM Firmware Version>
Example for M4 Servers:
~# stcli cluster upgrade –components hxdp,ucs-fw –location /tmp/storfs-packages-1.8.1c-19712.tgz –vcenter-user –ucsm-host eng-fi16.eng.storvisor.com –ucsm-user admin –ucsfw-version ‘3.1(2b)’.
Start Cluster and Power On VMs
After the upgrade is complete and the cluster has been upgraded, log out and log back in to vCenter to see upgrade changes.
Procedure
Step 1
After the upgrade is complete, start your cluster.
Step 2
Login to any controller VM through SSH.
# stcli cluster start
Example:
HyperFlex StorageController 1.8(1c) Last login: Wed Sept 21 23:54:23 2016 from pguo-dev.eng.storvisor.com – 384 -1;~# stcli cluster upgrade-status Cluster upgrade succeeded. Cluster version: 1.8(1c) ;~# stcli cluster start waiting for Cluster to start on nodes: [ucs-383, ucs-384, ucs-385, ucs-386]
This will start the cluster and mount the HX datastores. Wait for the cluster to come online. You will see the prompt:
Started cluster on nodes; [ucs-383, ucs-384, ucs-385, ucs-386] Cluster is online ;~#.
Step 3
Wait for cluster to become healthy before starting the VMs. Run command:
~# stcli clustr info| grep health
Example:
;~# stcli cluster info | grep health state: healthy.
state: healthy
storage cluster is healthy
Step 4
After the cluster is healthy, launch vSphere Web Client or Thick Client, navigate to Hosts and Cluster > Datacenter > Cluster > . Right-click, select Power> Power On to start the VMs.