It is very nice to have switches connected in a virtual switch, where link aggregations can span different stack members. And very nice to manage many switches as one multi slot chassis also.
However upgrading stack can be painful sometimes, when stack couldn't contain members with different software versions.
For example, when you try to add a member to a H3C stack, newcomer will be provided with the software from the stack master - the whole stack must have the same software version.
So in order to upgrade the whole stack you can either configure new boot files for all member and reboot the whole stack at once (then the first switch to come up already with the new software will accept all others also with the new software)
or
you can use special procedure named ISSU. ISSU stands for in service software upgrade. During this procedure the stack is kept online, while you perform one-by-one upgrade of members: when one has joined back the stack, you can proceed to the next one till the very last.
This procedure effectively provides you with a zero downtime for servers if they are connected with link aggregated interfaces to different stack member.
How to perform ISSU is very well described for example here:
What this guide fails to describe - possible problems you can face during upgrade. Yesterday I ran into one of such problems and decided to write a post about it, because I was unable to google for a solution.
So as per manual one have to initiate ISSU with the command: issu load file
I have done several upgrades before, so the output I saw came as a surprise:
[h3c]issu load file A5800_5820X-CMW520-R1211.bin slot 2
Error: Different running boot files on MPUs.
I decided to check boot files of the stack:
[h3c]disp boot-loader
Slot 1
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
Slot 2
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
Slot 3
The current boot app is: flash:/s5800_5820x-cmw520-r1211.bin
The main boot app is: flash:/s5800_5820x-cmw520-r1211.bin
The backup boot app is: flash:/
Slot 4
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
Slot 5
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
Slot 6
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
Slot 7
The current boot app is: flash:/s5800_5820x-cmw520-r1211.bin
The main boot app is: flash:/s5800_5820x-cmw520-r1211.bin
The backup boot app is: flash:/
Slot 8
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
You can see, that slots 3 and 7 had the same software as the whole stack, but the file name was different (thanks to HP re-branding - H3C s5800 series switches became HP A5800 series switches). The difference was due to recent replacement of 6 stack members with new models (I managed to replace 6 members one-by-one while keeping stack online). As the software version was the same R1211, stack recognized them as compatible and didn't provide with the boot files from master.
After checking that files on slot 1 and 3 have the same size and creation date I decided to rename boot files (and update boot-loaders) for slots 3 and 7 (slot 3 commands are shown):
<h3c>rename slot3#flash:/s5800_5820x-cmw520-r1211.bin slot3#flash:/A5800_5820X-CMW520-R1211.bin
Rename slot3#flash:/s5800_5820x-cmw520-r1211.bin to slot3#flash:/a5800_5820x-cmw520-r1211.bin?[Y/N]:y
%Renamed file slot3#flash:/s5800_5820x-cmw520-r1211.bin to slot3#flash:/a5800_5820x-cmw520-r1211.bin.
<h3c>boot-loader file slot3#flash:/A5800_5820X-CMW520-R1211.bin slot 3 main
This command will set the boot file of the specified board. Continue? [Y/N]:y
The specified file will be used as the main boot file at the next reboot on slot 3!
I have checked boot files again but saw that renaming corrected only main boot file (which will be effective after reboot) but current boot app remained s5800... (output for some slots is omitted):
<h3c>disp boot-loader
Slot 1
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
Slot 2
...
Slot 3
The current boot app is: flash:/s5800_5820x-cmw520-r1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
Slot 4
...
Slot 5
...
Slot 6
...
Slot 7
The current boot app is: flash:/s5800_5820x-cmw520-r1211.bin
The main boot app is: flash:/s5800_5820x-cmw520-r1211.bin
The backup boot app is: flash:/
Slot 8
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
I have rebooted slot 3 and after it rejoined the stack rebooted slot 7 (again slot 3 command only shown):
<h3c>reboot slot 3
This command will reboot the specified board, Continue? [Y/N]:y
Now rebooting, please wait...
Important: Rebooting two switches at once will lead to split brain scenario of a stack. To keep stack up and running reboot units only one by one and always check that rebooted switch rejoined the stack before rebooting the next.
After reboot boot files were the same (output for some slots is omitted):
<h3c>disp boot-loader
Slot 1
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
Slot 2
...
Slot 3
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
Slot 4
...
Slot 5
...
Slot 6
...
Slot 7
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
Slot 8
The current boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The main boot app is: flash:/A5800_5820X-CMW520-R1211.bin
The backup boot app is: flash:/
And ISSU allowed me to begin upgrade process:
[h3c]issu load file a5800_5820x-cmw520-r1211p04.bin slot 2
This command will begin ISSU, and the specified board will reboot and be upgraded. Please save the current running configuration first; otherwise, the configuration may be lost. Continue? [Y/N]:y
[h3c]