Disk Space Error when using HPSUM from HP Scripting Toolkit

By | July 1, 2015

I have a blog that I never published that detailed how I am using the HP Scripting Toolkit 10.04 (STK – a bootable SUSE image) to run hpsum in an ‘offline’ mode. Basically I would boot from the HP Scripting Toolkit environment, mount a nfs share that had the SPP, copy it locally and then run hpsum.

This has worked fine for a few years (last SPP I tried was the September SPP I believe) but I recently tried the June SPP and ran into some issues.

First issue was that the SPP does a space check on the /tmp directory. The STK loads up into ram, but some of it is rootfs and some ramfs. The end result is that /tmp (which is on rootfs) has 0 KB free as returned by df.

The workaround for this is to mount a small tmpfs filesystem as /tmp

http://www.thegeekstuff.com/2008/11/overview-of-ramfs-and-tmpfs-on-linux/

# mount -t tmpfs -o size=5g tmpfs /tmp

I then ran into another issue where hpsum was looking for space on / (which is also rootfs).
Apparently the SPP/HPSUM has moved to a a linux RPM based format for delivering the packages (what happened to the good ‘ol .scexe ?).
The packages seem to either extra into a folder on / or into /tmp/HPSUM/localhost.

I couldn’t mount on top of / so I did some more digging.

I ran rpm -qlp on a few of the rpms and I found that they try to install to:
/usr/lib/i386-linux-gnu
/usr/lib/x86_64-linux-gnu

# mkdir /usr/lib/i386-linux-gnu
# mkdir /usr/lib/x86_64-linux-gnu
# mount -t tmpfs -o size=1g tmpfs /usr/lib/i386-linux-gnu
# mount -t tmpfs -o size=1g tmpfs /usr/lib/x86_64-linux-gnu

I re-ran hpsum and everything started to work!

I did notice that HP may have changed some flags to hpsum. In previous versions most of the flags were like /romonly (with the forward slash). I think to differentiate between Windows/DOS-style and Linux-style command prompts, the Windows will use the slash and Linux will use “–“, I don’t think the change is being enforced though, I used the DOS-style flags just fine.

EDIT 7/1/2015: I have not been able to apply the SPP using the conventional method of attaching the ISO (both the auto and interactive method fail). The server ends up rebooting at a point in the deployment or hardware scan. I am also having issues with PXE booting on UEFI AND legacy (legacy worked before). I’m going to open a case or two tomorrow.

24 thoughts on “Disk Space Error when using HPSUM from HP Scripting Toolkit

  1. sten spans

    I used similar tricks to get 2015-04 to work, by remounting the initrd on tmpfs.

    Sadly 2015-06 fails for me too, please let me know if you find a fix.

    Reply
    1. Chris Post author

      I skipped the 2015-04 so this was my first encounter. What issues are you seeing? Applying the SPP has been a little hit or miss. One blade didn’t like auto and if I did interactive sometimes hpsum would launch and sometimes it didn’t. Uefi boot was a little more successful. After resetting the blade and VC profile my legacy pxe boot worked but uefi pxe (which I never really got working) was worse than the previous firmware. I’m also seeing issues with virtual connect vs fed for pxe booting.

      Reply
    2. Chris Post author

      I got 2015-10 to work with a number of tricks (copying files from the SPP initrd to toolkit initrd). I haven’t tested 2016-04 but that will probably be next.

      Reply
  2. Thomas

    I ran into the same issue, mounted spp iso image over nfs does not apply any updates, in the logs /var/hp/localhost/.. I found some strange messages about missing libraries (libstdc++, libpthread and so on)
    In my pxe setup I use a multivendor environment with hp and dell, I use not the dell deployment toolkit to apply hp firmwareupdates 🙂
    It works as I expected from hp, in summary I can say: Applying hp firmware with hp scripting toolkit does not work 🙁

    Reply
    1. Thomas

      sorry, typo:
      “use not the dell deployment” should be “use now the dell deployment”

      Reply
    2. Chris Post author

      Unfortunately I don’t think HP is putting much effort into the hp scripting toolkit as OneView and Synergy become their preferred methods.

      Reply
    3. Chris Post author

      I saw those errors as well. The issue is that they are including all the necessary libraries on the SPP but not on the scripting toolkit. I unpacked the initrd.img for both the SPP and scripting toolkit, did a diff on both /usr/lib and /usr/lib64 directories, and then did a one way rsync (without deleting extra files) from the SPP /usr/lib and /usr/lib64 to the Toolkit.

      I then repacked the initrd and booted off of that. I can post some commands if you’re interested.

      Reply
  3. Thomas

    Indeed, but the small scripting toolkit and its tools is a very flexible method for deploying systems in restricted environments and it has a small footprint without installing a lot of centralized software and their dependencies. I’m using STK (formerly SSSTK) since years with success. Until the new SPP came out with rpm’s instead of the good old .scexe files. As stated, give the dell toolkit a chance, it does some things even better as the hp’s one. But, for using the hp tools like conrep, hponcfg and hpssascripting you must use the hp toolkit. In this case you can set up a flexible toolkit. If you are interested in, I can give you an example.
    And, opening a support case will not be helpful, my experiences are mostly bad – especially if the topic is tools 🙁

    Reply
    1. Chris Post author

      I totally agree, I’ve been using my pxe based workflow with the toolkit for three years and have had similar experience with HP tickets

      Reply
  4. Thomas

    That’s just how it is. If HP is not willing to get their tools working as expected, we have to find workarounds furthermore :-).

    kind regards from germany, Thomas

    Reply
  5. Thomas

    I’ve found a very dirty hack and tested sucessfully:

    – Make “disk” space in hp toolkit environment (useful in an updatescript for hp stk):
    mkdir -p /usrtmp
    mkdir -p /sbintmp
    mount -t tmpfs -o size=1g tmpfs /usrtmp
    mount -t tmpfs -o size=1g tmpfs /sbintmp
    mount -t tmpfs -o size=1g tmpfs /opt
    cp -a /usr/* /usrtmp
    cp -a /sbin/* /sbintmp
    /sbintmp/rm -rf /usr
    /sbintmp/rm -rf /sbin
    /sbintmp/ln -s /sbintmp /sbin
    /sbintmp/ln -s /usrtmp /usr

    – Install hpsum itself (available as rpm package at hpe.com, use package for sles11 x86_64)
    – mount SPP iso image from nfs share or use a ftp target
    – use hpsum as it made for, see http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=c04774377
    the cli syntax for the installed hpsum has more options than hpsum from SPP, only hp knows why…
    But: the “old” call hpsum -s -romonly -use_location still works

    Reply
    1. Chris Post author

      I have been working on the 201604 SPP and I have found a couple of tricks
      I copied the ‘shutdown’ (/bin/shutdown I think) from one of the older scripting toolkit initrd.img files to the new one (hpsum wants that file).

      I then noticed that the rpm installs do file system checks on /, which sucks because the / directory is created as ramfs which doesn’t work with df.

      cd /
      mkdir /newroot
      mount -t tmpfs -o size=5g tmpfs /newroot
      cp -a bin /newroot/
      ls -al /newroot
      cp -a bootstrap.sh /newroot/
      cp -a dev /newroot/
      cp -a etc /newroot/
      cp -a init /newroot/
      ap -a lib* /newroot/
      cp -a linuxrc /newroot/
      cp -a loadnet.sh /newroot/
      cp -a media /newroot/
      cp -a mnt /newroot/
      cp -a /opt /newroot/
      cp -a phoenix /newroot/
      mkdir /newroot/proc
      cp -a root /newroot/
      cp -a run /newroot/
      cp -a sbin /newroot/
      mkdir /newroot/sys
      cp -a tmp /newroot
      cp -a usr /newroot
      cp -a var /newroot/
      mount –move /sys /newroot/sys
      mount –move /proc /newroot/proc
      mount –move /dev /newroot/dev
      exec switch_root /newroot /init

      So far this seems to have worked

      Reply
  6. Thomas

    cool stuff. You made HPE’s work once again 🙂
    Their toolkit is getting more and more useless. I tried today to run hpdiagnostics on a Gen9 system, but there were a message that hpdiagnostics (in initrd of the latest toolkit!) is not supported on this system.
    Another hack was needed to put the contents of the hpdiagnostis rpm to the target system in toolkit environment.

    Reply
    1. Chris Post author

      I’m not looking forward to when (or if) they move to squashfs like they have on the spp. Haven’t cracked that cookie yet.

      Reply
    2. Chris Post author

      I spoke to a few people who manage the scripting toolkit and they will be building the hpsum capability back in.

      Reply
    1. Chris Post author

      Not too sure, but the developer was right on top of it. He analyzed all of the stuff I sent them right away and included at least some of the modules. My guess is the next release.

      Reply
  7. slgray

    Have the same issue using STK10.30 with SPP201604.
    Since SPP is supported to boot from PXE, I try to boot SPP but it fail to mount network share to obtain SPP ISO file.
    I found the initrd.img of SPP don’t have nfs and cifs included, I think I maybe easy hack this initrd.img.

    Reply
    1. Chris Post author

      Let me know how it goes. Thanks for reminding me about the initrd.img in the pxe folder of the SPP. I remember comparing the contents of that img to the SPP in order to figure out what library files were missing. At some point I forgot about all of that and got confused with the squashfs format that they are now using.

      Reply
      1. slgray

        I have tried to use the pxe form spp, the init script in the initrd.img of pxe just chroot to the squashfs and run the init script of squashfs . And it works just like boot form iso file. I think your chroot way is better than this way and I now use this method in my environment.

        Thanks for your sharing.

        Reply
  8. Thomas

    Indeed, the squashfs format is the fail. The installation of the firmware rpm’s need some space in / absolutely, there is no space for. You can relocate the rpm installation manually, but not in hpsum and/or scripted environments. Unfortunately your chroot example (comment from June 27) is not working in my setup (fully scripted).
    Meanwhile I’ve extended my framework with UEFI Boot, there is the good old one dell toolkit unusable (kernel too old message).
    What about to create an own kernel/initrd (small livesystem with all needed hpe drivers) to get the hpe stuff working?
    Next month I have to update 50 servers in a HPC Cluster 🙁

    Reply
  9. Thomas

    I’ve got it. It works now with the hpe toolkit, I’ve created a chroot environment within a set of scripts.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *