Recording of my LCA2017 talk: Organizational Change: Challenges in shipping open source firmware
Reply
Tag Archives: OpenPower
Workaround for opal-prd using 100% CPU
opal-prd is the Processor RunTime Diagnostics daemon, the userspace process that on OpenPower systems is responsible for some of the runtime diagnostics. Although a userspace process, it memory maps (as in mmap) in some code loaded by early firmware (Hostboot) called the HostBoot RunTime (HBRT) and runs it, using calls to the kernel to accomplish any needed operations (e.g. reading/writing registers inside the chip). Running this in user space gives us benefits such as being able to attach gdb, recover from segfaults etc.
The reason this code is shipped as part of firmware rather than as an OS package is that it is very system specific, and it would be a giant pain to update a package in every Linux distribution every time a new chip or machine was introduced.
Anyway, there’s a bug in the HBRT code that means if there’s an ECC error in the HBEL (HostBoot Error Log) partition in the system flash (“bios” or “pnor”… the flash where your system firmware lives), the opal-prd process may get stuck chewing up 100% CPU and not doing anything useful. There’s https://github.com/open-power/hostboot/issues/67 for this.
You will notice a problem if the opal-prd process is using 100% CPU and the last log messages are something like:
HBRT: ERRL:>>ErrlManager::ErrlManager constructor. HBRT: ERRL:iv_hiddenErrorLogsEnable = 0x0 HBRT: ERRL:>>setupPnorInfo HBRT: PNOR:>>RtPnor::getSectionInfo HBRT: PNOR:>>RtPnor::readFromDevice: i_offset=0x0, i_procId=0 sec=11 size=0x20000 ecc=1 HBRT: PNOR:RtPnor::readFromDevice: removing ECC... HBRT: PNOR:RtPnor::readFromDevice> Uncorrectable ECC error : chip=0,offset=0x0
(the parameters to readFromDevice may differ)
Luckily, there’s a simple workaround to fix it all up! You will need the pflash utility. Primarily, pflash is meant only for developers and those who know what they’re doing. You can turn your computer into a brick using it.
pflash is packaged in Ubuntu 16.10 and RHEL 7.3, but you can otherwise build it from source easily enough:
git clone https://github.com/open-power/skiboot.git cd skiboot/external/pflash make
Now that you have pflash, you just need to erase the HBEL partition and write (ECC) zeros:
dd if=/dev/zero of=/tmp/hbel bs=1 count=147456 pflash -P HBEL -e pflash -P HBEL -p /tmp/hbel
Note: you cannot just erase the partition or use the pflash option to do an ECC erase, you may render your system unbootable if you get it wrong.
After that, restart opal-prd however your distro handles restarting daemons (e.g. systemctl restart opal-prd.service) and all should be well.
Compiling your own firmware for Barreleye (OpenCompute OpenPOWER system)
Aaron Sullivan announced on the Rackspace Blog that you can now get your own Barreleye system! What’s great is that the code for the Barreleye platform is upstream in the op-build project, which means you can build your own firmware for them (just like garrison, the “IBM S822LC for HPC” system I blogged about a few days ago).
Remarkably, to build an image for the host firmware, it’s eerily similar to any other platform:
git clone --recursive https://github.com/open-power/op-build.git cd op-build . op-build-env op-build barreleye_defconfig op-build
…and then you wait. You can cross compile on x86.
You’ve been able to build firmware for these machines with upstream code since Feb/March (I wouldn’t recommend running with builds from then though, try the latest release instead).
Hopefully, someone involved in OpenBMC can write on how to build the BMC firmware.
Compiling your own firmware for the S822LC for HPC
IBM (my employer) recently announced the new S822LC for HPC POWER8+NVLINK NVIDIA P100 GPUs server (press release, IBM Systems Blog, The Register). The “For HPC” suffix on the model number is significant, as the S822LC is a different machine. What makes the “for HPC” variant different is that the POWER8 CPU has (in addition to PCIe), logic for NVLink to connect the CPU to NVIDIA GPUs.
There’s also the NVIDIA Tesla P100 GPUs which are NVIDIA’s latest in an SXM2 form factor, but instead of delving into GPUs, I’m going to tell you how to compile the firmware for this machine.
You see, this is an OpenPOWER machine. It’s an OpenPOWER machine where the vendor (in this case IBM) has worked to get all the needed code upstream, so you can see exactly what goes into a firmware build.
To build the latest host firmware (you can cross compile on x86 as we use buildroot to build a cross compiler):
git clone --recursive https://github.com/open-power/op-build.git cd op-build . op-build-env op-build garrison_defconfig op-build
That’s it! Give it a while and you’ll end up with output/images/garrison.pnor – which is a firmware image to flash onto PNOR. The machine name is garrison as that’s the code name for the “S822LC for HPC” (you may see Minsky in the press, but that’s a rather new code name, Garrison has been around for a lot longer as a name).
Building OPAL firmware for POWER9
Recently, we merged into the op-build project (the build scripts for OpenPOWER Firmware) a defconfig for building OPAL for (certain) POWER9 simulators. I won’t bother linking over to articles on the POWER9 chip or schedule (there’s search engines for that), but with this commit – if you happen to be able to get your hands on a POWER9 simulator, you can now boot to the petitboot bootloader on it!
We’re using upstream Linux 4.7.0-rc3 and upstream skiboot (master), so all of this code is already upstream!
Now, by no means is this complete. There’s some fairly fundamental things that are missing (e.g. PCI) – but how many other platforms can you build open source firmware for before you can even get your hands on a simulator?
First POWER9 bits merged into skiboot master
I just merged in some base POWER9 support patches into skiboot. While this is in no way near complete or really enough to be interesting to anyone that isn’t heavily involved in POWER9 development, it’s nice to take upstream first and open source first so seriously that this level of base enablement patches is easy to merge in.
Other work that has gone on for POWER9 in open source projects include way back in November 2015 where work for the updated PowerISA 3.0 was merged into binutils and this year there’s been kernel work too.
OpenPOWER, OpenCompute and fostering a firmware development community
Recently, I was at the OpenPOWER Summit in San Jose where people could see the Barreleye server (specs and design here, initial Rackspace blog post here). Barreleye is an OpenCompute form factor POWER8 server. It’s not only an OpenPOWER machine, which means all of the host firmware is free and open source software, but there’s also OpenBMC meaning that the source to the OS and userspace running on the BMC (service processor) is also open!
In addition, the firmware enablement came from Foxconn (see this skiboot commit), which means we’re being successful in enabling people who aren’t part of IBM to join the development community for OpenPOWER firmware and get the changes needed to support their machines accepted upstream.
Granted, the size of a firmware development community is always likely to be relatively small, but I really like how Foxconn has shown leadership to other ODMs on interacting with and becoming part of the OpenPOWER firmware community.
1 Million SQL Queries per second: GA MariaDB 10.1 on POWER8
A couple of days ago, MariaDB announced that MariaDB 10.1 is stable GA – around 19 months since the GA of MariaDB 10.0. With MariaDB 10.1 comes some important scalabiity improvements, especially for POWER8 systems. On POWER, we’re a bit unique in that we’re on the higher end of CPUs, have many cores, and up to 8 threads per core (selectable at runtime: 1, 2, 4 or 8/core) – so a dual socket system can easily be a 160 thread machine.
Recently, we (being IBM) announced availability of a couple of new POWER8 machines – machines designed for Linux and cloud environments. They are very much OpenPower machines, and more info is available here: http://www.ibm.com/marketplace/cloud/commercial-computing/us/en-us
Combine these two together, with Axel Schwenke running some benchmarks and you get 1 Million SQL Queries per second with MariaDB 10.1 on POWER8.
Having worked a lot on both MySQL for POWER and the firmware that ships in the S882LC, I’m rather happy that 1 Million queries per second is beyond what it was in June 2014, which was a neat hack on MySQL 5.7 that showed the potential of MySQL on POWER8 but wasn’t yet a product. Now, you can run a GA release of MariaDB on GA POWER8 hardware designed for scale-out cloud environments and get 1 Million SQL queries/second (with fewer cores than my initial benchmark last year!)
What’s even more impressive is that this million queries per second is in a KVM guest!
PAPR spec publicly available to download
PAPR is the Power Architecture Platform Reference document. It’s a short read at only 890 pages and defines the virtualised environment that guests run in on PowerKVM and PowerVM (i.e. what is referred to as ‘pseries’ platform in the Linux kernel).
https://members.openpowerfoundation.org/document/dl/469
As part of the OpenPower Foundation, we’re looking at ensuring this is up to date, documents KVM specific things as well as splitting out the bits that are common to OPAL and PAPR into their own documents.
gcov code coverage for OpenPower firmware
For skiboot (which provides the OPAL boot and runtime firmware for OpenPower machines), I’ve been pretty interested at getting some automated code coverage data for booting on real hardware (as well as in a simulator). Why? Well, it’s useful to see that various test suites are actually testing what you think they are, and it helps you be able to define more tests to increase what you’re covering.
The typical way to do code coverage is to make GCC build your program with GCOV, which is pretty simple if you’re a userspace program. You build with gcov, run program, and at the end you’re left with files on disk that contain all the coverage information for a tool such as lcov to consume. For the Linux kernel, you can also do this, and then extract the GCOV data out of debugfs and get code coverage for all/part of your kernel. It’s a little bit more involved for the kernel, but not too much so.
To achieve this, the kernel has to implement a bunch of stub functions itself rather than link to the gcov library as well as parse the GCOV data structures that GCC generates and emit the gcda files in debugfs when read. Basically, you replace the part of the GCC generated code that writes the files out. This works really nicely as Linux has fancy things like a VFS and debugfs.
For skiboot, we have no such things. We are firmware, we don’t have a damn file system interface. So, what do we do? Write a userspace utility to parse a dump of the appropriate region of memory, easy! That’s exactly what I did, a (relatively) simple user space app to parse out the gcov gcda files from a skiboot memory image – something we can easily dump out of the simulator, relatively easily (albeit slower) from the FSP on an IBM POWER system and even just directly out of a running system (if you boot a linux kernel with the appropriate config).
So, we can now get a (mostly automated) code coverage report simply for the act of booting to petitboot: https://open-power.github.io/skiboot/boot-coverage-report/ along with our old coverage report which was just for the unit tests (https://open-power.github.io/skiboot/coverage-report/). My current boot-coverage-report is just on POWER7 and POWER8 IBM FSP based systems – but you can see that a decent amount of code both is (and isn’t) touched simply from the act of booting to the bootloader.
The numbers we get are only approximate for any code run on more than one CPU as GCC just generates code that does a load/add/store rather than using an atomic increment.
One interesting observation was that (at least on smaller systems, which are still quite large by many people’s standards), boot time was not really noticeably increased.
For more information on running with gcov, see the in-tree documentation: https://github.com/open-power/skiboot/blob/master/doc/gcov.txt
Building OpenPower firmware for use in POWER8 Simulator
Previously, I blogged on how to Run skiboot (OPAL) on the POWER8 Simulator. If you want to build the full Open Power firmware environment, including the Petitboot bootloader and kernel, you can now do so!
My pull request for an op-build target for the simulator has been merged, so you can now do the following three steps to compile a kernel+initramfs to use with your built skiboot for development purposes:
git clone --recursive git@github.com:open-power/op-build.git cd op-build . op-build-env op-build mambo_defconfig && op-build
Then you wait for a whole bunch of time while everything compiles! Afterwards, you should be left with a zImage.epapr in output/images/ that you can copy into your skiboot directory.
With zImage.epapr in your skiboot directory, when you run “make check”, the skiboot test suite will actually launch the simulator to verify that your skiboot code boots all the way to the petitboot prompt!
We now have two boot tests as part of “make check” for skiboot!
skiboot-4.1
I just posted this to the mailing list, but I’ve tagged skiboot-4.1, so we have another release! There’s a good amount of changes since 4.0 nearly a month ago and this is the second release since we hit github back in July.
For the full set of changes, “git log” is your friend, but a summary of them follows:
- We now build with -fstack-protector and -Werror
- Stack checking extensions when built with STACK_CHECK=1
- Reduced stack usage in some areas, -Wstack-usage=1024 now.
- Some functions could use 2kb stack, now all are <1kb
- Unsafe libc functions such as sprintf() have been removed
- Symbolic backtraces
- expose skiboot symbol map to OS (via device-tree)
- removed machine check interrupt patching in OPAL
- occ/hbrt: Call stopOCC() for implementing reset OCC command from FSP
- occ: Fix the low level ACK message sent to FSP on receiving {RESET/LOAD}_OCC
- hardening to errors of various FSP code
- fsp: Avoid NULL dereference in case of invalid class_resp bits
- abort if device tree parsing fails
- FSP: Validate fsp_msg in fsp_queue_msg
- fsp-elog: Add various NULL checks
- Finessing of when to use error log vs prerror()
- More i2c work
- Can now run under Mambo simulator (see external/mambo/skiboot.tcl) (commonly known as “POWER8 Functional Simulator”)
- Document skiboot versioning scheme
- opal: Handle more TFAC errors.
- TB_RESIDUE_ERR, FW_CONTROL_ERR and CHIP_TOD_PARITY_ERR
- ipmi: populate FRU data
- rtc: Add a generic rtc cache
- ipmi/rtc: use generic cache
- Error Logging backend for bmc based machines
- PSI: Drive link down on HIR
- occ: Fix clearing of OCC interrupt on remote fix
So, who worked on this release? We had 84 csets from 17 developers. A total of 3271 lines were added, 1314 removed (delta 1957).
| Developers with the most changesets | ||
|---|---|---|
| Stewart Smith | 24 | 28.6% |
| Benjamin Herrenschmidt | 17 | 20.2% |
| Alistair Popple | 8 | 9.5% |
| Vasant Hegde | 6 | 7.1% |
| Ananth N Mavinakayanahalli | 5 | 6.0% |
| Neelesh Gupta | 4 | 4.8% |
| Mahesh Salgaonkar | 4 | 4.8% |
| Cédric Le Goater | 3 | 3.6% |
| Wei Yang | 3 | 3.6% |
| Anshuman Khandual | 2 | 2.4% |
| Shilpasri G Bhat | 2 | 2.4% |
| Ryan Grimm | 1 | 1.2% |
| Anton Blanchard | 1 | 1.2% |
| Shreyas B. Prabhu | 1 | 1.2% |
| Joel Stanley | 1 | 1.2% |
| Vaidyanathan Srinivasan | 1 | 1.2% |
| Dan Streetman | 1 | 1.2% |
| Developers with the most changed lines | ||
|---|---|---|
| Benjamin Herrenschmidt | 1290 | 35.1% |
| Alistair Popple | 963 | 26.2% |
| Stewart Smith | 344 | 9.4% |
| Mahesh Salgaonkar | 308 | 8.4% |
| Ananth N Mavinakayanahalli | 198 | 5.4% |
| Neelesh Gupta | 186 | 5.1% |
| Vasant Hegde | 122 | 3.3% |
| Shilpasri G Bhat | 39 | 1.1% |
| Vaidyanathan Srinivasan | 24 | 0.7% |
| Joel Stanley | 21 | 0.6% |
| Wei Yang | 20 | 0.5% |
| Anshuman Khandual | 15 | 0.4% |
| Cédric Le Goater | 12 | 0.3% |
| Shreyas B. Prabhu | 9 | 0.2% |
| Ryan Grimm | 3 | 0.1% |
| Anton Blanchard | 2 | 0.1% |
| Dan Streetman | 2 | 0.1% |
| Developers with the most lines removed | ||
|---|---|---|
| Mahesh Salgaonkar | 287 | 21.8% |
| Developers with the most signoffs (total 54) | ||
|---|---|---|
| Stewart Smith | 44 | 81.5% |
| Vasant Hegde | 4 | 7.4% |
| Benjamin Herrenschmidt | 4 | 7.4% |
| Vaidyanathan Srinivasan | 2 | 3.7% |
| Developers with the most reviews (total 2) | ||
|---|---|---|
| Vasant Hegde | 2 | 100.0% |
skiboot/OPAL versioning
skiboot is boot and runtime firmware for OpenPower systems. There are other components that make up all the firmware you need, but if you’re, say, a Linux kernel, you’re going to be interacting with skiboot.
I recently committed doc/versioning.txt to skiboot to try and explain our current thoughts on versioning releases.
It turns out that picking version numbers is a bit harder than you’d expect, especially when you want to construct a version string to display in places that has semantic meaning. In fact, the writing on Semantic Versioning influenced us heavily.
Since we’re firmware, making incompatible API changes is something we should basically never, ever do. Old kernels should must boot and work on new firmware and new kernels should boot and function on old firmware (and if they don’t, it plainly be a kernel bug). So, ignore the Major version parts of Semantic Versioning for us :)
For each new release, we plan to bump the minor version for mostly bug fix releases, while bump the major version for added functionality. Any additional information is to describe the version on that particular platform – as everybody shipping OPAL is likely to build it themselves with possibly some customizations (e.g. YOUR COMPANY NAME HERE, support for some on board RAID card or on-board automated coffee maker). See doc/versioning.txt for details.
You may wonder why we started at 4.0 for our first real version number. Well… this is purely a cunning plan to avoid confusion with other things, the details of which will only be extracted out of my when plied with a suitable amount of excellent craft beer (because if I’m going to tell a boring story, I may as well have awesome craft beer).
Tyan OpenPower
Good news everyone! Tyan has announced the availability of their first OpenPOWER system! They call this a Customer Reference System, which means it’s an excellent machine to start poking at OpenPower and POWER8 (or deploying applications on).
Because it’s an OpenPower machine, it runs the open source Open Power firmware (all up on github) and will happily run Linux (feel free to port your other operating system kernels). I’ll be writing more on the OpenPower firmware soon as, well, technical details are fun!
Ubuntu 14.10 is listed as recommended as not only have they been building for POWER8 but have spent some time ensuring things work fairly well out-of-the-box (both as a KVM guest and running native on the bare metal). Or, you can always just boot whatever the mainline kernel is at – build for the POWERNV (POWER non-virtualized) platform (be sure to include all the required drivers) and have fun!
OpenPower firmware up on github!
With the whole OpenPower thing, a lot of low level firmware is being open sourced, which is really exciting for the platform – the less proprietary code sitting in memory the better in my books.
If you go to https://github.com/open-power you’ll see code for a bunch of the low level firmware for OpenPower and POWER8.
Hostboot is the bit of code that brings up the CPU and skiboot both sets up hardware and provides runtime services to Linux (such as talking to the service processor, if one is present).
Patches to https://github.com/open-power/skiboot/blob/master/doc/overview.txt are (of course) really quite welcome. It shouldn’t be too hard to get your head around the basics.
To see the Linux side of the OPAL interface, go check out linux/arch/powerpc/platforms/powernv -there you can see how we ask OPAL to do things for us.
If you buy a POWER8 system from IBM running PowerKVM you’re running this code.