Review: HP's Moonshot is flexible, manageable, amazing

Innovating in the server space has traditionally meant cranking out new systems with the latest Intel CPUs along with more memory and storage. While that delivery model has served the industry well for many years, the time is right for new approaches that focus on balancing processing power and energy consumption for specific use case scenarios. It's also a great time to bring innovations in processor architecture into the enterprise space in order to reduce operating costs.

HP set out to break the traditional mold with the Moonshot platform. In the initial Moonshot release, which stuffed as many as 45 Intel Atom server cartridges into the 4.3U chassis, HP addressed dynamic Web workloads with a strong focus on reducing the long-term costs associated with powering and cooling data center hardware. With the recent releases of AppliedMicro ARM, Intel Xeon, and Texas Instruments DSP+ARM boards, HP has unleashed the beast on additional workloads, including static Web, virtual desktop infrastructure, and Hadoop.

HP also recently introduced the 45XGc switch module, which provides 10GbE connections to cartridges within the Moonshot chassis. The 45XGc joins 45G and 180G models in the Moonshot switch lineup, which provide 45 1GbE and 180 1GbE internal connections, respectively. The Moonshot chassis can accommodate up to two switch modules.

This latest release of new Moonshot cartridges brings with it the ability to mix and match boards in the same chassis. Note there are a couple of restrictions with respect to networking, however. First, you can't mix boards with different networking speeds and expect to get 10G from the higher-speed boards (namely the ARM and Xeon cartridges). When mixed with 1G cartridges, the 10G-capable cartridges will operate at 1G. Second, the 45G and 45XGc switches do not support multinode cartridges (includes Atom, Xeon, and DSP+ARM cartridges). Using multinode cartridges requires the 180G switch.

Money spent on power and cooling constitutes much of today's data center operating expenses. HP's Moonshot team has come up with a new measure of cost versus performance that zeroes in on power consumed per application unit. For a VDI implementation the measure would be watts per VDI user. For Web servers the measure is watts per user session. The new 64-bit, eight-core ARM processor cartridge, the m400, consumes a measly 43W of power at its peak, less than half the power consumed by an eight-core Xeon CPU, and trumps the Xeon in computing power per watt.

Designer hardware

Building a general-purpose server capable of running any number of different workloads is one thing. Designing a server platform with an eye on specific applications and a focus on maximizing the effective usage of CPU, I/O, and memory requires a different mind-set. HP has a long history of hardware design excellence, and the Moonshot platform shows it off across the board.

HP used a number of methods to determine the optimum amounts of memory and CPU required to handle a number of different workloads. For example, the new m400 cartridge pairs a substantial 64GB of memory with the 64-bit, eight-core ARM processor to provide a platform for Web caching in a power-sipping form factor. HP used a wide range of benchmarking tools and multinode deployment scenarios to settle on this combination. The m400 also comes with 10G Ethernet on board to move large amounts of data between nodes and off chassis.

While HP designed each of the cartridges for a specific set of workloads, a few customers have used cartridges in ways that weren't anticipated. For instance, the m800 cartridge -- based on a Texas Instruments system-on-a-chip with four ARM cores and eight DSP cores -- was designed with telecommunications customers and audio/video transcoding in mind. However, PayPal uses this board to perform real-time analytics on text-based event streams.

The innovations in Moonshot start with the chassis. While you might wonder why the height of the case (4.3U) is nonstandard, there is a reason. One rack unit or 1U requires 1.75 inches or 44.45 millimeters of space. The longest dimension of a 3.5-inch hard disk is 5.75 inches or 146 millimeters long. In order to stand a 3.5-inch disk drive vertically in a rack and accommodate the rail and connector mounting, you would need an object slightly larger than 4U (7 inches) of space. HP already had other product offerings that used 4.3U, so it made sense to stick with that dimension.

Moonshot fabrics, networks, and interconnects

Connecting the 45 processor slots along with two additional long slots for the network switches required a significant amount of innovation. HP's engineers designed the backplane that connects all slots in the Moonshot chassis with 28 dedicated copper lines. Each of these lines, or lanes, can carry different types of signaling at very high data rates. If you know anything about high-speed communications, you understand the challenges posed by electromagnetic interference in a hardware platform as densely packed as Moonshot.

Four separate interconnect fabrics provide physical pathways for data and management communications within Moonshot. The three data pathways go by the names of Radial Communication, Proximal Array, and 2D Torus Mesh. The Radial Communication pathway provides high-speed interfaces between each cartridge and two radial fabric slots. These carry mainly networking traffic and provide a pathway to the outside world. The Proximal Array fabric was designed primarily for storage traffic, though it also interacts with the 2D Torus Mesh. The 2D Torus Mesh is a high-bandwidth cartridge-to-cartridge communication pathway that provides direct connections between each node and its four nearest neighbors.

Both the ARM-based m400 and the Xeon-based m710 boards incorporate the Mellanox MT27518 chip that provides two 10GbE ports. These boards require the new 45G switch module. With two 45G switch modules in the chassis, you have a total potential bandwidth within the Moonshot chassis of 900 gigabits. While you would never see that amount of data, it allows serious multinode operations to support applications such as Hadoop.

At the power management level, HP uses a part called an eFuse that provides the isolation needed for hot-swapping cartridges and at the same time measures the amount of power consumed by each individual cartridge. HP tested these devices extensively to ensure the accuracy of the power measurements and found them to consistently hit a very small margin well within the tolerance of the power system. As you would expect, all server cartridges and switch modules are hot-swappable, as are the five fans at the back of the unit. The only item missing at this point might be a storage cartridge of some type.

Moonshot management options

HP's approach to managing enterprise systems has traditionally combined a command-line interface with a GUI. The company has also traditionally supported standards-based methods, including IPMI and SNMP. However, newer initiatives such as HP OneView have taken a more Web-oriented approach, such as the use of open standards like JSON and REST.

As a part of this new initiative HP made a strategic decision to move to a REST interface for its entire set of management tools. This includes iLO (Integrated Lights-Out) and the Chassis Management and Cluster Management tools. A browser-based interface built on top of the REST APIs presents all pertinent information in an easy-to-read and easy-to-navigate format (see Figure 1). Operations such as power on/off can be performed on single or multiple cartridges with a few mouse clicks. The Web UI also offers a full graphical display using unique and innovative ways to present status information (see Figure 2).

The real power behind the REST interface comes from automation through scripting. HP makes it easy to use your favorite scripting language, be it PowerShell or Python, to fully control every aspect of the Moonshot system.

Canonical has used this same interface to bring automated provisioning and orchestration to Moonshot based on its Juju and MaaS (metal as a service) tools. Juju's graphical interface makes it possible to build out a multitiered Web service based on existing templates downloadable from the Charm store. You'll find Charms for all kinds of services, including Juju itself. I used MaaS and Juju to deploy a small Hadoop cluster on multiple Moonshot m400 cartridges in a matter of minutes (see Figure 3). Note that Ubuntu is currently the only operating system HP supports on the m400.

Data center in a can

HP has recently restructured the Moonshot pricing and delivery model, allowing for mixing and matching cartridges and the purchase of individual units. The initial release of the product shipped with a full complement of 45 Intel Atom cartridges and was not available any other way. Under the new model, you can purchase the Moonshot 1500 chassis and populate it as you see fit. Prices for the chassis range from $15,155 with one 45G switch and three power supplies to $55,589 with two 180G switches and four power supplies. Starting prices for the server cartridges (and starting pricing with M.2 interfaces for SATA SSDs) are as follows:

Thus, a maximum-configured Moonshot chassis stuffed with 45 64-bit ARM cartridges will cost you upward of $156,000. HP acknowledges that initial acquisition costs for a Moonshot platform versus a more traditional multiplatform or blade approach could be higher. The Moonshot advantage comes in the form of dramatically lower long-term operating costs.

HP took a bold step with the Moonshot platform and has built on the promise with new cartridges that expand the initial target workloads. The good news is that the company has plenty of room to innovate with new combinations and configurations coupled with enhanced features to take advantage of the framework already in place.

If there is an obvious missing link, it would be some type of in-box storage. HP designed support for storage into one of the three fabrics present in the backplane. The question then becomes what type of storage would you want in a box like this, and can HP build it using the same modular approach? Up to this point the answer would be no, although HP hasn't ruled out such a cartridge.

HP Moonshot definitely doesn't fit every scenario.You wouldn't throw a heavy-duty database workload at Moonshot, but for a variety of clustering and distributed computing scenarios, it combines amazing density and efficiency with excellent management tools. For the scenarios it was designed for, Moonshot knocks the ball right out of the park.