ASRock Industrial NUCS BOX-1360P/D4 Review: Raptor Lake-P Impresses, plus Surprise ECC

Setup Notes: The In-Band ECC Option

Upon completion of the hardware configuration of the NUCS BOX-1360P/D4 and freshly installing the OS, we took some time to look into its BIOS interface. The video below present the entire gamut of available options for the system.

ASRock Industrial primarily plays in the B2B space, where a functional feature-rich BIOS is valued over a fancy GUI-based one with fewer knobs. In that context, it is not hugely surprising that the BIOS interface is spartan in nature. Since the system is primarily meant for business deployment, the control knobs mainly relate to the activation of specific CPU and chipset features, along with the peripherals. The BIOS home screen provides a quick overview of the system configuration - processor details, along with the DRAM configuration (including the memory controller speeds).

Under the Advanced > Chipset option, we found an entry that was not present in the BIOS of the NUC BOX-1260P. By default, this 'In-Band ECC Support' was disabled. The constraint on enabling it was only related to both SODIMM slots having similar memory modules installed.

Since we had configured the SODIMM slots in a symmetric manner, toggling this option and booting into the OS was a straightforward affair. The screenshots below show the impact of this option on the memory information as reported by Windows Task Manager - the default settings on the left, and the ECC enabled version on the right.

It can be noted that the amount of hardware reserved memory in the 'In-Band ECC'-enabled case is 2GB higher than the default case. This points to 1/32 of the total memory capacity being reserved for ECC storage. At this juncture, an overview of in-band ECC is warranted.

In-Band ECC

Error-correction code (ECC) memory is typically used in applications where memory integrity is of paramount importance. Typical ECC memory associates a bit pattern with every block of data that can be used to ensure that the block did not experience any corruption when it was resident in external memory. In general, this feature has been restricted to high-end workstation and server systems. There are many ways to implement this type of memory protection. The most commonly used scheme involves the external memory having additional pins transmitting the protection bit pattern (ECC) along with the main data. Thus, instead of a 64-bit memory interface, the SoC / processor and the memory chips would have a (64 + 8)-bit interface, with the 8-bit interface for the ECC. The cited example of a 72-bit word with 64 data bits (block size) and 8 check bits (ECC) is most commonly used for single-bit error correction / double-bit error detection (SECDED), but other variants are also possible.

Going into the details of how ECC works and helps in SECDED are beyond the scope of this review (interested readers can start going down the rabbit hole here, in case one is not already familiar with them). If performance is of paramount importance, the sideband (extra bits in the bus between the processor and external memory) scheme is helpful because the ECC computation and checks can be carried out in parallel with the main data transfer. However, using sideband signals is not only wasteful of board real estate and component sizes, but also imposes power and cost penalties. In space-constrained systems and other applications where sacrificing a bit of performance is an acceptable trade-off for memory protection, architects have come up with the concept of in-band ECC.

'In-Band' refers to the fact that the ECC is stored within the same memory space as the main data (by reserving an address range, and disallowing its use by the memory clients inside the SoC / processor). In simple terms, whenever data is written out to the external memory, the ECC corresponding to it is also written out to a corresponding reserved address. Whenever data is read from the external memory, the corresponding ECC is also read back, and the memory controller inside the SoC / processor does the data integrity check as required. This scheme is not performance-friendly if operated with the same 64-bit data / 8-bit ECC granularity used in sideband configurations, as the effective memory bandwidth would get cut down by more than a factor of two. Instead, most in-band ECC schemes operate with data block sizes equivalent to the burst size of the external memory. DRAM is accessed in sets of cycles (termed as the burst length - BL). With BL8, and a 64-bit memory bus, each access set would be to 64 bytes (512-bits). I am hugely oversimplifying things here, but readers should be able to catch the drift. Now, SECDED for 512 bits can be achieved with 16 ECC bits. There is an additional complication here because accessing the external memory for reading and writing 2 bytes is highly wasteful (remember the BL8 / 64-byte access set). To improve memory bandwidth utilization for ECC accesses, the memory controller includes an 'ECC cache' where these ECC values are stored (and preferably flushed out only if they can be bunched together in a single write burst). Similar to any caching scheme, this can improve bandwidth utilization but can't always be guaranteed to avoid inefficiencies. Sometimes, it may be necessary to perform read-modify-writes to the external memory, and this can bring down overall memory bandwidth utilization. Intel's 2019 patent filing provides more detailed technical insights into the likely architecture of the in-band ECC block in the memory controller.

In-band ECC started appearing in Intel's processors recently in specific embedded Tiger Lake-U and Elkhart Lake parts. In fact, we did review Supermicro's SYS-E100-12T-H that included a TGL-UE processor with the capabilities, but the BIOS didn't allow control over this in-band ECC setting. Seeing the feature re-appear unannounced in Raptor Lake-P was a pleasant surprise, as it finally provided us with the opportunity to evaluate the performance impact of in-band ECC (something we were unable to do in the SYS-E100-12T-H review). Since Intel hadn't talked about this feature in their CES launch, we reached out to them regarding the official line on in-band ECC support for Raptor Lake-P. The official response, quoted verbatim:

In-band ECC is supported for Chrome designs but not Windows designs. Windows designs use in-band ECC for debug purposes only to identify failures in memory.

In order to figure out the impacts of activating in-band ECC, we processed our evaluation routine on the NUCS BOX-1360P/D4 twice - once with in-band ECC disabled, and once with it activated. The next few sections details the comparative benchmarks for the two configurations (and also includes a host of other UCFF systems to provide additional insights). Prior to that, a brief analysis of the platform is warranted.

Platform Analysis

The block diagram below presents the overall high-speed I/O distribution of the motherboard in the NUCS BOX-1360P/D4.

The architecture is similar to that of the NUC BOX-1260P despite the dropping of the second LAN port and the replacement of one of the Display Port outputs with HDMI. It must be noted that the retimer used in the Thunderbolt port path is still the same Burnside Bridge used in the NUC BOX-1200 series - this means that we don't get USB 3.2 Gen 2x2 support that could have provided 20 Gbps support in addition to the regular 40 Gbps Thunderbolt 4 support.

The rest of this review deals with the comparative benchmark numbers for the UCFF systems outlined in the table above. All of the systems are based on 4x4 motherboards, though the PL1 and PL2 configurations vary.