Today as part of HotChips 2020 we saw Marvell finally reveal some details on the microarchitecture of their new ThunderX3 server CPUs and core microarchitectures. The company had announced the existence of the new server and infrastructure processor back in March, and is now able to share more concrete specifications about how the in-house CPU design team promises to distinguish itself from the quickly growing competition that is the Arm server market.

We had reviewed the ThunderX2 back in 2018 – at the time still a Cavium product before the designs and teams were acquired by Marvell only a few months later that year. Ever since, the Arm server ecosystem has been jump-started by Arm’s Neoverse N1 CPU core and partner designs such as from Amazon (Graviton2) and Ampere (Altra), a quite different set of circumstances and alongside AMD’s successful return in the market, a very different landscape.

Marvell started off the HotChips presentation with a roadmap of its products, detailing that the ThunderX3 generation isn’t merely just a single design, but actually represents a flexible approach using multiple dies, with the first generation 60-core CN110xx SKUs using a single die as a monolithic design in 2020, and next year seeing the release of a 96-core dual-die variant aiming for higher performance.

The use of a dual-die approach like this is very interesting as it represents a mid-point between a completely monolithic design, and a chiplet approach from vendors such as AMD. Each die here is identical in the sense that it can be used independently as standalone products.

From a SoC-perspective, the ThunderX3 die scales up to 60 cores, with the 2-die variant scaling up to 96. The first thing question that comes to mind when seeing these figures is why the 2-die variant doesn’t scale up to the full 120-cores- Marvell didn’t cover this during the talk but there were a few clues in the presentation.

Marvell had made the performance improvement claim of 2-3x over a ThunderX2 at equal power levels. This latter had a TDP of 180W – if the TX3 maintains this thermal envelope then it would mean that a dual-die design would have had to grow TDPs to up to 360W which far beyond what one can air cool in a typical server form-factor and rack in terms of power density. Assuming just a linear cut-down to 96 cores as advertised we’d end up around 288W – which is more in line with the current high-end server CPU deployments without water-cooling. Of course – this is all our own analysis and take of the matter.

A single die supports 8 channels of DDR4-3200 which is standard for this generation of a server product and essentially in line with everybody else in the market. I/O wise, we see a disclosure of 64 lanes of PCIe 4.0 – which is again in line with competitors but half of what higher-end alternatives from Ampere or AMD can achieve.

One big unknown right now is how the dual-die product will segment the I/O and memory controllers – if this is going to be a 50-50 split in terms of resources between the two dies, or whether we’ll see an imbalanced setup – or if the platform can actually handle the full resources from each die and transform itself into a 16-channel 128 lane beast?

Comparison of Major Arm Server CPUs
  Marvell
ThunderX3
110xx
Cavium
ThunderX2
9980-2200
Ampere
Altra
Q80-33
Amazon
Graviton2
Process Technology TSMC
7nm
TSMC
16 nm
TSMC
7 nm
TSMC
7nm
Die Type Monolithic

or

Dual-Die MCM
Monolithic Monolithic Monolithic
Micro-architecture Triton Vulcan Neoverse N1 (Ares)
Cores 60 (1 Die)
Swiched 3x Ring

96 (2 Die)
32
Ring bus
80
Mesh
64
Mesh
Threads 240 (1 Die)
384 (2 Die)
128 80 64
Max. number of sockets 2 2 2 1
Base Frequency ? 2.2 GHz - -
Turbo Frequency 3.1 GHz 2.5 GHz 3.3 GHz 2.5 GHz
L3 Cache 90MB 32 MB 32 MB 32 MB
DRAM 8-Channel
DDR4-3200
8-Channel
DDR4-2667
8-Channel
DDR4-3200
8-Channel
DDR4-3200
PCIe lanes 4.0 x 64
(1 Die)
3.0 x 56 4.0 x 128 4.0 x 64
TDP ~180W (1 Die)
(unconfirmed)
180W 250 W ~110-130W
(unconfirmed)

On paper at least, the ThunderX3 seems quite similar to Amazon’s Graviton2 as they both share a similar amount of CPU cores and similar memory and IO configurations. The bigger differences that one can immediately point out to is that the ThunderX3 employs SMT4 in its CPU cores and thus supports up to 240 threads per die. There’s also a TDP difference, but I attribute this to the Graviton2 being conservative with its clock frequencies, whilst Ampere’s SKUs being more in line with the ThunderX3, particularly the 64-core 3.0GHz 180W Q64-30 being the closest match in specifications.

Another thing that stands out for the ThunderX3 is the 90MB of L3 cache that dwarfs the 32MB of the previous generation as well as the 32MB configurations of Ampere and Amazon.

Marvell here opted to evolve its own interconnect microarchitecture which has now evolved from a simple ring design, to a switched ring with three sub-rings, or columns. Ring stops consist of CPU tiles with 4 cores and two L3-slices with 3MB of cache. This gives a full die with 15 ring stops (3x5 columns) and the full 60 cores 90MB of total L3 cache which is a quite respectable amount.

In the Q&A sessions, Marvell disclosed that their rationale for a switched ring topology versus a single ring, or a mesh design was that a single ring wouldn’t have been able to scale up in performance and bandwidth at higher core counts. A mesh design would have been a big change, and it would have required a reduction in core count. A switched ring represented a good trade-off between the two architectures. Indeed, if this is what enabled Marvell to include up to 3x the cache versus its nearest competitors, it seems to have been a good choice.

One odd thing I noted is that the system is still using a snoop-based coherency algorithm which comes in contrast with other directory-based systems in the industry. This might reduce implementation complexity and area, but might lag behind in terms of power efficiency and coherency traffic for the chip.

The memory controllers tap into the rings, and Marvell’s inter-socket/die CCPI3 interface here serves up to 84GB/s of bandwidth.

Triton CPU Core - 30% Generational IPC Improvements
Comments Locked

27 Comments

View All Comments

  • McCartney - Tuesday, August 18, 2020 - link

    i just want to give quantumz0d credit for having the courage at expressing what is clearly the case. i'm tired of ARM trash being propounded by losers stuck in the stock market and trying to pump whenever they can.

    i foresaw this years ago when i was propositioned with "entering the market" and doing an IPO. i steadfastly refused since i looked at it as a "credit aggregation scheme" in which the quality of any monetary "cashout" would be directly dependent on those buying in. and as far as i can see, that's not the best way to secure my future.

    8 years later, my fears have been realised. the "market" has destroyed the enthusiast industry, where the latter has been enslaved by the former. all we hear about from today's "enthusiast" is how ARM processors are great, with these foolish expectations that x86 binaries can somehow be transitioned to ARM seamlessly. there is a lack of appreciation for both sides of the coin with today's enthusiast (learning the software side and the hardware side) and it is reflected by the lack of diverse offerings from the manufacturers.

    in the words of one of my favourite people in the embedded space, ralph baechle (a huge contributor to MIPS), it was never foreseen that ARM would even go multicore (https://www.tldp.org/HOWTO/SMP-HOWTO-3.html).

    in fact, it's hard enough to make a good multicore embedded processor (the SH4[A] is/was amasing, and stacking more cores introduces bigger challenges when you compare the physical restrictions of the embedded segment versus the desktop.

    now, on to what you're saying Gomez (and originally the reason i wanted to post): i agree. for my line of work, an x86 or a good MIPS (Kfc, not just Kc) is an absolute necessity. i need larger shared memory and my work (10^5 dimension matrices that involve eigendecompositions) is not able to use "high core low cache+memory" designs such as nVidias (which has an API in MATLAB) or ARM.

    i agree with you entirely. it would be very interesting from a GPU design standpoint if nVidia absorbed ARM. i would love to see what their 'shader units' would look like after getting more direction from ARM cores.
  • Spunjji - Wednesday, August 19, 2020 - link

    Courage? For posting an ill-informed, barely-grammatical rant that didn't come close to a rational argument? Okay... 🤪

    Based on the rambling off-topic content of your post, it's hard to tell whether you're a z0d sockpuppet or just equally delusional.
  • mkanada - Wednesday, August 19, 2020 - link

    With Apple going to ARM, many desktop software will be ported to this architecture. So, in the next 5 years, I hope to see ARM workstations with powerfull GPUs, competing head-to-head with x86 based computers.
  • Gomez Addams - Friday, August 21, 2020 - link

    Windows already runs on ARM and Visual Studio can target ARM code generation. All it takes is a re-compilation. There are already ARM-powered GPUs available now. This site reviewed one recently. This is only the start.
  • Rudde - Friday, August 21, 2020 - link

    "8-wife fetch unit"
    Now I'm intrigued.
  • Industry_veteran - Saturday, August 29, 2020 - link

    Just 10 days after this announcement, the Marvell management seems to have realized there is no market for general purpose server grade ARM!. They pulled the rug under the feet of this team.
    It is funny because just stuffing more cores in the SoC doesn't win new customers in server market.
    For hyper scale customers the name of the game is performance per watt numbers.
    This Marvell team should have known this for long time yet they keep making these superficial announcements about how they can stuff so many cores in an SoC. Only less experienced people fall for that. The hyper scale customers know better.
  • pawder - Monday, September 14, 2020 - link

    Wow! That is impressive, thanks for charing with us and making it clear! Now I want to help you back, do you know that most of spouses are cheating? I`m sure that you know it and you know that they are keeping their secrets in their cell phones. Today I give you an opportunity to spy on them without accessing with https://topspying.com/spy-on-a-cell-phone-without-... So follow the instuctions and spy on anybody you want

Log in

Don't have an account? Sign up now