The Next Wave of Deep Learning Architectures Posted on : Sep 10 - 2016

Intel has planted some solid stakes in the ground for the future of deep learning over the last month with its acquisition of deep learning chip startup, Nervana Systems, and most recently, mobile and embedded machine learning company, Movidius.

These new pieces will snap into Intel’s still-forming puzzle for capturing the supposed billion-plus dollar market ahead for deep learning, which is complemented by its own Knights Mill effort and software optimization work on machine learning codes and tooling. At the same time, just down the coast, Nvidia is firming up the market for its own GPU training and inference chips as well as its own hardware outfitted with the latest Pascal GPUs and requisite deep learning libraries.

While Intel’s efforts have garnered significant headlines recently with that surprising pair of acquisitions, a move which is pushing Nvidia harder to demonstrate GPU acceleration (thus far the dominant compute engine for model training) for deep learning, they still have some work to do to capture mindshare for this emerging market. Further complicating this is the fact that the last two years have brought a number of newcomers to the field—deep learning chip upstarts touting the idea that general purpose architectures (including GPUs) cannot compare to a low precision, fixed point, specialized approach. In fact, we could be moving into a “Cambrian explosion” for computer architecture — one that is brought about by the new requirements of deep learning. Assuming, of course, there are really enough applications and users in a short enough window that the chip startups don’t fall over waiting for their big bang.

Among the upstarts that fit the specialization bill for deep learning is Wave Computing, which in many ways could have served as a suitable acquisition target for an Intel (or other another party) over Nervana Systems. Although the execution and technology are different from Nervana, the fundamental belief that it is practical to do large-scale deep learning training on ultra-low precision hardware with low-level stochastic rounding and other techniques is the same. And while the company’s Jin Kim tells The Next Platform they see high value in their own technology for companies like Intel, the Nervana acquisition is actually a positive element for the field overall because it proves that there is a need for non-general purpose hardware for such a market.

One could make the argument that Intel was just as interested in Nervana for its Neon software framework as it was for the chip, but Wave Computing’s Kim says that there is another unmet need that has companies scrambling. “There are development boards and accelerator boards, but as we talk to people in the field they want a single system that is designed for the specific needs of deep learning.” Of course, something like this already exists in Nvidia’s DGX-1 appliance, which is outfitted with Pascal-generation GPUs and has all software ready to roll for both training and inference. However, Kim says that they have mastered both the hardware and software and can (in theory—they don’t yet have a DGX-1 appliance on hand) beat out Pascal with lower thermals and far faster training times. More on that in a moment; for now, however, the key point is this is one of the first systems to take on deep learning aside from DGX-1, but of course, it is based on a novel architecture.

The Wave Computing approach is based on a dataflow architecture via their DPU processing elements. Like Nervana, Wave has a highly scalable shared memory architecture (with hybrid memory cube or HMC) at the core. “We are both taking the perspective that there are some familiar characteristics of deep learning compute workloads—this means we can take advantage of the fact that these algorithms are resilient to noise and errors. That, coupled with data parallelism and the opportunity to reuse data are just a few things that give this an advantage over general purpose hardware.”

Instead of selling accelerator boards, Wave is focused on delivering a full system for both training and inference. This is designed as what Kim calls a “plug and play node in a datacenter network with native support of TensorFlow as well as Hadoop, Yarn, Spark, and Kafka.” The systems will come in 1U and 3U configurations starting in Q2 of 2017. They have a 28nm test chip that was delivered two years ago they have been validating with, but the forthcoming 16nm FinFET chips, which are being taped out now with delivery late in the year will offer a sizable boost. The first 28nm implementation could put 16,000 of the simple processing elements on a piece of silicon, but with the 16nm FinFET chips coming soon, they can scale up to 64,000, Kim says. These will first be available via a private cloud to foster early users who want to experiment before the systems become available. “Our business model is not to deliver services via our own cloud, but demand is high and private cloud offers a faster option for those who want early access.”

Get a load of all of that memory–and the all-important memory bandwidth capabilities here. This is actually just as (if not more) important than the dataflow processing elements themselves since, after all, even the cleverest of processors is useless without a way to move data. Of course, this is just moving instructions for the most part–another key feature that fits the bill for deep learning workloads where the reuse of data can lead to significant boosts in efficiency.

Each of the 8-bit RISC-oriented processors has an instruction RAM and local registers to hold on to a lot of the memory data being operated on locally so data is not being moved, rather, instructions are. There is an instruction queue with the basics (multiply, add, shifting, etc.). Each of the processing elements group together into a cluster with 16 of the elements sharing the arithmetic units (each has 2). These clusters make up the core compute engines and can be teamed together to do 8-bit work alone or with up to 64 bits together (variable length arithmetic).

The general idea is simple. Each of the DPUs are grouped together and can do multiple tasks. It is a NIMD architecture (versus control flow) and can support very high memory bandwidth. The first generation chips ran between 6.7 GHz and 8 GHz and were in the 150 watt to 200 watt range but Kim did not comment on the next generation ones due this winter. Each of the dataflow processing units (DPUs) becomes part of a cluster. It is scheduled statically via a toolkit the team developed for tactical scheduling without running into memory consistency issues because it is clear which portion of the algorithms are running on specific parts of the DPU.

There are 24 “compute machines” per DPU and as you can see above, the architecture can support serious random access memory with the four HMC and DDR4 memory comprising the core fabric element that tiles these together. In the case of Wave’s 3U configuration, they can put 16 such machines together. View More