FAI Summit 2022

Abstract: Deep learning technology has made significant progress on various cognitive tasks, once believed impossible for computers to do well as humans, including image classification, object detection, speech recognition, and natural language processing. However, the vast adaptation of deep learning also highlights its shortcomings, such as limited generalizability and lack of interpretability. In addition, application-specific deep learning models require lots of manually annotated training samples with sophisticated learning schemes. Witnessing the performance saturation of early models such as MLP, CNN, and RNN, one notable recent innovation in deep learning architecture is the transformer model introduced in 2017. It has two good properties towards artificial general intelligence over conventional models. First, the performance of transformer models continues to grow with their model sizes and training data. Second, transformers can be pre-trained with tons of unlabeled data either through unsupervised or self-supervised learning and can be fine-tuned quickly for each application.  In this talk, I will present a multi-FPGA acceleration appliance named DFX for accelerating hyperscale transformer-based AI models. Optimized for OpenAI’s GPT (Generative Pre-trained Transformer) models, it manages to execute an end-to-end inference with low latency and high throughput. DFX uses model parallelism and optimized dataflow that is model-and-hardware-aware for fast simultaneous workload execution among multiple devices. Its compute cores operate on custom instructions and support entire GPT operations including multi-head attentions, layer normalization, token embedding, and LM head. We implement the proposed hardware architecture on four Xilinx Alveo U280 FPGAs and utilize all of the channels of the high bandwidth memory (HBM) and the maximum number of compute resources for high hardware efficiency. Finally, DFX achieves 5.58× speedup and 3.99× energy efficiency over four NVIDIA V100 GPUs on the modern GPT-2 model. DFX is also 8.21× more cost-effective than the GPU appliance, suggesting that it can be a promising alternative in cloud datacenters.

Abstract: This talk will go over different AutoML methods that share the common themes of efficiency and hardware-awareness. We will present (1) a fast predictor-based search algorithm, (2) zero-cost NAS proxies that significantly speed up the evaluation phase of NAS, (3) accurate hardware latency prediction in NAS, (4) automated hardware-DNN codesign, and (5) real application case studies by which NAS delivered significant improvements. Our projects are all parts of an effort to enable on-device deployment of DNN within constrained hardware devices--an area in which AutoML can play a big role.

Abstract: Hardware architectures are undergoing major shifts due to slowing of Moore's law. Innovations in packaging technology has given rise to chiplet based architectures which we are seeing in CPUs and GPUs. In addition, we will in near future, see integration of heterogeneous chiplet modules on the same compute package thereby allowing a mix of accelerated workloads on the same architecture without offloading to PCIe attached accelerators. In addition, thanks to coherent fabrics like CXL, we will see more tighter integration between coherent accelerators and processors. This might also allow dis-aggregation of memory and storage at rack level. CXL will be an enabling technology for CPU-memory and CPU-accelerator allowing for disaggregation into fine grained resource pools. It will allow cloud vendors to provide precisely sized compute block for user workloads. Silicon photonics and in package optics will be able to ensure that latency induced by this dis-aggregated architecture is within tolerable limits and does not come at price of performance of latency sensitive application for most cases. All these hardware stack innovations will have massive impact on future accelerators and their development. It will be important to look at these underlying trends in hardware design and what they offer so that software can leverage performance gains accordingly. Architects will need to leverage the hardware and get performance with low overheads. The aim of the talk is to look at CXL enabled switch to pool memory, accelerator and storage with compute elements to provide customers economical and performance-oriented cloud infrastructure on demand. In addition, we aim to build a co designed serverless based API to provision these units and provide a cost-efficient experience to users.

Abstract: For programming FPGA-based accelerators, high level synthesis (HLS) is the mainstream approach. Unfortunately, HLS leaves a significant programmability gap in terms of reconfigurability, customization and versatility: 1. FPGA physical design can take hours, 2. FPGA reconfiguration time limits HLS from targeting complex workloads, and 3. HLS tools do not reason about cross-workload flexibility. Overlay approaches mitigate the above by mapping programmable designs (e.g. CPU, GPU, etc.) on top of FPGAs. However, the abstraction gap between overlay and FPGA leads to low efficiency/utilization.Our work develops a new FPGA programming paradigm, where an overlay architecture is automatically specialized to a set of representative applications. The key innovation is a highly-customizable overlay design space based on spatial architectures, which encompass a range of designs from application-specific to general purpose. We leverage and extend prior work on accelerator compilers, SoC generation, and fast design space exploration (DSE) to create an end-to-end FPGA acceleration system called OverGen. OverGen can compete in performance with state-of-the-art HLS techniques, while requiring 10,000x less compile time and reconfiguration time.

Abstract: In this project, we set out to find innovative ways to improve inference performance on CPU for better resource utilization. By utilizing sparse multiplication libraries and vendor provided optimized libraries, we can notably improve performance in ML inference

Abstract: Abstract: Entrepreneurs, researchers and AI practitioners that are climate-conscious (and receive monthly electricity bills) and in the search for a modern IT infrastructure to build their solutions need look no further. IBM Power server hardware paired with a RedHat OpenShift container software stack that stays “on” (with 99.999% availability) and a hardware-based AI accelerator (on-chip Matrix Math Accelerator) maintains a lower energy-footprint and Total Cost of Ownership (TCO) compared to x86 servers. Learn about the IBM Power servers, how you can leverage them to drive your AI roadmaps and benefit from the expanding ecosystem of vendors that support IBM Power at this session.

Abstract: In the space of hardware acceleration alternatives, FPGAs lie in the middle of the programmability-efficiency spectrum, with GPUs being more programmable and ASICs being more efficient. FPGAs provide massive parallelism and are reconfigurable, which makes them very well suited for the fast-changing needs of DL applications. But how can we minimize the gap between ASICs and FPGAs in terms of performance and efficiency, while retaining their strength - the reconfigurability? This talk will dive into our research that attempts to answer this question by exploring better reconfigurable fabrics for Deep Learning. We will discuss how FPGAs are evolving into domain-specific reconfigurable fabrics. Specifically, we will look at new blocks called Tensor Slices and CoMeFa RAMs. These blocks are a significant step towards closing the performance gap between FPGAs and ASICs. In this talk, we will take a peek into the architecture of these blocks and talk about the performance improvement and energy reduction that can be obtained for DL application by using modern FPGAs containing these blocks.

Abstract: HPC developers/users in scientific domains such as climate modeling, CAD for Manufacturing, CFD, molecular dynamics, histopathology, seismology, protein folding, high energy physics, astrophysics etc. are exploring ways to use AI models to augment and accelerate or develop AI solutions for HPC simulations. Their problems typically have extremely large, multi-dimensional input data that are unlike those used in popular DL domains of image recognition, recommendation engine, language and text translations etc. This leads to significantly large memory usage and I/O ingestion challenges in both the input data pipeline and end-to-end HPC-AI workload pipelines. The talk will feature Intel’s best practices for optimizing HPC/AI workloads including, code optimizations using Intel Optimized TensorFlow and Pytorch using Intel extensions for training and Inference. These optimizations results in up to 3-4X improvement over Intel’s 3rd Gen Scalable Processor using AVX-512 using Intel’s 4th Gen Processor code named Sapphire Rapids with HBM using AMX/TMUL instructions supporting mixed-precision FP32 and BFloat16 for Training and quantized Inference.