DELiRium: The New Deep Learning Compiler for FPGAs
Bringing Artificial Intelligence to the Edge
Deploying Deep Learning models on edge devices represents one of the most ambitious hardware and software challenges for AI in 2020. Sma-RTy, a French start-up based in Clermont-Ferrand, founded by three PhDs and a Full Professor, has decided to tackle this challenge by focusing on embedded systems for image processing and computer vision.
To optimize models in integrated operational scenarios, efficient software architectures (such as pruning and quantization) are needed alongside hardware capable of fully supporting them. FPGAs offer immense advantages for real-time Deep Learning: they guarantee reconfigurability, structural flexibility, and a fine-grain architecture ideal for quantized networks. However, the main problem for their large-scale adoption remains the high difficulty of programmability.
FPGAs offer the hardware flexibility essential for Edge AI, but actual programmability remains one of the greatest open challenges for developers.
Sma-RTy's Solution: DELiRium and Direct Hardware Mapping
To overcome the technical hurdle of programmability, Sma-RTy has developed DELiRium, a Deep Learning compiler for reconfigurable devices. DELiRium takes a pre-trained and optimized neural network in a standard ONNX format as input and automatically outputs target-independent VHDL code ready for synthesis.
The compiler relies on parameterized IPs (like ConvLayer.vhd, PoolLayer.vhd, and mathematical modules) and pushes physical implementation to the limits through Direct Hardware Mapping (DHM). Unlike classic streaming paradigms, DHM completely unrolls the Convolutional Neural Network (CNN) processing: every actor and every level of the computational graph is mapped to its own unique hardware instance.
| Advantages of DHM | Disadvantages of DHM |
|---|---|
| High computational throughput, outputting results per clock cycle. | Scalability issues: some extensive models might not fit entirely in the FPGA. |
| No memory bandwidth limitations; intermediate data stays local in FIFOs. | Lack of post-deploy flexibility: different networks correspond to different bitstreams. |
| Fully exploits pruned (fewer nodes) and quantized (low resource utilization) networks. | Requires preliminary software optimizations to minimize hardware footprint. |
Overcoming Physical Limits: Single Constant Multiplication
The biggest limitation of a direct 1:1 mapping lies in the available physical resources. For example, unrolling just the second layer of a LeNet5 (with C=6, N=16, K=5) would require 2400 multipliers, exceeding the 680 DSP blocks available in the largest Intel Cyclone V.
To make DHM feasible, DELiRium introduces an important architectural optimization called Single Constant Multiplication. Since network weights are constant during inference, multipliers are specialized:
- If the weight (𝚹) is 0, the multiplier and connection are simply removed.
- If the weight is 1, the hardware is replaced by a simple wire.
- If the weight is a power of 2, the compiler replaces the costly DSP with a shift register.
Results and Perspectives
Tests confirm that, thanks to these joint hardware/software optimizations, it is possible to effectively implement CNN inference on entry-level FPGAs. Devices like the Intel Cyclone V or the Xilinx Kintex 7 can maintain power consumption under 10 W while operating at over 50 FPS on 720p video streams.
And what about historically heavy architectures trained on ImageNet, such as AlexNet? Even when FPGA resources cannot host the entire depth of the network, DELiRium can physically map the first three convolutional layers, providing an unbeatable system for real-time pre-processing on the peripheral IoT node.
