CNN-Based DPI for Stronger 5G Security

Private 5G networks redefine secure connectivity for defense and critical infrastructure. To protect this infrastructure, Deep Packet Inspection (DPI) moves cybersecurity directly to the network edge, stopping malicious traffic before it impacts core operations.

Traditional cybersecurity approaches often rely on centralized cloud-based inspection. The problem? They can’t stop cyber attacks that hit the base station directly, a critical point in private 5G networks, especially for mission-critical applications.

To understand how Artificial Intelligence can help address this challenge, we spoke with Emilio Paolini, AI researcher at Scuola Superiore Sant’Anna and former PhD candidate with SMA-RTY, whose academic work contributed to the development of SMA-RTY's Deep Packet Inspection (DPI) engine for NGCI private 5G network.

Understanding Private 5G Cybersecurity Risks

First of all, what are the main cybersecurity risks for private 5G networks today?

One of the biggest threats comes from DoS and DDoS attacks. These try to overwhelm servers or network functions by flooding them with traffic, making services unavailable.

A DoS attack comes from a single source. A DDoS attack? Think of it like a traffic jam caused by thousands of cars coming from all directions at once, it’s much harder to manage.

In private 5G networks, compromised devices connected to the network itself can generate this traffic. Without fast detection, these attacks can knock out critical services. The challenge is spotting malicious traffic immediately, before it spreads.

Diagram showing DoS and DDoS attack vectors targeting a private 5G base station

Why cloud-based inspection fails at the 5G edge?

Traditional network security relies on centralized cloud inspection. While effective for standard IT networks where traffic converges centrally, this model introduces latency and security gaps in localized private 5G deployments.

Private 5G networks, however, often operate like “bubbles.” If inspection happens too late, malicious traffic may already affect the whole network. Cloud protections aren't an option in these cases.

From FC-NNs to CNNs: Choosing the Right AI Tools

So, the main problem of the traditional approaches is that they can't protect the bubble, that’s what led to your research?

Exactly.

The core idea was to make the base station itself intelligent, not just a transmission point, but also a security enforcement point.

Could we apply AI-based security right at the gNodeB, detecting threats the instant they appear? That was the starting point and this led us to explore Deep Learning Languages.

You used Neural Networks (NNs) and Convolutional Neural Networks(CNNs) in your studies. Why those?

We initially worked with NNs, specifically fully connected architectures, because they are a well-established solution for classification problems and had already been applied in cybersecurity.

During our analysis, however, we observed that network traffic can be well represented as a matrix, where each row corresponds to a packet and each column represents a feature, such as packet size or timing.

This structure looks like a sort of image. And since CNNs are designed to spot patterns in images, we wondered: could they detect patterns in network traffic more effectively?

So the first study tested NN, and the second focused on CNN?

Yes. The first study was a preliminary test to see how well NNs could detect DoS attacks. The results showed that a CNN could perform even better, so later we pursued that route.

How did you validate them?

In both studies, we trained the algorithms on datasets containing benign and malicious traffic*, then tested their performance by using the Keysight BreakingPoint software tools. This allowed us to evaluate the approaches in a realistic telecom environment.

  • In the first study, we trained on the DoS Hulk attack and tested it on Hulk and another attack, DoS Golden Eye. Accuracy was over 90%.
  • In the second study, we trained a customized CNN specifically for mobile traffic using the 5G-NIDD dataset, generated from a real 5G network.

The customized-CNN outperformed other state-of-the-art architectures, forming the basis of SMA‑RTY’s DPI engine.

Performance comparison between NN and CNN models in network threat detection

* dataset CIC-IDS 2017 for the first study, which is based on traditional internet traffic (not mobile), and the 5G-NIDD dataset, which is based on mobile traffic.

How the Studies Shaped the NGCI DPI Engine

What were the key outcomes?

Both studies confirmed that CNN-based DPI is highly effective at spotting application-layer DoS and DDoS attacks, and can do so at the core network premises. Another key point: the same architecture can be retrained with new datasets, making it adaptable to different networks.

How did these researches contribute to the NGCI DPI engine?

As I said just early, these researches (especially the second one) provided the foundation for the NGCI DPI engine.

The customized-CNN used consisted of 3 convolutional layers (8, 64, 128 filters) followed by 3 fully connected layers (512, 128, 9 units) with ReLU and SoftMax activations.

Let’s put it simple.

Think of it like a multi-level security checkpoint:

  • The convolutional layers act as scanners, each one more precise than the last: spotting patterns, details, and subtle anomalies.
  • The fully connected layers are the security officers, reviewing the information and deciding whether traffic is safe or malicious.
Diagram of the 3-layer CNN architecture used in the NGCI DPI engine

DPI is based exactly on this customized-CNN we tested in the study, even though updates have been made since then, of course.

This layered approach allows the NGCI DPI engine to detect threats instantly at the base station, stopping attacks before they escalate.

Why Edge-Based AI Security Is the Future of 5G?

As private 5G networks expand into mission-critical sectors, cybersecurity must evolve accordingly. Centralized protection alone is no longer enough.

Security needs to become distributed, intelligent, and integrated into the network infrastructure itself.

CNN-based DPI does exactly that: it turns base stations from passive infrastructure into active defenders, ensuring next-generation networks remain safe and reliable.

Moreover, private 5G cells can host third-party services directly at the edge. This means different AI-based applications, not only security, can run locally, enabling an AI-RAN approach where intelligence is embedded throughout the network, not just centralized.

This shift will play a key role in ensuring the security, flexibility, and reliability of next-generation connectivity.

References

back