Explainers

Encrypted traffic,

When the test changes the winner

Same six models, two ways of splitting the data

Macro-F1 on VNAT before and after keeping each recording session on one side of the split. Informer 95.1 to 79.9, first to last. NST 91.4 to 87.8, second to first. Autoformer 90.1 to 83.7. TimesNet 89.0 to 86.7. FEDformer 88.2 to 86.2. TimeMixer 86.7 to 83.1. sessions leak into the test sessions kept apart Informer 95.1 79.9 Informer NST 91.4 87.8 NST Autoformer 90.1 83.7 Autoformer TimesNet 89.0 86.7 TimesNet FEDformer 88.2 86.2 FEDformer TimeMixer 86.7 83.1 TimeMixer
Macro-F1 (out of 100) on the VNAT dataset. Left: the usual split, where flows from the same recording session can sit in both the training and the test set. Right: each session kept on one side. Informer goes from first to last and NST from second to first. From our CCNC 2027 paper.

A traffic dataset is recorded in sessions: someone streams for an hour, chats for twenty minutes, downloads a file. If the flows of one session land on both sides of the train/test split, the model is tested on traffic it has, in effect, already seen. It looks brilliant, but partly because it remembers the session.

We trained six modern deep time-series models 2,016 times on two public datasets. With the usual split, one model, Informer, led everywhere. Once we kept each recording session on one side only, Informer fell from first to last on the VNAT dataset, losing 15 points of macro-F1, and the runner-up took first place. The reason: 55.7% of VNAT’s flows come from just three recordings. On a dataset whose recordings are spread thin, the same change left the ranking untouched.

Two further lessons. Giving every model a slightly richer description of each moment of a flow (volume and timing per direction, not just packet counts) helped in 323 of 324 comparisons, more than any choice of architecture. And this echoes our earlier finding that a plain Random Forest can beat deep networks on malicious traffic. Before crediting a new model, check how it was tested.