Flow Exporter Provenance as a Major Confounder in Cross-Dataset IoT Intrusion Detection

Source abstract

Machine-learning intrusion detection for the Internet of Things (IoT) routinely exceeds 99% accuracy on single datasets but fails when transferred to new networks or flow exporters. We formalize five failure modes, define a 23-feature canonical schema, and adapt three datasets (CICIoT2023, TON_IoT, Bot-IoT) to build a full six-pair transfer matrix across three classifiers, each with three random seeds. Random forest attains the highest mean AUC (0.740), yet its balanced accuracy collapses to chance (0.50–0.51) exclusively on CICFlowMeter-sourced pairs while remaining strong (0.72–0.87) on Zeek- and Argus-sourced pairs. This exporter-dependent collapse is reproduced across structurally unrelated classifiers, establishing it as our central finding. A controlled synthetic test confirms that CICFlowMeter’s directional heuristic destroys variance (up to 104 reduction) and causes a small, consistent transfer cost (+0.004 AUC), though full degradation requires compounded effects. Aggregate distributional distance does not predict transfer success (r = −0.17), ruling out a simple divergence explanation. Prior correction provides the largest ablation gain. Source-domain coverage is necessary but not sufficient for transfer, and we observed no universal sample-count threshold. Flow exporter provenance emerges as a major upstream confounder and a directly implicated contributing mechanism, though exporter identity is confounded with dataset identity and is not established as the sole determinant of cross-dataset performance.