Many network monitoring tasks identify subsets of traffic that stand out, eg., top-kk flows for a particular statistic. A Protocol Independent Switch Architecture (PISA) switch can identify these "heavy hitter" flows directly in the data plane, by aggregating traffic statistics across packets and comparing against a threshold. However, network operators often want to identify interesting traffic on a network-wide basis. To bridge the gap between line-rate monitoring and network-wide visibility, we present a distributed heavy-hitter detection scheme for networks modeled as one-big switch. We use adaptive thresholds and approximate data structures to perform threshold monitoring and distinct counting directly in the data plane. We implement our system using the P4 language and Barefoot's Tofino hardware switch, and evaluate it using real-world packet traces. We demonstrate that our solution can accurately detect network-wide statistics with up to 75% savings in communication overhead.

Managing and securing networks requires collecting and analyzing measurement data. Current technologies do not make it easy to do so, typically because they separate data collection (e.g., packet capture or flow monitoring) from the analysis, producing either too much data to answer a general question or too little data to answer a detailed question. This paper presents Sonata, a network telemetry system that exposes a query interface that directs the joint collection and analysis of network traffic. Sonata allows operators to directly express queries in a high-level language, partitions each query into a portion that runs on the switch and another that runs on the streaming analytics platform and refines the query to capture only the traffic that satisfies a query. Sonata allows operators to express real network monitoring tasks using dataflow operators, a compact, familiar programming idiom. Evaluation using traffic traces from a large ISP backbone show that Sonata's ability to compile portions of these queries to the data plane can reduce traffic rates at the stream processor by up to seven orders of magnitude.

@inproceedings{gupta2016network,
title={Network Monitoring as a Streaming Analytics Problem},
author={Gupta, Arpit and Birkner, R{\"u}diger and Canini, Marco and Feamster, Nick and Mac-Stoker, Chris and Willinger, Walter},
booktitle={Proceedings of the 15th ACM Workshop on Hot Topics in Networks},
pages={106--112},
year={2016},
organization={ACM} }

Programmable switches make it easier to perform flexible network monitoring queries at line rate, and scalable stream processors make it possible to fuse data streams to answer more sophisticated queries about the network in real-time. Unfortunately, processing such network monitoring queries at high traffic rates requires both the switches and the stream processors to filter the traffic iteratively and adaptively so as to extract only that traffic that is of interest to the query at hand. Others have network monitoring in the context of streaming; yet, previous work has not closed the loop in a way that allows network operators to perform streaming analytics for network monitoring applications at scale. To achieve this objective, Sonata allows operators to express a network monitoring query by considering each packet as a tuple and efficiently partitioning each query between the switches and the stream processor through iterative refinement. Sonata extracts only the traffic that pertains to each query, ensuring that the stream processor can scale traffic rates of several terabits per second. We show with a simple example query involving DNS reflection attacks and traffic traces from one of the world's largest IXPs that Sonata can capture 95% of all traffic pertaining to the query, while reducing the overall data rate by a factor of about 400 and the number of required counters by four orders of magnitude.