Entropy Based Worm and Anomaly Detection in Fast IP Networks

An interesting approach to the problem of detecting worm outbreaks using scalable, wide-view network monitoring tools (like NetFlow from border routers). This approach relies on the fact that for any given sliding time window, the traffic seen in that time period will be very self-similar. Because of this, it will compress reasonably well, being highly redundant in nature and all. However, when a large change occurs, like a worm outbreak, the pattern will change, and you can detect this as a change in entropy. While this pinpoints that something happened, it has a bit to go to actually being useful. It's not sufficient to just say "the network has changed," you need to show the operator what changed.

Detecting massive network events like worm outbreaks in fast IP networks, such as Internet backbones, is hard. One problem is that the amount of traffic data does not allow real-time analysis of details. Another problem is that the specific characteristics of these events are not known in advance. There is a need for analysis methods that are real-time capable and can handle large amounts of traffic data. We have developed an entropy-based approach, that determines and reports entropy contents of traffic parameters such as IP addresses. Changes in the entropy content indicate a massive network event. We give analyses on two Internet worms as proof-of-concept. While our primary focus is detection of fast worms, our approach should also be able to detect other network events. We discuss implementation alternatives and give benchmark results. We also show that our approach scales very well.

Source: Entropy Based Worm and Anomaly Detection in Fast IP Networks, Arno Wagner, Bernhard Plattner, both from the Swiss Federal Institute of Technology Zurich.