Real-Time Data-to-Decision

The ability to analyze massive volumes of network traffic (several hundred Gbps) in real-time (with microsecond to sub-second latencies) is important for communication service providers as it enables them to optimize use of their service infrastructure and develop revenue-generating opportunities. In particular, the real-time analysis of perishable user traffic that is not stored due to regulatory and other constraints can provide insights that are useful in many applications. In this talk, we describe the design and implementation of a platform for real-time analysis of network traffic based on IBM InfoSphere Streams, a scalable stream-processing middleware, which provides comprehensive visibility on the data objects and communication patterns of users at the application layer in contrast to simple packet- and flow-based analysis that most current systems provide. We discuss our design considerations for such system and further describe analytics applications developed to showcase its capabilities: online identification of most-frequent objects, online social network discovery, and real-time sentiment analysis. We also present performance results from a pilot deployment of this platform and its applications that analyzed Internet traffic generated by users of a large corporate research lab.