StreamCentral and Big Data
The world is now producing around 2,500,000,000,000,000,000 bytes of data each day! An increasing proportion of this vast data volume comes from machines, sensors, applications and content creation tools that provide “Real-Time” information about events as they occur.
The Internet of Things already is today and will continue to be a massive generator of data. More and more of this data is becoming accessible to businesses. Tapping into this real-time data and connecting it with other organizational data provides context to business outcomes in a way that has never been possible before. This is the power of Big Data.
Two amazing things have happened that forever change the way value of data can be exploited:
- Web based standards like XML have been adopted widely making it easy for systems and applications to consume and exchange data. This means a wide variety of systems that once were closed have the potential of opening up and enabling data to be leveraged. SCADA in the Utilities sector is a great example of this.
- Data Markets like Microsoft’s Windows Azure Data Market offer a wide variety of data available for subscription. Today, businesses can subscribe to data like Country GDP numbers, Census information, real-time traffic, weather forecasts, labor statistics, road accidents, economic indicators, crime statistics, Dunn & Bradstreet company info and more.
There is just so much going on in the world today that impacts business outcomes, but until now it was just not possible to tap into a wide variety of data to see if patterns emerge that help better understand inefficiencies, opportunities or threats. What were gut feelings before can be proved by using data today. StreamCentral makes it easy to connect to web standard based data sources. With a few clicks, a data source can be created along with all necessary parameters.
The challenge at hand is how to make sense of all of this data? When is this data useful or more importantly when is the data relevant to your business? What does data from one source when combined with data from another source tell you about your business or your market? Big Data does not just refer to the ability to collect all the vast data available but also refers to the intelligent use of it for maximum business impact.
StreamCentral Workbench provides easy to use graphical tools that allow people to define how to interpret this data. Rules can be defined to combine and correlate the data so it can be looked at together. Rules can also be defined to look at the status of various data to see if patterns emerge that uncover events. The best part is that as people define these rules, StreamCentral does two things automatically :
- Builds a real-time Operational Intelligence solution that alerts you about events as they happen
- Builds a data warehouse that collects this processed data to give you decision making ability based on historical data
That’s enabling a wide variety of people within your business to now be able to participate in using data for making better and more educated decisions.
The challenge of consuming so much information and processing it in real-time is how will the solution scale? How will it meet the demands of the people wanting to analyse this data?
Another tenet of Big Data is the ability to manage and store all this information without requiring a significant investment in human capital. StreamCentral is built for web scale workloads. All collection and processing of data done by StreamCentral is distributed across multiple machines. In-memory operations enable tremendously fast processing, correlation and event detection on in-flight data.