Companies love big data but lack the strategy to use it effectively, according to the faculty at Harvard Business School. Researchers say that the problem is that everyone is better at collecting data — about their customers, about their products, about competitors — than analyzing it and designing strategy around it. The data is often not used well.
Solving “impossible” problems with stream processing
Sharpe Engineering Inc., a software engineering company specializing in the application of advanced technologies to address complex problem spaces, shares this philosophy. The reason that much of the data produced today is underutilized is in part because of its volume, velocity and variety.
Our consultancy helps clients understand and utilize data in motion by adopting stream processing and machine learning technologies to make sense of data that has been previously underutilized.
Sharpe uses IBM Streams as its platform of choice to build innovative solutions that can analyze data in motion, delivering results that clients previously thought to be impossible.
Choosing IBM Streams
The fundamental problem is in processing the huge volumes of data in most businesses or use cases fast enough to enable instant insight and decision support.
Over the years, I have seen many software products that claimed to be able to meet this challenge, but most of them quickly fell by the wayside.
I was considering building a solution myself until I heard about the IBM technology called System S nearly a decade ago. I contacted IBM and built a close working relationship with the development team for System S. Today, System S has evolved into IBM Streams, and much of my business has been built helping my clients use it to solve a wide range of novel and challenging problems.
I chose IBM Streams because of its computational efficiency, which saves my clients money, as well as its maturity and stability, which provides a highly reliable platform and the richness of the tools that have evolved around the platform.
Applying the technology across industries
We can use stream processing to run the algorithms while data is in motion. That’s a very powerful combination for near-real-time insight that could be useful for many industries.
For example, recently we worked with medical researchers at the University of Montana to analyze brainwave data from patients who have suffered traumatic brain injuries. By predicting which patients are likely to develop post-traumatic epilepsy, physicians can prescribe preventative drugs, possibly reducing the risk of the condition developing.
We also worked with a government research team on real-time analysis of underwater acoustics, using IBM Streams to process the raw data into a form suitable to feed machine learning algorithms that are capable of discerning previously undetectable signals, even with no prior examples. This approach could be used in a variety of government, commercial and research settings.
We used the same algorithms and nearly identical code in both projects.
Read the case study for more details.