The ashes of network monitoring products from the last 30 years is a Sauron-sized mountain of tears on which we must build the a new generation of tools. Analytics, machine learning, big data and user interfaces are the new hope.
Network as a Service
A key feature in “as a Service” products is transparency & visibility. Not only must a service “work” it MUST be able to demonstrate that it is working, the SLAs are honoured, that performance is within specification and so on
A key element of cloud design is this:
Its more important to know that a service isn’t working (properly) than for it to work well.
– Define “well” ? How will you know that “well” is good enough if you can’t measure it ?
– Why ? Over-engineering. Zero visibility means excess capacity is the only answer. For practical purposes, near enough is far better than too much.
– Spend money/time only when a problem exists. Not because you think it might need it.
– Analytics platform pay by reducing overspend on WAN, LAN and reducing time-to-resolution.
– Analytics proves that your decisions/spending/project is producing a return/delivering benefits.
– Network as a Service demands proof and validation. Thats the basis of any cloud operation.
In three bullet points:
- Data driven decisions
- Telemetry is the process of collecting data for the analysis
- Analytics extracts value from the data.
Monitoring is just one part of analytics. Graphs with threshold alerting is the very least that can be achieved with software.
Three Types of Analytics
Three D’s of Analytics:
- Descriptive: What happened ?
- Deductive : What will happen ?
- Determination: What should we do ?
Batch vs Real Time. Batch is useful for predictive outcomes. Real time is needed for operational issues i.e. security.
Batch processing is easy (and cheap). Real time analysis is hard. For example finding security events in real time from server and firewall logs.
IOT and Network Analytics
The “Internet of Things” is going to be driven by analytics platforms:
- High degree of similarity between IOT and network operations
- Monitoring, ownership and operation of IOT devices is closely aligned with network operations.
- Distributed edge, uncontrolled, unpredictable performance, user variation.
- The entire IOT market is supported by Analytics tooling that enables many millions to be effectively operated at minimal cost.
Most analytic platforms use the same technology e.g. Spark, Hadoop, Kibana, Elasticsearch etc.
The value is in two parts:
- Deep Learning or Machine Learning that extracts patterns, events or data points from the data
- The visual interface that presents data to the user varies according to customer need.
Data presentation or UI/UX is the key to value extraction. IP telephony, WAN monitoring, security events need different data presentation interfaces to be able to advise the customer of the Three D’s.
Failure of “ Network Monitoring”
The failure of network monitoring has left a bunch of deep scars across the industry. An ash mountain of engineer souls represents the last 30 years of network monitoring which destroyed value and belief.
This means that new products that claim to replace network monitoring have a very high bar for success & acceptance. (thats why analytics vendors are not doing that).
This new market is dramatically different – Collect, Parse, Ship, Extract, Present compared to the “Fault, Configuration, Accounting, Performance, Security”
Reflection on the issues related to data collection:
- How much data do I collect, does it need filtering at the source or on ingest ?
- how do I store (a lot of) data, for how long and in what format ?
- Open APIs and data formats vs vendor specific.
- How many ingestion formats can I reasonably support ?
- How do I get data from device/network to the ingestion point ? (On-prem collectors, cloud storage/analytics)
Machine Learning / Heuristics – which algorithms are suitable for extracting information from the telemetry.
Variation – Homogenising data sources to consistent handling and extracting consistent outcomes against algorithms.
Customers – will customers believe that analytics can work ? Will they accept that cloud.
The Etherealmind View
But thats enough for now. Analytics is my big hope and I get all excited when I think about potential for finally getting real-time information about the network with some sort of smarts.
Packet Pushers recorded a podcast on this topic with founders from companies who have network analytics platforms. You can listen to it beliow.