Network operation engineers must deal with log messages generated by multiple devices and technologies. Log message extraction is crucial because these messages carry valuable information, but their unstructured design makes it challenging to automate the process.
The standard procedure used so far is based on regular expressions (Regex or Grok or similar) that are used to find and extract the fields of interest. This has always been an arduous process for several reasons.
- Building those expressions can be complex.
- They need to be supported and evolved over time.
- Engineers who created them may not leave proper documentation, resulting in reverse-engineering efforts.
- Vendors may change the format for their syslog messages, breaking existing regular expressions and blocking log message extraction.
Moving Beyond Manual Log Message Extraction
The era of “human-based rules” to extract information has passed. We have evolved into a digital transformation of how data and algorithms are processed.
With data and algorithms, we can accomplish many of the tasks that previously required manually configured rules. This shift is transforming log message extraction for network operations teams.
NLP-Powered Log Message Extraction with Selector
Selector’s platform uses state-of-the-art Natural Language Processing (NLP) techniques to manage log messages and simplify log message extraction. Now, we are also using NLP techniques to identify and extract key information from within a log.
Named Entity Recognition (NEW) for Smarter Extraction
Named Entity Recognition (NER) is a well-known technique in the context of Natural Language Processing that allows the automatic extraction of key and relevant information from text.
In this case, the target text is a log message, and the relevant information can include:
- Hostnames
- IP addresses
- MAC addresses
- Fully qualified domain names
- Interface names
Selector’s Named Entity Recognition will train a model using the customer’s key sources of data, such as inventories, customer databases, and other relevant data. With that, Selector’s Log processing pipeline automatically performs log message extraction and enriches log messages with the new fields automatically identified by the NER algorithm.
From Enriched Logs to Actionable Insights
Once logs are enriched with the new labels, key analytics and insights can be generated so that the Selector Platform can surface multidimensional anomalies that are otherwise very difficult to identify.
Chasing fields with regular expressions is not an effective use of time. Advanced techniques like NLP and NER automate this process and allow network operation engineers to focus on anomalies and proactive management, not manual log parsing.
Key Benefits for Network Operations Teams
Selector’s Analytics platform offers several key benefits:
- No more complex regex maintenance – Operations teams are no longer required to create and maintain complex and numerous regular expressions to parse and extract data from logs.
- Automatic anomaly detection – By extracting key fields and performing multidimensional anomaly detection, Selector surfaces unseen anomalies from logs.
- Faster problem identification – Network operation engineers can quickly identify issues that might otherwise remain hidden.
See It in Action
Below are several screenshots of this unique feature:
Interested in learning more about this feature? Contact us today for a free demo!