Regular posts on all things StableNet® from a sales, techie, or marketing perspective
StableNet® BlogRegular posts on all things StableNet® from a sales, techie, or marketing perspective
Network Automation Insights
Stay tuned and keep informed with our blog series about the future of automation in network and service management
Automation Series – Part 4
Network Automation for Discovery & Inventory
July 5th 2021, Würzburg
In the last blog post, we had a look at the automation market and “discovered” that Network Automation starts with Discovery to build the inventory baseline as a foundation and that everything else proceeds from there.
So let’s start by having a look at some automated discovery use cases. Then we can explore customization options to structure a large, multi-vendor network environment. We’ll finish with a brief introduction of some of the main advantages of the XML discovery engine.
Network Discovery Use Cases:
The re-occurring discovery component in the StableNet® Workflow Cycle (see figure 1) is the most important function of the entire inventory-building and -maintenance process. Its programmability is built on the eXtended Markup Language (XML), which stores data in plain text format and provides platform-independent data sharing and transport. XML is human readable, easy to understand – even by novices – and not difficult to code.
The XML template provides the steering function for the discovery of an entire network (or can be directed towards specific parts of it). Its logic is structured as building blocks that determine the type of collected information, the included input sources, their properties, as well as the setup of all objects that are created and used within StableNet®.
Figure 1: StableNet® Network Discovery Use Cases
Network discovery automation use cases include inventory creation and maintenance with bidirectional communication to external sources, discovery, and selective re-discoveries. Device property defaults and auto-measurements collect the most common and typically used data points from the infrastructure to provide immediate, out-of-the-box results. This built-in discovery automation already covers a significant number of KPI requirements and provides systemwide network visibility with status updates on visualizations (weather maps).
However, the real power of network automation kicks in once we begin creating specific measurements (data collection) and monitors (KPI validation) to meet customization requirements. That programmability, in combination with meta data enrichments (tagging), encapsulates the core strength of the solution and is the key to generating a fully-tailored network inventory.
The ability to leverage existing data sets (i.e. technical measurements from the network infrastructure which are enriched with business-related aspects) can thus be aggregated into a single data structure automatically. This generates a truly valuable inventory which can be further leveraged as single “source of truth”.
Commonly used inventory data sets are structured by geography (country, regions etc.), organization (corporation, subsidiaries, departments etc.), topology (network hierarchy, aggregated links, critical interfaces etc.) and can be complemented by distinct business aspects like assets in the field, deployed vendors, used technologies, defined services, or even test cases in a lab environment.
Meta Data Enrichment (Tagging):
Tagging is a very powerful tool within StableNet® that has been automated to the largest extent possible. It generates “any and many” complementary data structures within the system. As a consequence, an operator can navigate, sort, combine, filter and query the infrastructure on any of those criteria with the help of the GUI by using boolean logic or regular expressions.
A tag consists of a tag category and a tag value. Tags can be attached to many different taggable objects and used to store any kind of information. To allow for a quick StableNet® configuration ramp up, taggable objects by default come with many preconfigured tags of different flavors (measurement, custom, attribute). Most of them can be freely added, deleted or changed. Tags have a parent-child relationship and are inherited by subordinate objects.
A single object gets a particular tag value assigned. For example, a measurement has a tag with the tag category Measurement Category. The value of this tag is Interfaces, which allows all measurements with this tag value to be grouped together. In a tree, this creates a folder “Interfaces” which contains all measurements with this tag value.
Figure 2: Screenshot how StableNet® visualizes Tag Trees
Tag categories organize tags and group similar functions. Once created, they can be used globally for multiple objects. For the previous example, the Measurement Category separates measurements into different kinds (e.g. Interfaces Processors, Routing, Syslog etc.).
Objects in StableNet® are aggregated in tag domains, e.g. devices are represented by a tag domain Device, while their interfaces are represented by the tag domain Interface. This structure allows for general relationship descriptions like Devices have Interfaces.
The below diagram shows all tag domains of StableNet® along with their relationships to one another. There are predefined read-only tags for each domain, which are automatically set by the system if the required information is available.
Figure 3: Tag Domains (Diagram)
By leveraging and maintaining tags within StableNet®, users are able to structure available and collected data sets according to their individual needs.
The key contributors to network automation are Discovery Jobs and the built-in XML Discovery Engine. Those components take command of the agent conversation. The below schema highlights the essential building blocks, which orchestrate inventory creation and regular update processes.
Figure 4: StableNet® Automated Discovery (Discovery Job and XML Engine)
Discovery Jobs are configured and scheduled in the StableNet® Device Automation Theme. They are a prerequisite to run the Discovery Engine. The XML template allows the definition of placeholders, which are then mapped during runtime executions to the corresponding values.
Various Input Sources:
Comma separated value (csv) files are the most common source of infrastructure data and are required by the agent in order to query information from deployed network devices. An input csv file typically contains IP host addresses, network ranges (subnets), host names, geo data, snmp version and hashed credentials with authentication information.
A more elegant solution is querying an external CMDB (Configuration Management Database) directly during the discovery process by using database variables. Quite often such a CMDB does not exist or is not properly maintained. Under those circumstances, StableNet® acts as an “Infrastructure Asset CMDB” or serves a synchronization function by automatically updating information that was queried from the network in bidirectional communication.
Another StableNet® discovery automation variant is provided by the Resource Management module. This concept uses a modelled resource pool inside of StableNet®, consisting of objects and tables, to organize a larger inventory. Resource Management allows users to define devices within StableNet® before they exist in the network and then leverage common database technologies like filtering, adding tags and assigning group/user rights for access.
Using the Resource Pool concept improves asset structures and organization into different inventory areas. It also simplifies the process of adding devices according to defined criteria and the usage of tags to delete them again. Typical use case examples are the decommissioning of network elements due to device replacements, network re-organizations or corporate mergers.
In summary: leveraging Resource Pools for the discovery process leads to network automation improvements and a better organized inventory structure that facilitates change management for defined assets.
XML Discovery Templates:
StableNet® XML Discovery templates are designed to automate and individualize inventory creation for a diverse range of network and IT infrastructures.
A template can be used with different values for parameters and tags to automatically set up measurements and monitors without manual intervention, resulting in a radical reduction in resource expenditures for recurring tasks for discovered devices. XML templates are auto-generated by StableNet® and the built-in XML editor provides syntax checks with auto-completion and version tracking during the customization process.
StableNet® Discovery Engine refers to the discovery logic which makes the automated “inventory creation magic” happen. The high-level overview in the flowchart (see figure 5) shows the process when a set of devices is discovered by using the previously outlined input sources.
Figure 5: StableNet® Discovery Process (1/2)
At the start of discovery, prefilters are checked. These are used to select the input sources before connections are established and thus reduce the runtime in large environments. The next step depends on the default property “Add Devices which are not reachable”. If enabled, all devices are discovered. Otherwise, a ping scan only discovers reachable devices.
The execution of device discoveries runs in parallel, and the number of threads can be specified in the Job Wizard of the respective Discovery Job. That parallel portion when discovering a single device is shown in the next flowchart (see figure 6).
Figure 6: StableNet® Discovery Process (2/2)
At first, the StableNet® Agent establishes a connection to the target device and determines, based on pre-defined conditions, whether it is relevant. This triggers additional sub-processes which identify the device relevance. After a connection has been established, device post-filters come into play. They use information that is only available on that particular device. These additional inputs make the post-filters very powerful and flexible.
The next step is for StableNet® to check if that device is already part of the inventory. The underlying logic for executing this sub-process is quite complex. We’ll skip it here to avoid making the blog post too long. Devices previously contained in the inventory are updated if necessary, while new devices are added.
The next step is to create the custom measurements from the device block of the XML Discovery template. The different StableNet® measurement types are categorized as Runtime, Application, SNMP, Flow Fault and Status. These categories in turn contain measurements for Common Elements and Attributes, TCP Connect, Ping, SNMP Trap, Syslog, Status, Script, SNMP Template, WMI Template, External and Multi Measurements, SNMP Auto Templates, IP SLA, Netflow, Interface, Bandwidth (Element, Interface), Derived Monitors and Link Generation as an outline of the available toolkit.
Finally, the last step in the discovery process is the addition of auto measurements and interface measurements (unless the auto attribute in the device block is set to false). The following measurements are added:
- A default measurement which monitors device availability (usually ping).
- SNMP template measurements like CPU load, disk space, temperature etc. (provided they are applicable for this device).
- Interface measurements that monitor the operational status, bandwidth utilization, etc. The installed interface monitors are defined by properties, which are set in the default property category to Auto Measure.
- Various additional measurements that depend on specific vendor and device models.
This blog post was intended to provide some background on why we believe that automated discovery and inventory creation is the most important starting point in network automation.
StableNet® does not only provide a vendor-independent framework to master complex operational challenges in multi-vendor and multi-technology infrastructures. It also gets your network automation project as a ”ready-to-use” solution quickly out of the gate, which translates into operational efficiency and short return on investment cycles.
However, the real power comes with the various built-in toolkits to automate workflows that incorporate business-relevant data and generate a customized network inventory, which is based on criteria that matter most to you. StableNet® Automated Network & Service Management equips you with the right network automation foundation to take everything else from there.
After having the network inventory properly in place the workflow circle leads us to automation potentials in configuration generation, management and maintenance. The next blog post explores that subject a bit further.
Made in Germany