Cloud computing has demonstrated dramatic benefits in accelerating and flexing the delivery of IT services thanks to automation from initial request to provisioning, self service, auto-scaling, and some level of API standardization through web services. It has offered as well an attractive financial proposition to specific organizations, projects, or for certain classes of applications. Even though large public clouds offer some unique appeal many companies today have expressed an interest in building their own private cloud to reap the aforementioned benefits without concern for multi-tenancy, lack of direct control, security, compliance and data migration issues that often come with a public cloud provider.

The initial public clouds were pretty much custom built as these early leaders invented them. But since then the market has evolved and flourished. New software companies have emerged to deliver off-the-shelf tools and platforms that fill in the multiple facets of building, operating and managing either public or private cloud environments and applications. Promising companies include Eucalyptus, Rightscale, Northscale, Puppet Labs, New Relic and Cloudswitch to name a few.

Along the way some issues have disappeared but new ones have popped up as well. I posit that a private cloud will not come to displace a huge legacy environment but enable a new breed of applications that support business and process innovation.  Yet IT departments will most likely be required to manage them with a dual challenge of delivering high availability, performance and reliability “as usual” and fulfilling cloud expectations of higher velocity, resource optimization, nimbleness, and better cost control. This in turn puts a new set of requirements on the IT Operations management system that legacy solutions were not designed to cope with. I’ll cover them in a future post here.

Solarwinds is a great network polling solution, and offers some event collection capabilities. However, googling ‘Solarwinds event correlation’ reveals the product’s shortcomings in the correlation and problem isolation space.

For example, many problems can occur in the IT environment of a business:

• Silent failures: many issues can go unnoticed such as link flapping, spanning tree changes, etc.
• High level of event noise, requiring time and resources to diagnose.
• No conciliation of collected events with performance degradation, and
• No correlation against application performance problems reported by 3rd party tools such as Microsoft SCOM, Hyperic, and CA Wily – essentially no cross-tool correlation.

RiverMuse Pro, our commercial IT operations management solution, accepts any event from any source. That includes Solarwinds Network Performance Manager, Syslog and SNMP traps from network devices, and alarms from other systems and application management tools. RiverMuse Pro addresses the correlation gap in Solarwinds.

A customer of mine recently experienced the scenario of a critical network link flapping. The problem went undetected for 60 minutes. Solarwinds was polling the interfaces every 5 minutes, and pinging for availability every 2 minutes. Coincidentally, the interface happened to be up during each poll. Syslog messages from the devices were noisy, but link down and link up traps were reported. However, given the volume of messages they went unnoticed. Without a real-time operations tool in place, it was not evident that the interface was down or flapping. On the 15-minute mark poll, Solarwinds reported heavy interface errors and packet discards. But the fault was cleared on the subsequent 20-minute mark poll. At the 60 minute mark, the link was down again, which coincided with the Solarwinds poll. It was only then that some smart operators manually sifted through Syslog messages, and determined that the link was flapping.

In contrast, RiverMuse offers a throttle based correlation that can detect link flaps. If 2 or more link down events are reported for the same interface in 3 minutes, RiverMuse can escalate the issue, and generate a summary event indicating the link flap. In addition to that, RiverMuse can perform event priority based correlation. Solarwinds events reporting high packet discards and errors would be correlated as symptoms of the ‘Link flapping’ event.

Lastly, RiverMuse can provide business context and automation on the ‘Link Flapping’ event. A dynamic lookup from an inventory management system would identify the carrier and circuit id. A second lookup from a change management system can provide the escalation contact name and email address. All this information can be viewable in the alert within the RiverMuse console. The escalation contact would also be automatically emailed.

With RiverMuse, a problem that went unnoticed could be detected in real-time. And previously manual procedures would also be automated. Both would safeguard business operation and save time and effort of the operations staff.

Download RiverMuse Pro  free for 30 days and check it out. And if there are additional types of correlations that you would like to set up in your environment, let us know. It would be great to discuss that in our RiverMuse Pro Community.

IT Operations Consoles can be a great asset to an organization, as all pertinent events are consolidated and accessible from a single pane of glass. An effective IT Operations Console also requires comprehensive correlation capabilities. This allows organizations to reduce alarm noise to only those that are meaningful and actionable. It also allows organizations to ensure that operator actions are efficient and impactful in business terms.

RiverMuse Pro offers a broad rules-based correlation engine. One of those rules can perform topology-based correlation by looking up relationships from a CMDB, then performing upstream or downstream suppression. Another rule can lookup procedures from a 3^rd party knowledgebase, and execute them through an external action. Through my years in data center operations, perhaps the 3 most popular correlations include:
-       Suppressing duplicate events
-       Sliding time window based correlation (escalate if X events occur in Y minutes), and last but not least
-       Mapping business impact of outages.

De-duplication has been so commonplace with enterprise-class event management systems that it can easily be taken for granted. But the fact remains: de-duplication is the first and primary method of reducing alarm volumes to manageable levels. Some leading tools in the market lose granularity when de-duplicating events. All they know is the first time the problem occurred, the last time it occurred, and how many times it occurred. RiverMuse addresses this issue with a 2-tier event/alert model. Raw incoming ‘events’ are managed separate from processed de-duplicated ‘alerts’, and reside in separate subsystems. This allows RiverMuse to scale and de-duplicate events without sacrificing granularity.

How many times have we seen a link flapping? Or numerous CPU spikes? The problems are auto-remedied within seconds or minutes, but their frequency hides other problems. User transactions might hang or fail. The 3-hour file transfer had 10 seconds left for completion, only to stop unexpectedly. Deploying a time-series or ‘throttle’ based correlation through RiverMuse Pro can detect these recurring flaps, and escalate the problem(s).

In most IT Operations Consoles, there is zero to little notion of business context. Perhaps the most popular correlation method is determining business impact of an outage. Under the covers, this correlation is based on parent-child relationships. These relationships can be based on a service topology present in a CMDB, or other service management tools. Through the use of dynamic variables, RiverMuse can query external tools, such as CMDBs to lookup impacted business services, and other dependent Configuration Items (CIs). Business impact analysis can then be performed.

RiverMuse offers a wealth of correlation capabilities to address an organization’s business and operational needs. Download RiverMuse available free for 30 days and check it out. And if there are specific type of correlations that you would like to set up in your environment let us know. It would be great to discuss that in our RiverMuse Pro Community.

Until ITIL V3 came to fruition, there was a common misconception that any reported event was an incident, and required a ticket be opened. That could be a CPU threshold exceedance, a process restarting, or a change in network topology. Service Desk vendors drooled like Tasmanian Devils, and some went as far as redefining terms such as Business Service Management.

On the opposite end of the spectrum, many organizations still send email notifications or pager alerts whenever any exception occurs in the monitoring infrastructure. IT administrators receive numerous notifications to the point that many emails are ignored or deleted – since the notifications were too common to be taken seriously. Some infrastructure monitoring tools incorporate a certain level of correlation, but they only correlate the data captured by the tool itself; and often, the correlation capabilities are too narrow or limited. Naturally, these tools cannot correlate information reported by other toolsets. So, the IT administrators are now bombarded with emails from multiple tools, not just one.

With an event management platform in place, correlation first occurs prior to taking any secondary action. Correlation can occur whether the events originated from the same source, or multiple sources. Tickets are then opened on real incidents and problems.  Email notification volumes are dramatically reduced. When an administrator receives an email, he/she can rest assured it’s regarding a real problem. It ensures that those 3 AM calls are actually worth getting up for.

Aside from correlation, an event management platform also acts as a central location to consolidate events across the infrastructure. It is ideal for data center operators since they can see the status of any component in real-time in a single pane of glass. That can include power supplies, HVAC units, servers, virtual machines, the network, the cloud, and virtually any other component. Operators can launch “in context” to other tools once incidents have been correlated to problems. This facilitates troubleshooting, and ensures operators drilldown to the right tool, rather than wasting their time swiveling between multiple tools, and manually trying to correlate problems.

Last but not least, a good event management system also makes business sense out of IT events. In a previous life, I recall instructing operators to ignore a set of messages from 1 – 2 AM due to system maintenance. And that was just for my systems! What about every other application, system, and network device out there? Alert enrichment to the rescue! By looking up the maintenance schedule from a change control system, alerts can be suppressed during maintenance periods, or automatically escalated otherwise. What if escalation contacts are automatically populated into alerts, but the contact information resides in an external inventory management system? Alert enrichment can automate the lookup, and automatically send notifications to the contact. How about populating carrier Circuit ID for ‘link down’ events? Or customers impacted by an outage? Perhaps looking up procedures from a knowledgebase, and automatically executing the verification steps? Alert enrichment allows for such intelligence, which facilitates correlation, automation, and visualization.

RiverMuse, the next generation real-time IT Operations platform enables operational efficiency through event consolidation, alert reduction, and business context enrichment. Rather than swiveling chairs from one tool to another, and looking up procedural information in spreadsheets and other systems, all pertinent information is automatically mapped to alerts. In the past such systems were only afforded by large corporations with large budgets. RiverMuse completely breaks through that barrier by delivering sophisticated event management technology at the right price and ease-of-use that can appeal to mid-market companies too.  Join our RiverMuse Pro Community to discuss your thoughts on IT event management.

When designing the RiverMuse architecture, we were fully cognizant of the constraints of other system management solutions that use only a single tier ‘alert’ model with no historical archival of ‘events’.  By adopting an architecture that supports a two-tier ‘event’ and ‘alert’ model at the basic level, freed RiverMuse from many of these past constraints.
 
For clarity, we describe an ’event’ as an infrastructure occurrence communicated as a message (e.g. trap or syslog), polling result (reachable/not reachable) or a rule (threshold below ‘x’ for a service). An ‘alert’ is generated based on how an event (and its related occurrences) is processed based on specified business rules. In essence, events are the raw incoming data stream; alerts are processed and de-duplicated events.
 
Legacy system management platforms for instance – only manage “alerts”; and therefore risk losing data. For example, in the case of de-duplication, the individual occurrences of the duplicate alert are discarded.
 
In contrast, with RiverMuse’s two-tier alert and event model means that you can select from the database all the events that have occurred for any given alert. Each alert has a unique “alert_id” (integer) and each event references an “alert_id”.
 
Through the desktop console you can retrieve all events relevant to a specific “alert_id”, thus providing an audit trail of every occurrence that has happened to a specific alert. For example, you can find out: create time, time of every duplicate, assignment time, acknowledgement time, how long it took to close the alert and so on.
 
Being able to detect every change to an alert creates a fully compliant system for regulations such as Sarbanes Oxley and Basel II, as no information is ever discarded. Consequently, in relation to ‘alert management’ RiverMuse offers the best solution for compliance sensitive environments.
 
The two level event and alert model is a key component of RiverMuse’s lightweight platform for event acquisition, correlation, alert generation and processing in dynamic IT environments.

We were pleased to complete the beta release of the next version of our commercial product RiverMuse Pro v2.0 last week. RiverMuse Pro replaces and builds on top of the RiverMuse ES (Enterprise Subscription) v1.0 product released in Nov 2009.

Now that RiverMuse PRO is in the hands of many beta customers and will be GA in a few weeks, I thought that this would be a good opportunity to explain what makes RiverMuse Pro different.

The first critical advantage of RiverMuse Pro is its ability to automatically fingerprint any new entity in an IT  infrastructure and configure a set of predefined rules against it for de-duplication,  polling, enrichment, and correlation without any manual intervention. From that point on the RiverMuse system automatically maintains its rules configuration in step with changes in the environment.  The second critical advantage is its power to integrate business logic from other application, data center an network management systems  using an open architecture.

This single capability differentiates RiverMuse from other IT Event and Fault Management software when dealing with the demands of highly dynamic operations environments such as virtualized or cloud systems.

RiverMuse Pro intelligently converts raw input of topology and external management data into configuration files called RiverMuse Objects (RMOs). By working with a hierarchical set of objects and analyzing the business logic, the entity modeling component (called modeld) automatically creates the right RMO representation, say, when device or system of type X is encountered.

Consequently, you can arrange for knowledge of ‘what is out there’ in the environment, to be automatically incorporated in the RiverMuse Pro system. If you have a large environment, no doubt you will agree that it is much easier to write one rule as opposed to 100’s or 1000’s of rules to cover every conceivable situation.

RiverMuse Pro thus removes in its entirety the job of manually configuring the changes to your configuration. In theory, you write it once and run it always.

Within the RMOs, the configuration is written in a meta-language so there is no reference to a physical device by name. You express configuration in pure business logic rather than ‘IP address = 192.168.42.36’. Hence, in theory – you can take your configuration, put it into a different system, with a different notion of what is out there, and it will still create a valid configuration.

Another innovation in the architecture is the RiverMuse presence topology acquirer (called topod). This tells the system when something is there and what it is. topod runs a linear augmentation style discovery through the use of seed Explorers, and Embellishing Agents. The Seed Explorer looks for devices it has never seen before, as you can be quite confident if you get an alert from something, it exists. topod then passes any discovered entities along to different Embellishing Agents, which find out and augment the RiverMuse database with more information as they are uncovered. This enables the system to dynamically manage new entities, like say, virtual machine entities as they are created, moved or torn down.

Together these capabilities make RiverMuse Pro uniquely suited for automated adaptation and configuration in dynamic cloud and IT environments.

RiverMuse PRO includes a Reusable Business Logic (RBL) engine to streamline the creation of all configuration components in a single reusable package.

The configuration is loaded through a text file; it is then parsed and converted by a back-end engine that updates the configuration of multiple components within the RiverMuse product. This is a vast change from traditional Manager of Managers (MoM) solutions as configurations from multiple components are incorporated in a single configuration.

The RBL engine serves to:
– Map organizational processes into the product through one configuration channel
– Fuel the creation of a community driven repository (aka App Store). Configuration packages can be shared or bought. (i.e. a configuration package for grouping events by event type, and performing isolated problem correlation; a configuration package for interpreting Cisco alarms and integrating with Cisco inventory tools to populate RiverMuse dynamic variables).

RiverMuse PRO also incorporates a presence management engine that can discover entities on demand.

This is most useful when new alarms are reported for devices/entities not yet present in a CMDB or inventory management system. A RiverMuse Business Logic Package can first lookup Configuration Item information from the CMDB, and if nothing is found, attempt to perform a discovery using the RiverMuse Presence Management System. This will provide many additional variables that can be leveraged through correlations, automations, and escalations.

RiverMuse PRO includes a centralized rules management wizard.

Whether there are 1 or 50 remotely deployed collectors, rules are configured in a central location through a GUI. Within a traditional Manager of Managers (MoM) solution, the process of obtaining events, performing correlation, and providing business context are usually separate and distinct. Legacy MoM architectures typically require business logic rules to be updated at various levels and multiple components within their system and frequently using different, proprietary scripting languages. This creates a management challenge – and makes it hard for operations teams to keep up with infrastructure shifts.

RiverMuse PRO provides the facility to consolidate your Data Center Operations in a single pane of glass, and achieve Operational Excellence by automating tasks and streamlining processes.

RiverMuse Core, the first enterprise-class open source Real-time Consolidated Operations Console system ideally collects information via SNMP traps and Syslog messages out-of-the-box. Additionally it supports 8 standards-based APIs to obtain data from virtually any source (gSOAP, Perl, Java, C++, XML, PHP, Python, and Ruby on Rails). RiverMuse PRO builds on top of RiverMuse Core and provides a presence management discovery engine, a powerful enterprise desktop console, dynamic alert enrichment from external systems, enhanced scalability, and additional functionalities to streamline organizational processes and dramatically simplify system maintenance.

Perhaps the most compelling reason for pursuing a Consolidated Operations Console solution is being buried in a myriad of tools. And more importantly having little or no business context mapped to the results. The need for a so-called Manager-of-Managers MOM solution becomes more evident the more complex and dynamic an infrastructure becomes – thus requiring various tools to manage and monitor the environment. While multiple monitoring tools are great for specific tasks such as application monitoring, transaction monitoring, or communication device monitoring, problems that affect more than one silo take longer to identify and isolate.

RiverMuse PRO solves this problem by centralizing data across all the different tools, and retrieving events directly from devices when needed. To perform this, RiverMuse PRO includes passive as well as active collectors such as the RiverMuse VMWare agent. This active collector natively interprets CIM (Common Information Model) formatted data streams. Events from different environments are consolidated in one repository, where cross-domain correlation can occur. This allows operations personnel to quickly identify the problem and associated symptoms.

RiverMuse PRO incorporates the event processing scalability of leading commercial Manager-of-Manager solutions without sacrificing granularily.

In other words, all events related to a specific alert are kept in our system and made available on demand including through a launch in context tool. Additionally, correlation can occur against events and alerts. Other leading Manager of Managers tools are also resource intensive and require several install instances to provide value. In contrast, RiverMuse PRO was incorporates a small footprint to curb the overhead and maintenance requirements of legacy MOM solutions without sacrificing elegance and functionality.

A common feature of current generation IT Event and Fault management systems is that you have to encode into the configuration of the system, a knowledge or representation of the logic that you use to manage the network, or device, or application. For example, a typical system management scenario is where you have a series of servers, routers and applications that you poll for specific data, which you then place conditions on to test for exceptions. Each exception can generate an alert that triggers actions.

All pre-existing approaches in legacy IT event and fault management systems prior to RiverMuse, encode the logic in a heavily scripted way, or, require detailed understanding of the topology or configuration of the managed system integrated in with the rules you write.

For example, in a typical probe rules file, to add an entry you need to include specific information, i.e. an IP address, for every device you want to ping. In others, you can consume an entire topology and output a code book that pattern matches the alert streams looking for particular root causes.

From a RiverMuse 
perspective, either of these approaches results in it being difficult to alter the configuration when the underlying network changes. In addition, if you want to collect all the intellectual property that has gone into building the business logic, and take it to a completely different network that is configured in a different way, i.e. different hostnames and IP addresses – this can also be complicated.

Hence, the work of configuring a IT event and fault management system becomes a consultancy driven exercise. You have teams of skilled people, with a deep understanding of the management system being used, and the systems or networks being managed, who execute something akin to a software development exercise.

The result is often a single purpose, environment specific configuration or a particular system or network. If you want to replicate the functionality, or solution elsewhere, you have to start over. If the network changes, you have to repeat large parts of the original exercise. All of these issues RiverMuse terms static and non transportable business logic.

(This is a Guest contribution on the RiverMuse Blog).

We loved your post and thanks for reading the Telesperience blog Microsperience!

We believe that even though we’re now officially post-crunch in telecoms, many CSPs have taken a good, long and hard look at how they buy and consume software. Interest in open source software by CSPs of all sizes is definitely on the increase. As you know, one of the “dirty little secrets” of the telecoms industry has been that open source tech can be found in virtually all CSPs’ IT stacks at some level – in fact many ISVs have been using it to lower their own costs for years while still publicly keeping on-message that it’s not really suitable for mission-critical BSS-OSS.

Our research has shown, however, that this situation is changing fast. Partly this is because a new generation of open source solutions like yours (not just technology or tools), are coming to market, and partly because CSPs in many markets are faced with having to do more with less. So to fund innovation they need to look hard at their cost base and reduce waste in the BSSOSS, using these savings to finance new projects.

We’ve also found that interest in open source tech is higher in the telecoms vertical than in other highly transactive industries. We recently asked a range of different types of large companies from telecoms, banking, automotive, computing/IT and government sectors whether they’d consider using open source technology. Eighteen per cent told us they were already using it, 9% are actively investigating using it, 55% said they would not rule out the possibility and 18% said they had no plans to use it. In a separate study we asked just telecoms SPs the same question and this found that, like the previous study, 18% were already using it. However, in contrast 27% were actively investigating using it, 46% would not rule out using it and only 9% said they had no plans to use it. This shows that telcos are more advanced in utilizing or evaluating open source solutions than the average enterprise.

Not only is interest in open source solutions high in the telecoms vertical but we discovered interesting trends amongst those adopting it. The (uninformed) consensus is that CSP adopt open source to lower costs. In fact, although this was certainly why many were attracted to open source initially, it’s not why they stayed. Far more attractive to them than just lower costs was the fact that many open source solutions are built using the most up-to-date technology available, and that they were capability-rich due to the sheer speed of innovation.

My advice to CSPs is therefore to not get hung up on what is essentially an alternative business model, but instead to evaluate open source solutions on exactly the same basis as any other software. One of the big hang ups in the telecoms software market has been that many CSPs are wary of being held hostage by a single vendor, and also that they realize that differentiation is increasingly going to come from software not networks. Open source seems to have a strong message for this fear – take control of your own destiny and don’t just accept a generic one-size fits all offering. There’s some really good software out there in the market today. Some of it is from ISVs and some from open source vendors. There’s no single ‘best’ solution for everyone – you have to evaluate the stand-out offerings against your own individual needs. What I’m saying is that dismissing a solution just because its open source is dumb and, if you take a look at the stats above, it’s certainly not what your rivals are doing.

Teresa Cottam

Research Director, Telesperience