Twice last week I sat in a meeting where there was a casual remark that seemed trivial in nature.  The remark went something like this – “I have to look at my spam folder, I registered for the community but didn’t get the confirmation email”.    Two people saying this within days of each other caught my attention.  After a little investigating (helped by the fact we knew the email addresses of the two people) we discovered about 12 RiverMuse Community membership applications had arrived by a non-standard path sitting in a queue requiring manual authorization.  Of course we fixed the problem and contacted each of the community members who had been in limbo.

This is an example of failing silently, a key concept that we set out to address and mitigate when architecting RiverMuse.  We thought we had our registration process clearly defined, simple and effective with no human intervention and with frequent automated checks that the application and SSO manager were working correctly.  Reporting shows that membership is growing on a daily basis.  What could go wrong…..

In the even more complex world of fault management platforms that have rules based workflows, the opportunities to fail silently increase with the scale of the network and systems being monitored.  An automation rule on a central server looks for a particular string or identifier but for this event to get this far there is often another rule engine sitting at the element management or probe level.  A simple error in rule creation, or an alteration of the string at this layer may well lead to an inability to process or recognize an event as significant enough to create an alert or undertake a trigger action.  Now move forwards to the new challenges of virtualization and grid or cloud infrastructures where critical events and services can be nomadic in nature and what was important yesterday is non-critical today or vice versa.

In my own real world example above I had no idea that it was possible to register using an alternative route that we had not planned, not knowing meant that for those 12 people the process had failed, and but for the chance conversation and a similar statement in an email we may never have realized, because all appeared to be well.  This is failing silently.  In my next post on this subject I will talk about the way we have designed RiverMuse to avoid many of the pitfalls that can lead to this type of scenario.

Meantime I would be interested to hear if you have any experiences of failing silently.

One month after the launch of our open source event and fault management platform  we want to hear how you are using our software and what it is that you are managing with it.

We are offering one lucky community member the chance to win an Apple IPHONE 3GS or HTC Hero Android phone (your choice).  All you have to do is tell us how you are using the RiverMuse software and what it is that you are monitoring with it.  We are looking for the most interesting and unusual customisation here, so let your imagination run wild in how you can deploy the system to monitor your coffee machine, coke dispenser,  highway toll booth or aircraft departure delays!  The sky is the limit with our flexible event capture and rules engineering. So show us what you can do!

Check out the rules here.

After generating a lot of debate in our earlier post on what is an event and what is an alert, RiverMuse founder and Chief Science Officer, Phil Tee gives his perspective:

Births; Marriages; Deaths; West Bromwich Albion scoring three goals to secure the “double” against Manchester United; Sterling reaches parity with the Euro; Barack elected first African American President; The Number 42 bus arrives on time today; Our first date; our last date.

All of these share something in common.

All of these are events.

What then is an event? This was a problem that an anonymous civil servant pondered some 104 years ago. He asked the simple question, what exactly is an event?

This anonymous civil servant was in fact Einstein. Einstein’s answer changed the world because it gave us relativity, and eventually, everything from the atom bomb to the GPS.

The fundamental insight of Einstein is that an event is nothing more than a set of measurements that are relevant to the measurer, together with, an agreed way to compare these measurements to those made by anybody else. Indeed, the physics that is being measured, is only relevant when you agree between measurers, how to compare.

For those of us in the world of systems management, we are surrounded by events, in fact, usually too many.

Some 18 years ago, during the design of Netcool, a good part of the motivation was to combat a major drawback in SunNet Manager. In brief, the product was able to hold 1000 traps (a happening or event in a TCP/IP network) in its trap inspection tool, or roughly 2 seconds worth of events in an average sized network.

At this point, we started to distinguish a special class of event, an event that actually meant something. Without too much originality, we termed this event an alert. For example, if you are “pinging” an interface and it goes offline, every time you try to “ping” it, you get an event in the trap interface. We interpreted this event as a recurrence of a “ping fail” alert.

A few lines of code later, we could reduce 1000 typical network events down to a few alerts, and radically extend the usefulness of the trap window from a few seconds of historical coverage, to several minutes. The addition of a scrollable alert list, and a database, gave us a tool that could represent the state of a network over several hours.

However, we neglected to appreciate that we had thrown away some very useful information. For instance, the reduction of events into alerts involved the disposal of the content of an event, in favour of updating attributes (such as count, last occurrence) of an alert.

Another one of Einstein’s insights relating to events was something referred to academically as “locality”, which is, that what happens instantaneously next, can only be affected by what is going on now. Reversing the argument, the future is dictated by the past, and an absence of information about the past, necessarily dims our ability to anticipate the future.

With advancements/progression in database technology, RiverMuse can abandon the alert centric view of Netcool, and now embrace the world of events.

RiverMuse defines an event as a point in the development of the state of a system. Some events warrant the creation of an alert. RiverMuse defines such alerts as a state of the system requiring intervention from an operator.

For every alert there may be many events, each being an agreed state of evolution of the alert (assignment, diagnosis, closure). There may be many events that never create an alert, but nevertheless must be recorded.

We make no assumptions about the retrospective importance of an event; we record everything.

The non-arrival of the number 42 bus, after the unsuspected victory of West Bromwich Albion may have led to that first date, or may not. It is impossible to identify which event in the present will be important in the future; however, we can say that events hold the key to understanding the future, and some end up more important than others.

Now what price the Baggies in the Premier League next season? A question way beyond the genius of Prof Einstein, I think!
Phil Tee