A recent IDG survey referenced in a Marketpulse whitepaper underscored the waning loyalty of a majority of users with their legacy log management solution. This was primarily stemming from a need for simplified management, real-time threat response, and reduced costs,  combined with an overwhelming requirement for integrating log management and security information and event management.

There is a parallel that can certainly be drawn here with the state of the NSM world, fractured but still dominated by a few legacy platforms designed in the 80s. This is particularly true of the IT operations management tools / Managers of Managers in the event correlation and analysis space where the top 5 vendors IBM, HP, EMC, BMC and CA provide a suite of complex products that require significant effort and several bolt-on components to deliver on their promises. This is perhaps fine in the large enterprise and service provider domain but not in the mid market where time to value is critical and simplicity paramount.

With the advent of agile and cost-effective network management suites like SolarWinds Orion and system management solutions from Nagios, Microsoft SCOM and NetIQ the mid market is favorably responding and fast adopting new products to solve their infrastructure management problems. In the process new challenges are being introduced to now cope with  a higher level of events and alarms generated by these tools, often poorly de-duplicated, with loads of false positives and clearly not consolidated and correlated across different domains.

 For instance our customers’ infrastructure already generate hundreds of thousands to millions of events and log messages per day that need to be processed, consolidated, and enriched to produce a few hundreds to a few thousand alarms that can be acted upon. Because we operate agnostically with other NSM tools and have the ability to add a consolidation layer that brings together entire infrastructure and application issues into one screen – we are able to simplify management and lower costs dramatically. This also puts us in a position to deliver a true service assurance solution for the mid market that helps manage IT at the service level and not just the device, in real-time through a simplified management interface and at reduced cost. 

It is our belief that what the IDG report captures in relation to the log management market is already happening in the IT operations management market too. Would you agree?

Cloud computing has demonstrated dramatic benefits in accelerating and flexing the delivery of IT services thanks to automation from initial request to provisioning, self service, auto-scaling, and some level of API standardization through web services. It has offered as well an attractive financial proposition to specific organizations, projects, or for certain classes of applications. Even though large public clouds offer some unique appeal many companies today have expressed an interest in building their own private cloud to reap the aforementioned benefits without concern for multi-tenancy, lack of direct control, security, compliance and data migration issues that often come with a public cloud provider.

The initial public clouds were pretty much custom built as these early leaders invented them. But since then the market has evolved and flourished. New software companies have emerged to deliver off-the-shelf tools and platforms that fill in the multiple facets of building, operating and managing either public or private cloud environments and applications. Promising companies include Eucalyptus, Rightscale, Northscale, Puppet Labs, New Relic and Cloudswitch to name a few.

Along the way some issues have disappeared but new ones have popped up as well. I posit that a private cloud will not come to displace a huge legacy environment but enable a new breed of applications that support business and process innovation.  Yet IT departments will most likely be required to manage them with a dual challenge of delivering high availability, performance and reliability “as usual” and fulfilling cloud expectations of higher velocity, resource optimization, nimbleness, and better cost control. This in turn puts a new set of requirements on the IT Operations management system that legacy solutions were not designed to cope with. I’ll cover them in a future post here.

Until ITIL V3 came to fruition, there was a common misconception that any reported event was an incident, and required a ticket be opened. That could be a CPU threshold exceedance, a process restarting, or a change in network topology. Service Desk vendors drooled like Tasmanian Devils, and some went as far as redefining terms such as Business Service Management.

On the opposite end of the spectrum, many organizations still send email notifications or pager alerts whenever any exception occurs in the monitoring infrastructure. IT administrators receive numerous notifications to the point that many emails are ignored or deleted – since the notifications were too common to be taken seriously. Some infrastructure monitoring tools incorporate a certain level of correlation, but they only correlate the data captured by the tool itself; and often, the correlation capabilities are too narrow or limited. Naturally, these tools cannot correlate information reported by other toolsets. So, the IT administrators are now bombarded with emails from multiple tools, not just one.

With an event management platform in place, correlation first occurs prior to taking any secondary action. Correlation can occur whether the events originated from the same source, or multiple sources. Tickets are then opened on real incidents and problems.  Email notification volumes are dramatically reduced. When an administrator receives an email, he/she can rest assured it’s regarding a real problem. It ensures that those 3 AM calls are actually worth getting up for.

Aside from correlation, an event management platform also acts as a central location to consolidate events across the infrastructure. It is ideal for data center operators since they can see the status of any component in real-time in a single pane of glass. That can include power supplies, HVAC units, servers, virtual machines, the network, the cloud, and virtually any other component. Operators can launch “in context” to other tools once incidents have been correlated to problems. This facilitates troubleshooting, and ensures operators drilldown to the right tool, rather than wasting their time swiveling between multiple tools, and manually trying to correlate problems.

Last but not least, a good event management system also makes business sense out of IT events. In a previous life, I recall instructing operators to ignore a set of messages from 1 – 2 AM due to system maintenance. And that was just for my systems! What about every other application, system, and network device out there? Alert enrichment to the rescue! By looking up the maintenance schedule from a change control system, alerts can be suppressed during maintenance periods, or automatically escalated otherwise. What if escalation contacts are automatically populated into alerts, but the contact information resides in an external inventory management system? Alert enrichment can automate the lookup, and automatically send notifications to the contact. How about populating carrier Circuit ID for ‘link down’ events? Or customers impacted by an outage? Perhaps looking up procedures from a knowledgebase, and automatically executing the verification steps? Alert enrichment allows for such intelligence, which facilitates correlation, automation, and visualization.

RiverMuse, the next generation real-time IT Operations platform enables operational efficiency through event consolidation, alert reduction, and business context enrichment. Rather than swiveling chairs from one tool to another, and looking up procedural information in spreadsheets and other systems, all pertinent information is automatically mapped to alerts. In the past such systems were only afforded by large corporations with large budgets. RiverMuse completely breaks through that barrier by delivering sophisticated event management technology at the right price and ease-of-use that can appeal to mid-market companies too.  Join our RiverMuse Pro Community to discuss your thoughts on IT event management.

It’s an exhilarating time at RiverMuse this week as we reach a notable milestone in our business with the launch of our new commercial product, RiverMuse Pro v2.0. It has taken us slightly longer than anticipated to get there but frankly it was worth the wait and the additional effort and intensity from our team, our partners and our beta customers. We are excited and proud to bring to the mid-market what we believe is genuine technical innovation in IT Operations management and I want to thank everyone that contributed to this major milestone.

Service providers and enterprises now have a viable solution to consolidate myriads of daily IT events from all infrastructure components and monitoring tools into one centralized console for rapid problem detection, isolation and resolution – thereby dramatically lowering cost of operations and improving customer experience. RiverMuse PRO builds on top of our open source platform with a number of unique innovative capabilities toward faster time to value, lower cost of ownership, and higher scalability. In addition to our free open source RiverMuse Core edition, we have decided to make RiverMuse Pro available for a 30-day free trial as well, so that users have the freedom to choose which edition is best for them. So feel free to check it out for yourself.

RiverMuse Pro includes unique capabilities around event capture and correlation, external data enrichment and centralized configuration that stays in step with infrastructure changes. I am also very excited with the prospect of further collaborating with our partners, VARs and system integrators who will benefit from a global marketplace setup by RiverMuse where they will be able to provide complementary products and services to our growing user community. I will cover this in detail in a separate post in the upcoming weeks.

In the meantime I invite you to download RiverMuse Pro, give it a test run in your environment and let me know what you think.

Ever since the early 1990’s many large xSP’s have implemented Manager of Managers (MoM’s) to consolidate their complex and disparate operations consoles. These separate consoles were windows into existing stovepipe monitoring systems that were split out across different technologies, systems, networks, applications, firewalls etc. All these stovepipe systems had a different look and feel, requiring custom training and worse still they only provided visibility of their part of the infrastructure. Individual operational consoles made it very difficult to understand the impact of failures in adjacent/dependent technology areas, or separate geographic locations.

MoM’s were the key to allow the operational teams the ability to create a horizontal (end to end) view across the infrastructure, so that the xSP’s could proactively monitor the infrastructure that the customer’s service depended upon. The descriptive term that many people used to describe this consolidation of consoles was “Single Pane of Glass”. Without this type of technology resource (people) costs to perform effective proactive monitoring quickly spiralled out of control.

In the last few years mid sized XSP’s are now hitting the same operational issues as their bigger brethren. In simple terms these issues are – a mixture of higher volume of events, increased complexity /diversity of equipment and the increasing dynamic nature of the infrastructure they need to monitor. As with the larger companies before them, the mid-sized xSP’s now see the need to gain control and improve the service they provide to their customers, by performing far more effective proactive monitoring. At the same time, they need to reduce the resource (people) costs involved in providing that monitoring, so that they can remain competitive.

Consolidated IT operations consoles achieve their return on investment (ROI) by dramatically improving the efficiency and accuracy with which they allow an ever-smaller number of people to proactively monitor a given size of infrastructure. The techniques that IT operations management platforms use are many and various, and achieve their results with different levels of success. If you look in our Video section you will see good explanations and examples of the three most successful methods/techniques, namely de-duplication (basic concepts video 4), correlation and automation (basic concepts video 5). For a specific demonstration of what RiverMuse functionality in this area, you can visit our Demo section (registration required).

Although from my experience there is little doubt that MoM’s improve proactive monitoring efficiency, in order to achieve significant and rapid ROI they must also be simple, as well as inexpensive to implement and maintain. In the early days many of the older MoM’s were far too complicated and expensive, which in turn often resulted in an ROI break-even points measured in years. This usually placed the MoM’s outside the reach of most xSP’s with medium sized infrastructures. However the recent generation of real-time IT Operations Management  platforms such as RiverMuse are designed with the primary goal of simplicity of implementation and operation, providing significant cost reduction. This results in a break-even ROI that can be achieved in weeks or months at worst. For the first time, RiverMuse is making the MoM technology an attractive and practical solution for the mid-sized xSP’s.

As always we would be interested in your thoughts and experiences.

I believe a lot of the problems stem from the fact that frameworks usually consist of a small number of original vendor solutions that have been added to over time with products obtained through acquisition, rather than via organic and planned growth in functionality. The issue I have with this approach is that all the individual products within the framework have a different look and feel, different workflows, different architectures, databases, messaging methods etc. This negates the single solution argument; it’s basically the same as best of breed, however without the concurrent benefits.

After visiting many customers who have implemented frameworks, the messages that come through time after time are:

• They are difficult and slow to deploy, often taking 2 to 3 years to implement, to a point where they are returning “real” value.
• Integrations within the framework are superficial at best. Often just designed to meet a tick in the marketing box, and often poorly thought through.
• The vendor will tell you all your problems will be fixed in the next version, in six months to a year’s time (and then in the next version and so on). This is partly because the development of the framework is analogous to steering a super tanker.

To be fair to the framework vendors, acquiring different products and stitching them together is always a nightmare. Partly because many of the original innovators move on (after acquisition), or products are developed on different continents etc, the list of hurdles to overcome is huge.

Just to be clear, I am only pointing out that for medium sized infrastructure customers the “no-brainer” choice does not exist.  I believe you have to examine the issue for yourself, and to a large extent ignore the slideware the larger vendors love to show you. Also be wary of “User Conferences”- I have seen a number of “this solution is fantastic” presentations from a senior sponsor in a company, only to visit operations staff on the ground a few months later to find out that they either never got it working, or are unhappy with the results.

I would be very interested in your experiences, good or bad, and I hope this topic generates some lively discussion.

Ever since I first became involved with systems and network management tools (back in 1990) the idea of using a single vendor “framework”, versus buying “best of breed” tools from multiple sources has been in and out of favour with customers and “experts” alike. As an influential customer (as part of BT) I have had the opportunity to get pitched by both sides and consider both perspectives. From a purely theoretical standpoint it’s very difficult to accurately compare the relative merits of these approaches.

For example a “framework” supplier will tell you things like:

• They are easier and quicker to deploy.
• The vendor deals with all the integration headaches, saving you integration costs and time.
• If you have any problems you only have to negotiate with one supplier.
• One vendor can’t pass the buck to another, by saying it’s not our product that’s causing the problem it’s all the fault of someone else.

Whereas a “best of breed” tools supplier will counter with claims such as:

• You get the best solution for the job, not a “Jack of all trades master of none”.
• You can save money by encouraging competition for the components you need. Basically pitching one hungry supplier against another.
• You are at less risk of your chosen vendor disappearing from the market (all your eggs in one basket).
• Best of breed products are more “innovative” and “agile”, developed by people who are both keen and understand their space. Usually because they are smaller and hungrier.

Why frameworks seem attractive to mid size customers

For many medium sized customers concentrating on the theoretical advantages alone usually makes the “framework” approach appear to be a no-brainer. Especially attractive are the claims that they are quicker/easier to deploy, and that the single vendor deals with all the integration issues.

This desire to implement a framework solution is often further amplified by the fact that many medium sized customers have grown from small infrastructures – either organically or via mergers. In the small infrastructure tools world, nearly all solutions are “integrated products” from one vendor, fulfilling 80% to 90% of customer requirements. For decision makers in mid size companies, previous experience tells them that a single vendor integrated product works well. Consequently, as they move beyond the capabilities of the small tools (usually because they don’t scale, or do not have enough horizontal coverage for a medium size infrastructure) they think frameworks are the answer.

Why frameworks really don’t work for mid size customers

As always though the devil is in the details, and often times the theory and real life experience are at odds. My advice would be to visit existing medium/large customers (similar in size as your organization) who have implemented frameworks and check first hand. When you do, talk to the people on the ground “using the tools” rather than only with the senior managers.

In my experience, most customers with medium sized infrastructures that I have visited over the years have experienced a number of significant issues implementing frameworks (regardless of who the supplier is). For example one customer I visited told me they had developed 75% of the solution in-house (over two years) to make the framework solution they had picked work in their environment.

- Con Blackett

As I mentioned in my last post, what’s needed is a new approach to IT Operations – one that is simpler, real-time in nature, event agnostic, and can easily leverage the existing tools and monitoring technology feeds. With millions of IT events raised on a daily basis in an environment that is increasingly chattier, it becomes paramount to deliver to Command/Service Center operators only the meaningful alerts and notifications through real-time consolidation, correlation and processing.

By capturing all incoming events at high throughput, and filtering them through sophisticated alarm processing mechanisms that retain both the granularity and scalability – real-time event consolidation systems can fulfill what has eluded existing mid-market tools to date. In addition, these systems must dynamically enrich alarms with the right contextual information so that they are easily prioritized and acted upon by operators.

Service providers differentiate from their competitors based on the quality of service and visibility they offer their end customers. To build this differentiation and yet ensure that they get immediate value from management tool investments – these systems must offer the right balance of out-of-the-box capabilities, and flexibility through configuration and integration abilities.

New real-time IT operations systems must also be modular and open throughout so that innovation can blossom outside the confines of the vendor’s development resources, benefiting from the contributions and inputs of a larger community of practitioners.

And finally and perhaps most importantly, such system must be outright simple and affordable to achieve mainstream adoption by mid-sized service providers.

These thoughts do not come from us but from the scores of service providers and practitioners we talk to everyday. We are convinced that mid size service providers are ready for a new approach based on real time event consolidation, simplification and intelligent alarm processing.

Whether you agree or disagree we invite you to comment on our blog or give us a call. We would be delighted to have this conversation.

For any IT service provider the service is the business, and assuring it is the first line of customer and revenue protection. Until now mid-sized service providers have had no choice but to resort to a multiplicity of tools and consoles to manage and monitor their services from the application down to the underlying infrastructure.  

Some illustrative areas covered by these tools include network monitoring, storage monitoring, log monitoring, system monitoring, virtualization monitoring, application monitoring, database monitoring, VoIP monitoring and others. Despite claims to the contrary by so called suite vendors, no single monitoring system provider can cover all aspects of management, across all technologies and across all layers, at any given time. There is simply too much out there to address and this market is too dynamic for any one vendor to address it all.

While higher order event consolidation frameworks exist, they are highly expensive, take months and years to configure and need armies of staff to maintain. Most mid-size service providers have bypassed using these framework platforms altogether. 

Consequently, mid-size service providers have ended up with a combination of monitoring tools whether they like it or not. These tools produce their own specific IT events and alarms in their own environments, leading to multiple consoles that command centers operators have to inevitably negotiate through inefficient ‘swivel chair’ management.

Furthermore, because of scalability constraints -most of the existing mid market tools apply only a rudimentary logic of IT event elimination and consolidation, and offer even more limited coverage of real-time events.

In some cases, they may simply discard events rather than process them so that these are neither seen, nor controlled by operators – leading to missed alarms that are eventually business impacting. Operators learn about such events when customers call in with a problem.

In other cases, they may easily overwhelm operations teams with non-correlated alarm floods that often lack business context – lengthening the time to prioritization, escalation and ultimately remediation.

In summary, most current monitoring systems in use in mid-size service providers sacrifice accuracy, granularity, and enrichment to operate within a limited scale. While this worked for low velocity of change environments, this is an unacceptable proposition as we enter a new era of computing with increasing demand for intelligence, elasticity and compliance.

In my next post I will discuss a new approach to Real Time IT operations that meets the needs of the mid size Service providers in this time of change.

I am pleased to report that RiverMuse was a Judges Choice winner in the category of “Cloud Infrastructure Management” at the select “Under The Radar” event held in Mountain View, CA last week. RiverMuse was one of the 30 vibrant startups pre-selected to present its business case around cloud innovation.

The event attracted over 350 movers and shakers from the IT industry with a keen interest in all things ‘cloud’ this year. The presenting companies had 6 minutes to make their case in front of a panel of industry experts before answering questions from the judges and the audience. In addition to infrastructure management the cloud categories included a range of topical themes including noSQL data management systems, compliance, analytics, virtualization, communication, and app development and management.

The day’s events were fast paced, filled with energy, ideas and edge. The audience and the judges represented a good mix of enterprise and service provider IT executives, vendors, and other key stakeholders, looking for the new generation of cloud computing innovators. It was interesting to note the serious level of interest to adopt cloud techniques and technologies by large enterprises and established service providers.

Open minded yet stringent in their enterprise requirements, it became clear early in the day that the audience meant business and was looking for players with differentiated value proposition, a solid offering and a viable business model. It was also apparent that judges were interested in companies with the potential to target mainstream organizations, not just emerging and atypical web 2.0 businesses that still only represent a sliver of the IT spend but tend to eclipse the thousands of tried-and-true enterprises who have similar needs.

All in all, it was an awesome day to connect with genuine buyers and practitioners of cloud computing. And despite differences in style and pitch, the participants exuded similar traits in presenting their business: true cloud believers, relentless in their quest to solve real problems, true innovation, strong passion, and great agility.

Being recognized as the Judge’s Choice in our category also re-affirmed our belief that managing Cloud systems needs a complete rethinking of the management approach and a new architecture that is equally dynamic, automated and flexible as the monitored environment. RiverMuse is dedicated to doing just that – delivering service assurance for both cloud and virtualized datacenters and networks across all manner of service providers – inside and outside the enterprise.