Conclusions and Lessons Learned
What are the key lessons we have learned? In no particular order:
Address the provision of systems management in prioritized, manageable units of work. Don't try to do everything at once.
Manage the rollout of your systems management application just like any other implementation. After all, availability and integrity of your management application should ideally exceed that of the most critical component that it manages.
Get buy in from all the stakeholders in the process, from Management to Shift Operators.
Think about who will use the tool. You probably don't want to send alerts regarding printer failures to your DBA team. Similarly it's probably not a good idea to send arcane messages about network topology changes to application support teams. Prioritize system alerts and trim out noise.
Don't use it a stick to beat developers, network admin or systems admin staff. If your network management tool highlights a problem, use that information as a justification to provide the resources to fix it.
Do your research. Adopting an Open Source solution requires just as much rigor in the selection and evaluation process as a proprietary solution. By all means download your candidate solution and try it out, but don't allow a machine under a Systems Administrator's desk to become a mission critical component.
It's not been a totally smooth ride. We had repeated problems with memory leaks within the Java virtual machine (admittedly, not OpenNMS's fault). We also had a few nasty problems with corruption of OpenNMS's back-end database, which are now fixed. There were also a lot of "d'oh!" moments along the way, as we got up to speed with what is a pretty complex application. None of these problems ever seemed like show stoppers at the time. This had much to do with help extended by the development team and user community, to whom we extend our thanks.