Designing and Building your CMDB, Part 2: Test, Test, and Test Again

With apologies for the large break between these two parts, a lot of project work has eaten up all my time and it's not been possible to post as often as I'd have liked!

So the questions that were left at the end of Part 1 is; how do we know we've correctly identified everything related to our system for the CMDB? The answer is, because we didn't write the software and so can't claim to fully understand it, that we don't. Now I'll explain why that doesn't matter as much as you'd think it would.

As with most things in business it's not about a 100% guarantee  - it's about doing exactly the right amount of work to minimising the risk of impact to the business from the "unknown unknowns" (those things we don't know we don't know). In order to work out the ways we've eliminated the risk we need to go back to the reason we're doing this; to take the knowledge out of people's heads and put it into a system so it can be shared. I don't need to create a CI for every item in a config file - the CMDB needs to know where the file is and what it's configuring. The rest is up to the person looking at it and the problem/ incident they're dealing with.

In order to test out that our CI's will meet our needs we need to look at how we are expecting them to be used after we've created them. Here are a few examples of the scenarios we might like to test our configuration with;

• A user X who has just joined the company needs to be added to the users list (1)
• User X has changed role and no longer needs access (2)
• User X needs access to the Admin Interface (3)
• User X can't access the system (4)
• User can access parts of the system but isn't seeing any maps (5)
• Microsoft has released a patch for a critical vulnerability in IIS  and Engineer Y needs to find all the boxes with IIS installed so he can patch them manually (6)
• Emails from "System X" don't seem to be being sent (7)

As you could see this list could go on forever but, as I pointed out earlier, we're not trying to capture every possible thing that could happen - we're just trying to cover the 90% of things that are most likely to occur and a few other things we (as developers) might like to worry about.

Now that we've got out list let's go through and see whether we have enough information so that someone who isn't familiar with the system but has access to the CI structure can solve the issues we've highlighted;

1. Looking at the CI list we have the CI "UK InfoMaps Standard Users" and "FR LocalMaps Standard Users" so if the new user joins in the UK we add them to the former, in France we add them to the latter
2. As 1, except rather than adding them we just remove them
3. Again we have two active directory groups "UK InfoMaps Administrators" and "FR LocalMaps Administrators" so we can add them to the right group depending on the Country
4. The first non-straight forward one! We have the two DNS entries, the person taking the call can quickly test these and see if the service is down for everyone or just the user, if it'd down for everyone is the box down? Is the Hyper-V host down? Is the database accessible? Is there an error message? In short there are lots of things to try - and with access to the CI list you can start to do clever things like look for other services using the same Hyper-V host - if they're running then the problem probably isn't with the entire host, etc
5. Two DNS entries provide some points for testing - is Google down (it has been known ...)? Has the firewall changed so the ports are blocked?
6. IIS is linked to WIN005 so it should just be a quick case of searching the CMDB for IIS and seeing which boxes have IIS components on them
7. Is the SMTP server accessible? Is the user account locked?

As you can see there is a lot here that can be done with relatively little technical experience and (trust me as a developer - this bit is key!) *if* the incident eventually gets escalated to a software engineer to look at then there is going to be a lot more information in the call so, rather than having to chase people and get answers to simple questions like "what box is it on?", a lot of that information will already be in the call because whoever answered the phone will have already done most of that work. The key here is what do you want your software engineers doing - chasing users for answers or fixing issues and then getting on with doing other work?

The next part of this series (which will hopefully not take so long to put together) will continue this example and look at metrics and the things you might like to consider doing to keep your CMDB up-to-date and relevant as your business changes.

Designing and Building your CMDB, Part 1: From System Description To Configuration Items

This series of posts is going to be a slight departure from normal in that it won't be showing you any code. We are going through the process of designing a CMDB (that's a Configuration Management Database) to hold details for all the systems (500+) that we administer. The point of this post is to, by means of an example, show you the sort of questions you should be asking yourself when you put together a CMDB.

So let's start with a description of a system;

"System X is a fairly simple VB.NET solution deployed using IIS and installed on the server WIN005. It consists of two  applications; User Interface (an application open to all users), and Admin Interface (only available to a few).

The Admin Interface works on a vanilla IIS install but the User Interface requires the installation of  the Visual Studio 2010 Tools for Office Runtime.

The installation files for the software are located on WIN080\Software (a file share) as are the bi-monthly patch files that are applied.

At the back end the database, SYSXDB, is SQL Server 2008R2 and is held on a SQL Server cluster called SQLC001.

The application uses Active Directory for authentication, and the User Interface renders some information  from Google Maps to which it requires access.

The users of the Solution are spread across two Countries; France and the United Kingdom. We have internally configured the system so that in the UK users know the solution as as 'InfoMaps' and in France it's known as 'LocalMaps'."

I'm sure there are probably parts in that description that you'll recognise from systems you've worked on. As you can see despite it being only a fairly simple VB.Net website with a couple of plugins there is already quite a lot of information here to capture in our CMDB. If we take this structure and put it into Visio then as a system overview we get something like this;

System X: An Overview

Now for most small organisations this is probably 95% of the information they're ever going to need. If you're a small company and aren't expecting do significantly increase in size and you're not planning on managing hundreds of systems across the globe then you can make do - let's be honest we all have our "go to guy" for a particular system and so long as they're not on holiday (or haven't left!) then they can keep the system ticking over quite happily from both the users and managements perspecitive.

The problem comes when you don't just have one system, or a few, you start to have tens of systems like this and each system takes some time to administer. Suddenly your team of 3/4 software engineers don't really have any time to do anything new because they're too busy keeping the systems that the business is already relying on working to put in anything new.

Once you approach this level you need to significantly increase the quality of information you are holding on each system; you stop needing "Bob" to fix your problem but instead you need "someone who knows IIS" or "someone who can fix SQL Server". If all the knowledge is in Bob's head then Bob quickly becomes a bottle-neck through which all issues have to go through - this isn't good for Bob (although he might think in the short term that it is!) and it's certainly not good for the company or the users.

So let's go back to the description for System X and look for all the items in the configuration that we might want to store information on in our CMDB. Each of these items will become a Configuration Items (CI) in the CMDB. It's fairly easy looking at the system description to just pick things out;

• IIS
• WIN005
• User Interface
• Admin Interface
• Visual Studio 2010 Tools for Office Runtime
• WIN080\Software
• SYSXDB
• SQLC001
• Active Directory
• maps.google.com

This is a fairly long list, but is only part of the story. We (as IT Professionals) then need to take this list and add in the non-obvious things that will help us troubleshoot the system when there's a problem six months after it's gone live and we've all moved on to other projects. Again there is no easy way to do this and you're heavily reliant on vendors to provide "full and complete" information.

The sort of questions that need to be picked out from the system description are; have both applications been installed into the same Application Pool in IIS? Is the Application Pool running as a local user or is it using network credentials? How are we connecting to the database? Are users typing in http://win005 to access the site or have we setup DNS entries (http://infomaps for example)? How are we deciding if a user has access to the Admin Interface? Etc.

So let's assume someone technical has gone through the system, had the discussions with the vendor, and found out how everything is not just connected but configured. Here's the list of things we might like to consider turning into CI's in additional the ones we've already identified;

• Application Pool: SystemXUserInterface (Installed on WIN005)
• Application Pool: SystemXAdminInterface (Installed on WIN005)
• SYSTEMXSERVER (Active Directory account Used by both Application Pools and SQLC001 to grant access to SYSXDB)
• "UK InfoMaps Standard Users" (Active Directory Group, Used By "System X User Interface")
• "FR LocalMaps Standard Users " (Active Directory Group, Used by "System X User Interface")
• "UK InfoMaps Administrators" (Active Directory Group, Used By "System X User Interface")
• "FR LocalMaps Administrators" (Active Directory Group, Used by "System X User Interface")
• DNS Entry: LocalMaps.ourcompany.org (Maps to WIN005)
• DNS Entry: InfoMaps.ourcompany.org (Maps to WIN005)
• SMTP.ourcompany.org (Used by System X Admin Interface to send email notifications)
• Firewall Ports: 80,443 (Required for access to WIN005)
• VT001 (Hyper-V server hosting WIN005 - a virtual server)

Now this list is looking a little more comprehensive!

But how do we know we've captured everything? or even captured enough details for us to be able to properly support the system after we've put it in?

In Part 2 we'll look at "testing" our configuration to try and identify the gaps.