A bit of IT in developer's world: services.exe high CPU usage

By jay at March 24, 2011 12:36 Tags: , , ,

One of the advantages of virtualization is the P2V (Physical to Virtual) process: Converting an "old" build machine to a VM so it can be moved around with the load as-is, snapshotted, backed-up and so on.

This is particularly useful when say, you have a build machine that's been there for a very long time, has a lot of dependencies over old third party software, has been customized by so many people (that have long left the company) that if you wanted to rebuild that machine from scratch, it would literally take you weeks of tweaking to get it to work properly. And that machine is running out of very old hardware that may break at any time. And that the edition of Windows that does not migrate easily to new hardware because of HAL or Mass Storage issues, requiring a reinstallation. That a lot of "ands".

That's the kind of choice you do not need to make: You just take the machine and virtualize it using SCVMM 2008 R2.

But still, even virtualized, the machine been there that long, and things have started falling apart, like having the services.exe process taking 100% of the CPU. And I did not want to have to rebuild that machine just because of that strange behavior.

If you read scott hanselman's blog, you've been recalled that Windows Server 2008 and later has the resource monitor that gives a wealth of information about the services running under services.exe. But if you're out of luck, like running under Windows Server 2003, you can still use Process Explorer. This will give you the similar kind of insight in the Windows Services that are running.

For my particular issue, this was actually the Event Log service that was taking all the resources.

 

How about I get my CPU back ?

After some digging around, I noticed that :

  • All Event Viewer logging sections in the MMC snap-in were all displaying the same thing, which was actually a mix of all the System, Application and Security logs.
  • Displaying any of these logs was taking a huge amount of time to display.
  • The HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\System\Source was containing something like "System System System System System System System" a hundred of times, the same thing for the keys for the other event logs
  • A whole bunches (thousands) of interesting sources named like some .NET application domain created by the application being built on this machine

To fix it, a few steps in that order :

  • Disable the Event Log service and reboot. You won't be able to stop it, but at the next reboot it will not start.
  • In the C:\WINDOWS\system32\config folder, move the files *.evt to a temporary folder, so they don't get picked up by the service when it'll restart
  • In the registry, for each HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\[System|Security|Application]\Source, replace the content with the one found on the same key on another very similar Windows Server 2003 machine. You can install a brand new machine and pick up the content.
  • If you have, like I did, a whole bunch of sources that look familiar and should not be there under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\[System|Security|Application], remove their keys if you removed them from the "Source" value.
  • Set the Event Log service to "Automatic" and reboot.

 

The interesting part about the virtualization of that build machine is like in many other occasions, the snapshots, where you can make destructive changes and go back if they were actually too destructive.

 

What's with the event log "interesting sources" ?

The application being built and tested is running tests on the build machine, and it makes use of application domains and log4net. Log4net has an EventLogAppender that allows the push of specific content to the Windows Event Log. Log4net defaults the name of the source to the application domain name, if there is no entry assembly.

Those tests were actually using a default configuration, and were logging Critical messages to the event log, but the domains were created using a new GUID to avoid supposed name collisions. This is something that did actually more harm than good in the long run, because each new appdomain that was logging to the event log was creating a new event source.

And the build system has been there for a long time. Hence the thousands of "oddly named" event sources.

 

Inter-Domain Trust Relationship and lmhosts Text Casing

By Jerome at October 13, 2004 11:40 Tags: ,

Among the things that are interesting in latest versions of Samba, there are NT4 Inter-Domain Trust Relationships. In the Windows world, the original NT4 domain model was not scalable enough when NT4 was omnipresent. Companies fusions did not mix well with the single windows domain to merge all user accounts and everything else.

One solution brought back then was the ability to have an NT4 domain to trust users authenticated in an other NT4 domain. This comes in multiple flavors known as incoming, outgoing, bidirectional trust to allow users to be authenticated in both or only selected domains. Other things like transitive and non-transitive trusts can also be used in at least a 3 domains interaction to allow users from domain, say, A to be authenticated in domain C through B, if the trust between A and B is transitive.

This facility is now fully integrated in Windows 2000/2003 domains and this allows to trust NT4 style domains, which includes Samba managed domains.

Here at epitech we a growing need to interconnect a number of small domains to allow users to connect to a variety of services and to allow an unified password management, a samba domain is used to map users between the Windows and the Unix world.

When establishing the trust between two domains , there are at least two possible scenarios :

  • The domain can be identified using a Netbios broadcast or WINS resolution, which implies that the domain is known and has registerd itself to the wins or is located on the same physical IPv4 subnet,
  • Or, the domain cannot be identified using a single broadcast and must be identified via a manual addition of the domain in the WINS or the domain and the associated domain controller have been added to the lmhosts file.

During a lot of trying, I have found that a combination of both the WINS and lmhosts modification are needed to establish the trust. If you don't do both, the domain controller that wants to establish the trust tries to partially resolve the remote DC by using a really weird NETLOGON/UDP packet that is of course rejected (and not even logged) by Samba. The rest is done by an attempt to locate the remote DC by a local subnet broadcast, which of course fails.

You might say : "Well, modifying the lmhosts file should be a piece of cake !". Actually yes, writing it is. So here is what you might want to add in your lmhosts file :

     10.0.0.1    mysamba-dc             #PRE   #DOM:midhearth
     10.0.0.1   "midearth       \0x1b"  #PRE

Which seems to be good. By the way, the \x1b is used to identify the midearth entry as a domain and should be placed at the 16 byte index. Historical laziness... What a shame.  Anyway, this does not work and produces the strange behavior I described earlier. Here is a version of the same block of text that really works :

     10.0.0.1    MYSAMBA-DC             #PRE   #DOM:MIDEARTH
     10.0.0.1   "MIDEARTH       \0x1b"  #PRE

Noticing anything ? Yes ! All names are uppercased... And no error or warning message notifies you about this when casing is not correct. After that, everything works fine and the trust can be establised. I kind of hate losing time with that kind of tricks, but hey, it works now :)

About me

My name is Jerome Laban, I am a Software Architect, C# MVP and .NET enthustiast from Montréal, QC. You will find my blog on this site, where I'm adding my thoughts on current events, or the things I'm working on, such as the Remote Control for Windows Phone.