Unattended Windows setup may fail due to wrongly encoded password

While testing unattended Windows deployment I encountered a problem.

After the first installation reboot, Windows setup would fail with an empty dialog box:

install_windows_empty

 

Followed by the error message:

install_windows_windows_could_not_complete_the_installation

Windows could not complete the installation. To install Windows on this computer, restart the installation.

 

Then the machine would countinually reboot and show the last message.

 

Started troubleshooting by booting into Windows PE.

Then examined setuperr.log and setupact.log under:

C:\Windows\Panther\

However these files contained no useful clues.

 

I checked the other files left by Windows setup, and examined the files under:

C:\Windows\Panther\UnattendGC\

Found this in setuperr.log:

[oobeldr.exe] [Action Queue] : Unattend action failed with exit code 4
[oobeldr.exe] Execution of unattend GCs failed; hr = 0x0; pResults->hrResult = 0x8030000b
[oobeldr.exe] User input error was detected in unattend file. Error: [0x0]

[windeploy.exe] Command [%windir%\system32\oobe\oobeldr.exe /system] failed with exit code [0x8030000b]
[windeploy.exe] Failure occured during online installation.  Online installation cannot complete at this time.; hr = 0x8030000b

 

But it was a warning in setupact.log which revealed the cause behind the error:

[Shell Unattend] Failed to decode password (0x8007000d)

 

My initial understanding of the user password format in AutoUnattend.xml turned out to be wrong.

This problem was not discovered by Windows System Image Manager, because AutoUnattend.xml was updated just before deployment by a custom program on the target computer.

 

Local user passwords are encoded like this in AutoUnattend.xml:

1. Text is initially encoded as unicode / UTF-16.

2. Then a “Password” string is appended to the password.
Example: The password “1234” is represented as “1234Password”

3. Finally the password string is base64 encoded.

Running .NET programs under Windows PE

Windows PE can be modified to include PowerShell as well as .NET like described here:

https://msdn.microsoft.com/en-us/windows/hardware/commercialize/manufacture/desktop/winpe-adding-powershell-support-to-windows-pe

 

Windows PE is limited to running one type of executable (either x86, AMD64 or ARM), depending on the type of Windows PE image.

This also affects .NET applications.

 

.NET programs compiled for any CPU may fail on AMD64 versions of Windows PE with:

The subsystem needed to support the image type is not present.

 

This can happen if the “Prefer 32-bit” build option is set, which is the default for some project types with Visual Studio 2015.

 

The solution is either to remove the “Prefer 32-bit” option before compiling.

vs_build_deselect_prefer_32-bit

 

Or using corflags to remove any 32-bit flags with:

corflags program.exe /32BITREQ- /32BITPREF-

Problems when reusing AutoUnattend.xml with new Windows image

I recently experienced problems when reusing AutoUnattend.xml, after having upgraded the Windows image:

From: Windows 10 Enterprise 2015 LTSB

To: Windows 10 Enterprise 2016 LTSB

 

The unattended installation would start and run, but eventually failed with:

windows_setup_windows_could_not_apply_unattend_settings_during_pass_offlineservicing

Windows could not apply unattend settings during pass [offlineServicing].

 

Examined the installation logfiles under:

\$WINDOWS.~BT\Sources\Panther

 

setuperr.log only contained:

2016-10-20 15:20:20, Error      [0x0606ae] IBS    [SetupCl library] Required profile hive does not exist: [\??\D:\WINDOWS\system32\config\systemprofile\NTUSER.DAT].
2016-10-20 15:20:37, Error      [0x0604a7] IBS    InstantiateCBSUnattendPass: dism.exe returned with failing exit code -2146498555
2016-10-20 15:20:37, Error      [0x060431] IBS    Callback_UnattendInitiatePass: An error occurred while initiating unattend passes; hr = 0x80004005

 

setupact.log contained no additional useful information.

 

However cbs_unattend.log contained an explanation:

2016-10-20 15:20:36, Error                 DISM   DISM Package Manager: PID=2348 TID=2368 Failed opening package Microsoft-Windows-Foundation-Package~31bf3856ad364e35~amd64~~10.0.10240.16384. - CDISMPackageManager::Internal_CreatePackageByName(hr:0x800f0805)
2016-10-20 15:20:36, Error                 DISM   DISM Package Manager: PID=2348 TID=2368 Failed to get the underlying cbs package. - CDISMPackageManager::OpenPackageByName(hr:0x800f0805)
2016-10-20 15:20:36, Error                 DISM   DISM Package Manager: PID=2348 TID=2368 The specified package is not valid Windows package. - GetCbsErrorMsg

 

I wondered why the Microsoft-Windows-Foundation-Package could not be opened, so I went back to Windows System Image Manager.

windows_system_image_manager_unknown_package

There I noticed that the package was unknown, because the version numbers had changed.

Be aware that I had run: Tools -> Validate Answer File

(Which did not show warnings about this)

 

Added the new Microsoft-Windows-Foundation-Package

Then copied all the settings

After verifying that all settings had been copied, I deleted the unknown package.

(It would probably have been easier to update the version number in AutoUnattend.xml, which I recommend trying first)

 

Then ran: Tools -> Validate Answer File

Which now showed these warnings:

Windows Feature is enabled but one or more of its dependencies have not been enabled in the answer file.

Packages/Foundation/amd64_Microsoft-Windows-Foundation-Package_10.0.14393.0__31bf3856ad364e35_/Client-DeviceLockdown/Client-EmbeddedBootExp
Packages/Foundation/amd64_Microsoft-Windows-Foundation-Package_10.0.14393.0__31bf3856ad364e35_/Client-DeviceLockdown/Client-EmbeddedLogon
Packages/Foundation/amd64_Microsoft-Windows-Foundation-Package_10.0.14393.0__31bf3856ad364e35_/Client-DeviceLockdown/Client-EmbeddedShellLauncher

 

I was challenged by this until I found the answer here:

http://blog.theatticnetwork.net/2014/08/windows-unattend-file-notes/

 

I had to right-click the feature in question and choose: Enable Parent Features

windows_system_image_manager_enable_parent_features

 

After making these changes new configuration sets / deployment images could be installed successfully.

Conclusion

When reusing AutoUnattend.xml with a new Windows image, please look for unknown packages in Windows SIM in addition to validating the answer file.

If any unknown packages are found, please update the version numbers in AutoUnattend.xml.

Debugging failing SSRS report caused by read-only report parameter

I recently experienced problems when trying to view a report for SQL Server Reporting Services with ReportViewer for MVC.

 

The report worked in some cases, but failed when certain parameters were set.

Exception and stack trace:

[ReportServerException: The report parameter 'DeviceTypes' is read-only and cannot be modified. (rsReadOnlyReportParameter)]
Microsoft.Reporting.WebForms.ServerReportSoapProxy.OnSoapException(SoapException e) +155
Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.ProxyMethodInvocation.Execute(RSExecutionConnection connection, ProxyMethod`1 initialMethod, ProxyMethod`1 retryMethod) +1256
Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.RSExecutionConnection.SetExecutionParameters(ParameterValue[] Parameters, String ParameterLanguage) +370
Microsoft.Reporting.WebForms.SoapReportExecutionService.SetExecutionParameters(IEnumerable`1 parameters, String parameterLanguage) +530
Microsoft.Reporting.WebForms.ServerReport.SetParameters(IEnumerable`1 parameters) +896
Microsoft.Reporting.WebForms.Report.SetParameters(ReportParameter parameter) +136
ReportViewerForMvc.ReportExtensions.SetParameters(Report report, ReportParameterInfoCollection collection) +114
ReportViewerForMvc.ReportViewerExtensions.SetProperties(ServerReport serverReport, ServerReport properties) +84
ReportViewerForMvc.ReportViewerExtensions.SetProperties(ReportViewer reportViewer, ReportViewer properties) +117
ReportViewerForMvc.ReportViewerWebForm.BuildReportViewer() +92
System.Web.UI.Control.OnLoad(EventArgs e) +109
System.Web.UI.Control.LoadRecursive() +68
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +4498

 

Checked the ReportServerService__*.log and ExecutionLog* database views, but they contained no additional clues about the error.

 

I attached WinDbg (x64) to the IIS worker process (w3wp.exe) and set a breakpoint with:

!bpmd   ReportViewerForMvc.dll   ReportViewerForMvc.ReportExtensions.SetParameters

Messages from WinDbg:

Found 2 methods in module 00007ff9bde79bb0...
MethodDesc = 00007ff9c00a30d0
MethodDesc = 00007ff9c00a30e0
Setting breakpoint: bp 00007FF9C0EE5590 [ReportViewerForMvc.ReportExtensions.SetParameters(Microsoft.Reporting.WebForms.Report, Microsoft.Reporting.WebForms.ReportParameterInfo)]
Setting breakpoint: bp 00007FF9C0EE53E4 [ReportViewerForMvc.ReportExtensions.SetParameters(Microsoft.Reporting.WebForms.Report, Microsoft.Reporting.WebForms.ReportParameterInfoCollection)]
Adding pending breakpoints...

 

Continued exection, reloaded the report and the breakpoint was hit.

Checked the .NET call stack with:

!clrstack -a

Part of the result:

OS Thread Id: 0x273c (45)
Child SP               IP Call Site
00000007dba9d030 00007ff9c0ee53e4 ReportViewerForMvc.ReportExtensions.SetParameters(Microsoft.Reporting.WebForms.Report, Microsoft.Reporting.WebForms.ReportParameterInfoCollection)
PARAMETERS:
report (<CLR reg>) = 0x0000000390e93650
collection (<CLR reg>) = 0x0000000490886678

 

Examined the collection object with:

!DumpObj /d 0000000490886678

Result:

Name:        Microsoft.Reporting.WebForms.ReportParameterInfoCollection
MethodTable: 00007ff9bff1cc98
EEClass:     00007ff9bff08d40
Size:        32(0x20) bytes
File:        C:\Windows\assembly\GAC_MSIL\Microsoft.ReportViewer.WebForms\11.0.0.0__89845dcd8080cc91\Microsoft.ReportViewer.WebForms.dll
Fields:
MT    Field   Offset                 Type VT     Attr            Value Name
00007ffa1b23d408  40017fa        8 ...Canon, mscorlib]]  0 instance 00000004908823f0 list
00007ffa1b8de068  40017fb       10        System.Object  0 instance 0000000000000000 _syncRoot

 

Examined the list array with:

!DumpArray /d 00000004908823f0

Result:

Name:        Microsoft.Reporting.WebForms.ReportParameterInfo[]
MethodTable: 00007ff9c0fd9e60
EEClass:     00007ffa1b2f0f10
Size:        120(0x78) bytes
Array:       Rank 1, Number of elements 12, Type CLASS
Element Methodtable: 00007ff9bff1cbc0
[0] 00000004908826c8
[1] 00000004908829d8
[2] 0000000490882ce8
[3] 00000004908832e0
[4] 0000000490884490
[5] 00000004908847a8
[6] 0000000490884ac0
[7] 0000000490884e38
[8] 00000004908853f8
[9] 0000000490885818
[10] 0000000490886250
[11] 00000004908865c8

 

Examined details for the list array with:

!DumpArray /d -details 00000004908823f0

Part of the result:

[11] 00000004908865c8
Name:        Microsoft.Reporting.WebForms.ReportParameterInfo
MethodTable: 00007ff9bff1cbc0
EEClass:     00007ff9bff08cc8
Size:        104(0x68) bytes
File:        C:\Windows\assembly\GAC_MSIL\Microsoft.ReportViewer.WebForms\11.0.0.0__89845dcd8080cc91\Microsoft.ReportViewer.WebForms.dll
Fields:
MT    Field   Offset                 Type VT     Attr            Value Name
00007ffa1b8dda88  4000120        8            System.String      0     instance     0000000490881950     m_name
00007ff9bff1c678  4000121       50             System.Int32      1     instance                    4     m_dataType
00007ffa1b8cd6f8  4000122       58           System.Boolean      1     instance                    0     m_isNullable
00007ffa1b8cd6f8  4000123       59           System.Boolean      1     instance                    1     m_allowBlank
00007ffa1b8cd6f8  4000124       5a           System.Boolean      1     instance                    1     m_isMultiValue
00007ffa1b8cd6f8  4000125       5b           System.Boolean      1     instance                    1     m_isQueryParameter
00007ffa1b8dda88  4000126       10            System.String      0     instance     000000038e411420     m_prompt
00007ffa1b8cd6f8  4000127       5c           System.Boolean      1     instance                    0     m_promptUser
00007ffa1b8cd6f8  4000128       5d           System.Boolean      1     instance                    1     m_areDefaultValuesQueryBased
00007ffa1b8cd6f8  4000129       5e           System.Boolean      1     instance                    1     m_areValidValuesQueryBased
00007ffa1b8dda88  400012a       18            System.String      0     instance     0000000000000000     m_errorMessage
00007ff9c0fda240  400012b       20     ...Viewer.WebForms]]      0     instance     0000000490886320     m_validValues
00007ffa1b23d128  400012c       28     ...tring, mscorlib]]      0     instance     0000000490886658     m_currentValues
00007ff9bff1c7c8  400012d       54             System.Int32      1     instance                    0     m_state
00007ff9bff1cc98  400012e       30     ...terInfoCollection      0     instance     0000000490886fb8     m_dependencyCollection
00007ff9bff1cc98  400012f       38     ...terInfoCollection      0     instance     0000000000000000     m_dependentsCollection
00007ffa1b8c1480  4000130       40          System.String[]      0     instance     0000000490881bb8     m_dependencies
00007ff9c0fda5d0  4000131       48     ...Viewer.WebForms]]      0     instance     0000000490886630     m_dependentsCollectionConstruction
00007ffa1b8cd6f8  4000132       5f           System.Boolean      1     instance                    1     m_visible

 

Examined the name of the last report parameter with:

!DumpObj /d 0000000490881950

Part of the result:

Name:        System.String
MethodTable: 00007ffa1b8dda88
EEClass:     00007ffa1b236a08
Size:        48(0x30) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      DeviceTypes

So the last parameter was DeviceTypes, the one causing the problem.

 

Looked for a read-only property among the report parameters. Noticed that m_promptUser was 0 for the DeviceTypes parameter, while m_promptUser was 1 for all other parameters.

 

I decided to use procdump to log exception messages from the ReportingServicesService.exe process with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

 

Reloaded the report to recreate the error.

Part of the result was:

[07:10:37] Exception: E0434F4D.Microsoft.ReportingServices.Diagnostics.Utilities.ReadOnlyReportParameterException ("The report parameter 'DeviceTypes' is read-only and cannot be modified.")
[07:10:37] Exception: E0434F4D.Microsoft.ReportingServices.Diagnostics.Utilities.RSException ("The report parameter 'DeviceTypes' is read-only and cannot be modified.")

 

Then loaded all DLL files from SSRS into ILSpy from:

C:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\ReportServer\bin

 

Searched for the ReadOnlyReportParameterException class in ILSpy.

Found it and noticed that it was apparently only used by:

Microsoft.ReportingServices.ReportProcessing.ParameterInfoCollection.Combine(...)

 

Read the decompiled code for the above method and noticed this part:

if (checkReadOnly && !parameterInfo.PromptUser)
{
  ParameterInfoCollection.ThrowReadOnlyParameterException(parameterInfo.Name, isSharedDataSetParameter);
}

Apparently the value of parameterInfo.PromptUser determined the read-only status.

 

I decided to examine the XML for the RDL report and noticed that the DeviceTypes report parameter did not have a Prompt section.

I added this:

<Prompt>DeviceTypes</Prompt>

Then uploaded the new version of the report (by overwriting).

However this did not seem to have any immediate effect, the error still occured…

 

I spent some time examining various other ideas and internals of SSRS.

 

Then I got an idea: I decided to delete the problematic report and then recreate the modified version with a Prompt section.

This fixed the problem! The report now worked in all cases, no matter which parameters were set.

This behavior was consistent across 2 SSRS instances.

Conclusion

It seems that read-only status for a report parameter is derived from the presence of a <Prompt>Description</Prompt> section.

However this status is not updated if the report is overwritten, it has to be deleted and then recreated…

 

Due to the way ReportViewerForMvc.dll is implemented all report parameters are copied from the initial instance to another instance of ReportViewer.

This means that report parameters can’t be read-only, so <Prompt>Description</Prompt> should always be specified for all report parameters, when using ReportViewer for MVC.

Separate, physical trackpoint buttons on Lenovo Thinkpad T440p

The Lenovo Thinkpad T440p (and other models of that generation) is delivered with a touchpad, without separate physical left, middle and right buttons.

Instead the entire pad clicks and reacts depending on the area touched.

 

In my subjective opinion these buttons feel spongy and imprecise.

In use it’s common to make mistakes by clicking another button than expected.

This makes the laptop less productive and frustrating to use.

 

However it’s possible to replace the touchpad with the one from the Lenovo Thinkpad T450, which has 3 separate, physical trackpoint buttons.

thinkpad_t440p_with_t450_trackpad

 

The first challenge is getting the right replacement part, with the dimensions 10 cm x 7,5 cm.

thinkpad_t440p_clickpad_horizontal

thinkpad_t440p_clickpad_vertical

It’s not available as a separate part from Lenovo, but is sold as part of the keyboard bezel.

The part number I found and used was: 00HN550

 

Be careful with online sellers claiming to sell touchpads that fit a long range of Thinkpad models.

They may fit electrically, but possibly not physically.

If you are considering performing this replacement, please verify that the part fits your particular Thinkpad model.

 

The next challenge is to disassemble the laptop and performing the replacement.

I refer to the hardware maintenance manual and online guides.

 

The final challenge is to solve driver problems on Windows.

The hardware ID for the touchpad is on the motherboard, which remains unchanged.

The default Synaptics Pointing Device drivers are not compatible and won’t work.

 

Simplest way to solve the driver problems on Windows:

1. Connect a USB mouse, because the trackpoint won’t work reliably until these steps have been completed.

2. Uninstall the Synaptics Pointer Device drivers using Programs and Features.

3. Restart the computer.

4. Remove any remaining Synaptics components by opening Control Panel -> Mouse

If asked: Do you want to uninstall the Synaptics driver now?

Then select yes and OK to the following dialogs:

synaptics_driver_uninstall1

synaptics_driver_uninstall2

synaptics_driver_uninstall3

5. Restart the computer.

6. Now the trackpoint and 3 physical buttons should work with a default mouse driver.

(Be aware that I have disabled the rest of the touchpad, so I don’t know if it works with the default mouse driver)

 

With Windows 10 extra steps are needed, because it can automatically install incompatible drivers.

This can be prevented by downloading and running the “Show or hide updates” program (wushowhide.diagcab) from:

https://support.microsoft.com/en-us/kb/3073930?utm_source=twitter

 

1. Click: Advanced

disable_driver_updates1

2. Deselect: Apply repairs automatically

disable_driver_updates2

3. Click: Next

4. Click: Hide updates

disable_driver_updates3

5. Select: Synaptics – Pointing Drawing – Synaptics Pointing Device

disable_driver_updates4

6. Click: Next

7. Confirm by clicking: Next

disable_driver_updates5

8. Click: Close the troubleshooter

disable_driver_updates6

 

Be aware that fully compatible drivers can be downloaded and installed from Lenovo, which will enable full touchpad functionality.

However I’m currently satisfied with a trackpoint and 3 physical buttons, so I have not found the correct drivers or procedure yet.

Debugging blank SSRS report caused by SecurityException

I was testing reports for SQL Server Reporting Services on a new computer, but the results were blank and no relevant error messages were logged by SSRS.

 

Investigated by using procdump to log exception messages from the IIS worker process with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

Part of the result was:

[13:22:13] Exception: E0434F4D.System.Security.SecurityException ("Request for the permission of type 'System.Net.WebPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed.")
[13:22:13] Exception: E0434F4D.System.Security.SecurityException ("Request for the permission of type 'System.Security.Permissions.SecurityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed.")

 

I knew that the reports performed HTTP/web service calls from VB code, so I had to modify rssrvpolicy.config to allow this.

It can be done by:

1. Appending a named permission set with both execution and web permission like:

<PermissionSet class="NamedPermissionSet"
version="1"
Name="ExecutionAndWebPermission"
Description="A special permission set that grants execution and web permission for specific site(s).">
  <IPermission class="SecurityPermission" version="1" Flags="Execution" />
  <IPermission class="WebPermission" version="1">
    <ConnectAccess>
      <URI uri="http://localhost/Test/ReportingUtilityService.svc"/>
    </ConnectAccess>
  </IPermission>
</PermissionSet>

 

2. Modifying the CodeGroup with Name=”Report_Expressions_Default_Permissions”

From:

PermissionSetName="Execution"

To:

PermissionSetName="ExecutionAndWebPermission"

 

3. Restarting Reporting Services.

 

I decided to verify that this was the issue before performing the changes.

Therefore I attached WinDbg (x64) to ReportingServicesService.exe and configured event filters to break on CLR exceptions.

 

Refreshed the report again and the first exception was thrown.

Checked the exception with:

!PrintException

 

The result was a very detailed (and useful) message, that confirmed the suspected issue:

Exception object: 0000000460198410
Exception type: System.Security.SecurityException
Message: Request for the permission of type 'System.Net.WebPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed.
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 8013150a
SecurityException Message:
The action that failed was:
Demand
The type of the first permission that failed was:
System.Net.WebPermission
The first permission that failed was:
<IPermission class="System.Net.WebPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1">
<ConnectAccess>
<URI uri="http://localhost/Test/ReportingUtilityService\.svc"/>
</ConnectAccess>
</IPermission>

The demand was for:
<IPermission class="System.Net.WebPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1">
<ConnectAccess>
<URI uri="http://localhost/Test/ReportingUtilityService\.svc"/>
</ConnectAccess>
</IPermission>

The granted set of the failing assembly was:
<PermissionSet class="System.Security.PermissionSet"
version="1">
<IPermission class="System.Security.Permissions.SecurityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1"
Flags="Execution"/>
<IPermission class="System.Security.Permissions.StrongNameIdentityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1"
PublicKeyBlob="..."
Name="expression_host_..."
AssemblyVersion="1.0.0.0"/>
<IPermission class="System.Security.Permissions.ZoneIdentityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1"
Zone="MyComputer"/>
</PermissionSet>

The assembly or AppDomain that failed was:
expression_host_..., Version=12.2.6020.0, Culture=neutral, PublicKeyToken=null
The method that caused the failure was:
System.Object CallWebService(System.String, System.String)
The Zone of the assembly that failed was:
MyComputer

 

After modifying rssrvpolicy.config and restarting Reporting Services the issue was resolved.

Windows Server may shut down if system time is wrong

I was testing some Reporting Services reports, but the test data was fairly old and some datasets were linked to the date, so I wasn’t getting proper results.

Instead of modifying the test data or the queries, the easy solution was to disable time synchronization and change the system time to 3 years in the past.

 

But changing the system time lead to an interesting problem:

Suddenly the server shut down automatically with no warning.

No chance was given to save open documents or to cancel shutdown.

 

I started up the virtual machine again and noticed that the system time had been reset.

Found this explanation for the behavior in Event Viewer:

Log Name:      System
Source:        User32
Date:          27-09-2013 11:14:19
Event ID:      1074
Task Category: None
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      TestServer
Description:
The process C:\Windows\system32\wlms\wlms.exe (TESTSERVER) has initiated the shutdown of computer TESTSERVER on behalf of user NT AUTHORITY\SYSTEM for the following reason: Other (Planned)
Reason Code: 0x80000000
Shutdown Type: shutdown
Comment: The license period for this installation of Windows has expired. The operating system is shutting down.

 

Be aware that Windows was activated and continued to be activated after it was restarted.

This happened with Windows Server 2012 R2 Standard, but I suspect it’s an issue with other versions and editions.

Conclusion

Apparently Windows doesn’t tolerate running with a wrong system time, even for testing purposes.

Fortunately wlms.exe is only present on evaluation versions of Windows, so this particular issue should never occur on production systems.

In any case, a correct system time is always desired for a number of things like logging dates, scheduling, synchronization, certificate validation and so on.

Debugging endless loop caused by missing SignalR performance counters

I recently experienced a problem with an IIS website:

After changing and building code, the site would hang indefinitely at startup.

It seemed like an endless loop, because CPU activity was high.

 

One workaround was to reset IIS with:

iisreset

Then the site became functional until the next change + build.

 

However I decided to find the cause of the problem and a real solution.

 

I checked the process activity for w3wp.exe with Process Monitor:

These parts of the registry were continually accessed:

process_monitor_w3wp_registry_signalr_performance

HKLM
HKLM\SYSTEM\CurrentControlSet\Services\SignalR\Performance
HKLM\System\CurrentControlSet\Services\SignalR\Performance

 

At this point I should have checked if the system actually had SignalR performance counters installed…

 

However I decided to examine the situation further by attaching WinDbg (x64) to the w3wp.exe process.

 

Because I was suspecting an endless loop I decided to check the thread CPU activity with:

!runaway

Part of the result was:

User Mode Time
Thread       Time
6:26cc      0 days 0:05:39.750
15:2e00      0 days 0:00:11.187
13:27b0      0 days 0:00:06.718

 

Thread 6 was definitely the one with the most activity, so I switched to that with:

~6s

 

Then checked the .NET call stack with:

!clrstack

Part of the result was:

OS Thread Id: 0x26cc (6)
Child SP               IP Call Site
000000f1a0c6b158 00007ff98b60b1a6 [HelperMethodFrame: 000000f1a0c6b158] System.Security.SecurityRuntime.GetSecurityObjectForFrame(System.Threading.StackCrawlMark ByRef, Boolean)
000000f1a0c6b260 00007ff98a4b584c System.Security.SecurityRuntime.RevertAssert(System.Threading.StackCrawlMark ByRef)
000000f1a0c6b290 00007ff98a4b9e3d System.Security.PermissionSet.RevertAssert()
000000f1a0c6b2c0 00007ff989cd5e6d System.Diagnostics.PerformanceCounterLib.FindCustomCategory(System.String, System.Diagnostics.PerformanceCounterCategoryType ByRef)
000000f1a0c6b310 00007ff989cd5db8 System.Diagnostics.PerformanceCounterLib.FindCustomCategory(System.String, System.Diagnostics.PerformanceCounterCategoryType ByRef)
000000f1a0c6b380 00007ff989cd735c System.Diagnostics.PerformanceCounterLib.IsCustomCategory(System.String, System.String)
000000f1a0c6b3e0 00007ff989cdd1af System.Diagnostics.PerformanceCounter.InitializeImpl()
000000f1a0c6b450 00007ff989cdc35f System.Diagnostics.PerformanceCounter..ctor(System.String, System.String, System.String, Boolean)
000000f1a0c6b4a0 00007ff92f81b7b5 Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.LoadCounter(System.String, System.String, System.String, Boolean)
000000f1a0c6b550 00007ff92f81b606 Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.SetCounterProperties()
000000f1a0c6b5c0 00007ff92f81b2d7 Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.Initialize(System.String, System.Threading.CancellationToken)
000000f1a0c6b640 00007ff92f81a23b Microsoft.AspNet.SignalR.Hosting.HostDependencyResolverExtensions.InitializeHost(Microsoft.AspNet.SignalR.IDependencyResolver, System.String, System.Threading.CancellationToken)
000000f1a0c6b680 00007ff92f81859b Owin.OwinExtensions.UseSignalRMiddleware[[System.__Canon, mscorlib]](Owin.IAppBuilder, System.Object[])
000000f1a0c6b770 00007ff92f818147 Owin.OwinExtensions+c__DisplayClass1.b__0(Owin.IAppBuilder)
000000f1a0c6b7b0 00007ff92f816c22 Owin.MapExtensions.Map(Owin.IAppBuilder, Microsoft.Owin.PathString, System.Action`1)
000000f1a0c6b800 00007ff92f816993 Owin.MapExtensions.Map(Owin.IAppBuilder, System.String, System.Action`1)
000000f1a0c6b850 00007ff92f816814 Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder)

 

After continuing and breaking execution a few times I was seeing similar call stacks.

I needed to identify the endless loop.

 

Experimented by setting breakpoints from the above call stack.

 

Set a single breakpoint at:

00007ff989cd735c System.Diagnostics.PerformanceCounterLib.IsCustomCategory(System.String, System.String)

With:

bc *
bp 00007ff989cd735c

Result: Breakpoint was continually hit.

 

Set a single breakpoint at:

00007ff989cdd1af System.Diagnostics.PerformanceCounter.InitializeImpl()

With:

bc *
bp 00007ff989cdd1af

Result: breakpoint was not hit within 1 minute.

 

This indicated that the endless loop was in:

System.Diagnostics.PerformanceCounterLib.IsCustomCategory(System.String, System.String)

 

I used ILSpy to decompile the method and noticed this loop:

for (CultureInfo cultureInfo = CultureInfo.CurrentCulture; cultureInfo != CultureInfo.InvariantCulture; cultureInfo = cultureInfo.Parent)
{
    performanceCounterLib = PerformanceCounterLib.GetPerformanceCounterLib(machine, cultureInfo);
    if (performanceCounterLib.IsCustomCategory(category))
    {
        return true;
    }
}

 

I noted that the loop condition likely continued to be true.

And that performanceCounterLib.IsCustomCategory(category) most likely continued to be false.

This was likely the endless loop I was looking for.

 

I set an additional breakpoint with:

!bpmd   System.dll   System.Diagnostics.PerformanceCounterLib.IsCustomCategory

Messages from WinDbg:

Found 2 methods in module 00007ff989421000...
MethodDesc = 00007ff989524020
MethodDesc = 00007ff989524040
Setting breakpoint: bp 00007FF989CD73EB [System.Diagnostics.PerformanceCounterLib.IsCustomCategory(System.String)]
Setting breakpoint: bp 00007FF989CD727A [System.Diagnostics.PerformanceCounterLib.IsCustomCategory(System.String, System.String)]
Adding pending breakpoints...

 

However when continuing only breakpoint 0 was hit, further indicating that the endless loop occured inside PerformanceCounterLib.IsCustomCategory(…).

 

Decided to look for the CultureInfo object in the loop.

 

Checked the call stack including locals and parameters with:

clrstack -a

Top part of the call stack was:

OS Thread Id: 0x26cc (6)
Child SP               IP Call Site
000000f1a0c6b380 00007ff989cd735c System.Diagnostics.PerformanceCounterLib.IsCustomCategory(System.String, System.String)
PARAMETERS:
machine (<CLR reg>) = 0x000000f4a155df18
category (<CLR reg>) = 0x000000f4a1584228
LOCALS:
<CLR reg> = 0x000000f1a158f130

 

Examined the local reference with:

!DumpObj /d 000000f1a158f130

Part of the result was:

Name:        System.Globalization.CultureInfo
MethodTable: 00007ff98a701420
EEClass:     00007ff98a111e48
Size:        128(0x80) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT    Field   Offset                 Type VT     Attr            Value Name
00007ff98a6ed6f8  4001208       74       System.Boolean  1 instance                1 m_isReadOnly
00007ff98a6e0698  4001209        8 ...ation.CompareInfo  0 instance 000000f1a1593b40 compareInfo
...
...
00007ff98a6ed6f8  4001212       75       System.Boolean  1 instance                0 m_isInherited
00007ff98a6ed6f8  4001213       76       System.Boolean  1 instance                1 m_isSafeCrossDomain
00007ff98a7003d0  4001214       70         System.Int32  1 instance                2 m_createdDomainID
00007ff98a701420  4001215       40 ...ation.CultureInfo  0 instance 0000000000000000 m_consoleFallbackCulture
00007ff98a6fda88  4001216       48        System.String  0 instance 000000f1a1551420 m_name
00007ff98a6fda88  4001217       50        System.String  0 instance 000000f1a1551420 m_nonSortName
00007ff98a6fda88  4001218       58        System.String  0 instance 000000f1a1551420 m_sortName
00007ff98a701420  4001223       60 ...ation.CultureInfo  0 instance 000000f1a158f130 m_parent
...
...

 

Noticed that m_parent pointed to itself.

 

Cleared all breakpoints with:

bc *

Set another breakpoint with:

!bpmd   System.dll   System.Diagnostics.PerformanceCounterLib.FindCustomCategory

 

This breakpoint was continually hit as expected.

 

Then checked the PerformanceCounterLib.FindCustomCategory(…) method with ILSpy.

Noticed that the method accessed the registry and that the return value depended on what was found.

 

Searched online for: HKLM\System\CurrentControlSet\Services\SignalR\Performance

And found this article: http://www.asp.net/signalr/overview/performance/signalr-performance

 

Based on the article I decided to install SignalR performance counters by:

1. Installing the NuGet package: signalr.utils

2. Then installing the SignalR performance counters with:

SignalR.exe ipc

 

This was the solution!

After continuing execution, the loop ended and site initialization continued normally.

(The issue occured with version 2.2.1 of SignalR and version 4.6.1 of .NET, but targeting version 4.5)

Conclusion

A site using SignalR can hang with an endless loop when initializing, if the SignalR performance counters are not installed.

A combination of tools like Process Monitor, ILSpy and WinDbg can be used to discover the cause of an endless loop in 3rd party or framework code.