Debugging #Error in SSRS report caused by attempts to convert non-numeric strings

While testing another modified SSRS report I noticed that certain values were not shown, only the message:

#Error

 

Checked the ReportServerService__*.log and ExecutionLog* database views, but they contained no clues about the error.

 

Then I used procdump to log exception messages from ReportingServicesService.exe with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

 

Part of the result was:

[13:15:59] Exception: E0434F4D.System.FormatException ("Input string was not in a correct format.")
[13:15:59] Exception: E0434F4D.System.FormatException ("Input string was not in a correct format.")
[13:15:59] Exception: E0434F4D.System.InvalidCastException ("Conversion from string " norm" to type 'Decimal' is not valid.")

 

I extracted and ran the dataset query in SQL Server Management Studio.

Then looked for “norm” strings and found them in a column normally used to store numbers.

Attempts to convert these caused the problem.

 

My solution was to exclude them from the dataset query by adding:

AND ISNUMERIC(ValueString) = 1

Debugging #Error in SSRS report caused by missing function overload

While testing a modified SSRS report I noticed that certain values were not shown, only the message:

#Error

 

Neither the ReportServerService__*.log or ExecutionLog* database views contained any clues about the cause.

 

I used procdump to log exception messages from ReportingServicesService.exe with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

Part of the result was:

[13:46:39] Exception: E0434F4D.System.InvalidOperationException ("Nullable object must have a value.")
[13:46:39] Exception: E0434F4D.System.FormatException ("Input string was not in a correct format.")
[13:46:39] Exception: E0434F4D.System.FormatException ("Input string was not in a correct format.")
[13:46:39] Exception: E0434F4D.System.InvalidCastException ("Conversion from string "Nullable object must have a valu" to type 'Decimal' is not valid.")

 

These exceptions were unexpected and puzzling. Additional examination was required.

 

I attached WinDbg (x64) to ReportingServicesService.exe and configured event filters to break on CLR exceptions.

 

Refreshed the report again and the first exception was thrown.

Verified the exception with:

!PrintException

The result was:

Exception object: 000000030d4c2f90
Exception type: System.InvalidOperationException
Message: Nullable object must have a value.
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131509

 

Checked the .NET call stack with:

!clrstack

Part of the result was:

Child-SP         RetAddr          Call Site
0000000016d1b150 00007ff98347a650 System.ThrowHelper.ThrowInvalidOperationException(System.ExceptionResource)
0000000016d1b190 00007ff9271033ef System.Nullable`1[[System.Double, mscorlib]].op_Explicit(System.Nullable`1<Double>)
0000000016d1b1c0 00007ff927103310 ReportExprHostImpl+CustomCodeProxy.UnitConvertSlim(Double, System.String, System.String, System.String, System.Nullable`1<Double>, System.Nullable`1<Double>)
0000000016d1b290 00007ff92622c452 ReportExprHostImpl+Textbox9_TextBoxExprHost+Paragraph00_ParagraphExprHost+TextRun00_TextRunExprHost.get_ValueExpr()
0000000016d1b360 00007ff92622bedf Microsoft.ReportingServices.RdlExpressions.ReportRuntime.EvaluateTextRunValueExpression(Microsoft.ReportingServices.ReportIntermediateFormat.TextRun)

 

Also checked the .NET call stack including parameters and locals with:

!clrstack -a

However the majority of these were optimized away, only showing: <no data>

 

It’s relevant to mention that the report code in question contains multiple UnitConvert(…) functions with different parameter overloads.

 

At this point I made a mistake by reading UnitConvertSlim as UnitConvert. I mistakenly thought that the right UnitConvert(…) function was called.

 

I wanted to set a breakpoint to inspect the status before the exception occured, but this was complicated by the fact that ReportExprHostImpl+CustomCodeProxy is generated code.

(Managed to set breakpoints, but they were not hit, most likely because they were on previous versions of the generated code)

 

Being challenged by this, I decided to do something else.

I compared the old and new version of the report and realised that UnitConvertSlim(…) was another function than expected.

 

Therefore I decided to run the VB code in a simple, separate VB application, to enable normal debugging.

This required simulating certain SSRS classes and properties, in this case Report.Parameters(…).

 

By debugging the separate VB application I realized that the expected UnitConvert(…) function did not exist!

Instead another UnitConvert(…) function was found and called. Some parameters were converted to other types and this was the root cause of the problem…

 

I implemented the missing UnitConvert(…) function and this solved the problem, both in the separate VB application and in SSRS.

Conclusion

SSRS VB code may fail in unexpected ways, if a required function overload does not exist.

If debugging VB code in SSRS is problematic, try debugging it outside of SSRS in a separate program.

Blank SSRS report caused by ScriptManager timeout

I was testing a fairly complex and long running report for SQL Server Reporting Services, but the results were blank when using the Report Viewer component.

 

Status in the ExecutionLog* views was rsSuccess in the ReportServer database.

 

The ReportServerService*.log showed no errors, only this:

library!ReportServer_0-1!a78!09/23/2016-11:19:31:: i INFO: RenderForNewSession('/Test report')
webserver!ReportServer_0-1!a78!09/23/2016-11:21:59:: i INFO: Processed report. Report='/Test report', Stream=''

 

I used procdump to log exception messages from the IIS worker process with:

procdump.exe -f "" -l -e 1 w3wp.exe

Part of the result was:

[11:21:01] Exception: E0434F4D.System.Net.WebException ("The request was aborted: The request was canceled.")
[11:21:01] Exception: E0434F4D.System.Net.WebException ("The request was aborted: The request was canceled.")
[11:21:01] Exception: E0434F4D.Microsoft.Reporting.WebForms.ReportServerException ("The operation was canceled.")
[11:21:01] Exception: E0434F4D.Microsoft.Reporting.WebForms.ReportServerException ("The operation was canceled.")

 

So the request was actively getting cancelled.

This definitely seemed like a timeout issue, but which timeout setting was causing the problem?

 

Checked the default value for ReportViewer.ServerReport.Timeout, which was 600 seconds.

 

Then I googled for: ReportServerException (“The operation was canceled.”)

And found a solution here: Report Viewer Control returns OperationCanceledException OnError event after 90 seconds

 

The solution was to modify the .aspx page with the ReportViewer control:

From:

<asp:ScriptManager ID="ScriptManager1" runat="server">

To:

<asp:ScriptManager ID="ScriptManager1" runat="server" AsyncPostBackTimeout="600">

Running 32-bit or 64-bit IIS Express

IIS Express runs x86 / 32-bit as standard, but there is also a 64-bit version.

 

The version used for all web projects can be configured in Visual Studio 2013 or 2015 in the Tools -> Options… dialog.

Under: Project and Solutions -> Web Projects

The setting is:  Use the 64 bit version of IIS Express for web sites and projects

visual_studio_options_web_projects

 

I encountered this while trying to debug some startup and initialization issues with IIS Express and Visual Studio.

The result from:

Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles)

Was the path: C:\Program Files (x86)

But I was expecting the path: C:\Program Files

This was a clear sign that IIS Express was running as x86 / 32-bit.

 

Found the Visual Studio setting here:

https://visualstudio.uservoice.com/forums/121579-visual-studio-2015/suggestions/3254745-allow-for-iis-express-64-bit-to-run-from-visual-st

Other possible solutions are to modify the registry or calling the desired version of IIS Express with a script.

Debugging blank SSRS report caused by invalid XML characters

I was testing a modified report for SQL Server Reporting Services, but the results were blank and no relevant error messages were logged by SSRS.

Only messages like these that didn’t reveal the root cause:

ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerHttpRuntimeInternalException: RsWorkerRequest::FlushResponse., Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerHttpRuntimeInternalException: An internal or system error occurred in the HTTP Runtime object for application domain ReportServer_MSSQLSERVER_0-1-131184109058523378.  ---> System.Runtime.InteropServices.COMException (0x800703E3): The I/O operation has been aborted because of either a thread exit or an application request. (Exception from HRESULT: 0x800703E3)

 

Started troubleshooting by comparing the modified report with the original version, but I could see no obvious problems in either version.

 

When testing the report through the Report Viewer component resulted in this exception:

[XmlException: '♫', hexadecimal value 0x0E, is an invalid character. Line 1, position 6512.]
   System.Xml.XmlTextReaderImpl.Throw(Exception e) +88
   System.Xml.XmlTextReaderImpl.ParseNumericCharRefInline(Int32 startPos, Boolean expand, StringBuilder internalSubsetBuilder, Int32& charCount, EntityType& entityType) +817
   System.Xml.XmlTextReaderImpl.ParseCharRefInline(Int32 startPos, Int32& charCount, EntityType& entityType) +98
   System.Xml.XmlTextReaderImpl.ParseText(Int32& startPos, Int32& endPos, Int32& outOrChars) +816
   System.Xml.XmlTextReaderImpl.ParseText() +126
   System.Xml.XmlTextReaderImpl.ParseElementContent() +438
   System.Xml.XmlReader.ReadElementString() +84
   Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderServerReportSoapProxy.Read5_ValidValue(Boolean isNullable, Boolean checkType) +489

 

I wanted to verify that it was the same problem in pure SSRS, so I used ProcDump to log exception messages with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

 

The result was:

[17:47:58] Exception: E0434F4D.System.Xml.XmlException ("'♫', hexadecimal value 0x0E, is an invalid character. Line 1, position 6508.")
[17:47:58] Exception: E0434F4D.System.InvalidOperationException ("There is an error in XML document (1, 6508).")
[17:47:58] Exception: E0434F4D.MissingEndpointException ("The attempt to connect to the report server failed. Check your connection information and that the report server is a compatible version.")
[17:47:58] Exception: E0434F4D.System.Runtime.InteropServices.COMException ("The I/O operation has been aborted because of either a thread exit or an application request. (Exception from HRESULT: 0x800703E3)")
[17:47:58] Exception: E0434F4D.System.Runtime.InteropServices.COMException ("The I/O operation has been aborted because of either a thread exit or an application request. (Exception from HRESULT: 0x800703E3)")
[17:47:58] Exception: E0434F4D.Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerHttpRuntimeInternalException ("An internal or system error occurred in the HTTP Runtime object for application domain ReportServer_MSSQLSERVER_0-1-131184109058523378. ")

 

That confirmed that the problem was caused by invalid XML characters somewhere.

 

I wanted to see the XML that contained the invalid characters, so I attached WinDbg to ReportingServicesService.exe and enabled the CLR Exception event filter.

 

Refreshed the report to reproduce the problem.

Tried looking at the .NET call stack including parameters and locals with:

!clrstack -a

First part of the stack was:

OS Thread Id: 0x1d84 (26)
        Child SP               IP Call Site
00000011d267d1d8 00007ffef2638a5c [HelperMethodFrame: 00000011d267d1d8] 
00000011d267d2c0 00007ffedc73e488 System.Xml.XmlTextReaderImpl.Throw(System.Exception)
    PARAMETERS:
        this = <no data>
        e = <no data>
    LOCALS:
        <no data>

00000011d267d300 00007ffedc3cfd11 System.Xml.XmlTextReaderImpl.ParseNumericCharRefInline(Int32, Boolean, System.Text.StringBuilder, Int32 ByRef, EntityType ByRef)
    PARAMETERS:
        this (0x00000011d267d380) = 0x0000000fae5d0c00
        startPos = <no data>
        expand (0x00000011d267d390) = 0x0000000000000001
        internalSubsetBuilder (0x00000011d267d398) = 0x0000000000000000
        charCount (0x00000011d267d3a0) = 0x00000011d267d3f0
        entityType = <no data>
    LOCALS:
        <no data>
        0x00000011d267d340 = 0x0000000000000a6b
        0x00000011d267d330 = 0x0000000fae5d1328
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <CLR reg> = 0x000000000000000e
        <no data>
        <no data>

 

I examined the local object 0x0000000fae5d1328 with:

!DumpObj /d 0000000fae5d1328

Which seemed to be used as a buffer for XML data:

Name:        System.Char[]
MethodTable: 00007ffee08debd0
EEClass:     00007ffee02f10d0
Size:        8218(0x201a) bytes
Array:       Rank 1, Number of elements 4097, Type Char (Print Array)
Content:     MultiValue>true</MultiValue><QueryParameter>true</QueryParameter><Prompt>Device Type</Prompt><PromptUser>true</PromptUser><Depen
Fields:
None

 

I decided to find the adresses involved by dumping the array with:

!DumpArray /d 0000000fae5d1328

Part of the result was:

Name:        System.Char[]
MethodTable: 00007ffee08debd0
EEClass:     00007ffee02f10d0
Size:        8218(0x201a) bytes
Array:       Rank 1, Number of elements 4097, Type Char
Element Methodtable: 00007ffee08dec38
[0] 0000000fae5d1338
[1] 0000000fae5d133a
...
...
[4095] 0000000fae5d3336
[4096] 0000000fae5d3338

 

Now knowing the address range the unicode characters in the buffer could be read with:

du 0000000fae5d1338 0000000fae5d3338

Part of the result was:

0000000f`ae5d1338  "MultiValue>true</MultiValue><Que"
0000000f`ae5d1378  "ryParameter>true</QueryParameter"
...
...
0000000f`ae5d27b8  "/Value></ValidValue><ValidValue>"
0000000f`ae5d27f8  "<Label>♫♫113897</Label><"
0000000f`ae5d2838  "Value>52b3bc3a-32a8-49fc-a658-66"
0000000f`ae5d2878  "c67d3b29f6</Value></ValidValue><"
...
...

 

Checked the XML for invalid characters and eventually found some.

In a more readable form:

<ValidValue>
  <Label>♫♫113897</Label>
  <Value>52b3bc3a-32a8-49fc-a658-66c67d3b29f6</Value>
</ValidValue>

 

I thought it would be interesting to see the last query before the exception occured, so I started a trace with SQL Server Profiler and found:

SELECT cast(cast(0 as binary) as uniqueidentifier) AS Id, 'UNKNOWN' AS Username
UNION
SELECT Id, Username FROM Users order by 2

 

Tried running the query in SQL Server Management Studio and part of the result was:

query_result_usernames_with_invalid_xml_characters

 

Certain usernames actually contained characters that are invalid in XML.

This was the underlying cause behind the problem.

Conclusion

When SSRS reports fail without a clear indication of the cause, I recommend using a variety of tools and techniques to debug the problem similar to the described approach.

Using ProcDump itself to debug programs

ProcDump is an excellent tool for capturing memory dumps.

It actually attaches to processes as a debugger and it can also be used to debug certain problems without using other tools.

 

This can be useful when debugging problems on systems that are currently in use.

(Maybe policies prevent installing and attaching a traditional debugger or live debugging will simply cause too much disruption)

 

ProcDump can be used to output exceptions and debug messages from a process with:

procdump.exe -f "" -l -e 1 Process_name_or_id

The options used are:

-f “” Exception filter with empty filter makes ProcDump output exception messages without creating memory dumps.
-l Shows debug messages from process.
-e 1 Monitors both handled and unhandled exceptions.

 

Of course ProcDump can be used normally to generate memory dumps, which can be examined on another system with a debugger.

Be aware that processes are paused when ProcDump takes a memory dump and also be aware of the disk usage, especially when taking full memory dumps.

 

Recently using this technique helped me debug a problem on a fresh installation of SQL Server Reporting Services 2012.

All reports were returning blank results and no error messages were shown or logged.

 

I started monitoring the ReportingServicesService.exe process with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

 

After refreshing the report the output from ProcDump was:

ProcDump v8.0 - Writes process dump files
Copyright (C) 2009-2016 Mark Russinovich
Sysinternals - www.sysinternals.com
With contributions from Andrew Richards

Process:               ReportingServicesService.exe (11180)
CPU threshold:         n/a
Performance counter:   n/a
Commit threshold:      n/a
Threshold seconds:     n/a
Hung window check:     Disabled
Log debug strings:     Enabled
Exception monitor:     First Chance+Unhandled
Exception filter:      Display Only
Terminate monitor:     Disabled
Cloning type:          Disabled
Concurrent limit:      n/a
Avoid outage:          n/a
Number of dumps:       1
Dump folder:           C:\bin\Procdump\
Dump filename/mask:    PROCESSNAME_YYMMDD_HHMMSS


Press Ctrl-C to end monitoring without terminating the process.

CLR Version: v2.0.50727

[18:22:24] Exception: E0434F4D.System.IndexOutOfRangeException ("QC")
[18:22:26] Exception: E0434F4D.System.Net.Sockets.SocketException ("No such host is known")
[18:22:26] Exception: E0434F4D.System.Net.WebException ("The remote name could not be resolved: 'reportingutility'")
[18:22:26] Exception: E0434F4D.System.Net.WebException ("The remote name could not be resolved: 'reportingutility'")
[18:22:26] Exception: E0434F4D.System.Net.WebException ("The remote name could not be resolved: 'reportingutility'")
[18:22:26] Exception: E0434F4D.System.Net.WebException ("The remote name could not be resolved: 'reportingutility'")

 

The report was trying to call a web service from VB code, however the host name was unknown.

The problem was resolved by specifying the host name in the system hosts file.

Conclusion

ProcDump can also be used on its own to debug programs.

Laptop computer freezes when power supply is connected

I recently experienced a problem on a Lenovo Thinkpad T440p computer running Windows 10:

If the power supply was connected when the computer was running it would seemingly freeze: Mouse and keyboard became non-responsive.

However it was not a full freeze or crash, because music from a mediaplayer would continue.

If the power supply was disconnected, then the computer became responsive again.

 

One way to avoid the problem was to connect the power when the computer was sleeping or turned off.

 

It was an annoying problem and some people would assume the computer had crashed.

So I searched for a solution and found this:

https://www.ifixit.com/Answers/View/70872/How+to+fix+a+notebook+that+freezes+when+plugged+to+AC+power

 

The solution that worked for me was to:

  1. Open Device Manager
  2. Find Batteries -> Microsoft AC Adapter
  3. Right click and disable the Microsoft AC Adapter

2016-09-12_laptop_freezes_when_connected_to_power