Wireshark can hang when stopping capture

I have experienced problems with recent Wireshark versions on Windows, including the current latest version 2.2.3.

 

The way I normally use Wireshark is to capture the traffic of interest, then stop the capture and finally analyze the traffic.

However Wireshark on Windows can hang when stopping the capture…

 

When it happens the Wireshark UI becomes non-responsive:

 

The main thread runs at 100% CPU utilization.

 

And Process Monitor shows that the Wireshark process reads the “End Of File” from the same temporary file over and over:

 

Started debugging the issue by taking multiple memory dumps of the wireshark.exe process using procdump with:

procdump -ma -n 10 -s 1 wireshark.exe

 

Opened the first memory dump in WinDbg (x64).

Checked the call stack for the main thread (0) with:

~0 kp

Result:

# Child-SP          RetAddr           Call Site
00 00000039`a00f9fc8 00007ffc`e5dbc264 ntdll!NtReadFile+0x14
01 00000039`a00f9fd0 00007ffc`d9ca9aa7 KERNELBASE!ReadFile+0x74
02 00000039`a00fa050 00007ffc`d9ca9782 msvcr120!_read_nolock(int fh = 0n8, void * inputbuf = 0x000001a8`3e57bd20, unsigned int cnt = 0x1000)+0x2cf [f:\dd\vctools\crt\crtw32\lowio\read.c @ 256]
03 00000039`a00fa0f0 00007ffc`a14f09b3 msvcr120!_read(int fh = 0n8, void * buf = 0x000001a8`3e57bd20, unsigned int cnt = 0x1000)+0xc6 [f:\dd\vctools\crt\crtw32\lowio\read.c @ 92]
04 00000039`a00fa140 00007ffc`a14efde4 libwiretap!file_tell+0xc93
05 00000039`a00fa170 00007ffc`a14ef98c libwiretap!file_tell+0xc4
06 00000039`a00fa1a0 00007ffc`a1517c8d libwiretap!file_read+0xac
07 00000039`a00fa1e0 00007ffc`a150abb3 libwiretap!wtap_read_bytes_or_eof+0x2d
08 00000039`a00fa210 00007ffc`a150a9a9 libwiretap!wtap_wtap_encap_to_pcap_encap+0xbd3
09 00000039`a00fa290 00007ffc`a1517b77 libwiretap!wtap_wtap_encap_to_pcap_encap+0x9c9
0a 00000039`a00fa310 00007ff7`c7929fd3 libwiretap!wtap_read+0x37
0b 00000039`a00fa350 00007ff7`c7b7cfd7 Wireshark+0x9fd3
0c 00000039`a00fa3a0 00007ff7`c7b963dd Wireshark+0x25cfd7
0d 00000039`a00fa3e0 00007ff7`c798889a Wireshark+0x2763dd
0e 00000039`a00fb470 00007ff7`c79d5144 Wireshark+0x6889a
0f 00000039`a00fb4c0 00000000`5b94f906 Wireshark+0xb5144
10 00000039`a00fb5c0 00000000`5b9c4d66 Qt5Core!QMetaObject::activate+0x5a6
11 00000039`a00fb6d0 00000000`5b95413a Qt5Core!QTimer::timeout+0x16
12 00000039`a00fb700 00000000`5bcf7d12 Qt5Core!QObject::event+0x6a
13 00000039`a00fb8b0 00000000`5bcf6c2f Qt5Widgets!QApplicationPrivate::notify_helper+0x112
14 00000039`a00fb8e0 00000000`5b92f689 Qt5Widgets!QApplication::notify+0x1b3f
15 00000039`a00fc000 00000000`5b977a8c Qt5Core!QCoreApplication::notifyInternal2+0xb9
16 00000039`a00fc080 00000000`5b976a32 Qt5Core!QEventDispatcherWin32Private::sendTimerEvent+0x10c
17 00000039`a00fc0e0 00007ffc`e6851c24 Qt5Core!QEventDispatcherWin32::processEvents+0xd82
18 00000039`a00fc1f0 00007ffc`e685156c user32!UserCallWinProcCheckWow+0x274
19 00000039`a00fc350 00000000`5b9761d9 user32!DispatchMessageWorker+0x1ac
1a 00000039`a00fc3d0 00007ffc`a14029b9 Qt5Core!QEventDispatcherWin32::processEvents+0x529
1b 00000039`a00ff760 00000000`5b92bf91 qwindows!qt_plugin_query_metadata+0x2499
1c 00000039`a00ff790 00000000`5b92e477 Qt5Core!QEventLoop::exec+0x1b1
1d 00000039`a00ff810 00007ff7`c7929ccd Qt5Core!QCoreApplication::exec+0x147
1e 00000039`a00ff880 00007ff7`c7ba2ac5 Wireshark+0x9ccd
1f 00000039`a00ffd50 00007ff7`c7ba22fd Wireshark+0x282ac5
20 00000039`a00ffde0 00007ffc`e87c8364 Wireshark+0x2822fd
21 00000039`a00ffe20 00007ffc`e8f470d1 kernel32!BaseThreadInitThunk+0x14
22 00000039`a00ffe50 00000000`00000000 ntdll!RtlUserThreadStart+0x21

 

Checked the main thread (0) call stack for all the memory dumps by scripting CDB, the console version of WinDbg.

Used this PowerShell script:

$dmpPath = 'C:\Bin\Procdump\Wireshark\'
$dmpFiles = Get-ChildItem -Path $dmpPath -Recurse -Include *.dmp

foreach ($dmpFile in $dmpFiles)
{
    & "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -z $dmpFile -c "~0 kp; Q"
}

 

Noticed that all the memory dumps had the same call stack for thread 0, which further indicated that WireShark was stuck or running in an endless loop.

 

Now I wanted to identify if this was a hang or endless loop.

Attached to the running, but non-responsive Wireshark process with WinDbg (x64).

Experimented by setting breakpoints from the initial call stack.

 

Set a single breakpoint at:

MSVCR120!_read_nolock(...)+0x2cf

With:

bc *
bp MSVCR120!_read_nolock+0x2cf

Result after continue: Breakpoint was continually hit.

 

This definitely indicated an endless loop.

 

Set a single breakpoint at:

libwiretap!wtap_read_bytes_or_eof+0x2d

With:

bc *
bp libwiretap!wtap_read_bytes_or_eof+0x2d

Result after continue: Breakpoint was continually hit.

 

Set a single breakpoint at:

Qt5Core!QTimer::timeout+0x16

With:

bc *
bp Qt5Core!QTimer::timeout+0x16

Result after continue: Breakpoint was not hit (within 1 minute)

 

Set single breakpoints at the various Wireshark functions.

The breakpoint was not hit until setting a single breakpoint at:

Wireshark+0x9fd3

With:

bc *
bp Wireshark+0x9fd3

Result after continue: The breakpoint was continually hit.

 

This indicated that the endless loop occured in the Wireshark module.

 

Had a quick look at the disassembly and tried single stepping with:

p

The code was definitely looping, but the problem was not obvious by looking at the machine code.

 

Went looking for debug symbols and found them here:

https://1.eu.dl.wireshark.org/win64/all-versions/Wireshark-pdb-win64-2.2.3.zip

Downloaded the debug symbols and unpacked them to a temporary folder.

 

Then modified the symbol path in WinDbg with:

.sympath C:\Temp\WSSymbols\;srv*c:\SymbolsCache*https://msdl.microsoft.com/download/symbols

And reloaded the symbols with:

.reload /f

 

Checked the call stack again with:

~0 kp

Result:

# Child-SP          RetAddr           Call Site
00 00000039`a00f9fc8 00007ffc`e5dbc264 ntdll!NtReadFile+0x14
01 00000039`a00f9fd0 00007ffc`d9ca9aa7 KERNELBASE!ReadFile+0x74
02 00000039`a00fa050 00007ffc`d9ca9782 MSVCR120!_read_nolock(int fh = 0n8, void * inputbuf = 0x000001a8`3e57bd20, unsigned int cnt = 0x1000)+0x2cf [f:\dd\vctools\crt\crtw32\lowio\read.c @ 256]
03 00000039`a00fa0f0 00007ffc`a14f09b3 MSVCR120!_read(int fh = 0n8, void * buf = 0x000001a8`3e57bd20, unsigned int cnt = 0x1000)+0xc6 [f:\dd\vctools\crt\crtw32\lowio\read.c @ 92]
04 00000039`a00fa140 00007ffc`a14efde4 libwiretap!raw_read(struct wtap_reader * state = 0x000001a8`4ad0ebd0, unsigned char * buf = 0x000001a8`3e57bd20 "vided by dumpcap???", unsigned int count = 0x1000, unsigned int * have = 0x000001a8`4ad0ec08)+0x43 [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\wiretap\file_wrappers.c @ 133]
05 00000039`a00fa170 00007ffc`a14ef98c libwiretap!fill_out_buffer(struct wtap_reader * state = 0x000001a8`4ad0ebd0)+0x44 [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\wiretap\file_wrappers.c @ 704]
06 00000039`a00fa1a0 00007ffc`a1517c8d libwiretap!file_read(void * buf = 0x00000039`a00fa250, unsigned int len = 8, struct wtap_reader * file = 0x000001a8`4ad0ebd0)+0xac [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\wiretap\file_wrappers.c @ 1237]
07 00000039`a00fa1e0 00007ffc`a150abb3 libwiretap!wtap_read_bytes_or_eof(struct wtap_reader * fh = 0x000001a8`4ad0ebd0, void * buf = <Value unavailable error>, unsigned int count = 8, int * err = 0x00000039`a00fa3a0, char ** err_info = 0x00000039`a00fa3b0)+0x2d [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\wiretap\wtap.c @ 1291]
08 00000039`a00fa210 00007ffc`a150a9a9 libwiretap!pcapng_read_block(struct wtap * wth = 0x000001a8`41f48790, struct wtap_reader * fh = 0x000001a8`4ad0ebd0, struct pcapng_t * pn = 0x000001a8`41f1ca40, struct wtapng_block_s * wblock = 0x00000039`a00fa2c0, int * err = 0x00000039`a00fa3a0, char ** err_info = 0x00000039`a00fa3b0)+0x53 [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\wiretap\pcapng.c @ 2303]
09 00000039`a00fa290 00007ffc`a1517b77 libwiretap!pcapng_read(struct wtap * wth = 0x000001a8`41f48790, int * err = 0x00000039`a00fa3a0, char ** err_info = 0x00000039`a00fa3b0, int64 * data_offset = 0x00000039`a00fa3a8)+0x79 [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\wiretap\pcapng.c @ 2589]
0a 00000039`a00fa310 00007ff7`c7929fd3 libwiretap!wtap_read(struct wtap * wth = 0x000001a8`41f48790, int * err = 0x00000039`a00fa3a0, char ** err_info = 0x00000039`a00fa3b0, int64 * data_offset = <Value unavailable error>)+0x37 [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\wiretap\wtap.c @ 1237]
0b 00000039`a00fa350 00007ff7`c7b7cfd7 Wireshark!capture_info_new_packets(int to_read = 0n14, struct _info_data * cap_info = 0x000001a8`3f1838b8)+0x43 [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\capture_info.c @ 211]
0c 00000039`a00fa3a0 00007ff7`c7b963dd Wireshark!capture_input_new_packets(struct _capture_session * cap_session = 0x000001a8`3f183878, int to_read = 0n23)+0xa7 [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\ui\capture.c @ 407]
0d 00000039`a00fa3e0 00007ff7`c798889a Wireshark!sync_pipe_input_cb(int source = 0n4, void * user_data = 0x000001a8`3f183878)+0x1bd [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\capchild\capture_sync.c @ 1775]
0e 00000039`a00fb470 00007ff7`c79d5144 Wireshark!MainWindow::pipeTimeout(void)+0x8a [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\ui\qt\main_window_slots.cpp @ 949]
0f 00000039`a00fb4c0 00000000`5b94f906 Wireshark!MainWindow::qt_static_metacall(class QObject * _o = 0x000001a8`3f1836e0, QMetaObject::Call _c = <Value unavailable error>, int _id = <Value unavailable error>, void ** _a = 0x00000039`a00fb5f8)+0x4c4 [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\cmbuild\ui\qt\moc_main_window.cpp @ 1436]
10 00000039`a00fb5c0 00000000`5b9c4d66 Qt5Core!QMetaObject::activate+0x5a6
11 00000039`a00fb6d0 00000000`5b95413a Qt5Core!QTimer::timeout+0x16
12 00000039`a00fb700 00000000`5bcf7d12 Qt5Core!QObject::event+0x6a
13 00000039`a00fb8b0 00000000`5bcf6c2f Qt5Widgets!QApplicationPrivate::notify_helper+0x112
14 00000039`a00fb8e0 00000000`5b92f689 Qt5Widgets!QApplication::notify+0x1b3f
15 00000039`a00fc000 00000000`5b977a8c Qt5Core!QCoreApplication::notifyInternal2+0xb9
16 00000039`a00fc080 00000000`5b976a32 Qt5Core!QEventDispatcherWin32Private::sendTimerEvent+0x10c
17 00000039`a00fc0e0 00007ffc`e6851c24 Qt5Core!QEventDispatcherWin32::processEvents+0xd82
18 00000039`a00fc1f0 00007ffc`e685156c USER32!UserCallWinProcCheckWow+0x274
19 00000039`a00fc350 00000000`5b9761d9 USER32!DispatchMessageWorker+0x1ac
1a 00000039`a00fc3d0 00007ffc`a14029b9 Qt5Core!QEventDispatcherWin32::processEvents+0x529
1b 00000039`a00ff760 00000000`5b92bf91 qwindows!qt_plugin_query_metadata+0x2499
1c 00000039`a00ff790 00000000`5b92e477 Qt5Core!QEventLoop::exec+0x1b1
1d 00000039`a00ff810 00007ff7`c7929ccd Qt5Core!QCoreApplication::exec+0x147
1e 00000039`a00ff880 00007ff7`c7ba2ac5 Wireshark!main(int argc = 0n1, char ** qt_argv = <Value unavailable error>)+0xe3d [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\wireshark-qt.cpp @ 853]
1f 00000039`a00ffd50 00007ff7`c7ba22fd Wireshark!WinMain+0x155
20 00000039`a00ffde0 00007ffc`e87c8364 Wireshark!__tmainCRTStartup(void)+0x149 [f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 618]
21 00000039`a00ffe20 00007ffc`e8f470d1 KERNEL32!BaseThreadInitThunk+0x14
22 00000039`a00ffe50 00000000`00000000 ntdll!RtlUserThreadStart+0x21

 

The symbols seemed to be correct and functional.

 

Wanted to verify the location of the endless loop by setting breakpoints again.

 

Set a single breakpoint at:

Wireshark!capture_input_new_packets

With:

bc *
bp Wireshark!capture_input_new_packets

Result after continue: Breakpoint was not hit (within 1 minute)

 

Set a single breakpoint at:

Wireshark!capture_info_new_packets

With:

bc *
bp Wireshark!capture_info_new_packets

Result after continue: Breakpoint was not hit (within 1 minute)

 

Set a single breakpoint at:

libwiretap!wtap_read

With:

bc *
bp libwiretap!wtap_read

Result after continue: The breakpoint was continually hit.

 

This result indicated that the endless loop was in Wireshark!capture_info_new_packets(…) calling libwiretap!wtap_read(…), specifically here:

Wireshark!capture_info_new_packets(int to_read = 0n14, struct _info_data * cap_info = 0x000001a8`3f1838b8)+0x43 [c:\buildbot\wireshark\wireshark-2.2-64\windows-2012r2-x64\build\capture_info.c @ 211]

 

Downloaded the source code for Wireshark from:

https://1.eu.dl.wireshark.org/src/wireshark-2.2.3.tar.bz2

 

Looked for and opened: capture_info.c

Found the method, which has been included here:

/* new packets arrived */
void capture_info_new_packets(int to_read, info_data_t* cap_info)
{
    int err;
    gchar *err_info;
    gint64 data_offset;
    struct wtap_pkthdr *phdr;
    union wtap_pseudo_header *pseudo_header;
    int wtap_linktype;
    const guchar *buf;

    cap_info->ui.new_packets = to_read;

    /*g_warning("new packets: %u", to_read);*/

    while (to_read > 0) {
        wtap_cleareof(cap_info->wtap);
        if (wtap_read(cap_info->wtap, &err, &err_info, &data_offset)) {
            phdr = wtap_phdr(cap_info->wtap);
            pseudo_header = &phdr->pseudo_header;
            wtap_linktype = phdr->pkt_encap;
            buf = wtap_buf_ptr(cap_info->wtap);

            capture_info_packet(cap_info, wtap_linktype, buf, phdr->caplen, pseudo_header);

            /*g_warning("new packet");*/
            to_read--;
        }
    }

    capture_info_ui_update(&cap_info->ui);
}

 

I have marked some points of interest with bold.

 

The while loop continues as long as to_read is greater than 0.

Noticed that the to_read variable continued to have the value 14.

to_read is only decremented in case wtap_read returns a non-zero value.

It seems that this never happens, when the endless loop occurs in Wireshark.

 

I have noticed that the problem apparently doesn’t occur with Wireshark Legacy based on GTK (at least with default settings).

I wanted to determine the cause for the difference.

 

Examined symbols for Wireshark (QT) with:

x /D /f Wireshark!capture_info_new_packets

Result:

00007ff7`c7929f90 Wireshark!capture_info_new_packets (int, struct _info_data *)

 

Examined symbols for Wireshark Legacy (GTK) with:

x /D /f Wireshark_gtk!capture_info_new_packets

Result:

00007ff7`c03310f0 Wireshark_gtk!capture_info_new_packets (int, struct _info_data *)

 

Both versions include the function capture_info_new_packets (…)

 

Checked where the function was mentioned in the source code with grep (from Cygwin):

grep "capture_info_new_packets" -r

Result:

capture_info.c:void capture_info_new_packets(int to_read, info_data_t* cap_info)
capture_info.h:extern void capture_info_new_packets(int to_read, info_data_t* cap_info);
ui/capture.c:    capture_info_new_packets(to_read, cap_session->cap_data_info);

 

Checked the content of ui/capture.c and found:

if(capture_opts->show_info)
    capture_info_new_packets(to_read, cap_session->cap_data_info);

 

So the capture_info_new_packets(…) function will only be called if the show_info option is true…

 

Looked for places where show_info was used with:

grep "show_info" -r

Part of the result was:

ui/gtk/capture_dlg.c:  gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(hide_info_cb), !global_capture_opts.show_info);
ui/gtk/capture_dlg.c:  global_capture_opts.show_info =
ui/qt/capture_interfaces_dialog.cpp:    global_capture_opts.show_info = checked;
ui/qt/capture_interfaces_dialog.cpp:    ui->cbExtraCaptureInfo->setChecked(global_capture_opts.show_info);

 

For Wireshark Legacy (GTK) I checked the content of ui/gtk/capture_dlg.c and found:

global_capture_opts.show_info =
    !gtk_toggle_button_get_active(GTK_TOGGLE_BUTTON(hide_info_cb));

 

For Wireshark Legacy (GTK) the endless loop problem can be avoided by enabling this option in the Capture Options dialog :

Hide capture info dialog   (default setting)

 

For Wireshark (QT) I checked the content of ui/qt/capture_interfaces_dialog.cpp and found:

void CaptureInterfacesDialog::on_cbExtraCaptureInfo_toggled(bool checked)
{
    global_capture_opts.show_info = checked;
}

 

For Wireshark (QT) the endless loop problem can be avoided by disabling this option in the Capture Interfaces dialog :

Show extra capture information dialog   (not default setting)

 

Conclusion

The endless loop problem occurs consistently depending on the identified settings, so the problem cause has likely been found along with a workaround.

Until the endless loop bug has been fixed in Wireshark, it is recommended to set the Wireshark options like described above.

Debugging failing SSRS report caused by read-only report parameter

I recently experienced problems when trying to view a report for SQL Server Reporting Services with ReportViewer for MVC.

 

The report worked in some cases, but failed when certain parameters were set.

Exception and stack trace:

[ReportServerException: The report parameter 'DeviceTypes' is read-only and cannot be modified. (rsReadOnlyReportParameter)]
Microsoft.Reporting.WebForms.ServerReportSoapProxy.OnSoapException(SoapException e) +155
Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.ProxyMethodInvocation.Execute(RSExecutionConnection connection, ProxyMethod`1 initialMethod, ProxyMethod`1 retryMethod) +1256
Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.RSExecutionConnection.SetExecutionParameters(ParameterValue[] Parameters, String ParameterLanguage) +370
Microsoft.Reporting.WebForms.SoapReportExecutionService.SetExecutionParameters(IEnumerable`1 parameters, String parameterLanguage) +530
Microsoft.Reporting.WebForms.ServerReport.SetParameters(IEnumerable`1 parameters) +896
Microsoft.Reporting.WebForms.Report.SetParameters(ReportParameter parameter) +136
ReportViewerForMvc.ReportExtensions.SetParameters(Report report, ReportParameterInfoCollection collection) +114
ReportViewerForMvc.ReportViewerExtensions.SetProperties(ServerReport serverReport, ServerReport properties) +84
ReportViewerForMvc.ReportViewerExtensions.SetProperties(ReportViewer reportViewer, ReportViewer properties) +117
ReportViewerForMvc.ReportViewerWebForm.BuildReportViewer() +92
System.Web.UI.Control.OnLoad(EventArgs e) +109
System.Web.UI.Control.LoadRecursive() +68
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +4498

 

Checked the ReportServerService__*.log and ExecutionLog* database views, but they contained no additional clues about the error.

 

I attached WinDbg (x64) to the IIS worker process (w3wp.exe) and set a breakpoint with:

!bpmd   ReportViewerForMvc.dll   ReportViewerForMvc.ReportExtensions.SetParameters

Messages from WinDbg:

Found 2 methods in module 00007ff9bde79bb0...
MethodDesc = 00007ff9c00a30d0
MethodDesc = 00007ff9c00a30e0
Setting breakpoint: bp 00007FF9C0EE5590 [ReportViewerForMvc.ReportExtensions.SetParameters(Microsoft.Reporting.WebForms.Report, Microsoft.Reporting.WebForms.ReportParameterInfo)]
Setting breakpoint: bp 00007FF9C0EE53E4 [ReportViewerForMvc.ReportExtensions.SetParameters(Microsoft.Reporting.WebForms.Report, Microsoft.Reporting.WebForms.ReportParameterInfoCollection)]
Adding pending breakpoints...

 

Continued exection, reloaded the report and the breakpoint was hit.

Checked the .NET call stack with:

!clrstack -a

Part of the result:

OS Thread Id: 0x273c (45)
Child SP               IP Call Site
00000007dba9d030 00007ff9c0ee53e4 ReportViewerForMvc.ReportExtensions.SetParameters(Microsoft.Reporting.WebForms.Report, Microsoft.Reporting.WebForms.ReportParameterInfoCollection)
PARAMETERS:
report (<CLR reg>) = 0x0000000390e93650
collection (<CLR reg>) = 0x0000000490886678

 

Examined the collection object with:

!DumpObj /d 0000000490886678

Result:

Name:        Microsoft.Reporting.WebForms.ReportParameterInfoCollection
MethodTable: 00007ff9bff1cc98
EEClass:     00007ff9bff08d40
Size:        32(0x20) bytes
File:        C:\Windows\assembly\GAC_MSIL\Microsoft.ReportViewer.WebForms\11.0.0.0__89845dcd8080cc91\Microsoft.ReportViewer.WebForms.dll
Fields:
MT    Field   Offset                 Type VT     Attr            Value Name
00007ffa1b23d408  40017fa        8 ...Canon, mscorlib]]  0 instance 00000004908823f0 list
00007ffa1b8de068  40017fb       10        System.Object  0 instance 0000000000000000 _syncRoot

 

Examined the list array with:

!DumpArray /d 00000004908823f0

Result:

Name:        Microsoft.Reporting.WebForms.ReportParameterInfo[]
MethodTable: 00007ff9c0fd9e60
EEClass:     00007ffa1b2f0f10
Size:        120(0x78) bytes
Array:       Rank 1, Number of elements 12, Type CLASS
Element Methodtable: 00007ff9bff1cbc0
[0] 00000004908826c8
[1] 00000004908829d8
[2] 0000000490882ce8
[3] 00000004908832e0
[4] 0000000490884490
[5] 00000004908847a8
[6] 0000000490884ac0
[7] 0000000490884e38
[8] 00000004908853f8
[9] 0000000490885818
[10] 0000000490886250
[11] 00000004908865c8

 

Examined details for the list array with:

!DumpArray /d -details 00000004908823f0

Part of the result:

[11] 00000004908865c8
Name:        Microsoft.Reporting.WebForms.ReportParameterInfo
MethodTable: 00007ff9bff1cbc0
EEClass:     00007ff9bff08cc8
Size:        104(0x68) bytes
File:        C:\Windows\assembly\GAC_MSIL\Microsoft.ReportViewer.WebForms\11.0.0.0__89845dcd8080cc91\Microsoft.ReportViewer.WebForms.dll
Fields:
MT    Field   Offset                 Type VT     Attr            Value Name
00007ffa1b8dda88  4000120        8            System.String      0     instance     0000000490881950     m_name
00007ff9bff1c678  4000121       50             System.Int32      1     instance                    4     m_dataType
00007ffa1b8cd6f8  4000122       58           System.Boolean      1     instance                    0     m_isNullable
00007ffa1b8cd6f8  4000123       59           System.Boolean      1     instance                    1     m_allowBlank
00007ffa1b8cd6f8  4000124       5a           System.Boolean      1     instance                    1     m_isMultiValue
00007ffa1b8cd6f8  4000125       5b           System.Boolean      1     instance                    1     m_isQueryParameter
00007ffa1b8dda88  4000126       10            System.String      0     instance     000000038e411420     m_prompt
00007ffa1b8cd6f8  4000127       5c           System.Boolean      1     instance                    0     m_promptUser
00007ffa1b8cd6f8  4000128       5d           System.Boolean      1     instance                    1     m_areDefaultValuesQueryBased
00007ffa1b8cd6f8  4000129       5e           System.Boolean      1     instance                    1     m_areValidValuesQueryBased
00007ffa1b8dda88  400012a       18            System.String      0     instance     0000000000000000     m_errorMessage
00007ff9c0fda240  400012b       20     ...Viewer.WebForms]]      0     instance     0000000490886320     m_validValues
00007ffa1b23d128  400012c       28     ...tring, mscorlib]]      0     instance     0000000490886658     m_currentValues
00007ff9bff1c7c8  400012d       54             System.Int32      1     instance                    0     m_state
00007ff9bff1cc98  400012e       30     ...terInfoCollection      0     instance     0000000490886fb8     m_dependencyCollection
00007ff9bff1cc98  400012f       38     ...terInfoCollection      0     instance     0000000000000000     m_dependentsCollection
00007ffa1b8c1480  4000130       40          System.String[]      0     instance     0000000490881bb8     m_dependencies
00007ff9c0fda5d0  4000131       48     ...Viewer.WebForms]]      0     instance     0000000490886630     m_dependentsCollectionConstruction
00007ffa1b8cd6f8  4000132       5f           System.Boolean      1     instance                    1     m_visible

 

Examined the name of the last report parameter with:

!DumpObj /d 0000000490881950

Part of the result:

Name:        System.String
MethodTable: 00007ffa1b8dda88
EEClass:     00007ffa1b236a08
Size:        48(0x30) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      DeviceTypes

So the last parameter was DeviceTypes, the one causing the problem.

 

Looked for a read-only property among the report parameters. Noticed that m_promptUser was 0 for the DeviceTypes parameter, while m_promptUser was 1 for all other parameters.

 

I decided to use procdump to log exception messages from the ReportingServicesService.exe process with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

 

Reloaded the report to recreate the error.

Part of the result was:

[07:10:37] Exception: E0434F4D.Microsoft.ReportingServices.Diagnostics.Utilities.ReadOnlyReportParameterException ("The report parameter 'DeviceTypes' is read-only and cannot be modified.")
[07:10:37] Exception: E0434F4D.Microsoft.ReportingServices.Diagnostics.Utilities.RSException ("The report parameter 'DeviceTypes' is read-only and cannot be modified.")

 

Then loaded all DLL files from SSRS into ILSpy from:

C:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\ReportServer\bin

 

Searched for the ReadOnlyReportParameterException class in ILSpy.

Found it and noticed that it was apparently only used by:

Microsoft.ReportingServices.ReportProcessing.ParameterInfoCollection.Combine(...)

 

Read the decompiled code for the above method and noticed this part:

if (checkReadOnly && !parameterInfo.PromptUser)
{
  ParameterInfoCollection.ThrowReadOnlyParameterException(parameterInfo.Name, isSharedDataSetParameter);
}

Apparently the value of parameterInfo.PromptUser determined the read-only status.

 

I decided to examine the XML for the RDL report and noticed that the DeviceTypes report parameter did not have a Prompt section.

I added this:

<Prompt>DeviceTypes</Prompt>

Then uploaded the new version of the report (by overwriting).

However this did not seem to have any immediate effect, the error still occured…

 

I spent some time examining various other ideas and internals of SSRS.

 

Then I got an idea: I decided to delete the problematic report and then recreate the modified version with a Prompt section.

This fixed the problem! The report now worked in all cases, no matter which parameters were set.

This behavior was consistent across 2 SSRS instances.

Conclusion

It seems that read-only status for a report parameter is derived from the presence of a <Prompt>Description</Prompt> section.

However this status is not updated if the report is overwritten, it has to be deleted and then recreated…

 

Due to the way ReportViewerForMvc.dll is implemented all report parameters are copied from the initial instance to another instance of ReportViewer.

This means that report parameters can’t be read-only, so <Prompt>Description</Prompt> should always be specified for all report parameters, when using ReportViewer for MVC.

Debugging blank SSRS report caused by SecurityException

I was testing reports for SQL Server Reporting Services on a new computer, but the results were blank and no relevant error messages were logged by SSRS.

 

Investigated by using procdump to log exception messages from the IIS worker process with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

Part of the result was:

[13:22:13] Exception: E0434F4D.System.Security.SecurityException ("Request for the permission of type 'System.Net.WebPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed.")
[13:22:13] Exception: E0434F4D.System.Security.SecurityException ("Request for the permission of type 'System.Security.Permissions.SecurityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed.")

 

I knew that the reports performed HTTP/web service calls from VB code, so I had to modify rssrvpolicy.config to allow this.

It can be done by:

1. Appending a named permission set with both execution and web permission like:

<PermissionSet class="NamedPermissionSet"
version="1"
Name="ExecutionAndWebPermission"
Description="A special permission set that grants execution and web permission for specific site(s).">
  <IPermission class="SecurityPermission" version="1" Flags="Execution" />
  <IPermission class="WebPermission" version="1">
    <ConnectAccess>
      <URI uri="http://localhost/Test/ReportingUtilityService.svc"/>
    </ConnectAccess>
  </IPermission>
</PermissionSet>

 

2. Modifying the CodeGroup with Name=”Report_Expressions_Default_Permissions”

From:

PermissionSetName="Execution"

To:

PermissionSetName="ExecutionAndWebPermission"

 

3. Restarting Reporting Services.

 

I decided to verify that this was the issue before performing the changes.

Therefore I attached WinDbg (x64) to ReportingServicesService.exe and configured event filters to break on CLR exceptions.

 

Refreshed the report again and the first exception was thrown.

Checked the exception with:

!PrintException

 

The result was a very detailed (and useful) message, that confirmed the suspected issue:

Exception object: 0000000460198410
Exception type: System.Security.SecurityException
Message: Request for the permission of type 'System.Net.WebPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed.
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 8013150a
SecurityException Message:
The action that failed was:
Demand
The type of the first permission that failed was:
System.Net.WebPermission
The first permission that failed was:
<IPermission class="System.Net.WebPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1">
<ConnectAccess>
<URI uri="http://localhost/Test/ReportingUtilityService\.svc"/>
</ConnectAccess>
</IPermission>

The demand was for:
<IPermission class="System.Net.WebPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1">
<ConnectAccess>
<URI uri="http://localhost/Test/ReportingUtilityService\.svc"/>
</ConnectAccess>
</IPermission>

The granted set of the failing assembly was:
<PermissionSet class="System.Security.PermissionSet"
version="1">
<IPermission class="System.Security.Permissions.SecurityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1"
Flags="Execution"/>
<IPermission class="System.Security.Permissions.StrongNameIdentityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1"
PublicKeyBlob="..."
Name="expression_host_..."
AssemblyVersion="1.0.0.0"/>
<IPermission class="System.Security.Permissions.ZoneIdentityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1"
Zone="MyComputer"/>
</PermissionSet>

The assembly or AppDomain that failed was:
expression_host_..., Version=12.2.6020.0, Culture=neutral, PublicKeyToken=null
The method that caused the failure was:
System.Object CallWebService(System.String, System.String)
The Zone of the assembly that failed was:
MyComputer

 

After modifying rssrvpolicy.config and restarting Reporting Services the issue was resolved.

Debugging #Error in SSRS report caused by attempts to convert non-numeric strings

While testing another modified SSRS report I noticed that certain values were not shown, only the message:

#Error

 

Checked the ReportServerService__*.log and ExecutionLog* database views, but they contained no clues about the error.

 

Then I used procdump to log exception messages from ReportingServicesService.exe with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

 

Part of the result was:

[13:15:59] Exception: E0434F4D.System.FormatException ("Input string was not in a correct format.")
[13:15:59] Exception: E0434F4D.System.FormatException ("Input string was not in a correct format.")
[13:15:59] Exception: E0434F4D.System.InvalidCastException ("Conversion from string " norm" to type 'Decimal' is not valid.")

 

I extracted and ran the dataset query in SQL Server Management Studio.

Then looked for “norm” strings and found them in a column normally used to store numbers.

Attempts to convert these caused the problem.

 

My solution was to exclude them from the dataset query by adding:

AND ISNUMERIC(ValueString) = 1

Debugging #Error in SSRS report caused by missing function overload

While testing a modified SSRS report I noticed that certain values were not shown, only the message:

#Error

 

Neither the ReportServerService__*.log or ExecutionLog* database views contained any clues about the cause.

 

I used procdump to log exception messages from ReportingServicesService.exe with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

Part of the result was:

[13:46:39] Exception: E0434F4D.System.InvalidOperationException ("Nullable object must have a value.")
[13:46:39] Exception: E0434F4D.System.FormatException ("Input string was not in a correct format.")
[13:46:39] Exception: E0434F4D.System.FormatException ("Input string was not in a correct format.")
[13:46:39] Exception: E0434F4D.System.InvalidCastException ("Conversion from string "Nullable object must have a valu" to type 'Decimal' is not valid.")

 

These exceptions were unexpected and puzzling. Additional examination was required.

 

I attached WinDbg (x64) to ReportingServicesService.exe and configured event filters to break on CLR exceptions.

 

Refreshed the report again and the first exception was thrown.

Verified the exception with:

!PrintException

The result was:

Exception object: 000000030d4c2f90
Exception type: System.InvalidOperationException
Message: Nullable object must have a value.
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131509

 

Checked the .NET call stack with:

!clrstack

Part of the result was:

Child-SP         RetAddr          Call Site
0000000016d1b150 00007ff98347a650 System.ThrowHelper.ThrowInvalidOperationException(System.ExceptionResource)
0000000016d1b190 00007ff9271033ef System.Nullable`1[[System.Double, mscorlib]].op_Explicit(System.Nullable`1<Double>)
0000000016d1b1c0 00007ff927103310 ReportExprHostImpl+CustomCodeProxy.UnitConvertSlim(Double, System.String, System.String, System.String, System.Nullable`1<Double>, System.Nullable`1<Double>)
0000000016d1b290 00007ff92622c452 ReportExprHostImpl+Textbox9_TextBoxExprHost+Paragraph00_ParagraphExprHost+TextRun00_TextRunExprHost.get_ValueExpr()
0000000016d1b360 00007ff92622bedf Microsoft.ReportingServices.RdlExpressions.ReportRuntime.EvaluateTextRunValueExpression(Microsoft.ReportingServices.ReportIntermediateFormat.TextRun)

 

Also checked the .NET call stack including parameters and locals with:

!clrstack -a

However the majority of these were optimized away, only showing: <no data>

 

It’s relevant to mention that the report code in question contains multiple UnitConvert(…) functions with different parameter overloads.

 

At this point I made a mistake by reading UnitConvertSlim as UnitConvert. I mistakenly thought that the right UnitConvert(…) function was called.

 

I wanted to set a breakpoint to inspect the status before the exception occured, but this was complicated by the fact that ReportExprHostImpl+CustomCodeProxy is generated code.

(Managed to set breakpoints, but they were not hit, most likely because they were on previous versions of the generated code)

 

Being challenged by this, I decided to do something else.

I compared the old and new version of the report and realised that UnitConvertSlim(…) was another function than expected.

 

Therefore I decided to run the VB code in a simple, separate VB application, to enable normal debugging.

This required simulating certain SSRS classes and properties, in this case Report.Parameters(…).

 

By debugging the separate VB application I realized that the expected UnitConvert(…) function did not exist!

Instead another UnitConvert(…) function was found and called. Some parameters were converted to other types and this was the root cause of the problem…

 

I implemented the missing UnitConvert(…) function and this solved the problem, both in the separate VB application and in SSRS.

Conclusion

SSRS VB code may fail in unexpected ways, if a required function overload does not exist.

If debugging VB code in SSRS is problematic, try debugging it outside of SSRS in a separate program.

Blank SSRS report caused by ScriptManager timeout

I was testing a fairly complex and long running report for SQL Server Reporting Services, but the results were blank when using the Report Viewer component.

 

Status in the ExecutionLog* views was rsSuccess in the ReportServer database.

 

The ReportServerService*.log showed no errors, only this:

library!ReportServer_0-1!a78!09/23/2016-11:19:31:: i INFO: RenderForNewSession('/Test report')
webserver!ReportServer_0-1!a78!09/23/2016-11:21:59:: i INFO: Processed report. Report='/Test report', Stream=''

 

I used procdump to log exception messages from the IIS worker process with:

procdump.exe -f "" -l -e 1 w3wp.exe

Part of the result was:

[11:21:01] Exception: E0434F4D.System.Net.WebException ("The request was aborted: The request was canceled.")
[11:21:01] Exception: E0434F4D.System.Net.WebException ("The request was aborted: The request was canceled.")
[11:21:01] Exception: E0434F4D.Microsoft.Reporting.WebForms.ReportServerException ("The operation was canceled.")
[11:21:01] Exception: E0434F4D.Microsoft.Reporting.WebForms.ReportServerException ("The operation was canceled.")

 

So the request was actively getting cancelled.

This definitely seemed like a timeout issue, but which timeout setting was causing the problem?

 

Checked the default value for ReportViewer.ServerReport.Timeout, which was 600 seconds.

 

Then I googled for: ReportServerException (“The operation was canceled.”)

And found a solution here: Report Viewer Control returns OperationCanceledException OnError event after 90 seconds

 

The solution was to modify the .aspx page with the ReportViewer control:

From:

<asp:ScriptManager ID="ScriptManager1" runat="server">

To:

<asp:ScriptManager ID="ScriptManager1" runat="server" AsyncPostBackTimeout="600">

Debugging blank SSRS report caused by invalid XML characters

I was testing a modified report for SQL Server Reporting Services, but the results were blank and no relevant error messages were logged by SSRS.

Only messages like these that didn’t reveal the root cause:

ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerHttpRuntimeInternalException: RsWorkerRequest::FlushResponse., Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerHttpRuntimeInternalException: An internal or system error occurred in the HTTP Runtime object for application domain ReportServer_MSSQLSERVER_0-1-131184109058523378.  ---> System.Runtime.InteropServices.COMException (0x800703E3): The I/O operation has been aborted because of either a thread exit or an application request. (Exception from HRESULT: 0x800703E3)

 

Started troubleshooting by comparing the modified report with the original version, but I could see no obvious problems in either version.

 

When testing the report through the Report Viewer component resulted in this exception:

[XmlException: '♫', hexadecimal value 0x0E, is an invalid character. Line 1, position 6512.]
   System.Xml.XmlTextReaderImpl.Throw(Exception e) +88
   System.Xml.XmlTextReaderImpl.ParseNumericCharRefInline(Int32 startPos, Boolean expand, StringBuilder internalSubsetBuilder, Int32& charCount, EntityType& entityType) +817
   System.Xml.XmlTextReaderImpl.ParseCharRefInline(Int32 startPos, Int32& charCount, EntityType& entityType) +98
   System.Xml.XmlTextReaderImpl.ParseText(Int32& startPos, Int32& endPos, Int32& outOrChars) +816
   System.Xml.XmlTextReaderImpl.ParseText() +126
   System.Xml.XmlTextReaderImpl.ParseElementContent() +438
   System.Xml.XmlReader.ReadElementString() +84
   Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderServerReportSoapProxy.Read5_ValidValue(Boolean isNullable, Boolean checkType) +489

 

I wanted to verify that it was the same problem in pure SSRS, so I used ProcDump to log exception messages with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

 

The result was:

[17:47:58] Exception: E0434F4D.System.Xml.XmlException ("'♫', hexadecimal value 0x0E, is an invalid character. Line 1, position 6508.")
[17:47:58] Exception: E0434F4D.System.InvalidOperationException ("There is an error in XML document (1, 6508).")
[17:47:58] Exception: E0434F4D.MissingEndpointException ("The attempt to connect to the report server failed. Check your connection information and that the report server is a compatible version.")
[17:47:58] Exception: E0434F4D.System.Runtime.InteropServices.COMException ("The I/O operation has been aborted because of either a thread exit or an application request. (Exception from HRESULT: 0x800703E3)")
[17:47:58] Exception: E0434F4D.System.Runtime.InteropServices.COMException ("The I/O operation has been aborted because of either a thread exit or an application request. (Exception from HRESULT: 0x800703E3)")
[17:47:58] Exception: E0434F4D.Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerHttpRuntimeInternalException ("An internal or system error occurred in the HTTP Runtime object for application domain ReportServer_MSSQLSERVER_0-1-131184109058523378. ")

 

That confirmed that the problem was caused by invalid XML characters somewhere.

 

I wanted to see the XML that contained the invalid characters, so I attached WinDbg to ReportingServicesService.exe and enabled the CLR Exception event filter.

 

Refreshed the report to reproduce the problem.

Tried looking at the .NET call stack including parameters and locals with:

!clrstack -a

First part of the stack was:

OS Thread Id: 0x1d84 (26)
        Child SP               IP Call Site
00000011d267d1d8 00007ffef2638a5c [HelperMethodFrame: 00000011d267d1d8] 
00000011d267d2c0 00007ffedc73e488 System.Xml.XmlTextReaderImpl.Throw(System.Exception)
    PARAMETERS:
        this = <no data>
        e = <no data>
    LOCALS:
        <no data>

00000011d267d300 00007ffedc3cfd11 System.Xml.XmlTextReaderImpl.ParseNumericCharRefInline(Int32, Boolean, System.Text.StringBuilder, Int32 ByRef, EntityType ByRef)
    PARAMETERS:
        this (0x00000011d267d380) = 0x0000000fae5d0c00
        startPos = <no data>
        expand (0x00000011d267d390) = 0x0000000000000001
        internalSubsetBuilder (0x00000011d267d398) = 0x0000000000000000
        charCount (0x00000011d267d3a0) = 0x00000011d267d3f0
        entityType = <no data>
    LOCALS:
        <no data>
        0x00000011d267d340 = 0x0000000000000a6b
        0x00000011d267d330 = 0x0000000fae5d1328
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <CLR reg> = 0x000000000000000e
        <no data>
        <no data>

 

I examined the local object 0x0000000fae5d1328 with:

!DumpObj /d 0000000fae5d1328

Which seemed to be used as a buffer for XML data:

Name:        System.Char[]
MethodTable: 00007ffee08debd0
EEClass:     00007ffee02f10d0
Size:        8218(0x201a) bytes
Array:       Rank 1, Number of elements 4097, Type Char (Print Array)
Content:     MultiValue>true</MultiValue><QueryParameter>true</QueryParameter><Prompt>Device Type</Prompt><PromptUser>true</PromptUser><Depen
Fields:
None

 

I decided to find the adresses involved by dumping the array with:

!DumpArray /d 0000000fae5d1328

Part of the result was:

Name:        System.Char[]
MethodTable: 00007ffee08debd0
EEClass:     00007ffee02f10d0
Size:        8218(0x201a) bytes
Array:       Rank 1, Number of elements 4097, Type Char
Element Methodtable: 00007ffee08dec38
[0] 0000000fae5d1338
[1] 0000000fae5d133a
...
...
[4095] 0000000fae5d3336
[4096] 0000000fae5d3338

 

Now knowing the address range the unicode characters in the buffer could be read with:

du 0000000fae5d1338 0000000fae5d3338

Part of the result was:

0000000f`ae5d1338  "MultiValue>true</MultiValue><Que"
0000000f`ae5d1378  "ryParameter>true</QueryParameter"
...
...
0000000f`ae5d27b8  "/Value></ValidValue><ValidValue>"
0000000f`ae5d27f8  "<Label>♫♫113897</Label><"
0000000f`ae5d2838  "Value>52b3bc3a-32a8-49fc-a658-66"
0000000f`ae5d2878  "c67d3b29f6</Value></ValidValue><"
...
...

 

Checked the XML for invalid characters and eventually found some.

In a more readable form:

<ValidValue>
  <Label>♫♫113897</Label>
  <Value>52b3bc3a-32a8-49fc-a658-66c67d3b29f6</Value>
</ValidValue>

 

I thought it would be interesting to see the last query before the exception occured, so I started a trace with SQL Server Profiler and found:

SELECT cast(cast(0 as binary) as uniqueidentifier) AS Id, 'UNKNOWN' AS Username
UNION
SELECT Id, Username FROM Users order by 2

 

Tried running the query in SQL Server Management Studio and part of the result was:

query_result_usernames_with_invalid_xml_characters

 

Certain usernames actually contained characters that are invalid in XML.

This was the underlying cause behind the problem.

Conclusion

When SSRS reports fail without a clear indication of the cause, I recommend using a variety of tools and techniques to debug the problem similar to the described approach.

Using ProcDump itself to debug programs

ProcDump is an excellent tool for capturing memory dumps.

It actually attaches to processes as a debugger and it can also be used to debug certain problems without using other tools.

 

This can be useful when debugging problems on systems that are currently in use.

(Maybe policies prevent installing and attaching a traditional debugger or live debugging will simply cause too much disruption)

 

ProcDump can be used to output exceptions and debug messages from a process with:

procdump.exe -f "" -l -e 1 Process_name_or_id

The options used are:

-f “” Exception filter with empty filter makes ProcDump output exception messages without creating memory dumps.
-l Shows debug messages from process.
-e 1 Monitors both handled and unhandled exceptions.

 

Of course ProcDump can be used normally to generate memory dumps, which can be examined on another system with a debugger.

Be aware that processes are paused when ProcDump takes a memory dump and also be aware of the disk usage, especially when taking full memory dumps.

 

Recently using this technique helped me debug a problem on a fresh installation of SQL Server Reporting Services 2012.

All reports were returning blank results and no error messages were shown or logged.

 

I started monitoring the ReportingServicesService.exe process with:

procdump.exe -f "" -l -e 1 ReportingServicesService.exe

 

After refreshing the report the output from ProcDump was:

ProcDump v8.0 - Writes process dump files
Copyright (C) 2009-2016 Mark Russinovich
Sysinternals - www.sysinternals.com
With contributions from Andrew Richards

Process:               ReportingServicesService.exe (11180)
CPU threshold:         n/a
Performance counter:   n/a
Commit threshold:      n/a
Threshold seconds:     n/a
Hung window check:     Disabled
Log debug strings:     Enabled
Exception monitor:     First Chance+Unhandled
Exception filter:      Display Only
Terminate monitor:     Disabled
Cloning type:          Disabled
Concurrent limit:      n/a
Avoid outage:          n/a
Number of dumps:       1
Dump folder:           C:\bin\Procdump\
Dump filename/mask:    PROCESSNAME_YYMMDD_HHMMSS


Press Ctrl-C to end monitoring without terminating the process.

CLR Version: v2.0.50727

[18:22:24] Exception: E0434F4D.System.IndexOutOfRangeException ("QC")
[18:22:26] Exception: E0434F4D.System.Net.Sockets.SocketException ("No such host is known")
[18:22:26] Exception: E0434F4D.System.Net.WebException ("The remote name could not be resolved: 'reportingutility'")
[18:22:26] Exception: E0434F4D.System.Net.WebException ("The remote name could not be resolved: 'reportingutility'")
[18:22:26] Exception: E0434F4D.System.Net.WebException ("The remote name could not be resolved: 'reportingutility'")
[18:22:26] Exception: E0434F4D.System.Net.WebException ("The remote name could not be resolved: 'reportingutility'")

 

The report was trying to call a web service from VB code, however the host name was unknown.

The problem was resolved by specifying the host name in the system hosts file.

Conclusion

ProcDump can also be used on its own to debug programs.

Neuron ESB service may hang due to RabbitMQ service

I recently experienced a situation where the Neuron ESB service would hang when restarting.

ESBService.exe was stuck at Stopping and didn’t finish within 5 minutes, so it seemed to be permanently stuck.

Troubleshooting

Resource Monitor’s “Analyze Wait Chain” feature showed that 3 threads were waiting to finish network I/O.

ESBService_Analyze_Wait_Chain_waiting_to_finish_network_IO

 

I took 5 memory dumps using procdump:

procdump -ma EsbService.exe

 

WinDbg showed similar call stacks for the waiting threads:

0:060> !clrstack
OS Thread Id: 0x4e58 (60)
Child SP IP Call Site
000000001cd6dd88 000000007757d3fa [InlinedCallFrame: 000000001cd6dd88] System.Net.UnsafeNclNativeMethods+OSSOCK.recv(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags)
000000001cd6dd88 000007fee9217e61 [InlinedCallFrame: 000000001cd6dd88] System.Net.UnsafeNclNativeMethods+OSSOCK.recv(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags)
000000001cd6dd60 000007fee9217e61 DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags)
000000001cd6de10 000007fee919c501 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)
000000001cd6de90 000007fee919c3b8 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags)
000000001cd6def0 000007fee919c132 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)
000000001cd6df50 000007feea1479bf System.IO.BufferedStream.ReadByte()
000000001cd6df90 000007feea13444f System.IO.BinaryReader.ReadByte()
000000001cd6dfc0 000007fe8c0ad406 RabbitMQ.Client.Impl.Frame.ReadFrom(RabbitMQ.Util.NetworkBinaryReader)
000000001cd6e020 000007fe8c0ad374 RabbitMQ.Client.Impl.SocketFrameHandler_0_9.ReadFrame()
000000001cd6e070 000007fe8c0ad27c RabbitMQ.Client.Impl.ConnectionBase.MainLoopIteration()
000000001cd6e0b0 000007fe8c0acfe2 RabbitMQ.Client.Impl.ConnectionBase.MainLoop()
000000001cd6e110 000007feea0c2490 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000001cd6e1e0 000007feea0c2327 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000001cd6e210 000007feea0c22e2 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
000000001cd6e260 000007feea162362 System.Threading.ThreadHelper.ThreadStart()
000000001cd6e548 000007feeb203753 [GCFrame: 000000001cd6e548]
000000001cd6e898 000007feeb203753 [DebuggerU2MCatchHandlerFrame: 000000001cd6e898]
000000001cd6ea28 000007feeb203753 [ContextTransitionFrame: 000000001cd6ea28]
000000001cd6ec48 000007feeb203753 [DebuggerU2MCatchHandlerFrame: 000000001cd6ec48]

 

The call stacks indicated that the problem was related to RabbitMQ.

 

I decided to forcefully stop the ESBService.exe process.

 

When starting ESBService.exe it was still trying to communicate with RabbitMQ and hanging.

I noticed this error in the Neuron ESB Event Log:

Event Info: An error occurred trying to update the status for RabbitMQ exchange "DEFAULT.Enterprise.FileContent": This request operation sent to net.tcp://localhost:50004/MasterControl/ did not receive a reply within the configured timeout (00:01:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.. Attempted to Query 'localhost' on port '15672'.

 

I decided to restart the RabbitMQ service, which solved the problem.

After this the Neuron ESBService.exe was able to start and work normally.

 

If the problem occurs again, I will take and examine memory dumps of the RabbitMQ service, in an attempt to learn the cause of the problem.

This occured with Neuron ESB version 3.5.3.411 and RabbitMQ version 3.2.2 (bundled with Neuron ESB).

 

I notice that the latest RabbitMQ version is currently 3.6.1:

https://www.rabbitmq.com/changelog.html

It seems relevant to update RabbitMQ, but compatibility with the current Neuron ESB version is currently unknown.