WideClient crashing due to ntdll.dll

[frazer84]

Hi Pete,

 

I'm having constant problems with my WideClient application that is crashing. This started a couple of weeks ago.

 

The error message are as follows:

 

Faulting application name: WideClient.exe, version: 6.9.9.9, time stamp: 0x522d9ee3

Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521ea8e7

Exception code: 0xc0000005

Fault offset: 0x0002e3be

Faulting process id: 0x8c4

Faulting application start time: 0x01cf4ded19bcc897

Faulting application path: C:\Program Files (x86)\WideClient\WideClient.exe

Faulting module path: C:\Windows\SysWOW64\ntdll.dll

Report Id: 687c953b-b9e0-11e3-b17b-60a44c314c13

 

Any idea what this could be? Is there any way to further log or debug what could be the reason? Ntdll.dll doesn't tell me much at least.

 

I'm using the latest version of FSUIPC (tried the latest beta-version as well), same thing with WideClient.

 

Thanks

 

Regards,

Daniel

[Pete Dowson]

'm having constant problems with my WideClient application that is crashing. This started a couple of weeks ago.

 

The only time I get a crash, and it only occurs on one out of 7 client PCs, is when FSX crashes. The other 6  PCs just go back into "waiting for connection" mode. I suspect the Networking hardware or driver in the one machine. BUT it never does this whilst things are running, only when FSX crashes, so I've never speared the time to try and find out why.

 

As far as I recall (from my last FSX crash, some long time ago) I think that crash is also in NTDLL, but this doesn't really help much because NTDLL is a collection of hundreds of Windows functions.

 

Perhaps you can describe specifically what circumstances this crash occurs in? 

 

Is there any way to further log or debug what could be the reason? Ntdll.dll doesn't tell me much at least.

 

The standard WideClient.log itself just possibly might be useful to start with. Otherwise, since we've no idea what the cause is, only full logging might be useful and that produces huge logs. So let's take that step afterwards.

 

Pete

 

[frazer84]

The only time I get a crash, and it only occurs on one out of 7 client PCs, is when FSX crashes. The other 6  PCs just go back into "waiting for connection" mode. I suspect the Networking hardware or driver in the one machine. BUT it never does this whilst things are running, only when FSX crashes, so I've never speared the time to try and find out why.

 

As far as I recall (from my last FSX crash, some long time ago) I think that crash is also in NTDLL, but this doesn't really help much because NTDLL is a collection of hundreds of Windows functions.

 

Perhaps you can describe specifically what circumstances this crash occurs in? 

 

I'm running Prepar3d 1.4, and that is not at all affected by the WideClient crash. All other clients are running fine. This happens during the startup of windows. I've added some applications to automatically start when WideClient get a connection to P3D, and it's when this happens WideClient crashes. If I remove the Prosim737 main application from the list of applications, then WideClient doesn't crash. Sometimes it works and sometimes it doesn't, which is also a bit strange. I talked to Marty, the developer behind Prosim737, and he hasn't changed anything in his FSUIPC integration code for long now.

 

It might be a driver problem, but in that case I need to know which driver that is the problem. I've updated all drivers to the latest version, but if I need to revert any of these, I need to know which.

 

The standard WideClient.log itself just possibly might be useful to start with. Otherwise, since we've no idea what the cause is, only full logging might be useful and that produces huge logs. So let's take that step afterwards.

 

I can't find anything in the log, and I guess that because the application crashes to the desktop and it doesn't have the time to log anything?

[Pete Dowson]

This happens during the startup of windows. I've added some applications to automatically start when WideClient get a connection to P3D, and it's when this happens WideClient crashes. If I remove the Prosim737 main application from the list of applications, then WideClient doesn't crash.

 

If you run ProSim737 manually, later (i.e. leaving WideClient to get itself fully sorted out first),does it still crash?

 

It might be a driver problem, but in that case I need to know which driver that is the problem. I've updated all drivers to the latest version, but if I need to revert any of these, I need to know which.

 

 

Sorry, no idea. but if you do find out please let me know. Maybe I could fix mine then,

 

I can't find anything in the log, and I guess that because the application crashes to the desktop and it doesn't have the time to log anything?

 

Well, it opens its log before doing anything so there certainly should be something there. Else it's Windows loading that's crashing! And if it doesn't crash till it is connected there with be log entries showing the connection attempts and the connection, as well as its action in running programs. I think you must be looking in the wrong place? The file is "Wideclient.log" and it's next to WideClient.exe and Wideclient.ini in your WideClient folder.

 

If there's no log there's no point in adding more logging, so let me know what the current reality is, please.

 

BTW, I thought ProSim's software used its own networking connections, not WideFS? Could there be a clash with ports? Either way, if it does, it still points to Networking problems, low level.

 

Pete

[frazer84]

If you run ProSim737 manually, later (i.e. leaving WideClient to get itself fully sorted out first),does it still crash?

 

Sometimes yes, sometimes not.

There are actually three possible scenarios that I've seen:

1. WC (WideClient) crashes as soon as Prosim737 is started (even if you wait 10 minutes after WC was launched)

2. WC crashes as soon as the Battery switch is turned on in the Prosim software

3. WC doesn't crash at all during the whole flight session

 

I haven't seen a connection when these scenarios apply, I tried to reboot the computer 4 times and it felt random.

 

Well, it opens its log before doing anything so there certainly should be something there. Else it's Windows loading that's crashing! And if it doesn't crash till it is connected there with be log entries showing the connection attempts and the connection, as well as its action in running programs. I think you must be looking in the wrong place? The file is "Wideclient.log" and it's next to WideClient.exe and Wideclient.ini in your WideClient folder.

 

If there's no log there's no point in adding more logging, so let me know what the current reality is, please.

 

I will check the logs in a few hours and post the result!

 

BTW, I thought ProSim's software used its own networking connections, not WideFS? Could there be a clash with ports? Either way, if it does, it still points to Networking problems, low level.

 

Well, it does for it's own modules. It uses a combination of FSUIPC and SimConnect to communicate with P3D though.

[frazer84]

I've got the log-files now finally.

 

Here's a log-file from a normal working startup of WideClient:

********* WideClient Log [version 6.999b] Class=FS98MAIN *********

Date (dmy): 16/04/14, Time 20:40:11.220: Client name is 737PROSIM

       63 LUA: "C:\Program Files (x86)\WideClient\Initial.LUA": not found

       63 Attempting to connect now

       63 Trying TCP/IP addr 192.168.2.50, port 8002 ...

       63 Error on client pre-Connection Select() [Error=10065] No route to host

       63 Ready to try connection again

     1123 Attempting to connect now

     1186 Connection made okay!

     1576 c:\program files (x86)\swesimappstarter

     1576 C:\Program Files (x86)\SwesimAppStarter\SwesimAppStarter.exe

     1701 New Client Application: "SwesimAppStarter" (Id=3272)

     2512 New Client Application: "USBFSUIPCLink" (Id=3492)

     6084 New Client Application: "TQControl" (Id=3700)

     9501 New Client Application: "YokeControl" (Id=3632)

    13806 New Client Application: "BFF_Control_Loader_v1_104" (Id=3188)

    17488 New Client Application: "sioc" (Id=3872)

    43103 New Client Application: "ProsimMCP" (Id=3780)

    62245 New Client Application: "Prosim737" (Id=4336)

   727433 New Client Application: "ProsimMCP" (Id=5712)

 

 

And here's the same for the crashing during startup scenario:

********* WideClient Log [version 6.999b] Class=FS98MAIN *********

Date (dmy): 17/04/14, Time 09:17:48.438: Client name is 737PROSIM

       47 LUA: "C:\Program Files (x86)\WideClient\Initial.LUA": not found

       47 Attempting to connect now

       47 Trying TCP/IP addr 192.168.2.50, port 8002 ...

       47 Error on client pre-Connection Select() [Error=10065] No route to host

       47 Ready to try connection again

     1061 Attempting to connect now

     1155 Connection made okay!

     1576 c:\program files (x86)\swesimappstarter

     1576 C:\Program Files (x86)\SwesimAppStarter\SwesimAppStarter.exe

     1685 New Client Application: "SwesimAppStarter" (Id=3284)

     2512 New Client Application: "USBFSUIPCLink" (Id=3532)

     6084 New Client Application: "TQControl" (Id=3736)

     9173 New Client Application: "YokeControl" (Id=3644)

    13447 New Client Application: "BFF_Control_Loader_v1_104" (Id=3280)

    17129 New Client Application: "sioc" (Id=3948)

    42760 New Client Application: "ProsimMCP" (Id=3040)

    61777 New Client Application: "Prosim737" (Id=4140)

 

As you can see, the log stops at the launch of the "Prosim737" application and doesnt proceed to register the launch of the ProsimMCP application since WideClient crashes before then. No error messages though.

[Pete Dowson]

As you can see, the log stops at the launch of the "Prosim737" application and doesnt proceed to register the launch of the ProsimMCP application since WideClient crashes before then. No error messages though.

 

Those log lines aren't registering launches of programs, since they aren't actually being launched by WideClient. They are registering the point where the application connects to FSUIPC via WideClient. So the crash could actually be when ProSimMCP starts. However, your comment does seem a little odd in relation to this log because the order shown is ProSimMCP first, then ProSim737. Should they be the other way around? In the example without the crash they were!

 

If you were loading these programs by WideClient parameters you could insert delays between each one loading, experimenting to get it right. Can the loader you are using do that? It might be worth trying to get the MCP module starting a few seconds later than the main ProSim module.

 

Pete

[frazer84]

Those log lines aren't registering launches of programs, since they aren't actually being launched by WideClient. They are registering the point where the application connects to FSUIPC via WideClient. So the crash could actually be when ProSimMCP starts. However, your comment does seem a little odd in relation to this log because the order shown is ProSimMCP first, then ProSim737. Should they be the other way around? In the example without the crash they were!

If you were loading these programs by WideClient parameters you could insert delays between each one loading, experimenting to get it right. Can the loader you are using do that? It might be worth trying to get the MCP module starting a few seconds later than the main ProSim module.

Pete

Sorry about that, was in a bit of a hurry when looking at the log files. You are right about the ProsimMCP application.

My launcher app allows for delay, it's a long delay (10 secs) between all the applications to minimise the risk of crashes etc.

[Pete Dowson]

Sorry about that, was in a bit of a hurry when looking at the log files. You are right about the ProsimMCP application.

My launcher app allows for delay, it's a long delay (10 secs) between all the applications to minimise the risk of crashes etc.

 

Okay, so if it consistently works when the MCP connects after the other, try to set the timings and load order to reflect that. Maybe even put some of the other loads between the two?

 

Ah, wait a minute. I see that the order at the start WAS the same in the "OK" log. It was just that you reloaded the MCP much later (0ver 10 minutes later, in fact), so it might be a red herring.

 

I think it must be some problem occurring when several programs simultaneously try to do things on the network. Do other ProSim users have problems?

 

You could enable more logging in WideClient, but with 8 clients it would be a massive log and is unlikely to show anything useful because the crash is deep into Windows code.

 

Pete

[frazer84]

Okay, so if it consistently works when the MCP connects after the other, try to set the timings and load order to reflect that. Maybe even put some of the other loads between the two?

 

Sorry if I missled you there. Either way doesn't work. I also tried to remove the Prosim737 application from the startup and started it manually at different stages and the crash could happen either way I did it. It happens with or without the ProsimMCP started, also independent of the startup order. 

To summarize, I haven't found any combination (except not starting Prosim737 at all) that does work all the come consistently.

[Pete Dowson]

Sorry if I missled you there. Either way doesn't work. I also tried to remove the Prosim737 application from the startup and started it manually at different stages and the crash could happen either way I did it. It happens with or without the ProsimMCP started, also independent of the startup order. 

To summarize, I haven't found any combination (except not starting Prosim737 at all) that does work all the come consistently.

 

Hmm. I'd like to what that program is doing. Can you set things up so you ONLY run ProSim737, no other clients? Then it might just be worth doing some logging.

 

Of course that might be a waste of time too if it never fails that way.

 

Pete

[frazer84]

Hmm. I'd like to what that program is doing. Can you set things up so you ONLY run ProSim737, no other clients? Then it might just be worth doing some logging.

Of course that might be a waste of time too if it never fails that way.

That can be set up fairly easy yes. It's probably worth a try since I can't find any more information the normal way.

[Pete Dowson]

That can be set up fairly easy yes. It's probably worth a try since I can't find any more information the normal way.

 

Let me know if you can get it to fail that way and I'll work out what logging parameters you might try. I'm still doubtful that it will show much. WideClient itself is very protective of itself, trapping errors (even access violations) and logging them. It just can't trap stuff deep in Windows, especially not in the Networking stuff.

 

Pete

[frazer84]

Let me know if you can get it to fail that way and I'll work out what logging parameters you might try. I'm still doubtful that it will show much. WideClient itself is very protective of itself, trapping errors (even access violations) and logging them. It just can't trap stuff deep in Windows, especially not in the Networking stuff.

 

Today I spend a couple of hours on trying all different ways of starting these applications.

It gave me some new info, but I didn't find any way for me to start the applications I want without loosing WideClient along the way.

 

Only starting the Prosim737 application and WideClient alone doesn't cause a crash. It's the combination of Prosim737 and the ProsimMCP application that causes this. These two applications are the only ones in the Prosim suite that uses a FSUIPC connection, the rest goes through the Prosim server to fetch data from FSUIPC.

 

I tried to start the application directly from WideClients run-when-ready-feature with a delay. The result was a bit different. Using this method, WideClient froze instead of crashing by definition. This is illustrated by Microsofts fancy "milky"-effect:

 

The log file created when this happened (happened on every start of WideClient using this config) has been attached as WideClient.log.txt.

 

I also tried this setup using the Debug mode of Logging. That file is attached as WideClient_debug.log.txt.

 

I'm not sure if this would indicate something to you. I found it interesting that a combination of two different applications using the FSUIPC link can cause this crash, and that the application freezes if they are launched from within WideClient.

 

Thanks!

WideClient.log.txt

WideClient_debug.log.txt

[Pete Dowson]

Only starting the Prosim737 application and WideClient alone doesn't cause a crash. It's the combination of Prosim737 and the ProsimMCP application that causes this.

These two applications are the only ones in the Prosim suite that uses a FSUIPC connection, the rest goes through the Prosim server to fetch data from FSUIPC.

 

I suppose without ProSim737 the MCP one doesn't do anything or won't load, so it isn't possible to isolate it just to the MCP one ... or is it?

 

I tried to start the application directly from WideClients run-when-ready-feature with a delay. The result was a bit different. Using this method, WideClient froze instead of crashing by definition. This is illustrated by Microsofts fancy "milky"-effect:

 WideClient_freeze.png

 

Hmm. Very strange. I don't understand why that would be any different.

 

The log file created when this happened (happened on every start of WideClient using this config) has been attached as WideClient.log.txt.

 

Doesn't show anything different, does it.

 

I also tried this setup using the Debug mode of Logging. That file is attached as WideClient_debug.log.txt.

I'm not sure if this would indicate something to you. I found it interesting that a combination of two different applications using the FSUIPC link can cause this crash, and that the application freezes if they are launched from within WideClient.

 

 

 

Well, interesting but it doesn't really help. Any the above log shows everything normal till the freeze, which oddly is NOT happening after a client has called WideClient for data, but just during normal polling. It's as if the Network part of Windows has simply stopped calling WideClient with the WSA_READ message which tells it it is okay to ask for more data. So, like the NTDLL crash, it implies the problem is actually occurring in the Networking part of Windows.

 

Can you reproduce the hang and/or the crash  with Log=Yes rather that Log=Debug please.

 

I'm not sure we are going to get anywhere with this, at least not without some sort of clever network analyser. Do you have another PC to try it on at all? As I said, the only similar crash I get with WideClient (and in my case only when FSX crashes) happens only on one out of 6 client PCs in my system. I've never got to the bottom of that.

 

Pete