FSX Crash with multiple SimConnect clients

[russdirks]

Hi Pete,

I've been working on a SimConnect addon, and I've discovered a bug that could have serious ramifications for all addon developers. I've found that FSX will CTD when one client has already been loaded, and has begun making calls to the SimConnect server, and a second client is in the process of being loaded, but has generated a security warning because it is not trusted. It seems that the combination of one client communicating with the SimConnect server in the background, while FSX is displaying the security warning dialog in the foreground causes FSX to crash. If the second client is trusted and does not generate a warning, everything works fine.

I discovered this working on my FS Force program, which is comprised of an .exe and a .dll. I found that whenever I recompiled the .dll and ran FSX, FSX would crash as soon as the security warning came up for the .dll. In this case, the .exe had already been loaded and had begun talking to SimConnect.

I just found out yesterday that the same thing happens with your FSUIPC program. I sent an update of FS Force to one of my beta testers (Bill Womack), who also has FSUIPC installed. His dll.xml file was configured to load FSUIPC first and FS Force second. When he tried to start FSX, FSForce.dll triggered a security warning and FSX immediately CTD'd. I checked his SimConnect.log file, and noticed that FSUIPC was making a lot of calls to the SimConnect server immediately after the SimConnect_Open call. I instructed him to disable FSUIPC from loading, and then he was able to start FSX normally, and tell FSX to trust FSForce.dll. Then he was able to re-enable FSUIPC, after which everything worked together just fine.

If this is a confirmed bug, then I think this is going to be a big problem for developers and users of addons. Every time a developer issues an updated version that does not have a signed security certificate, it could potentially cause FSX to crash if there is another addon that is being loaded ahead of the updated one. The user will have to go through the hassle of disabling all other addons except the new one so he can tell FSX to trust the new one, after which he will be able to re-enable all the other addons.

There is a workaround however. What seems to cause the problem is when the early loading addon makes calls to either RequestDataOnSimObject or RequestDataOnSimObjectType immediately after the call to SimConnect_Open. If instead the addon waits until the simulator has actually started (via a SimStart event, or RequestSystemState), then there are no problems.

I hope you will experiment with this to confirm. My suggestion to reproduce the problem is to set up your dll.xml file to load FSUIPC first, and some other .dll second. Make sure the second .dll is not trusted, and then start FSX.

If you can confirm all this, then we need to put pressure on MS to fix this as soon as possible. I've bugged it as ID 215790. I think it would also be good to try to get the word out to developers to implement the workaround.

Regards,

Russ Dirks

[Pete Dowson]

I've been working on a SimConnect addon, and I've discovered a bug that could have serious ramifications for all addon developers. I've found that FSX will CTD when one client has already been loaded, and has begun making calls to the SimConnect server, and a second client is in the process of being loaded, but has generated a security warning because it is not trusted. It seems that the combination of one client communicating with the SimConnect server in the background, while FSX is displaying the security warning dialog in the foreground causes FSX to crash. If the second client is trusted and does not generate a warning, everything works fine.

Ouchthis needs to be notified to MS immediately, as I think they are trying to work out what to do about the problems imposed by firewall things at present.

There is a workaround however. What seems to cause the problem is when the early loading addon makes calls to either RequestDataOnSimObject or RequestDataOnSimObjectType immediately after the call to SimConnect_Open. If instead the addon waits until the simulator has actually started (via a SimStart event, or RequestSystemState), then there are no problems.

Currently FSUIPC4 issues its main series of RequestDataOnSimObject after it gets a user aircraft ID. Before that it would have issued a RequestDataOnSimObjectType to get the user aircraft ID. It issues that as soon as it gets any System State return including a UserObjectID indicating the OBJECT_ID_USER. I thought that was safe enough, but possibly it is getting that when the default aircraft or flight is being loaded. I could try moving that code specifically into the case statement for the "SimStart" state.

I hope you will experiment with this to confirm. My suggestion to reproduce the problem is to set up your dll.xml file to load FSUIPC first, and some other .dll second. Make sure the second .dll is not trusted, and then start FSX.

I don't have a second one to experiment with, although I could use a second copy of FSUIPC with a different name I suppose. How about if I make a modification, as above, and email a version to you for you to test?

Oh, I also call RequestDataOnSimObject for each AI aircraft object addded, as I receive notifications for aircraft being added (OBJECT_ADDREMOVE event). Are these ever likely to occur too early or doesn't the traffic generation start till after "SimStart"? I don't want to have to stockpile AI object IDs ready to request their data later on.

Regards

Pete

[Pete Dowson]

If instead the addon waits until the simulator has actually started (via a SimStart event, or RequestSystemState), then there are no problems.

I've looked at this, and in fact a SimStart event occurs very early on, before or early during the progress bar showing the progress or loading of scenery, textures etc.

Are you sure delaying the requests for a few seconds it takes to get to that point works?

It looks like a SimStop event follows (about 40 seconds later in my case) and then, a lot later, after all the loading, another SimStart.

That's what is happening here with my FSX configuration anyway. Trouble is I don't know if I dare ignore the first SimStart in case the sequence varies according to configuration.

Regards

Pete

[russdirks]

If instead the addon waits until the simulator has actually started (via a SimStart event, or RequestSystemState), then there are no problems.

I've looked at this, and in fact a SimStart event occurs very early on, before or early during the progress bar showing the progress or loading of scenery, textures etc.

Are you sure delaying the requests for a few seconds it takes to get to that point works?

No, the problem doesn't occur when the flight is loading, but rather when FSX is loading, and all the addons are loading. The key question is, what does your code do right after a successful call to SimConnect_Open? Originally my code would immediately make a bunch of calls to AddToDataDefinition followed by calls to RequestDataOnSimObject. The workaround involved delaying the calls to RequestData until I received a SimStart event. I noticed from the SimConnect log that Bill gave me that your code makes a call to RequestDataOnSimObjectType about 2.9 seconds after SimConnect_Open. If you could delay that call, that might solve the problem.

[russdirks]

I don't have a second one to experiment with, although I could use a second copy of FSUIPC with a different name I suppose. How about if I make a modification, as above, and email a version to you for you to test?

I've done a fair bit of testing with my own modules. I think this is something that all developers are going to have to grapple with and understand. MS may come up with a fix, but that could be some time coming. I've attached an empty DLL I put together to test my code. You can use it if you want.

[Pete Dowson]

No, the problem doesn't occur when the flight is loading, but rather when FSX is loading, and all the addons are loading.

Okay, so it should be okay after the first SimStart, provided it is only on the RequestData calls that the problem occurs, and providing no AI traffic starts up before that first Simstart.

The key question is, what does your code do right after a successful call to SimConnect_Open? Originally my code would immediately make a bunch of calls to AddToDataDefinition followed by calls to RequestDataOnSimObject.

All my calls to SubscribeToSystemEvent first, then the Menu addition, then some RequestSystemState, then a heap of AddToDatadefinition followed by a bigger heap of MapClientEvent/AddClientEvent pairs are done. The RequestData cals are done after, but certainly before the first "SimStart" event at present. But I can change that, as I said.

Regards

Pete

[Pete Dowson]

I've attached an empty DLL I put together to test my code. You can use it if you want.

Okay. Thanks. I'll try it with and without my change.

Pete

[Pete Dowson]

I've attached an empty DLL I put together to test my code. You can use it if you want.

That doesn't runthe SimConnect log shows:

9.96803 DLL Failed to load. Check for missing dependencies.

9.96808 DLL Load Failed: Error=-3 Path="Modules\Test.dll"

I assume this is because you've got no manifest resource in the DLL?

Anyway I tried it with FSUIPC4.023 and FSX continued to load for quite some time after that, before it crashed with a normal error box. I didn't get the location**, but I'll try again. Is that what you get, even with the other DLL not being loaded?

[** LATER: got it! It is 20C6A07E, in Terrain.dll!

This is the same as the crash reported in http://forums.simflight.com/posting.phpe&p=352008]

I then tried it with the change to not RequestData till after the first SimStart event, and loaded okay, up to the point where I expected to see the flight screen, and then .... FSX just disappeard. CTD'd with no error at all.

SoI don't think the work-around is quite right yet. If it appeared to work for you I think that may just be luck.

I ran it in debug mode to see if I could trap the error. I get an access violation in API.DLL at location 200E7DA9. It is using EAX as a pointer and that is zero. Additionally, before this, on every "1sec" event, instead of the event reaching my code it is giving two identical Exception 23's" (failure to load flight plan) -- something completely spurious.

From where I'm sitting it looks like something is getting corrupted really badly early on, but what I cannot say.

I'm not sure what other experiments I can try for a work-around, but first I'll try my modified version in a non-debug version (in the debug version it too with bring up the non-approved dialogue).

Regards

Pete

[russdirks]

Well, I've done some more experimentation with your program and mine, and a new copy of the Test.dll, and it looks like the workaround I described isn't as effective as I had thought. I set my system to load my program first and yours second, with mine trusted and your's not, and FSX still crashed. Sometimes it does a CTD prior to the first UI screen, sometime it gives an error like "The instruction at "0x20eab106" referenced memory at "0x07bb1054". The memory could not be read". Sometimes I make it into the first UI page, but it crashes during the flight load. So there is clearly something going on here beyond what I understand.

MS hasn't even acknowledged this error yet. We really need them to give us some information as to what's going on so we can develope some effective workarounds.

In case you want it, I've attached a new copy of the Test.dll. The one I uploaded before was a debug build.

Regards,

Russ

Test.zip

[Pete Dowson]

... it looks like the workaround I described isn't as effective as I had thought.

Yes, it doesn't work for me at all. See my last message above (please see it again if you viewed it before as I've edited it a couple of times).

MS hasn't even acknowledged this error yet. We really need them to give us some information as to what's going on so we can develope some effective workarounds.

I'm trying direct contacts, but I won't hold my breath. I think the whole team must be up to their proverbials.

In case you want it, I've attached a new copy of the Test.dll. The one I uploaded before was a debug build.

With a manifest resource now? It won't load without in any case, though that seems irrelevant to this problem.

Regards

Pete

[Pete Dowson]

I may have found a better work-around!

My first tests look good, but it's late here now so I'm going to bed. I'll try some more in the morning, and maybe send you a test version of FSUIPC4.

What I did was remove almost EVERY call upon SimConnect to be executed AFTER I receive the first SimStart event. After the Open I'm only requesting the SimStart and SimEnd events (didn't see any point in the first without the second, even if it isn't needed yet).

On my first test with that and your TEST.DLL (still not loading of course), no errors! No crashes.

More tomorrow.

Regards

Pete

[russdirks]

With a manifest resource now? It won't load without in any case, though that seems irrelevant to this problem.

That's weird! I ran it on a clean partition and it worked fine. It has a manifest resource embedded. But like you say, the only thing needed is for it to generate a security warning.