[solved by Asobo] Reproducible MFS crashes with legacy FSUIPC config

[aurel42]

Hi,

I'm running FSUIPC7 with a configuration inherited from previous FSUIPC versions for P3D, because it knows all about my plethora of HID devices.

Sadly, I found that whenever FSUIPC is running, MFS will crash after a certain time. I'm saying it's reproducible, because when I redo the same flight with the same aircraft and the same route, the crash will very likely happen after covering roughly the same distance.

When MFS crashes, it just CTDs without hanging first (there are no Windows "not responding" or other dialogs). Most of the time, the Event Viewer assigns the blame ("faulting module") to "FlightSimulator.exe" itself, sometimes to "C:\WINDOWS\System32\ucrtbase.dll".

If I had to guess: some form of SimConnect-related resource exhaustion (read: leak) in MFS?

FSUIPC just states that "MSFS no longer running" and is otherwise unaffected. My FSUIPC version is a few weeks old. Since FSUIPC only talks to MFS using SimConnect, and no SimConnect application should ever be able to crash the sim it talks to, I didn't bother to retest this with the latest FSUIPC release. I will update before trying again.

How would I go about isolating the underlying issue of the crash? If I start over with a fresh .ini, can I transfer my controller setup to the new file instead of redoing the whole configuration, calibrations and assignments in FSUIPC7?

I've attached ini and log of the last flight.

Thanks
Marc

PS: I also came across an issue with large log files being written for random lua scripts after enabling and disabling separate LUA logs while debugging an issue with one of my scripts, but I'll have to retest that with the latest FSUIPC version. The MFS crashes are happening whether my lua scripts are active or not (I disable them by changing the file extensions and restarting FSUIPC).

FSUIPC7.ini FSUIPC7_prev.log

[John Dowson]

Hi Marc,

so MSFS is crashing, but why do you think this is related to FSUIPC? With FSUIPC now being an exe rather than a dll, it should not crash the sim. Even if its a problem in the server side of simconnect, it is still an issue with MSFS.

10 hours ago, aurel42 said:

How would I go about isolating the underlying issue of the crash? If I start over with a fresh .ini, can I transfer my controller setup to the new file instead of redoing the whole configuration, calibrations and assignments in FSUIPC7?

There will be nothing in your ini that would cause this.

You can try disabling the option 'Exit with FS', and FSUIPC7 will then still be running once MSFS has a CTD. 

[aurel42]

1 hour ago, John Dowson said:

so MSFS is crashing, but why do you think this is related to FSUIPC?

Because it exclusively happens while FSUIPC is running and connected. I've repeated two separate two hour flights (different routes, different aircraft) three times each under the same conditions while running FSUIPC and once each without. For both flights, MFS crashed in roughly the same stage of the flight after covering about the same distance when FSUIPC was active, while I could complete the flight without issues when FSUIPC was not running. This might be anecdotal evidence to you, but it was conclusive enough for me to bring it up here.

I did not add any community mods to my sim, and I'm not using any add-ons except TrackIR, the FS Economy client, Little Navmap and FSUIPC. I have used the FS Economy client and Little Navmap continuously since shortly after release of MFS without any issues and, before FSUIPC entered the equation, in the six weeks since the launch of MFS I have experienced a single crash in about 300 hrs of "time spent in game" (according to Steam), and that crash was caused by an "imported" aircraft on an official community group flight and affected many other players (after that crash, I switched multiplayer models to "generic", I'd rather see everybody flying in 172's than risk a crash), so I feel my system is stable enough to run MFS without issues.

I only started using FSUIPC a few days ago. At first I did mostly short flights (to test controller setup etc.) and saw no issues. Only in the past two days, when I tried longer flights with FSUIPC, MFS started crashing regularly, as described.

I actually pointed out explicitly that I don't believe FSUIPC is the culprit in this scenario. And that I believe that FSUIPC might be triggering a resource exhaustion in the SimConnect code. If that were the case, of course the INI settings would make a difference, because different settings lead to different usage of the SimConnect interface.

Also, I stated in my original post that FSUIPC was not affected by the crash.

While I understand your frustration about people coming in here complaining about their MFS crashes, there seems to be no way forward for me at this point, so I'll just stop using FSUIPC for the time being, hoping that the issue will affect enough other users that it will at some point be taken care of.

Thanks.

 

[Thomas Richter]

Hi,

if you could finish your test flights without FSUIPC7 running could you please do a test flight with FSUIPC7 running but having FSUIPC7.ini file removed from the folder before starting FSUIPC7, so it will create a fresh INI file. That would identify if the previous INI file you used contains something that triggers that.

Thomas

[John Dowson]

You could try activating SimConnect logging, to see if anything is reported when the crash occurs. other than that, you should raise a zendesk issue with MSFS/Asobo, and include the windows crash report from the event viewer.

If you want to test with a fresh ini, you could let FSUIPC7 generate a fresh/clean one, and then copy/replace all sections apart from the [General] section, and also remove the Path variable from the [Sounds] section, together with your missing yoke in the [JoyNames] section:

Quote

Y=CLSE NG Yoke << MISSING JOYSTICK >>
Y.GUID={BBAB7FA0-EC23-11E9-8003-444553540000}

 

[aurel42]

1 hour ago, John Dowson said:

You could try activating SimConnect logging, to see if anything is reported when the crash occurs.

Is that the "Events" category from the FSUIPC Log menu?

Thanks for the advice on how to keep my controller settings with a fresh ini. I'll do some more testing and will report the issue to Asobo.

[John Dowson]

12 minutes ago, aurel42 said:

1 hour ago, John Dowson said:

You could try activating SimConnect logging, to see if anything is reported when the crash occurs.

Is that the "Events" category from the FSUIPC Log menu?

No. 

To activate SimConnect logging, create a file called SimConnect.ini in the FLT UNC Path (or FLT Path), as reported at the beginning of your FSUIPC7.log file (for Steam installs this will be C:\Users\<YourName>\AppData\Roaming\Microsoft Flight Simulator\, and for MS Store installs, this maybe under the LocalCache folder)  with the following contents:

Quote

[SimConnect]
level=verbose
console=No
RedirectStdOutToConsole=No
OutputDebugString=no
file=<path>\SimConnect%01u.log
file_max_index=9

replacing <path> with the path specification (e.g. file=C:\SimConnect%01u.log).

 

[aurel42]

I filed a bug on Zendesk, and included a couple of examples of the crash information produced by the "Reliability History" tool. I seem to remember there's a way to get more detailed crash reports from Windows (than with Event Viewer or Reliability History) with the help of a tool that doesn't come with Windows (not sure if by Microsoft or by a 3rd party), but my Google-fu fails me.

Here's a short quantitative analysis of the SimConnect log of a flight ending in a MFS CTD:

Lines in file: 1498660

Clients:
> 3.99297 [320, 1]Open: Version=0x00000004  Name="Little Navconnect"
> 17.74603 [381, 1]Open: Version=0x00000005  Name="FSUIPC7"
> 17.79976 [382, 1]Open: Version=0x00000005  Name="FSUIPC7_AItraffic"
> 1041.62965 [63, 1]Open: Version=0x00000003  Name="FS Economy"

Usage by client:
63: 6973     # FS Economy
320: 2485    # Little Navmap's Little Navconnect
381: 1471255 # FSUIPC7
382: 7091    # FSUIPC7_AItraffic

Usage by event/class/command/object/whatever:
Open: 4
AddToDataDefinition: 1360
SubscribeToSystemEvent: 28
RequestDataOnSimObjectType: 2550
MapClientEventToSimEvent: 1107
SetInputGroupState: 1
ObjectDataByType: 2705
RequestSystemState: 1782
ObjectAddRemove: 837
RequestFacilitiesList: 330
Facilities Request: 10230
Event: 351474
EventFrame: 251861
ObjectData: 871878

Exceptions:
< 36.43413 [382] >>>>>  EXCEPTION=3, SendID=33, Index=-1  <<<<<
< 332.20962 [382] >>>>>  EXCEPTION=3, SendID=37, Index=-1  <<<<<
< 334.78455 [382] >>>>>  EXCEPTION=3, SendID=40, Index=-1  <<<<<
< 1800.05275 [382] >>>>>  EXCEPTION=3, SendID=792, Index=-1  <<<<<
< 3765.15386 [382] >>>>>  EXCEPTION=3, SendID=1903, Index=-1  <<<<<

Last lines of log:
< 4545.17444 [381] ObjectData: RequestID=14  DefineID=14  dwSize=508
< 4545.17448 [381] ObjectData: RequestID=24  DefineID=24  dwSize=96
< 4545.17459 [381] EventFrame: 2  58.013054
< 4545.17475 [381] ObjectData: RequestID=10  DefineID=10  dwSize=1096
< 4545.17476 [381] ObjectData: RequestID=9  DefineID=9  dwSize=48
< 4545.19208 [381] ObjectData: RequestID=14  DefineID=14  dwSize=460
< 4545.19211 [381] ObjectData: RequestID=24  DefineID=24  dwSize=96
< 4545.19223 [381] EventFrame: 2  55.304535
< 4545.19239 [381] ObjectData: RequestID=10  DefineID=10  dwSize=856
< 4545.19241 [381] ObjectData: RequestID=9  DefineID=9  dwSize=48

Relevant lines of FSUIPC log:
********* FSUIPC7, Version 7.0.0-Beta (6th October 2020) by John & Pete Dowson *********
[…]
   104953 Running in "KittyHawk", Version: 11.0.282174.999 (SimConnect: 11.0.62651.3)
[…]
   464640 -------------------- Starting everything now ----------------------
[NO log entries deleted here, there's really no output between 464640 and 4636390]
  4636390 Failed on SimConnect_CallDispatch for Message, return = 0xC000014B
  4636390 Failed on SimConnect_CallDispatch for Traffic Message, return = 0xC000014B
  4639312 MSFS no longer running - trying to find...
  4639312 === Calling SimConnect_Close ...

I can't see anything special perusing the end of the SimConnect log, everything that's happening in the log in the last few minutes before the crash is also happening during the whole duration of the flight.

Still, after a couple more attempts to do extended flights with all three SimConnect clients (FSE, LNM, FSUIPC) active, the pattern remains the same: MFS crashes when FSUIPC is running, even with a freshly generated and unmodified FSUIPC7.ini (one exception: I disabled "Exit with FS").

To me, this seems to be more evidence that the SimConnect code in MFS is causing these crashes on my system, and looking at the SimConnect usage of my three SimConnect clients, I feel it makes sense that a client causing the sim to do more work would be more likely to… erm… make the SimConnect problems visible (almost 1.5 mio SimConnect "events" are related to FSUIPC, compared to fewer than 10000 "events" for FSE and LNM combined).

I will continue to test after each new MFS patch, but, for now, I feel I've done enough testing and I'll return to flying without FSUIPC and without access to half of my buttons and knobs. 😭

Also, there's a typo in the menu: "Miscelleneous" ("Miscellaneous")

Cheers,
Marc

 

[aurel42]

In response to my Zendesk ticket, I just got the following advice from Asobo support:

Quote

Uninstall Visual C++ and (re)install the latest version of Visual C++ (Visual Studio 2015, 2017 and 2019) then reboot your computer

I did not have Visual C++ or Visual Studio installed when the crashes started to happen or when I reported the bug on Zendesk and here in the forums. My PC dedicated to simming and gaming is just a year old and never had any IDE installed.

Unless it's a component of the "Visual C++ Runtime" packages which are part of many application installers, I have no idea where the ucrtbase.dll on my system came from. If it is in fact a part of the Runtime package, I wonder why Asobo didn't tell me to update the Runtime only, instead of specifically mentioning Visual Studio.

But incidentally, I did in fact install Visual Studio 2019 recently, after reading this very informative post on the MFS forums about crash dumps (and how to write them and read them), but before I got the response from Asobo:

https://forums.flightsimulator.com/t/crash-to-desktop-known-issue-what-happens-from-now-on/297218/40

So, by pure chance, I had already done what Asobo recommended. And here's the funny part: it might have worked. Since setting my system up to write crash dumps and installing Visual Studio to be able to have a look at those dumps, I've been unable to reproduce the "SimConnect crash".

(btw, I found another MFS user with exactly the same issue on the MFS forums. In his case, the SimConnect applications are Little Navmap, Navigraph Charts and FSUIPC. So I'm really not hallucinating that there is a problem. 😄)

[John Dowson]

Interesting, and thanks for those links and reporting back. Let me know if you get any further crashes.

[aurel42]

Correction: ucrtbase.dll is indeed part of the Visual C++ runtime, and the reference to Visual Studio makes sense in the context of the download page for the runtime packages (the latest runtimes are labeled "Visual Studio 2015, 2017 and 2019" on that page). So they basically just told me to update the Visual C++ runtime – I'm not sure why I wouldn't have had the latest version already. I'll keep the crash dumps enabled for a while, so I can provide better information to Asobo if the runtime update didn't get rid of the crash after all.

 

[aurel42]

Yeah, the bug wasn't gone after all, I guess my test flights were just too short. MFS crashed earlier today, I provided the "user-mode dump" (aka application crash dump) to Asobo via Zendesk.

Thanks to Visual Studio, I learned that the issue is a classical segmentation fault:

Unhandled exception at 0x00007FF62D477B43 (FlightSimulator.exe) in FlightSimulator.exe.3440.dmp: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.

And this is what the stack looks like (not very informative, since Visual Studio is missing the debugging information for the executable), but, as I understand it, it definitely points to a problem in the MFS code (instead of a problem in the Visual C++ runtime dll):

[aurel42]

Since the release of MFS 1.9.5.0, I've not been able to reproduce this problem any longer; people stopped adding their stories to "my" thread on the MFS forums.

Perhaps this one is gone for good.