FSUIPC shutdown

[RobAins]

Hi Gents,

Installed 5.153 recently and having some odd issues with shutdown

Taskmgr ... the process never terminates after a shutdown.

I've attached the log, but don't see any issues.

Cheers, Rob.

FSUIPC5.log

[Pete Dowson]

5 hours ago, RobAins said:

Taskmgr ... the process never terminates after a shutdown.

I've attached the log, but don't see any issues.

FSUIPC is not getting as far as closing down completely. But where is P3D hung? Is it actually in an FSUIPC thread?

I'm certain that there are no recent changes in FSUIPC which would affect the closing sequence. Are you sure it is FSUIPC causing the process to hang and not another module? The only way we can tell is attaching to the process with Visual Studio and seeing what thread it is stuck in at the time. In the past this has usually been due to some device driver hanging, maybe the local network.

Is it consistent? Can you try a process of elimination please? For example, try disabling WideServer (on the About tab in FSUIPC Options). And what is your ipcInit.lua trying to do where you need an unavailable Socket support package?

Note that ipcInit.lua is run early on in FSUIPC's starting sequence, and some things won't be ready at that time. ipcReady.lua is generally the safer way to start off plug-ins.

Pete

 

[RobAins]

Hi Pete,

I thought you were on holiday?

I'm still trying to diagnose, I've eliminated many variables and I'm down to RealSimGear and RXP GNS 530 and A2A as possible triggers.  All the info I can get out of TaskMgr is "waiting for network I/O" which I think is generic and not useful.

In order to get the RXP to not trigger a CTD when added to an aircraft, I have to run P3D "as Admin" .. P3D is outside Program Files and both RSG and RXP are located in folders under Program Files.  I suspect it's some sort of security context issue.  Sadly many add-ons I have like to install themselves into Program Files without prompting install paths (SODE, RXP, CP, Immersion, etc.).  I was thinking of manually relocating them outside of Program Files, but requires a lot of registry work also and I'm trying to avoid that.

I'll do more diagnostics and get back to you, this is a difficult one because I'm not seeing any issues in your log, nothing in SimConnect log (even with Verbose), nothing in P3D logs, there is one "warning" in event viewer but I don't think it's related.

Cheers, Rob.

 

[Pete Dowson]

20 hours ago, RobAins said:

I thought you were on holiday?

Back last weekend.

20 hours ago, RobAins said:

All the info I can get out of TaskMgr is "waiting for network I/O" which I think is generic and not useful.

That's why I suggested turning off WideFS as a test.

20 hours ago, RobAins said:

I'll do more diagnostics and get back to you, this is a difficult one because I'm not seeing any issues in your log, nothing in SimConnect log (even with Verbose), nothing in P3D logs, there is one "warning" in event viewer but I don't think it's related.

Really annoying when you get problems like that. When running the debug build of FSUIPC with Visual Studio we can usually identify the last call made to a Windows API and that might help work out why.

One thing worth trying is checking that the Network drivers are fully up to date.

Keep us posted.

Pete

 

[RobAins]

 Hi Pete,

I should have listened ... it seems WideFS is the source of preventing P3D shutdown.  I suspect it's perhaps a combination of add-ons maybe? ... I'm using GoFlight, HiFi, SODE, ChasePlane, Immersion, Orbx, and several others as you can see in the list.

I tried to pin point exactly why using a tool called Process Hacker (ethics aside), it's a VERY good tool at determining who's got what, when, and where even down to file I/O.  But with that said ... I still couldn't positively ID ... here is my diagnostics work:

I've ran about 12 shutdown tests now with WideFS disabled and P3D is shutting down correctly.  May I ask why you suspected WideFS?

Cheers, Rob.

 

 

[Pete Dowson]

4 hours ago, RobAins said:

May I ask why you suspected WideFS?

The log ends at the point where WideFS threads are being closed, which includes efforts to shut down any open listener sockets and outstanding transmissions, AND you earlier reported that it seemed stuck in a network operation.

The only thing it can be is a hanging network driver or API.

I’ve only ever seen this rarely and it’s always been bad network drivers or corruption at lower levels, where it is out of our code’s control.

Check your drivers and maybe Windows updates or other things which may have changed recently.

ALSO I see a GoFlight driver in your stack list. What's that doing there? Have you tried without this?

Note that considering the numbers of WideFS users, have been very few — maybe a handful over the 20 years of WideFS (which hasn’t changed really in all that time -- except dropping support for the old Novell IPX protocol (which was much faster) and adding UDP instead, when MS removed Novell stuff from Windows).

If you are using TCP try UDP, or vice versa, just to see. And don’t forget to check drivers etc at the Client ends.

And check Intel drivers, not just the higher level Windows stuff.

 

Pete

 

 

 

[Pete Dowson]

Also, with WideFS enabled, add this line to the [WideServer] section in FSUIPC5.INI

Log=Debug

That will log quite a lot, but since you have no Clients connecting, not so much as it will sit waiting until you close down.

Please then show us the WideServer.Log file. This is all I have in mine, with FSUIPC 5.153 and with no clients:

********* WideServer.DLL Log [version 7.076] *********
Blocksize guide = 8192 (double allowed)
Date (dmy): 25/01/20, Time 15:02:39.246: Server name is LEFT
      531 WM_MYTIMER2 still okay
    15148 Starting service in 'MYTIMER' call
    15148 Initialising TCP/IP server
    15148 Serving socket obtained okay
    15148 Socket bound to address+port okay
    15148 Listening on socket now ...
    15148 Set to receive Accept message upon connection
    15148 Initialising UDP/IP server
    15148 Serving socket obtained okay
    15148 Socket bound to address+port okay
    15694 WM_MYTIMER2 still okay
    16131 Broadcasting service every 1000 mSecs
    20748 WM_MYTIMER2 still okay
  5312240 Closing down now ...
Memory managed:  Offset records: 2357 alloc, 2355 free
Read buffer usage:  0 alloc, 0 free, max in session: 0
Write buffer usage: 0 alloc, 0 free, max in session: 0
Throughput maximum achieved:  0 frames/sec, 0 bytes/sec
Throughput average achieved for complete session:  0 frames/sec, 0 bytes/sec
********* Log file closed *********

That's with this in the FSUIPC5.INI file:

[WideServer]
WideFSenabled=Yes
AdvertiseService=1
Port=8002
Port2=9002
Log=Debug

There's not a lot shown for the shutdown sequence as there isn't much to it. But I do need to know if your problem is truly just a 5.152 to 5.153 difference, because the source code for WideServer isn't changed. I'd need to check with John about whether the Visual Studio version and maybe libraries used were changed (I'm still on vS2017 but I think John moved on to VS2019).

Pete

 

[John Dowson]

I'm still using VS2017 for FSUIPC5, and the build shouldn't have changed. Worth trying with 5.152 though, but I would be surprised (and worried!) if it was the 5.153 build.

John

[RobAins]

Hi Pete/John,

I'm working on something else right now, but will do some more diagnostics later tonight.  

FYI, I'm using the "Aquantia 5G/10G Ethernet connection Driver for RS5" ... I can switch to old school "Intel(R) Gigabit Ethernet Driver" for testing, but I have been using the "Aquantia" drivers for many months now without issue.

Will report back later with your suggestions.

Cheers, Rob.

[RobAins]

I ran a very quick test flight (1.53) with WideFS enabled and logging (see attached).

I don't get to "Closing down now ..." in WideFS log.  Obviously I have to manually end task after I trigger P3D Exit  (I do wait about 3 mins just to make sure it's not shutting down normally).

The shutdown problem is 100% repeatable when I enable WideFS.

I'll revert to 1.52 later tonight and test again.

Cheers, Rob.

 

WideServer.log FSUIPC5.log

[Pete Dowson]

2 hours ago, RobAins said:

The shutdown problem is 100% repeatable when I enable WideFS.

I feel it is still a low level problem. What was that GoFlight entry doing in the Stack trace for the hung thread?  Have you tested without that drive at all?

Without being able to reproduce it it seems a rather intractable problem. I don't know if there's some other logging John or I can add.  I'll try taking a look when I get time.

Pete

 

[RobAins]

I don't think it's a low level problem as that would surface everywhere not just with WideFS (my PC's and add-on are heavily network reliant).

I'll test without GoFlight entry and see what happens ... but FYI, I've been using the same GoFlight entry/version for years (plural) without issue and the GoFlight entry doesn't cause any shutdown issues when WideFS is disabled.

Is it possible you can create a special build with additional logging points and email to me? 

I'll run the 1.52 version tonight and see if that makes a difference.

But like I said, it's not a big deal for me to just disable WideFS so I have a work-around.

Cheers, Rob.

[RobAins]

UPDATE:

Disable GoFlight entry and GIT entry:

  <Launch.Addon>
    <Disabled>True</Disabled>
    <Name>GoFlight P3D Data Bridge</Name>
    <ManualLoad>False</ManualLoad>
    <Path>D:\GoFlight\GFDevP3Dv4.exe</Path>
    <CommandLine>
    </CommandLine>
  </Launch.Addon>

and GIT entry:

  <Launch.Addon>
    <Name>GoFlight Interface Module</Name>
    <Disabled>True</Disabled>
    <Path>D:\GoFlight Interface Tool\GoFlight Interface Module 64.dll</Path>
  </Launch.Addon>

Unfortunately, same result if WideFS is enabled, P3D doesn't terminate.

Removed FSUIPC v1.53 and installed v1.52 and the same issue is present with v1.52.

Tested the Intel NIC (made changes to accommodate for IP change) and results where the same as the Aquantia NIC, P3D would not shutdown with WideFS enabled.

I understand trying to diagnose this is going to be extremely difficult, so I'm happy to just leave this as a mystery as I really don't mind disabling WideFS since I have nothing that relies on it.  Appreciate the assistance and support.

Cheers, Rob.

[Pete Dowson]

1 hour ago, RobAins said:

Removed FSUIPC v1.53 and installed v1.52 and the same issue is present with v1.52.

Ah, that's more understandable. So, unless you've only recently enabled WideFS, something else in your system has changed or been corrupted.

I still feel it must be a lower level Network function. Just because the network appears to be okay otherwise doesn't mean it isn't. Not all programs have the same sequences or timings.

I'd still like to try to resolve it, so I'll look to se where extra logging might help during the closing process.

Pete

 

[John Dowson]

Hi Rob,

I've added some extra logging in the attached dll - could you try this and post both your FSUIPC5 and WideServer log files:

FSUIPC5.dll

Thanks,

John

[RobAins]

Here ya go gents, attached logs.

TaskMgr termination of P3D was required so take that into account.

Cheers, Rob.

FSUIPC5.log WideServer.log

[John Dowson]

Hi Rob,

could you try the attached please:

FSUIPC5.dll

Thanks,

John

[RobAins]

Hi John,

See attached logs with new DLL.  

I did find something rather interesting.  I run HiFi AS4 from a networked PC. 

Here are the test scenarios and results:

Scenario 1

• WideFS enabled
• HiFi AS4 running from my network PC
• P3D shutdown correctly (all terminates)

Scenario 2

• WideFS enabled
• HiFi AS4 NOT running on my network PC
• P3D does NOT shutdown

Scenario 3

• WideFS disabled
• HiFi AS4 running from my network PC
• P3D shutdown correctly (all terminates)

Scenario 4

• WideFS disabled
• HiFi AS4 NOT running from my network PC
• P3D shutdown correctly (all terminate)

So there seems to be some interaction between WideFS and HiFi AS4 running on a network PC ... is it possible WideFS is looking for some sort of termination event from HiFi AS4 that never happens because it's never started from my networked PC?

Cheers, Rob.

FSUIPC5_HiFi_AS4_CONNECTED.log WideServer_HiFI_AS4_CONNECTED.log FSUIPC5_HiFi_AS4_NOT_Connected.log WideServer_HiFi_AS4_NOT_Connected.log

[Pete Dowson]

16 hours ago, RobAins said:

So there seems to be some interaction between WideFS and HiFi AS4 running on a network PC ... is it possible WideFS is looking for some sort of termination event from HiFi AS4 that never happens because it's never started from my networked PC?

I'm not sure what ASP4 does when running on a client PC. There's an AS DLL which runs internally in P3D4, and the main AS program talks to it. I'm not sure how it does this -- via TCP/IP or UDP on its own socket I suppose. But certainly, if that DLL is running then FSUIPC will connect to functions within it in order to receive assorted weather data and in particular the data for the weather radar file it produces for other add-ons (like ProSim).

Anyway, it's a good clue, so thanks.

Pete

 

[Pete Dowson]

1 hour ago, Pete Dowson said:

Anyway, it's a good clue, so thanks.

Hmmm. I just enabled the ASP4 DLL for installation into P3D4.5 on my test PC, so I cauld run a debug session on it. ASP4 itself isn't running anywhere.

Though WideFS is enabled, and FSUIPC is definitely linking to the ASP4 DLL (I have the log entries
ASN active function link set
Ready for ActiveSky WX radar with additional data
as in your logs), it closes fine.

My version of as_connect.dll is 17.1.1.126 dating from 26/09/2019. Can you check your please?

The other difference is that I am using Win7 still on this PC.

Looking in the as_srv folder, where as_connect is kept, I see the file "bstrp.txt" has this entry in it:

02/22/20 14:37:57 SHOW: Incompatible P3D version. This version of as_connect is designed to be used with P3D versions 4.0.23.21468 up to 4.5.13.32097

I am on the last 4.5 Beta, so maybe this means as_connect isn't doing much, though it is responding to FSUIPC's connection requests.

Maybe there's something different in the as_connect.dll setup when set for using ASP4 on a different PC, though I don't see any configuration file?

Pete

 

  

[RobAins]

4 hours ago, Pete Dowson said:

My version of as_connect.dll is 17.1.1.126 dating from 26/09/2019. Can you check your please?

I have the same version.

4 hours ago, Pete Dowson said:

02/22/20 14:37:57 SHOW: Incompatible P3D version. This version of as_connect is designed to be used with P3D versions 4.0.23.21468 up to 4.5.13.32097

I don't see that in my btstrp.txt?  This is what I have:

For Network setup, SimConnect needs to be installed on the Network PC and there needs to be a SimConnect.cfg defined on the Network PC in the Documents folder: 

The IP address needs to point to the main FS PC.

On the network PC I run AS_P3Dv4.exe which does all the weather process and communication to my main FS PC.

Running this program on the network PC and I get the main HiFi AS weather app: 

The ONLY component installed on my main FS PC is the ASConnect_P3Dv4_Install.exe (this file is copied over to my main FS PC from the Network PC during the AS4 installation on Network PC).

This file is executed on the my main FS PC and will create the following directory/files:

The as_connect_64.dll is referenced in the DLL.XML file on my main FS PC.

I think the KEY here is that WideFS seems to have a reliance on HiFi AS4 actually running on my network PC.  Hope this info helps.

Cheers, Rob.

 

[Pete Dowson]

3 hours ago, RobAins said:

I don't see that in my btstrp.txt?  This is what I have:

That looks like the log for a connected ASP4.  My total btstrp.txt is:

02/22/20 14:37:57 SHOW: Incompatible P3D version. This version of as_connect is designed to be used with P3D versions 4.0.23.21468 up to 4.5.13.32097
02/22/20 14:37:57 SimVersion=-1
02/22/20 14:37:57 p3dBuild=1432669
02/22/20 14:49:43 Dll stop called

but you are not running the latest P3D4.5, which may also explain it. Looks like I'll have to uninstall the Client and go back to the current release.

3 hours ago, RobAins said:

For Network setup, SimConnect needs to be installed on the Network PC and there needs to be a SimConnect.cfg defined on the Network PC in the Documents folder: 

I know all that. It was the remote ASP4 connection to as_connect which I was thinking might have been using a different socket, as i said. I don't need to sort out the true network connection if i am trying to repro your problem which occurs when ASP4 is NOT running. In fact I'm testing with no client PCs switched on in any case.

3 hours ago, RobAins said:

The ONLY component installed on my main FS PC is the ASConnect_P3Dv4_Install.exe (this file is copied over to my main FS PC from the Network PC during the AS4 installation on Network PC).

Okay, that's fine then.

I'll try going back to 4.5.13. I should really have noticed you weren't using the Beta from the FSUIPC log files you've posted.

Pete

[RobAins]

8 hours ago, Pete Dowson said:

I'll try going back to 4.5.13. I should really have noticed you weren't using the Beta from the FSUIPC log files you've posted.

On my main FS PC, I never install any of LM's Beta (too many add-ons would not work if I did).  Only on my development PC do I install Betas. 

Cheers, Rob.

 

 

[Pete Dowson]

1 hour ago, RobAins said:

On my main FS PC, I never install any of LM's Beta (too many add-ons would not work if I did).  Only on my development PC do I install Betas. 

Same here. And as you know, I don't have an operational "main FS PC" at present.  Also, my debugging tools (VS etc) are only on my development PC, so I put the Beta there -- except for those which must have Win10, in which case they are currently on my VFR PC (55" 4K screen).

Pete

 

[John Dowson]

Hi Rob,

I've added further logging in the attached if you could try please:

FSUIPC5.dll

Thanks,

John