performance issues with airport database on fs 2020

[Jason Fayre]

Hello,

We're having some significant performance issues when using the airports database in FS 2020. In P3d, things work fine. This is most likely due to the size of the database in 2020.

If we load the database, people are reporting stuttering audio and general lag in fs 2020. If we don't load it, everythin is fine. Is there any optimization that can be done to the airport database code?

It does seem to be more of an issue on systems that have lower CPU/GPU specs.

 

 

[Paul Henty]

Hi Jason,

I can have a look and see if anything can be done. I don't have MSFS however. Would you be able to zip and attach the MakeRunways files make with MSFS so I can test with those?

Runways.xml
F5.csv
R5.csv
G5.csv
Helipads.csv
T5.csv

Paul

[Andy B.]

Hi Paul,

 

We have a feature in Talking flight monitor that takes the ILS needles on the aircraft and turns them into bits of information the pilot can understand as a blind pilot. For example, the GS needle is announced in terms of feet relative to the aircraft. EX: 500feet up/down. The localizer needle is represented in terms of heading offsets. EX: 60 left/right. In our main process timer, I have the following line of code:

 

// Track the user so we don't have to do it on every request.

FSUIPCConnection.AirportsDatabase.SetReferenceLocation();

 

This line is ran every 250ms, which makes MSFS buffer. OBS log analyzer output tells the typical user that TFM is forcing the GPU into a 15% or larger lag. Is there any other way to constantly track the users location in the world? It is also worth noting that the makerunways output for MSFS is around 800MB in size.

[Paul Henty]

Hi Andy,

So if I understand correctly it's not the actual loading of the database (AirportsDatabase.Load()) that causes the issues, it's repeatedly calling the SetReferenceLocation() with a large airports database.

Quote

Is there any other way to constantly track the users location in the world?

SetReferenceLocation() updates the AirportsDatabase using the current location. It will calculate the distance to the nearby airports, what runway you are on (if you're in the boundary of the airport) etc.

If that's what you need every 250ms then there's no other way to do that.

If you only need to know the players Lon/Lat you can just use the offsets that provide that information.

If you can confirm that SetReferenceLocation() is the problem I will see if I can optimise it. It would be best if I have the large MSFS generated database to work with. If you have an easy way to get it to me (e.g. Google drive/Dropbox link) please send it to me in a private message on here. Otherwise I can maybe fake a larger database with the FSX one.

Also, are you using the AI Traffic services as well?

Paul

[Andy B.]

We can try by lat/long and see if we can calculate the distance/GS. I can dropbox you a link to a zipfile of my makerunways output a little later.

[Jason Fayre]

Hi Paul,

We're only using traffic services to pull a list of nearby aircraft on request. It isn't running on the main timer.

 

[Jason Fayre]

Hi Paul,

I just sent you a dropbox link to the fs2020 airport database in a PM.

 

[Paul Henty]

Got it, thanks.

Looking at the files, the biggest is 500mb (runways.txt) but I don't use that one.

Adding up the files that I do use, your MSFS data is 2.5 times larger than my FSX data (191mb vs 75mb).

I'll have a look this later today and see if anything can be done.

Paul

[Paul Henty]

So after some investigation, the problem is the mainly the MSFS taxiways. They seem to be made up of many more segments than FSX. 

When I wrote the airports database in the FSX days it was no problem loading everything into memory. The database with all taxiways only took about 600mb and the SetReferencePosition() only takes around 70ms on my machine.

With the MSFS data, the entire in-memory database (with Taxiways) is now 2.6Gb. This is quite a large program to be running on the MSFS computer.

The main problem however is that the SetReferencePosition() now takes 350ms. If you're trying to run that every 250ms you can see the problem - it's longer than the timer tick,

I assume you're loading the Taxiways because I remember you having a feature to give call outs of which taxiway the pilot was on. Perhaps you can confirm.

Loading the entire database into memory is just not feasible now with MSFS. It's not just a question of making optimisations, it really needs a rethink of how the database is handled and when things get loaded. 

I'll see if I can come up with a plan over the next few days.  I'll keep you updated.

Paul

[Andy B.]

We were planning on it at some point in time. Unfortunately, things didn't look too good for that feature.

[Paul Henty]

I've had a look at the Load() and SetReferenceLocation() functions. There is no more optimisation that can be done. The problem is just the amount of data it has to load and work through.

The solution to this is to rewrite the airports database to make is possible to only load the airports/data you need or is relevant at the time. This will do two things:

1. Limit the memory taken by your application

2. Make the SetReferenceLocation() much faster because it's not calculating distances to every runway on the planet.

This will mean rewriting your code airports database a little, but the current system is not viable.

I'm going to use a database to store the airport information on the disk in a way that it can be retrieved very quickly. (Parsing the CSV files is slow so this will only need to be done when the MakeRunways files change). A side benefit of this is that your application won't need to go through the lengthy Load() procedure every time it starts up.

This will probably take a few weeks as I also need to update the example code projects.

In the mean time you might be able to improve the situation in your application by not loading data you don't need. For example if you don't use the Taxiway information then turn it off with AirportsDatabase.LoadTaxiways = false. Same with any other data you don't use.

Paul

   

[Paul Henty]

Jason, Andy.

The new version 3.3.0 is now on NuGet with the changes outlined above.

You will need to rewrite some of your Airports Database code to use the new methods. The strategy is to limit the data you load into RAM and the number of airports you process when calling SetReferenceLocation().

Details are in this post:

More detailed examples are available in the Example Code application. Link in the post above.

Sorry you have to change your code, but it just could work the way it was. 

Let me know if you need any help with it.

Paul

 

[Andy B.]

Hi,

 

I tried rebuilding the main routine that loads the database for TFM. Unfortunately, we aren't getting any response from RebuildDatabaseAsync. It seems to get stuck for some reason. I then built and ran the sample code to see if pressing the rebuild database button would work. I ended up getting the unhandled exception window with the following contents. I imagine this is the same problem we are having in TFM. How should I fix this problem?

 

Note: I tried installing protobuf-net in gnuget, but it says it is already installed.

 

See the end of this message for details on invoking 
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.IO.FileNotFoundException: Could not load file or assembly 'protobuf-net, Version=3.0.0.0, Culture=neutral, PublicKeyToken=257b51d87d2e4d67' or one of its dependencies. The system cannot find the file specified.
File name: 'protobuf-net, Version=3.0.0.0, Culture=neutral, PublicKeyToken=257b51d87d2e4d67'
   at FSUIPC.DataStore.WriteList[T](String fileName, IEnumerable`1 ObjectsToStore)
   at FSUIPC.AirportsDatabase.importRunwaysAndAirports(Int32 taskIndex, IProgress`1 progress)
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at FSUIPC.AirportsDatabase.<RebuildDatabaseAsync>d__53.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at FSUIPCExampleCode_CS.AirportDatabase.DB001_ManagingTheDatabase.<btnRebuild_Click>d__0.MoveNext() in C:\Users\Andy Borka\source\repos\fsuipc samples\FSUIPCExampleCode_CS\AirportDatabase\DB001_ManagingTheDatabase.cs:line 71
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

************** Loaded Assemblies **************
mscorlib
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9105.0 built by: NET481REL1LAST_C
    CodeBase: file:///C:/Windows/Microsoft.NET/Framework64/v4.0.30319/mscorlib.dll
----------------------------------------
FSUIPCExampleCode_CS
    Assembly Version: 1.4.0.0
    Win32 Version: 1.4.0.0
    CodeBase: file:///C:/Users/Andy%20Borka/source/repos/fsuipc%20samples/FSUIPCExampleCode_CS/bin/Debug/FSUIPCExampleCode_CS.exe
----------------------------------------
System.Windows.Forms
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9075.0 built by: NET481REL1LAST_C
    CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9065.0 built by: NET481REL1LAST_C
    CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Drawing
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9032.0 built by: NET481REL1
    CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Configuration
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9032.0 built by: NET481REL1
    CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Core
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9115.0 built by: NET481REL1LAST_B
    CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Xml
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9032.0 built by: NET481REL1
    CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
fsuipcClient
    Assembly Version: 3.3.1.401
    Win32 Version: 3.3.1.401
    CodeBase: file:///C:/Users/Andy%20Borka/source/repos/fsuipc%20samples/FSUIPCExampleCode_CS/bin/Debug/fsuipcClient.DLL
----------------------------------------
Newtonsoft.Json
    Assembly Version: 13.0.0.0
    Win32 Version: 13.0.1.25517
    CodeBase: file:///C:/Users/Andy%20Borka/source/repos/fsuipc%20samples/FSUIPCExampleCode_CS/bin/Debug/Newtonsoft.Json.DLL
----------------------------------------
System.Numerics
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9032.0 built by: NET481REL1
    CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Numerics/v4.0_4.0.0.0__b77a5c561934e089/System.Numerics.dll
----------------------------------------
System.Runtime.Serialization
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9032.0 built by: NET481REL1
    CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Runtime.Serialization/v4.0_4.0.0.0__b77a5c561934e089/System.Runtime.Serialization.dll
----------------------------------------
System.Data
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9110.0 built by: NET481REL1LAST_B
    CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_64/System.Data/v4.0_4.0.0.0__b77a5c561934e089/System.Data.dll
----------------------------------------
Accessibility
    Assembly Version: 4.0.0.0
    Win32 Version: 4.8.9032.0 built by: NET481REL1
    CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/Accessibility/v4.0_4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
    <system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

 

[Paul Henty]

Hi Andy,

The example code solutions had the nuget paths messed up because I was using them in a larger project.

I've sorted them out and re-uploaded them. Please download the latest from the website - 1.4.1.

If you can get that to work then we'll look at the problem you're having in your app.

A couple of questions

What version of .NET is your application targeting?

Are you getting any callbacks to the progress class?

Paul 

[Andy B.]

Hi,

 

To answer your questions, We are using .net 4.8 and we aren't using the progress class. Our database loading routine uses tolk to speak the status of database loading.

 

[Paul Henty]

Okay, just tried here in a new 4.8 winforms app and works okay:

        private async void button1_Click(object sender, EventArgs e)
        {
            FSUIPCConnection.Open();
            AirportsDatabase db = FSUIPCConnection.AirportsDatabase;
            await db.RebuildDatabaseAsync();
            MessageBox.Show("Done");
        }

Quote

we aren't getting any response from RebuildDatabaseAsync

Are you using await? If not, it will just start the rebuild on a background thread and your code will just continue.

Paul

[Andy B.]

Hi,

 

Your sample worked and the following code will load the database once built. Unfortunately, it doesn't rebuild it for the first time.

 

public static async void LoadAirportsDatabase()
        {

            if (FSUIPCConnection.IsOpen)
            {
                AirportsDatabase database = FSUIPCConnection.AirportsDatabase;
                database.MakeRunwaysFolder = Properties.Settings.Default.P3DAirportsDatabasePath;
                
                if(database.MakeRunwaysFilesExist && database.DatabaseFilesExist == false)
                {
                    await database.RebuildDatabaseAsync();
                                                        }
                else if (database.DatabaseFilesExist)
                {
                                        database.Load();
                }

                if (database.IsLoaded)
                {
                    Tolk.Output("Airports database loaded.");
                }
                                                            } // open connection.
        } // LoadAirportsDatabase
 

[Paul Henty]

It takes about 23 seconds on my machine to rebuild the MSFS database (from the files Jason sent).

Can you add a progress to see if you get anything back, and also check your "%localappdata%\FSUIPCClientDLL" folder to see if there are any files there. They will be under a folder with a long hex string.

Paul

[Andy B.]

I got rebuild to work. The startup routine now works - lets you know if it failed and to see the logs for more, or that it successfully loaded. I also have rewritten a few database features and am getting used to how the new layout works. Thanks for the help and rewrite!