Illegal char in Runways.xml - MkRwy 5.11

[Pete Dowson]

I looked up Windows-1252. On this page there's a conversion table into UTF-8. That seems to show no useful conversions for the whole range 0x80-0xFF.

http://string-functions.com/encodingtable.aspx?encoding=1252&decoding=65001

So, are there more Escape codes in XML to be used instead?

==========================

Maybe Luke can revise that function I posted earlier to work as he thinks it should?

Pete

 

[pellelil]

The escaping of char in XML you have implemented appears (to me) to be just fine, so the only issue is the encoding of wide-chars. There are currently chars in the XML in the range 0x80 to 0xFF that works just fine, so not sure why these 2 chars are causing an issue

[Pete Dowson]

2 hours ago, pellelil said:

the only issue is the encoding of wide-chars. There are currently chars in the XML in the range 0x80 to 0xFF that works just fine, so not sure why these 2 chars are causing an issue

So, shall I just deal with those for now? i.e 0xA1 followed by 0xF6?

Pete

 

[pellelil]

That's a very good question Pete 🙂

It's a two-edged sword. If you only look for 0xA1 followed by 0xF6, chances are there will be other byte-combinations that might break it, and if using my suggested hack, you might end up replacing chars that are just fine.

As I am apparently the only one who have seen this issue, and so far only with 0xA1 followed by 0xF6 (for this one/single airport), it might be a good idea to only look for, and replace this particular combination.

[Pete Dowson]

2 minutes ago, pellelil said:

As I am apparently the only one who have seen this issue, and so far only with 0xA1 followed by 0xF6 (for this one/single airport), it might be a good idea to only look for, and replace this particular combination.

Wilco!

Pete

 

[pellelil]

Thanks Pete, sorry for the emotional roller coaster most likely caused by a "weird" value in a BGL 😉

 

[Luke Kolin]

4 hours ago, Pete Dowson said:

Sorry, I'm not learning any new programming methods at my age. I am retired, and this isn't even a hobby now, it's a favour. If you want to re-program it all I'll send you the source.

If I can load it into VS2019, I'd be willing to look at it. My point is that your challenge is upstream, when you load the strings out of the BGL. I suspect (being optimistic) it might be enough to either load the strings as wchar_t* instead of char* (or cast) and then have your encoding function that you shared here operate on *wchar_t instead of *char. The challenge is going to be how you read from the BGLs, but I suspect it's all bytes and the cast should be OK.

If you want a low-effort experiment, change the function signature of your encoding function to take wchar_t* instead of char*, cast the pointers before you call it and see if it works.

Cheers

Luke

[pellelil]

4 minutes ago, Luke Kolin said:

If I can load it into VS2019, I'd be willing to look at it. My point is that your challenge is upstream, when you load the strings out of the BGL. I suspect (being optimistic) it might be enough to either load the strings as wchar_t* instead of char* (or cast) and then have your encoding function that you shared here operate on *wchar_t instead of *char. The challenge is going to be how you read from the BGLs, but I suspect it's all bytes and the cast should be OK.

If you want a low-effort experiment, change the function signature of your encoding function to take wchar_t* instead of char*, cast the pointers before you call it and see if it works.

Cheers

Luke

Depending on how much code is reused and how "old" BGLs were encoded, switching to wchar might break things for FSX/P3D !?

[Luke Kolin]

19 minutes ago, pellelil said:

Depending on how much code is reused and how "old" BGLs were encoded, switching to wchar might break things for FSX/P3D !?

There are four areas I recognize I am ignorant in and leave it to the experts - thread synchronization, dates/times, encryption and character encoding. 😄

From what little I know, UTF-8 and UTF-16 should be a superset of ASCII, so regular ASCII characters should be encoded the same way in all three. It should a simple modification to change the signature for that function and any comparisons it does, then see what happens. I'd be slightly surprised if that's all it took to make it work, but even more surprised if that broke the ASCII case.

YMMV, of course.

Cheers

Luke

[Pete Dowson]

Pelle,

I can't test the change, but I attach MakeRwys 5.121 for you to test.

The reason I can't test the change is because my EDLP scenery appears different to yours. I don't know why. I have a fully-updated MSFS installation, with Japan, USA and UK updates installed, but no other add-on scenery.

From your first post, your EDLP is

<ICAO id="EDLP">
<ICAOName>Paderborn-Lippstadt</ICAOName>
<Country>Germany</Country>
<State>North Rhine-Westphalia</State>
<City>B    ören</City>
<File>G:\FS\2020Packages\Official\OneStore\fs-base\scenery\0601\APX50130.bgl</File>
<SceneryName>fs-base scenery 0601</SceneryName>

with that weird City name, whereas mine is:

<ICAO id="EDLP">
<ICAOName>Paderborn/Lippstadt</ICAOName>
<Country></Country>
<City>Paderborn/Lippstadt</City>
<File>C:\Users\Pete\AppData\Local\Packages\Microsoft.FlightSimulator_8wekyb3d8bbwe\LocalCache\Packages\Official\OneStore\fs-base\scenery\0601\APX50130.bgl</File>
<SceneryName>fs-base scenery 0601</SceneryName>

So -- same BGL file, but different result. Are you sure yours in not corrupted? That might well explain why no one else has complained about Runways.xml.

For interest I also installed Aerosoft's free Paderborn. That gives me:

<ICAO id="EDLP">
<ICAOName>Paderborn/Lippstadt</ICAOName>
<Country></Country>
<City>Paderborn/Lippstadt</City>
<File>C:\Users\Pete\AppData\Local\Packages\Microsoft.FlightSimulator_8wekyb3d8bbwe\LocalCache\Packages\Community\aerosoft-paderborn\scenery\global\scenery\EDLP_Placement.bgl</File>
<SceneryName>aerosoft-paderborn</SceneryName>

Assuming it is a corruption, before you do try a correction please test 5.121 on the corrupt BGL for me.

Mind you, if MakeRwys is to deal with all corruptions in strings, maybe I ought to make the change more severe after all. Or just remove it (discard this update)???

Pete

 

MakeRwys.exe

[pellelil]

Something weird is going on. Now with the new version it instead is <City>B÷ren</City> shown in Notepad++ x87xF7

I'll try to remove Paderborn and see when it scans then

[pellelil]

Pete I tried removing the free Paderborn and then I did a new scan, and the city-name was just fine. Then I started the sim again, installed Paderborn and closed the sim. Now when I scan I see the city-name as <City>BPören</City> (where the letter after "P" is shown as xF6 in Notepad++. The only "special" thing I do is I am calling MakeRwys via a link where I have the command-line argument: "/>1000"

EDIT: Executing MakeRwys directly (without the argument) does not change anything. The city-name from Paderborn still containx xF6 when viewed in NotePad++

However now it loads as XML just fine, eventhough Notepad++ still shows xF6 ???

[Pete Dowson]

5 hours ago, pellelil said:

Pete I tried removing the free Paderborn and then I did a new scan, and the city-name was just fine.

But in your original post it wasn't the Free PaderBorn from Aerosoft, but the default:

<ICAO id="EDLP">
<ICAOName>Paderborn-Lippstadt</ICAOName>
<Country>Germany</Country>
<State>North Rhine-Westphalia</State>
<City>B    ören</City>
<File>G:\FS\2020Packages\Official\OneStore\fs-base\scenery\0601\APX50130.bgl</File>
<SceneryName>fs-base scenery 0601</SceneryName>

which is the same file which reads fine here. Is your MSFS not up to date?

I posted the entries for yours, mine with default, and mine with the Aerosoft free Paderborn just a couple of message back. Please check.

Also, the code I added was exactly as eventually decided upon:

if ((*pszFrom == 0xA1) && (*(pszFrom + 1) == 0xF6))
{    strcpy(pszNow, "#");
     pszNow++;
     pszFrom++;
 }

 i.e. just replacing A1 followed by F6 by #.  (pszFrom is incremented by 2 because there's one for the whole loop already following as you'll see from the function I posted before).      

My APX50130.bgl is dated 11/03/2021 23:19 and is 1,952,960 bytes is size.

Sorry, I'm not sure what's going on with your system, nor why the aerosfot one isn't showing up for you. Are you sure it's installed correctrly?

Pete

 

[pellelil]

I'll have a look, but I am pretty sure the file is listed "wrong" in the runways.xml (or at least not what I would expect). I noticed this for another 3rd party airport as well. My MSFS (Premium Delux) is fully updated, all content downloaded, I currently have nothing in the community folder, and the only 3rd party scenery installed is the free Paderborn along with VidanDesign EKBI and FlyTampa EKCH.

I can't be sure if the Paderborn is installed "correctly" but had it not been originally, it should be been fixed yesterday when I first removed (shut down the sim and performed a scan with MkRwy), and then starting the sim again, and installed Paderborn once again. When not installed the city is shown as "Paderborn/Lippstadt", but once installed I see this wrong city-name.

I have NO IDEA why Paderborn is causing this issue (perhaps only on my system). Its the only airport I have seen with a weird value, and only in the city-name.

[Pete Dowson]

58 minutes ago, pellelil said:

My MSFS (Premium Delux) is fully updated, all content downloaded, I currently have nothing in the community folder, and the only 3rd party scenery installed is the free Paderborn along with VidanDesign EKBI and FlyTampa EKCH.

Well, those addons go in the Community folder. That's how I see Paderborn, anyway. I thought all add-on scenery and other add-ons, even John's new L:Var module) went there. Only the official release from Asobo goes into the Official\OneStore folder. Where is your Paderborn?

MakeRwys produces a SceneryList.txt file (it's own constructed version of the Scenery.cfg file, replacing the one it uses in P3D generated by the Lorbi-si program).  The scenery is processed in the order listed there -- all the fs-base stuff first, then the fs-base-nav files to add the frequency data, then all the other airport files in the Official\OneStore folder, and lastly all those in the Community folder in the order of the directory.

Please look at that file.

It seems there's certainly something really odd going on on your system.

Pete

 

[pellelil]

What I meant about community-folder is the folder "G:\FS\2020Packages\Community" into where I in the past have installed freeware content in form of scenery and community driven updates for the default aircraft (e.g. the work done by "Working Title" updating the CRJ4 and the Garmin 1000/3000). This is all content that cannot be installed through the MSFS-markedplace, hence its more easy to control having a separate folder for it ... but as I said this folder is currently empty.

The first two lines of SceneryList.txt contains the "link" to the Paderborn scenery:
001    Active    aerosoft-paderborn scenery
    G:\FS\2020Packages\Official\OneStore\aerosoft-paderborn\scenery\global\scenery

But Runways.xml list the the file as this:
<ICAO id="EDLP">
<ICAOName>Paderborn-Lippstadt</ICAOName>
<Country>Germany</Country>
<State>North Rhine-Westphalia</State>
<City>B+ören</City>
<File>G:\FS\2020Packages\Official\OneStore\fs-base\scenery\0601\APX50130.bgl</File>
 

In SceneryList.txt I can't see "links" to be 3rd party EKBI nor EKCH, and these are also listed with a default BGL-file in runways.xml
<ICAO id="EKBI">
<ICAOName>Billund</ICAOName>
<Country></Country>
<City>Billund</City>
<File>G:\FS\2020Packages\Official\OneStore\fs-base\scenery\0601\APX50120.bgl</File>

and:
<ICAO id="EKCH">
<ICAOName>Kastrup</ICAOName>
<Country></Country>
<City>Copenhagen</City>
<File>G:\FS\2020Packages\Official\OneStore\fs-base\scenery\0601\APX51120.bgl</File>

All these 3 airports works just fine in the sim. I can see that my EKCH (installed via the MSFS markedplace) have been installed into the folder: "G:\FS\2020Packages\Official\OneStore\flytampa-copenhagen" and my EKBI have been installed into: "G:\FS\2020Packages\Official\OneStore\vidandesign-billund".

[Pete Dowson]

25 minutes ago, pellelil said:

The first two lines of SceneryList.txt contains the "link" to the Paderborn scenery:
001    Active    aerosoft-paderborn scenery
    G:\FS\2020Packages\Official\OneStore\aerosoft-paderborn\scenery\global\scenery

How on Earth does it get there? It isn’t official scenery? 

Being processed before the Base scenery means it is overridden by that. The MakeRwys output will have never used it. It is your base scenery which is corrupted and producing those odd characters, not the addon.

There’s no way MakeRwys can determine the proper processing order with files being installed in different folders seemingly at random.

Maybe you can show me the Scenery list file please. I’ve no idea what’s going on.

Pete

 

 

 

[pellelil]

I just installed it through the markedplace, don't know how/why MSFS did what it did. You mean the SceneryList.txt?

http://www.liljendal.dk/portals/1/files/tmp2021/SceneryList.txt

 

[Pete Dowson]

5 minutes ago, pellelil said:

I just installed it through the markedplace, don't know how/why MSFS did what it did. You mean the SceneryList.txt?

http://www.liljendal.dk/portals/1/files/tmp2021/SceneryList.txt

 

This is awful! Why bothering with a Community folder if they install marketplace addons in the “official” folders?

I could change MakeRwys to make sure all of the Fs-base stuff was processed first, but what on Earth has happened to your other two addons?

MakeRwys merely enumerates all folders in each of those two main folders — Official\Onestore and Community. It uses the regular FindFirstFile and FindNextFile, so how can they be missed?

Sorry. I don’t know how to proceed. 
And what about your original problem? File corruption or what? Have you check or repaired?

Pete

 

[pellelil]

I've just started the sim up at Paderborn, The Aerosoft scenery looks fine ... don't see any default scenery bleed through or anything (that should indicate it have a lesser priority than the default). I then loaded up the aircraft at EKCH, and it is indeed the FlyTampa EKCH I see in the sim, even though it is lon at all listed in the SceneryList file ???

[Pete Dowson]

So how the heck can I find them?

 

[pellelil]

All my 3rd party scenery are in the same folder (or subfolders to this), so if it can find one it should find them all. Strange that MkRwy finds 1 (the Aerosoft) but not the 2 others (all 3 are in sub-folders to "G:\FS\2020Packages\Official\OneStore").

[Pete Dowson]

Very strange, and I don’t understand how it could omit those folders in the enumeration using the regular Windows functions. Are you sure they have normal properties like the others?

Are they freeware like Paderborn? If so I could try them here.

Making sure I scan fs-base first is easy. Finding invisible folders a lot less so.

[pellelil]

I becomes more and more weird. The two sub-folders containing EKCH and EKBI don't contain any .BGL files (hence probably why MkRwy don't have these) ??? They contain a lot of texture files and json files, but just no .BGL

EDIT: BOTH EKBI and EKCH are payware

[pellelil]

Pete I wrote the author of the EKBI scenery, telling him you and I were pulling out the last of our hairs, trying to work out this issue. He told me that scenery installed through the marketplace gets encrypted, and that is why MkRwy can find/read it