FSUIPC LUA scripting - UDP multicast

[chris178]

Hi All,

I'm having an issue setting up LUA script for UDP multicast communication.

The idea behind udp multicast implementation is that I could get independent from client/server machine's IP addresses and instead use pre-defined multicast addresses. This should save hassle setting correct FS destination IP address on all devices talking to FS lua multicast listener over multicast.

Here is the relevant part of my multicast listening script (instance that should join specified UDP multicast group and process messages sent to that group):

host = "230.1.1.1"
port = "7001"
udp = socket.udp()
udp:setpeername(host, port)
print(udp:setoption("ip-add-membership" , { multiaddr = "230.1.1.1", interface = "*"}))

IP traffic analysed with wireshark revealed that script isn't sending multicast join group message (to the network/switch), so no multicast traffic is distributed from the switch to the FS machine running above LUA script. There would be some other solutions on the switch setup to make it flood all ports with multicast data, but it would be very interesting to find out why above "join" command doesn't work?

Console debug window shows issue with multicast join command (udp:setoption("ip-add-membership" , { multiaddr = "230.1.1.1", interface = "*"})):

debug info is:  nil    "setsockopt failed"

I'm running Windows 7 64 Pro.

Would this be some sort of bug in lua's socket library not "talking" correctly to windows tcpip stack or something else?

Any advice would be much appreciated.

 

Chris

[Pete Dowson]

I'm afraid I don't know Lua's socket library well enough to help here. Not only that, I don't really know a lot about sockets and don't understand much of your post.

The stuff built into FSUIPC, augmented by the separate parts provided, is all directly from the Lua stuff freely available. If you can get it working independently, with the Lua run-time setup, then it should work identically in FSUIPC.

FSUIPC's WideServer does have an optional Broadcast mode with UDP for transmission of data to multiple clients, useful when most want the same data in any case, but I've looked at that code and I don't really understand it now even though I did write it myself. Most of my Network code is derived or even straight copied from MS examples. I think I'm also getting senile. :-(

The other broadcasting used by WideServer to broadcast its name, ip Address and protocol uses the Windows "MailShot" facility, which is easier to understand and works well.

Pete

 

[chris178]

Thanks Pete for quick response - I'm quite amazed.

To be honest - I don't know much about WideServer, but my idea is to use linux OS in device communicating with FS.

With regard to my original post - I've run a test on my ubuntu 14.04 machine and LUA 5.2 environment running below exact code:

local socket = require("socket")
host = "230.1.1.1"
port = "7001"
udp = socket.udp()
udp:setsockname(host, port)
print(udp:setoption("ip-add-membership" , { multiaddr = "230.1.1.1", interface = "*"}))

while 1 do
	data, ip, port = udp:receivefrom()
	if data then
		print("RX: ", data)
	end
	socket.sleep(0.01)
end

print(udp:setoption("ip-add-membership" , { multiaddr = "230.1.1.1", interface = "*"})) returns just "1" in console and I can see in network analyser "membership Report / Join group 230.1.1.1 for any source" packet sent via NIC.

Exactly same code run through FSUIPC on Win7 64 bit machine reports for above line "nil    setsockopt failed".

 

Perhaps older/different version of LuaSocket library used in FSUIPC? I'm obviously just guessing here. But it looks like lua not able to send/trigger right thing in Win7 tcpip stack. This is my guess.

I don't know if anything can be done on this from your point of view, but if not I'm going to turn back and use unicasts (that use normal arp/routing tables in the OS).

Thanks any way Pete.

 

Best regards

Chris

[Pete Dowson]

Quote

Perhaps older/different version of LuaSocket library used in FSUIPC? I'm obviously just guessing here. But it looks like lua not able to send/trigger right thing in Win7 tcpip stack. This is my guess.

Details are in the Lua PDF in FSUIPC Documents. To quote:

In addition to the built-in libraries, full LuaSockets support has been included, with all the major modules also built
in. This is a package by Diego Nehab, and thanks are due to him. For reference data and the full package, go to
http://www.tecgraf.puc-rio.br/~diego/professional/luasocket/

I don't know if that's been updated since I first added Lua (towards the end of 2007 -- at the time the Lua version was 5.1 and that is what is built in as well -- as also stated in the documents..

Maybe, if you need the later one you can use the modules externally? You might need to rename them from "Sockets" as else the "Require" for sockets will no doubt choose the one already loaded.

[LATER]

I've just looked at the website linked above, and it seems to still be the same version there, quote " LuaSocket version 2.0.2 is now available for download! It is compatible with Lua 5.1". The authors copyright date above is 2004-2007.

I see the current version of Lua on the main Lua site is 5.3.2.  I'd rather stick with 5.1 because most of the available books for Lua are 5.1 oriented, and it;s a LOT of error-prone work building the stuff into FSUIPC so it works smoothly in that threaded environment

Pete

 

[chris178]

Hi Pete

It looks like Diego Nehab has moved development of the luasocket to https://github.com/diegonehab/luasocket.

The latest version seems to be 3.0-rc1. And it looks like my ubuntu lua package is installed with the latest luasocket.

This could be reason why mcast join works in my linux environment, but of course there could be some other reasons.

Thanks for suggestion using external library capacity - I'll give it a try with the latest luasocket from github (renamed).

Best regards

[Pete Dowson]

28 minutes ago, chris178 said:

It looks like Diego Nehab has moved development of the luasocket to https://github.com/diegonehab/luasocket.
The latest version seems to be 3.0-rc1.

Thanks for the information. I note that the very first thing on that page is a link back to the one I'm using:

"Network support for the Lua language http://www.impa.br/~diego/software/luasocket"

Good luck!

Pete

 

[chris178]

Hi Pete,

Tried to use latest luasocket library but without success... I'm bit puzzled.

That's what I did:

1. Created lua dir in FSX modules directory.

2. Layout as follows (following luasocket installation manual):

Directory of C:\Program Files (x86)\Microsoft Games\Microsoft Flight Simulator X\Modules\lua

27/06/2008  11:18             8,469 ltn12.lua
01/06/2016  23:10    <DIR>          mime
01/06/2016  23:14             2,521 mime.lua
01/06/2016  23:18    <DIR>          socket3
01/06/2016  23:15             5,012 socket3.lua

Directory of C:\Program Files (x86)\Microsoft Games\Microsoft Flight Simulator X\Modules\lua\mime
15/10/2007  18:08            12,800 core.dll

Directory of C:\Program Files (x86)\Microsoft Games\Microsoft Flight Simulator X\Modules\lua\socket3

15/10/2007  18:08            29,184 core.dll
26/05/2016  11:20             9,243 ftp.lua
26/05/2016  11:20             3,698 headers.lua
26/05/2016  11:20            12,330 http.lua
26/05/2016  11:20             8,074 smtp.lua
26/05/2016  11:20             3,612 tp.lua
26/05/2016  11:20            11,036 url.lua               

3. As you see modified lib name to socket3 and relevant dir/files.

4. It also make sense to modify a line in socket3.lua :

local socket = require("socket3.core")

This unfortunately causes error message in debug window:

  4308389 *** LUA Error: error loading module 'socket3.core' from file 'C:\Program Files (x86)\Microsoft Games\Microsoft Flight Simulator X\modules\lua\socket3\core.dll':
        The specified module could not be found.

Although the file core.dll in fact exits there (taken from lua 5.1.4 clibs installation folder). Perhaps not matching exactly LUA version built into FSUIPC?

5. If I leave socket3.lua unmodified ( local socket = require("socket.core") ), then all seems to be running but my impression is that build in luasocket is run and my test script:

socket = require("socket3")
print(socket._VERSION)

returns:  5177284 LUA.0: LuaSocket 2.0.2

 

Unfortunately I'm not software engineer and my experience here is very limited.

Any ideas what I'm doing wrong?

 

BTW: Here is a bit from NEW file of recent luasocket package:

What's New

Main changes for LuaSocket 3.0-rc1 are IPv6 support and Lua 5.2 compatibility. 

    * Added: Compatible with Lua 5.2
        - Note that unless you define LUA_COMPAT_MODULE, package
          tables will not be exported as globals! 
    * Added: IPv6 support;
        - Socket.connect and socket.bind support IPv6 addresses;
        - Getpeername and getsockname support IPv6 addresses, and
          return the socket family as a third value;
        - URL module updated to support IPv6 host names;
        - New socket.tcp6 and socket.udp6 functions;
        - New socket.dns.getaddrinfo and socket.dns.getnameinfo functions; 
    * Added: getoption method;
    * Fixed: url.unescape was returning additional values;
    * Fixed: mime.qp, mime.unqp, mime.b64, and mime.unb64 could
      mistaking their own stack slots for functions arguments;
    * Fixed: Receiving zero-length datagram is now possible;
    * Improved: Hidden all internal library symbols;
    * Improved: Better error messages;
    * Improved: Better documentation of socket options. 
    * Fixed: manual sample of HTTP authentication now uses correct
      "authorization" header (Alexandre Ittner);
    * Fixed: failure on bind() was destroying the socket (Sam Roberts); 
    * Fixed: receive() returns immediatelly if prefix can satisfy
      bytes requested (M Joonas Pihlaja);
    * Fixed: multicast didn't work on Windows, or anywhere

....

so obviously I'm eager to get this lib working ... but perhaps there are unavoidable conflicts with build-in luasocket library?

Best regards

[Pete Dowson]

23 minutes ago, chris178 said:

This unfortunately causes error message in debug window:

  4308389 *** LUA Error: error loading module 'socket3.core' from file 'C:\Program Files (x86)\Microsoft Games\Microsoft Flight Simulator X\modules\lua\socket3\core.dll':
        The specified module could not be found.

Although the file core.dll in fact exits there (taken from lua 5.1.4 clibs installation folder). Perhaps not matching exactly LUA version built into FSUIPC?

This business of modules (DLLs) which are obviously present not being loaded with the error "not found" is annoying. It took me ages to find out what the problem was with my SimConnectP3D2.DLL when used with P3Dv3, but only on PCs which had never had previous versions nor FSX installed.

It was because of dependencies in the DLL on libraries which were not installed. The "not found" message appears to mean not only "this file isn't found" but also "though this file is found it cannot be loaded because it is dependent on other files not found".

There is a DEPENDS utility (http://dependencywalker.com/) which helped me work this out. Maybe you could check?

I got around it for my modules by compiling a new one using the P3D3 SimConnect.lib which was dependent on the same libraries as P3D3 itself. If this is the problem your new sockets module has, maybe the source is available and it can be recompiled with a different library set? Or, maybe easier, install the needed VS run-times.

FSUIPC, and its built-in Lua parts, is not dependent on any external libraries other than the standard Windows ones. I made sure of that long ago so that I could have one version which applied to FSX, FSX-SE (again different libraries), and all the P3D versions (so far -- obviously a new version would be required for any future 64-bit one).

These changes happened because of moves through from VS2005, VS2008, VS2010 and (so far) VS2013. I expect they'll be using VS2015 soon. Each of these has different language libraries.

If this isn't the problem I'm afraid I'm lost. As I said earlier, sockets are a puzzle to me in any case.

Pete

[chris178]

Thanks Pete for update. This all seems to be a little bit too much for my knowledge - i feel like "trying to fix a space rocket being only a plumber" :)

I've just taken a crash course on command line compiling with windows (quite frustrating). Tool I was hoping was going to help me - luarocks uses CLI cl.exe.

At the end of the task (and many problems with header and library files... from both VC and win SDK) I've got brand new core.dll for socket and mime modules (mime might not be needed...).

I've installed quickly VS Express 2008 for this task - and reading your above explanation - this could have been a bad choice.

There is good and bad news.

1. Good news - command line stand-alone lua (5.1.4) interpreter on my Win7 machine now reports latest luasocket version (LuaSocket 3.0-rc1). Sample mclistener.lua works well joining mcast group.

2. Running luasocket version test script with FSUIPC (new core.dll in place) now returns:

  1112318 *** LUA Error: error loading module 'socket3.core' from file 'C:\Program Files (x86)\Microsoft Games\Microsoft Flight Simulator X\modules\lua\socket3\core.dll':
        The specified procedure could not be found.

 

Different Visual Studio, libraries, etc?

Best regards

Chris

[Pete Dowson]

 

Quote

 

1. Good news - command line stand-alone lua (5.1.4) interpreter on my Win7 machine now reports latest luasocket version (LuaSocket 3.0-rc1). Sample mclistener.lua works well joining mcast group.

2. Running luasocket version test script with FSUIPC (new core.dll in place) now returns:

  1112318 *** LUA Error: error loading module 'socket3.core' from file 'C:\Program Files (x86)\Microsoft Games\Microsoft Flight Simulator X\modules\lua\socket3\core.dll':
        The specified procedure could not be found.

 

Different Visual Studio, libraries, etc?

Seems that the dependency issue to removed at least.  But the problem now won't be VS libraries. The error it refers to is a procedure the Lua 5.1. version I have built into FSUIPC-needs to call not being found, or not exported, within the module. 

Now this is not done by any code I understand. The loading of external modules seems to use one of a variety of loaders, being called via very dense Lua code which is beyond me. However, I think the gist of it will be that the module you've managed to build is not loadable by any of the loaders built into the version of 5.1.I used. Possibly a difference between 5.1 and 5.1.4? I don't know. The package I used dates from not long after 5.1 was first released in 2006. When was 5.1.4 released? 

FSUIPC's Lua is able to load external modules successfully -- there are other examples which work fine, and folks have managed to build and compile their own okay. So I'm lost here really. If you'd like to ZIP the module you've built and send it to me at petedowson@btconnect.com I could take a look with a Debugger and work out why it is wrong, but I cannot guarantee to fix it.

Pete

 

 

[Pete Dowson]

42 minutes ago, Mario DANTAS said:

Is it possible to integrate the latest lua library into FSUIPC ?

Only with a huge amount of work and very likely introducing lots of errors. It was at least a 4-6 week job back then, plus weeks of testing and debugging, and I was younger and more intelligent back then. I'm 73 and have now forgotten it all and getting more stupid daily, so would effectively have to start again from scratch. I'm really not willing to risk the stability of FSUIPC and its future to do it I'm afraid.

The whole reason for implementing Lua into FSUIPC at the time was so I can avoid major programming in a future where I knew I would not want to get into such nitty gritty, allowing others to implement new features.

I think it would be quicker and more reliable for you to learn more, and take the source of the module you need and rework it to fit with FSUIPC's Lua module interface. You have a good start as you seem to understand socket programming, which is more than I do.

Quote

Or let user install it aside and make FSUIPC detects if lua is installed in the system ? (some kink of registry key or search in %WINDIR%/windows32/) ?

I mean if FSUIPC detects lua librarries installed it loads them, if not, it loads the embedded ones ?

But this is exactly what your Require is doing, trying to load the library you have. It is exactly what happens with other external libraries, sch as saitek.dll for Saitek devices, gd.dll for graphics, luacom.dll for COM component access. It should do the same for your renamed socket DLL, but there's a linkage problem, the export needed to get the module to provide it's list of contained functions is not being seen or is declared differently.

I did offer to look at it, to see if I can identify why.

Pete

 

 

[Pete Dowson]

18 hours ago, chris178 said:

Although the file core.dll in fact exits there (taken from lua 5.1.4 clibs installation folder). Perhaps not matching exactly LUA version built into FSUIPC?

Thanks for the files. I thought I really only needed the DLL module, so I could check the export, and it does of course contain and export "luaopen_socket_core", so  the required function "luaopen_socket3_core" cannot be found -- which is what your error is saying.

So, it really needs recompiling with that name changed. The same probably applies to mime as well, though I noticed you've been inconsistent renaming mime to mime3 in some places but not others. You'll need to check the Lua sources too as well as folder and module names.

In order to run a test, without recompiling, I patched the name "luaopen_socket_core" in the socket "core.dll" to "sockeu" (I can path a one byte change easier, hense not socket3), and renamed it all sockeu throughout.

I did the same for the core.dll in mime, changing it to mimd throughout. I see ltn12 is also used, but there was no core.dll for that so i had to leave it be.

 That allows it to be loaded okay and the socket.lua (now of course sockeu.lua) runs as far as line 14, whis is:

local socket = require("sockeu")

another require call for the same module. Not sure why this arises here, but I suspect you understand this stuff more than I do. The error logged is:

  2171190 *** LUA Error: E:\Steam\steamapps\common\FSX\modules\lua\sockeu.lua:14: loop or previous error loading module 'sockeu'
 

Which is an error I don't really understand. However, I found more references to socket or mime and changed those too, and now I get this error:

  3684026 *** LUA Error: error loading module 'sockeu.core' from file 'E:\Steam\steamapps\common\FSX\modules\lua\sockeu\core.dll':
    %1 is not a valid Win32 application.

Now, the %1 here should have been replaced by the relevant filename, so there's something peculiar about the innards of Lua itself there. So I run PROCMON to find what fie it was trying to access, and it wants lua5.1.dll,  In fact the compiled core.dll imports all of its lua functions from lua5.1.dll. Probably the two other core.dll's do the same.

I did try including lua5.1.dll in the Lua subfolder, but got the same error. I notice it needs the VS library MSVCR80.dll whereas the modules you've built use MSVCR90.dll. from a later version (2010?) of MS. It looks like Lua5.1.dll was built with MSVC 2005, which makes sense given its age. You used, what, MSVS 2008? Maybe you need to build with the same version of Lua 5.1?

Maybe, doing all of the above, you can sort things out. If not, I wonder if you can seek assistance via the Lua website. There must be some user Forum over there? I've about exhausted things on my side I'm afraid.

Pete

 

[Pete Dowson]

One more look at this from me.

Do you know which part or parts of the Lua sockets programming has the bug stopping multicast working correctly?

I had a thought. If I could narrow down the problem to a specific part of the built-in sockets code then maybe I could fix your original problem -- instead of trying to upgrade my whole implementation of Lua and sockets to a later version.

But there's a lot of it, and I see a lot of changes listed apart from the support for lua 5.2 from 5.1, so locating the area to fix without any real knowledge of sockets or the problem is well nigh impossible. So any clues you have might help. Is it likely to be contained just in  that one call:

udp:setoption("ip-add-membership" , { multiaddr = "230.1.1.1", interface = "*"})

so if that didn't fail, it would work?

I'm going to start by adding the error number returned by that to the "setsockopt failed" message.

[LATER]

The error is 10042, which says:

WSAENOPROTOOPT 10042

Bad protocol option. An unknown, invalid or unsupported option or level was specified in agetsockopt or setsockopt call.

The parameters being sent to the setsockopt function are:

socket id - ok
level = 0,  This I don't understand. Shouldn't it be 0xFFFF for "SOL_SOCKET"?
optname = 5. I think this means SO_REUSEADDR and SO_DEBUG, as the options are bit-oriented.
optval points to 8 bytes { 0xC0, 0XA8, 0x00, 0xAA, 0x00, 0x00, 0x00, 0x00 }
optlen = 8

This doesn't really make sense to me. What would you expect the "setsockopt" function to be doing, for instance if you were programming this externally not in Lua?

Pete

 

Pete

 

[Pete Dowson]

One more thing I thought I might try. I don't hold out much hope, but i thought if I compared the source of the LuaSockets component files I used with those for the later version, I might be able to see what I need to change just that way -- in the hope that none are a complete re-write.

However, following the link you posted much earlier takes me to a directory structure which, yes, does have a "src" subfolder witrh source data, but they are all just text files on scren. I can't find a package to download the complete LuaSockets source as you can with the version I used.

So, how did you get them to recompile? Did you copy every one individually to your own files?

Pete

 

[chris178]

Hi Pete

Sorry for delay in response but had two very busy days at work... I did some 'building' exercises trying to build luasocket again yesterday evening but not much success - I will update tomorrow.

As to the github luasocket - there is a green button "clone or download" on the right top cornet where the code directory tree is displayed allowing to download whole package (use 'Download ZIP' sub-button).

Just in case url is: https://github.com/diegonehab/luasocket/archive/master.zip .

Other way is to use 'git' to clone project structure onto local file system.

Best regards

Chris

 

[Pete Dowson]

51 minutes ago, chris178 said:

As to the github luasocket - there is a green button "clone or download" on the right top cornet where the code directory tree is displayed allowing to download whole package (use 'Download ZIP' sub-button).

Just in case url is: https://github.com/diegonehab/luasocket/archive/master.zip 

Thanks. Got it now. I'll see if I have time next week to look into it further. I'm away all weekend (though I'll be using my iPad to keep up to date, I'll have no access to my development system).

Pete