Lua Socket 64 FSUIPC 5

[budman9mm]

I am running FSUIPC 5 1.03g, registered.  I am trying to run a LUA script (via Key Presses section) that I used with P3Dv3 that controls my PMDG 737NGX home made overhead panels, etc. in P3Dv4.  I exclusively use UDP to communicate between Arduinos and FSUIPC which uses Lua Sockets.  I now get the LUA error:

Quote

 *** LUA Error: error loading module 'socket' from file 'socket':   socket: bad header in precompiled chunk

I am assuming this is because I have not compiled a 64 bit version of LUA.  I installed the 32 bit version a long time ago and I muddled my way through it with Luarocks version 5.1.x, I believe.  My question is, am I on the right path in believing that I need to figure out how to compile it again with 64 bit?  I am not very strong on these sorts of things and I can't find any clear cut ways to do it on my Win7-64 box.  The searches I am coming up with are all over the map on ways to do it.  Has anyone done this yet that could advise on what I need to do or point me to a URL that explains the best way?

Thanks in advance,

Dustin

Edited July 16, 2017 by budman9mm

[Pete Dowson]

5 minutes ago, budman9mm said:

I am assuming this is because I have not compiled a 64 bit version of LUA.

Is this your own compiled LUA code?

Naturally compiled code needs to be 64-bit compiled for a 64-bit program. the Lua interpreter in FSUIPC5 is of course compiled in 64-bit mode.

7 minutes ago, budman9mm said:

My question is, am I on the right path in believing that I need to figure out how to compile it again with 64 bit?

If you compiled it with a 32-bit compiler, then, yes. If it's a precompiled Lua library then maybe there's a 64-bit one already available?

There's another thread near here where a user tried to use the 5.2 or 5.3 version 64-bit Lua compile, and the didn't work, but he did find a 5.1 version I think. FSUIPC's Lua version is 5.1, and i don't think later versions are compatible.

My only source for data on Lua is the main Lua website, but like you I don't know a lot about compiled lua nor external libraries. .FSUIPC just provides the basic interpreter and the most common libraries, which i think include much of what you need for Luasockets.

If there are other parts with source code available which should be bolted into the FSUIPC4 code to make all this easier I am willing to look at them.

Pete

 

[budman9mm]

1 hour ago, Pete Dowson said:

Is this your own compiled LUA code?

Thanks, Pete.  I haven't compiled any of my own code for the sim (I have never tried).  I just have the script (PMDGOH_1.LUA) in the modules folder.  I think you can tell my ignorance on the topic.  I just wrote enough LUA code (see below) from your included Lua Library Reference(aside from the UDP portion)  to do what I needed.  What I do have compiled is the LUA 32 bit 5.1 and lua sockets in a completely different folder.  If I understand correctly, there is a lua sockets already provided with FSUIPC, so it really should work as is.  I was wondering that since I have PATH variables that point to my own compiled LUA directory, is it possible that it's using that instead of the FSUIPC LUA facilities? So I tried removing all the LUA environment variables and seeing no change.

(Snippet)

--Overhead Panel #1
local socket = require("socket")
local lan = 1 --1 UDP enabled, 0 COM Port enabled 
local address, port = "*", 47019 --send port
local remoteip, remoteport = "192.168.1.142", 47018 --Arduino #1
local remoteip_2, remoteport_2 = "192.168.1.148", 47018 --Arduino #3
host = "192.168.1.5" or "*"
----
--ipc.display("PMDG Overhead/Aft Overhead")

if lan == 0 then
	dev_1 = com.open("COM23", 115200, 0) --Don't go faster than 115200
	dev_2 = com.open("COM7", 115200, 0) --Mega #3 Aft Overhead
--I don't use the COM ports because I tend to get bunches of buffer errors
elseif lan == 1 then 
	server = assert(socket.bind(host, port))
	server:settimeout(0)
	udp = socket.udp()
	udp:settimeout(0)
	udp:setsockname(host, port)
	
	server:setpeername(remoteip, remoteport)
	ipc.lineDisplay("UDP peer #1 opened", 10)
	server:settimeout(5) --Remove blocking
	udp:settimeout(5)
	
	server:setpeername(remoteip_2, remoteport_2) 
	ipc.lineDisplay("UDP peer #2 opened", 10)
	
end

Thanks,

 

Dustin

Edited July 16, 2017 by budman9mm

[Pete Dowson]

Sorry, I am lost with LINDA stuff. I thought there was an updated LINDA for 64-bit use? Isn't there a LINDA support site? 

One of the LINDA developers is a frequent visitor here and would no doubt have spotted your post if you'd have put LINDA in the title. I'll see if I can change that now ...

Pete

 

[Pete Dowson]

3 hours ago, budman9mm said:

What I do have compiled is the LUA 32 bit 5.1 and lua sockets in a completely different folder.  If I understand correctly, there is a lua sockets already provided with FSUIPC, so it really should work as is. 

Not sure whether you need that. I have you checked the network examples in the example plug-ins ZIP in your FSUIPC documents folder. I'll have to revise all that stuff myself as know little about it and have forgotten whay i did know!

Have you tried removing the 32-bit module? That certainly cannot be used.

Pete

 

[budman9mm]

I didn't even realize there are some network examples in there, so I tried running "testsrvr.lua"" which utilizes socket (socket = require("socket");) and it gives me the same exact error.  I have removed any traces to my own lua installation with no luck.  I think I'm at the point where I am going to try install P3Dv4 on another PC and see what transpires. 

1 hour ago, Pete Dowson said:

I'll have to revise all that stuff myself as know little about it and have forgotten whay i did know!

I certainly do NOT envy you having to do this!  I tend to get things in a working order and quickly forget how I did so.

Dustin

 

[Pete Dowson]

59 minutes ago, budman9mm said:

I didn't even realize there are some network examples in there, so I tried running "testsrvr.lua"" which utilizes socket (socket = require("socket");) and it gives me the same exact error.

Ah! Thanks ... I'll check. Maybe I'm inadvertently building in some 32-bit code myself

Pete

 

[Pete Dowson]

Well, I have good news and bad news.

I get the same results here as you. The bad news is that I cannot at present figure out why. All the code is c/c++ source re-compiled for 64-bits. There are no actual "precompiled chunks", at least not unless they are well buried in some of this lua source code. I've traced through to where it gets the error and I'm none the wiser at present -- the code is very dense, very obscure. 

If I don't get anywhere I'll have to seek assistance over in the Lua website.

Pete

 

[budman9mm]

I'm just glad you are aware of it.  It's certainly not urgent for me.  In all of my many searches, I did come across some files linked from github that are supposed to be compiled 64 bit luasocket, but unfortunately I have no idea what to do with them yet.  Maybe they can help you, maybe not.  https://download.zerobrane.com/luasocket-win64.zip

I'll keep trying some work arounds and update if I get anywhere.

Thanks for all your help,

Dustin

[Pete Dowson]

I'm afraid I will have to remove the built-in LuaSocket facilities in FSUIPC5. I have been trying to compile 64-bit versions of socket.core, mimi.core and ltn12.core (I did find I was including them -- wrongly as they were the 32-bit versions), but there are vital bits missing in the LuaSocket sources I downloaded and the things just won't compile.

There is a work-around. Install WideClient on the same PC asFSUIPC5. Change the ClassInstance parameter in the INI to 1 so it can run at the same time as P3D4, and connect to FSUIPC in the normal way, and use socket plug-ins in WideClient. It's still a 32-bit program and has the requisite modules built in.

I'll publish this as a note in the next "changes" notice for an FSUIPC5 update.

Pete

 

[budman9mm]

13 hours ago, Pete Dowson said:

There is a work-around. Install WideClient on the same PC asFSUIPC5.

I will give it a go.  I haven't used WideClient much yet so I am skimming over the documentation.  Does this need to be a registered version?

Dustin

EDIT:  Please disregard.  I see that WIDEFS7 is still the same version from FSUIPC4 and that registration is valid.

[budman9mm]

I was able to get WideFS registered and configured to allow connections on the same PC.  I have my lua script in the separate wideclient folder I created.  Unfortunately it is giving me "6989 *** LUA Error: not enough memory" and just repeats trying with the same.  I verified that it does execute properly in P3Dv3 still.  It's roughly a 45kb file and growing.  I guess since it's not running within the P3D thread that there is significantly less memory allocated?  Any thoughts?  This whole 64 bit thing we all wished for is starting to look fraught with caveats.

Thanks,

Dustin

[Pete Dowson]

5 hours ago, budman9mm said:

6989 *** LUA Error: not enough memory"

That's rather odd. There's no difference at all in WideClient, and the socket code is 100% within that and 32-bit still of course. It doesn't know or care that the FS you are linking to is 64 bit, so the FS version isn't relevant.

5 hours ago, budman9mm said:

I verified that it does execute properly in P3Dv3 still.  It's roughly a 45kb file and growing. 

What, WideClient? Space for execution of Lua threads and other things is allocated dynamically, not part of the EXE file. And as far as it is concerned, P3D3 is the same as P3D4 (and FSX, FS9, even FS98!)..

5 hours ago, budman9mm said:

I guess since it's not running within the P3D thread that there is significantly less memory allocated? 

It doesn't pre-allocate any memory. The memory is allocated as needed, and it wouldn't need any more or different memory arrangement depending on what FS it is talking to. As I said, it doesn't know nor care. Being a comnpletely separate process to FS it is only dependant upon Windows providing whatever resources it needs (which really are not many).

What is the Lua doing when this error occurs? How far is it getting? Can you please supply logs with problem reports. Maybe do it for both P3Dv3 and P3Dv4 both running.

There isn't the same Lua trace facility in WideClient  as in FSUIPC, but to nail the actual line in the plug-in you could try adding little log messages  like ipc.log("I am here #1") into the code and moving it up and down until you pinpoint it.

Maybe P3Dv4 is already grabbing most or all of your real memory before you get to run WideClient? Try running WideClient before starting P3D, and name your Lua file "Initial.lua" so that it starts without waiting for an FS connection.

Pete

 

[budman9mm]

6 hours ago, Pete Dowson said:

What, WideClient? Space for execution of Lua threads and other things is allocated dynamically, not part of the EXE file. And as far as it is concerned, P3D3 is the same as P3D4 (and FSX, FS9, even FS98!)..

Thank you for explaining how the memory is allocated.  To clarify, I ran the script as normal, not using WideClient ,but just FSUIPC4, it ran fine.  So, I tried using WideClient in P3Dv3 and the script runs, although an error occurs in the FSUIPC console...

 ***ERROR C0000005 at 1AF27076 ExWriteStateData(Offset=9BFC, Size=4)
 ***      Access violation trying to read address 000064C8
 ***      EAX 00000026  EBX 00000001  ECX 00000000  EDX 00000000  EDI 1

I have no offsets to 0x64C8.  Even so, and without tracing down what is causing this, it is not a fatal error and the script continues to run.  I'm not sure if it is relevant to the problem or not.

Going back to P3Dv4...I attempted to run WideClient before running P3D and it (WideClient) crashes.  I then execute WideClient again and it throws the out of memory error.

I will try and do as you suggested about tracing it down with ipc.log statements next, but time is short right now as I expect you can understand.

 

Dustin

[Pete Dowson]

30 minutes ago, budman9mm said:

To clarify, I ran the script as normal, not using WideClient ,but just FSUIPC4, it ran fine.  So, I tried using WideClient in P3Dv3 and the script runs, although an error occurs in the FSUIPC console..

...

I have no offsets to 0x64C8

64C8 is not an "offset", it's an actual memory address in the WideClient process. The offset being written at the time is shown in the line before:

Offset=9BFC, Size=4

which is used internally between WideClient and FSUIPC to handle the Lua display window on the FS PC. Are you using Lua display facilities?

Quote

Going back to P3Dv4...I attempted to run WideClient before running P3D and it (WideClient) crashes.

Where's the log please? Any crash details from Windows?

I really need such information. Just saying "crashes" is really no help.

30 minutes ago, budman9mm said:

I then execute WideClient again

So, is P3D4 running by then? If not what changed since the first time?

Can you show me the WideClient.INI file you use to run WideClient on the same PC as P3D, please?

Pete

 

[Pete Dowson]

I've tested here both scenarios: with an INITIAL.LUA starting before P3D4 runs, and a (restarted) WideClient running AFTER P3D4 is full loaded. No problems with the socket requeire call, or any subsequent socket activities, though i dodn't actually connect to it from another PC.

For tresting I'm using the Lua plug-in supplied for that -- "testsrvr.lua" in your Lua examples ZIP, in the FSUIPC Documents subfolder.

Maybe it is something specific to the plug-in you are using. You cannot always run the exact same Lua under WideClient that you can on FSUIPC, or vice versa, as some things are only applicable to their own setting.

Could you try with that testsrvr.lua instead, just to prove that it isn't related to the Networking side of things, which is what we were trying to deal with.  You can run testclnt.lua too -- they'll just connect to each other (see description in the example list in the Lua plug-ins document).

Pete

 

[budman9mm]

2 hours ago, Pete Dowson said:

Are you using Lua display facilities?

Yes.  Mostly for troubleshooting so I can see values of variables. I have now commented out all of the ipc.lineDisplay and ipc.display commands.

3 hours ago, Pete Dowson said:

Where's the log please? Any crash details from Windows?

I really need such information. Just saying "crashes" is really no help.

There was nothing in the log when I sent that except the windows event viewer.  However, I tried a few more times and WideClient did terminate with a message to check the log.  I attached it to this message along with the windows event viewer details.  These are from after removing the ipc.display cmds.

2 hours ago, Pete Dowson said:

Could you try with that testsrvr.lua instead, just to prove that it isn't related to the Networking side of things, which is what we were trying to deal with.

I did as you asked with tstsrver and tstclient and it ran without error and seems to be connecting just fine.  I've attached the log for that anyway as well.

I've started adding points in my code to log.  It does seem that the memory errors start once I start receiving event offset triggers...somewhere.  It's going to take some doing to find out just where.

Thanks,

D

WideClient.log

windows_event_view.txt

testclnt.log

[Pete Dowson]

1 hour ago, budman9mm said:

However, I tried a few more times and WideClient did terminate with a message to check the log.  I attached it to this message along with the windows event viewer details.  These are from after removing the ipc.display cmds.

Okay. Those are two different errors. I'll take a look, tomorrow probably.

1 hour ago, budman9mm said:

I did as you asked with tstsrver and tstclient and it ran without error and seems to be connecting just fine.

Okay. So it looks like it isn't related to Sockets itself. Must be something else. Are you sure the Lua plug-in is using only facilities supported by WideClient? Mind you, if it is using libraries or functions not included you should just get a Lua error report saying so.

Pete

 

 

[budman9mm]

Pete,

I'm sorry for missing this, but I was unaware until you said as much that some of the plug-in commands weren't available - I later see it in the User Guide.  I was able to find a case of event.timer being used being used to check my UDP connection for receive data every 2msec.  So I tried tried removing that and instead using a loop to execute my "receiveUDP()" function.  It now does not give any Lua out of memory error, but it results in a UDP connection refused.  With "event.timer" it was making connection with my Arduino nicely, but after a number of commands sent back and forth, it would start the memory issues.

I may keep trying to chase this down for a bit.  Maybe you have some idea as well.  However, I'm just wondering if you think the sockets issue will ever be able to be compiled into FSUIPC5?  I may just go back to working in the previous versions of software until that time.

Thanks

D

[Pete Dowson]

41 minutes ago, budman9mm said:

I'm sorry for missing this, but I was unaware until you said as much that some of the plug-in commands weren't available

You mean in WideClient? If you browse through the Lua Libray reference document, you'll see some functions or even libraries marked as "Not WideClient" or "WideClient only". below I append a list for you.

1 hour ago, budman9mm said:

I was able to find a case of event.timer being used being used to check my UDP connection for receive data every 2msec.

event.timer is okay. i use it quite a lot in plug-ins.

1 hour ago, budman9mm said:

It now does not give any Lua out of memory error, but it results in a UDP connection refused.

There's something else awry in your code. There's no way changing from one type of "loop" (by time) to another should do that. Unless your loop doesn't inlide an ipc.sleep so that you don't hog the processor. Try ipc.sleep(2) if you want to do it at the same speed as the timer, though i think you'd find the timer method only gives you 2 msecs approximately. You might need to allow more for a contained loop.

Quote

 

With "event.timer" it was making connection with my Arduino nicely, but after a number of commands sent back and forth, it would start the memory issues.

 

Ah. I don't think you mentioned that before? I would guess that, because the WideClient process won't be as "busy" as the FS process, your timer events were arriving much closer to the 2mSecs than when you were in the FS process. I'm pretty sure that with TCP protocol, and maybe UDP too, the Windows functions allocate buffers from system space for each block , or each block requested even if not arrived. Maybe these were accululating more rapidly that they were freed. Try a longer timer value. 500 times per second does sound rather extreme. Try a much longer time (say 20) then wor downwards if needed.

1 hour ago, budman9mm said:

However, I'm just wondering if you think the sockets issue will ever be able to be compiled into FSUIPC5?  I may just go back to working in the previous versions of software until that time.

It would need someone who can make some sense of the very dense, uncommented, code in the "socket.core", "mime.core" and "ltn12.core", which are the parts which are simply built into FSUIPC4 as ready-compiled code. Or if you can find 64-bit compiled versions of these which are Lua 5.1.x compatible.

By "go back", do you mean to P3Dv3?

Pete

 

Here's the list i mentioned:

[Not WideClient]
   ipc.ask
   ipc.axis
   ipc.buttons
   ipc.clearflag
   ipc.getLvarId
   ipc.getLvarName
   ipc.readLvar
   ipc.readPOV
   ipc.RestoreFriction (not in FSUIPC5 yet)
   ipc.setdisplay 9dummy provided in WideClient)
   ipc.setflag
   ipc.SetFriction
   ipc.setowndisplay
   ipc.testbutton
   ipc.testflag
   ipc.toggleflag
   ipc.writeLvar
   mouse: library not included
   event.button
   event.control
   event.gfd
   event.intercept
   event.key
   event.Lvar
   event.mousehoriz
   event.mousehoriztrap
   event.mouseleft
   event.mouselefttrap
   event.mousemiddle
   event.mousemiddletrap
   event.mousemove
   event.mousemovetrap
   event.mouseright
   event.mouserighttrap
   event.mousewheel
   event.mousewheeltrap
   event.param
   event.vriread
   

[WideClient ONLY]
   ipc.setbtncol
   ipc.setbtnstate
   ipcsetbtnstateonly
   com.gethidbuttoncount
   wnd complete library
   display complete library

 

[budman9mm]

EDIT:  Pete, please see my next message first!

 

Hello again,

11 hours ago, Pete Dowson said:

event.timer is okay. i use it quite a lot in plug-ins.

I misread the WideFS user guide where it mentioned this thinking it was an exception in the com library.  Anyway, I tried varying the duration of the timer same with ipc.sleep as mentioned (see below) and it still goes through a few messages passed back and forth and ends up with the same out of memory errors.

11 hours ago, Pete Dowson said:

There's something else awry in your code. There's no way changing from one type of "loop" (by time) to another should do that. Unless your loop doesn't inlide an ipc.sleep so that you don't hog the processor. Try ipc.sleep(2) if you want to do it at the same speed as the timer, though i think you'd find the timer method only gives you 2 msecs approximately. You might need to allow more for a contained loop.

I agree.  It should not make a difference.  However, I tried using ipc.sleep(2) within a loop that then calls my function to receive the UDP and I get the same connection refused every iteration.    I also tried varying the duration up from a few msec, to 50 msec, 100, etc. up to 2 full seconds...same result.  I will continue to try and track this down using event.timer.

11 hours ago, Pete Dowson said:

Or if you can find 64-bit compiled versions of these which are Lua 5.1.x compatible.

Did you see my previous post regarding what is "supposed" to be just that (copied below)?

On 7/16/2017 at 8:01 PM, budman9mm said:

I did come across some files linked from github that are supposed to be compiled 64 bit luasocket, but unfortunately I have no idea what to do with them yet.  Maybe they can help you, maybe not.  https://download.zerobrane.com/luasocket-win64.zip

I'm just asking because I was excited to see this, yet I have no idea what to make of whether it is applicable or not.  I would be certainly willing to investigate incorporating it if you think it is worthwhile.

11 hours ago, Pete Dowson said:

By "go back", do you mean to P3Dv3?

Affirmative.  I haven't given up yet though.  I just haven't had time yet, but I am going to try and rebuild the code step by step.

And thanks for the list.  I have a printed "FSUIPC Lia Library Reference" that I have referenced at least a hundred times and it just did not register that it specifically says "[Not WideClient]". I don't know why I'm forgetting these details...*sigh* I regret wasting my efforts as well as yours.

Thanks,

Dustin

[budman9mm]

I have made some progress with the UDP issues and WideClient.  First, without a doubt, the lua file of this size must be run as "initial.lua" and WC started before P3D in order to allocate enough memory (you mentioned this before, but I think it's absolutely required).  Secondly, changing the event.timer that checks the UDP connection for received data to somewhere around 50msec AND adjusting my udp socket.bind timeout to something other than zero (2) appears to have fixed the connection refused problems.  Best I can tell, with some limited testing, the UDP connections are working as they should, and quite well.  No more LUA out of memory errors!

A problem I am running into now is that all of my event.offsets are getting sent one time, at startup, as they should.  However, after that, no further state changes are reported by WideClient as well as the STR type offsets are reported as nil by WC.  I confirmed that "initial.lua" is still running.  This may well be a PMDG issue.  They have just released an update to the NGX on Tuesday, so I'm getting that now.  Will test this out tomorrow hopefully.

Thanks again,

Dustin

 

[Pete Dowson]

6 hours ago, budman9mm said:

A problem I am running into now is that all of my event.offsets are getting sent one time, at startup, as they should.  However, after that, no further state changes are reported by WideClient as well as the STR type offsets are reported as nil by WC. 

To make sure it isn't to do with FSUIPC or WideFS, could you add an event.offset for something which is known to change redularly, and log that so you can see it still running? For example, the time of dat "second" value, one byte at 023A. If that is okay it is down to the PMDG aircraft I think.

9 hours ago, budman9mm said:

Did you see my previous post regarding what is "supposed" to be just that (copied below)?

On 7/17/2017 at 1:01 AM, budman9mm said:

I did come across some files linked from github that are supposed to be compiled 64 bit luasocket, but unfortunately I have no idea what to do with them yet.  Maybe they can help you, maybe not.  https://download.zerobrane.com/luasocket-win64.zip

I'm just asking because I was excited to see this, yet I have no idea what to make of whether it is applicable or not.  I would be certainly willing to investigate incorporating it if you think it is worthwhile.

Yes, I looked at that. I even downloaded the source and tried compiling it (I need compiled modules fitting into the rst of my Lua interpreter). But there appear to be too many things missing which I couldn't find anywhere in the source provided. I assume it must be possible. i tried to work out what the missing bits should look like, but that's where I got lost in the very complex convoluted and uncommented code.

Pete

[budman9mm]

4 hours ago, Pete Dowson said:

To make sure it isn't to do with FSUIPC or WideFS, could you add an event.offset for something which is known to change redularly, and log that so you can see it still running? For example, the time of dat "second" value, one byte at 023A. If that is okay it is down to the PMDG aircraft I think.

This offset works as it should.  I discovered that during the last update from PMDG, the SDK was reinstalled and overwrote the .ini and the SDK was disabled.  I corrected that and all offsets are triggering with desired results.  I think we can put this to bed.

4 hours ago, Pete Dowson said:

But there appear to be too many things missing which I couldn't find anywhere in the source provided. I assume it must be possible. i tried to work out what the missing bits should look like, but that's where I got lost in the very complex convoluted and uncommented code.

I see.  That can't be easy to decipher.  I don't know if the author, Diego can help from his gitHub.  I don't want to force the issue down your throat more than I have. 

Thanks for giving this issue so much attention.  I probably would have given up on it long ago without your help.

Regards,

Dustin

[brian60]

So sockets don`t work with p3dv4 and fsuipcv5 ? did you get it working?