Lua capabilities

[Pete Dowson]

Somehow Lua is managing to load the module in such a way that none of its external dependencies are being filled in! The result, obviously, a crash as it jumps into nowhere in particular!

Okay, I think I've solved it.

The "LoadLibraryEx" Windows API call was explicitly telling Windows to NOT load any DLLs the module needs and so not fill in any references. I don't know why. I presume this was intended to avoid the need for a "DllMain" function in the module -- but it doesn't seem to fail even if there isn't one. So I changed the LoadLibraryEx call to explicitly load dependent DLLs and fill in the references.

That solved the bulk of the crash causes. The others were all due to struct member alignments being different. It works here now with "pragma pack( 8 )" settings for all Lua structs. This may only be because it is the current MSVC default alignment, so it is still an unknown for any old modules you may pick up. But FSUIPC no longer crashes FS when running either your "base64.dll" or "TestLua.dll". The base64 one complains of not being able to load a module, but it doesn't crash FS.

So, I'm now going to upload, to the Download Links subforum:

Version 4.705 of FSUIPC4

Version 3.995 of FSUIPC3.DLL

Version 6.89 of WideClient.EXE

with these changes. Hopefully they'll allow any properly written and compiled (!) DLL to be loaded as a module.

Perhaps you could try a few and let me know?

Regards

Pete

[Chrilith]

Great, sounds good!

I'll give it a try later today!

Thanks Pete!

[Pete Dowson]

Great, sounds good!

I'll give it a try later today!

Another user, see the thread next to this, has tested it already using the LuaFileSystem module.

One thing he found, and i've verified, is that when Windows fills in the dependencies for the loaded DLL module, it looks in the PROCESS's folder, not the folder from which it is loading the DLL module itself. So if you need lua5.1.dll, for example, it needs to go into FS's root folder (or in the usual Windows folders which are searched after).

After I did this "base64" didn't fail either. Mind you it didn't do anything but that's presumably because I made no calls upon it.

It is a bit annoying having to put DLLs into the FS root folder but I doubt there's any way around that -- though I'm looking.

[LATER]

Found it! There's a Windows API called "SetDLLDirectory", which adds a path to the list for DLL searching. I'll get FSUIPC to add "Modules\Lua" and "Modules\DLL" for the entire session. That should deal with it! But i'll leave this till i return from holiday, by which time hopefully the modules facility will have been well and truly tested! I might also then incorporate the LuaFileSystem (lfs) library into FSUIPC and Wideclient. It looks useful and won't add noticeably to FSUIPC's size.

Regards

Pete

[Chrilith]

Wow, things are really getting better and better! Quite cool!

Thank you for your hard work on this Pete!

[Chrilith]

Ok, just tested my previous test module and no crash! Great job!

Now, I'll continue the development of my Saitek module and let you know whether I have an issue.

Thank you again Pete.

[Artem Crum]

Pete,

When can we now expect the update with DLL paths and LFS?

I'm going to release a freeware project next days which is using FSUIPC/Lua at it's backend and need the LFS library to work. I can include it in the release package, but it will be much better if it will be compiled in FSUIPC dll.

Here is the project intro: http://forum.avsim.net/topic/336857-welcome-to-linda/

It a device assigning tool to replace the "Buttons and Switches" settings in FSUIPC.

It also implements it's own VRinsight MCP Combo dirver written in lua with much more capabilities than original SerialFP software.

[Pete Dowson]

When can we now expect the update with DLL paths and LFS?

Certainly this week, by next weekend. I'll release a test version here before Saturday I hope. I've caught up okay with my backlog from holiday but at present I'm busy with a cockpit engineer from the U.S. who is helping me upgrade my 737NG setup.

I'm going to release a freeware project next days which is using FSUIPC/Lua at it's backend and need the LFS library to work. I can include it in the release package, but it will be much better if it will be compiled in FSUIPC dll.

Yes. If you can hold off till the end of the week ... or alternatively release without the library but with a download link for it with a note ... then it should be fine.

It would be quicker if you could help test it for me when I've first done it, so which is best for you first: FSUIPC3, FSUIPC3 or WideClient? Or perhaps you have a Lua test script using the LFS functions, which I could use?

Regards

Pete

[Artem Crum]

Ok, I'll run the beta test this week and wait for your update before public release.

Sure I can do my tests with updeted FSUIPC as soon as you give me a download link (LINDA is using FSX/FSUIPC4).

Or you can test it yourself if it will be more convinient to you. The simple example of code using LFS lib:

lfs = assert(require('lfs'))

local dir_obj = lfs.dir ('.\\modules\\')
local dir = dir_obj ()

while dir ~= nil do

 	ipc.log(dir)
 	dir = dir_obj ()

end

This code should read the FSX/modules folder and show all it's entries in FSUIPC log.

[Pete Dowson]

Sure I can do my tests with updeted FSUIPC as soon as you give me a download link (LINDA is using FSX/FSUIPC4).

Or you can test it yourself if it will be more convinient to you. The simple example of code using LFS lib:

Okay, thanks. I'll do one then the other. I'll need to add the other functions too.

Pete

[Pete Dowson]

Sure I can do my tests with updeted FSUIPC as soon as you give me a download link (LINDA is using FSX/FSUIPC4).

Interim version FSUIPC 4.706 from this link:

FSUIPC 4.706

works okay with your test Lua program. Perhaps you'd like to download it and give it a bit of a work out before i add the library to FSUIPC3 and WideClient?

Thanks & Regards

Pete

[Artem Crum]

I'm on it...

[Artem Crum]

Works fine on my system!

Thank you, Pete

[Chrilith]

Hi Pete,

I think this is the only remaing problem. Here is the case:

In FSUIPC when you press the key to launch a Lua script again, the previous thread is killed and the script relaunched.

The problem is that you don't call the finalizers so that the script can quit properly releasing any allocated ressource from an external module and so, in some modules, the ref counter should become unreliable.

The common Lua engine calls finalizers(let's say pressing CTRL+C in command line) and the __gc functions are all properly called on interruption. The SciTE editor does this too.

Could you please check this?

Thanks!

[Pete Dowson]

In FSUIPC when you press the key to launch a Lua script again, the previous thread is killed and the script relaunched.

The problem is that you don't call the finalizers so that the script can quit properly releasing any allocated ressource from an external module and so, in some modules, the ref counter should become unreliable.

The common Lua engine calls finalizers(let's say pressing CTRL+C in command line) and the __gc functions are all properly called on interruption. The SciTE editor does this too.

Could you please check this?

I will certainly add code to call something, if present, before terminating the thread. How about an "event.terminate" function, so you can declare to me what function to call when i want to throw the program off?

I'd still need to ruthlessly terminate the thread if the Lua program doesn't voluntarily perform an exit -- how many miliiseconds should be allowed do you think?

If this is not in accord with the "common Lua engine" action, could you kindly explain what that should do? As far as I'm aware I've left nothing out of the Lua code in my program. Maybe there's some in that code which needs me to call it, but I need a clue please. You seem to assume I know a lot about Lua, which I don't -- certainly not as much as your good self. What do you mean, for instance, by "__gc functions"?

Regards

Pete

[Artem Crum]

Not my request, but it would be great to have something like event.terminate available.

I've had some problems in LINDA with closing the com-port on script restart - port becomes locked/unaccessible after restart.

[Chrilith]

I not so expert too :)

I will certainly add code to call something, if present, before terminating the thread. How about an "event.terminate" function, so you can declare to me what function to call when i want to throw the program off?

Would be interesting indeed if there is no other way to kindly terminate the script.

I'd still need to ruthlessly terminate the thread if the Lua program doesn't voluntarily perform an exit -- how many miliiseconds should be allowed do you think?

I understand that but I'm not sure that a script can voluntary perform an exit when the script is killed unless it receive some kind of event.

If this is not in accord with the "common Lua engine" action, could you kindly explain what that should do? As far as I'm aware I've left nothing out of the Lua code in my program. Maybe there's some in that code which needs me to call it, but I need a clue please. You seem to assume I know a lot about Lua, which I don't -- certainly not as much as your good self. What do you mean, for instance, by "__gc functions"?

I'll take an example related to my module. Hopefully this will help.

When my module is initialized, it creates a user data using lua_createuserdata() and store it in the LUA_REGISTRYINDEX using lua_ref(). Like this, the "memory" will not be collected unless the script ends. The user data is linked to a meta table so that a __gc function can be associated with it which will be called upon garbage collection. The function is pushed to the stack using lua_pushcfunction().

Now, with Lua, even if you don't explicitly end your script (let's say you have an infinite loop and the script is interrupted with CTRL+C), the garbage collector (seems) to do its job and then call all required __gc function of allocated objects.

Does this help?

[Pete Dowson]

Not my request, but it would be great to have something like event.terminate available.

I've had some problems in LINDA with closing the com-port on script restart - port becomes locked/unaccessible after restart.

Hmmm. That's odd, because if you are opening the port through the FSUIPC com library, FSUIPC knows about it and should be closing it for you when it terminates the thread. I'll double-check that.

On the "terminate" call, I'll see what the Lua expert has to say before deciding what to do. I can't do anything till next week, in any case.

Regards

Pete

[Chrilith]

Ok, I answeredyour questions before you post this message.

Again, no rush on this of course :)

[Pete Dowson]

I not so expert too

But judging by your code and advice, you are most certainly a lot more "expert" than me in Lua.

I understand that but I'm not sure that a script can voluntary perform an exit when the script is killed unless it receive some kind of event.

I think you misunderstood. the timeout to "exit" would start when FSUIPC sent the event to the program. It was an adjunct to what I had said just previously! What else could it mean? There's no point in just inserting a time out otherwise, of course,.

I'll take an example related to my module. Hopefully this will help.

When my module is initialized, it creates a user data using lua_createuserdata() and store it in the LUA_REGISTRYINDEX using lua_ref(). Like this, the "memory" will not be collected unless the script ends. The user data is linked to a meta table so that a __gc function can be associated with it which will be called upon garbage collection. The function is pushed to the stack using lua_pushcfunction().

Now, with Lua, even if you don't explicitly end your script (let's say you have an infinite loop and the script is interrupted with CTRL+C), the garbage collector (seems) to do its job and then call all required __gc function of allocated objects.

Er, are the allocated objects simply all assigned memory sections? I think all of those are freed automatically when the thread is terminated. Are you seeing a gradual increase in memory use indicating some sort of memory leak? The whole stack system used by Lua is involved, all in allocated memory objects, not just items you allocate yourself later.

I've not yet had time to look at the code, but as far as I understand the action should just be similar to that which occurs when exit() is executed?

I thought the concern was more for system resources like devices and files which might have been opened by modules not directly being controlled by FSUIPC or the Lua interpreter, like DLLs.

Does this help?

Sorry, no, not really. What is a "__gc" function? How is FSUIPC supposed to know about these so it can call them? I'm a bit lost in all of this, I must admit.

I shall try to find some references to all of this in the Lua books I have. But I won't have time till next week.

Regards

Pete

[Chrilith]

I think you misunderstood. the timeout to "exit" would start when FSUIPC sent the event to the program. It was an adjunct to what I had said just previously! What else could it mean? There's no point in just inserting a time out otherwise, of course,.

Makes more sense indeed, sorry :)

I thought the concern was more for system resources like devices and files which might have been opened by modules not directly being controlled by FSUIPC or the Lua interpreter, like DLLs.

Yes, this is more a resource related problem and everything is related to the garbage collector.

I think that the only thing to do is to force garbage collection before killing the thread so that allocated resources can be freed.

If this helps, I can try to compile Lua and see how it operates on script interruption.

Thanks!

[Pete Dowson]

Yes, this is more a resource related problem and everything is related to the garbage collector.

But if the garbage collector only deals with allocated memory, not other system resources like files and devices, then I don't think it is needed in this environment, because the thread is killed along with all of its allocated memory. It isn't like a normal Lua environment where the Lua program is part of the main program. the Lua threads are like little processes.

I will double check to make sure there are no memory leaks, of course

I think that the only thing to do is to force garbage collection before killing the thread so that allocated resources can be freed.

I assume this means calling some Lua routine from C? I'll check.

Next week.

If this helps, I can try to compile Lua and see how it operates on script interruption.

Any information would be useful. I'm a tad confused at present I must admit.

Regards

Pete

[Chrilith]

But if the garbage collector only deals with allocated memory, not other system resources like files and devices, then I don't think it is needed in this environment, because the thread is killed along with all of its allocated memory. It isn't like a normal Lua environment where the Lua program is part of the main program. the Lua threads are like little processes.

Not only allocated memory, some Lua objects (either from a Lua script or user data from a C module) have a defined __gc function which should be called upon garbage collection to free resources.

I assume this means calling some Lua routine from C? I'll check.

Yes, there are lots of functions related to GB but really don't know what to call right now.

Next week.

Of course! :)

[Pete Dowson]

Not only allocated memory, some Lua objects (either from a Lua script or user data from a C module) have a defined __gc function which should be called upon garbage collection to free resources.

Okay. Having looked it all over there are some conclusions.

First is that those __gc functions are called regularly in any case by the Lua interpreter. They will also be called by the "lua_close" function when it finally tidies up and frees the main Lua stack memory.

Second is that FSUIPC cannot call "lua_close" from its main thread. It has to be called from within the thread in which the Lua program to be terminated is running. The only way FSUIPC's thread can get that to happen in a running or looping or stuck Lua thread is to set a flag in the main Lua state structure, and change the interpreter to check that flag between every statement or block, aborting the statement or block is it is set. This should make it ultimately ignore all the following statements and blocks till it exits and tidies of its own accord.

Unless you have any better ideas, this is what I am doing right now.

I still need to place a time limit on this before forcibly throwing the thread off. Since the interpreter will be rushing through the statements like a dose of salts, ignore every one, I think 10 mSecs should be sufficient (elapsed) time for even the most complex programs. What do you think?

I will also implement that event.terminate facility, which will be given longer. not sure how much longer at present (maybe 30 mSecs?). If such an event function is provided, it will of course be called before the "abort" method mentioned above is used.

Regards

Pete

[Chrilith]

Sounds good Pete. Thank you. I'll will test as soon as this new version is available.

Thanks again

[Pete Dowson]

Sounds good Pete. Thank you. I'll will test as soon as this new version is available.

Okay. try these:

FSUIPC 3.997d

FSUIPC 4.715

When a plug-in is being Killed, a flag is set which caused all statements to be bypassed until the end is reached, which will terminate that script (or module), thus very soon ending the execution at which time a normal Close should sort out any resources used. Of course if C-modules are called it cannot achieve that sort of result so easily, but hopefully they don't keep control for too long.

I've allowed only 5 mSecs for this action to occur before forcibly throwing off the thread if it is still running. After experiments I found that 10 mSecs was sometimes just long enough to affect FS smoothness (I am controlling Lua plug-ins from FS's main thread because it makes it more efficient in accessing FS functions and data -- it would be far too much work and very complicated moving the whole system into a separate thread.

There's also a new event:

event.terminate("function name")

which can be used in event -driven plugins to do specific tidying before being thrown off. There's 5 mSecs allowed here too (additional to the 5 mSecs above when both apply). Note that events aren't called when the plug-in is running, so it isn't any good for looping plug-ins or those engaged in such heavy work that the 5 mSecs could easily expire before the terminate event can occur.

I've also changed the way the ipc.sleep(n) function works so that instead of using the Windows Sleep API directly with the value provided, making it non-interruptible for that time, is performs the sleep in small doses in a loop and checks the termination needs on each loop, finishing the sleep when an abort is indicated.

I've not released these formally in the Download Links subforum yet, pending your findings.

Regards

Pete