Jump to content
The simFlight Network Forums

Recommended Posts

Posted

Hello,

I'd like to know how to make Lua plug-ins work.

I'm building a Lua module to handle Saitek FIP but can make the provided FSUIPC+Lua examples to do or show anything on screen.

Does Lua require the reigstered version to operate? (I don't see any [lua script].log file generated but lines are added to the fsuipc.ini file)

Thanks!

Chris

  • Replies 54
  • Created
  • Last Reply

Top Posters In This Topic

Posted

Ok, I did configure a hot key to launch my custom Lua script and module.

Here is the sample script:

testlib = package.loadlib("C:\\Program Files (x86)\\Lua\\5.1\\chrilith\\Lua_SaitekLib.dll","luaopen_MyLuaDLL");

if(testlib)then
	testlib();
else
	-- Error
end

MyLuaDLL.HelloWorld();

Unfortunately, this code works fine using the classic Lua environement but crashes FS9. It seems to be the call to testlib() which makes it crash.

The Lua_SaitekLib.dll, for testing purpose only use this simple code:

#include <Windows.h>
#include "lua.h"
#include "lualib.h"
#include "lauxlib.h"

static int MyLuaDLL_HelloWorld(lua_State* L) {
	FILE *f;
	fopen_s(&f, "D:\\test.file", "w");
	fwrite("test", 4, 1, f);
	fclose(f);

	return 0;
}

static const luaL_reg MyLuaDLLFunctions [] = {
	{"HelloWorld", MyLuaDLL_HelloWorld},
	{NULL, NULL}
};

int __declspec(dllexport) luaopen_MyLuaDLL(lua_State* L) {
	luaL_register(L, "MyLuaDLL", MyLuaDLLFunctions);

	return 1;
}

Pete, since you wrote some extensions to Lua to be used in FSUIPC, do you see any problem here which make my code crashes under FS but not under classic Lua env?

Thanks!

Posted

Unfortunately, this code works fine using the classic Lua environement but crashes FS9. It seems to be the call to testlib() which makes it crash.

The Lua_SaitekLib.dll, for testing purpose only use this simple code

Well, you are doing something I've never tried, and you need to bear in mind that the environment you are running inside is as a thread in FS. Your program looks deceptively simply but it is using a load of C library routines. Have they been statically linked so that they are part of the code, or does your DLL in turn try to call the C run-time? If so is it the multithreaded or single-threaded version? I think there could be a huge tangle down there somewhere.

Since you evidently have a development system to make this, have you used the debugger to determine where exactly it crashes and why?

Since you wrote some extensions to Lua to be used in FSUIPC, do you see any problem here which make my code crashes under FS but not under classic Lua env?

See above. All my extensions use facilities in FSUIPC or in FS or in the libraries already in use by both. Bear in mind this is all one program. A "classic Lua environment" is simply the Lua interpreter, not the Lua interpreter running as one of many threads in some fairly complex multi-threaded program like FS.

If you want me to check what is happening on FSX, instead, email me the parts ready to run (petedowson@btconnect.com). I can't promise anything. I am not set up to do debugging on FS9 these days, but why don't you try doing something with that there?

Regards

Pete

Posted

Hi Pete,

Thank you very much for your answer. Unfortunately I wasn't able to make it work.

I recompiled the Lua static library in Multi-Threaded Debug and Multi-threaded Debug DLL, same for my own DLL of course but no luck. I also tried the dynamic version. It always crashes when executing something from luaopen_MyLuaDLL(). If it is empty, it doesn't crash but do nothing, of course :). As soon as a code in the function is executed whether a luaL_* call or something else like the fopen() stuff, FS9 crashes.

Also, I wasn't able to enter the debugger because I always have an error saying that a debugger has been detected as soon as FS is launched...

Hard problem... :(

All this is done using VC++ 2010.

Posted

OK, well, I did a simple test using an existing Lua module from the Lua package (base64.dll) I copied to the FS.../Modules/lua/ folder. I created a simple Lua script that I try to run from FSUIPC

require("base64");

Same result! This simple script makes FS crash, so I suspect that there is a problem in the Lua implementation built-in FSUIPC since it is impossible to load an external module.

Posted

Thank you very much for your answer. Unfortunately I wasn't able to make it work.

I said I'd debug it for you if you sent me something to test. I'll do it on FSX since I'm not set up for FS9 at present.

Also, I wasn't able to enter the debugger because I always have an error saying that a debugger has been detected as soon as FS is launched...

Hmmm. Why not launch FS first then attach to the process from the debugger?

All this is done using VC++ 2010.

I'm using that for FSUIPC4, but I'm still on VS 2003 for FSUIPC3 so that it's libraries etc are the same as those FS9 is using.. Saved a lot of trouble having folks loading more run-times.

From your subsequent message:

I suspect that there is a problem in the Lua implementation built-in FSUIPC since it is impossible to load an external module.

I told you already I'd not tested any of that. I state in the documentation that I've not taken possible incompatible parts out, but may have to do so when and if they are found to crash FS. I've made very few changes at all to the Lua interpreter itself (only things like routing 'print' to the log and making exit terminate only the thread not the EXE), and the module loading stuff is untouched just like most of the rest. It probably needs massaging to fit into the FS environment.

I will either remove it completely to avoid further folks complaining, OR I'll try to make it work, as I offered to do if you look back. I am not familiar enough with that part of Lua to construct a test, which is why I asked you to send me whatever it takes and I'll debug it for you here and see if I can fix it. I even gave you my email address so you could send me stuff.

Regards

Pete

Posted

The purpose is not to complain but to find out what is wrong and fix the problem if possible :)

I really think that Lua addition is a amazing new feature and really want to see it work perfectly.

I'll prepare you a very quick new project so that you can debug it.

Thanks!

Posted

I'll prepare you a very quick new project so that you can debug it.

Received, thank you.

It fails here too. However, I can't see how your test module can work in FSUIPC as it stands. It contains Lua C calls, presumably intended to call directly back into the.Lua interpreter bolted into FSUIPC. But it links (imports) from Lua5.1.dll which of course is not being used or supplied, and presumably wouldn't relate to the interpreter in FSUIPC in any case.

FSUIPC doesn't export any of the Lua procedures for external C modules to call. By an external DLL module I was of course assuming this was something completely non-Lua, a routine for accessing external hardware or some such. I've obviously been a little naive in this regard, because I now assume you need at least some such callbacks in order to provide the tables of functions or variables provided by the module in the Lua caller.

I'm afraid I don't know enough about this side of "normal" (freestanding) Lua to do much very quickly. Can you explain what you think ought to be provided? Do you expect all of the Lua 5.1 lib functions to be exported? And how would the standard Lib file you are linking hook into the resulting FSUIPC exports. You'd presumably need a specific FSUIPC.lib to link against?

At present I'm not sure I want to proceed down such a route. I'd need to understand it all first, then try to design something which will work easily without forcing folks to recompile modules for each new update of FSUIPC. It looks like it could be a large project -- as large as my original undertaking to incorporate Lua in fact.

I also noticed that the current code searches some weird and wonderful pathnames if you don't get the module correctly identified. I'll probably need to sort that too.

Regards

Pete

Posted

But it links (imports) from Lua5.1.dll which of course is not being used or supplied, and presumably wouldn't relate to the interpreter in FSUIPC in any case.

The Lua lib I use is a static library and is not supposed link to any external DLL since the whole required code is, in this case, part of the module DLL itself.

In my understanding, Lua is all about state and C Lua functions are just filling the lua_State structure to communicate.

In the first test code, indeed, the "base64" module should try to access Lua functions to initialize itself upon Lua request with the require() call.

Did you try to step into the internals of the "require" function in your Lua implementation to see where it fails and what exception is raised?

FSUIPC doesn't export any of the Lua procedures for external C modules to call. By an external DLL module I was of course assuming this was something completely non-Lua, a routine for accessing external hardware or some such. I've obviously been a little naive in this regard, because I now assume you need at least some such callbacks in order to provide the tables of functions or variables provided by the module in the Lua caller.

Indeed, for the first test which use the "base64" module this should be a problem and the call to luaL_openlib/luaL_register will always failed. For the VC one, it should not (it uses a static library) and the call to loadlib(), in the lua script, does work and returns a valid pointer. It seems that something goes wrong later in the call to luaL_openlib/luaL_register (luaL_register internally calls luaL_openlib) which is handled by the call to testlib() in the Lua script. Unfortunately I wasn't able to find what kind of exception is raised here (access violation?).

All external Lua module/extension have to call back to the Lua engine, at least once, in order to register itself and make its functions available to the Lua code. The modules you have embedded in FSUIPC are of course fully functional since there are all part of the same code and so, have access to all the needed functions.

Can you explain what you think ought to be provided? Do you expect all of the Lua 5.1 lib functions to be exported?

And how would the standard Lib file you are linking hook into the resulting FSUIPC exports. You'd presumably need a specific FSUIPC.lib to link against?

I think that "package" is the only part that may require this internal access. This includes all the functions required to initialize a module. But a module may want to call some other functions :(.

About the exports I really don't know right now. Perhaps, instead of adding the Lua code in FSUIPC you can link to the Lua DLL and create external modules for "ipc" or "gfd" for instance.

Hopefully this helps. I'm sorry but I believe I'm far behind you regarding Lua internals coding :). Anyway I want to help as much as possible. Let me know if you need more information.

Thank you

Chris

Posted

The Lua lib I use is a static library and is not supposed link to any external DLL since the whole required code is, in this case, part of the module DLL itself.

Odd then that when I look at its imports in PE Explorer the "Base64.dll" module you supply tells me it needs Lua5.1.dll to import several functions:

lua_pushlstring

luaL_buffinit

luaL_openlib

lua_settable

luaL_pushresult

luaL_addlstring

luaL_checklstring

Admittedly these aren't the ones I was worried about. But I'm not sure what happens when the lua5.1.dll isn't available to it.

In my understanding, Lua is all about state and C Lua functions are just filling the lua_State structure to communicate.

Hmmm. The state structure is of course owned by FSUIPC. I added some stuff to the end but the part used as standard should be manipulatable.

Did you try to step into the internals of the "require" function in your Lua implementation to see where it fails and what exception is raised?

Yes, but not using stepping all the way. It takes ages and ages that way, the Lua stuff is so convoluted!

The base64.dll you supplied (test#1) is crashing with an access violation with some daft address (0x000023E0), but the point in the code identified varies slightly.

Sometimes it is inside lua_call (part of lapi.c), at the luaD_call. The lua_call came from the ll_require function. At other times it is identified actually inside the luaD_call, at the luaD_precall line. I'm checking the registers and nearby values but I can't find where this "0x000023E0" is referenced, used, or appears -- seems to be nowhere.

There's a loop someplace. If I trap at the luaD_precall function I find it is executed about 14 times, from the same routines, before crashing on the 15th iteration. I've no idea what it is doing, it's either a stack corruption, or it could simply be a result of the dependence on the lua5.1.dll not being fulfilled. I don't know how Lua succeeded even in loading the DLL with such a dependency unsatisfied, but it is loaded.

Indeed, for the first test which use the "base64" module the should be a problem since the call to luaL_openlib/luaL_register will always failed. For the VC one, it should not (it uses a static library) and the call to loadlib(), in the lua script, does work and returns a valid pointer. It seems that something goes wrong later in the call to luaL_openlib/luaL_register (luaL_register internally calls luaL_openlib) which is handled by the call to testlib() in the Lua script. Unfortunately I wasn't able to find what kind of exception is raised here (access violation?).

Sorry, you've lost me. Are you saying I should discard the precompiled base64.dll yuu supplied and do something with the project instead? At present that won't compile, because although you said it was complete in itself, it contains:

#include "lua.h"

#include "lualib.h"

#include "lauxlib.h"

and I'll need to download those originals and put them into your project for you. My versions are specific to the FSUIPC implementation (though obviously almost identical).

I think that "package" is the only part that may require this internal access. This includes all the functions required to initialize a module. But a module may want to call some other functions sad.gif.

About the exports I really don't know right now. Perhaps, instead of adding the Lua code in FSUIPC you can link to the Lua DLL and create external modules for "ipc" or "gfd" for instance.

Uh, No way. I want everything integrated. Such an implementation would best be done by a third party using the standard FSUIPC interface, just like any other application.

Hopefully this helps. I'm sorry but I believe I'm far behind you regarding Lua internals coding smile.gif.

Well, not really. I've only skimmed the surface. I don't understand much of the internals, and not even the externals about this module / package part -- you obviously know a lot more about that part than I. I've concentrated on providing tools for Lua programmers to do things in FSUIPC, not really realising that the world of external C modules would be such a problem.

I'm not sure how to proceed at present. I'm not getting anywhere with base64.dll debugging so far. I'd need to understand more about what is going on, and it is horrible convoluted.

It would be a lot easier to do it my way, a different way, providing an FSUIPC Lua library which allows Lua programs to register a DLL and make the calls to it under FSUIPC control, as it does for all the other libraries I've added. It would mean that C modules for use with FSUIPC would be specifically built ones, not ready made ones.

Regards

Pete

Posted

I'm not sure how to proceed at present. I'm not getting anywhere with base64.dll debugging so far. I'd need to understand more about what is going on, and it is horrible convoluted.

Okay. I'm re-reading Chapter 15 in the Lua Programming book (the one I skimmed through a few years ago), and I am starting to appreciate some of the processes needed to get this to work. From what I've read so far, it should work okay, so I'm going to persevere a bit more. Looks like i need to track it through step by step:

"require" checks "package.loaded", then "findloader", and the loader. The base64.dll is actually getting loaded, so either it is failing somehow after that, when being called, or whatever it is returning (if anything) is being assumed to be something it is not.

I can see that if I get the main mechanics working I shouldn't really need to have FSUIPC export anything. The user would have to use the static or dynamic Lua libraries as they wish. (Seems your base64.dll uses both, because the imports from the lua DLL do not include the vital functions needed to build the table of its module functions and exported variables -- although I suppose they can be built in ordinary C if you know the structure.

Any comments so far?

I'm on holiday after Wednesday, not back till early June, so I doubt i'll solve this is any short period.

Regards

Pete

Posted (edited)

Odd then that when I look at its imports in PE Explorer the "Base64.dll" module you supply tells me it needs Lua5.1.dll to import several functions:

Sorry if I was unclear about that. "base64.dll" is extracted for the Official Lua distribution. Indeed, this module use the dynamic library.

I have put the Lua DLL in the same folder as the base64.dll module and it doesn't work either.

Hmmm. The state structure is of course owned by FSUIPC. I added some stuff to the end but the part used as standard should be manipulatable.

Indeed, it should be ok since thre is no aligment consideration with Windows dev AFAIK.

The base64.dll you supplied (test#1) is crashing with an access violation with some daft address (0x000023E0), but the point in the code identified varies slightly.

Looking at this low address, this sounds like a relocation problem. May be the globals are not properly relocated and so, the structure owning the function pointers isn't accessible. I'll do sometest to check this putting the structure in a function returned value instead of a global with the code from test#2.

EDIT: Just tried to return the opcodes table from a function returned value and it doesn't work better.

Sorry, you've lost me. Are you saying I should discard the precompiled base64.dll yuu supplied and do something with the project instead?

Yes, sorry again. test#1 use an official (let's call it like this) Lua module. This one is linked to the Lua DLL.

The project use a static library.

At present that won't compile, because although you said it was complete in itself, it contains:

Weird since I use relative paths for lib/include location. Everything is in the project in the Libs folder.

"require" checks "package.loaded", then "findloader", and the loader. The base64.dll is actually getting loaded, so either it is failing somehow after that, when being called, or whatever it is returning (if anything) is being assumed to be something it is not.

What I saw too. The DLL is loaded and Lua does have a pointer to it but the initialisation code failed to be called. I don't think it will be easy to debug this with base64.dll since you don't have the symbols. Using test#2 should do the trick. The function in this project is "luaopen_TestLua()". Again I'm sorry because I still unable to attach to the process because FS2004 refuse tolaunch when a debugger is running....

I can see that if I get the main mechanics working I shouldn't really need to have FSUIPC export anything. The user would have to use the static or dynamic Lua libraries as they wish.

I do agree with that.

I'm on holiday after Wednesday, not back till early June, so I doubt i'll solve this is any short period.

No rush :) take your time. I'm sure I'm the only one, as least for now, to have this problem and I can deal with it for now. I can easily (I think) create my module with the Lua distro and wait for the package loading to be fixed in FSUIPC. I just hope it can be so that I don't work on that for nothing :). But anyway I'll do my best to help

Thank you again

Chris

Edited by Chrilith
Posted

Sorry if I was unclear about that. "base64.dll" is extracted for the Official Lua distribution. Indeed, this module use the dynamic library.

I have put the Lua DLL in the same folder as the base64.dll module and it doesn't work either.

Same here.

Indeed, it should be ok since thre is no aligment consideration with Windows dev AFAIK.

Now here's a definite problem. The Lua state structure is decidedly wonky, in the sense that one byte values jostle with 4 and 8 bytes values with no consideration for alignment. I always compile with 1 byte alignment (ie. no padding), and simply ensure that I put spare bytes in where I want efficient word or dword or double alignment. I must use 1 byte because FS is compiled that way -- in fact most of it's internal doubles and so on are on odd boundaries. Obviously they cared more for size than efficiency.

The default MSVC alignment is different I think in different versions. The default in VC2010 seems to be 8 byte , as best as I can find out -- it doesn't make it obvious anywhere.

I did actually put some spare bytes into the Lua state to get it aligned properly. But I've tried patching them out and I still get the same error with "base64.dll". But it's going to depend entirely on what options are chosen when compiling the DLL, since there are no "pragmas" to force a fixed defined alignment! So, how is it supposed to work? Something doesn't make sense!

Weird since I use relative paths for lib/include location.

I don't have one for Lua.

Again I'm sorry because I still unable to attach to the process because FS2004 refuse tolaunch when a debugger is running....

Attach AFTER FS is launched, as I advised. FS doesn't re-check whilst it is running.

[LATER]

I've tried using "pragma pack(n)" and "pragma pack(1)" bracketing all of the Lua structs in the lstate.h header, with n=8, 4, and 2, but I get exactly the same error with all. Shame. I felt sure it had to be an alignment mismatch -- I suppose it still might be if there are some more weird structs in the other headers. I'll have a quick look, but at present it seems like a pretty tough brick wall I'm banging my head against.

It certainly calls the "base64.dll" exported function at the correct place and with the correct pointer to the State structure, but when it gets back into FSUIPC something is really screwed up good and proper. I was going to compare the state struct before and after -- with added code, not manually -- but first I'll move onto that test project you supplied. But I'm going to need to compile a debug version of the lib file if I'm going to find the discrepancy.

Regards

Pete

Posted

I did actually put some spare bytes into the Lua state to get it aligned properly. But I've tried patching them out and I still get the same error with "base64.dll".

I wasn't aware of this aligmnent stuff on windows. I was facing similar problems working on PalmOS. I think i do too much C# coding now.

Adding padding is indeed a good idea but we also have to deal with endianess. I always have a problem to know if I need to put this extra byte before or after the struct member :), don't know why...

I don't have one for Lua.

I meant there is a Libs folder containing all required files in the VC project I sent you.

Attach AFTER FS is launched, as I advised. FS doesn't re-check whilst it is running.

Well, I'll try again

Chris

Posted

But I'm going to need to compile a debug version of the lib file if I'm going to find the discrepancy.

I'll send you the VC2010 project I used to build the Lua lib later today if you can't find it on sourceforge. I'mcurrently not at home.

Posted

I wasn't aware of this aligmnent stuff on windows.

It's a pain.

The problem I have now is that I need a debug version of the lua5.1.lib so I can trace through its copy of the same code I'm using to see where it is corrupting my data, because corrupting it it certainly is. But if I compile my own copy of the lib in debug mode it will compile with all the structs aligned with my settings, so it may then work -- but that defeats the object.

So, have you got, or can you compile, a lua5.1.lib for debugging, check there that it works in a default Lua environment when bound into your test program, and then send it to me?

There's another slight worry which has occurred to me. Here I have two source versions of Lua, 5.1.3 and 5.1.4. I'm pretty sure the stuff in FSUIPC is based on 5.1.4. But how will different Lua versions affect this? Can we assume they never change the structs in a way which would affect compatibility?

Adding padding is indeed a good idea

The compile does that if the packing isn't 1. I only added padding because i must use 1. And it isn't good if it makes my Lua structs mismatch those used by external modules.

... but we also have to deal with endianess. I always have a problem to know if I need to put this extra byte before or after the struct member smile.gif, don't know why...

Endianness really is about whether numerical values have first byte lo (lo-hi) or last (hi-lo). Intel is always lo-hi. Not sure what the worry about struct padding involves endianness. The padding is placed to ensure the next member is aligned on a boundary appropriate to its type.

I meant there is a Libs folder containing all required files in the VC project I sent you.

No, the project you sent me contained 4 files in the "TestLua" folder and a Debug subfolder containing a load of files, mostly "tlog"s whatever they are.

Regards

Pete

Posted

So, have you got, or can you compile, a lua5.1.lib for debugging, check there that it works in a default Lua environment when bound into your test program, and then send it to me?

Just sent you the lib code.

But how will different Lua versions affect this? Can we assume they never change the structs in a way which would affect compatibility?

I do think so, at least from 5.1.3 to 5.1.4

Not sure what the worry about struct padding involves endianness. The padding is placed to ensure the next member is aligned on a boundary appropriate to its type.

I was talking about artificial padding adding dummy members to the struct. Not compiler or preprocessor options.

No, the project you sent me contained 4 files in the "TestLua" folder and a Debug subfolder containing a load of files, mostly "tlog"s whatever they are.

Oops, sorry. Sent again.

Posted

Just sent you the lib code.

Received two ZIPs, thank you, but in the lua one the vital folders, "Bin" and "Lib\static" seem to be ... empty? There's no ready-built debug LIB for me to build with. I can build one here, but then it'll be built with my settings.

Additionally, the full TestLua project doesn't actually include a compiled TestLua.DLL at all,. The TestLua\Debug folder is empty, and the TestLua\TestLua\Debug folder contain all the right stuff except the DLL. Really weird. It's as if something is removing anything which looks slightly like a program!

Regards

Pete

Posted

I'll send you both files.

Thanks.

I've put the larger of the two Lua5.1.lib files into your project and compiled it with it, in Debug mode, but whilst I get the source debugging in your "TestLua.c" code, the actual crash is with a corrupted stack top pointer in the State structure, occurring someplace pretty deep inside the Lua Lib in your program as a result of the "luaL_register" call (yes, with the correct state pointer, L) -- but it isn't showing me source for some reason, so stepping through it tells me nothing. I guess the LIB has to be compiled here or else it doesn't know where the source is? It doesn't complain though. Weird. I obviously don't know enough about MS's "studio" product.

I'm actually getting no where at present. I think I'll give it a rest till tomorrow at least.

Aren't you able to narrow it down there? You must surely be able to attach the MSVC debugger to the running FS9 process. I've always been able to!

Regards

Pete

Posted

Aren't you able to narrow it down there? You must surely be able to attach the MSVC debugger to the running FS9 process. I've always been able to!

Haven't tested again yet sorry. Didn't find time today :(

I'll try tomorrow when I get back from work.

Thanks!

Chris

Posted

Haven't tested again yet sorry. Didn't find time today :(

I'll try tomorrow when I get back from work.

Okay.

I did a bit more this evening. I think there is/was a struct alignment issue, and if folks compile their own Lua modules with different settings they are quite likely to be incompatible. I've not yet worked out whether it should be 4 or 8 byte alignment. Older versions of the compiler worked on the 32-bit principle of alignment, but recent versions default to 8 (64-bit0. It makes a difference to several structs, including the state. I'm testing with 4-byte (32-bit) at present.

However, the crash I'm now getting in TestLua.dll is inside luaI_openlib, called by the register call, and it happens where it calls a C library DLL API, "strchr", in the luaL_findtable function. And I'm not surprised ...

... the debug LIB you supplied, which I'm binding into your TestLua and compiling, imports a load of stuff from MSVCR100D.DLL and from KERNEL32.DLL. But the links to these are not made. Somehow Lua is managing to load the module in such a way that none of its external dependencies are being filled in! The result, obviously, a crash as it jumps into nowhere in particular!

I've no idea, yet, how this is supposed to work. It seems completely daft.

Note that nothing else is loading MSVCR100D.DLL either. FSX obviously predates this (it's on 80), and FSUIPC uses static versions to avoid versioning problems on different platforms.

Regards

Pete

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use. Guidelines Privacy Policy We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.