Lua for pixel analysis

[ckovoor]

Hi Pete,

I have recently started using an Arduino Mega to display outputs from FSX on LED's + 8-digit, 7-segment LED modules + 1602 LCD's. This is to augment (but not replace) my touchscreen-based simulator whose cockpit comprises 2-D panels of payware add-on aircraft across 6 touchscreen monitors. My present project is to construct as many different hardware MCP's as possible. For the inputs I use encoders and switches driven by Bodnar's BU0836X and Pokeys 56U in conjunction with FSUIPC Mouse Macros/aircraft-specific key assignments. This is a robust and stable solution and I will not change it. It is for the hardware MCP outputs (alone) that I am using Arduino.

(1) For the PMDG 737 NGX I am using this scheme, made possible by your detailed listing of FSUIPC offsets for the aircraft:
PMDG737NGX --> FSUIPC --> Jim's Link2FS --(USB/COM)--> Arduino --> LED's

(2) This also worked for the Level-D B767 as Nico Kaan's Lekseecon_f bridges the gap between the aircraft and FSUIPC:
LD B767 --> Lekseecon_f --> FSUIPC --> Link2FS --(USB/COM)--> Arduino --> LED's

(3) The first challenge came with the SimCheck A300B4. This aircraft uses standard SimConnect variables/FSUIPC offsets for MCP digital outputs of SPD/ALT/HDG etc, so displaying these outputs followed the same scheme as for the NGX. But offsets for the connected annunciator lights are not available. So I took a backdoor route:
- wrote an Autoit routine to detect pixel colour at the each on-screen annunciator location (every 100 ms) and assess whether that light was on or off, and write the results as a string to a .txt file
- wrote a lua plugin to read that .txt file (every 100 ms), and write the annunciator states to 15 free FSUIPC offsets starting at 0x66C0
- Link2FS was then programmed to read these offsets and transmit them to the Arduino via the USB/COM serial port

SC A300B4 --> Autoit --> FSUIPC --> Link2FS --(USB/COM)--> Arduino --> LED's
A full description of the method and photographs is at http://www.mycockpit.org/forums/showthread.php?t=28888
The method works, and there is very little lag between the operation of a switch and the response of the hardware.

(4) I then started work on the PMDG MD-11X MCP. This MCP has no annunciators as such, but it does have digital readouts. But, as you know, there are no available FSUIPC offsets at all for this aircraft, so I did the following:
- wrote an Autoit routine to detect critical pixel colours, one at each segment of each digit of the readouts (every 100 ms) to detect its state, analyze all the information across the 15 digits to produce an 'image' of what digits were displayed, and then transmit this information to the Arduino directly via the COM port.
Autoit has advanced window/pixel management functions as well as functions for serial communications, so everything was bundled into one script. Link2FS and FSUIPC are not involved.

PMDG MD-11--> Autoit ---(USB/COM)--> Arduino --> LCD's
I have attached a photo of the screen version of the MCP and its 'real-time' LCD equivalent (I would have uploaded a video if I had the means to produce a good one). Anyway, it works! And there is no noticeable lag between the parent display and its hardware 'derivation'.

(5) My next project will be the PMDG B747-400X MCP.Here the digital readouts are not 7-segment LED's but more like mechanical counters, so I will probably have to analyze more pixel states, and there are illuminated switches to consider as well. So it will be a larger project.

 

The method proposed is a solution for building hardware around add-on aircraft where offsets for the outputs are not available. The caveat is that one would need to have a screen running from which the data is to be extracted, and the panel windows cannot be hidden.
I find that this idea has been mooted before, though quite a while ago: http://forum.simflight.com/topic/58052-solution-to-detecting-light-state-changes-in-pmdg-overhead/?hl=%2Bpixel+%2Bcolour
I wonder how he has gotten on in the interim.
 
So far I have been using Autoit, and I have to say it works very well. However, FSUIPC does have a (lua) COM library, so communicating with the Arduino is certainly possible. So I am wondering if you might consider adding facilities for pixel analysis (pixel colour detection) at FS window locations via lua. I know that such libraries exist out there, and it would be a boon (to hardware cockpit-builders adopting this method) to be able to handle all this within FS/FSUIPC.

It's just my idea, and I look forward to your opinion.

Thank you for all that you are doing to support and enhance our hobby.

Warm Regards,

Chakko.

[Pete Dowson]

So far I have been using Autoit, and I have to say it works very well. However, FSUIPC does have a (lua) COM library, so communicating with the Arduino is certainly possible. So I am wondering if you might consider adding facilities for pixel analysis (pixel colour detection) at FS window locations via lua. I know that such libraries exist out there, and it would be a boon (to hardware cockpit-builders adopting this method) to be able to handle all this within FS/FSUIPC.

 

Hmm. Interesting and ambitious. But to me it does seem a little self-defeating. Surely the point of having the indicators and so on on real hardware is so that you can get rid on the on-screen stuff? For this reason I don't really see such facilities as "a boon".  My projected screen for my cockpit shows just the outside world.

 

If you have a library which works, then wrap it into a Lua-compatible interface and provide some documentation and I'll either include a link to it in the Lua documentation, or, if there are no copyright attachments, make it downloadable in the Download Links subforum here.

 

Pete

[ckovoor]

Hi Pete,

 

Thank you for your response.

 

Hmm. Interesting and ambitious. But to me it does seem a little self-defeating. Surely the point of having the indicators and so on on real hardware is so that you can get rid on the on-screen stuff? For this reason I don't really see such facilities as "a boon".  My projected screen for my cockpit shows just the outside world.

 

I agree that it would be self-defeating in the case of a B737 hardware cockpit. But that is because all the offsets to drive hardware are available, so one doesn't need this 'backdoor solution'. But consider builders who would like to build say an MD-11 cockpit, and see in the PMDG MD-11 software a superlative basis on which to do so, but are confronted by the TOTAL absence of published or 'discovered' offsets. In that case one could use this method: have the FS PC run a couple of monitors displaying the panels from which one extracts the pixel data I am speaking of, and send the information to the hardware cockpit which takes centre-stage. The monitors displaying the FS panels need not be visible at all; they might in fact be running in a corner of the room, out of view! If that is the 'cost' of building an MD-11 hardware cockpit, I feel there would be many builders who would consider it.

 

The lack of published offsets around reasonably detailed add-on aircraft simulations has been, I believe, a major deterrent to building hardware cockpits other than the B737.

 

So, the method proposed is a solution for building hardware around add-on aircraft where offsets for the outputs are not available. (The caveat is that one would need to have a screen running from which the data is to be extracted, and the panel windows cannot be hidden.)

 

Anyway....continuing my work with Arduino-driven hardware, I have successfully run lua plugins which open the COM port and send the required data to the Arduino. All that now remains to integrate this solution fully within FSUIPC is to find a Lua library with functions to read pixel data off the FS screen 'in real-time'.

 

If you have a library which works, then wrap it into a Lua-compatible interface and provide some documentation and I'll either include a link to it in the Lua documentation, or, if there are no copyright attachments, make it downloadable in the Download Links subforum here.

 

There is a lua package entitled Lua-GD at http://ittner.github.io/lua-gd/manual.html which has a gdImage:getPixel(x, y) function, however, as I understand it, it can only work with pre-loaded images, and not with on-screen displays generally. I have searched elsewhere, but so far have been unsuccessful in locating an appropriate library and function.

 

So if you, or anyone else knows of any such Lua library, or if there is some other way to build this functionality into FSUIPC Lua, I for one will be extremely grateful. :razz:

 

Thank you for your help and patience!

 

Warm regards,

 

Chakko.

[Pete Dowson]

The lack of published offsets around reasonably detailed add-on aircraft simulations has been, I believe, a major deterrent to building hardware cockpits other than the B737.

 

There have been many solutions found, usually via local panel variables (L:Vars) or "mouse macros". I believe the latter method was quite successful on the PMDG 737 and the only way before they published their SDK.  Browse through the User Contributions subforum here and see what folks have managed.

 

Also I think generally most cockpit builders who might be so serious they'd set up extra monitors for displays they are not looking at would more likely opt for one of the more fexible cockpit packages such as ProSim737, Sim-Avionics, or Project Magenta (and there are others), which actually emulate the full systems much more accurately and allow use of a much more VAS and frame-rate friendly cockpit-less aircraft.

 

There is a lua package entitled Lua-GD at http://ittner.github.io/lua-gd/manual.html which has a gdImage:getPixel(x, y) function, however, as I understand it, it can only work with pre-loaded images, and not with on-screen displays generally. I have searched elsewhere, but so far have been unsuccessful in locating an appropriate library and function.

 

Quite honestly, graphics and screens is not a subject I'm at all familiar with, and even if I had time I wouldn't know where to start. If you yourself are not a programmer I think you need to find someone who is and who is willing to take this up.

 

Regards

Pete