HID devices in Lua

[Artem Crum]

Hello again, Pete

Other topic and other question now..

I'm using FSUIPC/Lua to read data from HID devices (those com.openhid and other commands). My script is based on your example provided with Lua library documentation (hiddemo.lua). Everything works fine, but sometimes I get some kind of strange behavior.

My system consist of 3 CH devices - Eclipse yoke, Rudder pedals and Pro Throttle quadrant. All of them are working in programmed mode (not direct... if you know what it is in CH logic, but I don't think it really matter).

And from time to time (quite often) I see a noticeable delay about half of second between button press and HID library catch of it. This happens only on Pro Throttle (no ever delay happened on Eclipse). Then, if I move throttle axises somewhere from their current position delay disappears (but not always). And then delay could appear again. Eclipse keeps working without delays at the same time.

This delay is could be seen ONLY IN LUA. If I open CH Control panel I see that button presses detected as usual without any delay.

It seems that this delay depends on quadrant axises positions. If I put all of them full forward or full backward, the delay is lower. If all axises are in different positions about the middle, the delay is bigger. Sometime there no visible delay at all, but most of the time there is a feeling of some lag. And from time to time the lag becomes too much.

Do you have any ideas, why this could happen?

[Pete Dowson]

It seems that this delay depends on quadrant axises positions. If I put all of them full forward or full backward, the delay is lower. If all axises are in different positions about the middle, the delay is bigger. Sometime there no visible delay at all, but most of the time there is a feeling of some lag. And from time to time the lag becomes too much.

Very strange. some sort of timing causing pile-ups, like motorway traffic queues perhaps?

Are you using a separate Lua plug-in for each device? If not try it, because it might just be too much going on in one thread.

Also, to investigate further, uncomment the part logging the number of discards -- see if there's any correlation between those and the delays. The "readlast" function basically uses a loop (inside FSUIPC, in fast C code of course) to keep reading blocks till there are none left, and supplies the last one. If there's a lot of axis activity the button press which occurred a number of discarded blocks ago could be delayed, but half a second seems a little on the high side.

You could also see what happens by using "read" instead of "readlast" -- it would probably work better for button presses, but worse for axes where values are changing far too quickly for a humble interpreter to catch all of them.

Windows joystick drivers just store values and supply them on poll calls. You could try revising the code to do something similar. Just have one Lua plug-in storing values every n milliseconds to Globals, and one or more separate Lua threads reading values at a normal poll rate (FS's is 6/sec, FSUIPC's default is more like 20/sec). By having them in separate threads you might avoid a pile up in the one reading and storing values.

Regards

Pete

[Artem Crum]

Oh! So many ideas to play with :-)

I'll try to see the discards number and try the "read" instead of "readlast" because I don't need axises values at all (except for hats, but they behave like buttons). Will report the results. Thanks.

[Artem Crum]

Ok. Here are the results...

1. Using com.read instead of com.readlast makes things much worse on all devices

2. Discards counter shows values about 1-2 on problem device and 0 on other devices

3. CH Throttle axises are constantly jittering and it seems that this is the main problem - they are constantly send new values and this lead to the delay. As I said when all axises are full forward or full backward delay is reduced and this is because in extreme positions there is no jitter on axises values.

Also this problem was confirmed on other system with completely different set of devices. Older and now I'm suppose jittering Logitech Extreme device was working with the same delays.

So the conclusion is that while devices sending a lot of updates about axises values something inside FSUIPC HID library chokes and this cause button reaction delay.

Could it be changed somehow?

Does or can the FSUIPC axis filtering affect this (remove jitter I mean)?

[Pete Dowson]

So the conclusion is that while devices sending a lot of updates about axises values something inside FSUIPC HID library chokes and this cause button reaction delay.

Not sure how it "chokes". It simply does normal serial port reads. Maybe yu need the "GetInputReport" method for such devices, but when I experimented with that it didn't work well at all for fast polling (i.e. for smooth axes).

Could it be changed somehow?

I don't know. I'll have to think about it. Sorry, though, no time this week. Can you prompt me again next week?

Does or can the FSUIPC axis filtering affect this (remove jitter I mean)?

No. The HID facilities are bypassing all Windows joystick interfaces, drivers, everything. They are reading the raw data on the wires.

If all you want is buttons you'd be better off using the normal routes. The HID facilities in Lua were for taking advantage of features not otherwise supported -- like many more buttons, many more axes, and even control of displays if you can decode the specific protocols.

Regards

Pete

[Artem Crum]

I don't know. I'll have to think about it. Sorry, though, no time this week. Can you prompt me again next week?

Yes, sure.

If all you want is buttons you'd be better off using the normal routes. The HID facilities in Lua were for taking advantage of features not otherwise supported -- like many more buttons, many more axes, and even control of displays if you can decode the specific protocols.

This could be the way out... I stick with HID- approach because the GUI part of my application implements it's own internal "driver" for HID devices. And it was the easiest way to sync list of devices in GUI and Lua parts of it. In GUI I make device ennumeration and differentiate them by VendorID/ProductID (exactly as you do). If I change the lua part to work with common event.button approach then I need the way how to convert VID/PID to exact device number for event.button call.

The other problem could happen that i. e. Saitek X52 has 3 hats. One normal/common Hat Switch and two others with custom usage codes but with the same logic of bit encoding (they are Values, not the Buttons in HIDs notation). Will the event.button work with them correctly? And how I can know what values will event.button send to the handler function for these additional hats?

I need some kind of universal approach which will work with any types of joysticks and the HID library suits this goal the best.

[Pete Dowson]

This could be the way out... I stick with HID- approach because the GUI part of my application implements it's own internal "driver" for HID devices. And it was the easiest way to sync list of devices in GUI and Lua parts of it. In GUI I make device ennumeration and differentiate them by VendorID/ProductID (exactly as you do). If I change the lua part to work with common event.button approach ...

Well, let me see if I can sort out the HID method first. Sorry I can't get to it straight away.

... then I need the way how to convert VID/PID to exact device number for event.button call.

Yes, it's a registry thing, and different between XP and Vista/Win7.

The other problem could happen that i. e. Saitek X52 has 3 hats. One normal/common Hat Switch and two others with custom usage codes but with the same logic of bit encoding (they are Values, not the Buttons in HIDs notation). Will the event.button work with them correctly? And how I can know what values will event.button send to the handler function for these additional hats?

No, currently the buttons are only extended from 32-39 for the first Hat (POV). FSUIPC4 supports two POVs for axis assignment, but FSUIPC only has the one. This is because FSUIPC3 uses the old "joy" API whilst FSUIPC4 uses DirectInput 8. Only later DirectInput versions supported 3 or more Hats.

I need some kind of universal approach which will work with any types of joysticks and the HID library suits this goal the best.

We'll see if we can make it work better.

Regards

Pete

[Bob Church]

Hi Pete,

>> No. The HID facilities are bypassing all Windows joystick interfaces, drivers, everything. They are reading the raw data on the wires. <<

The CH controllers report every 10 ms. The CM drivers suppress the duplicates, but if you're tapping into the data at the top of the USB stack then the CM drivers aren't in the loop and you're looking at 100 reports per second per controller or 300 reports per second with the three controllers that the OP mentioned. The scripting would need to check for and discard the duplicates. If you think it's a "traffic jam", that's probably where it's coming from.

Best regards,

- Bob

The StickWorks

http://www.stickworks.com

[Artem Crum]

Bob,

Nice to meet you here (we talked earlier on CH-Hangar).

The problem with these delays does happen not only with my CH Throttle, but with Logitech and I believe with any other device with jittering axises. And more the axises device have leads to more jittering and the bigger "traffic jam" (bigger delays).

By the way. Is it normal that CH Throttle axises are so unstable?

Values constantly jumping +/-1 points from the current position.

Eclipse axises are rock-solid in comparision.

[Pete Dowson]

Well, let me see if I can sort out the HID method first. Sorry I can't get to it straight away.

I've just noticed that I implemented a different way to read the HID data -- "ReadReport".

data, len = com.readreport(devicehandle) for report ID 0, else

data, len = com.readreport(devicehandle, id) for other report IDs.

When testing here they didn't appear to work correctly (it calls the HidD_GetInputReport function in Windows). So i left it out of the documentation. But try it in place of "com.readlast" and see what you get. I won't have time to test it till the end of the week.

Regards

Pete

[Pete Dowson]

The CH controllers report every 10 ms. The CM drivers suppress the duplicates, but if you're tapping into the data at the top of the USB stack then the CM drivers aren't in the loop and you're looking at 300 reports per second for three controllers. The scripting would need to check for and discard those

This is why I added the "readlast" function, which takes it out of slow scripting and puts it into a fast C loop. I'd like do be able to do it all in one read, but it seems the HID code in Windows doesn't like that so I have to loop reading the input report size instead. Still, it should be fast in C, and it's in its own thread.

Maybe, conscious of affecting FS performance, I yield too much in the loop. I doubt it, but I'll check. My normal yields are by "sleeps" of 0 or 1 mSec. Which should be fine.

Regards

Pete

[Artem Crum]

data, len = com.readreport(devicehandle) for report ID 0, else

data, len = com.readreport(devicehandle, id) for other report IDs.

When testing here they didn't appear to work correctly (it calls the HidD_GetInputReport function in Windows). So i left it out of the documentation. But try it in place of "com.readlast" and see what you get. I won't have time to test it till the end of the week.

Have it tried by simple replacing readlast with readreport. Doesn't work. Len is always a Zero value.

Take your time if your are busy. These delays are not a critical issue at the moment.

[Bob Church]

Hi Pete,

It could be. The buffering isn't done in the drivers, though. HIDClass does it. It pends a read on the device, so the downstack trip (HIDClass -> HidUsb -> USBD and it's minions) have the data packet and just hold on to it. When the controller has data, it fills in the packet and sends it back up to HIDClass, DInput gets it's data there from that buffer in HIDClass. When HIDClass gets the packet back it immediately sends another one down, the round trip from the controller driver up to HIDClass and back is very fast. Less than a uS as I recall. On a PIII, the stuff usually ate maybe 3% of the CPU, I'd guess it's essentially negligible with today's multicore stuff. I don't know if that's any help, I'm not sure what you're doing, but FWIW. Maybe it will suggest something.

To the OP: +/- 1 is as good as it gets with any digital system. They don't do fractions, so if the setting should result in something like 127.5, for example, it will show you 127...128...127...128... to get it to average out to the 127.5 value that it's trying to represent.

Best regards,

- Bob

The StickWorks

http://www.stickworks.com

[Pete Dowson]

Have it tried by simple replacing readlast with readreport. Doesn't work. Len is always a Zero value.

What report ID are you using? Maybe it's not 0 but 1?

Regards

Pete

[Artem Crum]

As I'm working with several HID devices at once, I'm initially filling the array like this:

jdev[jid], rd[jid], wrf[jid], wr[jid], init[jid] = com.openhid(VID, PID, UID, Report)

And then I'm using this array data to make the com.readlast calls in the For Loop.

CurrentData, n = com.readlast(jdev, rd)

As you can see rd contains the data which was recieved on init stage from openhid call.

Actual values in rd table ar 11 for one device and 13 for another. (Is it normal?)

If I try to replace rd with 0 or 1 it doesn't work also.

[Pete Dowson]

As you can see rd contains the data which was recieved on init stage from openhid call.

Actual values in rd table ar 11 for one device and 13 for another. (Is it normal?)

It represents the size of the input report. It varies per device. I've seen as short as 2 and as long as 40.

If I try to replace rd with 0 or 1 it doesn't work also.

Of course not. You missed the definition of this in my earlier message?

The "readreport" function doesn't use a length -- you get the fixed length it provides, and like it! The second parameter is a Report ID. It's a guess. I think most devices use 1. Some devices have multiple report types, for instance for calibration data stuff like that.

Pete

[Artem Crum]

Yes, Pete, I'm missing something here.

I'm pure novice in all these HID- things. Ok. If readreport requires the report id as the second parameter, is this the same report id which is required in openhid call as the fourth parameter?

Then if I do the readreport call with correct device handle which is recieved from the openhid, then why I'm receiving the len == 0 no matter what report id I set to it?

I've tried both:

CurrentData, n = com.readreport(jdev)

CurrentData, n = com.readreport(jdev, 1)

In both cases n == 0 and CurrentData seems empty.

[Pete Dowson]

I'm pure novice in all these HID- things. Ok. If readreport requires the report id as the second parameter, is this the same report id which is required in openhid call as the fourth parameter?

Yes, though "required" is too strong. That is only used to do an initial 2readreport" to provide initial settings for toggle switches and axis positions, if the device cares to provide such things.

Then if I do the readreport call with correct device handle which is recieved from the openhid, then why I'm receiving the len == 0 no matter what report id I set to it?

Possibly because the device doesn't support readreport -- it is optional.

Shame. it would have been a quick answer if it had worked. I'll have to think more -- but it'll be next week.

Regards

Pete

[Pete Dowson]

I've tried both:

CurrentData, n = com.readreport(jdev)

CurrentData, n = com.readreport(jdev, 1)

In both cases n == 0 and CurrentData seems empty.

I just realised (and re-reading my own OpenHid documentation) that the default Report ID is actually 1, and there's a comment in the "hiddemo.lua" saying "I *think* all joystick types use Input Report 0", so it would be worth your while trying

CurrentData, n = com.readreport(jdev, 0)

Or more generally

CurrentData, n = com.readreport(jdev, Report)

because, at least in the version of the Lua I supplied, "Report" is set to the correct Report number at the top of the program.

I'm just trying it here ...

The other way of speeding things up might be to have separate Lua threads for each device. The problem with only having the one trying to handle several devices is that whilst the thread is running it won't receive more timing events, and lost ones are lost. I don't queue events. If you wanted a central control, to determine connected devices etc, you could have the one Lua running the others (by RunLua) but you'd need to have each with a unique name as FSUIPC only allows uniquely named plug-ins to run together.

Maybe there's a requirement for a plug-in to be able to spawn separately threaded copies of itself. There are some threading facilities in Lua, but I doubt they work in FSUIPC's environment (not tried them), and in any case that wouldn't deal with the event reception issue as it would only look like one to FSUIPC.

Regards

Pete

[Artem Crum]

Yes, I remember the comments inside hiddemo.lua, but com.readreport(jdev, 0) gives just the same result anyway: n == 0. I've tried it yesterday.

[Pete Dowson]

Yes, I remember the comments inside hiddemo.lua, but com.readreport(jdev, 0) gives just the same result anyway: n == 0. I've tried it yesterday.

Ah, shame. Did you see my amendments, about separate threads?

I've been looking at the code for ReadLast, and there are no delays or sleeps. It doesn't actually do the "reads" itself, all serial port reads are done in separate read threads, one for each device, which simply loop reading when there's data. Those are high priority threads, and work well for all comms devices not just HID. So I don't think there's any inefficiencies in my code.

I think this seems to be confirmed also by the low number of discards you reported.

2. Discards counter shows values about 1-2 on problem device and 0 on other devices

If the devices are sending frames at 10 mSec intervals, the worst case of 2 discards should only mean a max delay of 30 mSecs.

I will try digging a little deeper, but I think this is looking to be a Windows thing. This method of handling devices just isn't so good when they are jittering. The regular joystick drivers are working at a lower level and presumably bypass where the bottleneck is occurring.

Regards

Pete

[Artem Crum]

Ah, shame. Did you see my amendments, about separate threads?

At the moment I cannot imagine how to implement this in my case. If only to automatically generate the separate lua script for each HID-device, run them and make them interface with the main process through the global vars...

But I don't think, it will really help. I see the same delays even if I unplug all other HID-devices except the CH Throttle.

[Pete Dowson]

But I don't think, it will really help. I see the same delays even if I unplug all other HID-devices except the CH Throttle.

I agree, because the discard figures show as much. If it was catching up with a backlog I'd expect those to be much higher. I guess something much lower down in the heirarchy is discarding stuff before FSUIPC if there's only 1-2 discards for a 500 mSec delay!

Sorry, I'll keep thinking and looking but I can't hold out much hope.

Regards

Pete

[Artem Crum]

Here is a part of debug log from console:

 1141584 LUA: LINDA:: 1436953176 -- 1
 1141630 LUA: LINDA:: 1436953176 -- 2
 1141677 LUA: LINDA:: 1436953176 -- 1
 1141724 LUA: LINDA:: 1436953176 -- 1
 1141771 LUA: LINDA:: 1436953176 -- 2
 1141818 LUA: LINDA:: 1436953176 -- 1
 1141864 LUA: LINDA:: 1436953176 -- 1
 1141942 LUA: LINDA:: 1436953176 -- 4
 1141958 LUA: LINDA:: 1436953176 -- 0
 1142005 LUA: LINDA:: 1436953176 -- 1
 1142052 LUA: LINDA:: 1436953176 -- 2
 1142098 LUA: LINDA:: 1436953176 -- 1
 1142145 LUA: LINDA:: 1436953176 -- 1
 1142192 LUA: LINDA:: 1436953176 -- 2
 1142239 LUA: LINDA:: 1436953176 -- 1
 1142286 LUA: LINDA:: 1436953176 -- 1
 1142332 LUA: LINDA:: 1436953176 -- 2
 1142379 LUA: LINDA:: 1436953176 -- 1
 1142426 LUA: LINDA:: 1436953176 -- 1
 1142473 LUA: LINDA:: 1436953176 -- 2
 1142520 LUA: LINDA:: 1436953176 -- 1
 1142566 LUA: LINDA:: 1436953176 -- 1
 1142613 LUA: LINDA:: 1436953176 -- 2
 1142660 LUA: LINDA:: 1436953176 -- 1
 1142707 LUA: LINDA:: 1436953176 -- 1
 1142754 LUA: LINDA:: 1436953176 -- 2
 1142800 LUA: LINDA:: 1436953176 -- 1
 1142847 LUA: LINDA:: 1436953176 -- 1
 1142894 LUA: LINDA:: 1436953176 -- 2
 1142941 LUA: LINDA:: 1436953176 -- 1
 1142988 LUA: LINDA:: 1436953176 -- 1
 1143034 LUA: LINDA:: 1436953176 -- 2
 1143081 LUA: LINDA:: 1436953176 -- 1
 1143175 LUA: LINDA:: 1436953176 -- 4
 1143175 LUA: LINDA:: 1436953176 -- 0
 1143222 LUA: LINDA:: 1436953176 -- 1
 1143268 LUA: LINDA:: 1436953176 -- 2
 1143315 LUA: LINDA:: 1436953176 -- 1
 1143362 LUA: LINDA:: 1436953176 -- 1
 1143409 LUA: LINDA:: 1436953176 -- 2
 1143456 LUA: LINDA:: 1436953176 -- 1
 1143502 LUA: LINDA:: 1436953176 -- 1
 1143549 LUA: LINDA:: 1436953176 -- 2
 1143596 LUA: LINDA:: 1436953176 -- 1
 1143643 LUA: LINDA:: 1436953176 -- 1
 1143690 LUA: LINDA:: 1436953176 -- 2
 1143736 LUA: LINDA:: 1436953176 -- 1
 1143783 LUA: LINDA:: 1436953176 -- 1
 1143830 LUA: LINDA:: 1436953176 -- 2
 1143877 LUA: LINDA:: 1436953176 -- 1
 1143924 LUA: LINDA:: 1436953176 -- 1
 1143970 LUA: LINDA:: 1436953176 -- 2
 1144017 LUA: LINDA:: 1436953176 -- 1
 1144064 LUA: LINDA:: 1436953176 -- 1
 1144111 LUA: LINDA:: 1436953176 -- 2
 1144158 LUA: LINDA:: 1436953176 -- 1
 1144204 LUA: LINDA:: 1436953176 -- 1
 1144251 LUA: LINDA:: 1436953176 -- 2
 1144298 LUA: LINDA:: 1436953176 -- 1
 1144376 LUA: LINDA:: 1436953176 -- 3
 1144392 LUA: LINDA:: 1436953176 -- 0

1436953176 -- is a device handle, and the number after "--" is a discards counter.

If I remove jittering by moving all axises to the extreme position, discards drop to 0 at the same moment.

[Pete Dowson]

If I provided a way in Lua for you to relate joystick names, GUIDs, VIDs or PIDs to the FSUIPC joystick number, would you be able to move to using the joystick interface instead?

The names and GUIDs are actually available in the FSUIPC INI file of course, but it would be easy for me to return these given the unit number (0-15). Alternatively I could get the VID and PID hexadecimal numbers instead.

The problem of course is then multiple POVs and more than 64 buttons. To extend that for FSUIPC general assignment would be a nightmare, but it might be possible to do it for Lua access. I'd have to check. It might actually be best to implement a new "joy" Lua library -- so not an immediate project, just one I could think about adding.

Hmmm.

Pete