Rotary encoders not working as expected

[aurel42]

Hi Pete & everybody else,

a few days ago, I had a weird issue accessing my controller with a couple of axes, a couple of buttons, and a set of rotary encoders from hiddemo.lua. Pete graciously fixed that issue for me in the latest FSUIPC release. Now I've hit the next obstacle: while the rotary encoders work in principle -- FSUIPC can see them, I can assign FS functions to them, and when I turn a rotary encoder to the left, the assigned thingie often moves to the left -- they don't work reliably and smoothly.

• I'm still experimenting with a stripped down version of hiddemo.lua; I removed the support for axes, but left the button logic mostly untouched. I only use the script for the rotary encoders of my controller (everything else is mapped directly in FSUIPC). I have to use LUA, because the rotary encoder events are mapped to button events beyond the first 32 buttons (in my case, buttons 96 to 111 - and those are the only events I'm interested in handling with LUA).
• My controller uses two kinds of rotary encoder hardware, "normal" rotary encoders and "wedding cake"-style dual rotary encoders.
• For both types of hardware, the output of my device provides four events when turning the rotary encoders: cw slow, cw fast, ccw slow, ccw fast. So I am not using the rotaries.lua script which "emulates" the "fast" events for devices that don't provide them.

All the rotary encoders work in principle, I have assigned all the events to sim actions in FSUIPC's Button dialog.

I'm having trouble making the rotaries behave as they should, though.

This is what I expect:

1. When I turn the rotary encoder assigned to the heading bug so it clicks once, I expect the heading bug to move by one degree.
2. When I turn the same rotary encoder slowly so it clicks four times, I expect the heading bug to move by four degrees.
3. When I turn the same rotary encoder fast so it clicks many times, I expect the heading bug to move quickly (using the "incr/decr fast" actions, skipping several degrees per action).

This is what I'm experiencing instead:

1. When I turn the rotary encoder assigned to the heading bug so it clicks once, most of the time the heading bug will not move, rarely will it move by one degree. When I turn the encoder so it clicks twice, most of the time the heading bug will move by one degree, sometimes by two degrees.
2. When I turn the same rotary encoder slowly so it clicks four times, it will indeed move by three to four degrees if I find the correct turning speed. If I rotate the encoder too slow, it will move less than four degrees (sometimes not at all), if I turn too fast, it will start sending "fast" events and move more degrees than I intended.
3. When I turn the same rotary encoder fast so it clicks many times, it works mostly as expected, the heading bug is moving quickly, but it doesn't seem to move as smooth as it should.

I think this means I'm "losing" rotary encoder events, especially when turning the encoders slowly (let's define "slowly" as one to four "clicks per second"). The responsiveness of the rotary encoders is affected a lot by the Pollrate variable, I've tried a range of values from 10 (100ms) to 500 (2ms), I believe I'm seeing best results with 100 (10ms) or 200 (5ms), but pollrates that high seem like major overkill when the goal is to read 10 to 20 state changes per second.

I tried using com.read instead of com.readlast. The number of lost events seems to decrease, but com.reads needs high pollrates (100 or higher) or it introduces major delays (ie. I turn the encoder and it can take a second or longer until something happens in the sim). But even with com.read, single "clicks" are often lost. The "wedding cake" dual encoders seem to lose more events than the "simple" encoders, not sure what that means.

If I understand the docs for the com library, com.read reads fifo-style from a queue of events, while com.readlast reads the last value and discards the rest of the queue. And I assume that the queue is blowing up with all kinds of events from the controller, probably mainly axis events? If so, perhaps there a way to tell com.openhid or com.read to ignore controls I'm not interested in to keep the queue from filling up with their events?

Caveat: At this point, I cannot be sure that the issues aren't at least partially caused by the hardware -- I have only tried the controller with P3D/FSUIPC so far, I should probably see if I can get it to work with X-Plane to check whether it has similar issues there.

I would appreciate any wisdom you could share! Perhaps I should be using a different lua script? I searched the forum and read quite a few threads about issues with rotary encoders, but none seemed pertinent to my issue. Maybe someone can point me to the right thread?

Thanks,
Marc

[alaxus]

Simple answer - Be there - done that - did my head in - never got what I needed.

Encoders to joy clicks is simply not fast enough. What ever controller you have needs to be able to talk straight to the heading offset.

At the moment I am using phidgets but have some other solutions around the leo bodnar card and the arduino. These all talk direct to FSUIPC offsets via there own software.

For arduino you could look at this https://www.mobiflight.com/en/index.html

[Pete Dowson]

9 hours ago, aurel42 said:

when I turn a rotary encoder to the left, the assigned thingie often moves to the left -- they don't work reliably and smoothly.

Is the timing of the "button pressing" from the encoder a bit temperamental? Have you looked at what is happening in the log, with button logging enable?

With rotary encoders I usually find the you need to assign the same control to both "press" and "release", as eacg "click" is either a press or a release -- alternately.

If the encoders are not actually giving different button presses for each direction, they may be of the phased type, which are more difficult to deal with and need special conditionals in the assignments, done by editing the FSUIPC INI file. There are examples for this in the Advanced Guide.

9 hours ago, aurel42 said:

When I turn the rotary encoder assigned to the heading bug so it clicks once, most of the time the heading bug will not move, rarely will it move by one degree. When I turn the encoder so it clicks twice, most of the time the heading bug will move by one degree, sometimes by two degrees.

It certainly sounds like you only assigned to the "press" not also the "release".

9 hours ago, aurel42 said:

I think this means I'm "losing" rotary encoder events

Apart from alternate clicks, as above, you should note that assigning to contorls like INC and DEC results in a communication to SimConnect each and every time. Whether that loses some or not I don't know (it seems unlikely), but 'smoothness' is going to depend on the loading on that main core (usually core 0) at the time. That will vary. There's no way to do it synchronously.

An alternative approach is to use the FSUIPC Offset controls. You'd need to find the correct offset. You can assign to an increment or decrement, specifying the actual value of the increment/decrement and you can have it cyclic, so 0-359 and back, for example.

7 hours ago, alaxus said:

Encoders to joy clicks is simply not fast enough. What ever controller you have needs to be able to talk straight to the heading offset.

Exactly. So, why not?

Pete

 

[aurel42]

1 hour ago, Pete Dowson said:

With rotary encoders I usually find the you need to assign the same control to both "press" and "release", as eacg "click" is either a press or a release -- alternately.

This makes a major difference, the rotary encoders work more reliably this way, especially single "clicks", but they're still not "smooth".

Quote

Apart from alternate clicks, as above, you should note that assigning to contorls like INC and DEC results in a communication to SimConnect each and every time. Whether that loses some or not I don't know (it seems unlikely), but 'smoothness' is going to depend on the loading on that main core (usually core 0) at the time. That will vary. There's no way to do it synchronously.

An alternative approach is to use the FSUIPC Offset controls. You'd need to find the correct offset. You can assign to an increment or decrement, specifying the actual value of the increment/decrement and you can have it cyclic, so 0-359 and back, for example.

Every ipc.write is a distinct SimConnect call?

I'll look into this. So the idea is, at a lower poll rate, to com.read (possibly repeatedly until the queue is empty), cumulate the deltas, read the current values of the FS controls, and then write them back once after applying the cumulated changes, to reduce the number of interactions with SimConnect?

 

9 hours ago, alaxus said:

Encoders to joy clicks is simply not fast enough. What ever controller you have needs to be able to talk straight to the heading offset.

At the moment I am using phidgets but have some other solutions around the leo bodnar card and the arduino. These all talk direct to FSUIPC offsets via there own software.

My device uses a rather obscure board ("MJOY16"), not Arduino. The Arduino solution you pointed to doesn't seem to be Open Source (and even if it was, it would probably outside my expertise to adapt it). There's a "buttons to keyboard events" mapper program (called "SV Mapper") for the MJOY16 board (of course not Open Source, bummer). After mapping the rotary encoder events to key combinations, I can assign those keys to FS functions using FSUIPC just fine, and pretty reliable and almost smooth. This could be indicating that, when using hiddemo.lua, the number of SimConnect calls is the remaining issue for me, as Pete suggested.

Thanks for your help, I'm off to do some more experimentation!

*puts on rubber gloves and protective goggles*

Thanks,
Marc

Edited August 25, 2018 by aurel42
premature postaculation

[Pete Dowson]

59 minutes ago, aurel42 said:

I'll look into this. So the idea is, at a lower poll rate, to com.read (possibly repeatedly until the queue is empty), cumulate the deltas, read the current values of the FS controls, and then write them back once after applying the cumulated changes, to reduce the number of interactions with SimConnect?

I don't think that is the problem, because you go on to say:

59 minutes ago, aurel42 said:

I can assign those keys to FS functions using FSUIPC just fine, and pretty reliable and almost smooth. This could be indicating that, when using hiddemo.lua, the number of SimConnect calls is the remaining issue for me, as Pete suggested.

So smooth with direct assigns, just not with HidDemo?  Both are calling Simconnect, so it isn't the numbers of those which are different, it is the execution of the HidDemo. First, don't forget that it is all interpreted,. it isn't compiled into actual ccomputer native code. If you've not done so already, you need to try to streamline it to do only what you need it to do. The original example was very generalised.

Looking at HidDemo as supplied, do you see it using "com.readlast"? Or have you changed that? ReadLast will DISCARD any queue of inputs only retaining the last one. This is the normal requirement for switches. For fast button operations you want to use com.read instead, and keep looping in that the poll function till there's no more when you exit and wait for the next event.

I don't think there should be a need to accumulate them -- though maybe look at that if reading them all doesn't help. The problem with that is not seeing what you've changed it to whilst you are turning.

Pete

 

[aurel42]

10 hours ago, Pete Dowson said:

So smooth with direct assigns, just not with HidDemo?  Both are calling Simconnect, so it isn't the numbers of those which are different, it is the execution of the HidDemo.

I didn't realize that, thanks for the clarification.

10 hours ago, Pete Dowson said:

If you've not done so already, you need to try to streamline it to do only what you need it to do. The original example was very generalised.

I've played around with the LUA code. I stripped everything out except the handling of the buttons I'm interested in. The core loop looks sorta like this:

function Poll
  repeat
    com.read
    
    if queue was not empty then
      com.GetHidButtons

      if button state changed then
        ipc.writeUD
      end
    end
  until queue is empty

I added some logging and what I found is that com.read does not see all the rotary encoder events, while other applications do.

• I had the FSUIPC console log and the "SV Mapper" tool (that's the tool to map MJ16 events to keystrokes) open side-by-side.
• Whenever "SV Mapper" sees a button event, it highlights the button. Slowly turning one of the radials, I could see the matching button blink-blink-blink in the same rhythm as the radial encoder was clicking.
• I expected to see one or two lines of output in the FSUIPC window per click: one line per click if the radial encoder alternates between press and release; two lines per click, if the radial encoder presses and releases the button for each click. It turned out that the latter is the case, so I saw two lines whenever a click registered. [I guess that means, at least in theory, that I should not have to assign "release" actions for the rotary encoder events.]

I found that many of the clicks did not lead to a changed button state as seen by com.read. About half of the clicks are lost and never seem to end up in the queue that com.read reads from. And there's no rhythm to it, it's not every second click that gets lost. It's as if a coin was tossed for each click and if head comes up, the LUA script gets to see it, if tails comes up, the event is lost or invisible.

I also checked with X-Plane (to make sure that "SV Mapper" isn't doing anything magic specific to the MJOY16 board), and it also registers every single click without fail.

For reference, this is the code I used to test (so you can see I'm not the one throwing half of the events away, or maybe I am and you can point out my mistake)...

function Poll(time)
    -- counters for logging
	cntread = 0
	cntrad = 0

    -- the radial encoder events are limited to button[4], so I got rid of the loop over all buttons, but I kept the index for convenience
	i = 4

	repeat
		CurrentData, n = com.read(dev, rd)
    
		if n ~= 0 then
			buttons[1], buttons[2], buttons[3], buttons[4],
			buttons[5], buttons[6], buttons[7], buttons[8] = 
				com.GetHidButtons(dev, CurrentData)

			if buttons[i] ~= prevbuttons[i] then
				prevbuttons[i] = buttons[i]
				ipc.writeUD(0x3344 + ((i-1) * 4), buttons[i])
				cntrad = cntrad + 1
				ipc.log(string.format("last button state: %4X",
					buttons[4]))
			end
		end
		cntread = cntread + 1
	until n == 0
--	ipc.log(string.format("com.read called %d times, radial events sent %d, last button state: %4X",
--		cntread, cntrad, buttons[4]))
end

Unless there's something wrong with my code, something appears to be wrong with com.read...?

Confused as hell,
Marc

 

[aurel42]

Questioning my assumptions...

• com.openhid creates a queue which should receive a copy of all state changes of the device, correct?
• This queue is exclusive to the com.openhid handle, only the process holding the handle can read from this queue, correct?
• Is it weird that I feel like one of the seven dwarfs... "Who has been reading from my queue?"

[Pete Dowson]

9 hours ago, aurel42 said:

Unless there's something wrong with my code, something appears to be wrong with com.read...?

Unlikely -- com.read merely pulls data from a buffer is being continuously replenished as data arrives. This is code in a separate high priority thread. The same functions (though not from Lua) are used in my COM drivers for the older PFC equipment which used serial port access, and for their later HID devices. My 737 cockpit is almost totally dependent upon those COM routines.

You can use extra FSUIPC logging to see exactly what data arrives in the buffer. Just add this to the [General] section of the FSUIPC5.INI file:

Debug=Please
LogExtras=64

Also log changes to your Virtual Button set (offset appears to be

0x3344 + ((i-1) * 4

which, since i is fixed at 4, is actually 3350. Use the right-hand side of the Logging tab entering offset 3350, type U32 and 'hex' checked.

That will show what is actually received and what your plug-in 'catches'. We can then see if they match.

Pete

 

 

 

[aurel42]

Hi Pete,

I've attached the log.

What I did: I launched P3D and "SV Mapper", I confirmed the default scenario in the scenario dialog, I opened the FSUIPC dialog and set up logging, I started a new log file, then I started my "HidTRTL.lua" script and I turned one of the rotary encoders for about half a minute, about one click per second.

[For clarification: I would normally never run "SV Mapper" and the lua script simultaneously, because both are currently mapped in FSUIPC and that would obviously send duplicated controls to the sim. This is just for testing purposes.]

Here are the timestamps for the clicks, 
first column: when FSUIPC sees the keystrokes "SV Mapper" generated as response to the rotary encoder "clicking",
second column (only if present): HidTRTL.lua notices the "click",
additional colums: timestamps of "60 7D 20" showing up in the COMread dump, this appears to be the "signature" of the event during this test.

102000
103187 103453 103437
104109
105093
106125 106406 106390
107062
108156 108406 108375 108406
109203
110203
111156 111406 111375
112109 112375 112343*
113031
114062 114343 114312
114968 115218 115187
115937 116172 116156 116172
116843
117828
118797 119015 119015 
119750 120000 119968 120000
120734 121000 120984
121625 121875 121859*
122687 122968 122953
123656
124625
125656 125953 125922
126687 126968 126937 126953
127718 127953 127937
128547
129687 129968 129922
130718
131859
132875

[* Two times, the radial encoder sent a spurious "fast" event in addition to the correct event. This happens occasionally, I have no idea why. Probably a bug of the "MJOY16" homebrew board. It registers in the LUA script as two buttons being pressed simultaneously (0x2020 instead of 0x0020). If we can make the lua work, I could filter out these buggy events.]

So COMread sees 17 of the 32 events. Pretty much half of what it should see. As if someone else [Snow White?!] was stealing half of the events from the buffer before my lua script (or even com.read) could get to them.

Where do we go from here?

More confused than ever,
Marc

FSUIPC5.log

[Pete Dowson]

2 hours ago, aurel42 said:

So COMread sees 17 of the 32 events.

Is your other application sending the keystrokes, programmed in the sim? 

Please count the significant events.in the log.

There are 66 occasions in the log where the HEADING BUG INC control is sent.  There are 38 when "Button changed: bRef=0, Joy=68, Btn=5" occurs, half of them "pressed" and the other half "released". So, i make than 19 presses and 19 releases, all seen. How are you counting only 17 out of 32 when it is all of them?

As far as I can see from the log, the COMread function in FSUIPC is getting every input, and (miraculously?) your Lua is responding correctly to each one. How do you see it so differently?

Please, just count those events to see for yourself.

BTW, the device seems to only send its data in batches of 8. That splitting is not FSUIPC's doing! It uses a buffer which is 1024 bytes long. It is used cyclically, with a "head" and "tail", and it reads enough to fill to the previous "tail position" (or the end of the buffer)  -- if it gets to the end and there is more, it reads from the beginning to the tail. This works perfectly unless the device can supply more than 1024 bytes in the "poll" time -- the internal loop time, not yours -- which is 20 mSecs. That's never happened yet.

Of course, the 1024 byte buffer could "overspill" (i.e. lose data) if whatever was using it could not read the data in time. But it would need to be very slow to do that. And your log shows it is not losing anything and in fact giving good results.

So ... you say you are confused, but the log shows no reason for you to be puzzled, so your confusion rather puzzles me!

BTW in the rather artificial groups of 8 being sent by the device, those starting 03 are the button states. Those are supplied whether the buttons have been changed or not. The button you are testing is the 20 00 at the end, which is 00 00 if it isn't pressed.

Pete

 

[aurel42]

58 minutes ago, Pete Dowson said:

Is your other application sending the keystrokes, programmed in the sim? 

Yes, well, no, the keystrokes are mapped in FSUIPC. Quoting myself:

3 hours ago, aurel42 said:

[For clarification: I would normally never run "SV Mapper" and the lua script simultaneously, because both are currently mapped in FSUIPC and that would obviously send duplicated controls to the sim. This is just for testing purposes.]

 

Quote

Please count the significant events.in the log.

There are 66 occasions in the log where the HEADING BUG INC control is sent.  There are 38 when "Button changed: bRef=0, Joy=68, Btn=5" occurs, half of them "pressed" and the other half "released". So, i make than 19 presses and 19 releases, all seen. How are you counting only 17 out of 32 when it is all of them?

I was counting 17 out of 32 by looking closely at the changed button states and the keystrokes sent by "SV Mapper", and comparing that with the time when I actually manipulated the rotary encoder. I know that I turned the knob for half a minute at a pace of roughly one click per second. This resulted in 32 events in "SV Mapper" which were mapped to keystrokes. FSUIPC picked them up and turned them into HEADING BUG INC.

If you check the log, you will find that 32 of the 66 HEADING BUG INC controls follow a KEYDOWN event. That's the correct number of clicks (1 per sec for "about 30s").

Since I followed your advice to assign press and release events to the virtual buttons in FSUIPC for the lua script, we would now expect 64 more HEADING BUG INC controls. I forgot to change this for the test, but in a previous post I explained that the rotary encoders of my device do not need an event on release. Every click results in a button being pressed and released, they don't alternate states between pressed and released as other encoders do.

Because of this setup the heading bug should turn by three degrees for every click, one degree because the SV Mapper sends a keystroke, and one degree for the press and release events as seen by the LUA script.

So we should see 96 HEADING BUG INC. Instead we see 32 by "SV Mapper" und 34 by the lua script, and if we discard the dupes caused by the release, we end up with 17 of 32.

To see the difference between a lost event and one that got picked up, please look at the first two timestamps I provided, second 102 and second 103 in the log.

In second 102, only "SV Mapper" caused the heading bug to move by one degree.

In second 103, "SV Mapper" moves the heading bug by one degree, the lua script moves it by another two degrees.

 

Quote

BTW, the device seems to only send its data in batches of 8. That splitting is not FSUIPC's doing! 

I don't understand this. Is this a problem?

 

Quote

It uses a buffer which is 1024 bytes long. It is used cyclically, with a "head" and "tail", and it reads enough to fill to the previous "tail position" (or the end of the buffer)  -- if it gets to the end and there is more, it reads from the beginning to the tail. This works perfectly unless the device can supply more than 1024 bytes in the "poll" time -- the internal loop time, not yours -- which is 20 mSecs. That's never happened yet.

I don't believe in an overrun, either. There are no extraordinary amounts of data coming from the device, as you can see from the frame rates documented in the log, there were no performance issues during the test. Since my analysis points to a cointoss situation -- about half of the data is lost, but not every second piece of data -- my gut feeling is that somehow in my case two "entities" are polling from the same buffer and for every poll, it's a race between the two entities who gets the data.

That's why I asked those questions in that follow-up post, because I wanted to make sure that my lua script had exclusive access to its copy of the buffer, and didn't have to share it with something else.

 

Quote

So ... you say you are confused, but the log shows no reason for you to be puzzled, so your confusion rather puzzles me!

I hope I was successful sharing my confusion with you now.

 

Quote

BTW in the rather artificial groups of 8 being sent by the device, those starting 03 are the button states. Those are supplied whether the buttons have been changed or not. The button you are testing is the 20 00 at the end, which is 00 00 if it isn't pressed.

Yes, and in my previous post I provided time stamps for every time the button was seen as pressed by COMread.

Attached below are the matching lines from the log. There are 21 lines. Four times, the button was logged twice before the LUA script got to com.read it (two lines within the same fraction of a second). This leaves 17 times that the button has been seen as pressed down, while it should have been seen 32 times.

Thanks for sticking with me on this,
Marc

 

$ grep ": 03 .* 20" FSUIPC5.log # showing all COMread lines from log that start with "03" and contain "20"
   103437 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   106390 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   108375 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   108406 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   111375 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   112343 COMread:   8 bytes: 03 00 02 08 60 7D 20 20 
   114312 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   115187 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   116156 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   116172 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   119015 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   119968 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   120000 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   120984 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   121859 COMread:   8 bytes: 03 00 02 08 60 7D 20 20 
   122953 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   125922 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   126937 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   126953 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   127937 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 
   129922 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 

 

[Pete Dowson]

7 hours ago, aurel42 said:

I was counting 17 out of 32 by looking closely at the changed button states and the keystrokes sent by "SV Mapper", and comparing that with the time when I actually manipulated the rotary encoder. I know that I turned the knob for half a minute at a pace of roughly one click per second. This resulted in 32 events in "SV Mapper" which were mapped to keystrokes. FSUIPC picked them up and turned them into HEADING BUG INC.

If you check the log, you will find that 32 of the 66 HEADING BUG INC controls follow a KEYDOWN event. That's the correct number of clicks (1 per sec for "about 30s").

I count using a search and count feature in my text editor. I don't do it manually as the automated method is much more accurate and reliable. I counted all occurrences of Button changed: bRef=0, Joy=68, Btn=5 as well as the same followed by , Pressed and ,released 

And for the grand total, as I said, I counted HEADING BUG INC. You tell me why that might be wrong.

7 hours ago, aurel42 said:

Quote

BTW, the device seems to only send its data in batches of 8. That splitting is not FSUIPC's doing! 

I don't understand this. Is this a problem?

No, I was just explaining it in case it seemed odd to you. I certainly did to me. Most HID devices seem to supply the wrole of the data in one batch. What is the value of your variable "rd", the report size?

7 hours ago, aurel42 said:

my gut feeling is that somehow in my case two "entities" are polling from the same buffer and for every poll, it's a race between the two entities who gets the data.

Yes, that is probably true. You normally don't have two programs reading the same device. Why are you doing that? Do you have to? Can't you stop one? If they're competing then fairly obviously the one which is a compiled program running outside of the Sim has a better chsnce of "winning" than an interpreted one running inside the Sim.

7 hours ago, aurel42 said:

I wanted to make sure that my lua script had exclusive access to its copy of the buffer, and didn't have to share it with something else.

Er, I think there's a misunderstanding here. The "buffer" ComRead is using is its own. it is where the data RECEIVED from the device is put. The buffer in the device, if it has one at all, is its own affair. I expect it only retains it till someone reads it. Why woulsd it retain a copy in case something else read it? It has no idea how many programs are on the other end of the USB cable! How would it know when to dispense with it?

 

7 hours ago, aurel42 said:

Yes, and in my previous post I provided time stamps for every time the button was seen as pressed by COMread.

A rather pointless exercise and not helpful. Just counting the number of times the button changed was surely all that was needed.

7 hours ago, aurel42 said:

Attached below are the matching lines from the log. There are 21 lines. Four times, the button was logged twice before the LUA script got to com.read it (two lines within the same fraction of a second). This leaves 17 times that the button has been seen as pressed down, while it should have been seen 32 times.

Again, I count by search and count. Using 03 00 02 08 60 7D 20 00 I count 19 times, EXACTLY the same number as
Button changed: bRef=0, Joy=68, Btn=5, Pressed. For every change ComRead received in the button state there's a corresponding execution of the correct assignment! You are somehow getting mixed up. There's no way my code or your script is missing anything which has been read by COM.READ!

By the way, you are counting clicks as button down only events, but alternate clicks are button ups. At least every single rotary encoder I've ever met does that.

Pete

 

 

 

[aurel42]

I am not saying that com.read doesn't see everything COMread sees. I apologize if that's the impression I gave you.

I am saying that, after looking at the log, I am positive that COMread doesn't see about half of the button events generated by my rotary encoders (at least according to the log and the behaviour of the lua script), even though the Windows HID system picks them up reliably and other applications (e.g. X-Plane) can see all of them just fine. 

So if it's not a hardware issue (which would affect X-Plane as well) or a "driver" issue (because the device uses the default Windows thingie for HID devices, there is no special magic at work), and something happens between the Windows HID thingie and my LUA script which makes half of the events disappear, I think that's worth investigating.

You say that's unlikely because it works for you. All I can respond is that this problem might not be visible with manually operated buttons because they're pressed long enough. It might not be visible with those "phase-alternating" rotary encoders because they keep that alternating state long enough for COMread to pick it up. I don't know the internals of how Windows handles joysticks. Maybe the issue only happens when a LUA script is running and some other special FSUIPC setting or feature is in use, what do I know. All I can say is that I think I can prove that there is in fact a real issue if you indulge me and look at two pieces of the log file and then do your "search and count" after you've seen the context.

Let me try to make my point as digestible and concise as I can.

In the 102nd second covered in the log, I turned the rotary encoder and it clicked once.

// SV Mapper has registered the click and sends a key combination
   102000 KEYDOWN: VK=17, Waiting=0, Repeat=N, Shifts=2
   102000 .. Key not programmed -- passed on to FS
   102000 KEYDOWN: VK=16, Waiting=0, Repeat=N, Shifts=3
   102000 .. Key not programmed -- passed on to FS
   102015 KEYDOWN: VK=54, Waiting=0, Repeat=N, Shifts=3
// FSUIPC turns the heading bug by one degree
   102015 FS Control Sent: Ctrl=65879, Param=0 HEADING_BUG_INC
   102015 .. This key is programmed in FSUIPC4 'Keys' options
   102015 KEYUP: VK=54, Waiting=0
   102015 KEYUP: VK=16, Waiting=0
   102015 KEYUP: VK=17, Waiting=0
// If we remove all COMread lines that don't start with 03, for your convenience,
// all that is left are lines with this payload:
   102015 COMread:   8 bytes: 03 00 02 08 60 7D 00 00 
// There is not a single line that ends with "20 00",
// which means COMread does not see this button event. Why not?

At this point we've waited for a second and COMread did not see the button state change. This is not behaving as expected.

Since a second has passed, I turn the encoder again until it clicks once.

This time we get the result we would like to see every time the encoder clicks:

// just like in the previous second, "SV Mapper" sends the key combination after noticing the click
   103187 KEYDOWN: VK=17, Waiting=0, Repeat=N, Shifts=2
   103187 .. Key not programmed -- passed on to FS
   103203 KEYDOWN: VK=16, Waiting=0, Repeat=N, Shifts=3
   103203 .. Key not programmed -- passed on to FS
   103203 KEYDOWN: VK=54, Waiting=0, Repeat=N, Shifts=3
// FSUIPC turns the heading bug by one degree
   103203 FS Control Sent: Ctrl=65879, Param=0 HEADING_BUG_INC
   103203 .. This key is programmed in FSUIPC4 'Keys' options
   103203 KEYUP: VK=54, Waiting=0
   103203 KEYUP: VK=16, Waiting=0
   103203 KEYUP: VK=17, Waiting=0
// I removed all COMread lines not starting with "03"
   103265 COMread:   8 bytes: 03 00 02 08 60 7D 00 00 
   103297 COMread:   8 bytes: 03 00 02 08 60 7D 00 00 
   103312 COMread:   8 bytes: 03 00 02 08 60 7D 00 00 
   103406 COMread:   8 bytes: 03 00 02 08 60 7D 00 00 
   103437 COMread:   8 bytes: 03 00 02 08 60 7D 20 00 // here we are, "20" found, this is the button press
   103453 Button changed: bRef=0, Joy=68, Btn=5, Pressed
   103453 [Buttons] 182=P68,5,C65879,0
// FSUIPC turns the heading bug by another degree, because the virtual button has been "pressed"
   103468 FS Control Sent: Ctrl=65879, Param=0 HEADING_BUG_INC
   103468 LUA.0: last button state:   20
   103468 Button changed: bRef=0, Joy=68, Btn=5, Released
   103468 [Buttons] 209=U68,5,C65879,0
// FSUIPC turns the heading bug again, because the virtual button has been "released"
   103468 FS Control Sent: Ctrl=65879, Param=0 HEADING_BUG_INC
   103468 LUA.0: last button state:    0
// We increased the heading bug twice (once for pressed, once for released) which is a just mistake in my FSUIPC configuration.
// I realize this is a distraction, but I simply forgot to remove the "release" setting in FSUIPC after verifying that my
// encoders don't need it.
// All in all, the heading bug moved by three degrees for one click which has been seen by "SV Mapper" *and* the LUA script

So if COMread could see all the events and the radial encoder clicked 32 times, we should see HEADING_BUG_INC 96 times in the log file. We don't.

If you're following me so far, you're probably going to say that "SV Mapper" is the culprit.

But: I only ran "SV Mapper" so all the "clicks" of the rotary encoder would somehow manifest in the FSUIPC log.

If I don't run "SV Mapper", the behaviour of FSUIPC and my LUA script is exactly the same.

But then you would only have my word -- I would say "it should show 32 events because the rotary encoder clicked 32 times" and you would say "there were only 19 events".

If you use your search and count to look for KEYDOWN: VK=54 you will find exactly 32 matches. And only for 17 of those 32 events will you be able to find a matching COMread/LUA stanza. Again: if "SV Mapper" had not been running, COMread would still only have seen 17 of the 32 button presses. And this seems to be a bug. Somewhere. (Note how this is still consistent with my first post, before I even used "SV Mapper".)

[addition: Why 17/34 and not 19/38? Because the rotary encoder sent two spurious events ("20 20" instead of "20 00", probably because of a hardware flaw, this happens rarely (2 in 32 is an outlier)). They resulted in button 5 being pressed, button 13 being pressed, button 5 being pressed a second time (without being released!) and then all "three" buttons are released in the same order. See timestamps 112375 and 121875. These are the two "extra" events that I ignored because they don't seem relevant for the issue at hand, and the two "5 pressed" and two "5 released" events were caused by the same encoder click.]

To verify that "SV Mapper" is not the culprit and is a valid way to show the actual encoder clicks in the FSUIPC log, I ran "SV Mapper" together with X-Plane. Every single click of the rotary encoders resulted in the appropriate button being highlighted simultaneously in "SV Mapper" as well as X-Plane. X-Plane didn't lose a single click, let alone half of them.

If this is about trust and you doubt that I turned the rotary encoder 32 times and not just 17 or 19 times, I can try to make a video that shows me turning the knob, X-Plane and "SV Mapper" blinking for every click, and then I can reboot (to make sure nothing interferes) and launch P3D and demonstrate how COMread only sees about half of them, at least according to the log and the lua script...?

If you want me to shut up and go away, I can shut up and go away. it might just not be worth it if I'm the only person affected by this issue. But I would appreciate if you would at least acknowledge that, based on the log I provided and the argument I made, there is an issue -- at least with my hardware, on my system. [What if this is the reason why Alaxus and other users couldn't make their rotary encoders work either?]

Thanks,
Marc

 

[Pete Dowson]

I think we were talking at cross purposes. I was indeed thinking you were saying that the log proved that the lua function was not getting everything that FSUIPC was reading from the device.

There are two things I will tell you when I get back to my PC (I am walking the dog at present). But first let me say that your device's firmware is rather badly, even amateurly, written. It is the first HID device I've ever encountered like it, over 16 years since such devices were supported by FSUIPC, and other drivers I have written.

More soon, and possible solutions.

Pete

 

 

[Pete Dowson]

Right. Back at the PC. My wife has told me I shouldn't be working today (75th Birthday AND Bank Holiday), but i don't like things left outstanding.

Sorry about the long detailed misunderstanding. Now that we may be talking about the same things, I have had another look at the log. 

As I said, there are 19 occurrences of 03 00 02 08 60 7D 20 00, i.e. "button pressed". But counting all the ones where it isn't pressed (03 00 02 08 60 7D 00 00)  I get 575!

Now for a normal rotary encode (whether phased or the more expensive decoded type) there should be an equal number. After all with such an encoder alternate clicks set on and off, so it should be seen as pressed roughtly 50% of the time on average.

So, this does confirm your earlier hypothesis that the firmware in the device is acting on both changes (on and off) by sending a press followed by a release. It won't be the encoder itself, but how its output is being processed.

Thus there will be a timing problem. If my (internal) polling isn't frequent enough it will miss the presses. Looking at the log in that light it seems that the period that the button stays looking "pressed" averages around 90-110 mSecs. And that's the ones which are detected. So I assume the ones not detected are shorter.

The poll rate in FSUIPC is 50 per second (a sleep of 20 mS per loop). That should easily be enough for any device. But then we come to the second problme with your device's firmware: the number of polls needed per cycle! This appears to average 3-4 minimum between each 03 block of 8. By dividing the data into 8 byte chunks (01, 02, 03 and 04) it will on average poll and receive the values we want every 80 mSecs (12.5 per sec). Any press-release signalled which is shorter than that will be missed.

So, you see what I mean about poor firmware design. If it had simple signalled the on or off per click as I think it should there would be no problem. And if it supplied all the data each time it is asked for, again no problem. The combination of two poor design features is pretty bad, IMHO.

How to fix it?

Well, I can make the poll rate timing settable by parameter, which I will do. But when adjusting this, please bear in mind that it is in a high priority thread -- which means when it is active nothing else can run in that processor core. So it will need to you find the highest time which works consistently.. Try 10 first, then work from there, increase till it doesn't, decrease till it does.

5.151d to test, soon. ...

Pete

 

[aurel42]

2 hours ago, Pete Dowson said:

Right. Back at the PC. My wife has told me I shouldn't be working today (75th Birthday AND Bank Holiday), but i don't like things left outstanding.

OMG, now I feel so bad for being such a nuisance. Happy birthday, dear Pete. Please enjoy your day knowing we love you and are so grateful for everything you're doing for the FS community.

Thanks for the explanation of the quirks, or rather flaws of the MJOY16 controller. Now it all makes sense, and, yeah, I fully agree with your analysis that its behaviour is braindead.

I will test the dll tonight and get back to you by tomorrow!

THANK YOU!
Marc

[aurel42]

I know I said I would leave you alone for the day, but I think you will forgive me, because you were spot on.

With the test version of the dll you provided, I tested by decreasing ComReadLoopTime from 20ms in steps of 1ms.

I learned that only values divisible by 5 are worth trying:

at 19-20ms, about half the events were lost, 
at 18-16ms, about a third of the events were lost,
at 15ms, only about 10% (!) of the events were lost,
at 13-14ms, about a quarter of the events were lost,
at 11-12ms, about a third (!) of the events were lost,
and, finally, at 10ms, suddenly no events were lost at all.

I guess that means there's rhythm to the music, and somehow it helps to clap on the "multiple of 5ms" beat. 🙂

On my system, there is no noticeable performance impact whatsoever at the 10ms setting compared to the 20ms setting. I say "noticeable", because I'm not sure how to benchmark P3D reliably.

Thank you so much for turning my stupid hardware from half a brick into something that works as exactly as I imagined!
Marc

[Pete Dowson]

7 minutes ago, aurel42 said:

and, finally, at 10ms, suddenly no events were lost at all.

Okay. I was more imagining starting there. The odd order of lost events seems very strange. i don't understand that.

As I said earlier, with most of todays PCs, i can probably set 10 as default. that 20 has lasted very many years!

Pete

 

[aurel42]

1 minute ago, Pete Dowson said:

Okay. I was more imagining starting there.

Yeah, in your mail you suggested 10ms and I started with that value.

When I saw that 10ms solved the problem, I figured I would try to find the highest possible value by going from 20 to 10 (instead of the other way around), simply because I expected that with every step the result would become better and I like saving the best for last. To be honest, by the time I got to 15ms, I thought I would be able to stop testing before even getting to 10ms again, I did not expect that "anomaly" from 14ms down to 11ms and I have no clue at all what's happening there.

[scottb613]

Hi Folks,

Was a satisfactory solution ever found ?

I just bought a button box with six rotary encoders thinking it would be a snap to setup for things such as HDG Bug - CDI - Mixture - Prop - Altitude Select - and such... It seems I've made a mistake not realizing the complexity involved... It's easy to assign but far too fine in adjustment for any practical use... It would take all day to swing the HDG bug a full 360... I'm not a programmer - so I was wondering if there's any relatively easy solution available ? I guess you would need to monitor the rate of presses and increase the size of increment or decrement accordingly ?

Thanks...

Regards,

Scott

 

[Pete Dowson]

3 hours ago, scottb613 said:

I just bought a button box with six rotary encoders thinking it would be a snap to setup for things such as HDG Bug - CDI - Mixture - Prop - Altitude Select - and such... It seems I've made a mistake not realizing the complexity involved... It's easy to assign but far too fine in adjustment for any practical use... It would take all day to swing the HDG bug a full 360

Your idea "monitor the rate of presses and increase the size of increment or decrement accordingly" can work. An example of doing that sort of thing is provided in the Lua plug-in "tripleuse.lua" in your FSUIPC documents folder, though that's just for a single button of course.

GoFlight implement two speed operation in the rotaries in their RP48 units, which FSUIPC supports of course. I also have implemented similar actions in drivers I've made for PFC avionics stacks, and earlier for FlightLink and the GA28R cockpit unit built by Aerosoft Australia.

You should note that it is a lot more complicated than sending controls more rapidly,  That simply transfers the lag to the queue in Windows and the Sim, for those controls to be obeyed, and if this depends on the Gauge graphics update rate you can see why that could be slow. Instead you have to read and retain the current value in the code, and add to that, only sending the updated value back to the sim when there's a slight pause in the changes. The timing of that has to be adjusted to match the Sim's update speed. You wouldn't get constant continuous feedback on screen unless the frame rate is good, and even then the on-screen gauge can be jerky. It works best when you have a local hardware gauge being updated, the one in the sim not mattering until the adjustment is done or the pause is long enough.

Generally, though, for the more simple demands assigning one direction to INC and the other to DEC controls works fine, and it is just the same as for those using the usual two buttons unstead of rotaries. The fact that it's a rotary instead doesn't really make it different unless the number of clicks per roration is really pretty low. I suppose that depends on the make or quality of rotary encoder.

Pete

 

[scottb613]

Hi Pete,

As always - thank you for your time and valuable insights - - - I'll do some legwork on what you mentioned but it sounds a bit beyond my comfort zone...

Happy  Holidays - sir...

For the record - this is the button box I chose:

Regards,

Scott