Author Topic: Bug! NRPN implementation with data-ranges of 127 or less.  (Read 4272 times)

SMF

  • Team member
  • *
  • Posts: 43
    • View Profile
Bug! NRPN implementation with data-ranges of 127 or less.
« on: December 28, 2020, 08:19:49 PM »
Can someone please confirm, that these NRPNs (Algo, Velocity, NumberOfPolyphonicy, Glide) do actually work like they are supposed to with the current FW for the preenFM2? (Before I file a bug-report to Image Line and these NRPNs just don't work because the preen has a bug there and not FL Studio...)

I am following this list for setting up the CCs/NRPNs for the preenFM2 in FL Studio 20.8: https://ixox.fr/preenfm2/preenfm/midi/

All NRPNs but the lowest four seem to work... CC for Algo works, NRPN for Algo doesn't...

preenFM2-FW: 2.20

TIA,
Stefan
« Last Edit: December 29, 2020, 04:08:59 PM by SMF »

Xavier

  • Administrator
  • Hero Member
  • *
  • Posts: 2257
    • View Profile
Re: Possibe Bug? But maybe not in the PreenFM2... NRPN MSB=0 LSB={0,1,2,3}?
« Reply #1 on: December 28, 2020, 10:00:55 PM »

I'm not at home this week, and don't have a preenfm2.
I would be surprised if it doesn't work.

Have you tried with the preenfm2 editor ?
All the communication between the preenfm2 and the editor is done through NRPN.

If you modify these params on the editor, and if you see the new value on your preenfm2 display, it means the preenfm2 receives well the NRPN.
In the other direction it means it sends well the NRPN.

SMF

  • Team member
  • *
  • Posts: 43
    • View Profile
Re: Possibe Bug? But maybe not in the PreenFM2... NRPN MSB=0 LSB={0,1,2,3}?
« Reply #2 on: December 28, 2020, 11:25:19 PM »
Xavier,

I d' bet it would be like this... so, if the editor works and can change the parameters, the preen works like expected. Fine. In this case it's a bug in FL Studio, then...

I will (of course) verify this, but I must admit, that this is no-where unexpected. I have seen a lot(!) silly bugs in the midi-interface-plugin of FL Studio, so far...

will keep you informed,
Stefan

SMF

  • Team member
  • *
  • Posts: 43
    • View Profile
Re: Possibe Bug? But maybe not in the PreenFM2... NRPN MSB=0 LSB={0,1,2,3}?
« Reply #3 on: December 29, 2020, 03:50:02 PM »
I can confirm that this is a bug in FL Studio 20.8 (and most probably before...).


EDIT: OOOoopps... I am not so sure about this any more... I re-read my documentation on this topic and if I understand it correctly, then sending out the LSB is only required for data-ranges bigger than 127. Voice allocation per channel is 0...16 according to the above table and so, it should  be using Data Entry MSB, only. I may be wrong, but this now looks like an midi-implementation-bug in the PreenFM-Firmware...??


The editor send's this (the following PAN command is just shown for reference):

Code: [Select]
0000F7CA   2   4     B0    63    00    1  ---  CC: NRPN MSB     
0000F7CB   2   4     B0    62    02    1  ---  CC: NRPN LSB     
0000F7CC   2   4     B0    06    04    1  ---  CC: Data Entry MSB
0009F4EA   2   4     B0    0A    40    1  ---  CC: PAN           

Whiles the editor sends this:
Code: [Select]
000A8D48   2   4     B0    63    00    1  ---  CC: NRPN MSB     
000A8D49   2   4     B0    62    02    1  ---  CC: NRPN LSB     
000A8D4B   2   4     B0    06    00    1  ---  CC: Data Entry MSB
000A8D4B   2   4     B0    26    02    1  ---  CC: Data Entry LSB

So, FL Studio omits the "Data Entry LSB" and transmits the value as MSB °_O ...

Edit: I checked, that it sends out the LSB for data-ranges bigger than 0...127. So, this does not look like a bug in FL Studio, now.

Stefan
« Last Edit: December 29, 2020, 04:06:11 PM by SMF »

SMF

  • Team member
  • *
  • Posts: 43
    • View Profile
Re: Bug! NRPN implementation with data-ranges of 127 or less.
« Reply #4 on: December 29, 2020, 04:13:59 PM »
I am pretty sure about this now.

The Preen should not use/require "Data Entry LSB" for NRPN-data-ranges specified as being less or equal to 0...127.

As a work around I can declare the data range in FL Studio to be 128 instead of 16 (why 16? max nr of voices is 8 if I recall correctly?). But this makes the controller-knob behaving quite fiddly. But well, it's a working work around...

Stefan

Xavier

  • Administrator
  • Hero Member
  • *
  • Posts: 2257
    • View Profile
Re: Bug! NRPN implementation with data-ranges of 127 or less.
« Reply #5 on: December 29, 2020, 06:18:51 PM »
The Preen should not use/require "Data Entry LSB" for NRPN-data-ranges specified as being less or equal to 0...127.

I'm not sure the NRPN specification is so strong. In fact there's no real NRPN specification.

Yes CC#38 is known as optional.
But not for the preenfm,  it's required. And all the data values are sent in a much more consistant way., whatever the range is.

I have to double check when i'm at home, i'm almost sure the Novation keyboards also send the NRPN data using this consistant formula :
Value = CC#6 * 128 + CC#38

SMF

  • Team member
  • *
  • Posts: 43
    • View Profile
Re: Bug! NRPN implementation with data-ranges of 127 or less.
« Reply #6 on: December 29, 2020, 08:03:44 PM »
Well, Midi-Spec is quite clear about this (... well, if read twice or more times... I have a copy and it's NOT a nice read... this is the reason why I initially thought it was a FL Studio bug...).

NRPNs (as controllers with MSB/LSB in general) are defined as something like a 14bit fixed-point number with the LSB containing the fractional part. This is the reason for the statement that the LSB "optionally may be omitted". So MSB=39 and LSB=64 would be a controller value of 39.50 and not 5056. The LSB is optional, because it is just thought to be a fine-tuning-value.

The reason for this is also quite clear: Transmission speed/timing. It's three bytes less, which is a lot on a data-rate of only 31.250 bps...

I unfortunately neither can directly quote nor post the spec (thanks for that, MIDI Association,... *sigh*... but why? it's a stoneage-document... I don't get it...)...

Stefan

Xavier

  • Administrator
  • Hero Member
  • *
  • Posts: 2257
    • View Profile
Re: Bug! NRPN implementation with data-ranges of 127 or less.
« Reply #7 on: December 29, 2020, 08:49:08 PM »
"CC #38 is optional"... have at least 2 meanings.
Is it the device choice to decide if it needs it or not  and write it down in its midi implementation doc? or does the device should be able to accept without CC#38 in any case.

The later doesn't make any sense.
How does the device know if a last LSB CC#38 will arrive after a CC#6 ?
It has to wait for the following byte ? What if none arrive ? How long to wait before considering it will never arrive ? Latency will quick become unacceptable.
If the value is between 0 and 200. Is CC#38 mandatory ?
What if a device has a new version, and this new version accepts value between 0 and 150 although the previous one had a 100 upper limit. Changing the meaning of the Data MSB break all compatibility.

NRPNs only makes sense if that's the device (the preenfm) in its spec that decide how the DATA (LSB/MSB) must be sent and if CC#38 is used.

As you can guess, i disagree with the title of this thread   ;)

SMF

  • Team member
  • *
  • Posts: 43
    • View Profile
Re: Bug! NRPN implementation with data-ranges of 127 or less.
« Reply #8 on: December 30, 2020, 09:17:54 AM »
Quote
"CC #38 is optional"... have at least 2 meanings.

The term "optional" can have a lot meanings. But as the terms "should" and "should not" in RFCs do not have the "usual" ("common sense") meaning, it is with other standards as well. It is not about what a normal human being would think about the meaning of "optional" but what was defined in the technical papers and why.

Quote
Is it the device choice to decide if it needs it or not  and write it down in its midi implementation doc? or does the device should be able to accept without CC#38 in any case.

Is it "the devices choice" weather it needs CC#38 or ot?

Clearly: No.

MIDI-standard defines the term "optional" in the regard of control-messages like this: The MIDI-device shall still work without any flaws, but with reduced precision, even if the message for setting the LSB never arrives.

So, "optional" in this context means: It's upon anyone (including, but not exclusively to the playback-routine(!) producing the MIDI-data) to decide, that currently there is "just" not enough time to send it in full lenght and so it's LSB-value is left out to gain some time. All general controler values shall be defined accordingly.
 
Quote
The later doesn't make any sense.

Sorry, but it does. I just have given a meaningful senario for it. The reason for the irritation might be what a "controller" in terms of MIDI-spec really is: It's the value of a physical device like a rotary-encoder, a poti, a switch or a multi-switch. It's clearly not intendet as a general-purpose comms for the device.

Quote
How does the device know if a last LSB CC#38 will arrive after a CC#6 ?

It can't "know" and it should neither be dependend of nor wait for it. In terms of software development: Access to MSB and LSB are not atomic. They are thought to be absolutely independent from each other. And yes, this will produce artifacts. This is why it is not meant as a method of precise data-comm with the device.

Quote
It has to wait for the following byte ?

No. If the command for setting the MSB arrives, it sets the MSB, leaving the LSB untouched. If the command for setting the LSB arrives it sets the LSB, leaving the MSB untouched. It is not atomic. It can and it will contain garbage in between. It is not suited for precise data-communication with the device. That's what SYSEX is for...

The protocol in pseudo-code (for the sender side) including some errors or weirdo-things and how to deal wih them from the device-side is as follows:

Code: [Select]
set( NRPN_REG_MSB, 10 ); // set the MSB of the register-address. Block access to data-entry.
set( NRPN_REG_MSB,  0 ); // set the MSB of the register-address. Block access to data-entry. Overrides previous MSB-set
set( DATA_ENTRY_MSB, 10 ); // plainly ignored...
set( DATA_ENTRY_LSB,  0 ); // ignored as well...
set( NRPN_REG_LSB,  2 ); // set the LSB of the register-address. Release access to data-entry.
set( DATA_ENTRY_LSB, 10 ); // set the LSB to 10 whatever is in MSB... weird ... but OK...
sendOtherDataMaybeNoteOnOffs/(); // not a good idea but allowed
set( DATA_ENTRY_MSB,  0 ); // set MSB to 0, leaving LSB at 10
set( DATA_ENTRY_MSB, 10 ); // OK,... set the MSB again...
set( DATA_ENTRY_MSB,  9 ); // OK,... and again...
set( DATA_ENTRY_MSB,  8 ); // OK,... and again...
sendOtherDataMaybeNoteOnOffs/(); // Perfectly OK to do this
set( DATA_ENTRY_MSB,  7 ); // OK,... set the MSB again...
set( DATA_ENTRY_MSB,  6 ); // OK,... and again...
sendOtherDataMaybeNoteOnOffs/(); // Perfectly OK
set( DATA_ENTRY_LSB,  8 ); // weird ... but still OK...
set( DATA_ENTRY_LSB,  9 ); // even more weird ... but still OK...
set( DATA_ENTRY_LSB, 10 ); // even more, more weird ... but still OK...
...

Quote
What if none arrive ?

Then LSB is untouched. Whatever was in before...

Quote
How long to wait before considering it will never arrive ?

There is no passive or active waiting for setting Data-Entry-LSB. Realtime protocols simply do not permit this in different messages. I say it again: Access to MSB and LSB is completely indepentent from each other.

Quote
Latency will quick become unacceptable.

As there is no waiting...

Quote
If the value is between 0 and 200. Is CC#38 mandatory ?

No. According to MIDI-spec, there is not a single allowed situation, where CC#38 is mandatory.

Quote
What if a device has a new version, and this new version accepts value between 0 and 150 although the previous one had a 100 upper limit. Changing the meaning of the Data MSB break all compatibility.

I have not heard that a Montage is compatible to a DX-7 within the NRPN range of settings... Or that a Korg E2S is compatible to an Korg ES.

Quote
NRPNs only makes sense if that's the device (the preenfm) in its spec that decide how the DATA (LSB/MSB) must be sent and if CC#38 is used.

No... NRPNs are still "general controllers" in a MIDI-sense. Just like volume, panning and/or filter-cutoff. They may also carry the information "switch is in position Nr 0...127" (an no, 0...16129 is NOT allowed in this case, as you cannot prevent illegal intermediate switch-positions due to non-atomic access to LSB/MSB).

Quote
As you can guess, i disagree with the title of this thread

Xavier, it's your synth, so it's your decision. I have found a work-around which is not nice but works for me.

And I owned (and still own) "some" other midi-devices with grossly more serious glitches/flaws/bugs in their MIDI implementation. Most of the time this was done due to the precise and understandable wording in the MIDI spec (which in it's original 1.0 form excluded dealing with any corner-case...) leaving the manufacturers (too) much of room for own creativity. But also, sometimes breaking the standard was done by intention.

As you can guess, for me it is a bug in the PreenFM's firmware, still. I could tell you what I would do about it, but I can't tell what you will do about it -- or IF you will do something about it at all.

best,
Stefan

PS: Do I understand it correctly, that the PreenFM2 does actively wait for the LSB in this case? If so, what will happen if it never arrives? Maybe because the DAW died and was restarted?

PPS: If I had to fix it: I would use SYSEX for things which must be changed in an atomic way. I would keep using full-scale (MSB+LSB) NRPNs only for parameter-changes where small audible "glitches" are allowed. For that I would upscale the range (say you have a range of [-1; +22.3], maybe?) to the full 14 bits, so "leaving" off the LSB makes it less precise but it would still work...

Xavier

  • Administrator
  • Hero Member
  • *
  • Posts: 2257
    • View Profile
Re: Bug! NRPN implementation with data-ranges of 127 or less.
« Reply #9 on: December 30, 2020, 10:01:28 AM »
Thanks a lot for taking the time to write that down. :)
That's interesting.

I really think this is quite bad for the device. But Midi was created so long ago ;)
Or maybe it was not designed to work in real time (while playing sound).

For example the Pan parameter expect a value between 0 and 200, than it divides by 100 and substract 1, to have a value between [-1.00:1:00].
Or IM that accepts between 0 and 1600 and divides by 100 to get a [0.00:16.00] value.
Each time after receiving a CC#6, i would have to send a wrong parameter to the audio engine, and will produce some unexpected audio glitch.
Quote
No. According to MIDI-spec, there is not a single allowed situation, where CC#38 is mandatory.

In the current implementation that cannot work at all with the preenfm parameters.

You can also use the PC/mac editor to control any parameters. It knows how to talk to the preen hardware  ;)

Anyway, thanks again for this discussion.

« Last Edit: December 30, 2020, 10:26:51 AM by Xavier »

SMF

  • Team member
  • *
  • Posts: 43
    • View Profile
Re: Bug! NRPN implementation with data-ranges of 127 or less.
« Reply #10 on: December 30, 2020, 12:51:27 PM »
Quote from: Xavier
Thanks a lot for taking the time to write that down. :)
That's interesting.

I really think this is quite bad for the device. But Midi was created so long ago ;)
Or maybe it was not designed to work in real time (while playing sound).

It was... (designed to work in realtime)

The intention behind was just, that small (you could see this as some sort of lossy encoding/compression...) differences would not be too annoying. Esp. for something like filter-cutoff/-resonance, volume and panning... Could be a different thing on something like modulation-indices, though...

These values are lp-filtered on most midi-devices anyways to get more smooth transitions.

Quote
For example the Pan parameter expect a value between 0 and 200, than it divides by 100 and substract 1, to have a value between [-1.00:1:00].

Which quite easily could be translated into an NRPN following that lossy encoding: The range [-1.00:+1.00] would/could be translated like this:

Code: [Select]
int panMSB = (int)( (panValue+1.00)*16383.0 )/128;
int panLSB = (int)( (panValue+1.00)*16383.0 )&127;

.. of course this would also work with a range of [0:200] for the panning. Just other offsets 'n multipliers would be used. (Hey, what the heck am I telling you... Sorry :o ) If the LSB would be in a miss (for what reason ever) it would still (coarsely) work. If the LSB was delayed by some tens of milliseconds it wouldn't be noticeable that much -- if at all...

Quote
Or IM that accepts between 0 and 1600 and divides by 100 to get a [0.00:16.00] value.
Each time after receiving a CC#6, i would have to send a wrong parameter to the audio engine, and will produce some unexpected audio glitch.

What about defining the NRPN-Data-Entry for mod-indices like this (I am assuming a range of 0:1600 this time, which I guess -- have not looked into the code so far -- is the internal (int?) representation of the IM-values):

Code: [Select]
int modIndexMSB = (modIndex*10)/128;
int modIndexLSB = (modIndex*10)&127;

This would deliver a range of 0:125 in the MSB, resulting in a maximum error of 12.8 (for a range of [0:1600] or 0.128 in a range of [0.0:16.0]). Yes, this can (well, "could") be heard as a glitch. But it would be quite small, I think...

("Something"(TM) in my brain -- from having edited a lot of patches with the editor, lately ;D -- is telling me, that the spectral difference might in fact be so small for the above scenario, that I most probably wouldn't notice even if knowing the difference was there...)

Quote
In the current implementation that cannot work at all with the preenfm parameters.

You can also use the PC/mac editor to control any parameters. It knows how to talk to the preen hardware  ;)

Anyway, thanks again for this discussion.

I must admit, that I have not at all looked into the code on how the preenfm deals with incoming NRPNs but I guess it would simplify things a little bit if it were changed to a more compliant way of doing things. The audible glitches can be reduced (most probably to the level were they are completely masked when changed during playback) by changing the ranges of the NRPNs. And for the editor this wouldn't be much of a deal, too.

Essentially I do see two scenarios:
  • Changes of values when using the editor
  • Changes of values during playback

The changes I suggest would be:
  • For all values which can be expressed just within 7bit use the MSB and just ignore the LSB.

    Eg. for the voice-count-per-channel: It is most unlikely that we will ever encounter a preenfm being capable of doing 127(128) voice polyphony in a single channel. That would be a whopping 500+ voices polyphony for the device if only considering 4 channels. Same would apply to the velocity parameter.

  • For all parameters which cannot be expressed within 7bit maximise MSB significance and keep fingers crossed

    ...that delayed LSBs will not be audible too much. This is how all major manufacturers deal with it (some even stating that there rarely might be audible glitches in their manuals when changing values during playback)

Yes, this would somewhat break compatibility with the current implementation. But for the first usecase it is like you already said. The editor knows exactly of how to talk to the preenfm. A new version of the editor is required for nearly every new FW due to changes in the feature-set.

For the second case of changing the parameters during playback inside your favourite DAW, it's just a matter of changing the according midi-template to make things work again. And BTW: Is anyone else so insane like me to change these during playback?  :o I somewhat doubt it...  ;D

best,
Stefan

PS: See this as just what it is, please... a suggestion or personal opinion or so...
PPS: Please, be also forgiving with me on not looking into the code (not wanting to). I am coding for a professional life. I make music for private recreation. I dislike the idea of coding for music... (well,... sometimes,... I... am... not... that... steadfast...  ;D ;D)