[LV2] [RTID] proposal for a 'real time ID' extension

[Hanspeter Portner]

On 25.08.2016 03:49, David Robillard wrote:
> On Thu, 2016-08-25 at 00:28 +0100, Harry van Haaren wrote:
>> On Thu, Aug 25, 2016 at 12:12 AM, David Robillard <d at drobilla.net> wrote:
>>>> The extension is as simple as it can get: A single function which
>> requests the
>>>> next integer ID from the host whereby the function MUST be RT-safe,
>>>> non-blocking, lock-free and wait-free. New RTIDs may be requested in
>> any of the
>>>> many LV2 thread groups.
>>>
>>> I doubt the latter requirement is possible to achieve for anything but a
>>> trivial atomic integer increment.  Is that good enough for fancy use
>>> cases with many plugins operating in parallel on distinct event streams?
>>
>> That doesn't really scale well - at least not if we're talking lots of
>> messages. Assuming the RTIDs are not monotonically increasing, a host can
>> "reserve" a bunch of IDs for a particular core, use them from the reserved
>> pool until they run out, and reserve again.
>>
>> This implies that the RTID is *just* a tag, and *only* a tag. The result is
>> that recieving tags out of order, backwards, forwards, etc, is all valid
>> usage of RTIDs.
>>
>> If we're all cool with that, then no problem :) -Harry

I'd be cool with that. I don't see any need for RTIDs to be monotonically
rising, as sequential/temporal ordering is inherently included in the atom event
system. I need them just to be non-repeating in a session of finite length.

An atomic increment just happens to be the simplest implementation. But well,
host authors are free to make this more efficient by tying pools to threads, etc.

> This is the sort of thing that gets incredibly complicated very quickly.
> This is part of my hand-wavey instinct to at least stick a flags
> parameter in there so we can do things like reserve the top n bits for
> something or other later if necessary.

Makes sense to me.
You're thinking about something along these lines?

  typedef enum {
    ...
  } LV2_RTID_Flags;

  typedef void* LV2_RTID_Get_Handle;
  typedef int64_t LV2_RTID;

  typedef struct _LV2_RTID_Get {
    LV2_RTID_Get_Handle handle;
    LV2_RTID (*next)(LV2_RTID_Get_Handle handle, uint32_t flags);
  } LV2_RTID_Get;

Will adapt it ASAP. Instead of merging this into the patch extension, it may
better fit into the urid extension. The patch extension thus could remain
metadata only.

> I'm not (yet) familiar with the sorts of use cases that motivated
> Hanspeter, but it knee-jerk-seems to me that if you have complicated
> graphs of many plugins processing request/response streams, one of two
> things is bound to become necessary:

That's exactly the scenario that let me come up with the RTIDs in the first
place, e.g. graphs like this

        / pluginB - pluginC \
pluginA                       pluginF - pluginG
        \ pluginD           /          /
         \                            /
          \ pluginE                  /

Each one of those plugins may consume and produce arbitrary amount of events.
E.g. pluginD may produce a dozen events per incoming one, pluginF may consume 4
and produce 1, ...

Trying to do this for continuous dimensional event data based on MIDI is a real
pain. I thus use my own atom based event system. But don't worry, I won't
propose that as an official extension, ever :)

> 1) Requirement for a separate "stream ID" property to allow multiplexing
> without any loss of information, and allowing a simple monotonically
> increasing atomic sequence number to suffice

What information is lost? From which source a given event is coming from? Why
would we need that information?

Taking the above plugin graph as example and sending event with RTID #12 from
pluginA, one may think that pluginG could receive multiple events with RTID #12
as #12 gets tripled and merged in between. This indeed would be confusing.

pluginA -(#12)- pluginB -(#12)- pluginC -(#12)- pluginF
pluginA -(#12)- pluginD -(#12)- pluginF
pluginA -(#12)- pluginE -(#12)- pluginF
pluginF -(#12,#12,#12)- pluginG

What I do instead is this:

pluginA -(#12)- pluginB -(#13)- pluginC -(#16)- pluginF
pluginA -(#12)- pluginD -(#14)- pluginF
pluginA -(#12)- pluginE -(#15)- pluginF
pluginF -(#14,#15,#16)- pluginG

Plugins handling my events always consume an RTID at their receiving end and
issue new RTIDs at their sending end. This robustly prevents having events with
the same RTID in multiple parallel event paths.

> 2) Host rewriting of sequence numbers

For plugin-plugin communication this won't be needed if done like above.

> If 2 is even theoretically possible, it's certainly very unpleasant.
> 
> It's worth noting that as far as the directly plugin related transport
> mechanisms are concerned, preserving order is a given anyway, so the
> usual issues with this sort of thing over networks do not apply, but of
> course it's possible to stick a network hop in there (which is a key
> benefit of being able to serialize this stuff, and is how Ingen works
> remotely)

When having independent processes exchanging RTID tagged events, it may become
necessary for the host/ui to rewrite them.

Yes, I can see that to be a major drawback.