[LV2] [RTID] proposal for a 'real time ID' extension

Hanspeter Portner dev at open-music-kontrollers.ch
Tue Jun 20 10:28:49 PDT 2017

On 10.09.2016 17:45, David Robillard wrote:
> On Sat, 2016-09-10 at 12:42 +0200, Hanspeter Portner wrote:
>> On 25.08.2016 03:49, David Robillard wrote:
>>> On Thu, 2016-08-25 at 00:28 +0100, Harry van Haaren wrote:
>>>> On Thu, Aug 25, 2016 at 12:12 AM, David Robillard <d at drobilla.net> wrote:
>>>>>> The extension is as simple as it can get: A single function which
>>>> requests the
>>>>>> next integer ID from the host whereby the function MUST be RT-safe,
>>>>>> non-blocking, lock-free and wait-free. New RTIDs may be requested in
>>>> any of the
>>>>>> many LV2 thread groups.
>>>>> I doubt the latter requirement is possible to achieve for anything but a
>>>>> trivial atomic integer increment.  Is that good enough for fancy use
>>>>> cases with many plugins operating in parallel on distinct event streams?
>>>> That doesn't really scale well - at least not if we're talking lots of
>>>> messages. Assuming the RTIDs are not monotonically increasing, a host can
>>>> "reserve" a bunch of IDs for a particular core, use them from the reserved
>>>> pool until they run out, and reserve again.
>>>> This implies that the RTID is *just* a tag, and *only* a tag. The result is
>>>> that recieving tags out of order, backwards, forwards, etc, is all valid
>>>> usage of RTIDs.
>>>> If we're all cool with that, then no problem :) -Harry
>> I'd be cool with that. I don't see any need for RTIDs to be monotonically
>> rising, as sequential/temporal ordering is inherently included in the atom event
>> system. I need them just to be non-repeating in a session of finite length.
>> An atomic increment just happens to be the simplest implementation. But well,
>> host authors are free to make this more efficient by tying pools to threads, etc.
>>> This is the sort of thing that gets incredibly complicated very quickly.
>>> This is part of my hand-wavey instinct to at least stick a flags
>>> parameter in there so we can do things like reserve the top n bits for
>>> something or other later if necessary.
>> Makes sense to me.
>> You're thinking about something along these lines?
>>   typedef enum {
>>     ...
>>   } LV2_RTID_Flags;
>>   typedef void* LV2_RTID_Get_Handle;
>>   typedef int64_t LV2_RTID;
>>   typedef struct _LV2_RTID_Get {
>>     LV2_RTID_Get_Handle handle;
>>     LV2_RTID (*next)(LV2_RTID_Get_Handle handle, uint32_t flags);
>>   } LV2_RTID_Get;
> Yeah.
> I realized the other day that the same thing applies if you want to talk
> in terms of blank nodes without having their IDs (conventionally stuff
> like b12345) clash...
>> Will adapt it ASAP. Instead of merging this into the patch extension, it may
>> better fit into the urid extension. The patch extension thus could remain
>> metadata only.
> The weight of extensions is really getting old.  I'm seriously
> considering turning all of official LV2 into a single directory...
> In any case, I suppose it kind of fits into URID.  Especially if you
> call it "blank ID" (even though Atom no longer actually has a facility
> for that since it was simplified away...)
>>> I'm not (yet) familiar with the sorts of use cases that motivated
>>> Hanspeter, but it knee-jerk-seems to me that if you have complicated
>>> graphs of many plugins processing request/response streams, one of two
>>> things is bound to become necessary:
>> That's exactly the scenario that let me come up with the RTIDs in the first
>> place, e.g. graphs like this
>>         / pluginB - pluginC \
>> pluginA                       pluginF - pluginG
>>         \ pluginD           /          /
>>          \                            /
>>           \ pluginE                  /
>> Each one of those plugins may consume and produce arbitrary amount of events.
>> E.g. pluginD may produce a dozen events per incoming one, pluginF may consume 4
>> and produce 1, ...
>> Trying to do this for continuous dimensional event data based on MIDI is a real
>> pain. I thus use my own atom based event system. But don't worry, I won't
>> propose that as an official extension, ever :)
>>> 1) Requirement for a separate "stream ID" property to allow multiplexing
>>> without any loss of information, and allowing a simple monotonically
>>> increasing atomic sequence number to suffice
>> What information is lost? From which source a given event is coming from? Why
>> would we need that information?
>> Taking the above plugin graph as example and sending event with RTID #12 from
>> pluginA, one may think that pluginG could receive multiple events with RTID #12
>> as #12 gets tripled and merged in between. This indeed would be confusing.
>> pluginA -(#12)- pluginB -(#12)- pluginC -(#12)- pluginF
>> pluginA -(#12)- pluginD -(#12)- pluginF
>> pluginA -(#12)- pluginE -(#12)- pluginF
>> pluginF -(#12,#12,#12)- pluginG
>> What I do instead is this:
>> pluginA -(#12)- pluginB -(#13)- pluginC -(#16)- pluginF
>> pluginA -(#12)- pluginD -(#14)- pluginF
>> pluginA -(#12)- pluginE -(#15)- pluginF
>> pluginF -(#14,#15,#16)- pluginG
>> Plugins handling my events always consume an RTID at their receiving end and
>> issue new RTIDs at their sending end. This robustly prevents having events with
>> the same RTID in multiple parallel event paths.
>>> 2) Host rewriting of sequence numbers
>> For plugin-plugin communication this won't be needed if done like above.
>>> If 2 is even theoretically possible, it's certainly very unpleasant.
>>> It's worth noting that as far as the directly plugin related transport
>>> mechanisms are concerned, preserving order is a given anyway, so the
>>> usual issues with this sort of thing over networks do not apply, but of
>>> course it's possible to stick a network hop in there (which is a key
>>> benefit of being able to serialize this stuff, and is how Ingen works
>>> remotely)
>> When having independent processes exchanging RTID tagged events, it may become
>> necessary for the host/ui to rewrite them.
>> Yes, I can see that to be a major drawback.
> Yeah, but then again, an extension for "give me a unique number" is
> pretty sensible on its own and really doesn't need to get so
> rabbit-holey, especially if it solves a real problem.  I imagine there's
> all kinds of problems if you start having fancy feedback request/reply
> stuff in plugins which nobody has played with much yet, but probably
> orthogonal ones to this.

I hereby bluntly take the opportunity to bump this issue.

Most recent patch can be found at [1].

Changed spec can be previewed at [2], [3]. I think to remember that David
approved of the name urid:Blank on IRC...

[1] https://github.com/ventosus/lv2/commit/4112b89a369f0db7bee42432837b1aa506c23204
[2] https://paste.open-music-kontrollers.ch/lv2plug.in/ns/ext/urid/urid.html#blank

More information about the Devel mailing list