[LV2] LV2 event ports running out of space

[David Robillard]

On Sun, 2012-09-02 at 04:37 -0400, Jeremy Salwen wrote:
> 
> 
> On Sat, Sep 1, 2012 at 4:58 PM, David Robillard <d at drobilla.net>
> wrote:
>         On Thu, 2012-08-30 at 18:07 -0400, Jeremy Salwen wrote:
>         > Thinking about it more, it seems to be something that needs
>         an
>         > extension.  A plugin should be able to stop processing
>         partway through
>         > a chunk and say "Hey, I ran out of space for events, here's
>         how many
>         > frames I finished".  Ideally it would just be the return
>         value of the
>         > run function (think of an fwrite like interface), but
>         obviously we
>         > can't make an incompatible change like that.  Probably just
>         a tagged
>         > port like the way latency reporting works.  It can be an
>         optional
>         > feature, and if a host doesn't support it, it just loses
>         events, but
>         > it still works.  What do you think?
>         
>         
>         Something like http://lv2plug.in/ns/ext/resize-port/ ? :P
>         
>         Implementing this dynamically can be very difficult, though.
>          I think
>         more widespread support for the static properties needs to
>         happen, but I
>         don't know if dynamic resize is a realistic expectation...
>         
>         -dr
>         
> 
> I don't think resizing port buffers is the answer.  If you're resizing
> port buffers, then A. you basically have to implement a realtime safe
> malloc in the host. B. You still don't know how big your memory pool
> has to be.

I see.  I was on my way out reading your message and wanted to reply
quicky, I didn't catch that you wanted to abort processing and indicate
only part of the cycle has finished.

Sensible in some ways, but avoiding that situation is definitely very
highly preferable.  It kind of breaks the entire model of run()

> And having the plugin statically specify the buffer size is useful but
> only solves some cases.  What if the plugin has no idea how big the
> buffer needs to be, as it depends entirely on the inputs?
> 
> I think the real solution is that the run method should be allowed to
> terminate early if it runs out of room in a buffer.
> 
> For example suppose you have a program which is running a chain
> 
> Memory buffer=>Plugin 1=>Plugin 2=> Jack callback
> 
> Let's suppose Jack wants 1024 frames of audio.  You call
> Plugin1.run(1024), but the buffer you are using for events between
> Plugin1 and Plugin2 isn't big enough for any of the events past 512
> frames.  Rather than try to increase the size of this buffer in
> realtime, you simply have Plugin1.run() stop after 512 frames and
> return 512.  Now, instead of calling Plugin2.run(1024), you call
> Plugin2.run(512).  Plugin2 doesn't run out of space, so it returns
> 512.  You get 512 frames of audio output, and write it to the buffer.
> Then, seeing as you still are missing the last 512 frames, you go back
> and call Plugin1.run(512), and (assuming it doesn't run out of space
> again) Plugin2.run(512).  
> 
> Essentially, you free up the buffer again by pushing its contents to
> the end of the chain.  In this way, you don't need to worry about
> buffer explosion. Each plugin is responsible for statically specifying
> a minimum buffer size so that it can pass a reasonable number of
> events through each run, and if an unreasonable number of events come
> along, the buffers stay the same size, the data is just processed in
> smaller chunk sizes.  The only buffers you do have to worry about the
> size of is the endpoints, if you're going to store the output events
> in a buffer or something, but that's inevitable.  Plus, one large
> buffer at the end is not nearly as bad as a whole series of large
> buffers at every step of the way.

Isn't this effectively equivalent to the host just running with shorter
block lengths, which is something hosts can easily do currently without
any extension?

> Resizing has it backwards:  It's like you have a certain amount of
> cheese and and chips at your house, and you ask your friend to make
> nachos.  He asks you "how much" and you say "use up all of the chips",
> even though you don't know the ratio of chips to cheese he is going to
> use.  When he asks you to go to the store to buy him more cheese,  you
> shouldn't comply.  Instead, you should tell him to stop when he runs
> out of cheese.  My solution is like saying "keep making nachos until
> you run out of chips or cheese, whichever comes first".

Perhaps, but I can't really think of any cases where plugins might
sporadically generate massive amounts of event output and not be able to
come up with a reasonable upper limit on that (which is pretty
questionable in RT anyway)

Are you having this problem present itself in some real situation you
are working on?

I see what you are saying, but it is a very messy world of things that I
think:

1) May be easy to simply avoid entirely

2) Should not be tackled without active implementation work on both
plugin and host side keeping things realistic

3) Is quite outside the usual model of LV2, LADSPA, JACK, and nearly
every callback driven audio system, which makes me wary

Which is basically just me saying, in an abstract sense, I don't like
this problem/solution :)  Seems to add more issues than it resolves...
better avoided

If there's a concrete immediate problem here, describing it would be
useful.  Others (on IRC) are currently running up against buffer size
limits so it'd be good to get a good feel for field of issues out there
so we can be sure the solution covers everything.

Cheers,

-dr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.lv2plug.in/pipermail/devel-lv2plug.in/attachments/20120902/ca5e95f4/attachment-0002.pgp>