[Zope3-dev] Re: bitio routines licensing (fwd)

[Paul Everitt]

Sorry for the lack of response, Matt.  I'm not ignoring it, rather, I'm 
trying to come up with a policy.  More to follow...

--Paul

Matt Hamilton wrote:
> Jim/Paul,
>   I have had a response to an email I sent to the author of the header
> files needed for the CompressedList stuff, attached below. I've just had a
> thought though, if he releases to files under the ZPL, does he then need
> to sign any Zope committer forms, or not as I will be doing the actual
> checkin?
> 
>   I have replaced the stub code that Jim and I wrote at the sprint with
> actual compression code and it seems to be running quite well.  Very quick
> and dirty tests show it being twice as fast as an IIBTree at inserting
> ordered random integers, and the resulting pickle is 3-4 times smaller.
> However, of course it can't accept integers out of order.
> 
> Jim: We discussed creating some kind of python class to wrap around the
> CList for use as a wordlist in the indexes such that it can handle wordids
> out of order by storing them in some auxiliary data structure then
> periodically merging them into the CList.  A possible problem with this
> approach is that any kind of iterator would have to check both the CList
> and the other data structure on each iteration.  Either that or do a set
> union of the two lists first (possible with the IISet code).  I can see
> this as potentially slowing the system down quite a bit.  I am thinking
> about some way of storing the out of order integers at the C level
> somehow.  Given the speed at which the compression occurs, I might do a
> test to see how slow it would be to re-encode the entire list if an out of
> order integer arrives.  It may be possible to limit the time this would
> take, by internally storing not just one compressed list, but several
> (say, break the list in half each time it reaches a certain size) -- that
> way only the list in which the integer is going to be stored would need to
> be re-encoded.
> 
> -Matt
> 
> PS.  All the ObjectHub/EventChannel stuff has now gone pretty much over my
> head -- hence the silence :)
> 
>