[Zope3-dev] Re: bitio routines licensing (fwd)

[Matt Hamilton]

Jim/Paul,
  I have had a response to an email I sent to the author of the header
files needed for the CompressedList stuff, attached below. I've just had a
thought though, if he releases to files under the ZPL, does he then need
to sign any Zope committer forms, or not as I will be doing the actual
checkin?

  I have replaced the stub code that Jim and I wrote at the sprint with
actual compression code and it seems to be running quite well.  Very quick
and dirty tests show it being twice as fast as an IIBTree at inserting
ordered random integers, and the resulting pickle is 3-4 times smaller.
However, of course it can't accept integers out of order.

Jim: We discussed creating some kind of python class to wrap around the
CList for use as a wordlist in the indexes such that it can handle wordids
out of order by storing them in some auxiliary data structure then
periodically merging them into the CList.  A possible problem with this
approach is that any kind of iterator would have to check both the CList
and the other data structure on each iteration.  Either that or do a set
union of the two lists first (possible with the IISet code).  I can see
this as potentially slowing the system down quite a bit.  I am thinking
about some way of storing the out of order integers at the C level
somehow.  Given the speed at which the compression occurs, I might do a
test to see how slow it would be to re-encode the entire list if an out of
order integer arrives.  It may be possible to limit the time this would
take, by internally storing not just one compressed list, but several
(say, break the list in half each time it reaches a certain size) -- that
way only the list in which the integer is going to be stored would need to
be re-encoded.

-Matt

PS.  All the ObjectHub/EventChannel stuff has now gone pretty much over my
head -- hence the silence :)

-- 
Matt Hamilton                                         matth@netsight.co.uk
Netsight Internet Solutions, Ltd.          Business Vision on the Internet
http://www.netsight.co.uk                               +44 (0)117 9090901
Web Hosting | Web Design  | Domain Names  |  Co-location  | DB Integration


---------- Forwarded message ----------
Date: Wed, 27 Feb 2002 13:26:25 +1100
From: Alistair Moffat <alistair@cs.mu.oz.au>
To: Matt Hamilton <matth@netsight.co.uk>
Subject: Re: bitio routines licensing

Matt Hamilton said:

>   I am working on an indexing implementation for Zope (www.zope.org), an

Hi, I have your mail. I am basicly sympathetic -- from my point of
view, ZPL seems pretty much the same as GPL, but will need to check
with some other interested parties before confirming a response.

And glad to hear that we have been helpful.

	Alistair Moffat, alistair@cs.mu.oz.au

(PS -- if Zope wants to write some compression software, check out
http://www.cs.mu.oz.au/~alistair/caca/. I don't make much money out
of book sales, but it is more than I make out of software sales :-)