<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Have you looked at whether you are overflowing the zeocache and having to fetch your catalog from disk too often? The timing mentioned in this thread seems about right for that to be the case.<div><br><div>I have a client with about a 300k small documents, for whom I have separated the catalog from the rest of Data.fs. This means I can have different cache settings for small and large objects. Resulting in markedly faster catalog queries.</div><div><br></div><div>--r.</div><div><br></div><div><div><div><div>On 27 Oct 2008, at 12:32, Roché Compaan wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote:<br><blockquote type="cite">-----BEGIN PGP SIGNED MESSAGE-----<br></blockquote><blockquote type="cite">Hash: SHA1<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Oct 27, 2008, at 13:08 , Roché Compaan wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">- Plone uses too many indexes, and in particular, uses multiple text<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> indexes. Having extra indexes around "just in case" is a sure lose<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> a write time, and may even be expensive at query time (depending on<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> the query).<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">- Particular indexes have performance characteristics based on their<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> designed purpose: for instance, the stock FieldIndex <br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">implementation<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> assumes that the number of documents indexed will be >> the <br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">number of<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> discrete indexable values. Using such an index in an application<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> domain with a very large set of indexable values probably loses, <br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">and<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> in ways which don't show up in early / small-scale testing.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">- I'm pretty sure that we haven't yet found the best data structure <br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">for<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> "hierarchy indexes" (e.g., the Plone EPI index, or the stock Zope2<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> PathIndex, etc.). Something like a 'trie' might be optimal for<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> pure prefix searching of hierarchies.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">- I am confident that the TopicIndex is underutiliized: it does <br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">*all*<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> the work for a given query at write time, and can thus be <br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">blindingly<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> fast at query time.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">- Other special-purpose indexes (e.g., a "recent items" index) would<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> be worth a look, especially for applications with large volumes of<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> content.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I agree that one should look at improving performance without <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">caching as<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">well. But this is a lot harder and takes significantly more <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">development<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">and debugging time than introducing some form caching. So I'm not<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">convinced that it needs to happen in a certain order. If caching gives<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">you lots of performance with little effort now, then why shouldn't you<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">use it?<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">It's the typical trade-off. One course is expedient and fast for your <br></blockquote><blockquote type="cite">use case now. The other requires more resources, but benefits <br></blockquote><blockquote type="cite">everyone. Including those who don't want to depend on yet another <br></blockquote><blockquote type="cite">package, like memcached, for performance.<br></blockquote><br>I'm not tied to memcached. We started out using module level caches like<br>zope.cache.ram but that has obvious problems when using ZEO.<br><br><blockquote type="cite">When it comes to integrating anything in Zope itself I'd choose the <br></blockquote><blockquote type="cite">latter.<br></blockquote><br>Sure, we're not trying to get this into Zope, we're just sharing our<br>experience and exploring the territory so that one can produce a third<br>party package that really help people with the same use case (which I<br>suspect is quite common one).<br><br>-- <br>Roché Compaan<br>Upfront Systems <a href="http://www.upfrontsystems.co.za">http://www.upfrontsystems.co.za</a><br><br>_______________________________________________<br>Zope-Dev maillist - <a href="mailto:Zope-Dev@zope.org">Zope-Dev@zope.org</a><br><a href="http://mail.zope.org/mailman/listinfo/zope-dev">http://mail.zope.org/mailman/listinfo/zope-dev</a><br>** No cross posts or HTML encoding! **<br>(Related lists - <br> http://mail.zope.org/mailman/listinfo/zope-announce<br> http://mail.zope.org/mailman/listinfo/zope )<br></div></blockquote></div><br><div> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div id="sig" style="line-height: 13px; margin-top: 4px; margin-right: 0px; margin-bottom: 4px; margin-left: 0px; padding-top: 2px; padding-right: 4px; padding-bottom: 2px; padding-left: 4px; font-family: Verdana, Arial, Helvetica, sans-serif; font-weight: bold; font-size: 10px; color: black; "><div style="float: left; padding-top: 2px; padding-right: 6px; padding-bottom: 2px; padding-left: 6px; background-color: rgb(238, 238, 238); font-size: 10px; font-weight: bold; line-height: 13px; "><strong style="font-size: 10px; font-weight: bold; line-height: 13px; "><span class="Apple-style-span" style="font-size: 10px; font-weight: bold; line-height: 13px; ">Russ Ferriday</span></strong><span class="Apple-style-span" style="font-size: 10px; font-weight: bold; line-height: 13px; "><span class="Apple-converted-space"> </span>-<span class="Apple-converted-space"> </span></span><a href="http://topia.com/" title="visit topia.com" style="text-decoration: none; color: rgb(102, 102, 102); font-size: 10px; font-weight: bold; line-height: 13px; "><span class="Apple-style-span" style="color: rgb(102, 102, 102); font-size: 10px; font-weight: bold; line-height: 13px; ">Topia Systems</span></a><span class="Apple-style-span" style="font-size: 10px; font-weight: bold; line-height: 13px; "><span class="Apple-converted-space"> </span>- Open Source content management with Plone and Zope</span><br style="font-size: 10px; font-weight: bold; line-height: 13px; "><a href="mailto:russf@topia.com" style="text-decoration: none; color: rgb(102, 102, 102); font-size: 10px; font-weight: bold; line-height: 13px; "><span class="Apple-style-span" style="color: rgb(102, 102, 102); font-size: 10px; font-weight: bold; line-height: 13px; ">russf@topia.com</span></a><span class="Apple-style-span" style="font-size: 10px; font-weight: bold; line-height: 13px; "><span class="Apple-converted-space"> </span>- office: +44 2076 1777588 - mobile: +44 7789 338868 - skype: ferriday</span></div></div></div></span></div></span> </div><br></div></div></div></body></html>