[Checkins] SVN: Produts.RecentItemsIndex/ Import from cvs://cvs.zope.org/cvs-repository/Products/RecentItemsIndex trunk.
Tres Seaver
tseaver at palladion.com
Tue Mar 16 16:29:26 EDT 2010
Log message for revision 109997:
Import from cvs://cvs.zope.org/cvs-repository/Products/RecentItemsIndex trunk.
Changed:
A Produts.RecentItemsIndex/
A Produts.RecentItemsIndex/trunk/
A Produts.RecentItemsIndex/trunk/COPYRIGHT.txt
A Produts.RecentItemsIndex/trunk/LICENSE.txt
A Produts.RecentItemsIndex/trunk/README.txt
A Produts.RecentItemsIndex/trunk/__init__.py
A Produts.RecentItemsIndex/trunk/index.py
A Produts.RecentItemsIndex/trunk/test.py
A Produts.RecentItemsIndex/trunk/version.txt
A Produts.RecentItemsIndex/trunk/www/
A Produts.RecentItemsIndex/trunk/www/addIndex.dtml
A Produts.RecentItemsIndex/trunk/www/index.gif
A Produts.RecentItemsIndex/trunk/www/manageIndex.dtml
-=-
Added: Produts.RecentItemsIndex/trunk/COPYRIGHT.txt
===================================================================
--- Produts.RecentItemsIndex/trunk/COPYRIGHT.txt (rev 0)
+++ Produts.RecentItemsIndex/trunk/COPYRIGHT.txt 2010-03-16 20:29:26 UTC (rev 109997)
@@ -0,0 +1,9 @@
+Copyright (c) 2004 Zope Corporation and Contributors.
+All Rights Reserved.
+
+This software is subject to the provisions of the Zope Public License,
+Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+FOR A PARTICULAR PURPOSE.
Added: Produts.RecentItemsIndex/trunk/LICENSE.txt
===================================================================
--- Produts.RecentItemsIndex/trunk/LICENSE.txt (rev 0)
+++ Produts.RecentItemsIndex/trunk/LICENSE.txt 2010-03-16 20:29:26 UTC (rev 109997)
@@ -0,0 +1,54 @@
+Zope Public License (ZPL) Version 2.1
+-------------------------------------
+
+A copyright notice accompanies this license document that
+identifies the copyright holders.
+
+This license has been certified as open source. It has also
+been designated as GPL compatible by the Free Software
+Foundation (FSF).
+
+Redistribution and use in source and binary forms, with or
+without modification, are permitted provided that the
+following conditions are met:
+
+1. Redistributions in source code must retain the
+ accompanying copyright notice, this list of conditions,
+ and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the accompanying
+ copyright notice, this list of conditions, and the
+ following disclaimer in the documentation and/or other
+ materials provided with the distribution.
+
+3. Names of the copyright holders must not be used to
+ endorse or promote products derived from this software
+ without prior written permission from the copyright
+ holders.
+
+4. The right to distribute this software or to use it for
+ any purpose does not give you the right to use
+ Servicemarks (sm) or Trademarks (tm) of the copyright
+ holders. Use of them is covered by separate agreement
+ with the copyright holders.
+
+5. If any files are modified, you must cause the modified
+ files to carry prominent notices stating that you changed
+ the files and the date of any change.
+
+Disclaimer
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS''
+ AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT
+ NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
+ AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN
+ NO EVENT SHALL THE COPYRIGHT HOLDERS BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+ OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+ DAMAGE.
Added: Produts.RecentItemsIndex/trunk/README.txt
===================================================================
--- Produts.RecentItemsIndex/trunk/README.txt (rev 0)
+++ Produts.RecentItemsIndex/trunk/README.txt 2010-03-16 20:29:26 UTC (rev 109997)
@@ -0,0 +1,21 @@
+Recent Items Index
+
+ This index is designed to optimize queries which ask for the most
+ recent objects that match a certain value for an attribute. The designed
+ usage is a query for the most recent objects of a particular portal
+ type.
+
+ The index only retains up to a fixed number of items for each field
+ value which means that the performance of queries using the index
+ are independant of the size of the catalog.
+
+ The index also has a custom query interface so that applications
+ may query it directly for greatest efficiency since it handles both
+ the result selection and sorting simultaneously.
+
+ At the moment the index is not searchable through the standard ZCatalog
+ 'searchResults()' API. This is because the catalog does not yet support
+ indexes that can do searching and sorting simultaneously as this one
+ does.
+
+
Added: Produts.RecentItemsIndex/trunk/__init__.py
===================================================================
--- Produts.RecentItemsIndex/trunk/__init__.py (rev 0)
+++ Produts.RecentItemsIndex/trunk/__init__.py 2010-03-16 20:29:26 UTC (rev 109997)
@@ -0,0 +1,32 @@
+##############################################################################
+#
+# Copyright (c) 2002 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""RecentItemsIndex Zope product
+
+A ZCatalog plug-in-index for indexing recent items matching a specific field
+value.
+
+$Id: __init__.py,v 1.1.1.1 2004/07/19 17:46:21 caseman Exp $"""
+
+import index
+
+def initialize(context):
+
+ context.registerClass(
+ index.RecentItemsIndex,
+ meta_type='RecentItemsIndex',
+ permission='Add Pluggable Index',
+ constructors=(index.addIndexForm,),
+ icon='www/index.gif',
+ visibility=None
+ )
Added: Produts.RecentItemsIndex/trunk/index.py
===================================================================
--- Produts.RecentItemsIndex/trunk/index.py (rev 0)
+++ Produts.RecentItemsIndex/trunk/index.py 2010-03-16 20:29:26 UTC (rev 109997)
@@ -0,0 +1,299 @@
+##############################################################################
+#
+# Copyright (c) 2004 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""ZCatalog index for efficient queries of the most recent items
+
+$Id: index.py,v 1.5 2004/08/10 16:01:26 caseman Exp $"""
+
+import types
+from Globals import DTMLFile, InitializeClass
+from Acquisition import aq_inner, aq_parent
+from Persistence import Persistent
+from OFS.SimpleItem import SimpleItem
+from Products.PluginIndexes.common.PluggableIndex \
+ import PluggableIndexInterface
+from Products.ZCatalog.Lazy import LazyMap
+from AccessControl import Permissions
+from AccessControl.PermissionRole import rolesForPermissionOn
+from AccessControl.SecurityInfo import ClassSecurityInfo
+from BTrees.OOBTree import OOBTree
+from BTrees.IOBTree import IOBTree
+from BTrees.OIBTree import OIBucket
+from BTrees.Length import Length
+from zLOG import LOG, WARNING
+
+_marker = []
+
+def _getSourceValue(obj, attrname):
+ """Return the data to be indexed for obj"""
+ value = getattr(obj, attrname)
+ try:
+ # Try calling it
+ value = value()
+ except (TypeError, AttributeError):
+ pass
+ return value
+
+class RecentItemsIndex(SimpleItem):
+ """Recent items index"""
+
+ __implements__ = PluggableIndexInterface
+
+ meta_type = 'Recent Items Index'
+
+ manage_options = (
+ {'label': 'Overview', 'action': 'manage_main'},
+ )
+
+ manage_main = DTMLFile('www/manageIndex', globals())
+
+ security = ClassSecurityInfo()
+ security.declareObjectProtected(Permissions.manage_zcatalog_indexes)
+
+ def __init__(
+ self, id, field_name=None, date_name=None, max_length=None,
+ guard_roles=None, guard_permission=None, extra=None, caller=None):
+ """Recent items index constructor
+
+ id -- Zope id for index in
+
+ field_name -- Name of attribute used to classify the objects. A
+ recent item list is created for each value of this field indexed.
+ If this value is omitted, then a single recent item list for all
+ cataloged objects is created.
+
+ date_name -- Name of attribute containing a date which specifies the
+ object's age.
+
+ max_length -- Maximum length of each recent items list.
+
+ guard_roles -- A list of one or more roles that must be granted the
+ guard permission in order for an object to be indexed. Ignored if
+ no guard_permission value is given.
+
+ guard_permission -- The permission that must be granted to the
+ guard roles for an object in order for it to be indexed. Ignored if
+ no guard_roles value is given.
+
+ extra and caller are used by the wonderous ZCatalog addIndex
+ machinery. You can ignore them, unfortunately I can't 8^/
+ """
+ self.id = id
+ self.field_name = field_name or getattr(extra, 'field_name', None)
+ self.date_name = date_name or extra.date_name
+ self.max_length = max_length or extra.max_length
+ assert self.max_length > 0, 'Max item length value must be 1 or greater'
+ if guard_roles is None:
+ guard_roles = getattr(extra, 'guard_roles', None)
+ if guard_permission is None:
+ guard_permission = getattr(extra, 'guard_permission', None)
+ if guard_permission is not None and guard_roles:
+ self.guard_permission = guard_permission
+ self.guard_roles = tuple(guard_roles)
+ else:
+ self.guard_permission = self.guard_roles = None
+ self.clear()
+
+ ## Index specific methods ##
+
+ def getItemCounts(self):
+ """Return a dict of field value => item count"""
+ counts = {}
+ for value, items in self._value2items.items():
+ counts[value] = len(items)
+ return counts
+
+ def query(self, value=None, limit=None, merge=1):
+ """Return a lazy sequence of catalog brains like a catalog search
+ that coorespond to the most recent items for the value(s) given.
+ If value is omitted, then the most recent for all values are returned.
+ The result are returned in order, newest first. An integer value
+ can be specified in limit which restricts the maximum number of
+ results if desired. If no limit is specified, the indexes'
+ maximum length is used as the limit
+ """
+ catalog = aq_parent(aq_inner(self))
+ if value is None and self.field_name is not None:
+ # Query all values
+ value = list(self._value2items.keys())
+ elif value is not None and self.field_name is None:
+ # Ignore value given if there is no classifier field
+ value = None
+ if isinstance(value, (types.TupleType, types.ListType)):
+ # Query for multiple values
+ results = []
+ for fieldval in value:
+ try:
+ itempairs = self._value2items[fieldval].keys()
+ except KeyError:
+ pass
+ else:
+ results.extend(itempairs)
+ results.sort()
+ if merge:
+ results = [rid for date, rid in results]
+ else:
+ # Create triples expected by mergeResults()
+ results = [(date, rid, catalog.__getitem__)
+ for date, rid in results]
+ else:
+ # Query for single value
+ try:
+ items = self._value2items[value]
+ except KeyError:
+ results = []
+ else:
+ if merge:
+ results = items.values()
+ else:
+ # Create triples expected by mergeResults()
+ results = [(date, rid, catalog.__getitem__)
+ for date, rid in items.keys()]
+ results.reverse()
+ if limit is not None:
+ results = results[:limit]
+ if merge:
+ return LazyMap(catalog.__getitem__, results, len(results))
+ else:
+ return results
+
+ ## Pluggable Index API ##
+
+ def index_object(self, docid, obj, theshold=None):
+ """Add document to index"""
+ if self.guard_permission is not None and self.guard_roles:
+ allowed_roles = rolesForPermissionOn(self.guard_permission, obj)
+ for role in allowed_roles:
+ if role in self.guard_roles:
+ break
+ else:
+ # Object does not have proper permission grant
+ # to be in the index
+ self.unindex_object(docid)
+ return 0
+ try:
+ if self.field_name is not None:
+ fieldvalue = _getSourceValue(obj, self.field_name)
+ else:
+ fieldvalue = None
+ datevalue = _getSourceValue(obj, self.date_name)
+ except AttributeError:
+ # One or the other source attributes is missing
+ # unindex the object and bail
+ self.unindex_object(docid)
+ return 0
+ datevalue = datevalue.timeTime()
+ entry = self.getEntryForObject(docid)
+ if (entry is None or fieldvalue != entry['value']
+ or datevalue != entry['date']):
+ # XXX Note that setting the date older than a previously pruned
+ # object will result in an incorrect index state. This may
+ # present a problem if dates are changed arbitrarily
+ if entry is None:
+ self.numObjects.change(1)
+ else:
+ # unindex existing entry
+ self.unindex_object(docid)
+ self._rid2value[docid] = fieldvalue
+ try:
+ items = self._value2items[fieldvalue]
+ except KeyError:
+ # Unseen value, create a new items bucket
+ items = self._value2items[fieldvalue] = OIBucket()
+ items[datevalue, docid] = docid
+ while len(items) > self.max_length:
+ # Prune the oldest items
+ olddate, oldrid = items.minKey()
+ # Unindex by hand to avoid theoretical infinite loops
+ self.numObjects.change(-1)
+ del items[olddate, oldrid]
+ if not items:
+ # Not likely, unless max_length is 1
+ del self._value2items[fieldvalue]
+ try:
+ del self._rid2value[oldrid]
+ except KeyError:
+ LOG('RecentItemsIndex', WARNING,
+ 'Could not unindex field value for %s.' % oldrid)
+ return 1
+ else:
+ # Index is up to date, nothing to do
+ return 0
+
+ def unindex_object(self, docid):
+ """Remove docid from the index. If docid is not in the index,
+ do nothing"""
+ try:
+ fieldvalue = self._rid2value[docid]
+ except KeyError:
+ return 0 # docid not in index
+ self.numObjects.change(-1)
+ del self._rid2value[docid]
+ items = self._value2items[fieldvalue]
+ for date, rid in items.keys():
+ if rid == docid:
+ del items[date, rid]
+ if not items:
+ del self._value2items[fieldvalue]
+ return 1
+ return 1
+
+ def _apply_index(self, request, cid=''):
+ """We do not play in normal catalog queries"""
+ return None
+
+ def getEntryForObject(self, docid, default=None):
+ """Return a dict containing the field value and date for
+ docid, or default if it is not in the index. The returned dict
+ has the keys 'value' and 'date'
+ """
+ try:
+ fieldvalue = self._rid2value[docid]
+ except KeyError:
+ return default
+ for date, rid in self._value2items[fieldvalue].keys():
+ if rid == docid:
+ return {'value':fieldvalue, 'date':date}
+ # If we get here then _rid2values is inconsistent with _value2items
+ LOG('RecentItemsIndex', WARNING,
+ 'Field value found for item %s, but no date. '
+ 'This should not happen.' % docid)
+ return default
+
+ def hasUniqueValuesFor(self, name):
+ """Return true if the index holds the unique values for name"""
+ return name == self.field_name
+
+ def uniqueValues(self, name=None):
+ """Return the unique field values indexed"""
+ if name is None:
+ name = self.field_name
+ if name == self.field_name:
+ return self._value2items.keys()
+ else:
+ return []
+
+ ## ZCatalog ZMI methods ##
+
+ def clear(self):
+ """reinitialize the index"""
+ self.numObjects = Length()
+ # Mapping field value => top items
+ self._value2items = OOBTree()
+ # Mapping indexed rid => field value for unindexing
+ self._rid2value = IOBTree()
+
+
+InitializeClass(RecentItemsIndex)
+
+addIndexForm = DTMLFile('www/addIndex', globals())
Added: Produts.RecentItemsIndex/trunk/test.py
===================================================================
--- Produts.RecentItemsIndex/trunk/test.py (rev 0)
+++ Produts.RecentItemsIndex/trunk/test.py 2010-03-16 20:29:26 UTC (rev 109997)
@@ -0,0 +1,433 @@
+##############################################################################
+#
+# Copyright (c) 2004 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Tests for RecentItemsIndex
+
+$Id: test.py,v 1.5 2004/08/10 16:01:26 caseman Exp $"""
+
+import os
+from unittest import TestCase, TestSuite, main, makeSuite
+
+import ZODB
+from OFS.SimpleItem import SimpleItem
+from DateTime import DateTime
+
+
+class Doc:
+
+ def __init__(self, **kw):
+ self.__dict__.update(kw)
+
+class DummyCatalog(SimpleItem):
+
+ docs = {}
+
+ def __getitem__(self, item):
+ return self.docs[item]
+
+class Viewable(SimpleItem):
+
+ date = DateTime('2/21/2004')
+ type = 'Viewable'
+
+ def __init__(self, role=None):
+ self._addRole(role)
+ if role is not None:
+ self.manage_permission('View', [role])
+
+class RecentItemsIndexTest(TestCase):
+
+ def setUp(self):
+ from Products.RecentItemsIndex.index import RecentItemsIndex
+ self.test = DummyCatalog()
+ self.test.index = RecentItemsIndex('test', 'type', 'date', 10)
+ self.index = self.test.index
+
+ def test_construct_with_extra(self):
+ # Simulate instantiating from ZCatalog
+ from Products.RecentItemsIndex.index import RecentItemsIndex
+ class extra:
+ field_name = 'bruford'
+ date_name = 'wakeman'
+ max_length = 25
+ guard_roles = ['Anonymous']
+ guard_permission = 'View'
+ index = RecentItemsIndex('extra', extra=extra)
+ self.assertEqual(index.getId(), 'extra')
+ self.assertEqual(index.field_name, 'bruford')
+ self.assertEqual(index.date_name, 'wakeman')
+ self.assertEqual(index.max_length, 25)
+ self.assertEqual(tuple(index.guard_roles), ('Anonymous',))
+ self.assertEqual(index.guard_permission, 'View')
+
+ def test_construct_with_no_classifier_or_guard(self):
+ # Simulate instantiating from ZCatalog
+ from Products.RecentItemsIndex.index import RecentItemsIndex
+ class extra:
+ date_name = 'modified'
+ max_length = 30
+ index = RecentItemsIndex('nuttin', extra=extra)
+ self.assertEqual(index.getId(), 'nuttin')
+ self.assertEqual(index.date_name, 'modified')
+ self.assertEqual(index.max_length, 30)
+
+ def test_construct_with_bogus_max_length(self):
+ from Products.RecentItemsIndex.index import RecentItemsIndex
+ self.assertRaises(
+ Exception, RecentItemsIndex, 'test', 'type', 'date', 0)
+ self.assertRaises(
+ Exception, RecentItemsIndex, 'test', 'type', 'date', -20)
+
+ def test_index_single(self):
+ doc = Doc(type='fluke', date=DateTime('1/1/2004'))
+ self.failUnless(self.index.index_object(1, doc))
+ self.assertEqual(self.index.numObjects(), 1)
+ self.assertEqual(self.index.getItemCounts(), {'fluke': 1})
+
+ def test_exclude_obj_without_field_and_date(self):
+ doc = Doc()
+ self.failIf(self.index.index_object(1, doc))
+ self.assertEqual(self.index.numObjects(), 0)
+ self.assertEqual(self.index.getItemCounts(), {})
+ doc = Doc(type='cheetos')
+ self.failIf(self.index.index_object(1, doc))
+ self.assertEqual(self.index.numObjects(), 0)
+ self.assertEqual(self.index.getItemCounts(), {})
+ doc = Doc(date=DateTime('4/17/2004'))
+ self.failIf(self.index.index_object(1, doc))
+ self.assertEqual(self.index.numObjects(), 0)
+ self.assertEqual(self.index.getItemCounts(), {})
+
+ def test_unindex_single(self):
+ self.test_index_single()
+ self.failUnless(self.index.unindex_object(1))
+ self.assertEqual(self.index.numObjects(), 0)
+ self.assertEqual(self.index.getItemCounts(), {})
+
+ def test_index_many(self):
+ types = ['huey', 'dooey', 'looey', 'dooey'] * 15
+ date = DateTime('1/1/2004')
+ docs = {}
+ for docid, typ in zip(range(len(types)), types):
+ if not docid % 3:
+ date = date + 1
+ if not docid % 7:
+ date = date - (docid % 3)
+ doc = docs[docid] = Doc(docid=docid, type=typ, date=date)
+ self.index.index_object(docid, doc)
+ maxlen = self.index.max_length
+ self.assertEqual(self.index.getItemCounts(),
+ {'huey': maxlen, 'dooey':maxlen, 'looey':maxlen})
+ self.assertEqual(self.index.numObjects(), maxlen*3)
+ self.test.docs = docs
+ return docs
+
+ def test_index_many_no_classifier(self):
+ from Products.RecentItemsIndex.index import RecentItemsIndex
+ self.test.index = RecentItemsIndex('test', None, 'date', 10)
+ self.index = self.test.index
+ types = ['huey', 'dooey', 'looey', 'dooey'] * 15
+ date = DateTime('1/1/2004')
+ docs = {}
+ for docid, typ in zip(range(len(types)), types):
+ if not docid % 3:
+ date = date + 1
+ if not docid % 7:
+ date = date - (docid % 3)
+ doc = docs[docid] = Doc(docid=docid, type=typ, date=date)
+ self.index.index_object(docid, doc)
+ maxlen = self.index.max_length
+ self.assertEqual(self.index.getItemCounts(), {None: maxlen,})
+ self.assertEqual(self.index.numObjects(), maxlen)
+ self.test.docs = docs
+ return docs
+
+ def test_unindex_one_type(self):
+ docs = self.test_index_many()
+ for docid, doc in docs.items():
+ if doc.type == 'looey':
+ self.index.unindex_object(docid)
+ self.assertEqual(self.index.numObjects(), 20)
+ self.assertEqual(self.index.getItemCounts(), {'huey': 10, 'dooey':10})
+
+ def test_unindex_all(self):
+ docs = self.test_index_many()
+ for docid in docs.keys():
+ self.index.unindex_object(docid)
+ self.assertEqual(self.index.numObjects(), 0)
+ self.assertEqual(self.index.getItemCounts(), {})
+ self.assertEqual(list(self.index.uniqueValues()), [])
+
+ def _get_top_docs(self, docs):
+ top = {'huey':[], 'dooey':[], 'looey':[]}
+ for doc in docs.values():
+ top[doc.type].append((doc.date.timeTime(), doc.docid))
+ for typ, docs in top.items():
+ docs.sort()
+ top[typ] = docs[-10:]
+ return top
+
+ def test_getEntryForObject(self):
+ docs = self.test_index_many()
+ top = self._get_top_docs(docs)
+ for docid, doc in docs.items():
+ entry = self.index.getEntryForObject(docid)
+ if entry is not None:
+ self.assertEqual(entry,
+ {'value': doc.type, 'date': doc.date.timeTime()})
+ else:
+ self.failIf((doc.date.timeTime(), doc.docid) in top[doc.type])
+
+ def test_unindex_most_recent(self):
+ docs = self.test_index_many()
+ top = self._get_top_docs(docs)
+ item_counts = self.index.getItemCounts()
+ total_count = 30
+ for i in range(10):
+ for typ in ('huey', 'dooey', 'looey'):
+ nil, byebyeid = top[typ].pop()
+ self.failUnless(self.index.unindex_object(byebyeid))
+ item_counts[typ] -= 1
+ if not item_counts[typ]:
+ del item_counts[typ]
+ total_count -= 1
+ self.assertEqual(self.index.getItemCounts(), item_counts)
+ self.assertEqual(self.index.numObjects(), total_count)
+ self.assertEqual(self.index.numObjects(), 0)
+ self.assertEqual(self.index.getItemCounts(), {})
+
+ def test_unindex_bogus_rid(self):
+ self.test_index_many()
+ self.failIf(self.index.unindex_object(-2000))
+
+ def _get_indexed_doc(self, fromtop=0):
+ docs = self.test_index_many()
+ top = self._get_top_docs(docs)
+ items = docs.items()
+ if fromtop:
+ items.reverse()
+ for docid, doc in items:
+ entry = self.index.getEntryForObject(docid)
+ if entry is not None:
+ break
+ else:
+ self.fail('No objects in index')
+ self.assertEqual(entry, {'value':doc.type, 'date':doc.date.timeTime()})
+ return doc
+
+ def test_reindex_no_change(self):
+ # reindex with no change should be a no-op
+ doc = self._get_indexed_doc()
+ self.failIf(self.index.index_object(doc.docid, doc))
+ self.assertEqual(self.index.getEntryForObject(doc.docid),
+ {'value':doc.type, 'date':doc.date.timeTime()})
+
+ def test_reindex_change_date(self):
+ doc = self._get_indexed_doc()
+ doc.date = doc.date + 10
+ self.failUnless(self.index.index_object(doc.docid, doc))
+ self.assertEqual(self.index.getEntryForObject(doc.docid),
+ {'value':doc.type, 'date':doc.date.timeTime()})
+
+ def test_reindex_change_value(self):
+ doc = self._get_indexed_doc(fromtop=1)
+ oldtype = doc.type
+ for typ in self.index.uniqueValues():
+ if typ != oldtype:
+ doc.type = typ
+ break
+ self.failUnless(self.index.index_object(doc.docid, doc))
+ self.assertEqual(self.index.getEntryForObject(doc.docid),
+ {'value':doc.type, 'date':doc.date.timeTime()})
+
+ def test_reindex_change_date_and_value(self):
+ doc = self._get_indexed_doc(fromtop=1)
+ doc.date = doc.date + 4
+ oldtype = doc.type
+ for typ in self.index.uniqueValues():
+ if typ != oldtype:
+ doc.type = typ
+ break
+ self.failUnless(self.index.index_object(doc.docid, doc))
+ self.assertEqual(self.index.getEntryForObject(doc.docid),
+ {'value':doc.type, 'date':doc.date.timeTime()})
+
+ def test_query_empty_index(self):
+ result = self.index.query('foobar')
+ self.failIf(result)
+
+ def test_simple_query(self):
+ docs = self.test_index_many()
+ top = self._get_top_docs(docs)
+ result = self.index.query('huey')
+ expected = [docid for nil, docid in top['huey']]
+ expected.reverse()
+ self.assertEqual([doc.docid for doc in result], expected)
+
+ def test_query_bogus_value(self):
+ docs = self.test_index_many()
+ self.failIf(self.index.query('snacks'))
+
+ def test_query_limit(self):
+ docs = self.test_index_many()
+ top = self._get_top_docs(docs)
+ result = self.index.query('huey', limit=3)
+ expected = [docid for nil, docid in top['huey']]
+ expected.reverse()
+ expected = expected[:3]
+ self.assertEqual([doc.docid for doc in result], expected)
+
+ def test_query_no_merge(self):
+ docs = self.test_index_many()
+ top = self._get_top_docs(docs)
+ result = self.index.query('dooey', merge=0)
+ expected = [(date, docid, self.test.__getitem__)
+ for date, docid in top['dooey']]
+ expected.reverse()
+ for rrow, erow in zip(result, expected):
+ self.assertEqual(rrow[:2], erow[:2])
+
+ def test_query_multiple_values(self):
+ docs = self.test_index_many()
+ top = self._get_top_docs(docs)
+ result = self.index.query(['huey', 'dooey'])
+ expected = top['huey'] + top['dooey']
+ expected.sort()
+ expected = [docid for nil, docid in expected]
+ expected.reverse()
+ self.assertEqual([doc.docid for doc in result], expected)
+ return expected
+
+ def test_query_all_values(self):
+ docs = self.test_index_many()
+ top = self._get_top_docs(docs)
+ result = self.index.query()
+ expected = top['huey'] + top['dooey'] + top['looey']
+ expected.sort()
+ expected = [docid for nil, docid in expected]
+ expected.reverse()
+ self.assertEqual([doc.docid for doc in result], expected)
+ return expected
+
+ def test_query_no_classifier(self):
+ docs = self.test_index_many_no_classifier()
+ top = self._get_top_docs(docs)
+ result = self.index.query()
+ expected = top['huey'] + top['dooey'] + top['looey']
+ expected.sort()
+ expected = [docid for nil, docid in expected]
+ expected.reverse()
+ self.assertEqual([doc.docid for doc in result], expected[:10])
+
+ def test_query_no_classifier_ignores_value(self):
+ docs = self.test_index_many_no_classifier()
+ top = self._get_top_docs(docs)
+ result = self.index.query('ptooey')
+ expected = top['huey'] + top['dooey'] + top['looey']
+ expected.sort()
+ expected = [docid for nil, docid in expected]
+ expected.reverse()
+ self.assertEqual([doc.docid for doc in result], expected[:10])
+
+ def test_query_multiple_with_tuple(self):
+ expected = self.test_query_multiple_values()
+ result = self.index.query(('huey', 'dooey'))
+ self.assertEqual([doc.docid for doc in result], expected)
+
+ def test_query_multiple_bogus_values(self):
+ self.failIf(self.index.query(['fooey', 'blooey']))
+ result = self.index.query(['blooey', 'looey'])
+ expected = self.index.query('looey')
+ self.assertEqual(list(result), list(expected))
+
+ def test_query_multiple_limit(self):
+ expected = self.test_query_multiple_values()[:4]
+ result = self.index.query(['huey', 'dooey'], limit=4)
+ self.assertEqual([doc.docid for doc in result], expected)
+
+ def test_query_multiple_no_merge(self):
+ docs = self.test_index_many()
+ top = self._get_top_docs(docs)
+ result = self.index.query(['dooey', 'huey'], merge=0)
+ expected = [(date, docid, self.test.__getitem__)
+ for date, docid in top['huey'] + top['dooey']]
+ expected.sort()
+ expected.reverse()
+ for rrow, erow in zip(result, expected):
+ self.assertEqual(rrow[:2], erow[:2])
+
+ def test_apply_index(self):
+ # _apply_index always returns none since recent items index
+ # do not participate in the normal ZCatalog query as they
+ # handle both intersection and sorting
+ self.failUnless(self.index._apply_index({}) is None)
+ self.failUnless(self.index._apply_index({'query':'looey'}) is None)
+
+ def test_uniqueValues(self):
+ self.failIf(self.index.uniqueValues('type'))
+ docs = self.test_index_many()
+ values = list(self.index.uniqueValues('type'))
+ values.sort()
+ self.assertEqual(values, ['dooey', 'huey', 'looey'])
+ self.failIf(self.index.uniqueValues('carbtastic'))
+
+ def test_hasUniqueValuesFor(self):
+ self.failUnless(self.index.hasUniqueValuesFor('type'))
+ self.failIf(self.index.hasUniqueValuesFor('spork'))
+
+ def test_numObjects(self):
+ docs = self.test_index_many()
+ self.assertEqual(self.index.numObjects(), 30)
+
+ def test_numObjects_small_maxlen(self):
+ self.index.max_length = 1
+ docs = self.test_index_many()
+ self.assertEqual(self.index.numObjects(), 3)
+
+ def test_numObjects_empty_index(self):
+ self.assertEqual(self.index.numObjects(), 0)
+
+ def test_clear(self):
+ self.test_index_many()
+ self.failUnless(self.index.numObjects())
+ self.index.clear()
+ self.assertEqual(self.index.numObjects(), 0)
+
+ def test_role_permission_guard(self):
+ from Products.RecentItemsIndex.index import RecentItemsIndex
+ index = RecentItemsIndex(
+ 'test', 'type', 'date', 5, ['NerfHerder', 'Bloke'], 'View')
+ viewable = Viewable('NerfHerder')
+ index.index_object(0, viewable)
+ self.assertEqual(index.numObjects(), 1)
+ notviewable = Viewable()
+ index.index_object(1, notviewable)
+ self.assertEqual(index.numObjects(), 1)
+ bloke = Viewable('Bloke')
+ index.index_object(2, bloke)
+ self.assertEqual(index.numObjects(), 2)
+ bloke.manage_permission('View', [])
+ index.index_object(2, bloke)
+ self.assertEqual(index.numObjects(), 1)
+ dummy = Viewable('Dummy')
+ index.index_object(3, dummy)
+ self.assertEqual(index.numObjects(), 1)
+ viewable.manage_permission('View', [])
+ index.index_object(0, viewable)
+ self.assertEqual(index.numObjects(), 0)
+
+def test_suite():
+ return TestSuite((makeSuite(RecentItemsIndexTest),))
+
+if __name__=='__main__':
+ main(defaultTest='test_suite')
Added: Produts.RecentItemsIndex/trunk/version.txt
===================================================================
--- Produts.RecentItemsIndex/trunk/version.txt (rev 0)
+++ Produts.RecentItemsIndex/trunk/version.txt 2010-03-16 20:29:26 UTC (rev 109997)
@@ -0,0 +1 @@
+0.1
Added: Produts.RecentItemsIndex/trunk/www/addIndex.dtml
===================================================================
--- Produts.RecentItemsIndex/trunk/www/addIndex.dtml (rev 0)
+++ Produts.RecentItemsIndex/trunk/www/addIndex.dtml 2010-03-16 20:29:26 UTC (rev 109997)
@@ -0,0 +1,104 @@
+<dtml-var manage_page_header>
+
+<dtml-var "manage_form_title(this(), _,
+ form_title='Add RecentItemsIndex')">
+
+<p class="form-help">
+A recent items index classifies objects by an attribute field and sorts them by
+a date attribute. Only a fixed number of the most recent items are kept
+allowing fast queries whose performance does not degrade as the size of
+the catalog increases.
+</p>
+
+<form action="manage_addIndex" method="post">
+<input type="hidden" name="type" value="RecentItemsIndex" />
+<table cellspacing="0" cellpadding="2" border="0">
+ <tr>
+ <td align="left" valign="top">
+ <div class="form-label">
+ Id
+ </div>
+ </td>
+ <td align="left" valign="top">
+ <input type="text" name="name" size="40" />
+ </td>
+ </tr>
+
+ <tr>
+ <td align="left" valign="top">
+ <div class="form-label">
+ Classifier attribute
+ </div></td>
+ <td align="left" valign="top">
+ <input type="text" name="extra.field_name:record:ignore_empty" size="40" />
+ <br /><em class="form-help">For each value of this attribute, a
+ separate recent items list is indexed. Leave empty keep a single
+ list for all objects.</em>
+ </td>
+ </tr>
+
+ <tr>
+ <td align="left" valign="top">
+ <div class="form-label">
+ Date attribute
+ </div></td>
+ <td align="left" valign="top">
+ <input type="text" name="extra.date_name:record" size="40" />
+ <br /><em class="form-help">The date attribute which determines
+ each object's age.</em>
+ </td>
+ </tr>
+
+ <tr>
+ <td align="left" valign="top">
+ <div class="form-label">
+ Guard
+ </div></td>
+ <td align="left" valign="top">
+ <strong>Roles</strong><br />
+ <select name="extra.guard_roles:list:ignore_empty:record" size="4" multiple>
+ <dtml-in valid_roles>
+ <option value="&dtml-sequence-item;">&dtml-sequence-item;</option>
+ </dtml-in>
+ </select><br />
+ <strong>With Permission</strong><br />
+ <select name="extra.guard_permission:ignore_empty:record">
+ <option value="">(None)</option>
+ <dtml-in possible_permissions sort>
+ <option value="&dtml-sequence-item;">&dtml-sequence-item;</option>
+ </dtml-in>
+ </select>
+ <br /><em class="form-help">Only objects which have the selected
+ permission granted to one or more of the selected roles will
+ be included in the index.</em>
+ </td>
+ </tr>
+
+ <tr>
+ <td align="left" valign="top">
+ <div class="form-label">
+ Item count limit
+ </div></td>
+ <td align="left" valign="top">
+ <input type="text" name="extra.max_length:int:record" size="5"
+ value="20" />
+ <br /><em class="form-help">The maximum number of recent items
+ for each unique field value to hold in the index. Keep this value as
+ small as possible for best performance.</em>
+ </td>
+ </tr>
+
+ <tr>
+ <td align="left" valign="top">
+ </td>
+ <td align="left" valign="top">
+ <div class="form-element">
+ <input class="form-element" type="submit" name="submit"
+ value=" Add " />
+ </div>
+ </td>
+ </tr>
+</table>
+</form>
+
+<dtml-var manage_page_footer>
Added: Produts.RecentItemsIndex/trunk/www/index.gif
===================================================================
(Binary files differ)
Property changes on: Produts.RecentItemsIndex/trunk/www/index.gif
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: Produts.RecentItemsIndex/trunk/www/manageIndex.dtml
===================================================================
--- Produts.RecentItemsIndex/trunk/www/manageIndex.dtml (rev 0)
+++ Produts.RecentItemsIndex/trunk/www/manageIndex.dtml 2010-03-16 20:29:26 UTC (rev 109997)
@@ -0,0 +1,37 @@
+<dtml-var manage_page_header>
+<dtml-var manage_tabs>
+
+<p class="form-help">
+ Field Attribute: &dtml-field_name;
+</p>
+<p class="form-help">
+ Date Attribute: &dtml-date_name;
+</p>
+<p class="form-help">
+ Max Item Length: &dtml-max_length;
+</p>
+<dtml-if guard_permission>
+<p class="form-help">
+ Guard Permission: &dtml-guard_permission;<br/>
+ Guard Roles: <dtml-var expr="', '.join(guard_roles)" html_quote>
+</p>
+</dtml-if>
+<p class="form-help">
+ Item list sizes:
+ <table>
+ <tr>
+ <th>Field Value</th>
+ <th>Recent Item Count</th>
+ </tr>
+ <dtml-in expr="getItemCounts().items()">
+ <tr>
+ <td>&dtml-sequence-key;</td>
+ <td>&dtml-sequence-item;</td>
+ </tr>
+ </dtml-in>
+ <dtml-unless expr="getItemCounts().items()">
+ <tr><td><em>The index is empty</em></td></tr>
+ </dtml-unless>
+ </table>
+</p>
+<dtml-var manage_page_footer>
More information about the checkins
mailing list