[Checkins] SVN: lovely.remotetask/trunk/ - Randomized the generation of new job ids like intid does it: Try to allocate
Christian Zagrodnick
cz at gocept.com
Wed May 20 04:32:04 EDT 2009
Log message for revision 100161:
- Randomized the generation of new job ids like intid does it: Try to allocate
sequential ids so they fall into the same BTree bucket, and randomize if
stumble upon a used one.
Changed:
U lovely.remotetask/trunk/CHANGES.txt
U lovely.remotetask/trunk/src/lovely/remotetask/README.txt
U lovely.remotetask/trunk/src/lovely/remotetask/browser/README.txt
U lovely.remotetask/trunk/src/lovely/remotetask/ftests.py
U lovely.remotetask/trunk/src/lovely/remotetask/processor.txt
U lovely.remotetask/trunk/src/lovely/remotetask/service.py
U lovely.remotetask/trunk/src/lovely/remotetask/startlater.txt
U lovely.remotetask/trunk/src/lovely/remotetask/tests.py
U lovely.remotetask/trunk/src/lovely/remotetask/xmlrpc.txt
-=-
Modified: lovely.remotetask/trunk/CHANGES.txt
===================================================================
--- lovely.remotetask/trunk/CHANGES.txt 2009-05-20 08:24:08 UTC (rev 100160)
+++ lovely.remotetask/trunk/CHANGES.txt 2009-05-20 08:32:04 UTC (rev 100161)
@@ -5,6 +5,10 @@
unreleased (0.4):
-----------------
+- Randomized the generation of new job ids like intid does it: Try to allocate
+ sequential ids so they fall into the same BTree bucket, and randomize if
+ stumble upon a used one.
+
2009/04/05 (0.3):
-----------------
Modified: lovely.remotetask/trunk/src/lovely/remotetask/README.txt
===================================================================
--- lovely.remotetask/trunk/src/lovely/remotetask/README.txt 2009-05-20 08:24:08 UTC (rev 100160)
+++ lovely.remotetask/trunk/src/lovely/remotetask/README.txt 2009-05-20 08:32:04 UTC (rev 100161)
@@ -104,7 +104,7 @@
>>> jobid = service.add(u'echo', {'foo': 'bar'})
>>> jobid
- 1
+ 1392637175
The ``add()`` function schedules the task called "echo" to be executed with
the specified arguments. The method returns a job id with which we can inquire
@@ -401,7 +401,7 @@
For management purposes, the service also allows you to inspect all jobs:
>>> dict(service.jobs)
- {1: <Job 1>, 2: <Job 2>, 3: <Job 3>}
+ {1392637176: <Job 1392637176>, 1392637177: <Job 1392637177>, 1392637175: <Job 1392637175>}
To get rid of jobs not needed anymore one can use the clean method.
@@ -702,7 +702,7 @@
>>> r_jobid = root_service.add(
... u'echo', {'foo': 'this is for root_service'})
>>> r_jobid
- 1
+ 1506179619
.. [#2] We verify the root_service does get processed:
Modified: lovely.remotetask/trunk/src/lovely/remotetask/browser/README.txt
===================================================================
--- lovely.remotetask/trunk/src/lovely/remotetask/browser/README.txt 2009-05-20 08:24:08 UTC (rev 100160)
+++ lovely.remotetask/trunk/src/lovely/remotetask/browser/README.txt 2009-05-20 08:32:04 UTC (rev 100161)
@@ -57,15 +57,14 @@
>>> browser.reload()
>>> print browser.contents
- <!DOCTYPE
- ...
+ <!DOCTYPE ...
<tbody>
<tr class="odd">
<td class="">
- <input type="checkbox" name="jobs:list" value="1">
+ <input type="checkbox" name="jobs:list" value="1506179619">
</td>
<td class="tableId">
- 1
+ 1506179619
</td>
<td class="tableTask">
echo
@@ -123,7 +122,8 @@
>>> 'No jobs were selected.' in browser.contents
True
- >>> browser.getControl(name='jobs:list').getControl(value='1').click()
+ >>> browser.getControl(name='jobs:list').getControl(
+ ... value='1506179619').click()
>>> browser.getControl('Cancel', index=0).click()
>>> 'Jobs were successfully cancelled.' in browser.contents
True
Modified: lovely.remotetask/trunk/src/lovely/remotetask/ftests.py
===================================================================
--- lovely.remotetask/trunk/src/lovely/remotetask/ftests.py 2009-05-20 08:24:08 UTC (rev 100160)
+++ lovely.remotetask/trunk/src/lovely/remotetask/ftests.py 2009-05-20 08:32:04 UTC (rev 100161)
@@ -19,18 +19,27 @@
import unittest
from zope.app.testing import functional
import os
+import random
zcml = os.path.join(os.path.dirname(__file__), 'ftesting.zcml')
-functional.defineLayer('RemotetaskLayer', zcml)
+functional.defineLayer('RemotetaskLayer', zcml, allow_teardown=True)
+
+def setUp(test):
+ random.seed(27)
+
+
+def tearDown(test):
+ random.seed()
+
+
def test_suite():
- suite1 = functional.FunctionalDocFileSuite('xmlrpc.txt')
- suite2 = functional.FunctionalDocFileSuite('browser/README.txt')
+ suite1 = functional.FunctionalDocFileSuite(
+ 'browser/README.txt',
+ 'xmlrpc.txt',
+ setUp=setUp,
+ tearDown=tearDown,
+ )
suite1.layer = RemotetaskLayer
- suite2.layer = RemotetaskLayer
- return unittest.TestSuite((suite1, suite2))
-
-
-if __name__ == '__main__':
- unittest.main(defaultTest='test_suite')
+ return unittest.TestSuite((suite1, ))
Modified: lovely.remotetask/trunk/src/lovely/remotetask/processor.txt
===================================================================
--- lovely.remotetask/trunk/src/lovely/remotetask/processor.txt 2009-05-20 08:24:08 UTC (rev 100160)
+++ lovely.remotetask/trunk/src/lovely/remotetask/processor.txt 2009-05-20 08:32:04 UTC (rev 100161)
@@ -164,7 +164,7 @@
>>> jobid = proc.claimNextJob()
>>> jobid
- 5
+ 1392637179
We need to claim a job before executing it, so that the database marks the job
as claimed and no new thread picks up the job. Once we claimed a particular
Modified: lovely.remotetask/trunk/src/lovely/remotetask/service.py
===================================================================
--- lovely.remotetask/trunk/src/lovely/remotetask/service.py 2009-05-20 08:24:08 UTC (rev 100160)
+++ lovely.remotetask/trunk/src/lovely/remotetask/service.py 2009-05-20 08:32:04 UTC (rev 100161)
@@ -20,12 +20,13 @@
import datetime
import logging
import persistent
+import random
import threading
import time
import zc.queue
import zope.interface
import zope.location
-from BTrees.IOBTree import IOBTree
+import BTrees
from lovely.remotetask import interfaces, job, task, processor
from zope import component
from zope.app.appsetup.product import getProductConfiguration
@@ -51,13 +52,14 @@
_scheduledJobs = None
_scheduledQueue = None
+ _v_nextid = None
+ family = BTrees.family32
def __init__(self):
super(TaskService, self).__init__()
- self._counter = 1
- self.jobs = IOBTree()
+ self.jobs = self.family.IO.BTree()
self._queue = zc.queue.Queue()
- self._scheduledJobs = IOBTree()
+ self._scheduledJobs = self.family.IO.BTree()
self._scheduledQueue = zc.queue.Queue()
def getAvailableTasks(self):
@@ -68,8 +70,7 @@
"""See interfaces.ITaskService"""
if task not in self.getAvailableTasks():
raise ValueError('Task does not exist')
- jobid = self._counter
- self._counter += 1
+ jobid = self._generateId()
newjob = job.Job(jobid, task, input)
self.jobs[jobid] = newjob
if startLater:
@@ -87,8 +88,7 @@
dayOfWeek=(),
delay=None,
):
- jobid = self._counter
- self._counter += 1
+ jobid = self._generateId()
newjob = job.CronJob(jobid, task, input,
minute, hour, dayOfMonth, month, dayOfWeek, delay)
self.jobs[jobid] = newjob
@@ -119,7 +119,7 @@
job = self.jobs[key]
if job.status in status:
if job.status not in allowed:
- raise ValueError('Not allowed status for removing. %s' % \
+ raise ValueError('Not allowed status for removing. %s' %
job.status)
del self.jobs[key]
@@ -154,9 +154,9 @@
"""See interfaces.ITaskService"""
if self.__parent__ is None:
return
- if self._scheduledJobs == None:
- self._scheduledJobs = IOBTree()
- if self._scheduledQueue == None:
+ if self._scheduledJobs is None:
+ self._scheduledJobs = self.family.IOB.Tree()
+ if self._scheduledQueue is None:
self._scheduledQueue = zc.queue.PersistentQueue()
# Create the path to the service within the DB.
servicePath = [parent.__name__ for parent in getParents(self)
@@ -329,7 +329,24 @@
jobs = self._scheduledJobs[nextCallTime]
self._scheduledJobs[nextCallTime] = jobs + (job,)
+ def _generateId(self):
+ """Generate an id which is not yet taken.
+ This tries to allocate sequential ids so they fall into the
+ same BTree bucket, and randomizes if it stumbles upon a
+ used one.
+ """
+ while True:
+ if self._v_nextid is None:
+ self._v_nextid = random.randrange(0, self.family.maxint)
+ uid = self._v_nextid
+ self._v_nextid += 1
+ if uid not in self.jobs:
+ return uid
+ self._v_nextid = None
+
+
+
def getAutostartServiceNames():
"""get a list of services to start"""
Modified: lovely.remotetask/trunk/src/lovely/remotetask/startlater.txt
===================================================================
--- lovely.remotetask/trunk/src/lovely/remotetask/startlater.txt 2009-05-20 08:24:08 UTC (rev 100160)
+++ lovely.remotetask/trunk/src/lovely/remotetask/startlater.txt 2009-05-20 08:32:04 UTC (rev 100161)
@@ -61,7 +61,7 @@
>>> jobid = service.add(u'echo', {'foo': 'bar'}, startLater=True)
>>> jobid
- 1
+ 1392637175
The ``add()`` function schedules the task called "echo" to be executed with
the specified arguments. The method returns a job id with which we can inquire
@@ -87,7 +87,7 @@
>>> jobid = service.add(u'echo', {'foo': 'bar'}, startLater=True)
>>> jobid
- 2
+ 1392637176
It's still in the status ``start later``:
Modified: lovely.remotetask/trunk/src/lovely/remotetask/tests.py
===================================================================
--- lovely.remotetask/trunk/src/lovely/remotetask/tests.py 2009-05-20 08:24:08 UTC (rev 100160)
+++ lovely.remotetask/trunk/src/lovely/remotetask/tests.py 2009-05-20 08:32:04 UTC (rev 100161)
@@ -19,6 +19,7 @@
import doctest
import logging
+import random
import unittest
from zope.app.testing import placelesssetup
from zope.app.testing.setup import (placefulSetUp,
@@ -29,6 +30,7 @@
from lovely.remotetask import service
+
def setUp(test):
root = placefulSetUp(site=True)
test.globs['root'] = root
@@ -37,16 +39,47 @@
test.globs['log_info'] = log_info
test.origArgs = service.TaskService.processorArguments
service.TaskService.processorArguments = {'waitTime': 0.0}
+ # Make tests predictable
+ random.seed(27)
def tearDown(test):
+ random.seed()
placefulTearDown()
log_info = test.globs['log_info']
log_info.clear()
log_info.uninstall()
service.TaskService.processorArguments = test.origArgs
+
+
+class TestIdGenerator(unittest.TestCase):
+
+ def setUp(self):
+ random.seed(27)
+ self.service = service.TaskService()
+
+ def tearDown(self):
+ random.seed()
+
+ def test_sequence(self):
+ self.assertEquals(1392637175, self.service._generateId())
+ self.assertEquals(1392637176, self.service._generateId())
+ self.assertEquals(1392637177, self.service._generateId())
+ self.assertEquals(1392637178, self.service._generateId())
+
+ def test_in_use_randomises(self):
+ self.assertEquals(1392637175, self.service._generateId())
+ self.service.jobs[1392637176] = object()
+ self.assertEquals(1506179619, self.service._generateId())
+ self.assertEquals(1506179620, self.service._generateId())
+ self.service.jobs[1506179621] = object()
+ self.assertEquals(2055242787, self.service._generateId())
+
+
+
def test_suite():
return unittest.TestSuite((
+ unittest.makeSuite(TestIdGenerator),
DocFileSuite('README.txt',
setUp=setUp,
tearDown=tearDown,
Modified: lovely.remotetask/trunk/src/lovely/remotetask/xmlrpc.txt
===================================================================
--- lovely.remotetask/trunk/src/lovely/remotetask/xmlrpc.txt 2009-05-20 08:24:08 UTC (rev 100160)
+++ lovely.remotetask/trunk/src/lovely/remotetask/xmlrpc.txt 2009-05-20 08:32:04 UTC (rev 100161)
@@ -96,7 +96,7 @@
<methodResponse>
<params>
<param>
- <value><int>1</int></value>
+ <value><int>1392637175</int></value>
</param>
</params>
</methodResponse>
@@ -117,7 +117,7 @@
... <methodCall>
... <methodName>cancel</methodName>
... <params>
- ... <value><int>1</int></value>
+ ... <value><int>1392637175</int></value>
... </params>
... </methodCall>
... """)
@@ -166,7 +166,7 @@
... <methodCall>
... <methodName>getStatus</methodName>
... <params>
- ... <value><int>2</int></value>
+ ... <value><int>1392637176</int></value>
... </params>
... </methodCall>
... """)
@@ -193,7 +193,7 @@
... <methodCall>
... <methodName>getStatus</methodName>
... <params>
- ... <value><int>2</int></value>
+ ... <value><int>1392637176</int></value>
... </params>
... </methodCall>
... """)
@@ -224,7 +224,7 @@
... <methodCall>
... <methodName>getResult</methodName>
... <params>
- ... <value><int>2</int></value>
+ ... <value><int>1392637176</int></value>
... </params>
... </methodCall>
... """)
@@ -261,7 +261,7 @@
... <methodCall>
... <methodName>getError</methodName>
... <params>
- ... <value><int>2</int></value>
+ ... <value><int>1392637176</int></value>
... </params>
... </methodCall>
... """)
@@ -306,7 +306,7 @@
... <methodCall>
... <methodName>getError</methodName>
... <params>
- ... <value><int>3</int></value>
+ ... <value><int>1392637177</int></value>
... </params>
... </methodCall>
... """)
More information about the Checkins
mailing list