[Checkins] SVN: lovely.remotetask/branches/port-for-zope210/ Merged r86667:103299 from trunk
Radim Novotny
novotny.radim at gmail.com
Thu Aug 27 13:10:44 EDT 2009
Log message for revision 103309:
Merged r86667:103299 from trunk
Changed:
U lovely.remotetask/branches/port-for-zope210/CHANGES.txt
U lovely.remotetask/branches/port-for-zope210/buildout.cfg
U lovely.remotetask/branches/port-for-zope210/setup.py
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/README.txt
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/README.txt
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/job.py
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/service.py
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/ftests.py
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/interfaces.py
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/processor.py
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/processor.txt
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/service.py
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/startlater.txt
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/tests.py
U lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/xmlrpc.txt
-=-
Modified: lovely.remotetask/branches/port-for-zope210/CHANGES.txt
===================================================================
--- lovely.remotetask/branches/port-for-zope210/CHANGES.txt 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/CHANGES.txt 2009-08-27 17:10:44 UTC (rev 103309)
@@ -2,15 +2,52 @@
Changes for lovely.remotetask
=============================
+unreleased (0.5):
+-----------------
+
+- Fixed a bug with SimpleProcessor: if the job aborted the transaction, it would
+ never be removed from the queue, but re-tried over and over again.
+
+2009/05/20 (0.4):
+-----------------
+
+- Randomized the generation of new job ids like intid does it: Try to allocate
+ sequential ids so they fall into the same BTree bucket, and randomize if
+ stumble upon a used one.
+
+2009/04/05 (0.3):
+-----------------
+
+- Use dropdown widget with available tasks in the cron job
+ adding form, instead of text input.
+
+- Remove dependency on zope.app.zapi by using its wrapped api directly.
+
+- Use ISite from zope.location instead of zope.app.component
+
+- Use zc.queue.Queue instead of zc.queue.PersistentQueue because
+ PersistentQueue is only to be used by the CompositeQueue.
+
+- Changed URL to pypi.
+
+- Using the correct plural form of status (which is status) in
+ ITaskService.clean
+
+
+2008/11/07 0.2.15a1:
+--------------------
+
+- running could cause an AttributeError. added handling for it
+
2008/02/08 0.2.14:
-==================
+------------------
- commiting after each 100 jobs during 'clearAll' to avoid browser timeouts
while canceling a huge amount of jobs
2008/01/28 (new):
-=================
+-----------------
- Some bugs smashed, improved tests.
@@ -19,7 +56,7 @@
2007/12/?? (new):
-=================
+-----------------
- Switched index to Zope 3.4 KGS, so that we agree on used package versions.
@@ -32,14 +69,14 @@
2007/11/12 0.2.13:
-==================
+------------------
- added "cancel all" button
- fixed bug in associating threads with task service instances
2007/10/28 0.2.12:
-==================
+------------------
- make the startup more robust
If an already registered task service is remove via ZMI it's registration is
@@ -48,7 +85,7 @@
2007/10/28 0.2.11:
-==================
+------------------
- allow '*' to select all possible times in the cron job add/edit forms
@@ -56,25 +93,25 @@
2007/10/24 0.2.10:
-==================
+------------------
- avoided deprecation warnings
2007/10/08 0.2.9:
-=================
+-----------------
- don't push a cron job back into the queue if it's status is ERROR
2007/10/08 0.2.8:
-=================
+-----------------
- enhanced logging during startup
2007/10/02 0.2.7:
-=================
+-----------------
- added index to buildout.cfg
- enhanced autostart behaviour: Services can be started like: site@*,
@@ -82,25 +119,25 @@
2007/08/07 0.2.6:
-=================
+-----------------
- fix bug in sorting that causes column headers to never be clickable
2007/08/07 0.2.5:
-=================
+-----------------
- no longer require session support for "Jobs" ZMI view
2007/08/06 0.2.4:
-=================
+-----------------
- fix bug that caused processing thread to keep the process alive unnecessarily
2007/07/26 0.2.3:
-=================
+-----------------
- Now handles the use-case where a task service is registered directly at the
root. References to such services in the product configuration must begin
@@ -108,7 +145,7 @@
2007/07/02 0.2.2:
-=================
+-----------------
- ZMI menu to add cron jobs to a task service
- named detail views can be registered for jobs specific to the task
@@ -119,14 +156,14 @@
2007/06/12 0.2.1:
-=================
+-----------------
- Do not raise IndexError because of performance problems with tracebacks when
using eggs.
2007/06/12 0.2.0:
-=================
+-----------------
- added namespace declaration in lovely/__init__.py
- allow to delay a job
Modified: lovely.remotetask/branches/port-for-zope210/buildout.cfg
===================================================================
--- lovely.remotetask/branches/port-for-zope210/buildout.cfg 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/buildout.cfg 2009-08-27 17:10:44 UTC (rev 103309)
@@ -1,7 +1,6 @@
[buildout]
develop = .
parts = py test
-index = http://download.zope.org/zope3.4
[test]
recipe = zc.recipe.testrunner
Modified: lovely.remotetask/branches/port-for-zope210/setup.py
===================================================================
--- lovely.remotetask/branches/port-for-zope210/setup.py 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/setup.py 2009-08-27 17:10:44 UTC (rev 103309)
@@ -1,15 +1,25 @@
#!python
from setuptools import setup, find_packages
+import os.path
+
+def read(*names):
+ return open(os.path.join(os.path.dirname(__file__), *names)).read()
+
+
setup (
name='lovely.remotetask',
- version='0.2.14',
+ version='0.5dev',
author = "Lovely Systems",
author_email = "office at lovelysystems.com",
description = "A remotetask client utiltiy for zope 3",
+ long_description=(
+ read('src', 'lovely', 'remotetask', 'README.txt')
+ + '\n\n'
+ + read('CHANGES.txt')),
license = "ZPL 2.1",
keywords = "zope3 zope remotetask cache ram",
- url = 'svn://svn.zope.org/repos/main/lovely.remotetask',
+ url = 'http://pypi.python.org/pypi/lovely.remotetask',
packages = find_packages('src'),
include_package_data = True,
package_dir = {'':'src'},
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/README.txt
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/README.txt 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/README.txt 2009-08-27 17:10:44 UTC (rev 103309)
@@ -2,6 +2,8 @@
Remote Task Execution
=====================
+.. contents::
+
This package provides an implementation of a remote task execution Web service
that allows to execute pre-defined tasks on another server. It is also
possible to run cron jobs at specific times. Those services are useful in two
@@ -113,7 +115,7 @@
>>> jobid = service.add(u'echo', {'foo': 'bar'})
>>> jobid
- 1
+ 1392637175
The ``add()`` function schedules the task called "echo" to be executed with
the specified arguments. The method returns a job id with which we can inquire
@@ -162,7 +164,7 @@
registered at the root. We test this use-case in a separate footnote so that
the flow of this document is not broken. [#1]_
-To get a clean logging environment let's clear the logging stack::
+To get a clean logging environment let's clear the logging stack:
>>> log_info.clear()
@@ -177,7 +179,7 @@
>>> conn.root()[ZopePublication.root_name] = root
>>> transaction.commit()
-Fire the event::
+Fire the event:
>>> from zope.app.appsetup.interfaces import DatabaseOpenedWithRoot
>>> from lovely.remotetask.service import bootStrapSubscriber
@@ -189,7 +191,7 @@
>>> service.isProcessing()
True
-Checking out the logging will prove the started service::
+Checking out the logging will prove the started service:
>>> print log_info
lovely.remotetask INFO
@@ -206,9 +208,9 @@
To deal with a lot of services in one sites it will be possible to use
asterisks (*) to start services. In case of using site@* means start all
-services in that site::
+services in that site:
-But first stop all processing services::
+But first stop all processing services:
>>> service.stopProcessing()
>>> service.isProcessing()
@@ -220,18 +222,18 @@
>>> import time; time.sleep(STOP_SLEEP_TIME)
-And reset the logger::
+And reset the logger:
>>> log_info.clear()
-Reset the product configuration with the asterisked service names::
+Reset the product configuration with the asterisked service names:
>>> config.mapping['autostart'] = 'site1@*'
>>> setProductConfigurations([config])
>>> getAutostartServiceNames()
['site1@*']
-Firing the event again will start all services in the configured site::
+Firing the event again will start all services in the configured site:
>>> bootStrapSubscriber(event)
@@ -241,7 +243,7 @@
>>> root_service.isProcessing()
False
-Let's checkout the logging::
+Let's checkout the logging:
>>> print log_info
lovely.remotetask INFO
@@ -251,7 +253,7 @@
To deal with a lot of services in a lot of sites it possible to use
asterisks (*) to start services. In case of using *@* means start all
-services on all sites::
+services on all sites:
>>> service.stopProcessing()
>>> service.isProcessing()
@@ -259,18 +261,18 @@
>>> import time; time.sleep(STOP_SLEEP_TIME)
-Reset the product configuration with the asterisked service names::
+Reset the product configuration with the asterisked service names:
>>> config.mapping['autostart'] = '*@*'
>>> setProductConfigurations([config])
>>> getAutostartServiceNames()
['*@*']
-...and reset the logger::
+...and reset the logger:
>>> log_info.clear()
-And fire the event again. All services should be started now::
+And fire the event again. All services should be started now:
>>> bootStrapSubscriber(event)
@@ -280,7 +282,7 @@
>>> root_service.isProcessing()
True
-Let's check the logging::
+Let's check the logging:
>>> print log_info
lovely.remotetask INFO
@@ -292,8 +294,8 @@
To deal with a specific service in a lot of sites it possible to use
-asterisks (*) to start services. In case of using *@service means start the
-service called `service` on all sites::
+asterisks (*) to start services. In case of using \*@service means start the
+service called `service` on all sites:
>>> service.stopProcessing()
>>> service.isProcessing()
@@ -305,18 +307,18 @@
>>> import time; time.sleep(STOP_SLEEP_TIME)
-Reset the product configuration with the asterisked service names::
+Reset the product configuration with the asterisked service names:
>>> config.mapping['autostart'] = '*@TestTaskService1'
>>> setProductConfigurations([config])
>>> getAutostartServiceNames()
['*@TestTaskService1']
-...and reset the logger::
+...and reset the logger:
>>> log_info.clear()
-And fire the event again. All services should be started now::
+And fire the event again. All services should be started now:
>>> bootStrapSubscriber(event)
@@ -326,7 +328,7 @@
>>> root_service.isProcessing()
False
-Let's checkout the logging::
+Let's checkout the logging:
>>> print log_info
lovely.remotetask INFO
@@ -335,7 +337,7 @@
service TestTaskService1 on site site1 started
In case of configuring a directive which does not match any service on
-any site logging will show a warning message::
+any site logging will show a warning message:
>>> service.stopProcessing()
>>> service.isProcessing()
@@ -410,7 +412,7 @@
For management purposes, the service also allows you to inspect all jobs:
>>> dict(service.jobs)
- {1: <Job 1>, 2: <Job 2>, 3: <Job 3>}
+ {1392637176: <Job 1392637176>, 1392637177: <Job 1392637177>, 1392637175: <Job 1392637175>}
To get rid of jobs not needed anymore one can use the clean method.
@@ -616,7 +618,7 @@
the database.
Let's start the task services we have defined at this point, and see
-what threads are running as a result::
+what threads are running as a result:
>>> service.startProcessing()
>>> root_service.startProcessing()
@@ -635,7 +637,7 @@
<Thread(remotetasks.site1.++etc++site.default.testTaskService1, started daemon)>]
Let's add a second site containing a task service with the same name as the
-service in the first site::
+service in the first site:
>>> site2 = Folder()
>>> service2 = remotetask.TaskService()
@@ -647,18 +649,18 @@
>>> sm['default']['testTaskService1'] = service2
>>> service2 = sm['default']['testTaskService1'] # caution! proxy
-Let's register it under the name `TestTaskService1`::
+Let's register it under the name `TestTaskService1`:
>>> sm = site2.getSiteManager()
>>> sm.registerUtility(
... service2, interfaces.ITaskService, name='TestTaskService1')
The service requires that it's been committed to the database before it can
-be used::
+be used:
>>> transaction.commit()
-The new service isn't currently processing::
+The new service isn't currently processing:
>>> service2.isProcessing()
False
@@ -673,7 +675,7 @@
<Thread(remotetasks.site2.++etc++site.default.testTaskService1, started daemon)>]
Let's stop the services, and give the background threads a chance to get the
-message::
+message:
>>> service.stopProcessing()
>>> service2.stopProcessing()
@@ -681,7 +683,7 @@
>>> import time; time.sleep(STOP_SLEEP_TIME)
-The threads have exited now::
+The threads have exited now:
>>> print [t for t in threading.enumerate()
... if t.getName().startswith('remotetasks.')]
@@ -699,7 +701,7 @@
>>> component.provideUtility(root_service, interfaces.ITaskService,
... name='RootTaskService')
- The object should be located, so it get's a name::
+ The object should be located, so it get's a name:
>>> root['rootTaskService'] = root_service
>>> root_service = root['rootTaskService'] # caution! proxy
@@ -711,7 +713,7 @@
>>> r_jobid = root_service.add(
... u'echo', {'foo': 'this is for root_service'})
>>> r_jobid
- 1
+ 1506179619
.. [#2] We verify the root_service does get processed:
@@ -761,4 +763,4 @@
>>> verifyObject(interfaces.ICronJob, fakecronjob)
True
>>> interfaces.IJob.providedBy(fakecronjob)
- True
\ No newline at end of file
+ True
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/README.txt
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/README.txt 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/README.txt 2009-08-27 17:10:44 UTC (rev 103309)
@@ -57,15 +57,14 @@
>>> browser.reload()
>>> print browser.contents
- <!DOCTYPE
- ...
+ <!DOCTYPE ...
<tbody>
<tr class="odd">
<td class="">
- <input type="checkbox" name="jobs:list" value="1">
+ <input type="checkbox" name="jobs:list" value="1506179619">
</td>
<td class="tableId">
- 1
+ 1506179619
</td>
<td class="tableTask">
echo
@@ -123,7 +122,8 @@
>>> 'No jobs were selected.' in browser.contents
True
- >>> browser.getControl(name='jobs:list').getControl(value='1').click()
+ >>> browser.getControl(name='jobs:list').getControl(
+ ... value='1506179619').click()
>>> browser.getControl('Cancel', index=0).click()
>>> 'Jobs were successfully cancelled.' in browser.contents
True
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/job.py
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/job.py 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/job.py 2009-08-27 17:10:44 UTC (rev 103309)
@@ -30,6 +30,8 @@
from zope.app.pagetemplate import ViewPageTemplateFile
from zope.app.form.browser.textwidgets import TextWidget
+from zope.app.form.browser.itemswidgets import DropdownWidget
+from zope.schema.vocabulary import SimpleTerm, SimpleVocabulary
from lovely.remotetask.interfaces import CRONJOB, ICronJob
@@ -103,6 +105,12 @@
values = tuple(range(0,7))
+class TaskWidget(DropdownWidget):
+
+ def __init__(self, field, request):
+ terms = [SimpleTerm(name) for name in field.context.getAvailableTasks()]
+ vocabulary = SimpleVocabulary(terms)
+ super(TaskWidget, self).__init__(field, vocabulary, request)
class CronJobFormBase(object):
"""base settings for all cron job forms"""
@@ -116,6 +124,7 @@
'dayOfWeek',
'delay',
)
+ form_fields['task'].custom_widget = TaskWidget
form_fields['hour'].custom_widget = HourWidget
form_fields['minute'].custom_widget = MinuteWidget
form_fields['dayOfMonth'].custom_widget = DayOfMonthWidget
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/service.py
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/service.py 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/browser/service.py 2009-08-27 17:10:44 UTC (rev 103309)
@@ -380,17 +380,17 @@
self.status = u'Cleaned %r Jobs' % cleaned
elif 'CLEAN_ERROR' in self.request:
jobs = len(list(self.context.jobs.keys()))
- self.context.clean(stati=[interfaces.ERROR])
+ self.context.clean(status=[interfaces.ERROR])
cleaned = jobs - len(list(self.context.jobs.keys()))
self.status = u'Cleaned %r Jobs' % cleaned
elif 'CLEAN_CANCELLED' in self.request:
jobs = len(list(self.context.jobs.keys()))
- self.context.clean(stati=[interfaces.CANCELLED])
+ self.context.clean(status=[interfaces.CANCELLED])
cleaned = jobs - len(list(self.context.jobs.keys()))
self.status = u'Cleaned %r Jobs' % cleaned
elif 'CLEAN_COMPLETED' in self.request:
jobs = len(list(self.context.jobs.keys()))
- self.context.clean(stati=[interfaces.COMPLETED])
+ self.context.clean(status=[interfaces.COMPLETED])
cleaned = jobs - len(list(self.context.jobs.keys()))
self.status = u'Cleaned %r Jobs' % cleaned
elif 'CANCEL_ALL' in self.request:
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/ftests.py
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/ftests.py 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/ftests.py 2009-08-27 17:10:44 UTC (rev 103309)
@@ -19,18 +19,27 @@
import unittest
from zope.app.testing import functional
import os
+import random
zcml = os.path.join(os.path.dirname(__file__), 'ftesting.zcml')
-functional.defineLayer('RemotetaskLayer', zcml)
+functional.defineLayer('RemotetaskLayer', zcml, allow_teardown=True)
+
+def setUp(test):
+ random.seed(27)
+
+
+def tearDown(test):
+ random.seed()
+
+
def test_suite():
- suite1 = functional.FunctionalDocFileSuite('xmlrpc.txt')
- suite2 = functional.FunctionalDocFileSuite('browser/README.txt')
+ suite1 = functional.FunctionalDocFileSuite(
+ 'browser/README.txt',
+ 'xmlrpc.txt',
+ setUp=setUp,
+ tearDown=tearDown,
+ )
suite1.layer = RemotetaskLayer
- suite2.layer = RemotetaskLayer
- return unittest.TestSuite((suite1, suite2))
-
-
-if __name__ == '__main__':
- unittest.main(defaultTest='test_suite')
+ return unittest.TestSuite((suite1, ))
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/interfaces.py
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/interfaces.py 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/interfaces.py 2009-08-27 17:10:44 UTC (rev 103309)
@@ -113,7 +113,7 @@
This is neccessary if the cron jobs parameters are changed.
"""
- def clean(stati=[CANCELLED, ERROR, COMPLETED]):
+ def clean(status=[CANCELLED, ERROR, COMPLETED]):
"""removes all jobs which are completed or canceled or have errors."""
def cancel(jobid):
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/processor.py
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/processor.py 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/processor.py 2009-08-27 17:10:44 UTC (rev 103309)
@@ -76,7 +76,11 @@
@property
def running(self):
- return threading.currentThread().running
+ thread = threading.currentThread()
+ if thread is not None:
+ return thread.running
+ log.error('SimpleProcessor: no currentThread')
+ return False
def __init__(self, db, servicePath, waitTime=1.0):
self.db = db
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/processor.txt
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/processor.txt 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/processor.txt 2009-08-27 17:10:44 UTC (rev 103309)
@@ -120,7 +120,40 @@
lovely.remotetask INFO
Job: 4
+Transactions in jobs
+--------------------
+With the SimpleProcessor, jobs _should_ not change the transaction status, since
+both the administration of the jobs by the TaskService and the job itself run in
+the same transaction, so aborting it from inside the job could wreak havoc with
+the administrative part.
+
+This is a regression test that aborting the transaction inside the job does not
+lead to an infinite loop (because SimpleProcessor pulls the job inside the
+transaction, so if it is aborted, the job remains on the queue):
+
+ >>> counter = 0
+ >>> def count(arg):
+ ... global counter
+ ... counter += 1
+ ... transaction.abort()
+ >>> countTask = remotetask.task.SimpleTask(count)
+ >>> zope.component.provideUtility(countTask, name='count')
+
+ >>> jobid = tasks.add(u'count', ())
+ >>> transaction.commit()
+
+ >>> tasks.startProcessing()
+ >>> transaction.commit()
+ >>> time.sleep(0.5)
+ >>> tasks.stopProcessing()
+ >>> transaction.commit()
+ >>> time.sleep(0.5)
+ >>> transaction.abort() # prevent spurious conflict errors
+ >>> counter
+ 1
+
+
The Multi-thread Processor
--------------------------
@@ -164,7 +197,7 @@
>>> jobid = proc.claimNextJob()
>>> jobid
- 5
+ 1392637180
We need to claim a job before executing it, so that the database marks the job
as claimed and no new thread picks up the job. Once we claimed a particular
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/service.py
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/service.py 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/service.py 2009-08-27 17:10:44 UTC (rev 103309)
@@ -13,13 +13,20 @@
##############################################################################
"""Task Service Implementation
-$Id$
"""
__docformat__ = 'restructuredtext'
+from lovely.remotetask import interfaces, job, task, processor
+from zope import component
+from zope.app.container import contained
+from zope.app.publication.zopepublication import ZopePublication
+from zope.component.interfaces import ComponentLookupError
+from zope.traversing.api import getParents
+import BTrees
import datetime
import logging
import persistent
+import random
import threading
import time
import zc.queue
@@ -32,7 +39,9 @@
from zope.app.publication.zopepublication import ZopePublication
from zope.component.interfaces import ComponentLookupError
from lovely.remotetask import interfaces, job, task, processor
+import zope.location
+
log = logging.getLogger('lovely.remotetask')
try:
@@ -40,12 +49,14 @@
from App.config import getConfiguration
ZOPE2 = True
del Five
+ import sys
except ImportError:
from zope.app.appsetup.product import getProductConfiguration
ZOPE2 = False
storage = threading.local()
+
class TaskService(contained.Contained, persistent.Persistent):
"""A persistent task service.
@@ -63,14 +74,20 @@
_scheduledJobs = None
_scheduledQueue = None
+ _v_nextid = None
+ if not ZOPE2:
+ family = BTrees.family32
def __init__(self):
super(TaskService, self).__init__()
- self._counter = 1
- self.jobs = IOBTree()
- self._queue = zc.queue.PersistentQueue()
- self._scheduledJobs = IOBTree()
- self._scheduledQueue = zc.queue.PersistentQueue()
+ if not ZOPE2:
+ self.jobs = self.family.IO.BTree()
+ self._scheduledJobs = self.family.IO.BTree()
+ else:
+ self.jobs = IOBTree()
+ self._scheduledJobs = IOBTree()
+ self._queue = zc.queue.Queue()
+ self._scheduledQueue = zc.queue.Queue()
def getAvailableTasks(self):
"""See interfaces.ITaskService"""
@@ -80,8 +97,7 @@
"""See interfaces.ITaskService"""
if task not in self.getAvailableTasks():
raise ValueError('Task does not exist')
- jobid = self._counter
- self._counter += 1
+ jobid = self._generateId()
newjob = job.Job(jobid, task, input)
self.jobs[jobid] = newjob
if startLater:
@@ -99,8 +115,7 @@
dayOfWeek=(),
delay=None,
):
- jobid = self._counter
- self._counter += 1
+ jobid = self._generateId()
newjob = job.CronJob(jobid, task, input,
minute, hour, dayOfMonth, month, dayOfWeek, delay)
self.jobs[jobid] = newjob
@@ -122,16 +137,16 @@
def reschedule(self, jobid):
self._scheduledQueue.put(self.jobs[jobid])
- def clean(self, stati=[interfaces.CANCELLED, interfaces.ERROR,
- interfaces.COMPLETED]):
+ def clean(self, status=[interfaces.CANCELLED, interfaces.ERROR,
+ interfaces.COMPLETED]):
"""See interfaces.ITaskService"""
allowed = [interfaces.CANCELLED, interfaces.ERROR,
interfaces.COMPLETED]
for key in list(self.jobs.keys()):
job = self.jobs[key]
- if job.status in stati:
+ if job.status in status:
if job.status not in allowed:
- raise ValueError('Not allowed status for removing. %s' % \
+ raise ValueError('Not allowed status for removing. %s' %
job.status)
del self.jobs[key]
@@ -166,15 +181,17 @@
"""See interfaces.ITaskService"""
if self.__parent__ is None:
return
- if self._scheduledJobs == None:
- self._scheduledJobs = IOBTree()
- if self._scheduledQueue == None:
+ if self._scheduledJobs is None:
+ if not ZOPE2:
+ self._scheduledJobs = self.family.IOB.Tree()
+ else:
+ self._scheduledJobs = IOBTree()
+ if self._scheduledQueue is None:
self._scheduledQueue = zc.queue.PersistentQueue()
# Create the path to the service within the DB.
if not ZOPE2:
- servicePath = [parent.__name__ for parent in zapi.getParents(self)
- if parent.__name__]
- servicePath.append(self.__name__)
+ servicePath = [parent.__name__ for parent in getParents(self)
+ if parent.__name__]
else:
servicePath = [path for path in self.getPhysicalPath() if path]
servicePath.reverse()
@@ -212,7 +229,7 @@
# This name isn't unique based on the path to self, but this doesn't
# change the name that's been used in past versions.
if not ZOPE2:
- path = [parent.__name__ for parent in zapi.getParents(self)
+ path = [parent.__name__ for parent in getParents(self)
if parent.__name__]
path.append('remotetasks')
path.reverse()
@@ -257,6 +274,8 @@
job = self.jobs[jobid]
if job is None:
return False
+ if job.status == interfaces.COMPLETED:
+ return True
try:
jobtask = component.getUtility(self.taskInterface, name=job.task)
except ComponentLookupError, error:
@@ -349,7 +368,27 @@
jobs = self._scheduledJobs[nextCallTime]
self._scheduledJobs[nextCallTime] = jobs + (job,)
+ def _generateId(self):
+ """Generate an id which is not yet taken.
+ This tries to allocate sequential ids so they fall into the
+ same BTree bucket, and randomizes if it stumbles upon a
+ used one.
+ """
+ while True:
+ if self._v_nextid is None:
+ if not ZOPE2:
+ self._v_nextid = random.randrange(0, self.family.maxint)
+ else:
+ self._v_nextid = random.randrange(0, sys.maxint)
+ uid = self._v_nextid
+ self._v_nextid += 1
+ if uid not in self.jobs:
+ return uid
+ self._v_nextid = None
+
+
+
def getAutostartServiceNames():
"""get a list of services to start"""
@@ -394,7 +433,7 @@
sites = []
sites.append(root_folder)
for folder in root_folder.values():
- if ISite.providedBy(folder):
+ if zope.location.interfaces.ISite.providedBy(folder):
sites.append(folder)
else:
sites = [root_folder.get(siteName)]
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/startlater.txt
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/startlater.txt 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/startlater.txt 2009-08-27 17:10:44 UTC (rev 103309)
@@ -61,7 +61,7 @@
>>> jobid = service.add(u'echo', {'foo': 'bar'}, startLater=True)
>>> jobid
- 1
+ 1392637175
The ``add()`` function schedules the task called "echo" to be executed with
the specified arguments. The method returns a job id with which we can inquire
@@ -87,7 +87,7 @@
>>> jobid = service.add(u'echo', {'foo': 'bar'}, startLater=True)
>>> jobid
- 2
+ 1392637176
It's still in the status ``start later``:
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/tests.py
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/tests.py 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/tests.py 2009-08-27 17:10:44 UTC (rev 103309)
@@ -13,21 +13,20 @@
##############################################################################
"""Remote Task test setup
-$Id$
"""
__docformat__ = "reStructuredText"
+from lovely.remotetask import service
+from zope.app.testing import placelesssetup
+from zope.app.testing.setup import (placefulSetUp, placefulTearDown)
+from zope.testing.doctest import INTERPRET_FOOTNOTES
+from zope.testing.doctestunit import DocFileSuite
+from zope.testing.loggingsupport import InstalledHandler
import doctest
import logging
+import random
import unittest
-from zope.app.testing import placelesssetup
-from zope.app.testing.setup import (placefulSetUp,
- placefulTearDown)
-from zope.testing.doctestunit import DocFileSuite
-from zope.testing.doctest import INTERPRET_FOOTNOTES
-from zope.testing.loggingsupport import InstalledHandler
-from lovely.remotetask import service
def setUp(test):
root = placefulSetUp(site=True)
@@ -37,41 +36,55 @@
test.globs['log_info'] = log_info
test.origArgs = service.TaskService.processorArguments
service.TaskService.processorArguments = {'waitTime': 0.0}
+ # Make tests predictable
+ random.seed(27)
def tearDown(test):
+ random.seed()
placefulTearDown()
log_info = test.globs['log_info']
log_info.clear()
log_info.uninstall()
service.TaskService.processorArguments = test.origArgs
+
+
+class TestIdGenerator(unittest.TestCase):
+
+ def setUp(self):
+ random.seed(27)
+ self.service = service.TaskService()
+
+ def tearDown(self):
+ random.seed()
+
+ def test_sequence(self):
+ self.assertEquals(1392637175, self.service._generateId())
+ self.assertEquals(1392637176, self.service._generateId())
+ self.assertEquals(1392637177, self.service._generateId())
+ self.assertEquals(1392637178, self.service._generateId())
+
+ def test_in_use_randomises(self):
+ self.assertEquals(1392637175, self.service._generateId())
+ self.service.jobs[1392637176] = object()
+ self.assertEquals(1506179619, self.service._generateId())
+ self.assertEquals(1506179620, self.service._generateId())
+ self.service.jobs[1506179621] = object()
+ self.assertEquals(2055242787, self.service._generateId())
+
+
+
def test_suite():
return unittest.TestSuite((
+ unittest.makeSuite(TestIdGenerator),
DocFileSuite('README.txt',
+ 'startlater.txt',
+ 'processor.txt',
+ 'TESTING.txt',
setUp=setUp,
tearDown=tearDown,
optionflags=doctest.NORMALIZE_WHITESPACE
|doctest.ELLIPSIS
|INTERPRET_FOOTNOTES
),
- DocFileSuite('startlater.txt',
- setUp=setUp,
- tearDown=tearDown,
- optionflags=doctest.NORMALIZE_WHITESPACE
- |doctest.ELLIPSIS
- ),
- DocFileSuite('processor.txt',
- setUp=setUp,
- tearDown=tearDown,
- optionflags=doctest.NORMALIZE_WHITESPACE
- |doctest.ELLIPSIS
- ),
- DocFileSuite('TESTING.txt',
- setUp=placelesssetup.setUp,
- tearDown=placelesssetup.tearDown,
- optionflags=doctest.NORMALIZE_WHITESPACE|doctest.ELLIPSIS,
- ),
))
-
-if __name__ == '__main__':
- unittest.main(defaultTest='test_suite')
Modified: lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/xmlrpc.txt
===================================================================
--- lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/xmlrpc.txt 2009-08-27 16:34:47 UTC (rev 103308)
+++ lovely.remotetask/branches/port-for-zope210/src/lovely/remotetask/xmlrpc.txt 2009-08-27 17:10:44 UTC (rev 103309)
@@ -96,7 +96,7 @@
<methodResponse>
<params>
<param>
- <value><int>1</int></value>
+ <value><int>1392637175</int></value>
</param>
</params>
</methodResponse>
@@ -117,7 +117,7 @@
... <methodCall>
... <methodName>cancel</methodName>
... <params>
- ... <value><int>1</int></value>
+ ... <value><int>1392637175</int></value>
... </params>
... </methodCall>
... """)
@@ -166,7 +166,7 @@
... <methodCall>
... <methodName>getStatus</methodName>
... <params>
- ... <value><int>2</int></value>
+ ... <value><int>1392637176</int></value>
... </params>
... </methodCall>
... """)
@@ -193,7 +193,7 @@
... <methodCall>
... <methodName>getStatus</methodName>
... <params>
- ... <value><int>2</int></value>
+ ... <value><int>1392637176</int></value>
... </params>
... </methodCall>
... """)
@@ -224,7 +224,7 @@
... <methodCall>
... <methodName>getResult</methodName>
... <params>
- ... <value><int>2</int></value>
+ ... <value><int>1392637176</int></value>
... </params>
... </methodCall>
... """)
@@ -261,7 +261,7 @@
... <methodCall>
... <methodName>getError</methodName>
... <params>
- ... <value><int>2</int></value>
+ ... <value><int>1392637176</int></value>
... </params>
... </methodCall>
... """)
@@ -306,7 +306,7 @@
... <methodCall>
... <methodName>getError</methodName>
... <params>
- ... <value><int>3</int></value>
+ ... <value><int>1392637177</int></value>
... </params>
... </methodCall>
... """)
@@ -352,7 +352,6 @@
<value><string>getError</string></value>
<value><string>getResult</string></value>
<value><string>getStatus</string></value>
- <value><string>skin</string></value>
</data></array></value>
</param>
</params>
More information about the Checkins
mailing list