[Checkins] SVN: zc.async/trunk/src/zc/async/ progress on documenting new monitor functions.
Gary Poster
gary at modernsongs.com
Thu Sep 11 14:19:00 EDT 2008
Log message for revision 91063:
progress on documenting new monitor functions.
Changed:
U zc.async/trunk/src/zc/async/TODO.txt
U zc.async/trunk/src/zc/async/monitordb.py
-=-
Modified: zc.async/trunk/src/zc/async/TODO.txt
===================================================================
--- zc.async/trunk/src/zc/async/TODO.txt 2008-09-11 14:15:54 UTC (rev 91062)
+++ zc.async/trunk/src/zc/async/TODO.txt 2008-09-11 18:18:58 UTC (rev 91063)
@@ -3,6 +3,7 @@
- zc.monitor instead of zc.z3monitor
- improvements to thread teardown
- docstrings for monitordb functions
+- waiting thread testing notification improvements (Patrick)
Improvements
Modified: zc.async/trunk/src/zc/async/monitordb.py
===================================================================
--- zc.async/trunk/src/zc/async/monitordb.py 2008-09-11 14:15:54 UTC (rev 91062)
+++ zc.async/trunk/src/zc/async/monitordb.py 2008-09-11 18:18:58 UTC (rev 91063)
@@ -161,11 +161,316 @@
yield j
def jobs(context, *states, **kwargs):
- """Return jobs in one or more states."""
+ """Return jobs in one or more states.
+
+ Jobs are identified by integer OID and database name. These identifiers
+ can be used with the "asyncdb job" command to get details about the jobs.
+ The integer OIDs can be used in a database connection to get the job with
+ ``connection.get(ZODB.utils.p64(INTEGER_OID))``.
+
+ The asyncdb commands "jobs," "count," "jobstats," and "firstjob" all share
+ the same arguments, which are described below; after which usage examples
+ for this command are listed.
+
+ Arguments
+ =========
+
+ States
+ ------
+
+ You must provide at least one of the following states.
+
+ - "pending": the job is in a queue, waiting to be started.
+
+ - "assigned": a dispatcher has claimed the job and assigned it to one of
+ its worker threads. Work has not yet begun. Jobs are in this state very
+ briefly.
+
+ - "active": A worker thread is performing this job.
+
+ - "callbacks": the job's work is ended, and the thread is performing the
+ callbacks, if any.
+
+ - "completed": the job and its callbacks are completed. Completed jobs
+ stay in the database for only a certain amount of time--between seven and
+ eight days in the default agent implementation.
+
+ - "succeeded": the job completed successfully (that is, without raising an
+ unhandled exception, and without returning an explicit
+ twisted.python.failure.Failure). This is a subset of "completed,"
+ described above.
+
+ - "failed": the job completed by raising an unhandled exception or by
+ explicitly returning a twisted.python.failure.Failure. This is a subset
+ of "completed," described above.
+
+ You may use no more than one of the states "completed," "succeeded," and
+ "failed".
+
+ Optional Arguments
+ ------------------
+
+ You can further filter your results with a number of optional arguments.
+
+ "Shell-style glob wildcards," as referred to in this list, are "?", "*",
+ "[SEQUENCE]", and "[!SEQUENCE]", as described in
+ http://docs.python.org/lib/module-fnmatch.html .
+
+ A "duration-based filter" described in this list accepts an argument that
+ is of the form "sinceINTERVAL", "beforeINTERVAL", or
+ "sinceINTERVAL,beforeINTERVAL" (no space!). The "INTERVAL"s are of the
+ form ``[nD][nH][nM][nS]``, where "n" should be replaced with a positive
+ integer, and "D," "H," "M," and "S" are literals standing for "days,"
+ "hours," "minutes," and "seconds." For instance, you might use ``5M`` for
+ five minutes, ``20S`` for twenty seconds, or ``1H30M`` for an hour and a
+ half. Thus "before30M" would mean "thirty minutes ago or older."
+ "since45S" would mean "45 seconds ago or newer." "since1H,before30M" would
+ mean "between thirty minutes and an hour ago." Note that reversing the two
+ components in the last example--"before30M,since1H"--is equivalent.
+
+
+ - "callable": filters by callable name. Supports shell-style glob
+ wildcards. If you do not include a "." in the string, it matches only on
+ the callable name. If you include a ".", it matches on the
+ fully-qualified name (that is, including the module).
+
+ - "queue": filters by queue name. Supports shell-style glob wildcards.
+
+ - "agent": filters by agent name. Supports shell-style glob wildcards.
+
+ When applied to jobs in the "pending" state, this restricts the jobs to
+ the ones that the agent could perform, according to its filter.
+
+ - "uuid": filters by UUID string, or the special marker "THIS", indicating
+ the UUID of the current process' dispatcher. Supports shell-style glob
+ wildcards.
+
+ When applied to jobs in the "pending" state, this restricts the jobs to
+ the ones that the agents for that dispatcher could perform, according to
+ their filters.
+
+ - "requested_start": a duration-based filter for when the job was requested
+ to start.
+
+ Note that, if a job is not given an explicit start time, the time when it
+ was added to a queue is used. This is based on a job's ``begin_after``
+ attribute.
+
+ - "start": a duration-based filter for when the job started work.
+
+ Note that, if a job is restarted because of problems such as an
+ interruption or a conflict error, this is the most recent time that the
+ job started work. This is based on a job's ``active_start`` attribute.
+
+ To see the first time a job started work ever, the default retry policies
+ store a 'first_active' value in their ``data`` attribute
+ (``job.getRetryPolicy().data.get('first_active')``). Other information
+ about retries can also be found in this data dictionary.
+
+ - "end": a duration-based filter for when the job ended work (but not
+ callbacks).
+
+ This is based on a job's ``active_end`` attribute.
+
+ - "callbacks_completed": a duration-based filter for when the job
+ finished the callbacks it had just after it performed the job.
+
+ If subsequent callbacks are added, they are performed immediately, and
+ will not affect the value that this filter uses.
+
+ This is based on a job's ``initial_callbacks_end`` attribute.
+
+ Usage Examples
+ ==============
+
+ Here are some examples of the command.
+
+ asyncdb job pending
+ (lists the job identifiers for pending jobs)
+
+ asyndb job active agent:instance5
+ (lists the job identifiers of all jobs that any agent named instance5
+ is working on)
+
+ asyndb job pending agent:instance5
+ (lists the job identifiers of all pending jobs that agents named
+ "instance5" could perform)
+
+ asyncdb job completed end:since1H callable:import_*
+ (lists the job identifiers of completed jobs that ended within the
+ last hour that called a function or method that began with the string
+ "import_")
+
+ Here are some examples of how the duration-based filters work.
+
+ * If you used "start:since5s" then that could be read as "jobs that
+ started five seconds ago or sooner."
+
+ * "requested_start:before1M" could be read as "jobs that were supposed to
+ begin one minute ago or longer".
+
+ * "end:since1M,before30S" could be read as "jobs that ended their
+ primary work (that is, not including callbacks) between thirty seconds
+ and one minute ago."
+
+ * "callbacks_completed:before30S,since1M" could be read as "jobs that
+ completed the callbacks they had when first run between thirty seconds
+ and one minute ago." (This also shows that the order of "before" and
+ "since" do not matter.)
+ """
return _jobs(context, states, **kwargs)
def count(context, *states, **kwargs):
- """Count jobs in one or more states."""
+ """Count jobs in one or more states.
+
+ The asyncdb commands "jobs," "count," "jobstats," and "firstjob" all share
+ the same arguments, which are described below; after which usage examples
+ for this command are listed.
+
+ Arguments
+ =========
+
+ States
+ ------
+
+ You must provide at least one of the following states.
+
+ - "pending": the job is in a queue, waiting to be started.
+
+ - "assigned": a dispatcher has claimed the job and assigned it to one of
+ its worker threads. Work has not yet begun. Jobs are in this state very
+ briefly.
+
+ - "active": A worker thread is performing this job.
+
+ - "callbacks": the job's work is ended, and the thread is performing the
+ callbacks, if any.
+
+ - "completed": the job and its callbacks are completed. Completed jobs
+ stay in the database for only a certain amount of time--between seven and
+ eight days in the default agent implementation.
+
+ - "succeeded": the job completed successfully (that is, without raising an
+ unhandled exception, and without returning an explicit
+ twisted.python.failure.Failure). This is a subset of "completed,"
+ described above.
+
+ - "failed": the job completed by raising an unhandled exception or by
+ explicitly returning a twisted.python.failure.Failure. This is a subset
+ of "completed," described above.
+
+ You may use no more than one of the states "completed," "succeeded," and
+ "failed".
+
+ Optional Arguments
+ ------------------
+
+ You can further filter your results with a number of optional arguments.
+
+ "Shell-style glob wildcards," as referred to in this list, are "?", "*",
+ "[SEQUENCE]", and "[!SEQUENCE]", as described in
+ http://docs.python.org/lib/module-fnmatch.html .
+
+ A "duration-based filter" described in this list accepts an argument that
+ is of the form "sinceINTERVAL", "beforeINTERVAL", or
+ "sinceINTERVAL,beforeINTERVAL" (no space!). The "INTERVAL"s are of the
+ form ``[nD][nH][nM][nS]``, where "n" should be replaced with a positive
+ integer, and "D," "H," "M," and "S" are literals standing for "days,"
+ "hours," "minutes," and "seconds." For instance, you might use ``5M`` for
+ five minutes, ``20S`` for twenty seconds, or ``1H30M`` for an hour and a
+ half. Thus "before30M" would mean "thirty minutes ago or older."
+ "since45S" would mean "45 seconds ago or newer." "since1H,before30M" would
+ mean "between thirty minutes and an hour ago." Note that reversing the two
+ components in the last example--"before30M,since1H"--is equivalent.
+
+
+ - "callable": filters by callable name. Supports shell-style glob
+ wildcards. If you do not include a "." in the string, it matches only on
+ the callable name. If you include a ".", it matches on the
+ fully-qualified name (that is, including the module).
+
+ - "queue": filters by queue name. Supports shell-style glob wildcards.
+
+ - "agent": filters by agent name. Supports shell-style glob wildcards.
+
+ When applied to jobs in the "pending" state, this restricts the jobs to
+ the ones that the agent could perform, according to its filter.
+
+ - "uuid": filters by UUID string, or the special marker "THIS", indicating
+ the UUID of the current process' dispatcher. Supports shell-style glob
+ wildcards.
+
+ When applied to jobs in the "pending" state, this restricts the jobs to
+ the ones that the agents for that dispatcher could perform, according to
+ their filters.
+
+ - "requested_start": a duration-based filter for when the job was requested
+ to start.
+
+ Note that, if a job is not given an explicit start time, the time when it
+ was added to a queue is used. This is based on a job's ``begin_after``
+ attribute.
+
+ - "start": a duration-based filter for when the job started work.
+
+ Note that, if a job is restarted because of problems such as an
+ interruption or a conflict error, this is the most recent time that the
+ job started work. This is based on a job's ``active_start`` attribute.
+
+ To see the first time a job started work ever, the default retry policies
+ store a 'first_active' value in their ``data`` attribute
+ (``job.getRetryPolicy().data.get('first_active')``). Other information
+ about retries can also be found in this data dictionary.
+
+ - "end": a duration-based filter for when the job ended work (but not
+ callbacks).
+
+ This is based on a job's ``active_end`` attribute.
+
+ - "callbacks_completed": a duration-based filter for when the job
+ finished the callbacks it had just after it performed the job.
+
+ If subsequent callbacks are added, they are performed immediately, and
+ will not affect the value that this filter uses.
+
+ This is based on a job's ``initial_callbacks_end`` attribute.
+
+ Usage Examples
+ ==============
+
+ Here are some examples of the command.
+
+ asyncdb count pending
+ (counts pending jobs)
+
+ asyndb count active agent:instance5
+ (counts the jobs that all agents named instance5 are working on)
+
+ asyndb count pending agent:instance5
+ (counts the pending jobs that all agents named "instance5" could
+ perform)
+
+ asyncdb count completed end:since1H callable:import_*
+ (counts the completed jobs that ended within the last hour that called
+ a function or method that began with the string "import_")
+
+ Here are some examples of how the duration-based filters work.
+
+ * If you used "start:since5s" then that could be read as "jobs that
+ started five seconds ago or sooner."
+
+ * "requested_start:before1M" could be read as "jobs that were supposed to
+ begin one minute ago or longer".
+
+ * "end:since1M,before30S" could be read as "jobs that ended their
+ primary work (that is, not including callbacks) between thirty seconds
+ and one minute ago."
+
+ * "callbacks_completed:before30S,since1M" could be read as "jobs that
+ completed the callbacks they had when first run between thirty seconds
+ and one minute ago." (This also shows that the order of "before" and
+ "since" do not matter.)
+ """
res = 0
for j in _jobs(context, states, **kwargs):
res += 1
@@ -180,7 +485,162 @@
zc.async.interfaces.COMPLETED: 'completed'}
def jobstats(context, *states, **kwargs):
- """Return statistics about jobs in one or more states."""
+ """Return statistics about jobs in one or more states.
+
+ XXX describe statistics
+
+ Jobs are identified by integer OID and database name. These identifiers
+ can be used with the "asyncdb job" command to get details about the jobs.
+
+ The asyncdb commands "jobs," "count," "jobstats," and "firstjob" all share
+ the same arguments, which are described below; after which usage examples
+ for this command are listed.
+
+ Arguments
+ =========
+
+ States
+ ------
+
+ You must provide at least one of the following states.
+
+ - "pending": the job is in a queue, waiting to be started.
+
+ - "assigned": a dispatcher has claimed the job and assigned it to one of
+ its worker threads. Work has not yet begun. Jobs are in this state very
+ briefly.
+
+ - "active": A worker thread is performing this job.
+
+ - "callbacks": the job's work is ended, and the thread is performing the
+ callbacks, if any.
+
+ - "completed": the job and its callbacks are completed. Completed jobs
+ stay in the database for only a certain amount of time--between seven and
+ eight days in the default agent implementation.
+
+ - "succeeded": the job completed successfully (that is, without raising an
+ unhandled exception, and without returning an explicit
+ twisted.python.failure.Failure). This is a subset of "completed,"
+ described above.
+
+ - "failed": the job completed by raising an unhandled exception or by
+ explicitly returning a twisted.python.failure.Failure. This is a subset
+ of "completed," described above.
+
+ You may use no more than one of the states "completed," "succeeded," and
+ "failed".
+
+ Optional Arguments
+ ------------------
+
+ You can further filter your results with a number of optional arguments.
+
+ "Shell-style glob wildcards," as referred to in this list, are "?", "*",
+ "[SEQUENCE]", and "[!SEQUENCE]", as described in
+ http://docs.python.org/lib/module-fnmatch.html .
+
+ A "duration-based filter" described in this list accepts an argument that
+ is of the form "sinceINTERVAL", "beforeINTERVAL", or
+ "sinceINTERVAL,beforeINTERVAL" (no space!). The "INTERVAL"s are of the
+ form ``[nD][nH][nM][nS]``, where "n" should be replaced with a positive
+ integer, and "D," "H," "M," and "S" are literals standing for "days,"
+ "hours," "minutes," and "seconds." For instance, you might use ``5M`` for
+ five minutes, ``20S`` for twenty seconds, or ``1H30M`` for an hour and a
+ half. Thus "before30M" would mean "thirty minutes ago or older."
+ "since45S" would mean "45 seconds ago or newer." "since1H,before30M" would
+ mean "between thirty minutes and an hour ago." Note that reversing the two
+ components in the last example--"before30M,since1H"--is equivalent.
+
+
+ - "callable": filters by callable name. Supports shell-style glob
+ wildcards. If you do not include a "." in the string, it matches only on
+ the callable name. If you include a ".", it matches on the
+ fully-qualified name (that is, including the module).
+
+ - "queue": filters by queue name. Supports shell-style glob wildcards.
+
+ - "agent": filters by agent name. Supports shell-style glob wildcards.
+
+ When applied to jobs in the "pending" state, this restricts the jobs to
+ the ones that the agent could perform, according to its filter.
+
+ - "uuid": filters by UUID string, or the special marker "THIS", indicating
+ the UUID of the current process' dispatcher. Supports shell-style glob
+ wildcards.
+
+ When applied to jobs in the "pending" state, this restricts the jobs to
+ the ones that the agents for that dispatcher could perform, according to
+ their filters.
+
+ - "requested_start": a duration-based filter for when the job was requested
+ to start.
+
+ Note that, if a job is not given an explicit start time, the time when it
+ was added to a queue is used. This is based on a job's ``begin_after``
+ attribute.
+
+ - "start": a duration-based filter for when the job started work.
+
+ Note that, if a job is restarted because of problems such as an
+ interruption or a conflict error, this is the most recent time that the
+ job started work. This is based on a job's ``active_start`` attribute.
+
+ To see the first time a job started work ever, the default retry policies
+ store a 'first_active' value in their ``data`` attribute
+ (``job.getRetryPolicy().data.get('first_active')``). Other information
+ about retries can also be found in this data dictionary.
+
+ - "end": a duration-based filter for when the job ended work (but not
+ callbacks).
+
+ This is based on a job's ``active_end`` attribute.
+
+ - "callbacks_completed": a duration-based filter for when the job
+ finished the callbacks it had just after it performed the job.
+
+ If subsequent callbacks are added, they are performed immediately, and
+ will not affect the value that this filter uses.
+
+ This is based on a job's ``initial_callbacks_end`` attribute.
+
+ Usage Examples
+ ==============
+
+ Here are some examples of the command.
+
+ asyncdb jobstats pending
+ (gives statistics about the pending jobs)
+
+ asyndb jobstats active agent:instance5
+ (gives statistics about all jobs that any agent named instance5 is
+ working on)
+
+ asyndb job pending agent:instance5
+ (gives statistics aboutt all pending jobs that agents named "instance5"
+ could perform)
+
+ asyncdb job completed end:since1H callable:import_*
+ (gves statistics about completed jobs that ended within the last hour
+ that called a function or method that began with the string "import_")
+
+ Here are some examples of how the duration-based filters work.
+
+ * If you used "start:since5s" then that could be read as "jobs that
+ started five seconds ago or sooner."
+
+ * "requested_start:before1M" could be read as "jobs that were supposed to
+ begin one minute ago or longer".
+
+ * "end:since1M,before30S" could be read as "jobs that ended their
+ primary work (that is, not including callbacks) between thirty seconds
+ and one minute ago."
+
+ * "callbacks_completed:before30S,since1M" could be read as "jobs that
+ completed the callbacks they had when first run between thirty seconds
+ and one minute ago." (This also shows that the order of "before" and
+ "since" do not matter.)
+ """
now = datetime.datetime.now(pytz.UTC)
d = {'pending': 0, 'assigned': 0, 'active': 0, 'callbacks': 0,
'succeeded': 0, 'failed': 0}
@@ -292,6 +752,158 @@
def firstjob(context, *states, **kwargs):
"""Return summary of first job found matching given filters.
+
+ XXX describe what "first" means for different states.
+
+ The asyncdb commands "jobs," "count," "jobstats," and "firstjob" all share
+ the same arguments, which are described below; after which usage examples
+ for this command are listed.
+
+ Arguments
+ =========
+
+ States
+ ------
+
+ You must provide at least one of the following states.
+
+ - "pending": the job is in a queue, waiting to be started.
+
+ - "assigned": a dispatcher has claimed the job and assigned it to one of
+ its worker threads. Work has not yet begun. Jobs are in this state very
+ briefly.
+
+ - "active": A worker thread is performing this job.
+
+ - "callbacks": the job's work is ended, and the thread is performing the
+ callbacks, if any.
+
+ - "completed": the job and its callbacks are completed. Completed jobs
+ stay in the database for only a certain amount of time--between seven and
+ eight days in the default agent implementation.
+
+ - "succeeded": the job completed successfully (that is, without raising an
+ unhandled exception, and without returning an explicit
+ twisted.python.failure.Failure). This is a subset of "completed,"
+ described above.
+
+ - "failed": the job completed by raising an unhandled exception or by
+ explicitly returning a twisted.python.failure.Failure. This is a subset
+ of "completed," described above.
+
+ You may use no more than one of the states "completed," "succeeded," and
+ "failed".
+
+ Optional Arguments
+ ------------------
+
+ You can further filter your results with a number of optional arguments.
+
+ "Shell-style glob wildcards," as referred to in this list, are "?", "*",
+ "[SEQUENCE]", and "[!SEQUENCE]", as described in
+ http://docs.python.org/lib/module-fnmatch.html .
+
+ A "duration-based filter" described in this list accepts an argument that
+ is of the form "sinceINTERVAL", "beforeINTERVAL", or
+ "sinceINTERVAL,beforeINTERVAL" (no space!). The "INTERVAL"s are of the
+ form ``[nD][nH][nM][nS]``, where "n" should be replaced with a positive
+ integer, and "D," "H," "M," and "S" are literals standing for "days,"
+ "hours," "minutes," and "seconds." For instance, you might use ``5M`` for
+ five minutes, ``20S`` for twenty seconds, or ``1H30M`` for an hour and a
+ half. Thus "before30M" would mean "thirty minutes ago or older."
+ "since45S" would mean "45 seconds ago or newer." "since1H,before30M" would
+ mean "between thirty minutes and an hour ago." Note that reversing the two
+ components in the last example--"before30M,since1H"--is equivalent.
+
+
+ - "callable": filters by callable name. Supports shell-style glob
+ wildcards. If you do not include a "." in the string, it matches only on
+ the callable name. If you include a ".", it matches on the
+ fully-qualified name (that is, including the module).
+
+ - "queue": filters by queue name. Supports shell-style glob wildcards.
+
+ - "agent": filters by agent name. Supports shell-style glob wildcards.
+
+ When applied to jobs in the "pending" state, this restricts the jobs to
+ the ones that the agent could perform, according to its filter.
+
+ - "uuid": filters by UUID string, or the special marker "THIS", indicating
+ the UUID of the current process' dispatcher. Supports shell-style glob
+ wildcards.
+
+ When applied to jobs in the "pending" state, this restricts the jobs to
+ the ones that the agents for that dispatcher could perform, according to
+ their filters.
+
+ - "requested_start": a duration-based filter for when the job was requested
+ to start.
+
+ Note that, if a job is not given an explicit start time, the time when it
+ was added to a queue is used. This is based on a job's ``begin_after``
+ attribute.
+
+ - "start": a duration-based filter for when the job started work.
+
+ Note that, if a job is restarted because of problems such as an
+ interruption or a conflict error, this is the most recent time that the
+ job started work. This is based on a job's ``active_start`` attribute.
+
+ To see the first time a job started work ever, the default retry policies
+ store a 'first_active' value in their ``data`` attribute
+ (``job.getRetryPolicy().data.get('first_active')``). Other information
+ about retries can also be found in this data dictionary.
+
+ - "end": a duration-based filter for when the job ended work (but not
+ callbacks).
+
+ This is based on a job's ``active_end`` attribute.
+
+ - "callbacks_completed": a duration-based filter for when the job
+ finished the callbacks it had just after it performed the job.
+
+ If subsequent callbacks are added, they are performed immediately, and
+ will not affect the value that this filter uses.
+
+ This is based on a job's ``initial_callbacks_end`` attribute.
+
+ Usage Examples
+ ==============
+
+ Here are some examples of the command.
+
+ asyncdb firstjob pending
+ (describes the first (next-to-be-processed) pending jobs)
+
+ asyndb job active agent:instance5
+ (describes the first job that any agent named instance5
+ is working on; "first" doesn't mean much here.)
+
+ asyndb job pending agent:instance5
+ (describes the first (next-to-be-processed) pending jobs that agents
+ named "instance5" could perform)
+
+ asyncdb job completed end:since1H callable:import_*
+ (describes the first (most recently completed) completed job that ended
+ within the last hour that called a function or method that began with
+ the string "import_")
+
+ Here are some examples of how the duration-based filters work.
+
+ * If you used "start:since5s" then that could be read as "jobs that
+ started five seconds ago or sooner."
+
+ * "requested_start:before1M" could be read as "jobs that were supposed to
+ begin one minute ago or longer".
+
+ * "end:since1M,before30S" could be read as "jobs that ended their
+ primary work (that is, not including callbacks) between thirty seconds
+ and one minute ago."
+
+ * "callbacks_completed:before30S,since1M" could be read as "jobs that
+ completed the callbacks they had when first run between thirty seconds
+ and one minute ago." (This also shows that the order of "before" and
+ "since" do not matter.)
"""
for j in _jobs(context, states, **kwargs):
return jobsummary(j)
More information about the Checkins
mailing list