[Checkins] SVN: zc.icp/trunk/ initial import of zc.icp
Benji York
benji at zope.com
Thu Feb 7 11:24:55 EST 2008
Log message for revision 83613:
initial import of zc.icp
Changed:
_U zc.icp/trunk/
A zc.icp/trunk/bootstrap.py
A zc.icp/trunk/buildout.cfg
A zc.icp/trunk/setup.py
A zc.icp/trunk/src/
A zc.icp/trunk/src/zc/
A zc.icp/trunk/src/zc/__init__.py
A zc.icp/trunk/src/zc/icp/
A zc.icp/trunk/src/zc/icp/README.txt
A zc.icp/trunk/src/zc/icp/__init__.py
A zc.icp/trunk/src/zc/icp/datagram.txt
A zc.icp/trunk/src/zc/icp/interfaces.py
A zc.icp/trunk/src/zc/icp/tests.py
-=-
Property changes on: zc.icp/trunk
___________________________________________________________________
Name: svn:ignore
+ develop-eggs
bin
parts
.installed.cfg
Name: svn:externals
+ bootstrap svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap
Added: zc.icp/trunk/bootstrap.py
===================================================================
--- zc.icp/trunk/bootstrap.py (rev 0)
+++ zc.icp/trunk/bootstrap.py 2008-02-07 16:24:55 UTC (rev 83613)
@@ -0,0 +1 @@
+link bootstrap/bootstrap.py
\ No newline at end of file
Property changes on: zc.icp/trunk/bootstrap.py
___________________________________________________________________
Name: svn:special
+ *
Added: zc.icp/trunk/buildout.cfg
===================================================================
--- zc.icp/trunk/buildout.cfg (rev 0)
+++ zc.icp/trunk/buildout.cfg 2008-02-07 16:24:55 UTC (rev 83613)
@@ -0,0 +1,17 @@
+[buildout]
+develop = .
+parts = test interpreter
+find-links =
+ http://download.zope.org/ppix
+
+extensions = zc.buildoutsftp
+
+[test]
+recipe = zc.recipe.testrunner
+eggs = zc.icp
+defaults = '--tests-pattern [fn]?tests --exit-with-status -1'.split()
+
+[interpreter]
+recipe = zc.recipe.egg
+eggs = zc.icp
+interpreter = py
Added: zc.icp/trunk/setup.py
===================================================================
--- zc.icp/trunk/setup.py (rev 0)
+++ zc.icp/trunk/setup.py 2008-02-07 16:24:55 UTC (rev 83613)
@@ -0,0 +1,24 @@
+import os
+
+from setuptools import setup, find_packages
+
+setup(
+ name='zc.icp',
+ version='1.0dev',
+ packages=find_packages('src', exclude=['*.tests', '*.ftests']),
+ package_dir={'':'src'},
+
+ url='svn+ssh://svn.zope.com/repos/main/zc.icp',
+ zip_safe=False,
+ author='Zope Corporation',
+ author_email='benji at zope.com',
+ description='Pluggable ICP server',
+ license='ZPL',
+ install_requires=[
+ 'setuptools',
+ 'zope.component',
+ 'zope.interface',
+ 'zope.testing',
+ ],
+ include_package_data=True,
+ )
Property changes on: zc.icp/trunk/setup.py
___________________________________________________________________
Name: svn:keywords
+ Id
Name: svn:eol-style
+ native
Added: zc.icp/trunk/src/zc/__init__.py
===================================================================
--- zc.icp/trunk/src/zc/__init__.py (rev 0)
+++ zc.icp/trunk/src/zc/__init__.py 2008-02-07 16:24:55 UTC (rev 83613)
@@ -0,0 +1,7 @@
+# this is a namespace package
+try:
+ import pkg_resources
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+ __path__ = pkgutil.extend_path(__path__, __name__)
Property changes on: zc.icp/trunk/src/zc/__init__.py
___________________________________________________________________
Name: svn:keywords
+ Id
Name: svn:eol-style
+ native
Added: zc.icp/trunk/src/zc/icp/README.txt
===================================================================
--- zc.icp/trunk/src/zc/icp/README.txt (rev 0)
+++ zc.icp/trunk/src/zc/icp/README.txt 2008-02-07 16:24:55 UTC (rev 83613)
@@ -0,0 +1,275 @@
+=============================
+Internet Cache Protocol (ICP)
+=============================
+
+In many cases some set of web servers will be able to more quickly service an
+HTTP request than others. HTTP accelorators (reverse proxies) can use ICP_
+queries to find the most appropriate server to handle a request. One of the
+most popular reverse proxies (Squid_) includes support for making ICP requests
+to origin servers.
+
+.. [ICP] http://www.faqs.org/rfcs/rfc2186.html
+.. [Squid] http://www.squid-cache.org/
+
+
+When to use ICP
+===============
+
+When generating content dynamically, having all the data available locally to
+fulfil a request can have a profound effect on service time. One approach to
+having the data available is to have one or more caches. In some situations
+those caches are not large enough to contain the entire working set required
+for efficient servicing of incoming requests. Adding additional request
+handlers (servers or processes) doesn't help because the time to load the data
+from one or more storage servers (e.g., databases) is the dominant factor in
+request time. In those situations the request space can be partitioned such
+that the portion of the working set a particular handler (server or process) is
+responsible for can fit in its cache(s).
+
+Statically configuring that request space partitioning may be difficult,
+error-prone, or even impossible. In those circumstances it would be nice to
+let the origin servers provide feedback on whether or not they should handle a
+particular request. That's where ICP comes in.
+
+
+Hits and Misses
+===============
+
+When an ICP query request is recieved, the server can return one of ICP_OP_HIT,
+ICP_OP_MISS, ICP_OP_ERR, ICP_OP_MISS_NOFETCH, or ICP_OP_DENIED. The meanings
+of these result codes are defined in the ICP RFC as below.
+
+ICP_OP_HIT
+ An ICP_OP_HIT response indicates that the requested URL exists in
+ this cache and that the requester is allowed to retrieve it.
+
+ICP_OP_MISS
+ An ICP_OP_MISS response indicates that the requested URL does not
+ exist in this cache. The querying cache may still choose to fetch
+ the URL from the replying cache.
+
+ICP_OP_ERR
+ An ICP_OP_ERR response indicates some kind of error in parsing or
+ handling the query message (e.g. invalid URL).
+
+ICP_OP_MISS_NOFETCH
+ An ICP_OP_MISS_NOFETCH response indicates that this cache is up,
+ but is in a state where it does not want to handle cache misses.
+ An example of such a state is during a startup phase where a cache
+ might be rebuilding its object store. A cache in such a mode may
+ wish to return ICP_OP_HIT for cache hits, but not ICP_OP_MISS for
+ misses. ICP_OP_MISS_NOFETCH essentially means "I am up and
+ running, but please don't fetch this URL from me now."
+
+ Note, ICP_OP_MISS_NOFETCH has a different meaning than
+ ICP_OP_MISS. The ICP_OP_MISS reply is an invitation to fetch the
+ URL from the replying cache (if their relationship allows it), but
+ ICP_OP_MISS_NOFETCH is a request to NOT fetch the URL from the
+ replying cache.
+
+ICP_OP_DENIED
+ An ICP_OP_DENIED response indicates that the querying site is not
+ allowed to retrieve the named object from this cache. Caches and
+ proxies may implement complex access controls. This reply must be
+ be interpreted to mean "you are not allowed to request this
+ particular URL from me at this particular time."
+
+Because we want to use ICP to communicate about whether or not an origin server
+(as opposed to a cache server) wants to handle a particular request, we will
+use slightly different definitions for some of the result codes.
+
+ICP_OP_HIT
+ An ICP_OP_HIT response indicates that the queried server would prefer to
+ handle the HTTP request. The reason the origin server is returning a hit
+ might be that it has recently handled "similar" requests, or that it has
+ been configured to handle the partition of the URL space occupied by the
+ given URL.
+
+ICP_OP_MISS
+ An ICP_OP_MISS response indicates that the queried server does not have a
+ preference to service the request, but will be able to handle the request
+ nonetheless. This is normally the default response.
+
+ICP_OP_MISS_NOFETCH
+ An ICP_OP_MISS_NOFETCH response indicates that the requesting server may
+ not request the named object from this server. This may be because the
+ origin server is under heavy load at the time or some other policy
+ indicates that the request must not be forwarded at the moment.
+
+The response (hit, miss, etc.) to a particular ICP query is based on one or
+more configured policies. The mechanics of defining and registering those
+plolicies is explained in the next section.
+
+This package does not implement the deprecated ICP_OP_HIT_OBJ.
+
+
+Defining Policies
+=================
+
+To use this package one or more polices must be defined and registered. The
+Zope component architechure is used to manage the polices as "utilities".
+
+Policies must implement the IICPPolicy interface.
+
+ >>> from zc.icp.interfaces import IICPPolicy
+ >>> IICPPolicy
+ <InterfaceClass zc.icp.interfaces.IICPPolicy>
+
+At this point no policy is registered, so any URL will generate a miss.
+
+ >>> import zc.icp
+ >>> zc.icp.check_url('http://example.com/foo')
+ 'ICP_OP_MISS'
+
+Let's say we want to return an ICP_OP_HIT for all URLs containing "foo", we
+can define that policy like so:
+
+ >>> def foo_hit_policy(url):
+ ... if 'foo' in url:
+ ... return 'ICP_OP_HIT'
+
+Once we register this policy. We have to provide a name for this registration.
+Any subsequent registration with the same name will override it. The default
+name is the empty string.
+
+ >>> import zope.component
+ >>> zope.component.provideUtility(foo_hit_policy, IICPPolicy, 'foo')
+
+The registered policy is immediately available.
+
+ >>> zc.icp.check_url('http://example.com/foo')
+ 'ICP_OP_HIT'
+
+Non-foo URLs are still misses.
+
+ >>> zc.icp.check_url('http://example.com/bar')
+ 'ICP_OP_MISS'
+
+Now we can add another policy to indicate that we don't want any requests with
+"baz" in them.
+
+ >>> def baz_hit_policy(url):
+ ... if 'baz' in url:
+ ... return 'ICP_OP_MISS_NOFETCH'
+ >>> zope.component.provideUtility(baz_hit_policy, IICPPolicy, 'baz')
+
+ >>> zc.icp.check_url('http://example.com/foo')
+ 'ICP_OP_HIT'
+ >>> zc.icp.check_url('http://example.com/bar')
+ 'ICP_OP_MISS'
+ >>> zc.icp.check_url('http://example.com/baz')
+ 'ICP_OP_MISS_NOFETCH'
+
+The policies are prioritized by name. The first policy to return a non-None
+result is followed. Therefore if we check a URL with both "foo" and "baz" in
+it, the policy for "baz" is followed.
+
+ >>> zc.icp.check_url('http://example.com/foo/baz')
+ 'ICP_OP_MISS_NOFETCH'
+
+
+Running the Server
+==================
+
+Starting the server begins listening on the given port and IP.
+
+ >>> zc.icp.start_server('localhost', 3130)
+ info: ICP server started
+ Address: localhost
+ Port: 3130
+
+Now we can start sending ICP requests and get responses back. To do so we must
+first construct an ICP request.
+
+ >>> import struct
+ >>> query = zc.icp.HEADER_LAYOUT + zc.icp.QUERY_LAYOUT
+ >>> url = 'http://example.com/\0'
+ >>> format = query % len(url)
+ >>> icp_request = struct.pack(
+ ... format, 1, 2, struct.calcsize(format), 0xDEADBEEF, 0, 0, 0, 0, url)
+ >>> print zc.icp.format_datagram(icp_request)
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | ICP_OP_QUERY | Version: 2 | Message Length: 44 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Request Number: DEADBEEF |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Options: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Option Data: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Sender Host Address: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | |
+ | Payload: \x00\x00\x00\x00http://example.com/\x00 |
+ | |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+After sending the request we get back a response.
+
+ >>> import socket
+ >>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ >>> s.connect(('localhost', 3130))
+
+ >>> s.send(icp_request)
+ 44
+ >>> icp_response = s.recv(16384)
+ >>> icp_response
+ '\x03\x02\x00(\xde\xad\xbe\xef\x00\x00\x00\x00\...http://example.com/\x00'
+ >>> print zc.icp.format_datagram(icp_response)
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | ICP_OP_MISS | Version: 2 | Message Length: 40 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Request Number: DEADBEEF |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Options: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Option Data: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Sender Host Address: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | |
+ | Payload: http://example.com/\x00 |
+ | |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+That was a miss. We can also provoke a hit by satisfying one of our policies.
+
+ >>> url = 'http://example.com/foo\0'
+ >>> format = query % len(url)
+ >>> icp_request = struct.pack(
+ ... format, 1, 2, struct.calcsize(format), 0xDEADBEEF, 0, 0, 0, 0, url)
+ >>> print zc.icp.format_datagram(icp_request)
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | ICP_OP_QUERY | Version: 2 | Message Length: 47 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Request Number: DEADBEEF |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Options: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Option Data: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Sender Host Address: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | |
+ | Payload: \x00\x00\x00\x00http://example.com/foo\x00 |
+ | |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+ >>> s.send(icp_request)
+ 47
+ >>> print zc.icp.format_datagram(s.recv(16384))
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | ICP_OP_HIT | Version: 2 | Message Length: 43 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Request Number: DEADBEEF |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Options: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Option Data: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Sender Host Address: 0 |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | |
+ | Payload: http://example.com/foo\x00 |
+ | |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Property changes on: zc.icp/trunk/src/zc/icp/README.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: zc.icp/trunk/src/zc/icp/__init__.py
===================================================================
--- zc.icp/trunk/src/zc/icp/__init__.py (rev 0)
+++ zc.icp/trunk/src/zc/icp/__init__.py 2008-02-07 16:24:55 UTC (rev 83613)
@@ -0,0 +1,197 @@
+##############################################################################
+#
+# Copyright (c) 2008 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+import logging
+import asyncore
+import errno
+import os
+import socket
+import string
+import struct
+import sys
+import threading
+import zc.icp.interfaces
+import zope.component
+
+ICP_OP_INVALID = 0
+ICP_OP_QUERY = 1
+ICP_OP_HIT = 2
+ICP_OP_MISS = 3
+ICP_OP_ERR = 4
+ICP_OP_SECHO = 10
+ICP_OP_DECHO = 11
+ICP_OP_MISS_NOFETCH = 21
+ICP_OP_DENIED = 22
+
+HEADER_LAYOUT = '!BBHIIII'
+RESPONSE_LAYOUT = '%ds'
+QUERY_LAYOUT = 'I' + RESPONSE_LAYOUT
+MAX_DATAGRAM_LOG_LENGTH = 70
+
+def check_url(url):
+ policies = zope.component.getUtilitiesFor(zc.icp.interfaces.IICPPolicy)
+ for name, policy in sorted(policies):
+ result = policy(url)
+ if result is not None:
+ return result
+
+ return 'ICP_OP_MISS'
+
+
+def print_datagram(datagram):
+ try:
+ return format_datagram(datagram)
+ except:
+ return repr(datagram)
+
+
+def handle_request(datagram, check_url=check_url):
+ log_message = None
+ out_header = HEADER_LAYOUT + RESPONSE_LAYOUT % 1
+ out_group = [ICP_OP_ERR, 2, len(datagram), 0, 0, 0, 0, '\0']
+ try:
+ in_group = list(struct.unpack(HEADER_LAYOUT, datagram[:20]))
+ opcode, version, length, number, options, opdata, sender_ip = in_group
+ except struct.error:
+ log_message = 'Error unpacking ICP header'
+ else:
+ out_group[2:4] = [struct.calcsize(out_header), number]
+ if version == 2 and length == len(datagram) and length <= 16384:
+ if opcode == ICP_OP_QUERY:
+ if length > 24:
+ try:
+ (requester_ip, url) = struct.unpack(
+ '!' + QUERY_LAYOUT % (length - 24), datagram[20:])
+ except:
+ log_message = 'Error unpacking ICP query'
+ else:
+ in_group.extend([requester_ip, url])
+ out_header = HEADER_LAYOUT + RESPONSE_LAYOUT % len(url)
+ out_group[2] = struct.calcsize(out_header)
+ out_group[6:] = [sender_ip, url]
+ if url[-1:] == '\x00':
+ out_group[0] = globals()[check_url(url[:-1])]
+ else:
+ log_message = (
+ 'URL in ICP query is not null-terminated')
+ else:
+ log_message = 'Query is not long enough'
+
+ if log_message:
+ if len(datagram) > MAX_DATAGRAM_LOG_LENGTH:
+ chunk_size = MAX_DATAGRAM_LOG_LENGTH / 2
+ log_gram = datagram[:chunk_size] + '...' + datagram[-chunk_size:]
+ else:
+ log_gram = datagram
+
+ logging.error('%s:\n %r' % (log_message, log_gram))
+
+ result = struct.pack(out_header, *out_group)
+ return result
+
+# We want our own, independent map for running an asyncore mainloop.
+asyncore_map = {}
+
+class ICPServer(asyncore.dispatcher):
+
+ REQUESTS_PER_LOOP = 4
+
+ def __init__(self, ip, port):
+ asyncore.dispatcher.__init__(self, map=asyncore_map)
+ self.create_socket(socket.AF_INET, socket.SOCK_DGRAM)
+ self.set_reuse_addr()
+ self.bind((ip, port))
+ if ip=='':
+ addr = 'any'
+ else:
+ addr = ip
+
+ self.log_info(
+ 'ICP server started\n\tAddress: %s\n\tPort: %s' % (addr, port))
+
+ def handle_read(self):
+ for i in range(self.REQUESTS_PER_LOOP):
+ try:
+ datagram, whence = self.socket.recvfrom(16384)
+ except socket.error, e:
+ if e[0] == errno.EWOULDBLOCK:
+ break
+ else:
+ raise
+ else:
+ reply = handle_request(datagram)
+ if reply:
+ self.socket.sendto(reply, whence)
+
+ def readable(self):
+ return 1
+
+ def writable(self):
+ return 0
+
+ def handle_connect(self):
+ pass
+
+ def handle_write(self):
+ self.log_info('unexpected write event', 'warning')
+
+ def handle_error(self):
+ # don't close the socket on error
+ (file, fun, line), t, v, tbinfo = asyncore.compact_traceback()
+ self.log_info('Problem in ICP (%s:%s %s)' % (t, v, tbinfo), 'error')
+
+
+def start_server(ip='', port=3130):
+ server = ICPServer(ip, port)
+ thread = threading.Thread(target=asyncore.loop,
+ kwargs=dict(map=asyncore_map))
+ thread.setDaemon(True)
+ thread.start()
+
+
+template = """\
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+| %-13s| Version: %-3s| Message Length: %-10s|
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+| Request Number: %-27X|
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+| Options: %-34X|
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+| Option Data: %-30X|
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+| Sender Host Address: %-22s|
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+| |
+| Payload: %-50s|
+| |
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"""
+
+reverse_opcode_map = {
+ 0: 'ICP_OP_INVALID',
+ 1: 'ICP_OP_QUERY',
+ 2: 'ICP_OP_HIT',
+ 3: 'ICP_OP_MISS',
+ 4: 'ICP_OP_ERR',
+ 10: 'ICP_OP_SECHO',
+ 11: 'ICP_OP_DECHO',
+ 21: 'ICP_OP_MISS_NOFETCH',
+ 22: 'ICP_OP_DENIED',
+ }
+
+def format_datagram(datagram):
+ header_size = struct.calcsize(HEADER_LAYOUT)
+ parts = list(struct.unpack(HEADER_LAYOUT, datagram[:header_size]))
+ parts[0] = reverse_opcode_map[parts[0]]
+ payload = repr(datagram[header_size:])[1:-1]
+ parts.append(payload)
+ return template % tuple(parts)
Property changes on: zc.icp/trunk/src/zc/icp/__init__.py
___________________________________________________________________
Name: svn:keywords
+ Id
Name: svn:eol-style
+ native
Added: zc.icp/trunk/src/zc/icp/datagram.txt
===================================================================
--- zc.icp/trunk/src/zc/icp/datagram.txt (rev 0)
+++ zc.icp/trunk/src/zc/icp/datagram.txt 2008-02-07 16:24:55 UTC (rev 83613)
@@ -0,0 +1,109 @@
+ICP datagrams look like this::
+
+ 0 1 2 3
+ 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Opcode | Version | Message Length |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Request Number |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Options |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Option Data |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Sender Host Address |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | |
+ | Payload |
+ / /
+ / /
+ | |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+The handle_request function tests datagrams for correctness, uses the check_url
+function it is passed to determine what response to send, and packs up the
+response.
+
+This is the format for the payload of a query::
+
+ 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | Requester Host Address |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | |
+ / Null-Terminated URL /
+ / /
+ | |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+We'll want to set up some stuff for the testig later.
+
+ >>> import struct
+ >>> import pprint
+ >>> import zc.icp
+ >>> query = zc.icp.HEADER_LAYOUT + zc.icp.QUERY_LAYOUT
+ >>> response = zc.icp.HEADER_LAYOUT + zc.icp.RESPONSE_LAYOUT
+ >>> def checker(url):
+ ... return 'ICP_OP_HIT'
+ >>> import logging
+ >>> import sys
+ >>> stdout_log = logging.StreamHandler(sys.stdout)
+ >>> stdout_log.setFormatter(logging.Formatter("LOGGER: %(message)s"))
+ >>> logging.getLogger('').addHandler(stdout_log)
+
+If the query is too short to have a proper header, a generic error will be
+returned. the length will have the length of the sent datagram.
+
+ >>> truncated_header = '!BBH'
+ >>> truncated = struct.pack(
+ ... truncated_header, 1, 2, struct.calcsize(truncated_header))
+ >>> struct.unpack(
+ ... response % 1,
+ ... zc.icp.handle_request(truncated, checker))
+ LOGGER: Error unpacking ICP header:
+ '\x01\x02\x00\x04'
+ (4, 2, 4, 0L, 0L, 0L, 0L, '\x00')
+
+The header is 20 bytes long, and the payload has an 4 byte leader for the
+requester's ip and is null terminated, so a query datagram that is less than 21
+bytes long is malformed.
+
+ >>> no_payload = zc.icp.HEADER_LAYOUT
+ >>> header = struct.pack(
+ ... no_payload, 1, 2, struct.calcsize(no_payload), 0xDEADBEEF, 0, 0, 0)
+ >>> struct.unpack(
+ ... response % 1,
+ ... zc.icp.handle_request(header, checker))
+ LOGGER: Query is not long enough:
+ '\x01\x02\x00\x14\xde\xad\xbe\xef...\x00\x00\x00'
+ (4, 2, 21, 3735928559L, 0L, 0L, 0L, '\x00')
+
+If the query url is not NULL terminated, this is an error.
+
+ >>> url = 'http://metrowestdailynews.com'
+ >>> format = query % len(url)
+ >>> datagram = struct.pack(
+ ... format,
+ ... 1, 2, struct.calcsize(format), 0xDEADBEEF, 0, 0, 0, 0, url)
+ >>> format = response % len(url)
+ >>> struct.unpack(
+ ... format,
+ ... zc.icp.handle_request(datagram, checker))
+ LOGGER: URL in ICP query is not null-terminated:
+ '\x01\x02\x005\xde\xad\xbe\xef\x00\x00...http://metrowestdailynews.com'
+ (4, 2, 49, 3735928559L, 0L, 0L, 0L, 'http://metrowestdailynews.com')
+
+If the query is valid, then the determination of whether the request is a
+hit or not is delegated to the passed function. This function should accept
+one argument, the url we are looking for.
+
+ >>> url = 'http://metrowestdailynews.com\0'
+ >>> format = query % len(url)
+ >>> datagram = struct.pack(
+ ... format,
+ ... 1, 2, struct.calcsize(format), 0xDEADBEEF, 0, 0, 0, 0, url)
+ >>> format = response % len(url)
+ >>> struct.unpack(
+ ... format,
+ ... zc.icp.handle_request(datagram, checker))
+ (2, 2, 50, 3735928559L, 0L, 0L, 0L, 'http://metrowestdailynews.com\x00')
Property changes on: zc.icp/trunk/src/zc/icp/datagram.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: zc.icp/trunk/src/zc/icp/interfaces.py
===================================================================
--- zc.icp/trunk/src/zc/icp/interfaces.py (rev 0)
+++ zc.icp/trunk/src/zc/icp/interfaces.py 2008-02-07 16:24:55 UTC (rev 83613)
@@ -0,0 +1,26 @@
+##############################################################################
+#
+# Copyright (c) 2008 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+__docformat__ = "reStructuredText"
+
+import zope.interface
+
+class IICPPolicy(zope.interface.Interface):
+ """A policy to determine the response to an ICP request."""
+
+ def __call__(url):
+ """Inspect the given URL and return a mneumonic for an ICP opcode.
+
+ The result can be one of the strings: 'ICP_OP_HIT', 'ICP_OP_MISS',
+ 'ICP_OP_ERR', 'ICP_OP_MISS_NOFETCH', and 'ICP_OP_DENIED'.
+ """
Property changes on: zc.icp/trunk/src/zc/icp/interfaces.py
___________________________________________________________________
Name: svn:keywords
+ Id
Name: svn:eol-style
+ native
Added: zc.icp/trunk/src/zc/icp/tests.py
===================================================================
--- zc.icp/trunk/src/zc/icp/tests.py (rev 0)
+++ zc.icp/trunk/src/zc/icp/tests.py 2008-02-07 16:24:55 UTC (rev 83613)
@@ -0,0 +1,24 @@
+##############################################################################
+#
+# Copyright (c) 2008 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+__docformat__ = "reStructuredText"
+
+import unittest
+import zope.testing.doctest
+
+def test_suite():
+ return zope.testing.doctest.DocFileSuite(
+ "README.txt",
+ "datagram.txt",
+ optionflags=zope.testing.doctest.NORMALIZE_WHITESPACE|
+ zope.testing.doctest.ELLIPSIS)
Property changes on: zc.icp/trunk/src/zc/icp/tests.py
___________________________________________________________________
Name: svn:keywords
+ Id
Name: svn:eol-style
+ native
More information about the Checkins
mailing list