[Zope3-dev] ZODB with Blob support
Luis De la Parra
lparrab at gmx.net
Thu Aug 10 15:28:52 EDT 2006
Hi,
sorry for the delay, but I was almost 3 weeks without access to my
computer...
In case someone is interested, this is what I did to get zope.fle to work
with Blobs... like I said: it seems to work ok, but deleting a file
leaves the blob object lying on the filesystem (even after packing the
DB) -- oh, and I still haven't figured out how to make the security
declarations for the file handle, so for the test I just skipped security
altogether:
1) Modify the externals to use the BLOB-branch of ZODB:
2) put zope.file and zope.mimetypes in my instance, and replace the stringio
in zope.file with a Blob
regards. luis
patch 1: Zope3/file_blob_Zope3.patch:
Property changes on: src
___________________________________________________________________
Name: svn:externals
- docutils svn://svn.zope.org/repos/main/docutils/tags/0.4.0
ZConfig svn://svn.zope.org/repos/main/ZConfig/trunk/ZConfig
BTrees -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/BTrees
persistent -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/persistent
ThreadedAsync -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/ThreadedAsync
transaction -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/transaction
ZEO -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/ZEO
ZODB -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/ZODB
twisted -r 15340
svn://svn.twistedmatrix.com/svn/Twisted/branches/releases/2.1.x/twisted
zdaemon -r 40792
svn://svn.zope.org/repos/main/zdaemon/trunk/src/zdaemon
+ docutils svn://svn.zope.org/repos/main/docutils/tags/0.4.0
ZConfig svn://svn.zope.org/repos/main/ZConfig/trunk/ZConfig
BTrees -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/BTrees
persistent -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/persistent
ThreadedAsync -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/ThreadedAsync
transaction -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/transaction
ZEO -r 69196
svn://svn.zope.org/repos/main/ZODB/branches/3.7/src/ZEO
ZODB
svn://svn.zope.org/repos/main/ZODB/tags/blob-technical-preview/src/ZODB
twisted -r 15340
svn://svn.twistedmatrix.com/svn/Twisted/branches/releases/2.1.x/twisted
zdaemon -r 40792
svn://svn.zope.org/repos/main/zdaemon/trunk/src/zdaemon
Index: zope.conf.in
===================================================================
--- zope.conf.in (revision 69370)
+++ zope.conf.in (working copy)
@@ -46,10 +46,20 @@
# </sshserver>
# Standard Filestorage
+#<zodb>
+# <filestorage>
+# path Data.fs
+# </filestorage>
+#</zodb>
+
+# Filestorge with Blob support
<zodb>
- <filestorage>
- path Data.fs
- </filestorage>
+ <blobstorage>
+ blob-dir blobs
+ <filestorage>
+ path Data.fs
+ </filestorage>
+ </blobstorage>
</zodb>
<accesslog>
patch 2 : Zope3/src/zope/file/file_blob.patch
Index: configure.zcml
===================================================================
--- configure.zcml (revision 69216)
+++ configure.zcml (working copy)
@@ -27,6 +27,7 @@
<!-- set up permissions for the accessor objects -->
+<!-- **** FIXME: Do we still need this ?????
<class class=".file.Reader">
<require
permission="zope.View"
@@ -44,7 +45,14 @@
attributes="close flush write"
/>
</class>
-
+ ******* / FIXME -->
+
+ <!-- this is not working.. :-( -->
+ <class class="file">
+ <allow attributes="read" />
+ <require
+ permission="zope.View" attributes="write writelines truncate" />
+ </class>
<!-- Subscriber to update the mimeType field on content-type
changes. -->
<subscriber
Index: download.py
===================================================================
--- download.py (revision 69216)
+++ download.py (working copy)
@@ -23,6 +23,10 @@
import zope.publisher.http
+#FIXME: this is bad.. but I haven't found a way to make security
declarations
on the "file handle" object (inheriting from a normal python file object)..
so for this test we just turn off security all together =(
+from zope.security.proxy import removeSecurityProxy
+
+
class Download(object):
zope.interface.implements(
@@ -87,7 +91,9 @@
# application/database; ZODB BLOBs will provide a equivalent
# feature once available.
#
- data = context.open("rb").read()
+ file = context.open("rb")
+ file = removeSecurityProxy(file) #FIXME
+ data = file.read()
self.headers += ("Content-Length", str(context.size)),
self.body = bodyIterator(cStringIO.StringIO(data))
Index: file.py
===================================================================
--- file.py (revision 69216)
+++ file.py (working copy)
@@ -15,8 +15,13 @@
"""
__docformat__ = "reStructuredText"
-import cStringIO
+#import os
+#import stat
+from ZODB.Blobs.Blob import Blob
+#FIXME: this is bad.. but I haven't found a way to make security
declarations
on the "file handle" object (inheriting from a normal python file object)..
so for this test we just turn off security all together =(
+from zope.security.proxy import removeSecurityProxy #FIXME
+
import persistent
import zope.location.interfaces
@@ -34,10 +39,16 @@
__parent__ = None
mimeType = None
- _data = ""
- size = 0
+ _data = None
def __init__(self, mimeType=None, parameters=None):
+ self._data = Blob()
+ #open the blob for writing to make sure
+ #it is created... if someone tries to
+ #read this file otherwise before writing
+ #to it, he might get into problems... (?)
+ b = self._data.open('w')
+ b.close() # HACK??
self.mimeType = mimeType
if parameters is None:
parameters = {}
@@ -46,115 +57,14 @@
self.parameters = parameters
def open(self, mode="r"):
- if mode in ("r", "rb"):
- return Reader(self, mode)
- if mode in ("w", "wb"):
- return Writer(self, mode)
- raise ValueError("unsupported `mode` value")
+ b = self._data.open(mode)
+ return removeSecurityProxy(b) #FIXME
+
+ def getSize(self):
+ #os.fstat(self._data.open('r').fileno())[stat.ST_SIZE] #FIXME: why is
this not working?
+ f = self._data.open('r')
+ f.seek(0,2)
+ return int(f.tell()) #FIXME
+
+ size = property(getSize)
-
-class Accessor(object):
- """Base class for the reader and writer."""
-
- _closed = False
- _sio = None
-
- # XXX Accessor objects need to have an __parent__ to support the
- # security machinery, but they aren't ILocation instances since
- # they aren't addressable via URL.
- #
- # There needs to be an interface for this in Zope 3, but that's a
- # large task since it affects lots of Z3 code. __parent__ should
- # be defined by an interface from which ILocation is derived.
-
- def __init__(self, file, mode):
- self.__parent__ = file
- self.mode = mode
-
- def close(self):
- if not self._closed:
- self._close()
- self._closed = True
- if "_sio" in self.__dict__:
- del self._sio
-
- def __getstate__(self):
- """Make sure the accessors can't be stored in ZODB."""
- cls = self.__class__
- raise TypeError("%s.%s instance is not picklable"
- % (cls.__module__, cls.__name__))
-
- def _close(self):
- pass
-
-
-class Reader(Accessor):
-
- zope.interface.implements(
- zope.file.interfaces.IFileReader)
-
- _data = File._data
-
- def read(self, size=-1):
- if self._closed:
- raise ValueError("I/O operation on closed file")
- return self._get_stream().read(size)
-
- def seek(self, offset, whence=0):
- if self._closed:
- raise ValueError("I/O operation on closed file")
- if whence not in (0, 1, 2):
- raise ValueError("illegal value for `whence`")
- self._get_stream().seek(offset, whence)
-
- def tell(self):
- if self._closed:
- raise ValueError("I/O operation on closed file")
- if self._sio is None:
- return 0L
- else:
- return self._sio.tell()
-
- def _get_stream(self):
- # get the right string io
- if self._sio is None:
- # create if we don't have one yet
- self._data = self.__parent__._data
- self._sio = cStringIO.StringIO(self._data)
- elif self._data is not self.__parent__._data:
- # if the data for the underlying object has changed,
- # update our view of the data:
- pos = self._sio.tell()
- self._data = self.__parent__._data
- if pos > len(self._data):
- # need to check if this is really right
- pos = len(self._data)
- self._sio = cStringIO.StringIO(self._data)
- self._sio.seek(pos)
- return self._sio
-
-
-class Writer(Accessor):
-
- zope.interface.implements(
- zope.file.interfaces.IFileWriter)
-
- def flush(self):
- if self._closed:
- raise ValueError("I/O operation on closed file")
- if self._sio is not None:
- self.__parent__._data = self._sio.getvalue()
- self.__parent__.size = len(self.__parent__._data)
-
- def write(self, data):
- if self._closed:
- raise ValueError("I/O operation on closed file")
- self._get_stream().write(data)
-
- def _get_stream(self):
- if self._sio is None:
- self._sio = cStringIO.StringIO()
- return self._sio
-
- def _close(self):
- self.flush()
Index: upload.py
===================================================================
--- upload.py (revision 69216)
+++ upload.py (working copy)
@@ -34,6 +34,11 @@
from zope.file.i18n import _
+
+#FIXME: this is bad.. but I haven't found a way to make security
declarations
on the "file handle" object (inheriting from a normal python file object)..
so for this test we just turn off security all together =(
+from zope.security.proxy import removeSecurityProxy
+
+
_nameFinder = re.compile(r'(.*[\\/:])?(.+)')
def nameFinder(fileob):
@@ -158,5 +163,6 @@
ob.mimeType = mimeType
ob.parameters = {}
w = ob.open("wb")
+ w = removeSecurityProxy(w) #FIXME
w.write(data)
w.close()
On Sunday 23 July 2006 10:56, Christian Theune wrote:
> Hi,
>
> Luis De la Parra wrote:
> > Hello,
> >
> > has anyone tried the zodb with blob support in zope3 ?
> > I just was wondering if there are any plans to support it in the (near)
> > future.
>
> There might be, as soon as we get it into the main line of ZODB. Chris
> and I are working on this every now and then. Due to some feedback we
> got recently, we'll integrate the blob branch into main ZODB in
> approximately Mid-August.
>
> > Today I changed the externals in my checkout to point to the blob-branch
> > and replaced the stringio in zope.file with a blob object, and it really
> > (almost) worked out of the box. The only things I had to do was to add
> > some removeSecurityProxys in the upload/download views because I
couldn't
> > figure out how to make security declarations on the blob/file object,
and
> > replace the "size" attribute with a property to get the size from the
> > blob object.
> >
> > the only thing that is still not working right now, is that deleting
file
> > objects in the zodb leaves the files in the blob directory lying around,
> > and packing the database doesnt remove them either..
>
> Humm. That's either a bug or a problem in the pattern of how to use
> blobs. I'll try to look into it. Do you think you can check your changes
> on Zope 3 into a branch so I can have a look at it? Alternatively I can
> use a patch.
>
> Christian
More information about the Zope3-dev
mailing list