<div dir="ltr">Hi,<div><br></div><div style>Will be cool if you create a package with this script and with instructions to patch ZODB and maybe new major version could have the new API.</div><div style><br></div><div style>
If you need help I'm interested to do it. :-)</div><div style><br></div><div style>Cheers, </div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Feb 14, 2013 at 6:29 PM, Jürgen Herrmann <span dir="ltr"><<a href="mailto:Juergen.Herrmann@xlhost.de" target="_blank">Juergen.Herrmann@xlhost.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">Am 07.02.2013 17:11, schrieb Jim Fulton:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Thu, Feb 7, 2013 at 10:48 AM, Jürgen Herrmann<div><div class="h5"><br>
<<a href="mailto:Juergen.Herrmann@xlhost.de" target="_blank">Juergen.Herrmann@xlhost.de</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Am 06.02.2013 15:05, schrieb Jürgen Herrmann:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi there!<br>
<br>
I hav a relstorage with mysql backend that grew out of bounds<br>
and we're looking into different backend solutions now. Possibly<br>
also going back to FileStorage and using zeo...<br>
<br>
Anyway we'll have to convert the databases at some point. As these<br>
are live DBs we cannot shut them down for longer than the<br>
ususal maintenance interval during the night, so for maybe 2-3h.<br>
<br>
a full conversion process will never complete in this time so<br>
we're looking for a process that can split the conversion into<br>
two phases:<br>
<br>
1. copy transactions from backup of the source db to the destination<br>
db. this can take a long time, we don't care. note the last<br>
timestamp/transaction_id converted.<br>
2. shut down the source db<br>
3. copy transactions from the source db to the destination db, starting<br>
at the last converted transaction_id. this should be fast, as only<br>
a few transactions need to be converted, say < 1% .<br>
<br>
<br>
if i would reimplement copyTransactionsFrom() to accept a start<br>
transaction_id/timestamp, would this result in dest being an exact<br>
copy of source?<br>
<br>
source = open_my_source_storage()<br>
dest = open_my_destination_storage()<br>
dest.copyTransactionsFrom(<u></u>source)<br>
last_txn_id = source.lastTransaction()<br>
source.close()<br>
dest.close()<br>
<br>
source = open_my_source_storage()<br>
# add some transactions<br>
source.close()<br>
<br>
source = open_my_source_storage()<br>
dest = open_my_destination_storage()<br>
dest.copyTransactionsFrom(<u></u>source, last_txn_id=last_txn_id)<br>
source.close()<br>
dest.close()<br>
</blockquote>
<br>
<br>
I will reply to myself here :) This actually works, tested with a<br>
modified version of FileStorage for now. I modified the signature<br>
of copyTransactionsFrom to look like this:<br>
<br>
def copyTransactionsFrom(self, source, verbose=0, not_before_tid=None):<br>
</blockquote>
<br>
``start`` would be better to be consistent with the iterator API.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
not_before_tid is a packed tid or None, None meaning "copy all"<br>
(the default, so no existing API usage would break).<br>
<br>
Is there public interest in modifying this API permamently?<br>
</blockquote>
<br>
+.1<br>
<br>
This "API" is a bit of an attractive nuisance. I'd rather people<br>
learn how to use iterators in their own scripts, as they are very<br>
useful and powerful. This API just hides that.<br>
<br>
The second part, replaying old transactions is a bit more subtle,<br>
but it's still worth it for people to be aware of it.<br>
<br>
If I were doing this today, I'd make this documentation<br>
rather than API. But then, documentation ... whimper.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Anybody want to look at the actual code changes?<br>
</blockquote>
<br>
Sure, if they have tests. Unfortunately, we can only accept<br>
pull requests from zope contributors. Are you one?<br>
Wanna be one? :)<br>
<br>
Jim<br>
</div></div></blockquote>
<br>
1. SUCCESS. I migrated to FileStorage/ZEO successfully with the modified<br>
copyTransactionsFrom() with downtime of only 10 minutes. Very cool :)<br>
DB sizes are reasonable again and repozo backups are set up, just like<br>
before the migration to RelStorage. Feels robust again, a good feeling.<br>
I'd advise anybody thinking about migrating to RelStorage to think<br>
about a proper (incremental) backup strategy for the underlying sql db<br>
first, this might become crucial! repozo makes your life easy there<br>
when using FileStorage.<br>
<br>
2. Regarding the tests: I did not find any test for copyTransactionsFrom()<br>
in the current test suite, am I blind? If there isn't any test yet,<br>
maybe you could suggest a proper file for the test, maybe even a similar<br>
one to start from?<br>
<br>
3. Regarding my request to have this in ZODB 3.10.5:<br>
After reading my own statement I quickly realized that changing an API<br>
can only happen in a major version. Anyway, this was a one-shot action<br>
so I'll probably never need my own code again :)<br>
<br>
best regards,<br>
Jürgen<div class="im HOEnZb"><br>
-- <br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
XLhost.de ® - Webhosting von supersmall bis eXtra Large <<<br>
</blockquote></blockquote>
<br>
XLhost.de GmbH<br>
Jürgen Herrmann, Geschäftsführer<br>
Boelckestrasse 21, 93051 Regensburg, Germany<br>
<br>
Geschäftsführer: Jürgen Herrmann<br>
Registriert unter: HRB9918<br>
Umsatzsteuer-<u></u>Identifikationsnummer: DE245931218<br>
<br>
Fon: +49 (0)800 XLHOSTDE [0800 95467833]<br>
Fax: <a href="tel:%2B49%20%280%29800%2095467830" value="+4980095467830" target="_blank">+49 (0)800 95467830</a><br>
Web: <a href="http://www.XLhost.de" target="_blank">http://www.XLhost.de</a><br></div><div class="HOEnZb"><div class="h5">
______________________________<u></u>_________________<br>
For more information about ZODB, see <a href="http://zodb.org/" target="_blank">http://zodb.org/</a><br>
<br>
ZODB-Dev mailing list - <a href="mailto:ZODB-Dev@zope.org" target="_blank">ZODB-Dev@zope.org</a><br>
<a href="https://mail.zope.org/mailman/listinfo/zodb-dev" target="_blank">https://mail.zope.org/mailman/<u></u>listinfo/zodb-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Rudá Porto Filgueiras<br><a href="http://python-blog.blogspot.com">http://python-blog.blogspot.com</a><br><a href="http://twitter.com/rudaporto">http://twitter.com/rudaporto</a>
</div>