On Fri, Jun 11, 2010 at 11:48 AM, Paul Winkler <span dir="ltr"><<a href="mailto:slinkp@gmail.com">slinkp@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="im">On Fri, Jun 11, 2010 at 10:34 AM, Hanno Schlichting <span dir="ltr"><<a href="mailto:hanno@hannosch.eu" target="_blank">hanno@hannosch.eu</a>></span> wrote:</div><div class="gmail_quote"><div class="im">
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><br>
I tend to run rsync via "rsync -rP --rsh=ssh". The Data.fs is an<br>
append-only file, so rsync is very efficient at handling it. Only<br>
zeopack rewrites things all across the file and causes a subsequent<br>
rsync to be slow again.<br>
<br></blockquote></div></div></blockquote><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
</blockquote></div><div><br>Thanks. I'll do a trial run of this today.<br></div></div></blockquote><div><br>It seems that a second rsync isn't exactly blazing fast with a few changes on the end of the 32G Data.fs. Near as I can tell, it spends a good 10 minutes or so just comparing the files to see if it has any work to do.<br>
Once that phase is done, it seems to spend a lot of its time in IO since by default it builds a new file and replaces the existing file when it's done. Total time ~ 25 minutes.<br><br>The rsync man page paid off though: Using the --append option (or --append-verify on recent enough versions of rsync) seems to reduce the IO a lot, as it's tailor-made for this use case: updating in-place when the source file has only been appended to and potentially losing the target file on failure is OK. (We can manually make a pristine copy prior to starting our downtime, just in case we need to do it over for any reason).<br>
<br></div></div>FWIW total time for the second `rsync -z --append Data.fs` was:<br>real 7m50.253s <br><br>Last time I had to rebuild the index file it took ~ 30 minutes, so this looks like a win. We'll go with rsync.<br clear="all">
<br>-- <br><a href="http://www.slinkp.com">http://www.slinkp.com</a><br><br>