Hi Shane,<br><br>Thanks for pursuing this.<br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
I have lots of other ideas now, but I don't know which to pursue. I need a lot more information. It would be helpful if you sent me your database to analyze. Some possible causes:<br>
<br>
- Have you looked for filesystem-level corruption yet? I asked this before and I am waiting for an answer.<br>
<br></blockquote><div><br>Yep, I've checked for file system consistency and Mysql consistency without any error reported.<br><br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
- Although there is a pack lock, that lock unfortunately gets released automatically if MySQL disconnects prematurely. Therefore, it is possible to force RelStorage to run multiple pack operations in parallel, which would have unpredictable effects. Is there any possibility that you accidentally ran multiple pack operations in parallel? For example, maybe you have a cron job, or you were setting up a cron job at the time, and you started a pack while the cron job was running. (Normally, any attempt to start parallel pack operations will just generate an error, but if MySQL disconnects in just the right way, you'll get a mess.)<br>
<br></blockquote><div><br>That's not unlikely! I've actually seen traces of packing invoked TTW, however the cron job uses zodbpack. I will try to figure out if the PosKeys starts to surface right after that.<br>
</div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
- Every SQL database has nasty surprises. Oracle, for example, has a nice "read only" mode, but it turns out that mode works differently in RAC environments, leading to silent corruption. As a result, we never use that feature of Oracle anymore. Maybe MySQL has some nasty surprises I haven't yet discovered; maybe the MySQL-specific "delete using" statement doesn't work as expected.<br>
</blockquote><div><br>That could also be the case. In fact we have also seen Mysql locking up longer than expected, but that's another story.<br><br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
- Applications can accidentally cause POSKeyErrors in a variety of ways. For example, persistent objects cached globally can cause POSKeyErrors. Maybe Plone 4 or some add-on uses ZODB incorrectly.<font color="#888888"><br>
</font></blockquote><div><br>I was not aware of that. <br><br>Next step here would probably be to inspect log files further and grab a copy of the dabase before PosKeys started to appear and see if it is possible to recreate the incident.<br>
<br>Again, thanks.<br><br>Anton<br><br>
</div></div>