[ZODB4] IStorage: Require lastTransaction() to invalidate DB before r…

…eturning ...

... and to provide correct "current" view for load().

This is ZODB4 backport of zopefoundation#313,
which itself is just a more explicit language of
zopefoundation@4a6b0283#diff-881ceb274f9e538d4144950eefce8682R685-R695

It has been amended with ZODB4-specific

      It is guaranteed that after lastTransaction returns, "current" view of
      the storage as observed by load() is ≥ returned tid.

because on ZODB<5 - contrary to ZODB5 - there is "load current" for
which correct semantic must be also provided:
zopefoundation#307 (comment)

Original description follows:

---- 8< ----

Because if lastTransaction() returns tid, for which local database
handle has not yet been updated with invalidations, it could lead to
data corruption due to concurrency issues similar to
zopefoundation#290:

- DB refreshes a Connection for new transaction;
- zstor.lastTransaction() is called to obtain database view for this connection.
- objects in live-cache for this Connection are invalidated with
  invalidations that were queued through DB.invalidate() calls from
  storage.
- if lastTransaction does not guarantee that all DB invalidations for
  transactions with ID ≤ returned tid have been completed, it can be
  that:

	incomplete set of objects are invalidated in live cache

  i.e. data corruption.

This particular data corruption has been hit when working on core of
ZODB and was not immediately noticed:

zopefoundation#307 (review)

this fact justifies the importance of explicitly stating what IStorage
guarantees are / must be in the interface.

This guarantee

- already holds for FileStorage (no database mutations from outside of
  single process);
- is already true for ZEO4 and ZEO5
  zopefoundation#307 (review)
  zopefoundation#307 (comment)
- holds for RelStorage because it implements IMVCCStorage natively;
- is *not* currently true for NEO because NEO sets zstor.last_tid before
  calling DB.invalidate:

  https://lab.nexedi.com/nexedi/neoppod/blob/fc58c089/neo/client/handlers/master.py#L109-124

However NEO is willing to change and already prepared the fix to provide
this guarantee because of data corruption scenario that can happen
without it:

  zopefoundation#307 (comment)
  https://lab.nexedi.com/nexedi/neoppod/commit/a7d101ec
  https://lab.nexedi.com/nexedi/neoppod/commit/a7d101ec#note_112238
  https://lab.nexedi.com/nexedi/neoppod/commit/a7d101ec#note_113331

In other words all storages that, to my knowledge, are in current use
are either already providing specified semantic, or will be shortly
fixed to provide it.

This way we can fix up the interface and make the semantic clear.

/cc @jamadden, @vpelletier, @arnaud-fontaine, @jwolf083, @klawlf82, @gitzit, @jimfulton

src/ZODB/interfaces.py

@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 ##############################################################################
 #
 # Copyright (c) Zope Corporation and Contributors.
@@ -564,6 +565,18 @@ def isReadOnly():
     def lastTransaction():
         """Return the id of the last committed transaction.
 
+        Returned tid is ID of last committed transaction as observed from
+        some time _before_ lastTransaction call returns. In particular for
+        client-sever case, lastTransaction can return cached view of storage
+        that was learned some time ago.
+
+        It is guaranteed that for all IStorageWrappers, that wrap the storage,
+        invalidation notifications have been completed for transactions
+        with ID ≤ returned tid.
+
+        It is guaranteed that after lastTransaction returns, "current" view of
+        the storage as observed by load() is ≥ returned tid.
+
         If no transactions have been committed, return a string of 8
         null (0) characters.
         """