[Checkins] [zopefoundation/zope.interface] 8ac3bd: Move Interface hashing and comparison to C; 2.5 to...

Thu Mar 19 13:41:22 CET 2020

  Branch: refs/heads/master
  Home:   https://github.com/zopefoundation/zope.interface
  Commit: 8ac3bd088d9b924cfb3170b77b41effd2de39d23
      https://github.com/zopefoundation/zope.interface/commit/8ac3bd088d9b924cfb3170b77b41effd2de39d23
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M MANIFEST.in
    A benchmarks/micro.py
    M src/zope/interface/_zope_interface_coptimizations.c
    M src/zope/interface/interface.py
    M src/zope/interface/tests/test_interface.py
    M src/zope/interface/tests/test_sorting.py

  Log Message:
  -----------
  Move Interface hashing and comparison to C; 2.5 to 15x speedup in micro benchmarks

Included benchmark numbers:

Current master, Python 3.8:

.....................
contains (empty dict): Mean +- std dev: 198 ns +- 5 ns
.....................
contains (populated dict): Mean +- std dev: 197 ns +- 6 ns
.....................
contains (populated list): Mean +- std dev: 53.1 us +- 1.2 us

This code:

.....................
contains (empty dict): Mean +- std dev: 77.9 ns +- 2.3 ns
.....................
contains (populated dict): Mean +- std dev: 78.4 ns +- 3.1 ns
.....................
contains (populated list): Mean +- std dev: 3.69 us +- 0.08 us

So anywhere from 2.5 to 15x faster. Not sure how that will translate to
larger benchmarks, but I'm hopeful.

It turns out that messing with ``__module__`` is nasty, tricky
business, especially when you do it from C. Everytime you define a new
subclass, the descriptors that you set get overridden by the type
machinery (PyType_Ready). I'm using a data descriptor and a meta class
right now to avoid that but I'm not super happy with that and would
like to find a better way. (At least, maybe the data part of the
descriptor isn't necessary?) It may be needed to move more code into
C, I don't want a slowdown accessing ``__module__`` either; copying
around the standard PyGetSet or PyMember descriptors isn't enough
because they don't work on the class object (so
``classImplements(InterfaceClass, IInterface)`` fails).

  Commit: b9165b790c831b6d8a87a3f27d4f135143494ff8
      https://github.com/zopefoundation/zope.interface/commit/b9165b790c831b6d8a87a3f27d4f135143494ff8
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M src/zope/interface/interface.py

  Log Message:
  -----------
  Fix doctest by making sure the default type repr can be used.

  Commit: 01e0a7e36cecbd0a36fd49b08b1c21e0a5bba119
      https://github.com/zopefoundation/zope.interface/commit/01e0a7e36cecbd0a36fd49b08b1c21e0a5bba119
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M benchmarks/micro.py

  Log Message:
  -----------
  Benchmarks looking up adapters from components.

Current results (this branch vs master, 354faccebd5b612a2ac8e081a7e5d2f7fb1089c1):

| Benchmark                                 | 38-master | 38-faster                     |
|-------------------------------------------|-----------|-------------------------------|
| query adapter (no registrations)          | 3.81 ms   | 3.03 ms: 1.26x faster (-20%)  |
| query adapter (all trivial registrations) | 4.65 ms   | 3.90 ms: 1.19x faster (-16%)  |
| contains (empty dict)                     | 163 ns    | 76.1 ns: 2.14x faster (-53%)  |
| contains (populated dict)                 | 162 ns    | 76.9 ns: 2.11x faster (-53%)  |
| contains (populated list)                 | 40.3 us   | 3.09 us: 13.04x faster (-92%) |

Also need benchmarks using inheritance. The 'implied' data structures
are also hash/equality based.

  Commit: 5f4bb3f8ec7798b146c007041ff60aac2ca0566e
      https://github.com/zopefoundation/zope.interface/commit/5f4bb3f8ec7798b146c007041ff60aac2ca0566e
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M src/zope/interface/tests/test_interface.py

  Log Message:
  -----------
  Clean up linter errors in test_interface.py so new/real problems are more obvious.

  Commit: 7afd59d869e3391ce27a4a07ad0931eb5e7910a1
      https://github.com/zopefoundation/zope.interface/commit/7afd59d869e3391ce27a4a07ad0931eb5e7910a1
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M src/zope/interface/tests/__init__.py
    M src/zope/interface/tests/test_interface.py

  Log Message:
  -----------
  Fix tests when zope.component is also importable.

  Commit: 413e716f9fb48338829435c4f243133589eb9fe6
      https://github.com/zopefoundation/zope.interface/commit/413e716f9fb48338829435c4f243133589eb9fe6
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M benchmarks/micro.py
    M src/zope/interface/_zope_interface_coptimizations.c
    M src/zope/interface/declarations.py
    M src/zope/interface/interface.py
    M src/zope/interface/tests/test_interface.py

  Log Message:
  -----------
  Avoid use of a metaclass by implementeng __getattribute__.

This is pretty, but it slows down all attribute access to interfaces.
By up to 25%. I'm not sure that's acceptable for things like
Interface.providedBy.

+-------------------------------------------+------------+-------------------------------+
| Benchmark                                 | 38-master3 | 38-faster3                    |
+===========================================+============+===============================+
| read __module__                           | 41.1 ns    | 44.3 ns: 1.08x slower (+8%)   |
+-------------------------------------------+------------+-------------------------------+
| read __name__                             | 41.3 ns    | 51.6 ns: 1.25x slower (+25%)  |
+-------------------------------------------+------------+-------------------------------+
| read __doc__                              | 41.8 ns    | 53.3 ns: 1.28x slower (+28%)  |
+-------------------------------------------+------------+-------------------------------+
| read providedBy                           | 56.7 ns    | 71.6 ns: 1.26x slower (+26%)  |
+-------------------------------------------+------------+-------------------------------+
| query adapter (no registrations)          | 3.85 ms    | 2.95 ms: 1.31x faster (-23%)  |
+-------------------------------------------+------------+-------------------------------+
| query adapter (all trivial registrations) | 4.59 ms    | 3.65 ms: 1.26x faster (-20%)  |
+-------------------------------------------+------------+-------------------------------+
| contains (empty dict)                     | 136 ns     | 55.4 ns: 2.45x faster (-59%)  |
+-------------------------------------------+------------+-------------------------------+
| contains (populated dict)                 | 137 ns     | 55.0 ns: 2.49x faster (-60%)  |
+-------------------------------------------+------------+-------------------------------+
| contains (populated list)                 | 40.2 us    | 2.95 us: 13.62x faster (-93%) |
+-------------------------------------------+------------+-------------------------------+

  Commit: d9f06470f9c45d0710c00e680806a3577b5617f1
      https://github.com/zopefoundation/zope.interface/commit/d9f06470f9c45d0710c00e680806a3577b5617f1
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M src/zope/interface/_compat.py
    M src/zope/interface/_zope_interface_coptimizations.c
    M src/zope/interface/interface.py
    M src/zope/interface/tests/test_registry.py

  Log Message:
  -----------
  Use a descriptor for __module__

This makes the rest of the attribute access fast again, but slows down
__module__.

+-------------------------------------------+------------+-------------------------------+
| Benchmark                                 | 38-master3 | 38-faster-descr               |
+===========================================+============+===============================+
| read __module__                           | 41.1 ns    | 123 ns: 2.99x slower (+199%)  |
+-------------------------------------------+------------+-------------------------------+
| read __name__                             | 41.3 ns    | 39.9 ns: 1.04x faster (-3%)   |
+-------------------------------------------+------------+-------------------------------+
| read __doc__                              | 41.8 ns    | 42.4 ns: 1.01x slower (+1%)   |
+-------------------------------------------+------------+-------------------------------+
| query adapter (no registrations)          | 3.85 ms    | 2.95 ms: 1.30x faster (-23%)  |
+-------------------------------------------+------------+-------------------------------+
| query adapter (all trivial registrations) | 4.59 ms    | 3.67 ms: 1.25x faster (-20%)  |
+-------------------------------------------+------------+-------------------------------+
| contains (empty dict)                     | 136 ns     | 54.8 ns: 2.48x faster (-60%)  |
+-------------------------------------------+------------+-------------------------------+
| contains (populated dict)                 | 137 ns     | 55.7 ns: 2.46x faster (-59%)  |
+-------------------------------------------+------------+-------------------------------+
| contains (populated list)                 | 40.2 us    | 2.86 us: 14.03x faster (-93%) |
+-------------------------------------------+------------+-------------------------------+

Not significant (1): read providedBy

  Commit: a9d90f4418315098686bcff9b978ab2572000df9
      https://github.com/zopefoundation/zope.interface/commit/a9d90f4418315098686bcff9b978ab2572000df9
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M benchmarks/micro.py
    M src/zope/interface/interface.py

  Log Message:
  -----------
  Move to a metaclass for handling __module__.

This offers the absolute best performance at what seems like reasonable complexity.

+-------------------------------------------------------------+----------------+-------------------------------+
| Benchmark                                                   | 38-master-full | 38-faster-meta                |
+=============================================================+================+===============================+
| read __module__                                             | 41.8 ns        | 40.9 ns: 1.02x faster (-2%)   |
+-------------------------------------------------------------+----------------+-------------------------------+
| read __name__                                               | 41.8 ns        | 39.9 ns: 1.05x faster (-5%)   |
+-------------------------------------------------------------+----------------+-------------------------------+
| read providedBy                                             | 56.9 ns        | 58.4 ns: 1.03x slower (+3%)   |
+-------------------------------------------------------------+----------------+-------------------------------+
| query adapter (no registrations)                            | 3.85 ms        | 2.95 ms: 1.31x faster (-24%)  |
+-------------------------------------------------------------+----------------+-------------------------------+
| query adapter (all trivial registrations)                   | 4.62 ms        | 3.63 ms: 1.27x faster (-21%)  |
+-------------------------------------------------------------+----------------+-------------------------------+
| query adapter (all trivial registrations, wide inheritance) | 51.8 us        | 42.2 us: 1.23x faster (-19%)  |
+-------------------------------------------------------------+----------------+-------------------------------+
| query adapter (all trivial registrations, deep inheritance) | 52.0 us        | 41.7 us: 1.25x faster (-20%)  |
+-------------------------------------------------------------+----------------+-------------------------------+
| sort interfaces                                             | 234 us         | 29.9 us: 7.84x faster (-87%)  |
+-------------------------------------------------------------+----------------+-------------------------------+
| sort mixed                                                  | 569 us         | 340 us: 1.67x faster (-40%)   |
+-------------------------------------------------------------+----------------+-------------------------------+
| contains (empty dict)                                       | 135 ns         | 55.2 ns: 2.44x faster (-59%)  |
+-------------------------------------------------------------+----------------+-------------------------------+
| contains (populated dict: interfaces)                       | 137 ns         | 56.1 ns: 2.45x faster (-59%)  |
+-------------------------------------------------------------+----------------+-------------------------------+
| contains (populated list: interfaces)                       | 39.7 us        | 2.96 us: 13.42x faster (-93%) |
+-------------------------------------------------------------+----------------+-------------------------------+
| contains (populated dict: implementedBy)                    | 137 ns         | 55.2 ns: 2.48x faster (-60%)  |
+-------------------------------------------------------------+----------------+-------------------------------+
| contains (populated list: implementedBy)                    | 40.6 us        | 24.1 us: 1.68x faster (-41%)  |
+-------------------------------------------------------------+----------------+-------------------------------+

Not significant (2): read __doc__; sort implementedBy

  Commit: 1094bee45cc6d56f9f0e282dba6ca943577a1e7b
      https://github.com/zopefoundation/zope.interface/commit/1094bee45cc6d56f9f0e282dba6ca943577a1e7b
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M src/zope/interface/_zope_interface_coptimizations.c

  Log Message:
  -----------
  Several small tweaks to GC and deletion handling.

Several places needed to, essentially, call super.

  Commit: a2687a64253c76b0d7452d102f521e7f680b5306
      https://github.com/zopefoundation/zope.interface/commit/a2687a64253c76b0d7452d102f521e7f680b5306
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M CHANGES.rst
    M src/zope/interface/_zope_interface_coptimizations.c
    M src/zope/interface/interface.py
    M src/zope/interface/tests/test_declarations.py
    M src/zope/interface/tests/test_interface.py

  Log Message:
  -----------
  Add tests for comparing InterfaceClass/Implements objects to things without the required attributes.

And fix the C handling of this case.

  Commit: 6be183e34defe640bbad3d898fa3468bf292438a
      https://github.com/zopefoundation/zope.interface/commit/6be183e34defe640bbad3d898fa3468bf292438a
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M CHANGES.rst
    M src/zope/interface/tests/test_interface.py

  Log Message:
  -----------
  Add additional tests for assigning to Interface.__module__.

  Commit: dc719e296d6f8ccf5d989414d0af6d645a93c940
      https://github.com/zopefoundation/zope.interface/commit/dc719e296d6f8ccf5d989414d0af6d645a93c940
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M src/zope/interface/interface.py

  Log Message:
  -----------
  Remove untested except in the metaclass __new__.

Reviewers weren't sure how it could be raised.

  Commit: 42c5ad2d4f88610bb89d35152c1caa1c52dc6927
      https://github.com/zopefoundation/zope.interface/commit/42c5ad2d4f88610bb89d35152c1caa1c52dc6927
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-19 (Thu, 19 Mar 2020)

  Changed paths:
    M .gitignore
    A benchmarks/.gitignore
    M src/zope/interface/declarations.py
    M src/zope/interface/interface.py
    M src/zope/interface/tests/test_declarations.py
    M tox.ini

  Log Message:
  -----------
  Update comments and add a test for more coverage.

  Commit: 04152ae7d2687bc7f100f4f504a342e84fc5c167
      https://github.com/zopefoundation/zope.interface/commit/04152ae7d2687bc7f100f4f504a342e84fc5c167
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-19 (Thu, 19 Mar 2020)

  Changed paths:
    M src/zope/interface/interface.py

  Log Message:
  -----------
  Another comment update, referencing #163

  Commit: 6c98b3381a5fbe5023c72bb560aabffb1af63df3
      https://github.com/zopefoundation/zope.interface/commit/6c98b3381a5fbe5023c72bb560aabffb1af63df3
  Author: Jason Madden <jamadden at gmail.com>
  Date:   2020-03-19 (Thu, 19 Mar 2020)

  Changed paths:
    M .gitignore
    M CHANGES.rst
    M MANIFEST.in
    A benchmarks/.gitignore
    A benchmarks/micro.py
    M src/zope/interface/_compat.py
    M src/zope/interface/_zope_interface_coptimizations.c
    M src/zope/interface/declarations.py
    M src/zope/interface/interface.py
    M src/zope/interface/ro.py
    M src/zope/interface/tests/__init__.py
    M src/zope/interface/tests/test_declarations.py
    M src/zope/interface/tests/test_interface.py
    M src/zope/interface/tests/test_registry.py
    M src/zope/interface/tests/test_ro.py
    M src/zope/interface/tests/test_sorting.py
    M tox.ini

  Log Message:
  -----------
  Merge branch 'faster-eq-hash-comparison'

Compare: https://github.com/zopefoundation/zope.interface/compare/13de77d4edfc...6c98b3381a5f