[ZODB-Dev] grab zodb dict keys by search term

[Sean Upton]

On Wed, Dec 11, 2013 at 3:27 AM, Tamer Higazi <tameritoke2 at arcor.de> wrote:
> Perhaps it's my fault, the way I stored the data.
> The point is, that I am looking the fastest and performant way to grab the
> data from a big pool.

I do not think folks are suggesting you iterate over the "big pool"
keys/values/items.  Rather, folks are suggesting that you consider
using BTrees to make your own indexes, if you choose not to use
search/catalog/indexing tools like Hypatia or repoze.catalog (with or
without Souper).  For each field of data you care to search, you need
an index that can yield identifiers that match the keys in your "big
pool" of data (which I assume is also stored in a BTree).

I personally chose to solve this problem by building myself a small
container/retrieval framework uses a combination of BTrees, UUID
usage, zope.schema, and repoze.catalog to tackle problems (this
assumes that all my items are keyed by UUIDs).  Example:
https://github.com/upiq/uu.retrieval

> I don't want to iterate the whole thing over and over again to get the
> matched entries.
> If i would have 100.000 - 500.000 entries, where 150 would match, I don't
> want to go over those completely.

You need indexes, either use a built-in ZODB-native catalog/search
framework like repoze.catalog/hypatia (or Souper), external search
service like Solr/ElasticSearch, or build your own indexes using two
BTrees each (forward mapping of values to lists of matching ids,
reverse mapping of ids to values).

> So, if there is a pythonic way to get it solved on a performant way, or a
> ZODB way as well that would be wondefull.

You need indexes; most folks choose to use some sort of library to
solve this problem instead of DIY indexes, but that's up to you.

Sean