How do I gather Entities in Python?

It is common to want to get all Entity records in Python for processing or analysis.

A Data Lab query does the paging for you. Hand each_page a callback and it reads one page at a time, so nothing but the current page is held no matter how large the Entity Type is. Each page carries its own position, so progress needs no counting of your own.

You will want to install our Python library to facilitate this process.

pip install bruce-models

import bruce_models as BM

ENTITY_TYPE_ID = "YOUR_ENTITY_TYPE_ID"
# Either a session token from a login or a long-lived access token.
TOKEN = "YOUR_TOKEN"

# Setup the API instance to communicate with your account.
bruce = BM.BruceApi({
    "account_id": "YOUR_ACCOUNT_ID",
    "session_id": TOKEN
})

# The selection to read. Nothing is requested until you walk it.
query = BM.DataLabBuilder(bruce, page_size=1000).entity_type(ENTITY_TYPE_ID)

# How many there are, without transferring any of them.
print(f"{query.count():,} Entities to gather")

all_entities = []

def gather(page):
    """
    Handles one page of Entities.
    """
    all_entities.extend(page.items)
    print(f"page {page.number}: {len(page)} Entities, {page.seen:,} so far")

# Reads a page at a time, so only one page is in flight.
walked = query.each_page(gather)

print(f"Total Entities collected: {len(all_entities):,} "
      f"over {walked['pages']} page(s)")