Author Total incorrect when doing a topic search from home page

[lewlefton]

Search for Photovoltaics as a topic from the Home page. Data seems incorrect

Total 0 authors
Total 359149 works
Total 5918410 citations

Also the institution list looks incomplete (surprised that Georgia Tech isn't there)?

[gladishd]

I think I have made some API requests on uncovering some of the answers. I was ready to construct a more cohesive response, however, I think that it's quite compelling to see that yes, we're still getting some of the answers from the OpenAlex API. Here are some URLs that might provide some insight:

List of things that do work:

• https://openalex.org/subfields/3107
• https://api.openalex.org/subfields/3107
• https://api.openalex.org/works?filter=topics.subfield.id:3107
• https://api.openalex.org/works?filter=topics.subfield.id:3107&group_by=author.id
• https://api.openalex.org/works?filter=topics.subfield.id:3107&group_by=author.id&per-page=200&page=1
• https://openalex.org/subfields/3107
• https://openalex.org/subfields/2738
• https://api.openalex.org/subfields/3107
• https://api.openalex.org/subfields/3107
• https://api.openalex.org/works?filter=topics.subfield.id:3107&group_by=author.id
• https://explore.openalex.org/works?filter=primary_topic.subfield.id%3Asubfields%2F3107
• https://explore.openalex.org/works?filter=primary_topic.subfield.id%3Asubfields%2F3309
• https://api.openalex.org/works?filter=topics.subfield.id:3107&group_by=author.id&per-page=200
• https://explore.openalex.org/works?filter=primary_topic.subfield.id%3Asubfields%2F3107&page=1
• https://explore.openalex.org/works?filter=primary_topic.subfield.id%3Asubfields%2F3309%7Csubfields%2F3107&page=1
• https://api.openalex.org/works?filter=topics.subfield.id:3107
• https://api.openalex.org/topics?filter=subfield.id:3107&select=id,display_name
• https://openalex.org/works?filter=primary_topic.subfield.id%3Asubfields%2F3107
• https://api.openalex.org/topics?filter=subfield.id:3107&per-page=100&select=id,display_name
• https://api.openalex.org/topics/T11484
• https://api.openalex.org/authors?filter=topics.id:T10002
• https://api.openalex.org/authors?filter=topics.id:T12646
• https://openalex.org/works?page=1&filter=primary_topic.id%3At10523
• https://openalex.org/works?page=1&filter=concepts.id%3Ac184779094,primary_topic.id%3At10523
• https://explore.openalex.org/authors
• https://explore.openalex.org/sources/S124453322
• https://api.openalex.org/works?filter=publication_year:2020
• http://localhost:3000/search?topic=Nuclear+and+High+Energy+Physics
• http://localhost:3000/search?topic=Atomic+and+Molecular+Physics%2C+and+Optics
• https://api.openalex.org/authors
• https://api.openalex.org/concepts
• https://explore.openalex.org/authors
• https://openalex.org/concepts/C41008148
• https://openalex.org/concepts/C184779094
• https://api.openalex.org/authors?filter=x_concepts.id:C184779094
• https://docs.openalex.org/api-entities/works/filter-works
• https://docs.openalex.org/api-entities/authors/filter-authors
• https://help.openalex.org/hc/en-us/articles/24736129405719-Topics
• https://docs.openalex.org/api-entities/concepts/get-lists-of-concepts

List of things that don't work at least not as intended:

• https://openalex.org/concepts/
• https://api.openalex.org/authors?filter=topics.id:3107
• https://api.openalex.org/works?filter=concept.id:S3107
• https://api.openalex.org/works?filter=concept.id:C3107
• https://api.openalex.org/authors?filter=x_concepts.id:S3107
• https://api.openalex.org/authors?filter=x_concepts.id:T12004
• https://api.openalex.org/works?filter=concept.id:https://openalex.org/S3107
• https://api.openalex.org/works?filter=topics.subfield.id:3107&group_by=author.id&per-page=2

List of things that actually don't work:

• https://explore.openalex.org/subfields
• https://api.openalex.org/authors?filter=topics.subfield.id:3107
• https://api.openalex.org/authors?filter=subfield.topics.id:3107
• https://api.openalex.org/authors?filter=subfield.topics.id:S3107
• https://api.openalex.org/works?filter=topics.subfield.id:3107&group_by=results.id
• https://explore.openalex.org/authors?filter=primary_topic.subfield.id%3Asubfields%2F3107&page=1

I'm not going to elaborate on the "pyramidal structure" of "topics" which are really subfields the way we have it defined because fundamentally Domains are the same as Fields are the same as Subfields are the same as Topics, and back. There's too many too specific Topics (4516) to fit on our dropdown menu however the quantity of Subfields (252) is sufficient. So what we call "Topics" are really Subfields.

I think in order to reduce clutter what I can do now is ignore the above and provide a verifiable example of a branch I have created with the entirety of the changes that specifically can be made to the current Saturday May 17 2025 branch main, to acquire and reproduce these results. Because everything is interconnected, the pyramid of topics and subfields and fields and domains, they're all interconnected. We just have to query the URL parameters in the right sequence to find the answers to some questions, such as: authors #, works #, and citations #.

The following is a Pythonic example (save this with the .py extension and run it python openalex_topics_test_2_collabnext.py).

import requests, itertools

URL = "https://api.openalex.org/works"

params = {
    "filter": "topics.subfield.id:3107",
    "group_by": "author.id", 
    "per-page": 200,
    "page": 1
}

total_authors = 0

while True:
    page = requests.get(URL, params=params, timeout=20).json()
    rows = page.get("group_by", [])
    total_authors += len(rows)
    if len(rows) < 200:
        break
    params["page"] += 1
    
print("unique authors linked to sub-field 3107:", total_authors)

Which leads us to the result:

(base) ~/Downloads python openalex_topics_test_2_collabnext.py
unique authors linked to sub-field 3107: 199
(base) ~/Downloads

But what does this actually mean? It means that we can verify this result. Sure enough there are 199 authors:

Source: https://openalex.org/subfields/3107 -> https://openalex.org/works?filter=primary_topic.subfield.id%3Asubfields%2F3107

Now, here's the question. Do we want to include authors who aren't classified as "Top Authors"? I would actually say not because there are a lot of authors who aren't writing that much.

So for now, I would say if nothing else we've got the 199 Top Authors, a result which is easily integrable (from the Pythonic example) into our code.

Sure, the Authors list is Subfield-specific (the top 199 changes as a result of the fact that these are the current top 199 authors specifically for the chosen Subfield), but we always get the top 199 Authors for that particular subfield. What if we want to count ALL authors who have ever contributed to that particular subfield?

Let's look at all the Topics for the Subfield 3107. You could go to the website https://openalex.org/subfields/3107. Or, you could go to the API: https://api.openalex.org/subfields/3107. Either way, you'll see the list of all topics. So what? Is there a way to first fetch all topics programmatically and then find all authors within each topic and then sum them up? Yes, there is. And we can print out the intermediate topics' author counts before we sum them, for verification purposes.

import requests

API      = "https://api.openalex.org"
HEADERS  = {"User-Agent": "full-subfield-tally/0.1"}
TIMEOUT  = 20

SUBFIELD = 3107

def fetch_all_topic_ids(subfield_id):
    ids, page = [], 1
    while True:
        resp = requests.get(
            f"{API}/topics",
            params={
                "filter": f"subfield.id:{subfield_id}",
                "per-page": 200,
                "select": "id",
                "page": page
            },
            headers=HEADERS,
            timeout=TIMEOUT
        ).json()
        batch = [t["id"].split("/")[-1] for t in resp.get("results", [])]
        if not batch:
            break
        ids.extend(batch)
        if len(batch) < 200:
            break
        page += 1
    return ids

def author_count_for(topic_id):
    r = requests.get(
        f"{API}/authors",
        params={
            "filter": f"topics.id:{topic_id}",
            "per-page": 1,
            "select": "id"
        },
        headers=HEADERS,
        timeout=TIMEOUT
    )
    r.raise_for_status()
    return r.json()["meta"]["count"]

def union_count(topic_ids):
    orf = "|".join(topic_ids)
    r = requests.get(
        f"{API}/authors",
        params={
            "filter": f"topics.id:{orf}",
            "per-page": 1,
            "select": "id"
        },
        headers=HEADERS,
        timeout=TIMEOUT
    )
    r.raise_for_status()
    return r.json()["meta"]["count"]

if __name__ == "__main__":
    topics = fetch_all_topic_ids(SUBFIELD)
    print(f"Fetched {len(topics)} topics in subfield {SUBFIELD}\n")

    per_topic = {}
    print(f"{'Topic':<8}  Authors")
    print("-"*20)
    for tid in topics:
        c = author_count_for(tid)
        per_topic[tid] = c
        print(f"{tid:<8}  {c:,}")

    print("\nCalculating union…")
    u = union_count(topics)
    s = sum(per_topic.values())
    print(f"\nUNION (deduped):       {u:,}")
    print(f"SIMPLE SUM (overlap):  {s:,}")

(base) ~/Downloads python openalex_topics_test_5_collabnext.py
Fetched 37 topics in subfield 3107

Topic     Authors
--------------------
T10002    296,470
T10022    247,424
T11177    239,717
T10425    165,635
T10049    180,625
T10382    175,564
T10622    106,256
T10523    188,007
T10988    218,158
T11965    168,145
T11993    363,081
T10923    202,406
T10566    133,435
T11853    239,938
T11175    130,732
T10666    158,361
T11666    100,229
T10739    77,486
T11449    177,704
T11050    121,819
T13531    124,604
T10490    69,570
T10657    67,098
T11414    108,567
T11484    114,630
T11262    36,499
T13493    80,703
T12077    56,707
T11603    30,255
T11804    42,705
T12004    64,123
T12220    30,703
T11897    38,964
T12612    29,773
T11683    22,527
T14363    40,140
T14311    3,735

Calculating union…

UNION (deduped):       2,630,248
SIMPLE SUM (overlap):  4,652,495
(base) ~/Downloads

Based on what we know about the OpenAlex API, that https://api.openalex.org/authors?filter=topics.id:T10002&per-page=1&select=id is giving us the count of authors for Topic T10002, we just have to sum them up (for now this is an approximation; try removing &per-page=1). You see, OpenAlex's https://docs.openalex.org/api-entities/sources/get-lists-of-sources meta.count is not a fixed, transaction-level tally--it's computed via fast, eventually-consistent indexes..

But anyway, enough documentation on how I got here. It's time to integrate the authors works and citations fetches from OpenAlex.

[gladishd]

OKN-CollabNext/CollabNext_public at author-total-incorrect-when-doing-a-topic-search-from-home-page-inquiry-issue-26
https://github.com/OKN-CollabNext/CollabNext_public/tree/author-total-incorrect-when-doing-a-topic-search-from-home-page-inquiry-issue-26
Alright so results. The results are as follows:
And by the way that OpenStreetMap is spot on I was beaming across that map. That is exactly how I would do it. I can't believe we got a geo-location map.
Anyway, here's the "new" OpenAlex query.

git fetch origin
git checkout -b author-total-incorrect-when-doing-a-topic-search-from-home-page-inquiry-issue-26 origin/author-total-incorrect-when-doing-a-topic-search-from-home-page-inquiry-issue-26

The Total 2,630,248 authors is corroborated via our Python example openalex_topics_test_5_collabnext.py..
Total 2168028 works and Total 41240087 citations, that's corroborated via https://api.openalex.org/subfields/3107 .

When comparing http://localhost:3000/search?topic=Atomic+and+Molecular+Physics%2C+and+Optics with https://collabnext.io/search?topic=Atomic+and+Molecular+Physics%2C+and+Optics I do see some more people have joined. But that is all.

[gladishd]

If these results were useful at all ( in the sense of the OpenAlex API) then , here are the minimal viable changes that can in fact be made to the current main branch 95fab7119d06b9b1f1ab9ba06db4e239a6c22265: replace the entire get_subfield_results and the get_subfield_results function only, with the entire block of following code :

import math
import urllib.parse
from datetime import datetime, timedelta

OPENALEX_BASE    = "https://api.openalex.org"
OPENALEX_HEADERS = {
    "User-Agent": "CollabNext/1.0 (mailto:dgladish3@gatech.edu)",
    "Accept":     "application/json",
}
session = requests.Session()
session.headers.update({
    **OPENALEX_HEADERS,
    "X-USER-EMAIL": "dgladish3@gatech.edu"
})

def get_author_count_for_subfield_id(subfield_id: str) -> int:
    sid = str(subfield_id).lstrip("S")
    topic_ids: list[str] = []
    page = 1
    while True:
        resp = session.get(
            f"{OPENALEX_BASE}/topics",
            params={
                "filter": f"subfield.id:{sid}",
                "per-page": 200,
                "select": "id",
                "page": page,
            },
            timeout=20,
        ).json()
        batch = [t["id"].rsplit("/", 1)[-1] for t in resp.get("results", [])]
        if not batch:
            break
        topic_ids.extend(batch)
        if len(batch) < 200:
            break
        page += 1
    if not topic_ids:
        return 0
    or_filter = "|".join(topic_ids)
    r = session.get(
        f"{OPENALEX_BASE}/authors",
        params={
            "filter": f"topics.id:{or_filter}",
            "per-page": 1,
            "select": "id",
        },
        timeout=20,
    )
    r.raise_for_status()
    return r.json()["meta"]["count"]

_TYPE_CACHE: dict[str, dict] = {}
def _get_institution_ids_by_type(inst_type: str) -> set[str]:
    key = inst_type.upper()
    now = datetime.utcnow()
    hit = _TYPE_CACHE.get(key)
    if hit and now - hit["ts"] < timedelta(minutes=10):
        return hit["ids"]
    col = {
        "HBCU":"hbcu","HSI":"hsi","MSI":"msi","AANAPISI":"aanapisi",
        "ANNH":"annh","PBI":"pbi","TCU":"tcu",
        "CARNEGIE R1":"carnegie_r1","CARNEGIE R2":"carnegie_r2",
        "EMERGING":"emerging"
    }.get(key)
    if not col:
        return set()
    rows = execute_query(
        f"SELECT id FROM institutions_filtered WHERE {col} = TRUE", ()
    ) or []
    ids = {r[0] for r in rows}
    _TYPE_CACHE[key] = {"ts": now, "ids": ids}
    return ids

import pprint, requests

OA_HEADERS = {
    "User-Agent": "CollabNext/1.0 (mailto:dgladish3@gatech.edu)",
    "Accept": "application/json",
}

def get_subfield_by_id(subfield_id="3107"):
    url = f"https://api.openalex.org/subfields/{subfield_id}"
    r = requests.get(url, headers=OA_HEADERS)
    r.raise_for_status()
    return r.json()

def get_subfield_via_search(name, per_page=5):
    r = requests.get(
        "https://api.openalex.org/subfields",
        headers=OA_HEADERS,
        params={"search": name, "per-page": per_page},
    )
    r.raise_for_status()
    return r.json()["results"]

def get_subfield_filter_no_comma(name="Atomic and Molecular Physics and Optics"):
    filt = f"display_name.search:{name}"
    r = requests.get(
        "https://api.openalex.org/subfields",
        headers=OA_HEADERS,
        params={"filter": filt},
    )
    r.raise_for_status()
    return r.json()["results"]

def get_subfield_via_topics(name):
    t = requests.get(
        "https://api.openalex.org/topics",
        headers=OA_HEADERS,
        params={"search": name, "per-page": 1},
    ).json()["results"][0]
    subfield_url = t["subfield"]["id"]
    return requests.get(
        subfield_url.replace("openalex.org/S", "api.openalex.org/subfields/"),
        headers=OA_HEADERS,
    ).json()

def fetch_subfield_by_name(name):
    if name.isdigit():
        try:
            return get_subfield_by_id(name)
        except requests.HTTPError:
            pass
    try:
        hits = get_subfield_via_search(name)
        if hits:
            return hits[0]
    except requests.HTTPError:
        pass
    try:
        if "," in name:
            hits = get_subfield_filter_no_comma()
            if hits:
                return hits[0]
    except requests.HTTPError:
        pass
    try:
        return get_subfield_via_topics(name)
    except Exception:
        return None

def fetch_works_for_subfield_id(subfield_id, per_page=20, cursor=None):
    filt = urllib.parse.quote(f"topics.subfield.id:{subfield_id}", safe=':')
    url = f"{OPENALEX_BASE}/works?filter={filt}&per-page={per_page}"
    if cursor:
        url += f"&cursor={cursor}"
    resp = session.get(url)
    resp.raise_for_status()
    return resp.json()

def get_subfield_results(
        topic: str,
        page: int = 1,
        per_page: int = 20,
        map_limit: int = 100,
        institution_type: str | None = None,
):
    sub = fetch_subfield_by_name(topic)
    if not sub:
        app.logger.warning(f"No OpenAlex subfield for '{topic}'")
        return {"metadata": None, "graph": None, "list": None}
    oid = sub["id"].rsplit("/", 1)[-1]
    try:
        author_cnt = get_author_count_for_subfield_id(oid)
    except Exception as e:
        app.logger.warning(f"author-count lookup failed for subfield {oid}: {e}")
        author_cnt = None
    meta = {
        "name": sub["display_name"],
        "work_count": sub["works_count"],
        "cited_by_count": sub["cited_by_count"],
        "oa_link": sub["id"],
        "author_count": author_cnt,
        "researchers": author_cnt,
    }
    data = search_by_topic(topic)
    if data is None:
        app.logger.warning(f"No results found for topic: {topic}")
        return {"metadata": None, "graph": None, "list": None}
    nodes: list[dict] = [{'id': topic, 'label': topic, 'type': 'TOPIC'}]
    edges: list[dict] = []
    lst: list[tuple[str, int]] = []
    coords: list[tuple[str, str, int]] = []
    total_topics = len(data['data'])
    start, end   = (page - 1) * per_page, (page - 1) * per_page + per_page
    for row in data['data'][start:end]:
        inst, n_auth, inst_id = row['institution_name'], row['num_of_authors'], row['institution_id']
        lst.append((inst, n_auth))
        coords.append((inst_id, inst, n_auth))
        nodes.append({'id': inst,  'label': inst,  'type': 'INSTITUTION'})
        nodes.append({'id': n_auth, 'label': n_auth, 'type': 'NUMBER'})
        edges.append({'id': f'{inst}-{topic}',  'start': inst, 'end': topic,
                      'label': 'researches', 'start_type': 'INSTITUTION', 'end_type': 'TOPIC'})
        edges.append({'id': f'{inst}-{n_auth}', 'start': inst, 'end': n_auth,
                      'label': 'number', 'start_type': 'INSTITUTION', 'end_type': 'NUMBER'})
    for row in data['data'][:map_limit]:
        coords.append((row['institution_id'], row['institution_name'], row['num_of_authors']))
    graph = {'nodes': nodes, 'edges': edges}
    return {
        "metadata": meta,
        "metadata_pagination": {
            "total_pages": (total_topics + per_page - 1) // per_page,
            "current_page": page,
            "total_topics": total_topics,
        },
        "graph":       graph,
        "list":        lst,
        "coordinates": coords,
    }

That is all. Now when you run the application, the metadata ( # authors & # works & # citations) as well as the No of people per Organizations List as well as the Organizations List, will be "updated' per the "specifications, of OpenAlex. ./setup.sh (the next few times you might wanna lsof -i :5000 and kill -9 [PID] but that, clearing ports in use might be wholly unnecessary. No new branch needed.

[robertbriggsjr-morehouse]

This is great