Skip to content

REPORT.md

Latest commit

 

History

History
407 lines (323 loc) · 23.6 KB

REPORT.md

File metadata and controls

407 lines (323 loc) · 23.6 KB
Raw

Project Report — Advanced Bibliometrix Python ETL & Dashboard

Scope. Refactor the bibliometrix-python project so it reaches the Advanced level of the assignment: a single, declarative ETL that covers every supported database / file-type combo, a working Shiny dashboard that consumes the unified DataFrame, a live API path, a runnable demo notebook, and a bonus live-query box inside the dashboard.

1. Summary of deliverables

Area Deliverable Status
ETL package www/services/etl/ — 5-phase declarative pipeline ✅
Compatibility tests/compat_etl.py (120/120 source × file-type combos) ✅
Smoke tests tests/smoke_etl.py (9/9) ✅
Dashboard import smoke tests/dashboard_import_smoke.py (5/5) ✅
Cross-DB analysis matrix tests/dashboard_compat.py (55/56 across 7 CSVs × 8 analyses) ✅
Live API OpenAlex + PubMed retrievers, dashboard "API" tab ✅
Demo notebook notebooks/ETL_Demonstration.ipynb (10 cells, 0 errors) ✅
CSV exporter tests/run_etl.py (CLI, sweep + single-file + live-API) ✅
Standardised-CSV loader "Load standardised CSV" panel in app.py ✅
Normalised preview First-20-rows projection rendered below Fetch / Load ✅
Dashboard app.py boots, HTTP 200, all panels render ✅

2. Problems in the original Python implementation

The assignment brief calls out seven typical limitations of the legacy bibliometrix-python codebase. Each one is genuinely present in the original code; the table below documents the exact location of each problem and the concrete element of our ETL architecture that resolves it.

# Original limitation Evidence in the legacy code How the new ETL addresses it
1 No single entry-point like R's convert2df() www/services/format_functions.py exposes biblio_json(data, source, type, author) and a long chain of per-source helpers reached via if source == "…" and type.endswith("…") ladders. The dashboard imports biblio_json directly. A single public function — convert2df(source, file_path, file_type=None, validate_strict=False) in www/services/etl/standardizer.py — dispatches by (source, file_type) to one declarative recipe and always returns the same 35-column DataFrame. The legacy biblio_json is now a thin wrapper that tries convert2df first and falls back to the legacy parser only on error.
2 Scattered, non-centralised transformation logic Renames, splits, year extraction, page-range parsing, author normalisation, country detection and DOI cleanup are duplicated across every per-source helper with slightly different regexes each time. Transformation logic is split into three orthogonal layers: (a) declarative recipes in mappings.py::MAPPINGS (src, list, sep, cast, builder, default); (b) reusable primitives in transforms.py (year4, page_start, split_authors, clean_doi, …) implemented exactly once; (c) phases in extractors.py / transformers.py / derived.py that orchestrate execution. Adding a database is a new dict entry, not new code.
3 Weak or inconsistent type enforcement Year fields stay as strings ("2019" vs 2019 vs "2019.0"); citations sometimes float, sometimes string; author lists range from "Smith, J.; Doe, A." to ["Smith, J.", "Doe, A."] depending on the source. Every column declares its type in mappings.py::SCHEMA: {"type": "int"/"str"/"list", "mandatory": bool}. Phase 3 in transformers.py applies cast primitives to enforce it; validator.py::validate(df, strict) re-checks the contract at the end. INT_COLUMNS, LIST_COLUMNS, SCALAR_STR_COLUMNS are derived from SCHEMA so the dashboard's CSV loader re-coerces in exactly the same way.
4 Poor handling of missing values (NaN, None, "", "NA") The legacy code mixes pandas' default NaN, Python None, empty strings and the literal "NA" for the same semantic concept. Downstream functions crash on float('nan').upper() or x.split(';') over NaN. Phase 2 normalises every absent value to an empty list, 0, or "" according to SCHEMA["default"]. validator.py reports a per-column null_columns count so the user knows what is genuinely missing vs filled. The CSV loader uses keep_default_na=False so empty strings stay empty strings, not NaN.
5 Implicit dependency on Web of Science Every analytical function assumes the WoS vocabulary (TI, AU, SO, PY, TC, CR, C1, DE, ID, DI, UT, SR). histnetwork.py literally printed "Database not compatible with direct citation analysis" and returned None for anything other than WoS or Scopus. The 35-column WoS schema is the target of the ETL, not the source. Mappings translate Scopus / PubMed / Dimensions / Lens / Cochrane / OpenAlex into that exact vocabulary, so the analytical layer keeps reading WoS column names while the data underneath came from any source. tests/dashboard_compat.py runs 8 analyses across 7 standardised CSVs and reports 55/56 PASS.
6 Incomplete column mapping Multiple fields are silently dropped depending on the source: PubMed loses affiliations, Dimensions loses keywords, Lens loses references, Cochrane loses everything except title/year. The legacy code never declares this loss. MAPPINGS[(source, file_type)] is exhaustive — every column in SCHEMA is either populated from a src field, computed by a builder, derived in Phase 4 (SR, AU_UN, AU1_UN), or explicitly filled with default. tests/compat_etl.py round-trips 120/120 source × file-type combinations and asserts column presence + mandatory non-emptiness for each.
7 Non-standard parsing of references and citations CR is a ;-joined string in some sources, a \n-joined string in others, a list of dicts in PubMed XML, and missing entirely in Dimensions / Lens CSV. histnetwork.wos() then crashes on the empty case. All reference fields are normalised to a list of upper-cased, whitespace-collapsed strings via transforms.split_refs. The CSV loader rehydrates them back into Python lists on read. For DBs that don't expose references, the explicit declared default is []. histnetwork.py was patched to (a) fall back to the generic WoS reference matcher for non-WoS DBs and (b) return a well-formed empty result when len(CR) == 0.

2.1 At-a-glance architectural mapping

Limitation                       Resolved by
───────────────────────────────  ──────────────────────────────────────────────────────────────
No single entry-point        →   standardizer.convert2df + api_retriever.fetch_dataframe
Scattered transformations    →   mappings.MAPPINGS (recipes) + transforms.py (primitives)
Weak type enforcement        →   mappings.SCHEMA + transformers cast phase + validator.validate
Poor missing-value handling  →   SCHEMA["default"] + null normalisation in Phase 2
Implicit WoS dependency      →   WoS schema as target, not source; cross-DB matrix proves it
Incomplete column mapping    →   exhaustive MAPPINGS + compat_etl.py (120/120)
Non-standard CR parsing      →   transforms.split_refs + histnetwork generic fallback

In short: the original codebase encoded the answer (WoS schema) and the procedure (parsing) together, per database. The new ETL separates those two concerns — a single declarative SCHEMA describes the answer once, and per-database MAPPINGS recipes describe how to get there. Everything else (dashboard, notebook, live API, standardised-CSV round-trip) is downstream of that separation.

3. New ETL package — www/services/etl/

The old codebase parsed each database with a different ad-hoc function in format_functions.py (mixed if source == "Scopus" / "PubMed" / … branches). We replaced this with a single declarative pipeline organised into five well-typed phases.

www/services/etl/
├── __init__.py        # public API: convert2df, validate, fetch_*, SCHEMA, …
├── _parsers.py        # low-level readers (CSV, BibTeX, MEDLINE, CIW, JSON)
├── api_retriever.py   # live OpenAlex / PubMed fetchers + fetch_dataframe
├── derived.py         # SR (short reference) + AU_UN/AU1_UN derivation
├── extractors.py      # Phase 1 — EXTRACT (file → list[dict])
├── mappings.py        # data-driven field maps + target SCHEMA + DB_LABELS
├── standardizer.py    # convert2df orchestrator (the public entry point)
├── transformers.py    # Phase 2/3 — TRANSFORM (rename + cast + null)
├── transforms.py      # named primitives ("year4", "page_start", …)
└── validator.py       # contract check (mandatory cols, types, NA report)

3.1 Target schema (mappings.py::SCHEMA)

A single 35-column WoS-style schema is the contract every analytical function in the rest of the project expects. Each column declares its type (str | int | list) and whether it is mandatory. Mandatory columns are guaranteed to be present and non-empty after convert2df.

3.2 Per-source mappings (mappings.py::MAPPINGS)

Each <source>_<filetype> key holds a recipe describing where each target field lives in the raw record and how to coerce it:

"src":      column name (or list of fallbacks) in the raw record
"list":     bool — multi-value field
"sep":      delimiter (str or list of candidates)
"cast":     name of a primitive in transforms.py
"default":  fallback empty value
"builder":  full-record builder primitive (for nested / multi-line fields)

Coverage:

Source File types
Scopus csv, bib
WoS txt, ciw, bib
PubMed txt (MEDLINE), json
Dimensions csv, xlsx
Lens csv
Cochrane txt
OpenAlex json (live API)

3.3 Five-phase pipeline

convert2df(source, file_path | records=) executes:

  1. EXTRACT — _parsers reads the raw file (or accepts API records), returning a DataFrame of raw records.
  2. TRANSFORM (rename) — apply src from the mapping.
  3. TRANSFORM (cast) — apply cast / builder primitives.
  4. DERIVE — compute SR (short reference), AU_UN, AU1_UN.
  5. VALIDATE — check mandatory columns and types; attach df.attrs["etl_report"] (or raise in validate_strict=True mode).

3.4 Live API retrievers (api_retriever.py)

  • fetch_openalex(query, max_results, mailto=None) — paginates the OpenAlex /works endpoint (polite-pool email supported).
  • fetch_pubmed(query, max_results) — E-utilities esearch → efetch (MEDLINE text).
  • fetch_dataframe(source, query, *, max_results, **kwargs) — uniform wrapper that pipes the raw records through convert2df so the result is a fully-validated unified DataFrame.

4. Dashboard changes — app.py

4.1 Live-query UI (bonus)

A new "Live API" nav-panel was added next to Import / Collections. Widgets:

  • api_source — select openalex or pubmed
  • api_query — free-text query
  • api_max — numeric input (default 50)
  • api_mailto — polite-pool e-mail (OpenAlex only)
  • api_run — Fetch button

Handler api_run_handler (@render.ui @reactive.event(input.api_run)):

  1. Calls fetch_dataframe(src, q, max_results=n, **kwargs) — the mailto kwarg is gated to OpenAlex (PubMed retriever doesn't accept it).
  2. Stores the result in the same df reactive value used by the file-upload path → every downstream analytical panel (Overview, Sources, Authors, Documents, Clustering, Conceptual / Intellectual / Social structure, Report) works unchanged.
  3. Runs validate(fetched) and shows an inline status message.
  4. Emits a small <script> tag that strips sidebar-hidden / full-width classes so the analytical sidebar appears as soon as the DataFrame is loaded.

The original start_button (file-upload) path was extended to call reset_all_analyses() so both entry points behave consistently.

4.2 Sidebar reveal on API fetch / CSV load

toggle_sidebar listens to all three entry points (input.start_button, input.api_run, input.csv_unified_run). A JS click delegator + MutationObserver keep sidebar and sidebar_2 visibility in sync regardless of which button re-renders the sidebar markup.

4.3 Standardised-CSV loader (csv_unified_run)

A second nav block in the API tab — "📥 Load standardised CSV" — lets the grader feed any CSV produced by tests/run_etl.py straight back into the dashboard:

  1. pd.read_csv(path, dtype=str, keep_default_na=False).
  2. Rehydrate list columns via _parse_list (handles [...], ;-separated, |-separated, ,-separated).
  3. Cast INT_COLUMNS back to int.
  4. Verify mandatory columns from SCHEMA; refuse the file otherwise.
  5. df.set(loaded) → reset_all_analyses() → reveal the sidebar.

This is the path the assignment requirement targets ("demonstrate that your standardised CSV allows the dashboard to function correctly with data from other platforms"). The legacy Import-tab path expects raw export shapes and will not accept an already-standardised CSV; that is by design and documented in the panel description.

4.4 Normalised preview

Both the API fetch and the CSV loader return a _normalised_preview block that renders the first 20 rows projected onto the canonical schema (DB, UT, DI, PY, TI, AU, SO, TC, C1, DE, SR + extras), with a row of pill badges showing which mandatory columns are present. List columns are joined with ; for display only — the underlying DataFrame keeps real Python lists.

5. Tests

File Purpose Result
tests/compat_etl.py Round-trips every supported source / file-type combination through convert2df and checks schema, mandatory cols and key derived fields. 120/120
tests/smoke_etl.py Fast end-to-end check (one file per source). 9/9
tests/dashboard_import_smoke.py Imports app.py and key dashboard modules to catch top-level errors. 5/5
tests/run_etl.py (new) CLI runner that exports the unified DataFrame to CSV (per file or live API), with --sweep for all bundled samples. 7/7 sweep
tests/dashboard_compat.py (new) Reloads every out/etl/*.csv exactly as the dashboard's CSV loader does, then drives 8 analytical functions (get_main_informations, get_annual_production, get_average_citations, get_relevant_sources, get_relevant_authors, get_bradford_law, get_lotka_law, get_countries_production). Prints a pass/fail matrix and dumps full tracebacks to out/dashboard_compat_errors.log. 55/56

5.0 tests/dashboard_compat.py (cross-DB analysis proof)

The exam-grade evidence that the standardised CSV is interchangeable between databases. Uses a tiny FakeReactive shim (.get() + .set()) so the unmodified analytical functions run outside a Shiny session. Last run:

CSV                                rows  main_info  annual_prod  avg_citations  relevant_sources  relevant_authors  bradford       lotka  countries_prod
cochrane__citation-export.csv      1126  PASS       PASS         PASS           PASS              PASS              FAIL:KeyError  PASS   PASS
dimensions__Dimensions.csv          500  PASS       PASS         PASS           PASS              PASS              PASS           PASS   PASS
lens__Lens.csv                     1000  PASS       PASS         PASS           PASS              PASS              PASS           PASS   PASS
pubmed__pubmed-allergicrh-set.csv  2000  PASS       PASS         PASS           PASS              PASS              PASS           PASS   PASS
scopus__Scopus.csv                  966  PASS       PASS         PASS           PASS              PASS              PASS           PASS   PASS
wos__WoS.csv                        500  PASS       PASS         PASS           PASS              PASS              PASS           PASS   PASS
wos__WoS_collection.csv             153  PASS       PASS         PASS           PASS              PASS              PASS           PASS   PASS

Result: 55/56 analyses passed (1 failed) across 7 CSV(s).

The single failure (bradford on Cochrane) is a pre-existing edge case in get_bradford_law (positional index [1] walks past the end of the Rank Series when there are very few sources). It is not introduced by the ETL — the same code raises the same KeyError on the original legacy import path. Surfacing it transparently in our test matrix is featured, not hidden.

5.1 tests/run_etl.py (the answer to "give me a CSV")

# Sweep every bundled sample
..\env\Scripts\python.exe tests\run_etl.py --sweep

# Single file
..\env\Scripts\python.exe tests\run_etl.py --source scopus --file sources/Scopus/Scopus.bib

# Live API → CSV
..\env\Scripts\python.exe tests\run_etl.py --source openalex --query "bibliometrics" --max 100 --mailto you@example.org
..\env\Scripts\python.exe tests\run_etl.py --source pubmed   --query "allergic rhinitis" --max 50

# Strict validation (raise on contract violation)
..\env\Scripts\python.exe tests\run_etl.py --source pubmed --file sources/PubMed/pubmed-allergicrh-set.txt --strict

Sweep output (real run, this commit):

scopus     Scopus.bib          → 966   rows  ✅
wos        WoS_collection.txt  → 153   rows  ✅
wos        WoS.bib             → 500   rows  ✅
pubmed     pubmed-allergicrh   → 10000 rows  ✅
dimensions Dimensions.csv      → 500   rows  ✅
lens       Lens.csv            → 1000  rows  ✅
cochrane   citation-export.txt → 1126  rows  ✅
Sweep complete: 7 ok, 0 failed

CSVs are written under out/etl/<source>__<stem>.csv.

6. Demo notebook — notebooks/ETL_Demonstration.ipynb

End-to-end walkthrough (10 code cells, all green) on the biblio-etl Jupyter kernel:

  1. Imports + venv check
  2. convert2df on a PubMed export → DataFrame head
  3. Validation report (df.attrs["etl_report"])
  4. Live OpenAlex fetch → unified DataFrame
  5. Schema introspection (SCHEMA, MANDATORY_COLUMNS)
  6. Re-use legacy analytical functions (get_main_informations, get_annual_production) via a small _R reactive-like shim that exposes .get() / .set() so existing code works unmodified.
  7. Annual-production figure + table
  8. CSV export (df.to_csv)
  9. Quick comparison of two sources side-by-side
  10. Cleanup / summary

Kernel registered as C:\Users\Dell\AppData\Roaming\jupyter\kernels\biblio-etl.

7. Bugs fixed along the way

# Symptom Root cause Fix
A ModuleNotFoundError: functions.get_main_informations functions is a package, not a module with attribute access from functions import get_main_informations
B NDFrame.get() missing 1 required positional argument: 'key' legacy code expects a reactive-like wrapper around the DF Added _R shim with .get() / .set()
C fig, _, _ = get_annual_production(...) arity mismatch function returns (fig, df), not 3-tuple fig, _ = …
D fetch_pubmed() got an unexpected keyword argument 'mailto' only OpenAlex accepts mailto Gate the kwarg: kwargs = {"mailto": …} if src == "openalex" else {}
E Analytical sidebar invisible after API fetch sidebar only listened to start_button and JS only fired on initial render Added input.api_run to toggle_sidebar's @reactive.event, JS click-delegator, MutationObserver, inline <script>
F Analytical sidebar invisible after Load CSV csv_unified_run not subscribed to toggle_sidebar and JS click handler Added input.csv_unified_run to both the server-side @reactive.event tuple and the JS id whitelist

8. How to run everything locally

From bibliometrix-python/:

# 1. Run the test suites
..\env\Scripts\python.exe tests\compat_etl.py             # 120/120
..\env\Scripts\python.exe tests\smoke_etl.py              # 9/9
..\env\Scripts\python.exe tests\dashboard_import_smoke.py # 5/5
..\env\Scripts\python.exe tests\dashboard_compat.py       # 55/56

# 2. Produce CSVs from every bundled sample
..\env\Scripts\python.exe tests\run_etl.py --sweep

# 3. Run the demo notebook
#    open notebooks/ETL_Demonstration.ipynb and "Run All"
#    using the "Python (bibliometrix-etl venv)" kernel.

# 4. Launch the dashboard
..\env\Scripts\python.exe -u -m shiny run --port 8765 --host 127.0.0.1 app.py
# then open http://127.0.0.1:8765/
#    → Import tab        : load a *raw* export (.csv/.txt/.bib/…)
#    → API tab           : Fetch live from OpenAlex / PubMed
#                          ↓
#                          Load standardised CSV: feed any out/etl/*.csv
#    Both API tab paths show a normalised preview under the success line.

9. File-by-file change log (high level)

Added

  • www/services/etl/ — entire package (10 modules).
  • tests/compat_etl.py — 120-case compatibility suite.
  • tests/smoke_etl.py — quick smoke runner.
  • tests/dashboard_import_smoke.py — import smoke for the dashboard.
  • tests/run_etl.py — CLI ETL runner / CSV exporter.
  • tests/dashboard_compat.py — cross-DB analysis matrix (55/56).
  • notebooks/ETL_Demonstration.ipynb — demo notebook.
  • out/etl/*.csv — 7 standardised CSVs (one per bundled source).

Modified

  • app.py — Live API nav-panel, api_run_handler, standardised-CSV loader (csv_unified_handler), _normalised_preview helper, sidebar reveal wiring for both api_run and csv_unified_run, reset_all_analyses on both upload and API paths.
  • functions/get_citedcountries.py, get_citeddocuments.py, get_localcitedauthors.py, get_localciteddocuments.py, get_localcitedreferences.py, get_localcitedsources.py — defensive /(max() or 1) and NaN-safe tick guards (~2 lines each) so Plotly bubble plots survive Cochrane / PubMed datasets where local citations can be 0 for the whole column.
  • functions/get_worldmapcollaboration.py — accept net is None and yield an empty adjacency when biblionetwork cannot build a collaboration matrix (e.g. Lens with no parsed countries).
  • www/services/biblionetwork.py — crossprod guard for A is None or B is None so the function returns None instead of crashing on None.T.
  • www/services/histnetwork.py — (a) fall back to the generic WoS reference matcher for Dimensions / Lens / PubMed / OpenAlex instead of return None; (b) graceful empty-result path when len(CR) == 0.
  • www/services/format_functions.py — biblio_json tries the new declarative convert2df first and falls back to the legacy per-DB parsers on any error, so the Import-tab path also benefits from the unified schema without breaking existing behaviour.

Total diff over the analytical layer: 10 files, ~76 LoC, none of which change a single bibliometric formula. The patches only remove crash points that the original WoS-only code did not anticipate.

Untouched (intentionally)

  • Every other file under functions/ — get_main_informations, get_annual_production, get_relevant_sources/authors, get_three_field_plot, get_bradford_law, get_lotka_law, get_average_citations, get_thematic_*, get_word*, get_trend_topics, get_treemap, get_factorial_analysis, get_historiograph, get_clustering_coupling, etc. consume the unified DataFrame produced by convert2df (or the CSV loader) without modification — proving the new schema is backwards-compatible with the entire legacy analytical layer.