Add bulk-update command to index/delete records from TIMDEX parquet dataset

Why these changes are being introduced:
* The timdex-index-manager (TIM) needs to support the v2 parquet dataset,
which now contains records for both indexing and deleting. The new CLI
command performs a "bulk update" given a subset of the dataset
(filtered by 'run_date' and 'run_id') and uses the timdex-dataset-api
library to read records from the TIMDEXDataset.

By introducing a new CLI command, it doesn't require the feature
flagging approach, allowing the existing CLI commands and helper functions
to remain untouched for v1 purposes.

How this addresses that need:
* Implement 'bulk-update' CLI command
* Add unit tests for 'bulk-update'
* Move v1 CLI unit tests to 'test_cli_v1'
* Update README

Side effects of this change:
* TIM remains backwards v1 compatible but will now support v2 runs.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-428

Author: jonavellecuerdo

Makefile

@@ -90,4 +90,13 @@ dist-stage:
 publish-stage:
 	docker login -u AWS -p $$(aws ecr get-login-password --region us-east-1) $(ECR_URL_STAGE)
 	docker push $(ECR_URL_STAGE):latest
-	docker push $(ECR_URL_STAGE):`git describe --always`
+	docker push $(ECR_URL_STAGE):`git describe --always`
+
+##############################
+# Local Opensearch commands
+##############################
+
+local-opensearch: # Run a local instance of Opensearch via Docker Compose
+	docker pull opensearchproject/opensearch:latest
+	docker pull opensearchproject/opensearch-dashboards:latest
+	docker compose --env-file .env up

Pipfile

@@ -21,6 +21,7 @@ mypy = "*"
 pre-commit = "*"
 pytest = "*"
 ruff = "*"
+timdex-dataset-api = { git = "git+https://github.com/MITLibraries/timdex-dataset-api.git"}
 vcrpy = "*"
 
 [requires]

Pipfile.lock

@@ -9,6 +9,7 @@ TIMDEX! Index Manager (TIM) is a Python CLI application for managing TIMDEX indi
 - To update dependencies: `make update`
 - To run unit tests: `make test`
 - To lint the repo: `make lint`
+- To run local OpenSearch with Docker: `make local-opensearch`
 - To run the app: `pipenv run tim --help`
 
 **Important note:** The sections that follow provide instructions for running OpenSearch **locally with Docker**. These instructions are useful for testing. Please make sure the environment variable `TIMDEX_OPENSEARCH_ENDPOINT` is **not** set before proceeding.
@@ -92,34 +93,21 @@ For a more detailed example with test data, please refer to the Confluence docum
 
 ### Required ENV
 
-```
-# Set to `dev` for local development, this will be set to `stage` and `prod` in those environments by Terraform.
-WORKSPACE=dev 
+```shell
+WORKSPACE=### Set to `dev` for local development, this will be set to `stage` and `prod` in those environments by Terraform.
 ```
 
 ## Optional ENV
 
-```
-# Only needed if AWS region changes from the default of us-east-1.
-AWS_REGION=
-
-# Chunk size limit for sending requests to the bulk indexing endpoint, in bytes. Defaults to 104857600 (100 * 1024 * 1024) if not set.
-OPENSEARCH_BULK_MAX_CHUNK_BYTES= 
-
-# Maximum number of retries when sending requests to the bulk indexing endpoint. Defaults to 50 if not set.
-OPENSEARCH_BULK_MAX_RETRIES=
-
-# Only used for OpenSearch requests that tend to take longer than the default timeout of 10 seconds, such as bulk or index refresh requests. Defaults to 120 seconds if not set.
-OPENSEARCH_REQUEST_TIMEOUT=
-
-# The ingest process logs the # of records indexed every nth record. Set this env variable to any integer to change the frequency of logging status updates. Can be useful for development/debugging. Defaults to 1000 if not set.
-STATUS_UPDATE_INTERVAL= 
-
-# If using a local Docker OpenSearch instance, this isn't needed. Otherwise set to OpenSearch instance endpoint without the http scheme (e.g., "search-timdex-env-1234567890.us-east-1.es.amazonaws.com"). Can also be passed directly to the CLI via the `--url` option.
-TIMDEX_OPENSEARCH_ENDPOINT= 
-
-# If set to a valid Sentry DSN, enables Sentry exception monitoring This is not needed for local development.
-SENTRY_DSN=
+```shell
+AWS_REGION=### Only needed if AWS region changes from the default of us-east-1.
+OPENSEARCH_BULK_MAX_CHUNK_BYTES=### Chunk size limit for sending requests to the bulk indexing endpoint, in bytes. Defaults to 104857600 (100 * 1024 * 1024) if not set.
+OPENSEARCH_BULK_MAX_RETRIES=### Maximum number of retries when sending requests to the bulk indexing endpoint. Defaults to 50 if not set.
+OPENSEARCH_INITIAL_ADMIN_PASSWORD=###If using a local Docker OpenSearch instance, this must be set (for versions >= 2.12.0).
+OPENSEARCH_REQUEST_TIMEOUT=### Only used for OpenSearch requests that tend to take longer than the default timeout of 10 seconds, such as bulk or index refresh requests. Defaults to 120 seconds if not set.
+STATUS_UPDATE_INTERVAL=### The ingest process logs the # of records indexed every nth record. Set this env variable to any integer to change the frequency of logging status updates. Can be useful for development/debugging. Defaults to 1000 if not set.
+TIMDEX_OPENSEARCH_ENDPOINT=### If using a local Docker OpenSearch instance, this isn't needed. Otherwise set to OpenSearch instance endpoint without the http scheme (e.g., "search-timdex-env-1234567890.us-east-1.es.amazonaws.com"). Can also be passed directly to the CLI via the `--url` option.
+SENTRY_DSN=### If set to a valid Sentry DSN, enables Sentry exception monitoring This is not needed for local development.
 ```
 
 ## CLI commands
@@ -153,6 +141,7 @@ Usage: tim [OPTIONS] COMMAND [ARGS]...
 ╭─ Bulk record processing commands ───────────────────────────────────────────────────────────────────────────────────────────────────╮
 │ bulk-index                         Bulk index records into an index.                                                                │
 │ bulk-delete                        Bulk delete records from an index.                                                               │
+│ bulk-update                        Bulk update records from an index.                                                               │
 ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 

README.md

@@ -10,6 +10,7 @@ services:
       - discovery.type=single-node
       - bootstrap.memory_lock=true
       - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
+      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
     volumes:
       - opensearch-local-data:/usr/share/opensearch/data
     networks:
@@ -21,6 +22,7 @@ services:
     environment:
       - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true"
       - 'OPENSEARCH_HOSTS=["http://opensearch:9200"]'
+      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
     networks:
       - opensearch-local-net
 volumes:

compose.yaml

@@ -7,7 +7,7 @@ disallow_untyped_defs = true
 exclude = ["tests/"]
 
 [[tool.mypy.overrides]]
-module = ["ijson", "smart_open"]
+module = ["ijson", "smart_open", "timdex_dataset_api"]
 ignore_missing_imports = true
 
 [tool.pytest.ini_options]
@@ -27,8 +27,6 @@ select = ["ALL", "PT"]
 
 ignore = [
     # default
-    "ANN101", 
-    "ANN102", 
     "COM812", 
     "D107", 
     "N812", 
@@ -41,6 +39,7 @@ ignore = [
     "D102",
     "D103",
     "D104", 
+    "G004",
     "PLR0912",
     "PLR0913", 
     "PLR0915",