Skip to content

Bug: search_hdx_dataset fails with CKANAPIError / HDXError #52

@Claudio9701

Description

@Claudio9701

Running search_hdx_dataset() fails. The error seems to come from the HDX API (CKAN) returning a 202 status, which causes hdx-python-api to raise a CKANAPIError that UrbanPy doesn’t handle gracefully.


Code Snippet

import urbanpy as up

brazil_hdx_datasets = up.download.search_hdx_dataset("brazil")
brazil_hdx_datasets

Error:

---------------------------------------------------------------------------
CKANAPIError                              Traceback (most recent call last)
File ~/Documents/code/edu-brazil-data/.venv/lib/python3.11/site-packages/hdx/data/hdxobject.py:116, in HDXObject._read_from_hdx(self, object_type, value, fieldname, action, **kwargs)
    115 try:
--> 116     result = self.configuration.call_remoteckan(action, data)
    117     return True, result

File ~/Documents/code/edu-brazil-data/.venv/lib/python3.11/site-packages/hdx/api/configuration.py:360, in Configuration.call_remoteckan(self, *args, **kwargs)
    359 kwargs["apikey"] = apikey
--> 360 return self.remoteckan().call_action(*args, **kwargs)

File ~/Documents/code/edu-brazil-data/.venv/lib/python3.11/site-packages/ckanapi/remoteckan.py:97, in RemoteCKAN.call_action(self, action, data_dict, context, apikey, files, requests_kwargs)
     96     status, response = self._request_fn(url, data, headers, files, requests_kwargs)
---> 97 return reverse_apicontroller_action(url, status, response)

File ~/Documents/code/edu-brazil-data/.venv/lib/python3.11/site-packages/ckanapi/common.py:134, in reverse_apicontroller_action(url, status, response)
    133 # don't recognize the error
--> 134 raise CKANAPIError(repr([url, status, response]))

CKANAPIError: ['https://data.humdata.org/api/action/package_search', 202, '']

The above exception was the direct cause of the following exception:

HDXError                                  Traceback (most recent call last)
Cell In[115], line 1
----> 1 brazil_hdx_datasets = up.download.search_hdx_dataset("peru")
      2 brazil_hdx_datasets

File ~/Documents/code/edu-brazil-data/.venv/lib/python3.11/site-packages/urbanpy/download/download.py:330, in search_hdx_dataset(country, repository)
    299 """
    300 Dataset search within HDX repositories. Defaults to population density maps.
    301 
   (...)    327 12 | 2019-06-11	| PER_youth_15_24_2019-06-01_csv.zip	            | Youth (ages 15-24)	                    | 16.61	  | https://data.humdata.org/dataset/4e74db39-87f1...
    328 """
    329 # Get dataset list
--> 330 datasets = Dataset.search_in_hdx(f"title:{country.lower()}-{repository}")
    332 resources_records = Dataset.get_all_resources(datasets)
    333 resources_df = pd.DataFrame.from_records(resources_records)

File ~/Documents/code/edu-brazil-data/.venv/lib/python3.11/site-packages/hdx/data/dataset.py:1339, in Dataset.search_in_hdx(cls, query, configuration, page_size, **kwargs)
   1337 rows = min(rows_left, page_size)
   1338 kwargs["rows"] = rows
-> 1339 _, result = dataset._read_from_hdx(
   1340     "dataset",
   1341     query,
   1342     "q",
   1343     Dataset.actions()["search"],
   1344     **kwargs,
   1345 )
   1346 datasets = []
   1347 if result:

File ~/Documents/code/edu-brazil-data/.venv/lib/python3.11/site-packages/hdx/data/hdxobject.py:121, in HDXObject._read_from_hdx(self, object_type, value, fieldname, action, **kwargs)
    119     return False, f"{fieldname}={value}: not found!"
    120 except Exception as e:
--> 121     raise HDXError(
    122         f"Failed when trying to read: {fieldname}={value}! (POST)"
    123     ) from e

HDXError: Failed when trying to read: q=title:peru-high-resolution-population-density-maps-demographic-estimates! (POST)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions