add figshare functions to download, read, and process datasets available via the other private fighsare repository

[ernestguevarra]

Current figshare functions interacts with the figshare API and interfaces with the Pandemic PACT Figshare account that is under the University of Oxford Figshare services (see https://portal.sds.ox.ac.uk/pandemicpact). We are able to do this because this Figshare repository has appropriate identifiers to be accessed via the Fighsare API. However, this repository doesn't have the the Pandemic PACT datasets that are updated to current. This repository is also not the repository that is linked to by the Pandemic PACT dataset (website link to figshare is this - https://figshare.com/s/9e712aa1f4255e37b0db). Note that the s in this URL's path signifies a privately shared repository which means that the details to access this repository via Figshare API is private to those who owns the repository and to those given the permission to access it. However, the owners of this repository has made the datasets downloadable. This means that these datasets cannot be accessed via the Figshare API by someone who doesn't have the permission or the credentials but anyone can bulkd download the datasets. We describe this issue in this article - https://oxford-ihtm.io/pactr/articles/figshare-workflow.html#about-the-pandemic-pact-figshare-repository - on the pactr package website.

Given this, the current pactr functions only gives functionality to access older/dated datasets but not the up-to-date datasets provided by Pandemic PACT.

This feature proposal suggests to create a set of functions that will bulk download the datasets described in the second scenario above, extract, and process. We propose to call these add the suffice _private to these functions to distinguish from the figshare functions that interfaces with the Figshare API.

[ernestguevarra]

The reliable way is to compare a hash (or content) of the two files rather than just names or sizes. In R you have a few options depending on how strict you want to be:

1. Content hash (most robust) — base R ships tools::md5sum():

same_file <- function(path_a, path_b) {
  if (!file.exists(path_a) || !file.exists(path_b)) return(FALSE)
  unname(tools::md5sum(path_a) == tools::md5sum(path_b))
}

This catches any difference in bytes, regardless of timestamps or filename.

2. Cheap pre-check, then hash — hashing large files is wasteful if sizes already differ:

same_file <- function(path_a, path_b) {
  if (!file.exists(path_a) || !file.exists(path_b)) return(FALSE)
  sa <- file.info(path_a)$size
  sb <- file.info(path_b)$size
  if (sa != sb) return(FALSE)
  unname(tools::md5sum(path_a) == tools::md5sum(path_b))
}

3. Comparing an in-memory/just-downloaded object vs. the file on disk — if the "new" file isn't written yet, hash its raw bytes with the digest package:

new_md5  <- digest::digest(raw_or_object, algo = "md5", serialize = FALSE)
disk_md5 <- unname(tools::md5sum(existing_path))
identical(new_md5, disk_md5)

A couple of notes given your download workflow:

• Don't trust filename or modification time — same name says nothing about content, and mtimes differ on every download.
• If the server provides an ETag or Content-MD5 header, comparing that against the local file's hash lets you skip re-downloading entirely. With httr2 (which you're now using per your recent commit) you can read resp_header(resp, "etag").

[ernestguevarra]

Pattern A — download to a temp file, then compare and decide

This always fetches the bytes, but only replaces the existing file when the content actually differs. Safe and simple.

First, a small helper (good fit for utils.R):

#' @keywords internal

same_file <- function(path_a, path_b) {
  if (!file.exists(path_a) || !file.exists(path_b)) return(FALSE)
  ## cheap size pre-check before hashing
  if (file.info(path_a)$size != file.info(path_b)$size) return(FALSE)
  unname(tools::md5sum(path_a) == tools::md5sum(path_b))
}

Then the function downloads to a temp path and compares:

pact_download_figshare_private <- function(.url, path,
                                           overwrite = FALSE,
                                           quiet = TRUE) {
  download_url <- "https://figshare.com/ndownloader/articles/26937448?private_link=9e712aa1f4255e37b0db"

  if (!missing(.url)) {
    download_url <- .url
  }

  destfile <- file.path(path, "pandemic-pact-figshare.zip")

  ## Nothing there yet -> just download normally ----
  if (!file.exists(destfile)) {
    return(
      pact_download_url(.url = download_url, destfile = destfile, quiet = quiet)
    )
  }

  ## File exists -> download to temp, compare, then decide ----
  tmp <- tempfile(fileext = ".zip")
  on.exit(unlink(tmp), add = TRUE)
  pact_download_url(.url = download_url, destfile = tmp, quiet = quiet)

  if (same_file(tmp, destfile)) {
    ## identical content -> keep existing, skip the write
    message("Existing file is identical to the remote file; keeping local copy.")
    return(destfile)
  }

  if (!overwrite) {
    message("Remote file differs from local file. Set `overwrite = TRUE` to replace.")
    return(destfile)
  }

  file.copy(tmp, destfile, overwrite = TRUE)
  destfile
}

Note this changes the meaning of overwrite slightly (and pleasantly): it now only triggers when the content genuinely differs, so you never overwrite an unchanged file.

Pattern B — compare before downloading, using a server checksum

This is the efficient one — it avoids downloading at all if nothing changed — but it only works if Figshare gives you a checksum to compare against. The public Figshare API exposes a per-file computed_md5 (you already touch the API in pact_download_figshare() via pact_client$hostdata$files). The ndownloader/articles/... endpoint zips multiple files on the fly, so there's no single stable md5 for the zip — meaning Pattern B fits the per-file pact_download_figshare() better than the bundled private-zip download.

A sketch of that idea, were a checksum available:

## pseudo: only download if remote checksum != local checksum
remote_md5 <- pact_client$hostdata$files$computed_md5      # from Figshare API
if (file.exists(destfile) &&
    unname(tools::md5sum(destfile)) == remote_md5) {
  return(destfile)                                          # unchanged, skip
}

For a zip that's assembled on the fly you could fall back to the ETag / Content-Length headers via a HEAD-style request (httr2::req_method(req, "HEAD") then httr2::resp_header(resp, "etag")), but Figshare's ndownloader ETags aren't guaranteed stable, so I wouldn't rely on it for correctness.

[ernestguevarra]

just figured out that we don't need to separate private from oxford pages. The private share link from the website is the information that has caused the confusion. The appropriate link to the datasets from the Oxford repository is this - https://portal.sds.ox.ac.uk/articles/dataset/PandemicPACT_Dataset_Data_Dictionary_Files/26937448.

This resolves to a proper Figshare repository that has citations and can be interfaced with by the deposits package via the Figshare API.

The pull requests that resolves these issues/tasks intead refactors the existing pact_figshare_*() functions given this discovery.