Download particular files from a dataset

[CivilBengineer]

Hello, I think this is different to issue #73

It would be useful to only download some of the constituent files in a dataset. For example, the MRI-ESM2-0 r1i1p1f1 dataset for tas at 3hrPT resolution is made up of 27 files, but I only need the first 9. Is there any way to specify this that I've missed? Is it possible to add?

As esgpull already finds, lists and downloads the separate files, I'm hoping that this request is different to the more complicated subsetting features available in, for example, esgf pyclient.

Thanks in advance!

[svenrdz]

Hi @CivilBengineer , yes it should be possible! The difficult part is coming up with the right pattern, since there is no direct concept of "first N files".

In the regular workflow (add/update/download), esgpull queries only fetch lists of files. The fact that you always get full datasets is simply because all files in a dataset usually share (almost) all facets. The one they never share is the filename, which is called title in ESGF index nodes.

For your particular request, I guess you're talking about this dataset:

$ esgpull search source_id:MRI-ESM2-0 variant_label:r1i1p1f1 variable_id:tas frequency:3hrPt experiment_id:ssp534-over
Found 1 dataset.
 id │                                  dataset                                   │ #  │   size   
════╪════════════════════════════════════════════════════════════════════════════╪════╪══════════
  0 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108 │ 27 │ 83.6 GiB

You can get the list of files with --file, the filename is the rightmost part in the file column (I trimmed the output for readability):

$ esgpull search source_id:MRI-ESM2-0 variant_label:r1i1p1f1 variable_id:tas frequency:3hrPt experiment_id:ssp534-over --file
Found 27 files.
 id │                                                               file                                                               │   size    
════╪══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╪═══════════
  0 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_204001010… │   3.2 GiB 
  1 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_205001010… │   3.2 GiB
...
  8 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_211101010… │   3.2 GiB 
  9 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_212101010… │   3.2 GiB 
...

From there, you can either list all filenames that you want to download (separated with ,), or come up with 1 or more patterns that filter out what you don't want.

I tried the 2nd method and came up with the query below. I have not double checked the output, I was mostly aiming for 9 files. Note the single quotes are required to not trigger bash expansion on *

$ esgpull search --file source_id:MRI-ESM2-0 variant_label:r1i1p1f1 variable_id:tas frequency:3hrPt experiment_id:ssp534-over 'title:*gn_20*,*gn_210*,*gn_211*'
Found 9 files.
 id │                                                               file                                                               │   size    
════╪══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╪═══════════
  0 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_204001010… │   3.2 GiB 
  1 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_205001010… │   3.2 GiB 
  2 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_206001010… │   3.2 GiB 
  3 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_207001010… │   3.2 GiB 
  4 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_208001010… │   3.2 GiB 
  5 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_209001010… │   3.2 GiB 
  6 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_210001010… │ 327.7 MiB 
  7 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_210101010… │   3.2 GiB 
  8 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_211101010… │   3.2 GiB

It isn't the most practical way to select a subset of files from a dataset, but it's possible!

[CivilBengineer]

Hi @svenrdz , thanks very much for your help! The 'title' facet seems like just what I'm looking for. Unfortunately, limiting the search by the title facet isn't working for me; the search just keeps returning 0 files.

Doing the same search as you:

$ esgpull search source_id:MRI-ESM2-0 variant_label:r1i1p1f1 variable_id:tas frequency:3hrPt experiment_id:ssp534-over --file
Found 54 files.
 id │                                                                        file                                                                        │   size    │       data_node
════╪════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╪═══════════╪════════════════════════
  0 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_209001010000-209912312100.nc │   3.2 GiB │     esgf-node.ornl.gov
  1 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_210001010000-210012312100.nc │ 327.7 MiB │ esgf-data03.diasjp.net
  2 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_211101010000-212012312100.nc │   3.2 GiB │ esgf-data03.diasjp.net
  3 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_213101010000-214012312100.nc │   3.2 GiB │     esgf-node.ornl.gov
  4 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_227101010000-228012312100.nc │   3.2 GiB │     esgf-node.ornl.gov
  5 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_206001010000-206912312100.nc │   3.2 GiB │     esgf-node.ornl.gov
  6 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp534-over.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp534-over_r1i1p1f1_gn_216101010000-217012312100.nc │   3.2 GiB │ esgf-data03.diasjp.net

But then

$ esgpull search --file source_id:MRI-ESM2-0 variant_label:r1i1p1f1 variable_id:tas frequency:3hrPt experiment_id:ssp534-over 'title:*gn_20*,*gn_210*,*gn_211*,*v2019*'
Found 0 files.

The dataset I'm actually looking for is MRI-ESM2-0 ssp585, which has 29 files, not 27 as I incorrectly stated earlier.

$ esgpull search source_id:MRI-ESM2-0 variant_label:r1i1p1f1 variable_id:tas frequency:3hrPt experiment_id:ssp585 data_node:esgf-node.ornl.gov --file

Found 29 files.
 id │                                                                   file                                                                   │  size
════╪══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╪═════════
  0 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_209501010000-210012312100.nc │ 1.9 GiB
  1 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_203501010000-204412312100.nc │ 3.2 GiB
  2 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_215101010000-216012312100.nc │ 3.2 GiB
  3 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_217101010000-218012312100.nc │ 3.2 GiB
  4 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_225101010000-226012312100.nc │ 3.2 GiB
  5 │ CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_207501010000-208412312100.nc │ 3.2 GiB

I'm just trying to get the files up to 2100, so I use the following search, but again get 0 results

$ esgpull search --file source_id:MRI-ESM2-0 variant_label:r1i1p1f1 variable_id:tas frequency:3hrPt experiment_id:ssp585 data_node:esgf-node.ornl.gov 'title:*gn_20*'
Found 0 files.

I also tried searching for a particular file name, but that didn't work either:

$ esgpull search --file source_id:MRI-ESM2-0 variant_label:r1i1p1f1 variable_id:tas frequency:3hrPt experiment_id:ssp585 data_node:esgf-node.ornl.gov title:CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_209501010000-210012312100.nc
Found 0 files.

Do you have any idea what's going wrong here? I'm on version 0.9.3, if that helps. Thanks in advance!

[svenrdz]

For your last query, the value you are using is not the filename, only the last part is:

• file_id: CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_209501010000-210012312100.nc
• title: tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_209501010000-210012312100.nc

To know the exact facet key/value pairs, you can inspect a query's results (if there are any) using either --detail N (N being the row id in the search results, the leftmost column), or --dry-run to print the URL esgpull is using, which can be pasted in any web browser to view the raw output.

$ esgpull search --file source_id:MRI-ESM2-0 variant_label:r1i1p1f1 variable_id:tas frequency:3hrPt experiment_id:ssp585 data_node:esgf-node.ornl.gov --detail 0
FILE
│ added    2025-12-09T10:16:40+00:00Z
│ updated  2025-12-09T10:16:40+00:00Z
└── activity_id:        ScenarioMIP                                                                                                                
    data_node:          esgf-node.ornl.gov                                                                                                         
    dataset_id:         CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.v20191108|esgf-node.ornl.gov                                   
    experiment_id:      ssp585                                                                                                                     
    frequency:          3hrPt                                                                                                                      
    grid_label:         gn                                                                                                                         
    index_node:         us-index                                                                                                                   
    instance_id:        CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.v20191108.tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_209501010000-2…
    institution_id:     MRI                                                                                                                        
    master_id:          CMIP6.ScenarioMIP.MRI.MRI-ESM2-0.ssp585.r1i1p1f1.3hr.tas.gn.tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_209501010000-21001231210…
    member_id:          r1i1p1f1                                                                                                                   
    mip_era:            CMIP6                                                                                                                      
    nominal_resolution: 100 km                                                                                                                     
    project:            CMIP6                                                                                                                      
    realm:              atmos                                                                                                                      
    source_id:          MRI-ESM2-0                                                                                                                 
    table_id:           3hr                                                                                                                        
    title:              tas_3hr_MRI-ESM2-0_ssp585_r1i1p1f1_gn_209501010000-210012312100.nc                                                         
...

$ esgpull search --file source_id:MRI-ESM2-0 variant_label:r1i1p1f1 variable_id:tas frequency:3hrPt experiment_id:ssp585 data_node:esgf-node.ornl.gov --dry-run
https://esgf-node.ornl.gov/esgf-1-5-bridge?type=File&offset=0&limit=20&format=application%2Fsolr%2Bjson&query=data_node%3A%22esgf-node.ornl.gov%22+
AND+experiment_id%3A%22ssp585%22+AND+frequency%3A%223hrPt%22+AND+source_id%3A%22MRI-ESM2-0%22+AND+variable_id%3A%22tas%22+AND+variant_label%3A%22r1
i1p1f1%22&distrib=true&latest=true

---

The ORNL index node is a bridge API that mimicks the solr API but uses a different search engine on the backend, so your results vary from mine because I was using api.index_node = esgf.ceda.ac.uk. It reproduces solr behavior perfectly for all cases but one (as far as i know): unfortunately that case is the * patterns...

So for the ORNL index node, the list of exact filenames is the only method that works.
Note that URL have a fixed maximum length, it's pretty long but in earlier esgpull versions where lists of files were always used instead of queries, this limit was hit consistently. With 9 filenames you should be fine, but with longer lists it might be required to split the query into multiple smaller queries.

[CivilBengineer]

Thanks again, this is a big help. I changed the index node to esgf.ceda.ac.uk as well and constructed a query that successfully found all the files.

Please ignore previous statement of further error (since deleted) - I didn't realise I'd briefly lost connection to the remote drive where the tmp folder is stored.

Download of specific files now working successfully - thanks very much, @svenrdz !

[svenrdz]

I'll keep this issue open, as it might be interesting to more people, until we find time to update the documentation with usage tips.