PSS-697
Description
Filing this PSS after Alan’s experience running MetGenC with IOLVIS1A when the data producer delivered four campaigns of data to be ingested in one fell swoop. Normally they deliver one at a time. Based on Alan’s struggles though, even the granule count of just one campaign would have required him to divide the files into a number of data directories to allow metgenc to list a given data directory relatively efficiently so that it could then proceed to get on with processing.
Alan’s summary:
“Reporting back after completing ingest - I first tried breaking down the entire temporal update into the 4 campaigns. The smallest of them (26,914 granules) was still hanging an hour or two after running metgenc process . I ended up further breaking down into 75 directories. The largest of which had 15,836 granules. That one started and completed. So, for me somewhere between 15,000 and 27,000 granules metgenc starts to have problems.
Note: because of browse files, when I say 'granule' that means the data/ directory actually has double that number of files (1 sci, 1 brws).
Further:
Attempting a dry run of just a few granules hangs when the file count is large. Kicking off metgenc process -d -c /share/apps/metgenc/<some.ini> -n 3 was hanging due to MetGenC taking the easiest possible approach of reading everything in the directory before it’d then just process just the three specified. Alan’s account: “I did originally run with -n 3 … I killed it after a few hours and moved a few granules to a separate directory instead.”
PSS-697
Description
Filing this PSS after Alan’s experience running MetGenC with IOLVIS1A when the data producer delivered four campaigns of data to be ingested in one fell swoop. Normally they deliver one at a time. Based on Alan’s struggles though, even the granule count of just one campaign would have required him to divide the files into a number of data directories to allow metgenc to list a given data directory relatively efficiently so that it could then proceed to get on with processing.
Alan’s summary:
“Reporting back after completing ingest - I first tried breaking down the entire temporal update into the 4 campaigns. The smallest of them (26,914 granules) was still hanging an hour or two after running metgenc process . I ended up further breaking down into 75 directories. The largest of which had 15,836 granules. That one started and completed. So, for me somewhere between 15,000 and 27,000 granules metgenc starts to have problems.
Note: because of browse files, when I say 'granule' that means the data/ directory actually has double that number of files (1 sci, 1 brws).
Further:
Attempting a dry run of just a few granules hangs when the file count is large. Kicking off metgenc process -d -c /share/apps/metgenc/<some.ini> -n 3 was hanging due to MetGenC taking the easiest possible approach of reading everything in the directory before it’d then just process just the three specified. Alan’s account: “I did originally run with -n 3 … I killed it after a few hours and moved a few granules to a separate directory instead.”