Merge branch 'dev'

Author: temuller

README.md

@@ -166,21 +166,32 @@ If you make use of HostPhot, please cite the following [paper](https://joss.theo
 } 
 ```
 
-## What's new!
+## What's new
+
+v3.1.0
+
+* Using DES DR2 instead of DR1
+* Downloading images, for some surveys, trying to avoid timeout errors.
+* Adapting global aperture even if the survey is the same as the reference (mainly to avoid issues with JWST).
+* Fixing/adding features from different issues, mainly for HST/JWST ([#8](https://github.com/temuller/hostphot/issues/8), [#9](https://github.com/temuller/hostphot/issues/9), [#10](https://github.com/temuller/hostphot/issues/10), [#11](https://github.com/temuller/hostphot/issues/11) - thanks Lisa!).
+
 v3.0.0
-* Major internal restructuring of the code
-    * Convert the cutouts into a module with each survey as a different submodule
-    * Storage of input parameters (with timestamps)
-    * Storage of output parameters in a more friendly format
-    * Restructuring of output files
-* Fix the VISTA SCIENCE ARCHIVE url for downloading VISTA images
+
+* Major internal restructuring of the code:
+  * Convert the cutouts into a module with each survey as a different submodule
+  * Storage of input parameters (with timestamps)
+  * Storage of output parameters in a more friendly format
+  * Restructuring of output files
+* Fix the VISTA SCIENCE ARCHIVE url for downloading VISTA images.
 * Fix issue with VISTA image size when downloading cuouts
 * The download of VISTA images now automatically chooses the survey with overlap for the given coordinates, prioritising the survey with deeper coverage.
 * The user experience is kept approximately consistent with previous versions, but there are some changes
 * Download of HST images is now working!
 * Adding download of JWST images!
+
 ___
 v2.10.0
+
 * Adding JWST! (few NIRCam filters for now - thanks to Lisa Kelsey for this addition)
 * SkyMapper upgrade from DR2 to DR4 (note that there is still a problem with the photometric calibration of extended sources)
 * Fix HST functions, which includes working with Drizzlepac images
@@ -189,11 +200,13 @@ v2.10.0
 * Close some fits images that were previously kept opened
 
 v2.9.0
+
 * Fixing issue when applying/propagating common-mask parameters (e.g. from a coadd) to other images
 * Updated error propagation for Legacy Survey with inverse-variance maps (using it as a weight mask now)
 * Bumping up the DR version of VIKING and VIDEO for VISTA
 
 v2.8.0
+
 * Fix SED plotting with negative or too large values/errors, and improve plotting aesthetics
 * Fix problem downloading some 2MASS images giving internal server error (no image overlap)
 * Adding UKIDSS, SPLUS and SkyMapper surveys. WARNINGS: SkyMapper photometry (DR2) is not very accurate with extended sources!
@@ -208,31 +221,38 @@ v2.8.0
 * Kron radius is no longer "optimized", just the scale, which doesn't have any effect in practice for the user
 
 v2.7.1
+
 * Adding SED plotting
-* Fixed plotting angle of masked object 
+* Fixed plotting angle of masked object
 
 v2.7.0
+
 * MAJOR BUG: Convertion factor between different pixel scales fixed (it was previously inverted). This affects the common aperture photometry between different surveys!
 * Moving font family to `_constant.py` file
 
 v2.6.2
+
 * Prettier plots (now using `aplpy`), but also more informative!
-* Scale of the apertures for the masks is now a parameter (`r`) 
+* Scale of the apertures for the masks is now a parameter (`r`)
 * Raise exception is now `True` by default when calculating photometry
 
 v2.6.1
+
 * 2MASS cutouts improved (picking largest image)
 
 v2.6.0:
+
 * HST (WFC3 only - pseudo-trial) included
 * 2MASS cutouts fixed (it now downloads the image closest to the given coordinates)
 
 v2.5.1:
+
 * Using sfdmap2 instead of sfdmap to avoid issues with numpy version (requires Python>=3.9)
 
 v2.5.0:
+
 * Systematic error floor added to PS1 photometry
-* Added missing uncertainties in the error budget of DES (~5 mmag) 
+* Added missing uncertainties in the error budget of DES (~5 mmag)
 * Flux/counts have been added to output photometry
 * GALEX now downloads images with largest exposure time by default
 * Fixed image realignment/orientation between different surveys when using common apertures (for masking and global photometry)
@@ -244,15 +264,16 @@ v2.5.0:
 * Other minor bugs fixed
 
 v2.4.0:
+
 * Fixed and improved GALEX cutouts
 * Better galaxy identification
 * Better output plots for masked images
 * Option to return NaN or raise an exception when getting photometry
 
 v2.3.2:
-* Improved download of GALEX images (download even if they seem to be just background noise)
 
+* Improved download of GALEX images (download even if they seem to be just background noise)
 
 ## Acknowledgements
 
-I thank Yuchen LIU for helping me adding Spitzer as part of AstroHackWeek 2022.
+I thank Yuchen LIU for helping me adding Spitzer as part of AstroHackWeek 2022. I also thank Lisa Kelsay for helping add HST and JWST.

src/hostphot/_version.py

@@ -1,3 +1,3 @@
 # -*- coding: utf-8 -*-
 
-__version__ = "3.0.0rc1"
+__version__ = "3.1.0"

src/hostphot/cutouts/des.py

@@ -6,6 +6,7 @@
 from astropy.io import fits
 from astropy import units as u
 
+from hostphot.utils import open_fits_from_url
 from hostphot.surveys_utils import get_survey_filters, check_filters_validity
 
 
@@ -30,7 +31,7 @@ def get_DES_urls(ra: float, dec: float, fov: float, filters: str="grizY") -> tup
         filters = get_survey_filters("DES")
     check_filters_validity(filters, "DES")
     # get field-of-view table
-    des_access_url = "https://datalab.noirlab.edu/sia/des_dr1"
+    des_access_url = "https://datalab.noirlab.edu/sia/des_dr2"
     svc = sia.SIAService(des_access_url)
     imgs_table = svc.search(
         (ra, dec), (fov / np.cos(dec * np.pi / 180), fov), verbosity=2
@@ -101,12 +102,15 @@ def get_DES_images(ra: float, dec: float, size: float | u.Quantity = 3, filters:
     hdu_list = []
     for url, url_w in zip(url_list, url_w_list):
         # combine image+weights on a single fits file
-        image_hdu = fits.open(url)
+        #image_hdu = fits.open(url, timeout=120)
+        image_hdu = open_fits_from_url(url)
         hdu = fits.PrimaryHDU(image_hdu[0].data, header=image_hdu[0].header)
         if url_w is None:
             hdu_sublist = fits.HDUList([hdu])
         else:
-            weight_hdu = fits.open(url_w)
+            print(url_w)
+            #weight_hdu = fits.open(url_w, timeout=120)
+            weight_hdu = open_fits_from_url(url_w)
             hdu_err = fits.ImageHDU(
                 weight_hdu[0].data, header=weight_hdu[0].header
             )

src/hostphot/cutouts/galex.py

@@ -8,6 +8,7 @@
 from astropy.coordinates import SkyCoord
 from astroquery.mast import Observations
 
+from hostphot.utils import open_fits_from_url
 from hostphot.surveys_utils import get_survey_filters, check_filters_validity, survey_pixel_scale
 
 import warnings
@@ -92,7 +93,7 @@ def get_GALEX_images(ra: float, dec: float, size: float | u.Quantity = 3,
             hdu_list = []
             for file in files:
                 try:
-                    hdu = fits.open(file)
+                    hdu = open_fits_from_url(file)
                     hdu_list.append(hdu)
                 except:
                     pass

src/hostphot/cutouts/legacysurvey.py

@@ -2,6 +2,7 @@
 from astropy.io import fits
 from typing import Optional
 
+from hostphot.utils import open_fits_from_url
 from hostphot.surveys_utils import get_survey_filters, check_filters_validity, survey_pixel_scale
 
 def get_LegacySurvey_images(ra: float, dec: float, size: float | u.Quantity = 3, 
@@ -42,7 +43,7 @@ def get_LegacySurvey_images(ra: float, dec: float, size: float | u.Quantity = 3,
     params = f"ra={ra}&dec={dec}&layer=ls-{version}&pixscale={pixel_scale}&bands={filters}&size={size_pixels}&invvar"
     url = base_url + params
 
-    master_hdu = fits.open(url)
+    master_hdu = open_fits_from_url(url)
     master_header = master_hdu[0].header
     master_header_invvar = master_hdu[1].header
 

src/hostphot/cutouts/panstarrs.py

@@ -5,6 +5,7 @@
 from astropy import units as u
 from astropy.table import Table
 
+from hostphot.utils import open_fits_from_url
 from hostphot.surveys_utils import get_survey_filters, check_filters_validity, survey_pixel_scale
 
 def query_ps1(ra: float, dec: float, size: float | u.Quantity = 3, filters: Optional[str] = None) -> Table:
@@ -111,6 +112,6 @@ def get_PanSTARRS_images(ra: float, dec: float, size: float | u.Quantity = 3, fi
     # download FITS images
     hdu_list = []
     for url, filt in zip(fits_url, filters):
-        hdu = fits.open(url)
+        hdu = open_fits_from_url(url)
         hdu_list.append(hdu)
     return hdu_list

src/hostphot/cutouts/skymapper.py

@@ -5,6 +5,7 @@
 from astropy.io import fits
 
 from pyvo.dal import sia
+from hostphot.utils import open_fits_from_url
 from hostphot.surveys_utils import get_survey_filters, check_filters_validity
 
 def get_SkyMapper_urls(ra: float, dec: float, fov: float | u.Quantity = 3, 
@@ -86,7 +87,7 @@ def get_SkyMapper_images(ra: float, dec: float, size: float | u.Quantity = 3,
         if url is None:
             hdu_list.append(None)
         else:
-            hdu = fits.open(url)
+            hdu = open_fits_from_url(url)
             # add zeropoint
             hdu[0].header['MAGZP'] = hdu[0].header['ZPAPPROX']
             hdu_list.append(hdu)

src/hostphot/cutouts/spitzer.py

@@ -4,6 +4,7 @@
 import astropy.units as u
 from astropy.io import fits
 
+from hostphot.utils import open_fits_from_url
 from hostphot.surveys_utils import get_survey_filters, check_filters_validity, survey_pixel_scale
 
 def get_Spitzer_images(ra: float, dec: float, size: float | u.Quantity = 3, 
@@ -80,7 +81,7 @@ def get_Spitzer_images(ra: float, dec: float, size: float | u.Quantity = 3,
     hdu_list = []
     for filt, file in files_dict.items():
         if file is not None:
-            hdu = fits.open(file)
+            hdu = open_fits_from_url(file)
             hdu[0].header["MAGZP"] = hdu[0].header["ZPAB"]
             hdu_list.append(hdu)
         else:

src/hostphot/cutouts/splus.py

@@ -9,6 +9,7 @@
 
 import hostphot
 hostphot_path = Path(hostphot.__path__[0])
+from hostphot.utils import open_fits_from_url
 from hostphot.surveys_utils import get_survey_filters, check_filters_validity
 
 def get_SPLUS_urls(ra: float, dec: float, fov: float | u.Quantity = 3, 
@@ -88,7 +89,7 @@ def get_SPLUS_images(ra: float, dec: float, size: float | u.Quantity = 3,
         if url is None:
             hdu_list.append(None)
         else:
-            hdu = fits.open(url)
+            hdu = open_fits_from_url(url)
             # add zeropoint
             # file from https://splus.cloud/documentation/dr2_3
             zps_file = hostphot_path.joinpath('filters', 'SPLUS', 'iDR3_zps.cat')

src/hostphot/cutouts/twomass.py

@@ -9,6 +9,7 @@
 from astropy.nddata import Cutout2D
 from astropy.coordinates import SkyCoord
 
+from hostphot.utils import open_fits_from_url
 from hostphot.surveys_utils import get_survey_filters, check_filters_validity, survey_pixel_scale
 
 
@@ -63,7 +64,7 @@ def get_2MASS_images(
         for line in response.content.decode("utf-8").split("<TD><![CDATA["):
             if re.match(f"https://irsa.*{filter.lower()}i.*fits", line.split("]]>")[0]):
                 fitsurl = line.split("]]")[0]
-                fits_image = fits.open(fitsurl, cache=None)
+                fits_image = open_fits_from_url(fitsurl)
                 wcs = WCS(fits_image[0].header)
                 if coords.contained_by(wcs):
                     break