Accessing HEASARC data on the ODR¶

A brief tutorial on how to access HEASARC and LAMBDA data available through the Amazon AWS Open Data Registry.¶

In [1]:
#  Set up some useful astronomy libraries:
import pyvo as vo, os, astropy
import astropy.coordinates as coord
import matplotlib.pyplot as plt
%matplotlib inline  
from astropy.io import fits
import astropy.coordinates as coord

#  Set up AWS access with the boto3 and associated libraries:  
import boto3
from botocore import UNSIGNED
from botocore.client import Config
s3_client = boto3.client("s3", config=Config(signature_version=UNSIGNED))

import warnings
# There are a number of relatively unimportant warnings that show up, so for now, suppress them:
warnings.filterwarnings("ignore", module="astropy.io.votable.*")
warnings.filterwarnings("ignore", module="astropy.units.format.vounit")
warnings.filterwarnings("ignore", module="pyvo.utils.xml.*")

A simple HEASARC search for Chandra imaging data¶

The first step is to find the HEASARC service that you can use to search for Chandra images:

In [2]:
chandra_services = vo.regsearch(servicetype='image',keywords=['chandra heasarc'])
chandra_services.to_table()['ivoid','short_name','res_title']
Out[2]:
Table length=1
ivoidshort_nameres_title
objectobjectobject
ivo://nasa.heasarc/chanmasterChandraChandra Observations

Next, use Astropy to get the coordinates of your favorite source, in this case the Cas A supernova remnant. Then query the service for matching images.

In [3]:
coords = coord.SkyCoord.from_name("cas a")

im_table = chandra_services[0].search(pos=coords,size=0.1)
im_table.to_table()[0:5]
Out[3]:
Table length=5
obsidstatusnameradectimedetectorgratingexposuretypepipublic_datedatalinkt_mint_resolutiont_maxt_exptimeem_res_powers_regions_ras_decs_resolutionaccess_estsizes_fovo_ucdaccess_urlobs_publisher_didobs_idobs_collectiontarget_nameinstrument_namefacility_namepol_statescalib_levelaccess_formatdataproduct_typeem_minem_maxSIA_titleSIA_scaleSIA_naxisSIA_naxesSIA_formatSIA_referenceSIA_raSIA_decSIA_instrument
degreedegreemjdsmjddsdsdegdegdegarcseckbytedegmm
objectobjectobjectfloat64float64float64objectobjectfloat64objectobjectint32objectfloat64float64float64float64float64objectfloat64float64float32int32float64objectobjectobjectobjectobjectobjectobjectobjectobjectint32objectobjectfloat64float64objectobjectobjectint32objectobjectfloat64float64object
1038archivedCas A350.8660458.8116952171.2069HRC-SNONE50400GTOMurray5254523379:chandra.obs.img52171.2068981481--102571.206898148150400.0--350.8660458.81169--432185--https://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/8/1038/primary/hrcf01038N005_cntr_img2.fits.gzHEASARC1038CHANDRA HRC-SCas AHRC-SChandra3image/fitsImage1.24e-101.24e-08hrcf01038N005_cntr_img2.fits[-0.0013666666666667 0.0013666666666667][1024 1025]2image/fitshttps://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/8/1038/primary/hrcf01038N005_cntr_img2.fits.gz350.8660458.81169CHANDRA HRC-S
1038archivedCas A350.8660458.8116952171.2069HRC-SNONE50400GTOMurray5254523379:chandra.obs.img52171.2068981481--102571.206898148150400.0--350.8660458.81169--945041--https://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/8/1038/primary/hrcf01038N005_cntr_img2.jpgHEASARC1038CHANDRA HRC-SCas AHRC-SChandra3image/jpegImage1.24e-101.24e-08hrcf01038N005_cntr_img2.jpg[-0.0013666666666667 0.0013666666666667][1024 1025]2image/jpeghttps://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/8/1038/primary/hrcf01038N005_cntr_img2.jpg350.8660458.81169CHANDRA HRC-S
1857archivedCAS A350.8660658.8116951821.9464HRC-SNONE48880GTOMurray5221423380:chandra.obs.img51821.9463888889--100701.946388888948880.0--350.8660658.81169--426096--https://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/7/1857/primary/hrcf01857N007_cntr_img2.fits.gzHEASARC1857CHANDRA HRC-SCAS AHRC-SChandra3image/fitsImage1.24e-101.24e-08hrcf01857N007_cntr_img2.fits[-0.0013666666666667 0.0013666666666667][1024 1024]2image/fitshttps://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/7/1857/primary/hrcf01857N007_cntr_img2.fits.gz350.8660658.81169CHANDRA HRC-S
1857archivedCAS A350.8660658.8116951821.9464HRC-SNONE48880GTOMurray5221423380:chandra.obs.img51821.9463888889--100701.946388888948880.0--350.8660658.81169--959340--https://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/7/1857/primary/hrcf01857N007_cntr_img2.jpgHEASARC1857CHANDRA HRC-SCAS AHRC-SChandra3image/jpegImage1.24e-101.24e-08hrcf01857N007_cntr_img2.jpg[-0.0013666666666667 0.0013666666666667][1024 1024]2image/jpeghttps://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/7/1857/primary/hrcf01857N007_cntr_img2.jpg350.8660658.81169CHANDRA HRC-S
26655archivedCXOU J232327.8+584842350.8666758.8117859943.1372ACIS-SNONE30100GOPosselt6014123381:chandra.obs.img59943.1372106482--90043.137210648230100.0--350.8666758.81178--244996--https://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/5/26655/primary/acisf26655N001_cntr_img2.fits.gzHEASARC26655CHANDRA ACIS-SCXOU J232327.8+584842ACIS-SChandra3image/fitsImage1.24e-101.24e-08acisf26655N001_cntr_img2.fits[-0.0013666666666667 0.0013666666666667][1025 1025]2image/fitshttps://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/5/26655/primary/acisf26655N001_cntr_img2.fits.gz350.8666758.81178CHANDRA ACIS-S

Select an image and fetch its URL:

In [4]:
url = im_table[0].getdataurl()
print(url)

https://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/8/1038/primary/hrcf01038N005_cntr_img2.fits.gz

Now convert that URL into an S3 bucket address and use boto3 to download the object to a local file:

In [5]:
key_name = url.replace("https://heasarc.gsfc.nasa.gov/FTP/","")
s3_client.download_file("nasa-heasarc", key_name, "chandra_casA.fits")
hdu_list = fits.open("chandra_casA.fits")

Now you're ready to analyze this Cas A Chandra image:

In [6]:
plt.imshow(hdu_list[0].data, cmap='hot', origin='lower',vmax=1)
Out[6]:
<matplotlib.image.AxesImage at 0x116058430>

Note that in astropy>=5.2, you do not have to download the file locally to read it but can give the astropy.io.fits reader the S3 bucket URI directly. An example is shown below for WMAP. Obviously you cannot write to the file on the bucket.

A simple use case for fetching WMAP data¶

The LAMBDA community generally needs full sky datasets, so browsing the products is something that can be done interactively on the LAMBDA data portal. Once a user has selected, for example, the WMAP DR5 smoothed polarization map for the K band, the portal shows the archive's URL of the dataset. To find the file on AWS is almost as straightforward as for HEASARC:

In [7]:
url = "https://lambda.gsfc.nasa.gov/data/map/dr5/skymaps/9yr/smoothed/wmap_band_smth_iqumap_r9_9yr_K_v5.fits"
##  The "map" directory on LAMBDA has been renamed "wmap" on AWS to clarify that it refers to the mission:
s3_uri = url.replace("https://lambda.gsfc.nasa.gov/data/map/","s3://nasa-lambda/wmap/")

With astropy>=5.2, you can read the file directly out of the bucket. You must also install fsspec and s3fs. (Astropy will tell you this.) With these libraries, you can open the FITS file and even extract subsets of the data. For this case, we will just visualize the map of one of the columns:

In [8]:
try:
    import fsspec, s3fs
except:
    print("Please install fsspec and s3fs in order to use the cell below.")
    
if tuple(map(int, (astropy.__version__.split(".")))) < tuple(map(int, ("5.2".split(".")))):
    print(f"You have astropoy v{astropy.__version__}. Please update astropy to >= 5.2")
In [9]:
hdul = fits.open(s3_uri, use_fsspec=True, fsspec_kwargs={"anon": True})
In [10]:
import healpy
healpy.mollview(hdul[1].data['Q_POLARISATION'],nest=True,min=-0.1,max=0.1)
In [ ]: