CDS's on-the-fly HATS products
About
HATS stands for "Hierarchical Adaptive Tiling Scheme" anb is described in this IVOA Note.
It is a "format" developped by the LINCC, see their github repo.
The current structure of CDS on-the-fly HATS products is based on
the LSDB Gaia example.
This prototype will continue evolving to match the evolving LINCC-HATS format.
All CDS HATS products are generated on-the-fly from HEALPix indexed data.
Although so far the list of available product is limited, all VizieR large catalogues
can be added very quickly (matter of adding names in a list).
Data access
So far, one can use two URLs to access CDS HATS products:
Remark: the first URL is subject to change while the second should be permanent (but the default may change or vary from one catalogue to another in the futur).
So far, two query parameters can be added to those URLs (see CHANGELOG and or this LSDB/HATS issue):
- 'columns' with up to 8 coma separated column names, for column selection;
- 'filters' with a SQL WHERE like constraint, for row filtering.
Disclaimer
WARNING: This is still a proof-of-concept, in an beta version, requiring systematic testing for all added catalogues.
WARNING: Do not use as a way to harvest larges catalogues without the consent of the CDS (and possibly of original providers)!
Contact
For any question, please contact CDS or F.-X. Pineau.
Product list
Changelog
- 03/11/2025
- Add the possibility to remove ':n=xxx' in the URLs (and use n=1_000_000 by default)
- Add the upper limit 10_000_000 to n (reminder: the lower limit is 10_000)
- Add links to VizieR tables in the list of HATS products
- 15/10/2025
- Add apache rewrite rules to support row filtering with query parameter '?filters=CONSTRAINTS', with, e.g. the regular/URL encoded constraint 'Gmag>8.0 && o_Gmag>100' / 'Gmag%3E8.0+%26%26+o_Gmag%3E100'
- 13/10/2025
- Add apache rewrite rules to support column selection with query parameter '?columns=_healpix_29,X,X,X' or '?columns=X,X,X' (with up to 8 'X' column names) to parquet file URLs
- For complex column names 'X' (starting with a digit, containing minus, ...), use the '${X}' notation
- 02/09/2025
- Set row group size = 10_000 rows for smaller download time overhead and smaller memory footprint on server side (both due to row ordering to col ordering conversion)
- Remove xx_.with_border11.parquet files: collection with margins to be done!
- Rename the properties file hats.properties (and deprecated properties)
- Rename the point_map.fits file skymap.fits (and deprecate point_map.fits)
- Remove hats_coordinate_epoch from properties file
- Add hats_npix_suffix = .parquet in the properties file
- Add hats_skymap_alt_orders = 6 7 9 10 11 in the properties file
- Add hats_frame = equatorial in the properties file
- Add hats_version = v0.1 in the properties file
- Add hats_estsize in the properties file
- 22/11/2024
- Replace old keywords by newer ones in the properties file
- 21/11/2024
- Change _healpix_29 type from unsigned long to signed long
- 22/10/2024
- Add _healpix_29 column
- Remove JSON serialized VOTable metadata from parquet files
- Add the dataset dirrectory
- 30/09/2024
- 23/09/2024
- CatHAGrid (Catalogue in an Healpix Adaptative Grid) to HATS (Hierarchical Adaptive Tiling Scheme) renaming
- Add skymap.ORDER.png files
- Deprecate catalog_info.json and add properties
- 10/09/2024
- Add point_map.fits and skymap.ORDER.fits files
- Remove colums Dir and #rows in parition_info.csv (Dir was wrong with %n instead of /n)
- 04/09/2024
- Add VOTable metadata in Parquet schema (in addition to the Arrow schema)
- 14/08/2024
- Put n in the URL instead of in the query parameters
![[ICO]](/icons/blank.gif){kind=icon}
![[DIR]](/icons/folder.gif){kind=icon}