How to ... parse data from the Transient Name Server webpages

How to ... parse data from the Transient Name Server webpages

The Transient Name Server provides a search page, which lists all of the transients currently in the name server. The URL of this page can be manipulated either through the HTML form provided or manually via the URL request parameters to return a filtered set of transients.

Here’s the search URL and the parameters we can use to search the TNS:

url="http://wis-tns.weizmann.ac.il/search",
params={
   "page": "1",
   "name": "",
   "name_like": "0",
   "isTNS_AT": "all",
   "public": "all",
   "unclassified_at": "0",
   "classified_sne": "0",
   "ra": "",
   "decl": "",
   "radius": "",
   "coords_unit": "arcsec",
   "groupid[]": "null",
   "type[]": "null",
   "discoverer": "",
   "date_start[date]": "",
   "date_end[date]": "",
   "discovery_mag_min": "",
   "discovery_mag_max": "",
   "redshift_min": "",
   "redshift_max": "",
   "spectra_count": "",
   "associated_groups[]": "null",
   "display[redshift]": "1",
   "display[hostname]": "1",
   "display[host_redshift]": "1",
   "display[source_group_name]": "1",
   "display[programs_name]": "1",
   "display[isTNS_AT]": "0",
   "display[public]": "1",
   "display[spectra_count]": "1",
   "display[discoverymag]": "1",
   "display[discmagfilter]": "1",
   "display[discoverydate]": "1",
   "display[discoverer]": "1",
   "display[sources]": "0",
   "display[bibcode]": "0",
   "num_page": "40",
   "edit[type]": "",
   "edit[objname]": "",
   "edit[id]": "",
   "sort": "desc",
   "order": "discoverydate"
}

To parse the TNS data in a sensible and efficient manner I want to:

  • display all columns in the table by setting all display[XXX] parameters equal to 1.
  • order the results by discovery date by setting "order": "discoverydate"
  • display 1000 results at a time ("num_page": "1000") and paginate through them with the page parameter
  • set a lower limit to the discovery date and do not parse before that date. Currently I select everything in the last two years by setting "date_start[date]": "2014-01-28" and setting "date_end[date]": "2100-01-01" (i.e. some arbitrary time in the future).

Here’s a script you can use to parse the name-server. Please use with care and don’t overload the name-server by hitting it more than a few times an hour with this script.

Automation

The PESSTO Marshall’s TNS importer runs once an hour. Once the importer has run the Marshall’s name updater script runs to make any (a) SN designation or (b) AT name the master name for the object.