LSQ Lag Times

The Problem

We need to minimise time between a SN explosion and the PESSTO classification of the SN. We’ve minimised delays on the PESSTO side of the timeline (the easy bit!), but we need to help LSQ reduced their time delays if possible.

Timeline from Explosion to Classification

lsq timeline
lsq timeline
The 2 time blocks highlighted in pink are:
  1. Lag time between viable detections being added to the LSSNDB candidates table and human screening of those objects to find the ‘good’ LSQ targets.
  2. Lag time between a ‘good’ target being screened, forced photometry being performed on the target and data added to the LSSNDB ms_lightcurves table.

Components of Problem

1. Lag between LSQ ‘Discovery’ and Human Screening

I’ve done some digging around in the data to try and quantify this lag time. In the plot below I define ‘discovery’ as the earilest JD in the LSSNDB candidates table for a given object. The screening date comes from the screening_date in the LSSNDB summary table (translated to UT).
Screenshot 2015-06-03 15.14.11
Screenshot 2015–06–03 15.14.11
If this plot is correct, then you can see that ~30% of the discovered objects are screened within the first 24 hrs, and the remaining ~70% within around 3 days. We are guessing the 2 bottlenecks here are:
  • Processing the data at NERSC, and
  • Eyeballing all of the potential candidates
I also tried to work out how many candidates you typically have to eyeball from a single night’s observing, but I think I’ve probably got this wrong. This is data taken from the candidates table - so these are probably the good targets you select from some larger list of potential candidates?
Screenshot 2015-06-03 15.30.02
Screenshot 2015–06–03 15.30.02

2. Lag between Human Target Selection and Forced Photometry

From my estimates below you seem to be doing pretty well at getting the forced photometry done quickly after a taret is screened as a ‘good’ target. Our efforts are therefore probably best placed at trying to reduces the discovery to screening lag rather than here.
Screenshot 2015-06-03 15.39.52
Screenshot 2015–06–03 15.39.52
Some questions for the LSQ guys -
  • Is the data streamed to NERSC for processing throughout the La Silla night, or is it a bulk dump at the end of the night?
  • How many candidates do you typically have to screen from a single night’s data?
  • Can PESSTO help with the screening of candidates to reduce the load on you guys and speed up the eyeballing process?
  • How CPU intensive is the forced photometry? Would it be impossible to perform the force photometry on all LSQ candidates BEFORE they pass human screening? This should help find good targets sooner.

ESO Fits Binary Table Format

  • The spectral axis of spectral data are calibrated to proper physical units - erg/cm2/s/A.
  • The wavelength axis has monotonically increasing steps from TDMIN to TDMAX.
  • Each FITS file contains weighted science spectrum, flux error spectrum, unweighted science spectrum, sky-background spectrum.
  • The spectrum binary table format is made of:
    1. one primary header (no data in the primary HDU, that is NAXIS=0)
    2. one single extension with:
      • one header unit and
      • one BINTABLE with NAXIS=2.

Primary Header

  • Here are the mandatory primary header keywords included
    SIMPLE  =                    T / Standard FITS format (NOST-100-2.0) 
    BITPIX  =                   %d / Number of bits per data pixel 
    NAXIS   =                    0 / Number of data axes 
    EXTEND  =                    T / Extensions may be present  
    ORIGIN  = 'ESO'                / European Southern Observatory                   
    DATE    =                   %s / Date this file was written 
    TELESCOP=                   %s / ESO Telescope designation 
    INSTRUME=                   %s / Instrument name 
    DISPELEM=                   %s / Dispersive element name 
    SPECSYS =                   %s / Reference frame for spectral coordinates 
    OBJECT  =                   %s / Target designation 
    EXT_OBJ =                   %c / TRUE if extended 
    RA      =                   %f / [deg] Spectroscopic target position (J2000.0) 
    DEC     =                   %f / [deg] Spectroscopic target position (J2000.0) 
    EQUINOX =                 %.0f / Standard FK5 (years) 
    RADECSYS=                   %s / Coordinate reference frame 
    EXPTIME =                   %f / Total integration time per pixel (s) 
    MJD-OBS =                 %.8f / [d] Start of observations (days) 
    MJD-END =                 %.8f / [d] End of observations (days) 
    PROG_ID =                 %20s / ESO programme identification 
    OBID1   =                   %d / Observation block ID 
    M_EPOCH =                   %d / TRUE if resulting from multiple epochs 
    PROV1   =                   %s / Originating science file   
    OBSTECH =                   %s / Technique of observation        
    PRODCATG= 'SCIENCE.SPECTRUM'   / Data product category 
    FLUXCAL=                    %s / Type of flux calibration                      
    CONTNORM=                   %c / TRUE if normalised to the continuum 
    WAVELMIN=                   %s / [nm] Minimum wavelength 
    WAVELMAX=                   %s / [nm] Maximum wavelength  
    SPEC_BIN=                   %f / Wavelength bin size   
    TOT_FLUX=                   %c / TRUE if photometric conditions and all src flux is captured 
    FLUXERR =                   %f / Uncertainty in flux scale (%) 
    REFERENC=                   %s / Bibliographic reference 
    SNR     =                   %f / Average signal to noise ratio per pixel 
    SPEC_RES=                   %f / [nm] Reference spectral resolution (FWHM) 
    CHECKSUM=                   %s / HDU checksum 
    DATASUM =                   %s / Data unit checksum   
Note: the format strings %c, %d, %f, and %s correspond to the data types boolean, integer number, floating point number, and character string, respectively.

Extension BINTABLE

  • The data arrays are stored as vectors in single cells. As a consequence, there shall be only one row in the BINTABLE, that is NAXIS2=1.

Extension Header

  • To ensure compliance with the VO standard, the keywords listed here are present in the extension header.
    1. VOCLASS - The data model name and version, fixed to ‘SPECTRUM V1.0’.
    2. VOPUB - The name of the publisher, fixed ‘ESO/SAF’.
    3. TITLE - The value of the OBJECT keyword & short description of data included.
    4. APERTURE - Aperture angular size, in degrees [deg]. It shall be set to the width of the slit or the diameter of the fiber.
    5. TELAPSE - Total elapsed time in seconds [s], defined as MJD-END-MJD- OBS.
    6. TMID - Exposure midpoint (MJD). It shall be set to (MJD-OBS+MJD- END)/2.0.
    7. SPEC_VAL - Characteristic spectral coordinate value in nanometers [nm]. Should WCS information be available, it can be calculated as:
      Otherwise, it is set to:
    8. SPEC_BW - Width of the spectrum in nanometers [nm]. Should WCS information be available, it can be calculated as:
      Otherwise, it is set to:
  • Each field of the BINTABLE is further described in the extension header as specified in Table 11 of the ESO Phase III SDP Document. Mandatory fields included are WAVE, FLUX, and ERR, in that particular order.
  • Here are the mandatory extension header keywords included:
    XTENSION =BINTABLE           / FITS Extension first keyword                    
    BITPIX  =                    8 / Number of bits per data pixel                   
    NAXIS   =                    2 / Number of data axes                             
    NAXIS1  =                   %d / Length of data axis 1                           
    NAXIS2  =                   %d / Length of data axis 2
    VOCLASS = 'SPECTRUM V1.0'      / VO Data Model 
    VOPUB   = 'ESO/SAF'            / VO Publishing Authority  
    TITLE   =                   %s / Dataset title  
    OBJECT  =                   %s / Target designation 
    RA      =                   %f / [deg] Spectroscopic target position (J2000.0) 
    DEC     =                   %f / [deg] Spectroscopic target position (J2000.0) 
    APERTURE=                   %f / [deg] Aperture diameter 
    TELAPSE =                   %f / [s] Total elapsed time 
    TMID    =                   %f / [d] MJD mid exposure 
    SPEC_VAL=                   %f / [nm] Mean Wavelength 
    SPEC_BW =                   %f / [nm] Bandpass Width Wmax – Wmin 
    TFIELDS =                   %d / Number of fields in each row                    
    NELEM   =                   %d / Length of the data arrays 
    TTYPE1  =                   %s / Label for field 1                               
    TUTYP1  = 'Spectrum.Data.SpectralAxis.Value' 
    TFORM1  =                   %s / Data format of field1                           
    TUNIT1  =                   %s / Physical unit of field1                         
    TUCD1   =                   %s / UCD of field 1 
    TDMIN1  =                   %f / Start in spectral coord. 
    TDMAX1  =                   %f / Stop in spectral coord. 
    TTYPE2  = 'FLUX'               / Label for field 2
    TUTYP2  = 'Spectrum.Data.FluxAxis.Value' 
    TFORM2  =                   %s / Data format of field 2                          
    TUNIT2  =                   %s / Physical unit of field 2                        
    TUCD2   =                   %s / UCD of field 2   
    TTYPE3  = 'ERR'                / Label for field 3                               
    TUTYP3  = 'Spectrum.Data.FluxAxis.Accuracy.StatError' 
    TFORM3  =                   %s / Data format of field 3                          
    TUNIT3  =                   %s / Physical unit of field 3                        
    TUCD3   =                   %s / UCD of field 3 
    EXTNAME =                   %s / FITS Extension name                             
    INHERIT =                    T / Primary header keywords are inherited           
    CHECKSUM=                   %s / HDU checksum                                    
    DATASUM =                   %s / Data unit checksum