Data Reduction Pipeline for Imaging Data

Overview

SWIMS Imaging Data Reduction Pipeline, written in Python, processes SWSB*/SWSR* FITS data to produce a final stacked image in a standard manner.

Reduction Flow Chart
flowchart.png

Download

Core scripts

FileContents
fileswsred_20210517a.tgzPython core scripts (swsred/*py) and configuration/utility files

BPM

May 26, 2021: BPM file on B1 detector for S21A April is found to be incorrect. Please download newer tar ball created on May 26.

FileContents
fileswsred_calib_bpm_s18a_20210131.tgzBad pixel mask files for S18A data
fileswsred_calib_bpm_s18b_20210131.tgzBad pixel mask files for S18B data
fileswsred_calib_bpm_s20b_20210131.tgzBad pixel mask files for S20B data
fileswsred_calib_bpm_s21a1_20210526.tgzBad pixel mask files for S21A April data
fileswsred_calib_bpm_s21a2_20210526.tgzBad pixel mask files for S21A May data

Mosaic

FileContents
fileswsred_calib_mosaic_s18a_20210131.tgzMosaic parameter files for S18A data
fileswsred_calib_mosaic_s18b_20210131.tgzMosaic parameter files for S18B data
fileswsred_calib_mosaic_s21a_20210413.tgzTentative use only!! Mosaic parameter files for S21A data

Flat

BBF: Broad-band filters, MBF: Medium-band filters, NBF: Narrow-band filters

FileContents
fileswsred_calib_flat_bbf_s18a_20210131.tgzFlat files for S18A BBF data
fileswsred_calib_flat_bbf_s18b_20210131.tgzFlat files for S18B BBF data
fileswsred_calib_flat_mbf_s18b_20210131.tgzFlat files for S18B MBF data
fileswsred_calib_flat_nbf_blue_s18b_20210131.tgzFlat files for S18B Blue NBF data
fileswsred_calib_flat_nbf_red_s18b_20210131.tgzFlat files for S18B Red NBF data
fileswsred_calib_flat_bbf_s20b_20210131.tgzFlat files for S20B BBF data
fileswsred_calib_flat_bbf_s21a_20210418.tgzFlat files for S21A BBF data
fileswsred_calib_flat_mbf_s21a_20210418.tgzFlat files for S21A MBF data

Individual files.

S21A (May)S21A (April)S18B
Yflat_s21a2_y_b1.fits
flat_s21a2_y_b2.fits
flat_s21a1_y_b1.fits
flat_s21a1_y_b2.fits
flat_s18b_y_b1.fits
flat_s18b_y_b2.fits
Jflat_s21a2_j_b1.fits
flat_s21a2_j_b2.fits
flat_s21a1_j_b1.fits
flat_s21a1_j_b2.fits
--
--
J1flat_s21a2_j1_b1.fits
flat_s21a2_j1_b2.fits
flat_s21a1_j1_b1.fits
flat_s21a1_j1_b2.fits
flat_s18b_j1_b1.fits
flat_s18b_j1_b2.fits
J2flat_s21a2_j2_b1.fits
flat_s21a2_j2_b2.fits
flat_s21a1_j2_b1.fits
flat_s21a1_j2_b2.fits
flat_s18b_j2_b1.fits
flat_s18b_j2_b2.fits
NB1244--
--
--
--
flat_s18b_nb1244_b1.fits
flat_s18b_nb1244_b2.fits
NB1261--
--
--
--
flat_s18b_nb1261_b1.fits
flat_s18b_nb1261_b2.fits
NB1292--
--
--
--
flat_s18b_nb1292_b1.fits
flat_s18b_nb1292_b2.fits
NB1326--
--
--
--
flat_s18b_nb1326_b1.fits
flat_s18b_nb1326_b2.fits
Hflat_s21a2_h_r1.fits
flat_s21a2_h_r2.fits
flat_s21a1_h_r1.fits
flat_s21a1_h_r2.fits
--
--
KSflat_s21a2_ks_r1.fits
flat_s21a2_ks_r2.fits
flat_s21a1_ks_r1.fits
flat_s21a1_ks_r2.fits
--
--
H1flat_s21a2_h1_r1.fits
flat_s21a2_h1_r2.fits
--
--
flat_s18b_h1_r1.fits
flat_s18b_h1_r2.fits
H2flat_s21a2_h2_r1.fits
flat_s21a2_h2_r2.fits
--
--
flat_s18b_h2_r1.fits
flat_s18b_h2_r2.fits
H3flat_s21a2_h3_r1.fits
flat_s21a2_h3_r2.fits
--
--
flat_s18b_h3_r1.fits
flat_s18b_h3_r2.fits
K1flat_s21a2_k1_r1.fits
flat_s21a2_k1_r2.fits
flat_s21a1_k1_r1.fits
flat_s21a1_k1_r2.fits
flat_s18b_k1_r1.fits
flat_s18b_k1_r2.fits
K2flat_s21a2_k2_r1.fits
flat_s21a2_k2_r2.fits
flat_s21a1_k2_r1.fits
flat_s21a1_k2_r2.fits
flat_s18b_k2_r1.fits
flat_s18b_k2_r2.fits
K3flat_s21a2_k3_r1.fits
flat_s21a2_k3_r2.fits
flat_s21a1_k3_r1.fits
flat_s21a1_k3_r2.fits
flat_s18b_k3_r1.fits
flat_s18b_k3_r2.fits
NB1630--
--
--
--
flat_s18b_nb1630_r1.fits
flat_s18b_nb1630_r2.fits
NB1653--
--
--
--
flat_s18b_nb1653_r1.fits
flat_s18b_nb1653_r2.fits
NB1875--
--
--
--
flat_s18b_nb1875_r1.fits
flat_s18b_nb1875_r2.fits
NB1945--
--
--
--
flat_s18b_nb1945_r1.fits
flat_s18b_nb1945_r2.fits
NB2137--
--
--
--
flat_s18b_nb2137_r1.fits
flat_s18b_nb2137_r2.fits
NB2167--
--
--
--
flat_s18b_nb2167_r1.fits
flat_s18b_nb2167_r2.fits

Sample data

Use these test data (taken in S18B) to check if your swsred works.

FileContents
fileswsred_sample_b.tgzJ1-band data (4 frames x 2 arrays)
fileswsred_sample_r.tgzH1-band data (4 frames x 2 arrays)

The reduced images produced with two iterations (n_iter=2) and the execution log (swsred.log) should be like these.

FileContents
filefs12_j1_b1_stack1.fitsStacked image (after two iterations) on the blue array #1
filefs12_j1_b2_stack1.fitsStacked image (after two iterations) on the blue array #2
filefs12_h1_r1_stack1.fitsStacked image (after two iterations) on the red array #1
filefs12_h1_r2_stack1.fitsStacked image (after two iterations) on the redarray #2
filefs12_j1.fitsMosaicked image (after two iterations) on the blue arrays
Mosaicking parameters still under verification.
Use this image (and mosaicking methods) only to check if swsred works correcctly.
filefs12_h1.fitsMosaicked image (after two iterations) on the red arrays
Mosaicking parameters still under verification.
Use this image (and mosaicking methods) only to check if swsred works correctly.
fileswsred_H1.loglogs on reducing H1 data.

Installation

prerequisites

SWSRED requires the following python packages and astrOmatic softwares.

PackageTested version
Python2.7.173.7.7
astropy2.0.94.0
numpy1.16.61.18.1
scipy1.2.11.4.1
matplotlib2.2.33.1.3
sextractor2.19.52.19.5
scamp2.7.82.7.8
swarp2.38.02.38.0

Also, Internet connection is required during reducing data to access Pan-STARRS/2MASS catalogs.

Install SWSRED

  1. Download SWSRED files you need.
  2. Extract all the tar balls on a directory you want.
    % cd SWSRED_DIRECTORY
    % tar xvf swsred_20210414c.tgz
    % tar xvf swsred_calib_bpm_s21a_20210418.tgz
    % tar xvf swsred_calib_mosaic_s21a_20210413.tgz
    % tar xvf swsred_calib_flat_bbf_s21a_20210418.tgz
    % tar xvf swsred_calib_flat_mbf_s21a_20210418.tgz
    
  3. set PYTHONPATH to SWSRED_DIRECTORY.
    % export PYTHONPATH="SWSRED_DIRECTORY:${PYTHONPATH}
    

Usage

SWSRED has no documentations yet, but you can use a dirty script 'swsred/reduce_all.py' as a quick tutorial to learn how to use the software.

Be sure to use SWSRED within python interpreter (e.g., in ipython or jupyter, etc.), not via the command line as no command-line interfaces prepared.

Here is a quick introduction to use SWSRED.

  1. First of all, prepare a list of input FITS files including absolute paths as a python 'array-like' (list, tuple, or numpy array) object, for example as follows.
    > import glob
    > in_arr1 = glob.glob("DATA/PATH/SWSB*1.fits")  # blue chip1 data 
    > in_arr1.sort()  # must be sorted before using.
    
    • Note that as shown in the flow chart above, data of chip1 and chip2 should be reduced separately (before stacking).
    • Data taken within the same dithering set are grouped by '2_DSETID' header keyword. Using this, you can make a input list like follows,
      import glob
      import astropy.io.fits
      
      dct1 = {}
      dct2 = {}
      
      infiles = glob.glob("SWS*fits")
      infiles.sort()
      
      for infile in infiles:
          hdr = astropy.io.fits.getheader(infile)
      
          ch = hdr["2_DETCH"]
          if ch == 1:
              dct = dct1
          else:
              dct = dct2
      
          if hdr["2_DSETID"] not in dct.keys():
              dct[hdr["2_DSETID"]] = []
      
          dct[hdr["2_DSETID"]].append(infile)
      
  2. Make filenames of intermediate files.
    > import swsred.make_file_lst
    > file_lst1 = swsred.make_file_lst.make_file_lst(in_arr1)
    
    • 'file_lst1' is a dictionary which defines various filenames such as flat-fielded FITS filename. You can edit any values as you want.
      • For example, you get the following result with an input ["data/path/SWSB00000001.fits"] as in_arr1,
        KeyValueDescription
        indata/path/SWSB00000001.fitsinput filename
        ffffSWSB00000001.fitsfilename of flat-fielded data
        msmsSWSB00000001.fitsfilename of self-sky data
        ssssSWSB00000001.fitsfilename of sky-subtracted data
        wcwcSWSB00000001.fitsfilename of WCS-corrected data
        bpm/SWSRED_DIRECTORY/calib/bpm_sXXY_b1.fitsfilename of Bad-pixel mask (XXY denotes a semester)
        whtSWSB00000001_wht.fitsfilename of weight map (created by make_weight_map module)
        omskSWSB00000001_omsk.fitsfilename of object mask
        ldacssSWSB00000001.ldacfilename of SExtractor LDAC catalog
        asciissSWSB00000001.asciifilename of SExtractor ASCII catalog
        headssSWSB00000001.headfilename of SExtractor HEAD file
  1. Then, start your data reduction. For example, to perform flat-fielding,
    > import swsred.flat_field
    > swsred.flat_field.flat_field(file_lst1["in"], file_lst1["ff"])
    
    • By default, the values 'file_lst1["ff"]' do not have a path which means that 'file_lst1["ff"]' files will be created on the current directory. If you want to specify any directory to put intermediate files, edit and add your path in 'file_lst1'.

Refer to 'swsred/reduce_all.py' for all the procedures.

Questions & Answers

Known bugs

Changes

Version 2021.01.31

Contact

Please send your bug reports or questions to konishi__at__ioa.s.u-tokyo.ac.jp.


トップ   新規 一覧 検索 最終更新   ヘルプ   最終更新のRSS