Setting up the calibration run

The EHT-HOPS pipeline is driven by a series of shell scripts that execute the various stages of the calibration process step-by-step. Sample driver scripts for running the entire pipeline are provided in the scripts directory of the EHT-HOPS repository.

Setting up the repository

Clone the public git repository for the EHT-HOPS pipeline. All the calibrated output files will be created within this directory:

cd /home/user/calibration
git clone https://github.com/sao-eht/ehthops.git
cd ehthops
git submodule update --init --remote

The last line above ensures that the submodules containing metadata and summary notebooks are updated to the latest version.

The code repository consists of four directories named hops-bx where x stands for the EHT “zoom” frequency band. Conventionally, in order of increasing frequency, the bands are named 1, 2, 3, and 4.

Note

The 2017 campaign has only two bands named “lo” and “hi”. The shell scripts driving the calibration are aware of this and will make the appropriate substitutions for 2017 data, mapping “lo” -> “b3” and “hi” -> “b4”.

The main pipeline script ehthops/ehthops_pipeline.sh takes the filename of a configuration file as argument and proceeds to run the calibration. A sample configuration file ehthops/settings.config shows all the keywords that can be set to control the calibration process. The user can create a copy of this file and modify it as needed. ehthops_pipeline.sh must be run from within the hops-bx directories.

For this tutorial, we will use the “lo” band, so we will copy the driver script to hops-b3:

cd hops-b3/
cp ../ehthops_pipeline.sh .
cp ../settings.config .

Note

The script scripts/cleanup.sh helps to quickly delete all output files created during calibration, leaving the hops-bx directories in a clean state. Copy this to the hops-b3 directory and run source cleanup.sh to clean up the directory when necessary.

Updating the configuration file for calibration

The sample configuration file ehthops/settings.config contains the following keywords:

  • SET_SRCDIR: Base directory containing the Mk4 data to be processed.

  • SET_CORRDAT: List of paths (or simple directory names such as correlator data tags/releases) relative to SRCDIR separated by ‘:’.

  • SET_METADIR: Directory where campaign metadata are to be found. Normally found under ehthops/ehthops/meta.

  • SET_EHTIMPATH: Path to eht-imaging source code.

  • stages: Stages to run (i.e. directory names found under hops-bx) as a space-separated string.

  • SET_YEAR: 4-letter code representing year of observation.

  • SET_MIXEDPOL: Boolean value. Set this to true to request mixedpol calibration. This will assume that all ALMA data are in linear polarization basis while the rest are in circular polarization basis.

  • SET_HAXP: Boolean value. Set this to true to indicate that ALMA linear polarization data are present in -haxp directories and must replace circularly polarized ALMA data originally linked from -hops directories. Setting this to true will automatically set MIXEDPOL=true.

  • SET_CAMPAIGN: An EAT-recognizable code; currently EHT2017, EHT2018, EHT2021, EHT2022 are supported.

For this tutorial, we will assign the following values to the keywords:

SET_SRCDIR="/home/user/calibration/data/extracted"
SET_CORRDAT="2016.1.01154.V"
SET_METADIR="/home/user/calibration/ehthops/meta/eht2017/230GHz"
SET_EHTIMPATH="/home/user/software/eht-imaging"
stages="0.bootstrap 1.+flags+wins 2.+pcal 3.+adhoc 4.+delays 5.+close 6.uvfits"
SET_YEAR="2017"
SET_MIXEDPOL=true
SET_HAXP=true
SET_CAMPAIGN="EHT2017"

More information on how to determine the values of the command-line options can be found here.

Submitting the calibration job to SLURM

A sample configuration file for submitting the job to SLURM follows (also found in scripts/ehthops_slurm.job):

#!/bin/bash
#SBATCH -c 48 # Number of cores requested
#SBATCH -t 1-00:00:00 # Runtime
#SBATCH -p blackhole # Partition
#SBATCH --mem=64G # Memory per node in MB (--mem or --mem-per-cpu)
#SBATCH -e slurm-%j.err
#SBATCH -o slurm-%j.out

# Set up env -- this may be different for different systems; the following are reasonable guidelines

# source default bash settings from user's bashrc file.
source $HOME/.bashrc

# Activate the mamba environment with the necessary packages installed.
mamba activate nseht310

# Uncomment the following line if it is not present in your $HOME/.bashrc file or has not been run until now. In this case, the
# bashrc file above contains this line, so it has been commented out. This is required to set up the HOPS environment properly.
# source /n/holylfs05/LABS/bhi/Lab/doeleman_lab/inatarajan/software/installed/hops-3.24/bin/hops.bash

# Set up HOPS environment once again with HOPS_SETUP=false (necessary to pick up all the HOPS environment variables properly).
HOPS_SETUP=false source /n/holylfs05/LABS/bhi/Lab/doeleman_lab/inatarajan/software/installed/hops-3.24/bin/hops.bash

# run script
source ehthops_pipeline.sh -c settings.config

The environment setup lines may be different for different systems. The user should modify these lines as needed. The correct python environment and HOPS setup must be activated before running the calibration script. This config file can now be submitted to SLURM with sbatch:

sbatch ehthops_slurm.job