==============================
Setting up the calibration run
==============================

``ehthops`` pipeline is driven by a series of shell scripts that execute the various stages of the calibration process step-by-step.
Sample driver scripts for running the entire pipeline are provided in the ``scripts/`` directory of the EHT-HOPS repository.

.. note::
    
    This tutorial is intended only for those users who want to submit a job to run the entire calibration pipeline.
    Individual stages/steps of ``ehthops`` can also be run manually by executing only the relevant lines from the driver script
    found in ``scripts/ehthops_pipeline.sh``.

Setting up the repository
-------------------------

Clone the public git repository for the EHT-HOPS pipeline. All the calibrated output files will be created within this directory:

.. code-block:: bash

    cd /home/user/calibration
    git clone https://github.com/sao-eht/ehthops.git

The code repository consists of four directories named ``hops-bx`` where *x* stands for the EHT "zoom" frequency band.
Conventionally, in order of increasing frequency, the bands are named 1, 2, 3, and 4.

.. note::

    The 2017 campaign has only two bands named ``lo`` and ``hi``. We adopt the convention that ``b3 -> lo`` and ``b4 -> hi``.

Additionally, the repository contains the following directories:

- ``scripts``: Contains the driver scripts for running the calibration pipeline and sample configuration files.
- ``share``: ``Marimo`` noteboks that are run after each iteration of ``fourfit`` to generate summary plots of the results of the current stage.
- ``meta``: Contains metadata files for the campaigns, which are used to generate control files.

.. note::

    The ``meta`` directory is organized by campaign and frequency band, e.g., ``meta/eht2017/230GHz`` for the 230 GHz 2017 campaign.
    Note that the ``antab`` directory is not bundled with the repository and must be created and populated manually with the appropriate
    ``ANTAB`` files corresponding to the campaign, usually obtained from the metadata package generated by the EHTC.

The main pipeline script ``scripts/ehthops_pipeline.sh`` takes the filename of a configuration file as argument and proceeds to
run the calibration. A sample configuration file ``scripts/settings.config`` explains all the keywords that can be set to control the
calibration process. The user can copy and modify this file as needed. ``ehthops_pipeline.sh`` must be run from
within the ``hops-bx`` directories.

For this tutorial, we will use the "lo" band, so we will copy the driver script to ``hops-b3``:

.. code-block:: bash

    cd ehthops/ehthops/hops-b3/
    cp ../scripts/ehthops_pipeline.sh .
    cp ../scripts/settings.config .

.. note::

    The script ``scripts/cleanup.sh`` helps to quickly delete all output files created during calibration, leaving the ``hops-bx``
    directories in a clean state. Copy this to the ``hops-b3`` directory and run ``source cleanup.sh`` to clean up the 
    directory when necessary.

Updating the configuration file for calibration
-----------------------------------------------

The sample configuration file ``ehthops/settings.config`` contains the following keywords:

- ``SET_SRCDIR``: Base directory containing the Mk4 data to be processed.
- ``SET_CORRDAT``: List of paths (or simple directory names such as correlator data tags/releases) relative to SRCDIR separated by ':'.
- ``SET_METADIR``: Directory where campaign metadata are to be found. Normally found under ``ehthops/ehthops/meta``.
- ``SET_EHTIMPATH``: Path to eht-imaging source code.
- ``stages``: Stages to run (i.e. directory names found under ``hops-bx``) as a space-separated string.
- ``SET_OBSYEAR``: 4-letter code representing year of observation.
- ``SET_FILTERSTRING``: Optional substring used to filter which data directories are linked from the archive. Ignored if empty.
- ``SET_MIXEDPOL``: Boolean value. Set this to true to request mixedpol calibration. This will assume that all ALMA data are in linear polarization basis while the rest are in circular polarization basis.
- ``SET_HAXP``: Boolean value. Set this to true to indicate that ALMA linear polarization data are present in ``*-haxp`` directories and must replace circularly polarized ALMA data originally linked from ``*-hops`` directories. Setting this to true will automatically set MIXEDPOL=true.
- ``SET_CAMPAIGN``: An EAT-recognizable code; currently EHT2017, EHT2018, EHT2021, EHT2022 are supported.
- ``SET_INPUTDIR``: Input directory for post-processing stages. This is different for different stages and must be re-set for each post-processing stage. More details could be found in the sample ``settings.config`` file in the repository.

For this tutorial, we will assign the following values to the keywords:

  .. code-block:: bash

      SET_SRCDIR="/home/user/calibration/data/extracted"
      SET_CORRDAT="2016.1.01154.V"
      SET_METADIR="/home/user/calibration/ehthops/meta/eht2017/230GHz"
      SET_EHTIMPATH="/home/user/software/eht-imaging"
      stages="0.bootstrap 1.+flags+wins 2.+pcal 3.+adhoc 4.+delays 5.+close 6.uvfits"
      SET_YEAR="2017"
      SET_FILTERSTRING=""
      SET_MIXEDPOL=true
      SET_HAXP=true
      SET_CAMPAIGN="EHT2017"
      SET_INPUTDIR=""

More information on how to determine the values of the command-line options can be found :ref:`here <launching-the-pipeline>`.

Submitting the calibration job to SLURM
---------------------------------------

A sample SLURM script can be found at ``scripts/ehthops_slurm.job``. You should already have set up the
Python environment and installed the necessary dependencies for running the pipeline. Copy this SLURM
job file to ``ehthops/hops-b3`` and submit to SLURM with ``sbatch``:

.. code-block:: bash

    cp ../scripts/ehthops_slurm.job .
    
    # Take some time to update the resources being requested and the job parameters.

    # Submit to SLURM
    sbatch ehthops_slurm.job