Running ehthops

Launching the pipeline

Arguments are passed only to the 0.launch step which sets environment variables used by the other scripts. Some of the arguments define environment variables and are set using SET_ENVVAR=VALUE syntax. The rest of the arguments are passed to the pipeline as command-line arguments.

The launch script can be run with the -h option to display all available options.

The fastest and the simplest way to launch the pipeline is to use the ehthops_pipeline.sh script which runs all stages of the pipeline in sequence. This script can be used with a configuration file (e.g., settings.config) that defines the environment variables and command-line arguments for the launch stage. On a SLURM cluster, the pipeline can be launched by submitting the ehthops_slurm.job script with appropriate modifications.

The maximum number of concurrent fourfit jobs can be set using SET_JOBARRAY_CAP in settings.config.

Some notes on the environment variables and running the stages manually (the following are taken care of automatically by scripts/ehthops_pipeline.sh):

  • 0.launch attempts to set reasonable defaults for the environment variables if they are not specified. We recommend setting/verifying the values of at least SRCDIR, CORRDAT, METADIR, and OBSYEAR every time 0.launch is run.

  • In stages 0 to 5, SRCDIR points to the top level directory that hosts the archival data. In stage 6, SRCDIR must point to the directory 5.+close/data in the current band.

  • If not starting from 0.bootstrap, ensure that 0.launch and 9.next are run before stage 1 (1.+flags+wins). This copies all relevant scripts and control files.

  • The user can set SHRDIR to any directory containing runnable marimo notebooks to replace the default notebooks provided.

  • The METADIR is expected to be organized according to the description given in the Metadata organization section.

Todo

Instructions to run as Docker image to be added.

Helper scripts

The ehthops/scripts directory contains several helper scripts:

  • ehthops_pipeline.sh runs all stages of the pipeline, applicable to any band. It is a good starting point to quickly start running the pipeline.

  • settings.config provides a sample configuration that can be updated and passed to ehthops_pipeline.sh.

  • ehthops_slurm.job can be used to submit the pipeline to a SLURM cluster with appropriate modifications.

  • cleanup.sh deletes all data generated during a previous run and leaves the working area in a clean state.