How to submit a LLM Python Script created on Jupyter Notebook on HPC?

To submit a Large Language Model (LLM) Python script created on Jupyter Notebook on a High Performance Computing (HPC) system, follow these steps:

1. Create a Job Script:

   • Write a bash script that loads the necessary modules, such as Python and Jupyter, and executes the Jupyter notebook using nbconvert.
   • Ensure the script includes the necessary directives for the HPC job scheduler, such as SLURM commands.

2. Submit the Job:

   • Save the script with a .sh extension (e.g., job.sh).
   • Submit the job using the HPC job scheduler command, typically sbatch job.sh.

Here is an example of a job script:

#!/bin/bash
#SBATCH --nodes=1
module load conda
source activate jupyter-env
srun jupyter nbconvert --to notebook --execute mynotebook.ipynb

This script loads the necessary modules, activates the Jupyter environment, and executes the Jupyter notebook using nbconvert.

3. Monitor the Job:
   • Check the job status using the HPC job scheduler commands, such as squeue or sacct.
   • Verify the output file generated by the job, typically named slurm-<job_id>.out.

By following these steps, you can successfully submit and run your LLM Python script created on Jupyter Notebook on an HPC system.