From e869309600b69d932e39647f63c3553b8b71d34b Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 18 Feb 2025 10:38:56 -0600 Subject: [PATCH 01/24] added array example1 --- docs/cheaha/slurm/slurm_tutorial.md | 63 +++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 7827f69d3..a5fd79dd7 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -300,6 +300,69 @@ $ sacct -j 27101430 27101430_2.+ extern USER 1 COMPLETED 0:0 ``` +#### Example 4.1: Count Number of Words From a File List in Parallel + +This example job script is designed to count the number of words in multiple files in parallel using a job array. First, let us create a directory to store the input files: + +```bash +$mkdir input_files + +$cd input_files +``` + +Next, copy and paste the following script into your terminal to generate five random text files, each containing a different number of randomly selected words: + +```bash +for i in {1..5}; do + shuf -n $(($RANDOM % 500 + 1)) /usr/share/dict/words > "random_file_$i.txt" + echo "random_file_$i.txt" >> file_list.txt +done +``` + +The script selects a random number between 1 and 500, then generates a file containing that many random words sourced from the executable `words` found in `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of words. It also keeps track of the generated 5 filenames in `file_list.txt`. + +Next, copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `word_count.job`. You will have to create a directory named `logs` to redirect the output and error files using, + +```bash +$mkdir logs +``` + + + +```bash linenums="1" + +#!/bin/bash +#SBATCH --job-name=word_count +#SBATCH --cpus-per-task=1 +#SBATCH --mem=4G # Memory per task +#SBATCH --partition=express +#SBATCH --time=00:15:00 # Max job run time +#SBATCH --output=logs/%x_%A_%a.out # Output log for each task +#SBATCH --error=logs/%x_%A_%a.err # Error log for each task +#SBATCH --array=1-5 # Adjust based on the number of files + +# Load necessary modules if needed +#module load Anaconda3 + +# Define working directory +WORKDIR="$HOME/input_files" +FILELIST="$WORKDIR/file_list.txt" + +# Get the file corresponding to the current task ID +FILE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" $FILELIST) + +# Check if file exists +if [[ -f "$WORKDIR/$FILE" ]]; then + # Count words and save result + wc -w "$WORKDIR/$FILE" > "$WORKDIR/${FILE}.wordcount" + echo "Processed $WORKDIR/$FILE" +else + echo "File not found: $WORKDIR/$FILE" +fi +``` + +The above SLURM job script runs a word count operation in parallel on multiple files using a job array (1-5). It reads a list of filenames from `file_list.txt` in `$HOME/input_files`. Each task in the job array processes a different file based on its task ID (SLURM_ARRAY_TASK_ID). If the file exists, it counts the number of words using wc -w and saves the output as `.wordcount`, and logs standard output and errors for each task separately. + ### Example 5: Multithreaded or Multicore Job This Slurm script illustrates execution of a MATLAB script in a multithread/multicore environemnt. Save the script as `multithread.job`. The `%` symbol in this script denotes comments within MATLAB code. Line 16 runs the MATLAB script `parfor_sum_array`, with an input array size `100` passed as argument, using 4 CPU cores (as specified in Line 5). From 10ae95f11e7ad6f3722517a48651bf6a9ed0561d Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 25 Feb 2025 10:28:50 -0600 Subject: [PATCH 02/24] add example 2 --- docs/cheaha/slurm/slurm_tutorial.md | 41 +++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index a5fd79dd7..ca7b684da 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -300,7 +300,37 @@ $ sacct -j 27101430 27101430_2.+ extern USER 1 COMPLETED 0:0 ``` -#### Example 4.1: Count Number of Words From a File List in Parallel +#### Example 4.1: Slurm Array Job for Line-by-Line Word Count + +This SLURM job script processes a text file line by line using an array job. Each task in the job array handles a single line and counts the number of words in it. + +```bash linenums="1" +#!/bin/bash +#SBATCH --job-name=line_word_count ### Name of the job +#SBATCH --cpus-per-task=1 ### Number of Tasks per CPU +#SBATCH --mem=2G ### Memory required, 2 gigabyte +#SBATCH --partition=express ### Cheaha Partition +#SBATCH --time=00:10:00 ### Estimated Time of Completion, 10 minutes +#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index +#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index + +### Define the input file +INPUT_FILE="$HOME/input_files/random_file_2.txt" + +### Extract the line corresponding to the current Slurm task ID +LINE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" "$INPUT_FILE") + +### Count words in the line and print the result +WORD_COUNT=$(echo "$LINE" | wc -w) +echo "Task $SLURM_ARRAY_TASK_ID: $WORD_COUNT words" +``` + +```bash +MAX_TASKS=$(wc -l < $HOME/input_files/random_file_2.txt) +sbatch --array=1-"$MAX_TASKS" line_word_count.job +``` + +#### Example 4.2: Count Number of Words From a File List in Parallel This example job script is designed to count the number of words in multiple files in parallel using a job array. First, let us create a directory to store the input files: @@ -314,7 +344,11 @@ Next, copy and paste the following script into your terminal to generate five ra ```bash for i in {1..5}; do - shuf -n $(($RANDOM % 500 + 1)) /usr/share/dict/words > "random_file_$i.txt" + num_lines=$(($RANDOM % 10 + 1)) # Random number of lines (1-10) + for _ in $(seq 1 $num_lines); do + num_words=$(($RANDOM % 20 + 5)) # Random words per line (5-20) + shuf -n $num_words /usr/share/dict/words | paste -s -d " " + done > "random_file_$i.txt" echo "random_file_$i.txt" >> file_list.txt done ``` @@ -327,10 +361,7 @@ Next, copy the following SLURM array job script to count the number of words in $mkdir logs ``` - - ```bash linenums="1" - #!/bin/bash #SBATCH --job-name=word_count #SBATCH --cpus-per-task=1 From 3ac534dca15b63c58b712d43cee626d0ce77b792 Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 25 Feb 2025 11:16:28 -0600 Subject: [PATCH 03/24] added script --- docs/cheaha/slurm/slurm_tutorial.md | 53 ++++++++++++++++------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index ca7b684da..078357884 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -302,7 +302,30 @@ $ sacct -j 27101430 #### Example 4.1: Slurm Array Job for Line-by-Line Word Count -This SLURM job script processes a text file line by line using an array job. Each task in the job array handles a single line and counts the number of words in it. +This SLURM job script processes a text file line by line using an array job. First, let us create a directory to create and store the input files: + +```bash +$mkdir input_files + +$cd input_files +``` + +Next, copy and paste the following script into your terminal to generate five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. Additionally, the names of the generated files are tracked in `file_list.txt`. + +```bash +for i in {1..5}; do + num_lines=$(($RANDOM % 10 + 1)) # Random number of lines (1-10) + for _ in $(seq 1 $num_lines); do + num_words=$(($RANDOM % 20 + 5)) # Random words per line (5-20) + shuf -n $num_words /usr/share/dict/words | paste -s -d " " + done > "random_file_$i.txt" + echo "random_file_$i.txt" >> file_list.txt +done +``` + +Let us use these input files for [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [Example 4.2](#example-42-count-number-of-words-from-a-file-list-in-parallel). + +Save the following script as `line_word_count.job`. This SLURM job ensures that each task in the job array processes a single line from the input file and counts the number of words in that line. ```bash linenums="1" #!/bin/bash @@ -325,37 +348,21 @@ WORD_COUNT=$(echo "$LINE" | wc -w) echo "Task $SLURM_ARRAY_TASK_ID: $WORD_COUNT words" ``` -```bash -MAX_TASKS=$(wc -l < $HOME/input_files/random_file_2.txt) -sbatch --array=1-"$MAX_TASKS" line_word_count.job -``` - -#### Example 4.2: Count Number of Words From a File List in Parallel - -This example job script is designed to count the number of words in multiple files in parallel using a job array. First, let us create a directory to store the input files: +Before running the above SLURM job, determine the number of lines in the input file (e.g., random_file_2.txt) using the following command: ```bash -$mkdir input_files - -$cd input_files +MAX_TASKS=$(wc -l < $HOME/input_files/random_file_2.txt) ``` -Next, copy and paste the following script into your terminal to generate five random text files, each containing a different number of randomly selected words: +Next, submit the array job using the command below, which creates an array of tasks from 1 to the maximum number of lines in the file. ```bash -for i in {1..5}; do - num_lines=$(($RANDOM % 10 + 1)) # Random number of lines (1-10) - for _ in $(seq 1 $num_lines); do - num_words=$(($RANDOM % 20 + 5)) # Random words per line (5-20) - shuf -n $num_words /usr/share/dict/words | paste -s -d " " - done > "random_file_$i.txt" - echo "random_file_$i.txt" >> file_list.txt -done +sbatch --array=1-"$MAX_TASKS" line_word_count.job ``` -The script selects a random number between 1 and 500, then generates a file containing that many random words sourced from the executable `words` found in `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of words. It also keeps track of the generated 5 filenames in `file_list.txt`. +#### Example 4.2: Counting Words in Multiple Files Using a Slurm Job Array -Next, copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `word_count.job`. You will have to create a directory named `logs` to redirect the output and error files using, +This example job script is designed to count the number of words in multiple files in parallel using a job array. Copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `word_count.job`. You will have to create a directory named `logs` to redirect the output and error files using, ```bash $mkdir logs From 988e932399c60058cf39d819c0bd9dd068b9d2b6 Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 25 Feb 2025 11:45:51 -0600 Subject: [PATCH 04/24] typos --- docs/cheaha/slurm/slurm_tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 078357884..317106acb 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -323,7 +323,7 @@ for i in {1..5}; do done ``` -Let us use these input files for [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [Example 4.2](#example-42-count-number-of-words-from-a-file-list-in-parallel). +Let us use these input files for [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [Example 4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array). Save the following script as `line_word_count.job`. This SLURM job ensures that each task in the job array processes a single line from the input file and counts the number of words in that line. From 3f280fc7e2e8028793ca8a285bd6a4b05e08027a Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 28 Feb 2025 11:15:08 -0600 Subject: [PATCH 05/24] add output to example1 --- docs/cheaha/slurm/slurm_tutorial.md | 30 ++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 317106acb..cec9a90fc 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -310,7 +310,7 @@ $mkdir input_files $cd input_files ``` -Next, copy and paste the following script into your terminal to generate five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. Additionally, the names of the generated files are tracked in `file_list.txt`. +Next, copy and paste the following script into your terminal to generate five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. The input files along with its path are tracked in `file_list.txt`. ```bash for i in {1..5}; do @@ -319,13 +319,15 @@ for i in {1..5}; do num_words=$(($RANDOM % 20 + 5)) # Random words per line (5-20) shuf -n $num_words /usr/share/dict/words | paste -s -d " " done > "random_file_$i.txt" - echo "random_file_$i.txt" >> file_list.txt done + +# Use globbing to get all generated files and save them to file_list.txt +ls random_file_*.txt > file_list.txt ``` Let us use these input files for [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [Example 4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array). -Save the following script as `line_word_count.job`. This SLURM job ensures that each task in the job array processes a single line from the input file and counts the number of words in that line. +Save the following script as `line_word_count.job`. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. ```bash linenums="1" #!/bin/bash @@ -337,13 +339,13 @@ Save the following script as `line_word_count.job`. This SLURM job ensures that #SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index #SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index -### Define the input file +### Define the input file, for instance, random_file_2.txt INPUT_FILE="$HOME/input_files/random_file_2.txt" -### Extract the line corresponding to the current Slurm task ID +### Extract the line corresponding to the current Slurm array task ID LINE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" "$INPUT_FILE") -### Count words in the line and print the result +### Count words in the line and print the result for each task WORD_COUNT=$(echo "$LINE" | wc -w) echo "Task $SLURM_ARRAY_TASK_ID: $WORD_COUNT words" ``` @@ -360,6 +362,20 @@ Next, submit the array job using the command below, which creates an array of ta sbatch --array=1-"$MAX_TASKS" line_word_count.job ``` +The below output comes from the SLURM job array script (line_word_count.job), where each task processes one line from random_file_2.txt. Since the file has 3 lines, SLURM created 3 tasks (Task 1, Task 2, Task 3), each counting words in its respective line. + +```bash +$ wc -l ../input_files/random_file_2.txt #counts the number of lines in the file, for instance, random_file_2.txt +3 ../input_files/random_file_2.txt +``` + +```bash +$ cat line_word_count_$JOB_ID_*.out +Task 1: 8 words +Task 2: 19 words +Task 3: 6 words +``` + #### Example 4.2: Counting Words in Multiple Files Using a Slurm Job Array This example job script is designed to count the number of words in multiple files in parallel using a job array. Copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `word_count.job`. You will have to create a directory named `logs` to redirect the output and error files using, @@ -392,7 +408,7 @@ FILE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" $FILELIST) # Check if file exists if [[ -f "$WORKDIR/$FILE" ]]; then # Count words and save result - wc -w "$WORKDIR/$FILE" > "$WORKDIR/${FILE}.wordcount" + wc -w "$WORKDIR/$FILE" > "$WORKDIR/${BASENAME}.wordcount" echo "Processed $WORKDIR/$FILE" else echo "File not found: $WORKDIR/$FILE" From 98be6cf54d3624db717488d957faebb0f6fd5de6 Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 28 Feb 2025 11:43:04 -0600 Subject: [PATCH 06/24] add output to example 2 --- docs/cheaha/slurm/slurm_tutorial.md | 64 +++++++++++++++++++---------- 1 file changed, 42 insertions(+), 22 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index cec9a90fc..195aef041 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -327,7 +327,11 @@ ls random_file_*.txt > file_list.txt Let us use these input files for [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [Example 4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array). -Save the following script as `line_word_count.job`. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. +Save the following Slurm script as `line_word_count.job`. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. You will have to create a directory named `logs` to redirect the output and error files using the command: + +```bash +$mkdir logs +``` ```bash linenums="1" #!/bin/bash @@ -378,7 +382,7 @@ Task 3: 6 words #### Example 4.2: Counting Words in Multiple Files Using a Slurm Job Array -This example job script is designed to count the number of words in multiple files in parallel using a job array. Copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `word_count.job`. You will have to create a directory named `logs` to redirect the output and error files using, +This example job script is designed to count the number of words in multiple files in parallel using a job array. The input files are created using the script depicted in [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) Copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `file_word_count.job`. You will have to create a directory named `logs` to redirect the output and error files using the command: ```bash $mkdir logs @@ -386,36 +390,52 @@ $mkdir logs ```bash linenums="1" #!/bin/bash -#SBATCH --job-name=word_count -#SBATCH --cpus-per-task=1 -#SBATCH --mem=4G # Memory per task -#SBATCH --partition=express -#SBATCH --time=00:15:00 # Max job run time -#SBATCH --output=logs/%x_%A_%a.out # Output log for each task -#SBATCH --error=logs/%x_%A_%a.err # Error log for each task -#SBATCH --array=1-5 # Adjust based on the number of files - -# Load necessary modules if needed -#module load Anaconda3 - -# Define working directory +#SBATCH --job-name=file_word_count ### Name of the job +#SBATCH --cpus-per-task=1 ### Number of Tasks per CPU +#SBATCH --mem=4G ### Memory required, 4 gigabyte +#SBATCH --partition=express ### Cheaha Partition +#SBATCH --time=00:15:00 ### Estimated Time of Completion, 10 minutes +#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index +#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index +#SBATCH --array=1-5 ### Adjust based on the number of files + +### Define working directory WORKDIR="$HOME/input_files" FILELIST="$WORKDIR/file_list.txt" -# Get the file corresponding to the current task ID +### Get the file corresponding to the current task ID FILE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" $FILELIST) +BASENAME=$(basename "$FILE" .txt) -# Check if file exists -if [[ -f "$WORKDIR/$FILE" ]]; then +### Check if file exists and count words from file +if [[ -f "$FILE" ]]; then # Count words and save result - wc -w "$WORKDIR/$FILE" > "$WORKDIR/${BASENAME}.wordcount" - echo "Processed $WORKDIR/$FILE" + wc -w "$FILE" > "$WORKDIR/${BASENAME}.wordcount" + echo "Processed $FILE" else - echo "File not found: $WORKDIR/$FILE" + echo "File not found: $FILE" fi ``` -The above SLURM job script runs a word count operation in parallel on multiple files using a job array (1-5). It reads a list of filenames from `file_list.txt` in `$HOME/input_files`. Each task in the job array processes a different file based on its task ID (SLURM_ARRAY_TASK_ID). If the file exists, it counts the number of words using wc -w and saves the output as `.wordcount`, and logs standard output and errors for each task separately. +The above SLURM job script runs a word count operation in parallel on multiple files using a job array (1-5). It reads a list of filenames along with its path from `file_list.txt` located in `$HOME/input_files`. Each task in the job array processes a different file based on its task ID (SLURM_ARRAY_TASK_ID). If the file exists, it counts the number of words using wc -w and saves the output as `.wordcount`, and logs standard output and errors for each task separately. + +In the output below, each file was processed independently by a SLURM array task. The task IDs 1, 2, 3, 4, and 5 correspond to the five different files being processed. The SLURM Job ID is `31934540`. Each output file contains the word count for a specific text file handled by its respective SLURM array task. + +The command `cat random_file*.wordcount` displays the word count for all processed input files. + +```bash + +$ ls file_word_count*.out +file_word_count_31934540_1.out file_word_count_31934540_3.out file_word_count_31934540_5.out +file_word_count_31934540_2.out file_word_count_31934540_4.out + +$ cat random_file*.wordcount +142 random_file_1.txt +82 random_file_2.txt +55 random_file_3.txt +61 random_file_4.txt +92 random_file_5.txt +``` ### Example 5: Multithreaded or Multicore Job From d2f7a12925adfc2538184ae8326fef927d487f28 Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 28 Feb 2025 12:27:16 -0600 Subject: [PATCH 07/24] added usage of find command in example 4.2 --- docs/cheaha/slurm/slurm_tutorial.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 195aef041..18cd0ecad 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -397,7 +397,6 @@ $mkdir logs #SBATCH --time=00:15:00 ### Estimated Time of Completion, 10 minutes #SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index #SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index -#SBATCH --array=1-5 ### Adjust based on the number of files ### Define working directory WORKDIR="$HOME/input_files" @@ -419,6 +418,18 @@ fi The above SLURM job script runs a word count operation in parallel on multiple files using a job array (1-5). It reads a list of filenames along with its path from `file_list.txt` located in `$HOME/input_files`. Each task in the job array processes a different file based on its task ID (SLURM_ARRAY_TASK_ID). If the file exists, it counts the number of words using wc -w and saves the output as `.wordcount`, and logs standard output and errors for each task separately. +Before running the above SLURM job, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path `$HOME/input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. + +```bash +MAX_TASKS=$(find $HOME/input_files -type f -name "random_file_*.txt" | wc -l) +``` + +Next, submit the array job using the command below, which creates an array of tasks from 1 to the value of MAX_TASKS, where each task corresponds to processing a different file listed in the array. + +```bash +sbatch --array=1-"$MAX_TASKS" line_word_count.job +``` + In the output below, each file was processed independently by a SLURM array task. The task IDs 1, 2, 3, 4, and 5 correspond to the five different files being processed. The SLURM Job ID is `31934540`. Each output file contains the word count for a specific text file handled by its respective SLURM array task. The command `cat random_file*.wordcount` displays the word count for all processed input files. From 028e3e29e17ed541ad0c63d49aa5f6d4bd638876 Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 11 Mar 2025 09:58:52 -0500 Subject: [PATCH 08/24] revised structurin --- docs/cheaha/slurm/slurm_tutorial.md | 40 +++++++++++++++++++---------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 18cd0ecad..bf4f23710 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -302,7 +302,14 @@ $ sacct -j 27101430 #### Example 4.1: Slurm Array Job for Line-by-Line Word Count -This SLURM job script processes a text file line by line using an array job. First, let us create a directory to create and store the input files: +In the examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array), we will explore how to use a Slurm Array Job to process text files in parallel. To keep things organized, we divide the process into two main sections: + +(i) Setup: This section covers creating the necessary directories and input files. Since this setup is common for both examples, we will do it once before moving on. +(ii) Running the Slurm Array Job: Here, we focus on understanding and executing an example Slurm array job. + +##### Setup + +Before we run the Slurm array examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array), we need to prepare the environment. This involves creating a directory and generating the input files. First, let us create a directory to create and store the input files: ```bash $mkdir input_files @@ -310,9 +317,10 @@ $mkdir input_files $cd input_files ``` -Next, copy and paste the following script into your terminal to generate five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. The input files along with its path are tracked in `file_list.txt`. +Save the following script as `generate_input.sh`. This script generates five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. The input files along with its path are tracked in `file_list.txt`. For more details refer to [script concepts](../../workflow_solutions/shell.md#script-concepts) and [bash scripting example](../../cheaha/software/modules.md#best-practice-for-loading-modules). ```bash +#!/bin/bash for i in {1..5}; do num_lines=$(($RANDOM % 10 + 1)) # Random number of lines (1-10) for _ in $(seq 1 $num_lines); do @@ -320,19 +328,27 @@ for i in {1..5}; do shuf -n $num_words /usr/share/dict/words | paste -s -d " " done > "random_file_$i.txt" done - # Use globbing to get all generated files and save them to file_list.txt ls random_file_*.txt > file_list.txt ``` -Let us use these input files for [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [Example 4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array). +You can execute the script as shown below. The first command `chmod` grants execute permission to the script so it can be run directly. The second command runs the script to generate the required input files. For more details on usage of bash scripting, refer to [Script Concepts](../../workflow_solutions/shell.md/#script-concepts) + +```bash +$chmod +x generate_input.sh +$./generate_input.sh +``` -Save the following Slurm script as `line_word_count.job`. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. You will have to create a directory named `logs` to redirect the output and error files using the command: +Let us use these input files for [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [Example 4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array). You will also have to create a directory named `logs` to redirect the output and error files using the command: ```bash $mkdir logs ``` +##### Running the Slurm Array Job + +This SLURM job script processes a text file line by line using an array job. Save the following Slurm script as `line_word_count.job`. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. + ```bash linenums="1" #!/bin/bash #SBATCH --job-name=line_word_count ### Name of the job @@ -346,10 +362,10 @@ $mkdir logs ### Define the input file, for instance, random_file_2.txt INPUT_FILE="$HOME/input_files/random_file_2.txt" -### Extract the line corresponding to the current Slurm array task ID +### `sed` is a shell command and stream editor for manipulating text. Here, `sed` reads $INPUT_FILE and extracts the line corresponding to SLURM_ARRAY_TASK_ID. LINE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" "$INPUT_FILE") -### Count words in the line and print the result for each task +### The command `wc -w` counts the number of words in the extracted line ($LINE), while echo "$LINE" prints the line of text. The pipe (|) then passes this text from echo "$LINE" to wc -w, which counts the words in the line for each task. WORD_COUNT=$(echo "$LINE" | wc -w) echo "Task $SLURM_ARRAY_TASK_ID: $WORD_COUNT words" ``` @@ -369,8 +385,8 @@ sbatch --array=1-"$MAX_TASKS" line_word_count.job The below output comes from the SLURM job array script (line_word_count.job), where each task processes one line from random_file_2.txt. Since the file has 3 lines, SLURM created 3 tasks (Task 1, Task 2, Task 3), each counting words in its respective line. ```bash -$ wc -l ../input_files/random_file_2.txt #counts the number of lines in the file, for instance, random_file_2.txt -3 ../input_files/random_file_2.txt +$ wc -l $HOME/input_files/random_file_2.txt #counts the number of lines in the file, for instance, random_file_2.txt +3 $HOME/input_files/random_file_2.txt ``` ```bash @@ -382,11 +398,7 @@ Task 3: 6 words #### Example 4.2: Counting Words in Multiple Files Using a Slurm Job Array -This example job script is designed to count the number of words in multiple files in parallel using a job array. The input files are created using the script depicted in [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) Copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `file_word_count.job`. You will have to create a directory named `logs` to redirect the output and error files using the command: - -```bash -$mkdir logs -``` +This example job script is designed to count the number of words in multiple files in parallel using a job array. It utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. Copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `file_word_count.job`. ```bash linenums="1" #!/bin/bash From 1c3ec0d525a8fe1a49c9126b3e263cd5b1a1530f Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 18 Mar 2025 16:55:32 -0500 Subject: [PATCH 09/24] organizing data --- docs/cheaha/slurm/slurm_tutorial.md | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index bf4f23710..d2818d72c 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -309,12 +309,13 @@ In the examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) a ##### Setup -Before we run the Slurm array examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array), we need to prepare the environment. This involves creating a directory and generating the input files. First, let us create a directory to create and store the input files: +Before executing the Slurm array examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array), we need to set up the environment. This includes creating a main directory, `example_4`, to store all job files and scripts. Within it, we will create two subdirectories: `input_files` to store the generated input files and `logs` to organize output and error logs. Let us create and structure these directories. ```bash +$mkdir example_4 +$cd example_4 $mkdir input_files - -$cd input_files +$mkdir logs ``` Save the following script as `generate_input.sh`. This script generates five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. The input files along with its path are tracked in `file_list.txt`. For more details refer to [script concepts](../../workflow_solutions/shell.md#script-concepts) and [bash scripting example](../../cheaha/software/modules.md#best-practice-for-loading-modules). @@ -339,12 +340,6 @@ $chmod +x generate_input.sh $./generate_input.sh ``` -Let us use these input files for [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [Example 4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array). You will also have to create a directory named `logs` to redirect the output and error files using the command: - -```bash -$mkdir logs -``` - ##### Running the Slurm Array Job This SLURM job script processes a text file line by line using an array job. Save the following Slurm script as `line_word_count.job`. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. From d778ad8b17dd2d8d3b005362ca579f7c4b1fc3bf Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 21 Mar 2025 11:08:59 -0500 Subject: [PATCH 10/24] alter example4.2 --- docs/cheaha/slurm/slurm_tutorial.md | 62 +++++++++++++++++++++++++---- 1 file changed, 55 insertions(+), 7 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index d2818d72c..276a3fa80 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -316,9 +316,10 @@ $mkdir example_4 $cd example_4 $mkdir input_files $mkdir logs +$cd input_files ``` -Save the following script as `generate_input.sh`. This script generates five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. The input files along with its path are tracked in `file_list.txt`. For more details refer to [script concepts](../../workflow_solutions/shell.md#script-concepts) and [bash scripting example](../../cheaha/software/modules.md#best-practice-for-loading-modules). +The next step involves generating input files using a pseudo-random text generation script. Save the following script as `generate_input.sh` inside the `input_files` folder. This script generates five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. The input files along with its path are tracked in `file_list.txt`. For more details refer to [script concepts](../../workflow_solutions/shell.md#script-concepts) and [bash scripting example](../../cheaha/software/modules.md#best-practice-for-loading-modules). ```bash #!/bin/bash @@ -329,8 +330,9 @@ for i in {1..5}; do shuf -n $num_words /usr/share/dict/words | paste -s -d " " done > "random_file_$i.txt" done -# Use globbing to get all generated files and save them to file_list.txt -ls random_file_*.txt > file_list.txt + +# Use find and globbing to list all generated files in the current directory and subdirectories. The absolute path of each file is tracked in the file_list.txt +find $(pwd) -type f -name "random_file_*.txt" | sort > file_list.txt ``` You can execute the script as shown below. The first command `chmod` grants execute permission to the script so it can be run directly. The second command runs the script to generate the required input files. For more details on usage of bash scripting, refer to [Script Concepts](../../workflow_solutions/shell.md/#script-concepts) @@ -355,12 +357,15 @@ This SLURM job script processes a text file line by line using an array job. Sav #SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index ### Define the input file, for instance, random_file_2.txt -INPUT_FILE="$HOME/input_files/random_file_2.txt" +INPUT_FILE="$HOME/example_4/input_files/random_file_2.txt" -### `sed` is a shell command and stream editor for manipulating text. Here, `sed` reads $INPUT_FILE and extracts the line corresponding to SLURM_ARRAY_TASK_ID. +### `sed` is a shell command and stream editor for manipulating text. +### Here, `sed` reads $INPUT_FILE and extracts the line corresponding to SLURM_ARRAY_TASK_ID. LINE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" "$INPUT_FILE") -### The command `wc -w` counts the number of words in the extracted line ($LINE), while echo "$LINE" prints the line of text. The pipe (|) then passes this text from echo "$LINE" to wc -w, which counts the words in the line for each task. +### The command `wc -w` counts the number of words in the extracted line ($LINE), +### while echo "$LINE" prints the line of text. The pipe (|) then passes this text from echo "$LINE" to +### wc -w, which counts the words in the line for each task. WORD_COUNT=$(echo "$LINE" | wc -w) echo "Task $SLURM_ARRAY_TASK_ID: $WORD_COUNT words" ``` @@ -391,7 +396,50 @@ Task 2: 19 words Task 3: 6 words ``` -#### Example 4.2: Counting Words in Multiple Files Using a Slurm Job Array +#### Example 4.2: Dynamically Reading and Counting Words in Multiple Files dynamically Using a Slurm Job Array + +```bash linenums="1" +#!/bin/bash +#SBATCH --job-name=file_word_count ### Name of the job +#SBATCH --cpus-per-task=1 ### Number of Tasks per CPU +#SBATCH --mem=4G ### Memory required, 4 gigabyte +#SBATCH --partition=express ### Cheaha Partition +#SBATCH --time=00:15:00 ### Estimated Time of Completion, 15 minutes +#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index +#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index + +### Define working directory +WORKDIR="$HOME/example_4/input_files" + +### Find all files in the working directory ($WORKDIR) that match the pattern "random_file_*.txt" +### Pass the list of these files to the 'sort' command via the pipe ('|') +### The sorted file paths are then captured and stored in the FILES array. +FILES=($(find "$WORKDIR" -type f -name "random_file_*.txt" | sort)) + +### Selects the file corresponding to the current Slurm Array task ID. +### The task IDs are assigned starting from 1, but in bash, array indexing starts from 0. +### To align the task ID with bash’s 0-based array indexing, we subtract 1 from the task ID. +FILE="${FILES[$SLURM_ARRAY_TASK_ID-1]}" + +### Extract the directory part of the file path. This will give the full directory path of the selected file. +DIRNAME=$(dirname "$FILE") + +### Extract the base name of the file (without the extension) +BASENAME=$(basename "$FILE" .txt) + +### Check if file exists and count words from file +if [[ -f "$FILE" ]]; then + ### If the file exists, count the number of words in the file using 'wc -w' + ### The result is saved to a file with the same name as the original file, + ### but with a '.wordcount' extension in the same directory as the original file. + wc -w "$FILE" > "$DIRNAME/${BASENAME}.wordcount" + echo "Processed $FILE" +else + echo "File not found: $FILE" +fi +``` + +#### Example 4.3: Counting Words in Multiple Files from a File List Using a Slurm Job Array This example job script is designed to count the number of words in multiple files in parallel using a job array. It utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. Copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `file_word_count.job`. From 17637cc5ac0ca600147655688fb09edca2e0dce4 Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 21 Mar 2025 11:13:13 -0500 Subject: [PATCH 11/24] lint verified --- docs/cheaha/slurm/slurm_tutorial.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 276a3fa80..514ada7a3 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -302,14 +302,14 @@ $ sacct -j 27101430 #### Example 4.1: Slurm Array Job for Line-by-Line Word Count -In the examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array), we will explore how to use a Slurm Array Job to process text files in parallel. To keep things organized, we divide the process into two main sections: +In the examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count), [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array), and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list-using-a-slurm-job-array), we will explore how to use a Slurm Array Job to process text files in parallel. To keep things organized, we divide the process into two main sections: (i) Setup: This section covers creating the necessary directories and input files. Since this setup is common for both examples, we will do it once before moving on. (ii) Running the Slurm Array Job: Here, we focus on understanding and executing an example Slurm array job. ##### Setup -Before executing the Slurm array examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [4.2](#example-42-counting-words-in-multiple-files-using-a-slurm-job-array), we need to set up the environment. This includes creating a main directory, `example_4`, to store all job files and scripts. Within it, we will create two subdirectories: `input_files` to store the generated input files and `logs` to organize output and error logs. Let us create and structure these directories. +Before executing the Slurm array examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array) and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list-using-a-slurm-job-array), we need to set up the environment. This includes creating a main directory, `example_4`, to store all job files and scripts. Within it, we will create two subdirectories: `input_files` to store the generated input files and `logs` to organize output and error logs. Let us create and structure these directories. ```bash $mkdir example_4 From 493e25f83695a0240c4dc8399cfe222721df28de Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 21 Mar 2025 11:59:42 -0500 Subject: [PATCH 12/24] add ex-4.2 description --- docs/cheaha/slurm/slurm_tutorial.md | 50 +++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 6 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 514ada7a3..982ef914a 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -319,7 +319,7 @@ $mkdir logs $cd input_files ``` -The next step involves generating input files using a pseudo-random text generation script. Save the following script as `generate_input.sh` inside the `input_files` folder. This script generates five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. The input files along with its path are tracked in `file_list.txt`. For more details refer to [script concepts](../../workflow_solutions/shell.md#script-concepts) and [bash scripting example](../../cheaha/software/modules.md#best-practice-for-loading-modules). +The next step involves generating input files using a pseudo-random text generation script. Save the following script as `generate_input.sh` inside the `input_files` folder. This script generates five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. For more details refer to [script concepts](../../workflow_solutions/shell.md#script-concepts) and [bash scripting example](../../cheaha/software/modules.md#best-practice-for-loading-modules). ```bash #!/bin/bash @@ -330,9 +330,6 @@ for i in {1..5}; do shuf -n $num_words /usr/share/dict/words | paste -s -d " " done > "random_file_$i.txt" done - -# Use find and globbing to list all generated files in the current directory and subdirectories. The absolute path of each file is tracked in the file_list.txt -find $(pwd) -type f -name "random_file_*.txt" | sort > file_list.txt ``` You can execute the script as shown below. The first command `chmod` grants execute permission to the script so it can be run directly. The second command runs the script to generate the required input files. For more details on usage of bash scripting, refer to [Script Concepts](../../workflow_solutions/shell.md/#script-concepts) @@ -398,6 +395,8 @@ Task 3: 6 words #### Example 4.2: Dynamically Reading and Counting Words in Multiple Files dynamically Using a Slurm Job Array +This example job script performs the same function as [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job`. + ```bash linenums="1" #!/bin/bash #SBATCH --job-name=file_word_count ### Name of the job @@ -439,9 +438,48 @@ else fi ``` +The script begins by setting up the Slurm array job with required resources. The working directory is defined, and the script uses the `find` command to search for files matching the pattern random_file_*.txt within that working directory and its subdirectories, passing the resulting list of file paths to the sort command to ensure they are processed in order. The file corresponding to the current Slurm array task ID is selected, and its directory and base name are extracted. If the file exists, the script counts the words in the file using the `wc -w` command and saves the word count to a new file with a `.wordcount` extension in the same directory. If the file is not found, an error message is displayed. + +Before you run the script `dynamic_file_word_count.job`, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path`$HOME/example_4input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. + +```bash +MAX_TASKS=$(find $HOME/example_4/input_files -type f -name "random_file_*.txt" | wc -l) +``` + +Next, submit the array job using the command below, which creates an array of tasks from 1 to the value of MAX_TASKS, where each task corresponds to processing a different file listed in the array. + +```bash +sbatch --array=1-"$MAX_TASKS" dynamic_file_word_count.job +``` + +In the output below, each file was processed independently by a SLURM job array task. The task IDs 1, 2, 3, 4, and 5 correspond to the five different files being processed. The SLURM Job ID is `31934540`. Each output file contains the word count for a specific text file handled by its respective SLURM array task. + +The `find` command in the output shows the word counts for each of the random_file*.txt files, where each `.wordcount` file contains the word count for its corresponding input file. + +```bash + +$ ls file_word_count*.out +file_word_count_31934540_1.out file_word_count_31934540_3.out file_word_count_31934540_5.out +file_word_count_31934540_2.out file_word_count_31934540_4.out + +$ find input_files/ -type f -name "random_file*.wordcount" -exec cat {} \; +81 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_1.txt +117 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_5.txt +33 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_2.txt +104 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_3.txt +81 /home/prema/Tutorial/slurm_tutorial/example4/input_files/test/random_file_6.txt +114 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_4.txt +``` + #### Example 4.3: Counting Words in Multiple Files from a File List Using a Slurm Job Array -This example job script is designed to count the number of words in multiple files in parallel using a job array. It utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. Copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `file_word_count.job`. +This example job script is designed to count the number of words in multiple files in parallel using a job array. It utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. Copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `file_list_word_count.job`. + +# Use find and globbing to list all generated files in the current directory and subdirectories. The absolute path of each file is tracked in the file_list.txt + +find $(pwd) -type f -name "random_file_*.txt" | sort > file_list.txt + +The input files along with its path are tracked in `file_list.txt`. ```bash linenums="1" #!/bin/bash @@ -485,7 +523,7 @@ Next, submit the array job using the command below, which creates an array of ta sbatch --array=1-"$MAX_TASKS" line_word_count.job ``` -In the output below, each file was processed independently by a SLURM array task. The task IDs 1, 2, 3, 4, and 5 correspond to the five different files being processed. The SLURM Job ID is `31934540`. Each output file contains the word count for a specific text file handled by its respective SLURM array task. +In the output below, each file was processed independently by a SLURM job array task. The task IDs 1, 2, 3, 4, and 5 correspond to the five different files being processed. The SLURM Job ID is `31934540`. Each output file contains the word count for a specific text file handled by its respective SLURM array task. The command `cat random_file*.wordcount` displays the word count for all processed input files. From 839430bc63ef8d86a9816e27fcc767f8c217be4d Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 21 Mar 2025 14:29:45 -0500 Subject: [PATCH 13/24] added example 4.3 --- docs/cheaha/slurm/slurm_tutorial.md | 40 ++++++++++------------------- 1 file changed, 13 insertions(+), 27 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 982ef914a..ccea61f3a 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -395,7 +395,7 @@ Task 3: 6 words #### Example 4.2: Dynamically Reading and Counting Words in Multiple Files dynamically Using a Slurm Job Array -This example job script performs the same function as [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job`. +This example job script performs the same function as [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job`. It utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. ```bash linenums="1" #!/bin/bash @@ -458,11 +458,11 @@ The `find` command in the output shows the word counts for each of the random_fi ```bash -$ ls file_word_count*.out +$ ls $HOME/example_4/logs/file_word_count*.out file_word_count_31934540_1.out file_word_count_31934540_3.out file_word_count_31934540_5.out file_word_count_31934540_2.out file_word_count_31934540_4.out -$ find input_files/ -type f -name "random_file*.wordcount" -exec cat {} \; +$ find $HOME/example_4/input_files/ -type f -name "random_file*.wordcount" -exec cat {} \; 81 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_1.txt 117 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_5.txt 33 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_2.txt @@ -473,13 +473,15 @@ $ find input_files/ -type f -name "random_file*.wordcount" -exec cat {} \; #### Example 4.3: Counting Words in Multiple Files from a File List Using a Slurm Job Array -This example job script is designed to count the number of words in multiple files in parallel using a job array. It utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. Copy the following SLURM array job script to count the number of words in the five generated files in parallel. Save it as `file_list_word_count.job`. +This example job script is similar to [Example 4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array), but instead of dynamically reading files from the directory, it counts the number of words in multiple files in parallel using a SLURM job array, with the files listed in a separate file list. This example utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. -# Use find and globbing to list all generated files in the current directory and subdirectories. The absolute path of each file is tracked in the file_list.txt +To create a file list by tracking all the files in the `input_files` directory, use the find command along with globbing to list all generated files in the current directory and its subdirectories. The input files along with its absolute path are tracked in `file_list.txt`. -find $(pwd) -type f -name "random_file_*.txt" | sort > file_list.txt +```bash +find $HOME/example_4/input_files -type f -name "random_file_*.txt" -exec realpath {} \; | sort > file_list.txt +``` -The input files along with its path are tracked in `file_list.txt`. +Copy the following SLURM array job script to a file naming `file_list_word_count.job`. This Slurm array job script count the number of words across multiple files listed in a file list `file_list.txt`. ```bash linenums="1" #!/bin/bash @@ -487,13 +489,13 @@ The input files along with its path are tracked in `file_list.txt`. #SBATCH --cpus-per-task=1 ### Number of Tasks per CPU #SBATCH --mem=4G ### Memory required, 4 gigabyte #SBATCH --partition=express ### Cheaha Partition -#SBATCH --time=00:15:00 ### Estimated Time of Completion, 10 minutes +#SBATCH --time=00:15:00 ### Estimated Time of Completion, 15 minutes #SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index #SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index ### Define working directory WORKDIR="$HOME/input_files" -FILELIST="$WORKDIR/file_list.txt" +FILELIST="file_list.txt" ### Get the file corresponding to the current task ID FILE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" $FILELIST) @@ -514,7 +516,7 @@ The above SLURM job script runs a word count operation in parallel on multiple f Before running the above SLURM job, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path `$HOME/input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. ```bash -MAX_TASKS=$(find $HOME/input_files -type f -name "random_file_*.txt" | wc -l) +MAX_TASKS=$(wc -l < file_list.txt) ``` Next, submit the array job using the command below, which creates an array of tasks from 1 to the value of MAX_TASKS, where each task corresponds to processing a different file listed in the array. @@ -523,23 +525,7 @@ Next, submit the array job using the command below, which creates an array of ta sbatch --array=1-"$MAX_TASKS" line_word_count.job ``` -In the output below, each file was processed independently by a SLURM job array task. The task IDs 1, 2, 3, 4, and 5 correspond to the five different files being processed. The SLURM Job ID is `31934540`. Each output file contains the word count for a specific text file handled by its respective SLURM array task. - -The command `cat random_file*.wordcount` displays the word count for all processed input files. - -```bash - -$ ls file_word_count*.out -file_word_count_31934540_1.out file_word_count_31934540_3.out file_word_count_31934540_5.out -file_word_count_31934540_2.out file_word_count_31934540_4.out - -$ cat random_file*.wordcount -142 random_file_1.txt -82 random_file_2.txt -55 random_file_3.txt -61 random_file_4.txt -92 random_file_5.txt -``` +The output generated will be similar to [Example 4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array), as the functionality is the same, but the method of handling the input data differs. ### Example 5: Multithreaded or Multicore Job From ca0bd3387d43d47de339b382047b322131c63c6b Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 25 Mar 2025 10:40:11 -0500 Subject: [PATCH 14/24] example4.1 typos --- docs/cheaha/slurm/slurm_tutorial.md | 30 +++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index ccea61f3a..fa8ea7d48 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -302,14 +302,15 @@ $ sacct -j 27101430 #### Example 4.1: Slurm Array Job for Line-by-Line Word Count -In the examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count), [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array), and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list-using-a-slurm-job-array), we will explore how to use a Slurm Array Job to process text files in parallel. To keep things organized, we divide the process into two main sections: +In the examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count), [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array), and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list-using-a-slurm-job-array), let us explore how to use a Slurm Array Job to process text files in parallel. To keep things organized, we divide the process into two main sections: (i) Setup: This section covers creating the necessary directories and input files. Since this setup is common for both examples, we will do it once before moving on. + (ii) Running the Slurm Array Job: Here, we focus on understanding and executing an example Slurm array job. ##### Setup -Before executing the Slurm array examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array) and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list-using-a-slurm-job-array), we need to set up the environment. This includes creating a main directory, `example_4`, to store all job files and scripts. Within it, we will create two subdirectories: `input_files` to store the generated input files and `logs` to organize output and error logs. Let us create and structure these directories. +Before executing the Slurm array examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array) and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list-using-a-slurm-job-array), we need to set up the environment. This includes creating a main directory, `example_4`, to store all job files and scripts. Within it, we will create two subdirectories: `input_files` to store the generated input text files and `logs` to organize output and error logs. Let us create and structure these directories. ```bash $mkdir example_4 @@ -319,20 +320,27 @@ $mkdir logs $cd input_files ``` -The next step involves generating input files using a pseudo-random text generation script. Save the following script as `generate_input.sh` inside the `input_files` folder. This script generates five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. For more details refer to [script concepts](../../workflow_solutions/shell.md#script-concepts) and [bash scripting example](../../cheaha/software/modules.md#best-practice-for-loading-modules). +The next step involves generating input files using a pseudo-random text generation script. Save the following script as `generate_input.sh` within the `input_files` folder. This script generates five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. For more details refer to [script concepts](../../workflow_solutions/shell.md#script-concepts) and [bash scripting example](../../cheaha/software/modules.md#best-practice-for-loading-modules). ```bash #!/bin/bash + +### Loop to create 5 random files for i in {1..5}; do - num_lines=$(($RANDOM % 10 + 1)) # Random number of lines (1-10) + ### Generate a random number of lines (1-10) + num_lines=$(($RANDOM % 10 + 1)) for _ in $(seq 1 $num_lines); do - num_words=$(($RANDOM % 20 + 5)) # Random words per line (5-20) + ### Generate a random words per line (5-20) + num_words=$(($RANDOM % 20 + 5)) + ### Use 'shuf' to pick $num_words random words from the dictionary file. + ### The pipe ('|') sends these selected words to 'paste', which combines them into a single line, + ### separating each word with a space (using the '-d " "' option). shuf -n $num_words /usr/share/dict/words | paste -s -d " " done > "random_file_$i.txt" done ``` -You can execute the script as shown below. The first command `chmod` grants execute permission to the script so it can be run directly. The second command runs the script to generate the required input files. For more details on usage of bash scripting, refer to [Script Concepts](../../workflow_solutions/shell.md/#script-concepts) +You can execute the `generate_input.sh` script as shown below. The first command `chmod` grants execute permission to the script so it can be run directly. The second command runs the script to generate the required input files. For more details on usage of bash scripting, refer to [Script Concepts](../../workflow_solutions/shell.md/#script-concepts) ```bash $chmod +x generate_input.sh @@ -341,7 +349,7 @@ $./generate_input.sh ##### Running the Slurm Array Job -This SLURM job script processes a text file line by line using an array job. Save the following Slurm script as `line_word_count.job`. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. +The following Slurm job script processes a text file line by line using an array job. Save the following Slurm script as `line_word_count.job` within `example_4` folder. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. ```bash linenums="1" #!/bin/bash @@ -370,23 +378,25 @@ echo "Task $SLURM_ARRAY_TASK_ID: $WORD_COUNT words" Before running the above SLURM job, determine the number of lines in the input file (e.g., random_file_2.txt) using the following command: ```bash -MAX_TASKS=$(wc -l < $HOME/input_files/random_file_2.txt) +$MAX_TASKS=$(wc -l < $HOME/input_files/random_file_2.txt) ``` Next, submit the array job using the command below, which creates an array of tasks from 1 to the maximum number of lines in the file. ```bash -sbatch --array=1-"$MAX_TASKS" line_word_count.job +$sbatch --array=1-"$MAX_TASKS" line_word_count.job ``` The below output comes from the SLURM job array script (line_word_count.job), where each task processes one line from random_file_2.txt. Since the file has 3 lines, SLURM created 3 tasks (Task 1, Task 2, Task 3), each counting words in its respective line. ```bash -$ wc -l $HOME/input_files/random_file_2.txt #counts the number of lines in the file, for instance, random_file_2.txt +### counts the number of lines in the file, for instance, random_file_2.txt +$ wc -l $HOME/input_files/random_file_2.txt 3 $HOME/input_files/random_file_2.txt ``` ```bash +### Print and verify the output using the `cat` command $ cat line_word_count_$JOB_ID_*.out Task 1: 8 words Task 2: 19 words From 9ea93f09b7108065d7dc4cb98207f48786d7071b Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 25 Mar 2025 10:51:20 -0500 Subject: [PATCH 15/24] fix typos example4.2 --- docs/cheaha/slurm/slurm_tutorial.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index fa8ea7d48..5d3f13c97 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -405,7 +405,7 @@ Task 3: 6 words #### Example 4.2: Dynamically Reading and Counting Words in Multiple Files dynamically Using a Slurm Job Array -This example job script performs the same function as [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job`. It utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. +This example job script performs the same function as [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job` within the `example_4` folder. It utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. ```bash linenums="1" #!/bin/bash @@ -450,28 +450,30 @@ fi The script begins by setting up the Slurm array job with required resources. The working directory is defined, and the script uses the `find` command to search for files matching the pattern random_file_*.txt within that working directory and its subdirectories, passing the resulting list of file paths to the sort command to ensure they are processed in order. The file corresponding to the current Slurm array task ID is selected, and its directory and base name are extracted. If the file exists, the script counts the words in the file using the `wc -w` command and saves the word count to a new file with a `.wordcount` extension in the same directory. If the file is not found, an error message is displayed. -Before you run the script `dynamic_file_word_count.job`, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path`$HOME/example_4input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. +Before you run the script `dynamic_file_word_count.job`, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path`$HOME/example_4/input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. ```bash -MAX_TASKS=$(find $HOME/example_4/input_files -type f -name "random_file_*.txt" | wc -l) +$MAX_TASKS=$(find $HOME/example_4/input_files -type f -name "random_file_*.txt" | wc -l) ``` Next, submit the array job using the command below, which creates an array of tasks from 1 to the value of MAX_TASKS, where each task corresponds to processing a different file listed in the array. ```bash -sbatch --array=1-"$MAX_TASKS" dynamic_file_word_count.job +$sbatch --array=1-"$MAX_TASKS" dynamic_file_word_count.job ``` -In the output below, each file was processed independently by a SLURM job array task. The task IDs 1, 2, 3, 4, and 5 correspond to the five different files being processed. The SLURM Job ID is `31934540`. Each output file contains the word count for a specific text file handled by its respective SLURM array task. +In the output below, each file was processed independently by a Slurm job array task. The task IDs 1, 2, 3, 4, and 5 correspond to the five different files being processed. For instance, here the Slurm Job ID is `31934540`. Each output file contains the word count for a specific text file handled by its respective Slurm array task. The `find` command in the output shows the word counts for each of the random_file*.txt files, where each `.wordcount` file contains the word count for its corresponding input file. ```bash - +### Listing all the output log files generated by the Slurm $ ls $HOME/example_4/logs/file_word_count*.out file_word_count_31934540_1.out file_word_count_31934540_3.out file_word_count_31934540_5.out file_word_count_31934540_2.out file_word_count_31934540_4.out +### This command finds all files with names matching "random_file*.wordcount" in the specified directory and subdirectories. +### For each file found, concatenate and display its contents (using 'cat') to show the word count and their file path. $ find $HOME/example_4/input_files/ -type f -name "random_file*.wordcount" -exec cat {} \; 81 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_1.txt 117 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_5.txt From 5ad36bcc26ba99d1df501d160812873d0eaf4a54 Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 25 Mar 2025 11:32:34 -0500 Subject: [PATCH 16/24] typos example4.3 --- docs/cheaha/slurm/slurm_tutorial.md | 44 ++++++++++++++++++----------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 5d3f13c97..b9baac60b 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -409,13 +409,13 @@ This example job script performs the same function as [Example 4.1](#example-41- ```bash linenums="1" #!/bin/bash -#SBATCH --job-name=file_word_count ### Name of the job -#SBATCH --cpus-per-task=1 ### Number of Tasks per CPU -#SBATCH --mem=4G ### Memory required, 4 gigabyte -#SBATCH --partition=express ### Cheaha Partition -#SBATCH --time=00:15:00 ### Estimated Time of Completion, 15 minutes -#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index -#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index +#SBATCH --job-name=dynamic_file_word_count ### Name of the job +#SBATCH --cpus-per-task=1 ### Number of Tasks per CPU +#SBATCH --mem=4G ### Memory required, 4 gigabyte +#SBATCH --partition=express ### Cheaha Partition +#SBATCH --time=00:15:00 ### Estimated Time of Completion, 15 minutes +#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index +#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index ### Define working directory WORKDIR="$HOME/example_4/input_files" @@ -468,9 +468,10 @@ The `find` command in the output shows the word counts for each of the random_fi ```bash ### Listing all the output log files generated by the Slurm -$ ls $HOME/example_4/logs/file_word_count*.out -file_word_count_31934540_1.out file_word_count_31934540_3.out file_word_count_31934540_5.out -file_word_count_31934540_2.out file_word_count_31934540_4.out +$ ls $HOME/example_4/logs/dynamic_file_word_count*.out +dynamic_file_word_count_31934540_1.out dynamic_file_word_count_31934540_3.out +dynamic_file_word_count_31934540_2.out dynamic_file_word_count_31934540_4.out +dynamic_file_word_count_31934540_5.out ### This command finds all files with names matching "random_file*.wordcount" in the specified directory and subdirectories. ### For each file found, concatenate and display its contents (using 'cat') to show the word count and their file path. @@ -490,10 +491,15 @@ This example job script is similar to [Example 4.2](#example-42-dynamically-read To create a file list by tracking all the files in the `input_files` directory, use the find command along with globbing to list all generated files in the current directory and its subdirectories. The input files along with its absolute path are tracked in `file_list.txt`. ```bash +### Change to the "example_4" directory. +cd example_4 +### Search for all files in the "input_files" directory matching the pattern "random_file_*.txt". +### For each matching file, the 'realpath' command is used to get the absolute path of the file. +### The output is then sorted and written to "file_list.txt". find $HOME/example_4/input_files -type f -name "random_file_*.txt" -exec realpath {} \; | sort > file_list.txt ``` -Copy the following SLURM array job script to a file naming `file_list_word_count.job`. This Slurm array job script count the number of words across multiple files listed in a file list `file_list.txt`. +Copy the following SLURM array job script to a file naming `file_list_word_count.job` within the `example_4` folder. This Slurm array job script count the number of words across multiple files listed in a file list `file_list.txt`. ```bash linenums="1" #!/bin/bash @@ -510,12 +516,15 @@ WORKDIR="$HOME/input_files" FILELIST="file_list.txt" ### Get the file corresponding to the current task ID +### 'sed' picks the file from the $FILELIST based on the task ID (SLURM_ARRAY_TASK_ID). FILE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" $FILELIST) + +### Extract the base name of the file (without the .txt extension) BASENAME=$(basename "$FILE" .txt) ### Check if file exists and count words from file if [[ -f "$FILE" ]]; then - # Count words and save result + ### 'wc -w' counts the words in the file, and the result is saved in a new file with '.wordcount' added to the name. wc -w "$FILE" > "$WORKDIR/${BASENAME}.wordcount" echo "Processed $FILE" else @@ -523,18 +532,21 @@ else fi ``` -The above SLURM job script runs a word count operation in parallel on multiple files using a job array (1-5). It reads a list of filenames along with its path from `file_list.txt` located in `$HOME/input_files`. Each task in the job array processes a different file based on its task ID (SLURM_ARRAY_TASK_ID). If the file exists, it counts the number of words using wc -w and saves the output as `.wordcount`, and logs standard output and errors for each task separately. +The above SLURM job script runs a word count operation in parallel on multiple files using a job array (1-5). It reads a list of filenames along with its path from `file_list.txt` located in `$HOME/example_4/input_files`. Each task in the job array processes a different file based on its task ID (SLURM_ARRAY_TASK_ID). If the file exists, it counts the number of words using `wc -w` and saves the output as `.wordcount`, and logs standard output and errors for each task separately. -Before running the above SLURM job, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path `$HOME/input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. +Before running the above SLURM job, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path `$HOME/example_4/input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. ```bash -MAX_TASKS=$(wc -l < file_list.txt) +### Get the total number of tasks by counting the lines in 'file_list.txt' +### 'wc -l' counts the number of lines in the file, which represents the number of tasks. +$MAX_TASKS=$(wc -l < file_list.txt) ``` Next, submit the array job using the command below, which creates an array of tasks from 1 to the value of MAX_TASKS, where each task corresponds to processing a different file listed in the array. ```bash -sbatch --array=1-"$MAX_TASKS" line_word_count.job +### Submit a job array with tasks ranging from 1 to MAX_TASKS +$sbatch --array=1-"$MAX_TASKS" line_word_count.job ``` The output generated will be similar to [Example 4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array), as the functionality is the same, but the method of handling the input data differs. From b5a0b74c5f24ea7d9f7b0c7978b1a39b98879dc1 Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 28 Mar 2025 12:20:20 -0500 Subject: [PATCH 17/24] retested code and updated --- docs/cheaha/slurm/slurm_tutorial.md | 32 ++++++++++++++--------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index b9baac60b..0e69cd2cc 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -378,7 +378,7 @@ echo "Task $SLURM_ARRAY_TASK_ID: $WORD_COUNT words" Before running the above SLURM job, determine the number of lines in the input file (e.g., random_file_2.txt) using the following command: ```bash -$MAX_TASKS=$(wc -l < $HOME/input_files/random_file_2.txt) +$MAX_TASKS=$(wc -l < $HOME/example_4/input_files/random_file_2.txt) ``` Next, submit the array job using the command below, which creates an array of tasks from 1 to the maximum number of lines in the file. @@ -391,13 +391,13 @@ The below output comes from the SLURM job array script (line_word_count.job), wh ```bash ### counts the number of lines in the file, for instance, random_file_2.txt -$ wc -l $HOME/input_files/random_file_2.txt -3 $HOME/input_files/random_file_2.txt +$ wc -l /home/$USER/example_4/input_files/random_file_2.txt +3 $HOME/example_4/input_files/random_file_2.txt ``` ```bash ### Print and verify the output using the `cat` command -$ cat line_word_count_$JOB_ID_*.out +$ cat /home/$USER/example_4/logs/line_word_count_$JOB_ID_*.out Task 1: 8 words Task 2: 19 words Task 3: 6 words @@ -467,8 +467,9 @@ In the output below, each file was processed independently by a Slurm job array The `find` command in the output shows the word counts for each of the random_file*.txt files, where each `.wordcount` file contains the word count for its corresponding input file. ```bash -### Listing all the output log files generated by the Slurm -$ ls $HOME/example_4/logs/dynamic_file_word_count*.out +### Listing all the output files generated by the Slurm from the `logs` directory +$ cd $HOME/example_4/logs/ +$ ls dynamic_file_word_count*.out dynamic_file_word_count_31934540_1.out dynamic_file_word_count_31934540_3.out dynamic_file_word_count_31934540_2.out dynamic_file_word_count_31934540_4.out dynamic_file_word_count_31934540_5.out @@ -476,12 +477,11 @@ dynamic_file_word_count_31934540_5.out ### This command finds all files with names matching "random_file*.wordcount" in the specified directory and subdirectories. ### For each file found, concatenate and display its contents (using 'cat') to show the word count and their file path. $ find $HOME/example_4/input_files/ -type f -name "random_file*.wordcount" -exec cat {} \; -81 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_1.txt -117 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_5.txt -33 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_2.txt -104 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_3.txt -81 /home/prema/Tutorial/slurm_tutorial/example4/input_files/test/random_file_6.txt -114 /home/prema/Tutorial/slurm_tutorial/example4/input_files/random_file_4.txt +81 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_1.txt +117 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_5.txt +33 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_2.txt +104 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_3.txt +114 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_4.txt ``` #### Example 4.3: Counting Words in Multiple Files from a File List Using a Slurm Job Array @@ -503,7 +503,7 @@ Copy the following SLURM array job script to a file naming `file_list_word_count ```bash linenums="1" #!/bin/bash -#SBATCH --job-name=file_word_count ### Name of the job +#SBATCH --job-name=file_list_word_count ### Name of the job #SBATCH --cpus-per-task=1 ### Number of Tasks per CPU #SBATCH --mem=4G ### Memory required, 4 gigabyte #SBATCH --partition=express ### Cheaha Partition @@ -512,7 +512,7 @@ Copy the following SLURM array job script to a file naming `file_list_word_count #SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index ### Define working directory -WORKDIR="$HOME/input_files" +WORKDIR="$HOME/example_4/input_files" FILELIST="file_list.txt" ### Get the file corresponding to the current task ID @@ -539,14 +539,14 @@ Before running the above SLURM job, determine the number of files in the directo ```bash ### Get the total number of tasks by counting the lines in 'file_list.txt' ### 'wc -l' counts the number of lines in the file, which represents the number of tasks. -$MAX_TASKS=$(wc -l < file_list.txt) +$MAX_TASKS=$(wc -l < $HOME/example_4/file_list.txt) ``` Next, submit the array job using the command below, which creates an array of tasks from 1 to the value of MAX_TASKS, where each task corresponds to processing a different file listed in the array. ```bash ### Submit a job array with tasks ranging from 1 to MAX_TASKS -$sbatch --array=1-"$MAX_TASKS" line_word_count.job +$sbatch --array=1-"$MAX_TASKS" file_list_word_count.job ``` The output generated will be similar to [Example 4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array), as the functionality is the same, but the method of handling the input data differs. From f0d6adcc8f7d310d6dab0f17fba2c1161e5da7c4 Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 28 Mar 2025 12:28:32 -0500 Subject: [PATCH 18/24] headings changed --- docs/cheaha/slurm/slurm_tutorial.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 0e69cd2cc..abb48b214 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -300,9 +300,9 @@ $ sacct -j 27101430 27101430_2.+ extern USER 1 COMPLETED 0:0 ``` -#### Example 4.1: Slurm Array Job for Line-by-Line Word Count +#### Example 4.1: Line-by-Line Word Count -In the examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count), [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array), and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list-using-a-slurm-job-array), let us explore how to use a Slurm Array Job to process text files in parallel. To keep things organized, we divide the process into two main sections: +In the examples [4.1](#example-41-line-by-line-word-count), [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files), and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list), let us explore how to use a Slurm Array Job to process text files in parallel. To keep things organized, we divide the process into two main sections: (i) Setup: This section covers creating the necessary directories and input files. Since this setup is common for both examples, we will do it once before moving on. @@ -310,7 +310,7 @@ In the examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count), ##### Setup -Before executing the Slurm array examples [4.1](#example-41-slurm-array-job-for-line-by-line-word-count) and [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array) and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list-using-a-slurm-job-array), we need to set up the environment. This includes creating a main directory, `example_4`, to store all job files and scripts. Within it, we will create two subdirectories: `input_files` to store the generated input text files and `logs` to organize output and error logs. Let us create and structure these directories. +Before executing the Slurm array examples [4.1](#example-41-line-by-line-word-count) and [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files) and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list), we need to set up the environment. This includes creating a main directory, `example_4`, to store all job files and scripts. Within it, we will create two subdirectories: `input_files` to store the generated input text files and `logs` to organize output and error logs. Let us create and structure these directories. ```bash $mkdir example_4 @@ -347,8 +347,6 @@ $chmod +x generate_input.sh $./generate_input.sh ``` -##### Running the Slurm Array Job - The following Slurm job script processes a text file line by line using an array job. Save the following Slurm script as `line_word_count.job` within `example_4` folder. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. ```bash linenums="1" @@ -403,9 +401,9 @@ Task 2: 19 words Task 3: 6 words ``` -#### Example 4.2: Dynamically Reading and Counting Words in Multiple Files dynamically Using a Slurm Job Array +#### Example 4.2: Dynamically Reading and Counting Words in Multiple Files -This example job script performs the same function as [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job` within the `example_4` folder. It utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. +This example job script performs the same function as [Example 4.1](#example-41-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job` within the `example_4` folder. It utilizes the same input files from [Example 4.1](#example-41-line-by-line-word-count) as described in the [Setup](#setup) section. ```bash linenums="1" #!/bin/bash @@ -484,9 +482,9 @@ $ find $HOME/example_4/input_files/ -type f -name "random_file*.wordcount" -exec 114 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_4.txt ``` -#### Example 4.3: Counting Words in Multiple Files from a File List Using a Slurm Job Array +#### Example 4.3: Counting Words in Multiple Files from a File List -This example job script is similar to [Example 4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array), but instead of dynamically reading files from the directory, it counts the number of words in multiple files in parallel using a SLURM job array, with the files listed in a separate file list. This example utilizes the same input files from [Example 4.1](#example-41-slurm-array-job-for-line-by-line-word-count) as described in the [Setup](#setup) section. +This example job script is similar to [Example 4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files), but instead of dynamically reading files from the directory, it counts the number of words in multiple files in parallel using a SLURM job array, with the files listed in a separate file list. This example utilizes the same input files from [Example 4.1](#example-41-line-by-line-word-count) as described in the [Setup](#setup) section. To create a file list by tracking all the files in the `input_files` directory, use the find command along with globbing to list all generated files in the current directory and its subdirectories. The input files along with its absolute path are tracked in `file_list.txt`. @@ -549,7 +547,7 @@ Next, submit the array job using the command below, which creates an array of ta $sbatch --array=1-"$MAX_TASKS" file_list_word_count.job ``` -The output generated will be similar to [Example 4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files-dynamically-using-a-slurm-job-array), as the functionality is the same, but the method of handling the input data differs. +The output generated will be similar to [Example 4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files), as the functionality is the same, but the method of handling the input data differs. ### Example 5: Multithreaded or Multicore Job From fa9f39982f25e71e8b971710d5e3e35f98d6fa26 Mon Sep 17 00:00:00 2001 From: Premas Date: Mon, 31 Mar 2025 17:01:02 -0500 Subject: [PATCH 19/24] updated to four examples --- docs/cheaha/slurm/slurm_tutorial.md | 118 +++++++++++++++------------- 1 file changed, 64 insertions(+), 54 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index abb48b214..5a70e3f2b 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -239,7 +239,58 @@ $ sacct -j 27099591 Array jobs are more effective when you have a larger number of similar tasks to be executed simultaneously with varied input data, unlike `srun` parallel jobs which are suitable for running a smaller number of tasks concurrently (e.g. less than 5). Array jobs are easier to manage and monitor multiple tasks through unique identifiers. -The following Slurm script is an example of how you might convert the previous `multijob` script to an array job. To start, copy the below script to a file named, `slurm_array.job`. The script requires the input file `python_script_new.py` and the `conda` environment `pytools-env`, similar to those used in [example2](../slurm/slurm_tutorial.md#example-2-sequential-job) and [example 3](../slurm/slurm_tutorial.md#example-3-parallel-jobs). Line 11 specifies the script as an array job, treating each task within the array as an independent job. For each task, lines 18-19 calculates the input range. `SLURM_ARRAY_TASK_ID` identifies the task executed using indexes, and is automatically set for array jobs. The python script (line 22) runs individual array task concurrently on respective input range. The command `awk` is used to prepend each output line with the unique task identifier and then append the results to the file, `output_all_tasks.txt`. For more details on on parameters of array jobs, please refer to [Batch Array Jobs](../slurm/submitting_jobs.md#batch-array-jobs-with-known-indices) and [Practical Batch Array Jobs](../slurm/practical_sbatch.md#). +To keep things organized, let us first structure the directories and implement necessary preprocessing step required for the subsequent Slurm array examples. + +**(i) Setup**: This part of the section covers creating the necessary directories and input files. Since this setup is common for all the following examples, we will do it once before moving on. + +Before executing the Slurm array examples [4.1](#example-41-running-parallel-python-tasks-with-dynamic-input-ranges), [4.2](#example-42-line-by-line-word-count) and [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files) and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us set up the environment. This includes creating a main directory, `example_4`, to store all job files and scripts. Within it, we will create a directory `logs` to organize output and error logs. Let us create and structure these directories. + +```bash +$mkdir example_4 +$cd example_4 +$mkdir logs +``` + +**(ii) Input File Generation**: For the slurm array examples [4.2](#example-42-line-by-line-word-count) and [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files) and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us create subdirectory `input_files` within the `example_4` folder to store a set of generated input text files. + +```bash +$cd example_4 +$mkdir input_files +$cd input_files +``` + +The next step involves generating input files using a pseudo-random text generation script. Save the following script as `generate_input.sh` within the `input_files` folder. This script generates five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. For more details refer to [script concepts](../../workflow_solutions/shell.md#script-concepts) and [bash scripting example](../../cheaha/software/modules.md#best-practice-for-loading-modules). + +```bash +#!/bin/bash + +### Loop to create 5 random files +for i in {1..5}; do + ### Generate a random number of lines (1-10) + num_lines=$(($RANDOM % 10 + 1)) + for _ in $(seq 1 $num_lines); do + ### Generate a random words per line (5-20) + num_words=$(($RANDOM % 20 + 5)) + ### Use 'shuf' to pick $num_words random words from the dictionary file. + ### The pipe ('|') sends these selected words to 'paste', which combines them into a single line, + ### separating each word with a space (using the '-d " "' option). + shuf -n $num_words /usr/share/dict/words | paste -s -d " " + done > "random_file_$i.txt" +done +``` + +You can execute the `generate_input.sh` script as shown below. The first command `chmod` grants execute permission to the script so it can be run directly. The second command runs the script to generate the required input files. For more details on usage of bash scripting, refer to [Script Concepts](../../workflow_solutions/shell.md/#script-concepts) + +```bash +$chmod +x generate_input.sh +$./generate_input.sh +``` + +Now that the general environment and preprocessing setup are complete, the following sections will walk you through four detailed Slurm array job examples. + +#### Example 4.1: Running Parallel Python Tasks with Dynamic Input Ranges + +The following Slurm script is an example of how you might convert the previous [parallel job example](#example-3-parallel-jobs) script to an array job. To start, copy and save the below script to a file named, `slurm_array.job` within the `example_4` folder. The script requires the input file `python_script_new.py` and the `conda` environment `pytools-env`, similar to those used in [example2](../slurm/slurm_tutorial.md#example-2-sequential-job) and [example 3](../slurm/slurm_tutorial.md#example-3-parallel-jobs). Line 11 specifies the script as an array job, treating each task within the array as an independent job. For each task, lines 18-19 calculates the input range. `SLURM_ARRAY_TASK_ID` identifies the task executed using indexes, and is automatically set for array jobs. The python script (line 22) runs individual array task concurrently on respective input range. The command `awk` is used to prepend each output line with the unique task identifier and then append the results to the file, `output_all_tasks.txt`. For more details on on parameters of array jobs, please refer to [Batch Array Jobs](../slurm/submitting_jobs.md#batch-array-jobs-with-known-indices) and [Practical Batch Array Jobs](../slurm/practical_sbatch.md#). !!! important @@ -256,8 +307,8 @@ The following Slurm script is an example of how you might convert the previous ` #SBATCH --mem=4G ### Memory required, 4 gigabyte #SBATCH --partition=express ### Cheaha Partition #SBATCH --time=01:00:00 ### Estimated Time of Completion, 1 hour -#SBATCH --output=%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index -#SBATCH --error=%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index +#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index +#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index #SBATCH --array=1-3 ### Number of Slurm array tasks, 3 tasks ### Loading Anaconda3 module to activate `pytools-env` conda environment @@ -275,7 +326,7 @@ python python_script_new.py $start $end 2>&1 | awk -v task_id=$SLURM_ARRAY_TASK_ The output shows the sum of different input range computed by individual task, making it easy to track using a task identifier, such as array task 1/2/3. ```bash -$ cat output_all_tasks.txt +$ cat $HOME/example_4/output_all_tasks.txt array task 2 Input Range: 100001 to 200000, Sum: 14999850000 array task 3 Input Range: 200001 to 300000, Sum: 24999750000 @@ -300,54 +351,13 @@ $ sacct -j 27101430 27101430_2.+ extern USER 1 COMPLETED 0:0 ``` -#### Example 4.1: Line-by-Line Word Count - -In the examples [4.1](#example-41-line-by-line-word-count), [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files), and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list), let us explore how to use a Slurm Array Job to process text files in parallel. To keep things organized, we divide the process into two main sections: - -(i) Setup: This section covers creating the necessary directories and input files. Since this setup is common for both examples, we will do it once before moving on. - -(ii) Running the Slurm Array Job: Here, we focus on understanding and executing an example Slurm array job. +In the following three examples, we will explore additional Slurm array job scenarios that are commonly used in scientific simulations. -##### Setup +#### Example 4.2: Line-by-Line Word Count -Before executing the Slurm array examples [4.1](#example-41-line-by-line-word-count) and [4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files) and [4.3](#example-43-counting-words-in-multiple-files-from-a-file-list), we need to set up the environment. This includes creating a main directory, `example_4`, to store all job files and scripts. Within it, we will create two subdirectories: `input_files` to store the generated input text files and `logs` to organize output and error logs. Let us create and structure these directories. - -```bash -$mkdir example_4 -$cd example_4 -$mkdir input_files -$mkdir logs -$cd input_files -``` - -The next step involves generating input files using a pseudo-random text generation script. Save the following script as `generate_input.sh` within the `input_files` folder. This script generates five random text files, each containing a different number of randomly selected words. The script creates a file with 1 to 10 randomly generated lines, where each line contains 5 to 20 randomly selected words from `/usr/share/dict/words`. The output is saved in files named random_file_1.txt, random_file_2.txt, ..., random_file_5.txt, with each file containing a unique, randomly determined number of lines and words per line. For more details refer to [script concepts](../../workflow_solutions/shell.md#script-concepts) and [bash scripting example](../../cheaha/software/modules.md#best-practice-for-loading-modules). - -```bash -#!/bin/bash - -### Loop to create 5 random files -for i in {1..5}; do - ### Generate a random number of lines (1-10) - num_lines=$(($RANDOM % 10 + 1)) - for _ in $(seq 1 $num_lines); do - ### Generate a random words per line (5-20) - num_words=$(($RANDOM % 20 + 5)) - ### Use 'shuf' to pick $num_words random words from the dictionary file. - ### The pipe ('|') sends these selected words to 'paste', which combines them into a single line, - ### separating each word with a space (using the '-d " "' option). - shuf -n $num_words /usr/share/dict/words | paste -s -d " " - done > "random_file_$i.txt" -done -``` - -You can execute the `generate_input.sh` script as shown below. The first command `chmod` grants execute permission to the script so it can be run directly. The second command runs the script to generate the required input files. For more details on usage of bash scripting, refer to [Script Concepts](../../workflow_solutions/shell.md/#script-concepts) - -```bash -$chmod +x generate_input.sh -$./generate_input.sh -``` +In the examples [4.2](#example-42-line-by-line-word-count), [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files), and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us explore how to use a Slurm Array Job to process text files in parallel. -The following Slurm job script processes a text file line by line using an array job. Save the following Slurm script as `line_word_count.job` within `example_4` folder. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. +The following Slurm job script processes a text file line by line using an array job. Save the following Slurm script as `line_word_count.job` within `example_4` folder. Prior to running this job make sure to complete the [input file generation](#example-4-array-job) step. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. ```bash linenums="1" #!/bin/bash @@ -401,9 +411,9 @@ Task 2: 19 words Task 3: 6 words ``` -#### Example 4.2: Dynamically Reading and Counting Words in Multiple Files +#### Example 4.3: Dynamically Reading and Counting Words in Multiple Files -This example job script performs the same function as [Example 4.1](#example-41-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job` within the `example_4` folder. It utilizes the same input files from [Example 4.1](#example-41-line-by-line-word-count) as described in the [Setup](#setup) section. +This example job script performs the same function as [Example 4.2](#example-42-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job` within the `example_4` folder. It utilizes the same input files generated from the [Input File Generation](#example-4-array-job) section. ```bash linenums="1" #!/bin/bash @@ -482,9 +492,9 @@ $ find $HOME/example_4/input_files/ -type f -name "random_file*.wordcount" -exec 114 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_4.txt ``` -#### Example 4.3: Counting Words in Multiple Files from a File List +#### Example 4.4: Counting Words in Multiple Files from a File List -This example job script is similar to [Example 4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files), but instead of dynamically reading files from the directory, it counts the number of words in multiple files in parallel using a SLURM job array, with the files listed in a separate file list. This example utilizes the same input files from [Example 4.1](#example-41-line-by-line-word-count) as described in the [Setup](#setup) section. +This example job script is similar to [Example 4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files), but instead of dynamically reading files from the directory, it counts the number of words in multiple files in parallel using a SLURM job array, with the files listed in a separate file list. This example utilizes the same input files from [Example 4](#example-4-array-job) as described in the `Input File Generation` section. To create a file list by tracking all the files in the `input_files` directory, use the find command along with globbing to list all generated files in the current directory and its subdirectories. The input files along with its absolute path are tracked in `file_list.txt`. @@ -547,7 +557,7 @@ Next, submit the array job using the command below, which creates an array of ta $sbatch --array=1-"$MAX_TASKS" file_list_word_count.job ``` -The output generated will be similar to [Example 4.2](#example-42-dynamically-reading-and-counting-words-in-multiple-files), as the functionality is the same, but the method of handling the input data differs. +The output generated will be similar to [Example 4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files), as the functionality is the same, but the method of handling the input data differs. ### Example 5: Multithreaded or Multicore Job From 9311eabc4e91640e52998513c79fa5408e32deec Mon Sep 17 00:00:00 2001 From: Premas Date: Wed, 16 Apr 2025 11:10:59 -0500 Subject: [PATCH 20/24] rewording --- docs/cheaha/slurm/slurm_tutorial.md | 58 +++++++++++++++-------------- 1 file changed, 30 insertions(+), 28 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 5a70e3f2b..f8d573b93 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -237,24 +237,26 @@ $ sacct -j 27099591 ### Example 4: Array Job -Array jobs are more effective when you have a larger number of similar tasks to be executed simultaneously with varied input data, unlike `srun` parallel jobs which are suitable for running a smaller number of tasks concurrently (e.g. less than 5). Array jobs are easier to manage and monitor multiple tasks through unique identifiers. +Array jobs are more effective when you have a larger number of similar tasks to be executed simultaneously with varied input data, unlike `srun` parallel jobs which are suitable for running a smaller number of tasks concurrently (e.g. less than 5). Array jobs are easier to manage and monitor multiple tasks through unique identifiers. However, with the increased power come more moving parts, so the example here requires some setup. + +#### Environment Setup for Slurm Array Execution To keep things organized, let us first structure the directories and implement necessary preprocessing step required for the subsequent Slurm array examples. **(i) Setup**: This part of the section covers creating the necessary directories and input files. Since this setup is common for all the following examples, we will do it once before moving on. -Before executing the Slurm array examples [4.1](#example-41-running-parallel-python-tasks-with-dynamic-input-ranges), [4.2](#example-42-line-by-line-word-count) and [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files) and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us set up the environment. This includes creating a main directory, `example_4`, to store all job files and scripts. Within it, we will create a directory `logs` to organize output and error logs. Let us create and structure these directories. +Before executing the Slurm array examples [4.1](#example-41-running-parallel-python-tasks-with-dynamic-input-ranges), [4.2](#example-42-line-by-line-word-count) and [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files) and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us set up the environment. This includes creating a main directory, `array_example`, to store all job files and scripts. Within it, we will create a directory `logs` to organize output and error logs. Let us create and structure these directories. ```bash -$mkdir example_4 -$cd example_4 +$mkdir array_example +$cd array_example $mkdir logs ``` -**(ii) Input File Generation**: For the slurm array examples [4.2](#example-42-line-by-line-word-count) and [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files) and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us create subdirectory `input_files` within the `example_4` folder to store a set of generated input text files. +**(ii) Input File Generation**: For the slurm array examples [4.2](#example-42-line-by-line-word-count) and [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files) and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us create subdirectory `input_files` within the `array_example` folder to store a set of generated input text files. ```bash -$cd example_4 +$cd array_example $mkdir input_files $cd input_files ``` @@ -290,7 +292,7 @@ Now that the general environment and preprocessing setup are complete, the follo #### Example 4.1: Running Parallel Python Tasks with Dynamic Input Ranges -The following Slurm script is an example of how you might convert the previous [parallel job example](#example-3-parallel-jobs) script to an array job. To start, copy and save the below script to a file named, `slurm_array.job` within the `example_4` folder. The script requires the input file `python_script_new.py` and the `conda` environment `pytools-env`, similar to those used in [example2](../slurm/slurm_tutorial.md#example-2-sequential-job) and [example 3](../slurm/slurm_tutorial.md#example-3-parallel-jobs). Line 11 specifies the script as an array job, treating each task within the array as an independent job. For each task, lines 18-19 calculates the input range. `SLURM_ARRAY_TASK_ID` identifies the task executed using indexes, and is automatically set for array jobs. The python script (line 22) runs individual array task concurrently on respective input range. The command `awk` is used to prepend each output line with the unique task identifier and then append the results to the file, `output_all_tasks.txt`. For more details on on parameters of array jobs, please refer to [Batch Array Jobs](../slurm/submitting_jobs.md#batch-array-jobs-with-known-indices) and [Practical Batch Array Jobs](../slurm/practical_sbatch.md#). +The following Slurm script is an example of how you might convert the previous [parallel job example](#example-3-parallel-jobs) script to an array job. To start, copy and save the below script to a file named, `slurm_array.job` within the `array_example` folder. The script requires the input file `python_script_new.py` and the `conda` environment `pytools-env`, similar to those used in [example2](../slurm/slurm_tutorial.md#example-2-sequential-job) and [example 3](../slurm/slurm_tutorial.md#example-3-parallel-jobs). Line 11 specifies the script as an array job, treating each task within the array as an independent job. For each task, lines 18-19 calculates the input range. `SLURM_ARRAY_TASK_ID` identifies the task executed using indexes, and is automatically set for array jobs. The python script (line 22) runs individual array task concurrently on respective input range. The command `awk` is used to prepend each output line with the unique task identifier and then append the results to the file, `output_all_tasks.txt`. For more details on on parameters of array jobs, please refer to [Batch Array Jobs](../slurm/submitting_jobs.md#batch-array-jobs-with-known-indices) and [Practical Batch Array Jobs](../slurm/practical_sbatch.md#). !!! important @@ -326,7 +328,7 @@ python python_script_new.py $start $end 2>&1 | awk -v task_id=$SLURM_ARRAY_TASK_ The output shows the sum of different input range computed by individual task, making it easy to track using a task identifier, such as array task 1/2/3. ```bash -$ cat $HOME/example_4/output_all_tasks.txt +$ cat $HOME/array_example/output_all_tasks.txt array task 2 Input Range: 100001 to 200000, Sum: 14999850000 array task 3 Input Range: 200001 to 300000, Sum: 24999750000 @@ -357,7 +359,7 @@ In the following three examples, we will explore additional Slurm array job scen In the examples [4.2](#example-42-line-by-line-word-count), [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files), and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us explore how to use a Slurm Array Job to process text files in parallel. -The following Slurm job script processes a text file line by line using an array job. Save the following Slurm script as `line_word_count.job` within `example_4` folder. Prior to running this job make sure to complete the [input file generation](#example-4-array-job) step. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. +The following Slurm job script processes a text file line by line using an array job. Save the following Slurm script as `line_word_count.job` within `array_example` folder. Prior to running this job make sure to complete the [input file generation](#example-4-array-job) step. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. ```bash linenums="1" #!/bin/bash @@ -370,7 +372,7 @@ The following Slurm job script processes a text file line by line using an array #SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index ### Define the input file, for instance, random_file_2.txt -INPUT_FILE="$HOME/example_4/input_files/random_file_2.txt" +INPUT_FILE="$HOME/array_example/input_files/random_file_2.txt" ### `sed` is a shell command and stream editor for manipulating text. ### Here, `sed` reads $INPUT_FILE and extracts the line corresponding to SLURM_ARRAY_TASK_ID. @@ -386,7 +388,7 @@ echo "Task $SLURM_ARRAY_TASK_ID: $WORD_COUNT words" Before running the above SLURM job, determine the number of lines in the input file (e.g., random_file_2.txt) using the following command: ```bash -$MAX_TASKS=$(wc -l < $HOME/example_4/input_files/random_file_2.txt) +$MAX_TASKS=$(wc -l < $HOME/array_example/input_files/random_file_2.txt) ``` Next, submit the array job using the command below, which creates an array of tasks from 1 to the maximum number of lines in the file. @@ -399,13 +401,13 @@ The below output comes from the SLURM job array script (line_word_count.job), wh ```bash ### counts the number of lines in the file, for instance, random_file_2.txt -$ wc -l /home/$USER/example_4/input_files/random_file_2.txt -3 $HOME/example_4/input_files/random_file_2.txt +$ wc -l /home/$USER/array_example/input_files/random_file_2.txt +3 $HOME/array_example/input_files/random_file_2.txt ``` ```bash ### Print and verify the output using the `cat` command -$ cat /home/$USER/example_4/logs/line_word_count_$JOB_ID_*.out +$ cat /home/$USER/array_example/logs/line_word_count_$JOB_ID_*.out Task 1: 8 words Task 2: 19 words Task 3: 6 words @@ -413,7 +415,7 @@ Task 3: 6 words #### Example 4.3: Dynamically Reading and Counting Words in Multiple Files -This example job script performs the same function as [Example 4.2](#example-42-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job` within the `example_4` folder. It utilizes the same input files generated from the [Input File Generation](#example-4-array-job) section. +This example job script performs the same function as [Example 4.2](#example-42-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job` within the `array_example` folder. It utilizes the same input files generated from the [Input File Generation](#example-4-array-job) section. ```bash linenums="1" #!/bin/bash @@ -426,7 +428,7 @@ This example job script performs the same function as [Example 4.2](#example-42- #SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index ### Define working directory -WORKDIR="$HOME/example_4/input_files" +WORKDIR="$HOME/array_example/input_files" ### Find all files in the working directory ($WORKDIR) that match the pattern "random_file_*.txt" ### Pass the list of these files to the 'sort' command via the pipe ('|') @@ -458,10 +460,10 @@ fi The script begins by setting up the Slurm array job with required resources. The working directory is defined, and the script uses the `find` command to search for files matching the pattern random_file_*.txt within that working directory and its subdirectories, passing the resulting list of file paths to the sort command to ensure they are processed in order. The file corresponding to the current Slurm array task ID is selected, and its directory and base name are extracted. If the file exists, the script counts the words in the file using the `wc -w` command and saves the word count to a new file with a `.wordcount` extension in the same directory. If the file is not found, an error message is displayed. -Before you run the script `dynamic_file_word_count.job`, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path`$HOME/example_4/input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. +Before you run the script `dynamic_file_word_count.job`, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path`$HOME/array_example/input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. ```bash -$MAX_TASKS=$(find $HOME/example_4/input_files -type f -name "random_file_*.txt" | wc -l) +$MAX_TASKS=$(find $HOME/array_example/input_files -type f -name "random_file_*.txt" | wc -l) ``` Next, submit the array job using the command below, which creates an array of tasks from 1 to the value of MAX_TASKS, where each task corresponds to processing a different file listed in the array. @@ -476,7 +478,7 @@ The `find` command in the output shows the word counts for each of the random_fi ```bash ### Listing all the output files generated by the Slurm from the `logs` directory -$ cd $HOME/example_4/logs/ +$ cd $HOME/array_example/logs/ $ ls dynamic_file_word_count*.out dynamic_file_word_count_31934540_1.out dynamic_file_word_count_31934540_3.out dynamic_file_word_count_31934540_2.out dynamic_file_word_count_31934540_4.out @@ -484,7 +486,7 @@ dynamic_file_word_count_31934540_5.out ### This command finds all files with names matching "random_file*.wordcount" in the specified directory and subdirectories. ### For each file found, concatenate and display its contents (using 'cat') to show the word count and their file path. -$ find $HOME/example_4/input_files/ -type f -name "random_file*.wordcount" -exec cat {} \; +$ find $HOME/array_example/input_files/ -type f -name "random_file*.wordcount" -exec cat {} \; 81 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_1.txt 117 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_5.txt 33 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_2.txt @@ -499,15 +501,15 @@ This example job script is similar to [Example 4.3](#example-43-dynamically-read To create a file list by tracking all the files in the `input_files` directory, use the find command along with globbing to list all generated files in the current directory and its subdirectories. The input files along with its absolute path are tracked in `file_list.txt`. ```bash -### Change to the "example_4" directory. -cd example_4 +### Change to the "array_example" directory. +cd array_example ### Search for all files in the "input_files" directory matching the pattern "random_file_*.txt". ### For each matching file, the 'realpath' command is used to get the absolute path of the file. ### The output is then sorted and written to "file_list.txt". -find $HOME/example_4/input_files -type f -name "random_file_*.txt" -exec realpath {} \; | sort > file_list.txt +find $HOME/array_example/input_files -type f -name "random_file_*.txt" -exec realpath {} \; | sort > file_list.txt ``` -Copy the following SLURM array job script to a file naming `file_list_word_count.job` within the `example_4` folder. This Slurm array job script count the number of words across multiple files listed in a file list `file_list.txt`. +Copy the following SLURM array job script to a file naming `file_list_word_count.job` within the `array_example` folder. This Slurm array job script count the number of words across multiple files listed in a file list `file_list.txt`. ```bash linenums="1" #!/bin/bash @@ -520,7 +522,7 @@ Copy the following SLURM array job script to a file naming `file_list_word_count #SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index ### Define working directory -WORKDIR="$HOME/example_4/input_files" +WORKDIR="$HOME/array_example/input_files" FILELIST="file_list.txt" ### Get the file corresponding to the current task ID @@ -540,14 +542,14 @@ else fi ``` -The above SLURM job script runs a word count operation in parallel on multiple files using a job array (1-5). It reads a list of filenames along with its path from `file_list.txt` located in `$HOME/example_4/input_files`. Each task in the job array processes a different file based on its task ID (SLURM_ARRAY_TASK_ID). If the file exists, it counts the number of words using `wc -w` and saves the output as `.wordcount`, and logs standard output and errors for each task separately. +The above SLURM job script runs a word count operation in parallel on multiple files using a job array (1-5). It reads a list of filenames along with its path from `file_list.txt` located in `$HOME/array_example/input_files`. Each task in the job array processes a different file based on its task ID (SLURM_ARRAY_TASK_ID). If the file exists, it counts the number of words using `wc -w` and saves the output as `.wordcount`, and logs standard output and errors for each task separately. -Before running the above SLURM job, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path `$HOME/example_4/input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. +Before running the above SLURM job, determine the number of files in the directory and its subdirectories using the following command. The command counts the number of files matching the pattern random_file_*.txt in the path `$HOME/array_example/input_files` directory and its subdirectories, and stores the result in the variable `MAX_TASKS`. ```bash ### Get the total number of tasks by counting the lines in 'file_list.txt' ### 'wc -l' counts the number of lines in the file, which represents the number of tasks. -$MAX_TASKS=$(wc -l < $HOME/example_4/file_list.txt) +$MAX_TASKS=$(wc -l < $HOME/array_example/file_list.txt) ``` Next, submit the array job using the command below, which creates an array of tasks from 1 to the value of MAX_TASKS, where each task corresponds to processing a different file listed in the array. From 521465e6127f3eec6b8ecf46d5a3abdb39ad14a0 Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 29 Apr 2025 10:13:07 -0500 Subject: [PATCH 21/24] fix typos --- docs/cheaha/slurm/slurm_tutorial.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index f8d573b93..2fa987836 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -309,8 +309,8 @@ The following Slurm script is an example of how you might convert the previous [ #SBATCH --mem=4G ### Memory required, 4 gigabyte #SBATCH --partition=express ### Cheaha Partition #SBATCH --time=01:00:00 ### Estimated Time of Completion, 1 hour -#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index -#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index +#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index +#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index #SBATCH --array=1-3 ### Number of Slurm array tasks, 3 tasks ### Loading Anaconda3 module to activate `pytools-env` conda environment @@ -322,7 +322,16 @@ start=$((($SLURM_ARRAY_TASK_ID - 1) * 100000 + 1)) end=$(($SLURM_ARRAY_TASK_ID * 100000)) ### Run the python script with input arguments and append the results to a .txt file for each task -python python_script_new.py $start $end 2>&1 | awk -v task_id=$SLURM_ARRAY_TASK_ID '{print "array task " task_id, $0}' >> output_all_tasks.txt +python python_script_new.py $start $end 2>&1 \ + | awk -v task_id=$SLURM_ARRAY_TASK_ID '{print "array task " task_id, $0}' \ + >> output_all_tasks.txt +``` + +Submit the script `slurm_array` for execution using the command, + +```bash +$cd array_example +$sbatch slurm_array.job ``` The output shows the sum of different input range computed by individual task, making it easy to track using a task identifier, such as array task 1/2/3. From a00bcfa11e85d54c0366a06e8a0782632c09e382 Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 16 May 2025 11:57:35 -0500 Subject: [PATCH 22/24] links and comments --- docs/cheaha/slurm/slurm_tutorial.md | 53 +++++++++++++++++------------ 1 file changed, 32 insertions(+), 21 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 2fa987836..c496c5132 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -245,7 +245,7 @@ To keep things organized, let us first structure the directories and implement n **(i) Setup**: This part of the section covers creating the necessary directories and input files. Since this setup is common for all the following examples, we will do it once before moving on. -Before executing the Slurm array examples [4.1](#example-41-running-parallel-python-tasks-with-dynamic-input-ranges), [4.2](#example-42-line-by-line-word-count) and [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files) and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us set up the environment. This includes creating a main directory, `array_example`, to store all job files and scripts. Within it, we will create a directory `logs` to organize output and error logs. Let us create and structure these directories. +Before executing the Slurm array examples [parallel-python-tasks](#example-41-running-parallel-python-tasks-with-dynamic-input-ranges), [word-count-line-by-line](#example-42-line-by-line-word-count) and [dynamic-word-count-multiple-files](#example-43-dynamically-reading-and-counting-words-in-multiple-files) and [word-count-from-file-list](#example-44-counting-words-in-multiple-files-from-a-file-list), let us set up the environment. This includes creating a main directory, `array_example`, to store all job files and scripts. Within it, we will create a directory `logs` to organize output and error logs. Let us create and structure these directories. ```bash $mkdir array_example @@ -253,7 +253,7 @@ $cd array_example $mkdir logs ``` -**(ii) Input File Generation**: For the slurm array examples [4.2](#example-42-line-by-line-word-count) and [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files) and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us create subdirectory `input_files` within the `array_example` folder to store a set of generated input text files. +**(ii) Input File Generation**: For the slurm array examples [word-count-line-by-line](#example-42-line-by-line-word-count) and [dynamic-word-count-multiple-files](#example-43-dynamically-reading-and-counting-words-in-multiple-files) and [word-count-from-file-list](#example-44-counting-words-in-multiple-files-from-a-file-list), let us create subdirectory `input_files` within the `array_example` folder to store a set of generated input text files. ```bash $cd array_example @@ -309,8 +309,10 @@ The following Slurm script is an example of how you might convert the previous [ #SBATCH --mem=4G ### Memory required, 4 gigabyte #SBATCH --partition=express ### Cheaha Partition #SBATCH --time=01:00:00 ### Estimated Time of Completion, 1 hour -#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index -#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index +### Slurm Output file, %x is job name, %A is array job id, %a is array job index +#SBATCH --output=logs/%x_%A_%a.out +### Slurm Error file, %x is job name, %A is array job id, %a is array job index +#SBATCH --error=logs/%x_%A_%a.err #SBATCH --array=1-3 ### Number of Slurm array tasks, 3 tasks ### Loading Anaconda3 module to activate `pytools-env` conda environment @@ -366,7 +368,7 @@ In the following three examples, we will explore additional Slurm array job scen #### Example 4.2: Line-by-Line Word Count -In the examples [4.2](#example-42-line-by-line-word-count), [4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files), and [4.4](#example-44-counting-words-in-multiple-files-from-a-file-list), let us explore how to use a Slurm Array Job to process text files in parallel. +In the examples [word-count-line-by-line](#example-42-line-by-line-word-count), [dynamic-word-count-multiple-files](#example-43-dynamically-reading-and-counting-words-in-multiple-files), and [word-count-from-file-list](#example-44-counting-words-in-multiple-files-from-a-file-list), let us explore how to use a Slurm Array Job to process text files in parallel. The following Slurm job script processes a text file line by line using an array job. Save the following Slurm script as `line_word_count.job` within `array_example` folder. Prior to running this job make sure to complete the [input file generation](#example-4-array-job) step. This SLURM job ensures that each task in the job array processes a single line from an input file and counts the number of words in that line. @@ -377,8 +379,10 @@ The following Slurm job script processes a text file line by line using an array #SBATCH --mem=2G ### Memory required, 2 gigabyte #SBATCH --partition=express ### Cheaha Partition #SBATCH --time=00:10:00 ### Estimated Time of Completion, 10 minutes -#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index -#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index +### Slurm Output file, %x is job name, %A is array job id, %a is array job index +#SBATCH --output=logs/%x_%A_%a.out +### Slurm Error file, %x is job name, %A is array job id, %a is array job index +#SBATCH --error=logs/%x_%A_%a.err ### Define the input file, for instance, random_file_2.txt INPUT_FILE="$HOME/array_example/input_files/random_file_2.txt" @@ -388,8 +392,8 @@ INPUT_FILE="$HOME/array_example/input_files/random_file_2.txt" LINE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" "$INPUT_FILE") ### The command `wc -w` counts the number of words in the extracted line ($LINE), -### while echo "$LINE" prints the line of text. The pipe (|) then passes this text from echo "$LINE" to -### wc -w, which counts the words in the line for each task. +### while echo "$LINE" prints the line of text. The pipe (|) then passes this text from echo "$LINE" +### to wc -w, which counts the words in the line for each task. WORD_COUNT=$(echo "$LINE" | wc -w) echo "Task $SLURM_ARRAY_TASK_ID: $WORD_COUNT words" ``` @@ -424,7 +428,7 @@ Task 3: 6 words #### Example 4.3: Dynamically Reading and Counting Words in Multiple Files -This example job script performs the same function as [Example 4.2](#example-42-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job` within the `array_example` folder. It utilizes the same input files generated from the [Input File Generation](#example-4-array-job) section. +This example job script performs the same function as example [word-count-line-by-line](#example-42-line-by-line-word-count), but instead of counting the number of words in a single file line by line, it is designed to dynamically count the number of words across multiple files. Save the below script as `dynamic_file_word_count.job` within the `array_example` folder. It utilizes the same input files generated from the [Input File Generation](#example-4-array-job) section. ```bash linenums="1" #!/bin/bash @@ -433,8 +437,10 @@ This example job script performs the same function as [Example 4.2](#example-42- #SBATCH --mem=4G ### Memory required, 4 gigabyte #SBATCH --partition=express ### Cheaha Partition #SBATCH --time=00:15:00 ### Estimated Time of Completion, 15 minutes -#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index -#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index +### Slurm Output file, %x is job name, %A is array job id, %a is array job index +#SBATCH --output=logs/%x_%A_%a.out +### Slurm Error file, %x is job name, %A is array job id, %a is array job index +#SBATCH --error=logs/%x_%A_%a.err ### Define working directory WORKDIR="$HOME/array_example/input_files" @@ -449,7 +455,7 @@ FILES=($(find "$WORKDIR" -type f -name "random_file_*.txt" | sort)) ### To align the task ID with bash’s 0-based array indexing, we subtract 1 from the task ID. FILE="${FILES[$SLURM_ARRAY_TASK_ID-1]}" -### Extract the directory part of the file path. This will give the full directory path of the selected file. +### Get the full path to the directory containing the file DIRNAME=$(dirname "$FILE") ### Extract the base name of the file (without the extension) @@ -493,8 +499,8 @@ dynamic_file_word_count_31934540_1.out dynamic_file_word_count_31934540_3.out dynamic_file_word_count_31934540_2.out dynamic_file_word_count_31934540_4.out dynamic_file_word_count_31934540_5.out -### This command finds all files with names matching "random_file*.wordcount" in the specified directory and subdirectories. -### For each file found, concatenate and display its contents (using 'cat') to show the word count and their file path. +### Finds all files matching "random_file*.wordcount" in the specified directory and subdirectories. +### For each file, display contents with word count and file path. $ find $HOME/array_example/input_files/ -type f -name "random_file*.wordcount" -exec cat {} \; 81 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_1.txt 117 /home/$USER/Tutorial/slurm_tutorial/example4/input_files/random_file_5.txt @@ -505,7 +511,7 @@ $ find $HOME/array_example/input_files/ -type f -name "random_file*.wordcount" - #### Example 4.4: Counting Words in Multiple Files from a File List -This example job script is similar to [Example 4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files), but instead of dynamically reading files from the directory, it counts the number of words in multiple files in parallel using a SLURM job array, with the files listed in a separate file list. This example utilizes the same input files from [Example 4](#example-4-array-job) as described in the `Input File Generation` section. +This example job script is similar to example [dynamic-word-count-multiple-files](#example-43-dynamically-reading-and-counting-words-in-multiple-files), but instead of dynamically reading files from the directory, it counts the number of words in multiple files in parallel using a SLURM job array, with the files listed in a separate file list. This example utilizes the same input files from [Example 4](#example-4-array-job) as described in the `Input File Generation` section. To create a file list by tracking all the files in the `input_files` directory, use the find command along with globbing to list all generated files in the current directory and its subdirectories. The input files along with its absolute path are tracked in `file_list.txt`. @@ -515,7 +521,10 @@ cd array_example ### Search for all files in the "input_files" directory matching the pattern "random_file_*.txt". ### For each matching file, the 'realpath' command is used to get the absolute path of the file. ### The output is then sorted and written to "file_list.txt". -find $HOME/array_example/input_files -type f -name "random_file_*.txt" -exec realpath {} \; | sort > file_list.txt +find $HOME/array_example/input_files -type f \ + -name "random_file_*.txt" \ + -exec realpath {} \; | \ + sort > file_list.txt ``` Copy the following SLURM array job script to a file naming `file_list_word_count.job` within the `array_example` folder. This Slurm array job script count the number of words across multiple files listed in a file list `file_list.txt`. @@ -527,8 +536,10 @@ Copy the following SLURM array job script to a file naming `file_list_word_count #SBATCH --mem=4G ### Memory required, 4 gigabyte #SBATCH --partition=express ### Cheaha Partition #SBATCH --time=00:15:00 ### Estimated Time of Completion, 15 minutes -#SBATCH --output=logs/%x_%A_%a.out ### Slurm Output file, %x is job name, %A is array job id, %a is array job index -#SBATCH --error=logs/%x_%A_%a.err ### Slurm Error file, %x is job name, %A is array job id, %a is array job index +### Slurm Output file, %x is job name, %A is array job id, %a is array job index +#SBATCH --output=logs/%x_%A_%a.out +### Slurm Error file, %x is job name, %A is array job id, %a is array job index +#SBATCH --error=logs/%x_%A_%a.err ### Define working directory WORKDIR="$HOME/array_example/input_files" @@ -543,7 +554,7 @@ BASENAME=$(basename "$FILE" .txt) ### Check if file exists and count words from file if [[ -f "$FILE" ]]; then - ### 'wc -w' counts the words in the file, and the result is saved in a new file with '.wordcount' added to the name. + ### `wc -w` counts the words, saving the result in a new file with `.wordcount` appended. wc -w "$FILE" > "$WORKDIR/${BASENAME}.wordcount" echo "Processed $FILE" else @@ -568,7 +579,7 @@ Next, submit the array job using the command below, which creates an array of ta $sbatch --array=1-"$MAX_TASKS" file_list_word_count.job ``` -The output generated will be similar to [Example 4.3](#example-43-dynamically-reading-and-counting-words-in-multiple-files), as the functionality is the same, but the method of handling the input data differs. +The output generated will be similar to example [dynamic-word-count-multiple-files](#example-43-dynamically-reading-and-counting-words-in-multiple-files), as the functionality is the same, but the method of handling the input data differs. ### Example 5: Multithreaded or Multicore Job From a6517e8acf29e0e07b06b3bda0832765e034004a Mon Sep 17 00:00:00 2001 From: Premas Date: Fri, 16 May 2025 12:07:51 -0500 Subject: [PATCH 23/24] minor changes --- docs/cheaha/slurm/slurm_tutorial.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index c496c5132..2c60182fe 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -336,7 +336,7 @@ $cd array_example $sbatch slurm_array.job ``` -The output shows the sum of different input range computed by individual task, making it easy to track using a task identifier, such as array task 1/2/3. +The output shows the sum of different input range computed by individual task, making it easy to track using a task identifier, such as array task 1/2/3. While each task is clearly labeled (e.g., "array task 1"), the order in which these outputs appear in the file may not be in order. This is because Slurm array tasks run in parallel and write to the output file concurrently, so the write order is not guaranteed. ```bash $ cat $HOME/array_example/output_all_tasks.txt @@ -346,7 +346,7 @@ array task 3 Input Range: 200001 to 300000, Sum: 24999750000 array task 1 Input Range: 1 to 100000, Sum: 4999950000 ``` -The `sacct` report indicates that the job `27101430` consists of three individual tasks, namely `27101430_1`, `27101430_2`, and `27101430_3`. Each task has been allocated one CPU resource. +The `sacct` report indicates that the job `27101430` consists of three individual tasks, namely `27101430_1`, `27101430_2`, and `27101430_3`. Each task was allocated one CPU resource. ```bash $ sacct -j 27101430 From 048f658c0da0f6f60e5cd5d5431ee51e41b70b47 Mon Sep 17 00:00:00 2001 From: Premas Date: Tue, 20 May 2025 11:42:53 -0500 Subject: [PATCH 24/24] array indexing --- docs/cheaha/slurm/slurm_tutorial.md | 56 ++++++++++++++--------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 2c60182fe..e63c87926 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -292,7 +292,7 @@ Now that the general environment and preprocessing setup are complete, the follo #### Example 4.1: Running Parallel Python Tasks with Dynamic Input Ranges -The following Slurm script is an example of how you might convert the previous [parallel job example](#example-3-parallel-jobs) script to an array job. To start, copy and save the below script to a file named, `slurm_array.job` within the `array_example` folder. The script requires the input file `python_script_new.py` and the `conda` environment `pytools-env`, similar to those used in [example2](../slurm/slurm_tutorial.md#example-2-sequential-job) and [example 3](../slurm/slurm_tutorial.md#example-3-parallel-jobs). Line 11 specifies the script as an array job, treating each task within the array as an independent job. For each task, lines 18-19 calculates the input range. `SLURM_ARRAY_TASK_ID` identifies the task executed using indexes, and is automatically set for array jobs. The python script (line 22) runs individual array task concurrently on respective input range. The command `awk` is used to prepend each output line with the unique task identifier and then append the results to the file, `output_all_tasks.txt`. For more details on on parameters of array jobs, please refer to [Batch Array Jobs](../slurm/submitting_jobs.md#batch-array-jobs-with-known-indices) and [Practical Batch Array Jobs](../slurm/practical_sbatch.md#). +The following Slurm script is an example of how you might convert the previous [parallel job example](#example-3-parallel-jobs) script to an array job. To start, copy and save the below script to a file named, `slurm_array.job` within the `array_example` folder. The script requires the input file `python_script_new.py` and the `conda` environment `pytools-env`, similar to those used in [example2](../slurm/slurm_tutorial.md#example-2-sequential-job) and [example 3](../slurm/slurm_tutorial.md#example-3-parallel-jobs). Line 11 specifies the script as an array job, treating each task within the array as an independent job. For each task, lines 18-19 calculates the input range. `SLURM_ARRAY_TASK_ID` identifies the task executed using indexes, and is automatically set for array jobs. The python script (line 24) runs individual array task concurrently on respective input range. The command `awk` is used to prepend each output line with the unique task identifier and then append the results to the file, `output_all_tasks.txt`. For more details on on parameters of array jobs, please refer to [Batch Array Jobs](../slurm/submitting_jobs.md#batch-array-jobs-with-known-indices) and [Practical Batch Array Jobs](../slurm/practical_sbatch.md#). !!! important @@ -313,15 +313,15 @@ The following Slurm script is an example of how you might convert the previous [ #SBATCH --output=logs/%x_%A_%a.out ### Slurm Error file, %x is job name, %A is array job id, %a is array job index #SBATCH --error=logs/%x_%A_%a.err -#SBATCH --array=1-3 ### Number of Slurm array tasks, 3 tasks +#SBATCH --array=0-2 ### Number of Slurm array tasks, 3 tasks ### Loading Anaconda3 module to activate `pytools-env` conda environment module load Anaconda3 conda activate pytools-env ### Calculate the input range for each task -start=$((($SLURM_ARRAY_TASK_ID - 1) * 100000 + 1)) -end=$(($SLURM_ARRAY_TASK_ID * 100000)) +start=$((($SLURM_ARRAY_TASK_ID) * 100000 + 1)) +end=$((($SLURM_ARRAY_TASK_ID + 1) * 100000)) ### Run the python script with input arguments and append the results to a .txt file for each task python python_script_new.py $start $end 2>&1 \ @@ -341,27 +341,27 @@ The output shows the sum of different input range computed by individual task, m ```bash $ cat $HOME/array_example/output_all_tasks.txt -array task 2 Input Range: 100001 to 200000, Sum: 14999850000 -array task 3 Input Range: 200001 to 300000, Sum: 24999750000 -array task 1 Input Range: 1 to 100000, Sum: 4999950000 +array task 2 Input Range: 200001 to 300000, Sum: 24999750000 +array task 1 Input Range: 100001 to 200000, Sum: 14999850000 +array task 0 Input Range: 1 to 100000, Sum: 4999950000 ``` -The `sacct` report indicates that the job `27101430` consists of three individual tasks, namely `27101430_1`, `27101430_2`, and `27101430_3`. Each task was allocated one CPU resource. +The `sacct` report indicates that the job `33509888` consists of three individual tasks, namely `33509888_0`, `33509888_1`, and `33509888_2`. Each task was allocated one CPU resource. ```bash -$ sacct -j 27101430 +$ sacct -j 33509888 JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- -27101430_3 slurm_arr+ express USER 1 COMPLETED 0:0 -27101430_3.+ batch USER 1 COMPLETED 0:0 -27101430_3.+ extern USER 1 COMPLETED 0:0 -27101430_1 slurm_arr+ express USER 1 COMPLETED 0:0 -27101430_1.+ batch USER 1 COMPLETED 0:0 -27101430_1.+ extern USER 1 COMPLETED 0:0 -27101430_2 slurm_arr+ express USER 1 COMPLETED 0:0 -27101430_2.+ batch USER 1 COMPLETED 0:0 -27101430_2.+ extern USER 1 COMPLETED 0:0 +33509888_2 slurm_arr+ express prema 1 COMPLETED 0:0 +33509888_2.+ batch prema 1 COMPLETED 0:0 +33509888_2.+ extern prema 1 COMPLETED 0:0 +33509888_0 slurm_arr+ express prema 1 COMPLETED 0:0 +33509888_0.+ batch prema 1 COMPLETED 0:0 +33509888_0.+ extern prema 1 COMPLETED 0:0 +33509888_1 slurm_arr+ express prema 1 COMPLETED 0:0 +33509888_1.+ batch prema 1 COMPLETED 0:0 +33509888_1.+ extern prema 1 COMPLETED 0:0 ``` In the following three examples, we will explore additional Slurm array job scenarios that are commonly used in scientific simulations. @@ -388,8 +388,8 @@ The following Slurm job script processes a text file line by line using an array INPUT_FILE="$HOME/array_example/input_files/random_file_2.txt" ### `sed` is a shell command and stream editor for manipulating text. -### Here, `sed` reads $INPUT_FILE and extracts the line corresponding to SLURM_ARRAY_TASK_ID. -LINE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" "$INPUT_FILE") +### It extracts the line from $INPUT_FILE matching the task ID (adjusted for zero-based indexing) +LINE=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" "$INPUT_FILE") ### The command `wc -w` counts the number of words in the extracted line ($LINE), ### while echo "$LINE" prints the line of text. The pipe (|) then passes this text from echo "$LINE" @@ -404,26 +404,26 @@ Before running the above SLURM job, determine the number of lines in the input f $MAX_TASKS=$(wc -l < $HOME/array_example/input_files/random_file_2.txt) ``` -Next, submit the array job using the command below, which creates an array of tasks from 1 to the maximum number of lines in the file. +Next, submit the array job using the command below. It creates an array of tasks starting from 0 up to (MAX_TASKS – 1), matching the zero-based indexing used in the script. ```bash -$sbatch --array=1-"$MAX_TASKS" line_word_count.job +$sbatch --array=0-$(($MAX_TASKS - 1)) line_word_count.job ``` -The below output comes from the SLURM job array script (line_word_count.job), where each task processes one line from random_file_2.txt. Since the file has 3 lines, SLURM created 3 tasks (Task 1, Task 2, Task 3), each counting words in its respective line. +The below output comes from the SLURM job array script (line_word_count.job), where each task processes one line from random_file_2.txt. Since the file has 3 lines, SLURM created 3 tasks (Task 0, Task 1, Task 2), each counting words in its respective line. ```bash ### counts the number of lines in the file, for instance, random_file_2.txt -$ wc -l /home/$USER/array_example/input_files/random_file_2.txt +$ wc -l $HOME/array_example/input_files/random_file_2.txt 3 $HOME/array_example/input_files/random_file_2.txt ``` ```bash ### Print and verify the output using the `cat` command -$ cat /home/$USER/array_example/logs/line_word_count_$JOB_ID_*.out -Task 1: 8 words -Task 2: 19 words -Task 3: 6 words +$ cat $HOME/array_example/logs/line_word_count_$JOB_ID_*.out +Task 0: 8 words +Task 1: 19 words +Task 2: 6 words ``` #### Example 4.3: Dynamically Reading and Counting Words in Multiple Files