Differences

This shows you the differences between two versions of the page.

--- wiki:user_guide [2022/05/06 09:35]
phegde
+++ wiki:user_guide [2022/05/28 18:18] (current)
cnr-guest [Job preparation ans submission]
@@ Line 101: / Line 101: @@
 Complete documentation is avalailable at ''https://slurm.schedmd.com/''.
-SLUR is an open source software sytstem for cluster management; it is highly scalable and  integrates fault-tolerance and job scheduling mechanisms.
+SLURM is an open source software sytstem for cluster management; it is highly scalable and  integrates fault-tolerance and job scheduling mechanisms.
 ==== SLURM basic concepts ====
@@ Line 322: / Line 322: @@
 === Parallel computation in python ===
-The effective usage of the hpc can be done using parallelizing the processes. The codes can be parallelized by distributing the tasks among the available nodes and their respective CPUs and GPUs as well.  This information can be specified in a simple submission bash script as follows,
+   * The effective usage of the hpc can be done using parallelizing the processes. The codes can be parallelized by distributing the tasks among the available nodes and their respective CPUs and GPUs as well. This information can be specified in a simple submission bash script or a sub.sh file as follows,
 <code>#!/bin/bash
-#SBATCH --nodes=
+#SBATCH --nodes=[nnodes]           #number of nodes
-#SBATCH --ntasks-per-node=
+#SBATCH --ntasks-per-node=[ntasks per node] #number of cores per node
-#SBATCH --gres=gpu:
+#SBATCH --gres=gpu:[ngpu]        #number of GPUs per node</code>
-python example.py </code>
-Python offers many packages to parallelize the given process. The basic one among them [[https://docs.python.org/3/library/multiprocessing.html|multiprocessing]].
+=== Example of parallel jobs submission ===
+Suppose a given python code has to be executed for different values of a variable "rep". It is possible to execute the python codes parallelly during the job submission process by creating temporary files each file with rep=a1, a2,... The python code example.py can have a line:
+<code> rep=REP </code>
+The submission script sub.sh can be used to parallelize the process in following way:
+<code>#!/bin/bash
+#SBATCH --nodes=[nnodes]            #number of nodes
+#SBATCH --ntasks-per-node=[ntasks per node]  #number of cores per node
+#SBATCH --gres=gpu:[ngpu]         #number of GPUs per node
+NPROC=[nprocesses]                     #number of processing units to be accessed
+tmpstring=tmp               #temporary files generated
+count=0                     #begin counting the temporary files
+for rep in {1..10};         #The value of rep should run from 1 to 10
+do
+    tmpprogram=${tmpstring}_${rep}.py         #temporary file names for each of the values of rep
+    sed -e "s/REP/$rep/g"   #replace the variable REP in the .py with rep specified in the sub.sh file.
+    $program > $tmpprogram  #create the temporary files in parallel
+    python $tmpprogram &    #run the temporary files
+    (( count++ ))           #increase the count number
+    [[ $(( count % NPROC )) -eq 0 ]] && wait  #wait for the parallel programs to finish.
+done
+rm ${tmpstring}*            #optionally remove the temporary files after the execution of all the temporary files</code>
+   * Parallel job submissions can also be done by job array submission. More information about Job arrays can be found in [[https://slurm.schedmd.com/job_array.html]].
+   * Parallelization can be implemented within the python code itself. For example, the evaluation of a function for different variable values can be done in parallel. Python offers many packages to parallelize the given process. The basic one among them is [[https://docs.python.org/3/library/multiprocessing.html|multiprocessing]].
+   * The keras and Pytorch modules in tensorflow which are mainly used for machine learning detects the GPUs automatically.

ibisco HPC Wiki

User Tools

Site Tools

Differences

Page Tools