Blender 2.49 on Rocks 5.3 HPC (Centos 5.4) torque/PBS queue

[aa3025]

Here is the PBS/qsub script for blender rendering job submission into HPC queue running torque PBS queue manager with Rocks Cluster suite based on CentOS 5. Blender 2.49 is installed.
project folder (with blend file ) must be shared across all the nodes.
Paralelisation is frame-based, so each node rendering its own number of frames. All nodes save output into the same folder as sequence of JPEG files.

Some improvement is still needed (listed on the final lines of the code)

Code: Select all

#!/bin/bash
#PBS -S /bin/bash
#PBS -V
#
#PBS -l nodes=20:ppn=12
filename=testfile.blend

rsvr="qsub -I -l nodes="

cd ${PBS_O_WORKDIR}

#how many nodes we have to render on:
cat ${PBS_NODEFILE} | sort -u > ./hostsfile.txt
nr_of_nodes=`wc -l < hostsfile.txt`

#how manu frames we want to render
let total_frames=1000

#how many frames per node we have to render
let fpn=$total_frames/$nr_of_nodes

#global starting frame to render
let start_frame=1253

#start frame for the 1st node
let startf=$start_frame

#frame number to stop at
let final_frame=$start_frame+$total_frames

#create killing file
echo "" > kill_blenders.sh

#now we will launch blender render on each node via ssh:
for NODE in `cat hostsfile.txt`;
do
pl="+"
let endf=$startf+$fpn-1

if [ ${endf} -gt ${final_frame} ]; then
let endf=$final_frame
pl=""
fi

#for reserving node:
rsvr=${rsvr}${NODE}":ppn=12"${pl}


EXECUTABLE="ssh -f "$NODE" blender -b "${PBS_O_WORKDIR}"/"${filename}" -o //output/frame -F JPEG -x 1 -s "$startf" -e "$endf" -a "
echo ${EXECUTABLE} > $NODE.log

#here we actually launch rendering on the $NODE:
${EXECUTABLE} >> $NODE.log

#add command in killing script for this node
echo "ssh -f $NODE killall blender" >> kill_blenders.sh

#start frame for the next node:
let startf=$startf+$fpn

done 

#this script (kill_blenders.sh) will be generated in the project folder
#for interrupting all blenders on all nodes
chmod +x kill_blenders.sh


#the trouble is that after launching blender renderer through ssh, 
#torque thinks the job is done and removes it from queue
#but actually blenders are still running on the nodes
#therefore just after job is launched and removed form queue
# we have to reserve the same nodes on the cluster with
#generated here reserve.sh script

#have to guess walltime to reserve in the queue, say 1 hour
rsvr=${rsvr}",walltime=1:00:00" 
echo $rsvr > reserve.sh

chmod +x reserve.sh

#and launch it manually after present script (for submission) has finished

[Art3DWorks]

i have a rocks cluster and want to use it for a render farm with blender.

What is the procedure to do this?

[aa3025]

Here is updated version of the script, which may be run without particular scheduler on HPC (torque or whatever). In this case blender is installed into /share/apps/blender which is accessible from every node (NFS share). The script will launch MPI workers on each node rendering their own selection of frames.

Requirements:
1) blender installed in folder shared across all nodes (or in the same location on each node)
2) working folder (from which this script is launched) must be shared across all the nodes (e.g. NFS share).

You have to launch this script with the command (from e.g. master node of HPC ):

Code: Select all

mpirun -hostfile /full/path/to/hosts/file/hosts.txt  blenders.sh <BLENDERFILENAME> <START_FRAME> <TOTAL_FRAMES_TO_RENDER>

Example to render starting from frame 1 to 200:

Code: Select all

mpirun -hostfile ~/my/folder/hosts.txt ./blenders.sh ~/myblenderproject/scene.blend 1 200

Parameters of rendering (e.g. resolution of output or default camera) must be saved within .blend file itself. Otherwise you can change last line of the script to enter more command line parameters for blender executable.


text file hosts.txt must contain the list of all nodes for blender to start on (in column); if node has 8 CPU's, you have to repeat its name 8 times in order to start 8 MPI threads on the node.

Code: Select all

#!/bin/bash
# Usage ./blenders.sh <filename.blend> <startframe> <total_frames_to_render>

source ~/.bashrc

filename=$1
start_frame=$2
total_frames=$3

proc_num=${OMPI_COMM_WORLD_RANK}
total_proc_num=${OMPI_COMM_WORLD_SIZE}


echo $filename $start_frame $total_frames

##################USER PARAMETERS#####################
# output_folder
output=output_jpegs

#####################################################


#create output folder if does not exist

if [ ! -d "$output" ]; then
	mkdir $output
fi

workdir=`pwd`

let frames_per_proc=$total_frames/$total_proc_num
let startf=$start_frame+$proc_num*$frames_per_proc
let endf=$startf+$frames_per_proc-1

let proc_nr=$proc_num+1
if [ $proc_nr -eq $total_proc_num ]
then
    let endf=$start_frame+$total_frames-1
fi

/share/apps/blender/blender -b ${workdir}/${filename} -t 1 -o $workdir/$output/ -F JPEG -x 1 -s $startf -e $endf -a

[Art3DWorks]

Thanks for your response
now i have a problem, it's when i launch the script it can't find the files, because the data in the head node is in:
/export/home/FelixMatias

and in the nodes it is in:
/share/home/FelixMatias

how can i do to resolve this?