project folder (with blend file ) must be shared across all the nodes.
Paralelisation is frame-based, so each node rendering its own number of frames. All nodes save output into the same folder as sequence of JPEG files.
Some improvement is still needed (listed on the final lines of the code)
Code: Select all
#!/bin/bash
#PBS -S /bin/bash
#PBS -V
#
#PBS -l nodes=20:ppn=12
filename=testfile.blend
rsvr="qsub -I -l nodes="
cd ${PBS_O_WORKDIR}
#how many nodes we have to render on:
cat ${PBS_NODEFILE} | sort -u > ./hostsfile.txt
nr_of_nodes=`wc -l < hostsfile.txt`
#how manu frames we want to render
let total_frames=1000
#how many frames per node we have to render
let fpn=$total_frames/$nr_of_nodes
#global starting frame to render
let start_frame=1253
#start frame for the 1st node
let startf=$start_frame
#frame number to stop at
let final_frame=$start_frame+$total_frames
#create killing file
echo "" > kill_blenders.sh
#now we will launch blender render on each node via ssh:
for NODE in `cat hostsfile.txt`;
do
pl="+"
let endf=$startf+$fpn-1
if [ ${endf} -gt ${final_frame} ]; then
let endf=$final_frame
pl=""
fi
#for reserving node:
rsvr=${rsvr}${NODE}":ppn=12"${pl}
EXECUTABLE="ssh -f "$NODE" blender -b "${PBS_O_WORKDIR}"/"${filename}" -o //output/frame -F JPEG -x 1 -s "$startf" -e "$endf" -a "
echo ${EXECUTABLE} > $NODE.log
#here we actually launch rendering on the $NODE:
${EXECUTABLE} >> $NODE.log
#add command in killing script for this node
echo "ssh -f $NODE killall blender" >> kill_blenders.sh
#start frame for the next node:
let startf=$startf+$fpn
done
#this script (kill_blenders.sh) will be generated in the project folder
#for interrupting all blenders on all nodes
chmod +x kill_blenders.sh
#the trouble is that after launching blender renderer through ssh,
#torque thinks the job is done and removes it from queue
#but actually blenders are still running on the nodes
#therefore just after job is launched and removed form queue
# we have to reserve the same nodes on the cluster with
#generated here reserve.sh script
#have to guess walltime to reserve in the queue, say 1 hour
rsvr=${rsvr}",walltime=1:00:00"
echo $rsvr > reserve.sh
chmod +x reserve.sh
#and launch it manually after present script (for submission) has finished