Working renderfarm. - gui configuring. UPDATE

The interface, modeling, 3d editing tools, import/export, feature requests, etc

Moderators: jesterKing, stiv

Post Reply
ideasman
Posts: 0
Joined: Tue Feb 25, 2003 2:37 pm

Working renderfarm. - gui configuring. UPDATE

Post by ideasman »

Hi, this is a renderfarm script that works as long as all the pcs can accsess the same directory (SAMBA / NFS ETC.)
Uses menus... firs youyll need to set your render dir the you can send a file and start a client.
Tested on linux and win32.

Ctrl+Alt+Shift to exit the Client.

http://members.iinet.net.au/~cpbarton/blenderfarm.py
Last edited by ideasman on Fri Aug 13, 2004 3:19 pm, edited 3 times in total.

steve_t
Posts: 30
Joined: Tue Nov 05, 2002 7:10 am

Post by steve_t »

This is an AWESOME SCRIPT
though it didn't work when I copy and pasted

Code: Select all

#=======================================================# 
# Ask the user weather they want to start a client or   # 
# send the existing scene to the renderfarm             # 
#=======================================================# 

method = Draw.PupMenu(\ 
'BRenderfarm...%t|\ 
RenderFarm current .blend|\ 
Be a RenderFarm slave|\ 
Configure this Node|') 
I had to take out the backslashes for your popup menu and put it on one line.
I am not sure but I think your header doesn't work with the menu in 2.34

Great Work!
I can see this script being a good solution to integrated network rendering. This is the first blender network renderer that I have seen that can split up one frame and put it back together.

I would like to have an option to have one of the slaves wait for the frames and put them back together so I can keep working on my file if I want to.

Another addition would be to support rendering with yafray. I am not sure why but the client just renders the full frame for each piece when I use yafray.

It would be cool to support multiple image formats and animations.

Awesome work man!
-steve
[/code]

ideasman
Posts: 0
Joined: Tue Feb 25, 2003 2:37 pm

Post by ideasman »

Hi, thanks for your interest.
Short term TODO
- make the composite function look for imlib / PIL before doing the slow and inefficient blender/render based composite.
(Possible solution is to use the sequencer/python but would need to be coded- long term goal)

- Draw a GUI that displays client/server status.

- Automaticly generate Web based update (HTML file that refreshes all rendered tiles as it renders) - I have tested this and it looks way funky.

- Animation (dosent do any tricky image splitting stuff so its no fun but would be easy to do) but also isnt fun to code :)

- Yafray needs to support border rendering, when it does this will work with no problems. (sombode else will need to do this.)

Long term Goal.
- INternet based rendering

- Tricky binary diffing for updates to blend file accross the network to reduce the time to update a fiel accross a network.

- Some funky file sharing teachnology BitTorrent? blenderchat IRC file sharing? WEBDAV/FTP if I cant get anything tricky working.

For us linux users Id LOVE to just use lufs (makes it possible to mount FTP as a filesystem) and to a web Render test!!!!!!!
I have some ftp space that I dont use so I dont mind handing out the password to it.
This would work as is, with no changes to the script!


The resion you havnt seen ant blender renders that split the image is that I added the python hooks needed to automaticly set border rendering.

- Id realy like this to be more of a cummunity project since many people can benifit from this.

The code is commented fairly well so Id appretiate some people having a look and mabe add some of the features I have mentioned, bug reports are also usefull (there are some bugs but I havent pinned them down.)
steve_t wrote:This is an AWESOME SCRIPT
though it didn't work when I copy and pasted

Code: Select all

#=======================================================# 
# Ask the user weather they want to start a client or   # 
# send the existing scene to the renderfarm             # 
#=======================================================# 

method = Draw.PupMenu(\ 
'BRenderfarm...%t|\ 
RenderFarm current .blend|\ 
Be a RenderFarm slave|\ 
Configure this Node|') 
I had to take out the backslashes for your popup menu and put it on one line.
I am not sure but I think your header doesn't work with the menu in 2.34

Great Work!
I can see this script being a good solution to integrated network rendering. This is the first blender network renderer that I have seen that can split up one frame and put it back together.

I would like to have an option to have one of the slaves wait for the frames and put them back together so I can keep working on my file if I want to.

Another addition would be to support rendering with yafray. I am not sure why but the client just renders the full frame for each piece when I use yafray.

It would be cool to support multiple image formats and animations.

Awesome work man!
-steve
[/code]

calli
Posts: 38
Joined: Wed Oct 16, 2002 10:59 am
Contact:

Post by calli »

Hi,

I just did some quick test without much thinking so maybe I overlooked something....

- If I have a *.blend1 file in the folder, the slaves try to render theese...

- slaves does not reload the file, so changes need to walk to all slaves reloading the file by hand?!

- the master could do some rendering too?! No need then to start a slave on the same machine

- The sky of a world is not rendered

Other than that I can say it renders ;-)

Thanks,
Carsten.

ideasman
Posts: 0
Joined: Tue Feb 25, 2003 2:37 pm

Post by ideasman »

Hi, fixed many bugs and improved some things

Changed Name to BlenderFarm

- Old image Problem (due to images alredy being loaded and not ReLoaded from the file) (Yay I got to use my img.reload() function.)

- Configuration system improved and added TileTimeOut and TileNumber
... the number of tiles is now set per image rather then by pixel size.
... so you can say 'Split all my images into 9 tiles' - Would be good for 3-9 computers. - Since the number of PCs you have is useualy static unlike the dimensions of your image.
(Time out not working yet- will be implimented with render overt IP)
- Dosent render .blend1 files anymore.

Various other improvements- should be more stable in general.


- Other Image formats?
... not yet... it depends a lot on the selected compositing system.
- PNG is persect for this application, and a darn good image format.
... This is more on an issue for the composoting system sine the tile have to be rendered in a format that supports alpha (under the current system) so Im not going to worry about it now.

http://members.iinet.net.au/~cpbarton/blenderfarm.py
[/b]

ideasman
Posts: 0
Joined: Tue Feb 25, 2003 2:37 pm

Post by ideasman »

Now you can press CTRL+ALT+SHIFT to exit slave Mode.

- Can now send and be a client very easely.

method = Draw.PupMenu(\
'BlenderFarm...%t|\
Send Scene|\
Send & Render|\
View Rendered Tiles|\
Render Watch (Slave)|\
Configure this Node|')

calli
Posts: 38
Joined: Wed Oct 16, 2002 10:59 am
Contact:

Post by calli »

Just to let you know, tested on XP, W2K and Linux at the same time, worked well!

However still no Sky rendered (I guess its a Blender Border Render thingy?!)

And on XP Blender often crashes while rendering, also on W2K taking the whole machine down (only on W2K). COuld not prove if it is your script or Blender itself. The crashes also makes testing a bit hard, running between the machines and rebooting etc.

That also shows that a network renderer with a gui is maybe a bad idea, I guess I would opt for a commandline version (maybe possible with OnLoad Scripts?).

Thanks,
Carsten.

ideasman
Posts: 0
Joined: Tue Feb 25, 2003 2:37 pm

Post by ideasman »

calli wrote:Just to let you know, tested on XP, W2K and Linux at the same time, worked well!

However still no Sky rendered (I guess its a Blender Border Render thingy?!)

And on XP Blender often crashes while rendering, also on W2K taking the whole machine down (only on W2K). COuld not prove if it is your script or Blender itself. The crashes also makes testing a bit hard, running between the machines and rebooting etc.

That also shows that a network renderer with a gui is maybe a bad idea, I guess I would opt for a commandline version (maybe possible with OnLoad Scripts?).

Thanks,
Carsten.
Hi, thanks for testing- This script uses some new blender functions as well loading a blend file while running a python script (which makes it necessary to re import all modules including ones in renderfarm.py)

It also does a few odd things- (Combinations of functions that wouldent be used an many situations.)

The renderer isnt realy a GUI renderer- Its just a renderFarm script with graphical menu and input configuration. The client its self (slave desent accsess GUI functions.)

Rendering from the command line is Tricky and clunky since the blend file needs to be reloaded each time, so I think the solution is to find out where the errors are and patch them or work around them, also this could be your system- Its just possible since I havnt had trouble with my XP box.

0 Sky is rendered but set to 1 alpha (0 is no alpha 255 is opaque) even tho its not rendered, its not dispayed.- Wiil need a solution, but try different alpha methods... only dested the alpha value.

Thanks for testing the script. if you want I can make a debug version that writes the line numbers to a file as it runs so we can find the point of error.
- Cam

ideasman
Posts: 0
Joined: Tue Feb 25, 2003 2:37 pm

Post by ideasman »

Some usage Tips-
For faster response per tile do the following...

- Convert subserf mesh's to normal meshs.
- Use raytraced shadows, not shadowbuffer.
- Use the smallest ammount of tiles possible so each PC in rthe farm has 1 tile.

monocult
Posts: 17
Joined: Wed Oct 16, 2002 2:28 pm
Location: Siegburg (germany)
Contact:

Post by monocult »

Hi

If I try

Send & Render

Blender crash with

Code: Select all

monocult@gina monocult $ ./Programme/blender-2.34-linux-glibc2.2.5-i386/blender
BlenderFarm Client  0.4 started.
Press Ctrl+Alt+Shift to exit
Speicherzugriffsfehler
if I try only send it works but View Rendered Tiles crashs Blender too.

Code: Select all

monocult@gina monocult $ ./Programme/blender-2.34-linux-glibc2.2.5-i386/blender
Traceback (most recent call last):
  File "<string>", line 611, in ?
  File "<string>", line 443, in viewRender
KeyboardInterrupt
Speicherzugriffsfehler
[/code]

levon
Posts: 0
Joined: Thu Jul 31, 2003 6:06 am
Location: adelaide

Post by levon »

hey ideasman awsome script, just tried it out for an actual render across 3 computers on a network. works realy well.

all windows XP, i got access to some mac and linux box, but they are both quite slow compared to the 3 XP machines.

thanks for this awsome script ;)

levon
Posts: 0
Joined: Thu Jul 31, 2003 6:06 am
Location: adelaide

Post by levon »

who knows if anyone is still looking at this thread but anywho
http://projects.blender.org/pipermail/b ... 02931.html
this has just been comited, so that border render will work in yafray. so using this script in yafray MAY work. unfortunatly i cant get the yafray CVS to compile so i cant test it.

Poju
Posts: 0
Joined: Mon Oct 21, 2002 11:16 pm

Post by Poju »

Hi

This sounds exactly what i'm been looking for.
Only bad is that script won't run.

Here is Blender console output:

SyntaxError: invalid syntax
Traceback (most recent call last):
File "<string>", line 17, in ?
File "D:\PYTHON24\LIB\os.py", line 133
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep, altsep,



Do i have too new python or what?
I'm pretty lost with this python/script stuff

levon
Posts: 0
Joined: Thu Jul 31, 2003 6:06 am
Location: adelaide

Post by levon »

yeah, you need python 2.3

ideasman
Posts: 0
Joined: Tue Feb 25, 2003 2:37 pm

Post by ideasman »

Been itching to try this on my new amd64- now I have 3pc's all around 3ghz- Could make a nifty lil render farm, sadly I dont need to render anything yet...
- Tile based realy needs to be implimented in the Renderloop- so the scene isnt always being rebuilt.

Post Reply