wahn wrote:

Hi, Unfortunately the Python API is also not consistent between the game engine and the rest of Blender. Erwin and me wanted to work on that but there was just no time left before NaN went bancrupt the first time. And I had other problems to survive in London Jan

Thanks, Jan, for everything!

My priorities would be:

1. revise the python system to give scripts access to more parts of Blender's internals, including the GUI. Ton talked a bit about this at the conference, and the problems now revolve around Blender's "event system". He thought there might be a way to get python "hooks" into those areas, and this could allow all sorts of additions to the GUI. It might also help in "pythonizing" the plugin/material system.

2. smooth the gap between realtime and offline python. markluffel's comments are right on -- at least in terms of integrating realtime and offline. It seems that if you kept the python context (namespace) constant between running an offline script (gui in a text window) and the game engine, you could definitely handle his example of "baking" realtime behavior into ipo's (there's a script available somewhere that records game objects' properties to a file for offline translation into IPO's).

How about running your python gui script, being able to set variables in the GameLogic module. Then if the offline API had a "startGameEngine()" method, the gui script could start the game engine, maybe even advancing it frame by frame with "advanceGameFrame(). When the game was finished you could access whatever variables your game scripts put onto the GameLogic module, and the gui script could continue running. If the offline API also had access to the logic bricks, then much more would be possible. With the new ODE physics system, offline python programmers are really going to want this functionality!

These things, to me, would be the "low hanging fruit" that would pay off big time. Yet bigger things are good, too. Actually, Ton suggested that a version 2.5 could have an expanded "constraint" system, such that "parenting" is handled with a type of constraint possibly subject to IPO's like in v.2.25. The great thing about open source is that big and small projects can go on in parallel. Sign me up for 1. and 2.!

Cheers,
Tom

Hi,

OK, the reason why the game engine's Python scripts do NOT affect the "offline" (I don't like this term - maybe game vs. modeling is more appropriate) Python scripts is that both parts are starting their own interpreter. Erwin and me decided that we have to do this because we did NOT want that the game engine can "pollute" the namespace of the modeling interpreter. If your modeling script would define a variable "a" in global namespace and the game engine would use the same variable name in global name space your modeling script might behave differently AFTER running the game engine. We thought this would be VERY confusing. So we decided to have two SEPARATE interpreters for both parts.

The other reason why the game and the modeling API is NOT consistent is that the game engine was entirely written in C++ and the older modeling code (mainly written by Ton itself) is written in C. My idea was to make the API LOOK like an object-oriented language (which Python is) for both parts by using classes. This is straight forward for a C++ implementation but NOT for C. The problem is that I prefered to use the users perspective of Blender. Which menas that I tried to define an API which would be expected by a Blender user seeing the Object Window (e.g. Scene, Object, Mesh, Curve, Surface, etc. data) and the connections between the blocks (e.g. a material can be connected to a mesh OR an object). This is more how the data in Blender's modeling system is connected (stored in a kind of a database). In C++ it makes sense to have more or less the same Python classes like you have in C++. But this is a DIFFERENT view. So the question was:

Should we implement the whole Blender code in C++ (and start from scratch)?

The question was answered by the management and time constraints: We had NO time to start from scratch. Now Blender is open source. Which means that people COULD start from scratch taking the CONCEPTS of Blender and use plain C++ everywhere ...

I think Ton implemented a lot of very good ideas in plain C. If he could explain the main concepts people could help to start with a pure C++ implementation taking this concepts and define a good class structure. But this takes very experienced programmers and a lot of time discussing concepts NOT starting coding straight away. Now it's time to do this but I doubt it will happen

What do you think?

Cheers,

Jan

Yeah! C++!

ton wrote:

In the past, with each new developer coming in, we were confronted with a new vision on how Python should be implemented in Blender. This resulted in incompatibility among releases, and a non-uniform API (game engine vs. rest of Blender). Causing quite some frustrations with our users...

Hello there,

well, finally, I thought, I check out what the old blender fellows are
chatting about these days..

This discussion about Python has been very old, I guess. And many things have been repeatedly misunderstood at NaN, since there was little support around for the Python side. I must also say, that the concepts and strengths of Python have been very underestimated. At least and last, you're coming back to these points

The problem about implementing a good object oriented API is: the robust Kernel. Blender has no Kernel (at least you can't really call it one).
So, at least one point where all different python API people would agree:
There are things you just can't to to make it stable.
(The GameEngine is another topic - there was not even a concept of
an API. It got obviously just hacked in to 'make something work')


So: To design a proper API without implementing quirks and hacks on an existing System which was not designed for object handling with all consequences of proper reference counting, is a darn hard thing which involves a lot of planning, thinking, and discussions. And we all know that attempts to start these kind of discussions were cancelled due to funny ideas that people in the management suddenly brought up. Like: Can you make a VRML2 importer in 2 weeks ?

So the only approach I see to make a clean API:
1. Redesign a Blender object kernel
2. Create a good C API (C++ still is too problematic among different
platforms and compilers - *sigh*)
3. SWIG the darn thing, or use other approaches like BOOST (which involve a lot of C++ knowledge though. And they can make size explode...) More on BOOST at http://www.boost.org.

Too bad. I've got another job but it might be interesting to
follow some redesign ideas, Maya-alike.
Let's see what the current blender hackers work out.

Cheers,

- Strubi

strubi wrote:

ton wrote: In the past, with each new developer coming in, we were confronted with a new vision on how Python should be implemented in Blender. This resulted in incompatibility among releases, and a non-uniform API (game engine vs. rest of Blender). Causing quite some frustrations with our users... Hello there, well, finally, I thought, I check out what the old blender fellows are chatting about these days..

Hi, great to hear from you!

I'm currently one of the guys who is working on the Python API.

strubi wrote:

The problem about implementing a good object oriented API is: the robust Kernel. Blender has no Kernel (at least you can't really call it one). So, at least one point where all different python API people would agree: There are things you just can't to to make it stable.

Yes, that's where I'm struggling with as well

strubi wrote:

(The GameEngine is another topic - there was not even a concept of an API. It got obviously just hacked in to 'make something work')

I haven't even looked at that one yet, since the current focus is to get the 'offline' Python api to build without any problems on all platforms.

strubi wrote:

So the only approach I see to make a clean API: 1. Redesign a Blender object kernel

I've been thinking about this as well. But this involves quite a lot of work. Since this is a purely volunteer project, this may be a bit too much. Maybe it's possible to (re)design an object kernel that can be implemented and integrated in small steps. I have to look at that a little better. This has impact on all parts of blender (otherwise it wouldn't be a kernel, would it ).

strubi wrote:

2. Create a good C API (C++ still is too problematic among different platforms and compilers - *sigh*)

Could you give examples on which platforms problems arise? I mean, ghost is written in C++ as well as the game engine and these work on all platforms. Well, the game engine has problems with the physics engine, but that's another kind of problem.

strubi wrote:

3. SWIG the darn thing, or use other approaches like BOOST (which involve a lot of C++ knowledge though. And they can make size explode...) More on BOOST at http://www.boost.org.

Yep, I've tried that too. But I had problems with functions that have different return types. For example, the function Blender.Get() can return long, char* and float. This construction is impossible to use swig for.

I'm all for using swig or any other tool that helps with api implementation. So maybe a complete redesign of the Python api - that is not backward compatible - would be the only solution.

strubi wrote:

Too bad. I've got another job but it might be interesting to follow some redesign ideas, Maya-alike. Let's see what the current blender hackers work out.

Ah, I have a fulltime job aswell, but that doesn't stop me from working on something like Blender.

With regards,
Michel

Hi Strubi,

Good to hear from (read about) you. And congratulations on your new job

Let me know your private email address and let's talk a bit German outside of this forum ...

Hello Michel,

Good to see that someone took the job to design a new Python API. As I said before it would be much easier to start from scratch with having Python in mind from the very beginning. But maybe it does help you to see how others approach the integration of a scripting language. Have a look at the very good and well designed library VTK:

[ http://public.kitware.com ]

It wraps the C++ functionality for several scripting languages (Tcl, Python, Java, ...) and you can learn a lot from there BUT it would work only for a well designed kernel (as Strubi said).

My idea (while I was responsible for the Python API) was to have the API very close to what the USER sees (like the window where you can see the links between the objects, the meshes, curves, materials, etc.) and make the API look object-oriented (actually Blender's data structures ARE object-oriented) even so there are no classes in the Blender kernel which are similar to what you would expect from the stuff you see in the user interface. Anyway NaN did a great job confusing people about Python and my API disappeared within a few weeks (and yes, there was once access to NURBS in the old API when I started to work for NaN). I still get a lot of questions about my Python scripts (done several YEARS ago) but I was just tired to explain things I wasn't responsible for anymore. There was no official place to explain things and I couldn't update my webpages anymore. And I was really busy with other things.

I'm working in a much more professional environment now. I worked on Black Hawk Down and Harry Potter I and II. But I still like Blender very much and I have done some experiments which were never released (e.g. integrating the RIB export directly into Blender using C, some subdivision surfaces work [before Blender had them], using mental ray and other renderers with Blender). Anyway I'm using professional tools now (Houdini, XSI, Maya, ...) and I had the feeling that nobody was really interested to take Blender a step further into a professional tool. All the effort I started in the beginning was talked away by concepts like "Blender on mobile phones" and the direction of Blender being a "game engine" (which is GOOD, Erwin, don't get me wrong). If you look closely you will see that a lot of this concepts are now in other software packages. So the industry HAD a close look to Blender ...

If you look at the development of VTK (and other successful open source projects) there is a core team which talks much about concepts and OO design BEFORE they actually start coding. If you want to contribute to their code there are STRONG coding guidelines and very good documentation. I suggest (again) that some people try to start from scratch, make some basic tests (e.g. how to integrate several scripting languages), how to stay multi-platform, and take Blender as a source to learn about some things ...

I guess this will NOT happen. That's why I stop now but I'm willing to contribute some things I have done. Maybe I find time in the future to make my Python scripts work again with the latest Blender Python API. But I will wait for some stability before I even think about starting again with Blender coding. Because sometimes I have the feeling that all the effort I have done in the past was a waste of time

People take things I published, take it a step further without even mentioning my name. The whole thing about giving things away for free is that people give each other "credits" and respect each other. And I have long term relationships to developers of (commercial and not-commercial) software all over the world. The idea of giving examples of import/export scripts away was to convince others to do the same. This were examples how to do it. Make it nicer, user-friendlier, etc. Share experience.

Anyway, I hope the open source Blender will go well and I still support Blender (but less with Python scripts and coding). It was great to meet all this developers and artists and I wish them all the best

Cheers,

Jan

Hi Michel,

Michel wrote:

I'm currently one of the guys who is working on the Python API.

Cool. Hope you don't go nuts with this challenge
We've all wasted far too much time on trying to implement a stable
solution. I really hope you don't waste yours. Well, at least you can do with the source what you want, now

Michel wrote:

I haven't even looked at that one yet, since the current focus is to get the 'offline' Python api to build without any problems on all platforms.

I wouldn't worry about the game engine as well.. I don't think anyone would want to continue working on it, but rather design something fresh from scratch or make an interface for something like Crystal Space.

Michel wrote:

Could you give examples on which platforms problems arise? I mean, ghost is written in C++ as well as the game engine and these work on all platforms. Well, the game engine has problems with the physics engine, but that's another kind of problem.

It's not a big issue, but there are sometimes troubles with the STL (which I personally don't really like because of some overhead introduced. I can't help it, I sometimes have to look at the produced asm output...).
And when you want to modularize things, C++ DLLs are a horror among different compilers - on Windows.

Michel wrote:

Yep, I've tried that too. But I had problems with functions that have different return types. For example, the function Blender.Get() can return long, char* and float. This construction is impossible to use swig for.

Oh yes. I've had a close look at SWIG some time ago (at NaN) - it wasn't really usable at that time for all the complex object types in Blender, so I went back to hand coding the modules. But using functions that return a lot of different types can be avoided...(and can be handled faster, too).


I must admit: I've given up on Blender and concentrated on my own toolbox. So, the only thing that would really interest me at the moment is a complete redesign of the mesh & curve 'kernels'.
I think it would really pay off. It might be hell of a job to integrate a new mesh kernel into blender, but then you could program a lot of nice functionality on top with python. The current structures just make it (IMHO) impossible to write a powerful and robust API.

I've written a mesh library a year ago which allows full undo and handles all entities (verts, edges, faces) as single objects. The thing is in C++ and could be easily python-wrapped with BOOST, which is a rather intelligent C++ solution, but featured by massively nested templates, which produce a lot of code. It's really good to make a python wrap in a few lines though. Maybe you want to have a look at it..
The mesh library demo (win32) can still be downloaded at http://www.section5.ch.

I'm actually using it with Blender. But it's a bad bad hack and far from complete.

If you've ever looked at that terrible 'hacked in' VRML2 importer: The guy who has written all the underlying scenegraph code obviously had been thinking a lot about a well organized and flexible scenegraph. VRML2 might be seen as dead, but A LOT can be learnt from those formalisms introduced (see Scenegraph Walker concept, etc.) The same architecture would be great to have in a future blender kernel.
Anyway, I'm looking forward to new improvements in the good old crappy code. Keep on the good work!

- strubi

Hi,

Speaking about VRML2 - look at:

http://oss.sgi.com/projects/inventor

VRML is based on OpenInventor from SGI. It's open source today and as Strubi said: You can learn a lot from it. To be honest, I made some test with free libraries (PLIB, etc.) and had some very successful results with OpenInventor (for the scene graph, several actions [e.g. import/export/render], database for NURBS and polygons) and Qt (for the event handling, signal&slots, ...). In a few days I had a simple "Blender" version running. I mean I "borrowed" some ideas from Blender but the source code was completely written from scratch

OpenInventor books are online:

http://www.cs.indiana.edu/classes/b581/docs/inventor/Inventor_Mentor/sgi_html/index.html

http://techpubs.sgi.com/library/tpl/cgi-bin/browse.cgi?coll=0650&db=bks&cmd=toc&pth=/SGI_Developer/Inventor_TM

For Qt:

http://www.trolltech.com

PLIB:

http://plib.sourceforge.net

Sorry, the whole discussion is drifting away from Python but maybe someone wants to repeat this somewhere else (where people are discussing the future of Blender)

Cheers,

Jan

wahn wrote:

Good to see that someone took the job to design a new Python API. As I said before it would be much easier to start from scratch with having Python in mind from the very beginning.

I've been thinking about that too. But I'm having trouble with the size of the task. Therefore, what our current aim is, is to rewrite the entire Python api, which means looking at what has been implemented, take the good stuff out of it (which is a lot), and remove the parts that are giving some problems. This process has been started because of the problems with the freeze tool in intern/python.

wahn wrote:

But maybe it does help you to see how others approach the integration of a scripting language. Have a look at the very good and well designed library VTK: [ http://public.kitware.com ] It wraps the C++ functionality for several scripting languages (Tcl, Python, Java, ...) and you can learn a lot from there BUT it would work only for a well designed kernel (as Strubi said).

Thank you for the link. I have heard about it in the past, but never had a real look at it. This is definately something that I will look at when we decide to redesign the api. Our current effort is to get the new implementation to be 100% backward compatible with 2.23/2.25.

With regards,
Michel

wahn wrote:

Speaking about VRML2 - look at: http://oss.sgi.com/projects/inventor

Again, thanks for the link

wahn wrote:

In a few days I had a simple "Blender" version running. I mean I "borrowed" some ideas from Blender but the source code was completely written from scratch

great! Although I have very little experience with Qt, I do know it. Somewhere in the past I switched my interest to GTK+. But, I guess, that's simply a mather of taste

We had a little discussion about using a event library on IRC, but no real outcome was the result. Most of us are still trying to grasp the source base and maybe from there propose improvements to the structure.

Personally, I would like to do some redesign on the kernel, but this will influence the rest of blender in one way or another. This kernel would then consist of smaller components (think of all the structures in dna) and these would then be either accessible throught either the C or C++ interface or through a Python/.../.... api. I guess I just have to write down my ideas and see what others think of it.

Anyway, thank you for your ideas on the past, current and future

With regards,
Michel

strubi wrote:

Oh yes. I've had a close look at SWIG some time ago (at NaN) - it wasn't really usable at that time for all the complex object types in Blender, so I went back to hand coding the modules. But using functions that return a lot of different types can be avoided...(and can be handled faster, too).

When we decide at some point to design a complete new API, multiple return types will be named as evil and not allowed. However, as you may have read in one of my previous posts, our current effort is to replace the current implementation with something that is 100% backward compatible. I really respect everybodys effort that has gone into creating this api in the past and I don't want to sound offensive, but our humble opinion was that now was a good time to try to remove the intern/freeze stuff from Blender as it is causing a lot of confusion among all current developers and would-be developers. So, I merely was stupid enough to say "hey, here's a plan on how to fix that, and I'm willing to do it". . So far, it's looking good

strubi wrote:

I must admit: I've given up on Blender and concentrated on my own toolbox. So, the only thing that would really interest me at the moment is a complete redesign of the mesh & curve 'kernels'. I think it would really pay off. It might be hell of a job to integrate a new mesh kernel into blender, but then you could program a lot of nice functionality on top with python. The current structures just make it (IMHO) impossible to write a powerful and robust API.

Agreed, and as proposed by Jan, some/a redesign (may) need to be done to achieve this.

strubi wrote:

I've written a mesh library a year ago which allows full undo and handles all entities (verts, edges, faces) as single objects. The thing is in C++ and could be easily python-wrapped with BOOST, which is a rather intelligent C++ solution, but featured by massively nested templates, which produce a lot of code. It's really good to make a python wrap in a few lines though. Maybe you want to have a look at it.. The mesh library demo (win32) can still be downloaded at http://www.section5.ch. I'm actually using it with Blender. But it's a bad bad hack and far from complete.

Thanks for the link, I'll have a look at it at home

strubi wrote:

If you've ever looked at that terrible 'hacked in' VRML2 importer: The guy who has written all the underlying scenegraph code obviously had been thinking a lot about a well organized and flexible scenegraph. VRML2 might be seen as dead, but A LOT can be learnt from those formalisms introduced (see Scenegraph Walker concept, etc.) The same architecture would be great to have in a future blender kernel.

Well, we have the idea of removing the vrml import stuff from blender and create plugins of some sort. For example, the Yafray export script is something similar. No real plans have been made on how we could make these plugins available to the user. Maybe, nothing needs to be changed.

Thank you for your input

With regards,
Michel