Re: Ocean Simulator: The research continues...

[simonharvey]

Hello
I am currently playing around with the fftw fourier library with the intention
of coming out with some routines that would implement a deep ocean simulator using a
filtered frequency spectrum and phase shifting in frequency space (based on code
derived from mwave which has been put into the public domain). This is similar
to the wave effect but it gives a much more realistic result.

So far I have:
* back ported the effect over to effects.c - which compiles
* implemented an untested GUI
* added the OceanFFTEff to DNA_effect_types.h and BKE_effect.h

however when I come to compile the modified source tree I get an error in the
build process:

---------------------------------------------------------------------------------------

gcc -o /home/simon/develop/build/linux2/makesdna /home/simon/develop/build/linux2/source/blender/makesdna/intern/makesdna.o -L/home/simon/develop/build/linux2/lib -Llib -lblender_guardedalloc
../build/linux2/makesdna /home/simon/develop/build/linux2/source/blender/makesdna/intern/dna.c
Running makesdna at debug level 0
Program version: $Id: makesdna.c,v 1.8 2004/03/20 22:55:38 zuster Exp $
Align pointer error in struct: OceanFFTEff *h0
Error: alphalength is 84, alphalen mod8 4, mul 1, sizeof(fftw_complex)=8
Align pointer error in struct: OceanFFTEff *ht
Error: alphalength is 92, alphalen mod8 4, mul 1, sizeof(fftw_complex)=8
Align pointer error in struct: OceanFFTEff *result
Error: alphalength is 100, alphalen mod8 4, mul 1, sizeof(fftw_complex)=8
scons: *** [/home/simon/develop/build/linux2/source/blender/makesdna/intern/dna.c] Error 1
scons: building terminated because of errors.
[simon@lunar oceanblender]$

---------------------------------------------------------------------------------------

I have added extra debugging info to make it clearer.

The code for the OceanFFTEff is shown below:

---------------------------------------------------------------------------------------

Code: Select all

typedef struct OceanFFTEff {
	struct OceanFFTEff *next, *prev;
	short type, flag, buttype, stype; /* from the wave effect */
	
	/* this is to store the current effect context, the other values are
	 * cycled through when the 'Recalc' button is pushed.
	*/
	struct OceanFFTEff * current;
	
	int fftsamplesize;
	float waveRange;
	float heightScale;
	float windX;
	float windY;
	float gravity;
	float lambda;
	float 	offset_x;
	float 	offset_y;
	float 	offset_z;
	float 	size_x;
	float 	size_y;
	float 	size_z;
	fftwf_complex  *h0;
	fftwf_complex  *ht;
	fftwf_complex  *result;
	fftwf_plan fplan2;
} OceanFFTEff;

---------------------------------------------------------------------------------------

I have added the marked lines of code to the makesdna.c source file:

---------------------------------------------------------------------------------------

Code: Select all

	/* insertion of all known types */
	/* watch it: uint is not allowed! use in structs an unsigned int */
	add_type("char", 1);	/* 0 */
	add_type("uchar", 1);	/* 1 */
	add_type("short", 2);	/* 2 */
	add_type("ushort", 2);	/* 3 */
	add_type("int", 4);		/* 4 */
	add_type("long", 4);	/* 5 */
	add_type("ulong", 4);	/* 6 */
	add_type("float", 4);	/* 7 */
	add_type("double", 8);	/* 8 */
	add_type("void", 0);	/* 9 */
	
	/* FFTW ADD TYPES added to allow DNA code to record FFT data */
	add_type("fftwf_complex", sizeof(fftwf_complex)); /* 10? */
	add_type("fftwf_plan", sizeof(fftwf_plan)); /* 11? */
	

---------------------------------------------------------------------------------------

however there is a problem - I do not want blender to store the contents at the arrays of
fftwf_complex *h0;
fftwf_complex *ht;
fftwf_complex *result;

and also the fftw plan 'fplan2' because they are generated dynamically and (in case of the plan)
are machine dependant. My Ocean FFT rengeneration routine creates new arrays and plans if they are NULLed -
which is what I want.


I guess what I would want to know is: has anybody else had any experience with DNA that can help me?

Kind Regards
Simon Harvey

-------------------------
Post ChangeLog:
Modified the title, on Thursday 5th August
"Ocean Simulator: Compile Problems" to"Ocean Simulator: Update"

[eeshlo]

There is an alignment restriction for all DNA structs, the size must be a multiples of 8 (bytes). So you might need to add a padding variable. Can't see from your code what the size should be, since I don't know the size of fftwf_plan or fftwf_complex. I suppose the complex var size (if float) is 8.
Also, I don't know if adding another external library is 'accepted' so to speak, I guess you would have to ask Ton and the other coders about that.
A 2d fft routine is not that difficult to create yourself, It doesn't take that much code, and it would probably more easily accepted for inclusion in the cvs tree.
I know the fftw library is supposed to be the fastest, but your code wouldn't strictly require realtime. Not that it would be impossible to run in realtime even with your own code.
Not that I -really- know what I'm talking about anyway (as far as system stuff is concerned), you better ask the other coders, try the #blendercoders channel if you didn't do so already.

edit : Also, if by any chance adding new variables to DNA is part of the problem, make fftwf_complex and fftwf_plan void pointers. Then cast them to the correct type in your code. In any case, adding those variables to DNA shouldn't be necessary, or is it? I mean, do they need to be saved with the blendfile?

[simonharvey]

I have added some padding bytes between the last float variable and the first *hX pointer, which got rid of the alignment problems as well as changed the pointer types from fftwf_complex * to void * however this still requires experimentation to determine whether blender nulls these pointers, save them blindly or packs the memory referenced by these pointers into the blend file when it gets saved.

A screen shot below shows the initial layout of version 0.01 of the GUI:



What happens is that the user changes the values and then clicks 'Regen' which regenerates the FFTW arrays (which takes a second or two on a 1.6GHz linux machine). When the effects get applied to the mesh it displaces the Z component of the mesh just like that of the wave effect.

I was up to 11:34pm last night trying to figure out why it kept crashing whenever I changed a value of the sliders - as it turns out one of my FFTW array addressing macros gives a wrong value that address a value outside of the array - I will have to get on top of that tonight.

So this is no where complete but it is coming along and I am starting to understand blenders effects code better.

Kind Regards
Simon Harvey

p.s. I also crossposted the initial post on both the animation and blender general forums (1) not knowing what would be the most appiorate forum and (2) not 100% knowledgeable about forum equette, I am sorry if I have stood on anybodies toes or aggrivated anybody.

[kencanvey]

Hi Simon,

I am really looking forward to this being released, I have been hoping for somthing like this for ages, but am not well disposed to programming .

Great work, and good luck.

Ken

[jesterKing]

p.s. I also crossposted the initial post on both the animation and blender general forums (1) not knowing what would be the most appiorate forum and (2) not 100% knowledgeable about forum equette, I am sorry if I have stood on anybodies toes or aggrivated anybody.

No problem For coding issues, this forums is almost always the best for coding questions in Blender. No damage done here! You were just witnessing prompt moderation work Anyway, keep the work going with the ocean effect!

/jesterKing

[harkyman]

Yeah - keep working. I've been using textures, displacement and the noise effect to create a realistic ocean mesh, but it keeps coming up short of my requirements. This will be useful to many people.

[simonharvey]

Hi all

Well I managed to fix the memory accessing bug as reported above and now I actually get stuff forming on the screen!!!
see below:



however there is a small problem:
The Effect isnt recalculated again when the frame is advanced *or* when the object has to be rendered.

Initially I thought that the system called B_CALCEFFECT when it came to render it in object_buttons.c however the
only place that this gets called is from within object_buttons.c itself, just by UI widgets that only set values.

Then I ran the blender 2.33 code base through cscope to find all of the functions that call object_wave(Object*) but all it came up with was the following:

Code: Select all

0 blender.c        setup_app_data      306 if(ob->effect.first) object_wave(ob);
1 displist.c       makeDispList       1657 if(ob->effect.first) object_wave(ob);
2 displist.c       test_all_displists 2260 if(ob->effect.first) object_wave(ob);
3 buttons_object.c do_object_panels   1077 else if(eff->type==EFF_WAVE) object_wave(ob);

(please forgive the code tags, I just wanted the PHP parser to handle the tab spaces correctly)

The displist.c calls are just used to initalise the structures, they are not used to modify the mesh just before rendering or
when the frame is advanced.

So *if* anybody has had any experience with the effects system: how does it apply an effect when the frame is advanced and when the object is about to be rendered?

Kind Regards

Simon Harvey

[simonharvey]

I am going to take a closer look at B_NEWFRAME and MakeDispList() messages / functions. I guess when these leads dry up then Ill probably come screaming for some help. For thoes who didnt post, knowing what the answer was: thankyou - I need to learn myself.

BTW: In the above screen shot the reason why it looks so noisy is because I have the size mapping scalars set to such a small value meaning that the mapping functions returns a value from the FFTW array that isnt 100% smooth. If the ScaleX and ScaleY values are large enough you get something that looks wave like. However the next feature that I want to implement would be linear and maybe bicubic interpolation to give smooth interpolation between FFT values.

Kind Regards

Simon Harvey

[jesterKing]

I think you should look at how the particle effects are updated. If you look at the softbody effects in tuhopuu you might get some clues for the update, too. It's been a while since I ported those, so I can't tell you from the top of my head.

/jesterKing

[simonharvey]

Well, I had another look at the code and discovered that the place where blender creates its geometry is in the DispList function call, After adding my apply_oceanfft_effect(Object * ob) functions to it where the object_wave functions were being called I managed to get something that rendered well.

Naturally there were problems, The first being that if I save the scene with the OceanFFT effect then I reopen it blender crashes, so it must save the void * pointer, i'll guess that I will have to look and see what it does to the Particle effect since that also uses pointers to hold it's data.

I am going to use B-Spline and Bilinear interpolation to improve the output. and maybe add some UI comtrolls to decide which type should be used.

It was so cool seeing my effect finially come up in the 3D window, it was a great feeling

[iluvblender]

Congratulations on your sweet progress. Eagerly waiting for your feature to get into the official build.

- Satish.

[simonharvey]

Well here is another status update,

I have created another file that holds all of the C code in it for the Ocean FFT effect, this is to make it easier to port it to blender 2.34 (as well as later versions). Right now it works on 64x64 FFT sample size as seen by the screen shot below:



You can also see the updated GUI that shows my plans for more sophisticated interpolation.
However there is still a bug with the FFT code. If the FFT sample size is anything but 64x64 then the effect decays into this:



You can see that there are some waves forming but it looks noisy.

I have made a small video of a 'sample ocean', here:

**Link Removed**
- it is about 390Kb

I can't see which direction the waves are ment to be travelling, but they do look 'waveish' - a healthy start!

I thought that I had fixed blenders segfaulting when I load up a .blend with an OceanFFT effect saved in it, but alas it keeps on crashing. Which is another thing to to sort out.

anyway, that is it for now.

Kind regards
Simon Harvey

[harkyman]

Glad this is progressing! I think that Blender could use some real-world simulation capabilities, and this is one of them.

[simonharvey]

I knew that this would happen:
Due to the demanding nature of my masters I cannot focus on it and also on blender development at the same time, because of this I have to pospone the development of the ocean FFT effect perhaps until the beginning of next year.

I know that some were looking forward to its development/inclusion however it is providing to be too much of a distraction and a mental load at a time when I should really be getting my thesis done.

The Effect involves alot of math, in fourier space that is quite complicated and difficult to understand and I feel that the expectations that others have is binding me to something that is getting in the way.

If it turns out that I cannot keep going on at it, then I will tidy up the code and release it on my website with instructions with how to integrate it into the code base. However it is still incomplete and I do not know what 'minimal' functionality is needed to have it included even into the tophuhu build.

I am sorry if I falsely raised the hopes of anybody.

Kind Regards
Simon Harvey

[joostbouwer]

Man thats sad!!!!

Why don't you post your code right now. Maybe others could help you out??? And with helping out I do not mean steal the credits you deserve. Any way thanks for everything you've done and good luck!

Joost