major verse optimization

General discussion about the development of the open source Blender

Moderators: jesterKing, stiv

Post Reply
eskil
Posts: 140
Joined: Tue Oct 29, 2002 10:42 pm

major verse optimization

Post by eskil »

I received an E-mail form averse user who was having some issues with an application he is writing using verse. He sent me a small test application that attempted to upload an image to verse but failed.

As it turns out there where some bugs in the in the re-send mechanism. there where quite easily fixable, but is I tried to send larger images i found that verse is just very slow. sending a 256 * 256 8 bit image (4 channels) took about 20+ seconds! i removed some waits, but it still dint help.

So i have spent the last 5 nights trying to fix this, and as it turns out is wasn't that easy. One of the problems is that im trying to send 10.000+ packets over the need and trying to figure out why they don't run very good is hard with that much data to look threw.

First of all, i found that verse out going queue is just way too slow. when a new command is placed in the out going queue it has to be compared to every existing command in the que. with 16000 commands this turns out to be slow even if its just a check sum compare. so I split the queue in to 2192 queues, making it 2192 * 2192 times fewer compares. this made things a lot faster.

But still there are some problems. when you upload an image to verse you usually call the "set tile" command _a lot_. And you don't ever inside that loop call verse_callback_update (the commands that looks on the net for any incoming data). This means that you wont get any confirmation that any thing got threw, and you cant cut down the history, and basically the network chocks. This could be considered as just poorly written code by the app and not at all a verse problem, but in my opinion application developers should not have to care about this.

So I tried to fix this by making verse check the network for in coming data. pick out the acks and naks and store the command until the next time the user calls. verse_callback_update. this turned ot to be a good thing but only marginal.

So next i found that when verse tried to send 16000 packets at once almost none got there so i implemented a "sliding window" to allow the stream to bee a little bit more controlled. How ever this turned out to not wok at all! The packets just never got there at all,and both sides where waiting for the other side to send something. later i found that when both sides where receiving data and tried to confirm this, the confirmation commands (acks) got last in the queue, and since it dint send any data until it got any acks nothing happened.

So i made a separate "speed lane" and made all acks and nacks(negative confirmations) a priority. And all of a sudden the network started running like hell!

So i think the major issues are fixed now but there may still be problems, so if you have any please mail me.

Another issue since im using windows is that windows switches tasks so slow, and so seldom. Having two apps on two machines comunicate over a network may even be faster then having two apps comunicate on the same machine. I hope to get some feadback to see if this is better in linux.

the new version is up at: http://www.blender.org/modules/verse

E

Money_YaY!
Posts: 442
Joined: Wed Oct 23, 2002 2:47 pm

Post by Money_YaY! »

aww! your still working on this verse thing.

Hope you get it working on OSX some day, as I want to play with it a bit.

eskil
Posts: 140
Joined: Tue Oct 29, 2002 10:42 pm

Post by eskil »

I just uploaded a new version with a few fixes mainly to the tags system but I also found a bug that makes some subscription commands get lost some times.

E

emack
Posts: 0
Joined: Fri May 23, 2003 1:53 am

Enough, Seduce, and NGL?

Post by emack »

Hi Eskil,

I've been interested in following the work that you've been doing. Do you plan to update NGL, Enough, and Seduce now that OpenGL 1.5 (used to be 2.0) is now out? It would seem to make a very compelling development environment.

Do all of your apps (Quel Solaar, Nils Salentin, Loq Airou, and Connector) use these 3 key components? I'm curious which one you would recommend downloading to understand how the parts work together.

eskil
Posts: 140
Joined: Tue Oct 29, 2002 10:42 pm

Post by eskil »

Well, NGL Seduce and Enough are my own personal libs. and i don't really have too many plans for them. I wrote them because i needed them rather then trying to make something for every one to use. That's why there is very little documentation about them. Having said that yes you are free to use them. and i am planing on talking a bit about Enough on the Blender conference.

Yes All my apps use all of them.

If you want to download some thing, download connector, but as I am currently working hard on it there may be a lot of changes, expect a new version by the time of the conference.

About OpenGL 1.5 , i just don't know, it depends on what i will do in the future, but yes it is interesting although it will generate a lot of work.

E

emack
Posts: 0
Joined: Fri May 23, 2003 1:53 am

Post by emack »

The new version of connector sounds very interesting. What sorts of things are you adding/changing?

eskil
Posts: 140
Joined: Tue Oct 29, 2002 10:42 pm

Post by eskil »

Well its almost a complete re write. The idea is to have an application where you can see and modify any verse data.

It is the ultimate debug tool for verse development. you can do things like create a bit map node, set its name, create layers and then see how the app you are writing reacting. In the same way you can use it too see what you are doing.

I must admit that its not the most fun App i have ever written but it is very usefull.

E

leinad13
Posts: 192
Joined: Wed Oct 16, 2002 5:35 pm

Post by leinad13 »

I know that this is probably in the wrong place, but how do i compile verse?

I am using MSVC6, ive tried adding everying to the project and compiling, every file compiles, but when is come to linkings, i get lots of unressolved external symbols, such as _verse_session_set
-------------
Over to you boffins

L!13

eskil
Posts: 140
Joined: Tue Oct 29, 2002 10:42 pm

Post by eskil »

Well it is as good as any place. join the mailing list too.

Its hard to know what the issue is, but first lets see what files you are adding, all files starting with v_, verse, vs_ (if the server is what you are trying to build) and the wsock32.lib .

Then all you should need to do is to compile! If it doesn't work get back to me and i will see what i can do.

E

leinad13
Posts: 192
Joined: Wed Oct 16, 2002 5:35 pm

Post by leinad13 »

Right i added all the v_ vs_ and verse files, to my project, i also added wsock32.lib to be used when i Link.

Code: Select all

v_network_que_old.obj : error LNK2005: _verse_session_get_size already defined in v_network_que.obj
v_network_que_old.obj : error LNK2005: _v_nq_create_network_queue already defined in v_network_que.obj
v_network_que_old.obj : error LNK2005: _v_nq_destroy_network_queue already defined in v_network_que.obj
v_network_que_old.obj : error LNK2005: _v_nq_send_queue already defined in v_network_que.obj
v_network_que_old.obj : error LNK2005: _v_nq_send_buf already defined in v_network_que.obj
v_network_que_old.obj : error LNK2005: _callback_send_packet_ack already defined in v_network_que.obj
v_network_que_old.obj : error LNK2005: _callback_send_packet_nak already defined in v_network_que.obj
v_network_que_old.obj : error LNK2005: _std_queue_size already defined in v_network_que.obj
vs_node_bitmap_alt.obj : error LNK2005: _vs_b_create_node already defined in vs_node_bitmap.obj
vs_node_bitmap_alt.obj : error LNK2005: _vs_b_destroy_node already defined in vs_node_bitmap.obj
vs_node_bitmap_alt.obj : error LNK2005: _vs_b_subscribe already defined in vs_node_bitmap.obj
vs_node_bitmap_alt.obj : error LNK2005: _callback_send_b_unsubscribe already defined in vs_node_bitmap.obj
vs_node_bitmap_alt.obj : error LNK2005: _callback_send_b_init_dimensions already defined in vs_node_bitmap.obj
vs_node_bitmap_alt.obj : error LNK2005: _callback_send_b_layer_create already defined in vs_node_bitmap.obj
vs_node_bitmap_alt.obj : error LNK2005: _callback_send_b_layer_destroy already defined in vs_node_bitmap.obj
vs_node_bitmap_alt.obj : error LNK2005: _callback_send_b_layer_subscribe already defined in vs_node_bitmap.obj
vs_node_bitmap_alt.obj : error LNK2005: _callback_send_b_layer_unsubscribe already defined in vs_node_bitmap.obj
vs_node_bitmap_alt.obj : error LNK2005: _callback_send_b_layer_set_tile already defined in vs_node_bitmap.obj
vs_node_bitmap_alt.obj : error LNK2005: _vs_b_callback_init already defined in vs_node_bitmap.obj
Debug/Verse.exe : fatal error LNK1169: one or more multiply defined symbols found
Error executing link.exe.

Verse.exe - 20 error(s), 38 warning(s)
Heres the error i got.

Sorry i know that this isnt the right place for this, should i start a new thread in coding.

Also how do i join the mailing list i can't seem to find it anywhere.
-------------
Over to you boffins

L!13

leinad13
Posts: 192
Joined: Wed Oct 16, 2002 5:35 pm

Post by leinad13 »

Right sorrya bout this, i found the problem, perhaps this occured due to your optimization recently.

What happened was there are 2 sets of duplicate files, they might not be exact copies but it caused the break in the build.

One of them is vs_node_bitmap_alt.c and the other was v_network_que_old.c

I removed these from my build and it compiled with not one error.

I am not sure whether you knew this or not, but i thought i would bring it to your attention.
-------------
Over to you boffins

L!13

Post Reply