Import: Limit to file size?

The interface, modeling, 3d editing tools, import/export, feature requests, etc

Moderators: jesterKing, stiv

Reynolds
Posts: 19
Joined: Tue Mar 27, 2012 7:54 pm

Import: Limit to file size?

Postby Reynolds » Tue Nov 13, 2012 10:14 pm

Hello folks,
I'm back to making some nice renderings with the x3d to blender pipeline I created a while back (see my older posts if you are interested).

However, I have come across a problem when importing x3d-files into blender (both via python directly or via the GUI) that exceed a certain size (about 800 MB or so): blender will just use 100% cpu time, do nothing with it and just "freeze" when I start the import. Below 800 MB file size for the x3d, everything works fine.
I tried it on a linux system with 64 GB RAM (2.64a and 2.63) and on a Win 7 64bit system with 8 GB Ram (again 2.64a and 2.63). Same result....

So is there a limit on the file size of the x3d blender can import??? that would suck more than I can say, I hope you guys can help me out with this!

thanks in advance, all the best

Reynolds


ps: Sorry for the double post, I first posted this in the python section but noted later that this forum might be more suitable. so in case admins want to remove on post, please feel free! thanks!

CoDEmanX
Posts: 894
Joined: Sun Apr 05, 2009 7:42 pm
Location: Germany

Postby CoDEmanX » Tue Nov 13, 2012 10:58 pm

I experienced long loading times and a lot of memory usage when I imported a large OBJ (>50 MB, but still under 100). The reason was mainly the splitting into objects by material, without the splitting, it used little memory and went a lot faster.

Not sure about your problem... Does it already use much memory and take long if it's like 400 MB, or is it really starting to get stuck only if the import file is >800MB?
I'm sitting, waiting, wishing, building Blender in superstition...

Reynolds
Posts: 19
Joined: Tue Mar 27, 2012 7:54 pm

Postby Reynolds » Tue Nov 13, 2012 11:24 pm

CoDEmanX wrote:I experienced long loading times and a lot of memory usage when I imported a large OBJ (>50 MB, but still under 100). The reason was mainly the splitting into objects by material, without the splitting, it used little memory and went a lot faster.

Not sure about your problem... Does it already use much memory and take long if it's like 400 MB, or is it really starting to get stuck only if the import file is >800MB?


Hello,
And thank you for the reply! Under about 800MB, it doesnt take that long ( with a python script) to import the file, I would say about 1 minute fromthe start of the import to the start of the rendering. Above 800 MB or so, it really frezes the code, I let it try for 12h over night on the 64 GB RAM machine, all it did was slowly fill up the RAM but do nothing, neither via script nor via GUI.

So yes, I am pretty sure there must be a limit on the import file size...

stiv
Posts: 3646
Joined: Tue Aug 05, 2003 7:58 am
Location: 45N 86W

Postby stiv » Wed Nov 14, 2012 12:14 am

So yes, I am pretty sure there must be a limit on the import file size...


Color me skeptical. I cannot think of anything offhand that would create a hard limit on file sizes. However...

I can imagine some algorithmic limitations where a bit of code that runs in reasonable time takes forever on a big dataset. A bubble sort, for example, runs fast on 20 items. It takes forever on 5 million. (Yeah, we are talking Big O notation)

No idea about x3d format or what is in your file, but 800 MB is a large file.

Another possibility is a bug in the importer or bad data.

If the importer is a python script, it should be easy to instrument it with some progress reporting print statements.

CoDEmanX
Posts: 894
Joined: Sun Apr 05, 2009 7:42 pm
Location: Germany

Postby CoDEmanX » Wed Nov 14, 2012 8:09 am

maybe convert to another format and try that, or maybe get Bratwurst branch from graphicall.org and try that assimp import (requires different format!)

X3D conversion:
http://www.web3d.org/x3d/content/exampl ... onversions
I'm sitting, waiting, wishing, building Blender in superstition...

Reynolds
Posts: 19
Joined: Tue Mar 27, 2012 7:54 pm

Postby Reynolds » Thu Nov 15, 2012 12:01 pm

stiv wrote:
So yes, I am pretty sure there must be a limit on the import file size...


Color me skeptical. I cannot think of anything offhand that would create a hard limit on file sizes. However...

I can imagine some algorithmic limitations where a bit of code that runs in reasonable time takes forever on a big dataset. A bubble sort, for example, runs fast on 20 items. It takes forever on 5 million. (Yeah, we are talking Big O notation)

No idea about x3d format or what is in your file, but 800 MB is a large file.

Another possibility is a bug in the importer or bad data.

If the importer is a python script, it should be easy to instrument it with some progress reporting print statements.


Thanks a lot for your reply and sorry for the late response. I understand that these algorithms don't scale linearly, so my first try indeed was to wait for 24h, leaving the import running (below 800MB, it takes about 1 min, so 24h should cover all bad scaling :) ).

What I found is that blender just stays in the frozen state, allocates memory and deallocates it in a never-ending cycle (the memory consumption over time does indeed look like a saw-tooth curve), both on linux with 64 GB and on Win with 8 GB.

What I did then was to first import a 600 MB file (worked fine), and then another 600 MB file on top of that, worked fine as well. So the problem is indeed the single file size, not the total size of all files, which is probabaly just determined by your available RAM..

Reynolds
Posts: 19
Joined: Tue Mar 27, 2012 7:54 pm

Postby Reynolds » Thu Nov 15, 2012 12:06 pm

CoDEmanX wrote:maybe convert to another format and try that, or maybe get Bratwurst branch from graphicall.org and try that assimp import (requires different format!)

X3D conversion:
http://www.web3d.org/x3d/content/exampl ... onversions


thanks for the reply, I cannot easily convert the data to something else, because the overall data size is about 4 TB, and converting it would take forever...

Could you tell me about the bratwurscht branch a few words? What's so special about it?

thanks a lot,
please see the post one above for what I found about the file size limit in case you are interested!

Thanks and all the best!

CoDEmanX
Posts: 894
Joined: Sun Apr 05, 2009 7:42 pm
Location: Germany

Postby CoDEmanX » Thu Nov 15, 2012 12:30 pm

Bratwurst branch has assimp integrated, which allows to import couple file formats. Unlike the python importers, it should take full advantage of multicore systems, maybe even use less memory and compute more efficiently...
I'm sitting, waiting, wishing, building Blender in superstition...

Reynolds
Posts: 19
Joined: Tue Mar 27, 2012 7:54 pm

Postby Reynolds » Thu Nov 15, 2012 3:03 pm

CoDEmanX wrote:Bratwurst branch has assimp integrated, which allows to import couple file formats. Unlike the python importers, it should take full advantage of multicore systems, maybe even use less memory and compute more efficiently...



Oh that sounds good. I will try to find a build for linux, as I'm not to good at compiling complicated stuff myself :)

CoDEmanX
Posts: 894
Joined: Sun Apr 05, 2009 7:42 pm
Location: Germany

Postby CoDEmanX » Thu Nov 15, 2012 6:29 pm

At least one November build for 64bit Fedora:

http://graphicall.org/blender/linux?keywords=Bratwurst
I'm sitting, waiting, wishing, building Blender in superstition...

stiv
Posts: 3646
Joined: Tue Aug 05, 2003 7:58 am
Location: 45N 86W

Postby stiv » Fri Nov 16, 2012 3:50 am

I do love a mystery! But I like solutions even better.

That you can process a 600 MB file in reasonable time suggests this is not a Big O problem - like a linear search thru an ever-increasing data structure.

That leaves bad data - something like a loop or recursion that gets processed over and over, or a bug in the importer - possibly related handling badly-formed data.

I wonder if it is possible to pare down or split your large file?


Return to “Interface & Tools”

Who is online

Users browsing this forum: No registered users and 0 guests