Hello board, I have been brainstorming an idea for quite some time and am hoping to toss it out and see if anyone has any ideas for a solution to a problem I've been facing in my work as a video designer and director of theater.

I have been working on developing a case study for "augmented reality theater" using projectors to create immersive environments in my work for about 2 years now and have reached a limit of my knowledge in coding and engineering.

What I would like to accomplish, is using a 3d rendering program (like Blender) and a data manipulation program (like Isadora) to run a theatrical show that allows an actor to exist within a virtual world that is then video mapped with projectors into an actual live space/theater.

While researching the capabilities of Blender and Python this is the proposed solution I have come up with:

Utilize Blender to create a scale model of the theater/lab space

create both a suit and prop that transmits OSC position data (think wiimote)

Use Isadora to receive and transmit position data into game engine for Blender (compositing position data with game effects)

Hang projectors in lab space to match the position of virtual cameras in the game engine in order to video map the environment in the game engine directly to projectors that would then recreate the "in-game" environment to real space in real-time. (Thinking of using Syphon framework into Isadora which then send video out to projectors.)

Theoretically, this would allow a character to exist within and control the virtual world created in Blender. I have a theater in NYC downtown area that I am able to workshop these ideas in but have reached my limit in the programming end. If anyone feels like offering constructive input to this proposed setup (improvements/changes) or would like to offer assistance in building this framework I am in eager need of help with this. In the meantime I will be scanning the forums for any other aid and doing what I can to learn more Python :p


Reference:
Isadora
Syphon
Video Mapping Wiki Article
Videomapping Demo Video

Thanks!
-Jbrem003[/url]
_________________
Lucidity and know-how are required to be a revolutionary

~Eugenio Barba

Found this in a later posting and is also very applicable. Looking into the methods described here. Some of this capability is in Isadora, but not all of it. A visit to MIT sounds in the works

CoDEmanX wrote:

Paper about closed world tracking: http://vismod.media.mit.edu/pub/tech-reports/TR-403.pdf Paper Multi-cam multi-user tracking: http://research.microsoft.com/en-us/um/people/jckrumm/Publications/EZL%20Tracking/final01.pdf Paper Multi-Object tracking: https://docs.google.com/viewer?a=v&q=cache:p9hjmiohlQ4J:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.69.7070%26rep%3Drep1%26type%3Dpdf+&hl=de&pid=bl&srcid=ADGEEShFN311v0TcEbYj27GYxXRt_D4pNqx0WYA71pLYF0-yfq-16AhwBzhrIAIMr0k6g0o4riHMqG062WwV7oZ3UyyCywCyVP-1nPyeF5sGkI4k9IiPJ4lAs9iRXvuNTBp56iZAaMjR&sig=AHIEtbTbWFrrEOIkIA6xEZnv7AsQFAxpeA

_________________
Lucidity and know-how are required to be a revolutionary

~Eugenio Barba

Hey jbrem003, I'm working on something very similar, also through the game engine.

I've currently got perspective-corrected projections in a room (via a Kinect camera) and am trying to move the code over to the Game Engine to better control it in real-time.

Would be interested in chatting further about what you're up to. Feel free to email at tom.svilans@gmail.com.

Best,

Tom Svilans
tomsvilans.com

it depends, I don't know the details of your project but Blender uses Python, and Python, or CPython to be fair, is probably the slowest scripting solution that i know.

The blender game engine is also really old an it adopts a deprecated approach to old OpenGL APIs so what you produce with the Blender Game Engine it's not even granted to work.

jbrem003 wrote:

Hello board, I have been brainstorming an idea for quite some time and am hoping to toss it out and see if anyone has any ideas for a solution to a problem I've been facing in my work as a video designer and director of theater.

I am a playwright and software engineer who is interested in similar problems. It might be interesting to compare notes.

I am working on slightly more modest problems with Blender and theater.

First, making a 3D model of the theater in Blender.

Second, constructing backdrops, props, et cetera with monochromatic fabric/paint.

Third, make a video of a staged performance with cameras that are calibrated with respect to the location of the stage.

Fourth, using the known geometry of the objects to create UV maps which are used at compositing to project images or video onto the objects. A mask is created so that only the portion of the UV map which corresponds to a specified chroma key is used (so that actors occluding the backdrops and props would not have the UV mapped image composited over them).

So far I've done proof of concept work around doing each of these steps manually, but automating the process is something of a challenge.

Although I'm continuing to look into using Python within Blender for the whole process, I'm also seriously considering just using Python to export UV map information in some fashion to use with CImg or FFMPEG or some other more C++ centric platform. As a Software Engineer, developing large applications in Python offers some serious challenges that are easily overcome by using C++.

I am also interested in the whole process of Video Mapping in terms of using projectors within a traditional proscenium and outdoors at festivals, however I have been looking into a control process that is more along the lines of having the sound person (who has more free time than the light person) cue the specific video sequence using a customized panel of buttons on a dedicated computer based on annotations in the script (similar to the way Foley is done).

Blender figures into the Video Mapping process as a way of projecting onto irregular objects and/or accurately mapping onto the backdrops based upon limitations of projector placement.

Anyway, your project sounds more ambitious from a control standpoint.

One of the things I am trying to get at is efficiently being able to composite the video from the stage with the UV mapped images so that the video of a production on the stage can be composited and routed to the web as quickly as possible. Right now finding some way of using FFMPEG seems to be the most promising approach.

Would love to discuss your project further.

Cheers!

Robert