constructing a mesh from an AI generated set of images.

[1234d]

My question relates to how photogrammetry might be used to generate 3d meshes from AI generated images as opposed to more traditional photography. I am not very familiar with photogrammetry, Would you expect this to work?

In a 3D programme I can take a basic head model and generate reasonably consistent images using AI (through a process called controlNet) that have alot of new details. Each time I generate a new view of the model the image will change somewhat, but by and large will be quite similar to the previous view. I can generate lots of these images.

Do you suspect that it should be possible to generate a 3D mesh from these images using photogrammetry?

I have given this a test today, initially Meshroom failed at the camera init stage but on retrying it got passed this stage and failed at the structure from motion stage which I can see from the docs is where the feature matching happens.

I think this is an interesting question and would appreciate your expertise in this.

[natowi]

Photogrammetry is not the best solution for AI generated images, as it relies on similarities in the images for feature detection and matching to calculate the camera position and depth. AI Images will likely get much better and more consistent, but even models ("virtual turntables") trained for that purpose lack the consistency required for photogrammetry at the moment. "reasonably consistent" is not good enough.

You should probably take a look in AI approaches for Depth estimation from images like https://github.com/facebookresearch/pifuhd
or https://github.com/thygate/stable-diffusion-webui-depthmap-script + object turning model + depth map fusion

[1234d]

I suspected that might be the case and you have saved me the RnD time of me figuring this out for myself. I am aware of the AI approaches and have had some success with the depth map approach, though you obviously only get depth estimation in one direction so for more complex objects with occlusion etc. it breaks down pretty quickly.

Thanks for your insight on this, appreciate it.

[natowi]

I moved your questions to discussions, as this a little bit off topic. Do you have sample images to share? Maybe I can modify the tolerance to force matching and reconstruction.
There are similar problems when trying to reconstruct subjects that move slightly. Also I´d be interested to hear what you are trying to achieve :)