Pifu HD - 3d Character Object from Single 2d Photo

I saw someone post about this on Twitter. It can take a 2d image and convert it to a pretty decent (most of the time) 3d model using neural networks. It's on GitHub and there is some way to compile it for Windows and Linux, but I think an Nvidia card is needed. Luckily it can also be run in Google Collaborate right in the web browser.

Pifu HD Google Colab Link

There's a video showing how to use it, but the browser has an Asian language displayed.

So the steps to run this are:
Make sure you're using Google Chrome and signed in to a Google account. I couldn't get this to work in Safari.
1. Click on the "Copy to Drive" button near the top. This will open a new browser tab.
2. At the top right click the pull down "Connect" > "Connect to Hosted Runtime"
3. Scroll down and click on each of the [ ] brackets next to the code. You have to skip over some text and video links in each section. The brackets will be next to some code to run.
4. After clicking on the "Configure input data" bracket section. This code loads the sample images. (Skip the "If you want to upload your own picture,.." part. I never could get it to work. So uploading your own photo manually works better)
5. Click on the folder icon on the very far left. This will give a list of sub-folders. Pick pfiuhd>sample_images
6. You can hover to the right of "sample_images" and three dots will appear. Click this and you can upload your own photo. It's best to have a straight on shot if you can with a relaxed pose and no other people in the image. You can crop them out or you can isolate the person and just have them on a white or transparent background. The software tries to pick the person out, but sometimes it has trouble. So masking the background can help most of the time.
7. After you upload your image, go to the part of the code that has the "# example image" comment in green. You want to change "test.png" to whatever the name of the file you just uploaded is. Then click on the bracket [ ]. If you already clicked on that, you can just click it again.
8. Click on the [ ] cd/content section.
9. After this you can manually click on each [ ] bracket all the way down, or you can go up to the menu bar and click Runtime > Run After. This will automatically run all the code in sequence after the last one you ran. This can take a few minutes to run, so let it do it's thing.
10. Once this is done, it will create a .png file with some type of normal map that I have no idea how to use, an .obj file, and a movie.
11. When the computation is done, a new section will pop up called "results". Open each of these so you have results > pifuhd_final > recon. The finished files will be there and you can hover over to the right and three dots will appear. Just click download and then you will have the 3d object.

So of course this won't replace someone with a lot of skill in modeling and sculpting. But this may help develop some 3d characters more quickly. Some images don't work for some reason. So I've had some luck either just changing them from jpg to png for some reason, cropping them better, or removing the background. Sorry for the wall of text. I guess I could make a screen recording of how to do this. If anyone has any questions, just ask and I'll try to help.
 
Last edited:
Back
Top