Geometry registration with occlusion resolving:

Step 1. REGISTER the coordinate systems between the source and target videos using the camera parameters.

Step 2. DENOTE the perspective transformation between the two coordinate systems as a 4X4 matrix PA.

Step 3. RESOLVE the the potential occlusion between the sprite and the separated layers in the target video.

Here I picked an image as the target video, and another image as the sprite:

Monsters-University-Photos

The sprite.

 target

The target image.

target_z

The depth map of target image (generated by ACTS)

Results:

mu1

References: Prof. Guofeng ZhangThe Refilming project