Project 4A

Description

The main goal of this project was creating a photo mosaic from several images made from the same location but with different angle of view.

Step 1: Shoot and digitize pictures

For this project I took all photos with my iPhone 14 Pro Max and than compressed their resolution to 756x1008.
Here are some pictures I took for mosaic creation:

View from Study Hall Rooftop Lounge

University way photos

Night photos of The Campanile and other surroundings from the top of Campbell Hall

Step 2: Recover homographies

To recover homographies, I first marked correspondence points on image pairs with the help of https://cal-cs180.github.io/fa23/hw/proj3/tool.html

[wx']   [h11 h12 h13]   [x]
[wy'] = [h21 h22 h23] * [y]
[w]     [h31 h32 h33]   [1]

Based on the homography matrix operation above, for every pair of corresponding points I created two linear equations in the form:

h11 * x + h12 * y + h13 - (h31 * x + h32 * y + h33) * x' = 0
h21 * x + h22 * y + h23 - (h31 * x + h32 * y + h33) * y' = 0

I then used numpy.linalg.svd to solve the group of linear equations and picked the best solution based on smallest singular value.

Step 3: Warp the images

To warp an image with the homography matrix, I first warped the corners of the image to get the warped image size.
I then warped every pixel of the original image one by one, changing all neighboring pixels in the projection image in case the warped pixel had float coordinates.

Here are some examples of warped images for pictures from Study Hall Rooftop Lounge:

First image original

Second warped to first

First warped to second

Second image original

Example of Campanile warped to the first night image:

First image original

Campanile warped to first image

Step 3.5: Image Rectification

To rectify images, I followed followed project instructions for correspondence points.
I then used calculated homography and used image warping logic to warp the image into a rectangle.

Here are some examples of rectified images:

Project 4 instructions on my monitor in safari browser

Rectified instructions to wrong aspect ratio

Rectified instructions to correct aspect ratio (somewhat readable)

Properties company sign outside

Rectified company sign

Beautiful mosaic window of a neighoring house

Rectified window (top half)

Step 4: Blend images into a mosaic

All of my images were taken with slight rotation left/right.
To combine images into a mosaic, I accounted for the changed height of the warped image.

When blending images in case where pixels overlap, I used a weighted average to blend the images together.
Weight for blending was based on horizontal distance to center of the combined images.

Here are pairs of images and their mosaics:

View from Study Hall Rooftop Lounge

University way photos. This one has the most defects in the background as the distance to the road was way closer than background trees and images.

Night photos of The Campanile and other surroundings from the top of Campbell Hall. This is my personal favorite, I took these photos while watching stars for astronomy class.

Extra stuff

Here are some mistakes I made on my first attempts of homography calculation and image warping.

Wrong signs in linear equations.

Didn't account for pixels warping to float coordinates (just rounded them).

Project 4B

Description

The main goal of this project was creating a photo mosaic from several images made from the same location but with different angle of view.
Different from the 4A, manual image correspondence is not allowed.
To generate homographies, corresponding points on the images must be selected automatically.

Here are the images I will use for different step illustrations:

Step 1: Detecting corner features in an image

For this step I just used the provided implementation of Harris.

Step 1.5: Selecting best corners in images

For this step I implemented anms as suggested in the paper.
I used suggested param values: n_ip = 500, c_robust = 0.9

Here are the corners I had left after anms:

Step 2: Extracting a Feature Descriptor for each feature point

For this step as suggested in project instructions, for each left corner I extracted feature descriptor as
8x8 image block extracted from 40x40 image block centered at the corner coordinates.
I also did normalization as suggested.

Step 3: Matching these feature descriptors between two images

As suggested in lecture, to match points on two images,
for each corner of the first image I iterated over all corners of the second image
and calculated differences between corresponding feature descriptors.

Then I would only produce a match if the best match between fixed corner of the first image was significantly better than
any other possible match (for other corner of the second image), as suggested in lecture.

Here are the matches I got:

Step 4: Use a robust method (RANSAC) to compute a homography

I implemented RANSAC as suggested in lecture.
I would repeatedly pick 4 random points from the matches, compute homography based on them,
then check how many point pairs fit this homography with some allowed error.

Finally I would select matches depicted by the homography with the most inliers.

Here are the matches I got:

Step 5: Proceed as in the first part to produce a mosaic

Same images as in manual mosaics:

First image

Second image

Manual mosaic

Automatic mosaic

First image

Second image

Manual mosaic

Automatic mosaic

All automatic mosaics I got (more than manual since I didn't have to mark the points):

Cookies

I also tried generating mosaics out of more than two images.
I tried naive idea of computing homographies between image pairs
And then warping all images to the plane of the left most image

But it seems like the small errors in homographies would lead to great errors when multiplying so I got pretty bad results.
It was still fun to try though.

What I learned

I was surprised to see how automatic mosaics turned out same or even better than manual ones.
I think that manual selection of points leads to errors in coordinates that might be more significant than errors in automatic feature matching.

It was also interesting to see how RANSAC helped a lot to get rid of outliers in matches.
It's fascinating to me how this random process of picking points and computing homography worked so well.

Finally it was fun to be reminded how small errors in individual homography computations accumulate for matrices multiplication
and how this leads to pretty bad results when trying to warp more than two images together.
Of course this was a naive approach and there are more sophisticated ways to do this.