Module 18 : Pixel manipulations (part 2)

• Topics
• Resources
• In-Class Activities
• Sample Quiz Question

Topics

• Pixel-level manipulations

We are not technically introducing any new content today, but instead want to use our existing abilities to accomplish richer behaviors involving pixel-level manipulations of imagery. In the previous module on this topic, all of our activites involved a straightforward one-for-one replacement of each pixel's color. We used a basic loop structure such as the following to process all pixels:

for (int x=0; x < w; x++) {
  for (int y=0; y < h; y++) {
    color c = orig.get(x, y);
    color replacement = ?????;
    result.set(x, y, replacement);
  }
}

In today's challenges, we broaden the visual affects, either by analyzing larger regions of the original image at a time, or letting that information affect larger regions of the displayed image.

Resources

None (other than those provided with the previous module)

However, to provide some variety, we switch to the following 1024x768 image as our reference:

Available from wikipedia commons at: https://upload.wikimedia.org/wikipedia/commons/thumb/e/ed/SLU_portals.jpg/1024px-SLU_portals.jpg

In-Class Activities

Pointillism
We begin by intentionally going from a photo back to more abstract art. We create a randomized "point" version of the original image as follows. Rather than render the actual image, we begin with a blank canvas and we draw a series of circles (or fattened points) at random (x,y) locations of the canvas. For each such circle, we pick the color so that it matches precisely the color that is at the corresponding pixel location in the original reference image. We can effect the level of realism by varying the number of circles that we draw as well as the diameter of those circles.

Spoiler: my code

Mosaic
In this version, we create a mosaic effect from an original image. Unlike the previous pointillism where we relied on randomly placed circles, we will draw a full grid of evenly spaced squares, yet with the color of each square matching that pixel of the reference image which corresponds to the top-left corner of the square.

While it would be possible to create a static image, the following interactive software allows the user to use the up and down keys to alter the size of the mosaic squares.

Spoiler: my code

Smoothed Mosaic
One concern with the approach with the above mosaic is that it is very suseptible to noise since the entire square is based on a single pixel's color (which might be quite different from its neighboring pixel).

In this new version, we take additional means to smooth the mosaic. While we again tile the canvas in equally sized and spaced squares, for the color of each square, we compute the average of all pixels of the original image that lie within that square. Computing this average requires additional nested loops (and thus more computation time).

Note that some care is needed when averaging near the boundaries of the picture, as some "squares" might go beyond the boundaries of the original image.

Spoiler: my code

Basic Edge Detection
Although it might seem strange, one significant challenge in computer vision is to be able to detect the object within a picture. A significant step toward that end is to be able to effectively detect the edges that define the boundaries of those objects. This process is known as edge detection

We begin by exploring a very simple form of edge detection. For every pixel in the image, we compare its color to the pixel immediate to its right. On a component-by-component basis, we compute the difference between the two pixels (technically, the absolute value of that distance). Notice that if you have large regions having the same color, or nearly the same color, these differences will be zero and thus the difference is visualized as black. When you have a neighboring pixel that is quite different from its neighbor, the difference will be significant and that difference, as a color, will be brighter.

The simple rule of comparing each pixel to its neighbor on the right produces the following image, when plotting those differences as colors:

Spoiler: my code

Finding vertical lines
In the previous project, we chose to compare a pixel to the one on its right. That decision was somewhat arbitrary but it means that we do much better at detecting vertical edges (like flagpoles) because they demonstrate a strong difference between a pixel and that to its right. That unfortunately does poorly at detecting horizontal lines (such as the street's curbs, or the american flag's stripes).

We could choose to instead compare each pixel to the one immediately below it. We implement that and demonstrate the resulting image below. Notice that it does much better at finding the curb and the stripes on the flag (but it does not do well at finding vertical features such as the flagpoles).

Spoiler: my code

More robust edge detection
We can combine the two previous ideas as follows. For each pixel p, compute the component-wise color combination by taking 2*p - below(p) - rightOf(p). Note that if this computation causes a result that is greater than 255 or less than 0, Processing conveninetly renders the value as if it remained in that range.

We show our resulting image below. Notice that it captures horizontal lines such as the curb and U.S. flag, and vertical lines suc as the flag poles. In fact, this also does a good job on the text such as Saint Louis University, and text in the circular seal.

Spoiler: my code

More to come...