Welcome back. There are two main reasons we are considering color. The first one is that color is a powerful, discreet tool that can be used for various tasks, such as segmentation, object detection, tracking, and identification. The second one is that humans can distinguish thousands of color shades of intensities as compared to about only two dozen shades of gray. In this segment, we first discussed the characteristics of light based on the absorption characteristics of light, we define the primary colors of red, green, blue and the secondary colors of cyan, magenta, and yellow. We also discuss the primary colors of pigments which are cyan, magenta and yellow and the secondary colors of pigments which are RGB. We define the color distinguish characteristics of brightness, hue and saturation. After that we discuss color modules which allow us to specify colors in some standard and generally accepted way. We describe the most commonly encountered models of RGB, CMY, HSI, and YCbCr. We also briefly address the independent versus joint color processing approaches. For the various aspects of our discussion, we also show some experimental examples. So let us proceed this last exciting and colorful segment of week four. The use of color in image and video processing is motivated by two primary reasons. The first one is that color is a powerful descriptor which can be used for various tasks such as segmentation, object detection, tracking, and identification. We will see examples of these as we move along. The second reason is that humans can distinguish thousands of color shades and intensities As compared to about only two dozen shades of gray or gray scale images. For these two reasons, color is an important attribute in image and video processing, and we just want to cover some of the basic properties and basic ways to present color. In this segment. Isaac Newton, in the mid-17th century, discovered that when a beam of light passes through a glass prism the light that comes out is not white, but consists instead of a continuous spectrum of colors ranging from violet At the one end to red at the other end. So I showed in this figure the color spectrum may be divided into six broad regions. Violet, Blue, green, yellow, orange and red. No color in this spectrum ends abruptly, but instead each color blends smoothly into the next one. The color we perceive of an object has to do with the nature of the light reflected by the object. If the object, for example, appears green, then the object absorbs the energy in all regions of the visible part of the electromagnetic spectrum, except in this range, 495 to 570 nanometers, in which the light is reflected by the object. Light can be characterized as achromatic, without color, in which case the only attribute of achromatic light is intensity, or the amount of light or it can be characterized as chromatic, with color. In this case, the characteristics are the radiance, which is the total amount of energy that flows from the light source, and it's measured in watts. It's also characterized by luminance. Which is the perceived amount of energy and the unit is lumens. So for example if energy is emitted in the far infrared region, it might have a lot of energy, a lot of radiance, but it is hardly perceived by the viewer and therefore its luminance is zero. And it's also characterized by brightness. This is hard to measure and it includes the achromatic notion of intensity and by and large brightness is used to describe color sensation. Cones in the eye are responsible for color vision. There are about six to seven million cones. It has been found experimentally that the cones can be divided into three principle sensing categories which correspond roughly to the RGB Categories. So 65% of the cones are sensitive to red. 33% of the cones are sensitive to the green light and the remaining 2% are sensitive to this blue category of light although the blue cones are the most sensitive ones. So this figure shows the average experimental curves that deal with the absorption of light by the RGB cones in the eye. Due to these absorption characteristics of the human eye, Carlos has seen, as variable combinations of the so-called primary colors, the RGB. For the purpose of standardization, this body C I E, which stands for Commission International de l'Eclairage Designated in 1971, the three primary colors. For the blue, at 435.8 nanometers. For the green 546.1 nanometers and for the red, 700 nanometers. These standards were set before the experimental that are shown here became available and this is why there's this discrepancy. We see here that the absorption of the blue peaks at 445, while the designation by CIE was at 455.8, and so on, and so forth. So RGB of the primary colors, but if we think of this standardized colors at these fixed frequencies either this one, or this one, it doesn't matter. This fixed frequencies cannot generate all visible colors. The interpretation of primary colors is correct, if we allow the wavelength also to vary, as shown by the colors here. The three primary colors we just mentioned RGB, can be added to produce the secondary colors, which are the cyan, magenta, and yellow. If we mix the three primary colors we'll obtain white. We also obtain white if we mix a secondary color with its opposite primary color. So, magenta is the secondary color and the opposite primary color is green. So, this is again the intersection between magenta and green. It's the white color. So this is the additive model of the colors, and it's used for example in color television when we use CFT or a flat panel. This is an example where we have separate grams for RGB and it's their combination that will allow the generation of all possible additional colors. The primary color of pigment is defined as one that subtracts or absorbs the primary color of light. And reflector transmits the other two. So for example cyan equals one minus red, we assume here that all the values are normalized in the zero to one range. So if we have a surface that is coated by pure cyan then this surface reflects light that does not contain red. So red is absorbed. So the primary colors of pigments are the magenta, cyan, yellow and the secondary colors are red, green and blue. A proper combination of these three pigment primaries generates black. And black is also generated if I combine a secondary such as green with it's opposite primary, which is magenta. And this is referred also as the subtractive model of color. Here are the distinguishing characteristics of color. The first one is brightness, which we also showed a couple of slides earlier. As we said there, it's hard to measure but is used to describe color sensation. The second characteristic is hue. It indicates the dominant wavelength in this mixture of light waves. And the third one is saturation. It tells us about the relative purity of the color or the amount of white light that is mixed with a hue. The pure spectrum colors are fully saturated. Colors such as pink which is a combination of red and white is less saturated and in general the degree of saturation is inversely proportional to the amount of white light that is added so if we add a little bit of white light, then it's quite saturated. You add a lot of light then the glare situation is much less. The amount of RGB to form a particular color are called the tristimulus values and they're denoted by capital X, Y, Z. A color is then specified by its trichromatic coefficients, denoted by lowercase x, y, z. You can obtain the trichromatic coefficients according to these relationships. So, little x is the proportion of big X with respect to the sum of the tristimulus values. Clearly if we add up the trichromatical efficients they add up to one. So given a wavelength of light in the visible spectrum, the tri-stimulus values that are needed to produce the color that corresponds to this wavelength can be obtained directly either from curves or from tables that have been compiled from extensive experimental results. Another approach is to use the CIE chromaticity diagram shown here, which shows the color composition as a function of X in the horizontal axis. Which represents red and y in the vertical axis which represents green. Clearly the value of z which represents blue can be found from this relationship that results from the relationship down here. The point marked green here is about 62% green, 25% red, and the remaining 13% blue. The value spectrum colors from the violet down here at about 380 nanometers to red at 780 nanometers are located on the boundary of this tongue shaped diagram here. These are the pure colors in the spectrum, as we discussed earlier, and any point not on the boundary but within the diagram represents a mixture of spectral colors. A point of equal energy here corresponds to equal fractions of the three primary colors that presents the CIE standard for white light. Any point on the boundary of this chromatic chart is fully saturated as the point leaves the boundary and approaches this point of equal energy. More white light is added to this point and therefore it becomes less saturated. And the saturation of this point of equal energy is equal to 0. The purpose of the color model, which is also called the color space or the color system, is to facilitate the specification of colors in some standard and generally accepted way. So a color model specifies a coordinate system and the subspace within that system which colors are presented by single point. The living models in image and video processing are the RGB, red, green, blue model. Which is suitable to describe color monitors and video cameras. The cyan, magenta, yellow, and cyan, magenta, yellow, and black, K stands for black, which is suitable model for color printing. The HSI, hue saturation intensity model which corresponds closely with the way humans describe and interpret color, and also the YUV and YCbCr model. Which is used primarily for be the compression applications. The RGB model is based on the Cartesian coordinates system. The RGB primary colors are these corners, red, green and blue. The secondary colors are these other corners, cyan, magenta, yellow, white is at this corner, and black at the origin. The points of equal RGB extends from the black point to the white point, so these are the gray scale values of equal RGB. Here is the 24-bit color cube. The surface of it two to the 24 is equal to 16,777,216 different colors. As already discussed, cyan, magenta, yellow are the secondary colors of light or the primary colors of pigments. So when for example, a surface is coded with cyan pigments and is illuminated with white light, no red light is reflected from the surface since cyan Is one minus red. In this equation, we assume that all of the values are normalized in the zero to one range, right? So cyan equals one minus red. Now, since in principle, equal amounts of C, M, Y should produce black, but in practice, they produce a kind of muddy looking black. So because of that, a fourth color has been added, the black color denoted by K, this gives rise to the C, M, Y, K color model. The HSI model, the couple's intensity component from the color carrying information. It's an ideal tool for developing image processing algorithms based on color descriptions that are natural and intuitive to humans. They're nonlinear expressions, which allow us to go from RGB to HSI and back that can be found in most text or image processing, such as the one by Gonzalez and Woods. Instead of reproducing the equations here, let's try to get a feel about the HSI space. In this figure, we see the primary and secondary colors. We've seen this figure actually multiple times already. While in this figure we see the hue, the H component of the first one. The gray values in this figure correspond to angles. So red, for example, corresponds to zero degrees angle and, therefore, it's black. So, here's the red and it corresponds, too, and then go 0 degrees and it's in order by black. In this figure, we see the saturation component of the original image here. Clearly the primary colors are fully saturated, we've shown here in white. All the values here are marked in the 0 to 55 range. And finally, here is the I component, the intensity values, which are actually these values one-third of the sum of the RG and B values. So with a model like this, one can manipulate directly and independently the H, or S, or I component. And, this way one can change the individual colors. There's a family of color models where the luminance component Y is separated from the chrominance components. YUV is used by the PAL, NTSC, and SECAM color video standards. So Y Is again the aluminum's component. Related to that is the YIQ. Y is the aluminum, I the in face, and Q the quadrature component coming from a modulation scheme. IQ are linearly related to UV. And then YCbCr Y, again, the luminance, and these are the differential chrominance components. This is a recommendation out of ITU, BT.601, and it's a scaled and offset version of YUV. There are linear relationships relating these three color models. And each one of them to the RGB color model. Here are the equations for the YCbCr BT.601 recommendation. We see that the luminance component is 60% due to the green channel, 30% to the red and 10% to the B. And again, these are the differential blue and red channels. Clearly, since there's a linear relationship between YCbCr and RGB, I can solve for RGB in terms of YCbCr. These are the formula used by JPEG conversion. And in general, the RGB values for an 8-bit image are in this range and the prominence and luminance components are also in this range. So due to the fact, again, that luminance is separated from chrominance very often when processing that is represented by this model, we might carry the processing, with respect to the Y component while not processing the chrominance components and just adding them to the processed Y component. For example, when we perform motion estimation, we utilize the Y component only, the luminance component and the motion vectors that are found are also utilized to motion compensate the chrominance components. We consider here, a color image and show it's the composition or representation of utilizing the different color spaces we just covered. So this is sunrise over Lake Michigan here in the cold winter day, this is just south of the north west of compose. We see here the RGB components of the color image. There's a lot of red here where the sun is rising, so you see high intensity values. The green channel comes very close to representing the intensity image or the gray scale value of the colored image. We see here the cyan, magenta, yellow components of the original color image. These components are the negatives of the RGB components. So for example, cyan is equal to one minus red, so where the sun is rising, we have high red values and we have very low grows to zero cyan values. We see here the HSI components of the original color image, so this is the Intensity is the grey scale version, you might say, of the original color image, while the hue and saturation are the color elements or the color components of the original image. So we have this separation between luminance, intensity and chrominance. And here we show the luminance Y. And the differential color components, Cb and Cr. So this is, again, the grayscale image. It relates to the I image of the previous slide, and also to the green of the RGB decomposition. This is a different way to separate luminance and chrominance on the HSI representation system. The advantage, you might say, of YCBCR is that there's a liner transformation between RGB and YCBCR, while the transformation from HSI to RGB is a non-linear one. In person color images and videos, the first step is to choose the appropriate color space based on the analysis and the discussion in this segment. The second decision is whether to process each channel in this representation independently or jointly. Most of the work in the literature is performed by processing each channel independently and they call channel the RGB or YUV. For example, plains or representation of the color image. And therefore, there is no distinction, you might say, between a grayscale image processing algorithm and the color image processing algorithm, in the sense again, that when we deal with color, we'll apply the same algorithm we used for the grayscale image, three times independently to each of the three channels. Alternatively, one can utilize a multi-channel processing. So in many cases, gains have to be had by capitalizing of the correlation and complementary information conveyed by the various color components of the various channels. So for example, in image compression, the standard approach is to find the motion by working with the Y component of the video in a YCBCR decomposition. Then the same motion is applied to the chrominance component. Work has been done and showed that one can formulate a motion estimation problem by working with a three by one vector, where the RGB, for example, components form this three by one vector. And by and large, more accurate motion estimation can be found this way. Another example, and we'll touch a little bit of that when we talk about restoration, when we do multi-channel processing again. We tried to capitalize on the between channel correlation and the for example, a cross channel smoothness as well as within channel smoothness. We show here a simple example where RGB and HSI filtering is compared. So this is a bled version of the original image we showed earlier. A five by five flat filter was used embedding this image. And each of the channels, the RGB channels was filtered separately. Here is the image you obtain when we consider the HSI representation and only the I component. The intensity component was filtered by this same filter, while the H and S component were unaltered and just were combined with the I component. These two images look indistinguishable and we try to quantify that by finding finding the difference between the two, and this is the area of gray is equal to zero, and it's almost identical, the two images. So this is a simple example clearly, which demonstrates that if this filter or similar operation is what we are interested in, then it would make sense to utilize the HSI representation and filter one, the I channel instead of three channels, the RGB channels. So, we're reached the end of week four. During this week, we learned some of the basic techniques for estimating the motion for each pixel or a group of pixels in a video frame. And also ways to represent a color image in video. Some of the motion estimation techniques are more intuitive and easier to explain than others. So for example, the block matching or region matching technique, which is widely used, if not exclusively for video compression, might be the first idea that comes to everybody's mind, if you spend a few minutes thinking about the problem. The fine implementation details can always be worked out later. The face correlation or registration approaches are rather intuitively clear, and they're based on a simple property of the Fourier Transform that we covered in week three. The special temporal gradient method, on the other hand, seem to stem from mathematical manipulations, taking the theory of Taylor series approximations. And they're not intuitively straightforward, at least at first. At the end, however, similar considerations with the block matching techniques are taking place. That is, the modeling assumption that a region of pixels is experiencing the same motion. Such techniques are also used widely, especially when a dense motion field is required. We also talked about primary and secondary colors of light and pigments. And we also discussed various models to represent color. We described the most commonly encountered ones that is, RGB, CMY, HSI, and YUV, YCBCR. We know now, that there are different choices both in representing as well as in processing color images and videos. The material of this week is by and large not as demanding as material of the weeks two and three. If, however, you do not feel 100% confident about everything we've covered this week, do not worry. We can still use the final motions to measure an algorithm we derived in a rather straightforward manner, or perform a transformation from one color space to another. Clearly, we'll be making use of motion estimation in various parts of the course, especially when we cover video compression. Similarly, most data process nowadays are indeed in color. Next week, we'll cover an easy to explain and understand and fun to play with topic in image and video processing, that of enhancement. So, see you all next week.
