2010-03-13

Generating false color images on the Nexus One using only grayscale pixels

EDIT: I want to say up front, don't go out and sell your phones, people! And don't change your mind about purchasing a Nexus One without looking at the screen yourself. The N1 display is beautiful and vivid with dark blacks and incredible photo reproduction, and much better than the iPhone screen for text reproduction. Only when compared with an extremely high-res LCD screen like on the Droid are the text fuzziness comments even justified -- and that's a high standard to hold the screen to.  My only comment was that both screens were specced at almost the same resolution so it would be nice if the N1's screen looked just as sharp.
I just published an article on Ars Technica showing how the effective display resolution on the Nexus One is not 480x800 as repeatedly claimed by both Google and HTC -- at least not the way screen resolutions are normally calculated, with one pixel equal to one RGB triplet.  If you are interested in the arguments about display resolution on the Nexus One, please see my other post on that topic.

In the Ars Technica article, I also showed a few nefarious test patterns, formed only of pure grayscale pixels, that leverage color fringing on the N1 display to appear as color images. The color disappears as soon as you scale the image in or out from 100% by a few percent, and the images appear gray on a standard LCD screen such as on the Motorola Droid or on your laptop.  Lots of examples are given below.  [Since there is a lot of public discussion going on about the article, I'm collecting some of the most insightful feedback here.]
IMPORTANT NOTE: The greyscale stippled images in the Ars article cannot induce color on the N1 display because Ars decided to scale them down to 75% and not provide links to the full-res versions -- you must download the full-resolution example images at the links below if you want to try this on your own Nexus One!

 
Left: Closeup (200% zoom) of a pure grayscale stipple image that appears colored on a Nexus One's PenTile display when displayed at 100%.  Right: Perceived color.
Since a lot of people probably came here for the download links, I'll give those up front. Read on below for more details about what is going on.
DOWNLOAD LINKS: (N.B. here's a short-link to this blog post, so you can easily type it into the Nexus One browser to get to these links: http://goo.gl/zSzT )
  • Tap to download in N1 browser, then view in Gallery:
    Test patterns:
    1 | 2 | 3 | 4 | 5
    Example false-color stippled images:
    1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17
  • You can also download all images as a single zipfile if you want to unzip them all in a single step to your Nexus One's SD card.
  • Source code for the algorithm that generated the images is also available under the GPLv2 license, please drop me a note if you do anything interesting with it.  The algorithm is described below.

Note: the color fringing shown here is specific to the configuration of the Nexus One display, and may not work the same way or at all on other PenTile AMOLED configurations, as described in my other post.


Downloading and Viewing the Example Images on the Nexus One

Download and view the individual sample images above, or download all images as using the single zipfile link, unzip them to your N1 SD card, and open the images in the Gallery application.  Then double-tap or pinch-zoom to zoom in and out of 100% to see the colors magically appear at 100% and disappear at other scale values. The grayscale stipple images images must be displayed at exactly 100% (1-to-1) zoom on the Nexus One screen for the color artifacts to be observed.  On the Android browser, you may not be able to get the colors to appear, firstly because the images below are shown at 50% unless you click on them, but secondly when you do, the browser makes it particularly hard to view images at exactly 100%. In each of the examples shown here, the grayscale stipple image is shown on the left hand side at about 50% zoom, the way it appears on the N1 screen is shown on the right hand side for reference.  (Click to view full-sized versions in a desktop browser.)


Example Image

 
Stipple image (left) and how it appears on the N1 screen (right).  You can force the PenTile display to show color fringes in pretty much every color of the rainbow... albeit sometimes a bit washed out. How can Nouvoyance claim this display is exactly equivalent to a standard RGB-striped LCD panel in its color resolution?
This image on the left would show as grayscale on a standard LCD panel, for example the screen on the Motorola Droid (and probably the screen you are viewing this on right now), but in full rainbow color on the Nexus One display when viewed at 100%.  The Droid's screen has almost the exact same resolution as is claimed for the Nexus One (480x854 on the Droid vs. 480x800 on the N1).  If the N1 screen really were the resolution that is claimed, the image would show as a grayscale image on the N1 just like on the Droid.  Clearly the N1's screen is not capable of the same physical resolution as the Droid, or there would be no color artifacts.

How it works

I created the following two test images where a 3x3 stipple image is stretched to 4x4, with the phase (stipple pattern offset) continuously varies with angle about the center and the intensity of the stipple pattern varies with radius:

 
Test images
I then took a photo of the N1 screen (slightly blurred to remove moiré patterns), manually corrected the color curves so that the image on my desktop LCD screen matched what I saw on my N1 screen, and de-warped back to 480x800, producing the following two reference images.

  
Reference images (photos of the test images viewed at 100% on the N1 screen)
Then given an input image (e.g. the Mona Lisa), my algorithm finds the pixel in one or other reference image that most closely matches the the color of each pixel in the input image, and reads off phase and intensity.  These phase and intensity values are used to output a single stipple pixel in the output image.  The algorithm also smooths the resulting image by performing local averaging of phase values before outputting the final stipple pattern.
The reference images therefore give us a palette to work with.  The resulting colors are a bit washed out and dull (they each cover less than 1% of the total 16-bit colorspace), but as the image of the rainbow above demonstrates, every color can be generated this way, albeit over a limited saturation and/or lightness range. The reference image step using manual color adjustment is a real hack. :-)  The color mapping could be greatly improved using real color calibration hardware.  This would enable most color gradients shown in the test images here to appear completely smooth.

Note: The reference images above are real photos, but all other images in this article that show how a stipple image is perceived on an N1 screen (e.g. the righthand of the two rainbow images above) are actually a recoloring of the input image to use the closest color in the reference image palette for each pixel.

Different Ways of Stippling

Once those reference images have been obtained, for every input image, my algorithm outputs not only the stipple image that best reconstructs the color display using only grayscale pixels, but also outputs an approximation for how the image will look to a human observer on the N1 screen.  For example, given the following input image,


the algorithm outputs the following grayscale stipple image and an approximation of how the image will be perceived.  (The perceived colors are less saturated than the original.)

  

The above stippled version of the Mona Lisa image uses a pattern of a 1-pixel-wide diagonal black line set on a 3x3 white background (the first of the two test images from which the reference images are generated), stretched to a ratio of 4/3, and then offset appropriately at each pixel position to generate the correct color at that pixel location.  The following image inverts the colors of the stipple pattern, i.e. uses a 1-pixel-wide diagonal white line set on a 3x3 diagonal black background (the second of the two test images, which generates a different palette of observable colors in its corresponding reference image).  The colors in this second rendering of the Mona Lisa really jump out on the N1 screen, more so than the righthand image would indicate.

 

Lots more examples

Again, stippled image is on the left (click to view full-size), an approximation of how the image will look on the screen is on the right.

 
My algorithm takes a very rough cut at finding the closest color available, it could be substantially improved, probably to the point of making the above sunset image appear almost completely smooth.  I doubt I'll actually do that sort of fine-tuning, this algorithm was just developed for demonstration purposes, and it's good enough for this as it stands.  Here's another image that shows a color gradient that could be made much smoother.
 
Starry Night (stippled version) looks great on the PenTile display!  The blue and yellow colors in the sky look electric on the N1 screen, and the dark greens are really rich and dark, because in this image I use both of the two different stipple patterns used above (for the two different Mona Lisa images), and switch between them when it reduces the color error for a given pixel.
 
Not only the density of the stipple pattern but the overall average intensity of pixels in the stipple pattern can be varied to give colors with a range of different luminance values.  Note that changing the average luminance of a stipple pattern can change the apparent chrominance (apparent color), so the color selection algorithm has to take this into account.
 
Another image showing all the colors of the rainbow in one scene by exploiting color fringe artifacts of the PenTile matrix display.
 
An image of Yellowstone using the same white-on-black stipple pattern used for the first Mona Lisa image
 
The same image using the black-on-white stipple image used for the second Mona Lisa image (the colors are actually more significantly different on the N1 screen between these two examples than they appear here)
 
The color gradations in the sky could be smoother here again, it would just take a bit more time polishing the mapping from stipple phase offset to perceived color.
 
 
The first of these two examples uses the "white-on-black" stipple pattern, the second example switches dynamically from white-on-black to black-on-white, as with the Starry Night image above (again the difference in color depth can only be seen on the N1 screen).
 
 
 
 There are lots of different ways to dither the same image all to similar effect.

16 comments:

  1. I came some how code my way out of brown paper bag but this is way over my head. Excellent work. Cheers.

    ReplyDelete
  2. Typo correction: I can some how...

    ReplyDelete
  3. there is the human eye reception factor, I mean your algorithm to make the pics gray might just "hide" the colors so the human eyes cannot see it, .. but when u transfer it to N1, the colors factor gets amplified (due to the AMOLED and other factors) so, it appears more to the eye as colored.
    Am not into image processing though, just my 2 cents.
    nice article nevertheless
    cheers

    ReplyDelete
  4. Arkan Hadi,
    While this may be true, he wouldn't be able to photograph the phenomenon, and it would happen on more displays than just this one.

    ReplyDelete
  5. You have the determination to hack! I adore you.

    ReplyDelete
  6. What about full blue or red images? The resolution would be half of the specification? Is it perceivable?

    ReplyDelete
  7. Excellent job, mortal! You have embraced the true meaning of techno-worship.

    ReplyDelete
  8. and one more thing, ... the pixels count are 480*800, that is not something you can change, its in the hardware, ... however, ... each pixel is subdivided into 2 sub-pixels, instead of 3 sub-pixels as the tradition RGB rendering, so its just different way of rendering, if you have 2 sub-pixels in each pixel, that does NOT mean that you have less overall pixels. you have the same amount of pixels, but each pixel have 2 sub-pixels, not 3

    ReplyDelete
  9. I reckon the Samsung Galaxy S is probably going to suffer from this as well no?

    ReplyDelete
  10. Fabio: good question, not sure, it would be interesting to view pure red or blue surfaces and see if they look "holey"...

    Arkan: obviously -- that's the whole point of the post.

    Mori: No idea :/ However almost all high-end Android devices in the future will have AMOLED screens, and for now almost all AMOLED screens will be PenTile, so it's likely.

    ReplyDelete
  11. Hi. I couldn't figure out a way to PM you via Ars. Your statement that "The eye's spatial resolution in each of the three primary colors is approximately uniform" is not correct either. It's pretty easy to demonstrate that your eye has better resolution in green and worse in blue.

    http://nfggames.com/games/ntsc/visual.shtm

    This aspect of vision is exploited in NTSC, old console games (hooked up to a TV), and the Bayer filter ubiquitous in digital cameras. All of these devote more bandwidth to transmitting/capturing green than they do blue because your eyes simply can't tell you've degraded the blue channel much more badly. For scientific applications where you want to capture the same resolution in each color channel, you typically have to rotate a color filter in front of an unfiltered sensor (NASA does this). You can't just grab consumer-grade video hardware off the shelf and expect it to yield scientifically usable color data, at least not without significant downsampling.

    Another related bit of trivia is that the sensitivity of your cones does not fall at the red, green, and blue portions of the spectrum as you'd expect. The green and blue cones approximately sense green and blue. But your "red" cones are actually sensitive to yellow/orange. The strong perception we have of red is manufactured in our brains via a differential analysis of how much blue and green our eyes see relative to yellow/orange.

    http://en.wikipedia.org/wiki/Color_vision#Physiology_of_color_perception

    ReplyDelete
  12. Heya,

    Just curious - what program did you use to generate the stipple image, and to work with subsequent images?

    Great article, and very good job!

    Oz

    ReplyDelete
  13. Alexander: I wrote the program myself, the source code is linked from the blog post above.

    ReplyDelete
  14. Luke, I like your work, but it shows N1 display has problem with artifacts (under very special circumstances), not loss of resolution.
    I am not saying there is no loss, but I don't see any proof here.

    ReplyDelete
  15. Vlasac: all I can suggest is that you try looking at a Droid side-by-side with a Nexus One, and you will see that the Droid *looks* significantly higher-res than the Nexus One, particularly when looking at small text. Once you have seen it yourself, if you feel that my attempts to explain this phenomenon are inadequate, please feel free to suggest other reasons why this may be so.

    ReplyDelete
  16. Thanks for this great post. Some months ago i bought an Nexus One with an AMOLED screen and i was also very disturbed by the fuzzy look. Then i found your arstechnica article, where you hit the right point.

    I also got my hands on an Nexus One with SLCD and made a little comparison between both devices.

    My article is not scientific at all, but only from an enduser perspective.

    http://smartphoneblogging.com/2010/09/nexus-one-slcd-vs-nexus-one-amoled-showdown-which-screen-technology-is-better/

    I can strongly agree to all the points you mention. The effective resolution IS less than WVGA on an (S)LCD screen. And while the fact remains that the number of pixels is the same for both screens, the subpixel are the reason why the AMOLED screen is not as good in reproducing small text. I was also very disappointed because i had a Motorola Milestone/Droid prior to the NExus One "Superphone", and the first impression of the Nexus One display was catastrophic. How could HTC let this through their quality control?

    What i find funny is that there are really some people out there which claim that AMOLED screens have BETTER small text reproduction....i just don't get it.

    ReplyDelete