Why do Uncle Sam's eyes and finger follow you? The explanation is actually more scientific than most would think.
The key is that we have Two (2) eyes!
Imagine each of our eyes is a camera. When we capture the image with our left eye, it's a "picture" of our left field of view. When we capture the image with our right eye, it's a "picture" of our right field of view. Now look at the two "pictures", you will find an overlap of image in the middle. But why do we NOT see overlaps when we use both eyes together? Because our brain, just like a panoramic photo editing software, it combines the two "pictures" to form a single panoramic photo by deleting the overlapping region in between. The means what we "see" is not really a direct image from our eyes, but rather a rendered graphics by our CPU/GPU in our brain.
Sometimes our brain's graphics rendering is not so accurate, and can fool us with false perceptions and optical illusions. Depends on which eye is in focus, which eye is dominant, lighting effects in the environment, the depth of view, our mental concentration, biochemical influences, functionality of our sensory receptors, and many other factors, sometimes what we think we see is not actually what our eyes see, or even not able see what our eyes supposed to see.
As for the Uncle Sam illusion, when we turn our heads in front of Uncle Sam, the eye closest to him is now in focus. Our brain process the image from the focus eye as the priority, the eye not in focus as only a supporting image. When we turn our head left, Uncle Sam's pointing / looking at our right eye. When we turn our head right, Uncle Sam's pointing / looking at our left eye. When we looking straight, it's a combined image from both eyes, and he's pointing / looking at both eyes.
That's why don't always trust what your brain tell you what you see. Sometimes we see what we don't see and sometimes we don't see what we see. Seeing is not always believing, believing is not always seeing. Here's a scientific paper, you can see it with your heart's content if my layman's explanation is not enough for you to see.
If you find it hard to understand what you see, the easier way out is to use Google Glass, and just let Google tell us what we see. XD