Sony PMWF3
First Tests of Sony PMW-F3 in Low Light and Chroma Noise
Evaluation Seminar on March 10th at 10:00 am Went into greater depth (video comming soon)
We just received our first production version of the Sony PMW-F3 and after a few preliminary client demonstrations we had a few hours with the camera to run some preliminary tests to evaluate the low light performance and examine the chroma noise and its characteristics. Our first camera was leaving the next day to go to Panavison in Burbank for some various lens test (including the new Sony primes which arrived after this first test) so we had a very tight schedule . It will be back in one week and we will be able to run more tests.
Since Sony has stated a signal to noise ratio of 63dB which is unheard of in the HD world we were anxious to see if indeed this high specification could be seen in the real world by the unaided eye. Were were also interested to see if the high gain of 18dB was indeed usable for actual production footage.
F3 Lowlight Room Setup
The stated sensitivity of the PMW-F3 is F11 at 2,000 lux which is a broadcast type of rating which is at 60i (59.94) at 1/60 sec. and not at 24p at 180 degree shutter. All of our tests were performed at full 1920 resolution @ 24p with the shutter set at 180 degrees. For comparison we also tested the new Panasonic AG-AF100 and the Canon 5D since both of these cameras have large format sensors and have attracted a lot of interest as of late. Using the "syncro" shutter setting in the Panasonic we were able to set it at exactly 180 degrees (1/48 sec) and the Canon was set at 1/50 of a second which equates to about 181 degrees. The only light in the room was a Arri 150 aimed at the rear wall and ceiling pointed away from the objects. The measured light reflecting off the fruit was a very low 4 lux. To the eye the scene was very dark as can be seen in the photo above. Keep in mind a typical office has between 320 and 500 lux and a typical home living room is about 50 lux. The AG-AF100 had a maximum gain of 12 dB which Panasonic menus show to be an ISO 0f 1,000 and shows a brightness level of 20 IRE while the Canon at an ISO of 6400 and the Sony PMW-F3 was at 18dB which both reached an IRE rating of 80-85.
Full Scene Lowlight F3 @ 18dB
Disclaimers
Although this first test can give an insight to the actual performance of the PMW-F3 is is not possible to convey across the web the actual comparisons and look to the eye of the PMW-F3 and the other cameras tested. Still images do not show the motion and pattern and color of the noise and motion footage compressed for the web does not convey the real nature of the noise because of the compression and various re-encoding. Since we are also a dealer for each of the cameras tested every effort was made to ensure that each camera was set up properly and recording were made at each cameras best settings. The Sony PMW-F3 was at 35 Mps, the Canon at approximately 40-45 Mps and the Panasonic at 24 Mps using their native internal codecs. The images were taken directly off the media as .jpg and cropped as can be seen below to allow our attention to be on chroma noise since that is the most objectionable.
This cropped portion of the off-media of the PMW-F3 shows a closeup of the color chart, fruit and bottle which allows us to better examine the type and nature of the noise and processing of the internal codec. If you look carefully into the brown, purple and blue chips you will notice the absence of chroma phase modulation errors which traditionally begin to show up at gain settings of 6 or 12dB. Phase modulation errors which you can easily observe in the next two photos are errors in color phase or hue that are colors within a given that are not actually there. In other words if you look at the brown, purple or blue of the next photo you will also see specs of red, yellow and other colors which are not part of the original scene. When you see these errors in motion they are very noticable and most distracting to the intent of the scene. This is most easily scene on a Sony professional monitor in its "blue gun" position because Sony translates the blue gun into a black and white image which allows one ( as we did) to really see all of the chroma and luminance noise in the image. ( we'll try and post off-monitor shots in an upcoming test)
The other type of Chroma noise is amplitude errors which manifest themselves as different intensities of the same color. In other words if you look into the blue of the next photo you will see light and dark specs of blue within the blue chip, these are amplitude errors as they do not exists in the original scene. When you look at the PMW-F3 image above you will see very little amplitude errors and almost no phase errors. What you can see is the effects of jpg compression which was necessary for the web. The slight hint of noise in 18dB with the PMW-F3 was almost like a texture or sense of image rather than electronic chroma errors that we are all used to at 18db.
The Canon 5D was a surprise is some respects and to be expected in others. When we tried a ISO of 3200 (theoretically the F3 at 18dB) is was no where need the brightness of 80 IRE that we had seen on the Sony. When we put the Canon at ISO 6400 it did reach approximately 85 IRE and the setting below did not reach 80 so we left it at 6400. We would not expect the image quality of a under $5,000 camera to compete against a $16,000 camera but since the Canon has been widely used in high profile projects and is enjoying widespread use we wanted to see where it fell relative to chroma noise and sensitivity. As you can see to some degree from this photo the phase and amplitude chroma noise would prohibit normal use of this camera at these levels. Still, for certain documentation applications the Canon was able to make bright images at these levels of just 4 lux.
The highest gain setting for the Panasonic AG-AF100 was 12 dB or an ISO of 1000. As you can see from this portion of the off-media image it could not approach the 80 IRE for the white portions of the image. This however does tend to hide the chroma phase and amplitude errors as they are quite difficult to see in the darker image. If you look closely you will see they are there and if you were to electronically brighten the image the errors would be more noticeable. To Panasonic's credit they have chosen to limit the anount of electronic gain to keep these chroma erros from becoming too distractive.
Conclusion
Sony's claim of a high 63dB signal to noise appears to be founded in actual on-screen lack of noise and a new level of low light capability in the real word. Considering these tests were all done with the internal 4:2:0 codecs at very compressed rates, we did not expect to see such dramatic (you will want to see in person) differences in the amount and type of noise. The Sony PMW-F3 however should be significantly better as its cost ($13,300) would dictate. We are seeing new levels of performance that have yet to be achieved in the under $50,000 range and maybe even in much higher priced systems which we hope to begin testing in the coming weeks. Although this test was not designed to measure or compare color sharpness, or resolution you can also get a feel for the differences the PMW-F3 has. We will also compare the outputs recorded at 4:2:2 (not possible with the Canon) and eventually at 4:4:4 when the upgrade on the Sony becomes available. The PMW-F3 is certainly a new benchmark in performance for super 35mm digital image quality and we look forward to doing further tests and comparisons.
