screwdriver

Joined: 11th January 2006

All Forum Comments

Topics Created Topics Contributed Forum Comments Photo Comments

All Forum Comments

screwdriver
What your saying is true Smeggy, these are contained within ACR, but what your looking at here is the XMP file which is a look up table for the values of all the tools on opening the file. The preliminary start up values.

If all I wanted was to set the de-noise to 13 then all I would need here is two hexadecimal digits to set any value between 0 and 255 and start and end of data characters. The other hexadecimal digits are where to apply it, comprising the seed data for the algorithm within ACR for it's memory map flagging those pixels which are to have this setting applied.

There is no need to store all, in this case, 12Mp all it needs is the column (0 - 4000) and row (0 - 3000)in the memory map for the start pixel and then the algorithm within ACR will plot the others effected from that point and mark them to receive this data.

All Raw files open with the same tools in the same order, starting with brightness, then contrast and so on, once ACR has got these preliminary values in the XMP file it then moves onto the next tool then refers to this file for the parameters for that, the tool doesn't have to be called it's automatic in the program. There is no need to select the tool on opening as they always work in the same order, so the 'end of data' character in this XMP file makes ACR step through the program and move onto the next tool.

By this method it assembles the data for the XMP layer and creates it which, as I'm sure you know, sits above the DE-mosaiced file and adds the original values to those we have applied, we see the original data though this layer and modified by these settings, similar to how a colour profile works.

Any pixel can have all, none or some of the data applied from all, none or some of the tools within ACR.

It has to be stressed that this is just what happens on opening the Raw file within ACR, any further editing, such as altering the de-noise will alter the value within this XMP file, but it will still only effect those pixels tagged for that tool.

When you open the file into PS it flattens these two layers together and opens them as a PSD file, now, unless you make selections, any adjustments will be global.

The de-noise you refer to is under these conditions and then it's different.

Comment by screwdriver posted on K-3 20 page review out at 20/11/2013 - 11:53

screwdriver
And the purple defringeing gets applied globally too - interesting.

Comment by screwdriver posted on K-3 20 page review out at 20/11/2013 - 01:41

screwdriver
The outline in my image is the data from the camera giving the location in the memory map of what the camera thinks are the effected pixels, we can change those to by using simple addition or subtraction to those values, but they are usually pretty accurate, the value of 13 is what I set as a basic de-noise after looking at different values on lots of different images shot (usually) in our studio, it is a compromise, and it can be subsequently altered by just moving the, in this case, the Luminance noise reduction slider in ACR.

If you open the image in PS and then alter noise reduction, then that will apply globally, but while ever you are altering in ACR it's just these pixels that are effected.

Comment by screwdriver posted on K-3 20 page review out at 20/11/2013 - 00:38

screwdriver
Here is a screen grab of a Hex Editor file looking at the XMP file from a Panasonic G1, the reason I chose this is because I worked on this ACR for Panasonic before I retired.

[Comment Image

Hex example-1 by Chris_Judge, on Flickr]

The highlighted values are what we decided was a good pre de-noise for files from a Panasonic G1, the values surrounded by my rather crude outline are the pixels in the image to apply the de-noise to. These will vary from image to image, at this time we could only crudely identify a threshold value any speckled group of pixels, speaking to my former colleagues it's come a long way since then. The tool being applied is the Noise Reduction slider within ACR.

In this Hex Editor I could alter any value much the same as a text editor. We then save it and import it into the C++ assembler in our developers edition of CS5 to test and from then on ACR will apply these changes when it opens a Panasonic G1 Raw file and create the XMP file from it. The hex file was then sent to Panasonic who forwarded it to Adobe to be included in the next ACR update.

This just one parameter from hundreds which are applied to every Raw file when you open them in ACR, just because you don't know about it doesn't mean it's not happening.

When you edit in ACR and, may be, increase noise reduction, this is the parameter you are altering within the XMP file. If you select an area within ACR and alter, in this case, the noise reduction, then you are creating a new parametric and adding it to the XMP file. This will only apply to that one image, the ones we applied to the ACR will happen on every first conversion from Raw to XMP.

The whole XMP file is only 6KB as it only holds the changes from the de-mosiced image file and is a very small text document. It will swell when opened in PS, LR etc and those changes are applied to a PSD copy of the Raw file.

Everybody happy now?

Comment by screwdriver posted on K-3 20 page review out at 19/11/2013 - 23:31

screwdriver
Smeggypants wrote:
I'm not being a troll at all. When people claim a RAW file contains embedded metadata to tell an image editor which pixels contain noise and by how much, and also claim that metadata in a RAW is linked to content aware tools work then they show a complete lack of understanding of RAW files and indeed physics. Lest anyone believes your misinformation I think it's important to respond with correct information.
Couldn't agree more except that they do and have done for years, Raw files are pretty poor images without them. In the past these codes were only global, recently theses same codes can, and are, used on specific pixels. May be your a little out of your depth?

Smeggypants wrote:


PHP is a just s programming language, which has been popularly adopted for server side processing of dynamic webpages. Forums being
Technobabble like this for instance.

Smeggypants wrote:
Clearly, given you don't really know what PHP is anyway. PHP has dozens of Image Processing functions. http://php.net/manual/en/ref.image.php
Actually I do and none of which can open a Raw file and alter the code in it which is what I am talking about.

Smeggypants wrote:
I have several years experience in PIE (Parametric Image Editing) which is the application of algorithms to the master file created after you de-mosaic the Raw data based on embedded data created by the camera.

So do I. I've never heard someone describe their experience of using Lightroom, ACR, Aperture, etc in that manner though. more technobabble.
Good to know that you are conversant with PIE, the average user doesn't need to unless they want to know what the program is actually doing and how it's doing it. If you think that PHP is good for image editing it would be interesting to know what program you use to open and edit directly the code in a Raw files. It's only 'technobabble' to people that don't understand it.

Smeggypants wrote:
Look I may not win any awards for tact, but when people write nonsense on the internet then it must be challenged.
Certainly true, the same lame excuse for the bad behavior of every Troll I've ever come across.

Smeggypants wrote:
If and when I write nonsense on the Internet then I'm happy for people to challenge me just as directly.
As in the above example for instance?

Still it's good to know that we have our very own self appointed, self opinionated policeman to protect us from technobabble.

Comment by screwdriver posted on K-3 20 page review out at 19/11/2013 - 13:05

screwdriver
Smeggypants wrote:
screwdriver wrote:

The noise reduction in Lightroom is just blurring the image globally, the retention of detail, that's also applied in Lightroom, it can't do because Lightroom, as yet, can't read the hexadecimal codes in the Raw file which tell Lightroom and any other software which pixels are affected.

There is a lot of data in the Raw file that you seem unaware of. There is approximately 3-4Mb of data in the Raw file other than the image data, some of that is the EXIF data, around 100K, around 800Kb is the embedded Jpeg in a DNG file, the rest (several Mb) is data about that specific exposure, it's how content aware works in Photoshop or Lightroom. It's how Lightroom will know which pixels are effected by noise and by how much, then Lightroom can apply noise reduction based on data it can't currently read when it opens the image, when the algorithms are included for the K3, there may be no visible noise, or very little. As yet we don't know, but
Thankyou for your lecture on the extra information embedded in RAW files. I'm already well aware of it thanks. I've done some php programming to extract it and process it in the past.


I love the Internet, you don't half read some technobabble and this one is right up there!

"De-mosaiced" "Content aware" "algorithms" .. impressive


Crikey if Pentax ( or anyone ) could manufacture a camera that could tell which pixels are affected by noise and by how much then it would remove that noise at source and we'd have ISO51200 images as clean as ISO100 images. Wouldn't that be wonderful.

In the real world computer algorithms can't tell which is wanted signal and which is unwanted signal and that's why noise reduction algorithms always degrade the wanted signal as well. The trick in writing the algorithm is to minimise the affect no the wanted signal while reducing the. Retaining detail is done by detecting edges and lessening the amount of NR applied to those edges. ( Sharpening is also done by detecting edges and applying a higher contrast to those edges )

There is NO METADATA embedded in a raw file telling software which pixels are effected by noise. In reality ALL pixels are affected by noise, there's no such thing as noise free electronic circuits so it's a case of "by how much". You'd need at least 24MB of embedded metadata to describe the levels of noise to 8bit accuracy. but all this is irrelevant becuase there's no way of calculating how much of a signal is wanted and how much is not wanted.

If I present you with a number, for example, 250, how are you going to find out how much of that number is unwanted? You can't. And like I say, if you could we'd all have noise free images at any ISO and there'd be no need for NR Software.


There is no embedded metadata that is used by the Content aware tools in Photoshop or Lightroom either. These tools analyse the image itself. They work on all image formats. Lightroom never even had a content aware clone tool until LR5 anyway.






Quote:
you are comparing adjusted K5 images with, as yet, unadjusted K3 images, images that have only been De-mosaiced into a visible image with none of the embedded image data applied.
Basically..... no.

I compared DNG files for both cameras which contain any [actual, not absurd ] embedded metadata.

Will all respect using technobabble to cover up a lack of understanding is always going to come unstuck at some point.
I'm not sure what PHP has to do with images being as that is a Hypertext Preprocessor for HTML files usually so web pages appear the same across different platforms, personally I have several years experience in PIE (Parametric Image Editing) which is the application of algorithms to the master file created after you de-mosaic the Raw data based on embedded data created by the camera.

Currently all you can view in any Raw converter program is the master file, once the algorithms are in place in image editors and PIE gets to work to reinterpret these files it will be different, visibly different, which is all I'm saying.

To be accused of writing technobabble is a little severe, dare I say troll like. But from your other comments to other contributors it seems I am in good company.

Comment by screwdriver posted on K-3 20 page review out at 17/11/2013 - 19:04

screwdriver
gartmore wrote:
Was the first one very underexposed to begin with?
Yes I think that's the main reason for the noise Ken. Probably a combination of me not applying EV compensation in the heat of the moment and the camera being foxed by the bright sky. May be that's why Pentax spent the time and money to increase the accuracy of the exposure on the K3, because they knew there would be (slightly) more noise to deal with. Getting the exposure spot on reduces noise.

Smeggypants wrote:
screwdriver wrote:

The K3 has more Mp crammed onto the same sized sensor, so the pixels are smaller, will it have more noise - yup, but that's not the whole story, it will also have finer resolution, more detail to start with so noise reduction blur has less effect.
Disagree from my findings with high ISO K-3 Files

What findings were those?

Quote:

My final point is that as yet there is no software that optimizes K3 files for noise or anything else as the software hasn't yet caught up except for, may be, the Silkypics software bundled with the camera, so your comparing K5 images which have been optimized for noise reduction with K3 images that, as yet, haven't.
The noise reduction that features in Lightroom works as it should with 24MP or even higher MPs ( I wish it worked with members of parliament )
The noise reduction in Lightroom is just blurring the image globally, the retention of detail, that's also applied in Lightroom, it can't do because Lightroom, as yet, can't read the hexadecimal codes in the Raw file which tell Lightroom and any other software which pixels are affected.

There is a lot of data in the Raw file that you seem unaware of. There is approximately 3-4Mb of data in the Raw file other than the image data, some of that is the EXIF data, around 100K, around 800Kb is the embedded Jpeg in a DNG file, the rest (several Mb) is data about that specific exposure, it's how content aware works in Photoshop or Lightroom. It's how Lightroom will know which pixels are effected by noise and by how much, then Lightroom can apply noise reduction based on data it can't currently read when it opens the image, when the algorithms are included for the K3, there may be no visible noise, or very little. As yet we don't know, but you are comparing adjusted K5 images with, as yet, unadjusted K3 images, images that have only been De-mosaiced into a visible image with none of the embedded image data applied.

Comment by screwdriver posted on K-3 20 page review out at 15/11/2013 - 20:43

screwdriver
Noise is not as simple as "I turn ISO up - therefore I will get more noise", this shot was taken at an ISO of 400 on a K5 with a Sigma 70 - 300mm lens

[Comment Image

Vulcan 02 by Chris_Judge, on Flickr]

and is full of noise.

Same camera, same lens - different day shot at 3200ISO

[Comment Image

Botanical_Gdns_0199 by Chris_Judge, on Flickr]

No trace of noise.

The K3 has more Mp crammed onto the same sized sensor, so the pixels are smaller, will it have more noise - yup, but that's not the whole story, it will also have finer resolution, more detail to start with so noise reduction blur has less effect.

My final point is that as yet there is no software that optimizes K3 files for noise or anything else as the software hasn't yet caught up except for, may be, the Silkypics software bundled with the camera, so your comparing K5 images which have been optimized for noise reduction with K3 images that, as yet, haven't.

Comment by screwdriver posted on K-3 20 page review out at 15/11/2013 - 10:24

screwdriver
Adobe Camera Raw V8.2 doesn't, as yet, fully support the K3 so MAWB may not yet be fully implemented.

http://helpx.adobe.com/creative-suite/kb/camera-raw-plug-supported-cameras.html#

Addition

Currently the Pentax K3 isn't on the list for ACR V8.3 either, though it may be in the final release due 31st December 2013.

http://labs.adobe.com/technologies/cameraraw8-3-cc/?tabID=details#tabTop

It also talks about "Separate Auto Temperature and Auto Tint feature. Shift-double-click to invoke "auto temperature" and "auto tint" separately".

Comment by screwdriver posted on MAWB on the K3 at 11/11/2013 - 10:01

screwdriver
This is something that Ricoh has brought to the table, it looks like a very useful addition, now we can have different WB in different parts of the image.

Comment by screwdriver posted on MAWB on the K3 at 08/11/2013 - 15:33

Proudly supporting Pentax User

Samsung Logo Asahi Pentax Logo