Raw-files as original

[nickelas]

I would like to store raw-files as the original and have CPG create three different sizes of jpgs.

Similar but very old topics:
http://forum.coppermine-gallery.net/index.php/topic,18148.0.html
http://forum.coppermine-gallery.net/index.php/topic,12063.0.html

http://uppsalafoto.se/galleri/

[phill104]

Sorry, but that would require a RAW processor to be added. Currently afaik neither Imagemagik nor GD are capable of this. It would also swallow a lot of server resources as well as bandwidth uploading them all. If you really want it you will have to create a plugin to do so. That may be difficult as I am not aware of any scripts that will do it.

[nickelas]

Thanks,
Yes I understand it would take alot of resources. Just to clarify I didn't mean a raw-viewer, just being able to create the previews from the best source.
Filesize between 5 - 15 mb is big but not impossible, is it?
I took a quick look at imagemagick and as I understood it, it does raw-conversion. Anyways, this is not my league - just thought it would be a very useful feature if possible to implement

[phill104]

I didn't realise that IM now supported it. In theory then it would be possible for CPG to do it but I really do not think it is a good idea. The amount of memory the server would consume processing a RAW file would be prohibitive to say the least. The current average for raw files is about 9mb. To open the image it will be roughly 25mb. To manipulate that will require a lot more memory.

Give it a go on your server and see. Setup imagemagik and use the filetypes editor plugin to allow coppermine to accept whatever raw format you want to use.

[Stramm]

afaik memory usage is the same if processing a compressed file such as jpg. For memory usage just width x/y and colour depth are important.

[phill104]

I think my point is that most RAW files are compressed to about 9mb but even the simplest of RAW processor on my linux box needs to use about 25mb just to display it. I tried examining the mem usage on a RAW file while converting it to a jpeg and it rose to 68mb and really hit the processor hard albeit for a brief time. I think that processing lots of RAW files could cause some hosts to have a fit not to mention the amout of time it takes to upload such large files. That is my concern.

[Αndré]

Phill, please check the memory usage and processing time while converting a jpg file with the same dimensions and color depth, as afaik

For memory usage just width x/y and colour depth are important.

[phill104]

Partly true Andre. In the case of RAW files you already start with a much higher colour depth Working from a digital camera point of view the conversion to an 8bit jpeg also takes a lot of processing as demosaicing and white balance are not applied in the RAW image.

Maybe this article will explain a little more - http://www.cambridgeincolour.com/tutorials/RAW-file-format.htm

[roger_morgan]

I was discussing this very point with a colleague, and so found this topic again. Thought I would post my comments here rather than start a new topic.

I often use coppermine as an intranet solution due to its flexibility and good online support. In an intranet setting bandwidth requirements because of file size is not really a problem. A common requirement is to use the browsing/categorising/search facilities of coppermine, but still to have access to a raw file image if needed for photo-shopping purposes, also to avoid split storage. One of the main advantages of coppermine is that all the image files stay together in the same directory rather than being split among directories according to size.

Personally I always pre-process my images, create the "normal" and "thumbnail" sizes upload them into the appropriate sub-directory and use the import add-in which is then very quick as it finds the files already there. ie. my laptop does all the work away from the server.

To proove the point about processing raw files with ImageMagick I created a little bash script (most servers run a linux derivative) to show the time/ressource differences

#!/bin/bash
# Quick and dirty timing test using image magick with Nikon Raw, and jpg files
# refer http://forum.coppermine-gallery.net/index.php/topic,7033.0.html
#
# Note my camera (Nikon D700) gives me a large jpg and a raw file
#
#
# writen 20120717 roger.morgan@htl-steyr.ac.at 
#
# SuSE 12.1, ImageMagick 6.7.2-7 2011-10-19 Q16
#
echo -e "\n\n SuSE Version\n"
cat /etc/SuSE-release
echo -e "\n\n ImageMagick Version\n"
convert -version
#
PICTURE=QSD_8808
#
echo -e "\n## Picure : ${PICTURE}  ##\n\n"
ls -al ${PICTURE}.*
echo -e "\n## 1) Timings for \'normal\' pictures\n"
echo -e "\n## 1.1) JPG\n"
/usr/bin/time -f "\n%E elapsed,\n%U user,\n%S system,\n%M memory\n%x status" convert ${PICTURE}.JPG  -quality 80 -resize 400x400 normal_${PICTURE}_JPG.jpg
echo -e "\n## 1.2) NEF\n"
/usr/bin/time -f "\n%E elapsed,\n%U user,\n%S system,\n%M memory\n%x status" convert ${PICTURE}.NEF  -quality 80 -resize 400x400 normal_${PICTURE}_NEF.jpg
echo -e "\n\n jhead infos\n"
jhead normal_*
echo -e "\n## 2) Timings for \'thumbnail\' pictures\n"
echo -e "\n## 2.1) JPG\n"
/usr/bin/time -f "\n%E elapsed,\n%U user,\n%S system,\n%M memory\n%x status" convert ${PICTURE}.JPG  -quality 80 -resize 150x150 thumb_${PICTURE}_JPG.jpg
echo -e "\n## 2.2) NEF\n"
/usr/bin/time -f "\n%E elapsed,\n%U user,\n%S system,\n%M memory\n%x status" convert ${PICTURE}.NEF  -quality 80 -resize 150x150 thumb_${PICTURE}_NEF.jpg
echo -e "\n\n jhead infos\n"
jhead thumb_*
echo -e "\n\n -----------------------------------------------------------------------------------------\n"
ls -al *${PICTURE}*.*
   

Here are the results

wwwrun@bilder:/srv/www/htdocs/cpg_test_timing> ./doit.sh


SuSE Version

openSUSE 12.1 (i586)
VERSION = 12.1
CODENAME = Asparagus


ImageMagick Version

Version: ImageMagick 6.7.2-7 2011-10-19 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC
Features: OpenMP


## Picure : QSD_8808  ##


-rw-r--r-- 1 wwwrun www  5117022 Jun 17 18:28 QSD_8808.JPG
-rwxr--r-- 1 wwwrun www 14378897 Jun 17 18:28 QSD_8808.NEF

## 1) Timings for \'normal\' pictures


## 1.1) JPG


0:00.75 elapsed,
1.10 user,
0.04 system,
428688 memory
0 status

## 1.2) NEF


0:20.20 elapsed,
23.44 user,
0.28 system,
604640 memory
0 status


jhead infos

File name    : normal_QSD_8808_JPG.jpg
File size    : 63267 bytes
File date    : 2012:07:17 17:05:30
Camera make  : NIKON CORPORATION
Camera model : NIKON D700
Date/Time    : 2012:06:17 18:28:39
Resolution   : 266 x 400
Flash used   : No
Focal length : 24.0mm  (35mm equivalent: 24mm)
Exposure time: 0.0031 s  (1/320)
Aperture     : f/9.0
ISO equiv.   : 200
Whitebalance : Auto
Metering Mode: matrix
Exposure     : program (auto)
GPS Latitude : ? ?
GPS Longitude: ? ?

File name    : normal_QSD_8808_NEF.jpg
File size    : 19666 bytes
File date    : 2012:07:17 17:05:50
Camera make  : NIKON CORPORATION
Camera model : NIKON D700
Date/Time    : 2012:06:17 18:28:39
Resolution   : 266 x 400
Flash used   : No
Focal length : 24.0mm  (35mm equivalent: 24mm)
Exposure time: 0.0031 s  (1/320)
Aperture     : f/9.0
ISO equiv.   : 200
Whitebalance : Auto
Metering Mode: matrix
Exposure     : program (auto)
GPS Latitude : ? ?
GPS Longitude: ? ?


## 2) Timings for \'thumbnail\' pictures


## 2.1) JPG


0:00.73 elapsed,
1.06 user,
0.04 system,
403648 memory
0 status

## 2.2) NEF


0:20.19 elapsed,
23.42 user,
0.25 system,
604592 memory
0 status


jhead infos

File name    : thumb_QSD_8808_JPG.jpg
File size    : 48283 bytes
File date    : 2012:07:17 17:05:51
Camera make  : NIKON CORPORATION
Camera model : NIKON D700
Date/Time    : 2012:06:17 18:28:39
Resolution   : 100 x 150
Flash used   : No
Focal length : 24.0mm  (35mm equivalent: 24mm)
Exposure time: 0.0031 s  (1/320)
Aperture     : f/9.0
ISO equiv.   : 200
Whitebalance : Auto
Metering Mode: matrix
Exposure     : program (auto)
GPS Latitude : ? ?
GPS Longitude: ? ?

File name    : thumb_QSD_8808_NEF.jpg
File size    : 9473 bytes
File date    : 2012:07:17 17:06:11
Camera make  : NIKON CORPORATION
Camera model : NIKON D700
Date/Time    : 2012:06:17 18:28:39
Resolution   : 100 x 150
Flash used   : No
Focal length : 24.0mm  (35mm equivalent: 24mm)
Exposure time: 0.0031 s  (1/320)
Aperture     : f/9.0
ISO equiv.   : 200
Whitebalance : Auto
Metering Mode: matrix
Exposure     : program (auto)
GPS Latitude : ? ?
GPS Longitude: ? ?





-----------------------------------------------------------------------------

------------

-rw-r--r-- 1 wwwrun www  5117022 Jun 17 18:28 QSD_8808.JPG
-rwxr--r-- 1 wwwrun www 14378897 Jun 17 18:28 QSD_8808.NEF
-rw-r--r-- 1 wwwrun www    63267 Jul 17 17:05 normal_QSD_8808_JPG.jpg
-rw-r--r-- 1 wwwrun www    19666 Jul 17 17:05 normal_QSD_8808_NEF.jpg
-rw-r--r-- 1 wwwrun www    48283 Jul 17 17:05 thumb_QSD_8808_JPG.jpg
-rw-r--r-- 1 wwwrun www     9473 Jul 17 17:06 thumb_QSD_8808_NEF.jpg
wwwrun@bilder:/srv/www/htdocs/cpg_test_timing>

I have deliberately not added other possible options to the ImageMagick convert program. In my case the resized images from the raw format are not as vivid as the camera produced jpg's



The results show quite clearly the extra time/ressources needed to process the Nikon raw format files. Personally as I always take the JPG in the camera, I do not need to process the raw formats for coppermine, but just copy the raw files into the same subdirectory so I can find them again when needed.

Hope this is of interest to you

[Αndré]

Thank you very much! From your results it seems that processing RAW images needs about 50% more memory and 20-30 times more CPU time than processing JPG files. I assume both source files have the same resolution?

[phill104]

I think you can see the resolutions from the attached jpeg Andre. They findings above seem to tie up with mine. Thanks to Roger_Morgan for posting those. Very interesting results.

[roger_morgan]

Glad it is still of interest.

One thing somebody might be able to comment on is why are the resized images from the raw files considerably smaller than from the jpeg? (They both get the EXIF data , so that isn't the reason!)

Roger

[Joe Carver]

Glad it is still of interest.

It certainly is. Great work!
You have proven very well what limitations can be asked of a web server.

Personally I always pre-process my images, create the "normal" and "thumbnail" sizes upload them into the appropriate sub-directory and use the import add-in which is then very quick as it finds the files already there. ie. my laptop does all the work away from the server.

That's still the best method overall and even the consumer grade editing software allows for that to be done in batch routines.

Personally as I always take the JPG in the camera, I do not need to process the raw formats for coppermine, but just copy the raw files into the same subdirectory so I can find them again when needed.

The lowering costs and increasing speed of memory make that much more practical today.

One thing somebody might be able to comment on is why are the resized images from the raw files considerably smaller than from the jpeg? (They both get the EXIF data , so that isn't the reason!)

Could this be an artifact of ImageMagick or something you could command to change? I see the same effect with .bmp files using a consumer grade of editing software. It will return different file sizes depending on the source type - compressing compression?

Again, great work.