Uploading Big Photos in Large Quantities

[emperoradz]

We are running Coppermine + JUpload (latest version) on our Yearbook Server.

Here are our settings:
Max size for uploaded files (KB): 10000000000000
Max width or height for uploaded pictures/videos (pixels): 2048000
maxChunkSize: 2000
maxFileSize: 50000
maxNonTransformedPictureSize: 50000
pictureCompressionQuality: 1

We upload all kinds of photos (7MP, 8MP, 12MP, etc). The largest we usually upload is 12MP (7000-10000KB).
We want NO resizing and modification. Just plain uploads.

Now when we try to upload 100 or so 12MP files, it's fine. But when we try uploading 500 or so, after 127 it starts complaining that the files are too big, and we need to click OK repeatedly for the 400 remaining files.

Any way to turn off ALL warnings, turn of resizing of any kind, and let any picture through unblemished.

Also, when is metadata preservation coming....

[Joachim Müller]

We are running Coppermine + JUpload (latest version) on our Yearbook Server.

JUpload is a plugin that is not part of coppermine itself. It only get's supported within the very sub-board "cpg1.4 JUpload by etienne_sf"

Max size for uploaded files (KB): 10000000000000

Use settings that make sense: 10,000 TB is just plain stupid and might cause issues.

Max width or height for uploaded pictures/videos (pixels): 2048000

That's of course silly as well.

maxChunkSize: 2000
maxFileSize: 50000
maxNonTransformedPictureSize: 50000
pictureCompressionQuality: 1

Those are JUpload settings. I have no idea of their impact.

We want NO resizing and modification. Just plain uploads.

You don't even want thumbnails to be created? OK, that's an unusual setup.

Now when we try to upload 100 or so 12MP files, it's fine. But when we try uploading 500 or so, after 127 it starts complaining that the files are too big, and we need to click OK repeatedly for the 400 remaining files.

Happens when? Using JUpload or using coppermine's batch-add?

Any way to turn off ALL warnings

What warnings? Batch-add is an admin only feature. The admin should be able to live with some warnings.

turn of resizing of any kind

Sure: set the max pixel size for thumbnails and intermediates to the same value you have set your max resolution to.

Also, when is metadata preservation coming....

As far as Coppermine is concerned: as soon as the image libraries that Coppermine can use (GD or ImageMagick) will support meta tag preservation we'll add that feature to coppermine as well. We can't add features to coppermine that the underlying architecture don't have.

We have a strict "one question per thread" policy that you agreed to respect when signing up. Another policy says that it's absolutely mandatory to post a link to your gallery in each thread. And yet another policy says that you must have read the docs before being allowed to post on the support board. If you have issues with uploading (and as far as I am concerned batch-add is an uploading method just as well), do exactly as suggested in the docs, section "asking for support on upload issues".

If you have posted this in error on the upload sub-board and where looking for JUpload support only, say so in your reply, so a moderator can move your thread accordingly. Most of the recommendations posted above apply nevertheless though, as JUpload is using the server-side of coppermine.

[emperoradz]

Ah looks like I posted it in the wrong place. Can you please move this to JUpload?

Use settings that make sense: 10,000 TB is just plain stupid and might cause issues.That's of course silly as well.Those are JUpload settings. I have no idea of their impact.

Meant to be no limit, wasn't sure how to do it so just put 10 PB

You don't even want thumbnails to be created? OK, that's an unusual setup.

We do, we just don't want the original file to be edited.

Happens when? Using JUpload or using coppermine's batch-add?

JUpload, we only use JUpload

What warnings? Batch-add is an admin only feature. The admin should be able to live with some warnings.Sure: set the max pixel size for thumbnails and intermediates to the same value you have set your max resolution to.

Says size is too big so it won't be resized (JUpload warning)

We have a strict "one question per thread" policy that you agreed to respect when signing up. Another policy says that it's absolutely mandatory to post a link to your gallery in each thread. And yet another policy says that you must have read the docs before being allowed to post on the support board. If you have issues with uploading (and as far as I am concerned batch-add is an uploading method just as well), do exactly as suggested in the docs, section "asking for support on upload issues".

It's being a private network

If you have posted this in error on the upload sub-board and where looking for JUpload support only, say so in your reply, so a moderator can move your thread accordingly. Most of the recommendations posted above apply nevertheless though, as JUpload is using the server-side of coppermine.

yup sorry. Please move it to JUpload board

[Joachim Müller]

It's being a private network

Then there can only be generic advice: make sure that regular http uploads work as expected first before uploading with JUpload. Can't help you with JUpload issues, as I don't know/understand/use it, so I'm out of this thread. Bye.

It's being a private network

Then there can only be generic yup sorry. Please move it to JUpload board[/quote]Phill did as you suggested.

[etienne_sf]

Hi,

  I missed this message as I (apparently) get no alarm in my mail box, when a message is moved. Sorry for that.

Here are my answer, about JUpload :

We are running Coppermine + JUpload (latest version) on our Yearbook Server.

Here are our settings:
Max size for uploaded files (KB): 10000000000000
Max width or height for uploaded pictures/videos (pixels): 2048000
maxChunkSize: 2000
maxFileSize: 50000
maxNonTransformedPictureSize: 50000
pictureCompressionQuality: 1

We upload all kinds of photos (7MP, 8MP, 12MP, etc). The largest we usually upload is 12MP (7000-10000KB).
We want NO resizing and modification. Just plain uploads.

Now when we try to upload 100 or so 12MP files, it's fine. But when we try uploading 500 or so, after 127 it starts complaining that the files are too big, and we need to click OK repeatedly for the 400 remaining files.

Can you put here the JUpload debug output ?

I never managed so many such big files...

Any way to turn off ALL warnings, turn of resizing of any kind, and let any picture through unblemished.

I don't know exactly about he Warnings.

To turn off resizing:
- Keep the parameter you've choosen
- Put in maxFileSize a bigger value, like you did for 'Max size for uploaded file'

Also, when is metadata preservation coming....

  You can preserve metadata. Go to the JUpload configuration page (parameter pictureTransmitMetadata). But you'll have to test if it's ok, with your configuration. With (apparently) recent camera, preserving leads to 'strange' colors. See one of the three head messages in this list (and vote for its correction ! see explanation there).


Etienne

[adzempire]

USER:
------------------
Array
(
    [ID] => f9da9978139d0594aa491a6d05f594d0
    [am] => 1
    [lang] => english
    [lap] => 3
    [liv] => Array
        (
        )

    [search] => Array
        (
            [search] => speech
            [album] => search
            [plugin_search_album] => true
            [title] => on
            [newer_than] =>
            [caption] => on
            [older_than] =>
            [keywords] => on
            [type] => OR
            [album_title] => on
            [album_description] => on
            [params] => Array
                (
                    [title] => on
                    [caption] => on
                    [keywords] => on
                )

        )

)

==========================
USER DATA:
------------------
Array
(
    [user_id] => 61
    [user_name] => Thorp
    [groups] => Array
        (
           

• => 1
          )
  
      [disk_max] => 0
      [disk_min] => 0
      [can_rate_pictures] => 1
      [can_send_ecards] => 1
      [ufc_max] => 3
      [ufc_min] => 3
      [custom_user_upload] => 0
      [num_file_upload] => 5
      [num_URI_upload] => 3
      [can_post_comments] => 1
      [can_upload_pictures] => 1
      [can_create_albums] => 1
      [has_admin_access] => 1
      [pub_upl_need_approval] => 0
      [priv_upl_need_approval] => 0
      [group_name] => Administrators
      [upload_form_config] => 3
      [group_quota] => 0
      [can_see_all_albums] => 1
      [group_id] => 1
  )
  
  ==========================
  Queries:
  ------------------
  Array
  (
     
• => SELECT extension, mime, content, player FROM cpg14x_filetypes; (0s)
      [1] => select * from cpg14x_plugins order by priority asc; (0s)
      [2] => delete from `pic10`.cpg14x_sessions where time<1260472192 and remember=0; (0s)
      [3] => delete from `pic10`.cpg14x_sessions where time<1259266192; (0s)
      [4] => select user_id from `pic10`.cpg14x_sessions where session_id = '5ea0482526b053e6080b8074ddfcb9bb' (0s)
      [5] => select user_id as id, user_password as password from `pic10`.cpg14x_users where user_id=61 (0s)
      [6] => SELECT u.user_id AS id, u.user_name AS username, u.user_password AS password, u.user_group+100 AS group_id FROM `pic10`.cpg14x_users AS u INNER JOIN `pic10`.cpg14x_usergroups AS g ON u.user_group=g.group_id WHERE u.user_id='61' (0s)
      [7] => SELECT user_group_list FROM `pic10`.cpg14x_users AS u WHERE user_id='61' and user_group_list <> ''; (0s)
      [8] => SELECT MAX(group_quota) as disk_max, MIN(group_quota) as disk_min, MAX(can_rate_pictures) as can_rate_pictures, MAX(can_send_ecards) as can_send_ecards, MAX(upload_form_config) as ufc_max, MIN(upload_form_config) as ufc_min, MAX(custom_user_upload) as custom_user_upload, MAX(num_file_upload) as num_file_upload, MAX(num_URI_upload) as num_URI_upload, MAX(can_post_comments) as can_post_comments, MAX(can_upload_pictures) as can_upload_pictures, MAX(can_create_albums) as can_create_albums, MAX(has_admin_access) as has_admin_access, MIN(pub_upl_need_approval) as pub_upl_need_approval, MIN( priv_upl_need_approval) as  priv_upl_need_approval FROM cpg14x_usergroups WHERE group_id in (1) (0s)
      [9] => SELECT group_name FROM  cpg14x_usergroups WHERE group_id= 1 (0s)
      [10] => update `pic10`.cpg14x_sessions set time='1260475792' where session_id = '5ea0482526b053e6080b8074ddfcb9bb' (0s)
      [11] => SELECT user_favpics FROM cpg14x_favpics WHERE user_id = 61 (0s)
      [12] => DELETE FROM cpg14x_mod_online WHERE last_action < NOW() - INTERVAL 10 MINUTE (0s)
      [13] => REPLACE INTO cpg14x_mod_online (user_id, user_name, user_ip, last_action) VALUES ('61', 'Thorp', '10.127.2.238', NOW()) (0s)
      [14] => DELETE FROM cpg14x_banned WHERE expiry < '2009-12-10 15:09:52' (0s)
      [15] => SELECT * FROM cpg14x_banned WHERE (ip_addr='10.127.2.238' OR ip_addr='10.127.2.238' OR user_id=61) AND brute_force=0 (0s)
      [16] =>  SELECT category, count(1) nb  FROM cpg14x_albums  WHERE category < 10000 OR category = 10061 GROUP BY category  (0s)
      [17] =>  SELECT aid, title, description   FROM cpg14x_albums  WHERE category < 10000 ORDER BY UPPER(title) (0s)
      [18] =>  SELECT aid, title, description  FROM cpg14x_albums  WHERE category='10061'  ORDER BY UPPER(title) (0s)
      [19] => SELECT COUNT(*) FROM cpg14x_pictures WHERE approved = 'NO' (0s)
      [20] =>  SELECT cid, name, description, pos  FROM cpg14x_categories  WHERE parent = '0'  ORDER BY pos (0s)
      [21] =>  SELECT cid, name, description, pos  FROM cpg14x_categories  WHERE parent = '1'  ORDER BY pos (0s)
      [22] =>  SELECT cid, name, description, pos  FROM cpg14x_categories  WHERE parent = '2'  ORDER BY pos (0s)
      [23] =>  SELECT cid, name, description, pos  FROM cpg14x_categories  WHERE parent = '3'  ORDER BY pos (0s)
      [24] =>  SELECT cid, name, description, pos  FROM cpg14x_categories  WHERE parent = '4'  ORDER BY pos (0s)
      [25] =>  SELECT cid, name, description, pos  FROM cpg14x_categories  WHERE parent = '5'  ORDER BY pos (0s)
      [26] =>  SELECT cid, name, description, pos  FROM cpg14x_categories  WHERE parent = '6'  ORDER BY pos (0s)
      [27] =>  SELECT cid, name, description, pos  FROM cpg14x_categories  WHERE parent = '7'  ORDER BY pos (0s)
      [28] =>  SELECT cid, name, description, pos  FROM cpg14x_categories  WHERE parent = '8'  ORDER BY pos (0s)
      [29] =>  SELECT cid, name, description, pos  FROM cpg14x_categories  WHERE parent = '9'  ORDER BY pos (0s)
      [30] =>
              SELECT aid, title, description
              FROM cpg14x_albums
              WHERE category = 10061
              ORDER BY UPPER(title)
               (0s)
      [31] => SELECT sum(total_filesize) FROM cpg14x_pictures WHERE owner_id = 61 (0s)
  )
  
  ==========================
  GET :
  ------------------
  Array
  (
      [file] => jupload/jupload
  )
  
  ==========================
  POST :
  ------------------
  Array
  (
  )
  
  ==========================
  VERSION INFO :
  ------------------
  PHP version: 5.2.10-2ubuntu5 - OK
  ------------------
  mySQL version: 5.1.37-1ubuntu5
  ------------------
  Coppermine version: 1.4.25(stable)
  ==========================
  Module: GD
  ------------------
  GD Version: 2.0 or higher
  FreeType Support: 1
  FreeType Linkage: with freetype
  T1Lib Support: 1
  GIF Read Support: 1
  GIF Create Support: 1
  JPG Support: 1
  PNG Support: 1
  WBMP Support: 1
  XPM Support:
  XBM Support:
  JIS-mapped Japanese Font Support:
  
  ==========================
  Module: mysql
  ------------------
  MySQL Supportenabled
  Active Persistent Links 0
  Active Links 1
  Client API version 5.1.37
  MYSQL_MODULE_TYPE external
  MYSQL_SOCKET /var/run/mysqld/mysqld.sock
  MYSQL_INCLUDE -I/usr/include/mysql
  MYSQL_LIBS -L/usr/lib -lmysqlclient_r 
  ==========================
  Module: zlib
  ------------------
  ZLib Support enabled
  Stream Wrapper support compress.zlib://
  Stream Filter support zlib.inflate, zlib.deflate
  Compiled Version 1.2.3.3
  Linked Version 1.2.3.3
  ==========================
  Server restrictions (safe mode)?
  ------------------
  Directive | Local Value | Master Value
  safe_mode | Off | Off
  safe_mode_exec_dir | no value | no value
  safe_mode_gid | Off | Off
  safe_mode_include_dir | no value | no value
  safe_mode_exec_dir | no value | no value
  sql.safe_mode | Off | Off
  disable_functions | no value | no value
  file_uploads | On | On
  include_path | .:/usr/share/php:/usr/share/pear | .:/usr/share/php:/usr/share/pear
  open_basedir | no value | no value
  ==========================
  email
  ------------------
  Directive | Local Value | Master Value
  sendmail_from | no value | no value
  sendmail_path | /usr/sbin/sendmail -t -i  | /usr/sbin/sendmail -t -i
  SMTP | localhost | localhost
  smtp_port | 25 | 25
  ==========================
  Size and Time
  ------------------
  Directive | Local Value | Master Value
  max_execution_time | 100000 | 100000
  max_input_time | 10000 | 10000
  upload_max_filesize | 1000M | 1000M
  post_max_size | 8M | 8M
  ==========================
  Page generated in 0.061 seconds - 32 queries in 0 seconds - Album set : ; Meta set: ;

[etienne_sf]

adzempire,

  What's this message about

Is this linked to this (old) thread ?

Etienne

[emperoradz]

yup it is

The big photo error when uploading > 100ish files

[etienne_sf]

Hi,

  Ok. So what I need is the JUpload debug. Can you read the doc here, for an howto on this point :
http://forum.coppermine-gallery.net/index.php/topic,53168.0.html

Etienne