Importing large quantity of images, MySQL query error

[rivers]

Thanks for all the help so far, the boards have been very useful.

This is my second run at using CPGmassimport to add about 20,000 images.  I don't think its a CPGmassimport issue though.  Everything is working fine, but when I get close to 10,000 completed, I get a MySQL error (debug and notices below).  I would like to know if this is a limitaiton with my host (which is not supposed to have any limitations on MySQL size).   I have no clue    where to start.


Thanks in advance.

There was an error while processing a database query.

While executing query "SELECT aid FROM cpg14x_albums WHERE title='Animals__A-G' AND category=0 LIMIT 1" on 0 

mySQL error: Lost connection to MySQL server during query

Notices
/plugins/CPGMassImport/codebase.php

    * Notice line 15: Use of undefined constant GALLERY_ADMIN_MODE - assumed 'GALLERY_ADMIN_MODE'
    * Notice line 36: Undefined variable: CONFIG

/plugins/final_extract/include/init2.inc.php

    * Warning line 30: mysql_free_result(): supplied argument is not a valid MySQL result resource

/include/themes.inc.php

    * Warning line 1227: Missing argument 1 for pageheader()
    * Notice line 1241: Undefined variable: section

/plugins/CPGMassImport/import.php

    * Notice line 305: Undefined variable: cid

/include/functions.inc.php

    * Warning line 2327: mysql_fetch_row(): supplied argument is not a valid MySQL result resource

USER: 
------------------
Array
(
    [ID] => 9824c1a2d5e9bf2aff0f539eef966f1e
    [am] => 1
    [lang] => english
    [liv] => Array
        (
            
 => 19
            [1] => 22
            [2] => 18
            [3] => 17
            [4] => 20
        )

    [sort] => nd
    [search] => Array
        (
            [search] => flagellation
            [album] => search
            [title] => on
            [newer_than] => 
            [caption] => on
            [older_than] => 
            [keywords] => on
            [type] => AND
            [params] => Array
                (
                    [title] => on
                    [caption] => on
                    [keywords] => on
                )

        )

    [lap] => 2
)

==========================
USER DATA:
------------------
Array
(
    [user_id] => 1
    [user_name] => ifadmin
    [groups] => Array
        (
            
 => 1
        )

    [disk_max] => 0
    [disk_min] => 0
    [can_rate_pictures] => 0
    [can_send_ecards] => 0
    [ufc_max] => 3
    [ufc_min] => 3
    [custom_user_upload] => 0
    [num_file_upload] => 5
    [num_URI_upload] => 3
    [can_post_comments] => 0
    [can_upload_pictures] => 1
    [can_create_albums] => 1
    [has_admin_access] => 1
    [pub_upl_need_approval] => 0
    [priv_upl_need_approval] => 0
    [group_name] => Administrators
    [upload_form_config] => 3
    [group_quota] => 0
    [can_see_all_albums] => 1
    [group_id] => 1
)

==========================
Queries:
------------------
Array
(
    
 => SELECT extension, mime, content, player FROM cpg14x_filetypes; (0.001s)
    [1] => select * from cpg14x_plugins order by priority asc; (0s)
    [2] => delete from `mycp1357`.cpg14x_sessions where time<1197986530 and remember=0; (0s)
    [3] => delete from `mycp1357`.cpg14x_sessions where time<1196780530; (0.001s)
    [4] => select user_id from `mycp1357`.cpg14x_sessions where session_id=md5("43cb2f41d9e1cc919cf8efd290c9493a764c860982f506f303c6c0d189f53574"); (0.001s)
    [5] => select user_id as id, user_password as password from `mycp1357`.cpg14x_users where user_id=1 (0s)
    [6] => SELECT u.user_id AS id, u.user_name AS username, u.user_password AS password, u.user_group+100 AS group_id FROM `mycp1357`.cpg14x_users AS u INNER JOIN `mycp1357`.cpg14x_usergroups AS g ON u.user_group=g.group_id WHERE u.user_id='1' (0s)
    [7] => SELECT user_group_list FROM `mycp1357`.cpg14x_users AS u WHERE user_id='1' and user_group_list <> ''; (0s)
    [8] => SELECT MAX(group_quota) as disk_max, MIN(group_quota) as disk_min, MAX(can_rate_pictures) as can_rate_pictures, MAX(can_send_ecards) as can_send_ecards, MAX(upload_form_config) as ufc_max, MIN(upload_form_config) as ufc_min, MAX(custom_user_upload) as custom_user_upload, MAX(num_file_upload) as num_file_upload, MAX(num_URI_upload) as num_URI_upload, MAX(can_post_comments) as can_post_comments, MAX(can_upload_pictures) as can_upload_pictures, MAX(can_create_albums) as can_create_albums, MAX(has_admin_access) as has_admin_access, MIN(pub_upl_need_approval) as pub_upl_need_approval, MIN( priv_upl_need_approval) as  priv_upl_need_approval FROM cpg14x_usergroups WHERE group_id in (1) (0.001s)
    [9] => SELECT group_name FROM  cpg14x_usergroups WHERE group_id= 1 (0s)
    [10] => update `mycp1357`.cpg14x_sessions set time='1197990130' where session_id=md5('43cb2f41d9e1cc919cf8efd290c9493a764c860982f506f303c6c0d189f53574'); (0.001s)
    [11] => SELECT user_favpics FROM cpg14x_favpics WHERE user_id = 1 (0.001s)
    [12] => SELECT * FROM cpg14x_final_extract_config (0.001s)
    [13] => DELETE FROM cpg14x_banned WHERE expiry < '2007-12-18 15:02:10' (0.001s)
    [14] => SELECT * FROM cpg14x_banned WHERE (ip_addr='156.34.73.28' OR ip_addr='156.34.73.28' OR user_id=1) AND brute_force=0 (0.001s)
    [15] => SELECT COUNT(*) FROM cpg14x_pictures WHERE approved = 'NO' (0s)
    [16] => SELECT aid FROM cpg14x_albums WHERE title='Animals__A-G' AND category=0 LIMIT 1 (0s)
)

==========================
GET :
------------------
Array
(
    [file] => CPGMassImport/import
)

==========================
POST :
------------------
Array
(
    [directory] => 
    [sleep] => 1000
    [auto] => 1
    [hardlimit] => 10
    [start] => Begin
)

==========================
VERSION INFO :
------------------
PHP version: 4.3.11 - OK
------------------
mySQL version: 
------------------
Coppermine version: 1.4.14(stable)
==========================
Module: GD
------------------
GD Version: bundled (2.0.28 compatible)
FreeType Support: 1
FreeType Linkage: with freetype
T1Lib Support: 
GIF Read Support: 1
GIF Create Support: 1
JPG Support: 1
PNG Support: 1
WBMP Support: 1
XBM Support: 1
JIS-mapped Japanese Font Support: 

==========================
Module: mysql
------------------
MySQL Supportenabled
Active Persistent Links 0 
Active Links 1 
Client API version 5.0.18 
MYSQL_MODULE_TYPE external 
MYSQL_SOCKET /usr/local/mysql-5.0/data/mysql.sock 
MYSQL_INCLUDE -I/usr/local/mysql-5.0/include/mysql 
MYSQL_LIBS -L/usr/local/mysql-5.0/lib/mysql -lmysqlclient  
==========================
Module: zlib
------------------
ZLib Support enabled 
Compiled Version 1.1.4 
Linked Version 1.2.3 
==========================
Server restrictions (safe mode)?
------------------
Directive | Local Value | Master Value
safe_mode | Off | Off
safe_mode_exec_dir | no value | no value
safe_mode_gid | Off | Off
safe_mode_include_dir | ~ | ~
safe_mode_exec_dir | no value | no value
sql.safe_mode | Off | Off
disable_functions | no value | no value
file_uploads | On | On
include_path | .:/usr/local/lib/php | .:/usr/local/lib/php
open_basedir | no value | no value
==========================
email
------------------
Directive | Local Value | Master Value
sendmail_from | me@localhost.com | me@localhost.com
sendmail_path | /usr/sbin/sendmail -t -i  | /usr/sbin/sendmail -t -i 
SMTP | relay-hosting.secureserver.net | relay-hosting.secureserver.net
smtp_port | 25 | 25
==========================
Size and Time
------------------
Directive | Local Value | Master Value
max_execution_time | 30 | 30
max_input_time | 60 | 60
upload_max_filesize | 8M | 8M
post_max_size | 8M | 8M
==========================
Page generated in 23.599 seconds - 17 queries in 0.009 seconds - Album set : ; Meta set: ;[/size][/pre]

[Nibbler]

http://dev.mysql.com/doc/refman/5.0/en/gone-away.html

[rivers]

Strange... Nothing worked at all, so I removed the "Animals__A-G" folder which I noticed was listed during all the MySQL "lost connection" errors.  It worked, all folders on the server were created as galleries and files successfully added. 

Next, I added back the "Animals__A-G" folder and ran CPGmassimport again to add it, and got the MySQL errors again.  Something is wrong with this folder.  It has fewer pictures than some of the other folders, there are no strange filenames or characters. 


What could one particular folder/files be doing to cause the MySQL errors?


Thanks

[Nibbler]

Taking too long probably. Add code to reconnect to include/functions.inc.php cpg_db_query()

Code Select Expand


if (!is_resource($CONFIG['LINK_ID'])){
    $CONFIG['LINK_ID'] = cpg_db_connect();
}


[rivers]

Thanks for the help.  I split the folder into two and it worked.  I would have liked to try your code though.  May still have to in the future.  FOr now I'm going to backup the DB and keep it as it is.


Thanks again.