Batch add with automatic album/cat structure from directory tree.

[andrez1]

I just added the pieces that allow you to pass the intial directory as null and it will import everything under the "albums" directory.

Hello.

Nice script. 

It works well, imports everything under /albums.  But doing so with a null value to 'SubDirectory (under albums)' results in
'filepath' in 'cpg***_pictures' starting with a slash.

If the picture is already there, by other methods: it will be inserted twice.

It produce adresses like http://**/cpg133/albums//***/**/*.jpg , double slash after albums.

Any way of filtering out things like '../../../..'   as possible value   'SubDirectory (under albums)' ?





Mvh Andrez1

[flux]

i haven't analyzed how the directory structure is created but a quick and dirty way to solve the problem would be to simple remove a possible "/" first char in the path string just before insterting the picture..

i guess that having an openning "/" in the filepath data is not correct so the following simple patch would be okay.. although the proper thing to do would be to sanitize the file path when building the directory structure..

So using the latest Donoman file, below line 178 :

Code Select Expand

$dir_name = dirname($pic_file) . "/";

insert :

Code Select Expand

$dir_name = ( substr($dir_name,0,1) == "/" ) ? substr($dir_name,1) : $dir_name;

that should do it. If so please confirm it works fine, i have nothing to test it on here, the code works don't worry, just make sure it also works when using a subcategory info in the field..
then i'll post the updated file.

thx for your bug report

.flux

[andrez1]

So using the latest Donoman file, below line 178 :

Code Select Expand

$dir_name = dirname($pic_file) . "/";

insert :

Code Select Expand

$dir_name = ( substr($dir_name,0,1) == "/" ) ? substr($dir_name,1) : $dir_name;

that should do it. If so please confirm it works fine, i have nothing to test it on here, the code works don't worry, just make sure it also works when using a subcategory info in the field..
then i'll post the updated file.

That makes the script take a blank value.

It also works when  giving as a value a subdir containing another dir with imagefiles.

although the proper thing to do would be to sanitize the file path when building the directory structure..

Absolutely..

Tested a litle bit. 

the file (win):

c:\XXXXXX\htdocs\cpg133\albums\C370-3\TJA14\P8190003.JPG

in subdir C370-3\TJA14 is accepted at the  following values:

(aid,filepath,filename)      (given value to the script)

6236,\\c370-3\\/tja14/,P8190003.JPG        \c370-3\
6233,c370-3\\/tja14/,P8190003.JPG          c370-3\
6230,c370-3//tja14/,P8190003.JPG           c370-3/
6227,c370-3/tja14/,P8190003.JPG            c370-3
6224,\\c370-3//tja14/,P8190003.JPG        \c370-3/


The values given to the script results in 5 categories like the values given to the script. And a 'tja14' album in each.

The  'c370-3' value seems to be what the script is expecting. But it accepts  a lot (including '..', which might be a risk).



Mvh Andrez1

[andrez1]

That makes the script take a blank value.

But filtering on what is already in the database is broken.. That does not work with the newly added line to the script,  and a null value passed to the scripts 'SubDirectory (under albums)'.

Filtering does work when the script is given a subdir as value.


Mvh Andrez1

[flux]

okay,

based on your comments i've made a few modifications to the latest version of donnoman's script.

it will now handle blank "subcategory" field properly and store valid paths in the database, without starting "/".

Filtering also works with blank "subcategory" field.

Concerning your previously insterted pictures using a blank "subcategory" field they were all stored with a faulty filepath so you'll have to fix them. The following sql query will do that for you :

Code Select Expand

UPDATE prefix_pictures SET filepath=SUBSTRING(filepath,2) WHERE RIGHT(filepath,1)='/'
of course replace "prefix_pictures" table name by your own table name using your "prefix_" name..

that's about it..

oh yeah i've also added some more version checking in the script so it'll fail to start if you're not using cpg v1.3.x or 1.4.x
And it'll also work now for both versions 1.3.x and 1.4.x, i've just wrapped the db_query function BUT did some tests on a cpg1.3.5 version only !

So if you can confirm it works fine on a 1.4.1 beta too, i'll thank you for that ;)

.flux

[flux]

one last quick word, all my tests are done on a linux plateform so if there are some paths problems when running this script on windows, i'm afraid i won't be able to do much about it.

[andrez1]

it will now handle blank "subcategory" field properly and store valid paths in the database, without starting "/".

Filtering also works with blank "subcategory" field.

Works fine in cpg 1.33 an 1.41.

The 1.33 also gets filename as title.

The 1.41 do not getfilname as title. (and thats ok, can be set in config on thumbs)

Have not used the script for anything serious. Yet.  Nothing to repair.

As of now basic functionality is there, working. Fine to insert large amounts of pictures.

I have found 3000 pictures in a directory, presized and with thumbs_  present, to be the practical limit with 'batch add' on a local pentium4 - 3ghz -xp.  (and that is with standard values on script time in php).  More than that and 'reload' or new start on given directory is needed.

The 'batch add' gives possibility to set which album to insert pictures into.  This script does not. But it solves a problem with inserting large amounts of pictures. That is the use i consider. And when it's done something like -

UPDATE `cpg132_pictures` SET `aid` = 3 WHERE `filename` between 'xxx005.jpg' and 'xxx010.jpg'

(or whatever the WHERE clause and 'aid' is set to) - will  arrange the pictures in another order than the directory-structure.
(Hiding the albums originating from the script is part of idea.) 

The next time the script is run on the given directory it does not find the picture in db and do not filter it out. It tries to add it and reports something like 'picture already there'.  Thats OK, but not optimal.


Mvh Andrez1

[andrez1]

one last quick word, all my tests are done on a linux plateform so if there are some paths problems when running this script on windows, i'm afraid i won't be able to do much about it.

Not a windows-linux path problem. '..'  is understood as 'one directory up' in windows, linux and html.

'../themes' as a value to the scripts 'SubDirectory (under albums)'  gives a result i think not shold be allowed.

'/dir' or '/dir/'  or 'dir/subdir'  (given existing dirs) produces categories and albums and insert files into them. I think it should not.


I have a category 'c370-7/7' with the album 'tja22'  an a file like 'P8250003.JPG'

http://XXXX/cpg133/albums/c370-7/7/tja22/P8250003.JPG

That is produced by passing 'c370-7/7' to the script.  (dir and subdirs exist, there is nothing in the script to guide me,  stop me or warn me.)

I also have a  'c370-7' category with a '7' subcategory and 'tja22' album. Thats empty.  Should not be.


And so on. Gets messy.



Mvh Andrez1

[flux]

not sure i understand why your previously added pictures are not filtered but then skipped with an "already added" output.. shouldn't happen, something must be broken with the filtering, although what i've tested here was working fine as far as i could tell..


as for "../" it's not supported and not filtered either because it's simply a bad use of the subdirectory field..
The script and the whole cpg system only deals with pictures placed in your "album" (aka fullpath in the config) directory directly under the root directory..

I guess it must have been stated somewhere in the first explanations of this batch add tool that the subdirectory field was made to target a directory under the "album" directory..
thus using "../" entries makes no sence as it would mean leaving the "album" directory tree..

that's all i can say..

[andrez1]

not sure i understand why your previously added pictures are not filtered but then skipped with an "already added" output.. shouldn't happen, something must be broken with the filtering, although what i've tested here was working fine as far as i could tell..

That happens on a cpg133 when pictures are inserted using the script and a value to 'SubDirectory (under albums)' (f.ex 'c370-8')

The dir 'c370-8' exist and have a subdir 'tja23' which contains pictures.

'http://xxxxx/cpg133/albums/c370-8/TJA23/normal_P8280001.JPG' is visible in coppermine after use of the script. The image is found in the category 'c370-8', album 'tja23'.  It is as it should be.

But there is no need to stick to the somewhat rigid order imposed by the script. 

The mentioned album 'tja23' have an 'aid' like '11250' , and a new album created like 'fancyname' with an 'aid' like '11251'.

Running  ---  UPDATE `cpg133_pictures` SET `aid` = 11251 WHERE `aid` like '11250'  --- on the db empties the album created by the script and move the pictures to another album. The album 'fancyname'. That kind of use is neccesary if someone want another name on an album other than the subdir the file resides in. That kind of use make the script useable.

The only thing changed is the 'aid' field in `cpg133_pictures`. 

the pair of 'filepath' and 'filename' is the same.

as for "../" it's not supported and not filtered either because it's simply a bad use of the subdirectory field..
The script and the whole cpg system only deals with pictures placed in your "album" (aka fullpath in the config) directory directly under the root directory..

I guess it must have been stated somewhere in the first explanations of this batch add tool that the subdirectory field was made to target a directory under the "album" directory..
thus using "../" entries makes no sence as it would mean leaving the "album" directory tree..

It can be a feature  to be allowed to list the whole directorystructure of the disk (filesystem?) on a server running coppermine.

'../../..' to the script gives me a category '../../..' with a subcategory 'htdocs'.


'../../coppermine/albums/test1' to the script gives me a category '../../coppermine/albums/test1'  containing albums. From another coppermine-install that is..

If I ask for image properties on:

'http://xxxxx/cpg133/displayimage.php?album=7182&pos=0'

I get

'http://xxxxx/coppermine/albums/test1/alt/xxx001.jpg'

'filepath' on that picture is set to '../../coppermine/albums/test1/alt/'

A nice feature which can be exploited.

Mvh Andrez1

[Paver]

Should the image next to the thread say "1.4.x" as well?  Maybe the most current version should be added to the first message (with Nibbler's permission & an edit note of course)?

The current version is here:
http://forum.coppermine-gallery.net/index.php?topic=13949.msg105809#msg105809

I have some mods I'd like to suggest, like ignoring the "Thumbs.db" files Windows creates, a test run that doesn't create albums (to check things like when you rename an album in CPG and then timer.php wants to add a new album with the current folder name but now old album name - a mod I'm trying to decide how to address), and a mod to remove forbidden characters in directory names (like CPG does with filenames).  I'll add them here when I get the chance.  I'd also like to try to convert this into a plugin.  Has anyone attempted this yet?  Of course it needs to be more bulletproof and have more checks installed.

[donnoman]

I'd be happy to host this project over on cpg-contrib.org, and we could have it made as a plugin, and available from cvs.

Perhaps a better name than TIMER might be appropriate

When someone figures out a good name, I'll create the project.

[Juanito]

if its not too much to ask, would it be possible to get this nicely made timer script to include the mini thumbs and orgi back-up pix? im using Stramm's mod pack found here, http://forum.coppermine-gallery.net/index.php?topic=24341.0 and his mods add two additional files for each pix... one with the prefix mini_ and another with orig_ . And when I run the most recent timer script by flux, it detects these additional files as a completely new pictures and starts creating normal_ and thumb_ files for them. So would it be possible to get the script to include or perhaps simply just ignore the orig_ and mini_ files? If it involves alot of scripting, then no worries.. ill just stop using mini thumbs and watermarking. thanks!

[Paver]

So would it be possible to get the script to include or perhaps simply just ignore the orig_ and mini_ files?

I haven't used Stramm's mod pack, but I did modify the timer script to ignore the "Thumbs.db" files that Windows creates when you browse Thumbnails in a directory.  The script above already ignores the thumb_ and normal_ files that Coppermine creates, so you merely have to add orig_ and mini_ to the 2 if statements that do this.

In the createstructure function, look for lines similar to these and modify as shown:

Code Select Expand

foreach ($contents as $file) {
if ( (strncmp($file,$CONFIG['thumb_pfx'],strlen($CONFIG['thumb_pfx'])) != 0 ) && ( strncmp($file,$CONFIG['normal_pfx'],strlen($CONFIG['normal_pfx'])) != 0) && ( strncmp($file,'orig_',strlen('orig_')) != 0) && (strncmp($file,'mini_',strlen('mini_')) != 0)) {
$filelist["$path/$name/$file"] = $aid;
}
}

Then in the addpic function, modify the if statement similarly:

Code Select Expand

if ( (strncmp($file_name,$CONFIG['thumb_pfx'],strlen($CONFIG['thumb_pfx'])) != 0 ) && ( strncmp($file_name,$CONFIG['normal_pfx'],strlen($CONFIG['normal_pfx'])) != 0) && ( strncmp($file_name,'orig_',strlen('orig_')) != 0) && (strncmp($file_name,'mini_',strlen('mini_')) != 0)) {

[Juanito]

Thanks! Works like a charm. 

[fade2grey]

Hi guys, I'm have some problems with this - probably user error...

so far I've grabbed the file attached to the first post in this thread (updated Jan 4th 06)
copied it to my coppermine folder as timer.php
when I run it in a browser, I get a load of garbled PHP & a few boxes/fields to complete - nedless to say, it does nothing.
I've uploaded some test files in the format:

FTP_Upload
    Test
         10_01_2005
         11_01_2005
etc
the intention is for them to have the nesc catagory created & them to be posted in the Test Album.

Any suggestions or quick pointers?

A

[donnoman]

instead of grabbing the first post, why dont you go through the thread and grab the LAST updated copy, as most of the bugs and 1.3 and 1.4 compatability was worked out in the later versions.

[fade2grey]

Okay, I've grabbed the most recent one from this thread & I get the following when Debug is turned on:

While executing query "INSERT INTO cpg143_categories (pos, parent, name) VALUES ('10000', '0', 'Test')" on 0

mySQL error: Field 'description' doesn't have a default value

[donnoman]

sounds like your on Mysql 5, search the board, it's been discussed before.

[Prisoner_24601]

I was getting the same error as fade2grey (CPG 1.4.3, Win2K3, PHP 5, MySQL 5) and, at the advice of my webhost, I:

* Booted up phpMyAdmin
* Found the table "cpg143_categories"
* Set the DEFAULT column for "description" to NULL.

Ran the newest version of timer, again. 

Now, PROVIDED:

* I had a folder under "albums" called "testy" and
* A folder under "testy" called "testy2" (so the directory structure looks like: <root>\cpg143\albums\testy\testy2")

The TIMER script would create the category "testy" with an album in it called "testy2".

That's the good.  The bad is it wouldn't upload the pictures in "testy2".

When I turned the debug on, I'd get the following error:

Code Select Expand

While executing query "INSERT INTO cpg143_pictures (aid, filepath, filename, filesize, total_filesize, pwidth, pheight, ctime, owner_id, owner_name, title, caption, keywords, approved, user1, user2, user3, user4, pic_raw_ip, pic_hdr_ip, position) VALUES ('12', 'testy/testy2/', 'a.jpg', '14702', '17383', '362', '295', '1137635330', '1', '[my server name, edited out]','', '', '', 'YES', '', '', '', '', '', '', 'a.jpg')" on 0

mySQL error: Out of range value adjusted for column 'position' at row 1

File: D:\Inetpub\wwwroot\[my server]\cpg143\include\functions.inc.php - Line: 248

============

Notices:

\include\themes.inc.php

    * Warning line 1167: Cannot modify header information - headers already sent by (output started at D:\Inetpub\wwwroot\[my server]\cpg143\timer.php:10)
    * Warning line 1168: Cannot modify header information - headers already sent by (output started at D:\Inetpub\wwwroot\[my server]\cpg143\timer.php:10)

\include\functions.inc.php

    * Warning line 162: Cannot modify header information - headers already sent by (output started at D:\Inetpub\wwwroot\[my server]\cpg143\timer.php:10)

Little file named "a.jpg".  All of 12K.

I tried editing out the "pos" statements in timer.php and then deleting them altogether.  I tried changing the DEFAULT column to null and not null or other numbers in the cpg143_pictures table with phpMyAdmin and nothing helped.  I tried combinations of the above, too.

Any suggestions?  (My webhost does not wish to downgrade MySQL to 4.x.)