External Batch Load of Picture/Albums

[ZaZ]

Hi all,

I don't know if this request has already been posted (I tried to read the forums but I didn't find anything matching).

It Would be nice to have a "batch" load with the following characteristics
- DOS batch (or a Shell script) lanched from the command line or regularly scheduled
- log into Coppermine as Admin (and only available to Admin!)
- automatically load images from an external directory (local to the web server)
- automatically update the MySQL db with additional info (descriptions, keywords, user info) stored in a file (.CSV?)

This would help a lot loading massive number of photos (I'm thinking about a big digital archive setup/maintenance).

What do you think about?

Other topic... Just to save space why not dynamically resize images on the fly instead of saving a normal_*.jpg version on disk?

I just gave a look to LinPha (another photo archive/publishing system) and it has this nice feature.

Thanks all! An keep on doing your great job! really well done. I love it!

[tuxhedoh]

Other topic... Just to save space why not dynamically resize images on the fly instead of saving a normal_*.jpg version on disk?

I just gave a look to LinPha (another photo archive/publishing system) and it has this nice feature.

I'd like the command line utility as well.... and would love to know more about the dynamic resizing of the images... how much load would this put on the server?  Is it even something that's possible... and if possible, could it be modular in that it could be turned on and off as desired by the admin.

[Joachim Müller]

dynamic image creation is a huge resource eater. Take a look at the watermark hack of DJAxion that's around somewhere on this board (do a little search); it does on-the-fly creation/modification of pics (as a point to start with).
It can be done (but hasn't been done yet afaik).

GauGau

[Venik]

Command-line batch add of images would be extremely beneficial, particularly to users who have a lot of photos to add on a regular basis.

There seems to be an issue with adding a large number of photos from a single directory. Initially everything is going well and you see those blue "OK" icons popping up. However, once the number of added photos gets in the 400-500 range, all you get are the "red crosses" and the images are not added. Clicking on the "red cross" reveals no useful information. I tried to optimize MySQL, PHP, Apache2 settings as well as to use different browsers... Still, the problem remains and it's hard to figure out where it's coming from.

I suppose a Shell/SQL script can be put together with some effort to add photos, titles, and captions from command line, but it would be nice to have this option come standard with Coppermine.

As an example, I have photos, captions, and dates stored in separate files with matching names. I add the photos to the gallery via the Coppermine Web interface. However, the dates and the photos are added by this rather unsightly Sh/SQL script. It may be far from the ideals of efficiency, but it gets the job done quickly enough for my needs.

Code Select Expand


#!/bin/sh
#
MYSQL="mysql -U******** -P********"
DB="photodump"
TABLE="photoarc_pictures"

BASEDIR="/disk4/htdocs/photodump/albums"
NEWSDIR="news"
LOG=$BASEDIR"/"$NEWSDIR"/photodump.log"

if [ ! -f $LOG ]
then
touch $LOG
chown wwwrun $LOG
chgrp www $LOG
fi

ls $BASEDIR"/"$NEWSDIR | fgrep "_photos" | awk -F'_' '{print $1"_"$2}' | while read DIR
do
if [ `cat $LOG | fgrep -c $DIR` -eq 0 ]
then
PREPATH=$NEWSDIR"/"$DIR
FILEPATH=$PREPATH"_photos/"
COUNT=$(ls $BASEDIR"/"$PREPATH"_photos" | egrep -v "normal|thumb" | fgrep "photo_" | wc -l | awk '{print $1}')

i=1
while [ $i -le $COUNT ]
do
if [ -f $BASEDIR"/"$PREPATH"_photos/photo_"$i".jpg" ] && [ -f $BASEDIR"/"$PREPATH"_dates/date_"$i".txt" ] && [ -f $BASEDIR"/"$PREPATH"_captions/caption_"$i".txt" ]
then
FILENAME=photo_$i.jpg
echo "Processing $DIR - $FILENAME"
DATE=$(cat $BASEDIR"/"$PREPATH"_dates/date_"$i".txt")
CAPTION=$(cat $BASEDIR"/"$PREPATH"_captions/caption_"$i".txt")

#------------------------------
# BEGIN SQL QUERY
#------------------------------
eval $MYSQL <<EOF

CONNECT $DB;
UPDATE $TABLE
SET caption = "$DATE - $CAPTION"
WHERE filename = "$FILENAME" AND filepath = "$FILEPATH";
quit

EOF
#------------------------------
# END SQL QUERY
#------------------------------

fi
(( i = i + 1 ))
done
echo $DIR >> $LOG
fi
done



And a couple other wishes:

1) The "Categories" page gets very "heavy" and take a while to load even over LAN once as the number of categories grows. For me it's still bearable, but inevitably a point will be reached when it will be easier to add/edit categories directly via mysql.

2) Again the "batch add" feature - the list of directories seems to take an eternity to load if you have a large number directories/files in the "albums" folder. I suppose this can be fixed by a more experience end user. Still, it's a limitation of Coppermine that should be relatively simple to remove in the "production" version of Coppermine.

These are the three big Christmas wishes :-) And so I don't appear ungfrateful: Coppermine is a masterful combination of simplicity and powerful features. I chose it because it had everything I needed and nothing I didn't need. And that doesn't happen very often. Thanks.

[Casper]

We would never recommend adding as many as 500 pics in one go.  This is almost certain to cause timeouts.
The next version has a browsable interface for the batch add, which should make it much quicker.
The category manager issue has been discussed.  I cannot remember the outcome, but worth bringing back to the discussion table.

[Venik]

About the timeout issue I was having - I just stumbled upon an acceptable workaround. If I use the latest Mozilla, enter "about:config" in the URL box, and in the resulting configuration screen double or triple the values for: network.http.max-persistent-connections-per-server, network.http.max-persistent-connections-per-proxy, and network.http.max-persistent-connections-per-proxy the timeout issue goes away and I was able to add more than 4,000 photos in one try.

Also, I noticed that the average CPU load on the server during batch-add operations can be dropped from about 90+% to about 60+% by simply adding a "sleep(1);" after "$result = db_query($sql);" in addpic.php. Of course, the batch-add process takes longer to complete, but the server becomes much more responsive. It may be useful to have the ability to set some sort of a flow-control delay for batch-add runs in the CPG config menu.

And another batch-add thing: the process looks in "albums" directory (or whatever directory is set in the config). It would be nice to have an ability to select different "main" directories on the fly. For example, clicking on "Batch add files" would show a drop-down list of "album1, album2, album3, etc.) 'Cause once you get a million files in one directory tree, things tend to get a bit sluggish