External Images Links / Use "remote" storage servers

[flux]

okay as it seems we can't post in the admin functions forum i'll post my hack here, hopping it'll get noticed, rated and maybe posted where it belongs :)

so this hack is of course about the topic i investigated in this thread.

I made several versions of my hack to be able to store picture on remote servers and finally decided to only keep the very simplest one and post it here..

For the moment i'd say it's still in beta as it's only been tested by myself.. And you could call it "Pseudo Remote Gallery" hack as i'll explain how it works below :

Basically this hack will let the gallery use pictures that are located on a different server than the local server and still behave just as if the pictures where stored locally..

## Requirements ##
Php with "http" fopen support

CPG 1.4.x (tested on cpg 141 beta), my admin tool "files.php" should also work on cpg 1.3 with the proper "db_query" function modifications. The modifications to themes.inc.php could probably be ported to 1.3 too.

a properly formated remote file list.


## Installation ##
In the Config section "Custom fields for image description", set "Field 4 name" to "Remote URL". Be carefull no typo will be tolerated..

then put the file "files.php" into the root of your cpg gallery and put the file "themes.inc.php" in the include directory.
You can rename "files.php" to anything the script should still work.

## How does it work / What does it do ? ##

"files.php" is a batch add tool, to add remote pictures to your gallery. It's built on the "timer.php" script i found in this forum's admin tools scripts..
You can find details about that batch add script here : http://forum.coppermine-gallery.net/index.php?topic=13949.msg102336#msg102336

basicaly it creates categories/albums based on paths provided in the remote pictures list you'll provide.

So the important thing is to provide my script with a properly formated list of remote files and locations to store the local pictures to.

The list should always have the following format :

/relative/album/path/
http://www.domain.com/direct/path/to/the/picture.jpg
..
..
..
/relative/otheralbum/path/
http://www.domain.com/direct/path/to/the/picture.jpg*
..
..

and so on
"/relative/album/path/" is the path that will be used to both create the category/album structure in your gallery AND download/store the pictures under your albums/ directory..

so in my above example list, the first picture would be downloaded to albums/relative/album/path/, the categories "relative" > "album" would be created if not existing and the album "path" would be created inside the "album" category..
okay my choice of words for the path was really poor but anyways it's quite simple, right ?

Once the download of the remote picture is completed successfully the "thumb" and "normal" pictures will be created according to your gallery configuration, meaning the normal picture does not have to be created/stored locally if you don't use normal pictures or if it's not required according to the source picture size and your max sizes configuration..
Last but not least the source picture we've just downloaded will be removed and replaced by an empty file of the very same name. This empty file is necessary for later dupe checks to make sure the picture is already present in the gallery and use another filename to perform the download and storage of thumbs/normal pics..

Well that's about it.. Oh yeah of course the source URL is stored inside the "Field 4 name" field of the picture so it can be used to display the fullsize picture inside your gallery.

now when you browse your gallery, thumbs and normal pics are used locally but the fullsize picture is fetched from the source url..

That works both when using the fullsize popup from a normal picture display but also when not using any normal picture at all..

And of course local pictures can still be used !
The gallery behaviour towards local pics in unchanged, so you can potentially mix local and remote pictures as you wish..


## A few further explanations on the batch add script and the list format ##

any line starting with "/" will be considered a path and treated as such.
A path will be created prior to adding any pictures to it so even though no pictures follow the path it would still be created !

any line starting with "http" will be considered a source url, case sensitivity is not a matter (HttP would be recognized as well).

any other line will be ignored !

so if you screw up your paths some pictures could end up inside the previous path. Coz if the new path is faulty it will be ignored and the latest recognized path would still be used for the next pictures..

Pictures following a "/" path will be skipped. That would mean adding pictures with an album ID of 0 and those pics cannot be sorted with the current admin tools of CPG so i decided to skip those "unsorted" pictures..


## IMPORTANT LIMITATIONS ##

the main cpg hack is located in the include/themes.inc.php script which stores functions to display the pictures in the gallery.
As i see it those functions can be localized inside a CPG theme, meaning that with such a theme my modifications in the original include script would be overriden and the hack would no longer work !
If your theme is using its own display functions you have to manualy apply my hack to them. Search for "user4" in the include/themes.inc.php and patch you theme functions.


You can "see" a demo of the hack here : http://capsnscans.free.fr/cpg/
all the source pics are remote, thumbs and normals are local as explained above.


I still consider this hack beta because there must be some things i've missed or skipped being focused on my own way of using cpg..


Let me know if this hack works fine for you and if you have suggestions/questions, i'll be glad to take a look at them.

.flux


## Make sure you check my posts below they will have the latest versions of this hack ! ##

[Joachim Müller]

split the above post from the thread "[feat. req.] External Images Links / Use "remote" storage servers" on the "Feature requests" board that got turned down as the dev team doesn't want coppermine to be used for stealing other's bandwidth. However, we welcome the contribution. We strongly suggest that all who want to use this mod make up their minds on bandwidth theft and only use this mod to share pics from another webspace that they own or pay the bills for.

Joachim

[flux]

sorry i don't quite understand what you want me to do with my post.. should i (re)post it in another of your forums with a different topic ?


concerning bandwidth problems, here's my point of view and possible customization of this hack / remote system :

first yeah you can steal bandwidth when using remote urls, but recently the huge growth of free picture hosting pushes almost everyone to use thoses "remote" hostings anyway..

Now in order to give the credit to the host actually hosting the remote picture i could modify my hack to clearly reference / link the hosting company on top of every remote picture displayed in the gallery.

Moreover i could introduce a config parameter limiting the remote urls to some of the biggest and well known free picture hosts, so at least it shows a gesture to prove our good intentions..

of course it would be quite simple to go around that limitation but "burying" that data in the database would probably prevent newbies to do whatever they want with it..

[Joachim Müller]

sorry i don't quite understand what you want me to do with my post.. should i (re)post it in another of your forums with a different topic ?

Everything is fine, you don't need to do anything. I was justing pointing out that the dev team disagrees with storing pics remotely. So if people come here and ask for the mod, we'll tell them that it is there, but we will not recommend using it. Having nearly unlimited webspace only will make newbie users waste resources imo - they'll upload incredibly large pics.

Cheers

Joachim

[Aktiv]

Sounds great - just the right hack to use my university webspace 

But unfortunately it doesn't work. I've copied the files.php and themes.inc.php into the particular folders and I've got the following file uploadpictures.txt:

Code Select Expand

/Test/test/
http://www.informatik.uni-oldenburg.de/~jb007/derrest/000_0043.jpg
/Test/test/
http://www.informatik.uni-oldenburg.de/~jb007/derrest/000_0045.jpg
/Test/test/
http://www.informatik.uni-oldenburg.de/~jb007/derrest/000_0047.jpg
When pressing the button [Begin], I get the following:

Code Select Expand

File Uploaded : /tmp/phpCmhJuE
File Size : 236

Back to the main form : Back
It seems as if nothing happens - there are no new pictures in my album "test" or somewhere else.
Category Test and Album test are already existing and I use the theme "rainy day" and Coppermine 1.3.3 - maybe I should upgrade to 1.4.X or did I just miss something?

[Nibbler]

This mod is made for 1.4, it won't work on 1.3

[flux]

hi,

yeah the above files are only for 1.4.x, the script should work with 1.3.x if you replace the cpg_db_query function calls by db_query..

but you'd still have to hack your 1.3 themes.inc.php in order to get this to work..

I'll try to make a 1.3 version when i have time, shouldn't be too much of a problem.. I'll try to figure out how to determine what version of cpg is currently running my batch add tool, so you don't have to worry about thoses db_query functions.

[flux]

okay i just had a couple of minutes to update my batch script so it works with CPG 1.3 : TESTED ON 1.3.2 ONLY !

the files are in the "remote_cpg13_b2.zip"
both files.php and displayimage.php go into your cpg root

CPG Configuration for 1.3 is the same as for 1.4 :
In the Config section "Custom fields for image description", set "Field 4 name" to "Remote URL". Be carefull no typo will be tolerated..




And I've also updated the 1.4 version of the batch script and cpg hack : TESTED ON CPG 1.4.1 ONLY !
the files are in the "remote_cpg14_b2.zip".
files.php goes in the root and themes.inc.php goes into "include" directory.


The display now shows a remote host link above the picture, both in fullsize and intermediate versions.

the demo is still available here : http://capsnscans.free.fr/cpg/

Know Bug :
After analyzing a bit longer my hack for the 1.3 version i noticed the host address was not displayed in slide show mode.. Meaning that if you don't use intermediate pictures, it's most probable that your slideshow won't work with remote pictures..
I haven't found exactly how the slideshow display works and why my image display modifications are skipped in slideshow mode.

[Stramm]

have a look at include/slideshow.inc.php
to get the pic urls it's using function get_pic_url (in functions.inc.php). If you don't have the correct URLs here it won't work (all URLs in a javascript array that gets cycled)

[flux]

thx stram

okay i've modified slideshow.inc.php, but had to make a small modification to the get_picture_data function too in functions.inc.php..

i guess i'll have to give all the modification details so anyone can add them manually and not overwrite the whole file..
i'll try to do that when i have some spare time.

the good thing is that it works for slideshows too now when you don't use intermediate pics and the fullzise is remote one..

i'll try to make the slideshow hack for CPG 1.3 too.. gee i hope CPG 1.5 won't get released next week =)

[Aktiv]

Thank you!

[flux]

you're welcome ;)


So I have updated my hack with the slideshow thing, so it'll work with slideshows too now :

## Current Version is Beta 3 ##

You'll find the files for "CPG 1.4.1 Beta" AND "CPG 1.3.5" below

Installation requires manual modifications so make sure you read the INSTALL.txt File !
for 1.4 and 1.3 : 1 modification of include/functions.inc.php for the slideshow to work
for 1.3 only : 1 modification of include/iptc.inc.php to trap an error if you're using iptc

If everything works fine for all the user out there i'm not sure there'll be any further modifications / additions to this hack, maybe some to the batch add script although it's quite functionnal already, imho..

.flux

[flux]

could an admin reset the cpg compatibility version info on this thread, i won't edit the original message anymore as it seems to remove it.

so it's compatible with 1.3 and 1.4 now

thx

.flux

[digimic]

hello
if i understand with "remote_cpg141b_b3"
i can manage pictures on another server. Can i do exactly the same actions as the local server? like host all pictures format, upload picture on the remote server...?

[flux]

pictures have to be located on a remote server before adding them to cpg gallery thru my hack.

my cpg hack only makes cpg gallery use those remote pictures and produce thumbnails and intermediates versions of those remote pictures on the local server. It won't managed remote pictures in any ways, or store anything on those remote servers thru your cpg tools..

You can still use cpg as a local gallery just as you'd use a unhacked version..

[digimic]

Hello Flux!
DO you have a new release of your hack for the version 1.4.2 ?

[Nibbler]

Read the thread.

[Joachim Müller]

..and http://forum.coppermine-gallery.net/index.php?topic=24540.0 as well...

[flux]

humm, haven't tested my hack over version 1.4.2

i'll do that shortly and report problems and possibly new version of the hack for cpg142 soon..

i've also been trying to implement a better handling of the downloads in my batch add tool coz i've had some problems with some downloads never getting any timeouts and so making the batch process hang forever, unfortunately beeing no php geek i haven't managed to make something thru fopen although i did come accross a few examples making loops and counting the time but i couldn't get it to work.. so i anyone reads this and can help me out, please drop me a mail or pm ont this forum..

[flux]

New 1.4.x Version : Beta4

having finally some spare time to test the hack on 1.4.2 sources, i couldn't resist to add/improve some features, so here they are :

##########################
* Beta4 CHANGES
- hack code locations are clearly marked with "#REMOTE HACK START/END" blocks (not a feature but usefull :).
# batch add tool
- support for user1 to user4 extra info.
- added pictures log in the database.
# web display
- iframes & remote pages urls instead of only remote pictures urls, kinda anti-anti-hotlinking.
- width management of remote files when not using intermediate pictures.
- added links to popup fullsize remote pictures or iframes.
- alternate, nicer way to hack into your cpg themes.
##########################


basically the big new feature is support for iframes.
Remote pictures URLs can now be added along with their true web page URL and that web page URL can be used to display iframes in your gallery.
Displaying the page url, instead of the remote picture directly avoids almost any anti-hotlinking the hosts can use.
And is some way also gives the proper credits to the picture host as the entire web page is displayed, including possible advertisement and so on..
With mosts free picture hosting iframe doesn't make a difference and won't polute your gallery in any way..

here's the doc concerning the new features that is present in the INSTALL.txt file in the zip


##########################
* Using IFRAMES, "Remote Page URL" data and other user data:
In order to use the iframes you will have to provide my batch add too with a second data for your remote pictures which is the remote page url from where that remote picture can be seen.
Indeed as you probably already know many free pictures hosts give access to hosted pictures thru a web page of their own, so the following picture "http://www.freepichost.com/random/path/to/picture_name.jpg" can be accessed thru the following URL "http://www.freepic.host/page.php?picture=picture_name.jpg", or something like that.
It can happen that accessing the picture using its direct URL will return an "access denied" page or a different picture mentioning the fact that you can't access the picture directly. Then the only way to access that picture is done by accessing the web page with the second link.
So if you set your CPG user data 3 aka "Field 3 name" with the following name "Remote Page URL" and provide my batch add tool with the page URL then it'll be stored and used to display that picture thru it's webpage inside the gallery using an iframe.
You will also have to set the default "define('REMOTE_PIC_IFRAME',0);" to "define('REMOTE_PIC_IFRAME',1);" to enable iframes.

Then you can build your remote picture list as follows :
/path/to/album
http://www.freepichost.com/random/path/to/picture_name.jpg \ http://www.freepic.host/page.php?picture=picture_name.jpg
...

the page URL is placed on the very same line as the picture url and separated by a "\".

You can also place two more data on every line that will be stored insode the "Field 2 name" and "Field 1 name" of each of your added pictures.
So in a matter of speaking you could see each picture line with the following description :

Picture URL \ Page URL \ user data 2 \ user data 1

or if you're not using Page URL and iframes you can even see it like that :
Picture URL \ user data 3 \ user data 2 \ user data 1

You can of course put anything you want on each data, expected you won't use the "\" char in yout text strings..
Beware the "\" char cannot be escaped link in some programming language, the strings will be interpreted asis.


* Using the width management with remote pictures :
When the intermediate picture is disabled you can enforce the remote picture width, displayed on your pages with an additional define() as follows :
define('REMOTE_PIC_MAXW',1);

the width of the displayed remote picture will then be according to your Max Intermediate Width Setting in CPG config.
Of course i've made sure to add a link above the iframe or on the picture in order to display the fullsize one with the CPG popup you already know.


* Using the Added Pictures Log :
That log is simply an additional table you can create/purge/remove within my batch add tool, that will log every picture you add to your gallery.
You can very well leave that alone and never use it, it will not make any difference in the way my Remote hack works online..

The only idea behind that log was to avoid inserting the same remote picture more than once.
So if you insert a remote picture in your database and for some reason decide to delete it from you database manually then when the log is active the batch add tool won't add that very same picture ever again when it finds it in your lists.
The reason for that log is that i'm using some automated scripts that go over remote pictures lists that can have the same source from times to times and i don't want the pictures i've removed manually to be added ever again, so i keep a log of every picture i've added to the gallery and decide not to add them in the future.
Of course as nobody's perfect you can always make mistakes and remove pictures you shouldn't have, so you can override the log whenever you want during a single batch add session. The log will then be ignored.

##########################



enjoy and report bugs here.
The trickiest part was the slideshow so i expect some thing to look weird there especially with iframes.. The modified javascript should be clean enough for recent browsers and display is okay for Mozilla Browsers which was the only thing i had in mind when coding that thing.

.flux