Large File support?

[dcbrit2003]

Hi,

I'm currently trying to get CPG to add some video files to the database ( i.ve already uploaded them to my web server ).  The files range between 1.2 - 2.5 GB in size. I keep getting the message "no new file was found". I have checked and re-checked that I have used the correct syntax for the thumbnail image, i.e.:

file1.avi corresponds to thumb_file1.jpg

Initially I thought that it was Apache that was the problem as I was running ver 1.3.34 and so I installed ver. 2.2 which has Large File support. When I tried to access the files just via Apache ( i.e. from a basic directory and not using PHP, etc ) it now worked, whereas before I just got an error. I went back to trying to add the AVI files to the CPG database via the "batch add files" and still it was the same old message "no new file was found" yet the syntax for the files and thumds was fine. I then looked around to see if PHP was the problem and compiled in the following CFLAGS into PHP :

-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

I thought again that this would do the trick yet i'm no nearer solving this, it would seem. Can CPG handle files which have already been uploaded which are greater than 1GB in size? what else can I try?

p.s. I tried establishing beyond doubt that it is a file size issue buy copying over a 500mb file from another directory and renaming it it to the exact same filename in the exact same directory of the 1GB+ file that I was trying to add. This worked fine and CPG had no problem adding the smaller file so it is a file size issue

Regards, Mark

[dcbrit2003]

i'm using CPG 1.4.4, PHP 5.1.4 Apache 2.2.2 and MySQL 5.0.20a

Any help will be greatly appreciated

[Nibbler]

I don't have any files of that size to test with. It's more likely to be an underlying issue with server setup than anything directly related to Coppermine since the script only use the functions built-in to php to read and manipulate files

[dcbrit2003]

Hi,

I am using Apache 2.2.2 which has Large File support and I have compiled in Large File support to PHP 5.1.4. I did a test by replacing an existing movie file ( size of about 350MB ) which was succesfully added to the CPG database, with one of the 1GB+ movie files by using the exact same filename in the exact same directory. Unfortunetly, the encoding used in the two files is different ( WMV vs AVI ) so all I could expect was my browser to load the Windows Media Player Plugin and then just sit there donig nothing. This is exactly what happened so it is obviously not a filesystem, Apache or PHP issue. It seems to be a CPG issue ( From Apache log )

:xx.xxx.xx.xx - - [09/May/2006:10:04:00 -0500] "GET /members/albums/movies/512k/gdfgdf/dfgdfgdf-512k.wmv HTTP/1.1" 200 2256795648

The above request was done from within the PHP interface so large files are not a problem with my PHP + Apache installation yet when I try adding 1GB+ files to my CPG installation's database I cannot.

Regards, Mark

[Joachim Müller]

what happens if you try to load the file using your media player directly (using the player's "open" dialog), i.e. without embedding the player into your browser?

[dcbrit2003]

I just tried this and it downloads the file :

xx.xxx.xx.xx - - [10/May/2006:12:04:25 -0500] "GET /members/albums/movies/3000k/fghgfhfghfg/fghfghfg.avi HTTP/1.1" 200 3063927980

so i'm just wondering why CPG has a problem with large ( 1GB+ ) files.

Regards, Mark

[Joachim Müller]

Coppermine is a PHP script and therefor bound to any limitations set up on the server for PHP. As a guess, I'd look at the execution time / time-out settings in php.ini (if the server is yours to administer). There's no explicit code in Coppermine (at least none that I'm aware of) that artificially limits the size of a video file, that's why my guess is a PHP limitation.

[dcbrit2003]

I think that the solution here is to just split my 1GB+ movies up. This will be a PITA because i'll have to re-encode them and upload them ( all 32 GB's worth ) but I don't see any other way.

[Paver]

There might be a way to split the files byte-wise, upload, then put them back together again on the server.  I know you can do that with shell access, the question is can you do it in PHP and create a Coppermine add-on/mod/plugin to do this for you.

I know some people have considered this in threads on this forum before, but I don't remember what or if they decided anything.

[Tranz]

If it's a pain to upload huge files, it'll be a pain to download them, too...

[Paver]

Heh.  That's a really good point. 

[Joachim Müller]

I fully and whole-heartedly agree. I can't see any good reason why people should download videos that "weigh" Gibabytes or more using their browser (i.e. Coppermine) . If you must distribute such large files, use FTP or file sharing like BitTorrent - anything that allows downloads to be resumed. Even users on broadband will have issues downloading such huge files, no matter what the actual content may be.
You better think about codecs that have better compression (like DivX or similar). 32 GB for one video file just is a bit too much.

[Paver]

You could use Coppermine to organize the .torrent files if you have a need for 32 GB video files.

[dcbrit2003]

To answer several points above:

I don't have any files which are 32GB in size, someone seems to be guilty of not reading my posts. As i've said, all my video files are 1.2 - 2.5 GB in size. it's a PITA to split them up using VirtualDub and then re-upload them because my upload speed is restricted to about 300Kb/s, whereas my download speed is 10 Mb/s so it would not be much of a pain for people to download them. My movies are already encoded using Divx.

Regards, Mark

[dcbrit2003]

There might be a way to split the files byte-wise, upload, then put them back together again on the server.  I know you can do that with shell access, the question is can you do it in PHP and create a Coppermine add-on/mod/plugin to do this for you.

I know some people have considered this in threads on this forum before, but I don't remember what or if they decided anything.

I used an MSDOS version of the unix 'split' command to split the 1GB+ movie files up before uploading and re-combine them using 'cat' to reconsitute the files after uploading before I realised I had this Large File problem. I used two computers on the same network to upload each piece to minimise any data corruption problem stoping the whole upload process, which sometimes happens on my SFTP client. Now i'll have to split each file into seperate video files ( Part 1, 2,3, etc ) so as not to be more then 750MB just to be on the safe side.

Regards, Mark

[Paver]

I assume you are using something like MD5 checksums to verify each part and the final file, but in case not, I thought I'd mention it.  RFSync is a good tool to fix errors when found.

[Joachim Müller]

I don't have any files which are 32GB in size, someone seems to be guilty of not reading my posts.

I confess that I'm going to be guilty to ignore your postings from now on... Sorry for misreading your important posting, this can happen if you read thousands of postings In the end, 1 GB or 32 GB doesn't make a difference to me: such files are too large for http