Index.php currently takes over 20 seconds to load, and I've gotten a few email (okay, account suspensions) from my host about the load my site is putting on the server. It looks like debug is saying the page is executing over 700 queries, and most of them look the same, but I don't really know what they mean.
I've read the documentation for the Album List View setting, and tried what was suggested in this thread (http://forum.coppermine-gallery.net/index.php/topic,32791.msg152596.html#msg152596), but it didn't help. No random pics are used, and my only (?) blocks are minicms/breadcrumb/lastalb,2/catlist/alblist/lastcom.
Website: http://anime-fanservice.org/coppermine/index.php (Not safe for work)
User account: Jets0n
Password: Spac3ly
Thanks in advance.
Do you have detailed statistics turned one? That is well known to create a little extra load so you could try disabling that.
No detailed hit/vote statistics, sorry, though I do have file and album views turned on and turning them off does nothing. The vast majority of the queries are (and hopefully this doesn't technically count as posting debug without being asked, since that file is significantly longer):
=> SELECT filepath, filename, url_prefix, pwidth, pheight FROM cpg15x_pictures WHERE ((aid = 'XXX' ) ) AND approved='YES' ORDER BY RAND() LIMIT 0,1 [index.php:1049] (1 ms)
SELECT filepath, filename, url_prefix, pwidth, pheight FROM cpg15x_pictures WHERE pid='XXXX' [index.php:1055] (0 ms)
SELECT count(pid) AS link_pic_count, max(pid) AS link_last_pid FROM cpg15x_pictures WHERE ((aid != '1213' ) AND (keywords like '%Bradster%' )) AND approved='YES' [index.php:1018] (54 ms)
Turning lastalb,2 off doesn't fix the problem.
First thing you need to do is update to CPG1.5.24
Overall, the update helped a bit in page load time. It's down to about 11 seconds- at least with most of the default settings left intact. I don't know if that speed increase couldn't also be attributed to the settings instead of the update.
Number of queries does seem to vary wildly based on the settings, and I've gotten it down to about 500- though I can't remember exactly which settings were changed to get this number.
I did save my config page to PDF, in case it would be useful for someone to look at it and find the "smoking gun"- though I secretly hope there aren't too many more things I could try, since I'm at what I'd consider to be the minimum of functionality at this point.
Are there any plugins installed?
BBCode Control, FileMove, FileReplacer, MassImport, miniCMS, More Meta Albums, Move 2 Public, MP4 player, User Gallery Alphabetic Tabbing, phpMyAdmin
Since I am not familiar enough with those plugins and since I can't see anything obvious
on the page or in the source, the best I have at the moment is:
- Set debug ON for users, logout, copy the output, and attach it as a .zip file (no promises from me, but it might help)
- Try uninstalling plugins you think that you live without
Zip file attached. I'll start pruning plugins now- since I don't use them constantly maybe I'll just reinstall one when I need it. If it helps.
(I can't demand resolution since it's volunteer help for a free application, but your time is definitely appreciated)
For some reason it looks like it is trying to pull a lot of thumbnails
for the albums in the categories on your home page which are not shown.
However, those thumbnails are shown in the Categories.
Usually that is a setting in Config>> Album list view>> Show first level album thumbnails in categories.
Is there anything in your theme or elsewhere that could affect that setting?
Is the query count the same or high in a category? (ex. anime-fanservice.org/coppermine/index.php?cat=9)
Well, I'm not artistically-inclined enough to modify a default theme, and experimented with the "show first level" settings to make sure they were as low as possible without dropping the albums altogether. To my beginner's eye, the values in the config table appear to correlate with the settings I have on its page.
I haven't done an exhaustive line-by-line comparison, but it looks like all of the aid and pid calls are different. Most of these calls are made by lines 1078 and 1084 of index.php, and the text of both is:
$result = cpg_db_query($sql);
I assume it'd be bad form to copypasta the entire section that contains those lines, since they should appear in everybody's index.php and can be looked at independently.... unless mine has undergone a mysterious mutation which has somehow survived the update and overwriting of all files..
Okay, after an impulse made me look at the SQL table again, I discover that quite a few albums are still using "random" thumbnails (-1) instead of an assigned one or "last uploaded" (0). This is most likely the source of my problem, though I don't understand CM enough to know why it's already calling for thumbnails on the main page when I only display the most recent albums.
This is more than a little irritating, because I know I spent quite a few hours well over a year ago changing all of the thumbnail settings away from random. The only way I can think that this was undone would be something stupid and believable like me manually importing table values from a backup that didn't have the correction in it.
Changing all of these will take some time, and I'll get back to this thread when I've been able to fix them, and any change in load times/queries.
If you're looking at the database and noticing the value of `thumb`, then entering a simple SQL command will reset all the albums
UPDATE `cpg_albums` SET thumb = 0
Substitute your table name if it is not `cpg_albums`.
Thank you, I didn't know how to phrase that command.
Unfortunately it's still trying to query all of those thumbs. I haven't checked all of the pids, but quite a few of them belong to the thumbnail of the last updated albums in each category, but removing lastalb entirely (as a test) doesn't cut the number of queries down, though.
Quote from: Bradster on July 19, 2013, 04:47:32 PM
Thank you, I didn't know how to phrase that command.
Through phpMyAdmin (I use the version my host provides):
Open the db
Migrate to the
cpg_albums table (or whatever the prefix is for your db)
Click SQL near the top of the page (returns window to run SQL queries)
Copy this into the field (remember the prefix)
UPDATE `cpg_albums` SET `thumb`= -1
Click Go
I am not certain that will solve your problem, although it is worth a try.
Things I do now know after trying to reproduce your problem without success include:
It will pull "basic" album thumbs on index, even when they are not called for in Config or shown.
The behavior / count did vary and seemed inconsistent when the value in the thumb column was changed,
but only by small amounts. However, my test gallery is nowhere near the scale of your gallery.
I would recommend making a backup of your db and performing a repair.
Creating a test gallery might help too, along with the other suggestions.
[edit]
Are you using album keywords?
[/]
Unfortunately, replacing the thumb value with "-1" has no effect.
I have the "More Meta Albums" plugin installed, and have keywords for a few dozen albums, but not more than that because I've been too lazy about the project. However, something I noticed in the past is that CM itself assigns my username to all uploaded albums, even other peoples'. There are quite a few image keywords that other users have made for their stuff.
I've been making a test page from scratch, which is why I haven't replied earlier- the Mass Import plugin dies when trying to add the whole site (probably not surprising), so I've been going one letter at a time. I'm currently up to the letter S and can confirm the number of queries is almost back to "normal"- but the page response time is a fraction of what it was. I thought about bulking it up with old information to see if that would return a lot of the load time, but realized that since none of the images or albums have the same id numbers as before, it'd be a nightmarish mess.
http://anime-fanservice.org/coppertest/index.php
Okay, so reducing the number of categories also reduces the number of queries, and increasing the number of albums per page inside the categories increases the queries significantly, regardless of what you choose to show on the main page. That definitely cripples a visitor's navigation of a page this size, since a quick load time on index.php will mean a much larger number of pages within each category. I guess they'll have to deal with it, because there's no way I'm doing another account upgrade to keep things quick, and am not inclined to use a second platform for older material.
Should I just mark this as "solved", since it basically boils down to the gallery size itself? Others might at least get some useful info from what was suggested here, but didn't work for me personally.
Thanks for the help. (Feel free to post whatever final comment strikes your fancy)
Quote from: Bradster on July 22, 2013, 05:13:43 AM
Should I just mark this as "solved", since it basically boils down to the gallery size itself?
In my opinion, no. Although I can't be certain yet until a second opinion with better knowledge than mine is posted.
In the meantime try the mod. below. I am not too happy about it and have only performed limited testing,
(server & time issues) but the results are reduced queries when first level thumbs are turned off.
Backup index.php, then open / edit a copy.
Find:
//unused code {SaWey}
//$visibility = $album['visibility'];
Add directly below it:
if ($CONFIG['first_level'] == 1) {
Find below:
$alb_list[$aid]['thumb_pic'] = '<img src="' . $cpg_privatepic_data['thumb'] . '" ' . $cpg_privatepic_data['whole'] . ' class="image" border="0" alt="" />';
}
Add directly below it:
}
Save and upload.
Please reply if makes any difference.
That makes quite a bit of difference! Queries on index.php are down to about 80, and page generation/load are both under 1s. It does take a little longer to load the category pages now (presumably since they're no longer called on the main page), but that is a much better trade-off since more people start on the main page than they do a category.
Thanks for returning and marking the thread as solved (helpful for future searchers).
But I am curious...
Quote from: Bradster on July 27, 2013, 08:52:58 AM
It does take a little longer to load the category pages now
Can you post link to any particular page(s) with that behavior? And do you know if the
difference is due to increased query count? Or could you take a look at the count and time
differences? Again, my test setup is not as large by any means, so any large scale gallery effects
are impossible for me to test.
I'll be away from my PC for a week and my phone won't be the best test platform for the DOE I have in mind based on your request, so I'll have to put this off for a bit. I'll test 6 categories (a pair from 3 size ranges) with the old and new index.php file, and note the number of queries and generation time. Might also take the average of several refreshes, because I occasionally saw some noticeably different times and should figure out if they're outliers or not.
Joe: I just glanced through the thread, is there anything to fix in the Coppermine code or is this just an individual issue?
Αndré, there might be a need for a change or another tester. :) Here is what I have observed:
In index.php here:
// Inserts a thumbnail if the album contains 1 or more images
Thumbnail queries are run under this condition:
// Inserts a thumbnail if the album contains 1 or more images
Even when the thumbnails are not shown (this is not checked):
$CONFIG['first_level']
The apparent result is a larger number of sql queries than the page needs.
The original poster has reported no negative effects (yet) and in my limited testing gallery there was a decrease in the amount queries.
Since I am still not certain if adding the first_level check will break something else, I hope this can help to answer your question.
Thanks. I'll have a closer look soon.
We don't need to run list_cat_albums if $CONFIG['first_level'] is false. We should replace
// Check if you need to show first level album thumbnails
if ($level <= $CONFIG['subcat_level']) {
with
// Check if you need to show first level album thumbnails
if ($CONFIG['first_level'] && $level <= $CONFIG['subcat_level']) {
as it's the more logical and performant solution IMHO.
Thanks - I suspected something better could be done.
Sorry for not getting back to this thread after returning home and running a few tests. Though it looks like there has been another change to index.php proposed (and possibly rendering all of this moot), here's the outcome of the brief fiddling I did before my time was allocated elsewhere:
https://www.dropbox.com/s/4egymvjevac0j4i/afg%20page%20testing.xlsx
Values are the average of 3 page refresh.
Thanks again for all the help- I'm not sure how many others have sites with an image count that's this size, but I'm sure they'll benefit from this as well. :)
I don't understand that table, at least I don't understand how it's related to your gallery. As far as I can see you don't display any category thumbs in your gallery. Where did you get the "number of category thumbs" in this case?
Sorry, that table was made after I implemented Joe's modification of index.php. I noticed that that page loaded faster, but the category pages loaded slower, and he was curious to know which ones and how much slower.
Quote from: Joe Carver on July 30, 2013, 12:19:32 AM
Can you post link to any particular page(s) with that behavior? And do you know if the
difference is due to increased query count? Or could you take a look at the count and time
differences? Again, my test setup is not as large by any means, so any large scale gallery effects
are impossible for me to test.
I then changed the number of thumbs per category page, going in increments of 10 for S/M/L-sized galleries to see what the time and query differences were for each setting.
I kinda stopped after doing 10 entries of the new index.php 'cos each one is an average of 2-3 galleries, and I didn't dare try this test with the native file out of fear of getting the stink eye from my host.
I assume we're talking about pages like this one? http://anime-fanservice.org/coppermine/index.php?cat=30
Actually, those are album thumbnails, not category thumbnails. If you confirm, I'll check if we can optimize the corresponding code (decrease query amount).
Sorry, yes, they're album thumbnails (from "number of albums to display" under "Album List View" in Config). The smaller this number was, the (relatively) quicker the main page would load under the normal index.php- but I didn't want categories that have over 50 albums to be split into such tiny chunks.
Do you want me to give your code replacement suggestion from 8/30 a try, or is that a simple change that has no chance of affecting other code/config settings?
That code change just affects the category thumbnails. It has no effect on the album thumbnail pages. I'll check if we can improve your issue probably tomorrow.
Open index.php, find
foreach ($alb_stats as $key => $value) {
$cross_ref[$value['aid']] = &$alb_stats[$key];
if ($CONFIG['link_pic_count'] == 1 || $value['pic_count'] == 0) {
if (!empty($value['keyword'])) {
$keyword = ($value['keyword'] ? "AND (keywords like '%".addslashes($value['keyword'])."%' $forbidden_set_string )" : '');
$query = "SELECT count(pid) AS link_pic_count, max(pid) AS link_last_pid "
." FROM {$CONFIG['TABLE_PICTURES']} "
." WHERE ((aid != '{$value['aid']}' $forbidden_set_string) $keyword) $approved";
$result = cpg_db_query($query);
$link_stat = mysql_fetch_assoc($result);
mysql_free_result($result);
$alb_stats[$key]['link_pic_count'] = $link_stat['link_pic_count'];
$alb_stats[$key]['last_pid'] = ($alb_stats[$key]['last_pid'] > $link_stat['link_last_pid']) ? $alb_stats[$key]['last_pid'] : $link_stat['link_last_pid'];
}
}
}
and replace with
$last_pids = array();
$last_pid_data = array();
foreach ($alb_stats as $key => $value) {
$cross_ref[$value['aid']] = &$alb_stats[$key];
if ($CONFIG['link_pic_count'] == 1 || $value['pic_count'] == 0) {
if (!empty($value['keyword'])) {
$keyword = ($value['keyword'] ? "AND (keywords like '%".addslashes($value['keyword'])."%' $forbidden_set_string )" : '');
$query = "SELECT count(pid) AS link_pic_count, max(pid) AS link_last_pid "
." FROM {$CONFIG['TABLE_PICTURES']} "
." WHERE ((aid != '{$value['aid']}' $forbidden_set_string) $keyword) $approved";
$result = cpg_db_query($query);
$link_stat = mysql_fetch_assoc($result);
mysql_free_result($result);
$alb_stats[$key]['link_pic_count'] = $link_stat['link_pic_count'];
$alb_stats[$key]['last_pid'] = ($alb_stats[$key]['last_pid'] > $link_stat['link_last_pid']) ? $alb_stats[$key]['last_pid'] : $link_stat['link_last_pid'];
}
}
if ($alb_stats[$key]['last_pid']) {
$last_pids[] = $alb_stats[$key]['last_pid'];
}
}
if (count($last_pids)) {
$result = cpg_db_query("SELECT pid, filepath, filename, url_prefix, pwidth, pheight FROM {$CONFIG['TABLE_PICTURES']} WHERE pid IN (".implode(',', $last_pids).")");
while ($row = mysql_fetch_assoc($result)) {
$last_pid_data[$row['pid']] = $row;
unset($last_pid_data[$row['pid']]['pid']);
}
mysql_free_result($result);
}
unset($last_pids);
find
$sql = "SELECT filepath, filename, url_prefix, pwidth, pheight "
. "FROM {$CONFIG['TABLE_PICTURES']} WHERE pid='{$alb_stat['last_pid']}'";
$result = cpg_db_query($sql);
$picture = mysql_fetch_assoc($result);
mysql_free_result($result);
and replace with
$picture = $last_pid_data[$alb_stat['last_pid']];
This should reduce the query amount if you use "last uploaded" as album thumbnail. We still need to optimize it for albums that will be displayed through the function list_cat_albums, I'll post it as soon as possible.
Open index.php, find
foreach ($catdata['subalbums'] as $aid => $album) {
$approved = ' AND approved=\'YES\'';
$forbidden_set_string = ((count($FORBIDDEN_SET_DATA) > 0) ? ' AND aid NOT IN (' . implode(', ', $FORBIDDEN_SET_DATA) . ')' : '');
$keyword = ($album['keyword'] ? "AND (keywords like '%".addslashes($album['keyword'])."%' $forbidden_set_string)" : '');
if ($CONFIG['link_pic_count'] == 1 || $album['pic_count'] == 0) {
if (!empty($album['keyword'])) {
$query = "SELECT count(pid) AS link_pic_count, max(pid) AS link_last_pid "
." FROM {$CONFIG['TABLE_PICTURES']} "
." WHERE ((aid != '$aid' $forbidden_set_string) $keyword) $approved";
$result = cpg_db_query($query);
$link_stat = mysql_fetch_assoc($result);
mysql_free_result($result);
$album['link_pic_count'] = $link_stat['link_pic_count'];
$album['last_pid'] = !empty($album['last_pid']) && ($album['last_pid'] > $link_stat['link_last_pid']) ? $album['last_pid'] : $link_stat['link_last_pid'];
}
}
and replace with
$approved = ' AND approved=\'YES\'';
$forbidden_set_string = ((count($FORBIDDEN_SET_DATA) > 0) ? ' AND aid NOT IN (' . implode(', ', $FORBIDDEN_SET_DATA) . ')' : '');
$last_pids = array();
$last_pid_data = array();
foreach ($catdata['subalbums'] as $aid => $album) {
if ($CONFIG['link_pic_count'] == 1 || $album['pic_count'] == 0) {
if (!empty($album['keyword'])) {
$keyword = ($album['keyword'] ? "AND (keywords like '%".addslashes($album['keyword'])."%' $forbidden_set_string)" : '');
$query = "SELECT count(pid) AS link_pic_count, max(pid) AS link_last_pid "
." FROM {$CONFIG['TABLE_PICTURES']} "
." WHERE ((aid != '$aid' $forbidden_set_string) $keyword) $approved";
$result = cpg_db_query($query);
$link_stat = mysql_fetch_assoc($result);
mysql_free_result($result);
$catdata['subalbums'][$aid]['link_pic_count'] = $link_stat['link_pic_count'];
$catdata['subalbums'][$aid]['last_pid'] = !empty($album['last_pid']) && ($album['last_pid'] > $link_stat['link_last_pid']) ? $album['last_pid'] : $link_stat['link_last_pid'];
}
}
if ($catdata['subalbums'][$aid]['last_pid']) {
$last_pids[] = $catdata['subalbums'][$aid]['last_pid'];
}
if ($album['thumb'] > 0) {
$last_pids[] = $album['thumb'];
}
}
if (count($last_pids)) {
$result = cpg_db_query("SELECT pid, filepath, filename, url_prefix, pwidth, pheight FROM {$CONFIG['TABLE_PICTURES']} WHERE pid IN (".implode(',', $last_pids).")");
while ($row = mysql_fetch_assoc($result)) {
$last_pid_data[$row['pid']] = $row;
unset($last_pid_data[$row['pid']]['pid']);
}
mysql_free_result($result);
}
unset($last_pids);
foreach ($catdata['subalbums'] as $aid => $album) {
find
// Inserts a thumbnail if the album contains 1 or more images
if ($album['thumb'] > 0) {
$sql = "SELECT filepath, filename, url_prefix, pwidth, pheight "
." FROM {$CONFIG['TABLE_PICTURES']} WHERE pid='{$album['thumb']}'";
$result = cpg_db_query($sql);
$picture = mysql_fetch_assoc($result);
if (!is_array($picture)) {
$sql = "SELECT filepath, filename, url_prefix, pwidth, pheight "
. "FROM {$CONFIG['TABLE_PICTURES']} WHERE pid='{$album['last_pid']}'";
$result = cpg_db_query($sql);
$picture = mysql_fetch_assoc($result);
}
mysql_free_result($result);
and replace with
if (!empty($last_pid_data[$album['thumb']]['filename'])) {
$picture = $last_pid_data[$album['thumb']];
find
$sql = "SELECT filepath, filename, url_prefix, pwidth, pheight "
. "FROM {$CONFIG['TABLE_PICTURES']} WHERE pid='{$album['last_pid']}'";
$result = cpg_db_query($sql);
$picture = mysql_fetch_assoc($result);
mysql_free_result($result);
and replace with
$picture = $last_pid_data[$album['last_pid']];
As you maybe noticed I also optimized the query amount if you set a particular picture as album thumbnail (that's why we need to replace 3 blocks instead of 2 blocks).
Please report if everything works as expected. To test all changes, please enable "Show first level album thumbnails in categories" in the config, if not already done. Have a look at the query count and if any selected album thumbnail remains. Thank you.
If everything works as expected, I'll commit the changes to our SVN repository.
Sorry for the delay. Browser message is:
Parse error: syntax error, unexpected '}' in /home/content/27/3728527/html/coppermine/index.php on line 1143
line 1143 appears to be } // function list_cat_albums
Sorry, I forgot to copy 1 line. Please add
foreach ($catdata['subalbums'] as $aid => $album) {
after
foreach ($catdata['subalbums'] as $aid => $album) {
if ($CONFIG['link_pic_count'] == 1 || $album['pic_count'] == 0) {
if (!empty($album['keyword'])) {
$keyword = ($album['keyword'] ? "AND (keywords like '%".addslashes($album['keyword'])."%' $forbidden_set_string)" : '');
$query = "SELECT count(pid) AS link_pic_count, max(pid) AS link_last_pid "
." FROM {$CONFIG['TABLE_PICTURES']} "
." WHERE ((aid != '$aid' $forbidden_set_string) $keyword) $approved";
$result = cpg_db_query($query);
$link_stat = mysql_fetch_assoc($result);
mysql_free_result($result);
$catdata['subalbums'][$aid]['link_pic_count'] = $link_stat['link_pic_count'];
$catdata['subalbums'][$aid]['last_pid'] = !empty($album['last_pid']) && ($album['last_pid'] > $link_stat['link_last_pid']) ? $album['last_pid'] : $link_stat['link_last_pid'];
}
}
$last_pids[] = $catdata['subalbums'][$aid]['last_pid'];
if ($album['thumb'] > 0) {
$last_pids[] = $album['thumb'];
}
}
if (count($last_pids)) {
$result = cpg_db_query("SELECT pid, filepath, filename, url_prefix, pwidth, pheight FROM {$CONFIG['TABLE_PICTURES']} WHERE pid IN (".implode(',', $last_pids).")");
while ($row = mysql_fetch_assoc($result)) {
$last_pid_data[$row['pid']] = $row;
unset($last_pid_data[$row['pid']]['pid']);
}
mysql_free_result($result);
}
unset($last_pids);
I'll also update my above instructions accordingly.
Error on line 1164
}
I just tested the above code changes at a different testbed and don't get any parse errors. However, I got an MySQL issue which I just fixed in the above code.
I suggest to undo the changes from my 2 posts at 2013-09-06 and re-apply them.
Hello André,
QuoteI have tested the code before you edited your post today.
The code worked perfectly until I've removed a duplicate file of the gallery, and then, when loading the index.php page displays a critical error in the database.
Now when I put the code after you edit your post today, index.php page is blank.
---Edit---I think the code I had pasted wrongly, :-[
now works perfectly.
Certainly has greatly improved the speed of loading index.php
Regards.
Thanks for confirmation. If Bradster (and maybe also some other people) confirm that the fix works as expected, I'll commit it to our SVN repository and it will be part of the next cpg1.5.x release.
In my case:
Quote
Before: "Page query count 62"
Now: "Page query count 46"
Regards.
Sorry, now it's
Parse error: syntax error, unexpected $end in /home/content/27/3728527/html/coppermine/index.php on line 1262
at the very end of the document. If I'm making a mistake, I'm doing it over and over again without figuring out what it is...
Maybe Niecher or someone else can attach the successfully modified file to his/her reply, as I'm not able to access it before Monday.
Here is the file
Regards.
Thank you Niecher, this doesn't cause errors. After a third or fourth attempt I was finally able to get my version to match, according to diffchecker.com. (Honestly don't know how my Search and Replace was getting botched)
Queries are down to 105, and is the same whether or not "Show first level album thumbnails in categories" is selected- though with almost 900 albums I definitely won't have it selected.
Thanks again for the time and effort on the part of the devs.
What were you using as an editor Bradster? Some tools can mess with code. I recommend the free http://notepad-plus-plus.org/
Phill, I was using EditPad Lite (http://www.editpadlite.com/), and I rarely have problems with it. I'll just have to assume that somewhere along the line I wasn't selecting the full text in the Search/Replace windows when moving from one correction to another. I will also look at your suggested editor- never hurts to have two good and simple ones around.
Quote from: Αndré on September 06, 2013, 11:14:25 AM
If everything works as expected, I'll commit the changes to our SVN repository.
Committed changes in SVN revision 8605.
Well, I saw this change was part of 1.5.26 ... 1.5.26 in general seems to have speeded up my gallery, and Google Webmaster no longer complains (so far) of my gallery being slow, great !
Have you tried enabling CDN on your website and integrating caching. My website was slow as well, but once I activated CDN and integrated Varnish and Memcached, the performance really improved. I also changed from shared hosting of Godaddy to dedicated hosting on Cloudways.