Very large cpg install, suggestions, patches

[dleary]

I mean this post to be mostly targeted to the dev team, but I couldn't figure out where to post it, so I put it here...


I work at IMVU, http://www.imvu.com/

We have a very large cpg install, with 350,000 albums and just under 3,000,000 pictures.  I'm wondering, is this the largest coppermine install you guys have heard of?

Imvu homepages are a sort of myspace-like thing...  Everyone who wants one gets a 'gallery', which is created on demand, so some are nearly empty, and some get a lot of use.  Each 'user gallery' is currently just 1 cpg album, but my latest changes give each user their own category, so they can have multiple albums.

The current version on the site is cpg131, and it's not a very good integration.  I just finished the work for cpg144, and it will be pushed live sometime early next week, probably.  I have some suggestions, and I can generate some patches for the cpg dev team if you want.

I don't mind making up the patches, but maybe some of this work is already slated or solved in your cpgNG, I don't know.

To integrate with our site, here are the big changes I made:

• I put all of the DB functionality through commondb calls...  No more "mysql_num_rows" etc, now it's "cpg_db_num_rows".  I then made the cpg_db_* calls use our own DB layer, but I can send a patch which still uses mysql, if you like.
• I removed all the points in the code where you could get a list of all albums, categories, etc...  This is too painful to our DB.  I briefly tried to figure out if I could do this with a plugin, but I ended up just tweaking the "real" files directly.
• There are a couple of places where a query is built that ends up "AND aid NOT IN(1,2,3,4,5,6)", etc...  This could generate a huge query for us, and it was not amenable to our caching.  I removed these for now (which means we don't honor album visibility)...  I will address this with a better solution at some point.
• We converted all the tables from MyISAM to InnoDB (necessary for reliable replication), which also meant removing the fulltext index on cpg_pictures...  This is not a problem to us, because we have taken most searching out of the site anyway.  90% of our slow query logs end up being some search that wasn't properly constrained or indexed doing a table scan.  For our searching, we try to group related areas under a different domain (forums.imvu.com, images.imvu.com, etc), and make it spiderable.  This way, when we need to turn off our internal searching, we can just put up a google site search bar, and let them do the work

I would like to make modifications to our site, or to the CPG mainline, such that we can keep most of our changes in the theme and plugin.  This way we can easily keep abreast of new CPG developments.  If you have any advice on that (particularly helpful would be better ways to accomplish what I have above), I would love to hear it.

If you are interested in any of the changes I have outlined, I would be happy to prepare a patch...

I think the most useful change for CPG to do would be to consolidate all the DB stuff, and switch to using adodb for your data layer.  Not necessarily to give you the ability to deploy on other DBs, but to take advantage of the caching.
On a big install such as ours (imvu itself has recently passed 1,000,000 users, we have more than 50 servers, etc), caching data is a huge deal.  We use a modified adodb, which uses memcached for caching instead of the filesystem, and a system for grouping queries by "cache-class" and flushing them all at the same time.


Finally:  I feel a little bad whoring a job offering, but if my manager saw that I didn't I would get beaten .  So, if you're a good programmer who wants to come work in Silicon Valley, contact me so I can start the interview process.

[Joachim Müller]

We have a very large cpg install, with 350,000 albums and just under 3,000,000 pictures.  I'm wondering, is this the largest coppermine install you guys have heard of?

yes, it is. I'm impressed.

The current version on the site is cpg131, and it's not a very good integration.

I agree - the design you're being sent to differs completely from the overall look of the rest of your site.

I don't mind making up the patches, but maybe some of this work is already slated or solved in your cpgNG, I don't know.

cpgNG is miles away from being production-ready.

I think the most useful change for CPG to do would be to consolidate all the DB stuff, and switch to using adodb for your data layer.  Not necessarily to give you the ability to deploy on other DBs, but to take advantage of the caching.
On a big install such as ours (imvu itself has recently passed 1,000,000 users, we have more than 50 servers, etc), caching data is a huge deal.  We use a modified adodb, which uses memcached for caching instead of the filesystem, and a system for grouping queries by "cache-class" and flushing them all at the same time.

I can see the need to change the database patterns for such a huge install as yours, yet I can't see the benefit for the majority of coppermine users. Your changes sound interessting, and we have indeed discussed forking coppermine at a later stage for low-level use and for huge installs as yours, as the needs differ. The low-profile coppermine users need ease of modification, there's no additional benefit for them in database abstraction by adding another layer.

Finally:  I feel a little bad whoring a job offering, but if my manager saw that I didn't I would get beaten .  So, if you're a good programmer who wants to come work in Silicon Valley, contact me so I can start the interview process.

sounds promising, but probably won't apply for most coppermine devs, as only a few of them are US-residents ;-)


I have a bug report to make: I went through registration and then clicked on "Upload your first image" in the "my gallery" section, which led me to the coppermine page, where I was told that I wasn't logged in and therefor not able to perform that action. When clicking on "login" on that page and entering my login data on the subsequent page, I was sent to http://www.imvu.com/catalog/gallery.php?cat=427778 with the error message

Can only create user directory on home server AF001007.imvu.com this is AF001057.imvu.com:: gallery.php:20 (require) / init.inc.php:335 (require) / theme.php:28 (ensure_gallery_created) / galleries.php:77 (create_user_directory) PHP_SELF = /catalog/gallery.php

You appear to have removed the "Powered by Coppermine" tag at the bottom of the page, which is mandatory to have, so please restore it.

Joachim

[Stramm]

• There are a couple of places where a query is built that ends up "AND aid NOT IN(1,2,3,4,5,6)", etc...  This could generate a huge query for us, and it was not amenable to our caching.  I removed these for now (which means we don't honor album visibility)...  I will address this with a better solution at some point.

just turn off in config
Files and thumbnails advanced settings -> Albums can be private (Note: if you switch from 'yes' to 'no' any current private albums will become public)

[dleary]

I have a bug report to make: I went through registration and then clicked on "Upload your first image" in the "my gallery" section, which led me to the coppermine page, where I was told that I wasn't logged in and therefor not able to perform that action.

Yeah, that's a side effect of our current 'cpg limbo'.  I have the codebase for both cpg131 and cpg144 on the site, with cpg131 working right now, but new albums cannot be created.  (This was supposed to only be for a day or so, but it's taking longer than expected to get our new DB server online (which is when we'll migrate the data).

You appear to have removed the "Powered by Coppermine" tag at the bottom of the page, which is mandatory to have, so please restore it.

I don't know when that happened...  it used to be there, and it's there again in the cpg144 integration.