Re: Too many convert.exe at the same time use too much memory

[padde0711]

Hi there,

In my opinion, this is a critical bug!

Every user who is allowed to do batch uploads is able to accomplish a serious DoS attack against the server where coppermine is running on.

From many posts in this forum, I can see that it's not only me, who's concerned about this.

Another thing is, that most of the boxes coppermine is running on, are single CPU boxes anyway, and the parallelisation of the convert processes will slow down the conversion considerably (task switching overhead).
As soon as the box has to use virtual (swap) memory, because of the large amounts of memory each convert instance is eating up, the performance will go down to nearly 0, and it will take forever to complete the conversion.
If the virtual memory isn't enough (in my case, 1.5 GB virtual memory + 256 MB of real memory weren't, for parallely converting 30 high-res pictures (=60 convert processes)), the kernel (at least in Linux, that is) will start to kill random processes, for example mysqld, and other important things.

Regards, Padde.

[Nibbler]

Only admins should have batch add rights, if you fear a dos attack from your own admins then you have other issues to deal with. 256MB is not enough memory for a LAMP webserver imo, but i am no server admin so feel free to ignore me. If you have large pics to resize then either do so on your own PC before you upload or use the serial batch adding mod as already suggested.

Please don't use the 1.2 support board unless you are actually using Coppermine 1.2

[padde0711]

Umm, sorry that I posted to the wrong sub-section. Could perhaps a moderator cut off this part and make it a new posting in an appropriate section, please?

If you have large pics to resize then either do so on your own PC before you upload or use the serial batch adding mod as already suggested.
There is NO PROBLEM with resizing one image after another on the server, the machine is perfectly capable of that. There also wouldn't be a problem, resizing 3 at once. But not the whole batch!
What exactly is the reason that you made the conversion parallel? I can't think of ANY.

256MB is not enough memory for a LAMP webserver imo
Well, as I stated: It would even have happened with 1 GB of RAM. As soon as massive swapping occurs, the machine stands still.

Only admins should have batch add rights, if you fear a dos attack from your own admins then you have other issues to deal with.
There is something like rented servers, where clients run instances of coppermine, and some of those neither know nor care about system resources.

To make that clear: I really find the batch-upload feature very useful, and I'm using it a lot. Just that it eats up all resources at once (with no apparent reason) is hard to understand. And it's even harder to understand that I would need a modification (as suggested in another thread) to make coppermine act sanely on this behaviour.

[Nibbler]

I can't move your post if you don't post what version you are using.

They are done in parallel because doing them in serial would take too long and cause the script to timeout when it hits the default 30 seconds execution limit. Batch add is 'sane' for the majority of users, and we of course welcome code contributions.

[padde0711]

I'm using coppermine 1.4.5 here.

[padde0711]

Thanks for moving it to the right place!

Well, one very simple idea I have, is to use a dirty wrapper for convert, which checks if convert is already running, and if so, wait.

This wouldn't need any changes except search&replace of "convert" to "cpg_convert_wrapper.sh". Of course the wrapper could be done in PHP or what not, and I'm 100% sure it could be done in a better way, as I'm not much of a programmer But you'll get the idea:

Code Select Expand

#!/bin/sh

# Filename: cpg_convert_wrapper.sh

# A wrapper for convert which checks if convert is already running.
# Only if that's not the case, convert will be run with the original arguments,
# otherwise the script will wait in a loop, until there is no instance of convert
# running anymore.

while ((DONE==0)); do
        # Search for convert in the process list. If one or more instances
        # are already running, sleep and proceed to beginning of this loop.
        pgrep "^convert" >/dev/null && sleep 0.1 && continue

        # In case there was no convert running, do it.
        convert "$@"

        # Tell the loop that we're done
        DONE=1
done

Regards, Padde.

[Nibbler]

Solutions to this problem cannot be platform specific, that's why it's not a trivial problem to solve.

[padde0711]

Well, then how about a function that's being called instead of calling convert directly?

That function would take care of how many instances are run at once. Use a global variable or something, which is being incremented when convert is being started, and decremented after it's finished. Check that variable, and if it's >$number_of_desired_parallel_converts, then wait and check again later.

[Joachim Müller]

In my opinion, this is a critical bug!

Missing feature imo (if at all), not a bug.
If it were critical, webhosts would complain and ban coppermine from their servers. Instead, they often bundle it with their webhosting packages.

[padde0711]

Sure. That's why you wrote this in a sticky thread in this forum:

"[...] it very resource hungry, and some server administrators have deemed it 'insane'.  If your server administrator requests that you stop batch adding, we recommend that you consider using the XP Publish utility, multiple HTTP uploads (if suitable for your intent), or that you get a new host who can handle your needs."

I still say: It's a bug that needs to be fixed. My opinion.

Regards, Padde.