Yikes, I've been hacked! Now what?

[Joachim Müller]

There have been efforts of users who have fallen victim of hacking attacks to share their insight with others. However, you need to understand in the first place that not all hacking attacks are the same: once an attacker has managed to break your site's security, he can do virtually anything. Some hackers may just deface your site (i.e. display an unwanted message or ads on your page), others may abuse to store your site to store content (malware, warez, porn etc.).
There is no saying what the hacker may have done to your page, so I suggest you don't believe in simple recipes from a cooking book that say "delete file X and Y and you will be good". Instead, believe me: there's no saying what the attacker may have done, so you better clean your site thoroughly.

[Joachim Müller]

Scope of this article
I have created this article as a result of the attacks performed by the site owner of cdpuvbhfzz.com that has been discussed in the thread "Someone has Redirected my Site to cdpuvbhfzz.com-What do I do?", and I'll refer to a coppermine-driven gallery and how to clean it, but this thread should give you an idea as well how to clean any hacked site.
I will try to write the instructions for users with intermediate skills - I assume that you have read the coppermine documentation, that you know your way around in your FTP app and have good skills on your Windows-driven client PC. I'm sure that there are easier methods to clean your site for power users - Linux users might want to perform the cleaning directly on the server's shell. Those users probably know what to do anyway and don't need the advice posted here.
If the instructions posted here are over your head you better seek professional help from a computer specialist - consult your webhost for a start in that case, but don't expect them to perform the job for free: it's your task in the first place and you can't expect others to do your job. Your webhost might offer sanitization of the site as a paid extra.

[Joachim Müller]

The simple method
In an ideal world, you'd be making regular backups of everything, so you could just restore your backup to the stage before the hack. You'd then just have to close the hole that the attacker used to get into your site and you'd be good. But I'm aware that this is not an ideal world - you probably don't have a backup. You wouldn't be reading this article if you had one ;-). I understand that - we're all human. But as a result of the disaster with your site getting hacked, you might want to perform regular backups in the future.

[Joachim Müller]

So what do you have to do?
First of all: don't panic! You may be tempted to delete everything that looks funny or causes issues, but this might result in losing precious data. So here's what we'll do: first, create a backup of all files (including the infected ones). Then, create a backup of your database. Next, we'll close the hole the attacker used to get control over your site. Finally, we'll clean all files and unwanted database entries.

[Joachim Müller]

Back up ALL files that reside on your webhost to your client PC
No matter what security hole the attacker has used to break into your site, the payload of the hack (i.e. the files that the attacker has tampered with or created) might reside all over your site - even outside your gallery folder. Therefore, you'll need to make a backup of all of the files that reside on your webspace, even the files that reside outside of your webroot (one level up).
So you better use your FTP app (use a real FTP app, not a lame crutch like a web FTP application or something built into your editor) and start the backup. Depending on the amount of files on your webserver and your connection speed, this may take some time. You better use an FTP app that is capable of re-connecting and continuing the backup even if it gets interrupted. The coppermine dev team recommends FileZilla or Smart FTP (see "Tools recommended by the devs").
As a target for the backup, create a new folder on your desktop PC, e.g. c:\working_copy\ - I'll refer to that folder name in this thread accordingly.
For this article I will asume that your coppermine gallery doesn't reside in your webroot, but within a sub folder named "coppermine", so your gallery URL is http://your_site.tld/coppermine/. Subsequently, your actual coppermine files in your working copy should reside in c:\working_copy\coppermine\. It doesn't matter if this is not the case for you - probably, the folder is named differently or your gallery resides in the web root - I just needed a naming scheme in this article to follow; make the changes accordingly. Please note: I'm not suggesting that you rename your gallery's URL or the folder names of your local copy: keep them as they are. I'll just refer to the folders in this article accordingly.
We will use the folder c:\working_copy\ as a working copy (i.e. we will later modify the files that reside in that folder), so you should make a copy of that folder to another location to keep a forensic copy in case you need it later. That second copy should reside in a safe place - you might even burn it to a CD or DVD. You're welcome to store it inside a ZIP archive. For this article, I'll assume that you have stored this copy in c:\forensic_backup\

[Joachim Müller]

Backup your database
Many users shy away from performing a database backup because they are not used to it. However, it's mandatory that you learn how to perform such backups anyway, so at least now after you have issues with your site being hacked, you'll need to learn it "the hard way".

• phpMyAdmin
  For small to intermediate database sizes, the tool phpMyAdmin can be used to perform a backup. Your webhost may provide you with it in your website control panel. If your webhost doesn't provide it, you can install it for yourself - download it from the phpMyAdmin home page and install it on your webpage as suggested in the documentation that comes with it. Please understand that it is beyond this article to explain how to install phpMyAdmin - google for help if you need it.
  The main benefit of phpMyAdmin is that it is not only a tool to make a database backup, but for all sorts of database manipulation. We will need it later in this article to scan through the database for backdoors the attacker might have left.
• mySqlDumper
  For intermediate and large databases, you might experience time-outs when performing a database dump using phpMyAdmin. That's where mySqlDumper comes into play: it is only performing small steps of the backup, using your browser's refresh function to continue to the next step. The dev team recommends using mySqlDumper to perform backups of large databases.
• Coppermine backup plugin
  The main advantage of that plugin is the ease of install and use (refer to the coppermine documentation how to install a plugin). The main disadvantage is that although you can use it to make backups, you can't use it to restore all coppermine database tables. Another disadvantage is that you will have to use coppermine (which is as of now in a hacked stage) to perform the backup, so you can't be absolutely sure that the result will be functional.

I understand that newbies will find it hard to understand how to perform a backup, and you might find it frustrating being told to learn about additional apps like phpMyAdmin or mySqlDumper instead of going ahead and just sanitizing your gallery. Therefore, skip this step (backing up the database) if you're really busy, but perform it later - after having performed the other steps below.

[Joachim Müller]

Put your gallery into maintenance mode
Go to your gallery's URL, log in as admin, go to coppermine's config, scroll to the bottom and enable within the section "Maintenance settings" the option "Gallery is offline". Save your changes. This is meant to make sure that while we're in the process of sanitizing the files locally your users don't upload additional files that would get lost.
Putting your gallery into maintenance mode will not accomplish anything in terms of sanitizing your gallery - the only benefit will be that the legitimate visitors of your site won't be adding content after having performed the backup that might be lost later. If you can not access coppermine's config to set your gallery offline, skip this step; it's merely cosmetical.

[Joachim Müller]

Get the most recent coppermine release
Go to coppermine's download section (http://sourceforge.net/project/showfiles.php?group_id=89658) and get the most recent stable coppermine release (currently cpg1.4.19). Download it and unzip the file to a temporary folder on your hard drive. For this article, let's assume that you have downloaded and unzipped it to c:\cpg1419\.

[Joachim Müller]

Replace the files in your working copy with the files from the most recent release
As suggested in the upgrading section of the docs, copy the entire content of the most recent release you just downloaded (the content of c:\cpg1419\) over the working copy of your coppermine folder (c:\working_copy\) except the file anycontent.php. If your coppermine gallery resides in the webroot (e.g. your gallery is accessible under the URL http://your_site.tld/) and you have no folders one level above the root folder, you have to copy the content of c:\cpg1419\ into c:\working_copy\. If your gallery URL is http://your_site.tld/coppermine/, you'll have to copy the content of c:\cpg1419\ into c:\working_copy\coppermine\. Make sure to overwrite all files in your working copy (except anycontent.php) with the files from the new release, as explained in the upgrade section of the docs. This is "business as usual", i.e. it is the same as if you were upgrading without the need to sanitize anything. Don't upload the files from your working copy to your webserver yet, we still need to sanitize things.

[Joachim Müller]

Get a diff viewer
For sanitization purposes, we'll need a diff viewer. A diff viewer is a piece of software that will compare folders and files and will display the difference between files. The coppermine dev team recommends the free tool WinMerge (see "Tools recommended by the devs: WinMerge"). Download that tool and install it on your PC.
Linux users should use the free tool Meld.

[Joachim Müller]

Compare the working copy and coppermine's core files
Now this is the step most regular coppermine probably are not familiar with: we'll use the diff viewer to see what files that might be harmful exist in your working copy. For this, you'll need to understand some things first:

• We assume that your webserver is configured properly: it should be set up to parse PHP files based on the extension ".php". Files that have another extension (e.g. ".jpg") should not be parsed.
• There are files that are potentially dangerous because the can actually be executed on the webserver. You'll have to make sure that there will be no such files left of unknown origin. Those files that can actually be executed are: PHP files (extensions php, php4, php3, phtml), Perl files (extensions pl or cgi), Python files (extensions py, pyc, pyd, pyw), ASP files (extensions asp, ascx, aspx). Do not blindly delete those files - some of them are legitimate; your coppermine gallery will stop working if your delete all of those files. You'll have to be aware of those files though, particularly of php files of unknown origin.
• Zip archives or jpeg files are not harmful by themselves on the server, as they can not be executed on the server (at least if the server is configured properly). This being said, it doesn't hurt if a malevolent user manages to upload a file named "I_am_evil.jpg" to your webserver that actually isn't a jpeg image, but just a plain text PHP-file that contains malicious code that he renamed from I_am_evil.php to I_am_evil.jpg on his client before uploading it. Without the corresponding configuration, such a file can not do harm. However, it's a trick hackers frequently use to disguise their payload files from the eye of the legitimate site owner: if they manage to break your site's security by modifying an existing PHP file, they can inject code into that PHP file that uses PHP's include command to actually execute the code within I_am_evil.jpg.
  Let me give you an example: there is a legitimate PHP file http://your_site.tld/coppermine/upload.php - if an attacker manages to manipulate that file and add a code line like this: include('albums/userpics/100023/picture.jpg'); and then manages to upload the malicious file http://your_site.tld/coppermine/albums/userpics/100023/picture.jpg to your server that actually isn't a jpeg file, but a script file in disguise, the payload contained in that file will be executed. If you manage to sanitize the file http://your_site.tld/coppermine/upload.php (e.g. remove the offending include line), the malicious jpeg file can no longer do harm, so it won't hurt if it is still a leftover from the attack. The same trick can be used by attackers to disguise their payload in all other files that might look innocent (like zip files or similar).

Now, let's start our diff viewer (WinMerge) using the shortcut on your desktop or from your start menu. You'll notice that at the very start of the application, there is a dialog that asks you to specify two folders (the two folders we're going to compare). Use the "browse" button and navigate to the folders we want to compare. For the left pane, browse to the folder that contains the clean coppermine archive that you have extracted to c:\cpg1419\ previously. Make sure to choose "Folder selection". For the right pane, select the coppermine folder of your working copy (c:\working_copy\coppermine\), again with "Folder selection" for the file name. Once you set both folders to compare, tick the option "Include subfolders" and hit "OK". Winmerge will then start comparing your folders and after a while (depending on the size of your gallery) come up with a result screen.
On the result screen, click on the "folder" tab to sort the results by folder name, then click one "View" - and disable (untick) the option "Show identical files", since we don't want to display safe files (i.e. files that are identical in your working copy and your fresh coppermine package).


We'll now take care of potentially dangerous files. For this, you'll have two options later:

• First option:
  You could sanitize (i.e. delete) malicious files in your local working copy, which is easy and fast. However, this requires you to have faith in what you did, as sanitizing your actual webspace will then mean deleting the entire content of your webspace (all folders and files on your webspace) and then upload only the known-good stuff from your local working copy
• Second option:
  You just take a note of all files that need to be replaced on your webserver. You then later only use your FTP app to selectively delete the corrupted/infected files. This method requires less skills, but is more time-consuming. The drawback is: if you forget one malicious file it could re-infect your site

Decide for one method now (or later) - anyway, we'll delete the malicious files from our working copy right now, so let's start:

• In coppermine's root folder, only the file anycontent.php should be displayed to be different - let's check it manually to make sure it doesn't contain malicious code: double-click the file in WinMerge's results screen. You will then be sent to the comparison of the individual files (the fresh file from the clean package and the one from your working copy). The differences will be highlighted - scan them visually for anything that looks unusual, like an iframe definition that you never added by yourself or similar. If you think that the file is infected, manually clean it or replace the file in the working_copy folder with a clean file from the download. If you're done with comparing anycontent.php, close the new tab (using the right-click/close on the tab).
  If there are any other files within the gallery root folder that you haven't deliberately added by yourself, delete them. If you're not sure about them, review them manually in the same manner you reviewed anycontent.php.
• In the next step, it's time to sanitize the albums folder, so still within WinMerge's results screen, review all files (there might be lots of them) that are not meant to be there. This is comparatively easy, as we know what kind of files are supposed to reside within the albums folder: basically, images (usually jpg-files) are meant to be there. No PHP files, no HTML files, no JavaScript files (file extension ".js) should reside there. Delete all files that are not supposed to reside within the albums folder.
  Another option to clean the albums folder would be to specifically search it for potentially malicious files - all files that are not supposed to be there. For this, you have to review your settings for allowed file types in coppermine's config: usually, you allow images, movies, documents. Under no circumstances should you allow script files like PHP, and it's not a clever idea to allow HTML or JS files neither (out of the box, coppermine will not allow you to add those potentially malicious file types). This being said, you can use the search feature of your operating system to look for unwanted file types. To do so, start the windows explorer, navigate to the folder c:\working_copy\coppermine\albums\, click on the "Search"-button on the Explorer toolbar and enter into the "search for" field a list of files that you want to find (separated with semi colons), e.g. *.php;*.htm;*.html;*.js;*.pl;*.py and then click on the "search now" button. As a result, you should get some files at least: there should be a flie named index.php both within the albums folder as well as within the userpics folder. Both files come with a vanilla coppermine install, so you should be good if you did as suggested earlier and copied the clean coppermine files over your working copy. To make sure that the files in your working copy are not hacked, you're welcome to open them using a plain text editor (notepad.exe is fine) and review their content. Additionally to the index.php files within the albums and the userpics folder, you'll find a file named index.html within each sub-folder of the userpics directory. Those files get created by coppermine when a user uses http uploads. To make sure that the files are untampered with, you can again use your plain text editor. However, it won't hurt to delete all html and PHP files from within the albums folder if there are simply too many of them to check - this should not hurt your gallery at all.
• For the sub-folder bridge, you should apply the same mechanisms that apply for the coppermine root: as you have replaced all files in your working copy with the files of the fresh package and then chose not to display identical files in WinMerge, there should be no files displayed for the bridge folder. If there still are files displayed, delete them - they are definitely malicious.
• The very same thing that is the case for the bridge folder is true for the docs folder and the sub-folders within it (pics and theme)- there should be no other files within those folders except the ones from the fresh package, so they should be displayed as empty (with the WinMerge option "Show identical files" turned off). If this is not the case (i.e. there are still files in those folders), get rid of them.
• The very same thing is the case for the folder images within coppermine's root folder: the folder (and its sub-folders) should only contain images anyway, and they should be identical both for the working copy as well as the clean coppermine package, so you should not see additional files within that folder. If there still are surplus files within that folder, get rid of them as suggested above.
• The next folder "include" is a slight exception: there should be only two files in it that differ from a clean coppermine package: the files config.inc.php and install.lock are being created during coppermine's initial install and therefore don't exist in the clean package, so they should be displayed in WinMerge. To make sure that they have not been tampered with, double-click on config.inc.php within the include folder from WinMerge's result screen and review the content. This is what you should see then:

  <?php
  // Coppermine configuration file
  
  // MySQL configuration
  $CONFIG['dbserver'] =                         'xxx';        // Your databaseserver
  $CONFIG['dbuser'] =                         'xxx';        // Your mysql username
  $CONFIG['dbpass'] =                         'xxx';                // Your mysql password
  $CONFIG['dbname'] =                         'xxx';        // Your mysql database name
  
  
  // MySQL TABLE NAMES PREFIX
  $CONFIG['TABLE_PREFIX'] =                'xxxx';
  ?>

  (with the actual content, i.e. the stuff displayed as xxx filled with actual db details) and maybe the additional line

  define('SILLY_SAFE_MODE', 1);

  Everything else that maybe exists in that file is malevolent code and should be deleted. Do not delete the entire file, but only clean the contents if there is malicious code within that file.
  The file install.lock should be an empty file - if it contains content, get rid of the content, but don't delete the file.
  If there are other files being displayed except config.inc.php and install.lock, they must be harmful and you should delete them.
• The folder lang contains language files that should be identical both in the working copy as well as the clean coppermine folder, so you should not see any files in WinMerge's result screen. If there are files that are not supposed to be in it, delete them.
• The folder logs out of the box only contains the file log_header.inc.php, which should be identical in your working copy and the clean coppermine folder, so it should not be displayed. If you have enabled logging in coppermine's config, some log files might reside within this folder that might be legitimate. Anyway, it won't hurt to delete them. If you don't want to do this, take a look into the files by double-clicking them and reviewing the contents manually.
• The plugin folder is pretty hard to clean, as there might be legitimate folders with legitimate PHP files in it if you have uploaded/installed plugins. It's beyond the scope of this article to explain each and every plugin that might reside in that folder. If you have the skills, review all files within the plugins folder and all sub-folders within it manually by double-clicking them from WinMerge's results screen and hunting the code for malicious bits, but this might be a hard task for you. If you don't have those skills, go to your gallery in your browser and disable all your plugins from the plugin manager (as suggested in the docs). Then delete all folders and files in the local working copy. Once you're entirely done with cleaning up, re-upload the plugins you installed and then re-enable them using the plugin manager.
  However, it's not very likely that the hacker has infected the individual plugins you have installed, as he can not be sure that those plugins reside on each and every coppermine install the hacker has been attacking.
  Out of the box, only the sample plugin should reside in your plugins folder - if you're not using plugins at all, the plugin folder should be displayed as being empty - delete everything that is not supposed to reside in the plugins folder if you're not sure.
• The folder sql should only contain three files (basic.sql, schema.sql and update.sql) that should be identical both in your working copy and your clean coppermine folder, so nothing should be displayed within WinMerge's results screen. If this is not the case for you (i.e. there are other files than the ones mentioned), you should delete them.
• The themes-folder and it's sub-folders should be identical in your working copy and your clean coppermine folder except for your custom theme (if you're actually using a customized theme). Therefore, only your custom theme's folder and the files within it should be displayed in WinMerge's result screen. You will have to manually review those files (i.e. double-click each of the files from WinMerge's result screen) and inspect those files closely. Review the code carefully - chances are high that the attacker has entered his malicious code into your custom theme. If you have a local copy of your custom theme on your PC, use it to compare the WinMerge result and your clean copy that resided on your hard-drive all the time - preferably using WinMerge as well (by now, you should already be a WinMerge expert and know how to use it ;-)).
  As before, delete everything that looks suspicious.

[Joachim Müller]

Compare custom files outside of coppermine's root folder
You should by now know your way around in WinMerge. If Coppermine is not the only app on your webserver, you will have to scan everything else outside of coppermine in the same way you just scanned the coppermine folder: if you for example use phpbb, it's possible that the attacker has managed to break into your site using a vulnerability in coppermine, but has uploaded the payload (i.e. the malevolent code) not into coppermine folder, but into the phpbb folder to cloak his hack from you. Therefore, you have to sanitize the entire site. If you're using another pre-made script like phpbb or SMF, do exactly the same thing you did for coppermine: download the fresh package of your "other" application, then use WinMerge to compare the working copy (that corresponds to the live-content on your webserver) and the fresh (vanilla) copy of the "other" app.
The same thing is the case for custom content: if you have created "hand-made" content, you have to make sure that it hasn't been corrupted on your server, so you should use WinMerge to compare your local, untampered copy with the working copy that represents your webserver's content.

[Joachim Müller]

After sanitizing the working copy, clean up your actual webspace
So far, we have only cleaned the local copy that represents your webspace. We'll have to upload the sanitized files and get rid of the ones that we deleted locally. That's why I suggested to make a note of what you did before. In case that you don't want to delete the entire content of your webspace and then upload everything from your local hard-drive (maybe because you're not sure that you have done everything as you should or because you haven't understood parts of this article), you can use an alternative: you have already found out about the powers of WinMerge, so let's use it as well to figure out what needs to be uploaded to your server or deleted; for this, you should close WinMerge and re-start it. Then select the working copy (the folder that contains all your edits) for one pane and the backup that you have made of that folder before modifying it (we named it c:\forensic_backup\ in one of the above steps) for the other pane. Again, choose not to display identical files. What you have then in WinMerge's results screen is a list of files that only exist in your working copy, but not on the forensic backup and vice versa, and files that exist in both folder that differ. Files that only exist in the forensic backup folder, but not in your working copy need to be deleted from the webserver as well. Files that differ in both folders need to be uploaded from the working copy to your actual webspace. Now it's time to start your favorite FTP app and perform the needed transaction - deleting the unwanted files, replacing the updated files on your webserver. Make sure that your FTP app is configured to actually replace existing files (some FTP apps by default are set to not replace existing files - you may have to override that default behaviour).
This is the step you need to perform very thoroughly. Don't perform this if you're tired or not concentrating - making a mistake here might result in

• Your server not getting sanitized properly, leaving behind a backdoor that the hacker could use to re-infect your website
• Your website getting dysfunctional: if you delete content that you're not supposed to delete, your website will no longer work as expected

You need to really understand what you're doing.

[Joachim Müller]

Perform the database upgrade
So far, we have done two things: we have upgraded coppermine (as we copied the clean coppermine package over the local working copy) and sanitized the gallery. As suggested in the docs, you need to perform another step to finish the upgrade: you'll have to run the corresponding script in your browser at least once (it won't hurt to run it several times). Going to http://your_site.tld/coppermine/update.php should do the trick.

[Joachim Müller]

Scan the database for admin accounts
The attacker may have left backdoors that allow him to get into your gallery later and doing whatever malevolent things he's up to. This usually happens using two methods: they leave one or more script files on the webserver that allow them to take over control (we should by now have taken care of that by sanitizing all unwanted files in the working copy) and/or they create an admin account in your hacked application that allows them control over the site later. Therefore, you'll have to go to your gallery in your browser, log in as admin and review the user list - you need to make sure that none of the accounts that are not supposed to be admins actually have admin privileges. To accomplish this, click on "Users" from the admin menu (or go to http://your_site.tld/coppermine/usermgr.php directly) and then click on the arrow-up icon (to sort in ascending order) in the "Group" column to sort your user list by groups. All admin accounts (membership in the group "Administrators"). If you see an admin account that you haven't created, delete it.
This can be a time-consuming task for large galleries with a lot of user accounts, so this is where your mySQL database dump that you have created before comes into play; alternatively, you can use phpMyAdmin and browse your members table. What we're looking for are members that are members of the group number 1 - find them and get rid of all illegitimate accounts.

If you are running coppermine bridged to another app this step does not apply, but you should check your forum for added admin accounts just in case.

[Joachim Müller]

Review your config options
Some hacks (like the one that lead to the release of cpg1.4.18) perform changes of coppermine's config, so you'll have to undo those changes. In an ideal world, you would have a backup of your database, so you could just restore the config table. But sadly, you probably don't have that backup, so you'll have to log in as admin and go to coppermine's config. Go through all settings and review them - if you're not sure what a particular setting does, use the help icons or look the setting up in the documentation that comes with coppermine.
The notorious cdpuvbhfzz hack changed the size of the intermediate-sized image to one pixel, so you'll need to restore that to what you had before.

[Joachim Müller]

Review groups permissions
There is a slight chance that the hacker has changed the settings on the groups page as well - as an example, he might have enabled anonymous comments to be able to flood your gallery with comment spam. Although this is not directly related to security in the first place, you should go to your groups page as well and review the settings there.

[Joachim Müller]

Disable offline mode
Once you're done with all the sanitization works, don't forget that your gallery is still in offline mode, so you better go back to coppermine's config and disable offline mode there.

[Joachim Müller]

Alternative sanitization methods
Particularly for the cdpuvbhfzz hack, there are two sanitization scripts available, one as a PHP file and one as a shell script. I don't think that they will work in every case and haven't looked into them, so there are no guarantees.

[Joachim Müller]

Read on
This thread has got a second page. You just reached the end of the first page. The thread is not over yet. Yes, I admit that it's a long thread and that it has been a lot of reading so far, but it won't be much more - I promise.
Click on the tab at the very top or bottom of this page that will send you to page 2 of this thread. Alternatively, click here: go to page two of this thread.