Lets say you hacked Drupal core and now there's a new release. You want to upgrade, but you don't want to overwrite all your changes. Here's what I would do.
The basic idea is to first generate a patch file of *just your hacks* to core. Then upgrade Drupal to the latest version and reapply your hacks using the patch file.
Tools: I'm using some UNIX-style command-line tools. This means you'll need to pull up the Terminal on OS X, the command line on Linux/Unix, or install something like Cygwin on Windows.
We've got a hacked version of Drupal installed, but how do we extract just our changes?
Easy! First, get a stock (unmodified) version of Drupal in the same version as you have installed. In my case, 5.12.
$ mkdir ~/drupal_upgrade
$ cd ~/drupal_upgrade
$ wget http://ftp.drupal.org/files/projects/drupal-5.12.tar.gz
$ tar xfvz drupal-5.12.tar.gz
This creates a new directory, downloads and untar's the stock (unmodified) version of Drupal 5.12 from drupal.org.
Now we use the 'diff' utility to create a patch file of just our hacks. Note that the hacked version of 5.12 is in /var/www/drupal
$ diff -ru drupal-5.12 /var/www/drupal > our_hacks_to_512.patch
Now we've got a patch file of just our hacks. Incidentally, you can open this .patch file in a text editor to see exactly what you've changed,
Next step: grab a fresh copy of the *current* version of Drupal (5.19 for Drupal 5, as of this writing). and apply our changes to that
$ wget http://ftp.drupal.org/files/projects/drupal-5.19.tar.gz
$ tar xfvz drupal-5.19.tar.gz
Now apply our core hacks to the new version of drupal.
$ cd drupal-5.19
$ patch -p1 < ../our_hacks_to_512.patch
Now we've got a copy of Drupal 5.19 with our core hacks installed!
If you get any errors while applying, you can usually just hit enter. If there were any problems applying the hacks, there will be files named *.rej that you can inspect to see which changes it was unable to apply (perhaps there's a bugfix in Drupal core that you already hacked core to fix). If it all looks good, copy it over to upgrade the real site.
$ cp -rf . /var/www/drupal
At this point I typically inspect the changes by hand and thoroughly test them. If you're running Subversion for version control, you can type "svn diff" in the drupal directory to see exactly what has changed before committing it. Hacking core is dangerous business and it's entirely possible that something could have changed in core that breaks one of your hacks.