Chan Chen Coding...

JProfile or YourKit

Finding a memory leak can be difficult, even when you've got some good tools to give you an idea what's happening in your heap. If you can narrow down the operations that are leak suspects that goes a long way to finding the problem.

However, if you aren't sure where to start, your best bet is to find a good profiler that will let you walk the heap. My favorites are JProfiler and YourKit, though some of the others people here have suggested are decent as well. They both cost money, but getting a trial license isn't terribly difficult.

Once in there you'll want to look for what objects are taking up the most heap size. Unfortunately, it's almost always char[], byte[], and String taking up the most space, so you can walk the heap to find what objects are holding onto the largest instances of those objects. Also, the profiler should have the ability to show you the deep size of an object, so be on the lookout for instances of your own classes that have fairly large deep sizes.

Another technique is if you have a suspect operation, is to see if the heap is growing as you perform the operation. The profiler will have the ability to force a garbage collection (you can also do this through jconsole), so perform this operation before and after your operation and see if the heap has grown in size.

Sometimes the leak could be undetectable by these means though since you might see totally random results. A few months ago, I was looking at a leak that was caused by someone synchronizing the finalize method of an object which halted garbage collection. Another time, the leak wasn't a leak, but rather just a certain request was throwing out of memory because even though the heap was 50% free, there wasn't a large enough contiguous block for that request. So it's a lot of trail and error, and the best approach is to try to form a hypothesis and prove it to be true or false as quickly as you can.