Finding memory leaks in Java

Here is what we found out about finding memory leaks today - we have not fixed our problem yet so there may be another installment in the works :-)

Choose your tool

There are a number of good tools to observe and inspect memory use in a Java app.  We were alerted to our problem via AppDynamics, which is suite for monitoring live apps.  Back on the dev station, you can use VisualVM (free) or YourKit (with a 14 day free trial) to visualize Memory usage, CPU usage, Garbage Collection, etc.

Integration with IntelliJ

We found that YourKit integrates very well: Install the YourKit Java Profiler Integration plugin.  You get a new run option that starts your app and connects it to YourKit.  The alternative is to start the app normally and connect to it, like you would with any remote debugger.

Monitoring Local Server

This is the memory view before we start hitting the server:  Notice how the VM starts up and creates lots of heap objects (left), on the right you see the class meta data (metaspace, in light green) grow.  If you are unsure about Java memory spaces, check out YourKit help page,  and this page about the transition from PermGen to MetaSpace.

Monitoring Remote Server

Testing on a dev station is all nice and well but the real challenge was connecting to a Java app running on a CloudFoundry sandbox.  It turns out that the default Java buildpack has support for a large number of options, one of which is YourKit.

How it works:
The java buildpack has a your-kit-profiler.yml.  This Yaml file disables the profiler by default.  So the trick is to either deploy the app with a custom buildpack, or somehow override the contents of this yaml file.

It turns out that the buildpack has a interesting mechanism for (partially?) overriding the contents of yaml files:  You include a special environment variable in the env section of your manifest.  The name of the variable is 'JBP_CONFIG_' + the name of the yaml file but without '.yml'.  The value of the variable will be the new contents of the yaml file, overriding what was there before.

The only limitation is that you have to squeeze all your values on one line and be careful with quoting rules.  For us, the result looked like this:

applications:- host: HostAway
  memory: 1024M
  env:    SPRING_PROFILES_ACTIVE: sb
    JBP_CONFIG_YOUR_KIT_PROFILER: '{version: "2015.15086.0", repository_root: "{default.repository.root}/your-kit/{platform}/{architecture}", enabled: true, port: "10101", default_session_name: "sandboxdocservice"}'

The one line of yaml contents does not come out so well, it contains:
applications:
    JBP_CONFIG_YOUR_KIT_PROFILER: '{version: "2015.15086.0", repository_root: "{default.repository.root}/your-kit/{platform}/{architecture}", enabled: true, port: "10101", default_session_name: "HostAwayWithYourKit"}'

Bring up your app by pushing it to CF as normal.  To reach it, you will have to create a secure tunnel to your workstation:

cf ssh -N -T -L 10101:localhost:10101 HostAway

Back in YourKit, go to the Welcome screen and choose 'connect to remote application'. Enter 'localhost:10101' and you should be good to go.

Thanks!!

Tons of thanks to Tiffany Chang and Ajay Pillai for working with me on this.