Saturday, June 13, 2015

Integrating WSO2 Products with New Relic

Recently, I had a chance to play around with New Relic [1] and integrate WSO2 products with New Relic. It turns out, that integrating new relic is WSO2 servers is quite straight forward.
  1. Download the newrelic agent as instructed in [2] or in the NewRelic APM Getting Started guide.
  2. Unzip to <CARBON_HOME> directory. The agent should be installed under <CARBON_HOME>/newrelic
  3. Make sure the license key and application name are correct in <CARBON_HOME>/newrelic/newrelic.yml.
  4. Add the following line to <CARBON_HOME>/bin/wso2server.sh java command at the end.   

  5.  -javaagent:$CARBON_HOME/newrelic/newrelic.jar \  
    

    The last part of the wso2server.sh should be as follows.

     while [ "$status" = "$START_EXIT_STATUS" ]  
     do  
       $JAVACMD \  
       -Xbootclasspath/a:"$CARBON_XBOOTCLASSPATH" \  
       -Xms256m -Xmx1024m -XX:MaxPermSize=256m \  
       -XX:+HeapDumpOnOutOfMemoryError \  
       -XX:HeapDumpPath="$CARBON_HOME/repository/logs/heap-dump.hprof" \  
       $JAVA_OPTS \  
       -javaagent:$CARBON_HOME/newrelic/newrelic.jar \  
       -Dcom.sun.management.jmxremote \  
       -classpath "$CARBON_CLASSPATH" \  
       -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \  
       -Djava.io.tmpdir="$CARBON_HOME/tmp" \  
       -Dcatalina.base="$CARBON_HOME/lib/tomcat" \  
       -Dwso2.server.standalone=true \  
       -Dcarbon.registry.root=/ \  
       -Djava.command="$JAVACMD" \  
       -Dcarbon.home="$CARBON_HOME" \  
       -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \  
       -Djava.util.logging.config.file="$CARBON_HOME/repository/conf/etc/logging-bridge.properties" \  
       -Dcarbon.config.dir.path="$CARBON_HOME/repository/conf" \  
       -Dcomponents.repo="$CARBON_HOME/repository/components/plugins" \  
       -Dconf.location="$CARBON_HOME/repository/conf"\  
       -Dcom.atomikos.icatch.file="$CARBON_HOME/lib/transactions.properties" \  
       -Dcom.atomikos.icatch.hide_init_file_path=true \  
       -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true \  
       -Dcom.sun.jndi.ldap.connect.pool.authentication=simple \  
       -Dcom.sun.jndi.ldap.connect.pool.timeout=3000 \  
       -Dorg.terracotta.quartz.skipUpdateCheck=true \  
       -Djava.security.egd=file:/dev/./urandom \  
       -Dfile.encoding=UTF8 \  
       -DapplyPatches \  
       org.wso2.carbon.bootstrap.Bootstrap $*  
       status=$?  
     done  
    

  6. Start the WSO2 server. If the agent was picked up successfully, you should see additional log entries similar to following at startup.

  7.  May 14, 2015 14:17:54 +0530 [1866 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/home/lasantha/Staging/wso2am-1.8.0/newrelic/./newrelic.yml"May 14, 2015 14:17:55 +0530 [1866 1] com.newrelic INFO: New Relic Agent: Writing to log file: /home/lasantha/Staging/wso2am-1.8.0/newrelic/logs/newrelic_agent.log  
    

  8. After server is started up properly, use the server for sometime as you would normally use e.g. send requests to the server, browse the management console etc. You should be able to see statistics when you are logged into your new relic dashboard when you view APM stats under the application name you have given.
New Relic dashboard showing stats about the JVM

Dashboard for viewing application request stats
Now that you've setup your WSO2 product properly with stats being published to New Relic, its time to know about those little caveats and gotchas! New Relic uses JMX to collect statistics from any Java server. However, by default, it listens to stats from a pre-defined set of beans. This will give you stats about request to the management console of a WSO2 server. However, WSO2 products use a different set of ports and publishes stats to a different set of MBeans for traffic related to HTTP/HTTPS NIO transport. These MBeans are not captured by default in New Relic, dashboard. 

But to capture these stats, all you need to do is register an additional set of stats to a new relic custom dashboard. To do that you need to create custom jmx instrumentation when configuring New Relic. The following link [3] can provide more details about custom JMX instrumentation on New Relic. You can find some JMX examples as well in [4].

And for the final link...! A list of JMX MBeans that are available for monitoring HTTP/HTTPS synapse NIO transport is available in [5]


4 comments:

  1. Hi,

    would you be kind enough to share a newrelic.yaml config file with wso2 MBeans set up?

    ReplyDelete
  2. Hi

    Are you 100% sure that WSO2 started correctly after integrating with the NewRelic?
    Which versions are you using? (WSO2, NewRelic)
    I'm getting an error regarding Identity Server's humantask scheduler, that's why I'm asking

    ReplyDelete
  3. I have tried this.But I could not solve this problem. Can you any help me to solve ?

    Error bootstrapping New Relic agent: java.lang.RuntimeException: java.io.IOException: No such file or directory
    java.lang.RuntimeException: java.io.IOException: No such file or directory
    at com.newrelic.bootstrap.BootstrapLoader.load(BootstrapLoader.java:133)
    at com.newrelic.bootstrap.BootstrapAgent.startAgent(BootstrapAgent.java:152)
    at com.newrelic.bootstrap.BootstrapAgent.premain(BootstrapAgent.java:121)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:382)
    at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:397)
    Caused by: java.io.IOException: No such file or directory
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:1006)
    at java.io.File.createTempFile(File.java:1989)
    at java.io.File.createTempFile(File.java:2040)
    at com.newrelic.bootstrap.EmbeddedJarFilesImpl$1.load(EmbeddedJarFilesImpl.java:36)
    at com.newrelic.bootstrap.EmbeddedJarFilesImpl$1.load(EmbeddedJarFilesImpl.java:26)
    at com.newrelic.agent.deps.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.newrelic.agent.deps.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.newrelic.agent.deps.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.newrelic.agent.deps.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.newrelic.agent.deps.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.newrelic.agent.deps.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.newrelic.agent.deps.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.newrelic.bootstrap.EmbeddedJarFilesImpl.getJarFileInAgent(EmbeddedJarFilesImpl.java:66)
    at com.newrelic.bootstrap.BootstrapLoader.addBridgeJarToClassPath(BootstrapLoader.java:83)
    at com.newrelic.bootstrap.BootstrapLoader.load(BootstrapLoader.java:129)
    ... 8 more

    ReplyDelete
  4. @Vivek: Following might be the workaround, if you are using a Carbon 4.4.x product
    https://wso2.org/jira/browse/CARBON-15898

    ReplyDelete