Skip to main content

JVM multi-tenancy with ElastiCat

One of the speaker in last JavaOne Moscow mentioned about JVM multi-tenancy and declare it's availability in Java 9. End of the last year IBM also declare their IBM J9 JVM to host multiple applications. Out of curiosity i was interested on this topics and time to time read a few articles and see presentation. Who hasn't familiar with JVM multi-tenancy yet, JVM multi-tenancy is hosting more then one application on single JVM. Here you will found a very good articles about JVM multi-tanancy. Last Friday i have got an e-mail from Dzone mailing list about ElastiCat, which providing jvm multi tenancy and much more. After spending some time i download the products and deploy simple web service on it and play with JVM. Today i have found a few spare hours after lunch and decided to share my experience with ElastiCat.
ElastiCat, as they declared, provides Multitenancy, Isolation, Density and Elasticity through Tomcat. You can deploy more than one application on single tomcat and share infrastructure resources. ElastiCat provides virtualization layer that isolates applications in Java Virtual Containers, it's means when you are deploying your application in ElastiCat, you are deploying in Java Virtual Containers and you have control on this JVC, you can add heap memory on it or can add any extra resources. Ok lets try some quick test.
1) Download ElastiCat and extract it in any directory, please see the prerequisites first. Currently ElastiCat doesn't support JDK 1.7.
2) Run the servlet container.
3) Develop a quick web service for testing purpose.
4) Deploy the web service war in directory $ELASTICAT_HOME/cloudapps. If you want to deploy in your application host JVM, if can use $ELASTICAT_HOME/webapps
5) ElastiCat also provides a few examples to try, one of the test-app you can reach by http://localhost:8080/test-app
6) Now we can use jirsh shell, jirsh shell - is a command line administrative interface to the Waratek CloudVM for Java. The Waratek jirsh shell is based on the libvirt virsh shell.
7) Connect through terminals ssh -p 2222 super@localhost - use word super for password
8) Run the command list which will show the following output (it may be differ from yours)
JVCID    GROUP    STATUS          NAME                    COMMAND
0        0        Running         dom-0                   platform
1        0        Running         jvc-1                   /examples1.war
2        0        Running         jvc-2                   /examples2.war
3        0        Running         jvc-3                   /test-app.war
4        0        Running         jvc-4                   /test-infras.war
my webservice artifact test-infras.war is deployed in jvc-4, we can check the configuration by running the command dominfo
dominfo 4

JVCID:                          4
JVC-NAME:                       jvc-4
JVC type:                       ServletContainer
JVC command line:               /test-infras.war
Console log file:               /home/xx/ccc/elasticcat/waratek-elasticat-0.9.2/waratek/var/log/javad/elasticat/jvc-4/stdout
JVC status:                     Running
JVC persistence:                true
JVC priority:                   10
JVC elastic group:              0
JVC uptime:                     21 minutes, 3.237 seconds
JVC cpu usage:                  0.0010 GHz-hours (1.570 seconds)
Maximum heap memory:            0 (unlimited)
Allowed elastic memory:         0KiB
Used heap memory:               1.16MiB
Classloader count:              4
Total classes loaded:           2470
Thread maximum limit:           0 (unlimited)
Alive thread count:             1
Alive daemon threads:           1
Peak thread count:              1
Total started threads:          1
Alive thread IDs:               57
Number of host processors:      4
Number of JVC processors:       4
Cpu affinity:                   true, true, true, true,
File descriptor limit:          0 (unlimited)
File descriptor count:          10
File bytes written:             10781 (10.53KiB)
File bytes read:                21281 (20.78KiB)
Socket maximum limit:           0 (unlimited)
Active socket count:            0
Network bytes written:          5696 (5.56KiB)
Network bytes read:             1232 (1.2KiB)
Native library loading is:      Enabled
Virtual root directory:         "/"
9) Lets modify web service to do some memory leaks, perhaps it's very easy way to do OOM errors.
10) Now we are going to fixed the heap memory size for the JVC by running the command setmem
setmem 4 10000 # add about 10 MB heap 
dominfo 4

JVCID:                          4
JVC-NAME:                       jvc-4
JVC type:                       ServletContainer
JVC command line:               /test-infras.war
JVC status:                     Running
JVC persistence:                true
JVC priority:                   10
JVC elastic group:              0
JVC uptime:                     30 minutes, 3.445 seconds
JVC cpu usage:                  0.0009 GHz-hours (1.410 seconds)
Maximum heap memory:            9.77MiB
Allowed elastic memory:         0KiB
Used heap memory:               1.18MiB
Classloader count:              4
Total classes loaded:           2449
11) now we are ready to invoke the web service method to fill up the heap size for getting OOM error, run the web service client a few times
setmem 4 10000 # add about 10 MB heap 
dominfo 4

JVCID:                          4
JVC-NAME:                       jvc-4
JVC type:                       ServletContainer
JVC command line:               /test-infras.war
JVC status:                     Running
JVC persistence:                true
JVC priority:                   10
JVC elastic group:              0
JVC uptime:                     30 minutes, 3.445 seconds
JVC cpu usage:                  0.0009 GHz-hours (1.410 seconds)
Maximum heap memory:            9.77MiB
Allowed elastic memory:         0KiB
Used heap memory:               9.43MiB
Classloader count:              4
Total classes loaded:           2449
12) Check the ElastiCat log file
WARNING: Interceptor for TestService#sayHello has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault
 at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:122)
 at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)
 at org.apache.cxf.phase.PhaseInterceptorChain.resume(PhaseInterceptorChain.java:232)
 at org.apache.cxf.interceptor.OneWayProcessorInterceptor$1.run(OneWayProcessorInterceptor.java:130)
 at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$3.run(AutomaticWorkQueueImpl.java:371)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
 at java.lang.Thread.run(Thread.java)
 at java.lang.Thread.begin(Thread.java)
 at java.lang.Thread.invokeRun(Thread.java)
 at java.lang.reflect.VMReflection.invokeMethod(VMReflection.java)
 at java.lang.reflect.Method$NativeAccessor.invoke(Method.java)
 at java.lang.reflect.MethodNativeAccessorImpl.invoke(MethodNativeAccessorImpl.java)
 at java.lang.reflect.Method.invoke(Method.java)
Caused by: java.lang.OutOfMemoryError
Yes, we already stack with the OOM error on server. At these moment we can check several application deployed in ElastiCat, in my cases others application run well, however i have check only test-app.
Here we can add a more resource (Heap memory) to avoid the OOM error and continue to execute the web service method.
13) Add more 10 MB heap to the 4th JVC
setmem 4 10000
dominfo 4

JVCID:                          4
JVC-NAME:                       jvc-4
JVC type:                       ServletContainer
JVC command line:               /test-infras.war
JVC status:                     Running
JVC persistence:                true
JVC priority:                   10
JVC elastic group:              0
JVC uptime:                     30 minutes, 3.445 seconds
JVC cpu usage:                  0.0009 GHz-hours (1.410 seconds)
Maximum heap memory:            18.65MiB
Allowed elastic memory:         0KiB
Used heap memory:               9.43MiB
Classloader count:              4
Total classes loaded:           2449
Run the web service client and we check the log file, you should find that OOM error clear and the web service continue executing normally.
With Jirsh command line client you also stop, start and resume the JVC, the full list of the command you can find here.
In my experience it's works like a charm, however very hard to say what will be the actual performance in production environment. All the JVC's share the host JVM GC and other resources also, it's hard to say what will be the actual performance in production with high load app. At first glance, bottle neck would be the GC, but for the first step of JVM multi tenancy ElastiCat is good choice.

Comments

Popular posts from this blog

Send e-mail with attachment through OSB

Oracle Service Bus (OSB) contains a good collection of adapter to integrate with any legacy application, including ftp, email, MQ, tuxedo. However e-mail still recognize as a stable protocol to integrate with any application asynchronously. Send e-mail with attachment is a common task of any business process. Inbound e-mail adapter which, integrated with OSB support attachment but outbound adapter doesn't. This post is all about sending attachment though JavaCallout action. There are two ways to handle attachment in OSB: 1) Use JavaCallout action to pass the binary data for further manipulation. It means write down a small java library which will get the attachment and send the e-mail. 2) Use integrated outbound e-mail adapter to send attachment, here you have to add a custom variable named attachment and assign the binary data to the body of the attachment variable. First option is very common and easy to implement through javax.mail api, however a much more developer manage t

Tip: SQL client for Apache Ignite cache

A new SQL client configuration described in  The Apache Ignite book . If it got you interested, check out the rest of the book for more helpful information. Apache Ignite provides SQL queries execution on the caches, SQL syntax is an ANSI-99 compliant. Therefore, you can execute SQL queries against any caches from any SQL client which supports JDBC thin client. This section is for those, who feels comfortable with SQL rather than execute a bunch of code to retrieve data from the cache. Apache Ignite out of the box shipped with JDBC driver that allows you to connect to Ignite caches and retrieve distributed data from the cache using standard SQL queries. Rest of the section of this chapter will describe how to connect SQL IDE (Integrated Development Environment) to Ignite cache and executes some SQL queries to play with the data. SQL IDE or SQL editor can simplify the development process and allow you to get productive much quicker. Most database vendors have their own front-en

Load balancing and fail over with scheduler

Every programmer at least develop one Scheduler or Job in their life time of programming. Nowadays writing or developing scheduler to get you job done is very simple, but when you are thinking about high availability or load balancing your scheduler or job it getting some tricky. Even more when you have a few instance of your scheduler but only one can be run at a time also need some tricks to done. A long time ago i used some data base table lock to achieved such a functionality as leader election. Around 2010 when Zookeeper comes into play, i always preferred to use Zookeeper to bring high availability and scalability. For using Zookeeper you have to need Zookeeper cluster with minimum 3 nodes and maintain the cluster. Our new customer denied to use such a open source product in their environment and i was definitely need to find something alternative. Definitely Quartz was the next choose. Quartz makes developing scheduler easy and simple. Quartz clustering feature brings the HA and