[Rcpp-devel] RInside Using JNI

[Kota, Ravindra B]

Hi,

We have financial models written in C++, exposed to Java applications using the JNI. Lately, we have started writing these models in R, and trying to integrate them in C++ using RInside. Standalone integration with C++/RInside/R works fine. We get seg faults when integrated with JNI.

Questions.


1. Are there any limitations integrating RInside inside the JVM?

2. Do we have any best practices for this.?

3. Do we have tools available to debug this better?

We have narrowed down to the source of the seg fault. Both 'e' and 'rho' are invalid references.

/* Return value of "e" evaluated in "rho". */

/* some places, e.g. deparse2buff, call this with a promise and rho = NULL */
SEXP eval(SEXP e, SEXP rho)
{
SEXP op, tmp;
static int evalcount = 0;

R_Visible = TRUE;

/* this is needed even for self-evaluating objects or something like
'while (TRUE) NULL' will not be interruptable */
if (++evalcount > 1000) { /* was 100 before 2.8.0 */
R_CheckUserInterrupt();
#ifndef IMMEDIATE_FINALIZERS
/* finalizers are run here since this should only be called at
points where running arbitrary code should be safe */
R_RunPendingFinalizers();
#endif
evalcount = 0 ;
}

/* handle self-evaluating objects with minimal overhead */
switch (TYPEOF(e)) {
case NILSXP:
case LISTSXP:
case LGLSXP:
case INTSXP:
case REALSXP:
case STRSXP:
case CPLXSXP:
case RAWSXP:
case S4SXP:
case SPECIALSXP:
case BUILTINSXP:
case ENVSXP:
case CLOSXP:
case VECSXP:
case EXTPTRSXP:
case WEAKREFSXP:
case EXPRSXP:


#0 0x00002b1a1cd9866f in bcEval (body=0x0, rho=0x0, useCache=FALSE) at eval.c:5873
#1 0x00002b1a1cd8bc11 in Rf_eval (e=0x2b1a15190b38, rho=0x2b1a21180a28) at eval.c:624
#2 0x00002b1a1cd8b944 in forcePromise (e=0x2b1a21180a60) at eval.c:520
#3 0x00002b1a1cd950b7 in FORCE_PROMISE (value=0x2b1a21180a60, symbol=0x2b1a151751a8, rho=0x2b1a21180b40, keepmiss=FALSE)
at eval.c:4714
#4 0x00002b1a1cd95285 in getvar (symbol=0x2b1a151751a8, rho=0x2b1a21180b40, dd=FALSE, keepmiss=FALSE, vcache=0x2b1a14c57f50,
sidx=3) at eval.c:4756
#5 0x00002b1a1cd9ac1e in bcEval (body=0x2b1a151764f0, rho=0x2b1a21180b40, useCache=TRUE) at eval.c:6189
#6 0x00002b1a1cd8bc11 in Rf_eval (e=0x2b1a151764f0, rho=0x2b1a21180b40) at eval.c:624
#7 0x00002b1a1cd8e2e2 in R_execClosure (call=0x2b1a151910e8, newrho=0x2b1a21180b40, sysparent=0x2b1a21180a28, rho=0x2b1a21180a28,
arglist=0x2b1a21180a98, op=0x2b1a15176608) at eval.c:1614
#8 0x00002b1a1cd8e000 in Rf_applyClosure (call=0x2b1a151910e8, op=0x2b1a15176608, arglist=0x2b1a21180a98, rho=0x2b1a21180a28,
suppliedvars=0x2b1a14957188) at eval.c:1549
#9 0x00002b1a1cd9d2d1 in bcEval (body=0x2b1a1518eaf0, rho=0x2b1a21180a28, useCache=TRUE) at eval.c:6400
#10 0x00002b1a1cd8bc11 in Rf_eval (e=0x2b1a1518eaf0, rho=0x2b1a21180a28) at eval.c:624
#11 0x00002b1a1cd8e2e2 in R_execClosure (call=0x2b1a179b4100, newrho=0x2b1a21180a28, sysparent=0x2b1a211812c8, rho=0x2b1a211812c8,
arglist=0x2b1a211816f0, op=0x2b1a1518ebd0) at eval.c:1614
#12 0x00002b1a1cd8e000 in Rf_applyClosure (call=0x2b1a179b4100, op=0x2b1a1518ebd0, arglist=0x2b1a211816f0, rho=0x2b1a211812c8,
suppliedvars=0x2b1a14957188) at eval.c:1549
#13 0x00002b1a1cd9d2d1 in bcEval (body=0x2b1a179b3fb0, rho=0x2b1a211812c8, useCache=TRUE) at eval.c:6400
#14 0x00002b1a1cd8bc11 in Rf_eval (e=0x2b1a179b3fb0, rho=0x2b1a211812c8) at eval.c:624
#15 0x00002b1a1cd8b944 in forcePromise (e=0x2b1a211814f8) at eval.c:520
#16 0x00002b1a1cd950b7 in FORCE_PROMISE (value=0x2b1a211814f8, symbol=0x2b1a149af960, rho=0x2b1a21181610, keepmiss=FALSE)
at eval.c:4714


Help is greatly appreciated. You can reach me at 571-340-1578 for any kind of questions.

Regards
Ravindra

[Dirk Eddelbuettel]

Ravindra,

Thanks for posting here.

On 15 August 2017 at 08:45, Kota, Ravindra B wrote:
| We have financial models written in C++, exposed to Java applications using the JNI. Lately, we have started writing these models in R, and trying to integrate them in C++ using RInside. Standalone integration with C++/RInside/R works fine. We get seg faults when integrated with JNI.

Can you go multi-machine? Have one engine deal with C++ models and their R
integration. Have another engine deal with the Java UI and integration.

There are by now multiple ways to 'request' R answers over tcp/ip, starting
from the good old Rserve by Simon (initially written with a Java client, no
less) to Jeroen's opencpu and more. I actually like Google's Protocol
Buffers (package RProtoBuf) and the by-now added gRPC networking.

Methink you have less of an Rcpp / RInside question and more of an overall
architecting question. For which this list may be less relevant.

As for your questions:

| 1. Are there any limitations integrating RInside inside the JVM?

I prefer to stay as far from JVM and CLI and all that as I can. But maybe
that's just me.

| 2. Do we have any best practices for this.?

See above. Separation of concern?

| 3. Do we have tools available to debug this better?

Not really.

Cheers, Dirk

[Kota, Ravindra B]

Dirk,

Thanks for the reply. It was a pleasure speaking with you yesterday, over the phone.

Now, all our front end and web service applications are integrated with JNI with C++ Quant frameworks behind it. Going multi machine, would be a significant architectural change. This might not be feasible in the short run.

A couple of options to consider.

1. We would really like to know, why the seg faults are happening, before we give up this path. Rcpp/RInside protocol fits very well within our framework. It will be very sad to see if this option does not work.
1.1 We will be working on debugging this issue today.
2. Client Server Design:
2.1 We run large portfolios using monte carlo simulations with about 150-300 paths. This means at any given point of time we would be running 1000 cores in parallel, all running an R model. Does Rserve/opencpu/gRPC scale to this level?
2.2. If we do go down this path, our C++ quant library needs to be the client. Currently we are using Rcpp template data types to go back and forth between R. Which client server option is close to this model, so that we leverage the existing Rcpp investment.
3.What is your availability, if we need to bring you down to Northern Virginia for a day?


Regards
Ravindra

-----Original Message-----
From: Dirk Eddelbuettel [mailto:***@gmail.com] On Behalf Of Dirk Eddelbuettel
Sent: Tuesday, August 15, 2017 7:53 AM
To: Kota, Ravindra B <***@freddiemac.com>
Cc: rcpp-***@lists.r-forge.r-project.org
Subject: Re: [Rcpp-devel] RInside Using JNI

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


Ravindra,

Thanks for posting here.

On 15 August 2017 at 08:45, Kota, Ravindra B wrote:
| We have financial models written in C++, exposed to Java applications using the JNI. Lately, we have started writing these models in R, and trying to integrate them in C++ using RInside. Standalone integration with C++/RInside/R works fine. We get seg faults when integrated with JNI.

Can you go multi-machine? Have one engine deal with C++ models and their R integration. Have another engine deal with the Java UI and integration.

There are by now multiple ways to 'request' R answers over tcp/ip, starting from the good old Rserve by Simon (initially written with a Java client, no
less) to Jeroen's opencpu and more. I actually like Google's Protocol Buffers (package RProtoBuf) and the by-now added gRPC networking.

Methink you have less of an Rcpp / RInside question and more of an overall architecting question. For which this list may be less relevant.

As for your questions:

| 1. Are there any limitations integrating RInside inside the JVM?

I prefer to stay as far from JVM and CLI and all that as I can. But maybe that's just me.

| 2. Do we have any best practices for this.?

See above. Separation of concern?

| 3. Do we have tools available to debug this better?

Not really.

Cheers, Dirk

--
http://dirk.eddelbuettel.com | @eddelbuettel | ***@debian.org

[Dirk Eddelbuettel]

On 15 August 2017 at 12:17, Kota, Ravindra B wrote:
| Thanks for the reply. It was a pleasure speaking with you yesterday, over the phone.

As you now bring this up in public, I have no choice but to be VERY clear about this:

- I consider it totally inappropriate for you to cold call me at work.

- Please do not do this again. Please do not recommend it either.

- I do all this as a volunteer. Please respect that.

Dirk

[Kota, Ravindra B]

Dirk,

After compiling R with "--enable-java" , I am not getting any seg faults so far. Do we know what that option does to the JVM?

Regards
Ravindra

-----Original Message-----
From: Dirk Eddelbuettel [mailto:***@gmail.com] On Behalf Of Dirk Eddelbuettel
Sent: Tuesday, August 15, 2017 8:38 AM
To: Kota, Ravindra B <***@freddiemac.com>
Cc: Dirk Eddelbuettel <***@debian.org>; rcpp-***@lists.r-forge.r-project.org
Subject: RE: [Rcpp-devel] RInside Using JNI

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


On 15 August 2017 at 12:17, Kota, Ravindra B wrote:
| Thanks for the reply. It was a pleasure speaking with you yesterday, over the phone.

As you now bring this up in public, I have no choice but to be VERY clear about this:

- I consider it totally inappropriate for you to cold call me at work.

- Please do not do this again. Please do not recommend it either.

- I do all this as a volunteer. Please respect that.

Dirk

--
http://dirk.eddelbuettel.com | @eddelbuettel | ***@debian.org

[Dirk Eddelbuettel]

On 17 August 2017 at 11:14, Kota, Ravindra B wrote:
| After compiling R with "--enable-java" , I am not getting any seg faults so far. Do we know what that option does to the JVM?

AFAIK nothing, but it sets up the compilation options for R in order to allow
for R to be extensive via Rjava etc -- ie the effect of R CMD javareconf.

Again, here we deal chiefly with Rcpp (and sometimes with RInside) neither
one of which aims at java, so if you have troubles on the Java side you
probably have to address them there.

Hope this helps, Dirk