[Rcpp-devel] R.e. loadRcppModules

[Christian Gunning]

On Mon, Apr 11, 2011 at 3:00 AM,

This will load the declared modules, and populate them into your
package's namespace.
I will update the Rcpp-modules vignette accordingly.
Suggestions, comments, improvements, "what are you  ? nuts !", ... are
always welcome.

Works great for me -- very cool.

One question that's been bothering me -- how are module-based objects
*supposed* to behave across an R session quit/save/restart cycle? I'm
seeing no persistence, and a consistent segfault. Am I missing
something here (perhaps wrt finalizer)? If not, I'd like to add a
note to the vignette...

Using the supplied example, I get:

Rcpp.package.skeleton('testpkg1', module=T)
require(testpkg1)
modtest <- new(World)
## everything works
show(modtest)
modtest$set('hello')
modtest$greet()
q()
## Save workspace image? [y/n/c]: y
## exit, save, restart -- nothing works

# [Previously saved workspace restored]
#
# During startup - Warning message:
# Class "C++Object" is defined (with package slot ‘Rcpp’) but no
metadata object found to revise subclass information---not exported?
Making a copy in package ‘.GlobalEnv’

require(testpkg1)
modtest$greet()
#Error in .External(list(name = "CppMethod__invoke_notvoid", address =
<pointer: (nil)>, :
# NULL value passed as symbol address

show(modtest)

*** caught segfault ***
address 0x8, cause 'memory not mapped'

Traceback:
1: .Call(Class__name, cppclass)
2: sprintf("C++ object <%s> of class '%s' <%s>",
externalptr_address(pointer), .Call(Class__name, cppclass),
externalptr_address(cppclass))
3: show(modtest)
4: show(modtest)

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 1
aborting ...
Segmentation fault


best,
Christian

[Romain Francois]

Post by Christian Gunning
On Mon, Apr 11, 2011 at 3:00 AM,

This will load the declared modules, and populate them into your
package's namespace.
I will update the Rcpp-modules vignette accordingly.
Suggestions, comments, improvements, "what are you ? nuts !", ... are
always welcome.

Works great for me -- very cool.
One question that's been bothering me -- how are module-based objects
*supposed* to behave across an R session quit/save/restart cycle? I'm
seeing no persistence, and a consistent segfault. Am I missing
something here (perhaps wrt finalizer)? If not, I'd like to add a
note to the vignette...

Yes. This is a problem.

Not sure how this could be dealt with, but at least we should document
that there is no persistence yet.

Romain

Post by Christian Gunning
Rcpp.package.skeleton('testpkg1', module=T)
require(testpkg1)
modtest<- new(World)
## everything works
show(modtest)
modtest$set('hello')
modtest$greet()
q()
## Save workspace image? [y/n/c]: y
## exit, save, restart -- nothing works
# [Previously saved workspace restored]
#
# Class "C++Object" is defined (with package slot ‘Rcpp’) but no
metadata object found to revise subclass information---not exported?
Making a copy in package ‘.GlobalEnv’
require(testpkg1)
modtest$greet()
#Error in .External(list(name = "CppMethod__invoke_notvoid", address =
# NULL value passed as symbol address
show(modtest)
*** caught segfault ***
address 0x8, cause 'memory not mapped'
1: .Call(Class__name, cppclass)
2: sprintf("C++ object<%s> of class '%s'<%s>",
externalptr_address(pointer), .Call(Class__name, cppclass),
externalptr_address(cppclass))
3: show(modtest)
4: show(modtest)
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 1
aborting ...
Segmentation fault
best,
Christian

[Douglas Bates]

On Tue, Apr 12, 2011 at 1:11 AM, Romain Francois

Post by Romain Francois

Post by Christian Gunning
On Mon, Apr 11, 2011 at 3:00 AM,

This will load the declared modules, and populate them into your
package's namespace.
I will update the Rcpp-modules vignette accordingly.
Suggestions, comments, improvements, "what are you  ? nuts !", ... are
always welcome.

Works great for me -- very cool.
One question that's been bothering me -- how are module-based objects
*supposed* to behave across an R session quit/save/restart cycle? I'm
seeing no persistence, and a consistent segfault.  Am I missing
something here (perhaps wrt finalizer)?  If not, I'd like to add a
note to the vignette...

Yes. This is a problem.
Not sure how this could be dealt with, but at least we should document that
there is no persistence yet.

The documented behavior is that an external pointer is replaced by a
null pointer when an object is serialized. I was planning to
experiment with an object that has enough information saved as R
objects to be able to regenerate the C++ object and an external
pointer to the module instance. The methods that use the object will
check first to see if the pointer is null, in which case they will
regenerate the module instance, otherwise go directly to invoking
methods on the module instance.

I doubt that I will be able to get to this until next week as I have
our Master's exam reports to read starting at 11:00 this morning.

Post by Romain Francois

Post by Christian Gunning
Rcpp.package.skeleton('testpkg1', module=T)
require(testpkg1)
modtest<- new(World)
## everything works
show(modtest)
modtest$set('hello')
modtest$greet()
q()
## Save workspace image? [y/n/c]: y
## exit, save, restart -- nothing works
# [Previously saved workspace restored]
#
# Class "C++Object" is defined (with package slot ‘Rcpp’) but no
metadata object found to revise subclass information---not exported?
Making a copy in package ‘.GlobalEnv’
require(testpkg1)
modtest$greet()
#Error in .External(list(name = "CppMethod__invoke_notvoid", address =
#  NULL value passed as symbol address
show(modtest)
 *** caught segfault ***
address 0x8, cause 'memory not mapped'
 1: .Call(Class__name, cppclass)
 2: sprintf("C++ object<%s>  of class '%s'<%s>",
externalptr_address(pointer),     .Call(Class__name, cppclass),
externalptr_address(cppclass))
 3: show(modtest)
 4: show(modtest)
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 1
aborting ...
Segmentation fault
best,
Christian

--
Romain Francois
Professional R Enthusiast
[phone]
http://romainfrancois.blog.free.fr
http://romain-francois.com
|- http://bit.ly/fhqbRC : Rcpp workshop in Chicago on April 28th
|- http://bit.ly/dFyZGB : Hydraulique au Montpellier Comedie Club
`- http://bit.ly/eVXit9 : Eponyme : 40 minutes stand up
_______________________________________________
Rcpp-devel mailing list
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel

[Christian Gunning]

Post by Douglas Bates
The documented behavior is that an external pointer is replaced by a
null pointer when an object is serialized.

Thanks. Got it.

Post by Douglas Bates
 I was planning to
experiment with an object that has enough information saved as R
objects to be able to regenerate the C++ object and an external
pointer to the module instance.  The methods that use the object will
check first to see if the pointer is null, in which case they will
regenerate the module instance, otherwise go directly to invoking
methods on the module instance.

No rush on my part -- as usual, I'm just trying to figure out how the
pieces fit together :)

Just to clarify -- does your plan include storing user-modified
fields and properties in objects? E.g. below, modtest with the msg
set?

best,
Christian

Rcpp.package.skeleton('testpkg1', module=T)
require(testpkg1)
modtest <- new(World)
show(modtest)
modtest$set('hello')

[Romain Francois]

Post by Christian Gunning

Post by Douglas Bates
The documented behavior is that an external pointer is replaced by a
null pointer when an object is serialized.

Thanks. Got it.

Post by Douglas Bates
I was planning to
experiment with an object that has enough information saved as R
objects to be able to regenerate the C++ object and an external
pointer to the module instance. The methods that use the object will
check first to see if the pointer is null, in which case they will
regenerate the module instance, otherwise go directly to invoking
methods on the module instance.

No rush on my part -- as usual, I'm just trying to figure out how the
pieces fit together :)
Just to clarify -- does your plan include storing user-modified
fields and properties in objects? E.g. below, modtest with the msg
set?

yes. Although the way I see it, this will require more work for the
developper, who would have to provide a serialization/deserialization
couple for his/her classes.

This is a nice goal for a future version.

Post by Christian Gunning
best,
Christian
Rcpp.package.skeleton('testpkg1', module=T)
require(testpkg1)
modtest<- new(World)
show(modtest)
modtest$set('hello')

[Christian Gunning]

On Tue, Apr 12, 2011 at 11:07 PM, Romain Francois

Post by Romain Francois

Just to clarify --  does your plan include storing user-modified
fields and properties in objects?  E.g. below, modtest with the msg
set?

yes. Although the way I see it, this will require more work for the
developper, who would have to provide a serialization/deserialization couple
for his/her classes.
This is a nice goal for a future version.

Am I naive to think that a 1-to-1 mapping between C++ object fields &
properties and an S4 object might make sense? This reduces the module
developer's problem of serial/deserialization to 1) writing a
constructor (and perhaps with validator) that takes an S4 object, and
2) writing a finalizer that populates a specified S4 object in the
user's environment, tears down the C++ object (as per R Exts manual),
and let's save() do the rest?

It seems that, in theory, there's a canonical mapping between the
names/types of fields/properties exposed in the RCPP_MODULE call and
the proposed S4 object. The main gotcha I see here is that any
variable within a C++ object *not* exposed as a field/property must be
viewed as a const to ensure consistency of the C++ object between
write and re-construct...

If this is off-base, feel free to ignore or point out good reading material :)
-christian

[Douglas Bates]

Post by Christian Gunning
On Tue, Apr 12, 2011 at 11:07 PM, Romain Francois

Post by Romain Francois

Just to clarify --  does your plan include storing user-modified
fields and properties in objects?  E.g. below, modtest with the msg
set?

yes. Although the way I see it, this will require more work for the
developper, who would have to provide a serialization/deserialization couple
for his/her classes.
This is a nice goal for a future version.

Am I naive to think that a 1-to-1 mapping between C++ object fields &
properties and an S4 object might make sense?  This reduces the module
developer's problem of serial/deserialization to 1) writing a
constructor (and perhaps with validator) that takes an S4 object, and
2) writing a finalizer that populates a specified S4 object in the
user's environment, tears down the C++ object (as per R Exts manual),
and let's save() do the rest?
It seems that, in theory, there's a canonical mapping between the
names/types of fields/properties exposed in the RCPP_MODULE call and
the proposed S4 object.  The main gotcha I see here is that any
variable within a C++ object *not* exposed as a field/property must be
viewed as a const to ensure consistency of the C++ object between
write and re-construct...
If this is off-base, feel free to ignore or point out good reading material :)

I don't think it is off-base. I was planning something very similar
to that except that not every field/property would be saved. I have
classes in which many of the properties are calculated on the fly and
I don't need to save those. I just need to save enough information to
create the object again.

This approach would run into trouble if there were data fields in the
C++ object that could not be expressed as R objects. However, it
seems to me that if the C++ object can be generated from R objects in
the first place you should be able to serialize enough information to
reconstruct it.

[Dirk Eddelbuettel]

On 14 April 2011 at 07:49, Douglas Bates wrote:
| On Thu, Apr 14, 2011 at 12:43 AM, Christian Gunning <***@unm.edu> wrote:
| > On Tue, Apr 12, 2011 at 11:07 PM, Romain Francois
| > <***@r-enthusiasts.com> wrote:
| >>>
| >>> Just to clarify --  does your plan include storing user-modified
| >>> fields and properties in objects?  E.g. below, modtest with the msg
| >>> set?
| >>
| >> yes. Although the way I see it, this will require more work for the
| >> developper, who would have to provide a serialization/deserialization couple
| >> for his/her classes.
| >>
| >> This is a nice goal for a future version.
| >
| > Am I naive to think that a 1-to-1 mapping between C++ object fields &
| > properties and an S4 object might make sense?  This reduces the module
| > developer's problem of serial/deserialization to 1) writing a
| > constructor (and perhaps with validator) that takes an S4 object, and
| > 2) writing a finalizer that populates a specified S4 object in the
| > user's environment, tears down the C++ object (as per R Exts manual),
| > and let's save() do the rest?
| >
| > It seems that, in theory, there's a canonical mapping between the
| > names/types of fields/properties exposed in the RCPP_MODULE call and
| > the proposed S4 object.  The main gotcha I see here is that any
| > variable within a C++ object *not* exposed as a field/property must be
| > viewed as a const to ensure consistency of the C++ object between
| > write and re-construct...
| >
| > If this is off-base, feel free to ignore or point out good reading material :)
|
| I don't think it is off-base. I was planning something very similar
| to that except that not every field/property would be saved. I have
| classes in which many of the properties are calculated on the fly and
| I don't need to save those. I just need to save enough information to
| create the object again.
|
| This approach would run into trouble if there were data fields in the
| C++ object that could not be expressed as R objects. However, it
| seems to me that if the C++ object can be generated from R objects in
| the first place you should be able to serialize enough information to
| reconstruct it.

This really is darn close to what RProtoBuf does to fit into (Google's)
Protocol Buffers "everything to everyone" scheme reportedly used for all
intermachine (and interlanguage) communication. But that would of course
introduce a new non-R dependency so you probably do not want that here.

So we'd have to reinvent a suitable subset, leaning on R's serialization.

Too bad we didn't have that insight four weeks ago. This would have made a
swell project for Google Summer of Code. I blame Christian ;-)

Dirk