[Rcpp-devel] Rcpp module visible after check but not after install

[Christopher Genovese]

Hello,

I have a package under development that uses Rcpp modules to expose
some C++ classes. I hadn't touched the package for some months (close
to a year). At that time the package compiled and installed without problem
and worked well. I could access the classes as expected. Recently, having
updated Rcpp and R in the meantime, I reinstalled the package with
the following result:

1. If building and loading from within the package directory, all works
fine.
2. When installing the package, either from github, locally, or
otherwise,
the modules are *not* visible.

The package passes devtools:check() and compiles fine.

For instance, for one of those classes, LshParameterSetter, I do


RCPP_EXPOSED_CLASS(LshParameterSetter)

RCPP_MODULE(mod_params) {
class_<LshParameterSetter>("LshParameterSetter")

.constructor<int,int>()

.method("withDefaults", &LshParameterSetter::withDefaults,
"Fill with defaults")
.method("distance", &LshParameterSetter::distance,
"Set distance metric")
.method("numHashFunctions", &LshParameterSetter::numHashFunctions,
"Set numbers of hash functions and tables")
.method("numHashTables", &LshParameterSetter::numHashTables,
"Set numbers of hash functions and tables")
.method("storage", &LshParameterSetter::storage,
"Set LSH Hash Table Storage type")
.method("family", &LshParameterSetter::family,
"Set LSH Family")
.method("rotations", &LshParameterSetter::rotations,
"Number of rotations")
.method("asList", &LshParameterSetter::asList,
"List of parameter values by name")
;
}

and similarly for the other classes. This module is loaded in the R code
with
Rcpp::loadModule("mod_params", TRUE)
and similarly for the other modules.

The installed package has the .rdb, .rdx, and .so file, so it's hard to see
if something has been lost in the installation. There's no indication what
is wrong, but after loading the installed package, a class like
LshParameterSetter
is not defined. (Loading from within the package source directory it is,
showing

LshParameterSetter

C++ class 'LshParameterSetter' <0x7fde72fb1940>
LshParameterSetter(int, int)
Fields: ...

as it should be.) I've done this with simple skeletons and not had a
similar problem. But I don't see a qualitative difference between these
cases.

The full package code is at https://github.com/genovese/falconnr
and the session info is below.

I'd appreciate any pointers on this, as the problem is unclear to me,
especially since it worked without problem last year. I'm happy to
provide any useful information that I might have omitted here.

Thanks for your help, Chris

sessionInfo()

R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS Sierra 10.12.6

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] grid stats graphics grDevices utils datasets methods
base

other attached packages:
[1] falconnr_0.0.0.9001 MASS_7.3-45 lattice_0.20-34

loaded via a namespace (and not attached):
[1] Rcpp_0.12.13 codetools_0.2-15

[Dirk Eddelbuettel]

On 9 November 2017 at 01:54, Christopher Genovese wrote:
| Hello,
|
| I have a package under development that uses Rcpp modules to expose
| some C++ classes. I hadn't touched the package for some months (close
| to a year). At that time the package compiled and installed without problem
| and worked well. I could access the classes as expected. Recently, having
| updated Rcpp and R in the meantime, I reinstalled the package with
| the following result:
|
| 1. If building and loading from within the package directory, all works
| fine.
| 2. When installing the package, either from github, locally, or
| otherwise,
| the modules are *not* visible.

What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we are
meant to create a tar.gz in either way via 'R CMD BUILD' and then install it
via 'R CMD INSTALL'.

Are you doing that?

| The package passes devtools:check() and compiles fine.

Well if it passes check you should be fine.

| For instance, for one of those classes, LshParameterSetter, I do
|
|
| RCPP_EXPOSED_CLASS(LshParameterSetter)
|
| RCPP_MODULE(mod_params) {
| class_<LshParameterSetter>("LshParameterSetter")
|
| .constructor<int,int>()
|
| .method("withDefaults", &LshParameterSetter::withDefaults,
| "Fill with defaults")
| .method("distance", &LshParameterSetter::distance,
| "Set distance metric")
| .method("numHashFunctions", &LshParameterSetter::numHashFunctions,
| "Set numbers of hash functions and tables")
| .method("numHashTables", &LshParameterSetter::numHashTables,
| "Set numbers of hash functions and tables")
| .method("storage", &LshParameterSetter::storage,
| "Set LSH Hash Table Storage type")
| .method("family", &LshParameterSetter::family,
| "Set LSH Family")
| .method("rotations", &LshParameterSetter::rotations,
| "Number of rotations")
| .method("asList", &LshParameterSetter::asList,
| "List of parameter values by name")
| ;
| }
|
| and similarly for the other classes. This module is loaded in the R code
| with
| Rcpp::loadModule("mod_params", TRUE)
| and similarly for the other modules.
|
| The installed package has the .rdb, .rdx, and .so file, so it's hard to see
| if something has been lost in the installation. There's no indication what
| is wrong, but after loading the installed package, a class like
| LshParameterSetter
| is not defined. (Loading from within the package source directory it is,

What do you mean by 'loading from within the package source directory' ?

| showing
|
| > > LshParameterSetter
| > C++ class 'LshParameterSetter' <0x7fde72fb1940>
| > Constructors:
| > LshParameterSetter(int, int)
| >
| > Fields: ...
| >
| as it should be.) I've done this with simple skeletons and not had a
| similar problem. But I don't see a qualitative difference between these
| cases.
|
| The full package code is at https://github.com/genovese/falconnr
| and the session info is below.

Thanks for posting the link. Not a small package ...

| I'd appreciate any pointers on this, as the problem is unclear to me,
| especially since it worked without problem last year. I'm happy to
| provide any useful information that I might have omitted here.

Not sure what is going on there. Package clearly builds. Testing complains
some but hey, it is not a CRAN package yet...

You have

falconnr::LshTable

visible, but (what you probably want) only un-exported:

falconnr:::LshNnTable
falconnr:::LshParameterSetter

I have been using Modules for about as long as we've had them, but I have
always been a little mystified by the various UPPERCASE_MACROS exporting
things. Looking at what was probably my first Modules package, RcppBDT, I
seem to "simply" do

RCPP_MODULE(bdtDdMod) {
Rcpp::class_<bdtDd>("bdtDd")
// ... stuff omitted

and nothing else for the four or so different modules there. That worked. I
think you may rely on a helper macro which may have gotten stale as the
_underlying R behaviour_ may have changed. I think I use one or two of the
exporter macros in other places (ie RcppAnnoy) so you could try that.

In general, I always recommend to simplify as much as you can.

Hope this helps, Dirk

| Thanks for your help, Chris
|
| > sessionInfo()
| R version 3.3.2 (2016-10-31)
| Platform: x86_64-apple-darwin13.4.0 (64-bit)
| Running under: macOS Sierra 10.12.6
|
| locale:
| [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
|
| attached base packages:
| [1] grid stats graphics grDevices utils datasets methods
| base
|
| other attached packages:
| [1] falconnr_0.0.0.9001 MASS_7.3-45 lattice_0.20-34
|
| loaded via a namespace (and not attached):
| [1] Rcpp_0.12.13 codetools_0.2-15
| _______________________________________________
| Rcpp-devel mailing list
| Rcpp-***@lists.r-forge.r-project.org
| https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel

[Christopher Genovese]

Thanks, Dirk.

What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we are

Post by Dirk Eddelbuettel
meant to create a tar.gz in either way via 'R CMD BUILD' and then install it
via 'R CMD INSTALL'.
Are you doing that?

Yes. I usually use devtools, but I've done it directly as well.

In #1, for instance: I go to the package source directory, start R,
do devtools::build and devtools::check. This builds and compiles
the package. After library() at this point, the module components
are visible, e.g., LshParameterSetter.

In #2, for instance, I do devtools::install() or devtools::install_github()
or similar.
Quit R, go to some other directory, restart R, and call library(). The R
entry
points are visible, but none of the module classes.

I'm unclear as to what in the installation/export process can explain the
difference
between #1 and #2.

Before emailing yesterday, I had tried removing the RCPP_EXPOSED_CLASS()'s,
but then the code does not compile. There is a static_assert error in
wrap_dispatch_unknown_iterable in Rcpp's wrap.h (line 520). I did not see an
easy way to fix that. (I vaguely recall that I only included them
originally because
I had to.)

I will look RcppAnnoy and see if I can find a way around using these
macros.

Thanks again, Chris

Post by Dirk Eddelbuettel
| Hello,
|
| I have a package under development that uses Rcpp modules to expose
| some C++ classes. I hadn't touched the package for some months (close
| to a year). At that time the package compiled and installed without problem
| and worked well. I could access the classes as expected. Recently, having
| updated Rcpp and R in the meantime, I reinstalled the package with
|
| 1. If building and loading from within the package directory, all works
| fine.
| 2. When installing the package, either from github, locally, or
| otherwise,
| the modules are *not* visible.
What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we are
meant to create a tar.gz in either way via 'R CMD BUILD' and then install it
via 'R CMD INSTALL'.
Are you doing that?
| The package passes devtools:check() and compiles fine.
Well if it passes check you should be fine.
| For instance, for one of those classes, LshParameterSetter, I do
|
|
| RCPP_EXPOSED_CLASS(LshParameterSetter)
|
| RCPP_MODULE(mod_params) {
| class_<LshParameterSetter>("LshParameterSetter")
|
| .constructor<int,int>()
|
| .method("withDefaults", &LshParameterSetter::withDefaults,
| "Fill with defaults")
| .method("distance", &LshParameterSetter::distance,
| "Set distance metric")
| .method("numHashFunctions", &LshParameterSetter::numHashFunctions,
| "Set numbers of hash functions and tables")
| .method("numHashTables", &LshParameterSetter::numHashTables,
| "Set numbers of hash functions and tables")
| .method("storage", &LshParameterSetter::storage,
| "Set LSH Hash Table Storage type")
| .method("family", &LshParameterSetter::family,
| "Set LSH Family")
| .method("rotations", &LshParameterSetter::rotations,
| "Number of rotations")
| .method("asList", &LshParameterSetter::asList,
| "List of parameter values by name")
| ;
| }
|
| and similarly for the other classes. This module is loaded in the R code
| with
| Rcpp::loadModule("mod_params", TRUE)
| and similarly for the other modules.
|
| The installed package has the .rdb, .rdx, and .so file, so it's hard to see
| if something has been lost in the installation. There's no indication what
| is wrong, but after loading the installed package, a class like
| LshParameterSetter
| is not defined. (Loading from within the package source directory it is,
What do you mean by 'loading from within the package source directory' ?
| showing
|
| > > LshParameterSetter
| > C++ class 'LshParameterSetter' <0x7fde72fb1940>
| > LshParameterSetter(int, int)
| >
| > Fields: ...
| >
| as it should be.) I've done this with simple skeletons and not had a
| similar problem. But I don't see a qualitative difference between these
| cases.
|
| The full package code is at https://github.com/genovese/falconnr
| and the session info is below.
Thanks for posting the link. Not a small package ...
| I'd appreciate any pointers on this, as the problem is unclear to me,
| especially since it worked without problem last year. I'm happy to
| provide any useful information that I might have omitted here.
Not sure what is going on there. Package clearly builds. Testing complains
some but hey, it is not a CRAN package yet...
You have
falconnr::LshTable
falconnr:::LshNnTable
falconnr:::LshParameterSetter
I have been using Modules for about as long as we've had them, but I have
always been a little mystified by the various UPPERCASE_MACROS exporting
things. Looking at what was probably my first Modules package, RcppBDT, I
seem to "simply" do
RCPP_MODULE(bdtDdMod) {
Rcpp::class_<bdtDd>("bdtDd")
// ... stuff omitted
and nothing else for the four or so different modules there. That worked. I
think you may rely on a helper macro which may have gotten stale as the
_underlying R behaviour_ may have changed. I think I use one or two of the
exporter macros in other places (ie RcppAnnoy) so you could try that.
In general, I always recommend to simplify as much as you can.
Hope this helps, Dirk
| Thanks for your help, Chris
|
| > sessionInfo()
| R version 3.3.2 (2016-10-31)
| Platform: x86_64-apple-darwin13.4.0 (64-bit)
| Running under: macOS Sierra 10.12.6
|
| [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
|
| [1] grid stats graphics grDevices utils datasets methods
| base
|
| [1] falconnr_0.0.0.9001 MASS_7.3-45 lattice_0.20-34
|
| [1] Rcpp_0.12.13 codetools_0.2-15
| _______________________________________________
| Rcpp-devel mailing list
| https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
--

[Dirk Eddelbuettel]

On 9 November 2017 at 09:32, Christopher Genovese wrote:
| Thanks, Dirk.
|
| What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we are
| > meant to create a tar.gz in either way via 'R CMD BUILD' and then install
| > it
| > via 'R CMD INSTALL'.
| >
| > Are you doing that?
| >
|
| Yes. I usually use devtools, but I've done it directly as well.

Take this up with the devtools developers then, please.

I operate with R command, and possibly very thin littler wrappers such as
build.r and rcc.r to, respectively, build and check.

| In #1, for instance: I go to the package source directory, start R,
| do devtools::build and devtools::check. This builds and compiles
| the package. After library() at this point, the module components
| are visible, e.g., LshParameterSetter.
|
| In #2, for instance, I do devtools::install() or devtools::install_github()
| or similar.
| Quit R, go to some other directory, restart R, and call library(). The R
| entry
| points are visible, but none of the module classes.
|
| I'm unclear as to what in the installation/export process can explain the
| difference
| between #1 and #2.

Again, I have no idea what devtools does here. Other people may.

The Rcpp documentation does not cover devtools so you are on your own.

| Before emailing yesterday, I had tried removing the RCPP_EXPOSED_CLASS()'s,
| but then the code does not compile. There is a static_assert error in
| wrap_dispatch_unknown_iterable in Rcpp's wrap.h (line 520). I did not see an
| easy way to fix that. (I vaguely recall that I only included them
| originally because
| I had to.)

I think you need those. I just glance at the different variants on the train
in, they are mostly equivalent and just layers of each other.

| I will look RcppAnnoy and see if I can find a way around using these
| macros.

Sounds good.

Modules can be frustrating. I just tried with some larger code at work, and
it doesn't load :-/ Something else may get in the way but hard to tell.

Rcpp Attributes is just fine.

Dirk

| Thanks again, Chris
|
|
|
|
| On Thu, Nov 9, 2017 at 7:55 AM, Dirk Eddelbuettel <***@debian.org> wrote:
|
| >
| > On 9 November 2017 at 01:54, Christopher Genovese wrote:
| > | Hello,
| > |
| > | I have a package under development that uses Rcpp modules to expose
| > | some C++ classes. I hadn't touched the package for some months (close
| > | to a year). At that time the package compiled and installed without
| > problem
| > | and worked well. I could access the classes as expected. Recently, having
| > | updated Rcpp and R in the meantime, I reinstalled the package with
| > | the following result:
| > |
| > | 1. If building and loading from within the package directory, all
| > works
| > | fine.
| > | 2. When installing the package, either from github, locally, or
| > | otherwise,
| > | the modules are *not* visible.
| >
| > What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we are
| > meant to create a tar.gz in either way via 'R CMD BUILD' and then install
| > it
| > via 'R CMD INSTALL'.
| >
| > Are you doing that?
| >
| > | The package passes devtools:check() and compiles fine.
| >
| > Well if it passes check you should be fine.
| >
| > | For instance, for one of those classes, LshParameterSetter, I do
| > |
| > |
| > | RCPP_EXPOSED_CLASS(LshParameterSetter)
| > |
| > | RCPP_MODULE(mod_params) {
| > | class_<LshParameterSetter>("LshParameterSetter")
| > |
| > | .constructor<int,int>()
| > |
| > | .method("withDefaults", &LshParameterSetter::withDefaults,
| > | "Fill with defaults")
| > | .method("distance", &LshParameterSetter::distance,
| > | "Set distance metric")
| > | .method("numHashFunctions", &LshParameterSetter::numHashFunctions,
| > | "Set numbers of hash functions and tables")
| > | .method("numHashTables", &LshParameterSetter::numHashTables,
| > | "Set numbers of hash functions and tables")
| > | .method("storage", &LshParameterSetter::storage,
| > | "Set LSH Hash Table Storage type")
| > | .method("family", &LshParameterSetter::family,
| > | "Set LSH Family")
| > | .method("rotations", &LshParameterSetter::rotations,
| > | "Number of rotations")
| > | .method("asList", &LshParameterSetter::asList,
| > | "List of parameter values by name")
| > | ;
| > | }
| > |
| > | and similarly for the other classes. This module is loaded in the R code
| > | with
| > | Rcpp::loadModule("mod_params", TRUE)
| > | and similarly for the other modules.
| > |
| > | The installed package has the .rdb, .rdx, and .so file, so it's hard to
| > see
| > | if something has been lost in the installation. There's no indication
| > what
| > | is wrong, but after loading the installed package, a class like
| > | LshParameterSetter
| > | is not defined. (Loading from within the package source directory it is,
| >
| > What do you mean by 'loading from within the package source directory' ?
| >
| > | showing
| > |
| > | > > LshParameterSetter
| > | > C++ class 'LshParameterSetter' <0x7fde72fb1940>
| > | > Constructors:
| > | > LshParameterSetter(int, int)
| > | >
| > | > Fields: ...
| > | >
| > | as it should be.) I've done this with simple skeletons and not had a
| > | similar problem. But I don't see a qualitative difference between these
| > | cases.
| > |
| > | The full package code is at https://github.com/genovese/falconnr
| > | and the session info is below.
| >
| > Thanks for posting the link. Not a small package ...
| >
| > | I'd appreciate any pointers on this, as the problem is unclear to me,
| > | especially since it worked without problem last year. I'm happy to
| > | provide any useful information that I might have omitted here.
| >
| > Not sure what is going on there. Package clearly builds. Testing complains
| > some but hey, it is not a CRAN package yet...
| >
| > You have
| >
| > falconnr::LshTable
| >
| > visible, but (what you probably want) only un-exported:
| >
| > falconnr:::LshNnTable
| > falconnr:::LshParameterSetter
| >
| > I have been using Modules for about as long as we've had them, but I have
| > always been a little mystified by the various UPPERCASE_MACROS exporting
| > things. Looking at what was probably my first Modules package, RcppBDT, I
| > seem to "simply" do
| >
| > RCPP_MODULE(bdtDdMod) {
| > Rcpp::class_<bdtDd>("bdtDd")
| > // ... stuff omitted
| >
| > and nothing else for the four or so different modules there. That worked.
| > I
| > think you may rely on a helper macro which may have gotten stale as the
| > _underlying R behaviour_ may have changed. I think I use one or two of the
| > exporter macros in other places (ie RcppAnnoy) so you could try that.
| >
| > In general, I always recommend to simplify as much as you can.
| >
| > Hope this helps, Dirk
| >
| > | Thanks for your help, Chris
| > |
| > | > sessionInfo()
| > | R version 3.3.2 (2016-10-31)
| > | Platform: x86_64-apple-darwin13.4.0 (64-bit)
| > | Running under: macOS Sierra 10.12.6
| > |
| > | locale:
| > | [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
| > |
| > | attached base packages:
| > | [1] grid stats graphics grDevices utils datasets methods
| > | base
| > |
| > | other attached packages:
| > | [1] falconnr_0.0.0.9001 MASS_7.3-45 lattice_0.20-34
| > |
| > | loaded via a namespace (and not attached):
| > | [1] Rcpp_0.12.13 codetools_0.2-15
| > | _______________________________________________
| > | Rcpp-devel mailing list
| > | Rcpp-***@lists.r-forge.r-project.org
| > | https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
| >
| > --
| > http://dirk.eddelbuettel.com | @eddelbuettel | ***@debian.org
| >

[Christopher Genovese]

Sorry, let me be clear. I've done it both ways with the same basic
result (for #2). Specifically, doing the R CMD CHECK, R CMD BUILD,
R CMD INSTALL sequence gives an installed package with the
the two classes invisible. So I don't think this is a devtools issue.

This is rather frustrating, I agree, especially as it worked previously.

Is there any documentation (preferably short of going through the Rcpp
modules
code) on the process by which these symbols get exported into R. The
symbols are in the .so file, but for instance, where/when would the
function R_init_falconnr function, which is created by
Rcpp::compileAttributes
and seems to load the modules, be called?

Thanks, Chris

Post by Dirk Eddelbuettel
| Thanks, Dirk.
|
| What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we are
| > meant to create a tar.gz in either way via 'R CMD BUILD' and then install
| > it
| > via 'R CMD INSTALL'.
| >
| > Are you doing that?
| >
|
| Yes. I usually use devtools, but I've done it directly as well.
Take this up with the devtools developers then, please.
I operate with R command, and possibly very thin littler wrappers such as
build.r and rcc.r to, respectively, build and check.
| In #1, for instance: I go to the package source directory, start R,
| do devtools::build and devtools::check. This builds and compiles
| the package. After library() at this point, the module components
| are visible, e.g., LshParameterSetter.
|
| In #2, for instance, I do devtools::install() or
devtools::install_github()
| or similar.
| Quit R, go to some other directory, restart R, and call library(). The R
| entry
| points are visible, but none of the module classes.
|
| I'm unclear as to what in the installation/export process can explain the
| difference
| between #1 and #2.
Again, I have no idea what devtools does here. Other people may.
The Rcpp documentation does not cover devtools so you are on your own.
| Before emailing yesterday, I had tried removing the
RCPP_EXPOSED_CLASS()'s,
| but then the code does not compile. There is a static_assert error in
| wrap_dispatch_unknown_iterable in Rcpp's wrap.h (line 520). I did not see an
| easy way to fix that. (I vaguely recall that I only included them
| originally because
| I had to.)
I think you need those. I just glance at the different variants on the train
in, they are mostly equivalent and just layers of each other.
| I will look RcppAnnoy and see if I can find a way around using these
| macros.
Sounds good.
Modules can be frustrating. I just tried with some larger code at work, and
it doesn't load :-/ Something else may get in the way but hard to tell.
Rcpp Attributes is just fine.
Dirk
| Thanks again, Chris
|
|
|
|
|
| >
| > | Hello,
| > |
| > | I have a package under development that uses Rcpp modules to expose
| > | some C++ classes. I hadn't touched the package for some months (close
| > | to a year). At that time the package compiled and installed without
| > problem
| > | and worked well. I could access the classes as expected. Recently, having
| > | updated Rcpp and R in the meantime, I reinstalled the package with
| > |
| > | 1. If building and loading from within the package directory, all
| > works
| > | fine.
| > | 2. When installing the package, either from github, locally, or
| > | otherwise,
| > | the modules are *not* visible.
| >
| > What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we are
| > meant to create a tar.gz in either way via 'R CMD BUILD' and then install
| > it
| > via 'R CMD INSTALL'.
| >
| > Are you doing that?
| >
| > | The package passes devtools:check() and compiles fine.
| >
| > Well if it passes check you should be fine.
| >
| > | For instance, for one of those classes, LshParameterSetter, I do
| > |
| > |
| > | RCPP_EXPOSED_CLASS(LshParameterSetter)
| > |
| > | RCPP_MODULE(mod_params) {
| > | class_<LshParameterSetter>("LshParameterSetter")
| > |
| > | .constructor<int,int>()
| > |
| > | .method("withDefaults", &LshParameterSetter::withDefaults,
| > | "Fill with defaults")
| > | .method("distance", &LshParameterSetter::distance,
| > | "Set distance metric")
numHashFunctions,
| > | "Set numbers of hash functions and tables")
| > | .method("numHashTables", &LshParameterSetter::numHashTables,
| > | "Set numbers of hash functions and tables")
| > | .method("storage", &LshParameterSetter::storage,
| > | "Set LSH Hash Table Storage type")
| > | .method("family", &LshParameterSetter::family,
| > | "Set LSH Family")
| > | .method("rotations", &LshParameterSetter::rotations,
| > | "Number of rotations")
| > | .method("asList", &LshParameterSetter::asList,
| > | "List of parameter values by name")
| > | ;
| > | }
| > |
| > | and similarly for the other classes. This module is loaded in the R code
| > | with
| > | Rcpp::loadModule("mod_params", TRUE)
| > | and similarly for the other modules.
| > |
| > | The installed package has the .rdb, .rdx, and .so file, so it's hard to
| > see
| > | if something has been lost in the installation. There's no indication
| > what
| > | is wrong, but after loading the installed package, a class like
| > | LshParameterSetter
| > | is not defined. (Loading from within the package source directory it is,
| >
| > What do you mean by 'loading from within the package source directory' ?
| >
| > | showing
| > |
| > | > > LshParameterSetter
| > | > C++ class 'LshParameterSetter' <0x7fde72fb1940>
| > | > LshParameterSetter(int, int)
| > | >
| > | > Fields: ...
| > | >
| > | as it should be.) I've done this with simple skeletons and not had a
| > | similar problem. But I don't see a qualitative difference between these
| > | cases.
| > |
| > | The full package code is at https://github.com/genovese/falconnr
| > | and the session info is below.
| >
| > Thanks for posting the link. Not a small package ...
| >
| > | I'd appreciate any pointers on this, as the problem is unclear to me,
| > | especially since it worked without problem last year. I'm happy to
| > | provide any useful information that I might have omitted here.
| >
| > Not sure what is going on there. Package clearly builds. Testing complains
| > some but hey, it is not a CRAN package yet...
| >
| > You have
| >
| > falconnr::LshTable
| >
| >
| > falconnr:::LshNnTable
| > falconnr:::LshParameterSetter
| >
| > I have been using Modules for about as long as we've had them, but I have
| > always been a little mystified by the various UPPERCASE_MACROS exporting
| > things. Looking at what was probably my first Modules package, RcppBDT, I
| > seem to "simply" do
| >
| > RCPP_MODULE(bdtDdMod) {
| > Rcpp::class_<bdtDd>("bdtDd")
| > // ... stuff omitted
| >
| > and nothing else for the four or so different modules there. That worked.
| > I
| > think you may rely on a helper macro which may have gotten stale as the
| > _underlying R behaviour_ may have changed. I think I use one or two of the
| > exporter macros in other places (ie RcppAnnoy) so you could try that.
| >
| > In general, I always recommend to simplify as much as you can.
| >
| > Hope this helps, Dirk
| >
| > | Thanks for your help, Chris
| > |
| > | > sessionInfo()
| > | R version 3.3.2 (2016-10-31)
| > | Platform: x86_64-apple-darwin13.4.0 (64-bit)
| > | Running under: macOS Sierra 10.12.6
| > |
| > | [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
| > |
| > | [1] grid stats graphics grDevices utils datasets
methods
| > | base
| > |
| > | [1] falconnr_0.0.0.9001 MASS_7.3-45 lattice_0.20-34
| > |
| > | [1] Rcpp_0.12.13 codetools_0.2-15
| > | _______________________________________________
| > | Rcpp-devel mailing list
| > | https://lists.r-forge.r-project.org/cgi-bin/mailman/
listinfo/rcpp-devel
| >
| > --
| >
--

[Dirk Eddelbuettel]

On 9 November 2017 at 11:34, Christopher Genovese wrote:
| Sorry, let me be clear. I've done it both ways with the same basic
| result (for #2). Specifically, doing the R CMD CHECK, R CMD BUILD,
| R CMD INSTALL sequence gives an installed package with the
| the two classes invisible. So I don't think this is a devtools issue.
|
| This is rather frustrating, I agree, especially as it worked previously.

Agreed. But some things also changed in R ... and I am not entirely what the
actual cause was for the change you see.

| Is there any documentation (preferably short of going through the Rcpp
| modules
| code) on the process by which these symbols get exported into R. The
| symbols are in the .so file, but for instance, where/when would the
| function R_init_falconnr function, which is created by
| Rcpp::compileAttributes
| and seems to load the modules, be called?

Well, I also find that combining Modules and compileAttributes() can be
finicky. If output for this was was generated, you could place it e.g. in
src/init.c -- R calls the resulting function on package load. But, and that
is a big 'but' you don't have to unless you say .registration=TRUE in
useDynLib() in NAMESPACE. And I seem to recall I actually wrote the
src/init.c for RcppAnnoy "by hand" as the generator in R itself does not know
about Modules. We may need to add a tool here.

And again, note that you do not need src/init.c -- you only need it if you
set .registration=TRUE. So as first step, I'd say 'do not do that'. I know
it sounds silly -- but if I were you I'd start from the working Modules
example eg in the Rcpp unittests and would then re-add my code. Slowly, and
step by step. That way you will know which change breaks it.

And by all I means I don't claim this is ideal -- if we learn something along
the way we can try to make Modules better. It remains a useful yet somewhat
obscure and underdocumented tool.

Dirk

|
| Thanks, Chris
|
|
| On Thu, Nov 9, 2017 at 9:59 AM, Dirk Eddelbuettel <***@debian.org> wrote:
|
| >
| > On 9 November 2017 at 09:32, Christopher Genovese wrote:
| > | Thanks, Dirk.
| > |
| > | What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we
| > are
| > | > meant to create a tar.gz in either way via 'R CMD BUILD' and then
| > install
| > | > it
| > | > via 'R CMD INSTALL'.
| > | >
| > | > Are you doing that?
| > | >
| > |
| > | Yes. I usually use devtools, but I've done it directly as well.
| >
| > Take this up with the devtools developers then, please.
| >
| > I operate with R command, and possibly very thin littler wrappers such as
| > build.r and rcc.r to, respectively, build and check.
| >
| > | In #1, for instance: I go to the package source directory, start R,
| > | do devtools::build and devtools::check. This builds and compiles
| > | the package. After library() at this point, the module components
| > | are visible, e.g., LshParameterSetter.
| > |
| > | In #2, for instance, I do devtools::install() or
| > devtools::install_github()
| > | or similar.
| > | Quit R, go to some other directory, restart R, and call library(). The R
| > | entry
| > | points are visible, but none of the module classes.
| > |
| > | I'm unclear as to what in the installation/export process can explain the
| > | difference
| > | between #1 and #2.
| >
| > Again, I have no idea what devtools does here. Other people may.
| >
| > The Rcpp documentation does not cover devtools so you are on your own.
| >
| > | Before emailing yesterday, I had tried removing the
| > RCPP_EXPOSED_CLASS()'s,
| > | but then the code does not compile. There is a static_assert error in
| > | wrap_dispatch_unknown_iterable in Rcpp's wrap.h (line 520). I did not
| > see an
| > | easy way to fix that. (I vaguely recall that I only included them
| > | originally because
| > | I had to.)
| >
| > I think you need those. I just glance at the different variants on the
| > train
| > in, they are mostly equivalent and just layers of each other.
| >
| > | I will look RcppAnnoy and see if I can find a way around using these
| > | macros.
| >
| > Sounds good.
| >
| > Modules can be frustrating. I just tried with some larger code at work,
| > and
| > it doesn't load :-/ Something else may get in the way but hard to tell.
| >
| > Rcpp Attributes is just fine.
| >
| > Dirk
| >
| > | Thanks again, Chris
| > |
| > |
| > |
| > |
| > | On Thu, Nov 9, 2017 at 7:55 AM, Dirk Eddelbuettel <***@debian.org>
| > wrote:
| > |
| > | >
| > | > On 9 November 2017 at 01:54, Christopher Genovese wrote:
| > | > | Hello,
| > | > |
| > | > | I have a package under development that uses Rcpp modules to expose
| > | > | some C++ classes. I hadn't touched the package for some months (close
| > | > | to a year). At that time the package compiled and installed without
| > | > problem
| > | > | and worked well. I could access the classes as expected. Recently,
| > having
| > | > | updated Rcpp and R in the meantime, I reinstalled the package with
| > | > | the following result:
| > | > |
| > | > | 1. If building and loading from within the package directory, all
| > | > works
| > | > | fine.
| > | > | 2. When installing the package, either from github, locally, or
| > | > | otherwise,
| > | > | the modules are *not* visible.
| > | >
| > | > What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we
| > are
| > | > meant to create a tar.gz in either way via 'R CMD BUILD' and then
| > install
| > | > it
| > | > via 'R CMD INSTALL'.
| > | >
| > | > Are you doing that?
| > | >
| > | > | The package passes devtools:check() and compiles fine.
| > | >
| > | > Well if it passes check you should be fine.
| > | >
| > | > | For instance, for one of those classes, LshParameterSetter, I do
| > | > |
| > | > |
| > | > | RCPP_EXPOSED_CLASS(LshParameterSetter)
| > | > |
| > | > | RCPP_MODULE(mod_params) {
| > | > | class_<LshParameterSetter>("LshParameterSetter")
| > | > |
| > | > | .constructor<int,int>()
| > | > |
| > | > | .method("withDefaults", &LshParameterSetter::withDefaults,
| > | > | "Fill with defaults")
| > | > | .method("distance", &LshParameterSetter::distance,
| > | > | "Set distance metric")
| > | > | .method("numHashFunctions", &LshParameterSetter::
| > numHashFunctions,
| > | > | "Set numbers of hash functions and tables")
| > | > | .method("numHashTables", &LshParameterSetter::numHashTables,
| > | > | "Set numbers of hash functions and tables")
| > | > | .method("storage", &LshParameterSetter::storage,
| > | > | "Set LSH Hash Table Storage type")
| > | > | .method("family", &LshParameterSetter::family,
| > | > | "Set LSH Family")
| > | > | .method("rotations", &LshParameterSetter::rotations,
| > | > | "Number of rotations")
| > | > | .method("asList", &LshParameterSetter::asList,
| > | > | "List of parameter values by name")
| > | > | ;
| > | > | }
| > | > |
| > | > | and similarly for the other classes. This module is loaded in the R
| > code
| > | > | with
| > | > | Rcpp::loadModule("mod_params", TRUE)
| > | > | and similarly for the other modules.
| > | > |
| > | > | The installed package has the .rdb, .rdx, and .so file, so it's hard
| > to
| > | > see
| > | > | if something has been lost in the installation. There's no
| > indication
| > | > what
| > | > | is wrong, but after loading the installed package, a class like
| > | > | LshParameterSetter
| > | > | is not defined. (Loading from within the package source directory it
| > is,
| > | >
| > | > What do you mean by 'loading from within the package source directory'
| > ?
| > | >
| > | > | showing
| > | > |
| > | > | > > LshParameterSetter
| > | > | > C++ class 'LshParameterSetter' <0x7fde72fb1940>
| > | > | > Constructors:
| > | > | > LshParameterSetter(int, int)
| > | > | >
| > | > | > Fields: ...
| > | > | >
| > | > | as it should be.) I've done this with simple skeletons and not had a
| > | > | similar problem. But I don't see a qualitative difference between
| > these
| > | > | cases.
| > | > |
| > | > | The full package code is at https://github.com/genovese/falconnr
| > | > | and the session info is below.
| > | >
| > | > Thanks for posting the link. Not a small package ...
| > | >
| > | > | I'd appreciate any pointers on this, as the problem is unclear to me,
| > | > | especially since it worked without problem last year. I'm happy to
| > | > | provide any useful information that I might have omitted here.
| > | >
| > | > Not sure what is going on there. Package clearly builds. Testing
| > complains
| > | > some but hey, it is not a CRAN package yet...
| > | >
| > | > You have
| > | >
| > | > falconnr::LshTable
| > | >
| > | > visible, but (what you probably want) only un-exported:
| > | >
| > | > falconnr:::LshNnTable
| > | > falconnr:::LshParameterSetter
| > | >
| > | > I have been using Modules for about as long as we've had them, but I
| > have
| > | > always been a little mystified by the various UPPERCASE_MACROS
| > exporting
| > | > things. Looking at what was probably my first Modules package,
| > RcppBDT, I
| > | > seem to "simply" do
| > | >
| > | > RCPP_MODULE(bdtDdMod) {
| > | > Rcpp::class_<bdtDd>("bdtDd")
| > | > // ... stuff omitted
| > | >
| > | > and nothing else for the four or so different modules there. That
| > worked.
| > | > I
| > | > think you may rely on a helper macro which may have gotten stale as the
| > | > _underlying R behaviour_ may have changed. I think I use one or two
| > of the
| > | > exporter macros in other places (ie RcppAnnoy) so you could try that.
| > | >
| > | > In general, I always recommend to simplify as much as you can.
| > | >
| > | > Hope this helps, Dirk
| > | >
| > | > | Thanks for your help, Chris
| > | > |
| > | > | > sessionInfo()
| > | > | R version 3.3.2 (2016-10-31)
| > | > | Platform: x86_64-apple-darwin13.4.0 (64-bit)
| > | > | Running under: macOS Sierra 10.12.6
| > | > |
| > | > | locale:
| > | > | [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
| > | > |
| > | > | attached base packages:
| > | > | [1] grid stats graphics grDevices utils datasets
| > methods
| > | > | base
| > | > |
| > | > | other attached packages:
| > | > | [1] falconnr_0.0.0.9001 MASS_7.3-45 lattice_0.20-34
| > | > |
| > | > | loaded via a namespace (and not attached):
| > | > | [1] Rcpp_0.12.13 codetools_0.2-15
| > | > | _______________________________________________
| > | > | Rcpp-devel mailing list
| > | > | Rcpp-***@lists.r-forge.r-project.org
| > | > | https://lists.r-forge.r-project.org/cgi-bin/mailman/
| > listinfo/rcpp-devel
| > | >
| > | > --
| > | > http://dirk.eddelbuettel.com | @eddelbuettel | ***@debian.org
| > | >
| >
| > --
| > http://dirk.eddelbuettel.com | @eddelbuettel | ***@debian.org
| >

[Christopher Genovese]

Thanks again, Dirk. Great suggestions; I will certainly try that
approach.

I definitely agree that modules are very useful, and I very much
appreciate all your (and your team's) efforts in bringing this to the
community.

Regards, Chris

Post by Dirk Eddelbuettel
| Sorry, let me be clear. I've done it both ways with the same basic
| result (for #2). Specifically, doing the R CMD CHECK, R CMD BUILD,
| R CMD INSTALL sequence gives an installed package with the
| the two classes invisible. So I don't think this is a devtools issue.
|
| This is rather frustrating, I agree, especially as it worked previously.
Agreed. But some things also changed in R ... and I am not entirely what the
actual cause was for the change you see.
| Is there any documentation (preferably short of going through the Rcpp
| modules
| code) on the process by which these symbols get exported into R. The
| symbols are in the .so file, but for instance, where/when would the
| function R_init_falconnr function, which is created by
| Rcpp::compileAttributes
| and seems to load the modules, be called?
Well, I also find that combining Modules and compileAttributes() can be
finicky. If output for this was was generated, you could place it e.g. in
src/init.c -- R calls the resulting function on package load. But, and that
is a big 'but' you don't have to unless you say .registration=TRUE in
useDynLib() in NAMESPACE. And I seem to recall I actually wrote the
src/init.c for RcppAnnoy "by hand" as the generator in R itself does not know
about Modules. We may need to add a tool here.
And again, note that you do not need src/init.c -- you only need it if you
set .registration=TRUE. So as first step, I'd say 'do not do that'. I know
it sounds silly -- but if I were you I'd start from the working Modules
example eg in the Rcpp unittests and would then re-add my code. Slowly, and
step by step. That way you will know which change breaks it.
And by all I means I don't claim this is ideal -- if we learn something along
the way we can try to make Modules better. It remains a useful yet somewhat
obscure and underdocumented tool.
Dirk
|
| Thanks, Chris
|
|
|
| >
| > | Thanks, Dirk.
| > |
| > | What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we
| > are
| > | > meant to create a tar.gz in either way via 'R CMD BUILD' and then
| > install
| > | > it
| > | > via 'R CMD INSTALL'.
| > | >
| > | > Are you doing that?
| > | >
| > |
| > | Yes. I usually use devtools, but I've done it directly as well.
| >
| > Take this up with the devtools developers then, please.
| >
| > I operate with R command, and possibly very thin littler wrappers such as
| > build.r and rcc.r to, respectively, build and check.
| >
| > | In #1, for instance: I go to the package source directory, start R,
| > | do devtools::build and devtools::check. This builds and compiles
| > | the package. After library() at this point, the module components
| > | are visible, e.g., LshParameterSetter.
| > |
| > | In #2, for instance, I do devtools::install() or
| > devtools::install_github()
| > | or similar.
| > | Quit R, go to some other directory, restart R, and call library().
The R
| > | entry
| > | points are visible, but none of the module classes.
| > |
| > | I'm unclear as to what in the installation/export process can explain the
| > | difference
| > | between #1 and #2.
| >
| > Again, I have no idea what devtools does here. Other people may.
| >
| > The Rcpp documentation does not cover devtools so you are on your own.
| >
| > | Before emailing yesterday, I had tried removing the
| > RCPP_EXPOSED_CLASS()'s,
| > | but then the code does not compile. There is a static_assert error in
| > | wrap_dispatch_unknown_iterable in Rcpp's wrap.h (line 520). I did not
| > see an
| > | easy way to fix that. (I vaguely recall that I only included them
| > | originally because
| > | I had to.)
| >
| > I think you need those. I just glance at the different variants on the
| > train
| > in, they are mostly equivalent and just layers of each other.
| >
| > | I will look RcppAnnoy and see if I can find a way around using these
| > | macros.
| >
| > Sounds good.
| >
| > Modules can be frustrating. I just tried with some larger code at work,
| > and
| > it doesn't load :-/ Something else may get in the way but hard to tell.
| >
| > Rcpp Attributes is just fine.
| >
| > Dirk
| >
| > | Thanks again, Chris
| > |
| > |
| > |
| > |
| > |
| > | >
| > | > | Hello,
| > | > |
| > | > | I have a package under development that uses Rcpp modules to expose
| > | > | some C++ classes. I hadn't touched the package for some months (close
| > | > | to a year). At that time the package compiled and installed without
| > | > problem
| > | > | and worked well. I could access the classes as expected. Recently,
| > having
| > | > | updated Rcpp and R in the meantime, I reinstalled the package with
| > | > |
| > | > | 1. If building and loading from within the package directory, all
| > | > works
| > | > | fine.
| > | > | 2. When installing the package, either from github, locally, or
| > | > | otherwise,
| > | > | the modules are *not* visible.
| > | >
| > | > What exactly do you mean by 1. and 2. ? Per 'Writing R Extensions' we
| > are
| > | > meant to create a tar.gz in either way via 'R CMD BUILD' and then
| > install
| > | > it
| > | > via 'R CMD INSTALL'.
| > | >
| > | > Are you doing that?
| > | >
| > | > | The package passes devtools:check() and compiles fine.
| > | >
| > | > Well if it passes check you should be fine.
| > | >
| > | > | For instance, for one of those classes, LshParameterSetter, I do
| > | > |
| > | > |
| > | > | RCPP_EXPOSED_CLASS(LshParameterSetter)
| > | > |
| > | > | RCPP_MODULE(mod_params) {
| > | > | class_<LshParameterSetter>("LshParameterSetter")
| > | > |
| > | > | .constructor<int,int>()
| > | > |
| > | > | .method("withDefaults", &LshParameterSetter::withDefaults,
| > | > | "Fill with defaults")
| > | > | .method("distance", &LshParameterSetter::distance,
| > | > | "Set distance metric")
| > numHashFunctions,
| > | > | "Set numbers of hash functions and tables")
numHashTables,
| > | > | "Set numbers of hash functions and tables")
| > | > | .method("storage", &LshParameterSetter::storage,
| > | > | "Set LSH Hash Table Storage type")
| > | > | .method("family", &LshParameterSetter::family,
| > | > | "Set LSH Family")
| > | > | .method("rotations", &LshParameterSetter::rotations,
| > | > | "Number of rotations")
| > | > | .method("asList", &LshParameterSetter::asList,
| > | > | "List of parameter values by name")
| > | > | ;
| > | > | }
| > | > |
| > | > | and similarly for the other classes. This module is loaded in the R
| > code
| > | > | with
| > | > | Rcpp::loadModule("mod_params", TRUE)
| > | > | and similarly for the other modules.
| > | > |
| > | > | The installed package has the .rdb, .rdx, and .so file, so it's hard
| > to
| > | > see
| > | > | if something has been lost in the installation. There's no
| > indication
| > | > what
| > | > | is wrong, but after loading the installed package, a class like
| > | > | LshParameterSetter
| > | > | is not defined. (Loading from within the package source directory it
| > is,
| > | >
| > | > What do you mean by 'loading from within the package source directory'
| > ?
| > | >
| > | > | showing
| > | > |
| > | > | > > LshParameterSetter
| > | > | > C++ class 'LshParameterSetter' <0x7fde72fb1940>
| > | > | > LshParameterSetter(int, int)
| > | > | >
| > | > | > Fields: ...
| > | > | >
| > | > | as it should be.) I've done this with simple skeletons and not had a
| > | > | similar problem. But I don't see a qualitative difference between
| > these
| > | > | cases.
| > | > |
| > | > | The full package code is at https://github.com/genovese/
falconnr
| > | > | and the session info is below.
| > | >
| > | > Thanks for posting the link. Not a small package ...
| > | >
| > | > | I'd appreciate any pointers on this, as the problem is unclear to me,
| > | > | especially since it worked without problem last year. I'm happy to
| > | > | provide any useful information that I might have omitted here.
| > | >
| > | > Not sure what is going on there. Package clearly builds. Testing
| > complains
| > | > some but hey, it is not a CRAN package yet...
| > | >
| > | > You have
| > | >
| > | > falconnr::LshTable
| > | >
| > | >
| > | > falconnr:::LshNnTable
| > | > falconnr:::LshParameterSetter
| > | >
| > | > I have been using Modules for about as long as we've had them, but I
| > have
| > | > always been a little mystified by the various UPPERCASE_MACROS
| > exporting
| > | > things. Looking at what was probably my first Modules package,
| > RcppBDT, I
| > | > seem to "simply" do
| > | >
| > | > RCPP_MODULE(bdtDdMod) {
| > | > Rcpp::class_<bdtDd>("bdtDd")
| > | > // ... stuff omitted
| > | >
| > | > and nothing else for the four or so different modules there. That
| > worked.
| > | > I
| > | > think you may rely on a helper macro which may have gotten stale as the
| > | > _underlying R behaviour_ may have changed. I think I use one or two
| > of the
| > | > exporter macros in other places (ie RcppAnnoy) so you could try that.
| > | >
| > | > In general, I always recommend to simplify as much as you can.
| > | >
| > | > Hope this helps, Dirk
| > | >
| > | > | Thanks for your help, Chris
| > | > |
| > | > | > sessionInfo()
| > | > | R version 3.3.2 (2016-10-31)
| > | > | Platform: x86_64-apple-darwin13.4.0 (64-bit)
| > | > | Running under: macOS Sierra 10.12.6
| > | > |
| > | > | [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-
8
| > | > |
| > | > | [1] grid stats graphics grDevices utils datasets
| > methods
| > | > | base
| > | > |
| > | > | [1] falconnr_0.0.0.9001 MASS_7.3-45 lattice_0.20-34
| > | > |
| > | > | [1] Rcpp_0.12.13 codetools_0.2-15
| > | > | _______________________________________________
| > | > | Rcpp-devel mailing list
| > | > | https://lists.r-forge.r-project.org/cgi-bin/mailman/
| > listinfo/rcpp-devel
| > | >
| > | > --
| > | >
| >
| > --
| >
--