Skip test on M1mac in R

user_5967807 0
Viewed 185

According to Carl Boettiger in this thread, "...when tests fail. Like Solaris, some of these failures can occur when an upstream dependency installs on the platform but does not actually run." My code fails numerically on M1mac, but not on other platforms, while using stats::integrate on functions returning very small values.

Should I skip the test on M1mac (arm64)?

require(testthat)
test_that("correct numeric solution", {
     skip_on_os("mac", arch = "aarch64")

     # code of the test using expect_equal()
})

Alternatively, can adjusting the tolerance argument in expect_equal() help resolve the specific issue on the one system? If yes, to what value should tolerance be changed, if my results is at 1e-9 (using tolerance = 1e-6 now, and the test fails)?

In more general terms, what is the best coding practice (or a long term solution) for R packages on CRAN to solve problems with specific tests failing on one OS?

apple-m1 testthat arm64 r
2 Answers

How other programmers do it

I hesitate to tell about canonical packages, but there's a handful of pretty recognizable packages that skip certain operating systems for particular tests:

A couple of examples from the repositories above:

  1. Example 1. Skips core functionality tests on Solaris platform.

  2. Example 2. Skips installation tests for Linux (other tests in this project seemingly cover all operating systems).

What the documentation says

This is an official article of the testthat package. It clearly states the circumstances under which you might better skip the test; however, those statements tend to be more recommendatory rather than imperative:

You’re testing a web service that occasionally fails, and you don’t want to run the tests on CRAN. Or maybe the API requires authentication, and you can only run the tests when you’ve securely distributed some secrets.

You’re relying on features that not all operating systems possess, and want to make sure your code doesn’t run on a platform where it doesn’t work. This platform tends to be Windows, since amongst other things, it lacks full utf8 support.

You’re writing your tests for multiple versions of R or multiple versions of a dependency and you want to skip when a feature isn’t available. You generally don’t need to skip tests if a suggested package is not installed. This is only needed in exceptional circumstances, e.g. when a package is not available on some operating system.

I've highlighted everything that had been written regarding operating systems. From your question, I can conclude that your situation falls under the statement of "You’re relying on features that not all operating systems possess", since you've most likely encountered a bug in the M1 Mac operating system since macOS build has a slightly different way of calculating extended-precision floating-point numbers[1]:

The ‘native’ build is a little faster (and for some tasks, considerably so) but may give different numerical results from the far more common ‘x86_64’ platforms (on macOS and other OSes) as ARM hardware lacks extended-precision floating-point operations.

What the ideology adheres to

It's worth our time to recall why unit tests were invented in the first place.

In spite of all the talks about Wikipedia's unreliability, I believe this source is being considered "canonical" by the vast majority of my colleagues, which states:

Unit tests are typically automated tests written and run by software developers to ensure that a section of an application (known as the "unit") meets its design and behaves as intended.

In order to answer your question from the ideological perspective, we have to answer only one question: does your code currently work as intended?

If you consider your functionality to be comprehensive enough for the end user, even though it partially doesn't work properly on a particular OS, feel free to skip the test.

If not, it implies that your code contains bug that is to be fixed before going into the production. In this case, once you fix it, the tests should succeed with your problematic OS.

My code fails numerically on M1mac, but not on other platforms, while using stats::integrate on functions returning very small values.

My own (probably biased) opinion

Based on your words, it's hard to understand which one is the case, but I believe that if your package is viable for the 99.99% of the audience, go ahead and skip this annoying test for the 0.01 remaining percentile of the possible environments. Maybe you should note somewhere in the README.MD that your package has this issue.

This way other developers will be aware of it, and those are using M1Mac OS will most likely find a workaround or fix it themselves - in case you're creating an open-source project.

Notes:

[1]. Thanks to Roland's comment, I've updated my answer.

user_37856180

Speaking as a package developer (but not a CRAN expert by any means), my advice would be to reformulate (not skip) the test. You can use the test as an opportunity to document the issue. Here are the steps I would take:

  • Define platform-specific assertions. On M1 Mac, assert that the approximation is not equal to the exact value of the integral at the tolerance that you are using for other platforms, by wrapping the failing expect_equal call inside of expect_error. Then assert that the two are equal at some minimal, greater tolerance with a less strict expect_equal call. Your test block would contain something like this:

    x <- approximate_integral_value
y <- exact_integral_value

## If testing on M1 Mac
if (tolower(Sys.info()[["sysname"]]) == "darwin" && R.version[["arch"]] == "aarch64") {
  ## Expect strict test to fail
  expect_error(expect_equal(x, y, tolerance = 1e-9)) 
  ## Expect less strict test to pass
  expect_equal(x, y, tolerance = 1e-4)
## Otherwise
} else {
  ## Expect strict test to pass
  expect_equal(x, y, tolerance = 1e-9)
}

    This way, you will detect if the problem resolves itself (the first expect_equal will pass, causing the expect_error to fail) and you will detect if the problem gets worse (the second expect_equal will fail). In both cases, you would update the test code.

  • Leave a comment near the expect_error call explaining why the integral approximation is inaccurate on M1 Macs and describe any attempts you've made to work around the underlying numerical issue (or why it cannot be resolved).

  • If you think that functions in your package might not work on M1 Macs because this test is failing, then use warning or stop inside of those functions to let users know. If those functions are central to your package's functionality, then add a platform note somewhere more visible (e.g., on the landing page of your package website).

As for your numerical issue:

  • Try to find a more stable algorithm for computing the integrand. For example, it might help to replace prod(x) with exp(sum(log(x))).

  • Experiment with the optional arguments of integrate, namely rel.tol and abs.tol.

FWIW, there are dedicated forums for these kinds of questions, namely the R-package-devel and R-SIG-Mac mailing lists:

  • R-package-devel would be appropriate for the question, "What is the best practice for handling this package development problem?" There, you are more likely to get an answer directly from a CRAN maintainer.

  • R-SIG-Mac would be appropriate for the question, "Why am I encountering this behaviour only on M1 Mac?" There, you are more likely to get an answer from an R Core Team member who develops R for Macs. [Edit: @Roland points out in the comments that native R builds for ARM-based platforms do not support extended precision arithmetic.]

user_126857680
Related