Overview
The reticulate package provides an R interface toPython modules, classes, and functions. For example, this code importsthe Python os
module and calls some functions withinit:
library(reticulate)os <- import("os")os$listdir(".")
[1] ".git" ".gitignore" ".Rbuildignore" ".RData" [5] ".Rhistory" ".Rproj.user" ".travis.yml" "appveyor.yml" [9] "DESCRIPTION" "docs" "external" "index.html" [13] "index.Rmd" "inst" "issues" "LICENSE" [17] "man" "NAMESPACE" "NEWS.md" "pkgdown" [21] "R" "README.md" "reticulate.Rproj" "src" [25] "tests" "vignettes"
Functions and other data within Python modules and classes can beaccessed via the $
operator (analogous to the way you wouldinteract with an R list, environment, or reference class).
The reticulate package is compatible with allversions of Python >= 2.7. Integration with NumPy is optional andrequires NumPy >= 1.6.
Python Version
By default, reticulate uses the version of Python found on yourPATH
(i.e.Sys.which("python")
). Theuse_python()
function enables you to specify an alternateversion, for example:
library(reticulate)use_python("/usr/local/bin/python")
The use_virtualenv()
and use_condaenv()
functions enable you to specify versions of Python in virtual or condaenvironments, for example:
library(reticulate)use_virtualenv("myenv")
See the article on Python VersionConfiguration for additional details.
Python Packages
You can install any required Python packages using standard shelltools like pip
and conda
. Alternately,reticulate includes a set of functions for managing and installingpackages within virtualenvs and Conda environments. See the article onInstallingPython Packages for additional details.
Type Conversions
When calling into Python, R data types are automatically converted totheir equivalent Python types. When values are returned from Python to Rthey are converted back to R types. Types are converted as follows:
R | Python | Examples |
---|---|---|
Single-element vector | Scalar | 1 , 1L , TRUE ,"foo" |
Multi-element vector | List | c(1.0, 2.0, 3.0) , c(1L, 2L, 3L) |
List of multiple types | Tuple | list(1L, TRUE, "foo") |
Named list | Dict | list(a = 1L, b = 2.0) ,dict(x = x_data) |
Matrix/Array | NumPy ndarray | matrix(c(1,2,3,4), nrow = 2, ncol = 2) |
Data Frame | Pandas DataFrame | data.frame(x = c(1,2,3), y = c("a", "b", "c")) |
Function | Python function | function(x) x + 1 |
Raw | Python bytearray | as.raw(c(1:10)) |
NULL, TRUE, FALSE | None, True, False | NULL , TRUE , FALSE |
If a Python object of a custom class is returned then an R referenceto that object is returned. You can call methods and access propertiesof the object just as if it was an instance of an R reference class.
Importing Modules
The import()
function can be used to import any Pythonmodule. For example:
difflib <- import("difflib")difflib$ndiff(foo, bar)filecmp <- import("filecmp")filecmp$cmp(dir1, dir2)
The import_main()
and import_builtins()
functions give you access to the main module where code is executed bydefault and the collection of built in Python functions. Forexample:
main <- import_main()builtins <- import_builtins()builtins$print('foo')
The main module is generally useful if you have executed Python codefrom a file or string and want to get access to its results (see thesection below for more details).
Sourcing Scripts
The source_python()
function will source a Python scriptand make the objects it creates available within an R environment (bydefault the calling environment). For example, consider the followingPython script:
def add(x, y): return x + y
We source it using the source_python()
function and thencan call the add()
function directly from R:
source_python('add.py')add(5, 10)
[1] 15
Executing Code
You can execute Python code within the main module using thepy_run_file
and py_run_string
functions. Youcan then access any objects created using the py
objectexported by reticulate:
library(reticulate)py_run_file("script.py")py_run_string("x = 10")# access the python main module via the 'py' objectpy$x
Object Conversion
By default when Python objects are returned to R they are convertedto their equivalent R types. However, if you’d rather make conversionfrom Python to R explicit and deal in native Python objects by defaultyou can pass convert = FALSE
to the import
function. In this case Python to R conversion will be disabled for themodule returned from import
. For example:
# import numpy and specify no automatic Python to R conversionnp <- import("numpy", convert = FALSE)# do some array manipulations with NumPya <- np$array(c(1:4))sum <- a$c*msum()# convert to R explicitly at the endpy_to_r(sum)
As illustrated above, if you need access to an R object at end ofyour computations you can call the py_to_r()
functionexplicitly.
Getting Help
You can print documentation on any Python object using thepy_help()
function. For example:
os <- import("os")py_help(os$chdir)
Lists, Tuples, and Dictionaries
The automatic conversion of R types to Python types works well inmost cases, but occasionally you will need to be more explicit on the Rside to provide Python the type it expects.
For example, if a Python API requires a list and you pass a singleelement R vector it will be converted to a Python scalar. To overcomethis simply use the R list
function explicitly:
foo$bar(indexes = list(42L))
Similarly, a Python API might require a tuple
ratherthan a list. In that case you can use the tuple()
function:
tuple("a", "b", "c")
R named lists are converted to Python dictionaries however you canalso explicitly create a Python dictionary using the dict()
function:
dict(foo = "bar", index = 42L)
This might be useful if you need to pass a dictionary that uses amore complex object (as opposed to a string) as its key.
Numeric Types and Indexes
R and Python have different default numeric types. If you write42
in R it is considered a floating point number whereas42
in Python is considered an integer.
This means that when a Python API expects an integer, you need to besure to use the L
suffix within R. For example, if thefoo
function requires an integer as its index
argument you would do this:
foo$bar(index = 42L)
Python collections are addressed using 0-based indices rather thanthe 1-based indices you might be familiar with from R. So to address thefirst item of an array in R you would write:
items[[1]]
Whereas if you are calling a method in Python via reticulate thattakes an index you would write this to address the first item:
items$get(0L)
Note the use of the 0-based index as well as the L
toindicate t that the value is an integer.
Arrays
R matrices and arrays are converted automatically to and from NumPy arrays.
When converting from R to NumPy, the NumPy array is mapped directlyto the underlying memory of the R array (no copy is made). In this case,the NumPy array uses a column-based in memory layout that is compatiblewith R (i.e.Fortran style rather than C style). When converting fromNumPy to R, R receives a column-ordered copy of the NumPy array.
You can also manually convert R arrays to NumPy using thenp_array()
function. For example, you might do this if youneeded to create a NumPy array with C rather than Fortran stylein-memory layout (for higher performance in row-oriented computations)or if you wanted to control the data type of the NumPy array moreexplicitly. Here are some example uses of np_array()
:
a <- np_array(c(1:8), dtype = "float16")a <- np_array(c(1:8), order = "C")
Reasoning about arrays which use distinct in-memory orders can betricky. The Arrays in R and Python articleprovides additional details.
Also, always remember that when calling NumPy methods array indicesare 0 rather than 1 based and require the L
suffix toindicate they are integers.
Data Frames
R data frames can be automatically converted to and from Pandas DataFrames. By default,columns are converted using the same rules governing R array <->NumPy array conversion, but a couple extensions are provided:
R | Python |
---|---|
Factor | Categorical Variable |
POSIXt | NumPy array with dtype = datetime64[ns] |
If the R data frame has row names, the generated Pandas DataFramewill be re-indexed using those row names (and vice versa). Specialhandling is also available for a DatetimeIndex
associatedwith a Pandas DataFrame; however, because R only supports charactervectors for row names they are converted to character first.
Using Pandas nullable data types
Pandas has experimental support for nullable data types. Those datatypes have built-in support for missing values, represented bypd.NA
and using them allows us to better represent RNA
values.
Users can opt-in to use Pandas nullable data types instead of numpyarrays by setting the reticulate.pandas_use_nullable_dtypes
to TRUE
. For example:
df <- data.frame( int = c(NA, 1:4), num = c(NA, rnorm(4)), lgl = c(NA, rep(c(TRUE, FALSE), 2)), string = c(NA, letters[1:4]))options(reticulate.pandas_use_nullable_data_types = TRUE)r_to_py(df)#> int num lgl string#> 0 <NA> <NA> <NA> <NA>#> 1 1 -0.697855 True a#> 2 2 -0.253042 False b#> 3 3 0.385421 True c#> 4 4 0.519933 False d
Sparse Matrices
Sparse matrices created by Matrix R packagecan be converted ScipyCSC matrix, and vice versa. This is often useful when you want topass sparse matrices to Python functions that accepts Scipy CSC matrixto take advantage of this format, such as efficient column slicing andfast matrix vector products.
For example, we first create a sparse matrix usingMatrix::sparseMatrix()
:
library(Matrix)N <- 5dgc_matrix <- sparseMatrix( i = sample(N, N), j = sample(N, N), x = runif(N), dims = c(N, N))
The sparse matrix looks like this:
> dgc_matrix5 x 5 sparse Matrix of class "dgCMatrix" [1,] 0.2264952 . . . . [2,] . . . . 0.3927282[3,] . . . 0.9215908 . [4,] . . 0.01777771 . . [5,] . 0.05885743 . . .
Let’s convert it to Scipy CSC matrix usingr_to_py()
:
> csc_matrix <- r_to_py(x)> csc_matrix (0, 0) 0.226495201467 (4, 1) 0.0588574311696 (3, 2) 0.0177777127828 (2, 3) 0.921590822982 (1, 4) 0.392728160601
Note that the right-hand side contains the non-zero entries of thematrix while the left-hand side represents their locations in thematrix.
We can also use py_to_r()
to convert the CSC matrix backto Matrix::dgCMatrix
representation that can then bemanipulated easily in R which is the same as the original sparse matrixthat we created earlier using Matrix::sparseMatrix()
:
> py_to_r(csc_matrix)5 x 5 sparse Matrix of class "dgCMatrix" [1,] 0.2264952 . . . . [2,] . . . . 0.3927282[3,] . . . 0.9215908 . [4,] . . 0.01777771 . . [5,] . 0.05885743 . . .
With Contexts
The R with
generic function can be used to interact withPython context manager objects (in Python you use the with
keyword to do the same). For example:
py <- import_builtins()with(py$open("output.txt", "w") %as% file, { file$write("Hello, there!")})
This example opens a file and ensures that it is automatically closedat the end of the with block. Note the use of the %as%
operator to alias the object created by the context manager.
Iterators
If a Python API returns an iterator or a generator, you can interactwith it using the iterate()
function. Theiterate()
function can be used to apply an R function toeach item yielded by the iterator:
iterate(iter, print)
If you don’t pass a function to iterate
the results willbe collected into an R vector:
results <- iterate(iter)
Note that the Iterators
will be drained of their valuesby iterate()
:
a <- iterate(iter) # results are not emptyb <- iterate(iter) # results are empty since items have already been drained
Element Level Iteration
You can also iterate on an element-by-element basis using theiter_next()
function. For example:
while (TRUE) { item <- iter_next(iter) if (is.null(item)) break}
By default iter_next()
will return NULL
when the iteration is complete but you can provide a customcompleted
value it will be returned instead. Forexample:
while (TRUE) { item <- iter_next(iter, completed = NA) if (is.na(item)) break}
Note that some iterators/generators in Python are infinite. In thatcase the caller will need custom logic to determine when to terminatethe loop.
Generators
Python generators arefunctions that implement the Python iterator protocol. Similarly, thereticulate generator()
function enables you to create aPython iterator from an R function.
In Python, generators produce values using the yield
keyword. In R, values are simply returned from the function. One benefitof the yield
keyword is that it enables successiveiterations to use the state of previous iterations. In R, this can bedone by returning a function that mutates its enclosing environment viathe <<- operator. For example:
# define a generator functionsequence_generator <-function(start) { value <- start function() { value <<- value + 1 value }}# convert the function to a python iteratoriter <- py_iterator(sequence_generator(10))
If you want to indicate the end of the iteration, returnNULL
from the function:
sequence_generator <-function(start) { value <- start function() { value <<- value + 1 if (value < 100) value else NULL }}
Note that you can change the value that indicates the end of theiteration using the completed
parameter(e.g.py_iterator(func, completed = NA)
).
Functions
Signatures
By default R functions are converted to Python with a genericsignature (function(...)
), where there’s neither keywordargument nor default values for arguments.
For example, below we apply r_to_py()
to an R functionand then we use inspect Pythonmodule to get the converted function’s argument spec. You can seethat the signature of the wrapped function looks different than theoriginal R function’s signature.
> inspect <- import("inspect")> converted_func <- r_to_py(function(a, b = 1.5) {})> inspect$getargspec(converted_func)ArgSpec(args=[], varargs='args', keywords='kwargs', defaults=None)
This default conversion typically works fine, however some Pythonlibraries have strict checking on the function signatures of userprovided callbacks. In these cases the genericfunction(...)
signature will fail this checking.
For these cases you can use py_func()
to wrap the Rfunction so that the wrapped function has exactly the same signature asthat of the original R function, e.g.one argument a
without default value and another argument b
with defaultvalue 1.5.
> wrapped_func <- py_func(function(a, b = 1.5) {})> inspect$getargspec(wrapped_func)ArgSpec(args=['a', 'b'], varargs=None, keywords=None, defaults=(1.5,))
Note that the signature of the R function must not contain esotericPython-incompatible constructs. For example, we cannot have R functionwith signature like function(a = 1, b)
since Pythonfunction requires that arguments without default values appear beforearguments with default values.
Background Threads
In some cases Python libraries will invoke callbacks on a Pythonbackground thread. Since R code must run on the main thread, this won’twork by default when you pass an R function as a callback.
To work around this, you can use py_main_thread_func()
,which will provide a special wrapper for your R function that ensures itwill only be called on the main thread.
Advanced
There are several more advanced functions available that are usefulprincipally when creating high level R interfaces for Pythonlibraries.
Python Objects
Typically interacting with Python objects from R involves using the$
operator to access whatever properties for functions ofthe object you need. When using the $
, Python objects areautomatically converted to their R equivalents when possible. Thefollowing functions enable you to interact with Python objects at alower level (e.g.no conversion to R is done unless you explicitly callthe py_to_r
function):
Function | Description |
---|---|
py_has_attr() | Check if an object has a specified attribute. |
py_get_attr() | Get an attribute of a Python object. |
py_set_attr() | Set an attribute of a Python object. |
py_list_attributes() | List all attributes of a Python object. |
py_len() | Length of Python object. |
py_call() | Call a Python callable object with the specified arguments. |
py_to_r() | Convert a Python object to its R equivalent |
r_to_py() | Convert an R object to its Python equivalent |
Pickle
You can save and load Python objects (via pickle) using thepy_save_object
and py_load_object
functions:
Function | Description |
---|---|
py_save_object() | Save a Python object to a file with pickle. |
py_load_object() | Load a previously saved Python object from a file. |
Configuration
The following functions enable you to query for information about thePython configuration available on the current system.
Function | Description |
---|---|
py_available() | Check whether a Python interface is available on this system. |
py_numpy_available() | Check whether the R interface to NumPy is available (requires NumPy>= 1.6) |
py_module_available() | Check whether a Python module is available on this system. |
py_config() | Get information on the location and version of Python in use. |
Output Control
These functions enable you to capture or suppress output fromPython:
Function | Description |
---|---|
py_capture_output() | Capture Python output for the specified expression and return it asan R character vector. |
py_suppress_warnings() | Execute the specified expression, suppressing the display Pythonwarnings. |
Miscellaneous
The functions provide miscellaneous other lower-levelcapabilities:
Function | Description |
---|---|
py_set_seed() | Set Python and NumPy random seeds. |
py_unicode() | Convert a string to a Python unicode object. |
py_str() , py_repr() | Get the string representation of Python object. |
py_id() | Get a unique identifier for a Python object |
py_is_null_xptr() | Check whether a Python object is a null externalptr. |
py_validate_xptr() | Check whether a Python object is a null externalptr and throw anerror if it is. |
Learning More
The following articles cover additional aspects of usingreticulate:
R Markdown Python Engine
Python Version Configuration
Installing PythonPackages
Using reticulate in an RPackage
Arrays in R and Python