How to run R code in Python

R is freely available language and computational tool which is very popular among the statisticians and bioinformatics community. In this post I am going to list the steps to run R commands through python on Windows. On Linux it is much simple to setup but the steps are almost similar.

Requirements:

  1. R: https://cran.r-project.org/
  2. Python 2.7, 3.4, 3.5 or 3.6: https://www.continuum.io/downloads
  3. rpy2 pip package (Windows only): http://www.lfd.uci.edu/~gohlke/pythonlibs/#rpy2
My configuration is R version 3.4.1 (https://cran.r-project.org/bin/windows/base/R-3.4.1-win.exe) and Python 3.5 Anaconda x64: https://repo.continuum.io/archive/Anaconda3-4.2.0-Windows-x86_64.exe

Installation Instructions:

  1. Install R
  2. Install Anaconda
  3. Edit Environment Variables (Control Panel -> System -> Advanced system settings -> Environment Variables)
    • Select Path -> Edit and add R installation path (default location is "C:\Program Files\R\R-3.4.1\bin\x64") to Path variable and press OK
    • Select New and add R_HOME in variable name and base R directory (default is "C:\Program Files\R\R-3.4.1") to variable value
    • Select New and add R_USER in variable name and your Windows user name (You can check this by typing echo %username% in command window) to variable value
  4. Open Windows command window as admin (or launch terminal on linux as root) and enter the following commands.
    D:\Downloads> pip install rpy2-2.8.6-cp35-cp35m-win_amd64.whl
    *On Linux just use the command pip install rpy2, environment variables should already be set otherwise you can set them by using export command.
  5. After successful install you can run the following commands to test if you can run R code in python
     C:\> python
     >>> import rpy2
     >>> print(rpy2.__version__)
     2.8.6
     >>> from rpy2.rinterface import R_VERSION_BUILD
     >>> print(R_VERSION_BUILD)
     ('3', '3.3', '', 72310)
     >>> import rpy2.robjects as robjects
     >>> pi = robjects.r['pi']
     >>> pi[0]
     3.141592653589793
    
    You should be able to run R commands from Python after this step. See https://rpy2.readthedocs.io/en/version_2.8.x/index.html for further help.

How to run MATLAB code from Python

Python is a free open source programming language and can be used to solve a number of problems from research purposes to web development and big data analysis. MATLAB provides a list of professionally developed specialized toolboxes for a number of problems. It is sometimes required to use MATLAB code in your python script. I have listed below a set of instructions to run MATLAB code in python.
NOTE: To run MATLAB code using this method you will still need MATLAB licence.

Requirements:

  1. Matlab version 2014b or above.
  2. Python 2.7, 3.4 or 3.5
  3. My configuration is MATLAB 2017a and Python 3.5 Anaconda x64: https://repo.continuum.io/archive/Anaconda3-4.2.0-Windows-x86_64.exe

Installation Instructions:

  1. Run MATLAB and in command window enter the following command
    >> matlabroot 
    
    ans = 
    
       'C:\Program Files\MATLAB\R2017a'
    
  2. C:\Program Files\MATLAB\R2017a' is the default matlabroot folder on Windows. This may be different on your system.
  3. Open Windows command window (or launch terminal on linux) in matlabroot folder and enter the following commands.
    C:\Program Files\MATLAB\R2017a> cd extern\engines\python
    C:\Program Files\MATLAB\R2017a\extern\engines\python> python setup.py install 
    If you do not have access permissions then (a) you can run command window as administrator (or as root in linux) and run the above commands or (b) you can install MATLAB API in non default locations by using the following command
    C:\Program Files\MATLAB\R2017a\extern\engines\python> python setup.py build --build-base="builddir" install --prefix="installdir" 
    where "builddir" and "installdir" are non-default build and installation folders.
  4. After successful install you can run the following commands to test if you can run MATLAB code in python
     C:\> python
     >>> import matlab.engine
     >>> eng = matlab.engine.start_matlab()
     >>> tf = eng.isprime(37)
     >>> print(tf)
     True
     >>> eng.eval("figure", nargout=0) # To show graphics figure
    
    You should be able to see a graphics figure when you run the last command above.

Compile MatConvNet on Windows

Matconvnet is a convolutional neural network (CNN) toolbox for Matlab. Being integrated into Matlab it gives you the flexibility to use Matlab built in functions but it's execution time is slightly slower compared to C++ based caffe and other deep learning libraries. In addition, you need Matlab licence to run this library. Similar to other Matlab toolboxes it's relatively easier to compile compared to open source caffe and tensorflow. I have listed the set of instructions to compile MatConvNet on a Windows machine.

Requirements:

  1. Matlab version 2017a
  2. MatConvNet : http://www.vlfeat.org/matconvnet/, http://www.vlfeat.org/matconvnet/download/matconvnet-1.0-beta24.tar.gz
  3. Visual Studio 2015: https://www.visualstudio.com/downloads/, https://www.visualstudio.com/vs/older-downloads/
  4. CUDA 8.0: (For GPU Support only) https://developer.nvidia.com/cuda-downloads
  5. cuDNN v5.1 (For GPU support only): https://developer.nvidia.com/cudnn

Instructions:

  1. Extract MatConvNet to a folder.
  2. Install Visual Studio. Make sure you select the option for programming languages when you install visual studio otherwise Matlab will not be able to find the compiler.
  3. Run Matlab and setup MatConvnet extracted folder as working directory.
  4. In command window enter the following command to setup Visual Studio as your compiler
    >> mex -setup C++
    MEX configured to use 'Microsoft Visual C++ 2015 Professional' for C++ language compilation.
  5. Now add path to 'matlab' folder in the extracted library and compile
    >> addpath matlab 
    >> vl_compilenn('verbose', 1)
  6. At this point MatConvNet should be succesfully compiled to run on CPU. To test this run the following commands.
    >> run matlab\vl_setupnn.m 
    >> vl_testnn
  7. GPU Support

  8. Install CUDA v8.0
  9. Extract cuDNN
    • copy all files from bin folder to 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin'
    • similarly from include to 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include' 
    • and 'lib\x64' to  ''C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64'.
  10. Follow step 1 to 5 and enter the following command
    >> vl_compilenn('enableGpu', true, ...
    'cudaRoot', 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0', ...
    'cudaMethod', 'nvcc', ... 
    'enableCudnn', true, ... 
    'cudnnRoot', 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0');
  11. After succesfull build you can test the mex files by entering the command below
    >> vl_testnn('gpu', true); 
    

  

Install Tensorflow on Windows

The instructions below can be used to install precompiled version of tensorflow. To build tensorflow from source please follow the instructions in my previous post <http://www.shaneahmed.com/2017/07/build-tensorflow-on-windows.html>. After successful build you can follow the same instructions as for the precompiled version below. Make sure you use the correct path to tensorflow pip package .whl file.

Requirements:

  1. Python 3.5 Anaconda x64: https://repo.continuum.io/archive/Anaconda3-4.2.0-Windows-x86_64.exe
  2. CUDA 8.0 (For GPU Support):  https://developer.nvidia.com/cuda-downloads
  3. cuDNN v5.1 (For GPU support) : https://developer.nvidia.com/cudnn

Installation instructions:

  1. Install Anaconda in "C:\tools\", make sure you select Register Anaconda as system path for Python 3.5.
  2. Open command window. (Win+R -> cmd -> Enter)
  3. Enter the following commands
    C:\> conda create -n tensorflowCPU python=3.5
  4. Enter 'y' if asked to continue.
  5. Activate conda virtual environment by entering the following command.
    C:\> activate tensorflowCPU
  6. Enter the appropriate command below to install particular version of tensorflow.

    For latest version of tensorflow use the following command.

    (tensorflowCPU) C:\> pip install --upgrade tensorflow

    For version 1.2 of tensorflow use the following command. You can find the appropriate address for each version on pages with instructions at this link <https://www.tensorflow.org/versions/>

    (tensorflowCPU) C:\> pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.2.1-cp35-cp35m-win_amd64.whl
  7. Enter the following commands to run tensorflow using python
    (tensorflowCPU) C:\> python
     >>> import tensorflow as tf
     >>> hello = tf.constant('Hello, TensorFlow!')
     >>> sess = tf.Session()
     >>> print(sess.run(hello))
     'Hello, TensorFlow!'
    
  8. The command should print 'Hello, TensorFlow!' if you have successfully installed tensorflow.

      For GPU support:

  1. Install CUDA v8.0.
  2. Extract cuDNN.
    • copy all files from bin folder to 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin'
    • similarly from include to 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include' 
    • and 'lib\x64' to  ''C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64'.
  3. Open command window. (Win+R -> cmd -> Enter)
  4. Enter the following commands
    C:\> conda create -n tensorflowGPU python=3.5
  5. Enter 'y' if asked to continue.
  6. Activate conda virtual environment by entering the following command.
    C:\> activate tensorflowGPU
  7. Enter the appropriate command below to install particular version of tensorflow.

    For latest version of tensorflow use the following command.

    (tensorflowGPU) C:\> pip install --upgrade tensorflow-gpu

    For version 1.2 of tensorflow use the following command. You can find the appropriate address for each version on pages with instructions at this link <https://www.tensorflow.org/versions/>

    (tensorflowGPU) C:\> pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-1.2.1-cp35-cp35m-win_amd64.whl
  8. Enter the following commands to run tensorflow using python
    (tensorflowGPU) C:\> python
     >>> import tensorflow as tf
     >>> hello = tf.constant('Hello, TensorFlow!')
     >>> sess = tf.Session()
     >>> print(sess.run(hello))
     'Hello, TensorFlow!'
    
  9. The command should print 'Hello, TensorFlow!' if you have successfully installed tensorflow.
 

Build TensorFlow on Windows

Requirements:

  1. GIT for Windows: https://git-scm.com/download/win
  2. SWIG: https://sourceforge.net/projects/swig/files/swigwin/swigwin-3.0.10/
  3. Visual Studio 2015: https://www.visualstudio.com/downloads/, https://www.visualstudio.com/vs/older-downloads/
  4. CMake 3.4 or higher: https://cmake.org/download/
  5. Python 3.5 Anaconda x64: https://repo.continuum.io/archive/Anaconda3-4.2.0-Windows-x86_64.exe
  6. CUDA 8.0:  https://developer.nvidia.com/cuda-downloads
  7. cuDNN v5.1: https://developer.nvidia.com/cudnn

Instructions:

  1. Install Cmake and add it to system path.
  2. Install CUDA v8.0 (For GPU support only)
  3. Extract cuDNN (For GPU Support only)
    • copy all files from bin folder to 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin'
    • similarly from include to 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include' 
    • and 'lib\x64' to  ''C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64'.
  4. Install Anaconda in "C:\tools\", make sure you select Register Anaconda as system path for Python 3.5.
  5. Extract SWIG in "C:\tools\" folder.
  6. Open command window in the desired folder "D:\Downloads\tensorflow" (Shift+RightClick and Open Command Window here).
  7. Enter the following commands
    D:\Downloads>call "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" amd64
  8. Clone tensorflow directory and set up cmake configuration:
    D:\Downloads> git clone https://github.com/tensorflow/tensorflow.git
    D:\Downloads> cd tensorflow
    D:\Downloads\tensorflow> git checkout r1.2
    D:\Downloads\tensorflow> cd tensorflow\contrib\cmake
    D:\Downloads\tensorflow\tensorflow\contrib\cmake> mkdir build
    D:\Downloads\tensorflow\tensorflow\contrib\cmake> cd build
    D:\Downloads\tensorflow\tensorflow\contrib\cmake\build>  
    
  9. Invoke CMake and create Visual Studio solution by entering the following commands
    D:\Downloads\tensorflow\tensorflow\contrib\cmake\build> cmake .. -A x64 -DCMAKE_BUILD_TYPE=Release ^
     More? -DSWIG_EXECUTABLE=C:/tools/swigwin-3.0.10/swig.exe ^
     More? -DPYTHON_EXECUTABLE=C:/tools/Anaconda3/python.exe ^
     More? -DPYTHON_LIBRARIES=C:/tools/Anaconda3/libs/python35.lib ^
     More? -Dtensorflow_ENABLE_GPU=ON ^
     More? -DCUDNN_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0" 
    
  10. Build C++ tensorflow example
    D:\Downloads\tensorflow\tensorflow\contrib\cmake\build> MSBuild /p:Configuration=Release tf_tutorials_example_trainer.vcxproj
    D:\Downloads\tensorflow\tensorflow\contrib\cmake\build> Release\tf_tutorials_example_trainer.exe
    
  11. Build PIP package as .whl file
    D:\Downloads\tensorflow\tensorflow\contrib\cmake\build> MSBuild /p:Configuration=Release tf_python_build_pip_package.vcxproj
    
  12. PIP package is built in .\tf_python\dist folder. Enter the following commands to install tensorflow for python
    D:\Downloads\tensorflow\tensorflow\contrib\cmake\build> cd tf_python\dist
    D:\Downloads\tensorflow\tensorflow\contrib\cmake\build\tf_python\dist>
    D:\Downloads\tensorflow\tensorflow\contrib\cmake\build\tf_python\dist> pip install tensorflow_gpu-1.2.1-cp35-cp35m-win_amd64
    
  13. Enter the following commands to run tensorflow using python
    D:\Downloads\tensorflow\tensorflow\contrib\cmake\build\tf_python\dist> python
     >>> import tensorflow as tf
     >>> hello = tf.constant('Hello, TensorFlow!')
     >>> sess = tf.Session()
     >>> print(sess.run(hello))
     'Hello, TensorFlow!'
    
  14. The command should print 'Hello, TensorFlow!' if you have successfully installed tensorflow.

 

Build caffe, pycaffe and matcaffe on Windows

Requirements:

Download and install the following software's:
  1. Git for Windows: https://git-scm.com/download/win
  2. Visual Studio 2015: https://www.visualstudio.com/downloads/. To download older versions of visual studio you can go to https://www.visualstudio.com/vs/older-downloads/ 
  3. Cmake 3.4 or higher https://cmake.org/download/
  4. Python 3.5 Anaconda x64 (Required to build pycaffe). https://repo.continuum.io/archive/Anaconda3-4.2.0-Windows-x86_64.exe
  5. CUDA 8.0 https://developer.nvidia.com/accelerated-computing-toolkit
  6. cuDNN v5.1 (optional) https://developer.nvidia.com/cudnn
  7. Matlab (Required to build matcaffe)  http://mathworks.com

Instructions:

  1. Install Cmake and add it to system path.
  2. Install CUDA v8.0 (For GPU support only)
  3. Extract cuDNN (For GPU Support only)
    • copy all files from bin folder to 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin'
    • similarly from include to 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include' 
    • and 'lib/x64' to  ''C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib/x64'.
  4. Optional - Install Anaconda, make sure you select Register Anaconda as system path for Python 3.5 (required to build pycaffe).
  5. Optional - Install and activate Matlab using your Matlab licence (required to build matcaffe).
  6. Open command window in the desired folder "D:\Downloads\caffe" (Shift+RightClick and Open Command Window here).
  7. Enter the following commands
    D:\Downloads\caffe> git clone https://github.com/BVLC/caffe.git
    D:\Downloads\caffe> cd caffe
    D:\Downloads\caffe\caffe> git checkout windows
  8. Open scripts folder and edit build_win.cmd
    • set WITH_NINJA=0
    • set PYTHON_VERSION=3
    • set CONDA_ROOT to Anaconda folder, "C:\Program Files\Anaconda3".
    • set BUILD_MATLAB=1
    • Add -CUDNN_ROOT="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0" in cmake configuration in line 170.
  9. Save build_win.cmd and enter following command in command window
    D:\Downloads\caffe\caffe> scripts\build_win.cmd
  10. After successful build enter the following commands to run caffe
    D:\Downloads\caffe\caffe> cd build\tools\Release
    D:\Downloads\caffe\caffe\build\tools\Release> caffe -help
  11. Pycaffe instructions:

  12. Enter the following commands
    D:\Downloads\caffe\caffe\build\tools\Release> cd ..\..\..\
    D:\Downloads\caffe\caffe> conda config --add channels conda-forge
    D:\Downloads\caffe\caffe> conda install ninja --yes
  13. Copy the contents of D:\Downloads\caffe\caffe\python to site-packages folder [C:\Program Files\Anaconda3\]Lib\site-packages.
  14. Now run the following commands:
    D:\Downloads\caffe\caffe> python
    >>> import caffe
  15. You should be able to successfully import and run pycaffe.
  16. Matcaffe instructions:

  17. Enter the following command to download caffe bvlc model
    D:\Downloads\caffe\caffe> python scripts\download_model_binary.py models\bvlc_reference_caffenet
  18. Copy caffe_.mexw64 from D:\Downloads\caffe\caffe\matlab\+caffe\private\Release\ to D:\Downloads\caffe\caffe\matlab\+caffe\private.
  19. Add D:\Downloads\caffe\caffe\build\libraries to Matlab Path using setenv() command.
  20. Now you should be able to run classification_demo.m in C:\Projects\caffe\matlab\demo folder.

How to compile caffe on Windows 10

*UPDATE*: Updated instructions at this link < http://www.shaneahmed.com/2017/07/build-caffe-pycaffe-and-matcaffe-on.html >

Requirements:

Download and install the following software's:
  1. Visual Studio 2015 https://www.visualstudio.com/downloads/
  2. Make 3.4 or higher https://cmake.org/download/
  3. Python 3.5 Anaconda x64 (Required to build pycaffe). https://repo.continuum.io/archive/Anaconda3-4.2.0-Windows-x86_64.exe
  4. CUDA 8.0 https://developer.nvidia.com/accelerated-computing-toolkit
  5. cuDNN v5.1 (optional) https://developer.nvidia.com/cudnn
  6. Matlab (Required to build matcaffe)  http://mathworks.com

    Instructions:

    1. Install CUDA v8.0
    2. Extract cuDNN folder
      • copy all files from bin folder to 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin'
      • similarly from include to 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include' 
      • and 'lib/x64' to  ''C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib/x64'.
    3. Install Anaconda, make sure you select Register Anaconda as system path for Python 3.5
    4. Install and activate Matlab using your Matlab licence (only if you need to build matcaffe).
    5. Open command window in the desired folder "C:\Projects" (Shift+RightClick and Open Command Window here).
    6. Enter the following commands
    7. C:\Projects> git clone https://github.com/BVLC/caffe.git
      C:\Projects> cd caffe
      C:\Projects\caffe> git checkout windows
    8. Open scripts folder and edit build_win.cmd
      • set path for python and add path to Anaconda folder.
    9. set BUILD_MATLAB=1
    10. set -CUDNN_ROOT path to "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0" in cmake configuration in line 146.
    11. Edit CMakeLists.txt and make changes according to instructions in https://github.com/BVLC/caffe/pull/4928/files
    12. Enter following command
    13. C:\Projects\caffe> scripts\build_win.cmd
    14. You can find caffe.exe in C:\Projects\caffe\build\tools\release\caffe.exe
    15. Enter
    16. C:\Projects\caffe> call build\libraries\prependpath.bat
    17. Now you should be able to run 
    18. C:\Projects\caffe> build\tools\release\caffe.exe -help

      Pycaffe instructions:

    19. Enter the following commands
    20. conda config --add channels conda-forge
      conda install ninja --yes
    21. Copy the contents of C:\Projects\caffe\python to [C:\Program Files\Anaconda3\]Lib\site-packages.
    22. Now run the following commands:
    23. C:\Projects\caffe> python
      >>> import caffe
    24. You should be able to successfully import and run pycaffe.
    25. Matcaffe instructions:

    26. Copy caffe_.mexw64 from C:\Projects\caffe\matlab\+caffe\private\Release\ to C:\Projects\caffe\matlab\+caffe\private
    27. Add C:\Projects\caffe\build\libraries to Matlab Path using setenv() command.
    28. Now you should be able to run demo.m in C:\Projects\caffe\matlab\demo folder.

    How to run R code in Python

    R is freely available language and computational tool which is very popular among the statisticians and bioinformatics community. In this p...