flexmem allows users to transparently create out-of-core data structures that may be much larger than available RAM

Description

flexmem is a general, transparent tool for out-of-core (OOC) computing. It is launched as a command line utility, taking an application as an argument. All memory allocations larger than a specified threshold are memory-mapped to a binary file. When data are not needed, they are stored on disk. It is both process- and thread-safe.

flexmem will not make programs run faster. Allocations larger then the specified threshold are allocated onto the disk and retrieval from a disk is slower than for RAM. However, flexmem will allow you to run programs that would not run on a machine due to RAM constraints.

Requirements

  1. Linux
  2. Some version of openmp is installed on the machine.
  3. A C compiler

Installing flexmem

The flexmem package can be installed by navigating to the flexmem/standalone/src directory using the following commands:

mike@mike-VirtualBox:~/Projects/flexmem/standalone/src
> make all
mike@mike-VirtualBox:~/Projects/flexmem/standalone/src
> sudo make install

Note that make install will put the executable into /usr/local/bin and the required shared object file into /usr/local/lib.

Installing the R flexmem package

After installing xmem the R xmem package can be installed by navigating to the flexmem/Rpkg directory and using the following commands in a shell:

R CMD INSTALL flexmem

The package documentation provides more information about how to use the R flexmem package.

Using xmem

Start an application in xmem by specifying the application as an argument

mike@mike-VirtualBox:~/$ xmem R

The memory-mapped files are stored in /tmp by default. This parameter along with others can be interrogated using the Rxmem package, included in this project.

Benchmarks

In general flexmem is faster than swap when large chunks of contiguous memory is allocated. When a program makes many, small allocations, then swap is faster. You will get the best performance with flexmem when swap is left on.

A performance comparison of flexmem vs. page swapping of contiguous blocks of memory can be found here.

A more applied set of benchmarks comparing flexmem vs. swap to analyze genetic data, can be found here

Support

  1. flexmem is supported on Linux.
  2. Contributions are welcome.
  3. For more information contact Michael Kane at kaneplusplus@gmail.com or Bryan Lewis bwlewis@illposed.net.