The basic features of R are incredibly useful for someone coming from a non-R background, but experienced R users have sought faster and more effective data manipulation and improved plotting abilities, as well as multimedia options.

Hadley Wickham’s contribution to the R universe includes his ‘ tidyverse‘ concept, and the associated set of packages.  These packages contain R functions designed using a common data philosophy and graphical grammar.  The tidyverse is a framework of data concepts like ‘tidy data’ and ‘tibbles’ (extensions of R’s native data.frame).  The standard for ‘tidy data’ as defined by Hadley is this basic rectangular form that forms the basis for R:

“Tidy data is data where:

1. Each variable is in a column.
2. Each observation is a row.
3. Each value is a cell.”

These can all be installed collectively from within R Studio with install.package("tidyverse")
or you can install each one separately.  Installing the tidyverse means you can use the R command:
library("tidyverse")
to give you the advantage of making all the sub-components of that library available

Hadley explained these concepts in his books on R available online, including R for Data Science, R Packages and Advanced R.

It is useful to know the history and naming conventions of the tidyverse packages.  They use function names separated by an underscore instead of the dot that Base R uses.  For this reason, tidyverse has data_frame instead of data.frame, read_csv instead of read.csv,write_csv instead of write.csv etc.  reshape2 modified Base R’s reshape.   ‘dplyr’ was the 2014 evolution of an earlier package called plyr, also by Hadley.

1. dplyr

dplyr provides another abstraction layer for data, introducing new key verbs such as mutate, filter, summarise and the key adverb “by group”.  See Hadley’s own take on this.

R’s native group by and aggregate functions were common functions that took several steps to implement  – this package provided brevity in that context.  This is at the cost of more than one line entries.   Some discussion about pros and cons of dplyr v data.table.

2. tibble is the name of a package, but it is also the name of a data object/concept that is central to the tidyverse.   The use in script is ‘tb_df’ (tibble dataframe).  There is online PDF documentation.

Tibbles are an alternative to the data.table.  Tibbles extend R’s native data frames, not the data.tables (authored by a separate author).  Tibbles, in fact, form the central data type of the ‘tidyverse’.   data.frames, lists, vectors can be converted to tibbles with as_tibble(objectname).

Note that even though a tibble can be the output from a ‘tidyverse’ command ‘data_frame‘ this is NOT the basic R data.frame.

The concept of a modified data frame in the form of a tibble has been explored in several places, including Hadley’s R book, at the CRAN repository and at the tibble module in the tidyverse website.

3. tidyr

For data tidying.  It includes functions like ‘gather’, which collapses multiple columns into a key-value pair according to specification.  The complement is spread, in which a key-value pair is spread across multiple columns.

list
gather (unpivot) <->spread (pivot)
nest<->unnest

On-line documentation.

For an independent view, see Garrett Gman.

4. ggplot2

A ‘graphics of grammar‘ inspired extension to R’s native plot function.

One alternative to R’s native data frames is the data.table package first released to CRAN in 2006.  data table extends base R’s data.frame.  data.table was authored principally by Arun Srinivasan and Matt Dowle.

One of the central features of the data.table was that it replaced the focus on data.frame rownames with the concept of a ‘key‘, which could be a multi-column set of values, but only one of the ‘key’ columns would operate as the index column.  Further details here.

In the R ecosystem, data.table represents a different branch of the extensions to R’s basic data.frame object to that of the ‘tidyverse’ with its dplyr, readr and tibbles.  The pros and cons have been discussed by their respective authors.  Additionally, tidyverse’s author suggests data.table is faster for importing in some cases.  See also r-bloggers.

Due to the influence of R on python, some of the terminology is similar and could become confusing, but this is avoided by understanding that many of the concepts originated with R and have since been incrementally picked up in the python universe or elsewhere.

Here are some examples

A ‘data frame’ was introduced to python through the ‘pandas’ library.

Generally, the python libraries follow the lead of R. The data structures in R are generally known as ‘Data Frames’, with a two-dimensional structure.

Wes McKinney, a statistician, built a python library he called ‘pandas‘ that uses Series data structures (one dimensional) and a Data Frame data structure (two-dimensional) for time series processing (originally in a financial context).

Pandas is built on top of the python library numPy, so those wishing to improve performance, such as in the HPC context, may decide to proceed to numPy directly.

McKinney has written a book “Python for Data Analysis” (2nd Ed, 2017).

ggplot (python)

ggplot2, for example, is a package that exists within the R universe and uses the command ‘ggplot()’ in that context, but its later implementation within the python universe (as a library) is called ‘ggplot‘. ggplot (python) is unashamedly described as ‘a plotting system for Python based on R’s ggplot2 and the Grammar of Graphics‘ . To achieve its goals, it also requires the python implementation of a data frame, using the pandas library.

datatable (python)

A python library that has been based on Matt Dowle’s R library called ‘data.table’.  The library project started in 2017 at H20ai, not surprising since Matt Dowle also works there as a “Hacker” (since 2015).  H20 is a Silicon Valley machine-learning company in Mountain View, California.

Yihui Xie is a creative software engineer who works at RStudio.  His R packages are innovative and useful.

1. shiny. 

Shiny enables you to build interactive web applications straight from R.

2. knitR and animation.

Yihui Xie created a literate programming package for R, called knitR.  knitR enables R code to be embedded in other documents of different file types.  Both knitR and the animation package are the subject of a 2013 journal article by Xie.

The book ‘Dynamic Documents with R and knitr‘ is authored by Xie.

Another author used knitR and some other packages to create a package for an ‘iPython’-style R notebook using markdown to produce the HTML.

3.  htmlwidgets.

This package enables the use of javascript libraries in an HTML document generated R/Markdown or from Shiny.  Before this, it was necessary to work with R objects, and embed in script.  But now the html package includes everything with one line of R code.

4. Leaflets and DataTables.

There is a discussion video at RStudio in which Xie illustrates the use of htmlwidgets in two further packages, Leaflet and DataTables.

5. Xaringan

The xaringan package (with its own self-explanatory xaringan “Presentation Ninja” slideshow) is another of Yihui Xiu’s excellent output.  It is used to create browser-based slides, and relies on the remark.js package, which Xiu recommends you become familiar with before proceeding to use xaringan.   Here’s a link to a recent slideshow by Dr Alison Hill explaining her use of xaringan.

Xaringan has leveraged the work of the remark.js library, which converts markdown into HTML files ready for displaying in a browser.

6. Slidex

Daniel Anderson has recently written a neat package called slidex that will convert pptx (powerpoint) slides into HTML-ready powerpoint slides, by extracting the content and then using R’s xaringan/remark.js to complete the build of the slideshow ready for a browser.

The Grammar of Graphics (the ‘gg’ in the ‘ggplot2’)

Hadley Wickham is the author of several popular R packages and an employee of R Studio.  In developing his graphical plotting package for R, called ggplot2, he has applied many of the ideas in Leland Wilkinson’s influential Grammar of Graphics concept and book.  This 2010 article “A Layered Grammar of Graphics” explains ggplot2 and how it uses the Grammar of Graphics idea for R.  Hadley has also written his own ggplot2 book.

ggplot2 is one of several packages found in the set of R extensions known as the tidy verse.

The data life cycle and software

One of the requirements to work professionally with data is not only mathematical knowledge, but familiarity with data processing hardware and software.  Software packages can be useful through many stages of the life cycle, from data acquisition and cleaning, through to formatting and eventually, data visualisation.

Data-focussed scientists are often encouraged to be comfortable working with either R or python, though many will start with R for their statistical needs. Python is a scripted programming language that has built up many statistical and visualisation libraries that offer some alternative ways of working with data to working with R exclusively. Some cautions about using python have been raised.

See my separate post on how python is moving toward R.

Data science training and assistance

UWA’s HackyHour sessions are all about the programming needs of scientists.

A community of experienced scientists and computer scientists are now involved in training other scientists to work with data effectively, and to develop basic programming skills quickly, and effectively. Locally, training efforts are being made in HackyHours, and at the Pawsey Centre in Perth, where there are resources for High Performance Computing (HPC) available. There have also been Research Bazaars that combine Software Carpentry and other digital literacy sessions. The HackyHour sessions are in the spirit of the Software Carpentry training that was developed by Greg Wilson, whilst working at Mozilla.

Books

There are many introductory texts available for R and python.

e.g. An introduction to Data Cleaning with R

Computation and Performance

1. In general

One of the considerations when data files reach a certain size will be the time it takes to process the information, and the performance of the computing hardware and software that is being used.   I’ll have more details, for a range of use cases, on this website in the future.

Some of the considerations are data structures, choice of software data structures, and the opportunity for data-orientated processing (e.g. parallel computation).  In the context of R or python, some scientists explicitly compare the performance of some packages or libraries against others, for similar data processing activities, on their local hardware.

2.  High Performance Computing (HPC)

The availability of super computing facilities increases the need for scientists to become familiar with all aspects of the data processing life cycle, from the way in which they store their data, to the software manipulation of the data, and the parallel computing environment in which data can be processed in several cores simultaneously.

Locally, the Pawsey centre is trying to improve the deficit of knowledge in all these areas, as well as the workflow from scientist through to the performance of the supercomputing centre.  At the programming level, it is encouraging the use of python libraries like numpy, scipy, with group training conducted using a Jupyter Notebooks hub.

A further consideration for HPC will be universal data formats like HDF5, and in time, the provision of docker-based environments for providing the software to be used with the HPC hardware.

This is a working list of software that is useful for data science and visualisation projects.  I’ll have a separate page for useful data libraries.

R

Some useful R packages from the Institute of Bioinformatics, Johannes Kepler Uni, Linz are available at http://www.bioinf.jku.at/software/.

RStudio

Jupyter Notebooks (a dynamic/interactive programming environment for python, R and other languages).   It works by providing that environment in notebooks, which are html pages empowered to handle scripts and an input-output flow for data.  This is a link to what it is.  It can also be used in a shared way, by setting up a JupyterHub to deliver the same notebook document to many users.   There is a gallery of interesting jupyter notebooks, including one that is an introduction to python.

If you wish to work with Jupyter Notebooks and python in an integrated environment, one option is to use the Anaconda DataScience Distribution, which features a recent version of python, and most of the useful data science libraries for analytics (numpy, scipy and pandas), visualisation (matplotlib) and machine learning (theano, tensor flow).   There are some additional setup suggestions at Yale.

Static web pages in the form of an HTML file can be created from python (including the ipython notebooks) using the Sphinx program, originally written for the python documentation.

BORIS (Behavioural Observation Research Interactive Software).

I discovered BORIS talking to Erika at HackyHour this June 2018. Ethology and ethograms are very practical scientific concepts for a range of topics: something I’ve unknowingly been doing as part of my cricket scoring apps!

R is a software environment for statistical computing and graphics (https://www.r-project.org/). It is well established and scripting languages like python easily interface with its data structures (in fact, python library Pandas reproduces many of them), but R has its own universe of ideas and extensions as well.

One related application is RStudio (www.rstudio.com), which is intended to provide a graphical interface to R. RStudio itself is extended with packages like Shiny (https://shiny.rstudio.com/), to make it easier to build interactive web apps straight from RStudio.

R has its own scripting language, which is not python (clearly), but python can work with R data structures nonetheless.

The data structure within R (the ‘Data Frame’) is a rectangular (row, column) structure that forms the basis for many data applications.