My long-term professional interest centres around large-scale distributed storage and processing.

The chart below shows the main open-source projects I have worked on professionally over my career. Each dot is a week where I contributed in some way to the project code - typically a commit of some code I wrote, or (for the Hadoop projects) reviewing and committing someone else’s code. I have contributed to many more projects than the ones shown here, but these are ones to which I made sustained contributions, usually over multiple years.

Broadly speaking there are three main phases.

Hadoop

I was one of the first committers on Apache Hadoop, and from 2006 to 2015 I worked on it and many other distributed systems projects in the Hadoop ecosystem (mainly Avro, Parquet, Crunch, Kite, and Whirr), for the best part of a decade while working at Cloudera. I also wrote four editions of the bestselling book Hadoop: The Definitive Guide, published by O’Reilly, from 2008 to 2012.

Hadoop and Genomics

In 2015 I joined the data science team at Cloudera and shifted my focus to applications running on Hadoop and Spark. I was the primary author of Spark support in GATK, working with members of the Broad Institute GATK team from 2015 to 2019, and contributed to related projects (Hadoop-BAM, htsjdk). I also created Disq for reading and writing bioinformatics sequencing formats from Spark. I left Cloudera in 2018, but continued working on these projects through to the end of 2019.

During this time I also ported the single cell preprocessing pipeline in Scanpy so it could run in parallel using Dask, and on GPUs using RAPIDS, and made contributions to the dimensionality reduction software UMAP and the related project pynndescent (for calculating approximate nearest neighbours).

Python and Genomics

In June 2020, I started working for Related Sciences on sgkit, a statistical genetics toolkit based on PyData technologies like Dask, Xarray, and Zarr. This work culminated in a new Zarr-based file format for variant data (VCF Zarr), new tools for working with this data (bio2zarr, vcztools, and vczstore), and a published paper: ‘Analysis-ready VCF at Biobank scale using Zarr’.

In tandem with this I developed Cubed, a project I started in 2022 as a drop-in replacement for Dask, to make running large-scale array processing workloads more reliable. It has been used for workloads in sgkit (genomics) and Pangeo (geoscience).