Jacob Tomlinson

Guiding your contributor's agents to better behaviours

Thu, 19 Feb 2026 00:00:00 +0000

Many open source maintainers have noticed an uptick in low-effort AI generated PRs recently, myself included. The most frustrating of these is when someone prompts their agent to "Fix <url to issue> and make a PR with the changes". Reviewing these PRs can be time consuming because diffs can be large and the contributors rarely respond to review feedback, they just prompt and move on.

Backporting fixes to older releases on Conda Forge

Wed, 28 Jan 2026 00:00:00 +0000

Some quick notes on how to make a backport Conda Forge release.

Context

We recently had a security vulnerability in Dask which we patched in 2026.1.1 (see GHSA-c336-7962-wfj2 for more information on what happened).

Python package managers: uv vs pixi?

Tue, 18 Nov 2025 00:00:00 +0000

When I talk to people about Python package management in 2025 I see the following tools in active use; uv, pixi, pip, conda, mamba, micromamba and poetry. There may be others, but I don’t hear much about them.

Revisiting old contributions for Hacktoberfest

Wed, 01 Oct 2025 00:00:00 +0000

It’s hacktoberfest time again!

For the last few years hacktoberferst has been opt-in for project maintainers to avoid the bombardment of spam PRs across GitHub. To find participating projects you can view the hacktoberfest tag on github.

The Majority Of Your Users

Tue, 30 Sep 2025 00:00:00 +0000

The majority of your users don’t read your changelog.

The majority of your users only upgrade to new versions when forced to.

EffVer - Version your code by the effort required to upgrade

Thu, 21 Aug 2025 00:00:00 +0000

Version numbers are hard to get right. Maintainers want to communicate to users what the impact of adopting a new version will be, but poor communication can lead to a lot of frustration. There are a few popular version schemes in use today including Semantic Versioning (SemVer) and Calendar Versioning (CalVer). However, projects in the Python community often don’t strictly conform to these standards which leads to confusion.

Teaching Python with GPUs: Empowering educators to share knowledge that uses GPUs

Fri, 11 Jul 2025 00:00:00 +0000

In today’s world of ever-growing data and AI, learning about GPUs has become an essential part of software carpentry, professional development and the education curriculum. However, teaching with GPUs can be challenging, from resource accessibility to managing dependencies and varying knowledge levels.

EffVer - Version your code by the effort required to upgrade

Wed, 09 Jul 2025 00:00:00 +0000

GPU Accelerated Python

Fri, 06 Jun 2025 00:00:00 +0000

Accelerating Python using the GPU is much easier than you might think. We will explore the powerful CUDA-enabled Python ecosystem in this tutorial through hands-on examples using some of the most popular accelerated scientific computing libraries.

Intro to Kubernetes: 10 Years On

Thu, 15 May 2025 00:00:00 +0000

In October 2015 I gave a talk on Kubernetes at Tech Exeter (back when it was called the Exeter Web Meetup).

Generating useful titles for automated PRs in GitHub Actions

Tue, 29 Apr 2025 00:00:00 +0000

In kr8s I have a GitHub Actions workflow which runs a script nightly on a cron job. The workflow grabs a list of actively supported Kubernetes versions from endoflife.date and then cross-references them with the available kind container images for running the tests in CI.

Why don't my markdown titles work sometimes?

Fri, 14 Mar 2025 00:00:00 +0000

I write a lot of markdown. I use it on GitHub when creating issues/PRs, I use it in Obsidian when I take notes, I use it in Hugo when writing blog posts (like this one), I use it in Jupyter Notebooks when working with data and I use it in Sphinx with MyST when writing documentation.

100 Days of Coreutils

Wed, 05 Mar 2025 00:00:00 +0000

I consider myself an advanced Linux and macOS user. I’m currently a software engineer developing primarily for Linux systems, and I’ve previously worked as a Linux and Mac System Administrator. Over the years I’ve spent tons of time on the command line, however I bet there are a bunch of GNU Core Utilities (coreutils) commands I’ve never used before.

Using multiple config files with kubectl and other Kubernetes tools

Thu, 27 Feb 2025 00:00:00 +0000

If you want to point tools like kubectl to a config file other than ~/.kube/config you can set the environment variable KUBECONFIG. But did you know that KUBECONFIG behaves sort of like a path, and kubectl will load all the config files it finds?

An introduction to GPU programming in Python

Thu, 13 Feb 2025 00:00:00 +0000

Who Builds the PyData Ecosystem?

Mon, 27 Jan 2025 00:00:00 +0000

The PyData software ecosystem is made up of many open-source software libraries that are used heavily in Python Software Development, Data Science/Engineering, Traditional Sciences, Artificial Intelligence and beyond. They were used to fly a helicopter on Mars, drive new discoveries around climate change and generate the first image of a black hole.

Most stale bots are anti-user and anti-contributor, but they don't have to be

Thu, 05 Dec 2024 00:00:00 +0000

If you’ve been around open source projects on GitHub you may have encountered a project with a stale bot.

GPU Development in Python 101

Wed, 04 Dec 2024 00:00:00 +0000

Writing GPU code in Python is easier today than ever. You don’t need to learn C++ and there are many libraries available to get you started quickly. In this tutorial we will learn some GPU programming fundamentals and explore the ecosystem of GPU accelerated libraries that do the hard work for you.

The art of wrangling your GPU Python environments

Wed, 04 Dec 2024 00:00:00 +0000

Debugging software itself is a hard task, but debugging GPU software environments can be even more challenging. Understanding the intricate interactions between hardware, drivers, CUDA, C++ dependencies, and Python libraries can be far more complex.

Getting science done with accelerated Python computing platforms

Thu, 14 Nov 2024 00:00:00 +0000

Dask on HPC in 2024

Tue, 12 Nov 2024 00:00:00 +0000

GPU Acceleration in the PyData community

Tue, 12 Nov 2024 00:00:00 +0000

EffVer

Fri, 08 Nov 2024 00:00:00 +0000

GPU Accelerated Python

Wed, 06 Nov 2024 00:00:00 +0000

Accelerating Python on HPC with Dask

Thu, 29 Aug 2024 00:00:00 +0000

Dask is a popular Python framework for scaling your workloads, whether you want to leverage all of the cores on your laptop and stream large datasets through memory, or scale your workload out to thousands of cores on large compute clusters. Dask allows you to distribute code using familiar APIs such as pandas, NumPy and scikit-learn or write your own distributed code with powerful parallel task-based programming primitives.

EffVer

Wed, 28 Aug 2024 00:00:00 +0000

Zero Code Change GPU Acceleration

Wed, 28 Aug 2024 00:00:00 +0000

GPU Development in Python 101

Tue, 09 Jul 2024 00:00:00 +0000

Abstract

Since joining NVIDIA I’ve gotten to grips with the fundamentals of writing accelerated code in Python. I was amazed to discover that I didn’t need to learn C++ and I didn’t need new development tools. Writing GPU code in Python is easier today than ever, and in this tutorial, I will share what I’ve learned and how you can get started with accelerating your code.

GPU Development in Python 101

Fri, 14 Jun 2024 00:00:00 +0000

Abstract

Building a career in Open Source Software

Tue, 07 May 2024 00:00:00 +0000

Python version epochs are broken

Wed, 01 May 2024 00:00:00 +0000

In PEP440 Python introduced Version Epochs as a mechanism to allow projects to change versioning scheme. Unfortunately there’s no way I could see a project actually making use of this without confusing their users.

Creating GitHub Releases automatically on tags

Fri, 19 Apr 2024 00:00:00 +0000

GitHub Releases is a feature where you can create a page associated with a git tag that contains a description of the changes in that tag along with build artifacts for users to download.

Connect With Experts: RAPIDS Cloud Deployment

Thu, 21 Mar 2024 00:00:00 +0000

Reducing the Cost of your Data Science Workloads on the Cloud

Tue, 19 Mar 2024 00:00:00 +0000

By leveraging cloud computing resources, you can pay for just the computing power you need, when you need it. Additionally, GPU acceleration can significantly decrease the amount of time you need computing resources, reducing your overall cost.

A beginner's guide to managing Kubernetes resources in Python with kr8s

Mon, 11 Mar 2024 00:00:00 +0000

Managing Kubernetes resources with Python has never been easier thanks to the kr8s Kubernetes client for Python.

Dask on Databricks

Thu, 15 Feb 2024 00:00:00 +0000

Running Dask on Databricks

Thu, 15 Feb 2024 00:00:00 +0000

Databricks is a very popular data analytics platform used by data scientists, engineers, and businesses around the world. It was founded by the creators of Apache Spark, a powerful open-source data processing engine, and builds on top of Spark to provide a comprehensive analytics platform.

cudf.pandas: the Zero Code Change GPU Accelerator for pandas

Wed, 07 Feb 2024 00:00:00 +0000

Pandas is flexible, but often slow when processing gigabytes of data. Many frameworks promise higher performance, but they often support only a subset of the Pandas API, require significant code change, and struggle to interact with or accelerate third-party code that you can’t change. RAPIDS cuDF enables Pandas users to accelerate their existing workflows and third-party code with zero code change required. You can continue using Pandas on CPUs for small-scale local development and testing, and then enable “Pandas accelerator mode” to run on GPUs when you want better performance. Accelerator mode uses the GPU for supported operations and the CPU otherwise, dispatching under the hood as needed, and is compatible with most third-party libraries.

Running Dask workloads on multiple cluster backends with zero code changes using dask-ctl

Thu, 25 Jan 2024 00:00:00 +0000

Sometimes you want to write some code using Dask which can then be run against multiple different cluster backends. For example for local testing you might want to use LocalCLuster, but in production use KubeCluster. Or perhaps you want to easily switch between an on premise HPC with SLURMRunner or the cloud with Coiled.

EffVer: Version your code by the effort required to upgrade

Mon, 15 Jan 2024 00:00:00 +0000

Version numbers are hard to get right. Semantic Versioning (SemVer) communicates backward compatibility via version numbers which often lead to a false sense of security and broken promises. Calendar Versioning (CalVer) sits at the other extreme of communicating almost no useful information at all.

How to highlight lines in a Hugo code block

Thu, 11 Jan 2024 00:00:00 +0000

Sometimes when writing code in a blog post I want to emphasize a couple of lines in particular. Today I found out that Hugo has really nice syntax to do this in a regular markdown code-fence.

My Bio

Thu, 11 Jan 2024 00:00:00 +0000

From time to time I give talks and run workshops and folks will ask me for a bio. You may even be one of those folks and I’ve directed you to this page! Huzzah the system works!

How to get typer to show help by default

Wed, 10 Jan 2024 00:00:00 +0000

I love using typer for creating CLI tools in Python. It makes creating complex trees of subcommands really straightforward.

GitHub streaks and work/life balance

Tue, 02 Jan 2024 00:00:00 +0000

I recently read Loving and hating the Streak by Cassidy Williams. The post was all about committing code on GitHub every single day to maintain a streak.

Accelerating fuzzy document deduplication to improve LLM training with RAPIDS and Dask

Fri, 08 Dec 2023 00:00:00 +0000

Training Large Language Models (LLMs) requires a vast amount of input data, and the higher the quality of that data the better the model will be at producing useful natural language. NVIDIA NeMo Data Curator is a toolkit built with RAPIDS and Dask for extracting, cleaning, filtering and deduplicating training data for LLMs.

When to rebuild things that already exist

Sun, 24 Sep 2023 00:00:00 +0000

Abstract

This year I built a library that already exists. The existing solutions didn’t quite meet my needs, I wanted something that ticked all of my boxes. When thinking about building something new people referred me to xkcd #927. But I did it anyway.

Scaling the PyData Ecosystem with Dask

Sat, 23 Sep 2023 00:00:00 +0000

Dask is a flexible library for parallel computing in Python. Dask provides high-level interfaces to extend the PyData ecosystem to larger-than-memory or distributed environments, as well as lower-level interfaces to customise workflows. No previous experience is required, though knowledge of Python, NumPy and pandas is preferred.

Comparison of kr8s vs other Python libraries for Kubernetes

Mon, 04 Sep 2023 00:00:00 +0000

I’ve been working on kr8s for a while now and one of my core goals is to build a Python library for Kubernetes that is the most simple, readable and produces the most maintainable code. It should enable folks to write dumb code when working with Kubernetes.

The challenge of updating an aging blog

Wed, 23 Aug 2023 00:00:00 +0000

The Dask blog is a bit neglected these days. The website is an aging Jekyll blog and is well past it’s prime. Bringing it into current decade has been on my backlog for a while and today I decided to dedicate some time getting it up to date.

Deploying multi-GPU workloads on Kubernetes in Python

Thu, 17 Aug 2023 00:00:00 +0000

By using Dask to scale out RAPIDS workloads on Kubernetes you can accelerate your workloads across many GPUs on many machines. In this talk, we will discuss how to install and configure Dask on your Kubernetes cluster and use it to run accelerated GPU workloads on your cluster.

How I fixed my UniFi Devices intermittently showing as offline

Thu, 03 Aug 2023 00:00:00 +0000

Since upgrading to a UniFi Dream Machine (UDM) Pro I’ve had a problem with some of my UniFi devices showing as offline. TLDR It turned out that I accidentally had two controllers on my network and the devices were hopping back and forth between them.

Livestream notes: Replacing aiohttp with httpx in kr8s

Tue, 04 Jul 2023 00:00:00 +0000

This post will be updated with notes from the livestream throughout the day.

Today I will be streaming some open source code refactoring. Come and join in on Twitch!. Don’t forget to say hi in the chat 😊.

Introducing kr8s, a new Kubernetes client library for Python inspired by kubectl

Mon, 19 Jun 2023 00:00:00 +0000

For the last few months I’ve been tinkering with a new Kubernetes client library for Python called kr8s.

Avoid indirection in tests at all costs

Thu, 25 May 2023 00:00:00 +0000

When writing tests the balance between avoiding indirection and DRY-ness should be much more weighted towards avoiding indirection than in the code it is testing.

Mini demos

Wed, 17 May 2023 00:00:00 +0000

As software engineers we should all be able to communicate the things we have built to others, but giving a formal demo of something you’ve been working on can be daunting. Mini demos are a great way to build muscles around giving ad-hoc demos with little to no preparation.

Debugging Data Science workflows at scale

Fri, 12 May 2023 00:00:00 +0000

The more we scale up our workloads the more we run into bugs that only appear at scale. Reproducing these bugs can be expensive, time consuming and error prone. In order to report a bug on a GitHub repo you generally need to isolate the bug and come up with a minimal reproducer so that the maintainer can investigate. But what if a minimal reproducer requires hundreds of servers to isolate and replicate?

Dask on Kubernetes Overview

Thu, 16 Mar 2023 00:00:00 +0000

I was invited to join the Kubernetes Batch Working Group (k8s-batch-sig) to give an overview of how Dask runs on Kubernetes, especially for batch style workloads with DaskJob.

Running Jupyter in your Dask Kubernetes cluster

Fri, 10 Mar 2023 00:00:00 +0000

Did you know that the Dask scheduler has a --jupyter flag that will start a Jupyter server running within the Dask Dashboard?

Being intentional with container terminology

Thu, 09 Feb 2023 00:00:00 +0000

When writing and speaking about linux container technologies I’m trying to be more intentional with the words I use, which means often avoiding the word docker. My goal is to communicate clearly to both experts and novices alike.

Oversubscribing GPUs in Kubernetes

Tue, 07 Feb 2023 00:00:00 +0000

Sometimes I want to oversubscribe the GPUs in my Kubernetes cluster. This is especially useful when I’m developing but could also be useful in light workloads where you have ample GPU memory and don’t mind the occasional failure.

Deploying multi-GPU workloads on Kubernetes in Python

Thu, 02 Feb 2023 00:00:00 +0000

The RAPIDS suite of open source software libraries gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs with minimal code changes and no new tools to learn.

Quick and dirty way to pre-pull container images on Kubernetes

Thu, 02 Feb 2023 00:00:00 +0000

Sometimes when I give live demos with Kubernetes clusters I want to make sure that the container images I’m going to use are already pulled onto all of the nodes in my cluster. The last thing I want is for a Pod to be created to then sit in a Pending state while an image is pulled, especially given how large containers can be in the Data Science space.

Debugging Sphinx extensions in VSCode

Tue, 24 Jan 2023 00:00:00 +0000

This week I’ve been working on some custom Sphinx extensions for a documentation site.

Sphinx is a pretty complex tool with a broad ecosystem so documentation tends to be spread across the upstream project, dependencies like docutils and popular extensions like MyST. Therefore figuring out what is going on can be challenging, so I almost always resort to digging through state in a debugger and doing code spelunking on GitHub.

Sometimes I regret using CalVer

Mon, 16 Jan 2023 00:00:00 +0000

Over the last few years, many open-source Python projects that I work on have switched to CalVer. I’ve felt some pain around this, particularly in Dask and its subprojects. I want to unpack some of my thoughts and feelings around this trend.

Issue 6: Documentation and diagrams

Fri, 28 Oct 2022 16:00:00 +0000

1. JSON Patch

If you want to install the Dask Operator on a Kubeflow cluster you need to patch the kubeflow-kubernetes-edit cluster role to give users permission to create Dask custom resources.

Dask Kuberenetes Operator Demo

Thu, 27 Oct 2022 00:00:00 +0000

Issue 5: Collaborating in Germany

Fri, 14 Oct 2022 16:00:00 +0000

1. Offsites

This week I joined some colleagues from Coiled at their team offsite in Germany to talk about Dask Distributed maintenance.

Issue 4: Last minute talk rewrites

Fri, 30 Sep 2022 16:00:00 +0000

1. Dask overview for Stanford distributed computing group

This week I gave a presentation to folks at Stanford on Dask. I mistakenly thought that my target audience would be students who were studying distributed computing and so I pulled together some Dask tutorial material and prepared to give a 101 level introduction to distributed computing in Python.

Dask overview

Thu, 29 Sep 2022 00:00:00 +0000

Issue 3: How much CI is too much CI?

Fri, 23 Sep 2022 16:00:00 +0000

Last week I was on holiday, so I skipped a week of newsletter, and this one is a little light.

Narrative driven development

Mon, 12 Sep 2022 00:00:00 +0000

In July I published a blog post on using Dask on KubeFlow with the Dask Kubernetes Operator. I originally outlined that post in January before the Dask Operator even existed as part of my planning for that work.

Issue 2: Kubeflow, Dracula and more

Fri, 09 Sep 2022 16:00:00 +0000

1. Blog: Accelerating ETL on KubeFlow with RAPIDS

Issue 1: Five things each week

Fri, 02 Sep 2022 00:00:00 +0000

Thank you for reading my very first newsletter! Everyone and their dog seems to have a newsletter these days, so I thought why not me.

Accelerating ETL on KubeFlow with RAPIDS

Tue, 30 Aug 2022 00:00:00 +0000

In the machine learning and MLOps world, GPUs are widely used to speed up model training and inference, but what about the other stages of the workflow like ETL pipelines or hyperparameter optimization?

How to check your NVIDIA driver and CUDA version in Kubernetes

Fri, 12 Aug 2022 00:00:00 +0000

When using GPUs with Kubernetes it can be important to know which driver and CUDA versions are installed on the nodes.

Using Dask on KubeFlow with the Dask Kubernetes Operator

Wed, 27 Jul 2022 00:00:00 +0000

Kubeflow is a popular Machine Learning and MLOps platform built on Kubernetes for designing and running Machine Learning pipelines for training models and providing inference services. It has a notebook service that lets you launch interactive Jupyter servers (and more) on your Kubernetes cluster as well as a pipeline service with a DSL library written in Python for designing and building repeatable workflows. It also has tools for hyperparameter tuning and running model inference servers, everything you need to build a robust ML service.

Don't prematurely squash/rebase and force push your PRs

Tue, 26 Jul 2022 00:00:00 +0000

A big frustration for me when reviewing Pull Requests on GitHub is coming back to a PR you’ve already reviewed to check on recent changes and be greeted with “We went looking everywhere, but couldn’t find those commits”.

Introducing Container Canary

Thu, 14 Jul 2022 00:00:00 +0000

Commenting on Pull Requests with GitHub Actions

Tue, 12 Jul 2022 00:00:00 +0000

When someone opens a Pull Request (PR) on your GitHub project it can be helpful for a bot to comment on the PR. You might want to thank the user for the contribution, provide some useful information such as giving a binder link where folks can try out the PR, or providing more verbose output from some tests or other checks.

GPU development with Python 101

Tue, 12 Jul 2022 00:00:00 +0000

Dask Tutorial: Scaling the PyData Ecosystem

Mon, 11 Jul 2022 00:00:00 +0000

The secret to making code contributions that stand the test of time

Tue, 05 Jul 2022 00:00:00 +0000

When you contribute code to collaborative projects, whether they are open source community projects or large internal projects inside organisations, the feeling of having your code running inside a large application can be very rewarding.

Parallelizing Your ETL with Dask on KubeFlow

Wed, 29 Jun 2022 00:00:00 +0000

Kubeflow is a popular MLOps platform built on Kubernetes for designing and running Machine Learning pipelines for training models and providing inference services. Kubeflow has a notebook service that lets you launch interactive Jupyter servers (and more) on your Kubernetes cluster. Kubeflow also has a pipelines service with a DSL library written in Python for designing and building repeatable workflows that can be executed on your cluster, either ad-hoc or on a schedule. It also has tools for hyperparameter tuning and running model inference servers, everything you need to build a robust ML service.

GPU programming with Python

Tue, 28 Jun 2022 00:00:00 +0000

Writing GPU code in Python is easier today than ever!

I joined NVIDIA in 2019 and I was brand new to GPU development. In that time, I’ve gotten to grips with the fundamentals of writing accelerated code in Python. I was amazed to discover that I didn’t need to learn C++ and I didn’t need new development tools. Writing GPU code in Python is easier today than ever, and in this tutorial, I will share what I’ve learned and how you can get started with accelerating your code.

GPU Development with Python 101

Wed, 15 Jun 2022 00:00:00 +0000

Writing GPU code in Python is easier today than ever!

I joined NVIDIA in 2019 and I was brand new to GPU development. In that time, Ive gotten to grips with the fundamentals of writing accelerated code in Python. I was amazed to discover that I didnt need to learn C++ and I didnt need new development tools. Writing GPU code in Python is easier today than ever, and in this tutorial, I will share what Ive learned and how you can get started with accelerating your code.

Parallelizing Your ETL with Dask on KubeFlow

Wed, 08 Jun 2022 00:00:00 +0000

GPU accelerating your computation in Python

Wed, 25 May 2022 00:00:00 +0000

Talk abstract

There are many powerful libraries in the Python ecosystem for accelerating the computation of large arrays with GPUs. We have CuPy for GPU array computation, Dask for distributed computation, cuML for machine learning, Pytorch for deep learning and more. We will dig into how these libraries can be used together to accelerate geoscience workflows and how we are working with projects like Xarray to integrate these libraries with domain-specific tooling. Sgkit is already providing this for the field of genetics and we are excited to be working with community groups like Pangeo to bring this kind of tooling to the geosciences.

How to set environment variables on your Dask workers

Thu, 05 May 2022 00:00:00 +0000

When working with Dask clusters you often need the remote worker environment to match you local environment. This generally means having the same packages and data available.

Golang block until interrupt with ctrl+c

Mon, 21 Mar 2022 00:00:00 +0000

Today I found myself needing a Go application’s main thread to stop and wait until the user wants it to exit with a ctrl+c keyboard interrupt.

Goodbye Docker Desktop for Mac, Hello Colima

Mon, 31 Jan 2022 00:00:00 +0000

Today is the deadline for the license changes to Docker Desktop for Mac and Windows. This means that if you are employed at a company with more than 250 employees or your company makes more than $10m you need to start paying a subscription to continue using Docker Desktop.

Docker Desktop for Mac alternatives for developers

Fri, 28 Jan 2022 00:00:00 +0000

In a couple of days Docker will begin charging employees of companies with >250 employees to use Docker Desktop. I have no problem with paying for software that brings me value, but you wouldn’t believe how complex it can be for large companies to sign employees up to subscription services. Paperwork everywhere! To avoid this I’m evaluating alternatives for Docker Desktop to use on my MacBook.

Running Kubeflow inside Kind with GPU support

Thu, 27 Jan 2022 00:00:00 +0000

This week I’ve been playing around with Kubeflow as part of a larger effort to make it simpler to use Dask and RAPIDS in MLOps workflows.

Quick hack: Adding GPU support to kind

Tue, 25 Jan 2022 00:00:00 +0000

This post has been superseded with this tutorial that no longer requires any code changes. Please read that instead.

Don't be that open-source user, don't be me

Mon, 17 Jan 2022 00:00:00 +0000

Before I was a maintainer of open source software I was a user of open source software, and I sometimes behaved badly.

GPU code in Python

Thu, 04 Nov 2021 00:00:00 +0000

Writing GPU code in Python is easier today than ever!

GPU development with Python 101

Thu, 28 Oct 2021 00:00:00 +0000

Summary

Writing GPU code in Python is easier today than ever, and in this tutorial, I will share what I’ve learned and how you can get started with accelerating your code. You don’t need to learn C++ and you don’t need any special tooling. All you need is an NVIDIA GPU, Python and a few libraries (and we’ll provide that for the tutorial).

GPU development with Python 101 (extended)

Mon, 26 Jul 2021 00:00:00 +0000

Writing GPU code in Python is easier today than ever!

What is RAPIDS?

Wed, 14 Jul 2021 00:00:00 +0000

Branding your open source Python package

Fri, 09 Jul 2021 00:00:00 +0000

Having a brand can help give your open source project some legitimacy, and you don’t need to be a designer to see these benefits. However it is important to understand that you do not need to add branding to your project in order for it to be successful, and adding branding can even harm your project.

What is the difference between Dask and RAPIDS?

Tue, 22 Jun 2021 00:00:00 +0000

Both Dask and RAPIDS are Python libraries to scale your workflow and empower you to process more data and leverage more compute resources. Both use interfaces modeled after the PyData ecosystem, making them familiar to most data practitioners.

The evolution of a Dask Distributed user

Tue, 01 Jun 2021 00:00:00 +0000

This week was the 2021 Dask Summit and one of the workshops that we ran covered many deployment options for Dask Distributed.

Managing cluster lifecycle with dask-ctl

Thu, 20 May 2021 00:00:00 +0000

Dask on Kubernetes

Wed, 19 May 2021 00:00:00 +0000

At the Dask Summit I chaired a workshop called Deploying Dask. This workshop was comprised of multiple talks delivered by myself and others.

Deploying Dask Distributed

Wed, 19 May 2021 00:00:00 +0000

At the Dask Summit I chaired a workshop called Deploying Dask. This workshop was comprised of multiple talks delivered by myself and others.

Building a contributor community for your open source project

Fri, 30 Apr 2021 00:00:00 +0000

With our open source project published on GitHub we probably want to allow folks to contribute changes. Some users of the project may find bugs, or desire extra features and will open issues to tell you. Users who have the skills required to make that change can open a Pull Request on GitHub to propose it. As the maintainer you can then review and merge those changes.

Communicating with your open source community

Fri, 23 Apr 2021 00:00:00 +0000

Once your open source Python project has users and a community you will likely want to communicate with them in an official capacity. Perhaps you want to tell them about a new release, show a use case where someone is using your tool or solicit feedback on an upcoming feature.

Building a user community for your open source project

Fri, 16 Apr 2021 00:00:00 +0000

Now that our open source Python project exists and users can install it we will want to turn our attention to sustainability, reach and ongoing maintenance. By putting it out there and gaining users you are opening yourself up to questions, bug reports and feature requests.

Documenting Python projects with Sphinx and Read the Docs

Fri, 09 Apr 2021 00:00:00 +0000

In part four of this series we discussed documenting our code as we went along by adding docstrings throughout out project. In this post we will see that effort pay off by building a documentation site using Sphinx which will leverage all of our existing docstrings.

Monitoring Dask + RAPIDS with Prometheus + Grafana

Fri, 09 Apr 2021 00:00:00 +0000

Prometheus is a popular monitoring tool within the cloud community. It has out-of-the-box integration with popular platforms including Kubernetes, Open Stack, and the major cloud vendors, and integrates with dashboarding tools like Grafana.

Intro to distributed computing on GPUs with Dask in Python

Wed, 31 Mar 2021 00:00:00 +0000

RAPIDS is an end to end data science stack built entirely for CUDA GPUs. Faster analytics, at scale, for lower total cost of ownership. Dask natively scales Python and the RAPIDS ecosystem stack onto multiple servers and GPUs, supporting unprecedented scale. BlazingSQL is a distributed SQL engine built in Python. It performs incredibly fast SQL queries on the RAPIDS DataFrame and ensures optimal usage of GPU primitives.

Automating releases of Python packages with GitHub Actions

Fri, 26 Mar 2021 00:00:00 +0000

In this post we will cover automatically packaging and releasing our project when a new git tag is pushed to GitHub.

Testing and Continuous Integration for Python packages with GitHub Actions

Fri, 19 Mar 2021 00:00:00 +0000

In this post we will cover automatically running our tests when we push new code to GitHub, and when contributors raise Pull Requests against our project.

Awaitable Objects and Async Context Managers in Python

Wed, 17 Mar 2021 00:00:00 +0000

Python objects are synchronous by default. When working with asyncio if we create an object the __init__ is a regular function and we cannot do any async work in here.

Test driven development in Python

Fri, 12 Mar 2021 00:00:00 +0000

What is test driven development (TDD)?

Test driven development is a style of development where you write your tests before you write your code.

Testing your Python package

Fri, 22 Jan 2021 00:00:00 +0000

In this post we will cover testing our code.

Testing

There are many many great resources out there for learning about testing software. In this post I’m going to try and focus on simple examples that you can use to get started quickly. Once you have a good foundation for your tests you can then dive into mocking, replaying HTTP requests or even hypothesis testing.

High Throughput Computing with Dask: Intro Tutorial

Thu, 21 Jan 2021 00:00:00 +0000

High-throughput (task-based) computing is a flexible approach to parallelization. It involves splitting a problem into loosely-coupled tasks. A scheduler then orchestrates the parallel execution of those tasks, allowing programs to adaptively scale their resource usage. Individual tasks may themselves be parallelized using MPI or OpenMP, and the high-throughput approach can therefore enable new levels of scalability.

Documenting your Python code

Fri, 15 Jan 2021 00:00:00 +0000

This post will cover documenting our code. Specifically adding documentation within the code itself.

Docstrings

Right now our code is undocumented, so if the user inspects our function they will only see the interface (the way you call it) but with no other context. We can use IPython to quickly inspect this.

How to interactively debug GitHub Actions with netcat

Fri, 08 Jan 2021 00:00:00 +0000

Update: This was a fun experiment and I recommend you check out the post for a fun read on setting up reverse shells. But I’ve since discovered this awesome tmate action which lets you interactively debug in the browser or via SSH.

How to check out the default git branch

Wed, 09 Sep 2020 00:00:00 +0000

Many open source projects are taking steps to update terminology to be more inclusive. The largest of these changes has been renaming the “trunk” branch of git repositories from master to main.

Intro to GPU Development in Python

Wed, 09 Sep 2020 00:00:00 +0000

Writing code for GPUs has come a long way over the last few years and it is now easier than ever to get started. You can even do it in Python! This talk will cover setting up your Python environment for GPU development. How coding for GPUs differs from CPUs, and the kind of problems GPUs excel at solving. We will dive into some real examples using Numba and also touch on a suite of Python Data Science tools called RAPIDS.

Leveraging the Hacktoberfest community

Thu, 27 Aug 2020 00:00:00 +0000

Hacktoberfest is approaching once again. In previous years I have both participated and contributed to open source, and also tried to leverage the community in the open source projects I maintain by curating and labeling issues.

Running Dask tutorials

Fri, 21 Aug 2020 00:00:00 +0000

Originally published on the Dask blog on August 21st, 2020.

For the last couple of months we’ve been running community tutorials every three weeks or so. The response from the community has been great and we’ve had 50-100 people at each 90 minute session.

The current state of distributed Dask clusters

Thu, 23 Jul 2020 00:00:00 +0000

Originally published on the Dask blog on July 23rd, 2020.

Dask enables you to build up a graph of the computation you want to perform and then executes it in parallel for you. This is great for making best use of your computer’s hardware. It is also great when you want to expand beyond the limits of a single machine.

Dask Maintainers Update

Wed, 08 Jul 2020 00:00:00 +0000

How to use OBS Studio with Zoom, Hangouts, Teams and more on macOS

Fri, 19 Jun 2020 00:00:00 +0000

A popular tool with streamers and YouTubers is Open Broadcaster Software®️ Studio or OBS for short. It allows you to compose scenes with cameras, desktop sharing, video snippets, images, web pages and more and then stream that video to services like Twitch or Mixer. You can also save recordings locally if you want to upload them to YouTube.

Intro to Dask Tutorial

Fri, 19 Jun 2020 00:00:00 +0000

How to enable SSH on Binder

Wed, 17 Jun 2020 00:00:00 +0000

⚠️ This post is no longer valid.

Running SSH on Binder has not been possible since late 2020. Due to abuse from botnets Binder will now kill sessions running sshd.

Publishing open source Python packages on GitHub, PyPI and Conda Forge

Fri, 28 Feb 2020 00:00:00 +0000

In this post we will cover making our code available to people. This is the bit where we open the source! We will push our code to a code posting platform and then package up our library and submit it to a couple of repositories to make it easy for people to install.

Native Cloud Deployment with Dask-Cloudprovider

Thu, 27 Feb 2020 00:00:00 +0000

RAPIDS - Open GPU Data Science

Tue, 18 Feb 2020 00:00:00 +0000

The RAPIDS suite of open source software libraries (https://rapids.ai/) allow you to run data science and analytics pipelines entirely on GPUs, but following familiar Python APIs including Numpy, Pandas and SciKit Learn.

Versioning and formatting your Python code

Fri, 14 Feb 2020 00:00:00 +0000

In this post, we will cover a few project hygiene things that we may want to put into place to make our lives easier in the future.

Testing static sites with Lighthouse CI and GitHub Actions

Thu, 13 Feb 2020 00:00:00 +0000

When you build a website you want pages to load as quickly as possible for users. Google has a tool called PageSpeed Insights which you can run on your website to see various metrics about the page. I’ve used it in the past while working on my blog and other sites.

Creating an open source Python project from scratch

Fri, 07 Feb 2020 00:00:00 +0000

Have you had a great idea for an open-source Python library that you think people will find useful, but you don’t know where to begin in creating and publishing it?

Twenty Nineteen Roundup

Tue, 31 Dec 2019 00:00:00 +0000

Introduction

It has been a few years since I published a list of technology and media I enjoyed this year, so here we go for 2019.

5 Tips to help you ace your internship and entry-level job interviews

Mon, 16 Dec 2019 00:00:00 +0000

Applying for internships and entry-level positions can be tricky. Interviewers want to hear you talk about your experiences and things you’ve done that prove you’re a good fit for the job. However given that you are applying for an entry-level role you likely don’t have much real world experience in this space. It’s a chicken and egg situation that everyone faces when they are first starting out or wanting to make a shift to a new area.

Creating GitHub Actions in Python

Mon, 09 Dec 2019 00:00:00 +0000

Note: This post is also available in Go flavour.

GitHub Actions provide a way to automate your software development workflows on GitHub. This includes traditional CI/CD tasks on all three major operating systems such as running test suites, building applications and publishing packages. But it also includes automated greetings for new contributors, labelling pull requests based on the files changed, or even creating cron jobs to perform scheduled tasks.

Creating GitHub Actions in Go

Thu, 05 Dec 2019 00:00:00 +0000

Note: This post is also available in Python flavour.

How to run Jupyter Lab at startup on macOS

Mon, 02 Dec 2019 00:00:00 +0000

In my day to day work I generally access a variety of Jupyter installations. Sometimes these are short lived installations in conda environments on my laptop, sometimes they are running on a remote server, and sometimes I use a managed service like JupyterHub or Binder.

How to create a Helm chart repository with Chartpress, Travis CI, GitHub Pages and Jekyll

Mon, 28 Oct 2019 00:00:00 +0000

Helm has become a pervasive tool in the Kubernetes community for packaging, managing, upgrading and distributing applications. It uses a packaging format called charts which are a collection of templates that describe Kubernetes resources and can be configured by the user.

How to merge Kubernetes kubectl config files

Tue, 15 Oct 2019 00:00:00 +0100

Sometimes when working with a new Kubernetes cluster you will be given a config file to use when authenticating with the cluster. This file should be placed at ~/.kube/config. However you may already have an existing config file at that location and you need to merge them together.

The three types of fun

Tue, 10 Sep 2019 00:00:00 +0100

According to folks who enjoy outdoor activities there are three types of fun. I’ve been using this scale for a while to categorize my own enjoyment of things and wanted to share my version.

Why your profile picture is important

Tue, 27 Aug 2019 00:00:00 +0000

Choosing a good profile picture will make collaborating with others easier, especially if you haven’t met them yet. Here are some tips to help you pick a good one.

Cleaning up conda environments

Fri, 23 Aug 2019 00:00:00 +0100

Often when I’m developing or debugging in Python I end up creating throw away conda environments. They will be to test some package installation or combination of packages and once I’ve finished I will probably never use them again.

Setting up GPU Data Science Environments for Hackathons

Tue, 13 Aug 2019 00:00:00 +0000

Originally published on the RAPIDS AI blog on August 13th, 2019.

Background

In my first week working at NVIDIA, I have been spending some time with my previous colleagues at the Met Office to explore how the two organizations can collaborate.

A history of science on the cloud at the Met Office

Thu, 23 May 2019 00:00:00 +0000

The Informatics Lab is a technology, science and design research group at the Met Office. For the last four years we’ve been exploring, among other things, how we can do more science using the power of cloud computing. Using tools such as Jupyter, Dask and Kubernetes we’ve built a number of prototypes of cloud based scientific platforms starting with our JADE project through to current implementations of Pangeo. In this session we will cover how we evolved these systems, produced tools like dask-kubernetes and the Pangeo helm chart, helped build a community, and pushed our science and the Met Office forward.

Switching to Hugo

Tue, 21 May 2019 14:21:28 +0100

It has been nearly two years since I published a new blog post on this website. That doesn’t mean I haven’t been writing things. It’s just that much of my content has been posted on other platforms. I’ve decided recently to gather everything together and make this website the canonical source for the things I produce. This includes blog posts, talks, videos and more.

Hypothetical datasets

Wed, 08 May 2019 00:00:00 +0000

In Theo’s previous posts on storing high momentum data and its accompanying metadata we get some interesting insights into the future of cloud based data storage. In this post I’m going to cover how we are working with today’s NetCDF-based challenges, by making assumptions!

Met Office Informatics Lab Overview

Tue, 26 Mar 2019 00:00:00 +0000

Straight from our own Informatics Lab, Jacob will share how to perform large scale distributed data analysis on any cloud platform with Pangeo. Pangeo is a suite of open source tools created and curated by the geoscience community to make big scale scientific analysis possible on any cloud platform. It uses Kubernetes to bring consistency to the workflow and it brings popular scientific python libraries together to work interoperability.

Intro to Earth Information Workshop

Tue, 12 Mar 2019 00:00:00 +0000

This article was originally written for the the Met Office workshop run at the Intro to Earth Information event on the 12th of March 2019.

Cloud agnostic distributed data analysis with Pangeo

Tue, 12 Feb 2019 00:00:00 +0000

Containers Why and How to Use Them

Wed, 30 Jan 2019 00:00:00 +0000

Enable your development teams to be able to deploy software quickly and efficiently, as well as at great scale, with containerisation.

My pragmatic workshop format

Wed, 30 Jan 2019 00:00:00 +0000

Mozfest workshop facilitators meeting

Figuring out the right format for a workshop can be tricky. There are so many factors; what is the subject, do people need any equipment, how many people will attend, how many facilitators will there be, where will it be held, what level of expertise will the participants have, the list goes on…

Containers and orchestration 101 - Docker, Kubernetes and Helm

Fri, 07 Sep 2018 00:00:00 +0000

If you read reports from organizations like the CNCF they will tell you that 90% of people are developing in containers, 69% are running them in production and 77% of those are using Kubernetes to manage them. However those surveys are biased towards a self-selecting group of container converts.

Pangeo Principles

Mon, 13 Aug 2018 00:00:00 +0000

Debugging Kubernetes PVCs

Fri, 03 Aug 2018 00:00:00 +0000

Sometimes I find that something goes wrong in a container and some data stored in a persistent volume gets corrupted. This may result in me having to get my hands dirty and have a poke around in the filesystem myself.

Opsdroid

Thu, 02 Aug 2018 00:00:00 +0000

Server administration is an activity that often happens in an isolated context in a terminal. ChatOps is a way of bringing that work into a shared environment and unlocking more collaboration. This week Jacob Tomlinson talks about the work he has done on opsdroid, a new bot framework targeted at tying together the various services and environments that modern production systems rely on.

Met Office Informatics Lab Overview

Tue, 17 Jul 2018 00:00:00 +0000

Using Xiaomi door/window sensors as light switches

Wed, 06 Jun 2018 00:00:00 +0000

Introduction

For a while I’ve been searching for a decent light switch solution for my home automation setup. I’ve recently put in a pretty good solution using Xiaomi door/window sensors, I’m very happy with it and it ticks a lot of boxes.

Exploring Dask and Distributed on AWS Lambda

Fri, 13 Apr 2018 00:00:00 +0000

I spent some time this week exploring whether it would be possible to run Dask and Distributed on a function as a service platform like AWS Lambda.

Rise of the Bots

Wed, 21 Mar 2018 00:00:00 +0000

Helping Balloons Navigate the Weather: Alibaba Tianchi Award Presentation

Tue, 27 Feb 2018 00:00:00 +0000

In late 2017 the Met Office ran a joint Tianchi data science challenge with Alibaba. I was invited to present the winning team with their award at the Mobile World Congress in Barcelona.

Instant access to auto-scaling personal Python clusters

Wed, 07 Feb 2018 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on February 7th, 2018.

We are excited to announce that the work we’ve been doing with distributed Dask clusters running on Kubernetes has been absorbed into an awesome new tool called Daskernetes through our work on the Pangeo project.

Accessing S3 from FUSE

Mon, 08 Jan 2018 00:00:00 +0000

My data lives in an object store, but my tools expect a POSIX file path, what do I do? In the Informatics Lab we have been experimenting with using FUSE filesystems on a distributed compute cluster to provide parallel access to files on AWS S3. We have also created a library called pysssix which provides a slim and intentionally minimal access to S3 for fast data access.

ChatOps - Automation via chat

Tue, 19 Dec 2017 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on December 19th, 2017.

ChatOps - Automation via chat

This article is a companion to a workshop on using chat to automate ops workflows. This is a static version of a Jupyter Notebook which you can download here.

The Power of Big Data

Mon, 30 Oct 2017 00:00:00 +0000

This session gives an in-depth look at the current state of big data at AWS. Learn about the latest big data trends and industry use cases. We’ll focus on how other organizations are using the AWS big data platform to innovate and remain competitive. Met Office also joins us to offer an inside look on how they are using AWS to enable citizens, business, and governments to consume its data on demand.

Scaling Clusters to Thousands of Servers in the Cloud

Thu, 21 Sep 2017 00:00:00 +0000

In order to analyse the petabytes of data we have at the Met Office we need very large clusters of servers. However procuring these pieces of infrastructure takes months or even years of planning and large up-front capital expense.

Deploying opsdroid using ZEIT

Sat, 16 Sep 2017 00:00:00 +0000

ZEIT is a great platform for deploying your opsdroid instance. Particularly because it is free for light use, which many opsdroid deployments will be.

Autoscaling Distributed Compute With Dask Kubernetes and AWS

Mon, 21 Aug 2017 00:00:00 +0000

Article in Computer Weekly

Tue, 01 Aug 2017 00:00:00 +0000

This week Computer Weekly have published an article interviewing me about how the Met Office is tackling the vast amount of data we are producing. They reference the work the Informatics Lab have done on the Jade project and Met Office public datasets.

Interactive Big Data Analysis With Jupyter, Dask and more

Thu, 27 Jul 2017 00:00:00 +0000

Adaptive Dask clusters on Kubernetes and AWS

Fri, 21 Jul 2017 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on July 21st, 2017.

Introduction

This article assumes a basic understanding of Amazon Web Services (AWS), Kubernetes, Docker and Dask. If you are unfamiliar with any of these you should do some preliminary research before continuing.

Making Environmental Science and Data Beautiful and Useful

Tue, 18 Jul 2017 00:00:00 +0000

Asyncio in Depth

Wed, 28 Jun 2017 00:00:00 +0000

Generate git release notes automatically

Thu, 15 Jun 2017 00:00:00 +0000

It is common practice for release notes to consist of a list of the Pull Requests which have been merged since the last release. Some projects divide these into categories, for example breaking changes, enhancements and bug fixes. If you are a project maintainer you may want to be able to generate this automatically.

How to create a seal only token for Hashicorp Vault

Wed, 14 Jun 2017 00:00:00 +0000

Introduction

When using Hashicorp’s Vault you may want to have an authentication token which only has permissions to seal the vault. This can then be used in an emergency situation to seal the vault, perhaps through a chatbot.

RITA 2017 Innovation Award

Thu, 25 May 2017 00:00:00 +0000

My team won a Real IT Award for Innovation! Check out the full post here.

Empowering Environmental Science With Open Tools and Data

Mon, 22 May 2017 00:00:00 +0000

Abstract

The Met Office is a world leading weather and climate centre with the largest operational supercomputer in Europe. With their data growing rapidly every year, they’ve realized a need to change their approach to analysing it. This talk covers how they are working with a suite of open source tools to solve this problem. They have mixed popular projects, like Jupyter Notebooks, with new data processing libraries (like Dask) and their own bespoke weather analysis tools (like Iris). They are also opening large volumes of data to the public, which through these tools can be manipulated quickly and easily from the comfort of a web browser.

Monitoring scalable infrastructure

Mon, 08 May 2017 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on May 8th, 2017.

Recently we’ve been thinking a lot about monitoring. In a world of ephemeral servers, auto-scaling, spot instances and infrastructure-as-code, monitoring has to be tackled differently.

Using Jupyter notebooks for SysAdmin, CloudOps and DevOps workflows.

Mon, 08 May 2017 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on May 8th, 2017.

Jupyter notebooks are awesome. If you speak to a data scientist or analyst who writes Python there’s a very good chance that they use Jupyter notebooks. But I think there’s another community that would benefit hugely from including them in their standard arsenal of tools, and that’s folks in IT Infrastructure.

Moving large volumes of data to S3

Thu, 20 Apr 2017 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on April 20th, 2017.

We just moved ~80TB of data to S3 (stay tuned to hear what we’re doing with it).

Tue, 28 Mar 2017 00:02:24 +0100

Apple Airport Express Repair

Mon, 20 Mar 2017 00:00:00 +0000

Introduction

I recently acquired an Apple Airport Express wireless hotspot which wouldn’t power on. This was most likely down to a fault on the power supply board and so I decided to have a go at fixing it.

Building Telegraf for 32bit FreeBSD

Mon, 16 Jan 2017 00:00:00 +0000

Introduction

Currently InfluxData do not provide a 32bit FreeBSD build of Telegraf as part of their standard packages. Luckily it is easy to build yourself.

Cloud Utilisation in the Sciences

Fri, 04 Nov 2016 00:00:00 +0000

Processing Environmental Big Data in the Browser for Science

Sat, 29 Oct 2016 00:00:00 +0000

We should split the session into roughly three activities, with the final one being totally open ended. This caters for most ability levels and will allow everyone to fill the entire session.

SVG Animations

Sat, 08 Oct 2016 00:00:00 +0000

Build games for iOS 10 with Xcode 8 and Game Maker Studio 1.4

Tue, 20 Sep 2016 00:00:00 +0000

Introduction

Building games for iOS is straight forward with Game Maker. You create the game as normal in Game Maker, but in order to build it you must have a Mac with Xcode installed. You must configure Game Maker with the IP address of your Mac and the username and password. When you build the project Game Maker will produce an Xcode compatible project, copy it onto your Mac and open it in Xcode. You can find comprehensive instructions on the YoYo Games website.

A game on the perception of symbols

Fri, 24 Jun 2016 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on June 24th, 2016.

With some friends look out the window and each choose a weather symbol which represents what you see. Do you all agree?

Running Telegraf inside a docker container

Thu, 23 Jun 2016 00:00:00 +0000

Telegraf is an application for collecting server and application telemetry and metrics and sending them to a time series datastore like InfluxDB. Like me you may prefer running all of your applications in Docker containers, however this means Telegraf will only collect data for the container. This article will cover the configuration options to allow Telegraf to collect host metrics from inside a container.

Getting started with VMwares ESXi/vSphere API in Python

Wed, 22 Jun 2016 00:00:00 +0000

In 2013 VMware dropped their Python library for accessing the API for ESXi/vSphere on GitHub. This is great, however it isn’t the easiest library in the world to use. This quick guide will show you how to connect to an ESXi host or vSphere cluster and get some info about a virtual machine.

Cracking Enigma with Go

Thu, 02 Jun 2016 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on June 2nd, 2016.

Can I crack the Enigma code with Go on a MacBook? Yes!

A note on AWS disk performance testing

Fri, 15 Apr 2016 00:00:00 +0000

Here is an interesting note on testing the disk performance of your AWS instances. Before you can acurately test the performance of your EBS disk you need to read all the sectors of the disk at least once if that disk was created from a snapshot.

Interactive Docker containers

Thu, 31 Mar 2016 00:00:00 +0000

Interactive shell

In this post we are going to create a container we can interact with. We can then have a poke around inside the container and see what it is and how it works.

Running a Docker container

Tue, 29 Mar 2016 00:00:00 +0000

Installing Docker

Installing Docker on your machine is required but beyond the scope of this series. Getting Docker up and running is an ever evolving and improving process and anything put here will go stale reasonably quickly. As Docker uses linux kernel features you will need a running linux operating system. Therefore installing Docker on linux is easy, however installing it on Windows and Mac involve running a lightweight linux virtual machine.

Killing Containers to Make Weather Beautiful

Fri, 25 Mar 2016 00:00:00 +0000

The Met Office Informatics Lab includes scientists, developers and designers. We build prototypes exploring new technologies to make environmental data useful. Here we describe a recent project to process multi-dimensional weather data to create a fully interactive 4D browser application. We used long-running containers to serve data and web pages and short-running processes to ingest and compress the data. Forecast data is issued every three hours so our data ingestion goes through regular and predictable bursts (i.e. perfect for autoscaling).

What is Docker?

Tue, 22 Mar 2016 00:00:00 +0000

Welcome to the first post in ‘Intro to Docker’, a series I’m writing to get beginners started with Docker. Each post in this series will be no more than a 5 minute read and will cover exactly one topic surrounding Docker.

How to use an Xbox 360 controller with OS X El Capitan

Wed, 24 Feb 2016 00:00:00 +0000

Introduction

In order to use an Xbox 360 controller with OS X El Capitan you will need to install a driver for it. This is an update to my article on using an Xbox 360 controller with Yosemite.

MacBook Rebuild

Fri, 22 Jan 2016 00:00:00 +0000

Introduction

I’ve decided to wipe my MacBook and reinstall everything! The current installation has been hanging around for the last few years and has even migrated between hardware with Time Machine. Things are starting to feel a little slow, I’ve installed lots of things in the past which I don’t want any more, and I just want a fresh start. I’ve backed it up with Time Machine just in case and I’m ready to get started.

Pretty git logs with `git lg`

Mon, 18 Jan 2016 00:00:00 +0000

This is a repost of a Stack Overflow answer, mainly to preserve it for myself. Slipp Thompson posted some really nice aliases for showing branch topology in the git command line.

Fixing the SSH roaming vulnerability (CVE-2016-0777)

Fri, 15 Jan 2016 00:00:00 +0000

A vulnerability in the OpenSSH client has been discovered which means that if you SSH to a compromised server the server can steal your private key. This affects any operating system with OpenSSH client 5.4 and above, which is pretty much all flavors of linux and OS X.

Twenty Fifteen Roundup

Thu, 31 Dec 2015 00:00:00 +0000

Introduction

Here is a list of the technology and media I enjoyed this year, a follow up on last year’s post.

Thoughts on Star Wars: The Force Awakens

Mon, 21 Dec 2015 00:00:00 +0000

Spoilers ahead, you have been warned!

I thoroughly enjoyed the new Star Wars. Like I assume many people did I went to the cinema with concerns that the film would not live up to expectations. However roughly around the moment where Rylo Ken trapped the blaster shot in mid air those worries went away.

A Raspberry Pi Docker Cluster

Wed, 09 Dec 2015 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on December 12th, 2015.

Introduction

We are fortunate in the Lab to have a small stash of Raspberry Pis in our cupboard which are used at hackathons and other events. As there are no events using them currently I thought I’d take the opportunity to make a nice demonstration piece to show off clustering containers.

Intro to Kubernetes

Thu, 29 Oct 2015 00:00:00 +0000

Building with Kubernetes

Thu, 01 Oct 2015 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on October 1st, 2015.

For our 3D visualisation project we wanted to build a data processing service using Docker containers. We quickly found that once you are running more than a couple of containers you need a way to manage them. After looking into the different tools available we decided to give Kubernetes a go, this is what we learned.

Quick Tip - git delete merged branches

Thu, 01 Oct 2015 00:00:00 +0000

Here’s a quick line to run in your terminal to delete all local git branches which have already been merged into master.

Quick Tip - em vs rem

Fri, 04 Sep 2015 00:00:00 +0000

em and rem are used in CSS to set a size value relative to a font-size. This is useful in many situations such as increasing the font size relatively across your whole website by changing one value or adding padding which is larger or smaller depending on the font size.

govspeak: An open source markup language

Wed, 22 Jul 2015 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on July 22nd, 2015.

The Informatics Lab website is created with an application called Jekyll. Recently I made an enhancement to it which I’m very excited about. It allows us to write our articles in a markup language called Govspeak, which is an extension to the excellent markdown.

Lab School: Docker

Wed, 24 Jun 2015 00:00:00 +0000

Originally published on the Met Office Informatics Lab blog on June 24th, 2015.

Welcome to the first ever Lab School session. This session aims to give you an overview of docker and how we are currently using it in the Lab.

Updating flightradar24 with a Raspberry Pi

Fri, 29 May 2015 00:00:00 +0000

You can feed data into flightradar24 (from now on referred to as fr24) simply using a Raspberry Pi and a cheap USB DVB-T tuner.

Collaborative article corrections in Jekyll

Wed, 27 May 2015 00:00:00 +0000

Don’t you find it really useful when you publish an article on your blog and then someone comes up to you and says:

Simple reading speed estimate in Jekyll

Tue, 24 Feb 2015 00:00:00 +0000

When browsing through off-the-shelf Jekyll themes recently I stumbled across one I really like called Pixyll. There are lots of things I like about the theme but one thing in particular is the reading speed estimate at the top of each article. Not only is it a nice feature but the code is simple and concise too!

Test your Jekyll blog with Travis CI

Wed, 18 Feb 2015 00:00:00 +0000

Introduction

Testing your blog may sound like an odd thing to do, but if you’re running a Jekyll blog hosted on GitHub it is simple to set up and really useful for notifying you about broken links and other issues.

Bullet Journaling in 2015

Tue, 03 Feb 2015 00:00:00 +0000

Introduction

It is one month into 2015 and I am on page 31 of my latest bullet journal. I started using bullet journaling to organise myself in March last year and now I can’t imagine my life without it.

How to install and configure inadyn on CentOS 6

Mon, 05 Jan 2015 00:00:00 +0000

Introduction

Inadyn is a command line utility for periodically checking and updating your ip address with DynDNS.

How to install VMware Tools on Centos 6 with yum

Wed, 31 Dec 2014 00:00:00 +0000

Introduction

Often when managing a large number of systems you want to manage all software installs the same way. So when it comes to VMware Tools you may not want to follow the official instructions but instead install using yum, especially if you’re automating a large number of headless systems.

Twenty Fourteen Roundup

Wed, 31 Dec 2014 00:00:00 +0000

Introduction

I’ve decided to finish off the year by writing a roundup of all the technology I’ve used this year. Inspiration taken from sammcj’s “the best of”.

Run OpenVPN on non-standard port with SELinux and Centos 6

Mon, 08 Dec 2014 00:00:00 +0000

I recently installed OpenVPN on a Centos 6 server but found that I couldn’t get the service to start. Running service openvpn start failed despite being able to run openvpn --config /path/to/config without errors.

How to easy_install and pip through a proxy

Tue, 25 Nov 2014 00:00:00 +0000

If you’re trying to install a Python package using easy_install or pip and you connect to the internet via a proxy you’ll need to make a few changes to your setup.

How to install the vSphere 5.5 Client on Windows 8

Wed, 19 Nov 2014 00:00:00 +0000

If you’ve tried installing the vSphere 5.5 client on Windows 8 you may have received the following error message

Simple HTML Redirect

Fri, 14 Nov 2014 00:00:00 +0000

I often find myself in need of a quick html redirect page. Most of the time I use the example from Stack Overflow but it involves changing the url in 3 places.

How I value media and entertainment

Thu, 13 Nov 2014 00:00:00 +0000

I’ve been meaning to write about how I value media/entertainment and my thought process when purchasing it. This is different to my normal style of article but hopefully people will find it interesting.

How to use an Xbox 360 controller with OS X Yosemite

Fri, 17 Oct 2014 00:00:00 +0000

Update: There is a newer version of this article, see How to use an Xbox 360 controller with OS X El Capitan.

Learn How to Build Cool Things With Weather Data in Python

Tue, 30 Sep 2014 00:00:00 +0000

What is semalt and why are they in my analytics?

Wed, 03 Sep 2014 00:00:00 +0000

You may be among the many people having their Google Analytics stats skewed by fake referrals from semalt.

How to stop Google from scanning my site

Thu, 28 Aug 2014 00:00:00 +0000

Sometimes there may be occasions where you don’t want Google (and other search engines) to scan some or all of your website.

Amazon S3: s3cmd put ([Errno 32] Broken pipe)

Thu, 31 Jul 2014 00:00:00 +0000

Recently I decided to use Amazon’s S3 as another location to store some of my server backups. However I found when testing that I was unable to upload my backup tarballs to S3. I ended up with the following errors.

How to install OS X Yosemite Developer Preview Beta in Virtualbox

Sat, 07 Jun 2014 00:00:00 +0000

Like me you may be excited about the Developer Preview Beta of OS X 10.10 Yosemite and want to try it out, but you don’t want to deal with a buggy system between now and the general release. If that’s the case you’ll want to install Yosemite as a virtual machine on your Mac. Here’s how I’ve done it on mine using VirtualBox.

How to use text expansion in OS X 10.9 Mavericks

Thu, 15 May 2014 00:00:00 +0000

Since the upgrade to OS X 10.9 Mavericks and iOS 7 you may have noticed that your text expansion shortcuts from iOS have found their way onto your Mac. Thanks to iCloud all of your text shortcuts are now synchronised between your devices.

How to make screen recordings in OS X Mavericks 10.9

Wed, 14 May 2014 00:00:00 +0000

Did you know that you don’t need any additional software to make high quality, watermark free screen recordings in OS X Mavericks 10.9? Well this feature exists and it’s in a slightly unexpected place … QuickTime Player.

NASA Space Apps Challenge 2014 Roundup

Fri, 25 Apr 2014 00:00:00 +0000

A couple of weeks ago I attended a hackathon hosted by the Met Office as part of the 2014 NASA Space Apps Challenge. It was a weekend event and the basic idea was to meet with other technology professionals/enthusiasts, form teams with these like minded folk, select a challenge from a long list of challenges set by NASA and solve it during the weekend.

How to prepare for a hackathon

Tue, 08 Apr 2014 00:00:00 +0000

So this weekend I’ll be taking part in the 2014 NASA Space Apps Challenge. This time around I’ll be leading a team, rather than just joining one on the day, so I feel like I have to be extra prepared. Here is a checklist of all the things you should do before going to a hackathon/codeathon.

Python script: Recursively remove empty folders/directories

Sun, 16 Feb 2014 00:00:00 +0000

So as part of a script I’m writing I needed the ability to recursively remove empty folders/directories from a filesystem. After a bit of googling I found this very useful script by Eneko Alonso. However the script isn’t really in a usable state for what I want so I decided to make a few changes to it and publish it on GitHub.

Should I buy a cheap upgraded/reformatted SDHC micro SD card on eBay?

Thu, 23 Jan 2014 00:00:00 +0000

Short answer – No!

Now I must admit I am one for buying rubbish on eBay and usually fancy myself as someone who can spot the difference between a bargain and a scam. However this time I almost got scammed.

Convert tweet hashtags, at-tags and urls to links with PHP and Regular Expressions

Wed, 22 Jan 2014 00:00:00 +0000

Now of course if you’re using the Twitter API you can use Twitter entities but in this tutorial we’re going to use regular expressions.

Fixing VirtualBox verr_supdrv_component_not_found when selecting bridged networking on OS X 10.9

Fri, 10 Jan 2014 00:00:00 +0000

While installing CentOS in VirtualBox (version 4.2.4) on OS X (version 10.9.1) I came across the following error message when selecting bridged networking

Google Charts IE7 IE8 Issue: Date formatting problem

Thu, 28 Nov 2013 00:00:00 +0000

Just a quick post about an issue I’ve had with Google Charts on IE7/8.

When viewing my page in Firefox or Chrome my graph displayed as expected.

Mac OS X Terminal Theme: Piperita

Thu, 17 Oct 2013 00:00:00 +0000

UPDATE – This project is now at version 2. See the Piperita GitHub page for up to date documentation and information.

Bootcamp Windows 7 on a 2011 MacBook Pro without a SuperDrive

Mon, 30 Sep 2013 00:00:00 +0000

So recently I swapped out the SuperDrive in my early 2011 MacBook pro for an additional HDD caddy. I then moved my 1TB HDD into the caddy and put a new SSD in the HDD slot. As I was messing around with hard drives I decided to go for a fresh install of OS X and Windows on the SSD.

Fixing "ERROR: Error 35: error:14077458:SSL routines:SSL23_GET_SERVER_HELLO: reason(1112)"

Tue, 13 Aug 2013 00:00:00 +0000

So the other day I came across the following error when using the munki configuration tool for mac.

Sort top command by cpu usage and set to default in OS X

Tue, 04 Jun 2013 00:00:00 +0000

As I come from a linux background but seem to spend more and more of my free time using OS X I keep noticing little differences in the way the command line works on a mac. One difference which as been bugging me recently is that way that the top command orders itself. I’m used to having it ordered by highest processor usage at the top on linux, which I find the most useful as generally when I run top I’m looking to see what is chewing up my cpu. However when you run top in OS X it orders by pid, so the newest processes are at the top.

Why is there no space in the MySQL password parameter?

Fri, 31 May 2013 00:00:00 +0000

After troubleshooting a MySQL issue with a colleague we began discussing a “feature” of the MySQL command line which insists that you don’t put a space in the password parameter when using the short parameter. We both felt that it was rather inconsistent to allow the usage of -h hostname or -u username but insist on -ppassword instead of -p password. You can of course use the full parameter --password=password but as most people use the shorthand commands it just seems slightly unintuitive.

Using the AddThis Share Buttons wordpress plugin in a custom theme

Sun, 17 Mar 2013 00:00:00 +0000

There is an undocumented function for adding a custom AddThis widget to your Wordpress theme when using the Add This Share Buttons plugin, so I thought I would document it here.

Convincing Paypal Phishing Email

Sat, 02 Mar 2013 00:00:00 +0000

So I woke up yesterday morning to find a receipt from Paypal for $149.49 on my iPhone. I haven’t bought anything for $149.49 so right away I was worried that my Paypal account had been broken into. The first thing I did was to log into my Paypal account on my laptop to check to see more information about the transaction. When logging in there was no trace of this transaction.

Has Stack Overflow been hijacked?

Wed, 27 Feb 2013 00:00:00 +0000

Something funny seems to be happening with Google and Stack Overflow.

I first noticed this last night when on my Macbook at home. I went to www.google.co.uk, typed in “stackoverflow” and was presented by the usual page. However I noticed that the url displyed under the link on the search results said www.doioig.gov. When clicking the link it took me to www.doioig.gov instead of www.stackoverflow.com. I thought to myself that this was probably just a temporary issue and went to the correct url myself.

SSH without a password on OS X with ssh-copy-id

Thu, 24 Jan 2013 00:00:00 +0000

What is ssh-copy-id?

ssh-copy-id is a script that uses ssh to log into a remote machine (presumably using a login password, so password authentication should be enabled, unless you’ve done some clever use of multiple identities) It also changes the permissions of the remote user’s home, ~/.ssh, and ~/.ssh/authorized_keys to remove group writability (which would otherwise prevent you from logging in, if the remote sshd has StrictModes set in its configuration).

Album Review: Muse 2nd Law

Thu, 11 Oct 2012 00:00:00 +0000

Well I just got “2nd Law” by Muse and as I’m a big Muse fan I thought I’d write a quick review of it. Up until now I’ve enjoyed all of Muse’s albums with the exception of album 5 “The Resistance” which I felt was a little too experimental for my liking.

Using background-position and sprite sheets to stop icon hover flicker

Tue, 18 Sep 2012 00:00:00 +0000

While updating the theme on this blog I added some links to my social websites. I made these links in the form of images which were black and white and become coloured when hovered over. To create these icons I created empty divs which would then be styled in CSS.

Incorrect Gmail password when using exchange on iPhone

Wed, 12 Sep 2012 00:00:00 +0000

The Problem

Recently I’ve tried playing around a little with the email account settings on my iPhone to try and get as much to sync as possible with some of my other accounts. One thing I did was to change Gmail from the standard ‘Gmail’ setting that you get on iOS to the ‘Microsoft Exchange’ setting as recommended by Google.

How to query a database with AJAX and display as a tooltip

Tue, 11 Sep 2012 00:00:00 +0000

This post began as an answer on Stack Overflow to a question on ‘How to query a database with AJAX and display as a tooltip‘. I have put the answer here for future reference.

Download Festival 2012 Timetable

Fri, 01 Jun 2012 00:00:00 +0000

Download have now released the times for Download Festival 2012 in the form of an app. The only problem with this is it’s not exactly clear and easy to see who is clashing or to make a plan. For my own benefit and anyone else who wants it I’ve put the timetable into a spreadsheet and put it in google docs.

Shrinking SQL logs

Thu, 23 Feb 2012 00:00:00 +0000

While reading some articles about minimizing the size of data while leaving the data fully searchable I came across a lot of info about finding common data and replacing it with pointers to a dictionary or store of common strings. After thinking about it for a while I thought I would just write a little post about a way I try and minimize the size of databases in projects that I work on. Now I’m going to start with an example of a very bad database I wrote a while back and how I used relational tables to shrink the size considerably. Now if you know anything about databases you should find all of this quite obvious but if you don’t then it may be worth taking note of.

The lesser known browser war

Tue, 07 Feb 2012 00:00:00 +0000

Well it’s been quite a while since I’ve written a post on here but due to a suggestion from a friend I’m going to try and make an effort to write more about the projects I work on and the random thoughts that fall out of my brain. A few things have changed since my last post but the main thing would be my job. I am now a software test analyst, aka that guy who has to pick fault with your hard work. I’m enjoying this position and it is actually quite nice finding bugs and knowing you’re not going to have to be the one that fixes them.

Apple game center hacks

Sat, 04 Jun 2011 00:00:00 +0000

Ok I’m currently on holiday with my girlfriend in turkey. It started getting a bit too hot in the sun a few days ago so we went into the hotel lobby and found free wifi. We went to our room and both got out our iOS devices and went to the lobby to catch up with Facebook etc.

Additional HDD Philosphy

Sat, 25 Dec 2010 00:00:00 +0000

A friend has asked me what I think the best way for him to set up his new HDD is. He currently has a 500GB Internal HDD and has purchased a second one. So here is what I would do.

Guide to repairing TFT monitor scratches

Thu, 23 Dec 2010 00:00:00 +0000

About a year and a half ago I was working on a guitar modification and dropped part of the modified casing on my laptop. It made a pretty good scratch on the bottom of my monitor about an inch long (2.54cm for you metrics out there). It has mildly bothered me since then but I’ve never done anything about it. But this evening I decided to google around and see if there was any kind of specialist product that could be purchased and used to fill in the scratch. I read a few blogs and sites until I found the most amazing solution ever.

Ubuntu setup 2010

Wed, 15 Dec 2010 00:00:00 +0000

I am now happy with my Ubuntu installation. After a small hiccup earlier where it decided not to do an update properly, disable my mouse and keyboard and then ask for a password for them to be re-enabled. Solved with a trusty USB mouse and onscreen keyboard. I have managed to find a PHP/HTML/JS editor to my liking. I have looked through a few more since my post last night and have settled on Komodo. It highlights my code properly, auto completes functions and even highlights closing tags when you select the opening one which is going to make my life with tables and divs a hell of a lot easier. Definitly a viable replacement for Dreamweaver as the only thing I missed was the colour scheme which has easily been remedied in the settings.

Back to Ubuntu

Tue, 14 Dec 2010 00:00:00 +0000

For as long as I’ve been interested in computers I’ve been interested in linux. I was introduced to it by a friend at school when I was 13, we installed mandrake in vmware. We only admired it for a few minutes and then deleted it. He later introduced me to ubuntu and he began using it as his only operating system and still does to this day. I personally couldn’t move from Windows mainly due to gaming but also because I can’t stand to lose photoshop, dreamweaver etc. I also have a problem with wine, vmware and virtualbox, why emulate windows when you can just run it.