#dask
Tag

Running Dask on Databricks

Feb 15, 2024 4 minute read #python, #dask, #databricks, #distributed-computing, #cloud-deployment

Databricks is a very popular data analytics platform used by data scientists, engineers, and businesses around the world.
Running Dask workloads on multiple cluster backends with zero code changes using dask-ctl

Jan 25, 2024 3 minute read #python, #dask, #dask-ctl, #distributed-computing

Sometimes you want to write some code using Dask which can then be run against multiple different cluster backends.
The challenge of updating an aging blog

Aug 23, 2023 9 minute read #blog, #oss, #community, #dask, #notes

The Dask blog is a bit neglected these days. The website is an aging Jekyll blog and is well past it’s prime.
Debugging Data Science workflows at scale

May 12, 2023 15 minute read #python, #dask, #kubernetes, #apache-beam, #google-cloud, #google-kubernetes-engine

The more we scale up our workloads the more we run into bugs that only appear at scale.
Running Jupyter in your Dask Kubernetes cluster

Mar 10, 2023 2 minute read #dask, #kubernetes, #jupyter

Did you know that the Dask scheduler has a --jupyter flag that will start a Jupyter server running within the Dask Dashboard?
Accelerating ETL on KubeFlow with RAPIDS

Aug 30, 2022 11 minute read #dask, #etl, #kubeflow, #pandas, #rapids, #technical-walkthrough Archive

In the machine learning and MLOps world, GPUs are widely used to speed up model training and inference, but what about the other stages of the workflow like ETL pipelines or hyperparameter optimization?
Using Dask on KubeFlow with the Dask Kubernetes Operator

Jul 27, 2022 10 minute read #python, #dask, #kubernetes, #kubeflow, #operator, #data-science

Kubeflow is a popular Machine Learning and MLOps platform built on Kubernetes for designing and running Machine Learning pipelines for training models and providing inference services.
How to set environment variables on your Dask workers

May 5, 2022 2 minute read #python, #dask, #snippet

When working with Dask clusters you often need the remote worker environment to match you local environment.
What is the difference between Dask and RAPIDS?

Jun 22, 2021 3 minute read #rapids, #dask, #pydata Archive

Both Dask and RAPIDS are Python libraries to scale your workflow and empower you to process more data and leverage more compute resources.
The evolution of a Dask Distributed user

Jun 1, 2021 9 minute read #dask, #python, #distributed, #user-journey Archive

This week was the 2021 Dask Summit and one of the workshops that we ran covered many deployment options for Dask Distributed.
Monitoring Dask + RAPIDS with Prometheus + Grafana

Apr 9, 2021 6 minute read #rapids, #dask, #monitoring, #prometheus, #grafana Archive

Prometheus is a popular monitoring tool within the cloud community. It has out-of-the-box integration with popular platforms including Kubernetes, Open Stack, and the major cloud vendors, and integrates with dashboarding tools like Grafana.
Running Dask tutorials

Aug 21, 2020 20 minute read #python, #dask, #distributed-computing, #open-source, #community, #tutorials Archive

Originally published on the Dask blog on August 21st, 2020. For the last couple of months we’ve been running community tutorials every three weeks or so.
The current state of distributed Dask clusters

Jul 23, 2020 19 minute read #python, #dask, #distributed-computing Archive

Originally published on the Dask blog on July 23rd, 2020. Dask enables you to build up a graph of the computation you want to perform and then executes it in parallel for you.
Exploring Dask and Distributed on AWS Lambda

Apr 13, 2018 5 minute read #dask, #distributed, #aws, #aws-lambda Archive

I spent some time this week exploring whether it would be possible to run Dask and Distributed on a function as a service platform like AWS Lambda.
Instant access to auto-scaling personal Python clusters

Feb 7, 2018 5 minute read #dask Archive

Originally published on the Met Office Informatics Lab blog on February 7th, 2018. We are excited to announce that the work we’ve been doing with distributed Dask clusters running on Kubernetes has been absorbed into an awesome new tool called Daskernetes through our work on the Pangeo project.
Adaptive Dask clusters on Kubernetes and AWS

Jul 21, 2017 7 minute read #dask, #aws, #kubernetes, #autoscaling Archive

Originally published on the Met Office Informatics Lab blog on July 21st, 2017. Introduction This article assumes a basic understanding of Amazon Web Services (AWS), Kubernetes, Docker and Dask.

#dask Tag

#dask
Tag