Which system for cloud-based cluster in OpenStack? (Kubernetes, Slurm, others?)

I have professional access to a cloud platform (OpenStack) with the following quota:

  • 128 vCPUs
  • 40 vGPUs
  • 528 GB RAM
  • 125 TB storage
  • max. 10 virtual machines / instances
  • 5 public ips
  • ... There is also an S3 storage with 18 PB of data (remote sensing data) attached, which we are working with.

I want to set up a kind of small cluster on this platform to run data science with Python and R for my colleagues and me. I would like to create scripts on the platform in a JupyterHub or R server, for example, and then use the entire contingent to process the huge amount of data with machine learning.

The question I have is how can I create some sort of cluster? I'm currently learning about Docker and Kubernetes, but I also know about Slurm, which is used in HPCs.

Which system is right one for our purpose? Kubernetes, Slurm, others???

submitted by /u/beyerflorian
[link] [comments]