I have professional access to a cloud platform (OpenStack) with the following quota:
- 128 vCPUs
- 40 vGPUs
- 528 GB RAM
- 125 TB storage
- max. 10 virtual machines / instances
- 5 public ips
- … There is also an S3 storage with 18 PB of data (remote sensing data) attached, which we are working with.
I want to set up a kind of small cluster on this platform to run data science with Python and R for my colleagues and me. I would like to create scripts on the platform in a JupyterHub or R server, for example, and then use the entire contingent to process the huge amount of data with machine learning.
The question I have is how can I create some sort of cluster? I’m currently learning about Docker and Kubernetes, but I also know about Slurm, which is used in HPCs.
Which system is right one for our purpose? Kubernetes, Slurm, others???