Posts by /u/alexpwn

Fully managed web-based platform for running software jobs and experiments

Hey everyone. I’m a researcher based at a Computer Science department in Cape Town, South Africa. I do a lot of work in the field of Evolutionary Computation, and that means I’m constantly running computationally expensive code and repeated experiments. One thing I’m really tired of is the manual and tedious effort that goes into getting the code for my experiments up and running on some hardware, be it a cluster or AWS.

Currently, to run large code jobs and batches, you have a few options. You can use a cluster with some sort of job scheduler like Slurm, you can use a platform like AWS and Lambda, Fargate, etc. or you can manage things manually. Then you have to track and visualise all the jobs you’ve run for a project (Google sheets for example).

I’m building a web based platform with an intuitive UI that abstracts and automates a lot of the work needed to set up job scheduling, so that software jobs can easily be run on the cloud either serverlessly or through linked hardware. There’ll also be a nice component for visualising your projects and how the jobs in them are progressing, and viewing and downloading results.

I’ve already built a minimum version of this for running a set of repeated experiments in my research, using AWS instances. It helped a lot with running and managing large amounts of software jobs and I couldn’t help but feel like others may find something like this useful.

I’d love to hear what you guys think.

submitted by /u/alexpwn
[link] [comments]

Read More