I am looking for some recommendations on how to set up my homelab. Specifically with software/technologies

I have:

3x R630s with 512GB each and 44t/88c

1x R730 with 384GB 36c/72t and a 42x16TB drive JBOD DAS array attached, a 4x NVME 2TB pcie card, and a GTX1660 (currently running unraid, but might change that)

1x R420 with 96GB RAM and 32c/64t cpus (I think)

1x C4140 with 16c/32t, 256GB ram, and 4x P100 GPUs (just bought V100s to replace)

All servers have Connectx3 cards in them (40G/56G) and a SX6036 switch. I just got these and have no idea what I am doing yet.. All servers also have dual 10G SPF Nics that are connected to a switch for regular ethernet

and my workstation that has a threadripper 5995wx, 1TB Ram, and 4x 3090s (will be upgraded to 5090s when they drop). It is running windows and WSL (also dual booted to Ubuntu 22.04 due to a bug with WSL and 4 GPUs)

I have a large dataset taking up 70% of the 500TBs from commoncrawl. I was thinking K8s with the r420 as the master and 630s as worker nodes. I also might throw the 4140 and the 730 in the cluster too. I currently have Minio on a docker image on the 730 but I think it is slow for what I am trying to do, therefore I was going to move it to the K8s cluster but I only have 1 chassis for the drives. I see all this other technology (Hadoop, Spark, Minio, etc). I am doing this to learn primarily. The only way I really learn is hands on. My goal is to try to replicate what the big guys do, at a much smaller scale, but learning the technologies that I will need if I want to shift into this field. So given this layout, wanting to be able to build models and use the hardware as efficiently as possible (meaning if I am preprocessing, all CPUs are at full tilt until its done, if I am training all GPUs are at full tilt until its done) and storage access is as fast as I can make it, how would you configure this?

Also, if there is something I need to buy that is inexpensive to make this much better, I am open to suggestions.

edit:

I also need the dataset externally accessible (that is why I am using Minio)

tl;dr:

given this equipment, and the workload (also being a home lab) how would you configure it? Do i bring in the 730 into the cluster, or set it up as a trunas/unraid setup, or something else since I have 56GbE and IB(RDMA, RCoE)