Machine Learning Performance Engineer, Training
Tower Research CapitalAbout the Role
You will bridge the gap between quantitative research and high-performance computing, building and optimizing the systems used to train machine learning models at scale. You will focus on accelerating the end-to-end training lifecycle—from data ingestion and distributed execution to kernel performance and hardware utilization—enabling researchers to iterate more quickly across increasingly complex models and datasets.
- Benchmark model-training workloads across CPUs, GPUs, and other accelerator platforms to identify bottlenecks and guide Tower’s compute infrastructure decisions.
- Develop performance models and standardized benchmarks for measuring throughput, utilization, scalability, and time to convergence.
- Design and optimize distributed training strategies, including data, tensor, pipeline, and model parallelism.
- Improve communication efficiency across multi-GPU and multi-node environments by optimizing collective operations, topology awareness, and computation–communication overlap.
- Analyze and improve the full training pipeline, including data loading, preprocessing, memory management, forward and backward passes, optimizer execution, checkpointing, and experiment recovery.
- Develop and optimize GPU kernels and performance-critical framework components for quantitative machine learning workloads.
- Apply techniques such as mixed-precision training, gradient accumulation, activation checkpointing, operator fusion, and memory-efficient optimizers.
- Partner with HPC and infrastructure teams to optimize workload scheduling, resource allocation, observability, fault tolerance, and reproducibility across shared compute environments.
- Help define the architecture and tooling required to support large-scale experimentation across on-premises and cloud-based infrastructure.
- Work closely with ML Researchers, Quantitative Researchers, HPC Engineers, Systems Engineers, and hardware specialists to translate research requirements into highly efficient training systems.
Responsibilities
- Benchmark model-training workloads across CPUs, GPUs, and other accelerator platforms to identify bottlenecks and guide Tower's compute infrastructure decisions
- Develop performance models and standardized benchmarks for measuring throughput, utilization, scalability, and time to convergence
- Design and optimize distributed training strategies, including data, tensor, pipeline, and model parallelism
- Improve communication efficiency across multi-GPU and multi-node environments by optimizing collective operations, topology awareness, and computation–communication overlap
- Analyze and improve the full training pipeline, including data loading, preprocessing, memory management, forward and backward passes, optimizer execution, checkpointing, and experiment recovery
- Identify bottlenecks across compute, memory, storage, networking, and interconnects to increase accelerator utilization and researcher productivity
- Develop and optimize GPU kernels and performance-critical framework components for quantitative machine learning workloads
- Integrate specialized libraries, compilers, and execution techniques to improve throughput, memory efficiency, and numerical performance
- Apply techniques such as mixed-precision training, gradient accumulation, activation checkpointing, operator fusion, and memory-efficient optimizers
- Evaluate tradeoffs among training speed, numerical stability, reproducibility, model quality, and infrastructure cost
- Partner with HPC and infrastructure teams to optimize workload scheduling, resource allocation, observability, fault tolerance, and reproducibility across shared compute environments
- Help define the architecture and tooling required to support large-scale experimentation across on-premises and cloud-based infrastructure
- Work closely with ML Researchers, Quantitative Researchers, HPC Engineers, Systems Engineers, and hardware specialists to translate research requirements into highly efficient training systems
Requirements
- 3+ years of experience optimizing machine learning training workloads in high-performance, distributed, or large-scale computing environments.
- Deep knowledge of machine learning frameworks such as PyTorch or JAX, including their execution models, compilation paths, autograd systems, and distributed-training capabilities.
- Strong programming skills in Python and C++, with experience developing or optimizing performance-critical systems.
- Proven experience with GPU kernel development and optimization using technologies such as CUDA, Triton, CUTLASS, cuBLAS, cuDNN, or related libraries.
- Strong understanding of GPU architecture, including streaming multiprocessor execution, warp scheduling, tensor cores, and the memory hierarchy from registers through HBM.
- Experience with distributed-training technologies and communication libraries such as NCCL, FSDP, DeepSpeed, Megatron-LM, XLA, or equivalent systems.
- Proficiency with performance-analysis tools such as Nsight Systems, Nsight Compute, PyTorch Profiler, or comparable tracing and profiling platforms.
- Understanding of high-performance networking, storage, and accelerator interconnects, including technologies such as InfiniBand, RDMA, NVLink, or NVSwitch.
- Demonstrated ability to benchmark heterogeneous compute platforms and make rigorous, data-driven recommendations about performance, scalability, and cost.
- Experience optimizing training workloads for transformer-based, time-series, reinforcement-learning, or other computationally intensive models.
- Experience with cluster orchestration and scheduling technologies such as Kubernetes, Slurm, Ray, or similar platforms.
- Familiarity with fault-tolerant distributed training, large-scale checkpointing, experiment reproducibility, and GPU-cluster observability.
- Practical experience with specialized accelerators, custom hardware, or compiler technologies for machine learning.
What You'll Get
- $200k base salary
- Generous paid time off policies
- Savings plans and other financial wellness tools available in each region
- Hybrid working opportunities
- Free breakfast, lunch and snacks daily
- In-office wellness experiences and reimbursement for select wellness expenses (e.g., gym, personal training and more)
- Volunteer opportunities and charitable giving
- Social events, happy hours, treats and celebrations throughout the year
- Workshops and continuous learning opportunities
- Open concept workplace
- Casual dress code
- Well-stocked kitchens