AWS Batch

Install OpenSRE on AWS Batch (Multi-Node) under two minutes

Try OpenSRE in Your AWS Batch Workloads

CloudFormation

Use CloudFormation to set up AWS Batch infrastructure for bioinformatics and HPC pipelines.

1. Connect AWS Integration

Open the following link to connect your AWS account automatically to OpenSRE. This will allow OpenSRE to monitor your AWS Batch jobs.

→ Connect AWS

2. Locate the EC2 LaunchTemplate UserData

In your CloudFormation template, find the UserData field within the Launch Template used by your AWS Batch compute environment.

Reference example: CloudFormation Template

Resources:
  NextflowLaunchTemplate:
    Properties:
      LaunchTemplateData:
        UserData: !Base64 |
          #cloud-config
          runcmd:
            - ...

3. Append OpenSRE install in EC2 Launch Template UserData

Add these commands to the runcmd: section in your UserData:

- curl -sSL https://install.tracer.cloud | sh

- tracer init --token YOUR_TOKEN_HERE \
     --pipeline-name YOUR_PIPELINE_NAME \
     --pipeline-type YOUR_PIPELINE_TYPE \
     --environment aws_batch \
     --watch-dir="/var/log"

Go to our onboarding to get your own personal token

4. Validate, apply change set, then verify instance

aws cloudformation validate-template \
  --template-body file://YOUR_TEMPLATE.yaml

aws cloudformation create-change-set \
  --stack-name YOUR_STACK_NAME \
  --template-body file://YOUR_TEMPLATE.yaml \
  --change-set-name tracer-userdata-update \
  --capabilities CAPABILITY_NAMED_IAM

aws cloudformation execute-change-set \
  --stack-name YOUR_STACK_NAME \
  --change-set-name tracer-userdata-update

5. Configure Nextflow batch.config with Trace ID

In your Nextflow batch.config file, you need to:

1. Add a params section to generate a UUID that will be shared across all jobs in the workflow
2. Add containerOptions in the process section to pass the trace ID as an environment variable to each container

This allows OpenSRE to correlate all jobs from the same workflow execution.

// 1. Add params section to generate UUID
params {customUUID = java.util.UUID.randomUUID().toString()}

process {executor = 'awsbatch'
    queue = 'NextflowCPU'

// 2. Add containerOptions to pass trace ID to containers
containerOptions = "--env TRACER_TRACE_ID=${params.customUUID}"

resourceLabels = ['custom-session-uuid': "${params.customUUID}"]
}

aws {region = 'YOUR_REGION'
  batch
  {cliPath = '/usr/local/aws-cli/v2/current/bin/aws'}
}

Monitor and Optimize Your Pipeline

Watch your pipeline run in the OpenSRE dashboard

View real-time metrics, resource usage, and performance insights for your pipeline runs.

Terraform

Use Terraform to provision infrastructure and run RNA-seq pipeline on AWS Batch.

Prerequisites

Make sure these tools are installed locally before proceeding.

• AWS CLI v2

• Terraform

• Nextflow

• Java

• Docker (optional)

1. Connect AWS Integration

Open the following link to connect your AWS account automatically to OpenSRE. This will allow OpenSRE to monitor your AWS Batch jobs.

→ Connect AWS

2. Clone the repository

Clone the nextflow-test-pipelines repository to get access to the AWS Batch RNA-seq pipeline configuration.

git clone https://github.com/Tracer-Cloud/nextflow-test-pipelines.git
cd nextflow-test-pipelines/pipelines/aws-batch/rnaseq

3. Update user_data_mime.sh

Update the user_data_mime.sh file to include your OpenSRE user ID. Navigate to the terraform directory and edit the file:

cd nextflow-test-pipelines/pipelines/aws-batch/rnaseq/terraform
vim user_data_mime.sh

Important: Find line 120 in the file and update it with your user ID:

tracer init --token YOUR_TOKEN_HERE
--pipeline-name aws_batch_rnaseq
--pipeline-type RNA-SEQ
--environment aws-batch
--watch-dir="/var/log"

Go to our onboarding to get your own personal token

Note: If you need to change this later, you’ll need to refresh the infrastructure.

View the exact line to update

4. Setup infrastructure

Run the setup script to create the AWS infrastructure (VPC, S3, Batch, etc.):

cd nextflow-test-pipelines/pipelines/aws-batch/rnaseq
./run.sh setup

What this does: Creates VPC with public/private subnets, AWS Batch compute environment, S3 buckets for work directory and outputs, IAM roles and policies, security groups, and EC2 Instance Connect endpoint.

Alternative for OpenSRE AWS Account Users: If you have access to OpenSRE’s AWS account, you can skip the infrastructure setup and run the pipeline directly: bash cd nextflow-test-pipelines/pipelines/aws-batch/rnaseq ./run.sh tracer run

5. Run the RNA-seq pipeline

Execute the RNA-seq pipeline using the run script:

cd nextflow-test-pipelines/pipelines/aws-batch/rnaseq
./run.sh run

Monitor and Optimize Your Pipeline

Watch your pipeline run in the OpenSRE dashboard

View real-time metrics, resource usage, and performance insights for your pipeline runs.