Deploying Sourcegraph executors using Terraform on AWS

A Terraform module is provided to provision machines running executors on AWS.

See also: Deploying on Google Cloud

Basic Definition

The following is the minimum required definition to deploy an executor on AWS.

TERRAFORM
module "executors" {
  source  = "sourcegraph/executors/aws"

  # Find the latest version matching your Sourcegraph version here:
  # https://github.com/sourcegraph/terraform-aws-executors/tags
  version = "<version>"

  availability_zone = "<availability zone to provision resources in>"

  executor_sourcegraph_external_url            = "<external url>"
  executor_sourcegraph_executor_proxy_password = "<shared secret>"

  # Either:
  executor_queue_name                          = "<codeintel | batches>"
  # Or:
  executor_queue_names                         = ["codeintel", "batches"]

  executor_instance_tag                        = "<tag to filter in monitoring>"
  executor_metrics_environment_label           = "<label to filter custom metrics>"
  executor_use_firecracker                     = true
}

See the AWS Terraform module variables for additional configurations.

Terraform Version

The executor Terraform modules require Terraform >= 1.1.0, < 2.0.0.

Permissions

Access to get and create in the following resources.

• Auto Scaling
• CloudWatch Logs
• EBS (EC2)
• EC2 (Elastic Compute Cloud)
• IAM (Identity & Access Management)
• VPC (Virtual Private Cloud)

Supported Regions

• us-east-1
• us-east-2
• us-west-1
• us-west-2
• eu-west-1
• eu-west-2
• eu-west-3
• eu-north-1
• eu-south-1
• eu-central-1
• ap-northeast-1
• ap-northeast-2
• ap-southeast-1
• ap-southeast-2
• ap-southeast-3
• ap-east-1
• ap-south-1
• sa-east-1
• me-south-1
• af-south-1
• ca-central-1

Examples

Single Executor

Provisions a single executor to pull from the codeintel queue.

• AWS example

Multiple Executors

Provisions two executors, one to pull from the codeintel queue and the other for the batches queue.

• AWS example

Private Single Executor

Provisions a single executor in a private subnet (no public IP). A NAT Gateway is used for outbound internet traffic.

• AWS example

Step-by-step Guide

The following is a step-by-step guide on provisioning a single codeintel executor on AWS.

Provision

1. Install Terraform.
2. Install the AWS CLI.
3. Run aws configure to set up your credentials and default region.
4. Set up your Sourcegraph instance's Site configuration for executors
   1. Click on your profile picture in the top right corner
   2. Select Site admin
   3. Expand the Configuration section
   4. Select Site configuration
   5. Set the following,
      • "externalURL": "<URL>"
        • A URL that is accessible from the EC2 instance (e.g. a public URL such as https://sourcegraph.example.com)
      • "executors.accessToken": "<new long secret>"
        • Can be generated by running cat /dev/random | base64 | head -c 20
        • The secret will be displayed as REDACTED once it's saved.
      • "codeIntelAutoIndexing.enabled": true
        • This is only for codeintel executors.
5. Download the example files.
6. Change the following in providers.tf
   • region to the AWS region to provision the executor in
7. Change the following in main.tf
   • availability_zone to an availability zone within your chosen region (e.g. us-west-2a)
   • executor_sourcegraph_external_url to the URL configured in your instance's Site configuration
   • executor_sourcegraph_executor_proxy_password to the access token configured in your instance's Site configuration
8. Run terraform init to download the Sourcegraph executor modules.
9. Run terraform plan to preview the changes that will occur to your AWS infrastructure.
10. Run terraform apply and enter "yes" after reviewing the proposed changes to create the executor resources.
    • Ensure terraform apply exited with code 0 and did not print any errors
11. Go back to the site admin page, expand Executors, click Instances, and check to see if your executor shows up in the list with a green dot 🟢

Validation

The following can be done to troubleshoot or double-check that the executor has been properly provisioned.

Ensure the executor instance is running. Either go to EC2 → Instances in the AWS Console or run the following command.

SHELL
$ aws ec2 describe-instances \
    --filters "Name=tag:executor_tag,Values=<your executor_instance_tag>" \
              "Name=instance-state-name,Values=running" \
    --query "Reservations[].Instances[].[InstanceId,InstanceType,State.Name,PrivateIpAddress]" \
    --output table
-------------------------------------------------------------------
|                       DescribeInstances                         |
+----------------------+-------------+---------+------------------+
|  i-0abc123def456789  | c5n.metal   | running |  10.0.1.42       |
+----------------------+-------------+---------+------------------+

You can connect to the instance using AWS Systems Manager Session Manager. The module attaches the AmazonSSMManagedInstanceCore policy to the executor IAM role, so SSM works out of the box.

SHELL
aws ssm start-session --target i-0abc123def456789

Then run the following command to check if the service is running.

SHELL
ubuntu@ip-10-0-1-42:~$ systemctl status executor
● executor.service - User code executor
     Loaded: loaded (/etc/systemd/system/executor.service; enabled; preset: enabled)
     Active: active (running) since Thu 2021-11-18 02:28:48 UTC; 19s ago

To check the logs, you can either query CloudWatch Logs in the AWS Console or run the following command while connected to the instance.

SHELL
ubuntu@ip-10-0-1-42:~$ journalctl -u executor | less

Ensure the EXECUTOR_FRONTEND_URL and EXECUTOR_FRONTEND_PASSWORD in /etc/systemd/system/executor.env are correct.

SHELL
cat /etc/systemd/system/executor.env

Ensure the instance can reach your externalURL:

SHELL
ubuntu@ip-10-0-1-42:~$ curl <your externalURL here>
<a href="/sign-in?returnTo=%2F">Found</a>

Configure Auto-indexing

1. Go to the Site admin page
2. Expand Code graph,
3. Select Configuration
4. Click Create new policy, and fill in:
   • Name: TEST
   • Click add a repository pattern
   • Repository pattern #1: set this to an existing repository on your Sourcegraph instance ( e.g. github.com/gorilla/mux)
   • Type: HEAD
   • Retention: Disabled
   • Auto-indexing: Enabled
5. Expand Code graph
6. Select Auto-indexing, and check to see if an indexing job has appeared. If nothing is there:
   • Try clicking Enqueue
   • Try setting a higher update frequency: PRECISE_CODE_INTEL_AUTO_INDEXING_TASK_INTERVAL=10s
   • Try setting a lower delay: PRECISE_CODE_INTEL_AUTO_INDEXING_REPOSITORY_PROCESS_DELAY=10s
7. Once you have a completed indexing job, click Uploads and check to see that an index has been uploaded.
8. Once the index has been uploaded, you should see the PRECISE badge in the hover! 🎉

Auto-scaling

NOTE: Auto scaling is currently not supported when downloading and running executor binaries yourself, and on managed instances when using self-hosted executors, since it requires deployment adjustments.

Auto-scaling of executor instances can help to increase concurrency of jobs, without paying for unused resources. With auto-scaling, you can scale down to 0 instances when no workload exist and scale up as far as you like and your cloud provider can support. Auto-scaling needs to be configured separately.

Auto-scaling makes use of AutoScalingGroups on AWS. Sourcegraph's worker service publishes a scaling metric (that is, the number of jobs in queue) to CloudWatch. Then, based on that reported value, the auto-scaler adds and removes compute resources to match the required amount of compute. The autoscaler will attempt to hold 1 instance running per each executor_jobs_per_instance_scaling items in queue.

For example, if executor_jobs_per_instance_scaling is set to 20 and the queue size is currently 400, then 20 instances would be determined as required to handle the load. You might want to tweak this number based on the machine_type, maximum_num_jobs (concurrency per machine), and desired processing speed. See the AWS variable definitions for details.

With the Terraform variables executor_min_replicas and executor_max_replicas, you can configure the minimum and maximum number of compute machines to be run at a given time.

For auto-scaling to work, two things must be true:

1. executor_min_replicas must be >= 0 and executor_max_replicas must be > executor_min_replicas.
2. The Sourcegraph instance (its worker service, specifically) needs to publish scaling metrics to CloudWatch.

For the latter to work, the Sourcegraph instance needs to be configured with the correct credentials that allow it to access AWS.

The credentials submodule in the AWS executor module exists for that purpose. When used, the credentials module sets up an IAM user with permission to write CloudWatch metrics and returns the access key in the Terraform outputs.

Here's an example of how one would configure auto-scaling.

TERRAFORM
module "executors" {
  source  = "sourcegraph/executors/aws"
  version = "<version>"

  # Basic configuration...

  # Auto-scaling
  executor_min_replicas              = 0 # Spin down when not in use
  executor_max_replicas              = 30
  executor_jobs_per_instance_scaling = 20
}

module "my-credentials" {
  source  = "sourcegraph/executors/aws//modules/credentials"
  version = "<version>"

  resource_prefix          = "<optional prefix to add to created resources>"
  # permissions_boundary_arn = "<ARN of IAM permissions boundary policy>"
}

output "metric_writer_access_key_id" {
  value = module.my-credentials.metric_writer_access_key_id
}

output "metric_writer_secret_key" {
  value     = module.my-credentials.metric_writer_secret_key
  sensitive = true
}

After running terraform apply, retrieve the credentials by running the following commands.

SHELL
$ terraform output metric_writer_access_key_id
$ terraform output metric_writer_secret_key

Configuring the Sourcegraph instance

The AWS EC2 auto-scaling groups configured by the Sourcegraph Terraform module respond to changes in metric values written to CloudWatch. The target Sourcegraph instance is expected to continuously write these values.

To write the scaling metric to CloudWatch, the worker service must have defined the following environment variables.

Testing auto scaling

Once the environment variables have been set and the worker service has been restarted, you should be able to find the scaling metrics in CloudWatch.

To test if the metric is correctly reported: go to the CloudWatch metrics section. Under All metrics, select the namespace sourcegraph-executor and then the metric environment, queueName. Make sure there are entries returned.

Next, you can test whether the number of executors rises and shrinks as load spikes occur. Keep in mind that auto-scaling is not a real-time operation and usually takes a short moment and can have some delays between the metric going down and the desired machine count adjusting.

Upgrading executors

Upgrading executors is relatively uninvolved. Simply follow the instructions below. Also, check the changelog for any Executors related breaking changes or new features or flags that you might want to configure. See Executors maintenance for version compatibility.

Step 1: Update the source version of the terraform modules

NOTE: Keep in mind that only one minor version bumps are guaranteed to be disruption-free.

DIFF
module "executors" {
  source = "sourcegraph/executors/aws"
 
  # Find the latest version matching your Sourcegraph version here:
  # https://github.com/sourcegraph/terraform-aws-executors/tags
-  version = "7.3.0"
+  version = "7.4.0"
 
  availability_zone = "<availability zone>"
 
  executor_sourcegraph_external_url            = "<external url>"
  executor_sourcegraph_executor_proxy_password = "<shared secret>"
 
  # Either:
  executor_queue_name                          = "<codeintel | batches>"
  # Or:
  executor_queue_names                         = ["codeintel", "batches"]
 
  executor_instance_tag                        = "<tag to filter in monitoring>"
  executor_metrics_environment_label           = "<label to filter custom metrics>"
  executor_use_firecracker                     = true
}

Step 2: Reapply the terraform configuration

Simply reapply the terraform configuration and executors will be ready to go again.

BASH
terraform apply