Operation not permitted when executing an arbitrary binary

user_2564199 0
Viewed 651

In Running arbitrary binary, AWS explains:

Including your own executables is easy; just package them in the ZIP file you upload, and then reference them (including the relative path within the ZIP file you created) when you call them from Node.js or from other processes that you’ve previously started. Ensure that you include the following at the start of your function code: process.env[‘PATH’] = process.env[‘PATH’] + ‘:’ + process.env[‘LAMBDA_TASK_ROOT’] You can use all the usual forms of interprocess communication as well as files in /tmp to communicate with any of the processes you create.

I would like to use the Go Terraform library tfexec in my code, but I consistently get Permission denied.

Code in main/tf:

package main

import (
    "context"
    "fmt"
    "os"
    "path/filepath"
    "runtime"
    "strings"

    "github.com/aws/aws-lambda-go/lambda"
    "github.com/hashicorp/terraform-exec/tfexec"
)

func Start() error {
    fmt.Printf("Go version: %s\n", runtime.Version())
    fmt.Println()

    // set environment variable, cf https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/
    newPath := []string{
        os.Getenv("PATH"),
        ":",
        os.Getenv("LAMBDA_TASK_ROOT"),
    }
    os.Setenv("PATH", strings.Join(newPath, ""))

    // Get path to binary
    fmt.Println("Start terraform")
    currDir, err := os.Getwd()
    if err != nil {
        return err
    }
    tfBinary := filepath.Join(currDir, "terraform")

    stat, err := os.Stat(tfBinary)
    if err != nil {
        return err
    }
    fmt.Printf("Stat %s: ", tfBinary)
    fmt.Println(stat.Mode())

    // Start new instance of terraform
    tf, err := tfexec.NewTerraform(currDir, tfBinary)
    if err != nil {
        return err
    }

    tf.SetStdout(os.Stdout)
    tf.SetStderr(os.Stderr)

    // run terraform init
    fmt.Println("Tf init")
    if err = tf.Init(context.Background()); err != nil {
        return err
    }

    return nil
}

func main() {
    // Start()
    lambda.Start(Start)
}

Setup:

curl -X GET -o terraform https://releases.hashicorp.com/terraform/0.12.29/terraform_0.12.29_linux_amd64.zip
chmod 755 terraform
GOOS=linux GOARCH=amd64 go build create-vpc-tf.go
zip zip.zip terraform main
# Create Lambda running on go1.x with `main` as handler

Result:

START RequestId: f23e4122-3595-48ef-808f-ef7951531f59 Version: $LATEST
Go version: go1.16

Start terraform
Stat /var/task/terraform: -rwxr-xr-x
Tf init
fork/exec /var/task/terraform: operation not permitted
fork/exec /var/task/terraform: operation not permitted: PathError
null
END RequestId: f23e4122-3595-48ef-808f-ef7951531f59
REPORT RequestId: f23e4122-3595-48ef-808f-ef7951531f59  Duration: 2.26 ms   Billed Duration: 3 ms   Memory Size: 512 MB Max Memory Used: 32 MB  Init Duration: 83.85 ms

This works as expected locally (I have not provided a *.tf file so terraform correctly reports Terraform initialized in an empty directory!.

aws-lambda terraform go amazon-web-services
2 Answers

This issue relates to the tfexec library and its operability within an AWS lambda / Firecracker environment. It is not allowed to fork a process in a Firecracker environment.

Pdeathsig forces a fork, therefore it can't be used in AWS lambda.

Terraform itself doesn't use it, but tfexec does, here.

A quick Google shows that other projects ran into the same issue, example here.

We packaged the fix shown here in order to prevent Pdeathsig from being set. This is making a change to the tfexec library directly, which hopefully they will absorb upstream. Meanwhile, we packaged tfexec in our vendor directory directly with go mod and we made the change there directly. Works nicely.

go mod init [repo]
go mod download
go mod vendor

Update vendor/github.com/hashicorp/terraform-exec/tfexec/cmd_linux.go as follows:


if _, ok := os.LookupEnv("LAMBDA_TASK_ROOT"); !ok {
    cmd.SysProcAttr = &syscall.SysProcAttr{
        // kill children if parent is dead
        Pdeathsig: syscall.SIGKILL,
        // set process group ID
        Setpgid: true,
    }
}

Rebuild your project: go1.16 build main.go.

Re-package and upload to your lambda.

user_25641990

You've downloaded the Terraform binary as a zip file, so it isn't in a form that can be executed.

You'll need to decompress it before repackaging it with zip.

Replace:

curl -X GET -o terraform https://releases.hashicorp.com/terraform/0.12.29/terraform_0.12.29_linux_amd64.zip

With:

curl https://releases.hashicorp.com/terraform/0.12.29/terraform_0.12.29_linux_amd64.zip |zcat >terraform

When you run this locally, you should see something like:

Your version of Terraform is out of date!

Since the version used in the question is ... out of date.

If you aren't seeing a message to that effect, it's a strong clue that your local test isn't actually using the file you downloaded in the setup shown.

A safer way to download the Terraform binary

Alternatively, you can use tfinstall to download and implicitly decompress the latest version at runtime, as shown in this example.

execPath, err := tfinstall.Find(context.Background(), tfinstall.LatestVersion(tmpDir, false))
if err != nil {
    log.Fatalf("error locating Terraform binary: %s", err)
}

This approach uses Hashicorp's custom downloader go-getter, which simultaneously verifies the checksum and performs streaming decompression. tfinstall even uses Hashicorp's public key to verify that the checksum file itself hasn't been tampered with.

If you want to avoid downloading terraform every time you run, you could just use tfinstall locally, and capture the result for reuse (essentially just replacing the curl in you setup).

nobar_869670
Related