Unable to get Functionbeat to deploy successfully

[kaiyan-sheng]

Functionbeat failed to run with this error below:

2021-10-05T10:48:06.569-0400 DEBUG [aws] aws/cli_manager.go:130 Deploy finish for function 'cloudwatch'
Function: cloudwatch, could not deploy, **error:** bucket 'test' already exist and you don't have permission to access it: unknown endpoint, could not resolve endpoint, partition: "aws", service: "s3", region: "", known: [ap-northeast-2 eu-central-1 me-south-1 ap-southeast-2 us-west-1 af-south-1 ap-east-1 us-east-2 ap-southeast-1 us-east-1 us-west-2 ap-south-1 ca-central-1 eu-north-1 eu-south-1 eu-west-1 s3-external-1 sa-east-1 eu-west-2 eu-west-3 ap-northeast-1 aws-global]
2021-10-05T10:48:06.569-0400 DEBUG [cli-handler] cmd/cli_handler.go:62 Deploy execution ended
Fail to deploy 1 function(s)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[eXigentCoder]

Having the same issue, have checked and can definitely access S3 with the credentials I'm using.

I tried setting functionbeat.provider.aws.endpoint to a specific s3 bucket endpoint from here but no joy, the error message still has region as blank.

Same if I pass the exported environment variable using the --E setting=value arg, i.e. --E AWS_DEFAULT_REGION=<region>

[eXigentCoder]

@jlind23 I get a 404 when I try go to that URL, perhaps that repo is private?

[jlind23]

Sorry my bad:

`It is a known issue. We have to investigate the problem and release a fix in the future.

The workaround is to deploy the Lambda using the AWS console not with the manager. They should follow the first 2 steps of this guide: https://www.elastic.co/guide/en/beats/functionbeat/current/deploy-to-cloud-provider.html#own-deployment`

[jlind23]

@kvch feel free to jump if i am saying something wrong.

[yakeeliuliu]

Hi, running into another issue when deploying with cloudfomation:

{
  "errorMessage": "fork/exec /var/task/functionbeat-aws: no such file or directory",
  "errorType": "PathError"
}

My template:

...
    "fnbcloudwatch": {
      "Properties": {
        "Code": {
          "S3Bucket": "logging2elk",
          "S3Key": "functionbeat.zip"
        },
        "Description": "lambda function for cloudwatch logs",
        "Environment": {
          "Variables": {
            "BEAT_STRICT_PERMS": "false",
            "ENABLED_FUNCTIONS": "cloudwatch"
          }
        },
        "FunctionName": "cloudwatch",
        "Handler": "functionbeat-aws",
        "MemorySize": 128,
        "ReservedConcurrentExecutions": 5,
        "Role": "arn:aws:iam::096481171559:role/logging-functionbeat",
        "Runtime": "go1.x",
        "Timeout": 3
      },
      "Type": "AWS::Lambda::Function"
    },
..

[jlind23]

@yakeeliuliu How is your lambda configured on AWS? Could you provide me your settings?

• Could you please share the content of the zip file you uploaded?
• Is the template coming from functionbeat export template output?

[yakeeliuliu]

Hi, fixed the issue, but now have another one:

I can see I have errors :

Lambda Errors
--
FunctionName: | cloudwatch
Region: | ca-central-1
Period: | 1 Minute
Statistic: | Sum
Unit: | Count
Min: | 1
Max: | 2
Average: | 1.4
Sum: | 7
Last value: | 1
Last time: | 2021-10-29 11:53:00 UTC

but no logs under /aws/lambda/cloudwatch

Here is my template:

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "fnbcloudwatch": {
      "Properties": {
        "Code": {
          "S3Bucket": "logging2elk",
          "S3Key": "functionbeat-deployment/cloudwatch/jTR9lDaAG14YU1iqP7nLRp0bQynRW-Zffo1zP8HBI5U/functionbeat.zip"
        },
        "Description": "lambda function for cloudwatch logs",
        "Environment": {
          "Variables": {
            "BEAT_STRICT_PERMS": "false",
            "ENABLED_FUNCTIONS": "cloudwatch"
          }
        },
        "FunctionName": "cloudwatch",
        "Handler": "functionbeat-aws",
        "MemorySize": 128,
        "ReservedConcurrentExecutions": 5,
        "Role": "arn:aws:iam::096481171559:role/logging-functionbeat",
        "Runtime": "go1.x",
        "Timeout": 3
      },
      "Type": "AWS::Lambda::Function"
    },
    "fnbcloudwatchLogGroup": {
      "Properties": {
        "LogGroupName": "/aws/lambda/cloudwatch"
      },
      "Type": "AWS::Logs::LogGroup"
    },
    "fnbcloudwatchPermission0": {
      "Properties": {
        "Action": "lambda:InvokeFunction",
        "FunctionName": {
          "Fn::GetAtt": [
            "fnbcloudwatch",
            "Arn"
          ]
        },
        "Principal": {
          "Fn::Join": [
            "",
            [
              "logs.",
              {
                "Ref": "AWS::Region"
              },
              ".",
              {
                "Ref": "AWS::URLSuffix"
              }
            ]
          ]
        },
        "SourceArn": {
          "Fn::Join": [
            "",
            [
              "arn:",
              {
                "Ref": "AWS::Partition"
              },
              ":logs:",
              {
                "Ref": "AWS::Region"
              },
              ":",
              {
                "Ref": "AWS::AccountId"
              },
              ":log-group:",
              "/aws/lambda/email-prod-email",
              ":*"
            ]
          ]
        }
      },
      "Type": "AWS::Lambda::Permission"
    },
    "fnbcloudwatchSFawslambdaemailprodemail": {
      "Properties": {
        "DestinationArn": {
          "Fn::GetAtt": [
            "fnbcloudwatch",
            "Arn"
          ]
        },
        "FilterPattern": "mylog_",
        "LogGroupName": "/aws/lambda/email-prod-email"
      },
      "Type": "AWS::Logs::SubscriptionFilter"
    }
  }
}

When I test the lambda function with the "Hello World" template, it got this error:

{
  "errorMessage": "2021-10-29T12:04:05.850Z 7fa26953-5bc8-4aac-aa15-1af79fa63f91 Task timed out after 3.00 seconds"
}

[jlind23]

For your "Lambda Errors" don't you have something more explicit than this?

[yakeeliuliu]

I don't see any explicit logs from /aws/lambda/cloudwatch, it's empty there

[jlind23]

Could you run it locally or at least use debug mode? https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-debugging-additional-arguments.html

[yakeeliuliu]

Hi, I got the beat error logs now:

2021-11-01T16:33:08.150-04:00 | EOF: errorString null
2021-11-01T16:33:08.150-04:00 | 2021-11-01T20:33:08.150Z ERROR [cloudwatch_logs] aws/cloudwatch_logs.go:122 Could not parse events from cloudwatch logs, error: EOF
2021-11-01T16:33:08.151-04:00 | END RequestId: b2a376e4-3c99-40ca-aced-c16ed0a7d4d0
2021-11-01T16:33:08.151-04:00 | REPORT RequestId: b2a376e4-3c99-40ca-aced-c16ed0a7d4d0 Duration: 16.63 ms Billed Duration: 17 ms Memory Size: 128 MB Max Memory Used: 125 MB
2021-11-01T16:35:00.462-04:00 | START RequestId: 48d3049e-ba2f-4a9d-ab23-234a24eb077d Version: $LATEST

[jlind23]

Could you please put logging.level: debug in functionbeat.yml for detailed logs?

[Jeff-Klein]

Facing the same issue. Any updates on this?

[jlind23]

Hi @Jeff-Klein unfortunately we didn't have time to work on this so far.

[farrp]

I have run into the same issue. Can you please provide more guidance on how to deploy this manually? I am not an AWS export, nor do I want to be. You say "using the AWS console"... which console do I use? What files do I upload? What steps do I need to follow?

[yakeeliuliu]

Hi, after uploading the cloud formation template successfully, I fixed my issue by replacing the cloudwatch lambda "Execution role" policy created by funtionbeat with this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:*"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::*"
        }
    ]
}

hoe this would help.**

[kaykhancheckpoint]

I came across this same issue when migrating to version 7.16.* i had to roll back the functionbeat to 7.12.* until this issue is resolved.

[pushred]

I was able to workaround this issue by defining another config value that seems to be undocumented in both the docs and the reference config file.

functionbeat.provider.aws.region: us-east-1

Presumably it should be inferred from the endpoint but this works for now.