DynamoDB migration errors

[lukehoban]

I can't figure out what's going on here. If I deploy an example that uses DynamoDB through the containerized Ubuntu image I see lumi deploy failures inside the Terraform bridge. But I can't reproduce these locally. I believe versions of everything are the same - but very well could be that something is different (currently hard to create a stable environment with the complex sets of static and dynamic dependencies).

Here's the error I get (some of this may be out-of-order due to how the deployment service currently prints stdout/stderr):

debug: plugin: plugin address: unix /tmp/plugin240061921
debug: No assume_role block read from configuration
debug: Building AWS region structure
debug: Building AWS auth structure
debug: Setting AWS metadata API timeout to 100ms
debug: Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
debug: AWS Auth provider used: "SharedCredentialsProvider"
debug: Initializing DeviceFarm SDK connection
debug: Trying to get account ID via iam:GetUser
Applying step #1 [create]
+ aws:dynamodb/table:Table:
      [urn=urn:lumi:pulumi::todo:index::aws:dynamodb/table:Table::urls]
      attribute           : [
          [0]: {
                   name: "name"
                   type: "S"
               }
      ]
      hashKey             : "name"
      name                : "urls"
      readCapacity        : 5
      writeCapacity       : 5
debug: DynamoDB table create: urls-8ec8d8e9d8e114d2998bc3a1d6f8823b6316ac90
debug: Sleeping for 5 seconds for table to become active
debug: Sleeping for 5 seconds for table to become active
debug: Setting DynamoDB TimeToLive on arn: arn:aws:dynamodb:eu-west-1:153052954103:table/urls-8ec8d8e9d8e114d2998bc3a1d6f8823b6316ac90
debug: Loading data for DynamoDB table 'urls-8ec8d8e9d8e114d2998bc3a1d6f8823b6316ac90'
debug: Added Attribute: name
debug: Loaded TimeToLive data for DynamoDB table 'urls-8ec8d8e9d8e114d2998bc3a1d6f8823b6316ac90'
info: Found AWS DynamoDB Table State v0; migrating to v1
debug: DynamoDB Table Attributes before Migration: map[string]string(nil)
debug: panic: assignment to entry in nil map
debug:
debug: goroutine 153 [running]:
debug: github.com/terraform-providers/terraform-provider-aws/aws.migrateDynamoDBStateV0toV1(0xc420531270, 0x1, 0x1, 0xc4201dccf0)
debug: 	/go/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_dynamodb_table_migrate.go:64 +0x526
debug: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsDynamoDbTableMigrateState(0x0, 0xc420531270, 0x1d9eb80, 0xc4201fb200, 0x33faf00, 0x0, 0x18)
debug: 	/go/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_dynamodb_table_migrate.go:17 +0xb8
Step #1 failed [create]: this failure was catastrophic and the provider cannot guarantee recovery
debug: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc420302660, 0xc420531270, 0x1d9eb80, 0xc4201fb200, 0xc4203cb818, 0xc42064d401, 0x0)
info: no changes were made
debug: 	/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:308 +0x347
A catastrophic error occurred; resources states may be unknown
Error: topic.ts(22,25): warning LUMI504: Type 'T' could not be found: unrecognized primitive type name
topic.ts(22,25): warning LUMI504: Type 'T' could not be found: unrecognized primitive type name
error LUMI2003: Plan apply failed: rpc error: code = Unknown desc = Error reading aws:dynamodb/table:Table's state: unexpected EOF
error: rpc error: code = Unknown desc = Error reading aws:dynamodb/table:Table's state: unexpected EOF

Note the attempt to migrate state from v0 to v1, then the failure to read the state (which appears to have been passed as nil).

Any ideas what might be going on here?

For reference, here's current Dockerfile contents to try to create a deployment environment:

RUN go get -d -v github.com/terraform-providers/terraform-provider-aws
RUN go install github.com/terraform-providers/terraform-provider-aws

RUN mkdir -p /go/src/github.com/pulumi
COPY . /go/src/github.com/pulumi

# Build and install deployment binary
RUN go get -d -v github.com/pulumi/pulumi/deployment 
RUN go install github.com/pulumi/pulumi/deployment

# Build and install Lumi binaries (`lumi` and `lumijs`)
RUN go get -d -v github.com/pulumi/lumi/cmd/lumi 
RUN go install github.com/pulumi/lumi/cmd/lumi
RUN cd /go/src/github.com/pulumi/lumi/cmd/lumijs \
  && yarn run build \
  && ln -s /go/src/github.com/pulumi/lumi/cmd/lumijs/lumijs /usr/local/bin/lumijs

# Build and install Pulumi libraries (`lumirt`, `lumi`, `lumijs`, `aws`, `platform`)
RUN mkdir -p /usr/local/lumi/packs
RUN cd /go/src/github.com/pulumi/lumi/lib/lumirt \
  && rm -rf node_modules \
  && lumijs \
  && yarn link \
  && cp -RL ./.lumi/bin/ /usr/local/lumi/packs/lumirt
RUN cd /go/src/github.com/pulumi/lumi/lib/lumi \
  && rm -rf node_modules \
  && lumijs \
  && yarn link \
  && cp -RL ./.lumi/bin/ /usr/local/lumi/packs/lumi
RUN cd /go/src/github.com/pulumi/lumi/lib/lumijs \
  && rm -rf node_modules \
  && yarn link @lumi/lumirt \
  && lumijs \
  && yarn link \
  && cp -RL ./.lumi/bin/ /usr/local/lumi/packs/lumijs
RUN cd /go/src/github.com/pulumi/aws/pack \
  && rm -rf node_modules \
  && yarn link @lumi/lumirt @lumi/lumijs @lumi/lumi \
  && lumijs \
  && mkdir -p /usr/local/lumi/packs/aws \
  && cp VERSION /usr/local/lumi/packs/aws \
  && cp package.json /usr/local/lumi/packs/aws \
  && cp -RL .lumi/bin/* /usr/local/lumi/packs/aws \
  && cp -RL node_modules /usr/local/lumi/packs/aws \
  && cd .. \
  && go install github.com/pulumi/aws/cmd/lumi-tfbridge-aws \
  && cp /go/bin/lumi-tfbridge-aws /usr/local/lumi/packs/aws/lumi-resource-aws \
  && cd /usr/local/lumi/packs/aws \
  && yarn link
RUN cd /go/src/github.com/pulumi/aws-serverless/ \
  && rm -rf node_modules \
  && yarn link @lumi/lumirt @lumi/lumi @lumi/aws \
  && lumijs \
  && yarn link \
  && cp -RL ./.lumi/bin/ /usr/local/lumi/packs/aws-serverless
RUN cd /go/src/github.com/pulumi/lumi-platform/ \
  && rm -rf node_modules \
  && yarn link @lumi/lumirt @lumi/lumi @lumi/aws @lumi/aws-serverless \
  && lumijs \
  && yarn link \
  && cp -RL ./.lumi/bin/ /usr/local/lumi/packs/platform
WORKDIR /src

[lukehoban]

And another failure even just on a RolePolicyAttachment.

Applying step #3 [create]
+ aws:iam/rolePolicyAttachment:RolePolicyAttachment:
      [urn=urn:lumi:pulumi::todo:index::aws:iam/rolePolicyAttachment:RolePolicyAttachment::shortener3454b7045a88364b5cecad7c4d8bbef4513062e4-iampolicy-0]
      name     : "shortener3454b7045a88364b5cecad7c4d8bbef4513062e4-iampolicy-0"
      policyArn: "arn:aws:iam::aws:policy/AWSLambdaFullAccess"
      role     : "shortener3454b7045a88364b5cecad7c4d8bbef4513062e4-iamrole-9ff9c3"
debug: Instance Diff is nil in Diff()
Step #3 failed [create]: this failure was catastrophic and the provider cannot guarantee recovery
info: 2 deployed changes:
    + 2 resources created
Deployment duration: 8.160764173s
A catastrophic error occurred; resources states may be unknown
Error: topic.ts(22,25): warning LUMI504: Type 'T' could not be found: unrecognized primitive type name
topic.ts(22,25): warning LUMI504: Type 'T' could not be found: unrecognized primitive type name
error LUMI2003: Plan apply failed: rpc error: code = Unknown desc = Error reading aws:iam/rolePolicyAttachment:RolePolicyAttachment's state: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, GetRoleInput.RoleName.

error: rpc error: code = Unknown desc = Error reading aws:iam/rolePolicyAttachment:RolePolicyAttachment's state: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, GetRoleInput.RoleName.

[joeduffy]

I wonder if there is a version mismatch. Can you verify that the lumi and terraform-bridge revisions match the latest?

[lukehoban]

They should be the same sources as what's on my dev machine. I'm not synced to latest of everything, but I was in a working state across all the repos locally, and the same pulumi sources were failing inside the deployment container image. I will try brining everything up to date locally (I need to do that anyway), but it's not clear why these errors could be related to out-of-date lumi or terraform-bridge components.

Any tips on debugging these sorts of issues? Does the combination of the DynamoDB migration failure and the GetRoleInput failure suggest any root cause to you?

[joeduffy]

I've definitely had lots of problems getting versions to line up myself when bringing up the Travis CI work, primarily due to dep resolving to different versions.

The GetRoleInput error suggests that the GroupName field is missing on the call to Get, which should only happen if the role name wasn't mapped properly. This could be a missing AutoNameInfo for "role_name" -- I can't remember offhand but can take a look a little later -- however there have definitely been lots of bug fixes that mean the name changes may or may not have been working at a particular commit back in time.

[lukehoban]

Okay - I think I've gotten this working now by bringing everything up to date. I'm still unclear on what the root cause here was - but I'm no longer blocked for now.