[Bug]: reading AWS CloudTrail Trail (XXXXXX-cloudtrail): not found after creation

[killmepete]

Terraform Core Version

1.5.6

AWS Provider Version

aws v5.0.1

Affected Resource(s)

Cloudtrail

Expected Behavior

The cloudtrail module I have created (based of the example) should be able to deploy without issue.

Actual Behavior

I currently have a simple module created to deploy cloudtrail (using the example provided), I'm running into an error which seems to be intermittent where after running a few deploys (with resources nothing to do with Cloudtrail) I'll encounter the error message:

"reading AWS CloudTrail Trail (dev-test-cloudtrail): not found after creation"

I can confirm that if I log into the console or check via the CLI that the trail does exist and is logging as expected, I haven't been able to find any documentation around this error message and the only way I've managed to get around it is by intercepting my build, running a terraform state rm on cloudtrail and redeploying.

I'm inclined to believe this is a bug with the provider, I've deployed and written similar cloudtrail modules before and I've never encountered this problem. If I'm mistaken and it's an easy fix that would make me happy!

Relevant Error/Panic Output Snippet

Error: reading AWS CloudTrail Trail (dev-test-cloudtrail): not found after creation
│ 
│   with module.cloudtrail.aws_cloudtrail.cloudtrail,
│   on ../modules/cloudtrail/main.tf line 1, in resource "aws_cloudtrail" "cloudtrail":
│    1: resource "aws_cloudtrail" "cloudtrail" {

Terraform Configuration Files

resource "aws_cloudtrail" "example" {
  name                          = "example"
  s3_bucket_name                = aws_s3_bucket.example.id
  s3_key_prefix                 = "prefix"
  include_global_service_events = false
}

resource "aws_s3_bucket" "example" {
  bucket        = "tf-test-trail"
  force_destroy = true
}

data "aws_iam_policy_document" "example" {
  statement {
    sid    = "AWSCloudTrailAclCheck"
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }

    actions   = ["s3:GetBucketAcl"]
    resources = [aws_s3_bucket.example.arn]
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = ["arn:${data.aws_partition.current.partition}:cloudtrail:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:trail/example"]
    }
  }

  statement {
    sid    = "AWSCloudTrailWrite"
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }

    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.example.arn}/prefix/AWSLogs/${data.aws_caller_identity.current.account_id}/*"]

    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"
      values   = ["bucket-owner-full-control"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = ["arn:${data.aws_partition.current.partition}:cloudtrail:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:trail/example"]
    }
  }
}
resource "aws_s3_bucket_policy" "example" {
  bucket = aws_s3_bucket.example.id
  policy = data.aws_iam_policy_document.example.json
}

data "aws_caller_identity" "current" {}

data "aws_partition" "current" {}

data "aws_region" "current" {}

Steps to Reproduce

Unsure, problem happens intermittently and can happen a long time after the cloudtrail has been deployed.

Debug Output

ERROR

│ Error: reading AWS CloudTrail Trail (test-dev-cloudtrail): not found after creation
│ 
│   with module.cloudtrail.aws_cloudtrail.cloudtrail,
│   on ../modules/cloudtrail/main.tf line 1, in resource "aws_cloudtrail" "cloudtrail":
│    1: resource "aws_cloudtrail" "cloudtrail" {
│ 
╵
ERRO[0020] Terraform invocation failed in /root/app/production 
ERRO[0020] 1 error occurred:
        * [/root/app/production] exit status 1
 

Exited with code exit status 1

PLANNED CHANGES

  # module.guardduty.aws_cloudwatch_event_target.guardduty_event_target will be created
  + resource "aws_cloudwatch_event_target" "guardduty_event_target" {
      + arn            = "arn:aws:lambda:eu-west-2:xxxxxxxxxxxxxxx:function:datadog"
      + event_bus_name = "default"
      + id             = (known after apply)
      + rule           = "guardduty_event_rule"
      + target_id      = "guardduty_event_rule_target"
    }

  # module.guardduty.aws_guardduty_publishing_destination.gd_publishing_destination will be created
  + resource "aws_guardduty_publishing_destination" "gd_publishing_destination" {
      + destination_arn  = "arn:aws:s3:::xxxxxxxxxxxx-guardduty-findings"
      + destination_type = "S3"
      + detector_id      = "xxxxxxxxx"
      + id               = (known after apply)
      + kms_key_arn      = "arn:aws:kms:eu-west-2:xxxxxxxxxx:key/xxxxxxxxxxxxxxxxxxx"
    }

  # module.guardduty.aws_kms_key.gd_encryption_key will be updated in-place
  ~ resource "aws_kms_key" "gd_encryption_key" {
        id                                 = "xxxxxxxxxx"
      ~ policy                             = jsonencode(
          ~ {
              - Id        = "key-default-1"
              ~ Statement = [
                  ~ {
                      ~ Action    = "kms:*" -> "kms:GenerateDataKey"
                      ~ Principal = {
                          - AWS     = "arn:aws:iam::xxxxxxxxx:root"
                          + Service = "guardduty.amazonaws.com"
                        }
                      ~ Resource  = "*" -> "arn:aws:kms:eu-west-2:xxxxxxxxxxxx:key/*"
                      ~ Sid       = "Enable IAM User Permissions" -> "Allow GuardDuty to encrypt findings"
                        # (1 unchanged attribute hidden)
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        tags                               = {}
        # (11 unchanged attributes hidden)
    }

Plan: 2 to add, 1 to change, 0 to destroy.

Panic Output

No response

Important Factoids

Resource names, IDs and so and so have been changed for the purposes of the bug report.

References

The lines responsible for the logging message...

terraform-provider-aws/internal/service/cloudtrail/cloudtrail.go

Lines 366 to 385 in 8aebcb6

	// CloudTrail does not return a NotFound error in the event that the Trail
	// you're looking for is not found. Instead, it's simply not in the list.
	var trail *cloudtrail.Trail
	for _, c := range resp.TrailList {
	if d.Id() == aws.StringValue(c.Name) {
	trail = c
	}
	}
	if !d.IsNewResource() && trail == nil {
	create.LogNotFoundRemoveState(names.CloudTrail, create.ErrActionReading, ResNameTrail, d.Id())
	d.SetId("")
	return diags
	}
	if d.IsNewResource() && trail == nil {
	return create.DiagError(names.CloudTrail, create.ErrActionReading, ResNameTrail, d.Id(), errors.New("not found after creation"))
	}
	log.Printf("[DEBUG] CloudTrail received: %s", trail)

Would you like to implement a fix?

Happy too if it's a confirmed bug and can get some pointers!

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[killmepete]

Upgraded to the latest version of the provider (5.14.0) and still seeing the same problem.