Skip to content
This repository has been archived by the owner on Nov 3, 2021. It is now read-only.

AWS MediaLive Cloudformation Fn::GetAtt get array value #23

Open
samueleastdev opened this issue Feb 8, 2021 · 7 comments
Open

AWS MediaLive Cloudformation Fn::GetAtt get array value #23

samueleastdev opened this issue Feb 8, 2021 · 7 comments

Comments

@samueleastdev
Copy link

samueleastdev commented Feb 8, 2021

I am building a Cloudformation template for Media Live I have everything working with the AWS PHP SDK. I create a MediaPackage channel and return the data to be used in the next call.

Return example.

(
    [Arn] => arn:aws:mediapackage:us-east-1:000000000000:channels/000000000000
    [Description] => Tests
    [HlsIngest] => Array
        (
            [IngestEndpoints] => Array
                (
                    [0] => Array
                        (
                            [Id] => 000000000000
                            [Password] => 000000000000
                            [Url] => https://000000000000.mediapackage.us-east-1.amazonaws.com/in/v2/1e803c424d2947f58c07d9a6a5ff3d31/000000000000/channel
                            [Username] => 000000000000
                        )

                    [1] => Array
                        (
                            [Id] => 000000000000
                            [Password] => 000000000000
                            [Url] => https://2bcff136c2fbf1e5.mediapackage.us-east-1.amazonaws.com/in/v2/000000000000/10392b8387fd442eae56f29ac2656837/channel
                            [Username] => 000000000000
                        )

                )

        )

    [Id] => sdv
    [Tags] => Array
        (
        )
)

I can then use these values in PHP like this.

$destinationOneUrl      = $createChannel['HlsIngest']['IngestEndpoints'][0]['Url'];
$destinationOnePassword = $createChannel['HlsIngest']['IngestEndpoints'][0]['Password'];
$destinationOneUsername = $createChannel['HlsIngest']['IngestEndpoints'][0]['Username'];

And use these when creating the media live channel.

"Destinations": [
		    {
		      "Id": "destination1",
		      "Settings": [
		        {
		          	"Url": "' . $destinationOneUrl . '",
		          	"Username": "' . $destinationOneUsername . '",
          			"PasswordParam": "' . $streamName . '"
		        },
		      ]
		    }
		  ],

Which works but with Cloudformation you have to use the Fn::GetAtt like this.

"Destinations": [{
                    "Id": "destination1",
                    "Settings": [{
                            "Url": {
                                "Fn::GetAtt": ["MediaPackageChannel", "HlsIngest"]
                            },
                            "Username": {
                                "Fn::GetAtt": ["MediaPackageChannel", "HlsIngest"]
                            },
                            "PasswordParam": {
                                "Fn::GetAtt": ["MediaPackageChannel", "HlsIngest"]
                            },
                        },
                        {
                            "Url": {
                                "Fn::GetAtt": ["MediaPackageChannel", "HlsIngest"]
                            },
                            "Username": {
                                "Fn::GetAtt": ["MediaPackageChannel", "HlsIngest"]
                            },
                            "PasswordParam": {
                                "Fn::GetAtt": ["MediaPackageChannel", "HlsIngest"]
                            },
                        }
                    ]
                }],

You cannot go into the array with Fn::GetAtt is there another function that can be used in combination with the Fn::GetAtt to get the inner values.

Something along these lines

"Fn::GetAtt": ["MediaPackageChannel", "HlsIngest['IngestEndpoints'][0]['Url']"]

Here is my full template its hard to explain but basically I need to attach my medialive channel to the mediapackage channel as I mention I have done with the php code above Cloudformation doesn't make this easy.

You will see the Destinations section line 39 I need to populate the Url, Username, PasswordParam with the returned values from the mediapackage channel creation.

https://gist.github.com/samueleastdev/03b050f937855a81f36bb1e3260aa5d3

I started with this template https://docs.aws.amazon.com/solutions/latest/live-streaming/template.html and I am trying to do it myself they use custom lambda functions within the template so maybe it is not possible without lambda.

Any suggestions or help.

@morjoan
Copy link
Contributor

morjoan commented Feb 9, 2021

Hi,
I'm not sure there is a way to do what you are doing without the use of a Lambda function. However, one thing to consider might be to use the MediaPackageSettings (instead of Settings) for your MediaLive channel's destination (see: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-medialive-channel-outputdestination.html). By doing so, you'll only need to provide the MediaPackage's channel ID and not have to specify the Url, Username, etc. Hope that helps.

@samueleastdev
Copy link
Author

Thanks for the response @morjoan I have tried using the MediaPackageSettings and REF the mediachannel

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-medialive-channel-outputdestination.html

"Destinations": [{
                    "Id": "destination1",
                    "MediaPackageSettings": [{
                        "ChannelId" : "MediaPackageChannel"
                    }]
                }]

But this then give you errors about output groups.

output_groups[0].output_group_settings.destination.channel_id instance.destination additionalProperty "channel_id" exists in instance when not allowed; output_groups[0].output_group_settings.destination.uri instance.destination requires property "uri"; output_groups[0].output_group_settings Object does not match discriminator "hls_group"; output_groups[0].output_group_settings.destination.channel_id instance.destination additionalProperty "channel_id" exists in instance when not allowed; output_groups[0].output_group_settings.destination.uri instance.destination requires property "uri"; output_groups[0].output_group_settings Object does not match discriminator "hls_group" (Service: AWSMediaLive; Status Code: 422; Error Code: UnprocessableEntityException; Request ID: 74a4e4bc-3ffd-4ae2-83bc-62b9b342e261; Proxy: null)

I can get the template to run by inserting dummy settings like this.

"Destinations": [{
                    "Id": "destination1",
                    "Settings": [
                        {
                            "Url": "https://my_destination1/testing",
                            "Username": "user",
                            "PasswordParam": "my_destination1"
                        }
                    ]
                }],

Here is the full template that will complete.

https://gist.github.com/samueleastdev/926a0bfbe7b8af193643ddda2ac9cd73

But obviously it is not connected to the mediapackage channel.

Do you know how to get the full code for this template? https://docs.aws.amazon.com/solutions/latest/live-streaming/template.html to understand it and make updates to it in Lambda etc.

Thanks

@morjoan
Copy link
Contributor

morjoan commented Feb 11, 2021

For MediaPackageSettings, the ChannelId should be the Id of the MediaPackage channel you created -- in your example that is "MediaPackage2021", not MediaPackageChannel. Using REF won't give you the Id but instead the MediaPackage ARN (see https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-mediapackage-channel.html).

At any rate, you'll find the source code for that template here: https://docs.aws.amazon.com/solutions/latest/live-streaming/source-code.html

Good luck!

@samueleastdev
Copy link
Author

samueleastdev commented Feb 11, 2021

Hi @morjoan

Thanks I have tried that before pretty much tried every combination I could possible think of changing the id just gives this error.

output_groups[0].output_group_settings.destination.channel_id instance.destination additionalProperty "channel_id" exists in instance when not allowed; output_groups[0].output_group_settings.destination.uri instance.destination requires property "uri"; output_groups[0].output_group_settings Object does not match discriminator "hls_group" (Service: AWSMediaLive; Status Code: 422; Error Code: UnprocessableEntityException; Request ID: 3911a8f5-4e79-4fdb-b165-998a69253b25; Proxy: null)
"Destinations": [{
                    "Id": "destination1",
                    "MediaPackageSettings": [{
                        "ChannelId" : "MediaPackage2021"
                    }]
                }],

I have tried removing the Id.

"Destinations": [{
                    "MediaPackageSettings": [{
                        "ChannelId" : "MediaPackage2021"
                    }]
                }],

And it says its required.

Destination id can not be empty. (Service: AWSMediaLive; Status Code: 400; Error Code: BadRequestException; Request ID: 6c3bebe6-8bc9-444b-b6e3-94046f11a580; Proxy: null)

Even tho it says it is not required in the documentation.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-medialive-channel-outputdestination.html

Id
The ID for this destination.

Required: No

Type: String

Update requires: No interruption

Thought it maybe related to my EncoderSettings

"Destination": {
                                    "DestinationRefId": "destination1"
                                },

In my hls OutputGroups no luck.

You would think that adding the "ChannelId" : "MediaPackage2021" would be the right way to go unfortunately not.

Thanks

@richardcornelissen
Copy link

@samueleastdev ran into the same error message. For me, the issue wasn't in the Destinations config, but in the EncoderSettings. I've used HlsGroupSettings and HlsOutputSettings in the OutputGroups like this:

{
  "ChannelClass": "SINGLE_PIPELINE",
  "Destinations": [
    {
      "Id": "some-id",
      "MediaPackageSettings": [
        {
          "ChannelId": "media-package-channel-id"
        }
      ]
    }
  ],
  "EncoderSettings": {
    "OutputGroups": [
      {
        "Name": "some-name",
        "OutputGroupSettings": {
          "HlsGroupSettings": {
            ...
          }
        },
        "Outputs": [
          {
            "OutputName": "_some-name",
            "OutputSettings": {
              "HlsOutputSettings": {
                ...
              }
            }
          }
        ]
      }
    ]
  },
  "InputAttachments": [...],
  "InputSpecification": {...},
  "Name": "some-name",
  "RoleArn": "some-role",
  "Tags": {}
}

While I should've used MediaPackageGroupSettings and MediaPackageOutputSettings like so

{
  "ChannelClass": "SINGLE_PIPELINE",
  "Destinations": [
    {
      "Id": "some-id",
      "MediaPackageSettings": [
        {
          "ChannelId": "media-package-channel-id"
        }
      ]
    }
  ],
  "EncoderSettings": {
    "OutputGroups": [
      {
        "Name": "some-name",
        "OutputGroupSettings": {
          "MediaPackageGroupSettings": {
            "Destination": {
                "DestinationRefId": "media-package-channel-id"
            }
        }
        },
        "Outputs": [
          {
            "OutputName": "_some-name",
            "OutputSettings": {
              "MediaPackageOutputSettings": {}
            }
          }
        ]
      }
    ]
  },
  "InputAttachments": [...],
  "InputSpecification": {...},
  "Name": "some-name",
  "RoleArn": "some-role",
  "Tags": {}
}

@morjoan
Copy link
Contributor

morjoan commented Apr 16, 2021

Hi,
In case you're still running into problems, wanted to share a blog I recently published that provides a working simple CFN template using the MediaPackage resource: https://aws.amazon.com/blogs/media/sbpdx-part-2-anatomy-live-streaming-aws-cloudformation-template/

@samueleastdev
Copy link
Author

samueleastdev commented May 13, 2021

Hi @morjoan & @richardcornelissen

Thanks for the extra information I can now get somewhere with the following template I am trying to use RTMP push.

{
    "Resources": {
        "MediaPackageChannel": {
            "Type": "AWS::MediaPackage::Channel",
            "Description": "MediaPackage Channel",
            "Properties": {
                "Id": {
                    "Fn::Sub": "${AWS::StackName}-Channel"
                }
            }
        },
        "MediaPackageEndpoint": {
            "Type": "AWS::MediaPackage::OriginEndpoint",
            "Properties": {
                "Description": "MediaPackage HLS Endpoint",
                "ChannelId": {
                    "Fn::Sub": "${AWS::StackName}-Channel"
                },
                "Id": "HLSEndpoint",
                "HlsPackage": {},
                "Origination": "ALLOW"
            },
            "DependsOn": "MediaPackageChannel"
        },
        "MediaLiveSecurityGroup": {
            "Type": "AWS::MediaLive::InputSecurityGroup",
            "Properties": {
                "WhitelistRules": [{
                  "Cidr" : "0.0.0.0/0"
                }]
            }
        },
        "MediaLiveInput": {
            "Type": "AWS::MediaLive::Input",
            "Properties": {
                "Name": {
                    "Fn::Sub": "${AWS::StackName}-RTMPInput"
                },
                "Destinations": [{
                    "StreamName" : "app/key"
                }],
                "Type": "RTMP_PUSH",
                "InputSecurityGroups": [{
                    "Ref": "MediaLiveSecurityGroup"
                }]
            }
        },
        "SSMParameter": {
            "Type": "AWS::SSM::Parameter",
            "Properties": {
                "Description": "Automatically created by AWS Elemental",
                "Name": "my_destination1",
                "Type": "String",
                "Value": "my_destination1"
            }
        },
        "MediaLiveChannel": {
            "Type": "AWS::MediaLive::Channel",
            "Properties": {
                "Name": {
                    "Fn::Sub": "${AWS::StackName}-Channel"
                },
                "ChannelClass": "SINGLE_PIPELINE",
                "RoleArn": {
                    "Ref": "MediaLiveRoleArn"
                },
                "EncoderSettings": {
                    "AudioDescriptions": [
                        {
                            "Name": "audio_1",
                            "LanguageCodeControl": "FOLLOW_INPUT",
                            "AudioTypeControl": "FOLLOW_INPUT"
                        }
                    ],
                    "OutputGroups": [
                        {
                            "OutputGroupSettings": {
                                "MediaPackageGroupSettings": {
                                    "Destination": {
                                        "DestinationRefId": "destination1"
                                    }
                                }
                            },
                            "Outputs": [
                                {
                                    "AudioDescriptionNames": [
                                        "audio_1"
                                    ],
                                    "CaptionDescriptionNames": [],
                                    "OutputName": "EMPOutput1",
                                    "OutputSettings": {
                                        "MediaPackageOutputSettings": {}
                                    },
                                    "VideoDescriptionName": "video_1"
                                }
                            ]
                        }
                    ],
                    "VideoDescriptions": [
                        {
                            "CodecSettings": {
                                "H264Settings": {
                                    "AdaptiveQuantization": "MEDIUM",
                                    "AfdSignaling": "NONE",
                                    "ColorMetadata": "INSERT",
                                    "EntropyEncoding": "CABAC",
                                    "FlickerAq": "ENABLED",
                                    "ForceFieldPictures": "DISABLED",
                                    "FramerateControl": "SPECIFIED",
                                    "FramerateDenominator": 1,
                                    "FramerateNumerator": 30,
                                    "GopBReference": "DISABLED",
                                    "GopClosedCadence": 1,
                                    "GopNumBFrames": 2,
                                    "GopSize": 90,
                                    "GopSizeUnits": "FRAMES",
                                    "Level": "H264_LEVEL_AUTO",
                                    "LookAheadRateControl": "MEDIUM",
                                    "NumRefFrames": 1,
                                    "ParControl": "SPECIFIED",
                                    "ParDenominator": 1,
                                    "ParNumerator": 1,
                                    "Profile": "MAIN",
                                    "RateControlMode": "CBR",
                                    "ScanType": "PROGRESSIVE",
                                    "SceneChangeDetect": "ENABLED",
                                    "SpatialAq": "ENABLED",
                                    "SubgopLength": "FIXED",
                                    "Syntax": "DEFAULT",
                                    "TemporalAq": "ENABLED",
                                    "TimecodeInsertion": "DISABLED"
                                }
                            },
                            "Height": 720,
                            "Name": "video_1",
                            "RespondToAfd": "NONE",
                            "ScalingBehavior": "DEFAULT",
                            "Sharpness": 50,
                            "Width": 1280
                        }
                    ],
                    "TimecodeConfig": {
                        "Source": "EMBEDDED"
                    }
                },
                "Destinations": [
                    {
                        "Id": "destination1",
                        "MediaPackageSettings": [
                            {
                                "ChannelId": {
                                    "Fn::Sub": "${AWS::StackName}-Channel"
                                }
                            }
                        ]
                    }
                ],
                "InputAttachments": [{
                    "InputAttachmentName": "HLSInput",
                    "InputId": {
                        "Ref": "MediaLiveInput"
                    },
                    "InputSettings": {
                        "NetworkInputSettings": {
                            "ServerValidation": "CHECK_CRYPTOGRAPHY_AND_VALIDATE_NAME"
                        },
                        "SourceEndBehavior": "CONTINUE",
                        "InputFilter": "AUTO",
                        "FilterStrength": 1,
                        "DeblockFilter": "DISABLED",
                        "DenoiseFilter": "DISABLED",
                        "AudioSelectors": [],
                        "CaptionSelectors": [{
                            "SelectorSettings": {
                                "EmbeddedSourceSettings": {
                                    "Source608ChannelNumber": 1,
                                    "Source608TrackNumber": 1,
                                    "Convert608To708": "DISABLED",
                                    "Scte20Detection": "OFF"
                                }
                            },
                            "Name": "EmbeddedSelector"
                        }]
                    } 
                }]
            }
        }
    },
    "Parameters": {
        "MediaLiveRoleArn": {
            "Description": "ARN of the IAM role that MediaLive is allowed to assume in order to perform the operations on the resources specified in the policies.",
            "Type": "String"
        }
    },
    "Outputs": {
        "LiveStreamUrl": {
            "Description": "LiveStream Playback URL",
            "Value": {
                "Fn::Sub": "${MediaPackageEndpoint.Url}"
            }
        }
    }
}

It completes but there seems to be errors pushing to the MediaPackage channel hls endpoint that I cannot figure out.

I get the following errors.

OutputDataBackground failed to send file for URL [mediapackage://lopsss-Channel/channel_EMPOutput1_20210513T123222_00257.ts], after [3] attempts, error [failed to write output to mediapackage]

Failed to Create Output File or Socket

Failed to Write to Output

Failed to Write to Output

Failed to Write to Output

So it must be some permissions issue.

When using OBS software it makes the connection to the input.

Server: rtmp://54.000.000.243:1935/app
Stream Key: key

Any suggestions on where I might be going wrong?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants