AWS SSO legacy profile doesn't work with ansible

[marcoieni]

When running aws sso login --profile legacy the apply script doesn't work.

For example, ./apply staging dev-desktop -u marcoieni fails with the following error:

fatal: [dev-desktop-staging.infra.rust-lang.org]: FAILED! => {"msg": "The conditional check 'datadog_api_key is not defined and datadog_manage_config' failed.
The error was: An unhandled exception occurred while templating '{{ vars_datadog_api_key }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ ssm_all['datadog-api-key'] }}'.
Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ lookup('aws_ssm', '/staging/ansible/all/', region='us-west-1', shortnames=true, bypath=true, recursive=true) }}'.
Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: Failed to access SSM parameter path /staging/ansible/all/ (AccessDenied)\n\nThe error appears to be in '~/.ansible/roles/datadog.datadog/tasks/main.yml': line 17, column 3,
but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Fail if API key is missing\n  ^ here\n"}

By logging in with the aws-creds.py script, the apply script works as expected.

It would be nice to be able to use sso for everything and delete the aws-creds.py from the repo.

[Carbonhell]

Hello! I was looking for a way to start contributing to the infra-team repositories and I took a shot at investigating this issue.
I've never used Ansible before, but I believe the issue is due to the aws_ssm lookup plugin using the default AWS CLI profile unless one is specified through the profile argument (see here). If this assumption is correct, I believe you should be able to use the apply script directly without logging in with aws-creds.py by setting the AWS_PROFILE env var like this: AWS_PROFILE=legacy ./apply staging dev-desktop -u marcoieni. The env vars are propagated to Ansible in the apply script (here), so that should work. I did a quick test with Ansible and AWS SSO to be sure, and I had no issue reading a SSM parameter with a non-default profile.

Using the aws-creds.py works because the AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY take priority if set (see here).

If the usage of AWS_PROFILE solves the issue, to simplify its usage I can open a PR to allow passing a specific AWS CLI profile to the apply script, e.g., ./apply staging dev-desktop -u marcoieni -p legacy. Implementation-wise, I'd simply set up the AWS_PROFILE var in the copied env after those lines. Let me know if that's an acceptable solution :)

Regarding the deletion of aws-creds.py, I believe aws-rollback.py and promote-release.py also assume the default profile has the required permission, or that you have the AWS_ACCESS env vars set up through aws-creds.py. I can add the same parameter to those scripts too. Terragrunt, on the other hand, should not be affected by this based on this section in the README.