Repository navigation
write logs to NON-AWS S3 object storage #13961
Answered
by
ashb
tomrwillis
asked this question in
Q&A
|
My organization has an internally hosted S3 object storage that is compatible with AWS. I can access it with the botocore libraries outside of Airflow. However, I am unable to configure Airflow to write logs to it. The Airflow S3_hook appears to hardcode the endpoint URL to the amazon location and I cannot seem to override it. So it looks for my S3 bucket out there in AWS instead of inside my corporate network. Has anyone else tried to write logs to S3 storage that was NOT AWS? |
Answered by
ashb
Feb 5, 2021
Replies: 1 comment
|
It's not well documented, but it is possible if you put a https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html { "host": "https://my-s3-alike.internal/" }If that works a Doc PR to make this more obvious would be great! |
0 replies
Answer selected by
XD-DENG
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It's not well documented, but it is possible if you put a
hostextra in the connectionhttps://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html
{ "host": "https://my-s3-alike.internal/" }If that works a Doc PR to make this more obvious would be great!