potiuk / airflow

Apache Airflow

Independent Open-Source Contributor and Advisor

Committer and PMC Member of Apache Airflow

Member of The Apache Software Foundation

Member of the Security Commitee of the Apache Software Foundation

About me:

• 💡 The best decision ever - quit being CTO, go back to an engineering role
• 🌱 I’m currently learning to poke people more or less gently into doing great stuff
• 😄 Pronouns: He/Him
• 🆓 Affiliations: Member of the Apache Software Foundation
• 🎤 Fun fact: Sang in a choir for almost 30 years

You can sponsor my open-source work via Github Sponsors

airflow's Issues

Migrate Docker example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Docker provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• docker/example_dags/example_docker_swarm.py
• docker/example_dags/example_docker.py
• docker/example_dags/example_docker_copy_data.py

Migrate Amazon system tests to new design

There is a new design of system tests that was introduced by the AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all system tests related to Amazon provider.

It is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:

• tests/providers/amazon/aws/operators/test_ecs_system.py
• tests/providers/amazon/aws/operators/test_eks_system.py
• tests/providers/amazon/aws/operators/test_glacier_system.py
• tests/providers/amazon/aws/operators/test_s3_bucket_tagging_system.py
• tests/providers/amazon/aws/operators/test_emr_system.py
• tests/providers/amazon/aws/transfers/test_imap_attachment_to_s3_system.py
• tests/providers/amazon/aws/transfers/test_google_api_to_s3_system.py
• tests/providers/amazon/aws/transfers/test_s3_to_redshift_system.py
• tests/providers/amazon/aws/hooks/test_base_aws_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for Migrate Amazon example DAGs to new design

Migrate Postgres system tests to new design

There is a new design of system tests that was introduced by the
AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically
before releases.

This is an aggregated issue for all system tests related to Postgres provider,
and it is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:
    - [ ] tests/providers/postgres/operators/test_postgres_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate `Postgres` example DAGs to new design"

Prepare backport packages for providers: apache.hdfs

We would like to release the apache.hdfs backport package as described in using hooks and operators from master in Airflow 1.10

The providers package needs to have set of system tests that need to test operators/hooks/sensors released in the package.

How to prepare system tests is described in System tests.

• [example dags] must exist in airflow/providers/apache/hdfs package
• the example dags should be ready to execute them e2e
• the example dags should be configurable via environment variables
• pytest test running the DAGs must exist in tests/providers/apache.hdfs package
• the example dags should authenticate using
  • forwarded credentials or
  • the @pytest.mark.credential_file annotation
• The BACKPORT_README.md file must describe release notes for the package

Current status of backport packages is described in Backport packages status page

Migrate Airbyte example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all example DAGs related to Airbyte provider. It is created to track progress of their migration.

List of paths to example DAGs:

• airbyte/example_dags/example_airbyte_trigger_job.py

Migrate JDBC example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to JDBC provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• jdbc/example_dags/example_jdbc_queries.py

Prepare backport packages for providers: celery

We would like to release the celery backport package as described in using hooks and operators from master in Airflow 1.10

The providers package needs to have set of system tests that need to test operators/hooks/sensors released in the package.

How to prepare system tests is described in System tests.

• [example dags] must exist in airflow/providers/celery package
• the example dags should be ready to execute them e2e
• the example dags should be configurable via environment variables
• pytest test running the DAGs must exist in tests/providers/celery package
• the example dags should authenticate using
  • forwarded credentials or
  • the @pytest.mark.credential_file annotation
• The BACKPORT_README.md file must describe release notes for the package

Current status of backport packages is described in Backport packages status page

Migrate Microsoft system tests to new design

There is a new design of system tests that was introduced by the
AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically
before releases.

This is an aggregated issue for all system tests related to Microsoft provider,
and it is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:
    - [ ] tests/providers/microsoft/azure/operators/test_adls_delete_system.py
    - [ ] tests/providers/microsoft/azure/transfers/test_sftp_to_wasb_system.py
    - [ ] tests/providers/microsoft/azure/transfers/test_local_to_adls_system.py
    - [ ] tests/providers/microsoft/azure/transfers/test_local_to_wasb_system.py
    - [ ] tests/providers/microsoft/azure/hooks/test_fileshare_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate `Microsoft` example DAGs to new design"

Migrate DingTalk example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to DingTalk provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• dingding/example_dags/example_dingding.py

Migrate Jenkins example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Jenkins provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• jenkins/example_dags/example_jenkins_job_trigger.py

Migrate Airbyte example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all example DAGs related to Airbyte provider. It is created to track progress of their migration.

List of paths to example DAGs:

• airbyte/example_dags/example_airbyte_trigger_job.py

Migrate HTTP system tests to new design

There is a new design of system tests that was introduced by the
AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically
before releases.

This is an aggregated issue for all system tests related to HTTP provider,
and it is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:
    - [ ] tests/providers/http/operators/test_http_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate `HTTP` example DAGs to new design"

Migrate Cncf.Kubernetes example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Cncf.Kubernetes provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• cncf/kubernetes/example_dags/example_spark_kubernetes.py
• cncf/kubernetes/example_dags/example_kubernetes.py

Migrate Neo4j example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Neo4j provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• neo4j/example_dags/example_neo4j.py

Quarantined issues

Quarantined issues

Please do not update status or list of the issues manually. It is automatically updated during
Quarantine workflow, when the workflow executes in the context of Apache Airflow repository.
This happens on schedule (4 times a day) or when a change has been merged or pushed
to the relevant branch.

You can update "Comment" column in the issue list - the update process will read and preserve this column.

Status update

Last status update (UTC): 2020-08-03 23:51:37.966477

List of Quarantined issues

Test	Last run	Last 10 runs	Status	Comment
a	b	c	d	comment

Migrate Amazon system tests to new design

There is a new design of system tests that was introduced by the AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all system tests related to Amazon provider.

It is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:

• tests/providers/amazon/aws/operators/test_ecs_system.py
• tests/providers/amazon/aws/operators/test_eks_system.py
• tests/providers/amazon/aws/operators/test_glacier_system.py
• tests/providers/amazon/aws/operators/test_s3_bucket_tagging_system.py
• tests/providers/amazon/aws/operators/test_emr_system.py
• tests/providers/amazon/aws/transfers/test_imap_attachment_to_s3_system.py
• tests/providers/amazon/aws/transfers/test_google_api_to_s3_system.py
• tests/providers/amazon/aws/transfers/test_s3_to_redshift_system.py
• tests/providers/amazon/aws/hooks/test_base_aws_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate Amazon example DAGs to new design"

Test

Apache Airlfow Version

2.1.1

Operating System

a

Versions of Apache Airlfow Providers

No response

Deployment

Managed Services (Astronomer, Composer, MWAA etc.)

Deployment details

test

What happened

No response

What you expected to happen

No response

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit PR providing that I get guidance.

Code of Conduct

• I agree to follow this project's Code of Conduct

Migrate example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to `` provider.

It is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:

• apache/kylin/example_dags/example_kylin_dag.py
• apache/drill/example_dags/example_drill_dag.py
• apache/druid/example_dags/example_druid_dag.py
• apache/livy/example_dags/example_livy.py
• apache/cassandra/example_dags/example_cassandra_dag.py
• apache/spark/example_dags/example_spark_dag.py
• apache/beam/example_dags/example_beam.py
• apache/pig/example_dags/example_pig.py

Migrate Amazon system tests to new design

There is a new design of system tests that was introduced by the AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all system tests related to Amazon provider.

It is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:

• tests/providers/amazon/aws/operators/test_ecs_system.py (1)
• tests/providers/amazon/aws/operators/test_eks_system.py (4)
• tests/providers/amazon/aws/operators/test_glacier_system.py (1)
• tests/providers/amazon/aws/operators/test_s3_bucket_tagging_system.py (1)
• tests/providers/amazon/aws/operators/test_emr_system.py (2)
• tests/providers/amazon/aws/transfers/test_imap_attachment_to_s3_system.py (1)
• tests/providers/amazon/aws/transfers/test_google_api_to_s3_system.py (2)
• tests/providers/amazon/aws/transfers/test_s3_to_redshift_system.py (1)
• tests/providers/amazon/aws/hooks/test_base_aws_system.py (1)

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for Migrate Amazon example DAGs to new design

Migrate Amazon system tests to new design

There is a new design of system tests that was introduced by the
AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically
before releases.

This is an aggregated issue for all system tests related to Amazon provider,
and it is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:
    - [ ] tests/providers/amazon/aws/operators/test_ecs_system.py
    - [ ] tests/providers/amazon/aws/operators/test_eks_system.py
    - [ ] tests/providers/amazon/aws/operators/test_glacier_system.py
    - [ ] tests/providers/amazon/aws/operators/test_s3_bucket_tagging_system.py
    - [ ] tests/providers/amazon/aws/operators/test_emr_system.py
    - [ ] tests/providers/amazon/aws/transfers/test_imap_attachment_to_s3_system.py
    - [ ] tests/providers/amazon/aws/transfers/test_google_api_to_s3_system.py
    - [ ] tests/providers/amazon/aws/transfers/test_s3_to_redshift_system.py
    - [ ] tests/providers/amazon/aws/hooks/test_base_aws_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate `Amazon` example DAGs to new design"

Migrate Elasticsearch example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Elasticsearch provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• elasticsearch/example_dags/example_elasticsearch_query.py

Migrate GitHub example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to GitHub provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• github/example_dags/example_github.py

Migrate Google system tests to new design

There is a new design of system tests that was introduced by the
AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically
before releases.

This is an aggregated issue for all system tests related to Google provider,
and it is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:
    - [ ] tests/providers/google/leveldb/operators/test_leveldb_system.py
    - [ ] tests/providers/google/firebase/operators/test_firestore_system.py
    - [ ] tests/providers/google/suite/operators/test_sheets_system.py
    - [ ] tests/providers/google/suite/transfers/test_gcs_to_sheets_system.py
    - [ ] tests/providers/google/marketing_platform/operators/test_analytics_system.py
    - [ ] tests/providers/google/marketing_platform/operators/test_search_ads_system.py
    - [ ] tests/providers/google/marketing_platform/operators/test_display_video_system.py
    - [ ] tests/providers/google/marketing_platform/operators/test_campaign_manager_system.py
    - [ ] tests/providers/google/cloud/operators/test_dataprep_system.py
    - [ ] tests/providers/google/cloud/operators/test_kubernetes_engine_system.py
    - [ ] tests/providers/google/cloud/operators/test_compute_system.py
    - [ ] tests/providers/google/cloud/operators/test_functions_system.py
    - [ ] tests/providers/google/cloud/operators/test_spanner_system.py
    - [ ] tests/providers/google/cloud/operators/test_cloud_composer_system.py
    - [ ] tests/providers/google/cloud/operators/test_speech_to_text_system.py
    - [ ] tests/providers/google/cloud/operators/test_cloud_memorystore_system.py
    - [ ] tests/providers/google/cloud/operators/test_life_sciences_system.py
    - [ ] tests/providers/google/cloud/operators/test_bigquery_system.py
    - [ ] tests/providers/google/cloud/operators/test_stackdriver_system.py
    - [ ] tests/providers/google/cloud/operators/test_translate_speech_system.py
    - [ ] tests/providers/google/cloud/operators/test_pubsub_system.py
    - [ ] tests/providers/google/cloud/operators/test_cloud_build_system.py
    - [ ] tests/providers/google/cloud/operators/test_vertex_ai_system.py
    - [ ] tests/providers/google/cloud/operators/test_bigquery_dts_system.py
    - [ ] tests/providers/google/cloud/operators/test_dataflow_system.py
    - [ ] tests/providers/google/cloud/operators/test_datafusion_system.py
    - [ ] tests/providers/google/cloud/operators/test_vision_system.py
    - [ ] tests/providers/google/cloud/operators/test_datacatalog_system.py
    - [ ] tests/providers/google/cloud/operators/test_bigtable_system.py
    - [ ] tests/providers/google/cloud/operators/test_natural_language_system.py
    - [ ] tests/providers/google/cloud/operators/test_dataproc_system.py
    - [ ] tests/providers/google/cloud/operators/test_dataproc_metastore_system.py
    - [ ] tests/providers/google/cloud/operators/test_cloud_sql_system.py
    - [ ] tests/providers/google/cloud/operators/test_gcs_timespan_file_transform_system.py
    - [ ] tests/providers/google/cloud/operators/test_tasks_system.py
    - [ ] tests/providers/google/cloud/operators/test_workflows_system.py
    - [ ] tests/providers/google/cloud/operators/test_datastore_system.py
    - [ ] tests/providers/google/cloud/operators/test_text_to_speech_system.py
    - [ ] tests/providers/google/cloud/operators/test_video_intelligence_system.py
    - [ ] tests/providers/google/cloud/operators/test_gcs_system.py
    - [ ] tests/providers/google/cloud/operators/test_mlengine_system.py
    - [ ] tests/providers/google/cloud/operators/test_automl_system.py
    - [ ] tests/providers/google/cloud/operators/test_translate_system.py
    - [ ] tests/providers/google/cloud/operators/test_dlp_system.py
    - [ ] tests/providers/google/cloud/operators/test_cloud_storage_transfer_service_system.py
    - [ ] tests/providers/google/cloud/transfers/test_gcs_to_local_system.py
    - [ ] tests/providers/google/cloud/transfers/test_bigquery_to_bigquery_system.py
    - [ ] tests/providers/google/cloud/transfers/test_bigquery_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_gcs_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_salesforce_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_gcs_to_bigquery_system.py
    - [ ] tests/providers/google/cloud/transfers/test_azure_fileshare_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_s3_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_mysql_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_sheets_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_presto_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_gdrive_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_sftp_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_postgres_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_local_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_trino_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_calendar_to_gcs_system.py
    - [ ] tests/providers/google/cloud/transfers/test_gcs_to_sftp_system.py
    - [ ] tests/providers/google/cloud/transfers/test_facebook_ads_to_gcs_system.py
    - [ ] tests/providers/google/cloud/hooks/test_kms_system.py
    - [ ] tests/providers/google/cloud/hooks/test_bigquery_system.py
    - [ ] tests/providers/google/cloud/hooks/test_secret_manager_system.py
    - [ ] tests/providers/google/cloud/hooks/test_compute_ssh_system.py
    - [ ] tests/providers/google/cloud/secrets/test_secret_manager_system.py
    - [ ] tests/providers/google/cloud/log/test_gcs_task_handler_system.py
    - [ ] tests/providers/google/cloud/log/test_stackdriver_task_handler_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate `Google` example DAGs to new design"

Migrate Apache example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Apache provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• apache/hive/example_dags/example_twitter_dag.py
• apache/kylin/example_dags/example_kylin_dag.py
• apache/drill/example_dags/example_drill_dag.py
• apache/druid/example_dags/example_druid_dag.py
• apache/livy/example_dags/example_livy.py
• apache/cassandra/example_dags/example_cassandra_dag.py
• apache/spark/example_dags/example_spark_dag.py
• apache/beam/example_dags/example_beam.py
• apache/pig/example_dags/example_pig.py

Testing report provider bug

Body

testing

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Migrate Amazon system tests to new design

There is a new design of system tests that was introduced by the AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all system tests related to Amazon provider.

It is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:

• tests/providers/amazon/aws/operators/test_ecs_system.py (1)
• tests/providers/amazon/aws/operators/test_eks_system.py (4)
• tests/providers/amazon/aws/operators/test_glacier_system.py (1)
• tests/providers/amazon/aws/operators/test_s3_bucket_tagging_system.py (1)
• tests/providers/amazon/aws/operators/test_emr_system.py (2)
• tests/providers/amazon/aws/transfers/test_imap_attachment_to_s3_system.py (1)
• tests/providers/amazon/aws/transfers/test_google_api_to_s3_system.py (2)
• tests/providers/amazon/aws/transfers/test_s3_to_redshift_system.py (1)
• tests/providers/amazon/aws/hooks/test_base_aws_system.py (1)

For anyone involved in working with this issue - please, make sure to also check if all example DAGs are migrated. The issue for them is stored separately. Search for Migrate Amazon example DAGs to new design

Prepare backport packages for providers: amazon

We would like to release the amazon backport package as described in using hooks and operators from master in Airflow 1.10

The providers package needs to have set of system tests that need to test operators/hooks/sensors released in the package.

How to prepare system tests is described in System tests.

• [example dags] must exist in airflow/providers/amazon package
• the example dags should be ready to execute them e2e
• the example dags should be configurable via environment variables
• pytest test running the DAGs must exist in tests/providers/google package
• the example dags should authenticate using
  • forwarded credentials or
  • the @pytest.mark.credential_file annotation
• The BACKPORT_README.md file must describe release notes for the package

Current status of backport packages is described in Backport packages status page

Migrate Asana example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Asana provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• asana/example_dags/example_asana.py

Migrate Influx example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Influx provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• influxdb/example_dags/example_influxdb.py
• influxdb/example_dags/example_influxdb_query.py

Migrate Amazon system tests to new design

There is a new design of system tests that was introduced by the AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all system tests related to Amazon provider.

It is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:
- [ ] tests/providers/amazon/aws/operators/test_ecs_system.py
- [ ] tests/providers/amazon/aws/operators/test_eks_system.py
- [ ] tests/providers/amazon/aws/operators/test_glacier_system.py
- [ ] tests/providers/amazon/aws/operators/test_s3_bucket_tagging_system.py
- [ ] tests/providers/amazon/aws/operators/test_emr_system.py
- [ ] tests/providers/amazon/aws/transfers/test_imap_attachment_to_s3_system.py
- [ ] tests/providers/amazon/aws/transfers/test_google_api_to_s3_system.py
- [ ] tests/providers/amazon/aws/transfers/test_s3_to_redshift_system.py
- [ ] tests/providers/amazon/aws/hooks/test_base_aws_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate Amazon example DAGs to new design"

dasdasdasd

Description

asdada

Use case/motivation

sdasdaas

Related issues

dsdad

Are you willing to submit PR?

• Yes I am willing to submit PR providing that I get guidance.

Code of Conduct

• I agree to follow this project's
  Code of Conduct

Migrate Snowflake system tests to new design

There is a new design of system tests that was introduced by the
AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically
before releases.

This is an aggregated issue for all system tests related to Snowflake provider,
and it is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:
    - [ ] tests/providers/snowflake/operators/test_snowflake_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate `Snowflake` example DAGs to new design"

Migrate Alibaba example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Alibaba provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• alibaba/cloud/example_dags/example_oss_object.py
• alibaba/cloud/example_dags/example_oss_bucket.py

Prepare backport packages for providers: amazon

We would like to release the amazon backport package as described in using hooks and operators from master in Airflow 1.10

The providers package needs to have set of system tests that need to test operators/hooks/sensors released in the package.

How to prepare system tests is described in System tests.

• [example dags] must exist in airflow/providers/amazon package
• the example dags should be ready to execute them e2e
• the example dags should be configurable via environment variables
• pytest test running the DAGs must exist in tests/providers/amazon package
• the example dags should authenticate using
  • forwarded credentials or
  • the @pytest.mark.credential_file annotation
• The BACKPORT_README.md file must describe release notes for the package

Current status of backport packages is described in Backport packages status page

Migrate Asana system tests to new design

There is a new design of system tests that was introduced by the
AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically
before releases.

This is an aggregated issue for all system tests related to Asana provider,
and it is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:
    - [ ] tests/providers/asana/operators/test_asana_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate `Asana` example DAGs to new design"

Migrate HTTP example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to HTTP provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• http/example_dags/example_http.py

Prepare backport packages for providers: apache.hdfs

We would like to release the apache.hdfs backport package as described in using hooks and operators from master in Airflow 1.10

The providers package needs to have set of system tests that need to test operators/hooks/sensors released in the package.

How to prepare system tests is described in System tests.

• [example dags] must exist in airflow/providers/apache/hdfs package
• the example dags should be ready to execute them e2e
• the example dags should be configurable via environment variables
• pytest test running the DAGs must exist in tests/providers/google package
• the example dags should authenticate using
  • forwarded credentials or
  • the @pytest.mark.credential_file annotation
• The BACKPORT_README.md file must describe release notes for the package

Current status of backport packages is described in Backport packages status page

Migrate Amazon example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Amazon provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• amazon/aws/example_dags/example_s3_bucket_tagging.py
• amazon/aws/example_dags/example_emr_eks_job.py
• amazon/aws/example_dags/example_athena.py
• amazon/aws/example_dags/example_sqs.py
• amazon/aws/example_dags/example_eks_with_nodegroups.py
• amazon/aws/example_dags/example_dms_full_load_task.py
• amazon/aws/example_dags/example_emr_job_flow_manual_steps.py
• amazon/aws/example_dags/example_sagemaker.py
• amazon/aws/example_dags/example_eks_with_nodegroup_in_one_step.py
• amazon/aws/example_dags/example_dynamodb_to_s3.py
• amazon/aws/example_dags/example_s3_to_sftp.py
• amazon/aws/example_dags/example_local_to_s3.py
• amazon/aws/example_dags/example_s3_bucket.py
• amazon/aws/example_dags/example_google_api_to_s3_transfer_basic.py
• amazon/aws/example_dags/example_redshift_cluster.py
• amazon/aws/example_dags/example_lambda.py
• amazon/aws/example_dags/example_s3_to_redshift.py
• amazon/aws/example_dags/example_emr_job_flow_automatic_steps.py
• amazon/aws/example_dags/example_imap_attachment_to_s3.py
• amazon/aws/example_dags/example_ecs_fargate.py
• amazon/aws/example_dags/example_rds.py
• amazon/aws/example_dags/example_eks_with_fargate_in_one_step.py
• amazon/aws/example_dags/example_batch.py
• amazon/aws/example_dags/example_datasync_2.py
• amazon/aws/example_dags/example_dynamodb_to_s3_segmented.py
• amazon/aws/example_dags/example_ecs_ec2.py
• amazon/aws/example_dags/example_eks_with_fargate_profile.py
• amazon/aws/example_dags/example_redshift_to_s3.py
• amazon/aws/example_dags/example_redshift_sql.py
• amazon/aws/example_dags/example_sns.py
• amazon/aws/example_dags/example_google_api_to_s3_transfer_advanced.py
• amazon/aws/example_dags/example_datasync_1.py
• amazon/aws/example_dags/example_glacier_to_gcs.py
• amazon/aws/example_dags/example_redshift_data_execute_sql.py
• amazon/aws/example_dags/example_eks_templated.py
• amazon/aws/example_dags/example_sftp_to_s3.py
• amazon/aws/example_dags/example_salesforce_to_s3.py

Migrate Opsgenie example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Opsgenie provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• opsgenie/example_dags/example_opsgenie_alert.py

Prepare backport packages for providers: celery

We would like to release the celery backport package as described in using hooks and operators from master in Airflow 1.10

The providers package needs to have set of system tests that need to test operators/hooks/sensors released in the package.

How to prepare system tests is described in System tests.

• [example dags] must exist in airflow/providers/celery package
• the example dags should be ready to execute them e2e
• the example dags should be configurable via environment variables
• pytest test running the DAGs must exist in tests/providers/google package
• the example dags should authenticate using
  • forwarded credentials or
  • the @pytest.mark.credential_file annotation
• The BACKPORT_README.md file must describe release notes for the package

Current status of backport packages is described in Backport packages status page

Migrate Google example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Google provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• google/leveldb/example_dags/example_leveldb.py
• google/firebase/example_dags/example_firestore.py
• google/ads/example_dags/example_ads.py
• google/suite/example_dags/example_sql_to_sheets.py
• google/suite/example_dags/example_gcs_to_sheets.py
• google/suite/example_dags/example_gcs_to_gdrive.py
• google/suite/example_dags/example_sheets.py
• google/marketing_platform/example_dags/example_analytics.py
• google/marketing_platform/example_dags/example_display_video.py
• google/marketing_platform/example_dags/example_campaign_manager.py
• google/marketing_platform/example_dags/example_search_ads.py
• google/cloud/example_dags/example_bigquery_queries.py
• google/cloud/example_dags/example_gcs_to_sftp.py
• google/cloud/example_dags/example_oracle_to_gcs.py
• google/cloud/example_dags/example_compute_igm.py
• google/cloud/example_dags/example_automl_vision_classification.py
• google/cloud/example_dags/example_datacatalog.py
• google/cloud/example_dags/example_bigquery_to_mssql.py
• google/cloud/example_dags/example_mssql_to_gcs.py
• google/cloud/example_dags/example_mysql_to_gcs.py
• google/cloud/example_dags/example_bigquery_to_bigquery.py
• google/cloud/example_dags/example_natural_language.py
• google/cloud/example_dags/example_life_sciences.py
• google/cloud/example_dags/example_automl_nl_text_extraction.py
• google/cloud/example_dags/example_automl_video_intelligence_classification.py
• google/cloud/example_dags/example_trino_to_gcs.py
• google/cloud/example_dags/example_bigquery_transfer.py
• google/cloud/example_dags/example_cloud_storage_transfer_service_gcp.py
• google/cloud/example_dags/example_translate.py
• google/cloud/example_dags/example_local_to_gcs.py
• google/cloud/example_dags/example_compute_ssh.py
• google/cloud/example_dags/example_dataflow_sql.py
• google/cloud/example_dags/example_text_to_speech.py
• google/cloud/example_dags/example_presto_to_gcs.py
• google/cloud/example_dags/example_looker.py
• google/cloud/example_dags/example_sftp_to_gcs.py
• google/cloud/example_dags/example_facebook_ads_to_gcs.py
• google/cloud/example_dags/example_gcs_to_bigquery.py
• google/cloud/example_dags/example_postgres_to_gcs.py
• google/cloud/example_dags/example_spanner.py
• google/cloud/example_dags/example_gdrive_to_gcs.py
• google/cloud/example_dags/example_dlp.py
• google/cloud/example_dags/example_bigquery_dts.py
• google/cloud/example_dags/example_datastore.py
• google/cloud/example_dags/example_bigquery_operations.py
• google/cloud/example_dags/example_automl_nl_text_classification.py
• google/cloud/example_dags/example_bigquery_to_gcs.py
• google/cloud/example_dags/example_automl_vision_object_detection.py
• google/cloud/example_dags/example_bigtable.py
• google/cloud/example_dags/example_gcs_to_local.py
• google/cloud/example_dags/example_vertex_ai.py
• google/cloud/example_dags/example_automl_video_intelligence_tracking.py
• google/cloud/example_dags/example_gcs_timespan_file_transform.py
• google/cloud/example_dags/example_dataproc_metastore.py
• google/cloud/example_dags/example_salesforce_to_gcs.py
• google/cloud/example_dags/example_functions.py
• google/cloud/example_dags/example_compute.py
• google/cloud/example_dags/example_sheets_to_gcs.py
• google/cloud/example_dags/example_dataflow.py
• google/cloud/example_dags/example_datafusion.py
• google/cloud/example_dags/example_cloud_sql_query.py
• google/cloud/example_dags/example_translate_speech.py
• google/cloud/example_dags/example_automl_tables.py
• google/cloud/example_dags/example_cloud_memorystore.py
• google/cloud/example_dags/example_bigquery_sensors.py
• google/cloud/example_dags/example_gcs.py
• google/cloud/example_dags/example_mlengine.py
• google/cloud/example_dags/example_dataprep.py
• google/cloud/example_dags/example_cloud_storage_transfer_service_aws.py
• google/cloud/example_dags/example_pubsub.py
• google/cloud/example_dags/example_gcs_to_gcs.py
• google/cloud/example_dags/example_s3_to_gcs.py
• google/cloud/example_dags/example_gdrive_to_local.py
• google/cloud/example_dags/example_azure_fileshare_to_gcs.py
• google/cloud/example_dags/example_workflows.py
• google/cloud/example_dags/example_kubernetes_engine.py
• google/cloud/example_dags/example_dataproc.py
• google/cloud/example_dags/example_speech_to_text.py
• google/cloud/example_dags/example_cloud_composer.py
• google/cloud/example_dags/example_automl_translation.py
• google/cloud/example_dags/example_stackdriver.py
• google/cloud/example_dags/example_tasks.py
• google/cloud/example_dags/example_calendar_to_gcs.py
• google/cloud/example_dags/example_video_intelligence.py
• google/cloud/example_dags/example_cloud_build.py
• google/cloud/example_dags/example_dataflow_flex_template.py
• google/cloud/example_dags/example_automl_nl_text_sentiment.py
• google/cloud/example_dags/example_cloud_sql.py
• google/cloud/example_dags/example_vision.py

Migrate Microsoft example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Microsoft provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• microsoft/mssql/example_dags/example_mssql.py
• microsoft/azure/example_dags/example_azure_blob_to_gcs.py
• microsoft/azure/example_dags/example_azure_cosmosdb.py
• microsoft/azure/example_dags/example_fileshare.py
• microsoft/azure/example_dags/example_adls_delete.py
• microsoft/azure/example_dags/example_local_to_adls.py
• microsoft/azure/example_dags/example_azure_container_instances.py
• microsoft/azure/example_dags/example_local_to_wasb.py
• microsoft/azure/example_dags/example_sftp_to_wasb.py
• microsoft/azure/example_dags/example_adf_run_pipeline.py
• microsoft/winrm/example_dags/example_winrm.py

Migrate example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to `` provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• apache/kylin/example_dags/example_kylin_dag.py
• apache/drill/example_dags/example_drill_dag.py
• apache/druid/example_dags/example_druid_dag.py
• apache/livy/example_dags/example_livy.py
• apache/cassandra/example_dags/example_cassandra_dag.py
• apache/spark/example_dags/example_spark_dag.py
• apache/beam/example_dags/example_beam.py
• apache/pig/example_dags/example_pig.py

Migrate CNCF system tests to new design

There is a new design of system tests that was introduced by the
AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically
before releases.

This is an aggregated issue for all system tests related to CNCF provider,
and it is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:
    - [ ] tests/providers/cncf/kubernetes/operators/test_spark_kubernetes_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate `CNCF` example DAGs to new design"

asdadad

Apache Airlfow Version

2.1.3 (latest)

Operating System

adsadasda

Versions of Apache Airlfow Providers

dasdasdadasdasd

Deployment

Managed Services (Astronomer, Composer, MWAA etc.)

Deployment details

No response

What happened

No response

What you expected to happen

No response

How to reproduce

No response

Anything else

asdadasda

Are you willing to submit PR?

• Yes I am willing to submit PR providing that I get guidance.

Code of Conduct

• I agree to follow this project's Code of Conduct

Migrate Airbyte example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Airbyte provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• airbyte/example_dags/example_airbyte_trigger_job.py

Migrate MySQL example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to MySQL provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• mysql/example_dags/example_mysql.py

Prepare backport packages for providers: google

We would like to release the backport package as described in using hooks and operators from master in Airflow 1.10

The providers package needs to have set of system tests that need to test operators/hooks/sensors released in the package.

How to prepare system tests is described in System tests.

• [example dags] must exist in airflow/providers/google package
• the example dags should be ready to execute them e2e
• the example dags should be configurable via environment variables
• pytest test running the DAGs must exist in tests/providers/google package
• the example dags should authenticate using
  • forwarded credentials or
  • the annotation
• The BACKPORT_README.md file must describe release notes for the package

Current status of backport packages is described in Backport packages status page

Migrate Databricks example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all exmpled dags related to Databricks provider.

It is created to track progress of their migration.

List of paths to example DAGs:

• databricks/example_dags/example_databricks_sql.py
• databricks/example_dags/example_databricks.py

Migrate Apache Beam system tests to new design

There is a new design of system tests that was introduced by the
AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically
before releases.

This is an aggregated issue for all system tests related to Apache Beam provider,
and it is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:
    - [ ] tests/providers/apache/beam/operators/test_beam_system.py

For anyone involved in working with this issue - please, make sure to also check if all example
DAGs are migrated. The issue for them is stored separately.
Search for "Migrate `Apache Beam` example DAGs to new design"