Name already in use

…9954)

* Add option to show output of `SQLExecuteQueryOperator` in the log

Add basic serialization capabilities for the ORM SqlAlchemy models
that we use on the client side of the Internal API. Serializing
the whole ORM models is rather complex, therefore it seems much
more reasonable to convert the ORM models into serializable
form and use them - rather than the SQLAlchemy models.

There are just a handful of those models that we need to serialize,
and it is important to maintain typing of the fields in the objects
for MyPy verification so we can allow some level of duplication
and redefine the models as pure Python objects.

We only need one-way converstion (from database models to Python models),
because all the DB operations and modifications of the Database
entries will be done in the internal API server, so the server side
of any method will be able to use primary key stored in the serializable
object, to retrieve the actual DB model to update.

We also need to serialization to work both way - an easy way to convert
such Python classees to json and back - including validation.

We could serialize those models manually, but this would be quite an
overhead to develop and maintain - therefore we are harnessing the
power of Pydantic, that has already ORM mapping to plain Python
(Pydantic) classes built in.

This PR implements definition of the Pydantic classes and tests
for the classes testing:

* conversion of the ORM models to Pydantic objects
* serialization of the Pydantic classes to json
* deserialization of the json-serialized classes to Pydantic objects

* Delay ConnectionModelView init until it's accessed

This removed the "lazy" initialization of ConnectionModelView (and
ConnectionForm used by it) that happens when the webserver first
launches, and moves the corresponding logic to until the view is
actually being accessed.

This further delays the provider manager from loading connection form
widgets until the last possible moment and speeds up the webserver's
startup sequence.

While this does make the connection form view a bit slower (only the
first time since the result is cached), the slowdown should not be as
noticeable since the provider manager should generally be partially
loaded into memory at that point.

* Fix connection model view tests

* Use flask.has_request_context() to time the init

This makes sure we only init when the view is accessed (instead of on
startup) without relying on Flask-Appbuilder internals.

* Add EC2CreateInstanceOperator and EC2TerminteInstanceOperator
Change system test to use the new operators
Add unit tests for new operators

* Add support for multiple ids to EC2TerminateInstanceOperator
Change system test to terminate without stopping instances

* Fix failing tests for terminate operator

* Update doc strings to add that the operators can create/terminate multiple instances
Add tests for creating/terminating multiple instances

* Fix system test so it passes
Fix doc string on EC2TerminateInstanceOperator

---------

Co-authored-by: syedahsn <syedahsn@ud74d1a752d7e5b.ant.amazon.com>

* add a new config

* add remote_task_handler_kwargs conf and add its content as kwargs for remote task handlers

* add delete_local_logs to logging tasks doc

Co-authored-by: Niko Oliveira <onikolas@amazon.com>

---------

Co-authored-by: Niko Oliveira <onikolas@amazon.com>

* move validation to execute + change type hints

* revert type changes

* update tests

* move json normalisation in SubmitRunDeferrable too

---------

Co-authored-by: Hernan Resnizky <hernanresnizky@NEX-0003.local>
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

* wait until the container is not terminated instead of waiting while it's running

Pipx installation is cached between runs and it should use
python location in order to not reuse cache between different
python versions, otherwise pipx installation might fail.

… of using dag processing timeout (#29446)

* Scheduler, make stale DAG deactivation threshold configurable instead of using dag processing timeout

* Apply suggestions from code review

Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>

* stale_dag_threshold, make default 50

* Add stale_dag_threshold to stale dags test

* Fixing type of stale_dag_threshold

* Update airflow/config_templates/config.yml

Co-authored-by: Andrey Anshin <Andrey.Anshin@taragol.is>

* Remove redundant processor_timeout_seconds

* Reword description

* reparsed -> re-parsed

---------

Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
Co-authored-by: Andrey Anshin <Andrey.Anshin@taragol.is>

Before we attempt to run tests in parallel, we quickly check once
if Pytest collection works. This is in order to avoid costly
parallel test execution if that makes no sense to initialize all
the parallel machines. This check used to be done in "Wait for
CI Inages" step, but running it there has the undesireable
side effect that it is not obvious that it's the collection
that fails, also it prevents other jobs (for example
static checks and docs building) from running. This means that
the contributor does not get all the feedback that could be
given immediately.

This PR separates the collection into separate job and only
makes "test" jobs depend on it - all the other jobs that need
CI image depend on "wait for CI image" one and should continue
running even if pytest collection fails.

CI diagrams are also updated to reflect a bit better optionality
and parallelism of the CI jobs.

The DOCKER_BUILDKIT=1 was only set in some specific cases, but since
our images are using buildkit features, we should always use
buildkit when building the images.

The Importlib-metadata 5 breaks importing Celery < 5.3.0. Until
Celery 5.3.0 gets released (it is pre-release/beta now) we should
workaround the problem by limiting importlib-metadata to < 5.0.0.

…tion (#29694)

Previously the views code was hardcoded to look out for the
SequentialExecutor and warn to not use it in production. This change
makes that generalized to any non-production executor

…nt type conversion by specifying BIT fields explicitly (#29902)

This is the attempt to fix #28751 by setting a memo to the same instance of DEFAULT,
which is the stub for a Literal value type introduced to support Python 3.7 with mypy
in Google's API core Python client.

Without this any operator that has the parameter set to DEFAULT constant
(which is often the default value for retry parameters) will throw the
error in mini-scheduler after execution as the attemt to deepcopy this
project would fail.

Co-authored-by: Igor Kholopov <kholopovus@gmail.com>

- Also add more documentation on how to use the generated
Dockerfile for testing installation

* fix empty variables, config and connections paths

* add tests for empty paths with and without mount point

Changes to import paths:

`operators.kubernetes_pod` -> `operators.pod`
`triggers.kubernetes_pod` -> `triggers.pod`

This PR is backward compatible

…#29919)

The images for pgbouncer and pgbouncer exporter have been refreshed
with latest working for pgbounced alpine images and pgbpuncer
versions. This PR switches to those images by default.

Follow up after #29792