What is the best way to handle sql connection in http server (Flask) without ORM in Python?

Question

I am using Flask with MySQL (MariaDB) database. To handle sql connection and cursor I use self-made context manager. I open and close connection inside each Flask http request hadling function, so I can be sure that number of connections to db will not exceed the certain number, but it creates overhead. I am sure that the same mysql connections can be used by other users, what other approach to handle sql connection and cursor I can use, if I do not use ORM ?

Context managers to hangle cursor and connection:

from contextlib import contextmanager
import mysql.connector
from mysql.connector.errors import Error

@contextmanager
def mysql_connection(user, password, host, database, auth_plugin):
    _conn = mysql.connector.connect(user=user, password=password, host=host, database=database, auth_plugin=auth_plugin)
    try:
        yield _conn
    except (Exception, Error) as ex:
        # if error happened all made changes during the connection will be rolled back:
        _conn.rollback()
        # this statement re-raise error to let it be handled in outer scope:
        raise
    else:
        # if everything is fine commit all changes to save them in db:
        _conn.commit()
    finally:
        # close connection to db, do not wait for timeout release:
        _conn.close()


@contextmanager
def mysql_curs(user, password, host, database, auth_plugin) -> "curs":
    with mysql_connection(user=user, password=password, host=host, database=database, auth_plugin=auth_plugin) as _conn:
        _curs = _conn.cursor()
        try:
            yield _curs
        finally:
            _curs.close()  # close cursor when everything is done

Some random Flask http handler function:

@app.route('/admin_panel/repair', methods=["GET"])
def repair_show_all_menu_webpages():
    """The page exists to repair menu if not existent flask function was added"""
    try:
        with common_db_ops.mysql_curs() as curs:
            left_side_menu = []
            webpages = admin_ops.show_all_menu_webpages_to_repair(curs)
    except (Exception, Error) as err:
        app.logger.error(f"Failed to repair website: {err}")
        abort(500)

    return render_template('repair_menu.html', webpages=webpages, left_side_menu=left_side_menu)

Edit: I would like to add that I found the following article which discuss how to use Flask with PostgreSQL and create your customized sql connection context manager, but I have question where in Flask I should declare sql connectors Pool:

Manage RAW database connection pool in Flask

Answer 1

Try to pool connections

From offical docs:

A pool opens a number of connections and handles thread safety when providing connections to requesters

Implementing connection pooling, you can reuse existing connections

dbconfig = {
  "database": "test",
  "user":     "joe"
}

cnxpool = mysql.connector.connect(pool_name = "mypool",
                                  pool_size = 3,    # or any number to suit your need
                                  **dbconfig)


# then to get a connection from pool use
cnx = cnxpool.get_connection()

For more see: https://dev.mysql.com/doc/connector-python/en/connector-python-connection-pooling.html

Answer 2

If anybody is interested in the approach of handling sql connection without ORM, I made the following steps to combine MySQL Connections Pool, context manager and Flask:

SQL_CONN_POOL = pooling.MySQLConnectionPool(
    pool_name="mysqlpool",
    pool_size=10,
    user=DB_USER,
    password=DB_PASS,
    host=DB_HOST,
    database=DATABASE,
    auth_plugin=DB_PLUGIN
)


@contextmanager
def mysql_connection_from_pool() -> "conn":
    conn_pool = SQL_CONN_POOL  # get connection from the pool, all the rest is the same

    # you can add print(conn_pool) here to be sure that pool
    # is the same for each http request

    _conn = conn_pool.get_connection()
    try:
        yield _conn
    except (Exception, Error) as ex:
        # if error happened all made changes during the connection will be rolled back:
        _conn.rollback()
        # this statement re-raise error to let it be handled in outer scope:
        raise
    else:
        # if everything is fine commit all changes to save them in db:
        _conn.commit()
    finally:
        # actually it returns cursor to the pool, rather than close it
        _conn.close()


@contextmanager
def mysql_curs_from_pool() -> "curs":
    with mysql_connection_from_pool() as _conn:
        _curs = _conn.cursor()
        try:
            yield _curs
        finally:
            _curs.close()

I used the following links to answer the question:

Manage RAW database connection pool in Flask

MySQL docs