Docker build doesn't work on M1 mac but works perfectly fine in Mac Intel

Question

Currently, I'm migrating all work from Macbook Intel to Macbook Pro Apple M1 Series chip

When I build a dockerfile on an Intel Macbook, it is a success. However, when I build the same dockerFile on M1 Mac I get this error.

docker build --no-cache . -t apache/airflow:2.3.3                                             
[+] Building 47.1s (10/11)                                                                                                             docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                   0.0s
 => => transferring dockerfile: 1.98kB                                                                                                                 0.0s
 => [internal] load metadata for docker.io/apache/airflow:2.3.3-python3.8                                                                              0.6s
 => [auth] apache/airflow:pull token for registry-1.docker.io                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                        0.0s
 => [internal] load build context                                                                                                                      0.0s
 => => transferring context: 189B                                                                                                                      0.0s
 => CACHED [1/6] FROM docker.io/apache/airflow:2.3.3-python3.8@sha256:c6a69cf649ad6cbe5e06154b3625c5c8c960c32e605fda3ae6f3b206b7f2b96e                 0.0s
 => [2/6] COPY requirements.txt .                                                                                                                      0.0s
 => [3/6] RUN apt-get update && apt-get install -y build-essential gcc g++ git awscli                                                                 42.1s
 => [4/6] RUN sudo rm -rf /var/lib/apt/lists/* &&     mkdir /var/lib/apt/lists/partial &&     apt-get clean &&     apt-get update                      1.4s
 => ERROR [5/6] RUN python3 -m pip install --user -r requirements.txt     -c "https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/cons  2.9s
------
 > [5/6] RUN python3 -m pip install --user -r requirements.txt     -c "https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-source-providers-3.8.txt":
0.241 + python3 -m pip install --user -r requirements.txt -c https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-source-providers-3.8.txt
1.535 Collecting apache-airflow-providers-common-sql==1.4.0
1.656   Downloading apache_airflow_providers_common_sql-1.4.0-py3-none-any.whl (31 kB)
1.672 Requirement already satisfied: apache-airflow-providers-amazon==4.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from -r requirements.txt (line 2)) (4.0.0)
1.748 Collecting apache-airflow-providers-snowflake==3.2.0
1.777   Downloading apache_airflow_providers_snowflake-3.2.0-py3-none-any.whl (28 kB)
1.781 Requirement already satisfied: apache-airflow-providers-mysql==3.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (3.0.0)
1.838 Collecting apache-airflow-providers-microsoft-mssql==3.2.0
1.863   Downloading apache_airflow_providers_microsoft_mssql-3.2.0-py3-none-any.whl (16 kB)
1.868 Requirement already satisfied: apache-airflow-providers-odbc==3.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from -r requirements.txt (line 6)) (3.0.0)
1.930 Collecting apache-airflow-providers-mongo==3.0.0
1.954   Downloading apache_airflow_providers_mongo-3.0.0-py3-none-any.whl (16 kB)
1.958 Requirement already satisfied: apache-airflow-providers-slack==5.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from -r requirements.txt (line 8)) (5.0.0)
2.015 Collecting apache-airflow-providers-opsgenie==5.0.0
2.039   Downloading apache_airflow_providers_opsgenie-5.0.0-py3-none-any.whl (16 kB)
2.186 Collecting apache-airflow-providers-cncf-kubernetes==5.3.0
2.214   Downloading apache_airflow_providers_cncf_kubernetes-5.3.0-py3-none-any.whl (62 kB)
2.227 Requirement already satisfied: boto3==1.24.23 in /home/airflow/.local/lib/python3.8/site-packages (from -r requirements.txt (line 11)) (1.24.23)
2.228 Requirement already satisfied: botocore==1.27.23 in /home/airflow/.local/lib/python3.8/site-packages (from -r requirements.txt (line 12)) (1.27.23)
2.274 Collecting openpyxl==3.0.9
2.303   Downloading openpyxl-3.0.9-py2.py3-none-any.whl (242 kB)
2.341 Requirement already satisfied: pandas==1.4.3 in /home/airflow/.local/lib/python3.8/site-packages (from -r requirements.txt (line 14)) (1.4.3)
2.341 Requirement already satisfied: psycopg2-binary==2.9.3 in /home/airflow/.local/lib/python3.8/site-packages (from -r requirements.txt (line 15)) (2.9.3)
2.342 Requirement already satisfied: kubernetes==23.6.0 in /home/airflow/.local/lib/python3.8/site-packages (from -r requirements.txt (line 16)) (23.6.0)
2.343 Requirement already satisfied: requests==2.28.0 in /home/airflow/.local/lib/python3.8/site-packages (from -r requirements.txt (line 17)) (2.28.0)
2.387 Collecting snowflake-sqlalchemy==1.3.4
2.416   Downloading snowflake_sqlalchemy-1.3.4-py2.py3-none-any.whl (32 kB)
2.502 Collecting mysql-connector-python==8.0.29
2.524   Downloading mysql_connector_python-8.0.29-py2.py3-none-any.whl (342 kB)
2.571 Collecting mysqlclient==2.1.1
2.594   Downloading mysqlclient-2.1.1.tar.gz (88 kB)
2.607   Preparing metadata (setup.py): started
2.867   Preparing metadata (setup.py): finished with status 'error'
2.869   error: subprocess-exited-with-error
2.869
2.869   × python setup.py egg_info did not run successfully.
2.869   │ exit code: 1
2.869   ╰─> [16 lines of output]
2.869       /bin/sh: 1: mysql_config: Permission denied
2.869       /bin/sh: 1: mariadb_config: Permission denied
2.869       /bin/sh: 1: mysql_config: Permission denied
2.869       Traceback (most recent call last):
2.869         File "<string>", line 2, in <module>
2.869         File "<pip-setuptools-caller>", line 34, in <module>
2.869         File "/tmp/pip-install-wk9zky_3/mysqlclient_ca5924e783e548669d6fe900a78e8727/setup.py", line 15, in <module>
2.869           metadata, options = get_config()
2.869         File "/tmp/pip-install-wk9zky_3/mysqlclient_ca5924e783e548669d6fe900a78e8727/setup_posix.py", line 70, in get_config
2.869           libs = mysql_config("libs")
2.869         File "/tmp/pip-install-wk9zky_3/mysqlclient_ca5924e783e548669d6fe900a78e8727/setup_posix.py", line 31, in mysql_config
2.869           raise OSError("{} not found".format(_mysql_config_path))
2.869       OSError: mysql_config not found
2.869       mysql_config --version
2.869       mariadb_config --version
2.869       mysql_config --libs
2.869       [end of output]
2.869
2.869   note: This error originates from a subprocess, and is likely not a problem with pip.
2.870 error: metadata-generation-failed
2.870
2.870 × Encountered error while generating package metadata.
2.870 ╰─> See above for output.
2.870
2.870 note: This is an issue with the package mentioned above, not pip.
2.870 hint: See above for details.
------
Dockerfile:35
--------------------
  34 |     USER airflow
  35 | >>> RUN python3 -m pip install --user -r requirements.txt \
  36 | >>>     -c "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-source-providers-${PYTHON_RUNTIME_VERSION}.txt"
  37 |     RUN python3 -m pip install dbt-core dbt-snowflake snowflake-connector-python typing-extensions==4.6.3
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -e -u -x -c python3 -m pip install --user -r requirements.txt     -c \"https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-source-providers-${PYTHON_RUNTIME_VERSION}.txt\"" did not complete successfully: exit code: 1

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/icmt9k7rj6v2k9qhtzpgbqf95

How do we solve this problem?

My requirements.txt and dockerFile are below

# Global Docker Build arguments
ARG AIRFLOW_VERSION=2.3.3
ARG PYTHON_RUNTIME_VERSION=3.8
FROM apache/airflow:${AIRFLOW_VERSION}-python${PYTHON_RUNTIME_VERSION}
SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
USER 0
COPY requirements.txt .

RUN apt-get update && apt-get install -y build-essential gcc g++ git awscli
RUN sudo rm -rf /var/lib/apt/lists/* && \
    mkdir /var/lib/apt/lists/partial && \
    apt-get clean && \
    apt-get update

ARG AIRFLOW_VERSION
ARG PYTHON_RUNTIME_VERSION
ENV DEBIAN_FRONTEND=noninteractive \
    LANGUAGE=C.UTF-8 \
    LANG=C.UTF-8 \
    LC_ALL=C.UTF-8 \
    LC_CTYPE=C.UTF-8 \
    LC_MESSAGES=C.UTF-8 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    PIP_PROGRESS_BAR="off" \
    PIP_NO_CACHE_DIR=1

USER airflow
RUN python3 -m pip install --user -r requirements.txt \
    -c "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-source-providers-${PYTHON_RUNTIME_VERSION}.txt"
RUN python3 -m pip install dbt-core dbt-snowflake snowflake-connector-python typing-extensions==4.6.3

Requirements.txt

apache-airflow-providers-common-sql==1.4.0
apache-airflow-providers-amazon==4.0.0
apache-airflow-providers-snowflake==3.2.0
apache-airflow-providers-mysql==3.0.0
apache-airflow-providers-microsoft-mssql==3.2.0
apache-airflow-providers-odbc==3.0.0
apache-airflow-providers-mongo==3.0.0
apache-airflow-providers-slack==5.0.0
apache-airflow-providers-opsgenie==5.0.0
apache-airflow-providers-cncf-kubernetes==5.3.0
boto3==1.24.23
botocore==1.27.23
openpyxl==3.0.9
pandas==1.4.3
psycopg2-binary==2.9.3 
kubernetes==23.6.0
requests==2.28.0
snowflake-sqlalchemy==1.3.4
mysql-connector-python==8.0.29
mysqlclient==2.1.1
opsgenie-sdk==2.1.5
pymssql==2.2.5
pymongo==3.12.3
timeago
billiard==3.6.4.0
ntlm-auth==1.5.0
python-dateutil==2.8.2
pytz==2022.1
requests-ntlm==1.1.0
apache-airflow-providers-ftp
apache-airflow-providers-http
apache-airflow-providers-sqlite
apache-airflow-providers-postgres
apache-airflow-providers-ssh
jq
gspread==5.11.3
oauth2client==4.1.3

I suspect mysqlclient==2.1.1 is the culprit, because I just ran pip install mysqlclient==2.1.1 on my MacBook Pro (with an M1 chip, by the way) and I get the exact same error you're getting in your output. Have a look here for a possible solution. — fireshadow52
– fireshadow52, Commented Apr 3, 2024 at 4:08
Thanks a lot , from the solution link you provided , which is the right solution to update in the dockerFile ? — Kar
– Kar, Commented Apr 3, 2024 at 4:38

Andrey Anshin · Accepted Answer · 2024-04-03 08:38:22Z

You have to install MySQL/MariaDB dev headers for successfully build mysqlclient. This package prebuilt and preinstalled in the regular (non-slim) official Airflow Docker Image for ARM (experimental) since Airflow 2.6

You might want to customise your docker image from the official one, see: Building the image: Customizing the image

If you want to build your own image from the scratch, I would recommend to look on Dockerfile in Apache Airlflow repo, you also could have a look which system dependency installed in addition for build mysqlclient into the scripts/docker/install_mysql.sh. And as far as I remember you need to install pkgconf via apt for build recent versions of mysqlclient

Collectives™ on Stack Overflow

Docker build doesn't work on M1 mac but works perfectly fine in Mac Intel

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related