Geo: Accelerate container images by serving read request from secondary site
<!--IssueSummary start-->
<details>
<summary>
Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards.
</summary>
- [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=365864)
</details>
<!--IssueSummary end-->
<!-- This template is a great use for issues that are feature::additions or technical tasks for larger issues.-->
## Proposal
Accelerate container images by serving read request from secondary site
### Currently
The Container Registry in a secondary site stands on its own. Pulls and pushes are routed directly to the Container Registry, e.g. `registry.gitlab.example.com:5000`.
It is _technically_ possible to pull images from the secondary site's container registry. But it's not a good experience in general, and therefore we don't consider it a supported thing to do, yet.
1. If you push to the secondary site's registry, then the data does **not** replicate to the primary site. Additionally, the secondary site may delete that image/tag since it doesn't exist in the primary site registry.
1. Replication is asynchronous. If you attempt to pull something from the secondary site's registry after pushing it to the primary site's registry, it may or may not exist yet, so you may get a 404 or some error like that.
1. People need to remember two distinct registry URLs, that they must not push to one of them, and that replication is async.
### Initial ideas
- GitLab eats its Container Registry: Instruct sysadmins to route secondary site Container Registry requests to GitLab Workhorse? Then we could reuse "proxying" logic in GitLab Workhorse and GitLab Rails.
- Make Container Registry smarter: Add proxying logic to Container Registry itself? Does Container Registry already make internal API requests against GitLab Rails?
## More Details
This feature request [might require work in the GitLab Helm chart to configure Ingresses](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/3253/#note_1473528726).
### K8s flow for `docker login registry.gitlab.com`
```mermaid
sequenceDiagram
Client->>+LoadBalancer: GET registry.gitlab.com/
LoadBalancer->>+NGINX: GET registry.gitlab.com/
loop route
NGINX ->> NGINX: Route based on Host/ServerAlias
end
NGINX->>+Registry: GET /
Registry->>-NGINX: HTTP 401 (token_realm: gitlab.com)
NGINX->>-LoadBalancer: HTTP 401
LoadBalancer->>-Client: HTTP 401
Client->>+LoadBalancer: POST gitlab.com/jwt/auth
LoadBalancer->>+NGINX: POST /jwt/auth
loop route
NGINX ->> NGINX: Route based on Host/ServerAlias
end
NGINX->>+Webservice: POST /jwt/auth
Webservice->>-NGINX: HTTP 401/200 (signed token)
NGINX->>-LoadBalancer: HTTP 401/200 (signed token)
LoadBalancer->>-Client: HTTP 401/200 (signed token)
Client->>+LoadBalancer: GET registry.gitlab.com/
LoadBalancer->>+NGINX: GET registry.gitlab.com/
loop route
NGINX ->> NGINX: Route based on Host/ServerAlias
end
NGINX->>+Registry: GET /
Registry->>-NGINX: HTTP 200 (valid token)
NGINX->>-LoadBalancer: HTTP 200 (valid token)
LoadBalancer->>-Client: HTTP200 (valid token)
```
---
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
*This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
issue