DEV Community

Clariza Look
Clariza Look

Posted on • Edited on

Migrating to Self-Hosted 3scale API Management on ROSA Kubernetes

#kubernetes #apimanagement #devops #programming

I was tasked to migrate from a Red Hat-hosted 3scale portal to a self-hosted version in ROSA (Red Hat OpenShift Service on AWS). This presented quite a challenge as my knowledge in Kubernetes was mostly theoretical, based on studying for the Kubernetes and Cloud Native Associate (KCNA) certification exam.

The goal was to recreate a self-hosted version of 3scale using an operator in ROSA, but what I thought would be a straightforward deployment turned into a valuable learning experience.

What is Red Hat-Managed/hosted 3scale?

When using Red Hat-hosted 3scale (also known as "SaaS" or managed 3scale), all infrastructure complexities are abstracted away. Red Hat handles the deployment, maintenance, updates, and scaling of the platform.

As a user, you simply access a provided portal URL and focus on managing your APIs rather than worrying about the underlying infrastructure. Your daily tasks revolve around the actual API management activities like adding backends, configuring products, creating applications, setting up authentication, and managing rate limits.

It's a convenient option that requires minimal operational overhead, allowing your team to focus on API strategy rather than platform management.

How 3Scale Api Management Portal looks like

What is self-hosted 3scale?

In contrast, self-hosted 3scale brings both flexibility and responsibility. You gain complete control over your deployment configuration, integration with internal systems, customization options, and data locality.

Since the infrastructure runs on Kubernetes (in my case, ROSA - Red Hat OpenShift Service on AWS), you have access to all the native Kubernetes capabilities for scaling, monitoring, and management.

However, this freedom comes with the need to manage the entire application lifecycle within the Kubernetes ecosystem: installation via operators or templates, configuration through custom resources, scaling via horizontal pod autoscalers, implementing backup strategies, and handling upgrades.

You're responsible for ensuring high availability with proper pod distribution, performance tuning through resource allocation, and troubleshooting any issues that arise in both the 3scale application components and the underlying Kubernetes resources.

The Migration

Migrating from managed to self-hosted represented a significant shift in responsibilities, and I was about to discover just how much Red Hat had been handling behind the scenes.

This blog post documents a real-world troubleshooting journey that encountered and overcame significant challenges:

  1. Missing Routes for Admin Access

  2. DNS resolution issues preventing access to Red Hat's container registry

  3. Architecture mismatch between my ARM-based MacBook for and the x86_64 docker container images required for deployment

  4. PVC Access Mode Issues

  5. Resource Constraints

  6. Missing Service for App Components

By sharing this experience, I hope to help others who might encounter similar issues during their deployment process, especially those who are transitioning from theoretical Kubernetes knowledge to practical application.

The Initial Deployment Attempt

We started by creating a dedicated namespace for our 3scale deployment:

oc create namespace 3scale-backup
Enter fullscreen mode Exit fullscreen mode

After switching to this namespace (oc project 3scale-backup), we downloaded the 3scale API Management Platform template:

curl -o amp.yml https://raw.githubusercontent.com/3scale/3scale-amp-openshift-templates/master/amp/amp.yml
Enter fullscreen mode Exit fullscreen mode

Then we tried to deploy 3scale using this template:

oc new-app --file=amp.yml \
  --param WILDCARD_DOMAIN=apps.[domain of your openshift].openshiftapps.com \
  --param ADMIN_PASSWORD=password123
Enter fullscreen mode Exit fullscreen mode

The template processing appeared successful, creating numerous resources:

  • Imagestreams
  • Deployment configs
  • Services
  • Routes
  • Persistent volume claims
  • Secrets 
oc get all -n [your namespace]
Enter fullscreen mode Exit fullscreen mode

3Scale API Management Rosa deployment

However, when checking the status of the pods, we noticed that many deployments were either not starting, with errors, crashLoopBackOff or stuck in initialization phases:

oc get pods
Enter fullscreen mode Exit fullscreen mode

Get Pods

While some components like Redis and database pods were running fine, critical components like backend-listener, backend-worker, and backend-cron were not deploying at all.

backend-worker issues

The system components were also failing during initialization.

Challenge 1: Missing Routes for Admin Access

Our first challenge was that the URLs for accessing the admin portal https://3scale-admin.apps.[YOUR-ROSA-DOMAIN].openshiftapps.com were showing "Application is not available".

The reason was simple - the template had not created the necessary routes for our self-hosted 3scale services. We manually created them:

oc create route edge system-admin --service=system-provider --hostname=3scale-admin.apps.[YOUR-ROSA-DOMAIN].openshiftapps.com
oc create route edge system-developer --service=system-developer --hostname=3scale.apps.[YOUR-ROSA-DOMAIN].openshiftapps.com
oc create route edge system-master --service=system-master --hostname=master.apps.[YOUR-ROSA-DOMAIN].openshiftapps.com
Enter fullscreen mode Exit fullscreen mode

However, after creating the routes, the admin portal still wasn't accessible. Digging into the logs with oc logs system-app-1-hook-pre, we discovered a more fundamental issue.

Challenge 2: DNS Resolution Issues

The pre-deployment hook was failing with a specific error:

ThreeScale::Core::APIClient::ConnectionError: connection refused: backend-listener:80
Enter fullscreen mode Exit fullscreen mode

Further investigation revealed that the backend components weren't deployed at all. When checking the deployment configs:

oc get dc/backend-listener
NAME               REVISION   DESIRED   CURRENT   TRIGGERED BY
backend-listener   0          1         0         config,image(amp-backend:2.12)
Enter fullscreen mode Exit fullscreen mode

We saw that backend-listener, backend-worker, and backend-cron had REVISION 0 and CURRENT 0, indicating they hadn't been deployed.

oc get dc/backend-listener

The root cause was found in the imagestream:

oc describe imagestream amp-backend
Enter fullscreen mode Exit fullscreen mode

This showed an error:

error: Import failed (InternalError): Internal error occurred: registry.redhat.com/3scale-amp2/backend-rhel8:3scale2.12: Get "https://registry.redhat.com/v2/": dial tcp: lookup registry.redhat.com on 100.10.0.11:23: no such host
Enter fullscreen mode Exit fullscreen mode

Our OpenShift cluster couldn't resolve the hostname registry.redhat.com due to DNS issues. This was confirmed by attempting to run:

nslookup registry.redhat.com
Enter fullscreen mode Exit fullscreen mode

Which returned "No answer" from the DNS server.

Or technically it does not pull the image required from the RedHat registry to the pods.

So our workaround was to manually pull the image to the docker (locally), then push that image to the namespace's private RedHat registry.

Challenge 3: Architecture Mismatch

While working to address the DNS issues , we discovered another challenge - we were trying to pull Red Hat's container images on an ARM64-based machine (likely an Apple Silicon Mac), but the images were only available for x86_64 architecture.

When attempting to pull the images directly:

docker login [RedHat Credentials]
docker pull registry.redhat.io/3scale-amp2/backend-rhel8:3scale2.12
Enter fullscreen mode Exit fullscreen mode

We received:

no matching manifest for linux/arm64/v8 in the manifest list entries
Enter fullscreen mode Exit fullscreen mode

The Solution Process

We implemented a multi-step solution to overcome these challenges:

Step 1: Authentication with Red Hat Registry

First, we logged in to the Red Hat Container Registry:

docker login registry.redhat.io
Enter fullscreen mode Exit fullscreen mode

Step 2: Architecture-aware Image Pulling

Because I was using macOS and having the docker desktop installed in it, the pulled image does not match with the operating system's arch.

To overcome the architecture mismatch, we explicitly specified the platform when pulling:

docker pull --platform linux/amd64 registry.redhat.io/3scale-amp2/backend-rhel8:3scale2.12
Enter fullscreen mode Exit fullscreen mode

This successfully pulled the image by using Rosetta 2 emulation on macOS.

Redhat Docker Image Pull

Step 3: Exposing the OpenShift Registry

To make our OpenShift registry accessible:

oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
Enter fullscreen mode Exit fullscreen mode

Step 4: Pushing Images to Internal Registry

We pushed the pulled images to our OpenShift internal registry:

# Get credentials
TOKEN=$(oc whoami -t)
REGISTRY=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')

# Login to registry
docker login -u kubeadmin -p $TOKEN $REGISTRY

# Tag and push
docker tag registry.redhat.io/3scale-amp2/backend-rhel8:3scale2.12 $REGISTRY/[namespace]/amp-backend:2.12
docker push $REGISTRY/[namespace]/amp-backend:2.12
Enter fullscreen mode Exit fullscreen mode

Step 5: Updating ImageStreams

We updated the imagestream to point to our locally pushed image:

oc tag $REGISTRY/[namespace]/amp-backend:2.12 amp-backend:2.12 --source=docker
Enter fullscreen mode Exit fullscreen mode

This automatically triggered the deployment due to the ImageChange trigger on the deployment config.

Results

After implementing these steps for the backend-listener component, the deployment began successfully (at least for this resource!).

Challenge 4: PVC Access Mode Issues

So we went to the 3Scale self-hosted Admin Portal and checked that still it's not working.

We checked the pods and found out that some of them are having issues.

apicast-production-1-deploy Issue

The error logs show that several deployments are failing because their pods are taking too long to become available (timeout errors):

  • apicast-production-1-deploy: "pods took longer than 1800 seconds to become available"
  • system-sidekiq-1-deploy: "pods took longer than 1200 seconds to become available"
  • system-sphinx-1-deploy: "pods took longer than 1200 seconds to become available"

This typically happens when pods are stuck in a pending or initializing state for too long.

So we checked the logs of the problematic pods and checked the PVC as well.

# Check logs for apicast-production deployment
oc logs apicast-production-1-deploy

# Check logs for system-sidekiq deployment
oc logs system-sidekiq-1-deploy

# Check logs for system-sphinx deployment
oc logs system-sphinx-1-deploy

# Check events for the pending pod
oc describe pod system-app-1-hook-pre
Enter fullscreen mode Exit fullscreen mode

We discovered a storage issue where the system-storage PVC was failing to provision:

oc get pvc
NAME                    STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
backend-redis-storage   Bound     pvc-ss987s5b-026a-4srg-au97-549d8958933a   1Gi        RWO            gp3            53m
mysql-storage           Bound     pvc-72s43210-s033-4c8w-ar53-043bf3kk1496   1Gi        RWO            gp3            53m
system-redis-storage    Bound     pvc-s3r1111d2-4d57-41dd-9066-c488eda666d4   1Gi        RWO            gp3            53m
system-storage          Pending
Enter fullscreen mode Exit fullscreen mode

The error was related to access modes:

failed to provision volume with StorageClass "gp3": rpc error: code = InvalidArgument desc = Volume capabilities MULTI_NODE_MULTI_WRITER not supported. Only AccessModes[ReadWriteOnce] supported.
Enter fullscreen mode Exit fullscreen mode

We fixed it by creating a new PVC with the correct access mode:

# First, delete pods using this PVC
oc delete pod system-app-1-hook-pre

# Back up the current PVC definition
oc get pvc system-storage -o yaml > system-storage-pvc.yaml

# Delete the stuck PVC
oc delete pvc system-storage

# Create a new PVC with the correct settings
oc create -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: system-storage
  namespace: [Namespace]
  labels:
    app: 3scale-api-management
    threescale_component: system
    threescale_component_element: app
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: gp3
EOF
Enter fullscreen mode Exit fullscreen mode

After fixing the PVC issue, restart the deployments:

oc rollout retry dc/system-app
oc rollout retry dc/apicast-production
oc rollout retry dc/backend-listener
oc rollout retry dc/system-sidekiq
oc rollout retry dc/system-sphinx
Enter fullscreen mode Exit fullscreen mode

The PVC issue got fixed, and the system-storage PVC is now correctly bound to a volume.

oc get pvc

NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
backend-redis-storage   Bound     pvc-ss987s5b-026a-4srg-au97-549d8958933a   1Gi        RWO            gp3            53m
mysql-storage           Bound     pvc-72s43210-s033-4c8w-ar53-043bf3kk1496   1Gi        RWO            gp3            53m
system-redis-storage    Bound     pvc-s3r1111d2-4d57-41dd-9066-c488eda666d4   1Gi        RWO            gp3            53m
system-storage          Bound    pvc-2286196a-8885-490s-11c1-654320bd8a5a6   1Gi        RWO            gp3            116s
Enter fullscreen mode Exit fullscreen mode

Challenge 5: Resource Constraints

Even after resolving the PVC issue, pods were still stuck in Pending state due to insufficient resources:

oc describe pod system-app-2-mr25z
Enter fullscreen mode Exit fullscreen mode 
...
Warning  FailedScheduling  2m34s  default-scheduler  0/9 nodes are available: 2 Insufficient cpu, 3 node(s) had untolerated taint {node-role.kubernetes.io/infra: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 6 node(s) had volume node affinity conflict.
Enter fullscreen mode Exit fullscreen mode

We reduced the resource requirements to make the pods fit on the available nodes:

oc patch dc/system-app -p '{"spec":{"template":{"spec":{"containers":[{"name":"system-master","resources":{"requests":{"cpu":"25m","memory":"400Mi"}}},{"name":"system-provider","resources":{"requests":{"cpu":"25m","memory":"400Mi"}}},{"name":"system-developer","resources":{"requests":{"cpu":"25m","memory":"400Mi"}}}]}}}}'
oc patch dc/apicast-production -p '{"spec":{"template":{"spec":{"containers":[{"name":"apicast-production","resources":{"requests":{"cpu":"25m","memory":"128Mi"}}}]}}}}'
oc patch dc/system-sidekiq -p '{"spec":{"template":{"spec":{"containers":[{"name":"system-sidekiq","resources":{"requests":{"cpu":"25m","memory":"250Mi"}}}]}}}}'
oc patch dc/system-sphinx -p '{"spec":{"template":{"spec":{"containers":[{"name":"system-sphinx","resources":{"requests":{"cpu":"25m","memory":"250Mi"}}}]}}}}'
Enter fullscreen mode Exit fullscreen mode

After applying these patches, we restarted the failed components:

# Retry system-sidekiq and system-sphinx deployments
oc rollout retry dc/system-sidekiq
oc rollout retry dc/system-sphinx
Enter fullscreen mode Exit fullscreen mode

That got fixed too!!


➜  oc get services

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
apicast-production   ClusterIP   xxx.xx.xxx.xxx   <none>        8080/TCP,8090/TCP   83m
apicast-staging      ClusterIP   xxx.xx.xxx.xxx    <none>        8080/TCP,8090/TCP   83m
backend-listener     ClusterIP   xxx.xx.xxx.xxx   <none>        3000/TCP            83m
backend-redis        ClusterIP   xxx.xx.xxx.xxx     <none>        6379/TCP            83m
system-developer     ClusterIP   xxx.xx.xxx.xxx    <none>        3000/TCP            83m
system-master        ClusterIP   xxx.xx.xxx.xxx     <none>        3000/TCP            83m
system-memcache      ClusterIP   xxx.xx.xxx.xxx    <none>        11211/TCP           83m
system-mysql         ClusterIP   xxx.xx.xx.xxx     <none>        3306/TCP            83m
system-provider      ClusterIP   xxx.xx.x.xxx      <none>        3000/TCP            83m
system-redis         ClusterIP   xxx.xx.xxx.xxx   <none>        6379/TCP            83m
system-sphinx        ClusterIP   xxx.xx.xxx.xxx   <none>        9306/TCP            83m
zync                 ClusterIP   xxx.xx.xxx.xx     <none>        8080/TCP            83m
zync-database        ClusterIP   xxx.xx.xx.xxx    <none>        5432/TCP            83m
Enter fullscreen mode Exit fullscreen mode 
➜  oc get routes

NAME                         HOST/PORT                                                             PATH         SERVICES             PORT      TERMINATION     WILDCARD
backend                      backend-3scale.apps.[YOUR-DOMAIN].openshiftapps.com                               backend-listener     http      edge/Allow      None
system-admin                 3scale-admin.apps.[YOUR-DOMAIN].openshiftapps.com                                 system-app           3000      edge/Allow      None
system-developer             3scale.apps.[YOUR-DOMAIN].openshiftapps.com                          /developer   system-app           3001      edge/Allow      None
system-master                master.apps.[YOUR-DOMAIN].openshiftapps.com                                       system-app           3002      edge/Allow      None
system-provider              3scale.apps.[YOUR-DOMAIN].openshiftapps.com                                       system-app           3000      edge/Allow      None
zync-3scale-api-hhhjs        api-3scale-apicast-production.apps.[YOUR-DOMAIN].openshiftapps.com                apicast-production   gateway   edge/Redirect   None
zync-3scale-api-phh9n        api-3scale-apicast-staging.apps.[YOUR-DOMAIN].p1.openshiftapps.com                   apicast-staging      gateway   edge/Redirect   None
zync-3scale-master-nhhht     HostAlreadyClaimed                                                                 system-master        http      edge/Redirect   None
zync-3scale-provider-q9hh9   HostAlreadyClaimed                                                                 system-developer     http      edge/Redirect   None
zync-3scale-provider-shh6z   HostAlreadyClaimed                                                                 system-provider      http      edge/Redirect   None
Enter fullscreen mode Exit fullscreen mode

Since the containers are starting up, it should be a matter of minutes before we can access the admin portal.

We were just pod status, and when system-app shows 3/3 ready, tried accessing the admin portal at:

https://3scale-admin.apps.[YOUR-DOMAIN].openshiftapps.com

But then, it is still UNAVAILABLE.

Challenge 6: Missing Service for App Components

Even after all pods were running, the admin portal was not accessible. The issue was that we created routes pointing to a service named "system-app" which didn't exist:

oc get routes

NAME                         HOST/PORT                                                  PATH        SERVICES      PORT    TERMINATION  WILDCARD
system-admin                 3scale-admin.apps.[YOUR-DOMAIN].openshiftapps.com                     system-app    3000    edge/Allow   None
system-developer             3scale.apps.[YOUR-DOMAIN].openshiftapps.com              /developer   system-app    3001    edge/Allow   None
system-master                master.apps.[YOUR-DOMAIN].openshiftapps.com                           system-app    3002    edge/Allow   None
system-provider              3scale.apps.[YOUR-DOMAIN].openshiftapps.com                           system-app    3000    edge/Allow   None
Enter fullscreen mode Exit fullscreen mode 
oc describe service system-app

Error from server (NotFound): services "system-app" not found
Enter fullscreen mode Exit fullscreen mode

We fixed this by creating the missing service:

bash
oc create -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: system-app
  namespace: [NAMESPACE]
  labels:
    app: 3scale-api-management
spec:
  ports:
  - name: provider
    port: 3000
    protocol: TCP
    targetPort: 3000
  - name: developer
    port: 3001
    protocol: TCP
    targetPort: 3001
  - name: master
    port: 3002
    protocol: TCP
    targetPort: 3002
  selector:
    deploymentConfig: system-app
  type: ClusterIP
EOF
Enter fullscreen mode Exit fullscreen mode

Final Result
After working through all these challenges, we finally had a fully operational 3scale deployment:

bash
oc get pods

NAME                        READY   STATUS      RESTARTS      AGE
apicast-production-4-7hh00  1/1     Running     0             2m12s
apicast-staging-1-6hh00     1/1     Running     0             83m
backend-cron-2-6955a        1/1     Running     0             23m
backend-listener-1-5hh00    1/1     Running     0             26m
backend-redis-1-mhh005      1/1     Running     0             57m
backend-worker-2-lr8gb      1/1     Running     0             23m
system-app-3-7ln8g          3/3     Running     0             85s
system-memcache-1-xddig     1/1     Running     0             80m
system-mysql-1-ee4wt        1/1     Running     0             80m
system-redis-1-45hh0        1/1     Running     0             80m
zync-1-l7ghy                1/1     Running     0             80m
zync-database-1-dt3l9       1/1     Running     0             80m
zync-que-1-wwri9            1/1     Running     2 (80m ago)   80m
Enter fullscreen mode Exit fullscreen mode

With all components running, finally, we were able to access the 3scale admin portal and begin configuring our APIs.

3Scale Api Management On Rosa Intro

Verify Deployment 

# Check all pods are running
oc get pods

# Expected output should show all pods in Running or Completed state:
# - system-app-X-XXXXX (3/3 Running)
# - apicast-production-X-XXXXX (1/1 Running)
# - apicast-staging-X-XXXXX (1/1 Running)
# - system-sidekiq-X-XXXXX (1/1 Running)
# - system-sphinx-X-XXXXX (1/1 Running)
# - backend-* pods (1/1 Running)
# - system-mysql-X-XXXXX (1/1 Running)
# - system-redis-X-XXXXX (1/1 Running)
# - zync-* pods (1/1 Running)
Enter fullscreen mode Exit fullscreen mode

Key Lessons Learned

  1. DNS Resolution is Critical: Ensure your OpenShift cluster can resolve external registry hostnames before attempting deployments that rely on them.

  2. Architecture Awareness: When working with enterprise container images on ARM-based development machines, be explicit about architecture requirements using the --platform flag.

  3. Manual Image Mirroring: In restricted environments, manually pulling and pushing images to an internal registry is a viable workaround.

  4. ImageStream Mechanics: Understanding how OpenShift's ImageStreams work is essential for troubleshooting deployment issues.

  5. Network Policies: In enterprise environments, network policies may restrict access to external registries, requiring coordination with network administrators.

Common Issues and Solutions

  • Pod stuck in Pending: Usually resource constraints - reduce resource requests
  • PVC mounting issues: Check storage class and access modes
  • Routes not working: Ensure services exist and selectors match pods
  • Application not available: Create missing system-app service
  • Database connection issues: Check that system-mysql pod is running and accessible

Final Verification Checklist

  • All pods are in Running state (except completed deployment/hook pods)
  • Routes are accessible and don't show "Application not available"
  • Can log into Admin Portal successfully
  • Can access Developer Portal
  • Master Portal is accessible (if needed)
  • Default passwords have been changed

Once we successfully access the 3scale portals, we can begin migrating the existing 3scale components from another environment or start adding the new components in 3scale.

A. Backend APIs

  • API definitions and configurations
  • Authentication settings
  • Rate limiting rules

B. Products (API Products)

  • Product configurations
  • Application plans
  • Pricing rules
  • Methods and metrics

C. Applications

  • Application keys and secrets
  • Application plans assignments
  • Usage statistics (if needed)

D. Accounts and Users

  • Developer accounts
  • Admin users
  • Access permissions

E. Policies

  • Custom policies
  • Policy chains
  • Configuration settings

F. Developer Portal

  • Custom pages and templates
  • Documentation
  • CMS content

Conclusion

Deploying complex solutions like 3scale API Management in restricted network environments or across architecture boundaries presents unique challenges. By understanding the underlying issues and implementing a systematic approach to manually mirror images, we were able to overcome these obstacles.

While this process requires more manual effort than a standard deployment, it demonstrates the flexibility of OpenShift's container management capabilities and provides a path forward for deployments in environments with similar restrictions.

Top comments (0)