A secure Docker-registry behind a Kong Reverse Proxy - Part I

Time for another techie blog. In this 2-part write-up (part II here) we’ll show you 2 ways to achieve the same goal; running a docker-registry container on a kubernetes cluster behind a Kong api-gateway (functioning as a reverse proxy). Although docker-registry is used, this setup should work on anything that requires a https, ssl or tls pass-through setup.

There’s some info online, but we didn’t find any actual examples of the scenarios we’re covering, so it seemed worthy of a post. Note that you should know about helm charts, Kong, Custom Resource Definitions, and Kong’s CRDs (KongIngress, KongPlugin, …). Otherwise read up before reading on.

Some starting points:

• https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/guides/using-tcpingress.md

• https://discuss.konghq.com/t/tcpingress-how-can-i-use-it/5908

The scope

As there is plenty of online help for using Kong to terminate a secure connections, we will not cover this case. Instead we want to make sure that the pod running on our cluster can handle secure traffic. We know docker-registry can be set up to be used over http too, but where’s the fun in that!

So, what DO we want ?

• pod itself handles secure traffic

• external access via a DNS

• Kong does not terminate, but functions as a pass-through

• It has to work on a Kong that already has a http and https proxy function, and the root path is not available anymore

We believe this is quite a common case: you have a kubernetes cluster, already running a proxy (hopefully set up to reroute all http traffic to https, and acts as a termination point for all the services running on your cluster).

source: https://shadrin.org/nginx/blog/content/microservices-api-gateways-part-2-how-kong-can-help.html

So, you have all of that running, and then you want to add a secure docker registry. Your root domain is probably already pointing to something, or you have a number of subpaths in use. We’ll assume that yourdomain.com is owned by you, and used for this cluster, together with alias www.yourdomain.com.

We also assume that you have already taken care of hooking up cert-manager to get your certificates, so https://www.yourdomain.com/ works just fine (and is pointing to something).

The Goals

With the above setup, we would like to securely access our private docker registry via any of the following URIs:

• registry.yourdomain.com

• (www.)yourdomain.com:5000

• (www.)yourdomain.com/registry

Spoiler alert: we didn’t succeed in the third case, mainly due to the way docker client determines the protocol to use (http vs https), together with the inability of docker-registry to have a context path defined.

Solution 1: registry.yourdomain.com

This solution requires 5 ‘steps’:

1. Ensure Kong registers the subdomain and gets a certificate

2. adapt the container to use tls and the correct host

3. adapt the configuration to enforce https

4. adapt the ingress to use the proper kongIngress

5. create a KongIngress CRD (Custom Resource Definition)

In an overview it looks something like this:

Step 1: subdomain registration and certification

In the Kong helm chart (we’re using 1.7.0, more on that later) you should already have a section defining your domain. Of course you should also create an A Record in your DNS, pointing to the same IP (maybe you have wildcard subdomains configurated already).

# old situation
proxy:
  ingress:
    hosts:
      - yourdomain.com
    tls:
      - hosts:
         - yourdomain.com

# new situation
proxy:
  ingress:
    hosts:
      - yourdomain.com
      - registry.yourdomain.com
    tls:
      - hosts:
         - yourdomain.com
         - registry.yourdomain.com

Check in your logs that the cert-manager is picking it up correctly, and that a new acme challenge is met. Your certificate secret should become valid for both domains.

going to the url (https://registry.yourdomain.com) should at least give you a valid certificate

Step 2: Adapt the Container

Step 2 takes us to the other end of the chain; the container. To allow https protocol on the docker-registry (version 2.7.1) you need to provide the tls configuration.

If you use cert-manager, the certificate name has been configured through your kong helm chart, so you should know its name (proxy.tls[0].secretName). Let’s say it’s ‘mydomain.le.tls’.

# add this line to your custom values.yaml for docker-registry

tlsSecretName: yourdomain.le.tls

Step 3: enforce https

Step 3 is an important one, as we observed the default behaviour being that the registry reverted to http traffic internally. To avoid this from happening, we forced the service to use https. With the new version of Kong (it’s different in older ones, … just upgrade, really 🙂 ). This should have been easy, but there is no service.annotations section in the chart. Let’s not dwell into the impact such a thing has on your CI/CD automated setup, and just show what the service.yaml should contain (to test you can just ‘kubectl edit’ the service) .

metadata:
  annotations:
    konghq.com/protocol: https

Step 4: Adapt the ingress

And still not leaving the docker-registry chart and values.yaml file, the ingress modifications happen in that file as well. Mainly, we want to ensure the tls definition is set up, but also that a KongIngress CRD is used. It hasn’t been created yet, we’ll get to that in step 5.

So, your final modification of the file:

ingress:
  enabled: true
  hosts:
    - registry.yourdomain.com
  annotations:
    kubernetes.io/ingress.class: kong
    configuration.konghq.com: registry-ingress-config
  tls:
    - secretName: yourdomain.le.tls
      hosts:
        - registry.yourdomain.com

Step 5: KongIngress definition

Onto the all-important last step, the KongIngress CRD definition.

apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name: registry-ingress-config
proxy:
  protocols:
  - https
route:
  methods:
  - POST
  - GET
  - PUT
  - PATCH
  - DELETE
  - HEAD
  preserve_host: true
  protocols:
  - https
  regex_priority: 0

It’s pretty self-explanatory, but to recap, we specify the incoming traffic is https only, specify which methods should be routed (pretty much everything), and that the used protocol in the route is https (which is not sufficient by itself, also the service needs that specification).

That’s it. The following should work, over https, from your docker client (e.g. your local pc).

docker pull alpine
docker tag alpine registry.yourdomain.com/alpine
docker push registry.yourdomain.com/alpine

Solution 2: yourdomain.com:5000

As S.R. Hadden famously stated in the movie Contact: “why build one when you can have two at twice the price?”

In the next part of this write-up we will show you step by step how you can use the stream_listen configuration together with TCPIngress CRD to achieve this goal. You gotta love the new stuff 🙂 .

In the meantime, if you have managed to get scenario 3 working (subpathing), definitely let us know in the comments.