A secure Docker-registry behind a Kong Reverse Proxy - Part I
Updated: Jul 14, 2020
Time for another techie blog. In this 2-part write-up (part II here) we’ll show you 2 ways to achieve the same goal; running a docker-registry container on a kubernetes cluster behind a Kong api-gateway (functioning as a reverse proxy). Although docker-registry is used, this setup should work on anything that requires a https, ssl or tls pass-through setup.
There’s some info online, but we didn’t find any actual examples of the scenarios we’re covering, so it seemed worthy of a post. Note that you should know about helm charts, Kong, Custom Resource Definitions, and Kong’s CRDs (KongIngress, KongPlugin, …). Otherwise read up before reading on.
Some starting points:
As there is plenty of online help for using Kong to terminate a secure connections, we will not cover this case. Instead we want to make sure that the pod running on our cluster can handle secure traffic. We know docker-registry can be set up to be used over http too, but where’s the fun in that!
So, what DO we want ?
pod itself handles secure traffic
external access via a DNS
Kong does not terminate, but functions as a pass-through
It has to work on a Kong that already has a http and https proxy function, and the root path is not available anymore
We believe this is quite a common case: you have a kubernetes cluster, already running a proxy (hopefully set up to reroute all http traffic to https, and acts as a termination point for all the services running on your cluster).
So, you have all of that running, and then you want to add a secure docker registry. Your root domain is probably already pointing to something, or you have a number of subpaths in use. We’ll assume that yourdomain.com is owned by you, and used for this cluster, together with alias www.yourdomain.com.
We also assume that you have already taken care of hooking up cert-manager to get your certificates, so https://www.yourdomain.com/ works just fine (and is pointing to something).
With the above setup, we would like to securely access our private docker registry via any of the following URIs:
Spoiler alert: we didn’t succeed in the third case, mainly due to the way docker client determines the protocol to use (http vs https), together with the inability of docker-registry to have a context path defined.
Solution 1: registry.yourdomain.com
This solution requires 5 ‘steps’:
Ensure Kong registers the subdomain and gets a certificate
adapt the container to use tls and the correct host
adapt the configuration to enforce https
adapt the ingress to use the proper kongIngress
create a KongIngress CRD (Custom Resource Definition)
In an overview it looks something like this:
Step 1: subdomain registration and certification
In the Kong helm chart (we’re using 1.7.0, more on that later) you should already have a section defining your domain. Of course you should also create an A Record in your DNS, pointing to the same IP (maybe you have wildcard subdomains configurated already).
# old situation proxy: ingress: hosts: - yourdomain.com tls: - hosts: - yourdomain.com # new situation proxy: ingress: hosts: - yourdomain.com - registry.yourdomain.com tls: - hosts: - yourdomain.com - registry.yourdomain.com
Check in your logs that the cert-manager is picking it up correctly, and that a new acme challenge is met. Your certificate secret should become valid for both domains.
going to the url (https://registry.yourdomain.com) should at least give you a valid certificate
Step 2: Adapt the Container
Step 2 takes us to the other end of the chain; the container. To allow https protocol on the docker-registry (version 2.7.1) you need to provide the tls configuration.
If you use cert-manager, the certificate name has been configured through your kong helm chart, so you should know its name (proxy.tls.secretName). Let’s say it’s ‘mydomain.le.tls’.
# add this line to your custom values.yaml for docker-registry tlsSecretName: yourdomain.le.tls
Step 3: enforce https
Step 3 is an important one, as we observed the default behaviour being that the registry reverted to http traffic internally. To avoid this from happening, we forced the service to use https. With the new version of Kong (it’s different in older ones, … just upgrade, really 🙂 ). This should have been easy, but there is no service.annotations section in the chart. Let’s not dwell into the impact such a thing has on your CI/CD automated setup, and just show what the service.yaml should contain (to test you can just ‘kubectl edit’ the service) .
metadata: annotations: konghq.com/protocol: https
Step 4: Adapt the ingress
And still not leaving the docker-registry chart and values.yaml file, the ingress modifications happen in that file as well. Mainly, we want to ensure the tls definition is set up, but also that a KongIngress CRD is used. It hasn’t been created yet, we’ll get to that in step 5.
So, your final modification of the file:
ingress: enabled: true hosts: - registry.yourdomain.com annotations: kubernetes.io/ingress.class: kong configuration.konghq.com: registry-ingress-config tls: - secretName: yourdomain.le.tls hosts: - registry.yourdomain.com
Step 5: KongIngress definition
Onto the all-important last step, the KongIngress CRD definition.
apiVersion: configuration.konghq.com/v1 kind: KongIngress metadata: name: registry-ingress-config proxy: protocols: - https route: methods: - POST - GET - PUT - PATCH - DELETE - HEAD preserve_host: true protocols: - https regex_priority: 0
It’s pretty self-explanatory, but to recap, we specify the incoming traffic is https only, specify which methods should be routed (pretty much everything), and that the used protocol in the route is https (which is not sufficient by itself, also the service needs that specification).
That’s it. The following should work, over https, from your docker client (e.g. your local pc).
docker pull alpine docker tag alpine registry.yourdomain.com/alpine docker push registry.yourdomain.com/alpine
Solution 2: yourdomain.com:5000
As S.R. Hadden famously stated in the movie Contact: “why build one when you can have two at twice the price?”
In the next part of this write-up we will show you step by step how you can use the stream_listen configuration together with TCPIngress CRD to achieve this goal. You gotta love the new stuff 🙂 .
In the meantime, if you have managed to get scenario 3 working (subpathing), definitely let us know in the comments.