{"id":544,"date":"2018-08-14T10:40:46","date_gmt":"2018-08-14T10:40:46","guid":{"rendered":"https:\/\/www.peopleperhour.com\/engineering\/?p=544"},"modified":"2018-08-28T13:26:02","modified_gmt":"2018-08-28T13:26:02","slug":"working-around-the-lack-of-volumes-from-in-kubernetes","status":"publish","type":"post","link":"https:\/\/www.peopleperhour.com\/engineering\/2018\/08\/14\/working-around-the-lack-of-volumes-from-in-kubernetes\/","title":{"rendered":"Working around the lack of volumes-from in Kubernetes"},"content":{"rendered":"

We have been running containerised apps since early 2014 so in theory, moving to Kubernetes should be easy. The problem is that we currently make use of the --volumes-from<\/code> docker flag to share volumes between containers of the same app. For example, a common pattern we use is to put our webapp sourcecode in a data-container and then run both nginx and a app-server like php-fpm or nodejs with the --volumes-from<\/code> flag, referencing the data-container, thereby ensuring all processes have a common view of the sourcecode. This is very useful for things like dynamically generated asset files such as CSS. For example, if the PHP container creates a CSS file, the nginx container can serve it as they share the same volume. Kubernetes doesn’t have --volumes-from<\/code>. If you want a single Pod to have both nginx and php-fpm containers, how do you share data between them in a way where the data is visible at container launch AND can be modified by any container in a manner that the others get the changes immediately?<\/p>\n

One approach is to copy the data you want to share (e.g. sourcecode) to a volume in a initContainers<\/code> so that when the containers start they can mount the volume in order to share it. We use emptyDir<\/code> and this is what it looks like:<\/p>\n

---\r\napiVersion: apps\/v1\r\nkind: Deployment\r\nspec:\r\n  replicas: 2\r\n\r\n  template:\r\n\r\n    spec:\r\n\r\n      initContainers:\r\n\r\n      # Copy the webapp sourcecode into a volume that both nginx and fpm will mount\r\n      - name: init-sourcecode\r\n        image: peopleperhour\/data:{{ version }}\r\n        command: [\"bin\/sh\", \"-c\", \"cp -r \/var\/www\/ourapp\/. \/code && echo 'some other stuff, e.g. set permissions'\"]\r\n        volumeMounts:\r\n        - mountPath: \/code\r\n          name: app-volume\r\n\r\n      containers:\r\n\r\n      - name: fpm\r\n        image: peopleperhour\/fpm\r\n        volumeMounts:\r\n        - name: app-volume\r\n          mountPath: \/var\/www\/ourapp\r\n        - name: php-socket\r\n          mountPath: \/sock\r\n\r\n      - name: nginx\r\n        image: peopleperhour\/nginx\r\n        ports:\r\n        - containerPort: 80\r\n        volumeMounts:\r\n        - name: app-volume\r\n          mountPath: \/var\/www\/ourapp\r\n        - name: php-socket\r\n          mountPath: \/sock\r\n\r\n      volumes:\r\n\r\n      # For our webapp sourcecode, e.g. the executable PHP.\r\n      - name: app-volume\r\n        emptyDir: {}\r\n      # For a unix socket so nginx and fpm can communicate\r\n      - name: php-socket\r\n        emptyDir: {}\r\n<\/pre>\n
<\/p>\n
This mimics the old --volumes-from<\/code> quite well but it makes the startup time of the pod a little slow because if your app is 300M then you’re copying 300M to disk in the init phase for each Pod. This is not a problem for a few Pods but if you run dozens of Pods and if you run CronJobs that run every minute the disk I\/O is noticeable. We found we were running out of AWS EBS disk I\/O credits due to all the moving of bits about.<\/p>\n
One alternative idea we came up with is to use “Docker-outside-of-Docker” (DooD) to bake together the app with the data as a local docker image. This way, the app can start very quickly on subsequent startups<\/em> without needing to copy to disk. DooD is the name given to the technique of mounting the docker socket so you can run docker commands and get the same results as if you ran them directly on the host. In our case, it allow us to run docker build<\/code> from within a container running in Kubernetes. This is what it looks like in a Kubernetes CronJob<\/code>:<\/p>\n
---\r\napiVersion: batch\/v1beta1\r\nkind: CronJob\r\nspec:\r\n  schedule: \"* * * * *\"\r\n  startingDeadlineSeconds: 59\r\n  jobTemplate:\r\n    spec:\r\n      template:\r\n        spec:\r\n          initContainers:\r\n          # Create a local docker image that has php and the sourcecode in it\r\n          # Note: Only do the build IF not already done.\r\n          - name: build-php\r\n            image: docker:18.05\r\n            imagePullPolicy: IfNotPresent\r\n            command: [\"bin\/sh\", \"-c\", \"[ $(docker images -q mylocalimage:{{ version }})\\\" != '' ] || (echo -e \\\"FROM peopleperhour\/data:{{ version }}\\\\nFROM peopleperhour\/fpm\\\\nCOPY --from=0 \/var\/www\/ourapp .\\\" | docker build -t mylocalimage:{{ version }} -)\"]\r\n            volumeMounts:\r\n            - name: docker-sock\r\n              mountPath: \/var\/run\r\n\r\n          containers:\r\n          - name: cron\r\n            image: mylocalimage:{{ version }}\r\n            imagePullPolicy: Never\r\n            command: [\"start-crons.sh\"]\r\n\r\n          volumes:\r\n\r\n          # For doing docker builds within a docker container (Docker-outside-of-Docker)\r\n          - name: docker-sock\r\n            hostPath:\r\n                path: \/var\/run\r\n<\/pre>\n
<\/p>\n
The initContainer<\/code> runs a docker build using a self-created Dockerfile. This creates a local docker image that can be used next time the cron runs. It is not best practice to mount the docker socket into a Pod – it is akin to running in “privileged mode”, so it may not be the best approach for you.<\/p>\n
Notes:<\/p>\n