Caching dependencies with Google Cloud Build and GCS
Cloud Build is a managed CI/CD environment, of which we are happy users. However, our test suite was taking about ten minutes to run, so we thought about speeding things up by caching our dependencies on GCS.
$HOME/.npm) into GCS, by adding two additional steps to our
steps: # # Restore dependencies cache # You probably want to use your own container image extending # this cloud-sdk and installing gsutil # - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:alpine' id: 'restore-cache' entrypoint: '/bin/sh' args: ['-c', './restore-cache'] # Run other tasks here, launching JS and Java tests in parallel. # Backup dependencies into the cache for the next run - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:alpine' waitFor: ['compilation-step'] entrypoint: '/bin/sh' args: ['-c', './save-cache'] options: env: # Point gradle and npm caches to a persistent location - GRADLE_USER_HOME=/home/cache/.gradle - npm_config_cache=/home/cache/.npm volumes: - name: 'cache' path: '/home/cache' timeout: 900s
To make this work, we created a persistent volume mapped to
/home/cache to store the cached dependencies between build steps.
./restore-cache are quite straightforward:
#!/bin/bash # save-cache: Save the gradle and npm cache to GCS. set -e -u tar -czf /tmp/gradle.tgz -C $GRADLE_USER_HOME . echo "$(tar -tvzf /tmp/gradle.tgz | wc -l) files copied from $GRADLE_USER_HOME" tar -czf /tmp/npm.tgz -C $npm_config_cache . echo "$(tar -tvzf /tmp/npm.tgz | wc -l) files copied from $npm_config_cache" echo 'Saving dependencies to gs://my_cache_bucket/' gsutil -q -m cp /tmp/gradle.tgz /tmp/npm.tgz gs://my_cache_bucket/ echo 'Saving timestamp to gs://my_cache_bucket/timestamp' date +%s | gsutil -q cp - gs://my_cache_bucket/timestamp
We compressed both folders into
.tgz files, and you may have noticed the additional timestamp. This helps to avoid accumulating unused dependencies, for example if we change versions of gradle or individual libraries.
#!/bin/bash # restore-cache: Restore the gradle and npm cache from GCS. set -e -u # If there is a cache and the content is not older than a month TIMESTAMP=$(gsutil cat gs://my_cache_bucket/timestamp || echo 0) SECONDS_IN_A_MONTH=2629743 if (( $(date +%s) - $TIMESTAMP < $SECONDS_IN_A_MONTH )); then gsutil -q -m cp gs://my_cache_bucket/gradle.tgz gs://my_cache_bucket/npm.tgz /tmp mkdir -p $GRADLE_USER_HOME $npm_config_cache # copy gradle dependencies echo 'Restoring gradle cache' tar -xzf /tmp/gradle.tgz -C $GRADLE_USER_HOME echo "$(ls -pR $GRADLE_USER_HOME | grep -v / | wc -l) files restored to $GRADLE_USER_HOME" # copy npm dependencies echo 'Restoring npm cache' tar -xzf /tmp/npm.tgz -C $npm_config_cache echo "$(ls -pR $npm_config_cache | grep -v / | wc -l) files restored to $npm_config_cache" else if (( $TIMESTAMP == 0 )); then echo 'Skipping cache restore: timestamp not found at gs://my_cache_bucket/timestamp' else echo 'Skipping cache restore: timestamp at gs://my_cache_bucket/timestamp is older than a month' fi fi
If the timestamp is missing or the cache is too old, we skip the
restore-cache step and recreate the whole thing from scratch.
This process saved us 2 minutes of execution time per build, and it helped us reassess the size of our container images (saved 30s) and how many steps we could run in parallel (around 3 minutes per build). In the end, we could reduce from ten minutes to almost five, but your mileage may vary. Let us know your own results at @koliseoapi.