Gatsby Incremental build for self-hosted environments.

Incremental build functionality is now in Gatsby core. Although it may still be an experimental feature but a pretty significant and much awaited one.

What was the problem?

Every-time you run a Gatsby build, internally the following steps are processed:

• Delete css and html files at the very beginning.
• Source data from an end point.
• Create Pages and Queries.
• Build JS/CSS production files.
• Build HTML files.

For sites with large amount of content this process can be quiet time consuming. Suppose you make changes to 2 nodes in a site that has 20,000 nodes and whenever you run the Gatsby build it will assume changes for all the 20,000 nodes and process the above the steps.

The Solution?

For sites that are hosted on Gatsby cloud the solution to the problem is Gatbsy Cloud builds feature.

But for self-hosted environments since this is cannot not be the solution. Gatsby provides us with environmental variables that helps us to reduce the build time by persisting the .cache and .public directories between the builds.

So every-time you run Gatsby build with the environment variable flag. It compares the page data from previous build and creates a list of page directories that are finally passed to the build process. Which means it will only trigger build for the changed nodes.

The difference between the build time is quite visible. For our site that has no more than 100 nodes, there was a difference of 50 secs in the consequent builds which i think is pretty huge.

The first build will trigger build for all the pages:

On second build only those pages are updated that has the content change.

How to Implement?

1. Upgrade your Gatsby to the latest version:

npm install gatsby@latest

2. Add a flag to your build command in package.json.

"scripts": {
   "develop": "gatsby develop",
   "build": "GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true                  gatsby build --log-pages"
 },

or you can simply use the environment variable in your terminal everytime you run your build command.

To get this working with Netlify you need add the netlify-plugin-gatsby-cache to your netlify.toml config file. You can follow this blog to see how it is implemented.

Something to remember:

1. GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true flag must be used everytime you trigger a new build, otherwise Gatsby will not be able to persist the cache between the builds and you will not get the desired outcome.
2. Any code changes or change to a static query will trigger a new build. The environment variable only takes care of content changes.
3. Also If you add or delete any new content/node, a new build is triggered.

This is all. For more in depth to know how this works you can follow the pull request details here.

P.s:- Incase you have any Queries, you can mention them in the comments section.