GLSL Shader Live-Reloading

Live-reloading shaders can improve productivity in rendering projects by an order of magnitude or two, so definitely worth your time to set up. In this article, I’ll describe a design for a live-reloading system that is unintrusive to your code, and simplifies a lot of resource binding.

The Interface

The usage is as follows:


ShaderSet shaders;

// prepended to shaders as "#version 450"
shaders.SetVersion("450");

// File prepended to shaders (after #version)
shaders.SetPreambleFile("preamble.glsl");

// detects shader stage from extensions
// returns pointer to a program in the set
GLuint* program = shaders.AddProgramFromExts({"foo.vert", "foo.frag"});

while (running)
{
    // polls timestamps and reloads any shaders
    // (programs auto-updated through pointers)
    shaders.UpdatePrograms();

    glUseProgram(*program);
    render();
}

Separation of Concerns

Since the ShaderSet returns a plain GLuint* (which points to to the program), the code making use of the program is separated from the ShaderSet system, since it doesn’t have to know about any ShaderSet specific interface details. This is in contrast to creating a boilerplate “Shader” class and passing that around, which makes your everyday OpenGL code strongly tied to your custom shader system.

Shader Stage Detection

Since OpenGL needs to know the shader type (vertex, fragment, geometry, etc) to create a shader object, I rely on a convention for the shader extension. The convention is as follows:

  • Vertex shader: .vert
  • Fragment shader: .frag
  • Geometry shader: .geom
  • Tessellation Control shader: .tesc
  • Tessellation Evaluation shader: .tese
  • Compute shader: .comp

Update Method

In my current design, I poll timestamps at every frame. This might become a performance problem if you have hundreds or thousands of shaders, but my average research codebases don’t, so polling keeps things simple. In contrast, getting file change callbacks on Windows is very complicated, and hard to make robust. I poll the timestamp using Win32 GetFileTime, and stat can be used for Unixes.

Compile/Link Errors

If a shader fails to compile, I output the error to the console and just keep on trucking. The user can then fix the compile errors through live-reloading.

If the user keeps running code with a shader that failed to compile, it’s likely that they’ll get spammed with errors from every call that uses that bad shader. For this reason, the ShaderSet automatically sets bad shaders to 0 until the compilation is fixed. This makes it possible to  detect and handle failed compilation from user code. For example:

if (*program) {
    glUseProgram(*program);
    render();
}

Resource Binding

In order to link resources like vertex attributes, uniforms, textures, and buffers to the shader, I rely exclusively on in-shader specification of locations using layout directives. This removes the need to query or bind all the attribute/uniform locations every time the programs are re-linked, which removes a lot of tedious binding code. Also, it makes it easier to swap between shaders that use the same inputs.

Synchronizing Bindings

One major drawback of in-shader specification of locations is the risk of C++ and GLSL going out of sync, and the risk of introducing “magic numbers” for locations. In order to counter this, I define a “preamble” file, which contains code that is prepended to all compiled shaders. This file contains all binding locations. A preamble file might be as follows:


#ifndef PREAMBLE_GLSL
#define PREAMBLE_GLSL

#define POSITION_ATTRIB_LOCATION 0
#define TEXCOORD_ATTRIB_LOCATION 1
#define NORMAL_ATTRIB_LOCATION 2

#define MVP_UNIFORM_LOCATION 0

#define DIFFUSE_TEXTURE_BINDING 0

#endif // PREAMBLE_GLSL

The nice thing is that this preamble is both valid GLSL and C++. Since my GLSL shaders have access to it, they can set their bindings using them, for example as follows:


layout(location = POSITION_ATTRIB_LOCATION)
    in vec4 Position;
layout(location = TEXCOORD_ATTRIB_LOCATION)
    in vec2 TexCoord;
layout(location = NORMAL_ATTRIB_LOCATION)
    in vec3 Normal;

layout(location = MVP_UNIFORM_LOCATION)
    uniform mat4 MVP;

layout(binding = DIFFUSE_TEXTURE_BINDING)
    uniform sampler2D Diffuse;

From there, my C++ code can also include preamble.glsl, and can thus also access the bindings. For example:


#include "preamble.glsl"

// binding attributes
glEnableVertexArrayAttrib(vao, POSITION_ATTRIB_LOCATION);

// binding uniforms
glProgramUniformMatrix4fv(*program, MVP_UNIFORM_LOCATION, 1, 0, mvp);

// binding textures
glBindTextures(DIFFUSE_TEXTURE_BINDING, 1, &diffuseTex);

If you’re careful, you can even put simple structs inside the preamble, which will also compile as C++ if you’ve defined types like vec4 and mat4 on the C++-side. If you’re even more careful, you can also put simple functions to share code between C++ and GLSL. This works especially well with libraries designed to work like GLSL, like glm.

I’d like to emphasize that the static nature of these bindings is not a weakness. By establishing conventions about the input locations, it’s easy to plug in new shaders by using only the inputs you want, and the C++ code you have to write to bind resources is both simplified (no need to query and re-bind constantly) and more efficient (less overhead of switching bindings.)

Implementation

You can find a sample implementation here:
https://github.com/nlguillemot/ShaderSet

Advertisements

4 comments

  1. Mihai Sebea

    for file watching there is the https://github.com/apetrone/simplefilewatcher
    it’s not great but it gets the jobs done
    haven’t had many issues with it.

    Also ..what happens if you need to add a secondary uv channel let’s say ? i guess the c++ code needs to be recompiled . Or you define in the preamble all supported bindings ?

    It would be interesting to see a higher level integration .. how do “materials” respond to shader changes. what happens if the resources are reloaded as well and now you have more or less vertex attributes and samplers?

    All in all hot reloading is the future. i have worked on a small system that would reload everything even the lua scripts used to drive the whole prototype and MAN it was FAST and AWESOME . would uses again 10/10 !

    Like

    • nlguillemot

      I wanted this to be standalone, so didn’t want to use a third party file watching library. Though might be useful to improve performance if necessary.

      Yeah, if you want to add a new attribute, you put all your bindings in the preamble and recompile your cpp code. Ideally, make your cpp code manipulate generic lists of attributes and textures so this isn’t necessary, then the bindings exist only script side. That’s one way to implement materials and differing vertex formats.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s