placeholder
thoughts and learnings in software engineering by Rotem Tamir

Creating Terraform-like configuration languages with HCL and Go

PCL: The Pizza Configuration Language

In the past year and a half, I’ve been working on Atlas, a database schema management tool that we’re developing at Ariga. As part of this effort, I worked on implementing the infrastructure for the Atlas DDL, a data definition language that is the basis for Atlas’s declarative style workflow for managing database schemas.

The Atlas language is based on HCL, a toolkit for creating configuration languages with a neat and simple information model and syntax. HCL was created at HashiCorp and used in popular tools such as Terraform and Nomad. We chose HCL as the basis of our configuration language for multiple reasons:

  • It has a base syntax that is clear and concise, easily readable by humans and machines.
  • Popularized by Terraform and other projects in the DevOps / Infrastructure-as-Code space, we thought it would feel familiar to practitioners which are the one of the core audiences for our tool.
  • It’s written in Go, making it super easy to integrate with the rest of our codebase at Ariga.
  • It has great support for extending the basic syntax into a full-blown DSL using functions, expressions and context variables.

PCL: The Pizza Configuration Language

In the rest of this post, we will demonstrate how to create a basic configuration language using HCL and Go. To make this discussion entertaining, let’s imagine that we are creating a new PaC (Pizza-as-Code) product that lets users define their pizza in simple HCL-based configuration files and send them as orders to their nearby pizza place.

Orders and Contacts

Let’s start building our PaC configuration language by letting users define where they want their pizza delivered and who is the hungry contact waiting for the pizza to arrive. We’re aiming for something like:

contact {
  name  = "Sherlock Holmes"
  phone = "+44 20 7224 3688"
}
address {
  street  = "221B Baker St"
  city    = "London"
  country = "England"
}

To capture this configuration, we will define a Go struct Order with sub-structs for capturing the Contact and Address:

type (
	Order struct {
		Contact *Contact `hcl:"contact,block"`
		Address *Address `hcl:"address,block"`
	}
	Contact struct {
		Name  string `hcl:"name"`
		Phone string `hcl:"phone"`
	}
	Address struct {
		Street  string `hcl:"street"`
		City    string `hcl:"city"`
		Country string `hcl:"country"`
	}
)

The Go HCL codebase contains two packages with a fairly high-level API for decoding HCL documents into Go structs: hclsimple (GoDoc) and gohcl (GoDoc). Both packages rely on the user supplying Go struct field tags to map from the configuration file to the struct fields.

We will start the example by using the simpler one, with the surprising name, hclsimple:

func TestOrder(t *testing.T) {
	var o Order
	if err := hclsimple.DecodeFile("testdata/order.hcl", nil, &o); err != nil {
		t.Fatalf("failed: %s", err)
	}
	require.EqualValues(t, Order{
		Contact: &Contact{
			Name:  "Sherlock Holmes",
			Phone: "+44 20 7224 3688",
		},
		Address: &Address{
			Street:  "221B Baker St",
			City:    "London",
			Country: "England",
		},
	}, o)
}

Pizza sizes and toppings (using static values)

Next, let’s add the ability to order actual pizzas in our PaC application. To describe a pizza in our configuration language users should be able to do something like:

pizza {
  size = XL
  count = 1
  toppings = [
    olives,
    feta_cheese,
    onions,
  ]
}

Notice that to make our API more explicit, users do not pass string values to the size or toppings field, and instead they use pre-defined, static identifiers (called “variables” in the HCL internal API) such as XL or feta_cheese.

To support this kind of behavior, we can pass an hcl.EvalContext (GoDoc), which provides the variables and functions that should be used to evaluate an expression.

To construct this context we’ll create this ctx() helper function:

func ctx() *hcl.EvalContext {
	vars := make(map[string]cty.Value)
	for _, size := range []string{"S", "M", "L", "XL"} {
		vars[size] = cty.StringVal(size)
	}
	for _, topping := range []string{"olives", "onion", "feta_cheese", "garlic", "tomatoe"} {
		vars[topping] = cty.StringVal(topping)
	}
	return &hcl.EvalContext{
		Variables: vars,
	}
}

To use it we need to add the pizza block to our top level Order struct:

type (
    Order struct {
        Contact *Contact `hcl:"contact,block"`
        Address *Address `hcl:"address,block"`
        Pizzas  []*Pizza `hcl:"pizza,block"`
    }
    Pizza struct {
        Size     string   `hcl:"size"`
		Count    int      `hcl:"count,optional"`
		Toppings []string `hcl:"toppings,optional"`
    }
	// ... More types ...
)

Here’s our pizza block read using ctx() in action:

func TestPizza(t *testing.T) {
	var o Order
	if err := hclsimple.DecodeFile("testdata/pizza.hcl", ctx(), &o); err != nil {
		t.Fatalf("failed: %s", err)
	}
	require.EqualValues(t, Order{
		Pizzas: []*Pizza{
			{
				Size: "XL",
				Toppings: []string{
					"olives",
					"feta_cheese",
					"onion",
				},
			},
		},
	}, o)
}

How many pizzas to order? (Using functions in HCL)

The final conundrum in any pizza delivery order is of course, how many pizzas to order. To help our users out with this riddle, let’s level up our DSL and add the for_diners function that will take a number of diners and calculate for the user how many pizzas should be ordered. This will look something like:

pizza {
  size     = XL
  count    = for_diners(3)
  toppings = [
    tomato
  ]
}

Based on the universally accepted heuristic that one should order 3 slices per diner and round up, we can register the following function into our EvalContext:

func ctx() *hcl.EvalContext {
    // .. Variables ..
	
    // Define a the "for_diners" function.
    spec := &function.Spec{
        // Return a number.
        Type: function.StaticReturnType(cty.Number),
        // Accept a single input parameter, "diners", that is not-null number.
        Params: []function.Parameter{
            {Name: "diners", Type: cty.Number, AllowNull: false},
        },
        // The function implementation.
        Impl: func (args []cty.Value, _ cty.Type) (cty.Value, error) {
            d := args[0].AsBigFloat()
            if !d.IsInt() {
                return cty.NilVal, fmt.Errorf("expected int got %q", d)
            }
            di, _ := d.Int64()
            neededSlices := di * 3
            return cty.NumberFloatVal(math.Ceil(float64(neededSlices) / 8)), nil
        },
    }
    return &hcl.EvalContext{
        Variables: vars,
        Functions: map[string]function.Function{
          "for_diners": function.New(spec),
        },
    }
}

Testing the for_diners function out:

func TestDiners(t *testing.T) {
    var o Order
    if err := hclsimple.DecodeFile("testdata/diners.hcl", ctx(), &o); err != nil {
      t.Fatalf("failed: %s", err)
    }
	// For 3 diners, we expect 2 pizzas to be ordered.
    require.EqualValues(t, 2, o.Pizzas[0].Count)
}

Wrapping up

With these features, I think we can call it a day for this prototype of the world’s first Pizza-as-Code product. As the source code for these examples is available on GitHub under an Apache 2.0 license, I truly hope someone picks this up and builds this thing!

In this post we reviewed some basic things you can do to create a configuration language for your users using HCL. There’s a lot of other cool features we built into the Atlas language (such as input variables, block referencing and block polymorphism), so if you’re interested so if you’re interested in reading more about it feel free to ping me on Twitter.