Stencil

Note: This was written about commit 60603dba1e7c9076f8c74724b7d8a6e082084320 of the Stencil repo. Parts mentioned may have changed since this was written.

What Is This Project

Stencil was a challenge to build an application like Cookiecutter, and improve on it. I wanted to continue to use Cobra like in the BoxUp project, as I feel a CLI tool made the most sense (also I really like it!).

What Made Me Want To Do This Project

I felt that Cookiecutter was good, but didn’t support complex json objects or an alternative method for entering overrides other than the CLI. I wanted complex json objects so I could separate variables into appropriate sections, for example {user: {firstname: "Chris", surname: "Greaves"}} would allow the variable “.user.firstname”. I then wanted an alternative for entering overrides that would be easier for bigger project templates. The issue I found was having to sit there going through CLI prompts one after the other and making a mistake, the only way to fix it would be to start again. I wanted a local Web interface you could look at and edit.

I also wanted to set this up for some extra features for me to work on in the future. Cookiecutter is missing a VS-Code extension and a store for common templates (like Docker Hub for example). Although these would be separate projects, they would only exist because of this project.

What Was The Original Design and How Did It Change Over The Course Of The Project

At the beginning, all I wanted to do was create an application like Cookiecutter with the ability to pull down from github (I didn’t realise that Cookiecutter already had this feature until quite far into the development). I created a quick list of features that I wanted to start with ,knowing that I would want to add to it in the future. I wanted to have the Web interface and ability to specify the output directory of the template as part of the first release, however after several large refactors and a large amount of time I decided to reduce the MVP to lose some of these features. In the end I settled for just having the git integration and a working template engine.

Being able to use the complex json objects for the variables was extremely easy with the built-in template engine in Go. It also supports lots of other features built-in known as actions. See here for a list of the actions that come with the template lib.

What I Learnt

A ridiculous amount!

As part of this project, I wanted to learn and make use of the Unit Testing built into Golang. This lead to a few restructures throughout its development. I first started building the application like I had with BoxUp, but found unit testing to be extremely difficult given that I had no way of faking out dependencies to force them to behave how I wanted them to. This is where I began to learn Testify. Testify adds a number of packages to help with unit testing, and most importantly mocking.

Using a combination of Testify and a tool called Mockery, I was able to create a number of mock objects to help me test all possible scenarios. To do this I needed to split out my code and create interfaces. To show an example of this, I will use the “fetch” package from Stencil that can be found here. (It doesn’t have any interfaces or mocks in the source code as I don’t test the cmd code directly, but thought it would be good to show how you could do it.) This package contains 4 methods used to check and get templates where available. These methods include; IsPath, IsGitUrl, PullTemplate and IsGitInstalled. From this, the parameters they need and what they return, I can create a interface that I can mock out.

Below is what this interface would look like:

type Fetcher interface {
    IsPath(input string) bool
    IsGitInstalled() bool
    IsGitUrl(input string) bool
    PullTemplate(repo string) (string, error)
}

Now in order to use that interface, we need to change the code to rely on an object having the methods on it instead of calling the methods on the package directly. Below I have shown how I would do this.

package fetch

type TemplateFetcher struct {
    // Any properties you want on the object, if any.
}

func New() (TemplateFetcher, error) {
    // Setup your object and return it or return errors that occurred
    return TemplateFetcher{}, nil
}

func (f *TemplateFetcher) IsPath(input string) bool{
    // Do Something
}

func (f *TemplateFetcher) IsGitInstalled() bool{
    // Do Something
}

func (f *TemplateFetcher) IsGitUrl(input string) bool{
    // Do Something
}

func (f *TemplateFetcher) PullTemplate(repo string) (string, error){
    // Do Something
}

So instead of calling the methods straight from the package like fetch.IsPath(input), we need the TemplateFetcher object, like so:

fetcher := fetch.New()

result := fetcher.IsPath(input)

Now we have the dependency as an object, we need to refactor the code that needs this dependency to accept it rather than create it itself. There are a number of ways of doing this, you can either pass it into the methods that need it or pass it in when creating the object itself. Below is an example of passing it into a method:

func GetTemplate(fetcher fetch.Fetcher, input string) (string, error) {
    if fetcher.IsPath(input) {
        // Do Something
    } else {
        return nil, errors.New("Is not a path")
    }
}

Notice how I used the interface instead of the struct for the parameter. This is so when it comes to mocking it out, we can create a mock object that also satisfies that interface. Now all we need to do is create that mock object to pass in instead. To do this I used the Mockery tool I mentioned earlier. Once you have downloaded the tool and added it to your PATH variable, all you have to do is run mockery -name=Fetcher.

This will create the file /mocks/Fetcher.go and will look something like this:

package mocks

import fetch "github.com/Chris-Greaves/stencil/fetch"
import mock "github.com/stretchr/testify/mock"

// Fetcher is an autogenerated mock type for the Fetcher type
type Fetcher struct {
	mock.Mock
}

// IsPath provides a mock function with given fields:
func (_m *Fetcher) IsPath(input string) bool {
	ret := _m.Called()

	var r0 bool
	if rf, ok := ret.Get(0).(func() bool); ok {
		r0 = rf()
	} else {
		if ret.Get(0) != nil {
			r0 = ret.Get(0).(bool)
		}
	}

	return r0
}

We now have everything we need to do mock out this dependency and test the code with want.

Below is a test that checks what happens if the input isn’t a path:

func Test(t *testing.T) {
	mockFetcher := new(mocks.Fetcher)
    mockFetcher.On("IsPath", "BadPath").Return(false)
    
    result, err := GetTemplate(mockFetcher, "BadPath")

    assert.Error(t, err)
}

Despite all the mocking and unit testing, I also learnt about another new package called gabs. I think the README for gabs sums in up best:

Gabs is a small utility for dealing with dynamic or unknown JSON structures in golang. It’s pretty much just a helpful wrapper around the golang json.Marshal/json.Unmarshal behaviour and map[string]interface{} objects. It does nothing spectacular except for being fabulous.

As it mentions, this is nothing other than a helpful wrapper around the existing json packages in the standard lib. That means if I wanted to remove this dependency I could quite easily by moving the code over to rely on the lib packages completely. However I don’t think that will be necessary as the package is small and why bother trying to “re-invent the wheel” as it were.

I used this package to help with the handling of unknown json from the stencil.json file in the templates. I created a new object called Conf that has a property called raw which is a gabs container. This allowed me to create methods for an object I control, which in itself is a wrapper around the gabs object. This makes it easier when it comes to mocking, as I can built my own interface for the config handling.

// Conf encompasses the anonymous json object for a template config.
type Conf struct {
	raw *gabs.Container
}

I then added methods to the object to handle setting values and retrieving the object as an interface to be passed into the Template Engine. For example:

// SetValues will take an array of settings to put each one into the Conf. If the setting already exists it will update the value, else it will add the new setting.
func (c *Conf) SetValues(settings []Setting) error {
	for _, setting := range settings {
		_, err := c.raw.SetP(setting.Value, setting.Name)
		if err != nil {
			return err
		}
	}

	return nil
}

What I struggled with

So there were a number of things that a struggled with while developing this application, but the most notable one is definitely how to structure the application. I struggled with this so much, that even now I’m still not sure that it is correct. Its always hard to know how to specifically structure something before you have any idea what you are actually doing, but there is usually a common or community accepted way of doing it. However when it comes to Golang, I have seen a lot of different open source projects doing it different.

Ignoring all the Workspace structure and all that (for information on that see this blog post), I want to focus on the actual app folder. At the beginning it was simple, I knew it was going to be a cli and that I was going to use cobra. So I used the cobra code generation to create a new project with all the boiler plate code added in already. However the difficulty came when splitting the logic out to separate packages for unit testing. Originally I wrote most of the code in the cmd/root.go file, as this was the quickest and easiest place. But I know that having all the logic in one colossal file is an anti-pattern, so knew I needed to start separating the code out into single “concerns”. That’s when I decided to create the fetch and engine packages.

These package took the logic for Getting the template and executing the template with values and separated them ready for isolated unit testing. However when starting to write the tests I realised I had to jump through hoops to get test data in place just to test the smallest parts of the methods, that’s where I started looking into Testify and Mocking. Coming from using Moq in C# I know the power of mocking and start refactoring all my code to be interfaceable ready to create the mock objects. The issue with this was I was used to creating interfaces along side the objects using the naming convention IClass, however this isn’t the suggested way of doing it in Golang. Interfaces in Golang should follow the same naming convention as structs. This meant naming interfaces was really difficult as you almost always want to name an interface and struct to explain what it is doing. For example, take a struct called Engine what would you call the Interface? Naturally I want to also call it “Engine”, so I opted to change the structs to be something like DefaultEngine. Although this worked, it felt wrong and led to some quite unreadable code and you couldn’t find any examples of Non-Default Engines. Also because the Interfaces were along side the structs, so were the mocks when they were generated. This seemed weird as the mocks were only used when testing other packages.

Not content with the state of the code, I started looking up how other projects did it and found varied results. Then I happened upon this blog post that explains a common mistake made by Java (and apparently C#) developers moving to Golang. It explains a “Java-like” interface usage where you create an interface alongside and specifically for a struct. This is exactly what I did with Stencil and caused my code base to become considerably more complicated. The solution was to have the package state what it needs rather than the structs state what it CAN do.

Before:

package engine

type Engine interface {
	Execute(template string, values map[string]string) byte[]
}

type DefaultEngine struct {}
func (e DefaultEngine) Execute(template string, values map[string]string) byte[] { 
	// Do stuff
}

...

package workers

func DoWork(e engine.Engine) { 
	e.Execute("{{.Title}}", {Title: "HelloWorld"})
}

After:

package engine

type Engine struct {}
func (e Engine) Execute(template string, values map[string]string) byte[] { 
	// Do stuff
}

...

package main

type Engine interface {
	Execute(template string, values map[string]string) byte[]
}

func main() {
	DoWork(new(engine.Engine))
}

func DoWork(e Engine) {
	e.Execute("{{.Title}}", {Title: "HelloWorld"})
}

So now I understood how interfaces were supposed to work, the structure became a lot clearer. Interfaces should be used as a way for a method to say “This is what you need to be able to do”, instead of being “You have to have this contract”. Having the interfaces specified with the code that uses it, means that mocks are generated along side the code where the mocks are needed. Now each package follows this similar pattern:

Package
 ├─mocks
 │ └─dependency.go
 ├─code.go
 └─code_test.go

What Improvements Do I Want To Make / What Are The Chances Of That Happening

There are so many changes and improvements I want to make to this project.

Firstly I want to improve the useability of Stencil a lot. It currently only has one command that has very little parameters and thus is very restricted for the user. As I mentioned at the start, I want to add a locally hosted Web UI to allow for easy editing of the settings for the template. This is one of the original features I wanted to include and I still plan on adding it. However, there are a couple of other quality of life changes I want to make first. Firstly, I want the ability to specify the output directory where the template will be executed against. This is such a simple feature, but holds so much value to users. I want to add an override for ignoring the .git directory, as there might be some templates that rely on that still being included in the end product. Although this feature doesn’t seem that vital, the fact the code makes much a huge assumption that users wont want it, it seems safe to add a way of turning it off.

I want to create some templates for Stencil. I have created a project template executer with no templates to execute, so why would anyone download it? This is why I need to create some to possibly peak some interest in the project. But I will never be able to create enough templates for Stencil to compare to software like Cookiecutter, which is why I want Stencil to also be able to handle Cookiecutter templates. Cookiecutter has been around for a while and has a lot of support from the community, which has lead to a large amount of project templates being created for it. If Stencil can handle Cookiecutter templates, then it can also benefit from the communities efforts. But why would people choose Stencil over Cookiecutter? That brings me onto my next point.

The next big change I want to make to Stencil actually doesn’t live in the Stencil code itself, but in a separate project. I want to increase the accessability of Stencil by creating a Visual Studio Code extension that uses Stencil in the background to render templates. This combined with the ability to render Cookiecutter templates as well makes the extension quite appealing and might just help to get Stencil on the map.

Summary

I loved working on this project! It was the first project that I could get fully stuck into and dive deep into developing applications in Golang. Not only that, but I saw a large amount of potential in the application I was making which pushed me to keep going and finish it properly. At the end of this project I not only had a finished product, but also the opportunity for other new exciting projects to stem out of it. I Can’t wait to return to Stencil to add the features and see how far I can push this application into the world!