Tips On How To Save Time During A Static Analysis On Go

For the last two years, our team has been translating various parts of the project into micro services on Go. They are developed by several teams, so we needed to set a strict bar of code quality. To do this, we use several tools, this article will deal with one of them – a static analysis.

Static analysis is the process of automatically checking the source code with the help of special utilities. This article will tell you about its usefulness, briefly describe the popular tools and give instructions on how to implement it. It is worth reading if you have not encountered such tools at all or use them in an unsystematic manner.

In articles on this topic, the term “linter” is often encountered. For us, this is a convenient name for simple tools for static analysis. The task of the linter is to find simple errors and incorrect design.

Why do we need linter?

Working in a team, you most likely perform a code review. Errors missed on the review are potential bugs. Missed the unhandled error – do not get an informative message and look for the problem blindly. The wrong typecasting or turned to nil map – even worse, the binaries will fall from panic.

The above errors can be added to the Code Conventions, but finding them when reading the requester pool is not so easy, because the reviewer will have to read the code. If your head does not have a compiler, some of the problems will still go to battle. In addition, the search for small errors distracts from checking logic and architecture. At a distance, support for this code will become more expensive. We write in a statically typed language, strange not to use it.

Popular Tools

Most utilities for static analysis use go/ast and go/parser packages. They provide functions for parsing the syntax of .go files.

The standard execution thread (for example, for the golint utility) is:

• Loads the list of files from the required packages
• For each file is executed parser parseFile (…) (* ast.File, error)
• Runs a check of supported rules for each file or package
• The check goes through each instruction, for example, like this:

f, err := parser.ParseFile(/* ... */)

ast.Walk(func (n *ast.Node) {
switch v := node.(type) {
case *ast.FuncDecl:
if strings.Contains(v.Name, "_") {
panic("wrong function naming")
}
}
}, f)

In addition to AST, there is a Single Static Assignment (SSA). This is a more complex way of analyzing code that works with the flow of execution, rather than syntactic constructs. In this article we will not consider it in detail, you can read the documentation and take a look at the example of the stack check utility.

Further, only popular utilities that perform useful checks for us will be considered.

Gofmt

This is a standard utility from the go package, which checks for consistency with the style and can automatically fix it. The matching style is a must for us, so gofmt checking is included in all our projects.

Typecheck

Typecheck checks the type match in the code and supports vendor (unlike gotype). Its launch is mandatory to check the compilation but does not give full guarantees.

Go vet

The go vet utility is part of the standard package and is recommended for use by the Go command. Checks for a number of common errors, for example:

• A misuse of Printf and similar functions
• Incorrect build tags
• Comparison function and nil

Golint

Golint is developed by the Go team and checks code based on Effective Go documents and code review comments.

Unfortunately, there is no detailed documentation, but by code, one can understand that the following is checked:

f.lintPackageComment()
f.lintImports()
f.lintBlankImports()
f.lintExported()
f.lintNames()
f.lintVarDecls()
f.lintElses()
f.lintRanges()
f.lintErrorf()
f.lintErrors()
f.lintErrorStrings()
f.lintReceiverNames()
f.lintIncDec()
f.lintErrorReturn()
f.lintUnexportedReturn()
f.lintTimeNames()
f.lintContextKeyTypes()
f.lintContextArgs()

Staticcheck

The developers themselves represent a staticcheck as an improved go vet.

There are many tests, they are divided into groups:

• Misuse of standard libraries
• Problems with multithreading
• Problems with tests
• Useless code
• Performance issues
• Questionable constructions

Gosimple

Specializes in finding designs that are worth simplifying, for example:

Before (source code golint)

func (f *file) isMain() bool {
if f.f.Name.Name == "main" {
return true
}
return false
}

After

func (f *file) isMain() bool {
return f.f.Name.Name == "main"
}

The documentation is similar to staticcheck and includes detailed examples.

Errcheck

The errors returned by the functions cannot be ignored. The reasons are described in detail in the mandatory document.

Effective Go. Errcheck will not miss the following code:

json.Unmarshal(text, &val)
f, _ := os.OpenFile(/* ... */)

Gas

Finds vulnerabilities in the code: hard-coded accesses, SQL injections and the use of unsafe hash functions.

Examples of errors:

// access from all IP addresses
l, err: = net.Listen ("tcp", ": 2000")

// potential sql injection
q: = fmt.Sprintf ("SELECT * FROM foo where name = '% s'", name)
q: = "SELECT * FROM foo where name =" + name

// use another hash algorithm
import "crypto / md5"

Maligned

In Go, the order of the fields in the structures affects the memory consumption. Maligned finds a sub-optimal sort. In this order of fields:

struct {
a bool
b string
c bool
}

The structure will occupy 32 bits in memory due to the addition of empty bits after fields a and c.

If we change the sorting and put two bool fields together, then the structure will only take 24 bits:

Goconst

The magic variables in the code do not reflect the meaning and complicate the reading. Goconst finds literals and numbers that occur in code 2 times or more. Note, often even a single use can be a mistake.

Gocyclo

We consider the cyclomatic complexity of the code to be an important metric. Gocycle shows the complexity of each function. You can display only functions that exceed the specified value.

We chose the threshold value 7 for ourselves because we did not find the code with a higher complexity, which did not require refactoring.

gocyclo -over 7 package/name

The Dead Code

There are several utilities for finding unused code, their functionality may overlap.

• Ineffassign: checks useless assignments

func foo () error {
var res interface {}
log.Println (res)
res, err: = loadData () // the res variable is no longer used
return err
}

• Deadcode: finds unused functions
• Unused: finds unused functions, but does it better than the chaos

func unusedFunc() {
formallyUsedFunc()
}

func formallyUsedFunc() {
}

As a result, unused will point directly to both functions, and the deadcode only on unusedFunc. Due to this, the extra code is deleted in one pass. Also unused finds unused variables and field structures.

• Varcheck: finds unused variables
• Unconvert: finds useless casts

var res int
return int(res) // unconvert error

If there is no task to save on the time of launching checks, it’s best to run them all together. If optimization is necessary, we recommend using unused and unconvert.

How it all works

Running the tools listed above is consistently inconvenient: errors are issued in different formats, execution takes a lot of time. Checking one of our services with the size of ~ 8000 lines of code took more than two minutes. Install utilities, too, will have to separate.

To solve this problem, there are utilities-aggregators, for example, goreporter and gometalinter. Goreporter renders the report in HTML, and the gometalinter writes to the console.

• Gometalinter is still used in some large projects (for example, docker). He can install all utilities with one command, run them in parallel, and format errors with the template. The execution time in the service mentioned above was reduced to one and a half minutes.

Aggregation works only on the exact coincidence of the text of an error, therefore on an output, the repeated errors are inevitable.

In May 2018 a golangci-lint project appeared on the GitHub, which greatly exceeds the gometalinter inconvenience:

• The execution time on the same project was reduced to 16 seconds (8 times)
• There are almost no duplicate errors
• Friendly yaml config
• The nice output of errors with a code line and a pointer to the problem

• Do not need to install additional utilities

Now the speed increase is provided by the re-use of SSA and loader Program, in the future it is also planned to re-use the AST tree, about which we wrote at the beginning of the Tools section.

PACKAGE=package/name docker run --rm -t \
-v $(GOPATH)/src/$(PACKAGE):/go/src/$(PACKAGE) \
-w /go/src/$(PACKAGE) \
roistat/golangci-lint

At the time of writing the article on hub.docker.com there was no image with the documentation, so we made our own, customized according to our notions of convenience. In the future, the configuration will change, so for the production, we recommend that you replace it with your own. To do this, just add the .golangci.yaml file to the root directory of the project (an example is in the golangci-lint repository).

With this command, you can check the entire project. For example, if it is in ~/go/src/project, change the value of the variable to PACKAGE = project. The check works recursively on all internal packages.

Note that this command works correctly only when using a vendor.

Implantation

All of our development services use the docker. Any project runs without the go environment installed.

To run the commands, use the Makefile and add the lint command to it:

lint:
@docker run --rm -t -v $(GOPATH)/src/$(PACKAGE):/go/src/$(PACKAGE) -w /go/src/$(PACKAGE) roistat/golangci-lint

make lint

There is a simple way to block the code with errors from getting into the wizard – create a pre-receive-hook. It will work if:

• You have a small project and little dependencies (or they are in the repository)
• For you, no problem waiting for the git push command to run for a few minutes

Instructions for configuring hooks: Gitlab, Bitbucket Server, Github Enterprise.

In other cases, it is better to use CI and to disallow the code in which there is at least one error. We do just that, adding the launch of the linters before the tests.

Conclusion

The introduction of systematic inspections markedly shortened the review period. However, more importantly, another: now we can discuss the big picture and architecture most of the time. This allows you to think about the development of the project instead of plugging holes.