Go mod's lesser known features

Modules are how Go manages dependencies. A module is a collection of packages that are released, versioned, and distributed together. Each package within a module is a collection of source files in the same directory that are compiled together.

In this post, we will explore Go mododules design and learn how they support supply chain security. You can find the go mod documentation here: https://golang.org/ref/mod

Contents

Notes:

The statements here reference Go 1.17 and not all applies to older versions.

This post is accompanied by a talk at PackageCon2021. The video will be added once available.

Download the slides

Anatomy of a go.mod file

// Sample go.mod file
module github.com/org/module         // module name

require (                            // module dependencies
  github.com/foo/bar  v0.1.2
  github.com/cow/moo  v1.2.3
  mydomain.com/gopher v0.2.3-beta1
)

Go has restrictions on a module name which we will talk about later. The version is a specific and minimum version for your project. Note, there are no version ranges, only a single Semantic Version. As Go processes the dependency tree, it selects the maximum of the versions found. In doing so, MVS can create a reproducible dependency tree without a lockfile.

Minimum Version Selection

Golang uses the MVS (minimum version selection) ^{1 2} algorithm to select dependency versions. This deterministic algorithm has nice properties for reproducible builds and avoids the NP-complete runtime complexity. There is no need for a SAT solver because the dependency selection problem is constrained.

At its heart, hand waving away some details, MVS is a breadth-first traversal across modules and versions. The tree is defined by go.mod files for a module, its dependencies, its dependencies-dependencies, and so on.

The minimum terminology is due to the idea that this is a minimal implementation for a lockfile free, deterministic dependecy management system.

Directives in go.mod

The go.mod file has a number of directives for controlling versions and dependencies.

module example.com/my/thing

go 1.16

require example.com/other/thing v1.0.2
require example.com/new/thing/v2 v2.3.4
exclude example.com/old/thing v1.2.3
replace example.com/bad/thing v1.4.5 => example.com/good/thing v1.4.5
retract [v1.9.0, v1.9.5]

• go - sets the minimum Go syntax version
• require - specify direct module dependencies
• exclude - prevents a dependency version
• replace - substitutes without renaming
• retract - minor and patch versions of this module

//Deprecated can be used for major version of your modules. You add a comment and tag a new release.

// Deprecated: use example.com/mod/v2 instead.
module example.com/mod

As of Go 1.17, there are two require blocks, one for direct and indirect each, to support lazy loading. ³

Environment variables

Go supports a number of environment variables for controlling how modules and module aware commands ⁴ work. The following table contains the variables referenced in later sections. For a full list, more details, and examples, see: ^{5 6}

You can use go env to see the current settings.

Hashes and the go.sum file

When the go command downloads a module, it computes a cryptographic hash and compares it with a known value to verify the file hasn’t changed since it was first downloaded. Modules store these hashes in a go.sum file and the Go command verifies they match. Go also stores these hashes in mode module cache and will compare them with a global database. ⁸

Local module cache

Go maintains a shared module cache on your local system. ⁹ This is where downloaded modules are stored. The location is determined by the GOMODCACHE variable. Module code is read-only by default to prevent local modifications and “it works on my machine” issues. The shared cache also contains prebuilt artifacts. All of this means that multiple projects on your machine can reuse the same downloaded and preprocessed packages.

Global services modules and hashes

The Go team maintain global proxies for sumdb, cachedb, and global hash integrity and revocation checks. ¹⁰ The checksum database can be used to detect misbehaving origin and proxy servers. It has a merkel tree transparency log for hashes powered by the Trillian project. The cache database proxies public modules and will maintain copies even if the origin server removes them.

The Go team has taken privacy seriously. These services record very minimal information. You can read the privacy statemnt for sum.golang.org/privacy for details. Their communications in issues on GitHub reflects this. For example, only limited auth features have been enabled, because they are being careful in trying to maintain privacy in proxy.

Module naming

Go has a number of module naming rules. These are partially by design, in using code hosts rather than package registries, but also for security resaons.

Requires a domain name to be the first part of the module identifier

The domain requirement is itself required because Go resolves modules to the code host. It also prevents a class of dependency confusion, discussed in the next section.

Only contain ascii letters, digits, and limited punctuation ([.~_-])

Restrictions on allowed import path parameters prevents homograph or homoglyph attacks. ^{11 12 13}

$ go mod init ɢoogle.com/chrome
go: malformed module path "ɢoogle.com/chrome": invalid char 'ɢ'

Cannot begin or end with a slash or dot

Slash and dot restrictions prevent absolute and relative path from being part of imports. While this means they are more verbose, it also means that

1. you can always see the exact package being used
2. relative and absolute path attacks are not possible

There are more restrictions

For specific contexts, there are more rules

• The domain part has further restrictions
• Windows has reserved files to avoid
• Major version suffixes

See ¹⁴ for more details.

Only Secure Remotes

Go will only talk to secure code hosts, preferring https and git+ssh.

You can use GOINSECURE to list module patterns which can be fetched over http and other insecure protocols. Modules fetched insecurly will still be validated against the checksum database.

Consult the table of VCS Schemes ¹⁵ to find which tools and protocols are supported. You may also need to set GOVCS ¹⁶.

Private module support

Go supports modules developed in private. You can

• Fetch modules from private code repositories
• Prevent your private modules from being publicly indexed
• Run a private proxy and sumdb

See the private modules section ¹⁷ for details and necessary configuration.

To authenticate with private module hosts, Go defers to tool config like .gitconfig when downloading directly. For https BasicAuth is supported through the .netrc file. ¹⁸

Preventing dependency confusion

Dependency confusion ¹⁹ is when a public package with the same name as an internal package is fetched. Go helps to prevent this by

Requiring a domain to start module and import paths

This means that module names cannot overlap, such as when a malicious actor registers the same module in a public registry.

Ignoring replace directives in dependencies

A comprimised dependency cannot replace other dependencies with one hosted under a different domain.

Malicious version changes

There are two main version attacks, replacing or adding a tag with an exploit.

Replacing a tag

Retagging is practically impossible, given a module has been fetched once, by anyone. The original hash will be in the global sumdb and the validation will fail. This will of course depend on your GO[NO]SUM, GO[NO]PROXY, and GOPRIVATE settings.

Creating a new tag

In Go, versions are specific, not a range. Additionally, Go will only select from versions which are listed. By design, Go will not select newer modules and is relatively safe from malicious version increments.

No pre or post hooks

The go module system lacks any pre or post hooks for fetch, build, or install. This rules out a class of attacks, such as those seen with NPM.

Information in the binaries

Go adds the dependency information into the binary. This includes their path, version, and sumdb hash.

go version -m $(which binary)

With Go 1.18, it will also include the build flags, environment settings, and VCS information for the main module. ²⁰

Reproducible Builds

Go has a goal for 100% reproducible builds of artifacts. MVS dependency management is one part and core to this, ensuring that the source code is the same. While this is only the first step, the Go team has been able to reach this goal even when cross-compiling.

Learning more

• Module Reference
• Go & Versioning
• Original Proposal
• GitHub Issues