Modules are how Go manages dependencies. A module is a collection of packages that are released, versioned, and distributed together. Each package within a module is a collection of source files in the same directory that are compiled together.
In this post, we will explore Go mododules design and learn how they support supply chain security. You can find the go mod documentation here: https://golang.org/ref/mod
The statements here reference Go 1.17 and not all applies to older or newer versions.
Anatomy of a go.mod file
// Sample go.mod file module github.com/org/module // module name require ( // module dependencies github.com/foo/bar v0.1.2 github.com/cow/moo v1.2.3 mydomain.com/gopher v0.2.3-beta1 )
Go has restrictions on a module name which we will talk about later. The version is a specific and minimum version for your project. Note, there are no version ranges, only a single Semantic Version. As Go processes the dependency tree, it selects the maximum of the versions found. In doing so, MVS can create a reproducible dependency tree without a lockfile.
Minimum Version Selection
Golang uses the MVS (minimum version selection) [^vgo-mvs] [^ref-mvs] algorithm to select dependency versions. This deterministic algorithm has nice properties for reproducible builds and avoids the NP-complete runtime complexity. There is no need for a SAT solver because the dependency selection problem is constrained.
At its heart, hand waving away some details, MVS is a breadth-first traversal across modules and versions. The tree is defined by go.mod files for a module, its dependencies, its dependencies-dependencies, and so on.
The minimum terminology is due to the idea that this is a minimal implementation for a lockfile free, deterministic dependecy management system.
Directives in go.mod
The go.mod file has a number of directives for controlling versions and dependencies.
module example.com/my/thing go 1.16 require example.com/other/thing v1.0.2 require example.com/new/thing/v2 v2.3.4 exclude example.com/old/thing v1.2.3 replace example.com/bad/thing v1.4.5 => example.com/good/thing v1.4.5 retract [v1.9.0, v1.9.5]
- go - sets the minimum Go syntax version
- require - specify direct module dependencies
- exclude - prevents a dependency version
- replace - substitutes without renaming
- retract - minor and patch versions of this module
//Deprecated can be used for major version of your modules. You add a comment and tag a new release.
// Deprecated: use example.com/mod/v2 instead. module example.com/mod
As of Go 1.17, there are two require blocks, one for direct and indirect each, to support lazy loading. [^lazy-loading]
Environment variables
Go supports a number of environment variables for controlling how modules and module aware commands [^ref-aware] work. The following table contains the variables referenced in later sections. For a full list, more details, and examples, see: [^envvars] [^private]
You can use go env to see the current settings.
| variable | used for | | ------- | -------- | | GOMODCACHE | directory for module related files | | GOPRIVATE | module globs to handle as private | | GOPROXY | ordered list of module proxies to use | | GONOPROXY | module globs to fetch directly | | GOSUMDB | ordered list of sumdb hosts to use | | GONOSUMDB | module globs to omit remote sumdb checks on | | GOVCS | sets VCS tools allowed for public and private access | | GOINSECURE | globs to allow fallback to http on [^dontuse] |
Hashes and the go.sum file
When the go command downloads a module, it computes a cryptographic hash and compares it with a known value to verify the file hasn’t changed since it was first downloaded. Modules store these hashes in a go.sum file and the Go command verifies they match. Go also stores these hashes in mode module cache and will compare them with a global database. [^mod-authn]
Local module cache
Go maintains a shared module cache on your local system. [^mod-cache] This is where downloaded modules are stored. The location is determined by the GOMODCACHE variable. Module code is read-only by default to prevent local modifications and "it works on my machine" issues. The shared cache also contains prebuilt artifacts. All of this means that multiple projects on your machine can reuse the same downloaded and preprocessed packages.
Global services modules and hashes
The Go team maintain global proxies for sumdb, cachedb, and global hash integrity and revocation checks. [^global-proxies] The checksum database can be used to detect misbehaving origin and proxy servers. It has a merkel tree transparency log for hashes powered by the Trillian project. The cache database proxies public modules and will maintain copies even if the origin server removes them.
The Go team has taken privacy seriously. These services record very minimal information. You can read the privacy statemnt for sum.golang.org/privacy for details. Their communications in issues on GitHub reflects this. For example, only limited auth features have been enabled, because they are being careful in trying to maintain privacy in proxy.
Module naming
Go has a number of module naming rules. These are partially by design, in using code hosts rather than package registries, but also for security resaons.
Requires a domain name to be the first part of the module identifier
The domain requirement is itself required because Go resolves modules to the code host. It also prevents a class of dependency confusion, discussed in the next section.
Only contain ascii letters, digits, and limited punctuation ([.~_-])
Restrictions on allowed import path parameters prevents homograph or homoglyph attacks. [^homoglyph] [^homograph] [^unicode-legality]
$ go mod init ɢoogle.com/chrome go: malformed module path "ɢoogle.com/chrome": invalid char 'ɢ'
Cannot begin or end with a slash or dot
Slash and dot restrictions prevent absolute and relative path from being part of imports. While this means they are more verbose, it also means that
- you can always see the exact package being used
- relative and absolute path attacks are not possible
There are more restrictions
For specific contexts, there are more rules
- The domain part has further restrictions
- Windows has reserved files to avoid
- Major version suffixes
See [^ref-modpath] for more details.
Only Secure Remotes
Go will only talk to secure code hosts, preferring https and git+ssh.
You can use GOINSECURE to list module patterns which can be fetched over http and other insecure protocols. Modules fetched insecurly will still be validated against the checksum database.
Consult the table of VCS Schemes [^vcs-scheme] to find which tools and protocols are supported. You may also need to set GOVCS [^govcs].
Private module support
Go supports modules developed in private. You can
- Fetch modules from private code repositories
- Prevent your private modules from being publicly indexed
- Run a private proxy and sumdb
See the private modules section [^priv-mods] for details and necessary configuration.
To authenticate with private module hosts, Go defers to tool config like .gitconfig when downloading directly. For https BasicAuth is supported through the .netrc file. [^private-auth]
Preventing dependency confusion
Dependency confusion ^depcon1 is when a public package with the same name as an internal package is fetched. Go helps to prevent this by
Requiring a domain to start module and import paths
This means that module names cannot overlap, such as when a malicious actor registers the same module in a public registry.
Ignoring replace directives in dependencies
A comprimised dependency cannot replace other dependencies with one hosted under a different domain.
Malicious version changes
There are two main version attacks, replacing or adding a tag with an exploit.
Replacing a tag
Retagging is practically impossible, given a module has been fetched once, by anyone. The original hash will be in the global sumdb and the validation will fail. This will of course depend on your GO[NO]SUM, GO[NO]PROXY, and GOPRIVATE settings.
Creating a new tag
In Go, versions are specific, not a range. Additionally, Go will only select from versions which are listed. By design, Go will not select newer modules and is relatively safe from malicious version increments.
No pre or post hooks
The go module system lacks any pre or post hooks for fetch, build, or install. This rules out a class of attacks, such as those seen with NPM.
Information in the binaries
Go adds the dependency information into the binary. This includes their path, version, and sumdb hash.
go version -m $(which binary)
With Go 1.18, it will also include the build flags, environment settings, and VCS information for the main module. [^vcs-details]
Reproducible Builds
Go has a goal for 100% reproducible builds of artifacts. MVS dependency management is one part and core to this, ensuring that the source code is the same. While this is only the first step, the Go team has been able to reach this goal even when cross-compiling.
Learning more
[^dontuse]: You probably shouldn't use this [^envvars]: https://golang.org/ref/mod#environment-variables [^global-proxies]: https://golang.org/ref/mod#checksum-database [^govcs]: https://golang.org/ref/mod#vcs-govcs [^hofmod]: https://github.com/hofstadter-io/hof [^homoglyph]: https://blog.malwarebytes.com/101/2017/10/out-of-character-homograph-attacks-explained/ [^homograph]: https://www.securityweek.com/zero-day-homograph-domain-name-attack [^lazy-loading]: https://golang.org/ref/mod#lazy-loading [^mod-authn]: https://golang.org/ref/mod#authenticating [^mod-cache]: https://golang.org/ref/mod#module-cache [^priv-mods]: https://golang.org/ref/mod#private-modules [^private-auth]: https://golang.org/ref/mod#private-module-repo-auth [^private]: https://golang.org/ref/mod#private-modules [^ref-aware]: https://golang.org/ref/mod#mod-commands [^ref-modpath]: https://golang.org/ref/mod#go-mod-file-ident [^ref-mvs]: https://golang.org/ref/mod#minimal-version-selection [^sumdb-privacy]: https://sum.golang.org/privacy [^unicode-legality]: https://github.com/golang/go/issues/44970 [^vcs-details]: https://github.com/golang/go/issues/37475 [^vcs-scheme]: https://golang.org/ref/mod#vcs [^vgo-mvs]: https://research.swtch.com/vgo-mvs (part 4 of a series)