1. Initialize a values repo
Otomi needs a git repo to store its configuration. We call it a values repo.
In order to quickly get up and running it is advised to download the chart values and fill in the values.
The following commands bootstrap the values repo:
# map the chart values.yaml as initial input:
# point to a folder that is or will become the values repo
# and bootstrap all the files in there
2. Customize configuration
The essential otomi platform configurations is stored in
env/secrets.settings.yaml files. Inspect them and customize values to match your environment.
The minimum required input is found in the chart values.
3. Configure credentials from a KMS (optional)
If you don't need encryption straight away please continue to the next step
Otomi will encrypt any
secrets.*.yaml files with sops, but only if it finds
sops: configuration details. In order to have access to the KMS credentials to encrypt/decrypt, a
.secrets file needs to exist and have those credentials. Please
copy .secrets.sample .secrets and fill it in with the credentials from the prerequisites.
Then you can run
otomi bootstrap, which will result in the encryption and decryption of the secrets files.
Note to developers:
git diff to show unencrypted values, you must register the sops diffing routine once with git. To register it:
git config diff.sopsdiffer.textconv 'sops -d'
This only registers the sops differ, which is responsible for invoking sops. But sops still needs the credentials to the KMS service. Again, your AWS profile is always pointed and loaded, but in case of Google KMS you will need to point GOOGLE_APPLICATION_CREDENTIALS to the
gcp-key.json file holding your account information:
Now try a diff:
4. Start Otomi Console on your local machine (optional)
Bootstrap again and start the console:
The console allows for easy configuration of many settings but not all. Assuming the setup steps are completed, you need to now configure the Otomi values repository. This repo is the source configuration for Otomi.
Configuration can be performed much easier through the Otomi Console, so please refer to the Otomi Console documentation.
However, chart configuration is not (yet) exposed through the console, so please look at the values repo's
env/charts/* files to edit the configuration files.
Important things to note:
- Every configuration file can have a
secrets.*.yamlcounterpart, but these are optional.
- A json schema and vscode settings are imported by the bootrap (in
.vscode/*), so you will have automatic linting and hinting for the configuration when vscode is used (try CTRL+SPACE in the yaml).
.secretsis correctly configured then automatic de-/en-cryption will also be performed when in vscode and editing a
Please follow the guidance of the yaml hinting, as it has all the descriptions and example values you need to operate on these files.
VSCode automatically loads the '.vscode/values-schema.yaml' schema provided. Please inspect it or wire it up manually when using another editor.
If you wish to be sure of your changes, you can always do a
git diff. When you chose to use encryption and have correctly followed the corresponding instructions, then you should see a diff with the unencrypted values. That is, if you modified any ;)
When you are done with the configuration you can validate the results:
If you have made an error in the format of the values this will be reported.
To check if all the output manifests are valid for the target cluster's k8s version, and following best practices you can run another variation:
Charted vs uncharted resources
The output manifests generated by otomi are deployed in two ways:
- Uncharted: some base manifests are applied directly with
- Charted: manfests that are packaged up in helm charts.
Ideally, we would like to deploy as helm chart as it has many benefits such as rollback. But in some cases we can't or we don't wish to. The reasons for that are the following:
- Some resources we don't want governed by charts (as charts might get accidentally removed, erasing everything that was deployed with it).
- Some existing resources have to be patched (like pull secrets in service accounts), which helmfile won't do as it will not modify existing resources not annotated to be under control by a chart.
- Some resources need to exist before the charts are deployed (such as CRDs).
The manifests that are currently not charted are:
k8s/base(unparameterized, mostly rbac roles)
values/cloud(applies cloud specific "normalization" patterns, such as for storageclasses)
values/k8s(team resources, such as namespaces, service accounts, pull secrets)
Working with uncharted resources
Currently we don't have any subcommand that only works on uncharted resources, but we have the following commands that target the entire bundle.
otomi test: does a dry run, showing all manifests that will be deployed, and will also show any errors in the output manifests.
otomi apply: deploys all the manifests (uncharted first, then charted)
So after doing
otomi test, if all looks ok, go ahead and do the initial deployment of all resources:
This command executes two stages (please see
binzx/apply.ts). The first stage will deploy all uncharted resources with
kubectl apply, and the second stage will deploy all the charted resources with
Whenever you add a team, or change or add to these uncharted resources, you have to run
otomi apply to apply them. When you let Drone do the syncing for you, it will invoke that command to synchronize the cluster.
Working with charted resources
During development iterations you will probably not touch uncharted resources often, but instead you will add features in charts.
Otomi has these subcommands that only target charted resources:
You can always target a single chart like this:
otomi (diff|apply|sync|template) -l name=prometheus-operator
(For a list of all supported flags to use those subcommands, we defer to the helmfile documentation, as those are deferred to the helmfile cli.)
Let's do a diff of all the charts that are enabled:
Need to know quirks
Whenever you modify resources without using helm, its internal bookkeeping (the versioned secrets in the namespaces) will not change, and any subsequent
otomi apply commands will not modify anything. If you notice this, and want to overwrite with the output manifests, you can use
otomi sync, which will skip doing a diff, and instead apply all charted manifests as a new version.
8. Committing values
To commit values and run post processing tasks:
This will detect any version changes and then commit all files with a standardized message "Manual commit". (We believe all values repo configuration changes are equally meaningful and don't need explicit commit messages.) Directly doing a
git commit is discouraged with a git hook saying so, but whenever you did not touch any versions in
env/clusters.yaml you may bypass with
git commit -m "Manual commit" --no-verify to save development time.