Stateit is a library created to make it easier execute tasks that require to keep state
1. Installation
You can execute StateIT with a standalone CLI so that you can execute it any time from your terminal.
2. Concepts
2.1. Plan
A plan is a manifest file where you can:
-
describe a set of resources
-
describe how the state of these resources will be stored
In practice a plan is a groovy file with a .st suffix:
resource1('id1') { ... } (1)
resource2('id2') { ... }
resource2('id3') { ... }
state {
provider = fileState('/path/to/state.json') (2)
}
1 | resources declared in the plan |
2 | state storage, or where and how the state of the resources will be stored |
2.2. Resource
A resource represents something inside a system having state. For instance:
-
a Github repository (can be created, removed, having permissions….)
-
a local directory which (could be created or removed)
-
a database (could be created, could have tables…)
-
…you name it
A resource has at least and ID and some properties needed to create the resource.
resource1('id-of-the-resource') { (1)
property1 = "property value" (2)
property2 = "value"
propertyN = "value"
}
1 | ID |
2 | properties |
2.3. Catalog
A catalog is a set of plan files in the same directory sharing the same state provider.
+-catalog-dir
+
|
+-- plan1.st
|
+-- plan2.st
|
+-- stateit.state.json
|
+-- stateit.vars.toml
If StateIT detects more than one state provider, it will stop trying to execute the catalog, and it will show an error. |
2.4. State
When StateIT executes a plan it should decide whether:
-
to validate the resources declared in it
-
to apply the resources declared in it
-
to destroy all the resources declared in it
In order to be able to decide, StateIT should store the state of the resources each time it executes the same plan. That state is stored as JSON. It stores the list of the resources applied by the plan execution.
[
{
"id": "resource-id", (1)
"type": "qualified.type.class.of.Resource1", (2)
"props": { (3)
"property1": "value",
"property2": "value"
}
}
]
1 | ID of the resource |
2 | qualified class name of the resource |
3 | the properties used to build the state of the resource |
This JSON file can be stored in any type of storage. In order to be able to store the plan execution, the plan file should declare which type of state storage is going to use:
state {
provider = fileState('/path/to/state.json') (1)
}
1 | Declares a local file where the stage will be stored |
2.5. Dependency
A given resource could depend on another resource. When declaring a resource you can assign the resource to a variable and then use the resource output properties in another resource.
def res1 = resource1('id-resource1') { (1)
property = "value"
}
resource2('id-resource2') {
property = res1.x (2)
}
1 | assign resource to a variable to be able to use it in another resource |
2 | resource2 depends on x property from resource1 |
3. StateIT process
At 1000 feet the StateIT process is pretty simple:
-
Plan file is parsed
-
Each resource found in plan has its properties resolved including their dependencies with other resources
-
Once resources are build and populated, all of them are validated
-
If the process goal was only to validate the plan, the process ends showing the result of the validation
-
Anyway regardless of the goal if the plan is not valid the process ends there
-
If the goal of the process is other than validation resources are applied or destroyed and then the process ends
4. Tutorial
StateIT executes plans representing resource states. When a plan is executed StateIT stores its state so that it knows when a given resource has been applied successfully and there is no need to execute it again.
4.1. validate
The following plan file shows a resource representing a local filesystem directory (/tmp/sample-dir). It also shows that the execution state will be stored in a file at /tmp/state.json.
directory("sample-dir") {
path = "/tmp/sample-dir"
}
state {
provider = fileState("/tmp/state.json")
}
Although we know it’s properly configured lets
stateit --plan plan.st validate
Which outputs:
[stateit] - 03:12:21.538 - validating resources to apply
[stateit] - 03:12:21.542 - TO APPLY: 1
[stateit] - 03:12:21.542 - TO REMOVE: 0
[stateit] - 03:12:21.546 - VALIDATION SUCCEEDED!
It seems that the plan is ok and that 1 resource will be applied in case you carry on and execute the plan.
4.2. apply
In that case execute:
stateit --plan plan.st execute
What it does is:
-
checks that a previous state exists
-
shows how many resources will apply or which will destroy
-
show the resources removed/applied
-
saves the state once the execution has finished
[stateit] - 03:16:30.272 - state file found... loading resources
[stateit] - 03:16:30.297 - TO APPLY: 1
[stateit] - 03:16:30.297 - TO REMOVE: 0
[stateit] - 03:16:30.297 - create sample-dir
[stateit] - 03:16:30.301 - saving state
You can check that the task created a directory. Now if you execute the same script one more time, you’ll realize that the script doesn’t execute anything:
[stateit] - 03:16:56.022 - state file found... loading resources
[stateit] - 03:16:56.050 - TO APPLY: 0
[stateit] - 03:16:56.050 - TO REMOVE: 0
[stateit] - 03:16:56.051 - saving state
That’s because the state stored in the /tmp/state.json
path has registered the state of the script,
and it knows that the resource have been already applied.
4.3. destroy
Finally if we’d like to remove all applied resources we can execute the destroy command:
stateit --plan plan.st destroy
This will destroy all resources declared in the plan:
[stateit] - 03:26:09.109 - state file found... loading resources
[stateit] - 03:26:09.135 - destroying all resources apply
[stateit] - 03:26:09.135 - deleting ALL (1) resources
[stateit] - 03:26:09.136 - destroy sample-dir
[stateit] - 03:26:09.137 - saving state
5. Variables & Secrets
At some point you may parametrize your plans with variables instead of hardcoding those values. StateIT by default uses TOML files. Inside these files the convention is that:
-
NON SENSITIVE values should be placed under the vars group.
-
SENSITIVE values should be placed under secrets.
[vars]
...
[secrets]
...
5.1. Resolution
5.2. Variables
To reference a variable in the plan file you can use the var_ variable followed by the name of the variable you’d like to resolve:
repository("sample-dir") {
path = var_.sample_dir_path
}
The variable should be present in the vars section of the stateit.vars.toml file:
[vars]
sample_dir_path="/tmp/example_dir"
5.3. Secrets
Some variables like passwords, and credentials in general require to have a special consideration. These values won’t be shown in console output. This time to reference a secret in your plan you can use the sec_ variable followed by the name of the secret:
github_repository("sample-dir") {
username = sec_.github_username
password = sec_.github_token
}
The secret should be declared in the stateit.vars.toml file in the secrets section:
[secrets]
github_username=username
github_token=token
Secrets are not stored in the resource state |
6. Functions
Although StateIT tries to be very concise, however there are a set of utility functions available in every plan that will help to reduce even more the code of the plan.
6.1. Date
Date functions help to create dates for different tasks (e.g. generate a file name with the current date):
7. Providers
A provider represents a set of stateful resources and some stateful operations of a specific realm.
7.1. Files
The Files provider has a set of resources representing local filesystem items and stateful operations.
7.1.1. State
The Files provider provides a way of handling a plan state via a local file.
state {
implementation = file('/tmp/mystate.json')
}
7.1.2. Resources
directory
Represents a file system directory
directory('id-directory') {
path = '/tmp/directory-to-create'
}
Name | Description | Default value |
---|---|---|
path |
path to create the directory at |
Name | Description | Default value |
---|---|---|
path |
path to create the directory at |
targz
Represents a periodic tar.gz operation. It uses a cron expression to schedule the operation. It could be a compress or uncompress operation.
Cron scheduling only works in *nix systems with crontab, otherwise it will only execute once |
targz('id-directory') {
input = "/input/dir"
output = () -> "/anotherdir/output${now('yyyy-MM-dd')}.tar.gz"
cron = "*/5 * * * *"
action = compress()
overwrite = true
}
Property | Description | Default value |
---|---|---|
input |
directory to compress/uncompress |
|
output (string) |
tar.gz file to create (static) |
|
output (lambda expr) |
tar.gz file to create (dynamic) |
|
cron |
cron expression |
* * * * * |
action |
compress() / uncompress() |
compress() |
overwrite |
whether to overwrite the file in case it has the same name |
false |
7.2. Github
Provides resources that can be created in Github such as repository, security restrictions, branches…etc.
7.2.1. Credentials
export STATEIT_GITHUB_USERNAME="username"
export STATEIT_GITHUB_TOKEN="token"