StateIT

Stateit is a library created to make it easier execute tasks that require to keep state

1. Installation

You can execute StateIT with a standalone CLI so that you can execute it any time from your terminal.

2. Concepts

2.1. Plan

A plan is a manifest file where you can:

describe a set of resources
describe how the state of these resources will be stored

In practice a plan is a groovy file with a .st suffix:

plan.st

resource1('id1') { ... } (1)
resource2('id2') { ... }
resource2('id3') { ... }

state {
  provider = fileState('/path/to/state.json') (2)
}

1	resources declared in the plan
2	state storage, or where and how the state of the resources will be stored

2.2. Resource

A resource represents something inside a system having state. For instance:

a Github repository (can be created, removed, having permissions….)
a local directory which (could be created or removed)
a database (could be created, could have tables…)
…you name it

A resource has at least and ID and some properties needed to create the resource.

resource

resource1('id-of-the-resource') {  (1)
    property1 = "property value"    (2)
    property2 = "value"
    propertyN = "value"
}

1	ID
2	properties

2.3. Catalog

A catalog is a set of plan files in the same directory sharing the same state provider.

+-catalog-dir
      +
      |
      +-- plan1.st
      |
      +-- plan2.st
      |
      +-- stateit.state.json
      |
      +-- stateit.vars.toml

If StateIT detects more than one state provider, it will stop trying to execute the catalog, and it will show an error.

2.4. State

When StateIT executes a plan it should decide whether:

to validate the resources declared in it
to apply the resources declared in it
to destroy all the resources declared in it

In order to be able to decide, StateIT should store the state of the resources each time it executes the same plan. That state is stored as JSON. It stores the list of the resources applied by the plan execution.

state.json

[
    {
        "id": "resource-id", (1)
        "type": "qualified.type.class.of.Resource1", (2)
        "props": { (3)
           "property1": "value",
           "property2": "value"
        }
    }
]

1	ID of the resource
2	qualified class name of the resource
3	the properties used to build the state of the resource

This JSON file can be stored in any type of storage. In order to be able to store the plan execution, the plan file should declare which type of state storage is going to use:

file state provider

state {
  provider = fileState('/path/to/state.json') (1)
}

1	Declares a local file where the stage will be stored

2.5. Dependency

A given resource could depend on another resource. When declaring a resource you can assign the resource to a variable and then use the resource output properties in another resource.

dependency

def res1 = resource1('id-resource1') { (1)
   property = "value"
}

resource2('id-resource2') {
   property = res1.x (2)
}

1	assign resource to a variable to be able to use it in another resource
2	resource2 depends on x property from resource1

3. StateIT process

At 1000 feet the StateIT process is pretty simple:

Plan file is parsed
Each resource found in plan has its properties resolved including their dependencies with other resources
Once resources are build and populated, all of them are validated
If the process goal was only to validate the plan, the process ends showing the result of the validation
Anyway regardless of the goal if the plan is not valid the process ends there
If the goal of the process is other than validation resources are applied or destroyed and then the process ends

4. Tutorial

StateIT executes plans representing resource states. When a plan is executed StateIT stores its state so that it knows when a given resource has been applied successfully and there is no need to execute it again.

4.1. validate

The following plan file shows a resource representing a local filesystem directory (/tmp/sample-dir). It also shows that the execution state will be stored in a file at /tmp/state.json.

plan.st

directory("sample-dir") {
  path = "/tmp/sample-dir"
}

state {
   provider = fileState("/tmp/state.json")
}

Although we know it’s properly configured lets

validate

stateit --plan plan.st validate

Which outputs:

[stateit] - 03:12:21.538 - validating resources to apply
[stateit] - 03:12:21.542 - TO APPLY:     1
[stateit] - 03:12:21.542 - TO REMOVE:    0
[stateit] - 03:12:21.546 - VALIDATION SUCCEEDED!

It seems that the plan is ok and that 1 resource will be applied in case you carry on and execute the plan.

4.2. apply

In that case execute:

execute

stateit --plan plan.st execute

What it does is:

checks that a previous state exists
shows how many resources will apply or which will destroy
show the resources removed/applied
saves the state once the execution has finished

[stateit] - 03:16:30.272 - state file found... loading resources
[stateit] - 03:16:30.297 - TO APPLY:     1
[stateit] - 03:16:30.297 - TO REMOVE:    0
[stateit] - 03:16:30.297 - create sample-dir
[stateit] - 03:16:30.301 - saving state

You can check that the task created a directory. Now if you execute the same script one more time, you’ll realize that the script doesn’t execute anything:

[stateit] - 03:16:56.022 - state file found... loading resources
[stateit] - 03:16:56.050 - TO APPLY:     0
[stateit] - 03:16:56.050 - TO REMOVE:    0
[stateit] - 03:16:56.051 - saving state

That’s because the state stored in the /tmp/state.json path has registered the state of the script, and it knows that the resource have been already applied.

4.3. destroy

Finally if we’d like to remove all applied resources we can execute the destroy command:

destroy

stateit --plan plan.st destroy

This will destroy all resources declared in the plan:

[stateit] - 03:26:09.109 - state file found... loading resources
[stateit] - 03:26:09.135 - destroying all resources apply
[stateit] - 03:26:09.135 - deleting ALL (1) resources
[stateit] - 03:26:09.136 - destroy sample-dir
[stateit] - 03:26:09.137 - saving state

5. Variables & Secrets

At some point you may parametrize your plans with variables instead of hardcoding those values. StateIT by default uses TOML files. Inside these files the convention is that:

NON SENSITIVE values should be placed under the vars group.
SENSITIVE values should be placed under secrets.

stateit.vars.toml

[vars]
...

[secrets]
...

5.1. Resolution

5.1.1. default

By default StateIT looks for a TOML variable file named stateit.vars.toml in these locations:

At the StateIT directory at $HOME ($HOME/.stateit)
In the same directory as the executed plan file

5.1.2. specific file

You can always pass a variables file as an argument using --var-file:

specific file

stateit --plan /path/to/plan/plan.st --var-file /path/to/var/file/stateit.vars.toml

5.2. Variables

To reference a variable in the plan file you can use the var_ variable followed by the name of the variable you’d like to resolve:

variables example

repository("sample-dir") {
    path  = var_.sample_dir_path
}

The variable should be present in the vars section of the stateit.vars.toml file:

stateit.vars.toml

[vars]
sample_dir_path="/tmp/example_dir"

5.3. Secrets

Some variables like passwords, and credentials in general require to have a special consideration. These values won’t be shown in console output. This time to reference a secret in your plan you can use the sec_ variable followed by the name of the secret:

github example

github_repository("sample-dir") {
    username  = sec_.github_username
    password  = sec_.github_token
}

The secret should be declared in the stateit.vars.toml file in the secrets section:

stateit.vars.toml

[secrets]
github_username=username
github_token=token

Secrets are not stored in the resource state

6. Functions

Although StateIT tries to be very concise, however there are a set of utility functions available in every plan that will help to reduce even more the code of the plan.

6.1. Date

Date functions help to create dates for different tasks (e.g. generate a file name with the current date):

6.1.1. now()

This function converts a given date to a string using the ISO-8601 ("yyyy-MM-dd’T’HH:mm:ssZ")

now(pattern)

directory('todays-backup') {
  path  = "/backups/${now()}" (1)
}

1	path will become /backups/2022-12-31’T'10:00:00Z

6.1.2. now(pattern)

This function converts a given date to a string following the pattern passed as an argument.

now(pattern)

directory('todays-backup') {
  path  = "/backups/${now('yyyy-MM-dd')}" (1)
}

1	path will become /backups/2022-12-31

7. Providers

A provider represents a set of stateful resources and some stateful operations of a specific realm.

7.1. Files

The Files provider has a set of resources representing local filesystem items and stateful operations.

7.1.1. State

The Files provider provides a way of handling a plan state via a local file.

state {
    implementation = file('/tmp/mystate.json')
}

7.1.2. Resources

Table 1. arguments
Name	Description	Default value
path	path to create the directory at

Table 2. attributes
Name	Description	Default value
path	path to create the directory at

targz

Represents a periodic tar.gz operation. It uses a cron expression to schedule the operation. It could be a compress or uncompress operation.

Cron scheduling only works in *nix systems with crontab, otherwise it will only execute once

targz

targz('id-directory') {
  input     = "/input/dir"
  output    = () -> "/anotherdir/output${now('yyyy-MM-dd')}.tar.gz"
  cron      = "*/5 * * * *"
  action    = compress()
  overwrite = true
}

Table 3. arguments
Property	Description	Default value
input	directory to compress/uncompress
output (string)	tar.gz file to create (static)
output (lambda expr)	tar.gz file to create (dynamic)
cron	cron expression	* * * * *
action	compress() / uncompress()	compress()
overwrite	whether to overwrite the file in case it has the same name	false

7.2. Github

Provides resources that can be created in Github such as repository, security restrictions, branches…etc.

7.2.1. Credentials

credentials using environment variables

export STATEIT_GITHUB_USERNAME="username"
export STATEIT_GITHUB_TOKEN="token"

7.2.2. State

state {
  implementation = github {
     repository = "myorg/myrepository"
     file       = "mystate.json"
     branch     = "main"
  }
}

7.2.3. Resources

repository

github_repository('id-directory') {
  name  = "mycustomrepository"
  owner = "myorganization"
}

7.3. Create your own

TODO