Config

Introduction

The config module does not communicate with any Recorded Future dataset, it is used to configure PSEngine and the integration behaviour. Its usage is not mandatory in an integration development, it more a conveninece if a configuration file is needed.

The Config is a class of PSEngine that can be used to retrieve static information from a file or the environment variables of the system. The file extensions allowed are:

• .toml
• .json
• .env file

If the system cannot use any of those you can still use the environment variables or the parameters of the init method. (See example below)

The config has a very strict priority of reading values:

1. values passed via init method are the most important
2. values gathered in the environment variables
3. values from any config file

If you have a environment variable configured it will overwrite the value set in the config file.

The Config class is a singleton, which means that once initialized its values are immutable and every module will read them. The creation of the Config object is done via the init method. If you need to get the data of the config you need to use the get_config method.

The Config class manages a ConfigModel class by default, which is a pydantic.BaseSetting class which contains attributes of general needs, like proxy settings, HTTP timeout etc.

Warning

The Config.init creates the a global config that is unique at any point of your code execution. You need to define the Config before initialising the manager on your integration entrypoint. Once that is done you can reference the Config from anywhere. See example below.

See the API Reference for internal details of the module.

Examples

Warning

Below are some examples of usage of the module. Consider adding error handling as necessary. All the errors that can be raised by each method or function are specified in the API Reference page.

Also, you need to configure the RF_TOKEN environment variable before starting. See Learn.

Example 1: Read a Config from config.toml.

To run this example create a config.toml file with content:

my_value=5

We initialize the Config object with init passing the path (absolute or relative) of the config file we intend to use. This create an object but it does not return it.

Since we want to print the value of my_value we use get_config method to return the ConfigModel instance.

from psengine.config import Config, get_config

Config.init(config_path='config.toml')
config = get_config()
print(config.my_value)

This will print 5.

Example 2: Configure a Config from environment variables.

You can read only variables that are statically defined in the ConfigModel. They need to be prefixed with RF_ and need to be of the type specified in the model. The variables are:

• platform_id -> str
• app_id -> str
• rf_token -> RFToken
• http_proxy -> str
• https_proxy -> str
• client_ssl_verify -> bool
• client_basic_auth -> (str, str)
• client_cert -> str or (str, str)
• client_timeout -> int
• client_retries -> int
• client_backoff_factor -> int
• client_status_forcelist -> List of int
• client_pool_max_size -> int

So for example if we need to set the app_id and platform_id variables we can do the following:

export RF_APP_ID=example/1.0.0
export RF_PLATFORM_ID=Splunk/10.0.0

And read the config:

from psengine.config import Config, get_config

Config.init()
config = get_config()
print(config.app_id)
print(config.platform_id)

The sample code will print the values defined above.

Example 3: Configure a Config from python.

You can initialise your config from the init method directly:

from psengine.config import Config, get_config

Config.init(my_value=5)
config = get_config()
print(config.my_value)

This will print 5.

Example 4: Define your own config.

If you want to define your own config in an integration, you can. The steps are:

1. Define your model (IntegrationModel in the example), it has to inherit from psengine.ConfigModel

2. Change the Config.init to use the config_class and assign it to the IntegrationConfig model we just created

3. Use the config_class with IntegrationConfig in the get_config function as well

4. Keep doing everything else as usual.

To replicate this example, first create a custom_config.toml file with content:

simple_value=5

[complex_value]
data = [ "a", "list" ]
value = [ 1, 2, 3]

This should be placed in the same directory of the example python code. Once the file configuration is created the sample code will create the custom config in the IntegrationConfig class. We then call the init method passing as config_class the custom configuration model and the usual TOML path.

from pathlib import Path

from pydantic import BaseModel

from psengine.config import Config, ConfigModel, get_config

CONFIG_PATH = Path(__file__).parent / 'custom_config.toml'


class ComplexValue(BaseModel):
    """Model to define the `complex_value` table."""

    data: list[str]
    value: list[int]


class IntegrationConfig(ConfigModel):
    """The class of my integration config."""

    simple_value: int
    complex_value: ComplexValue


Config.init(config_class=IntegrationConfig, config_path=CONFIG_PATH)
config = get_config()

print(config)
print(config.simple_value)
print(config.complex_value.data)
print(config.complex_value.value)

Each property can be access using the dot notation, for example config.complex_value.data.

Example 5: Real example.

Let's assume that we are developing an integration that needs to fetch playbook alerts. The current requirements are that the alerts to be ingested to be:

• Domain Abuse
• with New status
• High priority
• No older than yesterday

Each domain that triggered this alert has to be enriched with the links field.

We can opt for a quick script using "free" variables around the code, or using the config.

Code 1 without the config:

In this example we are hardcoding the values that we need for fetching the alert and enriching the IOCs. This is a perfectly fine code, however in larger applications it might be challenging to maintain if the requirements change.

from psengine.enrich import LookupMgr
from psengine.playbook_alerts import PlaybookAlertMgr
from psengine.playbook_alerts.pa_category import PACategory

pba_mgr = PlaybookAlertMgr()
enrich_mgr = LookupMgr()

alerts = pba_mgr.fetch_bulk(
    category=PACategory.DOMAIN_ABUSE, statuses=['New'], priority='High', created_from='-1d'
)

domains = [alert.panel_status.entity_name for alert in alerts]
enriched_domains = enrich_mgr.lookup_bulk(domains, 'domain', fields=['links'])

for enriched in enriched_domains:
    print(enriched)

An alternative is to save the requirements to a config file and use them instead of the hardcoded values.

Code 2 with the config: With the int_config.toml file being:

[pba]
category="domain_abuse"
statuses=["New"]
priority="High"
lookback="-1d"

[enrich]
fields=["links"]

The script can be rewritten as below.

from pathlib import Path

from pydantic import BaseModel

from psengine.config import Config, ConfigModel, get_config
from psengine.enrich import LookupMgr
from psengine.playbook_alerts import PlaybookAlertMgr

CONFIG_PATH = Path(__file__).parent / 'int_config.toml'


class PBAConfig(BaseModel):
    """Config for playbook alerts."""

    category: str
    statuses: list[str]
    priority: str
    lookback: str


class EnrichConfig(BaseModel):
    """Config for IOC enrichment."""

    fields: list[str]


class IntegrationConfig(ConfigModel):
    """The main integration config."""

    pba: PBAConfig
    enrich: EnrichConfig


Config.init(config_class=IntegrationConfig, config_path=CONFIG_PATH)
config = get_config()

pba_mgr = PlaybookAlertMgr()
enrich_mgr = LookupMgr()

alerts = pba_mgr.fetch_bulk(
    category=config.pba.category,
    statuses=config.pba.statuses,
    priority=config.pba.priority,
    created_from=config.pba.lookback,
)

domains = [alert.panel_status.entity_name for alert in alerts]
enriched_domains = enrich_mgr.lookup_bulk(domains, 'domain', fields=config.enrich.fields)

for enriched in enriched_domains:
    print(enriched)

The code itself is definitely longer, however we gain on maintainability, since now a person without development experience or inner understanding of the application can change the config to meet the new requirements.