Running Plugins in Production

Plugins run in-process with the FreeSDN API. That is a deliberate trade: it makes the SDK fast and simple, and it means a plugin is application code, not a sandboxed guest. This page is about what the platform does to keep that trade honest, and what it does not do.

What supervision covers

Every entry into plugin-authored code goes through a supervisor: event handlers, REST routes, AI assistant tools, lifecycle hooks, and scheduled runs. For each call it records duration and outcome, keeps a rolling error budget per plugin and per operation kind, and quarantines a plugin that trips the budget.

Before this existed, a plugin whose on_event raised on every single event did so indefinitely. The event bus swallows handler exceptions by design, so nothing degraded and nothing surfaced. The log line read Error in event handler for device.updated - with no plugin in it.

Signal	Where
Per-plugin call counts, error rate, p95 latency	GET /api/v1/plugins/supervision
One plugin’s state, alongside its own health check	GET /api/v1/plugins/{id}/health
Quarantine state in the UI, with a lift button	Plugins page, expand a plugin
freesdn_plugin_operations_total{plugin_id,operation,status}	/metrics
freesdn_plugin_operation_duration_seconds{plugin_id,operation}	/metrics
freesdn_plugin_quarantined{plugin_id}	/metrics

/health also degrades while any plugin is quarantined, so the condition is visible to whatever already watches that endpoint.

Error budgets and quarantine

A plugin is quarantined after five consecutive failures of one operation kind. The threshold is not one on purpose: a transient upstream blip should not quarantine a plugin whose only fault was being downstream of it.

While quarantined:

• its event handlers are not called;
• its REST routes answer 503 with a Retry-After header;
• its AI tools return a non-retryable error to the assistant;
• its lifecycle hooks still run, so a broken plugin can always be disabled or uninstalled. Refusing on_uninstall because on_install failed would trap it in place.

The first quarantine lasts two minutes. Each repeat doubles it, up to an hour, so a plugin that is simply broken stops consuming resources at a growing interval instead of retrying forever. A single success clears the streak; the error rate stays visible, because an intermittent plugin should not be able to accumulate its way to a quarantine over a week while looking fine.

An operator who has fixed the cause can lift it immediately rather than waiting:

curl -X POST https://freesdn.example.com/api/v1/plugins/my-plugin/supervision/reset \
  -H "X-API-Key: $FREESDN_API_KEY"

Secret settings

Mark a credential in your settings_schema and the platform takes care of it:

settings_schema:
  type: object
  properties:
    api_token:
      type: string
      secret: true          # or: format: password
      description: API token for the upstream service

What that changes:

• the value is encrypted on write and stored under a separate entry;
• the plaintext twin is blanked, so a value written before the field was marked secret does not linger beside the encrypted one;
• GET /plugins/{id}/settings returns "***" when set and "" when not - never the value, and never the ciphertext either;
• settings.get("api_token") in your plugin keeps working, reading the encrypted value transparently.

That last point is what makes the declaration safe to add to a plugin that already ships. Without it, marking a field secret would turn the value your plugin reads into an empty string, and an empty credential is indistinguishable from an unconfigured one.

Mark credentials, not every string

A Discord or Slack webhook URL is a credential - the path is the secret, and anyone holding it can post as you. A Gotify base URL is not. Over-marking has a real cost: an operator can never see what they configured.

The mask means “unchanged”

A configured secret reads back as ***. If you are writing a client that reads settings and writes them back - a script, a config-management step, anything - send *** unchanged for any credential you are not deliberately replacing. The endpoint treats that exact value on a declared-secret field as leave this one alone.

This is the shape of a bug worth knowing about, because the natural thing to write is the thing that used to break:

GET  /plugins/zabbix-sync/settings     ->  {"zabbix_token": "***", ...}
# change one unrelated field, send the whole object back
PUT  /plugins/zabbix-sync/settings     ->  200

That used to encrypt the literal three asterisks as the token. The real credential was gone, the plugin then failed to authenticate, and the settings page still showed the field as configured - so the one place you would look said it was fine. The endpoint now refuses to store the mask.

Settings outlive an uninstall, unless you say otherwise

Uninstalling a plugin removes its files everywhere and keeps its stored settings, credentials included. That is what makes uninstall-reinstall a safe troubleshooting step: the plugin comes back configured.

The cost is that a token outlives the plugin that used it, so the uninstall dialog offers “Also delete this plugin’s settings and credentials”, and the API takes ?purge_settings=true. It purges every organization’s settings, not just yours, because a platform uninstall removes the files for everyone and leaving one tenant’s credentials behind would be the worst of both answers. The purge is audited.

Which to choose is simply which uninstall you are doing:

You are…	Choose
Fixing a misbehaving plugin	Keep settings (default) - it comes back configured
Done with the plugin	Purge - the credential goes with it

Configuring a plugin from the UI

Plugins → expand a plugin → Settings. The form is generated from the plugin’s own settings_schema, so a plugin gets a settings UI by declaring one:

• boolean becomes a switch, enum becomes a dropdown, integer/number become number fields, everything else a text field;
• each field’s description is rendered as help text, and required fields are marked;
• a field the operator has never set shows the schema default, so the form reflects what the plugin will actually do rather than a blank that implies “off”;
• a secret shows an empty box reading “Configured. Type to replace it.” Leave it alone and it is not sent at all. Type in it and the new value replaces the old one.

When you save, the plugin’s own async_validate_config runs and its verdict appears next to the form - so a bad token or an unreachable host is reported at the moment you save it rather than by a schedule failing quietly hours later. It is advisory: the settings are stored either way, because a remote that happens to be down must not stop you finishing a configuration.

The same page shows an Updates count and marks any plugin the registry has moved past, with an update action on the row.

Outbound HTTP

ctx.http is SSRF-guarded: HTTPS-preferred, redirects disabled, DNS-rebinding safe, response capped at 10 MB, timeout capped at 60 seconds, and a kwarg allowlist that stops a plugin supplying its own transport or proxy.

By default it can only reach public destinations. For a self-hosted controller that rules out most of what is worth integrating with, so the deploy owner can extend the reach:

PLUGIN_HTTP_ALLOWED_HOSTS=netbox.lan,10.0.0.50,gotify.internal

Same contract as FABRIC_WEBHOOK_ALLOWED_HOSTS: deploy-owner controlled, never reachable from plugin or operator input, still DNS-pinned and TLS-verified, and cloud-metadata endpoints are never reachable regardless.

The Authorization header reaches the destination. Proxy-Authorization, Host, X-Forwarded-*, X-Real-IP, Cookie and Transfer-Encoding are stripped, and a stripped header is logged, so a request that does not do what you wrote says so somewhere.

Scheduled runs are supervised too

A run declared in plugin.yaml counts against the same budget as everything else, and a quarantined plugin is skipped rather than run again every interval. This matters more for schedules than for anything else on this page: a plugin failing unattended every 60 seconds forever is exactly the shape of problem the budget exists to stop, and nobody is watching. See Scheduling.

Upgrades apply without a restart

Method bodies reload in place, so an upgrade is live on the next request. Only a version that changed get_router still needs a restart, and the install response says so. See Lifecycle and Resilience.

The previous behaviour, kept here because instances predating this still have it:

Before: upgrades needed a restart

FastAPI cannot unmount a route once it is mounted. Reinstalling or upgrading a plugin replaces its files and rebuilds its runtime instances, but its REST routes keep serving handlers bound to the previous version until the API restarts.

The install response tells you:

{
  "plugin_id": "my-plugin",
  "version": "2.0.0",
  "status": "installed",
  "restart_required": true,
  "note": "This plugin's REST routes were already mounted, so they will keep serving the previous version until the API restarts."
}

Everything else - event handlers, AI tools, lifecycle hooks - is on the new version immediately.

What this does not do

Supervision bounds accidental damage and makes its source obvious. It is not a security boundary, and the loader has never claimed otherwise.

An asyncio timeout only interrupts at an await. A plugin that spins in a tight synchronous loop is not stopped by any of this, and will block the event loop until it finishes. A plugin that allocates without bound will exhaust memory. A plugin runs with the API’s own database credentials and filesystem access.

The model is cooperative and trusted-author. Install a plugin on the same basis you would merge a pull request into the API: because you trust where it came from, not because the platform will contain it if you are wrong. Real isolation means a separate process, and that is a different piece of work.

The controls that are security boundaries are documented in Security Model: the capability vocabulary, the CAN-015 confused-deputy guard, the staged-write chokepoint, ZIP-slip and ZIP-bomb limits, hash-pinned dependencies, and the SSRF guard above.