Building a Plugin Based Workflow Automation Platform: A Microkernel Approach

Automation tools like Zapier and n8n solve a real problem: connecting different services together without writing glue code every time. But building one from scratch teaches you something those tools hide from you, which is how much architectural discipline goes into making "plug in a new service" actually mean plug in, not rewrite half the system.

This post walks through the design of a workflow automation platform I built using a microkernel plugin architecture, a FastAPI backend, and a React plus Vite visual flow editor. I will focus less on "here is the code" and more on "here is why the system is shaped the way it is."

Why Microkernel Architecture

The core idea behind a microkernel design is simple. Keep the center of the system as small and stable as possible, and push everything that changes often out to the edges as plugins.

In this project, the core does exactly three jobs:

1. Discover and load plugins

2. Route events between plugins through an event bus

3. Manage flow execution state and scheduling

Everything else, meaning every integration like Gmail, Slack, Telegram, WhatsApp, Gemini, and any future service, lives outside the core as an independent plugin. This matters for a practical reason. If Slack changes their webhook format, I only touch the Slack plugin. The event bus, the flow engine, and every other plugin stay untouched. Compare that to a monolithic automation script where every integration is a function in the same file. One bad edit and you risk breaking something completely unrelated.

System Architecture Walkthrough

The backend is organized around a clear separation between orchestration and execution.

api/          FastAPI routes and controllers
core/         Plugin manager, event bus, flow engine, scheduler
plugins/      Independent, hot pluggable integrations
www/          React and Vite visual flow editor

The core/ folder is where the actual engineering decisions live:

Each plugin implements a common Plugin interface defined in core/interfaces/plugin.py. That contract is the entire agreement between the core system and any plugin. As long as a plugin implements name() and initialize(event_bus), the core does not need to know anything else about it. This is the part that makes the system genuinely extensible rather than extensible in name only.

Synchronous and Asynchronous Event Buses

One detail worth highlighting is that the project ships with both a synchronous event bus (event_bus.py) and an asynchronous one (async_event_bus.py), along with two separate CLI entry points, main.py and main_async.py.

The reasoning here is about testing and comparison rather than indecision. Synchronous event dispatching is easy to reason about and easy to debug, since events fire and resolve in a predictable order. But it does not scale well once you have multiple plugins doing network calls, like polling Gmail while also waiting on a Gemini summarization request. The async event bus processes those events concurrently, so one plugin's API latency does not stall the others.

Having both versions available made it possible to directly compare throughput and behavior under load, which is a useful exercise if you are trying to understand asyncio's actual benefit rather than just assuming it helps.

The Visual Flow Editor

The frontend, built with React, Vite, and React Flow, gives users a canvas to drag, drop, and connect triggers and actions visually rather than writing configuration files by hand.

The component structure keeps this simple:

Communication with the backend goes through a typed api.ts client, which keeps the frontend from having to guess at response shapes. This matters more than it sounds like it should, because a visual editor is only as trustworthy as the accuracy of the state it displays. If the canvas shows a flow as running when the backend actually failed to start it, users lose confidence in the tool fast.

Secrets Without Leaking Them

Every plugin that talks to an external service, like Gmail or Slack, needs credentials. The system handles this by letting each plugin define which of its configuration fields are secrets inside its metadata file, either settings.xml or plugin.yml.

When a user enters credentials through the web UI, the backend writes them into a local .env file inside that specific plugin's folder, for example plugins/gemini_plugin/.env. On initialization, the plugin manager checks for that file and loads it into the execution environment automatically. A root level .gitignore rule ignores every .env match across the repository, so credentials never get committed by accident even if a contributor forgets to think about it.

This pattern keeps credential handling local to each plugin rather than centralized in one big secrets file, which reduces the blast radius if something goes wrong with any single integration.

Preventing Duplicate Processing

Trigger plugins that poll for new data, like the Gmail plugin, run into an obvious problem. If you poll an inbox every thirty seconds, how do you avoid reprocessing the same email over and over?

The approach used here has two layers:

1. Record the workflow's start timestamp the moment it is activated, and filter out any fetched message older than that timestamp.

2. Maintain a cache of already processed message IDs during the active session, so even messages that arrive within the same polling window are not handled twice.

Neither of these ideas is complicated on its own, but skipping either one leads to the classic automation bug where an old email suddenly triggers ten duplicate notifications the moment a workflow starts.

Installing Plugins Without Restarting

Because the system is built around dynamic discovery rather than hardcoded imports, new plugins can be installed while the server is running.

curl -X POST http://localhost:8000/plugins/install \
    -H "Content-Type: application/json" \
    -d '{"source_path": "external_plugins/sample_external_plugin", "force": true}'

There is also a path for uploading a packaged .zip archive directly, which the server extracts, validates, and registers automatically. This is the practical payoff of the microkernel decision from earlier. Because plugins only depend on the shared interface and not on internals of the core, the server can accept, load, and initialize a completely new integration without a restart or a redeploy.

What This Project Actually Demonstrates

Beyond the individual features, the value of this project is in the architectural pattern itself. A microkernel design forces a specific kind of discipline: the core has to stay boring and stable, while all the interesting, changing, service specific logic gets pushed to the edges. That tradeoff pays off directly in extensibility, since adding WhatsApp support or a new AI summarization plugin never required touching the flow engine or the event bus.

If you are building anything that needs to integrate with a growing list of external services over time, whether that is an automation platform, a notification system, or an internal tooling dashboard, this pattern is worth considering early. Retrofitting a plugin system onto a monolith later is far more painful than designing around one from the start.

Getting Started

# Backend
pip install -r requirements.txt
python api/server.py
# Swagger docs available at http://localhost:8000/docs

# Frontend
cd www
npm install
npm run dev
# Visual editor available at http://localhost:5173

From there, the plugin directory is the natural place to start exploring. Each plugin is small enough to read end to end in a few minutes, which is really the whole point of the architecture.