Architecture
This is the heart of the Didact Platform. I say platform because this is truly a comprehensive solution. There are several different focal points of the design that, cumulatively, offer a self-contained, scalable job orchestator that has persistent storage, a full backend, and a user interface.
The problem may be familiar, but the solution approach is novel.
Decoupled and Standalone
No more defining background jobs in the Program.cs files of other applications. No more hijacking their dependency injection systems. No more tightly-coupled job orchestration.
We also don't want our methods and arguments saved to a storage location elsewhere. We just want an engine to operate off of pipeline metadata and let us keep our code where we want.
Class Library Project
Instead, we want to define pipelines - called Flows in Didact - from a centralized .NET class library project. Then, when the class library project publishes its set of DLL files, we want an execution and task engine to dynamically detect and load the DLL files, write metadata to the database and setup job triggers, and begin asynchronously and intelligently executing our pipelines.
Blocks and Flows
We also want the ability to define our Flows how we want. We want a set of pre-made execution Blocks to run our code in for observability, tracebility, and standardization. But we also want the freedom to write whatever code we want inside a Flow. Think of the power of tasks in Apache Airflow without the strict requirement to use them.
Flexible Code
We also don't want a begruding system of passing data between our Blocks. We just want to write normal, sensible code where we typically use loops, pass data around, and express our true creativity as developers.
First Class Dependency Injection
For the sake of testing, for the sake of simplicity, and for the sake of SOLID principles, we want to utilize the native .NET dependency injection system in our Flows.
Persistent Storage
We want our Blockflows persisted across engine restarts and friendly to both on-premises and cloud-hosted environments. We want a battle-tested, highly-reliable RDBMS to persist our metadata, and what better choice than .NET's brother, SQL Server and Azure SQL?
Dashboard UI
And I'll be the first to admit: even though I'm a developer, I love seeing a good UI. If we are going to be firing thousands, or hundreds of thousands, or who knows how many jobs, then we would like a beautiful, intuitive user interface and realtime monitoring dashboard to come along with it. Take a brief survey of background job libraries in .NET, and you will see that hardly any of them even ship a UI.
Stay informed
Stay informed about the latest updates or follow the founder's journey. Also, there will eventually be a commercial offering for Didact, and you will receive discounted pricing.