fsagent

fsagent is a Golang Application to perform various standard actions triggered by various events. FSAgent is highly customizable. And there is a blockly based editor, too.

Name

The name FSAgent was originally the shorthand for File System Agent.
With the support of additional triggers and data sources outside of file system events in the monitored folders, a new meaning for the letters F and S must be found. Currently, I prefer the definition Free Service Agent.

Why

WARNING: sad true reality

When planning and developing new (open source) applications / systems, you usually use the latest technologies (containerization (e.g. Docker), message queues (e.g. RabbitMQ, ZeroMQ, ActiveMQ, ...), databases (e.g. PostgreSQL, MariaDB, Redis, MongoDB, ...), ...), but if you're working for a non-startup Company, you often have to deal with old legacy enterprise applications.
These applications do not have modern interfaces, many are decades old. The most modern communication channels of these applications are mostly FTP uploads and emails. I really mean FTP, not those new and fancy SFTP Servers.
However, many of these applications do not even have the functionality to upload, but can only store files in directories and need other applications such as Bat, Blat or Outlook to upload. Bat or Blat is not fundamentally bad, but if your whole business depends on software like Blat, you have a big problem.
In my free time, I have written this program for replacing such "interfaces". It is not just a replacement for Bat or Outlook, it monitors directories and executes predefined actions for new files. The configuration is kept as simple as possible (and will become even easier) to be done by anyone in IT departments, not just programmers.
The first goal was the elimination of the biggest pain in the ass. Gradually, however, it is also planned to extend fsagent for the service composition/orchestration of other protocols such as HTTP(S), AMQP, WebSub (PubSubHubbub).

Install

fsagent can easily installed by the go get-command:

go get simonwaldherr.de/go/fsagent

Quickstart / Schnellstart

English

Fast local test setup:

1. Install Go and, if you want ORCA workflow persistence or the dbwrite action, sqlite3.
2. Start FSAgent with the HTTP example config:

   go run ./cmd/fsagent ./cmd/fsagent/web-example-config.json
   

3. Open:
   • http://127.0.0.1:8080/orca for the ORCA editor
   • http://127.0.0.1:8080/upload for file uploads
4. Test an inbound trigger quickly:

   curl -X POST http://127.0.0.1:8080/mail -d 'subject: test'
   curl -X POST http://127.0.0.1:8080/irc -d 'PRIVMSG #test :hello'
   

5. If external CDNs are blocked, the ORCA page automatically switches to a local fallback editor so the workflow API remains testable offline.

Deutsch

Schneller lokaler Testaufbau:

1. Go installieren und für ORCA-Workflow-Persistenz oder die Action dbwrite zusätzlich sqlite3.
2. FSAgent mit der HTTP-Beispielkonfiguration starten:

   go run ./cmd/fsagent ./cmd/fsagent/web-example-config.json
   

3. Öffnen:
   • http://127.0.0.1:8080/orca für den ORCA-Editor
   • http://127.0.0.1:8080/upload für Datei-Uploads
4. Einen Trigger schnell testen:

   curl -X POST http://127.0.0.1:8080/mail -d 'subject: test'
   curl -X POST http://127.0.0.1:8080/irc -d 'PRIVMSG #test :hallo'
   

5. Falls externe CDNs blockiert sind, schaltet die ORCA-Seite automatisch auf einen lokalen Fallback-Editor um und bleibt so offline testbar.

Config

fsagent can do many things, these can be defined and configured with json files.

start the fsagent daemon with go run fsagent.go config.json or compile a binary (go build) and run it with ./fsagent config.json.

the config.json file could look like:

[
  {
    "verbose": true,
    "debounce": true,
    "folder": "/mnt/prod/Server/Transfer/701/%Y/%m/%d/",
    "trigger": "fsevent",
    "match": "^[0-9]+\\.[Tt][Xx][Tt]$",
    "action": [
      {
        "do": "sleep",
        "config": {
          "time": 500
        },
        "onSuccess": [
          {
            "do": "mail",
            "config": {
              "name": "mail",
              "subject": "Lorem Ipsum",
              "body": "dolor sit amet",
              "from": "notification@company.tld",
              "to": ["example@domain.tld"],
              "cc": ["example2@domain.tld"],
              "bcc": ["example3@domain.tld"],
              "user": "notification",
              "pass": "spring2018",
              "server": "webmail.domain.tld",
              "port": 587
            },
            "onSuccess": [
              {
                "do": "move",
                "config": {
                  "name": "success/$file_%Y%m%d%H%M%S"
                }
              }
            ],
            "onFailure": [
              {
                "do": "move",
                "config": {
                  "name": "error/$file_%Y%m%d%H%M%S"
                }
              }
            ]
          }
        ]
      }
    ]
  }
]

Trigger

Currently there are two triggers available, the most important trigger is filesystem event trigger based on fsnotify. If this is not possible (e.g. if you work on mounted drives and the fs event comes from a different system) you can use a ticker as trigger.

Trigger	Info
fsevent	file system event based on fsnotify
ticker	checks for new files at a customizable frequency
http	files can be uploaded via a web form
mail	receives raw mail payloads via HTTP POST and forwards them as files
irc	receives raw IRC message payloads via HTTP POST and forwards them as files

When using the http trigger, FSAgent now also serves an embedded ORCA editor at /orca with workflow APIs at /api/orca/workflows. Set "workflowDB": "orca.sqlite" in the HTTP config to persist graph workflows in SQLite.

Actions

There are some ready-made actions, but you can easily create others yourself.

Action	Info
Copy	creates a copy of a given file at the specified destination
Delete	removes a file
Move	moves a file to a new location
Decompress	decompresses a file
Compress	compresses a file
HttpPostR.	sends the content of a file in a HTTP Post Request Body
Webhook	sends file content to external webhook/API interfaces
OpenAI	sends file content to OpenAI chat completion APIs
LMStudio	sends file content to LM Studio OpenAI-compatible APIs
Ollama	sends file content to Ollama for local LLM workflows
RAG	runs retrieval-augmented prompts with additional context files
DAG	triggers DAG/workflow orchestrators with file-based run payloads
DBWrite	writes filename and content into a SQLite table
SendMail	sends the file as mail attachment
Sleep	waits for a specified duration

Todo / Contribute

Informations about the license, how to contribute and a list of improvements to do are in separate files.