Run Argo workflow from a VEBA event through OpenFaaS

I recently made posts about the VMware Event Broker (aka VEBA) to explain basic on-boarding in the FaaS and Event-Driven worlds.

As you may have noticed, the FaaS concept is perfect for use-case where the automation will be:

• Stateless
• Fast running
• No latency sensitive
• Responsible of a single thing
• Deterministic

BTW, there is a nice list of FaaS Best Practices on the VEBA documentation: Writing your own functions.

When you need to break one (or more) of the above rules, it may be necessary to rely on other kinds of automation, like Workflows.

In the following post, I will demonstrate how it is possible to forward VEBA events to a very powerful Workflow engine named Argo to run, for example:

• long-running automation
• multi steps automation
• stateful functions
• retry-able functions

This work relies on an OpenFaaS function: veba-to-argo-fn.

How does it works

This OpenFaaS function is a simple forwarder (or proxy) to execute a pre-definied Worklow Template by providing the incoming cloud-event as an input parameter of the Workflow excecution.

Pre-requisites

You need:

• A deployed VEBA instance (appliance based or K8S based): How-to on vUptime blog
• A deployed OpenFaaS instance (+faas-cli)
• A deployed Argo instance (+argo cli): Quick Start

A clone of the below repository:

1git clone https://github.com/lrivallain/openfaas-fn.git
2cd openfaas-fn/veba-to-argo-fn

Deploy the Argo echoer template WF

1argo template create echoer-argowf.yaml
2argo template list
3
4# Expected output
5NAME
6echoer

This echoer workflow template is a very simple workflow that just repeats the incoming data in its stdin (logs).

Configure the function

Argo config secret

Copy and customize the argoconfig.example.yaml file:

1cp argoconfig.example.yaml argoconfig.yaml

 1argoserver:
 2  server: argo.vlab.lcl
 3  protocol: http
 4  namespace: argo
 5  serviceaccount: argo-svc
 6  template: echoer
 7  event_param_name: message
 8  labels:
 9    through: openfaas
10    coming-from: veba
11    foo: bar

Deploy this configuration file as a new faas secret.

1faas-cli secret create argoconfig --from-file=argoconfig.yaml
2faas-cli secret list
3
4# Expected output
5NAME
6argoconfig

stack.yaml

Edit the stack.yaml according to your needs:

 1version: 1.0
 2provider:
 3  name: openfaas
 4  gateway: http://openfaas.vlab.local
 5functions:
 6  veba-to-argo-echoer:
 7    lang: python3
 8    handler: ./handler
 9    image: lrivallain/veba-to-argo
10    environment:
11      write_debug: true
12      read_debug: true
13    secrets:
14      - argoconfig
15    annotations:
16      topic: VmCreatedEvent, VmClonedEvent, VmRegisteredEvent, DrsVmPoweredOnEvent, VmPoweredOnEvent

Now we need to pull the OpenFaaS language template for the specified lang in our stack.yml file:

1faas template store pull python3

Deploy the function

1faas-cli deploy -f stack.yaml
2faas-cli list
3
4# Expected output
5Function                        Invocations     Replicas
6veba-to-argo-echoer            0               1

Test

You can also check the function from the UI and do a first test by running:

1echo '{"id": "test", "source": "sourcetest", "subject": "any", "data": {}}' \
2  | faas-cli invoke veba-to-argo-echoer

This should produce an excecution of a Worklow based on the echoer template in Argo.

Results

 1argo get @latest
 2
 3# Expected output
 4
 5Name:                echoer-7ldps
 6Namespace:           argo
 7ServiceAccount:      argo-svc
 8Status:              Succeeded
 9Conditions:
10 Completed           True
11Created:             Wed Jan 06 09:56:19 +0000 (44 seconds from now)
12Started:             Wed Jan 06 09:56:19 +0000 (44 seconds from now)
13Finished:            Wed Jan 06 09:56:22 +0000 (47 seconds from now)
14Duration:            3 seconds
15ResourcesDuration:   1s*(100Mi memory),1s*(1 cpu)
16Parameters:
17  message:           {"id": "test", "source": "sourcetest", "subject": "any", "data": {}}
18
19STEP             TEMPLATE     PODNAME       DURATION  MESSAGE
20 ✔ echoer-7ldps  echoer/echo  echoer-7ldps  26s

And in the logs:

1argo logs @latest
2
3# Expected output
4echoer-7ldps: {"id": "test", "source": "sourcetest", "subject": "any", "data": {}}

UI

Argo provide an UI to have a quick-view on the content status.

Here is the view of an echoer instance:

and a view of the printed logs:

Retries

If needed, you can re-run an existing instance of a workflow (with the same inputs) with the following kind of command:

1argo resubmit @latest --watch

VEBA event

In the same way, if you trigger on of the vCenter events configured in your stack.yaml file (like VmCreatedEvent, VmClonedEvent, VmRegisteredEvent, DrsVmPoweredOnEvent, VmPoweredOnEvent in the provided example) from you vCenter server:

1. VEBA event router will trigger the OpenFaaS function with event as an incoming data
2. OpenFaaS function will trigger the Argo worklow with the event as an incoming data

With the echoer workflow, you will be able to get the content of the event sent by VEBA and of course, you can now run a (more or less complex) workflow(s) catching the event data and making multiple actions.

Conclusion

The above example is just a very simple sample of the enabled capabilities and it was difficult to demonstrate in this blog post the behavior of the full setup without being too complex for readers.

But I strongly encourage you to test it by yourself and to provide me a feedback.

You can also fill a GitHub issue on the project if needed: New issue.