Originally published 2021-02-08 on blog.tlarsendataguy.com, and shared here with permission.
It has been almost seven months since I announced the alpha of goalteryx, the unofficial Go SDK for building custom Alteryx tools. Now with v0.5.6 it is time to go beta. There have been a lot of changes and some new features I am excited to announce! But first…
What is goalteryx?
I gave it away in the intro sentence, but goalteryx is the unofficial SDK for building custom Alteryx tools using the Go programming language. All of the tools you drag into an Alteryx workflow canvas are little plugins of code, even the built-in tools. The existing tool suite is expansive and well-designed. However, not every use-case can be anticipated and built ahead of time. One option for building a new tool is to build a macro from existing tools. But some tasks are not well-suited even for macros and are better expressed (and stabilized) in code. Historically, the coders among us have been able to use C, C++, .NET languages, or Python to create new tools. The .NET SDK is being deprecated this year, which gave me an excuse to create my own SDK for the Go programming language. The end result of that effort is goalteryx, now in beta!
How is the beta different from alpha?
In a word, very. The alpha was designed to be a very thin wrapper of cgo on top of the Alteryx C APIs. A LOT of plumbing was left to the developer to make sure tools behave properly and communicate appropriately with other tools and Designer. I was happy with this approach until last year, when Alteryx released a beta of their new Python SDK on pypi.
Alteryx’s new Python SDK is designed to handle much more of the plumbing and let toolmakers focus on the functionality of their tools. It did not take me long to fall in love with the new API, and so it was an easy decision to change the direction of goalteryx and implement this new SDK design.
My intention is to keep goalteryx in beta for as long as Alteryx keeps the new Python SDK in beta. Once the new Python SDK is released, I will do a final gap analysis and bring goalteryx in close alignment with it. After some final testing, I will then release a production version and transition to maintenance mode. Ultimately, I want goalteryx to have the same feel and (roughly) same experience as the other Alteryx SDKs.
But really, what’s different? Give me some details…
Almost everything has changed, but let’s start with the obvious: the API.
In alpha, plugins had to implement the following API:
In addition, all incoming connections had to implement a second interface:
In the beta SDK, plugins only have to implement a single interface with 4, simpler, methods:
Communicating with the engine and Designer is also easier. In the past, the SDK provided thin wrappers that required a lot of mental energy to use. Now, communications are handled via a Provider that is received during the tool’s Init method. Provider and its children provide simple entry points for sending messages and retrieving variables from the Alteryx runtime:
The last major change relates to how records are processed. Under the hood, goalteryx caches incoming and outgoing records in 4mb caches. This minimizes the number of calls into and out of the Go runtime, which has a lot of overhead. Because of this caching the SDK handles a lot of the management of outgoing connections, leaving the toolmaker a simple interface for defining and writing to outgoing connections.
There are many other, smaller, changes. I will leave it to the reader to explore the SDK on GitHub and discover them for themselves.
What’s new?
I updated the project logo to a cleaner design that more prominently features the beloved gopher. It looks great on a mask!
The biggest new feature, and the one I am most excited about, is a test harness for unit testing tools outside of Alteryx. In the alpha you would need to compile your tools and then test them in Alteryx to see if they worked. The beta comes with a test harness so you can fully test your tools end-to-end without leaving your dev environment. Even better, the test environment works whether or not Alteryx is installed, and even works on non-Windows operating systems (I have verified it works on macOS, my primary dev environment). A basic test looks something like this:
You start by instantiating your plugin and defining a configuration string you would expect your tool to receive (lines 4 and 5).
- You then register your tool with the test harness, which returns a runner object you can use to run your tool’s lifecycle (line 6).
- You can register output collectors which will collect your tool’s outgoing field information and data (line 7).
- Line 8 connects an input to your tool using the specified data file.
- Line 9 actually runs the test harness.
The output object has the following properties you can use to test your tool’s behavior:
- Config: contains the fields your tool sent downstream.
- Name: the name of the output anchor the collector is attached to.
- Data: a map containing the data the collector received from your tool. It is a map[string][]interface{} which allows you to look up values by field name and row index.
- Progress: the last progress value (between 0 and 1) the output collector received.
The input file is a text file that is loosely based on a pipe-delimited format. It requires header rows for field names and types. Rows of data follow and each field expects the data in specific formats. An example file is below:
Note that the extra whitespace to make the columns line up is not required; it is there purely for aesthetic human-readability. The idea behind this file format is to allow you to precisely define incoming data for your tests while also allowing you the freedom to format the file as needed to make it easily decipherable.
And that’s the new beta! For more details take a look at the Readme in the repository. Let me know your thoughts as you build out custom tools in Go and feel free to submit issues on GitHub. Happy coding!
Responses