Fivetran and Stitch are great. They extract your data from lots of popular software tools and databases. And they load it into your data warehouse, where you can transform, model, and enrich the data.
But you need data from a less popular tool. Managed ELT services like Fivetran and Stitch may not support the tool’s API. Or, they may not support the newest version of the API.
What can you do?
1. Check the roadmap
Reach out to your Fivetran or Stitch account rep. They might have the API you need on their roadmap. If so and you don’t mind the wait, that may be the best option.
2. Use a no- or low-code tool
A no-code or low-code API reader might be a good option if you don’t want to write code. And if the API you need is low volume and simple.
Consider Integromat, Parabola, or Paragon. They have a wide selection of connectors themselves. They also have a graphical interface that lets you setup a custom API connection. And they will periodically extract data for you.
3. Ask the vendor for a work around
Some vendors under advertise features to get data without the API.
For example, they might ship data to your AWS S3 bucket. Or they might give you access to a read replica of a database.
Fivetran or Stitch can extract data from AWS S3 or from the database. And then load the data to your destination.
Vendors usually reserve these work arounds for their larger customers. But even if you’re not, it doesn’t hurt to ask nicely.
4. Search for an unofficial connector to the API
Both Fivetran and Stitch have templates for writing custom connectors. But maybe some kind soul has already written one for the API you need!
Stitch sponsors Singer.io. It’s an open source specification for connectors. Try searching Github for unofficial Singer connectors, called “taps” and “targets”. I used the search query: “singer tap -user:singer-io”. And here’s an example of a result.
You might also search Github for unofficial Fivetran connectors, called Cloud Functions. I used the search query: “fivetran function -user:fivetran”. Cloud Functions are new, so there’s only one result! But hopefully more will be published over time.
No luck on Github or the web? You could ask in a community like Locally Optimistic or the Singer.io Slack.
If you find a connector to use, hurray! Now you have to host the code to hook into Stitch or Fivetran’s loading infrastructure. Read more about how Stitch’s custom connectors work and how Fivetran’s Cloud Functions work.
Word of caution. The author has no obligation to update the connector or to help you. If you don’t fully understand the code, you might not be able to fix it when it breaks. And worst of all you’re at risk of getting bad data and not knowing it.
5. Write your own connector
Let’s say you or someone on your team can code and are willing to figure out the details of the tool’s API. And if you haven’t found an existing connector, your best bet might be write one yourself.
Again, read more about how Stitch’s custom connectors work and how Fivetran’s Cloud Functions work.
For Stitch’s Singer taps, you can submit your work, and they might include it as an officially supported tap! Fivetran does not currently have a process to submit Cloud Functions to be officially supported.
In any case, you can open source your code on Github. Even if you don’t plan on maintaining or supporting the code, just make that clear in the project description. And perhaps the code itself will help others!
Like what you read?
(Photo by Jordan Heath on Unsplash)