Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Gil Feig is the co-founder of Merge, a San Francisco-based startup that helps users build customer-facing integrations with third-party tools, giving access to over 50+ systems and adding new platforms every week. Previously, Feig was head of engineering at Canvas and led projects at Wealthfront and LinkedIn. A graduate of Columbia University, he lives and works in San Francisco.
Software hasn’t always been connected.
The Path to APIs
Thirty years ago, just as applications became exponentially more ingrained in the core processes of companies, most programs existed in a silo. These on-premise solutions (on-prem), hosted either on personal computers (PCs) or within local data centers, were built to fill small, individual roles within the overall functionality of a business.
One example of this comes from Core Solutions Group (CSG), which built accounting software for mattress companies in the 1990s. As customer needs grew through the decade, CSG modified its software to support those growing demands. At the same time, mattress companies began purchasing software to power their machines for stitching mattress toppers. Before, mattress companies were forced to manually copy data on materials consumed and output produced from their machinery software to their accounting software. This left room for errors and added operational complexity. By the end of the 1990s, mattress companies had at least two critical pieces of software: accounting and machining, yet no immediate way for the programs themselves to communicate. While rudimentary solutions existed for communication between applications, it was nowhere near as robust as the era of application programming interfaces (APIs).
In the early 2000s, APIs started to blossom. Representational state transfer (REST), one of the most popular modern API protocols, was introduced. Yet, as thought leadership pushed the world towards the opinion that “data is the new oil,” and should therefore flow freely between applications, historically on-premise software tugged in the opposite direction. The requirements of enterprise operations — siloed data centers, strict security requirements, and inflexibility of computational scale — dragged incumbent software back. On top of that, the need to build APIs on top of software not originally designed with APIs in mind proved to be a massive undertaking.
In response to the growing need for interconnectedness between software, generations of companies emerged to provide specific solutions to the general problem of integrations. The following four types of API companies represent the changing challenges and approaches in the ongoing problem of integrating real-time data movement:
4 Types of API Companies
API Company Type #1: On-Prem to Cloud
The first wave of companies to emerge bridged on-prem solutions to the newly rising cloud.
MuleSoft and Boomi were, and still are, two of the most notable players to tackle the problem. They provided a combination of code toolkits and visual interfaces to remove a lot of the boilerplate work around building APIs. Suddenly, there was a bridge between on-prem and cloud that didn’t require profit-shaking investments.
However, software engineering work was still required to connect all of these new, emerging APIs. Companies never have enough software development resources, so connecting internal APIs, like their accounting software with their machinery software, still proved a challenge — despite all of the interfaces for communication available.
This lack of development resources opened the world up to another category of API companies.
API Company Type #2: RPA and Workflow Automation
Beginning in 2006, with APIs following relatively consistent standards across a few protocols, such as simple object access protocol (SOAP) and REST, and engineering resources both expensive and in short supply, companies emerged to allow anyone to connect APIs.
While slightly varied in their approaches, the goal of these companies was to enable non-engineers to automate tasks, data movement, and data manipulation. UiPath, Tray.io, and Workato are three notable examples.
These platforms provide visual tools to extract data from APIs and UIs. Users can then manipulate and insert extracted data into other APIs and UIs. This process, also known as extract transform load (ETL), became immediately cheaper to achieve without the need for full-time software engineering involvement. Now, someone working in accounting at a mattress company could simply open up Tray.io, select to pull machinery data every hour, transform it into standard accounting metrics, and insert it into their Core Solutions Group accounting software. No coding experience is necessary.
But internal processes were only one small part of the equation. As cloud-native software boomed, it became clear that forcing customers to build their own integrations, with or without no-code automation tools, was insufficient. Why force each customer to build their own automation if platforms, like Core Solutions Group, could build native integrations with complementary software and offer them to their own customers instead?
API Company Type #3: Embedded Workflow Automation
As many companies began building out native integrations with complementary software products, the work proved, once again, demanding. Software engineers were required to build out each integration and deal with the numerous complexities introduced by varying data models and protocols across APIs.
Companies like Tray.io and Workato introduced “embedded” solutions. With these, Core Solutions Group could easily embed a UI component from Tray.io in their application and then build a generic visual workflow themselves. At that point, customers of Core Solutions Group would simply enter their credentials into the embedded UI component, and the pre-created workflows, invisible to the customer, would begin to run. This greatly simplified the integration process: Connections between software were required to be built only once, and the responsibility for those connections moved from the customer, the end user, to the product, the connected software provider.
Unfortunately, one of the biggest challenges that remained was that a company like Core Solutions Group couldn’t predict which machinery software their customers were going to use. There could be thousands of solutions used by mattress companies around the world, so even though embedded workflow automation requires less engineering, Core Solutions is still required to maintain thousands of workflows for APIs that break, change, and add functionality over time.
API Company Type #4: Unified APIs
In response to the issue of industry fragmentation came unified APIs.
The most notable in this space is Plaid, which provides a single API to connect to over 11,000 financial institutions. Before Plaid and similar companies emerged, creating fintech apps proved highly difficult. Consumers’ data was split across so many different banks that creating an app, such as Mint, to help people understand financial health meant having a team solely focused on providing integrations with every possible bank a user could use. Even with tools like UiPath to scrape and pull from the sparsely available bank APIs, it was simply financially impossible to build all the integrations needed to serve the masses.
Since Plaid, unified APIs have popped up across numerous verticals. Nylas provides a single API for all email and calendar clients, and Kloudless tackles cloud storage, calendar, customer relationship management (CRM), email, chat, and accounting. Depending on your needs, there is one for almost every vertical, including HR APIs, payroll APIs, recruiting ATS APIs, and more.
When it comes to producing production-grade customer-facing integrations where the end users are distributed across a variety of competitors in a vertical, unified APIs provide the simplest and most robust way for companies to offer integrations with every one of them.
Integration Solutions for All
The integrations industry isn’t one-size-fits-all. Similar to how, within the financial industry, credit cards and hedge funds aren’t analyzed competitively, the same approach applies to the wide variety of solutions within the integrations space.
And while this article focuses on the use cases of real-time data movement, it doesn’t even mention solutions related to other fields, such as those that bridge APIs and data warehouses, Fivetran and Census, for example. As Core Solutions Group evolved to adapting customer demands, they were met with the ability to purchase new products.
Whatever a company’s use case is, there exists a solution to reduce the massive overhead produced by critical integrations. And as new issues and nuances emerge in the field of integrations, there will always be new companies to bridge the gap and allow information to flow freely.
See more: Top 10 API Management Tools
