Rev Your Engines: The SDK Generation Race is About to Begin!

Hey, speed demons and code maestros! 🚀 Buckle up for an adrenaline-fueled race through the SDK generation landscape featuring the contenders: APIMatic, Speakeasy, and OpenAPI Generator. As a seasoned SDK builder in the API realm, I'm taking these powerhouses for a spin to determine which emerges as the ultimate champion.

Today’s Race Schedule

• Pit Stop 1: Validation and Linting
• Building the Car: Code Generation
• Starting Line: Code Compilation
• Test Drive: Code Evaluation
• Victory Lap: Documentation

So, fellow code warriors, fasten your seatbelts and join me. Let's determine which contender will take home the coveted checkered flag in this epic battle of code prowess! 🏁

Disclaimer:

Some modifications were made to Lob’s OpenAPI definition to prepare for our race. We removed properties that are beta and premium and modified property values that are unique to a specific account, i.e., an ID for an existing object (template, address, etc.). This is advice we would provide a company to create runnable code snippets for all users. The modified version of Lob's OpenAPI definition we used is available here.

APIMatic

We found two idempotency keys defined with similar names, Idempotency-Key and idempdency_key, resulting in a duplicate variable. We removed one of the idempotency_key parameters.

Speakeasy

Three arrays did not have the items attribute defined, so we corrected that and removed a script tag that blocked our test.

OpenAPI Generator

The merge_variables default value broke code generation, so we corrected this in our API definition.

You can put your API definition to the test with APIMatic.

Pit Stop 1: Validation and Linting

In the world of SDK racing, precision is key. Before we hit the track, we're making a pit stop for validation and linting. Which SDK ensures a clean and error-free journey, smoothly navigating Lob's API definition with its oneOf, allOf, and anyOf twists and turns? Let's kick off the race by inspecting their linting and validation prowess.

OpenAPI Generator

In testing OpenAPI Generator, I found the validate command wasn’t very useful in surfacing issues. I recommend running the generate command to receive more complete linting and validation. The warnings centered on correcting reserved words, ignoring maxLength, pattern validation, and examples on the requestBody while acknowledging limitations around allOf support. Sadly, the warnings lack line information or suggestions to address these warnings.

Speakeasy

Speakeasy also provides a CLI with a validate command that works better than OpenAPI Generator. Their validation results are more readable with line information. Unfortunately, no suggestions for fixing issues are provided. I also received a warning: “X is potentially unused or has been orphaned.” My investigation found these warnings to be false positives.

APIMatic

APIMatic’s validation provides actionable feedback for your API definition to improve code quality. Each warning includes details and links to documentation about the violation with suggestions for fixing it.

Validation Score	🟩	🟨	🟥

Building the Car: Code Generation

Once the engine is finely tuned, it's time to build the race car. Code generation is our assembly line, where APIMatic, Speakeasy, and OpenAPI Generator vie for the title of the most reliable pit crew. Who constructs the SDK with the precision of a seasoned mechanic? Let's tighten those bolts and see which one builds the meanest machine.

Comparing SDK code generators in every language can be time-consuming. I’ve found from experience that strongly typed languages like C# and Java are suitable for uncovering issues other languages may let slip by. For this reason, I chose Java to compare the three generators.

SDK generators will let you pass if your OpenAPI definition is syntactically correct. You can generate code, but you’ll also see warnings that some aspects of your API definition could be problematic. To assist you, code generators will remove or ignore things in your API definition that may interfere with successfully generating code.

Why do SDK code generators allow you to proceed? It’s a choice that allows you to experience the tool and examine the output without requiring you to spend hours resolving warnings of varying severity.

Will you want to optimize your API definition before publishing your SDKs? Yes, and APIMatic’s VSCode extension with 1200 rules specifically around code generation is an excellent tool for the job.

For those still in the evaluation phase, don’t spend too much time optimizing your API definition, as each code generator enforces slightly different rules; you’ll go crazy trying to please them all.

Code Generation	🟩	🟩	🟩

Using Lob’s API definition, I successfully generated Java SDKs using each tool.  🎉

Speakeasy did have the following warnings during code generation.

• The component is potentially unused or has been orphaned.
• Only enum types of string or integer are supported. Enum type won't be generated and will be treated as base type
• Enum is nullable but does not contain a null value
• anyOf should only contain types which are compatible with each other, use oneOf if the response can only match one type at a time
• anyOf will not be supported by Speakeasy, treating as oneOf
• missing schema type for X component, but found properties treating as object

Starting Line: Code Compilation

Engines roar as we move to the starting line. Code compilation is the moment we ignite the fuel and unleash the horsepower. Which SDK delivers a seamless start, ensuring the code compiles with the precision of a pit crew orchestrating a Formula 1 pit stop? The contenders are revved up, and it's time to hit the track.

APIMatic compiled successfully without any errors or warnings. Unfortunately, both Speakeasy and OpenAPI generators generated SDKs that failed to compile with 32 and 100+ errors, respectively.

To keep the race going, I got behind Speakeasy’s car and pushed it. From my previous experiment, I found inline headers in the Petstore API broke Speakeasy, so I removed all referenced headers in Lob. To my surprise, the generated Java code with Speakeasy compiled.

Note: removing API features is not recommended, but it was necessary to continue our race.

Sadly, I couldn’t get OpenAPI generator past the failed compilation.

Code Compilation	🟩	🟥	🟥

Test Drive: Code Evaluation

The rubber meets the road as we take the remaining SDKs for a test drive. Navigating Lob's API definition is our challenging course, filled with oneOf, allOf, and anyOf curves. How well do our contenders handle the heat? We're evaluating the experience of using these SDKs. Can we make that first API call, and is the code idiomatic?

For this comparison, I created a sample application for APIMatic and Speakeasy to retrieve a list of addresses and print the address's street, city, and zip code.

APIMatic

I copied and pasted the code sample from the documentation into my IDE to retrieve a list of addresses. The code ran successfully and retrieved a list of addresses, which was a good sign. The addresses returned can be one of two types: a US or an international one. These models have slightly different properties. For example, the US has zip codes while International addresses don’t. APIMatic’s Java SDK has an address container class with a match method to handle this. Using the match method, I can determine the address type and then access the corresponding data using getters and setters, which is the idiomatic way to interact with models in Java.

Speakeasy

I copied and pasted the provided code sample for the list addresses endpoint from the ReadMe. Running the code, I didn’t receive a list of addresses but a 422 status code. It turns out the code sample provided included malformed parameters. Once I figured out the cause of my 422, I removed the optional parameters and successfully retrieved a list of addresses.

The address list response uses a generic object to hold the data. Because the data structure is unknown, I utilized a JSON tree structure to traverse and access address details. While the resulting code works, it is a suboptimal experience for Java developers.

OpenAPI Generator failed to compile and, therefore, was not included in the test drive.

Test Drive Java SDK	🟩	🟨	⬛

Victory Lap: Documentation

As we approach the finish line, documentation becomes our victory lap. Clear, concise, and user-friendly documentation is like a well-designed pit board guiding you to SDK mastery. APIMatic, Speakeasy, and OpenAPI Generator boast top-notch documentation, but who provides the smoothest ride toward victory? The final stretch is where we evaluate their documentation to determine the true champion of this high-speed SDK race.

What type of documentation do I include in my comparison? Any docs generated to support the SDK code library, including a getting started guide, SDK reference, code samples, and use case guides.

OpenAPI Generator

A ReadMe markdown file is generated with installation information for both Maven and Gradle and a complete code sample as a getting-started guide. The SDK reference generated is a collection of markdown files. That includes the following for each endpoint:

• Parameters
• Return type
• Authorization
• HTTP request headers
• HTTP response details
• Complete code sample

OpenAPI Generator’s code samples look runnable, but I noticed the create methods would fail as new objects don’t set any initial values.

Speakeasy

Speakeasy’s ReadMe only provides Gradle install instructions. A complete code sample is provided as a getting started. The Speakeasy SDK reference is a collection of markdown files and provides fewer details than the OpenAPI Generator.

• Parameters
• Response
• Code Sample

The code samples included in the SDK reference are complete files with import statements. Like OpenAPI Generator, the code samples look runnable, but when testing the listAddress method, the default parameters resulted in a 422 status code.

APIMatic

APIMatic’s ReadMe is similar to OpenAPI Generator and Speakeasy, focusing on installing and getting started using the code library. A big difference is the build and installation instructions walk a developer through setting up a sample Eclipse project with the SDK as a dependency.

In addition to a ReadMe, APIMatic provides a hosted developer portal. The portal includes API reference docs and an SDK reference for each language. It’s easy to navigate and explore the methods, models, enums, and exceptions supported by the SDK.

The developer portal includes interactive documentation. Modifying property values in the docs update code samples in real-time. You can make live API calls from the documentation, similar to a Postman collection. Lastly, code samples default to a condensed display, but through the config settings, can show a complete runnable code sample. While testing the code sample address list, it ran as expected.

Conclusion

In the thrilling race through the SDK generation landscape, we pushed APIMatic, Speakeasy, and OpenAPI Generator to their limits, exploring validation, code generation, compilation, test drive, and documentation. The checkered flag has been waved, and after navigating the twists and turns of Lob's API definition, APIMatic emerges as the undisputed winner of our high-speed showdown.

Final Score Card

Validation and Linting	🟩	🟨	🟥
Code Generation	🟩	🟩	🟩
Code Compilation	🟩	🟥	🟥
Test Drive Java SDK	🟩	🟨	⬛
Documentation	🟩	🟨	🟨

To further validate our findings around code generation and compilation, we put ten additional API definitions through these three SDK generators, and the results are below. The yellow warning symbol means there were problems with the languages listed.

It’s time to put your API definition to the test with APIMatic.

Your 2-week trial gives you access to all features to test the platform’s SDK generation capabilities.