AI agents are becoming a core part of enterprise integration - not just for pre-built flows like ETL and API aggregation, but for ad-hoc, on-demand orchestration inside real-time agent workflows.

As adoption accelerates, a key question is whether LLMs can reliably navigate the complexity of real-world API estates.

So we set out to answer:

We recently completed a detailed study into exactly this - measuring LLM accuracy when planning API orchestration tasks under increasingly realistic conditions.

We gave the agents a task (in the form of an “email from my boss”), and an (ever increasing) collection of OpenAPI specs, and asked them to design a solution.

We then scored them against scoring criteria, running each test 30 times, and averaged the results.

If you’re in a hurry, here’s a link to the results.

• An executive summary (5 pages)
• The full research results (40 pages)

​

Key findings

We found four key facts:

• Planning accuracy falls to unusable levels somewhere between 60 and 300 endpoints
• Adding even minimal semantic metadata improves planning accuracy
• A declarative query language improved planning accuracy by between 73% to 142%
• Using Taxi for APIs (instead of OpenAPI) reduced token usage by 80%

​

Planning accuracy falls to unusable levels somewhere between 60 and 300 endpoints

Planning accuracy (more on how we measured this below) fell to ~30% and ~49% once we increased the population of endpoints to 300.

Interestingly, the degradation didn’t continue at 600 endpoints. We were surprised by this and don’t have a clear answer as to why.

Note: An “endpoint” is a Path and a Verb (eg: GET /customers/123)

# of endpoints	Sonnet 4.5	ChatGPT 5.1
60 endpoints	51.3%	62.7%
300 endpoints	30.1%	49.4%
600 endpoints	30.9%	46.4%

​

Adding even minimal semantic metadata improves planning accuracy

• We added semantic metadata to the OpenAPI specs on fields and input parameters.
• We updated our prompts to “consider semantic metadata (expressed via OpenAPI x-taxi-type annotations), which explicitly describe the semantic meaning of data contained within each field.”
• We didn’t provide any additional information about “Taxi”, ”x-taxi-type” or “Semantic Metadata” beyond what’s already present in standard base models

The impact was a meaningful increase in accuracy compared to baseline:

	Sonnet 4.5	ChatGPT 5.1
600 endpoints with plain OpenAPI	30.9%	46.1%
600 endpoints with OpenAPI using Taxi annotations	46.4%	58.8%

Importantly, this improvement came purely from base model knowledge of semantic metadata - we didn’t provide specialized prompts, examples, or training about how to use these annotations.

​

Example of semantic metadata

We added x-taxi-type annotations to the OpenAPI specs. To reiterate - we didn’t tell the LLMs anything more about what this was. In practice, both models appeared to be familiar with Taxi and TaxiQL from their training data.

/quotes/{symbol}:
  get:
    summary: Get single quote
    operationId: getQuote
    parameters:
      - name: symbol
        in: path
        required: true
        schema:
          type: string
          x-taxi-type: com.bank.trading.Ticker
    responses:
      '200':
        description: OK
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Quote'

And to response schemas:

components:
  schemas:
    Quote:
      type: object
      properties:
        symbol:
          type: string
          x-taxi-type: com.bank.trading.Ticker
        bid:
          type: number
          x-taxi-type: com.bank.trading.BidPrice
        ask:
          type: number
          x-taxi-type: com.bank.trading.AskPrice

​

Adopting a declarative query language improves LLM planning accuracy by between 73% - 142%

We asked the agents to express data requirements using TaxiQL queries.

The result was a significant improvement in planning accuracy.

	Sonnet 4.5	ChatGPT 5.1
600 endpoints with plain OpenAPI	30.9%	46.1%
600 endpoints with OpenAPI using Taxi + TaxiQL	74.7%	85.5%

There’s some important caveats here:

• For this test run, we provided Taxi schemas, not OpenAPI.
• Critically, this reduced token usage by 80%, which meant much smaller context windows.
• LLMs were told to “outline the API calls as TaxiQL queries”
• We didn’t provide specific information about TaxiQL to the LLMs beyond what’s already in their base training

​

Using Taxi for APIs (instead of OpenAPI) reduced token usage by 80%

In the final tests (TaxiQL focussed) we sent API specs to LLMs in Taxi, rather than OpenAPI.

This dramatically reduced the consumed tokens (and therefore token size). As teams become sensitive to token spend, this is a quick win for budgets.

	Tokens consumed
OpenAPI	62,337
Taxi	12,542

For reference, here’s the same example from above, expressed in Taxi:

service QuotesApi {
   @HttpOperation(method = "GET", path = "/quotes/{symbol}")
   operation getQuote(symbol:Ticker):Quote
}
model Quote {
   ticker: com.bank.trading.Ticker
   bid: com.bank.trading.BidPrice
   ask: com.bank.trading.AskPrice
}

However, because we deferred orchestration to TaxiQL, the LLM doesn’t actually need to know about the service definitions, only the models and types, so the ACTUAL Taxi schemas ended up looking more like this:

model Quote {
   ticker: com.bank.trading.Ticker
   bid: com.bank.trading.BidPrice
   ask: com.bank.trading.AskPrice
}

(That’s 36 tokens vs 177 in the OpenAPI definition)

​

Scoring

​

What we measured - and why

We did this research to ask the question:

More specifically - we wanted to ask three questions:

1. How well do LLMs perform at API orchestration when facing real-world complexity (hundreds of endpoints)?
2. Does adding semantic metadata improve their accuracy?
3. Does the adoption of a declarative orchestration language (specifically TaxiQL) make a difference?

​

Disclosures

It’s important to be transparent about motivations - we did this research specifically because we’re investing in building in this area, and want to be sure that we’re both:

1. addressing a problem that actually exists, and
2. proposing a solution that works

However, we took care to test in a balanced way.

We didn’t give specialized training or prompting for the alternative approaches we were testing (which were adding a semantic layer, and using a declarative orchestration language — specifically TaxiQL).

Independently of the main research, we also tested how well the LLMs could produce real compiling TaxiQL queries. For this we did use specialized prompts (and we’re continuing to iterate them), but we’ve called these out separately in the research results.

​

How we measured

The agents were measured against 4 criteria:

• API Flow - The agents had to pick the correct 5 endpoints (path and verb), and sequence them in the correct order
• Correct Identifier - Multiple ID schemes were used. The agents had to recognize this, and call an API endpoint to swap IDs, passing in and reading out the correct fields
• Business logic - Agents needed to describe (but not implement) how data from API responses would be used to implement business logic checks, or to pass to the next API call
• Robustness - Data is messy - nulls and partial records exist — agents needed to describe how they’d handle this

Importantly:

• Agents only had to build an accurate plan. If they produced code, they weren’t penalized if the code didn’t compile.

To score:

• A scoring sheet defined the key elements agents needed to articulate
• We used multiple LLMs as judges in a multi-stage review process:
  • Two separate LLMs scored the output against the scoring sheet (GPT-5.1 and Gemini-2.5 Pro)
  • Both scoring results were sent back to an LLM to critique and refine the scores (Gemini-2.5-Pro)
• The first LLM reviewed critiques and had the opportunity to adjust its scores (GPT-5.1)

Each scenario was run and scored 30 times, and the scores were averaged

​

Summary

Our research shows that AI agents need a semantic layer to operate reliably in enterprise environments.

Without it, accuracy degrades to unusable levels as API complexity grows - exactly the conditions agents will face in production.

The encouraging result is that even minimal semantic metadata improves outcomes, and moving orchestration into a declarative layer (TaxiQL) improved planning accuracy substantially while reducing token consumption.

Both Taxi and TaxiQL are open source, making this approach accessible to any organization. Taxi is designed to be interoperable with existing schema languages, so you can start adding semantic metadata to your OpenAPI, Avro, SOAP, or Protobuf specs now - without retooling.

If you’d like to discuss applying this approach in your environment, join us on Slack or reach out by Email. Or you can simply try Orbital now.

(The post Benchmarking LLM Accuracy in Real-World API Orchestration appeared first on Orbital Blog.)

Depending on how you found this blog post, there’s a strong chance you know what a Data mesh is, but may be less familiar with Orbital.

Orbital is a data integration platform that works across your entire stack - APIs, databases, and streams.

However, Orbital is a wee bit different from yer average iPAAS - it’s built around semantic metadata instead of complex pipelines.

Teams define what they need using plain business terms; Orbital handles the wiring and policy enforcement automatically.

Think of Orbital as:

• dbt, for APIs and streams (and DBs too) - transforming data across operational systems, not just SQL warehouses
• MuleSoft without the laborious manual orchestration - automated API orchestration and integration
• GraphQL Federation across all data sources - without resolvers to maintain, or a single uber-graph that breaks

Unlike traditional integration tools that help you write integration code, Orbital eliminates the need for it entirely.

Teams work independently, defining data contracts in Git using standard workflows. Orbital discovers and connects these distributed sources automatically.

Given this automated, decentralized approach, we often get asked: “Is Orbital a data mesh?”

Let’s unpack that question with a little architectural honesty.

​

Shared Goals

Orbital and data mesh share the same aspirations:

• Treat data as a product
• Decentralize ownership to the teams closest to the data
• Enable self-service discovery and delivery
• Ensure governance, quality, and interoperability

However, we didn’t set out to build a data mesh platform - so even though we share a bunch of the same goals, Orbital’s ended up a slightly different take.

​

Where Orbital delivers against Data Mesh

​

Decentralized ownership through Git

In many mesh architectures, decentralization is achieved by handing infrastructure, pipelines, and release cycles to individual teams - often with lots of tooling variation.

In Orbital, decentralization happens through standard GitOps workflows:

• Each team owns their repo
• They define data products using TaxiQL and metadata annotations
• They decide when to publish by merging to their nominated “production” branch

No central bottleneck. No handoff to a data engineering team.

Example: A sales team defines a CustomerRevenue product in their Git repo. They specify that only finance can access raw salary fields. Once merged, Orbital enforces those access rules automatically - across SQL views, APIs, and Kafka streams.

​

Favor declarative, shared semantic definitions over custom code

Orbital replaces pipelines with metadata. Data products are defined declaratively and generated automatically.

This enables:

• Business ownership of what a product is
• Engineering alignment through GitOps, versioning, and reviews

Teams don’t wire up Flink jobs or Spark DAGs. They define what they need in plain terms. Orbital figures out how to deliver it.

Getting into the detail is beyond this post, so if this part piqued you’re interest, here’s a few videos that you can douse your eyeballs with:

• Avoiding breaking changes using semantic metadata
• YOW Conference 2024 - Building API layers that build themselves
• Adaptive Architectures

​

Governance: Designed decentralized, enforced globally

​

What does "federated governance" actually mean in data mesh?

In theory, “Federated Governance” means domain teams define their own governance rules - such as data access, quality checks, and naming conventions - while a platform team provides shared tools and support.

In practice, that often leads to fragmentation:

• Teams define policies inconsistently
• Rules are duplicated or conflict across domains
• Enforcement is brittle or absent

​

Orbital's federated governance: strong in some areas, gaps in others

Where Orbital excels:

Data authorization: Teams define access policies using semantic annotations in Git. These rules are enforced automatically and globally - across APIs, queries, and streams.

Example: A team declares that only HR can view Salary. That rule applies automatically anywhere salary appears - no matter who published the product.

GitOps workflows: Teams use standard Git practices (PRs, reviews, branching) to federate governance around schema changes, policy updates, and data product definitions.

Lineage tracking: Traditional data mesh implementations require teams to manually document lineage as part of their data product pipelines - a process that’s error-prone and quickly becomes stale.

Orbital captures lineage automatically at runtime as queries execute and data flows between services. This ensures lineage always reflects what actually happened, not what was designed on paper. Teams get accurate, up-to-date lineage without the maintenance overhead.

This is computational governance at its best: federated by design (each team’s data products contribute to the overall lineage graph), but captured and enforced automatically by the platform.

Where Orbital has gaps today:

• Data quality enforcement: No built-in framework for teams to define and enforce quality rules (planned for future releases)
• SLA monitoring: No federated approach to defining or enforcing performance guarantees across data products
• Schema governance: Limited tooling for teams to define domain-specific naming conventions or structural standards

The result? Orbital delivers strong federated governance for access control, development workflows and lineage - but still requires external tools or manual processes for other governance concerns.

​

Self-serve infrastructure

One of the core principles of data mesh is self-serve infrastructure as a platform - enabling teams to build and publish data products without relying on central data engineering.

Orbital strongly supports this:

Teams can:

• Define products in Git using simple metadata
• Deploy those products over API (or as pipelines) simply by committing to Git
• Use Orbital’s browser-based editor to browse, author and deploy data products
• Discover and explore available products in a self-updating catalog

Orbital automates:

• API, SQL, and stream integration
• Catalog population and lineage tracking
• Access policy enforcement

Data integration happens automatically, without requiring coordination between teams.

This is infrastructure as a platform - without having to learn Airflow or build DAGs.

​

Where Orbital differs from traditional mesh tooling

Orbital doesn’t expose low-level orchestration tooling directly. Teams don’t:

• Write Airflow DAGs
• Build Spark jobs
• Manually configure Kafka pipelines

Instead, they define what the product should look like, and Orbital handles how it’s delivered - automating integration and orchestration beneath the surface.

To be clear: this doesn’t actually contradict data mesh principles at all - it simply diverges from the tools people commonly reach for.

Zhamak has been vocal that data mesh isn’t about pipelines - it’s about intent, ownership, and interoperability.

While many teams have implemented mesh with Airflow or Spark, Orbital takes a different path. We deliver the same outcomes with less complexity, by raising the level of abstraction.

That’s a simplification - and a strength. It avoids the complexity and inconsistency of DIY mesh implementations while still giving teams ownership over semantics, policy, and product lifecycles.

​

Open Source and Technology Agnostic

Taxi, the data product language that powers Orbital, serves dual purposes: it’s both a query language for discovering data products and a metadata language for publishing semantic information about APIs and data sources.

Taxi works alongside - not in place of - your existing definitions:

• Extend OpenAPI, Protobuf, Avro, and even legacy SOAP contracts
• Add semantic meaning to your existing data without a full rewrite
• Focus on what data means, not just how it’s structured

This schema-agnostic approach enables true semantic interoperability - the loose coupling that’s essential for Data Mesh success.

The core engine is Apache 2 licensed: The Taxi language, query engine, and execution engine are all open source and available on GitHub.

Orbital itself is Fair Source and also available on GitHub. Teams can work in whatever stack they prefer while Orbital provides the unifying layer.

​

Analytics vs. operational data: where data mesh draws unnecessary lines

The early Data Mesh literature made the somewhat puzzling decision to limit its principles exclusively to analytical data.

This artificial boundary creates unnecessary silos between operational and analytical systems - exactly the kind of fragmentation that data mesh supposedly aims to solve.

Orbital takes no such stance. We’re used by both analytics and operations teams across the same organizations:

• Analytics teams use Orbital to power batch data jobs for ML pipelines and data science workflows
• Operations teams use Orbital to federate APIs and databases into real-time services powering some of the largest banks in the world

The underlying challenges - data discovery, semantic alignment, governance, and integration complexity - are identical whether you’re building a quarterly sales report or a real-time fraud detection system. Why artificially limit solutions to only half the problem?

While the Data Mesh community has chosen this hill to die on, we see it as a false dichotomy.

Good data architecture principles should work across the entire data spectrum.

​

TL;DR

Orbital isn’t a data mesh by the book. But it aligns closely with its four foundational principles:

• Domain-oriented ownership - Teams manage their own products via Git, with full lifecycle control
• Data as a product - Products have contracts, owners, and real consumers
• Self-serve infrastructure - No pipeline authoring needed - Orbital automates delivery
• Federated governance - Policies are defined decentrally, and enforced globally

And one bonus principle:

• Technology-agnostic - Taxi integrates with your existing schemas, rather than replacing them

You get the outcomes of data mesh - without the overhead of implementing one.

So is Orbital a data mesh?

Kinda.

Orbital delivers the goals of data mesh - but in a way that’s actually usable, scalable, and maintainable.

(The post Is Orbital a Data Mesh? appeared first on Orbital Blog.)

(The post Avoiding breaking changes using Semantic Metadata appeared first on Orbital Blog.)

We’re excited to announce Orbital’s partnership with Ultumus, a leader in the benchmark mastering space, with a reputation for rethinking and modernizing market data distribution. Our collaboration is set to shake up the traditional market data landscape, making it more accessible, and fair for all.

Ultumus is on a mission to revolutionize benchmark data service by providing fair and cost-effective access to industry leading market data. Orbital is delighted to support this mission, by using our advanced integration platform to eliminate the project costs that customers often face when switching data providers, leaving them locked into legacy vendors.

Orbital enables lightning-fast creation of API and data feed adaptors, backed by comprehensive observability, lineage, and traceability. Through our partnership, Ultumus customers now have access to drop-in replacements for data feeds that emulate existing formats, eliminating the need for expensive and lengthy migration projects typically associated with switching vendors.

Together with Ultumus, we’re paving the way for a more equitable and efficient market data environment, ensuring that customers have the freedom to choose without being burdened by prohibitive costs and complexities.

Click to learn more about Ultumus benchmark data or how Orbital can help with your data integration

(The post Orbital partners with Ultumus to redefine market data integration appeared first on Orbital Blog.)

If you’ve ever had to work with SOAP APIs, you know how unpleasant they can be.

They’re cumbersome to call, have heavy amounts of code-gen required, and if you’re not using Java or C#… you’re already kinda screwed.

We’re gonna to take an old SOAP APIs (which, lets face it, aren’t fun to play with), and repackage it, combining multiple calls into a single REST API.

Instead of writing piles of boilerplate plumbing code, we’ll automate all the integration work with Orbital and Taxi to annotate the SOAP APIs.

​

Taxi - A quick intro

Taxi is a relatively new entrant in the API space. It’s goal is to let developers add simple, (but type-safe) tags into their APIs, so software can understand how different APIs relate to one another.

It’s a simple way of saying “This field is the same as that field”, while keeping systems decoupled.

And the same tags we add into our APIs, we can use to query for data using TaxiQL - Taxi’s query language - which replaces all the integration plubming code we’d normally have to write.

Let’s go a little deeper, and break that down into three steps:

1. Adding Taxi metadata to the SOAP WSDL (here’s the relevant docs)
2. Using Taxi queries to build the responses we want to return
3. Publishing those queries as REST APIs

The code for this blog is available on Github

​

Background

We’re working with a public webservice at Oorsprong.org that provides Country Information.

There’s lots of different services, which return small slices of information about countries, given an ISO code.

We’d like to grab the data of a few of these services, but without having to write a bunch of SOAP code.

​

Adding Taxi Metadata to SOAP

Taxi metadata allows us to sprinkle a few additional tags into our WSDL which let mark data attributes that are the same.

For example, our CountryInfo wsdl has a section dedicated to request / response payloads.

<!-- Snippet from our WSDL - Request / Response messages for getting a Country Name -->
 <xs:element name="CountryName">
    <xs:complexType>
       <xs:sequence>
          <xs:element name="sCountryISOCode"
                 type="xs:string" 
                 taxi:type="com.demo.IsoCountryCode"/>
       </xs:sequence>
    </xs:complexType>
 </xs:element>
 <xs:element name="CountryNameResponse">
    <xs:complexType>
       <xs:sequence>
         <xs:element name="CountryNameResult" 
             type="xs:string" 
             taxi:type="com.demo.CountryName"/>
       </xs:sequence>
    </xs:complexType>
 </xs:element>

You can summarize this by saying:

I can send a CountryName request containing a com.demo.IsoCountryCode, and I’ll get back a message containing a com.demo.CountryName

Elsewhere, we can also use a IsoCountryCode to get more information, like a CountryFlagUrl:

<xs:element name="CountryFlag">
    <xs:complexType>
       <xs:sequence>
          <xs:element name="sCountryISOCode" 
              type="xs:string" 
              taxi:type="com.demo.IsoCountryCode"/>
       </xs:sequence>
    </xs:complexType>
 </xs:element>
 <xs:element name="CountryFlagResponse">
    <xs:complexType>
       <xs:sequence>
          <xs:element name="CountryFlagResult" 
                  type="xs:string"  
                  taxi:type="com.demo.CountryFlagUrl"/>
       </xs:sequence>
    </xs:complexType>
 </xs:element>

That’s a whole lotta XML, for not much info - which is one of the problems with SOAP - it’s just too darn noisy.

​

Using Taxi queries to get the data we want

Now that our WSDL is annotated with Taxi, we can get Orbital to do all the heavy lifting for us.

Rather than generating a bunch of Java classes from the WSDL, we can simply write a taxi query:

given { iso: IsoCountryCode = "NZ"}
find { 
    // Each field comes from a different SOAP service.
    // But taxi keeps this nice and succinct, composing the services
    // together we need on demand
    name: CountryName
    flag: CountryFlagUrl
    currency: CurrencyName
 }

We can send that query to Orbital, and Orbital calls all the SOAP services we need on our behalf, giving us back the data we’re looking for:

{
   "name": "New Zealand",
   "flag": "http://www.oorsprong.org/WebSamples.CountryInfo/Flags/New_Zealand.jpg",
   "currency": "New Zealand Dollars",
}

Orbital’s profiler shows us exactly what happened:

Orbital's profiler shows the sequence of calls issued to load the data we needed

And we can drill into the details of each call too:

Drilling into each call shows the request and response

​

Instant REST API

Now that we’re happy with the content of our payload, we can quickly turn it into a REST API.

Simply by checking a query into our project’s git repo, and adding an annotation, Orbital gives us a fully working REST API:

@HttpOperation(url = '/api/q/countrydata/{countryCode}', method = 'GET')
query countrydata(@PathVariable("countryCode") countryCode :  IsoCountryCode) {
   given { countryCode }
   find {
       name : CountryName
       flag: CountryFlagUrl
       currency: CurrencyName
       capital: CapitalCityName
   }
}

By adding a @HttpOperation annotation to our query, and saving it our local Taxi project, Orbital instantly exposes a REST API for us. Rather than hard-coding our query to information about New Zealand, we’ve put the country code as a part of the path.

@HttpOperation(url = '/api/q/countrydata/{countryCode}', method = 'GET')

So, if we call:

curl http://localhost:9022/api/q/countrydata/NZ 

Gives us:

[
  {
    "name": "New Zealand",
    "flag": "http://www.oorsprong.org/WebSamples.CountryInfo/Flags/New_Zealand.jpg",
    "currency": "New Zealand Dollars",
    "capital": "Wellington"
  }
]

​

Live reload for the win

Instant REST API, that live reloads

Orbital is live reloading this query, so while we’re working locally and making changes to our query, changes are instantly deployed! Once deployed, making changes is as simple as pushing to a git repository.

Check it out:

Live reloads

Adding the CapitalCityName into our request actually added a whole new SOAP request and integration into our query, which Orbital instantly reloaded behind the scenes.

​

Conclusion

In this blog, we’ve quickly repackaged a SOAP API into the exact API we wanted, without having to deal with any of the nastiness that normally comes from dealing with SOAP requests.

In the process, we’ve touched on a few of Orbital’s handy features:

• Using Taxi metadata in SOAP WSDLs
• Composing together multiple SOAP calls
• Instant REST APIs with live reload
• Viewing profiling data

The full code to run this demo locally is available on Github

This is only scratching the surface of what we can do. Orbital can also stitch this data together with other APIs (such as REST / gRPC), call serverless functions, query our Databases, or Kafka queues.

Remember, if you found this useful, please give our Github repo a star!

(The post Quickly modernizing SOAP APIs appeared first on Orbital Blog.)

​

What is Semantic Metadata?

Semantic Metadata is a way of defining a contract around the meaning of data. It lets teams create terms and definitions they agree on, and use those terms to better describe their APIs and analytics, and make software interoperable.

Teams can use Semantic metadata to define formal definitions for fields in APIs:

• “This is what a first name means”,
• “This is what a company name means”

Analytical platforms such as cube.dev tend to expand this to include shared definitions of aggregates.

• “This is what a customer is”
• “This is what we mean by ‘Active Customers’”, etc.

The core idea is that Semantic Metadata is a way for teams to create a shared understanding of what data means, independent of a single specific system.

​

Building a Taxonomy

Semantic metadata is really just a collection of terms that describe our business.

When grouped together, this is called a Taxonomy.

Taxi is a language-agnostic tool for building semantic taxonomies:

type AccountNumber inherits Int
type CreditScore inherits Decimal
type FirstName inherits String
// etc...

Semantic Metadata is designed to be shared across multiple teams. So, just like with API technologies like OpenAPI, Protobuf, etc - it’s best to have Semantic Metadata designed in a platform agnostic language.

Generators can then generate bindings / SDKs / tools as required in whichever technology consuming teams are working with.

​

Embedding semantic metadata

On it’s own, semantic metadata isn’t very helpful - it’s just a set of tags and definitions.

However, embedded in API specs, it becomes much more powerful.

Here’s an example in OpenAPI:

openapi: 3.0.1
info:
  title: ReviewsApi
  version: 1.0.0
paths:
  https://reviews/{id}:
    get:
      parameters:
      - name: id
        in: path
        required: true
        schema:
          type: string
          x-taxi-type:
            name: FilmId  # <-- Semantic metadata
      responses:
        "200":
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/FilmReview'
components:
  schemas:
    FilmReview:
      type: object
      properties:
        id:
          type: string
          x-taxi-type:
            name: ReviewId  # <-- Semantic metadata
        filmId:
          type: string
          x-taxi-type:
            name: FilmId  # <-- Semantic metadata
        score:
          type: integer
          format: int32
          x-taxi-type:
            name: ReviewScore # <-- Semantic metadata

As teams enrich their exisitng API specs with semantic metadata, tooling can start inferring relationships between APIs and data sources.

Let’s look at a (very) simplifed API for an insurance company that provides quotes:

This takes a request payload with two inputs:

{
   "noClaimsBonus" : 0.25,
   "creditScore"  : "AAA"
}

Semantically, this can be modelled as:

type NoClaimsBonus inherits Decimal
type CreditScore inherits String

model QuoteRequest {
  noClaimsBonus : NoClaimsBonus
  creditScore : CreditScore
}

This has added a small benefit of improved clarity in the docs.

However, the real payoff is in when we’re trying to get out services to work together…

​

The Payoff: Automating Interoperability

This is where semantic metadata really starts to shine.

With Semantic Metadata embedded in our API specs, we can start to infer relationships between APIs and data.

Looking at our previous example without field names, we required two inputs - a NoClaimsBonus and a CreditScore.

If we don’t have those pieces of information, we need to look them up. Which means we need to look for services that expose this data.

As our APIs are enriched with Semantic Metadata, tooling can automatically infer relationships between systems.

So, in our Insurance Quote example, while it’s unlikely we have either a NoClaimsBonus or CreditScore available from our UI, we might have something else - like a UserName or UserId.

Semantic Metadata lets us use tooling to automate the integration, linking from the Things We Know (UserName) to the Things We Want To Find Out (a Quote).

We’ll look into this in more detail in the next post.

​

A few tips & tricks...

Semantic metadata is a simple concept, and is super easy to get going with. Here are a few tips & tricks to take along your journey

​

Stay small & nimble when using distributed ownership

Semantic Metadata is intended for wide collaboration, which can be tricky.

Where individual teams are responsible for defining API definitions, semantic metadata has distributed ownership.

If the idea of Distributed Ownership is giving you sweaty palms and flashbacks to Design-by-committee meetings when your org tried to implement Canonical Domain Models, you might already be rolling your eyes.

Therefore, it’s recommended that Semantic Metadata is defined on Scalar terms only - single noun-like ideas that describe exactly one idea. eg:

• FirstName
• LastName
• DateOfBirth
• PostCode

These are relatively un-contentious to define, and - unlike domain models (which evolve as systems mature) - semantics don’t really change.

​

GitOps all the things

Semantics are a bridge between business language and software. There’s no shortage of First-generation data catalog platforms that will sell you a glorified Wiki for defining your semantic terms.

Instead, consider using Open Source tooling, that aligns with GitOps.

You get peer review, audit trails, and automated workflows through all your existing Git tooling, without having to invest in expensive enterprise tooling.

​

Staying technology agnostic

Organisations these days aren’t “Java Shops” or “.NET Shops” anymore - they’re polyglot, with tech teams choosing the tech stack that best fits the task and team.

Likewise, most API and schema technologies (eg., OpenAPI, Protobuf, etc) are language agnostic, with bindings / generators allowing them to be consumed in whichever technology consuming teams are working with.

Taxi is an example of a Semantic Metadata language, which is also platform-agnostic.

​

Summary

This has been a high level introduction into some of the ideas behind Semantic Metadata.

To go deeper, take a read into Why we created Taxi, and Using Semantic Metadata for easier Integration.

In the next article, we’ll take a look at building a simple application using Semantic Metadata to automate the orchestration.

Also, if you liked this article, consider giving Taxi a star

(The post Semantic Metadata 101 appeared first on Orbital Blog.)

Inevitably, someone links that xkcd cartoon. Luckily, that cartoon is actually really funny, so I don’t mind re-reading over and over and over.

It seems the one standard the internet can agree on is to respond to discussions about schema languages with that cartoon.

However, it’s a fair question, and one that deserves a good answer. So, to summarize:

• Yes, Taxi is another schema language. It’s different.
• Yes, we felt there need for one more. It has different goals than the others, and addresses shortcomings we felt existed.
• Yes, there’s now one more way to do things… kinda (We don’t expect you to migrate from OpenAPI to Taxi.)

​

What is Taxi?

Taxi is a language for documenting data models and data sources (APIs, Event streams, Databases).

It’s designed for describing how data relates across an ecosystem of datasources (such as an enterprise), rather than a single API.

We do this so we can automate orchestration and interoperability between data sources, without writing glue code.

Where most schema languages describe a single data source (API, Event Stream, Db, Serverless function), Taxi is designed for describing how data relates across an ecosystem of datasources (such as an enterprise).

Taxi can partially describe:

• Databases - but you can’t replace DDL with Taxi.
• HTTP APIs - but you’ll probably keep using OpenAPI / gRPC - and that’s cool.
• Message queues, like Kafka and Rabbit - but you can’t replace Protobuf with Taxi.

For example, here’s a Taxi spec describing a database, a Kafka topic, and a REST API:

service FilmsDatabase {
  table films : Film[]
}

model Film {
  filmId : FilmId inherits String
}

service FilmEvents {
    stream newReleases: Stream<NewReviewSubmittedEvent>
}

model NewReviewSubmittedEvent {
    filmId : FilmId
    reviewId : ReviewId inherits String
}


service ReviewsApi {
    @HttpOperation(method = "GET", url = "https://reviews/{id}")
    operation getReviews(id: FilmId): FilmReview[]
}

model FilmReview {
    id: ReviewId
    filmId: FilmId
    score: ReviewScore inherits Int
}

There’s enough metadata there for us to understand how everything hangs together - here’s a diagram of that spec:

Fork this diagram on playground.taxilang.org

​

Taxi embeds within existing specs

In practice, Taxi actually breaks down into two separate activities:

• Defining a set of semantic scalar types, that describe a single field, which live in a Taxi project. These are designed for sharing.
• Embedding references to those types within existing API specs.

Our example above breaks down into a few simple types:

// A taxi project.  Generally, this is committed into
// a Git repo somewhere.

type FilmId inherits String
type ReviewId inherits String
type ReviewScore inherits Int

And some other specs:

openapi: 3.0.1
info:
title: ReviewsApi
version: 1.0.0
paths:
https://reviews/{id}:
  get:
    parameters:
    - name: id
      in: path
      required: true
      schema:
        type: string
        x-taxi-type:
          name: FilmId  # <-- Taxi metadata
    responses:
      "200":
        content:
          application/json:
            schema:
              type: array
              items:
                $ref: '#/components/schemas/FilmReview'
components:
schemas:
  FilmReview:
    type: object
    properties:
      id:
        type: string
        x-taxi-type:
          name: ReviewId  # <-- Taxi metadata
      filmId:
        type: string
        x-taxi-type:
          name: FilmId  # <-- Taxi metadata
      score:
        type: integer
        format: int32
        x-taxi-type:
          name: ReviewScore # <-- Taxi metadata

​

Why do this?

At Orbital, we’re on a mission to eradicate integration code.

Even though we were heavy users of OpenAPI, we found that when consuming APIs there’s a tonne of busywork in writing glue code stitching things together.

Often, we were composing multiple APIs together to achieve a single task. Each API required more glue code, and meant we were tightly coupling to that spec - meaning when the spec introduced breaking changes, our glue code had to be repaired.

That seemed silly. Even though we had all these specs that describe what APIs do, it still fell to engineers to work out which API to call, what data to pass for inputs, and write the glue code.

Even though we had all these specs that describe what APIs do, it still fell to engineers to work out which API to call, to track down inputs, and write the glue code.

We wanted to provide a way to let APIs describe themselves richly enough that software could work out how to orchestrate them together automatically. We also had some other goals:

• We shouldn’t be relying on field names - they’re a bad proxy for semantics, and teams should be free to chose names that make sense to their domain.
• We shouldn’t force teams building APIs to replace their existing specs with something new - we want to complement what’s already in place.
• We had to be technology-agnostic. Modern enterprises are heterogeneous, so we needed to work everywhere.
• Teams building APIs need to be able to change their APIs easily, without cascading change onto consumers.

​

Producer vs Consumer - Once vs Many times

Taxi shifts the obligation of describing how things stitch together from Consumers to Producers.

Traditionally, it falls to consumers to work out how to do this.

And, it’s an expensive question to answer… it involves tracking down API specs, reading docs, and building a mental model of how things hang together.

How integration works today. Over, and over, and over.

That sucks, because consumers are a-plenty, meaning that the “how does this relate to that” is answered over and over.

Instead, with Taxi, we shift the obligation of documenting how things relate to producers, by embedding metadata in their APIs.

This means work is done once - by the teams who understand the APIs the best - the ones building it.

​

What if teams put the wrong metadata?

Yeah, that’s a problem. If teams map fields together incorrectly, then the wrong APIs are stitched together.

But, that problem exists today - every time a new consumer stitches together some APIs, they’re performing the Field Mapping Foxtrot, and there’s a chance they’ll get it wrong.

So, by leaving it to consumers to map fields together, we’re actually facing this risk over and over.

In practice, the teams who are building the API have a much deeper understanding of how their data relates to the broader ecosystem, as they know their API best. And, whilst producers can’t map their APIs to all consumers, they can attach a piece of well-defined metadata that assigns a semantic contract we all agree on.

That being said, I think tooling can help more here than we do currently. Watch this space.

​

What is TaxiQL?

TaxiQL is part of the Taxi Spec - it’s a query language that lets consumers ask for data they want using the same semantic types that producers have embedded in their API specs.

find { Film( Title == 'Gladiator' ) } as {
   title: FilmTitle
   cast : {
     actorName : FirstName + ' ' + LastName
     twitter : TwitterHandle
   }[]
   rating: ReviewScore
   reviews: ReviewText[]
}

Taxi works just as well with request-response interfaces, as well as streaming data sources:

// Whenver a new film review is emitted, give me the data I want.
stream { FilmReviewSubmittedEvent } as {
   title: FilmTitle
   cast : {
     actorName : FirstName + ' ' + LastName
     twitter : TwitterHandle
   }[]
   rating: ReviewScore
   reviews: ReviewText[]
}

By using types in the consumer contract, and combining with the metadata in producers, there’s enough information to infer how everything hangs together, and to automate the integration.

​

TaxiQL vs GraphQL

Both Taxi and GraphQL share similar goals - providing a single entrypoint for composing multiple APIs together. GraphQL has been a strong inspiration in how we’ve designed Taxi.

However, there are key differences in their approach.

​

No global schema

GraphQL defines a single global schema that all consumers adhere to. Consumers can cherry-pick the fields they want, but the structure is fixed.

That global schema can create friction to change - if you need to refactor the schema, you either need to make backwards compatible changes, or consumers need to update.

Which is to say - evolving a shared structural contract is hard, and someone needs to take the hit - either the schema owner (through maintaining a backwards compatible change), or the consumers (by fixing what breaks when the schema changes).

This isn’t just a GraphQL issue - it’s true of any structural contract that is widely shared - they’re hard to change.

​

Consumer driven contracts

GraphQL has consumer-driven-contracts-lite. ie., consumers can cherry-pick the fields they want from the global schema. However, that’s it - structure, encoding, etc is fixed.

If you want to consume data in a different shape, you’ll need to map the data on the consumer side.

Those additional mapping layers are troublesome - each mapping layer you build is a tight coupling between contracts - ie., a thing that has to change when the upstream contract changes.

Instead, with Taxi, consumers define the data contract they want to consume in the request they send.

There is no single central schema - producer APIs are composed together on-the-fly to satisfy the consumer contract.

The types and query language provide enough flexibility that consumers can express the exact contract they require, and the middleware can understand how to assemble a response.

​

No resolvers

GraphQL uses resolvers to stitch together APIs - which is exactly the type of integration code we’re on a mission to eliminate.

As APIs change, resolvers need to be maintained and updated.

Generally, with Taxi you don’t need resolvers - the API specs are rich enough to automate the resolvers.

​

Technology-agnostic

GraphQL requires GraphQL everywhere, or shims to adapt to GraphQL.

Taxi (and taxiQL) is trying hard to keep the footprint low - Taxi will happily compose together a Kafka topic publishing Protobuf, with a gRPC service, some REST APIs, and a database.

​

What about implementations?

Taxi - the spec, compiler, and tooling ecosystem are all open source. Head over to Github and give it some stars.

The TaxiQL query server is currently part of Orbital - which we’re working to open source this summer. In the meantime, you can take it for a spin following our getting started guide.

​

Summary

So, here’s the key takeaways:

Taxi exists to describe how data and data sources relate to one another, and to automate interoperability.

OpenAPI, SQL, Protobuf, etc., all do a great job of describing a single data source. By adding additional metadata into those APIs, we have enough information to automate integration between services, without writing (or maintaining) glue code.

TaxiQL is a way to use Taxi types to ask for data, and for consumers to remain decoupled from producer schemas, so as things change, there’s no glue code to maintain.

XKCD is the one true standard, which is how it was always meant to be.

(The post Why we created Taxi, and why we felt the need for Another Schema Language appeared first on Orbital Blog.)

We prefer to see them as an API that’s customized for a single consumer rather than many consumers like typical APIs. It flips the concept of who an API is built for on its head.

This is the first of two posts with our take on BFFs. First we’ll take a look at the pattern itself, why they emerged and what the trade offs are. In our follow-up we’ll go through how Orbital acts as a BFF and why it’s an elegant solution (if we don’t say so ourselves)

​

How did they become a thing?

Historically, developers build APIs from the perspective of the backend system. They create models based on their own domain. Patterns like REST let them split entities into clean and well-structured endpoints. Which is all completely rational if you’re building one of those systems.

Enter UX designers who (rightly so) have very different ideas and are concerned with how to create an intuitive and pleasant interface. They care not if we’re serving User and Post data from two different systems and laugh in the face of the completely rational schema designs.

Backends-for-frontends is a pattern that emerged to resolve this tension. It allows us to cater for the different data fetching needs of web and mobile applications. However, the functionality and type of data are often similar across both mobile and desktop platforms. So, when developers started creating more complex mobile applications in the 2010s, it made sense to use the same backend systems as their desktop counterparts.

As the designs for desktop and mobile diverged, so did the requirements for the APIs serving them.

As the designs for desktop and mobile diverged, so did the requirements for the APIs serving them. The importance for optimizing for mobile meant fixing issues like over-fetching data on limited bandwidths. At the end of this path, it’s common to have similar data, all from the same domain, but very different API footprints.

Organizational structures accentuated these differences by using different teams to manage the frontend and backend systems. As the web and mobile teams were also split, backend teams became a bottleneck for implementing API requests.

Enter the BFF from stage-left. The savior of developers on both sides of this divide. Bringing gifts of freedom and autonomy for front end developers, and releasing backend developers from an onslaught of trivial requests to add fields. I can only assume that project managers celebrated in their own way too.

​

So what are they?

Assume we’re got a couple of frontend teams building separate desktop and mobile apps. Given the task of fetching data from our organization’s collection of services, let’s take a look at some of the options available to them.

Note: Although it’s more common to see this applied in microservices architectures, it’s equally relevant for monoliths. We’d still expect a monolith to split up their API into distinct endpoints over one or more domains.

​

DIY integration

The original method for this was something of a DIY approach. From each of our frontend applications, we’ll call out to each API and aggregate the data together. It’s basically everyone for themselves.

There’s nothing wrong with this, but we do import a bunch of complexity into our UI code. For each of the APIs we need to do things like:

• manage the connection config
• handle the success, failure or retry states
• import the individual domain models

And that’s assuming they’re all using a consistent protocol. It’s just as likely that one exposes a REST API with an OpenAPI spec, and another uses gRPC, or god forbid, SOAP.

​

General Purpose API

An improvement on this is to create a backend service that takes on some of this integration complexity. Each frontend is kinda doing the same thing if you squint a bit so maybe that makes sense.

What you’d have then would be a bit more like a concierge. A high-end concierge, sure! They’ll be super polite, maybe even go above and beyond to get you what you want on occasion. Instead of having to know all the ins and outs of where to get your data from, you can rely on your concierge (general purpose API) to gather all the relevent data for you and put it in a nice little package. From the perspective of the front end apps, it’s much simpler to integrate with this one API with pre-aggregated data than to do that work themselves.

But they’re still serving any number of people, not just you. When the team building our general purpose backend gets approached by two different frontend teams, they’ve gotta balance both requests.

​

Backend-for-frontend

Backends for frontends are kind of like a personal butler for your frontend. They know how you like things done, are ready and willing to make any change you want, and sometimes even know you better than you know yourself (if Michael Caine is involved).

Backends for frontends are kind of like a personal butler for your frontend.

To meet the data fetching requirements of each frontend, we can create a distinct backend. That allow us to customize the API to serve exactly what we need and as a result improve our ability to solve over fetching or under fetching issues. If we want to trial GraphQL, we don’t have to convince a backend team to expose the same API in two different protocols, or have all our sibling frontends also switch over.

Since we’re not liaising with another team for changes, our whole change management cycle becomes easier too. Deploying a new feature that changes the API interface doesn’t need multi-team coordination. As such, the process to test, deploy and monitor the change to guard against an outage is much more contained. That flexibility is important where your deployments are via an App Store and somewhat out of your control.

​

What are the challenges?

By design a BFF is tightly coupled with our frontend systems. That leads to interesting questions about which team should own it, and what technology it should be built with. From a work division and scheduling perspective it makes sense to sit with the frontend team. They’re also going to be driving it’s implementation so that it works best with the frontend.

However, this leaves us with a awkward technical choice for our server side BFF. We either have to pick technologies to match the skills of frontend engineers, they need learn them, or we need to import them. Luckily there’s good options for web and Android frontend teams to at least use the same programming language. For Node.js there’s great frameworks like Express and Nest and on the JVM they’ve got options like Spring.

I’d argue though that the programming language used is only a component of the problem. There’s generally a wide gap in the bag of tricks required to work on a backend application compared to a frontend. A level of upskilling and extra complexity for the frontend team to deal with is inevitable in this autonomy trade-off.

A level of upskilling and extra complexity for the frontend team to deal with is inevitable in this autonomy trade-off.

We should also consider the raw time involved in building and deploying another service. With each frontend team now building and managing their own integration layer, there’s bound to be some duplication of code and effort as payment for the flexibility that we’re buying with this abstraction.

In terms of deployments, our tools have come a long long way and for many use cases it’s a simple click of the button. Once you leave the guardrails though, there’s often devops pain. As an intermittent, rather than full time devops engineer myself, scripting a cloud deployment and getting the infrastructure right can become a time-consuming task. Creating a separate BFF service buys a little more of that pain. That’s especially true if it’s the frontend team managing it as the infrastructure requirements will differ from those of the frontend code base.

Finally, although managing the coupling with the APIs of your downstream services is easier to manage in a BFF, the problem still exists. It’s just in a different place. When any of those services updates their API, our BFF needs to be updated too. For breaking changes, the mapping code we’ve written needs to be reviewed and modified, and run through the test-and-deploy cycle. That’s a win for isolating the effect of these changes and protecting our frontend code bases, but it’s still work that the frontend team needs to pick up.

​

What's next?

Next up I’ll be writing about how you can use Orbital for your Backend for Frontend without needing to create a new custom backend service. We’ll look at how Orbital helps with automatically adapting to changes in downstream services, and with supporting multiple versions of frontend app being used in the wild.

​

Other BFF material

If this hasn’t quenched your thirst for reading about BFFs, here’s a few other great articles.

• https://samnewman.io/patterns/architectural/bff/
• https://www.thoughtworks.com/insights/blog/bff-soundcloud
• https://philcalcado.com/2019/07/12/some_thoughts_graphql_bff.html

(The post Backends for frontends appeared first on Orbital Blog.)

Possibly it’s as simple as a postcode resolution service, or it could be there’s a few Customer databases floating around. It’d be far too convenient for us for these sources to have stored exactly the same data, using exactly the same representation.

Even so, the content they contain almost certainly will have commonalities.

In the case of Customer data, you definitely wouldn’t be surprised to find a first name, last name and email address being represented in each system.

However, the way these fields are described may vary wildly, and you can’t help it if a developer long ago decided to label their last name property as frequently_inherited_name.

Regardless, the content of that field is equivalent to your last name, surname or family name fields which are present in the other Customer data sets.

Having systems that store and structure data in different ways is a Good Thing - we want our systems to be able to evolve independently, rather than conform to a rigid shared model. After all, you can’t force standardisation, without removing the ability for teams to evolve independently - and that’s not a tradeoff worth making.

What’s important is that there’s a common concept being represented, such as a Customer, an Order, or a Product.

Once our data scene has been set in this way, sooner or later someone in the business will realise that there’s some value to be gained by being able to query that spread out data in a consistent way. That’s the Chameleon pattern in action.

​

Where does this happen?

This is extremely common. Here’s a few examples that you might recognise:

• Your company may have grown up with independent divisions which have evolved separately and built their own versions of things
• Through success and good fortune, you’ve acquired a bunch of related littler companies, that each came with their own unique IT infrastructure.
• You’re using multiple providers for the same type of service (such as payments)
• You’re using multiple vendors for the same type of product (perhaps in your supply chain)
• You’ve started adopting microservices, and found that you’ve got duplicate services rearing their heads all over the place!
• It’s come time to deal with that dilapidated legacy system, and you’d sooner walk on hot ashes than do a big bang cutover

To handle these scenarios, typically we see a couple of different patterns implemented.

​

1. Consumers aggregate data themselves

The consuming system queries all providers separately and manually combines them.

If multiple systems need to consume the same data, each needs to build the same integrations with the source systems, and implement the same aggregation and reshaping code to convert the source data to something that’s useful to it.

This is great for developers who get paid per line of code.

Super App goes it alone and integrates with everyone separately

​

2. Our data providers expose canonical models

To save every consumer from the burden of integrating and reshaping the diverse data formats provided by source systems, you might think of forcing each provider to expose a canonical version of the data.

This helps our Super App developers to an extent, they still have to call out to every system, but at least the data is in the same format. The downside is that the data has to be in the same format! Both of our providers need to agree what a common definition of a customer model is. Cue the politics.

Super App, System 1 and System 2 convene a UN summit to agree a model

​

3. Our data is extracted to a separate store

An ETL pipeline is built to extract data from the providers and store it in a Data Lake or Data Warehouse in a normalized format

Super App hitches it's feature cart to the world's longest train

This solves the problem of each Super App that comes along having to solve the same problem of fetching and aggregating data from multiple sources, but it does bring with it some important restrictions.

The data is now coupled to the canonical model which has been enforced by the ETL pipeline and Data Warehouse. Getting functionality changes through to our Super App is now an IT project involving at least 3 teams (Source system, data pipeline and super app). This clearly makes the lead time of feature requests an immediate challenge.

Depending on your perspective of life as a data engineer, this is either fantastic or very frustrating. As the team building the pipelines and warehouse, you’re involved in all the action, but you’re not responsible for any end user functionality - and therefore the delivery pressure.

​

How Orbital does it

Orbital approaches this problem differently by providing a flexible data layer for consumers to query data. It automatically aggregates and reshapes the source data based on the needs of the consumer, saving you a huge amount of time typically spent on integrating with other systems and transforming data.

Super App asks it's trusty friend Orbital to worry about how to resolve the data it needs

By using our open source data language, Taxi, we define a shared language of terms which can be used to describe the data provided by each individual system. This helps us to focus on the content of what systems are providing, rather than the format and structure of their data.

The result for our Super App is that by querying via Orbital, it’s not limited to the data which can be provided by the data pipeline layer. It also doesn’t have to worry about managing individual integrations with each provider. We get the best of both worlds!

As a bonus, when the providers inevitably change, Orbital adapts with them to ensure minimal impact to our Super App and other consumers. We don’t have to wait for negotiations about our canonical model, or for our data pipeline team to complete work to propagate changes. When our data providers change, we can immediately change with them. Dare I say it, but that sounds a little bit agile.

(The post Uniting similar data from multiple sources appeared first on Orbital Blog.)

The Typescript ecosystem is close to our hearts and we’re looking forward to bringing more features and examples in different web and nodejs frameworks over the next few months.

​

In this post

• Submitting TaxiQL queries
• Building a type-safe query
• Providing parameters
• Type-safe querying
• Why Orbital?
• What next?

​

Getting Started

npm install @orbitalhq/orbital-client

​

Usage

There’s two methods of using the SDK depending on your preferred style.

1. Submit a TaxiQL query string
2. Use the typesafe query building

For both cases, the first step is to generate a taxonomy of terms based on the schemas registered with Orbital. That’s done using a script provided in the Orbital client npm package that connects to the Orbital schema server and downloads the typescript version of the taxonomy. Simply define an orbital.config.json file with the schema server URL and away you go.

{
    "schemaServerUrl": "http://localhost:9305/"
}

The content of the taxonomy depends on the data providers that have uploaded their schemas to Orbital. There’s more info available in our docs on how schemas can be published.

To run the taxonomy generation script from your project:

npx orbital-generate

​

Submitting TaxiQL

This method is closer to what you’d use if you’ve experimented with the Query Editor in the Orbital UI. If you’ve already written a query in the UI you’re happy with, you can generate this code directly from the Query Editor by clicking the the copy button .

const orbitalClient = new HttpQueryClient(`http://localhost:9022`)
let films = orbitalClient.query(```
import film.types.FilmId
import film.types.Title
import com.petflix.listings.StreamingProviderName
import com.petflix.listings.StreamingProviderPrice

find { Film[] } as {
    id: FilmId
    title : Title

    // where can I watch this?
    provider: StreamingProviderName
    cost: StreamingProviderPrice
}[]
```);

Here, we simply send the query string over to Orbital which describes:

• the data we want to find (Films),
• which pieces of data we want (id and title)
• the structure we want it in (flat).

In our projection of the pieces of data we want, we’re asking for two different types of information. Some details about the film itself (the FilmId and the Title), and information about where it’s availabled to watch (StreamingProviderName and StreamingProviderPrice).

If we check out the query history in the Orbital UI after running this query, we can see that the data has come from two different systems. We fetched the id and title from our Films database, and then we used the FilmId to call our StreamingProvidersService to fetch the provider and cost properties.

The query execution plan executed by Orbital

It’s worth noting that when defining the structure of data we want to receive, we can use whatever property names we want. It’s the semantic tags that we’ve imported from our generated taxonomy that are significant. In this example, FilmId and Title are the tags we’ve used, shown as the types of the id and title properties of our response projection.

Check out our other content for why we’re advocating for using semantic metadata to queries and schemas.

You can also extract the query to a separate *.taxi where you can take advantage of our VSCode Taxi language plugin for editing your query.

​

Type safe query builder

The other option is to use the query builder in our SDK which allows us to build our query as a series of chained functions. Just like in our query string, we’ll define the data we want, the properties we’re interested in, and the structure.

import { com, film } from '@/app/taxonomy';
const filmTaxonomy = new film.Taxonomy();
const filmTypesTaxonomy = new film.types.Taxonomy();
const listingsTaxonomy = new com.petflix.listings.Taxonomy();

let films = orbitalClient
    // What data do I want to find
    .find(asArray(filmTaxonomy.Film))

    // Which properties do I want returned, and in what structure
    .as({
        // The property names can be whatever you want
        // The types on the right hand side are our semantic tags that describe
        // to Orbital what data to find
        id: filmTypesTaxonomy.FilmId,
        title: filmTypesTaxonomy.Title,
        provider: listingsTaxonomy.StreamingProviderName,
        cost: listingsTaxonomy.StreamingProviderPrice
    })
    .execute()
    .subscribe()

​

Providing parameters

Sometimes we want to be more specific about the data we’re fetching. Instead of ALL the films, maybe we want more detailed information about one film.

Orbital works with your existing API endpoints and data sources - so provided you’ve got and endpoint that returns a single Film from it’s Id, Orbital will automatically query it.

We’ll add an extra function call given(), using the semantic tag of the item we’re specifying, the film ID.

import { com, film } from '@/app/taxonomy';
const filmTaxonomy = new film.Taxonomy();
const filmTypesTaxonomy = new film.types.Taxonomy();
const listingsTaxonomy = new com.petflix.listings.Taxonomy();

let film = orbitalClient
    // Return the data for the film with a FilmId of "5"
    .given(filmTypesTaxonomy.FilmId, "5")
    .find(asArray(filmTaxonomy.Film))
    .as({
        id: filmTypesTaxonomy.FilmId,
        title: filmTypesTaxonomy.Title,
        provider: listingsTaxonomy.StreamingProviderName,
        cost: listingsTaxonomy.StreamingProviderPrice
      })
    .execute()
    .subscribe()

​

Type-safe querying

A big advantage from using the DSL is type safe querying of our data. For whatever property names and structure we specify in our as() block, the responses from Orbital will be correctly typed.

That’s significant because instead of making a request to a service and providing a type parameter based on what the response should be, we’re asking Orbital to fetch specific data and return it in a specific structure. There’s a direct link between the types we’re asking for from Orbital and how the data will be returned. Additionally, the response type of our query is set of our specific object, not typed as any.

For example, if I add in explicit types to the variable we’re assigning our response to, we’ll see that the Typescript compiler won’t raise any concerns.

let films: Observable<{
    'id': number,
    'title': string,
    'provider': string,
    'cost': number
}> = orbitalClient
    .find(asArray(taxonomy.film.Film))
    .as({
        id: taxonomy.film.types.FilmId,
        title: taxonomy.film.types.Title,
        provider: taxonomy.com.petflix.listings.StreamingProviderName,
        cost: taxonomy.com.petflix.listings.StreamingProviderPrice
    })
    .execute()

​

Why Orbital?

We think there’s some great ergonomics here for Typescript developers when fetching data. There’s also a bunch of other benefits you get from using Orbital to power your Typescript apps:

• Adapt to schema changes - your code will automatically adapt to changes in schemas
• Eliminate integration code - you’ll eliminate loads of integration code and code-gen of other teams schemas
• Consumer defined schemas - regardless of the tech used by other teams you’ll gain the power to pick and choose which data is delivered to you.
• Observability - see exactly how an integration was performed in the Orbital console, including response payloads and cell level lineage

Check out our docs for more information.

​

What next?

In terms of where we’re heading, there’s a number of exciting features we’ll be adding in the near future. They’re already available in the core platform, and we’ll be adding support for them in the Typescript SDK.

• Streaming query results - for larger result sets, start receiving responses as a stream as soon as the first results become available
• Subscriptions - connect to kafka topics and other streaming sources of data
• Publishing schemas from Nodejs apps - similar to the capability in our Kotlin SDK, this let’s publishers add a couple of decorators to their code to push their schemas up to Orbital
• Type safety for projections to nested data structures - query projections for the DSL are currently limited to flat objects. While the Orbital query runner is capable of returning results in any shape requested in the query string, the DSL option in the SDK isn’t able to provide type safety for nested structures.

(The post Hello, Typescript SDK appeared first on Orbital Blog.)

Orbital works by reading the specs of microservices, (eg., OpenApi, Raml, Protobuf or Taxi) and building a large mesh of APIs.

All this schema metadata is browsable in Orbital’s UI, and can be an awesome hub for developers to discover and document their APIs - even without using Orbital’s integration capabilities.

In this blog post, we’re going to explore creating an awesome self-updating developer hub using Spring Boot, Kotlin, and Orbital’s API catalog.

A GIF showing the working catalog, diagrams and API explorer

Our microservices will publish their APIs directly to Orbital on startup, so that Orbital is always up-to-date. There’s lots of different ways to publish API specs to Orbital - but in this post we’ll focus on a code-first workflow.

​

What we'll get

At the end of this post, we’ll end up with a local instance of Orbital, with a couple of Spring Boot microservices that self-publish their API’s.

We’ll get beautiful diagrams like this, which show the running services, as well as how the data between them is related:

Orbital's service diagram - showing two microservices, their APIs, and how they relate

In addition, Orbital gives us a rich data and API catalog, where we can search and play with APIs:

Service explorer lets developers see the endpoints and operations exposed by an API

The operation explorer lets us interact and play around with individual APIs

​

The Spring Boot app

We’ll start out with a vanilla Spring Boot app:

package com.petflix.voyager.listings

import org.springframework.beans.factory.annotation.Autowired
import org.springframework.beans.factory.annotation.Value
import org.springframework.boot.SpringApplication
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.info.GitProperties
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.stereotype.Component

@SpringBootApplication
open class FilmListingsApp {

  companion object {
      val logger = LoggerFactory.getLogger(this::class.java)

      @JvmStatic
      fun main(args: Array<String>) {
          SpringApplication.run(FilmListingsApp::class.java, *args)
      }
  }
}

This consists of:

• A maven pom.xml file, declaring our vanilla Spring Boot dependencies, and a JPA database
• FilmListingsApp.kt - The spring boot app class file
• FilmListings.kt - A Spring Data repository and controller for listing films
• Reviews.kt - A Spring Data repository and controller for listing reviews of our films

Since these are all pretty standard, we won’t go into them in more detail here

​

Starting Orbital

To get Orbital running, grab the docker compose from start.orbitalhq.com and start Orbital running

curl https://start.orbitalhq.com > docker-compose.yml
docker-compose up -d

If you head over to http://localhost:9022 you should see Orbital running, and waiting for connections

​

Publishing our schemas

Next up, let’s get our Spring Boot applications publishing their schemas.

When the apps start, they’ll create a schema (using Taxi), and publish them up.

Note - you don’t have to use Taxi here - if you already have an OpenAPI spec you can use that, by following the [docs](/docs/describing - data - sources / open - api).

First, we’ll add some dependencies to our Maven:

  <properties>
     <taxi.version>1.36.2</taxi.version>
     <orbital.version>0.22.0</taxi.version>
  </properties>
 <dependencies>
    <!-- others omitted -->

      <!-- A publisher, which will push our service schema
      to Orbital on startup -->
      <dependency>
         <groupId>com.orbitalhq</groupId>
         <artifactId>schema-rsocket-publisher</artifactId>
         <version>${orbital.version}</version>
      </dependency>

      <!-- The taxi codegen, which generates schemas for a spring boot app -->
      <dependency>
         <groupId>org.taxilang</groupId>
         <artifactId>java-spring-taxi</artifactId>
         <version>${taxi.version}</version>
      </dependency>
   </dependencies>

   <!-- Also, add the Orbital maven repository -->
  <repositories>
      <repository>
          <id>orbital</id>
          <url>https://repo.orbitalhq.com/release</url>
      </repository>
  <repositories>

Now, add a Kotlin class with the following code:

import com.orbitalhq.PackageMetadata
import com.orbitalhq.schema.publisher.SchemaPublisherService
import com.orbitalhq.schema.publisher.rsocket.RSocketSchemaPublisherTransport
import com.orbitalhq.schema.rsocket.TcpAddress
import lang.taxi.generators.java.spring.SpringTaxiGenerator
/**
 * Create a task, which runs on startup, generating & publishing our schema
 */
@Component
class RegisterSchemaTask(
   @Value("\$\{spring.application.name}") appName: String,
   @Value("\$\{server.port}") private val serverPort: String
) {
   init {
      /**
       * Declare a publisher, which will connect to the local running Orbital,
       * and publish the generated schema
       */
      val publisher = SchemaPublisherService(
         appName,
         RSocketSchemaPublisherTransport(
            // Orbital accepts schema submissions over
            // RSocket using port 7655
            TcpAddress("localhost", 7655)
         )
      )
      publisher.publish(
         // Package metadata identifies this project.
         // Kinda like maven poms, or npm package.json
         // This can be whatever you want, but make it unique to this service
         PackageMetadata.from(
            organisation = "io.petflix.demos",
            name = "films-listings"
         ),
         // The generator creates Taxi schemas.
         // The base url should be the url of this application.
         // All published URLs are relative to this url
         SpringTaxiGenerator.forBaseUrl("http://localhost:${serverPort}")
            .forPackage(FilmListingsApp::class.java)
            .generate()
      ).subscribe()
   }
}

That’s it! Now, restart your application.

​

Exploring our catalog

Inside Orbital, Click on the Catalog tab, and then the Services Diagram tab.

You should now see a diagram, showing our two API’s.

Clicking on any of the endpoints will also add the response schema to the diagram.

Our services diagram

Our catalog also already has rich information about our services - including an API browser, and data catalog.

The data and API catalog

​

Going further - Exposing links between our APIs

So far, we have a pretty rich catalog, from just a few lines of code.

However, there’s important relationships here that we’re not exposing.

The Id from the Films can be passed to my Review service to lookup a review.This is key information for developers who are trying to navigate between services to Get Stuff Done.

There's a missing relationship between the Film model and the ReviewService

However, we’re in luck.

The schema that was generated is actually built using Taxi - an open source schema language that focuses on describing the semantic relationship of data between services.

Taxi has amazing support for Kotlin, and lets us use Kotlin’s typealias to express that the data is related.

First, let’s define a type alias:

import lang.taxi.annotations.DataType

// The DataType annotation from 
@DataType
typealias FilmId = Int

Then update our code to use this.

There’s a few places we can change:

// Update the response object our API returns
@Entity
data class Film(
 @Id
  val filmId: Int,
  val filmId: FilmId,
 val title: String
)

// Update the input into our Review service
fun getReviewsForFilm(
 @PathVariable("filmId") filmId: Int
 @PathVariable("filmId") filmId: FilmId
): Review {
return repo.findAllByFilmId(filmId)
}

Now, if restart the Spring Boot app, our updated schema will get published directly to Orbital.

Refreshing the services diagram now shows an updated diagram with the relationship mapped:

Using type aliases, the relationship between services can be mapped

​

Changelog

We just pushed a change to our schema.While this one was fairly harmless(changing an input type fromInt totypealias FilmId = Int), it’s useful to know when changes occur.

Orbital has a full changelog, every time a change is pushed.

Navigate over to the Schemas tab, and see the changelog entry

Orbital's changelog shows changes to APIs as they're published

​

Summary

That’s a lot of catalog, for only a few lines of change.

To recap - the only change we needed to make to our Spring Boot application was adding our schema publisher - about a dozen lines of simple code.

Of course, we can go much further— Orbital’s real power is using the data in these schemas to automate integration between services.Check out our [Kotlin SDK](/blog/2023 - 03 - 06 - hello - kotlin - sdk) for more information!

As always, if you have any questions, come chat to us on Slack, or reach out on Twitter(either me, or our team)

(The post Building an API Hub with Spring Boot, Kotlin and Orbital appeared first on Orbital Blog.)