Skip to content

Integration

Groq Integration

Low latency inference over open models, integrated with typed tool calling, grounded retrieval, schema validated output and provider fallback.

Typical use cases
  • Customer facing assistants where response latency decides adoption
  • High volume classification and routing of inbound messages
  • Structured extraction from documents with schema validation
  • Retrieval grounded answering over policies and documentation
  • Draft generation inside an existing tool, with a person editing

Overview

What this integration involves

Groq serves open weight models with an emphasis on inference speed. For interactive features that difference is the product: a response that starts immediately reads as a conversation, and the same response after several seconds reads as a form submission.

The integration surface includes an OpenAI compatible chat completions API. A provider adapter can reduce migration work, but supported parameters, model behavior, tools and output formats still need compatibility tests before switching.

Speed does not remove the engineering. Grounding, validation, guardrails and evaluation are what make a model feature reliable, and none of them are provided by the endpoint.

Surface

Capabilities we use

  • Chat completions

    Standard message based completion with system, user and assistant roles, temperature and token controls, and streaming for immediate first token display.

  • Streaming responses

    Server sent events deliver tokens as they are produced, so the interface shows progress rather than a spinner. This is most of the perceived speed benefit.

  • Tool calling

    Functions described with a JSON schema. The model returns a structured call, your code validates the arguments and executes, and the result returns to the conversation.

  • Structured output

    Responses constrained to a schema for extraction and classification, validated again server side because a constraint is not a guarantee.

  • Open model choice

    Several open weight model families at different sizes, so a classification task can use a small fast model while a reasoning task uses a larger one.

Data flow

Step by step

  1. 01

    Resolve the request scope

    The authenticated user and their permissions are established before anything is retrieved, because retrieval must be filtered by what that user is allowed to see.

  2. 02

    Retrieve grounding material

    A hybrid search over your indexed content returns candidate passages, filtered by permission and metadata, with source references attached.

  3. 03

    Assemble the prompt

    System instructions, the retrieved passages, the conversation history within a token budget, and the tool definitions. Older turns are summarised rather than dropped silently.

  4. 04

    Call the model with a timeout

    The request streams, with a hard timeout and a cancellation path so an abandoned browser tab does not leave work running.

  5. 05

    Handle tool calls

    A returned tool call has its arguments validated against the schema and checked against the user permissions before execution. The result is appended and the model continues.

  6. 06

    Validate and record

    Structured output is checked against its schema and business rules. Record request identifiers, permitted diagnostic fields and token usage. Store transcripts only where required, with redaction, access controls and a defined retention policy.

Architecture

How the pieces connect

LLM feature data flow
  1. 01UsertoAssistant service

    message

  2. 02Assistant servicetoRetrieval

    query

  3. 03RetrievaltoIndex

    Application data exchange

  4. 04Assistant servicetoGroq

    prompt. Asynchronous delivery.

  5. 05GroqtoAssistant service

    stream and tool calls. Asynchronous delivery.

  6. 06Assistant servicetoTyped tools

    execute

  7. 07Assistant servicetoTranscript log

    record

Events

Webhooks we handle

  1. Completion events recorded to the transcript store for audit and evaluation
  2. Budget threshold alerts when a feature approaches its spend cap
  3. Failure alerts on sustained provider errors or timeouts
  4. Evaluation runs triggered by a prompt or model change in the pipeline

Reliability

What makes it hold

  • Timeouts and cancellation

    Every call carries a hard timeout. A streamed response is cancelled when the client disconnects, so an abandoned tab does not continue consuming tokens.

  • Provider fallback

    A tested provider adapter can route eligible requests to an approved fallback after sustained failures. Model behavior, tool support, data processing permissions and response formats must be verified first. If no approved fallback is available, return a clear unavailable state.

  • Retry with care

    Read only completions retry on transient errors. A completion that has already triggered a tool call with a side effect does not, because the side effect is not idempotent unless the tool made it so.

  • Token budgets per request

    Context is assembled to a budget, with older history summarised. Without a budget, a long conversation silently becomes the most expensive request in the system.

  • Response caching

    Deterministic work such as classification of identical input and summaries of unchanged documents is cached with an invalidation rule tied to the source record.

  • Evaluation gates changes

    A fixed set of real cases with expected outcomes runs before a prompt or model change ships, so a regression is caught by a number rather than by a user.

Security

How it is secured

  1. API keys live in the platform secret store and are used only in server code.
  2. Retrieval is filtered by the requesting user permissions before any passage enters the prompt.
  3. Personal data is excluded from prompts where the feature does not require it, and redacted where it does.
  4. Transcripts are stored in your database with a retention policy you set.
  5. Prompt injection is mitigated by treating retrieved content as data, keeping instructions outside it, and validating every tool argument rather than trusting a model decision.
  6. Where material cannot leave the environment, the same architecture runs against a private or on device model.

Evidence

Where we have built this

FAQ

Questions people actually ask

Why choose a fast provider over a larger model?

Because for interactive features the wait is part of the quality. A good answer that arrives immediately is used. A slightly better answer after eight seconds often is not. For non interactive work the calculation changes and we say so.

Are we locked in?

A provider adapter and model independent application code make changes easier. Migration still requires checking model availability, licences, hardware, API compatibility and evaluation results. It is not automatically a configuration only change.

How do you stop prompt injection?

Retrieved content is treated as data, never as instructions. Tool arguments are validated against a schema and checked against the user permissions before execution, so a model that has been persuaded to attempt something still cannot do it.

What happens during a provider outage?

The circuit breaker opens and requests route to a fallback provider or a smaller model. Where no fallback is acceptable, the feature returns a deterministic response and queues the work.

How is cost controlled?

Token budgets per request, caching of repeated deterministic work, model selection per task, and per feature spend caps with alerting.

Can this run without sending data to a provider?

Yes. The same architecture runs against a private or on device model, with the index kept inside your network. The tradeoffs in latency and quality are stated before you decide.

Let’s talk about the system you need.

Tell us where the current process breaks down. We can map the workflow, examine the integration requirements and define a practical scope for the build.

Discuss Your Projectsupport@sawhorsesoftware.tech
  • A mutual NDA is signed before project details are shared.
  • Scope is fixed in writing before any build starts.
  • You own the source code, the IP and the documentation at delivery.
  • Support plans are available and tailored on a discovery or technical call.