OmniHai gets motion

OmniHai 1.7 has been released! In 1.1 OmniHai grew ears and in 1.6 it got hands. In 1.7 it gets motion, in both directions: hand it a recording and ask what happens in it, or describe a scene and get one back. Relatively a lot of smaller things have been added around that, such as classification against your own labels, audio generation through OpenRouter, and a modality check which looks up what a model can do instead of guessing it from the name.

New: analyzeVideo()

How do you ask an AI when the goal happens in a match recording? Until now you had to cut the video into frames yourself and send those as images, which loses the timing and makes you pay per frame. OmniHai 1.7 sends the video itself:

String answer = service.analyzeVideo(Path.of("match.mp4"), "When does the goal happen?");

The video may also be passed as a byte[], and there is an async variant analyzeVideoAsync(), as with every other operation.

An AI does not watch a video the way you do. It samples frames from it, by default one per second, and reads those. On a long recording that is a lot of frames for a question about one minute of it. You can therefore state which part to sample and how densely, with AnalyzeVideoOptions:

String answer = service.analyzeVideo(Path.of("match.mp4"), "When does the goal happen?",
    AnalyzeVideoOptions.newBuilder()
        .fps(2)
        .startOffset(Duration.ofMinutes(1))
        .endOffset(Duration.ofMinutes(2))
        .build());

Video input is currently only accepted by Gemini (Google AI), by Muse Spark (Meta AI; note that Meta AI is now finally available for public whereas it was previously preview only) and by the video capable models which OpenRouter routes. The sampling options of AnalyzeVideoOptions are honored by Gemini only.

The other providers do not accept a video at all: OpenAI, Azure, Anthropic, xAI, Mistral and Ollama. This is not OmniHai holding something back. Their APIs simply do not support video analysis, so supportsModality() continues to answer false for VIDEO_ANALYSIS on those providers, so you can pick a service which can do it before you call.

Gemini does not take a video in the request itself. It must first be uploaded, and the AI provider then needs a moment to process it before it can be referred to, which on a larger file is longer than one round trip. OmniHai waits for that state to become active and only then asks the question, so you do not have to poll it yourself. A video which the provider rejects fails with an AIException stating why. OpenRouter takes the video as a plain data URI only and has nowhere to put them, so be careful with large videos.

New: generateVideo()

Making a video does not fit in one call. The AI provider answers within seconds with a job id and then takes minutes to actually produce the video, and it never calls you back. A method which just returns the video would sit and wait for those minutes, which you cannot do in a request. So generateVideo() hands you a handle on the job as soon as it is accepted:

VideoGeneration video = service.generateVideo("Sunrise over the colorful houses of Willemstad");
String jobId = video.jobId();

The handle is serializable and carries that job id, so you can submit in one request and look at it again in a later one, after a restart or on another node. Keep the id, and pick the job up again with findVideoGeneration():

VideoGeneration video = service.findVideoGeneration(jobId);

if (video.refresh().status() == VideoGeneration.Status.COMPLETED) {
    video.writeTo(Path.of("curacao.mp4"));
}

status() is a plain getter which does no I/O at all, so it costs you nothing to call it from a render pass. refresh() is the one which asks the AI provider, exactly one request per call, on your own schedule.

When you do want to wait, generateVideoAsync() does the polling for you and completes when the video is there:

CompletableFuture<VideoGeneration> pending = service.generateVideoAsync("Sunrise over the colorful houses of Willemstad");

pending.thenAccept(video -> video.writeTo(Path.of("curacao.mp4")));

Asking for a path writes the video there for you, asynchronously or not:

CompletableFuture<Void> written = service.generateVideoAsync("Sunrise over the colorful houses of Willemstad", Path.of("curacao.mp4"));

service.generateVideo("Sunrise over the colorful houses of Willemstad", Path.of("curacao.mp4"));

That last one blocks for as long as it takes, so it is for a batch job rather than for a request. The polling stops as soon as you stop watching, so a handle nobody looks at costs nothing. A job which never finishes fails after five minutes rather than keeping you there, which you can raise with GenerateVideoOptions, together with the aspect ratio, the resolution and the duration.

Video generation is offered by Google with Veo, by xAI with Grok Imagine, and by OpenRouter, which routes the generators of several labs. OpenAI and Azure both retire Sora without a successor to route to, so supportsModality() continues to answer false for VIDEO_GENERATION there, as it does on Anthropic, Mistral, Meta, Hugging Face and Ollama.

The generated video is hosted by the AI provider for about a day and is then deleted, upon which the status becomes EXPIRED. Some of them host it on a separate host and hand out a pre-signed URL for it; OmniHai downloads such a URL without your API key, which has no business leaving the endpoint it belongs to.

New: classify() and classifyAll()

Which queue does this support ticket belong in? You could ask that in a chat prompt, but then you have to parse whatever came back, and "This looks like a billing issue." is not a queue name. AIService now has a method for it:

ClassificationResult result = service.classify(ticket, "billing", "shipping", "technical");
route(result.label());

The labels you offer are the only values the AI may answer with. This uses structured outputs under the covers: a schema, which is a description of the exact shape the answer must have, is sent along with the question, and the AI provider holds the model to it. So label() is never anything else than one of your labels, and you can switch on it without a default branch for surprises. You do not write that schema yourself here; classify() derives it from the labels you pass in.

The AI must pick one even when nothing fits well, which is what confidence() is for. It runs from 0.0 to 1.0 and tells a certainty apart from a guess:

if (result.confidence() < 0.7) {
    queueForReview(ticket);
}

A ticket can of course be about two things at once. classifyAll() scores every label on its own merit and returns them with the best fitting one first:

List<ClassificationResult> results = service.classifyAll(ticket, "billing", "shipping", "technical");
List<String> tags = results.stream().filter(r -> r.confidence() > 0.5).map(ClassificationResult::label).toList();

The scores are not divided among the labels here, so several of them may score high, or none of them. A text which belongs to none of your labels scores low on all of them, which is the answer you want and which classify() cannot give you.

New: audio generation through OpenRouter

generateAudio() has been in OmniHai since 1.2, on OpenAI and Google. OpenRouter now joins them, by a different route. It does have an audio/speech endpoint, but that one serves a model catalog of its own which the model listing does not enumerate, so there is no telling which model to ask. Audio comes out of an audio capable chat model instead, which streams it back in chunks. OmniHai collects those and assembles a playable WAV file from it, so the call is the same one you already know:

service.generateAudio("Hello world", Path.of("hello.wav"));

The audio is written straight to the given path while it streams in, so a long text does not have to fit in memory first. Hugging Face still cannot do this; the text-to-speech models there are served by third parties with their own API rather than via Hugging Face's own API.

Modality lookup instead of name matching

A modality is a kind of input or output: text, image, audio, video. supportsModality() tells you whether a service can handle one, so you can check up front rather than by catching:

if (service.supportsModality(AIModality.VIDEO_ANALYSIS)) {
    return service.analyzeVideo(video, "Summarize this");
}

Most providers publish one model family per endpoint, so the answer follows from the predefined model names and versions. Aggregators do not; OpenRouter and Hugging Face route hundreds of models from many labs, and a name says nothing about them. Of the OpenRouter models which accept video, not one carries "video" in its name. Both of them do publish the input and output modalities per model, which OmniHai now looks up instead of guessing or even hardcoding.

That listing is fetched at most once a day per endpoint and shared by every service instance on it, so the first call blocks on one HTTP request and the rest are answered from memory. When it cannot be obtained, matching the model name is the fallback, and the last known listing is kept rather than cleared.

It stays a hint, not a guarantee. A published listing goes stale, a name matched guess is a guess, and a provider may still refuse a call it advertises, for this key, this region or this moment. The only guarantee is to make the call and handle its failure. The actual calls do namely not precheck the supportsModality().

Smaller things

analyzeImage() and generateAltText() now also accept a Path, as the video and audio methods already did.

ModerationOptions now also accepts a List of categories instead of only String... varargs.

Installation

Non-Maven users: download OmniHai 1.7 JAR and drop it in /WEB-INF/lib the usual way, replacing the older version if any.

Maven users: use the following coordinates.

<dependency>
    <groupId>org.omnifaces</groupId>
    <artifactId>omnihai</artifactId>
    <version>1.7</version>
</dependency>