spring-projects / spring-ai

An Application Framework for AI Engineering

Spring AI

Welcome to the Spring AI project!

The Spring AI project provides a Spring-friendly API and abstractions for developing AI applications.

Let's make your @Beans intelligent!

For further information go to our Spring AI reference documentation.

Project Links

• Documentation
• Issues
• Discussions - Go here if you have a question, suggestion, or feedback!
• Upgrade from 0.7.1-SNAPSHOT

Educational Resources

• Follow the Workshop material for Azure OpenAI
  • The workshop contains step-by-step examples from 'hello world' to 'retrieval augmented generation'

Some selected videos. Search YouTube! for more.

• Spring Tips: Spring AI
  

• Overview of Spring AI @ Devoxx 2023
  
• Introducing Spring AI - Add Generative AI to your Spring Applications
  

Getting Started

Please refer to the Getting Started Guide for instruction on adding your dependencies.

Note, the new Spring CLI project lets you get up and running in two simple steps, described in detail here.

1. Install Spring CLI
2. Type spring boot new --from ai --name myai in your terminal

Adding Dependencies manually

Note that are two main steps.

1. Add the Spring Milestone and Snapshot repositories to your build system.
2. Add the Spring AI BOM
3. Add dependencies for the specific AI model, Vector Database or other component dependencies you require.

Overview

Despite the extensive history of AI, Java's role in this domain has been relatively minor. This is mainly due to the historical reliance on efficient algorithms developed in languages such as C/C++, with Python serving as a bridge to access these libraries. The majority of ML/AI tools were built around the Python ecosystem. However, recent progress in Generative AI, spurred by innovations like OpenAI's ChatGPT, has popularized the interaction with pre-trained models via HTTP. This eliminates much of the dependency on C/C++/Python libraries and opens the door to the use of programming languages such as Java.

The Python libraries LangChain and LlamaIndex have become popular to implement Generative AI solutions and can be implemented in other programming languages. These Python libraries share foundational themes with Spring projects, such as:

• Portable Service Abstractions
• Modularity
• Extensibility
• Reduction of boilerplate code
• Integration with diverse data sources
• Prebuilt solutions for common use cases

Taking inspiration from these libraries, the Spring AI project aims to provide a similar experience for Spring developers in the AI domain.

Note, that the Spring AI API is not a direct port of either LangChain or LlamaIndex. You will see significant differences in the API if you are familiar with those two projects, though concepts and ideas are fairly portable.

Feature Overview

This is a high level feature overview. The features that are implemented lay the foundation, with subsequent more complex features building upon them.

You can find more details in the Reference Documentation

Interacting with AI Models

ChatClient: A foundational feature of Spring AI is a portable client API for interacting with generative AI models. With this portable API, you can initially target one AI chat model, for example OpenAI and then easily swap out the implementation to another AI chat model, for example Amazon Bedrock's Anthropic Model. When necessary, you can also drop down to use non-portable model options.

Spring AI supports many AI models. For an overview see here. Specific models currently supported are

• OpenAI
• Azure OpenAI
• Amazon Bedrock (Anthropic, Llama, Cohere, Titan, Jurassic2)
• HuggingFace
• Google VertexAI (PaLM2, Gemini)
• Mistral AI
• Stability AI
• Ollama
• PostgresML
• Transformers (ONNX)
• Anthropic Claude3

Prompts: Central to AI model interaction is the Prompt, which provides specific instructions for the AI to act upon. Crafting an effective Prompt is both an art and science, giving rise to the discipline of "Prompt Engineering". These prompts often leverage a templating engine for easy data substitution within predefined text using placeholders.

Explore more on Prompts in our concept guide. To learn about the Prompt class, refer to the Prompt API guide.

Prompt Templates: Prompt Templates support the creation of prompts, particularly when a Template Engine is employed.

Delve into PromptTemplates in our concept guide. For a hands-on guide to PromptTemplate, see the PromptTemplate API guide.

Output Parsers: AI model outputs often come as raw java.lang.String values. Output Parsers restructure these raw strings into more programmer-friendly formats, such as CSV or JSON.

Get insights on Output Parsers in our concept guide.. For implementation details, visit the OutputParser API guide.

Incorporating your data

Incorporating proprietary data into Generative AI without retraining the model has been a breakthrough. Retraining models, especially those with billions of parameters, is challenging due to the specialized hardware required. The 'In-context' learning technique provides a simpler method to infuse your pre-trained model with data, whether from text files, HTML, or database results. The right techniques are critical for developing successful solutions.

Retrieval Augmented Generation

Retrieval Augmented Generation, or RAG for short, is a pattern that enables you to bring your data to pre-trained models. RAG excels in the 'query over your docs' use-case.

Learn more about Retrieval Augmented Generation.

Bringing your data to the model follows an Extract, Transform, and Load (ETL) pattern. The subsequent classes and interfaces support RAG's data preparation.

Documents:

The Document class encapsulates your data, including text and metadata, for the AI model. While a Document can represent extensive content, such as an entire file, the RAG approach segments content into smaller pieces for inclusion in the prompt. The ETL process uses the interfaces DocumentReader, DocumentTransformer, and DocumentWriter, ending with data storage in a Vector Database. This database later discerns the pieces of data that are pertinent to a user's query.

Document Readers:

Document Readers produce a List<Document> from diverse sources like PDFs, Markdown files, and Word documents. Given that many sources are unstructured, Document Readers often segment based on content semantics, avoiding splits within tables or code sections. After the initial creation of the List<Document>, the data flows through transformers for further refinement.

Document Transformers:

Transformers further modify the List<Document> by eliminating superfluous data, like PDF margins, or appending metadata (e.g., primary keywords or summaries). Another critical transformation is subdividing documents to fit within the AI model's token constraints. Each model has a context-window indicating its input and output data limits. Typically, one token equates to about 0.75 words. For instance, in model names like gpt-4-32k, "32K" signifies the token count.

Document Writers:

The final ETL step within RAG involves committing the data segments to a Vector Database. Though the DocumentWriter interface isn't exclusively for Vector Database writing, it the main type of implementation.

Vector Stores: Vector Databases are instrumental in incorporating your data with AI models. They ascertain which document sections the AI should use for generating responses. Examples of Vector Databases include Chroma, Postgres, Pinecone, Qdrant, Weaviate, Mongo Atlas, and Redis. Spring AI's VectorStore abstraction permits effortless transitions between database implementations.

Cloning the repo

This repository contains large model files. To clone it you have to either:

• Ignore the large files (won't affect the spring-ai behaviour) : GIT_LFS_SKIP_SMUDGE=1 git clone [email protected]:spring-projects/spring-ai.git.
• Or install the Git Large File Storage before cloning the repo.

Building

To build with running unit tests

./mvnw clean package

To build including integration tests. Set API key environment variables for OpenAI and Azure OpenAI before running.

./mvnw clean verify -Pintegration-tests

To run a specific integration test allowing for up to two attempts to succeed. This is useful when a hosted service is not reliable or times out.

./mvnw -pl vector-stores/spring-ai-pgvector-store -Pintegration-tests -Dfailsafe.rerunFailingTestsCount=2 -Dit.test=PgVectorStoreIT verify

To build the docs

./mvnw -pl spring-ai-docs antora

The docs are then in the directory spring-ai-docs/target/antora/site/index.html

To reformat using the java-format plugin

./mvnw spring-javaformat:apply

To update the year on license headers using the license-maven-plugin

./mvnw license:update-file-header -Plicense

To check javadocs using the javadoc:javadoc

./mvnw javadoc:javadoc -Pjavadoc

spring-ai's Issues

PromptTemplate can't be reused with different model data

This was tested with Spring AI 0.7.0-SNAPSHOT.

Once a PromptTemplate has been created without model data and then used once via the render() method that takes model data, it cannot be reused with different model data. Attempting to do so will result in a prompt that contains the original model data. For example, I would expect the following test to pass. But it does not as described in the comments.

    @Test
    public void testPromptTemplate() {
        PromptTemplate pt = new PromptTemplate("Here is the input: {input}");

        String output1 = pt.render(Map.of("input", "Hello, world!"));
        Assertions.assertThat(output1).isEqualTo("Here is the input: Hello, world!");

        String output2 = pt.render(Map.of("input", "Goodbye, world!"));
        // This next assertion fails!!!
        // The rendered output is the "Hello world" from the first time!!!
        Assertions.assertThat(output2).isEqualTo("Here is the input: Goodbye, world!");
    }

This is appears to be because of the if statement at https://github.com/spring-projects-experimental/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/prompt/PromptTemplate.java#L141 that prevents new model data from being given to the ST instance unless that key is null.

Note that I've tested the same thing with both the Python and Node.js implementations of PromptTemplate from Langchain and they both work as I'd expect. That is, the PromptTemplate can be reused with new model data. For the sake of clarity, here's what I did in Python:

from langchain.prompts import PromptTemplate

prompt_template = PromptTemplate.from_template(
    "Here is the input: {input}"
)

output1 = prompt_template.format(input="Hello, world!")
print(output1)
# Here is the input: Hello, world!

output2 = prompt_template.format(input="Goodbye, world!")
print(output2)
# Here is the input: Goodbye, world!

And here is what I did in Node.js:

import { PromptTemplate } from "langchain/prompts";

const pt = PromptTemplate.fromTemplate('Here is the input: {input}');

const output1 = await pt.format({input: "Hello, world!"});
console.log(output1);
// Here is the input: Hello, world!

const output2 = await pt.format({input: "Goodbye, world!"});
console.log(output2);
// Here is the input: Goodbye, world!

In both cases, the result was (as expected):

Here is the input: Hello, world!
Here is the input: Goodbye, world!

I'd expect Spring AI's PromptTemplate to be reusable in the same way.

Retrieve embedding model dimensions

Expected Behavior

The EmbeddingClient interface should expose a method to report the dimension size of the configured embedding model.

Current Behavior

Currently the users must retrieve this information manually from the documentations of the used models.

Context

The dimensions are need to configure VectorStores and at the moment has to be specified explicitly.
Instead they could obtain this information directly from the provided EmbeddingClient instance.

Investigate Documents Used for VectorDB testing

I just noticed when writing the README for the Neo4j store that the sentence in the PgVector store is a little bit vague.
Yes, it's true that the document will be there but it is either to expect that it is the first in the collection or we should set the "top-k amount" to 1 in the example snippet, or?

https://github.com/spring-projects-experimental/spring-ai/blob/0a584f0dc2483ef9ad0741c8598998e489b64726/vector-stores/spring-ai-pgvector-store/README.md?plain=1#L141-L145

A little bit tweaked in the Neo4j store README:
https://github.com/spring-projects-experimental/spring-ai/blob/0a584f0dc2483ef9ad0741c8598998e489b64726/vector-stores/spring-ai-neo4j-store/README.md?plain=1#L114-L118

AI client library

Apologies for creating an issue to only ask a question, but I don't see the discussions enabled for this repository.

Why don't you use Azure client library instead of theokanning client? Did you consider it?

It provides sync and async clients. And it's very easy to switch from Azure API to Open AI API

Create a github pages web site with the reference docs for 0.7.1-SNAPSHOT docs

The project will move to 0.8.0-SNAPSHOT and have breaking changes. For people who want to continue to use 0.7.1-SNAPSHOT until that 0.8.0 release, these docs will still be available, but in another location since our use of antora can't yet support having two snapshot versions listed simultaneously.

Add Auto-Configurations and Boot starters for all Vector Store implementations

Most Spring-ai vector store implementations already have a proper boot auto-configurations in place.
Create the corresponding Spring Boot Starters under the spring-ai-spring-boot-starters and update the VectorStore's README.md with both starter usage instructions.

• Milvus Vector Store, using the milvus auto-configure
• Chroma Vector Store, using the chroma auto-configure
• PgVector Vector Store, using the pgvector auto-configure
• Pinecone Vector Store, using the pinecone auto-configure
• Azure Vector Store, using the azure vector store auto-configure
• Weaviate Vector Store - using the weaviate vector store auto-configure
• Redis Vector Store - using the redis vector store auto-configure
• Neo4j Vector Store - using the neo4j auto-config

Remove chain package

The chain idea from langchain involves 'workflow' and often type unsafe access to input and output. Instead of following down that path, first extract the key helper classes that are needed for use cases and later revisit how they can be composed into a chain/flow. There is some example usage of 'chaining' based on langchain's example using java.util.Function in #108

Helper classes envisioned are GenerationTemplate (similar to LLMChain) and ChatEngine (similar to conversationalchain)

Avoid duplicated entries in VectorStore(s) by allowing generation of Document ID based on the hashed document content.

Currently the Document if not provided with an explicit ID, generates a random UUID for every document.
Even if the document content/metadata haven't changed a new ID is generated every time.
This will lead to document content duplications in the Vector store.

To prevent this type of unnecessary duplications we can allow generation of Document ID based on the hashed document content+metadata.

Following snippet is inspired by a langchain4j vector store implementations.

import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

....

public static String generateIdFrom(String contentWithMetadata) {
    try {
	    byte[] hashBytes = MessageDigest.getInstance("SHA-256").digest(contentWithMetadata.getBytes(StandardCharsets.UTF_8));
	    StringBuilder sb = new StringBuilder();
	    for (byte b : hashBytes) {
		    sb.append(String.format("%02x", b));
	    }
	    return UUID.nameUUIDFromBytes(sb.toString().getBytes(StandardCharsets.UTF_8)).toString();
    }
    catch (NoSuchAlgorithmException e) {
	    throw new IllegalArgumentException(e);
    }
}

milvus向量库

为什么不用milvus作为向量数据库呢？

Integrate PGVector as a VectorStore

Allow using Postgres+PGVector as VectorStore

PGVector is Open-source vector similarity search for Postgres.
It comes with pgvector-java support.

Handle functions as an agent

Expected Behavior

I want to be able to provide Spring AI with one or more functions that could be used to provide additional information or functionality. (See https://platform.openai.com/docs/guides/gpt/function-calling). Moreover, when the response to a prompt indicates that a function should be called, I'd like it if that decision, the call to the function, and the reissuing of a new prompt were all handled internally by Spring AI. This behavior is what I believe is commonly referred to as an "agent".

In short, I'd like to be able to configure Spring AI with one or more functions (perhaps as lambdas or method references) that could be used later. Then, if I ask a question that the LLM needs more information to answer, Spring AI would call those functions for me and resubmit the prompt. I don't want to have to do what is shown as "step 2" in the example at https://platform.openai.com/docs/guides/gpt/function-calling. That decision making should be handled internally by Spring AI.

My primary desire is to have function support, but almost as importantly, I want to not have to deal with the back and forth interactions in my own code. I want Spring AI to handle that internally.

Current Behavior

Spring AI does not yet support functions, so at this point there's no need for agent behavior. Currently Spring AI only handles one-turn Q&A type interactions.

Context

As an example (drawn from the aforementioned documentation), if I were to ask what the weather is in Boston, most/all LLMs wouldn't know that answer because they're not trained on real-time or even relatively current data. But if I provide a function that can lookup the weather in a city, then instead of the LLM responding that it doesn't know what the weather is, it could respond with an instruction to call the function. The application (or the agent on behalf of the application) would call that function as instructed, then submit a new prompt with weather data from which the LLM would be able to generate the desired answer.

Create GenerativeTextTemplate that streamlines usage of AiClient, Prompts, and OutputParsers

There is an analogy to make between GenerationTemplate and JdbcTemplate that will help simplify the use of the lower level classes such as AiClient, Prompts, and OutputParsers. Note: AiClient should likely be renamed GenerationClient as it is focused on generation, specifcally text, so perhaps even TextGenerationClient as we haven't yet experimented with multi-modal apis.

DataSource <-> AiClient

String Query with placeholdres <-> Prompt with placeholders

RowMapper <-> OutputParser

The simplest method (which is a default method in AiClient) is string input and string output, that could be removed from the AiClient interface and used in GenerateTemplate instead. Here are a few sample signatures for consideration.

public interface GenerateOperations {

	String generate(String message);

	String generate(String message, Map<String, Object> model)

	String generate(PromptTemplate promptTemplate, Map<String, Object> model);
	
	<T> T generate(String message, Class<T> elementType, Map<String, Object> model);

	<T> T generate(String message, OutputParser<T> parser, Map<String, Object> model);

	<T> List<T> generateList(String message, Class<T> elementType, Map<String, Object> model);

	<T> List<T> generateList(String message, OutputParser<T> parser, Map<String, Object> model);

        AiResponse generateResponse(PromptTemplate promptTemplate, Map<String, Object> model);
}

Example usage

ActorsFilms actorsFilms = generateTemplate.generate("Generate the filmography for the actor {actor}",
 		                                    ActorsFilms.class, Map.of("actor", "Tom Hanks"));

Simple "chains" for flows can be done with standard Java functional programming, and we can see how our needs for a "chain" or a "flow" evolve over time. e.g. the example of a chain from langchain using functional programming

	@Test
	void functionalChains() {
		Function<String, String> combinedFunction = generateSynopsis.andThen(generateReview);
		System.out.println(combinedFunction.apply("Tragedy at sunset on the beach"));
	}

	private Function<String, String> generateSynopsis = title -> {
		String synopsisInput = """
				You are a playwright. Given the title of play, it is your job to write a synopsis for that title.

				Title: {title}
				Playwright: This is a synopsis for the above play:""";

		return generateTemplate.generate(synopsisInput, Map.of("title", title));
	};

	private Function<String, String> generateReview = synopsis -> {
		String synopsisInput = """
				You are a play critic from the New York Times. Given the synopsis of play, it is your job to write a review for that play.

				Play Synopsis:
				{synopsis}
				Review from a New York Times play critic of the above play:""";

		return generateTemplate.generate(synopsisInput, Map.of("synopsis", synopsis));
	};

Excuse me, can you support the proxy mode of openAI-java later?

Please do a quick search on Github issues first, the feature you are about to request might have already been requested.

Expected Behavior

I now want to use vpn to access gpt. However, it seems that we do not have this entrance yet. We hope that we can support a method similar to openAI-java sdk in the future.

Current Behavior

Add documentation for Ollama client

Add Weaviate VectorStore support

Please add support Weaviate as a vector store for existing vector dbs list.

Review Message interface

There is a method String getMessageTypeValue(); which is not needed. This was used in the implementation of the Microsoft ACME fitness store and is no longer needed.

Consider making the interface generic, interface Message<T> { T getContent(); ... } A quick spike using Whisper or DallE etc would show if this class is usable in the context of other types of content.

Also review if there is a need for Map<String, Object> getProperties(); It doesn't seem necessary and is likely just a hold over from the initial work to create the org.springframework.ai.prompt.messages when modifying the langhchain design.

Also review the naming of MessageType - perhaps Role is more succinct.

Add Boot autoconfigure for the existing vector-store implementations.

It would bring a lot of convenience to have Boot autoconfigurations for the various Vector Store implementations.
I would make it easier to bootstrap with just property configurations .

Add a DocumentReader implementation for GH Markdown

Github Markdown is a popular format that provides ready structural hints as to document content, e.g paragraph, table, etc and many document formats can be converted to markdown, for example pandoc can convert asciidoc to markdown which makes it easier to work with.

I've used https://github.com/commonmark/commonmark-java and the GH extension with success in spring-cli.

Add invocation retry capabilities for AiClient, EmbeddingClient and VectorStore

The AiClient, EmbeddingClient or VectorStore clients interaction with their remote service endpoints could suffer from transient errors such as a momentary network glitch or rate limitation errors.
Often, those communication issues are resolvable by repetitive service invocation or altering the invocation rate.

We should provide a retry-decorators that automatically re-invoke the failed operations according to pre-configured retry policies.

Consider moving `AiResponse` and `Generation` to a separate package from `AiClient`

The AiResponse and Generation types are fairly common across different AI providers, AFAICT. Although, where we refer to what the AI returns as a Generation, others refer to this as a Completion, or based on use, a Choice.

Regardless, like Prompt, I think that AiResponse and Generation warrant their own top-level package, such as org.springframework.ai.generation, or org.springframework.ai.response.

Food for thought.

Change `AzureOpenAiEmbeddingOptions` and `AzureOpenAiChatOptions` use of `setModel` to `setDeployentName` as that matches Azure OpenAI terminology.

Bug description
spring-ai-azure-openai-spring-boot-starter reports issue about DeploymentNotFound ( it requires Azure model deployment name)
with properties included
spring.ai.azure.openai.api-key= spring.ai.azure.openai.endpoint=https://abc.openai.azure.com/ spring.ai.azure.openai.embedding-model=text-embedding-ada-002

com.azure.core.exception.ResourceNotFoundException: Status code 404, "{"error":{"code":"DeploymentNotFound", "message":"The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again."}}" at com.azure.core.implementation.http.rest.RestProxyBase.instantiateUnexpectedException(RestProxyBase.java:347) ~[azure-core-1.41.0.jar:1.41.0] at com.azure.core.implementation.http.rest.SyncRestProxy.ensureExpectedStatus(SyncRestProxy.java:130) ~[azure-core-1.41.0.jar:1.41.0] at com.azure.core.implementation.http.rest.SyncRestProxy.handleRestReturnType(SyncRestProxy.java:213) ~[azure-core-1.41.0.jar:1.41.0] at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:81) ~[azure-core-1.41.0.jar:1.41.0] at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:109) ~[azure-core-1.41.0.jar:1.41.0] at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:91) ~[azure-core-1.41.0.jar:1.41.0] at jdk.proxy2/jdk.proxy2.$Proxy75.getChatCompletionsSync(Unknown Source) ~[na:na] at com.azure.ai.openai.implementation.OpenAIClientImpl.getChatCompletionsWithResponse(OpenAIClientImpl.java:897) ~[azure-ai-openai-1.0.0-beta.3.jar:1.0.0-beta.3] at com.azure.ai.openai.OpenAIClient.getChatCompletionsWithResponse(OpenAIClient.java:294) ~[azure-ai-openai-1.0.0-beta.3.jar:1.0.0-beta.3] at com.azure.ai.openai.OpenAIClient.getChatCompletions(OpenAIClient.java:430) ~[azure-ai-openai-1.0.0-beta.3.jar:1.0.0-beta.3] at org.springframework.ai.azure.openai.client.AzureOpenAiClient.generate(AzureOpenAiClient.java:60) ~[spring-ai-azure-openai-0.2.0-20230908.193706-31.jar:0.2.0-SNAPSHOT] at org.springframework.ai.openai.samples.helloworld.simple.SimpleAiController.completion(SimpleAiController.java:21) ~[classes/:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-6.0.11.jar:6.0.11] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-6.0.11.jar:6.0.11] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.0.11.jar:6.0.11] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884) ~[spring-webmvc-6.0.11.jar:6.0.11] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-6.0.11.jar:6.0.11] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-6.0.11.jar:6.0.11] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081) ~[spring-webmvc-6.0.11.jar:6.0.11] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974) ~[spring-webmvc-6.0.11.jar:6.0.11] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011) ~[spring-webmvc-6.0.11.jar:6.0.11] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.0.11.jar:6.0.11] at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564) ~[tomcat-embed-core-10.1.11.jar:6.0] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.0.11.jar:6.0.11] at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) ~[tomcat-embed-core-10.1.11.jar:6.0] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.11.jar:10.1.11] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.0.11.jar:6.0.11] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.11.jar:6.0.11] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.0.11.jar:6.0.11] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.11.jar:6.0.11] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109) ~[spring-web-6.0.11.jar:6.0.11] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.11.jar:6.0.11] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.0.11.jar:6.0.11] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.11.jar:6.0.11] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1740) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-10.1.11.jar:10.1.11] at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

Environment
spring-ai-azure-openai-spring-boot-starter : 0.2.0-SNAPSHOT
spring boot: 3.1.2

Steps to reproduce
Just run the application and access the endpoint /ai/sample

Expected behavior
It has to connect to Azure OpenAI and get the response.

Minimal Complete Reproducible example
https://github.com/rd-1-2022/ai-azure-openai-helloworld.git

Embedding clients in different maven modules share the same package name, this needs to change

For example, the module spring-ai-postgresml-embedding-client has the class PostgresMEmbeddingClient in the package org.springframework.ai.embedding and transformers-embedding module has TransformersEmbeddingClient in the same package org.springframework.ai.embedding

When making this change, create a more specific name for the transformers-embedding perhaps onnx-embedding-client?

Also, there is a convention of having the artifact contain the prefix spring-ai. The Postgres embedding clients doesn't follow this convention.

Add docs and examples for spring-ai-test

The spring-ai-test is meant to help evaluating generative responses using the the LLM itself.
But is it stands at the moment it is not obvious how to make use of it.

Add Azure Cognitive Search as a Vector Store

Azure Cognitive Search provides value add capabilities to users of Azure. In addition to supporting easy document upload and
indexing, indexes that are configured with Vector storage could be integrated and consumed via a VectorStore implementation in Spring AI.

Expected Behavior

Azure Cognitive Search supports a Java search API for uploading, deleting, and searching documents using vector based queries and storage. This API could be wrapped in a Spring AI Vector Store implementation giving Spring AI and Azure users the ability to integrate their existing Cognitive Search indexes with Spring AI.

Conceptually, a user would create a SearchClient instance using existing methods and inject the instance along with an EmbeddingClient into a VectorStore implementation. Using bean creation methods and a PropertiesSource class, the VectorStore creation could look like the following

@Bean
public SearchClient searchClient(AzureSearchClientProperties props) {

	return new SearchClientBuilder().endpoint(props.getEndpoint())
		.credential(new AzureKeyCredential(props.getApiKey()))
		.indexName(props.getIndex())
		.buildClient();
}

@Bean
public VectorStore vectorStore(SearchClient searchClient, EmbeddingClient embeddingClient) {
	return new AzureCognitiveSearchVectorStore(searchClient, embeddingClient);
}

Current Behavior

Spring AI currently does not support Azure Cognitive Search as a VectorStore implementation.

Context

An Azure Spring sample project currently exists that
implements a Cognitive Search based VectorStore. In addition, the Microsoft AI learning pages also demonstrate sample code
that utilizes existing documents already uploaded and indexed into CognitiveSearch. It would be desirable for Spring AI to integrate with Azure Cognitive Search as another VectorStore implementation.

Add Chroma Vector Store integration

Chroma is a popular open-source embedding database. It gives you the tools to store document embeddings, content and metadata and to search through those embeddings including metadata filtering.

Add Neo4j as a Vector Store

Current Version of the Spring AI allows InMemoryVectorStore but Neo4jVectorStore is expected along with it.

Expected Behavior

Neo4j should be able to use as VectorStore

Current Behavior

InMemoryVectorStore is supported.

Context

I am working on content writer suggestions use-case with azure openai embedding stored in Neo4j. So i request the team to implement Neo4j as Vector Store

Add support for @JsonPropertyDescription

It would be helpful to add additional descriptions to the JSON schema for better prompt results. The jsonschema-generator used by the BeanOutputParser could also add the jsonschema-generator/jsonschema-module-jackson
module to enable @JsonPropertyDescription annotations on fields.

Tidy up project directory structure

The generative AI client implementations are modules sitting at the top of the project, e.g. spring-ai-ollama. Create a directory, generative-clients and put the various client modules in that directory

Create a ChatEngine helper class

This would be akin to llamindex chatengine as well as what was done in canopy from pinecone.

This is the alternative to following the design in langchain which adds on workflow/chain elements that at the moment is too premature (in my mind) to delve into and can be added later. Functional programming chain in Java can also be useful in this context, will wait and see how it evolved.

DocumentReader for all kinds of documents

Current Version of the Spring AI allows JsonReader and TextReader.

Expected Behavior

More DocumentReader implementations should allow reading all kinds documents (PDF, word, html, ...)

Current Behavior

DocumentReader is implemented only for Json and Text content types.

Context

I am working on document querying use-case with azure openai to load all the documents and create embeddings to store them in Redis. So i request the team to implement DocumentLoader.

Improve the Document API

• Currently the Document mixes multiple concerns that should be isolated in dedicated classes.
• Encapsulate the content formatting into a separate ContentFormatter abstraction.
• Refactoring the document loading and indexing strategy.
• Moves toward ETL processing model.
• Add metadata enriching transformers as DocumentTranformer.

Remove use of Supplier/Consumer in org.springframework.ai.document and replace with Function

The use of Supplier/Consumer was found to be limiting in a few scenarios. In the case of loading documents from a Resource, the use of a Supplier made the class stateful, meaning a new class instance was needed per Resource. Instead using a Functional interface so that the Resource is the input makes it easier to process many files.

So move from

interface DocumentReader extends Supplier<List<Document>>

to

interface DocumentReader<I> extends Function<I, List<Document>>

and have an implementation that takes a Spring Resource as input:

public class ResourceNewDocumentReader implements NewDocumentReader<Resource> {
    @Override
    public List<Document> apply(Resource resource) {
        ...
    }
}

The case of consumer is a bit less clear, perhaps the current DocumentWriter can still exist, but we could have our current implementations that write to the vector store implement DocumentTransformer so that if we wanted to chain together writin to multiple vector stores, the output from one could be changed to another. Maybe DocumentProcessor would be better then instead of DocumentTransformer

Using Void in a function instead of consumer is also an option, though it looks a little odd now that I write it out.

public interface DocumentWriter extends Consumer<List<Document>>

to

public interface DocumentCreator<O> extends Function<List<Document>, O> {

}

and

public class DocumentWriter implements DocumentCreator<Void>{
    @Override
    public Void apply(List<Document> documents) {
        return null;
    }
}

We'd love to add Neo4j as a vector store to the project

Neo4j added vector support in August 2023 based on HNSW, we'd like to add support for that to spring-ai

Just wanted to open an issue first before sending in a PR, to see if there are any thoughts.

Improve Ollama IT by caching the image with model

Hi,

I have an improvement for OllamaAutoConfigurationIT which can help when working locally. The current issue as described in the test is the model is around 3GB which can take quite a while depending on internet speed.

My approach using the Testcontainers lifecycle:
First test execution

1. Pull image from registry
2. Start container
3. Once the container has started, pull the model
4. Test execution
5. When test is finishing, commit the changes in the container and create a local image.
6. Stop container, stop test

Second test execution

1. Start container using the local image that already contains the model
2. Test execution
3. Stop container, stop test

If there is interest on this, I can provide the PR.

Support AWS Bedrock

Please do a quick search on Github issues first, the feature you are about to request might have already been requested.

Expected Behavior
It would be nice to add support for AWS Bedrock.

Current Behavior
No support yet
Context
In general AWS Bedrock will probably be relevant since a lot of people are on AWS and will probably not migrate everything over.

I could pitch in time to make it happen.
As a first pitch I would probably start with inference

Create boot starter for Ollama client

Paragraph Pdf Document Reader issue

Bug description
ParagraphPdfDocumentReader causing NullPointerException when reading sample1.pdf
https://github.com/spring-projects-experimental/spring-ai/blob/main/document-readers/pdf-reader/src/test/resources/sample1.pdf

Environment
Spring Boot version: 3.1.4
Spring AI version: 0.7.0-SNAPSHOT
Java version: openjdk version "17.0.2" 2022-01-18

Steps to reproduce
Add dependency spring-ai-pdf-document-reader: 0.7.0-SNAPSHOT version to pom.xml

`

    <dependency>
     <groupId>org.springframework.experimental.ai</groupId>
     <artifactId>spring-ai-pdf-document-reader</artifactId>
     <version>0.7.0-SNAPSHOT</version>
   </dependency>

`

Code to read paragraphs:
`

    var documents = pdfReader.get();

   ParagraphPdfDocumentReader pdfReader = new ParagraphPdfDocumentReader(
            "file:\\C:\\Users\\test\\sample1.pdf",
            PdfDocumentReaderConfig.builder()
                    .build());

    var documents = pdfReader.get();

    for (Document document : documents) {
        System.out.println(document.getContent());
    }

`

Expected behavior
It should read each paragraph from the sample1.pdf file

Exception
`

  java.lang.NullPointerException: Cannot invoke "org.apache.pdfbox.pdmodel.interactive.documentnavigation.outline.PDOutlineNode.getFirstChild()" because "bookmark" is null
at org.springframework.ai.reader.pdf.config.ParagraphManager.generateParagraphs(ParagraphManager.java:131) ~[spring-ai-pdf-document-reader-0.7.0-20231019.142632-5.jar:0.7.0-SNAPSHOT]
at org.springframework.ai.reader.pdf.config.ParagraphManager.<init>(ParagraphManager.java:82) ~[spring-ai-pdf-document-reader-0.7.0-20231019.142632-5.jar:0.7.0-SNAPSHOT]
at org.springframework.ai.reader.pdf.ParagraphPdfDocumentReader.<init>(ParagraphPdfDocumentReader.java:109) ~[spring-ai-pdf-document-reader-0.7.0-20231019.142632-5.jar:0.7.0-SNAPSHOT]
at org.springframework.ai.reader.pdf.ParagraphPdfDocumentReader.<init>(ParagraphPdfDocumentReader.java:92) ~[spring-ai-pdf-document-reader-0.7.0-20231019.142632-5.jar:0.7.0-SNAPSHOT]

`

Stream support in `AiClient`

Context

Applications using AI often use a UI that generates text little by little in a stream instead of outputting texts all at once. The current spring-ai AiClient does not support stream output.

Expected Behavior

I hope that generateStream method will be added to AiClient. As for the return value, Flux<AiResponse> is promising, but if you want to avoid dependence on reactor, you can also consider the JDK standard Flow.Publisher<AiResponse>.

For example, openai-java used in OpenAiClient uses Flowable. This can be converted to Flux or Flow.Publisher.

https://github.com/TheoKanning/openai-java/blob/650d76b87a3246a52247b231f8d47f994ae63020/service/src/main/java/com/theokanning/openai/service/OpenAiService.java#L150

This is a little off topic, but openai-java uses outdated RxJava2 and has an unnecessary number of dependencies (e.g. kotlin sdk). Since spring-ai does not use all OpenAI APIs, I think it is better to have an in-house OpenAI client that implements only the minimum API with WebClient that supports the streaming.

Add MongoDB as a Vector Store

See https://www.mongodb.com/products/platform/atlas-vector-search.

Simplify the VectorStore#similaritySearch signature

Currently the VectorStore interface offers at least 5 variations of the similaritySearch() method, each with different set of arguments.

We should collapse all methods into a single similaritySearch(SearchRequest req) using a convenient SearchRequest builder like:

vectorStore.similaritySearch(
    SearchRequest.query("Hello World")
	.withTopK(5)
	.withSimilarityThreshold(0.6)
	.withFilterExpression("country == 'UK"));

Vector Store similarity search with metadata filters.

Most of the vector store implementations provide, usually proprietary, capabilities for embedding search with metadata filters.

• Pinecone offers filtering with metadata using custom (Mongo inspired) query language. The PineconeVectorStore implementation maps the document metadata to the native Pinecone json metadata format.
• PostgresSQL's JSONB column allows JsonPath filter expressions (e.g. @@). The PgVectorSore implementation uses jsonb column to store the document's metadata and can leverage this functionality.
• Neo4J ?
• Milvus ?

Ref:
[1] https://www.pinecone.io/learn/vector-search-filtering/

多个starter同时引用，出现问题，没有设置实例化优先级

Please do a quick search on Github issues first, there might be already a duplicate issue for the one you are about to create.
If the bug is trivial, just go ahead and create the issue. Otherwise, please take a few moments and fill in the following sections:

Bug description

Environment
Please provide as many details as possible: Spring AI version, Java version, which vector store you use if any, etc

Steps to reproduce
Steps to reproduce the issue.

Expected behavior
A clear and concise description of what you expected to happen.

Minimal Complete Reproducible example
Please provide a failing test or a minimal complete verifiable example that reproduces the issue.
Bug reports that are reproducible will take priority in resolution over reports that are not reproducible.

Update main readme and relevant getting started sections regarding 0.7.1-snapshot and 0.8.0-snapshot

Similarity search on PgStore might be wrong wrt threshold

As far as I understood the JavaDoc of VectorStore threshold is a lower bound.

The Postgres store uses the threshold as an upper limit to the distance, though.

In addition, the cosine distance must be inverted, see https://github.com/pgvector/pgvector#distances
I suggest fixing the comparison operator and also change the sql statement when using cosine.

Adapt interfaces that are part of the document page to be functional interfaces

Provide prompt options on a call by call basis

Our local OpenAI endpoint has a very small default max_tokens against the chat completion.
We like to make it adjustable

Expected Behavior

Hoping this code.

https://github.com/spring-projects-experimental/spring-ai/blob/main/spring-ai-openai/src/main/java/org/springframework/ai/openai/client/OpenAiClient.java

Looks something like this

	private ChatCompletionRequest getChatCompletionRequest(String text) {
		List<ChatMessage> chatMessages = List.of(new ChatMessage("user", text));
		logger.trace("ChatMessages: ", chatMessages);
		ChatCompletionRequest chatCompletionRequest = ChatCompletionRequest.builder()
			.model(this.model)
			.temperature(this.temperature)
			.messages(List.of(new ChatMessage("user", text)))
                         .maxTokens(4000) /// ADD HERE
 			.build();
		logger.trace("ChatCompletionRequest: ", chatCompletionRequest);
		return chatCompletionRequest;
	}

Current Behavior

max tokens not adjustable

Context

Support PostgresML as an `EmbeddingClient`

It would be nice to support PostgresML embeddings (https://postgresml.org/docs/guides/transformers/embeddings) as an EmbeddingClient that doesn't require OpenAI/Azure OpenAI account

Parent Document Retriever feature.

Expected Behavior

Lanchain's Parent Document Retriever feature is expected in Spring AI

Current Behavior
Files are chunked and cretead embeddings for them. We would like to have child chunks for parent chunks of the document.

Add Redis as a Vector Store

Current Version of the Spring AI allows InMemoryVectorStore but RedisVectorStore is expected along with.

Expected Behavior

Redis should be able to use as VectorStore

Current Behavior

InMemoryVectorStore is supported.

Context

I am working on document querying use-case with azure openai embedding stored in Redis. So i request the team to implement Redis as Vector Store

Review package org.springframework.ai.client

The current implementation is focused on text/code based GenerativeAI use cases as input and output. The name AiClient, is likely too broad. There isn't yet a ton of experience with multi-modal language models and such a generic name might be better suited in that use case.

Consider to rename AiClient. Here are some things to consider.

• There is friction with Java naming conventions and standard use of the abbreviation AI.
• Consider renaming AiClient to GenerativeClient and perhaps the package name to org.sf.ai.client.generative
• Consider making Generation an interface with a generic type.