Markdown support + UI for non-knowledge-base-answers (#82)

* markdown file handler, small refactorings, make the file delete immediately submit a request to the reconciler to delete the file

* "delete an empty data source with an empty global storage"

* "maybe ask to delete from docstore"

* fix deleting from doc store

* "markdown rendering"

* wip on markdown tables

* "now we're thinking with styled-components"

* "css styling of the markdown tables"

* update lock file

* use markdown for docling pdf, minor ui changes to source card

* readme updates

* wip on icon for no source nodes

* "now we're thinking with colors"

* "styling/coloration of the not-a-kb-response response"

* "now we're thinking with inference models"

* Set inference model when chatting with model directly

* it ain't round

* fill a little more of the "circle"

* it's still not a circle, but it's a little bigger

* mock the model api for the test

---------

Co-authored-by: Elijah Williams <[email protected]>
Co-authored-by: Michael Liu <[email protected]>

Author: 3 people

.DS_Store

@@ -6,15 +6,30 @@ An AMP that provides a no-code tool to build RAG applications
 
 ### Pre-requisites
 
-RAG Studio requires AWS for access to both LLM and embedding models. Please complete the following steps before using the RAG Studio:
-
-- A S3 bucket to store the documents
-- The following models configured and accessible via AWS Bedrock. Any of the models not enabled will not function in the UI.
-  - Llama3.1 8b Instruct v1 (`meta.llama3-1-8b-instruct-v1:0`) - This model is required for the RAG Studio to function
-  - Llama3.1 70b Instruct v1 (`meta.llama3-1-70b-instruct-v1:0`)
-  - Cohere Command R+ v1 (`cohere.command-r-plus-v1:0`)
-- For Embedding, you will need to enable the following model in AWS Bedrock:
-  - Cohere English Embedding v3 (`meta.cohere-english-embedding-v3:0`)
+RAG Studio can be used with both Cloudera Inference (CAII) or AWS Bedrock for selecting LLM and embedding models. 
+
+#### Cloudera Inference (CAII) Setup:
+
+To use CAII, you must provide the following environment variables:
+
+- `CAII_DOMAIN` - The domain of the CAII instance
+
+#### AWS Bedrock Setup:
+
+To use AWS Bedrock, you must provide the following environment variables:
+
+- `AWS_DEFAULT_REGION` - defaults to `us-west-2`
+- `AWS_ACCESS_KEY_ID`
+- `AWS_SECRET_ACCESS_KEY`
+
+#### Document Storage:
+
+RAG Studio can utilize the local file system or an S3 bucket for storing documents. If you are using an S3 bucket, you will need to provide the following environment variables:
+
+- `S3_RAG_BUCKET_PREFIX` - A prefix added to all S3 paths used by Rag Studio
+- `S3_RAG_DOCUMENT_BUCKET` - The S3 bucket where uploaded documents are stored
+
+S3 will also require providing the AWS credentials for the bucket.
 
 ### Cloudera DataFlow (Nifi) Setup:
 

README.md

@@ -99,7 +99,7 @@ public ReconcilerConfig reconcilerConfig(@Qualifier("testEnvironment") boolean t
   public HttpClient httpClient(OpenTelemetry openTelemetry) {
     return JavaHttpClientTelemetry.builder(openTelemetry)
         .build()
-        .newHttpClient(HttpClient.newHttpClient());
+        .newHttpClient(HttpClient.newBuilder().followRedirects(HttpClient.Redirect.NORMAL).build());
   }
 
   @Bean

backend/src/main/java/com/cloudera/cai/rag/configuration/AppConfiguration.java

@@ -61,6 +61,7 @@ public class RagFileService {
   private final RagFileIndexReconciler ragFileIndexReconciler;
   private final String s3PathPrefix;
   private final RagDataSourceRepository ragDataSourceRepository;
+  private final RagFileDeleteReconciler ragFileDeleteReconciler;
 
   @Autowired
   public RagFileService(
@@ -69,13 +70,15 @@ public RagFileService(
       RagFileUploader ragFileUploader,
       RagFileIndexReconciler ragFileIndexReconciler,
       @Qualifier("s3BucketPrefix") String s3PathPrefix,
-      RagDataSourceRepository ragDataSourceRepository) {
+      RagDataSourceRepository ragDataSourceRepository,
+      RagFileDeleteReconciler ragFileDeleteReconciler) {
     this.idGenerator = idGenerator;
     this.ragFileRepository = ragFileRepository;
     this.ragFileUploader = ragFileUploader;
     this.ragFileIndexReconciler = ragFileIndexReconciler;
     this.s3PathPrefix = s3PathPrefix;
     this.ragDataSourceRepository = ragDataSourceRepository;
+    this.ragFileDeleteReconciler = ragFileDeleteReconciler;
   }
 
   public RagDocumentMetadata saveRagFile(MultipartFile file, Long dataSourceId, String actorCrn) {
@@ -144,6 +147,7 @@ public void deleteRagFile(Long id, Long dataSourceId) {
       throw new NotFound("Document with id " + id + " not found for dataSourceId: " + dataSourceId);
     }
     ragFileRepository.deleteById(id);
+    ragFileDeleteReconciler.submit(document);
   }
 
   // Nullables stuff down here
@@ -155,7 +159,8 @@ public static RagFileService createNull(String... dummyIds) {
         RagFileUploader.createNull(),
         RagFileIndexReconciler.createNull(),
         "prefix",
-        RagDataSourceRepository.createNull());
+        RagDataSourceRepository.createNull(),
+        RagFileDeleteReconciler.createNull());
   }
 
   public List<RagDocument> getRagDocuments(Long dataSourceId) {

backend/src/main/java/com/cloudera/cai/rag/files/RagFileService.java

@@ -215,7 +215,8 @@ private RagFileService createRagFileService(
         tracker == null ? RagFileUploader.createNull() : RagFileUploader.createNull(tracker),
         RagFileIndexReconciler.createNull(),
         prefix,
-        dataSourceRepository);
+        dataSourceRepository,
+        RagFileDeleteReconciler.createNull());
   }
 
   private long newDataSourceId() {