Ingestion

Ingesting files with R2R.

Document Ingestion and Management

Ingest Files

Ingest files or directories into your R2R system:

const files = [
  { path: 'path/to/file1.txt', name: 'file1.txt' },
  { path: 'path/to/file2.txt', name: 'file2.txt' }
];
const metadatas = [{ key1: 'value1' }, { key2: 'value2' }];

const ingestResponse = await client.ingestFiles(files, {
  metadatas,
  user_ids: ['user-id-1', 'user-id-2'],
});

Response

response · object

The response from the R2R system after ingesting the files.

[{'message': 'Ingestion task queued successfully.', 'task_id': '6e27dfca-606d-422d-b73f-2d9e138661b4', 'document_id': 'c3291abf-8a4e-5d9d-80fd-232ef6fd8526'}, ...]

files · Array<string | File | { path: string; name: string }> · required

An array of file paths, File objects, or objects with path and name properties to ingest.

options · object

metadatas · Record<string, any>

An optional array of metadata objects corresponding to each file.

document_ids · Array<string>

An optional array of document IDs to assign to the ingested files.

user_ids · Array<string | null>

An optional array of user IDs associated with the ingested files.

ingestion_config · Optional[Union[dict, ChunkingConfig]]

The ingestion config override parameter enables developers to customize their R2R chunking strategy at runtime.

Show properties

provider · str · default: "unstructured_local"

Which chunking provider to use. Options are “r2r”, “unstructured_local”, or “unstructured_api”.

method · str · default: "by_title"

Which chunking method to apply. Options are “by_title”, “basic”, “recursive”, or “character”.

chunk_size · int · default: "512"

The average size of chunks, in tokens.

chunk_overlap · int · default: "20"

The default overlap between chunks.

max_chunk_size · Optional[int] · default: "None"

Sets a maximum size on output chunks.

combine_under_n_chars · Optional[int]

Combine chunks smaller than this number of characters.

max_characters · Optional[int]

Maximum number of characters per chunk.

coordinates · bool · default: "False"

Whether to include coordinates in the output.

encoding · Optional[str]

Encoding to use for text files.

extract_image_block_types · Optional[list[str]]

Types of image blocks to extract.

gz_uncompressed_content_type · Optional[str]

Content type for uncompressed gzip files.

hi_res_model_name · Optional[str]

Name of the high-resolution model to use.

include_orig_elements · Optional[bool] · default: "False"

Whether to include original elements in the output.

include_page_breaks · bool

Whether to include page breaks in the output.

languages · Optional[list[str]]

List of languages to consider for text processing.

multipage_sections · bool · default: "True"

Whether to allow sections to span multiple pages.

new_after_n_chars · Optional[int]

Start a new chunk after this many characters.

ocr_languages · Optional[list[str]]

Languages to use for OCR.

output_format · str · default: "application/json"

Format of the output.

overlap · int · default: "0"

Number of characters to overlap between chunks.

overlap_all · bool · default: "False"

Whether to overlap all chunks.

pdf_infer_table_structure · bool · default: "True"

Whether to infer table structure in PDFs.

similarity_threshold · Optional[float]

Threshold for considering chunks similar.

skip_infer_table_types · Optional[list[str]]

Types of tables to skip inferring.

split_pdf_concurrency_level · int · default: "5"

Concurrency level for splitting PDFs.

split_pdf_page · bool · default: "True"

Whether to split PDFs by page.

starting_page_number · Optional[int]

Page number to start processing from.

strategy · str · default: "auto"

Strategy for processing. Options are “auto”, “fast”, or “hi_res”.

chunking_strategy · Optional[str] · default: "by_title"

Strategy for chunking. Options are “by_title” or “basic”.

unique_element_ids · bool · default: "False"

Whether to generate unique IDs for elements.

xml_keep_tags · bool · default: "False"

Whether to keep XML tags in the output.

Update Files

Update existing documents:

const files = [
  { path: '/path/to/updated_file1.txt', name: 'updated_file1.txt' }
];
const document_ids = ['document-id-1'];
const updateResponse = await client.updateFiles(files, {
  document_ids,
  metadatas: [{ key: 'updated_value' }] // to overwrite the existing metadata
});

Response

response · object

The response from the R2R system after updating the files.

{'results': {'processed_documents': [{'id': '9f375ce9-efe9-5b57-8bf2-a63dee5f3621', 'title': 'aristotle_v2.txt'}], 'failed_documents': [], 'skipped_documents': []}}

files · Array<File | { path: string; name: string }> · required

An array of File objects or objects with path and name properties to update.

options · object · required

document_ids · Array<string> · required

An array of document IDs corresponding to the files being updated.

metadatas · Array<Record<string, any>>

An optional array of metadata objects for the updated files.

ingestion_config · Record<string, any>

The ingestion config override parameter enables developers to customize their R2R chunking strategy at runtime.

Show properties

provider · str · default: "r2r"

Which chunking provider to use, r2r or unstructured. Selecting unstructured is generally recommended when parsing with unstructured or unstructured_api.

method · str · default: "recursive"

Which chunking method to apply? When using unstructured, by_title or basic are supported.

chunk_size · int · default: "512"

The average size of chunks, in tokens.

chunk_overlap · int · default: "20"

The default overlap between chunks.

max_chunk_size · Optional[int] · default: "None"

Sets a maximum size on output chunks.

Documents Overview

Retrieve high-level document information, restricted to user files, except when called by a superuser where it will then return results from over all users:

const documentsOverview = await client.documentsOverview();

Response

response · Array<object>

An array of objects containing document information.

[
  {
    'document_id': '9fbe403b-c11c-5aae-8ade-ef22980c3ad1',
    'version': 'v1',
    'size_in_bytes': 73353,
    'metadata': {},
    'status': 'success',
    'user_id': '2acb499e-8428-543b-bd85-0d9098718220',
    'title': 'aristotle.txt',
    'created_at': '2024-07-21T20:09:14.218741Z',
    'updated_at': '2024-07-21T20:09:14.218741Z',
    'metadata': {'x': 'y'}
  },
  ...
]

document_ids · Array<string>

An optional array of document IDs to filter the overview.

Document Chunks

Fetch chunks for a particular document:

const documentId = '9fbe403b-c11c-5aae-8ade-ef22980c3ad1';
const chunks = await client.documentChunks(documentId);

Response

response · Array<object>

An array of objects containing chunk information.

[{
  'text': 'Aristotle[A] (Greek: Ἀριστοτέλης Aristotélēs, pronounced [aristotélɛːs]; 384–322 BC) was an Ancient Greek philosopher and polymath...',
  'user_id': '2acb499e-8428-543b-bd85-0d9098718220',
  'document_id': '9fbe403b-c11c-5aae-8ade-ef22980c3ad1',
  'extraction_id': 'aeba6400-1bd0-5ee9-8925-04732d675434',
  'fragment_id': 'f48bcdad-4155-52a4-8c9d-8ba06e996ba3'
  'metadata': {'title': 'aristotle.txt', 'version': 'v0', 'chunk_order': 0}}
},
...]

document_id · string · required

The ID of the document to retrieve chunks for.

Delete Documents

Delete a document by its ID:

const deleteResponse = await client.delete({ document_id: "91662726-7271-51a5-a0ae-34818509e1fd" });

Response

response · object

The response from the R2R system after successfully deleting the documents.

{'results': {}}

filters · { [key: string]: string | string[] } · required

A list of logical filters to perform over input documents fields which identifies the unique set of documents to delete (e.g., {"document_id": {"$eq": "9fbe403b-c11c-5aae-8ade-ef22980c3ad1"}}). Logical operations might include variables such as "user_id" or "title" and filters like neq, gte, etc.