S3-Compatible Storage

Required

Fields required for a valid configuration.

Path	Type	Required	Description	Default	Constraints
required	object	Yes	—	—	no extra properties
required.bucket	string	Yes	Bucket name for AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, and other S3-compatible endpoints	—	—

Masked

Sensitive fields under(secrets/credentials).

Path	Type	Required	Description	Default	Constraints
masked	object	No	Optional static credentials. Leave empty to use ambient AWS credentials chain.	—	no extra properties
masked.aws_access_key_id	string	No	S3-compatible access key ID	—	—
masked.aws_secret_access_key	string	No	S3-compatible secret access key	—	—
masked.aws_session_token	string	No	Optional session token for temporary credentials	—	—

Optional

Optional configuration fields.

Path	Type	Required	Description	Default	Constraints
optional	object	No	—	—	no extra properties
optional.connection	object	No	—	—	no extra properties
optional.connection.endpoint_url	string	No	Custom endpoint URL for MinIO/R2/B2/Garage and other S3-compatible providers	—	format uri
optional.connection.max_keys_per_page	integer	No	Maximum objects requested per provider list API call	200	min 1, max 1000
optional.connection.max_object_bytes	integer	No	Maximum bytes downloaded per object for MIME detection and text extraction	5242880	min 1024, max 52428800
optional.connection.region_name	string	No	Region (recommended for AWS; required by some S3-compatible providers)	—	—
optional.connection.request_timeout_seconds	number	No	Network timeout in seconds for list/download operations	30	min 1, max 300
optional.connection.verify_ssl	boolean	No	TLS certificate verification toggle	true	—
optional.scope	object	No	Object scope and filtering controls.	—	no extra properties
optional.scope.exclude_extensions	array	No	Optional extension denylist	—	—
optional.scope.exclude_extensions[]	string	No	—	—	—
optional.scope.include_content_preview	boolean	No	Download object bytes to infer MIME and extract detector-ready text previews	true	—
optional.scope.include_empty_objects	boolean	No	Include zero-byte objects in extraction results	false	—
optional.scope.include_extensions	array	No	Optional extension allowlist (for example, .pdf, .csv, .parquet)	—	—
optional.scope.include_extensions[]	string	No	—	—	—
optional.scope.include_object_metadata	boolean	No	Attach provider metadata (etag, size, content-type hints, timestamps) to asset checksums	true	—
optional.scope.prefix	string	No	Object key prefix filter (for example, exports/2026/)	—	—

Extracted Metadata

File

Field	Type	Required	Description
size_bytes	integer	Yes	Raw byte size of the content
mime_type	string	Yes	Resolved MIME type
parse_error	string	No	Set when content extraction failed
image_width	integer	No	Width in pixels
image_height	integer	No	Height in pixels
page_count	integer	No	Number of pages (pdf)
paragraph_count	integer	No	Number of paragraphs (docx)
table_count	integer	No	Number of tables (docx)
row_count	integer	No	Number of data rows
columns	object[]	No	Columns as {name, type} objects (type may be empty for csv/xlsx)
encoding	string	No	Detected character encoding
json_root_type	string	No	Root JSON type: object, array, or scalar
top_level_keys	integer	No	Number of top-level keys when the root is an object
array_length	integer	No	Length when the root is an array
provider	string	Yes	Storage provider label
object_key	string	Yes	Object key/path
etag	string	No	Object entity tag
source_hash	string	No	Hash of the parent asset (embedded images and archive members only)
location	string	No	Location within the parent (embedded image location or archive member path)

Image

Field	Type	Required	Description
size_bytes	integer	Yes	Raw byte size of the content
mime_type	string	Yes	Resolved MIME type
parse_error	string	No	Set when content extraction failed
image_width	integer	No	Width in pixels
image_height	integer	No	Height in pixels
provider	string	No	Storage provider label
object_key	string	No	Object key/path
etag	string	No	Object entity tag
source_hash	string	No	Hash of the parent asset (embedded images and archive members only)
location	string	No	Location within the parent (embedded image location or archive member path)

Audio

Field	Type	Required	Description
size_bytes	integer	Yes	Raw byte size of the content
mime_type	string	Yes	Resolved MIME type
parse_error	string	No	Set when content extraction failed
provider	string	Yes	Storage provider label
object_key	string	Yes	Object key/path
source_hash	string	No	Hash of the parent asset (embedded images and archive members only)
location	string	No	Location within the parent (embedded image location or archive member path)

Video

Field	Type	Required	Description
size_bytes	integer	Yes	Raw byte size of the content
mime_type	string	Yes	Resolved MIME type
parse_error	string	No	Set when content extraction failed
provider	string	Yes	Storage provider label
object_key	string	Yes	Object key/path
source_hash	string	No	Hash of the parent asset (embedded images and archive members only)
location	string	No	Location within the parent (embedded image location or archive member path)

Archive

Field	Type	Required	Description
size_bytes	integer	Yes	Raw byte size of the content
mime_type	string	Yes	Resolved MIME type
parse_error	string	No	Set when content extraction failed
provider	string	Yes	Storage provider label
object_key	string	Yes	Object key/path
etag	string	No	Object entity tag
source_hash	string	No	Hash of the parent asset (embedded images and archive members only)
location	string	No	Location within the parent (embedded image location or archive member path)

Examples

AWS S3 bucket scan

Scan an S3 bucket for document metadata and content previews using IAM credentials

Config Payload

{
  "type": "S3_COMPATIBLE_STORAGE",
  "required": {
    "bucket": "your-bucket-name"
  },
  "masked": {
    "aws_access_key_id": "your-access-key-id",
    "aws_secret_access_key": "your-secret-access-key"
  },
  "optional": {
    "scope": {
      "include_content_preview": true
    }
  },
  "sampling": {
    "strategy": "AUTOMATIC"
  }
}

Schedule

{
  "enabled": true,
  "preset": "nightly",
  "cron": "9 2 * * *",
  "timezone": "UTC"
}

MinIO document scan with prefix filter

Scan a MinIO bucket restricted to a key prefix and specific file types

Config Payload

{
  "type": "S3_COMPATIBLE_STORAGE",
  "required": {
    "bucket": "your-bucket-name"
  },
  "masked": {
    "aws_access_key_id": "your-minio-access-key",
    "aws_secret_access_key": "your-minio-secret-key"
  },
  "optional": {
    "connection": {
      "endpoint_url": "https://minio.your-company.internal:9000",
      "region_name": "us-east-1"
    },
    "scope": {
      "prefix": "exports/",
      "include_extensions": [
        ".csv",
        ".parquet",
        ".pdf"
      ],
      "include_content_preview": true
    }
  },
  "sampling": {
    "strategy": "AUTOMATIC"
  },
  "detectors": [
    {
      "type": "PII",
      "enabled": true
    },
    {
      "type": "SECRETS",
      "enabled": true
    }
  ]
}

Schedule

{
  "enabled": true,
  "preset": "nightly",
  "cron": "14 2 * * *",
  "timezone": "UTC"
}