> ## Documentation Index > Fetch the complete documentation index at: https://praison.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Cloud Storage Sources > Load knowledge from S3, Google Cloud Storage, and Azure Blob buckets Pass a cloud object storage URI as a knowledge source and PraisonAI downloads and indexes it automatically. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent agent = Agent( name="Handbook Assistant", instructions="Answer questions using the handbook.", knowledge=["s3://my-bucket/handbook.pdf"], ) agent.start("What's the leave policy?") ``` ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR subgraph "Cloud knowledge flow" URI[โ˜๏ธ s3:// gs:// az://] --> Detect{๐Ÿ” Cloud URI?} Detect -->|Yes| Fetch[โฌ‡๏ธ Download to temp] Fetch --> Read[๐Ÿ“– Existing reader] Read --> Index[๐Ÿ—ƒ๏ธ Vector store] Index --> Cleanup[๐Ÿงน Delete temp file] end classDef uri fill:#6366F1,stroke:#7C90A0,color:#fff classDef detect fill:#F59E0B,stroke:#7C90A0,color:#fff classDef process fill:#189AB4,stroke:#7C90A0,color:#fff classDef index fill:#10B981,stroke:#7C90A0,color:#fff classDef cleanup fill:#8B0000,stroke:#7C90A0,color:#fff class URI uri class Detect detect class Fetch,Read process class Index index class Cleanup cleanup ``` ## Quick Start Install `boto3` and pass an `s3://` URI. ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} pip install boto3 ``` ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent agent = Agent( name="Handbook Assistant", instructions="Answer from the handbook.", knowledge=["s3://my-bucket/handbook.pdf"], ) agent.start("What's the leave policy?") ``` Install `google-cloud-storage` and pass a `gs://` URI (`gcs://` also works). ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} pip install google-cloud-storage ``` ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent agent = Agent( name="Reports Assistant", instructions="Answer from the quarterly report.", knowledge=["gs://reports/q4.pdf"], ) agent.start("Summarise Q4 revenue.") ``` Install `azure-storage-blob` and pass an `az://container/blob` URI or an Azure Blob HTTPS URL. ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} pip install azure-storage-blob ``` ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent agent = Agent( name="Policy Assistant", instructions="Answer from the policy document.", knowledge=["https://acct.blob.core.windows.net/docs/policy.pdf"], ) agent.start("What's the refund window?") ``` *** ## Choose Your Provider Match where your file lives to the right URI form. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph TB Q{๐Ÿ“ Where does
your file live?} --> AWS[AWS S3] Q --> GCP[Google Cloud] Q --> AZ[Azure] AWS --> AWSU[s3://bucket/key] GCP --> GCPU[gs://bucket/key] AZ --> AZType{Have an
account URL?} AZType -->|Yes| AZURL[https://acct.blob.core.windows.net/
container/blob] AZType -->|No| AZShort[az://container/blob] classDef decision fill:#F59E0B,stroke:#7C90A0,color:#fff classDef provider fill:#6366F1,stroke:#7C90A0,color:#fff classDef uri fill:#10B981,stroke:#7C90A0,color:#fff class Q,AZType decision class AWS,GCP,AZ provider class AWSU,GCPU,AZURL,AZShort uri ``` *** ## Install the Right SDK Each SDK is imported only when a URI of that scheme is used โ€” installing PraisonAI does not pull in all three. ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} pip install boto3 # for s3:// ``` ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} pip install google-cloud-storage # for gs:// or gcs:// ``` ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} pip install azure-storage-blob # for az:// or Azure blob URLs pip install azure-identity # optional, for Azure managed identity ``` A missing SDK raises `CloudSourceError` with the exact `pip install โ€ฆ` command to run. *** ## Credentials Each provider uses its SDK's standard credential discovery โ€” no extra PraisonAI config. Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (plus optional `AWS_SESSION_TOKEN`), or use `~/.aws/credentials`, or an IAM role attached to your EC2 / ECS / Lambda environment. ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} export AWS_ACCESS_KEY_ID=AKIA... export AWS_SECRET_ACCESS_KEY=... ``` Point `GOOGLE_APPLICATION_CREDENTIALS` at a service-account JSON file, run `gcloud auth application-default login`, or use a workload-identity-attached environment. ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json ``` Set `AZURE_STORAGE_CONNECTION_STRING`. For an HTTPS Azure blob URL, `DefaultAzureCredential` (from optional `azure-identity`) is tried when the connection string is missing or names a different account. ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=..." ``` *** ## URI Forms | Provider | URI form | Notes | | -------------------- | ------------------------------------------------------------ | ---------------------------------------------- | | AWS S3 | `s3://bucket/key` | Keys are percent-decoded | | Google Cloud Storage | `gs://bucket/key` (or `gcs://โ€ฆ`) | | | Azure Blob (short) | `az://container/blob` (or `abfs://โ€ฆ`) | Account from `AZURE_STORAGE_CONNECTION_STRING` | | Azure Blob (URL) | `https://.blob.core.windows.net//` | Account bound to URL; wins over env | *** ## How It Works A cloud URI is resolved to a local temp file, read by the same reader used for local files, then the temp file is removed. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant User participant Agent participant Knowledge participant CloudFetch participant Reader participant VectorStore User->>Agent: knowledge=["s3://bucket/handbook.pdf"] Agent->>Knowledge: index source Knowledge->>CloudFetch: download to temp dir CloudFetch-->>Knowledge: local temp path (extension preserved) Knowledge->>Reader: read + chunk Reader->>VectorStore: store chunks (original URI in metadata) Knowledge->>CloudFetch: temp file deleted ``` | Step | What happens | | ---------------- | ---------------------------------------------------------------------------------- | | 1. Resolve | The cloud URI is parsed into provider, container, and key. | | 2. Fetch | The object is downloaded into a `TemporaryDirectory`, keeping its extension. | | 3. Reuse readers | The existing PDF/DOCX/text reader indexes the temp file โ€” no per-provider parsing. | | 4. Clean up | The temp file is deleted after indexing, so nothing accumulates on disk. | Design notes worth knowing: * **Extension is preserved.** Readers dispatch on file extension โ€” a PDF kept as `.pdf` is parsed correctly rather than read as plain text. * **Ordering matters.** The cloud branch runs **before** the `http(s)://` branch, so an Azure blob HTTPS URL is downloaded, not handed to the web fetcher. * **SDKs stay optional.** Each cloud SDK is imported only when its scheme is used. * **Temp file removed after indexing.** The download lives in a `TemporaryDirectory` and is cleaned up automatically. * **Original URI kept in metadata.** Citations reference the `s3://โ€ฆ` / `gs://โ€ฆ` URI, not the meaningless temp path. * **Azure account is bound to the URL.** A URL for account A is never served from configured account B โ€” the account named in the URL wins. *** ## Mix Cloud and Local Sources Cloud URIs, local files, and raw text can share one list. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent agent = Agent( name="Research Assistant", instructions="Answer from the handbook and local notes.", knowledge=[ "s3://my-bucket/handbook.pdf", "gs://reports/q4.pdf", "https://acct.blob.core.windows.net/docs/policy.pdf", "notes.md", "Key fact: launch is in March.", ], ) agent.start("When is the launch?") ``` *** ## Error Reference `CloudSourceError` reports the failure instead of silently indexing an empty document. | Error message contains | Meaning | Fix | | ------------------------------------------------------------ | --------------------------------------------------------------------------- | ------------------------------------------------------------- | | `pip install boto3` (or the GCS / Azure package) | The SDK for that scheme isn't installed | Run the exact `pip install` from the error | | `Not a supported cloud source` | The scheme isn't `s3`/`gs`/`gcs`/`az`/`abfs` and it isn't an Azure blob URL | Check the URI; supported schemes are listed above | | `must include a container and a blob name` (Azure) | Azure blob URL missing `/container/blob` path | Add both parts to the URL | | `container and an object` | URI names a bucket but no object key | Add the object key | | `AZURE_STORAGE_CONNECTION_STRING in the environment` | `az://` form used without env var | Set the env var, or use the account-bearing HTTPS URL form | | `Fetch reported success but wrote no file` | Silent no-op download (rare SDK bug or wrong path) | Verify the object exists and the credentials have `GetObject` | | `Could not fetch โ€ฆ AccessDenied` (or similar provider error) | Provider rejected the download | Check credentials and the object's ACL | *** ## Best Practices Each SDK is imported lazily. Install just `boto3`, `google-cloud-storage`, or `azure-storage-blob` for the provider you use โ€” don't bloat your app with all three. In production, attach a bucket-scoped IAM role (or workload identity) instead of exporting long-lived access keys. When multiple Azure accounts are in play, use the `https://.blob.core.windows.net/...` form โ€” the account in the URL wins over `AZURE_STORAGE_CONNECTION_STRING`. Cloud downloads happen on every index. Cache expensive parses by indexing into a persistent vector store once, not on every process start. The original cloud URI is preserved in metadata. Use it as your citation source โ€” the temp path is deleted after indexing. *** ## Related Chunking, embedding, and search options. What `Agent(knowledge=...)` accepts today.