Building Document Processing Pipelines for AI Agents with Amazon Bedrock

‍

Organizations in industries such as insurance, financial services, and healthcare process large volumes of documents every day. Claims forms, policy documents, customer correspondence, and supporting records often arrive as unstructured PDF files. While these documents contain valuable information, their unstructured nature makes them difficult to process consistently and introduces additional considerations on how sensitive data is handled when that information is later made available to generative AI applications.

‍

Building an agent-ready document repository requires more than simply storing PDFs in Amazon S3. Organizations need to extract the information that matters, remove personally identifiable information (PII) where appropriate, and make the resulting content available in a format that can be retrieved by applications and AI agents.

‍

Amazon Bedrock Data Automation (BDA), custom blueprints, AWS Step Functions, Amazon Comprehend, Amazon Bedrock Knowledge Bases, and Amazon S3 can be combined to create a managed pipeline for this process. In this post, we look at how a daily document-processing workflow can extract structure from unstructured documents, create a redacted document corpus, and synchronize that corpus into a Knowledge Base that can be used by Amazon Bedrock agents.

‍

Turning Unstructured Documents into Usable Data

Consider an insurance organization that receives thousands of claims documents every day. Each claim can contain a collection of PDFs, including incident reports, claim forms, invoices, correspondence, and other supporting documents. These files are uploaded to an Amazon S3 bucket as they arrive.

‍

The challenge is not just storing these documents. Downstream applications need to understand what is contained in them. A claims processing application may need information such as a claim number, policy number, incident date, claim type, location, claimed amount, and a description of the incident. Those fields may appear in different locations and formats depending on the document.

‍

Amazon Bedrock Data Automation is designed for this type of problem. BDA can process batches of documents asynchronously using a data automation project or blueprint and write the resulting output to an Amazon S3 location. The InvokeDataAutomationAsync API starts the operation, and the GetDataAutomationStatus API can be used to determine when processing has completed and retrieve the location of the results [1].

‍

Defining the Document Structure with Custom Blueprints

A key part of this architecture is defining what information should be extracted from each document type.

‍

Amazon Bedrock Data Automation supports custom blueprints that describe the fields to extract and the format in which those values should be returned. For a specific field within the document, a blueprint can define a field name, a natural-language description or instruction, a data type, and an inference type. BDA supports types such as strings, numbers, booleans, and arrays, while inference can be used to distinguish values that should be explicitly extracted from those that can be inferred from the document [2].

‍

For example, an insurance claims blueprint might define fields for claim_number, policy_number, incident_date, claim_type, claimed_amount, and incident_description. The description for each field provides additional context about what the value represents, which helps guide extraction. A field such as incident_date could be explicitly extracted when it is stated in the document, while another field could be inferred when the required information must be derived from the available content.

‍

Blueprints can be created and managed through the Amazon Bedrock console or APIs and can be associated with a Data Automation project. Project versions provide an immutable snapshot of the blueprint configuration, which can then be used when processing documents.

‍

This approach gives the document pipeline a defined output contract. Instead of asking a downstream application to interpret every PDF independently, the extraction stage produces a consistent representation of the information that the organization cares about.

‍

Removing PII Before Documents Reach the Knowledge Base

Document repositories often contain sensitive information. Even when the goal is simply to make documents searchable by an AI agent, organizations may not want customer names, addresses, phone numbers, or other PII to be included in the retrieval corpus.

‍

Amazon Comprehend provides asynchronous PII entity detection and redaction for collections of documents stored in Amazon S3. The StartPiiEntitiesDetectionJob operation accepts an S3 input location and an S3 output location. When the job is configured with Mode set to ONLY_REDACTION, Amazon Comprehend writes a copy of the input documents to the output location with identified PII entities redacted [3].

‍

This creates an important separation between the original source documents and the document corpus intended for AI retrieval. The original files can remain in a controlled S3 location, while a separate S3 prefix or bucket stores the redacted documents that are approved for downstream consumption.

‍

For example, an original claim document might contain a customer's name and address. The redacted copy can replace those entities according to the configured redaction policy while preserving the surrounding document content that an agent needs to answer questions about the claim. Amazon Comprehend supports configuration of the PII entity types to redact and how the redacted values are represented.

‍

Ingesting the Redacted Corpus into Amazon Bedrock Knowledge Bases

Once the redaction job has completed, the pipeline has a clean document set in Amazon S3 that can be synchronized into an Amazon Bedrock Knowledge Base.

‍

Amazon Bedrock Knowledge Bases support S3-backed data sources. The StartIngestionJob API begins an ingestion job for a specified Knowledge Base and data source, allowing changes in the S3 location to be synchronized and indexed for retrieval [4].

‍

At this point, the pipeline has transformed a collection of unstructured and potentially sensitive source documents into a redacted corpus that is available through a Knowledge Base. Agents can then use the Knowledge Base as a retrieval source when answering questions or completing tasks that depend on the organization's document content.

‍

It is useful to keep the two outputs of this architecture conceptually separate. BDA provides structured extraction for the fields defined by the blueprint, while Amazon Comprehend produces the redacted document corpus used by the Knowledge Base. If an application also needs the extracted fields to become searchable Knowledge Base content or metadata, an additional transformation or indexing step can be introduced between those stages.

‍

Orchestrating the Processing with Step Functions

The services described above each perform a specific part of the document-processing pipeline, but the workflow needs an orchestration layer to coordinate them. AWS Step Functions provides that layer, connecting the extraction, redaction, and ingestion stages into a single pipeline.

‍

For this workflow, we can use Amazon EventBridge Scheduler to start an AWS Step Functions state machine once a day at a specific time. EventBridge Scheduler supports recurring schedules and can target the Step Functions StartExecution API, making it a natural entry point for a batch-oriented document pipeline [5]. When the daily schedule starts the state machine, Step Functions can invoke Amazon Bedrock Data Automation to process the documents using the selected project and blueprint. Because the BDA processing is asynchronous, the state machine can retain the invocation ARN and periodically check the operation with GetDataAutomationStatus. When processing succeeds, BDA makes the results available in Amazon S3.

‍

Step Functions can then use that output to start the Amazon Comprehend PII redaction job. The state machine retains the returned job identifier and monitors the job with DescribePiiEntitiesDetectionJob until the redacted documents are available in the designated S3 location [6].

‍

Once redaction has completed, the workflow can start an Amazon Bedrock Knowledge Base ingestion job using StartIngestionJob. Step Functions can monitor the returned ingestion job identifier with GetIngestionJob until the job has reached status COMPLETE and the Knowledge Base has finished synchronizing the redacted corpus [7].

‍

This creates a clear sequence:

‍

EventBridge Scheduler → Step Functions → BDA → Comprehend → Knowledge Base

‍

Each stage has a defined input, output, and completion condition, while Step Functions controls the transitions between them. The workflow therefore does not need to keep any individual service request open while an asynchronous operation is running. Instead, it can wait, check the operation status, and continue only when the required output is available.

‍

This orchestration model also makes the pipeline easier to extend. Additional validation, transformation, routing, or approval steps can be introduced as separate states without changing the responsibilities of the underlying services.

‍

‍

A Better Shape for Document Processing

The value of this architecture is not simply that several AWS services are connected together. The services map naturally to different responsibilities in the document lifecycle.

‍

Amazon S3 provides durable storage for the document corpus. EventBridge Scheduler provides a cron trigger for recurring processing. Step Functions coordinates the asynchronous operations and controls the order in which they run. Amazon Bedrock Data Automation provides document understanding and structured extraction of unstructured data through custom blueprints. Amazon Comprehend provides a dedicated PII detection and redaction stage. Finally, Amazon Bedrock Knowledge Bases provides the retrieval layer that makes the sanitized content available to generative AI applications.

‍

Keeping those responsibilities separate also makes the pipeline easier to change. A new document type can be supported by creating or updating the appropriate blueprint. The PII policy can be adjusted independently from the extraction logic. The Knowledge Base can be updated without changing how documents are uploaded to S3.

‍

The architecture also gives organizations a clear place to enforce controls. The original documents and the redacted documents can be stored separately, and the workflow can ensure that only the approved output location is connected to the Knowledge Base.

‍

It is important to mention that there is an alternative approach to address PII exposure at the model interaction layer: Amazon Bedrock Guardrails can detect sensitive information in prompts and model responses and either block or mask it before it is returned to the application [8]. This provides an important additional layer of protection, particularly when sensitive information could otherwise appear in a generated response. However, it addresses the problem after the data has already entered the retrieval and generation pipeline. By detecting and removing PII before ingestion, this architecture takes a more preventative approach: sensitive information never needs to enter the Knowledge Base in the first place, so it cannot subsequently be retrieved and exposed to an agent or used as context for generation. Guardrails can therefore complement the pre-ingestion redaction stage rather than replace it, providing defense in depth across both the data-ingestion and model-response boundaries.

‍

What This Means for AI Agents

A retrieval-augmented agent is only as useful as the information it can retrieve. Giving an agent direct access to a large collection of unstructured documents can make retrieval less predictable and can introduce unnecessary exposure to sensitive information. By introducing an extraction and redaction pipeline before ingestion, organizations can create a more controlled foundation for their agents. The Knowledge Base contains content that has already passed through a defined processing workflow, while the structured output from BDA provides another representation of the same business information for potential downstream applications. For an insurance organization, this can support use cases such as claims investigation, policy research, internal knowledge assistants, or customer service workflows. Instead of building document-processing logic into every application, the organization can establish a reusable pipeline that prepares information once and makes it available to multiple downstream consumers.

‍

This pipeline is the data layer of an agent platform. The accounts, identity, networking, and monitoring underneath it matter just as much, and our Ethermind case study shows what that foundation looks like for AI knowledge agents on AWS.

‍

What’s Next

Every organization has different requirements when it comes to document processing, data protection, and generative AI. Amazon Bedrock Data Automation, Step Functions, Amazon Comprehend, and Amazon Bedrock Knowledge Bases provide a strong set of building blocks when you want document extraction, PII protection, and retrieval to work together as one controlled workflow.

‍

If your team is looking to build agent-based applications over existing document repositories, this architecture is worth considering early. It helps make document processing repeatable and traceable, keeps sensitive-data handling explicit, and gives you greater control over what information ultimately becomes available to your AI applications.

‍

We are just getting started with this topic. If you want to explore how this pattern could fit your own document-processing workflow, how to adapt the PII controls to your requirements, or how to structure the ingestion and retrieval stages around your existing data, we would be happy to help. Let’s look at your current architecture and identify where these AWS services can help you build a more secure and maintainable foundation for generative AI.

‍

References

‍

[1] “InvokeDataAutomationAsync”, AWS Docs,

https://docs.aws.amazon.com/bedrock/latest/APIReference/API_data-automation-runtime_InvokeDataAutomationAsync.html

‍

[2] “Creating blueprints”, AWS Docs,

https://docs.aws.amazon.com/bedrock/latest/userguide/bda-idp.html

‍

[3] “Redacting PII entities with asynchronous jobs (API)”, AWS Docs,

https://docs.aws.amazon.com/comprehend/latest/dg/redact-api-pii.html

‍

[4] “StartIngestionJob”, AWS Docs,

https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent_StartIngestionJob.html

‍

[5] “Using Amazon EventBridge Scheduler to start a Step Functions state machine execution”, AWS Docs,

https://docs.aws.amazon.com/step-functions/latest/dg/using-eventbridge-scheduler.html

‍

[6] “DescribePiiEntitiesDetectionJob”, AWS Docs,

https://docs.aws.amazon.com/comprehend/latest/APIReference/API_DescribePiiEntitiesDetectionJob.html

‍

[7] “GetIngestionJob”, AWS Docs,

https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent_GetIngestionJob.html

‍

[8] “Remove PII from conversations by using sensitive information filters”, AWS Docs,

https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-sensitive-filters.html

‍