CONCURRENT AND NON-BLOCKING OBJECT DELETION FOR CROSS-REGION REPLICATIONS

Smart overview of the Invention

Concurrent and non-blocking techniques for object deletion during cross-region replications are described. A target file system employs a replication pipeline and a deletion pipeline operating in parallel. The deletion pipeline removes processed objects reaching the last stage of the replication pipeline after each checkpoint. This method allows for efficient resource utilization and supports disaster recovery by reducing downtime.

Functionality

In the event of a non-recoverable failure, the replication process can restart without waiting for unused objects to be deleted. This is facilitated by a generation number associated with each object, allowing unused objects to be deleted in the background. The system operates by transferring objects between a source and target file system, with pipelines running in parallel to maintain consistent throughput.

Technical Details

The method involves performing a replication pipeline and a deletion pipeline in the target file system. Objects reaching the last pipeline stage are deleted by the deletion threads. The system is designed to handle failures by restarting the replication pipeline and continuing deletion processes in parallel. Checkpoints are created to ensure data is processed and deleted efficiently, maintaining the same throughput as the replication pipeline.

System Design

The system includes data processors and a non-transitory computer-readable medium containing instructions executable by processors. It supports multiple pipeline stages and handles both recoverable and non-recoverable failure events. Unused objects are identified by unique identifiers and deleted to free resources for new replications or restarts.

Benefits

This approach addresses the challenges of managing large data volumes during cross-region replications. By cleaning up unused data concurrently, it prevents storage overload and reduces unnecessary costs. The method ensures that the replication process meets typical RPO requirements, enhancing disaster recovery capabilities and ensuring business continuity.