Delete data from RDS on the onSuccess of a copyActivity in AWS Data Pipeline

Question

I am trying to backup data from RDS(postgres) to s3 incrementally. For this I'm using AWS Data Pipeline. I am able to copy the data, it all works.

The issue I'm facing is that I'm not able to find out a way to delete the already copied data in RDS. I can see there is an onSuccess action, but it only allows me to post a message to SNS or terminate the activity.

Is there a way I can delete data on successful copy of data from RDS to S3 in AWS Data Pipeline?

Answer 1

onSuccess/onFailure are primarily for status reporting. DPL doesn't really support the idea of branching or error handling. You set up your graph, and it runs through step by step. If something fails, it either pauses or aborts.

I would create a new activity that would run after your copy, using the dependsOn property:

{
  "id": "DeleteDataStep",
  ...
  "dependsOn": { "ref": "CopyRdsToS3Step" },
  ...
}