How to implement a time-to-live (TTL) in specific folders in an Azure Storage Account

In today’s world of big data, storage is one of the biggest challenges faced by organizations. With the amount of data growing rapidly, it becomes imperative to have a system in place to manage the data effectively. One way to do this is by implementing a Time to Live (TTL) in Azure Storage Account.

TTL is a feature that automatically deletes the data after a specified period of time. This helps in reducing storage costs, and also ensures that the data is not kept indefinitely.

Imagine you have created a container, and within that container, a folder with historical data that you don’t want to delete, and another folder that you only want to keep the data for the last 30 days, for example. Then we have to think of a way to configure this TTL.

In this post, I will show you how to implement a TTL in specific folders of an Azure Storage Account using Storage Account rules or Azure Automation.

Azure Storage Account rules

If you want to set a custom Time-to-Live (TTL) on specific folders within your storage account, rather than just at the container level, you can create prefix-match filtered storage account rules. Here’s how:

Open the Azure Portal and navigate to your storage account.
Go to the “Blob Service” section and select “Lifecycle management.”
Click on “Add a rule” to create a new rule.
Give your rule a name and select “Enabled” for the status.
Under “Actions,” select “Delete” for “When to transition.”
Under “Filter,” select “Only objects that meet the following conditions.”
In the “Filter” section, select “Prefix match” and enter the prefix of the folder you want to apply the TTL to. For example, if you want to apply the TTL to all objects within a folder named “private,” your prefix would be “private/”.
Enter the number of days you want the objects to live in the “Days after object creation” field.
Click on “Save” to apply the rule.

For more information on how to create prefix-match filtered storage account rules and manage the lifecycle of your objects in Azure Storage, please refer to the Microsoft Learn page on Azure Blob Storage Lifecycle Management.

With this prefix-match filtered storage account rule in place, any objects within the specified folder will automatically be deleted after the specified number of days, allowing you to have custom TTLs on specific folders within your storage account.

Additionally, creating prefix-match filtered storage account rules can also be done using infrastructure as code (IAC) tools such as Terraform, Azure Biceps, Azure Resource Manager (ARM) templates, and others. This allows you to manage your storage account configuration in a repeatable, version-controlled manner, making it easier to automate the deployment of new storage accounts or make changes to existing ones. These tools can also help to reduce the risk of human error, as well as increase consistency and reliability. Whether you prefer to use a graphical user interface like the Azure Portal or an IAC tool, you have multiple options for creating prefix-match filtered storage account rules to set custom TTLs on specific folders.

Azure Automation

Azure Automation provides a platform to automate tedious and repetitive tasks in the cloud. To implement TTL in Azure Storage Account using Azure Automation, follow these steps:

Create a new runbook in Azure Automation.
Add a new PowerShell script in the runbook to delete the data in the specified folders after a certain time period.
Set up a schedule for the runbook to run at the desired frequency.

You could be inspired by this PowerShell I provide:

# Variables to store the Azure Storage Account name, the time to live (in days), and the list of folders
$storageAccountName = "<Your Azure Storage Account Name>"
$timeToLiveInDays = 30
$foldersToApplyTTL = @("folder1", "folder2", "folder3")

# Connect to the Azure Storage Account
$storageAccount = Get-AzStorageAccount -ResourceGroupName "<Your Resource Group Name>" -Name $storageAccountName
$context = $storageAccount.Context

# Get the list of all containers in the Azure Storage Account
$containers = Get-AzStorageContainer -Context $context

# Loop through each container
foreach ($container in $containers)
{
    # Get the list of all blobs in the container
    $blobs = Get-AzStorageBlob -Container $container.Name -Context $context

    # Loop through each blob
    foreach ($blob in $blobs)
    {
        # Get the folder name from the blob path
        $folderName = (Split-Path $blob.Name -Parent).TrimEnd("/")

        # Check if the folder name is in the list of folders to apply the TTL
        if ($foldersToApplyTTL -contains $folderName)
        {
            # Get the date when the blob was last modified
            $lastModified = $blob.Properties.LastModified

            # Calculate the difference between the current date and the last modified date (in days)
            $differenceInDays = [math]::Round(((Get-Date) - $lastModified).TotalDays)

            # Check if the difference is greater than the time to live
            if ($differenceInDays -gt $timeToLiveInDays)
            {
                # Delete the blob
                Remove-AzStorageBlob -Blob $blob.Name -Container $container.Name -Context $context
            }
        }
    }
}

In conclusion, implementing a TTL in Azure Storage Account is a great way to manage the data effectively, reduce storage costs and ensure data is not kept indefinitely. The process is simple and straightforward.

And on a lighter note, if your data is like a hoarder, it could be time to kick it out by implementing a TTL.

Hope you found this post helpful. Happy data hoarding cleaning!