Assessing your storage activity and usage with Amazon S3 Storage Lens

The data provided will help you quickly understand your versioning costs and determine how to manage storage better by looking at the amount of storage for noncurrent versions and their tiers. The pricing is the same as for the current versioned copy. The following illustrates how noncurrent versions can build up quickly over time, adding to the cost.

You have an object that is 4GiB in size and gets updated five times daily. That means you would have five noncurrent versions daily and one current version for six. If we take the five-time daily changes and times them by an average of 30 days, we will get five changes daily x 30 average number of days in a month = 150 noncurrent version copies. This can add up quickly, and whatever storage tier the nonversioned objects are in, that is the price you are charged.

Another example is if you have a 1MB file and 200 copies (noncurrent versions), you pay for 200MB of storage.

Once you've identified the buckets that have Versioning enabled and which objects already have noncurrent versions, let's take a moment to consider the value of having copies of the object. It's important to ask yourself, "Do I need or want copies of the object?" If the answer is yes, then the question is how long you need to keep a copy and how many copies are required. These answers will help you determine the cost of maintaining the copies and the automation necessary to manage the object's transitions. So, take your time to decide what's best for you and your needs. Not all buckets are equal.

If you do not need versioning on a bucket, follow the process below to suspend Versioning, remove nonversioned copies, and delete markers that are not required. This will save costs and provide a pseudo-disabled versioned bucket.

NOTE: Before you proceed with the procedure below, once you delete the noncurrent files, all noncurrent copies of the object will be gone forever, and there's no way to recover them. So, please ensure you're in the right S3 bucket and consider whether you need the noncurrent versions before you delete anything. You can test first on a test bucket.

1. In the AWS S3 Console, find the bucket on which you would like to suspend Versioning and select it.
2. Go to the Properties tab.
3. In the Properties tab, find the section regarding Versioning and select Edit.

4. Select the radio button Suspend.

5. Select the box I acknowledge the outcomes of suspending Bucket Versioning

6. Click on Save changes

7. The Versioning on a bucket is now suspended.

• Suspending Versioning stops any new versions of the objects from being created.
• Suspending Versioning has no impact on existing objects in the bucket.

The next step is to remove the noncurrent versions and expired delete markers. One of the most common ways, especially with millions of files, is to use Lifecycle Configuration. We will use that in our example as recommended by AWS, but there are also other ways. See the following for options.

Options to delete millions of objects from a versioning-enabled AWS S3 bucket

Still, in the S3 bucket, select the Management tab.

In the Management window, there is a section called Lifecycle Rules. Click on the Create lifecycle rule.

1. In the Create lifecycle rule, fill in the following sections
   Lifecycle rule name: Use a name that describes what the rule does.
   For example DeleteNV-1days-DeleteMarker-IMU-7days
   In the rule's name, we say to delete nonversioned copies after one day, DeleteMarkers, and incomplete multipart uploads after seven days.
2. For Choose a rule scope, select the radio button, "Apply to all objects in the bucket." A box will appear with a warning message. Read the warning message and select "I acknowledge that this rule will apply to all objects in the bucket." Later in the article, the option to Limit the scope of this rule using one or more filters is discussed.
3. Under Lifecycle rule actions, select the boxes Permanently delete noncurrent versions of objects and Delete expired object delete markers or incomplete multipart uploads.
4. Two new sections will appear below: Permanently delete noncurrent versions of objects and Delete expired object delete markers or incomplete multipart upload.
5. Under Permanently delete noncurrent versions of objects, in the text boxes, Days after objects become noncurrent, enter 1. The number has to be greater than zero. This is one of the decisions discussed previously on how long to keep noncurrent objects. In the text box right of it, Number of newer version to retain—Optional, leave blank. Remember the question, "How many versions do I need to retain?". This is where you would set that number. It will always retain that number of copies or noncurrent versions.
6. Next is the Delete expired object delete marker and Delete incomplete multipart uploads section. Select the boxes Delete expired object delete markers, and Delete incomplete multipart uploads. A new selection will appear called Number of days. In the text box, enter 7. This is how many days before deleting any incomplete multipart uploads. This should give time for an incomplete multipart upload to finish if it needs to recover.

 

You may be curious about why we selected the option to delete incomplete multipart uploads. This is a simple and effective way to cut down on costs that can be set up automatically on your S3 buckets Lifecycle rule. Remember that you will be charged for any incomplete multipart upload parts based on the storage class specified when the parts were uploaded. You can again utilize the S3 Storage Lens to analyze incomplete multipart uploads. For more information on incomplete multipart uploads and analyzing them with Storage Lens, please refer to the following article.

Discovering and Deleting Incomplete Multipart Uploads to Lower Amazon S3 Costs | Amazon Web…

The following is further information from AWS regarding deleting noncurrent versions of objects.

A versioning-enabled bucket has one current version and zero or more noncurrent versions for each object. When you delete an object, note the following:

If you don’t specify a version ID in your delete request, Amazon S3 adds a delete marker instead of deleting the object. The current object version becomes noncurrent, and the delete marker becomes the current version.

If you specify a version ID in your delete request, Amazon S3 deletes the object version permanently (a delete marker is not created).

A delete marker with zero noncurrent versions is referred to as an expired object delete marker. (source).

Congratulations! You have successfully taken control of a versioned S3 bucket. Depending on the lifecycle configuration you have chosen for the use case, you should see a reduced cost after a week. However, it's important to note that Lifecycle Rules run asynchronously, so removing the noncurrent version objects may take some time. But don't worry; AWS won't charge you for the additional days it may take to complete the process.

You can enable versioning again now that a Lifecycle Management rule will handle noncurrent versions automatically. Alternatively, you can keep the versioning suspended if you don't want versions of the objects. Please note that any new objects placed in the S3 bucket while versioning is suspended will have a null ID marker and will not be eligible for versioning.

Sometimes, you need versioning enabled on an S3 bucket but only need Lifecycle Management on particular objects. Lifecycle Management allows you to apply the rule for specific prefixes. It is called "Limit the scope of this rule using one or more filters." Using filters, you can decide which noncurrent objects are subject to deletion. You can also use the filter feature to have different Lifecycle rules for other objects.

You can filter where the objects are stored. For example, if you have files in a logs folder, you can set the filter as logs/. The Lifecycle rule applies to logs/log.txt, logs/temp3.txt, and logs/test1.txt. The rule does not apply to an object in the root called example.jpg. You can also filter file extensions, tags, and the size of the objects. See the following article for more information regarding filtering and different examples.

Lifecycle configuration elements

You now have the knowledge to control S3 costs based on noncurrent versions of objects. It is always recommended to try changes in a test or development environment before moving to production.

---

Resources:

• Amazon S3 cost optimization for predictable and dynamic access patterns.
• Optimize storage costs with new Amazon S3 Lifecycle filters and actions.
• Examples of S3 Lifecycle Configuration.
• How can I reduce unexpected high costs for Amazon S3?
• Deleting object versions from a versioning-enabled bucket.
• Assessing your storage activity and usage with Amazon S3 Storage Lens.
• Options to delete millions of objects from a versioning-enabled AWS S3 bucket