An IT leaders guide to Cloud Storage

"Every person has their own path, but no one's journey is easy." – 19^th Century IT philosophers probably

TL: DR - The holy grail of cloud storage is Cheap AND Durable & consistent AND Low-latency AND high throughput AND meets the requirements of our business. AND SECURE. And the journey requires chooses the right combination of solutions offering the greatest flexibility – and an end to presenting IT leaders with either-or options.

17 ago Amazon launched AWS S3, the date was March 14th, 2006, the service has since become synonymous with cloud storage. The broad adoption of Amazon S3 and related tooling has created dozens (hundreds?) of services based on the S3 API.

Almost two decades later the shift to cloud storage still poses challenges for many IT leaders and their applications and workloads. And even for those grizzled IT vets who remember the day S3 launched, cloud storage is not what it used to be, it has grown up and has kids of its own. Cloud storage offerings are becoming more use case specific leading many IT leaders to question which offering is right for them.

To first understand the path to cloud storage it is good to first understand what cloud storage is.

What is cloud storage?

Cloud storage is a service model in which data is transmitted and stored on remote storage systems, where it is maintained, managed, backed up and made available to users over a network -- typically, the internet. Users generally pay for their cloud data storage on a per-consumption, monthly rate.

At its core cloud storage is built on virtualized storage infrastructure with accessible interfaces, designed for near-instant elasticity and scalability, multi-tenancy, consumed as a metered resource. Cloud-based data is stored in logical pools across disparate, commodity storage servers located on premises or in a data center managed by a third-party cloud provider.

Cloud service providers manage and maintain data transferred to the cloud. Storage services are provided on demand in the cloud, with capacity increasing and decreasing as needed. Organizations opting for cloud storage eliminate the need to buy, manage and maintain in-house storage infrastructure. All of this has led to cloud storage radically driving down the per-gigabyte cost of storage, but cloud storage providers have added operating expenses that can make the technology considerably more expensive, depending on how it's used.

As an IT leader, cloud storge is:

• A service model [managed infrastructure]
• Built on virtualized storage infrastructure [raw storage]
• Accessible via interfaces [object, file, block, blob]
• Made available via the network [the internet]
• Increased and decreased as needed [flexible/burst capacity]

As an IT leader there is a lot to like about cloud storage, the ability to lower CAPEX and flex OPEX based on my needs is attractive. Only accessible via the internet is not ideal for some organizations for reasons that range from risk tolerance to legacy workloads but there are bring you our cloud and private cloud options, so that’s good. The different types of interfaces are useful, and the industry has chosen S3 as the de facto cloud storage protocol so that reduces interoperability risk. This is all attractive and gives me flexibility but does require careful consideration against my actual business objectives. Namely, what use cases are best for my business to adopt cloud storage?

Some of the more common use cases for Cloud storage include:

• Data Backup and Disaster Recovery: Cloud storage is widely used for backing up critical data. In case of hardware failure, cyberattacks, or natural disasters, having data backed up in the cloud ensures quick recovery and continuity of business operations.
  
  
• File Sharing and Collaboration: Cloud storage facilitates easy sharing of files and collaborative work. Teams can access and work on documents simultaneously from different locations, enhancing productivity and streamlining workflows.
  
  
• Archiving and Long-Term Storage: For data that needs to be retained over long periods for compliance, regulatory, or historical reasons, cloud storage offers a cost-effective and scalable solution for archiving.
  
  
• Content Distribution: Cloud storage is used for storing and distributing digital content, such as videos, images, and software. It provides a centralized platform for managing content distribution to various channels or end-users efficiently.
  
  
• Data Storage for Training Models: ML and AI models require large datasets for training. Cloud storage provides a scalable solution to store and manage these extensive datasets, which can include images, videos, text, and more.
  
  
• Big Data Analytics: AI and ML algorithms often analyze vast amounts of data to derive insights and patterns. Cloud storage offers the necessary infrastructure to store and process big data efficiently, enabling more effective analytics.

Each of these use cases will have different requirements and this is why every IT leader will have their own path to cloud storage, and not although not every journey will be easy, it certainly does not have to be hard.

So, is there a holy grail of cloud storage? There already exist a growing range of cloud storage types that you can choose from based on the performance, data access, resiliency, and cost requirements of your workloads.

In his recent review of AWS’ newest storage class, Amazon S3 Express One Zone, here is how Jack Vanlightly, Staff Technologist at Confluent defined the holy grail of storage:

 The holy grail of [cloud] storage is simply:

• Cheap
• Durable and consistent
• Low-latency and high throughput

This sounds familiar… it’s the dreaded unattainable triangle. You know the one that says goods and services can never truly be perfect, and so you must choose between quality, speed, and price. Add to that I don’t implicitly see any mention of security being part of the holy grail. Surely any holy grail of cloud storage is secure. Right?

When confronted with either-or choices as IT leaders what we really see is a difficult journey ahead. Am I really to only choose between quality, speed, price AND security choices?

Gartner analyst Dean Lacheca recently shared research that stated 89% of business technologists would bypass cybersecurity guidance to meet a business objective.

I publicly posted the following in response:

Disagree. That number seems...low.

The unscientific data I have suggest it is 100%. Those leaders are not in fact bypassing security, they prioritize outcomes for the business.

Those business objectives are not divorced from cybersecurity objectives. The solutions presented are.

The idea that tech leaders are simply bypassing security is a bit insulting to every tech leader I know. It may just be how that question was worded to the respondents, but I hear similar all the time.

The problem is typically:

1) Either-or choices that are presented as solutions.
2) These choices are based on a false presumption of how the business operates.
3) The presented choices in turn limits the options tech leaders think they have.

I say all of this speaking as a teach leader myself...

Stop offering tech leaders either-or-choices.

The road to logical disjunction is paved with Boolean choices.

Cloud storage has historically presented a lot of either-or choices as solutions. Use Object Storage or use Block storage. Use AWS or use Azure. Choose a tier of cloud storage that is low latency or tier that is cheap.

The first step to finding the holy grail of cloud storage is breaking the Unattainable Triangle!

So how do we do that? I take back what I said about Boolean choices, what we need to do is update our choices to include AND. It is no longer a choice of Business Objectives OR Security Objectives. Our Business Objectives include our Security Objectives. Solutions must offer AND choices and not Either-Or choices.

The holy grail of cloud storage is Cheap AND Durable & consistent AND Low-latency AND high throughput AND meets the requirements of our use case. AND SECURE. What’s the point of any another requirement if your data can be ransomed out from underneath you?

I repeat, teach leaders are not in fact bypassing security, they prioritize outcomes for the business. Presenting either or choices limits the options tech leaders think they have.

Let’s start with cheap. The cost to store data on-premises is more expensive over a 5-year period than it is to store in the cloud. This cost does not extend linearly as data grows.

Estimated Cost to Store 1PB On Premises for 5 Years:

1. Note: Based on estimates of leading providers
2. *Assume 15% annual system maintenance over five years
3. **Assume the space needed to store 1PB is between 25%-50% of a standard rack (42U). Power and cooling costs average $1,200 per square foot per year. Assume 10 square feet or roughly $12,000 per year.
4. ***Assume personnel costs to manage a 1PB system are 0.5 IT/storage admin FTE at an annual salary of $150,000

Estimated Cost to Store 1PB in Public Cloud for 5 years:

Cloud storage offers better economies of scale than on-premises. But again, let’s not setup an either-or choice. The ability to store data in the cloud AND on-premises should be an option for IT leaders. The AND in this scenario can also mean integrating disparate systems across on-premises and various cloud. Ensuring seamless data movement and consistent management across these platforms requires sophisticated integration strategies and tools. Cost management drove an early decision by Myota to implement transparent tools to manage data movement across hybrid platforms. Data locked away on-premises far away from cloud tools (backup tools, data analyzation tools, <insert cloud tool here> setup yet another either-or choice. Either migrate my data to the cloud OR leave it on-prem where Snowflake for example cannot access it.

It is safe to say most cloud storage offerings will be cheaper than storing data on premises. It is also safe for me to say that this does not have to be an either-or choice with the Myota platform.

“The bitterness of poor quality remains long after the sweetness of low price is forgotten.” - 20th century IT philosophers – Benjamin Franklin.

Let’s move on to durable and consistent. In the context of data storage, "durability" and “consistency” usually refers to the long-term reliability and stability of stored data. Durability ensures that once data is written, it remains intact and uncorrupted over time, even in the face of various IT issues such as system failures, power outages, or hardware malfunctions. This is the opportunity to eliminate another either-or choice. Durability is the sum of Confidentiality AND Integrity AND availability (C.I.A Triad). When the business is presented with choices grounded in the principles of the C.I.A Triad, then durability naturally is inclusive of security, it is not either-or.

“Those who would give up essential business requirements, to purchase a little temporary safety, deserve neither." – Benjamin Franklin No IT Leader ever.

No IT Leader is bypassing security over business objectives. There are many IT leaders being presented with either or choices as solutions. Durability divorced from Security is number one the list of cloud storage either-or solutions.

How then do we solve for latency in search of a cloud storage holy grail?

Low-latency and high throughput. To break this link in the chain also requires us to remove either-or from our choices. We know low latency cloud storage exist – AWS S3 express boast a delivery of consistent single-digit millisecond data access. However, S3 express has a price tag of $0.16 per GB. That is significantly more than S3 Standard at a cost of $0.0125 per GB. Here comes the binary choice of either you get low latency, and low durability at a high cost or you get not so low latency, with ok durability at what an ok cost. Here is the thing, AWS alone offers 20+ different cloud storage options across 5 cloud storage products. There are even more offered by Google Cloud Platform, Microsoft Azure, Akamai, Digital Ocean, IBM Cloud, Wasabi…. The list goes on. There is no shortage of cloud storage options.

It requires thoughtful architecting based on your business requirements to decide if the cost of S3 express for example lowers your compute costs, AND subsequently increases the return on your total investment. As Jack Vanlightly points out in his coverage of S3 express, some IT leaders may want to build a replication layer in front of S3 such as they did with Kora and Neon. Great AND options in my opinion. This option can be expensive to build on your own however, impacting the cheap criteria of our holy grail.

At Myota we offer IT leaders choices – not either-or options. As a software defined secure storage layer you can choose your combination of storage options, cloud providers, inclusive of your on-premises environment with one system that manages data movement across these hybrid platforms and is purposely designed to deliver durability by combining the highest Confidentiality + Integrity + Availability controls.

The holy grail of cloud storage is Cheap AND Durable & consistent AND Low-latency AND high throughput AND meets the requirements of our use case. AND SECURE. And the journey requires chooses the right combination of solutions that offer the greatest flexibility – and an end to presenting IT leaders with either-or options.