What’s the difference? Elastic and Splunk data tiers

• Layer A: Data is usually first written here, and we have the best possible performance for indexing and search.

• Layer B: Data is moved here from other layers and it is searchable, although the performance is not as good as the above layer. Data is backed up to an object store and does not require any user action for archival or restore — it is done automatically by the platform.

• Layer C: Data moved to this layer does not incur in compute costs as it is not actively indexed. In order for data to be used (searched upon), it must be restored to a higher layer upon an explicit action from the user — until that happens, the data is virtually “invisible.”

Since Layer A is the most expensive of the three, it makes sense to keep data in there only for as long as it is needed. An example might be keeping the data for one week in Layer A then moving it to B, where it will remain for six months, then to Layer C where it gets stored for, say, three years, for compliance purposes. 

In our opinion, observability and security solutions must give users the ability to choose how they want to move their data up and down the layers, as much as possible, as it is ultimately their decision when it comes to how to balance performance and costs, considering several factors such as budget and business requirements.

Knowing the data tiers in both Splunk and Elastic helps in devising an appropriate data migration strategy. This includes determining which data should be migrated, how it should be transformed or restructured, and how to ensure data integrity and consistency during the migration process. Understanding the source and destination data tiers enables you to map data from one system to the other effectively.