Big Tech Data Platforms = Big Bills

Ask anyone that’s modernized their data platform to Big Tech platforms like Databricks and Snowflake, what they actually to spend ends up being several times what they expected. This isn’t an accident: Big Tech Platforms make their profits by quoting the lowest possible theoretical cost in their sales pitch, while convincing you to use options that spike your usage to many times that amount. We’ll provide the decades of experience needed to implement performant, low-cost solutions on your existing platform for a fraction of the cost.

Databricks and Snowflake

Built around the Spark framework, these ‘modernized’ platforms are built around a very old idea of what ‘Big Data’ means. A decade ago, Spark was the primary way to process hundreds of gigabytes in a performant way. Today however, tools like DuckDB and even modern Relational Database Systems like PostgreSQL provide big data performance at a fraction of the cost. We’ll review your existing pipelines, and identify key areas where more cost effective tooling can be used for processing, all while you keep your preferred interface in your existing data platform tool.

Still need Spark? We can create cost savings here as well, with one company seeing their bill go from $100k per month to $100k per year just from optimizing how their Spark pipelines run.

Astronomer

A powerful orchestration engine, Astronomer Airflow has in recent years changed its pitch from industry-standard pipeline scheduling platform to all-in-one behemoth. If you’re using Astronomer Airflow for running your pipelines end-to-end instead of simply scheduling them, your bill is likely several times what it needs to be. By using Airflow for what it was designed to do, and delegating pipeline execution to various ephemeral options, we can help you slash your spend to a fraction of what it is currently. One company saw their Astronomer bill go from $400 dollars a day to $40 a day with no decrease in the number of pipelines running.