Apache Mesos

One cluster. Any workload. Total resource efficiency.

Most organisations run separate infrastructure for different workloads - one cluster for containers, another for batch jobs, another for data pipelines. Apache Mesos collapses all of this into a single, unified resource pool. Node deploys Mesos as the cluster operating system that runs every workload - containerised services, Spark jobs, Kafka brokers and Airflow workers - on shared infrastructure with fine-grained resource isolation.

What Mesos does and why it matters

Apache Mesos was built at the University of California, Berkeley and later adopted at Twitter, Apple and Airbnb to solve a fundamental problem: how do you efficiently run diverse workloads on a large shared cluster without each team managing their own dedicated servers?

Mesos operates as a two-level scheduling system. The Mesos master manages the cluster, tracking available resources across every node and offering resource slots to application frameworks. The frameworks - whether Marathon for long-running services, Chronos for scheduled jobs, or direct Mesos frameworks for Spark and Kafka - accept those offers and schedule their tasks accordingly. This separation means Mesos can run entirely different types of workloads simultaneously without them interfering with each other.

The result is dramatically higher resource utilisation. Organisations that move from dedicated clusters to Mesos typically see server utilisation jump from 15-25% to 75-85%. Your hardware works harder, your infrastructure bill falls and your operational complexity decreases because you're managing one cluster instead of many.

How we deploy Mesos for business automation

We use Mesos as the infrastructure foundation that makes the rest of your automation stack more efficient. Apache Kafka brokers, Apache Spark executors, Apache Airflow workers and Apache NiFi nodes all compete for the same pool of resources under Mesos scheduling. When a Spark job finishes, those CPUs and memory are immediately available to Airflow workers that need to run the next pipeline stage. Nothing sits idle waiting for its dedicated slice of hardware.

For organisations running mixed workloads - some latency-sensitive services, some batch processing, some streaming jobs - Mesos provides the resource isolation guarantees that keep them from interfering with each other. A runaway batch job cannot starve a production API service of CPU. Resource quotas and priorities are enforced at the kernel level.

Key capabilities we implement

Fine-grained resource allocation - allocate CPU, memory, disk and network bandwidth to individual tasks rather than entire machines. Mesos tracks resources at the task level, enabling bin-packing algorithms that place workloads efficiently across your cluster and maximise utilisation.

Fault tolerance and high availability - Mesos masters run in a quorum using ZooKeeper for leader election. If the active master fails, a standby takes over in seconds. Agent failures trigger automatic task rescheduling to healthy nodes, ensuring your workloads continue running without manual intervention.

Container orchestration - run Docker containers and OCI-compliant images directly through Mesos with full support for volume mounts, network configuration and container health checks. Mesos predates Kubernetes and provides a lower-level, more flexible container execution model suited to mixed-workload environments.

Multi-framework scheduling - run Spark, Kafka, Cassandra, Elasticsearch and your own custom frameworks on the same cluster simultaneously. Each framework operates independently within its resource allocation, enabling you to adopt new technologies without provisioning new infrastructure.

Role-based resource quotas - assign guaranteed resource quotas to different teams, departments or workload types. Operations gets their guaranteed slice, data science gets theirs, and any spare capacity is distributed fairly according to configured weights.

Unified monitoring and observability - Mesos exposes detailed resource utilisation metrics through a REST API and built-in UI. We integrate these with Prometheus and Grafana to give you cluster-wide visibility into utilisation, task states and scheduling latency.

Use case: financial services batch and real-time processing

A mid-sized financial services firm was running separate dedicated clusters for their overnight batch reconciliation jobs (Spark), their real-time transaction event stream (Kafka), and their regulatory reporting pipelines (Airflow). Each cluster was provisioned for peak load, meaning all three sat at under 20% utilisation for most of the day.

Node migrated all three workloads onto a single Mesos cluster. Kafka brokers run continuously as persistent Mesos tasks with reserved resources. During off-peak hours, the Spark reconciliation jobs consume the surplus capacity that Kafka isn't using. Airflow workers spin up on demand when pipelines trigger and release their resources when complete.

The outcome: server count reduced by 60%, infrastructure cost fell by 55%, and the firm gained a single operations team managing one platform instead of three separate systems with three separate runbooks.