Sprout

a-deadline-and-a-dag

A Return Engagement

Nita got off the #2 bus just outside Infinity Co's mid-level office tower. It was located in a part of town that catered to the kind of company that Nita had found herself working for for much of her career. Non-descript building, typically three or four stories. Reminded her a lot of the public schools she'd attended as a child, themselves the product of a generation that believed windows a tool of the devil.

During her interview process, Nita's hiring manager had spend considerable time describing all of the perks offered to Infinity Co employees, including what was sold as "the world's first fully AI powered barista". Nita wasn't sure a laptop running a chatbot asking "What would you like to drink today?" next to a rather crusty looking Flavia machine counted but she was glad for second chance for caffeine. A rather unfortunately placed pot-hole near her apartment meant that her first attempt was showing as a small brown stain on her cream coloured skirt.

Sitting down at her desk, she settled in for what appeared to be another quiet day. It was 9:05 am and once again she was the sole proprietor of their platform enterprise, none of her coworkers nor her manager were in sight. "What the heck is going on around here?" she wondered as she sat down at her desk. Opening her laptop lid, she opened Ghostty and entered glow ~/working/notes/todo.md

She reviewed her notes - Kremis was the name she'd given to her new simple DAG engine, just enough structure to coordinate, not enough to complicate. The Pipeline class contained a #run method that just looped through the defined stages and called the #execute method on each. The Stage class was just a thin wrapper around an Actor to give it semantic meaning and delegate the #execute call. The Actors themselves were just thin wrappers around #system calls, something she'd have to take care of someday she thought but that was a platform problem and she was still writing pipelines.

module Actors
  class Build < Kremis::Actor
    def execute(context)
      app_dir = File.expand_path("../../../app", __dir__)
      tag = "infinity-service:#{context[:commit]}"

      puts "    docker build -t #{tag} #{app_dir}"
      result = system("docker build -t #{tag} #{app_dir}")
      raise "Docker build failed" unless result

      context.merge(image_tag: tag)
    end
  end
end

A Beginning, and Then Another One

Opening another tab in her terminal, she started another nvim session and created actors/deploy.rb and filled in the basic class definition.

module Actors
  class Deploy < Kremis::Actor
    def execute(context)
    end
  end
end

Yesterday, when reviewing the ticket for the first time she had thought through her list of deployment targets for the infinity-service application she was deploying. Her build stage was packaging it as an OCI container, so she was able to narrow the search space to a handful of possibilities. She decided to stand up one last time in a vain attempt to locate a coworker, a manager, or even a friendly looking facilities person. She was getting a bit desparate to find someone who might be able to tell her where everyone was. She thought "I really wish I could at least find a best practices document or even any documentation on how the applications that are already running were deployed", but alas none was to be found. She had tried the corporate wiki at https://wiki.infinity.co but it just rendered the boilerplate Django homepage. "Odd choice for a wiki" she knew, but she was starting to get the feeling that things were done a bit differently around here.

She ran down the list of potential deployment targets:

"K8S is definitely out, and I don't even know which cloud provider Infinity Co uses, so I guess I can't use ECS right now either", she decided while thinking through her options. That left Nomad and Docker Compose. In the interest of Keeping it Simple Silly, she decided she'd start with compose. Her experience at home lifting and shifting self-hosting environments into her own stack had given her confidence enough to know she could always make a minimal pivot later to a more complex orchestrator.

services:
  infinity-service:
    image: infinity-service:latest
    ports:
      - "9999:9999"
    restart: unless-stopped

A few moments later, she had a minimal version sketched out. Thinking through her pipeline, she knew that the Build actor had graciously added the new image tag to the context, so she decided that a simple templating system would be the easiest path forward. "That way", she thought, "we can archive the output files and age them off with the images they deployed, then we have a more deterministic rollback strategy than relying on environment variables from the build context". She knew this might be overthinking a bit at this point given the complete lack of configurability present in the service, but she also knew from experience how quickly the complexity of the runtime environment grew once developers had a simple button between their whims and production.

services:
  infinity-service:
    image: <%= image_tag %>
    ports:
      - "9999:9999"
    restart: unless-stopped

She was using Ruby for her engine and ERB was the natural choice for a templating language. "Good enough for Rails, good enough for me", she decided.

For her Deploy actor, the flow was simple. Load the templated compose file, load the image_tag from the context into the scope so it would be available in the binding passed to the ERB #result call and write the output file to a known path. From there, it was just a #system call to docker compose up -d. "Then", she murmured to herself, the quiet of the office starting to require noise, "I'll capture the path to the file and whatever service information I can capture to the context. For sure I'll need to capture the URL for the Validate state". Once again her thoughts not only able to express themselves clearly, but also to pre-format themselves as necessary to ensure a consistency of form.

module Actors
  class Deploy < Kremis::Actor
    def execute(context)
      template_path = File.expand_path("../templates/docker-compose.yml.erb", __dir__)
      output_path = File.expand_path("../output/docker-compose.yml", __dir__)

      image_tag = context[:image_tag]
      template = File.read(template_path)
      composed = ERB.new(template).result(binding)

      FileUtils.mkdir_p(File.dirname(output_path))
      File.write(output_path, composed)
      puts "    wrote #{output_path}"

      puts "    docker compose -f #{output_path} up -d"
      result = system("docker compose -f #{output_path} up -d", out: $stdout, err: $stderr)
      raise "Docker compose up failed" unless result

      context.merge(
        compose_file: output_path,
        service_url: "http://localhost:9999"
      )
    end
  end
end

Quickly reviewing both the ticket and the source repository it had directed her to, she was not terribly surprised to find no mention of any testing to be performed. "Nor", she supposed a bit louder this time, "did she see anything about a health check".

Testing the Waters

She decided she'd have to open up the code and dig out what details she could. She should remember to ask her manager if they had standard endpoints that application teams were expected to comply with. She remembered her boss talking about the early days of his platform and how they had decided upon a triptych of required endpoints:

She knew she was nearing the end of the work on her ticket and she still had no feedback. This was unusual for her coming from a place where her old boss took an hour each morning to offer open hours for a chat, for a problem, or to listen to him teach distributed systems through metaphor. He was a bit odd, but she missed him.

She decided to give her new boss another chance. Having thought she'd heard a noise earlier, she decided to wander down the row of cubicles to her manager's endcap hoping to finally catch a glimpse of another soul. Instead, she was greeted with her manager's monitor glaring up at her displaying "his login prompt?"

Not wanting to face the wrath of corporate security, and unsure if she'd ever even heard mention of an Employee Handbook, she returned to her desk defeated and opened up Alice's service controller.

package com.infinity;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
/////////////////////////////////////////////////////////////////////////////// 
//  
//  ___        __ _       _ _            ____  _             _               //
// |_ _|_ __  / _(_)_ __ (_) |_ _   _  / ___|| |_ __ _ _ __| |_ ___ _ __   //
//  | || '_ \| |_| | '_ \| | __| | | | \___ \| __/ _` | '__| __/ _ \ '__|  //
//  | || | | |  _| | | | | | |_| |_| |  ___) | || (_| | |  | ||  __/ |     //
// |___|_| |_|_| |_|_| |_|_|\__|\__, | |____/ \__\__,_|_|   \__\___|_|     //
//                               |___/                                        //
///////////////////////////////////////////////////////////////////////////////
//
//  Generated by Infinity Starter Kit v0.0.1-SNAPSHOT
//  https://start.infinity.co
//
//  Project:     infinite-money
//  Description: A division of Zero Industries Ltd.
//  Java:        8
//  Framework:   Spring Boot 3.2.3
//
//  Generated:   2025-03-10T09:15:00Z
//  Build:       probably.ai@infinity
//
//  DO NOT MODIFY THIS FILE — changes will be overwritten on next generation
//  (we know you will anyway, we just want you to feel bad about it)
//
///////////////////////////////////////////////////////////////////////////////

@RestController
public class DivisionController {

    @GetMapping("/")
    public String index() {
        return "Infinity Service — a division of Zero Industries Ltd.";
    }

    @GetMapping("/divide")
    public String divide(
            @RequestParam(defaultValue = "1") double a,
            @RequestParam(defaultValue = "0") double b) {
        double result = a / b;
        if (Double.isInfinite(result)) {
            return "Infinity. You divided by zero. Welcome to the company.";
        }
        return String.valueOf(result);
    }

    @GetMapping("/health")
    public String health() {
        return "OK";
    }
}

Thinking of Mutex her cat, and her own desire to eat occasionally, she pushed the sudden desire to run that had overtaken her deep down inside, where it would hopefully stay for at least a few more posts.

"Well at least there's one in there somewhere", she decided in a fully conversational tone, in an attempt to draw the attention of anything living in the building towards her location.

She looked at the clock on her phone, 10:45 am. She decided to knock out her new Validate actor and then head out for lunch.

In her mind, post-deployment validation should be divided into two phases.

First, check to see if it's even worth spinning up the full test infrastructure. This is where a well designed /status endpoint should be enough. "If a service has managed to at least get through to initialization and presents a reasonably stable surface, it should be clear to go but...", her thoughts drifted to the naked OK above. She'd had the idea of "fail fast, fail predictably, and most of all - fail in a manner that can be restarted if possible" drilled into her by her old boss.

Only once infrastructure and basic human error are ruled out as best as possible from the litany of potential errors that follows, should you proceed to a full regression suite. Here Nita had her own opinion on the matter. She had argued for and won the opportunity to evangelize the idea of trimming down the regression tests that ran during integration builds and instead running the full suite off hours where possible. The regression suite for the core application at her old gig could run for hours, especially if the search team were doing work on their development cluster. This meant restrictions on how frequently builds could be performed during the day, which Nita knew defeated a lot of the purpose of continuous integration. Her successful lobbying had meant that, while developers couldn't get a build for every commit they made - there was still a 45 minute cycle to get through - they could at least merge a commit and expect it to be deployed after a sprint planning meeting concluded.

In this case, all she had was a simple health check, and the only infrastructure that her current model required was a server somewhere able to run a compose stack. She decided on a simple curl loop, implemented with Net::HTTP#get with a break on OK and a small nap on failure.

module Actors
  class Validate < Kremis::Actor
    def execute(context)
      url = "#{context[:service_url]}/health"
      puts "    GET #{url}"

      retries = 0
      loop do
        begin
          uri = URI(url)
          response = Net::HTTP.get_response(uri)
          if response.code == "200"
            puts "    OK (#{response.code})"
            return context.merge(healthy: true, health_status: response.code.to_i)
          else
            raise "unexpected status: #{response.code}"
          end
        rescue StandardError => e
          retries += 1
          if retries > 10
            raise "Validation failed after #{retries} attempts: #{e.message}"
          end
          puts "    waiting for service... (#{e.message})"
          sleep 2
        end
      end
    end
  end
end

Thinking back to the pilfered poultry Mutex had run off with last night as her tummy growled for the third time, she decided to finish the rest of her pipeline after lunch.

A quick edit to todo.md left on screen was a reminder of what her afternoon would bring.