Having come from part of the country that were convinced that tacos were meant to come in a kit and be smothered in sour cream, she still struggled to contain herself when presented with the real thing. "If", she thought in a way that was sure to annoy the reader if it continued for too much longer, "I continue to eat tacos for lunch every day, I'm going start my own Great Oxygenation event". She'd been reading a great blog the other day that had explained the origins of life through an unfortunate series of bubbles and one was never quite too old for a bit of ptty humour.
Settling back in at her desk a bit more delicately than she'd gotten up, she opened her laptop and saw the indicator on the workspace she assigned to aerc indicated she had a new message. For what could be considered the first time an engineer has honestly and earnestly followed this procedure, she thought "I hope it's from my boss". This was the afternoon of her second day on the job and she'd still not seen another living soul.
Nita was a platform engineer who had been hired by Infinity Co, A Division of Zero Industries to help build their nascent developer platform. She had been through a grueling series of interviews, each intended to ensure that she understood that the Agile Coach on the screen of her first interview was practicing the modern version of the craft. She'd been through four interviews with four different sets of challenges embodied in the faces on the other side of her screen.
Una Fronte [Delivery Manager] :: Hypothetically, we have a new service going into production soon. It's going to be Infinity Co's big money maker and so we need it to be stable, in this scenario that is. Hypothetically, we've already seen a number of issues with it that we've not been able to track down and so in order to assess your ability to diagnose deployment pipeline issues, I'd like to ask you how you would start to trouble shoot this type of problem.
Nita [Interviewee] :: What types of issues have you encountered? What are the failure modes that have been exhibited and what are the symptoms of each? Are we talking about pipeline failures? Wrong code being pushed out? Downtime during deployments?
Una Fronte [Mildly unprepared Delivery Manager] :: Uh... mostly it breaks after deployment but all of our testing during the deployment reports healthy. We think it must be something in the runtime environment.
Disti Chous [SRE Manager] :: So hypothetically, how would you go about diagnosing an issue with a service that appears live in a load balancer but refuses to respond to live requests?
Nita [Starting to sense a pattern] :: What endpoint are you using for the liveness test? What are the characteristics of it's healthy and unhealthy state transitions?
Trinity Tertian [QA Manager] :: So, I'm not going to pretend this is a hypothetical because to be honest, I've just walked out of a directors meeting and we all got a good bollocksing. We've got this big product launch coming up and we've got this wierd bug in a core service that no one can figure out. My team have spent weeks writing up a full regression suite and Una's team spent another week scripting that in as the canary in their delivery cutover flow and we still go down hard in production everytime a new build is released. We just can't seem to figure out why the tests are all passing when the service clearly is not.
Nita [The meaning of her own name guiding her tongue] :: Does your regression test customer facing or internal endpoints? Does it use real or synthetic data and does the application contain any logic to route test data differently, such as to avoid writing to production data stores?
Bob [Possibly AI] :: So, how much development have you done? We have a big development first culture here at Infinity Co, everyone gets in on the act. For instance I developed the Infinity Starter Kit which is being used by every team here when they want to bootstrap a new project. It saves hours of time writing boilerplate and has increased developer productivity asymptotically.
Nita [Needing a job, needing to answer] : ...
Back in the present, Nita pressed Meta+1 to activate the workspace where she kept aerc her current email client of choice. She wasn't a huge fan of email, the format had gone the way of most things past their prime and had been reduced to a thing of noise. "Alhtough, there were a few things it was still good for", she reminded herself, updates from ticketing systems being one of them.
** KRM-1 : Deployment Pipeline Needed **
Submitted By: probably.ai@infinity
Assigned To: nita@ininity
Ticket #: KRM-1
Dear Nina,
You may be wondering where I am. I and the boys are still at the corporate kick-off down at Dave's Dude Ranch. You wouldn't believe the size of the hiefers they've got down here.
Anyways, if you need anything don't hesitate to reach out.
To infinity and no further,
Dave
After nearly two full days in her cubicle, this was the first contact she'd had from anyone in her reporting chain and Nita wasn't sure she felt any better. But at least she knew that someone was thinking about her, and "since he didn't mention the big production release, he must think I'm handling the ticket well" her thoughts chimed in to help settle her post-lunch stomach. But then after a moments thought, she realized she wasn't sure how he could given she hadn't yet worked out to file an update. Her brief hunt around for anything like a ticketing system hadn't even lived up to her limited success at finding a wiki and that itself was just an instance of Django unmaintained.
Instead she turned to her trusty terminal and typed glow ~/working/notes/todo.md
build and image stages that her old build system separated"Well then", the motto escaping her lips once again, "let's get on with it."
She opened up another terminal, hyprland kind enough to place it gently beside the first. The only saving grace of a 16:10 screen was its ability display two h-split terminals and show a decent number of columsn in each.
The final piece of her build was the Pipeline class that held everything together. She knew from experience that if she'd been true to her design, this piece should be nothing more than a overly chatty for loop.
module Kremis
class Pipeline
attr_reader :name, :stages
<snipped>
def run(context = Context.new, output_dir: nil)
run_id = SecureRandom.uuid
trace = nil
<snipped>
puts "Pipeline: #{@name} (#{run_id})"
puts "=" * 40
@stages do |stage, i|
puts "\nStage #{i + 1}/#{@stages}: #{stage.name}"
puts "-" * 30
before = context.to_h
stage.execute(context)
if trace
trace[:stages] << {
name: stage.name,
index: i + 1,
context: context.to_h,
added: context.to_h.reject { |k, v| before[k] == v }
}
end
end
if trace
trace[:finished_at] = Time.now.iso8601
path = File.join(output_dir, "#{run_id}.json")
File.write(path, JSON.pretty_generate(trace))
puts "\n -> #{path}"
end
puts "\n#{"=" * 40}"
puts "Pipeline complete."
puts "\nAccumulated context:"
context.data.each do |k, v|
puts " #{k}: #{v}"
end
context
end
end
end
"I really need to do something about those puts statements", she spoke aloud, her left hand reaching for the small rubber ducky she'd grabbed from the bodega near the taco shop at lunch. Nearly two full days in a completely quiet open plan office was driving her a bit mad. She hadn't thought open plan offices could get worse, and the quiet was starting make her go philosophical. "I bet you this is how the tree feels", she thought as silence was required for the bit. "I bet she falls over just to make noise and liven the place up", she exclaimed to the duck completing the joke.
"puts requires redirection at runtime for capture, which is fine for one offs" she remembered her old boss saying, "but when we're logging in production, we should use structured logging so that we can control the output globally and ensure it is consistent throughout". He loved to inject a Logger module into each of his classes that contained leveled emitters like info and trace that wrapped their payloads as a block that would only execute if the required level was set. "That way", her memory of him continued, "you can embed complex logic in your log statements if necessary and know that it will only trigger if the log output has actually been requested". A trick that Nita had used on occasion as often the only way to diagnose a distributed system like her boss' old platform was good old fashioned log dumps.
With her Pipeline class open in one terminal, she fired up another instance of neovim in the other with nvim ~/working/kremis/lib/kremis/logging.rb and replicated a module that she'd seen a thousand times.
module Kremis
# Centralized logging with block-based evaluation.
# Include this module to get logging methods that only eval when enabled.
#
# Usage:
# include Kremis::Logging
#
# debug { "Expensive #{computation} only runs if debug enabled" }
# info { "Processing #{item}" }
# warn { "Something looks off: #{details}" }
# error { "Failed: #{exception.message}" }
#
module Logging
def self.included(base)
base.extend(ClassMethods)
end
module ClassMethods
def logger
Kremis.logger
end
end
def logger
Kremis.logger
end
def debug(&block)
logger.debug(log_prefix) { block.call } if logger.debug?
end
def info(&block)
logger.info(log_prefix) { block.call } if logger.info?
end
def warn(&block)
logger.warn(log_prefix) { block.call } if logger.warn?
end
def error(&block)
logger.error(log_prefix) { block.call } if logger.error?
end
private
def log_prefix
self.class.name || "Kremis"
end
end
<snipped>
end
Code to her was strongly tied to memory. Often as she navigated a code base, the shape of a particular solution uncovered would trigger a memory like a waft of perfume recalls a stolen kiss. She blushed, thinking about a particularly naughty inversion she'd used once while trying to milk the last drops of functionality out of a kludge. "Sometimes", she knew, "the best ideas come at night after you two have been at it for a while. You're both a bit tipsy from all the raw exploration, and it's time to just get on with it and finish." She reached down to squeeze the duck once more. Her metaphors always seemed to give something more away than she intended, but she could never quite put her finger on it. She thought having been raised Catholic might have something to do with that.
This pattern was tied to warm memories of a team she wished she still had around her. Standing up to stretch, she thought "at this point, I'd take any team".
Returning to her refactored Pipeline she thought she could make one more improvement.
def run(context = Context.new, output_dir: nil)
run_id = SecureRandom.uuid
trace = nil
if output_dir
FileUtils.mkdir_p(output_dir)
trace = {
run_id: run_id,
pipeline: @name,
started_at: Time.now.iso8601,
stages: []
}
end
info { "Pipeline: #{@name} (#{run_id})" }
@stages do |stage, i|
info { "Stage #{i + 1}/#{@stages}: #{stage.name}" }
before = context.to_h
stage.execute(context)
if trace
trace[:stages] << {
name: stage.name,
index: i + 1,
context: context.to_h,
added: context.to_h.reject { |k, v| before[k] == v }
}
end
end
if trace
trace[:finished_at] = Time.now.iso8601
path = File.join(output_dir, "#{run_id}.json")
File.write(path, JSON.pretty_generate(trace))
info { "Trace written: #{path}" }
end
info { "Pipeline complete" }
debug { "Accumulated context: #{context.to_h}" }
context
end
This was better, but she knew that trace logic belonged in its own module. That was the kind of thing that started as a quick debugging tool and quickly grew to become structural. She knew she would eventually need an observability solution, but she always hated to reach for anything larger than she needed for the task at hand. For something like this, she knew tha the tracing points weren't likely to change and worst case, if she ever needed to swap out her homegrown solution for something like OT, it was a simple refactor with Netflix on in the backround. With Mutex, her Best Fury Friend on the couch beside her, she'd passed many a lazy evening rhythmically refactoring simple optimizations. Register based editors like vi were great for that, though she prefered emacs + doom with its evil bindings. vim couldn't cut it on its own any more and neovim just seemed a bit too thin and wobbly for her taste as a daily driver, though it was still her tool of choice for quick changes. Her nvim bindings as near to doom as time permitted. "With registers", she explained to the duck in her hand, "you could load up a bunch of snippets and let muscle memory take over". The platform she had worked on at her previous job had been very strongly pattern based, so a refactor generally meant loading up whatever new pattern was required and soon The Storybots would be learning of knights from a chin and computing from a Dogg of all things, and the work would take care of itself.
She decided on a simple span based tracing engine, with the ability to nest traces. This would come in handy as she built out the pipeline as she could trace the end-to-end flow within a single causal structure.
The Tracer itself was fairly standard:
tracer would be injected into every class#trace method provides manual or block blased span controlmodule Kremis
<snipped>
# Span: id, name, parent, children, records, timestamp
# The tracer itself — one per process, holds the span stack.
class Tracer
attr_reader :completed
def initialize
@stack = []
@completed = []
end
# Open a span. If a context is provided, the tracer snapshots it
# before and after the block, recording both the full context and
# the delta automatically.
#
# If no block, caller must call complete_span manually.
def trace(name, meta: {}, context: nil)
before = context&.to_h
span = Span.new(name: name, parent: current_span, meta: meta)
@stack(span)
if block_given?
begin
yield span
ensure
snapshot_context(span, before, context) if context
complete_span
end
end
span
end
# Record a key/value pair on the current open span.
def record(key, value)
current_span&.record(key, value)
end
# Complete the current span and pop it off the stack.
def complete_span
span = @stack
return unless span
span.complete!
if @stack?
@completed << span
end
span
end
# The currently open span (top of stack).
def current_span
@stack
end
# Is there an active trace?
def active?
!@stack?
end
# Full address of the current position in the trace tree.
# e.g. "pipeline:infinity-service/stage:build/actor:docker"
def current_address
@stack(&:name).join("/")
end
private
def snapshot_context(span, before, context)
after = context.to_h
added = after.reject { |k, v| before[k] == v }
span.record(:context, after)
span.record(:added, added) unless added.empty?
end
<snipped>
end
She opened up her Pipeline class again and cleaned up the main #run method. The new trace method allowed her to just wrap her main loop in a span and it would carry through each of the stages.
def run(context = Context.new, output_dir: nil)
trace("pipeline:#{@name}") do
info { "Pipeline: #{@name} (#{tracer.current_span.id})" }
@stages do |stage, i|
info { "Stage #{i + 1}/#{@stages}: #{stage.name}" }
stage.execute(context)
end
info { "Pipeline complete" }
debug { "Accumulated context: #{context.to_h}" }
end
if output_dir
path = tracer.write(output_dir)
info { "Trace written: #{path}" } if path
end
context
end
She'd been able to give her Stage class an upgrade in both aesthetics and function. The trace wrapping the actor's own #execute extends the one opened in the main loop and its context automatically snapshots deltas between stages for reporting.
module Kremis
class Stage
include Logging
include Tracing
attr_reader :name, :actor
def initialize(name:, actor:)
@name = name
@actor = actor
end
def execute(context)
trace("stage:#{@name}", context: context) do
info { "executing #{@actor}..." }
@actor(context)
end
context
end
end
end
Somehow the word "Yummy" had escaped her lips while reading over code again. She hoped her new duck friend hadn't heard. She had had enough relationships ruined by her love of a strict causal chain, and she and the duck had been getting on so well.
Finally, she opened her terminal to one last file. She had been using irb mainly to test her new tools, she liked the idea of being able to execute pipeline stages like functions inside an interactive environment and had tested all of these components individually as she'd worked on them. Now it was time to script them together and see if they'd all play as a team.
pipeline = Kremis::Pipeline.new(name: "infinity-service")
pipeline.add_stage(Kremis::Stage.new(name: "build", actor: Actors::Build.new))
pipeline.add_stage(Kremis::Stage.new(name: "deploy", actor: Actors::Deploy.new))
pipeline.add_stage(Kremis::Stage.new(name: "validate", actor: Actors::Validate.new))
context = Kremis::Context.new(
app: "infinity-service",
commit: "f4a7c2e"
)
pipeline.run(context, output_dir: File.join(__dir__, "output", "ticket-001"))
For now, this would be enough allow Alice to complete her deployment this week and to allow Nita to hopefully close her first ticket. She still wasn't entire sure what the point of the service was, but that was a decision without a distinction for Nita. Her job was infrastructure, a bottom line feeder in any organization. Revenue was the problem of another department.
Happy with her configuration, and after double checking her initial commit, she closed the file. This was another area she knew she was going to have to come back to - the commit hardcoded in a file was the right thing to do for repeatability she knew, but the proper capture point was in the context file that was produced by the pipeline with all of the rest of the data that was learned along the way. The input was rightly ephemeral, if guaranteed immutable and was best served by the environment since she was unaware of any CI platform that didn't make such things availble thusly.
Excited to see how many typos she'd left lying around, she kicked off her first official pipeline at Infinity Co, a division of Zero Industries
Nita had a sudden need to be in her bunk.
"Oh, if I leave now I can catch the 4:54 and make it home in time to get to that cute little grocers before it closes", she decided as she watched her pipeline complete successfully. She would normally send the requester of the ticket she was working on an update at this point, to let them know there was a pipeline available to be tested and that she'd be happy to sit with them and go through the first few runs together. However she had never met Alice and wasn't sure where she sat in the building. She also had no means of contacting her through the ticketing system, as it apparently was off at the Dude ranch riding hiefers with the rest of her team.
She threw the duck in her bag with her laptop and headed off to catch the early bus.
DING
You've got mail
-- Nita's phone, as her love of romantic comedy escaped her ability to laugh at her inability to find love in her own life
** KRM-1 : Deployment Pipeline Needed **
Submitted By: probably.ai@infinity
Assigned To: nita@ininity
Ticket #: KRM-1
Nita,
I am suddenly in need of the pipeline you have been working on for our shared ticket.
I trust it has been completed satisfactorily.
Please push your changes to `main` before leaving for the day.
Best Regards,
Alice
Nita spun around quickly, sure that someone must be pranking her. "How did she possibly know? I just ran it for the first time a moment ago", she thought running through the sequence in her mind. Nothing on her laptop should have pinged the outside world, at least nothing that was executed in the script that she had just triggered. "Maybe it was all just a big coincidence", she told the duck having freed it from the confines of her bag having decided that the afternoon had made them better than bag friends.
- [x] A build actor that can wrap Docker and execute multi-stage Dockerfiles to combine the `build` and `image` stages that her old build system separated
- [x] A deploy actor that could automatically update and deploy a Docker compose service definition with an updated image
- [x] A simple verification actor that can perform a post-deployment validation
- [x] A simple engine to chain them together and manage shared context