Is Your Team Busy or Productive?

I’m at my in-laws’ place in <redacted> North America, it’s 2am because my body still thinks it’s tomorrow afternoon in Sydney, my ankle has been recovering from a trimalleolar fracture/break and full dislocation for a month now, I’m laying on the floor watching my baby sleep next to me and I’m doing what any reasonable person does when jet-lagged taking care of their baby and unable to sleep: syncing files to my NAS on the other side of the planet.

Look, I had a reason. There were photos. The kid did something cute and the grandparents wanted copies and somehow this turned into “I should just pull down those files I’ve been meaning to organise” which turned into me staring at transfer speeds at 2am while everyone else sleeps.

The connection was surprisingly good. I’d expected the usual transpacific misery, with LEAST DOLLAR cost routing instead of least path cost, but somewhere between my in-laws’ surprisingly decent internet and whatever routing magic was happening across the Pacific, I was getting respectable bandwidth that could saturate my shitty NBN link back home; iperf3 even confirmed!

And yet.

The transfer was crawling.

So I did what any normal person would do: I SSH’d into my home lab from my father-in-law’s guest room and started investigating where my bottleneck actually was.

The Problem With Asking “Am I Max’d Out?”

The first thing I checked was network. Was I actually saturating the link? no. Not even close. The bandwidth was there. The route was clean. The packets were flowing. Send that switch some packets, Switches love packets

Then I remembered.

Right before we left for this trip, I’d replaced a drive in the NAS. One of the eight drives had started throwing SMART warnings, the kind that say “I’m not dead yet but I’m thinking about it” and I swapped it out the night before we flew. After an emergency trip to Centrecom. Responsible typical sysadmin behaviour.

Future Will would thank me. Present Will is LIVID.

The SHR2 array was still rebuilding.

For those fortunate enough to have never experienced this particular joy: SHR2 is Synology’s version of RAID 6. Two drives can fail before you lose data. Very safe. Very sensible - it's a home lab, nothing is sensible. But when you replace a drive, the array has to reticulate splines recalculate parity across all remaining drives and write it to the new one. This involves reading from every other disk and writing to the new one, continuously, for hours or days depending on array size.

Every single file I was trying to sync from North America was competing for disk IO with a rebuild operation that was touching all eight drives simultaneously.

It wasn’t slow because of the Pacific Ocean. It was slow because Past Will had done something responsible and Present Will was paying the price. Stupid Will.

IO Wait - AKA - Looking Busy While Accomplishing Nothing

There’s a concept in computing that’s been stuck in my head since that night. It’s called IO Wait, and it’s when your CPU is technically active and it’s managing requests, maintaining state, checking if data has arrived, but not doing any actual computation.

The utilisation graphs look healthy. The processor looks engaged. But productive throughput? Basically zero. The CPU is just... waiting. Waiting for data to arrive from somewhere slower than itself.

My NAS wasn’t overloaded in any traditional sense. The CPU wasn’t pegged. The RAM was fine. But the disk subsystem was absolutely hammered with eight drives all churning through a parity rebuild while simultaneously trying to serve my file requests and everything was spending more time waiting than working.

I’ve been thinking about that ever since. Specifically, I’ve been thinking about how many teams I’ve worked with that have the same problem.

Everyone looks busy. The calendars are full. Slack is buzzing. People are in meetings, on calls, typing furiously. The utilisation looks great.

But the actual work? It’s queued behind a parity rebuild that nobody remembered was happening.

The Cache Hierarchy

Here’s the thing about my file sync problem the raw speed was there. On paper, this should have been quick.

But “on paper” assumes the data is ready when you ask for it.

Modern CPUs have known about this problem for decades. The processor itself is absurdly fast Billions of operations per second.

AND that speed means nothing if there’s no data to operate on. So chip designers have spent enormous effort building cache hierarchies: L1, L2, L3 each layer a bit further from the CPU core, each a bit slower, each a bit larger.

The whole elaborate architecture exists for one reason: keep data as close as possible to where decisions get made.

I looked up the actual latency numbers for a modern CPU because apparently this is what I do instead of addressing my jet lag and the hierarchy is staggering:

Read that last row again. If reaching into L1 cache is one second of human time, then fetching from an NVMe drive (a fast one), PCIe attached, no spinning rust in sight is waiting nearly a week. A spinning hard drive? You’re waiting until next Spring/or Autumn.

And I had eight spinning hard drives, all of them busy doing something else.

This is why “each Graviton5 core has access to 2.6x more L3 cache than Graviton4, which translates to fewer delays waiting for data and faster application response times.” AWS didn’t just make the cores faster. They moved more data closer.

Meanwhile, I was trying to move data from North America to Sydney a journey of about 150 milliseconds in network terms but the real bottleneck was the spinning rust fighting over parity calculations.

The Organisational Parity Rebuild

I’m back in Sydney now, jet lag mostly resolved, I’m still here with my broken ankle and some sync’d files watching my baby sleep and I keep mapping this onto every team I’ve ever worked with.

The SHR2 rebuild is the perfect a metaphor, actually. It was important work but that necessary background work consumed so many resources that the foreground work, the actual thing I was trying to do, slowed to a crawl.

There’s the “happy” version of this:

• The “essential” reorganisation that has everyone updating spreadsheets instead of shipping

• The compliance initiative that’s consumed half of everyone’s week for the last month

• The tooling migration that seemed important but now has three people in meetings every day

• The process improvement that’s somehow made the process take longer

These might all be legitimate and important tasks. But they’re parity rebuilds. They’re consuming IO bandwidth that your actual work needs.

And then there’s the worse version, the sour flavour of organisational IO Wait:

• Waiting for approvals that sit in someone’s inbox for three days

• Blocked on a decision that requires scheduling a meeting with someone who’s in back-to-back meetings until next Thursday

• Needing information that lives in someone’s head, and they’re on holiday

• Writing status updates about why nothing has status updates

• Architectural meetings where 6 different teams give you requirements and you can’t make a decision but not because of indecision, but because there’s no single threaded owner and you’ve organized into role specific teams instead of product specific teams and the most responsible thing to do is not take accountability for anything more serious than ensuring the next meeting happens so you just design a playbook of 10,000 different patterns hoping someone in the future will see something they like and not bother you.

• Platform engineers answering tickets for product, SRE, and SDE(!!) teams because no one really knows how their code works in production because they can’t see it after it goes over the wall.

Everyone is doing something. The individual’s utilisation looks great. But the actual work is queued behind seventeen other requests, waiting for access to a contended resource.

Trust Is Just Pre-Distributed Decision Authority

Let’s go deeper into the metaphor.

You know that decision-making pattern where the SREs, Cloud team, Infrastructure team, Product folks, and SDEs all get in a room a sprawling Slack thread and hash out how to do something? They debate, they whiteboard, they reach consensus. It takes a while but they get there.

Then they take it to the Architecture team, who kick it back with concerns.

So everyone regroups, addresses the concerns, iterates. Gets back to consensus.

Then the CTO has a meeting with their vendor about it and comes back with a completely different direction.

Everyone goes back to their own drawing boards.

I’ve watched this happen so many times. Each handoff is a network round trip. Each escalation is a cache miss that goes all the way to disk. And the latency isn’t just the waiting it’s the context loss. The Architecture team wasn’t in the original conversations. The CTO wasn’t. Each hop loses fidelity, like a game of telephone played through Jira tickets.

Why can’t the person closest to the problem just make the call?

The answer, of course, is trust. Or rather, the absence of it.

Distributed Systems Have Distributed Trust

Think about how distributed computing actually works.

In a properly distributed system, each node has the authority to make local decisions. A replica in ap-southeast-2 doesn’t call home to us-east-1 for permission every time it needs to serve a read. That would be insane. The latency would destroy any benefit of having the replica in the first place.

Instead, you accept some tradeoffs. The ap-southeast-2 replica might serve slightly stale data. It might make a decision that’s locally optimal but not globally optimal. Occasionally, two nodes might make conflicting decisions that need to be reconciled later.

But the system works. It’s fast. It’s resilient. Each node has enough authority (enough trust) to do its job without constantly checking with a central coordinator.

Now think about your org chart.

You’ve got a team of DBAs over here. A team of architects over there. SREs in one silo, cloud specialists in another, product managers somewhere else entirely. When a product team needs to make a decision that touches infrastructure, they don’t have local cache. They have to make a network call to another team. That team has their own queue, their own priorities, their own parity rebuilds happening in the background.

Every decision that crosses a team boundary is a cache miss. And you’ve architected your organisation so that most decisions cross team boundaries.

Embedded Specialists as Local Cache

What if you embedded a DBA in each product team that needs one?

What if instead of a centralised architecture team that reviews everything, you had architects distributed into the teams, with enough trust and authority to make decisions locally?

What if the people closest to the problem had the context and the authority to just... solve it?

This is the organisational equivalent of putting an L1 cache on every core instead of having a single shared cache that everyone has to contend for.

Yes, you lose some economies of scale. Yes, sometimes different teams will make different decisions about similar problems. Yes, you’ll occasionally need to reconcile inconsistencies.

But the latency improvement is massive.

I’ve seen teams go from multi-week decision cycles to same-day decisions just by embedding the right specialist and giving them actual authority. Not “advisory input.” Not “must be consulted.” Actual decision-making power.

The product team with an embedded SRE who can say “yes, that’s fine, ship it” doesn’t have to wait for a ticket to be triaged by the platform team. The squad with an architect who was in the room for all the context doesn’t have to write a six-page document explaining the context to a review board.

Local cache. Local decisions. Local trust.

The Trust Latency Tax

The reason organisations don’t do this? Weirdly is also trust.

“What if they make the wrong decision?”

“What if different teams make inconsistent decisions?”

“What if someone approves something that causes an outage?”

These are legitimate concerns. But they’re also latency taxes. Every layer of review you add, every approval gate, every “just run it past the architects first”that’s a network hop. That’s IO Wait. That’s your team sitting idle while a request traverses the org chart and back.

And here’s the thing about centralised decision-making: it doesn’t actually prevent bad decisions. It just makes them slower. The Architecture team isn’t magic. They don’t have context your team doesn’t have. Often they have less context, because they weren’t in the room when the tradeoffs were discussed.

What centralised review actually provides is diffusion of responsibility. If the decision goes wrong, well, it was approved by Architecture. It went through the process. Nobody’s individually accountable because everybody was involved.

But that’s not trust. That’s the opposite of trust. That’s a system designed around the assumption that individuals can’t be trusted, so we need a bureaucratic apparatus to catch their mistakes.

In distributed systems terms, you’ve built a system that requires synchronous consensus for every write operation. It’s “safe.” It’s also so slow that nothing ever ships.

Where have I seen that system before? Oh right, you know about CAP Theorem?

CAP Theorem, But For Organisations

There’s a famous theorem in distributed systems called CAP: you can have Consistency, Availability, and Partition tolerance, but you can’t have all three. You have to pick two.(and if you don’t pick…)

Organizations have a similar tradeoff that nobody talks about.

You can have:

• Consistency: Every team makes the same decisions the same way. Strong central governance. Architectural purity.

• Velocity: Decisions happen quickly. Teams ship fast. Low latency between idea and execution.

• Scale: Lots of teams working on lots of things. Big organisation. Many concurrent efforts.

Pick two.

If you want Consistency and Scale, you need massive coordination overhead. Your Architecture team becomes a bottleneck. Everything queues. Velocity becomes a joke.

If you want Consistency and Velocity, you need to stay small. One team, one decision-maker, no handoffs. Works great until you need to scale.

If you want Velocity and Scale, you have to relax Consistency. You MUST accept that different teams will make different decisions. You have to trust the people closest to the problem to make reasonable calls, even if those calls aren’t globally optimal.

Most organizations say they want all three and then build structures that prioritize Consistency above all else. They put every decision through a centralised review process, wonder why nothing ships, and blame the individual teams for being slow.

The teams aren’t slow. The architecture is slow. You’ve built a system where every write has to achieve global consensus, and you’re surprised that writes are expensive? Take a step back, down or out. (just don’t break your ankle like me ok)

Build Better Cache Layers

The fix for my file sync was just... waiting. The rebuild finished eventually, the drives stopped thrashing, and subsequent syncs ran at proper speed. Sometimes you just have to let the parity rebuild complete. I picked Consistency and scale and I had ran out of scale.

But for teams, you have more options. You can think about where knowledge and authority actually live, and whether you’ve accidentally put everything on the slowest tier:

L1 Cache (instant): What can your team decide without asking anyone? What knowledge do they carry in their heads? What authority do they actually have? Every time someone has to send a Slack message and wait for a response, you’ve blown past L1.

L2 Cache (seconds): Documentation. Runbooks. Prior decisions written down somewhere findable. “How did we handle this last time?” shouldn’t require archaeology or a meeting with someone who was there.

L3 Cache (minutes): Shared team knowledge. The embedded specialist who has context and authority. The architect in the room, not the architecture review board three weeks from now.

RAM (acceptable delay): Input that genuinely requires synchronous communication. A meeting, but a short one. A decision that actually needs discussion because it’s genuinely novel.

Disk (danger zone): Multi-week approval processes. Information locked in someone’s head. Decisions requiring escalation through four layers of management. Centralised review boards with six-week backlogs. The CTO’s meeting with AWS.

Every time work travels from L1 to “disk,” you’re introducing months of latency for your team. The work might eventually complete. But at what cost in waiting?

The Metric That Actually Matters

What are you actually measuring when you say that your team is successful?

Most places measure load. Tickets in progress. Story points committed. Utilisation rates. Calendar density. Some even measure “engagement”whatever that means or count the number of pull requests opened.

These are all load metrics and load is one of the most misunderstood concepts in computing.

On a Linux system, there’s a number called “load average” that you’ll see if you run uptime or top. Three numbers, representing the average over 1, 5, and 15 minutes. People see these numbers and think they’re measuring CPU utilisation. They’re not

Load average counts processes in two states: those actively running on a CPU, and those waiting. Waiting to run. Waiting for IO. Waiting for a disk that’s busy rebuilding parity because Past You made a responsible decision three days ago.

A system with a load average of an “8” on a 4-core machine isn’t necessarily doing twice the work it can handle. It might have 7 processes sitting in IO Wait while one process actually runs. The load looks terrible. The CPU is basically idle. Everything is just... waiting for data.

You can have a load average through the roof and almost zero useful throughput.

I’ve seen teams that operate exactly like this.

A team with packed calendars, high “engagement” metrics, and stalled projects is the same thing. Everyone is working. Nobody is shipping. And somewhere in the background, there’s a parity rebuild that nobody remembered to mention or a decision traversing six teams on its way to someone with authority.

The question isn’t “Is everyone busy?”

It’s “Is the productive work actually executing, or is it waiting for data to arrive from somewhere else?”

“Do the people closest to the problem have the trust and authority to actually solve it? Or do they have to make a network call to someone who wasn’t in the room?”

I did briefly consider pausing the rebuild to prioritise my sync. You can do that, technically. But then I imagined explaining to my wife that I’d lost all our family photos because I was impatient at 2am and I decided that some latency is acceptable.

Sometimes the background work really is important. Sometimes you have to let the parity rebuild finish. Sometimes the Architecture review board genuinely needs to weigh in because the decision has organisation-wide implications.

The trick is knowing the difference. Knowing which decisions need global consensus and which ones can be made locally. Knowing when you’re adding safety and when you’re just adding latency. Knowing whether your org chart is an efficient distributed system or a bureaucratic apparatus that requires seventeen approvals to change a DNS record.

So there I was, standing at the front of a conference room in Manly, about to ask thirty forty grown adults to put their hands on their heads like they were being arrested. 65 ft 7.4 in Twenty meters from the beach, inside one of those hotel conference rooms that somehow manages to show you the beach and block out all evidence of natural beauty, and I'm about to demonstrate why expensive governance tools are probably a waste of money.

But I'm getting ahead of myself. Let me back up...

The hands-on-head incident example

Last Many weeks ago we had our team offsite - the whole extended crew, including some new joiners who were probably still trying to figure out if we were a normal company, an AWSome company, or one of those places where the Kool-Aid flows a little too freely. Our head of sales had this idea that we'd all share a "gift" during pod introductions - some wisdom or tip to impart to the team.

Everyone else was sharing sensible things. Practical advice about customer engagement. Tips for managing stakeholder expectations. You know, professional stuff that belongs in a conference room. Meanwhile, just beyond those hermetically sealed windows, the real world was happening. Waves were breaking. People were laughing. Life was being lived. It was like being served a perfectly plated photograph of food while slowly starving to death.

Then it's my turn.

"For me to share my gift, you need to put your hands on top of your heads and close your eyes."

The room goes quiet. Someone definitely thinks I've lost it. Others knew it well before now

"Go ahead, it'll make sense in a minute."

They all did it. Every single person.

Hands on heads, eyes closed, looking absolutely ridiculous compliant. If someone had walked past our conference room at that moment, they'd have thought we were doing some bizarre corporate mindfulness exercise or maybe being held hostage by the world's politest bank robber.

"Great - thanks - put your hands down."

Then I explained what just happened. And why it matters more than any governance tool you'll ever buy, sell, partner with, or build.

The Problem We're All Pretending to Solve

Here's what I told them, and what I'm telling you now: You are a trusted advisor for your customers. When we have a great culture and ask our teams to do something, they're generally capable of doing it properly. Not because we have the right tools. Not because we have the perfect processes documented in Confluence, Notion or 10,000 other relevant and useful tools

Culture creates compliance in ways that software never can.

I see this pattern everywhere. A company has a governance problem - maybe they're not following security protocols, or their documentation is a mess, or they can't get teams to follow the approved development process. So what do they do? They buy a SaaS tool. FANCY

"This'll fix it," they think, writing another check paying another invoice, to another vendor who promises their platform will magically transform organizational behavior.

It's like... imagine if after my little demonstration, someone hadn't put their hands on their head. And instead of asking why - instead of understanding what cultural or communication issue prevented compliance - I went and found a vendor. Paid for a tool that would come around and report on whether people did or didn't comply with hand-on-head requests.

What if the tool could somehow force you to do it? Would you be happy? Would you actually use the tool? Or would you find seventeen creative ways to work around it while technically maintaining compliance?

(I'm betting on option three. I've seen developers create entire shadow IT infrastructures just to avoid using the "approved" tools.)

Why We Keep Making This Mistake

The thing is, I get it. I really do. Buying software feels like solving a problem. It's concrete. You can put it on a roadmap. You can report to the board that you've "addressed the governance challenges" with a "best-in-class solution." There's a line item in the budget, contracts are signed, implementation is scheduled. It feels like progress.

Culture change? That's messy. It's slow. You can't really put "convince developers to actually care about documentation" on a Gantt chart. There's no vendor to blame when it doesn't work.

And let's be honest here - I've made this mistake myself. More than once. One time I had a massive problem with inconsistent deployment practices. Different teams doing different things, no visibility into what was actually happening in production, the usual chaos.

So naturally, we bought a tool. A really expensive tool. It had dashboards! And workflows! And integrations with everything!

Six months later, we had beautiful dashboards showing us that nobody was using the tool correctly. The teams that were already following good practices continued to do so. The teams that weren't... found creative ways to mark tasks as complete without actually doing them. We'd spent six figures to get prettier reports about the same problems we already had.

The Uncomfortable Truth About Tools

Here's what vendors won't tell you (and what I learned the expensive way): SaaS can't make people do the thing. Only culture can.

Tools are amplifiers. They take whatever culture you have and make it more visible, more measurable, maybe more efficient. But they don't change the underlying behavior. A great tool in a broken culture just gives you detailed metrics about how broken your culture is.

Think about it this way - and this is where my conference room demonstration comes in. When I asked everyone to put their hands on their heads, they did it because:

1. They trusted that I had a reason (even if they didn't know what it was yet)

2. The social contract of our team culture made non-compliance more uncomfortable than compliance

3. They were curious about where this was going

4. There was psychological safety - they knew they wouldn't be humiliated or punished for participating

Now imagine trying to achieve the same result with software. "Please log into our Hand Position Management System (HPMS) and update your status to 'Hands on Head.' Don't forget to attach photographic evidence and complete the post-hand-raising survey!"

You're laughing (I hope you're laughing), but this is literally what we do with governance tools. We try to software our way out of cultural problems.

How to Actually Do This (Without the Fluff)

Alright, enough philosophy. Let's get practical. How do you actually build a culture of governance before (or instead of) buying tools?

Start with One Team: Don't try to boil the ocean. Find one team that's either suffering from lack of governance or already naturally good at it. Work with them to understand what governance means in their context. What are they trying to prevent? What are they trying to enable? Build something that works for them, then expand.

Make It Visible: Culture thrives on visible examples. When someone follows good governance practices, make it visible. Not in a "gold star for Jimmy!" patronizing way, but in a "here's how the platform team's new tagging standard saved us from a production outage" way.

Connect Governance to Outcomes People Care About: Nobody cares about governance for governance's sake. They care about not getting fired. They care about not working weekends. They care about shipping features that don't break. Show how governance connects to these outcomes.

Start with Why, Not How: Before you document a single process or buy a single tool, make sure everyone understands why this governance matters. And "because compliance says so" isn't a why - that's a what.

Build Feedback Loops: Culture dies in darkness. You need mechanisms for people to say "this isn't working" without fear of retribution. And then - this is the critical part - you need to actually change things based on that feedback.

Lead by Example: If you're asking developers to follow secure coding practices, leadership better not be sharing passwords in Slack. If you want teams to document their architectures, executives better be able to articulate the company's technical strategy.

The Tool Paradox

Here's the thing that might surprise you after this whole rant: I'm not actually anti-tool. I love good tools. I've built my career on knowing which tools solve which problems.

But tools are tactics, not strategy. They're implementation details, not solutions.

The paradox is that the better your culture, the less sophisticated your tools need to be. A team with strong governance culture can maintain better compliance with a shared spreadsheet than a team with weak culture can achieve with a million-dollar platform.

I've seen teams maintain perfect infrastructure governance using nothing more than well-structured Terraform Cloudformation and strong code review culture. I've also seen teams with every governance tool imaginable still shipping credentials to public repos.

The tools aren't the difference. The culture is.

Bringing It Home

Remember my conference room demonstration? Everyone put their hands on their heads not because a tool made them, but because the culture of our team made it the natural thing to do. That's what real governance looks like - people doing the right thing because it's become the obvious thing to do.

You can't buy that. You can't install it. You can only build it, one interaction at a time, one team at a time, one leader at a time.

So next time someone suggests buying a tool to solve a governance problem, try this instead: Get everyone in a room. Ask them to metaphorically put their hands on their heads and describe their real problems. When they do (or don't), ask why.

The answer will tell you everything about whether you have a tools problem or a culture problem.

And I'm betting it's not a tools problem.

What's your experience with this? Have you seen organizations successfully build governance culture, or have you watched them fail trying to buy their way there? Drop me a line - I'm genuinely curious about how others are tackling this challenge. Because despite what I've written here, I definitely don't have all the answers. I'm just someone who's made these mistakes enough times to recognize the pattern.

And if you're wondering - yes, everyone at the offsite got the message. Sometimes the best way to teach about compliance is to make people voluntarily look ridiculous for thirty seconds. The lesson tends to stick.

I’m strict judicious careful conscious with my time trying to get home from work before 6:30 PM. It’s a 25min commute and I’ll take the train, the T2 Line from Town Hall towards the Inner West. We have an open concept office floor-plan so I pack up my laptop, put on the sunnies, pop in the AirPods and badge out. I’m on my way to do nothing.

Well not actually nothing.
As the air moves in and out of my lungs, my heart is picking up the pace as I rush towards the train, my thoughts are on which pocket my Opal card is crammed into, and my stomach is grumbling. I’m decidedly doing something.

As humans we’re not really familiar with the concept of doing nothing. We always have a little “Status:” tag on the taskbar of life.

Subscribe now

Status: Rushing

As I pass the gates to the train platform, I think “Later tonight, melatonin willing, I’ll be Status: Sleeping” (Yes, I can think in bold and italics)

Your computer has a different experience. It “understands” the concept of fully unplugged.

Earlier today, my Outlook client locked up, not responding to ⌘ + Q I opted for a force quit. Why can’t I gracefully stop this program. It’s stuck, the OS knows it’s stuck, I know it’s stuck, we both hope it gets unstuck, and without my guidance, it will stay stuck indefinitely.

“pkill -9 Outlook” is what I’m thinking as I Option + Two Finger Click on the icon and jab at “Force Quit”. It’s gone. I dismiss the, so far useless, Microsoft Crash Reporter.

Outlook is no more. It’s doing nothing. Took it long enough.

Status: Waiting

The T2 train line at this time is typically served every 3-9min. I’m viewing the train’s progress on an app. I’m observing a system that I can’t change, I’m getting on the train regardless. A watched pot does boil, but there really isn’t a reason to watch it happen. I check my notifications, I never clear them, they’re AI summarized and ordered mostly by priority.

Swipe - 2 posts on Reddit -Swipe - Mail checked - Swipe

We really don’t do “nothing” well. It’s probably not possible to be alive and doing nothing at the same time - put it in the comments if I’m wrong. Some might say we made computers to think like we do - sure - but you can turn a computer back on after its been unplugged for a few years.

Status: Sitting

I sit in the caboose last train-car. I still have my headphones in, I’m listening to nothing.
My AirPods are playing an inverted waveform of what my ears should be hearing. This nothing takes quite a lot of something to exist. I’m very much enjoying the silence and blissfully unaware of the tiny ARM processors whirring away. They're working harder to create nothing than they would to play something. It's a manufactured nothing, an artificial absence.

The train lurches forward, and my mind drifts to system states. In computing, we have clear delineations: running, stopped, sleeping, terminated. But human consciousness exists in fuzzy states, always processing something, even when we think we're doing nothing.

Status: Transit

My phone shows

6:12 PM

I could check my email again, but I recognize this as the brain's equivalent of a polling loop - an inefficient use of resources. Better to implement an interrupt-driven architecture: the important emails will notify me. The unimportant ones can wait until tomorrow's inevitable Status: switch back to work mode.

Between stations, I notice my reflection in the window, superimposed on the passing graffiti of RAYGUN and other indecipherable, yet artistic, analogue manifestos - each one a pixel-perfect act of beautiful destruction, sprayed in defiance of the cleaning crews' eternal war on expression. The writers' tags float by like deprecated commands in a legacy codebase: still executing, still meaningful, just not to everyone.

Status: Background Processing

Even as I try to do nothing, my mind performs garbage collection on the day's events, defragments memories, optimizes neural pathways.

I’m thinking about the interactions I had with my team and clients, the random stranger with the full body tattoos on the sidewalk yesterday with the bold letters on his knuckles that smiled at me before we spontainously fist-bumped - whats his story?

The human equivalent of ‘system maintenance’ runs whether we consciously initiate it or not. Maybe that's why true nothing eludes us - we're always running some background process, always maintaining some minimal viable state of consciousness.

If we all somehow hit a metaphorical pause button at the same time then no one would notice. Right?

Status: Macdonaldtown Station

This is my least favorite station - an island platform floating in a sea of rail junction switches, servicing what feels like a pointer to a deallocated suburb. Two minutes of execution overhead between Redfern and Newtown that any decent optimizer would eliminate. Even its name throws a 404 error - there is no Macdonaldtown - just a dangling reference to some long garbage-collected gone estate.

I’ve realized I’m doing something again - this station is the bane of my quasi-meditative return home.

The train doors remain open. Through the window, I watch exactly three passengers disembark into the liminal space, their footsteps would echo on the empty platform if I could hear them.

In a way I'm jealous of the station; it's achieved what I'm struggling towards - a near-perfect implementation of doing nothing. Most trains pass through without triggering its interrupt handler, leaving it in its preferred idle state, executing its null operation with flawless efficiency.

Status: Mindwandering

I've had my head deep into event driven architecture patterns today - there's no human equivalent to Lambda functions, those ephemeral pieces of code that spring into existence, execute their purpose, and vanish without a trace. We can't just spin up a new instance of ourselves to handle peak load, or gracefully terminate when our task is complete. Our consciousness persists, an always-on service with no shutdown command. OOO look at me I’m Mr Meeseeks - At least existence isn’t pain.

Maybe that’s why it feels so unnatural to build with functions the first few times, it’s a new experience that we can’t understand ourselves. Our brains are stateful monoliths, carrying years of accumulated technical debt and deprecated memories. I can't just garbage collect the embarrassing things I said at last year's Christmas party.

I watch as the train reaches platform after platform, its wheels silently clicking against the rails like a broken mechanical clock. The sun catches the metallic reflection of a passing train on the opposite track - a perfect metaphor for parallel processing that I'll probably forget before I can write it down.

Status: Preparing

I'm indexing through stations, waiting for length-1 so I can run my exit routine. My internal countdown hits one station to go. Where is my opal card again? No-one is seated next to me and I have a clear path to the isle. Zero resistance.

Zero is a weird concept.

I have 5 apples and I give you 5 apples now I have zero apples. Zero is an intellectual abstraction, a placeholder, a mathematical sleight of hand that revolutionized how we think about quantity. Think about your bank account balance. When If it shows $0.00, that's not nothing - it's a very specific something that means the absence of money. It's a pointer that points to nothing, but the pointer itself exists. The account of the zero exists but holds nothing itself. You still have an account if there’s nothing in it. Can you only have zero of something if you had more than zero of it previously?

As I depart the train, my noise-cancelling headphones create a near-zero noise environment, but only because there was noise to cancel. You can't cancel nothing. The AirPods aren't creating silence - they're creating anti-noise, a deliberate something that results in a perceived nothing.

I’m tapping my opal card on the reader so the system acknowledges I’m no longer in transit. The state of my card will read that I’m “tapped-off” if one of those pesky ticket inspectors showed up.

In memory management, null is not the same as unallocated. A null pointer is still a pointer - it's explicitly pointing to nothing, which is different from a pointer that doesn't exist. My opal card doesn’t have an unallocated status - it’s something closer to a null status. It’s still maintaining its identity even in its empty state. It's a container labeled "nothing here right now, but there could be."

Maybe that's why zero is such a strange concept - it's not really nothing, it's the deliberate acknowledgment of nothing.

Status: Walking

The evening air hits my face as I remove my AirPods, their manufactured nothing gives way to the suburb's symphony of something. Really it was always something I heard. Hearing nothing is still to acknowledge the existence of nothing and if it exists is it really absent? As I finish my last-kilometer commute on foot I think about how the world doesn’t “do zero”. Even in sleep, our minds play maintenance scripts dreams and defrag consolidate our memories. We invented zero, just like we invented the concept of nothing, just like we invented the idea that we could somehow achieve a status of doing nothing.

By the time I reach my home I think about if I have been successful in doing nothing. By my admission that nothing is something. I have to agree that I have succeeded.

After all, doing nothing is just something we collectively made up.

This blog post about Termites is a stub - you can help one twenty three cloud street grow by expanding upon your experience in the comments.

What is a Termite?

We know a few things about the Termite

• Termites are insects from the order Isoptera

  • Isoptera: Eusocial, wood-eating, colony-forming insects.

• Over 3,000 species, with many yet to be discovered.

• Incomplete metamorphosis, nymphs develop within the colony.

  • Ew

• Critical decomposers and soil engineers

  • Ever met a soil architect?

• ”Useful” as food for some animals and even humans in some cultures

• Can be destructive to wooden structures.

  • The choice is their own - like the honey badger, they take what they want.

• Termite Jokes need a lot of citation to understand(1)

  • Termite 1: "Hey, guess what? I'm feeling multi-potent today! I could become a worker or even a soldier!"

    Termite 2: "Oh yeah? Well, I'm totipotent! I could become anything - even the queen!"

    Termite 1: "Wait a second... aren't you just a first instar?"

    Termite 2: "Maybe. But at least I'm not stuck being a pseudergate like you!"

    Termite 1: "Ouch. Way to rub salt in my wingless wounds."

• Termites are causing your production outages.

I’m not here to talk to you about a group of detritophagous eusocial insects or the Rhinotermitidae, or the Nasutitermes triodiae of the Northern Territory (located in the southern hemisphere, east of Western Australia, and west of New South Wales). I’m here to talk to you about the hidden problems in your architecture that have become your “unknown unknown’s”. These problems blind side you like a rogue wave of fermented shark hitting your unsuspecting palate in a dimly lit Reykjavik bar, leaving you questioning every life choice that led you to this moment of gastronomic revelation and existential crisis I do when you read my strikethrough text.

When was the last time you spent more than 2 minutes discovering the root cause of a failure? When was the last time you spent more than 2 days? 2 weeks? How long were you down?

Resiliency should be a 1st class citizen in your architecture, design and planning and a major part of resiliency is observability.

Let me guess - the last time you built something, you thought about scaling, maintenance patterns, architecture of your messaging and eventing, data patterns - oh and only then did you think about monitoring it?

You should have thought about that all along the way - how will you know if you have termites deep in the soul of your house application if you don’t have the ability to poke around those parts and check them routinely?

Termites are a symptom of the lack of observability

They hide and can hide for a long time before becoming a lasting problem and that can lure you into a false sense of stability.

Lets look at this basic 3-tier architecture:

Fantastic - we’ve all seen this before. Now let’s add our logging system.

Looking good - what could go wrong?

Well if you deploy’d this with Docker (or ECS, or likely most any container platform). Then you’ve just introduced a single point of failure by introducing observability as an afterthought.

Docker provides two modes for delivering messages from the container to the log driver:

• (default) direct, blocking delivery from container to driver

• non-blocking delivery that stores log messages in an intermediate per-container buffer for consumption by driver

Quote Source: https://docs.docker.com/engine/logging/configure/

In blocking mode - which is the default in Docker, and ECS - if your logging provider/service is having a bad time, then so are your containers. Can’t write to your logging provider? Well then your container falls over and restarts over and over again until the logging provider is back online.

In non-blocking mode - which you need to change away from the default configuration to use - a logging provider failure does not impact the functionality of the container!

When you (hopefully) tested your failure modes or the initial system before the introduction of logging you wouldn’t have been any wiser to this failure case. You might be able to run for a very long time before your logging provider has any issues causing an outage. You’ve got a termite infestation now - it’s just a matter of time before this fails and takes your whole stack down.

This is just one example of a termite infestation - your mileage may will vary!

How can I check for termites in my stack?

Simple. Test your failure cases. In your staging environment make something fail, see how the system reacts and plan around that failure case. Then do it again and again and again. You can have great diagrams showing multiple arrows going to multiple boxes to describe your resiliency - but just like a backup - you don’t know its real until you test the restoration - you don’t know your failure modes until you test them. This is how to discover your “unknown unknown’s”.

It’s great to be able to recover from a failure quickly, it’s great to detect a failure quickly - but these pale in comparison to your ability to INCREASE THE TIME BETWEEN FAILURES. To do this - you must understand what could fail(everything), how it will fail WHEN (not if) it does fail and how your system will react. Then plan for that failure case.

How can I prevent termites from Day 1?

In Australia, at a cost of more than A$1.5 billion per year,^[242] termites cause more damage to houses than fire, floods and storms combined.^[243] In Malaysia, it is estimated that termites caused about RM400 million of damages to properties and buildings.^[244] The damage caused by termites costs the southwestern United States approximately $1.5 billion each year in wood structure damage, but the true cost of damage worldwide cannot be determined.^[236][245]

Quote Source: https://en.wikipedia.org/wiki/Termite

You treat your observability and resilience as first class citizens along side your planning for hardware, software and cost management. Early in the design process you think about the failure cases and implement mechanisms to test them routinely.

Experiencing 'termites' can be a tough lesson, as many of us can attest.

I’d be eager to hear how you've faced off with these nuisances—share your 'termite' tales in the comments below.

This post is meant as a reference guide and a starting place for deploying basic websites quickly and securely.

Average Read Time: 18min

Manifest

• Introduction

• Root Domain (123cloud.st) to Subdomain (www.123cloud.st) redirects

  • What Doesn’t Work

  • Option 1 - CloudFront with S3 - redirect via index.html

  • Option 2 - CloudFront with S3 Public Website

  • Option 3 - CloudFront with Lambda@Edge

  • Option 4 - CloudFront with CloudFront Functions

  • Root Domain to Subdomain Redirects Conclusion

• Standalone Public S3 Buckets

• S3 + CloudFront Overview

  • Option 1 - S3+CloudFront with OAI(Origin Access Identity)

  • Option 2 - S3+CloudFront with OAC(Origin Access Control)

Do you want to just look at the code? Then go to https://github.com/jeeshofone/Public-AWS-Website-Patterns/tree/main)

Introduction

I love building stupid websites. I also love using AWS for deploying them. There are a lot of different way too many ways mechanisms to host a basic website on AWS. From the absurdly complicated reference architecture for Wordpress to a public S3 bucket and everything in between.

In this post I’m going to provide a comprehensive set of patterns (and provide working, out of the box deployable CloudFormation templates) that I use everyday to make websites secure, cheap and fast.

Share

Subscribe now

Root Domain to Subdomain Redirects

Why would I want a root domain to a subdomain redirect? You used one getting to this blog post (if you’re reading this on 123cloud.st). Substack requires the use of a subdomain for its custom domain names. So I have to have www at the front of my URL. I don’t want you to type out the /dʌbəl.juː/ /dʌbəl.juː/ /dʌbəl.juː/ /dɒt/ so I need a way to direct you from 123cloud.st to www.123cloud.st.

Let’s talk quickly about what doesn’t work.

You may be familiar with CNAMES which “redirect” one domain name to another. This won’t work for us because of something very cool called SNI.

www.123cloud.st points to target.substack-custom-domains.com - in fact every website on Substack with a custom domain name points to this URL. We can dig(groaning intensifies) deeper. What is at target.substack-custom-domains.com?

When we look at the ANSWER SECTION of our dig output, we see 2 IP addresses.

104.18.40.87 and 172.64.147.169 (although likely depending on where I am, I will get different addresses)

These IP’s are owned by Cloudflare which provide edge caching and distribution for Substack. Awesome. Now let’s look at SNI.

Server Name Indication, or SNI, is an extension of the TLS protocol. SNI allows a server to present multiple certificates on the same IP address and port number and hence allows multiple secure (HTTPS) websites (or any other service over TLS) to be served by the same IP address without requiring all those sites to use the same certificate.

If we did not have SNI, we would need a separate IP address for every single site that needs a secure connection using a unique https certificate, which is obviously not scalable with the limited number of IPv4 addresses and would be a huge waste of IPv6 addresses as well.

For example, when your browser attempts to establish a secure connection with www.123cloud.st, it includes the hostname in the TLS handshake so that the server knows which website's certificate to present. If SNI were not used, Cloudflare would have no way to know which certificate to present because it hosts possibly thousands of different domains on a single IP address. As such, SNI is crucial for a multi-tenant hosting environment like Cloudflare's edge network.

This is why when I setup “ww.123cloud.st” (I did set it up, try it!) to point at “www.123cloud.st” with a CNAME record you get an error from Cloudflare. It doesn’t know what you’re talking about as the origin subdomain is ww and not www.

Subdomain Redirection (Option 1) Finally he’s showing us a solution

Since we can’t use DNS to direct our users around, we need to actually use an http redirect. There are many different kinds of http redirects. Lets look at this one first

<html><head><meta http-equiv="refresh" content="0; url=https://{redirection_url}"></head></html>

The “redirect” above is technically just refreshing your browser window and loading a new page. Some SEO experts will not recommend doing this; but how about we do it anyway? Well since we’re exploring AWS Native mechanisms; how will we host this redirect?

Let me explain how the template works.

1. Parameters: Inputs required for the template to work.

   • RootDomainName: The root domain you intend to redirect from.

   • WWWDomainName: The 'www' or other subdomain you want to redirect to.

   • HostedZoneID: The identifier of the hosted zone within Route 53 for the RootDomainName.

2. Resources that will be created:

   • RedirectBucket: An Amazon S3 bucket configured with public access blocked that will store the redirect page.

   • RedirectBucketPolicy: A policy applied to the S3 bucket that allows CloudFront to access the objects in the bucket.

   • CloudFrontOriginAccessControl: A resource that controls access to the S3 bucket from CloudFront using Origin Access Control (OAC).

   • CloudFrontDistributionForRedirect: A CloudFront distribution setup to redirect the root domain to the 'www' subdomain.

   • RedirectAcmCertificate: An ACM certificate resource representing the SSL certificate for the root domain.

   • Route53RecordSetGroup: A set of Route 53 DNS records (A and AAAA Types) that point your root domain to the CloudFront distribution.

   • LambdaExecutionRole: An AWS Identity and Access Management (IAM) role that gives Lambda very limited permissions to interact with other AWS services.

   • S3FileUploaderLambda: A Lambda function that is triggered by a Custom Resource to upload the redirect HTML page to the S3 bucket.

   • S3FileUploadCustomResource: A custom resource that triggers the Lambda function to upload the redirect page to the S3 bucket upon stack creation.

3. Outputs: Named references to the values returned by the template. In this case:

   • CloudFrontDistributionDomainName: The domain name of the CloudFront distribution.

   • RedirectAcmCertificateArn: The Amazon Resource Name (ARN) of the ACM certificate.

The overall functionality of the template is to create an S3 bucket that contains a simple HTML file that redirects our visitors to the 'www' subdomain. The bucket is made accessible only via the CloudFront distribution, which is linked to the root domain using Route 53 DNS records. An SSL/TLS certificate from ACM ensures secure browsing using HTTPS.

Upon stack creation, the Lambda function is invoked to upload the necessary HTML file into the bucket, enabling the redirection. The IAM role is provided to grant the only necessary permissions for the Lambda function to carry out its operations.

Please feel free to use this template but make sure you read and understand it before deploying it into your own account. You’ll want to deploy this in us-east-1 as CloudFront can only use ACM certificates from that region.

Link to Cloudformation template

Subdomain Redirection (Option 2)

Theoretically, a 301 redirect is the “best” - 301 is for “Moved Permanently” and many believe it is better for SEO. So how does a 301 redirect actually work?

1. Client Request: A user enters a URL into their browser or clicks on a link to navigate to a page.

2. Initial Server Response: The server that hosts the original URL receives the request for the resource (a web page, an image, etc.).

3. Redirect Instruction: The server checks its configuration for the requested URL. If a 301 redirect is set up, the server knows that this resource has been moved permanently to a new location. Instead of serving the original resource, the server sends back a response to the client with the status code 301, which means "Moved Permanently".

4. Location Header: Along with the 301 status code, the server includes a "Location" header in the response. This header specifies the new URL where the requested resource now resides.

5. Client Receives Redirect: The client (browser) receives the 301 status code and the new URL from the "Location" header. Understanding that the resource has been permanently moved, the browser automatically initiates a new request to the URL specified in the "Location" header.

6. Accessing the New Resource: The server at the new URL receives the request and, providing the resource is there, serves the desired web page or file to the client.

7. Updating Links: Search engines and browsers may update their links or cache to reflect the new URL. This is because a 301 redirect indicates a permanent move, and so the old URL is essentially replaced by the new one in indexes and bookmarks.

This feels a bit heavier than the simple html redirect in Option 1; however, it actually uses fewer resources but does require a public bucket.

It requires you to have a PUBLIC BUCKET due to the way that S3 redirects work with CloudFront. We can’t use an OAI or an OAC to control access via CloudFront and we must use the S3 Bucket Website endpoint which requires a public bucket. This bucket should have restrictions! However, it means you can’t put the BLOCK ALL rules on your AWS Account. This method requires this risk. I use a separate account for this use case across all my sites to prevent data from accidentally being added to these buckets.

2 important things to note when using a 301.

• A client’s browser can cache the response for a period of time. These redirections should not be used for frequently changed links.

• Many believe that a A 301 redirect is critical for maintaining SEO rankings. When a page is moved, the 301 redirect tells search engines that any rankings and link equity the old page had should be transferred to the new page.

So how do we do a 301 redirect with an AWS Native solution?

Link to Cloudformation template

1. Conditions:

   • CreateDNSRecordsCondition: Determines whether DNS records will be created based on the CreateDNSRecords parameter.

   • UseExistingCertificateCondition: Checks if an existing ACM Certificate ARN has been provided.

   • CreateNewCertificateCondition: Determines whether a new ACM certificate should be created based on CreateACMCertificate and the presence of an existing certificate.

2. Resources: The AWS resources that will be created and managed by the stack.

   • ACMCertificateResource: An SSL certificate managed by AWS Certificate Manager, created if needed per the conditions.

   • S3BucketForRedirect: A PUBLIC S3 bucket configured to redirect all incoming requests to the destination domain using HTTP 301 redirects.

   • S3BucketPolicyForRedirect: A bucket policy enabling public read access to the objects in the S3 bucket.

   • CloudFrontDistribution: A CloudFront distribution configured to set http-only for the custom origin (since S3 websites do not support HTTPS) and redirect all viewer protocols to HTTPS. It uses the provided ACM certificate or the one created by the stack.

   • Route53RecordSetGroup: DNS records in Route 53 (A and AAAA records) pointing the source domain to the CloudFront distribution, created if DNS record creation is enabled.

Our First request to http://123cloud.st takes us on a 301 to https://123cloud.st which is served via CloudFront’s ViewerProtocolPolicy: redirect-to-https

That request (below) takes us via 301 to https://www.123cloud.st which is configured by
WebsiteConfiguration:
RedirectAllRequestsTo:
HostName: !Ref DestinationDomain
Protocol: 'https'

Both Option 1 and Option 2 for subdomain redirection are relatively inexpensive, each serving very little data and using mostly free tier levels of resources (depending on your traffic volume). The main trade off is Option 1 gives you poor SEO performance, and Option 2 gives you a Public S3 bucket.

So is there an Option 3 that prioritizes SEO and Security?

Subdomain Redirection Option 3

Lets get fancy with Lambda@Edge

Lambda@Edge is a feature of Amazon CloudFront that lets you run code closer to users of your application, which can improve performance and reduce latency. With Lambda@Edge, you don't have to provision or manage infrastructure in multiple locations around the world. You pay only for the compute time you consume - there is no charge when your code is not running.

You can test this with 123cloudstreet.com or on your own with the Cloudformation template below.

Link to Cloudformation Template

Here's how we can use Lambda@Edge for a 301 redirect from a root domain to a subdomain.

Now when a request comes into your CloudFront distribution for the root domain, the Lambda function will be invoked. It will check the host header and return a 301 redirect to the www subdomain.

This approach has several benefits:

• It's SEO-friendly. 301 redirects are the recommended way to permanently redirect URLs for SEO purposes.

• It's secure. You don't need to make your S3 bucket public. You don’t even need an S3 Bucket. This uses an empty private S3 bucket because CloudFront needs an origin configured.

• It's scalable. Lambda@Edge scales automatically with your CloudFront traffic.

• It's customizable. You have full control over the redirect logic in the Lambda function which could allow you to setup multiple redirects or control different patterns.

The main drawback is that Lambda@Edge can be a bit more complex to set up compared to the other options. Also, keep in mind that Lambda@Edge is priced differently than regular Lambda functions.

Lambda@Edge charges for the total number of request for the function and for every GB-second.
Request pricing is $0.60 per 1 million requests ($0.0000006 per request).
Duration pricing is $0.00005001for every GB-second used

This means that “if you allocate 128MB of memory available per execution with your Lambda@Edge function, then your duration charge will be $0.00000625125 for every 128MB-second used, metered in 1ms granularity.”

Subdomain Redirection Option 4 - You probably want this one

Let's get even fancier with CloudFront Functions!

CloudFront Functions are a lightweight serverless computing service designed specifically for content delivery, enabling you to run code closer to your users for optimal performance and reduced latency. This service is integral to CloudFront and helps in executing simple functions rapidly at CloudFront edge locations worldwide. You pay only for the compute time you consume, so there is no charge when your code is not running.

You can test this with 123street.cloud or on your own with the provided CloudFormation template.

Link to CloudFormation Template

Here’s how we can utilize CloudFront Functions for a 301 redirect from a root domain to a subdomain or from a root domain to another root domain.

When a request reaches your CloudFront distribution for the root domain, the CloudFront Function will be executed. This function simply checks the request and returns a 301 redirect to the target subdomain.

This approach offers several benefits:

• SEO-friendly: Implements 301 redirects, which are preferred for permanent URL redirection in SEO.

• Secure: The simplicity of CloudFront Functions eliminates the need for additional security configurations typically required for server-based setups.

• Scalable: CloudFront Functions are designed to handle high request volumes, scaling automatically with the increase in CloudFront traffic.

• Customizable: Though much simpler than Lambda@Edge, CloudFront Functions still offer the flexibility to define specific redirection behaviors based on the incoming request characteristics.

The main advantage over Lambda@Edge is the simplicity and lower cost, particularly for simple tasks like redirections. CloudFront Functions are priced at a MUCH lower rate than Lambda@Edge, costing $0.10 per 1 million invocations!

This means that using CloudFront Functions, you benefit from reduced complexity and cost while achieving functionality specific to content delivery scenarios such as URL redirection!

Root Domain to Subdomain Redirects Conclusion

In conclusion, options for redirecting traffic from a root domain to a subdomain on AWS vary in complexity, cost, SEO impact, and security considerations. The four methods outlined offer scalable solutions ranging from simple HTML meta-refresh redirects, to S3 and CloudFront-based 301 redirects, and finally to the more complex but highly flexible Lambda@Edge approach.

• Option 1, using a meta-refresh HTML page, is the simplest to implement but is generally not recommended for SEO. Despite this, it serves as a quick and straightforward method for redirection, suitable for temporary or less critical redirection needs.

• Option 2 leverages S3 for a 301 redirect, which is better for SEO as it conveys to search engines that the move is permanent. This method does require a public-facing S3 bucket, which may raise security concerns, though these can be mitigated with careful configuration.

• Option 3, utilizing Lambda@Edge, represents a sophisticated approach that combines scalability, security, and SEO-friendliness. This method does not require a public bucket and allows for complex redirection logic to be implemented. While it is a bit more complex and very slightly more expensive regarding AWS charges, it arguably offers the best balance between functionality and best practices for professional use.

• Option 4, utilizing CloudFront Functions provides SEO - no S3 bucket and the lowest cost with the same level of security. This is most likely your best option for redirects in all cases.

Want my generic prescriptive guidance? Use Option 4 if you care about SEO and Option 1 if you just want it done dirty. Get all the CloudFormation here.

Standalone Public S3 Buckets

Ah, the audacious charm of deploying standalone S3 public bucket websites—a culinary equivalent of serving a sizzling, rare steak in a vegan commune. It’s not just about throwing caution to the wind; it’s about embracing the grandiose belief that the open web is still a place of purity, much like believing in the sanctity of a street-side taco in Tijuana at 3 AM. This approach is the digital antithesis of a locked safe in a bank vault; it's more akin to leaving your front door wide open, inviting every passerby into the warm embrace of your data. Imagine striding confidently through the back alleys of Bangkok, your server logs fluttering in the breeze behind you, a testament to your unshakeable faith in the digital goodness of mankind. It's a reckless culinary adventure, liberally seasoned with potential calamities, yet thrilling in its defiance of the mundane best practices. So, pour yourself a glass of absinthe, toast to the boundless optimism of your pioneering spirit, and watch as the flavors of your bold decision mingle with the unpredictable spices of the world wide web, hoping the result is palatable, or at least, an unforgettable tale.

S3 + CloudFront Overview

A match made in heaven, S3 and CloudFront will allow you to host a static website in a cost conscious, secure way without the need to maintain any infrastructure.

There are 2 main patterns here for securing access between S3 and CloudFront: OAI(legacy) and OAC.

Origin Access Control (OAC) represents an significant advancement over Origin Access Identity (OAI) for securing Amazon S3 content delivered via CloudFront, primarily due to its integration with S3's Access Point policies and the simplification of access management. OAC enables precise control by leveraging AWS’s advanced policy-based management system, allowing for more granular and secure configurations that are aligned with least privilege access principles. While the transition to OAC offers improved security practices and streamlined configurations, it necessitates a comprehension of S3 Access Point policies and a complex migration for those already invested in OAI configurations. However, the benefits, including enhanced security measures, compliance capabilities, and future-proofing against evolving AWS features and best practices, strongly advocate for OAC’s adoption in modern cloud architectures.

Another topic we now need to discuss is CORS. So far, we have only been discussing redirection architecture, now that we’re hosting content, we need to make sure we use CORS. S3 and CloudFront both support Cross-Origin Resource Sharing (CORS), but they do so in slightly different manners, catering to their respective roles in content storage and distribution within the AWS ecosystem.

Amazon S3 and CORS:

Amazon S3 allows you to store and retrieve any amount of data at any time, from anywhere on the web. When it comes to CORS, S3 provides the capability to configure CORS settings for your buckets directly. This is crucial when your application hosted on one domain (e.g., www.example.com) needs to directly access or manipulate resources stored in an S3 bucket that's not on the same origin.

Configuring CORS on an S3 bucket involves setting up a CORS configuration rule as JSON in the bucket's permissions settings. This configuration includes specifying which origins are allowed to access the bucket, what HTTP methods (GET, PUT, POST, DELETE, etc.) are permitted, whether credentials are supported, and what headers can be exposed to the browser.

For example, if you’re hosting a web application on www.example.com that needs to retrieve assets from your S3 bucket (myapp-assets.s3.amazonaws.com), you would configure the S3 bucket's CORS settings to allow GET requests originating from www.example.com. This would block someone else from trying to have badactorexample.com from using your bucket to serve content on their site.

Amazon CloudFront and CORS:

CloudFront is a global content delivery network (CDN) service that accelerates the delivery of your websites, APIs, video content, or other web assets. It integrates with other Amazon Web Services products, like S3, to you an easy way to distribute content to end-users with low latency and high data transfer speeds.

When you use Amazon CloudFront to serve content stored in an S3 bucket, CORS configurations become slightly more complex since you're dealing with two services.

Here's how CORS works with CloudFront:

1. CORS Configuration on S3: First, you must set the CORS configuration on the S3 bucket as described above.

2. CloudFront Behavior: Next, when configuring your CloudFront distribution, you can set specific behaviors to forward Origin headers to your S3 bucket. CloudFront can either forward all headers to your origin (which can impact caching), forward a whitelist of headers (including Origin for CORS), or not forward headers at all.

3. Handling Preflight Requests: For preflight OPTIONS requests, you need to ensure your S3 bucket or application backend appropriately handles these requests and returns suitable CORS headers. CloudFront itself doesn't automatically respond to OPTIONS requests; the response depends on your origin’s CORS setup.

4. Caching based on CORS Headers: You can configure CloudFront to cache responses based on the Origin header, allowing you to efficiently serve requests for different origins with the correct CORS headers.

Ok great - so lets look at a best practice least privilege design for our first option.

S3 + CloudFront with OAI (Option 1)

OAI or Origin Access Identity is the legacy way of providing authentication between CloudFront and S3; however, it is still used and works just fine. This pattern is well documented across the internet but in the interest of providing a one-stop guide I’ll cover it here.

You can test this pattern at oai.onetwentythree.cloud

Link to CloudFormation template https://github.com/jeeshofone/Public-AWS-Website-Patterns/tree/main/S3%2BCloudfront%20with%20OAI

This AWS CloudFormation template creates a secure, private AWS S3 bucket paired with a CloudFront distribution, utilizing an Origin Access Identity (OAI) to strictly control access. It also sets up a Lambda function for lifecycle management tasks such as initially populating and optionally clearing the S3 bucket. The S3 bucket is configured with a stringent public access block policy and a CORS rule that only allows GET requests from the specified domain. An SSL certificate managed by AWS Certificate Manager is used to enforce HTTPS connections, while Route 53 is employed to set up DNS routing for the domain.

This is pretty straight forward; however, there are some things in the Cloudformation template I want to draw your attention to:

1. CORS is provided by the S3 bucket configuration

2. Nothing but the Lambda function has permissions to write the the S3 bucket.

3. A custom resource exists to trigger the lambda function on stack creation and removal to make and delete the index.html file.

4. You must have a domain in your aws account and be using route53 for this template to work.

5. This template uses ACM for certificates (free) but must be deployed in US-East-1 even though Cloudformation will provide your website at the edge globally.

Now lets look at what needs to change to migrate this from OAI to OAC

S3 + Cloudfront with OAC (Option 2) - You probably want this one

This is the difference between the OAI and the OAC template. We remove the Origin Access Identity and add the Origin Access Control in its place. Then inside the Cloudfront template we undefined the OAI and define the ID of the OAC. In the bucket policy we remove the access from the OAI and provide the access to the OAC.

“SO WHAT?” you might ask - “good question” I might reply.

Why pick OAC over OAI? The short answer is - it’s more secure, and is the supported method going forward. The long answer is here: https://aws.amazon.com/blogs/networking-and-content-delivery/amazon-cloudfront-introduces-origin-access-control-oac/

Link to OAC Cloudformation template - it is exactly the same as the OAI template, but with security between S3 and Cloudfront via OAC instead of OAI.

Conclusion

This blog post has outlined various strategies for deploying basic websites and domain redirects on AWS, utilizing services such as S3, CloudFront, Lambda@Edge, and more.

I hope this guide gets you started faster on your journey to deploying your next website.

This blog post about flies is a stub - you can help one twenty three cloud street grow by expanding upon your experience in the comments.

What is a Fly?

We know a few things about the Fly

• Flies are insects from the order Diptera

  • Diptera: Two-winged, highly maneuverable, diverse insects.

• Over 1 million species, many still undescribed.

• Complete metamorphosis, larvae develop inside food sources.

  • yuck

• Critical pollinators and disease vectors

• Useful as fishing bait

• Can be annoying.

• A scrub is a guy that think’s he’s fly and is also known as a busta(1)

• Now I’m feeling so fly, like a G6 (2)

  • The mathematical property of transitive relation shows that a G6 is officially a Scrub.(3)

    • It also shows that a G6 is a busta. Remember that the next time your rich friend jets off to Monaco.

• Flies are symptoms of expired technical debt

I’m not here to talk to you about the common house fly(Musca domestica) or the Culex Quinquefasciatus, or the Cylindromyia Brassicaria. I’m here to talk to you about the flies hovering around your stand-up table or in the break room.

When we/our/your business takes out a technical line of credit, tech debt will always exist, this is arguably good and unavoidable; however, sometimes things can smell and then the flies get in.

While the irritations of biological flies may be a nuisance in the office setting, they are nothing compared to the metaphorical 'flies' that buzz around the realm of our projects and code repositories. Such flies represent the disproportional accumulation of technical debt in a specific area– those pesky issues we know exist yet often endlessly defer, impacting the daily life and productivity of a development team.

So what really is a fly?

These 'flies' can be remnants of expedited decisions, quick fixes, or postponed updates that we plan to address "later."

They are the exceptions we temporarily accept amid pressing deadlines and shifting priorities. Think of them as minor design flaws, instances of non-optimal code, lingering security patches, or even outdated documentation – each representing a small portion of technical debt that accrues interest over time.

Though technical debt is an inevitable part of the development process, not all of it buzzes with the same intensity. Nonetheless, just as leaving food out too long invites trouble, allowing these 'flies' to loiter in your tech environment can lead to compounded challenges. They might seem trivial, easy to swat away or ignore, yet in a swarm, they can diminish a team's velocity and the overall quality of a project.

Addressing the 'flies' is not just about maintaining cleanliness; it's an exercise in vigilance and discipline. It means regularly revisiting and revising our architectural and developmental choices to ensure they continue to serve our evolving goals effectively. By doing so, we keep the 'flies' at bay, allowing our projects to thrive in a healthy, high-performing ecosystem.

Story Time

My favorite story about a fly starts with a company that was using Python Django to build their core product. Year after year, feature after feature flies flocked to this one particular area.

“We can’t upgrade Django, we forked it 9 years ago and it would be too hard to bring those required changes forward.“

We believed this fly to be true for 9 years - the Django version remained unpatched, insecure, and people began saying “well if we could upgrade Django, my feature wouldn’t be so hard to implement and it would take me half the time”

Eventually someone investigated the root cause of the fly. The forked version of Django. It was indeed forked, and the only change - the README.md file.

The fly pointed to the mess part of the project that didn’t get its fair share of the interest payments.

Swatting the Flies

Flies are symptoms of technical debt with a juicy rotten root cause.

Listening is often the first step to finding “flies”. You’ll hear them buzzing around the office, sometimes they die out on their own, sometimes you find one stuck on the break room ceiling holding on for dear life, they’re around every tech company in varying amounts.

Typically, they are less disruptive than a rumor about your company's super-secret upcoming merger but also deemed less critical than the slated features and sensitive tech debt looming overhead, right? Maybe not.

Sure, it's all tech debt, and we do pay it down now and again, but why should we prioritize an issue simply because it's irritating someone?

It's because a fly symbolizes a more significant underlying issue.

Flies are drawn to places where they can lay their eggs—after all, the goal of a fly is to produce more flies, right? Ask yourself: Why is this part of our technical line of credit attracting flies? From my experience, it’s often due to two main issues: a lack of resources or a gap in skills.

There is usually an outdated technology or practice in the mix that a gap in skills or a lack of resources has compounded. This overlooked segment of our 'technical line of credit' isn't receiving its due care.

Such laments hallmark the emergence of a 'fly': a problem that may have once seemed manageable or insignificant, yet with time, it matures to stifle efforts and hinder progress.

When team members suggest that tasks "would be simpler if not for X," or "could be done in half the time if Y were resolved," it’s time to take action. These expressions of frustration are symptomatic, and akin to an exterminator homing in on the problem, developers and project managers need to track these 'flies' to their source.

I once nurtured a fly of my own, making excuses for why I hadn't upgraded our systems by saying,

“It might be easier to re-write the whole project then upgrade our forked version of Django” - Will

yeah. flies lay eggs.

The key is to find them before they hatch. Foster a culture of transparent communication within your team from all directions. Don't shy away from confronting flies; recognize that they’re indicative of wider issues.

Experiencing 'flies' can be a tough lesson, as many of us can attest.

I’d be eager to hear how you've faced off with these nuisances—share your 'fly' tales in the comments below.

While building findyourfivepm.com, I found myself knee-deep in the world of mapping services. The heart and soul of the site, the map, had to be more than just functional - it had to be a cheetah on steroids.

This map had to be the trifecta of digital cartography - it's like the age-old dilemma: cost, speed, quality. pick any two; however, two will not do. For I am not made of money, have no desire to pan around slowly, and I refuse to provide you a mediocre product.

After a thorough comparison of many map providers, Amazon Location Service emerged as the winner. But it had one drawback.

(I'm going to save myself some carpal tunnel syndrome and refer to Amazon Location Service as ALS from here on out.)

When my map is zipping around at warp speed, it needs to serve up image tiles faster than a sushi chef on a Saturday night. Every millisecond is a race against time. This became glaringly obvious during a quick trip from Sydney to the US. I had initially set up my ALS-based map in Sydney (ap-southeast-2), and upon touching down in the US, I was eager to show my unimpressed friends this beautiful and exceptionally useful website. But the map was moving so fast that I couldn’t even see any of the tiles until it stopped. What had happened? This worked perfectly at my house in Sydney!

My latency to the mapping service was a mere 350ms, which might not sound like much, but in the world of real-time map tile delivery, it's a lifetime.

Subscribe now

Share One Twenty Three Cloud Street

What doesn’t work

Using Amazon Location Services, you can create location-enabled, and mapping applications without the operational burden and complexity of building and maintaining your own location-based services. Maintaining your own mapping service is hard, needs constant updating and takes up a ton of disk space. https://github.com/maptiler/tileserver-gl While Amazon Location Services exists in 13 (as of December 2023) different AWS regions and 2 GovCloud regions, there is no native way to direct end users to use the API in the region closest to them.

So let’s first look at some obvious but non-functional options.

• What if I simply put a Route53 Latency Based Routing DNS entry for each API endpoint provided by AWS (geo.us-east-1.amazonaws.com, geo.ap-southeast-2.amazonaws.com, etc) to be mapped to geo.mydomain.com ?

  • SSL validation will fail on the client side because Amazon Location Services provides the SSL certificate and it won’t match geo.mydomain.com

• What if I put client side logic into the website to select the closest region and then direct the user there??

  • Amazon Location Service uses either IAM, Cognito, or API Keys to facilitate access. If you wanted to have un-authenticated users access your location service, you would need to setup an API Key and enable CORS, or anyone could use your API to load maps on their services (expensive for you, cheap for them) You also need a way to manage all those API key in your client side and rotating those keys would become a fun task.

What does work

Let me show you why and how it works.

In each region - I have created a copy of the same architecture.

• ALS with an Esri Navigation style map, and an API Key (you can’t use the same API keys across multiple regions and you can’t use Cloudformation to make API Keys for ALS!!)

• API-Gateway with a request path proxy rule that injects the right API key for this region into the query string, and a certificate for geo.findyourfivepm.com

In Route53 I have created a set of A Records for geo.findyourfivepm.com that uses latency based routing to automatically direct you to the closest region. Additionally I have health checks on these endpoints. If a region is having a bad day, I can send you over to one that isn’t, keeping those 5pm moments flowing.

Now I just need to update my javascript to use this new technique properly.

Above is what I was using to call ALS on a single region. I provide the API Key, the region name, the map name and then combine those into a style URL.

Now I only need to define the mapName. Route53 will direct the user to the location with the least latency between them and the request will land on API-Gateway which will inject ?key=”v1.public.abcdefg123…” into the end of the URL string and pass the request onto ALS.

Let’s explore this in more depth.

🎵Simply having a wonderful mapping time🎵

Remember when I mentioned that I care about cost?

Let’s review how this changes the price of the solution.

For HTTP APIs, the API Gateway free tier includes one million API calls per month for up to 12 months. (1)

Now if I became really popular (probably because you’re going to share this post and visit findyourfivepm.com every day?) how much would it cost me?

Share

In the last 30 days I’ve had 60.8k tiles served and 941 page views/cities returned.

I get about 400k map tiles served every 6months. You can also see in the graph below the period of time where I was only using ap-southeast-2 to serve my map tiles. When I switched to the architecture in this post, you can see both how the number of views increased, but also from which locations they originate. Most of my current user base is closer to us-east-2 than not, and traffic in eu-central-1 is picking up as well.

So pricing. It could get expensive, if I had many million API calls, and after you make my website popular, and I have 100 million tiles served a month will I be paying $3.50/million * 99 Million = $346/month just for this localized routing via API Gateway?

No, I will not and here’s why.

Lets inspect the initial request to geo.findyourfivepm.com.

Our call goes out to the URL of my latency based Route53 query and is handled by our API Gateway to:

geo.findyourfivepm.com/maps/v0/maps/Geo-5pm-Map/style-descriptor

We then get back this style descriptor document. Let’s inspect that.

Fantastic. We sent a single request thru the API-Gateway, it returned a style document from Amazon Location Service and in that style document we find paths directly to ALS (and not our API-Gateway!) with our API key pre-injected. No further requests will be sent to this API-Gateway endpoint as this document configures the local client to make calls directly to these endpoints on ALS. This should be free for me to use up to about 1 million page views a month. The ratio of page views to tile’s served is currently 60800 Tiles / 941 cities viewed or about 64.6 tiles served per city. I will be more concerned about the cost of serving ALS tiles well before I begin to pay anything for my global API-Gateway setup.

Let’s do that math.

If I have the maximum free tier usage of API-Gateway at 1,000,000 requests in a month thats 64,000,000 tiles served. I am using Esri maps, which is a commercial data provider so I pay $0.040 per 1,000 tiles. This is about $2,560 for 64 million tiles and $0.00 for 1 million requests to API-Gateway. While I hope that if my site is that popular I can make some return on investment, I will be much more concerned about the tiles than the efficient, local and speedy way I can serve them to you.

Let’s look at the other costs.

I have an ACM certificate in each region for geo.findyourfivepm.com but all ACM certs are free.

There’s one more cost to consider and it might surprise you. Route53.

You might think I’m paying $0.20/million queries more than a standard query for a latency based query but they’re actually free. These are alias queries and alias queries are all FREE.

Free is good and if my math is right, this solution costs no more than if all my map tiles are served from a single region - unless you finally make me popular and start sending me over a million page views a month.

Share

Subscribe now

This solution is ingeniously simple yet effective if i do say so myself. By replicating the Amazon Location Service setup across various regions(I’m using 7) and utilizing API-Gateway with custom proxy rules, I can dynamically inject the correct regional API key. All the while, Route53's latency-based routing seamlessly directs users to the closest endpoint, ensuring optimal performance.

This architecture not only solved the latency issue but also proved to be cost-efficient. Leveraging API Gateway's free tier and the fact that additional requests are handled directly by ALS!

I hope you’ve found this post while working on your own mapping solution and that it can guide you towards a fast, good and cheap win.

Try it for yourself!

Take a look at https://github.com/jeeshofone/LocationServiceGateway for my full guide on deploying this solution yourself. You’ll find fully functional Cloudformation Templates and deployment scripts. Happy Mapping!

Part 0 - Introduction
Part 1 - Failing Fast (Building with prompts)
Part 2 - MVP and Deployments
Part 3 - MVPolish
Part 4 - So what. *You are Here 💁‍♀️

As we stand at the precipice of conclusion, looking back over the time-zones we've traversed together, it's evident that our journey was a slow success. With findyourfivepm.com, we've learned that the sun never truly sets; it simply shifts, reminding us that the end of one adventure is the dawn “5PM” of another.

The Good

Our expedition was illuminated by the brilliance of ChatGPT 4, a technological marvel a chatbot that often turned code into conversation and problems into puzzles waiting needing to be solved. In those moments when the AI's responses clicked perfectly into place, it felt as if we had harnessed a piece of the future, a sliver of some advanced civilization's normality. I honestly think we’re a long way away from a simple conversation turning into a production ready product.

The Not-So-Good

But as any seasoned traveler architect knows, “it depends”. Our AI companion, for all its prowess, sometimes led us astray with misinterpretations and oversights that only a any human eye could catch and correct. These were the times when the project felt Sisyphean, the summit always just out of reach, the sun dipping below the horizon just as we thought it should be 5pm forever.

The Verdict

There is no short-cut for experience. ChatGPT 4, for all the times it hastened our pace, it also reminded forced us to slow down. Code completion AI, like CodeWhisperer, and even GitHub Copilot, those are ready for prime-time. Filling in a line or two, or twelve in an existing well thought out project is less of an art, and more of an auto-complete bot. Having a conversation with an LLM and pretending along with them that they're the sole developer, engineer, and architect on a project, that’s insane — delightfully so, but insane nonetheless.

The Conclusion

And now, as the final lines are written and the last echoes of our digital odyssey fade, we understand that "The Sun Never Sets on 5pm" is more than a title; it's a truth. Our efforts, our passions, our relentless pursuit of what's next — these are the constants, the never-ending day of our world.

The project findyourfivepm.com stands not just as a product but as a testament to the journey of discovery. It is a reminder that we should take time to relax from our work and find what makes us happy. Whether we leave the office at 5pm, 5am or any time in-between - it's only right that we find what makes us happy and take a moment each day to relax, learn and discover again what makes us tick.

Thank you for walking reading this path with me. May your Five PM be forever.

Oh and don’t use ChatGPT, or any another LLM for that matter, as the only tool in your belt - there is no shortcut for experience and you’ll just deprive yourself of an opportunity to learn.

Part 0 - Introduction
Part 1 - Failing Fast (Building with prompts)
Part 2 - MVP and Deployments
Part 3 - MVPolish *You are Here 💁‍♀️
Part 4 - So what.

Procrastination is important (a quick tangent)

Procrastination, oddly enough, serves as my guiding compass. Many see it as a vice, an impediment to productivity, but for me, it's an indicator, a sign that I'm on the brink of something truly meaningful. The gravitational pull towards distraction isn’t a sign of laziness, but a manifestation of the underlying tension of creation. When that urge to delay kicks in, it’s often because I'm subconsciously recognizing the magnitude of what lies ahead.

The initial 80% of a task can be deceiving. It gives a false sense of accomplishment, luring one into thinking that the bulk of the work is done. But that’s the easy part. It’s the linear path, the expected journey. The final 20%? That’s where the real magic happens. That’s where the nuances come alive, the finesse is applied, and the true essence of the creation is realized. And that's terrifying. Because it demands vulnerability, authenticity, and a willingness to confront potential failure head-on.

Procrastination, in this light, isn't a weakness; it’s an emotional barometer. It measures the depth of investment in a project. When I feel that familiar itch to defer, to delay, to put off, it's my internal alarm sounding off, reminding me that I'm on the cusp of birthing something genuinely remarkable.

So, instead of chastising myself for these moments of hesitancy, I've learned to embrace them. I take a step back, reflect, and understand the root of the resistance. Is it fear of criticism? The daunting nature of perfection? Or the weight of expectation? Once identified, it becomes easier to navigate through the fog of uncertainty.

In a world that often measures worth by worthless relentless productivity, I’ve found solace in understanding my patterns of procrastination. It’s become a beacon, illuminating the projects and pursuits that truly matter. And in that clarity, I find the strength to push past the inertia and dive headfirst into that thrilling, daunting, and transformative final stretch.

The final stretch?

If you haven’t accidentally discovered it already from the numerous hints I’ve been dropping this website is called FindYour5pm.com and below is the final(?) version. It’s time for me to cheat, one final time.

To achieve this marvelous design - I paid a guy on fiverr.com a bit more than $5AUD to get me setup. ChatGPT wasn’t cutting it. I could never get it to do the layout properly, or work with multiple files containing different parts of the code.

Sometimes it felt like this was because I couldn’t keep enough code in the available context space; sometimes I felt like it just wasn’t up to the task. What I got from my new friend on Fiverr was basically a website template with a full width map and what I assume is the right folder structure and layout for a front end.

I’ve spent the last 15 years working on backend infrastructure, both on-premise and in the AWS Cloud and have entirely ignored everything there is to know about front-end. I’ve found my talents best placed in the specialities of architecture, networking, containers and event driven systems. Also I bought a logo - sue me.

That logo? It might sound trivial, but it's a symbol. A symbol of my willingness to venture outside my comfort zone. It's not just about aesthetics; it's about embracing change, adapting, and realizing that even a backend guy can appreciate the beauty of a well-designed interface. So here's to new beginnings, to taking leaps, and to the incredible Fiverr talents that help bridge the gaps in our knowledge.

Now back to the story.

What’s Next?

The broad strokes of my architecture are completed. I have 2 primary tasks left. Put a bit of work into my new frontend and unfortunately add some more backend services.

“What backend service is missing?” You might ask - and rightly so.

After all, our PoC had every component in the “final” state of the website, but not as pretty, functional or inspiring as the quality you come to expect from me. If you remember from Part 2 - I can’t use OpenStreet maps here. It’s against the ToS and it’s going to rate-limit my users. It’s time to evaluate other map providers:

• Self-Hosted OpenStreet Maps

• Google Maps - Tiles API

• Amazon Location Service

OpenStreet Maps helpfully provides a list of commercial tile servers that I can use! https://wiki.openstreetmap.org/wiki/Tile_servers - Lets evaluate them!

To use these tile servers, I would need to host an always on backend that contained the entire planet worth of tiles. This sounds really fun!

I have done this in the past for other projects that I won’t get into here and the problem with this system is that I have to maintain all updates to the tiles, and I have to host an always on service regardless of how many users find my page. This will significantly increase my costs and not allow my website to scale dynamically in price with user volume.

Instead, I want someone else to host this non-trivial component for me. Thankfully there are numerous Raster and Vector Tile providers that do just this!

https://wiki.openstreetmap.org/wiki/Raster_tile_providers

https://wiki.openstreetmap.org/wiki/Vector_tiles#Providers

They all cost money. I’m ok with paying; however, most of the plans are not pay-as-you-go, but instead use an upfront plan with X number of tiles available to consume in a month. I don’t want to keep track of consumed tiles and work out how to update my plan accordingly and the prices are kinda expensive.

So what about Google Maps Tiles API?
This is pay as you go pricing!

Google Maps Tiles API is priced at $0.60 USD per 1000 tiles (up to 1,000,000 tiles) per month. and then $0.48 USD per 1000 tiles from 1,000,001 to 5,000,000 tiles. (Before becoming famous from writing this blog post I used about 128,000 tiles per month) 1,000,000 tiles would cost $0.60 * 1,000 = $600USD/month. (128,000 tiles would cost $76/month)

That’s looking better - since right now hardly anyone uses this site, and maybe I’ll have a good conversion rate on my silly designs I sell on the site - this will very much be my biggest cost!

I work at AWS. Let’s checkout Amazon Location Service then?

If I use a Commercial Data provider, I’ll pay $0.04 USD per 1000 tiles.

Updating my math this becomes:

$40 per million tiles and $5.12 for 128k tiles! Sold. (15x cheaper than Google Maps)

I also get a choice of ESRI, HERE, OpenStreetMaps, and GrabMaps (among others) and I can deploy this in my infrastructure as code templates. This is by far the cheapest and easiest, pay-as you go solution.

As of this writing(2023-11-02) there is partial support for Amazon Location Services in Cloudformation. I am able to deploy a map service but cannot deploy an API key for the service.

I’ve discovered a way (and as far as I’m concerned this “innovation” is well outside the scope of asking ChatGPT for help) to put Amazon Location Service in each relevant AWS region and direct a user to the closet region while injecting the correct API key for that region into the request without putting the API key in the front end javascript. This will get its own blog post later(if it’s later enough, this is now a link!) as I hope I can help others with this task since it’s off topic for this post. Here’s a glimpse of what that looks like:

I ask ChatGPT to help me implement this. Amazon Location services has multiple different authentication methods. You can (launched after GPT4’s knowledge cut off) use an API Key, IAM or Cognito to provide auth. We’re going to be using an API Key.

Yeah, you can’t do imports on a client side javascript package (without using something like Browserify to do 🪄✨magic✨) as far as I can tell at least.

Yeah this is not working - we already lost the context that I can’t really do this import in my client side javascript. This is very frustrating and an area where an LLM trained on my code base would really help me out. Too bad this isn’t a thing. Imagine true pair-programming with an AI that has all the context of your repo. https://aws.amazon.com/codewhisperer/customize/

After many back and forth’s and lots of trial and error at the request of ChatGPT, I end up with this error to troubleshoot.

What I’m seeing is a blank map with no tiles. There is a sense of motion in void as we pan and zoom from one location to another; however, all requests to the tile endpoint are getting error messages.

Vector and Raster maps. Let’s talk about the differences quickly.

Raster maps are composed of pixels, each representing a specific geographic area, much like a digital photograph. These maps are essentially images where each pixel has a defined geographical position. They are best suited for intricate visual representations, such as satellite imagery or detailed terrain maps. Their main limitation is that as you zoom in, the image can become pixelated or require additional tiles at higher resolutions.

In contrast, vector maps are made up of points, lines, and polygons that represent real-world features like roads, rivers, and boundaries. The data in vector maps is mathematically defined, allowing for smooth scaling and crisp representations at any zoom level. These maps are often more interactive, as individual elements can be easily highlighted, colored, or labeled. However, they may require more computational power to render, especially when displaying complex data.

While raster maps offer a fixed visual, vector maps are more dynamic and customizable, making them ideal for interactive web applications or when data overlay and manipulation are required.

Choosing between the two often depends on the application's needs: the level of detail required, the degree of interactivity, and the available computational resources. For this project I’m using a Vector map since its fewer map tiles and in my opinion fits into the design language of the page more eloquently.

After some back and forth with ChatGPT I we end up using both MapLibreGL and Leaflet together to handle the map. Here is what that looks like:

I then work through a few minor issues with the integration of MapLibre GL and Leaflet (taking maybe 10-15 messages) and I’m finally done with migrating the map to Amazon Location Service.

Yet another task that should have taken less than half this much time had I done it alone.

E-Commerce

If you remember back to Part 0:

Well right now all I have is a grid of static pictures with a box for text under them. That is neither slick nor integrated. Lets polish.

Together, ChatGPT and I go thru the same process as before. But let's skip the technical back-and-forth this time?

I have a few products on Redbubble - I want to show 6 random products on my page, this is what the HTML currently looks like - write a javascript function that puts them here - help me make a Lambda function that reads a json file of all the products and returns it via an API-Gateway endpoint - make the javascript function call the endpoint and show the 6 products that are returned.

What's important is the result: a living, breathing front-end that now cycles through products with a vigor that static images could never achieve. The once bland grid has evolved into a carousel of creativity, a vibrant display that beckons the onlooker with the promise of novelty and excitement site-supporting-revenue with each visitor.

I, on the other hand, am happy. The satisfaction of seeing this in action is twofold: the visible, tangible product that speaks to the efforts of development shouting at a computer for months on end, and the invisible, immeasurable growth frustration that comes from tackling a challenge head-on. This part won't be seen by the users, but it doesn't diminish its value to me.

If anything, it's the secret ingredient that makes the visible elements that much more compelling.

Subscribe now

Share

As we near the end of this blog series, I can't help but reflect on the journey. From the humble beginnings of an idea to the intricate web of services that now comprise FindYour5pm.com, it's been a trek through unknown territories, through valleys of doubt, and over peaks of achievement and I’m tired of yelling at the machine.

The next and final installment, will pivot from the 'how' and 'why' and into the “so what?”

But for now, allow yourself a moment of procrastination — it might just lead you to your next big idea.

Manifest

Part 0 - Introduction
Part 1 - Failing Fast (Building with prompts)
Part 2 - MVP and Deployments
Part 3 - MVPolish *You are Here 💁‍♀️
Part 4 - So what.

Part 0 - Introduction
Part 1 - Failing Fast (Building with prompts)
Part 2 - MVP and Deployments *You are Here 💁‍♀️
Part 3 - MVPolish
Part 4 - So what.

re: Cap

We just got the backend Lambda function to return a valid JSON object and now it’s time to build out an MVP. We’re only using GPT-4 to write code and it’s taking way more questions than it really should to build what we need.

Here’s what we have so far:

Here’s where we’re going:

I’m calling this practice LLM-Driven-Development. It's not exactly a virgin birth new kind of an idea, but I'm pushing it to the brink, to the edge of the precipice where people with too much time on their hands the daring, and reckless tread.

When used to accelerate your own efforts, LLM’s in a development setting work very well! Take a gander at CodeWhisperer. Its been forged in the fires of this very crucible, purpose-built for this kind of heavy lifting.

But let's not get ahead of ourselves. As of the current date, mid-2023, with the state of LLMs as they stand, it's a bit of a stretch, a leap of faith, to anticipate an entire project to spontaneously spring forth from the ether.

It's like expecting a perfect Coq au Vin to materialize out of thin air - you need the right ingredients, the right chef, and the right amount of time.

So, let's roll up our sleeves, get our hands prompts dirty, and see what we can cook up with LLM-Driven-Development.

Frontend

Now let’s try making an MVP frontend served by API-Gateway. API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale, allowing communication between an application and AWS services using HTTP, REST, and WebSocket protocols.

GPT-4 hands me a lengthy guide for setting up an API-Gateway via the AWS console. Until now, Infrastructure as Code (IaC) hasn't been employed, and truth be told, we've already rolled past the second third weekend.

As of this writing, GPT-4 has a limit of 25 messages every three hours, and I'm not going to resort to GPT-3.5. (Hi this is Will, from the future, I started writing this back in late March of 2023. That limit is gone now and also Code Interpreter Exists; additionally, I’ve moved on to using AWS Foundational Models, CodeWhisperer, and Bedrock for the majority of my LLM Driven Development exploration)

I'm sure I'd face a chorus of remarks about the superiority of GPT-4 over 3.5 and how “this isn’t a test of the latest version” and blah.. blah.. blah so I waited the 3 hours to get 25 more questions out so I can keep you happy.

In the mix of tasks, I've also requested GPT-4 to create a basic index.html file for backend testing purposes. After wrestling with the 'float vs decimal' conundrum again for a good ten messages, we've finally arrived at our first version of the index.html. Let's take a look at our initial test page.

You can try this page out yourself here:
https://findyourfivepm.com/index-original.html

I’ve kept the future API backwards compatible.

Feeling bold, I decide to spice things up. I request the integration of a theme and a map in the page that navigates to the specified coordinates. Immediately, GPT-4 responds with a barrage of CSS and JavaScript.

Feeling confident, I ask GPT-4 to add a unique background effect, making it look like it's raining 🍺 emojis.

Remarkable.
This page is also still online here: https://findyourfivepm.com/index-old.html

Two consecutive commands, and the execution is spot on. Surprisingly, not only has GPT-4 implemented OpenStreetMaps with Leaflet aptly without any explicit instruction to use those tools, but the page also presents itself rather nicely. I’ve officially reached the 'proof of concept' or MVP stage. Honestly, I'm rather impressed.

It's been two three weekends of collaborative coding with GPT-4, and we've already barely, just barely, crafted a compelling webpage.

Our Lambda is still pretty, pretty slow. Let’s see how Chat GPT can improve it

Improving the Lambda

Don’t worry too much about the first line “I don’t think I want to do that:” - I was experimenting with Step Functions and ultimately decided to keep the scope simpler. I’ve asked for performance improvements for the lambda.

You’ve probably noticed I keep pasting in the code. It’s annoying, and necessary.

I constantly refresh the context of the LLM with the content of the code we’re talking about - I also include details about how the data is structured in the DynamoDB table - this really seems to help when debugging as it is likely that the code I have provided previously is no longer fully contained in the context of my messages.

Oh yeah? How does Context Length work? Glad you asked, dear reader. Glad you asked.

Imagine that you and your friend go to dinner. You start the meal talking about one thing and then as the meal progresses, so does your conversation. You discuss many different topics and without a defined agenda you both end up just talking about what was on your mind. By the end of the dinner you probably can’t remember verbatim what was said but you can remember how you felt, many important selective details, and a broad idea of how the conversation went. Unless you’re Mike Ross then context length functions very similarly to how humans do.

What is Context Length?

In simple terms, context length refers to the amount of recent information a language model can “remember” or “consider” during its current operation. This information could be a couple of sentences, a full paragraph, or even more.

The longer the chatbot's memory, the better it can understand and respond to multi-sentence queries, ensuring more satisfying user interactions.

Introducing Langchain

As we rely more on chatbots and LLMs, ensuring they have a continuous understanding of the conversation becomes crucial. Enter 'langchain' - a conceptual tool that chains the dialogue, making sure the model remains updated with the ongoing context.

Langchain ensures that even when a conversation spans over multiple turns, the chatbot can retain the context. It's like giving the chatbot a tool to revisit past conversations and stay updated, allowing it to provide answers that are not just relevant to the last sentence but to the entire conversation.

Tips for Keeping the Chatbot Current with Context

1. Be Explicit: While LLMs are designed to understand context, sometimes being more explicit in your statements can help.

2. Chunk Information: If you're providing a lot of details, try breaking it into smaller, related chunks. This way, you allow the chatbot to process each bit of information more efficiently.

3. Recap Occasionally: Just like in a human conversation, recapping can be beneficial.

Now Back to the Lambda.

The very first item in the list is really the only one that is going to help us with performance. We should move things outside the lambda handler if possible. This won’t help with cold starts, but it will help with each subsequent invocation. For instance, if you’re using an SDK that needs to be initialized just once, like a database connection, or need to load some resources like machine learning models, keeping this kind of code outside the handler improves performance.

The real laziest performance gain comes from Lambda Power Tuning.

AWS Lambda Power Tuning is an optimization tool for balancing the speed and cost of Lambda functions by automating memory allocation tests. It employs AWS Step Functions to run multiple function versions concurrently at different memory allocations, measuring performance through live HTTP calls and SDK interactions. Graphing the results helps visualize performance and cost trade-offs, aiding in better configuration decisions.

Here are my results.

This is because Lambda's resources (CPU, memory, disk, and network) are allocated in proportion to each other. Therefore, by increasing the memory size, you not only get more memory but also a proportional increase in CPU power, which can lead to faster execution times for CPU-bound tasks. I highly recommend using Lambda Power Tuning on all your lambda functions especially after a big update.

Now that our main Lambda function is working well; lets move on to IaC

IaC - CloudFormation

Well, I’ve now asked my inaugural IaC question. Remember when I talked to you about refreshing the context to ensure maximum results? The resulting CloudFormation template thinks the lambda is written in Node and forgot that way earlier in the project, literally 3-4 weeks ago, I suggested we use AWS CodeBuild and CodePipeline. We’re now using GitHub Actions. I spend a few messages refreshing ChatGPT and telling it that we’re using Python (and pasting in the relevant code) and that I’d like to use CodeBuild and CodePipeline with my CloudFormation template.

OK. That got us somewhere, the first CloudFormation template now has:

1. LambdaRole for IAM permissions

2. CityFinderLambda - the actual Lambda function

3. CitiesTable - in DynamoDB

I also have a basic buildspec.yaml that will install our requirements for the lambda and deploy the CloudFormation template.

I ask to add the S3 Bucket for packaged CloudFormation templates and the API Gateway.

This does not add the requested API-Gateway explicitly. Instead it defines the API-Gateway as part of the AWS::Serverless::Function resource within the Events section. This is perfectly valid, but does not do what I want for the future maintainability of this project, as I want the flexibility to configure the API-Gateway in other ways. A simple follow up message and this is cleared up.

I spend the next 2 weeks here.

I failed flailed around with a stupid idea. In my index.html I have a section that looks like this

  <script>
    function createRain() {
      const numberOfDrops = 100;
      const rainContainer = document.querySelector(".rain");
  
      for (let i = 0; i < numberOfDrops; i++) {
        const drop = document.createElement("span");
        drop.innerHTML = "🍺";
        drop.classList.add("drop");
        drop.style.left = Math.random() * 100 + "vw";
        drop.style.animationDuration = Math.random() * 6 + 4 + "s";
        drop.style.animationDelay = Math.random() * 2 + "s";
        rainContainer.appendChild(drop);
      }
    }
  
    document.addEventListener("DOMContentLoaded", function () {
      const apiUrl = "https://abcde12345.execute-api.ap-southeast-2.amazonaws.com/prod/find_location";
      createRain();
  
      // Initialize the map
      const map = L.map("map").setView([0, 0], 5);
      L.tileLayer("https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png", {
        attribution: '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors',
      }).addTo(map);
  
    // Fetch city information and update the map and text
    fetch(apiUrl)
      .then((response) => response.json())
      .then((data) => {
        const { city, local_time, latitude, longitude, timezone } = data;
        const initialZoomLevel = 19;
        const finalZoomLevel = 9;

        map.setView([latitude, longitude], initialZoomLevel);
        const marker = L.marker([latitude, longitude]).addTo(map);
        marker.bindPopup(`<b>${city}</b><br>Local time: ${local_time}`).openPopup();
        document.getElementById("city-info").innerText = `City: ${city}, Timezone: ${timezone}, Local Time: ${local_time}`;

        // Smooth zoom out using flyTo method
        map.flyTo([latitude, longitude], finalZoomLevel, { duration: 10 });
      })
      .catch((error) => {
        console.error("Error fetching data:", error);
      });
  });
</script>

The part we’re concerned with is:

    document.addEventListener("DOMContentLoaded", function () {
      const apiUrl = "https://abcde12345.execute-api.ap-southeast-2.amazonaws.com/prod/find_location";

Since we’re deploying as IaC into a blank account, I don’t know the URL of the API-Gateway before the index.html file exists. I’m not really interested in using parameter store or doing anything correctly fancy with this part since this is likely to never change once deployed; but I do something stupid anyway. Since I have free labour with ChatGPT, I setup the audacious goal of having my CloudFormation Stack change this code (not the build spec, which would have been SOOO much easier) at deployment time. This is where I spend over a week. Completely stalled out.

ChatGPT leads me into a world many know of but few love. CloudFormation Custom Resources. More on this later? No, it’s boring. I’ll wrap it up here.

I end up getting my stack in a stuck state for multiple hours at a time. There is nothing you can do if your stack is stuck. I even ended up paying for AWS Support to tell me so.

I abandon the idea.

After I’m done with my migraine inducing journey, I keep going with actual development.

I spend the next weekend updating the CloudFormation stack. Getting CodeBuild and CodePipeline Setup and dialing in the IAM permissions via Trial and Error. I paste all deployment errors into ChatGPT and make only its suggested changes.

This is infuriating for me. I’ve been using CloudFormation for years, and I have the issue right in front of me. I know the answer and I’m powerless to do anything about it. Yet I watch as we brute force our way thru solutions that are simple changes.

Here is a montage - there are at least 100 more messages like this.

That last one was particularly frustrating! There are over 100 more messages involved in getting the IaC setup properly. This took me over 2 weeks.

With a completed CloudFormation Stack deployed into the main account I also get to work at setting up a second account for my Staging environment.

ChatGPT makes a few minor changes to my page and we’re now looking like we have an MVP. The biggest change is a custom javascript function that can search poorly Wikipedia for the most relevant article of the current location. (more on this later)

We might as well teach our viewer something while they look for an excuse to drink a beer. Surprisingly this only takes 2 12 ask’s for ChatGPT to get it right.

You can still view this version of the page here: https://findyourfivepm.com/index-old.html

Fun fact - Rikitea is the subject of my first GitHub Issue in this project. During sometimes of the year, it’s actually the only place in the world with a population over 500 observing 5pm! I thought this was a bug, or shitty data, but it’s actually a real phenomenon.

They have the world’s loneliest time zone at UTC-9. You can read more about that here: https://findyourfivepm.com/blog/rikitea-the-sometimes-exclusive-celebrant-of-five-pm-at-utc-9.html

I should probably close my GitHub Issue…

Current Status

I have now created:

1. A full CloudFormation Stack deploying automatically in 2 AWS accounts, Production and Staging

2. CityFinder Lambda Serving the API Request to get the Current City at 5pm

3. A DynamoDB table with over 198,000 cities and their locations

4. A Hydrate Lambda that gets data from http://www.geonames.org and hydrates the DynamoDB table

5. An API-Gatway to interface the CityFinder Lambda.

6. A Basic HTML front end that allows me to visualize the results.

7. All relevant IAM Roles and Permissions

8. Custom Domain Names for the service with CloudFront serving SSL certificates and API-Gateway serving Cities and in the production environment using CORS and SSL.

It's now time to polish this up a bit. I am using the “free” version of OpenStreet Maps. I can’t use this in production at scale, it’s against the terms of service and my users if I ever get any will get rate-limited. I also found that after spending a lot of time working with ChatGPT that it failed to get any better at web page creation. Organizing visual elements and making something appealing for a user to look at is well beyond its current capabilities.

It’s time to take our MVP to the next level.

See you in Part 3.

Manifest

Part 0 - Introduction
Part 1 - Failing Fast (Building with prompts)
Part 2 - MVP and Deployments *You are Here 💁‍♀️
Part 3 - MVPolish
Part 4 - So what.

Share

Subscribe now

I’m sure you want to see each and every prompt; but there are hundreds if not thousands of prompts and I couldn’t possibly paste them here…. oh, ChatGPT has link sharing - as a special offer for paid subscribers, you can access the enthralling conversations below.

Read more

Part 0 - Introduction
Part 1 - Failing Fast (Building with prompts) *You are Here 💁‍♀️
Part 2 - MVP and Deployments
Part 3 - MVPolish
Part 4 - So what.

be sure to read Part 0 for the “why”

Getting Started

(For those who prefer not to read the text in the images, rest assured, I'll continue to provide a summary of the progress after each visual update.)

My initial goal is to make a lambda function that will use a timezone database (which I did not specify any details about), compare the current time to the database and return the location, longitude and latitude of the result. I first try to ask GPT-4 to do this like I was writing comments in a python file.

First Fail Fast

Ah, our initial query is also a pivotal realization in our AI-powered odyssey. Indeed, approaching tasks with LLMs like GPT-4 requires a different mindset. Asking too much in one go is a common pitfall, and it's one that I've just encountered firsthand.

Helpfully, ChatGPT has used nominatim which is a nice library - however, its purpose is to find places by their name or address and turn them into GPS Coordinates like “New York” → 40°42′46″N 74°00′22″W. It can also take a longitude and latitude and turn them into the nearest address like 36°49′25.6″S 139°51′47.5″E → 17 Princes Hwy, Kingston SE SA 5275 Australia.

I want to turn the time into a place and I don’t know where that place is or what its address is.

To attempt this - ChatGPT has taken the current time, iterated over all the timezones provided by the pytz library (converting to UTC), finds the (doesn’t handle the case were there are multiple “named” timezones where its 5pm) singular timezone where its 5pm and then finds the singular longitude and latitude of the timezone. Can you even do that?

I’ve obviously not explained myself well.

So, the key is to break tasks into bite-sized, ordered steps. This approach mitigates reduces the chance of GPT-4 'hallucinating' - an AI's tendency to generate outputs that are coherent but detached from reality or the provided data. Let's recalibrate and proceed with this refined approach.

Fixing the first error

Remember - I can’t test this code, yet, I know it's wrong. I just told you it's wrong too - but I must push on since I need to give ChatGPT the error the code generates and allow it to make changes.

I quickly get an error - after those initial questions we have the following code for the inaugural lambda function:

import datetime
import pytz
from geopy.geocoders import Nominatim
import json

def get_location_with_time_between_5_and_6():
    geolocator = Nominatim(user_agent="myGeocoder")

    current_utc_time = datetime.datetime.utcnow()
    for tz in pytz.all_timezones:
        try:
            local_time = current_utc_time.astimezone(pytz.timezone(tz))
            if local_time.hour == 17:  # Check if the local time is between 5 PM and 5:59 PM
                location = geolocator.timezone(tz)
                city_info = geolocator.reverse((location.latitude, location.longitude), language='en')
                city = city_info.raw.get('address', {}).get('city', '')
                return {
                    "latitude": location.latitude,
                    "longitude": location.longitude,
                    "timezone": tz,
                    "city": city
                }
        except Exception as e:
            pass

    return None

def lambda_handler(event, context):
    location_data = get_location_with_time_between_5_and_6()

    if location_data:
        response_body = {
            "latitude": location_data["latitude"],
            "longitude": location_data["longitude"],
            "timezone": location_data["timezone"],
            "city": location_data["city"],
        }
        status_code = 200
    else:
        response_body = {"error": "No location found where the time is between 5 PM and 5:59 PM."}
        status_code = 404

    return {
        "statusCode": status_code,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps(response_body),
    }

Finally something I can pass back to Chat GPT to iterate on - I passed the error into the prompt and the soon to be classic response “My apologies for the confusion…”

If you’re familiar with Lambda, you know that the lambda_handler function looks like this in python:

def lambda_handler(event, context):

It has been shown the door.

In its place, we're charting a less-traveled path - defining a custom lambda handler function name.

This decision, while it certainly dials up the complexity quotient, might seem like an exercise in masochism. Especially when we're also keeping a function named 'lambda_handler' in the code and not using it as the lambda handler, setting up a veritable obstacle course for future readability.

But sometimes, in the pursuit of innovation, conventional wisdom is tossed to the wind, and we are made to navigate the labyrinthine I hope you’re studying for the SAT intricacies of code. This is one of those times?

I update the lambda function to use the find_location.lambda_handler, as poorly prescribed, and move on.

Now, with two 'lambda_handlers' we push ahead. Changing gears, I contemplate simplifying our function. I've planned on using a DynamoDB table to store the list of cities and timezones, but the data source is still unclear. Instead of the DynamoDB table, could a Python library be the answer? I just need latitude, longitude, name, and timezone. Let's delve into Python's rich ecosystem and see if it already has what we need.

My lucky day. A pre-built library that does exactly what I need. I’m overjoyed with enthusiasm as I have GPT-4 update my lambda to use this newly discovered library. Why hadn’t it been suggested to me previously?

In this example, the get_cities_in_timezone function takes a target timezone as an argument and returns a list of cities in that timezone. The cities package provides a dictionary of cities and their related information, including the timezone.

I have it update the script to randomly select a city from the list and also return the longitude and latitude.

Another much longer function emerges. I test for the second time.

My test can’t find the cities module. ChatGPT claims that I’m facing compatibility issues between a Python package and Lambda?

That's unusual.

I've also spared you from reading quite a few interactions in which GPT-4 assisted me in packaging a zip file for Lambda deployment. It guided me through creating a virtual environment and installing a few packages - pytz, cities, and geopy.

Even with the best tools at our disposal, We still encounter stumbling blocks. It's all part of the journey, reminding us that problem-solving is an integral part of any coding expedition - being human or AI for that matter.

I’m pretty sure the problem is that there is no such thing as cities or citytimezones packages - but I’m not allowed to say that.

I break my first rule and strike gold

I had to break the rule.

I was stuck for at least 30 min and could not discover a way forward here without being suggestive. I hated doing this, but I needed wanted progress.

Instead ChatGPT suggests that we use the GeoNames Database! They curate lists of cities and timezones!

I check and the GeoNames Database is real! The provided URL works and this dataset has exactly what I need.

What’s more is that the output of the script parse_geoname.py is actually pretty good - Spoiler alert: by the end of the project I’m still using some of this code.

GPT-4 then suggests that I update my code for the lambda to use the resulting json file and query it. After 3 errors it starts returning something; although its slow (due to locally processing a rather large json file, but I’m not supposed to know this) typically 8-10 seconds to get an answer. I ask it how I can understand which parts are slow. It suggests I print the time it takes after each section to discover this from the logs:

print(f"Time taken to load cities: {time.time() - start_time:.2f} seconds")

I’m going to speed this part up a bit as there are a lot of prompts, back and forth’s and otherwise boring yet helpful interactions, over the course of this project I asked 443 questions in this chat alone! There’s not enough time and let’s be honest, you’re not interested enough to read all of those messages (if you are a paid subscriber you can read the whole chat log in the paid content section at the end of this post.)

• I ask if an AWS service can do this instead of time

  • Suggests X-Ray

• I ask to implement X-Ray

  • Implements X-Ray in the lambda

• I give it the logs from x-ray and CodeGuru Profile (which it implemented too)

  • It determines which areas in the functions take the longest (but outputs fictitious times)

• I ask if I can use DynamoDB to store the json instead - There I go again breaking the rules

  • Yes you can

• Write a script that can do that

  • Outputs Script !! using a Lambda Function!

  • Script uses Provisioned Capacity (more expensive than I’m willing to tolerate for a test so I ask it to use PAY_PER_REQUEST, I know, I know, I’m bad)

  • The Lambda function takes 10 minutes to run at the largest Lambda Size 🥵

• 8 different questions about why the table isn’t populating

  • 7 different wrong answers

  • “stop using floats” - ChatGPT - this works.

• 2 questions about failed hydration

  • track duplicate cities (there are none 😉 but I decide not to tell it this)

• 22 questions to update the Lambda to use the DynamoDB table instead of the json file

  • working now!

I remind it another 10 times I don’t want provisioned capacity and we have to fix the float issue again and again until it works (it kept either reverting the code or not implementing strings in the right places). And it’s still slow, maybe 10seconds? (Its slow because we’re doing table scans instead of using a GSI in our DynamoDB table)

Here is the code for the lambda at this stage:

def lambda_handler(event, context):
    all_timezones = pytz.all_timezones

    five_pm_timezones = [
        timezone for timezone in all_timezones
        if 17 <= datetime.now(pytz.timezone(timezone)).hour < 18
    ]

    random_timezone = random.choice(five_pm_timezones)

    try:
        response = table.scan(
            FilterExpression=Key('timezone').eq(random_timezone)
        )
    except ClientError as e:
        print(e.response['Error']['Message'])
        return {
            'statusCode': 500,
            'body': 'An error occurred while fetching data from DynamoDB.'
        }

    if not response['Items']:
        return {
            'statusCode': 404,
            'body': f'No cities found in timezone: {random_timezone}'
        }

    random_city = random.choice(response['Items'])

    local_time = datetime.now(pytz.timezone(random_city['timezone'])).strftime('%Y-%m-%d %H:%M:%S')

    return {
        'statusCode': 200,
        'body': {
            'city': random_city['city_name'],
            'local_time': local_time,
            'latitude': random_city['latitude'],
            'longitude': random_city['longitude'],
            'timezone': random_city['timezone']
        }
    }

Here's a step-by-step explanation of what it does:

1. Initialize all_timezones: It takes all the time zones available through the pytz library and stores them in the all_timezones list.

   all_timezones = pytz.all_timezones

2. Find 5 PM Timezones: It filters all_timezones to create a new list, five_pm_timezones, containing only the timezones where the current local time is between 5 PM and 6 PM.

   five_pm_timezones = [
       timezone for timezone in all_timezones
       if 17 <= datetime.now(pytz.timezone(timezone)).hour < 18
   ]
   

3. Random Timezone Selection: It randomly selects one timezone from five_pm_timezones.

   random_timezone = random.choice(five_pm_timezones)

4. Query DynamoDB: It queries an AWS DynamoDB table using the table.scan method, filtering records where the 'timezone' key matches the random_timezone.

try:
    response = table.scan(
        FilterExpression=Key('timezone').eq(random_timezone)
    )
except ClientError as e:
    # Error handling for DynamoDB client error

5. Handle Errors: If there is a client error with DynamoDB, it prints the error message and returns a response with status code 500.

6. Check Response: It checks if the query returned any items (response['Items']). If not, it returns a response with status code 404 and a message saying no cities were found in the random_timezone.

if not response['Items']:
    return {
        'statusCode': 404,
        'body': f'No cities found in timezone: {random_timezone}'
    }

7. Random City Selection: If there are cities found in the random_timezone, it randomly picks one.

random_city = random.choice(response['Items'])

8. Local Time: It finds the local time in the timezone of the randomly picked city and formats it.

local_time = datetime.now(pytz.timezone(random_city['timezone'])).strftime('%Y-%m-%d %H:%M:%S')

8. Return Data: Finally, it returns the response with status code 200 and the details of the randomly picked city, including its name, local time, latitude, longitude, and timezone.

   return {
       'statusCode': 200,
       'body': {
           'city': random_city['city_name'],
           'local_time': local_time,
           'latitude': random_city['latitude'],
           'longitude': random_city['longitude'],
           'timezone': random_city['timezone']
       }
   }

IT’S WORKING!

We now have a Lambda function, and a DynamoDB instance returning a random place in the world where it is between 5:00 PM and 5:59:59 PM- now to build the first version of the front end and make some IaC.

To get to this point I’ve only reviewed 1/7 of my chat logs with you.

In the next part I’ll tackle my MVP and Infrastructure as code.

Manifest

Part 0 - Introduction
Part 1 - Failing Fast (Building with prompts) *You are Here 💁‍♀️
Part 2 - MVP and Deployments
Part 3 - MVPolish
Part 4 - So what.

Share

Subscribe now

I’m sure you want to see each and every prompt; but there are hundreds if not thousands of prompts and I couldn’t possibly paste them here…. oh, ChatGPT has link sharing - as a special offer for paid subscribers, you can access the enthralling conversation below

Read more

We entertain ideas of owning a carwash, then steering an ATM empire, before leaping onto the digital stage - creating a website. As our ideas crystallize, we add constraints: a lightning-quick launch within a weekend, no collection of user data, and a smooth user experience without any pesky logins. And most importantly - drop shipping.

Our brainstorming produces a flurry of concepts, some whimsical, others borderline genius.

A humorous calculator that never quite gets it right, with advertisement driven financials.

A one-click solution for baby-shower registries that could bring relief to countless confused friends and relatives.

An emergency haven, "OopsIForgotMothersDay.com", designed for those who have inadvertently let Mother's Day sneak up on them.

And finally, we land on the best idea so far: a site that tells you where, at this very moment, it's 5pm. Because no matter where you find yourself, it's comforting to know there's always a place where it's time to kick back and relax.

The vision is refreshingly straightforward - a single webpage presenting a map. On this map, an infectious wave of beer emojis will cascade across the screen, infusing an element of lighthearted humor into a predominantly informational site.

An accompanying section of the page introduces you to each location, courtesy of handpicked automated Wikipedia excerpts. It's a snapshot into the life and culture of the place where it's currently happy hour. You become a virtual globe trotter, one beer emoji at a time.

Rounding out the features is a slick e-commerce integration, offering branded merchandise. Drop-shipped with minimal fuss and a modest markup, these trinkets and keepsakes add a physical touch to the digital experience, and offer a way to keep the lights on. It's an inviting blend of education, entertainment, and commerce - a small slice of the globe served up on a silver web platter, with a side of fun.

As we get home from our boozy morning and I disappear into my office, I’ve got a secret mission for this very real idea. I’m not going to build it.

ChatGPT is. 🍿

Subscribe now

Share

My covert operation emerges from the swirling fog of speculation surrounding Large Language Models (LLMs). The rumors are rife, painted with broad, audacious strokes.

Will they usurp our jobs?

Take charge of our taxes?

As these whispers continue, my mission is to penetrate the murkiness and discern the true capabilities of these LLMs. More specifically, can I employ one to build a production-quality website without minimal manual human intervention?

I'm more than equipped to construct this website without ChatGPT and to make it interesting I'm imposing this self-restraint.

I am not permitted to suggest answers.

Every line of generated code must be executed 'as is', and the inevitable bugs (of which there will be plenty) must be handed back to the model for rectification.

It's the model's call on how to proceed, crafting its own solutions to the hurdles it stumbles upon creates for itself. This is an experiment in AI problem-solving, an exploration of the model's ability to learn and improve from on its own errors.

I am going to insist on a few things.

• Use AWS

• Use Serverless wherever possible

My architecture for this endeavour will be entirely serverless:

• DynamoDB for the NoSQL database

• Lambda for running the serverless “computations”

• API Gateway to handle the REST requests

In addition, I’ll employ Infrastructure as Code (IaC) and automated pipelines for seamless deployment and robust infrastructure management.

Again, I am not permitted to suggest answers. Let the mission commence.

As a Sneak Peak to the end of this series - here is our end-state architecture.

Subscribe now

Manifest

Part 0 - Introduction *You are Here 💁‍♀️
Part 1 - Failing Fast (Building with prompts)
Part 2 - MVP and Deployments
Part 3 - MVPolish
Part 4 - So what.

If you’ve ever made schema changes on large tables you know that these changes can take some time, use a lot of cpu and often cause production outages. Some schema changes will impact production workloads for the duration of the change as well.

I’m going to demonstrate how to use AWS RDS for MySQL’s Blue/Green feature to make a schema change, migrate from 5.7.43 → 8.0.34 and keep the availability as high as possible. I’ve recently helped several businesses evaluate their 5.7→8.0 upgrade paths as the EOL for 5.7 on RDS is mid-September 2023 approaching quickly on December 2023 February 29th 2024. I’ve recommended and tested this style of migration successfully over the last few months and I wanted to raise awareness for the success that I’ve been able bring to others. Please note, if you’re using MySQL in AWS, you should really be looking into Aurora MySQL instead of RDS MySQL. The flexibility, durability and resiliency options in Aurora far out weigh what’s possible in RDS and given proper architecture and design patterns for your data, you’re most likely to have a better experience - check out my Session at the AWS ANZ Summit 2023 for more information about how Aurora works under the hood to keep resiliency at its best:

Share One Twenty Three Cloud Street

Subscribe now

Building the Testing Environment

Lets start with my initial configuration.

I’m beginning with a MySQL database on version 5.7.43 with a instance identifier “old-busted”. It’s a very respectable db.t3.micro (Hey I’m frugal, but this will work on any instance family and size) with 20GB of GP2 storage (say frugal one more time and I swear..). I’m going to use the default parameter group and name the database “testing” with a default user of admin and a password so weak that a Commodore 64 could brute force it in less than a minute that I had my password manager create. While that RDS MySQL instance is creating, lets spin up an EC2 instance.

Again, we’re going with Free Tier eligible instances and I’ve made the executive decision to use Graviton - so t4g.small, 30GB GP2 storage, a new Key Pair and a locked down security group to my home IP. I’m going simple here and put a public IPv4 address directly on the host. Please don’t do this in production - there are much better ways.

Now I just install a few of my favorite packages…

Tmux on terminals, htop's dynamic views, 
Python scripts running, a coder's best news, 
Packages in Linux, tied up with string, 
These are a few of my favorite things.

MySQL in databases, Node.js in servers, 
Swift Git commits, no room for reverses, 
Docker containers like birds on the wing, 
These are a few of my favorite things.

SSH key pairings and Nginx's might, 
Bash scripting under the soft server light, 
Vim and Emacs, the joy that they bring, 
These are a few of my favorite things.

Home sweet home

Lets check the connection to the RDS MySQL instance (after remembering to update the database security group to include our EC2 instance which we definitely didn’t forget to do..)

Simple - we’re in. Now lets setup our simulation.

from faker import Faker
from multiprocessing import Process
import mysql.connector
import time

fake = Faker()

def create_insert_commands(start, end, process_number):
    # Establish a new connection for each process
    cnx = mysql.connector.connect(user='yourusername', password='yourpassword', host='localhost', database='testing')
    cursor = cnx.cursor()

    invite_data = []  # List to hold the data for all invites

    for i in range(start, end):
        name = fake.name().replace("'", "\\'")  # Generate a fake name
        user = fake.user_name()  # Generate a fake user name
        invite_data.append((name, user))

        # Perform the bulk insert every 10000 records
        if i % 10000 == 0:
            cursor.executemany("INSERT INTO invitations (invite_name, invite_user) VALUES (%s, %s)", invite_data)
            cnx.commit()
            print(f"Process {process_number}: Inserted records up to {i}")
            invite_data = []  # Clear the invite_data list

    # Insert any remaining data
    if invite_data:
        cursor.executemany("INSERT INTO invitations (invite_name, invite_user) VALUES (%s, %s)", invite_data)
        cnx.commit()
        print(f"Process {process_number}: Finished inserting records")

    cursor.close()
    cnx.close()

def main():
    total_records = 3919483539
    processes = 1
    records_per_process = total_records // processes

    for i in range(processes):
        start = i * records_per_process
        if i == processes - 1:  # If it's the last process, make sure it goes all the way up to total_records
            end = total_records
        else:
            end = start + records_per_process
        process = Process(target=create_insert_commands, args=(start, end, i))
        process.start()

if __name__ == "__main__":
    main()

(sudo yum install mysql-devel python3.9-devel ; sudo yum groupinstall "Development Tools" ; pip install faker; pip instal mysql-connector-python; pip install pymysql)

This is a quick script designed to create a problem for us. We want a simple table with fake data like this:

CREATE TABLE invitations (
    invite_number INT AUTO_INCREMENT,
    invite_name VARCHAR(255),
    invite_user VARCHAR(255),
    PRIMARY KEY (invite_number)
);

mysql> describe invitations;
+---------------+--------------+------+-----+---------+----------------+
| Field         | Type         | Null | Key | Default | Extra          |
+---------------+--------------+------+-----+---------+----------------+
| invite_number | int(11)      | NO   | PRI | NULL    | auto_increment |
| invite_name   | varchar(255) | YES  |     | NULL    |                |
| invite_user   | varchar(255) | YES  |     | NULL    |                |
+---------------+--------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)

The problem we could create with a table like this is to “run out of int’s” and want to switch to “bigint” (you should be using DynamoDB for this use case most likely).

Lets use the fake-data.py script I showed you earlier and give our table some data.

(We don’t actually need to cause the table to run out of integers to finish testing this; its just more fun with some data involved and simulated writes/reads)

Now lets have another script test our connection for us.

import pymysql
import time
import logging

logging.basicConfig(level=logging.INFO)

def connect_to_db():
    try:
        # Change the parameters based on your MySQL configurations
        connection = pymysql.connect(host='yourrdsinstance',
                                     user='admin',
                                     password='password',
                                     db='testing',
                                     charset='utf8mb4',
                                     cursorclass=pymysql.cursors.DictCursor)
        return connection
    except pymysql.MySQLError as e:
        logging.error(f"Error while connecting to MySQL: {e}")
        return None

def fetch_latest_row(connection):
    try:
        with connection.cursor() as cursor:
            sql_query = "SELECT * FROM invitations ORDER BY invite_number DESC LIMIT 1"
            cursor.execute(sql_query)
            latest_row = cursor.fetchone()
            connection.commit() 
            return latest_row
    except pymysql.MySQLError as e:
        logging.error(f"Error while fetching data from MySQL: {e}")
        return None

if __name__ == "__main__":
    connection = connect_to_db()
    disconnection_time = None

    while True:
        if connection:
            latest_row = fetch_latest_row(connection)
            if latest_row:
                logging.info(f"Latest row data: {latest_row}")
                if disconnection_time:
                    reconnection_time = time.time()
                    elapsed_time = reconnection_time - disconnection_time
                    logging.info(f"Database reconnected. Connection was lost for {elapsed_time:.2f} seconds.")
                    disconnection_time = None
            else:
                logging.warning("Failed to fetch latest row. Trying to reconnect.")
                connection.close()
                connection = None
                disconnection_time = time.time()
        else:
            logging.warning("Connection to database failed. Trying to reconnect.")
            connection = connect_to_db()
            if not disconnection_time:
                disconnection_time = time.time()

        # Sleep for 1 second before the next iteration
        time.sleep(1)

There we go - now we’re able to monitor the RDS Instance. Let’s test the outage of the instance by having the DB reboot.

Our DB monitoring script shows that we lost the connection for about 9 seconds ( I did need to start the fake-data.py script again) not bad for a reboot.

Creating the Blue/Green Deployment

Now lets make a Blue/Green Deployment.

We’ll click create and select MySQL 8.0.34

Then we’ll just create the staging environment and wait for it to provision.

If we had read replicas on old-busted, this will also create new read-replicas in the green side of the “new-hotness” deployment. When we fail over in a blue green deployment in RDS we get to keep our old endpoint names and use our new databases. However we leave our Green environment is what will become our new production “Blue” environment after the switch over.

While we’re waiting, let’s create the parameter group for the Green Instance. I’ll go to the “new-hotness” Blue/Green Deployment and find our Green instance then find the parameter group.

Oh. I don’t have the default mysql-8 parameter group in my parameter group page. Cause I’ve never had a mysql 8 instance in this account before.

The Green instance will actually start off as a 5.7 instance and then perform an upgrade adding the default:mysql8-0 options and parameter groups! Fun!

Once the 8-0 parameter group is created we need to make a copy of it and then change a single (dangerous) setting.

We’ll start by creating a parameter group like below:

Then we’ll edit the parameter group, we want to change the parameter “read_only” from “{TrueIfReplica}” to “0”

This will allow us to make changes in the Green instance. Please Please know that if you make incompatible changes in the green instance, you will stop replication and you’ll need to sort out the issues before you can make replication work again!

Now that the Blue Green Deployment is created. I’ll assign our new parameter group to the Green instance by modifying it and assigning the custom-green-mysql80 group. Changes to the green database are kept when promoting to blue; however, changes to green do not propagate to blue before the promotion.

Let's wait for the Green instance to finish modifying and take a look at the Deployment itself.

We can see a side by side comparison of our Blue “Active Production” Instance and our Green “Staged Production” Instance. (These databases are offline at the time of publishing, don’t bother trying to DDOS my t3.micro instances at those Endpoints)

We can also see some stats about the instance in the Blue/Green Deployment:

Now that our instances are available and replicating, let’s connect to the green instance and make our schema change! After we definitely remember to reboot the instance first to apply the parameter group changes….

Replication will kick back in after the reboot, don’t worry!

mysql> use testing;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> describe invitations;
+---------------+--------------+------+-----+---------+----------------+
| Field         | Type         | Null | Key | Default | Extra          |
+---------------+--------------+------+-----+---------+----------------+
| invite_number | int          | NO   | PRI | NULL    | auto_increment |
| invite_name   | varchar(255) | YES  |     | NULL    |                |
| invite_user   | varchar(255) | YES  |     | NULL    |                |
+---------------+--------------+------+-----+---------+----------------+
3 rows in set (0.03 sec)

mysql>ALTER TABLE invitations MODIFY invite_number BIGINT NOT NULL AUTO_INCREMENT;
Query OK, 11100002 rows affected (50.63 sec)
Records: 11100002  Duplicates: 0  Warnings: 0

mysql> describe invitations;
+---------------+--------------+------+-----+---------+----------------+
| Field         | Type         | Null | Key | Default | Extra          |
+---------------+--------------+------+-----+---------+----------------+
| invite_number | bigint       | NO   | PRI | NULL    | auto_increment |
| invite_name   | varchar(255) | YES  |     | NULL    |                |
| invite_user   | varchar(255) | YES  |     | NULL    |                |
+---------------+--------------+------+-----+---------+----------------+
3 rows in set (0.06 sec)

We can see that our Green instance has completed our schema change and is on mysql 8.0! We also induced some replica lag while the schema change was taking place. The amount of replica lag will depend on how big your tables are. If its going to take DAYS to do your schema change here, be wary of how much disk you have available and when you plan your fail over, the replica lag must be near 0 for a switch over to happen.

We should switch to the desired parameter group (the default 8.0 group in this case before our switchover takes place. We wouldn’t want any future read replicas having “write” permissions!

Our Blue instance and Green instance now have our desired changes.

Let’s proceed with the switch over and observe the downtime.

Understanding the Switchover

A few things happen before and during the switch over, you can read all about them here: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments-switching.html but I’ll put a summary in below.

Key Features:

1. Timeout Setting: You can set a switchover timeout ranging from 30 seconds to one hour. The default is 300 seconds. If the switchover exceeds this limit, all changes are rolled back.

2. Guardrails: Before initiating a switchover, RDS performs a series of checks on both the blue and green environments to ensure their readiness. These include checks on replication health, replication lag, and active writes. Any discrepancies will halt the switchover.

3. Switchover Actions: During the switchover, RDS runs guardrail checks, stops new write operations, drops and disallows new connections, waits for replication to sync, and renames DB instances and endpoints in both environments.

4. Post-Switchover: After a successful switchover, the new production environment allows write operations while the old one allows only read operations until rebooted.

5. Event Monitoring: Amazon EventBridge can be used to monitor the status of a switchover.

6. Tagging: Tags configured in the blue environment are transferred to the green environment post-switchover.

7. Rollback: If the switchover is halted for any reason, all changes are reverted to maintain the integrity of both environments.

Best Practices:

• Thoroughly test resources in the green environment.

• Monitor CloudWatch metrics.

• Identify low-traffic periods for the switchover.

• Ensure DB instances in both environments are healthy and available.

• Keep DNS cache TTL to a maximum of five seconds.

• Complete data loading before switchover.

Note: Modifications to DB instances are not allowed during a switchover.

Performing the Switchover

To perform the switchover we need to select our Blue/Green Deployment in the console and then select “Switch Over” from the actions menu. I have my monitoring script running and I am still inserting records into the table on the Blue endpoint.

We get an opportunity to review our changes and then we can just click “Switch over”

During the switchover, RDS runs guardrail checks, stops new write operations, drops and disallows new connections, waits for replication to sync, and renames DB instances and endpoints in both environments.

Let’s see how long that takes.

Writes were unavailable in the video from 0:17 → 1:22 for a total of 1:05. Not too shabby. Reads were unavailable from 0:27→ 1:22 for a total of 55seconds. My monitoring script saw about 2 seconds of outage, and due to bugs in the script that I’m not willing to resolve, its missed that its last check was stalled and waited too long for a timeout as the endpoint flipped over to the new instance. At the end I’m left with my production endpoint on 8.0 with my schema changes and my old 5.7 instance renamed to “old-busted-old1” still running 5.7!

Conclusion: Navigating Changes Seamlessly with AWS RDS's Blue/Green Feature

Embarking on database schema changes or version upgrades is often fraught with risks that could affect the availability and performance of your applications. This blog post has been a somewhat comprehensive guide on how to leverage the Blue/Green deployments feature in AWS RDS for MySQL to make these transitions as smooth as possible.

By following this approach, we've been able to update the MySQL version from 5.7.43 to 8.0.34 and alter the database schema—all while maintaining high availability. The Blue/Green feature, with its guardrails and rollback capabilities, ensures that the switchover is as risk-free as possible. Monitoring tools and best practices were also discussed to provide a holistic strategy for database migration and schema modification.

Key Takeaways:

1. Efficiency: The Blue/Green deployment feature allows you to make schema changes and version upgrades without significant downtime. In our test, the total connection unavailability was just 1 minute and 5 seconds.

2. Safety Nets: The guardrails and timeout settings are invaluable for preventing unwanted disruptions and data inconsistencies. If something does go wrong, the process rolls back, ensuring data integrity.

3. Monitoring: Real-time monitoring allows you to keep tabs on the switchover process and quickly react to any issues.

4. Flexibility: This method is not just for large enterprises but is scalable to any size, as demonstrated using a db.t3.micro instance.

5. Future-proof: The Blue/Green deployment strategy makes future changes easier to manage and less risky, setting you up for long-term success.

As we near the End-of-Life for MySQL 5.7 on AWS RDS, this strategy offers a solid pathway to upgrade to newer versions. However it's worth considering a move to Aurora MySQL for even more robustness and flexibility, as discussed in my session at the AWS ANZ Summit 2023.

With these tools and strategies in your arsenal, may your databases always be available, and your schema changes forever be smooth!

Share

Subscribe now

Share One Twenty Three Cloud Street

The APC ECU-3 is an old and basic tool for those harnessing solar power. I’ve found that in Australia it’s unfortunately the device you’re likely to get stuck with installed with your panels. It very slowly provides a wealth some data about your solar power system. However, its lack of an API presented a significant challenge when it came to integrating this data with Home Assistant.

The system shows a screen like this when visiting its web page. The goal is to scrape all the info out of this table and import it into Home Assistant sensors.

I set out to build this bridge between this silly page and Home Assistant. This tool retrieves, parses, and formats the data from the APC ECU-3 device, transforming it into a JSON file that Home Assistant can interpret.

With no API to call, I had to get creative old school. I wrote a simple script using Python's requests module that fetches the data directly from the APC ECU-3 device's parameters page. The data returned was a dense basic HTML page filled with valuable basic information hidden directly exposed within an HTML table.

Next, I had to extract the required data points from this HTML table. BeautifulSoup, a Python library for parsing HTML, was instrumental here. It allowed me to sift through the HTML structure, identify the data table, and pull out the rows containing the information I needed.

I then transformed this data into a format that Home Assistant could understand. Using Python, I created a dictionary where the keys were the Inverter IDs and the values were lists containing the power data for each inverter. Then I had to remove the units since Home Assistant will put its own units in place and I would get “140 W W” in my graphs. The final step in this process was to convert this dictionary into a JSON file and save it on the Home Assistant server in an accessible location so other scripts can access it.

To help others with the integration, I wrote another Python script that automates the generation of the relevant section of HA’s configuration.yaml file. This script reads the inverter IDs from the ECU and produces the corresponding sensor entries for the configuration.yaml file.

Not exactly difficult, but it is tedious if you’re unfamiliar with my janky setup.

In Home Assistant, I set up an automation to run the Python script every minute. This ensured that the solar power data in Home Assistant was always up-to-date as updated as possible since the ECU doesn’t seem to provide regular updates and often fails to provide any data. The automation, coupled with the sensor entries generated by the Python script, allowed the raw APC ECU-3 data to be leveraged within Home Assistant.

Upon completion, I successfully managed to plot the power output of each solar panel!

Developing the APC ECU-3 Data Reader was a journey filled with challenges boring scripting and hack-y methods. There's still plenty of room for improvement - but at this point, I need your help!

There's an existing project, the APSystems ECU-R component for Home Assistant, which supports models with APIs. Unfortunately, the APC ECU-3 model isn't yet supported. But with our collective effort, we can expand its support to include the APC ECU-3.

So, if you're a fellow tech enthusiast, a developer, or someone with a knack for tackling challenges, I have an ask for you! I invite you to contribute. Whether it's improving the current project or adding support for the APC ECU-3 to the APSystems ECU-R component, your contribution can make a difference.

You can find the source code here: https://github.com/jeeshofone/ha-apc-ecu-3 Please reach out if you have an APC ECU-3 and found this helpful or want to help improve the project.

(I also don’t want to have sole ownership of this technical line of credit)

I’ve been using vim since I started using linux back in the knoppix 3.x days and I wanted to share a quick list of vim related tips. I’ll start off easy and get more advanced as I go.

Basic

1. Opening Vim:

   1. To open Vim in your terminal, simply type vim. To open a specific file, type vim filename.

2. Vim modes:

   1. Vim has several modes, but the most important ones are Normal mode, Insert mode, and Command-line mode.

      • Normal mode: The default mode when you start Vim. You can use commands and navigate the text without editing it.

      • Insert mode: In this mode, you can insert text like a regular text editor. To enter Insert mode, press i in Normal mode.

      • Command-line mode: Used to execute commands, save files, and more. Press : in Normal mode to enter Command-line mode.

3. Basic navigation:

   1. In Normal mode, you can navigate using the following keys:

      • h - move left

      • j - move down

      • k - move up

      • l - move right You can also use the arrow keys, but using h, j, k, l allows you to keep your hands on the home row.

4. Inserting text:

   1. Press i to enter Insert mode and type as you would in any other text editor. To exit Insert mode and return to Normal mode, press Esc.

5. Saving and quitting:

   1. In Normal mode, press : to enter Command-line mode, then:

      • To save a file, type w and press Enter.

      • To quit without saving, type q and press Enter.

      • To save and quit, type wq and press Enter.

      • To force quit without saving, type q! and press Enter.

6. Basic editing:

   1. In Normal mode, you can perform various editing operations, such as:

      • dd - delete the current line

      • yy - yank (copy) the current line

      • p - paste the yanked text after the current line

      • u - undo the last change

      • Ctrl + r - redo the last undone change

7. Searching:

   1. In Normal mode, you can search for text:

      • /search_term - search forward for the term

      • ?search_term - search backward for the term

      • n - move to the next occurrence of the search term

      • N - move to the previous occurrence of the search term

Pro

1. Visual block selection:

   1. To select a block of text, first press Ctrl + v in Normal mode to enter Visual Block mode. Then, move the cursor to define the rectangular selection. You can use h, j, k, l, or the arrow keys to move the cursor.

      • I really really really really love visual block mode. Block editing in Vim can be powerful and efficient when working with columns or multiple lines.

2. Inserting text:

   1. With a visual block selected, press I to insert text before the selected block. Type the text you want to insert, then press Esc. Vim will insert the text on each line of the block.

3. Appending text:

   1. Similarly, with a visual block selected, press A to append text after the selected block. Type the text you want to append, then press Esc. Vim will append the text on each line of the block.

4. Deleting a block:

   1. To delete the selected block of text, press d or x in Visual Block mode. This will delete the rectangular selection.

5. Changing text:

   1. To change the selected block of text, press c in Visual Block mode. This will delete the block and enter Insert mode, allowing you to type new text. After typing, press Esc to apply the change to all lines in the block.

6. Indenting and dedenting:

   1. To indent or dedent a visual block, select the block in Visual Block mode and press > to indent or < to dedent. You can also prefix these commands with a number to specify the number of indentation levels, like 3> for three levels of indentation.

7. Columnar search and replace:

   1. Perform a search and replace operation within the selected visual block by first selecting the block and then typing :s/search/replace/g. Press Enter to execute the command.

Advanced

1. Macros:

   1. Macros allow you to record a series of keystrokes and replay them to automate repetitive tasks. To record a macro, press q followed by a register name (a lowercase letter) in Normal mode, e.g., qa. Perform the desired actions, then press q again to stop recording. To replay the macro, press @ followed by the register name, e.g., @a.

2. Split windows:

   1. Vim supports split windows, allowing you to view and edit multiple files or different parts of the same file simultaneously. Use the following commands to manage split windows:

      • :split or :sp - open a horizontal split

      • :vsplit or :vs - open a vertical split

      • Ctrl + w followed by h, j, k, or l to navigate between splits

3. Tab pages:

   1. Tab pages are another way to work with multiple files. Each tab page can contain multiple split windows. Use the following commands to manage tab pages:

      • :tabnew or :tabe - open a new tab page

      • gt or :tabn - go to the next tab page

      • gT or :tabp - go to the previous tab page

      • :tabc - close the current tab page

4. Folding:

   1. Folding allows you to collapse and expand sections of code or text, making it easier to navigate and manage large files. There are several folding methods available in Vim, including manual, syntax-based, and expression-based folding. Use the following commands to manage folds:

      • za - toggle the fold at the cursor

      • zR - open all folds

      • zM - close all folds

      • zj - move to the next fold

      • zk - move to the previous fold

5. Text objects:

   1. Text objects allow you to operate on specific portions of text, such as words, sentences, paragraphs, or blocks. They can be used with commands like d, c, y, and v. Examples of text objects include:

      • aw - a word

      • as - a sentence

      • ap - a paragraph

      • a( or a) - a block enclosed by parentheses

      • a[ or a] - a block enclosed by square brackets

6. Command-line window:

   1. The command-line window allows you to edit and navigate command-line history. Access it by pressing q: in Normal mode. You can edit and execute previous commands or search through your command history.

7. Persistent undo:

   1. Vim can store your undo history across sessions, allowing you to undo and redo changes even after closing and reopening a file. To enable this feature, add the following lines to your .vimrc file:
      set undofile

      set undodir=~/.vim/undodir

   2. Ensure that the undodir directory exists or create it if necessary.

vimspert

1. Omni completion:

   1. Vim offers context-aware autocompletion through omni completion. Press Ctrl + x followed by Ctrl + o in Insert mode to trigger omni completion. For programming languages, you may need to install additional plugins or configure Vim to provide relevant suggestions.

2. Quickfix list and location list:

   1. Vim's quickfix list stores errors, warnings, or search results from various sources, such as compilers or grep-like tools. The location list is similar but specific to individual windows. Use the following commands to work with quickfix and location lists:

      • :copen - open the quickfix window

      • :cclose - close the quickfix window

      • :cnext - move to the next item in the quickfix list

      • :cprev - move to the previous item in the quickfix list

      • :lopen, :lclose, :lnext, :lprev - same commands for location lists

3. Vim sessions:

   1. Vim sessions allow you to save your current editing environment, including open files, window layout, and options, so you can quickly restore it later. Use the following commands to manage sessions:

      • :mksession session_name.vim - save the current session

      • :source session_name.vim - restore a saved session

4. Marks:

   1. Marks are used to bookmark specific positions in a file or across multiple files. Lowercase marks are local to a file, while uppercase marks are global. Use the following commands to work with marks:

      • m{a-zA-Z} - set a mark at the cursor position, e.g., ma or mA

      • '{a-z} or '{a-z} - jump to a lowercase mark in the current file

      • '{A-Z} or '{A-Z} - jump to an uppercase mark in any file

5. Mappings:

   1. Mappings allow you to define custom keybindings or modify existing ones in Vim. You can create mappings for different modes using commands like :map, :nmap, :imap, :vmap, etc. For example, to map jj to Esc in Insert mode, add the following line to your .vimrc file:

      inoremap jj <Esc>

6. Registers:

   1. Registers are storage locations that can hold text, macros, or other data. Vim has multiple registers, such as unnamed, named, numbered, read-only, and expression registers. To access or manipulate registers, use the following commands:

      • "{a-z0-9}%#:-=. - specify a register, e.g., "ayy to yank the current line into register a

      • :reg or :registers - display the contents of all registers

      • :echo @a - display the contents of a specific register, e.g., register a

7. Plugins:

   1. Vim has a vibrant plugin ecosystem that can extend its functionality. Popular plugin managers include Vim Plug, Pathogen, and Vundle. Some notable Vim plugins are NERDTree (file explorer), CtrlP (fuzzy file finder), and YouCompleteMe (code completion).

   2. Checkout

      https://vimawesome.com/ for some truely epic plugins.

Conclusion

This is just a brief introduction to using Vim. There are many more commands, features, and customizations available. To learn more about Vim, you can refer to its extensive documentation by typing vimtutor in the terminal! And NO! I won’t tell you how to quit vim, its too good to stop using.

ChatGPT from OpenAI is quickly becoming a household name, and if you haven't heard of it yet, you're in for a treat. Its amazing and accurate responses are captivating audiences, but there's still a lot of potential for growth and improvement.

If you're not familiar with xkcd's "What If?" series, it's a popular webcomic that tackles all sorts of science- and math-related questions with a humorous and informative approach. In this post, we'll be using ChatGPT to answer a selection of questions from “What If?” and see how far it can go.

Lets start with something simple.

Compare that with the xkcd about page:

“It's not actually an acronym. It's just a word with no phonetic pronunciation -- a treasured and carefully-guarded point in the space of four-character strings.”

“For those of us pedantic enough to want a rule, here it is: The preferred form is "xkcd", all lower-case. In formal contexts where a lowercase word shouldn't start a sentence, "XKCD" is an okay alternative. "Xkcd" is frowned upon.”

https://xkcd.com/about/

The description of the comic may be accurate, but the origin of the name appears to be entirely fabricated. Now, lets get to the fun part

Dropping A Mountain

What if a huge mountain—Denali, say—had the bottom inch of its base disappear? What would happen from the impact of the mountain falling 1 inch? What about 1 foot? What if the mountain's base were raised to the present height of the summit, and then the whole thing were allowed to drop to the earth?

Get ready to laugh.

Mountains can float because gravity will hold them in place. To be fair, no one has tried to make a 1 foot gap at the base of a mountain to test this. [Citation Needed]

Saliva Pool

How long would it take for a single person to fill up an entire swimming pool with their own saliva?

Ok lets give it some more information..

This math is off by about an order of 100. It should take ~8300 years not 8.3. ChatGPT is not good at math. This response is after I taught it some proper multiplication.

Please don’t rely on ChatGPT for math.

Keyboard Power

As a writer, I'm wondering what would be the cumulative energy of the hundreds of thousands of keystrokes required to write a novel.

Even if you help it along by providing missing information, it can’t handle the analysis.

A Mole of Moles

What would happen if you were to gather a mole (unit of measurement) of moles (the small furry critter) in one place?

Hard to believe but this is actually a fun answer.

Relativistic Baseball

What would happen if you tried to hit a baseball pitched at 90% the speed of light?

Lets help it out a bit with some of the conditions laid out in Randall’s original post.

Someone forgot to teach it how to play baseball?

hmmmm, ok so it does know baseball

now we’re getting somewhere….

Everybody Jump

What would happen if everyone on earth stood as close to each other as they could and jumped, everyone landing on the ground at the same instant?

Supersonic Stereo

What if you somehow managed to make a stereo travel at twice the speed of sound, would it sound backwards to someone who was just casually sitting somewhere as it flies by?

This is a fun one. Randall actually cites a scientific paper that confirms the sounds would be backwards if you could hear it over the sonic boom.

Glass Half Empty

What if a glass of water was, all of a sudden, literally half empty?

Before I ask it the What If question, lets see if it can handle the concept of “Half”.

Apparently 500ml is more than half of 1000ml. Again, do not use ChatGPT for math.

On to the real question

Now if only ChatGPT could draw.

Letter To Mom

What’s the fastest way to get a hand-written letter from my place in Chicago to my mother in New Jersey?

Hmm, not very innovative, lets try a follow up question.

Conclusion

I think it’s safe to say that ChatGPT won’t be taking our jobs anytime soon. For now it’s just a fun tool that can do some decent writing and answer general knowledge questions.

I went to my favorite car dealer several years ago and leased a brand new car. We worked out option packages and mileage, down payment and monthly payments. I gave them my down payment of only $5,000USD and I walked away with a $50,000 automobile.

Let’s translate that exchange into application ownership.

I went to my development team several years ago and asked for a brand new application. We worked out the amount of sprints, time and cost to build. I gave them the time to build it and after only 6 months my startup walked away with an application capable of handling our million dollar idea.

After I bought my car, my bank followed up with me every month asking for a payment. I didn’t walk away from the dealership for just my initial $5,000 payment and no follow up payments on principal and interest. You must service your debt.

Our technical workloads don’t only accumulate debt. Instead, they accumulate interest on existing debt.

This unpaid interest is what many people are often referring to when they think of tech debt. From the first line of code, a new project begins accumulating debt; but the interest hasn’t come due, yet.

We don’t buy our applications, even from internally sourced development teams.

We lease them. They require interest payments in the form of package upgrades, architecture changes and security posture alignments. The debt and interest rates we take on at the inception of the idea can be known and must be planned for.

There is no “bank” that’s going to ask for the work to be completed to pay them back. If the interest isn’t paid, the debt will not be serviced and you won’t be sent to collections. This lack of external accountability, a vacuum of required responsibility, is a driving force in the creation of this unpaid interest in many businesses.

“You don’t own a house, a house owns you.” You are beholden to the house to not only pay for it but to maintain it and even improve it. If you don’t then:
The bank will come looking for payments.
The counsel will get after you for dues.
The neighbors will complain at you for your unkempt yard.
Your insurance will get on you for not mending the damages.

With tech development there are similar, yet different driving forces.

The business will ask the development team to add features and reduce bugs.
The customers will ask the business to add features and reduce bugs.
The partner integrators will ask the business to add features and reduce bugs.

It is very unlikely that anyone will ask the development team to pay any of their time towards the technical interest payments if they don’t understand the mountain of pain that is coming for them.

Technical leaders must be accountable to themselves and the business to understand how large their line of credit is so that they can service it properly.

Interest re-payment is part of the total cost of ownership.

Several years ago I was tasked with architecting a new product(lets call it “MultiTower”). Our team, like many “start-ups” consisted of just a hand full of absolute rockstars. Over the next year we built our initial proof of concept into a viable sellable product. We picked up a couple customers and set about reducing bugs and adding features.

Our team now had a technical line of credit that we had to keep servicing. Every new feature increased the amount of ownership, and increased the amount of credit we’ve chosen to take out.

We kept up with frameworks, database engine versions, security patches and released often. I implemented a micro-service architecture and insisted on high standards for testing. 20% of our time was easily and without pushback spent on interest re-payment. We never backed down from our code coverage target or our automation. At the close of the second year of development we had a very well maintained application and no extra customers to show for it. The product just wasn’t something our customers wanted.

Developing the proper way isn’t going to get your startup off the ground.

“Well, Will,” you might say “thats why you should skip ‘proper’ and build a monolith at your startup - spend that 20% on your features!”

Developing in an expedient way isn’t what gets you off the ground either.

Servicing the interest on your technical line of credit is the best practice regardless of the architectural complexity.

How much money do you need to make a year to own a Ferrari? How about a private jet? How much time do you need to spend maintaining your Kubernetes cluster to keep the versions up to date and the plugin versions all compatible?

If you don’t have the resources for “fancy”, ignoring the maintenance to stretch your budget to afford “fancy” does instantly give you “fancy”. You won’t have that fancy thing, in a couth state, for very long if you don’t increase your budget for maintenance.

We were fortunate enough to have enough staffing, with the right levels of experience to support the repayment for our chosen level of complexity. “MultiTower” was created out of a established scaleup as a new product. We didn’t operate like a startup because we didn’t work at one. We had the benefit of successful, existing products to fund the work and an existing customer base to begin selling to.

There are a few inflection points on the way up. Especially within applications that have a “cold start” problem to overcome.

The things that make a startup successful are rarely the exact mechanisms that make them successful as a scale-up and are rarely, if ever, the correct path to creating a successful enterprise.1

The square of velocity is acceleration. The square of shortcut is debt.

A business that has reached its next stage of growth without planning for adaptation will continue to pile on unserviced debt to an ever growing line of credit. This will make new features harder to build, increase time to deployment, and frustrate the developers. The larger the unserviced debt the more time it will take to correctly bring new business decisions, features, and security fixes into production. EVERY change will take longer. This pressure on the development team has a tendency to force their hand to meet deadlines.

How are deadlines met under pressure? Shortcuts are taken. The more shortcuts that are taken, the more often that specific shortcut will effect a dependent system or feature. When shortcuts are taken, the metaphorical Interest Rate goes UP in addition to the expected increase in the size of the line of credit.

Why debt is good

You’re going to be taking out this line of credit whether you want to or not.

Unfortunately the workload now exists. Having a business with a well serviced technical line of credit is incredibly attractive to your technical team.

Would you rather:

• Pay the bill on a well maintained car doing preventative maintenance at predictable intervals

• Be late to work cause it “suddenly” broke down and also have irregular maintenance bills that unexpectedly delay your journey

Treat your workloads the same way and give them the care they require.

Go ahead - Decide to build the monolith, or the micro service or the serverless or the IIS stack of your dreams.
Understand the line of credit you are taking out on behalf of your business.
Understand what the interest payments look like and service your technical line of credit accordingly.

Your investors, developers, and customers will thank you later.

This blog is about to document its own creation.

Yesterday I spent about 2hours discovering a “good” domain name.  I had requirements that it be clever (no .com/.org/.net domains) easy to say and remember and cost very little. Many options were considered before settling on 123cloud.st - the domain meets all my requirements except I cant purchase it on my preferred registrar.

Selecting a secondary registrar has never been something I quite enjoy. I actually like having all my eggs in one basket sometimes, it makes them easier to count.  I’ve placed my money into Gandi and they have sent some of that to São Tomé and Príncipe (owners of the .st TLD) and step one is to update my Name Servers to use Route53. I’m greeted with a delightful delay indicating that I must now wait 12-24hr’s for some lines in a database to be updated.

These messages are typically written by a marketing team to keep support levels low in the event of “higher than normal” call volumes. I can check if this is completed with:

dig NS 123cloud.st | grep NS

Annnnd it was completed instantly.

I build on the cloud and I have an infinite to-do list.

Here’s how I’m going so far.

• Buy a Domain Name

• Static Site or CMS?

• How do you contact me?

  • Email address?

    • do i get a lot of spam now?

    • Free email or paid?

  • Request Forum that sends me an email?

  • Does it need a captcha?

  • Forums and discussion groups?

  • Are posts allowed to have comments? - not if we have a static page?

• How do/Should I integrate social media(and which if any platforms)

• Do I want/need to market this?

• I need a logo

• I need a theme

• Do I need an editor?

• Start a github and build with IaC before we go too far - or is click ops ok?

• What security risks am I willing to take to save money?

• Write an About Page

• Define a budget?

• Can this create revenue or will it only consume it?

• Total Cost of Ownership

• Prioritize this list

• How will I handle customer data? How will I handle customer privacy?

As much as I want to pick between a static site, and a plethora of CMS’s available - I need to prioritize my list first.  A savvy reader might notice that I’ve forgotten an item on my list - “write the content”.  I’m currently writing this in Slack - I haven’t made any decisions so far except for purchasing the domain.  My intention is to write this as I build and think about the project and that means writing before I have a reasonable tool to draft content.

I researched 5 of the top headless and traditional CMS’s - Self hosted WordPress, Ghost, Strapi, and Netlify for about 4 hours. This wasn’t a waste of time but it was the wrong item on my list.

IM SUPPOSED TO BE PRIORITIZING THE LIST

I have ADHD and I jumped straight into the most interesting (at the moment) item and I hyper focused. Let’s start again.How about this for a better list?

• Set a deadline to get this done

• Find a way to publish your content that includes

  • Fixed Cost Ownership

  • Hands-Off (mostly) Security

  • An Editor (still writing this in slack yo)

  • Understand privacy of user data

• Get A Logo/Theme (Async)

• Get an Email Address Setup (Async)

• Make a Branded Twitter (Async?)

Progress?

I often think in cycles like this. Constantly refactoring my thoughts into more accurate and manageable pieces of work. My first list was way too specific. At work I would call that jumping straight into solution-ing. It is not an effective way to create a sustainable $anything.

No progress.

That was the end of the first day. I hadn’t done my first priority to set a deadline and over a week has passed with no more words written.

Welcome to my life… oh wait, procrastination paid off…

While this project wasn’t occupying time at the keyboard, it was occupying time in my mind. I learned that I didn’t want to maintain the daily maintenance of a blog’s serverless or serverfull infrastructure. I didn’t need to start drafting shared responsibility models. There had to be options out there for non-tech centric writers who wanted all the same things I did. So I did what anyone would do and registered my email address.

I’ve used Amazon WorkMail (exchange) in the past for myself and love the pricing model at $4USD/user/month. In about 5minutes I was able to add my 123cloud.st domain and WorkMail automatically added all my DNS entries, validated my SNS configuration and I was done.

I finally shared my idea to begin a blog with one of my longest held mentors. He immediately mentioned that he reads a lot of content on Substack and that I should check it out.

Substack looked promising because it had a mechanism to complete this entire task!

• Find a way to publish your content that includes

  • Fixed Cost Ownership

  • Hands-Off (mostly) Security

  • An Editor (still writing this in slack yo)

  • Understand privacy of user data

I also looked into Medium.

The main differences that I could see were that Substack lacked built-in SEO and Medium lacked email lists. I care about both of these things and with Substack being the new kid on the block, maybe I can give it a chance to scale and grow with me? Every time I visit a medium article, I find so many distracting promotions around the page. Substack seems to give me the ability to keep my brand and not distract you with other content. I guess I’m selfish, so starting with this paragraph, I’m writing in Substack’s editor.

There are some problems with the editor.

I can’t control the alignment of text. I found some issues in putting multiple images in the same “row” and some “Home” screen bugs when you have fewer than 2 posts (Keeping that Hello World post up for now) which hide the navigation bar.

After spending a few hours playing around with the style settings and link settings its time to get a logo. I head over to Fiverr and 99designs to price some logo creation. I’m not very good with Adobe Illustrator unfortunately.

I immediately get distracted and sign up for twitter before quickly realizing that I don’t have a header or profile image and I’m supposed to be working on those.

Logos are hard

Substack gives its users a tiny logo in the top right corner. I found some interesting designs on Fiverr and quickly realized a cheaper option is to use their LogoMaker product to efficiently source a designer curated icon. 2 hours later and I’ve got what you’re seeing today. Probably not the best design for a Substack icon; however, I’m assuming that most readers will care more about the words than my logo?

After a quick signup for Stripe (the only payment platform on Substack). All I need to do is some custom domain work.

Custom Domains are ESSENTIAL for SEO

Adding a custom domain to Substack is pretty simple with Route53. Since Substack lacks the documentation on their own site for R53, I’ve provided it here.

1. Tell Substack your domain name (and put a www as a subdomain because they make you - we’ll fix that later)

2. Add the CNAME record that Substack provides to redirect your traffic in Route53

   

3. Create an S3 Bucket with the root name of your website (123cloud.st) without the www and enable Static Website hosting

   

4. Wait for Substack to update their systems. They seem to use Cloudflare for their CDN and they likely setup an origin for each domain:

   

5. You’ll get an email when its done. Then you will have www.123cloud.st and 123cloud.st directing to your Substack blog

/me wipes sweat from brow

This entire exercise has been incredibly thought provoking for me. I’ve already started working on several other problems that I must tackle and I can’t wait to share with you how I go about solving them. You’re not going to find me writing rote documentation without commentary. I’m going to be strongly opinionated and frequently insist on high standards. I hope you enjoyed this first post!

oh yeah and

• write the content