2025/05/11

Full source code for the latest iteration of this project is here, and here and here.

Introduction and Clarification

In Part 0 I described the design for a system to find unavailable tracks in a Tidal playlist, select suitable replacement tracks, and publish a corrected playlist.

The system uses the workflow orchestration engine Flyte, and will eventually run on my home Kubernetes infrastructure.

In Part 2 we can all go back to pretending this solution is the simplest. For now I will acknowledge, for the sake of my professional reputation, that introducing Flyte and Kubernetes into this problem is actually the exact opposite of “obviously the simplest solution”.

Secrets

Secrets management is no joking matter.

My system involves one secret: my Tidal credentials. The initial working prototype read credentials from a file specified by an environment variable.

Considerations

The system has four execution modes/environments:

1. Local plain old Python mode: uv run ./src/workflows/remedy_tidal.py
2. Local pyflyte mode
3. Local Flyte sanbdbox mode
4. Production mode in Kubernetes

Implementation

Flyte manages getting Kubernetes secrets into the Task execution pods. The first step is to add secrets_requests to the Flyte task decorator.

# orchestration/secrets.py

TIDAL_CREDS_GROUP = "tidal-creds"
TIDAL_CREDS_KEY = "json"
TIDAL_CREDS_PATH = "/etc/flyte/secrets/tidal-creds/json"

# tasks/fetch_playlist.py

from orchestration import image_spec, secrets

@fl.task(
    container_image=image_spec.DEFAULT,
    secret_requests=[
        fl.Secret(
            group=secrets.TIDAL_CREDS_GROUP,
            key=secrets.TIDAL_CREDS_KEY,
            mount_requirement=fl.Secret.MountType.FILE,
        )
    ],
    cache=True,
    cache_version="v5",
)
def fetch_playlist(playlist_id: str, path_to_creds: str) -> List[model.Track]:

Then create the secret in the local sandbox cluster:

% kubectl --context flyte-sandbox \
    create secret -n ${FLYTE_PROJECT}-development \
    generic \
    ${SECRET_GROUP} \
    --from-file=${SECRET_KEY}=${SECRET_PATH}

When I get to deploying to my cluster, I’ll have to create the secret in the namespace of the Flyte domain where I execute the workflow, either in my IaC or using kubectl.

For Local plain old Python mode and local pyflyte mode, I can continue using a local filesystem path to the credentials.

In the case of Kubernetes execution, no path_to_creds is passed to the workflow so we default to the location where Flyte mounts the secret. Nice and tidy!¹

make: It was the best of task runners, it was the worst of task runners

Much digital ink has been spilt on how bad Makefiles are, and how we’re Doing It Wrong with Makefiles. It’s all true, make is terrible and nobody should use it.

Anyways, I use it all the time in personal projects. For one, it’s installed on all my machines. For another, it works. And finally, unlike pretty much every other part of my personal projects, I don’t care if my Makefiles are “correct”. If I end up needing a “build system” then I’ve architected my personal project incorrectly.

👉 None of this applies to my professional systems, where the build system is just as important and gets as much attention as the rest.

For personal projects, “the build system” is cargo or uv or go. The Makefile is a repository of cargo or uv or go commands that I can refer to later, especially years later.

If I avoid using all but the most basic Makefile syntax there’s a high probability I can understand the content a decade later. And I can refer to it later when I need similar functionality in a new project.

👉 The more complex your build system / task runner config gets, the harder it will be to use it for reference in the future

As I’m learning, I’m not going to remember all the arguments to pyflyte. I could put them in a README, or I could put them in a task runner with shell autocomplete. So at that point, I can:

1. Learn a new task runner, slowing my main project’s velocity
2. Focus on Using Make Properly, slowing my main project’s velocity and self-inflicting severe mental distress
3. Just throw together something that works well enough, ignoring .PHONY and other boilerplate, finishing my project earlier, never having directly called a C compiler

I chose option 3. In this case it really is the simplest solution!

Success!

After all this, I could execute make run, make sandbox-run-local, and make sandbox-run-remote to execute the workflow in plain old python mode, pyflyte local mode, and local Kubernetes mode, each using my actual Tidal credentials to successfully fetch a real playlist.

Next Up

This iteration of the project was short. In it, I designed and implemented a secrets handling strategy, a multi-platform execution strategy, implemented and validated integration with the music provider, and wasted no time on crafting exquisitely correct Makefiles.

At the end of it, I had a working playlist fetcher and stubs for most of the rest of the workflow.

In the next iteration/post, I’ll talk about finishing the workflow and tasks, the Flyte console, and adding tests to the project.

1. Just because the design is intentionally overcomplex doesn’t mean the implementation should be. ↩