servius/sukr
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: update to latest astro ink version

Browse Source
This commit is contained in:
Timothy DeHerrera
2024-07-06 14:41:28 -06:00
parent 97aa45c32e
commit e4620874f9
84 changed files with 13949 additions and 4620 deletions
Download Patch File Download Diff File Expand all files Collapse all files

294
src/content/blog/NixOS Policy Breakdown.md Normal file
View File

@@ -0,0 +1,294 @@
---
title: "Political Bikeshedding: NixOS Edition"
description: On Social Dynamics and Leadership
tags:
- nix
- politics
author: Tim D
authorGithub: nrdxp
authorImage: https://avatars.githubusercontent.com/u/34083928?v=4
authorTwitter: nrdxp52262
date: "2024-07-02"
category: politics
---
This piece offers a perspective on recent NixOS project challenges from
a long-term contributor. As one of the authors of [RFC 175][175], which
attempted to address moderation issues but faced obstacles in adoption,
the writer brings both experience and a commitment to improving the
project's governance.
Given the complexity of the situation, this article aims to provide a
high-level analysis rather than an exhaustive account. While specific
examples are limited for brevity, future pieces may explore more
detailed case studies. The current goal is to establish a framework for
understanding and addressing broader issues that are increasingly
prevalent across the open-source world.
## The Silent Majority and the Myth of "Community Consensus"
A significant portion, if not the majority, of people likely despise
politics and deliberately disengage from it, focusing on more enjoyable
pursuits. By nature, these individuals aren't necessarily interested in
having their unrelated political or ideological views "represented" in
groups or forums they join for specific interests, such as hacking.
Crucially, this silent majority challenges the notion of a unified
"voice of the community." Many claim to speak on behalf of "the
community," advocating for actions or bans based on supposed community
consensus. However, if a silent majority exists, such claims of
representing the entire community are inherently flawed and potentially
misleading, at best.
The concept of a silent majority and the questionable nature of claimed
community consensus lead us to examine another critical issue: the
misuse of marginalization claims, which many of these voices use as a
foundation. Understanding the contextual and temporal nature of
marginalization is key to addressing this problem.
## Marginalization is Contextual and Temporal
The term "marginal" has no fixed definition outside a specific context.
Consider this scenario: someone stands on the far side of a room while
others gather at a table. This person detects a threat, perhaps a small
fire, only visible from their position. They alert the group and come to
the table to address the issue. Everyone appreciates their input, and by
joining the table, they physically become part of the majority.
Now, imagine another fire starts under the table with everyone seated.
Those at the table, including the previously "marginal" individual,
can't detect this new threat. Their once unique position is lost, and
they're now part of the group that's unaware of the new danger.
It's crucial to note that even this scenario is relative. To another
group or from a broader perspective, everyone at this table could be
considered marginal. This underscores the importance of context: a
marginal position in one setting may be quite common in another.
This relativity is particularly relevant when considering claims of
marginalization within specific communities or projects. Even if
individuals are marginalized in broader society, they may hold majority
or influential positions within a particular project or community. In
such cases, their claims of marginalization within that specific context
may not be accurate or relevant.
In essence, marginalization is a temporary state, not a fixed identity.
It's fluid and can shift with changing situations and contexts,
highlighting the importance of diverse perspectives and the danger of
assuming any one group always holds a privileged viewpoint or unique
insight in all settings.
The misuse of marginalization claims has serious consequences.
Individuals wield this notion of perpetual marginalization not only to
speak for others, but also to justify a degradation of professional
standards. This false moral authority has become a shield for behavior
that would otherwise be unacceptable, leading us to examine the pitfalls
of such unchecked conduct.
## The Pitfall of Unchecked Behavior and False Marginalization
Traditionally, public displays of childish behavior were not tolerated
in professional settings. Recently, however, a troubling trend has
emerged: the justification of bullying behavior based on claimed
marginalized status. This justification often escalates rapidly,
creating untenable situations.
Crucially, these individuals are exploiting an identity that lacks a
concrete, technical definition. They are not inherently or permanently
marginalized; rather, they're hiding behind a facade to maintain special
privileges, particularly the ability to "shout down" others without
consequence.
This false claim of static marginalization ignores the contextual and
temporal nature of marginalization we discussed earlier. It allows
certain individuals or groups to maintain a position of perceived moral
authority, even when they've become part of, or aligned with the
majority. This misuse of claimed status creates an environment where
bullying is not only tolerated but sometimes even encouraged, as long as
it comes from the "right" sources.
Such behavior undermines the principles of professionalism, open
dialogue, and merit-based contribution that should be the hallmarks of
any healthy community, especially in technical fields. It's essential
to recognize and address this manipulation to maintain a truly fair and
productive environment.
## A Call for Maturity and Productive Incentives
As an adult and parent, such behavior is more disappointing than
surprising. Society might benefit from being more forgiving of mistakes,
allowing for course correction. In political terms, both sides often
have valid points and could learn from each other if they moved past
superficial differences. We should encourage more mature, productive
motivations, especially in contexts where many are willing to
collaborate constructively.
Importantly, we must consider the role of incentives in shaping
behavior. Creatures, including humans, are primarily motivated by
incentive structures. It's crucial not to inadvertently reward or
empower those who engage in divisive, derogatory, or unproductive
behavior, as this can quickly lead to a self-reinforcing cycle of
negative actions.
Thankfully, the solution is straightforward: we simply need to
incentivize civilized behavior. Instead of discouraging constructive
engagement by labeling it as "sea lioning" or "concern trolling," we can
cultivate an environment that rewards respectful disagreement and
collaborative problem-solving irrespective of personal or political
differences.
The alternative and apparent status quo seems to be a perpetual
witch-hunt for an ever-growing list of "wrong" opinions. Surely it is
clear which strategy is more sustainable?
## The Dangers of History Modification
The core issue lies in social manipulation through selective moderation
and, crucially, the modification of historical records. When moderation
teams, often claiming to represent marginalized groups, are empowered to
alter, delist, or delete past conversations, posts, or decisions, they
gain the ability to distort the narrative. This practice is, by
definition, a form of rewriting history.
By condoning or failing to address poor behavior from those claiming
marginalized status, these moderators further enable and entrench the
misuse of such claims. This creates a self-reinforcing cycle of
manipulation and degraded standards, undermining the integrity of
discourse and eroding trust among members.
A relevant example in the Nix project involves Jon Ringer, a
long-standing contributor and public figure. Recent controversies have
portrayed Jon as an instigator and "mean offensive person." However, a
more balanced view reveals him as a scapegoat for broader project
tensions. Crucially, Jon was permanently banned from the project, while
many who openly degraded him and made ad hominem attacks on his character
faced no consequences. This stark contrast highlights the uneven
application of moderation standards.
While Jon, like anyone, may have had imperfect moments under extreme
pressure, these pale in comparison to the systematic narrative
manipulation by others. The situation exposes a coordinated effort to
distort facts, issue threats, and employ bullying tactics to control the
project's direction.
The issue isn't that Jon was never wrong, but that he was consistently
painted as the primary instigator, regardless of reality. Even when
heated, Jon generally avoided the name-calling and derogatory behavior
that often went unchecked from other parties. His permanent ban, in this
context, underscores the troubling double standard at play in the
project's governance.
This manipulation of context and conversation history not only
misrepresents the overall dynamics but also serves to gaslight both
individuals and the wider perspective. The impact of this distortion is
evident in public platforms like Reddit, where observers unfamiliar with
Jon often express views that align with the manipulated narrative. These
casual observers, swayed by the dominant portrayal and the absence of
meaningful dissenting arguments, tend to perceive Jon as a far greater
problem than he actually was.
Crucially, while Jon may have contributed to some tension, he is far from
the epicenter of the controversy. In fact, the current issues surrounding
him have been brewing for years, consistently instigated by the same
individuals who have largely escaped scrutiny as they continue to
perpetuate divisive narratives.
The most tangible and regrettable outcome of this scapegoating is that
the Nix project has lost a long-standing, highly productive, and
professional contributor. Jon was often very helpful to newcomers, and
his departure represents a significant loss to the project. This
illustrates the real cost of allowing manipulated narratives to drive out
valuable members.
Such power to modify history is dangerous. It allows for the erasure of
context, the silencing of dissenting voices, and the creation of a false
consensus. This not only undermines transparency but also erodes trust
within project spaces and misleads those on the periphery. Setting a
clear precedent against this practice is vital. We must recognize that
allowing any group to "clean up" or selectively edit the historical
record is tantamount to endorsing propaganda[^1]. True professionalism in
project management involves facing our history honestly, learning
from it, and moving forward transparently.
Solving these problems requires strong leadership with a commitment to
preserving the integrity of shared discourse. Leaders must establish
clear principles that prioritize transparency and resist the temptation
to sanitize the past. While challenging, this approach is essential for
maintaining fairness, fostering genuine progress, and building a
trustworthy environment.
## The Solution: Embracing Leadership Principles
The real solution to Nix's or any project suffering from such childish
incursion lies in embracing fundamental principles of leadership. Being
a genuinely good leader is challenging. It requires holding oneself to a
higher standard than everyone else, and having the courage and conviction
to guide others to be their best selves, even when they resist.
Good leadership is the only way to be fair to all sides when there is a
genuine disagreement. It involves:
1. Setting clear, unambiguous goals and standards of behavior that align
with the project's core values. This clarity respects everyone's time,
allowing individuals to easily decide whether they align with and wish
to participate in the project.
2. Maintaining transparency and resisting the urge to manipulate
historical records.
3. Fostering respectful, merit-based dialogue while considering the
silent majority, not just vocal special interests.
4. Making decisions based on technical merit and the project's best
interests, not personal or ideological biases.
5. Being willing to address conflicts directly and fairly, without
scapegoating individuals or giving special privileges to allies.
6. Consistently enforcing these standards, making it clear what kind of
behavior and contributions are valued in the project.
By embracing these leadership principles, any project can create an
environment where technical excellence and collaborative spirit thrive.
It's a path that requires courage and commitment but offers the best hope
for resolving current tensions and preventing future ones.
However, implementing these principles requires a conscious choice from
all contributors, especially from those who have remained silent until
now.
## The Great Purge or Professionalism?
The Nix project faces a critical juncture. A long-standing moderator has
publicly expressed a desire for a ["purge"][purge] of supposed
"undesirables." This stark reality forces us to confront a fundamental
choice: do we embrace professionalism and mutual respect, or do we
allow divisive, exclusionary behavior to dominate and ultimately derail
the entire project?
This isn't just about Nix; it's a choice many now face. The silent
majority, those who typically avoid controversy, may now have to decide
what kind of project space they want to cultivate, and what sort of
leaders they wish to follow. Inaction is itself a choice; one that may
lead to the continued erosion of the project's ethic.
We must ask ourselves: Do we want a forum driven by technical merit and
collaborative spirit, or one ruled by ideological purity? The answer to
this question will shape the future of Nix and could set a precedent for
open-source projects at large.
It's time for those who value professionalism, open collaboration, and
technical excellence to stand up and be counted. The alternative - an
ecosystem stifled by ideological cleansing - is too high a price to pay
for our silence.
While this piece has focused on Nix, the issues discussed are
symptomatic of a growing and worrying trend across the open-source
world. Many projects face similar challenges with ideological divisions,
manipulated narratives, and the silencing of dissenting voices.
Open source is far too important to be ruled by narrow-minded and
exclusionary ideologies. By embracing strong leadership principles and
fostering environments of mutual respect and professionalism, we can
ensure that open source continues to thrive as a bastion of innovation
and collaboration.
[175]: https://github.com/NixOS/rfcs/pull/175
[purge]: https://chaos.social/@hexa/112711384631096150
[^1]: https://en.wikipedia.org/wiki/Historical_negationism

270
src/content/blog/NixOS-Flakes_and_KISS.md Normal file
View File

@@ -0,0 +1,270 @@
---
title: NixOS, Flakes and KISS
description: A simpler way to manage the OS Layer
tags:
- nix
author: Tim D
authorGithub: nrdxp
authorImage: https://avatars.githubusercontent.com/u/34083928?v=4
authorTwitter: nrdxp52262
date: "2020-12-19"
category: dev
---
## Introduction
This marks the first post of my very own developer blog, and it comes much later
than I had originally anticipated thanks to the ongoing pandemic, coupled with
some unforeseen life challenges. My original intent was to start by introducing
the concept of Nix Flakes, however, an excellent blog series over at
[tweag.io](https://www.tweag.io/blog/2020-05-25-flakes) has emerged, expanding
on just that premise. If you are new to flakes, it is highly recommended that
you check it out before continuing with this post.
Now, I'd like to introduce a project I've been slowly building up since
flakes were introduced called [DevOS][DevOS].
# So what is it anyway?
After years of working with NixOS, I strongly felt that the community as a whole
could benefit from a standardized structure and format for NixOS configurations
in general. It appears that every developer is essentially reinventing the
wheel when it comes to the "shape" of their deployments, leading to a lot of
confusion as to what the idioms and best practices should be, especially for
newcomers.
Having a mind share to collect the best ideas concerning structure and
method would be valuable, not only for its pragmatic implications, but also to
help ease adoption and onboarding for new NixOS users; something that has
traditionally been difficult up to now.
Of course this really hinges on wider community support, as my ideas alone
definitely shouldn't be the final word on what constitutes a correct and well
organized NixOS codebase. Rather, I am hoping to cajole the community forward
by providing useful idioms for others to expand on.
Even if my ideas lose out in the end, I sincerely hope they will, at the very
least, push the community toward some level of consensus in regards to the way
NixOS code repositories are structured and managed.
That said, DevOS appears to be gaining a bit of popularity among new flake
adopters and I am really quite excited and humbled to see others engage the
repository. If you have contributed to the project, thank you so much for your
time and support!
# An Arch KISS
I moved over to NixOS after a decades long love affair with Arch Linux. I found
their brand of KISS to be pragmatic and refreshing compared to alternatives
such as Ubuntu or Red Hat. This isn't to dog on those distributions, which I
also have used and enjoyed for years, but rather to accentuate my affection
for the simplified, and developer focused workflow that Arch Linux enabled for
my work stations.
However, over the years, I came to resent the several hours of tedious work
spent doing what amounted to the same small tasks over and over, any time
issues arose.
My first attempt to alleviate some of this work was by using Ansible to deploy
my common configuration quickly whenever it became necessary. However, I ran
into a ton of issues as the Arch repositories updated, and my configurations
inevitably became stale. Constant, unexpected breakage became a regular
nuisance.
I then became aware of Nix and NixOS, and hoped that it would live up the
promise of reproducible system deployment, and after a brief stint of
procrastination, I dove head first.
# Great but Not Perfect.
At first everything seemed almost perfect. NixOS felt like Ansible on steroids,
and there was more than enough code available in nixpkgs to meet my immediate
needs. Getting up to speed on writing derivations and modules was fairly
straightforward and the DevOps dream was in sight.
It wasn't all sunshine and rainbows, as channel updates sometimes caused
the same sort of breakage I moved to NixOS to avoid. But simple generation
rollbacks were a much more welcome interface to this problem than an unbootable
system. It was a measurable improvement from the busy work experienced with Arch. All in all, I felt it was
well worth the effort to make the transition.
It wasn't long before the [rfc][rfcs] that eventually became flakes emerged.
It seemed like the solution to many of my few remaining gripes with my
workflow. An officially supported and simple way to lock in a specific revision
of the entire system. No more unexpected and unmanaged breakage!
Of course it took a while for an experimental implementation to arrive, but I
found myself digging into the Nix and Nixpkgs PR's to see how flakes worked
under the hood.
Around the same time, the ad hoc nature of my NixOS codebase was starting to
bug at me, and I wanted to try my hand at something more generalized and
composable across machines. I had a first iteration using the traditional
"configuration.nix", but ended up feeling like the whole thing was more
complex than it really needed to be.
My eagerness to get started using flakes was the perfect excuse to start from
scratch, and so began DevOS. An attempt to address my concerns, using flakes.
## How does it work?
First and foremost, I want to point out that the bulk of the credit goes to the
amazing engineer's who have designed and implemented Nix and the ecosystem
as a whole over the last decade.
I see a lot of new users struggling to dive in and get up to speed with the Nix
language, and particularly, getting up and running with a usable and productive
system can take some serious time. I know it did for me.
The hope for DevOS is to alleviate some of that pain so folks can get to
work faster and more efficiently, with less frustration and more enthusiasm for
the power that Nix enables. I especially don't want anyone turning away from
our amazing ecosystem because their onboarding experience was too complex
or overwhelming.
# Everything is a profile!
At the heart of DevOS is the [profile][profiles]. Of course, these profiles
are really nothing more than good ol' NixOS [modules][modules]. The only reason
I've decided to rebrand them at all is to draw a distinction in how they are
used. They are kept as simple as possible on purpose; if you understand modules
you don't _really_ have anything new to learn.
The only limitation is that a profile should never declare any new NixOS module
options, we can just use regular modules for that elsewhere. Instead, they
should be used to encapsulate any configuration which would be useful for more
than one specific machine.
To put it another way, instead of defining my entire NixOS system in a
monolithic module, I break it up into smaller, reusable profiles which can
be themselves be made up of profiles. Composability is key here, as I don't
necessarily want to use every profile on every system I deploy.
As a concrete example, my [develop][develop], profile pulls in my preferred
developer tools such as my shell, and text editor configurations. It can be
thought of as a meta-profile, made up of smaller individual profiles. I can
either pull in the whole thing, which brings all the dependent profiles along
with it, or I can just import a single profile from within, say my zsh
configuration, leaving all the rest unused. Every profile is a directory with
a "default.nix" defining it. You can have whatever else you need inside the
folder, so long as it is directly related to the profile.
Let's draw the obvious parallel to the Unix philosophy here. Profiles work
best when they do one thing, and do it well. Don't provision multiple programs
in one profile, instead split them up into individual profiles, and then if you
often use them together, import them both in a parent profile. You can simply
import dependent profiles via the "imports" attribute as usual, ensuring
everything required is always present.
The key is this, by simply taking what we already know, i.e. NixOS modules, and
sticking to the few simple idioms outlined above, we gain composability and
reusability without actually having to learn anything new. I want to drill this
point home, because that's really all there is to DevOS!
Besides a few simple convenience features outlined below, profiles are the star
of the show. It's really nothing revolutionary, and that's on purpose! By
keeping things simple and organized we gain a level of control and granularity
we wouldn't have otherwise without adding real complexity to speak of.
# Really? Everything?
Yes! Thanks to built in [home-manager][home-manager] integration, users are
profiles, a preferred graphical environment is a profile. Anything that you
could imagine being useful on more than one machine is a profile. There are
plenty of examples available in the "profiles" and "users" directories, and
you can check out my personal "nrd" branch, if you want to see how I do things
on my own machines.
# Anything else I should know?
As mentioned briefly above, DevOS also has some convenience features to make
life easier.
For starters, you might be wondering how we actually define a configuration for
a specific machine. Simple, define the machine specific bits in a nix file
under the [hosts][hosts] directory and import any relevant profiles you wish to
use from there. The flake will automatically import any nix files in this folder as NixOS
configurations available to build. As a further convenience, the hostname of
your system will be set to the filename minus the ".nix" extension. This makes
future "nixos-rebuilds" much easier, as it defaults to looking up your current
hostname in the flake if you don't specify a configuration to build explicitly.
Now what if we actually just want to define a NixOS module that does declare
new NixOS options, you know, the old fashioned way? We'll also want to define
our own pkgs at some point as well. These are both structured closely to how
you might find them in the nixpkgs repository itself. This is so that you can
easily bring your package or module over to nixpkgs without much modification
should you decide it's worth merging upstream.
So, you'd define a package or module the exact same way you would in nixpkgs
itself, but instead of adding it to all-packages.nix or module-list.nix, you add
it to pkgs/default.nix and modules/list.nix. Anything pulled in these two files
will become available in any machine defined in the hosts directory, as well as
to other flakes to import from DevOS!
This setup serves a dual purpose. For people who already know the nixpkgs
workflow, it's business as usual, and for individuals who aren't familiar with
nixpkgs but wish to become so, they can quickly get up to speed on how to add
packages and modules themselves, in the exact same way they would do so upstream
proper.
Now what about overlays? Well, any overlay defined in a nix file under the
overlays directory will be automatically imported, just as with packages and
modules, and are available to all hosts, as well as to other flakes.
What if I want to pull a specific package from master instead of from the
stable release? There is a special file, pkgs/override.nix. Any package listed
here will be pulled from nixpkgs unstable rather than the current stable release.
Simple, easy.
What about cachix? It's super easy to add your own cachix link just as you
would a regular NixOS configuration. As a bonus, it will be wired up as a flake
output so other people can pull in your link directly from your flake! My
personal cachix repo is setup by default. It provides the packages the flake
exports so you don't have to build them.
That should just about do it for DevOS's current quality of life features, but
there are more ideas brewing.
# What's next?
I'm working on a system for seamlessly importing modules, packages and
overlays from other flakes, which isn't too hard as it is, but it's messy
because the current "flake.nix" has a lot of business logic that gets in the way.
Also, I would like to start programmatically generating documentation for
everything. So users can quickly find what goes where and not have to read
drawn out blog posts like this to get started. 😛 Nixpkgs is currently
transitioning to CommonMark for all documentation, and we will probably follow
suite.
Additionally, I want to implement an easy way to actually install NixOS on the
bare metal from directly within the project. I know the [deploy-rs][deploy-rs]
project is working on this, and I'm interested in supporting their project
in DevOS so as to add extra flexibility and power to installation and
deployment!
Also, certain parts of the flake should be tested to ensure things don't break.
We really have no tests to speak of as is. The auto import functions for the
"hosts" and "overlays" directory are good examples.
## A call to arms!
If you'd like to help, please jump in. I am very much open to any ideas that
could reduce the complexity or simplify the UI. If you have a profile you
believe would be useful to others, please open a [Pull Request][pr].
If you think I am crazy and wasting my time, please don't hesitate to say so! I
typically find critical feedback to be some of the most helpful. Most of all,
if you made it this far, thanks for taking some time to read about my efforts
and please consider giving DevOS a shot!
[nix]: https://nixos.org
[DevOS]: https://github.com/divnix/DevOS
[rfcs]: https://github.com/NixOS/rfcs
[modules]: https://nixos.org/manual/nixos/stable/index.html#sec-writing-modules
[profiles]: https://github.com/divnix/devos/tree/template/profiles
[develop]: https://github.com/divnix/devos/tree/template/profiles/develop
[hosts]: https://github.com/divnix/devos/tree/template/hosts
[deploy-rs]: https://serokell.io/blog/deploy-rs
[home-manager]: https://github.com/nix-community/home-manager
[pr]: https://github.com/divnix/devos/pulls

182
src/content/blog/std-action.md Normal file
View File

@@ -0,0 +1,182 @@
---
title: Standard Action
description: Do it once, do it right.
tags:
- std
- nix
- devops
- github actions
author: Tim D
authorGithub: nrdxp
authorImage: https://avatars.githubusercontent.com/u/34083928?v=4
authorTwitter: nrdxp52262
date: "2022-12-09"
category: dev
---
## CI Should be Simple
As promised in the [last post](./std), I'd like to expand a bit more on what we've
been working on recently concerning Nix & Standard in CI.
At work, our current GH action setup is rather _ad hoc_, and the challenge of optimizing that path
around Nix’s strengths lay largely untapped for nearly a year now. Standard has helped somewhat
to get things organized, but there has been a ton of room for improvement in the way tasks are
scheduled and executed in CI.
[Standard Action][action] is our answer. We have taken the last several months of brainstorming
off and on as time allows, experimenting to find a path that is versatile enough to be useful
in the general case, yet powerful enough for organizations who need extra capacity. So without
any further stalling, let's get into it!
## The Gist
The goal is simple, we want a CI system that only does work once and shares the result from there.
If it has been built or evaled before, then we want to share the results from the previous run
rather than start from scratch.
It is also useful to have some kind of metadata about our actions, which we can use to build
matrices of task runners to accomplish our goals. This also allows us to schedule builds on
multiple OS trivially, for example.
Task runners shouldn't have to care about Nix evaluation at all, they should just be able to get
to work doing whatever they need to do. If they have access to already reified derivations, they
can do that.
So how can we accomplish this? Isolate the evaluation to its own dedicated "discovery" phase, and
share the resulting /nix/store and a json list describing each task and its target derivations.
From there it's just a matter of opimizing the details based on your usecase, and to that end we
have a few optional inputs for things like caching and remote building, if you are so inclined.
But you can do everything straight on the runner too, if you just need the basics.
## How it Works
Talking is fine, but code is better. To that end, feel free to take a look at my own personal CI
for my NixOS system and related packages: [nrdxp/nrdos/ci.yml][nrdos].
What is actually evaluated during the discovery phase is determined directly in the
[flake.nix][ci-api].
I am not doing anything fancy here at the moment, just some basic package builds, but that is
enough to illustrate what's happening. You can get a quick visual by look at the summary of
a given run: [nrdxp/nrdos#3644114900](https://github.com/nrdxp/nrdos/actions/runs/3644114900).
You could have any number of matrices here, one for publishing OCI images, one for publishing
documentation, one for running deployments against a target environment, etc, etc.
Notice in this particular example that CI exited in 2 minutes. That's because everything
represented by these builds is already cached in the specified action input `cache`, so no work is
required, we simply report that the artifacts already exist and exit quickly.
There is a run phase that typically starts after this build step which runs the Standard action,
but since the "build" actions only duty is building, it is also skipped here.
This is partially enabled by use of the GH action cache. The cache key is set using the following
format: [divnix/std-action/discover/action.yml#key][key]. Coupled with the guarantees nix already
gives us, this is enough to ensure the evaluation will only be used on runners using a matching OS,
on a matching architecture and the exact revision of the current run.
This is critical for runners to ensure they get an exact cache hit on start, that way they pick
up where the discovery job left off and begin their build work immediately, acting directly
on their target derivation file instead of doing any more evaluation.
## Caching & Remote Builds
Caching is also a first class citizen, and even in the event that a given task fails (even
discovery itself), any of its nix dependencies built during the process leading up to that failure
will be cached, making sure no nix build _or_ evaluation is ever repeated. The user doesn't have
to set a cache, but if they do, they can be rest assured their results will be well cached, we
make a point to cache the entire build time closure, and not just the runtime closure, which is
important for active developement in projects using a shared cache.
The builds themselves can also be handed off to a more powerful dedicated remote builder. The
action handles remote builds using the newer and more efficient remote store build API, and when
coupled with a special purpose service such as [nixbuild.net](https://nixbuild.net), which your
author is already doing, it becomes incredibly powerful.
To get started, you can run all your builds directly on the action runner, and if that becomes
a burden, there is a solid path available if and when you need to split out your build phase to a
dedicated build farm.
## Import from What?
This next part is a bit of an aside, so feel free to skip, but the process outlined above just so
happened to solve an otherwise expensive problem for us at work, outlining how thinking through
these problems carefully has helped us improve our process.
IOG in general is a bit unique in the Nix community as one of the few heavy users of Nix’s IFD
feature via our [haskell.nix][haskell] project. For those unaware, IFD stands for
"import from derivation" and happens any time the contents of some file from one derivations output
path is read into another during evaluation, say to read a lock file and generate fetch actions.
This gives us great power, but comes at a cost, since the evaluator has to stop and build the
referenced path if it does not already exist in order to be able to read from it.
For this reason, this feature is banned from inclusion in nixpkgs, and so the tooling used there
(Hydra, _et al._) is not necessarily a good fit for projects that do make use of IFD to some extent.
So what can be done? Many folks would love to improve the performance of the evaluator itself, your
author included. The current Nix evaluator is single threaded, so there is plenty of room for
splitting this burden across threads, and especially in the case of IFD, it could theoretically
speed things up a great deal.
However, improving the evaluator performance itself is actually a bit of a red herring as far as
we are concerned here. What we really want to ensure is that we never pay the cost of any given Nix
workload more than once, no matter how long it takes. Then we can ensure we are only ever
building on what has already been done; an additive process if you will. Without careful
consideration of this principle beforehand, even a well optimized evaluator would be wasting cycles
doing the same evals over and over. There is the nix flake evalulation cache, but it comes with
a few [caveats][4279] on its own and so doesn't currently solve our problem either.
To give you some numbers, to run a fresh eval of my current project at work takes 35 minutes from a
clean /nix/store, but with a popullated /nix/store from a previous run it takes only 2.5 minutes.
Some of the savings is eaten up by data transfer and compression, but the net savings are still
massive.
I have already begun brainstorming ways we could elimnate that transfer cost entirely by introducing
an optional, dedicated [evaluation store](https://github.com/divnix/std-action/issues/10) for those
who would benefit from it. With that, there is no transfer cost at all during discovery, and the
individual task runners only have to pull the derivations for their particular task, instead of the
entire /nix/store produced by discovery, saving a ton of time in our case.
Either way, this is a special case optimization, and for those who are content to stick with the
default of using the action cache to share evaluation results, it should more than suffice in the
majority of cases.
## Wrap Up
So essentially, we make due with what we have in terms of eval performance, focus on ensuring we
never do the same work twice, and if breakthroughs are made in the Nix evaluator upstream at some
point in the future, great, but we don't have to wait around for it, we can minimize our burden
right now by thinking smart. After all, we are not doing Nix evaluations just for the sake of it,
but to get meaningful work done, and doing new and interesting work is always better than repeating
old tasks because we failed to strategize correctly.
If we do ever need to migrate to a more complex CI system, these principles themeselves are all
encapsulated in a few fairly minimal shell scripts and could probably be ported to other
systems without incredible effort. Feel free to take a look at the source to see what's really
goin on: [divnix/std-action](https://github.com/divnix/std-action).
There are some places where we could use some [help][7437] from [upstream][2946], but even then, the
process is efficient enough to be a massive improvement, both for my own personal setup, and for
work.
As I mentioned in the previous post though, Standard isn't just about convenience or performance,
but arguable the most important aspect is to assist us in being _thorough_. To ensure all
our tasks are run, all our artifacts are cached and all our images are published is no small feat
without something like Standard to help us automate away the tedium, and thank goodness for that.
For comments or questions, please feel free to drop by the official Standard [Matrix Room][matrix]
as well to track progress as it comes in. Until next time...
[action]: https://github.com/divnix/std-action
[haskell]: https://github.com/input-output-hk/haskell.nix
[nrdos]: https://github.com/nrdxp/nrdos/blob/master/.github/workflows/ci.yml
[key]: https://github.com/divnix/std-action/blob/6ed23356cab30bd5c1d957d45404c2accb70e4bd/discover/action.yml#L37
[7437]: https://github.com/NixOS/nix/issues/7437
[3946]: https://github.com/NixOS/nix/issues/3946#issuecomment-1344612074
[4279]: https://github.com/NixOS/nix/issues/4279#issuecomment-1343723345
[matrix]: https://matrix.to/#/#std-nix:matrix.org
[ci-api]: https://github.com/nrdxp/nrdos/blob/66149ed7fdb4d4d282cfe798c138cb1745bef008/flake.nix#L66-L68

217
src/content/blog/std.md Normal file
View File

@@ -0,0 +1,217 @@
---
title: From DevOS to Standard
description: Why we made Standard, and what it has done for us.
tags:
- std
- nix
- devops
author: Tim D
authorGithub: nrdxp
authorImage: https://avatars.githubusercontent.com/u/34083928?v=4
authorTwitter: nrdxp52262
date: "2022-10-31"
category: dev
---
## Update: A Video is Worth 1000 Blogs
For those who would rather watch than read, a colleague of mine has whipped up a great video series
exploring Standard in depth, so drop by the [media secition](../media) for links.
## Two years later...
DevOS started as a fun project to try and get better with Nix and understand this weird new thing
called flakes. Since then and despite their warts, Nix flakes have experienced widespread use, and
rightfully so, as a mechanism for hermetically evaluating your system & packages that fully locks
your inputs and guarantees you some meaningful level of sanity over your artifacts.
Yet when I first released it, I never even imagined so many people would find DevOS useful, and I
have been truly humbled by all the support and contributions that came entirely spontaneously to the
project and ultmately culminated in the current version of [digga][digga], and the divnix org that
maintains it.
## Back to Basics
For whatever reason, it really feels like time to give a brief update of what has come of this
little community experiment, and I'm excited to hopefully clear up some apparent confusion, and
hopefully properly introduce to the world [Standard](https://github.com/divnix/std).
DevOS was never meant to be an end all be all, but rather a heavily experimental sketch while
I stumbled along to try and organize my Nix code more effectively. With Standard, we are able to
distill the wider experience of some of its contributors, as well as some new friends, and design
something a little more focused and hopefully less magical, while still eliminating a ton of
boilerplate. Offering both a lightly opinionated way to organize your code into logically typed
units, and a mechanism for defining "standard" actions over units of the same type.
Other languages make this simple by defining a module mechanism into the language where users are
freed from the shackles of decision overload by force, but Nix has no such advantage. Many people
hoped and even expected flakes to alleviate this burden, but other than the schema Nix expects
over its outputs, it does nothing to enforce how you can generate those outputs, or how to organize
the logical units of code & configuration that generate them.
## A Departure from Tradition
It is fair to say that the nixpkgs module system has become the sort of "goto" means of managing
configuration in the Nix community, and while this may be good at the top-level where a global
namespace is sometimes desirable, it doesn't really give us a generic means of sectioning off our
code to generate both configuration _and_ derivation outputs quickly.
In addition to that, the module system is fairly complex and is a bit difficult to anticate the
cost of ahead of time due to the fixed-point. The infamous "infinite traces" that can occur during
a Nix module evaluation almost never point to the actual place in your code where the error
originates, and often does even contain a single bit of code from the local repository in the trace.
Yet as the only real game in town, the module system has largely "de facto" dictated the nature
of how we organize our Nix code up til now. It lends itself to more of a "depth first" approach
where modules can recurse into other modules ad infinitum.
## A Simpler Structure
Standard, in contrast, tries to take an alternative "breadth first" approach, ecouraging code
organization closer to the project root. If true depth is called for, flakes using Standard can
compose gracefully with other flakes, whether they use Standard or not.
It is also entirely unopionated on what you output, there is nothing stopping you from simply
exporting NixOS modules themselves, for example, giving you a nice language level
compartmentalization strategy to help manager your NixOS, Home Manager or Nix Darwin configurations.
Advanced users may even write their own types, or even extend the officially supported ones. We
will expand more on this in a later post.
But in simple terms, why should we bother writing the same script logic over and over when we can be
guaranteed to recieve an output of a specific type, which guarantees any actions we define for the
type at large will work for us: be it deploying container images, publishing sites, running
deployments, or invoking tests & builds.
We can ensure that each image, site, or deployment is tested, built, deployed and published in
a sane and well-defined way, universally. In this way, Standard is meant to not only be convenient,
but comprehensive, which is an important property to maintain when codebases grow to non-trivial
size.
There is also no fixed-point so, anecdotably, I have yet to hit an eval error in Standard based
projects that I couldn't quickly track down; try saying that about the module system.
## A CLI for productivity
The Nix cli can sometimes feel a little opaque and low-level. It isn't always the best interface
to explain and explore what we can actually _do_ with a given project. To address this issue in
a minimal and clean way, we package a small go based cli/tui combo to quickly answer exactly this
question, "What can I do with this project?".
This interface is entirely optional, but also highly useful and really rather trivial thanks to a
predicatable structure and well typed outputs given to us in the Nix code. The schema for anything
you can do follows the same pattern: "std //$cell/$block/$target:$action". Here the "cell" is the
highest level "unit", or collection of "blocks", which are well-typed attribute sets of "targets"
sharing a colleciton of common "actions" which can be performed over them.
### At a Glance
The TUI is invaluable for quickly getting up to speed with what's available:
```console
┌────────────────────────────────────────────────────────────────────────────────┐┌───────────────────────────────────┐
│| Target ││ Actions │
│ ││ │
│ 176 items │││ build │
│ │││ build this target │
│ //automation/packages/retesteth ││ │
│ testeth via RPC. Test run, generation by t8ntool protocol ││ run │
│ ││ exec this target │
││ //automation/jobs/cardano-db-sync ││ │
││ Run a local cardano-db-sync against our testnet ││ │
│ ││ │
│ //automation/jobs/cardano-node ││ │
│ Run a local cardano-node against our testnet ││ │
│ ││ │
```
## A Concise Show & Tell
The central component of Standard is the cell block API. The heirarchy is "cell"→"block", where
we defined the individual block types and names directly in the flake.nix.
The function calls in the "cellBlocks" list below are the way in which we determine which "actions"
can be run over the contents of the given block.
```nix
# flake.nix
{
inputs.std.url = "github:divnix/std";
outputs = inputs: inputs.std.growOn {
inherit inputs;
systems = ["x86_64-linux"];
# Every file in here should be a directory, that's your "cell"
cellsFrom = ./nix;
# block API declaration
cellBlocks = [
(std.functions "lib")
(std.installables "packages")
(std.devshells "devshells")
];
};
}
# ./nix/dev/packages.nix
# nix build .#$system.dev.packages.project
# std //dev/packages/project:build
{
inputs, # flake inputs with the `system` abstracted, but still exposed when required
cell # reference to access other blocks in this cell
}: let
inherit (inputs.nixpkgs) pkgs;
in
{
project = pkgs.stdenv.mkDerivation {
# ...
};
}
# ./nix/automation/devshells/default.nix
# nix develop .#$system.dev.devshells.dev
# std //automation/devshells/dev:enter
{
inputs,
cell
}: let
inherit (inputs) nixpkgs std;
inherit (nixpkgs) pkgs;
# a reference to other cells in the project
inherit (inputs.cells) packages;
in
{
dev = std.mkShell { packages = [packages.project]; };
}
```
## Encouraging Cooperation
Standard has also given us a useful mechanism for contributing back to upstream where it makes
sense. We are all about maintaining well-defined boundaries, and we don't want to reimplement the
world if the problem would be better solved elsewhere. Work on Standard has already led to several
useful contributions to both nixpkgs and even a few in nix proper, as well as some in tangentially
related codebases, such as github actions and go libraries.
One very exciting example of this cooperation is the effort we've expended integrating
[nix2container][n2c] with Standard. The work has given us insights and position to begin defining an
officially supported specification for [OCI images][oci] built and run from Nix store paths, which
is something that would be a huge win for developers everywhere!
We believe interoperability with existing standards is how Nix can ultimately cement itself into
the mainstream, and in a way that is unoffensive and purely additive.
## CI simplified
Instead of making this a mega post, I'll just leave this as a bit of a teaser for a follow-up post
which will explore our recent efforts to bring the benefits Standard to GitHub Actions _a la_
[std-action][action]. The target is a Nix CI system that avoids ever doing the same work more than
once, whether its evaluating or building, and versatile enough to work from a single user project
all the way up to a large organization's monorepo. Stay tuned...
[digga]: https://github.com/divnix/digga
[nosys]: https://github.com/divnix/nosys
[action]: https://github.com/divnix/std-action
[grow]: https://std.divnix.com/guides/growing-cells.html
[harvest]: https://github.com/divnix/std/blob/main/src/harvest.nix
[n2c]: https://github.com/nlewo/nix2container
[oci]: https://github.com/opencontainers/image-spec/issues/922

21
src/content/config.ts Normal file
View File

@@ -0,0 +1,21 @@
import { z, defineCollection } from "astro:content";
const blogCollection = defineCollection({
schema: z.object({
title: z
.string()
.max(100, "The title length must be less than or equal to 100 chars"),
description: z.string(),
tags: z.array(z.string()),
author: z.string(),
authorImage: z.string().optional(),
authorTwitter: z.string(),
date: z.string(),
image: z.string().optional(),
category: z.string(),
}),
});
export const collections = {
blog: blogCollection,
};