So, I've been using Thanos to receive and store my prometheus metrics long term in a self hosted S3 bucket. Thanos also acts as a datasource for my dashboards in Grafana, and provides a Ruler, which evaluates alerting rulers and forwards them to my alertmanager. It's ok. It's certainly got it's downsides, which I can go into later, but I've thinking... what about Mimir?

How do you all feel about Grafana's Mimir (source on GitHub)? It's AGPL and seems to literally be a replacement of Thanos, which is Apache 2.0.

Thanos description from their website:

Open source, highly available Prometheus setup with long term storage capabilities.

Mimir description from their website:

...open source software project that provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus and OpenTelemetry metrics.

Both with work with alloy and prometheus alike. Both require you to configure initially confusing hashrings and replication parameters. Both have a bunch of large companies adopting them, so... now I feel conflicted. Should I try mimir? Poll in reply.

Hello, hachyderm! we've been working hard on building up our ansible runbooks and improving hachyderm's overall resilience. Recently, we've been focusing on is database resilience.

We're getting close to retiring our original database server (finally!) and preparing to move to a fully ansible-managed set of databases servers, primary and replica on new hardware. We'll send another announcement when we do the cut over. The team has done excellent work to make this highly automated, quick, and painless!

Done:

author ansible roles for managing postgresql, pgbackrest (backups), pgbouncer, and primary/replica failover
decide to continue with pgbouncer and *not* use pgcat
rotate database passwords
order new replica database hardware
order new future primary database hardware

To do soon:

rebuild replica database with ansible scripts
prepare primary database with ansible scripts
start replicating to new database replica
cut over to new database server

We're also planning on open-sourcing our ansible roles in the coming weeks - just a little housekeeping & tidying up before we do!

hey, fediverse friends - i'm excited that we're finally announcing our Fediverse Security Fund over at @nivenly to help make fedi software more secure.

we're starting off super small to see if the Fund is a thing that can help. along the way we'll learn and improve our intake/payout process. and if there's solid interest and we see good impact, we'll hold a member vote near the end of the experiment to decide if we'll renew/expand the program.

thanks to @thisismissem for her contributions and being the first disclosure to validate the process.

let's close some vulns!

If you're at #KubeCon today (or the rest of the week?) do go say hi to the #CUE folks I help write their open-source docs (https://cuelang.org/docs) 'cos I *really* want the awesome tech to succeed!

If you're a #SysAdmin or #DevOps in #YAML config hell go and have a chat with them - next to the CNCF corner store at stall S761

And that concludes my job in infosec.
Tomorrow, I will be in IT working on Linux.
#infosec #cybersecurity #SRE #linux #foss

“This place can't function without me” #devops #sysadmin #sre #job

Pushing core workout lately and being rewarded with more mornings free of migraine.

I played deeply into my music the past few nights, awaking the next morning scrubbed of a migraine.

Having those who listen and witness allows me to let go of emotions when I am having them, not carry them around. Less migraine activity ensues.

This week I learned that my anxiety about others is entwined with a particularly evil symptom of religious trauma, I saw both but never saw hiw they were connected.

I can recognize it now. And the feeling of not needing to "save" someone is a really powerful emotion - or lack of one - that, today, I am thankful for contributing to a clear head and no migraine.

Also feeling self-assured that fixing failures in our systems look a lot more like treating a migraine than using quick-fixes and low-hanging-fruit.

So hand-wavy ; you still need #OnCall for critical services with defined SLAs :

“Breaking Up With On-Call”, Alexey Karandashev (https://reflector.dev/articles/breaking-up-with-on-call/).

Via HN: https://news.ycombinator.com/item?id=43378671

howdy, #hachyderm!

over the last week or so, we've been preparing to move hachy's #DNS zones from #AWS route 53 to bunny DNS.

since this could be a pretty scary thing -- going from one geo-DNS provider to another -- we want to make sure *before* we move that records are resolving in a reasonable way across the globe.

to help us to do this, we've started a small, lightweight tool that we can deploy to a provider like bunny's magic containers to quickly get DNS resolution info from multiple geographic regions quickly. we then write this data to a backend S3 bucket, at which point we can use a tool like #duckdb to analyze the results and find records we need to tweak to improve performance. all *before* we make the change.

then, after we've flipped the switch and while DNS is propagating -- -- we can watch in real-time as different servers begin flipping over to the new provider.

we named the tool hachyboop and it's available publicly --> https://github.com/hachyderm/hachyboop

please keep in mind that it's early in the booper's life, and there's a lot we can do, including cleaning up my hacky code.

attached is an example of a quick run across 17 regions for a few minutes. the data is spread across multiple files but duckdb makes it quite easy for us to query everything like it's one table.

hello, #SRE friends. looking for good resources for SRE-related content, specifically for #observability please share your fave resources, TY

Ugh slept like shit. The drama and stress from work is giving me insomnia. The boss who quit didn't give leave ANYTHING for my new boss to understand what I do with DevEx and incidents.

I had to bust my ass yesterday to basically hold the ground that I had worked extremely hard over the past four months to make it so that I could manage incidents. New boss wanted to yank me out of it and put me back on "SRE" infrastructure.

CTO says incidents are staying where they are, with Engineering. So I told him I want to transfer to Engineering.

And then I get the question "do want to be an SRE?"

Ridiculous. Makes me want to scream.

Saw this very nice article on slack about making standalone #python scripts that use #uv to automatically create their own virtual environments and install their dependencies there.

https://thisdavej.com/share-python-scripts-like-a-pro-uv-and-pep-723-for-easy-deployment/

#sre @sre #devops @devops

Some great #SRE war stories have been told this week at #srecon. Two of the most compelling were told by #Bluesky and #Squarespace engineers, as detailed here. #postmortem #googledomains #incidentpostmortem #projectpostmortem

ttps://www.techtarget.com/searchitoperations/news/366621615/BlueSky-Squarespace-pros-give-incident-project-postmortems

I really liked this informal community poll and thematic analysis on SLO usage. It does a better job at highlighting the hurdles to adopting them at a Company Who Is Not Google than a lot of "Here's how to do SLOs" pieces that just don't cover it.

If there is ever a "Seeking SLOs" book, this should be the first chapter.

https://ericmustin.substack.com/p/notes-on-service-level-objectives

Hey Go developers! We're still for hiring for a remote position in the US (needs to be in the US already)!
Feel free to reach out to me for a referral.
#Remote #RemoteWork #RemoteJobs #Go #Golang #SRE #Cloud #CloudComputing #FediHire
https://hachyderm.io/@badnetmask/114104286047688732