Show newer

After a couple of months spent working with #Amazon #Kinesis streams in my main job, I'm wondering why on earth anybody would use this sh*t over a #Kafka/#Mosquitto broker.

If you choose Kinesis, you get:

- Bad documentation

- Limited support when it comes to languages/SDKs

- Bad latency (Kinesis isn't really an asynchronous broker, it's an inefficient polling beast disguised as an asynchronous stream)

- Limited throughput (a shard can ingest a maximum of 1 MB/s)

- A heavily limited quota (you can get a maximum of 10 MB or 10,000 records per read, and you can only do a maximum of 5 reads per seconds, or retrieve 10,000 items every 5 seconds). Any quota extension requires an expensive product upgrade through an annoying "contact our customer support" process

- You can create a maximum of 50 data streams - any extension requires, again, the "contact Amazon support" bullsh*t

- An environment that is hard to replicate and test locally - you need to use their closed localstack Docker image, which breaks very frequently

- Additional configuration required to connect to the correct cloud service (region, environment, default topics that should be available etc.)

- Pay Amazon to provide you this shit

If you choose Kafka installed on your premises, you get:

- Top-level documentation and community support

- Support for virtually any programming language and tons of SDKs and libraries

- A true asynchronous broker, with a latency that is as good as the latency of your infrastructure

- A throughput as good as the throughput of your infrastructure/server

- No quota or restrictions whatsoever, unless you want to set them explicitly in your configuration

- You can create as many topics as you like. You don't need to spend time on phone with Amazon's customer service if you want to have more than 50 topics

- An environment that can be easily tested and replicated locally. Just pull and run a Kafka image, and you're good to go. No configuration about regions, clusters, shards and default topics is required, unless you need it or want it

- You don't have to pay anybody for anything, except the costs of running your own broker on your own servers. Everything is free and open-source

My company has realized the mistake it made by relying on some severely limited enterprise shit instead of readily available open-source solutions, and it's now in the process of migrating its data stream infrastructure to Kafka.

A lot of time has been wasted both by the infra and development teams in the meantime. A lot of business value has been held behind by a bad data stream product. A lot of developers got frustrated by all these unneeded constraints. A lot of time will be required to migrate the whole codebase to a new data infrastructure - we're talking of several months. All because the management was like "let's use this off-the-shelf AWS solution instead of running our data stream infrastructure using 'amateur' open-source products".

Amazon is literally producing copycats of open-source products that are stable and free for anyone to use. But those copycats are much worse, much more limited, and you have to pay money for them. Why on earth would anybody pay for this crap?

Are senior managers so dumb that if you sprinkle a bit of marketing and enterprise powder on top of some crap they're ready to pay for it (and even pay A LOT) rather than use something that is better and free?

Even the "it scales better" argument makes no sense. Amazon provides you the worst of the two worlds: you have to think of your whole application as working in an opaque cloud even when you're just running on your local machine, but real throughput is severely limited, the strict quotas mean that you won't be able to easily go from 100K to 10M records per day (let alone per hour) within a day, and every infrastructure upgrade to ensure that it can handle more traffic requires a human contacting another human at Amazon and requesting an upgrade.

This is the kind of shit that is holding back technological innovation today.

«Europe registered the fewest new passenger cars in the month of June since 1996 at just over 1.06 million vehicles, as problems from logistics bottlenecks to an ongoing chip shortage curbed production.»
reuters.com/business/autos-tra

For the longest time in Europe (IT/FI) most of my downloads from Internet Archive seemed to settle around 200 kB/s, now I get 50 MiB/s at home.

Show thread

Overjoyed to see that the just got more bandwidth!

After months running at full capacity (80 Gbps) now it's only at... some 90 % (of 140 Gbps). Download those big torrents while you can! (And keep seeding.)

Ho pubblicato in CC-0 un elenco dei circa 504 000 domini in cui ho trovato , con qualche dettaglio aggiuntivo.
doi.org/10.5281/zenodo.6815697

Show thread

Political optics.

July 2012: Italy pays up to 221 €/MWh to consumers for their solar energy. The lone minister Greens had in 2007 is widely ridiculed and branded a public menace.
escholarship.org/content/qt6nz

🇪🇺🇭🇷 Croatia will adopt the euro as of next year!

The country fulfils all convergence criteria, the Council of the European Union agreed today. Croatia is now formally approved to join the euro area on 1 January 2023.

Press release ecb.europa.eu/press/pr/date/20

🐦🔗: nitter.eu/ecb/status/154682020

In both desktop and mobile views, Twitter will nowadays show you a login prompt you can't bypass after a few seconds of browsing a profile. (That's not true for showing threads.) Services like nitter.it/ get around that, but frequently get hit by rate limits.

Bear that in mind when people talk about "the town square". It's a gated community.

L'articolo più lucido, profondo, umano e onesto che ho letto in questi mesi è l'analisi di Carlo #Rovelli sull'ipocrisia che circonda il militarismo europeo vestito di nobili ideali.

volerelaluna.it/rimbalzi/2022/

@EURACTIV@twitter.com 🇬🇧 Good news: The Commission is backing away from its proposal to introduce a unique, lifelong identifier for every EU citizen as part of the reform of the #eIDAS Regulation. I spoke to @euractiv@twitter.com about the issue: euractiv.com/section/digital/n #eID

Show thread

So instead of simply porting forward a pristine archival copy of the first website, I had to preserve technical and curatorial decisions that were made in 2014, about content that was collected in 1997, from the original website in 1991. Forget those white gloves, this is what archival work really looks like.

github.com/sul-dlss-labs/slac-

Show thread

#SysadminLife, microchip shortage 

Further to this, I would just like to add that contrary to what you might have heard, SNMP is amazing, a work of corporate cooperation against all the odds, and you can and should see what things your servers and appliances are making available over SNMP, if only so that you can disable unauthenticated SNMPv1 and enable authenticated and encrypted SNMPv3.

It's also comical to see how many companies (Hi again Canon) try to hide their SNMP description ("MIB") files behind logins, subscriptions, and threats of legal action for unlicensed use. Sharing paywalled MIBs is the bootleg market for sysadmins.

Show thread

«climate risk stress test published today show that banks do not yet sufficiently incorporate climate risk into their stress-testing frameworks and internal models, despite some progress made since 2020.»
bankingsupervision.europa.eu/p

philosophy of language 

I wrote something for 1000-Word Philosophy on Grice's theory of conversational implicature.

Their articles are intended as introductions for beginning students. I hope that I managed to say something useful and not misleading.

1000wordphilosophy.com/2022/07

#philosophy

💬 noyb commented on the EDPB draft Guidelines on administrative fines: While we do consider the document to be useful to Supervisory Authorities, fines should still be efficient and deterrent enough to keep controllers from violating the law:
noyb.eu/sites/default/files/20

📢Update 101 complaints: The Italian DPA has joined the consensus shared by the EDPS, as well as the French and Austrian DPA and has banned the use of Google Analytics
➡ Read our full statement here: noyb.eu/en/update-further-eu-d

We are supporting the winds of change in #Denmark🇩🇰 investing €475 million for wind energy, in a loan to @Vestas.
The project will support low-carbon technology, #RenewableEnergy and #EnergyEfficiency🌬️

👉bit.ly/3ykxbEA

🐦🔗: nitter.eu/EIB/status/154422665

Show older