In recent years every spring seems to turn into a period of _massive_ refactoring & restructuring in #ThingUmbrella — maybe it's a form of spring cleaning, even though the reasons[1] are not seasonal... Currently spending my nights reworking the https://thi.ng/vectors package (likely one of the most comprehensive vector packages available for TS/JS) and trying out different splits/structures, testing their impact on package sizes and usability in existing downstream packages. Currently over 3000 source files with uncommitted changes... aaaarrrgghh!

Most functions (vector operations) in this package exist in multiple versions (many code generated, but now in need to be updated): Generic n-dimensional, loop-free, optimized 2D/3D/4D versions and strided versions for manipulating vectors views of larger nD data buffers (supporting all kinds of data layouts, incl. AOS, SOA, hybrid...)

[1] https://mastodon.thi.ng/@toxi/114296442006935718

Thanks to @made I recently learned that dynamic code generation doesn't play nice with certain Content Security Policies (CSP). This has a major impact on a few core packages in #ThingUmbrella, like the https://thi.ng/vectors package which contains ~900 vector functions, most of them code generated and optimized for different vector sizes/dimensions (incl. n-dimensional versions). This package (and some others using a similar approach) are key dependencies for dozens of other geometry/visualization related packages... However, I found code generation the only way to practically manage & maintain the sheer amount of functionality provided.

Because of this (CSP impact), I've been working on a new code generator, which converts the dynamically generated code into statically generated source code files. This will make the overall initial package size bigger, but this shouldn't be a major problem in practice, since there're also very positive effects, including:

- The new format allows for doc strings for _all_ generated vector ops (with the dynamic approach there was no way to properly attach those in TypeScript)
- The new file structure (single function per source file) massively helps with dead code elimination when using a bundler, resulting in smaller final file/bundle sizes. When NOT using a bundler, similar filesize savings can be had by using direct imports (to individual functions) rather than full package imports
- None of the unused versions need to be code generated at runtime anymore, so also improving startup time

The new codegen is already covering around more than a third of the 900 ops. If you want to keep an eye on progress & discussion, follow this issue:

https://github.com/thi-ng/umbrella/issues/497

#ReleaseWednesday Just pushed a new version of https://thi.ng/block-fs, now with additional multi-command CLI tooling to convert & bundle a local file system tree into a single block-based binary blob (e.g. for bundling assets, or distributing a virtual filesystem as part of a web app, or for snapshot testing, or as bridge for WASM interop etc.)

Also new, the main API now includes a `.readAsObjectURL()` method to wrap files as URLs to binary blobs with associated MIME types, thereby making it trivial to use the virtual filesystem for sourcing stored images and other assets for direct use in the browser...

(Ps. For more context see other recent announcement: https://mastodon.thi.ng/@toxi/114264980961483146)

#ThingUmbrella #ReleaseTuesday... New package (initial alpha release):

https://thi.ng/block-fs provides highly customizable & extensible block-based storage with an optional hierarchical filesystem layer. This is useful everywhere you might need virtual filesystem, though the storage providers can also be used without the filesystem layer (e.g. for #Forth-style block data/editors).

The default configuration provides:

- arbitrarily nested directories
- filenames (UTF-8) of max. 31 bytes per directory level
- max. 32 owner IDs
- file locking
- creation/modification timestamps (64 bit)
- efficient append writes

Currently included storage providers: TypedArray-based in-memory and host filesystem based file storage (one block per file). More are planned (e.g. IndexedDB, remote endpoint)...

The readme is currently still lacking various diagrams to illustrate the filesystem internals. I will add those ASAP...

Just made my day: "I somehow made it to March 2025 before being aware of https://thi.ng/ - an incredible mountain of code created primarily by a prolific genius, full of ideas that are like catnip to me."

https://kylecordes.com/2025/typescript-html-generation-on-the-server

(Also, to clarify, even though thi.ng/hiccup and a small selection of other thi.ng libs started out porting concepts widely used in Clojure (the language I spent 7 years with previously), in many cases the scope, features, usability & potential use cases have been far extended far beyond their "originals" and it sometimes saddens me that these are often just plainly ignored or mis-labeled/described...)

To avoid a massive OpenCV dependency for a current project I'm involved in, I ended up porting my own homemade, naive optical flow code from 2008 and just released it as a new package. Originally this was written for a gestural UI system for Nokia retail stores (prior to the Microsoft takeover), the package readme contains another short video showing the flow field being utilized to rotate a 3D cube:

https://thi.ng/pixel-flow

I've also created a small new example project for testing with either webcam or videos:

https://demo.thi.ng/umbrella/optical-flow/

#ReleaseMonday — One of the recent (already very useful!) new package additions to #ThingUmbrella is:

https://thi.ng/leaky-bucket

Leaky buckets are commonly used in communication networks for rate limiting, traffic shaping and bandwidth control, but are equally useful in other domains requiring similar constraints.

A Leaky Bucket is a managed counter with an enforced maximum value (i.e. bucket capacity). The counter is incremented for each a new event to check if it can/should be processed. If the bucket capacity has already been reached, the bucket will report an overflow, which we can then handle accordingly (e.g. by dropping or queuing events). The bucket also has a configurable time interval at which the counter is decreasing (aka the "leaking" behavior) until it reaches zero again (i.e. until the bucket is empty). Altogether, this setup can be utilized to ensure both an average rate, whilst also supporting temporary bursting in a controlled fashion...

Related, I've also updated/simplified the rate limiter interceptor in https://thi.ng/server to utilize this new package...

Rejoice, #Zig v0.14.0 is out and the release notes show quite a lot of nice updates...

https://ziglang.org/download/0.14.0/release-notes.html

(Just hoping/bracing the build script for https://thi.ng/wasm-api modules isn't impacted too much, recent Zig updates caused quite a bit of rework there...)

PSA: DNS changes for thi.ng and various subdomains and aliases are currently propagating, for example:

https://asm.thi.ng/ - ARM/STM32 baremetal resources
https://demo.thi.ng/ - Hundreds of example projects for various thi.ng libraries
https://dependencies.thi.ng/ - Dependency graph/matrix
https://docs.thi.ng/ - API documentation
https://forth.thi.ng/ - Online Forth REPL[1] (also check out other shader examples from the https://thi.ng/charlie readme)
https://shadergraph.thi.ng/ - Node based shader graph[2]

So happy I completed this migration (now fully outside US) ! You should hopefully not see any downtime, but if so, please try again later...

[1] Example: https://mastodon.thi.ng/@toxi/109677667622245586
[2] Example: https://mastodon.thi.ng/@toxi/111838559833947868

Current status: Spending the day moving all my stuff off AWS and outside USA, setting up new buckets, servers, DNS... Other nice result: All 100% green energy powered...

#ReleaseTuesday Just released a new version of https://thi.ng/scenegraph which has some minor breaking changes and also includes a new feature to zoom/scale and translate a node with a given reference point, such that this reference point remains constant in global space. This is very useful for creating zoomable UIs where we want to zoom towards a cursor position (rather than the scene's origin). I got tired to remember the maths and recreate this functionality each time I need it, so now it's just a single `node.scaleWithReferencePoint()` call...

New example:
https://demo.thi.ng/umbrella/scenegraph-pan-zoom/

Source code:
https://github.com/thi-ng/umbrella/blob/develop/examples/scenegraph-pan-zoom/src/index.ts

Click & drag to pan, touchpad/mousewheel to zoom towards the cursor pos...

As for the breaking changes: Since the scene node constructors had quite a few (optional) arguments, I refactored them as options objects (aka named params), also improving readability. All other related/bundled examples have already been updated too, see a list in the package readme...

#ReleaseWednesday New (old actually, been dormant since 2021) #ThingUmbrella package addition: https://thi.ng/tsne - see [1] and [2] for further info/context...

A little bare bones example showing the solver in action to visualize 1000 random 4D data points:

https://demo.thi.ng/umbrella/tsne-colors/

Example source code:
https://github.com/thi-ng/umbrella/blob/develop/examples/tsne-colors/src/index.ts

cc/ @anandphilipc

[1] https://mastodon.thi.ng/@toxi/114030865988513140
[2] https://mastodon.thi.ng/@toxi/114030992180858620

Even with all the well known shortcomings and notoriously fiddly configuration of t-SNE visualizations, I'm still often reaching for it to just get quick overview of some datasets I'm exploring/working with. It's fascinating to observe the process and the structures unfolding, even with simple dummy data...

The attached example animations show different variations of projecting 500 random data points with different dimensionalities & configurations into a fixed 2D space:

1) [r, g, b] only
2) [r, g, b, radius]
3) [r, g, b, radius, outline]

Latest attempt at building better documentation for https://thi.ng/umbrella (also to make it available offline!): Having noticed that recent versions of TypeDoc support extracting & merging of doc strings from monorepos, over the past few weeks I've been updating/cleaning docstrings in hundreds of source files across all 200+ packages and started building a small tool to assemble a single/mega-page documentation (currently ~4.3MB of just HTML). The tool translates existing docstrings and references contained therein (and still used for the existing API docs) to support proper cross-package references.

I've uploaded an early preview here:
https://docs.thi.ng/umbrella/

Please be aware that so far this is only an early stage prototype and only contains very limited docs. I.e. there are no generics/typeparams, no details about classes/interfaces... But at least I know now HOW to add this all, as well as all the additional metadata I've already got (currently still only available via other custom tools/examples).

For example, there're links to the tag-based browser[1] and I'm also planning to add the fuzzy doc search engine/index[2] to this new documentation... The tag browser integration still needs more work in terms of correctly matching package names to tags. The underlying system is there already, just needs more work in terms of actually doing/assigning the concept mapping. Since most package names in thi.ng/umbrella are very plain/boring (for a reason), for many (most?) packages this already works pretty well:

Example: Visiting the WebGL package docs: https://docs.thi.ng/umbrella/#webgl and then clicking on "examples" for this package, then opens the tag browser for WebGL: https://demo.thi.ng/umbrella/thing-browser/#webgl where you can then see all other packages and examples related to this topic...

More updates on this all soon! Excited!

(EDIT: added screenshots...)

[1] https://demo.thi.ng/umbrella/thing-browser/
[2] https://demo.thi.ng/umbrella/rdom-search-docs/

Another workshop done and glad I got a chance to introduce colleagues to the I Ching hexagrams as an example for #BottomUpDesign and as system for constructing/composing higher level concepts/meanings from just a small set of fundamental archetypes (in the I Ching there're only 8 fundamental trigrams)... It's been hugely influential and a guiding principle for my own work & design philosophy (incl. for most of my #OpenSource projects) and I keep finding ever new uses for applying this approach, here to devise a composable classification system, eventually allowing us to define more complex concepts... In world defined by #TopDownThinking, going the opposite way and showing people the benefits/flexibility is sometimes/oftentimes _very_ hard, but to me unquestionably more powerful...

Ps. Also reminded again of the whole Seed technology and essence/function (aka "ti-yong") contrast/discussion/subplots in Stephenson's Diamond Age... Yong is the outer manifestation of something. Ti is the underlying essence...

https://en.wikipedia.org/wiki/Hexagram_(I_Ching)

PSA: I just updated https://thi.ng/monopub (a tool for synchronized publishing of packages in a monorepo) to add a new option to truncate generated changelogs with a configurable cut-off date (i.e. to exclude older versions). I've set the default cut off to 1st of January 3 years ago. As a result this saves around 1MB of just changelog files per full #ThingUmbrella release (for 200 packages). Older version history can of course still be obtained via Git...

Up since 5AM... already did a bunch of #ThingUmbrella stuff (incl. new release(s) and repo-wide maintenance), then morning walk in the still crispy air to marvel at the beautiful hoar frost (but forgetting to put the SD card in the camera, so phone pics only... )...