Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
mamot.fr is one of the many independent Mastodon servers you can use to participate in the fediverse.
Mamot.fr est un serveur Mastodon francophone, géré par La Quadrature du Net.

Administered by:

Server stats:

3.2K
active users

mamot.fr: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.4

#Ktistec

2 posts2 participants0 posts today
Public
Todd Sundsted

Release v2.4.6 of Ktistec is out. As mentioned in an earlier post, this release focuses on database performance improvements. This means caching the results of expensive queries (like counting all posts with a particular hashtag or mention). On my instance at least, pages like the notifications page are now snappier.

There are still slow queries (queries that take more than 50msec). Most of those are requests for pages of old posts where none of the necessary database pages are in the page cache. I have increased the page cache size, and that reduces the frequency, but I don't see an immediate fix.

Fixed

  • Add missing database query logging.

Changed

  • Improve query performance for hashtags and mentions.
  • Make less costly updates to tag statistics.
  • Improve anonymous session management.
  • Cache the Nodeinfo count of local posts.

Removed

  • Remove support for X-Auth-Token.

Other

  • Add timeout values for POST socket operations.

I don't have an immediate plan for the next release. There have been a bunch of feature requests that I think have merit. I'll probably get started on some of those.

#ktistec#fediverse#activitypub
Public
Todd Sundsted

You can specify the SQLite database and pass options (pragmas like cache_size, journal_mode, ...) on the command line when you start the Ktistec server. The following example sets the cache size to 20,000 pages (up from the default of 2,000 pages) which improves performance on larger instances.

KTISTEC_DB=~/ktistec.db\?cache_size=-20000 ./server

You can also enable the write-ahead log (but make sure you know what that means).

KTISTEC_DB=~/ktistec.db\?journal_mode=wal\&synchronous=normal ./server

Pragmas supported are limited to those listed here.

GitHubGitHub - toddsundsted/ktistec: Single user ActivityPub (https://www.w3.org/TR/activitypub/) server.Single user ActivityPub (https://www.w3.org/TR/activitypub/) server. - toddsundsted/ktistec
#ktistec#fediverse#activitypub
Public
Todd Sundsted

A new release of ktistec that improves database performance is imminent. In the past, database optimization usually meant "fixing a bunch of poorly constructed queries", and I'm sure there's more of that to do—I'm not an expert. But this time, I found most of the queries were as good as they were going to get on my watch (I'm not an expert). If you have a million records and you need to filter and count them, that's just going to take some time...

So this time, I focused on caching the results of queries like that (which really means I focused on cache invalidation, right). A case in point is commit d544b1af. Previously, the nodeinfo endpoint filtered and counted posts on every request, and it took +80msec to do that. Worse, the filtering pushed everything else out of the sqlite page cache, which made the next, unrelated database query slow!

Caching this value, and only recounting when I post something, not only dropped the service time for the request to ~1msec but actually improved database performance, generally!

More to come...

#ktistec#fediverse#activitypub
Public
Todd Sundsted

If you're running an instance of Ktistec and want to see what other ActivityPub instances are sending you, turn on JSON-LD processing debug logging.

  1. Go the the /system URL.
  2. Find the ktistec.json_ld setting.
  3. Select "Debug" and save.

Ktistec will dump received activities to the log, after the activity has been parsed into JSON but before JSON-LD expansion.

2025-01-22 14:53:17 UTC 409 POST /actors/toddsundsted/inbox 4.29ms
2025-01-22T14:53:17.597172Z  DEBUG - ktistec.json_ld: {"@context" => ["https://www.w3.org/ns/activitystreams", "https://w3id.org/security/v1"],
"id" => "https://random.site/users/FooBar#delete", "type" => "Delete", "actor" => "https://random.site/users/FooBar", "object" => "https://random.site/users/FooBar", "to" => ["https://www.w3.org/ns/activitystreams#Public"], 
"signature" => {"type" => "RsaSignature2017", "creator" => "https://random.site/users/FooBar#main-key", "created" => "2025-01-22T14:52:40Z", "signatureValue" => "01234567890abcdefghijklmnopqrstuvwxyz=="}}

Answer to a FAQ:
The server returns HTTP status code 409 ("Conflict") if it has already received an activity.

#ktistec#fediverse#activitypub
Public
Todd Sundsted

Crystal is fast because methods are monomorphized at compile time. In simple terms, that means that at compile time, a polymorphic method is replaced by one or more type-specific instantiations of that method. The following polymorphic code...

def plus(x, y)
  x + y
end

...is effectively replaced by two methods—one that does integer addition if called with two integers, and one that does string concatenation if called with two strings.

This extends to inherited methods, which are implicitly also passed self. You can see this in action if you dump and inspect the symbols in a compiled program:

class FooBar
  def self.foo
    puts "#{self}.foo"
  end

  def bar
    puts "#{self}.bar"
  end
end

FooBar.foo
FooBar.new.bar

class Quux < FooBar
end

Quux.foo
Quux.new.bar

Dumping the symbols, you see multiple instantiations of the methods foo and bar:

...
_*FooBar#bar:Nil
_*FooBar::foo:Nil
_*FooBar@Object::to_s<String::Builder>:Nil
_*FooBar@Reference#to_s<String::Builder>:Nil
_*FooBar@Reference::new:FooBar
_*Quux@FooBar#bar:Nil
_*Quux@FooBar::foo:Nil
_*Quux@Object::to_s<String::Builder>:Nil
_*Quux@Reference#to_s<String::Builder>:Nil
_*Quux@Reference::new:Quux
...

The optimizer in release builds is pretty good at cleaning up the obvious duplication. But during my optimization work on Ktistec, I found that a lot of duplicate code shows up anyway.

Most pernicious are weighty methods that don't depend on class or instance state (don't make explicit or implicit reference to self). As I blogged about earlier, this commit replaced calls to the inherited method map on subclasses with calls to the method map defined on the base class and reduced the executable size by ~5.8%. The code was identical and the optimizer could remove the unused duplicates.

So, as a general rule, if you intend to use inheritance, put utility code that doesn't reference the state or the methods on the class or instance in an adjacent utility class—as I eventually did with this commit.

(The full thread starts here.)

The Crystal Programming LanguageCrystalCrystal is a general-purpose, object-oriented programming language. With syntax inspired by Ruby, it’s a compiled language with static type-checking. Types are resolved by an advanced type inference algorithm.
#ktistec#crystallang#optimization
Public
Todd Sundsted

Ktistec release v2.4.5 rolls out the build time and executable size optimizations I've been blogging about here. It also fixes a few small bugs.

Fixed

  • Handle @-mentions with hosts in new posts.
  • Handle HEAD requests for pages with pretty URLs.
  • Destroy session after running scripts.

Changed

  • Delete old authenticated sessions.

I've started a branch full of query optimizations. My general rule—as highlighted in the server logs—is if a query takes longer than 50msec, it takes too long. It's time to address some problems...

GitHubRelease v2.4.5 · toddsundsted/ktistecFixed Handle @-mentions with hosts in new posts. Handle HEAD requests for pages with pretty URLs. Destroy session after running scripts. Changed Delete old authenticated sessions. Other Reduct...
#ktistec#fediverse#activitypub
Public
Todd Sundsted

The Ktistec executable is now ~24.7% smaller and build times are 28% faster.

I've been blogging about optimizations here, here, and here. This is the summary of the final outcome, with links to commits for the curious. I have one more post planned with a summary of my thoughts.

Here's my approach. Use nm to dump the symbols in a release build executable and then look for things that seem redundant. The first change and associated post below is a great example of what I mean—my original implementation led to the specialization of the #== method for every pairwise combination of model classes even though the result of the comparison was just false.

This might seem like a strange approach if you come from a compiled language where you mostly write all of the code yourself or invoke generics explicitly, but Crystal takes your code and does that for you. And it's not always obvious up front (to me, at least) what the final cost will be.

I've include counts of the lines added/removed because the point of this whole post is to say if you measure first and then optimize, a small change can have a big impact.

Here are the changes:

  • Specialize model #==. (+7 -5)
    I talked about this here but didn't have the commit to link to. This change results in a large reduction in executable size on regular builds (~4.0%) and a small difference on release builds (~0.2%).
  • Remove conversion to Hash. (+2 -2)
    This commit eliminates specialization of methods like __for_internal_use_only that get passed both named tuples and hashes by going all in with named tuples. It also eliminates instantiations of the Hash generic type itself for these cases. Reduces executable size by ~2.2%.
  • Eliminate duplicate code in the executable. (+3 -3)
    This small change reduces the size of the executable by a further ~0.4% by eliminating redundant definitions of __for_internal_use_only entirely.
  • Make InstanceMethods instance methods. (+1 -5)
    This was a goofy design I picked up somewhere. It's unnecessary. Changing this saves ~0.2% on release build executable size.
  • Move the code for digging through JSON-LD. (+246 -281)
    It looks like a lot of lines of code changed here, but the large numbers are the result of moving code line-by-line from an included module to a utility class. Invoking these as methods on the utility class rather than as instance methods on each including class reduces the executable size by ~0.5%.
  • Use map from base ActivityPub model classes. (+10 -2)
    map is a class method defined on each ActivityPub base model class. Each definition maps JSON-LD to a hash that is used to instantiate the class. Class methods defined on a base class are available on subclasses, as well. Calling the method on the subclass results in a copy of the method. This change reduces the executable size by ~5.8%.
  • Move map into helper. (+104 -88)
    The map method does not depend on class/instance state. This change ensures that the mapping code is not duplicated even if a subclass's map method is accidentally again called. It looks like a lot of changes but this commit is mostly reorganization. It reduces executable size by ~0.4%.
  • Replace classes with aliases. (+62 -148)
    Implementing ActivityPub's vocabulary with discrete model classes is expensive because every model class comes with machinery for type-specific CRUD operations. Enumerate aliases on each base model class (e.g. a "Service" is an "Actor"). This change reduces executable size by ~16.9%.

I'm off to optimize some queries now...

#ktistec#crystallang
Public
Todd Sundsted

Ktistec release v2.4.4 fixes a few things in the prior release and introduces at least one killer feature!

Fixed

  • Always get the attachments. (fixes #119)
  • Don't run scripts until the server has been configured.

Changed

  • ⭐️ Make the editor toolbar sticky.
  • Clear the cached translator when the settings change.

I'm spending some cycles looking at the size of the server executable. You can read about my approach to reducing Crystal Language executable size and build time here.

GitHubRelease v2.4.4 · toddsundsted/ktistecFixed Always get the attachments. (fixes #119) Don't run scripts until the server has been configured. Changed Make the editor toolbar sticky. Clear the cached translator when the settings change.
#ktistec#fediverse#activitypub
Public
Todd Sundsted

The prologue to this post is here.

Investigating commit e2327eea might be a bust.

I dumped the symbols before and after this change. The new symbols were all specializations of the core library Hash class introduced by adding JSON parsing support for the "language" property.

So what does that mean and why is this commit a dead end?

You can think of Crystal classes and methods as being implicitly generics. If you have a method foo with one parameter bar and call it with an Array, Crystal creates a version of that method specialized to handle an Array type as an argument. If you call it with a Hash, Crystal creates another version of that method specialized to handle a Hash type as an argument. If the method has 20 lines of code, you effectively get two copies of those 20 lines of code. There is no runtime polymorphic dispatch, which is one of the reasons Crystal is so fast. You can make all of this explicit with Crystal type restrictions, method overloading, and generics, of course, but you don't have to.

This path is a dead end (for now) because any improvements that I can see that I can make (replacing hash construction with a more fluent sequence of attribute assignments) will need to be made to other classes where this is a problem, and there are only a few of those, so the net potential for improvement seems small.

epiktistes.comEpiktistes
#ktistec#crystallang
Public
Todd Sundsted

After I release a new version of ktistec, I build the server commit-by-commit to see which commits increase the server executable size and build time the most. I do this because I’ve learned that small implementation details (inlined code, small methods, using blocks) can have large impacts on these numbers.

Here's the output:

Commit         Size          Time
======== ========== ======= ===== =======
248850b1   36426264          10.3
47268073   36425688  -0.00%  10.5  +1.60%
344de272   36425688  +0.00%  10.8  +3.24%
ef561f52   36425944  +0.00%  10.8  -0.08%
8ae2cbd4   36429128  +0.01%  10.8  -0.01%
3e425f3b   36429128  +0.00%  10.8  +0.22%
1487d903   36427704  -0.00%  11.0  +1.42%
935c9ceb   36427016  -0.00%  11.0  +0.14%
de37dc6a   36427016  +0.00%  10.9  -0.97%
a660a326   36427016  +0.00%  10.8  -1.12%
ff3d990e   36427016  +0.00%  10.8  +0.54%
5724a58d   36523192  +0.26%  11.0  +1.78%
7b5057d4   36523640  +0.00%  11.0  -0.44%
30ca6a3f   36541352  +0.05%  11.6  +5.73%
e2327eea   36671592  +0.36%  11.0  -5.36%
ad0d76eb   36671592  +0.00%  10.9  -0.48%
d388e74f   36671592  +0.00%  11.4  +4.59%
dacea7ad   36671592  +0.00%  11.0  -3.76%
03d5dfd8   36671592  +0.00%  10.8  -1.63%
79d9d89f   36671576  -0.00%  11.0  +1.82%
b65d292f   36792376  +0.33%  11.1  +0.95%
0ef53365   36808904  +0.04%  11.6  +4.88%
b3766e7b   36808904  +0.00%  11.1  -4.50%
56ba79ce   36825416  +0.04%  11.1  -0.50%
4824df58   36825736  +0.00%  11.1  +0.31%
c4705143   36837544  +0.03%  11.1  -0.03%
e3d37ef7   36837768  +0.00%  11.5  +3.52%
4509fa0d   36837768  +0.00%  11.0  -3.83%
0ff9237b   36837768  +0.00%  11.0  -0.55%

Overall, the server executable size increased by about 1.1% and the build time increased by about 6.8%. Maybe that's not too bad for a major feature, but let's dig in.

It's nice to see that three commits account for almost all of the increase in server executable size:

  • 5724a58d Add `language` to `Object`.
    2 files +19  loc
  • e2327eea Render `contentMap` on ActivityPub objects.
    2 files +17 -1 loc
  • b65d292f Add translation actions to the objects controller.
    1 file +35 loc

But, compare 5724a58d to 8ae2cbd4 (Add `language` to `Account`). It added +22 loc but didn't increase the server executable size as much.

In any case, I'll look at e2327eea first. I'd like to understand why this relatively small change adds 130,240 bytes to the server executable size!

GitHubGitHub - toddsundsted/ktistec: Single user ActivityPub (https://www.w3.org/TR/activitypub/) server.Single user ActivityPub (https://www.w3.org/TR/activitypub/) server. - toddsundsted/ktistec
#ktistec#crystallang
Public
Todd Sundsted

Ktistec release v2.4.3 supports language translation.

animation demonstrating the translation of text from Japanese to English

Inspiration for this feature comes from Mastodon.

In order to enable translation, you need an API key for either DeepL or LibreTranslate. These are the only services Ktistec supports at this time.

Posts from properly configured accounts on supported servers, like Mastodon, include the content language. On posts like these, Ktistec will display a button to translate the content if the language differs from your language.

Unfortunately, not all Fediverse/ActivityPub servers explicitly support language (I mean, Ktistec didn't until just now). And not all users correctly set their posts' language, so ymmv... but it has been hugely useful for me.

I'm going to focus on site customization next (colors, etc.).

GIF
#ktistec#fediverse#activitypub
Public
Todd Sundsted

Release v2.4.2 fixes a few more bugs. Only one is a regression—I found the others while testing. In this release:

Fixed

  • Fix metrics chart line labels.
  • Permit del, ins, and s elements in sanitized HTML.
  • Only store the redirect_after_auth_path on browser navigation.
  • Add "content" property to editor JSON error response.
  • Use FileUtils.mv to move uploaded files. (fixes #117)

Thanks to @jayvii for help with troubleshooting the last one!

GitHubRelease v2.4.2 · toddsundsted/ktistecFixed Fix metrics chart line labels. Permit del, ins, and s elements in sanitized HTML. Only store the redirect_after_auth_path on browser navigation. Add "content" property to editor JSON error r...
#ktistec#fediverse#activitypub
Public
Todd Sundsted

Eh, I didn't test enough and released v2.4.0 of Ktistec with a few annoying regressions. Release v2.4.1 fixes them! ☹️ 😠 😡 🤬

Fixed

  • Handle edge cases and race conditions in script runner.
  • Reconnect editor autocomplete support.
  • Fix permissions on uploaded files.
GitHubRelease v2.4.1 · toddsundsted/ktistecFixed Handle edge cases and race conditions in script runner. Reconnect editor autocomplete support. Fix permissions on uploaded files.
#ktistec#fediverse#activitypub
Public
Todd Sundsted

I just released v2.4.0 of Ktistec. This release encompasses a few things that I've been working on for a while: improved support for operating without JavaScript available/enabled and support for running scripted automations.

Except for a few items, Ktistec now works without JavaScript. Obviously, things like WYSIWYG editing of HTML don't work—I plan to add support for Markdown to compensate. Running in Lynx is a stretch, but...

lynx on osx displaying the authenticated home page

Since the early days, most controller actions supported both text/html and application/json.  I cleaned up support for the latter and have officially documented the Ktistec API in the README.

In addition, I've added support for running bots/automations (prior announcement). The Ktistec server will periodically run any executable script in the etc/scripts directory. These scripts have access to the Ktistec API and can post, follow, share, like, etc. This is experimental and obviously introduces an attack surface, though that shouldn't be a problem on correctly configured hosts.

Here's the full changelog:

Added

  • Support running scripted automations.
  • "Fetch Once" button on hashtag and thread pages. (fixes #108)
  • Support navigation to a post's threaded view. (fixes #108)
  • Add support for post name and summary.

Fixed

  • Improve support for operating without JavaScript available/enabled.
  • Only enforce CSRF protection on "simple" requests.

Changed

  • Replace use of multi-action controller with formaction. (fixes #101)

Other

  • API usability improvements.


#ktistec #fediverse #activitypub #crystallang

Public
Todd Sundsted

I've been thinking about the demise of botsin.space. Running a site for bots is hard (and expensive) but writing and running an ActivityPub-based bot should be easy.

To prove this was the case I added experimental support for bots/automations to Ktistec in the form of scripts that the server periodically runs. These scripts can be in a programming language of your choice. The server provides credentials for its API in the process environment (if you can use curl you can publish posts), simple interaction happens via stdin/stdout/stderr, and the complexity of using ActivityPub is abstracted away.

The code is only available on the following branch for the moment:

    https://github.com/toddsundsted/ktistec/commits/run-scripts/

There are a couple example shell scripts here:

    https://github.com/toddsundsted/ktistec/commit/4982925a...

I have a few enhancements in mind, but it's already proven useful as a means to periodically log data from my server host, and I'll use it, when finished, to publish release notes.

muffinlabs.comRIP botsin.space
#ktistec#activitypub#fediverse
Public
Todd Sundsted

TIL scripting media queries... gone are the days of the no-js class on the body tag...

i'm working on improving the ktistec user experience when javascript is disabled.

ktistec uses trix as its rich text editor.  of course. trix doesn't work without javascript. behind the scenes, however, trix uses a hidden textarea to hold the body of the post being edited. using scripting media queries, ktistec can now show (or hide) either the editor or the textarea based on the availability of javascript (2030b26b). 

MDN Web Docsscripting - CSS: Cascading Style Sheets | MDNThe scripting CSS media feature can be used to test whether scripting (such as JavaScript) is available.
#ktistec#activitypub#fediverse
Public
Todd Sundsted

i'm working toward providing a decent experience in ktistec with javascript turned off.  the first step was emitting minified css instead of using the webpack style loader plugin to apply css via javascript (fd7a5369).

GitHubGitHub - toddsundsted/ktistec: Single user ActivityPub (https://www.w3.org/TR/activitypub/) server.Single user ActivityPub (https://www.w3.org/TR/activitypub/) server. - toddsundsted/ktistec
#ktistec#activitypub#fediverse
ExploreLive feeds
About