Kicks Condor

LEECHING AND LINKING IN THE HYPERTEXT KINGDOM

I FIRST DISCOVERED
THE 【TECHS-MECHS】WHO
ARE A CLAN OF SOUTH
OF THE BORDER GUNDAM
BREAKING DOWN
IMMIGRATION FENCES
WITH THEIR
IMPRESSIVE MANOS
MECANICAS

PLUNDER THE ARCHIVES

This page is also on dat.

MOVING ALONG LET'S SEE MY FAVORITE PLACES I NO LONGER LINK TO ANYTHING THATS VERY FAMOUS

philosopher.life, the 'wiki'/'avatar'/'life' of h0p3. serious rabbithole.

indieseek blog, bumped into brad somehow and we crosstalk a ton about the web.

linkport by joe jenett---blogs at i.webthings.

an eye on: ᛝ ᛝ ᛝ — lucid. whimsy.space, caesar naples.

indieweb: .xyz, eli, c.rwr, boffosocko.

nostalgia: geocities.institute, bad cmd.

true hackers: ccc.de, fffff.at, voja antonić, cnlohr, esoteric.codes.

chips: zeptobars, scargill, 41j.

dwm, julia, tridactyl these are things you'll want on linux.

neil c very famous but should be a world icon.

the world or cate le bon you pick.

sammyclassicsonicfan the original teen rage adventure.

innovation.isotropic.org probly the best carl chudyk game.

and opinionated gamers for non-chudyk game analysis.

my twitter. my github. minor things.

#dat

I use three main tags on this blog:

  • hypertext: linking, the Web, the future of it all.

  • garage: art and creation, tinkering, zines and books, kind of a junk drawer—sorry!

  • elementary: schooling for young kids.

02 Oct 2018

Taming Outlandish TiddlyWikis

A prototype for the time being.

I’m sorry to be very ‘projecty’ today—I will get back to linking and surfing straightway. But, first, I need to share a prototype that I’ve been working on.

Our friend h0p3[1] has now filled his personal, public TiddlyWiki to the brim—a whopping 21 MEGAbyte file full of, oh, words. Phrases. Dark-triadic memetic, for instance. And I’m not eager for him to abandon this wiki to another system—and I’m not sure he can.

So, I’ve fashioned a doorway.

This is not a permanent mirror yet. Please don’t link to it.

Screenshot of the h0p3 archive page.

Yes, there is also an archive page. I took these from his Github repo, which appears to go all the way back to the beginning.

Ok, yes, so it does have one other feature: it works with the browser cache. This means that if you load snapshot #623 and then load #624, it will not reload the entire wiki all over again—just the changes. This is because they are both based on the same snapshot (which is #618, to be precise.) So—if you are reading over the course of a month, you should only load the snapshot once.

Snapshots are taken once the changes go beyond 2 MB—though this can be tuned, of course.

  • Total size of the raw archive: 6.2 gigs.
  • Size of my kicksnap’d archive: 736 megs.

Shrunk to 11% of its original size. This is done through the use of judicious diffs (or deltas). The code is in my TiddlyWiki-loader repository.

A Few Lessons I Picked Up

I picked up this project last week and kind of got sucked into it. I tried a number of approaches—both in snapshotting the thing and in loading the HTML.

I ended up with an IFRAME in the end. It was just so much faster to push a 21 MB string through IFRAME’s srcdoc property than to use stuff like innerHTML or parseHTML or all the other strategies.

Also: document.write (and document.open and document.close) seems immensely slow and unreliable. Perhaps I was doing it wrong? (You can look through the commit log on Github to find my old work.)

On the Snapshot Technique

I originally thought I’d settled on splitting the wiki up into ~200 pieces that would be updated with changes each time the wiki gets synchronized. I got a fair bit into the algorithm here (and, again, this can be seen in the commit log—the kicksplit.py script.)

But two-hundred chunks of 21 MB is still 10k per chunk. And usually a single day of edits would result in twenty chunks being updated. This meant a single snapshot would be two megs. In a few days, we’re up to eight megs.

Once I went back to diffs and saw that a single day usually only comprised 20-50k of changes (and that this stayed consistent over the entire life of h0p3’s wiki,) I was convinced. The use of diffs also made it very simple to add an archives page.

In addition, this will help with TiddlyWikis that are shared on the Dat network[2]. Right now, if you have a Dat with a TiddlyWiki in it, it will grow in size just like the 6 gig folder I talked about in the last box. If you use this script, you can be down to a reasonable size. (I also believe I can get this to work directly from TiddlyWiki from inside of Beaker.)

And, so, yeah, here is a dat link you can enjoy: dat://38c211…a3/

I think that’s all that I’ll discuss here, for further technical details (and how to actually use it), see the README. I just want to offer help to my friends out there that are doing this kind of work and encourage anyone else who might be worried that hosting a public TiddlyWiki might drain too much bandwidth.


  1. philosopher.life, dontchakno? I’m not going to type it in for ya. ↩︎

  2. The network used by the Beaker Browser, which is one of my tultywits. ↩︎

This post accepts webmentions. Do you have the URL to your post?

You may also leave an anonymous comment. All comments are moderated.

08 Sep 2018

Web Recorder

An incredibly sick tool for archiving—maybe this is already popular and beloved, but it doesn’t hurt to wave it around a bit here.

A modern WWW archiver service—just was overhauled and the bleeding-edge can save the archive to Dat. (Makes me want a ‘record’ ⏺ button in my URL bar that I can just leave on! Any ideas if this exists??)

(INCIDENTALLY discovered this on the Code for Society Agenda notes on Etherpad, which I hadn’t seen in many years—it’s fantastic that it’s still around! This is a tool surely in the vein of what our little internet surf club here has been discussing recently. (Video here, haven’t watched this, so this is also a TODO.)

This post accepts webmentions. Do you have the URL to your post?

You may also leave an anonymous comment. All comments are moderated.

28 Aug 2018

Sharing and Archiving with Dat

This is a technical overview of how to use/understand Dat. It covers how useful it is for ‘backing up’ websites—which is how I intend to use it.

So, this article (and the comments) cleared up a few things for me.

Dat can currently be configured to either track all changes (history) of files in a folder (at the cost of a full duplication of all files and all historical changes), or track only the most recent version of files with no duplication (at the cost of losing all history). There is not (yet?) any fancy dat mode which efficiently tracks only deltas (changes) to files with no other file overhead.

From my examination of the Beaker code yesterday, I noticed that the browser only downloads the specific version of a file that you need—I like this! (Rather than having to download the whole history of a file to put it back together.)

One advantage that Dat has over IPFS is that it doesn’t duplicate the data. When IPFS imports new data, it duplicates the files into ~/.ipfs. For collections of small files like the kernel, this is not a huge problem, but for larger files like videos or music, it’s a significant limitation. IPFS eventually implemented a solution to this problem in the form of the experimental filestore feature, but it’s not enabled by default. Even with that feature enabled, though, changes to data sets are not automatically tracked. In comparison, Dat operation on dynamic data feels much lighter. The downside is that each set needs its own dat share process.

I think this is a great benefit of Dat’s design. Because it basically just boils down to a distributed append-only log—a giant, progressively longer file that many people can share, and which you can build stuff like file folders or a database on top of—it’s incredibly flexible.

It certainly has advantages over IPFS in terms of usability and resource usage, but the lack of packages on most platforms is a big limit to adoption for most people. This means it will be difficult to share content with my friends and family with Dat anytime soon, which would probably be my primary use case for the project.

I totally disagree with this sentiment! Dat has the Beaker Browser—which is an incredible thing for a novice to use. Yes, it would (will?) be even better when it can be found on iOS and Android. But, for now, I’m happy to recommend it to friends and family: “Yeah, you can share your own websites—we can even have our own private Twitter-type-thing together—with this thing.”

I know the Beaker team has said that their goal is to get Dat accepted by the major browsers—but I think Beaker’s ability to customize itself to the decentralized web is an advantage. I could see it finding a lot of users.

This post accepts webmentions. Do you have the URL to your post?

You may also leave an anonymous comment. All comments are moderated.

Browser-Side Includes in Beaker Browser

A proof-of-concept for enjoying HTML includes.

It seems like the Beaker Browser has been making an attempt to provide tools so that you can create JavaScript apps that function literally without a server. Their Twitter-equivalent (‘fritter’) runs entirely in the browser—it simply aggregates a bunch of static dats that are out there. And when you post, Beaker is able to write to your personal dat. Which is then aggregated by all the others out there.

One of the key features of Beaker that allows this is the ‘fallback_page’ setting. This setting basically allows for simplified URL rewriting—by redirecting all 404s to an HTML page in your dat. In a way, this resembles mod_rewrite-type functionality in the browser!

What I’ve been wondering is: would it be possible to bring server-side includes to Beaker? So, yeah: browser-side includes. My patch to beaker-core is here. It’s very simple—but it works!

Quick Example

Beaker editing index.html

Here is Beaker editing the index.html of a new basic Website from its template. I’m including the line:

<!--#include file="inc.html"-->

This will instruct beaker to inline the inc.html contents from the same dat archive. Its contents look like this:

<p style="color:red">TEST</p>

Beaker displaying index.html

And here we see the HTML displayed in the browser.

But Does Beaker Need This?

I’m not sure. As I’ve been working with static HTML in dat, I’ve thought that it would be ‘nice’. But is ‘nice’ good enough?

Here are a few positives that I see:

Appeal to novices. Giving more power to HTML writers can lower the bar to building interesting things with Dat. Beaker has already shown that they are willing to flesh out JavaScript libraries to give hooks to all of us users out here. But there are many people who know HTML and not JavaScript. I think features for building the documents could be really useful.

Space savings. I think static blogs would appreciate the tools to break up HTML so that there could be fewer archive changes when layouts change subtly.

Showcase what Beaker is. Moving server-side includes into Beaker could demonstrate the lack of a need for an HTTP server in a concrete way. And perhaps there are other Apache/Nginx settings that could be safely brought to Beaker.

The negative is that Dat might need its own wget that understands a feature like this. At any rate, I would be interested if others find any merit to something like this. I realize the syntax is pretty old school—but it’s already very standard and familiar, which seems beneficial.

This post accepts webmentions. Do you have the URL to your post?

You may also leave an anonymous comment. All comments are moderated.