Omnivore Alternatives

Nov 4, 2024

As I’m not currently self-hosting there is every chance the hosted service may go away or my data get lost

It is with great prescience that I sadly relay news that Omnivore is going away:

We’re excited to share that Omnivore is joining forces with [some AI thing] […] All Omnivore users will be able to export their information from the service through November 15 2024, after which all information will be deleted.

So I’ve gone back to comparing self-hostable alternatives to avoid this happening again. My main criteria are:

Here are the options I’ve found:

Name Highlights/annotations Mobile app Notes
Linkwarden ❌ ❌ Demo instance. General UX is actually pretty nice. Seems to be primarily a bookmark management service. Does save archived snapshots of pages in various formats, but these are a little fiddly to navigate to - the default interaction is to just open the live link
wallabag ✅ ✅ I was using wallabag for a while before moving to Omnivore. The feature set is perfect, but both the web interface and mobile app feel a little clunky and the Android context menu to annotate/highlight doesn’t behave consistently
Readeck ✅ ❌ Despite not having a native mobile app, there’s a lot of great stuff here. The UX is clean and simple (quite similar to Omnivore’s), it’ll grab video transcripts where available, it can export to epub and there’s a browser extension (which works on Firefox Android) to send the contents of weird JavaScript festivals that it can’t scrape directly from the HTML alone. As a bonus, the dev knocked together an importer from Omnivore right after they announced they were shutting down

Wallabag ticks all the boxes, but Readeck feels so much nicer to work with…it’s just not going to be with me on the tube. Between the browser plugin and some HTTP Shortcuts workflow, sharing to the service is covered. It’s really just the offline syncing I’m missing. I’ll be sticking with Readeck for a while, and ultimately if the lack of apps becomes a deal breaker, I’m sure I’ll be able to move everything to wallabag easily enough.

I’d really love someone to fork the open source Omnivore mobile apps and make them talk Readeck instead. Maybe if I can find some time…

2024-10-06 Sunday Links

Oct 6, 2024

2024-09-29 Sunday Links

Sep 29, 2024

lists have been really foundational for the transmission of information. Writing in New York Mag, Kyle Chakra explores how Wikipedia, in some ways a logical extension of Otlet’s original idea for a a Universal Bibliography, has even gone as far as creating meta “lists of lists,” which help organize and rearrange the lists of information. There’s even a lists of lists of lists.

Creativity requires a mélange of experiences that cannot be quantifiable. They include but are not limited to: quiet contemplation, boredom, inspiration, taste, style, genre, motif, and dozens of others. These experiences do not come on-demand. They require a specific state of being that can only be acquired away from the measured and regimented processes that society loves too much.

we are working with the developers of the third-party client “ClassicUO” with the goal of making available an officially sanctioned version for use on all shards

Jekyll Short URL Generator

Sep 25, 2024

I was reading this post by J about setting up a simple URL shortener with no server-side logic and it got me wondering if there was a Jekyll/GitHub Pages “native” way to implement something similar. It turned out to be very simple to put together!

redirect.html layout

Basically my regular layout with a couple of changes:

<script>
	const url = new URL(window.location.href);
	const stay = url.searchParams.get("stay") != null;
	if (!stay) {
		window.location = ""
	}
</script>

<noscript>
	<meta http-equiv="refresh" content="0; url=" />
</noscript>

Some simple JavaScript to redirect to the target page, unless a stay query parameter exists. If JS isn’t available it’ll use the meta tag to redirect.

_config.yml

Add a new scope

  - scope:
      path: ""
      type: "redirects"
    values:
      layout: "redirect"

…so that redirect entries use the new layout

Add a new collection type

collections:
  redirects:
    output: true
    permalink: /s/:name/index.html

…which will cause redirect pages to be generated at http://ste.vet/[redirectname]/index.html. This allows them to be accessible at http://ste.vet/[redirectname] with no special serverside logic or web server config to handle these paths.

Add the redirect entries

Now I’m able to add a [redirectname.md] to my a _redirects folder, and all they need to contain is:

---
redirect: [url]
---

So you can now hit http://ste.vet/s/mastodon to get to my Mastodon page, or http://ste.vet/s/mastodon?stay to hold on the redirect page! I didn’t have any particular usecase for this and it was more idle noodling, but J’s application of theirs for QR codes to stick on luggage pointing to a Lost & Found page is very smart!

2024-09-22 Sunday Links

Sep 22, 2024

System 11 is based on a prototype of the PlayStation

2024-09-15 Sunday Links

Sep 15, 2024

Feediverse

Mar 15, 2024

Ben Werdmuller ponders:

What if we had a great experience that ties together both short-form discussion and re-sharing and long-form reading, in a way that better showcases both kinds of content and realizes that the way we consume both is different? What if it had a beautiful, commercial-level design? And what if it remained tied to the open social web at its core, and pushed the capabilities of the protocols forward as it released new features and discovered new user needs?

I've had an idea rattling around in my head for a while. The idea of a Fediversey RSS reader and read later platform. Each RSS feed would be an "account" you can follow/share, and then a nice client to save off articles for later and add highlights/annotations to share. If I had Ben's year and funding, I'd do that!

Copy and paste from Obsidian with formatting

Oct 21, 2023

tl;dr: Copy as HTML plugin

Copy/pasting from Obsidian has some issues. The first problem I hit was that pasting into Slack was coming through as unformatted text - the raw Markdown. After a bit of fiddling, switching from Edit to Reading mode resolved this. Pasting from Reading mode properly formats the text.

Pasting into Google Docs created some more interesting problems though. While Reading mode carried the formatting through, it also brought elements of my Obsidian theme, like background colour which, naturally, wasn’t what I was looking for. Then I found the Copy as HTML plugin, which handles its own Markdown to HTML conversion and copies the output from that, avoiding bringing along any unintended elements from the editor. This leads to a nice workflow where you can draft and edit documents in Obsidian, keeping them close to other notes to allow for future remixing/cross-referencing etc, while making it easy to transfer it to Google docs to share or collaborate on.

Omnivore, the free, open source read later app

Oct 12, 2023

An article in the reader view of Omnivore web

I’ve been using Wallabag as a self-hosted read later service for a while. Like a free, open source Pocket/Instapaper. It works well enough, but the general UX is a little rough around the edges and development doesn’t seem super speedy - the Android app hasn’t seen a release in a couple of years. I recently stumbled upon Omnivore and it’s already won me over.

The good

The less good

I’m finding it very enjoyable to use and I think I’m a convert! As I’m not currently self-hosting there is every chance the hosted service may go away or my data get lost, but I’ll keep exporting my highlights and spin up my own instance once they’ve ironed some of the kinks out!

Arc Browser And Tim Berners-Lee's Dream of Intercreativity

Jul 29, 2023

As I was reading Tim Berners-Lee’s book on the origins of the web it became clear that TBL has, or had, a real passion for democratising the creation of the web and for people to be able to create content directly in the browser, even editing the content they’re viewing.

We ought to be able not only to find any kind of document on the Web, but also to create any kind of document, easily. We should be able not only to follow links, but to create them - between all sorts of media. We should be able not only to interact with other people, but to create with other people. If interactivity is not just sitting there passively in front of a display screen, then intercreativity is not just sitting there in front of something ‘interactive’.

Throughout the book TBL seems frustrated that his idea that a web browser should also be a web editor never took off:

A long-standing goal of mine had been to find an intuitive browser that also, like my WorldWideWeb, allows editing. A few such browsers/editors had been made, such as AOLpress, but none were currently supported as commercial products. Few items on the wish list for collaboration tools had been achieved. At the consortium we wondered what was wrong. Did people not want these tools? Were developers unable to visalise them? Why had years of preaching and spec writing and encouragement got hardly anywhere? p183

The W3C even developed the Amaya browser/editor, which is able to create and edit content in place - the Github repo was archived in 2018.

I saw that the Arc browser came out of invite-only testing on Tuesday and looked around at the feature set.

I doubt any of this is exactly what TBL had in mind in the 90s, but this idea of being in control of the web and generating and sharing content directly in the browser definitely feels like it fits into his philosophy.

Hostname Aliases on a LAN In Linux

Jul 27, 2023

I have a little computer running Linux in my home that I use a home server, running various services such as FreshRSS and linkding. I wanted to setup a bunch of memorable hostnames I could use to point to different services on that machine. This was my journey.

My primary OS is Windows. I’m used to using the machine name (hostname) of the computer to resolve to its local IP address by some sort of magic with no human intervention (apparently this is NetBIOS? I dunno, I’ve literally never had to think about it), but that wasn’t working for this Linux server PC. After trawling through a bunch of StackExchange posts suggesting the user edit their hosts file…on each and every device on their LAN, which is obviously the wrong way around, I found posts such as this helpfully indicating that the functionality is offered in zero-conf(iguration) services, such as mDNS and that installing Avahi on the Linux machine should broadcast the hostname as hostname.local and then be available by name rather than just IP. After a quick

sudo apt install avahi-daemon

this all worked pretty promptly!

I then got to thinking “wouldn’t it be neat if different web services, running on different ports, could be referred to directly by a name?”. After some more frantic Googling I found this amazing post from Andrew Dupont walking through basically my exact usecase. The Python mdns-publisher seemed perfect for what I needed, but unfortunately it bound to the wrong network interface/IP address and as far as I can see there’s no parameter to change that. I ended up using something inspired by Andrew’s clearly less-perfect solution but it totally did the job, basically boiling down to a bunch of calls to

avahi-publish -a freshrss.local -R 192.168.1.100 &
avahi-publish -a linkding.local -R 192.168.1.100

(etc).

All hostnames were now mapping to the server’s IP address correctly, but the port still needed to be specified - this was effectively basic aliases for an IP. The next step in Andrew’s article is to setup nginx proxies. Having seen nginx config syntax before, I didn’t really fancy that so looked around for more accessible solutions. I eventually stumbled on Nginx Proxy Manager. I threw together a docker-composse.yml based on their example and the UI was super-legible and looked easy to configure. After making some entries to map the Avahi mDNS names to http://hostname.local:[port] everything was working!

The Nginx Proxy Manager web UI showing some services mapped from server.local to http://server.local:port

Weaving The Web, by Tim Berners-Lee

Jul 13, 2023

An image of "Weaving The Web, by Tim Berners-Lee"

Weaving The Web - The Past, Present and Future of the World Wide Web by its Inventor by Tim Berners-Lee, 1990

A couple of years ago I went on a quest to find the perfect note-taking software for me. I dabbled with Roam, but was looking for something less siloed. I tried TiddlyWiki, but found the tech stack quite clunky, as straight out the box it’s just an HTML file that has no way to write your changes anywhere. Eventually I stumbled on Obsidian, which was exactly what I needed: local notes, Wiki-style interconnected-ness with simple and (relatively) standardised Markdown for formatting. It got me thinking about the structure of the web and how fundamental links and connections should be. I was inspired to read Tim Berners-Lee’s book on the origins of the web. I took a bunch of notes, but these are the ones that resonated most strongly.

I’ll refer to Tim Berners-Lee as TBL to save us all some characters.

From the foreword by Michael L. Dertouzos, then-director of MIT Laboratory for Computer Science.

When I first met Tim, I was surprised by another unique trait of his. As technologists and entrepreneurs were launching or merging companies to exploit the Web, they seemed fixated on one question: ‘How can I make the Web mine?’ Meanwhile, Tim was asking, ‘How can I make the Web yours?’

Origins

Early iterations

Enquire

Tangle

R&D and beyond

The WorldWideWeb in production

The Future

Some observations

I was reading the book out of an interest in the origins of the web and the tech that drives it. I didn’t take notes around the formation and operation of the W3C, or some of the loftier crystal ball-gazing as it wasn’t relevant to my specific context. This was a nice quote though (emphasis mine)

Bias on the Web can be insidious and far-reaching. It can break the independence that exists among our suppliers of hardware, software, opinion and information, corrupting our society.

Gemini Why And How

Jan 21, 2022

I thought I’d start off by explaining why and how I setup a Gemini server.

Why

We were chatting at work about the issues with the modern web - speed, cruft and general usability. Someone dropped a link to the Gemini project page.

Gemini project page

It resonated with me, particularly because it doesn’t expect to be, and isn’t trying to be, a replacement to the web, but an alternative with a focus on signal-to-noise ratio. The markup for formatting text, gemtext, is Markdown-like and minimal, both in terms of characters/bytes used and range of functionality. There are 3 levels of (sub-)headings, links, blockquotes and pre-formatted text. And that’s it. It’s all making room for that textual content. I thought it’d be a fun project to setup my own Gemini server, ideally for as close to free as possible.

How

After a little research I settled on gmnisrv as the server program- I found a bunch of people using it and it doesn’t have a ton of dependencies.

gmnisrv

I’ve wanted to learn Docker for a while, so I spent enough time researching it to cobble together something that would build gmnisrv and host a placeholder page locally

My (probably bad) Dockerfile

Then I had to figure out where to host it. There are many, many free-or-almost-free options for hosting something running over http/https but a lot fewer with the flexibility to serve stuff on not-port-80-or-443. However, I found that Google Cloud Compute’s free tier offers a low-end VM that should be more than capable to serve any throughput I’ll get on this page. Specifically they offer:

GCP compute free tier details

I spun up an instance, setup SSH keys, SCPd over my Dockerfile, setup port forwarding and got gmnisrv running. I opened Gemini client Lagrange and pointed it at the VM’s IP address and was served my page!

I then wrote a (very) small Python script to run over all the gemtext files in a folder (e.g. blog posts) and build an index page. All the metadata is encoded into the filename- this is 2022-01-21-why-and-how.gmi, which will go on to generate the date and title/heading for the post. As the markup is so similar to Markdown, I thought I’d also generate a Markdown representation to mirror the posts over in my Github pages page, which is, let’s be honest, where you’re probably reading this.

I’m hoping to use this as a place to form a habit of writing posts to synthesise and clarify my thoughts on topics I’m reading and thinking about. Let’s see!

Debugging Unity Android Apps

Jul 23, 2018

This is pretty much a repost/summary of a StackOverflow post, as it took me a long while to find this information; hopefully Google will do its magic and surface this for future generations.

I had a need to debug a Unity game on an Android device. The build running on the device was a Development build with debugging enabled etc etc. For reasons I had neither time nor inclination to investigate, Visual Studio was not discovering the device and it was not showing in the Attach Unity Debugger list, despite the device being on the same Wifi network, and physically attached by USB cable. I knew the IP address of the device, but not the port the debugger process was listening on, which apparently changes with each launch.

To summarise the StackOverflow post, you want to get hold of the ADB logcat output for your game. I did (something like):

adb logcat -c
adb logcat > out.txt
(launch game on device)
(wait some period of time)
(CTRL + C)

…which will dump the logcat output to a file called out.txt . If you now search for monoOptions in the file you should see a line like:

Using monoOptions --debugger-agent=transport=dt_socket,embedding=1,defer=y,address=0.0.0.0:56785

If you add the device’s IP address with the port from above to the Visual Studio Attach Unity Debugger window, it should now connect, obey breakpoints and the like!

How and Why Is This Object Being Destroyed, Unity?!

Jul 17, 2017

I recently ran up against a problem in a Unity project I’m working on: a GameObject was being Destroyed, but I didn’t know why or from where. The codebase, naturally, has many calls to Destroy() and contains its own methods with that name, which made both Find References and text-based searches impractical. I just wanted a breakpoint in UnityEngine.Object.Destroy().

Round 1: OnDestroy()

Spoilers: you may well know this method is a dead-end.

Unity will automatically call OnDestroy() on all components on a destroyed GameObject. I thought this might allow me to set a breakpoint, but OnDestroy() is deferred to the end of the frame, so the callstack doesn’t go back to the original Destroy() call. Next!

Round 2: Hacking Unity’s IL

A discussion with a friend led to the idea of modifying the .NET IL in Unity’s DLLs to modify the contents of UnityEngine.Object.Destroy(). I Googled upon Simple Assembly Explorer, which allows you to view and modify the IL of compiled .NET binaries. Without any prior knowledge of .NET IL I was quickly able to insert a ldarg.0, to push this onto the stack as the argument for the next function call, and a call instruction, to call out to UnityEngine.Debug.LogWarning, which would give me a stack trace. I booted up my project in Unity and sure enough, every call to Destroy produced the log I hacked in there. Amazing! While this worked, it felt very fragile: any future update to Unity would stamp over this, and I didn’t fancy learning IL to build this out further.

Round 3: Harmony

I was aware of the concept of hooking/detours from lower level, C++/assembly code, and was interested whether something similar existed for .NET/C#/Mono/Unity. Google led me to Harmony on GitHub:

A library for patching, replacing and decorating .NET and Mono methods during runtime.

Not only does it support Unity, it’s built with Unity in mind- the example code is a mod for a Unity game. I was able to copy and paste the example code into a fresh project referencing the Harmony DLL, change the target class to UnityEngine.Object, the target method to Destroy and the target parameter to a UnityEngine.Object, then change the hooked method implementation to a log call. After a build, all I needed to do was drop my new DLL, along with the provided Harmony DLL, into the Unity project call the setup function to initialise the hooks and boom: the log was again produced on every Destroy. This has a bunch of benefits over the previous method, such as writing the patch in the language I was already using, not having to screw with binary file (so the patch’s code could happily live in source control), and being forwards compatible (unless Unity make any breaking API changes). In theory any C# Unity could be hooked in this fashion, which could be great for mocking functions or gaining a bit more control over what’s going on under the hood!

Sanitarium HD Patcher On GitHub

Dec 9, 2015

Just a quick update: I’ve open sourced my Sanitarium HD Patcher. The code and latest release can now be found on GitHub! Enjoy!

Sanitarium HD Patcher

Feb 7, 2015

It is now available! Please find v0.1.0 here. This version will only work with executables with MD5 of 0A09F1956FCA274172B0BB90AA5F08A7. If people turn out to have other versions I’ll try to get hold of the exes and get them supported. Enjoy!

UPDATE: Permanent page at sanitariumhd.

New Year's Resolution Patch

Feb 1, 2015

I guess that title would have worked better a month ago.

Anyhow. I’ve always been fascinated by software reverse engineering and general binary hackery, but had never really thought of a project to try it out on. Then I remembered the Fallout High Resolution Patch and the Infinity Engine Widescreen Mod, which apply cracking/patching techniques to allow old games designed at 1990s resolutions to run at glorious 1080p. I decided to do something similar.

Targeting

I wanted a target for which no fan patch already existed. I was browsing GOG, and saw that the game Sanitarium had recently been added. I remembered that I already had a copy installed on my PC - perfect! Target acquired.

Hacking Commences

The first step was to figure out what resolution the game runs at out of the box. To do this, I took a screenshot of the game running windowed (command line param -w, if you’re interested), and highlighted the rectangle excluding the standard windows border stuff. I’m sure there are more scientific ways to tell the size of a window, but this worked for me.

Original window

I deduced the game to be rendering at a 90s-classic 640x480. I opened up game in the trusty debugger, OllyDBG, and began investigating the heck out of it.

I searched for the number constant 480, set a breakpoint on each reference and ran the game. One of these was hit very early on in the initialization, and was proceded by a reference to 640 - strong candidate! (ignore the fact that the offsets are from patched - I’d already backed up the original executable)

Reference to 640x480

I used Olly to patch the 640/480 values to 1280/720 respectively, and ran the game. The window was now 720p, with the main menu occupying the upper-left corner, but once in game it was rendering a much larger visible area. See below for comparison

Original playfield

720p playfield

If you’re familiar with the game you’ll notice that all the game objects outside of the 640x480 camera the game is expecting aren’t drawn. I’ll address this later, but at this point I got ambitious(/distracted). The changes made in Olly can be saved out as a modified .exe, which can be used in the future. This would technically let me distribute the patched executable, allowing the wider internet to play the game at high-res. However, there are a couple of drawbacks:

Rolling a debugger

A quick bit of googling showed me that in order to modify executable code on the fly in Windows you basically have to write a debugger. This sounded very intimidating. I continued my research and it turned out to be conceptually very simple. All that’s required is a C++ project to do the following:

The target’s executable code is memory mapped to an offset from a base address. For 32 bit Windows programs, this address is 0x00400000. I referred to the patches I made in Olly to get the address which needed to be modified. As can be seen in the screenshot of the debugger, we started with a PUSH 1E0, followed by a PUSH 280 (480 and 640 in hexadecimal). The compiled x86 machine code for PUSH [some 4 byte value] is 68 [some 4 byte value in little-endian] - 68 E001000 in our exaple. In this case, and most cases we’ll need to deal with, we can leave the PUSH (68) part untouched, and only change the operand (E001000). The program I wrote takes the desired resolution (x and y) as command line arguments and parses them as an unsigned 16 bit integer. We can then take a pointer to one of these values, cast it to a pointer to a byte, and treat it as a little-endian 2-byte array, like so:

uint16_t resY = parseInt(resYString);
uint8_t* resYBytes = (uint8_t*)&resY;

The PUSH 1E0 happens at 0x0041A5FF. We can leave the first byte as 68 for PUSH, and just modify the 2 bytes at 0x0041A600/0x0041A601, to the 2 bytes of resYBytes. To do this we can use WriteProcessMemory, passing the offset we found with Olly as the lpBaseAddress param, the 2 byte array representing the dimension (e.g. resYBytes) as lpBuffer, and then the size to write as 2. That’s basically all there is to it. Once the patch for setting resolution width and height are applied, my program closes and lets the game carry on as normal.

Culling me softly

As I mentioned earlier, even with the resolution patches applied there are still some objects inside the newly-embiggened viewport which are not being drawn. Jumping back into Olly, I continued searching for 640/480. This lead me to the area of code below:

Some rect math

To ease both rendering and logic load, games often skip (or cull) objects which aren’t visible. I could see some calls to functions operating on Rects (IntersectRect/OffsetRect), and figured this could be the logic for culling offscreen objects, still using the hardcoded 640x480. Applying a couple more patches to bring these up to 720p I was presented with this:

Less culling

Note the extra dudes in the bottom right. Amazing! I then jumped over to my project and made the code a bit more generic, using a std::map<uint32_t, const uint8_t*> to store arrays of bytes to be patched in, indexed by their memory address. And that’s where I’m at. There is still one pretty glaring issue:

Smudging with camera pan

Previously the camera was restricted so it would never draw beyond the edge of the level. Now we’re drawing a bigger area around the player, empty space is visible. It looks like the surface the game draws to isn’t cleared every frame, leaving the remnants of the previous frame hanging around. I’ll need to figure out a way to clear it before the background is drawn to it, then we should be all set!

I also still need to add some validation of command line arguments, and I’ll make a follow up post with it (and hopefully the full source code) attached once it’s ready.