Casey Primozic's Notes

WebGL `blitFramebuffer` Performance Issues on M3 Mac

Fri, 24 Apr 2026 14:44:30 -0500

background

I’ve recently been working on some procedural background systems for a browser-based game I’ve been working on.

The idea was to create a sort of Shadertoy-style 100% per-fragment-procedural background. The background is what displays when there’s no other actual scene object/mesh at a given pixel on the screen. So, in order to work efficiently, this pass reads from the scene depth buffer and skips all of its relatively expensive procedural logic in the case that the background won’t be visible anyway.

investigation

When I was testing out the procedural background system in a live scene with other effects and postprocessing, I was noticing that there were some pretty significant performance dips. It wasn’t unplayable or anything, but my FPS dropped from a stable 120 to ~80-90 and felt somewhat jittery/hitchy as well.

My first thought was that the procedural shader stuff in the background was too heavy. I started trimming different pieces and optimizing the shader code as much as I could, but nothing had any effect.

At some point, I added an early return at the beginning of all the procedural stuff, essentially making the shader just return pure black. Even after doing that, the FPS didn’t move at all!

This made me realize that the perf regression wasn’t related to the fancy background shader at all but instead was something about the plumbing and the way the procedural background fit into the rendering pipeline and interacted with other passes.

framebuffer blit performance issues

After some trial and error, I discovered that the framebuffer blits I was doing to copy scene depth in and copy the procedural background output out were responsible for the vast majority of the perf regression.

This was surprising to me. I was previously under the impression that framebuffer blits were relative cheap since they were specialized and heavily-optimized hardware-level routines. This finding challenged that assumption.

To be fair, I was doing a pretty high amount of blits, definitely more than necessary:

1 to copy the scene depth in (I’ve dealt with a lot of issues with recursive depth texture bindings in the past, so copying it to a fresh buffer was a conservative safety measure)
2 to copy the output of the procedural background shader back to main scene color output buffers (my rendering pipeline uses two color buffers to differentiate normal diffuse color and special bloomed emissive colors)
1 to sync the depth out to the emissive render target

One interesting thing about these blits is that they didn’t seem to actually cause any measurable load that I could track in the devtools performance tab, even in the GPU category. It’s definitely possible that the GPU profiling just wasn’t able to accurately track that.

However, another possibility is that the blits were especially expensive due to the hardware of the computer I was testing on (M3 Mac). The M3 GPU uses tile-based deferred rendering (TBDR) which has some different performance characteristics then other platforms like discrete GPUs on desktops.

It’s possible that the framebuffer blits were acting as synchronization points for the pipeline, reducing hardware utilization or causing frame pacing issues. I’m not an expert at all on low-level hardware stuff like this, so take that assessment with a grain of salt.

avoiding the blits

The main trick I used to remove the need for the two color blits was to do some hacky manipulation of the Three.JS render targets to just bind the existing output buffers directly as the color attachments of the background pass’s MRT rather than copying into them after the fact.

For the depth-out blit, I used a slightly different technique: rather than copying depth into the emissive render target each frame, I aliased the scene depth texture directly as the emissive RT’s depth attachment so they share the same underlying texture.

Together these got rid of three of the four blits; the one that remains is the initial copy of scene depth into a standalone buffer, which needs to stay in order to prevent the recursive depth texture binding problem on the main effect composer.

This all required doing some raw WebGL operations and bypassing Three.JS for a few things. It needed to be done carefully to avoid putting the WebGL pipeline or Three.JS into a bad state.

I also made some additional small optimizations like only calling framebufferTexture2D to re-bind textures to the framebuffer in the case that the textures actually changed.

As a result of all these changes, I was able to almost entirely mitigate the performance dip caused by this procedural background pass and get back to a stable 120FPS even with some decently expensive shader code in the background itself.

conclusion + takeaways

WebGL framebuffer blits are apparently not as cheap as I thought they were - especially on certain architectures like Mac and mobile.
There are opportunities to get extra performance by doing things manually in a way that top-level Three.JS APIs can’t
Old and often-repeated advice, but you have to actually profile rather than making assumptions about what’s causing something to be slow

Fixing Rust LLD Linker Undefined Symbol Error

Thu, 23 Apr 2026 19:04:37 -0500

the problem

After upgrading my Rust toolchain to the latest nightly, I started getting errors when building a WebAssembly project of mine:

error: linking with `rust-lld` failed: exit status: 1
  |
  = note:  "rust-lld" "-flavor" "wasm" "--export" "create_terrain_gen_ctx" "--export" "free" "--export" "gen_heightmap" "--export" "malloc" "--export" "set_params" "--export=__heap_base" "--export=__data_end" "-z" "stack-size=1048576" "--stack-first" "--no-demangle" "--no-entry" "<2 object files omitted>" "<sysroot>/lib/rustlib/wasm32-unknown-unknown/lib/libpanic_abort-*.rlib" "/home/casey/dream/src/viz/wasm/target/wasm32-unknown-unknown/release/deps/{libcommon-a24e0de9c9a36cad,librand_pcg-96aa495088ea06d3,librand-dd7ab4b12369f67a,librand_core-5f1e6ad6780daf68,libopensimplex_noise_rs-29c02d98fe844459,libnanoserde-5b273dd919df410e}.rlib" "<sysroot>/lib/rustlib/wasm32-unknown-unknown/lib/{libstd-*,libdlmalloc-*,libcfg_if-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-L" "<sysroot>/lib/rustlib/wasm32-unknown-unknown/lib/self-contained" "-o" "/home/casey/dream/src/viz/wasm/target/wasm32-unknown-unknown/release/deps/terrain.wasm" "--gc-sections" "--no-entry" "-O3"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: rust-lld: error: /home/casey/dream/src/viz/wasm/target/wasm32-unknown-unknown/release/deps/terrain.terrain.1508d28c2e9bb756-cgu.0.rcgu.o: undefined symbol: log_error
          rust-lld: error: /home/casey/dream/src/viz/wasm/target/wasm32-unknown-unknown/release/deps/terrain.terrain.1508d28c2e9bb756-cgu.0.rcgu.o: undefined symbol: log_error

The place in my code causing these issues was this:

mod imports {
  #[link(wasm_import_module = "env")]
  extern "C" {
    #[allow(dead_code)]
    pub fn log_msg(msg: *const u8, len: usize);
    pub fn log_error(msg: *const u8, len: usize);
  }
}

It’s the same pattern I’ve used to import functions from JS into my Rust WebAssembly modules since I first started working with Rust + Wasm several years ago. I’d compile + instantiate my Wasm module with those imports at runtime like this:

WebAssembly.compile(bytes as Uint8Array<ArrayBuffer>).then(module =>
  WebAssembly.instantiate(module, {
    env: {
      log_error: (ptr: number, len: number) =>
        void getEngine().then(engine => {
          const memory = new Uint8Array((engine.exports.memory as WebAssembly.Memory).buffer);
          const buf = memory.slice(ptr, ptr + len);
          const str = new TextDecoder().decode(buf);
          console.error(str);
        }),
      log_msg: (ptr: number, len: number) =>
        void getEngine().then(engine => {
          const memory = new Uint8Array((engine.exports.memory as WebAssembly.Memory).buffer);
          const buf = memory.slice(ptr, ptr + len);
          const str = new TextDecoder().decode(buf);
          console.log(str);
        }),
    },
  })
)

the cause

I hadn’t changed anything in that code in a long time nor had I updated my lld linker, so the issue was almost certainly the updated rust toolchain.

Apparently, at some point, Rust changed its behavior wrt. the default import and you now have to manually specify the Wasm import module to pull those imports instead of having it default to env like before.

the fix

I just added one line to add a magic directive to tell it to pull these imports from env:

#[cfg(target_arch = "wasm32")]
mod imports {
  #[link(wasm_import_module = "env")]
  extern "C" {
    #[allow(dead_code)]
    pub fn log_msg(msg: *const u8, len: usize);
    pub fn log_error(msg: *const u8, len: usize);
  }
}

After that, the build and functionality worked like normal.

Fixing Spotify OAuth 500 "Oops! Something went wrong" Error

Tue, 07 Apr 2026 21:21:08 -0800

The Problem

When trying to complete the OAuth flow for my web app spotifytrack.net, I started getting sent to an error page with a 500 status code and the following unhelpful message:

Oops! Something went wrong, please try again or check out our help area.

The app had been running successfully for years with no changes to the OAuth configuration. The error was easily reproducible just by visiting the /authorize URL directly. It persisted after trying incognito windows, different browsers, different devices, logging out and back in, etc.

Looking at the my service’s logs, it seemed to only be affecting my account; other users were still able to complete the OAuth flow and generate tokens successfully. Maybe other accounts were running into it as well, but I couldn’t easily tell.

The authorize URL I was using looked like this:

https://accounts.spotify.com/authorize?client_id=...&response_type=code&redirect_uri=...&scope=user-top-read&state=

The Fix

The issue was the empty state parameter at the end of the URL.

Spotify’s OAuth /authorize endpoint apparently can’t handle an empty state and returns a 500 error. Setting any non-empty placeholder value for state makes the flow work as expected.

This is definitely a bug on Spotify’s side, but that’s the workaround if you run into it.

A Unique Performance Optimization for a 3D Geometry Language

Sat, 10 Jan 2026 19:14:35 -0600

For the past several months, I’ve been working on a programming language called Geoscript. It’s specialized for generating and manipulating 3D geometry for use in a Shadertoy-inspired web app called Geotoy.

Geoscript uses a pretty simple tree-walking interpreter for its execution. That being said, I’ve put in a decent amount of effort optimizing it for runtime speed as well as building up an optimization pipeline for the language itself.

While working on this, I discovered a pretty neat optimization opportunity that's uniquely suited to this use case.

But first, let me give a little background on Geoscript’s optimization pipeline to give some history and context.

(or just skip to the “Cross-Run Expr Cache Persistence” section below if you want)

Geoscript Optimization Pipeline

The first optimization I added for geoscript was basic constant folding. Nothing special or unique so far; just the usual approach of finding things like { op: Add, lhs: Literal(1), rhs: Literal(1) } in the AST and replacing it with Literal(2). I got the initial version working pretty quickly and started extending it to support more operations and data types. As I continued, I quickly came to realize something:

Almost everything in a typical Geoscript program is constant and doesn't depend on any external or dynamic input

Unlike programs from other languages, Geoscript doesn’t accept user input, perform DB calls, or interact with any external system at all. There are some small exceptions to this like PRNG calls and side effects to render meshes or print debug messages. But since the RNG is seeded, a Geoscript program is really just a pure function with zero arguments; it produces the exact same output every time it’s run.

Because of this fact, once I made the constant folding smart enough, it turned out that most or even all of the program would end up getting ran during the “optimization” process. This even applies to closures, as long as the variables they capture from the outer scope are themselves constant (it took some pretty complicated plumbing to get that part working correctly). The AST would often get replaced with just render(precomputed_mesh_literal).

Despite this, there weren’t really any significant user-visible changes to the way the language/interpreter worked, and there weren’t very dramatic speed-ups to program execution. The optimizations did enable some speedups for cases like this though:

spheres = 0..1000
  -> || {
    # random `vec3` where each element is in the range [-1000, 1000]
    pos = randv(-1000, 1000)
    s = icosphere(radius=10, resolution=4)
    s | translate(pos)
  }
  | join

In that case, the s = icosphere(...) statement doesn’t depend on any input in the closure or anything other than its literal arguments. So, the constant folding will replace it with a literal mesh in the AST. This saves the sphere mesh from having to get re-created each iteration of the loop and also saves on memory since the underlying mesh representation is reference-counted and translate just updates the transform matrix without mutating the mesh’s geometry itself.

Anyway, once I got the constant folding mostly wrapped up, I moved on to trying to add some other optimizations to the pipeline. The optimization I chose to pursue next was Common Subexpression Elimination, which is commonly referred to as CSE. This involves identifying identical expressions in the AST and replacing them with a single instance in a variable. This saves the value from having to get computed multiple times and can also unlock further optimization opportunities.

The first thing you need to do when implementing CSE is to set up a way to determine if two expressions (AST sub-trees) are identical. The approach I used for that is tree-based structural hashing, which I’m pretty sure is the standard solution. It was a little fiddly and tedious to set up, but I eventually got it working so any arbitrary AST could be deterministically hashed to a unique u128.

The next step I planned for implementing was to walk the AST, hash each node, identify duplicates, and replace them with references to a new temporary variable I’d create. This would require consideration for scopes and context, but if implemented correctly it would yield effective intra-expression CSE.

I also wanted to see if I could implement inter-expression, full-program CSE. For example, I wanted to be able to handle cases like this:

s = icosphere(radius=10, resolution=4)

some_fn = || {
  // ...
  another_sphere = icosphere(radius=10, resolution=4)
  // ...
}

Ideally, the CSE pass would identify that those two icosphere expressions were both identical and replace them both with a reference to the same pre-computed mesh. This would only work for fully-const cases. So if the radius of another_sphere depended on an argument of some_fn, for example, this optimization wouldn’t apply.

In order to implement this, I developed a dynamic programming approach that would memoize the evaluation of all fully constant expressions in the AST as the program is executed.

So each time an expression is evaluated, it’s hashed and the resulting value is inserted into the memoization cache. The next time an identical expression is hit, we get a cache hit and instead of running the interpreter we just clone the value out of the cache and use that.

Cross-Run Expr Cache Persistence

It was at this point that I suddenly had a very exciting idea:

There's nothing stopping me from persisting this constant expression cache across multiple runs of the interpreter!

Geotoy, Shadertoy are essentially live-coding environments. As a developer working in them, you usually iterate by making one small tweak or addition, re-running the code to see the updated output, and then evaluate based on that what to change next. The vast majority of the program usually stays exactly the same. By making the constant expression cache persistent, we can just re-use the results from the previous run for all the unchanged portions of the program.

In some cases this isn’t going to provide a lot of benefit. If you change a variable on line 1 which is depended on by every other line after it, there will be no cache hits and this strategy won’t provide any value. However, in very many real cases, a very high percentage of the program’s runtime can be cut out.

Here’s an example from a real Geotoy composition I was working on:

distance_to_circle = |p: vec3, radius: num| {
  sqrt(p.y*p.y + pow(sqrt(p.x*p.x + p.z*p.z) - radius, 2))
}

radius = 8

0..
  -> || randv(-radius*1.1, radius*1.1)
  | filter(|p| distance_to_circle(p, radius) < 2)
  | take(1550)
  | alpha_wrap(alpha=1/100, offset=1/100)
  | smooth(iterations=2)
  | simplify(tolerance=0.01)
  | render

alpha_wrap is a function from the CGAL library which works kind of like a sophisticated “concave hull” algorithm. It generates a mesh that encloses a set of arbitrary 3D points. It’s used for repairing meshes with bad or incomplete topologies and for things like processing LiDAR point cloud data. I’ve found that it’s also extremely useful for generating really cool-looking organic shapes from procedurally generated point clouds:

The downside of this function’s power and versatility is that it’s very expensive to run. On my computer, that alpha_wrap call takes around 900ms to run. The more points you pass and the higher you turn up the detail level, the bigger this gets as well.

Now, consider a case where you’re just tuning the tolerance param of the simplify function, trying to find a good balance between mesh quality and vertex count. Previously, this would require re-running the entire program each time - including the expensive alpha_wrap call - even though its output remains exactly the same. With the persistent expression cache, that whole part of the AST will just get pulled out directly and only the changed simplify() call gets run.

Unsurprisingly, this translates to huge reductions in execution time and dramatic improvements to the developer experience working in Geotoy:

PRNG Handling

One thing you may have noticed in the source code for that composition is that there’s a conspicuous randv call right at the beginning of the pipeline. Since this call both mutates the RNG’s state as well as relies on its hidden starting state, you might think that this would end up making the whole thing impure and non-const. And you’d be right.

This is something I had to handle explicitly in my expression cache in order to make caching it possible. A large number of Geotoy compositions make use of RNG calls, and just turning this optimization off for all those cases seemed like a big waste to me.

To support caching these kinds of expressions, I include the RNG start state as part of the cache key for expressions that read or write to Geotoy's built-in PRNG.

This essentially treats those expressions as pure functions of (rng_state) -> (T, new_rng_state) instead of the usual () -> T for most const expressions. Since RNG is re-seeded each run and the PRNG is fully deterministic, this means that as long as everything runs in the same order, the PRNG will transition to the same states after each call and the expression cache will be hit successfully on subsequent runs.

Conclusion

This persistent expression memoization optimization has ended up becoming the most impactful optimization I’ve added to Geoscript by far. It’s pretty funny since I came up with the idea for it completely out of the blue while working on CSE, which I haven’t even bothered finishing up yet.

That being said, this is obviously in no way a completely novel idea I’ve invented here. In fact, it very closely models the kinds of caching techniques used to speed up build systems like Nix and Bazel. Instead of caching object files or software packages, it’s caching the intermediate values produced while evaluating a program’s AST. This approach is just uniquely suited to Geoscript/Geotoy’s use case, where the same program is run repeatedly with small changes and no external or dynamic inputs.

Updating three-good-godrays for Three.JS 0.182

Thu, 01 Jan 2026 03:18:23 -0600

I maintain the three-good-godrays library, which adds support for volumetric raymarched screen-space godrays to Three.JS.

The most recent version of Three.JS (0.182.0) made some breaking changes to its depth packing and shadow internals that the library relied on, and it needed to be updated in order to work with it. I figured I’d create a little summary of what changes were needed for anyone else updating similar code for newer Three.JS versions.

Most of my fixes are implemented in this commit: https://github.com/Ameobea/three-good-godrays/commit/2295609acb7cadbf9415edd51bcebbab766012d5

Most of the breaking changes in Three.JS can be traced back to this PR: https://github.com/mrdoob/three.js/pull/32303

This PR is described as “modernizes the WebGLRenderer shadow mapping with several significant improvements” and it makes several distinct changes that touch many different parts of Three.JS’s internals.

Cubemaps for `PointLight`s

The first relevant change is “Native Cube Depth Texture Support for PointLight Shadows”. Previously, Three.JS used a texture atlas kind of approach that stored the shadow data for the point light in a normal 2D texture. This then had to be manually transformed to get the relevant shadow data out, which my code handled.

Version 0.182 changes it to using an actual cubemap texture. I updated three-good-godrays to check for this case and use the built-in cubemap read functions to deal with it in that case:

// this `USE_CUBE_SHADOWMAP` is set on the JS side if a cubemap texture is found
#if defined(USE_CUBE_SHADOWMAP)
  vec3 lightToPos = worldPos - lightPos;
  float lightDist = length(lightToPos);
  float shadowMapDepth = textureCube(shadowMap, lightToPos).r;
  float depth = lightCameraNear + (lightCameraFar - lightCameraNear) * shadowMapDepth;
  return vec2(float(lightDist > depth + 0.005), lightDist);
#else

Simpler Depth Packing

Previously, Three.JS’s BasicDepthPacking used a custom function that packed the depth value into the full RGBA range to help improve precision. This new version switches to a simpler scheme that just uses a single value. I’m not sure what the reason for this is, but anyway my code had to change.

I checked if the Three.JS release was >= 182 and if so use the simpler depth packing which is just 1.0 - depth:

// `USE_UNPACKED_DEPTH` is set if Three.JS version >= 182
#if defined( USE_UNPACKED_DEPTH )
  #if defined(IS_DIRECTIONAL_LIGHT)
    float depth = packedDepth.x;
  #else
  // packing uses: gl_FragColor = vec4( vec3( 1.0 - fragCoordZ ), opacity );
  float depth = 1. - packedDepth.x;
  #endif
#else
  float depth = unpackRGBAToDepth(packedDepth);
#endif

`OrthographicCamera` Now Default in `postprocessing` for `Pass`

The final breaking change that I needed to deal with wasn’t in Three.JS itself but rather the postprocessing library.

It was introduced in this commit: https://github.com/pmndrs/postprocessing/commit/443043c816ba7e1c3001a615e6a36648d234a1ef

This seems to impact postprocessing versions >= 6.38.

That commit supposedly fixes an exception that happens when using Three.JS’s new reversed depth mode - which is supposedly an alternative to logarithmic depth buffer that offers better precision + performance. Anyway, it changes the default camera passed to the Pass constructor from a simple THREE.Camera to a THREE.OrthographicCamera.

It was tricky to track this down, but the fix was as simple as just explicitly passing a new THREE.Camera() instead of relying on the new default:

-    super('GodraysCompositorPass');
+    super('GodraysCompositorPass', undefined, new THREE.Camera());

Conclusion

It was rather annoying making these fixes and getting everything working, but at least it worked and three-good-godrays now fully works with the latest versions of both Three.JS and postprocessing.

Usually these updates aren’t that bad luckily, and most of the time there’s nothing that needs to be updated at all.

Generating 3D Meshes From Text

Thu, 27 Nov 2025 23:41:35 -0600

I recently had a desire to convert text to 3D meshes that I could render and manipulate as part of my Geotoy project and Geoscript language. I did some research into tools and libraries that could solve different pieces of this, and I put together a pipeline that implements the whole thing - yielding nice, 2-manifold 3D meshes with arbitrary fonts, text styles, and more.

This post gives an overview of the whole setup and aims to give anyone else looking to implement something similar everything they need to get it working themselves.

`svg-text-to-path`

The first part of the setup uses a JavaScript library called svg-text-to-path. It handles taking arbitrary input text and font params and generating a SVG which contains paths that match the text as closely as possible.

Internally, this library handles both fetching + loading the user-specified font as well as performing the text->path conversion itself. It supports different backends for each of these steps.

For my use case, I made use of the Google Fonts provider. It was easy to set up and only requires a Google Fonts API key, which can be generated for free. This allows me to use almost any font on Google Fonts to create my meshes. Some failed to load, but only a few and they seemed to be more obscure ones, and I didn’t bother to dig into why.

For the text->path conversion, svg-text-to-path defaults to using the fontkit backend. Fontkit is another pure JavaScript library that implements a font engine. I didn’t look into it too deeply, but it seems feature rich and has support for many advanced font features.

For my use case, my app runs in the browser. I could have used svg-text-to-path directly within it to generate these paths. However, this text->mesh feature isn’t core to my use case and I didn’t want to bloat the app with it. I also wanted to make it as easy as possible for users to set up, and wanted to be able to use my Google Fonts API key in a secure way.

So, I opted to create a tiny little backend service to take input text + params and return the generated path as a string. It’s a very minimal Bun webserver using Bun’s built-in Bun.serve. It exposes a single HTTP/JSON endpoint.

svg-text-to-path also includes a minimal built-in webserver, but I opted to create my own so that I could set up some custom caching and post-process the generated SVG to just extract the path. I opted to use an LLM to scaffold out most of this app and it worked pretty well. I feel like this kind of low-stakes one-off/standalone app is an ideal use case for them.

Here’s the source code if you’re interested, but I promise it’s nothing special: https://github.com/Ameobea/sketches-3d/tree/main/geoscript_backend/text-to-path

Anyway, the output of this is an SVG path which encodes a sequence of draw commands used to generate the text like this:

{
  "path": "M5.86 24L5.86 9.53L1.15 9.53L1.15 7.2L13 7.2L13 9.53L8.28 9.53L8.28 24ZM18.8 24.29Q17.09 24.29 15.85 23.47 ...."
}

lyon

Now that I had the path generation working, I needed a way to turn it into triangles for the mesh. Luckily, the excellent lyon Rust libraries (which I’ve used several times in the past for various projects) solve this problem perfectly.

The lyon_extra crate includes an SVG path parser which handles parsing that path into the underlying draw commands.

Then, the lyon_tessellation crate takes those draw commands and converts them into triangles. It handles all the hard parts and edge cases with concave shapes, hollow inner areas, discretizing bezier curves, and everything else.

I implemented a tiny WebAssembly wrapper that takes the input path and returns vertex and index buffers: https://github.com/Ameobea/sketches-3d/blob/main/src/viz/wasm/path_tessellate/src/lib.rs

There is a little bit of extra stuff for handling custom scaling, but other than that it’s really just a very thin wrapper over lyon functionality.

One note here is that I had to change the default FillTessellator options to set the fill-rule to non-zero, which I believe is the default for SVGs. This fixes the output for some fonts that contain self-intersecting paths, going from this:

to this:

extrusion

So at this point, I had two buffers containing vertices and indices defining a 2D mesh matching the path for the text. The only part that remains is extruding it into 3D. This is a pretty straight-forward and common operation to do on a triangle mesh.

To start, you first convert all the vertices from 2D to 3D by filling in the new axis with zeroes (so like (5, 10) -> (5, 0, 10)).

Then, you flip the winding order of all the triangles in your mesh. WebGL and almost all other rendering systems use counter-clockwise winding orders, and that defines which direction the triangle is visible from. To flip them, you can just swap the first and third index of each triangle in the index buffer like this:

1,2,3,5,7,9,1,4,2

to

3,2,1,9,7,5,2,4,1

Then, you create a duplicate of each of the vertices offset n units in the new axis (so like (5, 0, 10) -> (5, 2, 10)).

Then, join those new vertices with triangles but in the original (unflipped) winding order. That will make the top and the bottom face in opposite directions - the top facing up and the bottom facing down.

Finally, you generate triangle strips to join the border edges of the top and bottom faces. A border edge is any edge that is only part of exactly one face. Usually a graph representation like a half-edge data structure is used when working with meshes, which helps with this part.

The result should look something like this:

Here’s my source code if you’re interested, but note that it’s using my own internal mesh representation: https://github.com/Ameobea/sketches-3d/blob/main/src/viz/wasm/geoscript/src/mesh_ops/extrude.rs

If you did everything correctly and took care to keep track of the vertex indices carefully to avoid creating duplicate vertices at the same position, the resulting mesh should be well-formed and 2-manifold/watertight. This is a very important topological property and is a requirement for a variety of other mesh processing algorithms including CSG (constructive solid geometry).

The fact that the output meshes are manifold means that they can be combined with other meshes using boolean operations or sent through additional processing like smoothing. I’m not 100% positive that all paths generated from all glyphs using all fonts will end up producing manifold outputs, but everything I tested did.

conclusion

That’s it! After all of this, the output is a set of vertices and indices that define a 3D mesh representing the input text.

I integrated this functionality into my Geoscript language as a builtin function:

There are few steps to manage, but the powerful libraries under the hood (svg-text-to-path, fontkit, and lyon) handle all the complex stuff and heavy lifting.

Even though some of the critical libraries are in JavaScript and the fact that the generation happens on a remote webserver, I’ve found that for the (relatively short) text I convert it works quite fast - fast enough to work on-demand without waiting.

I’ve also not yet found any fonts that produce broken output or buggy meshes. It even works for complicated non-English scripts:

It was a fun little side-quest and I’m very happy with the results overall.

Fixing Linux Perf Inject Hanging

Tue, 18 Nov 2025 11:38:39 -0600

The Problem

I was working on optimizing some WebAssembly, and wanted to use the Linux perf utility to profile Google Chrome to analyze where the most time was getting spent in my code at the assembly level.

I was following a guide to do this: https://v8.dev/docs/linux-perf

There is one step that involves processing some artifacts generated during the profiling process. This involves running the following command:

perf inject --jit --input=perf.data --output=perf.data.jitted;

When I ran this command, it would start but then hang indefinitely. It seemed to be making no progress even after several minutes of running, and there was no noticeable CPU activity to indicate it was doing work.

The Cause

I eventually figured out that perf has a -v flag which enables verbose logging. When I re-ran the perf inject command with this flag included, I saw that perf was trying to download hundreds (maybe more) of debug info files for some of the symbols in the binary I was profiling.

I was running a normal out-of-the box Google Chrome browser without any debug info attached. It’s possible that this wouldn’t happen in the case that a profiling build of Chrome was used that included debug symbols built-in.

I’m not sure if these requests were timing out, getting throttled/rate-limited, or if they were just taking a long time to return. However, the result was that they were being processed very slowly and I had no idea if/when they would finish.

The Fix

For my particular case, I didn’t have any need to download this debug info, whether it was even working or not; the code I was interested in was JIT-compiled WebAssembly.

I found that this automatic debug symbol downloading was controlled by an environment variable: DEBUGINFOD_URLS

On my system, this was set to https://debuginfod.debian.net by default.

I was able to fix the problem by re-running the perf inject command with the environment variable cleared like this:

DEBUGINFOD_URLS= perf inject -v --jit --input=perf.data --output perf.data.jitted

This fix made the perf inject command finish in less than a second, yielding me a perf.data.jitted file that I could then analyze directly with perf report like normal.

Compiling the Boundary-First-Flattening Library to Wasm

Mon, 04 Aug 2025 01:47:56 -0500

Here is an account of the process I developed to get the boundary-first-flattening library building for use on the web via WebAssembly.

Boundary First Flattening (I refer to it as BFF throughout this article) is a powerful algorithm and library for “surface parameterization” - or projecting 3D surfaces into 2D. It also includes built-in support for other parts of a full UV unwrapping pipeline like bin-packing texture islands into a square. I was using it for my Geotoy project - a browser-based, Shadertoy-inspired web app for procedural geometry.

Getting this library compiled to Wasm was a very tricky process, mostly because BFF relies on some underlying linear algebra and math libraries to function. These both require some special tweaks and modifications of their own to compile to Wasm. Specifically, it uses SuiteSparse - which in turn requires a BLAS + LAPACK implementation.

Building OpenBLAS for Wasm

I was lucky here in that someone else had already done most of the hard work for compiling a BLAS implementation (OpenBLAS) to Wasm. It was done to support a Wasm version of an open-source speech recognition project called Kaldi.

They were kind enough to provide some pretty detailed notes: https://github.com/msqr1/kaldi-wasm2

These include some instructions compiling OpenBLAS for Wasm specifically. I followed them mostly, but had to tweak the compilation flags a bit in order to include LAPACK in the build as well (which is required for SuiteSparse).

Here’s an overview:

Clone OpenBLAS and check out commit 5ef8b19
Apply three small patches to trick the RISC64 target into working with Emscripten
Source your Emscripten emsdk installation, then build with this:

make CC=emcc FC=emcc HOSTCC=gcc \
    TARGET=RISCV64_GENERIC \
        ONLY_CBLAS=1 NOFORTRAN=1 NO_LAPACK=0 NO_LAPACKE=0 \
        C_LAPACK=1 BUILD_WITHOUT_LAPACK=1 USE_THREAD=0 \
        BUILD_BFLOAT16=0 BUILD_COMPLEX16=0 BUILD_COMPLEX=0 \
         CFLAGS="-O3 -ffast-math -msimd128 -mavx"

This should run for a while and then spit out a .a file containing the built library in the current directory.

I end up getting errors at the end like unable to find library -lgfortran, but it seems to be late enough in the build that it doesn’t matter. As long as you wind up with a libopenblas_riscv64_generic-r0.3.28.a file in the openblas root dir, everything should be good.

Install with:

mkdir /home/YOURUSERNAME/blas-build
PREFIX=/home/YOURUSERNAME/blas-build NO_SHARED=1 make install

(Note that ~ doesn’t seem to work in paths in CMake as a shorthand for home directory; you have to provide the full explicit path in order for it to work.)

Building SuiteSparse

As I mentioned before, SuiteSparse depends on both BLAS and LAPACK. The OpenBLAS build from before (as configured) includes LAPACK symbols as well.

However, try as I might, I couldn’t get CMake’s built-in support for finding and configuring BLAS to work. I ended up having to update the CMakeLists.txt and add these lines right before if ( SUITESPARSE_USE_SYSTEM_GRAPHBLAS ) (line 116):

set(BLAS_LIBRARIES "/home/YOURUSERNAME/blas-build/lib/libopenblas.a")
set(LAPACK_LIBRARIES ${BLAS_LIBRARIES})

^ This overrides the default behavior of trying to search all over for any kind of valid BLAS/LAPACK implementation it can find and points it directly to the one we just built instead.

I also had to apply this additional patch to get rid of some conflicting symbols that were getting re-defined for some reason from OpenBLAS/LAPACK:

diff --git a/SuiteSparse_config/SuiteSparse_config.h b/SuiteSparse_config/SuiteSparse_config.h
index 7d7d3f3f6..b073183f1 100644
--- a/SuiteSparse_config/SuiteSparse_config.h
+++ b/SuiteSparse_config/SuiteSparse_config.h
@@ -612,11 +612,11 @@ int SuiteSparse_version     // returns SUITESPARSE_VERSION
 #define SUITESPARSE_BLAS_DSCAL      SUITESPARSE_BLAS ( dscal  , DSCAL  )
 #define SUITESPARSE_BLAS_DNRM2      SUITESPARSE_BLAS ( dnrm2  , DNRM2  )

-#define SUITESPARSE_LAPACK_DPOTRF   SUITESPARSE_BLAS ( dpotrf , DPOTRF )
-#define SUITESPARSE_LAPACK_DLARF    SUITESPARSE_BLAS ( dlarf  , DLARF  )
-#define SUITESPARSE_LAPACK_DLARFG   SUITESPARSE_BLAS ( dlarfg , DLARFG )
-#define SUITESPARSE_LAPACK_DLARFT   SUITESPARSE_BLAS ( dlarft , DLARFT )
-#define SUITESPARSE_LAPACK_DLARFB   SUITESPARSE_BLAS ( dlarfb , DLARFB )
+// #define SUITESPARSE_LAPACK_DPOTRF   SUITESPARSE_BLAS ( dpotrf , DPOTRF )
+// #define SUITESPARSE_LAPACK_DLARF    SUITESPARSE_BLAS ( dlarf  , DLARF  )
+// #define SUITESPARSE_LAPACK_DLARFG   SUITESPARSE_BLAS ( dlarfg , DLARFG )
+// #define SUITESPARSE_LAPACK_DLARFT   SUITESPARSE_BLAS ( dlarft , DLARFT )
+// #define SUITESPARSE_LAPACK_DLARFB   SUITESPARSE_BLAS ( dlarfb , DLARFB )

 // double complex
 #define SUITESPARSE_BLAS_ZTRSV      SUITESPARSE_BLAS ( ztrsv  , ZTRSV  )

This isn’t necessary to get the build to work, but it’s necessary to prevent runtime errors when using the final boundary-first-flattening library.

Now, SuiteSparse can be built with:

mkdir build && cd build

emcmake cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=emcc -DCMAKE_CXX_COMPILER=em++ -DEMSCRIPTEN=True -DBUILD_SHARED_LIBS=OFF -DBUILD_STATIC_LIBS=ON -DCHOLMOD_USE_CUDA=OFF -DSUITESPARSE_USE_CUDA=OFF "-DSUITESPARSE_ENABLE_PROJECTS=suitesparse_config;amd;colamd;cholmod" -DSUITESPARSE_USE_FORTRAN=OFF -DBLA_STATIC=ON -DBLAS_VENDOR=OpenBLAS -DBLA_VENDOR=OpenBLAS -DSUITESPARSE_USE_OPENMP=OFF -DCMAKE_FIND_DEBUG_MODE=OFF -DBLA_F95=OFF ..

emmake make -j12

At some point, the build will freak out and break with errors about undefined symbols and conflicting function declarations. HOWEVER, it should have gotten far enough along to produce some nice .a files in places like build/AMD/libamd.a.

These are what we’re really after, and as long as you have a build/CHOLMOD/libcholmod.a you should be golden.

Once the build finishes, the output static library .a files need to be moved somewhere that Emscripten can find them.

I discovered that emscripten will look at this location in its search path, which is nice and isolated:

/home/YOURUSERNAME/emsdk/upstream/emscripten/cache/sysroot/usr/local/lib/

I created that directory and then copied the following files into there:

/home/YOURUSERNAME/blas-build/lib/libopenblas.a to libopenblas.a and also to liblapack.a
- it should be duplicated to both locations, as this library contains symbols for both BLAS and LAPACK but the build system looks for both separately
SuiteSparse/build/AMD/libamd.a to libAMD.a
SuiteSparse/build/CAMD/libcamd.a to libcamd.a
SuiteSparse/build/CCOLAMD/libccolamd.a to libccolamd.a
SuiteSparse/build/CHOLMOD/libcholmod.a to libcholmod.a
SuiteSparse/build/SuiteSparse_config/libsuitesparseconfig.a to libsuitesparseconfig.a

Building `boundary-first-flattening`

I had to make some changes to the CMake config for the BFF library itself to get it building with Wasm.

(Full diff I applied: https://github.com/GeometryCollective/boundary-first-flattening/commit/680a2fd384f736b7300f73126796dd0001970294)

I edited cmake/FindSuiteSparse.cmake and updated the list of suitesparse libraries to this:

## Default behavior if user doesn't use the COMPONENTS flag in find_package(SuiteSparse ...) command
if(NOT SuiteSparse_FIND_COMPONENTS)
	list(APPEND SuiteSparse_FIND_COMPONENTS AMD CAMD CCOLAMD COLAMD CHOLMOD suitesparseconfig)  ## suitesparse and metis are not searched by default (special case)
endif()

This removes some of the libraries which aren’t needed (and aren’t actually built with the given config) and adds in the suitesparseconfig library explicitly which seemed to be missing.

I also edited the root CMakeLists.txt file to add these three lines before the # suitesparse section:

include_directories("/home/casey/SuiteSparse/CHOLMOD/Include")
include_directories("/home/casey/SuiteSparse/SuiteSparse_config")
include_directories("/home/YOURUSERNAME/blas-build/include")

^ these just help the compiler find the header files for SuiteSparse and BLAS/LAPACK that it needs in order to build.

Code Fixes

There were also a few fixes I had to make to the BFF source code itself to make it work.

I had to update several files to add imports for #include <cstdint> (this was required for me to even be able to build the library without emscripten). I imagine it has something to do with C++ compiler versions.

There’s one issue in the boundary-first-flattening library that caused runtime issues when compiling for 32 bit targets (like Wasm). It was caused by assuming that size_t matches the integer sizes used by some underlying SuiteSparse SparseMatrixes.

In reality, those SuiteSparse matrix indices were always 64 bits even on 32-bit targets, but size_t was dropping to 32. This predictably created a lot of issues when casting pointers and copying memory around.

I fixed it by patching this file: https://github.com/GeometryCollective/boundary-first-flattening/commit/680a2fd384f736b7300f73126796dd0001970294#diff-1d5499e970a46aba46111951b7a9f6b526989e47973b06d12d6fdd02f0826e76

Building

Once those CMake config files and code changes were applied, I built the library like this:

mkdir build && cd build

# This configures it to just build a `libbff.a` file.  This might be all that
# you need for your specific use case.
emcmake cmake -DBFF_BUILD_GUI=OFF -DBFF_BUILD_CLI=OFF -DCMAKE_BUILD_TYPE=Release ..

# You can also configure it to compile the CLI to Wasm, but this has limited utility:
emcmake cmake -DBFF_BUILD_GUI=OFF -DBFF_BUILD_CLI=ON -DCMAKE_BUILD_TYPE=Release ..

# Then run the build
emmake make -j12

Custom Wrapper Library

There’s not a ton of value in having the raw CLI itself compiled to Wasm, and the libbff.a file exposes a C++ interface that isn’t directly exposed to Wasm.

A more useful solution is to create a custom wrapper library that exposes some high-level bindings to JS.

I set up one that handles this. It exposes a single unwrapUVs function which takes input mesh positions and vertices along with a few params controlling the BFF algorithm. It then computes the projection and returns a class holding the unwrapped UVs and new vertices and indices.

You can find the full implementation in this commit: https://github.com/GeometryCollective/boundary-first-flattening/commit/680a2fd384f736b7300f73126796dd0001970294#diff-67e14663983ede25ad744af48bbd4faff5ba01e893c956c0e49d10a4cc25eefd

One thing to note is that I have a little extra script in the Justfile to inject a line into the generated JS module so it can be import as an ES Module.

The only other piece I have is a little TypeScript shim to handle the async initialization of the UV unwrap module, move data in and out of Wasm, and handle errors: https://github.com/Ameobea/sketches-3d/blob/main/src/viz/wasm/uv_unwrap/uvUnwrap.ts

If you’re interested in running this yourself, I’ve included the whole thing with the Emscripten-generated Wasm and JS file along with the TypeScript shim here: https://github.com/Ameobea/sketches-3d/tree/main/src/viz/wasm/uv_unwrap

Results

After all that work, you’re rewarded with a fully-functional BFF implementation that can handle the whole process of generating valid UVs for arbitrary input meshes:

I’m afraid I won’t be able to provide support to people if they run into trouble with this method or have issues getting this to work for their own use cases. As you can probably see, this whole thing is quite brittle and required a ton of hacks and workarounds to get working.

That being said, I hope this writeup serves as a decent guide or starting point and helps people make use of the amazing boundary-first-flattening library in the browser.

Fixing Google Chrome Using Wrong GPU on Linux

Fri, 01 Aug 2025 19:53:24 -0500

Well, here’s another chapter in the saga of graphics and GPU issues with Google Chrome on Linux.

I have a 7900 XTX GPU, and I think at least part of these issues are a result of AMD GPU drivers being pretty messed up on Linux.

The Problem

One day after updating my packages and rebooting my system, WebGL web apps were running at very low FPS in my Google Chrome browser. It was really slow - like sub-10 FPS when I usually get 165.

The Cause

After some debugging, I realized that my integrated GPU was being used instead of my dedicated GPU. I figured this out using the radeontop application, which has an argument to pick which GPU you’re recording performance for. My integrated GPU had zero usage, but the integrated GPU was maxed out.

The Fix

I eventually noticed this line in the output from the google-chrome process:

[14533:14533:0801/194836.930457:ERROR:ui/ozone/platform/wayland/gpu/wayland_surface_factory.cc:233] '--ozone-platform=wayland' is not compatible with Vulkan. Consider switching to '--ozone-platform=x11' or disabling Vulkan

I then tried launching Chrome with this argument:

google-chrome --ozone-platform=x11

And that fixed the problem. It went back to using my discrete GPU and WebGL performance went back to normal.

I use Wayland for my desktop environment. I don’t know why this is necessary, but whatever I guess.

I do have Vulkan enabled in chrome://flags but that’s required for me to avoid other issues I get with my max FPS getting limited.

Note that I tried setting “Preferred Ozone platform” to X11 in chrome://flags but it had no effect. The fix only worked when using the argument when launching Chrome.

Anyway, this exact of flags and configs makes Chrome work well for me on my system. I dread the day when one something gets changed to break this delicate balance.

Fixing Sveltekit `fetch` error `ERR_SSL_WRONG_VERSION_NUMBER` with NGINX reverse proxy

Tue, 08 Jul 2025 02:51:18 -0500

The Problem

I am using SvelteKit and importing a WebAssembly module on the server side inside a +page.server.ts file like this:

const WasmPromise = import('src/viz/wasmComp/geoscript_repl').then(
  async (engine) => {
    await engine.default(fetch('/geoscript_repl_bg.wasm'));
    return engine;
  }
);

It works when I run for local development, but it fails when running on my VPS in production. Some notes about my deployment environment:

I’m Sveltekit’s adapter-node
I’m deploying the Sveltekit app inside Docker behind an NGINX reverse proxy

This Wasm module as built using wasm-bindgen for Rust, and it needs the path to the Wasm to be explicitly provided since it’s running on the server side rather than the client.

The fetch('/geoscript_repl_bg.wasm') call was the one that was failing. This is the error that I got in the Sveltekit logs:

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async fetch (file:///app/build/server/index.js:4467:80) {
  cause: [Error: 40AC6882457F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:
  ] {
    library: 'SSL routines',
    reason: 'wrong version number',
    code: 'ERR_SSL_WRONG_VERSION_NUMBER'
  }
}

The Cause

The issue seems to be due to the fact that my NGINX server was setting the X-Forwarded-Proto header to https. This is correct behavior, but the Sveltekit node adapter was setting the protocol on the internal request for the Wasm module to HTTPs because of it.

Here’s the code responsible for that: https://github.com/sveltejs/kit/blob/96ce0f9735c4d1a1d467c1f9641e91b5858eea93/packages/adapter-node/src/handler.js#L185

The Fix

Luckily, there’s a way to override this auto-detection behavior. By setting an ORIGIN environment variable when running the Sveltekit server, I can force these internal requests to get made back to localhost with HTTP rather than HTTPS:

ORIGIN="http://127.0.0.1:5814" node ./build/index.js

This successfully resolved the issue for me, and my server-side Wasm worked in production.

Fixing ltree syntax error in Postgres 15

Wed, 02 Jul 2025 16:06:30 -0500

The Problem

I was recently working with some Postgres tables that have an ltree column. I had written tests for my new feature that made use of them and all the tests passed locally.

However, the tests were failing in CI with a vague error:

ERROR: ltree syntax error at character 9 (SQLSTATE 42601)

The query that was producing this error was quite simple as well:

INSERT INTO "my_table" ("path","value") VALUES ('22e8f4e1-437f-448b-a8fb-0d7fbed8de7a','foo')

In this case, the path column was of type ltree. Running that exact query locally worked fine and I was confused as to why it wasn’t working in CI.

The Cause

It turns out that I was running a more modern version of Postgres locally (17.5) than what was being run in CI (15).

In older versions of Postgres including 15, there is a limitation that values inserted into ltree columns can’t contain the - character. This limitation was removed in more recent versions.

I was able to work around this issue by just stripping all non-alphanumeric characters out of my UUIDs before inserting them into the column.

Fixing Google Chrome WebGL Low Frame Rate After Turning Monitors Off on Linux

Wed, 14 May 2025 15:00:57 -0500

The Problem

I have two monitors: my main one at 165hz and a side monitor at 60hz. When I first boot up my computer, I’m able to run web-based games and visualizations at full 165hz on my main monitor.

However, after I turn my monitors off for the night and then get back on the next morning, all my WebGL-based applications are locked to 60 FPS or lower (and they feel stuttery and generally worse than even what I’d expect from stable 60 FPS)

Attempted Fixes

I’ve tried closing all chrome windows/tabs and re-starting it, tried using a different version of chrome (google-chrome-unstable and I even tried Microsoft Edge), tried launching chrome with a variety of different flags and settings, but nothing works. The only fix is to log out and log back in from scratch, which is a hassle since I have to set everything back up.

I’m running Wayland. If I log in with X, then things run at like 40FPS average from the start. I’m using an AMD GPU with amdgpu drivers.

Other non-browser graphics apps work fine and run at the correct frame rate. Firefox doesn’t have this particular issue, but there are other issues I run into with input handling among other things that make me really want to avoid using it here unless necessary.

I tried launching Chrome with a variety of different flags including various combinations of --ozone-platform=wayland, --enable-features=UseOzonePlatform, --use-angle=gl, and --enable-features=VaapiVideoDecodeLinuxGL but all of these resulted in either no change or complete loss of hardware acceleration for WebGL.

The Real Fix

After messing around for a while, I discovered that I could fix the problem entirely by enabling Vulkan in the Chrome settings via chrome://flags/:

I could have sworn that this was broken in the past so I wasn’t able to do it, but now it works fine and completely fixed my issue.

EDIT 2025-05-18:

After some time, this fix stopped worked. I tried a bunch of stuff and what finally worked was swapping which monitor was “Primary” in my system settings:

After I set my higher-FPS monitor to be primary and rebooted Google Chrome, my WebGL went back to working at 165 FPS again.

Fixing WHEA_UNCORRECTABLE_ERROR Bluescreen with Windows + Linux Dual Boot

Tue, 15 Apr 2025 14:50:19 -0700

The Problem

I dual-boot multiple Linux installs along with Windows. At some point, possibly coinciding with an update to one of my Linux installs, my Windows partition failed to boot with a bluescreen error showing a WHEA_UNCORRECTABLE_ERROR error message. I tried booting multiple times with no success.

Most info I found online about this error seemed to indicate that it was related to hardware issues, but I hadn’t made any hardware changes recently and my Linux installs continued to boot and work fine.

The Fix

My boot setup is a bit unique since the Linux partition I use primarily isn’t the one that manages the bootloader.

To fix the problem, I booted into the Linux install that controls the bootloader and ran sudo update-grub.

After doing that and rebooting, my Windows partition booted and worked fine without any further changes necessary.

Getting Kinematic Object Movement Interpolation Working in Bullet Physics/ammo.js

Fri, 14 Mar 2025 16:07:46 -0700

Background

I’m working on a browser-based game engine that uses Three.JS for rendering and a WebAssembly port of the Bullet physics engine called ammo.js for its physics engine and character controller.

Bullet is a very old physics engine and Ammo.JS is more or less a direct copy of it with a few changes to support usage in the browser. There are a lot of rough edges and things you have to figure out yourself in order to use it.

The Problem

One problem I ran into was getting interpolated position changes to work for kinematic objects. Let’s way that your game’s animation frame is firing at 60FPS and you have the Bullet physics engine configured to run at a static 180hz tick rate. This means that for every animation frame, there will be 3 simulated physics engine ticks.

I had some kinematic objects that I controlled manually from the JS side. I computed and set the positions of the objects each animation frame, applying the new position to both Three.JS and ammo.js/bullet simultaneously. I set their transforms in bullet via the btMotionState interface which, according to code comments and docs, should help support interpolated movement and continuous collision detection.

I was running into problems where fast-moving kinematic objects would clip through the player or otherwise bug out when colliding.

No matter how high I turned up the physics engine tick rate, the problem wouldn’t improve. Theoretically, at some point turning the tick rate high enough should resolve the issues since the movement between each internal tick will eventually be small enough to prevent the clipping.

The Cause

It turns out that continuous collision detection (referred to as CCD in bullet code and docs) is complicated and doesn’t really work by default.

For starters, I was using the btDiscreteCollisionWorld for my game engine. This has “discrete” in the name, so I guess it’s not really much of a surprise that it doesn’t do much continuous stuff out of the box. For what it’s worth, there is no continuous collision world that I could find which is shipped with ammo.js.

After much digging through the code, I found some other places where continuous collision detection is used when solving individual collisions between objects. If an object has an associated btMotionState, the new position of the object is queried at the beginning of each call to btDiscreteDynamicsWorld::stepSimulation. This happens in the saveKinematicState function.

That function computes linear and angular velocities for objects that have moved. These velocities are then used by CCD, but weren’t really having any effect as far as I could tell. This is probably because the actual worldTransform of the object isn’t interpolated for the internal sub-ticks simulated by bullet (internalSingleStepSimulation). As a result, my character controller was seeing the objects at their final point at the end of the simulated frame rather than seeing its movement interpolated along sub-ticks.

The Solution

To resolve this, I manually interpolated the world transform of kinematic collision objects each subtick.

Then, I added a new applyManualMotionStateInterpolation method to btDiscreteDynamicsWorld which I call before each time internalSingleStepSimulation is run. applyManualMotionStateInterpolation uses that m_lastWorldTransform variable along with the velocities computed in saveKinematicState to simulate the transform of the object dt seconds after the end of the last frame.

Here’s the full function:

/**
 * This is some custom stuff I added to get interpolated positions of kinematic objects for internal
 * subticks.  The built-in interpolation doesn't seem to work for kinematic objects, and this forces
 * interpolation between the past and most recently set transforms for kinematic objects.
 */
void btDiscreteDynamicsWorld::applyManualMotionStateInterpolation(btScalar dt) {
	for (int i = 0; i < m_nonStaticRigidBodies.size(); i++){
		btRigidBody* body = m_nonStaticRigidBodies[i];
		if (body->isActive() && body->isKinematicObject()) {
			auto startPos = body->getLastFrameWorldTransform();
			btTransform dstPos;
			btTransformUtil::integrateTransform(
				startPos,
				body->getInterpolationLinearVelocity(),
				body->getInterpolationAngularVelocity(),
				dt,
				dstPos
			);
			body->setWorldTransform(dstPos);
			body->setLastFrameWorldTransform(dstPos);
		}
	}
}

It uses the pre-existing integrateTransform to apply the velocities which is where the math and heavy lifting happens. Then, it sets the simulated destination position into m_worldTransform via setWorldTransform - which is the actual live position of the object used in physics simulations.

I needed to make one additional change to the saveKinematicState function in order for this to be valid when using fixed tick-rate mode for bullet:

void btRigidBody::saveKinematicState(btScalar timeStep)
{
	//todo: clamp to some (user definable) safe minimum timestep, to limit maximum angular/linear velocities
	if (timeStep != btScalar(0.))
	{
+		m_interpolationWorldTransform = m_worldTransform;

		//if we use motionstate to synchronize world transforms, get the new kinematic/animated world transform
		if (getMotionState())
			getMotionState()->getWorldTransform(m_worldTransform);

		btTransformUtil::calculateVelocity(m_interpolationWorldTransform,m_worldTransform,timeStep,m_linearVelocity,m_angularVelocity);
		m_interpolationLinearVelocity = m_linearVelocity;
		m_interpolationAngularVelocity = m_angularVelocity;
		m_interpolationWorldTransform = m_worldTransform;
	} else {
		printf("Zero timestep???\n");
	}
}

Without that change, the transforms of objects affected by this manual interpolation wouldn’t reflect the portion of the movement from the remainder of the dt. m_interpolationWorldTransform is the transform that the object should have by the end of the current tick, but it doesn’t quite make it there due to that remainder.

Setting m_interpolationWorldTransform equal to m_worldTransform works around this by computing the motion as starting from the last simulated point of the object rather than the last set point.

Results

As a result of these fixes, I was able to actually benefit from turning up the simulation tick rate and my collision issues with fast-moving kinematic objects vastly improved.

Note that this fix might not be valid for all bullet physics configurations. If you do stuff like setting your btDispatcher’s m_dispatchFunc to DISPATCH_CONTINUOUS (it defaults to DISPATCH_DISCRETE) or maybe even just use certain types of collision objects, this might result in incorrect physics if the CCD (which seems to be inactive and do nothing in my case) actually gets used somewhere.

Other Misc. Notes

Use specialized collision objects like btBoxShape when possible rather than btTriangleMesh. These seem to behave much better (less clipping through fast-moving objects, jitter, inaccurate displacements, etc.) and are much more performant.
The built-in btKinematicCharacterController doesn’t work well out of the box. They note this in their official docs, and its code hasn’t been touched in a decade or more. It also requires careful tuning of constants like jump height, step height, max penetration depth, physics engine tick length, and more in order to get decent results.

I had to make numerous tweaks and customizations to get the character controller working well for my use case. I also added a variety of extra features like clamping the character to moving platforms, adding external velocity for things like directional dashes and jump pads, etc. These changes can be found on my fork.

Creating Constrained Bezier Curves for an Envelope Generator

Wed, 25 Dec 2024 19:27:03 -0600

I recently finished some work involving constrained bezier curves for use in my browser-based digital audio workstation. Specifically, I used them in its envelope generator, which looks like this:

The circles are draggable handles that allow the user to create whatever shape they desire for the envelope. The green handles define the start and end point of each curve segment and the blue circles control its shape/steepness.

Motivation

Originally, I used the function x^n to build the curves that made up the envelope. This curve type worked decently well and allowed for curves of varying steepness to be represented, but it had some problems:

The x^n function can produce numerically unstable results with very large or very small exponents. This can result in very rapid changes in underlying value or even produce NaN or Infinity values. As you might imagine, isn’t very good for audio use cases.
Curves aren’t symmetrical going in vs. going out which makes it impossible to create symmetrical peaks/dips, which is something often desired.

So the main goal was to add support for a different curve type that would solve these problems and ideally add options for even more expressive shapes than before.

Bezier curves were an obvious choice due to how simple they are to implement and their wide adoption. However, they also pose some challenges of their own:

If x^n didn’t offer enough control over the shape of the curve, Bezier curves offer too much control. For example, it’s possible to create bezier curves that loop back in the X dimension like this:

This clearly makes no sense for an envelope generator use case. I needed a way to constrain the generated curves in a way that ensures they’re valid.

The control points - which are used to adjust the shape of the curve - don’t lie on the curve itself. To work nicely in my existing UI, I needed to be able to add a handle somewhere on the curve that could be dragged to adjust its shape directly.
Additionally, for cubic bezier splines, there are two different control points which need to be selected to shape the curve. I needed a way to pick values for these behind the scenes from a single handle position.

Constrained Bezier Curves

A cubic bezier curve is defined by four points: A start point (P0), end point (P3), and two control points (P1 and P2). The start and end point of the curves are fixed, so all we have to worry about is picking values for the two control points.

I experimented with some bezier curve editors and manually tested out some different control point patterns. I eventually realized that setting both control points to the same position worked quite well and get me very close to what I was looking for.

It turns out that this produces curves with a lot of desirable characteristics. Crucially, the curves’ X values all monotonically increase/decrease along the span of the curve, so no looping back or other degenerate curve shapes can be created.

One important property of bezier curves which I learned is that all points on the curve will fit within the convex hull of the points that define it. So, if the control points are set to a position within the rectangle bounded by the start and end points, all resulting curves should be valid.

Although a lot of possible curve shapes aren’t creatable with coincident control points, it still provides more than enough control over the shape of the curve. So, I decided to stick with this method to constrain the curves.

Computing Control Points

The final piece to figure out was a way to control the shape of the curve by dragging a handle on the curve itself rather than moving a floating control point directly.

I needed to pick a point on the curve on which to attach the handle. I experimented with a few options and quickly discovered that the midpoint of the curve by length was the obvious best choice.

Bezier curves can be defined as a linear combination of lower-degree bezier curves. At the midpoint of the curve (t=0.5), the direction of the curve is evenly balanced between heading towards the shared control point and heading towards the endpoint. This works out to producing an instantaneous direction that is parallel to the line from the start point to the endpoint.

Anyway, now remainder of problem is quite well defined. P0 and P3 are already defined and fixed. There’s a constraint that the resulting curve must pass through the handle’s position (H) at t=0.5. Given that, we have to solve for the shared control point (C).

Here is the formula for a constrained cubic Bezier curve using coincident control points:

$$ConstrainedBezier(t) = (1 - t)^3 P_0 + 3 (1 - t)^2 t C + 3 (1 - t)t^2 C + t^3 P_3$$

This can be simplified using some algebra into the following:

$$ConstrainedBezier(t) = (1 - t)^3 P_0 + 3t(1 - t)C + t^3P_3$$

Then, to solve for the position of the handle H, we can plug its t=0.5 value into the equation:

$$H = ConstrainedBezier(t) = (1 - 0.5)^3 P_0 + 3 \times 0.5 (1 - 0.5)C + 0.5^3P_3$$

Those constants collapse down, and after re-arranging H and C and applying a bit more algebra, we get the following equation yielding the position of the shared control point:

$$C = \frac{1}{6} P_0 + \frac{4}{3} H + \frac{1}{6} P_3$$

I was pretty surprised that the solution ended up being so simple - a linear combination of the start point, end point, and handle position. Sure enough, when I plugged them in, I reliably got out control points that generated curves which accurately intersected the handles at the curves’ midpoints.

Here’s how it looks all put together:

Dragging the handle adjusts the curve in a way that feels intuitive and responsive. I was very happy with how it ended up feeling to use this.

The pink circle is the computed shared control point for the curve. As a final step, I clamp that control point to be within the bounds of the envelope segment and then re-calculate the handle position using that clamped value. This ensures that the curve always remains valid and produces values that are in bounds.

One other thing I noticed while setting this up is that the control point and handle always end up on the same line as the midpoint between the start and end point (this is drawn in pink in the animation above).

Conclusion

That’s about it! I just wrote this up because I thought the properties of cubic bezier curves with coincident control points ended up being very neat and potentially useful in other domains.

I’m very happy with how this ended up working in the envelope generator itself as well. The curves look beautiful and seem to be working very well for generating musically interesting envelopes.

Fixing Rust+WebAssembly Memory Access Out of Bounds Errors in Debug Mode

Mon, 23 Dec 2024 12:18:52 -0600

The Problem

In my project built with Rust compiled to WebAssembly, I started seeing errors like this that caused a crash - but only when it was compiled in debug mode:

wavetable.wasm-011dbbd6:0x17b89 Uncaught RuntimeError: memory access out of bounds
    at wavetable.wasm._ZN9wavetable2fm7effects14EffectInstance10from_parts17h321e4433e1216e56E (wavetable.wasm-011dbbd6:0x17b89)
    at wavetable.wasm._ZN9wavetable2fm7effects11EffectChain10set_effect17h969349e414104e9eE (wavetable.wasm-011dbbd6:0x1ba60)
    at wavetable.wasm.fm_synth_set_effect (wavetable.wasm-011dbbd6:0x314d3)
    at FMSynthAWP.port.onmessage (FMSynthAWP.js:171:37)

When compiling in release mode, the code ran just fine.

The decompiled debug-mode Wasm at the point of the crash looked like this:

...
(local $var833 i32)
(local $var834 i32)
(local $var835 i32)
global.get $__stack_pointer
local.set $var22
i32.const 1767008
local.set $var23
local.get $var22
local.get $var23
i32.sub
local.set $var24
local.get $var24
global.set $__stack_pointer
local.get $var24
local.get $var1
i32.store offset=20 // <- this is the instruction which produced the invalid access

The Cause

The only thing going on in this function up to that point is that it’s bumping the stack pointer down to make space for the local variables used by the function.

When I hovered over the $__stack_pointer variable in Chrome Devtools, I saw that it currently had this value stored:

type: "i32"
value: -792224

The stack pointer is definitely not supposed to be negative.

So the actual thing going on here was actually a stack overflow. That constant 1767008 earlier on in the function was how much stack space the function was trying to request - around 1.75 MB. This was more than what was available, and that caused the stack pointer to go negative and produce an invalid memory access when trying to read it.

This tracked with the code for the function itself. Within it, I was initializing some large buffers in memory using code like this:

let buffer = Box::new(CircularBuffer::new());

That CircularBuffer type was very large and stored the entire buffer inside itself directly without any indirection:

pub struct CircularBuffer<const LENGTH: usize> {
  buffer: [f32; LENGTH],
  /// Points to the index that the most recently added value was written to
  head: usize,
}

When Rust initializes Boxes in debug mode, it first creates the value on the stack and then copies it to the heap. In release mode, this copy is optimized out in most cases, so the large value are written directly into the allocated memory. That’s why this stack overflow only happens in debug mode.

A bit of a side note:

It used to be possible to work around this issue by using the unstable box syntax in Rust to force the stack-to-heap copy to not occur, like this:

let buffer = box CircularBuffer::new();

However, box syntax has since been removed with no plans on bringing it back.

The comment linked above mentions some alternatives that would bring the same functionality back (they call it “placement new”). But as far as I can tell, as of writing this (Dec. 2024) none of these alternatives are available yet - stable or otherwise.

A Workaround

I found two different fixes for this issue.

The first one is to manually bump the stack size to work around it. This can be done by editing the .cargo/config.toml file in your project’s workspace:

[target.wasm32-unknown-unknown]
rustflags = [
  "-C", "link-args=-z stack-size=15000000",
]

However, this has the effect of increasing the stack size for release mode as well which isn’t necessary.

I tried to configure it to only increase it in dev mode, which required me to enable the profile-rustflags feature for my workspace. After doing that, I got compilation errors for some of my dependencies which complained about the -z stack-size argument not being supported by my linker, which didn’t happen before.

I’m not sure what was going on with that, but I decided to stop trying this approach and go for what turned out to be a better solution.

A Better Fix

Rather than work around the problem by just increasing the stack size, I decided to solve the underlying problem that caused it by just reducing the amount of data being written to the stack.

To do this, I explicitly implement the optimization that box syntax/placement new would provide by allocating uninitialized memory and then writing my data into it manually. This requires a bit of unsafe code, but looks something like this:

let mut buffer: Box<MaybeUninit<CircularBuffer<_>>> =
  Box::new_uninit();
let buf_ptr = buffer.as_mut_ptr();
unsafe {
  let inner = &mut (*buf_ptr).buffer;
  inner.fill(0.);
  (*buf_ptr).head = 0;
}
let buffer = unsafe { buffer.assume_init() };

For my situation, there was an even better solution.

Since every single field of the struct I was initializing gets filled with data which has a bit representation of all zeroes, I can use a shortcut to initialize my buffer:

let feedback_buffer: Box<CircularBuffer<_>> =
  unsafe { Box::new_zeroed().assume_init() };

I can just ask for the memory to be pre-filled with all zeroes from the allocator and leave it at that!

Once I made this change and fixed a couple of other places where I was initializing some large boxed values on the stack, the invalid memory access errors went away and my program started working in debug mode again.

Fixing Cypress Tests Failing with 404 Errors in Github Actions CI but Working Locally

Sun, 24 Nov 2024 18:58:31 -0800

I have some very basic Cypress tests set up for one of my personal projects - a single-page application created with React. I have some simple Github Actions configured to automatically run those tests every time I push to the repository.

At some point somewhat randomly, those tests started failing in CI. It was probably due to some dependency getting upgraded or a change in the environment in which Github Actions runs. The tests still ran fine locally when using the same commands and configuration, though, and I struggled to figure out why.

I was using the official Cypress Github Action to run the tests. The config I was using is very basic with little to nothing special going on

  cypress-test:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - uses: actions/checkout@v4

      - name: Download built site artifacts
        uses: actions/download-artifact@v4.1.7
        with:
          name: dist
          path: ./dist

      # Run Cypress tests
      - name: Cypress Run
        uses: cypress-io/github-action@v6
        with:
          browser: chrome
          start: yarn cypress:serve
          wait-on: 'http://localhost:9000'

Here’s the error I was getting in CI:

  2) Entrypoint
       Should render composition 46 without errors:
     CypressError: `cy.visit()` failed trying to load:

http://localhost:9000/composition/46

The response we received from your web server was:

  > 404: Not Found

This was considered a failure because the status code was not `2xx`.

When running Cypress locally, all the tests pass without issue.

The application I was testing is a single-page app. The /composition/46 path doesn’t map to anything on the filesystem and instead is handled by the application itself. I had configured my cypress:serve script using http-server to proxy missing URLs back to the root index.html file, just like other SPA dev servers do.

And here’s what I had in my package.json for that cypress:serve script:

{
  "scripts": {
     "cypress:serve": "http-server dist -p 9000 -P http://localhost:9000? -c-1"
  }
}

To re-iterate, this command works totally fine when running locally. I can run yarn cypress:serve, navigate to http://localhost:9000/composition/43, and view the expected page which is returned with a 200 response code.

The Problem + The Fix

After multiple hours of banging my head against this problem, I finally figured out what was causing the issue.

For whatever reason, using localhost in the proxy path for the http-server command doesn’t work in Github Actions. I’m not sure if it has something to do with DNS, ipv6, the networking that GH Actions uses to communicate between commands, or something else.

In any case the fix turned out to be as easy as replacing localhost with 127.0.0.1 in my http-server command:

diff --git a/package.json b/package.json
index 1331d39..c6ba50e 100644
--- a/package.json
+++ b/package.json
@@ -51,7 +51,7 @@
     "prettier": "prettier --write \"src/**/*.{ts,js,tsx}\"",
     "cypress:open": "cypress open",
     "cypress:run": "cypress run --browser chrome",
-    "cypress:serve": "http-server dist -p 9000 -P http://localhost:9000? -c-1"
+    "cypress:serve": "http-server dist -p 9000 -P http://127.0.0.1:9000? -c-1"
   },
   "dependencies": {
     "@pixi/app": "^7.4.2",

So if you’re having similar issues and use a similar setup, give that a try.

Notes on Self Hosting a Bluesky PDS Alongside Other Services

Wed, 13 Nov 2024 19:03:37 -0800

I’ve recently set up a Bluesky Personal Data Server (PDS) to store the data for my Bluesky account. I wanted to host it on my server alongside the many other web apps, databases, and many other services. I additionally wanted to use my top-level domain as my handle.

I started out following the install guide on the official PDS repo and it initially started out pretty well. However, I pretty quickly ran into some issues where the default config didn’t work for me.

I eventually managed to solve all of the issues and get it running smoothly, and I figured I’d record my experience here for anyone else struggling with similar problems.

Manual Container Management

As I mentioned previously, I started out by running the default installer.sh script which worked most of the way, but some of the Docker containers it spawned failed to start. Additionally, when I tried to set up an account through that script after the main services were initialized, I got 400/500 server errors and it failed.

The default install script for the PDS (as of November 2024 anyway) sets up the following services launched as Docker containers managed by docker-compose:

The PDS itself
A Caddy reverse proxy
containrrr/watchtower which seems to facilitate zero-downtime upgrades

It has code to auto-install Docker and some other packages, pull images, and launch containers. The compose.yml file that it uses can be found here: https://github.com/bluesky-social/pds/blob/main/compose.yaml

For my server, I already had a NGINX reverse proxy set up which I use as the front door for all my services. The default setup that installer.sh uses launches the Caddy container on the host network and has it try to listen on port 80/443, so that conflicted with NGINX and caused Caddy to fail to start.

To work around these issues, I tore down and deleted all of the auto-launched containers and created a custom compose.yml file that skips the Caddy reverse proxy and watchtower containers entirely, only launching the PDS. It also sets up the PDS to expose itself on a different port than 3000 since I was already using that for a different service.

Here’s what my compose.yml file looks like:

version: '3.9'
services:
  pds:
    container_name: pds
    image: ghcr.io/bluesky-social/pds:0.4
    restart: unless-stopped
    volumes:
      - type: bind
        source: /opt/pds
        target: /pds
    ports:
      - '6010:3000'
    env_file:
      - /opt/pds/pds.env

One other thing to note is that by default, the install script creates the data directory on the host at /pds. This is where all of the state for your PDS lives (and is probably a good thing to set up automated backups for).

I moved it to /opt/pds to fit better with the rest of my server setup and updated the volume mount to match, but you can skip that if you prefer.

NOTE: If you do change the location of this directory, do not update the paths in your pds.env file accordingly. These paths are read from inside the container where it is mapped to /pds no matter where the directory is located on the host, so leave it as is.

I put the compose.yml file in a bsky directory and can launch it by running docker compose up -d.

NGINX Reverse Proxy

Once I had the manual compose.yml file set up, I had the PDS container up and healthy and exposed on port 6010. Now, I needed to configure my NGINX reverse proxy to expose that to the internet.

One caveat was that I had other things available on the domain I wanted to use (ameo.dev) and couldn’t just let the PDS take over the whole domain - which is the behavior that the install script expects.

I did some research and discovered that luckily, the PDS only needs to control a few paths - so I could leave all my other sites + services on the domain intact and just map a few paths through.

The paths that it needs are:

/xrpc/: This is the main path under which all of the APIs/RPCs that the PDS exposes are available
/.well-known/atproto-did: This is used to verify domain ownership in addition to DNS methods
- Note: You may not need this if you use DNS to verify your domain ownership. However, I was having issues with verifying my domain/handle and set up both methods.

Here’s the NGINX config I ended up with to expose these two paths and route them through to the PDS:

server {
    server_name ameo.dev www.ameo.dev;

    # ... rest of your server config ...

    location /xrpc {
        proxy_pass http://localhost:6010;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Host            $host;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    location /.well-known/atproto-did {
        proxy_pass http://localhost:6010/.well-known/atproto-did;
        proxy_set_header Host            $host;
    }

    # ...
}

It’s pretty straightforward, but there are a few pieces that are important.

For the well-known/atproto-did path, the Host header needs to be forwarded through in order for it to work. The handler for that path in the PDS looks at the Host header to figure out which DID to return, so if it’s not set the path will return an error/not found response.

I also added this line to my pds.env file to try to make the /.well-known/atproto-did route work:

PDS_SERVICE_HANDLE_DOMAINS=.ameo.dev

(replace with your domain of course)

I’m not sure if this was actually necessary, but if you are still having issues with that route even after forwarding the Host header correctly then maybe give it a try.

Additionally, the Upgrade and Connection headers need to be set in order for the PDS to function. These support websocket proxying - and you might need to set some other config in your reverse proxy in order to get websocket proxying to work.

I initially figured that it was a similar situation to Mastodon and the websocket functionality was optional and only used for things like realtime notifications and live feeds.

However, I learned that having a working websocket feed to your PDS is 100% necessary in order for it to work correctly and interface with the rest of the Bluesky network.

To make sure this was working, I used an online websocket tester and put this URL in (modify to your domain if you use it): wss://ameo.dev/xrpc/com.atproto.sync.subscribeRepos?cursor=0

If your proxy is set up correctly, it should report that the connection is successfully established when you test it.

Issues with Top-Level Domain Handles

Once I had everything up and running and proxied out to the public internet, I was able to set up an account using the pdsadmin command on the server - mostly following the official docs in the PDS repo’s guide.

I made it as far as logging in through the main bsky.app Web UI and saw that requests were successfully going to and getting answered by my PDS on my custom domain. However, there were issues with my account that quickly became apparent.

On my profile page, my handle was showing up as “Invalid Handle ⚠`. I had already performed the steps to set up my domain’s DNS to verify the domain, but it didn’t seem to be working. I tried changing my handle between my main domain and subdomain to no avail.

For me, the cause turned out to be that I didn’t have the websocket proxying working properly. As I mentioned in the previous section, a working websocket connections is necessary for a PDS to work. Once I set that up, I ran this command on my server to trigger the network to re-scrape my PDS:

pdsadmin request-crawl bsky.network

Once I did all that, my “Invalid Handle” issue went away after less than a minute and my handle was successfully set to ameo.dev - just what I wanted.

Other Debugging Resources

While trying to figure out this issue, I came across some resources that might be useful to you if you’re having similar issues verifying your handle or domain.

The first is an official debugging app to help check your domain verification and handle status:

https://bsky-debug.app/

The second is a semi-official Discord server with a community of people to help debug issues with self-hosted Bluesky PDSes:

https://discord.gg/NDATMNx2

I didn’t end up chatting there myself, but there seems to be a pretty active group who are eager to help out fellow PDS admins with their issues.

Top-Level Domain Handle Issues

One other thing to note is that it seems to currently be impossible to start off with a top-level domain as your handle for the initial account created on a self-hosted PDS.

As I understand it, it’s just a limitation of the current setup and not something inherent with the AT Protocol or Bluesky.

To work around this, I started off by using a subdomain as a handle (I used casey.ameo.dev) and then later switching it to my top-level domain (ameo.dev) via the settings in the main bsky.app web UI. Once I got my websocket proxying and DNS set up, it worked just fine.

SMTP Server

One last thing to note is that I needed to set up a custom SMTP server to make email verification work. To be honest, I’m not 100% sure if email verification is even required or not since this is a self-hosted PDS and I’m the only user. However, I did end up setting it up to make the message in the UI go away if nothing else.

I initially tried setting up a free account through Resend as suggested in the official PDS repo docs. However, I already have MX records set for my domain since I used it for my own email, so I didn’t want to mess with them in order to verify my domain through Resend.

For my specific case, I manage email for my domain through Google Workspace. Because of this, I was able to use Google’s SMTP relay to send emails and it worked fine.

I added these lines to my pds.env file to make that happen:

PDS_EMAIL_SMTP_URL=smtp://smtp-relay.gmail.com/
PDS_EMAIL_FROM_ADDRESS=casey@ameo.dev

Once that was set, my verification email was delivered successfully and I was able to verify it.

Conclusion

That’s all I have to note. It was a bit tricky but now that I’ve figured it all out, my PDS has been running smoothly for over a day and I’m interacting with Bluesky completely normally.

I hope this writeup helps save you some effort if you’re having these or similar issues yourself!

Feel free to tag or message me on Bluesky (@ameo.dev) if you have any questions as well :)

Fixing Artifacts Caused by Negative Color Sampling in WebGL

Sun, 10 Nov 2024 16:22:35 -0800

The Problem

I was working on a screen-space reflection shader. This involves reading pixels out of the main scene’s framebuffer/texture to be reflected.

I was seeing strange grainy-looking artifacts showing up in my reflections that I couldn’t trace to anything in my shader code. Here’s what they looked like:

One thing I noticed was that the artifacts seemed tied to specific positions in the texture being read, meaning that the artifacts were tied to individual texels in the texture being read from.

Another interesting thing I noticed was that not all of the reflections had these artifacts; it depended on their color.

The Cause

In my shader, I was returning sentinel values of -1 in my reflection-finding function to indicate that there was nothing to reflect. I then had a check in the main function like this to just use the existing fragment color in that case:

vec4 raymarchReflections(...) {
  // ...

  if (!hasReflection) {
    return vec4(-1.);
  }

  // ...
}

void main() {
  // ...

  vec4 reflectedColor = raymarchReflections(...);

  // if no reflections found, pass through existing color
  if (reflectedColor.r < 0.) {
    gl_FragColor = diffuse;
    return;
  }

  // ...
}

After a good deal of debugging, I discovered that changing the check to this made the artifacts go away completely:

if (reflectedColor.r == -1.) {

As far as I can tell, there is just inherent imprecision when reading values from certain textures in WebGL.

When reading some parts of the image with small values for some channel (red in my case), the sampled value is sometimes slightly negative. This also explains why the artifacts only showed up in some of the reflections. The blue one in the screenshot above had a color of #09f0f9 which has a small but non-zero red channel. Other reflections with larger red channel values didn’t suffer from the imprecision issue.

For my case, I was using WebGL textures with a type of HALF_FLOAT and an internal format of RGBA16F. It’s possible that this is the reason for the imprecision and that this issue might not happen if storing textures in some other format like unsigned integer. It might also be a quirk of my particular hardware, drivers, WebGL params, or one of the other hundreds of variables in play.

Anyway, the main thing I learned is that sampling values from textures isn’t guaranteed to be exact and to take more care when using special-case values like these in shaders.

A Small Change to Significantly Improve Triplanar Mapping

Sun, 01 Sep 2024 15:25:14 -0700

Triplanar Mapping is a method I make use of all the time in my 3D projects. Over time, I’ve experimented with tweaks and alterations to it with the goal of making it look better, run more performantly, and be useful in more situations.

The main change I present here is switching from using a linear mix of texture weights from each axis to a non-linear mix.

To explain what I mean by this, I’ll just show some code.

Here’s a basic triplanar mapping implementation:

vec4 triplanarTexture(sampler2D map, vec3 pos, vec3 normal) {
  vec3 weights = abs(normal);
  weights /= (weights.x + weights.y + weights.z);

  vec4 outColor = vec4(0.);
  outColor += texture2D(map, pos.yz) * weights.x;
  outColor += texture2D(map, pos.zx) * weights.y;
  outColor += texture2D(map, pos.xy) * weights.z;
  return outColor;
}

And here’s a modified version that uses a non-linear mix of the same texture samples:

vec4 triplanarTexture(sampler2D map, vec3 pos, vec3 normal) {
  vec3 weights = abs(normal);
  // non-linear scaling of weights
  weights = pow(weights, 8);
  weights /= (weights.x + weights.y + weights.z);

  vec4 outColor = vec4(0.);
  outColor += texture2D(map, pos.yz) * weights.x;
  outColor += texture2D(map, pos.zx) * weights.y;
  outColor += texture2D(map, pos.xy) * weights.z;
  return outColor;
}

It turns out that this one-line change has a lot of impact - especially for certain types of models/scenes.

First of all, it makes the transition regions smaller and reduces the amount of visible overlap between different planes. This is most obvious on smooth meshes like this sphere:

The left side shows the result of triplanar mapping with default linear weights, and the right shows the result when using the pow(weights, 8) change above.

As you can see, there is no more visible layering of the texture. Instead, there are some small areas where the texture faces smoothly between two different planes which looks much better.

Performance Improvements

In addition to the visual improvements, this change also opens up the possibility for some significant performance improvements.

Applying this non-linear transformation to weights serves to drive small weights to zero and large weights to one. This makes it possible to ignore small weights entirely in some cases since they go from making a small impact to a negligible/unnoticeable impact on the output.

So to optimize the shader, it’s possible to skip some texture lookups entirely if the weight is small enough:

vec4 triplanarTexture(sampler2D map, vec3 pos, vec3 normal) {
  vec3 weights = abs(normal);
  // non-linear scaling of weights
  weights = pow(weights, 8);
  weights /= (weights.x + weights.y + weights.z);

  vec4 outColor = vec4(0.);
  if (weights.x > 0.01) {
    outColor += texture2D(map, pos.yz) * weights.x;
  }
  if (weights.y > 0.01) {
    outColor += texture2D(map, pos.zx) * weights.y;
  }
  if (weights.z > 0.01) {
    outColor += texture2D(map, pos.xy) * weights.z;
  }
  return outColor;
}

This optimization would be fine to include without the non-linear weights transformation, but it would rarely activate. In the modified version, there is much more area where individual weights end up being very close to zero.

Also, the higher the power chosen to raise the weights to, the more aggressive the sharpening will be and the more effective the optimization will become. I usually choose as high of a power as I can that avoids any visible discontinuities in the texture.

And that’s it! I highly recommend giving this a try in your own projects when using triplanar mapping. It makes an already extremely effective technique even better.

A Basic Graphviz Dark Theme Config

Thu, 20 Jun 2024 16:14:03 -0700

I regularly use Graphviz to generate… graph visualizations. Since my website and blog are all dark-themed, I usually want to set it up so that the generated SVG is dark-themed as well.

Here’s the basic config I use to make graphviz produce outputs with dark backgrounds and light content:

digraph G {
  bgcolor="#181818";

  node [
    fontcolor = "#e6e6e6",
    style = filled,
    color = "#e6e6e6",
    fillcolor = "#333333"
  ]

  edge [
    color = "#e6e6e6",
    fontcolor = "#e6e6e6"
  ]

  # The rest of your code goes here...
  A -> B;
  B -> C;
  C -> A;
}

That results in an output that looks like this:

You can of course tweak the hex color to your liking. There may be some fancy elements you can add that this doesn’t cover, but it should be all you need for most cases.

Fixing `regl` Error Invalid Dynamic Attribute

Wed, 01 May 2024 01:43:06 -0700

Recently when working with the regl WebGL wrapper library, I encountered a weird error that I didn’t understand the cause of:

(regl) invalid dynamic attribute "position" in command

I checked my code, and I was indeed passing a position attribute and the data array I was using to construct the buffer looked correct. The error also happened intermittently, happening after my application hot-reloaded.

The Cause

It turns out that I had accidentally created two regl instances and was trying to use buffers allocated on one with commands created on a different one.

When my app hot-reloaded, it was incorrectly creating a new instance for the command while re-using the old buffers.

The Fix

I corrected the issue that was causing duplicate regl instances from getting created, and the error went away and my app worked again!

It was a bit tricky to figure out due to the rather vague error message, though.

Fixing Sveltekit Failed to Load URL SERVER/internal.js

Tue, 30 Apr 2024 12:36:56 -0800

I was working on a new SvelteKit project with Svelte 5 recently, and I kept getting an error that prevented anything from loading in the browser:

[vite] Pre-transform error: Failed to load url __SERVER__/internal.js (resolved id: /Users/casey/osu-embeddings/frontend/.svelte-kit/generated/server/internal.js) in /Users/casey/osu-embeddings/frontend/node_modules/@sveltejs/kit/src/runtime/server/index.js. Does the file exist?
[vite] Error when evaluating SSR module /node_modules/@sveltejs/kit/src/runtime/server/index.js: failed to import "__SERVER__/internal.js"
|- Error: Cannot find module '__SERVER__/internal.js' imported from '/Users/casey/osu-embeddings/frontend/node_modules/@sveltejs/kit/src/runtime/server/index.js'
    at nodeImport (file:///Users/casey/osu-embeddings/frontend/node_modules/vite/dist/node/chunks/dep-DkOS1hkm.js:55067:25)
    at ssrImport (file:///Users/casey/osu-embeddings/frontend/node_modules/vite/dist/node/chunks/dep-DkOS1hkm.js:54976:30)
    at eval (/Users/casey/osu-embeddings/frontend/node_modules/@sveltejs/kit/src/runtime/server/index.js:5:37)
    at async instantiateModule (file:///Users/casey/osu-embeddings/frontend/node_modules/vite/dist/node/chunks/dep-DkOS1hkm.js:55036:9)

The Cause

It turns out that I had a broken symlink in my static/ directory. I had cloned the project down to a new computer, and the file only existed on the old one.

The Fix

I created a file at the location pointed to by the broken symlink which made it valid. Then, when I re-started the server, things worked.

It’s possible that this was a fluke and some other thing happened which caused this error, but if you’re seeing it and have symlinks in your project, it might be worth looking into that.

Fixing Svelte VS Code `e.children.findLastIndex` Is Not a Function

Mon, 15 Apr 2024 16:21:08 -0700

The Problem

I recently created a new SvelteKit project using the Svelte 5 preview and kept getting an error at the beginning of every Svelte component:

The error was e.children.findLastIndex is not a function (svelte). It was showing up even on the basic SvelteKit skeleton starter project as soon as I added a <style></style> block.

The Fix

I looked around online, and I didn’t see anyone having this problem. This led me to believe that it was likely an issue with my local environment or configuration.

I looked in my VS Code configuration and found this line:

{
  "svelte.language-server.runtime": "/Users/casey/.nvm/versions/node/v16.13.0/bin/node"
}

I didn’t remember setting that or why I did it. However, once I removed it and reloaded my VS Code, the error went away!

So if you’re having this issue yourself:

Check for a config option similar to this in your VS Code settings.json and delete it if you find one
Try updating your Node.JS to a new version. I’m using version 21.1.0 myself.

Investigating Bogus Plausible Analytics Traffic

Wed, 20 Mar 2024 10:42:24 -0700

I run self-hosted Plausible analytics to keep track of how many people are visiting my various websites and web apps. I’m largely very happy with it - it gives me all of the info I need with only a miniscule JS payload, no cookies or other invasive tracking, and full control over the data.

However, recently I’ve been running into an issue with fake/bogus traffic getting submitted for my personal homepage. Usually, I’m more than happy to see a bump in traffic to one of my sites, but something was off about it this time.

First, all of this traffic seemed to be limited to the homepage of my site. Traffic spikes usually correspond to some blog post I’ve written getting posted to Reddit or similar, so having everything limited to just the homepage was quite strange.

Next, all of the traffic had no set referrer/source. This is strange for the same reason: usually people are coming to my sites from somewhere else rather than just typing in the URL manually.

I’ve seen some spam analytics events in the past with their referrer header set to spam domains. The intention seemed to be to advertise to me in my analytics dashboard or something along those lines. That’s not the case here, though.

When filtering my site’s traffic to only include events that match that criteria, it’s clear to see when this started happening in early March:

This was a very strange pattern that I hadn’t seen before, so I dug in deeper.

Since I host my homepage myself, I have direct access to the raw server logs for the NGINX webserver that serves it. Plausible submits all of its events to a single URL (/api/event), so it was pretty easy to filter down the logs to find the Plausible requests.

Since there were so many of these events coming in to the point where it was a large portion of my site’s total traffic, it was pretty easy to match the requests on the server to the traffic displayed in the Plausible UI. I identified several of these no-referrer homepage requests and quickly identified some other interesting patterns with those requests:

They were all coming from different IP addresses.

This does make sense since Plausible suggested a wide spread of different countries that these requests were originating from. However, this is pretty surprising if this is automated traffic; it would require the perpetrator to have control over dozens-hundreds of unique IPs.

They all had similar user agents.

Here’s a random sampling of some of the user agents that I saw from these requests:

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.1392.1086 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.1010.1725 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.5871.1821 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.4338.1594 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2734.1416 Mobile Safari/537.36

There are some clear similarities between them, but also some randomized parts as well. This makes it clear that whatever is sending these events is at the minimum more sophisticated than some simple script.

I searched my server logs for the string Mobile Safari/537.36. There were a lot of results there other than the plausible requests. Although there were some that looked like legitimate user-originating traffic, it looked like the majority of these requests were coming from bots like Googlebot or PetalBot.

Although Mobile Safari/537.36 seems to be something that can come up in real user traffic, it also seems to be a very common thing to include in bot/scraper user agents. This lends even more credence to the idea that these events are fake and don’t represent real user traffic.

All of the IPs I checked originate from residential/non-datacenter sources

I ran a collection of the IP addresses sending these requests through the Maxmind GeoIP service, and here are the results:

As far as I can tell, these look like legitimate residential IPs that could belong to real users. I would have guessed that they would have belonged to public clouds or datacenters given the apparent automated source of the traffic, so this was pretty surprising to me.

I know that there are available residential IP proxies that people use for scraping and similar use cases. It’s also possible that this traffic is coming from devices compromised by a virus or malicious browser extension. Both of these seem pretty extreme and unlikely to me, though, for something such at this.

I did some basic googling of some of these IPs looking for entries in abuse blocklists or similar, but nothing came up. They looked entirely like real user IPs in every way I could see.

All of the IPs I checked only make requests to the Plausible event submission URL (/api/event) without making any other requests to my webserver

This was the smoking gun for me that indicated that this traffic was almost certainly not real. When I searched through the server logs, there would always be a single request from that IP. They didn’t even download the plausible.js script that is used to kick off these requests!

I checked multiple IP addresses and this was the case for all of them. I double-checked the logs of the CDN I use for some of the static assets on my site (it’s not used for the index.html nor the plausible.js script though) and there was no trace of those IPs there either.

It’s probably technically possible that these requests came from users that had my entire homepage cached from like a week ago, but that’s extremely unlikely and doesn’t seem remotely realistic.

The most likely explanation for these requests I can think of is intentionally submitted artificial traffic from a relatively sophisticated source.

Conclusion

So to summarize:

I’m seeing a lot of fake traffic getting submitted to my Plausible analytics for one of my websites
All of the requests are to the homepage, bounce immediately, and have no referrer/source listed
The requests all come from unique residential-looking IPs
The requests all have similar but randomized user agents
There’s no other requests or traffic to other pages or content from any of these IPs whatsoever; they all just hit the Plausible event submission endpoint a single time each

Very strange indeed! This leaves the one big question remaining: Why?

I have no idea! This is my personal website that I mainly use as a portfolio and blog. I have no ads on the site at all, don’t sell anything on it, and have no paywalls or login system of any kind. It’s all static content, and the site itself is open source.

If someone out there is trying to inflate my ego by making it look like hundreds of people are coming every day to stare at my homepage before leaving, I appreciate the thought! But now that I’m almost positive it’s fake, it’s much more annoying than anything else; I’d prefer to have an accurate as possible view of real traffic on my sites.

Given how little of a reason for this traffic I can see, I still feel like there’s probably some explanation for this other than someone setting it up intentionally to give my website fake hits. Someone hotlinking or duping my site maybe, or perhaps some web cache or internet archive-like site acting in a weird way? Some kind of ad fraud bot that got lost or mis-targeted?

In any case, I’m thoroughly intrigued and would love to know the truth. In my experience, these kinds of mysterious web traffic events tend to clear up on their own after a while, so I wouldn’t be surprised if this fake traffic just vanishes in a few days or weeks.

If anyone has seen something similar or has any ideas, I’d be very curious to hear!

Trying Out `sea-orm`

Mon, 22 Jan 2024 13:56:15 -0800

For a new project at my dayjob, I’ve had the opportunity to try out sea-orm for the database layer. In the past, I’ve tried out other Rust SQL solutions including diesel and sqlx, so I have some context to compare this one to.

At a high level, sea-orm provides a fully-featured solution for managing your database setup in Rust. It provides a framework and CLI for setting up and maintaining migrations, code-gen’ing entities and relations, and writing + running queries. Like most other Rust DB options, it is fully typed and integration into Rust’s type system.

sea-orm is built on top of several other sea-* crates that make up the ecosystem. They include crates like sea-query, sea-schema, sea-orm-cli, and some internal crates like sea-orm-migration.

Note: At the time of writing this (Jan. 2024), I don’t have extensive experience with or knowledge about sea-orm; this is more of a “first impressions” writeup than a detailed review. Also, sea-orm and the rest of the sea-query ecosystem is under active development, so things may change significantly over time.

Setup

For this project, we started with a fresh Postgres database with nothing in it. This is the best case for frameworks like these since they like to have full ownership + control over everything.

I opted to go for the full setup with migrations, code-generated entities, and managed DB connections.

One thing I noticed right away is that sea-orm is pretty prescriptive/opinionated about the crate layout and setup of the different components. It expects you to create separate crates for both the migrations and entities and join them together into a workspace.

For one thing, it’s set up to pull in async-std by default for the migration crate. We (and pretty much everything else in the modern Rust ecosystem) use tokio which isn’t compatible with async-std for some things. I was able to manually change it to use tokio without seeming to breaking anything, though.

Having to split up the database functionality into multiple crates is a bit awkward for our setup. We have a very large workspace with several dozen different crates at the top level. The project we’re adding this database to has a crate of its own, so that means adding two additional crates to manage the database stuff and then importing them.

Compare this to something like diesel. It has a single schema.rs file which is generated based on the schema detected from the database. It contains a bunch of macros which define the various tables available and DSL for constructing queries. Since this all uses procedural macros, that definitely contributes to much of Diesel’s reputation for slow compile times, though.

It might be possible to set sea-orm up to avoid these extra crates, but the docs I read definitely don’t have any info on that. sea-orm-cli seems to require that at least the migrations are in their own crate, since it compiles it as a binary and runs it to run migrations.

Migrations

The reason for this is that migrations themselves are defined using sea-orm’s builder syntax rather than as raw SQL files like other libraries. So the migrations you write look something like this:

 manager
  .create_table(
    Table::create()
      .table(Session::Table)
      .if_not_exists()
      .col(ColumnDef::new(Session::Id).uuid().not_null().primary_key())
      .col(ColumnDef::new(Session::UserId).string())
      .col(ColumnDef::new(Session::UserEmail).string())
      .col(
        ColumnDef::new(Session::CreationTimestamp)
          .timestamp()
          .not_null(),
      )
      .to_owned(),
  )
  .await?;

The main benefit of this, according to the sea-orm docs, is that it allows migrations to be independent of the backend database used - so you can theoretically re-use your migrations across Postgres, MySQL, Sqlite, and any other backends supported by sea-query.

Personally, I’m not a big fan of this. I find the builder syntax very verbose and hard to work with. sea-orm does have support for running migration queries as raw SQL, but you have to write them as Rust strings in the code anyway, so you don’t get syntax highlighting in your editor, have to deal with annoying escaping issues, etc.

In addition, I’m not sure just how “backend-independent” these migrations are. I tried to set up an index by calling the .index() method on a TableCreateStatement. This caused my migrations to start failing with a very unhelpful error:

syntax error at or near "("

I eventually figured out that there’s a --verbose flag on the migration binary which prints out the raw queries being run, which helped.

It turns out that .index() function is actually only supported for the MySQL backend. For other databases, you have to create indexes using SchemaManager::create_index().

It was annoying having to spend the time to figure this all out when I could have done it in 15 seconds if I was using raw SQL.

To sea-orm’s credit, though, pretty much every feature you could want is available through the Rust interface itself. Even exotic types, postgres-specific built-in functions, typed JSON/array columns and operators, and more are all there.

Entities

As I mentioned before, sea-orm handles auto-generating models/entities based on the schema detected from the database.

After you run your migrations, you run sea-orm-cli generate entity to codegen a bunch of rust files that are dumped into your entities crate.

That codegens a bunch of Rust that looks something like this:

//! `SeaORM` Entity. Generated by sea-orm-codegen 0.12.10

use sea_orm::entity::prelude::*;

#[derive(Clone, Debug, PartialEq, DeriveEntityModel, Eq)]
#[sea_orm(table_name = "session")]
pub struct Model {
    #[sea_orm(primary_key, auto_increment = false)]
    pub id: Uuid,
    pub user_id: Option<String>,
    pub user_email: Option<String>,
    pub org_id: Uuid,
    pub creation_timestamp: DateTime,
    pub project_id: Uuid,
}

#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {
    #[sea_orm(has_many = "super::active_session::Entity")]
    ActiveSession,
    #[sea_orm(has_many = "super::session_step::Entity")]
    SessionStep,
}

impl Related<super::active_session::Entity> for Entity {
    fn to() -> RelationDef {
        Relation::ActiveSession.def()
    }
}

impl Related<super::session_step::Entity> for Entity {
    fn to() -> RelationDef {
        Relation::SessionStep.def()
    }
}

impl ActiveModelBehavior for ActiveModel {}

This includes derives for a bunch of traits for getting/updating/inserting/deleting/joining these entities via sea-orm.

Overall, the entity system seems… OK. You can do all the usual ORM stuff like finding related entities, filtering with .eq(), and inserting new stuff. The APIs mostly seem pretty clean and make sense.

There’s some of the familiar confusion due to the inclusion of Active variants of the models and entities. This system is designed to let you leave some portion of the model’s fields unchanged/default when updating or inserting entities by wrapping them all in an ActiveValue enum. The API makes sense conceptually and does indeed work, but I find that it makes for a verbose and clunky developer experience.

My main grips with this are mostly rooted in the fact that I’m not really a big fan of heavy entity/model-based ORMs in general. I usually find it easier to write my queries by hand or using the more minimal query builder patterns from diesel or similar.

For most of the database stuff I end up building, it feels like the abstractions leak very quickly out of what the ORM is expecting. For some apps with simpler or more constrained database hierarchies, I feel like this kind of system would be a better fit. It’s also very possible that I’m just doing something wrong or overlooking something.

In any case, there’s not a ton here I have to complain about that could be blamed on sea-orm specifically.

`sea-*` Crate Organization

One other pain point I kept running into came from the fact that there are so many different sea-* crates across which the functionality I need is split across.

It was quite difficult to figure out where to import things from. There are even some cases where there are structs of the same name exist in different crates making it even harder to figure things out.

I also ran into problems like this when trying to set up an enum type + field for one of my tables. My goal was to create a single Postgres enum type and use it for a field on one of my tables.

In raw SQL, it would look something like this:

CREATE TYPE customer_status AS ENUM ('inactive', 'pending', 'active');

CREATE TABLE customers (
  id uuid NOT NULL,
  cur_status customer_status
);

sea-orm certainly has support for generating custom types including enum types, and they have two docs pages about it: Generating Entities -> Enumerations and Schema Statement -> Create Enum.

However, there’s an issue. Both of those docs pages start from entities. They expect you to be manually defining these entities and then reference the entity in the code that performs the CREATE TABLE. This poses a chicken and egg problem for actually creating and referencing custom enum types within migrations.

It’s extremely unclear to me how to actually set up an enum type and reference it in a table I’m creating from within a migration. I really feel like it’s possible, but there’s so much complexity involved.

There are several different derivable traits on things. The migration examples show you creating a different enum for each table and deriving DeriveIden on it so you can refer to the table and its fields in queries. Then, for the codegen’d entities, there’s DeriveEntityModel which ‘is the ‘almighty’ macro which automatically generates Entity, Column, and PrimaryKey’. Then there’s an ActiveEnum which is the Rust type which corresponds to a custom enum type living in the DB. However, to define a table, you reference fields on that enum which are Idens.

I tried a bunch of things, but yeah I couldn’t figure it out. Diesel has a third-party solution for this which I’m not particularly a fan of, but at least it works and is comparatively easy to set up.

Programmatic Query Generation

Although not part of sea-orm itself, one thing I have to mention is how good sea-query is at doing programmatic query manipulation.

One project I built in the past year for my job at Osmos is an interactive query builder web interface. It lets users design complicated with joins, filters, and complex aggregates. It does all of this in a sandboxed way, allowing us to make sure they can only query resources we give them access to.

Building this would have been extremely difficult if it weren’t for the powerful tools provided by sea-query and sea-schema. I strongly believe these libraries are the best tools available in the Rust ecosystem to perform advanced manipulation or generation of SQL queries. The amount of situations where something “just worked” after I found the right function was very high.

sea-schema is also a lovely gem of a crate. It’s what sea-orm uses under the hood to do its codegen. sea-schema can connect to a live database and automatically discover the full schema for a given database. This includes all the details you would every need like data types, foreign keys, constraints, and all that kind of stuff. There are a few bugs and edge cases we’ve run into which honestly isn’t surprising given how big that problem space is, but for the majority of cases it again “just works” and is very easy to work with code-wise.

Conclusion

Overall, my impression of sea-orm is pretty lukewarm. The developer experience of setting up the crates, writing migrations, and making queries isn’t my favorite compared to other alternatives. It’s quite difficult for me to find the right way to do things due to a large number of magic traits required to do everything.

Again, a large part of this sentiment is personal preference. I think that a developer coming from the Rails/ActiveRecord world might enjoy an entity-based ORM like sea-orm much more than me. One of my co-workers advocated for us to try sea-orm in the first place which is why I ended up using it, so you might enjoy it as well.

I just want to say one more time that the underlying libraries like sea-query/sea-schema and the broader sea-* ecosystem really are excellent, though. Even if you don’t go for the whole sea-orm experience, they’re extremely useful on their own.

Handling Gamma Correction for Three.JS pmndrs `postprocessing` Effects

Mon, 22 Jan 2024 11:31:08 -0800

I recently received a bug report on a library I built - three-good-godrays - which implements screen-space raymarched godrays for Three.JS as a pass for the pmndrs postprocessing library. One of the problems pointed out was that colors seemed washed out/desaturated when my pass was used, even when the pass wasn’t rendering any godrays.

Here’s how things look by default without the effect (and are supposed to look with it on):

But here’s how they looked with the godrays pass enabled:

The bug report is right - there’s definitely something wrong.

The Cause

The reason this is happening is due to differences in color space handling. According to the docs on the pmndrs postprocessing library, the internal buffers used by postprocessing passes are expected to store data in sRGB format (relevant docs). To handle this, popular effects such as n8ao perform linear -> sRGB conversion by default: https://github.com/N8python/n8ao?tab=readme-ov-file#usage

To implement this, n8ao calls LinearTosRGB() on the pixel data output if gammaCorrect is enabled (which is the default).

The Fix

To fix three-good-godrays, I simply added the exact same solution that n8ao used. I added the same gammaCorrection flag to the pass params, set it enabled by default, and added the same call to LinearTosRGB() to the shader.

However, after adding this change, I noticed that my demos were experiencing some severe artifacts:

After digging into this, I determined that the artifacts went away if I disabled the SMAAEffect and its EffectPass that I had added to the postprocessing pipeline after the GodraysPass.

It turns out that output encoding is performed by default by EffectPass in some circumstances. This was causing my colors to get double-converted to sRGB which produced those artifacts.

To work around that, I … just set gammaCorrection: false in the godrays pass params. This had the result of making things look the same way they did before for my demos. Another method of achieving this is to set outputEncoding = false on the SMAAEffect’s EffectPass which disables the double-encoding.

One thing to note is that I have also set the frameBufferType to HalfFloatType on the framebuffers used by the postprocessing EffectComposer. The reason I do this is to avoid color banding that results from the loss of precision by the default 8-bit color format.

new EffectComposer(renderer, { frameBufferType: THREE.HalfFloatType });

Other Notes

The color space/encoding handling in the Three.JS ecosystem is very complex and confusing to me at the moment. Three.JS recently made some internal changes to Color Management which might be interfering/conflicting with pmndrs postprocessing in some ways.

If you’re still running into color issues in your Three.JS projects, some other things you can try are:

Toggling THREE.ColorManagement.enabled = true;. I set this to true for my demos, and I believe this is the “right way” to do things in modern Three.JS.
Changing the outputColorSpace of your WebGLRenderer. This is set to THREE.SRGBColorSpace by default, and I think it’s suggested to not change that.

I’m honestly not really sure about the details of these things myself. For my projects, I generally end up trying various combinations of these settings until things look right :p

In any case, I hope this helps give you some info on possible solutions if you’re running into these problems yourself.

Changing Linux Select to Paste Menu `fcitx` Keyboard Shortcut

Tue, 16 Jan 2024 21:09:46 -0800

For a long time - at least a couple of years - I’ve been cursed with an issue on my KDE Plasma Linux desktop where my PgUp key doesn’t work. Instead of scrolling up in my terminal or editor, it pops open a menu with the title “Select to paste” and a listing of my most recent clipboard entries:

I tried briefly a couple of times to fix this and get my page up to work again but to no avail. Today, I finally figured it out.

The Cause

I figured out that the menu getting opened is from a project called fcitx which is some utility for input handling on Linux. I found it by doing a code search on Github for the title of the menu - the only identifying information I could see.

The problem remained that I couldn’t find the shortcut anywhere in my system settings, though, so I had no idea how to disable it.

The Fix

After much digging, I finally found a part of the system settings which controlled Fcitx. Then, hidden two levels deep in sub-menus, I found the “Trigger Key for Clipboard History List” setting which was set to PgUp:

When I hit the “Defaults” button, it restored the shortcut to Ctrl + ; and suddenly I remembered how it got broken in the first place.

I wanted to use the control + semicolon shortcut for VS Code, so I switched the existing shortcut in my settings to something random (PgUp, apparently) and promptly forgot about it. Then, KDE Plasma must have moved that setting in some system update (or maybe I just missed it all these times).

Anyway, I changed the trigger key shortcut to ctrl + alt + shift + { and the problem is solved for good this time :)

PIXI.JS Optimizations

Tue, 02 Jan 2024 10:19:53 -0600

I was recently working on speeding up a MIDI editor UI written in PIXI.JS which is part of my web synth project. The UI is fairly simple itself, but it needs to be efficient in order to render potentially thousands+ notes on the screen at once.

Here’s what the MIDI editor UI looks like:

As I mentioned, I built this whole UI with Pixi.JS (except for the toolbar which is vanilla HTML/CSS/React). For the most part, it was working well. It supports zooming, scrolling/panning, and adding/removing/moving/resizing notes. All of the interactivity was handled by PIXI.JS through its events/interaction system.

I was looking to add a new feature to the MIDI editor for adding custom labels and annotations to some notes to support use cases where the MIDI editor controls a sampler or something similar where only some notes were active. However, when I went to start implementing it, I noticed that the MIDI editor’s performance was quite bad. It was using >80% of my GPU on my M1 Macbook Max to just render the base MIDI editor UI and a single note.

For an almost completely static UI, this was absurd. I felt strongly that there would be some low-hanging fruit to speed it up, so I went looking.

I tried several things to get this UI to render faster. Here’s a list of some of the things I tried to speed it up.

Avoiding Mask Usage

For the piano keyboard on the left, I rendered the whole thing inside of a PIXI.Container and then used a simple rectangular mask to cut off the top part so that it wouldn’t render underneath the cursor gutter bar.

I used code like this to set that up:

this.container.mask = new PIXI.Graphics()
  .beginFill(0xffffff)
  .drawRect(conf.PIANO_KEYBOARD_WIDTH, 10, this.width - conf.PIANO_KEYBOARD_WIDTH, this.height)
  .endFill();

According to the Pixi.JS docs, this kind of mask is the cheapest and should be applied using WebGL’s scissor testing. However, I found that in practice it slowed down my rendering significantly. It seemed to be going into some kind of rendering slow path that was slowing down the whole application.

I opted to just get rid of the masking entirely and instead extend the cursor gutter to the left and force it to render on top of the piano keyboard using zIndex and app.stage.sortableChildren = true. This achieved the same effect but made it easier to PIXI to render.

Using Spector.JS for Profiling + Debugging

To try to figure out where the application was spending its time rendering, I used a browser extension called Spector.JS. It’s a WebGL profiler/debugger that I’ve used in the past to debug and profile Three.JS applications.

Since PIXI.JS uses WebGL under the hood to render, Spector.JS can be used to analyze all the render passes. I loaded it up, selected the main canvas, and collected a profile. Here’s what I saw:

Overall, it was taking 238 commands to render the whole UI. This isn’t an enormous number (some complex 3D scenes can use thousands or more commands to render) but it’s a lot more than should be necessary for such a simple UI.

One thing that was clear was that each of the lines was being rendered one by one. This surprised me; I was using a PIXI.Graphics with cacheAsBitmap = true to render those lines and ticks, and I was using the same instance for each line. I expected that this would activate PIXI’s optimizations and cause those to all get rendered together as sprites, but that evidently wasn’t the case.

If there were multiple notes in one line, they were all getting rendered together in one pass, though, even when the little handles on either side for resizing them were included.

Upgrading PIXI.JS

I also took the opportunity to bump PIXI.JS to the latest version (7.3 at the time of writing this). Sometimes you get lucky and get performance for free when doing that.

Unfortunately, this was not one of those cases. Performance seemed pretty much the same after upgrading. Still a good thing to do, though, and I’ve been meaning to get to it for a while now.

Pre-Rendering to Sprites

My main goal after seeing that info from Spector.JS was to try to get the note lines rendering more efficiently. I knew that PIXI.JS had put extensive optimization into sprites, so I figured using real sprites rather than the cached graphics was a good idea to try.

I switched my note lines + tick markings to render into a PIXI.RenderTexture explicitly ahead of time, and then created sprites out of that shared texture:

const g = new PIXI.Graphics();

// ... rendering line and ticks based on current view + pan ...

const renderTexture = PIXI.RenderTexture.create({
  width: this.app.width,
  height: conf.LINE_HEIGHT,
});
this.app.app.renderer.render(g, { renderTexture });

MarkersCache.set(markersCacheKey, renderTexture);
return new PIXI.Sprite(renderTexture);

Whenever the zoom level of horizontal scroll of the UI changed, I’d re-construct the texture for the markers and re-create the sprites. After a bit of tweaking, I got it working as it was before, and performance was greatly improved!

Pre-Rendering Containers vs. Children

The main bottleneck remaining that I could see was the piano keyboard. Each of the keys consisted of a colored PIXI.Graphics to render the key background + border as well as a PIXI.Text for the label.

For each of the keys, I rendered the background and the label separately and set .cacheAsBitmap = true for each of them. Then, I set them into containers, added those containers to the base piano keys container, and set them up to render in the correct order.

As it turns out, this was a sub-optimal pattern for Pixi to work with. Pixi had to create separate textures for all of the dozens of text labels and render each of the keys separately.

The piano keyboard is mostly static; the only thing that changes is the Y offset with the current vertical scroll, and some pink transparent indicators are rendered on top of it when notes are actively playing.

Taking advantage of this, I pre-rendered the entire piano keyboard into a bitmap by setting the parent container .cacheAsBitmap = true. This means that PIXI was able to save the whole thing as a single big texture and then render it all at once.

This unsurprisingly sped things up significantly. I had to add a little bit of extra handling to the active note indicators, including the label in them again since they were rendering between the labels layer and keys layer previously, but it wasn’t that big of a task.

Results

After all that optimization, I ran some more profiling to see how effect it had:

That’s what I’m talking about! The GPU is basically idle - as it should be for this UI.

I also ran another Spector.JS profiling run to see how the render pass count had changed:

The whole thing renders in a single render pass now!!

I was quite shocked to see that honestly. PIXI.JS was finally able to show off its impressive capabilities here. I’d be interested check out their code to see how it manages to do that at some point.

So yeah - a very successful optimization journey indeed. PIXI.JS needs some things to get set up in the right way in order for it to really shine, but it excels when they are.

Fixing Svelte Seo `Cannot find module './transformers/"application/ld+json">${'`

Tue, 28 Nov 2023 23:25:56 -0800

The Problem

After upgrading svelte-seo to the latest version (1.5.4 at the time of writing this), I encountered this error when trying to run my SvelteKit dev server:

Error while preprocessing /home/casey/dream/node_modules/svelte-seo/index.svelte - Cannot find module './transformers/"application/ld+json">${'
Require stack:
- /home/casey/dream/node_modules/svelte-preprocess/dist/autoProcess.js
- /home/casey/dream/node_modules/svelte-preprocess/dist/index.js
Error while preprocessing /home/casey/dream/node_modules/svelte-seo/index.svelte - Cannot find module './transformers/"application/ld+json">${'
Require stack:
- /home/casey/dream/node_modules/svelte-preprocess/dist/autoProcess.js
- /home/casey/dream/node_modules/svelte-preprocess/dist/index.js

The Fix

The issue turned out to be that my svelte version was too old.

I upgraded svelte to the latest version (svelte ^3.50.1 → ^4.2.7), re-started the dev server, and the issue went away.

Building a Realistic Raindrop-Covered Window Pane Material in Three.JS

Sun, 12 Nov 2023 15:50:16 -0800

Recently, I’ve been working on a rainy scene in Three.JS. One of the most important parts of this scene is a greenhouse with big glass windows.

I decided to take a stab at making some realistic raindrop-covered window panes for them using Three.JS’s built-in transmission shader. The result turned out pretty well if I do say so myself, so I thought I’d write up my process for building it from start to finish.

Here’s how the finished material looks when included in my scene:

As you can see, the window looks like it has lots of raindrops on its outer surface. Notice how buildings in the background get distorted and twisted in areas where the rain drops are heaviest.

In addition, the whole background is quite blurred and out of focus - as if the window was quite thick or covered with condensation.

Finally, there are lots of small black flecks on the window surface, making it appear dirty or old. This helps make the window feel more tangible rather than having it be completely clear and clean.

I’ll break down how I built each of these components and how they all work together to produce the final Three.JS material.

Transmission via `MeshPhysicalMaterial`

Transmission is the key to making this material look good. If you’re not familiar with it, transmission is a property available on MeshPhysicalMaterial that functions kind of like a more advanced and powerful version of transparency/opacity. It simulates different properties of light and exposes several controls for altering the way light passes through objects.

For my needs with this window, there are a few properties which are key for getting the effect I wanted.

Transmission

The transmission parameter controls the amount of transmission. It does pretty much the same thing as opacity, but for transmissive materials rather than transparent ones.

For my material, I just set transmission to 1. This means that the whole material is 100% transmissive/transparent. That value makes sense for a window.

Setting it to a value lower than 1 will cause the material’s base color to be mixed in with the colors transmitted through from the background. Here’s how a value of 0.7 looks for my window material:

The window’s base color is white, so it looks milky-white as its base color is mixed in with the transmitted background.

IOR

ior stands for index of refraction. This is a number which represents how much the material bends light. The default value is 1.5, and setting it higher will add more bending.

ior was a bit harder to figure out. For my window, I ended up with a value of 1.6 - slightly above the default of 1.5.

ior is very closely linked with roughness; changing one will cause pretty drastic differences in the other’s functionality. Because of this, I found it best to adjust them together.

Overall, I found that higher IOR values tend to make the blur more pronounced and as the value approaches its minimum of 1, the raindrops almost completely disappear.

Roughness / Roughness Map

Now we get into the meat of the implementation.

roughness is usually used to control how reflective a material is, but it has a special purpose on materials with transmission. For transmissive materials, roughness controls how much blur the material applies to the background (higher = more blur).

I chose a base roughness of 0.64 for my window. This provides a good bit of blur without totally obscuring the background.

However, the real important part is the roughnessMap. This allows the blur factor of the material to be varied across its surface. Here’s the texture I’m using for the roughness map of the window material:

A bit of a surprising choice for use on a window!

This is actually a texture I generated myself with stable diffusion for use in a metallic gold material I used in a different scene. I converted it into a full-fledged PBR material using a process I detailed in a different post.

One important property of this texture is that it’s seamless - meaning it repeats in both the X and Y axis without any gaps or discontinuities. This is important for my material since the texture is pretty small (1024 x 1024) and so it needs to repeat multiple times across the large surface of all the windows.

Anyway, although it might seem surprising at first, this material actually works very well for creating raindrops.

When Three.JS interprets textures as roughness maps, it only looks at the green channel. So, that texture will actually be the same as this grayscale image:

Notice the white streaks across the texture. Those come from the specular highlights (shiny parts where lots of light is reflecting) from the original texture. At those parts of the texture, the roughness will be higher, causing there to be more blur when transmission is applied. These spots correspond to the raindrops themselves.

Here’s the Three.JS material containing all the properties we set to far:

To be honest, I’d say that it they already sort of look like raindrops! They look more like smudges than anything, though, and they lack a feeling of depth, but the effect is starting to take shape already.

Note About Three.JS Transmission Shader Blur

One important thing to note is that Three.JS made some major improvements to its transmission shader back in version 150 thanks to work by n8programs. If you try to replicate this material or honestly if you make use of transmission with blur at all, make sure you’re using a version of Three.JS with those upgrades.

Here’s a before-and-after screenshot from the PR implementing the changes (left = before, right = after):

As you can see, it’s a massive improvement. My material looked pretty bad without it!

Normal Map

Now it’s time to give those raindrops the depth they’re lacking.

Normal maps define the angle of the surface at every point on the material. It can be used to add additional details to materials without having to add additional geometry.

I used a web service called Poly to generate PBR maps - including a normal map - from my gold texture using AI. The result is a normal map which looks like this:

This normal map has a very prominent impact on the way the transmission shader runs.

A key part of the math behind transmission relies on the angle at which light enters and exits the material. The normal map causes this to vary greatly around the raindrop streaks themselves and causes the path of light to vary greatly on them - just like in real life.

Here’s how the windows look after adding in the normal map:

And there you go - that’s the core of the effect!

Light is strongly refracted around the raindrops - especially around their edges - which causes light from different parts of the background scene to get pulled through and creating the cool-looking distortions. It’s a real testament to the power of physically-based rendering that this works as well as it does.

Thickness / Thickness Map

Now we just need to put on some finishing touches.

The thickness parameter controls how thick the material is assumed to be when doing transmission computations. The thicker it is, the more refraction and other effects will take place.

I chose a base thickness value of 0.8 for my material. I’m pretty sure that’s way higher than anything physically accurate, but I found that it seems to exaggerate the raindrops and make the end result look better to me.

The thicknessMap is multiplied to the base thickness across the material’s surface just like roughnessMap and others. I used the same texture as for the roughness map for the thickness map.

It has a subtle but noticeable effect and helps make the raindrops even more prominent, treating the material as thicker at places where the raindrops are on its surface.

Texture Map

The final piece for this material is the small black flecks on the window’s surface.

To implement it, I used an almost-pure-white texture with some black flecks on its surface and set it as the material’s map.

Here’s the actual image I use for it:

To make the texture, I also used Stable Diffusion. I generated a pretty generic texture like this to start out:

This texture is also 1024x1024 and seamless like the gold texture, although that doesn’t matter as much in this case.

I then processed it in GIMP, adjusting the color balance, exposure, and other things until it was almost perfectly white and only a few tiny flecks were visible.

I found that it’s important to keep the map subtle for this material; making it too busy or opaque quickly makes the window look messy and greatly takes away from the cool transmission effects on the raindrops.

Final Result

Here’s the whole code I ended up with for the final rain-covered windows material:

const goldTextureNormal = await loadTexture('https://i.ameo.link/be2.jpg');
const goldTextureAlbedo = await loadTexture('https://i.ameo.link/be0.jpg');
const windowSurface = await loadTexture('https://i.ameo.link/bn8.jpg');

goldTextureNormal.repeat.set(34, 34);
goldTextureAlbedo.repeat.set(34, 34);
windowSeamless.repeat.set(40, 40);

const greenhouseWindowsMaterial = new THREE.MeshPhysicalMaterial({
  map: windowSurface,
  transmission: 1,
  roughness: 0.64,
  roughnessMap: goldTextureAlbedo,
  normalMap: goldTextureNormal,
  ior: 1.6,
  thickness: 0.8,
  thicknessMap: goldTextureAlbedo,
});

There’s really not a lot to it code-wise; Three.JS does all the heavy lifting for us with its fancy transmission shader.

Once you have good textures to use for your maps, it’s mostly just a matter of playing around with the IOR and roughness until you’re happy with the way it looks.

Feel free to use these textures yourself directly; they’re public domain. You can copy this whole material as well if you want. If you do, I’d love to hear if you build something cool with it!

My twitter is @ameobea10 and my mastodon is @ameo@mastodon.ameo.dev; feel free to let me know there. I also post updates when I work on stuff like this, so you can give me a follow if you’d like to see those.

Fixing Anker USB Hub Not Connecting to M1 Mac

Mon, 06 Nov 2023 12:02:42 -0800

I have an Anker USB hub that I use with my work laptop - an M1 Mac Pro. I use it to plug in two USB-A peripherals (mouse and keyboard) as well as to plug in a HDMI monitor. The hub itself connects to my laptop via USB-C. In addition to the hub, I also have a second HDMI monitor, a USB-C internet adapter, wired headphones, and my charging cable connected to the laptop.

The Problem

I’ve been having an issue where the USB hub refuses to connect to the laptop after plugging it in after first setting it up for the day. When I plug the hub into the laptop, nothing will happen for a few seconds. Then, the white circular light on the hub will start blinking indefinitely and the hub will still not connect and none of the peripherals plugged into it nor the monitor are usable.

This happens in spurts of a few days/weeks, seemingly randomly. In the past, I’ve fixed the problem to repeatedly un-plugging and re-plugging the hub - sometimes several times over the course of 15+ minutes - until it finally decides to connect.

The Fix

I’ve since figured out a more consistent fix for this issue.

I plug both the hub and my second HDMI monitor into the right side of my laptop, and the plugs are right next to each other:

I figured out that if I disconnect the HDMI cable plugged directly into the laptop and then unplug and re-plug the USB hub, it will almost always connect successfully after less than a minute. Then, I can re-connect the HDMI cable without issue and use both my monitors as well as my external mouse and keyboard with no problems.

I don’t know why this happens; it might be some bug with my USB hub, some issue with the M1 Mac itself with my specific peripheral configuration, some static electricity or interference thing going on - who knows. I do know that I’ve ended up burning multiple hours waiting for this stuff to connect over the past couple of years and I’m happy to have finally figured out a solution.

Investigating Fidget Spinner Bot

Mon, 06 Nov 2023 11:50:37 -0800

While watching some logs for my webserver recently, I’ve noticed a significant amount of requests coming from a bot I didn’t recognize with the user agent of fidget-spinner-bot. It seems to be pretty aggressively crawling my personal network of sites that I maintain, following links and downloading page contents. Some requests are also coming from user agents including my-tiny-bot, thesis-research-bot, test-bot which seems to be the same or related. I usually recognize most of the user agents of bots making significant amounts of requests to my server, so these stood out to me.

I first noticed this bot around the beginning of November 2023, but it’s possible it started earlier than that. I looked around online to see if other people were noticing traffic from this bot, and there are a few discussions but not many. There’s a forum thread on a webmaster forum where people were noticing traffic from these bots as well, but that’s pretty much all I could find.

As mentioned on that thread, these bots are both hosted on AWS IP space. Requests are spread out over multiple IP addresses which would indicate that there are multiple coordinated instances of these bots doing this crawling rather than just some little project someone set up on a VPS.

They don’t seem to be doing anything that seems malicious as far as I can tell. Their request counts are high but very much reasonable - at least for my server. They only seem to be requesting the HTML body of pages and not downloading any CSS, images, scripts, etc. There was some speculation on that webmasterworld thread that they might gathering training data for AI, but most people just go with commoncrawl for that rather than doing the collection themselves.

These bots follow 301 redirects and seem to spider links pretty naively. I have some densely linked doc pages for a browser-based synthesizer I built with a lot of densely inter-linked pages, and it’s spending a lot of time crawling those pages in circles. I’m not 100% sure it’s re-crawling the same pages, but it does seem to be.

All of its requests are made via HTTP 1.1. Individual requests are spaced 30 seconds to 2 minutes apart from what I can see, and that’s across all of my different domains. It’s made requests to at least 3 of my domains so far which makes sense since many of them are linked to from this website.

Anyway, it’s an interesting bot and I’d be very curious to know what it’s doing, who runs it, etc. If someone knows, feel free to get in contact and I’ll update this page with anything people find out.

UPDATE 2023-11-12

I was contacted via email from someone who works as a network administrator at a hosting company. He found this post and told me that he’s also been seeing high levels of traffic across many of his customers’ websites hosted there - to the point that he felt it was malicious.

He says that the bot seemed to be making specialized requests to various types of sites (woocommerce, drupal, wordpress, etc.) with the intention of using up as many resources as possible. I can’t corroborate that myself as I don’t really host any of those kinds of sites and hadn’t noticed malicious-level traffic on my own sites/server.

That being said, I did another search online and found some other people that were reporting that the bot was making huge numbers of requests and was causing service disruptions on their sites: https://community.cloudflare.com/t/getting-tons-of-bad-bots-attack-recently/578502/14

I just scanned my server logs and for today, I don’t see any requests from these bots. Maybe they’ve gone, or maybe they’ve changed their user agents.

Exporting Minecraft Objects to Three.JS

Fri, 03 Nov 2023 03:47:51 -0700

Recently, I’ve been working on some interactive sketches/games in Three.JS. For one of the levels I was building, I had the idea of importing something I built from one of my old MineCraft survival worlds in to use as part of it.

I figured that there was a pretty good chance of some software existing to export MineCraft levels to some 3D model format for 3D rendering or other purposes, and that indeed is the case. There are multiple options out there, but the one I chose to go with was jmc2obj.

Exporting with `jmc2obj`

jmc2obj is a fairly minimalistic application which lets you export parts of a MineCraft world into .obj format - a very common and simple 3D object format. I chose it because it’s very simple to install (just a standalone .jar file you can download), it’s fully open source, and it has some pretty good docs and usage guides.

Its UI is pretty straightforward to use. You load your world, select the dimension to export, select the region of the world you want to export, set some export settings, and get a .obj file as output.

There are a good deal of options, but most are fine at their defaults and there are some thorough docs on them as well. Here are a few you might want to take a look at though:

“Select blocks to export” if you want to filter out some blocks from the selection. I used this to get rid of torches, ladders, and some other stuff that I didn’t want included in the generated model.
“Export Textures” if you want the generated model to be textured with MineCraft textures. It will use the vanilla resource pack textures by default, but you can also load in your own resource pack.
- Apparently there are also some special PBR resource packs that have things like normal maps built in. jmc2obj has support for these, and if you use one the resulting model’s materials will be full-fledged PBR materials themselves.

After you run the export, you should have a .obj file and optionally some associated .mtl files. These contain the geometry and textures (if applicable) for your exported model respectively.

Loading the .obj Model in Three.JS

Three.JS has built-in support for loading .obj and .mtl files, so it’s actually possible to load the exported model into Three.JS directly. After serving the generated files (.obj, .mtl, and the tex directory containing texture images), they can be loaded with code something like this:

import { OBJLoader } from 'three/addons/loaders/OBJLoader.js';
import { MTLLoader } from 'three/examples/jsm/loaders/MTLLoader.js';

const mtlLoader = new MTLLoader();
mtlLoader.load('/obj/mc-demo.mtl', materials => {
  materials.preload();
  const objLoader = new OBJLoader();
  objLoader.setMaterials(materials);
  objLoader.load('/obj/mc-demo.obj', obj => {
    scene.add(obj);
  });
});

After adding lights, orbit controls, and some light postprocessing, the result for my export looks like this:

Loading + Modifying the Model in Blender

This indeed works! You can load these models into your world, add other stuff, and even integrate them with some physics engine like rapier or ammo.js.

For my purposes, though, I wanted to do a bit more and move away from the default MineCraft aesthetic. Since the exported .obj file is a standard 3D model, it’s possible to work with it like any other 3D model. I use Blender for all my 3D modelling, so I imported the .obj into Blender. It’s extremely easy - just File -> Import -> Wavefront (.obj) and you should see it loaded with textures:

Re-Exporting to glTF + Building a Scene

Now that we’re working in Blender, we can do a ton of stuff to the model. I re-textured it completely to hide the fact that it’s made out of individual blocks. In blender, I hand-modelled some other objects to flesh out the scene. Then, I re-exported it as a gLTF file which is a more modern and feature-rich model format which is very commonly used in Three.JS.

In Three.JS, I added in some more lighting, procedural terrain, and volumetric fog. Here’s the result:

It’s the same wall as the one from MineCraft, but the aesthetics are completely different. I’m a big fan of the creative possibilities of using this kind of technique.

Google Cloud Platform Gems

Wed, 25 Oct 2023 13:36:15 -0700

I was reading a Hacker News thread for an article comparing GCP to some alternatives like AWS. I’ve been a GCP user for a good while now, and it’s definitely my go-to public cloud. We also use it at my dayjob at Osmos.

Reading the article and comments got me thinking about some of my favorite GCP features. GCP has a few excellent gems which are better than pretty much any competing cloud offering:

Cloud Run. Best way to deploy containers hands down. All of the benefits of a serverless/containerized workload with all the ease of a traditional VPS deployment. Extremely cheap (pay $0 for side projects with little traffic).
BigQuery. Very easy to use with immense power without having to deal with the details yourself. Billing can be either extremely cheap or rather expensive depending on what you’re doing, though.
GCS Archive Storage. 1/3 the cost of glacier for storage ($0.0012/GB/Month lol), but more for retrieval. Has none of the annoyances of Glacier (you can download your files instantly, upload using normal APIs). Perfect for extremely cheap backups.
Cloud Spanner. I’ve not had a reason to use this yet myself, but there really aren’t any comparable offerings that I know of.

Fixing Nginx Reverse Proxy Server Not Compressing gltf/glb Files

Tue, 24 Oct 2023 03:39:17 -0700

I’ve been building some interactive sketches/games in Three.JS, and I wanted to deploy it on my server. I export the models used by the level from Blender in glTF format, which is a modern, well-supported, and commonly used format for this. Specifically, I exported the models as a .glb file.

The Problem

When I loaded my levels in the browser, I noticed that the .glb file wasn’t getting compressed with gzip. This was a problem because the file is quite large - over 4MB - and glb is a format that should benefit greatly from compression.

For my particular server setup, I use NGINX as a reverse proxy to a different Apache2 server hosting the raw files. Somewhere along the line, the compression was not taking place for some reason.

NGINX Config Changes

There were a few things causing this issue for me.

First, I had to add some MIME type definitions to my NGINX server manually to tell it that gltf/glb files are compressible. I added this to my nginx.conf file:

  gzip_proxied any;
  gzip_types text/plain text/css ... ... model/gltf model/gltf-binary;
  brotli_types text/plain text/css ... ... model/gltf model/gltf-binary;

This gives a list of MIME types that NGINX will treat as compressible. I also added some entries to the mime.types file to instruct it to map .gltf and .glb files to those mime types:

  model/gltf                            gltf;
  model/gltf-binary                     glb;

However, even after doing that, it still wasn’t working. The gzip_proxied directive should be causing it to compress any compressible responses from the upstream Apache2 server even if they weren’t compressed natively, but it wasn’t happening.

Apache2 Config Changes

It turns out that I also needed to make a config change on the upstream Apache2 server to get it to work.

NGINX bases its checks on whether or not to compress a proxied response based on the Content-Type header of that response, which is set by the upstream. It turns out that the upstream Apache2 server serving the .glb file didn’t know about its MIME type, so I had to add config entries there as well to map the extension to the MIME type:

  AddType model/gltf .gltf
  AddType model/gltf-binary .glb

After doing that, the Apache2 upstream server properly set the Content-Type: model/gltf-binary header on the response, even though it didn’t compress it. Then, the NGINX reverse proxy server saw that content type and matched it to the compressible MIME types I configured earlier and properly compressed it before forwarding it to the user.

After compressing with gzip, my glb response size went down from 4.2MB to under 1MB; very much worth the effort!

Speeding Up Geodesic Tracing in `geometry-central`

Thu, 12 Oct 2023 01:21:13 -0700

As part of some recent work in procedural mesh generation, I’ve been working with a computational geometry library called geometry-central to trace geodesic paths on the surface of 3D meshes. geometry-central is written in C++, and I compiled the library to WebAssembly with Emscripten in order to use it in the browser.

As I recently learned, a geodesic path is the straightest path along a surface. It’s the path you would take if you were to walk in a straight line across the surface of some manifold for some distance in some direction.

I’m using them to wrap a 2D mesh onto the surface of a 3D mesh, kind of like inverse UV mapping or a sort of variation of the “shrink wrap” modifier from Blender. After extruding the output from that, the result looks pretty cool:

I need trace up to millions of geodesic paths to generate this, and I want to do so as quickly as possible in order to make it possible to generate it dynamically. However, the initial performance numbers I was seeing weren’t very optimism-inspiring:

Drilling in further, it became clear that some functions from the Eigen linear algebra library named ColPivHouseholder were the bottleneck of the process:

I found the function in the geometry-central code where this was getting called. It was part of the logic for converting cartesian coordinate to barycentric coordinates. Barycentric coordinates are a way of representing some point within a triangle as a normalized mixture of each of the triangle’s points. It is used by the geodesic path tracing process.

Here’s the code itself:

inline Vector3 normalizeBarycentricDisplacement(Vector3 baryVec) {
  double s = sum(baryVec);
  return baryVec - Vector3::constant(s / 3.);
}

inline Vector3 cartesianVectorToBarycentric(const std::array<Vector2, 3>& vertCoords, Vector2 faceVec) {
  // Build matrix for linear transform problem
  // (last constraint comes from chosing the displacement vector with sum = 0)
  Eigen::Matrix3d A;
  Eigen::Vector3d rhs;
  const std::array<Vector2, 3>& c = vertCoords; // short name
  A << c[0].x, c[1].x, c[2].x, c[0].y, c[1].y, c[2].y, 1., 1., 1.;
  rhs << faceVec.x, faceVec.y, 0.;

  // Solve
  Eigen::Vector3d result = A.colPivHouseholderQr().solve(rhs);
  Vector3 resultBary{result(0), result(1), result(2)};

  resultBary = normalizeBarycentricDisplacement(resultBary);

  return resultBary;
}

geometry-central chose to implement the conversion by solving the system of equations directly via one of Eigen’s built-in solvers. This is a totally valid and correct way to handle it, but as you might imagine it leaves a lot of performance on the table.

The data has to be copied into the Eigen matrices, the solver has to do whatever initialization and cleanup it needs to, and then there’s that additional normalizeBarycentricDisplacement operation at the end that I don’t quite understand the purpose of.

Since we know we’re doing Cartesian -> barycentric coordinate conversion, we can use a bit of a more specific algorithm than what Eigen might be doing here.

I found some code on Gamedev Stack Exchange that itself was based on code from http://realtimecollisiondetection.net/ by Christer Ericson that implements the conversion process using simple math operations inline. I made some small tweaks to get the coordinates looking exactly like they did using Eigen, and wound up with this equivalent code:

inline Vector3 cartesianVectorToBarycentric(const std::array<Vector2, 3>& vertCoords, Vector2 faceVec) {
  Vector2 v0 = vertCoords[1] - vertCoords[0];
  Vector2 v1 = vertCoords[2] - vertCoords[0];
  Vector2 v2 = faceVec - vertCoords[0];

  double d00 = dot(v0, v0);
  double d01 = dot(v0, v1);
  double d11 = dot(v1, v1);
  double d20 = dot(v2, v0);
  double d21 = dot(v2, v1);
  double denom = d00 * d11 - d01 * d01;

  double v = (d11 * d20 - d01 * d21) / denom;
  double w = (d00 * d21 - d01 * d20) / denom;
  double u = 0. - v - w;
  return Vector3{u, v, w};
}

Results

Well is it faster? Yep!

The entire function is over 3x faster and there’s other stuff going on besides this barycentric coordinate conversion, so the speedup for that is even greater. This should now be fast enough for my purposes.

If I need more, the answer on stack exchange points out that a lot of the computations in that function can be pre-computed per triangle since they don’t depend on the input Cartesian coordinates. I’m not sure how much of an impact that would actually make for this case, though, but it’s an option to pursue if needed.

It was very nice to find an easy to spot bottleneck and a pre-made optimization I was able to drop in with little effort.

I pushed up these changes to a fork of geometry-central if you’re interested in using them directly.

Fixing rust-analyzer to work with `uuid_unstable` for v7 UUIDs

Mon, 02 Oct 2023 20:28:57 -0700

Recently, one of my coworkers added code that uses v7 UUIDs. v7 UUIDs are a new type of UUID that contains a timestamp along with randomness, making them useful for DB primary keys and similar things.

However, that code broke my rust-analyzer install for local development. I already was running rust nightly locally, so there was some other issue. It said that Uuid::new_v7() wasn’t a function even though the v7 feature was enabled for the uuid crate and it was at the latest version.

Looking at the code, it seems that the UUID v7 functions are only available if the uuid_unstable feature is set:

#[cfg(all(uuid_unstable, feature = "v7"))]
mod v7;

We already had a .cargo/config.toml file in our workspace root that had this config in it to enable that cfg item:

[build]
# required to enable the v7 feature in the uuid crate
# https://docs.rs/uuid/latest/uuid/#unstable-features
rustflags = ["--cfg", "uuid_unstable"]

However, rust-analyzer wasn’t picking that up and was showing errors in the code during development.

The Fix

I added these two items to my VS code config:

{
  "rust-analyzer.check.extraEnv": {
    "RUSTFLAGS": "--cfg uuid_unstable"
  },
  "rust-analyzer.cargo.extraEnv": {
    "RUSTFLAGS": "--cfg uuid_unstable"
  }
}

After setting those and reloading VS code, rust-analyzer properly finds that function, the error goes away, and I have inline documentation for it as well:

Fixing `HSA_STATUS_ERROR_OUT_OF_RESOURCES` Error with Stable Diffusion

Sun, 01 Oct 2023 22:45:34 -0700

I’ve been using Stable Diffusion XL via the Automatic1111 web UI to generate PBR textures for use in my Three.JS projects, as I’ve written about previously. Everything was going great until at random, the generation started crashing at 100% and I got this error in my console:

:0:rocdevice.cpp :2786: 58285575154 us: 154003: [tid:0x7f59ee1216c0] Callback: Queue 0x7f583ec00000 Aborting with error : HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events. Code: 0x1008 Available Free mem : 13386 MB ./webui.sh: line 254: 154003 Aborted (core dumped) "${python_cmd}" "${LAUNCH_SCRIPT}" "$@"

My GPU didn’t seem to be close to running out of memory, I hadn’t changed anything notable with my system or with stable diffusion; I was stumped.

I tried several things which didn’t work:

Updating ROCm from 5.5 to 5.7
Rebooting my computer multiple times
Closing all other running applications
Disabling all my stable diffusion extensions and resetting all settings to default

The Solution

What finally fixed it was pulling the latest automatic1111 stable diffusion web UI and using that. I’m now able to generate images without issue again.

So yeah - if you’re getting this issue or one like it, pulling the latest code for that and see if it fixes it.

Investigating Hill Noise for Terrain Generation

Tue, 26 Sep 2023 22:21:51 -0700

I’ve been working on some procedural terrain generation for my 3D work in Three.JS lately. I wanted to try out a fresh noise function for generating terrain; everyone has used Perlin noise or some variant of it for decades.

I came across a blog post by Bruce Hill describing a noise function he designed himself - called Hill Noise. It works by combining a bunch of sine waves together with different offsets and rotations. If you add enough of them together, the results can look pretty organic and random.

The code is very short to implement this, so I ported it to Rust with minimal effort. I then tested it out generating some terrain. Here’s one of the results:

It’s alright. This example uses 11 octaves of hill noise with wavelengths of [220, 160, 120, 100, 75, 40, 20, 10, 5, 2, 1]. So that requires 11 sin operations per lookup; not the cheapest thing in the world, but it’s not horrible.

Here’s closer-up view with a wireframe to see a bit more detail:

You can kinda see the higher-frequency sine waves showing through on the surface of the terrain. It looks quite regular with all the lumps about the same size, shape, and spacing. It’s probably possible to alleviate this by fine-tuning/adding more wavelengths or manually setting amplitudes.

When I expand the terrain out further, here’s how it looks:

The sine waves at the lower frequencies show through and the result looks pretty grid like and inorganic - at least it does to me.

One of the demos from Hill’s blog (https://blog.bruce-hill.com/media/hill-noise/hill_noise_2dshader.html) uses 31 octaves of noise, but if you zoom out far enough you get the same issue.

You can probably find combinations of wavelengths to make it look at least decent for most settings, but it seems harder to work with than a full-fledged Perlin or Simplex noise.

Conclusion

This is an incredibly simple noise function to implement, and it’s not that expensive to compute. It suffers from some visible tilings due to lower wavelengths of sine waves showing through.

It might be useful for some terrain generation uses, but probably not by itself.

Some Notes and Docs on the Rust Blackjack Procedural Geometry Tool

Sun, 24 Sep 2023 15:26:03 -0700

I’ve been trying out a tool written in Rust called Blackjack for procedural, node-based 3D modelling. It’s a lot like Blender’s geometry nodes.

There aren’t a lot in terms of docs, and it looks like the project isn’t being actively developed right now. Nonetheless, I think it’s an extremely cool project, and the code is very high quality. So, I’ve been spending a bit of time getting familiar with it and trying it out.

Note: This was written in September 2023. If development of this project ever picks back up, it’s likely that much of this will go out of date fast.

Selections

Some nodes like bevel, extrude, and others operate on individual edges or faces. Here’s “Edit Geometry” for example:

The way to specifying which edge/face to operate on seems to be entering their indices manually.

In the code, I found that it’s possible to use some special syntax to do things like select all and select ranges of indices:

0, 1, 2 // Select elements 0, 1 and 2
* // Select all elements
0..1 // Select a range of elements
0..5, 7..10, 13, 17, 22 // Select multiple ranges, and some single faces
 // (empty string), selects nothing

Another issue I had at first was figuring out the indices of things. I found a part of the UI under the “Mesh Visuals” button which has options to display text on the model itself to make that clear:

I looked a little bit to see if there was a way to programmatically define selections, like pick all edges that have an angle greater than some value, but I didn’t find anything for that.

Terrain

One thing I was eager to try out with this tool was terrain generation. I saw in the code that there’s some stuff there for perlin noise-based terrain generation, and I wanted to try that out.

However, as far as I can tell, that isn’t currently hooked up/accessible from the blackjack UI. The only thing that’s there is a code box to enter some lua code that’s evaluated for each point on the heightmap to generate it:

This isn’t very useful for much, unfortunately, and if I do end up using this I’ll probably end up writing some code to either pass through the perlin noise-based terrain.

Also something to note: If you create a named function, it won’t work; you have to use an anonymous lua function. This might be obvious to people familiar with Lua, but it was not for me.

Compiling to Wasm

Since Blackjack is written in Rust, it should be possible to compile it to Wasm. There’s one draft PR up on the repo with some changes aimed at facilitating that, but it’s extremely stale at this point and will likely never be merged.

I spent some time yesterday working to get blackjack_engine - the crate which implements the actual graph and node operations - compiled to Wasm and running in the web browser. After some significant effort, I was able to accomplish that. It’s able to load a .bjk file (the format produced when saving Blackjack graphs) and evaluate the graph. This includes getting the whole lua engine (which is written in C or C++) compiled to Wasm as well.

Note that this is just the engine - not the UI and editor. It should be possible to do that in the future, though, because as I understand it the UI library that Blackjack uses does have support for running in the browser.

I plan to write a separate post about this in the future.

Fixing WebGL/Three.JS `GL_INVALID_ENUM : glDrawElements: type was GL_FLOAT` error

Sat, 23 Sep 2023 11:25:33 -0700

The Problem

I was working on some procedural dynamic LOD terrain in Three.JS. As part of this, I was manually constructing indexed BufferGeometry instances. Things were working alright, but when I added in a depth pre-pass I started getting errors like this in the console and the terrain failed to render:

GL ERROR :GL_INVALID_ENUM : glDrawElements: type was GL_FLOAT

The code I had looked something like this:

const vertices = new Float32Array((segments + 1) * (segments + 1) * 3);
const indices = new Float32Array(segments * segments * 6);

// ... compute terrain and populate vertices and indices ...

const geometry = new THREE.BufferGeometry();
geometry.setAttribute('position', new THREE.Float32BufferAttribute(vertices, 3));
geometry.setIndex(new THREE.BufferAttribute(indices, 1));

const mesh = new THREE.Mesh(geometry, material);

The Cause

This error is caused because I accidentally used a Float32Array to store the indices, but I should have used an integer array type. I’m not sure why it originally worked and then stopped working when I added the depth pre-pass though.

The Fix

I switched the code creating the arrays to this:

const vertices = new Float32Array((segments + 1) * (segments + 1) * 3);
const indexCount = segments * segments * 6;
const u16Max = 65_535;
const indices = indexCount > u16Max ? new Uint32Array(indexCount) : new Uint16Array(indexCount);

and left the rest of the code the same.

After doing this, the error went away and the terrain rendered correctly again!

Debugging WebGL GLSL Shader Behavior Differences Between M1 Mac and AMD GPU

Fri, 22 Sep 2023 12:02:03 -0700

The Problem

I’ve been working on a shader in GLSL for implementing volumetric fog via raytracing. I did the majority of the work for it it on my M1 Macbook laptop while traveling, but I was eager to try it out on my powerful 7900 XTX when I got home to see how it performed.

To my surprise, the results looked extremely different! The lighting was very low-detail on my desktop with the AMD GPU compared to how it looked on my Macbook.

Here’s how it looked on the Mac:

And here’s how it looked on my desktop:

I tried changing a few things to make the cases as identical as possible. Macbooks have a High DPI screen with a pixel ratio of 2, and I thought maybe that was maybe causing the change in behavior. However, when I disabled High DPI rendering, the results on the mac looked pretty much the same - so that wasn’t it.

The Cause

To generate the fog, I use a series of composed noise functions. Each subsequent layer, or octave, is rendered at a finer scale and adds more detail to the fog. The noise function I used (psrdnoise) has built-in support for cheaply computing the gradient/derivative at each sampled point. I make use of this functionality for my lighting computations.

Here’s an excerpt from the code where I accumulate that gradient:

float sampleFogDensityLOD(vec3 worldPos, out vec3 gradient, const int lod) {
  float weight = LODWeights[lod];
  float scale = LODScales[lod];

  vec2 xzGradient;
  float noise = psrdnoise(worldPos.xz * scale, vec2(0.), 0., xzGradient);
  gradient += vec3(xzGradient.x, 0., xzGradient.y) * weight;
  return noise * weight;
}

float sampleFogDensity(vec3 worldPos, out vec3 gradient) {
  gradient = vec3(0.);
  float density = 0.;

  for (float lod = 0; lod < 4; lod += 1) {
    density += sampleFogDensityLOD(worldPos, gradient, lod);
  }

  return density;
}

It turns out that there’s an issue in this code.

The gradient parameter is specified as out vec3 gradient, but the sampleFogDensityLOD function reads and writes its value. In GLSL, arguments marked out have their value uninitialized by default.

The GLSL compiler or something else in the graphics driver stack on Mac was being lenient and preserving the value. The AMD compiler on the other hand was setting gradient to all zeroes in sampleFogDensityLOD, or something like that, and causing only the lowest level of detail’s gradient to be sampled.

This is an instance of undefined behavior. The initial value of an out argument is undefined, so the compiler is free to do whatever it wants when optimizing the code.

The Solution

The fix was simple: just change out gradient to inout gradient. That way, the value passed in is preserved and the gradient is accumulated properly.

This is something I’ll certainly watch out for when writing shaders in GLSL in the future.

Fixing "Could Not Load Openssl" Mastodon Build Error

Fri, 22 Sep 2023 11:15:03 -0700

While trying to update my Mastodon server to the latest v4.2.0 release, I kept running into build failures when running docker-compose build. I got errors like this in the logs:

#20 27.04 Bundler 2.4.10 is running, but your lockfile was generated with 2.4.13. Installing Bundler 2.4.13 and restarting using that version.
#20 27.21 There was an error installing the locked bundler version (2.4.13), rerun with the `--verbose` flag for more details. Going on using bundler 2.4.10.
#20 27.29 Could not load OpenSSL.
#20 27.29 You must recompile Ruby with OpenSSL support.

It was very confusing since the official CI builds seemed to be working fine. Nevertheless, there were multiple issues opened about this, so it’s something people were running into.

I bisected this build issue to commit https://github.com/mastodon/mastodon/commit/b749de766f5a6158fd0b5f3c3201943083fc7979 which bumps the tag of the Docker node image from Debian bullseye to bookworm.

I dug into it further. I tried to repro the build failure on my desktop, but it works there; it only fails when building on the server on which I run my Mastodon instance.

After more investigation, I found that the hashes of the ruby-jemalloc image used by the build process are different between the server and my local desktop

Server:

[
    {
        "Id": "sha256:c9d180a360e5d39e463cbb202eb10bc4e6d44af3f6ba53fbd26874027aea9ec9",
        "RepoTags": [
            "ghcr.io/moritzheiber/ruby-jemalloc:3.2.2-slim"
        ],
        "RepoDigests": [
            "ghcr.io/moritzheiber/ruby-jemalloc@sha256:9f452d67da4ffdc9abc103015ca447f442d324946a78cefcea9d35f558ce9a93"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2023-06-05T07:47:27.743594335Z",

Desktop:

[
    {
        "Id": "sha256:6bebe65ca9cf60a9d55145104943af2493c46a593c36d9ada11f0940db7a280e",
        "RepoTags": [
            "ghcr.io/moritzheiber/ruby-jemalloc:3.2.2-slim"
        ],
        "RepoDigests": [
            "ghcr.io/moritzheiber/ruby-jemalloc@sha256:9a1487af836fada451d706b4b9b4f17a42e82cab5bcef1d77cb577134473f9a6"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2023-09-18T09:01:47.627305986Z",

My guess is that whoever maintains that ruby-jemalloc image force-pushed a new version of the package to the same tag. My server has the old version cached, and my desktop picked up the new one.

The Fix

I ran docker image rm ghcr.io/moritzheiber/ruby-jemalloc:3.2.2-slim on the server. This cleared out that old stale ruby-jemalloc image, re-fetched the new one, and fixed my build.

So yeah - there’s a workaround for that @NahNick and @CSDUMMI. The issue isn’t technically on mastodon’s side, and it should go away the next time they upgrade their ruby version (as long as the ruby-jemalloc maintainer doesn’t do this again in the future).

Fixing pmndrs Postprocessing Recursive Depth Texture Binding

Tue, 29 Aug 2023 22:55:42 -0700

I’ve been working on building 3D scenes and environments in the browser using Three.JS. As part of those, I make pretty heavy use of the pmndrs postprocessing library for post-processing and effects.

I’ve also implemented a custom godrays effect that works with postprocesing called three-good-godrays. It creates a custom pass that is added to the postprocessing EffectComposer which renders volumetric screen-space godrays by reading the depth buffer and shadow map for a light.

The Problem

For one of my scenes, the postprocessing chain had gotten pretty long with several different effects in use. At some point, I started seeing errors like this in the JS console:

[.WebGL-0x16c00334d00]GL ERROR :GL_INVALID_OPERATION : glDrawArrays: Source and destination textures of the draw are the same.

In addition, my godrays effect stopped working. I used the Spector.JS browser extension to debug the WebGL rendering sequence for one of my frames and I saw that the shaders were getting launched, but nothing seemed to be getting rendered into the destination buffers.

The Cause

As I mentioned previously, the shaders used internally by three-good-godrays and some of the other effects in my pipeline needed access to scene depth information. The Passes added to the EffectComposer have a needsDepthTexture attribute that they can set to indicate that they need access to the depth buffer. If set, the EffectComposer will call their setDepthTexture() method and provide them a texture containing it.

Internally, EffectComposer has two buffers that it swaps back and forth between when rendering the passes. I looked at the code and the logic looks roughly like this:

input_buffer, output_buffer = build_buffers()

if any_pass_needs_depth_texture:
  input_buffer.bind_depth_texture(build_depth_texture())

for i, fx_pass in enumerate(passes):
  is_last = i == len(passes) - 1
  render_to_screen = is_last or pass.render_to_screen
  # Passing None indicates that the pass should render to the canvas framebuffer
  # which puts its output directly onto the screen
  fx_pass.render(input_buffer, None if is_last else output_buffer)

  # needs_swap defaults to true
  if pass.needs_swap and not render_to_screen:
    output_buffer, input_buffer = input_buffer, output_buffer

When the depth texture is first initialized, it gets set on the input buffer. Since these buffers swap back and forth each frame, it’s possible that the depth buffer will be attached to the output buffer while processing the effect. If the pass makes use of the depth texture as input for some shader by passing it as a uniform or similar, it will cause the WebGL error I pasted before and cause the pass to fail to render.

This is a known bug/limitation of postprocessing. There are multiple issues about this:

They plan to address it in a release of version 7, which is not yet out at the time of writing this.

The Fix

Luckily, it’s possible to work around this issue. I had to update three-good-godrays to detect and handle case where the provided depth texture is the same as the one bound to the provided output buffer.

If it is the same, then I allocate an additional framebuffer the same size of the depth texture, run a CopyPass to copy the contents of the depth texture into it, and then bind that copied buffer as the input for the sceneDepth uniform of the shader instead. This fixes the “Source and destination texture sof the draw are the same” error and allows the pass to render without issue.

I made that change in this commit. I do hope that pmndrs postprocessing gets that v7 rework; it was very hard to figure out what was causing this and debugging it took several hours.

Fixing "performance is not defined" Issue with Svelte/SvelteKit

Fri, 25 Aug 2023 00:32:30 -0700

The Issue

When building my SvelteKit project, I get lots of errors like this in my console:

ERROR in ./src/graphEditor/nodes/CustomAudio/MIDIToFrequency/MIDIToFrequencySmallView.svelte
Module build failed (from ./node_modules/svelte-loader/index.js):
ReferenceError: performance is not defined
    at now (/home/casey/web-synth/node_modules/svelte/compiler.cjs:7:31)
    at new Stats (/home/casey/web-synth/node_modules/svelte/compiler.cjs:47:21)
    at compile (/home/casey/web-synth/node_modules/svelte/compiler.cjs:45535:16)
    at injectVarsToCode (/home/casey/web-synth/node_modules/svelte-preprocess/dist/transformers/typescript.js:85:45)
    at mixedImportsTranspiler (/home/casey/web-synth/node_modules/svelte-preprocess/dist/transformers/typescript.js:257:26)
    at transformer (/home/casey/web-synth/node_modules/svelte-preprocess/dist/transformers/typescript.js:336:11)
    at transform (/home/casey/web-synth/node_modules/svelte-preprocess/dist/autoProcess.js:46:12)
    at async /home/casey/web-synth/node_modules/svelte-preprocess/dist/autoProcess.js:117:29
    at async script (/home/casey/web-synth/node_modules/svelte-preprocess/dist/autoProcess.js:147:33)
    at async process_single_tag (/home/casey/web-synth/node_modules/svelte/compiler.cjs:45984:21)
    at async Promise.all (index 0)
    at async replace_in_code (/home/casey/web-synth/node_modules/svelte/compiler.cjs:45708:23)
    at async process_tag (/home/casey/web-synth/node_modules/svelte/compiler.cjs:46001:26)
    at async preprocess (/home/casey/web-synth/node_modules/svelte/compiler.cjs:46059:25)

They’re originating from within the Svelte compiler.

The Cause

Svelte/SvelteKit dropped support for NodeJS v14. I still had this old version going locally which caused the build to fail because it was trying access stuff that wasn’t available in that version.

The Fix

I upgraded to NodeJS v18 which is much newer and works fully with Svelte/SvelteKit:

nvm use 18.9.0

You might have to use a different method to do this if you’ve got Node installed differently on your system, but yeah after doing that the build went through fine for me.

Fixing "can't resolve 'svelte/internal'" Error After Upgrading `svelte`

Thu, 24 Aug 2023 17:00:08 -0700

I bumped several of the dependencies for one of my projects. It uses Svelte and Webpack, and makes use of svelte-loader to facilitate importing .svelte files.

I upgraded Svelte from v3.57.0 to v4.2.0, and bumped svelte-loader, svelte-preprocess, prettier-plugin-svelte, and many other libraries to their latest versions as well.

After the upgrade, my Webpack dev server started up but failed to load with many errors like these displayed in the console:

ERROR in ./src/welcomePage/DemoTile.svelte 2:0-28:25
Module not found: Error: Can't resolve 'svelte/internal' in '/home/casey/web-synth/src/welcomePage'
 @ ./src/welcomePage/WelcomePage.svelte 27:0-55 118:17-25 132:17-25 146:17-25 160:17-25 495:34-42
 @ ./src/welcomePage/WelcomePage.ts 3:0-49 14:10-23
 @ ./src/engine_bg.js 17:0-124 639:2-19 643:2-22 647:2-19 651:2-21
 @ ./src/engine.js 2:0-31 2:0-31
 @ ./src/index.tsx 14:13-31

ERROR in ./src/welcomePage/DemoTile.svelte 30:0-42
Module not found: Error: Can't resolve 'svelte/internal/disclose-version' in '/home/casey/web-synth/src/welcomePage'
 @ ./src/welcomePage/WelcomePage.svelte 27:0-55 118:17-25 132:17-25 146:17-25 160:17-25 495:34-42
 @ ./src/welcomePage/WelcomePage.ts 3:0-49 14:10-23
 @ ./src/engine_bg.js 17:0-124 639:2-19 643:2-22 647:2-19 651:2-21
 @ ./src/engine.js 2:0-31 2:0-31
 @ ./src/index.tsx 14:13-31

ERROR in ./src/welcomePage/WelcomePage.svelte 2:0-24:25
Module not found: Error: Can't resolve 'svelte/internal' in '/home/casey/web-synth/src/welcomePage'
 @ ./src/welcomePage/WelcomePage.ts 3:0-49 14:10-23
 @ ./src/engine_bg.js 17:0-124 639:2-19 643:2-22 647:2-19 651:2-21
 @ ./src/engine.js 2:0-31 2:0-31
 @ ./src/index.tsx 14:13-31

ERROR in ./src/welcomePage/WelcomePage.svelte 26:0-42
Module not found: Error: Can't resolve 'svelte/internal/disclose-version' in '/home/casey/web-synth/src/welcomePage'
 @ ./src/welcomePage/WelcomePage.ts 3:0-49 14:10-23
 @ ./src/engine_bg.js 17:0-124 639:2-19 643:2-22 647:2-19 651:2-21
 @ ./src/engine.js 2:0-31 2:0-31
 @ ./src/index.tsx 14:13-31

The Fix

I found the fix in a Github issue thread after extensive searching.

In my Webpack config file webpack.config.js, I had this code:

    alias: {
      svelte: path.dirname(require.resolve('svelte/package.json')),
    },

I don’t remember why I put it there originally; probably copy-pasted from a README of template.

In any case, this code is broken after updating from Svelte 3->4 when using Webpack and needs to be updated.

Here was the diff I applied:

diff --git b/webpack.base.js a/webpack.base.js
index e73c6d2..da52655 100644
--- b/webpack.base.js
+++ a/webpack.base.js
@@ -74,7 +74,7 @@ const config = {
     extensions: ['.ts', '.tsx', '.js', '.jsx', '.wasm', '.svelte', '.mjs'],
     modules: [path.resolve('./node_modules'), path.resolve('.')],
     alias: {
-      svelte: path.dirname(require.resolve('svelte/package.json')),
+      svelte: path.resolve('node_modules', 'svelte/src/runtime'),
     },
     conditionNames: ['require', 'node', 'svelte'],
   },

After re-starting the webpack dev server, everything works just fine!

`amdgpu_top`: A Modern `radeontop` Alternative

Wed, 23 Aug 2023 21:57:55 -0700

I’ve been using a tool called radeontop for years to monitor the performance and utilization of my AMD GPUs on Linux. It’s a TUI-based application that renders the value of different performance counters as bar charts:

For the most part, it does a good job and it provides a concise overview of GPU utilization.

However, it seems that radeontop is no longer actively developed/updated.

It’s received ~7 commits in the past ~3 years and although it does still mostly work even with the latest GPUs like the 7900 XTX, but it’s not under active development.

Enter `amdgpu_top`

Today, I discovered an alternative tool that does all of the same things as radeontop and more: amdgpu_top. It’s actively developed with commits in the past couple of weeks at the time of writing this.

It is written in Rust, and so was extremely easy to install:

cargo install amdgpu_top

Here’s what its UI looks like:

As you can see, there’s a ton more data available! One immediate benefit is the process list which shows which processes are consuming what GPU resources. This is extremely useful and something I had no ability to see with radeontop.

The sensors view is also extremely useful. radeontop had no way to track GPU temperature, but here amdgpu_top provides that along with block rates, power metrics, and more.

For the displayed performance counters, I think they’re using the same underlying data with some differences in naming and grouping. I think that radeontop is slightly more granular with the performance counters, but I honestly have no idea what half of those counters even track so having a bit of a coarser view is actually beneficial to me.

GUI View

Another big upgrade that amdgpu_top provides is a fully-fledged GUI application as an alternative to their default TUI view. It can be accessed by launching amdgpu_top --gui:

It’s quite dense, but that’s because it’s packed with so much information. Having the ability to track GPU utilization over time is a hugely useful feature, and as such I’ll probably end up using the GUI mode most often when I use amdgpu_top.

Conclusion

So yeah, personally I see no way at all in which radeontop wins over amdgpu_top. amdgpu_top seems like a clear upgrade in every way, and I’ll not be looking back personally.

I’ll admit that as an avid Rust programmer, I’m slightly biased since amdgpu_top is written in Rust and radeontop is in C, but even besides that it’s just a better, more feature-rich, and modern tool.

If you’re interested in tracking performance for your AMD GPU, give amdgpu_top a try for sure!

Generating 4K PBR Textures Using Stable Diffusion XL

Sat, 19 Aug 2023 17:05:24 -0700

I’m picking back up the work that I started last year building 3D scenes and sketches with Three.JS.

At that time, it was just after AI image generators like DALL-E and Stable Diffusion were really taking off. I had success running Stable Diffusion locally and using it to generate textures for terrain, buildings, and other environments in the 3D worlds I was building.

I was using Stable Diffusion v1 back then. I had found some prompts that I liked which created images with styles kind of like this:

SDv1 was very good at generating textures like this, and I generated hundreds/thousands of images while playing around with different prompts.

It is now August 2023, and the AI image generation ecosystem has continued to improve by leaps and bounds. I’ve been trying out Stable Diffusion XL for texture generation, starting out with mostly terrain, and I’ve been getting some great results.

At first, I was getting some discouraging results since the prompts I was using before weren’t working very well. However, after some experimentation, I’ve been able to get results that are way better than anything I’d accomplished in the past.

Stable Diffusion XL v1.0 PBR Texture Generation Workflow

I’ve come up with a pretty solid workflow for generating high-quality 4K PBR textures using Stable Diffusion XL and some other tools. I’m extremely impressed with the results so far, and I’ve only been playing around with this stuff for a few days now.

Here’s the kind of results I’m able to get:

Installing + Running Stable Diffusion XL with 7900 XTX

I recently bought a 7900 XTX graphics card. It has 24GB of VRAM which is enough to generate 1024x1024 images with Stable Diffusion with no upscaling or other tricks required.

I chose the AUTOMATIC1111 WebUI for installing and running Stable Diffusion. It seems to be the most feature-rich and popular, and it supports AMD GPUs out of the box. I cloned the repository, ran the ./run_webui.sh script, and it handled automatically installing dependencies and downloading Stable Diffusion weights and other model files.

To run it, I needed to do a few manual things. I got a segfault on my first run but was able to work around this by setting two environment variables:

export CL_EXCLUDE=gfx1036
export HSA_OVERRIDE_GFX_VERSION=11.0.0

I’m currently using ROCm v5.5 which I believe is why this is required. That version adds support for AMD 7000-series GPUs, but the support isn’t quite complete. These environment variables trick Pytorch and other libraries into thinking it’s some other model and making it work.

I previously wrote some notes about installing ROCm and TensorFlow with the 7900 XTX. They might already be out of date, though.

Anyway, after installing and exporting those environment variables, the web UI started up and images were generating!

Prompts

The first step of the process was finding some good prompts for texture generation. This is of course the most creative part of the whole thing and has the biggest impact on your results.

I don’t have a ton of guidance here; it largely depends on what you’re trying to generate and the style you’re looking to get. However, here’s an example of a prompt that I’ve had a lot of success with:

top-down image of rough solid flat dark, rich slate rock. interspersed with bright ((flecks)) of ((glinting)) metallic spots like mica. high quality photograph, detailed, realistic

Negative Prompt: blurry, cracks, crevice, round, smooth, distinct, pebble, shadows, coin, circle

Note the parenthesis around some words. That’s a feature of the AUTO1111 webui I’m using, and it tells the model to pay more attention to the parts of the prompt surrounded by them. There are some other fancy prompt syntax tricks as well; tons of flexibility and options to explore.

Params

The next step is coming up with good parameters for the generation.

I’ve found that Stable Diffusion XL is actually more sensitive to the chosen params than older versions. On top of that, there are WAY more things to tweak than there was a year ago. As a result, it did take a good while to come up with settings that resulted in images I was happy with.

To help with this, I made extensive use of the “X/Y/Z Plot” feature of the AUTO1111 webui. It’s located in the “Script” dropdown of the UI:

It generates grids of output images where each cell is generated with a different combination of parameters. Here’s what the output looks like when using it:

It was extremely useful for zeroing in on a good set of baseline params for my images.

After a lot of experimentation, here’s the baseline params I now use to start with when generating new textures:

Sampling method: Euler
Sampling steps: 60
Width/Height: 1024x1024
CFG Scale: 6.5 (I found that getting this right is especially important)
Tiling: Enabled (needed to generate textures that tile seamlessly; extremely important)
Hires. fix: Disabled (I had bad luck with everything I tried with this enabled)

You’ll probably have to experiment yourself to find params that work well for your use-case, but these might work as a good baseline when getting started.

One other thing I observed is that enabling tiling causes generated images to be pretty significantly different overall. This makes it tricky when, say, generating images with the Stable Diffusion Dream Studio online to take advantage of their fast GPUs and then trying to replicate those results locally. They don’t have the tiling option in their hosted UI and don’t allow tuning some params like sampler, so getting an exact match is difficult.

Converting Textures from 1K to 4K

Once I found a good prompt and good params, I just set it to generate a bunch images and let it go! I let it run for around 2 hours and god a selection of ~50 images. A lot are bad, but there are more than enough gems in the bunch to work with.

One thing you may have noted is that I said “4K textures” in the title of this post but all the images I’ve generated so far are 1K. Well if you thought I was going to say that I did something along the line of upscaling using one of the many possible methods, you’d be wrong actually!

I created a unique method for combining AI-generated textures together to generate higher-resolution outputs. The resulting textures retain the seamless/infinitely tiling property of the source images.

I built a tool that runs in the browser to do this in an easy way. You can use it here (it’s 100% free and open source): https://texture-utils.ameo.design/seamless-stitcher/

The process is very simple. You drag and drop 4 similar-looking seamless-tiling Stable Diffusion-generated textures and it combines them together. As I mentioned, the output will also be seamless-tiling as well and go up in resolution by 4x.

Here’s what the UI for the tool looks like:

You can right-click the generated image and select “Save Image As” to download the result as a PNG. The outputs by default are extremely large, so you may want to use a tool like squoosh to compress/optimize it.

Compared to Stable Diffusion and other AI tools, the implementation is actually quite low-tech. But I’ve found that the results are very good overall!

It works best with images that look quite similar to each other. If there are big differences in style, color, etc. it can be obvious in the resulting output and look bad.

A great way to get very similar-looking source images is to find a generated image that you like and then generate variations of it. To do this, take the generated image and upload it to the “PNG Info” tab of the AUTO1111 web UI. Then, click the “send to txt2img” button and it will pre-populate the UI with all the params used to generate it.

Then, tick the “Extra” checkbox on the txt2img tab. Set “Variation strength” to something small like 0.1-0.3 and generate a dozen or so images. They should look similar to the original overall but with different details. If you choose 4 of those to upload to the seamless texture stitcher tool, I’ve found that the output usually looks good.

Building PBR Textures

Once you have a stitched 4K image you’re happy with, it’s time to make them into full PBR textures.

For this, I use a tool called Poly. They provide an AI-powered texture generator that takes an image as input and generates normal, height, ambient occlusion, metalness, and roughness maps for it. They also provide their own prompt-based generation tools, but I personally prefer the control that generating it myself provides.

Their tool lets you generate normal and height maps for free, but they charge you $20/month to generate others. I do personally pay for that subscription right now, but you don’t have to in order to get good results. The normal map is the most important part, can you can either set global roughness/metalness values for your texture (this is all you need a lot of the time) or write a custom shader to generate them on the fly from pixel values.

Another method you can use to generate normal maps is SmartNormap. It uses a non-AI approach to programmatically generate normal maps for any source image. It has some params you can tweak, and the results are OK but generally less good than the AI-powered Poly tool overall in my experience.

Anyway, yeah at that point you can download your maps and plug them into Blender, Three.JS, or whatever other 3D software you want.

If you want to get even crazier with it, the seamless 4K output texture can be used with a hex tiling algorithm to make it tile infinitely with no repetition whatsoever. I’m working on a Three.JS library to handle this automatically, and here’s what it looks like laid out on some terrain I generated in Blender:

Conclusion

It really feels like the floodgates are opened here. I personally find that using AI image generates for textures and other building-block assets rather than full images or artwork to be the way to go. There’s a lot of room for my creativity and input to guide the entire process. There are infinite possibilities to explore, and the whole process is quite fun in my opinion.

Anyway, I hope you found some of this helpful, and I wish you good luck if you decide to try this out for yourself.

Building RNN Architecture Visualizations With TikZ

Tue, 15 Aug 2023 00:42:47 -0700

I recently finished a big blog post about growing sparse computational graphs with RNNs.

An important part of that work involved creating a custom RNN architecture to facilitate the growth of extremely sparse networks. To help explain that custom RNN architecture in the blog post, I created some visualizations that looked like this:

These images are SVGs, so they scale infinitely without getting pixelated or blurry. I looked into a few different options for generating these including tools like draw.io, manually drawing them in a vector editor like Inkscape, and Graphviz.

It seems that most of the really good-looking RNN visualizations I’ve seen in popular blog posts were created by hand in Inkscape, but I really wanted to avoid doing that if I could. I got some decent results when I tried using Graphviz, but I struggled to get some of the fine-grained styling and layout controls that I wanted.

TikZ

The tool/language I ended up using to build my visualizations is called TikZ. TikZ is a sort of programming language and toolkit for programmatically generating vector graphics. It’s a sort of extension to LaTeX, so it’s commonly used for creating graphics and visualizations for research papers.

To create a standalone SVG using TikZ, it starts by writing some LaTeX code in a .tex file. Here’s the start of the code for the visualization above :

\documentclass[tikz,border=3mm]{standalone}
\usetikzlibrary{positioning, shapes, fit}

\begin{document}
\begin{tikzpicture}[
    every node/.style={draw, thick},
    operation/.style={circle, inner sep=2pt},
    kernel/.style={rectangle, fill=gray!20},
    point/.style={coordinate},
    cell/.style={rectangle, draw, thick, dashed, inner sep=2mm, fit=#1}
]

This sets up the LaTeX document, imports TikZ + some other helper libraries, defines some global default styling for the graph. kernel, point, cell, etc. are sort of like CSS classes and can be used to apply styles to elements without having to copy-paste them for each one.

For my purposes, everything else I needed to visualize was either a node or an edge. When creating nodes in TikZ, you can specify their positions relative to other objects in the graphic like this:

\node[kernel] (c0_kernel) {Kernel};
\node[operation, below=5mm of c0_kernel] (c0_dot1) {·};
\node[operation, right=15mm of c0_dot1, scale=0.73] (c0_add1) {+};
\node[kernel, above=5mm of c0_add1] (c0_bias) {Bias};
\node[operation, right=8mm of c0_add1, fill=blue!20] (c0_activation1) {$A$};

This creates 5 nodes and positions them all relative to each other, except the c0_kernel.

Edges are created like this:

\draw[->] (c0_recur_kernel) -- (c0_dot2) -- (c0_add2) -- (c0_activation2);
\draw[->] (c0_recur_bias) -- (c0_add2);
\draw[-, dashed, draw=blue!90] (c0_new_state) -- (c0_new_state_bridge);
\draw[->, dashed, draw=blue!90] (c0_new_state_bridge) -- (c0_state);

As you can see, it’s possible to choose whether to use arrows or lines to join nodes and set styles on individual edges. It’s also possible to define multiple edges on the same line, similar to Graphviz.

However, for some of the edges my visualization, I wanted to manually control the routing of edges in order to make it look better. There are a few different ways to accomplish this in TikZ, but the method I chose was to define coordinates.

In TikZ, coordinates are invisible objects that are not displayed in any way but can instead be used as sources/destinations for edges. So, to create an angled edge like the one from New State to State in the visualization, I did this:

% Define new state and state nodes
\node[kernel, left=3mm of c0_state_bridge] (c0_state) {State};
\node[kernel, below=5mm of c0_activation2] (c0_new_state) {New State};

% Define coordinate at the bend of the edge
\coordinate[left=49.4mm of c0_new_state] (c0_new_state_bridge) {};

% Draw the edge between New State and State in two parts
\draw[-, dashed, draw=blue!90] (c0_new_state) -- (c0_new_state_bridge);
\draw[->, dashed, draw=blue!90] (c0_new_state_bridge) -- (c0_state);

The final touch I did to create the visualization was draw the dashed bounding box around the whole cell:

\node[cell=(c0_kernel)(c0_state)(c0_new_state), label=above:Cell 0] {};

I followed a pretty similar process for the other RNN visualization in the blog post.

By using this method, it’s possible to have fine-grained control of the visualization’s layout and the rendering of all of its components. There also seem to be a pretty selection of resources available for TikZ which helped out a lot. It definitely looks way better than the graphviz version I had before. It did take a good amount of time tweaking the node positions to be just right, but certainly worth it for the result.

Stale Queries When Hitting BigQuery Streaming Buffer When Using BI Engine

Thu, 06 Jul 2023 16:12:49 -0500

UPDATE 2023-08-15:

Some engineers at Google reached out to me via e-mail after I submitted some feedback about this issue on the GCP console and linked to this blog post.

After a few back and forth messages, they were able to diagnose the problem and put out a mitigation that completely fixed it for us!

The issue seems to have stemmed from cursors tracking the position in the streaming buffer getting out of sync between the BI engine and base BigQuery.

This was happening due to a complex self-join in the query we were running; half the query ran in BI engine, and the other half ran on backend BigQuery.

I’m not 100% sure if the mitigation is deployed for all users at this time, but it should be soon if not.

If you’re still running into this issue, I suggest reaching out to Google via the feedback form in GCP that comes up when you disable BI engine.

Background

We use BigQuery extensively at my job at Osmos. We use it for a variety of use cases and both read and write to it very often using complex dynamically generated queries.

To increase performance and reduce costs, we use BI Engine for some of our tables. As described in the BI engine docs,

BigQuery BI Engine is a fast, in-memory analysis service that accelerates many SQL queries in BigQuery by intelligently caching the data you use most frequently.

The table on which we ran into this issue uses ingestion time partitioning. It lumps data together in its storage backend based on the timestamp at which it was inserted.

When new data is inserted into ingestion time partitioned tables in BigQuery, it is added to a streaming insert buffer. Its _PARTITIONTIME field is set to null until a point in the future when it’s asynchronously written to a partition and given a permanent _PARTITIONTIME.

We explicitly create our queries to include data in the streaming buffer like this:

SELECT ...
FROM ...
WHERE _PARTITIONTIME IS NULL
  OR _PARTITIONTIME >= TIMESTAMP_TRUNC(TIMESTAMP('2023-07-06 22:23:38'), HOUR);

This used to work just fine. Recently inserted data would show up in query results seconds after it was written. However, at some point recently, that stopped happening.

The Problem

We were running into issues where some of our queries would fail to include data from recently inserted rows. We’d send the insert rows request to BigQuery, get a 200 response code, but then queries that look at that data would fail to include some of it until much later - sometimes up to an hour.

The only setup where we noticed this happening was when we had all of the following:

BigQuery table set up using ingestion time partitioning
BI Engine enabled for the table
Inserting data dynamically into the table and query that tries to read it from the streaming insert buffer (_PARTITIONTIME IS NULL)

The Fix

… We shut of BI engine and the problem immediately stopped.

The thing is, this definitely used to work for us in the past. We’re almost certain we made no changes to our code or our infrastructure that would cause this change in behavior, so our only remaining explanation is that something changed on GCP’s end.

I’m not sure if it can be considered a bug or not. Maybe the previous behavior was incorrect and this is just BI Engine’s caching working as intended.

In any case, if you’re running into strange issues with data not showing up and have a scenario like this with BI engine enabled, try shutting it off and seeing if it fixes the issue for you.

Machine Learning Benchmarks on the 7900 XTX

Sat, 01 Jul 2023 10:09:06 -0700

I recently upgraded to a 7900 XTX GPU. Besides being great for gaming, I wanted to try it out for some machine learning.

It’s well known that NVIDIA is the clear leader in AI hardware currently. Most ML frameworks have NVIDIA support via CUDA as their primary (or only) option for acceleration. OpenCL has not been up to the same level in either support or performance.

That being said, the 7900 XTX is a very powerful card. It has 24GB of VRAM, a theoretical 60 TFLOPS of f32, and 120 TFLOPS of f16. The recent AI hype wave is also incentivizing AMD to beef ML support on their cards, and they seem to be making real investments in that space.

I ran some benchmarks to get a feel for its real-world performance right now.

These benchmarks were done in June 2023. It’s very possible that these results will change dramatically over time, even in the short term.

Simple TensorFlow Neural Network Training Benchmark

I wanted to get a feel for the actual performance that I can get with the 7900 XTX for a realistic ML training scenario. I started out by looking for some existing benchmarks that I could pull in to compare.

I found a Reddit post by cherryteastain that uses TensorFlow to run a few different ML benchmarks on a 6900XT card.

One of the cases they benchmarked is training a very simple multi-layer neural network using random data. Although it’s simple, it has over 7 million parameters:

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 dense (Dense)               (None, 2500)              252500

 dense_1 (Dense)             (None, 2500)              6252500

 dense_2 (Dense)             (None, 250)               625250

 dense_3 (Dense)             (None, 10)                2510

=================================================================
Total params: 7,132,760
Trainable params: 7,132,760
Non-trainable params: 0

Once I set up ROCm and TensorFlow, I was able to run the exact script as-is. Just in case that pastebin gets deleted in the future, I’ll include the whole thing here since it’s so short anyway:

import tensorflow as tf
import numpy as np
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  try:
    for gpu in gpus:
      tf.config.experimental.set_memory_growth(gpu, True)
  except RuntimeError as e:
    print(e)

nclasses = 10
nsamples = 3000000
bsize = nsamples//20
inp_units = 100
mod = tf.keras.Sequential([tf.keras.layers.InputLayer(inp_units), tf.keras.layers.Dense(2500, activation='relu'), tf.keras.layers.Dense(2500, activation='relu'), tf.keras.layers.Dense(250, activation='relu'), tf.keras.layers.Dense(nclasses, activation='softmax')])
mod.compile(loss='sparse_categorical_crossentropy', optimizer='adam')

inpt = np.random.rand(nsamples,inp_units)
gtt = np.random.randint(0,nclasses-1,nsamples)
dset = tf.data.Dataset.from_tensor_slices((inpt,gtt)).batch(bsize)

mod.fit(dset, epochs = 20)

Results

On my 7900 XTX GPU, I achieved 24 seconds per epoch. The author of the Reddit post included some timings for a variety of other cards that they tested:

6900XT/System 1	Titan V/System 2	V100/System 3	7900 XTX (Mine)
26s (1254ms/step)	19s (946ms/step)	20s (996ms/step)	24s (~1180ms/step)

… Not what I wanted to see. My card is performing barely better than the 6900 XT tested by the author. The 6900 XT has a theoretical max of 23 TFLOPS of FP32 performance - less than 40% of the 7900 XTX which has 61 TFLOPS of FP32 performance.

I’m not sure why the performance is so bad. One possibility is that it’s something to do with the hacky way I compiled TensorFlow to work with ROCm 5.5 and the 7900 XTX. I feel like it’s quite possible that there will be some changes to the ROCm TensorFlow fork in the future or ROCm drivers themselves that fix this performance to be more in line with the card’s actual power.

`resnet50` via `tf_cnn_benchmarks`

# Needed by my hacky TensorFlow setup
> export HSA_OVERRIDE_GFX_VERSION=11.0.0
> git clone https://github.com/tensorflow/benchmarks.git
> cd benchmarks/scripts/tf_cnn_benchmarks

# I edited the `tf_cnn_benchmarks.py` file add the bit of code to
# enable GPU memory growth so that my computer doesn't entirely
# lock up while it's running

> python3 tf_cnn_benchmarks.py --num_gpus=1 --batch_size=128 --model=resnet50

... debug output ...

Running warm up
Done warm up
Step    Img/sec total_loss
1       images/sec: 126.3 +/- 0.0 (jitter = 0.0)        7.442
10      images/sec: 126.3 +/- 0.1 (jitter = 0.3)        7.423
20      images/sec: 126.1 +/- 0.1 (jitter = 0.3)        7.469
30      images/sec: 125.5 +/- 0.4 (jitter = 0.2)        7.568
40      images/sec: 125.7 +/- 0.3 (jitter = 0.2)        7.515
50      images/sec: 125.8 +/- 0.3 (jitter = 0.3)        7.452

So, around 126 images/sec for resnet50. A Reddit thread from 4 years ago that ran the same benchmark on a Radeon VII - a >4-year-old card with 13.4 TFLOPS FP32 performance - resulted in a score of 147 back then.

This leads me to believe that there’s a software issue at some point. Maybe it’s my janky TensorFlow setup, maybe it’s poor ROCm/driver support for the 7900 XTX, or maybe it’s some some obscure boot param I added to my system 3 years ago. I really don’t know.

One thing is clear though: My TensorFlow performance is not anywhere near where it should be for this hardware.

Benchmarking 7900 XTX Raw FP32 FLOPS

After that bad result, I wanted to see if I could actually reach the 61 TFLOPS of FP32 performance advertised for the card.

While poking around online, I discovered the tinygrad library. It’s a minimalist ML framework built from the ground up on a very tiny foundation of basic operations. That being said, it’s still quite capable and is able to run full-scale complex networks like Stable Diffusion natively.

Tinygrad targets AMD GPUs as one of their backends. They support both OpenCL-based kernels as well as a work-in-progress RDNA3 Assembler backend.

The native RDNA3 backend was very interesting to me. Pretty recently after the 7900 XTX was released, Chips and Cheese put out a detailed microbenchmarking post for the 7900 XTX that used OpenCL to test various different microarchitectural properties of the card and raw performance numbers. One thing they noted was that the OpenCL compiler at the time was doing a poor job of making use of the “dual issue” mode of RRDNA3 to execute multiple instructions in parallel:

“I’m guessing RDNA 3’s dual issue mode will have limited impact. It relies heavily on the compiler to find VOPD possibilities, and compilers are frustratingly stupid at seeing very simple optimizations.”

Using dual issue, RDNA 3 GPUs like the 7900 XTX are able to kick off two v_dual_fmac_f32 (fused multiply-accumulate) instructions at once. That results in assembly that looks like this:

v_dual_fmac_f32 v108, v109, v110 :: v_dual_fmac_f32 v111, v112, v113

Since each of these fmac instructions results in 2 floating point operations (multiply + add) and two of them are dispatched per cycle, that comes out to 4 FLOPS of throughput per cycle. And since GPUs are SIMD machines, many of these instructions are executed in parallel across the GPU at the same time.

Tinygrad RDNA3 Matrix Multiplication Benchmark

I don’t remember how I found it, but Tinygrad has a benchmark script for raw RDNA3 floating point throughput.

It’s about as low-level as it gets. There’s a raw assembly file that has some boilerplate and loop handling code, and then the script dynamically generates the loop body out of nothing but v_dual_fmac_f32 :: v_dual_fmac_f32 instructions. There is no data loading or memory interaction at all; all operations read from and write to registers directly. This is far from anything “real-world” of course, but since the goal is to determine the max F32 throughput it’s perfect.

Somewhat miraculously, I was able to run the script as-is on my own 7900 XTX directly. Here’s what it output:

> # Exclude iGPU from being used by OpenCL; run code on the 7900 XTX only
> export CL_EXCLUDE=gfx1036
> DEBUGCL=5 DEBUG=5 GPU=1 OPTLOCAL=1 PRINT_KERNEL=1 python3 asm.py

... lots of debug info and disassembly ...

ran in 77.72 ms, 56663.33 GFLOPS
ran in 73.30 ms, 60082.85 GFLOPS
ran in 71.29 ms, 61775.70 GFLOPS

Well there it is! After a couple warmup iterations, it’s able to hit the 61 TFLOPS mark.

As I said before, this is far from any kind of a real-world benchmark. Memory is usually the bottleneck for any real model. However, this does validate the claims of peak theoretical floating-point throughput.

16-bit Floating Point

After seeing that result, I looked into testing the FP16 support as well. It turns out that the only way to achieve that massive 120 TFLOPS of FP16 advertised is by using the tensor/“AI” cores. These are specialized for matrix multiplication using FP16, BF16, IU8, and IU4 data types.

They can be interacted with via WMMA instructions and compiler intrinsics. So, benchmarking FP16 on the 7900 XTX would require me to write some assembly code.

At the time of writing this, that’s not something I want to devote the (possibly copious amounts of) time into right now, so I’ll put that in the future work pile.

Porting the TensorFlow benchmark to TinyGrad

Now TensorFlow seems to be the weak link, I wanted to see if I could replicate the simple neural network training example with Tinygrad. There is great interest in that community to support AMD cards natively and give them a more first-class-like experience for machine learning, so I figured it might be able to offer better performance.

I did my best to port the TensorFlow code over to Tinygrad directly. Tinygrad does have support for everything needed by the benchmark (which is a good thing since it’s so simple).

When I ran it, though, the Tinygrad port was very slow - around ~2x slower than the TensorFlow version.

After researching the project and learning more about it, I created an improved version which uses Tinygrad’s built-in JIT compilation to speed it up. After tweaking and tuning that a bit, I eventually got a version that beat TensorFlow by ~10%. However, that really did require me to do a lot of experimentation and delving into the Tinygrad internals and source code.

There’s a lot of heuristics and magic going on that greatly impacts the performance of Tinygrad. I think that if I spent a similar amount of time modifying the TF benchmark and tuning it similarly, I’d be able to get that or more of a boost as well.

Also, Tinygrad changes dramatically from day to day. When I re-ran the benchmark a week later without changing the benchmark code at all, performance had again regressed significantly to the point where it’s many times slower than TensorFlow.

I also tried out the RDNA3 Assembler backend. There are known limitations in the OpenCL compiler for AMD used by the default Tinygrad backend, so it’s possible that generating RDNA3 assembly code directly could yield results much closer to theoretical maximums.

This backend is far from complete, though, and failed to compile the small benchmark example. I went as far as to put up a PR myself to Tinygrad to try to extend its functionality enough to compile the benchmark. However, I hit another roadblock that was too complex for me to implement myself, so I gave up on that work.

The takeaway is that even if Tinygrad is capable of beating Tensorflow sometimes, there are a good deal of quirks and issues that make it difficult to recommend right now. That being said, development on the project really is proceeding at an immense pace, and I wouldn’t be surprised if in a few months it’s able to handily beat TensorFlow on AMD GPUs across a majority of benchmarks.

Conclusion

There’s obviously a lot to be desired for machine learning on AMD GPUs at the current point in time. A couple days ago, ROCm 5.6 has been released. I’ve not been able to install it and try it out yet, but it’s possible it might bring some performance improvements.

AMD has also said that they plan on adding official RDNA3 support to ROCm by this fall of 2023. Since RDNA3 isn’t even technically supported by ROCm right now, I feel like there’s definitely a ton of room for improvement on that side of things and a lot of hope things will get better.

I’ll continue to keep an eye on this space until then! `

Fixing GCS REST API Error "Your client has issued a malformed or illegal request"

Sat, 01 Jul 2023 09:39:30 -0700

We ran into this error at my job at Osmos. We upload files to GCS using their JSON-based REST API. Everything was working just fine until we tried uploading a large-ish file of ~2.5GB.

We upload to this API route: https://storage.googleapis.com/upload/storage/v1/b/bucket-name/o?name=file_name.csv&uploadType=media

The Problem

When we tried to upload the data, we got a HTML page as a response with a 400 error code and this unhelpful error message:

400. That's an error.

Your client has issued a malformed or illegal request.  That's all we know.

The Fix

We were about to fix the issue by adding a Content-Length header to the request. It seems that for some requests, it isn’t necessary, but it is for others.

The interface that we use to make this request the reqwest library from Rust to make the HTTP request. We use a Rust stream as the body, which may contribute to the issue.

For this particular case, we do know the size of the data ahead of time so we can provide the Content-Length header accurately. However, for cases where large amounts of data needs to be uploaded to GCS as a stream without a known size ahead of time, I’m not sure what the solution is.

Fix for League of Legends Lutris Mesa DRI Driver "Not From This Mesa Build" Error

Sun, 18 Jun 2023 23:22:46 -0700

The Problem

I recently updated the packages on my Debian Linux install with sudo apt upgrade.

After that, I rebooted and tried to launch League of Legends through Lutris as I have hundreds of times. The client failed to launch with this error printed in the logs:

DRI driver not from this Mesa build ('23.1.0-devel' vs '23.1.2-1')

The Cause

I recently installed AMD ROCm using amdgpu-install so I could do some machine learning with my AMD 7900 XTX GPU.

I installed amdgpu-install version 5.5.50500-1. This seems to force a specific version of some part of Mesa that’s incompatible with the one needed by Lutris, or Wine, or whatever other layer of the arcane Linux Graphics Stack is causing this issue.

The Fix

I uninstalled ROCm by running:

sudo amdgpu-install --uninstall

I then rebooted, and now I’m back tearing it up on the rift.

I tried installing a newer version of amdgpu-install, but that failed with dozens of dependency version conflicts, so it doesn’t seem like a viable fix right now. Hopefully one or the other will be updated to make the versions compatible in the future.

I will re-install ROCm tomorrow when I get in the mood to be a productive human again.

Fix for AMD OpenCL Devices Showing Up as the Same Name

Sun, 18 Jun 2023 14:10:17 -0700

I’ve been experimenting with OpenCL via pyopencl recently. They provide a nice interface for enumerating available devices and getting information about them, and then using them to run OpenCL code:

>>> import pyopencl as cl
>>> platform = cl.get_platforms()[0]
>>> platform.get_devices()
[
  <pyopencl.Device 'gfx1100' on 'AMD Accelerated Parallel Processing' at 0x56353125bd70>,
  <pyopencl.Device 'gfx1036' on 'AMD Accelerated Parallel Processing' at 0x5635312ec670>
]
>>>

There are two devices for me because I have one discrete 7900 XTX GPU as well as an integrated GPU on my 7950X CPU. gfx1100 is the Shader ISA for the 7900 XTX, and gfx1036 is for the iGPU.

The Problem

At some point, pyopencl started returning gfx1100 as the name for both of my devices:

>>> import pyopencl as cl
>>> platform = cl.get_platforms()[0]
>>> platform.get_devices()
[
  <pyopencl.Device 'gfx1100' on 'AMD Accelerated Parallel Processing' at 0x555cc5a8ed60>,
  <pyopencl.Device 'gfx1100' on 'AMD Accelerated Parallel Processing' at 0x555cc5b1f5f0>
]
>>> [d.hashable_model_and_version_identifier for d in platform.get_devices()]
[
  ('v1', 'Advanced Micro Devices, Inc.', 4098, 'gfx1100', 'OpenCL 2.0 '),
  ('v1', 'Advanced Micro Devices, Inc.', 4098, 'gfx1100', 'OpenCL 2.0 ')
]
>>>

There was no way to tell them apart. I needed my code to only run on my 7900 XTX and not on the iGPU, and this caused the filtering my code was doing on device.name to not work since I had no way to tell the devices apart.

The Cause + The Fix

It turns out that I had exported the environment variable HSA_OVERRIDE_GFX_VERSION=11.0.0. I did this to support building a custom TensorFlow package that works on the 7900 XTX with ROCm 5.5.

It turns out that setting that environment variable causes some driver in the stack to pretend that all the devices support gfx1100 and changes their names as well.

For this OpenCL code, I wasn’t using TensorFlow and so didn’t need to export that anymore. I expect that in the near future, that hack won’t be necessary anymore anyway.

unsetting that environment variable fixed the problem, and my cards are now properly identified by OpenCL again.

Setting Up TensorFlow with ROCm on the 7900 XTX

Fri, 09 Jun 2023 13:13:58 -0700

I recently upgraded to a 7900 XTX GPU. The upgrade itself went quite smoothly from both a hardware and software perspective. Games worked great out of the box with no driver or other configuration needed - as plug and play as it could possibly get.

However, I wanted to try out some machine learning on it. I’d been using TensorFlow.JS to train models using my GPU all in the browser, but that approach is limited compared to what’s possible when running it natively.

Since TensorFlow is the framework I’m most familiar with, it’s what I wanted to focus on getting working first.

AMD’s provided method for doing this is called ROCm. It’s a combination of mostly-open libraries, toolkits, frameworks, compilers, and other software to facilitate heterogenous/GPGPU compute on AMD hardware.

Installing ROCm

This part of the process took me the longest to figure out, but it ended up being quite simple. AMD provides a tool called amdgpu-install which handles installing ROCm and other AMD software and drivers.

ROCm version 5.5 is the most recent version available at the time of release. It contains improvements for RX 7000 series / RDNA3 GPU support which includes the 7900 XTX. Version 5.6 is in the works as well, and there are rumors it will bring even more mature RDNA3 support and hopefully better performance as well.

I installed amdgpu-install by running curl https://repo.radeon.com/amdgpu-install/5.5/ubuntu/jammy/amdgpu-install_5.5.50500-1_all.deb > /tmp/amdgpu-install.deb && sudo dpkg -i /tmp/amdgpu-install.deb.

The ROCm install tutorial says to run amdgpu-install --usecase=rocm. For me, this ended up causing a lot of issues.

The reason for that is that by default, that command attempts to build and install the amdgpu-dkim kernel module. I don’t understand how/if that differs from the amdgpu kernel module that comes built-into the Linux kernel already.

Anyway, the build for that kernel module failed for me because of differences in the kernel version targeted by ROCm 5.5 and my current kernel (Linux 6.3).

It turns out that building the kernel module isn’t necessary at all to get ROCm installed. Running sudo amdgpu-install --usecase=rocm --no-dkms worked for me.

I followed the rest of the instructions setting some user groups and rebooting, and things seemed to be in good shape.

Building 7900 XTX-Compatible TensorFlow

The next step was building a custom TensorFlow that works with ROCm version 5.5 and the 7900 XTX.

AMD maintains a TensorFlow fork for this, but at the time of writing this (June 9, 2023) it’s not yet updated for ROCm 5.5. So, we have to compile our own.

Luckily, some other people have done the vast majority of the hard work already!

@Mushoz posted a comment with a Dockerfile and instructions for handling this whole process. It patches TensorFlow to work with gfx1100 (used by the 7900 XTX), compiles it all, and produces a Docker image as output.

I followed their instructions exactly and was able to get it to work with no issues.

When I went to test TensorFlow’s GPU functionality from inside the container, though, I ran into some more problems:

>>> import tensorflow as tf
>>> tf.test.is_gpu_available()
WARNING:tensorflow:From <stdin>:1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2023-06-09 09:16:04.455832: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-09 09:16:04.456591: E tensorflow/compiler/xla/stream_executor/rocm/rocm_driver.cc:302] failed call to hipInit: HIP_ERROR_InvalidDevice
2023-06-09 09:16:04.456599: I tensorflow/compiler/xla/stream_executor/rocm/rocm_diagnostics.cc:112] retrieving ROCM diagnostic information for host: devitra
2023-06-09 09:16:04.456602: I tensorflow/compiler/xla/stream_executor/rocm/rocm_diagnostics.cc:119] hostname: devitra
2023-06-09 09:16:04.456631: I tensorflow/compiler/xla/stream_executor/rocm/rocm_diagnostics.cc:142] librocm reported version is: NOT_FOUND: was unable to find librocm.so DSO loaded into this program
2023-06-09 09:16:04.456635: I tensorflow/compiler/xla/stream_executor/rocm/rocm_diagnostics.cc:146] kernel reported version is: UNIMPLEMENTED: kernel reported driver version not implemented
False
>>>

This seemed to indicate some kind of driver version mismatch or other issue. I still don’t know what the exact issue was (Mushoz didn’t seem to run into this), but I was able to work around it.

The workaround involves copying the built TensorFlow Python wheel out of the container and running it directly on my host inside a virtual environment. Here’s how I accomplished that:

On host:

chmod 777 ~/dockerx

In container:

cp /tmp/tensorflow_pkg/tensorflow_rocm-2.11.1.-cp310-cp310-linux_x86_64.whl ~/dockerx/

Back on host:

pyenv install 3.10.6
pyenv virtualenv 3.10.6 tf-test
pyenv activate tf-test
pip3 install ~/dockerx/tensorflow_rocm-2.11.1.-cp310-cp310-linux_x86_64.whl
export HSA_OVERRIDE_GFX_VERSION=11.0.0
python3

After doing that and re-running the same Python commands to test TensorFlow’s GPU integration, it properly detected my GPU as supported. Being able to do all of that directly on the host is actually much better for me as well since I can use that virtual environment directly for notebooks from VS Code.

I did run into segfaults from inside libamdhip64.so.5 when I actually tried to run some TensorFlow code using the GPU:

#0  0x00007fa1d3cd9d25 in ?? () from /opt/rocm-5.5.0/lib/libamdhip64.so.5
#1  0x00007fa1d3cd9ead in ?? () from /opt/rocm-5.5.0/lib/libamdhip64.so.5
#2  0x00007fa1d3cdda41 in ?? () from /opt/rocm-5.5.0/lib/libamdhip64.so.5
#3  0x00007fa1d3c8f9de in ?? () from /opt/rocm-5.5.0/lib/libamdhip64.so.5
#4  0x00007fa1d3e404ab in ?? () from /opt/rocm-5.5.0/lib/libamdhip64.so.5
#5  0x00007fa1d3e117f6 in ?? () from /opt/rocm-5.5.0/lib/libamdhip64.so.5
#6  0x00007fa1d3e1db86 in hipLaunchKernel () from /opt/rocm-5.5.0/lib/libamdhip64.so.5
#7  0x00007fa1b7197496 in Eigen::internal::TensorExecutor<Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<float, 1, 1, int>, 16, Eigen::MakePointer>, Eigen::TensorCwiseNullaryOp<Eigen::internal::scalar_const_op<float>, Eigen::TensorMap<Eigen::Tensor<float, 1, 1, int>, 16, Eigen::MakePointer> const> const> const, Eigen::GpuDevice, true, (Eigen::internal::TiledEvaluation)0>::run(Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<float, 1, 1, int>, 16, Eigen::MakePointer>, Eigen::TensorCwiseNullaryOp<Eigen::internal::scalar_const_op<float>, Eigen::TensorMap<Eigen::Tensor<float, 1, 1, int>, 16, Eigen::MakePointer> const> const> const&, Eigen::GpuDevice const&) () from /home/casey/.pyenv/versions/tf-test/lib/python3.10/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#8  0x00007fa1b7192a51 in tensorflow::functor::FillFunctor<Eigen::GpuDevice, float>::operator()(Eigen::GpuDevice const&, Eigen::TensorMap<Eigen::Tensor<float, 1, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::TensorFixedSize<float const, Eigen::Sizes<>, 1, long>, 16, Eigen::MakePointer>) () from /home/casey/.pyenv/versions/tf-test/lib/python3.10/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so

The fix turned out to be adding that export HSA_OVERRIDE_GFX_VERSION=11.0.0 line. I learned that this is needed from a blog post about running Stable Diffusion on the 7900 XTX. The author of that writes:

HSA_OVERRIDE_GFX_VERSION defaults to 10.3.0 and will fail our gfx1100 if we don’t set it explicitly

There are a lot of other good tips and troubleshooting info for running PyTorch and other ML libraries on the 7900 XTX - I highly suggest checking it out if you run into any other problems.

That was the final piece I needed. I am now able to use full GPU-accelerated TensorFlow with my 7900 XTX GPU:

>>> import tensorflow as tf
2023-06-09 17:05:27.588883: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-09 17:05:27.654869: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
>>> tf.add(tf.ones([2,2]), tf.ones([2,2])).numpy()
2023-06-09 17:05:32.378116: I tensorflow/compiler/xla/stream_executor/rocm/rocm_gpu_executor.cc:843] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-06-09 17:05:32.378152: I tensorflow/compiler/xla/stream_executor/rocm/rocm_gpu_executor.cc:843] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-06-09 17:05:32.385399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2006] Ignoring visible gpu device (device: 1, name: AMD Radeon Graphics, pci bus id: 0000:1a:00.0) with core count: 1. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
2023-06-09 17:05:32.385608: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-09 17:05:32.386372: I tensorflow/compiler/xla/stream_executor/rocm/rocm_gpu_executor.cc:843] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-06-09 17:05:32.386509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 23986 MB memory:  -> device: 0, name: Radeon RX 7900 XTX, pci bus id: 0000:03:00.0
2023-06-09 17:05:32.506584: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
2023-06-09 17:05:37.839580: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
array([[2., 2.],
       [2., 2.]], dtype=float32)
>>>

Limiting TensorFlow Memory Usage

One final thing I ran into when testing out TensorFlow on the 7900 XTX was that my desktop was basically unusable while TensorFlow was running. Clicks took many seconds to go through, windows would freeze, etc.

This is because TensorFlow allocates all of the GPU’s memory by default, leaving nothing for the desktop environment and any other apps to use.

Luckily, the fix is easy. Just add this Python code to the start of the script/notebook before any other TensorFlow code is run:

import tensorflow as tf

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  try:
    for gpu in gpus:
      tf.config.experimental.set_memory_growth(gpu, True)
  except RuntimeError as e:
    print(e)

This is from a StackOverflow answer that Mushoz linked in that same Github issue from before. Huge thanks again to him for all the help getting this working!

Conclusion

This whole process was quite difficult. It took me more than a full day of effort.

I think/hope that this will be getting much better soon. Once the TensorFlow ROCm fork gains true ROCm 5.5 or better yet 5.6 support, none of that custom build process should be necessary and should be installable directly from a Python package registry.

That being said, machine learning on modern AMD hardware is very much possible. Once the setup is out of the way, TensorFlow seems to be working just like normal and running existing code with no modifications needed.

Fix for Broken Boot After Failed amdgpu-dkim Install

Thu, 08 Jun 2023 17:35:41 -0700

I recently upgraded to the 7900 XTX GPU which was a totally issue-free experience. Then today, I tried to install AMD ROCm so I could try out AMD’s TensorFlow fork that works with AMD GPUs.

I ran into a lot of issues with this that resulted in my computer not being able to boot for a while. I eventually figured it out, but it was quite a struggle.

It started after I downloaded and ran amdgpu-install - AMD’s tool for installing drivers and other software for use with their hardware.

I ran a variety of different commands with that - sudo amdgpu-install --usecase=rocm, sudo amdgpu-install --uninstall, sudo amdgpu-install --usecase=graphics,rocm through different stages of debugging stuff.

The install itself failed because the kernel version needed by the amdgpu-dkim component of ROCm (5.x) was different than my Kernel version (6.3) so the module build failed. amdgpu-dkim is a kernel module for amdgpu, and I didn’t and still don’t really understand how or if it differs from the amdgpu kernel module that comes built-in to the kernel.

Symptoms

At some point, I rebooted my computer. When I tried to reboot, the boot hung at the dmesg output which is displayed before my desktop environment pops up. I looked into a few red herring errors in the logs that turned out to have nothing to do with the failure to boot.

I eventually figured out a way to get the boot to work: Pressing the “e” key on the grub menu option and adding nomodeset to the list of boot args.

Although it did boot and most things worked alright, it was clear that nothing was GPU accelerated. Only one of my three monitors worked, Xorg had high CPU usage since it was clearly not accelerating anything with the GPU, and glxgears was running with software rasterizer.

The Cause

At some point, I ran lsmod | grep gpu to try to figure out if maybe there was some weird alternate kernel module running that was dropped by amdgpu-install which was conflicting with the kernel’s built-in one.

However, what I saw was that there were no kernel modules at all for amdgpu.

The Fix

After a good bit of googling, I found a blog post written in Japanese which talks about this exact situation.

It turns out that when the amdgpu-dkim kernel module build fails, a file /etc/modprobe.d/blacklist-amdgpu.conf will get created. This results in the amdgpu kernel module getting forced to not load during boot and results in the boot failing (unless the nomodeset boot param is set).

After deleting that file, the computer booted normally.

I’ve given up on getting ROCm working for now, but might give it another go in the future. I have a hope that it will maybe work without installing the kernel module which caused all of these issues, but we’ll see!

Upgrading from a 5700 XT to 7900 XTX on Linux

Wed, 07 Jun 2023 22:06:45 -0700

Just today, I switched to the 7900 XTX GPU. I mostly just wanted an upgrade, but I also secretly hoped it would fix a lot of the weird GPU-related issues I’ve had over the past years.

The 5700 XT is a rather buggy GPU as far as I can tell - especially on Linux which is my only OS on my desktop. I’ve run into multiple bugs with drivers and other mysterious green-screen crashes:

I opted to go for another AMD card rather than switch to NVIDIA despite the crashes. I had decent reason to believe that the crashes were mostly limited to teh 5700 family cards, and I hoped that the 9000 series would be safe. The card is quite close in performance to the comparable NVIDIA card, but a couple hundred dollars cheaper.

Results

This turned out to be the easiest ~~and most successful~~ GPU upgrades I’ve ever done. ~~No issues whatsoever so far.~~

~~There were zero issues through the whole thing. I actually replaced my computer’s PSU as well with a 1000 watt unit, and even with that it all went perfectly.~~

There were zero software changes I needed to do. I rebooted it after installing the new card and everything worked perfectly. No driver issues at all, ~~no crashes so far~~, 10/10.

My radeontop doesn’t know what model the card is, but it works just fine for measuring its utilization.

So yeah - here’s to hoping the driver bugs and crashes don’t come back at some point in the future, but I’m very happy with everything so far.

EDIT 2023-06-18:

I’ve been getting blackscreen crashes and kernel panics. They mostly tend to happen when I’m AFK, but still very annoying and an issue for sure.

One is a very specific issue that seems to happen only if you have >=2 monitors plugged in that have significant differences between them in frame rate and/or resolution: https://gitlab.freedesktop.org/drm/amd/-/issues/2609

The black-screen no-log full computer reboots are still happening as well. No idea what’s causing them.

EDIT 2023-08-15:

Well, I did a few sudo apt full-upgrades and reboots later, and… my crashes just stopped!

I don’t know what happened, but for the past ~month I’ve not had a single driver crash even with heavy gaming in an 85 degree Fahrenheit apartment.

My Thoughts on the uPlot Charting Library

Tue, 06 Jun 2023 11:50:23 -0700

TL;DR:

uPlot is a Spartan charting library that focuses intensely on minimalism and performance. It feels very much like a tool made for hackers, and it lacks many of the features and embellishments of fully-featured charting libraries.

The main downside is that it has quite terrible docs and sometimes has confusing APIs

I’m personally a big fan of its aesthetic and design goals, and I will probably be sticking with it as my primary charting library for the web for the forseeable future.

A lot of my projects end up needing to render some kind of chart in the browser, often using dynamic data. Over the years, I’ve explored a range of different charting libraries. I started off using Highcharts, then ECharts was my favorite for a few years, and most recently my go-to choice is uPlot.

uPlot self-describes as “A small, fast chart for time series, lines, areas, ohlc & bars”

It’s certainly a very minimal charting library compared to complex offerings like ECharts, but it more than makes up for that. Its bundle size is under 50KB for one which is really impressive. Compare that to the ~1MB bundle size for ECharts.

That being said, I can’t think of a time I’ve hit a wall with uPlot because it was lacking some feature I needed. I feel that it covers >95% of the surface area for common and uncommon charting needs for the chart types it supports.

It’s also intensely focused on performance. It renders to a canvas using WebGL which makes it possible to scale way further than libraries like D3 - which render to SVG - can ever reach.

uPlot is optimized to both have an extremely fast initial render as well as scale to supporting huge amounts of data points at 60FPS+. It is seriously fast - here’s one of their demos that effortlessly renders millions of data points: https://leeoniya.github.io/uPlot/bench/uPlot-10M.html

Its UI design is quite minimalistic, and that’s certainly their goal. The library explicitly avoids things like fancy animations when rendering things in or design flourishes in favor of minimizing latency and responsiveness. Here’s an example of the kind of line plot you will get out of it with more or less default settings for things:

Personally, I’m a big fan of its aesthetic. It fits well with a lot of the kinds of web apps I end up building, and there’s very little work necessary configuring styles or theming. For comparison, bigger libraries like Highcharts or ECharts often require me to set broader theming settings in order to make the plots look decent on a dark background.

With uPlot, there’s simply less things to style, so I end up only really having to define series colors - if even that.

Drawbacks

The main downside that uPlot has is that the docs are not good at all.

Pretty much the only docs that actually exist consist of one small Markdown doc. Other than that, you have to rely on reverse engineering the examples on their website or trying to figure things out based on the TypeScript type definitions (which are quite complete to be fair).

This is a real annoyance, though, and probably makes the library very unattractive to certain people. Setting up a basic chart is unsurprisingly very simple, but as soon as you want to do something like add a custom label formatter, tweak the tick spacing behavior, or use a non-standard line drawing mode, it’s a slog of finding an applicable example and reverse engineering it every time.

Compare that to the ECharts Docs. They’re extremely complete, very easy to search + browse, and have examples and codepens for pretty much every possible use case.

Conclusion

As I’ve said before, I really like uPlot and I use it all the time. It’s the first choice I reach for in all my new projects, and I’m largely very happy with it. For the kinds of things I build, I have no need of the vast majority of the features that bigger libraries provide, and I’m happy to pay the tradeoff in exchange for the extremely tiny bundle size and great performance.

I really wish the docs were better, though. I’ve burned many hours digging through examples and trying to cobble together functionality from “View Source” in Chrome, and it feels like a big waste of time.

Using a Ramdisk for Rust Dev

Sun, 04 Jun 2023 00:43:49 -0700

The main Rust workspace for my job at Osmos is very large. It has several thousand dependencies, does copious compile time codegen from gRPC protobuf definitions, and makes extensive use of macros from crates like serde, async-stream, and many others.

While it’s really convenient having all of our code in one place, this results in a lot of work being done by the Rust compiler as well as rust-analyzer during normal development. For the most part, this isn’t too much of an issue.

I’m lucky to have very good hardware for the computer I develop on, but even so compile times and editor features like go to definition can be quite slow at times.

Another thing I worry about is wear on my SSD from the large amount of reads and writes that occur. The generated intermediate compilation artifacts and output binaries are very large. Just opening the project up in VS code and saving a file results in ~2.5GB of writes:

I decided to try out using a Ramdisk to speed things up and maybe reduce the amount of disk activity incurred during Rust development. Luckily, there’s a tool that exists which makes it easy called cargo-ramdisk. Installing it was as simple as running cargo install cargo-ramdisk, and activating it for a project is as easy as running cargo ramdisk mount.

Results

On the plus side, cargo-ramdisk did indeed work to pretty much eliminate disk reads and writes as a result of compilation and development. Unfortunately, that was pretty much the only advantage for my setup.

There was very minimal reduction in compile time - on the order of a few percent even when doing a full clean cargo check of the project. I do have a very fast SSD, so this may be different for other people.

Another downside is that the build cache is lost every time I restart my computer, so I end up having to wait for all the dependencies to re-compile from scratch whenever I start developing again after a reboot.

The final issue I ran into was the ramdisk getting so big that my computer starts to run out of memory. This only happened after having the project open for a few consecutive workdays, but it did end up being a problem a few times.

Conclusion

After trying it out for a couple of weeks, I ended up dropping the ramdisk from my development workflow. For my specific case, it created more hassle than it solved. Maybe I’ll regret that, though, if my SSD fails in a year or two!

If you have a less-than-amazing SSD and are struggling with high compile times or rust-analyzer functionality, cargo-ramdisk might be worth checking out.

Rust Tonic Request/Response Size Limit Footgun

Sat, 03 Jun 2023 18:26:33 -0700

I recently encountered a bug in one of our services at my job at Osmos. Our service is written in Rust and connects to GCP PubSub via its gRPC interface.

We were running into errors in our logs like this:

Error, message length too large: found 5360866 bytes, the limit is: 4194304 bytes

The service in question had been running for over 2 years without seeing this issue before, and the message size limits shown are smaller than the PubSub message size cap of 10MB.

Root Cause

After a good bit of investigation, we discovered that the error message was actually coming from inside tonic.

tonic is a popular Rust gRPC client and server. We use it extensively, including for the GCP PubSub client.

It turns out that we’d recently upgraded to tonic version 0.9. This release included a breaking change that adds size limits of 4MB by default when decoding messages.

This is what was causing the bug. As far as I can tell, these limits are in place to prevent DOS attacks on gRPC servers, so it’s probably a good idea to have them. Before, the limits were either very high or nonexistent, so we never ran into this issue in the past.

tonic is still on major version 0, so as per semver, minor version bumps can introduce breaking changes. tonic did also include this in their changelog, so they didn’t do anything wrong in this case in my opinion.

That being said, it was a very unexpected break and I figured I’d write this up to maybe show up in search results for other people hitting this issue.

The Fix

The fix was simple - just specify a higher limits for message decoding and encoding:

SubscriberClient::new(auth_service)
  .accept_compressed(CompressionEncoding::Gzip)
  // added these two following lines
  .max_decoding_message_size(256 * 1024 * 1024)
  .max_encoding_message_size(256 * 1024 * 1024)

Conclusion

This bug actually made me realize that things have gotten a lot more stable in the Rust async ecosystem since I started using it extensively ~3 years ago.

There used to be lots of showstopping bugs, annoying incompatibilities, and stuff like that. Now, things are stable to the point where bugs like this are out of the ordinary - and this bug wasn’t even technically tonic’s fault.

My Self-Hosted Ngrok Alternative

Sat, 03 Jun 2023 15:03:59 -0700

I often need to expose some service running locally on my computer to the public internet for some reason or another. Demoing a website, exposing an API, giving someone access to download some local files, stuff like that.

The popular solution for this is tools like ngrok. You download their CLI application to your computer, specify a port, and your local service is available at a URL like https://a78f837.ngrok.io/.

The downside of this is that there are limits for the free versions of these services. You’ll get limits on session duration, get a random subdomain every time, have network speed limits, or other stuff. You can pay for them as well to make most of these limits go away, of course, and that’s what I used to do with an alternative service called <tunnelto.dev>.

However, I ended up encountering many problems with the quality of their service, and I ended up giving up on that as well.

Self-Hosted Alternative

Since the core concept of these services seems so simple, I figured it wouldn’t be too hard to self host my own.

My setup uses a tool called frp. frp described as:

A fast reverse proxy to help you expose a local server behind a NAT or firewall to the internet.

Exactly what I was looking for.

Server Configuration

Setting it up turned out to be pretty simple. I have a dedicated server on which I host all my websites and services, and I set up an instance of frpc there via Docker:

docker run -d --name frps \
  --restart=always \
  -p 5100:7000 \
  -p 5101:7500 \
  -p 5102:6000 \
  -p 5103:9888 \
  -v /opt/conf/frp/frps.ini:/etc/frp/frps.ini \
  snowdreamtech/frps:0.37.1

The required config for that in frps.ini is very minimal:

[common]
bind_port = 7000
vhost_http_port = 9888
authentication_method = token
token = <secret token>

The only other part on the server I needed to set up was adding some config to my main NGINX instance to create a reverse proxy to the frps server’s port:

server {
    server_name subdomain-name.yoursite.com

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://localhost:5103/;
    }

    # <letsencrypt, QUIC, and other common config>
}

Client Configuration

Setting up the client was even easier. All I had to do was download the latest release of the client portion of frp called frpc and create another tiny config file for it:

[common]
server_addr = <address of server running `frps`>
server_port = 5100
authentication_method = token
token = <same secret token as the server>

[web]
type = http
local_port = 4298
custom_domains = localhost

Whenever I want to expose a local service, I edit that config file (which I put at ~/frpc.ini) and set the local_port to whatever port the service is running on.

Then, I start frpc using this command:

frpc -c ~/frpc.ini

That’s all there is to it! The service will be proxied to the subdomain and available publicly.

Definitely a bit more complex than just paying ngrok or someone else to do it, but this way you get full control over the server, the subdomain, and the whole setup.

Tunnelto Woes

Fri, 02 Jun 2023 16:15:46 -0700

There’s a service https://tunnelto.dev/ which I’ve used in the past. It’s an ngrok alternative for exposing local services publically for things like demos, testing, etc.

I heard about tunnelto.dev when it was announced on Hacker News and gave it a try along with several others on my team at Osmos. It looked fresh, was written Rust, and was at least partially open source - all great. I even signed up for the (very cheap) paid plan which gives custom subdomains and some other stuff.

Issues

I really wanted to like this service and use it, but it’s been a source of constant issues. It’s a highly unreliable service as far as I can tell, and that hasn’t really improved over the past ~3 years we’ve been using it. I kind of feel bad writing this, but I think it’s important to say at this point:

If you’re having issues with tunnelto.dev (incredibly slow responses, networking issues, weird jank, etc.) it’s probably not you, and tunnelto (or one of tunnelto’s hosts) is to blame.

I’ve emailed the dev behind tunnelto multiple times in the past when I was encountering issues with their service. I did get a response, saying that they were related to an upstream provider issue.

I know that tunnelto.dev is deployed on fly.io, and fly.io has been experiencing some bad growing pains and having a lot of difficulty scaling their service. I think there’s a very good chance a lot of the issues tunnelto is experiencing can be attributed to fly.io, but I have no way of being sure.

Anyway, the point of this post wasn’t to flame tunnelto or fly.io or anything, but more to put some written record that these issues with tunnelto do exist for anyone else that might be running into them.

I’ve not used tunnelto myself in over a year, and with this latest round of issues I’ve been helping my coworkers debug I think they’re all going to be switching off as well.

As an alternative, I’m self-hosting an instance of [frp] on my personal VPS and using that for all my tunneling needs. It’s pretty simple to set up, but definitely more work than an off-the-shelf solution like ngrok or tunnelto.

Maybe I’ll write something up about my setup for that in the future.

Drawing Graphviz Edge Splines

Wed, 31 May 2023 21:32:17 -0700

Background

I recently had the need to build a custom renderer for graphs generated by graphviz for a project visualizing computation graphs for RNNs.

Graphviz has an output format called plain and plain-ext which generates a very simple text-based layout specification for all nodes and edges in the input graph. It’s pretty well spec’d out in the official docs for the most part: https://graphviz.org/docs/outputs/plain/.

Here’s an example of what plain-ext output format looks like for a simple graph:

graph 1 5.875 8.3333
node output_0 1.5694 0.47222 1.2604 0.5 output_0 solid ellipse black lightgrey
node layer_0_state_0 1.5694 4.7833 2.0049 0.5 layer_0_state_0 solid ellipse black lightgrey
node layer_0_state_1 3.8194 4.7833 2.0049 0.5 layer_0_state_1 solid ellipse black lightgrey
node layer_0_recurrent_0 1.5694 6.5444 2.4812 0.5 layer_0_recurrent_0 solid ellipse black lightgrey
node layer_0_recurrent_1 4.3056 6.5444 2.4812 0.5 layer_0_recurrent_1 solid ellipse black lightgrey
node layer_0_output_1 1.5694 3.0222 2.1989 0.5 layer_0_output_1 solid ellipse black lightgrey
node post_layer_output_0 1.5694 1.7056 2.5119 0.5 post_layer_output_0 solid ellipse black lightgrey
node input_0 4.3056 7.8611 1.1263 0.5 input_0 solid ellipse black lightgrey
edge layer_0_state_0 layer_0_output_1 4 1.5694 4.531 1.5694 4.2394 1.5694 3.7559 1.5694 3.415 -1 1.6504 3.9028 solid black
edge layer_0_state_1 layer_0_recurrent_0 4 3.5129 5.0233 3.1202 5.3307 2.4377 5.8649 1.9949 6.2114 1 2.8542 5.6639 solid black
edge layer_0_recurrent_0 layer_0_state_0 4 1.5694 6.2921 1.5694 6.0005 1.5694 5.517 1.5694 5.1761 1 1.6181 5.6639 solid black
edge layer_0_recurrent_1 layer_0_state_1 4 4.2359 6.2921 4.155 5.9992 4.0208 5.5127 3.9266 5.1715 1 4.1319 5.6639 solid black
edge layer_0_output_1 post_layer_output_0 4 1.5694 2.7684 1.5694 2.5789 1.5694 2.3165 1.5694 2.0999 -1 1.6504 2.3222 solid black
edge post_layer_output_0 output_0 4 1.5694 1.45 1.5694 1.2819 1.5694 1.0584 1.5694 0.8676 1 1.6181 1.0889 solid black
edge input_0 layer_0_recurrent_1 4 4.3056 7.6072 4.3056 7.4178 4.3056 7.1554 4.3056 6.9388 -1 4.3865 7.2444 solid black
stop

I used graphviz to consume the input graph in native graphviz dot language and then generate plain-ext output text.

Graphviz is an excellent library and largely does a great job creating good-looking graphs. One of the things it handles doing is drawing complex edges that twist and turn to avoid colliding with nodes or labels:

Here’s what the docs page says about the data provided for edge layout:

n is the number of control points defining the B-spline forming the edge. This is followed by 2*n numbers giving the x and y coordinates of the control points in order from tail to head.

I had trouble figuring out how to render these edges with PIXI.JS, the library I was using to build the visualization.

I did some research, and it seems that B-splines a.k.a. Basis Splines are relatives of bezier curves but with more power. They’re able to express curves that bezier curves can’t create on their own.

However, I knew that it was possible to render the edge paths generated by graphviz with just plain bezier curves since that’s what the generated SVGs were using for their paths.

I looked for other solutions, and I only really found one post on Stack Overflow from someone who was able to render them using curveBasis from D3. The D3 source code for that function only really has calls to bezierCurveTo, but they were doing weird things I didn’t understand to compute bezier control points from the spline control points.

Solution

After much more research and trial and error, I figured out how to do it. The list of control points generated by graphviz for each edge spline always totaled 3n + 1 which was a pretty big hint. The first point is the start point of the spline and the last is the end point.

It turns out that the “b-splines” used by graphviz here are quite simple and constrained and in fact are just composed out of a bunch of cubic bezier curves connected together end to end.

So, to render the edge path, all I had to do was move to the starting point of the edge, loop through the points in windows of 3, and draw a bezier curve for each set. The starting point of the next curve is the end point of the last one.

Here’s the code I ended up with to accomplish that:

const start = controlPoints[0];
graphics.moveTo(start.x, start.y);

const startPoint = controlPoints[0];
graphics.moveTo(startPoint.x, startPoint.y);

let p1: Point, p2: Point, p3: Point;
for (let i = 1; i < controlPoints.length; i += 3) {
  p1 = controlPoints[i];
  p2 = controlPoints[i + 1];
  p3 = controlPoints[i + 2];
  graphics.bezierCurveTo(p1.x, p1.y, p2.x, p2.y, p3.x, p3.y);
}

None of those fancy “knot vectors” or other crazy b-spline features are needed at all - just plain old cubic bezier curves.

The results look great and exactly match the SVG generated by graphviz natively!

Drawing Arrowheads

One additional piece that I wanted was arrowheads. My graphs are directed with lots of loops since they represent the computation graphs of RNNs, so knowing the direction of edges was important.

I accomplished this by computing the angle of the last spline at its endpoint where it touches the target node and rendering a triangle pointing in that same direction.

Here’s the code I wrote to do that:

// Computes the direction of the last bezier curve in the spline to determine
// the arrowhead direction
const dx = end.x - prev1.x;
const dy = end.y - prev1.y;
const angle = Math.atan2(dy, dx);

const point1 = {
  x:
    end.x -
    Conf.ArrowheadSize *
      Math.cos(angle - (Math.PI / 6) * Conf.ArrowheadHeightRatio),
  y:
    end.y -
    Conf.ArrowheadSize *
      Math.sin(angle - (Math.PI / 6) * Conf.ArrowheadHeightRatio),
};
const point2 = {
  x:
    end.x -
    Conf.ArrowheadSize *
      Math.cos(angle + (Math.PI / 6) * Conf.ArrowheadHeightRatio),
  y:
    end.y -
    Conf.ArrowheadSize *
      Math.sin(angle + (Math.PI / 6) * Conf.ArrowheadHeightRatio),
};

graphics.lineStyle(1, color);
graphics.beginFill(color);
graphics.drawPolygon([end.x, end.y, point1.x, point1.y, point2.x, point2.y]);
graphics.endFill();

It works well, and the arrowheads look good! Note that config option used to elongate the triangles for the arrowheads a bit so that they are longer since equilateral triangles looked too fat.

That’s all! I hope this ends up helping someone out doing something similar in the future.

Casey Primozic's Notes

WebGL `blitFramebuffer` Performance Issues on M3 Mac

background

investigation

framebuffer blit performance issues

avoiding the blits

conclusion + takeaways

Fixing Rust LLD Linker Undefined Symbol Error

the problem

the cause

the fix

Fixing Spotify OAuth 500 "Oops! Something went wrong" Error

The Problem

The Fix

A Unique Performance Optimization for a 3D Geometry Language

Geoscript Optimization Pipeline

Cross-Run Expr Cache Persistence

PRNG Handling

Conclusion

Updating three-good-godrays for Three.JS 0.182

Cubemaps for PointLights

Simpler Depth Packing

OrthographicCamera Now Default in postprocessing for Pass

Conclusion

Generating 3D Meshes From Text

svg-text-to-path

lyon

extrusion

conclusion

Fixing Linux Perf Inject Hanging

The Problem

The Cause

The Fix

Compiling the Boundary-First-Flattening Library to Wasm

Building OpenBLAS for Wasm

Building SuiteSparse

Building boundary-first-flattening

Code Fixes

Building

Custom Wrapper Library

Results

Fixing Google Chrome Using Wrong GPU on Linux

The Problem

The Cause

The Fix

Fixing Sveltekit `fetch` error `ERR_SSL_WRONG_VERSION_NUMBER` with NGINX reverse proxy

The Problem

The Cause

The Fix

Fixing ltree syntax error in Postgres 15

The Problem

The Cause

Fixing Google Chrome WebGL Low Frame Rate After Turning Monitors Off on Linux

The Problem

Attempted Fixes

The Real Fix

Fixing WHEA_UNCORRECTABLE_ERROR Bluescreen with Windows + Linux Dual Boot

The Problem

The Fix

Getting Kinematic Object Movement Interpolation Working in Bullet Physics/ammo.js

Background

The Problem

The Cause

The Solution

Results

Other Misc. Notes

Creating Constrained Bezier Curves for an Envelope Generator

Motivation

Constrained Bezier Curves

Computing Control Points

Conclusion

Fixing Rust+WebAssembly Memory Access Out of Bounds Errors in Debug Mode

The Problem

The Cause

A Workaround

A Better Fix

Fixing Cypress Tests Failing with 404 Errors in Github Actions CI but Working Locally

The Problem + The Fix

Notes on Self Hosting a Bluesky PDS Alongside Other Services

Manual Container Management

Cubemaps for `PointLight`s

`OrthographicCamera` Now Default in `postprocessing` for `Pass`

`svg-text-to-path`

Building `boundary-first-flattening`

Fixing Sveltekit Failed to Load URL SERVER/internal.js

`sea-*` Crate Organization

Transmission via `MeshPhysicalMaterial`

Exporting with `jmc2obj`