This involved compiling the code, and running a common set of tests on the compiled executable to determine the score.

When examining the output, I noticed that some submissions were failing to compile.

Diagnosis

Though there was a 30s timeout on the compile step to prevent broken submissions from running infinitely, I didn’t expect submissions to actually time out given the relatively small scale of the project.

After performing a couple of runs, I determined that this issue was intermittent, and occurring for different submissions each time. I also noticed that my reference submission which was written in Rust was failing more frequently.

I tried to compile some of these submissions outside of Docker, but was unable to reproduce the issue.

By chance, I then found that invoking clang (with no arguments) after restarting the machine was taking more than 20 seconds. Given that it was slow on the first run but not subsequent invocations, I thought to caching.

Perhaps there was an issue or limit with Disk I/O?

I filed a ticket with the provider.

Workaround 1

After reading documentation from previous years, I noticed that the documented setup had Docker on an external volume rather than the boot volume.

I added an external volume for Docker, and the timeouts went away.

Given that I was no longer experiencing the issue, I replied to the email from the provider with my solution and they closed the ticket.

Workaround 2

A few weeks later, I was scoring a new set of submissions for a new project.

When examining the output, I found that Rust submissions with external crates were often failing to compile, and many submission were failing on a test which indirectly required reading many small files from disk.

Another observation from watching docker ps was that though I requested parallel(1) to run 16 submissions at once, it was never running the full 16, and even dropped to 4 at one point.

Since it was a weekend, I didn’t immediately create another ticket and looked for alternative solutions.

The first thing that I tried was setting up Software RAID 0, and mounting Docker on the array. While this seemed to have improved results somewhat, some tests were still failing unexpectedly.

After some thinking, it occurred to me that since Disk I/O latency may be the issue, it might be worthwhile to try putting /var/lib/docker in memory instead.

This can be achieved with a ram disk, such as:

mount -o size=16G -t tmpfs none /var/lib/docker

Surprisingly, this worked on the first attempt. The parallelism increased, and tests were now passing consistently.

Since the RAM allocation for the VM was 64 GB, this solution was feasible for my use case and I was able to complete the testing, albeit with the minor inconvenience of recreating the ram disk after every restart.

The issue was that the client certificate popup no longer appeared when interacting with the extension.

I confirmed that visiting the URL of the Vaultwarden instance directly (i.e. in a browser tab) still triggers the prompt. But this, alongside reinstalls, restarts and clearing the cache had no effect.

After configuring the instance URL for login, the extension will try to reach the config endpoint (which returns information about server configuration) and immediately fail with 403 Forbidden without (the browser) prompting the user to select the client certificate.

As the laptop isn’t my primary device, and the Chromium version of the extension still worked, I tolerated the occasional annoyance of having to switch between browsers to copy passwords.

A workaround

At some point, possibly because it also stopped working on my main device, this became a much bigger annoyance.

After repeating the basic troubleshooting, I eventually tried to downgrade Firefox to an older version from the package cache.

Given that newer profiles cannot be used on older versions, I needed to reimport the certificate and reinstall the extension. This time, the prompt popped up and I was able to log in.

I then upgraded the browser to the latest stable version, and it continued to work in this logged in state with (what I assume to be) the certificate selection cached.

Reproduced

I ended up taking a closer look at the issue in December last year, with the aim of gathering more evidence to create a bug report.

I was able to reproduce the issue consistently on Firefox 120 and 121, while showing that the extension was working on Firefox 118.

I also created a simple test extension, which fetches a page requiring client authentication when visiting example.com:

{
  "manifest_version": 2,
  "name": "Client Cert test",
  "version": "1.0",
  "description": "Test that client cert works properly in extension.",
  "content_scripts": [
    {
      "matches": ["https://example.com/*"],
      "js": ["test.js"]
    }
  ],
  "permissions": [
    "<all_urls>"
  ]
}

// test.js
document.body.style.border = "5px solid red";
async function test() {
    const x = await fetch("https://<redacted>",
        { credentials: 'include' });
    console.log(x);
}
test();

Since the certificate prompt always pops up for this test extension when visiting example.com, I thought that perhaps a browser API has changed and that this somehow caused the extension to break in newer versions.

Believing that this was a bug with the Bitwarden extension, I created bitwarden/clients Issue #7077.

After trying to edit the code of the extension and failing to fix the issue, I set it aside and hoped that there was enough information for someone else to take a look.

Browser Bug

Unfortunately, the issue remained unsolved for a few months. Having this on my todo list for a while, I decided to investigate further on a free weekend.

After digging around in the code again, and having no luck, I browsed to the extension manifest.

Reading through, I found that there was a background key. After looking this up on MDN, it struck me as something which could make a difference, given that it is used to define background scripts and pages.

Since the extension can be used independently of a particular page or tab, perhaps it will be loaded differently compared to an extension which is activated on a page?

After extending the test app with a background page and adding the background key to the manifest:

"background": {
    "page": "background.html",
    "persistent": true
},

<html>
  <head>
    <meta charset="UTF-8" />
  </head>
  <body></body>
  <script src="test.js"></script>
</html>

I was able to reproduce the bug, but this time with the test extension.

This meant that it wasn’t an extension bug like I had previously thought, but a browser bug instead!

Having this minimal reproducible example and after confirming that it was still working on older versions, I reported Firefox Bug 1888751.

Confirmed

After providing a script to create a client certificate setup (upon request), the bug was confirmed by the triager.

I didn’t follow too closely on the discussion or the fix that followed, being busy with the teaching semester.

From reading the comments and code, my understanding is that the dialog failed to pop up because the variable holding the dialog box browserWindow.gDialogBox was undefined, where browserWindow is the “most recent main window”.

if (!browserWindow?.gDialogBox) {
    browserWindow = Services.wm.getMostRecentBrowserWindow();
}
if (browserWindow) {
    browserWindow.gDialogBox
        .open(clientAuthAskURI, { hostname, certArray, retVals })
        ...;
    return;
}

Then, the fix involved adding a further fallback of opening a new window on top of the most recent window, with the client certificate prompt as a modal.

const clientAuthAskURI = "chrome://pippki/content/clientauthask.xhtml";

// if cannot find window with browserWindow.gDialogBox...
let mostRecentWindow = Services.wm.getMostRecentWindow("");
Services.ww.openWindow(mostRecentWindow, clientAuthAskURI, "_blank",
    "centerscreen,chrome,modal,titlebar", args);

Conclusion

The fix from the Mozilla team made it to Firefox 126 Beta, and then Firefox 126 Stable on May the 14th, 2024.

After confirming that the fix applied to my workflow, I closed the GitHub issue on the Bitwarden clients repository.

Many thanks to those who were involved!

To compile using nasm:

nasm -felf64 <fn>.asm && ld <fn>.o -o <fn>

Hello World

A hello world program, which:

1. Writes out the message.
2. Exits with status code 0.

; Mark _start symbol as global
global _start

section .rodata
    message:        db "Hello World!", 0x0A
    messageLen:     equ $-message

section .text

_start:
    ; Write to stdout message of length messageLen
    mov         rax, 1          ; write syscall for x64
    mov         rdi, 1          ; stdout
    mov         rsi, message
    mov         rdx, messageLen
    syscall

    ; exit 0
    mov         rax, 60         ; exit
    xor         rdi, rdi        ; status code 0
    syscall

The syscall table for x64: https://chromium.googlesource.com/chromiumos/docs/+/master/constants/syscalls.md

Conditionals

A small program which prints out whether the number of CLI arguments is even or odd:

1. Pop argc into register.
2. Bitwise AND the value with 1.
3. Compare the result with 0, and jump to the corresponding label.
4. Print the relevant message and exit.

; Mark _start symbol as global
global _start

section .rodata
    oddMessage:        db "argc indicates odd number of arguments", 0x0A
    oddMessageLen:     equ $-oddMessage
    evenMessage:       db "argc indicates even number of arguments", 0x0A
    evenMessageLen:    equ $-evenMessage

section .text

_start:
    ; Pop top of stack (argc) to r8
    pop     r8

    ; Bitwise AND of r8 and 1
    mov     rax, 1
    and     rax, r8

    ; If rax == 0, even number, jump to even
    cmp     rax, 0
    je      even

odd:
    ; Prepare write of odd message
    mov         rsi, oddMessage
    mov         rdx, oddMessageLen
    jmp         end

even:
    ; Prepare write of even message
    mov         rsi, evenMessage
    mov         rdx, evenMessageLen

end:
    ; Write message
    mov         rax, 1          ; write
    mov         rdi, 1          ; stdout
    syscall

    ; exit 0
    mov         rax, 60         ; exit
    xor         rdi, rdi        ; status code 0
    syscall

Iterations

Loops can be achieved through a conditional and some jumps.

Here’s FizzBuzz, compiled with:

nasm -felf64 fizzbuzz.asm && gcc fizzbuzz.o -o fizzbuzz

In main:

1. The loop variable is initialised to 1.
2. Evalulate the condition, and jump to exit if the loop variable becomes greater than 100.
3. Call the fizzbuzz function (The body of the loop).
4. Increment the loop variable.
5. Jump back to the condition.

global main
extern printf
extern fflush

section .rodata
    fizz:       db "Fizz"
    fizzLen:    equ $-fizz
    buzz:       db "Buzz"
    buzzLen:    equ $-buzz
    nl:         db 0x0A
    printNum:   db "%d", 0x0

section .text

main:
    ; Initialise loop variable to 1
    ; Note that r12 is callee-saved
    mov         r12, 1

condition:
    ; Break if > 100
    cmp         r12, 100
    jg          exit

    ; Call function
    ; If register for loop variable is caller-saved,
    ; need to save value before call
    mov         rax, r12
    call        fizzbuzz

    ; Increment
    add         r12, 1
    ; Jump back to condition
    jmp         condition

exit:
    mov         rax, 60         ; exit
    xor         rdi, rdi        ; status code 0
    syscall

; FizzBuzz function
fizzbuzz:
    ; copy rax
    mov         r11, rax

    ; Divide by 3
    xor         rdx, rdx
    mov         r9, 3           ; divisor to r9
    mov         rax, r11        ; dividend to rax
    idiv        r9
    mov         r9, rdx         ; remainder to r9

    ; Divide by 5
    xor         rdx, rdx
    mov         r10, 5          ; divisor to r10
    mov         rax, r11        ; dividend to rax
    idiv        r10
    mov         r10, rdx        ; remainder to r10

    ; Flag to indicate no printing
    xor         r8, r8

; Divisible by 3?
div3:
    cmp         r9, 0
    jne         div5

    ; Print Fizz
    or          r8, 1           ; set flag
    mov         rax, 1          ; write
    mov         rdi, 1          ; stdout
    mov         rsi, fizz
    mov         rdx, fizzLen
    syscall

; Divisible by 5?
div5:
    cmp         r10, 0
    jne         printnum

    ; Print Buzz
    or          r8, 1           ; set flag
    mov         rax, 1          ; write
    mov         rdi, 1          ; stdout
    mov         rsi, buzz
    mov         rdx, buzzLen
    syscall

printnum:
    ; If number is not multiple of 3/5
    cmp         r8, 0
    jne         printline

    ; call printf
    lea         rdi, [rel printNum]
    mov         rsi, r11
    xor         eax, eax
    call        [rel printf wrt ..got]

    ; fflush
    xor         rdi, rdi
    call        [rel fflush wrt ..got]

    ; Print '\n'
printline:
    mov         rax, 1          ; write
    mov         rdi, 1          ; stdout
    mov         rsi, nl
    mov         rdx, 1
    syscall

ret

While I think I’ve experienced a similar issue with opening links before, I didn’t really want to take the ‘wait it out and hope for someone else to fix it’ approach this time.

Yesterday, I finally set out to identify the root cause and (hopefully) fix this issue.

Initially, I tried to compile the latest version of NewsFlash.
Unfortunately, it had the same issue.

I then cloned NewsFlash to search for how the Open Browser menu item is handled, tracing it to the following code:

use ashpd::desktop::open_uri::OpenFileRequest;
...

let ctx = glib::MainContext::default();
ctx.spawn_local(async move {
    if let Err(error) = OpenFileRequest::default().ask(ask).send_uri(&url).await {
        App::default().in_app_notifiaction(&i18n_f("Failed read URL: {}", &[&error.to_string()]));
    }
});

I tried to reproduce this usage of ashpd for opening links externally:

use ashpd::{desktop::open_uri::OpenFileRequest, url::Url};

#[tokio::main]
async fn main() {
    let url = "https://example.com";
    let url = Url::parse(url).unwrap();
    if let Err(error) = OpenFileRequest::default().ask(true).send_uri(&url).await {
        println!("{:?}", error);
    }
}

Unfortunately (or fortunately), it produced the same error.
This meant that it’s not a bug in NewsFlash.

Zbus(MethodError(OwnedErrorName(ErrorName(Str(Owned("org.freedesktop.DBus.Error.UnknownMethod")))), Some("No such interface “org.freedesktop.portal.OpenURI” on object at path /org/freedesktop/portal/desktop"), Msg { type: Error, sender: UniqueName(Str(Borrowed(":1.44"))), reply-serial: 15, body: Signature("s") }))

So I cloned ashpd. After struggling to produce a dev build of ashpd-demo, I eventually gave up and installed it using Flathub.

flatpak install com.belmoussaoui.ashpd.demo
flatpak run com.belmoussaoui.ashpd.demo

I clicked on the first couple of demos out of curiosity, and they all failed…

Unsurprisingly, the Open URI demo fails also.

ERROR ashpd_demo::portals::desktop::open_uri: Failed to open URI: ZBus Error: org.freedesktop.DBus.Error.UnknownMethod: No such interface “org.freedesktop.portal.OpenURI” on object at path /org/freedesktop/portal/desktop

At this stage, I performed a search on the Issues of ashpd and found no relevant results. A dead end… Perhaps it’s not an issue with ashpd either?

I went back to searching and eventually found a forum post, which links to a GitHub issue flatpak/xdg-desktop-portal #1077 No more OpenURI when using XDG_CURRENT_DESKTOP=sway and gtk portal.

This is great!

With some more trial and error, I settled on:

~/.config/xdg-desktop-portal/portals.conf

[preferred]
default=*

From one of the linked PRs.

After restarting, the links are working again! (for now…)

Edit: Note that this requires the package xdg-desktop-portal-gtk.

Here are the steps which I used to bypass the error:

1. Install edk2-ovmf for UEFI and swtpm for Software TPM.
2. Select Windows 11 ISO. virt-manager 3.2.0 still detects it as Windows 10.
3. Ensure allocation of at least 4 GB of RAM, 2 CPU cores, and 64 GB of disk.
4. Tick ‘Customize configuration before install’.
5. Add Hardware, TPM, TIS, Emulated device, 2.0.
6. Overview, Firmware to UEFI x86_64 with secure boot, Apply (important, or this will get reset!).
7. Boot options, tick CDROM, increase priority (to top), Apply again.
8. Begin installation.

Possibly unnecessary:
I encountered this strange problem where the screen was black after booting, so I changed video to VGA.
After installation, I installed spice-guest-tools and changed it back to QXL for auto-resizing.

Note: A version of this post was previously published as part of coursework for the subject SCIE90012 (Science Communication).

At some stage, you may have entered a 6-digit code from a 2FA app like Google Authenticator. But where did this code come from?

The answer lies within the QR code and the inner workings of the Time-based One-time Password algorithm (TOTP).

Here is an example of the algorithm in action.

The preparations

To prepare, I performed a MFA reset on my university account.

This was the QR code:

Then, I enrolled the QR code into my 2FA app and took the following screenshot on the 27th of September, at 9:21:19 pm UTC+10 Australia/Melbourne time:

With this, we have all the information necessary to recompute and verify the code.

Decode the QR

First, we must decode and make sense of the QR.

I’ve used a decoder to find the following URI:

otpauth://totp/sso.unimelb.edu.au:stevent2%40student.unimelb.edu.au?secret=XE7ZREYZTLXYK444&issuer=sso.unimelb.edu.au

From the URI, we see that the secret is XE7ZREYZTLXYK444.

The time?

Time undoubtedly plays an important role in the TOTP algorithm, which is defined as TOTP = HOTP(K, T), where K is the secret key and T = (Current Unix Time - T0) / X, with floor division being applied.

Current Unix Time is an integer counting the number of seconds (excluding leap seconds) since the unix epoch (January 1st, 1970). The date above converts to 1632741679 (18897 days, (21 - 10) hours, 21 minutes, 19 seconds).

T0 is the unix time from which counting starts and X is the time step in seconds, which controls how often the code changes. By default, T0 is the unix epoch (the value 0) and X is 30 seconds (if not specified).

This, along with the use of floor division means that T will stay the same across every 30 second period for this example.

Substituting in these values, we get the counter value T = (1632741679 - 0) / 30 = 54424722.

HOTP

From before, we have K = XE7ZREYZTLXYK444 and T = 54424722. By default, HOTP uses HMAC-SHA1. Taking K and T, HMAC-SHA1(K, T) produces an irreversible fixed-size value (a hash) from the input.

Convert the secret

The secret is in Base32, which uses the digits A-Z and 2-7 to represent the values 0-25 to 26-31 respectively.

1. Convert from Base32 digits to decimal (e.g. using a lookup table).

   X  E  7  Z  R  E  Y  Z  T  L  X  Y  K  4  4  4
   23 04 31 25 17 04 24 25 19 11 23 24 10 28 28 28
   

2. Convert decimal to binary (each will be less or equal to 5 bits, because values 0-31 are represented).

   10111 00100 11111 11001 10001 00100 11000 11001
   10011 01011 10111 11000 01010 11100 11100 11100
   

3. Rewrite this into groups of 4 bits, and convert binary to hexadecimal using the trick that 4 binary digits is one hexadecimal digit.

   1011 1001 0011 1111 1001 1000 1001 0011 0001 1001
   B    9    3    F    9    8    9    3    1    9
   1001 1010 1110 1111 1000 0101 0111 0011 1001 1100
   9    A    E    F    8    5    7    3    9    C
   

Convert the counter value

1. Convert to binary and then hexadecimal.

   54424722 =
   0011 0011 1110 0111 0100 1001 0010
   3    3    e    7    4    9    2
   

2. Pad this number to 8 bytes (16 hexadecimal digits) by adding leading zeros, per the RFC.

   00000000033e7492
   

HMAC & Truncation

With the converted values, we can calculate the SHA1-HMAC.

The values from above produces the following hash: 38056c8282a236a5ad624ff292b984d20138aba8.

Next, we follow a dynamic truncation algorithm to extract a section of the hash:

1. Given the 20 byte SHA1 hash String, extract the “low-order 4 bits of String[19]” to use as Offset.
   When splitting the hash into 20 segments, String[19] is the last segment (since we start from index 0). This is a8.
   Then, because one hexadecimal digit represent 4 binary bits, and the least significant digit is the rightmost digit, it is 8 in this example.

2. Convert the 4 bits into a number.
   8 in hexadecimal is the same in decimal (base 10), and so we move on.

3. Return the last 31 bits of String[Offset]...String[Offset+3].
   Substituting in 8, this becomes String[8]...String[11].

   38 05 6c 82 82 a2 36 a5 ad 62 4f f2 92 b9 84 d2 01 38 ab a8
   0  1  2  3  4  5  6  7  ^  ^  ^  ^  12 13 14 15 16 17 18 19
   

   We can then convert this into binary, by replacing each hexadecimal digit with its 4-digit binary equivalent.

   a    d    6    2    4    f    f    2
   1010 1101 0110 0010 0100 1111 1111 0010
   

   Since it’s the last 31 bits, we unset the first bit.

   0010 1101 0110 0010 0100 1111 1111 0010
   

The next step involves converting this 31-bit binary number into decimal.

2²⁹ + 2²⁷ + 2²⁶ + 2²⁴ + 2²² + 2²¹ + 2¹⁷ + 2¹⁴ + 2¹¹ + 2¹⁰ + 2⁹ + 2⁸ + 2⁷ + 2⁶ + 2⁵ + 2⁴ + 2¹ = 761417714

When not specified in the URI, 6-digit codes are used by default. Hence, we can (in the final step) write down the last six digits of the number.

And voila, 417714!

References

• RFC 4226: HOTP
• RFC 6238: TOTP
• Key URI format

Verified with:

$ oathtool --totp --verbose --now "2021-09-27 11:21:19 UTC" --base32 XE7ZREYZTLXYK444
Hex secret: b93f9893199aef85739c
Base32 secret: XE7ZREYZTLXYK444
Digits: 6
Window size: 0
TOTP mode: SHA1
Step size (seconds): 30
Start time: 1970-01-01 00:00:00 UTC (0)
Current time: 2021-09-27 11:21:19 UTC (1632741679)
Counter: 0x33E7492 (54424722)

417714

Last year, I had some trouble setting up Android builds on self-hosted GitHub Actions runners for smart-brocolli. This post serves to document this process.

1. Provision a Linux VM. I will be using Ubuntu 20.04 on Azure.
2. Install Flutter SDK.
   https://flutter.dev/docs/get-started/install/linux#install-flutter-manually
3. Add flutter to $PATH and run $ flutter doctor.
4. Download Android command-line tools.
   https://developer.android.com/studio#command-tools
5. Extract tools:

   $ sudo apt install unzip && unzip commandlinetools-linux-*.zip
   

6. Set up Android directory:

   $ mkdir android-sdk && mkdir android-sdk/cmdline-tools && \
    mv cmdline-tools/ android-sdk/cmdline-tools/latest
   

7. Add android-sdk/cmdline-tools/latest/bin/ to $PATH, set $ANDROID_HOME.
   e.g.

   export ANDROID_HOME=`pwd`/android-sdk
   export PATH="$PATH:`pwd`/flutter/bin:`pwd`/android-sdk/cmdline-tools/latest/bin"
   

8. Install Java:

   $ sudo apt install openjdk-8-jdk
   

9. Install Platform SDK and build-tools using sdkmanager, e.g.:

   $ sdkmanager --install "platforms;android-30"
   $ sdkmanager --install "build-tools;30.0.2"
   

10. Review licenses:

    $ flutter doctor --android-licenses
    

11. Run $ flutter doctor again.
    Ensure that a tick is given for Android Toolchain.
12. Finally, add an appropriate GitHub Actions workflow.

mongoose

• PR #9760 is a followup PR to #9688. It configures Actions to trigger on pull requests and changes the README badge from travis to Actions.
• PR #10393 replaces mongodb-topology-manager with an adapted implementation (for future CI upgrades).
• PR #10679 adds test for latest Ubuntu LTS alongside pre-existing older version through strategy.matrix.

wakapi

• PR #85 reduces minimum username length to 1.
• PR #86 reduces docker image layers by moving some steps to the build stage.
• PR #95 allows static assets to be embedded using pkger.
• PR #97 is a followup PR to #95, and fixes CI releases.
• PR #107 removes legacy configuration code.
• PR #111 adds GitHub Action to publish docker image to Docker Hub on release.
• PR #130 is a simple fix for a migration which fails for sqlite.
• PR #167 switches from pkger to golang’s embed.
• PR #217 adds Alpine image.
• PR #220 adds removal of unix socket on start.
• PR #233 fixes some javascript errors caused by conditional display of summary page (has data vs. no data).

And more…

Misc

• PR #311 docker_auth fixes arm64 Docker builds.
• PR #6 comp90024 fixes a dead link and some typos.
• PR #1 mail.katmail.xyz fixes TLSA script typos.
• MR #59 newsflash-gtk fixes a meson build issue.

MongoDB & mongoose

• PR #6 mongo-sanitize adds recursive object sanitisation, to mitigate mongoose/MongoDB injection attacks (which can occur on e.g. unsanitised JSON input).
• Issue #8521 mongoose was created after I found that calling syncIndex() after updating the collation index for a model (for case-insensitive username matching) had no effect.
• PR #9688 mongoose adds GitHub Actions CI configuration (after travis.org announcement). This was somewhat challenging due to environment (os, node, and dependency) differences and timing issues which exists in the test suite. In hindsight, it probably would have been easier if I tested it using Docker first.
• PR #9696 mongoose adds single document populate to the new internal type definitions.
• PR #9705 mongoose is a small CSS responsive design fix for documentation.
• PR #374 connect-mongo (not yet merged) fixes type definitions errors caused by upstream type definition updates.

CouchDB

For project 2 of COMP90024, we were required to use CouchDB and its MapReduce functions to analyze harvested Twitter data. One of the tasks is to use ansible to automate the deployment of fresh nodes. Though Docker was optional, I decided to use it to reduce complexity in making changes. During this process, I ran into several non-obvious details and issues.

• PR #185 couchdb-docker adds documentation about the default ports which are exposed by vm.args and the Dockerfile.
• PR #186 couchdb-docker (not merged) adds a sample docker compose file.

• PR #3036 couchdb fixes an issue which causes the finish_cluster API call to fail when executed on a fresh CouchDB node with no UUID. I helped by describing and reproducing the issue (using scripts and 2 networked VMs) and submitting the final PR.

• PR #225 couchdb-nano fixes typescript definitions for generic user-defined requests (request, relax, dinosaur functions). I discovered this issue when attempting to use the request function to get metadata about the instance using the / route.
• PR #226 couchdb-nano adds nano.info(), a method specifically used to retrieve the metadata. This PR came about because I was expecting a method to access the / route, but didn’t find any. As I didn’t have enough time before the project was due, I used nano.request() for the project and submitted this PR during the winter break.

tokei

• PR #576 (my first Rust PR!) was a small fix to remove a duplicated occurrence of json in tokei --help. I came across the attached issue by chance and thought that it might be a fun problem to tackle.
• PR #580 adds summary information (i.e. the Total) to output formats. Some minor refactoring was done to accommodate this change.

Misc self-hosted web apps

• PR #35 wakapi adds BASE_PATH as an environment variable.
• PR #1495 trilium was a small change to rename the session cookie from the default. This PR was inspired by a clash of express-session cookies which occurred on one of my subdomains.
• PR #252 linx-server removes entrypoint from the sample docker-compose.yml file suggested by the documentation. This prevents the default ENTRYPOINT from the Dockerfile from being overridden.

Rose compiler

• Issue #141 reports a compilation error.

Summary

2020 is the first year where I have started to look at and dive more deeply into the code of the software and packages that I use. Though I have mostly focused on small bugs and minor issues, I have found the process to be highly rewarding and something worthwhile to repeat for the future.

Though I doubt that many people (if there are any at all) read my blog, I wish you all the best for 2021!

After finishing my exams, I started to look at the miscellaneous tasks which were on my todo list. While messing around on my VM, I thought that it was probably a good time to upgrade to Ubuntu 20.04.

I first checked do-release-upgrade. For some reason, it listed eoan instead of focal as the release to upgrade to. Because I didn’t have many services running on the VM (only Gitea and ZNC), I thought that it wouldn’t be a bad idea to start fresh on a fresh VM.

After stopping all the running services, I moved all the files which weren’t on the attached block storage volume to the volume (misc files in /home like /home/ubuntu/.znc, configs from /etc/nginx, letsencrypt certificates from /etc/letsencrypt etc).

Then, I swapped this volume from the old machine to the new one, and started installing docker and docker-compose. This went quite smoothly now that the focal source respository for docker is present. After that, I disabled the snap and Oracle related services which were on the machine.

Because my domains pointed to the IP address of the original VM, I reassigned the reserved IP of the old VM and to the new one. This turned out to be a good decision, as it became possible to ProxyJump from the new machine to the old machine using its internal IP address, and I used this to double check some configuration files (e.g. iptables).

ssh -J vm ubuntu@10.0.0.7

Then, it was just a matter of reconfiguring ZNC, gitea and the other relevant pieces (e.g. nginx, iptables, sshd_config, fail2ban).

Containerising ZNC

I use znc as an IRC bouncer for freenode. Because I configured the old instance in a rush, I didn’t bother investigating whether it had a docker image or not and just let it run on the machine. This time round, I thought that it would be a good idea to run it as a container, if only to make the migration process easier the next time.

I referred to the reference for the official docker image and the compose file from linuxserver and adapted it into the following docker-compose file. Note that 6667 is the port which I originally chose.

---
version: "2.1"
services:
  znc:
    image: znc:latest
    container_name: znc
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Australia/Melbourne
    volumes:
      - ./data:/znc-data
    ports:
      - 6667:6667
    restart: unless-stopped

I renamed the original .znc folder to data and ran docker-compose start, immediately running into the following error:

Unable to locate pem file: /home/ubuntu/.znc/znc.pem

After playing around with the configuration files, I eventually realised that znc.conf contains a reference to the original location of the pem file. Therefore, I changed the corresponding line to reflect the new location, and all was well.

Gitea with SSH passthrough

I use gitea for hosting some of my personal git repositories. Because it is desirable to use SSH to perform operations on remote repositories, it becomes necessary to use SSH passthrough to pass SSH from host to container.

Though the guide on Gitea provides a step by step process, it isn’t trivial to set it up correctly.

Received disconnect from <IP> port 22:2: Too many authentication failures
Disconnected from <IP> port 22
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Here are some strategies for debugging such issues:

• Get the SSH client to output some logs when performing operations on remote repositories. e.g.

GIT_SSH_COMMAND="ssh -v" git clone <url>

When having many SSH keys, use a ~/.ssh/config file to specify the keys that should be used.

Host git.domain
    HostName git.domain
    IdentityFile ~/.ssh/<private_key>
    User git

• Check the SSH logs (e.g. /var/log/auth.log). Often, you’ll find something intuitive, e.g.

Jun 20 09:58:31 sshd[13579]: message repeated 5 times: [ Authentication refused: bad ownership or modes for directory /volume/gitea/gitea]

• Change to the git user (e.g. sudo su git) and see if the authorized_keys file is accessible.