defn.io

One Billion Row Challenge in Racket

Wed, 10 Jan 2024 21:51:00 +0200

I decided to have some fun tonight and work on a Racket solution to the One Billion Row Challenge. I came up with a (pretty gnarly) solution that completes in about 45 seconds on my machine, a 2023 12-core Apple M2 Max with 96GB of RAM. This is about on par with the lower end of the optimized Java solutions shown in the challenge repo's README, when I run them on the same machine.

You can find the solution in this GitHub Gist. It works by splitting the work across several places¹, where each place iterates over the input file in full. I initially attempted an approach where the main place read the whole file into a shared-bytes value and then dispatched work to separate places, but that turned out to have too much overhead from the places accessing the shared bytes. I had also tried an approach where the main place sent individual work units (i.e. lines) to the worker places, but that was killed by the overhead of place-channel-put².

On boot, each place is given the path to the input file, the total number of places (shards), its shard number and a place channel where it can publish the aggregated data once it's done reading the file. Even though each place iterates over the full input file, it only processes those entries that match its shard number, skipping the rest and thereby reducing the total number of work per place by the total number of places.

The data gets read in in 10MB chunks, but the difference between buffer sizes 1MB and over is negligible. The place-local data is stored in a hash from location names to state structs that contain the number of entries, the lowest temperature, the highest temperature and the sum of all temperatures at that location, as seen by that particular place. Any one location's data may be spread across different places, and there is a final step to combine the data from every place by location and print out the results.

Originally, I had used a regular Racket hash to store the place-local data, but I eventually rolled my own open-addressed hash table to avoid the cost of copying location names from the input buffer to individual bytes values for use with hash-ref!. I don't remember exactly how much time this saved, but I believe it was on the order of about 10-15 seconds. This is probably less because my hash is any good (it's not), and more because copying so many little strings gets expensive fast. The code goes to great lengths to avoid copying as much as possible and, thankfully, the built-in bytes procedures are set up to help as much as they can.

A couple more minor, but potentially-interesting, things about the code are its use of #%declare to disable some contract checks, and the filtered-in require to rename imports from racket/unsafe/ops to drop the unsafe- prefix.

I'm not completely satisfied with the result, so I may spend some more time in the coming days trying to come up with ways to make this code faster. Let me know if you have any ideas!

You might have noticed I spawned one place for every two CPU cores on my machine. This turned out to be more performant than trying to use one place per core, presumably because there was less synchronization overhead between places. ↩
My guess would be the issue here was partly contract checks and partly the overhead of copying the bytes between places. It would be nice if we had support for some kind of shared immutable bytes with less overhead than regular shared bytes. That way one could send the shared immutable bytes over to the worker places once and subsequently send pairs of start and end indexes representing the work units. ↩

Extensible Linting for Racket

Sun, 7 Jan 2024 18:37:00 +0200

Several years ago, I released review, a little linter for Racket. Recently, I added support for extending the linter from arbitrary Racket packages.

Review analyzes code by pattern-matching on input programs' surface-level syntax, rather than by analyzing their fully-expanded forms. This is a nice property because it means that it can provide advice about how to use certain syntactic forms before those forms are expanded to core Racket forms. It is, however, a double-edged sword because it means that the linter has to have a rule for every syntactic form it wants to lint, including ones that it can't possibly have any knowledge of (eg. because they're user-defined). By default, review simply ignores such unknown forms and does its best to analyze the rest of the program. This works reasonably well, but may cause it to return false-positive results or miss certain issues altogether.

I had had it in the back of my mind to make review user-extensible for a while but never got around to it until this winter break. It turned out to be much easier than I expected!

How it Works

When review is run, it looks for installed Racket packages that have a review-exts entry in any of their info.rkt files. Every entry must be a list of 3-element lists of this form:

[module-path should-review?-proc-id review-ext-proc-id]

On boot, it dynamically requires the should-review?-proc-id and the review-ext-proc-id from each module-path of each entry it discovers. For every expression in the input program, it applies each should-review?-proc against the expression's syntax object. If any of the should-review? procedures returns #t, that procedure's associated review-ext-proc is then called on that syntax object. The first match wins and, if there are no matches, then review falls back to its own analysis of the expression.

Review extension procedures can use the bindings provided by the review/ext module to track any errors or warnings they find or to manage scopes and bindings.

A Concrete Example

The above description probably makes things sound more complicated than they really are. So, let's try to write a custom linter. Say we have a custom form for defining records:

(define-record RECORD-ID
  (FIELD ...))

Where:

FIELD
  = [FIELD-ID : TYPE-ID]

TYPE-ID
  = Int
  | Str

The form can be used as follows:

(define-record Person
 ([name : Str]
  [age : Int]))

For every record, the above definition generates a constructor procedure (make-Person), a predicate (Person?), accessors for every field (Person-name and Person-age), and setters for every field (set-Person-name and set-Person-age).

We can start by adding a module called review.rkt to the root of our collection. In it, we'll have to define our should-review? and review-syntax procedures:

#lang racket/base

(require review/ext
         syntax/parse)

(provide
 should-review?
 review-syntax)

(define (should-review? stx)
  (syntax-parse stx
    #:datum-literals (define-record)
    [(define-record . _rest) #t]
    [_ #f]))

(define (review-syntax stx)
  stx)

The should-review? procedure returns #t whenever it is given a list whose first item is the literal symbol define-record and #f in every other case. For now, the review-syntax procedure is just a stub that does nothing. With this module in hand, we can now update our info.rkt file to add a review-exts entry:

#lang info

...

;; Assuming our collection is named "example".
(define review-exts
  '([example/review should-review? review-syntax]))

As soon as we do this, running raco review against a program should defer to the review-syntax procedure above any time it encounters a (define-record ...) form. So, let's extend it to perform some checks:

#lang racket/base

#|review: ignore|#

(require review/ext
         syntax/parse/pre)

(provide
 should-review?
 review-syntax)

(define (should-review? stx)
  (syntax-parse stx
    #:datum-literals (define-record)
    [(define-record . _rest) #t]
    [_ #f]))

(define-syntax-class field-definition
  #:datum-literals (: Int Str)
  (pattern [id:id : {~or Int Str}])
  (pattern [id:id : type] #:do [(track-error #'type "invalid type")])
  (pattern e
           #:attr id #'stub
           #:do [(track-error this-syntax "invalid field definition")]))

(define-syntax-class record-definition
  (pattern (define-record record-id:id
             (record-field:field-definition ...))))

(define (review-syntax stx)
  (syntax-parse stx
    [d:record-definition #'d]
    [_ (begin0 stx
         (track-error stx "invalid record definition"))]))

Above, we define syntax classes to pattern match on record and field definitions. Match failures, including using invalid types for field definitions, are reported as errors. This already gets us a useful little linter that checks that our uses of define-record are syntactically valid. With a little more work, we can extend our record-definition class to track the bindings generated by define-record:

(define-syntax-class record-definition
  (pattern (define-record record-id:id
             (record-field:field-definition ...))
           #:do [(track-binding #'record-id "make-~a")
                 (track-binding #'record-id "~a?")
                 (define record-id-sym (syntax->datum #'record-id))
                 (for ([field-id-stx (in-list (syntax-e #'(record-field.id ...)))])
                   (track-binding field-id-stx (format "~a-~~a" record-id-sym))
                   (track-binding field-id-stx (format "set-~a-~~a" record-id-sym)))]))

Now, our linter will complain if we have any unused fields. You can find the full example on GitHub. For a slightly more complex example, check out the custom linter for deta on GitHub.

Limitations

This approach has some obvious limitations. The binding tracking is approximate and done as a side-channel to the way Racket actually does things. Syntactic forms are recognized symbolically rather than by binding, so linters have no way of recognizing renamed imports, nor do they have a way of distinguishing the same names across different libraries.

Despite all that, I get valuable support from review in the 80% ¹ of cases where it does work well, and I expect to get more out of it as I wrote more extensions for bits of custom syntax that I have here and there. Maybe you will too!

A made-up number. Probably, it fails less than 20% of the time, but I haven't actually kept track. ↩

Advent of Racket 2023/07 - Camel Cards

Thu, 7 Dec 2023 10:00:00 +0200

Quite a fun one today. The example input looks like this:

32T3K 765
T55J5 684
KK677 28
KTJJT 220
QQQJA 483

The first column represents a poker hand and the second a bid. We're to sort the hands, multiply the bid and the hand's rank (its position after sorting), then sum up the results.

We can read the data as a vector of (hand . bid) pairs:

(define hands
  (call-with-input-file "day07.txt"
    (lambda (in)
      (for/vector ([line (in-lines in)])
        (match-define (regexp #rx"([^ ]+) (.+)"
                              (list _ hand (app string->number bid)))
          line)
        (cons hand bid)))))

To compute the type of hand we have, we can count the kinds of cards in a hand, sort the results and pattern match on them, since each hand is guaranteed to be of exactly one type:

(define (hand-counts h)
  (for/fold ([counts (hasheqv)])
            ([c (in-string h)])
    (hash-update counts c add1 0)))

(define (hand-type h)
  (define counts
    (sort
     (hash->list
      (hand-counts h))
     #:key cdr >))
  (match counts
    [`(,_) 'five-of-a-kind]
    [`((,_ . 4) ,_) 'four-of-a-kind]
    [`((,_ . 3) ,_) 'full-house]
    [`((,_ . 3) ,_ ,_) 'three-of-a-kind]
    [`((,_ . 2) (,_ . 2) ,_) 'two-pair]
    [`((,_ . 2) ,_ ,_ ,_) 'one-pair]
    [`(,_ ,_ ,_ ,_ ,_) 'high-card]))

Next, let's define a sort order for each type of hand:

(define (hand-type-numeric h)
  (case (hand-type h)
    [(five-of-a-kind) 6]
    [(four-of-a-kind) 5]
    [(full-house) 4]
    [(three-of-a-kind) 3]
    [(two-pair) 2]
    [(one-pair) 1]
    [(high-card) 0]))

In order to break ties, we're going to have to compare individual cards in a hand, left-to-right. Higher cards beat lower cards. So, we can define a procedure to get a card's score:

(define (card-score c)
  (match c
    [#\2 2]
    [#\3 3]
    [#\4 4]
    [#\5 5]
    [#\6 6]
    [#\7 7]
    [#\8 8]
    [#\9 9]
    [#\T 10]
    [#\J 11]
    [#\Q 12]
    [#\K 13]
    [#\A 14]))

Finally, we can define a procedure for sorting hands:

(define (hand> a b)
  (define an (hand-type-numeric a))
  (define bn (hand-type-numeric b))
  (if (= an bn)
      (for/fold ([ok? #t])
                ([ca (in-string a)]
                 [cb (in-string b)])
        (define cas (card-score ca))
        (define cbs (card-score cb))
        #:break (or (> cas cbs)
                    (not ok?))
        (and ok? (= cas cbs)))
      (> an bn)))

When there's a tie, we fall back to comparing the cards in both hands positionally using card-score. Otherwise, the better hand wins.

To compute the solution for part one, we can sort the hands then sum up the winnings for each hand:

(define (compute-winnings hands [hand> hand>])
  (let ([hands (vector-copy hands)])
    (vector-sort! hands hand> #:key car)
    (for/sum ([(h idx) (in-indexed (in-vector hands))])
      (define rank (- (vector-length hands) idx))
      (* (cdr h) rank))))

(compute-winnings hands)

For part two, we need to reinterpret joker cards such that whenever we compute the hand type for a hand that contains jokers, we have to replace the jokers in that hand with whatever cards would make it the best hand it can be. When comparing individual cards, jokers are now the weakest card type.

Given any hand that contains jokers, we can find the best hand it can be by iterating through all the possible replacements of non-joker cards:

(define (find-strongest h)
  (define counts
    (hand-counts h))
  (cond
    [(hash-has-key? counts #\J)
     (define non-jokers
       (remv #\J (hash-keys counts)))
     ;; When
     ;;  non-jokers = '(#\Q #\2)
     ;; Then
     ;;  replacementss = '((#\Q #\Q) (#\Q #\2) (#\2 #\2))
     (define replacementss
       (remove-duplicates
        (map (λ (cards) (sort cards char>?))
             (apply cartesian-product (make-list (hash-ref counts #\J) non-jokers)))))
     (for/fold ([res #f] #:result (or res h))
               ([replacements (in-list replacementss)])
       (define replacement-hand
         (for/fold ([chars null]
                    [replacements replacements]
                    #:result (apply string (reverse chars)))
                   ([c (in-string h)])
           (if (char=? c #\J)
               (values (cons (car replacements) chars) (cdr replacements))
               (values (cons c chars) replacements))))
       (if (or (not res)
               (> (hand-type-numeric replacement-hand)
                  (hand-type-numeric res)))
           replacement-hand
           res))]
    [else h]))

Next, we can update card-score to take an optional argument representing the score for joker cards:

(define (card-score c [j-score 11])
  (match c
    [#\2 2]
    [#\3 3]
    [#\4 4]
    [#\5 5]
    [#\6 6]
    [#\7 7]
    [#\8 8]
    [#\9 9]
    [#\T 10]
    [#\J j-score]
    [#\Q 12]
    [#\K 13]
    [#\A 14]))

And we can update the signature for hand> to make it possible for the caller to pass in custom procedures for computing hand types and card scores:

(define (hand> a b
               #:hand-type-numeric-proc [hand-type-numeric hand-type-numeric]
               #:card-score-proc [card-score card-score])
  ...)

Its body remains unchanged. With these changes in place, we can solve part two by passing in a custom sort procedure to compute-winnings:

(compute-winnings
 hands
 (lambda (a b)
   (hand>
    #:hand-type-numeric-proc (compose1 hand-type-numeric find-strongest)
    #:card-score-proc (λ (c) (card-score c 1))
    a b)))

This will be my last post in this series, but I'll probably keep solving puzzles for another week or so. Check out my solutions repo if you want to follow along!

Advent of Racket 2023/06 - Wait For It

Wed, 6 Dec 2023 10:00:00 +0200

A really quick one today. The example input looks like this:

Time:      7  15   30
Distance:  9  40  200

Every pair of rows represents a race, where the distance is the record distance so far. By pausing at the beginning of the race we gain one distance unit per time unit paused, but lose that time unit in the process. We're to determine the distinct durations we could have paused for during every race in order to beat the record distance, then multiply the results.

We can read the races as a list of pairs:

(define (read-integers in start)
  (map string->number
       (string-split
        (substring (read-line in) start))))

(define races
  (call-with-input-file "day06.txt"
    (lambda (in)
      (define times (read-integers in (string-length "Time:")))
      (define distances (read-integers in (string-length "Distance:")))
      (map cons times distances))))

And write a procedure to determine whether or not a given hold time would beat the record distance:

(define (win? r hold-time)
  (match-define (cons race-time distance) r)
  (define travel-time
    (- race-time hold-time))
  (> (* hold-time travel-time) distance))

Then all we have to do is count the number of times a race could be won within its allotted time:

(define (winning-hold-times r)
  (for/sum ([i (in-range (add1 (car r)))]
            #:when (win? r i))
    1))

And multiply those counds for every race together:

(for/fold ([res 1])
          ([r (in-list races)])
  (* res (winning-hold-times r)))

For part two, we need to append all the race times and durations together into one long race. So, instead of interpreting our example input as three separate races, we need to interpret it as if it were written without any spaces:

Time:      71530
Distance:  940200

Let's append the races together into our input for part two:

(define one-race
  (let ([m (λ (n) (expt 10 (exact-ceiling (log n 10))))])
    (for/fold ([t 0] [d 0] #:result (cons t d))
              ([r (in-list races)])
      (match-define (cons r-t r-d) r)
      (values (+ (* t (m r-t)) r-t)
              (+ (* d (m r-d)) r-d)))))

Finally, we can just call our winning-hold-times procedure on the one-race value to find the solution for part two. The input is small enough to brute force in a couple hundred milliseconds.

If the input for part two were larger, we could use a closed form solution. We've already expressed a race's solution as:

x(t - x) > d

Where x is the hold time required to beat the record d. We can expand that expression to:

-x^2 + tx - d > 0

And we can solve for x using the quadratic formula and get all possible values of x for any given distance:

(define (winning-hold-times* r)
  (match-define (cons t d) r)
  (define discriminant (sqrt (- (* t t) (* 4 d))))
  (define hi (exact-ceiling (/ (+ t discriminant) 2)))
  (define lo (exact-floor (/ (- t discriminant) 2)))
  (- hi lo 1))

Advent of Racket 2023/05 - Fertilizer

Tue, 5 Dec 2023 10:00:00 +0200

The example input for day five looks like this:

seeds: 79 14 55 13

seed-to-soil map:
50 98 2
52 50 48

soil-to-fertilizer map:
0 15 37
37 52 2
39 0 15

fertilizer-to-water map:
49 53 8
0 11 42
42 0 7
57 7 4

water-to-light map:
88 18 7
18 25 70

light-to-temperature map:
45 77 23
81 45 19
68 64 13

temperature-to-humidity map:
0 69 1
1 0 69

humidity-to-location map:
60 56 37
56 93 4

The seeds line contains the set of inputs we're supposed to pass through the given maps and each map feeds into the next. The entries in a map represent the ranges of values that can be converted by the map. Values outside the given ranges are passed through unchanged. After feeding every seed value through all the maps, we need to report the minimum value that comes out the other end.

I decided to make a struct to represent the individual ranges in a map:

(struct mapping (dst src len)
  #:transparent)

(define (parse-mapping str)
  (apply mapping (map string->number (string-split str))))

> (parse-mapping "50 98 2")
(mapping 50 98 2)

To read a map, we ignore the empty line before its definition, ignore the line that names the map and then read the ranges until we see an empty line or reach the end of file:

(define (read-map in)
  (void (read-line in))
  (void (read-line in))
  (let loop ([mappings null])
    (define c (peek-char in))
    (cond
      [(or (eof-object? c)
           (eqv? c #\newline))
       (reverse mappings)]
      [else
       (loop
        (cons
         (parse-mapping (read-line in))
         mappings))])))

We read the initial seeds and then the seven maps. Since the maps feed into each other, there's no need to keep track of which map is which, so a list of maps is enough.

(define-values (seeds maps)
  (call-with-input-file "day05.txt"
    (lambda (in)
      (define seeds
        (map
         string->number
         (string-split
          (substring (read-line in) 6))))
      (values
       seeds
       (for/list ([_ (in-range 7)])
         (read-map in))))))

Every map is just a list of mapping struct instances and for any given mapping, we can map a value by checking if it's within the specified range and adding to it the delta between the dst and src values:

(define (mapping-map m v)
  (match-define (mapping dst src len) m)
  (and (>= v src)
       (<= v (+ src len))
       (+  v (- dst src))))

Given a set of mappings (i.e. a map), we can write a lookup procedure that returns the mapped value for any given value:

(define (look-up mappings v)
  (or
   (for*/first ([m (in-list mappings)]
                [mapped-v (in-value (mapping-map m v))]
                #:when mapped-v)
     mapped-v)
   v))

With that, we can write a procedure to run a seed through all the maps:

(define (find-seed-location maps seed)
  (for/fold ([value seed])
            ([mappings (in-list maps)])
    (look-up mappings value)))

And, finally, go through all the seeds and find the minimum location:

(for/fold ([res +inf.0])
          ([s (in-list seeds)])
  (define loc
    (find-seed-location maps s))
  (if (< loc res) loc res))

For part two, we're asked to reinterpret the initial list of seeds as pairwise ranges of seeds instead of individual seed numbers. So 79 14 in our initial example now represents the seeds from 79 to 93. The actual input is large enough that brute forcing a solution would take a while.

Instead of iterating over all the seeds in every range, we can split our ranges against all the mappings in a map, then map the values of the split ranges and then feed the mapped ranges into the subsequent maps.

First, let's collect the ranges into a list of pairs:

(define ranges
  (let loop ([pairs null]
             [seeds seeds])
    (cond
      [(null? seeds)
       (reverse pairs)]
      [else
       (loop
        (cons
         (cons (car seeds)
               (+ (car seeds)
                  (cadr seeds)))
         pairs)
        (cddr seeds))])))

> (car ranges)
(79 . 93)

Next, let's write a procedure to split a range for any given mapping:

(define (mapping-split-range m r)
  (match-define (mapping _dst src len) m)
  (match-define (cons lo hi) r)
  (define src-lo src)
  (define src-hi (+ src len))
  (cond
    [(and (< lo src-lo)
          (< hi src-lo))
     (list r)]
    [(and (> lo src-hi)
          (> hi src-hi))
     (list r)]
    [(and (< lo src-lo)
          (> hi src-hi))
     (list
      (cons lo (sub1 src-lo))
      (cons src-lo src-hi)
      (cons (add1 src-hi) hi))]
    [(< lo src-lo)
     (list
      (cons lo (sub1 src-lo))
      (cons src-lo hi))]
    [(> hi src-hi)
     (list
      (cons lo src-hi)
      (cons (add1 src-hi) hi))]
    [else
     (list
      (cons lo hi))]))

And a procedure to map a range for a mapping:

(define (mapping-map-range m r)
  (define m-lo (mapping-map m (car r)))
  (define m-hi (mapping-map m (cdr r)))
  (and m-lo m-hi (cons m-lo m-hi)))

Let's write a procedure to map the ranges for a given set of mappings:

(define (map-ranges mappings ranges)
  ;; Split the ranges against all the mappings.
  (define split-ranges
    (let loop ([ranges ranges]
               [mappings mappings])
      (if (null? mappings)
          ranges
          (loop
           (apply append
                  (for/list ([r (in-list ranges)])
                    (mapping-split-range (car mappings) r)))
           (cdr mappings)))))
  ;; Then map the split ranges.
  (for/list ([r (in-list split-ranges)])
    (or
     (for*/first ([m (in-list mappings)]
                  [m-r (in-value (mapping-map-range m r))]
                  #:when m-r)
       m-r)
     r)))

And a procedure to feed the ranges through all the maps:

(define (find-location-ranges maps ranges)
  (for/fold ([ranges ranges])
            ([mappings (in-list maps)])
    (map-ranges mappings ranges)))

Finally, we can compute the result for part two by keeping track of the smallest start value of every funneled range:

(for*/fold ([res +inf.0])
           ([r (in-list (find-location-ranges maps ranges))])
  (define loc (car r))
  (if (< loc res) loc res))

An alternative, and probably less finicky, but less efficient, way to solve part two would've been to iterate up from 0 and pass the numbers backwards through the funnel, stopping on the first number that fit in one of the seed ranges. In my case that approach would've found the solution in a little under ten million iterations.

Advent of Racket 2023/04 - Scratchcards

Mon, 4 Dec 2023 10:00:00 +0200

Day four starts out very simple. We're given an input where we have two lists of numbers per line. Per line (or "card"), we get one point for the first number in the second list that is also found in the first, and the score doubles for every subsequent match. The example input looks like this:

Card 1: 41 48 83 86 17 | 83 86  6 31 17  9 48 53
Card 2: 13 32 20 16 61 | 61 30 68 82 17 32 24 19
Card 3:  1 21 53 59 44 | 69 82 63 72 16 21 14  1
Card 4: 41 92 73 84 69 | 59 84 76 51 58  5 54 83
Card 5: 87 83 26 28 32 | 88 30 70 12 93 22 82 36
Card 6: 31 18 13 56 72 | 74 77 10 23 35 67 36 11

Since part one seemed a little too easy, I decided to parse the input into a struct, in anticipation of a harder part two.

;; id : int
;; winning : listof int
;; have : listof int
(struct card (id winning have)
  #:transparent)

Parsing the cards is straightforward enough, though I did spend a minute scratching my head because I forgot to escape the | character:

(define cards
  (call-with-input-file "day04.txt"
    (lambda (in)
      (for/vector ([line (in-lines in)])
        (match-define (regexp #rx"Card +([0-9]+): ([^|]+) \\| (.+)"
                              (list _ (app string->number id) winning-str have-str))
          line)
        (card
         id
         (map string->number (string-split winning-str))
         (map string->number (string-split have-str)))))))

I had originally stored the set of cards as a list, but changed it to a vector for part two. We'll see why in a bit. In the mean time, computing part one is just a matter of determining the number of matches in each card:

(define (card-matches c)
  (for/sum ([n (in-list (card-have c))]
            #:when (memv n (card-winning c)))
    1))

And computing the score per card:

(define (card-score c)
  (define matches
    (card-matches c))
  (cond
    [(zero? matches) 0]
    [else (expt 2 (sub1 matches))]))

Putting those functions together, we get:

(define part1
  (for/sum ([c (in-vector cards)])
    (card-score c)))

For part two, the problem goes exponential. For every card, the number of matches that we find represents subsequent cards that we have to check for matches. We have to recursively add up all the cards we see.

We need a function that returns the won cards for any given card:

(define (card-wins c)
  (match-define (card id _winning _have) c)
  (for/list ([i (in-range 0 (card-matches c))])
    (vector-ref cards (+ id i))))

This function is the reason why I stored the cards as a vector earlier on. With this function in hand, we can now write a function to compute the number of cards seen when starting from any given card:

(define (add-counts cs)
  (apply hash-union cs #:combine +))

(define (card-counts c)
  (add-counts
   (list*
    (hasheqv (card-id c) 1)
    (map card-counts (card-wins c)))))

Running card-counts on the first card in the example input yields:

> (card-counts (vector-ref cards 0))
'#hasheqv((1 . 1) (2 . 1) (3 . 2) (4 . 4) (5 . 7))

To compute the result for part two we just have to add up all the counts for all the cards we have:

(apply + (hash-values
          (add-counts
           (for/list ([c (in-vector cards)])
             (card-counts c)))))

This works fine for the example input, but the real input is much larger and requires many more iterations. The trick to notice here is that calling card-counts on an individual card will always return the same result, so we can simply memoize the result for every card and greatly reduce the number of iterations required to compute the solution.

All we have to do is change card-counts to:

(define card-counts
  (let ([memo (make-hasheqv)])
    (lambda (c)
      (hash-ref!
       memo (card-id c)
       (lambda ()
         (add-counts
          (list*
           (hasheqv (card-id c) 1)
           (map card-counts (card-wins c)))))))))

The memo hash keeps a mapping of card ids to the number of cards seen when starting from that card. If a card's id is already in the hash, we return its associated value, otherwise we compute the result and store it in the hash for subsequent lookups.

That's it for day four!

Advent of Racket 2023/03 - Gear Ratios

Sun, 3 Dec 2023 10:00:00 +0200

Let's get right into day three. The first part of the puzzle today is to find any numbers adjacent to a non-numeric, non-period symbol in a table and add them all up. The example input looks like this:

467..114..
...*......
..35..633.
......#...
617*......
.....+.58.
..592.....
......755.
...$.*....
.664.598..

The first thing I did was count the number of columns and rows. I noticed the example input was a square, so I took a peek at my own input and it, too, was square. So, I decided to read the whole input into a unidimensional vector:

(define table
  (call-with-input-file "day03.txt"
    (lambda (in)
      (for*/vector ([line (in-lines in)]
                    [char (in-string line)])
        char))))

Since we're going to be looking for adjacent positions, we need to know what the table's stride is. Since the input table is square, that's easy:

(define s (sqrt (vector-length table)))
(define -s (- s))

Next, we can define a function to match special symbols:

(define (engine-symbol? c)
  (and (not (eqv? c #\.))
       (not (char-numeric? c))))

And a function to determine if a position has any adjacent special symbols:

(define (has-adjacent? t pos [ok? engine-symbol?])
  (for*/first ([d (in-list (list (- -s 1) -s (+ -s 1)
                                      -1           1
                                 (-  s 1)  s (+  s 1)))]
               [idx (in-value (+ pos d))]
               #:when (and (>= idx 0)
                           (<  idx (vector-length t))
                           (ok? (vector-ref t idx))))
    idx))

For any given index into the table, has-adjacent? checks its 8 adjacent indexes for a position for which ok? returns #t, returning the first match or #f if no matches are found.

To compute part one, all we have to do is iterate over the table, piece any run of digits into a number and add any number for which we've found an adjacent symbol to the total:

(for/fold ([num 0]
           [ok? #f]
           [total 0]
           #:result (if ok? (+ num total) total))
          ([(c idx) (in-indexed (in-vector table))])
  (if (char-numeric? c)
      (values (+ (* num 10) (char->decimal c))
              (or ok? (has-adjacent? table idx))
              total)
      (values 0 #f (if ok? (+ num total) total))))

The one edge case to watch out for here is that a table may end in a number, so we need to make sure the final result accounts for that and add the last-collected number to the total if necessary. That case is covered by the #:result clause above.

As in day two, the second part of today's puzzle flips the problem on its head and asks us to instead find any two numbers that are adjacent to a * symbol, multiply them together and sum them up.

To solve part two, we can just iterate over the table and whenever we see a * symbol, check if it has any adjacent numbers. If it has exactly two of them, we can multiply them together and add them to the total. It would be nice if we had a variant of has-adjacent? that returns the indexes of adjacent symbols, so let's go ahead and write that function:

(define-syntax-rule (define-finder id for*-id)
  (define (id t pos [ok? engine-symbol?])
    (for*-id ([d (in-list (list (- -s 1) -s (+ -s 1)
                                     -1           1
                                (-  s 1)  s (+  s 1)))]
              [idx (in-value (+ pos d))]
              #:when (and (>= idx 0)
                          (<  idx (vector-length t))
                          (ok? (vector-ref t idx))))
      idx)))

(define-finder find-adjacent for*/list)
(define-finder has-adjacent? for*/first)

I wanted to preserve the short-circuiting behavior of has-adjacent?, so I decided to abstract over its implementation using a macro. In this case, the define-finder macro lets us plug in different for loop behaviors into the same implementation. So, find-adjacent works the same way as has-adjacent?, but it returns all the adjacent positions for which ok? is true instead of just the first one.

One problem to consider, though, is that a symbol may be adjacent to multiple digits belonging to the same number. For example:

123
.*.
456

Given the above example, the find-adjacent procedure would return the position of every digit surrounding the * symbol. So, we need a way get the numbers at a given set of positions that is aware of this issue and skips redundant positions.

; invariant: indexes always start out as valid digit positions in the table
(define (get-numbers t is)
  (let loop ([is is]
             [nums null])
    (cond
      [(null? is) nums]
      [else
       (define-values (num rem-is)
         (let get-number ([i (car is)])
           (if (or (< i 0)
                   (not (char-numeric? (vector-ref t i))))
               (for/fold ([n 0] [rem-is is])
                         ([(c idx) (in-indexed (in-vector t (add1 i)))])
                 #:break (not (char-numeric? c))
                 (values
                  (+ (* n 10) (char->decimal c))
                  (remq (+ (add1 i) idx) rem-is)))
               (get-number (sub1 i)))))
       (loop rem-is (cons num nums))])))

The get-numbers procedure takes a table and a list of indexes into that table. It then tries the indexes in order and walks backwards from each index to find the start of a number, then it reads the number forwards, removing any indexes that it sees along the way from the pending list of indexes, and collecting the numbers into a list.

> (define t
    (vector #\1 #\2 #\.
            #\3 #\4 #\.))
> (get-numbers t '())
'()
> (get-numbers t '(0))
'(12)
> (get-numbers t '(1))
'(12)
> (get-numbers t '(0 1))
'(12)
> (get-numbers t '(0 1 3))
'(34 12)
> (get-numbers t '(0 1 3 4))
'(34 12)

With find-adjacent and get-numbers in hand, we can just iterate over the table and compute the result:

(for/fold ([total 0])
          ([(c idx) (in-indexed (in-vector table))]
           #:when (eqv? c #\*))
  (define adjacent-numbers
    (get-numbers table (find-adjacent table idx char-numeric?)))
  (if (= (length adjacent-numbers) 2)
      (+ total (apply * adjacent-numbers))
      total))

Any time we see a *, we find all the adjacent numeric positions and get the numbers at those positions. If we have exactly two adjacent numbers, then we multiply them together and add them to the sum. Easy peasy.

Advent of Racket 2023/02 - Cube Conondrum

Sat, 2 Dec 2023 10:00:00 +0200

Today's puzzle was quick and easy. For the first part, we're to take a list of "games" as input where each game has an id and a set of semicolon-separated sets of plays and report the sum of the game ids where the sets match a certain condition. The example input looks like:

Game 1: 3 blue, 4 red; 1 red, 2 green, 6 blue; 2 green
Game 2: 1 blue, 2 green; 3 green, 4 blue, 1 red; 1 green, 1 blue
Game 3: 8 green, 6 blue, 20 red; 5 blue, 4 red, 13 green; 5 green, 1 red
Game 4: 1 green, 3 red, 6 blue; 3 green, 6 red; 3 green, 15 blue, 14 red
Game 5: 6 red, 1 blue, 3 green; 2 blue, 1 red, 2 green

I decided to make a struct to represent each game:

(struct game (id sets)
  #:transparent)

And to parse each set into a hash from colors to the number of blocks:

(define (parse-set set-str)
  (for/hasheq ([reveal-str (in-list (string-split set-str ","))])
    (match-define (regexp #rx"([0-9]+) ([a-z]+)"
                          (list _
                                (app string->number blocks)
                                (app string->symbol color)))
      reveal-str)
    (values color blocks)))

The parse-set procedure splits the input on commas and extracts the block count and color of each reveal using a regular expression. The for/hasheq form then collects the results of each iteration into a hash. The match form's app syntax comes in handy when you want to transform a matched value before binding it.

> (parse-set "")
'#hasheq()
> (parse-set "10 red, 3 blue")
'#hasheq((blue . 3) (red . 10))
> (parse-set "10 red, 3 blue, 5 green")
'#hasheq((blue . 3) (green . 5) (red . 10))

To parse a game, we do a similar kind of pattern matching to extract the game id and the set of reveals to pass to parse-set:

(define (parse-game line)
  (match-define (regexp #rx"Game ([^:]+): (.+)"
                        (list _ (app string->number id) sets-str))
    line)
  (define sets
    (map parse-set (string-split sets-str ";")))
  (game id sets))

With the id and parsed sets in hand, we construct an instance of the game struct.

> (parse-game "Game 5: 10 red, 3 blue; 10 green, 5 red")
(game 5 '(#hasheq((blue . 3) (red . 10))
          #hasheq((green . 10) (red . 5))))

The goal of part one is to sum up the game ids where the games would be valid. A valid game is defined as any game where every set of reveals had fewer than 12 red cubes, 13 green cubes and 14 blue cubes. So, I defined a generic procedure for determining if a game was valid:

(define (game-ok? g proc)
  (for/and ([s (in-list (game-sets g))])
    (proc s)))

The for/and form returns #t when all of the iterations are truthy and #f otherwise. Next, we define a procedure representing the valid condition for part 1:

(define (part1-ok? s)
  (and
   (<= (hash-ref s 'red 0) 12)
   (<= (hash-ref s 'green 0) 13)
   (<= (hash-ref s 'blue 0) 14)))

And, with that, we can put everything together:

(call-with-input-file "day02.txt"
  (lambda (in)
    (for*/sum ([line (in-lines in)]
               [game (in-value (parse-game line))]
               #:when (game-ok? game part1-ok?))
      (game-id game))))

We use the sum variant of for* to sum up the valid game ids. The in-value form generates a sequence of one element representing each game for each line and the body of the loop is skipped whenever game-ok? is false.

In part two, the problem is flipped on its head and we're asked to find the minimum constraint that would make each game valid. That's just a matter of iterating over every set in each game and keeping track of the maximum number of blocks of each color:

(define (game-minimums g)
  (for*/fold ([minimums (hasheq 'red 0 'green 0 'blue 0)])
             ([s (in-list (game-sets g))]
              [c (in-list '(red green blue))])
    (hash-update minimums c (λ (blocks) (max blocks (hash-ref s c 0))))))

For example:

> (game-minimums (parse-game "Game 5: 10 red, 3 blue; 10 green, 5 red"))
'#hasheq((blue . 3) (green . 10) (red . 10))

To compute the puzzle solution, we have to sum up the result of multiplying the number of colors in every game (its "power"). So, we define a procedure to compute a game's power:

(define (game-power g)
  (apply * (hash-values (game-minimums g))))

Put it all together:

(call-with-input-file "day02.txt"
  (lambda (in)
    (for*/sum ([line (in-lines in)]
               [game (in-value (parse-game line))])
      (game-power game))))

And that's it for day two!

Advent of Racket 2023/01 - Trebuchet?!

Fri, 1 Dec 2023 10:00:00 +0200

The 2023 Advent of Code advent calendar has started and I'm doing it in Racket again this year. I'll probably stick with it for a couple of weeks or until the puzzles start taking me more than 10-15 minutes to finish. I've also decided to write some short posts about my solutions, so here's the first one.

The first part of today's challenge is to take a list of strings, combine the first and last decimal digit in each string into a decimal number and sum them all up. The example input looks like:

1abc2
pqr3stu8vwx
a1b2c3d4e5f
treb7uchet

In Racket, we can open a file as an input port using call-with-input-file and iterate over its lines using in-lines.

(call-with-input-file "day01.txt"
 (lambda (in)
   (for/sum ([line (in-lines in)])
     ...)))

The for/sum variant of for returns the sum of all intermediate iteration results. With the above skeleton in place, all we have to do is extract the first and last digit in every line and convert them to a number (the so-called "calibration value" in the problem statement).

We're going to need to convert characters to digits, so let's first write a helper procedure to do that.

(define (get-digit s i)
  (define c (string-ref s i))
  (and (char-numeric? c)
       (- (char->integer c)
          (char->integer #\0))))

The get-digit procedure takes a string and an index into that string, grabs the character at the given index and converts it into a decimal digit if it is numeric. We're only interested in the ASCII character set for this particular problem, so there's no need to worry about other unicode numerals.

Next, we can define the procedure to extract the calibration value from a line.

(define (calibration-value s)
  (define-values (d0 d1)
    (for/fold ([d0 #f]
               [d1 #f])
              ([i (in-range 0 (string-length s))])
      (define digit (get-decimal-digit s i))
      (values (or d0 digit)
              (or digit d1))))
  (+ (* d0 10) d1))

It uses the for/fold form to iterate over all the indices in a given string and keep track of the first and last-seen digits. Finally, it combines the two digits into a decimal number.

We can plug the two helpers into our skeleton to solve part one:

(call-with-input-file "day01.txt"
  (lambda (in)
    (for/sum ([line (in-lines in)])
      (calibration-value line))))

For part two, we get a new example and the puzzle gets a little bit harder. We now need to account for digits that are spelled out on each line. The example input looks like:

two1nine
eightwothree
abcone2threexyz
xtwone3four
4nineeightseven2
zoneight234
7pqrstsixteen

As in the first part, let's first write a helper to extract digits out of a string.

(define digit-rxs
  (for/list ([s (in-list '(zero one two three four five six seven eight nine))])
    (regexp (format "^~a" s))))

(define (get-spelled-out-digit s i)
  (or
   (for/first ([(rx n) (in-indexed (in-list digit-rxs))]
               #:when (regexp-match? rx s i))
     n)
   (get-decimal-digit s i)))

Like get-decimal-digit, the get-spelled-out-digit procedure takes a string and a starting index into that string. It then iterates over a list of regular expressions to return the index of the first match. The regexp-match? procedure takes an optional starting index that tells it where in the given string it should start matching from¹. If none of the regular expressions match, the for/first form produces #f and we fall back to the get-decimal-digit procedure.

Now, we can update calibration-value to take its digit-extracting procedure as an argument instead of calling get-decimal-digit directly.

(define (calibration-value s [get-digit get-decimal-digit])
  (define-values (d0 d1)
    (for/fold ([d0 #f]
               [d1 #f])
              ([i (in-range 0 (string-length s))])
      (define digit (get-digit s i))
      (values (or d0 digit)
              (or digit d1))))
  (+ (* d0 10) d1))

Since the get-digit argument to calibration-value defaults to get-decimal-digit, we don't need to change our solution to part one to account for this refactoring. The solution to part two is the same as part one, but we pass the get-spelled-out-digit procedure to calibration-value:

(call-with-input-file "day01.txt"
  (lambda (in)
    (for/sum ([line (in-lines in)])
      (calibration-value line get-spelled-out-digit))))

That's it for day one!

Why not just pass in a substring? I prefer to avoid copying where possible and many Racket built-ins that work on sequences support this pattern of passing in beginning and end indices. ↩

Distributing Apps to the Mac App Store With GitHub Actions

Sun, 22 Oct 2023 23:00:00 +0300

This weekend, I decided to try and get Franz published on the Mac App Store. Since I'm already using GitHub Actions to build distributions for my customers, I figured I'd extend that same workflow to handle build submissions to the Mac App Store.

Most of the steps are the same as for manual distribution, but some details are different and documentation is sparse, so I wanted to jot down some notes.

You can find the workflow definition on GitHub.

Apple Developer Certificates

Mac App Store apps are distributed as .pkg installers, so I generated a "Mac Installer Distribution" certificate in Xcode and exported it. I also needed an "Apple Distribution" certificate and a "Mac Development" certificate. As in my previous post, these are stored as base64-encoded secrets that are then added to a keychain during the workflow run.

Provisioning Profile

This mode of distribution requires a provisioning profile. I tried generating a profile from the Certificates, Identifiers & Profiles section of the Apple Developer site, but I couldn't get Xcode on the runner to recognize the profile no matter what I tried. So, I gave up and I copied the provisioning profile Xcode generated from ~/Library/MobileDevice/Provisioning Profiles and added it as a base64-encoded secret under GHA. The workflow writes the secret to the provisioning profiles folder on the build machine:

mkdir -p ~/Library/MobileDevice/Provisioning\ Profiles
echo -n "$MAC_PROVISIONING_PROFILE" | \
  base64 --decode -o ~/Library/MobileDevice/Provisioning\ Profiles/franz.provisionprofile

Building the App

To build the app, I run the xcodebuild archive command:

xcodebuild \
  archive \
  -project FranzCocoa.xcodeproj/ \
  -scheme 'Franz MAS' \
  -destination 'generic/platform=macOS' \
  -archivePath dist/Franz.xcarchive

And to generate the .pkg, I run xcodebuild with the -exportArchive flag:

xcodebuild \
  -exportArchive \
  -archivePath dist/Franz.xcarchive \
  -exportOptionsPlist FranzCocoa/MASExportOptions.plist \
  -exportPath dist/

The exportOptionsPlist file has to have a method key whose value is app-store. When this method is specified, xcodebuild implicitly exports a .pkg instead of an .app directory. All other options are the same as in the developer ID method.

Finally, to upload the build to App Store Connect, I use altool:

xcrun altool \
  --upload-package dist/Franz.pkg \
  --type macos \
  --asc-public-id '69a6de7a-5947-47e3-e053-5b8c7c11a4d1' \
  --apple-id '6470144907' \
  --bundle-id 'io.defn.Franz' \
  --bundle-short-version-string "$(/usr/libexec/PlistBuddy -c 'Print ApplicationProperties:CFBundleShortVersionString' dist/Franz.xcarchive/Info.plist)" \
  --bundle-version "$(/usr/libexec/PlistBuddy -c 'Print ApplicationProperties:CFBundleVersion' dist/Franz.xcarchive/Info.plist)" \
  --username 'bogdan@defn.io' \
  --password "$APPLE_ID_PASSWORD"

To find my public ID, I had to run xcrun altool --list-providers. To get an Apple ID for my app, I had to first create it in App Store Connect. The ID can be found under "General" -> "App Information".

Release Process

Since every submitted build must have a unique build number, I made this part of the workflow optional. It only runs when the workflow is run manually and the masBuild flag is set. So, my release process for the Mac App Store is:

Bump the build number.
Run the workflow manually.
Go to App Store Connect and publish the new version.

Franz for Windows

Sun, 15 Oct 2023 10:00:00 +0300

Today marks the first beta release of Franz for Windows!

Franz is a desktop client for Apache Kafka.

I had been planning to work on a Windows version of Franz since last year. Originally, I was going to implement the UI in C# (or possibly F# after a strong recommendation from Michael Sperber), but then Ben and I wrote a paper ¹ about my declarative wrapper around racket/gui earlier this year and that inspired me to try using it to implement the Windows (and, soon, Linux!) version.

On macOS, Franz is implemented using a combination of Racket for the core logic and Swift for the UI. The Windows version reuses the core and implements all the app's views afresh using gui-easy. The end result is about 5k lines of GUI code, including many small component "tests" that let me easily exercise views in isolation (think SwiftUI Previews, but, well, functional² and interactive). For comparison, the Swift version has about 8.2k lines of Swift code — a mix of AppKit and SwiftUI — and about 4.5k of XML representing XIBs built using the Xcode Interface Builder. Being that racket/gui is a cross-platform library, and therefore it has to target the least common denominator of the three platforms it supports in terms of features, the end result isn't as polished as the Mac version or as a Windows Forms³ version could be, but it gets reasonably close.

Since the source is all Racket, distribution is trivial: just call raco exe to generate an executable and then raco dist to package it up. The only snag has been Microsoft's SmartScreen. I'm not willing to pay for an EV certificate⁴ from one of the blessed vendors, so I need to manually submit new Windows releases to Microsoft to check for malware, but that hasn't been a huge hassle so far, apart from the processing times.

In my original announcement post for Franz I had mentioned that I think this kind of separation between backend and frontend is a very productive way of developing desktop apps and I think that bears out here. I only had to make minimal changes to the core code in order to support this new version and I was able to get everything done in about two months' worth of working in my free time. I think a small team using this model can move very fast and build great native apps for every platform.

Franz is source available so you can take a peek at the implementation yourself if you're interested. If you use Franz for your work, I'd love to get your feedback!

See also Ben's talk at ICFP. ↩
As in "working," not the programming paradigm. ↩
Or whatever the current Windows GUI framework du jour is. ↩
What a racket! ↩

Powered by •

Mon, 25 Sep 2023 20:30:00 +0300

Following in the long tradition of programmers writing blog engines instead of blogging, this site is now powered by Joel Dueck's Punct.

I'd been meaning to make the switch ever since Joel posted a video of Punct in Practice to the Racket Discord and I finally took the plunge this weekend. You can find the source code for the new site on GitHub.

The old site was implemented using Hugo and it served me well, but I never learned how to use Hugo properly because I could never make my way effectively around their documentation, so I was always a bit wary of making changes to the structure of the site (for example, making my own theme and other kinds of enhancements).

Punct is similar to other Racket publishing tools like Pollen and Scribble in that documents (i.e. pages) are just Racket modules with a reader that makes it convenient to input regular markup, but that lets you escape to and use inline Racket when necessary. This means that, at any point in a document, you can fall back on the full power of Racket. It's hard to overstate how nice this model is so I highly encourage you to check out Joel's video and see for yourself.

Restoring the Old Dashboard Feed on GitHub

Sat, 23 Sep 2023 13:00:00 +0300

A couple weeks ago, GitHub changed its Dashboard feed implementation and this new version has a lot less relevant information with respect to the repositories I follow compared to the old one. Fortunately, the old feed is still available at /dashboard-feed. So, for now, you can restore the old functionality using this quick and dirty user script:

document.addEventListener("DOMContentLoaded", () => {
  const $news = document.querySelector("#dashboard .news");
  if (!$news) return;
  let node = $news.firstChild;
  while (node !== null) {
    $news.removeChild(node);
    node = $news.firstChild;
  }

  const req = new XMLHttpRequest();
  req.open("GET", "/dashboard-feed", true);
  req.addEventListener("load", (e) => {
    let $frame = document.createElement("iframe");
    $frame.style.display = "none";
    $frame.onload = () => {
      $news.appendChild($frame.contentDocument.querySelector(".application-main"));
    };
    $frame.srcdoc = req.responseText;
    $news.appendChild($frame);
  });
  req.send(null);
})

In Arc, you can enable this script by going to github.com, creating a new boost and placing the script in the boost's code section. In Firefox, you should be able to use a plugin like Tampermonkey to achieve the same thing.

Hopefully, they'll keep the old feed around for a while, or bring the new feed's functionality up to par with the old one in future.

Distributing Mac Apps With GitHub Actions

Fri, 22 Sep 2023 21:00:00 +0300

This week, I spent some time automating the build & distribution process for Franz and I wanted to jot down some quick notes about how it works. Most of the steps are not specific to GitHub Actions so you could replace it by your favorite CI.

The Workflow

Take a look at the workflow to follow along.

The build_core_arm64 and build_core_x86_64 jobs are concerned with building the Racket core of the application and are relatively uninteresting: install Racket, install the core dependencies, and compile an object file with the core implementation. Finally, upload the core objects and supporting files for use in build_app.

The build_app job first downloads the core objects and installs a Swift package the application depends on, then proceeds to build the app, create a disk image containing the app, notarize the image, and, finally, save the notarized .dmg.

Apple Developer Certificates

This part is based on GitHub's own documentation for "Installing an Apple certificate on macOS runners for Xcode development", though I found I didn't need to export a provisioning profile and could just rely on Xcode to automatically handle that for me.

I distribute the app using my Apple Developer ID (i.e. folks download a .dmg file directly from my website, not via the Mac App Store), so I had to generate a couple certificates to use with the workflow. I did this directly from Xcode by going to "Settings" -> "Accounts" -> "Manage Certificates...".

I created a new "Apple Development Certificate" and a new "Developer ID Application Certificate", then exported both to disk and assigned each a strong password.

I converted the certificates to base64 and stored them as GitHub Secrets under my repository's settings. To make the certificates available to Xcode during workflow runs, I create a keychain and import the certificates into it:

MAC_DEV_CER_PATH=$RUNNER_TEMP/madev.p12
DEVELOPER_ID_CER_PATH=$RUNNER_TEMP/devid.p12
KEYCHAIN_PATH=$RUNNER_TEMP/app-signing.keychain-db
echo -n "$MAC_DEV_CER" | base64 --decode -o $MAC_DEV_CER_PATH
echo -n "$DEVELOPER_ID_CER" | base64 --decode -o $DEVELOPER_ID_CER_PATH
security create-keychain -p "$KEYCHAIN_PASSWORD" $KEYCHAIN_PATH
security set-keychain-settings -lut 21600 $KEYCHAIN_PATH
security unlock-keychain -p "$KEYCHAIN_PASSWORD" $KEYCHAIN_PATH
security import $MAC_DEV_CER_PATH -P "$MAC_DEV_CER_PASSWORD" -A -t cert -f pkcs12 -k $KEYCHAIN_PATH
security import $DEVELOPER_ID_CER_PATH -P "$DEVELOPER_ID_CER_PASSWORD" -A -t cert -f pkcs12 -k $KEYCHAIN_PATH
security list-keychain -d user -s $KEYCHAIN_PATH

Building the App

To build the app, I first run xcodebuild to generate an .Xcarchive of the app's compiled objects and runtime support files. Just run the archive subcommand with the Xcode scheme to build and input and output paths:

xcodebuild \
  archive \
  -project FranzCocoa.xcodeproj/ \
  -scheme Franz \
  -destination 'generic/platform=macOS' \
  -archivePath dist/Franz.xcarchive

Next, I run xcodebuild again to export the archive to an .app:

xcodebuild \
  -exportArchive \
  -archivePath dist/Franz.xcarchive \
  -exportOptionsPlist FranzCocoa/ExportOptions.plist \
  -exportPath dist/ \
  -allowProvisioningUpdates

Figuring out the contents of the ExportOptions.plist file was a bit tricky. The set of available options is printed at the end of the output for xcodebuild -help. The right combination of options for my app turned out to be:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>compileBitcode</key>
    <false/>
    <key>method</key>
    <string>developer-id</string>
    <key>signingStyle</key>
    <string>automatic</string>
    <key>stripSwiftSymbols</key>
    <true/>
    <key>teamID</key>
    <string>H3YE679B58</string>
    <key>thinning</key>
    <string>&lt;none&gt;</string>
  </dict>
</plist>

Once the app is exported, I use create-dmg to create a nice-looking disk image to distribute it with and then proceed to notarization. I considered building up the image manually using hdiutil, but generating output as nice as what create-dmg produces is relatively hard (and involves, for example, editing .DS_Store files), so create-dmg it is.

Notarizing the App

To notarize the app, I use Xcode's notarytool utility:

xcrun notarytool submit \
  --team-id 'H3YE679B58' \
  --apple-id 'bogdan@defn.io' \
  --password "$NOTARY_PASSWORD" \
  --wait \
  dist/Franz.dmg

In order to make notarization requests from within the workflow, I had to create an app-specific password using the Apple ID website.

Once notarization succeeds, I run the stapler utility to staple the notarization onto the disk image:

xcrun stapler staple dist/Franz.dmg

And that's it. The final step after this is just to upload the image artifact so I can grab it and manually¹ release it when I'm ready.

A process I'll automate some other time. ↩

Franz now Source Available

Thu, 10 Aug 2023 08:20:00 +0300

The source code for Franz, a desktop client for Apache Kafka, is now available for all to read on on GitHub.

One of my goals with writing software in Racket is to help expand the Racket ecosystem. I try do that by making parts of the apps I write Open Source where possible¹ and by making small contributions back to Racket itself. Additionally, I sometimes see new users ask for examples of real-world applications built with Racket so I've been trying to make the source code of my own apps available as well (where possible; see also Nemea and Remember). My hope is that folks interested in using Racket can get an idea of what using it in practice looks like by viewing the code for these apps and I like the thought of letting my own users see what code they're running when they use my apps.

In the process of writing Franz, I've written a Kafka client, libraries for serializing and deserializing Avro, MessagePack and Protocol Buffer data, native decompressors for LZ4 and Snappy, and a hash-lang for Lua among other things. ↩

Deploying Racket Web Apps With Docker

Fri, 19 May 2023 07:00:00 +0300

Since there was a question about deploying Racket code using Docker on the Racket Reddit this morning, I figured I'd write a quick follow-up to my post about Deploying Racket Web Apps. My preference is still to avoid using Docker and just use the method described in that post by default. But, if I have to use Docker for some reason, then I'll typically use a two-stage build to build a distribution in the first stage and then copy that distribution into the second stage in order to get a minimal Docker image that I can easily ship around.

Given the following app, saved as app.rkt:

#lang racket/base

(require racket/async-channel
         web-server/http
         web-server/servlet-dispatch
         web-server/web-server)

(define ch (make-async-channel))
(define stop
  (serve
   #:dispatch (dispatch/servlet
               (lambda (_req)
                 (response/xexpr
                  '(h1 "Hello!"))))
   #:port 8000
   #:listen-ip "0.0.0.0"
   #:confirmation-channel ch))

(define ready-or-exn (sync ch))
(when (exn:fail? ready-or-exn)
  (raise ready-or-exn))

(with-handlers ([exn:break? void])
  (sync/enable-break never-evt))

(stop)

I would write the following Dockerfile:

FROM racket/racket:8.9-full AS build

COPY app.rkt /code/
WORKDIR /code
RUN raco exe -o app app.rkt
RUN raco dist dist app

FROM debian:bullseye-slim AS final

RUN apt-get update -y && \
    apt-get install -y --no-install-recommends dumb-init && \
    rm -rf /var/lib/apt/lists/*

COPY --from=build /code/dist /app
CMD ["dumb-init", "/app/bin/app"]

When this image gets built, the build stage creates a distribution of the app that gets copied into the final stage. At the end, the build stage is discarded and the end result is a roughly 150MB Docker image with just my code in it and a minimal Debian system. Not quite as minimal as you can get out of using a similar method with Go, but Racket distributions have a high baseline, so a real app wouldn't be much bigger than this.

Breaking Racket

Fri, 31 Mar 2023 15:00:00 +0300

If you've looked at some of the concurrency and networking procedures available in Racket, you might've noticed a bunch that follow the naming pattern <name>/enable-break and it might not have been immediately obvious why or when you would use these variants of these procedures over the regular ones. Let's look at a snippet from the documentation of tcp-accept/enable-break:

If breaking is disabled when tcp-accept/enable-break is called, then either ports are returned or the exn:break exception is raised, but not both.

So, the procedure guarantees that either it will accept a connection and return a pair of ports, or it will raise a break exception, if breaking is disabled when it is called. Sounds straightforward enough, but it still might not be clear where this would come in handy. The part of the quote about "if breaking is disabled" is an important hint. Consider this simplified piece of code based on my previous post re. the protohackers challenge:

(define listener (tcp-listen 8000))
(with-handlers ([exn:break? void])
  (let loop ()
    (parameterize-break #f
      (define-values (in out)
        (tcp-accept listener))                     ;; •1
      (define conn-thd
        (thread (make-connection-handler in out))) ;; •2
      (thread (make-supervisor conn-thd in out)))  ;; •3
    (loop)))
(tcp-close listener)

We set up a listener, then enter a loop to start accepting new connections and spawn threads to handle every connection. For every connection-handling thread, we spawn a supervising thread to clean up resources once the handling thread is done.

We want to ensure that a handling thread gets spawned for every accepted connection, so we want to guarantee that no breaks can be received between •1 and •2. Likewise, we want to guarantee that every handler thread has an associated supervision thread, so no breaks should be allowed between •2 and •3 either. So, we've wrapped •1, •2, and •3 in the parameterize-break form to disable breaks. However, accepting a new connection will block the thread until a client actually tries connecting to the server. This means that we can only stop this server if we happen to send it a break at exactly the time after •3 returns, or by killing the process.

By replacing the call to tcp-accept with tcp-accept/enable-break, we can preserve the same guarantees as before while also allowing breaks to be raised before new connections are accepted.

Protohackers Challenge in Racket Part 2

Sun, 26 Mar 2023 07:15:00 +0300

I'm currently stuck waiting for a flight, so I figured I'd pick up where I left off yesterday ¹ and implement a couple more of the Protohackers challenges.

`run-server`

Since the accept loop was common between all the challenges so far, I decided to extract and reuse it:

(define (run-server host port handle
                    #:backlog [backlog 511]
                    #:reuse? [reuse? #t]
                    #:timeout-evt-proc [make-timeout-evt values]
                    #:listen-proc [listen tcp-listen]
                    #:accept-proc [accept tcp-accept/enable-break]
                    #:close-proc [close tcp-close])
  (define listener
    (listen port backlog reuse? host))
  (define server-custodian
    (make-custodian))
  (define server-thd
    (parameterize ([current-custodian server-custodian])
      (thread
       (lambda ()
         (with-handlers ([exn:break? void])
           (let loop ()
             (parameterize-break #f
               (define-values (in out)
                 (accept listener))
               (define client-custodian
                 (make-custodian))
               (define client-thd
                 (thread
                  (lambda ()
                    (break-enabled #t)
                    (parameterize ([current-custodian client-custodian])
                      (handle in out)))))
               (thread
                (lambda ()
                  (sync (make-timeout-evt client-thd))
                  (close-output-port out)
                  (close-input-port in)
                  (custodian-shutdown-all client-custodian))))
             (loop)))
         (close listener)))))
  (lambda ()
    (break-thread server-thd)
    (thread-wait server-thd)
    (custodian-shutdown-all server-custodian)))

This version abstracts over the TCP procedures (you'll see why in a bit), runs the accept loop in a thread and returns a procedure that can be used to stop the server. For convenience, I also wrote a version that blocks the calling thread until a break is received:

(define run-server*
  (make-keyword-procedure
   (lambda (kws kw-args . args)
     (define stop (keyword-apply run-server kws kw-args args))
     (with-handlers ([exn:break? void])
       (sync never-evt))
     (stop))))

With this in place, the 0th challenge from yesterday now looks like this:

#lang racket/base

(require racket/match)

(define (handle in out)
  (let loop ()
    (match (read-bytes 4096 in)
      [(? eof-object?)
       (void)]
      [bs
       (write-bytes bs out)
       (loop)])))

(module+ main
  (require "common.rkt")
  (run-server* "0.0.0.0" 8111 handle))

4: Unusual Database Program

This challenge switches things up and uses UDP instead of TCP.

#lang racket/base

(require racket/match
         racket/port)

(define db
  (make-hash `(("version" . "Ken's Key-Value Store 1.0"))))

(define (handle in out)
  (match (port->string in)
    [(regexp #rx"^([^=]*)=(.*)$" (list _ key value))
     (unless (equal? key "version")
       (hash-set! db key value))]
    [key
     (define value (hash-ref db key #f))
     (display key out)
     (when value
       (display "=" out)
       (display value out))]))

(module+ main
  (require racket/udp
           "common.rkt")

  (define (udp-listen port _backlog reuse? host)
    (define socket (udp-open-socket))
    (begin0 socket
      (udp-bind! socket host port reuse?)))

  (define (udp-accept listener)
    (define buf (make-bytes 65536))
    (parameterize-break #f
      (define-values (len hostname port)
        (udp-receive!/enable-break listener buf))
      (define client-in (open-input-bytes (subbytes buf 0 len)))
      (define-values (pipe-in client-out)
        (make-pipe))
      (thread
       (lambda ()
         (let loop ()
           (define len (read-bytes! buf pipe-in))
           (unless (eof-object? len)
             (udp-send-to listener hostname port buf 0 len)
             (loop)))))
      (values client-in client-out)))

  (run-server* "0.0.0.0" 8111 handle
               #:listen-proc udp-listen
               #:accept-proc udp-accept
               #:close-proc udp-close))

Since run-server now abstracts over the listen, accept, and close procedures, we can massage the UDP API to fit into the run-server model. To listen, we bind a socket to the given host and port. To accept, we receive a packet, wrap the data into an input port and use a pipe to pump responses back to the client through the same UDP socket.

The handle procedure is straightforward: it uses a shared hash to store and retrieve the entries. Hashes are thread-safe so it's fine to share the same hash between clients given that we don't have to worry about data races for this particular problem.

5: Mob in the Middle

Back to TCP, this time implementing a MITM attack.

#lang racket/base

(require racket/tcp)

(define (handle in out)
  (define-values (proxied-in proxied-out)
    (tcp-connect "chat.protohackers.com" 16963))
  (thread
   (lambda ()
     (pump proxied-in out)))
  (pump in proxied-out))

(define (pump in out)
  (define buf (make-bytes 4096))
  (let loop ([data #""])
    (define len
      (read-bytes-avail! buf in))
    (cond
      [(eof-object? len)
       (write-bytes (rewrite data) out)
       (flush-output out)]
      [else
       (loop (drain (bytes-append data (subbytes buf 0 len)) out))])))

(define (drain data out)
  (let loop ([data data])
    (cond
      [(bytes-index-of data 10)
       => (λ (idx)
            (write-bytes (rewrite (subbytes data 0 idx)) out)
            (write-byte 10 out)
            (flush-output out)
            (loop (subbytes data (add1 idx))))]
      [else data])))

(define (rewrite data)
  (regexp-replace*
   #px#"(.?)(7[a-zA-Z0-9]{25,34})(.?)" data
   (λ (bs pre _addr post)
     (cond
       [(and (member pre '(#"" #" "))
             (member post '(#"" #" ")))
        (bytes-append pre #"7YWHMfk9JZe0LM0g1ZauHuiSxhI" post)]
       [else bs]))))

(define (bytes-index-of bs b)
  (for/first ([o (in-bytes bs)]
              [i (in-naturals)]
              #:when (= o b))
    i))

(module+ main
  (require "common.rkt")
  (run-server* "0.0.0.0" 8111 handle))

This is the first challenge where we act as a TCP client. For every connection, we make our own connection to the upstream chat server. Then, we pipe all output from the chat server to the client, rewriting any lines containing cryptocurrency addresses. We do the same for input from the client to the chat server.

The tricky part about this challenge is that we can't just use read-line because the client might terminate the connection in the middle of writing a line and read-line makes no distinction between the end of input and the end of a line. So, we pump input from one side to the other in 4k chunks, searching for newlines after every chunk and rewriting complete lines.

6: Speed Daemon

This challenge cranks up the difficulty, so I'll split the solution into sections below.

Common Bits

The imports and a logger are shared between the three sections.

#lang racket/base

(require (for-syntax racket/base
                     racket/syntax
                     syntax/parse)
         racket/match
         racket/math
         racket/random
         threading
         (prefix-in proto: "006.bnf"))

(define-logger protohackers)

Message Parsing

Once again, I've opted to use binfmt to handle the message parsing, but this time around I've decided to parse the data into actual structs.

(define *message-readers*
  (make-hasheqv))

(define-values (prop:message-writer _prop:message-writer? message-writer)
  (make-struct-type-property 'message-writer))

(define-syntax (define-message stx)
  (syntax-parse stx
    [(_ message-id:id
        ([field-id:id field-sym:id] ...)
        #:tag tag-number:expr
        {~alt
         {~optional {~seq #:parser parser-proc}}
         {~optional {~seq #:unparser unparser-proc}}} ...)
     #:with (field-accessor-id ...) (for/list ([field-id-stx (in-list (syntax-e #'(field-id ...)))])
                                      (format-id field-id-stx "~a-~a" #'message-id field-id-stx))
     #:with read-message-id (format-id #'message-id "read-~a" #'message-id)
     #:with write-message-id (format-id #'message-id "write-~a" #'message-id)
     #:with default-parser-id (format-id #'message-id "proto:~a" #'message-id)
     #:with default-unparser-id (format-id #'message-id "proto:un-~a" #'message-id)
     #'(begin
         (struct message-id (field-id ...)
           #:transparent
           #:property prop:message-writer
           (λ () write-message-id))
         (define (read-message-id in)
           (define data ({~? parser-proc default-parser-id} in))
           (define msg (message-id (and~> (assq 'field-sym data) cdr) ...))
           (begin0 msg
             (log-protohackers-debug "read ~s" msg)))
         (define (write-message-id m out)
           (log-protohackers-debug "write ~s" m)
           (define msg `((num_1 . ,tag-number) (field-sym . ,(field-accessor-id m)) ...))
           ({~? unparser-proc default-unparser-id} msg out)
           (flush-output out))
         (hash-set! *message-readers* tag-number read-message-id))]))

(define-message Error
  ([message Str_1])
  #:tag #x10)

(define-message Plate
  ([plate PlateStr_1]
   [timestamp Timestamp_1])
  #:tag #x20)

(define-message Ticket
  ([plate PlateStr_1]
   [road Road_1]
   [mile1 Mile_1]
   [timestamp1 Timestamp_1]
   [mile2 Mile_2]
   [timestamp2 Timestamp_2]
   [speed Speed_1])
  #:tag #x21)

(define-message WantHeartbeat
  ([interval Interval_1])
  #:tag #x40)

(define-message Heartbeat ()
  #:tag #x41
  #:unparser (λ (_ out)
               (write-byte #x41 out)
               (flush-output out)))

(define-message Camera
  ([road Road_1]
   [mile Mile_1]
   [limit Limit_1])
  #:tag #x80
  #:parser proto:IAmCamera
  #:unparser proto:un-IAmCamera)

(define-message Dispatcher
  ([roads Road_1])
  #:tag #x81
  #:parser proto:IAmDispatcher
  #:unparser proto:un-IAmDispatcher)

(define (read-message in)
  (match (peek-byte in)
    [(? eof-object?) eof]
    [tag-number
     (with-handlers ([exn:fail?
                      (λ (e)
                        (begin0 #f
                          (log-protohackers-error "failed to read message with tag ~s: ~a" tag-number (exn-message e))))])
       ((hash-ref *message-readers* tag-number) in))]))

(define (write-message m out)
  (((message-writer m)) m out))

(define (write-message* m out)
  (with-handlers ([exn:fail?
                   (λ (e)
                     (log-protohackers-warning "failed to write message: ~s~n  error: ~a" m (exn-message e)))])
    (write-message m out)))

(define (write-error msg out)
  (write-message* (Error (string->bytes/utf-8 msg)) out))

The define-message form generates a struct with the given fields, a reader procedure that parses data off of a port and destructures it onto a struct instance and a writer procedure that does the reverse. It then registers the reader procedure with the global read registry by tag and associates the write procedure with the struct using a custom struct type property.

The read-message and write-message procedures use the aforementioned registry and struct type property to generically read and write messages.

Dispatcher

Similar to challenge 3, we keep track of shared state using an actor implemented with a thread and a channel.

(struct dispatcher (ch thd))

(define (make-dispatcher)
  (define ch (make-channel))
  (define thd
    (thread
     (lambda ()
       (let loop ([st (make-state)])
         (let-values ([(tickets ticketed-st) (get-tickets st)])
           (cond
             [(null? tickets)
              (apply
               sync
               (handle-evt
                ch
                (lambda (msg)
                  (log-protohackers-debug "handling dispatcher message: ~s" msg)
                  (match msg
                    [`(add-port ,out ,res-ch ,nack)
                     (define-values (id next-st)
                       (add-port st out))
                     (loop (add-request next-st `(,id ,res-ch ,nack)))]
                    [`(add-dispatcher ,id ,roads ,res-ch ,nack)
                     (loop (~> (add-dispatcher st id roads)
                               (add-request `(ok ,res-ch ,nack))))]
                    [`(remove-port ,id ,res-ch ,nack)
                     (loop (~> (remove-port st id)
                               (add-request `(ok ,res-ch ,nack))))]
                    [`(update-heartbeat ,id ,interval ,res-ch ,nack)
                     (loop (~> (update-heartbeat st id interval)
                               (add-request `(ok ,res-ch ,nack))))]
                    [`(snap ,plate-number ,road ,mile ,limit ,timestamp ,res-ch ,nack)
                     (loop (~> (track-plate st plate-number road mile limit timestamp)
                               (add-request `(ok ,res-ch ,nack))))]
                    [_
                     (log-protohackers-error "unexpected dispatcher message: ~s~n" msg)])))
               (handle-evt
                (heartbeat-evt st)
                (lambda (id interval)
                  (define out (ref-port st id))
                  (write-message* (Heartbeat) out)
                  (loop (update-heartbeat st id interval))))
               (append
                (for/list ([req (in-list (state-requests st))])
                  (match-define `(,res ,res-ch ,_) req)
                  (handle-evt
                   (channel-put-evt res-ch res)
                   (lambda (_)
                     (loop (remove-request st req)))))
                (for/list ([req (in-list (state-requests st))])
                  (match-define `(,_ ,_ ,nack) req)
                  (handle-evt nack (λ () (loop (remove-request st req)))))))]
             [else
              (for ([ticket (in-list tickets)])
                (define road (Ticket-road ticket))
                (define id (random-ref (hash-ref (state-dispatchers st) road)))
                (define out (ref-port st id))
                (write-message* ticket out))
              (loop ticketed-st)]))))))
  (dispatcher ch thd))

(define (make-dispatcher-evt d command . args)
  (match-define (dispatcher ch thd) d)
  (define res-ch (make-channel))
  (nack-guard-evt
   (lambda (nack)
     (begin0 res-ch
       (thread-resume thd (current-thread))
       (channel-put ch `(,command ,@args ,res-ch ,nack))))))

The actor receives messages on a channel and updates its internal state accordingly, checking to see if it has to dispatch any tickets after every update. When it has tickets to dispatch, it first writes them all out to their destination dispatchers and then resumes handling messages.

The actor's state is implemented using a set of structs and several functions that operate on them:

(struct heartbeat (deadline interval))
(struct observation (mile limit timestamp) #:transparent)
(struct plate (observations-by-road ticket-days) #:transparent)
(struct state
  (id-seq      ;; nonnegative-integer
   ports       ;; id -> output-port
   heartbeats  ;; id -> heartbeat
   dispatchers ;; road -> listof id
   plates      ;; plate-number -> listof plate
   requests    ;; listof (res res-ch nack)
   )
  #:transparent)

(define (make-plate)
  (plate (hasheqv) ;; observations-by-road
         (hasheqv) ;; ticket-days
         ))

(define (add-observation p road obs)
  (match-define (plate roads _) p)
  (define next-roads
    (hash-update
     roads road
     (λ (obss)
       (cons obs obss))
     null))
  (struct-copy plate p [observations-by-road next-roads]))

(define (add-ticket-days p timestamp1 timestamp2)
  (define ticket-days
    (for/fold ([ticket-days (plate-ticket-days p)])
              ([d (in-range (->day timestamp1)
                            (add1 (->day timestamp2)))])
      (hash-set ticket-days d #t)))
  (struct-copy plate p [ticket-days ticket-days]))

(define (ticketed-on? p timestamp)
  (hash-has-key?
   (plate-ticket-days p)
   (->day timestamp)))

(define (make-state)
  (state 0         ;; id-seq
         (hasheqv) ;; ports
         (hasheqv) ;; heartbeats
         (hasheqv) ;; dispatchers
         (hash)    ;; plates
         null      ;; requests
         ))

(define (add-port st out)
  (define id (state-id-seq st))
  (values id (struct-copy state st
                          [id-seq (add1 id)]
                          [ports (hash-set (state-ports st) id out)])))

(define (ref-port st id)
  (hash-ref (state-ports st) id))

(define (remove-port st id)
  (struct-copy state st
               [ports (hash-remove (state-ports st) id)]
               [heartbeats (hash-remove (state-heartbeats st) id)]
               [dispatchers (for/hash ([(road ids) (in-hash (state-dispatchers st))])
                              (values road (remq id ids)))]))

(define (update-heartbeat st id interval)
  (cond
    [(zero? interval)
     (struct-copy state st [heartbeats (hash-remove (state-heartbeats st) id)])]
    [else
     (define beat (heartbeat (+ (current-inexact-monotonic-milliseconds) interval) interval))
     (struct-copy state st [heartbeats (hash-set (state-heartbeats st) id beat)])]))

(define (heartbeat-evt st)
  (define heartbeats
    (state-heartbeats st))
  (cond
    [(hash-empty? heartbeats)
     never-evt]
    [else
     (match-define (cons id (heartbeat deadline interval))
       (car (sort (hash->list (state-heartbeats st)) #:key (compose1 heartbeat-deadline cdr) <)))
     (handle-evt
      (alarm-evt deadline #t)
      (lambda (_)
        (values id interval)))]))

(define (add-dispatcher st id roads)
  (define next-dispatchers
    (for/fold ([dispatchers (state-dispatchers st)])
              ([road (in-list roads)])
      (hash-update dispatchers road (λ (ids) (cons id ids)) null)))
  (struct-copy state st [dispatchers next-dispatchers]))

(define (track-plate st plate-number road mile limit timestamp)
  (define obs (observation mile limit timestamp))
  (define next-plates
    (hash-update
     (state-plates st) plate-number
     (λ (p) (add-observation p road obs))
     make-plate))
  (struct-copy state st [plates next-plates]))

(define (get-tickets st)
  (define dispatchers (state-dispatchers st))
  (define-values (tickets plates)
    (for*/fold ([tickets null]
                [plates (state-plates st)])
               ([(plate-number p) (in-hash (state-plates st))]
                [(road observations) (in-hash (plate-observations-by-road p))]
                #:unless (null? (hash-ref dispatchers road null)))
      (define sorted-observations
        (sort observations #:key observation-timestamp >))
      (define-values (next-tickets next-plate)
        (for/fold ([tickets tickets] [p p])
                  ([obs2 (in-list sorted-observations)]
                   [obs1 (in-list (cdr sorted-observations))]
                   #:unless (ticketed-on? p (observation-timestamp obs2))
                   #:unless (ticketed-on? p (observation-timestamp obs1))
                   #:unless (= (observation-timestamp obs2)
                               (observation-timestamp obs1)))
          (match-define (observation mile1 limit timestamp1) obs1)
          (match-define (observation mile2 _     timestamp2) obs2)
          (define speed
            (/ (abs (- mile2 mile1))
               (/ (- timestamp2 timestamp1) 3600)))
          (cond
            [(<= speed limit)
             (values tickets p)]
            [else
             (define ticket
               (Ticket plate-number road mile1 timestamp1 mile2 timestamp2 (exact-round (* speed 100))))
             (values
              (cons ticket tickets)
              (add-ticket-days p timestamp1 timestamp2))])))
      (values next-tickets (hash-set plates plate-number next-plate))))
  (values tickets (struct-copy state st [plates plates])))

(define (add-request st req)
  (struct-copy state st [requests (cons req (state-requests st))]))

(define (remove-request st req)
  (struct-copy state st [requests (remq req (state-requests st))]))

(define (->day ts)
  (quotient ts 86400))

This makes it easy to test the state and ticketing implementation in isolation:

(module+ test
  (require racket/port
           rackunit)
  (define (get-tickets* st)
    (define-values (tickets _)
      (get-tickets st))
    tickets)
  (check-equal?
   (get-tickets* (make-state))
   null)
  (check-equal?
   (~> (make-state)
       (track-plate "ABC" 1 0 80 0)
       (track-plate "ABC" 1 80 80 3600)
       (get-tickets*))
   null
   "within limit")
  (check-equal?
   (~> (make-state)
       (track-plate "ABC" 1 0 80 0)
       (track-plate "ABC" 1 80 80 1800)
       (get-tickets*))
   null
   "speed exceeded but no dispatcher")
  (check-equal?
   (let*-values ([(st) (make-state)]
                 [(id st) (add-port st (open-output-nowhere))])
     (~> (add-dispatcher st id '(1))
         (track-plate "ABC" 1 0 80 0)
         (track-plate "ABC" 1 80 80 1800)
         (get-tickets*)))
   (list (Ticket "ABC" 1 0 0 80 1800 16000))
   "speed exceeded with dispatcher")
  (check-equal?
   (let*-values ([(st) (make-state)]
                 [(id st) (add-port st (open-output-nowhere))]
                 [(_tickets st)
                  (~> (add-dispatcher st id '(1))
                      (track-plate "ABC" 1 0 80 0)
                      (track-plate "ABC" 1 80 80 1800)
                      (get-tickets))]
                 [(tickets _st)
                  (~> (track-plate st "ABC" 1 160 80 3600)
                      (get-tickets))])
     tickets)
   null
   "speed exceeded twice in same day")
  (check-equal?
   (let*-values ([(st) (make-state)]
                 [(id st) (add-port st (open-output-nowhere))]
                 [(_tickets st)
                  (~> (add-dispatcher st id '(1))
                      (track-plate "ABC" 1 0 80 0)
                      (track-plate "ABC" 1 80 80 1800)
                      (get-tickets))]
                 [(tickets _st)
                  (~> (track-plate st "ABC" 1 16000 80 86400)
                      (track-plate "ABC" 1 32000 80 (* 2 86400))
                      (get-tickets))])
     tickets)
   (list (Ticket "ABC" 1 16000 86400 32000 172800 66667))
   "speed exceeded next day"))

I'm using functional updates to alter the state between actor ticks. This is fine for the purposes of this challenge, but if performance were to become an issue, I'd probably switch to a mutable implementation.

The challenge is unclear on what should happen should an observation arrive arbitrarily late, so we keep all observations around perpetually. In a real system, we'd want to expire these.

Server

The server implementation simply accepts connections and sends messages to the dispatcher on their behalf.

(define-syntax-rule (send d command . args)
  (sync (make-dispatcher-evt d 'command . args)))

(define ((make-handler d) in out)
  (define id (send d add-port out))
  (dynamic-wind
    void
    (lambda ()
      (let loop ([ob #f])
        (match (read-message in)
          [(? eof-object?)
           (void)]
          [(? Camera? camera)
           #:when (not ob)
           (loop camera)]
          [(and (Dispatcher roads) dispatcher)
           #:when (not ob)
           (send d add-dispatcher id roads)
           (loop dispatcher)]
          [(WantHeartbeat interval)
           (send d update-heartbeat id (* (/ interval 10.0) 1000))
           (loop ob)]
          [(Plate plate-number timestamp)
           #:when (Camera? ob)
           (match-define (Camera road mile limit) ob)
           (send d snap plate-number road mile limit timestamp)
           (loop ob)]
          [_
           (write-error "unexpected message" out)])))
    (lambda ()
      (send d remove-port id))))

(module+ main
  (require "common.rkt")
  (run-server* "0.0.0.0" 8111 (make-handler (make-dispatcher))))

Every new client is registered with the dispatcher and receives a unique id. We use dynamic-wind to ensure that each client is deregistered on failure or on connection close.

That's all I have for this post. As before, you can find these solutions in full on GitHub.

I wrote this post last Sunday, but only published it today (Wednesday, March 26) after tweaking the post and solutions a bit. ↩

Protohackers Challenge in Racket Part 1

Sat, 25 Mar 2023 11:23:00 +0200

Someone on the Racket Discord recently mentioned the Protohackers project and I figured it'd be fun to write about Racket solutions to the challenges available on the website.

0: Smoke Test

The 0th challenge is a basic echo server.

#lang racket/base

(require racket/match
         racket/tcp)

(define (handle in out)
  (let loop ()
    (match (read-bytes 4096 in)
      [(? eof-object?)
       (void)]
      [bs
       (write-bytes bs out)
       (loop)])))

(module+ main
  (define listener
    (tcp-listen 8111 512 #t "0.0.0.0"))
  (define server-custodian
    (make-custodian))
  (with-handlers ([exn:break? void])
    (parameterize ([current-custodian server-custodian])
      (let loop ()
        (parameterize-break #f
          (define-values (in out)
            (tcp-accept/enable-break listener))
          (define client-custodian
            (make-custodian))
          (define client-thd
            (parameterize ([current-custodian client-custodian])
              (thread
               (lambda ()
                 (break-enabled #t)
                 (handle in out)))))
          (thread
           (lambda ()
             (sync client-thd)
             (close-output-port out)
             (close-input-port in)
             (custodian-shutdown-all client-custodian))))
        (loop))))
  (custodian-shutdown-all server-custodian)
  (tcp-close listener))

All we have to do is start a TCP listener, then accept new connections in a loop. For every new connection, we open a thread to handle it and a thread to close the connection and shut down the client custodian after the handling thread is done. Wrapping every client in a custodian ensures the handlers cannot leak resources (other threads, ports, etc.) after they exit. We disable breaks during connection setup so that an ill-timed break won't leave connections in a half-set-up state. On break (SIGINT, SIGTERM, or other signals), we terminate all running handler threads and their associated resources then close the TCP listener.

The handle procedure reads data in up to 4096 byte chunks and writes it back to the client. On EOF, read-bytes returns a special EOF value and, in that case, we simply exit the loop.

One notable thing about this server is we have no limit on the number of concurrent clients, so it is easy to flood. We also have no limit on connection duration, though that would be easy to add to the handler-supervising thread by using sync/timeout instead of sync.

1: Prime Time

This challenge requires us to do some JSON parsing and primality checking.

#lang racket/base

(require json
         racket/match
         racket/tcp)

(define (prime? n)
  (let ([n (truncate n)])
    (and (not (negative? n))
         (not (= n 0))
         (not (= n 1))
         (or (= n 2)
             (not
              (for/or ([i (in-range 2 (add1 (sqrt n)))])
                (zero? (modulo n i))))))))

(define (handle in out)
  (with-handlers ([exn:fail?
                   (λ (e)
                     ((error-display-handler) (format "client error: ~a" (exn-message e)) e)
                     (displayln "request malformed" out))])
    (let loop ()
      (define line (read-line in))
      (match (string->jsexpr line)
        [(hash-table
          ['method "isPrime"]
          ['number (? number? n)])
         (write-json (hasheq 'method "isPrime" 'prime (prime? n)) out)
         (newline out)
         (flush-output out)
         (loop)]
        [_
         (displayln "request malformed" out)]))))

The TCP listening bits haven't changed from the first challenge, so I've elided them here. The handle proc now reads one line at a time from the client and attempts to parse it as a JSON value. Any parsing error, as well as any validation error causes the handler to exit the loop and close the connection after writing a message to the client. Well-formed messages are validated in the first match clause and every valid request is followed by a valid response and a newline.

Note that read-line will happily buffer data in memory until it sees a linefeed character, meaning an adversarial client could easily crash this server by sending it a very long stream of non-linefeed characters¹. Also note how the error handlers are set up outside the loop. The body of the with-handlers form is not in tail position, so if we'd have set up the handlers inside the loop, we'd be creating unnecessary frames on every iteration, increasing memory consumption.

2: Means to an End

This challenge requires us to parse a custom binary format and store a little bit of state for each client.

#lang racket/base

(require racket/match
         racket/math
         racket/port
         racket/tcp
         "002.bnf")

(define (get-avg-price prices min-time max-time)
  (for/fold ([n 0] [s 0] #:result (if (zero? n) 0 (exact-round (/ s n))))
            ([(timestamp price) (in-hash prices)]
             #:when (and (>= timestamp min-time)
                         (<= timestamp max-time)))
    (values (add1 n) (+ s price))))

(define (handle in out)
  (let loop ([prices (hasheqv)])
    (define data (read-bytes 9 in))
    (unless (eof-object? data)
      (match (call-with-input-bytes data Message)
        [`((char_1 . #\I)
           (Timestamp_1 . ,timestamp)
           (Price_1 . ,price))
         (loop (hash-set prices timestamp price))]
        [`((char_1 . #\Q)
           (MinTime_1 . ,min-time)
           (MaxTime_1 . ,max-time))
         (un-Price (get-avg-price prices min-time max-time) out)
         (flush-output out)
         (loop prices)]))))

Just like with challenge 1, the listener bits have not changed. However, I did decide to overengineer this solution a little by using binfmt to parse the binary format. The contents of "0002.bnf" are:

#lang binfmt

Message = Insert | Query;

Insert = 'I' Timestamp Price;
Timestamp = i32be;
Price = i32be;

Query = 'Q' MinTime MaxTime;
MinTime = i32be;
MaxTime = i32be;

The handler proc reads 9 bytes from the client at a time and tries to parse a Message out of them. When it receives an insert message, it updates the price slot for that timestamp and when it receives a query message, it computes an average price and responds to the client.

Like the previous challange, this server is susceptible to an attack where an adversarial client could send it enough prices to exhaust available memory and cause it to crash.

3: Budget Chat

This challenge requires us to implement a chat room.

#lang racket/base

(require racket/match
         racket/string
         racket/tcp)

(define room-ch (make-channel))
(define room-thd
  (thread/suspend-to-kill
   (lambda ()
     (let loop ([users (hasheq)]
                [reqs null])
       (apply
        sync
        (handle-evt
         room-ch
         (lambda (msg)
           (match msg
             [`(join ,name ,out ,res-ch ,nack)
              (cond
                [(hash-has-key? users name)
                 (define req `((fail "username-taken") ,res-ch ,nack))
                 (loop users (cons req reqs))]
                [else
                 (define req `((ok ,(hash-keys users)) ,res-ch ,nack))
                 (broadcast users (format "* ~a has joined the room~n" name))
                 (loop (hash-set users name out) (cons req reqs))])]
             [`(broadcast ,name ,message ,res-ch ,nack)
              (define req `((ok) ,res-ch ,nack))
              (broadcast (hash-remove users name) (format "[~a] ~a~n" name message))
              (loop users (cons req reqs))]
             [`(leave ,name ,res-ch ,nack)
              (define req `((ok) ,res-ch ,nack))
              (define remaining-users (hash-remove users name))
              (broadcast remaining-users (format "* ~a has left the room~n" name))
              (loop remaining-users (cons req reqs))]
             [_
              (log-warning "malformed message: ~s" msg)
              (loop users reqs)])))
        (append
         (for/list ([req (in-list reqs)])
           (match-define `(,res ,res-ch ,_) req)
           (handle-evt
            (channel-put-evt res-ch res)
            (λ (_) (loop users (remq req reqs)))))
         (for/list ([req (in-list reqs)])
           (match-define `(,_ ,_ ,nack) req)
           (handle-evt nack (λ (_) (loop users (remq req reqs)))))))))))

(define (make-room-evt command . args)
  (define res-ch (make-channel))
  (nack-guard-evt
   (lambda (nack)
     (begin0 res-ch
       (thread-resume room-thd (current-thread))
       (channel-put room-ch `(,command ,@args ,res-ch ,nack))))))

(define (handle in out)
  (fprintf* out "Welcome to budgetchat! What shall I call you?~n")
  (match (read-line in 'any)
    [(regexp #px"^[a-zA-Z0-9]{1,16}$" (list (app string->symbol name)))
     (match (sync (make-room-evt 'join name out))
       [`(fail ,message)
        (fprintf out "error: ~a~n" message)]
       [`(ok ,names)
        (fprintf* out "* The room contains: ~a~n" (string-join (map symbol->string (sort names symbol<?))))
        (with-handlers ([exn:fail? (λ (e) ((error-display-handler) (format "client error: ~a" (exn-message e)) e))])
          (let loop ()
            (define data
              (read-line in 'any))
            (unless (eof-object? data)
              (sync (make-room-evt 'broadcast name data))
              (loop))))
        (sync (make-room-evt 'leave name))])]
    [_
     (fprintf* out "error: invalid name~n")]))

(define (broadcast users message)
  (for ([out (in-hash-values users)])
    (with-handlers ([exn:fail? void])
      (fprintf* out message))))

(define (fprintf* out msg . args)
  (apply fprintf out msg args)
  (flush-output out))

The implementation for this challenge is more complex since it involves keeping track of shared state and broadcasting messages to multiple clients. I opted to make a shared thread to keep track of user sessions. The contents of the loop inside room-thd is a common pattern I use when implementing stateful actors in Racket, stolen from this paper.

The room is itself a sort of server. It receives messages via the room-ch where each message is expected to have a channel on which responses can be sent, and a negative acknowledgement evt. For every message it receives, it updates its internal state and responds to whatever requests it can, or removes any requests whose negative acknowledgement event is ready for synchronization (i.e. requests that have been abandoned ). The make-room-evt procedure generates a synchronizable event that sends a request to the room and receives a response when synchronized.

The handler follows the initialization sequence by asking the client for a nickname, validating it and then entering the messaging loop. When a client joins the room, it sends its nick along with its output port to the room. The room then writes to the client's output port whenever new information is broadcast.

This server is susceptible to the same attacks as the ones before it: it uses read-line to read messages and it doesn't limit the number of concurrent users. The former we can fix by writing a custom read-line* function that rejects lines longer than 1024 bytes (eg, by using peek-bytes) and the latter we can fix by making the room reject join requests when the number of users exceeds some limit.

That's all for today. You can find these servers in full on GitHub.

Ryan Culpepper pointed out on the Racket Discord that we could combine read-line with make-limited-input-port in order to limit the amount of data read-line will buffer in memory. ↩

Racketfest 2023 Talk: Native Apps with Racket

Sun, 19 Mar 2023 19:00:00 +0200

Racketfest 2023 was held yesterday and I gave a short talk about building native apps with Racket. Nothing new if you've read my recent posts, but below is a transcript. A recording might also be posted later, in which case I'll update this post to link to it.

Transcript

Native Apps

My name is Bogdan Popa, and today I will be talking about an approach I've been using to build native desktop apps with Racket.

Native Applications?

First, what do I mean by "Native Application"? I mean an application that uses the system libraries and frameworks for building applications. Applications that look and feel like other applications that ship with the system and that implies they have access to all available widgets on the system.

With Racket?

Why with Racket? Because I like using the language and I've built up a large collection of libraries over the years and because I'd like the core logic to be portable between operating systems.

Approaches

So far, I've used three approaches to building desktop apps in Racket:

racket/gui
Embedding Racket as a subprocess.
Embedding Racket directly, which is the focus of this talk.

racket/gui{,easy}

racket/gui comes with Racket and supports a combination of native and custom widgets on Linux, macOS and Windows. But, the set of widgets it supports out of the box is limited, and not all widgets (eg. input fields) are truly native. Additionally, because it aims to support all of the aforementioned platforms, it's hard to extend it with new widgets because analogs of a widget on one platform might not exist on others.

Embedding as Subprocess

Another approach I've used is embedding Racket as a subprocess. The idea here being that the GUI app runs Racket in a subprocess and communicates with it via pipes. I've actually shipped an app to the Mac App Store using this approach.

One downside with this approach is that memory consumption is relatively high (but that's more of a Racket problem in general, than one particular to this approach). Another is that, since these are two separate processes, the Racket code can't call back into Swift without help.

Embedding Directly

The approach I used with Franz is to compile Racket as a static library and link it into a Swift app. Like in the subprocess approach, I've opted to run the Racket runtime in its own thread and communicate with it via pipes¹. By running Racket in its own OS thread, it can keep scheduling its own threads as normal and I can run an RPC server that listens on a pipe for requests and serves responses asynchronously. An advantage over the previous approach is here the Swift and Racket sides share memory so it's possible for the Racket side to call out to Swift directly.

Memory use is still a downside, though slightly better than the subprocess approach because process overhead is reduced and things like shared libraries can be shared between the two runtimes.

Demo

[No transcript for the demo portion, but see the Franz website.]

Code Stats

In terms of code, the Swift portion is about 9k lines and the Racket portion about 18k lines, but the Racket portion also includes the Kafka client.

How it Works

As mentioned, this approach works by compiling Racket to a static library and linking it directly into a Swift application. The Racket code is then compiled using raco ctool to a .zo bytecode file and shipped alongside the app.

How it Works (cont'd)

On boot, the Swift application starts Racket in a background thread, loads the .zo code from the application bundle, and there's a small interface for setting up the RPC system between the two languages. A "main" procedure is loaded from the .zo code, then that procedure is called with a set of pipe file descriptors, which are then converted into ports on the Racket side.

Noise

To abstract over some of this stuff, I've written a set of Swift and Racket libraries (with plans to add C# support for Windows soon). They live under the Noise repo, linked at the top, and they are split up in roughly three layers. The lowest layer handles embedding libracketcs and provides a Swift wrapper for its C ABI. The layer above that implements a protocol for serializing and deserializing data between the two languages and the layer above that implements a protocol for communicating via pipes.

NoiseRacket

NoiseRacket is the lowest layer and, as mentioned, it handles the embedding of libracketcs. From Swift code, you just import Noise, then create an instance of the Racket class to initialize the Racket runtime. Then, every time you want to call Racket, you pass a closure to that object's bracket method. In this example, we load a .zo file, construct a module path and require a function named fib then apply it and print the result. As you can see, this is pretty laborious.

NoiseSerde

NoiseSerde provides a set of macros on the Racket side and a code generator that produces Swift code from record and enum definitions. The example on the left expands to a struct definition that knows how to serialize and deserialize itself. Additionally, it stores information for the code generator so that it can produce a matching struct on the Swift side so that the two sides can pass data around transparently.

NoiseBackend

This layer provides a macro for defining remote procedure calls in Racket. On the Racket side, defined RPCs expand to regular functions, but they get registered with the RPC server. On the Swift side, they expand to method declarations that handle the details of serializing arguments and returning a Future value representing the to-be-received response.

Performance

This might sound like it's a lot of overhead, but in practice it isn't. The cost of the RPCs themselves is negligible, and the cost of ser/de on average is on the order of 1 to 100 microseconds. All of this is dwarfed by the overhead of the business logic (i.e. communicating with Kafka, which takes on the order of 1ms+ even on loopback).

Closing Thoughts

I'm very happy with this approach. It feels natural to write apps in this way, and the app that I demoed is already in customers' hands (some even paid for it!). In future, I plan to work on Windows support and we'll have to see how that pans out. All that said, if you want to make cross-platform desktop apps, probably something like Electron is a safer bet, despite not being native. This approach is still quite a lot of work. But, if you're a crazy-person who really wants to use Racket to make desktop apps for some reason, give it a try.

Thanks

Thank you for attending my talk! You can find Franz at franz.defn.io, and Noise on GitHub.

A couple folks asked about why I opted to serialize the data between Racket and Swift instead of just sharing the objects directly between the two languages. After thinking about it for a bit, I remembered two main reasons why I preferred the serde approach: 1) I didn't want to have to interrupt the Racket runtime every time the Swift side needed to access a Racket object and 2) the Racket CS GC is free to move objects in memory. While there are ways to tell the runtime not to move individual values, it just doesn't seem worth it as long as the serde approach doesn't add tons of overhead. ↩

Announcing racket-protocol-buffers

Wed, 22 Feb 2023 11:00:00 +0200

A couple of releases back, I added support for schema registries to Franz. Some of its users use Protocol Buffers to serialize their data, so I needed to be able to support that use case. So, I wrote a parser for the proto2 and proto3 specs and a minimal serializer/deserializer implementation that doesn't require code generation on top of that.

You can find the package and docs on the Package server and the source on GitHub.

Safe Foreign Callouts from Racket to Swift

Sat, 4 Feb 2023 13:47:00 +0200

In anticipation of working on the Windows & Linux versions of Franz, I've wanted to move its auto-update implementation from Swift into Franz' Racket core. The reason I implemented the auto-update code in Swift in the first place is because of the way the Swift part normally communicates with the Racket part: the core Racket code runs in its own thread and the Swift part communicates with it asynchronously via pipes. So, until a couple of days ago, I didn't have an easy way for Racket code to trigger the execution of Swift code on its own.

All of the code that handles embedding Racket inside Swift, code generation and the general communication mechanism is open source and lives in Noise, so that's where you can find the full implementation of the approach I describe in this post (specifically, commits 0a585be and 2f6c37e).

Low Level Bits

Swift has its own calling convention, but it supports declaring procedures (but not closures) as using the C calling convention via the @convention(c) attribute. For example:

var add1: @convention(c) (Int) -> Int = { x in x + 1 }

This attribute makes it so that you can transparently pass such procedures around in places where you would normally store C function pointers. In my case, though, I didn't want to have to write any C code to support this functionality. Instead, I needed to be able to get the raw pointer addresses of the procedures so that I could serialize them and send them (over the aforementioned pipes) to the Racket side. Thankfully, there is a way to do this in Swift, via unsafeBitCast:

let ptr = unsafeBitCast(add1, Optional<UnsafeRawPointer>.self)
let addr = Int(bitPattern: ptr!)

With a raw pointer in hand, all I have to do is make an RPC to the Racket side to tell it to register a callout at that pointer's address. The Racket side can then take that address and construct a foreign procedure using its FFI facilities:

(require ffi/unsafe)

(define add1-type
  (_func _int -> _int))       ;; 1
(define add1
  (let ([p (malloc _intptr)]) ;; 2
    (ptr-set! p _intptr addr) ;; 3
    (ptr-ref p add1-type)))   ;; 4

There's no direct way to convert an address to a pointer using the FFI library. Instead, I have to allocate a bit of memory (2), write the address to that memory (3) and then read the address out as a foreign procedure (4). Additionally, I have to know what the signature of that procedure is (1) to be able to call it later. ¹

Putting these bits together, I came up with a small abstraction on the Racket side to wrap the FFI code needed to turn a raw address into a foreign procedure:

(struct callout-box (type [proc #:mutable])
  #:property prop:procedure (λ (b . args)
                              (define proc (callout-box-proc b))
                              (unless proc
                                (error 'callout-box "procedure not installed"))
                              (apply (callout-box-proc b) args)))

(define (make-callout-box type)
  (callout-box type #f))

(define (callout-box-install! b addr)
  (define p (malloc _intptr))
  (ptr-set! p _intptr addr)
  (set-callout-box-proc! b (ptr-ref p (callout-box-type b))))

And, on the Swift side, I devised a little interface for installing arbitrary callbacks (on the Swift side) as callouts (on the Racket side) by using a trampoline (some details, such as locking around callbacks, elided for brevity):

public func installCallback(id: UInt64, proc: @escaping (Data) -> Void) -> Future<String, Void> {
  callbacks[id] = proc
  let ptr = unsafeBitCast(callbackHandler, to: Optional<UnsafeRawPointer>.self)
  let addr = Int(bitPattern: ptr!)
  installCallback(id, addr)  // RPC to Racket
}

fileprivate var callbacks = [UInt64: (Data) -> Void]()
fileprivate let callbackHandler: @convention(c) (UInt64, Int, UnsafePointer<CChar>) -> Void = { id, len, ptr in
  let data = ...  // based on len and ptr
  let proc = callbacks[id]
  proc!(data)
}

Whenever installCallback is called with a closure, it stores the closure in a global hash and calls an RPC on the Racket side to register the callbackHandler as the foreign procedure for that closure. The callback handler ends up always being the same, which is a little wasteful, but this keeps the implementation really straightforward so I'm not too bothered by it.

Mid Level Bits

You've probably noticed the id argument to installCallback. The Racket and Swift sides need to sync on these ids to know which callout connects to with callback. So, on the Racket side there is a syntactic form for declaring callouts:

(define-callout (hello-cb [name : String] [age : Varint]))

The callouts themselves can have arbitrary arguments and the data is automatically serialized when a callout procedure is executed, which is why the callback handler's type contains a size and a data pointer in addition to the callback id. We'll get to how this works on the Swift side toward the end of the article.

The implementation of define-callout is fairly straightforward. It starts with the well-known signature for callout handlers (the prototype of callbackHandler):

(define callout-type
  (_func _int _size _bytes -> _void))

Following that, there are some structure definitions to keep track of the callout metadata at runtime, a global registry for this metadata (indexed by callout id), and a helper function to perform the callouts:

(struct callout-arg (name type))
(struct callout-info ([id #:mutable] name args cbox))
(define callout-infos (make-hasheqv))

(define (do-callout info arg-pairs)
  (define id (callout-info-id info))
  (define cbox (callout-info-cbox info))
  (define bs
    (call-with-output-bytes
     (lambda (out)
       (for ([p (in-list arg-pairs)])
         (write-field (car p) (cdr p) out)))))
  (cbox id (bytes-length bs) bs))

The arg-pairs argument to do-callout is a list of cons pairs that contains the serializable type of each argument and the argument's runtime value. It takes those arguments, serializes them into a byte string and then executes the callout using the callout's id and the serialized data.

The definition of define-callout itself is as follows (with a couple small details simplified):

(define-syntax (define-callout stx)
  (syntax-parse stx
    #:literals (:)
    [(_ (name:id [arg-name:id : arg-type:expr] ...+))
     #:fail-unless (valid-name-stx? #'name)
     "callout names may only contain alphanumeric characters, dashes and underscores"
     #'(begin
         (define (name arg-name ...) ;; 1
           (do-callout info (list (cons (->field-type 'Callout arg-type) arg-name) ...)))
         (define args
           (for/list ([n (in-list (list 'arg-name ...))]
                      [t (in-list (list arg-type ...))])
             (callout-arg n (->field-type 'Callout t))))
         (define cbox
           (make-callout-box callout-type))
         (define info ;; 2
           (callout-info #f 'name args cbox))
         (hash-set! callout-infos (next-callout-id!) info) ;; 3
         )]))

Every use of define-callout expands to a definition of a procedure with the given name (1) that delegates to the do-callout helper when it itself is called and a metadata definition for the callout that is registered with the global registry (2, 3).

Finally, the RPC to install these callbacks that I mentioned at the end of the first section looks like this:

(define-rpc (install-callback [internalWithId id : UVarint]
                              [andAddr addr : Varint])
  (define cbox (callout-info-cbox (hash-ref callout-infos id)))
  (callout-box-install! cbox addr))

When applied, it looks up the runtime info for the callout with the given id, extracts its box and installs the procedure at that address into the box.

High Level Bits

You might be wondering why the callout-info needs to remember the argument types for the callout, since we never used them again above. This leads us to the final piece of this system, namely the code generation part.

Noise generates Swift code to handle data type serialization and deserialization, RPCs and, now, callouts. To do this, it uses that same runtime callout metadata described above.

When generating the Backend class for a project, it produces methods for all the RPCs and then it turns to callouts, the code for which looks like this:

(define sorted-callout-ids (sort (hash-keys callout-infos) <))
(for ([id (in-list sorted-callout-ids)])
  (match-define (callout-info _ name args _cbox)
    (hash-ref callout-infos id))
  (define proc-name (~name name))
  (define proc-type
    (format "@escaping (~a) -> Void"
            (string-join
             (map (compose1 swift-type callout-arg-type) args)
             ", ")))
  (fprintf out "~n")
  (fprintf out "  public func installCallback(~a proc: ~a) -> Future<String, Void> {~n" proc-name proc-type)
  (fprintf out "    return NoiseBackend.installCallback(id: ~a, rpc: self.installCallback(internalWithId:andAddr:)) { inp in~n" id)
  (fprintf out "      var buf = Data(count: 8*1024)~n")
  (fprintf out "      proc(~n")
  (define last-idx (sub1 (length args)))
  (for ([(arg idx) (in-indexed (in-list args))])
    (match-define (callout-arg _name type) arg)
    (define maybe-comma (if (= idx last-idx) "" ","))
    (fprintf out "        ~a.read(from: inp, using: &buf)~a~n" (swift-type type) maybe-comma))
  (fprintf out "      )~n")
  (fprintf out "    }~n")
  (fprintf out "  }~n"))

That is, for every known callout, it generates a Swift method named installCallback(calloutName:). To give a concrete example, here is what the installer for the hello-cb example from earlier in this article would look like:

public func installCallback(helloCb proc: @escaping (String, Varint) -> Void) -> Future<String, Void> {
  return NoiseBackend.installCallback(id: 0, rpc: self.installCallback(internalWithId:andAddr:)) { inp in
    var buf = Data(count: 8*1024)
    proc(
      String.read(from: inp, using: &buf),
      Varint.read(from: inp, using: &buf)
    )
  }
}

Which you would use from the Swift side like so:

Backend.shared.installCallback(helloCb: { name, age in
  print("hello \(name), I hear you're \(age) years old!")
})

And you would call from the Racket side like so:

(hello-cb "Bogdan" 30)

And you wouldn't need to worry about most of the details I've written about above.

Sam Phillips pointed out on the Racket Discord that there is actually a helper for this in ffi-lib, namely cast. So this let can be replaced with (cast addr _intptr add1-type). ↩

This is Fine

Sun, 18 Dec 2022 15:25:00 +0200

This post is a dumb rant, but I needed to vent.

As I often do when I'm bored or procrastinating, I decided to update some of the software on my machine today. As usual, this was a mistake.

I ran the following command:

$ sudo port upgrade yubikey-manager ykpers

And its output was:

--->  Computing dependencies for py310-semantic_version
--->  Fetching archive for py310-semantic_version
--->  Attempting to fetch py310-semantic_version-2.10.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-semantic_version
--->  Attempting to fetch py310-semantic_version-2.10.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-semantic_version
--->  Installing py310-semantic_version @2.10.0_0
--->  Activating py310-semantic_version @2.10.0_0
--->  Cleaning py310-semantic_version
--->  Computing dependencies for py310-typing_extensions
--->  Fetching archive for py310-typing_extensions
--->  Attempting to fetch py310-typing_extensions-4.4.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-typing_extensions
--->  Attempting to fetch py310-typing_extensions-4.4.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-typing_extensions
--->  Installing py310-typing_extensions @4.4.0_0
--->  Activating py310-typing_extensions @4.4.0_0
--->  Cleaning py310-typing_extensions
--->  Computing dependencies for jemalloc
--->  Fetching archive for jemalloc
--->  Attempting to fetch jemalloc-5.3.0_2.darwin_22.arm64.tbz2 from https://packages.macports.org/jemalloc
--->  Attempting to fetch jemalloc-5.3.0_2.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/jemalloc
--->  Attempting to fetch jemalloc-5.3.0_2.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/jemalloc
--->  Fetching distfiles for jemalloc
--->  Attempting to fetch jemalloc-5.3.0.tar.bz2 from https://distfiles.macports.org/jemalloc
--->  Verifying checksums for jemalloc
--->  Extracting jemalloc
--->  Applying patches to jemalloc
--->  Configuring jemalloc
--->  Building jemalloc
--->  Staging jemalloc into destroot
--->  Installing jemalloc @5.3.0_2
--->  Activating jemalloc @5.3.0_2
--->  Cleaning jemalloc
--->  Computing dependencies for libssh2
--->  Fetching archive for libssh2
--->  Attempting to fetch libssh2-1.10.0_0.darwin_22.arm64.tbz2 from https://packages.macports.org/libssh2
--->  Attempting to fetch libssh2-1.10.0_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/libssh2
--->  Attempting to fetch libssh2-1.10.0_0.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/libssh2
--->  Fetching distfiles for libssh2
--->  Attempting to fetch libssh2-1.10.0.tar.gz from https://distfiles.macports.org/libssh2
--->  Verifying checksums for libssh2
--->  Extracting libssh2
--->  Configuring libssh2
Warning: Configuration logfiles contain indications of -Wimplicit-function-declaration; check that features were not accidentally disabled:
  strchr: found in libssh2-1.10.0/config.log
--->  Building libssh2
--->  Staging libssh2 into destroot
--->  Installing libssh2 @1.10.0_0
--->  Activating libssh2 @1.10.0_0
--->  Cleaning libssh2
--->  Computing dependencies for libgit2
--->  Fetching archive for libgit2
--->  Attempting to fetch libgit2-1.5.0_0+threadsafe.darwin_22.arm64.tbz2 from https://packages.macports.org/libgit2
--->  Attempting to fetch libgit2-1.5.0_0+threadsafe.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/libgit2
--->  Attempting to fetch libgit2-1.5.0_0+threadsafe.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/libgit2
--->  Fetching distfiles for libgit2
--->  Attempting to fetch libgit2-1.5.0.tar.gz from https://distfiles.macports.org/libgit2
--->  Verifying checksums for libgit2
--->  Extracting libgit2
--->  Applying patches to libgit2
--->  Configuring libgit2
--->  Building libgit2
--->  Staging libgit2 into destroot
--->  Installing libgit2 @1.5.0_0+threadsafe
--->  Activating libgit2 @1.5.0_0+threadsafe
--->  Cleaning libgit2

That's more stuff than I'd expect, but whatever. So far so good...

--->  Computing dependencies for rust
--->  Fetching archive for rust
--->  Attempting to fetch rust-1.61.0_2.darwin_22.arm64.tbz2 from https://packages.macports.org/rust
--->  Attempting to fetch rust-1.61.0_2.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/rust
--->  Attempting to fetch rust-1.61.0_2.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/rust
--->  Fetching distfiles for rust
--->  Attempting to fetch rustc-1.61.0-src.tar.gz from https://static.rust-lang.org/dist
--->  Attempting to fetch rust-std-1.60.0-aarch64-apple-darwin.tar.gz from https://static.rust-lang.org/dist
--->  Attempting to fetch rustc-1.60.0-aarch64-apple-darwin.tar.gz from https://static.rust-lang.org/dist
--->  Attempting to fetch cargo-1.60.0-aarch64-apple-darwin.tar.gz from https://static.rust-lang.org/dist
--->  Attempting to fetch addr2line-0.16.0.crate from https://crates.io/api/v1/crates/addr2line/0.16.0/download?dummy=
--->  Attempting to fetch adler-0.2.3.crate from https://crates.io/api/v1/crates/adler/0.2.3/download?dummy=
--->  Attempting to fetch ahash-0.7.4.crate from https://crates.io/api/v1/crates/ahash/0.7.4/download?dummy=
--->  Attempting to fetch aho-corasick-0.7.18.crate from https://crates.io/api/v1/crates/aho-corasick/0.7.18/download?dummy=
--->  Attempting to fetch ammonia-3.1.3.crate from https://crates.io/api/v1/crates/ammonia/3.1.3/download?dummy=
--->  Attempting to fetch annotate-snippets-0.8.0.crate from https://crates.io/api/v1/crates/annotate-snippets/0.8.0/download?dummy=
--->  Attempting to fetch ansi_term-0.12.1.crate from https://crates.io/api/v1/crates/ansi_term/0.12.1/download?dummy=
--->  Attempting to fetch anyhow-1.0.51.crate from https://crates.io/api/v1/crates/anyhow/1.0.51/download?dummy=
--->  Attempting to fetch array_tool-1.0.3.crate from https://crates.io/api/v1/crates/array_tool/1.0.3/download?dummy=
--->  Attempting to fetch arrayvec-0.7.0.crate from https://crates.io/api/v1/crates/arrayvec/0.7.0/download?dummy=
--->  Attempting to fetch askama-0.11.0.crate from https://crates.io/api/v1/crates/askama/0.11.0/download?dummy=
--->  Attempting to fetch askama_derive-0.11.0.crate from https://crates.io/api/v1/crates/askama_derive/0.11.0/download?dummy=
--->  Attempting to fetch askama_escape-0.10.2.crate from https://crates.io/api/v1/crates/askama_escape/0.10.2/download?dummy=
--->  Attempting to fetch askama_shared-0.12.0.crate from https://crates.io/api/v1/crates/askama_shared/0.12.0/download?dummy=
--->  Attempting to fetch atty-0.2.14.crate from https://crates.io/api/v1/crates/atty/0.2.14/download?dummy=
--->  Attempting to fetch autocfg-1.1.0.crate from https://crates.io/api/v1/crates/autocfg/1.1.0/download?dummy=
--->  Attempting to fetch bitflags-1.2.1.crate from https://crates.io/api/v1/crates/bitflags/1.2.1/download?dummy=
--->  Attempting to fetch bitmaps-2.1.0.crate from https://crates.io/api/v1/crates/bitmaps/2.1.0/download?dummy=
--->  Attempting to fetch block-buffer-0.7.3.crate from https://crates.io/api/v1/crates/block-buffer/0.7.3/download?dummy=
--->  Attempting to fetch block-buffer-0.10.2.crate from https://crates.io/api/v1/crates/block-buffer/0.10.2/download?dummy=
--->  Attempting to fetch block-padding-0.1.5.crate from https://crates.io/api/v1/crates/block-padding/0.1.5/download?dummy=
--->  Attempting to fetch bstr-0.2.13.crate from https://crates.io/api/v1/crates/bstr/0.2.13/download?dummy=
--->  Attempting to fetch byte-tools-0.3.1.crate from https://crates.io/api/v1/crates/byte-tools/0.3.1/download?dummy=
--->  Attempting to fetch bytecount-0.6.2.crate from https://crates.io/api/v1/crates/bytecount/0.6.2/download?dummy=
--->  Attempting to fetch byteorder-1.3.4.crate from https://crates.io/api/v1/crates/byteorder/1.3.4/download?dummy=
--->  Attempting to fetch bytes-1.0.1.crate from https://crates.io/api/v1/crates/bytes/1.0.1/download?dummy=
--->  Attempting to fetch bytesize-1.0.1.crate from https://crates.io/api/v1/crates/bytesize/1.0.1/download?dummy=
--->  Attempting to fetch camino-1.0.5.crate from https://crates.io/api/v1/crates/camino/1.0.5/download?dummy=
--->  Attempting to fetch cargo-platform-0.1.2.crate from https://crates.io/api/v1/crates/cargo-platform/0.1.2/download?dummy=
--->  Attempting to fetch cargo_metadata-0.14.0.crate from https://crates.io/api/v1/crates/cargo_metadata/0.14.0/download?dummy=
--->  Attempting to fetch cc-1.0.69.crate from https://crates.io/api/v1/crates/cc/1.0.69/download?dummy=
--->  Attempting to fetch cfg-if-0.1.10.crate from https://crates.io/api/v1/crates/cfg-if/0.1.10/download?dummy=
--->  Attempting to fetch cfg-if-1.0.0.crate from https://crates.io/api/v1/crates/cfg-if/1.0.0/download?dummy=
--->  Attempting to fetch chalk-derive-0.80.0.crate from https://crates.io/api/v1/crates/chalk-derive/0.80.0/download?dummy=
--->  Attempting to fetch chalk-engine-0.80.0.crate from https://crates.io/api/v1/crates/chalk-engine/0.80.0/download?dummy=
--->  Attempting to fetch chalk-ir-0.80.0.crate from https://crates.io/api/v1/crates/chalk-ir/0.80.0/download?dummy=
--->  Attempting to fetch chalk-solve-0.80.0.crate from https://crates.io/api/v1/crates/chalk-solve/0.80.0/download?dummy=
--->  Attempting to fetch chrono-0.4.19.crate from https://crates.io/api/v1/crates/chrono/0.4.19/download?dummy=
--->  Attempting to fetch clap-2.34.0.crate from https://crates.io/api/v1/crates/clap/2.34.0/download?dummy=
--->  Attempting to fetch clap-3.1.1.crate from https://crates.io/api/v1/crates/clap/3.1.1/download?dummy=
--->  Attempting to fetch cmake-0.1.44.crate from https://crates.io/api/v1/crates/cmake/0.1.44/download?dummy=
--->  Attempting to fetch colored-2.0.0.crate from https://crates.io/api/v1/crates/colored/2.0.0/download?dummy=
--->  Attempting to fetch combine-4.6.3.crate from https://crates.io/api/v1/crates/combine/4.6.3/download?dummy=
--->  Attempting to fetch commoncrypto-0.2.0.crate from https://crates.io/api/v1/crates/commoncrypto/0.2.0/download?dummy=
--->  Attempting to fetch commoncrypto-sys-0.2.0.crate from https://crates.io/api/v1/crates/commoncrypto-sys/0.2.0/download?dummy=
--->  Attempting to fetch compiler_builtins-0.1.70.crate from https://crates.io/api/v1/crates/compiler_builtins/0.1.70/download?dummy=
--->  Attempting to fetch compiletest_rs-0.7.1.crate from https://crates.io/api/v1/crates/compiletest_rs/0.7.1/download?dummy=
--->  Attempting to fetch core-foundation-0.9.0.crate from https://crates.io/api/v1/crates/core-foundation/0.9.0/download?dummy=
--->  Attempting to fetch core-foundation-sys-0.8.0.crate from https://crates.io/api/v1/crates/core-foundation-sys/0.8.0/download?dummy=
--->  Attempting to fetch cpufeatures-0.2.1.crate from https://crates.io/api/v1/crates/cpufeatures/0.2.1/download?dummy=
--->  Attempting to fetch crc32fast-1.2.0.crate from https://crates.io/api/v1/crates/crc32fast/1.2.0/download?dummy=
--->  Attempting to fetch crossbeam-channel-0.5.2.crate from https://crates.io/api/v1/crates/crossbeam-channel/0.5.2/download?dummy=
--->  Attempting to fetch crossbeam-deque-0.8.1.crate from https://crates.io/api/v1/crates/crossbeam-deque/0.8.1/download?dummy=
--->  Attempting to fetch crossbeam-epoch-0.9.6.crate from https://crates.io/api/v1/crates/crossbeam-epoch/0.9.6/download?dummy=
--->  Attempting to fetch crossbeam-utils-0.8.6.crate from https://crates.io/api/v1/crates/crossbeam-utils/0.8.6/download?dummy=
--->  Attempting to fetch crypto-common-0.1.2.crate from https://crates.io/api/v1/crates/crypto-common/0.1.2/download?dummy=
--->  Attempting to fetch crypto-hash-0.3.4.crate from https://crates.io/api/v1/crates/crypto-hash/0.3.4/download?dummy=
--->  Attempting to fetch cstr-0.2.8.crate from https://crates.io/api/v1/crates/cstr/0.2.8/download?dummy=
--->  Attempting to fetch ctor-0.1.15.crate from https://crates.io/api/v1/crates/ctor/0.1.15/download?dummy=
--->  Attempting to fetch curl-0.4.41.crate from https://crates.io/api/v1/crates/curl/0.4.41/download?dummy=
--->  Attempting to fetch curl-sys-0.4.51+curl-7.80.0.crate from https://crates.io/api/v1/crates/curl-sys/0.4.51+curl-7.80.0/download?dummy=
--->  Attempting to fetch datafrog-2.0.1.crate from https://crates.io/api/v1/crates/datafrog/2.0.1/download?dummy=
--->  Attempting to fetch derive-new-0.5.8.crate from https://crates.io/api/v1/crates/derive-new/0.5.8/download?dummy=
--->  Attempting to fetch derive_more-0.99.9.crate from https://crates.io/api/v1/crates/derive_more/0.99.9/download?dummy=
--->  Attempting to fetch diff-0.1.12.crate from https://crates.io/api/v1/crates/diff/0.1.12/download?dummy=
--->  Attempting to fetch difference-2.0.0.crate from https://crates.io/api/v1/crates/difference/2.0.0/download?dummy=
--->  Attempting to fetch digest-0.8.1.crate from https://crates.io/api/v1/crates/digest/0.8.1/download?dummy=
--->  Attempting to fetch digest-0.10.2.crate from https://crates.io/api/v1/crates/digest/0.10.2/download?dummy=
--->  Attempting to fetch directories-3.0.2.crate from https://crates.io/api/v1/crates/directories/3.0.2/download?dummy=
--->  Attempting to fetch dirs-2.0.2.crate from https://crates.io/api/v1/crates/dirs/2.0.2/download?dummy=
--->  Attempting to fetch dirs-next-2.0.0.crate from https://crates.io/api/v1/crates/dirs-next/2.0.0/download?dummy=
--->  Attempting to fetch dirs-sys-0.3.6.crate from https://crates.io/api/v1/crates/dirs-sys/0.3.6/download?dummy=
--->  Attempting to fetch dirs-sys-next-0.1.2.crate from https://crates.io/api/v1/crates/dirs-sys-next/0.1.2/download?dummy=
--->  Attempting to fetch dlmalloc-0.2.3.crate from https://crates.io/api/v1/crates/dlmalloc/0.2.3/download?dummy=
--->  Attempting to fetch either-1.6.0.crate from https://crates.io/api/v1/crates/either/1.6.0/download?dummy=
--->  Attempting to fetch elasticlunr-rs-2.3.9.crate from https://crates.io/api/v1/crates/elasticlunr-rs/2.3.9/download?dummy=
--->  Attempting to fetch ena-0.14.0.crate from https://crates.io/api/v1/crates/ena/0.14.0/download?dummy=
--->  Attempting to fetch enum-iterator-0.6.0.crate from https://crates.io/api/v1/crates/enum-iterator/0.6.0/download?dummy=
--->  Attempting to fetch enum-iterator-derive-0.6.0.crate from https://crates.io/api/v1/crates/enum-iterator-derive/0.6.0/download?dummy=
--->  Attempting to fetch env_logger-0.7.1.crate from https://crates.io/api/v1/crates/env_logger/0.7.1/download?dummy=
--->  Attempting to fetch env_logger-0.8.4.crate from https://crates.io/api/v1/crates/env_logger/0.8.4/download?dummy=
--->  Attempting to fetch env_logger-0.9.0.crate from https://crates.io/api/v1/crates/env_logger/0.9.0/download?dummy=
--->  Attempting to fetch expect-test-1.0.1.crate from https://crates.io/api/v1/crates/expect-test/1.0.1/download?dummy=
--->  Attempting to fetch fake-simd-0.1.2.crate from https://crates.io/api/v1/crates/fake-simd/0.1.2/download?dummy=
--->  Attempting to fetch fallible-iterator-0.2.0.crate from https://crates.io/api/v1/crates/fallible-iterator/0.2.0/download?dummy=
--->  Attempting to fetch filetime-0.2.14.crate from https://crates.io/api/v1/crates/filetime/0.2.14/download?dummy=
--->  Attempting to fetch fixedbitset-0.2.0.crate from https://crates.io/api/v1/crates/fixedbitset/0.2.0/download?dummy=
--->  Attempting to fetch flate2-1.0.16.crate from https://crates.io/api/v1/crates/flate2/1.0.16/download?dummy=
--->  Attempting to fetch fnv-1.0.7.crate from https://crates.io/api/v1/crates/fnv/1.0.7/download?dummy=
--->  Attempting to fetch foreign-types-0.3.2.crate from https://crates.io/api/v1/crates/foreign-types/0.3.2/download?dummy=
--->  Attempting to fetch foreign-types-shared-0.1.1.crate from https://crates.io/api/v1/crates/foreign-types-shared/0.1.1/download?dummy=
--->  Attempting to fetch form_urlencoded-1.0.1.crate from https://crates.io/api/v1/crates/form_urlencoded/1.0.1/download?dummy=
--->  Attempting to fetch fortanix-sgx-abi-0.3.3.crate from https://crates.io/api/v1/crates/fortanix-sgx-abi/0.3.3/download?dummy=
--->  Attempting to fetch fs-err-2.5.0.crate from https://crates.io/api/v1/crates/fs-err/2.5.0/download?dummy=
--->  Attempting to fetch fs_extra-1.1.0.crate from https://crates.io/api/v1/crates/fs_extra/1.1.0/download?dummy=
--->  Attempting to fetch fst-0.4.5.crate from https://crates.io/api/v1/crates/fst/0.4.5/download?dummy=
--->  Attempting to fetch futf-0.1.4.crate from https://crates.io/api/v1/crates/futf/0.1.4/download?dummy=
--->  Attempting to fetch futures-0.1.31.crate from https://crates.io/api/v1/crates/futures/0.1.31/download?dummy=
--->  Attempting to fetch futures-0.3.19.crate from https://crates.io/api/v1/crates/futures/0.3.19/download?dummy=
--->  Attempting to fetch futures-channel-0.3.19.crate from https://crates.io/api/v1/crates/futures-channel/0.3.19/download?dummy=
--->  Attempting to fetch futures-core-0.3.19.crate from https://crates.io/api/v1/crates/futures-core/0.3.19/download?dummy=
--->  Attempting to fetch futures-executor-0.3.19.crate from https://crates.io/api/v1/crates/futures-executor/0.3.19/download?dummy=
--->  Attempting to fetch futures-io-0.3.19.crate from https://crates.io/api/v1/crates/futures-io/0.3.19/download?dummy=
--->  Attempting to fetch futures-macro-0.3.19.crate from https://crates.io/api/v1/crates/futures-macro/0.3.19/download?dummy=
--->  Attempting to fetch futures-sink-0.3.19.crate from https://crates.io/api/v1/crates/futures-sink/0.3.19/download?dummy=
--->  Attempting to fetch futures-task-0.3.19.crate from https://crates.io/api/v1/crates/futures-task/0.3.19/download?dummy=
--->  Attempting to fetch futures-util-0.3.19.crate from https://crates.io/api/v1/crates/futures-util/0.3.19/download?dummy=
--->  Attempting to fetch fwdansi-1.1.0.crate from https://crates.io/api/v1/crates/fwdansi/1.1.0/download?dummy=
--->  Attempting to fetch generic-array-0.12.4.crate from https://crates.io/api/v1/crates/generic-array/0.12.4/download?dummy=
--->  Attempting to fetch generic-array-0.14.4.crate from https://crates.io/api/v1/crates/generic-array/0.14.4/download?dummy=
--->  Attempting to fetch getopts-0.2.21.crate from https://crates.io/api/v1/crates/getopts/0.2.21/download?dummy=
--->  Attempting to fetch getrandom-0.1.14.crate from https://crates.io/api/v1/crates/getrandom/0.1.14/download?dummy=
--->  Attempting to fetch getrandom-0.2.0.crate from https://crates.io/api/v1/crates/getrandom/0.2.0/download?dummy=
--->  Attempting to fetch getset-0.1.1.crate from https://crates.io/api/v1/crates/getset/0.1.1/download?dummy=
--->  Attempting to fetch gimli-0.25.0.crate from https://crates.io/api/v1/crates/gimli/0.25.0/download?dummy=
--->  Attempting to fetch gimli-0.26.1.crate from https://crates.io/api/v1/crates/gimli/0.26.1/download?dummy=
--->  Attempting to fetch git2-0.14.2.crate from https://crates.io/api/v1/crates/git2/0.14.2/download?dummy=
--->  Attempting to fetch git2-curl-0.15.0.crate from https://crates.io/api/v1/crates/git2-curl/0.15.0/download?dummy=
--->  Attempting to fetch glob-0.3.0.crate from https://crates.io/api/v1/crates/glob/0.3.0/download?dummy=
--->  Attempting to fetch globset-0.4.5.crate from https://crates.io/api/v1/crates/globset/0.4.5/download?dummy=
--->  Attempting to fetch gsgdt-0.1.2.crate from https://crates.io/api/v1/crates/gsgdt/0.1.2/download?dummy=
--->  Attempting to fetch handlebars-4.1.0.crate from https://crates.io/api/v1/crates/handlebars/4.1.0/download?dummy=
--->  Attempting to fetch hashbrown-0.11.2.crate from https://crates.io/api/v1/crates/hashbrown/0.11.2/download?dummy=
--->  Attempting to fetch hashbrown-0.12.0.crate from https://crates.io/api/v1/crates/hashbrown/0.12.0/download?dummy=
--->  Attempting to fetch heck-0.3.1.crate from https://crates.io/api/v1/crates/heck/0.3.1/download?dummy=
--->  Attempting to fetch hermit-abi-0.1.19.crate from https://crates.io/api/v1/crates/hermit-abi/0.1.19/download?dummy=
--->  Attempting to fetch hermit-abi-0.2.0.crate from https://crates.io/api/v1/crates/hermit-abi/0.2.0/download?dummy=
--->  Attempting to fetch hex-0.3.2.crate from https://crates.io/api/v1/crates/hex/0.3.2/download?dummy=
--->  Attempting to fetch hex-0.4.2.crate from https://crates.io/api/v1/crates/hex/0.4.2/download?dummy=
--->  Attempting to fetch home-0.5.3.crate from https://crates.io/api/v1/crates/home/0.5.3/download?dummy=
--->  Attempting to fetch html5ever-0.25.1.crate from https://crates.io/api/v1/crates/html5ever/0.25.1/download?dummy=
--->  Attempting to fetch humantime-1.3.0.crate from https://crates.io/api/v1/crates/humantime/1.3.0/download?dummy=
--->  Attempting to fetch humantime-2.0.1.crate from https://crates.io/api/v1/crates/humantime/2.0.1/download?dummy=
--->  Attempting to fetch idna-0.1.5.crate from https://crates.io/api/v1/crates/idna/0.1.5/download?dummy=
--->  Attempting to fetch idna-0.2.0.crate from https://crates.io/api/v1/crates/idna/0.2.0/download?dummy=
--->  Attempting to fetch if_chain-1.0.0.crate from https://crates.io/api/v1/crates/if_chain/1.0.0/download?dummy=
--->  Attempting to fetch ignore-0.4.17.crate from https://crates.io/api/v1/crates/ignore/0.4.17/download?dummy=
--->  Attempting to fetch im-rc-15.0.0.crate from https://crates.io/api/v1/crates/im-rc/15.0.0/download?dummy=
--->  Attempting to fetch indexmap-1.8.0.crate from https://crates.io/api/v1/crates/indexmap/1.8.0/download?dummy=
--->  Attempting to fetch indoc-1.0.3.crate from https://crates.io/api/v1/crates/indoc/1.0.3/download?dummy=
--->  Attempting to fetch instant-0.1.12.crate from https://crates.io/api/v1/crates/instant/0.1.12/download?dummy=
--->  Attempting to fetch itertools-0.10.1.crate from https://crates.io/api/v1/crates/itertools/0.10.1/download?dummy=
--->  Attempting to fetch itoa-0.4.6.crate from https://crates.io/api/v1/crates/itoa/0.4.6/download?dummy=
--->  Attempting to fetch jobserver-0.1.24.crate from https://crates.io/api/v1/crates/jobserver/0.1.24/download?dummy=
--->  Attempting to fetch json-0.12.4.crate from https://crates.io/api/v1/crates/json/0.12.4/download?dummy=
--->  Attempting to fetch jsonpath_lib-0.2.6.crate from https://crates.io/api/v1/crates/jsonpath_lib/0.2.6/download?dummy=
--->  Attempting to fetch jsonrpc-client-transports-18.0.0.crate from https://crates.io/api/v1/crates/jsonrpc-client-transports/18.0.0/download?dummy=
--->  Attempting to fetch jsonrpc-core-18.0.0.crate from https://crates.io/api/v1/crates/jsonrpc-core/18.0.0/download?dummy=
--->  Attempting to fetch jsonrpc-core-client-18.0.0.crate from https://crates.io/api/v1/crates/jsonrpc-core-client/18.0.0/download?dummy=
--->  Attempting to fetch jsonrpc-derive-18.0.0.crate from https://crates.io/api/v1/crates/jsonrpc-derive/18.0.0/download?dummy=
--->  Attempting to fetch jsonrpc-ipc-server-18.0.0.crate from https://crates.io/api/v1/crates/jsonrpc-ipc-server/18.0.0/download?dummy=
--->  Attempting to fetch jsonrpc-pubsub-18.0.0.crate from https://crates.io/api/v1/crates/jsonrpc-pubsub/18.0.0/download?dummy=
--->  Attempting to fetch jsonrpc-server-utils-18.0.0.crate from https://crates.io/api/v1/crates/jsonrpc-server-utils/18.0.0/download?dummy=
--->  Attempting to fetch kstring-1.0.6.crate from https://crates.io/api/v1/crates/kstring/1.0.6/download?dummy=
--->  Attempting to fetch lazy_static-1.4.0.crate from https://crates.io/api/v1/crates/lazy_static/1.4.0/download?dummy=
--->  Attempting to fetch lazycell-1.3.0.crate from https://crates.io/api/v1/crates/lazycell/1.3.0/download?dummy=
--->  Attempting to fetch libc-0.2.116.crate from https://crates.io/api/v1/crates/libc/0.2.116/download?dummy=
--->  Attempting to fetch libgit2-sys-0.13.2+1.4.2.crate from https://crates.io/api/v1/crates/libgit2-sys/0.13.2+1.4.2/download?dummy=
--->  Attempting to fetch libloading-0.7.1.crate from https://crates.io/api/v1/crates/libloading/0.7.1/download?dummy=
--->  Attempting to fetch libm-0.1.4.crate from https://crates.io/api/v1/crates/libm/0.1.4/download?dummy=
--->  Attempting to fetch libnghttp2-sys-0.1.4+1.41.0.crate from https://crates.io/api/v1/crates/libnghttp2-sys/0.1.4+1.41.0/download?dummy=
--->  Attempting to fetch libssh2-sys-0.2.23.crate from https://crates.io/api/v1/crates/libssh2-sys/0.2.23/download?dummy=
--->  Attempting to fetch libz-sys-1.1.3.crate from https://crates.io/api/v1/crates/libz-sys/1.1.3/download?dummy=
--->  Attempting to fetch linked-hash-map-0.5.4.crate from https://crates.io/api/v1/crates/linked-hash-map/0.5.4/download?dummy=
--->  Attempting to fetch lock_api-0.4.5.crate from https://crates.io/api/v1/crates/lock_api/0.4.5/download?dummy=
--->  Attempting to fetch log-0.4.14.crate from https://crates.io/api/v1/crates/log/0.4.14/download?dummy=
--->  Attempting to fetch lsp-codec-0.3.0.crate from https://crates.io/api/v1/crates/lsp-codec/0.3.0/download?dummy=
--->  Attempting to fetch lsp-types-0.60.0.crate from https://crates.io/api/v1/crates/lsp-types/0.60.0/download?dummy=
--->  Attempting to fetch lzma-sys-0.1.16.crate from https://crates.io/api/v1/crates/lzma-sys/0.1.16/download?dummy=
--->  Attempting to fetch mac-0.1.1.crate from https://crates.io/api/v1/crates/mac/0.1.1/download?dummy=
--->  Attempting to fetch macro-utils-0.1.3.crate from https://crates.io/api/v1/crates/macro-utils/0.1.3/download?dummy=
--->  Attempting to fetch maplit-1.0.2.crate from https://crates.io/api/v1/crates/maplit/1.0.2/download?dummy=
--->  Attempting to fetch markup5ever-0.10.1.crate from https://crates.io/api/v1/crates/markup5ever/0.10.1/download?dummy=
--->  Attempting to fetch markup5ever_rcdom-0.1.0.crate from https://crates.io/api/v1/crates/markup5ever_rcdom/0.1.0/download?dummy=
--->  Attempting to fetch matchers-0.1.0.crate from https://crates.io/api/v1/crates/matchers/0.1.0/download?dummy=
--->  Attempting to fetch matches-0.1.8.crate from https://crates.io/api/v1/crates/matches/0.1.8/download?dummy=
--->  Attempting to fetch md-5-0.10.0.crate from https://crates.io/api/v1/crates/md-5/0.10.0/download?dummy=
--->  Attempting to fetch mdbook-0.4.15.crate from https://crates.io/api/v1/crates/mdbook/0.4.15/download?dummy=
--->  Attempting to fetch measureme-9.1.2.crate from https://crates.io/api/v1/crates/measureme/9.1.2/download?dummy=
--->  Attempting to fetch measureme-10.0.0.crate from https://crates.io/api/v1/crates/measureme/10.0.0/download?dummy=
--->  Attempting to fetch memchr-2.4.1.crate from https://crates.io/api/v1/crates/memchr/2.4.1/download?dummy=
--->  Attempting to fetch memmap2-0.2.1.crate from https://crates.io/api/v1/crates/memmap2/0.2.1/download?dummy=
--->  Attempting to fetch memoffset-0.6.5.crate from https://crates.io/api/v1/crates/memoffset/0.6.5/download?dummy=
--->  Attempting to fetch minifier-0.0.43.crate from https://crates.io/api/v1/crates/minifier/0.0.43/download?dummy=
--->  Attempting to fetch minimal-lexical-0.2.1.crate from https://crates.io/api/v1/crates/minimal-lexical/0.2.1/download?dummy=
--->  Attempting to fetch miniz_oxide-0.4.0.crate from https://crates.io/api/v1/crates/miniz_oxide/0.4.0/download?dummy=
--->  Attempting to fetch mio-0.7.14.crate from https://crates.io/api/v1/crates/mio/0.7.14/download?dummy=
--->  Attempting to fetch miow-0.3.7.crate from https://crates.io/api/v1/crates/miow/0.3.7/download?dummy=
--->  Attempting to fetch new_debug_unreachable-1.0.4.crate from https://crates.io/api/v1/crates/new_debug_unreachable/1.0.4/download?dummy=
--->  Attempting to fetch nom-7.1.0.crate from https://crates.io/api/v1/crates/nom/7.1.0/download?dummy=
--->  Attempting to fetch ntapi-0.3.6.crate from https://crates.io/api/v1/crates/ntapi/0.3.6/download?dummy=
--->  Attempting to fetch num-integer-0.1.43.crate from https://crates.io/api/v1/crates/num-integer/0.1.43/download?dummy=
--->  Attempting to fetch num-traits-0.2.12.crate from https://crates.io/api/v1/crates/num-traits/0.2.12/download?dummy=
--->  Attempting to fetch num_cpus-1.13.1.crate from https://crates.io/api/v1/crates/num_cpus/1.13.1/download?dummy=
--->  Attempting to fetch object-0.26.2.crate from https://crates.io/api/v1/crates/object/0.26.2/download?dummy=
--->  Attempting to fetch object-0.28.1.crate from https://crates.io/api/v1/crates/object/0.28.1/download?dummy=
--->  Attempting to fetch odht-0.3.1.crate from https://crates.io/api/v1/crates/odht/0.3.1/download?dummy=
--->  Attempting to fetch once_cell-1.7.2.crate from https://crates.io/api/v1/crates/once_cell/1.7.2/download?dummy=
--->  Attempting to fetch opaque-debug-0.2.3.crate from https://crates.io/api/v1/crates/opaque-debug/0.2.3/download?dummy=
--->  Attempting to fetch opener-0.5.0.crate from https://crates.io/api/v1/crates/opener/0.5.0/download?dummy=
--->  Attempting to fetch openssl-0.10.35.crate from https://crates.io/api/v1/crates/openssl/0.10.35/download?dummy=
--->  Attempting to fetch openssl-probe-0.1.2.crate from https://crates.io/api/v1/crates/openssl-probe/0.1.2/download?dummy=
--->  Attempting to fetch openssl-src-111.17.0+1.1.1m.crate from https://crates.io/api/v1/crates/openssl-src/111.17.0+1.1.1m/download?dummy=
--->  Attempting to fetch openssl-sys-0.9.65.crate from https://crates.io/api/v1/crates/openssl-sys/0.9.65/download?dummy=
--->  Attempting to fetch ordslice-0.3.0.crate from https://crates.io/api/v1/crates/ordslice/0.3.0/download?dummy=
--->  Attempting to fetch os_info-3.0.7.crate from https://crates.io/api/v1/crates/os_info/3.0.7/download?dummy=
--->  Attempting to fetch os_str_bytes-6.0.0.crate from https://crates.io/api/v1/crates/os_str_bytes/6.0.0/download?dummy=
--->  Attempting to fetch output_vt100-0.1.2.crate from https://crates.io/api/v1/crates/output_vt100/0.1.2/download?dummy=
--->  Attempting to fetch packed_simd_2-0.3.4.crate from https://crates.io/api/v1/crates/packed_simd_2/0.3.4/download?dummy=
--->  Attempting to fetch parity-tokio-ipc-0.9.0.crate from https://crates.io/api/v1/crates/parity-tokio-ipc/0.9.0/download?dummy=
--->  Attempting to fetch parking_lot-0.11.2.crate from https://crates.io/api/v1/crates/parking_lot/0.11.2/download?dummy=
--->  Attempting to fetch parking_lot_core-0.8.5.crate from https://crates.io/api/v1/crates/parking_lot_core/0.8.5/download?dummy=
--->  Attempting to fetch pathdiff-0.2.0.crate from https://crates.io/api/v1/crates/pathdiff/0.2.0/download?dummy=
--->  Attempting to fetch percent-encoding-1.0.1.crate from https://crates.io/api/v1/crates/percent-encoding/1.0.1/download?dummy=
--->  Attempting to fetch percent-encoding-2.1.0.crate from https://crates.io/api/v1/crates/percent-encoding/2.1.0/download?dummy=
--->  Attempting to fetch perf-event-open-sys-1.0.1.crate from https://crates.io/api/v1/crates/perf-event-open-sys/1.0.1/download?dummy=
--->  Attempting to fetch pest-2.1.3.crate from https://crates.io/api/v1/crates/pest/2.1.3/download?dummy=
--->  Attempting to fetch pest_derive-2.1.0.crate from https://crates.io/api/v1/crates/pest_derive/2.1.0/download?dummy=
--->  Attempting to fetch pest_generator-2.1.3.crate from https://crates.io/api/v1/crates/pest_generator/2.1.3/download?dummy=
--->  Attempting to fetch pest_meta-2.1.3.crate from https://crates.io/api/v1/crates/pest_meta/2.1.3/download?dummy=
--->  Attempting to fetch petgraph-0.5.1.crate from https://crates.io/api/v1/crates/petgraph/0.5.1/download?dummy=
--->  Attempting to fetch phf-0.8.0.crate from https://crates.io/api/v1/crates/phf/0.8.0/download?dummy=
--->  Attempting to fetch phf_codegen-0.8.0.crate from https://crates.io/api/v1/crates/phf_codegen/0.8.0/download?dummy=
--->  Attempting to fetch phf_generator-0.8.0.crate from https://crates.io/api/v1/crates/phf_generator/0.8.0/download?dummy=
--->  Attempting to fetch phf_shared-0.8.0.crate from https://crates.io/api/v1/crates/phf_shared/0.8.0/download?dummy=
--->  Attempting to fetch pin-project-lite-0.2.8.crate from https://crates.io/api/v1/crates/pin-project-lite/0.2.8/download?dummy=
--->  Attempting to fetch pin-utils-0.1.0.crate from https://crates.io/api/v1/crates/pin-utils/0.1.0/download?dummy=
--->  Attempting to fetch pkg-config-0.3.18.crate from https://crates.io/api/v1/crates/pkg-config/0.3.18/download?dummy=
--->  Attempting to fetch polonius-engine-0.13.0.crate from https://crates.io/api/v1/crates/polonius-engine/0.13.0/download?dummy=
--->  Attempting to fetch ppv-lite86-0.2.8.crate from https://crates.io/api/v1/crates/ppv-lite86/0.2.8/download?dummy=
--->  Attempting to fetch precomputed-hash-0.1.1.crate from https://crates.io/api/v1/crates/precomputed-hash/0.1.1/download?dummy=
--->  Attempting to fetch pretty_assertions-0.7.2.crate from https://crates.io/api/v1/crates/pretty_assertions/0.7.2/download?dummy=
--->  Attempting to fetch pretty_env_logger-0.4.0.crate from https://crates.io/api/v1/crates/pretty_env_logger/0.4.0/download?dummy=
--->  Attempting to fetch proc-macro-crate-0.1.5.crate from https://crates.io/api/v1/crates/proc-macro-crate/0.1.5/download?dummy=
--->  Attempting to fetch proc-macro-error-1.0.4.crate from https://crates.io/api/v1/crates/proc-macro-error/1.0.4/download?dummy=
--->  Attempting to fetch proc-macro-error-attr-1.0.4.crate from https://crates.io/api/v1/crates/proc-macro-error-attr/1.0.4/download?dummy=
--->  Attempting to fetch proc-macro2-1.0.30.crate from https://crates.io/api/v1/crates/proc-macro2/1.0.30/download?dummy=
--->  Attempting to fetch psm-0.1.16.crate from https://crates.io/api/v1/crates/psm/0.1.16/download?dummy=
--->  Attempting to fetch pulldown-cmark-0.9.1.crate from https://crates.io/api/v1/crates/pulldown-cmark/0.9.1/download?dummy=
--->  Attempting to fetch punycode-0.4.1.crate from https://crates.io/api/v1/crates/punycode/0.4.1/download?dummy=
--->  Attempting to fetch quick-error-1.2.3.crate from https://crates.io/api/v1/crates/quick-error/1.2.3/download?dummy=
--->  Attempting to fetch quick-error-2.0.0.crate from https://crates.io/api/v1/crates/quick-error/2.0.0/download?dummy=
--->  Attempting to fetch quine-mc_cluskey-0.2.4.crate from https://crates.io/api/v1/crates/quine-mc_cluskey/0.2.4/download?dummy=
--->  Attempting to fetch quote-1.0.7.crate from https://crates.io/api/v1/crates/quote/1.0.7/download?dummy=
--->  Attempting to fetch racer-2.2.1.crate from https://crates.io/api/v1/crates/racer/2.2.1/download?dummy=
--->  Attempting to fetch rand-0.7.3.crate from https://crates.io/api/v1/crates/rand/0.7.3/download?dummy=
--->  Attempting to fetch rand-0.8.4.crate from https://crates.io/api/v1/crates/rand/0.8.4/download?dummy=
--->  Attempting to fetch rand_chacha-0.2.2.crate from https://crates.io/api/v1/crates/rand_chacha/0.2.2/download?dummy=
--->  Attempting to fetch rand_chacha-0.3.0.crate from https://crates.io/api/v1/crates/rand_chacha/0.3.0/download?dummy=
--->  Attempting to fetch rand_core-0.5.1.crate from https://crates.io/api/v1/crates/rand_core/0.5.1/download?dummy=
--->  Attempting to fetch rand_core-0.6.2.crate from https://crates.io/api/v1/crates/rand_core/0.6.2/download?dummy=
--->  Attempting to fetch rand_hc-0.2.0.crate from https://crates.io/api/v1/crates/rand_hc/0.2.0/download?dummy=
--->  Attempting to fetch rand_hc-0.3.0.crate from https://crates.io/api/v1/crates/rand_hc/0.3.0/download?dummy=
--->  Attempting to fetch rand_pcg-0.2.1.crate from https://crates.io/api/v1/crates/rand_pcg/0.2.1/download?dummy=
--->  Attempting to fetch rand_xorshift-0.2.0.crate from https://crates.io/api/v1/crates/rand_xorshift/0.2.0/download?dummy=
--->  Attempting to fetch rand_xoshiro-0.4.0.crate from https://crates.io/api/v1/crates/rand_xoshiro/0.4.0/download?dummy=
--->  Attempting to fetch rand_xoshiro-0.6.0.crate from https://crates.io/api/v1/crates/rand_xoshiro/0.6.0/download?dummy=
--->  Attempting to fetch rayon-1.5.1.crate from https://crates.io/api/v1/crates/rayon/1.5.1/download?dummy=
--->  Attempting to fetch rayon-core-1.9.1.crate from https://crates.io/api/v1/crates/rayon-core/1.9.1/download?dummy=
--->  Attempting to fetch redox_syscall-0.2.10.crate from https://crates.io/api/v1/crates/redox_syscall/0.2.10/download?dummy=
--->  Attempting to fetch redox_users-0.4.0.crate from https://crates.io/api/v1/crates/redox_users/0.4.0/download?dummy=
--->  Attempting to fetch regex-1.5.4.crate from https://crates.io/api/v1/crates/regex/1.5.4/download?dummy=
--->  Attempting to fetch regex-automata-0.1.10.crate from https://crates.io/api/v1/crates/regex-automata/0.1.10/download?dummy=
--->  Attempting to fetch regex-syntax-0.6.25.crate from https://crates.io/api/v1/crates/regex-syntax/0.6.25/download?dummy=
--->  Attempting to fetch remove_dir_all-0.5.3.crate from https://crates.io/api/v1/crates/remove_dir_all/0.5.3/download?dummy=
--->  Attempting to fetch rls-data-0.19.1.crate from https://crates.io/api/v1/crates/rls-data/0.19.1/download?dummy=
--->  Attempting to fetch rls-span-0.5.3.crate from https://crates.io/api/v1/crates/rls-span/0.5.3/download?dummy=
--->  Attempting to fetch rls-vfs-0.8.0.crate from https://crates.io/api/v1/crates/rls-vfs/0.8.0/download?dummy=
--->  Attempting to fetch rustc-demangle-0.1.21.crate from https://crates.io/api/v1/crates/rustc-demangle/0.1.21/download?dummy=
--->  Attempting to fetch rustc-hash-1.1.0.crate from https://crates.io/api/v1/crates/rustc-hash/1.1.0/download?dummy=
--->  Attempting to fetch rustc-rayon-0.3.2.crate from https://crates.io/api/v1/crates/rustc-rayon/0.3.2/download?dummy=
--->  Attempting to fetch rustc-rayon-core-0.3.2.crate from https://crates.io/api/v1/crates/rustc-rayon-core/0.3.2/download?dummy=
--->  Attempting to fetch rustc-semver-1.1.0.crate from https://crates.io/api/v1/crates/rustc-semver/1.1.0/download?dummy=
--->  Attempting to fetch rustc_tools_util-0.2.0.crate from https://crates.io/api/v1/crates/rustc_tools_util/0.2.0/download?dummy=
--->  Attempting to fetch rustc_version-0.4.0.crate from https://crates.io/api/v1/crates/rustc_version/0.4.0/download?dummy=
--->  Attempting to fetch rustfix-0.5.1.crate from https://crates.io/api/v1/crates/rustfix/0.5.1/download?dummy=
--->  Attempting to fetch rustfix-0.6.0.crate from https://crates.io/api/v1/crates/rustfix/0.6.0/download?dummy=
--->  Attempting to fetch rustversion-1.0.5.crate from https://crates.io/api/v1/crates/rustversion/1.0.5/download?dummy=
--->  Attempting to fetch ryu-1.0.5.crate from https://crates.io/api/v1/crates/ryu/1.0.5/download?dummy=
--->  Attempting to fetch same-file-1.0.6.crate from https://crates.io/api/v1/crates/same-file/1.0.6/download?dummy=
--->  Attempting to fetch schannel-0.1.19.crate from https://crates.io/api/v1/crates/schannel/0.1.19/download?dummy=
--->  Attempting to fetch scoped-tls-1.0.0.crate from https://crates.io/api/v1/crates/scoped-tls/1.0.0/download?dummy=
--->  Attempting to fetch scopeguard-1.1.0.crate from https://crates.io/api/v1/crates/scopeguard/1.1.0/download?dummy=
--->  Attempting to fetch security-framework-2.0.0.crate from https://crates.io/api/v1/crates/security-framework/2.0.0/download?dummy=
--->  Attempting to fetch security-framework-sys-2.0.0.crate from https://crates.io/api/v1/crates/security-framework-sys/2.0.0/download?dummy=
--->  Attempting to fetch semver-1.0.3.crate from https://crates.io/api/v1/crates/semver/1.0.3/download?dummy=
--->  Attempting to fetch serde-1.0.125.crate from https://crates.io/api/v1/crates/serde/1.0.125/download?dummy=
--->  Attempting to fetch serde_derive-1.0.125.crate from https://crates.io/api/v1/crates/serde_derive/1.0.125/download?dummy=
--->  Attempting to fetch serde_ignored-0.1.2.crate from https://crates.io/api/v1/crates/serde_ignored/0.1.2/download?dummy=
--->  Attempting to fetch serde_json-1.0.59.crate from https://crates.io/api/v1/crates/serde_json/1.0.59/download?dummy=
--->  Attempting to fetch serde_repr-0.1.6.crate from https://crates.io/api/v1/crates/serde_repr/0.1.6/download?dummy=
--->  Attempting to fetch sha-1-0.8.2.crate from https://crates.io/api/v1/crates/sha-1/0.8.2/download?dummy=
--->  Attempting to fetch sha-1-0.10.0.crate from https://crates.io/api/v1/crates/sha-1/0.10.0/download?dummy=
--->  Attempting to fetch sha2-0.10.1.crate from https://crates.io/api/v1/crates/sha2/0.10.1/download?dummy=
--->  Attempting to fetch sharded-slab-0.1.1.crate from https://crates.io/api/v1/crates/sharded-slab/0.1.1/download?dummy=
--->  Attempting to fetch shell-escape-0.1.5.crate from https://crates.io/api/v1/crates/shell-escape/0.1.5/download?dummy=
--->  Attempting to fetch shlex-1.0.0.crate from https://crates.io/api/v1/crates/shlex/1.0.0/download?dummy=
--->  Attempting to fetch signal-hook-registry-1.2.2.crate from https://crates.io/api/v1/crates/signal-hook-registry/1.2.2/download?dummy=
--->  Attempting to fetch siphasher-0.3.3.crate from https://crates.io/api/v1/crates/siphasher/0.3.3/download?dummy=
--->  Attempting to fetch sized-chunks-0.6.4.crate from https://crates.io/api/v1/crates/sized-chunks/0.6.4/download?dummy=
--->  Attempting to fetch slab-0.4.2.crate from https://crates.io/api/v1/crates/slab/0.4.2/download?dummy=
--->  Attempting to fetch smallvec-1.7.0.crate from https://crates.io/api/v1/crates/smallvec/1.7.0/download?dummy=
--->  Attempting to fetch snap-1.0.1.crate from https://crates.io/api/v1/crates/snap/1.0.1/download?dummy=
--->  Attempting to fetch socket2-0.4.1.crate from https://crates.io/api/v1/crates/socket2/0.4.1/download?dummy=
--->  Attempting to fetch stable_deref_trait-1.2.0.crate from https://crates.io/api/v1/crates/stable_deref_trait/1.2.0/download?dummy=
--->  Attempting to fetch stacker-0.1.14.crate from https://crates.io/api/v1/crates/stacker/0.1.14/download?dummy=
--->  Attempting to fetch string_cache-0.8.0.crate from https://crates.io/api/v1/crates/string_cache/0.8.0/download?dummy=
--->  Attempting to fetch string_cache_codegen-0.5.1.crate from https://crates.io/api/v1/crates/string_cache_codegen/0.5.1/download?dummy=
--->  Attempting to fetch strip-ansi-escapes-0.1.0.crate from https://crates.io/api/v1/crates/strip-ansi-escapes/0.1.0/download?dummy=
--->  Attempting to fetch strsim-0.8.0.crate from https://crates.io/api/v1/crates/strsim/0.8.0/download?dummy=
--->  Attempting to fetch strsim-0.10.0.crate from https://crates.io/api/v1/crates/strsim/0.10.0/download?dummy=
--->  Attempting to fetch structopt-0.3.25.crate from https://crates.io/api/v1/crates/structopt/0.3.25/download?dummy=
--->  Attempting to fetch structopt-derive-0.4.18.crate from https://crates.io/api/v1/crates/structopt-derive/0.4.18/download?dummy=
--->  Attempting to fetch strum-0.18.0.crate from https://crates.io/api/v1/crates/strum/0.18.0/download?dummy=
--->  Attempting to fetch strum_macros-0.18.0.crate from https://crates.io/api/v1/crates/strum_macros/0.18.0/download?dummy=
--->  Attempting to fetch syn-1.0.80.crate from https://crates.io/api/v1/crates/syn/1.0.80/download?dummy=
--->  Attempting to fetch synstructure-0.12.6.crate from https://crates.io/api/v1/crates/synstructure/0.12.6/download?dummy=
--->  Attempting to fetch tar-0.4.37.crate from https://crates.io/api/v1/crates/tar/0.4.37/download?dummy=
--->  Attempting to fetch tempfile-3.2.0.crate from https://crates.io/api/v1/crates/tempfile/3.2.0/download?dummy=
--->  Attempting to fetch tendril-0.4.1.crate from https://crates.io/api/v1/crates/tendril/0.4.1/download?dummy=
--->  Attempting to fetch term-0.6.1.crate from https://crates.io/api/v1/crates/term/0.6.1/download?dummy=
--->  Attempting to fetch term-0.7.0.crate from https://crates.io/api/v1/crates/term/0.7.0/download?dummy=
--->  Attempting to fetch termcolor-1.1.2.crate from https://crates.io/api/v1/crates/termcolor/1.1.2/download?dummy=
--->  Attempting to fetch termize-0.1.1.crate from https://crates.io/api/v1/crates/termize/0.1.1/download?dummy=
--->  Attempting to fetch tester-0.9.0.crate from https://crates.io/api/v1/crates/tester/0.9.0/download?dummy=
--->  Attempting to fetch textwrap-0.11.0.crate from https://crates.io/api/v1/crates/textwrap/0.11.0/download?dummy=
--->  Attempting to fetch textwrap-0.14.2.crate from https://crates.io/api/v1/crates/textwrap/0.14.2/download?dummy=
--->  Attempting to fetch thiserror-1.0.30.crate from https://crates.io/api/v1/crates/thiserror/1.0.30/download?dummy=
--->  Attempting to fetch thiserror-impl-1.0.30.crate from https://crates.io/api/v1/crates/thiserror-impl/1.0.30/download?dummy=
--->  Attempting to fetch thorin-dwp-0.2.0.crate from https://crates.io/api/v1/crates/thorin-dwp/0.2.0/download?dummy=
--->  Attempting to fetch thread_local-1.1.4.crate from https://crates.io/api/v1/crates/thread_local/1.1.4/download?dummy=
--->  Attempting to fetch tikv-jemalloc-sys-0.4.1+5.2.1-patched.crate from https://crates.io/api/v1/crates/tikv-jemalloc-sys/0.4.1+5.2.1-patched/download?dummy=
--->  Attempting to fetch time-0.1.43.crate from https://crates.io/api/v1/crates/time/0.1.43/download?dummy=
--->  Attempting to fetch tinyvec-0.3.4.crate from https://crates.io/api/v1/crates/tinyvec/0.3.4/download?dummy=
--->  Attempting to fetch tokio-1.8.4.crate from https://crates.io/api/v1/crates/tokio/1.8.4/download?dummy=
--->  Attempting to fetch tokio-stream-0.1.7.crate from https://crates.io/api/v1/crates/tokio-stream/0.1.7/download?dummy=
--->  Attempting to fetch tokio-util-0.6.7.crate from https://crates.io/api/v1/crates/tokio-util/0.6.7/download?dummy=
--->  Attempting to fetch toml-0.5.7.crate from https://crates.io/api/v1/crates/toml/0.5.7/download?dummy=
--->  Attempting to fetch toml_edit-0.13.4.crate from https://crates.io/api/v1/crates/toml_edit/0.13.4/download?dummy=
--->  Attempting to fetch topological-sort-0.1.0.crate from https://crates.io/api/v1/crates/topological-sort/0.1.0/download?dummy=
--->  Attempting to fetch tower-service-0.3.1.crate from https://crates.io/api/v1/crates/tower-service/0.3.1/download?dummy=
--->  Attempting to fetch tracing-0.1.29.crate from https://crates.io/api/v1/crates/tracing/0.1.29/download?dummy=
--->  Attempting to fetch tracing-attributes-0.1.18.crate from https://crates.io/api/v1/crates/tracing-attributes/0.1.18/download?dummy=
--->  Attempting to fetch tracing-core-0.1.21.crate from https://crates.io/api/v1/crates/tracing-core/0.1.21/download?dummy=
--->  Attempting to fetch tracing-log-0.1.2.crate from https://crates.io/api/v1/crates/tracing-log/0.1.2/download?dummy=
--->  Attempting to fetch tracing-subscriber-0.3.3.crate from https://crates.io/api/v1/crates/tracing-subscriber/0.3.3/download?dummy=
--->  Attempting to fetch tracing-tree-0.2.0.crate from https://crates.io/api/v1/crates/tracing-tree/0.2.0/download?dummy=
--->  Attempting to fetch typenum-1.12.0.crate from https://crates.io/api/v1/crates/typenum/1.12.0/download?dummy=
--->  Attempting to fetch ucd-parse-0.1.8.crate from https://crates.io/api/v1/crates/ucd-parse/0.1.8/download?dummy=
--->  Attempting to fetch ucd-trie-0.1.3.crate from https://crates.io/api/v1/crates/ucd-trie/0.1.3/download?dummy=
--->  Attempting to fetch unic-char-property-0.9.0.crate from https://crates.io/api/v1/crates/unic-char-property/0.9.0/download?dummy=
--->  Attempting to fetch unic-char-range-0.9.0.crate from https://crates.io/api/v1/crates/unic-char-range/0.9.0/download?dummy=
--->  Attempting to fetch unic-common-0.9.0.crate from https://crates.io/api/v1/crates/unic-common/0.9.0/download?dummy=
--->  Attempting to fetch unic-emoji-char-0.9.0.crate from https://crates.io/api/v1/crates/unic-emoji-char/0.9.0/download?dummy=
--->  Attempting to fetch unic-ucd-version-0.9.0.crate from https://crates.io/api/v1/crates/unic-ucd-version/0.9.0/download?dummy=
--->  Attempting to fetch unicase-2.6.0.crate from https://crates.io/api/v1/crates/unicase/2.6.0/download?dummy=
--->  Attempting to fetch unicode-bidi-0.3.4.crate from https://crates.io/api/v1/crates/unicode-bidi/0.3.4/download?dummy=
--->  Attempting to fetch unicode-normalization-0.1.13.crate from https://crates.io/api/v1/crates/unicode-normalization/0.1.13/download?dummy=
--->  Attempting to fetch unicode-script-0.5.3.crate from https://crates.io/api/v1/crates/unicode-script/0.5.3/download?dummy=
--->  Attempting to fetch unicode-security-0.0.5.crate from https://crates.io/api/v1/crates/unicode-security/0.0.5/download?dummy=
--->  Attempting to fetch unicode-segmentation-1.6.0.crate from https://crates.io/api/v1/crates/unicode-segmentation/1.6.0/download?dummy=
--->  Attempting to fetch unicode-width-0.1.8.crate from https://crates.io/api/v1/crates/unicode-width/0.1.8/download?dummy=
--->  Attempting to fetch unicode-xid-0.2.2.crate from https://crates.io/api/v1/crates/unicode-xid/0.2.2/download?dummy=
--->  Attempting to fetch unicode_categories-0.1.1.crate from https://crates.io/api/v1/crates/unicode_categories/0.1.1/download?dummy=
--->  Attempting to fetch unified-diff-0.2.1.crate from https://crates.io/api/v1/crates/unified-diff/0.2.1/download?dummy=
--->  Attempting to fetch unindent-0.1.7.crate from https://crates.io/api/v1/crates/unindent/0.1.7/download?dummy=
--->  Attempting to fetch url-1.7.2.crate from https://crates.io/api/v1/crates/url/1.7.2/download?dummy=
--->  Attempting to fetch url-2.2.2.crate from https://crates.io/api/v1/crates/url/2.2.2/download?dummy=
--->  Attempting to fetch utf-8-0.7.5.crate from https://crates.io/api/v1/crates/utf-8/0.7.5/download?dummy=
--->  Attempting to fetch utf8parse-0.1.1.crate from https://crates.io/api/v1/crates/utf8parse/0.1.1/download?dummy=
--->  Attempting to fetch vcpkg-0.2.10.crate from https://crates.io/api/v1/crates/vcpkg/0.2.10/download?dummy=
--->  Attempting to fetch vec_map-0.8.2.crate from https://crates.io/api/v1/crates/vec_map/0.8.2/download?dummy=
--->  Attempting to fetch vergen-5.1.0.crate from https://crates.io/api/v1/crates/vergen/5.1.0/download?dummy=
--->  Attempting to fetch version_check-0.9.3.crate from https://crates.io/api/v1/crates/version_check/0.9.3/download?dummy=
--->  Attempting to fetch vte-0.3.3.crate from https://crates.io/api/v1/crates/vte/0.3.3/download?dummy=
--->  Attempting to fetch walkdir-2.3.1.crate from https://crates.io/api/v1/crates/walkdir/2.3.1/download?dummy=
--->  Attempting to fetch wasi-0.9.0+wasi-snapshot-preview1.crate from https://crates.io/api/v1/crates/wasi/0.9.0+wasi-snapshot-preview1/download?dummy=
--->  Attempting to fetch wasi-0.11.0+wasi-snapshot-preview1.crate from https://crates.io/api/v1/crates/wasi/0.11.0+wasi-snapshot-preview1/download?dummy=
--->  Attempting to fetch winapi-0.3.9.crate from https://crates.io/api/v1/crates/winapi/0.3.9/download?dummy=
--->  Attempting to fetch winapi-i686-pc-windows-gnu-0.4.0.crate from https://crates.io/api/v1/crates/winapi-i686-pc-windows-gnu/0.4.0/download?dummy=
--->  Attempting to fetch winapi-util-0.1.5.crate from https://crates.io/api/v1/crates/winapi-util/0.1.5/download?dummy=
--->  Attempting to fetch winapi-x86_64-pc-windows-gnu-0.4.0.crate from https://crates.io/api/v1/crates/winapi-x86_64-pc-windows-gnu/0.4.0/download?dummy=
--->  Attempting to fetch xattr-0.2.2.crate from https://crates.io/api/v1/crates/xattr/0.2.2/download?dummy=
--->  Attempting to fetch xml5ever-0.16.1.crate from https://crates.io/api/v1/crates/xml5ever/0.16.1/download?dummy=
--->  Attempting to fetch xz2-0.1.6.crate from https://crates.io/api/v1/crates/xz2/0.1.6/download?dummy=
--->  Attempting to fetch yaml-merge-keys-0.4.1.crate from https://crates.io/api/v1/crates/yaml-merge-keys/0.4.1/download?dummy=
--->  Attempting to fetch yaml-rust-0.3.5.crate from https://crates.io/api/v1/crates/yaml-rust/0.3.5/download?dummy=
--->  Attempting to fetch yaml-rust-0.4.4.crate from https://crates.io/api/v1/crates/yaml-rust/0.4.4/download?dummy=
--->  Attempting to fetch yansi-term-0.1.2.crate from https://crates.io/api/v1/crates/yansi-term/0.1.2/download?dummy=
--->  Verifying checksums for rust
--->  Extracting rust
--->  Applying patches to rust
--->  Configuring rust
--->  Building rust
      [                                        ]  18.3 %

Wait, what? Not only did Rust make it into the dependency chain here, but whatever thing (or things) it's building needs (need) 393 Rust packages to build? Including things like winapi-x86_64-pc-windows-gnu-0 (I am on macOS-arm64) and two different versions of a WebAssembly runtime? What the fuck?!

Rust finishes building 35 minutes later, and now we're on to cargo (the Rust package manager). Progress!

--->  Staging rust into destroot
--->  Installing rust @1.61.0_2
--->  Activating rust @1.61.0_2
--->  Cleaning rust
--->  Computing dependencies for cargo
--->  Fetching archive for cargo
--->  Attempting to fetch cargo-0.62.0_2.darwin_22.arm64.tbz2 from https://packages.macports.org/cargo
--->  Attempting to fetch cargo-0.62.0_2.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/cargo
--->  Attempting to fetch cargo-0.62.0_2.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/cargo
--->  Fetching distfiles for cargo
--->  Attempting to fetch cargo-0.62.0.tar.gz from https://distfiles.macports.org/cargo-crates
--->  Attempting to fetch cargo-1.60.0-aarch64-apple-darwin.tar.gz from https://static.rust-lang.org/dist
--->  Attempting to fetch anyhow-1.0.57.crate from https://crates.io/api/v1/crates/anyhow/1.0.57/download?dummy=
--->  Attempting to fetch arrayvec-0.5.2.crate from https://crates.io/api/v1/crates/arrayvec/0.5.2/download?dummy=
--->  Attempting to fetch bitmaps-2.1.0.crate from https://crates.io/api/v1/crates/bitmaps/2.1.0/download?dummy=
--->  Attempting to fetch bytesize-1.1.0.crate from https://crates.io/api/v1/crates/bytesize/1.1.0/download?dummy=
--->  Attempting to fetch cc-1.0.73.crate from https://crates.io/api/v1/crates/cc/1.0.73/download?dummy=
--->  Attempting to fetch clap-3.1.18.crate from https://crates.io/api/v1/crates/clap/3.1.18/download?dummy=
--->  Attempting to fetch clap_lex-0.2.0.crate from https://crates.io/api/v1/crates/clap_lex/0.2.0/download?dummy=
--->  Attempting to fetch combine-4.6.4.crate from https://crates.io/api/v1/crates/combine/4.6.4/download?dummy=
--->  Attempting to fetch commoncrypto-0.2.0.crate from https://crates.io/api/v1/crates/commoncrypto/0.2.0/download?dummy=
--->  Attempting to fetch commoncrypto-sys-0.2.0.crate from https://crates.io/api/v1/crates/commoncrypto-sys/0.2.0/download?dummy=
--->  Attempting to fetch core-foundation-0.9.3.crate from https://crates.io/api/v1/crates/core-foundation/0.9.3/download?dummy=
--->  Attempting to fetch core-foundation-sys-0.8.3.crate from https://crates.io/api/v1/crates/core-foundation-sys/0.8.3/download?dummy=
--->  Attempting to fetch crc32fast-1.3.2.crate from https://crates.io/api/v1/crates/crc32fast/1.3.2/download?dummy=
--->  Attempting to fetch crossbeam-utils-0.8.8.crate from https://crates.io/api/v1/crates/crossbeam-utils/0.8.8/download?dummy=
--->  Attempting to fetch crypto-hash-0.3.4.crate from https://crates.io/api/v1/crates/crypto-hash/0.3.4/download?dummy=
--->  Attempting to fetch curl-0.4.43.crate from https://crates.io/api/v1/crates/curl/0.4.43/download?dummy=
--->  Attempting to fetch curl-sys-0.4.55+curl-7.83.1.crate from https://crates.io/api/v1/crates/curl-sys/0.4.55+curl-7.83.1/download?dummy=
--->  Attempting to fetch env_logger-0.7.1.crate from https://crates.io/api/v1/crates/env_logger/0.7.1/download?dummy=
--->  Attempting to fetch env_logger-0.9.0.crate from https://crates.io/api/v1/crates/env_logger/0.9.0/download?dummy=
--->  Attempting to fetch fastrand-1.7.0.crate from https://crates.io/api/v1/crates/fastrand/1.7.0/download?dummy=
--->  Attempting to fetch filetime-0.2.16.crate from https://crates.io/api/v1/crates/filetime/0.2.16/download?dummy=
--->  Attempting to fetch flate2-1.0.23.crate from https://crates.io/api/v1/crates/flate2/1.0.23/download?dummy=
--->  Attempting to fetch fnv-1.0.7.crate from https://crates.io/api/v1/crates/fnv/1.0.7/download?dummy=
--->  Attempting to fetch foreign-types-0.3.2.crate from https://crates.io/api/v1/crates/foreign-types/0.3.2/download?dummy=
--->  Attempting to fetch foreign-types-shared-0.1.1.crate from https://crates.io/api/v1/crates/foreign-types-shared/0.1.1/download?dummy=
--->  Attempting to fetch form_urlencoded-1.0.1.crate from https://crates.io/api/v1/crates/form_urlencoded/1.0.1/download?dummy=
--->  Attempting to fetch fwdansi-1.1.0.crate from https://crates.io/api/v1/crates/fwdansi/1.1.0/download?dummy=
--->  Attempting to fetch git2-0.14.2.crate from https://crates.io/api/v1/crates/git2/0.14.2/download?dummy=
--->  Attempting to fetch git2-curl-0.15.0.crate from https://crates.io/api/v1/crates/git2-curl/0.15.0/download?dummy=
--->  Attempting to fetch globset-0.4.8.crate from https://crates.io/api/v1/crates/globset/0.4.8/download?dummy=
--->  Attempting to fetch hashbrown-0.11.2.crate from https://crates.io/api/v1/crates/hashbrown/0.11.2/download?dummy=
--->  Attempting to fetch hex-0.3.2.crate from https://crates.io/api/v1/crates/hex/0.3.2/download?dummy=
--->  Attempting to fetch hex-0.4.3.crate from https://crates.io/api/v1/crates/hex/0.4.3/download?dummy=
--->  Attempting to fetch home-0.5.3.crate from https://crates.io/api/v1/crates/home/0.5.3/download?dummy=
--->  Attempting to fetch humantime-1.3.0.crate from https://crates.io/api/v1/crates/humantime/1.3.0/download?dummy=
--->  Attempting to fetch idna-0.2.3.crate from https://crates.io/api/v1/crates/idna/0.2.3/download?dummy=
--->  Attempting to fetch ignore-0.4.18.crate from https://crates.io/api/v1/crates/ignore/0.4.18/download?dummy=
--->  Attempting to fetch im-rc-15.1.0.crate from https://crates.io/api/v1/crates/im-rc/15.1.0/download?dummy=
--->  Attempting to fetch indexmap-1.8.1.crate from https://crates.io/api/v1/crates/indexmap/1.8.1/download?dummy=
--->  Attempting to fetch itertools-0.10.3.crate from https://crates.io/api/v1/crates/itertools/0.10.3/download?dummy=
--->  Attempting to fetch itoa-1.0.2.crate from https://crates.io/api/v1/crates/itoa/1.0.2/download?dummy=
--->  Attempting to fetch kstring-1.0.6.crate from https://crates.io/api/v1/crates/kstring/1.0.6/download?dummy=
--->  Attempting to fetch libc-0.2.126.crate from https://crates.io/api/v1/crates/libc/0.2.126/download?dummy=
--->  Attempting to fetch libgit2-sys-0.13.2+1.4.2.crate from https://crates.io/api/v1/crates/libgit2-sys/0.13.2+1.4.2/download?dummy=
--->  Attempting to fetch libnghttp2-sys-0.1.7+1.45.0.crate from https://crates.io/api/v1/crates/libnghttp2-sys/0.1.7+1.45.0/download?dummy=
--->  Attempting to fetch libssh2-sys-0.2.23.crate from https://crates.io/api/v1/crates/libssh2-sys/0.2.23/download?dummy=
--->  Attempting to fetch libz-sys-1.1.6.crate from https://crates.io/api/v1/crates/libz-sys/1.1.6/download?dummy=
--->  Attempting to fetch log-0.4.17.crate from https://crates.io/api/v1/crates/log/0.4.17/download?dummy=
--->  Attempting to fetch matches-0.1.9.crate from https://crates.io/api/v1/crates/matches/0.1.9/download?dummy=
--->  Attempting to fetch memchr-2.5.0.crate from https://crates.io/api/v1/crates/memchr/2.5.0/download?dummy=
--->  Attempting to fetch miniz_oxide-0.5.1.crate from https://crates.io/api/v1/crates/miniz_oxide/0.5.1/download?dummy=
--->  Attempting to fetch once_cell-1.11.0.crate from https://crates.io/api/v1/crates/once_cell/1.11.0/download?dummy=
--->  Attempting to fetch opener-0.5.0.crate from https://crates.io/api/v1/crates/opener/0.5.0/download?dummy=
--->  Attempting to fetch openssl-0.10.40.crate from https://crates.io/api/v1/crates/openssl/0.10.40/download?dummy=
--->  Attempting to fetch openssl-macros-0.1.0.crate from https://crates.io/api/v1/crates/openssl-macros/0.1.0/download?dummy=
--->  Attempting to fetch openssl-probe-0.1.5.crate from https://crates.io/api/v1/crates/openssl-probe/0.1.5/download?dummy=
--->  Attempting to fetch openssl-src-111.20.0+1.1.1o.crate from https://crates.io/api/v1/crates/openssl-src/111.20.0+1.1.1o/download?dummy=
--->  Attempting to fetch openssl-sys-0.9.73.crate from https://crates.io/api/v1/crates/openssl-sys/0.9.73/download?dummy=
--->  Attempting to fetch os_info-3.4.0.crate from https://crates.io/api/v1/crates/os_info/3.4.0/download?dummy=
--->  Attempting to fetch os_str_bytes-6.0.1.crate from https://crates.io/api/v1/crates/os_str_bytes/6.0.1/download?dummy=
--->  Attempting to fetch percent-encoding-2.1.0.crate from https://crates.io/api/v1/crates/percent-encoding/2.1.0/download?dummy=
--->  Attempting to fetch pkg-config-0.3.25.crate from https://crates.io/api/v1/crates/pkg-config/0.3.25/download?dummy=
--->  Attempting to fetch pretty_env_logger-0.4.0.crate from https://crates.io/api/v1/crates/pretty_env_logger/0.4.0/download?dummy=
--->  Attempting to fetch proc-macro2-1.0.39.crate from https://crates.io/api/v1/crates/proc-macro2/1.0.39/download?dummy=
--->  Attempting to fetch quick-error-1.2.3.crate from https://crates.io/api/v1/crates/quick-error/1.2.3/download?dummy=
--->  Attempting to fetch rand_xoshiro-0.6.0.crate from https://crates.io/api/v1/crates/rand_xoshiro/0.6.0/download?dummy=
--->  Attempting to fetch regex-1.5.6.crate from https://crates.io/api/v1/crates/regex/1.5.6/download?dummy=
--->  Attempting to fetch regex-syntax-0.6.26.crate from https://crates.io/api/v1/crates/regex-syntax/0.6.26/download?dummy=
--->  Attempting to fetch remove_dir_all-0.5.3.crate from https://crates.io/api/v1/crates/remove_dir_all/0.5.3/download?dummy=
--->  Attempting to fetch rustc-workspace-hack-1.0.0.crate from https://crates.io/api/v1/crates/rustc-workspace-hack/1.0.0/download?dummy=
--->  Attempting to fetch rustfix-0.6.0.crate from https://crates.io/api/v1/crates/rustfix/0.6.0/download?dummy=
--->  Attempting to fetch ryu-1.0.10.crate from https://crates.io/api/v1/crates/ryu/1.0.10/download?dummy=
--->  Attempting to fetch schannel-0.1.20.crate from https://crates.io/api/v1/crates/schannel/0.1.20/download?dummy=
--->  Attempting to fetch semver-1.0.9.crate from https://crates.io/api/v1/crates/semver/1.0.9/download?dummy=
--->  Attempting to fetch serde-1.0.137.crate from https://crates.io/api/v1/crates/serde/1.0.137/download?dummy=
--->  Attempting to fetch serde_derive-1.0.137.crate from https://crates.io/api/v1/crates/serde_derive/1.0.137/download?dummy=
--->  Attempting to fetch serde_ignored-0.1.3.crate from https://crates.io/api/v1/crates/serde_ignored/0.1.3/download?dummy=
--->  Attempting to fetch serde_json-1.0.81.crate from https://crates.io/api/v1/crates/serde_json/1.0.81/download?dummy=
--->  Attempting to fetch shell-escape-0.1.5.crate from https://crates.io/api/v1/crates/shell-escape/0.1.5/download?dummy=
--->  Attempting to fetch sized-chunks-0.6.5.crate from https://crates.io/api/v1/crates/sized-chunks/0.6.5/download?dummy=
--->  Attempting to fetch socket2-0.4.4.crate from https://crates.io/api/v1/crates/socket2/0.4.4/download?dummy=
--->  Attempting to fetch strip-ansi-escapes-0.1.1.crate from https://crates.io/api/v1/crates/strip-ansi-escapes/0.1.1/download?dummy=
--->  Attempting to fetch strsim-0.10.0.crate from https://crates.io/api/v1/crates/strsim/0.10.0/download?dummy=
--->  Attempting to fetch syn-1.0.95.crate from https://crates.io/api/v1/crates/syn/1.0.95/download?dummy=
--->  Attempting to fetch tar-0.4.38.crate from https://crates.io/api/v1/crates/tar/0.4.38/download?dummy=
--->  Attempting to fetch tempfile-3.3.0.crate from https://crates.io/api/v1/crates/tempfile/3.3.0/download?dummy=
--->  Attempting to fetch termcolor-1.1.3.crate from https://crates.io/api/v1/crates/termcolor/1.1.3/download?dummy=
--->  Attempting to fetch textwrap-0.15.0.crate from https://crates.io/api/v1/crates/textwrap/0.15.0/download?dummy=
--->  Attempting to fetch thread_local-1.1.4.crate from https://crates.io/api/v1/crates/thread_local/1.1.4/download?dummy=
--->  Attempting to fetch tinyvec-1.6.0.crate from https://crates.io/api/v1/crates/tinyvec/1.6.0/download?dummy=
--->  Attempting to fetch tinyvec_macros-0.1.0.crate from https://crates.io/api/v1/crates/tinyvec_macros/0.1.0/download?dummy=
--->  Attempting to fetch toml_edit-0.13.4.crate from https://crates.io/api/v1/crates/toml_edit/0.13.4/download?dummy=
--->  Attempting to fetch typenum-1.15.0.crate from https://crates.io/api/v1/crates/typenum/1.15.0/download?dummy=
--->  Attempting to fetch unicode-bidi-0.3.8.crate from https://crates.io/api/v1/crates/unicode-bidi/0.3.8/download?dummy=
--->  Attempting to fetch unicode-ident-1.0.0.crate from https://crates.io/api/v1/crates/unicode-ident/1.0.0/download?dummy=
--->  Attempting to fetch unicode-normalization-0.1.19.crate from https://crates.io/api/v1/crates/unicode-normalization/0.1.19/download?dummy=
--->  Attempting to fetch unicode-xid-0.2.3.crate from https://crates.io/api/v1/crates/unicode-xid/0.2.3/download?dummy=
--->  Attempting to fetch url-2.2.2.crate from https://crates.io/api/v1/crates/url/2.2.2/download?dummy=
--->  Attempting to fetch utf8parse-0.2.0.crate from https://crates.io/api/v1/crates/utf8parse/0.2.0/download?dummy=
--->  Attempting to fetch vcpkg-0.2.15.crate from https://crates.io/api/v1/crates/vcpkg/0.2.15/download?dummy=
--->  Attempting to fetch vte-0.10.1.crate from https://crates.io/api/v1/crates/vte/0.10.1/download?dummy=
--->  Attempting to fetch vte_generate_state_changes-0.1.1.crate from https://crates.io/api/v1/crates/vte_generate_state_changes/0.1.1/download?dummy=
--->  Attempting to fetch windows-sys-0.36.1.crate from https://crates.io/api/v1/crates/windows-sys/0.36.1/download?dummy=
--->  Attempting to fetch windows_aarch64_msvc-0.36.1.crate from https://crates.io/api/v1/crates/windows_aarch64_msvc/0.36.1/download?dummy=
--->  Attempting to fetch windows_i686_gnu-0.36.1.crate from https://crates.io/api/v1/crates/windows_i686_gnu/0.36.1/download?dummy=
--->  Attempting to fetch windows_i686_msvc-0.36.1.crate from https://crates.io/api/v1/crates/windows_i686_msvc/0.36.1/download?dummy=
--->  Attempting to fetch windows_x86_64_gnu-0.36.1.crate from https://crates.io/api/v1/crates/windows_x86_64_gnu/0.36.1/download?dummy=
--->  Attempting to fetch windows_x86_64_msvc-0.36.1.crate from https://crates.io/api/v1/crates/windows_x86_64_msvc/0.36.1/download?dummy=
--->  Verifying checksums for cargo
--->  Extracting cargo
--->  Configuring cargo
--->  Building cargo
      [  ...  ]

Thankfully, cargo is relatively faster to build, and now we're ready to make progress on other things. Shout-out to the other Windows packages that were just brought in. Now we also get a hint into what exactly brought Rust in:

--->  Staging cargo into destroot
--->  Installing cargo @0.62.0_2
--->  Activating cargo @0.62.0_2
--->  Cleaning cargo
--->  Computing dependencies for py310-setuptools-rust
--->  Fetching archive for py310-setuptools-rust
--->  Attempting to fetch py310-setuptools-rust-1.5.2_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-setuptools-rust
--->  Attempting to fetch py310-setuptools-rust-1.5.2_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-setuptools-rust
--->  Installing py310-setuptools-rust @1.5.2_0
--->  Activating py310-setuptools-rust @1.5.2_0
--->  Cleaning py310-setuptools-rust
--->  Computing dependencies for py310-packaging
--->  Fetching archive for py310-packaging
--->  Attempting to fetch py310-packaging-22.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-packaging
--->  Attempting to fetch py310-packaging-22.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-packaging
--->  Installing py310-packaging @22.0_0
--->  Cleaning py310-packaging
--->  Computing dependencies for py310-packaging
--->  Deactivating py310-packaging @21.3_0
--->  Cleaning py310-packaging
--->  Activating py310-packaging @22.0_0
--->  Cleaning py310-packaging
--->  Computing dependencies for py310-pep517
--->  Fetching archive for py310-pep517
--->  Attempting to fetch py310-pep517-0.13.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-pep517
--->  Attempting to fetch py310-pep517-0.13.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-pep517
--->  Installing py310-pep517 @0.13.0_0
--->  Activating py310-pep517 @0.13.0_0
--->  Cleaning py310-pep517
--->  Computing dependencies for py310-build
--->  Fetching archive for py310-build
--->  Attempting to fetch py310-build-0.8.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-build
--->  Attempting to fetch py310-build-0.8.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-build
--->  Installing py310-build @0.8.0_0
--->  Activating py310-build @0.8.0_0
--->  Cleaning py310-build
--->  Computing dependencies for py310-installer
--->  Fetching archive for py310-installer
--->  Attempting to fetch py310-installer-0.6.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-installer
--->  Attempting to fetch py310-installer-0.6.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-installer
--->  Installing py310-installer @0.6.0_0
--->  Activating py310-installer @0.6.0_0
--->  Cleaning py310-installer
--->  Computing dependencies for py310-wheel
--->  Fetching archive for py310-wheel
--->  Attempting to fetch py310-wheel-0.38.4_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-wheel
--->  Attempting to fetch py310-wheel-0.38.4_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-wheel
--->  Installing py310-wheel @0.38.4_0
--->  Activating py310-wheel @0.38.4_0
--->  Cleaning py310-wheel
--->  Computing dependencies for py310-pycparser
--->  Fetching archive for py310-pycparser
--->  Attempting to fetch py310-pycparser-2.21_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-pycparser
--->  Attempting to fetch py310-pycparser-2.21_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-pycparser
--->  Installing py310-pycparser @2.21_0
--->  Activating py310-pycparser @2.21_0
--->  Cleaning py310-pycparser
--->  Computing dependencies for py310-cffi
--->  Fetching archive for py310-cffi
--->  Attempting to fetch py310-cffi-1.15.1_0.darwin_22.arm64.tbz2 from https://packages.macports.org/py310-cffi
--->  Attempting to fetch py310-cffi-1.15.1_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/py310-cffi
--->  Attempting to fetch py310-cffi-1.15.1_0.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/py310-cffi
--->  Fetching distfiles for py310-cffi
--->  Attempting to fetch cffi-1.15.1.tar.gz from https://distfiles.macports.org/py-cffi
--->  Verifying checksums for py310-cffi
--->  Extracting py310-cffi
--->  Applying patches to py310-cffi
--->  Configuring py310-cffi
--->  Building py310-cffi
--->  Staging py310-cffi into destroot
--->  Installing py310-cffi @1.15.1_0
--->  Activating py310-cffi @1.15.1_0
--->  Cleaning py310-cffi
--->  Computing dependencies for py310-cryptography
--->  Fetching archive for py310-cryptography
--->  Attempting to fetch py310-cryptography-38.0.3_0.darwin_22.arm64.tbz2 from https://packages.macports.org/py310-cryptography
--->  Attempting to fetch py310-cryptography-38.0.3_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/py310-cryptography
--->  Attempting to fetch py310-cryptography-38.0.3_0.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/py310-cryptography
--->  Fetching distfiles for py310-cryptography
--->  Attempting to fetch cryptography-38.0.3.tar.gz from https://distfiles.macports.org/cargo-crates
--->  Attempting to fetch Inflector-0.11.4.crate from https://crates.io/api/v1/crates/Inflector/0.11.4/download?dummy=
--->  Attempting to fetch aliasable-0.1.3.crate from https://crates.io/api/v1/crates/aliasable/0.1.3/download?dummy=
--->  Attempting to fetch android_system_properties-0.1.5.crate from https://crates.io/api/v1/crates/android_system_properties/0.1.5/download?dummy=
--->  Attempting to fetch asn1-0.12.2.crate from https://crates.io/api/v1/crates/asn1/0.12.2/download?dummy=
--->  Attempting to fetch asn1_derive-0.12.2.crate from https://crates.io/api/v1/crates/asn1_derive/0.12.2/download?dummy=
--->  Attempting to fetch base64-0.13.0.crate from https://crates.io/api/v1/crates/base64/0.13.0/download?dummy=
--->  Attempting to fetch bumpalo-3.10.0.crate from https://crates.io/api/v1/crates/bumpalo/3.10.0/download?dummy=
--->  Attempting to fetch chrono-0.4.22.crate from https://crates.io/api/v1/crates/chrono/0.4.22/download?dummy=
--->  Attempting to fetch iana-time-zone-0.1.47.crate from https://crates.io/api/v1/crates/iana-time-zone/0.1.47/download?dummy=
--->  Attempting to fetch indoc-0.3.6.crate from https://crates.io/api/v1/crates/indoc/0.3.6/download?dummy=
--->  Attempting to fetch indoc-impl-0.3.6.crate from https://crates.io/api/v1/crates/indoc-impl/0.3.6/download?dummy=
--->  Attempting to fetch js-sys-0.3.59.crate from https://crates.io/api/v1/crates/js-sys/0.3.59/download?dummy=
--->  Attempting to fetch libc-0.2.132.crate from https://crates.io/api/v1/crates/libc/0.2.132/download?dummy=
--->  Attempting to fetch lock_api-0.4.8.crate from https://crates.io/api/v1/crates/lock_api/0.4.8/download?dummy=
--->  Attempting to fetch num-integer-0.1.45.crate from https://crates.io/api/v1/crates/num-integer/0.1.45/download?dummy=
--->  Attempting to fetch num-traits-0.2.15.crate from https://crates.io/api/v1/crates/num-traits/0.2.15/download?dummy=
--->  Attempting to fetch once_cell-1.14.0.crate from https://crates.io/api/v1/crates/once_cell/1.14.0/download?dummy=
--->  Attempting to fetch ouroboros-0.15.4.crate from https://crates.io/api/v1/crates/ouroboros/0.15.4/download?dummy=
--->  Attempting to fetch ouroboros_macro-0.15.4.crate from https://crates.io/api/v1/crates/ouroboros_macro/0.15.4/download?dummy=
--->  Attempting to fetch parking_lot-0.11.2.crate from https://crates.io/api/v1/crates/parking_lot/0.11.2/download?dummy=
--->  Attempting to fetch parking_lot_core-0.8.5.crate from https://crates.io/api/v1/crates/parking_lot_core/0.8.5/download?dummy=
--->  Attempting to fetch paste-0.1.18.crate from https://crates.io/api/v1/crates/paste/0.1.18/download?dummy=
--->  Attempting to fetch paste-impl-0.1.18.crate from https://crates.io/api/v1/crates/paste-impl/0.1.18/download?dummy=
--->  Attempting to fetch pem-1.1.0.crate from https://crates.io/api/v1/crates/pem/1.1.0/download?dummy=
--->  Attempting to fetch proc-macro-hack-0.5.19.crate from https://crates.io/api/v1/crates/proc-macro-hack/0.5.19/download?dummy=
--->  Attempting to fetch proc-macro2-1.0.43.crate from https://crates.io/api/v1/crates/proc-macro2/1.0.43/download?dummy=
--->  Attempting to fetch pyo3-0.15.2.crate from https://crates.io/api/v1/crates/pyo3/0.15.2/download?dummy=
--->  Attempting to fetch pyo3-build-config-0.15.2.crate from https://crates.io/api/v1/crates/pyo3-build-config/0.15.2/download?dummy=
--->  Attempting to fetch pyo3-macros-0.15.2.crate from https://crates.io/api/v1/crates/pyo3-macros/0.15.2/download?dummy=
--->  Attempting to fetch pyo3-macros-backend-0.15.2.crate from https://crates.io/api/v1/crates/pyo3-macros-backend/0.15.2/download?dummy=
--->  Attempting to fetch quote-1.0.21.crate from https://crates.io/api/v1/crates/quote/1.0.21/download?dummy=
--->  Attempting to fetch redox_syscall-0.2.16.crate from https://crates.io/api/v1/crates/redox_syscall/0.2.16/download?dummy=
--->  Attempting to fetch smallvec-1.9.0.crate from https://crates.io/api/v1/crates/smallvec/1.9.0/download?dummy=
--->  Attempting to fetch syn-1.0.99.crate from https://crates.io/api/v1/crates/syn/1.0.99/download?dummy=
--->  Attempting to fetch unicode-ident-1.0.3.crate from https://crates.io/api/v1/crates/unicode-ident/1.0.3/download?dummy=
--->  Attempting to fetch unindent-0.1.10.crate from https://crates.io/api/v1/crates/unindent/0.1.10/download?dummy=
--->  Attempting to fetch wasm-bindgen-0.2.82.crate from https://crates.io/api/v1/crates/wasm-bindgen/0.2.82/download?dummy=
--->  Attempting to fetch wasm-bindgen-backend-0.2.82.crate from https://crates.io/api/v1/crates/wasm-bindgen-backend/0.2.82/download?dummy=
--->  Attempting to fetch wasm-bindgen-macro-0.2.82.crate from https://crates.io/api/v1/crates/wasm-bindgen-macro/0.2.82/download?dummy=
--->  Attempting to fetch wasm-bindgen-macro-support-0.2.82.crate from https://crates.io/api/v1/crates/wasm-bindgen-macro-support/0.2.82/download?dummy=
--->  Attempting to fetch wasm-bindgen-shared-0.2.82.crate from https://crates.io/api/v1/crates/wasm-bindgen-shared/0.2.82/download?dummy=
--->  Verifying checksums for py310-cryptography
--->  Extracting py310-cryptography
--->  Configuring py310-cryptography
--->  Building py310-cryptography
--->  Staging py310-cryptography into destroot
--->  Installing py310-cryptography @38.0.3_0
--->  Activating py310-cryptography @38.0.3_0
--->  Cleaning py310-cryptography
--->  Computing dependencies for py310-fido2
--->  Fetching archive for py310-fido2
--->  Attempting to fetch py310-fido2-1.1.0_0.darwin_22.arm64.tbz2 from https://packages.macports.org/py310-fido2
--->  Attempting to fetch py310-fido2-1.1.0_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/py310-fido2
--->  Attempting to fetch py310-fido2-1.1.0_0.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/py310-fido2
--->  Fetching distfiles for py310-fido2
--->  Attempting to fetch fido2-1.1.0.tar.gz from https://distfiles.macports.org/py-fido2
--->  Verifying checksums for py310-fido2
--->  Extracting py310-fido2
--->  Configuring py310-fido2
--->  Building py310-fido2
--->  Staging py310-fido2 into destroot
--->  Installing py310-fido2 @1.1.0_0
--->  Activating py310-fido2 @1.1.0_0
--->  Cleaning py310-fido2
--->  Computing dependencies for py310-openssl
--->  Fetching archive for py310-openssl
--->  Attempting to fetch py310-openssl-22.1.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-openssl
--->  Attempting to fetch py310-openssl-22.1.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-openssl
--->  Installing py310-openssl @22.1.0_0
--->  Activating py310-openssl @22.1.0_0
--->  Cleaning py310-openssl
--->  Computing dependencies for gsed
--->  Fetching archive for gsed
--->  Attempting to fetch gsed-4.9_0.darwin_22.arm64.tbz2 from https://packages.macports.org/gsed
--->  Attempting to fetch gsed-4.9_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/gsed
--->  Attempting to fetch gsed-4.9_0.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/gsed
--->  Fetching distfiles for gsed
--->  Attempting to fetch sed-4.9.tar.xz from https://distfiles.macports.org/gsed
--->  Verifying checksums for gsed
--->  Extracting gsed
--->  Configuring gsed
Warning: Configuration logfiles contain indications of -Wimplicit-function-declaration; check that features were not accidentally disabled:
  alignof: found in sed-4.9/config.log
  re_search: found in sed-4.9/config.log
  re_compile_pattern: found in sed-4.9/config.log
  re_set_syntax: found in sed-4.9/config.log
  MIN: found in sed-4.9/config.log
  __fpending: found in sed-4.9/config.log
  strchr: found in sed-4.9/config.log
  free: found in sed-4.9/config.log
--->  Building gsed
--->  Staging gsed into destroot
--->  Installing gsed @4.9_0
--->  Activating gsed @4.9_0
--->  Cleaning gsed
--->  Computing dependencies for swig
--->  Fetching archive for swig
--->  Attempting to fetch swig-4.1.1_0.darwin_22.arm64.tbz2 from https://packages.macports.org/swig
--->  Attempting to fetch swig-4.1.1_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/swig
--->  Attempting to fetch swig-4.1.1_0.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/swig
--->  Fetching distfiles for swig
--->  Attempting to fetch swig-4.1.1.tar.gz from https://distfiles.macports.org/swig
--->  Verifying checksums for swig
--->  Extracting swig
--->  Configuring swig
Warning: Configuration logfiles contain indications of -Wimplicit-function-declaration; check that features were not accidentally disabled:
  snprintf: found in swig-4.1.1/CCache/config.log
  strcmp: found in swig-4.1.1/CCache/config.log
  exit: found in swig-4.1.1/CCache/config.log
  vsnprintf: found in swig-4.1.1/CCache/config.log
--->  Building swig
--->  Staging swig into destroot
--->  Installing swig @4.1.1_0
--->  Cleaning swig
--->  Computing dependencies for swig
--->  Deactivating swig @4.0.2_2
--->  Cleaning swig
--->  Activating swig @4.1.1_0
--->  Cleaning swig
--->  Computing dependencies for swig-python
--->  Fetching archive for swig-python
--->  Attempting to fetch swig-python-4.1.1_0.darwin_22.arm64.tbz2 from https://packages.macports.org/swig-python
--->  Attempting to fetch swig-python-4.1.1_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/swig-python
--->  Attempting to fetch swig-python-4.1.1_0.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/swig-python
--->  Fetching distfiles for swig-python
--->  Verifying checksums for swig-python
--->  Extracting swig-python
--->  Configuring swig-python
Warning: Configuration logfiles contain indications of -Wimplicit-function-declaration; check that features were not accidentally disabled:
  snprintf: found in swig-4.1.1/CCache/config.log
  strcmp: found in swig-4.1.1/CCache/config.log
  exit: found in swig-4.1.1/CCache/config.log
  vsnprintf: found in swig-4.1.1/CCache/config.log
--->  Building swig-python
--->  Staging swig-python into destroot
--->  Installing swig-python @4.1.1_0
--->  Cleaning swig-python
--->  Computing dependencies for swig-python
--->  Deactivating swig-python @4.0.2_2
--->  Cleaning swig-python
--->  Activating swig-python @4.1.1_0
--->  Cleaning swig-python
--->  Computing dependencies for py310-pyscard
--->  Fetching archive for py310-pyscard
--->  Attempting to fetch py310-pyscard-2.0.5_0.darwin_22.arm64.tbz2 from https://packages.macports.org/py310-pyscard
--->  Attempting to fetch py310-pyscard-2.0.5_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/py310-pyscard
--->  Attempting to fetch py310-pyscard-2.0.5_0.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/py310-pyscard
--->  Fetching distfiles for py310-pyscard
--->  Attempting to fetch pyscard-2.0.5.tar.gz from https://distfiles.macports.org/py-pyscard
--->  Verifying checksums for py310-pyscard
--->  Extracting py310-pyscard
--->  Configuring py310-pyscard
--->  Building py310-pyscard
--->  Staging py310-pyscard into destroot
--->  Installing py310-pyscard @2.0.5_0
--->  Activating py310-pyscard @2.0.5_0
--->  Cleaning py310-pyscard
--->  Computing dependencies for py310-zipp
--->  Fetching archive for py310-zipp
--->  Attempting to fetch py310-zipp-3.11.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-zipp
--->  Attempting to fetch py310-zipp-3.11.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-zipp
--->  Installing py310-zipp @3.11.0_0
--->  Activating py310-zipp @3.11.0_0
--->  Cleaning py310-zipp
--->  Computing dependencies for py310-importlib-metadata
--->  Fetching archive for py310-importlib-metadata
--->  Attempting to fetch py310-importlib-metadata-5.1.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-importlib-metadata
--->  Attempting to fetch py310-importlib-metadata-5.1.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-importlib-metadata
--->  Installing py310-importlib-metadata @5.1.0_0
--->  Activating py310-importlib-metadata @5.1.0_0
--->  Cleaning py310-importlib-metadata
--->  Computing dependencies for py310-more-itertools
--->  Fetching archive for py310-more-itertools
--->  Attempting to fetch py310-more-itertools-9.0.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-more-itertools
--->  Attempting to fetch py310-more-itertools-9.0.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-more-itertools
--->  Installing py310-more-itertools @9.0.0_0
--->  Cleaning py310-more-itertools
--->  Computing dependencies for py310-more-itertools
--->  Deactivating py310-more-itertools @8.14.0_0
--->  Cleaning py310-more-itertools
--->  Activating py310-more-itertools @9.0.0_0
--->  Cleaning py310-more-itertools
--->  Computing dependencies for py310-jaraco.classes
--->  Fetching archive for py310-jaraco.classes
--->  Attempting to fetch py310-jaraco.classes-3.2.3_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-jaraco.classes
--->  Attempting to fetch py310-jaraco.classes-3.2.3_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-jaraco.classes
--->  Installing py310-jaraco.classes @3.2.3_0
--->  Activating py310-jaraco.classes @3.2.3_0
--->  Cleaning py310-jaraco.classes
--->  Computing dependencies for py310-keyring
--->  Fetching archive for py310-keyring
--->  Attempting to fetch py310-keyring-23.11.0_0.darwin_any.noarch.tbz2 from https://packages.macports.org/py310-keyring
--->  Attempting to fetch py310-keyring-23.11.0_0.darwin_any.noarch.tbz2.rmd160 from https://packages.macports.org/py310-keyring
--->  Installing py310-keyring @23.11.0_0
--->  Activating py310-keyring @23.11.0_0
--->  Cleaning py310-keyring
--->  Computing dependencies for yubikey-manager
--->  Fetching archive for yubikey-manager
--->  Attempting to fetch yubikey-manager-5.0.0_0.darwin_22.arm64.tbz2 from https://packages.macports.org/yubikey-manager
--->  Attempting to fetch yubikey-manager-5.0.0_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/yubikey-manager
--->  Attempting to fetch yubikey-manager-5.0.0_0.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/yubikey-manager
--->  Fetching distfiles for yubikey-manager
--->  Attempting to fetch yubikey_manager-5.0.0.tar.gz from https://distfiles.macports.org/yubikey-manager
--->  Verifying checksums for yubikey-manager
--->  Extracting yubikey-manager
--->  Configuring yubikey-manager
--->  Building yubikey-manager
--->  Staging yubikey-manager into destroot
--->  Installing yubikey-manager @5.0.0_0
--->  Cleaning yubikey-manager
--->  Computing dependencies for yubikey-manager
--->  Deactivating yubikey-manager @3.1.2_0
--->  Cleaning yubikey-manager
--->  Activating yubikey-manager @5.0.0_0
--->  Cleaning yubikey-manager

Several minutes later, and about an hour from when I ran that original command, we're done installing everything. Except, one of the things I tried to upgrade failed:

--->  Fetching archive for ykpers
--->  Attempting to fetch ykpers-1.20.0_1.darwin_22.arm64.tbz2 from https://packages.macports.org/ykpers
--->  Attempting to fetch ykpers-1.20.0_1.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/ykpers
--->  Attempting to fetch ykpers-1.20.0_1.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/ykpers
--->  Computing dependencies for ykpers
--->  Fetching distfiles for ykpers
--->  Attempting to fetch ykpers-1.20.0.tar.gz from https://distfiles.macports.org/ykpers
--->  Verifying checksums for ykpers
--->  Extracting ykpers
--->  Applying patches to ykpers
--->  Configuring ykpers
--->  Building ykpers
--->  Staging ykpers into destroot
--->  Unable to uninstall ykpers @1.20.0_1, the following ports depend on it:
--->    yubikey-manager @3.1.2_0
Warning: Uninstall forced.  Proceeding despite dependencies.
--->  Deactivating ykpers @1.20.0_1
--->  Cleaning ykpers
--->  Uninstalling ykpers @1.20.0_1
--->  Cleaning ykpers
--->  Computing dependencies for ykpers
--->  Fetching archive for ykpers
--->  Attempting to fetch ykpers-1.20.0_1.darwin_22.arm64.tbz2 from https://packages.macports.org/ykpers
--->  Attempting to fetch ykpers-1.20.0_1.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/ykpers
--->  Attempting to fetch ykpers-1.20.0_1.darwin_22.arm64.tbz2 from https://mse.uk.packages.macports.org/ykpers
--->  Installing ykpers @1.20.0_1
--->  Activating ykpers @1.20.0_1
--->  Cleaning ykpers
--->  Updating database of binaries
--->  Scanning binaries for linking errors

You win some, you lose some. At least now we get to find out exactly why we needed Rust:

$ port rdependents rust
The following ports are dependent on rust:
  cargo
    py310-setuptools-rust

The graph ends unexpectedly. Probably some bug in MacPorts that I'm going to ignore for now. Based on the earlier log, I can see py310-cryptography depends on py310-setuptools-rust¹, and py310-cryptography is, in turn, depended on by yubikey-manager. I also notice it brings in several wasm-bindgen-* packages, for what I'm sure are totally reasonable reasons².

$ port info py310-cryptography
py310-cryptography @38.0.3 (python, devel)
...
Build Dependencies:   diffutils-for-muniversal, py310-setuptools-rust, py310-build, py310-installer, py310-setuptools, py310-wheel, rust, cargo

There's no moral to this story, except maybe avoid upgrading your software. I think Rust is Good, but this sucks.

What we've got here is a cryptography package that transitively depends on at least 393 other packages, every single one of which is a potential attack vector. Is this fine? I don't know. Maybe? ↩
Vaguely, I remember something about some Rust packages that use Wasm to perform macro-expansion safely during builds, or something along those lines, so these may in fact be reasonable uses of these Wasm packages, by some definition of reasonable. It feels like extreme bloat to me, though. ↩

Announcing racket-{avro,iso-printf,lz4,messagepack}

Mon, 5 Dec 2022 10:15:00 +0200

Some of the feedback I've received on Franz so far has been that folks need support for more compression and serialization formats. In that vein, here are some Racket libraries I've released in the past couple of weeks:

racket-avro (docs, src) -- an implementation of the Apache Avro serialization protocol. This is fairly complete, but it elides support for Avro RPC for now.
racket-iso-printf (docs, src) -- an implementation of the standard C family of printf functions. This is used in racket-lua to implement the string.format procedure. Originally, this was going to just be an internal part of #lang lua, but I figured it might have some use beyond it. I've certainly wanted C-style printf in the past in Racket.
racket-lz4 (docs, src) -- a pure-Racket LZ4 decompressor. It doesn't yet support compression, but I may add it if there is interest.
racket-messagepack (docs, src) -- an implementation of the MessagePack serialization format. There is an existing msgpack package on the Package Server, but it is GPL-licensed, so I wanted to avoid distributing it with my app.

Announcing Franz

Sun, 20 Nov 2022 20:55:00 +0200

I've been using Apache Kafka for about a year now and I've wanted a desktop client from the beginning. I couldn't find one I liked -- the best I've found is Confluent's Cloud UI, but that's neither a desktop app, nor is it a great experience¹ -- so, a couple months ago I started building a native macOS GUI for Kafka in my spare time and, today, I'm proud to announce the first beta release of Franz.

This release covers all the basic functionality you might expect: you can manage topics and consumer groups, publish & consume records, and even hook in and script part of the consumption process. So, if that sounds appealing to you, please give it a try and let me know what you think. I've had tons of fun working on this so far and I'd love to make it even better.

In terms of tech, obviously, the sane approach would've been to make a straight Swift app and embed librdkafka and I'd be off to the races. But, that wouldn't be very fun. So, instead, Franz is a Swift app backed by Racket (via the same approach described in Screencast: SwiftUI + Racket, though with many improvements to the process since I recorded that video -- expect an updated screencast sometime soon!), where the underlying Kafka client is the same one I've been working on off-and-on since the beginning of the year. The aforementioned scripting support is provided by racket-lua, which I announced last week.

I've found this approach² to building apps to be hugely productive (and damn fun!). I get to alternate between quickly bashing out core functionality in Racket, interacting with said code at the racket-mode REPL, and switching to Swift where I can bang out whatever GUI idea comes to mind pretty quickly at this point. It all makes for some very productive moments, even when those moments are relatively few and far-between (I have a dayjob so this was all done during my free time and mostly during weekends).

Unless you love being unconditionally logged out once an hour for some reason. ↩
I've used a similar approach to ship an app on the Mac App Store a couple years ago. ↩

#lang lua

Sat, 12 Nov 2022 20:00:00 +0200

I'm currently working on a macOS app that's built with Racket and allows the user to write small scripts to process and filter some data. While Racket is definitely my preferred language and I could easily use it for these scripts, my target audience for this app would probably appreciate a more familiar language. I decided to use Lua. So, last weekend I was faced with a choice¹ between writing FFI bindings for Lua or implementing a #lang in Racket. Of course, I picked the harder option. Partly because it seemed like the more fun option, and partly because this way I don't have to worry about making two (three? did I mention Swift's also in the mix?) very different runtimes cooperate.

It's pretty far along at this point, though some Lua standard library functionality is still missing and a couple things are still missing lexer support (long brackets, scientific notation for numbers). If you want to play around with it, you can get it from the package server by installing lua-lang or lua-lib if you don't want the docs. You can find the code on GitHub and the docs on the package server.

Before I started working on the #lang, I had checked the documentation server to see if there was already an implementation. I didn't find anything. Unfortunately, I forgot to check the package server until I was ready to upload my own library. Long story short, there are now (at least) two² independent implementations of Lua as a #lang for Racket. Whoops. ↩
The other one you can find here. ↩

Screencast: SwiftUI + Racket

Sun, 21 Aug 2022 19:45:00 +0300

I've been playing with embedding Racket CS in desktop apps off and on for a while and today I recorded a little screencast demoing some of the stuff I've been working on. Here it is on YouTube:

This is all pretty experimental and I'm just playing around, so nothing is particularly stable, but if this stuff interests you, check out Noise and NoiseBackendExample.

Announcing racket-crontab

Sat, 20 Aug 2022 12:00:00 +0300

Earlier this week, Jesse Alama brought up the topic of scheduling cron jobs with koyo and we both agreed that it would be nice if koyo had built-in support for that sort of thing. So, I wrote crontab, a little library for parsing cron-style schedules and executing code based on them. On top of that functionality, I've added a new koyo/crontab module to koyo that integrates with the component system.

Announcing racket-kafka

Sat, 12 Mar 2022 10:00:00 +0200

For the past month or so, I've been working on implementing a pure-Racket client for Apache Kafka. Yesterday, it reached a point where it can do the bare minimum you would expect from it: produce data and join consumer groups to consume data. Kafka has a fairly large feature-set so there's a ton left to do, but I figure this is a good time to announce the library and get feedback. If you're interested in using Kafka with Racket, please give it a try and let me know what you think.

You can find the source code on GitHub and the docs on the Racket Package server.

PS: While writing the library, I needed an easy way to write lots of little wire procotol parsers, so I stacked my yaks and wrote a binary format parser generator called binfmt (docs).

Parallelizing the Racket Web Server

Thu, 30 Dec 2021 13:45:00 +0200

Racket provides support for concurrency via lightweight threads, which the web server leverages to handle requests, spawning one such thread per incoming request. At the runtime level, these threads run concurrently but not in parallel (i.e., only one thread is active at any one time). Parallelism is available in Racket via Places: distinct instances of the Racket runtime running in separate OS threads that communicate via message passing.

The web server doesn't do anything with places, so, by default, all Racket web servers run in a single OS thread. That isn't a big deal since you can run one web server process per core and place a reverse proxy like Nginx in front to load balance between the processes. But what if you don't want to do that? Is there a way to use the web server in conjunction with places despite the web server lacking explicit support for them?

The answer is "yes." Otherwise, I wouldn't be writing this post! Doing so can lead to a decent reduction in memory usage over the multi-process approach since some resources (such as code, shared libraries, allocation segments, etc.) are shared between places.

One approach to solving this problem might be to spawn multiple places, each running a web server bound to the same port. Unfortunately, it's not possible in Racket to re-use TCP ports (primarily because not all platforms have an equivalent of Linux's SO_REUSEPORT flag). Thankfully, the web server's serve function takes an optional tcp@ argument. We can leverage that argument to provide the server with a custom implementation of the tcp^ signature. So, our main place can spawn one place for every parallel web server that we want to run, then run a TCP server of its own, accept new connections on that server, and send each connection to the web server places one by one.

Take this minimal application -- saved on my machine as app.rkt -- for example:

#lang racket/base

(require web-server/dispatch
         web-server/http
         web-server/servlet-dispatch
         web-server/web-server)

(provide
 start)

(define-values (app _)
  (dispatch-rules
   [("")
    (λ (_req)
      (response/output
       (λ (out)
         (displayln "hello, world" out))))]
   [else
    (λ (_req)
      (response/output
       #:code 404
       (λ (out)
         (displayln "not found" out))))]))

(define (start host port)
  (serve
   #:dispatch (dispatch/servlet app)
   #:listen-ip host
   #:port port))

(module+ main
  (define stop (start "127.0.0.1" 8000))
  (with-handlers ([exn:break? (λ (_)
                                (stop))])
    (sync never-evt)))

Without modifying app.rkt, we can create a second module, called main.rkt, that spawns multiple instances of the server, each bound to different ports:

#lang racket/base

(require racket/match
         racket/place
         "app.rkt")

(define (start-place)
  (place ch
    (let loop ([stop void])
      (match (sync ch)
        [`(init ,host ,port)
         (loop (start host port))]
        [`(stop)
         (stop)]))))

(module+ main
  (define places
    (for/list ([idx (in-range 4)])
      (define pch (start-place))
      (begin0 pch
        (place-channel-put pch `(init "127.0.0.1" ,(+ 8000 idx))))))

  (with-handlers ([exn:break? (λ (_)
                                (for ([pch (in-list places)])
                                  (place-channel-put pch '(stop)))
                                (for-each place-wait places))])
    (sync never-evt)))

Next, we can define our custom tcp@ unit in main.rkt:

  #lang racket/base

- (require racket/match
+ (require net/tcp-sig
+          racket/match
           racket/place
+          (prefix-in tcp: racket/tcp)
+          racket/unit
           "app.rkt")

+ (struct place-tcp-listener ())
+
+ (define (make-place-tcp@ accept-ch)
+   (unit
+     (import)
+     (export tcp^)
+
+     (define (tcp-addresses _p [port-numbers? #f])
+       (if port-numbers?
+           (values "127.0.0.1" 1 "127.0.0.1" 0)
+           (values "127.0.0.1" "127.0.0.1")))
+
+     (define (tcp-connect _hostname
+                          _port-no
+                          [_local-hostname #f]
+                          [_local-port-no #f])
+       (error 'tcp-connect "not supported"))
+
+     (define (tcp-connect/enable-break _hostname
+                                       _port-no
+                                       [_local-hostname #f]
+                                       [_local-port-no #f])
+       (error 'tcp-connect/enable-break "not supported"))
+
+     (define (tcp-abandon-port p)
+       (tcp:tcp-abandon-port p))
+
+     (define (tcp-listen _port-no
+                         [_backlog 4]
+                         [_reuse? #f]
+                         [_hostname #f])
+       (place-tcp-listener))
+
+     (define (tcp-listener? l)
+       (place-tcp-listener? l))
+
+     (define (tcp-close _l)
+       (void))
+
+     (define (tcp-accept _l)
+       (apply values (channel-get accept-ch)))
+
+     (define (tcp-accept/enable-break _l)
+       (apply values (sync/enable-break accept-ch)))
+
+     (define (tcp-accept-ready? _l)
+       (error 'tcp-accept-ready? "not supported"))))

  (define (start-place)
    (place ch
      (let loop ([stop void])
        (match (place-channel-get ch)
          [`(init ,host ,port)
           (loop (start host port))]
          [`(stop)
           (stop)]))))

  (module+ main
    (define places
      (for/list ([idx (in-range 4)])
        (define pch (start-place))
        (begin0 pch
          (place-channel-put pch `(init "127.0.0.1" ,(+ 8000 idx))))))

    (with-handlers ([exn:break? (λ (_)
                                  (for ([pch (in-list places)])
                                    (place-channel-put pch '(stop)))
                                  (for-each place-wait places))])
      (sync never-evt)))

It may look daunting at first glance, but make-place-tcp@ is straightforward: it takes a channel of TCP connections as input and produces an instance of a unit that implements the tcp^ signature that accepts new connections off of that channel. The web server doesn't use the client-specific functions, so we don't need to bother with their implementation. The tcp-listen function returns new instances of a stub struct, and tcp-accept synchronizes on the input channel to receive new connections (each a list of an input port and an output port).

Next, let's change start-place to instantiate the unit for each web server place and to pass that unit along to the app:

  (define (start-place)
    (place ch
+     (define connections-ch (make-channel))
+     (define tcp@ (make-place-tcp@ connections-ch))
      (let loop ([stop void])
        (match (sync ch)
          [`(init ,host ,port)
-          (loop (start host port))]
+          (loop (start host port tcp@))]
          [`(stop)
           (stop)]))))

Now we need to change app.rkt's start function to take the tcp@ argument and pass it to serve:

- (define (start host port)
+ (define (start host port tcp@)
    (serve
     #:dispatch (dispatch/servlet app)
     #:listen-ip host
-    #:port port))
+    #:port port
+    #:tcp@ tcp@))

Next, we can change start-place to accept new connections on its place channel:

  (define (start-place)
    (place ch
      (define connections-ch (make-channel))
      (define tcp@ (make-place-tcp@ connections-ch))
      (let loop ([stop void])
        (match (sync ch)
          [`(init ,host ,port)
           (loop (start host port tcp@))]
+         [`(accept ,in ,out)
+          (channel-put connections-ch (list in out))
+          (loop stop)]
          [`(stop)
           (stop)]))))

Finally, we have to change the main place to make it spawn a TCP server to accept new connections and dispatch them to the server places:

  (module+ main
+   (require racket/tcp)
+
+   (define num-places 4)
    (define places
-     (for/list ([idx (in-range 4)])
+     (for/list ([_ (in-range num-places)])
        (define pch (start-place))
        (begin0 pch
-         (place-channel-put pch `(init "127.0.0.1" ,(+ 8000 idx))))))
+         (place-channel-put pch `(init "127.0.0.1" 8000)))))

+   (define listener
+     (tcp-listen 8000 4096 #t "127.0.0.1"))
+   (with-handlers ([exn:break? (λ (_)
+                                 (for ([pch (in-list places)])
+                                   (place-channel-put pch '(stop)))
-                                 (for-each place-wait places))])
+                                 (for-each place-wait places)
+                                 (tcp-close listener))])
-     (sync never-evt)))
+     (let loop ([idx 0])
+       (define pch (list-ref places idx))
+       (define-values (in out)
+         (tcp-accept listener))
+       (place-channel-put pch `(accept ,in ,out))
+       (tcp-abandon-port out)
+       (tcp-abandon-port in)
+       (loop (modulo (add1 idx) num-places))))

Now the main place spawns four other places, each running a web server that accepts new connections via the custom TCP unit, then it launches a TCP server on port 8000 and dispatches incoming connections to the server places in round-robin order. I used this approach earlier this week to improve the implementation of the Racket TechEmpower benchmark.

You can find the final version of the code in this post here.

LISP GUI Examples

Tue, 16 Nov 2021 08:00:00 +0200

I recently ran across Matthew D. Miller's "Survey of the State of GUI Programming in Lisp" series on implementing a small GUI application across various LISP implementations. The first article in that series uses racket/gui, so I figured I'd take a stab at porting that implementation to gui-easy. You can find my port here.

Porting the code was straightforward, but it uncovered a common problem with bidirectional inputs: updating the field's value observable on every change meant that the text (and cursor position) changed as the user typed because every change would trigger an update (and thus a re-rendering of the text) to the underlying text field. To work around those sorts of problems, I introduced the #:value=? and #:value->text arguments in commit ce190608. Input views with a #:value=? function only re-render the text field's contents when the current value of the input observable is different (according to the #:value=? function) from the previous one. This means that you can use that argument to control whether or not partial edits end up triggering a re-rendering of the text, so instead of:

(define/obs @n 42)
(render
 (window
  #:size '(200 #f)
  (input
   (@n . ~> . number->string)
   (λ (_event text)
     (cond [(string->number text) => (λ:= @n)])))))

You can write:

(define/obs @n 42)
(render
 (window
  #:size '(200 #f)
  (input
   @n
   #:value=? =
   #:value->text number->string
   (λ (_event text)
     (cond [(string->number text) => (λ:= @n)])))))

In the first example, typing a . after 42 re-renders the text as 42.0 and places the cursor at the end. In the second, it doesn't re-render the text at all since 42.0 and 42 are =. Still, the second example isn't perfect since string->number parses 42. to 42.0 so, if you type 42.5 into the text field and then delete the 5, it will re-render the value as 42.0. You can work around this problem by avoiding partial updates in the input's action:

   (λ (_event text)
--   (cond [(string->number text) => (λ:= @n)]))
++   (unless (string-suffix? text ".")
++     (cond [(string->number text) => (λ:= @n)])))

Perhaps a better way to handle this would be to make the #:value->text argument smarter and have it pass the current text to the rendering function when it has an arity of two. That way the rendering function can decide whether or not it needs to change the text. I'll have to experiment with that.

(eleventh RacketCon) talk: Declarative GUIs

Sun, 7 Nov 2021 16:00:00 +0200

Yesterday, I gave a talk about gui-easy at the eleventh RacketCon! You can find a recording of the talk on YouTube and a transcript below. Day two of the conference is starting in a little under a couple of hours so join us if you like!

Transcript

Declarative GUIs

My name is Bogdan Popa, and today I will be talking about gui-easy, a library for declaratively building graphical user interfaces in Racket.

racket/gui

Racket comes with racket/gui as part of its main distribution. The racket/gui library is a toolkit for building cross-platform graphical user interfaces. It's powerful and flexible, having been used to implement the DrRacket IDE. One of the reasons for that flexibility is that it's built on top of the racket/class library. The downside of that is that it exposes an imperative API. Additionally, it is agnostic concerning state management, which means it's up to you to decide how you're going to keep track of state within your application and how you're going to keep the GUI and the application's state in sync.

gui-easy

gui-easy is my attempt at adding a declarative layer on top of racket/gui. It achieves this in two ways. Firstly, by hiding the details of the class system from the user so that regular function calls form the view hierarchy. Secondly, by providing an abstraction for managing state and automatically propagating state changes to views. These two properties make it less flexible than racket/gui. In particular, you cannot opt out of its state management abstraction.

Counter

Here is an example application built with racket/gui on the left and gui-easy on the right. I can run both, and both produce roughly the same result.

The racket/gui version constructs the UI hierarchy incrementally by instantiating each widget individually and passing them around as parents of other widgets. The frame holds the panel, and the panel contains the two buttons and the message. In contrast, the gui-easy version has a closer correspondence between the final structure of the UI and the structure of the code. The window holds the panel, which holds the other three views.

The application state is managed in the racket/gui version using a mutable variable and a function that mutates that variable. In addition to changing the counter's value, the update-count! function is in charge of updating the message to reflect the change.

In the gui-easy version, an observable wraps the counter, and the library takes care of propagating changes to the relevant views (in this case, the text view).

Views

Views are regular Racket functions that combine to form the GUI hierarchy. They know how to respond to Observable changes in ways that make sense for the respective widgets they represent. For example, text views change their text when their input changes. Choice views change their current choice when their selection changes, and canvas views call their draw functions when their data changes.

Observables

An observable is like a box that can broadcast changes to its contents to observer functions. We can define an observable value, then subscribe a couple of functions to it. When we push a change to the observable, the two observers trigger. In this case, both print the new value of the observable to standard out.

The obs-map function produces derived observables by applying a function to the contents of an existing observable. Just like regular observables, we can observe derived ones. If we push a change to the original @count now, we can see both its observers trigger and the observer we added to the derived one.

While you can observe mapped observables, you cannot update them. Doing so results in a contract error.

Custom Views

Sometimes you may need to implement custom views. Doing this is straightforward. Views in gui-easy implement the view<%> interface. The interface is just four methods. Every view<%> must be able to list its dependencies. Its create method must instantiate the underlying racket/gui widget. It needs to know how to respond to changes in its dependencies and alter the underlying racket/gui widget. When it's no longer needed, it can perform any teardown actions it needs to in its destroy method.

Here is a custom text view. It depends on an observable message. To create the underlying widget, it instantiates a message%. When the @msg observable changes, it updates the label on the message% widget, and it doesn't need to perform any teardown actions, so its destroy method is a no-op. Once we have the view implementation, we can declare a constructor function to hide away the class details from users, and then we can use the new view just like we would any of the views built into gui-easy.

Demo

Next, I will live-code a small GUI to give you a feel for what it's like to use the library in practice.

[No transcript for the demo portion, sorry!]

Thanks

Thank you for attending my talk. The library and its documentation are available on the package server, and you can find the source code on my website at defn.io. Alongside the source code, you will find several example applications, so I encourage you to check those out if gui-easy appeals to you. Thanks!

Announcing dbg

Mon, 4 Oct 2021 09:10:00 +0300

I recently started working on a remote debugging/monitoring tool for Racket programs. It comes with a TCP server for exposing debugging information, a client implementation, and a GUI that builds upon those two things. You run the server as part of your application and then connect to it via the UI to debug things. Currently, it provides insights into GC pauses, current memory usage by data type, and a way to run and visualize performance profiles.

Below, you can see a short demo of dbg in action. The library is available on the package server and you can find the source code on GitHub.

Announcing gui-easy

Sun, 1 Aug 2021 10:25:00 +0300

These past couple of months, I've been working on a library for building declarative GUIs in Racket. It's called gui-easy and it works by wrapping racket/gui. See the examples directory to get an idea about how it's meant to be used, or watch the demo below:

The library is still a work in progress so expect some breaking changes. If you use it, let me know what you think!

Improvements in koyo 0.9

Fri, 30 Jul 2021 10:40:00 +0300

Recently, Daniel Holtby improved the implementation of racket/rerequire, which, in turn, inspired me to improve koyo's own code-reloading implementation. Version 0.9 (released today) no longer restarts the application process on every change and, instead, uses dynamic-rerequire to only reload the modules that change as well as any modules that depend on them. On Matchacha, which is about 11k lines of Racket code (excluding whitespace), this improves reload times by 5 to 10x, depending on the modules being reloaded.

Additionally, I added support for augmenting existing REPL sessions with the functionality of raco koyo console via start-console-here.

You can see the two changes in action in this short screencast.

Screencast: Writing a Resource Pool Library for Racket

Tue, 6 Apr 2021 07:42:00 +0300

After hacking on redis-lib for a bit on Sunday, I decided to write a general-purpose resource pooling library that I can re-use between it and http-easy and I recorded the process. You can check it out on YouTube:

You can find the library on GitHub. One particularly interesting bit about the library, that I did not to record, is that the tests are all property-based. I might do another screencast at some point to talk about how they work and the bugs they found in my original implementation (from the video).

Screencast: Building a Redis Session Store for Koyo

Sun, 4 Apr 2021 13:37:00 +0300

I decided to write a library for storing koyo sessions in Redis today and I recorded the process. If that sounds appealing, you can check it out on YouTube:

Running Racket CS on iOS

Tue, 19 Jan 2021 10:25:00 +0200

As of iOS 14.4, non-debugged builds (i.e. ones run outside of XCode) fail with a dynamic code signing error and there is no way to work around this at the moment.

A couple of weeks ago, I started working on getting Racket CS to compile and run on iOS and, with a lot of guidance from Matthew Flatt, I managed to get it working (with some caveats). Those changes have now been merged, so I figured I'd write another one of these guides while the information is still fresh in my head.

Compile Racket for macOS and for iOS

To build Racket for iOS, clone the Racket repository and follow the cross-compilation instructions under "racket/src/README.txt". The easiest approach is to create a "build" directory under "racket/src", then configure the build from within that directory by running

../configure \
  --host=aarch64-apple-darwin \
  --enable-ios=iPhoneOS \
  --enable-racket=auto

and then

make && make install

to compile Racket and set up a distribution. After running this series of commands, you should end up with a cross-compiled Racket distribution at "racket/" inside the source repository. Additionally, under "racket/src/build/local/", you'll have a compiled version of Racket CS for your host machine. You'll use that version of Racket to cross-compile Racket sources for iOS.

Cross-compile Racket modules for iOS

I added a section on how to cross-compile Racket modules to the "Inside Racket" docs so refer to that. In short, if you save the following module under "app.rkt" somewhere

#lang racket/base

(provide echo)

(define (echo m)
  (displayln m))

then you can run

/path/to/racket/src/build/local/cs/c/racketcs \
  --cross-compiler tarm64osx /path/to/racket/racket/lib \
  -MCR /path/to/racket/src/build/cs/c/compiled: \
  -G /path/to/racket/racket/etc \
  -X /path/to/racket/racket/collects \
  -l- \
  raco ctool --mods app.zo app.rkt

to produce "app.zo", a binary object containing the cross-compiled code for that module and all of its dependencies.

Set up your XCode project

To link against and use Racket CS within an XCode project, copy "racketcs.h", "racketcsboot.h" and "chezscheme.h" from "racket/include/" into a sub-directory of your project, then add that sub-directory to the "Header Search Paths" section under your project's "Build Settings" tab.

Then, disable Bitcode from the same section.

Next, copy "libracketcs.a", "petite.boot", "scheme.boot" and "racket.boot" from "racket/lib" into a sub-directory of your project called "vendor/" and drag-and-drop the "vendor/" directory into your XCode project. Then, instruct XCode to link "libracketcs.a" and "libiconv.tbd" with your code from the "Build Phases" tab. You'll have to add "libracketcs.a" to your project using the "Add Other..." sub-menu.

Next, add a new C source file called "vendor.c" and answer "yes" if prompted to create a bridging header for Swift. I tend to re-name the bridging header to plain "bridge.h" because I don't like the name that XCode generates by default. If you do this, you'll have to update the "Objective-C Bridging Header" setting in your "Build Settings" tab. From "bridge.h", include "vendor.h" and inside "vendor.h" add definitions for racket_init and echo

#ifndef vendor_h
#define vendor_h

#include <stdlib.h>

int racket_init(const char *, const char *, const char *, const char *);
void echo(const char *);

#endif

then, inside of vendor.c, implement them

#include <string.h>

#include "chezscheme.h"
#include "racketcs.h"

#include "vendor.h"

int racket_init(const char *petite_path,
                const char *scheme_path,
                const char *racket_path,
                const char *app_path) {
    racket_boot_arguments_t ba;
    memset(&ba, 0, sizeof(ba));
    ba.boot1_path = petite_path;
    ba.boot2_path = scheme_path;
    ba.boot3_path = racket_path;
    ba.exec_file = "example";
    racket_boot(&ba);
    racket_embedded_load_file(app_path, 1);
    ptr mod = Scons(Sstring_to_symbol("quote"), Scons(Sstring_to_symbol("main"), Snil));
    racket_dynamic_require(mod, Sfalse);
    Sdeactivate_thread();
    return 0;
}

void echo(const char *message) {
    Sactivate_thread();
    ptr mod = Scons(Sstring_to_symbol("quote"), Scons(Sstring_to_symbol("main"), Snil));
    ptr echo_fn = Scar(racket_dynamic_require(mod, Sstring_to_symbol("echo")));
    racket_apply(fn, Scons(Sstring(message), Snil));
    Sdeactivate_thread();
}

Take a look at the Inside Racket CS documentation for details on the embedding interface of Racket CS. The gist of racket_init is that it takes the paths to "petite.boot", "scheme.boot", "racket.boot" and "app.zo" as arguments in order to initialize Racket and then load the "app.zo" module, which you can do from the AppDelegate's application(_:didFinishLaunchingWithOptions:) method:

let vendorPath = Bundle.main.resourcePath!.appending("/vendor")
let ok = racket_init(
    vendorPath.appending("/petite.boot"),
    vendorPath.appending("/scheme.boot"),
    vendorPath.appending("/racket.boot"),
    vendorPath.appending("/app.zo"))
if ok != 0 {
    print("failed to initialize racket")
    exit(1)
}

Upon successful initialization, you should be able to call the Racket echo function from Swift:

echo("Hello from Racket!".cString(using: .utf8))

Compile and run the project on a device and you should see "Hello from Racket!" get printed in your debug console.

Some XCode gotchas

If you copy "vendor/" into your project instead of creating "folder references" when you drag-and-drop it, then code signing may fail with an ambiguous error.

Avoid using symbolic links for any of your resources (like the stuff in "vendor/"). Doing so makes copying the code over to the device fail with a "security" error that doesn't mention the root problem at all.

neko.app

Sat, 2 Jan 2021 02:00:00 +0200

I was watching Systems with JT the other day and he demoed a hobby operating system called skiftOS. During the demo he ran one of the built-in apps called "neko" which looks like a clone of an old Windows "pet" program I remember from my childhood, also called "neko" (or "neko32").

It's a really simple program: when you start it up, a cute little kitten shows up on your screen and starts running around, trying to catch your mouse cursor. I figured it would be fun to clone the program for macOS and so I went ahead and spent most of yesterday doing it. Here's the code and if you're feeling nostalgic and want to run it yourself, you can grab a build from the releases tab.

Racketeering Gophers

Tue, 17 Nov 2020 12:30:00 +0200

Close enough.

I've been working on a Wasm implementation in Racket for the past couple of weeks and have recently reached a neat milestone.

I can take this Go program,

package main

import (
        "log"
        "net/http"
)

func main() {
        log.Println("GETing https://defn.io...")
        resp, err := http.Get("https://defn.io")
        if err != nil {
                log.Fatal(err)
        }
        defer resp.Body.Close()
        log.Println(resp.Status)
}

compile it to Wasm

$ env GOARCH=wasm GOOS=js go build -o http.wasm http.go

and end up with a 7MiB Wasm file

-rwxr-xr-x  1 bogdan  staff   7.1M Nov 17 12:43 http.wasm*

that this Racket program can run:

#lang racket/base

(require wasm/private/binary
         wasm/private/validation
         wasm/private/vm
         "go-runtime.rkt")

;; Read the module.
(define m (call-with-input-file "http.wasm" read-wasm))

;; Typecheck.
(define-values (valid? error-message)
  (mod-valid? m))
(unless valid? (error error-message))

;; Create an interpreter.
(define v (make-vm m (hash "go" *go*)))

;; Grab the entrypoint and run it.
;; run(argc, argv)
(define run (vm-ref v "run" #f))
(parameterize ([current-vm v])
  (run 0 0))

$ racket run.rkt
2020/11/17 12:48:19 GETing https://defn.io...
2020/11/17 12:48:19 200 OK

Tada! The compiled Go code requires some runtime support that's specific to it, which is where the go-runtime.rkt module above comes in. I've only implemented the parts of the Go runtime support that I needed to get the above program to work and that code is pretty bad, but it gets the job done as a test for the Wasm implementation.

There's still a lot to do until this is ready to be used by others (note the lack of any sort of public API or documentation so far), but I thought this was a cool little result worth sharing.

Credit: Rocketeering Gopher by Egon Elbre on GitHub.

Racket Web Development with Koyo

Sun, 18 Oct 2020 21:00:00 +0300

Inspired by Brian Adkins' RacketCon talk from yesterday, I decided to record a screencast on what it's like to write a little web application using my not-quite-a-web-framework, koyo. You can watch it over on YouTube and you can find the resulting code on GitHub. It's unscripted and I don't go too deep on how everything works, but hopefully it's easy enough to follow and I've left the various mistakes I've made in since it's usually helpful to watch someone get out of a tricky situation so look forward to those if you watch it!

Deploying Racket Web Apps

Sun, 28 Jun 2020 15:00:00 +0300

Someone recently asked about how to deploy Racket web apps on the Racket Slack. The most common answers were

install Racket on the target machine, then ship your code there or
use Docker (basically a "portable" variant of option 1).

I wanted to take a few minutes today and write about my preferred way of deploying Racket apps: build an executable with the application code, libraries and assets embedded into it and ship that around. I prefer this approach because it means I don't have to worry about installing a specific version of Racket on the target machine just to run my code. In fact, using this approach I can have different versions of each application, each built with a different version of Racket and easily switch between them.

raco exe embeds Racket modules along with the runtime into native executables for the platform it's run on. Take this program for example:

#lang racket/base

(require racket/async-channel
         web-server/http
         web-server/servlet-dispatch
         web-server/web-server)

(define ch (make-async-channel))
(define stop
  (serve
   #:dispatch (dispatch/servlet
               (lambda (_req)
                 (response/xexpr
                  '(h1 "Hello!"))))
   #:port 8000
   #:listen-ip "127.0.0.1"
   #:confirmation-channel ch))

(define ready-or-exn (sync ch))
(when (exn:fail? ready-or-exn)
  (raise ready-or-exn))

(with-handlers ([exn:break?
                 (lambda (_)
                   (stop))])
  (sync/enable-break never-evt))

If I save it to a file called app.rkt and then call raco exe -o app app.rkt, I'll end up with a self-contained executable called app in the current directory.

$ file app
app: Mach-O 64-bit executable x86_64

The resulting executable may still refer to dynamic libraries only available on the current machine so it's not quite ready for distribution at this stage. That's where raco distribute comes in. It takes a stand-alone executable created by raco exe and generates a package containing the executable, dynamic libraries referenced by it and any run-time files referenced by the app (more on this in a sec). The resulting package can then be copied over to other machines running the same operating system.

Running raco distribute dist app produces a directory with the following contents:

$ raco distribute dist app
$ tree dist/
dist/
├── bin
│   └── app
└── lib
    ├── Racket.framework
    │   └── Versions
    │       └── 7.7.0.9_CS
    │           ├── Racket
    │           └── boot
    │               ├── petite.boot
    │               ├── racket.boot
    │               └── scheme.boot
    └── plt
        └── app
            └── exts
                └── ert
                    ├── r0
                    │   └── error.css
                    ├── r1
                    │   ├── libcrypto.1.1.dylib
                    │   └── libssl.1.1.dylib
                    └── r2
                        └── bundles
                            ├── es
                            │   └── srfi-19
                            └── srfi-19

15 directories, 10 files

I can take that directory, zip it up and ship it to any other machine running the same version of macOS as I am and it will run unmodified. The same would be true if I built the code on a Linux machine and then shipped it to other Linux machines to run on and that's exactly what I do when I distribute my web apps. I have a CI job in every project that builds and tests the code, then generates distributions that it copies to the destination servers.

At this point you might be thinking "that's nice, but what about files needed by the app at run-time?" Let's modify the app so it reads a file from disk then serves its contents on request:

#lang racket/base

(require racket/async-channel
         racket/port
         web-server/http
         web-server/servlet-dispatch
         web-server/web-server)

(define text
  (call-with-input-file "example.txt" port->string))

(define ch (make-async-channel))
(define stop
  (serve
   #:dispatch (dispatch/servlet
               (lambda (_req)
                 (response/xexpr
                  `(h1 ,text))))
   #:port 8000
   #:listen-ip "127.0.0.1"
   #:confirmation-channel ch))

(define ready-or-exn (sync ch))
(when (exn:fail? ready-or-exn)
  (raise ready-or-exn))

(with-handlers ([exn:break?
                 (lambda (_)
                   (stop))])
  (sync/enable-break never-evt))

If I just take this app, build an executable and then a distribution then try to run it, I'll run into a problem:

$ raco exe -o app app.rkt
$ raco distribute dist app
$ cd dist
$ ./bin/app
open-input-file: cannot open input file
  path: /Users/bogdan/tmp/dist/example.txt
  system error: No such file or directory; errno=2
  context...:
   raise-filesystem-error
   open-input-file
   call-with-input-file
   proc
   call-in-empty-metacontinuation-frame
   call-with-module-prompt
   body of '#%mzc:s
   temp35_0
   run-module-instance!
   perform-require!
   call-in-empty-metacontinuation-frame
   eval-one-top
   eval-compiled-parts
   embedded-load
   proc
   call-in-empty-metacontinuation-frame

Had I not cd'd into the dist directory, this would've worked, because example.txt would've been in the working directory where the application would have been run from. The problem is we're passing a path to call-with-input-file that Racket doesn't know anything about at compile time.

To ship the example.txt file along with the application, we have to use define-runtime-path to tell Racket that it should embed the file in the distribution and update the code so that it references the embedded file's eventual path.

 #lang racket/base

 (require racket/async-channel
          racket/port
+         racket/runtime-path
          web-server/http
          web-server/servlet-dispatch
          web-server/web-server)
+
+(define-runtime-path example-path "example.txt")

 (define text
-  (call-with-input-file "example.txt" port->string))
+  (call-with-input-file example-path port->string))

 (define ch (make-async-channel))
 (define stop
   (serve
    #:dispatch (dispatch/servlet
                (lambda (_req)
                  (response/xexpr
                   `(h1 ,text))))
    #:port 8000
    #:listen-ip "127.0.0.1"
    #:confirmation-channel ch))

 (define ready-or-exn (sync ch))
 (when (exn:fail? ready-or-exn)
   (raise ready-or-exn))

 (with-handlers ([exn:break?
                  (lambda (_)
                    (stop))])
   (sync/enable-break never-evt))

The use of define-runtime-path in the above code tells raco distribute to copy example.txt into the distribution and makes it so that the example-path binding refers to the path that file will eventually have.

If I build a distribution now and inspect its contents, I can see that example.txt is copied into it:

$ raco exe -o app app.rkt
$ raco distribute dist app
$ tree dist
dist/
├── bin
│   └── app
└── lib
    ├── Racket.framework
    │   └── Versions
    │       └── 7.7.0.9_CS
    │           ├── Racket
    │           └── boot
    │               ├── petite.boot
    │               ├── racket.boot
    │               └── scheme.boot
    └── plt
        └── app
            └── exts
                └── ert
                    ├── r0
                    │   └── example.txt
                    ├── r1
                    │   └── error.css
                    ├── r2
                    │   ├── libcrypto.1.1.dylib
                    │   └── libssl.1.1.dylib
                    └── r3
                        └── bundles
                            ├── es
                            │   └── srfi-19
                            └── srfi-19

16 directories, 11 files

If you want more information about how this all works, the links I gave for raco exe, raco distribute and define-runtime-path should have you covered!

Announcing http-easy

Sun, 14 Jun 2020 18:00:00 +0300

Yesterday I released http-easy, a high-level HTTP client for Racket. I started working on it after getting annoyed at some of the code in my racket-sentry package. The same day I wrote that code, someone started a mailing list thread asking for a "practical" HTTP client so that served as additional motivation to spend some time on this problem.

Here's a basic example:

(require net/http-easy)
(response-xexpr (get "https://example.com"))

It might not seem like much, but even just that gets you automatic connection pooling. Want to stream response bodies instead of reading them up front? Just pass in #t for the #:stream? argument:

(define inp
  (response-output
   (get "https://example.com" #:stream? #t)))
(read-bytes 10 inp)

Want to POST some JSON somewhere? Use the #:json keyword argument:

(post
 #:json (hasheq 'hello "world")
 "https://example.com")

Need to upload a file? It's got you covered:

(post
 #:data (multipart-payload
         (file-part "f" (open-input-file "example-1.txt"))
         (file-part "f" (open-input-file "example-2.txt")))
 "https://example.com")

You can find these examples and more in the documentation. The only big feature that's currently missing is proxy support, but I plan to add that soon. The library is pre-1.0 so, if you do start using it, keep in mind that its API might change before it stabilizes.

Continuations in Racket's Web Server

Mon, 11 May 2020 11:55:00 +0300

In The Missing Guide to Racket's Web Server, I said that dispatch/servlet is equivalent to:

(lambda (start)
  (lambda (conn req)
    (output-response conn (start req))))

That was an oversimplification. It does apply its start argument to incoming requests and it does take care of writing the responses to the appropriate connections, but it has another important job: to handle responses returned from continuations and to dispatch incoming requests to captured continuations.

With a number of details omitted, the essence of dispatch/servlet is actually the following:

(define servlet-prompt
  (make-continuation-prompt 'servlet))

(define (dispatch/servlet start)
  (define servlet (make-servlet start))
  (lambda (conn req)
    (output-response conn (call-with-continuation-barrier
                           (lambda ()
                             (call-with-continuation-prompt
                              (lambda ()
                                ((servlet-handler servlet) req))
                              servlet-prompt))))))

First, it creates a servlet value that wraps the request-handling function that it is given. The servlet contains some internal state that maps request URIs to captured continuations. The servlet's handler field is what decides which code to run when a request comes in: if the request URI matches a known continuation, then that continuation is resumed, otherwise the start function is applied to the request.

After creating the servlet, it returns a dispatcher that applies the servlet's handler to the request and writes the resulting response to the connection. Before it applies the servlet handler, it sets up a continuation barrier so that continuations captured within the servlet cannot be resumed from outside of the request-response cycle, guaranteeing that you can't resume such a continuation when the client isn't prepared to receive a response. After installing the continuation barrier, it installs a continuation prompt so that the various "web interaction" functions can abort to it.

The simplest of the web interaction functions, send/back, looks like this:

(define (send/back resp)
  (abort-current-continuation servlet-prompt (lambda () resp)))

With that in mind, consider the following request handler:

(define (hello req)
  (send/back (response/xexpr "sent"))
  (response/xexpr "ignored"))

When execution reaches the send/back function call, it aborts to the nearest¹ servlet-prompt handler, which happens to be the one that dispatch/servlet installs with call-with-continuation-prompt, so the execution of the request handler short circuits and the response passed to send/back is immediately sent to the client.

The send/suspend function, on the other hand, looks roughly² like this:

(define (send/suspend f)
  (call-with-composable-continuation
   (lambda (k)
     (define k-url (store-continuation! k))
     (send/back (f k-url)))
   servlet-prompt))

Rather than immediately sending a response back to the client, it captures the current continuation, associates it with a URL and then passes that URL to a function, f, that generates a response. The resulting response is then sent back to the client.

Using send/suspend, you can write request handlers that can be suspended in the middle of execution and then resumed upon subsequent requests:

(define (resumable req)
  (define req-2
    (send/suspend
     (lambda (k-url)
       (response/xexpr
        `(a ([href ,k-url]) "Resume")))))
  (response/xexpr "done"))

When resumable is executed, the first response is generated and returned to the client and when the client visits the anchor, the continuation is resumed from where the first request left off, with the new request bound to req-2.

If you're wondering whether or not you can install your own intermediary servlet-prompt handlers, the answer is yes! ↩
For clarity, I'm omitting a number of implementation details once again. ↩

Using GitHub Actions to Test Racket Code (Revised)

Tue, 5 May 2020 10:00:00 +0300

A little over a year ago, I wrote about how you could use the GitHub's new-at-the-time Actions feature to test Racket code. A lot has changed since then, including the release of a completely revamped version of GitHub Actions and so I thought it was time for an update.

A Basic Package

Let's say you're working on a Racket package for computing Fibonacci sequences. Your main.rkt module might look something like this:

#lang racket/base

(require racket/stream)

(provide
 fibs)

(define (fibs)
  (stream*
   1
   1
   (let ([s (fibs)])
     (for/stream ([x (in-stream s)]
                  [y (in-stream (stream-rest s))])
       (+ x y)))))

(module+ test
  (require rackunit)
  (check-equal? (stream->list (stream-take (fibs) 8))
                '(1 1 2 3 5 8 13 21)))

You'd like to make it so that every time you push a change to this package to GitHub the test in this module gets run and you get notified of any problems that occur. To do that, all you have to do is add a workflow configuration file under .github/workflows. The file can be called anything you like as long as it ends with the yml extension. In this case you might call it push.yml, because its contents will get run whenever code is pushed to the repository. A basic workflow file looks like this:

on:
  - push

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@master
      - name: Install Racket
        uses: Bogdanp/setup-racket@v1.6.1
        with:
          architecture: 'x64'
          distribution: 'full'
          version: '8.2'
      - name: Run Tests
        run: raco test main.rkt

This workflow triggers whenever a push event occurs within the repository. When that happens, it'll run through its jobs one-by-one. The test job in this workflow sets up a Ubuntu VM where it'll go through each of its steps in order.

The first step uses the actions/checkout action to clone the repository inside the VM. Once checked out, the working directory for all the subsequent actions will be in the root of the checked-out repository, unless otherwise specified within a step.

The second step uses my own Bogdanp/setup-racket action to install Racket CS version 8.2 in the VM. This adds the racket and raco executables to the PATH.

Finally, the last step runs the tests in the main.rkt module.

Installing Dependencies

Say you're not that confident in that one test that you have for the fibs function and you'll like to throw some property-based testing in the mix. Your test submodule becomes:

(module+ test
  (require rackcheck
           rackunit)

  (check-property
   (property ([n (gen:integer-in 3 100)])
     (define numbers (stream->list (stream-take (fibs) n)))
     (for ([n (cddr numbers)]
           [y (cdr numbers)]
           [x numbers])
       (check-eqv? (+ x y) n)))))

When you push this change, your action will fail because rackcheck won't be installed on the VM. To work around this, you can update the Install Racket step to tell it to install rackcheck for you:

       - name: Install Racket
         uses: Bogdanp/setup-racket@v1.6.1
         with:
           architecture: 'x64'
           distribution: 'full'
           version: '8.2'
+          packages: 'rackcheck'
       - name: Run Tests
         run: raco test main.rkt

A better solution, however, would be to add a info.rkt file to your repository to specify what dependencies your package has:

#lang info

(define build-deps '("rackcheck" "rackunit-lib"))

Then, between the Install Racket and the Run Tests steps, you can add another step to install your package into the VM:

       - name: Install Racket
         uses: Bogdanp/setup-racket@v1.6.1
         with:
           architecture: 'x64'
           distribution: 'full'
           version: '8.2'
+      - name: Install Package and its Dependencies
+        run: raco pkg install --auto --batch
       - name: Run Tests
         run: raco test main.rkt

This way, you won't have to worry about updating your workflow every time you change your dependencies.

Matrix Testing

At this point you might be fairly confident that your implementation of fibs is correct, but you want to guarantee that it works not only on Racket CS version 8.2, but also on Racket BC as well. To do this, you can add a matrix strategy to your job, specifying that the job should be parameterized over the racket-variant values:

 jobs:
   test:
     runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        racket-variant: ['BC', 'CS']
+    name: Test on ${{ matrix.racket-variant }} Racket
     steps:

You can then update the install step to be parameterized over the variant:

       - name: Install Racket
         uses: Bogdanp/setup-racket@v1.6.1
         with:
           architecture: 'x64'
           distribution: 'full'
+          variant: ${{ matrix.racket-variant }}
           version: '8.2'

You can go one step further and also parameterize the versions of Racket that you want your tests to run on:

 jobs:
   test:
     runs-on: ubuntu-latest
     strategy:
       matrix:
         racket-variant: ['BC', 'CS']
+        racket-version: ['8.1', '8.2']
     name: Test on ${{ matrix.racket-variant }} Racket
     steps:

And then plug that parameter into the install step, as before:

       - name: Install Racket
         uses: Bogdanp/setup-racket@v1.6.1
         with:
           architecture: 'x64'
           distribution: 'full'
           variant: ${{ matrix.racket-variant }}
+          version: ${{ matrix.racket-version }}

Following these steps will make it so that every change you push will get tested against versions 8.1 and 8.2 of both variants of Racket.

This only scratches the surface of what you can do with GH Actions so, if you're interested to learn more, I'd recommend reading through the docs. You can find a working example of everything I've mentioned in this article in this repo.

Announcing racksnaps

Sun, 3 May 2020 14:00:00 +0300

Racket's package manager doesn't currently have the notion of locking package sets to specific versions¹ per project so, as someone who operates a couple production Racket applications, I've been concerned about the possibility that new deployments could introduce bugs in production due to changing dependencies.

To solve this problem, over the past weekend I've put together a service that creates daily snapshots of the official package catalog. You can find it at racksnaps.defn.io.

Every day at 12am UTC, the service queries all the packages on pkgs.racket-lang.org for metadata and source locations. It then creates a source package archive for each package whose sources are still valid.

Once all the source package archives are created, "built" packages (packages that contain source code, docs and compiled .zo files) are created from those archives. Each of these is compiled in isolation and any packages that don't compile cleanly are excluded from the final snapshot.

Snapshots are never modified once they succeed and a content addressing scheme is used for the individual packages to avoid using up too much disk space over time.

I plan to keep snapshots around indefinitely and I may add support for paid features like custom package sets eventually -- if there is interest -- to help support the hosting costs.

An Example

Say you've just started working on a new application. To develop against the snapshot from May 2nd, 2020 using Racket 7.6, you might run the following command:

raco pkg config --set catalogs \
    https://download.racket-lang.org/releases/7.6/catalog/ \
    https://racksnaps.defn.io/snapshots/2020/05/02/catalog/ \
    https://pkgs.racket-lang.org \
    https://planet-compats.racket-lang.org

This will make it so that any packages you install will first look up the 7.6 release catalog (for packages in the "main distribution", like rackunit and typed-racket), then it'll look up packages in the snapshot and fall back to the package catalog for any packages not in the snapshot.

When building the application in CI you might limit the catalog list to just the release catalog (for packages in the main distribution) and the snapshot:

raco pkg config --set catalogs \
    https://download.racket-lang.org/releases/7.6/catalog/ \
    https://racksnaps.defn.io/snapshots/2020/05/02/catalog/

To speed up builds, you might layer in the built-snapshot for that day:

raco pkg config --set catalogs \
    https://download.racket-lang.org/releases/7.6/catalog/ \
    https://racksnaps.defn.io/built-snapshots/2020/05/02/catalog/ \
    https://racksnaps.defn.io/snapshots/2020/05/02/catalog/

Days/weeks/months/years later, when you're ready to deal with upgrading your dependencies, you can update to a more recent snapshot and repeat the cycle.

I'm receptive to feedback on how to improve the service so don't hesitate to reach out if you think of anything!

Technically, it does support pinning a specific sha per package when using git sources, but that is pretty cumbersome and it means that packages always have to be installed from source, which increases install times by 2 to 3x. ↩

Converting byte arrays to UUIDs in Postgres

Sun, 5 Apr 2020 19:00:00 +0300

For a project that I'm working on, I have a custom flake id spec that allows me to generate unique, sortable identifiers across computers without any sort of synchronization. The ids themselves can be encoded down to 16 bytes and I wanted to store them in Postgres. A good way to do that is to leverage Postgres' UUID data type, which lets you efficiently store any 16 byte quantity in a way that can be indexed reasonably well.

The problem I ran into was that my DB library of choice only supports inserting UUID values that follow the standard UUID format, so queries like

INSERT INTO the_table(uuid_column) VALUES ($1)

would get rejected at runtime unless $1 actually looked like a UUID.

I considered converting the ids into the standard UUID format within my application code but that didn't feel like the right thing to do. Instead, I found that Postgres has a standard function called encode that is able to take any byte array and encode it into a hex string so all I had to do was change my query into

INSERT INTO the_table(uuid_column) VALUES (CAST(ENCODE($1, 'hex') AS UUID))

and that worked great!

Testing a Web API using rackcheck

Fri, 13 Mar 2020 11:00:00 +0200

Yesterday, I announced rackcheck, my new property-based testing library for Racket and I wanted to do a quick dive into one of the examples in the rackcheck repo where a simple web API is integration tested using PBT.

You can find the full example here.

The app being tested is a simple leaderboard HTTP API with 3 endpoints:

GET /players: lists all the registered players in order from highest score to lowest. Ties are broken by a secondary sort on the names of the players in ascending order.
POST /players: expects a JSON object containing a player name. If a player with that name does not exist, then it is created. If it does, then a 400 response is returned.
POST /scores/{name}: increments the score of the player identified by {name}. Does nothing if a player with that name cannot be found.

The details of the API implementation don't matter much so I won't cover it apart from pointing out that the code is intentionally tightly coupled: all of the business logic is directly tied to the request handling code. One criticism I've seen of PBT is that it isn't usable in contexts where the code you want to test isn't well-factored so I wanted to show that this isn't true.

The Less Interesting Bits

Reading through the code from the top of the test submodule we have...

(define (reset)
  (query-exec (current-conn) "DELETE FROM players"))

reset ensures the database is in a clean slate. It gets called before every test so that the tests themselves don't interfere with one another.

(define (request path [data #f])
  (define-values (status _headers out)
    (http-sendrecv "127.0.0.1" path
                   #:port 9911
                   #:method (if data #"POST" #"GET")
                   #:headers (if data '(#"Content-type: application/json") null)
                   #:data (and data (jsexpr->bytes data))))

  (match status
    [(regexp #rx#"^HTTP.... 200 ")
     (read-json out)]

    [(regexp #rx#"^HTTP.... 4.. ")
     (error 'client "bad request: ~s" (read-json out))]

    [_
     (error 'server (port->string out))]))

request is used to make requests to the API from within the tests.

(run-tests
 (test-suite
  "web-api"
  #:before
  (lambda ()
    (current-conn (sqlite3-connect #:database 'memory))
    (init-db)

    (define ch (make-async-channel))
    (set! stop (start ch))
    (sync ch))

   ...))

The test suite initializes the database then starts the web server on a well-known port and waits for it to finish starting up. The server itself listens for connections a background thread.

...

#:after
(lambda ()
  (stop))

...

After the tests are all done, the suite gracefully shuts down the server.

The Interesting Bits

The approach I've taken to test the API is to come up with a simple model for what the state of the API should be at any point and then run arbitrary operations against the API, modifying both the API and the model of its state at the same time. After every operation, I check that the state of the model matches that of the API.

To begin with, I define a struct for the model:

(struct model (scores-by-name)
  #:transparent)

The model is just a mapping from player names to their scores at some point in time.

Next is a generator for player names:

(define gen:player-name
  (gen:let ([given (gen:string gen:char-letter)]
            [family (gen:string gen:char-letter)])
    (format "~a ~a" given family)))

When sampled, it produces values like:

web-api.rkt/test> (sample gen:player-name)
'(" "
  " "
  "vOVu "
  "FSIHd lly"
  "GvbsC JHdLeHmegT"
  "qWs sxsRXIxyZZGOtNVZwtdghwEY"
  "hKxIwwFZZDVoMirDig qpiGrJkbugmyodzXYxYnesIiS"
  "GikMSXKgMozVFWkDhWYduvyjTiSOJaTyNERaKhjPwTrerhoNM goHUhdziwTHzBnJeTrQUGcsLWKQYPGGqLSBntHWBtxw"
  "rylcoMnEtAMmdwsqvZiHqx ZgnOYbxJdeZ"
  "LGzrZIHZjnaZebCAvzPmzhvkbTL zxBzKdIbKumrXptYPEeQuPNqhAOiqczGb")

Next is the generator for operations:

(define gen:ops
  (gen:let ([names (gen:no-shrink
                    (gen:resize (gen:filter (gen:list gen:player-name)
                                            (compose1 not null?))
                                10))]
            [ops (gen:list
                  (gen:choice
                   (gen:tuple (gen:const 'create) (gen:one-of names))
                   (gen:tuple (gen:const 'increase) (gen:one-of names))))])
    (cons '(init) ops)))

It pulls from the same set of up to 10 names generated by gen:player-name to generate lists of operations that always start with '(init) followed by zero or more randomly-selected '(create ...) or '(increase ...) operations. Sampling it three times produces:

web-api.rkt/test> (sample gen:ops 3)
'(((init))
  ((init))
  ((init)
   (create "CveqBE K")
   (create "CveqBE K")
   (increase "ESXrkpSC uS")
   (create "sLvsTrsr ZKcVQr")))

Next is the interpreter:

(define/match (interpret s op)
  ...)

interpret takes the current state and the operation it's supposed to run, checks any pre-conditions, runs the operation, checks any post-conditions and returns the new state.

  ...

  [(_ (list 'init))
   (reset)
   (model (hash))]

  ...

When interpret receives an '(init) operation, it resets the database and returns a fresh model.

  ...

  [((model scores) (list 'create name))
   (define (create-player)
     (with-handlers ([exn:fail? void])
       (request "/players" (hasheq 'name name))))

   (define (player-names)
     (sort (for/list ([player (in-list (request "/players"))])
             (hash-ref player 'name))
           string<?))

   (define (scores->names s)
     (sort (hash-keys s) string<?))

   (cond
     [(regexp-match-exact? " *" name)
      (begin0 s
        (create-player)
        (check-equal? (player-names) (scores->names scores)))]

     [(hash-has-key? scores name)
      (begin0 s
        (create-player)
        (check-equal? (player-names) (scores->names scores)))]

     [else
      (define scores* (hash-set scores name 0))
      (begin0 (model scores*)
        (create-player)
        (check-equal? (player-names) (scores->names scores*)))])]

  ...

When interpret receives a '(create "player name") operation, it sends a request to create the player to the API and then grabs all the players in a subsequent request. Finally, it makes sure they match the updated model.

  ...

  [((model scores) (list 'increase name))
   (define scores*
     (if (hash-has-key? scores name)
         (hash-update scores name add1)
         scores))

   (request (format "/scores/~a" name) (hasheq))
   (check-equal?
    (for/list ([player (in-list (request "/players"))])
      (cons (hash-ref player 'name)
            (hash-ref player 'score)))
    (sort (sort (hash->list scores*) string<? #:key car) > #:key cdr))
   (model scores*)])

When interpret receives an '(increase "player name") operation, it sends a request to increase the player's score and then grabs the leaderboard in a subsequent request to ensure it matches the model.

(check-property
 (make-config #:tests 30)
 (property ([ops gen:ops])
   (for/fold ([s #f])
             ([op (in-list ops)])
     (interpret s op))))

Finally, I plug everything together by calling check-property on a property whose inputs are operation lists generated using gen:ops. The property just interprets every command in sequence and interpret will raise an exception if the application ends up in a bad state.

If I uncomment the check in the API that ensures no two players can have the same name and then run the tests I get:

; FAILURE
; /Users/bogdan/sandbox/rackcheck/examples/web-api.rkt:209:6
location:   web-api.rkt:209:6
name:       unnamed
seed:       1485163264
actual:     '(" UUVDlrhi" " UUVDlrhi")
expected:   '(" UUVDlrhi")

Failed after 4 tests:

  ops = ((init) (create "ZUNEQq k") (create "OPKmoJRUyl IYkkSON") (create "DrfMu pMLxwX") (increase "ZUNEQq k") (increase "OPKmoJRUyl IYkkSON") (create "tHsrGne IRVcaNpt") (create "ZUNEQq k"))

Shrunk:

  ops = ((init) (create " UUVDlrhi") (create " UUVDlrhi"))

--------------------
0 success(es) 1 failure(s) 0 error(s) 1 test(s) run
1

Which is pretty great if you ask me!

In Closing

You might think this was a lot of work compared to just writing example tests, but at the end of all this I have a straightforward specification for my API by way of the interpret function and the tests that get thrown at the API are far more diverse than anything I'd ever have taken the time to write by hand.

Extending the interpreter or adding new interpreters as the API grows is also very easy once you get the hang of this pattern.

Announcing rackcheck

Thu, 12 Mar 2020 14:00:00 +0200

I've been playing around with property-based testing in Racket this past week. I started by forking the existing quickcheck library to try and add support for shrinking, but I quickly realized that I'd have to make a number of breaking changes to get it to work the way I wanted so, in the end, I decided to start a new library from scratch.

The library is called rackcheck and you can grab it off of the package server. The reference docs should show up on there soon and there are a few examples in the repo. I'm pretty happy with the result so far, but I may end up making some small API adjustments as I use it more so keep that in mind!

The Missing Guide to Racket's Web Server

Wed, 12 Feb 2020 12:00:00 +0200

Racket's built-in web-server package is great, but parts of it are low-level enough that it can be confusing to people who are new to the language. In this post, I'm going to try to clear up some of that confusion by providing some definitions and examples for things beginners might wonder about.

Servlets

A servlet is a function from a request to a response. It has the contract:

(-> request? can-be-response?)

Here's a servlet that replies with "Hello, world!" regardless of what the request looks like:

#lang racket/base

(require web-server/http)

(define (hello req)
  (response/output
   (lambda (out)
     (displayln "Hello, world!" out))))

And here's one that dynamically constructs a response based on the request's query parameters:

#lang racket/base

(require racket/match
         web-server/http)

(define (age req)
  (define binds (request-bindings/raw req))
  (define message
    (match (list (bindings-assq #"name" binds)
                 (bindings-assq #"age" binds))
      [(list #f #f)
       "Anonymous is unknown years old."]

      [(list #f (binding:form _ age))
       (format "Anonymous is ~a years old." age)]

      [(list (binding:form _ name) #f)
       (format "~a is unknown years old." name)]

      [(list (binding:form _ name)
             (binding:form _ age))
       (format "~a is ~a years old." name age)]))
  (response/output
   (lambda (out)
     (displayln message out))))

serve/servlet is a convenience function that configures a server to run whatever servlet you give it.

Here's how you'd run the age servlet using serve/servlet:

#lang racket/base

(define age ...)

(serve/servlet
 age
 #:listen-ip "127.0.0.1"
 #:port 8000
 #:command-line? #t
 #:servlet-path ""
 #:servlet-regexp #rx"")

While very convenient for quick things, it obscures a lot of what's going on under the hood from the caller. An invocation of the lower-level serve function that achieves the same result would look like:

#lang racket/base

(require racket/match
         web-server/http
         web-server/servlet-dispatch
         web-server/web-server)

(define age ...)

(define stop
  (serve
   #:dispatch (dispatch/servlet age)
   #:listen-ip "127.0.0.1"
   #:port 8000))

(with-handlers ([exn:break? (lambda (e)
                              (stop))])
  (sync/enable-break never-evt))

This sets up a web server with a single dispatcher that runs a single servlet, running in a background thread. The return value of the serve function is a function that can be used to stop the server and, since the server runs in a background thread, I need to do something on the main thread to prevent it from terminating. I've chosen to wait on an event that never terminates and to capture breaks (such as the SIGINT and SIGTERM signals (the former is sent when you press Ctrl+C on a running process)). When such a break is received, the stop function gets called and the server terminates gracefully.

Dispatchers

You may have noticed that, unlike with serve/servlet, I couldn't just pass my age servlet directly to serve. I had to turn it into a dispatcher by calling dispatch/servlet. This is because a dispatcher, not a servlet, sits at the root of every server.

A dispatcher is a function that takes a connection object and a request and either services that request or calls next-dispatcher. Its contract is:

(-> connection? request? any)

Dispatchers' return values are ignored. They operate directly on the connection objects that they are given. If I wanted to make my own dispatcher to run the age servlet instead of using dispatch/servlet, it'd look something like this:

#lang racket/base

(require web-server/http/response)

(define (age-dispatcher conn req)
  (output-response conn (age req)))

output-response takes a connection and a response and serializes the response over the connection to the client end.

This is equivalent¹ to:

(define age-dispatcher (dispatch/servlet age))

There are a number of built-in dispatchers that you'd normally make use of in a real world project. The most important of which are:

`dispatch-sequencer`

This dispatcher takes a list of dispatchers and runs through them in order on every request, until it reaches the first one that doesn't call next-dispatcher.

#lang racket/base

(require net/url
         racket/string
         web-server/dispatchers/dispatch
         (prefix-in sequencer: web-server/dispatchers/dispatch-sequencer)
         web-server/http
         web-server/http/response
         web-server/web-server)

(define (request-path-has-prefix? req p)
  (string-prefix? (path->string (url->path (request-uri req))) p))

(define (a-dispatcher conn req)
  (if (request-path-has-prefix? req "/a/")
      (output-response conn (response/output
                             (lambda (out)
                               (displayln "hello from a" out))))
      (next-dispatcher)))

(define (b-dispatcher conn req)
  (output-response conn
                   (response/output
                    (lambda (out)
                      (displayln "hello from b" out)))))

(define stop
  (serve
   #:dispatch (sequencer:make a-dispatcher
                              b-dispatcher)
   #:listen-ip "127.0.0.1"
   #:port 8000))

(with-handlers ([exn:break? (lambda (e)
                              (stop))])
  (sync/enable-break never-evt))

The above server runs the a-dispatcher on every request. If the request path doesn't start with "/a/", then it moves on to the b-dispatcher.

`dispatch-filter`

Filtering the request path like I did in the previous snippet is pretty cumbersome so the web-server provides the filtering dispatcher for this exact purpose. The above code could be rewritten as:

#lang racket/base

(require (prefix-in filter: web-server/dispatchers/dispatch-filter)
         (prefix-in sequencer: web-server/dispatchers/dispatch-sequencer)
         web-server/http
         web-server/http/response
         web-server/web-server)

(define (a-dispatcher conn req)
  (output-response conn
                   (response/output
                    (lambda (out)
                      (displayln "hello from a" out)))))

(define (b-dispatcher conn req)
  (output-response conn
                   (response/output
                    (lambda (out)
                      (displayln "hello from b" out)))))

(define stop
  (serve
   #:dispatch (sequencer:make (filter:make #rx"^/a/" a-dispatcher)
                              b-dispatcher)
   #:listen-ip "127.0.0.1"
   #:port 8000))

(with-handlers ([exn:break? (lambda (e)
                              (stop))])
  (sync/enable-break never-evt))

`dispatch-files`

This dispatcher can be used to serve files off of the filesystem. You can combine it with the other dispatchers to generate a server that can either serve files off of the filesystem or fall back to a servlet:

#lang racket/base

(require net/url
         (prefix-in files: web-server/dispatchers/dispatch-files)
         (prefix-in filter: web-server/dispatchers/dispatch-filter)
         (prefix-in sequencer: web-server/dispatchers/dispatch-sequencer)
         web-server/dispatchers/filesystem-map
         web-server/http
         web-server/servlet-dispatch
         web-server/web-server)

(define (homepage req)
  (response/xexpr
   '(html
     (head
      (link ([href "/static/screen.css"] [rel "stylesheet"])))
     (body
      (h1 "Hello!")))))

(define url->path/static
  (make-url->path "static"))

(define static-dispatcher
  (files:make #:url->path (lambda (u)
                            (url->path/static
                             (struct-copy url u [path (cdr (url-path u))])))))

(define stop
  (serve
   #:dispatch (sequencer:make
               (filter:make #rx"^/static/" static-dispatcher)
               (dispatch/servlet homepage))
   #:listen-ip "127.0.0.1"
   #:port 8000))

(with-handlers ([exn:break? (lambda (e)
                              (stop))])
  (sync/enable-break never-evt))

This dispatcher needs to know how to map the current request URL to a path on the filesystem.

First, I create a function that maps URLs to file paths within the static directory (a relative path from where the server happens to be run (the current working directory)). This function automatically removes things like .. from the paths it is given, ensuring that no request paths can "escape" out of the static directory.

Then, I pass files:make a function that maps URLs to file paths. Since I'm going to serve all static files from URLs that start with /static/, I need to drop that prefix from the URL before I pass it to the url->path/static function because it expects a file path relative to the static directory.

Finally, I sequence the static dispatcher along with a servlet dispatcher that serves the home page and the end result is a web server that can serve static files from a directory and run dynamic Racket code!

Routing

You could route requests by sequencing together multiple dispatch-filter dispatchers, but that wouldn't be very ergonomic. The web server provides the dispatch-rules macro as a convenient way to declare servlets -- not dispatchers! the overloading of terms here can be a bit confusing -- that perform different actions based on the request method and path.

#lang racket/base

(require net/url
         web-server/dispatch
         (prefix-in files: web-server/dispatchers/dispatch-files)
         (prefix-in filter: web-server/dispatchers/dispatch-filter)
         (prefix-in sequencer: web-server/dispatchers/dispatch-sequencer)
         web-server/dispatchers/filesystem-map
         web-server/http
         web-server/servlet-dispatch
         web-server/web-server)

(define (response/template . content)
  (response/xexpr
   `(html
     (head
      (link ([href "/static/screen.css"] [rel "stylesheet"])))
     (body
      ,@content))))

(define (homepage req)
  (response/template '(h1 "Home")))

(define (blog req)
  (response/template '(h1 "Blog")))

(define-values (app reverse-uri)
  (dispatch-rules
   [("") homepage]
   [("blog") blog]))

(define url->path/static (make-url->path "static"))

(define static-dispatcher
  (files:make #:url->path (lambda (u)
                            (url->path/static
                             (struct-copy url u [path (cdr (url-path u))])))))

(define stop
  (serve
   #:dispatch (sequencer:make
               (filter:make #rx"^/static/" static-dispatcher)
               (dispatch/servlet app))
   #:listen-ip "127.0.0.1"
   #:port 8000))

(with-handlers ([exn:break? (lambda (e)
                              (stop))])
  (sync/enable-break never-evt))

Using dispatch-rules as I've done above produces two values: a servlet that maps requests made to / to the homepage servlet and requests made to /blog to the blog servlet, and a function that can produce reverse URIs when given either of those functions.

Plugging that in via dispatch/servlet into the main servlet sequence gets you a server that can serve files off of disk and also dynamically dispatch requests to multiple servlets.

One final tweak we might want to make here is to plug another servlet after the app servlet into the sequencer to handle requests to paths that don't exist:

#lang racket/base

(require net/url
         web-server/dispatch
         (prefix-in files: web-server/dispatchers/dispatch-files)
         (prefix-in filter: web-server/dispatchers/dispatch-filter)
         (prefix-in sequencer: web-server/dispatchers/dispatch-sequencer)
         web-server/dispatchers/filesystem-map
         web-server/http
         web-server/servlet-dispatch
         web-server/web-server)

(define (response/template . content)
  (response/xexpr
   `(html
     (head
      (link ([href "/static/screen.css"] [rel "stylesheet"])))
     (body
      ,@content))))

(define (homepage req)
  (response/template '(h1 "Home")))

(define (blog req)
  (response/template '(h1 "Blog")))

(define (not-found req)
  (response/template '(h1 "Not Found")))

(define-values (app reverse-uri)
  (dispatch-rules
   [("") homepage]
   [("blog") blog]))

(define url->path/static (make-url->path "static"))

(define static-dispatcher
  (files:make #:url->path (lambda (u)
                            (url->path/static
                             (struct-copy url u [path (cdr (url-path u))])))))

(define stop
  (serve
   #:dispatch (sequencer:make
               (filter:make #rx"^/static/" static-dispatcher)
               (dispatch/servlet app)
               (dispatch/servlet not-found))
   #:listen-ip "127.0.0.1"
   #:port 8000))

(with-handlers ([exn:break? (lambda (e)
                              (stop))])
  (sync/enable-break never-evt))

I am simplifying things here for the purposes of this guide. The dispatch/servlet function does some additional work to support continuations. See Continuations in Racket's Web Server for details. ↩

Announcing Try Racket

Fri, 31 Jan 2020 12:00:00 +0200

I'd been meaning to play with Racket's built-in sandboxing capabilities for a while so yesterday I sat down and made Try Racket. It's a web app that lets you type in Racket code and run it. The code you run is tied to your session and each session is allocated up to 60 seconds of run time per evaluation, with up to 128MB of memory used. Filesystem and network access is not permitted and neither is access to the FFI. The application itself runs inside a further-restricted Docker container.

You can find the source code on GitHub. Contributions and improvements are totally welcome!

Running Racket BC on iOS

Sun, 5 Jan 2020 20:00:00 +0200

As of 2021-01-18, it is possible to run Racket CS on iOS.

/u/myfreeweb pointed out to me in a lobste.rs thread yesterday that Racket compiles just fine on aarch64 and that led me down a rabbit hole trying to get Racket running inside an iOS application. I finally succeeded so I figured I'd write down my findings in hopes of helping future Racketeers (myself included) going down this path!

Compile Racket for macOS

A recent-enough version of Racket is required in order to compile Racket for iOS. The best way to do that is to clone the Racket repo and follow the build instructions which should be as simple as running make in the repository root.

Assuming you're following along in a terminal session, run

export RACKET_SRC=$(pwd)

You'll need to reference this directory in the following steps.

Compile Racket for iOS

Once you've successfully compiled Racket for macOS, clone the Racket repository again, this time under a different directory. I called this directory racket-ios to differentiate the two, but you can call it whatever. Make sure the same commit is checked out in both repos and run through the following build steps starting at the repository root:

mkdir racket/src/build \
  && cd racket/src/build \
  && ../configure \
        --host=aarch64-apple-darwin \
        --enable-ios=iPhoneOS \
        --enable-racket="$RACKET_SRC/racket/bin/racket"

This will configure the build to create objects that can run on a physical device. To build Racket for the simulator instead, change the host to x86_64-apple-darwin and enable-ios from iPhoneOS to iPhoneSimulator. For details on these flags, see the cross compiling instructions in the Racket repo.

~~Although the instructions currently don't mention that pthread support is required when configuring the build, the code will fail to compile without it.~~ Matthew Flatt pushed a fix for this today!

Next, run make cgc && make install-cgc to compile the code and the packages. This builds the conservative GC variant of Racket. I started out trying to get everything running using the 3m variant of Racket (with a precise GC), but I ran into a number of roadblocks, including an LLVM bug from 2015 so I eventually gave up and switched to the CGC variant.

Create the Xcode Project

Create a new iOS-based project in Xcode. Inside that project, add a new group called "Frameworks" and then drag and drop racket/libmzgc.a, racket/libracket.a and rktio/librktio.a from the racket/src/build directory into the "Frameworks" group. Make sure "Copy items if needed" is toggled.

Open the project settings and, from the "Build Phases" -> "Link Binary with Libraries" section, add libiconv.tbd. Racket depends on this library.

Copy racket/include into your project and then from the "Build Settings" -> "Search Paths" -> "Header Search Paths" section add $(SRCROOT)/include.

Add a new header file called BridgingHeader.h and then from the "Build Settings" -> "Swift Compiler - General" -> "Objective-C Bridging Header" add the path to the file. Everything defined in this file will be made available to the Swift code. Inside the file add:

#include "Interop.h"

Create a new C file called Interop.c and add

static int run(Scheme_Env *e, int argc, char *argv[]) {
    return 0;
}

void run_racket() {
    scheme_main_setup(1, run, 0, NULL);
}

In its associated header file, Interop.h, add

#include "scheme.h"

void run_racket(void);

Finally, inside the AppDelegate's application:didFinishLaunchingWithOptions method, add a call to run_racket and run your project on your device to ensure everything compiles and runs properly.

If everything runs correctly, then pat yourself on the back, you've just run Racket on iOS! Of course, Racket's not really doing much at this point. Let's embed a Racket module into the project.

In the project source directory, create a new file called hello.rkt with the following contents:

#lang racket/base

(displayln "Hello, World!")

From the command line, compile hello.rkt and all its transitive dependencies into a C file:

"$RACKET_SRC/racket/bin/raco" ctool --c-mods hello.c hello.rkt

Next, update Interop.c to include the resulting C file:

#include "hello.c"

And then update run in that same file to initialize and run that module:

static int run(Scheme_Env *e, int argc, char *argv[]) {
    Scheme_Object *a[2];
    declare_modules(e);
    a[0] = scheme_make_pair(scheme_intern_symbol("quote"),
                            scheme_make_pair(scheme_intern_symbol("hello"),
                                             scheme_make_null()));
    a[1] = scheme_false;
    scheme_dynamic_require(2, a);
    return 0;
}

Note that the name of the module passed to scheme_intern_symbol is "hello", the same as the file name.

Run your project, and you should see "Hello, World!" in your console.

It took a little bit of work, but we got there! From here, you should be able to do more interesting stuff. I'll get into some of that when I start porting Remember to iOS. In the mean time, you can read up on this stuff over here!

Native Applications with Racket

Sat, 4 Jan 2020 13:00:00 +0200

A couple of days ago, I released a native macOS application called Remember. It is a small, keyboard-driven application for stashing away notes/reminders for later. One of the cool things about it from a programming nerd perspective is that, while it is a completely native Cocoa application whose frontend is built with Swift, the core business logic is all in Racket!

Why not use racket/gui?

I started out with a proof of concept that used Racket for the GUI, but I realized that I'd have to write a bunch of Objective-C FFI code to get the UI to look the way I wanted (a carbon copy of Spotlight) and it seemed like it would be a pain to try and integrate DDHotKey and to add support for launching at login into a package that is easy to distribute. I was also unsure about how I could notarize the distribution for macOS Catalina (more on this later).

Why not do it all in Swift?

I don't know Swift particularly well, nor do I like it very much. I find Apple's documentation lackluster and Xcode is surprisingly buggy (renaming a class and its associated file fails to rename the file on disk, which causes Xcode to fail silently, for example). I wouldn't mind the documentation being bad if the core cocoa code was open source/source available; at least then I could look at the implementation to try and understand what's going on.

More importantly, I plan to support Windows and Linux which means that writing the core in a portable language is going to minimize the amount of work I have to do as well as the differences between the implementations on each platform.

How it Works

The Racket core runs a custom JSONRPC server that listens for commands on stdin and sends responses on stdout. Using Racket's raco exe and raco distribute commands, that core gets built into a native executable and copied into the Swift app's resources folder. The Swift application runs the core as a subprocess on startup and communicates with it via pipes.

RPC commands are asynchronously handled by the core and the core may also send asynchronous notifications to the frontend to let it know when entries are due.

Notarization

It took me a couple of hours to figure out how to get everything notarized. I had to enable App Sandboxing for both the frontend and the core application, figure out via trial and error which entitlements were necessary, realize that I needed a separate set of entitlements for the core application and that the com.apple.security.inherit entitlement, for whatever reason, doesn't let the subprocess inherit its parents entitlements, meaning that I had to also explicitly assign the core application the "Allow JIT" and "Allow Unsigned Executable Memory" entitlements, else the process would get killed with a SIGINT and a red herring error message about how the executable doesn't have a valid bundle identifier.

Would I do this again?

I'll have to see how porting to other platforms goes, but so far I'm very happy with this approach. The result is fast and now that I've built the RPC infrastructure I can easily copy all that code into other projects. Writing the business logic in Racket means that I can iterate very quickly and writing the GUI code using the native tools for each platform is advantageous in terms of look and feel and distribution.

What about iOS?

Unfortunately, the RPC approach breaks down on iOS where you're not allowed to run subprocesses. An approach that could work there is building the app into a shared library, linking against it and doing the RPC in-process. I think that approach could work, but Racket would have to be able to target arm64 for that to be feasible. Fortunately, now that Racket is able to run on top of Chez Scheme, which already has backends for many platforms, including arm32le, that might be a possibility in the future. ¹

myfreeweb pointed out on lobste.rs that this is already supported by Racket BC! ↩

Announcing Remember for macOS

Thu, 2 Jan 2020 18:00:00 +0200

I've been using org-mode capture templates for years and I've always wished I had something like that for the whole system. I took advantage of the holiday break to build Remember, a little reminders application with Spotlight-like UX. It's available on Gumroad and you can pay what you want for it (inlcuding $0!) at the moment so I hope you'll give it a go!

Although the app isn't Open Source, it source code is available on GitHub. Feel free to build, modify and run it yourself, but please don't redistribute it.

P.S. the app is currently on Product Hunt so if you have an account there, feel free to leave a comment!

Announcing setup-racket

Sun, 3 Nov 2019 10:00:00 +0200

GitHub Actions are going to become generally-available next week so I created an action for installing Racket. You can find it on the marketplace. Here's what a minimal CI configuration for a Racket package might look like:

on: [push, pull_request]
name: CI
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
      - uses: Bogdanp/setup-racket@v0.1
        with:
          architecture: x64
          distribution: full
          variant: regular
          version: 7.4
      - run: raco test --drdr my-package-test/

And a more involved one using a test matrix:

on: [push, pull_request]
name: CI
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        racket-version: ["7.3", "7.4"]
        racket-variant: ["regular", "CS"]
    name: "CI using Racket ${{ matrix.racket-version }} (${{ matrix.racket-variant }})"
    steps:
      - uses: actions/checkout@master
      - uses: Bogdanp/setup-racket@v0.1
        with:
          architecture: x64
          distribution: full
          variant: ${{ matrix.racket-variant }}
          version: ${{ matrix.racket-version }}
      - run: raco test --drdr my-package-test/

The above configuration will cause the tests to be run against regular Racket 7.3 and 7.4 as well as Racket-on-Chez 7.3 and 7.4.

One limitation right now is that the Action only supports installing Racket on Linux targets, but I can update it to support Windows and macOS if there is interest.

Check it out an let me know what you think!

Announcing nemea

Thu, 31 Oct 2019 11:00:00 +0200

I just open sourced one of the very first Racket code bases I've worked on. The project is called nemea and it's a tiny, privacy-preserving, website analytics tracker. It doesn't do anything fancy, but it does enough for my needs and, possibly, yours too so check it out!

Announcing redis-rkt

Sun, 6 Oct 2019 19:00:00 +0300

Another Racket thing! redis-rkt is a new Redis client for Racket that I've been working on these past few weeks. Compared to the existing redis and rackdis packages, it:

is fully documented,
is safer due to strict use of contracts,
is faster,
supports more commands and
its API tries to be idiomatic, rather than being just a thin wrapper around Redis commands.

Check it out!

Announcing chief

Fri, 20 Sep 2019 20:00:00 +0300

I made a new Racket thing today! chief is a port of a subset of foreman's functionality to Racket. It lets you run sets of processes together based on a Procfile. If that sounds useful to you, do check it out!

Generators from Scratch

Thu, 5 Sep 2019 17:00:00 +0300

Generators in Python

One of the nice things about Python is that it comes with in-built support for "generators", functions that can suspend themselves and be resumed in the middle of processing. Here's a generator that produces the fibonacci series:

def fib():
    x = 0
    y = 1
    while True:
        y, x = x + y, y
        yield x

A generator is instantiated every time you call the fib function. Once you have a generator object, you can run it until the next time it suspends itself by passing it to the next function:

>>> fibber = fib()
>>> next(fibber)
1
>>> next(fibber)
1
>>> next(fibber)
2
>>> next(fibber)
3
>>> next(fibber)
5
>>> next(fibber)
8
>>> next(fibber)
13

The very first time we call next on fib, all of the code within it runs and gets suspended at the yield, returning the first value. Every time we call next on it after that, it resumes execution after the yield, which happens to be in a while loop, so execution jumps to the top of the loop. x and y are modified and the function yields again, suspending itself and returning the new value for y.

This can go on and on until the machine runs out of memory.

Another property of generators is that you can send values into them every time they are resumed. Here's a generator that computes the sum of two numbers sent from the outside:

def add():
    x = yield "expecting x"
    y = yield "expecting y"
    return x + y

When you run this generator, execution first suspends before x is assigned and then it suspends again before y is assigned.

>>> adder = add()
>>> adder.send(None)
'expecting x'
>>> adder.send(1)
'expecting y'
>>> adder.send(2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration: 3

Think of the send method as a lower level way of resuming a generator than next, with the added benefit that it allows you to send values into the generator (whereas next always sends None).

Above, I first call send with None, which signals to the generator that it can start executing. Then I send it the value that should be assigned to x, followed by the value for y. Once it hits the return statement, the generator raises StopIteration with the returned value.

Generators in Racket

Racket's standard library also comes with built-in support for generators via the racket/generator module. But what if it didn't? Could we implement Python-like generators in the language itself? The answer, of course, is "yes."

Racket comes with an even more general mechanism for capturing and resuming computation, called continuations. Continuations let you capture the execution of a program at a particular point as a function. When applied, that function will cause the program flow to jump back to the point it refers to. Here's an example:

(+ 1
   (call/cc
    (lambda (k)
      (displayln "before k is applied")
      (k 2)
      (displayln "after k is applied")))
   3)

call/cc captures the current continuation and passes it to another function -- in this case the lambda we created -- and, as mentioned above, when the continuation, k, is applied the program will jump to the spot that call/cc was itself applied at, effectively "return"ing from it with the value passed into the continuation.

So the above program:

evaluates 1,
evaluates the call/cc, reaching the line where k is applied,
jumps back to the point where it started evaluating the call/cc,
replaces the entire call/cc block with the value passed to k,
evaluates the 3, and, finally,
evaluates (+ 1 2 3).

The (displayln "after k is applied") expression is never evaluated by this program.

Here's another example, where we escape an infinite loop by way of a continuation:

(call/cc
 (lambda (k)
   (define (loop i)
     (when (= i 5)
       (k i))

     (loop (add1 i)))

   (loop 0)))

Here we:

evaluate the call/cc and capture the current continuation, k
recursively loop until i is 5, when we apply k and
jump back to the point where call/cc was called and replace it with the value passed to k, which is 5.

Here's the same program with k renamed to return:

(call/cc
 (lambda (return)
   (define (loop i)
     (when (= i 5)
       (return i))

     (loop (add1 i)))

   (loop 0)))

Does that look familiar?

Delimited Continuations

To implement our generators, we're going to use a more general type of continuations called "delimited continuations." These types of continuations allow us to install "prompts", continuation frames associated with specific prompt tags, and then later on in the execution of the program, abort to the nearest enclosing prompt with a particular tag without needing to have a reference to the captured continuation.

Here's an example where I install a prompt and then abort to it:

(define the-tag
  (make-continuation-prompt-tag))

(+ 1
   (call-with-continuation-prompt
    (lambda ()
      (displayln "before abort")
      (abort-current-continuation the-tag 2)
      (displayln "after abort"))
    the-tag
    (lambda (x)
      (displayln "received x")
      x))
   3)

When abort-current-continuation is applied, execution jumps to the abort handler (the third argument to call-with-continuation-prompt) and then returns from the point where call-with-continuation-prompt is applied. So the above program prints

before abort
received x
6

That shared prompt tag is how the abort function knows which continuation prompt it should jump to.

With this in mind, we can define a prompt tag for our continuations:

(define yield-tag
  (make-continuation-prompt-tag))

Followed by the yield function:

(define current-yielder
  (make-parameter
   (lambda _
     (error 'yield "may only be called within a generator"))))

(define (yield . args)
  (apply (current-yielder) args))

yield looks up the value of the current yielder and then applies its arguments to that function. The default value of current-yielder raises an error so that yield will fail if applied outside of a generator.

Next, we implement the generator function itself:

(define (generator proc)
  (lambda _
    (define cont proc)

    (define (next . args)
      (call-with-continuation-prompt
       (lambda _
         (parameterize ([current-yielder yield])
           (begin0 (apply cont args)
             (set! cont (lambda _
                          (error 'generator "exhausted"))))))
       yield-tag))

    (define (yield . args)
      (call-with-current-continuation
       (lambda (k)
         (set! cont k)
         (abort-current-continuation yield-tag (lambda _
                                                 (apply values args))))
       yield-tag))

    next))

This function takes another function, proc, as an argument and returns a function that will create a generator "instance" (really, it's just another function!) when applied.

Let's break down what happens inside the generator "factory" that is created. First, the original function is assigned to cont. Next, an inner function called next is defined.

When applied, next installs a new continuation prompt with the yield-tag. Inside the extent of the prompt, next then sets the current yielder to yield, which we'll talk about in a second, and returns the value of applying cont to its arguments.

Next, the yield function that next references is defined. When applied, it captures its own continuation (the continuation of the place that it itself was applied at), it then updates cont to point to that continuation and, finally, it aborts to the nearest prompt with the yield-tag (i.e. the place where the next function was applied), passing a function that returns yield's arguments to the continuation prompt handler that call-with-continuation-prompt installed. Because we didn't pass a handler function to call-with-continuation-prompt, the default handler, which applies whatever argument it's given, is used.

Finally, the next function is returned from the "factory."

And that's it! That's pretty much all you need to do to get a working implementation of generators.

Here's the definition of our fib generator built using this function:

(define fib
  (generator
   (lambda ()
     (let loop ([x 1]
                [y 1])
       (yield x)
       (loop y (+ x y))))))

and here it is in use:

> (define fibber (fib))
> (fibber)
1
> (fibber)
1
> (fibber)
2
> (fibber)
3
> (fibber)
5
> (fibber)
8
> (fibber)
13

And here's the add generator:

(define add
  (generator
   (lambda ()
     (+ (yield "expecting x")
        (yield "expecting y")))))

in action:

> (define adder (add))
> (adder)
"expecting x"
> (adder 1)
"expecting y"
> (adder 2)
3

Pretty cool, eh?

Here's a video of me stepping through the fib example with a debugger:

P.S. This is actually pretty close to how the generator implementation in Racket itself works. Feel free to check that out, the main differences between the two stem from some additional error handling that the core implementation does.

Racket for e-commerce

Tue, 20 Aug 2019 21:00:00 +0300

I had originally shared a version of this post with a small, private mailing list, but then I figured there'd be no harm in sharing it with a larger audience so here it is.

My girlfriend and I recently launched matchacha.ro, a small e-commerce site selling Japanese green tea here in Romania (the site defaults to Romanian, but you can change the language by scrolling to the bottom -- we don't ship outside of Romania, though, sorry!) and I figured I'd write a little experience report for this group.

I'll get the obvious stuff out of the way first: building the website from scratch was by far the easiest part. Finding suppliers, getting all the documentation right to be able to import the tea from Japan -- nothing quite like Romanian bureaucracy to make you want to give up on everything --, finding users and convincing them to buy the product have all been significantly harder.

That said, as someone who wrote his first line of Racket almost exactly a year ago, I have to say I found the whole process of actually building the application delightful and only very rarely frustrating (I'll get to that!).

My reasoning for building from scratch was threefold:

I wanted to avoid being locked in to VC-backed SaaS so that excluded most of the large players, like Shopify and BigCommerce,
I wanted to avoid anything written in PHP, and
I wanted to have some fun.

The app is a server-rendered "classic" web application with minimal JS and it does most of the things you would expect from an e-commerce app. There are products, product variants, collections of products, shopping carts, invoices that are generated on the fly, user accounts (hidden right now), blog posts and dynamic pages, promo codes, affiliate links, credit card payments, and a pretty powerful administration panel. It is backed by a Postgres database and it comes in at a neat ~10k cloc of Racket code (including unit and end-to-end tests):

$ cloc matchacha/ matchacha-tests/ migrations/ resources/css/ resources/js/
     295 text files.
     294 unique files.
      88 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.18 s (1182.1 files/s, 102674.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Racket                          87           1924            136          10374
Sass                            35            626              3           2928
JavaScript                      20            142             29            646
SQL                             65            145            454            573
-------------------------------------------------------------------------------
SUM:                           207           2837            622          14521
-------------------------------------------------------------------------------

That number does not include all of the supporting libraries that I've written, which come in at about another ~13k cloc including unit tests.

Speaking of libraries, here are all the ones the application currently depends on:

base
component-lib
crypto-lib
db-lib
deta-lib
forms-lib
gregor-lib
koyo-lib
koyo-sentry
marionette-lib
mobilpay
net-cookies-lib
postmark-client
rackunit-lib
sql
srfi-lite-lib
struct-define
struct-plus-plus
threading-lib
twilio
web-server-lib

That doesn't include transitive dependencies, because I don't know how to list them all, but I want to thank everyone who has worked on any of these and on any of these libraries' own dependencies.

The deployment process is fairly simple. GitHub Actions picks up all commits to master, runs their tests, and, on success, for commits that are tagged a certain way, packages up a distribution then ships it up to my dedicated server, unzips it, updates a symlink and restarts a service. Deployments are not zero-downtime at this point and I'm planning to switch to an AB-style deployment model once the site has enough traffic to warrant it, but a second of downtime during each deployment is fine for now. Rolling back to an earlier version means updating the aforementioned symlink and restarting the service.

The ability to package up the application so that it can be distributed easily is one of my favorite things about Racket. This is a distinguishing feature that most of the dynamic languages that I have used either lack entirely or it's not nearly as well integrated as one would like. With koyo, all it takes is

$ raco koyo dist

and all that really does under the hood is shell out to raco exe and then raco dist.

Production runtime errors (of which there has been exactly one so far, triggered by me, not one of my users) are sent to the SaaS version of Sentry. The low runtime error rate is due to the amount of stuff that Racket manages to catch at compile time compared to other dynamic languages and to some fairly extensive automated testing.

I use rackunit for all my tests. It gets the job done, but I find it's not quite as good as, say, Python's pytest. My main complaint is I like to organize tests in suites, but those aren't run automatically, so in each of my test modules I have to call (run-tests some-suite) within a test submodule, which splits up the reporting that raco test generates. If I centralize things so that I have a single run-all-tests.rkt file, then I can't leverage the -j parameter to raco test, meaning my tests run significantly slower. Apart from that, I would like to be able to run a particular test suite or test case from the command line, but I haven't been able to figure out how to do that yet. In addition, it would be nice to be able to tell raco test that I want to list the top n slowest tests along with their runtimes, which is something that pytest can easily do.

For end-to-end and screenshot tests, I use my own library, marionette, together with rackunit to control a live Firefox instance and make assertions. Some of those assertions generate screenshots of a web page and compare them with old versions committed to the repo by shelling out to ImageMagick.

Error reporting during test runs or when I'm manually testing stuff is the most frustrating part of working with Racket. I run the tests as well as the app (in local development mode) with errortrace turned on, but I still find myself often scratching my head trying to figure out where certain errors originate. This is an area where Racket really needs to improve. It absolutely has to be able to show the programmer where every error originated and that "where" can't be "somewhere inside this 20-line-long for-loop in file x.rkt, good luck!"

racket-mode for Emacs is a joy. I can easily spin up a REPL for any module that I'm working on and tinker with it, with some pretty good facilities for cleaning up imports, running tests and a macro stepper.

Continuations are fantastic for rapid development. I use them for small things like increasing/decreasing item quantities in the shopping cart and preserving state between the steps of the checkout process, but I will eventually be moving off them as the site grows because, from the user's perspective, it's bad UX to have URLs expire randomly just because a deployment happened to occur when you were browsing the site. At this stage, though, that's not a problem, and continuation expiration is handled gracefully: the user is redirected to the original page they were on before clicking the continuation link (or submitting the form, etc.) and shown a flash message.

If you visited the website (and assuming you live in Europe), you may have noticed how fast it is. Part of that is because of how fast Racket itself is, and part of it is me being judicious about what code runs on each page. koyo has an instrumenting profiler and, in development mode, all pages render a little floating widget that I can expand to show a trace of all the instrumented places (inspired by MiniProfiler).

I didn't find the lack of libraries to be a problem -- I just wrote whatever it was I needed at the time -- and the documentation for the stuff that does exist is pretty good. Documentation for Racket itself tends to be great, although a bit "stuffy". That's a challenge for someone who, like me, tends to skim a lot. I would like to see more code examples intended for people in a hurry.

One area where I didn't build my own thing was I18n/L10n. For that, the app relies on SRFI-29. I would have preferred to use the more standard tooling, like gettext, but there's no library for that. Yet.

The community is welcoming and thoughtful. Although I don't participate frequently, I enjoy reading racket-users, racket-dev, the subreddit and the Slack channel. I've learned a ton by reading blog posts by Alexis King, Alex Harsanyi, Greg Hendershott, and others. I love it whenever a new issue of Racket News comes out.

In terms of development time, this was all spread out across the past year, but I estimate I only spent about a full "work" month on the application itself and about another one working on all of those supporting libraries I mentioned. I think that's pretty good, given all the things the application does.

Overall, I'm really happy with my choice to use Racket so far!

Announcing deta

Tue, 16 Jul 2019 10:00:00 +0300

I just released the first version of deta, a Racket library for mapping between database tables and structs. A bit like Elixir's Ecto.

You can install it from the package server with:

raco pkg install deta

Check it out!

Announcing racket-sentry

Tue, 2 Jul 2019 10:00:00 +0300

I just released the first version of sentry, a Racket library that lets you capture exceptions using the Sentry API.

You can install it from the package server with:

raco pkg install sentry

And the docs should show up on the package server within the next couple of days. In the mean time, you can run

raco docs sentry

after installing the package to read its docs locally.

racket/gui saves the day

Mon, 17 Jun 2019 10:00:00 +0300

Yesterday, I bought an icon pack containing over 3,000 (!) SVG files and macOS utterly failed me when I tried to search the unarchived folder.

So I did what any self-respecting Racketeer would do. I used this as an excuse to play around with Racket's built-in GUI library!

Diving in

The Racket GUI toolkit is a retained mode style API for building user interfaces, meaning that you work with it by instantiating objects that represent the things that ought to be drawn on your screen and then the system draws them for you and, in order to add custom behavior when the user interacts with those objects, you register callbacks that react to certain events.

To render a window on the screen, all you have to do is:

#lang racket/gui

(require racket/class)

(define window
  (new frame%
       [label "Hello World!"]))

(send window show #t)

The above code instantiates a new frame object -- whose label (title) is "Hello World!" -- and then tells it to make itself visible. Easy as that!

`racket/class` crash course

The GUI library is built on top of Racket's class system. All you need to know to make use of it in this context is:

class names conventionally have a % suffix,
interface names conventionally have a <%> suffix,
you instantiate a class by using the new macro, giving it the name of the class you want to instantiate followed by zero or more field values and
you send objects messages using the send macro.

Laying things out

With the above in mind, we can go ahead and lay out our interface:

#lang racket/gui

(require racket/class)

(define window
  (new frame%
       [label "Icon Viewer"]
       [width 800]
       [height 600]))

(define panel
  (new vertical-panel%
       [parent window]))

(define search-box
  (new text-field%
       [parent panel]
       [label #f]))

(define list-box
  (new list-box%
       [parent panel]
       [choices empty]
       [label #f]))

(define canvas
  (new canvas%
       [parent panel]))

(send window show #t)

Assigning a parent to a widget ensures that said widget renders within said object. So, above, we have defined the following hierarchy:

window
└── panel
    ├── search-box
    ├── list-box
    └── canvas

The panel lays out its children underneath one another in a single column and its children should be fairly self-explanatory:

search-box is where we're going to type our search filters,
list-box is where we're going to list the filtered files and
canvas is where we're going to draw the selected file.

If you run the above code, then you should get a UI that's nearly identical to the one I showed at the beginning of this article.

Adding behavior

Despite looking like the screenshot, the code above doesn't implement any of the behavior of the final product yet. So let's add that!

You may have noticed that the list-box% class takes a choices field. Let's start there.

At the top of the file, we can gather all of the names of the SVG files in the current directory into a list:

(define folder-path
  (current-directory))

(define filenames
  (for/list ([filename (directory-list folder-path)]
             #:when (equal? (path-get-extension filename) #".svg"))
    (path->string filename)))

And then we can pass that list to the list-box when we instantiate it via the aforementioned choices field:

(define list-box
  (new list-box%
       [parent panel]
       [choices filenames]
       [label #f]))

Run the code from a folder that contains SVG files and you should now see those files being listed in the UI.

On to filtering. When someone enters text into the search-box, we want the list of filenames to be narrowed down to only those filenames that contain the search string. To do this, we can give the search-box a callback that it should execute whenever its contents change:

(define search-box
  (new text-field%
       [parent panel]
       [label #f]
       [callback (lambda (sb e)
                   (define text (send sb get-value))

                   (send list-box clear)
                   (for ([filename filenames]
                         #:when (string-contains? filename text))
                     (send list-box append filename)))]))

The callback takes the search-box object itself and an object representing the event that occurred as arguments. It then extracts the text from the search-box, empties out the list-box and only adds back in those filenames which contain the search string.

Finally, it's time to display the selected SVGs. For that, we're going to need the rsvg library. To install it, you can run:

raco pkg install rsvg

It relies on librsvg so you're going to need to have that available as well. On macOS, you can install it using homebrew:

brew install librsvg

To hook things up, we require rsvg then add a callback to the list-box so that, when an item is clicked, we can read the file from disk and display it on the canvas:


(require rsvg)

;; ...

(define list-box
  (new list-box%
       [parent panel]
       [label #f]
       [choices filenames]
       [callback (lambda (tb e)
                   (define selection (send tb get-string-selection))
                   (define filename (and selection (build-path folder-path selection)))
                   (when filename
                     (define svg (load-svg-from-file filename 3))
                     (define dc (send canvas get-dc))
                     (send dc clear)
                     (send dc draw-bitmap svg 0 0)))]))

And that's it! At this point, you should have a working icon viewer. Not bad for about 50 lines of code.

The final version is a little longer because I added support for debouncing and copying selected files elsewhere by double clicking on them, but it still clocks in at only about 100 lines of code!

You can find the final version of the code here.

Announcing marionette

Sat, 8 Jun 2019 15:45:00 +0300

I just released the first version of marionette (named after the protocol it implements), a Racket library that lets you remotely control the Firefox web browser. Think "puppeteer", but for Firefox.

You can install it from the package server with:

raco pkg install marionette

And the docs should show up on the package server within the next couple of days. In the mean time, you can run

raco docs marionette

after installing the package to read its docs locally.

It's still early days, but the library already supports most of the basic things you would expect it to. If you try it out, let me know what you think!

Using GitHub Actions to Test Racket Code

Wed, 1 May 2019 17:00:00 +0300

This article is outdated as of 2020/05/05 because it refers to the previous implementation of GitHub Actions. I've put together a revised version at Using GitHub Actions to Test Racket Code (Revised) so you should read that instead.

Like Alex Harsányi, I've been looking for a good, free-as-in-beer, alternative to Travis CI. For now, I've settled on GitHub Actions because using them is straightforward and because I saves me from creating yet another account with some other company.

GitHub Actions revolves around the concept of "workflows" and "actions". Actions execute arbitrary Docker containers on top of a checked-out repository and workflows describe which actions need to be executed when a particular event occurs. During the execution of a workflow, all actions, including ones running in parallel, share the same physical workspace. All of this stuff is declaratively specified using HCL.

Here's an example workflow:

workflow "demo" {
  on = "push"
  resolves = ["echo"]
}

action "make-a" {
  uses = "docker://alpine"
  runs = ["sh", "-c", "echo Hello > $GITHUB_WORKSPACE/a"]
}

action "make-b" {
  uses = "docker://alpine"
  runs = ["sh", "-c", "echo World > $GITHUB_WORKSPACE/b"]
}

action "echo" {
  needs = ["make-a", "make-b"]
  uses = "docker://alpine"
  runs = ["sh", "-c", "cat $GITHUB_WORKSPACE/a $GITHUB_WORKSPACE/b"]
}

This is saying that there is a workflow called "demo" that is executed whenever anything is pushed to the repository. That workflow's goal is to execute the "echo" action, which happens to depend on actions "make-a" and "make-b". When the workflow is triggered, those two actions are run first and the "echo" action only runs after they both succeed. In this particular example, each of the actions runs a shell command on top of the alpine docker image, but I could've picked any other image from Docker Hub or any existing GitHub Action repository. This page describes all of the various workflow configuration options.

If you save the above config in a file called .github/main.workflow in any GitHub repository and visit the "Actions" tab, then -- assuming you have access to the GitHub Actions beta -- you should see the pipeline execute almost immediately and output:

Hello
World

We can leverage all of this to test a Racket package is by using Jack Firth's racket image (or you could roll your own and host it on Docker Hub yourself):

workflow "main" {
  on = "push"
  resolves = ["test"]
}

action "test" {
  uses = "docker://jackfirth/racket:7.2"
  runs = "/github/workspace/ci/test.sh"
}

Here are the contents of ci/test.sh:

#!/usr/bin/env bash

set -euo pipefail

pushd "$GITHUB_WORKSPACE"
raco pkg install --batch --auto testpackage/
raco test testpackage/

The script sets the working directory to $GITHUB_WORKSPACE, installs all of testpackage's (a hypothetical package for the purposes of this article) dependencies and then runs its tests.

That's all there is to it! With about 12 LOC we've put together a basic workflow that tests a package on every commit.

Gotchas & Limitations

Like I mentioned before, all actions in a workflow share the same workspace so it's possible for actions to clash with one another when they operate on the filesystem. That's something you have to keep in mind when designing your workflows.

Workflow and action names share a namespace. If you call your workflow "test" and your action "test", you'll get an error saying the workflow is invalid, but it won't point out exactly why.

Finally, there's no built-in support for notifications. When builds fail, you won't notice unless you visit the Actions tab. For one of my non-OSS projects, I've set up a Telegram bot that I can use to notify a particular channel whenever builds succeed or fail. That works, but it's fairly ugly because there doesn't seem to be any way to conditionally execute actions (i.e. things like "if action a fails then run action b"), so I had to bundle the notification-handling code into each of my action scripts.

The Problem with SSH Agent Forwarding

Fri, 12 Apr 2019 14:00:00 +0300

After hacking the matrix.org website today, the attacker opened a series of GitHub issues mentioning the flaws he discovered. In one of those issues, he mentions that "complete compromise could have been avoided if developers were prohibited from using [SSH agent forwarding]."

Here's what man ssh_config has to say about ForwardAgent:

Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's Unix-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.

Simply put: if your jump box is compromised and you use SSH agent forwarding to connect to another machine through it, then you risk also compromising the target machine!

Instead, you should use either ProxyCommand or ProxyJump (added in OpenSSH 7.3). That way, ssh will forward the TCP connection to the target host via the jump box and the actual connection will be made on your workstation. If someone on the jump box tries to MITM your connection, then you will be warned by ssh.

Here's what a config file that uses ProxyCommand might look like:

Host bastion
  HostName bastion.example.com

Host target
  HostName target.example.internal
  Port 2222
  ProxyCommand ssh -W %h:%p bastion

And here's the equivalent config using ProxyJump:

Host bastion
  HostName bastion.example.com

Host target
  HostName target.example.internal
  Port 2222
  ProxyJump bastion

Of course, you can also specify either option via the command line:

ssh -o 'ProxyJump=bastion.example.com' target.example.internal -p 2222

Continuations for Web Development

Sun, 7 Apr 2019 17:00:00 +0300

One of the distinguishing features of Racket's built-in web-server is that it supports the use of continuations in a web context. This is a feature I've only ever seen in Smalltalk's Seaside before, though Racket's version is more powerful.

I've been leveraging continuations in an e-commerce application I've been building these past couple of months and I wanted to write down my thoughts. Let's dive right in.

First, an Example

#lang web-server/insta

(define (render-counter counter)
  (send/suspend/dispatch
   (lambda (embed/url)
     (response/xexpr
      `(div
        (h1 "Current count: " ,(number->string counter))
        (a ((href ,(embed/url (lambda (req)
                                (render-counter (+ counter 1))))))
           "Increment"))))))

(define (start req)
  (render-counter 0))

Note: I'm using #lang web-server/insta here, which I realize may be off-putting to some readers (I know it rubbed me the wrong way when I was first reading about web development in Racket). There are more familiar (less magical) ways to implement web servers in Racket than this, but this is the most succinct.

The entry point for this server (the start function) calls render-counter with a starting value of 0 for every new request that comes in. render-counter then demarcates the start of the continuation with the call to send/suspend/dispatch and, finally, it renders some HTML that displays the current value of the counter as well as a link that, when clicked, will recursively render-counter with an incremented counter value.

Notice how naturally this code flows and the fact that the continuation (the anonymous function that is passed to embed/url) is able to reference bindings in the scope of its parent.

Here's what that short piece of code gets you:

I think this is cool as hell. In essence, continuations let you write code that manipulates objects (a counter, a shopping a cart, a form, etc.) local to the current web page without having to do duplicate work -- image a shopping cart where you only retrieve a product from the database when you render the product page, but you close over the product when adding it to the cart -- and without having to give said code an explicit route. The latter is both a strength and a weakness, as I argue below.

Then, the Bad

As with most things, continuations on the web come with a set of trade-offs.

Continuations are local to a Racket process, meaning that any web server that leverages them is stateful. That's not inherently a bad thing, but it does mean that you have to be careful when it comes to load balancing and deploying new versions of your application: all new deployments invalidate all existing continuations and you have to rely on session affinity (cookie each user and use that to determine which server they connect to) to tie individual users to particular instances of the web server.

Racket's particular implementation of continuations stores the continuation id as a parameter in the URI, meaning that if someone guesses the id of your continuation then they can effectively steal your session. This is fairly easy to work around by leveraging dynamic binding in racket:

for each request, read the continuation security token from the user agent; if it doesn't exist then generate a large unique value and store that in a cookie on the user agent,
dynamically bind the current continuation security token for the request,
before executing each continuation, ensure that the value of the user's continuation security token cookie is the same as the value of the current continuation security token parameter; if the value is different then return a 403 Forbidden or similar response.

Because of how the continuation machinery works, each continuation will "remember" the continuation security token of the request that created it, ensuring that each continuation is tied to the browser session that it was created by. You can find an implementation of this pattern in my racket-webapp-template project. All in all, it takes less than a hundred lines of code to implement.

Because of the session-stealing issue and the fact that continuation URLs are fairly ugly, they're not a good match for URLs that should be shareable between users (or for SEO, for that matter). I tend to limit my use of continuations to "actions" that a user may perform on an object local to the current page (eg. adding an item to the shopping cart, or increasing the quantity of said item in the cart, etc.).

Finally, because continuations close over local scope, any objects captured are going to live for the duration that the continuation exists. So if you're not careful, then you may end up leaking memory. To combat this and to prevent servers' memory usage from growing indefinitely, the web-server library has a robust implementation of an LRU continuation manager that expires continuations quicker the more memory pressure there is. In addition, you can write your own manager implementation to suit your application if the built-in ones don't cut it.

And a Conclusion

That may seem like a lot of "bad", but all of those points are straightforward tradeoffs that I believe are worth it in the long run given the ergonomics that continuations on the web buy you. Plus, writing code in this way is just so. much. fun!

Google Groups Without Google Accounts

Tue, 5 Feb 2019 20:00:00 +0200

It turns out that when you delete your Google accounts, Google unsubscribes you from any (public and private) Google Groups you're a member of. I found out about this only because my inbox traffic these past couple of days felt unusually light so I went and looked at racket-users and, lo and behold, there were a bunch of new posts I hadn't received.

I get it. They want to avoid sending emails to an address that, from their perspective, no longer exists. Makes sense, although they could certainly tell you that you're gonna be unsubscribed when you cancel, maybe even throw in a list of all the groups you're currently subscribed to for good measure.

What annoys me, though, is how hard they make it for someone who doesn't have a Google account to join a Google Group. If you visit a group and you're not logged in, there's no way for you to join via the UI. This makes it seem like you can't join unless you create an account, but that's not the case! You can still join groups by sending an e-mail to group-name+subscribe@googlegroups.com. For example, to join the racket-users group, you would send an e-mail to racket-users+subscribe@googlegroups.com. If the group you want to join has spaces in its name, replace the spaces with dashes in the e-mail address.

If you were wondering about how to join a group without a Google account, now you know! And if you're considering starting a new mailing list, I urge you to take this factor into consideration and maybe use something like groups.io or lists.sr.ht instead.

Bye, Bye, Google

Mon, 4 Feb 2019 09:00:00 +0200

I spent this past weekend de-Google-ifying my life and, despite my expectations, it wasn't too hard to do.

I started by moving all of my websites off of Google App Engine and onto a dedicated box that I had already owned. That was straightforward enough. Next, I removed any Google Analytics snippets from each of them and replaced those with my own analytics server that I had built a while back (it doesn't store any PII, only aggregate information (and very little of that, too)). Afterwards, I replaced any Google Fonts that I had been using with system fonts and, finally, I moved the screencasts for Dramatiq and molten from YouTube to peertube.

Next, it was time for e-mail. I had already been using isync to download all my mail so it was easy for me to switch my @defn.io and @cleartype.io e-mail addresses over to FastMail (who really deserve their name!). I created my accounts, updated my .mbsyncrc, ran a sync and successfully imported all my mail into the new accounts. So far so good! The problem was my @gmail.com account that I had been using since 2005. Most of my internet accounts were tied to that e-mail address. Thankfully, 1Password lets you filter your accounts by username and so I did that and, one by one, I updated all of my internet accounts to use my @defn.io e-mail address (and deleted a number of accounts in the process -- some companies make this process really annoying (I shit you not, one in particular made me go through about 20 dialogs before it finally let me delete the account)). That was a time-consuming, but satisfying, process; sorta like spring cleaning. In the end, I was able to switch about 130 internet accounts to use my new e-mail address after a few hours of work. There were only 5 instances where I could neither change my e-mail address or delete the account so, for all of those, I e-mailed support and I'm currently waiting to hear back.

I've debated deleting the @gmail.com e-mail address, but I think it's wiser to keep it and essentially squat the username lest someone else take it over and cause me trouble down the line. I've made it forward and delete any new mail it gets to @defn.io. As time goes on, the amount of incoming email to that address should tend toward 0.

Why go through all this trouble? I've grown increasingly concerned this past year with how much access Google has to our lives. They are the world's biggest advertising company and they have access to most of our web browsing via Google Chrome (62.5% market share -- although given the amount of broken websites (some explicitly Chrome-only!) I've found since switching to Firefox, I believe this number may actually be higher), all our website visitors via Google Analytics and Google Fonts. Much of our communication via GMail and Google Apps and much of the content we consume every day via YouTube. I'm not even going to get into all the information they gather from people who use Android phones. While I don't believe that folks working at Google are actively trying to do harm, I believe that, due to the sheer size of the company, no one is truly at the helm and this massive organism will tend toward maximizing profits at whatever expense so I've decided to do my best to support the smaller alternatives that are out there.

P.S. Here's a funny response I got to an account close e-mail from Kraken just as I was writing this post! They're offering me the privilege to keep my user account and receive their news.

We can offer you to keep your account with us instead of closing it. You'll get the newest updates about us/our services that you might be interested in.

Aren't I lucky?

Announcing north

Thu, 31 Jan 2019 07:00:00 +0200

A couple of days ago, I released north, a database schema migration tool written in Racket. It currently supports PostgreSQL and SQLite, but new adapters are extremely easy to add and my main reason for building it was because I wanted not only a CLI utility but also programmatic access to do migrations from Racket. I'm going to make that last part easy for everyone with the next release after I clean up some of the internals and write more docs.

If you do check it out, let me know what you think!

Announcing forms

Mon, 21 Jan 2019 21:00:00 +0200

Today marks the first public release of forms, a Racket library for web form validation. Racket's formlets module from the standard library already does something similar, but, unfortunately, it lacks any facilities for easily showing validation errors to end users which is a big part of what I want from this kind of library. Another nice thing about this new library is it'll be able to validate things other than forms -- like JSON -- soon!

I hope you check it out, and, if you do, let me know what you think!

Try Firefox

Mon, 17 Dec 2018 02:00:00 +0200

Since Microsoft officially announced that they will switch Edge's rendering engine to Chromium, many people have written about how this poses a danger to the future of the web. I'm not going to repeat those same arguments, as I feel others have done a good job of it.

What I want to do is urge you to try Firefox for a couple of days this week. That's it. Give it a try. More than the 15 minutes you gave it that one time. I bet you you'll find some things are better, some are worse; but you might realize you like it. You might also find that that thing you only ever tested in Chrome doesn't quite work as it should -- I found out my stock broker only supports Chrome -- and you might fix it and that might prompt you to test things in multiple browsers in the future and we just might avoid a future where the web is controlled by an ad-powered super-org. Or don't. It's up to you.

Advent of Racket 2018

Sat, 1 Dec 2018 07:45:00 +0200

I decided to do this year's Advent of Code in Racket and stream the whole thing. We'll see how far I make it (getting up this early is rough!), but so far I finished day one. The code is here and the playlist for the recordings is here.

If you want to get notified as soon as I jump on to start streaming, you can follow me on Twitch.

Announcing geoip

Thu, 29 Nov 2018 12:00:00 +0200

I released geoip today. It's a Racket library for working with MaxMind's geolocation databases. It's got a tiny API surface (3 functions!), but it should be useful to anyone needing to do geolocation in Racket. As always, check it out and let me know what you think!

BTW, I streamed the whole process on Twitch so, if that's your thing, you can check out the recordings here.

Announcing net-ip

Fri, 16 Nov 2018 13:00:00 +0200

I released net-ip -- a small Racket library for working with IP (v4 and v6) addresses and networks -- today. I needed this to be able to work on another library I'm going to release at some point in the future for doing geo-location based on Maxmind's databases. Check it out and let me know what you think!

Announcing component

Wed, 14 Nov 2018 21:10:00 +0200

I released component the other day. It's a Racket library for managing the lifecycle of stateful objects in long-running applications and for doing dependency injection. It was inspired by the Clojure library of the same name by Stuart Sierra. Check it out and let me know what you think!

P.S. Expect more Racket libraries from me in the coming weeks and months. I'm really enjoying the language so far!

Announcing cursive_re

Mon, 22 Oct 2018 12:04:55 +0300

I released cursive_re today. It's a tiny Python library made up of combinators that help you write regular expressions you can read and modify six months down the line.

Here's what it looks like:

>>> from cursive_re import *
>>> domain_name = one_or_more(any_of(in_range("a", "z") + in_range("0", "9") + text("-")))
>>> domain = domain_name + zero_or_more(text(".") + domain_name)
>>> path_segment = zero_or_more(none_of("/"))
>>> path = zero_or_more(text("/") + path_segment)
>>> url = (
...     group(one_or_more(any_of(in_range("a", "z"))), name="scheme") + text("://") +
...     group(domain, name="domain") +
...     group(path, name="path")
... )
>>> str(url)
"(?P<scheme>[a-z]+)://(?P<domain>[a-z0-9\-]+(?:\.[a-z0-9\-]+)*)(?P<path>(?:/[^/]*)*)"

Racket

Sun, 21 Oct 2018 20:00:00 +0300

I've been playing around with Racket every chance I got since early September of this year. This post is going to serve as a sort of experience report of my foray into Racket so far.

Things I Like

Editor Support

Greg Hendershott's racket-mode for emacs has been wonderful to work with. It provides syntax highlighting, REPL integration and auto-completion as well as a macro stepper. All of which are highly useful while working on Racket code.

Documentation

There seems to be a huge emphasis on documentation within the community, which is great. Everything is thoroughly documented and the documentation for installed packages is all built locally (with cross-references!) so you have access to it regardless of connectivity. And it's pretty! The Racket ecosystem has some of the most readable websites I've ever been to.

Consistency

The language is remarkably consistent, from the way modules and packages are organized to the way documentation is written.

Interactive Development

Despite primarily working with dynamic languages every day, I find that I don't use the REPL with those languages nearly as much or as tightly as I do with lisps. I don't know for sure why that is but it probably boils down to the nature of s-expressions: it's just so convenient to eval-last-sexp in lisps in a way that you can only approximate in a language like Python (and, believe me, I've tried!).

Contracts

Contracts are a way to specify and validate the boundaries between parts of a system. They are highly expressive (eg. in-range?) and composable and produce great error messages when the invariants they describe get broken. Unfortunately, they do incur a runtime cost and there's no way to disable them automatically for production, but they can be specified separately from the implementation of the things they describe such that you can add them to the functions and structures that your module exports and even have "unsafe" flavors of your modules from which you export versions of the same functions and structs without contracts for those times when performance really is an issue.

For a cool-but-unrelated-to-Racket talk on contracts, check out "Contracts For Getting More Programs Less Wrong" by Rob Simmons from this year's Strange Loop.

Modules

I really like the fact that you have to be explicit about what things your modules export. This reminds me a lot of ML-style languages and is great for encapsulation.

Things I Dislike

"Dislike" may be too strong a word for what I'm about to describe. Most of these things simply represent different tradeoffs to what I'm normally used to.

Error Reporting

This may just be because I'm not used to the error messages yet, but I find Racket's exception reporting hard to decipher and I often run into errors that contain no information regarding where in the source code the error occurred.

Lack of Docstrings

There's no built-in support for function docstrings and most of the code I've read seems to separate code and documentation. On the one hand this makes it so looking up a function's documentation via the REPL is not possible which is slightly annoying (though racket-mode provides racket-describe for this purpose), but, on the other hand, I like that there is a single documentation format that everyone agrees on (Scribble) and the prevalence of cross-references makes looking up a function's documentation easy enough.

Unit Testing

Racket's rackunit is lacking support for a few things I'd expect from such a library. There doesn't seem to be any built-in way to do set up and teardown (I've been using dynamic-wind for this purpose), the facilities for grouping tests together feel flimsy and I don't understand why test-suites have to be run manually.

Prometheus with Molten

Sun, 21 Oct 2018 03:00:00 +0300

molten has built-in support for exporting the following Prometheus metrics:

http_request_duration_seconds{method,path} -- a histogram of the request duration percentiles by request method and path,
http_requests_total{method,path,status} -- a counter of the total number of requests by method, path and status,
and http_requests_inprogress{method,path} -- a gauge of the number of requests in progress by method and path.

Let's say that you have a basic molten app. Something like this:

from molten import App, Route


def index():
    return {}

app = App(
    routes=[
        Route("/", index),
    ],
)

To start tracking metrics, add prometheus_middleware to your app's middlewares list:

from molten import App, ResponseRendererMiddleware, Route
from molten.contrib.prometheus import prometheus_middleware


def index():
    return {}

app = App(
    middleware=[
        prometheus_middleware,
        ResponseRendererMiddleware()
    ],
    routes=[
        Route("/", index),
    ],
)

The app will then begin keeping track of metrics in memory, but won't expose them anywhere. Prometheus uses a pull-based model to gather metrics from servers, which means you have tell it where (i.e. which servers/ports) to look for metrics and your server needs to be able to expose those metrics in a way that Prometheus can understand.

To start exposing metrics, add expose_metrics to your routes:

from molten import App, ResponseRendererMiddleware, Route
from molten.contrib.prometheus import expose_metrics, prometheus_middleware


def index():
    return {}

app = App(
    middleware=[
        prometheus_middleware,
        ResponseRendererMiddleware()
    ],
    routes=[
        Route("/", index),
        Route("/metrics", expose_metrics),
    ],
)

If you run your app now and visit the index handler a couple of times and then visit /metrics, you should see all the metrics that were gathered up until that point.

One caveat related to expose_metrics is it should only be used in single-process configurations. If your application uses multiple processes to server requests (for example, multiple gunicorn workers) then you should use expose_metrics_multiprocess instead.

Assuming you're running the app on localhost:8000, you can then use a scrape config such as this one to let Prometheus know where to look for metrics:

scrape_configs:
  - job_name: 'a-molten-app'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:8000']

Save that to a file called prometheus.yml, run prometheus and then visit http://localhost:9090 and you should be able to start querying the metrics I mentioned above. Check out Prometheus' querying documentation to find out how to construct queries.

While Prometheus' dashboard is nice for inspecting metrics, it's not meant for long-lived graphs and dashboards. For that, you should take a look at Grafana.

Announcing Molten

Sat, 23 Jun 2018 03:00:00 +0300

Today marks the first public release of molten, a modern API framework for Python I've been working on over the past couple of weeks.

Check it out and let me know what you think!

Web application from scratch, Part IV

Sat, 12 May 2018 03:00:00 +0300

This is the fourth post in my web app from scratch series. If you haven't read them yet, you should check out parts 1 through 3 first!

In this part we're going to cover building an Application abstraction. If you're following along at home, you can find the source code for part 3 here. Let's get to it!

Housekeeping

Type checking

So far we've used type annotations mostly as documentation and haven't worried about type checking our code. In commit 6fa54c4, I introduced mypy as a dev dependency and started type-checking the code using it. I won't describe the changes I had to make here because the process was mostly mechanical:

run mypy server.py,
make necessary changes,
repeat.

Unit testing

In commit 4455e04, I moved the modules from the repo root into a package called scratch and added a tests package at the top level.

The Application

Last time, we added support for mounting request handlers to specific paths and today we're going to build upon that work by writing an abstraction that can hold multiple request handlers and route to them based on the request path.

In scratch/application.py add the following:

from .request import Request
from .response import Response


class Application:
    def __call__(self, request: Request) -> Response:
        return Response("501 Not Implemented", content="Not Implemented")

If you remember from last time, we defined request handlers as functions that take a Request and return a Response. This means that our Applications are themselves going to be request handlers.

Let's remove the old server instantiation code from scratch/server.py -- delete everything from wrap_auth onward -- and create a new CLI entrypoint for our package in __main__.py:

import sys

from .application import Application
from .server import HTTPServer


def main() -> int:
    application = Application()

    server = HTTPServer()
    server.mount(application)
    server.serve_forever()
    return 0


if __name__ == "__main__":
    sys.exit(main())

We can now run our server from the repo root with python -m scratch and if we try to visit http://127.0.0.1:9000, we should get a 501 response back.

The Router

Each application is going to contain an instance of a Router. The router's responsibility will be to map incoming request method, path pairs like POST /users or GET /users/{user_id} to request handlers. Here's what it looks like (in scratch/application.py):

import re
from collections import OrderedDict, defaultdict
from functools import partial
from typing import Callable, Dict, Optional, Pattern, Set, Tuple

from .request import Request
from .response import Response
from .server import HandlerT

RouteT = Tuple[Pattern[str], HandlerT]
RoutesT = Dict[str, Dict[str, RouteT]]
RouteHandlerT = Callable[..., Response]


class Router:
    def __init__(self) -> None:
        self.routes_by_method: RoutesT = defaultdict(OrderedDict)
        self.route_names: Set[str] = set()

    def add_route(self, name: str, method: str, path: str, handler: RouteHandlerT) -> None:
        assert path.startswith("/"), "paths must start with '/'"
        if name in self.route_names:
            raise ValueError(f"A route named {name} already exists.")

        route_template = ""
        for segment in path.split("/")[1:]:
            if segment.startswith("{") and segment.endswith("}"):
                segment_name = segment[1:-1]
                route_template += f"/(?P<{segment_name}>[^/]+)"
            else:
                route_template += f"/{segment}"

        route_re = re.compile(f"^{route_template}$")
        self.routes_by_method[method][name] = route_re, handler
        self.route_names.add(name)

    def lookup(self, method: str, path: str) -> Optional[HandlerT]:
        for route_re, handler in self.routes_by_method[method].values():
            match = route_re.match(path)
            if match is not None:
                params = match.groupdict()
                return partial(handler, **params)
        return None

Its add_route method iterates over all the parts of the path it's given and generates a regular expression in the process, replacing all dynamic path segments with named capture groups in the regex ("/users/{user_id}" becomes "^/users/(?P<user_id>[^/]+)$").

When a path is looked up via its lookup method, it iterates over all of the available routes for that method and checks the path against the regex for matches. When it finds a match, it partially applies the dynamic capture groups (if any) to the handler function and then returns that value.

Hooking the router into our application class is pretty straightforward. We instantiate a router on application init, add a method to proxy adding routes and update our __call__ method to look up and execute handlers when a request comes in:

class Application:
    def __init__(self) -> None:
        self.router = Router()

    def add_route(self, method: str, path: str, handler: RouteHandlerT, name: Optional[str] = None) -> None:
        self.router.add_route(method, path, handler, name or handler.__name__)

    def __call__(self, request: Request) -> Response:
        handler = self.router.lookup(request.method, request.path)
        if handler is None:
            return Response("404 Not Found", content="Not Found")
        return handler(request)

As an added bit of sugar, we're also going to define a route decorator on the Application class:

    def route(
            self,
            path: str,
            method: str = "GET",
            name: Optional[str] = None,
    ) -> Callable[[RouteHandlerT], RouteHandlerT]:
        def decorator(handler: RouteHandlerT) -> RouteHandlerT:
            self.add_route(method, path, handler, name)
            return handler
        return decorator

With that all in place, we can go ahead and update our code in __main__ to register handlers for various routes.

import functools
import json
import sys
from typing import Callable, Tuple, Union

from .application import Application
from .request import Request
from .response import Response
from .server import HTTPServer

USERS = [
    {"id": 1, "name": "Jim"},
    {"id": 2, "name": "Bruce"},
    {"id": 3, "name": "Dick"},
]


def jsonresponse(handler: Callable[..., Union[dict, Tuple[str, dict]]]) -> Callable[..., Response]:
    @functools.wraps(handler)
    def wrapper(*args, **kwargs):
        result = handler(*args, **kwargs)
        if isinstance(result, tuple):
            status, result = result
        else:
            status, result = "200 OK", result

        response = Response(status=status)
        response.headers.add("content-type", "application/json")
        response.body.write(json.dumps(result).encode())
        return response
    return wrapper


app = Application()


@app.route("/users")
@jsonresponse
def get_users(request: Request) -> dict:
    return {"users": USERS}


@app.route("/users/{user_id}")
@jsonresponse
def get_user(request: Request, user_id: str) -> Union[dict, Tuple[str, dict]]:
    try:
        return {"user": USERS[int(user_id)]}
    except (IndexError, ValueError):
        return "404 Not Found", {"error": "Not found"}


def main() -> int:
    server = HTTPServer()
    server.mount("", app)
    server.serve_forever()
    return 0


if __name__ == "__main__":
    sys.exit(main())

jsonresponse is a little helper decorator that converts handler results to JSON responses. Other than that, everything is pretty straightforward: we create an application instance, register a couple route handlers and then mount that application inside our server. And, with that, we have a little JSON API for listing users.

Winding down

That's it for part 4. Next time we're going to cover extending the Request object to parse query strings and cookies as well as to be able to hold user-defined data per request. If you'd like to check out the full source code and follow along, you can find it here.

See ya next time!

Announcing dramatiq version 1.0!

Sat, 31 Mar 2018 01:00:00 +0300

Today marks the 1.0 release of dramatiq! With this release, the project has been re-licensed from AGPL to the much more permissive LGPL. Check it out and let me know what you think!

Announcing dramatiq_sqs

Tue, 27 Mar 2018 01:00:00 +0300

I just released dramatiq_sqs, an Amazon SQS broker for dramatiq. Check it out an let me know what you think!

Web application from scratch, Part III

Tue, 20 Mar 2018 02:00:00 +0200

This is the third post in my web app from scratch series. If you haven't read them yet, you can find the first part here and the second part here. You'll want to read them first.

This part is going to be short and sweet. We're going to cover request handlers and middleware. Here's the source for part 2 so you can follow along. Let's get to it!

Handlers

Last time we implemented all our request handling logic inside the HTTPWorker class. That's not an appropriate place for application logic to live in so we need to update that code to run arbitrary application code that it knows nothing about. To do this, we're going to introduce the concept of a request handler. In our case a request handler is going to be a function that takes in a Request object and returns a Response object. Expressed as a type, that looks like this:

HandlerT = Callable[[Request], Response]

Let's modify our HTTPServer so that it stores a set of request handlers, each one assigned to a particular path prefix so that we can host different applications at different paths. In HTTPServer's constructor, let's assign an empty list to the handlers instance variable.

class HTTPServer:
    def __init__(self, host="127.0.0.1", port=9000, worker_count=16) -> None:
        self.handlers = []
        ...

Next, let's add a method that we can use to add handlers to the handler list. Call it mount.

    def mount(self, path_prefix: str, handler: HandlerT) -> None:
        """Mount a request handler at a particular path.  Handler
        prefixes are tested in the order that they are added so the
        first match "wins".
        """
        self.handlers.append((path_prefix, handler))

Now we need to update the HTTPWorker class to take advantage of these handlers. We need to make the workers' constructor take the handlers list as a parameter.

class HTTPWorker(Thread):
    def __init__(self, connection_queue: Queue, handlers: List[Tuple[str, HandlerT]]) -> None:
        super().__init__(daemon=True)

        self.connection_queue = connection_queue
        self.handlers = handlers
        self.running = False

And then we need to update the handle_client method to delegate request handling to the handler functions. If none of the handlers match the current path, then we'll return a 404 and if one of the handlers raises an exception then we'll return a 500 error to the client.

    def handle_client(self, client_sock: socket.socket, client_addr: typing.Tuple[str, int]) -> None:
        with client_sock:
            try:
                request = Request.from_socket(client_sock)
            except Exception:
                LOGGER.warning("Failed to parse request.", exc_info=True)
                response = Response(status="400 Bad Request", content="Bad Request")
                response.send(client_sock)
                return

            # Force clients to send their request bodies on every
            # request rather than making the handlers deal with this.
            if "100-continue" in request.headers.get("expect", ""):
                response = Response(status="100 Continue")
                response.send(client_sock)

            for path_prefix, handler in self.handlers:
                if request.path.startswith(path_prefix):
                    try:
                        request = request._replace(path=request.path[len(path_prefix):])
                        response = handler(request)
                        response.send(client_sock)
                    except Exception as e:
                        LOGGER.exception("Unexpected error from handler %r.", handler)
                        response = Response(status="500 Internal Server Error", content="Internal Error")
                        response.send(client_sock)
                    finally:
                        break
            else:
                response = Response(status="404 Not Found", content="Not Found")
                response.send(client_sock)

Lastly, we have to make sure we pass the handler list to the HTTPWorkers when we instantiate them in serve_forever.

    def serve_forever(self) -> None:
        workers = []
        for _ in range(self.worker_count):
            worker = HTTPWorker(self.connection_queue, self.handlers)
            worker.start()
            workers.append(worker)

        ...

Now, whenever an HTTPWorker receives a new connection, it'll parse the request and try to find a request handler to process it with. Before the request is passed to a request handler, we remove the prefix from its path property so that request handlers don't have to be aware of what prefix they're running under. This'll come in handy when we write a handler that serves static files.

Since we haven't mounted any request handlers yet, our server will reply with a 404 to any incoming request.

~> curl -v 127.0.0.1:9000
* Rebuilt URL to: 127.0.0.1:9000/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9000 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:9000
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< content-length: 9
<
* Connection #0 to host 127.0.0.1 left intact
Not Found

Let's mount a request handler that always returns the same response.

def app(request: Request) -> Response:
  return Response(status="200 OK", content="Hello!")


server = HTTPServer()
server.mount("", app)
server.serve_forever()

Whatever path we visit now, we'll get the same Hello! response. Let's mount another handler to serve static files from a local folder. To do this, we're going to update our old serve_file function and turn it into a function that takes the path to some folder on disk and returns a request handler that can serve files from that folder.

def serve_static(server_root: str) -> HandlerT:
    """Generate a request handler that serves file off of disk
    relative to server_root.
    """

    def handler(request: Request) -> Response:
        path = request.path
        if request.path == "/":
            path = "/index.html"

        abspath = os.path.normpath(os.path.join(server_root, path.lstrip("/")))
        if not abspath.startswith(server_root):
            return Response(status="404 Not Found", content="Not Found")

        try:
            content_type, encoding = mimetypes.guess_type(abspath)
            if content_type is None:
                content_type = "application/octet-stream"

            if encoding is not None:
                content_type += f"; charset={encoding}"

            body_file = open(abspath, "rb")
            response = Response(status="200 OK", body=body_file)
            response.headers.add("content-type", content_type)
            return response
        except FileNotFoundError:
            return Response(status="404 Not Found", content="Not Found")

    return handler

Finally, we're going to call serve static and mount the result under "/static" before we mount our application handler.

server = HTTPServer()
server.mount("/static", serve_static("www")),
server.mount("", app)
server.serve_forever()

All requests that begin with "/static" will now be handled by the generated static file handler and everything else will be handled by the app handler.

Middleware

Given that our request handlers are plain functions that take a request and return a response, writing middleware -- arbitrary functionality that can run before or after every request -- is pretty straightforward: any function that takes a request handler as input and itself generates a request handler is a middleware.

Here's how we might write a middleware that ensures that all incoming requests have a valid Authorization header:

def wrap_auth(handler: HandlerT) -> HandlerT:
    def auth_handler(request: Request) -> Response:
        authorization = request.headers.get("authorization", "")
        if authorization.startswith("Bearer ") and authorization[len("Bearer "):] == "opensesame":
            return handler(request)
        return Response(status="403 Forbidden", content="Forbidden!")
    return auth_handler

To use it, we just pass it the app handler and mount the result.

server = HTTPServer()
server.mount("/static", serve_static("www")),
server.mount("", wrap_auth(app))
server.serve_forever()

Now all requests to the root handler will have to contain an authorization header with our super secret hard-coded value, otherwise they'll get back a 403 response.

Winding down

That's it for part 3. In part 4 we're going to cover extracting an Application abstraction and implementing request routing. If you'd like to check out the full source code and follow along, you can find it here.

See ya next time!

Web application from scratch, Part II

Sun, 4 Mar 2018 02:00:00 +0200

This is the second post in my web app from scratch series. If you haven't read it yet, you can find the first part here. You'll want to read that one first.

In this part we're going to cover improvements to the Request data structure, adding Response and Server abstractions and making the server able to serve concurrent requests.

Requests

The Request class we wrote last time was able to store the request method, its path and its headers. Let's first improve it by making it possible to store multiple values for a single header. To do that, we're going to define a class called Headers that acts as a mapping from case-insensitive header names to lists of header values.

from collections import defaultdict


class Headers:
    def __init__(self) -> None:
        self._headers = defaultdict(list)

    def add(self, name: str, value: str) -> None:
        self._headers[name.lower()].append(value)

    def get_all(self, name: str) -> typing.List[str]:
        return self._headers[name.lower()]

    def get(self, name: str, default: typing.Optional[str] = None) -> typing.Optional[str]:
        try:
            return self.get_all(name)[-1]
        except IndexError:
            return default

Pretty straightforward: each instance of the class has an underlying dict whose keys are lower-cased header names and whose values are lists of header values. If we plug Headers into our Request class now, it should look something like this:

class Request(typing.NamedTuple):
    method: str
    path: str
    headers: Headers

    @classmethod
    def from_socket(cls, sock: socket.socket) -> "Request":
        """Read and parse the request from a socket object.

        Raises:
          ValueError: When the request cannot be parsed.
        """
        lines = iter_lines(sock)

        try:
            request_line = next(lines).decode("ascii")
        except StopIteration:
            raise ValueError("Request line missing.")

        try:
            method, path, _ = request_line.split(" ")
        except ValueError:
            raise ValueError(f"Malformed request line {request_line!r}.")

        headers = Headers()
        for line in lines:
            try:
                name, _, value = line.decode("ascii").partition(":")
                headers.add(name, value.lstrip())
            except ValueError:
                raise ValueError(f"Malformed header line {line!r}.")

        return cls(method=method.upper(), path=path, headers=headers)

Next up, let's make it possible to read request bodies using our request class. Since reading the full request body on every request is potentially wasteful (and an attack vector!), we're going to define a BodyReader class that will behave as a read-only file object so that users of the Request class can decide when and how much data they want to read from the request body.

class BodyReader(io.IOBase):
    def __init__(self, sock: socket.socket, *, buff: bytes = b"", bufsize: int = 16_384) -> None:
        self._sock = sock
        self._buff = buff
        self._bufsize = bufsize

    def readable(self) -> bool:
        return True

    def read(self, n: int) -> bytes:
        """Read up to n number of bytes from the request body.
        """
        while len(self._buff) < n:
            data = self._sock.recv(self._bufsize)
            if not data:
                break

            self._buff += data

        res, self._buff = self._buff[:n], self._buff[n:]
        return res

The BodyReader wraps a socket and reads data in bufsize chunks into an in-memory buffer. To understand why its buffer can be pre-filled (the buff parameter in the constructor), lets take another look at the iter_lines function we defined last time:

def iter_lines(sock: socket.socket, bufsize: int = 16_384) -> typing.Generator[bytes, None, bytes]:
    """Given a socket, read all the individual CRLF-separated lines
    and yield each one until an empty one is found.  Returns the
    remainder after the empty line.
    """
    buff = b""
    while True:
        data = sock.recv(bufsize)
        if not data:
            return b""

        buff += data
        while True:
            try:
                i = buff.index(b"\r\n")
                line, buff = buff[:i], buff[i + 2:]
                if not line:
                    return buff

                yield line
            except IndexError:
                break

You can see that once it finds the marker for the end of the request headers (if not line:), it returns any additional data that it may have read. To give a concrete example, say a request contains 4KiB of header data and 100KiB of body data, since iter_lines reads data in chunks of 16KiB by default, it might read all the header data plus an additional 12KiB of body data in one call. The 4KiB of header data will be split into lines and yielded by the generator and the additional data that was read past the request headers will be returned from the generator. We'll use the returned data to pre-populate the RequestReader's internal buffer.

We're going to have to change our header parsing logic to use a while loop instead of a for loop so that we can capture the return value of the generator. Once we have the return value, we can construct a body reader and pass that to the Request constructor.

class Request(typing.NamedTuple):
    method: str
    path: str
    headers: Headers
    body: BodyReader

    @classmethod
    def from_socket(cls, sock: socket.socket) -> "Request":
        """Read and parse the request from a socket object.

        Raises:
          ValueError: When the request cannot be parsed.
        """
        lines = iter_lines(sock)

        try:
            request_line = next(lines).decode("ascii")
        except StopIteration:
            raise ValueError("Request line missing.")

        try:
            method, path, _ = request_line.split(" ")
        except ValueError:
            raise ValueError(f"Malformed request line {request_line!r}.")

        headers = Headers()
        buff = b""
        while True:
            try:
                line = next(lines)
            except StopIteration as e:
                # StopIteration.value contains the return value of the generator.
                buff = e.value
                break

            try:
                name, _, value = line.decode("ascii").partition(":")
                headers.add(name, value.lstrip())
            except ValueError:
                raise ValueError(f"Malformed header line {line!r}.")

        body = BodyReader(sock, buff=buff)
        return cls(method=method.upper(), path=path, headers=headers, body=body)

Now that we have a body reader, we can update our main server loop to read and print out incoming request bodies:

    while True:
        client_sock, client_addr = server_sock.accept()
        print(f"Received connection from {client_addr}...")
        with client_sock:
            try:
                request = Request.from_socket(client_sock)
                try:
                    content_length = int(request.headers.get("content-length", "0"))
                except ValueError:
                    content_length = 0

                if content_length:
                    body = request.body.read(content_length)
                    print("Request body", body)

                if request.method != "GET":
                    client_sock.sendall(METHOD_NOT_ALLOWED_RESPONSE)
                    continue

                serve_file(client_sock, request.path)
            except Exception as e:
                print(f"Failed to parse request: {e}")
                client_sock.sendall(BAD_REQUEST_RESPONSE)

If you send the server some body data now, you should see it print the whole thing to standard out. Cool!

We're done with the Request data structure for now. We'll come back to it later when we add query string parameter parsing but, for now, let's move it, BodyReader and iter_lines into a module called request.py. We'll also move Headers into its own module called headers.py since the Response class is also going to use the Headers data structure.

headers.py:

import typing

from collections import defaultdict


class Headers:
    def __init__(self) -> None:
        self._headers = defaultdict(list)

    def add(self, name: str, value: str) -> None:
        self._headers[name.lower()].append(value)

    def get_all(self, name: str) -> typing.List[str]:
        return self._headers[name.lower()]

    def get(self, name: str, default: typing.Optional[str] = None) -> typing.Optional[str]:
        try:
            return self.get_all(name)[-1]
        except IndexError:
            return default

request.py:

import io
import socket
import typing

from collections import defaultdict
from headers import Headers


class BodyReader(io.IOBase):
    def __init__(self, sock: socket.socket, *, buff: bytes = b"", bufsize: int = 16_384) -> None:
        self._sock = sock
        self._buff = buff
        self._bufsize = bufsize

    def readable(self) -> bool:
        return True

    def read(self, n: int) -> bytes:
        """Read up to n number of bytes from the request body.
        """
        while len(self._buff) < n:
            data = self._sock.recv(self._bufsize)
            if not data:
                break

            self._buff += data

        res, self._buff = self._buff[:n], self._buff[n:]
        return res


class Request(typing.NamedTuple):
    method: str
    path: str
    headers: Headers
    body: BodyReader

    @classmethod
    def from_socket(cls, sock: socket.socket) -> "Request":
        """Read and parse the request from a socket object.

        Raises:
          ValueError: When the request cannot be parsed.
        """
        lines = iter_lines(sock)

        try:
            request_line = next(lines).decode("ascii")
        except StopIteration:
            raise ValueError("Request line missing.")

        try:
            method, path, _ = request_line.split(" ")
        except ValueError:
            raise ValueError(f"Malformed request line {request_line!r}.")

        headers = Headers()
        buff = b""
        while True:
            try:
                line = next(lines)
            except StopIteration as e:
                buff = e.value
                break

            try:
                name, _, value = line.decode("ascii").partition(":")
                headers.add(name, value.lstrip())
            except ValueError:
                raise ValueError(f"Malformed header line {line!r}.")

        body = BodyReader(sock, buff=buff)
        return cls(method=method.upper(), path=path, headers=headers, body=body)


def iter_lines(sock: socket.socket, bufsize: int = 16_384) -> typing.Generator[bytes, None, bytes]:
    """Given a socket, read all the individual CRLF-separated lines
    and yield each one until an empty one is found.  Returns the
    remainder after the empty line.
    """
    buff = b""
    while True:
        data = sock.recv(bufsize)
        if not data:
            return b""

        buff += data
        while True:
            try:
                i = buff.index(b"\r\n")
                line, buff = buff[:i], buff[i + 2:]
                if not line:
                    return buff

                yield line
            except IndexError:
                break

100 Continue

If you use cURL to send more than 1KiB of data to the server you might notice it hangs for about a second before it reads all the data. That's because cURL uses the 100 Continue status code to figure out if and when it should send large request bodies to the server.

When you make a cURL request with a payload larger than 1KiB, it automatically sends the server an Expect: 100-continue header and waits until either

it receives a HTTP/1.1 100 Continue response status line from the server, at which point it sends the request body to the server, or
it receives some other response status line from the server, in which case it doesn't send the request body to the server at all, or
its 1 second timeout lapses with the server having not sent it any data, at which point it sends the request body to the server.

This mechanism lets clients pause the request until the server determines whether or not it wants to process it. Our server will accept all requests for now so we'll just make it send the client a 100 Continue status every time it gets such an Expect header:

    while True:
        client_sock, client_addr = server_sock.accept()
        print(f"Received connection from {client_addr}...")
        with client_sock:
            try:
                request = Request.from_socket(client_sock)
                if "100-continue" in request.headers.get("expect", ""):
                    client_sock.sendall(b"HTTP/1.1 100 Continue\r\n\r\n")

                try:
                    content_length = int(request.headers.get("content-length", "0"))
                except ValueError:
                    content_length = 0

                if content_length:
                    body = request.body.read(content_length)
                    print("Request body", body)

                if request.method != "GET":
                    client_sock.sendall(METHOD_NOT_ALLOWED_RESPONSE)
                    continue

                serve_file(client_sock, request.path)
            except Exception as e:
                print(f"Failed to parse request: {e}")
                client_sock.sendall(BAD_REQUEST_RESPONSE)

Send the server more than 1KiB of data using cURL now. It should be much faster than it was before.

Responses

So far all the responses we've returned have been "hand-written". It's time for us to write a Response abstraction to make that side of things a little more manageable.

Back in server.py, let's define our Response class:

class Response:
    """An HTTP response.

    Parameters:
      status: The resposne status line (eg. "200 OK").
      headers: The response headers.
      body: A file containing the response body.
      content: A string representing the response body.  If this is
        provided, then body is ignored.
      encoding: An encoding for the content, if provided.
    """

    def __init__(
            self,
            status: str,
            headers: typing.Optional[Headers] = None,
            body: typing.Optional[typing.IO] = None,
            content: typing.Optional[str] = None,
            encoding: str = "utf-8"
    ) -> None:

        self.status = status.encode()
        self.headers = headers or Headers()

        if content is not None:
            self.body = io.BytesIO(content.encode(encoding))
        elif body is None:
            self.body = io.BytesIO()
        else:
            self.body = body

    def send(self, sock: socket.socket) -> None:
        """Write this response to a socket.
        """
        raise NotImplementedError

We'll keep it relatively simple for now: a response is just a class that holds the response status, headers and a file representing the response body. In addition to that, it knows how to write itself to a socket (rather, it will know, since we haven't implemented that part yet). We can use this to rewrite serve_file and our main loop.

def serve_file(sock: socket.socket, path: str) -> None:
    """Given a socket and the relative path to a file (relative to
    SERVER_ROOT), send that file to the socket if it exists.  If the
    file doesn't exist, send a "404 Not Found" response.
    """
    if path == "/":
        path = "/index.html"

    abspath = os.path.normpath(os.path.join(SERVER_ROOT, path.lstrip("/")))
    if not abspath.startswith(SERVER_ROOT):
        response = Response(status="404 Not Found", content="Not Found")
        response.send(sock)
        return

    try:
        with open(abspath, "rb") as f:
            content_type, encoding = mimetypes.guess_type(abspath)
            if content_type is None:
                content_type = "application/octet-stream"

            if encoding is not None:
                content_type += f"; charset={encoding}"

            response = Response(status="200 OK", body=f)
            response.headers.add("content-type", content_type)
            response.send(sock)
            return
    except FileNotFoundError:
        response = Response(status="404 Not Found", content="Not Found")
        response.send(sock)
        return


with socket.socket() as server_sock:
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

    while True:
        client_sock, client_addr = server_sock.accept()
        print(f"Received connection from {client_addr}...")
        with client_sock:
            try:
                request = Request.from_socket(client_sock)
                if "100-continue" in request.headers.get("expect", ""):
                    response = Response(status="100 Continue")
                    response.send(client_sock)

                try:
                    content_length = int(request.headers.get("content-length", "0"))
                except ValueError:
                    content_length = 0

                if content_length:
                    body = request.body.read(content_length)
                    print("Request body", body)

                if request.method != "GET":
                    response = Response(status="405 Method Not Allowed", content="Method Not Allowed")
                    response.send(client_sock)
                    continue

                serve_file(client_sock, request.path)
            except Exception as e:
                print(f"Failed to parse request: {e}")
                response = Response(status="400 Bad Request", content="Bad Request")
                response.send(client_sock)

Before we can implement Response.send, we have to add a method to Headers that'll let us iterate over all the headers.

HeadersDict = typing.Dict[str, typing.List[str]]
HeadersGenerator = typing.Generator[typing.Tuple[str, str], None, None]


class Headers:
    def __init__(self) -> None:
        self._headers = defaultdict(list)

    def add(self, name: str, value: str) -> None:
        self._headers[name.lower()].append(value)

    def get_all(self, name: str) -> typing.List[str]:
        return self._headers[name.lower()]

    def get(self, name: str, default: typing.Optional[str] = None) -> typing.Optional[str]:
        try:
            return self.get_all(name)[-1]
        except IndexError:
            return default

    def __iter__(self) -> HeadersGenerator:
        for name, values in self._headers.items():
            for value in values:
                yield name, value

With this in place, we can now write Response.send:

    def send(self, sock: socket.socket) -> None:
        """Write this response to a socket.
        """
        content_length = self.headers.get("content-length")
        if content_length is None:
            try:
                body_stat = os.fstat(self.body.fileno())
                content_length = body_stat.st_size
            except OSError:
                self.body.seek(0, os.SEEK_END)
                content_length = self.body.tell()
                self.body.seek(0, os.SEEK_SET)

            if content_length > 0:
                self.headers.add("content-length", content_length)

        headers = b"HTTP/1.1 " + self.status + b"\r\n"
        for header_name, header_value in self.headers:
            headers += f"{header_name}: {header_value}\r\n".encode()

        sock.sendall(headers + b"\r\n")
        if content_length > 0:
            sock.sendfile(self.body)

send tries to figure out what the size of the request body is, then it joins up the status line with the headers and sends them over the socket. Finally, it writes the body file to the socket if there is at least one byte in it.

Python's socket.sendfile has the nice property that it figures out if its parameter is just a regular file or not. If it is, then it uses the high-performance sendfile system call to write the file to the socket and if it isn't then it falls back to regular send calls.

We're done with Response for the time being so let's move it into its own module, called response.py.

Server

At this point, our server loop in server.py is pretty short and sweet:

with socket.socket() as server_sock:
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

    while True:
        client_sock, client_addr = server_sock.accept()
        print(f"Received connection from {client_addr}...")
        with client_sock:
            try:
                request = Request.from_socket(client_sock)
                if "100-continue" in request.headers.get("expect", ""):
                    response = Response(status="100 Continue")
                    response.send(client_sock)

                try:
                    content_length = int(request.headers.get("content-length", "0"))
                except ValueError:
                    content_length = 0

                if content_length:
                    body = request.body.read(content_length)
                    print("Request body", body)

                if request.method != "GET":
                    response = Response(status="405 Method Not Allowed", content="Method Not Allowed")
                    response.send(client_sock)
                    continue

                serve_file(client_sock, request.path)
            except Exception as e:
                print(f"Failed to parse request: {e}")
                response = Response(status="400 Bad Request", content="Bad Request")
                response.send(client_sock)

Let's wrap it in a class called HTTPServer:

class HTTPServer:
    def __init__(self, host="127.0.0.1", port=9000) -> None:
        self.host = host
        self.port = port

    def serve_forever(self) -> None:
        with socket.socket() as server_sock:
            server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            server_sock.bind((self.host, self.port))
            server_sock.listen(0)
            print(f"Listening on {self.host}:{self.port}...")

            while True:
                client_sock, client_addr = server_sock.accept()
                self.handle_client(client_sock, client_addr)

    def handle_client(self, client_sock: socket.socket, client_addr: typing.Tuple[str, int]) -> None:
        with client_sock:
            try:
                request = Request.from_socket(client_sock)
                if "100-continue" in request.headers.get("expect", ""):
                    response = Response(status="100 Continue")
                    response.send(client_sock)

                try:
                    content_length = int(request.headers.get("content-length", "0"))
                except ValueError:
                    content_length = 0

                if content_length:
                    body = request.body.read(content_length)
                    print("Request body", body)

                if request.method != "GET":
                    response = Response(status="405 Method Not Allowed", content="Method Not Allowed")
                    response.send(client_sock)
                    return

                serve_file(client_sock, request.path)
            except Exception as e:
                print(f"Failed to parse request: {e}")
                response = Response(status="400 Bad Request", content="Bad Request")
                response.send(client_sock)


server = HTTPServer()
server.serve_forever()

This is still the exact same logic, but now it's a little more reusable. serve_forever sets up the server socket and accepts new connections on a loop, it then sends those connections over to handle_client which processes the request and sends a response over the socket.

Next, let's throw a little bit of concurrency into the mix by defining an HTTPWorker class.

class HTTPWorker(Thread):
    def __init__(self, connection_queue: Queue) -> None:
        super().__init__(daemon=True)

        self.connection_queue = connection_queue
        self.running = False

    def stop(self) -> None:
        self.running = False

    def run(self) -> None:
        self.running = True
        while self.running:
            try:
                client_sock, client_addr = self.connection_queue.get(timeout=1)
            except Empty:
                continue

            try:
                self.handle_client(client_sock, client_addr)
            except Exception:
                print(f"Unhandled error: {e}")
                continue
            finally:
                self.connection_queue.task_done()

    def handle_client(self, client_sock: socket.socket, client_addr: typing.Tuple[str, int]) -> None:
        with client_sock:
            try:
                request = Request.from_socket(client_sock)
                if "100-continue" in request.headers.get("expect", ""):
                    response = Response(status="100 Continue")
                    response.send(client_sock)

                try:
                    content_length = int(request.headers.get("content-length", "0"))
                except ValueError:
                    content_length = 0

                if content_length:
                    body = request.body.read(content_length)
                    print("Request body", body)

                if request.method != "GET":
                    response = Response(status="405 Method Not Allowed", content="Method Not Allowed")
                    response.send(client_sock)
                    return

                serve_file(client_sock, request.path)
            except Exception as e:
                print(f"Failed to parse request: {e}")
                response = Response(status="400 Bad Request", content="Bad Request")
                response.send(client_sock)

HTTP workers are OS threads that wait for new connections to pop up on a queue and then act on them. Let's plug workers into HTTPServer:

class HTTPServer:
    def __init__(self, host="127.0.0.1", port=9000, worker_count=16) -> None:
        self.host = host
        self.port = port
        self.worker_count = worker_count
        self.worker_backlog = worker_count * 8
        self.connection_queue = Queue(self.worker_backlog)

    def serve_forever(self) -> None:
        workers = []
        for _ in range(self.worker_count):
            worker = HTTPWorker(self.connection_queue)
            worker.start()
            workers.append(worker)

        with socket.socket() as server_sock:
            server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            server_sock.bind((self.host, self.port))
            server_sock.listen(self.worker_backlog)
            print(f"Listening on {self.host}:{self.port}...")

            while True:
                try:
                    self.connection_queue.put(server_sock.accept())
                except KeyboardInterrupt:
                    break

        for worker in workers:
            worker.stop()

        for worker in workers:
            worker.join(timeout=30)

Now all the HTTP server class does is it spins up some number of worker threads, then it sets up the server socket and starts accepting new connections. It pushes connections onto the shared connection queue so that the workers can pick them up and handle them.

Just for fun, here's what running Apache Bench on the server yields on my machine:

$ ab -n 10000 -c 32 http://127.0.0.1:9000/
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:
Server Hostname:        127.0.0.1
Server Port:            9000

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      32
Time taken for tests:   3.320 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      790000 bytes
HTML transferred:       150000 bytes
Requests per second:    3011.73 [#/sec] (mean)
Time per request:       10.625 [ms] (mean)
Time per request:       0.332 [ms] (mean, across all concurrent requests)
Transfer rate:          232.35 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       2
Processing:     2   11   5.1     10      73
Waiting:        1    9   4.6      8      72
Total:          3   11   5.1     10      73

Percentage of the requests served within a certain time (ms)
  50%     10
  66%     11
  75%     11
  80%     11
  90%     12
  95%     12
  98%     14
  99%     17
 100%     73 (longest request)

3k requests per second. Not bad considering this is a threaded web server implemented in pure Python!

Winding down

Whew! That was a lot of stuff to cover in one post. I didn't think you'd make it, but here you are! That's it for part 2. In part 3 we're going to cover request handlers and middleware. If you'd like to check out the full source code and follow along, you can find it here.

See ya next time!

Web application from scratch, Part I

Sun, 25 Feb 2018 03:00:00 +0200

This is the first in a series of posts in which I'm going to go through the process of building a web application (and its web server) from scratch in Python. For the purposes of this series, I'm going to solely rely on the Python standard library and I'm going to ignore the WSGI standard.

Without further ado, let's get to it!

The web server

To begin with, we're going to write the HTTP server that will power our web app. But first, we need to spend a little time looking into how the HTTP protocol works.

How HTTP works

Simply put, HTTP clients connect to HTTP servers over the network and send them a string of data representing the request. The server then interprets that request and sends the client back a response. The entire protocol and the formats of those requests and responses are described in RFC2616, but I'm going to informally describe them below so you don't have to read the whole thing.

Request format

Requests are represented by a series of \r\n-separated lines, the first of which is called the "request line". The request line is made up of an HTTP method, followed by a space, followed by the path of the file being requested, followed by another space, followed by the HTTP protocol version the client speaks and, finally, followed by a carriage return (\r) and a line feed (\n) character:

GET /some-path HTTP/1.1\r\n

After the request line come zero or more header lines. Each header line is made up of the header name, followed by a colon, followed by an optional value, followed by \r\n:

Host: example.com\r\n
Accept: text/html\r\n

The end of the headers section is signaled by an empty line:

\r\n

Finally, the request may contain a "body" -- an arbitrary payload that is sent to the server with the request.

Putting it all together, here's a simple GET request:

GET / HTTP/1.1\r\n
Host: example.com\r\n
Accept: text/html\r\n
\r\n

and here's a simple POST request with a body:

POST / HTTP/1.1\r\n
Host: example.com\r\n
Accept: application/json\r\n
Content-type: application/json\r\n
Content-length: 2\r\n
\r\n
{}

Response format

Responses, like requests, are made up of a series of \r\n-separated lines. The first line in the response is called the "status line" and it is made up of the HTTP protocol version, followed by a space, followed by the response status code, followed by another space, then the status code reason, followed by \r\n:

HTTP/1.1 200 OK\r\n

After the status line come the response headers, then an empty line and then an optional response body:

HTTP/1.1 200 OK\r\n
Content-type: text/html\r\n
Content-length: 15\r\n
\r\n
<h1>Hello!</h1>

A simple server

Based on what we know so far about the protocol, let's write a server that sends the same response regardless of the incoming request.

To start out, we need to create a socket, bind it to an address and then start listening for connections.

import socket

HOST = "127.0.0.1"
PORT = 9000

# By default, socket.socket creates TCP sockets.
with socket.socket() as server_sock:
    # This tells the kernel to reuse sockets that are in `TIME_WAIT` state.
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    # This tells the socket what address to bind to.
    server_sock.bind((HOST, PORT))

    # 0 is the number of pending connections the socket may have before
    # new connections are refused.  Since this server is going to process
    # one connection at a time, we want to refuse any additional connections.
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

If you try to run this code now, it'll print to standard out that it's listening on 127.0.0.1:9000 and then exit. In order to actually process incoming connections we need to call the accept method on our socket. Doing so will block the process until a client connects to our server.

with socket.socket() as server_sock:
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

    client_sock, client_addr = server_sock.accept()
    print(f"New connection from {client_addr}.")

Once we have a socket connection to the client, we can start to communicate with it. Using the sendall method, let's send the connecting client an example response:

RESPONSE = b"""\
HTTP/1.1 200 OK
Content-type: text/html
Content-length: 15

<h1>Hello!</h1>""".replace(b"\n", b"\r\n")

with socket.socket() as server_sock:
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

    client_sock, client_addr = server_sock.accept()
    print(f"New connection from {client_addr}.")
    with client_sock:
        client_sock.sendall(RESPONSE)

If you run the code now and then visit http://127.0.0.1:9000 in your favourite browser, it should render the string "Hello!". Unfortunately, the server will exit after it sends the response so refreshing the page will fail. Let's fix that:

RESPONSE = b"""\
HTTP/1.1 200 OK
Content-type: text/html
Content-length: 15

<h1>Hello!</h1>""".replace(b"\n", b"\r\n")

with socket.socket() as server_sock:
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

    while True:
        client_sock, client_addr = server_sock.accept()
        print(f"New connection from {client_addr}.")
        with client_sock:
            client_sock.sendall(RESPONSE)

At this point we have a web server that can serve a simple HTML web page on every request, all in about 25 lines of code. That's not too bad!

A file server

Let's extend the HTTP server so that it can serve files off of disk.

Request abstraction

Before we can do that, we have to be able to read and parse incoming request data from the client. Since we know that request data is represented by a series of lines, each separated by \r\n characters, let's write a generator function that reads data from a socket and yields each individual line:

import typing


def iter_lines(sock: socket.socket, bufsize: int = 16_384) -> typing.Generator[bytes, None, bytes]:
    """Given a socket, read all the individual CRLF-separated lines
    and yield each one until an empty one is found.  Returns the
    remainder after the empty line.
    """
    buff = b""
    while True:
        data = sock.recv(bufsize)
        if not data:
            return b""

        buff += data
        while True:
            try:
                i = buff.index(b"\r\n")
                line, buff = buff[:i], buff[i + 2:]
                if not line:
                    return buff

                yield line
            except IndexError:
                break

This may look a bit daunting, but essentially what it does is it reads as much data as it can from the socket (in bufsize chunks), joins that data together in a buffer (buff) and continually splits the buffer into individual lines, yielding one at a time. Once it finds an empty line, it returns the extra data that it read.

Using iter_lines, we can begin printing the requests we get from our clients:

with socket.socket() as server_sock:
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

    while True:
        client_sock, client_addr = server_sock.accept()
        print(f"New connection from {client_addr}.")
        with client_sock:
            for request_line in iter_lines(client_sock):
                print(request_line)

            client_sock.sendall(RESPONSE)

If you run the server now and visit http://127.0.0.1:9000, you should see something like this in your console:

Received connection from ('127.0.0.1', 62086)...
b'GET / HTTP/1.1'
b'Host: localhost:9000'
b'Connection: keep-alive'
b'Cache-Control: max-age=0'
b'Upgrade-Insecure-Requests: 1'
b'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36'
b'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8'
b'Accept-Encoding: gzip, deflate, br'
b'Accept-Language: en-US,en;q=0.9,ro;q=0.8'

Pretty neat! Let's abstract over that data by defining a Request class:

import typing


class Request(typing.NamedTuple):
    method: str
    path: str
    headers: typing.Mapping[str, str]

For now, the request class is only going to know about methods, paths and request headers. We'll leave parsing query string parameters and reading request bodies for later.

To encapsulate the logic needed to build up a request, we'll add a class method to Request called from_socket:

class Request(typing.NamedTuple):
    method: str
    path: str
    headers: typing.Mapping[str, str]

    @classmethod
    def from_socket(cls, sock: socket.socket) -> "Request":
        """Read and parse the request from a socket object.

        Raises:
          ValueError: When the request cannot be parsed.
        """
        lines = iter_lines(sock)

        try:
            request_line = next(lines).decode("ascii")
        except StopIteration:
            raise ValueError("Request line missing.")

        try:
            method, path, _ = request_line.split(" ")
        except ValueError:
            raise ValueError(f"Malformed request line {request_line!r}.")

        headers = {}
        for line in lines:
            try:
                name, _, value = line.decode("ascii").partition(":")
                headers[name.lower()] = value.lstrip()
            except ValueError:
                raise ValueError(f"Malformed header line {line!r}.")

        return cls(method=method.upper(), path=path, headers=headers)

It uses the iter_lines function we defined earlier to read the request line. That's where it gets the method and the path, then it reads each individual header line and parses those. Finally, it builds the Request object and returns it. If we plug that into our server loop, it should look something like this:

with socket.socket() as server_sock:
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

    while True:
        client_sock, client_addr = server_sock.accept()
        print(f"Received connection from {client_addr}...")
        with client_sock:
            request = Request.from_socket(client_sock)
            print(request)
            client_sock.sendall(RESPONSE)

If you connect to the server now, you should see lines like this one get printed out:

Request(method='GET', path='/', headers={'host': 'localhost:9000', 'user-agent': 'curl/7.54.0', 'accept': '*/*'})

Because from_socket can raise an exception under certain circumstances, the server might crash if given an invalid request right now. To simulate this, you can use telnet to connect to the server and send it some bogus data:

~> telnet 127.0.0.1 9000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello
Connection closed by foreign host.

Sure enough, the server crashed:

Received connection from ('127.0.0.1', 62404)...
Traceback (most recent call last):
  File "server.py", line 53, in parse
    request_line = next(lines).decode("ascii")
ValueError: not enough values to unpack (expected 3, got 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "server.py", line 82, in <module>
    with client_sock:
  File "server.py", line 55, in parse
    raise ValueError("Request line missing.")
ValueError: Malformed request line 'hello'.

To handle these kinds of issues a little more gracefully, let's wrap the call to from_socket in a try-except block and send the client a "400 Bad Request" response when we get a malformed request:

BAD_REQUEST_RESPONSE = b"""\
HTTP/1.1 400 Bad Request
Content-type: text/plain
Content-length: 11

Bad Request""".replace(b"\n", b"\r\n")

with socket.socket() as server_sock:
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

    while True:
        client_sock, client_addr = server_sock.accept()
        print(f"Received connection from {client_addr}...")
        with client_sock:
            try:
                request = Request.from_socket(client_sock)
                print(request)
                client_sock.sendall(RESPONSE)
            except Exception as e:
                print(f"Failed to parse request: {e}")
                client_sock.sendall(BAD_REQUEST_RESPONSE)

If we try to break it now, our client will get a response back and the server will stay up:

~> telnet 127.0.0.1 9000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello
HTTP/1.1 400 Bad Request
Content-type: text/plain
Content-length: 11

Bad RequestConnection closed by foreign host.

At this point we're ready to start implementing the file serving part, but first let's make our default response a "404 Not Found" response:

NOT_FOUND_RESPONSE = b"""\
HTTP/1.1 404 Not Found
Content-type: text/plain
Content-length: 9

Not Found""".replace(b"\n", b"\r\n")

#...

with socket.socket() as server_sock:
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

    while True:
        client_sock, client_addr = server_sock.accept()
        print(f"Received connection from {client_addr}...")
        with client_sock:
            try:
                request = Request.from_socket(client_sock)
                print(request)
                client_sock.sendall(NOT_FOUND_RESPONSE)
            except Exception as e:
                print(f"Failed to parse request: {e}")
                client_sock.sendall(BAD_REQUEST_RESPONSE)

Additionally, let's add a "405 Method Not Allowed" response. We're going to need it for when we get anything other than a GET request.

METHOD_NOT_ALLOWED_RESPONSE = b"""\
HTTP/1.1 405 Method Not Allowed
Content-type: text/plain
Content-length: 17

Method Not Allowed""".replace(b"\n", b"\r\n")

Let's define a SERVER_ROOT constant to represent where the server should serve files from and a serve_file function.

import mimetypes
import os
import socket
import typing

SERVER_ROOT = os.path.abspath("www")

FILE_RESPONSE_TEMPLATE = """\
HTTP/1.1 200 OK
Content-type: {content_type}
Content-length: {content_length}

""".replace("\n", "\r\n")


def serve_file(sock: socket.socket, path: str) -> None:
    """Given a socket and the relative path to a file (relative to
    SERVER_SOCK), send that file to the socket if it exists.  If the
    file doesn't exist, send a "404 Not Found" response.
    """
    if path == "/":
        path = "/index.html"

    abspath = os.path.normpath(os.path.join(SERVER_ROOT, path.lstrip("/")))
    if not abspath.startswith(SERVER_ROOT):
        sock.sendall(NOT_FOUND_RESPONSE)
        return

    try:
        with open(abspath, "rb") as f:
            stat = os.fstat(f.fileno())
            content_type, encoding = mimetypes.guess_type(abspath)
            if content_type is None:
                content_type = "application/octet-stream"

            if encoding is not None:
                content_type += f"; charset={encoding}"

            response_headers = FILE_RESPONSE_TEMPLATE.format(
                content_type=content_type,
                content_length=stat.st_size,
            ).encode("ascii")

            sock.sendall(response_headers)
            sock.sendfile(f)
    except FileNotFoundError:
        sock.sendall(NOT_FOUND_RESPONSE)
        return

serve_file takes the client socket and a path to a file. It then tries to resolve that path to a real file inside of the SERVER_ROOT, returning a "not found" response if the file resolves outside of the server root. Then it tries to open the file and figure out its mime type and size (using os.fstat), then it constructs the response headers and uses the sendfile system call to write the file to the socket. If it can't find the file on disk, then it sends a "not found" response.

If we add serve_file into the mix, our server loop should now look like this:

with socket.socket() as server_sock:
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(0)
    print(f"Listening on {HOST}:{PORT}...")

    while True:
        client_sock, client_addr = server_sock.accept()
        print(f"Received connection from {client_addr}...")
        with client_sock:
            try:
                request = Request.from_socket(client_sock)
                if request.method != "GET":
                    client_sock.sendall(METHOD_NOT_ALLOWED_RESPONSE)
                    continue

                serve_file(client_sock, request.path)
            except Exception as e:
                print(f"Failed to parse request: {e}")
                client_sock.sendall(BAD_REQUEST_RESPONSE)

If you add a file called www/index.html next to your server.py file and visit http://localhost:9000 you should see the contents of that file. Cool, eh?

Winding down

That's it for part 1. In part 2 we're going to cover extracting Server and Response abstractions as well as making the server handle multiple concurrent connections. If you'd like to check out the full source code and follow along, you can find it here.

See ya next time!

Automatic retries with Celery

Sun, 25 Feb 2018 02:00:00 +0200

One of the things that I think Celery could be doing better out of the box is to provide support for automatically retrying tasks on failure (thereby forcing users to write idempotent tasks by default).

Fortunately, you can achieve this using signals, specifically the task-failure signal. All you have to do is connect to it and call the retry method on your task:

import logging

from celery import signals


@signals.task_failure.connect
def retry_task_on_exception(*args, **kwargs):
  task = kwargs.get("sender")
  einfo = kwargs.get("einfo")
  logging.warning("Uncaught exception: %r for task: %s", einfo, task)
  task.retry(countdown=3600)

The above will retry failing tasks once an hour. The task object also contains information about how many times it's been retried and you can use that information in order to retry tasks with exponential backoff and to make them stop after a while. For example:

@signals.task_failure.connect
def retry_task_on_exception(*args, **kwargs):
  task = kwargs.get("sender")
  einfo = kwargs.get("einfo")
  logging.warning("Uncaught exception: %r for task: %s", einfo, task)

  # Backoffs: 60, 120, 240, 480, 960, 1920, 3600, 3600, ...
  backoff = min(60 * 2 ** task.request.retries, 3600)
  task.retry(countdown=backoff)

Not too bad. One caveat with this approach is you might run into issues if you have a large volume of failing tasks due to Celery's weak support for delayed tasks.

Check out dramatiq if you'd rather not have to worry about this kind of stuff.

Dramatiq cron with APScheduler

Thu, 11 Jan 2018 02:00:00 +0200

Here's a quick way you can combine Dramatiq and APScheduler to automatically schedule tasks to execute at certain times.

Install Dramatiq and APScheduler using pipenv:

pipenv install dramatiq apscheduler

Next, declare a task and decorate it with @cron. We'll define the cron function afterwards. In a module called tasks.py, add the following code:

import dramatiq

from cron import cron
from datetime import datetime


@cron("* * * * *")  # Run once a minute
@dramatiq.actor
def print_current_datetime():
    print(datetime.now())

Then define the decorator in cron.py:

import importlib

from apscheduler.triggers.cron import CronTrigger

JOBS = []


def cron(crontab):
    """Wrap a Dramatiq actor in a cron schedule.
    """
    trigger = CronTrigger.from_crontab(crontab)

    def decorator(actor):
        module_path = actor.fn.__module__
        func_name = actor.fn.__name__
        JOBS.append((trigger, module_path, func_name))
        return actor

    return decorator

JOBS is where the job definitions are stored. When we run the scheduler from the command line, we'll iterate over this list and register jobs based on entries made here.

cron just builds a cron trigger and adds a job definition to JOBS.

Now that we have all the components in place, we just need a way to run a scheduler from the command line. In a file called run_cron.py add the following:

import cron
import logging
import signal
import sys
import tasks  # imported for its side-effects

from apscheduler.schedulers.blocking import BlockingScheduler

logging.basicConfig(
    format="[%(asctime)s] [PID %(process)d] [%(threadName)s] [%(name)s] [%(levelname)s] %(message)s",
    level=logging.DEBUG,
)

# Pika is a bit noisy w/ Debug logging so we have to up its level.
logging.getLogger("pika").setLevel(logging.WARNING)


def main():
    scheduler = BlockingScheduler()
    for trigger, module_path, func_name in cron.JOBS:
        job_path = f"{module_path}:{func_name}.send"
        job_name = f"{module_path}.{func_name}"
        scheduler.add_job(job_path, trigger=trigger, name=job_name)

    def shutdown(signum, frame):
        scheduler.shutdown()

    signal.signal(signal.SIGINT, shutdown)
    signal.signal(signal.SIGTERM, shutdown)

    scheduler.start()
    return 0


if __name__ == "__main__":
    sys.exit(main())

Here we set up logging, instantiate a blocking scheduler and register all the jobs that were declared in tasks.py -- which is why we import it in the first place, if we didn't, then the code that registers the jobs would never run. Finally, we add a signal handler to shut down the scheduler whenever the process receives a SIGINT or a SIGTERM.

Run rabbitmq-server then python run_cron.py and dramatiq tasks in a separate terminal and you're done! You should see your workers print the current time once a minute.

Podcast dot init interview

Sun, 24 Dec 2017 02:00:00 +0200

I was interviewed by Tobias Macey of Podcast.__init__ a few days ago. The episode was just published a few hours ago. Check it out if you want to hear me ramble about Dramatiq and task queueing in Python! :-)

Prometheus metrics and API Star

Sat, 25 Nov 2017 02:00:00 +0200

This past week I started playing with API Star and I'm kind of in love with it right now. Being a new project, its docs are a bit lacking -- its source code, however, is high quality and easy to understand -- so it took me a little time to figure out a way to automatically track request and response metrics using Prometheus.

The Goal

My goal was to globally track request durations, request counts and the number of requests in progress at any given time. AFAICT, there are two major ways to do this with API Star: either write a WSGI middleware and wrap the API star App object or leverage a component along with BEFORE_REQUEST and AFTER_REQUEST hooks. I ended up going with the latter.

The Component

First I defined a Prometheus component. In the app's lifecycle this is a singleton (preload=True) object that can be injected in the before and after response hooks. It is used to keep track of some thread-local state (like the start time of each request) and to update the prom metrics.

import time

from apistar import http
from http import HTTPStatus
from prometheus_client import Counter, Gauge, Histogram
from threading import local

REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "Time spent processing a request.",
    ["method", "handler"],
)
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Request count by method, handler and response code.",
    ["method", "handler", "code"],
)
REQUESTS_INPROGRESS = Gauge(
    "http_requests_inprogress",
    "Requests in progress by method and handler",
    ["method", "handler"],
)


class Prometheus:
    def __init__(self):
        self.data = local()

    def track_request_start(self, method, handler):
        self.data.start_time = time.monotonic()

        handler_name = f"{handler.__module__}.{handler.__name__}"
        REQUESTS_INPROGRESS.labels(method, handler_name).inc()

    def track_request_end(self, method, handler, ret):
        status = 200
        if isinstance(ret, http.Response):
            status = HTTPStatus(ret.status).value

        handler_name = "<builtin>"
        if handler is not None:
            handler_name = f"{handler.__module__}.{handler.__name__}"
            duration = time.monotonic() - self.data.start_time
            del self.data.start_time
            REQUEST_DURATION.labels(method, handler_name).observe(duration)

        REQUEST_COUNT.labels(method, handler_name, status).inc()
        REQUESTS_INPROGRESS.labels(method, handler_name).dec()

Thread-local data is stored in the data property of the singleton and the track_request_start and track_request_end methods are meant to be called at the beginning and end of each request, respectively.

The Hooks

The hooks request the Prometheus component along with information about the current request method and handler. All they do is pass that information along to track_request_start and track_request_end.

def before_request(prometheus: Prometheus,
                   method: http.Method,
                   handler: Handler):
    prometheus.track_request_start(method, handler)


def after_request(prometheus: Prometheus,
                  method: http.Method,
                  handler: Handler,
                  ret: ReturnValue):
    prometheus.track_request_end(method, handler, ret)
    return ret

One odd thing I ran into was the fact that for builtin request handlers, such as the 404 handler, the before_request hook doesn't get called. This is why track_request_end checks whether or not the handler is None before trying to compute the time spent running it. In those cases, self.data.start_time is never set because track_request_start was never called.

The Exposition Handler

One problem I ran into while trying to expose the metrics was the fact that API Star doesn't seem to provide a plaintext renderer. Defining my own was straightforward enough:

from apistar import http
from apistar.renderers import Renderer


class PlaintextRenderer(Renderer):
    def render(self, data: http.ResponseData) -> bytes:
        return data

Finally, the exposition handler just calls the prom client's generate_latest function and renders it using the PlaintextRenderer.

from apistar import Response, annotate
from prometheus_client import CONTENT_TYPE_LATEST


@annotate(renderers=[PlaintextRenderer()])
def expose_metrics():
    return Response(generate_latest(), headers={
        "content-type": CONTENT_TYPE_LATEST,
    })

The App

To hook it all up, I added all the bits I defined previously to the appropriate spots in the WSGIApp's config, making sure the before_request and after_request hooks were the first and last ones to be added to the configuration, respectively.

import prometheus_component

from apistar import Component, Route, hooks
from apistar.frameworks.wsgi import WSGIApp as App

components = [
    Component(prometheus_component.Prometheus, preload=True),
]

routes = [
    Route("/metrics", "GET", prometheus_component.expose_metrics),
]

settings = {
    "BEFORE_REQUEST": [
        prometheus_component.before_request,
        hooks.check_permissions,
    ],
    "AFTER_REQUEST": [
        hooks.render_response,
        prometheus_component.after_request,
    ],
}

app = App(
    components=components,
    routes=routes,
    settings=settings,
)

Epilogue

In the end I packed all of this up in a library so I could reuse the functionality across apps. You can find it here.

Like I said at the beginning, API Star has been a joy to use so far and I can't wait to play around with it some more. I'm likely going to put it into production soon so expect more content to come about it.

Building a PDF API with Django and Dramatiq

Sun, 12 Nov 2017 20:12:43 +0200

In this post I talk about how you can use Django, Dramatiq and h2p to create a simple HTTP API that can turn any URL into a PDF.

What is Dramatiq?

Dramatiq is a distributed task processing library for Python 3 that I've been working on as an alternative to Celery. Using Dramatiq, you can transparently run functions in the background across a large number of machines. In this post I'm going to use it to offload the work of generating PDFs from the web server onto a background processing server.

Why use a task queue?

Long-running or computationally-intensive tasks in the middle of the request-response cycle of a web server can severily impact the latency and throughput of that server. A common pattern to work around this issue is to use a task queue to offload the parts of the request that can be done later and in the background off to a different fleet of servers known as workers. This has other advantages, too: tasks may easily be retried later in case there's an error and you can run tasks completely outside of the request-response cycle (eg. using a cron job).

Generating PDFs from web pages is a slow process so we want to take that out of the request and give the requester a way to poll for the result of the operation.

Setup

First things first, we're going to need a message broker. Dramatiq currently works with Redis and RabbitMQ, but for this post I'm going to use RabbitMQ. To install it on macOS, you can run:

$ brew install rabbitmq

Run it with rabbitmq-server.

Next, we're going to create a new virtual environment and, inside of that environment, use pipenv to install all the prerequisite libraries:

$ pipenv install django djangorestframework django_dramatiq "dramatiq[rabbitmq, watch]" h2p

django_dramatiq is a small Django app that makes integrating Dramatiq and Django easy.

After that's done, we're going to create a Django project called pdfapi:

$ django-admin.py startproject pdfapi .

Finally, we need to configure django_dramatiq to use RabbitMQ. In pdfapi/settings.py, add django_dramatiq and rest_framework to your INSTALLED_APPS:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'django_dramatiq',
    'rest_framework',
]

And configure the broker in the same file:

DRAMATIQ_BROKER = {
    "BROKER": "dramatiq.brokers.rabbitmq.RabbitmqBroker",
    "OPTIONS": {
        "url": "amqp://localhost:5672",
    },
    "MIDDLEWARE": [
        "dramatiq.middleware.Prometheus",
        "dramatiq.middleware.AgeLimit",
        "dramatiq.middleware.TimeLimit",
        "dramatiq.middleware.Retries",
        "django_dramatiq.middleware.AdminMiddleware",
        "django_dramatiq.middleware.DbConnectionsMiddleware",
    ]
}

Let's run the migrations and then the server to make sure everything's working so far:

$ python manage.py migrate
$ python manage.py runserver

If you visit http://127.0.0.1:8000, you should now see the familiar "Congratulations on your first Django-powered page" view. Kill the server and create a new app called pdfs:

$ python manage.py startapp pdfs

The API

The API we're going to define is going to be very simple. It will accept POST requests to /v1/pdfs containing the url we're expected to convert into a PDF, these requests will submit a task to generate the PDF and immediately return a JSON object with an id and a status field that the caller can then use to keep track of the job.

Using the id field from the response, the caller will be able to poll /v1/pdfs/{id} to find out what the status of the task is.

The PDF model

In pdfs/models.py declare the following model:

class PDF(models.Model):
    STATUS_PENDING = "pending"
    STATUS_FAILED = "failed"
    STATUS_DONE = "done"
    STATUSES = [
        (STATUS_PENDING, "Pending"),
        (STATUS_FAILED, "Failed"),
        (STATUS_DONE, "Done"),
    ]

    source_url = models.CharField(max_length=512)
    status = models.CharField(
        max_length=10,
        default=STATUS_PENDING,
        choices=STATUSES,
    )

    @property
    def filename(self):
        raise NotImplementedError

    @property
    def pdf_url(self):
        raise NotImplementedError

We're going to skip the implementations of the filename and pdf_url properties for now.

Build and run the migrations:

$ python manage.py makemigrations
$ python manage.py migrate

Then add a serializer for that model in pdfs/serializers.py:

from rest_framework import serializers

from .models import PDF


class PDFSerializer(serializers.ModelSerializer):
    source_url = serializers.URLField(max_length=512)
    pdf_url = serializers.URLField(read_only=True)

    class Meta:
        model = PDF
        fields = ("id", "source_url", "pdf_url", "status")
        read_only_fields = ("status",)

We're going to use this serializer to render PDF models as JSON and to validate incoming requests.

The Views

In pdfs/views.py add the following views:

from django.views.decorators.csrf import csrf_exempt
from rest_framework import status
from rest_framework.decorators import api_view
from rest_framework.response import Response

from .models import PDF
from .serializers import PDFSerializer


@csrf_exempt
@api_view(["POST"])
def create_pdf(request):
    serializer = PDFSerializer(data=request.data)
    if serializer.is_valid():
        serializer.save()
        return Response(serializer.data)
    return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)


@api_view(["GET"])
def view_pdf(request, pk):
    try:
        pdf = PDF.objects.get(pk=pk)
        serializer = PDFSerializer(pdf)
        return Response(serializer.data)
    except PDF.DoesNotExist:
        return Response(status=status.HTTP_404_NOT_FOUND)

And then hook them up in pdfs/urls.py:

from django.conf.urls import url

from . import views

app_name = "pdfs"
urlpatterns = [
    url(r"^$", views.create_pdf, name="create_pdf"),
    url(r"^(?P<pk>\d+)$", views.view_pdf, name="view_pdf"),
]

Add the pdfs app to pdfapi/settings.py:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'django_dramatiq',
    'rest_framework',

    'pdfs',
]

Finally, include the pdfs urls in pdfapi/urls.py:

from django.conf.urls import url, include
from django.contrib import admin

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^v1/pdfs/', include("pdfs.urls")),
]

At this point if you run the development server and visit /v1/pdfs you should be able to interact with the API.

The Task

So far we've declared the model and an API that lets us interact with it, but we haven't done anything to actually generate PDFs so every PDF we create using the API is going to be in a perpetual pending state. Let's fix that.

In pdfs/tasks.py add the following task:

import dramatiq
import h2p

from .models import PDF


@dramatiq.actor
def generate_pdf(pk):
    pdf = PDF.objects.get(pk=pk)

    try:
        h2p.generate_pdf(
            pdf.filename,
            source_uri=pdf.source_url,
        ).result()

        pdf.status = PDF.STATUS_DONE
    except h2p.ConversionError:
        pdf.status = PDF.STATUS_FAILED

    pdf.save()

Let's break this down a little bit. generate_pdf is just a normal Python function that we've decorated with @dramatiq.actor. This makes it possible to run the function asynchronously.

generate_pdf takes a pk parameter representing the id of a PDF, this is important because tasks are distributed and we wouldn't want to send entire PDF objects over the network. It delegates the work of actually creating the PDF to h2p and updates the PDF object's status based on the result of that operation.

We're passing the filename property of PDF to h2p.generate_pdf but we haven't implemented it yet so let's fill it and pdf_url in on the PDF model in pdfs/models.py:

    @property
    def filename(self):
        return f"{settings.MEDIA_ROOT}{self.pk}.pdf"

    @property
    def pdf_url(self):
        return f"{settings.MEDIA_URL}{self.pk}.pdf"

Don't forget to add MEDIA_ROOT and MEDIA_URL to pdfapi/settings.py:

MEDIA_ROOT = os.path.join(BASE_DIR, "files/")
MEDIA_URL = "/media"

Create the files folder and then add a static handler to pdfapi/urls.py:

from django.conf import settings
from django.conf.urls import url, include
from django.conf.urls.static import static
from django.contrib import admin

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^v1/pdfs/', include("pdfs.urls")),
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Hooking 'em up

At this point we've created a task that can generate PDF files and an API that can submit and keep track of that work. Let's hook them up!

In the create_pdf view from pdfs/views.py change the serializer.save() call to:

    if serializer.is_valid():
        pdf = serializer.save()
        generate_pdf.send(pdf.pk)
        return Response(serializer.data)

Now every time someone creates a PDF object using the API, we'll enqueue a generate_pdf task. Spin up the API server and some Dramatiq workers and test it out.

$ python manage.py runserver
$ python manage.py rundramatiq  # in a separate terminal window

To test it out, send a create request using curl:

$ curl -d"source_url=http://example.com" http://127.0.0.1:8000/v1/pdfs/
{"id":1,"source_url":"http://example.com","pdf_url":"/media/1.pdf","status":"pending"}

Then poll using GET requests until it's ready:

$ curl http://127.0.0.1:8000/v1/pdfs/1
{"id":1,"source_url":"http://example.com","pdf_url":"/media/1.pdf","status":"done"}

Finally, visit http://127.0.0.1:8000/media/1.pdf to view the generated PDF.

Next Steps

You can find the full code on GitHub. If you want to learn more about Dramatiq (and hopefully you do!) head on to the docs. I've put a lot of work into making them as accessible as possible.

Happy coding!

Announcing h2p

Sun, 5 Nov 2017 02:00:00 +0200

This past week I released h2p, a simple python frontend to libwkhtmltox that lets you generate PDF files from web pages without needing to spawn subprocesses.

You can use pip or pipenv to install it:

pipenv install h2p

And the API is straightforward:

import h2p

websites = ["https://google.com", "https://example.com", "https://defn.io"]

tasks = []
for i, website in enumerate(websites, 1):
   filename = f"output-{i}.pdf"
   tasks.append(h2p.generate_pdf(filename, website))
   print(f"Enqueued task for {website!r} -> {filename!r}.")

# ... do other stuff while your pdfs are being generated ...

for task in tasks:
   task.result()

print("All tasks are done.")

Each call to generate_pdf returns an asynchronous task that represents the action of generating that pdf. Calling result on that task will block until it's done.

Announcing django_dramatiq

Sat, 4 Nov 2017 23:00:00 +0200

This past week I released django_dramatiq, an app that helps you integrate Dramatiq and Django. Additionally, I released these two example repos:

django_dramatiq_example, a small Django app that shows off django_dramatiq, and
flask_dramatiq_example, a small Flask app that shows off how you can integrate Flask and Dramatiq.

Announcing dramatiq

Thu, 29 Jun 2017 01:00:00 +0300

Today marks the first public release of Dramatiq, a distributed task queueing library for Python with a focus on reliability, simplicity and performance.

Check it out and let me know what you think!

Announcing elm-cookiecutter

Fri, 24 Jun 2016 03:00:00 +0300

Since most of my Elm apps tend to follow a similar structure, I've decided to create and open source a cookiecutter template that deals with all of the boilerplate required to set up one of these apps the way I like it. The repository is available here.

Quickstart

To get started immediately:

pip install cookiecutter
cookiecutter gh:Bogdanp/elm-cookiecutter

You will be prompted for a project name and description. Once you've completed the process, cd into $PROJECT_NAME and run:

npm install
make serve

The latter will build your sources, start up a web server and open the app in a web browser. Once you've made changes to any of the sources, run make again to rebuild or run make watch in a separate terminal to have it automatically rebuild the app when any of the sources change.

To produce a production build, run env NPM_ENV=prod make.

Structure

Some of the more interesting files are described below.

.
├── Makefile               # Rules for building the app. See `make help`
├── README.md              # The app's README
├── bin
│   └── server.js          # The development server. Points all nonexistent paths to `index.html`
├── build
│   ├── index.html
│   └── js                 # The final product ends up in js/app.js. This includes styles
├── css
│   ├── app.scss           # The entrypoint for styles
│   └── normalize.scss     # A slightly modified Normalize.css
├── elm
│   ├── Main.elm           # The entrypoint for Elm code
│   ├── Model.elm          # Definitions for Flags, the Model and the root Msg. Referenced by Update and View
│   ├── Pages
│   │   └── Dashboard.elm
│   ├── Routes.elm
│   ├── Update.elm
│   ├── Util.elm
│   └── View.elm
├── elm-package.json
├── js
│   └── app.js             # Sets up the Elm app and deals with flags, subscriptions and ports
├── package.json
└── webpack.config.js

7 directories, 17 files

Announcing elm-datepicker

Wed, 1 Jun 2016 03:00:00 +0300

I found myself needing a reusable date picker component for an app so I went ahead and built one. You can find the package here and a demo here (source).

elm-mode refactoring demo

Sat, 27 Feb 2016 02:00:00 +0200

I recorded a short demo of elm-mode's new refactoring features. You can view it on Youtube.

Announcing elm-route

Sun, 21 Feb 2016 02:00:00 +0200

Today marks the first release of elm-route, a type safe route parsing DSL built on top of elm-combine. Its main additions to the world of type safe route parsing are:

A generic DSL for expressing arbitrarily-nested dynamic routes (at the cost of uglier route constructors as the depth increases).
An automatic way to do reverse routing that when coupled with a small amount of boilerplate should provide the safest approach to reverse routing that the Elm language can currently support.

You can see a working demo here (source). Note that direct linking to routes in the demo does not work due to a limitation of Gihub Pages.

Related work

elm-route-parser is another type safe route parsing library. In contrast to elm-route, its more rigid matchers make it possible to have cleaner route constructors (for example, HomeR instead of HomeR ()). It does not yet provide automatic reverse routing support.

Example

Here's a short taste of what the DSL looks like:

type Sitemap
  = HomeR ()
  | UsersR ()
  | UserR Int
  | UserPostR (Int, String)

homeR = HomeR := static ""
usersR = UsersR := static "users"
userR = UserR := "users" <//> int
userPostR = UserPostR := "users" <//> int </> string
sitemap = router [homeR, usersR, userR, userPostR]

match : String -> Maybe Sitemap
match = Route.match sitemap

route : Sitemap -> String
route r =
  case r of
    HomeR () -> reverse homeR []
    UsersR () -> reverse usersR []
    UserR id -> reverse userR [toString id]
    UserPostR (uid, pid) -> reverse userPostR [toString uid, pid]

For more check out the README and the examples folder.

elm-mode TAGS demo

Sun, 17 Jan 2016 02:00:00 +0200

I recorded a short demo of elm-mode's new automatic TAGS file generation feature. You can view it on Youtube.

elm-mode demo

Wed, 13 Jan 2016 02:00:00 +0200

I recorded a short screencast showing off some of the features available in elm-mode. You can view it on Youtube.

ido-mode

Mon, 12 Oct 2015 03:00:00 +0300

Ido mode is one of those Emacs packages that you can't imagine living without once you embed it into your workflow. It stands for "interactive do" and in this post I'm going to talk about what it does, some of the configuration options that come bundled with it and how you can enhance it further.

What it does

Ido extends the built-in find-file and switch-to-buffer commands with interactive completion. It displays all the items that are available to you based on the current selection, narrowing down to the most relevant ones as you type to expand said selection. You can find a brief demonstration of this functionality here.

Turn it on by calling ido-mode.

(require 'ido)
(ido-mode)

Calling ido-everywhere will ensure that ido is used for all buffer and file selections in Emacs.

(ido-everywhere)

If, for some reason, you'd prefer to only apply ido to one of those commands but not the other you can do so by calling ido-mode with a 'buffer or 'file argument to limit ido to buffer switching and finding files respectively.

Completion

As you type, ido narrows the result set based on which items your selection is a substring of. You can select the first result by pressing RET and you can cycle through the result set at any point with C-s and C-r. This can pose a problem if, for example, you want to create a new file whose name is a substring of another file in the current directory. You can use C-j to tell ido to use whatever you typed verbatim.

Ido limits the list of visible completions to a few at a time, but if you want to view the full list of available completions you can hit ?. Doing so will display all of the available completions in a separate buffer.

Prefix matching

Toggle prefix matching inside an ido buffer by hitting C-p. This is similar to the standard Emacs completion method in that it will only display results that start with your selection.

You can make prefix matching the default behavior by setting ido-enable-prefix to a truthy value.

(setq ido-enable-prefix t)

Flexible matching

Ido's flexible matching makes it so that any items containing all of the selection's characters in order will appear in the result set.

Turn it on by setting ido-enable-flex-matching to a truthy value.

(setq ido-enable-flex-matching t)

Finding files

ido-find-file comes with a few file-specific bindings. Find more information about this command with C-h f ido-find-file RET.

I most commonly use C-d to open the current matching directory in dired and C-k to delete the current matching file.

To ignore certain file extensions when finding files set ido-ignore-extensions to a truthy value and add the extension to the completion-ignored-extensions list.

(setq ido-ignore-extensions t)
(add-to-list 'completion-ignored-extensions ".pyc")

Sometimes it is convenient to use the filename under the cursor as a starting point for ido completion, ido-use-filename-at-point tells ido to do just that.

(setq ido-use-filename-at-point 'guess)

By default, ido will ask for confirmation every time you attempt to create a new buffer. This can become annoying if you like to create many buffers. Turn it off with:

(setq ido-create-new-buffer 'always)

Switching buffers

As with finding files, switching buffers using ido comes with a number of commands you can run on the current selection. Find out more about this command with C-h f ido-switch-buffer RET.

I most commonly use C-k to kill the current matching buffer. If you find that the buffer you meant to switch to isn't open, you can switch to find-file by hitting C-f.

You can use ido-switch-buffer to switch to recently-used buffers by enabling ido-use-virtual-buffers. Doing so will turn recentf mode on which means you will be able to open a file in Emacs, close it then start it back up and switch to that file's buffer as if it were already open.

(setq ido-use-virtual-buffers t)

Extensions

smex

smex is an ido-based replacement for M-x. In addition to bringing ido completion to M-x, smex maintains a list of your most-used commands so that it can order results by frequency.

You can grab it off of MELPA and bind it to your preferred key.

(require 'smex)
(smex-initialize)

;; My personal preference is C-;
(global-set-key (kbd "C-;") #'smex)
;; but you can also override M-x
(global-set-key (kbd "M-x") #'smex)

Smex comes with command completion for the current major mode. You can use this by binding smex-major-mode-commands to a key.

(global-set-key (kbd "M-X") #'smex-major-mode-commands)

ido-ubiquitous

Like the name implies, ido-ubiquitous is a package that attempts to weave in ido completion wherever it can. With this package, most functions that use completing-read will automatically start using ido for completion.

The package is available on MELPA and you can turn it on by calling ido-ubiquitous-mode.

(require 'ido-ubiquitous)
(ido-ubiquitous-mode)

If you run into any issues because of ido-ubiquitous, view the documentation for ido-ubiquitous-command-overrides and ido-ubiquitous-function-overrides by calling describe-variable (bound to C-h v by default). You can use those variables to turn ido off for specific functions or commands.

Note that ido-ubiquitous does not turn ido completion on for packages that come with built in ido support (even if it is not turned on by default) like Magit and Org mode. I have included a section below on how you can turn ido on for both of those modes.

ido-vertical-mode

ido-vertical-mode modifies the ido completion buffer so that it displays vertically rather than horizontally, making it so that the most relevant completions are displayed at the top.

ido-vertical-mode is available on MELPA.

(require 'ido-vertical-mode)
(ido-vertical-mode)

ido-clever-match

Finally, ido-clever-match is a simple package I wrote that wraps the built-in ido matching function in order to try to provide predictable prefix, substring and flex matching. You can find more information about how it works on its Github page but the gist of it is it ranks matches based on class (exact, prefix, substring or flex) and then some sub-metric within that class. The package ensures that prefix matches always come before substring which always come before flex matches.

It is available on MELPA and you can enable it with:

(require 'ido-clever-match)
(ido-clever-match-enable)

If you find that you prefer ido's standard matching behavior and would like to go back simply call ido-clever-match-disable.

Other packages

I've found that the following packages work particularly well when paired with ido.

Magit

Magit comes with its own completion function which you can replace with ido by setting magit-completing-read-function to magit-ido-completing-read.

(setq magit-completing-read-function #'magit-ido-completing-read)

Org mode

Like Magit, Org mode comes with its own completion function which you can replace with ido by setting org-completion-use-ido to a truthy value.

The documentation recommends that you turn off org-outline-complete-in-steps if you switch to ido completion.

(setq org-completion-use-ido t
      org-outline-path-complete-in-steps nil)

Projectile

Ido is Projectile's default completion method. The maintainers recommend you install flx-ido for its flexible matching but you can also use Projectile with ido-clever-match.

Wrapping up

In closing, ido-mode is an extremely versatile package that can massively enhance one's workflow when using Emacs.

I highly recommend you try it out!