Jekyll2021-09-28T10:18:41+00:00https://blog.fredrikmeyer.net//feed.xmlFredrik MeyerMathematics is fun.Filtering JSON logs with JQ2021-09-19T18:33:00+00:002021-09-19T18:33:00+00:00https://blog.fredrikmeyer.net//2021/09/19/jq-tricks<p>One of my favorite command line tools is <code class="language-plaintext highlighter-rouge">jq</code> (<a href="https://stedolan.github.io/jq/">homepage</a>). It is often very useful to look at structured JSON data.</p>
<p>My most common usage of <code class="language-plaintext highlighter-rouge">jq</code> is very simple. Whenever some command returns a long JSON, it is very handy to pipe the result to <code class="language-plaintext highlighter-rouge">jq</code>. Like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl https://api.github.com/users/fredrikmeyer | jq
</code></pre></div></div>
<p>Then I get a syntax highligted response that is easy to read (in this case the response was already formatted, but sometimes a JSON response is unformatted).</p>
<p>Another use-case is for looking through logs. As a first step, list all log groups in your AWS account:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">></span> aws logs describe-log-groups | jq <span class="s1">'.logGroups | .[] | .logGroupName'</span>
<span class="s2">"/aws/apigateway/welcome"</span>
<span class="s2">"/aws/lambda/my-first-lambda"</span>
<span class="s2">"/aws/lambda/my-second-lambda"</span>
...
</code></pre></div></div>
<p>A little explanation is in order: the format of the response is a JSON like this:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"logGroups"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"logGroupName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/aws/apigateway/welcome"</span><span class="p">,</span><span class="w">
</span><span class="nl">"creationTime"</span><span class="p">:</span><span class="w"> </span><span class="mi">1630789959982</span><span class="p">,</span><span class="w">
</span><span class="nl">"metricFilterCount"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nl">"arn"</span><span class="p">:</span><span class="w"> </span><span class="s2">"arn:aws:logs:eu-west-1:12345678910:log-group:defalt-vpc-flow-logs:*"</span><span class="p">,</span><span class="w">
</span><span class="nl">"storedBytes"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>The first part (<code class="language-plaintext highlighter-rouge">jq '.logGroups'</code>) chooses the content of the <code class="language-plaintext highlighter-rouge">.logGroups</code> key. The seconds unpacks the array and prints the items after each other. The final <code class="language-plaintext highlighter-rouge">.logGroupName</code> picks out the name of the log group.</p>
<p>To pretty-print JSON logs from AWS Lambda functions, one can run this little beast:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws logs <span class="nb">tail</span> <span class="s2">"<log-group-name>"</span> <span class="se">\</span>
<span class="nt">--since</span> 1d <span class="se">\</span>
<span class="nt">--format</span> short <span class="se">\</span>
| <span class="nb">sed</span> <span class="s1">'s/^.\{20\}//'</span>
| jq <span class="nt">-R</span> <span class="s2">"fromjson? | . "</span>
</code></pre></div></div>
<p>The first thing we do is to get all log messages in the last day from our log group. Since all log messages are prepended with the timestamp, we cut out the timestamp with <code class="language-plaintext highlighter-rouge">sed</code> (thanks, Google!). Then we parse all JSON logs and ignore the rest (<a href="https://stackoverflow.com/a/41599388/1013553">thank you Stackoverflow</a>).</p>
<p>If we want to see all error logs, we can just change the above command to</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws logs <span class="nb">tail</span> <span class="s2">"<log-group-name>"</span> <span class="se">\</span>
<span class="nt">--since</span> 1d <span class="se">\</span>
<span class="nt">--format</span> short <span class="se">\</span>
| <span class="nb">sed</span> <span class="s1">'s/^.\{20\}//'</span>
| jq <span class="nt">-R</span> <span class="s1">'fromjson? | . | select(.level =="ERROR")'</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">jq</code> also supports sorting and lots of much more advanced manipulation of JSON input. Take a look at the <a href="https://stedolan.github.io/jq/manual/">manual</a> for more.</p>One of my favorite command line tools is jq (homepage). It is often very useful to look at structured JSON data.How to connect to localhost from a Docker container2021-08-23T20:03:00+00:002021-08-23T20:03:00+00:00https://blog.fredrikmeyer.net//2021/08/23/docker-connect-to-host<p>At work I recently spent <em>a lot</em> of time trying to get a running Docker container to connect to a service running on <code class="language-plaintext highlighter-rouge">localhost</code>.</p>
<p>The setup is this: we have a Docker container that runs inside a Docker <a href="https://docs.docker.com/network/bridge/">bridge network</a> together with other Docker containers it can communicate with. On the host machine there is a service which listens on <code class="language-plaintext highlighter-rouge">127.0.0.1:8088</code>. For the purposes of this blog post, let’s just assume this service’s job is to stream messages to some server.</p>
<p>I wanted one of the containers to connect to the service running on <code class="language-plaintext highlighter-rouge">localhost</code>. In this post I will explain how I got it to work (only on Linux).</p>
<p>The picture is something like this:</p>
<p><img src="/assets/localhost_docker.png" alt="Docker Bridge Network" /></p>
<p>Docker containers inside the same bridge network can easily communicate with each other by using their container ID as the host name.</p>
<p>However, Docker Container A cannot easily talk to something on <code class="language-plaintext highlighter-rouge">localhost</code> on the host machine. There is even someone on <a href="https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach#comment109410097_24319662">StackOverflow</a> saying that this is not possible!</p>
<blockquote>
<p>Without network: host you can’t go back from a container to the host. Only host to container. This is the main ideology behind containers. They are isolated for both stability and security reasons.</p>
</blockquote>
<p>This is a truth with modifications. The way Docker manages its networking is <a href="https://docs.docker.com/network/iptables/">by suitably modifying</a> <code class="language-plaintext highlighter-rouge">iptables</code>. Which means that we also can do the same!</p>
<p>We are going to use <code class="language-plaintext highlighter-rouge">iptables</code> to route traffic to the bridge gateway inside a container to traffic on <code class="language-plaintext highlighter-rouge">localhost</code> on the host machine (the next steps are thanks to <a href="https://stackoverflow.com/a/57833573/1013553">this</a> very helpful StackOverflow answer). The IP of the bridge gateway can be found by running <code class="language-plaintext highlighter-rouge">docker network inspect <bridge name></code>. To not have to worry about the actual IP, you can start the container with <code class="language-plaintext highlighter-rouge">--add-host host.docker.internal:host-gateway</code>. This will map the host name <code class="language-plaintext highlighter-rouge">host.docker.internal</code> to the IP of the bridge gateway.</p>
<p>First we have to find the name of the Docker bridge network. This is not the same as the ID you see when you do <code class="language-plaintext highlighter-rouge">docker network ls</code>, but almost. If you use the default bridge network, then the name should be <code class="language-plaintext highlighter-rouge">bridge</code>. If you made your own network, the name will be <code class="language-plaintext highlighter-rouge">br-<first 12 chars of network ID></code> (see the <a href="https://github.com/moby/libnetwork/blob/b3507428be5b458cb0e2b4086b13531fb0706e46/drivers/bridge/bridge.go#L551">original code</a>). You can also get the name of the bridge by running <code class="language-plaintext highlighter-rouge">ifconfig</code>.</p>
<p>You can set the bridge name yourself when you create the network with the following command (<a href="https://stackoverflow.com/a/43981857/1013553">source</a>): (just remember that it can be at most 15 characters long)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker network create --opt com.docker.network.bridge.name=<my_name> test-net
</code></pre></div></div>
<p>We need to allow routing from the bridge to localhost:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sysctl -w net.ipv4.conf.<bridge_name>.route_localnet=1
</code></pre></div></div>
<p>Note that this has some <a href="https://security.stackexchange.com/questions/137602/what-are-the-security-implications-of-net-ipv4-conf-eth0-route-localnet-1-rout">mild security implications</a>.</p>
<p>The next step is to use <code class="language-plaintext highlighter-rouge">iptables</code> to forward traffic to the bridge gateway to localhost.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>iptables -t nat -A PREROUTING -p tcp -i <bridge-name> --dport 8088 -j DNAT --to-destination 127.0.0.1:8088
</code></pre></div></div>
<p>Once this is done, you can connect to the service on <code class="language-plaintext highlighter-rouge">localhost</code> on the host machine by calling <code class="language-plaintext highlighter-rouge">host.docker.internal:8088</code>.</p>
<p>Happy coding!</p>At work I recently spent a lot of time trying to get a running Docker container to connect to a service running on localhost.Github Action for checking if a changelog is updated2021-05-12T21:01:00+00:002021-05-12T21:01:00+00:00https://blog.fredrikmeyer.net//2021/05/12/github-actions-changelog<p>At a work project I’m working on, we work in a monorepo. We have a simple way to keep track of changes: a <code class="language-plaintext highlighter-rouge">version.txt</code> and a <code class="language-plaintext highlighter-rouge">CHANGELOG.md</code> in the root of the repository, that should be updated for every merge to master.</p>
<p>It is easy to forget this, so I wrote a little <a href="https://github.com/features/actions">Github Action</a> check that checks if these two files have been updated.</p>
<p>To use Github Actions, just put a workflow recipe in <code class="language-plaintext highlighter-rouge">.github/workflows/</code>.</p>
<p>Here is a workflow to check if both these files have changed:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Check changelog and version.txt</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">pull_request</span><span class="pi">:</span>
<span class="na">types</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">review_requested</span><span class="pi">,</span> <span class="nv">ready_for_review</span><span class="pi">,</span> <span class="nv">synchronize</span><span class="pi">]</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">check-changelog-version-txt</span><span class="pi">:</span>
<span class="na">timeout-minutes</span><span class="pi">:</span> <span class="m">10</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">fetch-depth</span><span class="pi">:</span> <span class="m">0</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Check if files have changed</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">FILES_CHANGED=$(git diff --name-only origin/master...HEAD | grep -E 'CHANGELOG\.md|version\.txt' -c)</span>
<span class="s">if [ "$FILES_CHANGED" != "2" ]; then echo "Remember to update CHANGELOG.md and version.txt"; exit 1; fi;</span>
</code></pre></div></div>
<p>This check runs for every pull requests and every commit on that PR. The first step checks out the repository. Note that we have to set <code class="language-plaintext highlighter-rouge">fetch-depth: 0</code>. This fetches all Git history and not just the current commit: it is needed to be able to run <code class="language-plaintext highlighter-rouge">git diff</code>.</p>
<p>Then in the next step, we run a <code class="language-plaintext highlighter-rouge">git diff</code> against <code class="language-plaintext highlighter-rouge">origin/master</code> to get the list of changes files. We then filter anything that matches <code class="language-plaintext highlighter-rouge">CHANGELOG.md</code> or <code class="language-plaintext highlighter-rouge">version.txt</code> (the <code class="language-plaintext highlighter-rouge">-E</code> flag allows us to use regular expressions with <code class="language-plaintext highlighter-rouge">grep</code>), and then count the the result (that’s the <code class="language-plaintext highlighter-rouge">-c</code> flag).</p>
<p>If not both of these files have changed, we return a non-zero exit code, and your build will automatically fail :).</p>
<p>Note: there is already a publicly available action that does something similar, see <a href="https://github.com/tj-actions/changed-files">here</a>. The reason I didn’t go for that one was partly for security reasons (in principle, the action can read and copy all your code), and partly because I wanted something simple.</p>At a work project I’m working on, we work in a monorepo. We have a simple way to keep track of changes: a version.txt and a CHANGELOG.md in the root of the repository, that should be updated for every merge to master.K3 surfaces, definition and a few examples2021-01-02T18:50:00+00:002021-01-02T18:50:00+00:00https://blog.fredrikmeyer.net//2021/01/02/k3-intro<p>I’m reading Daniel Huybrechts’ <a href="https://www.math.uni-bonn.de/people/huybrech/K3Global.pdf">book on K3 surfaces</a> now (because I miss doing mathematics), and so I thought I could write some short blog posts about it as I read through it.</p>
<p>We’ll start off with the definition. I’ll assume we’re working over $\mathbb C$ unless I say something else. Then a K3 surface is a complete non-singular variety $X$ of dimension $2$ such that $\Omega_{X/\mathbb C}^2 \simeq \mathcal O_X$ and $H^1(X, \mathcal O_X) = 0$ (the last condition is to separate K3 surfaces from Abelian surfaces). In a sense, they are <em>Calabi-Yau varieties of dimension 2</em>.</p>
<h2 id="the-easiest-example">The easiest example</h2>
<p>The by far easiest example of a K3 surface is a smooth quartic in $\mathbb P^3$. Indeed:</p>
<p>[
0 \to \mathcal O_{\mathbb P^3} (-4) \to \mathcal O_{\mathbb P^3} \to \mathcal O_X \to 0,
]</p>
<p>which gives us $H^1(X, \mathcal O_X) = 0$ by chasing some long exact sequences. The triviality of the canonial bundle can be found by the <a href="https://en.wikipedia.org/wiki/Adjunction_formula">adjunction formula</a>: $\omega_X = \omega_{\mathbb P^3} \otimes \mathcal O(4) \simeq \mathcal O(-4) \otimes \mathcal O(4) \simeq \mathcal O_X$.</p>
<p>An interesting special case is given by the <em>Fermat quartic</em> defined by $x_0^4+x_1^4+x_2^4+x_3^4=0$. It has a lot of discrete symmetries, given by scaling the coordinates by fourth roots of unity.</p>
<h2 id="another-example-not-so-easy">Another example, not so easy</h2>
<p>Another class of examples comes from double coverings of $\mathbb P^2$, branched along a curve of degree $6$. We can be very explicit: let $\mathbb P = \mathbb P(1,1,1,3)$ be a <em>weighted projective space</em> (this is defined by dividing out by $\lambda \cdot (a,b,c,d) \sim (\lambda a, \lambda b, \lambda c, \lambda^3 d)$). Let $f \in k[x,y,z]$ be a degree $6$ polynomial. Then let $X$ be the zeroes of $w^2=f(x,y,z)$ in $\mathbb P$, where we view $f$ as a polynomial in $k[x,y,z,w]$ in the natural grading.</p>
<p>This is a homogeneous polynomial in the grading on $\mathbb P$, so $X$ is well-defined as a variety in $\mathbb P$. We have a natural map $\pi: X \to \mathbb P^2$ by forgetting the last coordinate. It is easy to see that $X$ is smooth as long as $C=Z(f) \subset \mathbb P^2$ is smooth.</p>
<h3 id="little-aside">Little aside…</h3>
<p>Here Huybrechts uses the following fact:</p>
<blockquote>
<p>If $\pi: X \to Y$ is an affine morphism and $\mathscr F$ is a coherent sheaf on $X$, then $H^i(X, \mathscr F) \simeq H^i(Y, \pi_\ast \mathscr F)$.</p>
</blockquote>
<p>I have seen this many times, but never seen a proof. Looking it up, it is (of course) an exercise in Hartshorne. Turns out the proof is easy, so here’s a sketch: cover $Y$ by affines $U_i$. Then $\pi^{-1}(U_i)$ is an affine covering of $X$ since $\pi$ is an affine morphism. Then one can compare the Čech complexes of $\mathscr F$ and $\pi_\ast \mathscr F$ using this covering, and observe that they are in fact equal. Then since $\mathscr F$ is quasi-coherent, Čech cohomology computes sheaf cohomoogy, and so the the statement follows.</p>
<h3 id="back-to-the-example">Back to the example…</h3>
<p>Now a local computation (or by some other means?) we see that $\pi_\ast \mathcal O_X \simeq \mathcal O_{\mathbb P^2} \oplus \mathcal O_{\mathbb P^2}(-3)$ (one way is to notice that the projective coordinate ring of $X$ is $k[x,y,z,w]/(f(x,y,z)-w^2)$, which is equal to $k[x,y,z] \oplus k[x , y ,z] (-3)$ as a $k[x,y,z]$-module. But I’m not positive that this is a valid argument).</p>
<p>But now by the aside above, this lets us see that $H^1(X,\mathcal O_X) \simeq H^1(\mathbb P^2, \mathcal O_{\mathbb P^2} \oplus \mathcal O_{\mathbb P^2}(-3)) = 0$.</p>
<p>Finally, we must argue that the canonical bundle on $X$ is trivial. This follows from Hurwitz’ formula for branched coverings, which says that $K_X=\pi^\ast(K_Y) + R$, where $K_X$ is the class of the canononical line bundle on $X$, and $R$ is the ramification divisor on $X$. The map ramifies along the curve defined by $f=0$. On $X$, this is defined by the zeros of $w=0$. But $w$ has degree 3 in the graded coordinate ring of $X$, so $\mathscr O_X(-R)=\mathscr O_X(-3)$. On the other hand $K_Y=-3H$, and it follows that $K_X=-3H’ + 3H’ = 0$, where $H’$ is the class of a hyperplane in $X$.</p>
<p>Now we have two examples of K3 surfaces. As I progress further in the book, I hope to write more blog posts explaining other interesting stuff about these fascinating creatures.</p>I’m reading Daniel Huybrechts’ book on K3 surfaces now (because I miss doing mathematics), and so I thought I could write some short blog posts about it as I read through it.A topological space that is not first countable2020-12-14T06:58:00+00:002020-12-14T06:58:00+00:00https://blog.fredrikmeyer.net//2020/12/14/first-countable<p>Recently me and some friends were discussing <a href="https://en.wikipedia.org/wiki/First-countable_space">first countability</a>, and noticed that Wikipedia gave an example of space that was <em>not</em> first countable, but with no proof. So here’s my attempt at a proof.</p>
<p>Let $X=\mathbb R / \mathbb N$, where we identify $1 \simeq 2 \simeq \ldots$. Geometrically, this space is an “infinite wedge product” of circles. It looks like a flower with a countable infinite number of “petals”.</p>
<p>The open sets in $X$ are of two types: those containing $[1]$ (the equivalence class of $1 \in X$), and those not containing $[1]$. The latter ones are just shorts intervals not containing any natural numbers. Let $\pi: \mathbb R \to X$ be the projection map. Note that if $[1] \in V$, then $\mathbb N \subset \pi^{-1}(V)$ (since $\pi^{-1}([1])=\mathbb N$, by definition of $X$). This implies that any open set in $X$ containing $[1]$ must intersect all petals (and so must be a union of intervals around each natural number).</p>
<h1 id="proposition-x-is-not-first-countable">Proposition: $X$ is not first countable.</h1>
<p>Recall that a space is first countable if every point $x \in X$ has a countable <a href="https://en.wikipedia.org/wiki/Neighbourhood_system#Basis">neigbourhood basis</a>. A neigbourhood basis is a collection of open subsets that is “arbitrarily small”.</p>
<p>Note that we can consider a neigbourhood basis as indexed by the natural numbers, i.e. a function $f: \mathbb N \to \mathscr P(X)$. We will denote $f(i)$ by $N_i$, and its preimage in $\mathbb R$ by $N_i’$.</p>
<p>We will construct an open set $U \subset X$ such that $N_i \not \subset U$ for all $i$ given any proposed neigbourhood basis ${ N_i }$ around $[1]$.</p>
<p>For $N_1$, since $\mathbb N \subset N_1’$, we can find a small interval $I_1’$ in $\mathbb R$ around $1 \in \mathbb R$ strictly contained in $N_1’$ (make it small enough so that it doesn’t intersect any other natural numbers).</p>
<p>We continue: choose a small interval $I_2’$ around $2 \in \mathbb R$ strictly contained in $N_2’$.</p>
<p>And so on… We define $U’$ by taking the union of all the $I_n’$’s. We denote its image in $X$ by $U$. Note that $U=\pi(U’)$ is also open, since $\pi^{-1}(\pi(U))=U’$. Geometrically, $U$ looks like a small interval around every petal on $X$, made in such a way that $N_i \not \subset U$ for every $i$.</p>
<p>We conclude that $X$ cannot be first countable, since given any indexed collection of open subsets containing $[1] \in X$, we can find an open $U$ that is “too big”.</p>
<h1 id="comments">Comments</h1>
<p>Note that this proof wouldn’t work if we took a <em>finite</em> wedge product of circles, because then the countable neigbourhood basis could be big enough.</p>
<p>Also note that the technique in the proof is basically <a href="https://en.wikipedia.org/wiki/Cantor%27s_diagonal_argument">Cantor’s diagonal argument</a>.</p>Recently me and some friends were discussing first countability, and noticed that Wikipedia gave an example of space that was not first countable, but with no proof. So here’s my attempt at a proof.Making Emacs use correct Python interpreter in a virtual environment2020-08-26T19:33:00+00:002020-08-26T19:33:00+00:00https://blog.fredrikmeyer.net//2020/08/26/emacs-python-venv<p>I like to be able to test things interactively in a Python shell when exploring a new package. I also don’t like to exit Emacs when I don’t have to. So I’d like to be able to run <code class="language-plaintext highlighter-rouge">C-c C-p</code> (or <code class="language-plaintext highlighter-rouge">run-python</code>) directly from Emacs.</p>
<p>I use the <code class="language-plaintext highlighter-rouge">pyvenv</code> package to manage virtual environments within Emacs. It works well, but it doesn’t automaticlly set the correct Python interpreter that I expect when I change virtual environments.</p>
<p>So I added this to my <code class="language-plaintext highlighter-rouge">init.el</code>:</p>
<div class="language-elisp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="nb">use-package</span> <span class="nv">pyvenv</span>
<span class="ss">:ensure</span> <span class="no">t</span>
<span class="ss">:config</span>
<span class="p">(</span><span class="nv">pyvenv-mode</span> <span class="no">t</span><span class="p">)</span>
<span class="c1">;; Set correct Python interpreter</span>
<span class="p">(</span><span class="k">setq</span> <span class="nv">pyvenv-post-activate-hooks</span>
<span class="p">(</span><span class="nb">list</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">()</span>
<span class="p">(</span><span class="k">setq</span> <span class="nv">python-shell-interpreter</span> <span class="p">(</span><span class="nv">concat</span> <span class="nv">pyvenv-virtual-env</span> <span class="s">"bin/python3"</span><span class="p">)))))</span>
<span class="p">(</span><span class="k">setq</span> <span class="nv">pyvenv-post-deactivate-hooks</span>
<span class="p">(</span><span class="nb">list</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">()</span>
<span class="p">(</span><span class="k">setq</span> <span class="nv">python-shell-interpreter</span> <span class="s">"python3"</span><span class="p">)))))</span>
</code></pre></div></div>
<p>Now, when I press <code class="language-plaintext highlighter-rouge">C-c C-p</code>, the Python shell interpreter that starts is the one in the virtual environment.</p>
<p>So my workflow is this: activate the virtual environment with <code class="language-plaintext highlighter-rouge">M-x pyvenv-activate</code>, experiment in a Python shell (started with <code class="language-plaintext highlighter-rouge">C-c C-p</code>), profit.</p>
<p>(I use <code class="language-plaintext highlighter-rouge">lsp-mode</code> and <a href="https://github.com/palantir/python-language-server">Python Language Server</a> for IDE-like features, but that is another post…)</p>I like to be able to test things interactively in a Python shell when exploring a new package. I also don’t like to exit Emacs when I don’t have to. So I’d like to be able to run C-c C-p (or run-python) directly from Emacs.Proof of concept chat app2020-04-02T08:05:00+00:002020-04-02T08:05:00+00:00https://blog.fredrikmeyer.net//2020/04/02/hobby-project<p>I had some time in between projects at work, so I have spent some spare time trying to learn new stuff. I have for a while wanted to learn about web sockets, so I decided to try to write a simple chat app. Turns out, I had to learn about a lot of other stuff as well. This is a short write up to summarise (mostly) for myself what I’ve learned.</p>
<p>I chose Clojure as the backend language, because I have a fascination for languages in the LISP (<em>list processing</em>) family. On the frontend I used <a href="https://create-react-app.dev/">Create React App</a> to bootstrap a React + <a href="https://www.typescriptlang.org/index.html">Typescript</a> app with sane defaults.</p>
<p>Before I explain the implementation, open it <a href="https://gifted-antonym-271008.firebaseapp.com/">here</a>. Try to open it in two browser windows, and type something.</p>
<h2 id="technologies-used">Technologies used</h2>
<p>To get some context and list some buzzwords, here’s the complete tech stack. The backend is written in <a href="https://clojure.org/">Clojure</a> running in a <a href="https://www.docker.com/">Docker</a> container on <a href="https://cloud.google.com/appengine">Google App Engine</a>. It hosts a <a href="https://graphql.org/">GraphQL</a> server using the <a href="https://github.com/walmartlabs/lacinia">Lacinia</a> Clojure library. Incoming chat messages are pushed onto a <a href="https://redis.io/">Redis</a> pub/sub stream.</p>
<p>The frontend is written in Typescript and React, hosted on Google Firebase. It uses the <a href="https://www.apollographql.com/">Apollo GraphQL</a> library to talk with the backend.</p>
<p>The code is at <a href="https://github.com/fredrikmeyer/chat-app/">Github</a>, and I use <a href="https://github.com/features/actions">Github Actions</a> to automatically deploy any changes to the backend and frontend directories.</p>
<h2 id="architecture">Architecture</h2>
<p>Below is a sketch of the architecture:</p>
<p><img src="/assets/chat_app_structure.png" alt="Chat app architecture" /></p>
<p>The frontend is relatively straight forward (at least with the current functionality). The only UI is a input field for the name of sender, and a input field for the message. New messages are received by subscribing to GraphQL subscriptions.</p>
<p>What complicates the backend most is actually the choice of hosting. The backend lives in Docker containers on Google App Engine (GAE), but GAE does automatic scaling of the number of running instances. This had the weird effect that users only saw some of the messages received, because it was essentially random which instance your request would connect to.</p>
<p>The solution was to use a single source of truth, and because GAE has Redis support, I chose Redis for the job. Every time a message is received, it is pushed onto a pub/sub Redis channel. Then a subscribed client (perhaps the same) reads the message, and pushes it onto the GraphQL subscription. This essentially makes the Clojure part a stateless server - no state is ever saved anywhere.</p>
<h2 id="lessons-learned">Lessons learned</h2>
<p>The answer to the question “It can’t be that hard, can it?” is always yes. There were several technologies I knew, but not well enough to avoid Googling a lot of solutions. The number of technologies involved probably increased the time from start to finish to some degree.</p>
<h2 id="onwards">Onwards</h2>
<p>Technically, this is a working chat app. But it is not good. It would be fun to add authentication, chat rooms, and a prettier UI.</p>I had some time in between projects at work, so I have spent some spare time trying to learn new stuff. I have for a while wanted to learn about web sockets, so I decided to try to write a simple chat app. Turns out, I had to learn about a lot of other stuff as well. This is a short write up to summarise (mostly) for myself what I’ve learned.