diff options
Diffstat (limited to 'blog/2021-07-15.md')
| -rw-r--r-- | blog/2021-07-15.md | 141 |
1 files changed, 141 insertions, 0 deletions
diff --git a/blog/2021-07-15.md b/blog/2021-07-15.md new file mode 100644 index 0000000..f53009e --- /dev/null +++ b/blog/2021-07-15.md @@ -0,0 +1,141 @@ +# CircleCI :circleci: in daily life: how I improved Matrix stickerpicker + +[CircleCI](https://circleci.com) is a great tool that everyone seems to skip and I don't know exactly why. +I'm gonna quickly show you how I've automated adding stickers to the [Matrix](https://matrix.org) messenger. + +## What is CircleCI? +`CircleCI` is a cloud platform that can run automated jobs when different events occur. The most common use-case is +running tests in the cloud when you push a commit to the `GitHub` repo. But the possibilities are endless, +you can literally automate everything with it - I even used it to compile `.exe` file **in the cloud** and include +a **zipped binary** into the release package. Don't ask me why I had to compile `.exe`, better ask why the guys from my +University **made a deal with** [**the Devil**](https://microsoft.com). + +## Matrix +[Matrix](https://matrix.org) is a messaging protocol which I'm currently trying to replace **Telegram** with. +It's in itself a separate topic for discussion, but what's important for today is that **Matrix** lacks such +a huge collection of stickers as there is in **Telegram**. I'm a big fan of **Wojak** stickers and many other ones, +so I had to do something about it. + + + +### Matrix Stickerpicker +Stickers work in a slightly weird way in **Matrix**. Basically you have to deploy your own sticker "server" +(*don't worry, it's just a static site*) where you can add your stickers - it will be used as a widget inside your client. + +There's an amazing tool for that - [stickerpicker](https://github.com/maunium/stickerpicker). It's written in `Python` and the author +of course took care of the import feature - you can provide a link to a **Telegram** sticker pack - it will re-upload it +to your `Matrix` server and then generate a web UI using GitHub Pages that you can integrate as a widget in your client. Your friends +can use it too. + +The problem is that each time you want to add new sticker pack to your collection you have to run a `Python` script. +I'm lazy and I don't wanna do that - so let's automate it! + +## Automation +Main idea of my solution is to have a single plaintext file `telegram-packs.txt` which will contain links to all +sticker packs I import from **Telegram**. Then every time I update it `CircleCI` will run a job that will run this `Python` script for me. Deadly simple. + +This approach ships with few bonuses by default: + - You can edit the file from anywhere (even from phone :laughing:) + - If someone from your friends uses your stickers collection and wants to add a new pack, they simply have to open a PR which adds the pack URL to `telegram-packs.txt`. + +There are two problems of running this `Python` script automatically: + - You have to log into your `Matrix` account + - You have to log into your `Telegram` account + +### Config +Here's my fork of the repo: https://github.com/eug-vs/stickerpicker + +`CircleCI` is fully controlled by a config file `.circleci/config.yml`. It uses `YAML` which is really picky about indentation, so be careful with that. + +Ok so let's start outlining our job, we'll be using default CircleCI `Python` image: +```YAML +defaults: &defaults + working_directory: ~/repo + docker: + - image: cimg/python:3.9.5 + +jobs: + upload_stickers: + <<: *defaults + steps: +``` + +The `defaults` part might look scary, but you could just move everything from defaults directly under `upload_stickers` job. +Defaults are useful when you have multiple jobs, and I planned to have 2 initially. Let's now move to the `steps`: +```YAML + - checkout + + - run: + name: Install python dependencies + command: pip3 install . +``` +Obviously we checkout our code first, and then install all the dependencies for our `Python` script. + +Now let's solve the first problem we found. Instead of logging into `Matrix` manually, you can supply a `config.json` file with your credentials. So I prepared a template for this file (`config.template.json`): +```JSON +{"homeserver": "$HOMESERVER", "user_id": "$USER_ID", "access_token": "$ACCESS_TOKEN"} +``` +You might think how am I gonna put the actual variables in here? Well, there's a tool for that called `envsubst`. It does exactly what you think - substitutes environment variable names with the actual values. +We will put the actual values of `$HOMESERVER`, `$USER_ID` and `$ACCESS_TOKEN` in our `CircleCI` environment (as secrets), and the template can be safely pushed to the GitHub repo. + +There's a little problem - `envsubst` is not installed in the image that we use (`cimg/python:3.9.5`). Solution to it is [orbs](https://circleci.com/developer/orbs) - basically plugins for your CircleCI needs. +Let's add `envsubst` orb at the top of our config: +```YAML +orbs: + envsubst: sawadashota/envsubst@1.1.0 +``` + +Now back to the steps. Let's install the tool and use it to generate our `config.json`: +```YAML + - envsubst/install + + - run: + name: Create config.json from template + command: envsubst < config.template.json > config.json +``` + +Ok so we can now login into `Matrix`, but what about `Telegram`? The script uses `Telethon` package under the hood which works with a `sticker-import.session` file. This is basically an `SQL` database, and I don't really +wanna mess around with creating it from scratch (*although I could!*). Instead, I will just encrypt it with `GPG` and push to the repo. That's just me being lazy, but you know, that is what automation is about! +Of course, someone could try to decrypt my `Telethon` session since it's now publicly available, but I can just revoke it at any time. I will probably improve this at some point, but now it's not a big deal for me. + +Our next steps in config will be to decrypt the session and finally pipe our `telegram-packs.txt` to the script using `xargs`: +```YAML + - run: + name: Decrypt telethon session + command: gpg --batch --passphrase $PASSPHRASE --decrypt sticker-import.session.gpg > sticker-import.session + + - run: + name: Reupload and install stickers + command: cat telegram-packs.txt | xargs sticker-import +``` + +At this point, the stickers are uploaded to the server, but the UI is updated via it's own `.json` file which contains information about uploaded stickers. It's already updated by the script, we just have to push it to our repo. +For that I've created a new SSH key and added it to `CircleCI` secrets so that it can push commits to my repository. So the final steps would be: +```YAML + - add_ssh_keys + + - run: + name: Add github to known_hosts + command: | + mkdir -p ~/.ssh + ssh-keyscan github.com >> ~/.ssh/known_hosts + - run: + name: Commit stickerpack JSON's + command: | + git config --global user.email "eug-vs@keemail.me" + git config --global user.name "eug-vs" + git add . + git commit -m "feat: add stickerpacks with CircleCI [ci skip]" + git push -u +``` + +The `[ci skip]` flag tells Circle to avoid running on this commit, otherwise it would have to run once again since we pushed a new commit with it. + +One thing that I didn't mention is that I did not actually +need to check the **new** packs in the `telegram-packs.txt` - the script is actually smart enough to skip already existing ones. Otherwise I would have to use `git diff` to find only the new ones. + +And that's it! Now every time I add new URLs to the `telegram-packs.txt`, they will automatically appear in my widget within a minute or so. Enjoy your stickers! + + + +You can find full version of the config here: https://github.com/eug-vs/stickerpicker/blob/master/.circleci/config.yml |