Since blocktogether.org is going away, there is a need for people to create new instances of @j4cob‘s code if the concept of shared blocklists is to continue. I remember when I opened the code for @TheBlockBot I wanted people to create their own block bots ideally, but there was a barrier to getting it to work – no install instructions (As well as my code being terrible!)

So… This is the procedure I followed to re-purpose theblockbot.com as a blocktogether instance. Includes the code I forked for my version and the set up prerequisites on Ubunti 20.04 server edition. I’ve installed this on a low power server at home, but if you want to install in a cloud based instance you can – the memory requirements are not insignificant though, so it may be pricey – my home server is an 8GB one. As my installation runs from my home broadband effectively, I’ve included instructions on how to set up cloudflare to proxy connections and hide my origin IP address.

Main change – /settings is password protected and the “share your blocklist” is only on that page. This is to only allow people I know to create blocklists on this server. It’s low powered so I can’t make it a full on blocktogether.org clone where anyone can create a blocklist!

Set up a small BT node on a local web server, using your router. (Not recommended for hosting lots of load / lots of blocklists)
– Assumption is you have a server / PC with Ubuntu 20.04 installed on it and port 80/443 forwarded by your router to this server, check your router instructions on how to do this. Setting a static IP for it in your router is probably a good idea.
– It is also assumed your domain is set up with the IP set to your home IP address (whatsmyip.com will tell you) – as part of these instructions we will be changing management of your domain to Cloudflare, you can do that now if you wish.

1. Install node 10 …
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash
sudo apt install nodejs
2. Install mysql
sudo apt-get install mysql-server
sudo mysql_secure_installation utility
– Add root password
– rest is for security, set as appropriate
3. Add blocktogether user and table to mysql
mysql -u root -p
– Enter root password
CREATE USER ‘blocktogether’@’localhost’ IDENTIFIED BY ‘password’;
CREATE DATABASE IF NOT EXISTS blocktogether;
GRANT ALL PRIVILEGES ON blocktogether.* TO ‘blocktogether’@’localhost’;
ALTER USER ‘blocktogether’@’localhost’ IDENTIFIED WITH mysql_native_password BY ‘password’
– Above is for mysql v8, sequelize client needs a native password format to communicate with mysql … Replace “password” with your password!
4. Install nginx
sudo apt install nginx
5. in /etc/ clone git repo
git clone https://github.com/jdbillingham/blocktogether.git
– make sure a local non-root user owns the files
– Or clone your own fork
– Code should be located at /etc/blocktogether
6. Copy files from /etc/blocktogether/config/etc/nginx to nginx directory
– Assuming edited to change “theblockbot.com” to your domain
– If you base on blocktogether one make sure you delete the SSL bit, certbot will add that
cd /etc/blocktogether/
sudo cp -r config/etc/nginx/ /etc/nginx/
– Test your config
sudo nginx -t
– Reload nginx if it is OK
sudo systemctl reload nginx
7. Set up LetsEncrypt and get a cert for your domain
– These instructions work – https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-20-04
– Need to make sure your domain is pointing to the server now, DNS away!
– Check www.domain.com and domain.com are pointing to your IP.
8. Check certbot renewal will work with –
sudo certbot renew –dry-run
9. If not done already – Set up cloudflare to manage your domain, we want to hide your home IP so it can’t be DDoS’d etc
– It’s free! https://dash.cloudflare.com/sign-up
– SSL tab in control panel needs to be set to Full (Strict) or you’ll get an infinite redirect.
– Make sure the records are set to “proxied” so it goes through Cloudflare
10. Create maintenance.html to test domain/SSL
– vi /etc/blocktogether/static/maintenance.html and add a message
– Should now be able to access your domain over SSL and see the maintenance message come up!
11. We want to password protect the /settings url, to stop just anyone creating a blocklist (This is likely not perfect, TODO come up with a more secure way of managing this). The standard settings page has the “Share your blocklist” link disabled.
– sudo apt install apache2-utils
– htpasswd -c /etc/blocktogether/static/.htpasswd <user>
– Choose a password and save it in your password management tool
– Check the nginx config, a password is defined for the /settings URL
12. As we are using home router, the DNS record could become invalid on reboot of router and a new IP being allocated. No need to do this if you have a static IP at home.
– in /home run – git clone https://github.com/jdbillingham/Cloudflare-Public-Dynamic-IP-Update.git
13. Above requires jq
– sudo apt-get install jq
14. Edit the cloudflare_config file in the cloned directory
– Add your own API key, Domain, Domain Zone is on the cloudflare dashboard home bottom right and email address.
– Test by running -> ./cloudflare_dynamic_ip_update.bash
15. Add cronjob to run this every hour (Or more if you want)
– crontab -e
– Add -> * 0 * * * /bin/bash /home/james/Cloudflare-Public-Dynamic-IP-Update/cloudflare_dynamic_ip_update.bash >> /home/james/Cloudflare-Public-Dynamic-IP-Update/cloudflare.log 2>&1
– EDIT the above to your own home directory
– TODO: Need logrotate/other to clean up logs
16. in /etc/blocktogether run
npm -i
– to install dependancies
17. Run in /etc/blocktogether – to set up DB ->
./node_modules/.bin/sequelize –config config/sequelize.json db:migrate
18. Create a config.json in /etc/blocktogether/config
– Need to edit to add your callback URL, the development one is “localhost:3000”, yours needs to be your domain. This also needs to be set up in your Twitter app as a valid callback – with SSL
19. Run blocktogether … in /etc/blocktogether
nohup ./run-dev.sh >> /etc/blocktogether/logs/run-dev.log 2<&1 &
– TODO use something else to run in a more production like manner…
20. Remove the maintenance HTML
mv /etc/blocktogether/static/maintenance.html /etc/blocktogether/static/maintenance.html.old

That’s it! 21 steps and you have your own slightly hacky blocktogether instance running. It needs work, no logrotate, I was going to use “forever” to run the node scripts and manage restart etc. But couldn’t get that working and thinking about it just using Jacob’s init scripts would probably be better anyway. Not convinced it is very secure to just add a basic auth login for /settings, or that exposing the key for rpc in github is a very good idea. But as it is all locked down with only port 80/443 open on my router, I should be OK 😀

BUG – So far the blocks that are pulled in for a user don’t ever seem to update beyond just Twitter IDs in “show-blocks”. Can’t see any errors, but presumably something in update-users.js not working correctly…
^^ This seems to have fixed itself, updating nicely now!
^^ NOW not updating nicely for a new blocklist just added, see below

There are probably bugs in this set up doc, I’ve not been through it all from a clean slate to test it. I did have to go back a couple of times and update the instructions. Let me know @oolon

Updates!
1. Running with the /config/development log4js was a mistake, updated the git so the production one is copied over the top of this. Downside of running in “development” mode. Hammering the disks with log writing seemingly killed the processes somehow.

2. Log rotate, all you need to do now is –
cp /etc/blocktogether/etc/logrotate.d/blocktogether /etc/logrotate.d

3. Systemd, turn the node processes into services so they are auto-restarted
sudo cp config/etc/systemd/system/* /etc/systemd/system/
sudo systemctl enable update-users.service
sudo systemctl enable update-blocks.service
sudo systemctl enable deleter.service
sudo systemctl enable blocktogether.service
sudo systemctl enable actions.service
– This also changes the mode it runs in to “production”, although the only difference in the code is how it caches mustache templates and production uses a different sequelize config. The production blocktogether one was pointing to a remote mysql server instance, mine is on the same server so configured the same as development mode.

4. BUG, the update-users.js bug mentioned above has reappeared, any uids inserted into the TwitterUsers tables are updated by this script to get the full info on the users. BUT for some reason when adding a shared blocklist these IDs are not being pushed from Blocks -> TwitterUsers. Work-around at the moment is just to manually push them in with mysql –
Find the “BlockBatchID” for the shared blocklist and run:
Insert into TwitterUsers (uid,createdAt,updatedAt) Select distinct sink_uid,now(),now() from Blocks Where BlockBatchId = “xxx” and sink_uid not in (select uid from TwitterUsers);

Get block batch ID –
Select * from BtUsers where shared_blocks_key is not null;
– get uid for your list
Select * from BlockBatches where source_uid = “<uid>”
– The block batch is the latest one by date

NB – I found after some time running the Actions table can get very large, making sure any over 30 days are pruned and bin logs are off seems to have fixed the stability problems. I found I was having to truncate the table to get the processes to start again – or they’d get stuck in 100% CPU.