Using Celery with Rabbitmq

Rabbitmq is a message broker and celery is a task queue. When you run a celery app, by default, it will open as many processes as there are cores of cpu on the machine. These processes are workers. When you have a task which needs to be done outside of a normal HTTP request-response cycle,  you can use a task queue. Rabbitmq can be configured to decide (and deliver) which worker the task has to go and celery will help in the actual execution of the tasks.

Celery supports a lot of message brokers but, Rabbitmq is the default one. So, setting up celery for using with rabbitmq doesn’t require any effort. If you have rabbitmq already installed then all you need to do is create a rabbitmq user, a virtual host and give the user access to the virtual host. It is given in the celery documentation here.

Then you need to specify the broker url in this format in the celery app.

broker_url = 'amqp://myuser:mypassword@localhost:5672/myvhost'

The default exchange that celery listens to is named ‘celery‘ and routing key is also ‘celery‘. The ‘celery‘ exchange is direct type exchange. AMQP is the protocol that rabbitmq follows. Username, password and virtual host here is of rabbitmq that you want celery to use. Based on the given broker url, celery attempts to know which message broker is being used.

Advertisements

Using Syntastic for Python development

I use Synstastic plugin of vim for syntax checking in vim. Syntastic offers syntax checking for a LOT of languages. But, there is a problem that i had been facing with it. For a file with larger than 4k lines, it takes a lot of time to check the syntax and it used to happen every time you save the file. Syntax checking on write operation is the default behavior.

So, i did some changes in my .vimrc so that i could still use Syntastic for larger files. Do note that syntastic checking still takes a long time but, i have configured it to be called whenever i want to rather than on every write operation or opening of file.

” show list of errors and warnings on the current file
nmap <leader>e :Errors<CR>
” Whether to perform syntastic checking on opening of file
” This made it very slow on open, so don’t
let g:syntastic_check_on_open = 0
” Don’t check every time i save the file
” I will call you when i need you
let g:syntastic_check_on_wq = 0
” By default, keep syntastic in passive mode
let g:syntastic_mode_map = { ‘mode’: ‘passive’ }
” Use :Sc to perform syntastic check
:command Sc :SyntasticCheck
” Check pylint for python
let g:syntastic_python_checkers = [‘pylint’]
” For jsx – React and React native
let g:syntastic_javascript_checkers = [‘eslint’]

This change made opening of a larger python file ~25s (yes, seconds) faster. It still takes a lot of time for syntax checking though. I will have to find out why and if i could do anything about it. I don’t want to leave out this plugin because it offers so much. I could simply use Python-mode for python syntax checking but, what about the rest of the languages which i am going to use.

Sending Emails using Django and Sendgrid

Recently, i setup a django app which uses  sendgrid to send emails. I will go through the steps in this short blog.

  1. Register at sendgrid
  2. Choose SMTP for sending emails
  3. Get an API key. The last step will redirect you to this. This key will also be your password. The username that they gave me was apikey so, i guess this remains same for everyone.
  4. Configure your django settings to this:
    EMAIL_HOST_USER = ‘<your username here>’
    EMAIL_HOST = ‘smtp.sendgrid.net’
    EMAIL_HOST_PASSWORD = ‘<your password here>’
    EMAIL_PORT = 587
    EMAIL_USE_TLS = True
    EMAIL_BACKEND = ‘django.core.mail.backends.smtp.EmailBackend’  (this is the default value of EMAIL_BACKEND btw)
  5. Use django.core.mail.send_mail for sending emails now

Django with uwsgi and nginx on fedora

Today, i deployed a django project using uwsgi and nginx on a fedora 26 instance on AWS. I will talk about the same here.

I had used gunicorn in the past but never uwsgi. Getting started with gunicorn was a little bit easier for me than uwsgi primarily because i didn’t know i had to install uwsgi-plugin-python package to use uwsgi with django. This took me a while because, there were no errors. There was a “no app could be loaded” problem but, on internet most of this kind of error is for flask. Flask exposes it’s application object as app and uwsgi looks to load application which it fails to find.

The steps are:

  1. Install dependencies and the project itself:
    • sudo dnf install uwsgi uwsgi-plugin-python nginx
  1. Create a configuration file for uwsgi: uwsgi.ini
  2. Change nginx config to pass on to uwsgi for incoming requests on /ethcld/ (the mount point that i used)

 

Here is my uwsgi file:

[uwsgi]
chdir = /home/fedora/ethcld
plugin = python
# Django’s wsgi file
# module = ethereal.wsgi:application
mount = /ethcld=ethereal.wsgi:application
manage-script-name = True
master = True
# maximum number of worker processes
processes = 4
# Threads per process
threads = 2
socket = 127.0.0.1:8001
# clear environment on exit
vacuum = true

uwsgi asks for the directory of the project. In my case, it was /home/fedora/ethcld/. Mount is optional, if you want to run the application under some namespace, you will have to use mount. Also, if mount is getting used, you should not need module.

Manage Script name (manage-script-name)is important while mounting otherwise you will get a bunch of 404s. Usually, the request comes for something like: GET /username/  but, without manage-script-name option, nginx will not map /username/ to /ethcld/username.

For socket, i could have used unix socket instead of port one. But, somehow i settled for port.

On the nginx side, i did the following changes:

http {

       include /etc/nginx/mime.types;

       server {

                location ^~/static/ {
                       autoindex on;
                       alias /opt/ethcld/static/;
                }

                location ^~ /ethcld {
                        include uwsgi_params;
                        uwsgi_pass 127.0.0.1:8001;
                }

         }

}

Nginx supports uwsgi protocol by default, so options are already there, we just need to call them. For serving static files, for django, it is recommended that we do:

  • python manage.py collectstatic

This will copy (by default) all the static files in the django app to the location specified in STATIC_ROOT in settings. From there we can serve the static files. There are possible optimizations that can be done like gzip. But, i did this for testing only.

You need to include mime.types, otherwise browsers will keep rejecting files. By default, the mime type is ‘text/plain’.

Using IRC from Mobile

In this blog post, i will talk about how i use IRC from mobile.

I have been using weechat as my irc client for about a year now and i am happy with it. Generally, i used to open weechat in one tmux session and project (or projects) in other tmux sessions in my machine and stay on IRC while working. This serves well except for the fact that when you are not connected with internet, nobody on irc can leave a message for you.

I didn’t know about weechat’s relay feature until very recently when maxking was talking about this in #dgplug. In this feature, weechat listens to client connections to a specified port that allows two way communication between connected clients and weechat. The clients can be any device with internet access. The only thing left then for this setup to work was an android app. There were two options, glowing bear and weechat-android. The former didn’t support ssl so it was out of picture.

A few weeks ago, i got a free tier fedora 26 instance on AWS. I had to use this for testing other applications that i was working on. Also, AWS doesn’t allow(atleast for my instance) HTTPS connections for ports other than 443. I wanted to use ssl and that too for more than one application and thus decided using nginx as reverse proxy.

Here are the list of things that i did:

  1. Installed nginx, tmux and weechat on aws instance.
  2. Created self-signed certificate and pointed nginx to that.
  3. Configure weechat to relay on port 9001 and configure nginx for websocket connections on 443 and proxy it to 9001.
  4. Use weechat-android to connect to the relay using websocket(ssl) as connection type.

I used tmux so that i could ssh into the aws instance and join the tmux session. This is where the client-server architecture of tmux helped. I couldn’t use let’s encrypt or ACM for ssl certificate because i didn’t have domain name for that public IP. Creating self-signed certificate is surprisingly easy and this Digital Ocean blog helped.

I also use urlserver.py plugin for weechat which shortens url for you. It runs a small server on the system and provides redirects to the original link. With nginx, i am able to configure urlserver.py to run on one port and point nginx to that.

I am not fully happy with weechat-android though. Most of the time it works but, sometimes it disconnects right after connecting without saying anything. Other people have found this too here. Atleast, people can drop any message for me now.