Token based Websocket Authentication

At my day job, i had to implement websockets and thus authentication of the websocket connection came up. There were two different types of clients but, the authentication for browser client was the biggest headache.

For a normal HTTP request, we use cookies for authentication of a user. Websocket protocol in itself doesn’t define how a websocket connection should be authenticated. According to RFC 6455, under section 10.5:

   This protocol doesn't prescribe any particular way that servers can
   authenticate clients during the WebSocket handshake. The WebSocket
   server can use any client authentication mechanism available to a
   generic HTTP server, such as cookies, HTTP authentication, or TLS
   authentication.

So, the first thing that comes to mind is: Why not use the same cookies that we use for an HTTP request? I thought the same too but, eventually, decided to use token based authentication.

Why not use the cookie?
We are using Django for our main web application. Django and WSGI based python frameworks in general, are not built for long lived connections. So, for websockets we are using Tornado.

In Django, by default, cookies are not readable by javascript. They are marked as HTTP only and thus the browser uses the cookie only for making http/https requests to the origin server. It can be turned off by using:

SESSION_COOKIE_HTTPONLY = False

Screenshot from 2017-10-15 00-16-41

The above image is when you have SESSION_COOKIE_HTTPONLY = True .

Screenshot from 2017-10-15 00-42-28

This is when you set SESSION_COOKIE_HTTPONLY  to False. The “`sessionid“` is the one which will be used by the server to identify the user.

The main benefit of not exposing sessionid to js in the browser is that it if someone successfully performs a XSS attack they won’t be able to hijack the session. Setting the cookie to be not http only would have been the easiest option for me but, as it was not recommended, i went for token based authentication.

Token based authentication

For token based authentication to work, the Django server will have to generate a token on every request (for the endpoints which requires the websocket connection). Once the browser gets the token, it can initiate a websocket connection to the tornado server. While opening the websocket connection, the browser will send the token as well. On the server side, there should be a common store where Django can store the token and Tornado can retrieve the token to verify the request.

Generating the token on server side for multiple views can be done by making a python decorator. But, if you are making a decorator and want to pass on a variable to the original function itself, you will have to add an extra parameter on the function to receive the variable’s value. This was a big task and would have meant a lot of changes across the project. Instead, i went on to make project wide template tags.

Making a project wide template tag in django for creating tokens

  1. Create a folder under the project’s main directory and create two files: __init__.py and create_ws_tokens.py
  2.  In create_ws_tokens.py, you can put something like this.:
    import uuid
    import json
    import datetime
    
    from django import template
    from project_name import redis_conn
    
    register = template.Library()
    
    @register.simple_tag(takes_context=True)
    def create_ws_token(context):
        request = context['request']
        if not request.user.is_authenticated():
            return 'Not authed'
        user = request.user.username
        current_time = datetime.datetime.strftime(
                datetime.datetime.utcnow(),
                "%d:%m:%Y:%H:%M:%S"
        )
        token = 'wstoken' + uuid.uuid4().hex
        output = {
           'user': user,
           'time': current_time
        }
        redis_conn.set(token, json.dumps(output))
        return token
    
    
  3. Put the following snippet inside the Templates -> Options in settings of the project.
    'libraries': {
    'create_ws_token': 'project_name.templatetags.create_ws_token',
    },
    
  4. Now to use this template tag in any template, you will need to load it.
    {% load create_ws_token %}
    and

    <script>
        var token = '{% create_ws_token %}';
        if (token.startsWith('wstoken')) {
            socket(token);
        }
    </script>
    

socket is a function which is defined in other js file which creates a websocket connection.


ws = create_ws("ws://localhost:8080/wsb?ws_token="+ws_token);

From tornado side, we need to get the ws_token and query redis for a verification.


def open(self):
    ''' Called by tornado when a new connection opens up '''
    self.user = None
    if 'ws_token' in self.request.arguments:
        token = self.request.arguments['ws_token'][0]
        self.user = self.authenticate(token)
        if self.user:
             tsockets.add_socket(self.user, self)
             print 'New connection from browser!'
        else:
             self.close()
    else:
        self.close()

The authenticate method would be like:


def authenticate(self, token):
    ''' Check for authentic token in redis '''
    inredis = self.application.redis_conn.get(token)
    if inredis:
        inredis = json.loads(inredis)
        self.application.redis_conn.delete(token)
        current_time = datetime.datetime.utcnow()
        valid_time = current_time - datetime.timedelta(seconds=20)
        inredis_time = datetime.datetime.strptime(
           inredis['time'], "%d:%m:%Y:%H:%M:%S"
         )
        if valid_time <= inredis_time:
            return inredis['user']
    return False

I chose redis because, Tornado is a single threaded server and connecting to db, if it’s not async will result in a blocking connection which means the real time features will get affected.

That’s it.

Advertisements

Using C function from Python

ctypes is a python library which allows using C data types, functions from a python script. It’s in the standard python library. To use C functions using ctypes, you will need to compile the C code and create a shared library.

add.c


#include <stdio.h>

int add_two_numbers(int num1, int num2) {
    return num1 + num2;
}

I will be using a very simple C function in this case which adds two given numbers

Now compile this file using:
gcc -fPIC -shared -o libadd2nums.so add.c

This will create a shared library named libadd2nums.so which, for now, contains only one function.

add.py


# coding=utf-8

import ctypes

_add = ctypes.CDLL('/home/vivek/ctypestuts/libadd2nums.so')
_add.add_two_numbers.argtypes = (ctypes.c_int, ctypes.c_int)

def add_two_numbers(num1, num2):
   ''' Adds two numbers '''

   return _add.add_two_numbers(ctypes.c_int(num1), ctypes.c_int(num2))

I am using fedora 26. If you are using Windows, you will need to use ctypes.WinDLL.

_add here, is the shared library and we can access the C function using dot(.) .

main.py


# coding=utf-8

import add

num1 = int(raw_input("Enter num1: "))
num2 = int(raw_input("Enter num2: "))
print add.add_two_numbers(num1, num2)

That’s it.

Realtime Events using Tornado and Rabbitmq

At my day job, i needed a way to send real time events to clients that would in turn trigger some action on their side. The clients could ask for some computation from server which may take time.

To tackle this situation, i ended up making Tornado as a websocket server which will be different from our web app server (and both behind nginx). There are a couple of other services which the client may ask for indirectly. Since, those computations won’t have normal request – response cycle, the results from the computations will have to pushed to the clients. Since, the communication between the client and server is two-way so, websocket seemed fitting. For routing of messages internally, i decided to use Rabbitmq
and Celery for actual execution of tasks.

The problem with this is: Rabbitmq consumer and Tornado both run their own I/O loop. That confused me a little because i had heard this combo worked for zulip when i was randomly reading about their architecture. So, i duckduckgoed(:D) and found this article: https://reminiscential.wordpress.com/2012/04/07/realtime-notification-delivery-using-rabbitmq-tornado-and-websocket/   . It turns out he also had a similar doubt and he got a solution.

Pika library comes with a tornado adapter named TornadoConnection. This makes running the rabbitmq consumer loop inside the tornado IOloop itself. The code for tornado connection is fairly simple. As the code given in the blog wasn’t fully functional, i had to contact the source code of pika a couple of times.

Each websocket connection in tornado gets a unique WebSocketHandler object and these are not directly accessible from the tornado application object. But, the reverse is true. Each websocket handler has access to the application object. So, using TorandoConnection, we tie up one pika consumer to the tornado application object.

server.py


def main():
    ''' The main method to run the tornado application '''

    io_loop = tornado.ioloop.IOLoop.instance()

    pc = PikaConsumer(io_loop)

    application.pc = pc
    application.pc.connect()
    application.listen(8080)
    io_loop.start()

consumer.py


class PikaConsumer(object):
    ''' The pika client the tornado will be part of '''

    def __init__(self, io_loop):
        print 'PikaClient: __init__'
        self.io_loop = io_loop
        self.connected = False
        self.connecting = False
        self.connection = None
        self.channel = None
        self.event_listeners = {}

    def connect(self):
        ''' Connect to the broker '''
        if self.connecting:
           print 'PikaClient: Already connecting to RabbitMQ'
           return

        print 'PikaClient: Connecting to RabbitMQ'
        self.connecting = True

        cred = pika.PlainCredentials('someuser', 'somepass')
        param = pika.ConnectionParameters(
            host='localhost',
            port=5672,
            virtual_host='somevhost',
            credentials=cred)
        self.connection = TornadoConnection(
                    param,
                    on_open_callback=self.on_connected)
        self.connection.add_on_close_callback(self.on_closed)

    def on_connected(self, connection):
        print 'PikaClient: connected to RabbitMQ'
        self.connected = True
        self.connection = connection
        self.connection.channel(self.on_channel_open)

    def on_channel_open(self, channel):
        print 'PikaClient: Channel open, Declaring exchange'
        self.channel = channel
        # declare exchanges, which in turn, declare
        # queues, and bind exchange to queues
        self.channel.exchange_declare(
              exchange='someexchange',
              type='topic')
        self.channel.queue_declare(self.on_queue_declare, exclusive=True)

    def on_queue_declare(self, result):
        queue_name = result.method.queue
        self.channel.queue_bind(
        self.on_queue_bind,
        exchange='someexchange',
        queue=queue_name,
        routing_key='commands.*')
        self.channel.basic_consume(self.on_message)

    def on_queue_bind(self, is_ok):
        print 'PikaClient: Exchanges and queue created/joined'

    def on_closed(self, connection):
        print 'PikaClient: rabbit connection closed'
        self.io_loop.stop()

    def on_message(self, channel, method, header, body):
        print 'PikaClient: message received: %s' % body
        self.notify_listeners(body)
        # important, since rmq needs to know that this msg is received by the
        # consumer. Otherwise, it will be overwhelmed
        channel.basic_ack(delivery_tag=method.delivery_tag)

    def notify_listeners(self, event_obj):
        # do whatever you wish
        pass

    def add_event_listener(self, listener):
        # listener.id is the box id now
        self.event_listeners[listener.id] = {
                'id': listener.id, 'obj': listener}
        print 'PikaClient: listener %s added' % repr(listener)

    def remove_event_listener(self, listener):
        try:
            del self.event_listeners[listener.id]
            print 'PikaClient: listener %s removed' % repr(listener)
        except KeyError:
            pass

    def event_listener(self, some_id):
        ''' Gives the socket object with the given some_id '''

        tmp_obj = self.event_listeners.get(some_id)
        if tmp_obj is not None:
            return tmp_obj['obj']
        return None

That’s it. In your WebSocketHandler objects, you can access the consumer via: self.application.pc

Although, this is working fine for me right now but, i am not fully satisfied with this. At present each connection is listening to a single queue because in rabbitmq one consumer cannot listen to multiple queues.

Running Firefox as kiosk application on RPi3

At my day job, i had to run firefox as a kiosk application on RPi3. In this blog post, i will note down the steps that i did so that i or my team members can refer to it when needed.

I have not used any display manager or Desktop Environment but had to use matchbox-window-manager to make firefox run on full screen.

  1. sudo apt-get install vim (this is just for me, i can’t help it)
  2. sudo apt-get install xorg xutils matchbox-window-manager 
  3. sudo apt-get install iceweasel (this is firefox :p)
  4. sudo raspi-config
    1. Go to boot options and setup auto login for user pi
    2. Change the keyboard layout if you wish to
  5. As a sudo user, do the following steps:
    1. cp -r /home/pi  /opt/
    2. cd /opt/pi
    3. chmod -R a+r .
    4. touch .xsessionrc
    5. chmod a+x .xsessionrc
  6. Open .xsessionrc as a sudo user and put the following lines there:
    1. xset s off # no screen saver
      xset -dpms  # disable some power consumption thingy
      xset s noblank # don’t blank the rpi screen
      matchbox-window-manager &
      while true; do
        firefox –url http://127.0.0.1
      done
  7. Copy .xsessionrc file to /home/pi/ 
    1. cp .xsessionrc /home/pi
  8. Configure .bash_profile to start X server when user logs in:
    1. if [ -z “$DISPLAY” ] && [ -n “$XDG_VTNR” ] && [ “$XDG_VTNR” -eq 1 ]; then
      exec startx
      fi
  9. Install an extension in firefox to apply kiosk mode. The first extension that comes up when you search kiosk in addons works fine

The blog post that helped most in coming up this setup was: http://www.alandmoore.com/blog/2011/11/05/creating-a-kiosk-with-linux-and-x11-2011-edition/

Star a project on pagure

This feature was marked as “wishful” and it was supposed to be a low priority but i implemented it out of frustration. This should be there in the next feature release.

Star feature is already there on github and gitlab. I use this feature on github a lot. If I like an open source project, i star it. You also have a list of all the projects which you have starred which can be helpful if you have come across a project sometime ago and had starred it and you want to know more about it (given that you don’t exactly remember the name, otherwise you can just search). Also, if the project author/maintainer is anything like me, he would love to see the star count rising.

For sometime now, i had been asking people who use pagure often (and hopefully like pagure) to star it on github. The star count of pagure was 96 at the time i started my work on star project feature. Last year, at this time, it was in late 60s.

If you star a project on github, your followers come to know that you have liked a project. They can see that on their github homepage. If they see the project, like it, you already have helped pagure reach more people with almost zero effort. I can’t see one good reason if you like a project that you won’t star it.

Pagure doesn’t have the follow feature and i am not sure it will have in near future. This means the star project won’t have it’s full effect. But, one can star a project, there is a star count, there is a list of people who have starred, there is a list of starred projects of a user.

Here is how you can use this feature:

  1. Log in to pagure and go to a project’s home page.
  2. There is a star button, just beside the fork button. It has a star count just beside it.
  3. Star it if you like the project.

Screenshot from 2017-09-05 00-28-45

Here is where you will find your starred projects:

  1. Log in to pagure.
  2. The drop down on the top right corner will be “My Stars”

Screenshot from 2017-09-05 00-43-16

Here is where you can see who all have starred a particular projects:

  1. Right beside the star button on repo page, we have a star count which actually links to a page which lists all the users who have starred the project.

Screenshot from 2017-09-05 00-42-01

 

Using Celery with Rabbitmq

Rabbitmq is a message broker and celery is a task queue. When you run a celery app, by default, it will open as many processes as there are cores of cpu on the machine. These processes are workers. When you have a task which needs to be done outside of a normal HTTP request-response cycle,  you can use a task queue. Rabbitmq can be configured to decide (and deliver) which worker the task has to go and celery will help in the actual execution of the tasks.

Celery supports a lot of message brokers but, Rabbitmq is the default one. So, setting up celery for using with rabbitmq doesn’t require any effort. If you have rabbitmq already installed then all you need to do is create a rabbitmq user, a virtual host and give the user access to the virtual host. It is given in the celery documentation here.

Then you need to specify the broker url in this format in the celery app.

broker_url = 'amqp://myuser:mypassword@localhost:5672/myvhost'

The default exchange that celery listens to is named ‘celery‘ and routing key is also ‘celery‘. The ‘celery‘ exchange is direct type exchange. AMQP is the protocol that rabbitmq follows. Username, password and virtual host here is of rabbitmq that you want celery to use. Based on the given broker url, celery attempts to know which message broker is being used.

Using Syntastic for Python development

I use Synstastic plugin of vim for syntax checking in vim. Syntastic offers syntax checking for a LOT of languages. But, there is a problem that i had been facing with it. For a file with larger than 4k lines, it takes a lot of time to check the syntax and it used to happen every time you save the file. Syntax checking on write operation is the default behavior.

So, i did some changes in my .vimrc so that i could still use Syntastic for larger files. Do note that syntastic checking still takes a long time but, i have configured it to be called whenever i want to rather than on every write operation or opening of file.

” show list of errors and warnings on the current file
nmap <leader>e :Errors<CR>
” Whether to perform syntastic checking on opening of file
” This made it very slow on open, so don’t
let g:syntastic_check_on_open = 0
” Don’t check every time i save the file
” I will call you when i need you
let g:syntastic_check_on_wq = 0
” By default, keep syntastic in passive mode
let g:syntastic_mode_map = { ‘mode’: ‘passive’ }
” Use :Sc to perform syntastic check
:command Sc :SyntasticCheck
” Check pylint for python
let g:syntastic_python_checkers = [‘pylint’]
” For jsx – React and React native
let g:syntastic_javascript_checkers = [‘eslint’]

This change made opening of a larger python file ~25s (yes, seconds) faster. It still takes a lot of time for syntax checking though. I will have to find out why and if i could do anything about it. I don’t want to leave out this plugin because it offers so much. I could simply use Python-mode for python syntax checking but, what about the rest of the languages which i am going to use.