Sending Emails using Django and Sendgrid

Recently, i setup a django app which uses  sendgrid to send emails. I will go through the steps in this short blog.

  1. Register at sendgrid
  2. Choose SMTP for sending emails
  3. Get an API key. The last step will redirect you to this. This key will also be your password. The username that they gave me was apikey so, i guess this remains same for everyone.
  4. Configure your django settings to this:
    EMAIL_HOST_USER = ‘<your username here>’
    EMAIL_HOST = ‘smtp.sendgrid.net’
    EMAIL_HOST_PASSWORD = ‘<your password here>’
    EMAIL_PORT = 587
    EMAIL_USE_TLS = True
    EMAIL_BACKEND = ‘django.core.mail.backends.smtp.EmailBackend’  (this is the default value of EMAIL_BACKEND btw)
  5. Use django.core.mail.send_mail for sending emails now

Django with uwsgi and nginx on fedora

Today, i deployed a django project using uwsgi and nginx on a fedora 26 instance on AWS. I will talk about the same here.

I had used gunicorn in the past but never uwsgi. Getting started with gunicorn was a little bit easier for me than uwsgi primarily because i didn’t know i had to install uwsgi-plugin-python package to use uwsgi with django. This took me a while because, there were no errors. There was a “no app could be loaded” problem but, on internet most of this kind of error is for flask. Flask exposes it’s application object as app and uwsgi looks to load application which it fails to find.

The steps are:

  1. Install dependencies and the project itself:
    • sudo dnf install uwsgi uwsgi-plugin-python nginx
  1. Create a configuration file for uwsgi: uwsgi.ini
  2. Change nginx config to pass on to uwsgi for incoming requests on /ethcld/ (the mount point that i used)

 

Here is my uwsgi file:

[uwsgi]
chdir = /home/fedora/ethcld
plugin = python
# Django’s wsgi file
# module = ethereal.wsgi:application
mount = /ethcld=ethereal.wsgi:application
manage-script-name = True
master = True
# maximum number of worker processes
processes = 4
# Threads per process
threads = 2
socket = 127.0.0.1:8001
# clear environment on exit
vacuum = true

uwsgi asks for the directory of the project. In my case, it was /home/fedora/ethcld/. Mount is optional, if you want to run the application under some namespace, you will have to use mount. Also, if mount is getting used, you should not need module.

Manage Script name (manage-script-name)is important while mounting otherwise you will get a bunch of 404s. Usually, the request comes for something like: GET /username/  but, without manage-script-name option, nginx will not map /username/ to /ethcld/username.

For socket, i could have used unix socket instead of port one. But, somehow i settled for port.

On the nginx side, i did the following changes:

http {

       include /etc/nginx/mime.types;

       server {

                location ^~/static/ {
                       autoindex on;
                       alias /opt/ethcld/static/;
                }

                location ^~ /ethcld {
                        include uwsgi_params;
                        uwsgi_pass 127.0.0.1:8001;
                }

         }

}

Nginx supports uwsgi protocol by default, so options are already there, we just need to call them. For serving static files, for django, it is recommended that we do:

  • python manage.py collectstatic

This will copy (by default) all the static files in the django app to the location specified in STATIC_ROOT in settings. From there we can serve the static files. There are possible optimizations that can be done like gzip. But, i did this for testing only.

You need to include mime.types, otherwise browsers will keep rejecting files. By default, the mime type is ‘text/plain’.

Using IRC from Mobile

In this blog post, i will talk about how i use IRC from mobile.

I have been using weechat as my irc client for about a year now and i am happy with it. Generally, i used to open weechat in one tmux session and project (or projects) in other tmux sessions in my machine and stay on IRC while working. This serves well except for the fact that when you are not connected with internet, nobody on irc can leave a message for you.

I didn’t know about weechat’s relay feature until very recently when maxking was talking about this in #dgplug. In this feature, weechat listens to client connections to a specified port that allows two way communication between connected clients and weechat. The clients can be any device with internet access. The only thing left then for this setup to work was an android app. There were two options, glowing bear and weechat-android. The former didn’t support ssl so it was out of picture.

A few weeks ago, i got a free tier fedora 26 instance on AWS. I had to use this for testing other applications that i was working on. Also, AWS doesn’t allow(atleast for my instance) HTTPS connections for ports other than 443. I wanted to use ssl and that too for more than one application and thus decided using nginx as reverse proxy.

Here are the list of things that i did:

  1. Installed nginx, tmux and weechat on aws instance.
  2. Created self-signed certificate and pointed nginx to that.
  3. Configure weechat to relay on port 9001 and configure nginx for websocket connections on 443 and proxy it to 9001.
  4. Use weechat-android to connect to the relay using websocket(ssl) as connection type.

I used tmux so that i could ssh into the aws instance and join the tmux session. This is where the client-server architecture of tmux helped. I couldn’t use let’s encrypt or ACM for ssl certificate because i didn’t have domain name for that public IP. Creating self-signed certificate is surprisingly easy and this Digital Ocean blog helped.

I also use urlserver.py plugin for weechat which shortens url for you. It runs a small server on the system and provides redirects to the original link. With nginx, i am able to configure urlserver.py to run on one port and point nginx to that.

I am not fully happy with weechat-android though. Most of the time it works but, sometimes it disconnects right after connecting without saying anything. Other people have found this too here. Atleast, people can drop any message for me now.

GSoC Wrap Up

GSoC 2016 finished last week and i am writing this blog to list the work done by me in last three months for Fedora. My project was to adjust pagure and write script(s) so that we can have pkgs.fedoraproject.org on a pagure instance. We have it in staging currently http://pkgs.stg.fedoraproject.org/pagure/

https://pagure.io/pagure/pull-request/1007

https://pagure.io/pagure/pull-request/1035

https://pagure.io/pagure/pull-request/1036

https://pagure.io/pagure/pull-request/1045

https://pagure.io/pagure/pull-request/1050

https://pagure.io/pagure/pull-request/1058

https://pagure.io/pagure/pull-request/1071

https://pagure.io/pagure/pull-request/1094

https://pagure.io/pagure/pull-request/1095

https://pagure.io/pagure/pull-request/1097

https://pagure.io/pagure/pull-request/1114

https://pagure.io/pagure/pull-request/1120

https://pagure.io/pagure/pull-request/1149

https://pagure.io/pagure/pull-request/1150

https://pagure.io/pagure/pull-request/1151

https://pagure.io/pagure/pull-request/1157

https://pagure.io/pagure/pull-request/1177

https://pagure.io/pagure/pull-request/1210

https://pagure.io/pagure/pull-request/1211

https://pagure.io/pagure/pull-request/1219

https://pagure.io/pagure/pull-request/1218

https://pagure.io/pagure/pull-request/1158

Besides these, there is a script for getting user acls from pkgdb:

https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=de67bcbea22bb4539e32d195a10448948bc6d765

For me, the experience has been perfect. I like the work environment at #fedora-apps. My mentor, Pierre-Yves Chibon is nice to everyone and i hope i haven’t annoyed or disappointed him in last 3 months. It’s hard to find a person who can guide so patiently. I am saying this not because i see one of my friend working for FOSS Asia but, because he is genuinely good.

If GSoC wasn’t there, even then i would have spent my last 3 months in the same way (without mine and my father’s new mobile phone). I contribute here because i like the work environment that they have created and i get to learn new things while working on real life projects.

So, thanks Google for the money and Fedora for such an awesome experience.

User’s project watch list on pagure

Not long ago, Gaurav added watch feature on pagure. But, It had one thing missing from it: a user could not see what all projects he/she is watching. So, with pull request #1158, i tried to solve that problem.

For those of you are not aware of this feature, a user can now subscribe for emails for development of a project. He/She will receive emails for any changes on the issue tracker or if anything happens in any pull request. By default, the admin of a project is watching the project. However, if he wishes, he can unwatch it and he won’t receive emails for that project anymore. This can help in situations when the user is admin of a lot of projects and is no longer interested in some of them.

I won’t go in details of how this was implemented since, it involves simple function calls and minor addition in the UI (plus, there is a link to the PR). I will, however, attach screenshot of how it will look, when it will be live on pagure.io.

Screenshot from 2016-08-16 17-03-29

pkgs.fedoraproject.org on a pagure instance

pkgs.fedoraproject.org currently has more than 18k git repositories and it’s relying on cgit which is not capable of git collaboration. On the other hand, we have pagure which is a git collaboration tool and it’s live on pagure.io. As part of my Google Summer of Code project, i was supposed to prepare script and adjust pagure so that we can have pkgs.fp.org on a pagure instance.

For those who are not familiar with pagure, it’s a free and open source git collaboration tool written on top of pygit2 by pingou. It has a similar workflow as Github . One can fork a project, make changes and ask to merge the changes to the main project by creating a pull request. pagure.io already has more than 250 projects hosted on it and it’s increasing everyday. Feel free to play with pagure on stg.pagure.io.

The idea is, we will make pkgs.fedoraproject.org a pagure instance. With this change, it will be easier for anybody to contribute to any of the git rpms hosted on pkgs.fp.org . This is the major reason for the shift. Currently, only the rpm maintainers can make changes to the repository. But, once we have these rpms on a pagure instance, anybody with a FAS account can fork the project and make a pull request.

Here are a few points which makes pagure adaptable for pkgs.fp.org :

  • Turn on/off user management: Since, the acls for the git repositories come from pkgdb, user management should be turned off on the instance level.
  • Turn on/off Issue Tracker: The git rpms are not exactly projects, so it’s turned off at the instance level as well.
  • Turn on/off project creation: The git rpms need approval for creation of a project, thus a user shouldn’t be able to create a project.
  • Pseudo namespace: Pagure doesn’t have namespaces for a project (although, forks have). Thus, pseudo namespace was introduced. We can have a list of namespaces allowed for an instance.

Honestly speaking, almost all the above mentioned work was already done. After this, we just needed to adjust the script which currently gets acls from pkgdb so that it updates pagure database as well.

The work is almost complete and the shift should not take long once pingou returns from his vacation.

For any further query, you can ping me on #fedora-apps (nick: vivek_)

Happy coding,

Vivek

Access levels of user/group in a pagure project

Currently in pagure, we have only two access levels in a project – either you have no access or admin access. There have been a few discussions of more levels of access – #892,  #792

With the new changes, there will be four different levels of access –

  • None: As the name suggests, no special power. Just a normal user.
  • Ticket: Edit the metadata of issues which includes tagging, assigning and changing the status of the issue. They can’t delete or edit the issue itself.
  • Commit: All the access an admin of the project has except access to settings of the project.
  • Admin: He is the boss of the project.

As you can see, the levels are hierarchical in nature. The committers have all the access a user with ticket access has and similarly, the admin has all the access a committer has plus, some other accesses.

Ticket access is sort of “entry” level access to the project. They can play with the metadata of the issue and that’s it. The committers can merge/close a pull request, can commit to the project repository directly, edit/remove issues/comments. Basically, they can do anything in the project but touch the settings. Since, the admins are the only ones with access to settings, they are the only ones who can add/remove a user/group from a project or update their accesses.

In pagure, groups are dealt in a similar way as a user is. So, they too can have all the mentioned access levels. If you provide a group with some access in a project, all the users in the group will automatically get that access. In cases when a user already has some access “a”, and is also present in a group with some other access “b”, he gets greater level of access out of “a” and “b”.

Implementation (in short)

We already have admin access level in pagure so adding two more levels wasn’t a big deal. It has been implemented in a similar way. Here are few of the points:

  • Introduction of a new table: access_levels in the pagure db and foreign keys of the same in user_projects and projects_groups tables.
  • project.users now returns the list of users with at least ticket access , earlier it returned the list of users with admin access. Similar is the case with project.groups . Similar relations: project.admins, project.committers, project.admin_groups, project.committer_groups.
  • Two functions: is_repo_committer  and is_repo_user  to check for the rights of the user on the project. is_repo_admin was already there.
  • Update code to allow RW+ right for committers in gitolite file.
  • Changes in the conditional statements across the pagure code.

Pull request related to this is here. Tests and docs haven’t been updated yet.