life of a gizmo

You may be doing your daily business, as I was, then get an email from a client saying there’s a problem with the (django) site you created. You think to yourself, hmm, that’s odd, nothing has changed in that since since X years.

You check out the site and try to login only to discover this nasty error:

FORBIDDEN
CSRF verification failed. Request aborted.

This is most likely caused by an intentional or unintentional upgrade to Django 1.2.x. For me this upgrade was unintentional. My host is dreamhost and they decided to upgrade their django version to 1.2.1. Once this happened I started getting that error out of the blue! If you did upgrade to django 1.2.x intentionally, you need to make sure you read the release notes and pay close attention to the backward incompatible changes. You will probably have to make changes to your code.

I already had a virtual-env with the old version of django installed, so all I had to do was change these two lines of code in my ./dispatch.fcgi.

Before the change:

sys.path += ['/home/myuser/django_src']
sys.path += ['/home/myuser/beta_django_projects']

After the change:

sys.path.insert(1, '/home/myuser/django_src')
sys.path.insert(1, '/home/myuser/beta_django_projects')

This sets the your django_src path to higher priority in your PYTHONPATH which results in your django version being used over the one available at dreamhost.

These instructions are very tailored to my dreamhost issue. If you something is not clear or this doesn’t solve your CSRF problem, just leave a comment and I’ll try to help you out.

The Story

I had some work to do but instead found a perfect distraction. I decided to upgrade my textbook website to Django 1.2.x which is an updated version of the python framework I used to build it.

Unlike in the past, where such a task would have not phased me and which I would have attempted to do on the live site, I decided to setup a beta site that runs along the main GizmoBooks website as a sandbox so to speak.

I started upgrading and it turned out there were many changes that needed to be made. I came across this very helpful guide on intalling python 2.6, virtualenv, and virtualenvwrapper. After a few attempts and tweaks it worked and beautifully too! Not only did I have python 2.6 installed but I also had virtualenv and virtualenvwrapper which I until I had it, I had no idea what I was missing. It’s quite brilliant because it allows you to setup virtualenvironments like this *snaps fingers*.

So far so good, I had my virtual environment setup but then came time to find all the different python PIP packages and other software that my website needed. This was okay for a bit, but then came time to install xapian-core, xapian-binding, blah blah blah. I quickly got tired of all this.

The Script

So, I did what all lazy programmers do: I wrote a script about it! The script is called deploy_pysite.sh and can be found at GitHub.

You see, the beauty of virtualenv is that it allows you to quickly create a virtual environment from scratch. The problem, of course, is that (often) the virtualenv is empty and you need to go install all the packages to deploy your website. This is what the scripts aims to simplify.

Script Guide

Follow these steps to run it:

1. If you’re not using virtualenvwrapper, set VIRTUAL_ENV environment variable to your absolute virtual environment path. If you are using virtualenvwrapper, just “workon” the virtual env you wish to install the packages in.
2. Add to, or remove from nonpip_packages and pip_packages arrays in SETTINGS section of deploy_pysite.sh source code. Use the formats specified in comments.
3. (Optional) The default directory where software is downloaded and built is $HOME/downloads. This is set in dg_downloads_dir in deploy_pysite.sh SETTINGS section. Feel free to modify it.
4. Execute the script.

That’s pretty much all there is to it.

Not surprisingly, it proved usefulx100 immediatly. You see, I was following another guide from andrew.io on installing Django with virtualenv on dreamhost and low and behold the guide said to “[i]nstall any packages you may need for your project.” ./deploy_pysite.sh was all I had to say at that point .

Script Source

I’ll post the source here, but it’s probably best to just grab it from GitHub.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195

#!/bin/bash
 
# ensure that exceptions are treated as such and program exists!
set -e
 
# =================================================
# = deploy_pysite version 0.1.0
# = 
# = Created by Gezim Hoxha
# = License: GPL 2
# =
# = If you're using virtualenv, this script will
# =     install PIP packages and even build 
# =     build packages from source then install
# =     them into your virtual environment.
# = See SETTINGS section for instructions.
# =
# = July, 2010
# =
# = TO DO
# = - 
# =================================================
# = REQUIRES
# = bash >=3.1
# = curl
# = pip
# =================================================
 
usage="
USAGE: \n
Modify nonpip_packages and pip_packages according to your needs. These can be found in the source
of this script in SETTINGS section.\n
Once SETTINGS have been modified, activate your virtual environment, then call this script. \n
If you're not using virtualenvwrapper, set VIRTUAL_ENV environment variable 
to your virtual environemt directory. \n
\n
Also, you may want to ensure that the download dir is set to where you want it in SETTINGS section.
"
 
# ensure we're in a virtual environment
 
if [[ "$VIRTUAL_ENV" = "" ]]
then
    echo -e $usage
    exit 5
fi
 
# =================================================
# ====================SETTINGS=====================
# =================================================
 
# Directory to store source archives
dg_downloads_dir="$HOME/downloads"
 
# Add/remove non-pip packages you [don't] want to install here.
# IMPORTANT: Only .tar.gz packages are supported.
#
# You must add a package URL and function pair to nonpip_packages array.
# If no custom install is required (i.e. normal ./config, make, make install will do) leave 
#   function call empty. For example:
#   "http://www.example.com/pack.tar.gz" ""
# If you need custom install calls, create a function that makes those calls in CUSTOM FUNCTIONS
#   area below. The function calls can include arguments.
#   For example:
#   "http://www.example.com/pack02.tar.gz" "install_pack02"
#   Or with a function call argument:
#   "http://www.example.com/pack04.tar.gz" "install_package four"
#
# MAKE SURE YOU USE QUOTES.
#
nonpip_packages=(
    "http://oligarchy.co.uk/xapian/1.0.21/xapian-core-1.0.21.tar.gz"
        ""
    "http://oligarchy.co.uk/xapian/1.0.21/xapian-bindings-1.0.21.tar.gz"
        "function_xapian_bindings"
    )
 
# Add/remove pip packages you [don't] want to install here.
# Version is included to ensure you get the same packages
#   whenever you install.
# Example:
#   "django" "1.2.1"
#
# MAKE SURE YOU USE QUOTES.
#
pip_packages=(
    "django" "1.2.1"
    "django-haystack" "1.0.1-final"
    "xapian-haystack" "1.1.3beta"
    "MySQL-python" "1.2.3c1"
    )
 
# =================================================
# =================CUSTOM FUNCTIONS================
# You should use variables such as $VIRTUAL_ENV.
# =================================================
function_xapian_bindings()
{
    ./configure --with-python XAPIAN_CONFIG="$VIRTUAL_ENV/bin/xapian-config" --prefix="$VIRTUAL_ENV"
    make
    make install
}
 
 
 
# =================================================
# ====================REAL CODE====================
# ===Don't touch unless you are sure you need to.==
# =================================================
# =================================================
 
# Create the downloads dir.
# Nothing will happen if it already exists.
mkdir -p "$dg_downloads_dir"
 
temp_file="/tmp/$(basename $0).$$.tmp"
 
# Build and install non-pip packages
build_install()
{
    for ((i=0; i<${#nonpip_packages[@]}; i+=2))
    do
        #pip_install "${pip_packages[i]}" "${pip_packages[i+1]}"
        #get package name
        package_name="${nonpip_packages[i]##*/}"
        package_dir="${package_name%.tar.gz}"
 
        echo "Installing $package_name to $VIRTUAL_ENV with..."
        cd "$dg_downloads_dir"
 
        # Check if directory exists.
        # -d test doesn't work on case insensitive FS's (e.g. OS X)
        if (cd "$package_dir" 2>/dev/null)
        then
            cd "$package_dir"
        elif [ -f "$package_name" ]
        then
            # if package is already downloaded
            tar xzf "$package_name"
            cd "$package_dir"
        else
            curl -sS "${nonpip_packages[i]}" -O > /dev/null
            tar xzf "$package_name"
            cd "$package_dir"
        fi
 
        # if odd array index has no function, do a standard make
        # Otherwise, run custom function.
        custom_function=${nonpip_packages[i+1]}
        if [[ "$custom_function" = "" ]]
        then
            set -x
            ./configure --prefix="$VIRTUAL_ENV" >>"$temp_file" 2>&1
            make >>"$temp_file" 2>&1
            make install >>"$temp_file" 2>&1
            set +x
        else
            set -x
            "$custom_function" >>"$temp_file" 2>&1
            set +x
        fi
        echo "...done."
 
    done
}
 
# Install pip packages
# param: $1 is the package name
# param: $2 is the package version
# The package version is required
#  to ensure the exact setup is installed everytime.
pip_install()
{
    if (( $# != 2 ))
    then
        echo "pip_install requires 2 arguments."
        return 25
    fi
 
    echo "Installing $1 to $VIRTUAL_ENV with..."
    set -x
    pip install "$1"=="$2" >>"$temp_file" 2>&1
    set +x
    echo "...done."
}
 
# Now install the packages if they're enabled.
build_install
 
for ((i=0; i<${#pip_packages[@]}; i+=2))
do
    pip_install "${pip_packages[i]}" "${pip_packages[i+1]}"
done
 
echo -e "\nInstall was successful. See $temp_file for details."

Now, what was I doing?

I came across this excellent and very useful installation guide on installing python 2.6, virtualenv, and virtualenwrapper.

Ahh, that’s a nice and descriptive title.

Needless to say, I had the darnest time doing this. Upgrading mysql was not too big of an issue, but getting MySQLdb to work was a whole other game.

Anyway, here are the steps I took.
The instructions for upgrading MySQL came from entropy.ch and are as follows:

1. Save your current database data before doing anything: cd /usr/local/mysql; sudo tar -cvf /tmp/mysql-data.tar data
2. Then download the MySQL installer from the MySQL website. At first I chose to download “Mac OS X ver. 10.6 (x86, 32-bit), DMG” which, I believe, caused all my problems with MySQLdb. So download Mac OS X ver. 10.6 (x86, 64-bit), DMG unless you know you have a 32 bit machine.
3. Once downloaded, unpack and install the MySQL package and even the prefPane to make your life easier.
4. To restore your old data, run this command: cd /usr/local/mysql; sudo tar -xf /tmp/mysql-data.tar
5. I’m not sure if this is needed, in the attempt that worked, I followed the instructions to the dot so I did reboot my computer (although the MySQL install didn’t require me to).
6. Start MySQL either through the System Preferences (if you installed the prefPane above) or by running: sudo /usr/local/mysql/bin/mysqld_safe

That’s all there was to installing MySQL. Note that in my system I didn’t have to change the ownership of the database data when it was “imported” into the new install. If you have to do that (i.e. can’t login with the usual username/password combo) see the instructions in the link I posted above.

Now onto MySQLdb. These instructions are taken from Geert JM Vanderkelen.

1. Download MySQL-python-1.2.3c1.tar.gz from the Python Package Index.
2. Untar it and ensure “/usr/local/mysql/bin” is in your PATH, if not see the above link on how to add it.
3. Run this command: ARCHFLAGS="-arch x86_64" /usr/bin/python setup.py build (Ensure you don’t have a 64 bit machine before changing it.)
4. And finally, install it with the following command: sudo /usr/bin/python setup.py install

Once all that is done. You can test it like this:

$ python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>


If that gives you no errors, you’re set. Now get back to doing actual work .

I just finished listening (actually still not finished but just watching the question period ) to this DjangoCon talk by James Bennet. Found it very motivating and useful. There are a lot of apps out there! Before you start to write some feature for a django site, ensure that you’re not rewriting something that someone already wrote. Here are a few site to look at:

• djangosnippets
• djangoplugables
• Django Applications

I came across and incredible piece on MVC and web development:

it should be abundantly clear that the traditional “MVC” is back with a vengeance: the blurred lines between which bits of code are responsible for “Model business logic” validation are gone, and the blurred lines regarding which bits of code are responsible for generation of the “View” are also gone – but not completely. It’s up to the developer to decide how far they want to deviate from the traditional ‘HTML’ style of display / information rendering. Not everybody is comfortable with the use of Widgets that are executed as Javascript to do the job that HTML used to do (even if it’s behind a javascript framework such as mochikit, extjs or prototype), and fortunately, both the GWT and the Pyjamas frameworks do an extremely good job of supporting and interacting with straight HTML (investigate the HTMLPanel widget, for example, for details).

Wouldn’t it be cool to be able to replace Javascript as the only way to manipulate the DOM in browsers. Wouldn’t it be cool if instead of Javascript you could throw Java, python, into an HTML document! Let the dreams begin.

pyjamas is a new way to create web applications. Unlike the traditional method of creating an HTML page, adding CSS to it, spicing things up with javascript, pyjamas has a different way of doing things. You build your HTML page, by using only javascript (expect in this case python which compiles into javascript). This is totally different and something that will take a bit of time to get used to it; seems promising though (assuming that performance doesn’t take a huge hit).

pyjamas diagram

Programming with pyjamas needs a little bit of rethinking about the way that you do Web application development. Primarily, you need to forget virtually everything you’ve ever learned and come to expect Web development to be. The reason for this is very simple: Pyjamas is pure javascript. Although written in python, not javascript, it is essential to bear in mind that virtually 100% of your web application will be javascript – not HTML. The programming style you may be accustomed to for HTML programming involves placing as much HTML into one page as you can stand, and making the minimum number of exceptions and allowances for dynamic content that you can manage, without making the HTML page “too complicated” to be readable.

Pyjamas makes it possible for you to break pages down into concepts. classes. widgets. maybe some CSS styling is added, almost as an afterthought, on top of the “real” functionality. In other words, Pyjamas is actually more like Desktop application development than it is Web development.

That’s pretty exciting and I intend to try it, after I finish my current project.