This is a series of three posts on my experience with the PSP.
PSP stands for Personal Software Process. For an introduction, see the wikipedia page on PSP.
PSP℠ has been developed with training in mind. It starts you off with PSP 0 which is essentially your existing software process but adds measurements and through a series of six steps exposes the student to all of the PSP processes and measures. Below I will describe each step in more detail and explain the benefits of each.
PSP 0
Description
As mentioned above, this level attempts to establish a baseline for your existing development practices. It does, however, “force” you to split your development into phases: planning, development (design, code, compile, test) and postmortem. It also teaches you how to record development time and defects introduced. The only ‘planning’ aspect of it involves predicting how long each phase will take you. As you approach PSP 2.1, this data will show you how your predictions are better with PSP than without it–or not!
Takeaway
It’s very tempting to start developing a project (especially when it starts out small) by opening an editor and coding. Compile. Fix errors. Repeat until good enough. From personal experience, this leads to a lot of minuscule bugs because no attention to detail has been paid. It’s easier to let the compiler find the bugs for you! Using PSP however, you’re forced to plan, design, code, compile, and test. When I started using PSP 0, I noticed I was a lot more careful even with the planning phase. I gave some thought to what the design and coding will encompass in order to estimate how long each phase would take me. Normally these steps would have been performed implicitly, if at all. In the design phase, I started giving more importance to defects and payed close attention to design completeness as I didn’t want defects to appear on the defect logs! Something magical happens when you actually track even your defects that don’t make it into the “field.”
In the postmortem phase, it was definitely eye opening to find that I spent 49 minutes in testing! That’s 46 more minutes than I predicted!
PSP 0.1
Description
This level adds program size measurements, process improvement proposal, and a coding standard. The process improvement proposal is “a form that engineers use to record the process problems they encounter as well as their ideas for addressing them.” [4]
Takeaway
This level was a little more frustrating than the first. I think the challenge of doing a self-study of PSP is that instructions tend to lack a great deal (from the online materials). I initially didn’t understand (and still might not) the difference between base, deleted, modified, added, reused, and added and modified program size aspects. The especially annoying bit here is the reused category. In my opinion, since, I’m not designing libraries, my code is never 100% reusable like a library is meant to be.
In addition, the process improvement proposal (PIP) form, I found, was not very useful as again, I wasn’t sure what parts of the process I could question or what I could do about it to begin with. In hindsight, I’m glad I stuck through with most of the process suggestions as I now have a better understanding and appreciation of some of them.
On a more positive note, the coding standard was useful. As developers, we often have a personal coding style but, unfortunately, if it’s not documented we often end up straying from it.
PSP 1.0
Description
This level was exciting. You will finally reap some benefits. “In this level, engineers are introduced to the PROBE method, which uses historical data to estimate size and determine the accuracy of the estimate. PROBE is a regression-based size-estimating method developed specifically for PSP.”[4]
In addition to previous data, this level asks you to predict program size by breaking down the design into parts added and reused.
Takeaway
At this phase, you will get a lot more involved in the design as you are asked to list out the parts (functions, classes, modules) you’ll need to design your program. The notes on this level tell you to create a conceptual design in the planning phase in order to create somewhat of a parts inventory to allow you to predict how long each part and thus the whole program will take. Up to this point I was using a word processor to keep track of all my data which made the PROBE calculations a little cumbersome. These in the later phases lead to me dreading the planning phase the most! Luckily though, there’s a tool that makes all of this really easy, so use it: Process Dashboard.
PSP 1.1
Description
“This level adds resource and schedule estimating and earned-value tracking. Engineers often have trouble tracking their work because they do tasks in an order different from their plan. Earned-value tracking allows them to weight the relative importance of each task and to judge their progress as they finish some tasks early and others late.” [4]
Takeaway
The additions in this level seem really promising. It’s quite easy to get distracted by different projects at different times while letting other priorities slip. However, with the schedule and task planning you can keep track of your progress and your projected finish date. This allows you to renegotiate a deadline if you’re sure you will miss it. Having said that, the task planning and schedule planning template are meant to be used in project that will take more than one day. I attempted to use this in my 5th project, however, it didn’t prove too useful as it did not take more than one day. Interestingly enough, in the project that I should have used these tools (Project 6), I didn’t, since I anticipated to spend about 8 hours on it instead of the 45 hours the PROBE methods suggested! (Hint: I spent about 50 hours on Project 6!)
PSP 2.0
Description
“This level introduces design and code reviews, as well as quality measurement and evaluation. Using defect data from their earlier exercises, engineers also develop personal design and code review checklists.” [4]
PSP 2.0 does come with prebuilt design and code review checklists.
Takeaway
When completing the checklists you’re not supposed to work on more than one item at a time. For example, if you’re checking for include statements, that’s all you’re doing. These checklists require a lot of discipline to not rush and get them over with. Perhaps, if feasible, it’s a good idea to postpone these reviews for some time (be it hours or days), in order not to speed through them and miss critical defects. From my personal experience, although I have not followed these to the dot (because some didn’t apply and some seemed too burdensome) they have proved very useful. A much better practice, which I shall do in the future, is to modify these checklists to suite your needs by examining the time of defects you’ve produced in the past.
PSP 2.1
Description
This level breaks down the design phase into functional specification, operational specification, logic specifications, state specification. The functional specification is essentially a UML diagram with method specifications. An operational specification can be compared to a use-case diagram. The logic specification is, as the name suggests, the logic of an item (e.g. method) and finally a state specification is a state machine.
Takeaway
I used PSP 2.1 for Project 6 which was essentially maintenance work on my textbook website. Instead of following these different specifications, I decided that a UML diagram representing the changes I intend to make to the software would be sufficient. I also added documentation to the classes and functions in order to cover functional specifications. As the use cases for my website are well defined as it’s already public, I didn’t feel a need for the operational specification or use-case diagrams. Likewise with the task at hand, a logic specification wasn’t needed as I was replacing a module with another one and not fixing any existing logical defects. And finally, a state chart diagram seemed too cumbersome to even start–so I didn’t! These pain points of the PSP will be reflected in my own process which I will introduce below.
Part 3 will be posted shortly. It basically describes some changes I intend to make to PSP to make it work for me. PSP is meant to be personalized and cusotmized, hence the name, Personal Software Process.
This is a series of three posts on my experience with the PSP.
PSP stands for Personal Software Process. For an introduction, see the wikipedia page on PSP. This is part 1. Part 2 can be found here.
What is the Personal Software Process℠?
The Personal Software Process (PSP℠) was developed by Watts S. Humphrey from the Software Engineering Institution of Carnegie Mellon University. PSP℠ is a highly disciplined software development process. However that’s not all–it’s even described as the process that helps you develop software processes and not necessary a software process in itself. PSP℠ focuses on strict measurement of software development activities starting from planning, designing, coding, compiling, testing, and postmortem.
Why should you care?
In my Software Engineering class, through a number of papers (see References for a list), I was introduced to the Personal Software Process℠. These papers made some very seductive claims on improved completion estimates and software quality and they got my attention. Here are PSP success stories:
At Advanced Information Services (AIS), before PSP training, schedule estimating error averaged 394 percent; afterward PSP training the average was -10.4 percent. That is, projects were being completed even before their estimated deadline.[4]
Also at AIS, the software for a PSP trained team had 0.76 defects per 1,000 lines of code before PSP training and 0.17 defects per 1,000 lines of code after training, a 78 percent improvement. [4]
In addition to the above, AIS compared the results of project with PSP trained engineers to those with engineers not trained in PSP. Here are the results [4]:
Project C (completed by non-PSP trained engineers) was scheduled for two months, but took five to reach the acceptance test stage.
Likewise Project D (again, completed by non-PSP trained engineers) was scheduled for 10 months, but took 19.
Project B, however, was scheduled for seven months, but took five. This was completed by PSP trained engineers.
At Motorola, of eighteen projects completed by PSP-trained engineers, only one defect had been found after release.[2]
At Union Switch and Signal, PSP-trained engineers completed five projects and no defects had been found after release.
And finally, in my PSP training Project 1 consisted of a trivial 31 lines of code (LOC) which took 2.38 hours to complete but my schedule estimation error 26.55 percent. Project 6, on the other hand was a much more complex maintenance project with 247 LOC which took 50.42 hours to complete and I only had 10.36 percent schedule estimation error. (In reality if I had estimated Project 6 without the PSP tools like I estimated Project 1, I would have estimated development time to be about 8 hours which would have resulted in a 92.85 percent in estimation error!)
Further, you might wonder, while those numbers look really good, why are they so important. When it comes to schedule estimation, it’s important to get it right because you likely will be paying people (or are getting paid) to work on the software. These costs are covered by the client of the software (otherwise you’re not making a profit). If you underestimate how long it will take to develop a particular product (be it custom or packaged software), you’re bound to charge less than feasible prices and end up incurring a loss. This can lead your business (or employer) into trouble.
Likewise, quality is very important. Removing defects gets more expensive as you progress in the development of your software (e.g. a defect that might cost one dollar to fix in the requirements phase may cost $10 during design, $100 during coding, $1,000 in test and $10,000 in the field.)[2] Again, if you reduce the number of defects that make it to the latter phases of a project you are contributing to your company’s (or your employer’s) bottom line.
In a nutshell, that’s why schedule estimation and product quality are important in a software project. That is, without even considering the marketing benefits reaped from a quality product and good estimation.
How do I start?
If you’ve come to the conclusion, as I did, given these potential benefits it is worth trying PSP out, I recommend you start with the Self-Study PSP material you can download from the SEI website.
That was part 1. I’d love to hear your opinion on this especially if you’ve tried PSP or a “competing” development process like SCRUM, etc. Until part 2!
I had some work to do but instead found a perfect distraction. I decided to upgrade my textbook website to Django 1.2.x which is an updated version of the python framework I used to build it.
Unlike in the past, where such a task would have not phased me and which I would have attempted to do on the live site, I decided to setup a beta site that runs along the main GizmoBooks website as a sandbox so to speak.
I started upgrading and it turned out there were many changes that needed to be made. I came across this very helpful guide on intalling python 2.6, virtualenv, and virtualenvwrapper. After a few attempts and tweaks it worked and beautifully too! Not only did I have python 2.6 installed but I also had virtualenv andvirtualenvwrapper which I until I had it, I had no idea what I was missing. It’s quite brilliant because it allows you to setup virtualenvironments like this *snaps fingers*.
So far so good, I had my virtual environment setup but then came time to find all the different python PIP packages and other software that my website needed. This was okay for a bit, but then came time to install xapian-core, xapian-binding, blah blah blah. I quickly got tired of all this.
The Script
So, I did what all lazy programmers do: I wrote a script about it! The script is called deploy_pysite.sh and can be found at GitHub.
You see, the beauty of virtualenv is that it allows you to quickly create a virtual environment from scratch. The problem, of course, is that (often) the virtualenv is empty and you need to go install all the packages to deploy your website. This is what the scripts aims to simplify.
Script Guide
Follow these steps to run it:
If you’re not using virtualenvwrapper, set VIRTUAL_ENV environment variable to your absolute virtual environment path. If you are using virtualenvwrapper, just “workon” the virtual env you wish to install the packages in.
Add to, or remove from nonpip_packages and pip_packages arrays in SETTINGS section of deploy_pysite.sh source code. Use the formats specified in comments.
(Optional) The default directory where software is downloaded and built is $HOME/downloads. This is set in dg_downloads_dir in deploy_pysite.sh SETTINGS section. Feel free to modify it.
Execute the script.
That’s pretty much all there is to it.
Not surprisingly, it proved usefulx100 immediatly. You see, I was following another guide from andrew.io on installing Django with virtualenv on dreamhost and low and behold the guide said to “[i]nstall any packages you may need for your project.” ./deploy_pysite.sh was all I had to say at that point .
Script Source
I’ll post the source here, but it’s probably best to just grab it from GitHub.
#!/bin/bash# ensure that exceptions are treated as such and program exists!set-e# =================================================# = deploy_pysite version 0.1.0# = # = Created by Gezim Hoxha# = License: GPL 2# =# = If you're using virtualenv, this script will# = install PIP packages and even build # = build packages from source then install# = them into your virtual environment.# = See SETTINGS section for instructions.# =# = July, 2010# =# = TO DO# = - # =================================================# = REQUIRES# = bash >=3.1# = curl# = pip# =================================================usage="
USAGE: \n
Modify nonpip_packages and pip_packages according to your needs. These can be found in the source
of this script in SETTINGS section.\n
Once SETTINGS have been modified, activate your virtual environment, then call this script. \n
If you're not using virtualenvwrapper, set VIRTUAL_ENV environment variable
to your virtual environemt directory. \n\n
Also, you may want to ensure that the download dir is set to where you want it in SETTINGS section.
"# ensure we're in a virtual environmentif[["$VIRTUAL_ENV" = ""]]thenecho-e$usageexit5fi# =================================================# ====================SETTINGS=====================# =================================================# Directory to store source archivesdg_downloads_dir="$HOME/downloads"# Add/remove non-pip packages you [don't] want to install here.# IMPORTANT: Only .tar.gz packages are supported.## You must add a package URL and function pair to nonpip_packages array.# If no custom install is required (i.e. normal ./config, make, make install will do) leave # function call empty. For example:# "http://www.example.com/pack.tar.gz" ""# If you need custom install calls, create a function that makes those calls in CUSTOM FUNCTIONS# area below. The function calls can include arguments.# For example:# "http://www.example.com/pack02.tar.gz" "install_pack02"# Or with a function call argument:# "http://www.example.com/pack04.tar.gz" "install_package four"## MAKE SURE YOU USE QUOTES.#nonpip_packages=("http://oligarchy.co.uk/xapian/1.0.21/xapian-core-1.0.21.tar.gz""""http://oligarchy.co.uk/xapian/1.0.21/xapian-bindings-1.0.21.tar.gz""function_xapian_bindings")# Add/remove pip packages you [don't] want to install here.# Version is included to ensure you get the same packages# whenever you install.# Example:# "django" "1.2.1"## MAKE SURE YOU USE QUOTES.#pip_packages=("django""1.2.1""django-haystack""1.0.1-final""xapian-haystack""1.1.3beta""MySQL-python""1.2.3c1")# =================================================# =================CUSTOM FUNCTIONS================# You should use variables such as $VIRTUAL_ENV.# =================================================
function_xapian_bindings(){
./configure --with-pythonXAPIAN_CONFIG="$VIRTUAL_ENV/bin/xapian-config"--prefix="$VIRTUAL_ENV"makemakeinstall}# =================================================# ====================REAL CODE====================# ===Don't touch unless you are sure you need to.==# =================================================# =================================================# Create the downloads dir.# Nothing will happen if it already exists.mkdir-p"$dg_downloads_dir"temp_file="/tmp/$(basename $0).$$.tmp"# Build and install non-pip packages
build_install(){for((i=0; i<${#nonpip_packages[@]}; i+=2))do#pip_install "${pip_packages[i]}" "${pip_packages[i+1]}"#get package namepackage_name="${nonpip_packages[i]##*/}"package_dir="${package_name%.tar.gz}"echo"Installing $package_name to $VIRTUAL_ENV with..."cd"$dg_downloads_dir"# Check if directory exists.# -d test doesn't work on case insensitive FS's (e.g. OS X)if(cd"$package_dir"2>/dev/null)thencd"$package_dir"elif[-f"$package_name"]then# if package is already downloadedtar xzf "$package_name"cd"$package_dir"else
curl -sS"${nonpip_packages[i]}"-O>/dev/null
tar xzf "$package_name"cd"$package_dir"fi# if odd array index has no function, do a standard make# Otherwise, run custom function.custom_function=${nonpip_packages[i+1]}if[["$custom_function" = ""]]thenset-x
./configure --prefix="$VIRTUAL_ENV">>"$temp_file"2>&1make>>"$temp_file"2>&1makeinstall>>"$temp_file"2>&1set +x
elseset-x"$custom_function">>"$temp_file"2>&1set +x
fiecho"...done."done}# Install pip packages# param: $1 is the package name# param: $2 is the package version# The package version is required# to ensure the exact setup is installed everytime.
pip_install(){if(($#!= 2))thenecho"pip_install requires 2 arguments."return25fiecho"Installing $1 to $VIRTUAL_ENV with..."set-x
pip install"$1"=="$2">>"$temp_file"2>&1set +x
echo"...done."}# Now install the packages if they're enabled.
build_install
for((i=0; i<${#pip_packages[@]}; i+=2))do
pip_install "${pip_packages[i]}""${pip_packages[i+1]}"doneecho-e"\nInstall was successful. See $temp_file for details."
Needless to say, I had the darnest time doing this. Upgrading mysql was not too big of an issue, but getting MySQLdb to work was a whole other game.
Anyway, here are the steps I took.
The instructions for upgrading MySQL came from entropy.ch and are as follows:
Save your current database data before doing anything: cd /usr/local/mysql; sudo tar -cvf /tmp/mysql-data.tar data
Then download the MySQL installer from the MySQL website. At first I chose to download “Mac OS X ver. 10.6 (x86, 32-bit), DMG” which, I believe, caused all my problems with MySQLdb. So download Mac OS X ver. 10.6 (x86, 64-bit), DMG unless you know you have a 32 bit machine.
Once downloaded, unpack and install the MySQL package and even the prefPane to make your life easier.
To restore your old data, run this command: cd /usr/local/mysql; sudo tar -xf /tmp/mysql-data.tar
I’m not sure if this is needed, in the attempt that worked, I followed the instructions to the dot so I did reboot my computer (although the MySQL install didn’t require me to).
Start MySQL either through the System Preferences (if you installed the prefPane above) or by running: sudo /usr/local/mysql/bin/mysqld_safe
That’s all there was to installing MySQL. Note that in my system I didn’t have to change the ownership of the database data when it was “imported” into the new install. If you have to do that (i.e. can’t login with the usual username/password combo) see the instructions in the link I posted above.
Untar it and ensure “/usr/local/mysql/bin” is in your PATH, if not see the above link on how to add it.
Run this command: ARCHFLAGS="-arch x86_64" /usr/bin/python setup.py build (Ensure you don’t have a 64 bit machine before changing it.)
And finally, install it with the following command: sudo /usr/bin/python setup.py install
Once all that is done. You can test it like this:
$ python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>
If that gives you no errors, you’re set. Now get back to doing actual work .
I just finished listening (actually still not finished but just watching the question period ) to this DjangoCon talk by James Bennet. Found it very motivating and useful. There are a lot of apps out there! Before you start to write some feature for a django site, ensure that you’re not rewriting something that someone already wrote. Here are a few site to look at:
I wanted to blog about this issue a while ago, however I didn’t get a chance until now, thanks to Dreamhost. You see, I rely on subversion (hosted on my dreamhost account) for my revision control and my repositories are unaccessible for the second time in 9 days! With the exception of this latest issue I’ve been happy with dreamhost. Update: they fixed this issue about 16 hours after I contacted them.
The Problem
Now, getting back (or starting) on topic. A long time ago (dinosaurs used to roam the streets then) I was faced with the issue of designing Object Oriented Code (OOC, hmm haven’t seen that acronym before. Yeah, I just made it up on the spot . Yah, I’m that good). Normally we’re taught that in OOP the classes you create should represent (or go as close to it as possible) real world objects (containing information in fields and defining possible action in the methods). For example, a car class could have model, manufacturer, and owner as fields and driveTo(…) as a method. This is all fine and dandy for little school assignments where your objects are all created in the code and nothing is stored on databases or even flat files. However as soon as you start thinking about using a database a problem arises. When an object is created and manipulated and all that fun stuff, it needs to live somewhere after the program stops executing (can’t live on RAM, obviously) so a database is often needed to store the existing objects. In other words these fields (or the state of an object needs to be stored in a database). So, long ago, I was working on a project and needed to represent members (of a website). I had come up with the database and fields for my class (see Figure 1).
Figure 1 - Representation of my class and the database table for it. Don't pick on my UML diagram as I still don't know how to properly make one. I make them so I understand them, just like my code
The table columns simply mirrored the fields of my member class and there were methods to retrieve member information such as getAge(), etc. and I could nicely do this (PHP):
$newMember = new member(5)
echo $newMember->getAge()
Now that looks okay, except for one problem. Problem is that every time you have a new class or object you have to:
create the database table to represent that object
create the class in code (using language of choice, of course)
and define methods to link the two, including the necessary SQL
DataAccess layer–deals with querying the database and making the results available to the object
DataTransferObjects layer–like the member class in Figure 1
and the GUI layer–can interact directly with the object we are dealing with and doesn’t have to worry about the database
This method might work for some people but my main criticism for it was that code duplication would have to happen much like my initial solution (this was one of the problems with the initial solution) and second, the resulting code in the GUI methods wasn’t very object-oriented looking:
Products p = new Products() { Name = txtName.Text, ... };
ProductsOperations products_op = new ProductsOperations();
bool isRegistered = products_op.Register(p);
In addition, I don’t find that easy to read. What I’d rather have is something like this:
Products p = new Products() { Name = txtName.Text, ... };
bool isRegistered = p.isRegistered(); // this is more like it
The best solution
It turns out this problem has existed for a while (I guess I don’t get credit for finding the problem, let alone the solution!). The nice solution that will allow you to just relax a bit more (see Figure 2 for a demonstration) is Object-relational mapping (or ORM).
Figure 2 - Relax.
ORM simply maps your class (or object, if you will) to a database table(s) and if the ORM system you’re dealing with is powerful enough, it’ll handle even the relationships between your objects! Sweet, right? Okay, I’m beginning to sound like a snake oil salesman, so let’s look at Django’s ORM implementation (I looked at some PHP ORMs such as propel and repose but they didn’t seem nearly as nice as Django’s ORM implementation).
In Django you first define a model (this is all python):
from django.db import models
class Member(models.Model):
first_name = models.CharField(max_length=30)
last_name = models.CharField(max_length=30)
Then you simply execute this command: python manage.py syncdb and Django creates all the necessary tables for the database that you’re running (could be sqlite, MySQL, or PostgreSQL).
Once Django creates the database, you’ll be able to use the Model in normal python code and not even worry about the database communication and so on. The ORM even protects you from SQL injection attacks as you won’t be executing SQL commands directly.
So, now we have this:
newMember = Member("John", "Smith")
newMember.save()
# This creates a person with first name John and last name Smith. It's that simple.
or for Milot’s problem:
#define the model
from django.db import models
class Product(models.Model):
# The ID is auto-generated as the primary key,
# so we don't have to create it.
name = models.CharField(max_length=30)
description = models.CharField(max_length=100)
price = price = models.FloatField()
then we can access Product information like this:
newProduct = Product("Audio R8", "Only the awesomest car in the World!", 110000.00)
newProduct.save()
#then we could retrieve the recored we just created
sameProduct = Product.objects.get(name="Audio R8")
I should mention that Django is a python based Web framework, or as its developers claim The Web framework for perfectionists with deadlines. Who am I to argue
Thanks for reading. Your comments or corrections are much appreciated.
Where Am I?
You are currently browsing the programming category at
life of a gizmo.
.
