Tag Archives: python

Why Tendenci Chose Python over PHP

This blog is a WordPress blog written in PHP. And WordPress, when secured properly, is a great platform.

So why did our team choose to rewrite Tendenci Open Source and in the Python Programming language? It is a question I get asked a lot. We’ve never been a company that likes to talk in the negative if at all possible, yet it is important to talk about the megatrends going on given we work with associations and nonprofits.

which_web_programming_language_is_the_most_secure_

Source: https://www.upguard.com/blog/which-web-programming-language-is-the-most-secure

security-report1

Source: http://info.whitehatsec.com/rs/whitehatsecurity/images/statsreport2014-20140410.pdf

security-report

Source: http://info.whitehatsec.com/rs/whitehatsecurity/images/statsreport2014-20140410.pdf

Popularity of a language is a trend, and what you want is as many developers familiar and liking the language of your open source project as possible. This means you have a better chance to have a secure web site and therefore a more secure future.

To be fair – as Disraeli said – “lies, damn lies and statistics” – so there is no one perfectly secure language any more than there is a perfectly “safe” hammer. There will always be operator error and programmers make mistakes.

So we’re not saying Python is perfect, and all of us have used most of the other languages on those charts at some point. We’re just saying we are pleased so many other programmers also like Python and Open Source. THAT is the best that can be done to secure your future online. Secure code that you can examine yourself and even host yourself!

state

State
Impure functions are often more efficient but also require that the programmer “keep track” of the state of several variables. Keeping track of this state becomes increasingly difficult as programs grow in size. By eschewing state programmers are able to conceptually scale out to solve much larger problems. The loss of performance is often negligible compared to the freedom to trust that your functions work as expected on your inputs.

Maintaining state provides efficiency at the cost of surprises. Pure functions produce no surprises and so lighten the mental load of the programmer.

http://toolz.readthedocs.org/en/latest/purity.html

Installing iPython Notebook on a Mac OS X Mountain Lion

Installing iPython Notebook on a Mac OS X Moutain Lion 10.8.1 for Development and Testing

The “Quickstart” is anything but that leaving off details like the recomended dependencies are basically required for a functional notebook. Here is the sequence that worked for me.

[apps]$ mkvirtualenv ipythonvm –distribute
[apps]$ workon ipythonvm

at this point pip freeze shows

[apps]$ pip freeze
distribute==0.6.28
wsgiref==0.1.2

next up the big instAll

[apps]$ easy_install ipython[zmq,qtconsole,notebook,test]

the instructions suggest running iptest. Don’t bother yet. It will fail the tests without more dependencies like “nose’ installed. Keep on. Also note the “easy_install” in front of readline is specific to set the sequence to pass the tests. Not sure why. No time to question today and I’m in a VM so it can’t do much harm. Proceed with.

[apps]$ pip install nose
[apps]$ easy_install readline
[apps]$ easy_install pexpect
[apps]$ easy_install ipython[zmq]

At this point checking packages shows

(ipythonvm)LOCAL:~/dropbox/code/apps
[apps]$ pip freeze
distribute==0.6.28
ipython==0.13
nose==1.2.1
pexpect==2.4
pyzmq==2.2.0.1
wsgiref==0.1.2

but

[apps]$ ipython qtconsole

fails. Keep trying.

pip install Tornado

Tornado works, but still errors on ZeroMQ. It comes down to this. We need ZeroMQ and PyZMQ. PyZMQ is installed (see pip freeze lista bove) but is missing the dependencie ZeroMQ doesn’t work with pip or easy_install as far as I can tell. Now we go old school. I also didn’t read the docs and was a bit tired so rather than specifying to intsall in the virtualenv I accepted the defaults and installed it in /usr/local/bin globally.

for ZeroMQ, I downloaded the Mac latest stable release from this page:
http://download.zeromq.org/

On 9/15/2012 I downloaded this one and extracted it:
zeromq-2.2.0.tar.gz 14-Apr-2012 09:53 1.8M

Unpack the .tar.gz source archive and cd into that directory. Remember I installed globally for this one package after the battle so first I had to “deactivate” in the virtual environment I was in (iphythonvm for me). Then change into that directory someplace you have “write” rights. For me I copied it from downloads to “code/contribs” which is where I put random stuff I haven’t modified but may or may not be using. Thus the next command was:

cd zeromq-2.2.0/

Run ./configure, followed by make.

But remember, outside of a VM we are back to sudo so this looks like:

sudo ./configure
password:
sudo make
sudo make install

switch back to my VM

[apps]$ workon ipythonvm

the environment now looks like this:

(ipythonvm)LOCAL:~/dropbox/code/apps
[apps]$ pip freeze
distribute==0.6.28
ipython==0.13
nose==1.2.1
pexpect==2.4
pyzmq==2.2.0.1
wsgiref==0.1.2

now we try again.

[apps]$ ipython notebook

Success! Now let’s go get somethign to look at. I created a folder in my apps folder to put the downloads and grabbed a git repository from blogger Titus of Living in an Ivory Basement http://ivory.idyll.org/blog/teaching-with-ipynb.html

mkdir iPythonNotebooks
cd iPythonNotebooks
git clone git://github.com/ngs-docs/ngs-notebooks.git

Now we test it again:

[iPythonNotebooks]$ ipython notebook
[NotebookApp] Using existing profile dir: u’/Users/eschipul/.ipython/profile_default’
[NotebookApp] Serving notebooks from /Users/eschipul/Dropbox/Code/apps/iPythonNotebooks
[NotebookApp] The IPython Notebook is running at: http://127.0.0.1:8888/

I then imported the ngs-10-blast notebook. There are codeblocks in the notebook, so for a proof of concept I just picked one that imported “blast” knowing I had not imported it. Inside of Chrome, inside of the notebook, I can click on a codeblock that begins:

import csv
import blastparser

# open the output file for reading
fp = open(‘out.txt’)

and then I select “cell run” and it runs it as if I was in Eclipse. Properly giving an error of

—————————————————————————
ImportError Traceback (most recent call last)
in ()
1 import csv
—-> 2 import blastparser
3
4 # open the output file for reading
5 fp = open(‘out.txt’)

ImportError: No module named blastparser

That is huge. Think about it. Running code inline in the middle of a web page with a compiler and debugging. Not javascript by Python in a sandbox that can import modules and do everything else you would do with idyl. And saved in JSON based notebooks that can be shared and used for testing. It sort of blows your mind.

I’m not suggesting this is a replacement for a good training video. But it is a great addition to the educators arsenal of tools for online learning in richer environments with greater interactivity. I’m impressed to say the least.

Other randome take-aways. I did not know about the pexpect package and it is pretty compelling if you work at a company. It removes the need for the C libraries for builds which means that not everyone needs to install xCode if they have to /configure, make, make install, etc.

Thus I recommend you take a look at what pexpect can do as it was new to me. Pretty cool actually.
http://www.noah.org/wiki/Pexpect#Description_of_Pexpect

pycrypto install fails on mac osx mountain lion in virtualenv

Receiving the following errors trying to install pycrypto in a virtualenv on mac osx mountain lion. Assuming the name of the virtualenv is “PROJECTVIRTUALENV” and that your project is PROJECT my first try was:

[powershell]
$ mkvirtualenv PROJECTVIRTUALENV
……… change into the project folder
$ pip install -r requirements.txt
……… everything installs except pycrypto
$ sudo pip install pycrypto
[/powershell]

that also fails resulting in:

[powershell]
File "/Users/eschipul/.virtualenv/PROJECTVIRTUALENV/build/pycrypto/setup.py", line 278, in run
raise RuntimeError("autoconf error")
RuntimeError: autoconf error

[/powershell]

This was a new one on me so what changed? A few things. First virtualenv and virtualenv wrapper now default the –no-site-packages, but you were using that already.

Second the upgrade from OSx Lion to OSx Mountain Lion required an update of the command line tools with xCode. I’ve tried the stand alone xCode tools and they don’t keep up with Macs updates so I have the whole bloody xCode installed on my machine. To resolve the install problem with pycrypto it needed the new compiler. Simply updating to xCode 4.4 does not solve the problem by itself.

Per a note half way down this blog post on similar installation troubles by jiaaro you need to re-download the command line tools. From the post:

You need to install Xcode 4.4 (from the app store) and then, within xcode open Xcode > Preferences (or press Cmd +,) then open the downloads tab and install the Command Line Tools.

which looks like this:

and results in this:

[powershell]
sudo pip install pycrypto
password:
…. yada yada yada …. a warning or two …. then
Successfully installed pycrypto
Cleaning up…
[/powershell]

Note that I *did* already have the command line tools installed on xCode 4.3. The upgrade didn’t match my previous config and I had to explicitly download them again. Hope this saves some other poor soul some time as it makes no sense to me. But there it is.

mind as water as programming

There is a natural conflict between youth and programming. Frequently the best programming ideas, like in mathematics or chess, come from young people. Sometimes under 30. And in my experience focus becomes harder as you age. Young people don’t know what they can’t do and this often allows them hit the ball out of the park! A very good thing indeed!

The contradiction is the young guns with the fresh mental horsepower have the least experience and frequently charge around randomly between the latest cool “philosophy” and “tool” making less progress than an older programmer who might quickly determine what to avoid. And youth can sometimes be unwilling to compromise the reality of programming economics (code actually runs on hardware I’m told) and the beauty of full normalization. (Hint: your college prof was wrong. Talk to an experienced dba and she will speak the truth. Normalization is a goal like reaching the axis on an exponential curve.)

Currently I am trying to catch up with my team on Python and Django (they have really left me in the dust at the moment) I enjoyed this quote from DRY:

My biggest programming headaches have always come from abstractly struggling with “how can I write a good general solution to this problem”, even though I only know of one place where it’s definitely going to be used. “I’d better think of a general solution now,” I think, “or I’ll have to copy-and-paste-and-change code!” But it’s absurd to try to come up with a general solution without knowing more about the different varieties of the problem that exist (or will exist) in the system.

It’s a battle of two really strong urges - OnceAndOnlyOnce vs avoiding Premature Generalization. Do I duplicate for now and try to live with the duplication for a while, or violate YagNi and come up with some half-cocked generalized solution? It’s a tough one, because almost all programmers hate duplication; it’s a sort of primordial programming urge.

However, even though CopyAndPasteProgramming can be expensive to clean up, so is a botched “general solution” – and copy and paste is far cheaper up front. So I also am in favour of temporary duplication, to be refactored when you have a clearer view of the situation. – MatthewBennett

In our shop I have always called it “do it once for all time for all users.” But it means the same thing. Repeating yourself and not solving issues at the root level is a waste of time. I can’t tell you how awesome it is to not have to explicitly declare getters and setters in Python (vs classic ASP where classes don’t even have inheritance.).

Yet there is a catch. At the beginning of a project or approaching a new industry you can’t really do “root cause analysis” because you don’t have any data. To requote the quote above –

However, even though CopyAndPasteProgramming can be expensive to clean up, so is a botched “general solution” – and copy and paste is far cheaper up front. So I also am in favour of temporary duplication, to be refactored when you have a clearer view of the situation. – MatthewBennett

– this balance is the solution and yes it is a bit messy. If the goal of programming is to make a profit then you can’t just argue philosophy without discussing the economics of it. Economics as in money. And economics as in sometimes duplicate code or duplication of data just runs faster on the servers, increases your search engine rankings and lowers your operating costs. I can live with a bit of duplication for that reward.

Often the best solution is to collect data and slam it together and then identify the best organizational structure before the corpus gets too large. Programming classes or actual data both. A filing system for terrabytes of photos and videos for example. It is a balance between the default import settings of all of the hardware you use and the need to compartmentalize source and work product for portability and backups. Yes, you have to compromise to the machines somewhat because you can’t change the settings on every other camera in the world.

Let the data build up a bit. Don’t fight the hardware. Live with some duplication. Then generalize and normalize as best you can. Balance. “Mind as Water” as Lee would say.

the rules driving python

  1. Borrow ideas from elsewhere whenever it makes sense.
  2. “Things should be as simple as possible, but no simpler.” (Einstein)
  3. Do one thing well (the ‘UNIX philosophy’). And that was to create a great programming language that can be used anywhere.
  4. Don’t fret too much about performance – plan to optimise later when needed.
  5. Don’t fight the environment and go with the flow.
  6. Don’t try for perfection because ‘good enough’ is often just that.
  7. (Hence) it’s okay to cut corners sometimes, especially if you can do it right later.
  8. The Python implementation should not be tied to a particular platform. It’s okay if some functionality is not always available, but the core should work everywhere.
  9. Don’t bother users with details that the machine can handle.
  10. A large complex system should have multiple levels of extensibility. This maximizes the opportunities for users, sophisticated or not, to help themselves.
  11. Errors should not be fatal. That is, user code should be able to recover from error conditions as long as the virtual machine is still functional. At the same time, errors should not pass silently.
  12. A bug in the user’s Python code should never be allowed to lead to undefined behavior of the Python interpreter; a core dump is never the user’s fault.

LinuxUser, Python, the Universal Programming Language paraphrasing Guido van Rossum