Why Tendenci Chose Python over PHP

This blog is a WordPress blog written in PHP. And WordPress, when secured properly, is a great platform.

So why did our team choose to rewrite Tendenci Open Source and in the Python Programming language? It is a question I get asked a lot. We’ve never been a company that likes to talk in the negative if at all possible, yet it is important to talk about the megatrends going on given we work with associations and nonprofits.

which_web_programming_language_is_the_most_secure_

Source: https://www.upguard.com/blog/which-web-programming-language-is-the-most-secure

security-report1

Source: http://info.whitehatsec.com/rs/whitehatsecurity/images/statsreport2014-20140410.pdf

security-report

Source: http://info.whitehatsec.com/rs/whitehatsecurity/images/statsreport2014-20140410.pdf

Popularity of a language is a trend, and what you want is as many developers familiar and liking the language of your open source project as possible. This means you have a better chance to have a secure web site and therefore a more secure future.

To be fair – as Disraeli said – “lies, damn lies and statistics” – so there is no one perfectly secure language any more than there is a perfectly “safe” hammer. There will always be operator error and programmers make mistakes.

So we’re not saying Python is perfect, and all of us have used most of the other languages on those charts at some point. We’re just saying we are pleased so many other programmers also like Python and Open Source. THAT is the best that can be done to secure your future online. Secure code that you can examine yourself and even host yourself!

django errors using boot2docker tutorial

If you are doing the docker-compose / boot2docker tutorial on the docker site at:

https://docs.docker.com/compose/django/

There are a few typos and items that will stop you, at least on a Mac although I suspect they are an issue on a PC as well. These are my notes, out of context but hopefully they will help someone else having problems with errors and configuration of Django Dockers for their dev environment.

boot2docker up
docker-compose up
# the following line runs it in the background
docker-compose up -d
To connect the Docker client to the Docker daemon, please set:
 export DOCKER_HOST=tcp://192.168.59.103:2376
# this tells you your environment variables
docker-machine env
http://192.168.59.103:5000/
# The following is incorrect in the docs
docker-compose run web python composeexample/manage.py syncdb

Rough notes, but hopefully enough to help you google for more if you are hitting the same wall I did with boot2docker and docker-compose for Django.

Update: also helpfule https://github.com/ambitioninc/container-transform

Yes, software really is a mission

I quoted what I believe is a great blog post below by Michael Stanton. It relates to our company as well because we work primarily with non-profit organizations. Business or NPO, you work with a company online and it’s a relationship and not a transaction. You share a vision, or it won’t work. Period.

Does your RFP ask what their vision is? What their mission is? Or just for a copy of their financial statements?

Yes, software really is a mission. For our company I call our overarching beliefs and values our vision statement. The reason I look forward to work versus showing up early but only for a paycheck and jetting out the door at 5:51 is the vision. You can’t sustain 15 years for the money. Here is our vision statement:

“To Connect and Organize the World’s People. Do Good.”

When reading the article below replace the word “mission” with “vision” and it is a fair test on if you should do business with us.

If you don’t share that vision, then it won’t work. It just won’t. Sure we will make you money because we are really good at the whole marketing thing after years of study (15 years old, 30+ employees, 400+ clients, there is a reason agencies are constantly trying to steal our people and buy our company! Curious why I say no? Because I don’t believe in THEIR vision. I did the 9 to 5 thing at companies I didn’t believe in. It was hellish.)

Call it a vision. Or a mission. Whichever, yet THAT is why you should or should not do business with a company. Because I promise you as our CEO that the vision drives Tendenci 100%. And we don’t hold much back from the Tendenci open source for non-profits either. Here is the full quote from the article:

Software as a Service is no longer an accurate description of the paradigm of innovation, of the relationship between customer and service provider. We need a more accurate term.

Software as a Mission.

Software can move so fast that customers are not only not buying a static product anymore, they are also not subscribing to a defined service, they are now believers in a mission and hanging on for the ride. And the ride is fast enough to be a bullet train, but can also be a roller coaster. Companies that seem promising can suddenly get acquired, or go down in flames from premature scaling. They can get a strong competitor coming out of left field.

The question is no longer “Do you like the product?” As much as: “Do you believe in the company? Do you believe in their direction? Do you believe in the team?”

And if you bet on the wrong horse, it’s not as big of a deal as it used to be. You just take your credit card to the next one doing the thing you wanted doing. No big deal. The cost of implementation is usually just people hooking up their identities and choosing a password, at most uploading a spreadsheet.

By the way, this also means you won’t just have one vendor for what your communities or teams need. You’ll likely have several, and functionality will overlap. We’re going to have to be Zen about that.

So, let me ask you this question: think about your vendors. Picture them. Do you believe in the company? Do you believe in their mission, their direction, their team? Believing is so important because great teams can ship software really quickly, and what you have next year will not be what you have this year. Believing is important because small teams of people can now produce software that millions of people use. (At one point there were almost 2 Million Twitter users for every Twitter employee, same goes for Instagram.)

(excerpt from )

And I’d like to highlight one part again. Because oddly enough it applies in both directions. We have a sales incentive program that does not discern between clients who share our values and those that don’t. You can look back at every deal that unwound and it is either communication or a lack of aligned values. In the next paragraph, repeated from above, i have replaced the word “vendors” with the word “clients” and changed the audience to our employees, outsourcers, vendor partners and to the extended tribe. To (slightly) misquote Mr. Stanton again:

So, let me ask you this question: think about your vendors. Picture them. Do you believe in the company? Do you believe in their mission, their direction, their team?

While it is a common saying for us to “get to “no” fast if we aren’t a good fit.” I think the same goes for prospects. IBM was the biggest so the saying in the 80s was “nobody ever got fired for buying IBM hardware” even though it cost a premium. Yet that also wasn’t the key to great success. Dell seemed invincible. And right now the only PC company I see that I think “cares” about what I do, that I believe every employee shares the vision with, is Apple. And Steve isn’t there anymore. Yet the vision remains.

Purchase the proprietary market leader for your NPO if you are there for the paycheck and not the vision. And while the message might be self serving given Tendenci was started by our company, it is worth noting that the White House is now powered by Drupal. They believe that the company that powers Drupal, and that Dries himself, believes in the importance of open government from top to bottom. I believe it too. It really is software as a mission as

pycrypto install fails on mac osx mountain lion in virtualenv

Receiving the following errors trying to install pycrypto in a virtualenv on mac osx mountain lion. Assuming the name of the virtualenv is “PROJECTVIRTUALENV” and that your project is PROJECT my first try was:

[powershell]
$ mkvirtualenv PROJECTVIRTUALENV
……… change into the project folder
$ pip install -r requirements.txt
……… everything installs except pycrypto
$ sudo pip install pycrypto
[/powershell]

that also fails resulting in:

[powershell]
File "/Users/eschipul/.virtualenv/PROJECTVIRTUALENV/build/pycrypto/setup.py", line 278, in run
raise RuntimeError("autoconf error")
RuntimeError: autoconf error

[/powershell]

This was a new one on me so what changed? A few things. First virtualenv and virtualenv wrapper now default the –no-site-packages, but you were using that already.

Second the upgrade from OSx Lion to OSx Mountain Lion required an update of the command line tools with xCode. I’ve tried the stand alone xCode tools and they don’t keep up with Macs updates so I have the whole bloody xCode installed on my machine. To resolve the install problem with pycrypto it needed the new compiler. Simply updating to xCode 4.4 does not solve the problem by itself.

Per a note half way down this blog post on similar installation troubles by jiaaro you need to re-download the command line tools. From the post:

You need to install Xcode 4.4 (from the app store) and then, within xcode open Xcode > Preferences (or press Cmd +,) then open the downloads tab and install the Command Line Tools.

which looks like this:

and results in this:

[powershell]
sudo pip install pycrypto
password:
…. yada yada yada …. a warning or two …. then
Successfully installed pycrypto
Cleaning up…
[/powershell]

Note that I *did* already have the command line tools installed on xCode 4.3. The upgrade didn’t match my previous config and I had to explicitly download them again. Hope this saves some other poor soul some time as it makes no sense to me. But there it is.

convert legacy django mysql dbs from myisam to innodb

The Problem and the Virtue of Transactions

Disclaimer: a long time programmer, old guy, learning new tricks. If I made a mistake please let me know in the comments and I’ll credit and update the post. Thanks!

Short version: For Django – MYISAM BAD. INNODB GOOD. Fight it if you wish, but for me, this was the bug and the fix. (Oh, and everything below relates to a Mac OS X 10.7.3 so translate to your OS.)

Django supports transactions. This is a good thing. And recommended. In my case I had a bug where something in admin delete worked on my local machine but did NOT work in staging. Tres embarrassant!

Specifically (for the now open source Tendenci software) I was struggling with building a Tendenci Django plugin with related objects (categories, etc) and attached images (files). Basic CRUD stuff. CRUD worked on my django dev environment locally, but only CRU worked and D failed in staging. Specifically Django failed on delete with a relationship error. Can’t delete three things at once. Why? Hmmmm. Stumped.

Thanks to debugging by JMO, he found the difference is our staging and production database servers have tables that are set to INNODB as the storage engine and my local mySQL defaulted database tables to MYISAM (MySQL 5.1). Thus trying to delete an object through the Django admin failed in staging because it could not delete three things at once in INNODB which strictly enforces relationship rules. OK, that seems fair. MYISAM is more willy-nilly-do-whatever. Which means you have scraps of relational data left in your tables. Yuck. Long-term-data-mining-hell.

Bottom line – transactions in Django are not supported at a database level with Django middleware with MYISAM and thus I couldn’t wrap a (multi-)delete into one call in admin.py. Sure we could programmatically delete the objects in sequence as a quick bug fix, Yet I’d rather let Django handle the heavy lifting (and shorter code). And long term there is a strong need for transactions. Especially in the age of RESTFUL APIs (I miss you soap! NOT!) In current web development with API calls dependencies can easily be on 30 routers and 20 servers just so you can post a pic of your chicks to Instagram Flickr. Thus you have to be able to roll back transactions. That is part of what a framework does. And transaction rollbacks clearly should NOT be the application layer’s job IMHO. MVC and all even if C is in V in Django. Another debate…. So transactions it is.

How to upgrade the dev mysql database environment from MYISAM to INNODB?

The fix that worked for me came from this post on converting a mysql database to INNODB for Drupal as well as several others credited below. While this post is on Django, Drupal still supports both MYISAM and INNODB, the default in Drupal 7 is now INNODB. So don’t fight the man. I’m moving to INNODB for Django too.

What steps are required for Django on MYISAM on a Mac OSX to get transactions working in Django? Combining a few of the posts (all linked below) and the django transactions help file you wind up with the following:

First convert your database. To do this create a temp directory and change (cd temp) into it using terminal. Run the commands from this post. I repeated them below slightly changed and I am skipping the prompt indicator to make it easier to copy.

First check if this is even a problem by getting a list of databases that have tables that use MYISAM

mysql -u USERNAME -p
#it will prompt for your password here
SELECT table_schema, table_name FROM INFORMATION_SCHEMA.TABLES WHERE engine = 'myisam';

You will get some results from “information_schema” and “mysql” and I’d recommend not changing those. Just look for your dev databases.

Second create the SQL file to change the offending databases that are your Django mysql databases.

cd temp
mysql -u USERNAME -p -e "SHOW TABLES IN YOURDATABASENAME;" | tail -n +2 | xargs -I '{}' echo "ALTER TABLE {} ENGINE=INNODB;" > alter_table.sql

It will prompt you for your password and replace the USERNAME and the YOURDATABASENAME with your own. Side note, don’t put a dash in your database name or it won’t work. I want those 30 minutes of my life back. Moving on….

I’m a curious guy so I wanted to see the contents of the sql file. (Replace “Sublime Text 2” with the text editor of your choice. I just use Sublime because Glen told me to).

open -a "Sublime Text 2" alter_table.sql

Ooooh. Aaaaah. Looks fine. OK, close that. From there I prefer to do the update one database at a time to be sure so this:

mysql -u USERNAME -p YOURDATABASENAME < alter_table.sql

Then to confirm run the myisam table query again (which we hope does NOT show our now converted DB.)

SELECT table_schema, table_name FROM INFORMATION_SCHEMA.TABLES WHERE engine = 'myisam';

All good? Cool.

For all I know INNODB might break another database so I am only focusing on your Django mysql databases. Now in your settings.py file in your Django project be sure you have the following line as part of your DATABASES setting.

'OPTIONS': {"init_command": "SET storage_engine=INNODB",}

The whole settings.py mysql connection setup now looks like this because I hate when people leave off the context of where to put code or the details (I’m looking at you StackOverFlow.) /rant/Actually the only thing I hate more are code examples that use sqllite because they are close to useless. /rant/

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'YOURDATABASENAME',
        'USER': 'USERNAME',
        'PASSWORD': 'BLAH',
        'HOST': 'BLAH',
        'PORT': 'BLAH',
        'OPTIONS': {"init_command": "SET storage_engine=INNODB",}
    }
}

Now when you run from your project directory the usual “python manage.py syncdb“ command it doesn’t use the database defaults regardless. Anything new should be automatically created INNODB.

The following posts are the entire basis of the content above. I just combined it all in one place specific to Django and MySQL on Mac OS X. Comments and corrections are welcome.

  1. https://docs.djangoproject.com/en/dev/topics/db/transactions/
  2. http://djangosaur.tumblr.com/post/357759467/django-transaction-mysql-engine-innodb
  3. http://www.electrictoolbox.com/find-innodb-tables-mysql/
  4. http://www.mikebernat.com/blog/MySQL_-_InnoDB_vs_MyISAM
  5. http://highervisibilitywebsites.com/convert-your-mysql-database-myisam-innodb-and-get-ready-drupal-7-same-time
  6. http://stackoverflow.com/questions/9947671/performance-difference-between-innodb-and-myisam-in-mysql
  7. http://tag1consulting.com/MySQL_Engines_MyISAM_vs_InnoDB

 

mind as water as programming

There is a natural conflict between youth and programming. Frequently the best programming ideas, like in mathematics or chess, come from young people. Sometimes under 30. And in my experience focus becomes harder as you age. Young people don’t know what they can’t do and this often allows them hit the ball out of the park! A very good thing indeed!

The contradiction is the young guns with the fresh mental horsepower have the least experience and frequently charge around randomly between the latest cool “philosophy” and “tool” making less progress than an older programmer who might quickly determine what to avoid. And youth can sometimes be unwilling to compromise the reality of programming economics (code actually runs on hardware I’m told) and the beauty of full normalization. (Hint: your college prof was wrong. Talk to an experienced dba and she will speak the truth. Normalization is a goal like reaching the axis on an exponential curve.)

Currently I am trying to catch up with my team on Python and Django (they have really left me in the dust at the moment) I enjoyed this quote from DRY:

My biggest programming headaches have always come from abstractly struggling with “how can I write a good general solution to this problem”, even though I only know of one place where it’s definitely going to be used. “I’d better think of a general solution now,” I think, “or I’ll have to copy-and-paste-and-change code!” But it’s absurd to try to come up with a general solution without knowing more about the different varieties of the problem that exist (or will exist) in the system.

It’s a battle of two really strong urges - OnceAndOnlyOnce vs avoiding Premature Generalization. Do I duplicate for now and try to live with the duplication for a while, or violate YagNi and come up with some half-cocked generalized solution? It’s a tough one, because almost all programmers hate duplication; it’s a sort of primordial programming urge.

However, even though CopyAndPasteProgramming can be expensive to clean up, so is a botched “general solution” – and copy and paste is far cheaper up front. So I also am in favour of temporary duplication, to be refactored when you have a clearer view of the situation. – MatthewBennett

In our shop I have always called it “do it once for all time for all users.” But it means the same thing. Repeating yourself and not solving issues at the root level is a waste of time. I can’t tell you how awesome it is to not have to explicitly declare getters and setters in Python (vs classic ASP where classes don’t even have inheritance.).

Yet there is a catch. At the beginning of a project or approaching a new industry you can’t really do “root cause analysis” because you don’t have any data. To requote the quote above –

However, even though CopyAndPasteProgramming can be expensive to clean up, so is a botched “general solution” – and copy and paste is far cheaper up front. So I also am in favour of temporary duplication, to be refactored when you have a clearer view of the situation. – MatthewBennett

– this balance is the solution and yes it is a bit messy. If the goal of programming is to make a profit then you can’t just argue philosophy without discussing the economics of it. Economics as in money. And economics as in sometimes duplicate code or duplication of data just runs faster on the servers, increases your search engine rankings and lowers your operating costs. I can live with a bit of duplication for that reward.

Often the best solution is to collect data and slam it together and then identify the best organizational structure before the corpus gets too large. Programming classes or actual data both. A filing system for terrabytes of photos and videos for example. It is a balance between the default import settings of all of the hardware you use and the need to compartmentalize source and work product for portability and backups. Yes, you have to compromise to the machines somewhat because you can’t change the settings on every other camera in the world.

Let the data build up a bit. Don’t fight the hardware. Live with some duplication. Then generalize and normalize as best you can. Balance. “Mind as Water” as Lee would say.