Eclipse and Python and Django on Mac OSx 10.8.1

New laptop so I figured I might as well double check that I was using the latest packages and development tools. i was very very close, but not quite. So I went for a fresh install of Eclipse and PyDev for Django development.

Taking the lead from this blog post:
http://garmoncheg.blogspot.com/2011/06/django-better-way-to-install-eclipse.html

Installed the 64 bit version of Eclipse for Javq Developers (yes it’s Python, but Eclipse was originally designed for Java. It’s OK, you won’t have to type any squiggly things in your Python code….)
http://www.eclipse.org/downloads/

automatic termination – thanks apple

Mac OSx Mountain Lion (10.8) has a new “feature” called auto-termination. Even the sound of it is ominous. Auto termination. This means you.

Here’s the thing Apple. I’ve been programming a lot lately. I also do photography. And build robots. And program using two or three different apps at a time. And I actually read about politics instead of listening to talk radio. And I check email. Sometimes go find weird music. I do these things on this here little Mac. Typically I have six desktops open which then become 12 desktops when I get to work and hook it up to that monitor I paid y’all a bunch to purchase.

Then I upgrade to Mac OSx Lion. Released the day before DefCon guaranteeing I could not use the laptop I had with me (not this one) because you couldn’t download the upgrade and security fixes. And the updates to all of the apps. And XCode 4.3 and now 4.4 with the Command Line Tools. Because you can’t download the command line tools alone anymore.

I leave Lightroom open if I am working on a project. It can stay open for days as that isn’t my real job. It’s just something I chew on like a bone to keep my sanity. But I want to know when I leave the bone one desktop 6 that it will be there when I get back.

Or, I used to do all of that. I did buy extra RAM after all. Now, I am auto terminated. I close the lid on my laptop and it’s like a mystery game when I get back. Hey, guess which one (1) app we decided to leave running for you!? Usually none.

So this is me venting. Stop with the auto terminating if there are any open windows anywhere on any desktop. No open windows I get (like Preview) but if Sublime is open with three files THERE IS A REASON. Leave me be.

why are programmers arrogant? perception or warranted?

Running a company of 30+ employees with account executives, designers, sales people, support, operations, programmers, execs, etc, it is very apparent that every role is different. Even if the same person moves from one role to another, their attitude quickly can be morphed by their new role.

Every group has it’s stereotypical achilles heal, but today I’d like to talk about the perception of the “arrogant programmer.”  To wit; how about a readme file that starts:

  • edit your bash file to alias /your/project/file/path to the download
  • add the package to your path.

Baroo? Bash file? Path? Which friggin’ path? I’ll get back to that. (But don’t worry, the answer is definitely “no” that doesn’t make sense to an experienced programmer either, except at a high level.  More on that at the end.)

Given I started the company as a programmer who learned to sell, as opposed to a salesman who learned to code or a designer who learned to code, I suspect I have a few swords to fall on myself in the process. Yet first let us consider what others say about the perception of programmers being arrogant, earned or unearned, as a whole.

One poster on Yahoo Answers asks

Why are so many programmers arrogant?

I seriously don’t get it. I am a programmer myself, but whenever I’m having an issue with my code and need help and enter an IRC help channel, everyone is so rude and just acts like if I don’t know the answer to my question already, I’m stupid… the people are always so mean and aren’t there to help but rather to show off their alleged knowledge.

And a few excerpts from the answers:

It seems that many programmers have a hard time with interpersonal skills… Because their lives lack a significant amount of human social interaction [something that makes our lives pleasurable], they derive their pleasure by feeling superior over others

And…

There is a certain elitism amongst programmers, it’s probably knowing that they understand something a lot of other people don’t.

OK, Yahoo answers isn’t a great source for anything. So let’s consider this more reasoned post by Jay Fields:

At SpeakerConf 2009, one of the speaker’s wives asked me: Why is it that most speakers are confident, and some are simply arrogant. The question was also in the context of “software is nerdy, how can you speakers be so cocky”. I pondered for a minute and replied: When no one knows what’s “correct”, people with confidence generally win the discussion….

As I’ve said before, we’re still figuring this stuff out. I constantly try to improve my processes for delivering software. I share those processes via speaking and blogging. However, I’m definitely aware that I could be doing it entirely wrong.

Emphasis added. And to take it a step further, consider the three modes a programmer goes through by John Byrd.

All programmers of any competence go through three modes of professional development: an Arrogant mode, a Sponge mode, and a Mentor mode.

All good game programmers are born into Arrogant mode. An Arrogant programmer believes that he is personally capable of writing the smallest, the fastest, the “best” code possible, and that other programmers don’t have the depth of understanding that an Arrogant programmer does. This extreme self-confidence is the fundamental motivator for the Arrogant programmer to produce high-quality results.

Arrogant programmers maintain and defend control over all aspects of their code.

…..

Given enough time and experience, an Arrogant mode programmer will grow out of this phase of professional development and turn into a Sponge programmer.

Sponge programmers believe that other programmers exist who have a deeper understanding of the code than they do, and Sponge programmers are committed to learning from these more experienced programmers.

Wow, that’s depressing. So any good programmer has to start out being arrogant before they can approach self actualization and… oh, I don’t know…. humility? Major bummer.

And I am *sure* that I was that arrogant programmer. I can think back on specific interchanges and realize that “yup, you were an ass that day lording your almighty knowledge over everyone to win the argument.” And how many UI suggestions did I miss? How many actual client requests that might have made the company many more millions were not mentioned because of the perception of arrogance? I don’t and will not ever know my losses from arrogance. Hindsight is so 20/20. And humility is so profitable.

Back to our not-so-victimless-crime-of-arrogance.  The readme file that starts:

  • edit your bash file to alias /your/project/file/path to the download
  • add the package to your path.

Why are those two lines useless? Because they do not compile. They completely lacks exactness. They waste time. It is pseudo code at best and assumes you have time to waste that the writer does not to give you the details. They want you to fail or prove yourself like it’s Lord of the Flies or something.

Any programmer will tell you we are too lazy to remember the syntax to do those edits.  I know I am.  They say Einstein didn’t know the value of pi  because he had it written down and could look it up. As programmers look stuff up when faced with poor documentation. Logic is what programming is about. Languages and operating systems, on the other hand, are about syntax. Humans add value with higher level logic that machines are incapable of.

Thus glazing over the details of syntax, taking time away from someone else’s higher level thinking, is motivated, in my opinion primarily (thankfully) by laziness.  Not arrogance. The truth is we can’t remember the syntax to edit your bash file and don’t want to document the variables so we glaze over it. We skip over it bluffing, pretending “you should know” when in fact we aren’t linking it because we can’t remember ourselves!

That my friends, if it is intended as arrogant or not, clearly comes across as arrogant. No two ways about it.

But to suggest that a real programmer has the method of doing so memorized would suggest that only a stupid programmer would create a short-cut in their bash file called “editbash” that opens the file in their editor of choice. My point? A real programmer doesn’t memorize that crap. Therefore, if we don’t have it memorized, the documentation isn’t really anything more than a fishing assignment that wastes someone’s time  unnecessarily. #lame

Oblique assumed knowledge to have someone “prove themselves” is nothing more than a hurdle. A mental “pay your dues or you can’t use my code” type of statement. Or possibly just lazy documentation – which when I see it internally is my hope. Arrogant people are terrible security risks because they are so easy to manipulate. Thus I prefer to think of it as lazy as opposed to defensive when I observe it within my team. Yet the external perception, regardless of motive, is still unfortunate. Working on that.

The bottom line is arrogance, be it real or perceived, sucks. If you  really  want people to use your code, then you don’t let your documentation’s excuse for brevity hide behind arrogance or  laziness.

a user is a user is a user – database design fundamentals for marketers and programmers

Even in the mainframe days we had the golden rule that every user, person, student, teacher, whatever, on the system had a record in one (1) location.

Sure there might be descriptive data in another location. It might be a central table as simple as userid-username-password-role with a one to one relationship to a “role” table that then linked to the appropriate table for that role. Kung Fu Master versus Adjunct Professor. You get the idea. In Django this is the auth_users table with a relationship to the profiles table. But it is sooooo tempting to ignore the intention of the designers of Django, to break away from their DRY principles, and come up with a different system to reduce the number of records in auth_users. Despite lessons learned from Discus, the world’s largest Django application.

I followed that law, that a user is a user is a user, and wish I could credit it to the correct professor from my mainframe days on VAX using FORTRAN 4 and then FORTRAN 77.  I was taught as early as the 80s that there was always one (1) place for humans (in a database at least). When Microsoft Access came out and everyone became an amateur DBA they suddenly created tables for “students” and “teachers” and “staff” which required duplication of data all over the place. It became so brittle that if another programmer worked on your file they had to know every field in every table. In other words – it was pathetic. I swore I would never do that again.

But I did forget. My fault. You can’t blame the young guns with two or three years of experience as they haven’t suffered the “mile in shoes over broken glass” of permissions for humans in disparate tables. Which might work, until you hire the next programmer and they go “what the F were you people thinking….?” This one is all on me.

One user. One record….. – so simple and yet so profound. It is the singularity in database design. No exceptions

That rule, “one user, one record” served Schipul.com and our clients well recurring revenue and a nice margin with true database integrity for over 10 years. We all made a profit. And the software? “It just worked.”

As a public post so I don’t do something this stupid again in the future, I am posting these two pages from the book  Direct and Database Marketing by Graeme McCorkell. I used it with Tendenci 1,2,3,4.x+ and it served us well for 10 years. And then I forgot. Uuuuuugh. Double+plus+dumb points for me. And unfortunately this is hurting our clients through delays which is what really kills me.

One user. One record.

A user is a user is a user.

Here is to Tendenci 5.1 being pushed out in a few weeks. Major strides in the right direction. Back towards DRY and simplicity. Simplicity is best.

pycrypto install fails on mac osx mountain lion in virtualenv

Receiving the following errors trying to install pycrypto in a virtualenv on mac osx mountain lion. Assuming the name of the virtualenv is “PROJECTVIRTUALENV” and that your project is PROJECT my first try was:

[powershell]
$ mkvirtualenv PROJECTVIRTUALENV
……… change into the project folder
$ pip install -r requirements.txt
……… everything installs except pycrypto
$ sudo pip install pycrypto
[/powershell]

that also fails resulting in:

[powershell]
File "/Users/eschipul/.virtualenv/PROJECTVIRTUALENV/build/pycrypto/setup.py", line 278, in run
raise RuntimeError("autoconf error")
RuntimeError: autoconf error

[/powershell]

This was a new one on me so what changed? A few things. First virtualenv and virtualenv wrapper now default the –no-site-packages, but you were using that already.

Second the upgrade from OSx Lion to OSx Mountain Lion required an update of the command line tools with xCode. I’ve tried the stand alone xCode tools and they don’t keep up with Macs updates so I have the whole bloody xCode installed on my machine. To resolve the install problem with pycrypto it needed the new compiler. Simply updating to xCode 4.4 does not solve the problem by itself.

Per a note half way down this blog post on similar installation troubles  by jiaaro you need to re-download the command line tools. From the post:

You need to install Xcode 4.4 (from the app store) and then, within xcode open  Xcode > Preferences  (or press  Cmd  +,) then open the  downloads  tab and install the  Command Line Tools.

which looks like this:

and results in this:

[powershell]
sudo pip install pycrypto
password:
…. yada yada yada …. a warning or two …. then
Successfully installed pycrypto
Cleaning up…
[/powershell]

Note that I *did* already have the command line tools installed on xCode 4.3. The upgrade didn’t match my previous config and I had to explicitly download them again. Hope this saves some other poor soul some time as it makes no sense to me. But there it is.

never fathomed the idea that we can use git to send runnable code to the cloud

…we associate git only as a tool for managing source code (some of us only use git in conjunction with github, and have never fathomed the idea that we can use git to send runnable code to the cloud).

So, I was forced to decouple my notion of “git” as a version control tool from “git” as a generic source code commisioning utility, when I decided I wanted to experiement with deploying a blog … I quickly found that I didn’t fully understand heroku’s model.

jayunit100 (source)

Yup. Me neither. Great blog post if you are interested in experimenting with Heroku.

RoR vs Django – just one opinion but worth a read

“From reading up on both frameworks, there is a subtle, but noticeable, difference between their philosophies. Rails focuses on having the framework do stuff in the background for the developer, making it all seem simple and easy. The Django framework does just as much heavy lifting for you, but emphasizes on the fact that it is saving you from having to do these things yourself. These two ideas seem to be the same, but the Rails point of view has more of a “magic“ visage whereas Django has more of a convenience one. I personally prefer Django’s approach to this, but I’m sure there are plenty of people out there who would disagree.”

–  Huan Lai – CTC Labs

Regarding Django best practices, a truly elusive subject with much disagreement in the Django community (just my opinion), I found this suggestion of a more modular approach (delete settings.py! use a folder! specify environments!) compelling. The local_settings.py thing and gitignore bugs me. But I guess a lot of things bug me. Anyway, this one is also worth a read.

And I just learned from this post about Django Rosetta. Learning. Slowly. Uuuuuugh.

 

sometimes you learn more from the comments than the article

I was reading the article “Solving the Mystery of Black Holes” on CNN by Meg Urry. The subject is satellites and x-rays and black holes and stuff: Or as her subject paragraph states much better than I can:

Many years in the making,  NuSTAR  carries an important scientific instrument designed to look for energetic X-rays from cosmic sources like black holes and exploded stars.

Most of us know about X-rays used for diagnostic imaging of broken limbs or for security scans at the airport. They are a high-energy form of light, energetic enough to penetrate clothing or flesh.

I barely understand the concept of black holes and if they want to emit X-rays then, from what I have seen on Star Trek, they can pretty well do whatever they want because even Captain Kirk avoided them. So you gotta figure black holes are the bad-asses of terrestrial objects. (Or are they anti-objects? Makes my brain hurt.) Anyway, I get lost about half way through the article so I scan down.

At the bottom I scan the comments quickly and see the usual trolls who somehow mistook CNN for youTube. But then I read one of the best summaries of what the concept of “science” is about. That it is iterative. It isn’t absolute until proven and replicated. But that forming a model is a necessary part of the process. Here is a section of the comments from Glorifundel:

That is how science works, we take what we currently know and create models that describe that reality.  We then vigorously test those models, and adjust them to fit those reproducible results as needed.  We can then use those models to make new hypothesis about the world around us, and test those new scenarios to get an even more robust picture of our world.  The end goal being to know as much as we can about this place we all live in.

I’m not qualified to jump in on their debate about the mystery of black holes. But I recognize exactness of speech and Glorifundel nailed it. Observe, create a model, make a prediction, test it, adjust, repeat, learn.

Not that different from software development if you think about it. It sounds very agile to me.

“Blame” doesn’t replace “Pride of Craftsmanship”

I have been using Tower as my Git client lately, as trained by my programming team. I’m the old dog this time learning syntax and style guides from them (and hopefully they are listening and learning design patterns from me). While reviewing the Tendenci code recently one question that I asked was “Where are the comments? And when someone does comment, why do they not put their initials at the end?”

To this the response was:

“We don’t need to put our initials on things. Version control takes care of that and we can use  the “blame” function to find out who did what on any given line.”

I said OK at the time. But it has been bugging me. I couldn’t quite put my finger on why it bothered me, but it did. Who can argue against accountability? I see the business value in assigning blame and using that to create a teachable moment.

Having programmed a large number of products over the years (MS Access anyone? VB 4? Ahhhh, good times.) I was taught that software should be “self healing.” The end user should NEVER see an error message. If the user sees an error, the programmer didn’t do their job or they were just sloppy.  There is a reason Select Case statements end in Else. There is a reason If statements have both an Else and an ElseIf option. There is a reason you trap errors.

Error trapping is not a science it is an art. And again, programming just isn’t for everyone like any other trade. But if you are gonna do it, then do it right. Less really is more in programming. I love the DRY principle in Python.  Our process was one of complete accountability. You, the programmer, tested it first and when you took it live, there was no third party tester to fall back on. There was no tester to blame. No Linus security blanket. For years when developing Tendenci we didn’t even have a staging environment. When you play with live ammo you learn to pay attention. High stress? Sure. But much more profitable than cascading code through four different environments before it goes live.

Back to the blame game. I was taught starting in my first programming class in 1983 that putting your name in the comments, or initials as the case may be, was a way of saying

“I wrote this. I am proud of this code. There are no bugs in it. And I DEFY you to find a bug.”

This goes back to Basic, Fortran, Pascal and my mainframe days. The only time I didn’t use comments with my initials was when programming in Assembler and I think I should get a pass on that one, as hellish as it is an all.

Initials and comments were so serious that my professor my Freshman year of college would mock us saying that if we didn’t put our name in the comments he knew it was going to be C or B code at best, but definitely not A+ code. A+ coders are proud of simple clean code and they want credit for it. They comment it and put their name on it. Now to be fair, this particular prof did mark you down a letter grade if you didn’t comment your code so his statement was somewhat of a self fulfilling prophecy.

Pride. Ownership. Creators. These are very positive things.  And that is why we put our initials in the comments on functions and classes we wrote. It was a way of saying “I am proud of this code.”

And that is a very different attitude than the source control “BLAME” function which is used mostly to say “you f’d this up.”

Blame and pride-of-craftsmanship are two very different things. That is why it bothers me so much I think. Maybe the world has changed and programming is taught like dodgeball – the less you do and the more you hide the less you might get blamed for something? I really don’t know.

I do know that I do not like the negative connotations of “blame.” Where is the joy if you can be blamed for stuff but never credited? Yes I like having the “blame” functionality. It is just a bummer to see a craft go from a positive to a negative feedback loop. I’m still thinking about that. I do know craftsmen and artists should be proud of their work. And I believe my programming team is proud of their work. I just don’t understand how we got from pride to blamesmanship-only when we could have both.

I know the pride aspect isn’t totally gone because GitHub prominently shows how many commits you have made and which projects you work on. Yet quantity is not quality, and clean programming can be subtly claimed with initials to indicate pride of ownership without relying solely on “blame.” It doesn’t all have to be negative. Programming is about creating. And creating things gives us meaning. Which is beautiful.

Like I said, I’m still thinking about this one. I’m probably missing something. Feel free to blame me if I got it wrong.

In Python, does “favor composition over inheritance” – really apply at all?

Googling best practices for classes in Python for a Django project I kept coming across the phrase “favor composition over inheritance” and the  Strategy Design Pattern. First let me quote wikipedia on composition over inheritance:

Composition over inheritance  in  object-oriented programming  is a technique by which classes may achieve polymorphic behavior and code reuse by  containing other classes  which implement the desired functionality instead of through  inheritance.[1]

The Strategy Design Pattern is idea is to decouple  as much as possible to maximize code use. It looks kinda like this code from S-Lott on  Stack Overflow on Strategy:

class AUsefulThing( object ):
    def __init__( self, aStrategicAlternative ):
        self.howToDoX = aStrategicAlternative
    def doX( self, someArg ):
        self. howToDoX.theAPImethod( someArg, self )

class StrategicAlternative( object ):
    pass

class AlternativeOne( StrategicAlternative ):
    def theAPIMethod( self, someArg, theUsefulThing ):
        pass # an implementation

class AlternativeTwo( StrategicAlternative ):
    def theAPImethod( self, someArg, theUsefulThing ):
        pass # another implementation

Now you can do things like this.

t = AUsefulThing( AlternativeOne() )
t.doX( arg )

S-Lott then provides another Python example pattern that replicates the above with a bit less code.

This had me scratching my head. Not at the example and the helpful response, but at the question itself. Why are we creating functions, then assigning a variable to a class and passing the function in as a parameter? Barooo?

Then on the same stack overflow post there was a follow up response from Christian Groleau  as follows:

This pattern is practically non-existent in languages that support first class functions. You may want to consider taking advantage of this feature in Python:

def strategy_add(a, b):
    return a + b

def strategy_minus(a, b):
    return a - b

solver = strategy_add
print solver(1, 2)
solver = strategy_minus
print solver(2, 1)

“This approach is very clean and simple.”

I agree with him. I have made the mistake, granted it was out of desperation, of putting a literal in a public property in a class called “objecttype”. This let me pass the object to a function and make decisions on what to do next. Pseudo vbscript-ish code example:

function generate_invoice(object, d)
    select case(object.objecttype)
        case "jobs"
        #do job invoice stuff
    case "resumes"
        #do resume stuff here
    case  else

Ok, so it’s not quite as bad as concatenating a string of vbscript and calling the exec(str) method. But it ain’t pretty either. But both got the job done. And in the case of my fake function above, that sure looks like the composition pattern to me. Maybe not exactly, but it’s close. Functions taking classes as parameters and then calling methods within the class in the name of portability. And to reduce code in the classes themselves of course.

I like Christian’s example above a LOT better. Which makes sense because Python has, as noted by bitwise, both First Class Functions and allows for Meta-programming. “Python is not java without the compile” indeed. First Class Functions and Meta Programming used  in combination obviate the need for the Strategy design pattern in Python IMHO (see caveats below). And for further support I found this:

From pycon 2009 – Joe Gregorio – The (lack of) design patterns in Python

“Yes, design patterns are good, but they’re also a sign of weakness in a language.“

The patterns are built in [to Python].  “No one talks about the “˜structured programming’ pattern, or the “˜object-oriented’ pattern, or the “˜function’ pattern anymore.“

………

Conclusions:

  1. look to features before patterns
  2. reduce patterns -> shorter code
  3. needed patterns -> language feature

I’m still a Python newb, but definitely not a programming newb. And So  far it looks like you can inherit meta classes with properties to your hearts content to DRY. Yet still, make a thoughtful decision when it comes to how you should implement methods/functions in base classes. Sometimes perhaps a function is better than inheriting a method. Just stop and think. Odds are our fear of inheritance is a learned behavior from being burned by java or another similar lower-level language. I think I need to get over that.

And be thankful that most design patterns are already built into Python. Less code is better.

In closing, and I forget the post I read it on, but just because programming can do ORM (object relational mapping) does NOT mean that a database can. In fact it can’t. And a database deals with the multi-stand-alone class issue by creating a bunch of tables in Django. And one programming matra that I am 100% sure is true is “JOINS are EXPENSIVE.”

Django responds to inherited meta classes by keeping all of the fields in one table, not 20. And perhaps this is me just trying to get over my fear of multi-inheritance and break years of practice. To take advantage of Python being “First Class Everything”. To write code that is  Better, faster, stronger than before.

convert legacy django mysql dbs from myisam to innodb

The Problem and the Virtue of Transactions

Disclaimer: a long time programmer, old guy, learning new tricks. If I made a mistake please let me know in the comments and I’ll credit and update the post. Thanks!

Short version: For Django – MYISAM BAD. INNODB GOOD. Fight it if you wish, but for me, this was the bug and the fix. (Oh, and everything below relates to a Mac OS X 10.7.3 so translate to your OS.)

Django supports transactions. This is a good thing. And recommended. In my case I had a bug where something in admin delete worked on my local machine but did NOT work in staging. Tres embarrassant!

Specifically (for the now open source Tendenci software) I was struggling with building a Tendenci Django plugin with related objects (categories, etc) and attached images (files). Basic CRUD stuff. CRUD worked on my django dev environment locally, but only CRU worked and D failed in staging. Specifically Django failed on delete with a relationship error. Can’t delete three things at once. Why? Hmmmm. Stumped.

Thanks to debugging by JMO, he found the difference is our staging and production database servers have tables that are set to INNODB as the storage engine and my local mySQL defaulted database tables to MYISAM (MySQL 5.1). Thus trying to delete an object through the Django admin failed in staging because it could not delete three things at once in INNODB which strictly enforces relationship rules. OK, that seems fair. MYISAM is more willy-nilly-do-whatever. Which means you have scraps of relational data left in your tables. Yuck. Long-term-data-mining-hell.

Bottom line – transactions in Django are not supported at a database level with Django middleware with MYISAM and thus I couldn’t wrap a (multi-)delete into one call in admin.py. Sure we could programmatically delete the objects in sequence as a quick bug fix, Yet I’d rather let Django handle the heavy lifting (and shorter code). And long term there is a strong need for transactions. Especially in the age of RESTFUL APIs  (I miss you soap! NOT!) In current web development with API calls dependencies can easily be on 30 routers and 20 servers just so you can post a pic of your chicks to Instagram  Flickr. Thus you have to be able to roll back transactions. That is part of what a framework does. And transaction rollbacks clearly should NOT be the application layer’s job IMHO. MVC and all even if C is in V in Django. Another debate…. So transactions it is.

How to upgrade the dev mysql database environment from MYISAM to INNODB?

The fix that worked for me came from this post on converting a mysql database to INNODB for Drupal  as well as several others credited below. While this post is on Django, Drupal still supports both MYISAM and INNODB, the default in Drupal 7 is now INNODB. So don’t fight the man. I’m moving to INNODB for Django too.

What steps are required for Django on MYISAM on a Mac OSX to get transactions working in Django? Combining a few of the posts (all linked below) and the django transactions help file  you wind up with the following:

First convert your database. To do this create a temp directory and change (cd temp) into it using terminal. Run the commands from this post. I repeated them below slightly changed and I am skipping the prompt indicator to make it easier to copy.

First check if this is even a problem by getting a list of databases that have tables that use MYISAM

mysql -u USERNAME -p
#it will prompt for your password here
SELECT table_schema, table_name FROM INFORMATION_SCHEMA.TABLES WHERE engine = 'myisam';

You will get some results from “information_schema” and “mysql” and I’d recommend not changing those. Just look for your dev databases.

Second create the SQL file to change the offending databases that are your Django mysql databases.

cd temp
mysql -u USERNAME -p -e "SHOW TABLES IN YOURDATABASENAME;" | tail -n +2 | xargs -I '{}' echo "ALTER TABLE {} ENGINE=INNODB;" > alter_table.sql

It will prompt you for your password and replace the USERNAME and the YOURDATABASENAME with your own. Side note, don’t put a dash in your database name or it won’t work. I want those 30 minutes of my life back. Moving on….

I’m a curious guy so I wanted to see the contents of the sql file. (Replace “Sublime Text 2” with the text editor of your choice. I just use Sublime because Glen told me to).

open -a "Sublime Text 2" alter_table.sql

Ooooh. Aaaaah. Looks fine. OK, close that. From there I prefer to do the update one database at a time to be sure so this:

mysql -u USERNAME -p YOURDATABASENAME < alter_table.sql

Then to confirm run the myisam table query again (which we hope does NOT show our now converted DB.)

SELECT table_schema, table_name FROM INFORMATION_SCHEMA.TABLES WHERE engine = 'myisam';

All good? Cool.

For all I know INNODB might break another database so I am only focusing on your Django mysql databases. Now in your settings.py file in your Django project be sure you have the following line as part of your DATABASES setting.

'OPTIONS': {"init_command": "SET storage_engine=INNODB",}

The whole settings.py mysql connection setup now looks like this because I hate when people leave off the context of where to put code or the details (I’m looking at you StackOverFlow.) /rant/Actually the only thing I hate more are code examples that use sqllite because they are close to useless. /rant/

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'YOURDATABASENAME',
        'USER': 'USERNAME',
        'PASSWORD': 'BLAH',
        'HOST': 'BLAH',
        'PORT': 'BLAH',
        'OPTIONS': {"init_command": "SET storage_engine=INNODB",}
    }
}

Now when you run from your project directory the usual “python manage.py  syncdb”  command  it doesn’t use the database defaults regardless. Anything new should be automatically created INNODB.

The following posts are the entire basis of the content above. I just combined it all in one place specific to Django and MySQL on Mac OS X. Comments and corrections are welcome.

  1. https://docs.djangoproject.com/en/dev/topics/db/transactions/
  2. http://djangosaur.tumblr.com/post/357759467/django-transaction-mysql-engine-innodb
  3. http://www.electrictoolbox.com/find-innodb-tables-mysql/
  4. http://www.mikebernat.com/blog/MySQL_-_InnoDB_vs_MyISAM
  5. http://highervisibilitywebsites.com/convert-your-mysql-database-myisam-innodb-and-get-ready-drupal-7-same-time
  6. http://stackoverflow.com/questions/9947671/performance-difference-between-innodb-and-myisam-in-mysql
  7. http://tag1consulting.com/MySQL_Engines_MyISAM_vs_InnoDB

 

mind as water as programming

There is a natural conflict between youth and programming. Frequently the best programming ideas, like in mathematics or chess, come from young people. Sometimes under 30. And in my experience focus becomes harder as you age. Young people don’t know what they can’t do and this often allows them hit the ball out of the park! A very good thing indeed!

The contradiction is the young guns with the fresh mental horsepower have the least experience and frequently charge around randomly between the latest cool “philosophy” and “tool” making less progress than an older programmer who might quickly determine what to avoid. And youth can sometimes be unwilling to compromise the reality of programming economics (code actually runs on hardware I’m told) and the beauty of full normalization. (Hint: your college prof was wrong. Talk to an experienced dba and she will speak the truth. Normalization is a goal like reaching the axis on an exponential curve.)

Currently I am trying to catch up with my team on Python and Django (they have really left me in the dust at the moment) I enjoyed this quote from DRY:

My biggest programming headaches have always come from abstractly struggling with “how can I write a good general solution to this problem”, even though I only know of one place where it’s definitely going to be used. “I’d better think of a general solution now,” I think, “or I’ll have to copy-and-paste-and-change code!” But it’s absurd to try to come up with a general solution without knowing more about the different varieties of the problem that exist (or will exist) in the system.

It’s a battle of two really strong urges –  OnceAndOnlyOnce  vs avoiding  Premature Generalization. Do I duplicate for now and try to live with the duplication for a while, or violate  YagNi  and come up with some half-cocked generalized solution? It’s a tough one, because almost  all  programmers hate duplication; it’s a sort of primordial programming urge.

However, even though  CopyAndPasteProgramming  can be expensive to clean up, so is a botched “general solution” – and copy and paste is  far  cheaper up front. So I also am in favour of  temporary  duplication, to be refactored when you have a clearer view of the situation. —  MatthewBennett

In our shop I have always called it “do it once for all time for all users.” But it means the same thing. Repeating yourself and not solving issues at the root level is a waste of time. I can’t tell you how awesome it is to not have to explicitly declare getters and setters in Python (vs classic ASP where classes don’t even have inheritance.).

Yet there is a catch. At the beginning of a project or approaching a new industry you can’t really do “root cause analysis” because you don’t have any data. To requote the quote above –

However, even though  CopyAndPasteProgramming  can be expensive to clean up, so is a botched “general solution” – and copy and paste is  far  cheaper up front. So I also am in favour of  temporary  duplication, to be refactored when you have a clearer view of the situation. —  MatthewBennett

– this balance is the solution and yes it is a bit messy. If the goal of programming is to make a profit then you can’t just argue philosophy without discussing the economics of it. Economics as in money. And economics as in sometimes duplicate code or duplication of data just runs faster on the servers, increases your search engine rankings and lowers your operating costs. I can live with a bit of duplication for that reward.

Often the best solution is to collect data and slam it together and then identify the best organizational structure before the corpus gets too large. Programming classes or actual data both. A filing system for terrabytes of photos and videos for example. It is a balance between the default import settings of all of the hardware you use and the need to compartmentalize source and work product for portability and backups. Yes, you have to compromise to the machines somewhat because you can’t change the settings on every other camera in the world.

Let the data build up a bit. Don’t fight the hardware. Live with some duplication. Then generalize and normalize as best you can. Balance. “Mind as Water” as Lee would say.

p = [“groups”, “pairs”, “solitary”] cr/lf p.append(“common sense”)

Pair Programming Seen as Harmful?  

One of the more  extreme  calls for all development to be performed by pairs of programmers: two coders at one keyboard, at all times, with almost no exceptions.

and

“Research strongly suggests that people are more creative when they enjoy privacy and freedom from interruption “¦ What distinguished programmers at the top-performing companies wasn’t greater experience or better pay. It was how much privacy, personal workspace and freedom from interruption they enjoyed,“ says a  New York Times  article  castigating “the new groupthink.“

and

Working alone is good for creativity “” but being paired with someone who thinks differently from you can lead to more creativity yet. So should your takeaway here be that pairing is the answer after all?

No, no,  hell  no. The true answer is that there is no one answer; that what works best is a dynamic combination of solitary, pair, and group work, depending on the context, using your best judgement.

Judgement. Use it. As for me I work best alone. But I benefit from a pre-build conversation and post-build code review. Be it a physical bot or a programming method. I need other people’s brains to rise to the next level. But I need them to leave me completely alone to get my sh*t done in-between.

(via  Steve Purkiss  from  the Drupal Association)

those who remain determined “mouse-avores“ starve

“IV. At the same time, living systems adapt themselves to changes in their environment they learn, grow, develop, evolve. When the mouse population in a region suddenly declines because of an epidemic, the predators who adapt to a new prey survive; those who remain determined snake bit“mouse-avores“ starve. Life events affect us and change us, and we can see these changes reflected in the nevertheless familiar faces of our friends. The ability of living systems to adapt and self-organize allows them to defy the second law of thermodynamics, which insists that everything runs down and returns to a state of disorganization and homogeneity. Not so for living systems! They continuously reorganize themselves into ever more complex patterns and interrelationships.”

Molly Young Brown, Patterns, Flows and Interrelationships

I Just Need a Website, That’s All I Need

caller: I was calling about a web site

developer: cool, that’s what we do. how can I help?

caller: I have a simple site and I need some updates. not much really, just a few changes. is that something you do?

developer: yes, what is the url?

caller: well I’ll need you to sign an NDA before I give you the url. can I fax it over?

developer: no (thinks: “do I still have a fax to email gateway working? hmmm”)

caller: what?

developer: we just met and you want to fist bump attorneys?

caller: no, i just don’t know how else to protect my intellectual property!

developer: you have a site now, right?

caller: yes

developer: live on the internet?

caller: yes

developer: …… long pause…..

caller: ok, I see your point.

…. 45 minutes of spec requests and contract pre-negotiation convo takes place here….

caller: so basically that’s it. my brother-in-law said he could build it for $225 dollars but I wanted to call around and get a few options to see if I could reduce the cost. He’s not very good actually.

Thoughts going through your head as the dev:

  1. developer (option 1): so you have a job board and you want to enhance a few features of monster.com to allow for a commission and affiliate structure?
  2. developer (option 2): so you have a great e-commerce idea, have been reviewing amazon.com and found a few ways to improve on their theories to sell widgets?
  3. developer (opti0n 3): so you want to have a self sustaining site that makes you money with no effort invested while you work at your current job realizing the money-for-nothing potential advertised on TV?

Resolution: have you heard of Crowdspring?

#respect #withAllDueRespect

the rules driving python

  1. Borrow ideas from elsewhere whenever it makes sense.
  2. “Things should be as simple as possible, but no simpler.” (Einstein)
  3. Do one thing well (the ‘UNIX philosophy’). And that was to create a great programming language that can be used anywhere.
  4. Don’t fret too much about performance – plan to optimise later when needed.
  5. Don’t fight the environment and go with the flow.
  6. Don’t try for perfection because ‘good enough’ is often just that.
  7. (Hence) it’s okay to cut corners sometimes, especially if you can do it right later.
  8. The Python implementation should not be tied to a particular platform. It’s okay if some functionality is not always available, but the core should work everywhere.
  9. Don’t bother users with details that the machine can handle.
  10. A large complex system should have multiple levels of extensibility. This maximizes the opportunities for users, sophisticated or not, to help themselves.
  11. Errors should not be fatal. That is, user code should be able to recover from error conditions as long as the virtual machine is still functional. At the same time, errors should not pass  silently.
  12. A bug in the user’s Python code should never be allowed to lead to undefined behavior of the Python interpreter; a core dump is never the user’s fault.

LinuxUser, Python, the Universal Programming Language paraphrasing Guido van Rossum