The prototype and first beta version of my startup was built entirely on Rails. The framework enabled us to build our site significantly faster than we would have been able to without it. For that, I still love Rails. It was critical in helping us get to where we are today.
It’s the perfect hammer for the v1, nebulous direction, rapid development, nail.
That said, we’re not using Rails for any new components.
After getting a lot of feedback from our first beta, we realized we had the wrong approach to the UI. Now, in our second attempt, we’ve got a much richer client-side front-end, written in CoffeeScript and bootstrapped by Node.js. Ruby/Rails is simply not the right tool for that job. In the new world of the browser app, even the folks of 37signals have been accused of propping up their successor.
Luckily, we had a pretty good handle on the models in the problem domain. So I ran git rm -r app/views and effectively turned the Rails app into a Json API backend. The models have evolved, but not by much. In fact, there were a few months where the Ruby code remained practically unchanged while we developed the frontend.
As time goes on, we’ve been breaking our API layer apart into several services. It makes development, deployment, and versioning easier. Service Oriented Architecture is working great for us… now that we know what services we need. These services are each too simple to justify the overhead of Rails. Some are Node.js, some are Sinatra, some are cron jobs and shell scripts. We’re polyglots, so we’re careful to choose the right tools for each job.
We’d have gotten it very wrong if we tried to build services from the start. The monolithic Rails app was the correct evolutionary intermediate form.
Given a time machine, would I have done anything differently?
I guess I should have seen that one coming. ORMs are an insidious, broken abstraction. At first, they save you time and make you feel productive. They work out splendidly for a while, but like a virus, they infect every part of your application. One day, you hit that abstraction brick wall and it’s too late to do anything about it without significant effort.
Are there things I hate about Rails?
Ignoring ActiveRecord, only one thing really stands out: Performance.
I’m not even talking about page load times. I’m talking about development time.
Even with our modestly sized application, the impact on productivity is atrocious. Our test suite runs far too slowly to be waited on for every checkin. Rake’s startup time is too great to for its usage to be considered “interactive”. Simple shell scripts have replaced practically all of our custom tasks; they are instantaneous.
I still stand by my original criticisms of Django (although, I’m sure much as changed in the last year). I’m also re-affirmed in my belief that big frameworks are generally bad. It just so happens that Rails is good enough and matches my mindset well enough to justify the framework pains.
I’m going through using Django (1.2) coming from Rails to maintain and extend a legacy web app. Generally I find Django more limited and painful than the Rails equivalent. Maybe it’s the way we create apps or just personal preference.
I think it really depends what kind of web application you’re building as to what works out well.
Most of the web apps we deploy have pretty limited usage and are closer to the prototype side of things. They are niche, custom, broadly educational-market things that have a lot of customized UI flourishes, but underneath everything are more or less standard data driven forms-type apps.
The indentation of Python that some people fuss about doesn’t bother me at all since I indent correctly anyway. Python’s methods often feel misplaced with the wrong objects to me, though.
The Ruby ‘end’ statements take a bit more space but Ruby optimizes in other ways such as with blocks and things that make the code shorter in other ways. It’s more or less a wash as far as I can tell for code brevity.
A lot of people I know and respect love Python and Django. I can’t tell them they’re wrong, but I’m personally much more productive in Ruby even after getting used to Python for a number of months.
Python performance is better than Ruby’s, but Rails 3.1/Ruby 1.9.2 feels much faster than previous older versions of Ruby and Rails to me.
I don’t think there is a clear winner for all situations in the Ruby vs. Python and Rails vs. Django religious wars, but I know I prefer and work much better with Rails than with Django even after some significant ramp-up time.
Recently discovered a stupid brain trick that I thought was worth sharing.
When writing code with unit tests, I find it helpful to view the tests and the
implementations side by side. I use vertical splits in Vim to do this.
The stupid brain trick is this: Put your tests in the left-hand split!
Why? I traditionally used the left-hand split for the “main” thing I’m working
on and the right-hand split for reference. I’ve watched many other developers
do the same thing. The right split is constantly sub-split horizontally and
always changing to various different files.
In trying to be better about writing tests first, I created a test code file
before the implementation file, so it just happened to be in my left-hand
split. After an hour or so of development, I realized that the “main” file
placement of the tests meant that I focused more on them. I viewed the actual
implementation file as just some ancillary file I had to tweak to get my tests
to pass. It changed the focus of what I was doing from “Write a method that
does X” to “Describe X”. Describing something involves breaking it up in to
smaller and smaller pieces until the implementations are so trivial that they
basically write themselves.
Here’s a list of things I like about Ruby on Rails. Almost every single one of
these are available in at least one Python framework in some form, but the best
thing about Rails is how polished the pieces are and how well they all fit
together. The whole is truly greater than the sum of its parts.
Fantastic OOBE
Rails has a fantastic out-of-box experience. Simply execute rails new and you’re off to the races with a directory full of goodies that every real web app needs: dependency management, build system, multi-environment configuration, development storage, logging, custom error pages, robots.txt file and more. See section 3.2 of the getting started guide for a complete list. Hell, they even fill out a .gitignore file for you! A place for everything and everything in its place.
Dependency Management: RubyGems
In the Python world, easy_install and its ilk are all kinds of broken. You can’t even uninstall a package! Luckily, there is Pip and Virtualenv, but assuming Pip supports all your packages, you’ve still got to make sense of dependency freezing and manually setup a virtual environment. With Rails, that all just works: rails server runs in a virtualized environment and loads gems as defined by your Gemfile; rails console for a repl. No need to muck with sys.path! Any seasoned developed has spent many an hour in dependency hell, but it is still hard to justify the infrastructure investment before writing a line of HTML. Bonus points for the “vendor” directory, in case you need to patch a third party package (or Rails itself) and want to keep it under version control.
Build System: Rake
gnu.org/software/make/manual/make.html”>Makefiles. A lot of people really hate them, but they are just so damn useful. Frequently, you need to transform some set of files into some other set of files, but only when they change. Makefiles help you do that reliably. They accomplish their feat explicitly by not being a programming language, but a dependency and build rule declaration language. I’ve seen a lot of attempts at build systems that aim to be idiomatic with respect to their host languages. Java has Ant and Maven. Python has Scons and others. Scala has Simple Build Tool. None of them are as good as plain old Makefiles. Then, I discovered Rake. Rakefiles rock. They are 90% Makefiles, and 90% Ruby. Rake expertly uses the features of Ruby to craft an API/DSL that looks a lot like Make declarations without compromising on general purpose programmability.
Markup and Style Sheets: Haml and Sass
Haml and Sass are so amazing, they ought to be illegal. Just click those links and check out the HTML and CSS comparisons. It is no contest. I never want to write HTML or CSS ever again. As far as I’m concerned, HTML and CSS are object code: to be generated by a compiler. Whenever I have to look at the generated code, it feels like I am reading assembly. Simply add gem ‘haml’ to your Gemfile, then start writing .html.haml files instead of .html.erb and .sass files instead of .css. Use html2haml and sass-convert to upgrade your crusty old markup and style sheets instantly.
Object Relational Mapper: ActiveRecord
I’ve discussed my preference for database schema reflection at great length, so I won’t reiterate here. I will, however, say that ActiveRecord makes simple things simple and complex things possible while embodying successful best practices in its various subsystems, like migrations. It doesn’t try to be everything to everyone, unlike SqlAlchemy. And it seems to be nicely onion layered for pealing back when necessary. For example, ActiveModel validations can be used on plain old Ruby objects. Which brings me to…
Form Handling: FormBuilder, params and mass assignment
I don’t need to learn and manage three different data marshaling techniques to map a database record to a web form. In fact, I don’t need to manage any. Schema reflection spares me in the model layer, FormBuilder/FormHelpers in the views, and the hash & array structured params in the controllers layer. Almost every user interaction boils down to interpreting a form’s worth of JSON-like data structures as an API call, implemented via a (secured) mass assignment to virtual attributes.
Templating: Layouts, Partials, and Helpers
When I’m getting my Haml on, I frequently want to run some non-trivial code. Feel free to shout about your separation of concerns religion, but I’ve got real things to worry about. Furthermore, we as engineers should be writing our production views/templates, not some hypothetical designer who is technical enough to write loops and branching statements, but not to be trusted with a function definition. Views are composed of markup and presentation declarations in the templates, presentation logic goes in helpers (which conveniently share scope with templates and controllers), and are neatly organized into partials. Simple, but effective.
URL Design: Routes
Rail’s routing system makes clever use of blocks to provide a hierarchical, declarative URL map that just makes sense to read. I used to design URLs in an indented text file and then map it to a list of regexps, but the routes.rb file is close enough to a spec and doesn’t require me to even think about regexes, ‘nuff said.
Authentication: Authlogic
Authentication is hard. Really, really hard. And boring. Really, really boring. But important. Really, really important. Everyone knows that the sign-up funnel is critical to get right and security is easy to get wrong. Most quality sites have a little bit of domain-specific special sauce in their authentication model. Trivial sign-up and log-in forms just don’t cut it. One size simply does not fit all for authentication front-ends, but the authentication back-ends tend to be pretty repeatable. Authlogic turns repeatable into re-usable, without the baggage of views you’re going to need to rewrite anyway. Thinking about authentication as CRUD operations on a user-session model blew my mind slightly. I had a full authentication system tuned to my needs up and running within two hours.
Authorization: CanCan
I hate ACLs. Role-based authorization simply has too much impedance mismatch for most problem domains. CanCan is a clever little library which makes zero assumptions about how permissions are represented. Simply declare a list of everything a user can do and how to tell if they may. Query with the can? function, or assert with the authorize! method. Beautiful.
Rope to Hang Yourself: Ruby Itself
One keyword in my previous post triggered much discussion: magic. The concept of magic is a topic for a whole ‘nother blog post (as I’ve already been hung a few times), but as several commenters pointed out: it’s all just Ruby. Between mixins, include, require, blocks, method_missing, symbols, and many other features, Ruby is a very capable internal domain specific language development toolkit. Most of the time, you don’t want a DSL. If you’re not an expert in both the host language and the DSL, there is almost certain confusion and maintenance danger. Most projects are either small or distinct enough to justify writing more verbose code that can more readily be understood by successors or even your future self. The gains of a DSL do not always outweigh the costs. Python excels at not supporting DSLs; it is basically executable pseudocode. However, when it comes to a task as common as web application development, it’s worth the time to develop or learn a DSL or two. Rails is loaded with DSL features and “magic” behavior that increases the learning curve, but also increases peak productivity.
Community
These sites have been invaluable: The Ruby Toolbox, RailsPlugins, Railscasts. The quantity and quality of Ruby libraries on Github continues to impress. The educational infrastructure is extensive. Somehow, the Ruby community seems to consistently be at the forefront of the biggest trends in software development. Much like Rails compared to other frameworks, the community distinctions are subtle, but meaningful. For example, Rubyists championed Git while Python adopted Hg. To the untrained eye, they are virtually identical tools, but extensive experience with both has convinced me that Git is significantly superior. I’m new to the community, but it seems like Rubyists tend to insist on quality and bet on the winning horses.
Other Random Things
Javascript and CSS minification are trivially easy with plug-ins
Static files are served with version stamps for better caching behavior
Console and file logging are usefully configured by default
Only been working with Rails for two weeks; lots more goodness to uncover
Just who the hell do I think I am?
I’m nobody. Just some opinionated computer geek working on a start-up. Not unlike many of you! My co-founder and I are currently participating in TechStars Seattle and hope to launch something killer come demo day, November 11th, 2010. We’ve got a grand vision for next generation enterprise collaboration software, but we’re starting small by focusing on making weekly status reports less painful and more useful. More details soon, but please visit http://www.thinkfuse.com. Thanks!
Imported Comments
Steve
You’ll probably remove this too (at least you read like a fan-boy) but … you
don’t show where Python is lacking at all in this article. I’m not going to
start a war but TurboGears has an equivalent to almost everything you listed
there (Although I’m not so sure about Rake), and I was looking forward to the
“comparisons” your title implies.
Brandon Bloom
I removed the previous comment because the author deleted it himself which left
a vestigial block in the comment stream saying that the post was deleted.
He wrote “Why would you compare a framework to a programming language?” to
which the answer is: because I’ve tried dozens of libraries and several
frameworks in Python. Then I tried Rails. Python as an ecosystem lost out to
Rails as a particular framework community for my needs.
To your comment: I don’t really think TurboGears does. Or at least, they are
not first class citizens. For example, the documentation on form handling
presents several options include FormEncode, ToscaWidgets, etc. It requires
that I create an additional schema class where I declare widgets outside of the
templates — or awkward @validate decorators.
Ixmatus
This is a poorly researched article. You’re comparing the language Python and
it’s vast sea of tools to Rails the framework written in Ruby.
Pylons (which is also the foundation for TurboGears 2) provides all you have
listed above, if not more. I’ve also noticed a strong tendency for Rails,
Django, CakePHP, and many other “kitchen sink” frameworks to do much of the
work for the developer and if I wanted to get around it – it becomes a day long
hackathon of digging through docs, source code, and IRC logs.
Pylons, has its own quirks and dirty sides too; but I highly prefer having to
spend a lot of up-front time programming/configuring my own framework
environment “flavor” (and being able to use it across many different projects)
then having it just be “Rails”.
Next time you post something like this, compare Rails to Django, or TurboGears,
or Pylons; not just “Python”.
Brandon Bloom
@lxmatus: I am quite aware of the distinctions between a language and a
framework. Thanks.
In this case, I am genuinely making the case that the Python web-development
ecosystem as a whole has a story to tell which is less coherent and less
attractive than the single framework: Rails.
I make this argument because I was able to use everything in this article
without having to write a lot of glue code. I spent months researching and
experimenting with Python libraries. Then I spent a week using Rails. Each of
these things worked beautifully together without any heavy setup. The resulting
complete stack aligned very closely with my preferences. And I’m a picky
bastard: I tried a huuuge list of Python libraries before I found a stack I
liked and I’m still not totally satisfied.
You can say “TurboGears has these” or “You should try Web.py” but that’s
precisely the problem. If I install Ruby, install Rails, and then visit
http://ruby-toolbox.com/ and choose the most popular plugin for each thing I
need: I’ve got a highly pleasant, well integrated, stack in hours instead of
months.
Steve
@Brandon … sorry for the accusation! I’ve watched far too many baseless
arguments between the Ruby and Python groups, so maybe I’m a bit sensitive.
In any case, the example you gave is exactly the type of information I was
expecting to find in the article. It’s very valuable to get the kind of
comparative information you provide in your comment.
As noted by Ixmatus, I find both the frameworks to have their quirks … it’s
the learning curve that you’re paying for when using either.
So I generally recommend TurboGears to people that already know Python and Ruby
with Ruby on Rails for those that want to learn a 4th generation scripting
language and use it with a RAD web framework. RoR definitely has the media
hype and resulting community (or it could be the other way around).
John
Great list of reasons to love rails (may I add: migrations, UJS defaults, and
restful controllers). The “python ecosystem” is a appropriate phrase and the
whole “comparing framework to language” debate is out of context for this
article. It is clear you taking the various frameworks and
I-can-glue-this-library-to-this-one solutions into account in your support of
the rails framework.
As far as rails doing too much work for the developer, that is the point. That
is the whole point.
Paddy3118
Horses for courses. Python frameworks have their fanbois too!
I’d recommend devise over authlogic. Devises is build around warden which is
kind of an authentication framework of sorts. Allows for pluggable auth
strategies and already has a lot of extras(fb, twitter, imap, cas, openid).
Also in rails 3.1, you’ll be able to flush the buffer before the page is done
rendering, which should greatly increase load times. It will also come with a
built in sprite system and css compression
I tend to want to customize pretty much everything heavily :–) Also, I also
explicitly did not want any views/templates/etc done for me. However, I didn’t
look at Devise closely. If I ever revisit our authentication, I’ll
reinvestigate it.
Looking forward to those new features too. Thanks!
One week ago, my co-founder
and I found ourselves staring down a thick stack of paper UI
prototypes; almost 50 pages total. One week before that, we threw
out a perfectly good, working beta product because customer
feedback suggested a significantly better path. The tale of how we
ended up at that point is entertaining as well, but it will have to
wait for another time. This is the tale of how our paper prototype
turned a pair of long time
Python developers (and
documented framework haters) into
Ruby on Rails developers.
Paper prototyping proved to be an extremely worthwhile exercise,
but we really wanted something we could click around. We devised a
strategy to build a clickable prototype that would evolve into
production code. The plan was to stub out all of the views with
static HTML and CSS. The CSS could be directly carried over to the
product and the HTML could be retrofitted as templates. Having
become religious about Sass
about a year ago, I fired up the file watcher (which automatically
rebuilds CSS) and set to work filling a directory with sass and
html files.
After some time, I asked myself “I wonder if Haml
has a --watch argument?” (it doesn’t) because Sass
makes me hate Html’s verbosity. Armed with
practically zero Ruby knowledge, I hobbled together a simple
Rakefile
and Haml watcher script and got back to work. A few Git pushes
later, my co-founder, being the clever bastard that he is, simply
installed Rails and the Haml plugin to replace my hacky scripts.
“Neat”, I said and thought nothing of it.
Then we designed our URLs in an indented plain text file, which
pretty much directly mapped into the
routes.rb file and some trivial (empty method) controllers. Whoops! Now
we’re writing Ruby.
“At this point, we might as well try Rails proper.”
Generated some models and corresponding migrations:
“Wow, explicit schema with reflective model
objects. None of this
declarative schema bullshit.”
Wired up some controllers. “The
structured params infrastructure is pretty cool.”
Integrated Authlogic.
“Wow, that was easy. I really like how it does not force any front-end decisions on me.”
“Rails is full of evil black magic that I do not understand, but it seems to
read my mind, so I don’t really care.”
Started to apply some polish and work on the 10% that takes 90% of the time. “I
think I can safely add Ruby to my resume now.”
Late at night, at the very end of the week, exhausted, we stared at each other.
“Dude, I think we can show this off to people tomorrow.” Walking out the door
of the office, my co-founder said to me “I feel so dumb for having ignored
Rails for so long. I just assumed it was Ruby’s Django”.
We accidentally implemented our entire beta product in one week, with zero
prior Ruby or Rails experience.
When I have some more time (and experience), I’ll try to write about specific
design decisions that set Rails apart.
Imported Comments
Veezus Kreist
Great story! I love the black magic quote – but I think it will feel more
apropos when you use rspec as a test framework.
Joe Lallouz
Sweet. A very similar situation happened when my co-founder and i switched to
Groovy and Grails.
Dan
How did I never notice you on Hacker News before? I read it every day!
Andrew Ingram
Weird, I moved to Django because I was fed up with building sites using Rails :p
I guess everyone has their own preferred style.
Eric Floehr
The best language for you is the one in which you are most productive in, so
awesome!
My first public website was in Rails, and I assumed Django was just Python’s
Rails.
I have since converted it to Django because Django allows me to be more
productive, and ActiveRecord didn’t work well for me.
Best of luck, and congrats again for finding such productivity!
Rives
lol, do you always call your co-founder a bastard?
Max
Did you give web2py a try? I believe it provides in Python what you found in
Rails.
JonnieCache
“lol, do you always call your co-founder a bastard?”
I’m guessing he’s in the UK :)
Brandon Bloom
@Dan: I haven’t been hiding :–)
@Andrew: I think that’s why these topics are such hot issues. It is more
preference and religion than anything else. Django is a solid piece of code and
a stellar community, but it just doesn’t mesh with my brain.
@Rives & JonnieCache: I called him a clever bastard. I consider that a
complement. And no, I’m not from the UK. I’m from NY, he’s from Philly, and we
are currently located in Seattle.
@Max: I’ve played with web2py, but wasn’t super impressed. But hey, I thought
the same thing about Rails. I guess you need to really try to make something to
understand. What about it is interesting in particular?
Nicko
Just out of interest, which version of rails are you using?
Brandon Bloom
Rails 3
Psylinse
And here I am going from Rails (because of black magic :) to Pylons, which took
a lot from Rails, but its in Python!
Perhaps you should give Pylons a look over, just some food for thought.
jbaker
I originally began programming in Python with Django myself, but I saw Rails
and immediately went.. WTF!? to Django and Python in general.
fitzgeraldsteele
I think the real lesson here is to find the right tool for the right job. Both
Rails and Django are pretty strong pieces of framework, allowing you to do a
lot in a little amount of time.
I had a similar experience, where I quickly prototyped a website in Rails. I
showed it off and said, “see…this is pretty much done.” And then developers
on my team felt the need to rebuild the site in PHP since that’s what our
company ‘officially supports.’ :)
On another project, I was able to quickly use the Django ORM to introspect an
existing MySQL database, and build an admin interface for some of our internal
users. That was a HUGE win for me to be able to quickly make the admin site.
So, the right tool for the right job…
Max
@Brandon
Given your article I thought your would like the fact that in web2py everything
has a default. For example is you just type:
db.define_table('post',Field('body'))
web2py creates the table (if it does not exist, or alters it as necessary), a
web based database administrative interface, and crud.create, crud.update,
crud.read, crud.select, crud.search forms that you can insert in your app like
It also includes role based access control out of the box with pluggable
authentication mechanisms.
Web2py also has a component/plugin architecture that is best described by this
video http://vimeo.com/13485916
Since you are coming from Python, I though you may be interested. Anyway, I too
really like Rails and web2py was originally designed to be in Python but closer
to Rails than other Python frameworks.
brokenladder
Semantic indenting is far and away the best thing about Python, and Haml/Sass
prove Ruby should adopt it too.
Brandon Bloom
@brokenladder: Hell yes.
coulix
Well in my case I am way more productive in Python, especially when prototyping
with Django. It all depends on the mastering level of the tool. One point I
will give to rails is haml tho. We need a proper equivalent well integrated in
Django/Python.
Rafael Rosa Fu
Hi,
Next time you need to authentication, try Devise instead of Authlogic, and
check Ruby Toolbox for more interesting Ruby tools.
Could you expand on your file watcher script? Running a background process
(e.g. re-compile) on change is a common pattern that I haven’t found a tidy
solution to…
Brandon Bloom
@kevin: I’ve written a bunch of watchers in the past. Each had their pros and cons.
FSSM has the advantage that it uses the platform’s native file notification
mechanism, so it tends to respond more quickly (no polling delay).
T T
Awesome post! I’ve been comfortable with Django though I am dabbling into Rails
but I’m having trouble with getting to grips with Ruby. Would you recommend
anything you’ve read that allowed you build a working prototype? I’m curious
about how you managed to learn it so quickly. Thanks!
Brandon Bloom
@TT: I took advantage of the fact that many people switch to or from Python. I
searched for phrases like “python’s foo in ruby”. Ruby is so similar to Python,
it was pretty easy to just pretend it’s the same and go back for cleanup as you
learn the idiomatic approaches.
Also, I generally just have an interest in programming languages. I’m pretty
fluent with about a half dozen languages, so each marginal new language is that
much easier to pick up.
As an aside: the “end” keyword is just sooo unnecessary!
NoScript does not publish their individual add-on usage statistics, but the global download/usage ratio can be calculated from the statistics on the Firefox Add-on home page:
1,962,617,946 add-ons downloaded
157,090,095 add-ons in use
About 8% of downloaded add-ons are still in use. Assuming NoScript’s usage
ratio is comparable to the average, approximately 5.4 million installations of
FireFox are running NoScript. Let’s ignore the fact that NoScript’s usage ratio
is probably much lower than the average, due to the fact that it breaks most
web pages.
I’d wager that the average NoScript user has at least two machines, so the
total number of NoScript users is probably less than 2.7 million.
Even if 100% of NoScript users were Americans, they form 1% or less of the
general population. If you, like me, believe that I have been generous to
NoScript here, it is likely that no script users are no more numerous than 1 in
1,000.
Even if the user is using NoScript, they can whitelist your site. You can
probably add <noscript>WARNING: THIS SITE IS BUSTED WITHOUT JS</noscript> to
the top of your page and call it a day. If you are feeling generous, redirect
no-script users to the mobile version of your site and tell them why.
tldr: Assume that human user agents have Javascript.
Imported Comments
Anthony DiSanti
What’s the inspiration behind this post? Did you think you actually needed to
support a strong noscript experience? JS is mandatory at this point, screw
anyone that’s turning it off.
Brandon Bloom
Facebook’s BigPipe tech announcement prompted some Javascript-required
discussions on Hacker News.
My prior post was more controversial than I anticipated. In hindsight, I
should have realized what a hot button issue web frameworks are. One assertion
went all but unnoticed. I expect it may be even more controversial:
the schema-generative ORM paradigm is fundamentally flawed.
Disclaimer
I came to this conclusion while working with Django’s ORM, but this post is
completely object-relational mapper agnostic. We are now using SqlAlchemy, but
we are not using any of the many available declarative layers. Instead, we are
using schema reflection and semi-automatically configured mappers. This is not
an argument against ORMs. It is an argument against generating database schemas
from ORM declarations. By extension, this is an argument against Django’s ORM
because Django uses an exclusively schema declarative model. That said,
Django’s ORM is far from alone in this camp.
Data Outlives Code
When code is dead and gone — be it through rewrite, obsolescence, or by other
means — the data will still be there. Longevity implies slower evolution; data
is always more difficult and riskier to change. Data is also more valuable.
What if Facebook rebooted their database? They’ve already rebooted their
software several times.
Schemas are data. As data, schemas are longer lived, less flexible, and more
valuable than code. These factors alone suggest that the database itself should
hold the authoritative schema, not a class declaration in the code.
If you have inherited data from another project, you already know this lesson.
You can’t generate the schema from code because the schema already exists. You
can mimic the authoritative schema in your declarations, but it is easier and
more accurate to use reflection.
Ineffective Domain Objects
Object relational mapping is primarily a serialization problem. Every
serialization solution has its quirks. The scale and number of quirks seem
directly proportional to the absolute difference between the runtime and
storage representations. Since a database is a completely distinct type system,
rather than an opaque byte array, serialization to a database can have
particularly quirky quirks.
Modeling is one of the fundamental challenges of software development. Capable
developers prefer highly expressive or unconstrained type systems to aid with
modeling. Generally, runtime type systems are more expressive than those found
in databases. Declarative relational mappers, however, constrain the
programming language type system to its less expressive counterpart. When
building domain objects, the developer must think in terms of the database’s
type system, not the programming language’s.
While using a declarative object relational mapper, developers are effectively
trying to design storage and runtime models simultaneously. On average,
superior results are achieved by modeling these two concerns separately and
then solving an additional subproblem: serialization mapping. You might wind up
with slightly more code, but it will be easier to understand and maintain.
SQL Is Not Going Away
Despite forcing a less expressive type system on to the developer, declarative
ORM layers attempt to treat the database as an implementation detail. However
much we wish this were true, the database is not a detail which can be ignored.
Sooner or later, you are going to have to open your database shell and write a
SQL expression. This requires knowledge of your database’s particular SQL
dialect and idiosyncrasies. Exacerbating this issue, generated schemas are
typically full of name mangling and other ugliness. It is far more pleasant to
work with a carefully designed schema than one that compromises for the ORM or
the runtime type system.
Schema and Declarations Diverge
Data migrations present the most pressing need to work with SQL directly. As an
widely unsolved problem, automation can not be trusted. Unless you are
painstakingly simulating the schema generator, the production database schema
will slowly diverge. Most deviations are tolerable as they will not affect the
runtime behavior, but it is wise to minimze differences between production,
staging, and development environments. To faciliate this, store migration
scripts and backed-up schemas in version control.
Some deviations will directly affect runtime behavior. For example, consider
the case where two versions of an application are running in production. An
is_read boolean column was added to a message table in the database. It’s
default value is false. When a row’s page is viewed, the is_read column is set
to true. The old version of the application doesn’t know about the column, so
it can not set the flag. When the new version is rolled out to everyone,
affected user will see a bunch of read items marked as unread! The solution is
to set the column’s default to true, but initialize it to false in the
application. Declarations must either deliberately deviate from the schema or
present a misleading default value to be overridden during initialization. This
is just a simple example, real world schema migrations can be significantly
more complex and suffer from numerous more subtle problems.
Imported Comments
Martin Diers
These are all good points, and largely irrelevant for someone developing to
Django. Yes, if you are using Django to develop to a long-term data store,
enterprise data, etc., you are dead right, and you should be using SQLAlchemy
with Django instead of the Django ORM.
However, consider Django’s original target use: a newspaper, on the cutting
edge, putting up quick apps on-the-fly with a very short development cycle. A
CMS. A Polling app. A Blog. Think about the type of apps that people actually
build with Django, and you will find that the Django ORM makes perfect sense.
Short development cycle. Easy deployment. “Good enough” query
optimizations.
Ezequiel
I agree with Martin, Jacob Kaplan-Moss himself said that, when you get to a
point where you’re building something really big (e.g. Twitter) you basically
have to throw your framework away.
Django is really good, but, like any technolgy, sometimes is not enough.
Brandon Bloom
@Martin: The “original target use” argument is no longer relevant. Django is
being pushed as a general purpose web framework for developing sites or apps,
big and small.
@Ezequiel: I’m not building something Twitter big. I’m building something that
has to deal with real user data and survive real design evolution. The problems
I describe affect any application with any longevity of any kind.
Julien
I agree I’d rather go schema first than model first. As a side note though it’s
interesting to think about it in the context of the whole NOSQL mouvement
that’s been gaining a lot of traction over the last year. Facebook, LinkedIn or
Amazon for some subset of its data and others are doing away with the
relational stores and going with pure key/value pairs persistence.
Not relevant for every application but very interesting nonetheless.
Ergo
Have you ever considered something else than django? Like pylons or repoze.bfg?
Django is really not so good solution for more complicated tasks if you want to
work efficiently.
As promised, I’d like to elaborate on the URL routing system I came up with.
Weighing at less than 200 lines of code (including example), I’ll let it speak for itself:
download it.
This approach seems to be working great for us. Love it?
Hate it? Feel free to let me know what you think.
Imported Comments
Dead Man
You are hardcoding child views in the children method of each view which means
the view can’t decide what (type of) children it has based on the current path
segment. For example, I have a use case in which some content types can contain
children of more than 1 type, so for example /foobar/child/ can either be a
TypeAView or TypeBView depending on whether a TypeA with name “child” can be
found or not – if it cannot be found, TypeBView will be returned. If your
children() method took a path segment as an argument, it would probably solve
this problem.
BTW have you checked out how Restish
resolves/dispatches URLs? I think they have it right when it comes to dynamic
URL trees.
Brandon Bloom
Child views are not “hard coded”. They are determined by the children method.
You could pass segment, but it would be easier for the bind method to just
store something on self for use later.
I looked at Restish a bit. It seems like a nice library; definitely the closest
to my taste from what I’ve seen.
Thomas
I’d recommend taking a look at Werkzeug for your routing & other wsgi basic
needs.
As a follow-up to the previous post, Jinja 2 makes for an excellent templating
engine (think django but much faster and without the stupid limitations).
I am sorry I still don’t get how this serves as an improvement over Django’s
url system?
Can you provide me with a concrete example of what you cannot do in Django in
terms or url-routing?
Andriy Drozdyuk
in terms of* I meant. Spelling is not one of my strongest areas eh?
Brandon Bloom
There is nothing that Django’s routing can’t do. In fact, the first version of
this simply generated Django routes. This is about making is faster and easier
to define hierarchical permissions and trees of URLs in a way that queries the
database on a per-url-segment basis.
My co-founder and I built a non-trivial web site on Django. When the next
version ships, there might not be a single Django module imported.
We’re not trying to drop Django; it is just sort of happening. Piece by piece,
it is failing to meet our needs. Despite the marketing copy on the Django site,
most components of the framework are tightly coupled enough to make
customization frustrating. It is often easier to rewrite core framework
components than to implement them on top of the existing extensibility points.
What follows is a loose chronology of our migration away from Django.
URL Routing
A flat list of patterns violates the DRY principal when creating nested URLs.
Trees are a superior representation. Having a tree of views also enabled us to
optionally associate a “binder” function with each node. These bind functions
are executed for each URL component from left to right, filling the template
context as they go. Breadcrumbs are automatically generated as each node binds,
but only the last node executes its full view logic.
Authorization
Our site enforces permissions on every resource, but Django’s database ACLs
would have been prohibitively numerous. Instead, views or their URL binders may
raise an AccessDenied exception. Upon catching such an exception, a middleware
layer serves a login form. This ensures users have permission to access the
current resource, as well as all ancestor resources bound to the URL.
Authentication
Both of Django authentication’s key extensibility points are flawed. These two
extensibility points are “user profiles” (storing additional per-user data) and
custom credentials (such as for logging in via email address instead of
username). Django’s documentation and numerous internet sources cover both
topics, but all of the guidance lacks important caveats. The admin UI, in
particular, is very easy to break with either extensibility mechanism.
Extending the User model with the ORM requires a one-to-one database
relationship. This relationship can be implemented with a “user profile”
setting, an explicit foreign key, or model inheritance. Each approach has its
own strengths and weaknesses in terms of performance, API semantics, subtle
behavioral changes, and outright bugs.
Enabling custom credentials requires implementing a trivial authorization
“backend” object. Unfortunately, it is non-trivial to replace usernames with
email addresses. The admin UI’s login form refuses to accept email addresses
without hacking the template. Even if you hacked the template, the User model
would still enforce a non-null constraint on the username field and the
generated database schema enforces a uniqueness constraint as well. It turns
out to be easier to fill the username field with a dummy value and “support”
both forms of authentication with your backend, but you won’t come to that
conclusion until your head has already bore a hole in your desk.
Templating
We do our best to keep view and template logic separate. Django’s templates are
targeted at designers, who aren’t implementing any real logic anyway. However,
we’re a pair of hackers. Sometimes it is just more convenient to put a little
bit of logic in the views. Besides, templates are code; code needs to be
reviewed and tested. We wouldn’t ever hire a designer who couldn’t pass a code
review for some trivial template logic.
We needed a pragmatic template language to replace Django’s idealistic one. Any
template language with greater expressive power would have been welcome, but
Jinja2 fit the requirements and provided the easiest migration path.
Ultimately, we’d prefer to use something like HAML, but there doesn’t seem to
be a Python equivalent besides the inactive GHRML project. We are, however,
using SASS. I will never write CSS by hand again.
ORM and Admin UI
One of Django’s most touted features is the Admin UI. For simple “active
record” style database models, the Admin UI is a huge time saver. Sadly, it
struggles a little bit with nullable fields and is tricky to customize. You’ll
definitely need to write custom UI for complex models, but by and large the
admin solves the problem at hand: viewing, creating, updating, and deleting
database rows.
After using the Admin for a little, I found myself missing Microsoft Access. I
never thought I’d say that, but it is true. Django’s admin does not support
sorting, filtering, or other impromptu queries.Edit: It turn’s out
I was mistaken about sorting and filtering, but I stand by the core message of
this section. I found myself writing impromptu queries in the
database and Python shells. After a while, I just gave up and installed a
desktop client. I haven’t visited the Admin UI since.
Django’s ORM has shortcomings with respect to querying, especially for joins
and aggregation. It has been improving over time, but it will likely never
reach the capability of projects solely focused on databases, such as
SqlAlchemy. With the admin having fallen into disuse, the Django ORM lost all
advantage. Beyond Django’s specific weaknesses, I’ve come to believe that the
schema-generative ORMs paradigm is fundamentally flawed. That is a topic that
deserves an entire (Django-agnostic) post of it’s own. We are now using
SqlAlchemy via schema reflection; no declarative layer.
Form Validation and Generation
Here is where our chronology meets present day. We are still using Django form
validation, but never used form generation beyond scaffolding. Nearly all of
our templates customize labels and display of errors. Additionally, embedding
widget information in the Python code is cumbersome during template
development. Django forms is a quality validation library, but there are some
inconsequential style things that I like better about FormEncode. Preferences
aside, the difference isn’t large enough to justify switching.
While I like FormEncode, I’m still not sold on its anti-form-generation
companion, htmlfill. I think there is a middle ground with form generation that
provides scaffolding during development, smoothly transitions to production
use, and cooperates with validation. As we implement more complex client views,
I’ll be on the lookout for ways to improve our form development toolbox.
So, ugh… What’s left?
Besides a few isolated helper functions, not much is left of Django.
The last big ticket item is the HTTP framework and WSGI server. We could
continue using Django as if it were CherryPy or Paste, but Django has this
nasty habbit of insisting on running your code for you. The settings and
manage.py infrastructure are fiddly for deployment and don’t really add any
value over simple scripts using our application like a library. Might as well
use a simpler WSGI library, and replace those over-engineered
management/commands/foo.py files with vanilla scripts/foo.py files.
Moral of the Story
I’m sure there are numerous lessons to be generalized from this journey.
Personally, I’ve developed a moderate fear of the word “framework”, as well as
altered the way I think about software abstractions. I think the most important
lesson, however, is one I already knew: choose the right tool for the job.
Unfortunately, we had no idea what the right tool was when we started. I’m not
sure we know any better now.
Imported Comments
Austin
Maybe it worked out exactly like it should have. Django bootstrapped your app
to a certain point. Got you further faster than you would have if you
implemented everything from scratch. Then from there, you identified the
things you considered inadequate and replaced them. If it all goes away who
cares. You have learned something, shared it with us and moved on.
You seem to have successfully moved away from it to your own custom code. I am
impressed that it sounds like your decoupling your app from Django was not as
traumatizing as it might have been.
Brandon Bloom
Yeah, everything seemed to work out OK. Django was definitely a big help,
especially with my initial limited web experience (previously, I was a game and
desktop application developer).
If I were to do it all again, I probably wouldn’t choose Django. That’s not to
say Django wouldn’t be the right choice for someone else. I just think that I’d
choose a collection of tools that are less likely to get stuck glued together.
Eric
I think Pylons would be a great fit for you. It matches the way you want to
work.
Varikin
Your URL routing sounds intreging. I like have been looking at Werkzeug,
Routing and Django. I prefer Django to all three, but still I feel like
something is missing.
Could you elaborate on your solution?
Juan_Pablo
Take a look at Werkzeug. Despite the weird name is the most modular and yet
easy to use wsgi wrapper and dispatcher I know. Combine it with Jinja2 and
SQLAlchemy… and voilà.
WhatDoYouThink?
Good thing there are people like you who would share your improvements back to
the community!
Brandon Bloom
Thanks Juan, I’ll take a look at Werkzeug.
Eric: I’ve been looking at Pylons, but I think I want to go with something more like Juan’s suggestion.
Varikin: I was able to implement my URL routing over Django by simple matching
(.*) and then writing a view which splits by ‘/’. I then loop over each
component and perform a depth-first search of a tree structure. Each node has
an optional bind(request, template_context, path_component) function. After I
find the node which matches the path, I call the ancestor chain’s bind
functions and then finally the current node’s view function. We’ve got some
decorators and helper functions to make building this tree easy. I’d share it,
but it is too tightly tied to our application. Maybe I’ll write more about this
later.
WhatDoYouThink: My improvements are application specific. I’m sharing my
experiences to help the community. That is far more than most people do.
Jeremiah
This comment has been removed by the author.
gaspode
The tree stuff and permission scheme you’d like perfectly matches up how
repoze.bfg works with traversal and security based on traversal schemes (also
known as object dispatch).
I’d highly recommend giving it a shot as it seems to mesh quite well with the
way you’ve laid out the app.
Tucanae Services
Think Web2Py…..
Chris Shenton
Ditto what gaspode said. I’ve been fighting Django for an app whose content is
basically hierarchical and BFG’s traversal would really be a relief.
Cesar
I’d like to add my 2 cents. Restish has no threadlocals (compatiable with
eventlet/spawning) and focuses on resources/content negotiation. I believe the
URL routing is similar to what you want
handi
Enabling custom credentials requires implementing a trivial authorization
“backend” object. Unfortunately, it is non-trivial to replace usernames with
email addresses. The admin UI’s login form refuses to accept email addresses
without hacking the template. Even if you hacked the template, the User model
would still enforce a non-null constraint on the username field and the
generated database schema enforces a uniqueness constraint as well. It turns
out to be easier to fill the username field with a dummy value and “support”
both forms of authentication with your backend, but you won’t come to that
conclusion until your head has already bore a hole in your desk.
We wanted to let users log into our app using their email address through
contrib.auth.view.login. We don’t care about having email addresses log into
the Django Admin.
I think you were looking for Pylons not Django. Each has its own advantages. I
learnt that Django should be taken as is, once you start stripping it apart,
IMHO you should consider using Pylons.
If you like SQLAlchemy, you will probably like FormAlchemy for form generation
from SQLA models.
adeel
I’ve written my last few web apps on top of
Berry,
which just lets you map URLs to methods (and lay them out however you want).
And obviously you can import SQLAlchemy, Jinja, and whatever else you want
yourself. I’ve found this gives me the most flexibility possible without
actually writing a raw WSGI app.
Thanks Michael. Despite the distracting laughter that was an interesting
read.
gunzip
After using the Admin for a little, I found myself missing Microsoft Access.
it still happens to me every time i start a new data-entry project on the web.
sometimes i think something went wrong in the latest 20 years…
Dead Man
I’m also building a project on Django (a medium sized, multi language web
encyclopedia about a Northern-European state Estonia) and I have to admit that
I’ve run into mostly the same problems. What surprised me the most is that I’m
implementing a nearly identical to yours solution for URL mapping as our
content is also in a (generic) tree structure. The difference being that I’m
going to use class based views that allow nesting/composition of other views
for the purposes of chaining that you described.
I’m also thinking of open sourcing the code as the project matures more as it’s
already written in a clean way and decoupled from project specific code.
ars
Beyond Django’s specific weaknesses, I’ve come to believe that the
schema-generative ORMs paradigm is fundamentally flawed. That is a topic that
deserves an entire (Django-agnostic) post of it’s own. We are now using
SqlAlchemy via schema reflection; no declarative layer.
Well you caught my attention with that. Are you planning to write such a post
in the near future, say in the next two weeks?
Andrew Shultz
One of the big things I appreciated about django is the unit test supports.
What’s your plan to replace that stuff?
proteusguy
Almost all of your technical feasibility issues are easily addressed by Django.
URLs, authN/authZ, etc are all as flexible or more than any other python
environment out there because it is so simple to leverage python to implement
this things in an incredibly simple manner. This blog entry demonstrates it better than I could. It’s really
just a matter of perspective and context. Django may not be the context that
best fits your perspective but it most certainly has the capability and power
to address all the technical concerns you brought up. Good luck finding the one
that fits you best. I actually thought I’d be a TurboGears guy but ended up
Django. Both are great and others are good as well.
Alexei
I am working on building a medium size application in Django too. It is indeed a good way to start. But I am hitting issues too. I have multiple “agencies” so need to have users with multiple roles in agencies. These are within single site. That also means multiple “admins” with one “super admin.” So Django Admin interface is out of the question and role/permission infrastructure is built from scratch.
Another thing is importing data. I started writing scripts within Django framework (using settings.py, etc.) but need some tables/entities created during import and then dropped. Oops. Have to look at how South does it.
I realize that I am moving out of Django domain just because of the nature and complexity of the application I am building. Django cannot and should not be everything for everybody.
If you are looking for a “glue” framework, I recommend checking out Cherrypy: I like its configuration and request/response handling better than Paster + WebOb. It also has several types of routing, pick what you like or mix it (I would not recommend the latter though), one of them is Routes.
My next move will most likely be to Cherrypy + SQLAlchemy + FormAlchemy + Jinja2.
By the way, repoze.bfg is not microframework either, it pulls quite a large chunk of Zope code, primarily for managing configuration but for templating too.
RJ Ryan
Got a few issues with the things you said — I’ll list them by section.
URL routing is not a flat list. You can make them hierarchical. How do you
think all of the admin site routes get added? It’s in the template urls.py for
any starter project, so I really don’t see how you missed this.
Authorization — I don’t think I understand what you are saying is wrong —
“too numerous?” But designing a permission system using the Django auth system
has usually been pleasant for me. To require certain permissions just use a
decorator on your views which checks for it.
Authentication. There’s nothing that says you have to make a one-to-one with
User and whatever profile system you have. That’s just how most people do it
because most projects want 1-1. Make a ManyToMany from your Profile to User if
you want. There’s nothing stopping you. Also, using email address as your login
was simple — swap the auth backend. It’s like 10 lines of code.
Templating: Fair enough, I’ve wnted the templates to allow a little bit more in
the way of doing logic only for purposes of output, but you realize if they
make allowances for that, then it opens the templates up to abuse — and then
you get PHP. I’ve actually come to appreciate the template strictness now, and
when I come across something I just have to do in the template, I write a
templatetag.
ORM/Admin — I can’t really understand what you’re trying to say here. The
Admin UI is a very nice piece of software, and it is extensible at every point.
Though, the documentation on doing so is skimpy. Take a look at Satchmo — they
do amazing things to the Admin UI that really showcase its flexibility. As for
the ORM — if it really doesn’t suit your needs, upgrade to django-sqlalchemy
or something.
Forms — it’s true forms are sort of a go-between between views and templates.
It does seem odd to specify the widget in the form declaration. Shrug, though
the form is more of a prototype of a form, so you design the prototypical form
that is instantiated in a view. This makes sense to me.
I’m sure you actually did run into problems with Django that caused it to be
frustrating for you, and to that I say ymmv — I just couldn’t help reading
your post and being like “but that’s not true, because I’ve done X and he said
it can’t do X”.
Shantanu Kumar
Did you consider Jython? You could then probably use any framework from the
Java-space.
EH
Faced same problems with Django during 2 years of its using. Looked also at
Paste, web.py, Werkzeug, Pylons, WebOb to replace over-weight framework with
more flexible tools. My current best python tools is Werkzeug, Mako,
SQLAlchemy.
reedobrien
Alexei said> By the way, repoze.bfg is not microframework either, it pulls
quite a large chunk of Zope code, primarily for managing configuration but for
templating too.
I guess we will have to agree to disagree. By alexei’s definition 20 packages
isn’t ‘micro’. By mine 4 packages I wire together isn’t a framework…but to
clarify some misnomers:
repoze.bfg is a microframework. it consists of 20 packages, 8 of which are zope
packages. Although, BFG uses them you never have to.
The default templating package implements TAL, but it is NOT a zope package.
You are not bound to it either. You can plug in Jinha2, Mako, Genshi or
whatever you want.
There is no default database, form library or anything else.
FWIW zope is about 170 packages. 8 is not a ‘large chunk’ of that by any
definition.
Sébastien
If you’re looking for a HAML Python equivalent, there is PAML (Pamela) at
http://github.com/sebastien/pamela — also, you can check CleverCSS as a SASS
replacement.
Gabriel
I totally agree with RJ Ryan. No framework will suit every project, but Django
covers a lot of the basics in a way that’s good enough and doesn’t get in the
way when I need to extend it. Still, thanks for telling us why it wasn’t the
tool for you.
Tony Landis
I use pylons + mako + sqlalchemy and love it
Brandon Bloom
@Dead Man: I’ll probably publish something about our approach to URL routing
soon. It seems that several people have shown interest in it.
@ars: Yes, I will probably be writing about declarative ORM layers in the next
week or so.
@Andrew Shultz: We’re not doing a whole lot of automated testing yet. Small
helper functions typically have doctests, we’ve got a couple unit tests, and
we’re currently playing with Selenium for full tests. I’m much more confident
in tests which drive the UI than tests which only poke at the views.
@proteusguy: I just read that blog entry and will respond to it separately.
Again, I’m not bashing Django so much as I am saying we’ve been slowly choosing
tools that better fit each need as we identify those needs. Technical
feasibility aside, we’ve been looking for a path of least resistance to
accomplishing our goals and this post was just to describe them.
@Alexei: That’s the impression I got too; Django would have had greater
longevity on my tool shelf if I needed a highly customizable CMS. I don’t need
more flexibility, I need less constraints.
@RJ Ryan: Like I said about, I’ll try to follow up with more info about our URL
routing system. What needs we had and why I felt I had to subvert Django to get
them.
@Shantanu Kumar: There are already too many choices of Python frameworks! :–)
@Sebastien: Thanks for the PAML link! I’ll take a look at it. As for CleverCSS,
I don’t see a need to switch from SASS. We only need Ruby and SASS installed on
dev machines, the server only ever works with the compiled CSS.
dkubb
“Beyond Django’s specific weaknesses, I’ve come to believe that the
schema-generative ORMs paradigm is fundamentally flawed.”
I too would like to see some more detail on this. I maintain a Ruby ORM called
DataMapper, and while the API is completely different, it is still declarative.
I prefer the declarative style to ActiveRecord, since I can specify rich types
and constraints than I could reflect from a database. The way I see it, at
some point the schema needs to be defined, and I’d rather do it in a richer DSL
and have the schema, validation and other constraints generated from that.
You can have non-flat URL routing. After your first point demonstrated a
failure to grasp more than the basics before whining, I tuned out.
Nested URL routing confs and includes (new in 1.1) are what you should have used instead of bitching on the internet. It wouldn’t have made you seem as cool, but you would have gotten the job done instead.
Brandon Bloom
Without the changes in Django 1.1, modules were the only recursive mechanism;
too heavyweight for my taste. The 1.1 beta was released in March. I had
developed my own solution to this problem several months before then.
Even with the 1.1 improvements, Django doesn’t meet my other requirements:
namely binding URL segments to the database and creating breadcrumbs while
filling a template context dictionary. Since that work involves parsing
request.path myself, I might as well just bind (.*)+ and do my own routing. It
was relatively trivial.
Shawn’s AppWeek post inspired me to write one too. AppWeek is our chance
to be creators for a little while and it was a lot of fun. I didn’t set out to
build something nearly as ambitious as “Super Avatar Sample Smashup EXTREME! –
‘Capture the Cat’ edition”, but I did get to take a swing at a game I’ve wanted
to build for a while: Rock’em Sock’em Avatars Avatar Boxing. Avatars, being
a new feature in this release, were an unwritten requirement for all of the
AppWeek games. Between SASSECTCE, my game, and the many others, Avatars were
chasing cats, beating each other up, play futuristic sports, falling off
buildings, dancing in a cloud of gems, being launched from canons to save the
world, and much more. All this excitement was almost too much for a bunch of
exhausted engineers, but that’s what the beer was for during the game
unveilings.
Here’s what the game looked like with the basic animations wired up. You’ll
notice that the avatars have been hitting the gym. That’s because their arms
were too short to reach each other! I added a little extra bulk because I was
laughing too hard not to. I directly bound the game pad triggers to the
shoulder and elbow joints and rigged up the chase camera sample to inspect my
work. There wasn’t much game play yet, but it was already fun. That’s always a
good sign.
IMAGES MISSING :–(
Even with just one week, I decided to invest some time into debugging
visualizations. That turned out to be a really great idea.
IMAGE MISSING :–(
Then, I added some collision spheres for the heads, hands, and upper bodies.
This was a hacky, trial and error process. Thankfully, C# compiles quickly.
IMAGES MISSING :–(
At this point, I spent an entire day working on the physics. I wanted the
avatars to bounce/wobble when they got hit, so I rigged up some complex spring
systems. Things were starting to work, but I’m generally pretty bad at this
sort of thing and my simulation routinely exploded. The avatars arms went
shooting off into space and I was getting pretty frustrated. No screen shots of
that chaos because I am embarrassed.
With half a day to go, I added the obligatory damage bars and some rudimentary
hand-to-head collision detection.
IMAGE MISSING :–(
I was feeling pretty good about the game, despite my physics failures, it was
pretty fun anyway. I wondered down the hall to chat with Jace, who had just
added sound effects to his game. His game was hilarious before, but the sound
effects were priceless. I ejected the sound effect CD out of his machine,
yoinked it, and took off running. An hour later (and 10 minutes after the
deadline), my game had some sweet punch and miss sounds. I also made the
avatars’ heads pop up when their damage bar was full, accompanied by an awesome
zip-tie sound.
IMAGE MISSING :–(
At our team happy hour, I’d like to think Avatar Boxing was a fan favorite. I
certainly had fun making it! I hope everyone enjoys Avatar support in the new
XNA Game Studio.
Imported Comments
Danny Tuppeny
I really wish you guys were releasing these games – they sound so funny and it
would be great to see how you did some of the stuff with Avatars.
Brandon Bloom
As Shawn said: “We did an AppWeek shortly before shipping Game Studio 1.0,
which produced minigames by Dean and Minjie, plus a bunch of stuff that was too
unfinished or too much of an IP violation to release.”
Believe me, the code for my game isn’t remotely useful :–) Hacks upon kludges
upon hacks galore. It would be a significant chunk of work to turn it into
something educational. Even if it was interesting, it is technically not mine
to release. It would be a conflict of interest to publish it on Xbox LIVE
Indie Games, since I created it during the course of my job.
Since our games are only being seen by the team, many developers use
copyrighted assets in the interest of time. Fair use allows us to show them at
our happy hour, but someone would certainly come knocking if we released them.
It is very expensive to clean that up because now you need to involve artists
and rework the code for the new art (AppWeek games aren’t robust, art changes
are breaking).
Everyone would love to share, but it isn’t universally practical. Luckily,
several cool Avatar-centric samples were inspired by app week, so we will have
something to share. I’ll also see about recording some videos of the IP-safe
games…
Danny Tuppeny
No worries, it makes sense. I’m sure we’ll see more Avatar samples soon
that aren’t hacked together we can learn from :-D
Kyle
This looks like it could be turned into a real fun game. It’s too bad that we
can’t buff our Avatars up like this =(
Brandon Bloom
Yeah, sadly it appears that scaling individual body parts is against the rules
(which were unwritten when I did this project). I believe the motivation is for
accommodating future avatar accessories which might look awful with unknown
joint transforms. Don’t quote me on that :–)
I tried PowerShell when it was first released, but never used it for real work.
I recently attended a “brown bag” presentation about PowerShell. This
presentation spurred me to augment our team’s environment with PowerShell and I
have been using it every day since.
In the past weeks using and abusing PowerShell, I have drawn two conclusions:
PowerShell has a killer set of standard tools with brilliantly designed
usability.
The PowerShell team doesn’t understand UNIX and therefore were condemned to
reinvented it, poorly — with apologies to Henry Spencer.
First things first: if you spend any time working with Windows, get PowerShell.
Now. Stop reading my blog and go download it immediately. It mops the
floor with cmd.
The key premise behind PowerShell is that it operates on live .NET objects.
This is beneficial because it eliminates a lot of the text cutting and
manipulation common in shell scripts. Additionally, it puts the full .NET Base
Class Library into your scripting toolbox. PowerShell tools, known as
commandlets, typically only render the most common fields for their objects,
but the less common fields are easily available in memory. By convention,
Commandlets are named with a verb-noun pattern and support a common command
line parsing behavior. The repository of commandlets and the command line
options of each are easily queried and highly consistent. All this meta-data
makes PowerShell a breeze to learn.
I fell in love with the the discoverablity and ease of use when I tried to kill
a collection of runaway processes:
PS> get-command -noun process
CommandType Name Definition
----------- ---- ----------
Cmdlet Get-Process Get-Process [[-Name] <String[]>] [-Verbo...
Cmdlet Stop-Process Stop-Process [-Id] <Int32[]> [-PassThru]...
PS> get-process notepad | stop-process
PS> get-alias | where { $_.definition.contains("Process") }
CommandType Name Definition
----------- ---- ----------
Alias kill Stop-Process
Alias ps Get-Process
PS> ps someotherapp | kill
OK, that’s pretty cool and oh-so-very Unixy — right? Wrong. Notice the
“CommandType” column in the results of get-command. There are many other types
of commands besides commandlets: functions, filters, scripts, applications,
etc. Each of these has slightly different semantics for pipes and parameters.
Applications, for example, have no way of accepting .NET object pipes. You must
develop a separate commandlet. Yikes!
Compare to Unix: all commands are applications which accept a command line and
pipe byte streams in and out. Much simpler, but byte streams aren’t as
friendly, discoverable, and maintainable as object streams. However, the
brilliantly simple thing about Unix is that, when you get right down to it,
object streams are just byte streams! There is absolutely nothing stopping you
from implementing get-process and stop-process as Unix programs which pipe
object references, JSON, pickled Python objects, XML, S-expressions, or any
other data format you fancy. Doug Mcllroy, the inventor of Unix pipes, was
right: text streams are the universal interface.
Actually, this is no different on Windows. All of the PowerShell commandlets
could have been implemented as applications which import a library. This
library would replace main in much the same way as winmain, provide a
metadata enriched implementation of getopt, man, etc. There is no need to
invent a new shell in order to acquire the power of piping objects. Sure, cmd
is old and needed to be retired for many other reasons, but it is a real shame
that the PowerShell toolset is not available to those of us stuck in batch
scripts.
Personally, I would really like to see such a library developed. Microsoft has
certainly proved one thing with PowerShell: steep learning curves are not
intrinsic to command line interfaces. Unfortunately, commandlets are two steps
forward and one step backwards. I have no doubt that we can retake that forward
step.
Imported Comments
klumsy
I find the 2nd point interesting given that most of the powershell team were
unix only guys before this project.
Brandon Bloom
I don’t know the history of PowerShell or any of the team members. I’m sure
they are all very smart and have a strong grasp of Unix in general. Maybe they
understand exactly what I’m getting at, but explicitly rejected it for some
non-obvious reasons (such as scheduling, implementation complexity, target
users, etc). They probably know better than I do. My apologies for exaggerating
to make a point :–)
Howard T. Snidbiscuits
I agree, PowerShell is way better than the primitive, DOS style Windows CMD.
However, it’s nothing like a normal Unix shell for me… and I’d rather just
use cygwin.
Kosta
Sorry but no: Pipelining objects is much smarter. You can still pipe strings or
binary data if you want, but you don’t have all the drawbacks.
And then there’s the security implications. While theoratically possible, are
all your scripts safe against whitespace|“‘`` injection? Piping objects is sooo
much easier AND it does not kill your system if the bytestream containsrm -rf /`.
Just my two cents…
Powershell Jedi
Unix is optimized around flat files windows is optimized around objects. It’s
not the they don’t understand Unix. The Powershell Team has a strong unix
Background. It’s just just that M$ has 87% market share and a trillion dollars
in the bank share that says F@ck unix.
Brandon Bloom
Piping objects has draw backs too. Here’s a case that came up on the internal
PowerShell mailing list…
Unix:
cat foo | wc -l
PowerShell:
Get-Content foo | Measure-Object -line
Ignore the line length, as I didn’t use any aliases for the PowerShell version.
The issue here is that Get-Content returns a String, not a StreamReader. The
result is that for large files, the entire String must be read into memory and
for really large files, swapped back out to disk!
You could easily work around this with objects; this example is just to show
that objects have drawbacks just as byte streams do. Everything is always a
balance. I like the Unix approach because it is a simple system with drawbacks
that are well understood. I dislike the object approach because it is a complex
system with less obvious drawbacks that don’t bite you until much later.
Len
Are you kidding? MS couldn’t even get the simplest thing right. Their
“compare-object” is supposed to be their “equivalent” to diff. Not so. Not
even close. Try comparing these two files:
file 1:
a
b
c
d
file 2:
d
c
b
a
Unix diff correctly shows how they are different. MS compare-objects says they
are the same. Why? Because the latter doesn’t care about order.