I first have to admit that I have not written a program longer than maybe 500 lines of code in a dynamic language. I am very interested in DynamicLanguages, though and Rainer’s comment on another post made me investigate how to develop large scale applications in DynamicLanguages:
In my eyes the main danger in programming lisp is the lack of variable typing. If you are not a disciplined programmer you will soon loose the control over your code.
I never thought about that seriously. Since I only hacked small scripts in Python, Javascript and PHP it never occurred to me that the lack of type declaration may pose a problem. Until now. In this entry I’d like to summarize my findings of the past week.
Possible shortcomings of dynamic typing
Thinking about the loose the control over your code-part of the comment I realized that type information is helpful in situations where that information supports your IDE or editor in certain tasks.
Advantages of having type information…
Just for completeness I will briefly mention other features of type information that can be realized in DynamicLanguages by other means (not type). The list was taken from OliverSteele’s article The Type Declaration Compromise:
- Documentation, type serves as some kind of documentation of the source.
- Runtime introspection, for connecting systems (COM) or interoperation with other ProgrammingLanguages.
- Compiler optimization, the old hat.
- Program verification, fetching type mismatches, typos (e.g. in method names)…
just to ensure you’re not trying to do something silly like take the square root of a hash table. [source]
… that dynamic languages also have
In DynamicLanguages these features can be realized in ways other than providing type declarations. Sometimes these ways even improve on type information. Most of the times they are technically more interesting.
- Optimization can be achieved via TypeInference , JustInTimeSpecialization or optional static typing. For examples see Nice/Boo, Psyco and Lisp.
- Verification is done by putting the application into a harness of UnitTests and the like. Some argue that because you have to write UnitTests regardless of whether you use a static or a dynamic language you could just as well write some more tests that check for type consistency in a dynamic environment.
- Documentation, giving examples of using functionality, can be combined with automatic verification (UnitTests), e.g. using the doctest-module in python. The code examples in PEP 246 (Object Adaptation) nicely display using that possibility.
Things that are difficult to support for IDEs in dynamic languages
What I really found lacking in IDEs for dynamic language is support for CodeCompletion, FindUses and ReFactoring:
to support CodeCompletion, FindUses (e.g. of a method) and ReFactoring.
- CodeCompletion, aka AutoCompletion: Let the IDE complete a method-, object-, class-, variable or whatever-name. Or providing information on (the number of) parameters a function-call expects.
- FindUses: Let the IDE show you where a certain attribute is used. This is somewhat useful in…
- ReFactoring: Changing the structure of code without altering its functionality. Renaming methods, extracting methods, extracting interfaces, etc..
Example
The following (python-)code examplifies the difficulty IDEs have in supporting these operations for dynamic languages:
def my_function(a, b):
a.append(b)
Without type declaration it is difficult (impossible) for an IDE to decide whether a actually supports the method append. So an IDE cannot provide CodeCompletion here. It may guess what methods are available, the user must choose then, though.
To find out the IDE may run the code in the background, watching for calls to my_function and extracting the types of the parameters from that call. But: Is that function accessible by automatically running the code? I can hardly imagine that such an approach would work for many places in your code, if you are writing large applications (and that’s what I am talking about here.).
For FindUses the IDE cannot decide whether append is the append you are looking for starting from the declaration in some class MyClass.
The same but the other way around will be the case if you decide to rename append to add. The IDE then would have to decide whether the append declared in MyClass is the same append and therefore will have to be changed.
Writing code you can try to clarify certain things. E.g. how about dropping some HungarianNotation:
def function(listA, stringB):
listA.append(stringB)
Now you could know what types the parameters of the function should have. But does your editor? And does your naming correctly reflect what’s going on?
Heck, someone must have had these problems before….
How Smalltalk supports refactoring
Why should ReFactoring not work with dynamic languages? I have never used Smalltalk but I found that Smalltalk was the first language to support ReFactoring. And Smalltalk is a dynamic language. So how does Smalltalk do it? Refactoring “support” for Ruby? cites A Refactoring Tool for Smalltalk and explains that Smalltalk uses method wrappers to trace calls of a renamed method: The method definition is replaced by a wrapper that reacts on calling the method, logging the call. In Smalltalk the wrapper then goes up the stack trace to replace the call to the renamed method with the new call. The first part – replacing the renamed method with a wrapper – is rather straightforward to implement with e.g. Python. __getattr__ may help here to catch calls to methods that are no longer implemented. The referenced article explains a way of doing this in Ruby.
Still, renaming in this manner only one direction is supported: From the definition to the calls. You cannot go the other way: Suppose you find some call to a method you’d like to rename. You’ll have to find the place where that method is defined. Suppose that name is not unique and you are into trouble.
The wrapper example is a good illustration of the dynamism of languages like Python. It’s hard to force that kind of MetaProgramming (adding/removing class/object-attributes at runtime) into types.
Optional Static Typing
Some languages support adding type information where needed. Those are mostly added after the code has stabilized and is no longer changed in wide ways. Then performance bottlenecks are spotted by profiling the code and fixed by adding type information so that the compiler can optimize. At least that is how OST is used in Lisp. Adding type information right from the beginning is considered PrematureOptimization and therefore to be avoided.
The article Re: Optional Static Typing by IanBicking is recommended reading on that topic. Speaking of adding type information to python-code (like done in Boo):
But we don’t need that level of static analysis to support source checkers (like PyChecker or PyLint) or to allow auto-completion in IDEs.
Instead of using type declarations it is put forward to use interfaces. In contrast to interfaces in languages like Java, where an interface has to be implemented by a type, interfaces in (here) Python would be used to express the fact that a object supports the methods and fields declared in the interface. The class of such an object would not have to declare the implementation of that interface, though. Whether an object provides the promised attributes will be determined at runtime.
The BDFL has put up a series of articles dealing with the subject of optional static typing in Python. The most recent one is Optional Static Typing — Stop the Flames!
Dynamic IDEs
Coming back to my initial question of how certain operations may be supported by IDEs for DynamicLanguages I started to look for such an IDE, especially for Python. None of the ones I found support CodeCompletion, FindUses and ReFactoring in the strict way mentioned above. Some support these operations in a less strict way, though. It seems the following IDEs can be recommended:
- Wing IDE
- SPE, Stani’s Python Editor
- PyDev, plugin for Eclipse. Uses BicycleRepairMan for refactoring.
The article A review of 6 Python IDEs briefly mentions what can be expected from sophisticated IDEs for dynamic languages:
Wing is the only IDE here to give you code completion for more than function and class variables; it also completes keywords, modules (in import statements), and locals. This is more useful than it sounds;
Note that SPE was not included in the review of 6 IDEs and seems to support these features, too. PyDev is heavily developed. Version 1.0 was releases February 6th.
Though these IDEs may not be as sophisticated as my favorite – Eclipse – there are…
Awful IDEs for static languages
There are IDEs that don’t knock off the easy task of supporting the developer in static languages. I am thinking of Visual Studio 2003 here:
- no refactoring (no method-names, no variables)
- incomplete CodeCompletion (almost never working correctly)
I was quite amazed that an IDE like VS used by so many developers could be such crap. I had to buy (or rather, make by boss buy) a plugin to get rid of the most disturbing bugs and shortcomings of VS2003. So, there are quite some developers out there that are used to not using the functionality I am in search of for DynamicLanguages. If VS2003 does not keep developers from writing large scale applications, DynamicLanguages certainly will not either:
Large Applications implemented in dynamic languages
Several large applications in dynamic languages (here: Python) exist:
- Feeding a Large-scale Physics Application to Python
- Google was initially written in large parts in Python
- Zope
- PyPy – a Python-interpreter written in Python, compiling to C and doing other magic. It currently consists of more than 130000 lines of code. See their Status of the implementation.
- Python Success Stories lists an impressive list of other projects written in Python.
- Lisp Success Stories does the same for Lisp.
- Developing large scale applications in Python, a talk held on EuroPython in 2005 by Marc-André Lemburg.
Since I decided to dive into PyPy’s source code, I plan to do so with different IDEs, comparing them. So most likely this entry will be continued…
Post a Comment