Plan for preliminary work

My project was accepted, and therefore its time to do some preliminary work before official coding time starts.

First and foremost project goal is to make PyJS to be able to translate and run itself.
Basically, this means that it should be possible to do this:
#> $PyJSRepo/pyv8/pyv8run.py --strict translator --strict -o /tmp/translator.js $PyJSRepo/pyjs/src/pyjs/translator.py
And it should:
1. Translate translator.py and all its dependencies
2. Load result in pyv8
3. Use loaded in pyv8 translated Translator to translate translator.py again
4. Result of (3) has to be completely identical to result of (1)
This goal has to be achieved for mid-term evaluation (July 12). 

In order to achieve this goal we have to make pyjamas support all 180 modules on which translator.py depends.

At this moment, most of them fail with some kind of error either during translation or execution. These errors are caused by one of 3 issues:
1) Lack of support for specific syntax, like a[::] (ternary slice, bugs #364, #434, #577, #582)
2) Lack of support of specific features in pyjslib, like type(lambda x:x).
3) Restrictions of environment, like weakref.py. Such modules cannot be supported directly, and we have to provide replacement with compatible API or remove dependency on it from other modules.

First and second issues could be resolved via either fixing pyjs/pyjslib or providing replacement module, which does not use this particular feature.

To help myself track progress and properly prioritize work, I'll make simple pyjamas web-app, which will show list of all modules Translator uses, and show results of translation/evaluation of each of them.

Therefore plan for next week is as follows:

1) Find/make tool to produce dependency graphs for translate.py, compiler.py and other base modules we would need.
2) Add simple compile/eval test suite for all modules from (1)
3) Make pyjamas-GAE web app to show results from (2)
4) Fix import path issues in pyv8* tools
5) Fix issues in master for pyv8run

This will be useful for tracking stdlib support as well and providing good overview of current pyjs compatibility with cpython.

Introduction

PyJS, which is part of Pyjamas package, is some very under-appreciated tool in Python world, and generally is not recognized as 'python implementation' and often not considered as separate from widget set.
This is partially due to lack of documentation and clear API for use-cases other than Pyjamas itself. Translator class has rigid templates, limiting its applications, and Linker/Builder are very pyjamas-centric, pulling in DOM/Widget libraries by default right now.

Moreover, right now translator lacks ability to compile itself, and do compilation in run-time independent of CPython (or other base implementation), making it look far less capable, than it actually is.

As I'm one of those who are interested in PyJS itself rather than whole Pyjamas suite, I felt that GSoC would be a great opportunity to fix these issues and bring some public attention to PyJS as stand-alone tool, capable of running arbitrary python on javascript VMs like Google V8 and SpiderMonkey, having potential to be even faster than CPython thanks to advanced JIT of these platforms.

While this may sound as big undertaking, most of needed infrastructure is already there - we have pure-python python parser, translator itself is pure python as well, and road to making it independent of CPython is straight.

Improving API and supporting statement/expression-level compilation is harder, as currently translator lacks globals()/locals() support, and implementing it isn't trivial, but definitely possible.

Another step would be to update and improve CLI utilities we ship, and include typical 'python' utility to make it familiar for non-pyjamas users. I've already worked on this part, but API update for translator/linker should come first. While I'm at this, I would also need to add tests for CLI tools themself, as breaking one of them and not noticing is too common right now.

Aside from confidence increase and improvement of public image, there would be enough direct benefits for Pyjamas as well:

• Ability to do compilation in run-time could let us send raw python code to the browsers, significantly reducing total javascript size (up to 10 times).
• Improvement and simplification of widgets development, bringing full power of python introspection and interactivity to browsers environment.

Another goal is to evaluate ability to support python 3 syntax for parser and translator, and ability to do initial compilation with python 3 interpreters.
While its too early to talk about full python 3 support for Pyjamas, as that would take too much time and would be too hard to maintain afterwards, parser and translator themselves are much more feasible target for GSoC.
Since we are translating python into not-so-pythonic javascript, its possible to translate most of python 3 into same javascript with (mostly) same pyjslib, with some prior AST transformations.
3to2 project achieved a lot in figuring what can be translated and what not, so first step here would be to study their findings and see what else could be resolved by python-to-javascript translation.
Most likely we could not support all of python 3 at once, but bottom line is to document all changes needed for full support, and keep python 3 support in mind while re-designing translator, so we don't end up dead-locked with python 2 later.