Poo-tee-weet

XPath expert needed

Lukas Kahwe Smith — Fri, 24 May 2013 20:27:07 +0200

In the PHPCR implementation using Doctrine DBAL we support search queries by converting the SQL2/QOM statements into XPath queries that we run on the XML stored in an RDBMS. Sounds insane, yes .. but it works pretty well .. obviously will not scale very well .. but it works for smaller data sets and there will be ways to improve performance later. In terms of functionality we have everything working quite well including JOIN support that was added recently.

However we are stuck with multivalue queries. What makes it particularly tricky is that we do not know which properties are multivalue and which are not as illustrated by these tests.

As you can see in the test we have a multivalue property "tags" and in the SQL2 statement we want to test if that multivalue property contains both "foo" and "bar". However inside the query engine we do not know that this property is multivalued or not.

The relevant code is in QOMWalker:: sqlXpathValueExists() and QOMWalker:: sqlXpathExtractValue(). As you can see it currently only looks at "sv:value[1]".

You can find some more information here:
https://github.com/jackalope/jackalope-doctrine-dbal/issues/88

What is needed to REST in Symfony2

Lukas Kahwe Smith — Sun, 05 May 2013 22:40:47 +0200

I think we already have quite a nice toolchain for REST in Symfony2 with Bundles like FOSRestBundle, JMSSerializerBundle, NelmioApiDocBundle, FSCHateoasBundle and HautelookTemplatedUriBundle. What is great about these Bundles is that they are all nicely integrated with each other. But there are still some limitations which should be addressed. The following is not an exhaustive unordered list and also not detailed enough but just something to illustrate where I feel there is still work left:

1) Content type negotiation inside the Routing layer
2) Dynamic content in NelmioApiDocBundle and/or some way to integrate with JMSSerializer exclusion strategies (f.e. so that I can version my API docs together with my API)
3) Dynamic content in FSCHateoasBundle like being able to add Request data into URL patterns or being able to set URLs based on model data etc.
4) Behat Context for testing REST APIs (specifically Context for JSON/XML "assertions)
5) Integrate Behat test reports with NelmioApiDocBundle
6) Wrap up JSONP support
7) Fix open FOSRestBundle issues, especially edge cases in the Routing loader
8) Provide a simple solution to offer token based authentication including API throttling
9) Integrate with the SensioGeneratorBundle (and other RAD solutions like KnpRadBundle)
10) Better support for XML responses in NelmioApiDocBundle when using the "output" support together with JMSSerializer (right now it basically shows you can array like structure which makes sense for JSON only)
11) ..

What I would like is for us to work on creating a prioritized product backlog of all of these features and then do a kickstarter project (anyone used catincan.com yet?) to get funding for this. Ideal candidates for implementing this would imho be Adrien and William but there would be of course also others that could step up here. I imagine with budget for about 3 man months we could provide and incredible experience for the Symfony2 community. If we get 100 companies/people to donate 100 Euros each we would have a very good funding basis for this. I know bringing money into open source is tricky but there are lots of people who we know and trust to produce awesome stuff if they get some time to focus on open source. WDYT?

To put a bit of money where my mouth is .. I would donate 250 Euros for this out of my own pocket ..

Small update (30-05-2013):
There was lack luster interest in such a sponsoring initiative.

3) see https://github.com/TheFootballSocialClub/FSCHateoasBundle/issues/52 and https://github.com/TheFootballSocialClub/FSCHateoasBundle/issues/53
4) see https://github.com/sanpii/behatch-contexts
8) maybe integrate with http://code.google.com/p/varnish-apikey/

Good design is no excuse for wasting time

Lukas Kahwe Smith — Wed, 27 Mar 2013 19:37:36 +0100

Symfony 1.x I would put into a category of frameworks focused on RAD, aka rapid application development. There are many other frameworks I would put into this category like CakePHP, Yii, CodeIgniter. All of these frameworks have their roots in the pre 5.3 days of PHP. I say this mostly because there has been a considerable shift in development methodology since then towards more decoupling and application of design patterns. As such these older RAD frameworks to me cut corners to be able to focus on the use cases they wanted to focus on. Now this is a quite legitimate approach, we all know that in order to make a solution truly generic it takes a considerable increase in effort and complexity. So focusing on 80% of what web developers need to do on a daily basis makes sense. This is mostly achieved by providing either base classes or the ability to use configuration files and conventions to reduce the amount of code needed to achieve functionality. However these frameworks tend to become quite painful when dealing with those other 20%. At that point they can actually become a hinderance and so you start to loose some of the benefits you gained for the 80% you worked on before. With Symfony2 we took a different approach. I would not call it a rapid application development framework. All it really does is add some more high level abstractions but it stays short of providing an out of the box solution. You still need to write the code and that code tend to be very explicit.

So for those people who were happy focusing on the 80% use case Symfony2 is a step back. Suddenly the same features take longer to implement, take longer to modify later on and on top of that the learning curve is steeper. I think the people that most appreciate Symfony2 today are therefore people that keep themselves busy with projects that mostly deal with the 20%. These people are in fact more productive with Symfony2 than they were with any of the RAD options I mentioned above. For these people the learning curve was also not that steep, because the type of work they were doing already required a deeper understanding of architecture and design patterns.

Now even those mostly dealing in the realm of 20% will spend at least some of their time with repetitive tasks like whipping up a few CRUD forms, ie. exactly the type of stuff the RAD frameworks cover pretty well. Furthermore just because Symfony2 is verbose at its core, doesn't mean that one cannot add RAD layers on top. The nice thing with a RAD layer on top of such a powerful framework like Symfony2 is that if you reach the end of what the conventions cover you can drop down one layer and keep going without hitting a brick wall as one tends to hit with a RAD framework like symfony 1.x. There have been some initiatives in this direction. For example FOSRestBundle provides a solution to generate routes from Controller method names. SensioFrameworkExtraBundle provides similar capabilities powered by Annotations and helps with mapping request data into model instances. SonataAdminBundle can do a lot of the heavy lifting when dealing with CRUD forms. KnpRadBundle is one of the few approaches that tries to really provide a comprehensive solution for RAD development. In general I was hoping that now almost 2 years after the first stable release we would be much further. That the solutions would be much better polished, mature, well documented and widely in use. So lets stop wasting time and lets get to work on providing a proper RAD tools chain!

Where have all the core devs gone?

Lukas Kahwe Smith — Wed, 27 Mar 2013 19:22:10 +0100

Symfony2 is without a doubt a very lively open source project. No small part thanks to Github, there is a flurry of tickets and more importantly pull requests coming in daily with dozens merged every week. These changes cover everything from small typos, to bigger refactorings, new test cases, documentation, performance optimizations. What even more positively stunning is that while there are a bunch of "usual suspects" the bulk of all of this comes from people that just submit the occasional change. Which means we have managed to make the community very inviting for new contributors. Something many other open source projects struggle with. So in a way one could say all is well. However I think its also important to define a direction for a project as a whole. I guess at a high level we are fine there too. We rarely have fundamental issues with determining what should go into Symfony2 and what should not. But what I feel is severely lacking is a prioritization and focus what should be happening within the next few months.

Back when we were preparing the very first stable release, there were several developers that knew the bulk of the Symfony2 code well and that were actively participating in discussing most of the proposed changes. They were also the ones who actually proposed the bulk of the changes. Due to their general overview of the code base even bigger changes tended to more quickly turn into mergeable change sets. Of course the fact that we could more liberally break backwards compatibility was also helpful. However it seems like many of these people have now shifted their focus to other areas. I, for example, am devoting most of my open source time to the Symfony2 CMF initiative and what time I have left I use to work on FOSRestBundle and the various Liip Bundles. Jordi is busy with Composer, while Johannes is busy with scrutinizer-ci.com and the serializer, Konstantin with Behat. People like Benjamin, Bulat, Jon and Kris also seem less visible. Even Stof seems to have cut down a bit, though he remains one of the more visible developers from back then. Additionally we had weekly (or was it bi-weekly) IRC meetings and a much more active developer mailing list. Heck the mailing list is seeing so little traffic and good questions and proposals often remain unanswered for days that we even considered killing the mailing list entirely.

The end result is to me that right now I have no clue how things make it or not make it into the next release. Fabien is tasked with going through all these pull requests of varying quality. This obviously takes a lot of time. There are people that are helping with the review, but if the PR is even slightly more involved it simply takes someone that understands the entire code base to make a call on if its ready or not. At the same time there is no way to know about Fabien's prioritizing. For example there is an open PR for adding default CLI events. It was almost ready for 2.1, it was definitely ready for 2.2 but its still not merged. Will it be merged before 2.3? Similarly we have gotten worse and worse and pooling resources to finish more complex pull requests. The resource watcher pull requests has been there for ages too. The issue have long been identified yet nobody is picking up the ball.

In a way we have moved from an onion structure of Fabien at the core and a group of 10 developers as a layer around Fabien and a hundred additional contributors around those 10, to a community with Fabien still at the core but with 500+ contributors directly around him. Again in many ways this is great because we have managed to prevent the creation of old boys clubs that put themselves above other contributors. But at the same time we have failed in networking all of the contributors into smaller units that can coordinate working and thereby pushing specific topics. And the end result is that such more complicated things fall into the lap of Fabien who remains a miracle in productivity, but even he has limits.

Now I have proposed some steps to solve this. Namely I believe there should be a release manager, similar to PHP core, that maintains a list of mid to large topics we want to focus on for the next release, that reminds people of the release process etc. This person should try to actively network developers to ensure that those topics get the attention they need to actually get done. Furthermore I think we should go back to at least bi-weekly IRC meetings to realign ourselves on focus topics. I am of course open to other suggestions. It could also be that I am just realizing that as I do not read all pull requests and tickets anymore that I have simply dropped out of that first layer around Fabien and that other people have now taken this role and that these people do indeed know what the focus areas are for the next release..

Update: I wrote this post last week and send it to Fabien in advance. He does not agree with all of my points but I will let him answer himself. Well he did already answer by reviewing the CLI events PR and figuring out a much better solution, writing it and getting it merged. Fabien-style!

On predictable PHP release cycles

Lukas Kahwe Smith — Thu, 28 Feb 2013 20:52:38 +0100

There is currently a vote going on to include Zends Optimizer+ opcode cache into PHP core. I am very happy with finally adding an opcode cache to the core distribution if only because it then forces to actually ensure that there is a working opcode cache together with every new release. What troubles me though is that its being proposed very late in the game for PHP 5.5, therefore causing a likely delay of 5.5 of at least about 2 months in the best case scenario if it were included. The other option of including it in 5.6 does not seem to be as popular at this point. This saddens me quite a bit since I believe that predictable release cycles would carry several advantages. First and foremost it will ensure that developers that propose a new feature will have a better idea of when their feature will be part of a release. This can encourage developers to actually bother with sharing code and it also prevents developers from getting disgruntled if their feature is deemed not yet ready for the upcoming release or even more importantly prevents half baked solutions from being pushed in just to prevent an unknown further wait time. Furthermore it makes it easier for PHP end users, especially larger frameworks and application maintainers, to plan their releases. Finally it poses the chance that distributions will better schedule the freeze time for PHP versions, rather than the current rather random choice they tend to do.

Now people (ok Zeev in particular) are throwing around numbers in regards to adoption rates. They are talking about how quickly new features are adopted by frameworks etc. However none of the presented data holds much use for this argument. We have never had predictable releases over a long period of time. So right now we are still not predictable. Furthermore it takes time for these frameworks/applications/distributions, even if they have the trust that PHP actually follows its release cycles, to actually start getting used to this and leveraging this in their planning.

The reality is that its quite easy to find a new must have feature for every release that makes it worth in the short term to delay the release a little bit, thereby getting a feature a feature 10-11 months earlier. But if we do this even every 2nd or 3rd release, we loose the benefit of a predictable release cycle. What is even stranger for this case is that we are just talking about an extension here. Its not a language feature, there is no engine level integration. So even if its not added to core, people can easily get Optimizer+ via PECL. So in this case we are not talking about people having to wait another 10-11 months. Don't get me wrong I think getting an opcode cache into core is awesome, but the reality is that shared host users will probably still not have access to it (correct me if Optmizer+ is more shared host friendly than APC) and the rest can still get it, albeit with a bit more effort. So I really do not understand the rush here ..

Update: Anthony makes some additional very good arguments why there is no reason to delay.