Poo-tee-weet

Making PHP 5.3 happen

Lukas Kahwe Smith — Thu, 03 Jul 2008 13:25:24 +0200

So I have now joined Johannes as a co-RM for PHP 5.3. Its quite an honor and a challenge. Especially since I still need to find enough OSS time to continue to work on emPHPower and write out a proper proposal for a "constitution" (expect a mailinglist to be setup soon). We hope together we have enough brain cycles to push put what is probably the biggest minor release in the history of PHP. Just take a look at the todo list and the scratchpad detailing all the additions. Speaking of the scratchpad, we really really really need more people helping to identify the changes in PHP 5.3.

Scavenge the NEWS file, test your apps against current 5.3 CVS. Do whatever it takes to find all the places where PHP 5.3 adds functionality and more importantly where it breaks existing functionality. Let the php-doc team know your findings, post comments on this blog, send me an email or write them directly into the wiki page. If you have a cvs.php.net account, you are already setup. If not then just register yourself and I will grant the necessary permissions. Please please .. help out in this critical effort. Together we can make sure that everybody has all the information to make a painless transition to this feature filed release.

PostgreSQL is already there!

Lukas Kahwe Smith — Fri, 27 Jun 2008 09:56:35 +0200

So I took my concerns over prepared statements to the #postgresql IRC channel on freenode. I pointed out that I think there should be a way to get server side handling of placeholders in statements but without the additional overhead of a second round trip or the draw backs of overly generic query plans due to not being able to use the parameters in the planning stage. Some people have noted that this feature is available in MSSQL. It also seems to be available in PostgreSQL in the form of PQexecParams and its even exposed in ext/pgsql, though it's only used for sequence reading in PDO_PGSQL. It would be really cool if it could be used when emulating prepared statements (probably with a PostgreSQL PDO attribute).

As you may know PostgreSQL uses $[name] to denote placeholders, while Oracle uses :[name] and MySQL uses the ? syntax that I think ODBC used first. Again there is no clear standard for this. I prefer the approach with a name. It makes for more readable code and makes it possible to use the same parameter multiple times in a statement. However as more and more libraries are build on top of PDO, even if I only want to ever work with one RDBMS, when using these libraries, I currently have no choice but to let PDO parse and decide on its own if to rewrite the query. So it would be nice if PDO could also have a mode where it does not attempt to parse and rewrite placeholders at all.

UPDATE [27/06/2008]:
Expanded the request to have PDO provide a mode to just pass through the prepared statement.

Musings on ordered lists inside RDBMS (part II)

Lukas Kahwe Smith — Wed, 25 Jun 2008 16:20:04 +0200

So I took Roland's comment and tried to get it integrated into my code. For the tab management it worked well, but for portlet management it was a lot harder. Actually I only have a partial implementation finished. What's missing is the necessary logic to be able to move a portlet in the same tab from one column to another (there are 3 columns a portlet can be in for each tab). The thing that was most important to me was cleaning up the pruning operation. This took some trickery (aka hackery). I think its a better implementation but it does make me a bit nervous. Of course its all still very MySQL only.

Anyways so here goes the final query for moving a tab:

UPDATE user_tabs ut
    JOIN user_tabs ut2
        ON (ut.user_id = ut2.user_id
            AND ut2.id = :id
            AND ut2.pos != :pos
            AND ut.pos >= LEAST(ut2.pos, :pos)
            AND ut.pos <= GREATEST(ut2.pos, :pos)
        )
    SET ut.pos = CASE
        WHEN ut.id = :id THEN :pos
        WHEN ut2.pos > :pos THEN ut.pos + 1
        WHEN ut2.pos < :pos THEN ut.pos - 1
    END
    WHERE ut.user_id = :user_id;

Like I said the pruning operation took some hackery. What I do now is essentially redoing the entire positions for the tabs and all portlets for the given user. If a portlet is marked as to be removed, I update the position to NULL. This way I can easily remove the processed removed tabs and portlets afterwards. The tricky part was in updating the positions for tabs. I update all portlets in the order of the tabs. Of course whenever I start processing the next tab, I need to reinitialize the position counters for the columns. Since I am not aware of any way to do this directly, I created a fake condition that actually just ensures that the counters are reinitialized if the tab_id has changed since the last processed row.

Also note that I have to do two separate UPDATE's, because MySQL does not support JOIN's in UPDATE's together with an ORDER BY. Since I cannot work without the ORDER BY, I had to split things in two queries. This also means that I need to SELECT the tab_id's for the current user, so that I can filter on them in the UPDATE of the portlets, where I do not have a column with the user_id.

$tab_ids = SELECT id FROM user_tabs WHERE user_id = :user_id;

SET @pos := -1;
UPDATE user_tabs
    SET pos =
    (
    CASE
        WHEN is_removed = 1 THEN NULL
        ELSE (@pos := @pos + 1)
        END
    )
    WHERE id IN ($tab_ids) ORDER BY id, pos;

SET @tab_id := 0;
UPDATE user_tab_portlets
    SET pos =
    (
    CASE

        WHEN (tab_id = @tab_id OR ((@tab_id := tab_id) > 0 AND (@poscolleft := -1) >= -1 AND (@poscolmiddle := -1) >= -1 AND (@poscolright := -1) >= -1)) THEN

            CASE
                WHEN is_removed = 1 THEN NULL
                WHEN col = :colleft THEN (@poscolleft := @poscolleft + 1)
                WHEN col = :colmiddle THEN (@poscolmiddle := @poscolmiddle + 1)
                ELSE (@poscolright := @poscolright + 1)
                END

    END
    )
    WHERE tab_id IN ($tab_ids) ORDER BY tab_id, pos;

DELETE ut, utp
    FROM user_tabs ut LEFT JOIN user_tab_portlets utp
        ON (ut.id = utp.tab_id)
    WHERE (ut.pos IS NULL OR (utp.id AND utp.pos IS NULL)) AND ut.user_id = :user_id;

Thanks again Roland for your feedback. Lets see if I find time to also rewrite the other queries.

Bootstrapping emPHPower

Lukas Kahwe Smith — Tue, 24 Jun 2008 23:58:37 +0200

The following is more or less just a thought stream dump from my train ride back from frisbee practice. I just wanted to jump start a discussion on how emPHPower should be organized. I think the general idea of establishing a communication hub for core developers, end users and companies has been received with sufficient enthusiasm to start working on some of the boring details. The next logical step seems to be to start to fleshing out how the organization behind this communication hub should be structured. I want to leave as much power as possible with the individual members, which is one of the main reasons why I do not want companies to be come members. When planning all of this I also need to differentiate the "bootstrapping process" and my vision for how things should be afterwards. Actually I will focus on the bootstrapping process, since I think for the most part the members should have the opportunity to decide on the structure later on.

So initially there has to be a core group of people that dedicate a lot (and when I say a lot, then I really mean a lot) of their time to make this thing fly. This means working hard at getting the PHP world at large to hear and take note of this organization. Also it means building up the necessary infrastructure so that we can have the level of bottom up participation that is key to ensure that this organization is an enabler and not a centralized power. This means to me that we will probably need a dozen people in various roles (probably half technical and the other half evengelists, though there will be people working on both sides) and about 6 months of development time.

The necessary infrastructure to really enable a basic level of collaboration on a basic level should be in place within 3 months (probably the ability to quickly setup mailinglists, wikis and news feeds). After this point has been achieved we should try to get as many people to tentatively sign up as possible. I think for the initial period there will be no membership fee (or a really low fee at the most) to sign up, with the option of giving an anonymous donation. I am also open to code contributions to setup the infrastructure from anyone provided that the code is available under a very liberal open source license (and of course meets our needs and expectations in terms of quality).

During this time all members have the opportunity to collaboratively work on hopefully multiple proposals for our "constitution". The process will probably be more or less a free for all with the bootstrapping team serving as moderators if necessary (hopefully we will not have to be referees). After 3 more months we should be ready from a technological level to claim that we are a technology hub. That being said I envision that some parts in the "constitution" may need to be implemented as part of the infrastructure as well (for example certain voting mechanisms, or how articles make it to the frontpage or into the newsletter). At this point the bootstrapping team will try to find a good point in time over the course of the next 3 months to freeze the proposals and hold a vote on the actual constitution. We will probably need to define how long the voting should be allowed (I think 2 weeks would be ideal), but this can be decided depending on how many proposals there are and more importantly how many people have signed up.

I envision there could be a lot of people signed up at this point already. If there are less than 1000 I think I would not even bother calling for a vote. I would stop the project and accept failure of the attempt. I am hoping to have about 10000 people signed up. To become eligible to vote one would have to pay the fee for the first 6 months. So todays estimates this would be 25 Euros for someone living in western Europe. The fee would be scaled accordingly by the average income in the home country. We might adjust this amount to match a trend we are seeing in the constibution proposals that are up for vote. Until a constitution has been accepted the bootstrapping team will effectively work as benevolent despotes. We are the people that put in our heart to making this stuff work for the greater good. During the 6 months or more before the initial vote and having to pay a first membership fee, we will either have earned the trust of the community or we will have failed.

Each constitution should provide a method to manage the next 6 months in order to move from the bootstrapping process to the final structure, holding any further elections necessary as defined by the constitution. But the second the constitution is accepted, all paying members will immediately get whatever rights the constitution gives them. So I have said at the beginning of this blog post that I will mostly discuss the bootstrapping phase, but I do of course have some ideas of what kind of structure I would hope for later on:

there would be a team of people (eventually at least partially paid) that manage the infrastructure
there would be a board of about 7 people that manages the day to day business (this would include defining the infrastructure team) as well as proposing a yearly budget
the board is elected by the members for a term of one year
the board has to provide a monthly run down of what they have done and what they plan to do in the next month
any decision that has a large impact (for example the yearly budget etc.) will need to announced in time, so that any community member can get enough people to sign to hold a referendum on the decision
any board decision can be overturned by a referendum
board members can be replaced with someone else via a referendum

As I said before the constitution might also want to regulate things like how certain groups of members can use the organization to exert influence. For example getting a wiki namespace or a mailinglist could just be done on a per request basis. Getting a top level navigation point on the front page (for example for phpwomen) might require a certain number of members that support this request. Same for how a news item makes it to the frontpage. I also think we will probably want a monthly newsletter. Again there would need to be some definition of who controls what goes into the newsletter. Maybe we will need additional positions that need to be voted for?

Focusing on the bootstrapping part has also enabled me to sidestep some of the tricky topics entirely. One of the top issues on that list would be the question of having closed lists to enable discussion between people that feel they are not ready to talk in public just yet. Most likely some companies could be interested in such lists. But in the bootstrapping phase we will have no use for closed lists and I will leave it to the constitutions to wonder about this one. Based on feedback from people I have become convinced that it would be best to never provide closed lists. Companies should find enough documentation on how to interact with the PHP community and what to expect on the emPHPower website, so that they can already make a solid decision on if they should get in contact or not. Making it too easy to go to closed lists will only harm our ultimate goal: To have all discussions out in the open.

Again I envision that quite a few points in the constitution needs to be hardcoded into our infrastructure. We also need to make sure that like in any democracy, minorities get sufficient rights. Especially since "emPHPower is a mediator and catalyst that empowers members of the community to follow their own ideas". So there is no problem with sending mixed messages to the outside or inside, as long as the noise of the message remains proportional to the numbers of members supporting the given message.

Update [25/06/2008]: I just noticed I omitted one important topic. How should the role of PHP core developers be within the organization. Initially I suggested for example that they all would be members without paying the membership fee. Some people have said that these people should instead be the early adopters that show with their commitment to paying the membership fees, that this is a viable proposition. Felix (Testfest role model and newest member in the PHP QA team) suggested that maybe the best is to give the membership for free but optionally invite them to pay nonetheless. More tricky however is the question of how PHP core gets to setup polls in the community. They will likely have to be able to side step the normal "limits" in number of supporting members to create a community wide poll for example.

Prepared statements are dead, long live prepared statements?

Lukas Kahwe Smith — Sun, 22 Jun 2008 15:20:39 +0200

So everybody and their dog hopefully knows about SQL injection attacks these days. Most people should have also heard someone telling them that using prepared statements is the magic super fix to all of these issues. People slightly more in the know will have read that prepared statements lead to all sorts of issues. Some of which can be fixed with hacks (or eventually at the source). Some of which can only be solved of the source also exist of course. Some others can only be fixed with certain assumptions (like using the first set of data for generating the query plan), which might break the original use of prepared statements.

Original use? I guess their original intended use had nothing to do with SQL injection attacks. I can only guess since I was not doing database when they were invented. Heck maybe I was not even born back then. My guess is that prepared statements were invented back when people had long running connections and fairly static data that at most grow slowly over time. So it did not matter to them that their query plans would remain static with prepared statements. That additional round trip to the database to prepare the query probably also did not bother them, since they only did that round trip once in a few hours (or maybe even days?). Data entered the system mostly from known friendly people. Prepared statements were good and users were happy.

Well today everybody with a clue knows that data moves fast. That applications servers are a sure fire way to make your life miserable since every little stupid mistake can bring your entire happy system down, requiring hours to rebuild your caches before operation is back to normal. Shared nothing architectures rule the web world. So connections have a short life time. Even if we ever get connection pooling, we will get worried about connections going haywire in subtle ways as they are rarely created from scratch again (the same reason why application servers are a bad idea comes to haunt us again with prepared statements). So it seems like all the reasons why prepared statements were created are no longer relevant.

Oddly enough they are more popular today than a few years ago, when people were still learning the lesson that application servers suck. The reason is that contrary to back then, today most users are potentially evil, or at least stupid enough to accidentally be evil. Or whatever .. at any rate we want to protect against SQL injection or breaking queries because the user entered a single quote where he was not supposed to. And we have learned the lesson that making it easy to code defensively is the name of the game. So we began to advocate prepared statements again. And the masses flocked to them.

However we were only advocating prepared statements because it seemed like the best tool for the job. But we realized our faultiness, so we added buggy (some more than others) client side prepared statement emulation. But the database guys did not go back to the drawing board to solve the real issue: To have some way to send the SQL statement separated from the data in a single roundtrip.

So how about it? How about inventing a syntax that will do this for us? I am sure the guys on the board for SQL2008 (does anyone even know what they are doing .. I would not even know where to link to .. neither does wikipedia) have much more useless things to cook up, so I think it will be left to the more ad hoc OSS style (LIMIT, REPLACE, ON DUPLICATE KEY UPDATE, ENUM) of extending the SQL standard. Well lets hope that this time we can maybe do this together in order to ensure that we do not end up with a situation like with LIMIT where we had some tiny weeny syntax differences in the different OSS databases. So far MySQL has been the company to drive these kinds of extensions, but my hope is that maybe with the OSDB consortium (maybe one day more than "Just a web page") this could be done at a more universal level?