Yesterday, the City of Portland released the results of Uptown Services’ analysis of the Unwire PDX project, which is planned to provide 95 percent of the city with wireless internet access.

From the looks of the executive summary—the only thing released—MetroFi (the private company providing the service) passed; the city’s Bureau of Technology Services signed a certificate of acceptance allowing the company to expand beyond the downtown/central city area.

Uptown’s executive report and the acceptance letter can be found here.

But members of Portland’s geekocracy are still skeptical—last month, I wrote about the Personal Telco meeting in which an independent review of MetroFi’s project was less than glowing. Now, one of Personal Telco’s volunteers and an employee at a local ISP, Michael Weinberg, has written a letter to Logan Kleier, the city’s Unwire PDX manager, expressing doubts over the acceptance of MetroFi’s service.

Weinberg’s full letter is after the jump, but what it boils down to is that the executive summary doesn’t provide enough information to justify issuing the certificate of acceptance. Plus, the executive summary appears to say that the service only really worked with one user connected—hardly a ringing endorsement.

Check out more from UnwirePDX-Watch.org, written by PTP volunteers who ran their own evaluation. I’ve got a call in to Kleier, but haven’t yet heard back.

Dear Logan,

I'm writing to express my concern over the City's recent sign-off on the MetroFi Proof of Concept deployment. Based on the only publicly released document from Uptown, there is little reason for the public to have faith in the City's decision to issue the Certificate of Acceptance.

As someone who has worked with outdoor wireless network design and deployment for many years, initially as a volunteer with Personal Telco and later as the administrator for that group's first grant, a wireless network along Mississippi Avenue; as a private contractor deploying networks between buildings in downtown Portland and most recently at Stephouse Networks, an ISP operating a number of wireless services in downtown Portland, St. Johns and the N. Marine Dr. area, I believe I can speak with some authority about the issues facing outdoor, urban wireless deployment, particularly those delivering service over the 2.4GHz band.

Throughput:

The Uptown Services Executive Summary, the only document available to the public from Uptown, begins by stating that 69 of 70 tested access points were shown to be capable of passing downstream traffic at 1Mbps and 70 of 70 were able to pass upstream traffic at 256 kbps. The opening figure for bandwidth capability is not indicative of performance over the 802.11g tier, where users experience it, but rather is an assessment of the performance of the 802.11a backhaul. Any client capable of connecting to an 802.11g AP ought to see these speeds between the client and AP.

Uptown goes on to state that their testing indicated that this level of performance was supported for 2 downstream clients, but only 1 upstream client, suggesting that the advertised performance is only available when a single user is on each AP. In short, the throughput assessment, as described in the Executive Summary, appears to ignore the presumable average load on the network, and instead presents a standard for success that virtually any functioning network should be able to meet.

Availability:

As many formal and informal usage tests have shown, when a reliable connection to the MetroFi network is found, it tends to remain reliable, as such, the 91% figure, as presented, is neither surprising nor impressive. It is entirely conceivable that a single device, placed in a location that had reliable coverage and connected continuously to the network, would have service availability of 91% or more. However, a single device connected in a location chosen for network reliability, is neither a rigorous nor a scientific means of testing average network performance.

The Uptown "drive testing", like the throughput testing before it, presents success criteria that seem to ignore real-world use to the point of being meaningless. The ability of a single user to achieve the advertised performance is of little value if performance degrades with the second user on any given AP.

Of all the availability results, the most unsettling is the -79dBm figure, upon which network coverage has been judged. As anyone familiar with wireless networking can attest, a single signal strength reading is absolutely meaningless. All that this assessment means is that the MetroFi network can be "heard" at a certain level in 95% of the coverage area; it provides no insight whatsoever into connection availability. This is especially troubling, since MetroFi has repeatedly blamed end-users (and their hardware) for the inability to connect to the network. The inclusion of this statistic, is irresponsible and misleading without additional information about the strength or other technical assessment of the receiving equipment.

I can only presume that prior to issuing the Certificate of Acceptance, you received a more in-depth report from Uptown Services that clarifies some or all of my concerns regarding the testing methodology. It would behoove the City to release all or part of that report to demonstrate that the decision to sign-off on the MetroFi PoC was based on appropriate data and strong testing methodologies. The Executive Summary raises serious questions about this, especially in light of many assessments from frustrated users and the excellent research by Russell Senior and Caleb Phillips which suggest that usable coverage and performance are highly variable and MetroFi's own statements regarding the probable need for additional hardware in order to connect.

Thank you for your time,

Michael Weinberg