I was just reading an article about one of the G-20 arrests that seems relevant. In part, the article reported:

Police testified Puddy had a 15-centimetre knife attached to his belt.

Puddy, 32, from London, Ont., was eventually charged with obstructing police, concealing a weapon and possession of a prohibited weapon.
...
Justice Melvyn Green accepted Puddy's defence that he was not carrying the knife as a weapon and further ruled that since the arrest was illegal, so too was the search that uncovered the knife.

This seems to confirm a couple things.

First, having a knife is not the same as having a weapon; a knife only becomes a weapon if you intend to use it as such. This is likely a good thing about the way the law is actually worded. While the police may sometimes use terms like "legitimate purpose" with respect to knives, the law simply says you can't carry one with the intention of using it as a weapon.

Second, as dougwalkabout said, there must be a valid reason for a search to be considered legal.