What features are critical to the editor (wysiwyg or otherwise)

A thread level editor should have the following:

• Bold
• Italics
• Create a Link
  
• Unordered Bullets (used right here)
• Quote (indents post/snippet of text)
• SpellCheck (many libraries exist for this)
• Emoticons
• Image Uploader/Linker
• Paste flash embed code
• HTML view
• Clear Formatting (for when things get fucked up)

An inline reply doesn't need any buttons, but should recognize when an image, embed code, or link is pasted in.  (i'd assume that both the inline reply and thread level editor would auto convert any http://, www phrases into links.)

I think all it needs to be is out of the way. That is, not hinder your ability to make a quick post. With that said it should help you along the way. Definitely posting media needs to be made as simple as pasting a link. Font formatting I see as being pretty much entirely useless, but it also doesn't really hinder anything.

However, the problem seems to be that there's no real middle ground as far as the technology is concerened.

I agree..

the inline reply should be as minimal and smart as possible, to the point of -looking- trivial, but be safe to paste media links and pre-formatted text. This is what I believe you do most of the time, and one of the most common but annoying problems with forums and comment systems is all the quirks with formatting, like something triggering bold or smileys and replacing your text. So inline reply should affect your post in the minimal way.

As far as the wysiwyg editor, these would be mostly used by people starting their own thread, so in big part by forum owners and contributors, and there has to be a way to give them more control, so they don't end up adding formatting in variable width text and so on.

A thread level editor will be triggered <1% times compared to the rest of the posts, so if it's more complicated and/or slow, I don't think this is as big of a deal as long as it doesn't prevent people from posting altogether.