I'm trying out the hubbub parser and I gave it a test run on the google homepage's html, but it thinks a piece of javascript as the beginning script tag. I used the tokeniser.c in the test directory to test it out. Here's the relevant output: CHARACTERS: '&' CHARACTERS: 'd!=this;d=d.parentNode){var E=d,u,e=E;u=a;var v=e.__jsaction;if(!v){var v=e.__jsaction={},l=f;"getAttribute"in e' CHARACTERS: '&' CHARACTERS: '&' CHARACTERS: '(l=e.getAttribute("jsaction"));if(e=l)for(var e=e.split(p),l=0,n=e?e.length:0;l' START TAG: 'n;l++){var' attributes: 'j' = 'e[l];if(j){var' 'g' = 'j.indexOf(":"),m=-1!=g,w=m?j.substr(0,g).replace(/^\s+/,"").replace(/\s+$/,""):"click",j=m?j.substr(g+1).replace(/^\s+/,"").replace(/\s+$/,""):j;v[w]=j}}}if(u=v[u]){a={eventType:a,event:b,targetElement:t,action:u,actionElement:E};break' 'a}}a' = 'f}a&&(b.stopPropagation?b.stopPropagation():b.cancelBubble=!0,"A"==a.actionElement.tagName&&"click"==h&&(b.preventDefault?b.preventDefault():b.returnValue=!1),c.d?c.d(a):(t=a,b=(d=i.document)&&!d.createEvent&&d.createEventObject?d.createEventObject(b):b,t.event=b,c.c.push(a)))};var' 'g;g' = 'function(b){var' 'a' = 'h,c=n,d=!1;if(b.addEventListener){if("focus"==a||"blur"==a)d=!0;b.addEventListener(a,c,d)}else' 'if(b.attachevent){"focus"' = '=a?a="focusin":"blur"==a&&(a="focusout");var' Essentially there's some javascript in there that looks like this: for(var e=e.split(p),l=0,n=e?e.length:0;l<n;l++){var j=e[l];if(j){var g=j.indexOf(":"),m=-1!=g,w=m?j.substr(0,g).replace(/^\s+/,"").replace(/\s+$/,""):"click",j=m?j.substr(g+1).replace(/^\s+/,"").replace(/\s+$/,""):j;v[w]=j}}}if(u=v[u]){a={eventType:a,event:b,targetElement:t,action:u,actionElement:E};break And it thinks that this is the beginning of an html tag: <n;l++) -- it's a comparison statement inside the for-loop. Any clues where I might start looking to see why this might be?

On Wed, Oct 24, 2012 at 02:54:49AM -0700, Dean Mao wrote: > Here's a more compact test: > > <script>for(var i=0;i<n;i++);</script> > > Outputs: > > START TAG: 'script' > CHARACTERS: 'for(var i=0;i' > START TAG: 'n;i++);<' attributes: > 'script' = '' > > Essentially everything inside a <script> tag should be treated as > characters until a </script> tag is seen. Yes. This behaviour you're seeing is expected. The HTML5 tokeniser has a number of modes, which are selected by the token handler callback provided by the client. The trivial token handler in test/tokeniser.c does not manipulate the tokeniser mode, thus it does not handle the contents of script (and other, similar) elements in the expected fashion. The treebuilder implementation in Hubbub does manipulate the tokeniser mode in the correct way. In most cases, you'll want to use the built-in treebuilder, as it handles all the complexity of coping with junk input for you. See examples/libxml.c for a demonstration of how to use the built-in treebuilder. If you do only wish to use the tokeniser, then you need to ensure that your token handler changes the tokeniser mode in the same way that an HTML5 treebuilder would. J.

Btw, thanks for the help on this. The project is for a nodejs native extension that brings the love of libhubbub to the nodejs world: https://github.com/deanmao/node-hubbub There are other html parsers in the nodejs world, but none are as good as libhubbub. I considered using the parser from webkit or firefox, but libhubub was definitely the easiest to use since it was completely standalone and used very few external libraries.

Yeah sorry, I will submodule the stuff soon-ish, however there were a few things that can't be easily submoduled, like the entities.inc & aliases.inc which were generated from a perl script. I can't count on perl being on the system, or in the path. The build system is the nodejs standard, node-gyp so it's easier if makefiles are not used in this case. The user will simply do a "npm install hubbub" and node-gyp would download, compile, and install the binaries. What are your suggestions regarding some of these things?

You guys won't have to make any guarantees about compilation or runtime here as I'll take it on my own to get things working for users should they have questions, however I doubt this would be the case as there are very few nodejs developers compared to other languages, and it's pretty unlikely that many would use my library over the numerous html parsing libraries already available for nodejs. I would be very surprised if more than 20 developers use this library.