HTML parser java library, jericho

For an R&D project I needed something that would help in scraping our existing web pages, specifically pages with forms. I wanted it to be preferably a java library to easily plug into ColdFusion. After trying a few (jTidy, Cobra and two HtmlParsers) I stumbled on Jericho. http://sourceforge.net/projects/jerichohtml/. It did all I needed to do and then some. It did great parsing HTML and getting the values I was after. Projects java docs helped a lot.

There is only one .jar file (as of this writing jericho-html-2.6.jar). So put it somewhere in the CLASSPATH. To check if you put it in the right place in ColdFusion 8 administrator look in Settings Summary and see if it's listed under Java Class Path.

I didn't go as far as to write full on wrapper for it. I was after form fields so here is the code that get's what I need.

<br /><cffunction name="load" access="public" output="false" returntype="void"><br /> <cfargument name="formUrl" required="true" type="string" /><br /> <cfscript><br /> this.setFormUrl(createObject("java","java.net.URL").init(arguments.formUrl));<br /> this.setSource(createObject("java","au.id.jericho.lib.html.Source").init(this.getFormUrl()));<br /> this.setFormFields(this.getSource().findFormFields());<br /> parseFormValues();<br /> </cfscript><br /></cffunction><br />
Here is the code for parseFormValues() where you can see some of the API Jericho provides in action. As I looked through this, I noticed where I collect lists values with listAppend() if the values have commas it would create a problem.
So keep it in mind if you plan to use it.
I am sure there are some other improvements that can be made since it's a first pass at this.
<br /><cffunction name="parseFormValues" access="public" output="false" returntype="void"><br /> <cfscript><br /> var fieldStruct = {}; <br /> var formFields = this.getFormFields();<br /> var field_i = formFields.iterator();<br /> var value_i = '';<br /> var option_i = '';<br /> var fieldName = '';<br /> <br /> // itterate through fields<br /> while(field_i.hasNext()) {<br /> formField=field_i.next();<br /> <br /> fieldName = formField.getName();<br /> <br /> fieldStruct[fieldName] = {};<br /> <br /> fieldStruct[fieldName].name = fieldName;<br /> fieldStruct[fieldName].type = formField.getFormControl().getFormControlType().toString();<br /> <br /> //CHECKBOX, RADIO, BUTTON, SUBMIT, IMAGE, SELECT_SINGLE or SELECT_MULTIPLE. <br /> if(formField.getFormControl().getFormControlType().hasPredefinedValue()) {<br /> value_i=formField.getPredefinedValues().iterator();<br /> <br /> fieldStruct[fieldName].value = formField.getValues();<br /> fieldStruct[fieldName].optionvalues = '';<br /> <br /> while(value_i.hasNext()) {<br /> fieldStruct[fieldName].optionvalues = listAppend(fieldStruct[fieldName].optionvalues,value_i.next());<br /> } <br /> <br /> //get values and labels for OPTION tag<br /> if(listFindNocase("SELECT_SINGLE,SELECT_MULTIPLE", fieldStruct[fieldName].type)) {<br /> fieldStruct[fieldName].optionlabels = '';<br /> option_i = formField.getFormControl().getOptionElementIterator();<br /> while(option_i.hasNext()) {<br /> fieldStruct[fieldName].optionlabels = listAppend(fieldStruct[fieldName].optionlabels,option_i.next().getContent().getTextExtractor().toString());<br /> } <br /> }<br /> //TEXT, TEXTAREA <br /> } else {<br /> fieldStruct[fieldName].value = formField.getValues();<br /> fieldStruct[fieldName].value = fieldStruct[fieldName].value[1];<br /> }<br /> }<br /> <br /> this.setFieldStruct(fieldStruct);<br /> //dump(fieldStruct); <br /> </cfscript><br /></cffunction><br />
Note on "this" scope usage. The component this code is in, extends BaseComponent (thank you Hal Helms)
with generic (can you say "lazy" :-) ) set and get implemented with onMissingMethod. You have to use "this" for it to work inside the component.
I did find accidentally later in the project that using this (ooh cool pun) technique is slower then actually creating a setter and a getter, which kind of makes sense.