The developing process and some technical details

The general process

Stage 1:
- Build a single gadget with all the basic functions.
- Use a *.csv file database (Excel comma separated file) - not connected to the website yet.

Stage 2:
- Implementation of HTML parsing and connectivity to *.csv files.

Stage 3:
- Separate to one main gadget and several secondary ones using the pubsub framework.
- Addition of a search engine to the main gadget.

Stage 4:
- Design improvement and addition of summaries to the secondary gadgets.

---

Stage 1:

Example of the gadget at stage 1:

This stage was about getting familiar with the IGoogle API. First we wanted to create a basic version of the gadget with the main gadget’s functionalities, meaning a stand alone gadget that gets its data from a text file (written by us as a .csv file), display it, update from the area selection and includes links to the events and locations.

Stage 2:

Our challenge at this section was how to extract the information that is stored at the Mouse.co.il site in different places and deliver it to the gadget in a way that will make it as easy as possible.
At the beginning we tried to use the DOM way to manipulate HTML pages (looks at the HTML page as a tree where its root is the "document" and its first direct siblings are the "head" and "body" tabs).
After an inconceivable number of tries we gave up this way. we have realized that it is easy to work with the page you are in but approaching to the information in a different page is much complex and doesn't always allow using the wide majority of action we've been planning on executing quicly enough.

In addition we have come to an understanding that the right solution would be using a simple, daily updated data base that is personalized to our needs. The main reason is that we were concerning that each access to the gadget would take more time then a regular gadget upload would. This might have happen manipulating the original site directly (with the DOM methods). It would also be inefficient cause the data that would be fetched by a certain client (an action that would require time) won't be available for others, something that would have improved the usability.

After we have decided to focus on the “data base” way, we have set JAVA as our programming language for conducting the task of fetching the data and creating the data base, and turned on to look for ways for extracting the information.
First we try to use the “extractText” library, which is a “jar” library that includes, among others, a method that extract all the test fields from a site according to a url that is given to it as a parameter.
Soon after establishing the base code we have encounter two problems: (1) the site Mouse.co.il is written in a Unicode that is not supported by the String library in JAVA. We got all the text but in weird characters (“gibrish” ). (2) this way made us understand that due to the demands that we set for the gadget at the beginning, extracting the text won’t be enough. This is because first of all the page that holds the data we needed includes a lot of irrelevant text for us. Second and much more problematic is that all the event names and locations contain links to the relevant pages at the Mouse.co.il site, and we wanted to keep this option of use. These links, of course, are hidden with in the source code, there for they won’t be extracted by the text extract method. Even if we find a way to extract the links separately, it would be very difficult to match between them and the text fields.

Any way we went back to square one the second time only this time our goal was to extract the source code it self as a String and manipulate it.
After searching the web for a while and cross examine a few relevant results we have managed to come up with the method we needed. Now all we had left to do was to build on top of it an engine that is based on method from the String library of JAVA in order to get separate strings, each one holds the information about a certain type of events in a certain area in the country. These strings are saved in files (each one in a different file) that are the data base of the gadget.
This engine is meant to run once a day and update the data in the data base (the running takes a few seconds and the data base includes seventy small text files). Timing the run is simple using “crontab” command.
The gadget it self fetch the data from the files with any client access or change of area. Because all the files holds specific data (for certain event type and area in the country) the loading time is not perceivable (actually similar to the running time we got at the first stage of developing) and the user experience is very smooth.
The data base, that also includes the JAVA engine, is saved on our own private library at the CS computer lab.

Stage 3:

Example of the gadget at stage 3:

At this phase we were left with connecting between the gadget and the files being built by the JAVA engine, see this connection is fast and stable enough, design the gadget elements themselves and the main effort of separating the gadget into a main gadget that is connected to a few secondary gadgets (each shows different type of events).
The connection between the second gadgets and the main one is being done with sub-pub (an IGoogle tool).

Stage 4:

Example of the gadget at stage 4:

At this phase we focused on improving the design, closing loose ends and in addition we decided to add the events summaries.
With this application the user no longer needs to link to the source site in order to read the event's summary.

---

Source code

---

Back to the main page Back to the last page