Processing Webpages
Directed Search
The intended use of a Directed Search is to quickly process the Image Galleries linked to by a Thumbnail Post.
With this purpose in mind, Image Surfer Pro only crawls webpage links in a specific way and cannot be used as a general web spider to crawl the entire world wide web. At most a deep search will crawl three pages deep. While this seems to be a limitation, it keeps searches constrained to a reasonable processing time and efficiently finds the image content for the intended use.
A Directed Search is done in two stages. First the pages are searched for image content. Once all of the pages have been searched the image content found is added to the fusker collection.
Searching For Images
The search for image information builds three lists. A page search list is maintained during the process to assure pages are only searched once and to keep track of how far the page being searched is from the original page. The primary goal of the search is to find Direct Image Links. A list of unique Direct Image Links is maintained throughout the search for addition to the fusker collection. In some cases the desired image content is never found in a Direct Image Link - instead it is embedded into the pages. The third list contains unique image information found embedded on some of the searched pages.
The way the lists are built and maintained depends on the "level" of the page being searched. The following describe how the search progresses through the page levels.
The First Page
Though you do not have to start from a Thumbnail Post, having a graphic to use in the example will help explain the process. Say each small image on the thumbnail post is a link to an Image Gallery. The Image Galleries are hosted by different domains.
|
Using both the
If you chose to add image content or direct image links to your fusker collection by selecting the "Add Image" check boxes, those image references will be added to the collection prior to searching subsequent pages for image content. |
Second Level of Pages
Internet Explorer will navigate to each of the search links you selected in the ISP form created from the first page. It will navigate to each in the order they were listed in the form from top to bottom. Once the search page is loaded, Image Surfer Pro will process the page looking for image content. Image Surfer Pro processes 2nd level pages in one of three ways based on the type of content and links it finds on the pages.
|
Direct Image Links: The most common type of Image Gallery provides Direct Image Links from each thumbnail to the actual image files. Image Surer Pro will pull Direct Image Links from the gallery page directly and add them to the list of images and no new links are then added to the Search List because the desired image content has been found.
Image Page Links:
Non Image Gallery Pages: |
Third Level of Pages
While searching the second set of pages, Image Surfer Pro may have inserted additional pages into to the search list. This is done if it could find no Direct Image Links on the page but did find images which referenced other pages. Image Surfer Pro attempts to limit the number of these additional pages by not crossing domains. Only links which are found to be referenced by images and are in the same URL path as the referencing page will be added to the search list. In some cases this may cause Image Surfer Pro to miss some content you were expecting it to find but prevents a significant amount of wasted search time.
|
When processing this third level of webpages, Image Surfer Pro is expecting
to find an Image Page. A webpage where the primary content is a single image.
However it will still add any Direct Image Links to the list of images
found.
No links will be added to the search list when processing these pages. If no Direct Image Links are found on these pages, Image Surfer Pro will add any images embedded in the pages to the list of possible images. |
What The Search Looks Like
As each page is searched it is displayed in the browser window. Thus the search also works as a slide show of the pages. You may configure a pause in the viewing of these pages in your user preferences on the Views Tab.
As the pages flip by an interactive Image Surfer Pro task progress window is shown at the top right corner of your primary screen. This progress window can be moved and you can continue to work in other applications or even other Internet Explorer windows while the search runs.
This progress window provides some very useful information, including what operation is being performed and which page it is being performed on. The total number of pages which have been searched along with the current count of total pages to search is provided along with the number of Direct Image Links found and the number of possible images found embedded on the pages.
While the Elapsed Time counter is quite accurate, the Estimated Time Remaining may not be quite as accurate. The estimate is based upon the most recently processed pages and load time as well as processing time may vary greatly between pages. Since the number of pages to be searched may also grow the estimate of time remaining may suddenly become longer and the percentage of the progress bar completed may seem to move backwards at times during the search.
You may stop the search at any time by clicking the stop button on the status window. When you
do click the stop button it will change to a disabled "Stopping" button. The search will stop
after processing the next search page. When the search is stopped Image Surfer Pro will begin
adding the image content found prior to the stop command.
Adding Images
After the search for image content has completed or been stopped, the image content stored in the Direct Image Link list and the Embedded Image list will be added to the fusker collection.
Direct Image Links found in the search are automatically added to the fusker collection as that
was the primary purpose of running the search. This image information is added without checking
the size of the image. You may use the Stop button to skip some or all of these images and move
on to adding the Embedded Images.
Embedded images may or may not contain information you originally intended to find in your search.
It is not uncommon to have banner adds, headers, buttons, and assorted other graphics found by
a large search. In some cases data displayed as an image on the pages may not even link to a file
which could be directly accessed. To attempt to limit this clutter in your fusker collection, Image
Surfer Pro will compare the file size of each embedded image to your user preference. The
{Min image file size in Kbytes for auto collection add}
configuration on the Processing Tab allows you to tune how large embedded images need to be before
adding them to your fusker collections.
When All Is Done
Once all of the pages have been searched and all of the images have been added to your fusker collection, this popup window will let you know the processing has completed and the browser will return to the original ISP Form where you started the Directed Search.
You will notice that most or all of the pages searched are not in "back history" of the browser. They do however exist in the browser history - you will need to clear your browser history to remove them from there.
The