Enabling your web site for Content Recommendations requires that you implement the Maropost Web Tracking script on your web site, and set the schedule for the Maropost Web Spider to crawl the site. If there are portions of your site that you do not wish the spider to crawl, then add that information to your robots.txt file. Check with your web site administrator to ensure that pages you want to be included in content recommendations are NOT being blocked from the web spider.
To set the schedule for the web spider, click the “Enable Recommendations” check box in the Create/Edit Web Site dialog box.
- Parallel hits – Specify the number of concurrent threads that will be spidering your site. The more the threads, the faster the site will be cataloged. However, the default setting of 5 concurrent threads is sufficient to spider sites having millions of pages.
- Crawl day – Set which days of the week you want the web spider to crawl your site. If you are constantly adding new content, then a daily crawling is highly advised.
- Crawl time – Set the hours of the day (in Eastern Time Zone) that you want the spider to run. You probably wouldn’t need to schedule it every hour, but if you are constantly adding new content or products to your site, then every 4-6 hours is recommended.