This is an exported version of the JIRA issue tracker. Please use the Google Code site to open new tickets or report updates to these existing tickets. Feel free to contact the mailing list with any questions.

[SCB-18] Add userAgent configuration
Created: Tue, 29 Jun 2004 12:11:09 -0700 (PDT)  Updated: Tue, 9 Aug 2005 21:33:22 -0700 (PDT)

Status:Closed
Project:Scraping
Component/s:scraping-engine
Affects Version/s:scraping-engine-0.3
Fix Version/s:scraping-engine-0.4

Type:New FeaturePriority: Major
Reporter:Henri YandellAssignee:Henri Yandell
Resolution:Fixed 
Environment:


 Description   
Need to get around crappy sites that block user agents.
Comment by bayard [ Tue, 29 Jun 2004 13:14:48 -0700 (PDT) ]
Initial header setting ability added.

Problems with User-Agent in particular:

Norbert is not using the User-Agent it is given correctly. It defaults to Java/1.4.2_03.

FileStore is also not using the User-Agent.
Comment by bayard [ Wed, 7 Jul 2004 10:47:19 -0700 (PDT) ]
Norbert 0.3 supports user-agent correctly.
Comment by bayard [ Sun, 1 Aug 2004 19:35:12 -0700 (PDT) ]
Scraping-engine now uses the userAgent in its Norbert usage.
Comment by bayard [ Tue, 2 Aug 2005 15:51:27 -0700 (PDT) ]
So should be closeable according to the last comment.
Comment by bayard [ Tue, 9 Aug 2005 21:32:05 -0700 (PDT) ]
Doesn't look like it does use the user-agent properly. Seems to use a hardcoded value.
Comment by bayard [ Tue, 9 Aug 2005 21:33:22 -0700 (PDT) ]
I tell a lie, the code is there :)