This is an exported version of the JIRA issue tracker. Please use the Google Code site to open new tickets or report updates to these existing tickets. Feel free to contact the mailing list with any questions.

[SCB-19] Set a referer for initial scrape
Created: Tue, 29 Jun 2004 12:11:45 -0700 (PDT)  Updated: Wed, 17 Jan 2007 13:16:50 -0800 (PST)

Status:Open
Project:Scraping
Component/s:scraping-engine
Affects Version/s:scraping-engine-0.3
Fix Version/s:scraping-engine-0.6

Type:New FeaturePriority: Major
Reporter:Henri YandellAssignee:Henri Yandell
Resolution:Unresolved 
Environment:


 Description   
Sometimes need to set referer's when scraping that first time. Maybe any http header.
Comment by bayard [ Tue, 29 Jun 2004 12:52:17 -0700 (PDT) ]
This is handled by being able to set http headers I think.
Comment by cisco77 [ Wed, 17 Jan 2007 13:16:49 -0800 (PST) ]

http://cisco21.myblogvoice.com/?sri-lankan-nude-girls-free-picture-gallery-sri-lankan-nude-girls-pictures-sri-lankan-nude-girls-sexy-pictures-7322
http://cisco21.myblogvoice.com/?pornographicpictures-pornographics-picture-galery-pornographis-pictures-pornography-free-picture-gallery-7321
http://cisco21.myblogvoice.com/?spy-cams-babysitter-pictures-spy-pictures-celebrities-breastfeeding-spy-satillite-pictures-squirter-women-picture-7320
http://cisco21.myblogvoice.com/?pornographic-pictures-uncencored-pornographic-pictures-women-lactating-pornographic-picturesex-pornographic-picturres-7319
http://cisco21.myblogvoice.com/?sporty-car-picturers-spread-asshole-pictures-spred-pussy-pictures-spring-break-pictures-freaknik-girls-giving-head-7318
http://cisco21.myblogvoice.com/?ameteur-video-teen-tastyhotpussy-teengirls-pics-entireweb-ameteur-wife-pics-ameteurs-pics-ametuer-feet-pics-7317
http://cisco21.myblogvoice.com/?amiture-pics-carol-amiture-pics-nude-amiture-poontang-pics-amiture-porn-pics-amiture-pussy-pics-amiture-sex-pics-7316
http://cisco21.myblogvoice.com/?ameture-sex-pics-ameture-titt-pics-ameturepics-ameuter-nude-pics-amine-nude-pics-amiture-bikini-pics-amiture-daily-pics-7315
http://cisco21.myblogvoice.com/?rapeympics-animate-lesbian-pics-animated-animae-pics-animated-anime-tentacle-pics-animated-gifs-cartoon-anime-pics-porn-free-download-7314
http://cisco21.myblogvoice.com/?american-teen-ager-porn-pics-american-university-beirut-lesbian-girls-pics-amertur-porn-pics-ameteur-dicks-pics-ass-7313
http://cisco21.myblogvoice.com/?ameture-sex-pics-ameture-titt-pics-ameturepics-ameuter-nude-pics-amine-nude-pics-amiture-bikini-pics-amiture-daily-pics-7312
http://cisco21.myblogvoice.com/?ametur-pics-ametur-upskirt-pics-ameture-nude-pics-ameture-office-sex-pics-ameture-pics-ameture-pics-naked-older-women-7311
http://cisco21.myblogvoice.com/?ametuer-free-pussy-pics-ametuer-nude-pics-ametuer-nude-teen-pics-ametuer-panty-pics-ametuer-pics-horny-housewives-7310
http://cisco21.myblogvoice.com/?analsexfreepics-anamail-girl-pics-anamal-sex-free-pics-anamated-gif-pics-anamated-pics-anamation-cum-pics-7309
http://cisco21.myblogvoice.com/?amsterdan-sex-super-quality-pics-amstradam-pics-amteur-pics-amtur-pics-amture-black-pussy-pics-amture-pics-7308
http://cisco21.myblogvoice.com/?anorexic-sex-free-pics-anorexic-sex-pics-antimated-gay-pics-anul-free-pics-anul-pics-anul-sex-free-galley-pics-7307
http://cisco21.myblogvoice.com/?amputee-pics-free-amputee-pics-sex-free-amputee-sex-pics-amputee-sex-porn-pics-amputee-sexpics-amputee-women-pics-7306
http://cisco21.myblogvoice.com/?anorexic-chics-pics-anorexic-erotic-pics-anorexic-freepics-anorexic-girls-porn-pics-anorexic-nude-pics-anorexic-people-pics-7305
http://cisco21.myblogvoice.com/?anna-nicole-smith-skyscraper-nude-pics-anna-paquin-naked-shower-pics-anna-rusia-pics-annal-hardcore-pics-7304
http://cisco21.myblogvoice.com/?amiture-pics-amiture-porn-pics-amiture-pussy-pics-amiture-rape-pics-amiture-sex-pics-amiture-teen-pics-amiture-transexual-pics-7303
http://cisco21.myblogvoice.com/?analsex-analpics-analsex-free-pics-analsex-freepics-analsex-freepics-sample-gallery-analsex-menfree-pics-7302
http://cisco21.myblogvoice.com/?anime-porno-pics-little-virgin-chicks-gettin-fucked-anal-style-anime-schol-girls-pics-anime-scoolgirl-pics-7301
http://cisco21.myblogvoice.com/?ameture-male-teen-pics-ameture-pics-ameture-sex-pics-ameture-sluts-pics-ameture-teen-nude-pics-ameture-teen-pics-7300
http://cisco21.myblogvoice.com/?amuter-girl-pics-amuter-nude-pics-amuter-pics-amuter-porn-pics-amuter-sex-pics-amuteur-pics-amuture-nude-women-pics-7299
http://cisco21.myblogvoice.com/?fat-toes-tied-pics-lolita-galleries-fat-vagina-pics-fat-wemen-pics-fat-wemen-pictors-pusseys-fat-wemen-pictures-7298
http://cisco21.myblogvoice.com/?completley-free-lesbian-pictures-completly-free-porno-picts-completly-nakid-girls-pic-completly-nude-spice-girls-7297
http://cisco21.myblogvoice.com/?egyptian-sexpic-egyption-adult-pic-egyption-babe-nude-pictures-egyption-female-feet-pics-egyption-naked-pic-7296
http://cisco21.myblogvoice.com/?filippina-free-pics-filippino-nudes-pictures-fillipino-girls-pics-fillipino-nude-pics-fillipino-pics-fillipino-pictures-7295
http://cisco21.myblogvoice.com/?filipino-lolita-pics-filipino-male-sexy-pic-filipino-milky-titts-pics-filipino-nude-picture-filipino-nude-pictures-7294
http://cisco21.myblogvoice.com/?filipino-celebs-nude-pictures-filipino-cumshot-pic-filipino-female-celebs-cam-pic-filipino-free-hardcore-pics-7291
http://cisco21.myblogvoice.com/?filipina-pic-hubad-filipina-sexpics-filipina-sexy-pictuer-filipina-teen-sex-pictures-filipina-teenagers-girls-sex-pictures-7290
http://cisco21.myblogvoice.com/?fifteen-year-old-girl-suck-cock-pics-fifteen-year-old-girls-amsterdam-nude-pics-fifteen-year-old-girls-free-porn-pics-7289
http://cisco21.myblogvoice.com/?fetish-pic-speculum-fetish-pictrure-galleries-fetish-pussy-picture-gallary-fetish-stirrup-foot-free-pics-7288
http://cisco21.myblogvoice.com/?comic-cartoon-pictures-marijuana-comic-picture-erotic-comic-picture-gallery-jap-comic-pictures-comic-pictures-free-7287
http://cisco21.myblogvoice.com/?extreme-penatration-pics-extreme-rape-picture-extreme-sex-free-pictures-galeries-dirty-brutal-extreme-trasexual-pics-7286
http://cisco21.myblogvoice.com/?female-nudes-thumbnail-pictures-female-orgasam-pics-female-orgasam-pictures-female-orgasem-pic-female-orgasem-pics-7285
http://cisco21.myblogvoice.com/?extreme-hardcore-forced-toons-fucking-free-rape-pictures-extreme-hardcore-toon-prono-hentai-forced-fucking-free-pictures-7284
http://cisco21.myblogvoice.com/?female-naked-body-incest-birth-picher-female-naturists-pics-female-ndude-pics-female-niplles-pics-female-nude-models-free-pictorials-7283
http://cisco21.myblogvoice.com/?extreme-ass-fisting-free-picks-extreme-dilldo-pics-extreme-ertica-live-picture-extreme-funny-sexpics-extreme-hardcore-cartoon-prono-forced-fucking-fre-7282
http://cisco21.myblogvoice.com/?female-masturbation-pictures-female-masturbation-thumbnail-picsl-female-menstration-pictures-female-menstruating-pics-7281
http://cisco21.myblogvoice.com/?female-mastubation-pictures-female-mastubation-techniques-pics-female-masturbation-ejaculation-pictures-7280
http://cisco21.myblogvoice.com/?defacation-pictures-defication-pictures-demi-moore-nude-pics-demia-nude-pics-demo-asian-babes-peeing-pictures-free-7279
http://cisco21.myblogvoice.com/?exoctic-pictures-exotic-cars-pictuers-exotic-gay-men-live-pictures-fucking-exotic-pictures-dimple-kapadia-7278
http://cisco21.myblogvoice.com/?dauter-nude-pic-dauther-daddy-sexy-pics-dauther-pic-dave-matthews-picures-david-hamilton-pictures-dayli-picks-7277
http://cisco21.myblogvoice.com/?daddy-daughter-sex-family-incest-picture-daily-edition-pic-galery-daily-free-voyuer-pic-daily-gay-picture-tumbnails-7276
http://cisco21.myblogvoice.com/?felecia-pics-felecia-xxx-pics-felicia-hentai-pictures-fellatio-blowjobs-freepics-fellatio-free-pics-fellatio-free-pictures-7275
http://cisco21.myblogvoice.com/?eroticpics-erotics-pictures-hot-older-naked-women-mothers-eroticteens-free-pictures-erotik-cartoon-picture-story-hantai-free-7274
http://cisco21.myblogvoice.com/?feets-pictures-calista-flockhart-feexxxpics-felatio-pics-felatio-pictorial-felatio-pictures-felching-free-pic-7273
http://cisco21.myblogvoice.com/?cyna-pic-cyper-pic-cyper-spice-czech-nudist-pics-dad-daugther-oral-sex-pics-dad-dauhgter-incest-pictures-free-7272
http://cisco21.myblogvoice.com/?cunt-wide-cunt-eating-lesbo-sample-pics-cuntpic-cuntpics-cunts-fingering-free-pics-pissing-cup-pics-curved-penis-pictures-7271
http://cisco21.myblogvoice.com/?cumshot-facials-pics-cumshot-freepics-cumshot-freepics-teen-cumshot-freepicture-cumshot-pic-cumshot-pics-cumshot-pics-thumbnail-gallories-7269
http://cisco21.myblogvoice.com/?cumpicturescumpictures-cumpussy-pictures-cumshoots-free-pics-xxxx-cumshoots-xxxx-pics-cumshot-blowjob-freepic-cumshot-blowjobs-free-pictures-7268
http://cisco21.myblogvoice.com/?erotic-freepics-erotic-freepicture-hispanic-erotic-gallery-free-facial-pictures-erotic-garterbelt-pictures-7267
http://cisco21.myblogvoice.com/?fee-maria-tachibana-pics-fee-nake-pictures-fee-nude-celebrity-pic-fee-nude-pictues-fee-nude-picturesof-black-women-7266
http://cisco21.myblogvoice.com/?cumpictures-cumpussy-pictures-cumshoots-free-pics-xxxx-cumshoots-xxxx-pics-cumshot-blowjob-freepic-cumshot-blowjobs-free-pictures-7265
http://cisco21.myblogvoice.com/?eroitic-hardcore-sex-pictures-stories-erot-free-pic-erotc-picture-erotic-adult-suckling-pictures-erotic-amine-picture-7264
http://cisco21.myblogvoice.com/?fee-dragonball-pics-girls-pics-entireweb-fee-ebony-porn-pictures-fee-fat-pussy-pics-fee-gay-celeb-pics-men-naked-gay-poofs-7263
http://cisco21.myblogvoice.com/?cumface-pics-cumface-pictures-cumfacial-pics-cumm-blow-jobs-free-pics-cumm-free-pictures-cumm-shot-pics-cumming-shehe-pics-7261
http://cisco21.myblogvoice.com/?erica-boyer-pict-erica-eliniak-nude-pictures-eritic-pics-ermaphrodites-blowjob-free-pics-ermaphrodites-pics-7260
http://cisco21.myblogvoice.com/?fedish-rape-pictures-fedish-sex-pictures-fee-adult-comicpics-fee-amature-sex-pictures-fee-black-african-pussy-pic-galary-7259
http://cisco21.myblogvoice.com/?cum-shot-tumbnail-pics-cum-sluts-free-picx-cum-smaple-pics-cum-swallowing-pics-cum-swap-pics-cum-swapping-sluts-free-sample-pics-7258
http://cisco21.myblogvoice.com/?cocks-erection-pictures-gay-cocks-nude-pictuers-jennifer-love-hewitt-cocks-pics-thumbnails-gay-mens-cocks-assholes-7257
http://cisco21.myblogvoice.com/?free-porno-phto-free-sex-phto-girl-bizaree-free-sex-phtot-free-sexe-phto-freexxx-phto-fuke-phto-fuking-phtos-sex-7256
http://cisco21.myblogvoice.com/?piss-galeries-phots-gratuite-pissing-phots-playboy-phots-porno-phots-pregnancy-free-nude-phots-prono-phots-7255
http://cisco21.myblogvoice.com/?galleries-photosxxx-gallerry-photos-gallery-photos-amteur-models-gallery-photos-pregnat-gallery-photos-sex-7254
http://cisco21.myblogvoice.com/?nude-lolita-photography-nude-photography-nude-photography-female-genitals-galleries-nude-photography-gallaries-7253
http://cisco21.myblogvoice.com/?free-teen-pornography-photo-gallerys-free-teen-pusy-photos-free-teen-sex-photo-kerala-free-teen-sex-photos-gallaries-7252
http://cisco21.myblogvoice.com/?hardcore-cartoon-sex-photos-hardcore-celebritiy-sex-photo-gallery-hardcore-free-photo-hardcore-free-photo-irish-women-fucking-7251
http://cisco21.myblogvoice.com/?free-teenboys-free-photos-free-teens-lesbins-photo-free-teensex-photoes-free-thumbnail-ametuer-naked-photos-7250
http://cisco21.myblogvoice.com/?hardcore-nude-photos-tpg-couples-hardcore-photo-adele-stephens-hardcore-photo-galary-hardcore-photo-gallaries-7249
http://cisco21.myblogvoice.com/?hindi-actress-nagma-sexy-photos-hindi-actress-nude-photograph-bollywood-hindi-actress-nude-photographs-7248
http://cisco21.myblogvoice.com/?free-women-ejaculating-photo-free-women-porno-photoes-free-xrated-photos-free-xrated-simpons-photos-free-xray-camera-sex-photos-7247
http://cisco21.myblogvoice.com/?highland-dancers-photo-galery-highschool-celebratie-photos-highschool-toilet-photos-hight-grade-sex-xxx-photos-7246
http://cisco21.myblogvoice.com/?nacktphotos-paparazzi-nake-amatuer-photos-nake-photo-cartton-nake-schoolgirl-photo-naked-african-girls-photo-smples-7245
http://cisco21.myblogvoice.com/?nacked-females-photos-nacked-gays-photo-nacked-gays-photos-nacked-girls-photograph-nacked-photo-debra-nacked-photo-gallerynacked-females-photos-nacked-7244
http://cisco21.myblogvoice.com/?naked-photes-naked-photes-girls-naturist-photes-nba-photes-none-gay-male-photes-notty-nude-photes-nude-beach-photes-7243
http://cisco21.myblogvoice.com/?big-photes-sex-girls-biggest-breast-photes-ever-bollywood-actress-neked-photes-booty-photes-boy-nelly-photes-7242
http://cisco21.myblogvoice.com/?porno-prick-fuck-fanny-free-slut-cunt-sex-penthouse-row-penthouse-magizine-row-row-alias-satin-lace-penthouse-video-download-7241
http://cisco21.myblogvoice.com/?massive-penitrations-naked-jewish-penitration-painfull-sex-penitration-penile-penitration-penis-penitration-7240
http://cisco21.myblogvoice.com/?triple-penitration-upclose-penitration-vagina-penitration-vagina-penitration-pics-vagina-penitration-pictures-7239
http://cisco21.myblogvoice.com/?uncircumsised-penises-women-raped-sucking-penises-women-sucking-mens-penises-worlds-bigest-penises-hung-long-dick-pictures-7238
http://cisco21.myblogvoice.com/?backstreet-boys-penises-bdsm-click-thumbnail-enlarge-penis-beautiful-teengirl-fuc-little-penis-bec-penise-7237
http://cisco21.myblogvoice.com/?dwarfs-who-have-large-penises-pics-educational-erect-penis-picyures-educational-pictures-male-penis-elarge-penis-7236
http://cisco21.myblogvoice.com/?teen-penis-picure-totally-free-adult-pics-penises-longest-totalyfree-penis-uncesored-penis-uncircumcise-penis-7235
http://cisco21.myblogvoice.com/?free-voyer-exhibitionist-peep-free-xxxmovie-peep-gallery-peepingtoms-golden-showers-piss-peepee-hiden-peep-7234
http://cisco21.myblogvoice.com/?thai-pantyhose-thumbnails-blonds-pantyhose-thumnail-galarie-pantyhose-thumnail-gallery-mature-lesbian-pantyhose-movie-free-7233
http://cisco21.myblogvoice.com/?fuckvideo-pamela-tommy-download-fuking-pam-full-lenght-mpegs-pamala-anderson-fulltext-pamsuck-galeria-pamela-andersonxxx-7232
http://cisco21.myblogvoice.com/?megaspider-nude-pictuers-pamela-anderson-lee-megaspider-pamela-anderson-fucking-tommy-mpegs-men-pamelaanderson-nude-7231
http://cisco21.myblogvoice.com/?jpegs-naked-pakistani-girls-licking-pakistan-women-cunts-magezines-pakistan-men-who-love-licking-pakistan-wwomen-ass-7230
http://cisco21.myblogvoice.com/?celeberaty-xxx-indian-pakistani-chat-pakistani-gril-clebrity-pakistan-girls-cumshots-pakistani-mpg-cunts-pakistani-7229
http://cisco21.myblogvoice.com/?callgirls-pakistan-chatrooms-pakistani-girls-desi-baba-pakistani-nude-dirsrch-horny-indian-pakistani-babes-7228
http://cisco21.myblogvoice.com/?free-downlaodable-pakistani-music-free-download-erotic-desi-pakistani-stories-free-download-vidio-pakistani-songs-7227
http://cisco21.myblogvoice.com/?free-indian-pakistan-nacked-girls-picture-free-indian-pakistani-flim-actress-photo-gallery-free-indian-sri-lankan-pakistan-pussy-thumbnails-7226
http://cisco21.myblogvoice.com/?free-pic-paki-clit-free-sex-pics-pakistani-gay-male-pakistan-teenagers-gay-pakistani-boys-girls-nude-pakistan-india-girls-nude-france-7225
http://cisco21.myblogvoice.com/?tourture-pain-wemon-showing-painties-woman-painties-women-showing-painties-young-girls-verry-wet-painties-7224
http://cisco21.myblogvoice.com/?russian-teen-pusy-illegal-pink-minor-year-olds-russian-yrs-old-no-lolitas-only-teens-best-incest-rape-sites-family-rape-mother-and-son-sister-and-brot-7223
http://cisco21.myblogvoice.com/?rusian-girls-fucked-old-men-russian-golden-sexy-wemon-russian-older-weman-russian-older-women-sex-russian-scat-girls-golden-showers-7222
http://cisco21.myblogvoice.com/?wet-panties-oics-wet-pussy-oics-xxx-nude-oics-xxx-oics-xxx-sex-oics-yong-twinks-free-oics-young-lolita-nude-oics-7221
http://cisco21.myblogvoice.com/?vintage-nudist-camps-vintage-nudist-colonies-vintage-nudists-virgin-nudist-virgin-nudist-very-young-girls-older-men-7217
http://cisco21.myblogvoice.com/?teennudist-teennudist-camp-teens-nudists-pics-ten-year-old-nudists-tenage-girls-nudist-tender-nudist-sample-pics-7216
http://cisco21.myblogvoice.com/?free-amuter-nudes-free-asain-nudes-pics-free-asian-nudes-free-aussie-nudes-free-avi-nudes-free-beach-nudes-7213