Mobile/PDA About Us Advertise here
User User name Password  
  Not registered yet? Register here.
Lost your password? Click here. 		 
  Home   News   Guides   Downloads   Glossary   Hardware   Profiles   Forums   v4  
 Advertise at AfterDawn.com     Link to us!     Private messages     Compose a private message     List of all updated threads     Threads without replies     Search messages
Tuesday 3.12.2024 / 12:19
Search AfterDawn Forums:        In English   Suomeksi   På svenska
afterdawn.com > forums > software, operating systems and more > linux - general discussion > help with wget downloading website
Show topics
 
Forums
Forums
help with wget downloading website
  Jump to:
 
Posted Message
kk0425
Junior Member
_ 		23. December 2009 @ 22:06 _ Link to this message    Send private message to this user   
So, I'm having a small problem with wget. I'm trying to download a website but, due to the nature of the site, not all the images are available all the time and I can't download just the folder with the images I want because of the robots.txt file (I found a way around that).

I need wget to act like a crawler, but it should download the pictures. But from my understanding the --spider command will not download anything. It just crawls the site looking for "stuff".

I'm using the command 'wget -r -e robots=off sitename.com' and that gets some of it, but only the links available. If I try 'wget -r sitename.com/images/' I get a 404.

If I use 'wget --spider -r -e robots=off sitename.com' it doesn't download anything.

Thank you for your time!
[ + quote]
afterdawn.com > forums > software, operating systems and more > linux - general discussion > help with wget downloading website
 

Digital video: AfterDawn.com | AfterDawn Forums
Music: MP3Lizard.com
Gaming: Blasteroids.com | Blasteroids Forums | Compare game prices
Software: Software downloads
Blogs: User profile pages
RSS feeds: AfterDawn.com News | Software updates | AfterDawn Forums
International: AfterDawn in Finnish | AfterDawn in Swedish | AfterDawn in Norwegian | download.fi
Navigate: Search | Site map
About us: About AfterDawn Ltd | Advertise on our sites | Rules, Restrictions, Legal disclaimer & Privacy policy
Contact us: Send feedback | Contact our media sales team
 
  © 1999-2024 by AfterDawn Ltd.

  IDG TechNetwork