Author Topic: Is there a way to see an index of all pages of a website  (Read 1599 times)

Offline Definitions

  • Dansdeals Lifetime Platinum Elite
  • *******
  • Join Date: Jan 2018
  • Posts: 1748
  • Total likes: 480
  • DansDeals.com Hat Tips 3
    • View Profile
Is there a way to see an index of all pages of a website
« on: June 01, 2018, 06:12:39 PM »
I have a website that I want to see all the pages that it has  (example.com/pageone, example.com/page2, example.com/page3....). Is it possible to do that with some program?
My Tapatalk notifications don't always work.

Offline Alexsei

  • Dansdeals Lifetime Presidential Platinum Elite
  • *********
  • Join Date: Sep 2011
  • Posts: 5789
  • Total likes: 1421
  • DansDeals.com Hat Tips 5
    • View Profile
    • Travel & Kivrei Zadikim
  • Location: Truckistan
  • Programs: COVID-23
Jews ≠ Zionists
Palestinians ≠ Hamas
Satmar ≠ SatmarHQ

Offline yesitsme

  • Dansdeals Lifetime Presidential Platinum Elite
  • *********
  • Join Date: Dec 2014
  • Posts: 5020
  • Total likes: 2237
  • DansDeals.com Hat Tips 4
  • Gender: Male
    • View Profile
Re: Is there a way to see an index of all pages of a website
« Reply #2 on: June 02, 2018, 09:53:36 PM »
What are you looking for directory browsing or site map generator?
["-"]

Offline Definitions

  • Dansdeals Lifetime Platinum Elite
  • *******
  • Join Date: Jan 2018
  • Posts: 1748
  • Total likes: 480
  • DansDeals.com Hat Tips 3
    • View Profile
Re: Is there a way to see an index of all pages of a website
« Reply #3 on: June 02, 2018, 11:52:35 PM »
What are you looking for directory browsing or site map generator?
IDK what those mean. I just want to know all pages that a website has. Don't need the contents of the page.

Situation is as follows. I found a website that has a bunch of unlock codes for a certain model phone. But I only know the URL of that page and there are most probably more pages for different models I just don't know the URL. There are no links to other pages on the page that I do have.

And is it possible to do the same with subdomains (pageone.example.com, pagetwo.example.com...)?
My Tapatalk notifications don't always work.

Offline yelped

  • Dansdeals Lifetime 10K Presidential Platinum Elite
  • *******
  • Join Date: Mar 2015
  • Posts: 10481
  • Total likes: 3836
  • DansDeals.com Hat Tips 43
    • View Profile
Re: Is there a way to see an index of all pages of a website
« Reply #4 on: June 03, 2018, 01:02:45 AM »
IDK what those mean. I just want to know all pages that a website has. Don't need the contents of the page.

Situation is as follows. I found a website that has a bunch of unlock codes for a certain model phone. But I only know the URL of that page and there are most probably more pages for different models I just don't know the URL. There are no links to other pages on the page that I do have.

And is it possible to do the same with subdomains (pageone.example.com, pagetwo.example.com...)?

Try this. A bit old, but should give you the whole website organized into a directory.

http://softbytelabs.com/en/BlackWidow/

Offline Yisroel Tech

  • Dansdeals Lifetime Platinum Elite
  • *******
  • Join Date: Dec 2015
  • Posts: 2162
  • Total likes: 365
  • DansDeals.com Hat Tips 2
  • Gender: Male
    • View Profile
  • Location: Monsey, NY
Re: Is there a way to see an index of all pages of a website
« Reply #5 on: June 03, 2018, 01:56:02 PM »
The answer is that it is virtually impossible unless one page links to others (in which case you can crawl through them).

Offline Alexsei

  • Dansdeals Lifetime Presidential Platinum Elite
  • *********
  • Join Date: Sep 2011
  • Posts: 5789
  • Total likes: 1421
  • DansDeals.com Hat Tips 5
    • View Profile
    • Travel & Kivrei Zadikim
  • Location: Truckistan
  • Programs: COVID-23
Re: Is there a way to see an index of all pages of a website
« Reply #6 on: June 03, 2018, 02:24:11 PM »
Or if the url's have a structure  like site.com/galaxys7 site.com/galaxys8 etc. you can write a script to look for potential pages
Jews ≠ Zionists
Palestinians ≠ Hamas
Satmar ≠ SatmarHQ

Offline yelped

  • Dansdeals Lifetime 10K Presidential Platinum Elite
  • *******
  • Join Date: Mar 2015
  • Posts: 10481
  • Total likes: 3836
  • DansDeals.com Hat Tips 43
    • View Profile
Re: Is there a way to see an index of all pages of a website
« Reply #7 on: June 03, 2018, 03:31:52 PM »
Try this. A bit old, but should give you the whole website organized into a directory.

http://softbytelabs.com/en/BlackWidow/
The answer is that it is virtually impossible unless one page links to others (in which case you can crawl through them).

Offline Yisroel Tech

  • Dansdeals Lifetime Platinum Elite
  • *******
  • Join Date: Dec 2015
  • Posts: 2162
  • Total likes: 365
  • DansDeals.com Hat Tips 2
  • Gender: Male
    • View Profile
  • Location: Monsey, NY
Re: Is there a way to see an index of all pages of a website
« Reply #8 on: June 03, 2018, 03:50:00 PM »

Are you sure this crawls even unlinked pages?

With advanced programming and scripting you can obviously technically brute force the site for pages but that is not something for someone without experience.

Offline yelped

  • Dansdeals Lifetime 10K Presidential Platinum Elite
  • *******
  • Join Date: Mar 2015
  • Posts: 10481
  • Total likes: 3836
  • DansDeals.com Hat Tips 43
    • View Profile
Re: Is there a way to see an index of all pages of a website
« Reply #9 on: June 03, 2018, 05:13:33 PM »
Are you sure this crawls even unlinked pages?

With advanced programming and scripting you can obviously technically brute force the site for pages but that is not something for someone without experience.
Not sure. It worked for me. I just used the trial or free version.