0

I've been reading stackexchange for years now, and always managed to find an answer to the problems I had to deal with. This time however, I'm going to ask a question as I could not find a solution to the issue I'm facing now.

I'm currently working on a research project, and need to create a sitemap for a website. I have more than 10GB of php files, downloaded from an online forum (now deceased). The files are all in one folder, and therefore I can't reconstruct the original structure based on folders.

I was wondering if there is a tool/script out there (I don't really care about the language it is written in) to analyse the php files and output some sort of sitemap/structure (XML, HTML, or else)

The files were created with phpBB; I'm using Debian and Ubuntu (two different PCs). Any suggestions are welcome. Many thanks in advance

Improve this question
5
  • Are you sure its 10GB of PHP files? I can imagine a few hundred MB's of PHP code perhaps and then many GB's of data - user comments etc. which usually would be stored on a DB. but in your case the data might be in flat files in some data format, i.e. possibly XML. Commented Jul 6, 2016 at 11:16
  • 1
    You say "downloaded from an online forum" - do you mean they were downloaded off a website (with a browser's "Save Page" function, or wget or similar)? Because that means they're HTML, and the .php extension is irrelevant. Commented Jul 6, 2016 at 11:39
  • Search engines do crawling a lot better than you. Sitemaps only make sense if you can produce a list directly/dynamically (from a database) with additional info (in a forum, date of the last post in a thread etc.). Or perhaps if you have dead spots (link is in the sitemap but no where else). Commented Jul 6, 2016 at 12:21
  • Are you intending to get this site back on-line as a static archive? or maybe extract the data from the pages and try to rebuild a phpbb site from the static html pages? If the latter, this will be a difficult but not impossible task and you'll need to write a web-bot to do the extraction - look at libwww-perl aka LWP and DBI for the database inserts. I like to use HTML::TokeParser with LWP as a reasonably simple way of extracting data from HTML pages or files. Commented Jul 6, 2016 at 12:49
  • Yes @JigglyNaga, sorry for the mistake; they are actually HTML files with .php extension. Thanks for pointing it out. Commented Jul 7, 2016 at 8:49

1 Answer 1

Reset to default
1

Package: webcheck

Description-en: website link and structure checker

webcheck is a website checking tool for webmasters. It crawls a given website and generates a number of reports in the form of html pages. It is easy to use and generates simple, clear and readable reports.

Features of webcheck include:
 * support for http, https, ftp and file schemes
 * view the structure of a site
 * track down broken links
 * find potentially outdated and new pages
 * list links pointing to external sites
 * can run without user intervention

Homepage: http://arthurdejong.org/webcheck/

One of the reports it generates is a site map.

Improve this answer
3
  • Looks like a cool tool! If I understand the OP right, the original/actual website is offline -- can this tool work with offline/saved files to recreate the site map? Commented Jul 6, 2016 at 12:22
  • 1
    Yep. I've used it for "off-line" sites many times in the past. The feature-list above says "support for http, https, ftp and file schemes". relative URLs in the HTML will be fine. absolute URLs won't be. In any case, you can always set up an apache or whatever vhost, with the relevant domain name, and edit your /etc/hostsor hack up a bogus domain in dnsmasq or something to point it at 127.0.0.1 or wherever. Commented Jul 6, 2016 at 12:32
  • @cas many thanks for the suggestions; I've tried it on a bunch of files and webcheck seems to be working pretty well. I'll keep on searching however, as if I could find something even more precise it would really make my day Commented Jul 7, 2016 at 8:50

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.