WatchNon-threaded

Forums » The Lounge » Read Message

A place to sit back, hang out, and talk about anything you'd like.

Scraping for data for a project!

28 days ago
Hey guys, so I'm very new to this website and I was hoping to get some help with some data I'm trying to collect. I'm trying to create a fantasy AI agent for an RPG game sort of like AiDungeon and decided this was a nice place to get the fantasy text I need. But I'm having an issue with getting just the text from the stories, is it possible to get just the text without going through the choices? This may be a noob question, sorry I'm very very new to the site.

Scraping for data for a project!

28 days ago
Not possible to just get the text without clicking all the links. AIDungeon did this already with many of the games from the site. IIRC it was not welcomed very warmly due to the lack of consent by authors and usage of peoples work without giving any credit where it's due. You could probably get a lot from the AID repo if they have their dataset open source.

Scraping for data for a project!

28 days ago

Not really, I'm afraid. The games on the site are all choice based, so there's no way of progressing without the choices. I mean, some authors may have saved their stories as word documents, but that would literally be a case of asking individual authors if they have a word document for each of the stories that you're interested in using (keeping in mind that a lot of authors don't actually use the site anymore.)

Scraping for data for a project!

28 days ago

I dunno if any of us, strictly speaking, know how to do that, but it was something we just sort of tolerated the first time it happened. And as amusing as it is getting surprise porn about Semra and Eowyn in AI dungeon every now and then, I'm pretty sure people just routinely taking our shit like this is not a norm we'd like for this site in the future. 

Scraping for data for a project!

28 days ago
Just to clarify, the data will not be used publicly, commercially or be spread anywhere at all whether through the game or otherwise. It is solely for my coursework in class and is not permanent by any means, I now understand that the consent issue here is quite iffy so I'm going to reconsider using this platform.

Other than that, thanks for clarifying about the choices. I was previously getting data while navigating the choices and I see that's the only way. I have tried looking for AID's data but it's hard to find anything fantasy related, it's all mundane stuff. I'll look some more anyway, thank you!

Scraping for data for a project!

28 days ago
There actually is a way to do this, but frankly I don't think you're up to coding an AI on the first place if you can't figure it out.

You could always try clicking the links though. In fact, if you click all of them you get a cool piece of commissioned art.

Good luck with your project though.

Scraping for data for a project!

28 days ago
I appreciate the confirmation that there is a way to do this but it would really help if you pointed out what that was, even just the name of the method is enough. While I am definitely not an expert (or even remotely close to that) I am also very confident that I'm "up to coding an AI" because the whole point of this project is to learn. If I don't know something, I'll learn it.

To that effect, once again, if you do know of the way to do it please let me know.

Thanks for the well wishes.