Forums » The Lounge » Read Thread

AKA "the type of shit Mousecore would rather do than their contest entry." One thing I noticed about this quirky site is that the forums don't have a search feature! Maybe it's just a side effect of the place's age. I noticed that people usually just used Google searches set to only scrawl CYS for results, which is definitely an option. Out of sheer boredom and lack of motivation for my contest entry, I decided it would be kind of cool to have a dedicated CYS search. So that's basically what I did this entire weekend. Here is CYScraper. Like a Google search, you can query anything directly from CYS' forums, but CYScraper lets you filter results based on author, excluded author (all posts made NOT by an author), forum(s), and date before/after/on, as well as if you want to search only post replies vs. only threads (i.e. thread titles). While I'm no Google algorithm engineer, I did try to create a VERY basic relevancy scoring algorithm that ranks posts by default based on a few factors. You can also do a strict query (i.e. if you search for "heir", you will ONLY see "heir" and never "their") by wrapping your search in "quotations." Oh, hey, and there's dark mode too. Some Things to Consider - THIS IS AN IMPORTANT ONE. The data I have on hand for people to search is very limited. I didn't want to scrape all of the CYS forums in one sitting because I didn't want to get my IP flagged or literally blow up the website. Right now, the data you can query through is any post made within the timeline given on the CYScraper site, as well as any post on the front page of any forum (so as an example even if the Reading Corner's first page has some threads from beyond the timeline, you can still search through those on CYScraper). Over time, if people see this as a legitimate tool, I will collect more data from further back to put into the search service. - New posts are not automatically entered into the CYScraper database as soon as they are posted. They have to be scraped by my own service first in order to be queried on CYScraper. I am still in the process of figuring out an acceptable automatic scraping schedule that toes the line between having a good amount of fresh data vs. not harassing the site with requests. (I know it says it has a schedule on the CYScraper site itself, but that was an artifact from when I was still brainstorming how the hell I wanted to do this and I'm too lazy to update my entire repo just to remove that one sentence.) - With that being said, the majority of performance/optimization decisions are made with favor towards CYS. CYS a small site and I don't need or want to flood its servers excessively just for a performance buff on my end, even if that means the tool gets data slower/less frequently. - Downtime on the CYScraper page itself is definitely possible while I'm still figuring out its hosting configurations. - The mobile version doesn't look super pretty. I will be going back to test the UI specifically on mobile, but it should look good on Chromium and Firefox desktop browsers. So yeah, just a funky little project I made to test my Python skills and to solve an incredibly niche problem. Go ham, let me know if you find any problems, and if you have any suggestions, I'm all ears.

This is fucking dope.

Thanks Chris :)

I'm testing a recent scrape from all the way back to March 2019, so once I check that the data is valid from that push, you will surely get more lol, fags.

Searched up End saying "fag" and saw posts from 2017, did you go further back for posts from specific members or something?

Nah I'm too lazy for that. The scraping cutoff date determines "freshness" by the last post in the thread rather than the thread's creation date. So if there's a thread from 2017 that got bumped in late 2025, it got picked up. The reason I kept the rest of the thread (regardless of those post dates) instead of only the posts within a certain date is so that I don't end up with "impartial" or half-scraped threads in the system already. Just makes it easier for my scraper to more optimally skip threads that are already determined to be stale, which means less work for the scraper and less requests to handle on CYS' end. Also more shit for you to search through. Everyone wins.

That was actually one of my inspirations for the tool haha. I mean I'm sure the noteworthy stuff is just kept in memory but it might be nice to use for a quick receipts lookup.

Oh damn that's nuts. Well I'll definitely be keeping this running if it's going to be useful for everyone. It costs me a whopping total of $0 to host and run anyway. Glad it can do something good :)

Aw thanks End!

Holy shit! Thank you so much Thara!

So I knocked out a few bugs with the scraper tonight that didn't like the concept of pagination. I'll be rerunning a scrape tomorrow to grab data back to March 2019. Periodically I'll keep you guys posted whenever there's a new data drop, as I'm only going back a few years at a time, but I think over half a decade of CYS forum data is enough toys to play with in the meantime.

So I figured I would write this here so it's more visible. Scrape Progress 2019-2026 DONE 2016-2019 DONE 2014-2016 DONE 2012-2014 DONE 2010-2012 DONE 2007-2010 DONE 2002-2007 DONE

Feature To-do List

• CYScraper extension: it would be another neat project to have an extension that puts the CYScraper searchbar and filter dropdown right on the forum main page. I'll have to work out the details once the OG search page is where I want it to be. One of my larger concerns with this is making sure it works fine in tangent with Brad's very popular CYS extension, which I also use.

Other Considerations

As it says on the site, CYScraper is an unofficial tool. While I have access to all the scraped data, at the end of the day, those posts are the site's posts. With that being said, privacy is a big thing to keep in mind. Admins, if there are any posts/entire threads that the scraper picks up that need to be deleted (for example, the scraper runs while there's a heated gamer-word frenzy going on in the Lounge that's going to all be deleted by the mods), please contact me with the post/thread IDs and I will delete them. I can also modify individual entries if you only need some parts of a thread or post removed. This is also a big reason why images/embeds are not saved in the scraper, in the off chance that it picks up something extremely unsafe (think highly illegal content). Additionally, these posts are your posts too. If at any point, for privacy concerns, you would like to not have any of your posts scraped by the CYScraper, please contact me and I will add your data to an exclusion list and delete any previous posts/threads in the CYScraper system if necessary. I might point and laugh though, but I won't outright say no. And as if it needs to be said, no I am not selling or distributing your post data. I am a one-man show that's doing this for fun outside of their full-time job and irl obligations. I literally could not care less.

Expect a bit of downtime right now, I am currently launching a version with the full March 2019-March 2026 scrape.

Goddamn y'all, sorry for the wait. The 2019 scrape is officially ready. However I will take a break from scraping to migrate the data to a better databasing solution as the current setup is going to get exponentially slower on the user's end the more data there is. So yeah, the search should be functional right now, just a bit sluggish.

So I spent this week forgoing my usual hobbies and human socialization to drop the new CYScraper update. It is actually a big enough change that fuck it I'm calling it CYScraper 2.0. What's New?

• New database: the original solution worked for a hundred thousand or so posts but was in no way scalable (for context, CYS has about 500,000 posts right now). Over the week I migrated all of CYScraper's data into a relational database to make use of the serious performance and querying advantages. On average, CYScraper 2.0 searches are already three times faster than the original. I am continuously finding ways to make things faster of course.
• New ways to filter: you can now use search operators, similar to a Google search. Alongside strict searching with quotations, you can also omit phrases with a dash, and you can use 'and' and 'or' operators to narrow or broaden your search, respectively. I have examples on how to use these operators on the CYScraper main page.
• More relevant results: because of some really nifty SQL features, CYScraper now has even more relevant results showing up first. Additionally, the lexicon capabilities are improved to widen your search - for example, searching for "running" can also yield "runs," "run," "runner," etc.
• Date limit: the default limit for general searches is up to 5 years ago. You can of course change this in the "After" field, but be warned that broadening your search can lead to longer wait times for results.

In terms of scrape progress, the scraper is about halfway through the 2016-2019 job, so anticipate that sometime soon.

So my silly ass made a little mistake that caused about 65,000 posts in the DB from the 2016 scrape to not show up in the search results. Fixed that. As recompense, I wrapped up the 2014 scrape and delivered that to the DB today too. So have fun with ~12 years of data as of right now. Wow, that's like, older than some of our site members already. An unfortunate update: we are reaching the storage limit for our free tier very quickly, which means it is likely that I will need to pay a monthly fee for the cloud DB hosting if I want to continue scraping. I will think it over for a bit, but I'm leaning towards biting the bullet and forking over the fee for the greater good.

And as my final update for this week: Sorting and pagination customization: you can now sort your search results on the basis of relevancy, date, author, title, and threads/posts (replies) first. You can also retrieve 5, 10, 20 (default), 50, or 100 posts per page at a time. I'm fucking beat but I'm happy to see my pet project to almost-fruition. See you fuckers later.

Aw thanks Hatter!! Your donation to my 1,000 point total will not go unnoticed o7

Well, another half-week of foregoing my usual hobbies and CYScraper 3.0 (also known as The Rest Of It) has arrived. What's New?

• Full CYS scrape: as of this afternoon, theoretically every single accessible, non-deleted post is now visible on CYScraper from 2002 to 2026. Please let me know if there are any posts you are certain are missing from the scraper database. This is out of my own genuine curiosity as the whole site is actually hundreds of thousands of rows below my estimate (though tbf post IDs aren't that accurate since threads can be deleted). So yeah, either there really aren't as many still accessible posts as I thought, or my scraper missed something.
• Daily CYS scrapes: this one actually isn't done yet, but I'm trialing a new run method for my scraper and the scheduler for it, so this should be happening pretty soon. As I've said before, it'll case any CYS posts from within the past ~2 months and insert new or updated posts into the database every midnight EST.
• Database upgrade 2: The Squeakuel: once again we were approaching a limit with our sheer amount of data. The DB provider I was going through was asking for unfathomable amounts of money to give me some more storage, so I told them to fuck off and bought a VPS to self-host this thing myself. It now has noticeably greater processing power and enough storage to last multiple lifetimes. The downside is that I am paying monthly for this VPS instance, but it is less money than what I otherwise would have paid.
• Fixed mobile: this was admittedly an extremely easy fix but I was dealing with more important things like, y'know, the data. It should look nice on phones now.
• Better dark mode: the dark mode was ass. It's no longer ass.

Bonus: Some Fun Data! The following are some just-for-fun things my database generated automatically. Be curious or, even better, let me know if those totals for each year seem to track (as I'm particularly side-eyeing the 2002, 2004, and 2012 post totals). A little data visualization of the aggregated post counts by year, with 2013 in the lead at nearly 45,000 posts. Maybe it's cheating if I did this in LibreCalc but it was easier to read here than the file the DB spat out. So. CYScraper is practically "done." At least, the parts I 100% wanted implemented. Now my greatest hope is someday this tool might be retired... someday. Well anyways, can't wait to go back to punching people again as a means of slacking off my writing instead of worrying about this.

Part of me is anxious that my scraper blew up or some shit while getting into pre-2013 territory. But thanks Mystic! :D Honestly the most time-consuming part was the initial programming of the scraper and connecting it to a database; subsequent changes have been more or less streamlined with how modular the system is behind-the-scenes, which is how I've been able to pump out these updates so quickly (the first two months of researching and experimenting were the hardest by far).

Oh shit, I didn't realize that Forum Games was *that* huge. Well, guess that's where ~450,000 of my estimate went lol.

Everything about that feels so 2015. Wow. Creative Corner is more interesting anyway, we love a Thunderdome brawl. Casing all of Forum Games in this site's heyday on top of the already bustling Lounge would've been a pain anyway.

*35,000 word

I'm not actually that familiar with the WC lore aside from the fact that it ate up 2013 and the whole series got banned from the site. Was there a planned movement into CYS by a group of WC fans, or was it just completely incidental that Warrior Cats became a trend among elementary-aged noobs that year?

Surely we'll soon get our next punching bag that gets publicly stoned in the Lounge again. Let's see those numbers go up!

Thank you! :)

It honestly hasn't been in the question because there is already a storygame search that's more than serviceable (contrary to what some noobs may think about it). Obviously the story search can be improved, but it at least exists and functions as intended.

I didn't read this before responding to TV Girl's thing in the other forum and did a triple take on the "Commend" button. I thought someone somehow fucked up. But thanks a bunch Mizal!!!!! :) I promise I will keep my foiling and fraud to a minimum.

Oh so that's where those queries were coming from. There were also some interesting searches regarding some... unfun taboos a few days ago. Were you searching at around midnight EST? I'm wondering if the auto-scraper had something to do with it. Otherwise I can check this out in better detail during my lunch break or something because that is a very curious error.

I think it's funny that someone flooded the requests with foot fetish related searches after I mentioned it. Also something along the lines of "how to use CYScraper to blow up CYS asking for my friend Rider." And that's all fun and games since I didn't bring up the whole "hey I can see every single search timestamped btw" thing until now, but the more interesting searches were the one-off queries for incest and necrophilia back when the tool was still in its infancy and hardly used more than 4-5 times a day (mostly me testing things). And ofc the requesting platform was Linux.

I'm just throwing stones from a glass house lol. If I weren't so used to the Windows environment for literally everything else but programming, I probably would've fully switched to GNOME Fedora a while ago, it was easily my favorite OS to develop on.

I did go and retry all the queries you mentioned in this post myself (so now the log looks like a Discord chat between edgy unfunny 14 year-olds), it seemed to be sorting the gamer-words fine. Those posts from 2008 popped up immediately when sorting by oldest date across all years. I am noticing a funny discrepancy with the "posts only" search as you've seen for yourself though. I'll be checking into that and see what's up. UPDATE: I somehow forgot my own data model. Duh. "Posts only" only fetches replies to threads rather than the OPs of threads themselves. I'm still iffy on if I like this logic though. I think I would either change the frontend to explicitly say "replies only" or make it a "search title only / search content only" sort label to make things less confusing. But yeah, not a bug, just a misleading frontend label on my part. Good catch Mizal.

Goddamn it Corgi.

It's just the API request logs for the tool.

Quick question, is it intentional that the search filters start out only searching for posts after 3/22/2021?

It's a feature (not a bug!) from the 2.0 update; default searches are capped to search posts from up to 5 years ago for the fastest immediate response, though this can just be changed in the filter panel. This limit came about before I made significant improvements to the content indexing system though, not to mention that the VPS the search is hosted on now has way more processing power than when it was using the free tier database provider. At this point having a default date limit seems negligible for performance, but I'll have to see how much of a difference it makes while I'm developing the "search all by member / in timeframe" feature (which will definitely take more time to process).

Implying Sindri is gonna do anything is a pretty good joke even if you didn't intend for it to be one.

Wahoo! Another post-worthy CYScraper update! What's New? Wildcard searches are here! Search for a solitary asterisk * to initiate a wildcard search. This will grab all posts that match the filter criteria. By default, wildcard searches are sorted by newest date (sorting by "relevance" wouldn't really work in this case because there's no content matching being done). To avoid the possibility of someone trying to grab every single post in the DB at once and causing the connection to timeout, you must meet AT LEAST ONE of the following limitations:

• At least one author is specified
• There is an On date
• There is a Before and After date AND these two dates are no greater than one year apart

So now you can search through all of your still-accessible posts and cringe at your young selves, or see how many posts/threads were made by any user in a given day/week/month/etc., or some combination of all that. As long as at least one of the limitations are met, you're good to go. And for those who are curious about the project itself, I added a little tech stack blurb in the header modal. Finally, do not be afraid to give me feedback if you think something doesn't make sense or would work better some other way. If you think a one-year limit is unreasonable or too restrictive, tell me you want something else. The worst I can say is "no."

I said it once and I'll say it again, thank THE LORD that I was too much of a wimp to make an account back when I first found the site. I would've never started this project if it meant broadcasting my elementary school digital footprint.