Bioware forums are closing

[Baaleos]

               

Am thinking best stuff for save is all fixing and setting for running new WinOS, Linux, Mac nd with many problum that video mayb has too.  That matter much for new player who try running NWN with Win8 or 10 and fancy vcard/drivers.

 

Also think all design stuff where they figger how to do tricky things shud archive.  So many script ppl post here in many years for help it sad to think all that gud work just poof away end month.

 

I wouldnt worry about specific types of posts that we want to save.

When this is done, everything will be archived - regardless of importance.

               
               

               
            

[Baaleos]

               

According to the tool I am making, there will be 6964 topics to archive in Neverwinter Nights 1 forums.

I think it got the wrong number some how.... as manual adding of the topics from the summary on the main forum page suggests there is much more.

It managed to get the Topic title and start date/time without any exceptions or errors being thrown, so I think im ready to move onto the all important job of grabbing post information.

Then I can just leave it running.

 

Here is a pic to show my progress

 

 

 

Strange.

After it parses all the Sub-Forums, it gets this figure for each of them.

Finished Parsing SubForum:Hordes of the Underdark Official Campaign : Topics present:117

Finished Parsing SubForum:Shadows of Undrentide Official Campaign : Topics present:44

Finished Parsing SubForum:Dungeon Masters Realm : Topics present:79

Finished Parsing SubForum:Guilds and Registry - Ye Olde Tavern : Topics present:101

Finished Parsing SubForum:Official Campaign : Topics present:185

Finished Parsing SubForum:Tools and Plugin Developers : Topics present:165

Finished Parsing SubForum:Modules : Topics present:501

Finished Parsing SubForum:Persistent Worlds & Multiplayer : Topics present:577

Finished Parsing SubForum:Builders - Toolset : Topics present:701

Finished Parsing SubForum:Technical Support : Topics present:809

Finished Parsing SubForum:General Discussion : Topics present:1274

Finished Parsing SubForum:Custom Content and Community Expansion Pack : Topics present:2411

Finished Parsing SubForum:Builders - Scripting : Topics present:2984

 

Which matches the main displayed counts.

Not sure where it got the 6000 number before.

 

Turned out it was due to the way I was threading.

It was spitting out the count, before it had finished  counting the last Sub Forum.

               
               

               
            

[Legacy_omen_shepperd]

               

We also have members constantly discussing a wide range of topics  over on the NWN community group we have on facebook. We will continue to have great discussions as long as we can. 

https://www.facebook.com/groups/nwncp/

 

               
               

               
            

[Legacy_meaglyn]

               

Thanks Baaleos! That will be excellent to have. I'll be sad to see this go though.

               
               

               
            

[Baaleos]

               

Hey guys,

Just an update - the Scraper is go.

Its able to scrape around 186 Topics per 5 minutes. (Size of Topics will make that rate variable)

Eg: It will iterate over all pages within a topic.

We have over 9000 topics, with over 100,000 posts among them.

So this might take a day or two to capture all the information.

 

Some screenies can be found here:

 

 

 

For the post content - I basically capture the innerhtml from the post_body per post element on a page.

This is the example content I capture.

 

<div itemprop="commentText" class='post entry-content '>   <!--cached-Sat, 30 Jul 2016 13:25:47 +0000--><blockquote class="ipsBlockquote" data-author="WiredWraith" data-cid="20270158" data-time="1464020440"> <div> <p>I checked and double-checked all the stats of my henchman and no negative status effects show up other than that annoying Movement Speed Decreased one.&nbsp; Made sure conversation options weren't enabling Stealth or Search mode too.&nbsp; She's just... bugged.&nbsp; How can I kill her to try and reset her?&nbsp; She appears to have some sort of "I'm dying but I can't die" flag set.</p> </div> </blockquote> <p>Try kill her in shadow plane cuz script make her ded for gud on mythlar level wher snakehead lady stay. Hench respawn in SoU suppose to happen next to wher last master was.&nbsp; Am thinking hench respawn in Ch1 at dwarf house mayb specal for module but shud work like I say in all other mods/area cept the wind04 (mythlar).&nbsp; Mayb u rite that her bic went corrupt.&nbsp; Kill/rez dun mostly to fix faction but mayb work to fix bad effect.</p> <p>&nbsp;</p> <p>I know too late now to help much but save LOTS LOTS in NWN SP.&nbsp; CC not debug real gud sometime so it alway a gud idea to save so not have to redo lots.&nbsp; Guess mayb SoU have some bug left too. <img src='https://lvlt.forum.b...)</span></span>' class='bbc_emoticon' alt=''>&nbsp; PW is better debug cuz alway want many ppl there so will fix bug very fast.</p>     <br>   </div>         <div class='ipsLikeBar right clearfix' id='rep_post_20270217'> <ul class='ipsList_inline'> <li id="like_post_20270217" class='ipsLikeBar_info' style="display:none">   </li>   </ul> </div> <script type='text/javascript'> ipb.global.registerReputation( 'rep_post_20270217', { domLikeStripId: 'like_post_20270217', app: 'forums', type: 'pid', typeid: '20270217' }, parseInt('0') ); </script>             <ul id='postControlsNormal_20270217' class='post_controls clear clearfix'> <li class='top hide'><a href='#ipboard_body' class='top' title='Back to top'><img src="https://lvlt.forum.b...)</span></span>" alt=""> Back to top</a></li>                                </ul>  

 

 

Its my hope that by saving the html source - we can keep image links intact.

Unfortunately it pulls through some bioware javascript code. That can be removed after we scrape it all though.

 

 

The progress bar is increasing, albeit slowly.

               
               

               
            

[Legacy_kamal_]

               

Can you also scrape the nwn2 forums please?

               
               

               
            

[Legacy_azaz1234]

               

this is great work that's being done to preserve the content already on the bio boards !

 

however, as a few posters have pointed out, there's still quite a bit of activity on these boards. there's good info coming out here every single day, new ideas, and new creative offerings. making sure we have an archive of existing content is an excellent idea, but i think doing only that and nothing else would be tantamount to consigning nwn to davy jones.

 

is it just me, or do other people also see a need for a continued, active forum ?

 

one poster suggested the nwn forums at nexusmods. i took a look and there are something like... 30 posts on that forum [which isn't to say it couldn't grow... '> ]  does anyone know if they could handle a forum similarly-configured to this one ?  i.e., categories for general discussion, the games themselves, scripting, custom-content developers, etc.

 

another obvious location would be http://www.neverwinternights.info ; it's already a central repository of good info on nwn, and there's a forum link -- albeit a link that happens to point here at the moment. so similar question to the above : does anyone know if that site could host a phpbb ?

 

imho we can't give up this valuable resource !

 

thoughts, ideas ?

               
               

               
            

[Legacy_Cassandra Saturn]

               try uBSN: http://biowaresocial...ork.boards.net/
               
               

               
            

[Legacy_Neverwinter_Knight77]

               I think the local vault and server vault players were pretty much two entirely different communities, so y'all don't know me (local vault roleplayer), but... Through all the ups and downs over the years, it's been good.
               
               

               
            

[Legacy_Proleric]

               

I suggest the new vault as the best place for future discussion.

 

It's controlled by the NWN community. Most of us are there already.

 

The Nexus isn't a friendly place, in my experience. I don't think a relatively open community like ours would feel welcome there.

               
               

               
            

[Baaleos]

               Btw guys, it looks like the scrape finished successfully.

It claims to have captured all 9000+ topics.

I will do a few tests to be sure, then I will make the mdf (ms sql database) available for download. It's currently at over 500mb in size,

I can also run this against other forums too, it's been suggested to capture the nwn2 forums.

Shall i capture to the same db (increasing its size) or to a new one?
               
               

               
            

[Legacy_Faerinaal]

               

Hi Baaleos, awesome work man.  I would vote for a separate database for NWN and NWN2 data, respectively. Cheers!

               
               

               
            

[Legacy_PLUSH HYENA of DOOM]

               

Thanks, Baaleos...

It would be a shame to see it all just cease to exist.

 

Vive la intransigence!

               
               

               
            

[Baaleos]

               

On checking it looks like it might have missed a few posts.

I am going to re-run it, this time adding a message hash field to the post entity.

This way, I should be able to more efficiently detect the presence of existing posts in the database, and be able to append to topics, instead of having to regenerate them from scratch.

 

It was unlikely to be completed perfectly the first time around.

This time when it runs, it will have to generate the topics and posts again from scratch, but it should be the case that it will allow subsequent re-runs to append, rather than replace.

Meaning subsequent re-runs should be fast.

 

I was a little concerned about the db size - given it didnt seem to get all of the topic posts, it was still half a gb.

Im wondering if that is space taken up by Microsoft transaction info.

               
               

               
            

[Legacy_Grani]

               

I'm definitely FOR migrating to the Vault. We don't want to move to another place we have no control of.

 

We've been there already. We've struggled to rescue content that was about to be deleted from the old Vault exactly because it was in someone else's hands, IGN's.

Now the same story happens with the forums, once again because someone else made a decision to shut everything down.

 

Let's not move to another place where this can happen. NWVault truly belongs to the community, that's what we should call "home".