Migrating Points Data

We wanted to give you all a heads up, we will be migrating the wiki points data to a more reliable data store tomorrow. While the migration is happening the top contributors and wiki rollup pages will temporarily display incorrect data. Once the migration is complete those pages should actually work much better and reflect much more accurate data.

When: Tomorrow Afternoon (July 13)

How long will it take? A few hours probably (There is a LOT of points data)

We are making backups of all the points data as a precautionary measure so don't worry :)

I'll post here again when everything is complete!

@wcarle: will points earned during the migration be lost?

also did you know another two months of association data was lost when you migrated it to the new storage? I emailed about it but nobody ever acknowledged that it happened.

@pikahyper: Nope, points earned during the migration will not be lost. Numbers will be off until the migration finishes but when it does they will all add up.

I'm not sure if I ever saw that email. It may have slipped through, sorry about that. If you could provide an example of one of the missing associations that will help us pin down which data to recover. It's just a matter of recovering that data from the base level associations, if you can help pin down the exact window of data that's missing then we won't have to run the recovery command on the whole database, and it'll be fixed much quicker. Any info you can give would be super helpful, but if not we can just run the whole thing... that'll just take some time

@wcarle: I sent it to Rorie actually and I sent a tweet to both Rorie and Edgework but it was ignored like usual, I also tried to get your email or twitter handle but that didn't get a response either.

Basically whatever date you changed the storage system start there and go backwards at least two months, everything was lost from the previous loss all the way to the change over which was about two months. If you can do a full recovery that could be better though since a ton of data is still missing going all the way back to the launch of the new platform, it is spotty and sometimes it will just be a single section of a page like top characters missing but not top creators. About how long would it take to do a full recovery on the whole database? and would the wiki be unusable while it happens? would the wiki be unusable just recovering the two months worth and if so for how long?

@wcarle: looks like something happened with the points migration, the number on profiles doesn't match the number for the user popup, like if you mouse over my avatar here in this thread my wiki point total is way lower then it should be, that amount of points is probably from December of 2015.

Yeah the mouseover text is showing the wrong number, it's not a data problem it's just pointing at the wrong column in our database. We'll have that fixed really soon. There are a lot of places in the UI that display point totals, we just missed this one.

Your profile should be displaying your proper point value: http://comicvine.gamespot.com/profile/pikahyper/

In fact it looks like your points went up a bit because we found some extra points that were previously not added to your total.

Edit:

Full disclosure, we actually have one other bug with our command that we're working on too. A few submissions that happened between the code push and the run of the command are missing from the wiki top contributors pod on some pages. We are writing a quick command to sum those up and they'll be back soon

Sorry I forgot to reply to this one too:

A full recovery would take a week or two, just because we run it one association type (eg. character->issue) at a time. Once we get the points sorted out and fully working I'll kick off the process. Again all this data is totally recoverable so it's just a matter of taking the time to run the rebuild commands. The site would definitely be usable while running the rebuild association commands. It's just a matter of having one of us babysit the commands and restart them if they start slowing down or running out of memory.

@wcarle: I've got another point related glitch for you, if I manually change a users points it isn't updating on the profile or anywhere else that I can see, the only way I can see that its changed is by looking at the point changer box.

@pikahyper: Ok yeah, forgot to mention that. We need to address that feature. The change we made makes points actually accountable, we can run a query and know exactly where the points came from and where they got them. This feature kind of flies in the face of that. For now that feature doesn't work because it is adjusting that magic point value that has no backing.

What use case do the mods have for using that? When exactly would you adjust a users points? We could potentially keep that feature around, we would just change it so it inserts a negative points record in the submission table so we can account for where those points went

@wcarle: it doesn't come up as often anymore since we have fewer editors these days but it was mainly used for punishments, for example if we found an editor plagiarized a bunch we would remove the points or since we can't just ban a user from the wiki while allowing them to stay in the forum we put them at -100 points that way we know they are wiki banned, there are also times when we get editors that we need to keep in moderation so we manually lower their points regularly so they stay in moderation. There is one other situation where it happens but it is much rarer, users ask us to lower their points for various reasons.

All the reasons we manually adjust points though are so we can work around site weaknesses so it'll be easier most likely to just do what you're suggesting instead of fixing the other aspects of the site that would normally handle this stuff like having a switch on users profiles were we could keep them in moderation no matter how many points they have or having the ability to ban users just from one or more of the three sections of the site news/forum/wiki.

@pikahyper: I actually just tested and it looks like it is actually updating the correct column. I was able to change the point value of my alt account here: http://comicvine.gamespot.com/profile/wcarle88/

Are you having trouble exiting a specific account?

Also in regards to your use cases, I understand those. I'll put them on the list as things to investigate as we develop the new wiki. A couple flags for 'keep user in moderation' and 'wiki ban' definitely sound like they're worth including. For now at least though it seems like the edit points feature is working ^ (let me know if it's not working for you though, that is strange)

Although I would like to change it to add a negative record in the submission table

@wcarle: ok I just tried it again for a users account and its not updating on their profile but it is updating for the avatar popup in the forum and the point changer box, I'll PM you the account name.

@wcarle: should the Top Contributors box have disappeared from pages added to the site this week?

@wcarle:

Just another thought ... or not sure if it qualifies as a thought ...

Occasionally CV does hire others staff writers, and it might be in the interest not to have them in moderation if they didn't have enough points? Or for a moderator that mostly focuses on the forums? In which a moderator could just bump them to 5000

Not sure if that would be a concern though

While we are on the matter, someone bump me up to 5000 :P

@pikahyper@wcarle The migration has ended like 6 days ago, but i am still having trouble related to my posts. Please read this thread and help :

@lonewanderer67: just means the two problems were not related, you will have to wait until a staff member reads your thread.

@wcarle@pikahyper I thought the missing Top Contributors bug was fixed? A lot of the pages i'm working on is missing some or all of the information.

@cloudguy: If I'm remembering correctly they had to run a script to do it and since the association scripts would take weeks to run I'm sure the top contributor one would take just as long and they can't run them at the same time, I did ask about it further up and via PM I think but I haven't heard back from wcarle about anything in eighteen days.

@pikahyper Sorry, just re-read my previous post and I realised I wasn't clear on what I meant. What I meant was that the top contributors aren't appearing for things that are only a few weeks old.

@cloudguy: it goes back farther then that, I see stuff years old with the box missing, the script goes page by page and recalculates who the top editors are to re-add them, I'm not sure what order the script goes in but I'm guessing it is in chronological order of page creation and since there are thirteen page types and hundreds of thousands of pages with a top contributors box it probably takes a long time since it is a background process. All I know is wcarle is the one running these scripts and he has to be around to watch them and if he's not around they probably are not running.

Edit: I take it you had high hopes/expectations for wcarle as well but like usual they are not panning out, it's like I told another mod the other day "seems like wcarle has sunk into the CBS communication quicksand like all the other engineers and Rorie, they all start out great with the communication but eventually nothing." The site is what it is and it is a waste of time at this point to expect any better.

@pikahyper: Wow man, easy there. I was on vacation for a while, sometimes I neglect my PMs but I get a lot of them and I have a lot of work to do around here. We are a small team working on three sites and are in the middle of a complete rewrite of the wiki. I promise we're not sitting around doing nothing. Honestly man I communicate with you guys because I care, and when all my hard work is dismissed like that really hurts. Please have patience with us, we're real people on the other side of this screen and we really are doing our best.

@cloudguy: In response to your missing top contributors pod problem, that is caused by the way the wiki is designed, when you add a character (for example) to the wiki as a stub say from the issue page, the points you get are associated to the issue page you added him to.

For example this character has an empty contributor pod: http://comicvine.gamespot.com/zhong-lei/4005-131916/

That's because he was added as a stub in this edit to an issue page, therefore the points for creating him are counted on that page: http://comicvine.gamespot.com/wiki/moderation/1030430/

That's why the pod is empty. I know that's confusing and it shouldn't be that way, but that's why we are working on rewriting the whole wiki right now, so things start making sense. Now, if you have an example of a missing contributors pod to a page that isn't a stub let us know and we'll investigate but for now I can't seem to find any problems like that.

Again sorry for the delay, I'm trying my best to get my work done and respond to you guys.

@wcarle: I'm sorry but you have to realize that we've been getting jerked around with empty promises for at least five years now, five years of editors contributing to this site so that it can bring in more people and hits, all the work the community puts into this site and we get next to nothing in the way improvements to the site, the wiki barely gets acknowledged by staff period, we've lost tons of wiki editors over the years because of the wiki problems, we see engineers come and go and everything I said in that post and many others like it is true, I know it is a lot of work and I know you are not just sitting around but we've seen next to nothing when it comes to results over all these years, just look at the Bug Reporting forum Rories been MIA from there for what 3-4 months, it's been even longer then that since I've been able to get anyone other then you to reply to me on CV, via email or on twitter. If you want what I and others say to change we need to see visual results not just behind the scenes server changes, if things change we will change but until then this is where we are.

As for the top contributor box missing cloudguy wasn't talking about stub pages, we've known how those work for a couple years now, he means on regular pages of which there are still lots, here is an example page.

@pikahyper: Ok that page you linked is definitely missing points that we have a record for. I just talked to one of our devs who was going to run the command to update those, and he wanted to wait until the new points moderation features went out so we wouldn't have to run it again. That was a miscommunication on our end, he plans to run it either tonight when he gets home or tomorrow. I'll post here when it's done running.

As far as visual results, that's why we're doing the beta. It's been a while since we pushed a new build but that's because we are waiting on more UI work to be done (we are currently running on one designer for 3 sites.) You can see in the beta that we've rebuilt the backend completely and have totally redone the way the editing experience works. I know we're not moving fast, but we are moving.

and if you want to see a page where "Most issue credits" and "Most issue appearances" is still incorrect here is an example, each of the creators and characters should show 3 not 2, the one creator that shows 3 right now is cause he was missing and I just added him so that bumped him up from 2.

Ok cool thanks, one of our guys has been looking into redoing that association rollup process so we don't have to keep running these damn association recovery commands.

@wcarle: ok see that is one of the things I don't get and that no one has ever been willing to explain to me cause everything has to be so secretive, I get that you guys are a tiny team and you work on multiple sites, I know from experience that it is demanding, but I look at these sites and I see that none of them visually change, the features and visuals have basically been the same since we got bought and the platform changed, I also see that the Bug Reporting forums for the sites get no attention from the engineers or Rorie so we don't see you guys do anything, obviously you are doing things so what is the designer working on that's keeping him busy? I see you have your unknown projects and you are working on the beta, Rorie I guess is answering emails 9-5 every day and I have no clue what edgework does, so since the designer can't work on the beta what else is there? does he have to make the visuals for articles or do double duty editing videos? Are you guys stuck in like useless daily meetings? Chalk it up to me always being freelance and never in a corporate environment but is it really that different? The sites I build are much smaller but I still don't get very long to make them, I don't think I've ever had a web design project last more then six weeks and that was on a team with three other people, most of mine are solo and done in 2-3 weeks.

Seriously if you guys ever need example pages just let me know, I really do know this site better then anyone, I look at pretty much every wiki page that gets created and my memory is crazy good so I can find glitched pages in a snap.

@pikahyper: I get that you don't see many things change in the core functionality of the site and that's because these are very feature heavy sites already. Thinking about all the functionality of GB CV and GS makes my head spin sometimes. Not just user facing features and the wiki but all the behind the scenes CMS code, the special events like E3, too many features to name. I realize we move like the damn Titanic sometimes but there are so many plates we have to keep spinning sometimes those larger features and projects don't move as fast as we would like them. I really wish I could come in and do nothing but work on the wiki all day trust me, but the reality is that keeping the sites running has to be our priority.

@wcarle: that's understandable, that's one of the main reasons I stopped wanting to join the engineering team, not being able to work on what I want or what I see as important would drive me nuts. I wish you could have seen the old platform, it had its flaws but even by todays standards there is no better wiki/cms platform, we lost so many features and so much functionality that it just gutted the community, we had a hundred times as many wiki editors in those days.