Update on roll back issue - 10/16 - **FIXED**

[Ryan Pennington 359]

UPDATE (7/13 9:23PM) Temporary fix has been fully implemented. Please see the news article for more information:

 

Rolled out a fix (no pun intended) for rollback issue for ARK: Survival Evolved. Please restart your game server ASAP for these changes to apply. A new variable has been appended to launch parameters - "?AltSaveDirectoryName=SavedArkXXXX" where XXXX is your game server ID. This should prevent executable attaching / locking on to TheIsland.ark files.

 

Edit: It seems this fix is not fixing it for 100% of customers. Still researching.

 

UPDATE (7/14 1:31PM)

 

Regarding the ARK data loss, here's the current / latest scoop:

 

There was an issue with the ARK database restore which was inserting the data into the SavedArks folder instead of the new SavedArkXXXX (where XXXX = your game server id) format from the last night fix. It has been resolved.
​
​After the fix from last night, you may have tried to restart your game server to apply the fix. The fix has been applied successfully. This does not mean that your saved ARK data up to that point is safe from roll backs, it means that future restarts should be safe from the roll back issue. If there is data loss up to the point in which you restarted, that would be from the roll back issue and unfortunately is not likely restorable. You should be able to restore a backup now but the data from within the roll back bug time frame is likely not restorable.
​
​I apologize for this issue. When the developers created ARK they did not forsee the issue of TheIsland.ark file being locked by other processes which caused this roll back issue. I've been working directly with the dedicated server developer from ARK and he is aware of the issue and working on a fix. He was also the one who gave me the workaround (the ?AltSaveDirectoryName=SavedArkXXXX) and we are hoping that this is a permanent resolution to the issue.

 

UPDATE (7/14 1:31PM)

 

Please note that this is an isolated issue to a small set of customers and does not have to do with the hardware or anything. Most of our customers have not had roll back issues before or after the fix.

 

UPDATE (7/29 2:28PM) 

 

What I've found out, as of today, is that your TheIsland.ark file (which has all of your data) is being locked by another process on the dedicated server. I have not pinpointed as to WHY this is happening, but it is blocking TheIsland.ark file from writing and creating a TheIsland.tmp file instead with your most recent saves.

In the mean time, a fix is for our staff to come in and remove the file lock from the process that is blocking your TheIsland.ark file and the worldsaves will properly work.

Step #1: Disable automatic restarts, set it to NEVER

Step #2: Check if there is an existing block on your TheIsland.ark file

To check this, you can open up the ShooterGame\Saved\SavedArkXXXX (where XXXX is your game server ID) folder and check to see if a TheIsland.tmp file exists in this folder **WHILE IT IS RUNNING**

** If it does exist, create a support ticket and do not restart your server, we have to remove the lock before you restart or run an ARK update.. after we remove the lock it will function normally

** If it does not exist, you are good to go and can perform a server restart or ARK update.

I understand the frustration here but this is our current best method so that you don't lose any data. =)

 

UPDATE (8/5 5:48PM)

 

I've implemented a more permanent fix that will close any TheIsland.ark locks before a server is stopped and it will take the greater file size of TheIsland.ark vs TheIsland.tmp and rename to TheIsland.ark. This should result in no data loss. This fix has also been applied to our backup script. Crossing my fingers, but I think this is a big step forward.

 

UPDATE (8/7 2:51PM)

 

We've gone through a few update cycles and we've yet to see a single roll back after 8/5. I am not completely ruling it out, but this is a huge considering the last few weeks of issues around update cycles.

 

UPDATE (8/11 7:35PM)

 

.tmp check now when server is stopped as well as start / restart, in an effort to curb the roll backs during ARK Updates.

 

UPDATE (8/12 5:45PM)

 

.tmp check now when server is backed up during routine backup procedure.

Located a major bug that was causing the fix from 8/5 to not work 100% of the time.

 

UPDATE (8/13 1:27PM)

 

There was some TheIsland.ark handle closing hit misses on the roll back script fix from yesterday. In order to pinpoint the exact code change that needs to happen, I have implemented a feature that will completely "lock" your server if the TheIsland.ark lock cannot be removed, that way a roll back should not (in theory) occur. 

 

UPDATE (10/12 5:07PM)

 

There are still some people getting this issue. The number is not big but it is present. I want to get it down to zero.

 

A fellow server owner recently told me that he was getting the same issue and changed some of the Windows security permissions on the folder of the saves. I have made this change on several customer servers that have had recent roll backs and will be closely monitoring their servers. So far no roll backs, but it's not as easy as that (requires more time to validate the data)

 

That said, I will be implementing the folder permissions "fix" (even if it may or may NOT work, that will be my BIG disclaimer here) across the board later today. Let's hope for some success! 

 

EDIT 11:23PM - fix implemented.

 

UPDATE (10/16 12:31PM) -- LATEST UPDATE!

 

No roll backs reported. There are Data Corruption crash messages occasionally when ARK is updated, a mod is updated, or something breaks on the server - but this is not a "roll back" as described in this thread.

[Ryan Pennington 359]

I've unlocked this thread for discussion.

[typeForced 0]

I have a support ticket open and would like a response to it please.

[atatsu 1]

Just letting you know your file lock fix appears to be working. Earlier this morning I checked the state of my server and the `TheIsland.tmp` file was present and the `TheIsland.ark` file hadn't been modified since 7/29. I backed up the tmp file just in case. 

 

Fast forward to now and the v191 udpate and the tmp file was gone, and I updated the server to v191 successfully.

[Proxus 1]

Just had this issue happen to our server.

 

Ticket #723412

[zanith86 1]

This happened to our server as well.

Please help.

Ticket #561058

[zanith86 1]

The tech help hasn't helped at the moment, as all he said was to do a data-restore. Even though all the recent file saves are going back to July 30th, 2015 1:40 AM CST.

[typeForced 0]

Issue has been going on for near 4hrs, ticket for over 3.

 

Still no response.

[zanith86 1]

Still no response regarding this issue. I have posted this issue at 5:00 PM CST and the issue still has not been resolved.

Update:
It seems that all the saves that occurred in the last two days, were just duplicating the final save that was made on July 30th, 2015 at 1:40 AM in the morning. After that, they're all duplicates of that save...Unless there is another way to fix this my server randomly lost 2 days worth of data? 

[Ryan Pennington 359]

Just letting you know your file lock fix appears to be working. Earlier this morning I checked the state of my server and the `TheIsland.tmp` file was present and the `TheIsland.ark` file hadn't been modified since 7/29. I backed up the tmp file just in case. 

 

Fast forward to now and the v191 udpate and the tmp file was gone, and I updated the server to v191 successfully.

 

It's not necessarily a permanent fix but I am glad to hear that it is working.

 

We are going through all of the tickets right now. You all should receive a response shortly.

[Eltopo 3]

Whats the right way to use a save file after this? Update settings or restart with the date and time of the save selected?

[GillebLand 0]

Had this happen to us last night @ about 2:30AM EDT. Server reverted about 6 hours back, tried 3 different recent backups and all load the same one. I'd post our ticket # but I've tried twice and still haven't received the email with it.

[Ryan Pennington 359]

Had this happen to us last night @ about 2:30AM EDT. Server reverted about 6 hours back, tried 3 different recent backups and all load the same one. I'd post our ticket # but I've tried twice and still haven't received the email with it.

 

 

Whats the right way to use a save file after this? Update settings or restart with the date and time of the save selected?

 

All tickets are up to date.

[Bluvian 1]

Today I update my server to v193 with no problem but it did rollback to about 2 days ago... backup restoration seem not work at this moment... I did few time on backup but it still going rollback to 2 days ago... I really need get this server fixed... Server #7621 

 

Support will take so long time to respond so I need someone to fix this backup or stop the rollback please

[Ryan Pennington 359]

Updated this thread to reflect a new fix that I implemented today on every machine.

 

Still in testing, but this is a very promising step towards full resolution.

[Bluvian 1]

Ryan... this does not help with my rollback because the backup is not working for my server... how can I fix it or can you please fix it please?? 

 

Server ID: 7621

[Ryan Pennington 359]

This does not fix any issues before around 5PM today, if there was a roll back before then or a backup file that did not have accurate data to that date, this would also not apply. This is for future roll back avoidance and future backup accuracy.

[Bluvian 1]

My server got rollback twice tonight... Thought you fixed it? The backup is not working as well.

[Litup 0]

My server got rollback twice tonight... Thought you fixed it? The backup is not working as well.

Same Here, I noticed with FileZilla that when ark does its auto save it is saving to a new file called TheIsland.tmp instead of saving to TheIsland.ark. So I stopped the server, Downloaded the save folder and tried to delete the files on the server thru filazilla, I can delete all files on the server except for TheIsland.ark

 

Something with the server isnt allowing TheIsland.ark to be overwritten nor deleted and this is the main save file. I have my server in NC if I move it to dallas I am able to delete this file no problem but if i move it back to NC TheIsland.ark file is still in the folder unable to be deleted.

 

Command:    DELE TheIsland.ark

Response:    501 Internal error.

[hawk20011 0]

Is it possible a minor revision messed up the fix Ryan had implemented.

Everything seemed to be working for a bit there.

[Ryan Pennington 359]

It happens every time an ARK update happens and restoring data before the ARK update seems to be problematic.

 

This is affecting a lot less users than last time but it is being checked into.

[Bluvian 1]

Every time I restart server or update server... it always rollback to specific time. Server #7621 ... can you please stop that rollback or fix it please? 

 

This is 8th rollback now since I start this server...

[DiB 0]

Yeah, my server #7467 updated and rolled back like 8 hours and I didn't know until way too late to rollback to a proper backup.

[Ryan Pennington 359]

Yeah, my server #7467 updated and rolled back like 8 hours and I didn't know until way too late to rollback to a proper backup.

 

When was this?

[DiB 0]

 

 

When was this?

 

Happened at 7:15-7:30ish this morning right when the patch came out PDT. Gotta open a ticket now for that .tmp file.

[Ryan Pennington 359]

Updated the topic with the latest changes.

[Donelosh 0]

Had a rollback happen to me twice today. One rollback happened this morning was almost a 48 hour rollback, and the other about 6 hours. Server #7964.

[ChiliMag 1]

Happening on server 6533 for about the last 10 days.  I submitted a ticket but don't have a ticket number yet.

[Lancer88 0]

6821 huge roll backs as well

[Blitzking 0]

My server 8396 keeps rolling back after updates to the same time like two weeks ago.  I submitted a ticket days ago about this and I still don't have any response to it.