Alpha 12 / HWS 12 Current State and Intervention

[RexXxuS]

Good Morning HWS Community,
after almost 2 weeks of our HWS 12 Reloaded season start I draw a line and make a summary.

Galaxy

As everyone else I was hyped about it - even though time was missing to fully utilize it.
Nevertheless I expected issues month ago already and came up with the Big Bang idea. Natural limitation and slow increase of Solar Systems along the way. With the intention in mind that if something goes wrong, we can stop the Big Bang expansion.

The Galaxy is hands down a Singleplayer Feature, that is for sure after everything I saw and analyzed. At least more than 1000 Stars or how Eleon just dropped it live.
With heavy customization within the Galaxy config files it might work in a better way but that needs a lot, a lot of time to do.

To be honest exploration, with some other Alpha 12 features is awesome in this game. If each Planet and space would look different. If on each planet and space would be different POIs and then on top of it, every time different Missions / Quest… it would blow our all minds.
Combined with some more Multiplayer additions like Territories for factions? Conquering whole Star Classes it would create that deep gameplay we are all expected to see for Alpha 12 already, maybe.
But no, some of that isn’t possible yet and the rest needs time, like a lot guys.
I love Empyrion as Owner cause no other game gives you so much possibilities to customize it for players. But 99% of customization in this game is manufacture work. No automatism really.

I am already working on HWS 13 and to at least give you 100 different playfields instead of the 10 right now.
Still, the current state is horrible and I hoped for patches but no, there is no wonder helping anymore it seems.
Why? Because it is technically not even possible. Let me show you:
The Template folders on HWS NA (if a Star is entered, all playfield.yaml files are loaded and written on the disk):

The Playfield folders on HWS NA (that many playfields [Planet/Moon/Space] were visited by you, the players so far):

On HWS EU it’s few thousands less.

Now, I’m not a Software Architect nor a Hardware Architect but skimming¹ the² web³, you get roughly the idea that the more folders / files you have the more strain for Windows it becomes. Especially if you work with them (making backups, if you fly over terrain, the area files needs updating inside those files, etc.)

Okay, so that are internal Windows issues with all of that Galaxy stuff.
On top Alpha 12 gave us Databases. A Database is like an optimized Container. You put stuff inside and it gets categorized, sorted, filtered - ideally on the right place…
Looking up for those sections inside the Database is called a SQL Query.
If you press B for example and check the Statistics who drilled the most Gold => it’s a SQL query, looking for that specific data in the Database.
And the more data there and the more players calling this SQL Query, the more time and performance it consumes.
In relation to the Galaxy, there is just a lot of data incoming, right.

I mean the Database is only after 2 weeks already 1,1GB big

It really just adds up in count, time, performance drain.

All of that tells me at least: nope, so many stars for MP is nonsense and I make the well-founded statement, that I tested it and does not approve it for HWS for the future.

So, that brings me to the forced reaction to stop the Big Bang expansion as explained above.

From the 450 Light Years radius we are currently at 144. That is about 4000 Stars from the 7000 Stars in total. There are still hundreds of Stars not explored yet. All Star Fragments will be put within those 4000 Stars, so you still get the chance to fully discover this new feature.

The moral of the story is really: better to have less but unique and interesting Stars, than thousands of repeating Stars, causing nothing than problems.

And yes, while this is a case for Captain Obvious, it was my responsibility to test it until the bitter end.

Overall I don’t want to do a full wipe though and or because I can’t tell you the details of HWS 13 about that yet.
I will do tests with 100-500 Stars but ideally each of them has a purpose. We will see!

Disconnects, Missing Ship (parts), Corrupt logins

And then there are still those issues we can’t hope for a fix either. In theory all issues I explained above might be the result to them but there are year old bugs, which are still in the game.
However we did some more tests and crawling and @Jascha found out that one issue is coming again from those CPU Process Weight System.
If for example ECC Sector, ECC and Binomi Sector all share one Process and suddently ECC becomes dead, then ECC Sector and Binomi Sector will be forcefully killed after 10 seconds. Even though the other 2 playfields were fine!! This is absurd and the whole Weight System becomes redundant.
Additionally, if one playfield gets killed, even though it was running fine, THAT is the reason why your structures sometimes have huge missing chunks in them! Or worse…

Today and tomorrow I try to forcefully set each playfield.yaml in the inner fixed HWS Galaxy to 100 PfWeight, so each of them get their own CPU process.
The main benefit of that in Alpha 11 and why we used it was because our HWS NA server ran out of RAM. But now… we don’t need to care anymore.

The corrupt logins and disconnects might get better with the next patch but I don’t know really. As I said I’m tired of spending 6+ hours a day in bug finding and doing a Developers job.
I finally want to balance HWS, improve HWS, create a HWS campaign and much more.

That being said improvements will come still but I hope you have the patience and endurance for a “lesson learned” HWS 13 season, where I clean up and make it better.
Sorry to those (including me) who had much higher expectations…

Round up

Last but not least you can expect all HWS Events coming this week/weekend as well as some NPC Trader returning and some other polishes.

Thank you all for sticking through this and be sure… we really give our best, day and night, to give you a great time on HWS!

Your HWS Team

[Fleshbits]

@RexXxuS

Now, I’m not a Software Architect nor a Hardware Architect but skimming¹ the² web³, you get roughly the idea that the more folders / files you have the more strain for Windows it becomes

I am a software engineer and have worked with Windows for two decades. I don’t know if I can agree that simply having a lot of files and folders effects system performance at all. The link is vert very old. It all depends how EGS is accessing them and how often, really.

That said, I do think that the majority of planets are a waste of space and resources. I see maybe an interesting planet every 200 jumps. There simply is no point in having so many that people will spend less than a few minutes on.

What the heck is the point of the “empty space” playfields too? Why did Eleon even generate those? I thought maybe I was missing something new.

I was hoping handcrafted playfields would be injected into the galaxy as it expands, so we could reignite that sense of exploration. Sounds like next season will have them. I am sure its a ton of work.

Databases are a whole different story. It takes really smart people to get good caching and optimizations for a database that will be on a multiplayer game with more than a hundred people. I can imagine it is a mess.

I hope things make sense next season. Thanks for working so hard and keeping us up to date Mr Rexxus!

[SwissSteel1]

Thankyou for taking time to, and providing in-depth explanation.

[Ligator_swe]

The galaxy is a wonderful idea. But we need custom playfields in it. At the “six corners” of the universe. With tradegoods and resources and reasons to FLY there (i personally am not so fond of teleporters, but that is how eleon has made the game).

Trade and travel between the center and the outskirts.

That would make the universe less “single player”

[Wiseman738]

I have some interesting ideas for a much smaller and more intense kind of ‘expansion’ which will force players to interact together, will share it with you once the major issues subside.

But I for one am super grateful you tried to experiment with this idea Rexxus. It is really appreciated and your efforts have helped to bring GMC, ACM back into the game as a result!

Best regards,
Wise.

[Wiseman738]

Also in the spirit of greater customisation, might be worth a competition to crowd-source more custom POIS and perhaps even playfields? We have a huge community and I’m sure many would LOVE to participate in helping you customise systems, playfields, and POIs further!

[Fallen_Angel]

It might be faster for Rex and his team to create playfields than make sure that community created playfields don’t contain anything exploitable.

If it’s somewhat efficient to properly check playfields, then this idea is really good. I know at least @Altimus would love this opportunity to create content.

[Wiseman738]

Yeah I doubt anyone would be stupid enough hopefully to submit exploitable and/or broken playfield files. Obviously if they did then I imagine then they would be barred from future crowd-sourcing contributions.

And yes I for one would be interested to get to know the software for making custom playfields and scenarios. As Rex does most of it solo, player contributors could help form part of that ‘team’ to make our HWS even more special than it already is!

Best regards,
Wise.

[Achilles]

For those of you that might be interested in creating custom playfields keep and eye out for the updated version of the EPD tool here

[TOOL] Empyrion Playfield Designer Alpha8+ v1.95.5 EXP

back in the day we used to just manually manipulate the YAML files but over time as things have been added it can be quite taxing to do this, also it can be a bit tricky if you haven’t had too much experience with it.

hopefully the playfield tool will be updated soon and then people can have a blast.

in the meantime if any one knows how to/enjoys making playfields feel free to send to me via discord and i will happily test them and if they are cool/quality tested we can look at the possibility of having them for HWS 13.

HWS has got permissions to use playfield designs/concepts from project Eden so you wouldn’t need to submit those type of playfields, i know a few people are familiar with this scenario.

[SwissSteel1]

About the OCD. (TL:DR — it is Swiss, no short version.)

Yes, looking at OCD quantities may appear many are “rich” and can afford ships, etc. in a “lean” resource universe. This is a flawed conclusion.

For most, what you see in OCD required multiple seasons to accumulate. It is easy to burn through more resources from the OCD than can be replaced by small factions in (especially this) current season. Gold is being saved for next OCD upgrade.

Many players use their OCD for fast season start, and ready reserve for operations. Only using from OCD what they anticipate can be replaced by season end. I consider this to be “best practice” use of OCD.

Sustaining game play from the OCD will use up OCD resources faster than they can be replaced. A player will run out of key resources in the OCD quite quickly. Ending the season with less in OCD than held at the beginning.

(Example: My PVP CV requires 85k XENO substrate. Yes, I could spawn one from my OCD. However, my approach is to 1st obtain the needed XENO in game. Reserving the XENO in OCD for repairs. Using this approach, increases the likelihood I’ll end the season with at least as much as I began. The grind for 85k XENO substrate this season is in my opinion, too steep.)

If I 1st spawn the CV from my OCD, very good chance will end the season with far less XENO than begun. Especially if the CV is lost, which is very likely given the mega alliances/factions it will face. Therefore, I likely will not bring in the PVP CV this season to offer up some CV PVP.)

Severely limiting the necessary resources in game reduces the amount of PVP the small factions are able to offer up to the mega alliances/factions.

[420Inc_Space_Ranger]

We’ll be here. Waiting…

[aeonbug]

@RexXxuS I agree with you that the large Universe is not working because of the limitations in the game software. On the same subject it seems that the game software has become even worse at handling small PvP fights as well. Yesterday we (AAA and GMC) went to Golden Globe to have a fight with RED and JVI. I would estimate that at the height of battle there must have been about 15 vehicles (mainly HVs) in the battle area and 3 bases. This battle caused 10 to 15 minute lag in the battle area and untold lag elsewhere. In the end the GG playfield crashed. I would think that it would be a good idea if Eleon were made aware of this appalling state of affairs in their game so that they might actually be prompted to do something about the whole game performance issues. This game has so much potential that it would be a real shame to lose it because the developers are hiding their heads in the sand over its performance shortcomings.
As for you @RexXxuS I would suggest that you take some time off before you make yourself really ill. You are doing a good job, but do not make yourself unwell over it.

[Zackey_TNT]

@Fleshbits

@RexXxuS

Now, I’m not a Software Architect nor a Hardware Architect but skimming¹ the² web³, you get roughly the idea that the more folders / files you have the more strain for Windows it becomes

I am a software engineer and have worked with Windows for two decades. I don’t know if I can agree that simply having a lot of files and folders effects system performance at all. The link is vert very old. It all depends how EGS is accessing them and how often, really.

Hi,
I’m not a software engineer or really anyone with real work experience, but I do have a bachelor’s in network engineering, and I suspect that certain SAN topologies can slow performance when accessing a large amount of files.

It was mentioned that we have NVMe drives on both servers, but there’s no way to ensure that it’s actually on host and with the dedicated platform. Most cloud platforms centralize their storage for easier allocation of hosts, and it could be accessed over the network.

If it is accessed over the network, there may be a chance (through incompetence or other) that we’re not getting desirable performance from random read/write requests when it comes to 30,000+ files/folders.

Looking at your spec page, I see some weird storage termonlogy going on such as “SoftRaid” and “INFRA-3 Storage” which indicates to me that they are doing something other then simply presenting real disks to your server. Perhaps clarify with your provider how they are presenting the disks to your machine.

I would like to propose you investigate this, and maybe look into disabling “last accessed attribute for NTFS” if its enabled. By default on windows 10 it’s disabled, but you can check to ensure this via (note that it may not be disabled on server additions, due to the added file integrity it offers): Enable Last Access timestamp for files and folder in Windows
(should still work in server 2016 under command prompt)

Ask your provider how they are doing their disks and if your usecase is a potential bottleneck for their system. Usually cloud hosts are happy to discuss this because they can also usually offer a solution.

[mgaag]

Biggest performance killer still is badly tuned database queries. After performance tuning these kind of queries gains of 20 x or more are not rare. Bad indexes, no indexes or stale statistics etc can slow down performance a lot and you will perceive lag if this is the issue.

[TacoIsland]

All lies, I bet Eleon made their database in Access97 with all the glamorous flaws it brings. And yes, large amounts of files\folders in a folder itself does cause a degradation of performance in certain situations, temp folders are notorious for slowing system performance in extreme cases.

[Bob]

awesome discussion on this. the developers need to be included with a conversation of this magnitude. too bad we can’t clockwork orange the dev team into fixing their mistakes

[sacredglade]

I believe one of HWS principle attractions, has always been its very different experience from vanilla.

So I would agree with others that a small (or smaller), highly custom game world is far more attractive for HWS than vanilla’s massive procedural generated (early No Mans Sky like) universe.

However I am not so sure about community designed playfields as such. As I feel the hardwork and play value is in creation of unique POI’s/missions/instances. A pretty playfield is great, but if filled with more of the same stock vanilla POI’s, then the experience will feel vanilla very quickly.

But perhaps a more top down led community design system could work. So if Rex or another HWS “staffer” could post a POI “wish list” (need some Egyptian style constructs for a desert world etc) , then community builders can sign up to help fill the playfield.

Or players can submit ideas on POI/Instances/missions they would like to work on, again HWS team can advise/plan for the use, ask others to contribute more in the same theme etc.

Last thought, Rex always adds unique “gameplay features/mechanics” to HWS seasons. I would suggest that for the next season, no new features are required. Building out a custom designed universe that works in a multiplayer setting, is “where it’s at”

PS: its is great to hear that Project Eden playfields can be used in HWS, I think Raivens one man output of unique designs has been awesome.

[Wiseman738]

That’s what I had in mind. Custom playfields to me have to have custom POIS as well in order to remain interesting. After all, there’s only so much we can do with sliders and such! In regards to setting a theme, this idea has merit, but i’d prefer it if it was less exclusively focused initially so people could fully let their imagination run wild.

Perhaps there could then later be a sub competition for more specific POI requests akin to what Rex did in the past by asking for trade station and alien station POIS.

For me this exercise is all about unlocking the creativity of the HWS playerbase, all the while having some sensible restrictions such as not having a planet with an all-gold surface or exclusively utilising gold deposits and such!

Either way, great to have you back and posting on the forum again Sacred!

Best regards,
Wise.

[TacoIsland]

Tempting to remake Darkworld, near pitch black in the day and heavy fog as the only weather option!

[Godwayz]

I can not relate to be fair - I was having the best time ever with this update and Universe - The official server offers a small or smaller experience which is why it’s populated most of the time. I welcome the new idea rex implemented and do not welcome going back to the smaller experience which just makes PVE junk. Having space and a balance that encourages PVE which leads to PVP is the best experience your going to get.

One of the pulls of HWS is the OCD feature - remove that and with a smaller galaxy then people will look elsewhere if there was a better option. I think this was was in the right direction. On this news - I’ll pack up and go to official where I wont get wiped until A13 but at least they won’t make it any smaller and already has a custom experience.

Much of what HWS has had are Payed prvt spaces and planets, I don’t wanna go back to a restricted pay to play set again where the grind is tricky and the place is just darn well small.

Just my opinion but ya