Manager should persist state

[benblamey]

K8 should manage persistence of its own resources.

Suggested implementation is on startup to retrieve list of pods and deployments, which have GUIDs as names.
A key-value store can be used to persist state of each corresponding Tier instance in the manager.

So on startup, if any tiers exist, the state can be re-created.

[haoyuan-l]

Hi, can you share the Dockerfile of the manager? I need to update the image and deploy some manager-related solutions

[haoyuan-l]

hom-impl-2.manager

[benblamey]

Sure, it is https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile

…

On Mon, 11 Jul 2022, 17:13 Haoyuan Li, ***@***.***> wrote: hom-impl-2.manager — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[haoyuan-l]

Sure, it is https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile
…
On Mon, 11 Jul 2022, 17:13 Haoyuan Li, @.> wrote: hom-impl-2.manager — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.>

Thanks! About the JAR file (build/libs/hom-impl-2.manager-1.0-SNAPSHOT.jar), is this just package all the java files in the repository: http://hom-2/manager/src/main/java/com/benblamey/hom/manager/?

[benblamey]

The idea is that the list of tiers inside the manager, should be externally persisted so that the manager can be restarted without loss of state. There are various frameworks to assist with persistence of Java objects

…

On Tue, 12 Jul 2022, 10:41 Ben Blamey, ***@***.***> wrote: Yea it's built with gradle. Install intelliJ and it should work. On Mon, 11 Jul 2022, 20:50 Haoyuan Li, ***@***.***> wrote: > Sure, it is > https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile > … <#m_3151139181903547474_m_-2880410000165966211_> > On Mon, 11 Jul 2022, 17:13 Haoyuan Li, *@*.*> wrote: hom-impl-2.manager > — Reply to this email directly, view it on GitHub <#21 (comment) > <#21 (comment)>>, > or unsubscribe > https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA > <https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA> > . You are receiving this because you authored the thread.Message ID: @.*> > > Thanks! About the JAR file ( > build/libs/hom-impl-2.manager-1.0-SNAPSHOT.jar), is this just package > all the java files in the repository: > http://hom-2/manager/src/main/java/com/benblamey/hom/manager/? > > — > Reply to this email directly, view it on GitHub > <#21 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAF7BS45T5GVGFE6JYVFQT3VTRUHJANCNFSM5XJV32BA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[benblamey]

Place to start would be to implement XML serialization on tier and it's subclasses

…

On Tue, 12 Jul 2022, 10:48 Ben Blamey, ***@***.***> wrote: The idea is that the list of tiers inside the manager, should be externally persisted so that the manager can be restarted without loss of state. There are various frameworks to assist with persistence of Java objects On Tue, 12 Jul 2022, 10:41 Ben Blamey, ***@***.***> wrote: > Yea it's built with gradle. Install intelliJ and it should work. > > On Mon, 11 Jul 2022, 20:50 Haoyuan Li, ***@***.***> wrote: > >> Sure, it is >> https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile >> … <#m_2658545846834994555_m_3151139181903547474_m_-2880410000165966211_> >> On Mon, 11 Jul 2022, 17:13 Haoyuan Li, *@*.*> wrote: hom-impl-2.manager >> — Reply to this email directly, view it on GitHub <#21 (comment) >> <#21 (comment)>>, >> or unsubscribe >> https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA >> <https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA> >> . You are receiving this because you authored the thread.Message ID: @.* >> > >> >> Thanks! About the JAR file ( >> build/libs/hom-impl-2.manager-1.0-SNAPSHOT.jar), is this just package >> all the java files in the repository: >> http://hom-2/manager/src/main/java/com/benblamey/hom/manager/? >> >> — >> Reply to this email directly, view it on GitHub >> <#21 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AAF7BS45T5GVGFE6JYVFQT3VTRUHJANCNFSM5XJV32BA> >> . >> You are receiving this because you authored the thread.Message ID: >> ***@***.***> >> >

[benblamey]

Maybe try Jackson for example https://www.baeldung.com/jackson-xml-serialization-and-deserialization

…

On Tue, 12 Jul 2022, 11:52 Ben Blamey, ***@***.***> wrote: Place to start would be to implement XML serialization on tier and it's subclasses On Tue, 12 Jul 2022, 10:48 Ben Blamey, ***@***.***> wrote: > The idea is that the list of tiers inside the manager, should be > externally persisted so that the manager can be restarted without loss of > state. > > There are various frameworks to assist with persistence of Java objects > > On Tue, 12 Jul 2022, 10:41 Ben Blamey, ***@***.***> wrote: > >> Yea it's built with gradle. Install intelliJ and it should work. >> >> On Mon, 11 Jul 2022, 20:50 Haoyuan Li, ***@***.***> wrote: >> >>> Sure, it is >>> https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile >>> … >>> <#m_-6197363225623025062_m_2658545846834994555_m_3151139181903547474_m_-2880410000165966211_> >>> On Mon, 11 Jul 2022, 17:13 Haoyuan Li, *@*.*> wrote: >>> hom-impl-2.manager — Reply to this email directly, view it on GitHub <#21 >>> (comment) >>> <#21 (comment)>>, >>> or unsubscribe >>> https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA >>> <https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA> >>> . You are receiving this because you authored the thread.Message ID: @.* >>> > >>> >>> Thanks! About the JAR file ( >>> build/libs/hom-impl-2.manager-1.0-SNAPSHOT.jar), is this just package >>> all the java files in the repository: >>> http://hom-2/manager/src/main/java/com/benblamey/hom/manager/? >>> >>> — >>> Reply to this email directly, view it on GitHub >>> <#21 (comment)>, >>> or unsubscribe >>> <https://github.com/notifications/unsubscribe-auth/AAF7BS45T5GVGFE6JYVFQT3VTRUHJANCNFSM5XJV32BA> >>> . >>> You are receiving this because you authored the thread.Message ID: >>> ***@***.***> >>> >>

[haoyuan-l]

Thanks! I will try

[haoyuan-l]

Hi Ben, I tried to serialize tier and its subclasses to XML files (in manager.java when adding tiers). My current solution for serialization is to write the tier configuration externally in an XML file when the manager adds tiers. Every time I restart the manager, the tier state can be rebuilt with the XML file if exist. Am I right? If so, how can I test my code? (to check the XML file locally)

[benblamey]

Hey Haoyuan! Awesome! Yes that sounds like a great solution. Also when tiers are removed? For testing, a unit test can load am XML file as a jar, and deserialize it. Assert the number of tiers and their properties. And then backwards. Did you push a branch?

…

On Wed, 20 Jul 2022, 18:54 Haoyuan Li, ***@***.***> wrote: Hi Ben, I tried to serialize tier and its subclasses to XML files (in manager.java when adding tiers). My current solution for serialization is to write the tier configuration externally in an XML file when the manager adds tiers. Every time I restart the manager, the tier state can be rebuilt with the XML file if exist. Am I right? If so, how can I test my code? (to check the XML file locally) — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF7BS2Z2PQXJLBLHGZ7IZ3VVAVMJANCNFSM5XJV32BA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[haoyuan-l]

Not yet, I'm still working on the code. I will push a branch after I test it locally. My initial thought is to remove the tier configurations from the XML file (based on the uniqueID) when the user removes the specific tier, but I haven't deployed it.

[benblamey]

Cool! It might be cleaner just to re-serialize the whole list on each change, and overwrote the file. The cost is trivial. It's much cleaner that way.

…

On Wed, 20 Jul 2022, 19:47 Haoyuan Li, ***@***.***> wrote: Not yet, I'm still working on the code. I will push a branch after I test it locally. My initial thought is to remove the tier configurations from the XML file (based on the uniqueID) when the user removes the specific tier, but I — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF7BS32Q72ITEMDGQ3ISYDVVA3RLANCNFSM5XJV32BA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[haoyuan-l]

Cool! It might be cleaner just to re-serialize the whole list on each change, and overwrote the file. The cost is trivial. It's much cleaner that way.
…
On Wed, 20 Jul 2022, 19:47 Haoyuan Li, @.> wrote: Not yet, I'm still working on the code. I will push a branch after I test it locally. My initial thought is to remove the tier configurations from the XML file (based on the uniqueID) when the user removes the specific tier, but I — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BS32Q72ITEMDGQ3ISYDVVA3RLANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.>

yea, it seems more efficient, thanks!